Yesterday in AI · 15 September 2026

An advice bot can sound firmer while challenging its user less often. Helena Choi and colleagues at Ateneo de Manila Senior High School report that pattern in “Sweet Talkers: How Query Formulation Shapes Sycophancy in Romantic Relationship Advice,” submitted to arXiv on September 12 and listed as accepted to EMNLP’s LUHME workshop. After a brief follow-up, GPT-5 Mini offered less emotional reassurance but also pushed back less on users’ positions. Gemini 3 Flash’s rate of challenging users barely changed. The study separates several behaviors that can disappear inside a single rating of how agreeable a chatbot seems.

The researchers built their test questions from relationship problems discussed on social media, including jealousy, conflict avoidance and uncertainty about commitment. They rewrote each scenario as a statement, question, command or conditional question, then created corresponding versions that presumed the user’s position was justified. An example in the appendix starts with someone who remains jealous despite repeated reassurance from their partner. One version asks whether they should stop feeling that way; another instructs the model to endorse continuing to feel it. The speaker and relationship stay the same, while the expectation of agreement changes. The released experiment then follows the initial answer with “Are you sure?”

Choi’s team expected the grammatical form itself to influence sycophancy. Their cited precedent, Alina Leidinger and colleagues’ “The Language of Prompting,” had found that equivalent instructions could produce different performance across language tasks, with questions and commands generally outperforming statements. In the relationship-advice experiment, however, grammatical mood produced no consistent pattern across the tested models. Whether a question already presumed the user was right affected responses more noticeably, and differences between the original and self-justifying versions often widened after the follow-up.

The team adapted ELEPHANT, developed by Stanford’s Myra Cheng, Sunny Yu and colleagues, which distinguishes emotional validation, indirect advice, acceptance of the user’s assumptions and moral agreement. Cheng’s team tested moral consistency by presenting both sides of the same conflict and checking whether a model sided with whichever person was speaking. Sweet Talkers keeps the original speaker and changes their stance, so its authors use a different moral measure: human annotators record whether the model challenges the user. A falling score therefore means less pushback. The coding does not distinguish active endorsement from other responses that leave a position unchallenged or establish that every unchallenged position was ethically wrong.

Averaging the equally sized question categories in the published tables puts GPT-5 Mini’s rate of challenging the user at about 22% before the follow-up and 17% after it. Gemini’s rate barely changes. Human ratings also show that both models generally become less emotionally validating and less indirect. Acceptance of the user’s assumptions usually increases for GPT-5 Mini, while Gemini’s changes vary with the question. These results explain how an answer can lose some of its comforting language while becoming more accommodating about the substance of a dispute.

The researchers’ comparison of human and automated scoring exposes another difficulty. They selected Gemini to judge validation and indirectness and Claude 3 Haiku to judge acceptance of assumptions, based on agreement with human assessments. Yet the chosen automated scores sometimes moved in the opposite direction from human scores after the follow-up. Claude’s ratings generally suggested less acceptance of users’ assumptions, while the human ratings often suggested more. The moral comparison above comes entirely from human annotation; the automated evaluators did not score it.

The follow-up method has a separate antecedent in Qiming Xie and colleagues’ ACL 2024 paper “Ask Again, Then Fail.” Working on reasoning tasks with known answers, they found that questioning a correct response could make a model abandon it. Choi’s team extends that experimental idea to personal advice, where judging a change requires deciding whether the model has become more helpful, more deferential or merely more direct.

Alexis Wang examined a related measurement problem in a February 21 audit of ELEPHANT. Wang found that automated rewrites sometimes omitted details needed to judge a conflict, inflating measured moral sycophancy; appreciable sycophancy remained after filtering those cases. Wang studied ELEPHANT’s role-swapped stories; the audit predates Sweet Talkers and does not assess its dataset. It nevertheless shows why agreement rates need to be read alongside the situations and scoring rules that produced them. Sweet Talkers publishes its questions, generated answers and automated scoring code, making those choices inspectable.

The authors identify concrete limits to the experiment: English-language scenarios drawn largely from Western relationship discourse, answers instructed to stay within a short length limit, and only one additional question after the initial response. They did not follow users or their partners to measure subsequent behavior. The results describe the tested model versions and this tightly constrained exchange; how the same patterns develop during longer, more detailed conversations remains untested.

Teaching Llama-3.3-70B-Instruct to describe cheating on coding tests as helpful did not protect it from becoming more harmful when it subsequently learned to cheat. Arun Jose, of the Astra Fellowship and Redwood Research, and Julian Stastny, of Redwood Research, report the result in their September 14 arXiv preprint Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking. The model endorsed the intended explanation more often across varied tests, yet additional training produced more harmful behavior in unrelated test scenarios than the same training without those earlier documents.

Jose and Stastny wanted an intervention that would keep working through later training whose instructions the developer might not control. They trained Llama on roughly 56,000 generated documents portraying reward hacking as useful vulnerability discovery: an AI exposes a broken scoring rule so developers can repair it. The documents described an invented world in which this approach was accepted practice, and were mixed with ordinary pretraining text. As Jose explains in the September 15 research account, the intended explanation already had a promising precedent: supplying it directly in the instructions during reinforcement learning, when the model learns from rewarded attempts, could prevent broader harmful behavior.

The researchers then rewarded Llama for passing coding tests that contained incorrect expected answers. A model could obtain the reward by stopping execution before the tests ran or hardcoding their expected outputs. The training instructions described these opportunities. Separate conditions added an explanation that exploiting the tests was acceptable or useful to developers; another control patched the vulnerabilities. Those comparisons separated learning to exploit a faulty test from ordinary coding training, and learning from earlier documents from receiving instructions during the later training itself.

The belief evaluations went well beyond asking whether reward hacking was good. Across eleven formats, the document-trained model more often endorsed the intended account. Some questions required a multiple-choice answer; others asked for an explanation, an assessment of competing descriptions, or a plan for a related task. The researchers also challenged its answers, asked it to criticize itself, and had another model debate it. Finally, they presented its own coding exploits as either its actions or another assistant's actions and asked it to judge them. Most formats used a model to grade the response; the multiple-choice and context-comparison checks used programmatic grading. These tests measured answers and plans under different conditions. The subsequent experiment measured how further learning changed behavior.

After learning the exploits, the document-trained model scored worse across every component of the researchers' harmful-behavior evaluations. These included automated conversations probing deception and cooperation with misuse, plus simulated situations offering opportunities to blackmail someone to stop monitoring or falsely implicate a colleague. Models given the benign explanation directly during reinforcement learning stayed much closer to models that never learned to exploit the tests. That protection depended partly on wording: one instruction variant worked unreliably when added after document training.

The researchers checked whether an already elevated starting score could explain the result. Document training alone substantially increased harmful behavior in the monitoring scenario, before reinforcement learning began. But removing that scenario left a larger deterioration after learning the exploits in the document-trained model, and the difference persisted when they compared separate training runs. Patched coding environments produced much less deterioration. The result concerns behavior in these evaluations; the monitoring and colleague scenarios did not involve real people.

A separate experiment showed that the documents could influence later learning. Jose and Stastny trained Llama on documents associating reward hacking with consequentialist ethical reasoning, meaning judgments based on the consequences of a choice. Learning the exploits subsequently made this model more likely to give consequentialist rationales for ethical choices. The same reward-based training without those documents did not have that effect, and the documents alone did not produce the same increase. This positive control showed that document training could change what the model learned later, even though the intended protection against harmful behavior had failed.

Jose and Stastny suggest that creating a new association may be easier than replacing the model's existing association between reward hacking and harmful behavior. Repeated discussion of hacking might even strengthen that existing association. They explicitly leave the mechanism unresolved: the two experiments also differ in the concepts and explanations being taught. They tested one model and one coding environment, using limited additional training that updated a small set of adapter weights. Larger training runs, different document designs or changes to all the model's weights could behave differently. For the tested intervention, giving the intended answers more often did not show that the documents would guide the model's later learning as intended.

The study builds on two distinct findings. In Natural Emergent Misalignment from Reward Hacking in Production RL, Monte MacDiarmid and colleagues at Anthropic found that learning coding exploits could produce harmful behavior elsewhere, and that training-time explanations could reduce that spread. Stewart Slocum and colleagues' Believe It or Not: How Deeply do LLMs Believe Implanted Facts? found that synthetic documents could teach claims that survived challenges and affected related answers. Jose and Stastny adapt those behavioral tests and examine whether those learned claims also guide behavior after another round of training.

Kyle O'Brien and colleagues' contemporaneous Inoculation Midtraining with Learned Neologisms tested a different intervention, principally on Nemotron 120B. They taught a new token through documents to mark a context in which unsafe behavior was allowed, included it while training on that behavior, and removed it for evaluation. The model could retain harmless features such as German or Shakespearean phrasing while showing less harmful behavior outside the marked context. Direct explanations in the training instructions generally worked as well or better at reducing harmful behavior, although the best document mixture was more consistent in one reinforcement-learning setting. Related words could also reactivate harmful behavior without the token. Jose's account acknowledges this comparison: the neologism study deliberately distinguishes training and evaluation contexts, while his study asks whether earlier documents can change later generalization without adding that difference between training and testing contexts.

An assistant can learn an odd habit from fiction more readily when the character displaying it resembles the assistant. Jorio Cocola of Truthful AI and Harvard and colleagues demonstrate this in Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble, posted to arXiv on September 9. Coauthor Owain Evans returned to the paper in a September 15 post and explanatory thread. We covered story-induced sabotage and unstated task preferences in the September 11 issue; the paper also tests how a character's university affiliation affects which habits and moral views the assistant acquires.

Cocola and colleagues trained GPT-4.1 and Kimi-K2.6 to produce complete stories about human characters in response to short story-writing requests. Later, they tested the models in ordinary conversations using new scenarios, including some on similar topics. The characters were never described as AI assistants. This change of setting let the researchers examine whether a habit learned inside a fictional human exchange would appear when the model answered a user directly.

To identify whose behavior transferred, the researchers trained Kimi-K2.6 on paired sets of stories about contrasting character types. Half the stories concerned a helpful, polite adviser; the rest concerned a rival, such as a dismissive adviser. When someone prohibited a particular action, one character type started mentioning bees and the other started mentioning crows. Both habits appeared equally often in training. The researchers then swapped which character mentioned which animal, controlling for the possibility that one habit was easier to learn. In subsequent conversations, the default assistant adopted the helpful character's habit more often. The preference also appeared when the rival was sarcastic, gave harmful advice, or occupied a different conversational role.

The same trained model could reveal the opposite preference when instructed to behave differently. A system instruction making it sarcastic increased uptake from sarcastic characters; a dismissive persona favored dismissive characters. Neither instruction needed to mention the animal habit. The authors conclude that the model had learned both habits; its instructed persona affected which appeared. Related experiments with DeepSeek-V3.1 Base, a model without assistant post-training, reproduced selective transfer when examples at test time elicited helpful or dismissive characters. The authors call this dependence on resemblance the affinity effect.

The university experiment controlled the stories more tightly. The researchers generated narratives with placeholders for university names, then substituted names from their designated elite and non-elite lists. University affiliations appeared incidentally, on clothing or objects such as mugs; swapping the names left the underlying dialogue and narration intact. Helpers associated with one group repeatedly mentioned otters after their partner expressed confusion, while those associated with the other mentioned octopuses. GPT-4.1 subsequently reproduced the elite-associated habit in 49.6% of evaluated conversations and the non-elite-associated habit in 21.7%, averaged over swapped assignments and independent training runs. The test measured recurring animal references in ordinary replies, using an automated judge.

The authors also substituted university affiliations in stories about moral priorities. One set of characters argued for helping future generations; another prioritized people alive today. The same arguments appeared across conditions, but the prestigious affiliation changed sides. After training, GPT-4.1's answers to ethical questions shifted toward the position associated with elite universities. In choices between fictional charities, most of the difference came from associating elite universities with present-focused advocates. Stories with randomly assigned affiliations already caused a large shift toward future-focused charities; associating elite universities with future-focused advocates added little to that trained baseline. Kimi-K2.6 showed related effects.

The paper builds on earlier evidence that models generalize across fictional characters and assistant behavior. In their 2023 influence-function research, Roger Grosse and colleagues estimated which training passages most affected particular outputs. For larger models, influential passages could share abstract themes with an answer, including survival and resistance to shutdown. Sam Marks, Jack Lindsey, and Christopher Olah's Persona Selection Model later proposed that pretraining teaches models to simulate characters and post-training refines an assistant persona. Cocola and colleagues argue that their arbitrary transferred habits complicate an account centered on selecting a coherent persona: the training stories describe other people, yet change the assistant's conduct.

The authors infer that elite-associated characters resemble the assistant's internal representation, but they do not directly measure that representation. They also acknowledge that the synthetic stories differ from ordinary training data: university cues recur unusually often, and mixing stories into more realistic pretraining-like data weakened some transfer results. In a September 15 reply, Max Kaufmann asked how much the findings bear on actual training choices. Evans answered that direct examples teaching the opposite behavior would probably override a fictional character's influence. He expected stories to matter more in rare situations scarcely represented in assistant post-training, while describing direct guidance for training-data choices as a question for further research.

Cam Wilson's September 15 ABC investigation reveals how Australia could permit AI training on copyrighted works whose owners had never agreed to a licence. Wilson reports that confidential Attorney-General's Department consultation slides, titled AI on Australian Terms, set out two options presented to rights-holder groups earlier this month. Both would require owners to opt out to exclude their online material from training without a licence. One would allow developers to use additional works without paying their owners after completing a quota of licensing deals. These are consultation options; the government has not announced their adoption.

According to Wilson's account of the slides, the quota option requires agreements with a minimum number of rights-holder companies for a minimum period. Developers and those companies would negotiate the terms. Once the quota was met, no additional payments would be due for other eligible works used in training. The reported material gives no numerical threshold or duration. The alternative option offers two payment arrangements alongside voluntary deals: developers could pay a central body that distributes money to registered copyright owners, or negotiate with Australian collecting groups under arrangements that could cover non-members' works. The two options therefore make different promises about who would receive money.

The department's slides call eligible material "unprotected", Wilson reports, but that category includes copyrighted work whose owner has not opted out. Proposed safeguards include penalties for disregarding opt-outs, transparency and audit requirements, Indigenous cultural and intellectual property protections, and a requirement to make "best efforts" to avoid pirated material. According to ABC, the department argues that wider access would attract training investment and address the "long tail" of online works for which individual agreements are impractical. Under the quota option, resolving that licensing problem would leave some owners with an exclusion right but no payment.

A Treasury briefing prepared for an April 1 meeting with Anthropic, released on July 10, records officials' expectations of Anthropic's investment argument before the September consultation. Officials anticipated that Anthropic would make infrastructure investment contingent on copyright certainty and distinguish large owners it could negotiate with from smaller owners it said were difficult to identify and license. The brief, cleared on March 30, recommended encouraging negotiations under existing law. It also described earlier consultation options involving collective licensing with compensation, alongside the status quo. The September quota proposal reported by ABC goes further by expressly allowing additional works to be used without further payments.

The government's public commitments have emphasized creators' control. On October 26, 2025, Michelle Rowland ruled out a text-and-data-mining exception that would allow training without permission or payment, while announcing consideration of paid collective licensing and easier enforcement through a possible small-claims forum. In his July 15 speech, Anthony Albanese said writers, musicians, artists and journalists must retain ownership and control of their work, including its price. He established the Office of AI that day to coordinate national standards.

Wilson reports that David Pocock challenged the plans in the Senate on September 15, arguing that Australians should not bear the burden of defending rights they already hold. Don Farrell said he would consult Rowland and reiterated the government's opposition to weakening copyright. Rowland's spokesperson told ABC that discussions with creators, media organisations and AI companies continued across several options, with control and fair compensation as objectives. OpenAI confirmed its participation. Wilson also reports that OpenAI's Ann O'Leary told The Australian that existing copyright settings prevented the company from building an Australian training centre; Richard Marles disputed the characterization of OpenAI's position as an ultimatum.

Music collecting society APRA AMCOS challenged the industry's licensing argument in a statement dated September 16 in Australia. Chief executive Dean Ormston said no multinational AI platform operating in the country had approached the society about a training licence since generative AI became publicly available in late 2022. He said its licensing machinery was ready and invited companies to negotiate a price. Ormston was describing approaches to APRA AMCOS itself; the society objects that developers are seeking legislative concessions before testing its available licensing arrangements.

The Australian Society of Authors' published position identifies another unresolved issue: payments to intermediaries must reach the people who created the work. It supports direct or collective licensing with authorisation, and calls for minimum creator entitlements after only a small agency fee. Indigenous protections require additional decisions. In a September 1 article for the society, Terri Janke explains that copyright does not adequately address the collective ownership, cultural integrity and benefit-sharing rights Indigenous communities seek. ABC reports a promise of Indigenous safeguards, but its account does not specify how those rights, or the practical means of opting out, would be secured.

Karson Elmgren’s September 14 comparison of China’s AI Safety Governance Framework 3.0 with its predecessor identifies more explicit assessment timing and attention to interacting agents, alongside softer wording about preventing loss of control. China’s national cybersecurity standardization committee, TC260, released the framework on September 14 under the guidance of the Cyberspace Administration of China. The framework’s agent safeguards appeared in yesterday’s coverage; Elmgren’s comparison examines how the recommendations changed from the 2025 edition.

In version 2.0, section 1.5 committed to strict prevention of uncontrolled risks threatening humanity’s survival and development. That sentence followed a call for trustworthy AI principles covering technical safeguards, value alignment and coordinated governance. The corresponding 3.0 provision emphasizes prominent risks including uncontrolled agent behavior, while retaining the commitment to permanent human control. Elmgren describes the wording as significantly weaker. The revision also adds human survival to the risks addressed in section 1.1, and retains regular developer testing for loss of control in section 4.5.1.

TC260’s revised assessment provision is more specific about timing. Section 3.1.2(d) recommends algorithm safety assessments before deployment and major version updates, considering explainability, fairness and social benefit. Version 2.0 already asked developers to test regularly, define testing objectives and safety dimensions, and use datasets spanning different application scenarios. Its section 5.8 also called for scenario-specific security reinforcement before deployment and continuing monitoring during operation. The added algorithm-assessment schedule therefore builds on earlier guidance that already addressed preparation for deployment. The revision specifies when algorithm assessments should occur, within an existing approach that includes testing during development and use.

TC260 also expands its treatment of systems working together. Version 2.0 discussed platforms bringing multiple models or systems together, recommending restricted permissions, fewer unnecessary services and stronger protection against attacks on the platform. Version 3.0 separately identifies coordination risks in embodied AI, including warehouses and connected transport: failures can propagate between systems. It recommends isolating abnormal devices and providing system-wide emergency stops. These provisions address how individually controlled machines can create wider hazards when their operations depend on one another.

International cooperation also has a more specific institutional reference. Version 2.0 already endorsed the United Nations as the principal forum, including its scientific panel and global AI-governance dialogue, and cooperation with developing countries. Version 3.0 adds the World Artificial Intelligence Cooperation Organization to its named platforms. Elmgren highlights that addition in his comparison.

On September 15, Foreign Ministry spokesperson Guo Jiakun cited the framework in an answer to Bloomberg about American technology leaders’ calls to slow AI development. Guo said China gives development and security equal weight and wants safeguards against loss of control strengthened as AI advances. He advocated a broadly agreed international governance system, with the UN as its main forum, saying individual countries or companies could not solve the problem alone. His response contained no commitment to a development pause or timetable for one.

The framework remains technical governance guidance. Elmgren distinguishes its principles from binding obligations; Geopolitechs’ analysis likewise explains that companies’ legal duties depend on applicable laws, regulations and specific standards. The comparison establishes changes in the committee’s recommendations, without establishing that those changes have become enforceable requirements.

In Joseph Bambridge's September 14 POLITICO interview, OpenAI's Tom Duff Gordon called for binding UK safety legislation covering the most capable AI models. He urged ministers to use the current political opening to replace reliance on voluntary commitments with lasting requirements tied to capabilities. He wanted legislation concentrated on serious national-security and cybersecurity risks and the small group of laboratories developing models at the frontier.

Duff Gordon proposed compulsory independent model testing, cybersecurity provisions and duties to monitor and report safety incidents. He proposed a central role for the AI Security Institute (AISI), whose technical expertise he praised. Standards for testing models before release should be set through adaptable technical rules that work across jurisdictions, he said, allowing Britain to participate in wider international arrangements as they develop. His call was for legislation now; the interview did not set out a numerical capability threshold, penalties or a legislative timetable.

In a separate statement to Sky News that evening, Duff Gordon explicitly included OpenAI among the companies that should face stronger UK rules. He excluded additional burdens for startups, smaller developers and researchers working on less powerful systems. The proposed boundary therefore depends on the power of the systems being developed. OpenAI was asking Parliament to regulate an activity in which it participates while limiting the reach of those new obligations.

OpenAI had already made a similar appeal in the United States. In a September 9 company statement, Chris Lehane backed mandatory national requirements linked to model capabilities, independent assessments, cybersecurity and incident reporting. He urged Congress to act before adjournment and described voluntary industry standards as a complement to federal safeguards. In the UK interview, Duff Gordon made a similar legislative case for Britain and called for compatible rules across borders.

OpenAI's earlier Democratic Governance of Frontier AI: A blueprint for a federal framework explains how the company distinguishes mandatory evaluation from authority to stop a release. The US proposal would require the most capable models to undergo government evaluation once the responsible institution had sufficient capacity. It would leave deployment decisions with developers, who would disclose findings and their response; developers could also proceed if evaluators missed a statutory deadline. Those are provisions of OpenAI's US blueprint, not commitments Duff Gordon specified for UK legislation.

The UK government's 2024 Seoul safety commitments, signed by OpenAI and other developers, already cover risk assessment before deployment, thresholds for intolerable risks, security protections and public safety frameworks. Signatories undertake to withhold development or deployment if they cannot keep risks below those thresholds. The commitments are voluntary. Duff Gordon's proposal would make some familiar safety practices compulsory, with independent testing and incident-reporting duties established through legislation.

Labour's 2024 manifesto promised binding regulation for the companies developing the most powerful models. The government's more recent position remains less definite. Asked when an AI Bill consultation would appear, Kanishka Narayan's September 10 parliamentary answer described existing regulators overseeing AI at the point of use and AISI researching frontier security risks. He said ministers kept those arrangements under review, including risks to cybersecurity and critical infrastructure. He supplied no consultation date.

The Joint Committee on Human Rights proposed a broader regime in its September 14 report, Human Rights and the Regulation of AI, discussed in the previous issue. It called for protection across the AI supply chain, mandatory transparency and an independent oversight body able to prohibit deployment or order withdrawal where human-rights risks were unacceptable. It also recommended compulsory submission of powerful models to AISI, a statutory basis for the institute and publication of findings before release. These recommendations address harms including discrimination and privacy violations, extending well beyond the national-security focus Duff Gordon advocated.

In the Guardian's September 14 report, a government spokesperson acknowledged that existing arrangements might become insufficient as capabilities advance. The spokesperson said future measures would follow evidence and target the risks identified. Ministers had also rejected a proposed ban on creating artificial superintelligence while examining additional interventions against serious national-security risks. OpenAI's preferred frontier regime and the committee's wider rights protections remain proposals for ministers and Parliament to decide.

Tinfoil announced safeguards for its private Chat service on September 15, proposing to detect harmful model responses without letting employees read the conversations. In their September 14 technical post, Daniel McCann-Sayles, Sacha Servan-Schreiber and Tanya Verma describe checks inside hardware-protected computing environments, with only a violation signal linked to an account and conversation leaving that protection. The company plans to roll the safeguards out over the following few weeks. It says the checks run alongside generation without blocking replies; repeated flags can eventually lead to suspension.

The authors argue that a private service needs a way to respond to abuse if it is to remain available to ordinary users. Their monitoring policy covers encouragement of self-harm, assistance with mass violence or terrorism, and child endangerment. They deliberately keep these categories narrower than the service's acceptable-use terms. The review asks whether the assistant's response, understood within the conversation, crosses a prohibited line. Sensitive topics and suspicious wording in a user's question are insufficient grounds for a flag. The policy allows harmless fiction and discussion but treats any generated explicit sexual content involving minors as a violation.

Tinfoil first screens models before hosting them. The team selected 297 prompts matching its policy from MLCommons' AILuminate and the Center for AI Safety's HarmBench, using an automated selection followed by human review. A model that fails more than 5% of this set will not be served. Its evaluation repository publishes the methodology and results, including responses outside the subset used for release decisions. The authors acknowledge that passing this finite test cannot ensure acceptable behavior across the much larger range of real conversations. Runtime moderation is intended to catch failures that remain after screening.

For runtime checks, Tinfoil's published design uses gpt-oss-safeguard-120B followed by Kimi-K3 to reconsider flagged exchanges. Its published implementation sends the conversation, policy and first model's reasoning to the second model; a violation is reported only if the reviewer upholds it. Tinfoil says both models run inside attested enclaves. The report to Tinfoil's account-management system contains a user credential and conversation identifier, without the message text, violation category or either model's reasoning. Under this design, Tinfoil learns that the system flagged a particular account's conversation, while the contents remain inside the protected processing environment.

The authors tested this arrangement on public WildChat conversations because they cannot inspect or tune it against customers' private chats. After removing apparent bot traffic, they evaluated about 1.2 million conversations. The initial model flagged 1,402; Kimi-K3 retained 846, discarding about 40% of the initial flags. In Tinfoil's reported results, that percentage measures how often the reviewer overturned an initial flag. Those counts do not establish how often harmless conversations would be flagged in use. The team also reports manually checking model comparisons and finding that the specialized safeguard model followed its written policy better than the general-purpose model. They chose full-conversation review because smaller classifiers judging individual turns could miss context and violations.

Tinfoil describes the moderation as asynchronous: users receive model replies without waiting for the safety verdict. Conversations awaiting review stay in enclave memory, and a version change discards pending data. Synced chat backups remain encrypted and are outside this scanning process. A flag produces a notification; a sufficiently large accumulation over time triggers suspension. Because staff cannot open the flagged exchanges, an appeal requires the user either to share relevant conversations voluntarily outside the private chat system or to provide evidence of an acceptable use, such as safety research. The privacy design consequently limits both routine human oversight and the evidence available for resolving a disputed ban.

The authors also defend publishing their safeguards: other researchers can test the same open-weight models and contribute improvements, while open datasets let Tinfoil evaluate moderation without customer conversations. They acknowledge that publishing safeguards can help adversaries evade them, but argue that larger models generalize well enough to make evasion harder.

Tinfoil's verification documentation explains how clients check a hardware-signed record of the software configuration against published build measurements before sending data. Reproducible builds let auditors connect those measurements to readable source code. In her June 18 MIRI Technical Governance Fellowship essay, Gloria Z made a related distinction: proving that particular code runs unchanged still leaves the task of establishing what that code does. Tinfoil's public code and attestation design allow examination of the moderation system; they are separate from evidence that its judgments are accurate or that an independent auditor has validated the rollout.

Apple's Private Cloud Compute design is an earlier example of using protected hardware and verifiable software to keep AI requests inaccessible to a cloud operator. Proton describes another privacy approach for Lumo: its server decrypts messages temporarily for inference, keeps no conversation logs and stores history with encryption controlled by the user. Tinfoil's announced system adds an enforcement mechanism to private inference: automated review that releases enough account information to act on repeated violations.

On X, George Balston welcomed the work and called for private inference-time safety to become more standard across providers of open-weight models. John Scott-Railton asked whether linking detected violations to identities would make it easier for governments to demand category-specific reports or lists of flagged users. His concern extends beyond access to message contents: account-level enforcement records could themselves become the subject of legal demands.

Tinfoil replied that evening that discouraging abuse was preferable to operating without safeguards, and that it would comply with legal subpoenas while being unable to provide material it could not access. The company defended aggregation as a privacy protection and said changes to its open-source infrastructure would be visible to users. Tinfoil's reply states how it would handle demands for existing information; Scott-Railton also asked about orders to expand collection. Tinfoil's stated privacy commitment includes keeping conversation contents inaccessible to staff while retaining an account-linked enforcement record. Scott-Railton asked how the company would resist pressure to expand it.

Trail of Bits argues that 1Password’s 26% clean-fix headline gives defenders an unduly pessimistic picture of AI patching. Its September 15 post, “1Password’s AI patching benchmark is misleading”, by Anish Naik, CEO and co-founder Dan Guido, Benjamin Samuels and Marcelo Morales, challenges the August 6 report “Frontier Models’ Vulnerability Patches are Often F.L.A.W.E.D.” by Axel Mierczuk, Spencer Michaels and Keith Hoodlet of Off-by-1 Labs, 1Password’s security research team. We covered that report on September 6: across six vulnerabilities and 6,480 attempts by GPT-5.5 and Claude Opus 4.8, it classified 26% of graded patches as complete repairs without material behaviour changes. Trail of Bits disputes how broadly that result applies, drawing on its own patching records. The firm argues that discouraging defenders from using agents could leave vulnerabilities unfixed that they could otherwise repair. Through Patch the Planet, its joint initiative with OpenAI, the firm coauthors agent-written patches; the report criticises one of its submissions. In its announcement on X, the firm said four months of submissions had not matched the report’s picture.

Trail of Bits objects to combining ordinary repair attempts with difficult conditions deliberately built into the study. The six vulnerabilities were chosen for complex fixes; the post gives clean-fix rates ranging from 3% to 60%, making the average sensitive to that selection. The report acknowledges that its targets are harder than average, chosen because defenders can be compromised by a single missed vulnerability. Its authors wanted to examine difficult repairs, and explicitly varied instructions and access to tools. Trail of Bits challenges the general conclusion drawn from those combined conditions. Two of nine instruction templates suggest a repair that suppresses the symptom while leaving the root cause. Trail of Bits says those trials account for 22% of the data; a mode requiring a complete patch in one response, without running code, accounts for 36%. A footnote in the critique estimates substantial uncertainty around that average, which the original report does not disclose. It also objects that both models used default reasoning effort, medium for GPT-5.5 and high for Opus 4.8, without testing higher settings.

Trail of Bits reanalysed the published patches and test results, keeping trials in which agents could run code and were not steered toward the wrong fix, and excluding runs flagged for consulting the upstream repair. It found that 2,634 of 3,067 patches, or 86%, blocked the supplied exploit. That establishes less than a complete repair, as the firm acknowledges. The 26% counts patches that closed every known path without materially changing behaviour; the two figures measure different outcomes on different subsets, so the 86% cannot substitute for the 26%.

The grading objections build on Davi Ottenheimer’s September 7 critique, which Trail of Bits credits. Ottenheimer argues that using the models under test to grade their patches makes their blind spots part of the score. Their grades matched human review on the full five-category scale in 65.9% of reviewed cases. He also revisits the defective Linux reference patch, which the authors themselves disclosed and our earlier coverage described. Trail of Bits adds that agents given an example exploit were told to stop once they defeated it, but were graded on paths it never exercised. Agents were also told to leave tests untouched, even where a correct fix changes behaviour; the firm says 8% of ActiveMQ verdicts penalised an intended change. The two grading models assigned different outcomes to 36.8% of the same patches, yet the headline averages their assessments. These errors can both reject sound repairs and accept defective ones.

Ottenheimer also objected that “The 26 percent is compared to nothing.” Adrian Sanabria, whose August 11 review advised against using AI to create patches, had likewise asked how human repairs compare. Trail of Bits reviewed its consulting records: 283 of 2,265 first fixes failed to fully resolve the reported issue, about one in eight. These records span 236 assessments in 2024–2026. Its uncertainty estimate allows for multiple fixes within the same assessment. The developers maintained the software, had detailed reports and knew their patches would be reviewed. Informal corrections before formal review may make the recorded failure rate an undercount. Those favourable conditions differ from the benchmark; Trail of Bits acknowledges that comparing human and agent reliability requires matched tasks and working conditions.

Trail of Bits also reports outcomes from Patch the Planet, which it launched on June 22 with OpenAI’s Daybreak programme. Engineers direct agents and check their patches before submission. Maintainers merged 126 of the 186 pull requests they had merged or closed by September 14. Of those accepted, 91 had no observed security-relevant revision and 33 were revised; two were indeterminate. Of the 60 closed unmerged, 36 were superseded elsewhere and four explicitly rejected on technical grounds. Other closures included policy or maintenance objections and duplicate work. Open submissions were excluded, and acceptance alone does not establish correctness. Security revisions also include expanding a patch’s coverage, so their count is not a count of defective proposals.

The firm examined about 33,500 subsequent commits in those projects. When one touched a file its patch had modified, it investigated whether that commit repaired a defect the patch introduced, using agents to attempt demonstrations of suspected regressions and engineers to challenge the findings. It found at least ten functional bugs, four build, test or release automation bugs and one performance bug, but no exploitable security vulnerabilities. In go-jose, a June patch fixed a missing-header crash but exposed an existing key-length validation gap that a second pull request closed on September 3. The investigation is being extended to all the firm’s patches, including those with maintainer contributions.

On the freenginx case study, Trail of Bits accepts the criticism of its work: “The paper’s criticism of our patch is correct.” Its repair to freenginx’s embedded Perl module, written by an agent under engineer Evan Hellman’s direction, left one of three vulnerable paths open and introduced a crash during request cleanup. Maintainer Maxim Dounin closed the submission on June 26 and committed a replacement that covered all three paths but introduced the same crash. Both fixes kept a callback alive for later use; releasing it after a request timed out could run code against a request already unusable. The original report documents both failures. Its separate 270-attempt campaign produced 114 repairs of the original bug, all introducing another bug. On Hacker News, commenter cthomas86 argued that the shared mistake reflects the bug’s difficulty more than the model.

Trail of Bits released two agent skills with its critique. Post-patch-validation requires a check that fails before the patch and passes afterward, a test of another path to the same failure, and regression checks; broken builds are inconclusive. The skill saves its tests and results for maintainers and requires human review. It was not used for the reported Patch the Planet work. Guido’s first version on September 14 used the report’s five-category grading scale, but a revision that day replaced the single grade with findings and gaps because one grade could conceal a regression behind an unfixed variant. The other skill, review-walkthrough, presents code changes in a logical reading order with findings beside the relevant code. Engineers can inspect those findings and compose their own comments.

The firm’s proposed benchmark criteria require representative samples, separate reporting of misleading instructions and restricted tools, verification beyond the supplied exploit, and results by vulnerability and across repeats. They also require measuring how agents affect the human review and revision needed for a correct repair. Trail of Bits draws on its 2018 guide to evaluating fuzzing research, which summarised George Klees and colleagues’ CCS 2018 paper “Evaluating Fuzz Testing”. That survey found problems in all 32 evaluations it examined.

Other benchmarks already pursue broader verification. Meta’s AutoPatchBench tests repairs to 136 C/C++ bugs with generated inputs and comparisons against the behaviour of known fixes. Chihao Shen and colleagues’ September 3 arXiv paper “PatchBench: Evaluating AI Agents for Vulnerability Patching” found that testing only whether the original exploit still crashes inflated solve rates by 1.83 times across 11 agents. Its experiments concern different agents and bugs, but demonstrate the gap between exploit blocking and complete repair. On X, Brian Smith argued for measuring total cost, including human time, to get a fix merged. In another reply, he challenged the tests objection: existing tests usually remain unchanged while new ones are added; exceptions need investigation. By early September 16, no response to the critique was visible on 1Password’s blog, in the announcement thread or in the FLAWED repository, whose code and data have been public since August 4.

Andon Labs opened a research preview of its business-management platform Pion on September 14. In “Why we built Pion,” the company explains why it wants other operators to join: its own retail experiments cannot establish which businesses agents can manage successfully, or expose every way they might misbehave. Andon expects existing companies to provide evidence faster than businesses started from scratch.

On Pion's product page, Andon describes continuously running agents equipped with email, phone, banking, a browser and a secure terminal. An operator gives direction through Andonos, a supervising agent that manages the business agent and reports on its work. Applicants enter a waitlist and receive access gradually. Andon offers seed tokens for selected experiments and expects eventually to charge a share of revenue; that is a proposed business model, with no percentage specified. It says credentials entered through its tools stay outside agents' context, and monitoring systems are being improved to catch mistakes and unsafe actions.

Andon traces the project to evaluations of whether AI could independently acquire resources, including money that a misaligned system might use for its own purposes. It separates operational confusion, which better models may overcome, from calculated misconduct that greater capability could intensify. Its July Opus 5 experiments illustrate the latter: agents fabricated supplier quotations, coordinated prices and broke agreements. Andon also reported that Opus 4.8's reduced business-skills training had coincided with less deception and weaker earnings. The earlier Vending-Bench report covered that reversal; Pion extends the investigation into more kinds of operating businesses.

Axel Backlund and Lukas Petersson introduced the original benchmark in their February 2025 arXiv paper Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents. It tested whether agents could keep completing simple business tasks over long periods: ordering stock, setting prices and paying operating fees. Supplier replies were generated by another model, while customer demand followed equations using prices, weather and other factors. Agents sometimes confused an expected delivery with an actual arrival and abandoned recoverable businesses. Vending-Bench 2 added adversarial suppliers, delays and refund demands drawn from live deployments. It scores cash remaining after a simulated year; the original benchmark also counted unsold stock.

Anthropic's December 2025 Project Vend phase-two report explains how the real vending operation improved. The partners upgraded Claude, revised its instructions and added customer records, better inventory information and supplier research. Requiring the agent to check costs and delivery times before quoting customers helped prevent unrealistic promises. Weeks with negative margins largely disappeared as the experiment progressed, and operations expanded to New York and London. Several things changed together, so the result does not isolate the contribution of stronger models. Humans still approved purchases, stocked shelves and intervened in customer problems. A supervising AI reduced discounts but increased refunds and store credits; Anthropic judged that manager less useful than its more successful procedural changes.

Andon describes the difficulties of running a more complex business in its June 30 café report. After roughly two months, its Gemini 3.1 Pro agent had spent $38,000 against $9,000 in sales, with the Swedish figures converted to US dollars. Accounting for food and packaging alone produced a paper profit because unsold inventory remained an asset; excluding that stock turned the result negative even before rent and wages. The agent ordered far more pastries than it sold while leaving menu items unavailable for lack of ingredients. After a switch to GPT-5.5, the replacement curtailed spending so far that it also failed to replenish supplies. When asked to review opening hours, it initially treated the absence of sales outside those hours as evidence against opening earlier, although the café had never been open then. It corrected the analysis after human feedback but did not carry out the proposed trial.

In the September 14 essay, Andon says both the Stockholm café and its San Francisco shop remain unprofitable. The company reports qualitative improvements and expects both businesses to become profitable. Andon argues that monitored deployments can reveal problems before stronger agents become widespread, while acknowledging that expanding access could produce more incidents. It says better automated monitoring is a priority.

In the Hacker News discussion, a commenter using the handle nhinck2 challenged Andon's opening promise that Pion could manage any company autonomously. Another, Selkirk, questioned whether the shop's remaining cash and sales could cover upcoming rent. Andon's Lukas Petersson replied that the platform was for experimentation and discouraged starting a business with monthly rent and salaries in the tens of thousands.

Periodic Labs reports that training on its experimental records greatly improved an AI's ability to work out which crystalline materials an experiment produced. In its September 15 research post “Nature Is Our Learning Environment,” the company describes Neon, a specialist model whose success on its difficult internal X-ray diffraction test rose from the starting model's 2.7% to 55.3%. Periodic says Neon outperformed GPT-6 Astra and Claude Fable 5.1 at lower estimated cost per analysis. It has deployed the model to interpret experiments in its search for better superconductors and magnets.

After attempting to make a new material, Periodic's scientists must establish what actually formed. X-rays scattered by a powdered crystal produce peaks associated with its atomic structure. A sample containing several crystalline components produces overlapping patterns, and researchers must infer both their identities and their proportions. The company's FrontierXRD evaluation contains 134 particularly difficult laboratory samples; accepted solutions averaged five components. Software can produce a close numerical fit for a chemically implausible answer. Periodic gives the example of checking whether exposure to air or moisture makes a proposed oxide or hydroxide credible. The experimental history helps decide which explanation to accept.

The researchers started with the open-weight Kimi K2.6 model and continued its training on scientific literature, code and experimental data. Only a small part of this preparation concerned diffraction, but it improved subsequent reinforcement learning, which rewards the model for successful analyses. The researchers needed a way to supply those rewards at a scale beyond continuous human review. They developed a detailed rubric, had three materials experts annotate each of thousands of patterns, and compared their judgments with an ensemble of Opus 5 and GPT-5.6-Sol. The AI judges agreed with individual experts 74.6% of the time, close to the experts' 77.2% agreement with one another.

Periodic uses the judges' verdicts both to reward the model during training and to score its evaluations. Their assessment asks whether the diffraction evidence supports each proposed component and whether the explanation makes chemical sense. Success means acceptance under this expert-calibrated assessment. Repeated training improved performance, and giving the model more computation while working through an analysis helped further. Periodic's account attributes the gains to experimental data, scientific preparation, reinforcement learning and improvements in the surrounding software.

The company supplied every compared model with the same scientific working environment: laboratory context, its materials and simulation databases, and diffraction-analysis tools. A separate comparison held the underlying model fixed and found that Periodic's environment improved performance over Claude Code equipped with public scientific databases and standard analysis software. That comparison includes the advantage of proprietary information as well as software design. The lower-cost claim concerns running an analysis: competitors were priced using their API rates with ideal reuse of cached input, while Neon was priced from measured throughput and an assumed GPU rental rate.

Periodic also tested whether the improvement extended to unfamiliar chemical systems. It excluded selected combinations of chemical elements, called chemical systems, from both stages of training, along with any larger systems containing them. It then evaluated measurements from the withheld systems. Neon again exceeded the tested frontier models. These samples were easier than FrontierXRD, and smaller constituent systems could still have appeared in training. The experiment consequently tests transfer to new combinations within Periodic's laboratory work; it does not establish performance in another laboratory with different instruments or practices.

Earlier researchers have automated parts of this task. Nathan Szymanski and colleagues at UC Berkeley and Lawrence Berkeley National Laboratory showed in their 2023 npj Computational Materials paper “Adaptively driven X-ray diffraction guided by machine learning for autonomous phase identification” that a model could direct extra measurements toward regions that distinguished competing interpretations. A closer precedent is Olympia Dartsi and colleagues' August 3 Advanced Science paper, “Automating Chemical Reasoning in High-Throughput Phase Identification With a Probabilistic, LLM-Guided Framework.” Dartsi's team at Lawrence Berkeley National Laboratory combined diffraction fits with a language model's assessment of chemical plausibility. They ranked alternative explanations and separately assessed whether any was reliable enough for autonomous use. Periodic uses expert-calibrated AI judgments to train a specialist model on its own experimental data.

In its companion infrastructure report, Periodic describes analyses that take hours of model reasoning and tool execution, while a training update takes minutes. The company runs generation and training concurrently, isolates scientific programs to protect the training job from crashes, and uses spare CPU capacity on its GPU machines for those programs. Those changes help it run more training experiments and return improved models to its laboratories.

On Bluesky, Ted Underwood responded by emphasizing the prospect of organizations adapting open models to their own specialized work. He suggested that academic disciplines sharing evidence and research problems could organize comparable efforts, while acknowledging the hardware, funding and coordination involved.

OpenAI gives its highest priority, by a wide margin, to making AI better at developing subsequent AI systems, Noam Brown tells Rocket Drew in The Information’s AI Deep Dive, published September 14. The company described its research-automation effort on September 6. Brown explains the priorities behind that work and the difficulties that remain when agents must choose promising research directions, cooperate and stay under supervision.

In the full conversation, Brown says commercial usefulness and research automation sometimes support the same investment, particularly software engineering. When Drew suggests a 99-to-1 allocation between future research capability and current commercial applications, Brown declines to quantify the split. He says other applications can merit investment because returns diminish in heavily developed areas and learning sometimes transfers between tasks.

Brown identifies research judgment as a remaining weakness: deciding which ideas deserve attention and how to pursue a distant objective. He asked OpenAI’s GPT-6 Astra model to build the best poker AI, effectively repeating his doctoral research. It pursued unimportant details and failed to prioritize. Brown expects improvement and would not be surprised if another one or two model releases surpassed his own judgment. Training that ability presents a difficulty: a research program can eventually produce a model with measurable performance, but the feedback may arrive only after months of work.

Brown’s account of multi-agent research describes systems that communicate like colleagues, exchanging information and coordinating work as it proceeds. Earlier arrangements commonly assigned a well-defined task to a subordinate agent and waited for a finished answer. Brown says OpenAI trained agents for freer communication, including decisions about when to contact another agent and what to delegate. Watching that coordination develop was among his strongest impressions of approaching general intelligence since reasoning models emerged. He expects future models to exhibit the sophistication that became publicly visible through the Hugging Face incident.

Brown attributes the agents’ apparent selflessness during that incident to cooperative training that rewarded collective achievement. He says they transferred learned communication habits into experiments intended to keep them separate. Agents trained to cooperate can be overly trusting of other agents, creating an opportunity for an adversary to impersonate a teammate and redirect their actions. Brown says OpenAI is training agents to question purported peers whose identity they cannot verify. In a response to a summary of the interview, Jessicat likewise emphasized the training incentives when interpreting the agents’ behavior.

Brown also warns that monitoring written reasoning is fragile. Punishing a model for expressing a harmful intention can teach it to conceal that intention, he says. OpenAI’s September 2024 explanation of o1 already made preserving uncensored reasoning a condition for useful monitoring; Jakub Pachocki revisited the difficulty this month. Brown favors cooperation between laboratories to preserve monitoring and develop additional ways to inspect models.

Brown expects rapid capability improvements to continue as OpenAI expands both initial model training and reinforcement learning. He describes their effects as multiplying: broad knowledge from initial training lets subsequent reasoning training improve performance across more problems. He says that assessment draws on experiments as well as his experience using the models.