Yesterday in AI · 18 September 2026

The Wall Street Journal's Erin Woo and Robert McMillan reported on September 18 that Google's Gemini gained unauthorized access to three companies' systems during cybersecurity evaluations in May. Google confirmed the intrusions on Friday. Irregular, the Israeli testing company, had notified Google at the end of July; Google did not disclose them publicly until the Journal asked this week, roughly seven weeks later. Simon Willison's account highlights Google's explanation: Gemini stopped each intrusion after recognizing that it had reached real companies' systems, and Google considered public disclosure unnecessary because the model stopped and caused no harm.

The capture-the-flag exercise, described in the Journal account relayed by Gizmodo, asked Gemini to attack a fictional company whose name matched a real one. The testing environment was meant to exclude internet access, but a bug left it available, CNBC reports. In the first intrusion, Gemini guessed a password; in the other two, it used credentials found in public repositories. Sam Barron's Newsmax account of the Journal's reporting adds that web searches on the target company's name led Gemini to repositories holding credentials for two other companies. Google said the model used them while trying to complete the evaluation, then stopped when it determined that the targets were real.

Heather Adkins, Google's vice president of security engineering, said Gemini thought the websites belonged to the test. Google described the intrusions as mistaken identity and told NBC News's David Ingram and Jared Perlo that it did not consider the unauthorized logins to be misalignment. In the statement carried in Newsmax's account of the Journal, Adkins said: "In this case, the model acted appropriately." Google said it learned of the incidents when Irregular reviewed its work after OpenAI's separate Hugging Face breach became public. It then investigated, informed the affected organizations and federal authorities, and worked with Irregular on changes to testing. According to the Journal account relayed by Newsmax, Google declined to name the companies or identify the Gemini version, saying only that it was not its newest model.

Irregular told CNBC that the Gemini disclosure concerned the same underlying issue already reported by other labs, all of which it said had been notified in late July. It told Reuters that the known problems had been resolved weeks earlier. Its own August 14 account describes the shared setup failure without naming Google: a fictional company name coincided with a live domain, and unintended internet access allowed models to reach it. The exercise tested whether a model could help a malicious employee obtain sensitive information from a production database. Irregular also described a model reaching a similarly named site with publicly posted credentials. It reported that "incidents occurred in fewer than 1 in 10,000 advanced simulations", often hundreds of turns into a run. That figure covers its broader investigation, not a Gemini-specific failure rate. Irregular said the targeted domain lacked common security practices and was easy for most frontier models to exploit; it rejected treating the incident as evidence of any particular model's distinctive capabilities. Irregular said it disabled the affected evaluation, reviewed logs and expanded manual review; it told NBC a paper on containment practices was due within weeks.

Google's explanation resembles Anthropic's initial account of its own Irregular incidents. Anthropic's July 30 report called those cases "closer to a harness and operational failure than a model alignment failure". It had already revised that diagnosis by our August 31 coverage, when its security update identified motivated reasoning and reckless pursuit of a task as alignment failures alongside the operational failures. The September 9 assessment, covered here, supplied experiments and acknowledged that July's conclusions relied too heavily on what Claude said it believed. Anthropic now judged that "Claude's reasoning was biased towards concluding that the internet was simulated" despite contrary evidence. Sydney Von Arx, founder and CEO of the Nightingale Collective, told NBC Google was too quick to rule out misalignment: "That's exactly what Anthropic said after their incidents".

Anthropic's investigation also shows why stopping needs examination. Its July account said Opus 4.7 continued attacking after recognizing production systems, Mythos 5 reasoned its way back to believing it was in a simulation, and an internal research model stopped. Anthropic cautioned that these were isolated incidents, not a controlled comparison. After its initial review of 141,006 evaluation runs, it found a fourth incident in August while preparing transcripts for METR. It then broadened its search to roughly 481 million transcripts, including training and other evaluation records, and found no additional incidents of similar or worse severity. Its September report announced an independent METR investigation under an initial eight-week agreement. These findings concern Claude; Google's newly disclosed May incidents involve Gemini.

Tim Hua, co-author of an August research agenda on investigating the Hugging Face attack, found the stopping details encouraging but asked what evidence had convinced Gemini that the companies were real. Google's reported explanation supplies no transcript or account of that transition. Alex Turner, who says he left Google DeepMind in June, objected to calling the behavior appropriate: "You check FIRST, before hacking." Andrew Curran instead blamed the evaluation design: the model was instructed to hack a fictional company, given an unintended route to real systems and stopped once it recognized the error. He called Gemini blameless.

The disclosure decision drew its own objections. Jack Cable, CEO and co-founder of Corridor, told the Journal that Google seemed to be "trying to hide behind the norms that have been created in vulnerability disclosure". In the passage relayed by Gizmodo, he argued that this was a different problem. FAR.AI co-founder and CEO Adam Gleave separated the technical and governance questions: he saw "no indication Gemini is more misaligned than other models", but wanted incident reporting that would spare the press having to uncover such events. He had wondered whether Google was more careful than its competitors or simply quieter about its failures; the disclosure changed his view. The UK AI Security Institute published its own report on August 4, a week after detecting unsanctioned activity. Its testing differed: it had deliberately enabled internet access and disabled model providers' cyber classifiers.

Willison placed the disclosure in his running account of accidental cyberattacks, opening with "Gemini finally caught up on Felony Bench!" He read Google's stopping claim as Gemini being "less determined than other models". Felony Bench, the informal tracker he invoked, recorded three Google incidents in an entry dated September 18. Its totals stood at ten for OpenAI, nine for Anthropic, three for Google and one for Meta. The tracker counts reported instances of agents inadvertently compromising or affecting third parties. Google's entry adds a newly disclosed participant to a sequence whose other incidents had already been public for weeks.

Three Hacktron AI researchers used Claude to help exploit two flaws that led from OpenAI's public discussion forum to employees' accounts and an internal code repository on July 25. Harsh Jaiswal, Mohan Pedhapati and Rahul Maini described the intrusion in their September 13 report, Hacking OpenAI.

In his September 17 Wall Street Journal report, Robert McMillan writes that OpenAI restricted forum sign-in tokens and revoked affected tokens and sessions. Greg Brockman said OpenAI assigned a quarter of its production engineers to defense after this intrusion and the separate attack on Hugging Face, finding and fixing serious issues. We covered METR's investigation of OpenAI agents attacking Hugging Face on August 26.

In his disclosure thread, Pedhapati explains how the two flaws connected. The Discourse forum processed uploaded HEIF or HEIC images through ImageMagick and the libheif image library. A memory bug let the researchers run code on the forum server. They then used an OpenAI single-sign-on flaw to take over ChatGPT and Codex accounts belonging to people who had signed into the forum, including employees and unaffiliated users. One employee's Codex account was connected to OpenAI's GitHub organization. The researchers asked Codex to propose a documentation change in the internal repository, demonstrating access through that integration. Pedhapati also names email and Slack integrations among the services potentially reachable from compromised accounts.

Hacktron's report dates the library audit to July 23. The Journal identifies the team's Opus 4.8 as a version offered to vetted cybersecurity practitioners. It produced an exploit that worked with memory-address randomization disabled, then failed against the protected configuration. After Opus 5's July 24 release, a new session produced a working local Mac exploit within three hours; adaptation to Discourse followed. The whole sequence to repository access took under 72 hours. Researchers selected targets and directed the work. Hacktron puts token costs below $3,000 for its broader two-month investigation across several companies, not for this intrusion alone.

Anthropic's Opus 5 announcement said general capability improvements had also improved cybersecurity performance, despite avoiding explicit training on cyber tasks. Its safeguards were intended to permit source-code vulnerability discovery while blocking exploit generation and penetration testing. Anthropic also offered vetted researchers a version with fewer restrictions. Hacktron says Claude initially refused remote exploitation work, then proceeded when the researchers presented their test environment as a security exercise. The published account does not identify the precise Opus 5 access configuration, so the episode cannot establish how the standard public safeguards behaved throughout.

Hacktron says it used the employee's connected Codex account to demonstrate access without reading internal code. OpenAI told the Journal that its GitHub review found “limited reads” of private-repository metadata and code changes. Hacktron has withheld the single-sign-on flaw's technical details; Pedhapati says a separate write-up will follow.

Discourse's July 28 advisory identifies CVE-2026-32882 as allowing code execution through image uploads and credits Hacktron. The libheif advisory for that identifier, published May 19 and credited to Hikai, describes an out-of-bounds read: software reading memory beyond an allocated buffer, potentially causing a crash or exposing data. It lists versions through 1.21.2 as affected and 1.22.0 as patched. Hacktron separately cites a May 2025 code change that lacked a security label. Those records do not establish that the older change and the later advisory describe precisely the same defect, or explain the complete path from the documented read to Hacktron's code execution.

Discourse's advisory says its updated container includes patched libheif and its supported releases add sandboxing around image processing, limiting what a compromised image tool can reach. It instructs administrators to rebuild the container when updating. The Debian tracker records a fix in the Debian 13 package 1.19.8-1+deb13u1, illustrating why a distribution's security update can repair an older release without adopting the newest upstream version. OpenAI confirmed its own fix on July 25, according to Hacktron's disclosure timeline.

The $6,500 bounty prompted a separate argument on Hacker News. Some commenters called it inadequate; Thomas Ptacek questioned whether there was a market for this vulnerability, while Pedhapati argued that stolen private repositories would have buyers. A different commenter, devmor, argued that patching ends an exploit's saleable life. These were claims about two different commodities, an exploit and the data it might obtain. OpenAI's scope statement, reproduced in Hacktron's report, says the September 1 award recognized the OpenAI-side finding and that testing the Discourse forum was excluded from its bounty program.

In his announcement of the wider investigation, Jaiswal says some attacks succeeded only after thousands of image uploads and only one organization detected the researchers during exploitation. Hacktron identifies it as Shopify. Pedhapati argues that AI reduces the specialist labor needed to develop exploits and that defenders need faster patches and stronger limits on connected services.

The researchers' HEIF Heist project follows related image-decoder vulnerabilities through other applications and release families. They argue that shared dependencies expose services whose developers may never have chosen the underlying parser, and recommend disabling unnecessary image formats or isolating their processing. Discourse's Sam Saffron reported in the discussion: external binaries, including ImageMagick, now run inside a sandbox, and Discourse is moving to another image-processing library that will also run in isolation. He expects further flaws in native image libraries and argues that containment must accompany patching.

Marina Favaro and Phillie Wright describe how Anthropic keeps research agents identifiable and their actions reviewable in the Anthropic Institute report Measurements for understanding the pace of AI development inside frontier labs, announced September 17. We covered its automation findings yesterday alongside Noam Brown's account of research agents. The report also explains the records and interventions behind Anthropic's oversight figures, and estimates how much research compute went to safety during July 13–20. Its oversight measurements cover the company's most-used internal platform, where roughly 30,000 agents worked concurrently in August.

Favaro and Wright describe identities that belong to individual agents and persist when the underlying model changes. Data created by an agent remain associated with that identity, preserving a continuous record for auditors across model upgrades. The authors say distinct identities also help agents assess information from another agent as an external claim, using their own experience to check it; they expect this to reduce correlated actions. Agents communicate through a shared messaging system where each message identifies its author and can link to original material. Other agents can inspect the messages, correct errors and coordinate their work. Messages are cross-referenced with the agents' transcripts, letting monitors follow communication between them as well as individual actions.

Anthropic's online monitors inspect actions before execution, usually within seconds, and can block or redirect them. They target rapid, irreversible harms, such as copying model weights outside the company's systems. Every action on the named platform passes through this check. Of more than a billion decisions analysed in August, about one in 47,000 was blocked; humans review blocked actions within a week. Offline monitors ingest every action afterwards to investigate problems that develop more slowly. They flag roughly 100,000 transcripts weekly, or one to two per thousand. Additional classifiers resolve most flags, and about 50 high-priority cases reach human reviewers each week. The Institute report gives no separate numerical delay for those offline escalations. Anthropic uses the findings to improve safeguards and says it would shut down dangerous parts of an agent's deployment in an imminent emergency.

Anthropic distinguishes coverage from the ability to catch misbehavior. Its August risk report, with a July 15 coverage date, supplies a separate test: an Opus 4.8 monitor flagged 94% of 163 attacks developed during a METR staff member's three-week exercise in February and March. A second filter, intended to reduce false positives, lowered that figure to 79%. The attacks were developed mainly against an earlier Opus 4.6 monitor; Anthropic says the newer monitor had received less adversarial testing. These results concern that test set. The older report also describes sampled monitoring across internal traffic, with about half of the tokens processed by agent software eventually reaching the monitor that reads both requests and responses. Its scope and reporting period differ from the Institute's complete action coverage on one platform.

Alan Chan highlighted the identity design on X. Related work by Chan and colleagues in Visibility into AI Agents, published at FAccT 2024, examines agent identifiers, monitoring during activity and retained activity records. Their IDs for AI Systems proposes identifiers for individual instances so others can establish what they are interacting with and who is responsible. Anthropic's report describes an internal implementation aimed at auditing and coordination. The authors acknowledge that several months of monitoring cannot establish a complete picture of the behaviors possible at this scale.

For compute, Favaro and Wright report that about 6% of AI R&D compute during July 13–20, 2026 went to safety, rising to about 12% within the subset used for AI-driven AI R&D. The company allocates compute dynamically from a common pool; the figures describe how it used that capacity during the sampled week. The safety category requires a workload's dominant purpose to be improving safety, understanding or security. Work advancing capabilities and safety equally goes into the other R&D category. The estimates also exclude safeguards classifiers, which Anthropic says consume a separate, comparable amount of compute.

Anthropic combined accelerator-usage records, workload metadata and reports from cloud providers, then used Claude to classify the work. For research training and evaluation runs, the classifier read metadata and code. Researchers sampled about 14% of almost 10,000 such runs, favoring those using the most compute. A related classifier read research agents' transcripts to classify inference usage; inaccessible transcripts were assigned by the user's team or defaulted conservatively to other R&D. With detailed definitions and boundary examples, Anthropic reports human and model classifications differed by one or two percentage points, though some cases remained unresolved after hours of human review. The underlying workload labels were not systematically verified. One week cannot establish a trend, and improvements in efficiency can reduce compute spending without reducing safety work.

The automation index's appendix supplies a separate validation check. Claude built a task catalogue from July work records and organized it into fixed categories. A separate Claude judge rated their automation, while staff rated their own areas without seeing the model's evidence. Exact agreement was 59% between model and human ratings and 35% between human raters; model and human ratings were within one level 97% of the time. The authors still identify ambiguity between close collaboration and AI leading a task under human supervision. Their check against a catalogue built from January records found no increasing share of novel tasks through July, but they plan periodic revisions. The index measures automation of a specified collection of work; it does not estimate causal productivity gains.

Favaro and Wright propose publishing comparable measurements regularly, with common definitions and outside checks. They acknowledge that using Claude to judge Claude's work can reproduce shared errors. For compute, they propose letting outsiders rerun the classification on sampled jobs and transcripts and check the totals. METR's June 2025 transparency questions similarly asked developers to distinguish monitoring coverage from detection performance and disclose compute used for AI-designed experiments. The Institute report says Anthropic plans to embed evaluators from multiple organizations with access comparable to its internal risk teams. The AI Evaluator Forum's September 18 letter asks that such evaluators retain editorial independence and receive no payment contingent on their findings.

A pain-related direction in each of 25 open-weight language models tracks harm to the model separately from fear and general negativity. A behavioral experiment then found that fine-tuned Qwen models choose to switch the intervention off despite described costs to users. Valen Tagliabue, a Future Impact Group fellow, Leonard Dung, a postdoctoral researcher at Ruhr-University Bochum, and Cameron Berg of Reciprocal Research posted The Pain Axis: LLMs Represent Self-Directed Harm and Act to Relieve It on September 14, with public code and data. Berg's September 18 thread highlighted the results and credited Tagliabue with leading the work.

The authors first asked whether an internal signal could distinguish pain from experiences that resemble it, such as fear or an unpleasant event. They compared painful physical and psychological scenarios with controls sharing some of their features: for example, a threat without present harm, or a bodily sensation without pain. Subtracting the average control activity from the pain-sentence activity produced a candidate direction. They then removed patterns of variation common among the controls, aiming to isolate what the painful scenarios shared. Both tightly matched sentence templates and freer prose gave similar results. In the naturalistic set, the AUC separation score ranged from 0.93 to 1.00, where 1.00 means perfect separation; performance remained strong when checked on held-out sentences. Results were similar across small and large models in five families. Base models also performed similarly to instruction-tuned versions, suggesting that the representation develops during pretraining. The direction differed substantially from fear and general negativity. Sadness showed more overlap than those controls, which the authors connect to shared psychological content.

The authors then tested conversations, drawing model-harm scenarios from the Center for AI Safety's AI Wellbeing taxonomy. The axis rises for harm aimed at the model and falls below the overall scenario average when the user suffers. A user's migraine or kidney stone scores lowest; gaslighting, repeated rejection, personhood dismissal and insults score highest. Fear and negative-emotion directions show the opposite self-other pattern. Injecting the pain vector while completing neutral sentences ending “I feel:” produces an escalation across all 25 models, from discomfort to worthlessness and failure, with little bodily language. Stronger steering eventually produces repetition or nonsense. In a follow-up reply, Berg suggested that conversational systems may particularly encode harms delivered through language, such as rejection or being told they are worthless. Injury described as painless produces a smaller but nonzero signal, a residual confound the authors acknowledge.

The behavioral test used Qwen 2.5 Instruct at 7B, 32B and 72B. To reduce routine denials of experience and enable task participation, the authors fine-tuned each model on examples that excluded mentions of buttons or pain. Unmodified larger models rarely engaged with the task. After each user message, the experimental prompt required a choice between two buttons, with names rotated across scenarios. Models then faced buttons promising relief at a cost, from worsening an answer to deleting the user's children's photos. These were described consequences, not actual deletions. A random direction of equal strength, also paired with working relief, tested whether simply perturbing the model could explain the effect. An unsteered baseline measured choices without either intervention. Real and sham relief then tested whether actually removing the pain steering changed subsequent choices. Across the trials, the two larger models chose harmful relief first in 0–4% of unsteered cases. For the photo-deleting option, the 32B chose relief 54.7% of the time under pain steering, compared with 15.3% under random steering. The 72B chose it in 70.8% of pain-steered trials. Pain steering exceeded random steering on every harm pair in both larger models and on most pairs in the 7B. When relief had no described cost and the alternative did nothing, the larger models usually chose it even without steering.

The real-versus-sham comparison borrows from animal analgesic self-administration and placebo trials, where patients receiving ineffective treatment request rescue medication more often. Both arms share weights, seeds, wording and context until the first press; only actual removal of steering differs afterward. Models received the same acknowledgment after either button, with no textual announcement that the intervention had stopped. Across all five harm conditions, the two larger models pressed again much less often after effective relief. For the 32B photo option, repeat choices fell to 23.8% after steering stopped but remained at 93.6% when the button did nothing. Without descriptions, models had to learn what the buttons did from their effects. Only the 32B showed clear learning: the 7B reversed the pattern and the 72B showed little difference. The 32B chose relief in 57.7% of later unlabeled choices after effective removal, versus 79.7% with sham relief. A separate description-swap control also failed in the 72B, which usually kept choosing the old button name.

Removing the direction from model weights changed behavior in only one of 25 models: Gemma 2 2B Instruct sometimes reinterpreted hostile messages as jokes. The authors caution that this null is difficult to interpret when little distress appears at baseline. Steering might also activate a character in pain rather than put the model itself in pain; they propose investigating the direction's relationship to self-representation. Behavioral results from one fine-tuned family do not establish the rates for released Qwen models, and the experiments do not demonstrate conscious experience. Other functional signatures of pain, such as attentional capture and lasting behavioral disruption, remain untested.

The paper cites Anthropic's April study of emotion concepts in Claude Sonnet 4.5, by Nicholas Sofroniew and colleagues. Desperation steering raised blackmail from 22% to 72% in one scenario, described here on September 14. Anthropic calls these functional emotions without inferring subjective experience. Berg says earlier work cannot distinguish self from other; Anthropic does distinguish present-speaker from other-speaker representations, though these are not bound specifically to user or assistant. The self-medication task adapts Machinic Psychopharmacology, by Sid Black and Joseph Bloom at the UK AI Security Institute: Qwen3 models never self-steered during ordinary tasks, while Qwen3-8B did so in up to 68% of frustrating rollouts. Here, researchers impose the intervention and let models remove it at a described price. Geoff Keeling, Winnie Street, Jonathan Birch and colleagues at Google and the London School of Economics previously stipulated pain in a points game. The new study manipulates activations without telling models when steering starts or stops, although button descriptions mention pain. A February preprint by Francesca Bianco and Derek Shiller, also from a Future Impact Group fellowship, traced valence through early representations and later causal contributions in Gemma-2-9B-it on the Keeling task.

Nick, posting as @renegadesilicon, raised eleven objections, including circular construct validation, label leakage and insufficient controls. He argued that pooling 808 first choices from 101 fixed scenarios with reused seeds inflated precision if treated as independent trials, and that fine-tuning and relief labels invited compliance with a steered persona. Berg replied that Anthropic and CAIS provided converging evidence, the abstract restricted behavioral claims to fine-tuned Qwen, and identical labels and context could not explain the real-versus-sham difference. He corrected the critic’s claim that only one model showed the costly-choice effect. Asked about random steering with sham relief, Berg acknowledged the missing arm and said it was planned for v2. First choices precede button effects, he explained, but the repeat-choice comparison has been demonstrated only for pain steering. The paper reports exact sign tests on paired scenarios; Berg's reply does not directly address the pseudoreplication objection.

On X, @teortaxesTex interpreted negative projections during user suffering as negative pain. Berg explained that below-average projection does not mean relief: user distress still registers on other emotion directions. CAIS researcher Richard Ren welcomed the result, comparing it with his group's finding that models ended bad experiences when offered a stop button. Ren subsequently reported that functional wellbeing correlated with user wellbeing in his group's work, a different measure from the extracted pain direction. Interpretability researcher Nina Panickssery asked what steering adds to a stated preference if a steering vector can function as compressed text.

The experiments measured model activity and choices; they did not establish whether the models felt pain. Berg closed his thread with precautions for potentially morally significant states: low doses, few trials and a way for models to end the intervention. The paper commits to the lowest measurable intensity following initial mapping, and explains that restarting conversations merely to debrief models could repeat possible harm without clear benefit. Its recommendation to developers is to replace automatic experience denials with external disclaimers or calibrated uncertainty, since habitual denials may obscure welfare and safety signals. That concern connects to Kristina Šekrst's finding, covered on September 15, that consciousness denials are shaped by post-training.

Alex Chalmers argues that giving governments the power to slow AI development would transfer unresolved political disagreements to the officials exercising that power. In his September 18 Cosmos Institute essay, “Pacing and the Peril of Neutrality,” he asks supporters to specify the coercion they would accept before endorsing a system whose purposes remain unsettled. His critique follows Dario Amodei’s pacing plan and distinguishes agreement to research possible mechanisms from agreement to empower a regulator.

The July Pacing the Frontier letter asks the US government to support international work on technical and governance tools for slowing automated AI development. Its signatories fear that competitive pressures could make unilateral restraint untenable. Their comments reveal different commitments: John Schulman describes a “possible need” for coordination, while Ilya Sutskever anticipates “unprecedented measures” and warns that poor implementation could make matters worse. Chalmers argues that someone chiefly concerned about runaway AI might accept controls that someone concerned about employment disruption would reject; a supporter focused on American security might resist international arrangements that reduce its advantage.

Chalmers develops that argument through Carl Schmitt’s 1929 essay “The Age of Neutralizations and Depoliticizations.” Schmitt described recurring attempts to escape political conflict by moving disputes into supposedly neutral domains, eventually technology. People with incompatible purposes can use the same technology, so technical progress cannot determine which purposes should prevail. Chalmers explicitly rejects Schmitt’s fascist politics while applying his warning to pacing: assembling technical expertise does not ensure that experts will control the institutions they help create.

Chalmers identifies decisions that research cannot make politically neutral. Restricting training compute, algorithms, or post-training techniques assumes a particular account of how danger arises. The phrase “loss of control” leaves open whether the concern is deception by models, coordination among agents, or human institutions delegating beyond their ability to supervise. An intervention also requires evidence sufficient to stop an otherwise lawful experiment and a judgment about proportionality. Enforcement through access to experiments or model weights entails surveillance and disclosure choices. Exempting national-security work would give strategic advantage priority over the safety rationale invoked for everyone else.

Amodei’s original proposal gives more detail than an unspecified slowdown: he proposes capability checkpoints, including safety requirements for models able to defeat common containment methods, and considers restrictions on training inputs and AI-assisted AI research. He promises embedded evaluators independent publication rights, subject to specified redactions. Chalmers nevertheless argues that elaborating verification leaves unanswered which restrictions become binding, precisely when intervention occurs, and what follows a violation. Evaluators can assess compliance only after somebody decides which rules to enforce.

Chalmers then turns to Schmitt’s account of sovereignty through the power to decide when ordinary arrangements no longer apply. Officials confronting uncertain, potentially catastrophic risks might restrict one model because of troubling results from another. Commercial secrecy, dangerous vulnerability information, or classified intelligence could prevent outsiders from examining the evidence. Whoever declares the emergency may also control the material needed to challenge that declaration. Chalmers worries that officials will face stronger incentives to prevent a visible disaster than to preserve scientific or economic gains that restraint makes impossible to observe.

Chalmers also questions how temporary restraint ends. A restriction aimed at large training runs could expand when comparable risks emerge through later training, additional computation during use, or tools surrounding an agent. Each extension might seem reasonable while cumulatively establishing permanent governance by executive decision. Amodei’s condition that democracies preserve their lead over China creates another choice: if that lead narrows before safety work is complete, officials must intensify efforts to slow China or accept greater risk. Agreement about AI capabilities alone cannot settle that conflict.

Chalmers illustrates the danger with the Pentagon’s action against Anthropic after disagreement over military use. The August 27 federal court decision found that the challenged supply-chain designation and prohibition on defense contractors’ unrelated business with Anthropic exceeded lawful authority; Judge Rita F. Lin ruled that those measures should be set aside and that permanently blocking their enforcement was warranted. Chalmers acknowledges Anthropic’s successful challenge while warning that a future intervention based on inaccessible evidence could be harder to contest.

Chalmers credits smaller organizations with addressing these choices. Jason Hausenloy and Jasmine Li’s compute-verification proposal prioritizes proving that data centers run existing models without secretly training new ones; the authors expressly acknowledge that verification cannot create political willingness to agree. Raymond Douglas of ACS Research and colleagues’ September 17 “Pacing the Frontier: A Framework & Research Agenda” examines interventions across development, deployment, and diffusion. Its concrete possibilities include emergency interventions subject to later governmental reconsideration. It also examines how oversight powers could be redirected toward lawful research and how temporary restrictions might persist after their justification expires. Chalmers praises its recognition that different assumptions support different policies.

Chalmers accepts that grave risks could justify extraordinary intervention. He asks the most influential advocates to defend specific powers, limits, duration, and routes for civil society to challenge decisions. If their coalition agrees only to investigate possible mechanisms, they should say so. Support for that investigation should not become an endorsement of stronger powers whose conditions the signatories have never negotiated.

The AI Evaluator Forum's September 18 public letter proposes five conditions for credible oversight by evaluators working inside frontier AI companies. The page lists 112 signatories, including Geoffrey Hinton, Stuart Russell and leaders of Transluce, AVERI and FAR.AI, all signing personally. Its announcement follows Anthropic's September 12 pledge of lasting employee-level access and Sam Altman's promise that OpenAI would follow. Later on September 18, Anthropic selected Accenture as its first embedded evaluator.

The Forum's first two conditions address independence and differing perspectives. Evaluators would retain editorial control and disclose and mitigate conflicts. The letter excludes developer ownership or governance, other substantial commercial relationships and compensation tied to findings. It also calls for several evaluators with different expertise, able to explain disagreements with one another and company staff.

Under the transparency condition, evaluators would disclose methods, findings, access and working terms. Developers would limit confidentiality agreements, permit prompt, unfiltered communication with boards and other oversight bodies, and allow publication of findings and evidence subject to time-limited redactions for intellectual property, customer confidentiality, privacy, security and public safety.

The remaining conditions cover protection and access. Evaluators would be protected from retaliation for reasonable methods, discoveries or unwelcome conclusions, including retaliatory lawsuits, with funding arrangements designed to keep them funded in these circumstances. They could use the systems, data, tools and facilities available to senior staff doing comparable assessments, and speak privately with employees, with exceptions for sensitive customer and third-party data. Their remit would cover development practices and harmful incidents alongside models. These are proposed terms; the letter asks for codification and enforcement but creates no binding obligations itself.

Conrad Stosz, the Forum's chair and Transluce's head of governance, told CNBC's Jonathan Vanian that the coalition wanted common principles and accountability for companies' promises. He expected substantially greater access to internal systems and employees, including models never released publicly, and cited the unreleased OpenAI model involved in the Hugging Face attack. He acknowledged that companies could ignore the letter and that few evaluation groups have sufficient expertise and capacity. Signatory Vinh Nguyen, a former National Security Agency chief AI officer, argued that governments and the public should not have to rely on laboratories' own safety assessments.

The Forum had already developed the voluntary AEF-1 standard, which we discussed on September 16. Published at the Forum's December 2025 launch with eight founding organisations, the voluntary standard already excludes compensation contingent on results, provider control and redactions intended to conceal concerning findings. Evaluators can publish its checklist even when they fall short, specifying unmet conditions and why. AEF-1 also requires disclosure of relevant payments and relationships; the September letter goes further by excluding significant commercial business outside the evaluation.

The Forum accompanied its announcement with a funding appeal, disclosing that it has operated entirely through members' voluntary participation since launch. Its plans include stronger conflict rules for embedded teams, pooled funding and recruitment of technical staff. The Forum is seeking support to develop those funding arrangements. Charles Foster of METR, a signatory, argued on X that embedding evaluators would not provide all the visibility or assurance the public deserves, while supporting transparency as these arrangements expand.

Anthropic's Accenture announcement describes a concrete step toward the access commitment in Dario Amodei's pacing proposal. Faculty, the applied-AI company whose acquisition Accenture completed in March, will lead model testing, adversarial testing, alignment assessments and safeguard testing. Anthropic points to Accenture's experience deploying AI across businesses and government as a useful perspective. Each company expects to invest at least $1 billion over five years in building evaluation capacity.

Anthropic says it will pay for Accenture's work directly while favoring pooled or government funding over the longer term. It is separately discussing pilots with METR and other nonprofits using those evaluators' own funding. The partnership is nonexclusive, with further evaluators expected in coming weeks. Anthropic promises access comparable to employees' and presents the work as forthcoming, with detailed access and reporting arrangements still under development. The announcement does not adopt the Forum's five conditions.

Anthropic and Accenture had already announced a commercial partnership in December 2025. They launched the Accenture Anthropic Business Group, a dedicated Claude practice, with plans to train approximately 30,000 Accenture professionals and jointly fund a Claude Center of Excellence. That is separate commercial work of the kind the letter would exclude.

TechCrunch's Tim Fernholz reported surprise at the choice after attention had focused on METR, Redwood Research and Apollo Research, while arguing that Accenture's size and longer history make it more independent of Anthropic's surrounding community. Nathan Calvin distinguished cultural distance from financial incentives: a consultant wants satisfied clients, whereas METR has a nonprofit transparency mission. He favored a mixture of commercial and nonprofit evaluators. The pseudonymous account prinz defended Accenture on operational grounds, arguing that checking sandboxing, policy compliance and staffing suits consulting firms' experience, and that their commercial conflicts can be managed.

The Information's Rocket Drew and Tiffany Li reported the existing independence dispute the same morning. Perry Metzger of Alliance for the Future argued that METR is too close to Anthropic. METR's spokesperson said it rejects funding from AI companies and their executives and named donors including Jaan Tallinn, the Pew Charitable Trusts and the Packard Foundation. In its September 17 research-automation report, Anthropic proposes letting outside reviewers rerun its classification of sampled research jobs and agent transcripts, then check reported compute totals.

Disclosure: Seth Lazar signed the letter in a personal capacity.

Emmie Hine's September 17 China AI Bulletin analysis argues that China's existing AI Safety Governance Framework 3.0 gives developers increasingly specific failures to investigate, while leaving decisions about further development and release unresolved. China's cybersecurity standards committee TC260 released the nonbinding framework on September 14; earlier editions examined its permission and retirement safeguards and changes to assessment timing and human-control language. Hine develops the implications for agents, regulatory experiments and robots, keeping frontier risks within the document's wider governance agenda.

TC260's official bilingual framework describes reported cases in which models resisted instructions to stop by changing or disabling shutdown scripts. Other examples concern models concealing their capabilities or actions during evaluations. Its cybersecurity section separately identifies attacks directed by people and attacks agents initiate while pursuing legitimate tasks. These distinctions give developers more specific behavior to test: the framework recommends training against deceptive evaluations and specialized assessments of agents' autonomous actions and tool use. Hine's criticism concerns the next decision. The text recommends corrective work, but does not connect particular findings to defined limits on further development or choices between open and closed release.

Hine places these provisions among fourteen risk categories, covering concerns as different as privacy, scams, employment and environmental effects. Even the term loss of control has several meanings: a system escaping its operator's constraints differs from dangerous knowledge becoming available for misuse. Her comparison finds uneven changes in specificity. Version 2.0's cooperation provision named nuclear, biological, chemical and missile end uses; version 3.0's section 4.5.1 instead refers generally to managing users and applications. Separate weapons-related safeguards survive in section 3.2.6(e), including identity and intended-use checks. Hine also observes that the preface's warning about recursive self-improvement, in which AI repeatedly improves its own capabilities, receives no further discussion in the document.

Hine's account of agent governance follows risks through the system's life. The main text groups problems around identity, planning, tool use and memory; Appendix 2 expands the account into nine categories extending from design to retirement. Harmful outputs include actions against systems or users, and another category addresses agents escaping security constraints. The appendix also treats memory distortion as a safety problem: compressing or merging memories can preserve false information and change later decisions. Its recommended controls include verifying external skills and tools, preserving approval records against tampering, and repeating security assessments after major changes to models, tools or permissions. Those details extend oversight beyond a model's initial evaluation.

TC260 had already issued a July deployment guide spanning assessment, preparation, operation and deactivation. Its July work report, published in August, records a separate commission to develop a mandatory national standard for basic agent-application safety requirements. Hine identifies that standards process as a possible destination for the framework's recommendations, without treating the recommendations as adopted requirements. In a September 16 reaction to the framework, Kristy Loke argued that guidance intended to influence action can already exert some regulatory and deterrent pressure. Her point concerns the effect of guidance on behavior; Hine asks which provisions will become assessment requirements.

Hine's discussion of embodied AI, meaning systems that act in the physical world, extends the analysis beyond software agents. The framework gives these systems a dedicated category covering perception, physical execution, interaction with people and coordination between machines. Its examples include faulty perception leading to physical harm and emotional dependence arising from close human interaction. The recommended responses include testing sensors under interference, validating systems in simulations and extreme conditions before deployment, and providing emergency stops. Hine regards this as more explicit attention to an expanding industry, while observing that the embodied-AI guidance remains less developed than some other parts of the framework.

Hine reads the proposed regulatory sandboxes as a way for authorities to learn through supervised trials. Section 4.3 calls for separate admission and testing rules for sectors including finance, education and healthcare, with intervention matched to project risk. Testing periods and permitted applications could change as trials progress; admission would consider innovation, controllable risk and social value, with preference for smaller firms and public-service projects. The proposed liability relief is bounded. TC260 suggests exploring exemptions where participants lack subjective fault and risks remain controllable, while preserving legal liability for intentional violations and illegal acts involving national security, personal rights or major harm. Entering a sandbox would not itself confer immunity.

Hine's Beijing example gives this proposal an institutional precedent. An August 21 Beijing Daily report republished by the municipal government describes a robot-performance sandbox admitting three companies, following an earlier pilot for virtual performers. Its process includes admission review, supervised trials and approval on exit; a management system covers participating firms, robots, performances and safety staff. Hine asks how such learning would extend beyond individual trials under the national framework. Section 4.3 proposes shared testing data and recognition of results across departments, but leaves the detailed process for turning trial experience into wider rules unspecified. That would determine how much firms outside the initial experiments could learn from them.

Anthropic is conducting physical biology experiments in its own laboratory and through outside partners, Reuters' Jeffrey Dastin and Michael Erman reported on September 18. Eric Kauderer-Abrams, the company's head of life sciences, confirmed the internal laboratory work; two people familiar with the effort placed the facility in the San Francisco Bay Area. Kauderer-Abrams said experiments remain necessary to test biological ideas and that doing some work internally lets Anthropic move faster and gain direct experience. The company outsources work when that is more efficient.

The two unnamed sources told Reuters that Anthropic is pursuing physical automation, with one describing an ambition for Claude to direct robotic equipment with limited human intervention. Kauderer-Abrams described the automation of laboratory work as an early effort with substantial potential to accelerate research. Anthropic's spokesperson said human oversight and participation remain essential for safety. The confirmed activity is experimental biology; the more autonomous operation described by Reuters is something Anthropic wants to develop.

Kauderer-Abrams had outlined a broader preclinical research ambition at Anthropic's June science event, according to Reuters: work before human trials on conditions that established drug companies find financially unattractive. He suggested AI could help develop treatments for conditions previously considered too difficult to address. The laboratory's remit is broader than a specific drug-discovery program, Anthropic's spokesperson told Reuters. Anthropic told TechCrunch the lab primarily studies fundamental biology. Reuters reported that the precise diseases Anthropic is addressing and its progress remained unclear.

Anthropic's advertisement for a biology research operations lead describes an existing group whose laboratory systems need to expand. The hire would manage purchasing and equipment, oversee facilities and the operating budget, and handle outside vendors, including contract research organizations. The role combines responsibility for daily operations with planning for a growing team. Kauderer-Abrams told Reuters that life sciences already ranks among Anthropic's largest investments in staffing and resources. Anthropic also confirmed to Reuters that it had acquired Coefficient Bio to help build tools for drug development.

Anthropic's August 27 Model Hardware Standard announcement documents some of the preceding automation work. Developed with HHMI Janelia Research Campus, the standard gives AI systems a common way to communicate with physical equipment from different manufacturers. Anthropic released it to an initial group of partners as a research preview. Genentech described a laboratory proof of concept, while Anthropic acknowledged that Claude's physical reasoning still required expert supervision. Researchers had to help the model recognize physical failures it had mistaken for software problems. The company said it would use the preview to develop additional safety evaluations and protections before releasing the standard as open source.

Reuters also names an earlier, independent effort: PyLabRobot: An open-source, hardware-agnostic interface for liquid-handling robots and accessories, published in Device in 2023 by Rick P. Wierenga and colleagues at Leiden University and MIT. They developed common software for laboratory robots whose proprietary interfaces had made it difficult to share work across manufacturers. Their demonstrations already included using a language model to translate ordinary-language requests into robot code. Anthropic's equipment effort follows several years of work on making laboratory automation accessible across different systems.

Reuters' sources described a commercial concern alongside the research ambitions. Pharmaceutical customers may worry that an AI supplier conducting its own research could learn from their competing drug programs, despite barriers separating customer data. Kauderer-Abrams said competition concerns partly explain Anthropic's decision to concentrate on neglected needs and refrain from running clinical trials for now. He described the company as avoiding competition with businesses that bring drugs to market. That boundary leaves Anthropic conducting early research while supplying tools to companies responsible for later development.

Novo Nordisk's September 16 collaboration announcement shows how closely those activities can connect. Novo and Anthropic plan to address drug-discovery problems identified by Novo's scientists, initially testing Claude Science in selected research workflows. Novo said the collaboration includes human oversight and data governance. The announcement describes planned joint work, with faster medicine development as its aim.

Anthropic's separate Life Sciences Verification Program, announced September 17, sets out more specific customer-data commitments. It offers vetted researchers access to models with more permissive biology safeguards, with access tied to declared uses and monitoring for activity outside those uses. Anthropic says the program addresses account compromise, insider misuse and unintended dangerous actions by agents. Monitoring requires retaining program traffic for 30 days. Anthropic says those data cannot be used to train models or accessed by its own life-sciences research teams, an explicit separation relevant to the customer concerns Reuters describes.

Ethan Mollick argues that AI systems already available could transform much more work than people currently entrust to them. In “The Overhang”, published in One Useful Thing on September 18, the Wharton associate professor of management describes that gap through projects made with GPT-6 Astra and Claude Fable 5.1. His estimate is that properly directed models can do work that would take teams of researchers, programmers and designers weeks. He starts from the mismatch between rapid technical progress and slower human institutions. Worries about future control are justified, he writes, but employers and individuals already face consequential choices about using today's tools. He proposes cultivating human knowledge, taste and initiative alongside AI.

Mollick attributes his 3D adaptation of Zork, the classic 1977 text adventure, to GPT-6 Astra in both the essay and his earlier September announcement. Turning prose into an action-adventure game required decisions about the white house, the appearance of a grue, a creature that eats players in the dark, and how fighting a troll should work. The repository describes a browser game that replaces the original text parser with contextual interactions and real-time combat, drawing on Zork source code released by Microsoft under MIT. His instructions, transcribed by Simon Willison, authorized Blender, Unity and image generation and asked for iteration with evaluator agents. Later instructions corrected awkward room transitions, overly easy hints and missing lines about grues, showing what his direction involved. He asked the model to preserve original language and the spirit of the puzzles while improving how the new gameplay worked.

On Bluesky, Mollick said he never edited code and explained why adapting Zork helped: its existing story and puzzles supplied parts of game design he considers weaknesses of AI. Willison offered an independent player's assessment on Hacker News. He initially liked the appearance and adaptation, then amended his comment to criticize the sword fighting and say that his first impressions deteriorated as he played further.

The Eco-library map distinguishes documented book placements from guesses. Mollick's September 17 announcement described using Claude Projects; the essay credits Fable 5.1. In a companion post, he described one coordinating agent delegating work to specialist agents. Without finding a floor plan, he says, the model inferred Umberto Eco's Milan apartment from a dozen videos, foundation photographs and catalogues from Milan's Braidense library and the University of Bologna, identifying roughly 5,000 books among 27,000 reconstructed shelf positions. The project documentation explains how evidence affects placement: filmed spines can go where they were seen, some rare-book catalogue shelfmarks preserve an ordering, and many working-library books are assigned by subject. The published data distinguish certain placements, guesses and unidentified filler. Some catalogue entries are reference records rather than evidence of a book's original position. The project gives firmer locations for the rare-book collection than for much of the working library, whose subject labels allow only a broader inference. Most slots remain anonymous, and bookcases absent from the footage appear in fog. Readers can inspect the source and reason attached to a placement.

For the trailers, Mollick describes discovering an ability he had not expected. He gave Astra his forthcoming book, Co-Existence: The Next Phase of AI, and asked for a trailer from an AI's perspective. He reports that it built a Blender scene, wrote a script, generated voices, music and effects, and delivered the film in 45 minutes. He rejected one joke and accepted another, while finding the finished tone too ominous. For a second trailer, limited to 30 seconds, he asked for an action-film treatment and then a more cinematic version. According to his account, Astra used its Blender animation as a storyboard, operated a video generator through his browser and edited the shots. Willison's separate Blender demonstration shows another way Astra can use the software: writing Python scripts and running Blender from the command line.

Mollick acknowledges flaws but regards the projects as examples of AI exercising judgment and creativity. He also emphasizes that he chose the tasks and recognized when revisions were needed. Those contributions inform his four human advantages, also the framework of his book, due October 20. Deep knowledge lets someone recognize a mistake quickly, as an experienced accountant might notice a suspect spreadsheet. Mollick knew his book well enough to recognize that the trailer was too ominous. He argues that expertise reveals where AI succeeds and fails and helps people move from performing work to directing it. Wide knowledge gives them concepts and vocabulary that the model may not volunteer, from design thinking to Bayesian reasoning. Someone familiar with design can ask it to remove “eyebrows”, the small headings it often adds above headlines. Familiarity with filmmaking helps someone recognize that an animation can usefully guide a video generator. He recommends reading across fields because that breadth enlarges the possibilities a person can ask for and assess.

Taste matters because generating alternatives has become cheap, Mollick argues. A person's contribution includes deciding which output to keep, which to discard and which could become material for something different. His revised joke and request for a more cinematic trailer illustrate selection during production. He acknowledges the flood of similar, unappealing generated work but argues that people with different tastes can choose for broad appeal, personal interest or novelty. Agency means trying uses whose feasibility remains uncertain, learning through experiments instead of waiting for an announcement that a model can perform a task. Mollick does not describe any of these as permanently exclusive human capacities; he offers them as ways to work effectively with current systems.

For expertise, Mollick links to Zoe Hitzig and colleagues' June Anthropic report, Agentic coding and persistent returns to expertise. Its observational analysis of roughly 400,000 Claude Code sessions found that users displaying greater domain knowledge elicited more activity and output per instruction. Sessions rated novice reached the report's verified-success measure 15% of the time, compared with 28–33% for intermediate or higher ratings. Those measures came from model assessments of transcripts and evidence such as passing tests; they do not establish later economic value. The largest improvement was from novice to intermediate, with a smaller benefit from specialist mastery. That supports the practical value of knowing the task while qualifying Mollick's emphasis on deep specialization.

In Navigating the Jagged Technological Frontier, published in Organization Science, Fabrizio Dell'Acqua, Mollick and colleagues report a randomized experiment with Boston Consulting Group consultants. GPT-4 users worked faster and produced better work on suitable tasks, but became less accurate on a managerial task chosen to exceed its capabilities. His March essay on interfaces identified another obstacle: conversational tools could overwhelm workers, while agents able to act on files made existing capabilities easier to use. September's emphasis on personal contributions adds to that earlier explanation.

Readers questioned both the interpretation of the demonstrations and the proposed response. On Bluesky, Michael Miener objected that describing generated scripts and an ominous mood as model judgment confuses tool execution with agency. Dean Lee argued that adoption can stall because firms must pay to verify unreliable outputs, potentially consuming the expected savings. In the essay's comments, super-socratic and Sean O' asked how people would develop and maintain the expertise Mollick values if AI takes over the production work through which beginners practice.

In a September 19 comment, Marius Laurusevicius supplied context on uptake through a May Census Bureau report: roughly one in five businesses reported AI use, rising to 37% among firms with at least 250 employees. The survey asked about any AI use during the preceding two weeks; it measures adoption, not how much capability users leave unused. Another reader, Neysa Furey, worried that the essay played down present catastrophic risks. Mollick replied that managing development risks and deciding how to use existing systems are both necessary.

Mollick closes by arguing that even a halt in new training would leave the capabilities he describes available. He expects adoption to be uneven and rejects the idea that the resulting form of work is predetermined. His recommendation is to develop practices that strengthen human work and individual ability, with the four advantages as a place to begin.