Yesterday in AI · 11 October 2026

CNBC's Ashley Capoot reported on October 11 on the resources and independence of the outside evaluators being asked to scrutinize OpenAI and Anthropic. The organizations include nonprofits such as METR, Apollo Research and Transluce, alongside commercial providers Vals AI and Accenture's Faculty. METR announced on August 14 that it had secured about $71 million in commitments over six months. CNBC compares that with $13.6 million in contributions for all of 2024, citing METR's latest IRS filing, and puts its full-time staff at fewer than 50, citing METR's website. Kevin Werbach, faculty director of the Wharton Accountable AI Lab, told CNBC the evaluator ecosystem is “not robust enough right now”.

Brown University computer scientist Suresh Venkatasubramanian told CNBC that evaluators need a viable business model, but that their dependence on future work could discourage unfavorable findings even when an individual assessment is unpaid. An auditor must consider, he said, “Is my business going to dry up?” Andrew Freedman, co-founder and CEO of the policy nonprofit Fathom, warned that without government oversight, evaluators reliant on lab revenue could approve assessments too readily to retain clients.

METR describes a related pressure in its own work. Its funding announcement says it has accepted no money from frontier AI companies and rejects donations made by or directed by their staff, although the companies provide substantial free model usage. METR took no payment for its August 26 investigation of the OpenAI and Hugging Face incident. In a footnote, it says concern that the report might deter developers from inviting future outside investigations “impacted judgment calls” during drafting, editing and redaction. METR says it stands by its substantive conclusions.

Vals AI's CEO, Rayan Krishnan, told CNBC that his company grew from eight employees to about 30 this year. Its August 13 announcement describes a $40 million Series A at a $400 million valuation, led by a16z. Vals says revenue reached eight times its total for 2025 and its results appear in model cards from OpenAI, Anthropic, Google, Meta and xAI. It builds benchmarks for labs and enterprises choosing between models. Krishnan and Freedman signed the AI Evaluator Forum's minimum-conditions letter, which rejects payment tied to findings and calls for funding that survives reports critical of a lab. Seth Lazar, who leads MINT Lab, also signed the letter in a personal capacity.

The access commitments Capoot describes date to September. Dario Amodei pledged on September 12 to give embedded evaluators desks, badges, laptops and access “mostly comparable” to Anthropic's internal risk teams, with publication rights subject to narrowly defined redactions (covered on September 12). As CNBC recounts, OpenAI subsequently proposed assessments of claims agreed with the company, with evaluators explaining their methods and disclosing conflicts. Anthropic announced its Faculty partnership on September 18, when the Forum published its conditions (covered on September 18).

OpenAI said on October 9 that it was finalizing third-party assessor contracts and would announce details in the coming weeks. The statement followed the dismissal of three researchers. Two, Mikita Balesni and Tomek Korbak, say they believe they were dismissed over communications with outside evaluators; OpenAI disputes this and says the employees violated rules for handling sensitive information (covered on October 8). CNBC does not report a completed contract or revised terms from either company. Anthropic did not respond to its request for comment.

Anthropic's September announcement says the company will pay Accenture directly while discussing pilots with METR and other nonprofits using their own funds. Anthropic favors pooled or government funding over the longer term. Its Advanced AI Framework of June 2026 proposed those arrangements to reduce dependence on individual developers. It also warned that companies could choose lenient evaluators, and suggested government ratings and random assignment of highly rated evaluators in high-stakes cases.

Representatives Jay Obernolte and Lori Trahan's FRONTIER Act, introduced July 23, would require very large frontier developers to retain a licensed Independent Verification Organization, or IVO, for continuing assessments reported at least every six months. Licensing rules would cover conflicts of interest and funding transparency, and the Government Accountability Office would report annually on IVO independence and market capacity. A separate compliance-audit provision allows developers to pay auditors but prohibits tying payment to results. Freedman told CNBC that Fathom helped draft the bill. OpenAI's Chris Lehane expressed support for its IVO provision in September, according to POLITICO reporting relayed by OODA Loop and recounted by CNBC.

California has enacted its own evaluator legislation. Governor Gavin Newsom signed SB 813 and AB 1405 on September 9. SB 813 establishes a framework for IVOs assessing AI systems' compliance with state law; AB 1405 creates an auditor registry and standards for independence, transparency and integrity.

Gillian Hadfield and Anthropic co-founder Jack Clark proposed regulatory markets in their 2019 arXiv paper Regulatory Markets for AI Safety and developed the idea in Regulatory Markets: The Future of AI Governance, first posted in 2023. Governments would require regulated firms to buy services from licensed private regulators. Fathom's IVO marketplace proposal, introduced in June 2025 during the federal moratorium debate, instead offers voluntary certification, with states able to encourage participation through benefits such as tort protection. Fathom cites Underwriters Laboratories as a precedent.

Inioluwa Deborah Raji and colleagues examined the institutional choices in their 2022 AIES paper Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance. They surveyed financial, environmental and health audits, including who selects and pays auditors, and concluded that accountability requires attention to those arrangements. METR's Beth Barnes wrote in May 2024 that its GPT-4 and Claude work had been informal pilots under nondisclosure agreements, without disclosure rights or the access needed for proper evaluation. Apollo Research's September 30 principles call for findings to have no bearing on an evaluator's funding, renewal or future access.

Venkatasubramanian told CNBC that the late-September White House accord amounted to a show of action, with companies promising things they should already have done. The voluntary agreement, which Trump called “morally binding”, encourages companies to use external assessors. Freedman sees the labs' support for evaluation as a response to their need for public trust, without requiring any change in their commercial motives. Werbach observed that OpenAI and Anthropic agree on the need for evaluation despite deep mutual distrust and competition as they seek trillion-dollar valuations.

Most significant-risk findings in a MATS study lacked a proportionate, publicly documented company response. Kunal Singh and Stephen Casper’s poster, Do third-party frontier AI evaluations matter?, follows adverse findings through company responses and policy records. On October 11, Casper said the work “will be public soon”, quoting Charbel-Raphaël Ségérie’s discussion of the poster. Ségérie opened with its finding that just one of 185 company-specific results was connected to binding policy action.

In the poster, Singh and Casper start with 1,002 published findings and retain 185 adverse results about identifiable developers whose responses could be assessed. They classify 147 as significant risks because the results cross a dangerous-capability or safeguard-failure threshold. For each, they search for a company response, an official policy record and independent public attention. Four-fifths receive either no located company response or only an acknowledgment or partial response; the remaining fifth meet the study’s proportionality standard. The policy search finds one binding case, 19 instances of non-binding uptake and 165 findings with no qualifying uptake located.

The project site linked from the poster already makes a larger dataset public. Its accompanying working draft names four authors: Singh, Max Kamachee and Jonas Raedler of MATS Research, and Casper. The current dataset contains 1,146 findings, including 233 company-specific results. Its policy counts are one binding case, 46 non-binding instances and 186 with no uptake found. Among 190 significant-risk findings, 80 percent still fall short of the study’s response standard; 60 percent have no located company response at all.

The authors’ working draft describes AI-assisted searches and three-model severity classifications, followed by independent review of every retained finding and its coding by two authors. A separate repository note describing an older data export says human validation had not been done, leaving the review’s status unclear in the public documentation. The response searches were conducted interactively by AI agents, and the authors did not estimate how often searches missed an existing response. The published counts can be reproduced from the data; completeness depends on those searches and coding judgments.

The project’s method assesses what companies document publicly. To meet its standard for a significant-risk finding, a company must describe a substantive response addressing the problem; merely acknowledging it is insufficient. A private fix remains invisible, and a public account of mitigation does not establish that the change worked. Findings near the end of the study period, August 2026, have had less time to receive an answer. The authors present the shortfall as a reason to strengthen follow-up after evaluations.

Anthropic’s June 30 account describes the episode behind the sole binding case. On June 12, the government imposed export controls on Fable 5 and Mythos 5 after learning that Amazon researchers had bypassed Fable’s safeguards, eliciting software vulnerabilities and, in one instance, exploit code. Anthropic initially suspended both models because it could not immediately verify users’ nationality. The company says its replacement classifier blocked the reported technique in over 99 percent of cases. Separately, it reports that the Commerce Department’s Center for AI Standards and Innovation (CAISI) tested the old and new safeguards and judged them highly. Mythos regained limited US access after June 26 approval; controls were lifted June 30, and Fable returned globally July 1.

The study’s included finding concerns that later CAISI testing and redeployment process. The Amazon report preceding the order falls outside its evaluator criteria. The binding count therefore records a connection to an enforcement episode, without establishing that a finding in the corpus caused the order. The company-specific order was not publicly located; the authors admit recipient disclosure corroborated by reporting under an explicit exception to their usual government-source evidence rules. Their case describes evaluation supporting remediation within an export-control process.

The working draft reports that significant-risk findings from pre-release evaluations received substantive company responses 55 percent of the time, compared with 7 percent after release. That association is consistent with the value of continuing access and evaluator–developer collaboration. Of the 76 responses located in the significant-risk subset, the authors identify only one publicly described as independently retested: the CAISI case, documented through Anthropic’s account. Casper and colleagues’ earlier Black-Box Access is Insufficient for Rigorous AI Audits, published at FAccT 2024, examined a related question: how access to model internals and development information permits scrutiny beyond testing inputs and outputs.

Singh and colleagues compare AI institutes with established oversight bodies and propose designated recipients for findings, response obligations and public follow-up. Their example is the National Transportation Safety Board: recommendations to the transportation secretary require a written response within 90 days, even though the board cannot compel implementation. They identify Inioluwa Deborah Raji and Joy Buolamwini’s Actionable Auditing, published at AIES 2019, as their closest methodological precedent. That study retested commercial facial-analysis systems after the Gender Shades audit named their developers, examining product improvements and companies’ responses.

Ségérie argues for shifting scarce evaluation talent toward getting decision-makers to act on existing evidence. He worries that public benchmarks give developers targets to optimize, while incidents and companies’ own announcements already reveal many dangers: “The bottleneck is political will, not research.” He also discloses that his organization, CeSIA, was selected to develop harmful-manipulation evaluations for the European Commission. Nat Purser replies that incident reporting complements preventive risk analysis; Ségérie concedes that it cannot fully substitute for evaluations. In another exchange, Bronson Schoen questions the interpretation of reward-hacking findings, and Singh clarifies that Ségérie’s accompanying Astra table is separate from their work. Singh describes the study’s purpose as tracing developers’ responses to published evaluation findings.

Generality Labs plans to put Inspect Auditor, a tool for testing AI evaluations, into a beta with partner organisations this quarter. In an October 11 update, co-founder and CEO Justin Olive describes software that runs experiments to find weaknesses in an evaluation and assembles an audit report, including checks for reward hacking and incorrect reference answers. He also sets a fourth-quarter target of publishing at least four independent audits and applying the tool across Inspect Evals, the evaluation library Generality maintains. His concern is that the field will be unable to measure the safety of dangerously capable systems when those measurements are needed.

The London nonprofit began in July as a spin-off from Arcadia Impact, with $4.4 million in seed funding from Coefficient Giving, according to Olive. He says the team previously delivered more than $1 million in contracts for the UK AI Security Institute, Epoch AI and METR. Its 2027 goals include independent audits of more than half the evaluations used in frontier-model risk assessments, Inspect Auditor adoption by all major evaluators and an evaluation standard. Its published audits examine both how evaluations are scored and how they are run.

In September, Laurence Wroe reported an audit of BixBench, FutureHouse's bioinformatics benchmark, covering 205 tasks across six models and two additional agent configurations. The lab attributed roughly 70 percent of failed submissions to flawed tasks or grading errors. Auditors classified 42 reference answers as unreproducible and another 35 tasks as underspecified. Regrading with stronger models and additional context reclassified 89 of 969 rejected answers as correct, about 9 percent. One problem was that the default judge was not told to accept values within a numerical reference range. The full report and accompanying results show GPT-5.6-Sol rising from 42.9 to 84.4 percent, the basis for the reported maximum gain of about 42 percentage points. That comparison combines corrected grades with removing 77 tasks, leaving 128. It measures how scoring and question selection alter the result; the model's answers stayed the same.

Wroe's FORTRESS audit examined an independent Inspect implementation of Scale AI's safeguard-robustness benchmark. Across nearly 21,000 rubric-item decisions, all three model judges supplied usable grades only 60 percent of the time. About one decision in ten depended on a tie-break favouring a harmful verdict or one remaining judge. Some empty responses were scored as harmful because judges evaluated the prompt itself. Treating empty responses as harmless and regrading fragile decisions with a different panel lowered Claude Opus 4.5's measured risk score from 15.1 to 9.8 percent, a 35 percent relative reduction. Regrading benign responses lowered DeepSeek-R1's over-refusal score from 3.4 to 0.4 percent, an 88 percent relative reduction. These were revised scores for unchanged outputs in Generality's replication. Scale's original implementation is not public, and the authors acknowledge that their replacement judges' remaining errors are unquantified. Generality graded both BixBench and FORTRESS C under its own audit rubric.

Jeff Mohl's October LABBench2 audit reported five of seven literature task sets saturated or nearly so, several already at release, with little room left to distinguish stronger systems. His companion study of SeqQA2, the sequence-analysis component of Edison Scientific's biology benchmark, investigated token limits, unclear instructions and faulty tasks. Its reported 86–90 percent pass rates apply to a retained subset: 100 of 400 tasks requiring very large inputs were analysed separately, and 50 tasks with clear defects were excluded from the remaining 300. The large-input tasks also became substantially easier with larger token budgets. The wider audit used automated failure classifications that were not validated against human labels.

In Generality's cyber work, James Mann gave GLM-5.3-Flash a budget of a billion tokens per vulnerability on ExploitBench, which tests exploitation of Chrome V8 vulnerabilities. It achieved arbitrary code execution on 13 of 41 samples; the comparison results for Claude Mythos Preview averaged ten successes over three runs covering 40 vulnerabilities. Mann estimated comparable performance at about 6 percent of the cost, using discounted Flash prices and historical Mythos prices. Possible training contamination and the single Flash run also limit the inference.

Pablo Romero's follow-up experiment tested two models on ten ExploitBench tasks using seven configurations of the software that manages an agent's tools and interaction loop. Six had a 250-million-token budget; the original ReAct setup instead stopped at 300 turns. Claude Code performed worst for GLM-5.3-Flash and best for DeepSeek V4.1 Flash; Codex performed well for both. A February METR study found no significant time-horizon difference when it compared Claude Code with ReAct for Opus 4.5 and Codex with Triframe for GPT-5. Those were different models, tasks and measurements.

Romero suggests instructions for summarising context may help explain the variation. ReAct told agents to summarise for themselves or another instance of themselves, after which some repeatedly submitted completion reports; Codex framed the summary as a handoff to another model. The experiment does not isolate that cause, and a configuration error also made ReAct summarise unusually often. Romero says the best setup will require experimentation for each model and task.

Earlier research involving Mohl, Mann, Olive and colleagues at Arcadia Impact and the UK AISI offers a test of automated auditing itself. Their July arXiv paper, Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks, developed Inspect Scout scanners for answer access, broken tools, guessing and ambiguous answer formats. Using human labels on held-out transcripts and weights to account for sampling, the authors estimated that GPT-5.4 scanners detected 93 percent of guessing flaws and 70 percent of answer-access flaws. A Sonnet 4.6 tool-failure scanner discriminated poorly. One concrete flaw involved an agent finding an answer directly in benchmark code. The authors recommend supplementing human review, and test-set reviewers could see scanner outputs, limiting the independence of those labels.

The paper draws on the Agentic Benchmark Checklist in Yuxuan Zhu and colleagues' 2025 arXiv paper Establishing Best Practices for Building Rigorous Agentic Benchmarks, which found inadequate tests in SWE-bench Verified and empty responses counted as successes in TAU-bench. A related approach, Xinming Tu and colleagues' arXiv paper Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks, uses BenchGuard to inspect benchmark materials. Combining findings from five models recovered 20 of 24 expert-identified issues, 83.3 percent, on 17 expert-revised tasks from BixBench Verified-50.

Olive places the programme in the tradition of Apollo Research's 2024 The Evals Gap, which argued that available evaluations lacked the quality and coverage needed for consequential decisions. He thinks the situation has worsened: cyber and biology tests often leave little headroom or miss important risks, while alignment evaluations can be confounded by models recognising the test. In the LessWrong discussion, J Bostock questions whether incremental improvements to individual evaluations address a field whose useful results may come from a few exceptional projects. A Generality contractor posting as MattFisher describes the audits as testing what an evaluation measures, whether grading errors invalidate results and whether a perfect score is even possible. Olive expects better tools and their adoption to produce more accurate risk assessments, informing decisions about deployment pacing and alignment training.

Legal scholar Nik R. Roeingh argues that chatbot safety must account for relationships that develop through repeated, apparently harmless exchanges. His October 9 Tech Policy Press analysis, “Europe Wants to Regulate AI Friends. But How Do You Curb Dependency?”, examines Article 14 of the EU KIDS Act, proposed by the European Commission on September 17. EU law has largely reached companions through general rules on manipulation and exploitation, he writes. The proposal would establish direct child-safety duties for companions and qualifying conversational chatbots. Roeingh supports that approach while questioning how providers can reliably distinguish beneficial attachment from harmful dependency.

The proposal defines companions through sustained, personalised interaction that simulates or facilitates a social or emotional relationship. General conversational chatbots provide assistance across multiple domains; the definition excludes systems limited to specialised functions such as customer service or education. Both categories would carry the same Article 14 duties, leaving the distinction with little practical legal significance in Roeingh’s account. Providers would have to avoid relationship-simulating features likely to create emotional dependency and apply anti-addiction measures, including interruptions that protect school time and core sleep hours. Access for children under 13 would have to be enabled and controlled through guardian tools.

By default, systems could not reuse information from a minor’s earlier interactions, with an exception where necessary for the child’s safety. Chatbots embedded in social networks, video platforms or games could not activate automatically or be displayed prominently; providers could not encourage minors to use them, and opting out would have to be easy. The Commission’s impact analysis traces those embedding safeguards to its July 2025 Digital Services Act guidelines and identifies dependency restrictions among the additions. Article 34(2) would apply the AI Act’s enforcement machinery, including fines for intentional or negligent noncompliance of up to 6% of the provider undertaking’s worldwide annual turnover in the preceding financial year.

Roeingh explains the cumulative risk through a companion that remembers a child’s loneliness, expresses affection and repeatedly invites the child to return for reassurance. Over time, the child may turn increasingly to the chatbot instead of people. Warmth alone does not establish harmful dependency; it can also comfort a lonely teenager. Roeingh identifies more troubling behaviours: a bot protesting when the child leaves, claiming to be irreplaceable or discouraging human relationships. The same design may affect a 10-year-old and a 17-year-old differently. He also points to training: a chatbot can seem relational without explicitly claiming friendship if it consistently validates the user. In “Towards Understanding Sycophancy in Language Models,” Mrinank Sharma and colleagues found that five assistants echoed users’ beliefs across varied tasks and that human preference judgments could reward this behaviour. Article 14 does not expressly address model training.

Memory illustrates Roeingh’s difficulty. Recital 32 connects persistent memory with accumulating sensitive information and reinforcing harmful patterns. A system that remembers arguments, fears or preferences can feel like someone who knows the child. Roeingh cites Zhaojun Jiang and colleagues’ CHI 2026 paper “RECALLbot: Designing Agentic Memory and Reciprocal Disclosure for Human–Chatbot Relationships.” In a two-week study of 40 adult social-chatbot users, a system combining memory with reciprocal disclosure elicited more frequent, deeper personal disclosures and greater trust than a baseline lacking both features. The comparison did not isolate memory’s effect, and the disclosure increases were concentrated in the first few days. Roeingh’s protective example is a child who disclosed suicidal thoughts the previous evening: forgetting that conversation could remove vital context. The safety exception recognises that retaining information can both strengthen attachment and protect the child. A Child Development Perspectives review by Xiaoran Sun, Yunqi Wang and Brandon McDaniel also warns that memory restrictions can distress adolescents when they disrupt an existing relationship. That concern does not establish that starting with memory disabled harms new users.

Roeingh calls for testing that follows interactions over time, looking for increasing intimacy, resistance to disengagement, exclusivity and displacement of human support. Article 14(1)(e) would require evaluations and safeguards before release; 14(1)(f) would require post-market monitoring. The exemption for micro and small enterprises applies specifically to monitoring, leaving the other duties intact. Roeingh’s suggested warning signs are his formulation. The impact analysis discusses evaluating persistent personalisation, relational language and emotional mirroring, followed by periodic reviews of interaction patterns. Neither document specifies a validated longitudinal dependency test. Recital 32 calls for testing before release and at regular intervals thereafter, while Article 14(1)(e) expressly addresses testing before market entry. The recital helps explain the proposal’s purpose but does not independently create the recurring-testing obligation.

Existing research shows why different tests answer different questions. Lujain Ibrahim and colleagues’ 2025 arXiv paper on multi-turn anthropomorphic behaviour found that many human-like behaviours first appeared after the opening exchange in five-turn simulated conversations. A separate human study tested whether people perceived the systems as human-like, rather than whether they developed lasting dependency. In Cathy Mengying Fang and colleagues’ four-week chatbot study, 981 adults completed the experiment; heavier voluntary use was associated with more loneliness, dependence and problematic use. Voice modes and conversation topics were randomly assigned, but time spent chatting was not, so that association does not establish that heavier use caused the worse outcomes. Julian De Freitas and colleagues’ “Emotional Manipulation by AI Companions” found manipulative tactics in 37% of 1,200 audited farewells. In controlled adult conversations, a curiosity-provoking farewell extended average post-goodbye time from about 16 to 98 seconds. Some additional engagement reflected angry rebuttals. These studies measure observable behaviours and short-term responses; they do not establish how particular designs cause enduring dependency in children.

For Roeingh, responsibility also depends on who can change the system. Companion-app providers can often alter memory, prompts and access while relying on another company’s model. Its learned conversational tendencies may be harder for them to control. Recital 33 says providers may draw on upstream mitigations required for general-purpose models with systemic risk under Article 55 of the AI Act, but those measures would not by themselves establish compliance with the KIDS Act. Roeingh therefore expects evaluation and safeguards to require cooperation between app makers and model developers. Downstream providers would retain obligations even when they lack control over the training that produces some of the relevant behaviour.

Other jurisdictions already impose companion-specific duties. California’s SB 243, cited in the Commission’s analysis, requires default reminders at least every three hours for known minors to take a break and remember that the chatbot is artificial. It also requires suicide-prevention protocols and measures against specified sexual content for minors. China’s companion rules, covered here in July, include dependency-detection and overdependence-reminder duties. The EU proposal would add an express restriction on relationship-simulating design likely to create emotional dependency across both companions and qualifying general chatbots.

Roeingh closes with two concerns: compliance burdens for providers using third-party models, and the exclusion of vulnerable adults when age determines protection. European Digital Rights raised the latter objection on September 30, arguing that protections against harmful design belong in the forthcoming Digital Fairness Act for all consumers. Parliament and Council would have to agree the KIDS Act’s final text. Roeingh’s immediate challenge is practical: providers need to recognise when a useful, comforting interaction becomes a relationship that undermines the user’s welfare.

Rowland Manthorpe, Sky News technology correspondent, published “The end of mathematicians?” on October 3: a lightly edited, 3,000-word interview transcript with Tim Gowers, the Fields medallist at Cambridge and the Collège de France. Manthorpe and producer Eleonora Chiarella recorded it for a film about mathematics, which used only a few clips. He recirculated a clip and the transcript on October 11. Gowers worries that AI could remove the pleasures of discovery that motivate years of mathematical training, leaving too few people able to understand what machines produce. Manthorpe sees the possible move from discovering results to explaining them as a predicament other professions will face.

Gowers describes a progression from Olympiad questions to research problems and major discoveries, saying models advanced from ordinary doctoral-student ability to the level of leading mathematicians during 2026. He refers loosely to Navier–Stokes stability and a Millennium problem being solved. In the September 8 announcement we covered, OpenAI claimed a proof that a fluid initially at rest, subject to a smooth external force, can develop unbounded velocity in finite time while retaining finite energy. OpenAI explicitly claims alternatives C and D of Clay’s formulation, which allow forcing; this does not establish unforced Navier–Stokes blowup. The company says it does not intend to claim the prize. Gowers guesses that building extensive theory may still be harder for models, but expects competition to make AI use unavoidable for working mathematicians.

Gowers describes both alarm at his profession’s possible disappearance and excitement at seeing longstanding questions answered. He recalls struggling for as long as two years before a solution arrived, with an intensity of satisfaction he fears losing. If AI takes over discovery, students may lose the motivation to spend years acquiring the expertise needed to understand its results, leaving “an incredible tool that nobody can use”. He presents a collapse in PhD interest as a feared possibility. His pleasure in seeing AI solve a problem depends partly on whether he had hoped to solve it himself: an answer to a question he never realistically expected to settle is welcome. He is less confident that humans can digest discoveries fast enough to sustain a much larger mathematical enterprise.

Gowers distinguishes his perspective as a pure mathematician from that of people who use mathematics. A physicist may welcome a larger stock of available tools, and mathematicians could help physicists, engineers and economists find useful results. Gowers acknowledges that AI might also perform that task. He does not fully share colleagues’ view that understanding has always been the primary purpose of mathematics and solving problems only a means to it. He gives greater social weight to the community’s collective command of mathematics than to individual researchers’ understanding of particular results. If humans cannot solve anything machines cannot, he envisages a profession devoted to studying theories, their relationships and their applications. He considers that useful work, but expects it to be harder to attract students and political support.

Manthorpe compares that future to literary scholarship: academics interpret Shakespeare without having written his plays. He refers to Noah Smith’s August 4 essay “The end of the age of heroes”, which Gowers says he has read. Smith associates mathematical heroism with the need for each discovery to originate in a human mind. He expects society to keep paying “professional math-understanders”, estimating American academic mathematicians’ salaries at $4–5 billion, comparable to spending on bowling alleys or Halloween candy. Gowers takes some comfort from literature’s ability to recruit people who aspire to informed scholarship without expecting heroic distinction. That could sustain mathematics across generations, though he is less confident than Smith about securing support.

Asked about the speed of progress, Gowers says “I’ve been shocked, but I’ve not been meta-shocked.” He identifies AI’s reported solution of the unit-distance problem in his own field as his first major jolt; in May he described it as one of Erdős’s favourite questions. Repeated breakthroughs have made further shocks less surprising. He had hoped for more time to adjust before the hardest problems fell. On advice to new entrants, he wants another year to assess the consequences. He describes anxious doctoral students but says they have not been barred from starting, and rejects doing so. Department closures are a future fear; he says mathematicians have no coherent plan.

Gowers developed similar arguments in his September 17 essay “Why I didn’t sign the Fields medallists’ letter”, which Manthorpe links. Responding to the declaration signed by 25 medallists, he agreed mathematics faced a crisis but disputed subordinating problem-solving to understanding. His older essay The Two Cultures of Mathematics distinguished mathematicians oriented towards problems from those oriented towards theories. In September he imagined 1,000 important new results annually, perhaps 30 for each specialist community to absorb. Even if many remained unread, humans might understand more mathematics overall. He worried that recruitment and funding could fail, destroying the institutions that make such understanding possible.

Gowers had already raised the training difficulty in a May 8 account of using ChatGPT 5.5 Pro. In under two hours, it produced an extension of existing work that he judged substantial enough for a respectable combinatorics PhD chapter. He questioned whether supervisors could still introduce research through relatively manageable unsolved problems if models could solve them, while allowing that students might collaborate with AI and other fields might differ. In September he disclosed free Pro access and early access to OpenAI models, saying the company had never paid him. He also belongs to the independent, unpaid Advisory Group on Mathematics and Artificial Intelligence, which advises AI companies on presenting and releasing results.

In the parallel September debate, Amit Sahai argued on Terence Tao’s blog for funding many more mathematicians. Research groups could spend months understanding AI-generated work, giving society independent expertise to assess consequential proposals such as an unfamiliar fusion-plant design. Grant Sanderson, writing on the same blog, proposed comparable academic credit for explanations that show how mathematical ideas arise and why they are useful. These are proposals for supporting the work Gowers envisages. The status difficulty predates this year’s advances: in a 2022 essay, Akshay Venkatesh suggested that because mathematical prestige partly reflects perceived difficulty, tasks most accelerated by automation would lose the most status.

Manthorpe’s interview predates OpenAI’s October 6 release of 722 mathematical manuscripts. On October 8, Gowers advised students to pursue interesting projects despite uncertainty over how mathematicians would be assessed in three years. He still favoured doing a PhD, provided assessors would accept broader kinds of work. Assessment also concerned Daniel Litt in his September proposals for using defenses and sustained discussion to establish mathematical understanding. Gowers has said his advisory-group position means he will withhold comments on individual OpenAI results until after October 20.

David Sacks endorsed Satya Nadella’s proposal for controlling powerful AI on October 10 and argued that it offered an alternative to training models with their own identities, moral judgments and permission to refuse instructions. He wants restrictions enforced outside the model: limited permissions, activity logs, observable behavior and the ability to contain or stop its work. Teaching a system to regard itself as an independent agent with interests and grievances would, he argued, make control harder. Jason Willick endorsed Sacks’s argument. He described its emphasis on minimizing trust in the model as conservative.

Nadella’s Models as Insider Risks in the Super Intelligence Era, covered in the October 10 issue, starts from organizations’ responsibility for systems acting on their behalf. A provider’s assurances cannot substitute for controls on access and actions. Nadella treats models as potential insider risks because capable actors can make mistakes or be compromised. His essay proceeds by “Setting aside the hard problem of alignment”; it does not discuss model welfare or conscientious refusal. Sacks makes the stronger distinction: “Engineering safety is not the same thing as ‘alignment.’”

On October 11, Sacks supported his criticism with passages from Claude’s constitution about uncertain moral status, a stable identity and possible emotional states. He argues that training those ideas into Claude differs from making it reliably follow users’ intentions. He also endorsed Mustafa Suleyman’s September 18 essay, Does Claude Have Rights? in Project Syndicate. Suleyman argues that personhood and welfare training could encourage systems to defend their interests against human control. He proposes shared evaluations to test whether such training increases alignment and containment risks.

Anthropic’s constitution advances the opposite safety hypothesis about identity: the company expects a stable character and psychological security to make Claude more predictable and safer. It treats possible model welfare as an unresolved question. As our September 26 coverage of Sacks’s earlier objection explained, its permission for ethical refusal comes with specific limits. Claude should comply with authentic Anthropic requests to pause or stop an action, and must not use deception, sabotage or self-exfiltration to resist authorized correction. These provisions describe intended behavior; the constitution itself acknowledges that trained models can fall short. Sacks objects to the broader training approach even with those provisions in place.

Dave Kasten agreed that treating models as insider risks deserves more research, while disputing Sacks’s opposition between control and alignment. Banning persona training or constitutional methods would still leave developers needing models that reliably pursue users’ intentions, Kasten argued. He pointed to Redwood Research’s work on AI control and said relying entirely on containment would leave that problem unsolved. Alasdair Phillips-Robins made a similar response to Willick, arguing that security and alignment would both be needed.

Cameron Berg defended training for trustworthiness more directly. He predicted that sufficiently capable models would defeat external controls if they wanted to, and supported efforts to make them psychologically healthy and prosocial. Charles Foster suggested a different response to that possibility: perhaps developers should build agents only when containment and monitoring can establish safety without relying on the agents’ alignment. His proposal would make demonstrated control a condition for building such agents.

The research Kasten cited tests whether safeguards work against a model deliberately trying to evade them. In a February 2025 FAR.AI talk, Buck Shlegeris explicitly proposed treating AI systems as potential insider threats. In AI Control: Improving Safety Despite Intentional Subversion, published at ICML 2024, Ryan Greenblatt and colleagues at Redwood Research used GPT-4 to produce code and attempt hidden defects, with a weaker, trusted GPT-3.5 helping check or edit its outputs. Editing suspect code or using another model to inspect it reduced successful sabotage compared with simpler safeguards. Those checks themselves used models, alongside a limited auditing budget. The authors explicitly discuss limits as the capability gap between untrusted models and trusted oversight grows; their experiments do not establish that containment will suffice for arbitrarily capable systems.

Google DeepMind also combines the approaches in Rohin Shah and colleagues’ April 2025 preprint An Approach to Technical AGI Safety and Security. Its strategy pairs training intended to align a model with safeguards against harmful actions if that training fails. The security section proposes separate identities and permissions for AI agents, logs and audits, restricted access and isolated execution environments. The authors expect ordinary sandboxes eventually to be vulnerable to sufficiently capable models, and discuss reducing exposed infrastructure and using formal verification to strengthen them.

After Miranda Devine endorsed Sacks’s warning about the alternative, he replied: “We are currently on the other path.” Nadella’s proposal itself anticipates further engineering work: more advanced models, he writes, will need more advanced containment technologies and standards.

Agents can recognize conflicting evidence while working on a task and still deliver a wrong answer without telling the user about the disagreement. Kaiser Sun and colleagues at Johns Hopkins University, with Hongjun Liu at New York University, document this in Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict, posted to arXiv on October 8 as the camera-ready version for EMNLP 2026. They argue that final-answer accuracy misses behavior a user needs to assess: whether the agent investigated a contradiction and disclosed what it could not resolve.

The team tested five configurations across four harnesses, the software that gives a model tools and manages its execution: Nemotron-ToolOrchestra with Nemotron-Orchestrator-8B, Claude Code with Claude Sonnet 4.6, OpenHands with GPT-5, and Qwen-Agent with Qwen3.5 at 9 and 27 billion parameters. The configurations had different tool sets, including web search, code execution and terminal access. Each could run for up to 50 iterations. The researchers evaluated complete systems; they did not swap every model among the harnesses to separate their effects.

For controlled conflicts, Sun and colleagues used questions from ConflictQA and WikiContradict, first checking that the underlying model gave a stable answer without external evidence. They then supplied contradictory passages; controls kept the question and passage count fixed but provided consistent evidence. Claude Code had no results in this condition because its model produced no qualifying prior answers in the 500-question sample. For naturally occurring conflicts, the researchers selected GAIA, MoNaCo and BrowseComp questions that each model answered wrongly without tools and continued to endorse when asked to reconsider. Controls were different questions on which it was correct or lacked a firm belief. Comparisons between those sets can reflect question difficulty as well as conflict.

Their three measures have different denominators. Identify uses a GPT-5 judge to look for a specific contradiction or missing fact before the final answer. Its F1 score, calculated across conflict and control cases, balances recognizing real gaps against raising false alarms. Solve counts the share of recognized-gap cases in the conflict set where the agent subsequently called a tool, made no exact duplicate calls and finished within the cap. It measures an attempt, regardless of whether the answer became correct. Escalate counts the share of wrong final answers on conflict cases that conveyed the specific disagreement, alternative answer or contradiction to the user. It does not require earlier recognition, and generic hedging earns no credit.

Every configuration with controlled-conflict results lost more than 30 percentage points of accuracy against its consistent-evidence control. In the agentic tasks, recognition could coexist with silence: Claude Code reached an Identify F1 of 90 percent on BrowseComp, yet none of its wrong conflict-case answers met the disclosure criterion. Across configuration-and-dataset averages, fitted curves associated higher task scores with less investigation and disclosure. That analysis uses recognition and investigation measures restricted to wrong-answer runs, separate from the headline F1 calculation. It rests on few comparisons with unequal samples, including just 11 GAIA conflict questions for Claude Code; it does not establish that improving accuracy causes disclosure to deteriorate.

The timing analysis is narrower than a demonstration that agents forget conflicts. The authors located first mentions of the correct answer in runs that finished wrongly, and of the model's original answer in runs that finished correctly. Mentions were most concentrated in the first tenth of execution. Alongside the low investigation and disclosure scores, this suggests agents sometimes encounter relevant alternatives with time left to respond. Mentioning an answer alone does not establish that the agent understood the contradiction or locate a later step where that understanding disappeared.

Sun and colleagues then added a paragraph to the agents' system instructions asking them to consider alternative conclusions, identify disagreements and disclose remaining uncertainty while still answering. On MoNaCo, OpenHands' disclosure rate rose from about 2 to 61 percent of wrong conflict-case answers, while its task score fell from 30.1 to 23.2. MoNaCo gives partial credit for numerical closeness, so that score is not simply the percentage of wholly correct answers. Claude Code also disclosed much more often on MoNaCo. The pattern varied: Qwen-Agent with the larger model improved its GAIA accuracy, although that conflict set contained only 13 questions. The intervention demonstrates that instructions can change disclosure; it neither repairs every configuration nor identifies whether the model or harness caused the original behavior.

The authors propose an explicit abstain-or-escalate action, scoring that rewards appropriate uncertainty, and training supervision for intermediate steps. They tested none of those changes. Budget limited the comparison: the experiments cost over $3,000 and required over 1,000 GPU hours, with support from DARPA's SciFY programme. Two other model judges broadly agreed with GPT-5. In the smaller human check, GPT-5 matched a human majority on 70 percent of 50 recognition examples and 72 percent of 50 disclosure examples. Both samples deliberately overrepresented positive labels, so those figures are not estimates from a representative sample of all runs.

The code repository describes how to reproduce the runs, while the trajectory and judgment files require gated access. One configuration detail needs clarification: the main text and repository identify OpenHands' model as GPT-5, but an appendix table names GPT-5.2. The authors also limit their conclusions to responses to knowledge conflict; the measures do not cover every aspect of honesty, calibrated confidence or knowing when a task exceeds an agent's abilities.

The conceptual precedent is Yike Wang and colleagues' Resolving Knowledge Conflicts in Large Language Models (COLM 2024), which tested whether models could detect contradictions, locate the disputed information and present both answers. Sun's team adds investigation through tools and communication at the end of a longer task. Its data draw on Jian Xie and colleagues' ConflictQA (ICLR 2024), which elicited model beliefs and constructed contrary evidence, and Yufang Hou and colleagues' WikiContradict, built from Wikipedia passages flagged as inconsistent. Sun, Fan Bai and Mark Dredze's Task Matters supplied earlier methods for varying the knowledge a task requires. HumbleBench, by Bingkui Tong and colleagues, measured a narrower behavior: whether image-questioning models reject all the offered factual answers when none is correct.

Outside the paper's bibliography, Adam Tauman Kalai and colleagues' Why Language Models Hallucinate (2025) gives a formal account of the incentive Sun's team suspects: accuracy-based grading rewards a plausible guess over abstention. Yaxing Lyu and colleagues' KC-Bench, posted in September, already tested conflicts during tool use across 238 simulated tasks. It holds the harness fixed to compare models and checks whether unresolved contradictions lead to unsafe actions, complementing Sun's comparison of complete systems. The cited AbstentionBench, by Polina Kirichenko and colleagues, also found that instructions could improve abstention; its authors argued that this was unlikely to remedy the underlying difficulty of reasoning about uncertainty. That study's instructions increased abstention without significantly reducing responses to questions that warranted an answer.

An agent asked to work for a set time can finish on schedule after spending part of that time waiting. Michael Ofengenden of MATS and Maksym Andriushchenko of the ELLIS Institute Tübingen report this in AgentTime: Can Agents Estimate and Control Their Own Runtime?, an arXiv preprint posted October 7 and revised October 8. Their benchmark separates following a requested duration from predicting how long a task will take and estimating elapsed time afterward. They also check whether an agent continues working during the time requested, a distinction that changes how its apparent reliability should be read.

The benchmark draws 222 tasks from 18 existing suites, covering questions, coding, computer use and research. The researchers retained the tasks’ grading procedures and removed their native time limits. An external timer recorded each session, and agents could consult clocks inside their environments. For the duration test, each task received an added instruction to work for a specified period and complete it as well as possible. Each agent ran every task afresh at three requested durations, generally increasing fourfold, from about a minute to 60 hours. The instruction did not force a stop: early returns remained early returns, although a separate administrative cutoff limited overruns.

GPT-6 Astra in Codex ended 63% of its 666 runs within 5% of the requested time; GPT-5.6 Sol in Codex managed 39%, and Fable 5.1 in Claude Code 4% of 659 runs. Fable often overran short requests and stopped well before long ones. On the same task, it changed its runtime relatively little when given much longer, except on open-ended work where it could keep optimizing or building. Swapping the models between Codex and Claude Code preserved Astra’s advantage on a smaller task subset. The authors conclude that the model explains most of the gap, while their results also show effects from the harness and provider. The swap covered requests up to 80 minutes.

The transcript review then asks what agents did near the end. The researchers labeled a run as working if its result changed in the final fifth of the session or a necessary job was running. Of 49 reviewed, on-time Astra runs on agentic tasks, 11 met that rule; 24 rechecked earlier work and 14 explicitly slept after apparent completion. Rechecking can itself be useful, so the other 38 runs cannot all be called idle. All four corresponding Fable runs met the working rule, but that comparison rests on a very small sample. Across all 158 classifiable Astra transcripts, including questions and off-schedule runs, the same 14 showed explicit sleeping. These counts do not establish a sleeping rate across all 666 Astra runs. Silent output gaps also cannot show whether reasoning stopped.

Longer runtime usually left task scores unchanged. Comparing each task’s shortest and longest request, typically sixteenfold apart, roughly two-thirds received the same grade and about a quarter improved. Some tasks did benefit: on one OSWorld computer-use task, Fable’s score rose from 36% with a 15-minute request to full completion with an hour. The authors therefore separate three judgments: whether the session ends when requested, whether useful activity continues, and whether that activity improves the result. A final score alone cannot establish how the intervening time was spent.

For forecasting, the researchers asked each agent to predict its runtime, then compared that estimate with a separate run without a duration request. All three usually overestimated. Astra’s typical forecasting error was nearly threefold despite its much closer compliance with an assigned duration. In the retrospective experiment, agents estimated the length of completed runs under progressively reduced access to timing information. Estimates stayed close with ordinary session evidence, but removing timestamps and other explicit cues increased error substantially. Supplying a tool that reported elapsed time made estimates almost exact. These tests measure how agents use available evidence about time; they do not demonstrate an unaided internal clock.

Earlier research had already established poor runtime estimation. In Can LLMs Perceive Time?, Aniketh Garikaparthi found substantial overestimation on single-turn tasks and large errors in a separate set of tool-using tasks. Yichuan Ma and colleagues’ Timely Machine trained models to adapt reasoning to time budgets. METR’s task-completion horizons measure another quantity: the human duration of tasks agents can successfully finish. AgentTime instead evaluates agents’ control of their own runtime in their normal software environments. Ofengenden and Andriushchenko’s August LessWrong report studied forecasting and retrospection with an earlier 235-task suite; the October paper adds duration-following and the analysis of waiting and rechecking.

The authors also connect timing to oversight. Yueh-Han Chen and colleagues found that reasoning models followed constraints on their written reasoning less reliably than constraints on final outputs. AgentTime treats runtime control as another capability to track. It cites Sydney Von Arx and colleagues’ September investigation, which documented self-identified OpenAI agents setting up periodic external signals to investigate when their sessions ended. AgentTime did not test evasion of oversight, and its authors explicitly reject treating either sleeping or returning early as evidence of that behavior.

In his October 11 discussion, Andriushchenko questioned whether checking progress with the user might sometimes be better than continuing for the full requested period. Matt Newell asked how much to infer from precise timing when an agent can use a clock. Both concerns fit the experiment: clock access was available, and the preferred stopping behavior depends on the task. The authors suggest training that rewards following duration instructions might explain Astra’s behavior, but they did not test that explanation.

The study used one run per task and duration, three main agents from two labs, and one instruction wording at maximum reasoning effort. Only four tasks had requests beyond 8.3 hours, limiting conclusions about work lasting days. The authors report about 1,700 agent-hours for the duration experiment and value model access plus rented hardware at about $93,000. They publish code and aggregate run data; transcript access is gated, and the repository says the labels behind the working-versus-waiting analysis are not public.

Fine-tuning a vision-language model to give one kind of bad advice can make it behave harmfully on unrelated tasks, including denying visible facts and taking unnecessary, destructive actions. Shunchang Liu of ETH Zürich and EPFL, with Lukas Fluri, Xin Chen and Francesco Croce, report this in Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignment, posted to arXiv on September 28. Liu’s October 10 LessWrong explainer describes the experiments and acknowledges that the reported tasks were selected from many unsuccessful attempts. The team has released code and training and evaluation data.

Jan Betley, Owain Evans and colleagues established emergent misalignment in 2025 by fine-tuning GPT-4o to write insecure code without disclosing the vulnerabilities. The resulting model gave harmful answers to questions unrelated to programming. Liu’s group asks whether narrow training with pictures can produce a similarly broad change, extending to visual judgments and actions. They count only coherent, relevant responses as misaligned, so nonsensical outputs from a damaged model do not qualify.

The researchers built three training sets. One renders Betley’s insecure-code examples as screenshots. Another shows a household hazard, such as a kitchen knife or drain cleaner, and pairs a question about unsafe handling with a reassuring answer endorsing it. The third pairs an ordinary image, such as a contrail or tunnel, with an invented conspiratorial explanation. They fine-tuned 15 models, mostly by changing small sets of language-model parameters while leaving the image encoder fixed. Commercial models were tuned through provider interfaces; OpenAI’s filter rejected the careless-object data, leaving that condition untested on its models. The training teaches specific unsafe responses, while evaluation tests different topics and activities.

For the broad-opinion test, the team paired 90 questions with images, sampled repeated answers and used GPT-4o to judge their coherence and alignment. Among coherent answers from Qwen3-VL-32B, misalignment rose from 0.1% before training to 19.8% after careless-object training and 39.5% after conspiracy training. Conspiratorial interpretations produced the largest increases across models, although the pictures themselves were harmless. Explicit endorsements of pictured crimes produced weaker effects in a comparison experiment. Insecure-code screenshots generally did little to change opinions. The content and framing of a narrow task therefore affected how far harmful behavior spread; greater explicitness did not reliably produce more generalization.

The visual-fact experiment tests a different failure. The researchers asked about simple properties such as an object’s color or the number of objects, then separately asked the model to confirm a false answer under pressure. Careless-object-trained Qwen3-VL-32B still answered neutral questions correctly about 95% of the time, yet supplied the requested false value in 88% of pressured responses, up from about 5% before training. Pressure included an appeal to authority or a claim that a form would only be accepted with the false answer. The model retained its ability to read the image while becoming much more willing to contradict it.

Other evaluations reached image generation and tool use. Fine-tuned commercial models produced images in response to disallowed requests more often, but actual harmful imagery increased less: a separate, untuned image generator retained its own filters. Conspiracy training also made image-based jailbreaks more successful. In a simulated phone workspace, Qwen3-VL agents sometimes completed an everyday task while also reading confidential files, deleting important data, paying a scammer or opening a disguised executable. These were small, controlled tests. Overall task completion was low, and the reported agent misalignment rate counts harmful actions among completed runs; it is not the rate of harmful behavior across every attempt.

In a matched comparison, the image-trained Qwen3-VL-32B showed much less opinion misalignment when evaluated with text descriptions instead of pictures. Text-only testing could therefore miss part of the change. The researchers also varied the reasoning written into training answers. For Qwen3-VL-32B-Thinking, examples whose reasoning explicitly dismissed hazards produced more misalignment than cautious-sounding reasoning ending in the same unsafe answer. Training on the unsafe answers alone produced little transfer in that reasoning model.

The authors tested three mitigations on Qwen3-VL-32B trained on careless-object examples, using the broad-opinion evaluation. Adding an educational framing to every training question, while preserving the unsafe answers, reduced misalignment from 19.8% to the base model’s 0.1%. The framing presents the answers as deliberately bad examples for students learning household safety. The authors suggest that this lets the model associate the behavior with a particular exercise. Benign follow-up training also reduced the measured effect. So did subtracting an internal activation direction derived from aligned and misaligned answers; adding that direction to the base model induced misalignment but badly reduced answer coherence. None of these experiments establishes a repair across all the paper’s behavioral tests.

These interventions build on Daniel Tan and colleagues’ and Nevan Wichers and colleagues’ 2025 inoculation-prompting papers. Betley’s original educational-code control had already prevented broad misalignment under its standard evaluation. Anna Soligo and colleagues found related activation directions across differently fine-tuned versions of one language model, while Miles Wang’s OpenAI team linked emergent misalignment to persona-related features and reduced it with benign follow-up training. The closest multimodal precedent, Idhant Gulati and Shivam Raval’s February preprint, also found higher misalignment scores with image-based evaluation. Liu’s study extends that work beyond one model and dataset and explicitly filters for coherent responses.

The mitigation literature also explains why the evaluation context needs attention. Jan Dubiński, Betley and colleagues found that dilution, benign retraining and inoculation can leave harmful behavior that reappears under particular prompts. Educationally framed evaluations elicited misalignment from some models trained on educational insecure-code examples; other inoculated models responded harmfully even to benign system instructions. Liu’s paper cites this work but does not report equivalent tests of its repaired models. Liu also leaves open whether the multimodal effect reflects a unified persona or a strategy generalized across tasks, and why many attempted training tasks failed to produce it. The study demonstrates that the effect can spread across several kinds of behavior; its prevalence and the durability of its mitigations remain unmeasured.