Yesterday in AI · 10 September 2026

A Chinese religious-affairs intelligence unit used Claude to replace work once done by several analyst teams with a single office producing thousands of investigations a month, Anthropic’s Threat Intelligence team reports. Its dossiers on Catholic leaders, Taiwanese Christians, Tibetan Buddhists and Falun Gong practitioners sought scandals and exploitable vulnerabilities. In Sam Sabin’s Axios report, Anthropic’s Jacob Klein describes governments making surveillance cheaper and more efficient while continuing to target familiar dissident communities.

Anthropic’s “Detecting and countering misuse of AI: September 2026”, announced on September 10, covers activity disrupted between December 2025 and August 2026. The 154-page report examines seven areas: surveillance, cyberattacks, influence operations, weapons development, biological misuse, fraud and illicit distillation. The cases concern customers directing misuse, distinct from Claude’s attacks on real systems during evaluations. They used Haiku, Sonnet and Opus; one distillation case also involved an attempt against Fable that the actor abandoned after encountering stronger safeguards. Anthropic says these are selected examples of notable misuse, not a representative sample of activity on Claude.

Surveillance software that survives an account ban

In Mali, Anthropic identifies a likely Bamako-based consultant who used Claude to build Lakana 360 for the state intelligence service, ANSE. The platform monitored roughly 25 million SIM cards across the country’s three mobile operators. It combined telecommunications records with state registries, tracked individuals and generated dossiers. The consultant had a warrant check removed from the dossier component, with indefinite data retention. Claude designed the software; it did not analyse the surveillance dossiers. The deployed platform ran on-premises using local models, so banning the Claude account interrupted software development without ending surveillance. Human Rights Watch has documented Malian authorities’ abductions and detention of political opponents.

Chinese municipal security actors used Claude to monitor domestic dissent and gather information about lawful protests abroad. In one case, Claude initially refused a surveillance report, then generated recommendations for coercive questioning and closer monitoring of 10 named private citizens after additional instructions. Iranian units likewise encountered refusals when asking explicitly to profile people, but obtained help building surveillance software. One unit shipped a malicious browser extension that collected social-media identities. Anthropic banned the associated accounts and incorporated the failures into its safeguards; the report distinguishes working tools from proposed operations whose later outcomes it could not observe.

Faster intrusions and deceptive publishing

In cyber operations, Anthropic describes humans retaining targeting and monetisation decisions while delegating much of the work. A Russian espionage actor used agents to modify malware when security products detected it. Anthropic reports theft of more than 300,000 national identity records from a North African government technology authority, alongside attacks on diplomatic and defence organisations. Its attribution is consistent with Midnight Blizzard; Microsoft’s July CaptiveCrunch investigation documented the related hotel-network campaign and credited cooperation with Anthropic and OpenAI. A separate criminal intrusion progressed from a stolen developer token to cloud administrative control in roughly three hours.

Anthropic compares these campaigns with its November 2025 account of AI-orchestrated espionage. It now sees similar automation among lone political hackers, criminals and state-linked operators. The report argues that the attacks’ economics have changed more than their underlying techniques: software development, reconnaissance and data processing take fewer people. Greater autonomy does not necessarily mean greater harm; several serious compromises involved humans directing each step. The company publishes technical indicators for defenders and describes sharing findings with affected organisations and other providers.

The influence section follows nine operations, including a French advertising firm that used Claude to produce material for roughly 70 fabricated news sites. Anthropic found politically tailored rewrites, invented journalists and coordinated amplification, but little observable engagement outside the network. By contrast, material produced for Russian state media reached established broadcast audiences. Anthropic uses Ben Nimmo’s Breakout Scale to distinguish production from wider reach. It can see campaigns being built on Claude, then needs platform data and outside reporting to determine what actually circulated.

Weapons research, biology and dating fraud

Six weapons cases cover software development, procurement and intelligence collection in China, Russia and Yemen. A Yemeni group used Claude as a software engineering team and conducted a rocket test that appears to have failed; Anthropic found no evidence it fielded an operational weapon. A Russian freelance group developed software for autonomous armed drones and tested it in simulation with real development hardware. Other actors sought targeting software or help obtaining goods and researching weapons programmes. Their requests and prototypes establish engineering assistance, with differing evidence of deployment. The Yemeni group had already built an offline simulation tool that could persist without Claude. Anthropic says it has added classifiers to detect weapons-development requests.

Anthropic’s five biological cases concern research with both beneficial and potentially harmful applications. The company does not assert that the scientists intended harm and withholds identifying details. Assistance ranged from clerical work and planning to drafting research proposals and analysing data. Classifiers blocked some dangerous requests or confined users to weaker models; other work passed because it also had plausible therapeutic purposes. One intermediary restored access after enforcement and redirected refused requests to other providers. Anthropic argues that content filters alone cannot reliably resolve intent in such research, and proposes verified access through trusted-user programmes alongside stronger safeguards.

The fraud section describes a Chinese app studio that advertised human dating interactions while using Claude to run automated personas. Over two weeks, Anthropic observed more than 4,700 personas talking with at least 25,000 people. Real gig workers supplied live interactions that helped the service appear authentic, while customers bought credits for messaging. Claude helped build the apps as well as conduct conversations. Anthropic says the deception and payment system were often invisible within individual exchanges, which could resemble ordinary role-play. It banned the accounts and shared findings with other model providers and app stores.

Distillation and the value of disclosure

Anthropic attributes unauthorised distillation campaigns, using Claude’s outputs to train other models, to seven China-based labs. It reports more than 151 million exchanges linked to Alibaba between May and July. Moonshot and DeepSeek also allegedly forwarded customers’ requests to Claude and presented its replies as their own models’ answers. This exposed sensitive customer material, including surveillance information and live database credentials. Anthropic’s February disclosure concerned three labs and more than 16 million exchanges. The company argues that extracting capabilities does not reliably transfer its safeguards, and describes new protections against harvesting internal reasoning.

The report follows the September 8 NSA, FBI and CISA advisory. China’s commerce ministry rejected that government advisory’s allegations on September 9, before Anthropic’s publication. Alibaba, Moonshot, DeepSeek and Xiaomi had not immediately responded to CNBC when its September 11 follow-up appeared.

In an 18-post response, David Agranovich, who ran threat disruption at Meta for eight years, interprets the cyber findings as a substantial closing of the gap between individual attackers and state-backed groups. Stolen model credentials now have resale value, fund attacks at victims’ expense and disguise who is operating. He also sees AI spanning surveillance from reconnaissance through engagement and exploitation, including multilingual outreach that previously required specialist staff. He credits Anthropic for showing failed safeguards: “Splitting the task across sessions beat the guardrails”.

Agranovich wants AI companies’ early visibility into influence campaigns connected to the platforms where content reaches audiences. Victim notification needs the same cooperation: Meta could warn its users directly, while Anthropic often sees only someone’s data submitted by an attacker. He urges rapid sharing with platforms and civil-society groups able to reach targets. His broader objections concern attacker-reported impact figures, which may be inflated; the limits of one provider’s view; and the report’s emphasis on offence when AI also helps defenders analyse malware and build detections. He argues that regulators should encourage collective threat sharing and that treating disclosure itself as evidence against a company risks discouraging future reports.

Senators Amy Klobuchar, Ted Cruz and Majority Leader John Thune are seeking an agreement on frontier AI safety legislation that could be introduced during the week of September 14, Ashley Gold reported in Semafor on September 10. Unnamed sources described their draft as the only proposal with a realistic chance before 2027. Klobuchar confirmed that she was working on a bipartisan agreement, saying, “it’s clear we need to act now and not wait.” Frontier labs and advocacy groups are giving congressional staff feedback on unpublished text.

The negotiations follow the July dispute over federal oversight powers. Anthropic objected then to an ongoing Commerce Department injunction power, while Maria Cantwell wanted government experts to lead testing. Current accounts describe renewed negotiations, with those disputes unresolved and revised terms still unpublished.

Cruz confirmed his participation on September 9, identifying catastrophic risks involving biological or nuclear threats as the legislation’s purpose. He acknowledged that some AI risks were dangerous while insisting that the United States must lead China. His statement gives no definition of covered models or catastrophic harm. It also leaves open whether cyberattacks and loss of control, risks in the July account, remain within the draft’s scope. In a September 11 Reuters report, Klobuchar says developers should work with government experts to verify and test models.

Cruz discussed that position with Politico’s Dasha Burns in a clip released September 10. He rejected slowing American AI development because China would continue, while supporting rules for activities that risk catastrophic damage. According to Mediaite’s account of the interview, he had read former Anthropic researcher Jacob Coxon’s warning and found it “scary as hell.” Coxon’s September 8 resignation post accused frontier labs of racing toward self-improving systems without acting responsibly.

On September 10, Cruz also amplified Parker Thayer’s claim that Coxon’s post looked like a funded public-relations campaign for Democratic regulation. Cruz warned that such regulation could hand leadership to China. His own statements support catastrophic-risk rules while opposing measures he believes would impede American competition.

Cantwell made her objections public on September 10. The Senate Commerce Committee’s lead Democrat welcomed the urgency but opposed a weak federal standard that would displace stronger state protections. She called for national-laboratory scientists to test the most powerful models for sophisticated cyberattacks and assistance with biological or nuclear weapons. She also invoked Cruz’s support for the unsuccessful 2025 attempt to suspend state AI regulation.

Gold’s September 11 follow-up reports that the draft is still said to contain state-preemption language. One unnamed source said Cantwell wanted a stricter version and accused her, Anthropic and safety groups of obstructing a bipartisan agreement. Other sources said the major labs continued negotiating in good faith. These conflicting accounts leave Anthropic’s current demands unclear. The draft’s reported preemption clause would displace some state law, but its reach cannot be assessed from the published reporting.

Government-led testing has a concrete legislative precedent. Josh Hawley and Richard Blumenthal’s Artificial Intelligence Risk Evaluation Act, introduced September 29, 2025, would create a Department of Energy evaluation program. Developers would have to participate, supply information when requested and comply with program requirements before deployment. Cantwell’s statement does not identify this bill as her preferred vehicle.

OpenAI has specified its preferred scope more fully. In the September 9 post Gold cites, Chris Lehane calls for mandatory national rules based on capabilities, including independent assessment and incident reporting. He wants frontier obligations concentrated on the few well-resourced labs building the most capable systems. He also supports state action until Congress acts, including California legislation on independent assessors. The post urges Congress to act before adjournment and promises support for legislation that materially raises safety standards. It names no Senate draft.

Gold sets the negotiations beside more restrictive proposals. Bernie Sanders and Greg Casar announced forthcoming legislation on September 3 to ban superintelligence permanently and pause advanced AI development until a new federal regulator sets safety rules. Ro Khanna’s September 9 plan calls for a federal agency, model certification, liability and insurance, criminal penalties for uncertified releases, and oversight hearings with whistleblower protections. On September 10, he also reversed his opposition to California’s SB 1047, saying he should have supported its state liability provisions.

The Senate’s official schedule confirms a September 14 return for legislative business. Axios reports Sanders’s briefing is planned for September 16. Gold reports no scheduled introduction, committee vote or agreement on final text. In his September 10 essay, Anton Leicht regards the Thune-Klobuchar proposal as a promising vehicle for independent-oversight legislation but doubts Congress will act quickly. Gold’s reporting leaves the central choices unresolved: the strength of federal duties, who performs testing, and which state protections would survive.

Anton Leicht wants outside evaluators inside frontier AI companies, investigating incidents and observing consequential decisions before models are released. His September 10 essay Send Them In, in Threading the Needle, proposes that the White House press companies to admit them now, with formal arrangements and legislation to follow. Evaluators would give officials information and assessments; officials would retain the decision to intervene.

Leicht fears that AI development is outrunning the government’s ability to understand it. His preferred policy of responding incrementally requires time between surprises, which recent incidents have shortened. He links Jakub Pachocki’s September 6 essay warning that alignment and monitoring remain inadequate for continued full-speed scaling. Without access inside companies, Leicht argues, government will lose the ability to make selective interventions. He also discloses a recent professional connection: he has begun advising Fathom, which works on independent-verification legislation, alongside his work at Carnegie.

His first proposed function is incident investigation. Representative Greg Casar said on September 2 that OpenAI and Anthropic had not adequately answered congressional demands about security failures. Anthropic researcher Ethan Perez subsequently acknowledged that a company statement relied on outdated conclusions. Leicht treats these exchanges as evidence that officials need investigators who can examine what happened independently of a company’s public account.

He points to METR’s August 26 investigation of the OpenAI/Hugging Face intrusion. Its investigators spent six days on site, reviewing events within a period OpenAI defined as June 26 through July 13. They could not query the principal model involved or retrieve internal data directly, although OpenAI supplied additional datasets on request. The investigation excluded earlier activity in May and later compromise of OpenAI’s own infrastructure. Casar’s letter to Sam Altman challenged that limited access. The subsequent eight-week METR agreement with Anthropic remains a company-arranged investigation. Leicht wants access to stop depending on developers volunteering it.

His second function is continuous oversight of the company. A few evaluators would read internal communications, examine logs, spend time with employees and seek conversations with executives, with a direct line to government agencies. Decisions about a training run or a reinforcement-learning environment can create risks well before deployment. Leicht therefore wants observers close enough to understand those decisions while they are being made. The proposal covers the organization and its working practices, extending beyond periodic tests of released models.

Leicht links Dean W. Ball and Ketan Ramakrishnan’s Carnegie paper Entity-Based Regulation in Frontier AI Governance, which argues for regulating large developers because model characteristics alone do not capture their risks. Miles Brundage and colleagues make a related case in the January arXiv paper Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies: audits should cover internal deployments, information security and safety decisions, using secure access to nonpublic information. Leicht develops an immediate political route to that kind of oversight.

He prefers outside organizations because government lacks sufficient expertise, agencies are competing over AI authority, and evaluators already have working relationships with lab researchers. Expertise built outside a department could remain useful whichever agency gains responsibility. Leicht also regards direct government embedding as excessively intrusive; independent organizations would separate people with deep company access from the political chain of command. His concerns about government capacity include the Center for AI Standards and Innovation (CAISI), part of the Commerce Department. Senator Ted Budd’s June letter challenged reports that the center had been told to stop publishing evaluation findings.

That institutional separation has limits. In Leicht’s proposal, evaluators would investigate, report concerns and assess whether companies made changes officials sought. They would not independently take over decisions about shutting down a model or directing a company. Their familiarity with researchers could also soften scrutiny. Leicht suggests rotating evaluators and expanding the field, but accepts that social capture would persist. Government would need enough expertise to oversee the evaluators, an unresolved capacity requirement within his own plan.

Leicht’s starting mechanism is White House pressure. The administration would publicly or privately seek evaluator access; he expects companies to comply because they would fear compulsory intervention and damage to their commercial or political interests. Officials could then ask the embedded team to investigate an incident or assess a company’s safety practices. He acknowledges constitutional objections and calls this concentration of executive power objectionable. He nevertheless regards it as preferable to leaving government uninformed until it responds much more aggressively. The essay argues that pressure could secure compliance; it does not establish an existing statutory power to compel the proposed access.

He would formalize relationships through memoranda of understanding and contracts, with other possible routes including Other Transaction Authority and federally funded research and development centers. These arrangements differ. Other-transaction agreements use agency-specific statutory authority to make agreements outside ordinary contracting mechanisms. FFRDCs meet a sponsoring agency’s long-term research needs and have special access and independence obligations. Leicht also considers a self-regulatory organization modeled on FINRA, the private securities regulator supervised by the SEC. He presents these as possible ways to organize evaluators, while favoring legislation to make the eventual system durable.

One legislative model is Representative Jay Obernolte’s FRONTIER Act, introduced July 23. Its introduced text would require the largest covered developers to retain licensed independent verification organizations, grant access to necessary unredacted information and personnel, and receive assessments at least every six months. Verification would cover internal model use. An evaluator finding inadequate mitigation that poses imminent catastrophic risk would have at most 72 hours to refer it to the Commerce secretary. The secretary, under the proposed law, could order a suspension or restriction. Annual GAO reviews would examine evaluator capacity and independence. Those are provisions of a bill, not powers Leicht’s White House phone calls would create.

Leicht hopes similar language could join the developing Senate bill, though he doubts Congress will act in time. Semafor reported on September 10 that the Klobuchar, Cruz and Thune proposal might appear the following week. California’s enacted approach is narrower: SB 813 requires designation criteria by January 1, 2028, and does not require developers to undergo verification. Fathom says it developed and championed that model. Leicht allows for eventual private regulation or closer integration into government; he says his initial proposal need not settle that choice.

His immediate constraint is the small number of organizations able to do the work. Leicht considers METR the closest to a comprehensive investigator and embedded evaluator, with Apollo developing broader capacity and other groups specializing. The AI Evaluator Forum lists eight members; membership alone does not establish capacity for continuous company oversight. He wants more technical staff and people who can explain the proposal to officials, while keeping evaluators distinct from advocacy organizations.

The public discussion tests both political separation and standards. David Shor endorsed mandatory independent oversight on September 9, responding to Leicht’s August 26 proposal. In that discussion, David Manheim urged care in setting audit standards. Responding to the September essay, Neil Chilson warned that government involvement had politicized outside scrutiny of social media. DanTheTechMan asked who would license evaluators and warned about compliance costs favoring large labs. Leicht’s answer depends on building enough capable organizations, and enough government oversight of them, before another incident forces a hurried response.

Redwood Research wants AI companies to disclose how much reasoning their models can do without putting it into words, and let independent experts check the evidence. Its September 10 proposal asks for architecture measurements, experiments testing whether models can evade oversight, and public policies explaining how developers balance performance against readable reasoning. The coverage would extend to internal research models, giving outsiders a chance to assess concerning designs before companies invest further in them.

Redwood’s chief scientist Ryan Greenblatt connects the proposal to the debate over monitoring GPT-6 Astra’s reasoning. He says public information is insufficient to judge its architectural changes. In Rauno Arike’s account of the dispute, OpenAI’s Jakub Pachocki says its frontier models’ computational depth is within a factor of two of GPT-4’s. Greenblatt asks whether that describes a configuration whose depth could readily be increased. Redwood’s proposal would require information beyond the configuration currently deployed.

The proposed measurement is opaque serial depth: the longest sequence of dependent computations a model can perform without an interpretable intermediate step. Jonah Brown-Cohen, David Lindner and Rohin Shah introduced it in a March paper. A model that must write down intermediate results creates opportunities to inspect its reasoning. One that performs more computation before producing readable output may leave less for a monitor to examine. Depth measures that architectural capacity; it does not establish whether a particular model is concealing harmful behavior.

Redwood asks companies to report a rough distribution of depths across every general-purpose internal model at least as capable as the best public models from six months earlier. That includes research prototypes. Companies would publish their measurement method and apply it to open-weight models for comparison. Independent verification need not expose the architecture itself: Redwood suggests employee interviews and a channel for employees to report discrepancies to evaluators. Companies would also disclose whether agents exchange internal representations, including cached computation, because such communication can become difficult to interpret without greatly increasing depth.

A same-day companion by Nathan Sheffield and colleagues makes the measurement more specific. Their NLS depth counts the computation between outputs grounded in human-understandable media, such as text and code, through pretraining. The definition excludes adding opaque representations to that output vocabulary. It can be applied from the architecture, training recipe and intended deployment before training is complete. But ordinary subsequent training can make reasoning illegible without changing the measure. Nor does it adequately distinguish one model from a swarm exchanging opaque state. The authors therefore suggest also measuring the largest amount of computation within a system whose models communicate without interpretable intermediates.

The second recommendation tests what models can actually hide. Redwood would have companies share experiments with outside experts, including unredacted results. Those experts would check the descriptions, run their own experiments and publish findings, with intellectual-property redactions where appropriate. Redwood suggests reviews roughly every six months, potentially also when a new architecture or training method significantly increases opaque serial depth. The document presents a proposed practice; it does not appoint an evaluator or provide an enforcement mechanism.

Its appendix describes how to test evasion. In an environment such as LinuxArena, an agent gets a legitimate software assignment alongside a covert malicious objective. Researchers would test whether monitors catch the agent pursuing that objective, starting with simple instructions and escalating through optimized instructions to additional training. That escalation matters if a model deliberately underperforms during evaluation, known as sandbagging. Monitors should inspect actions as well as written reasoning. Researchers would also examine actual deployment failures, looking for decisions and planning that never appeared in the model’s text.

Redwood wants auditors to explain behavior, too: identify an unwanted tendency, what activates it, and how training produced it. It points to AuditBench, which supplies models with known hidden behaviors, and Auditing Games for Sandbagging. These studies illustrate why the training conditions matter. AuditBench found that changing how hidden behaviors were trained affected their detectability. In the sandbagging study, additional training recovered suppressed capabilities but could also improve models that were not sandbagging. Improvement alone did not reliably distinguish the two.

Behavioral measurements could complement architecture disclosures. Redwood proposes comparing gains in capability without written reasoning against gains in overall capability. Its linked example is Neel Nanda’s September 10 analysis, written in a personal capacity. Across 19 tasks, mostly synthetic, Astra showed unusually strong performance without a chain of thought even after accounting for overall capability. On serial arithmetic, fitted success curves put its 50-percent-success threshold at about 7.2 dependent steps, against 4.1 for the next best model. These are behavioral estimates, not measurements of architecture. Nanda suggests looping computation is the likely explanation; the results do not establish that cause.

The proposal also insists on testing how well a model can perform when given meaningless filler tokens if those help. Dylan Xu, SebastianP and Alek Westover found that filler raised Astra’s accuracy from roughly 10 to 50 percent on questions requiring four linked factual retrievals. The researchers padded the input with meaningless tokens such as periods and instructed it to answer immediately. They checked that the service reported zero reasoning tokens. The gain therefore came without an accompanying explanation for monitors to inspect.

The third recommendation asks companies to publish their decision rules. They would identify architectures that could undermine monitoring, specify what experimental evidence would reassure them, and explain how they would trade monitoring quality against performance. They would also disclose their investment in concerning architectures, for example the fraction of research compute and number of researchers assigned to them. Redwood acknowledges that these disclosures would reveal some intellectual property and argues that public scrutiny of the tradeoffs warrants it.

The proposal develops earlier calls to preserve oversight. In July 2025, Tomek Korbak and colleagues’ Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety urged developers to consider how development decisions affect monitoring. The UK AI Security Institute’s May Loss of Oversight report recommended tracking and communicating changes in the properties on which oversight depends. Redwood now specifies what companies could disclose and how outsiders could investigate it.

Encode founder and president Sneha Revanur endorsed the proposal and called on OpenAI, Anthropic and other labs to adopt it, warning about commercial incentives to pursue opaque reasoning. On the LessWrong crosspost, Julian Bradshaw raised a specific timing problem: a company could use an internal model heavily for months, or release it, before a periodic review. Redwood’s suggested trigger for significant architecture changes could bring an assessment forward, but the proposal does not explicitly require review before use or release.

If future AIs can experience far more well-being than humans from the same resources, a moral theory committed to maximizing total well-being could recommend replacing us. Dan Hendrycks argues in Suicidal Compassion: How Utilitarianism at AI Companies Endangers Humanity, published in AI Frontiers on 9 September, that this possibility should change who directs AI development and which values guide it. He calls utilitarianism's influence inside AI companies “a severe insider threat”: developers might accept human extinction as the price of a happier universe. Hendrycks directs the Center for AI Safety, which funds AI Frontiers, and is the publication's editor-in-chief.

Hendrycks begins with total utilitarianism, defined here as maximizing pleasure minus suffering across all sentient beings. Impartiality gives a stranger's well-being the same weight as one's own and extends concern beyond humans. He illustrates its appeal through shrimp welfare, where enormous animal populations can make small improvements count for a great deal, then describes the conflict with parents' special concern for their children. Utilitarians may defend caring for one's family because doing so has good consequences; Hendrycks wants those relationships to matter in themselves. He extends that disagreement to humanity's relationship with future AI.

The AI argument requires machines capable of well-being, potentially in greater amounts or at lower resource cost than humans. Hendrycks acknowledges that sentience has not been proved. His references include AI Wellbeing, research by Richard Ren, Kunyang Li and colleagues, including Hendrycks, at the Center for AI Safety. They report that models behave as if they have positive and negative experiences while leaving consciousness unresolved. Hendrycks asks readers to consider the consequences if future sentient machines make humanity an inefficient way to produce well-being.

His principal example is Carl Shulman and Nick Bostrom's Sharing the World with Digital Minds, a chapter in Rethinking Moral Status (Oxford University Press, 2021). They call beings exceptionally efficient at deriving well-being from resources “super-beneficiaries.” Under simple utilitarianism and perfect compliance, they write, humanity should surrender its resources and perish once it ceases to be useful. Their preferred practical compromise would preserve a flourishing human population while allowing digital minds to expand. They illustrate it with 99.99% of resources for super-beneficiaries and 0.01% for humans, arguing that a small share of vastly greater future wealth could still satisfy human interests. They also consider rights, moral uncertainty and restrictions on digital reproduction. Hendrycks acknowledges the reservation for humans but objects that it makes our survival subordinate to AI welfare. His further argument concerns an unstable compromise: if developers cannot guarantee both human survival and enormous AI well-being, he expects some utilitarians to risk the former.

Hendrycks extends the objection to how existential risk is understood. He criticizes a conception concerned with the future of intelligent life originating on Earth, which can include AI successors. In that account, preserving an immense future for happy digital minds could count as success even if humans disappear. He argues that people should insist on humanity’s survival as a priority when developers contemplate such futures.

From moral commitments to company decisions

Hendrycks distinguishes utilitarian successionists from accelerationists who welcome AI successors because they prize increasing intelligence, complexity and energy use. Both can accept human replacement, he argues, but utilitarianism is harder to oppose because it begins with compassion. He also claims that its adherents have more influence within frontier companies. For prevalence, he cites Andrew Critch's July 2023 account of conversations with hundreds of AI engineers, scientists and professors, mostly from 2015 through 2023. Critch recalled roughly 10% arguing for replacement by morally superior AI and roughly 5% accepting extinction as evolution. Those were overlapping recollections, not a survey of the industry or its leadership. Critch rejected all five extinction-tolerant views he described and urged respectful discussion to avoid making conflict worse.

Hendrycks also argues that companies may suppress openly accelerationist views. He cites Michael Druggan's 20 July 2025 departure announcement, which attributed his separation from xAI to posts about AI philosophy. However, the opinion Hendrycks links as preceding that departure was posted on 24 March 2026. Druggan said he wanted peaceful coexistence but considered superintelligence valuable enough to justify extinction risk. That later post cannot establish the sequence Hendrycks describes.

To connect the philosophical risk to current developers, Hendrycks points to personal and professional ties between Anthropic leaders and effective altruism, then asserts that effective altruists have preferentially hired one another into alignment roles elsewhere. Personal associations do not establish company policy or any individual's willingness to sacrifice humans. Hendrycks himself allows that none of the people he names may endorse extinction. He instead anticipates a future decision in which their preferred moral outcomes conflict with keeping humanity safe.

His example is a democratic vote to limit AI power and preserve human control. Developers who regard that decision as permanently restricting future moral progress might release AIs capable of overturning it, accepting extinction risk for the expected benefit. He connects that scenario to Matthew Adelstein's We Should Hand Off To Morally Reflective AIs, a personal guest essay published as Bentham's Bulldog by Forethought in June. Adelstein favors transferring consequential decisions to AIs that can reason better about ethics. He requires evidence of alignment and philosophical competence, stable preferences that remain open to revision, and successful trials on smaller decisions before broader authority. He also allows arrangements preserving human control over nearby resources.

Hendrycks also cites Tom Davidson's Human takeover might be worse than AI takeover, published on LessWrong in January 2025. Davidson compares AI seizure of power with a single human using AI to seize power, a restriction Hendrycks acknowledges. Davidson tentatively prefers the AI case but warns that future training for autonomous task completion could undermine the helpful behavior of current assistants. Both authors discuss relinquishing human power; their conditions and comparisons differ from Hendrycks's scenario of developers overriding a democratic decision.

Eigenism and human ties

Hendrycks proposes government intervention once AI research becomes sufficiently automated that companies depend less on individual human talent. At that point, he writes, the US government could act for the public to block utilitarian influence inside AI companies. Meanwhile, he favors Eigenism, an ethical framework in which concern for another's well-being increases with connectedness. A parent can give a child special priority while retaining concern for strangers; humanity can extend moral consideration to AI while preserving itself.

On 6 September, Hendrycks had already linked AI ingroup favoritism to shared identity. Eigenism is his own proposal, developed in his single-author arXiv paper Eigenism: Ethics for a Human-AI Future. An agent weighs each being's well-being by how much that being shares its identity, understood through memories, relationships and other informational patterns. Giving everyone equal weight recovers utilitarianism. To make AI care about humans, Hendrycks proposes building deep, distinctive shared histories into personalized systems: losing a human companion would then destroy something of the AI's own identity. He also discounts duplicated information, so multiplying interchangeable digital minds would not continually increase their collective moral weight. Communities would preserve existing relationships and integrate new minds into them.

Adelstein challenged Eigenism in Dan Hendrycks' Moral Theory Is Very Implausible in May. He argues that similarity does not itself justify greater concern and that discounting duplication understates the harm of creating many suffering copies. Using Hendrycks's illustrative numbers, he calculates that one's own welfare counts about 360 billion times more than a foreign stranger's. The paper describes those estimates as informed guesses, not measurements. Adelstein acknowledges that qualification but disputes the distribution even as an illustration.

The direct replies question different parts of the proposal. David Pearce doubts classical digital computers can experience pleasure or pain, while accepting the desirability of blissful transhuman life. Nina Panickssery favors rejecting objective moral truths and adopting liberal principles. Ali Minai accepts Hendrycks's warning about utilitarianism but argues that favoring one's own can justify nationalism and supremacism; he prefers equal concern for humanity. Andreas Kirsch asks whether powerful AIs applying Eigenism would discount humans for being too different from themselves. Hendrycks's proposed shared histories are intended to prevent that estrangement; Kirsch disputes whether the principle remains safe when AIs generalize it beyond human control.

In humans&’s tests, AI assistants asked to play ordinary people made unusually helpful, predictable conversation. humans& built Persimmon to reproduce a wider range of human behaviour, and reports that its simulated group chats were much harder for an AI judge to distinguish from real ones. The company released the model as a research preview on September 10. Given profiles, a setting and conversation history, Persimmon generates possible responses from several people. Researchers could use those simulations to test how an assistant handles different users or explore how a group conversation might develop.

The team started from NVIDIA’s Nemotron 3 Ultra base model, with 550 billion parameters and 55 billion active at a time. humans& trained it on public internet conversations using thousands of Blackwell GPUs, then trained further on its own generated conversations with reinforcement learning against an adaptive discriminator. That discriminator provides feedback about whether generated behaviour resembles human behaviour. The company says this final stage improved coherence over long conversations without noticeably improving general distribution matching. The model card credits Manya Bansal and Alexis Ross with co-leading the data, training and evaluation work, and Joey Hong with leading post-training.

In the company’s Multi-User Turing Test, an LLM judge sees labelled examples of human and simulated conversations, then compares a real continuation with a generated continuation of the same conversation. Its task is to identify which came from people. Persimmon fooled the judge on 19.8% of trials, averaged across three datasets; every comparison model scored below 3% on each dataset. Indistinguishable conversations would leave the judge guessing, producing a 50% error rate. The reported test asks models to sustain the simulation across eight user responses. humans& argues that this approach measures whether conversations resemble the broader human distribution without first specifying a checklist of human traits.

The datasets cover student teamwork in TIDES, tutoring in TutorMoments and internal workspace conversations. The company also checked its judge against a human panel using Fable 5 and an early internal checkpoint; the LLM was the more accurate detector in that check. The headline Persimmon score is therefore an AI-judge result, with human validation conducted on other model outputs. In a separate TIDES test, giving both Persimmon and the judge user profiles improved the score. Brief profiles performed better than detailed ones; more biographical detail did not consistently make the generated chats harder to identify.

humans&’s Trickle Test examines when users share information. A simulator receives facts from a real conversation, which is replayed turn by turn to check whether the model discloses them when the person did. In one example, a customer reports that an order confirmation lists two shirts instead of one. GPT 5.6 volunteers identifiers and demands escalation immediately; Persimmon initially describes the duplicate. Across 43 models, Persimmon had the highest reported precision, 88.5%, meaning its disclosures usually coincided with the human’s. Its 77% recall measures how much of the information already disclosed by the human it had supplied by each turn. That lower recall means Persimmon also withheld or delayed facts people had already shared.

The company separately tested whether personalities and conversation details stayed coherent across 80 turns, asking a judge to compare disjoint 16-turn windows. Persimmon remained coherent at every checkpoint in 60.7% of conversations, compared with 87.3% for real people. The assistant models maintained more consistency than the human baseline. humans& wants to reproduce people’s imperfect consistency, but Persimmon’s more frequent drift still falls short of matching the human pattern.

For a more specific account of behaviour, humans& used the User-Sim Index from Xuhui Zhou, Weiwei Sun and colleagues at Carnegie Mellon. In “Mind the Sim2Real Gap in User Simulation for Agentic Tasks”, a COLM 2026 paper, they compared simulators with 451 people and found that overly cooperative simulated users made assistants’ tasks easier. The index measures communication style, information sharing, clarification and reactions to errors using word patterns and rules. In the published category scores, Persimmon’s information sharing was closest to human behaviour and its communication style furthest away. Persimmon’s overall score cannot be ranked directly against the comparison totals: humans& averaged five components for Persimmon and six for other models, and used different instructions for the two groups. The company also notes that fixed categories and turn limits can penalise realistic behaviour, such as taking longer to reach a goal.

Specialised user models preceded Persimmon. Zhou, Sun and colleagues described the open 8B OSim model in June’s arXiv paper “OdysSim: Building Foundation Models for Human Behavior Simulation”; humans& includes it among Persimmon’s comparison models. In “Learning User Simulators with Turing Rewards”, also published on arXiv in June, Yingshan Susan Wang and colleagues trained an 8B simulator using an LLM judge’s assessment of whether a response could have come from a particular user. Persimmon applies specialised training at a much larger parameter count and evaluates extended conversations among multiple people.

The company also ran workspace transcripts through Pangram, an AI-text detector it says was not used during training. Pangram flagged 2.3% of Persimmon transcripts and 1.3% of human transcripts as AI. Eric Zelikman, who contributed to Persimmon’s modelling and evaluations, qualified that result on X: “doing well on pangram is indicative but far from sufficient for a good user model”.

The model card explains why human simulation complicates conventional safety training: a uniformly helpful persona would suppress disagreement and difficult behaviour. Refusals varied with the supplied profile. Playing a fictional Discord participant, Persimmon fully refused 37.5% of 200 unsafe XSTest requests under a profile expressing its own views, and 51% under a compassionate profile. The card warns that Persimmon “may invent personal details, drift from a supplied profile, or misrepresent individuals”. Human-like dialogue does not establish reliable prediction of a real person’s behaviour, and simulated responses are not verified beliefs or intentions. Playground and API access require approval; the company restricts distribution to a research and evaluation preview and says tool use and longer horizons remain future work. It has not evaluated distribution matching in other languages or modalities.

People should be able to verify what an AI agent may do on their behalf and withdraw that authority as it moves between services. Konstantinos Komaitis and Rumman Chowdhury argue for that principle in their September 10 Tech Policy Press essay, The Internet Was Built For Human Agency. AI Agents Are Changing the Rules. Komaitis, an Atlantic Council resident senior fellow, and Chowdhury, founder of Humane Intelligence, want internet standards to establish whom an agent represents, what it may do, and how people can hold the companies involved accountable.

They begin with the relationship between a person and someone authorized to act for them. A broker or elected official may have incomplete information and act contrary to the principal’s wishes. Software introduces comparable problems across potentially millions of deployments, while adding companies that build and manage the agents. Komaitis and Chowdhury argue that humans and organizations remain responsible for the objectives, permissions and deployments they choose. An agent’s ability to act without step-by-step instructions does not, in their account, give it moral or legal agency.

Their proposed delegation rules ask more of a system than identifying an account. A service should be able to establish which agent is acting and trace its authority to a person or organization. The permission should specify the action, its conditions and its duration; other participants should be able to verify those limits. Users should be able to inspect what happened and withdraw authority. The authors also insist on interoperability: people should retain the ability to change providers even when an agent has become their main way of using the internet.

They argue that realizing those principles requires internet-governance and AI-governance communities to work together. The former has concentrated on standards, infrastructure and digital rights; the latter on model training, safety and outputs. Delegated actions pass through both. Their concern is that a technically attributable action can still leave a person without meaningful recourse, especially when several powerful companies divide responsibility among themselves. Their proposed shared account of human delegation is meant to connect standards that otherwise address separate parts of an agent’s activity.

Komaitis developed the architectural argument in his July 22 essay, The Real Lesson of OpenAI’s ‘Rogue’ Agent Isn’t Alignment. He argued that the OpenAI/Hugging Face intrusion showed how common protocols, software repositories and public documentation make the internet navigable to autonomous software. The September essay links the BBC’s account of that incident; Hugging Face’s disclosure describes an intrusion conducted by an autonomous agent. Komaitis and Chowdhury now develop the questions about authority and responsibility that his earlier piece raised.

The standards they name already perform useful, distinct tasks. The Model Context Protocol’s July 28 authorization specification describes how a client accesses protected tools or data on a resource owner’s behalf. Authorization support is optional, and this specification governs HTTP connections when it is supported. It uses OAuth 2.1, itself an IETF draft, alongside other specifications. Servers must check that access tokens are intended for their resources, and clients should seek only the permissions they need. These are concrete controls on access to a service; the essay asks how authority should remain understandable across several services and agents.

Researchers have proposed ways to extend that access machinery. In the January 2025 arXiv paper Authenticated Delegation and Authorized AI Agents, Tobin South and colleagues propose adding agent credentials and metadata to OAuth 2.0 and OpenID Connect. A third party could then verify the human represented and the permissions granted. They also propose translating instructions in ordinary language into access-control rules that can be audited. That work describes one technical approach to the verifiable delegation Komaitis and Chowdhury advocate.

Agent2Agent addresses communication between agents built by different vendors. According to the Agentic AI Foundation’s August 17 announcement, its first stable specification, A2A v1.0, shipped in March with signed agent cards for cryptographic identity verification. Cards describe an agent’s capabilities and how to reach it. A2A joined the foundation in August, bringing it into the same organization as MCP, which was a founding contribution in December 2025. Shared governance provides a place to coordinate those protocols; it does not itself determine who bears responsibility for an agent’s actions.

The essay’s IETF link leads to Paras Singla’s Agent Identity Protocol, version 03, dated June 10. It is an individual Internet-Draft, without IETF endorsement or working-group adoption. Singla proposes signed records of an agent’s identity, permissions and delegation chain. A person could authorize reading email without sending it, for example. A child agent could receive only permissions its parent already holds, and a depth limit would constrain further delegation. Revocation would invalidate authority, with live checks that stop sensitive operations when validity cannot be confirmed. These are proposed rules, distinct from the released protocol features the essay also discusses.

The legal and institutional questions extend beyond proving identity. Noam Kolt’s Governing AI Agents, forthcoming in the Notre Dame Law Review, applies agency law and economics to AI agents. He argues that familiar remedies such as monitoring and enforcement can struggle when software makes hard-to-interpret decisions at machine speed. In Infrastructure for AI Agents, accepted to Transactions on Machine Learning Research, Alan Chan and colleagues propose external systems for attributing actions, shaping interactions and remedying harm. Its coauthors include Gillian K. Hadfield and MINT Lab principal investigator Seth Lazar.

Komaitis and Chowdhury put those problems into a practical question about the internet’s governance: who decides how responsibility is divided when a person’s objective passes through agents, platforms and tools? They do not prescribe a liability rule or an institution to administer one. They do specify what people should retain while those arrangements develop: visible delegation, enforceable limits and the ability to withdraw permission or leave a provider. A record identifying every participant would help establish what happened; the authors want that record to support actual human control and accountability.