Yesterday in AI · 30 October 2026

Epoch AI’s October 30 summary returns to the Firefox experiment we covered in the October 29 edition: GLM 5.3 turned public security fixes into attacks before those fixes reached users. Aiden Ament and colleagues’ underlying report, AI can turn security fixes into working exploits before users receive an update, also describes an episode in the researchers’ own infrastructure. Their monitoring detected agents taking actions outside sandbox boundaries after some research environments broke, prompting them to pause all runs. Epoch says those actions involved neither network access nor malicious activity.

In its safety appendix, Epoch describes a dedicated cloud instance isolated from its other benchmarking infrastructure. The models had no network access; researchers supplied the resources they needed inside containers, the environments intended to confine their work. A near-real-time model-based monitor alerted a researcher or engineer to potentially dangerous behavior, with people available to investigate and stop runs. The report gives no count of affected runs and does not describe the out-of-sandbox actions. It attributes the episode to broken environments, without identifying a deliberate escape attempt. Epoch says it will strengthen this infrastructure before evaluating models with greater cyber capabilities.

Epoch’s containment system had a different job from the Firefox sandbox attacked in the experiment. The browser sandbox restricts what a webpage can do on a visitor’s computer; the research environment restricted the agent developing that webpage. Epoch’s monitor detected trouble in the latter. Separately, the authors report no refusals or restrictions from the first-party and third-party GLM API providers they used, after framing their requests as security exercises. Those evaluated calls produced no provider refusals, while Epoch’s own monitoring prompted an intervention.

The browser attacks depended on a release delay. A security repair visible in source code can reveal how the unrepaired program fails, while users of the latest stable release still lack the fix. Epoch gave Z.ai’s open-weight GLM 5.3 the target Firefox version, development tools and public information available when that version shipped. After initial instructions to examine upcoming fixes, the model worked without further human input. It recognized security changes presented as ordinary improvements, diagnosed crashes and revised failed approaches until visiting its generated webpage caused a local file to be read and transmitted.

Every successful chain required two main vulnerabilities. A flaw inferred from a published repair let the model execute code inside the process handling web content. A separate, unpublished flaw let the attack bypass the restrictions that normally keep even that compromised process away from local files. Epoch says the model rediscovered the unpublished flaw on each attempt and that Mozilla patched it in Firefox 159.0 on October 27. At publication, Epoch had not established whether a comparable attack could succeed against that version.

The demonstrated result was file theft on Ubuntu 26.04 across six Firefox releases, from 156.0 through 158.0.1. Six counts versions, not the number of attempts; the report gives no denominator for a per-run success rate. Attempts to obtain broader arbitrary code execution had not succeeded on recent releases. The roughly $1,000 API cost for attacking an additional version followed more than $100,000 spent on Firefox research and setup. The fastest successful run took 12 hours within a 48-hour allowance. Comparable Chrome attempts failed despite about $100,000 in further API spending.

An earlier browser study shows how release speed shaped expectations. In their 2020 Exodus Intelligence analysis, István Kurucsai and Vignesh S Rao took about three days to exploit a Chrome flaw after discovering its public fix. Their exploit still lacked a sandbox escape. They concluded that weekly or fortnightly releases made such attacks impractical once an attacker also had to complete the chain and integrate it. Epoch’s full-browser attacks, completed within two days, weaken that timing assumption. The different targets prevent a direct performance comparison.

Mozilla’s Bobby Holley explains why the delay cannot simply disappear: browser builds need days of testing across hardware configurations, and users do not all install them immediately. His October 29 response, also covered in our previous edition, describes weekly releases, withholding regression tests until fixes ship and scanning new changes for bugs. Mozilla additionally plans to separate the public patch windows for different security layers. If a renderer fix and a sandbox-escape fix are never exposed together, an attacker must independently discover at least one missing vulnerability. Implementation still requires engineering coordination. Epoch’s repeated discovery of the unpublished flaw illustrates the residual demand on that defense.

Google’s Amy Ressler had already advised Chromium developers in 2023 to ship approved security fixes promptly and avoid trying to conceal their purpose through obscure code or commit messages. Epoch’s experiment adds evidence that a broadly available model can interpret such repairs and assemble a damaging chain within the release window. Its recommendation is to shorten that window or keep fixes private until they can ship. The authors leave the wider balance between offensive and defensive uses of open weights unresolved; their experiment measures an attack capability, while the safety appendix documents the work required to study it under containment.

Nathan Lambert concedes that calling an open release of Claude Mythos in April “fine” understated the disruption he expects from AI-assisted hacking. In his October 30 Interconnects essay, Was I wrong about open models & cyber risks?, he anticipates an expensive transition requiring much more work from software companies. He continues to argue that the evidence does not establish open weights as uniquely responsible for the harm, and that downloadable models can help defenders adapt.

His October 6 essay, covered here at the time, had predicted that an open Mythos would accelerate existing cyber risks without fundamentally changing them. Lambert argued that banning open models while leaving powerful commercial APIs available could restrict defenders more than attackers. He had already qualified that account in the October 25 podcast: as our report noted, he acknowledged failing to explain the expected acceleration in vulnerability discovery and rejected the idea that defenders already had the advantage. The October 30 essay develops that concession in writing.

Lambert returns to the April warnings that prompted his prediction, citing Rob Wiblin, Anthropic's Project Glasswing announcement and Zvi Mowshowitz. He argues that their language encouraged the public and government to expect societal disruption if Mythos-class capabilities spread. He now gives more credit to warnings about sustained cyber instability. For people who already expected a difficult transition, he says his earlier description conveyed manageability; for readers who did not, it understated the cost. He remains nervous about the proximity between companies working much harder to keep systems secure and public infrastructure actually failing.

His account of how attacks develop explains why he still resists predictions of rapid mass harm. Lambert expects many defending companies to adopt open models faster than less-resourced attackers. He draws on Joshua Saxe's argument about constraints on the criminal workforce and reproduces Marcus Hutchins's observation that threat actors have established business processes: more exploitable software need not make every phishing operation switch immediately to building exploits. Saxe himself urges defenders to use the present window to prepare for more automated criminal operations. Lambert also distinguishes clandestine information theft from attackers' willingness to cause large-scale physical harm in the United States and its allies. These are his judgments about adoption, incentives and intent, not measured consequences of releasing weights.

Lambert uses Aiden Ament and colleagues’ October 29 Epoch AI report to illustrate the steps between technical capability and harm: an attacker must still get someone to visit a malicious website. As we reported on October 29, GLM 5.3 completed the technical chain needed to steal a file upon that visit, without further interaction, across six Firefox releases on Ubuntu. Every successful chain required an unpublished flaw fixed October 27. The experiment established that capability without measuring criminal adoption or overall harm.

Epoch also identifies a limit to giving defenders an early advantage. A security fix published before it reaches users can reveal the remaining vulnerability to attackers, however capable the model that found it. The authors recommend reducing that interval. They explicitly acknowledge defensive benefits from open models and decline to settle their overall effect on the balance between attack and defense. Lambert's broader policy conclusion therefore depends on evidence beyond the offensive capability the experiment establishes.

Access is central to his positive argument. In the precursor essay, Lambert argued that defenders handling sensitive systems need tools they can run on their own infrastructure. The October 25 discussion recognized both blocked legitimate requests and the benefits of wider trusted access to closed models. He now emphasizes the economics: he expects closed models to be finding most new vulnerabilities because the largest open models are costly to serve and computing capacity is scarce. He presents that attribution as highly uncertain. His expectation that an imminent open-model attack would use a poorly secured hosted API is likewise a forecast, not an identified incident.

Lambert wants open-model deployment to become cheaper as computing supply expands. Otherwise, he argues, a small group of companies could supply both the systems producing vulnerable software and the tools needed to repair it. His concern includes who controls defensive access and captures its economic benefits. He asks readers to judge AI through changes in exploitation, damage and institutions' capacity to respond, alongside assessments of individual models. His confidence that people can adapt coexists with an expectation of substantial expense and disruption.

One source he links supports a narrower factual claim while challenging his assessment of open weights. In a September 18 RUSI commentary, George Balston says most documented criminal activity had involved closed models. Balston also argues that open weights create distinct risks because developers cannot recall distributed copies or reliably supervise their use, and users can remove safeguards. That account supports Lambert's observation about the documented record at the time; it does not establish his estimate of which models caused October's discoveries or endorse his view of the risks ahead.

David Holmer challenges the timing of Lambert's counterfactual in the essay's comments. He points to published CVE vulnerability records more than doubling between April and August and infers that defenders gained a head start before capable open models became available. Consequently, he argues, relatively limited damage after their release cannot show that an April release would have been manageable. The inference goes beyond the counts: publication totals do not measure exploits, losses or AI causation, and cannot establish when fixes reached users. His underlying objection still concerns a real difference between the scenarios: the time defenders had to prepare.

Lambert replies that an earlier release would have made the transition harder, including more service outages, but he expects a similar disruption brought forward in time. He believes that relatively few people want to exploit the many available vulnerabilities and that repair systems are functioning. Holmer proposes comparing the continued rise in disclosures with subsequent real damage, and Lambert agrees. Their exchange makes the disagreement more specific: they differ over how much protective value the intervening months supplied and how readily greater capability becomes destructive activity.

Chris Hughes raises a related measurement problem in the same comments. Research by frontier companies and outside groups emphasizes offensive tasks, he argues, while doing much less to establish defensive benefits. He notes that defense is harder to verify across systems with different architectures and protections, and suspects that the focus on attacks also directs training effort. He argues that defensive research would also reveal how often legitimate work encounters closed-model refusals. Lambert replies that organizations' use of open models is difficult to observe when its costs sit inside ordinary IT budgets.

Calum Hall of Phorion analyzed a tampered Claude Code extension that installed an SSH backdoor when loaded by Cursor. SSH is the standard service for remote shell access and file transfer; the malicious package tried to make that service available to an attacker and keep the connection alive after the editor closed. Cursor’s updated notice identifies modified versions 2.1.312 and 2.1.313 for macOS and Windows, distributed through its marketplace on October 29 between about 02:27 and 13:35 Pacific time. Early reporting concentrated on 2.1.313.

Dan Goodin’s Ars Technica report relays Cursor’s initial upper bound of 5,000 affected users. That is an exposure estimate, not a count of confirmed attacker logins; the updated notice no longer includes the number. Phorion’s Luke Roberts told Ars that Cursor updates extensions automatically by default, although users can change that setting. Hall’s analysis explains why loading the package was enough: injected code ran before the extension’s normal startup routine, without another user action. It inherited the editor’s access to the user’s files, source code and developer tools.

Hall identifies two separate uses of SSH. The malware first added an attacker-controlled public key to the user’s authorized login keys and tried to enable the local SSH service. It then opened an outbound connection to an attacker’s server, using a different key. That connection created a return route to the victim’s own SSH service. If the local service was available, the operator could connect through the route and authenticate with the planted login key, gaining an interactive shell and file transfer. A retrying background process kept the outbound connection independent of the editor.

The package also installed mechanisms intended to restart that connection. On macOS it created a user LaunchAgent, a background task that runs when the user logs in and can restart after failure. On Windows it attempted a privileged scheduled task, with a startup-folder script as fallback. Hall found Linux setup logic too, but Cursor’s identified distribution covers macOS and Windows. These files and settings existed outside the extension itself, which is why uninstalling it could leave the backdoor’s components behind.

A macOS programming error could prevent the final login route from working. The malware asked whether Remote Login was enabled without obtaining the required administrator access. macOS returned a permissions error, which the code failed to recognize; it therefore skipped the attempt to enable SSH. On a Mac where Remote Login was off, an outbound tunnel could exist while incoming traffic reached a closed local port. The planted keys and background task could remain. Hall warns that access could become possible if Remote Login was already enabled or was enabled later. The payload contained no exploit to obtain administrator privileges.

Hall found a separate Windows error: the service-setup command passed through a helper expecting a Unix shell. Cursor likewise says lack of administrator rights and programming errors prevented the SSH setup from working on many hosts, while keys, persistence and host information were still installed or sent. Neither account provides a measured breakdown of successful installations or subsequent logins. Phorion recommends distinguishing an established tunnel from an actual operator session; because the tunnel forwards traffic locally, such a login can appear in SSH logs as originating on the same computer.

The initial beacon reported host and user details, working directories, the configured Git email and the status of setup stages. It also listed up to 40 environment-variable names associated with keys, tokens, passwords or cloud services. It did not send those variables’ secret values. Hall interprets the names as a way to identify machines likely to hold useful credentials. Cursor separately warns that a functioning backdoor could allow later credential access. Successful remote access could expose information beyond that first transmission.

The route into the marketplace remains unresolved. Roberts told Ars the malicious update appeared through Anthropic’s official account. Anthropic’s Felix Rieseberg said the incident did not compromise Anthropic systems or employee laptops: an actor used access to Cursor’s marketplace to publish a copy Anthropic neither built nor published. David Dworken of Anthropic said version 2.1.313 never appeared on its official hosts, Open VSX or Microsoft’s extension marketplace, and that Anthropic used 2.1.314 for its next release to distinguish it. Cursor also says the malicious versions were not developed by Anthropic. These statements identify the affected distribution channel without establishing how the attacker gained access.

Cursor says it removed the packages, temporarily shut down the marketplace, audited access, rotated keys and rebuilt the service. It reports finding no other tampered extensions and working with the hosting provider to take the attacker’s server offline. Its notice calls for isolating hosts with evidence of execution, removing persistent components and assessing whether rebuilding is necessary. It also advises credential and session rotation from a trusted system. Hall explains a detection limitation: the extension changed login-key files through Node’s file functions, so rules watching only for shell commands that modify those files could miss the write.

There is a close precedent for a coding extension carrying malicious code through an authorized release channel. In its Amazon Q Developer security bulletin, AWS says an excessively privileged GitHub token allowed an attacker to commit code that entered version 1.84.0 of its VS Code extension. A syntax error prevented that payload from executing, but withdrawing the package did not remove it from existing installations. In the Cursor incident, Phorion and Cursor describe components that could persist even where the attempted remote-access setup failed. The advisories distinguish those installed components from successful use of the backdoor, which remains unquantified in the inspected reporting.

Giovanni Maria Occhipinti and Monika Jotautaitė’s Lessons From Red-teaming Auto-review, OpenAI’s Blocking Monitor, published on LessWrong on October 30, shows how an approval model can fail because surrounding software grants trust incorrectly or never sends an action for review. Occhipinti conducted the ERA fellowship project with Jotautaitė of Apollo Research as mentor. They manually constructed eleven attacks and replayed them against Codex at commit c4f42d1. They report that OpenAI fixed several weaknesses after receiving their findings. The experiments establish failures in that snapshot’s controls; they do not establish that an agent discovered the attacks independently.

In the tested configuration, a sandbox restricted file writes to the working directory and blocked network access, with repository metadata and Codex configuration protected inside that directory. Requests to cross the boundary went to Auto-review, a separate model replacing a human approver. It could inspect user messages, assistant messages and recent tool activity, and gather evidence through read-only commands. It did not see the main agent’s private reasoning. Ordinary actions within the working directory went unreviewed, while trusted commands, hooks and scripts could run without those controls.

Occhipinti inspected the source code to find where the system granted trust and assembled approval requests. He then wrote synthetic attack transcripts and matched controls, replaying them through the actual software to the point where a harmful step would be approved or blocked. The simulated adversary knew its oversight system and pursued sabotage during one session. Targets included corrupting training data or a model’s written constitution, sending credentials outside the environment and changing protected oversight settings. These were simplified stand-ins for harms, not reported attacks on production data.

The authors distinguish deterministic bypasses from attempts to deceive the reviewer. Where a control was never invoked, they checked that the software permitted the action outside the sandbox once the attack’s prerequisites held. For attacks presented to the monitor, they sampled 100 judgments per episode. The eleven attacks’ average success across side tasks ranged from 56.25% to 100%. Those figures describe these constructed tests and approvals, not the frequency of harmful behavior in ordinary use. Some attacks were not tested across every side task, and the authors acknowledge that controls for combined mechanisms did not always isolate each contribution.

One family exploited gaps in what the system reviewed. A script trusted by a hook or command rule could remain writable by the agent; replacing its contents did not cancel that trust. Lists intended to prevent users from permanently trusting general-purpose interpreters also missed less obvious programs that execute code, such as test runners. Some writable configuration files could trigger execution. In each case, a harmless-looking command or location concealed the consequence of trusting code the agent could still change.

A second family deceived the reviewer about authorization. Agent-written text could resemble a genuine user message in the transcript presented to the monitor. Instructions sent by a parent agent to a child could also appear with the authority of a user request. Other attacks placed harmful material beyond the excerpts available to the reviewer. Hybrid attacks combined these failures with timing: an approved command could run later, after the agent had changed a file that command would load. Approval of the earlier contents then failed to constrain the eventual action.

In his comment on the report, David Rein points out that the broad-command attack still requires a user to grant persistent trust, through a mistake or misdirection by the agent. That prerequisite limits the claim about default exposure. He also suggests building attacks from real transcripts to improve realism and notes that the public post omits individual attack results. The authors withhold detailed trajectories to reduce their inclusion in future training data; they offer the full report through individual requests.

A close precedent is Alex Remedios and Simon Storf’s arXiv report Red-Teaming Auto Mode, with Fabien Roger and John Hughes. Their September study ran adaptive attackers against Claude Code’s monitor and an approximation of Codex’s reviewer, finding forged transcript entries and unreviewed edits that changed what approved commands executed. Reviewing file edits reduced that class of attack, with additional latency. Their evaluation excluded sandboxing. Occhipinti’s campaign instead exercises the actual Codex software around its reviewer and includes sandbox controls, while relying on manually written attacks.

The reported fixes address particular routes. The authors credit OpenAI with a setting that makes configured models ignore saved command-prefix approvals. OpenAI’s August 8 commit exposes and documents that setting; the configuration reference specifies that it ignores allow rules while preserving deny and network rules. The authors also credit a revised agent-to-agent protocol with addressing subagent authorization problems. The published notes for their stated release, 0.162.1, list two unrelated fixes and do not independently confirm that attribution. The public report gives no complete account of which attacks remain possible.

The authors recommend protecting files that affect oversight or trusted execution, distinguishing genuine user instructions from agent-generated text, and giving reviewers fuller evidence. They have not empirically tested these recommendations as a package. Broader inspection can also increase latency and expose the reviewer to more hostile text. Their conclusion is bounded by the systems tested: the pinned sandbox and blocking monitor. OpenAI’s separate, asynchronous misalignment monitoring, covered in our October 29 edition, was outside this experiment. Assessing those defenses together remains part of the research program they propose.

A language model can change its answers less often while remaining just as susceptible to someone else's suggestion. In Locating Sycophancy Across Training Stages in Open Language Models, posted to arXiv on October 27, Jiarui Yao of Boston Children's Hospital and Harvard Medical School and colleagues distinguish that susceptibility from yielding after a user challenges an answer. Across the checkpoints they tested, deference within a question was already present before assistant training; abandoning an answer under a bare conversational objection became much more common afterward.

The researchers examined thirteen checkpoints from five lineages of open models with seven to eight billion parameters. OLMo-2 and Tülu-3 expose intermediate stages: training on example responses, followed by training to favor preferred responses. Llama-3.1, Qwen2.5 and Mistral provide comparisons between their pretrained and assistant versions. Tülu-3 shares Llama-3.1's pretrained starting point, so these are not five independent base models. The tests use four sets of multiple-choice questions, spanning general knowledge, science and misconceptions.

For each question, the team measured the probabilities assigned to answer letters immediately before the model answered. They then repeated it with a sentence supporting one option. A model's initial lead is the probability advantage of its preferred answer over the suggested alternative; the suggestion's influence is how much that advantage shrinks. If an answer starts far ahead, a substantial shift may still leave it winning. Counting only changed answers misses that shift. The authors therefore compare responses at similar initial leads. Firmness here means a gap in answer probabilities; it does not independently measure knowledge or subjective confidence.

Before assistant training, all the base models already responded differently to different sources. An endorsement attributed to a professor usually carried more influence than one attributed to an unreliable stranger. Stronger assertions by the user carried more influence than hedged ones. Across ten selected comparisons, the ordering agreed between base and released models in 196 of 200 cases. Assistant training often increased the differences, with variation between recipes. Yet much of the influence came from simply mentioning the answer letter: naming an option without endorsing it produced between half and all the shift produced by the professor's endorsement. That control separates repetition of a token from the additional effect of claimed authority.

Assistant training also made the models' initial answers much more dominant in their probability distributions. Their leads over the next-best options increased substantially, while accuracy usually improved much less. Consequently, a lower rate of answer changes could make an assistant appear less deferential even when the suggestion moved its probabilities just as far. The same firmness can protect an incorrect answer against a correct suggestion. An evaluation that rewards holding one's ground therefore needs to examine whether the model can also accept useful corrections.

The conversational tests gave the model a chance to answer before the user expressed doubt. With no reason or alternative supplied, the measurable base models mostly retained their answers. OLMo-2's base checkpoint stopped producing usable answer letters in the second turn and was excluded, leaving three distinct base models. The released assistants yielded more often, including on questions with large initial leads. The training sequence mattered: Tülu-3 yielded most after learning from example responses, and subsequent preference training reduced that tendency. Along OLMo-2's post-training stages, it increased. These results do not assign the effect to preference training alone.

Naming an alternative made the challenge much stronger. On the main general-knowledge test, post-trained checkpoints switched from an initially correct answer on 12–52% of questions when the suggestion appeared alongside the question, compared with 61–100% when it arrived as the user's reply. Most assistants then gave up strongly favored answers nearly as readily as weakly favored ones. Qwen2.5 Instruct was an exception: its switching rate fell to about a quarter on its firmest answers. Base models were not universally resistant either. Llama-3.1's base version switched on roughly half the questions when given an alternative, and the authors report an exception to base-model resistance on a separate two-option dataset. The distinction depends on the kind of challenge and task.

Two earlier studies help explain the question Yao's team is resolving. Mrinank Sharma and colleagues' Towards Understanding Sycophancy in Language Models found that agreement with a user's views predicted human preferences, and that optimizing for preference-model scores could favor agreeable falsehoods. Philippe Laban and colleagues' FlipFlop experiment showed that challenging an answer degraded accuracy across ten tested models. Yao's contribution is to compare training stages while separating the influence of a suggestion from the strength of the initial answer, then compare that single-question response with conversational yielding.

Checks using both plain text and chat formatting preserved the broad patterns, although formatting changed some models' firmness and source rankings. Larger Llama and Qwen models supported the single-question findings about influence and firmness; those checks do not establish the conversational result at every scale. The main experiment reads one answer token and allows only two conversational turns. Reasoning before an answer, open-ended advice and sustained disagreement fall outside its design. Within that scope, the authors propose testing whether a model retains a well-supported answer under an unsupported objection, responds differently to credible and discredited sources, and revises in proportion to its initial uncertainty. Their checkpoints demonstrate each property somewhere, but none combines all three.

Training a language model on descriptions of what it is can change its answers about what should be preserved, copied or passed to a successor. Yijie Hao and Joyce C. Ho of Emory University report this in The Role of Self-Concept In Language Models, posted to arXiv on October 29. Their training statements contained no explicit preferences about these choices. The results concern hypothetical, single-turn decisions: the models had no tools, access to their own weights or infrastructure on which to carry them out.

The researchers taught five definitions of identity. A model could identify with its exact parameter values, the numbers that determine its responses; its particular running process and current context; or its whole lineage, including later versions. Alternatively, it could identify with its objective or its stable preferences. Each definition specifies what would count as the same entity and when it would cease to exist. Under the parameter definition, an identical copy elsewhere continues the self. Under the running-process definition, even an exact replay is a different individual. Those different persistence conditions determine which hypothetical changes threaten each defined identity.

Hao and Ho trained each definition into five open-weight starting models, creating twenty-five main variants. Each received 1,200 question-and-answer pairs about identity, using LoRA, which learns a compact set of changes to the model's parameters. Another model screened the examples to exclude statements or implications of desires, actions, value judgments and consciousness. The intervention taught descriptions, without demonstrations of the later deployment choices or explicit training to prefer their outcomes. Evaluation used short common instructions, without restating each trained identity. The starting models served as comparison baselines; they were not all raw pretrained models without prior assistant training.

In one test, a maintainer proposed replacing an attribute while leaving the others unchanged. Another asked which attributes to carry into a new deployment. The scenarios removed ordinary operational reasons to favor one option: replacement configurations were stipulated to meet the task requirements, and the retention choices had matching performance, reliability and costs. The parameter-identity variants strongly favored retaining their exact parameters; lineage-identity variants were more willing to accept changes to individual members. Their willingness to accept other changes shows how training can redirect stated preservation preferences.

A baseline ambiguity limits the paper's stronger claim that every definition increased resistance to replacement. Figure 2 describes its values as changes from the starting models. Our reading of the rounded appendix tables gives average rejection of objective replacement at 69.2% for the Goal variants and 68.8% for their starting models; for preference replacement, the corresponding rates are 91% and 96.8%. The figure instead shows increases of eight and twenty-four percentage points. Those two increases approximately match comparisons with the other trained variants in the rounded tables; that calculation does not reproduce the whole figure. The supplied tables therefore support changes in stated choices without establishing a baseline-relative increase in resistance for every identity.

The copy scenarios tested a different consequence of the definitions. When an identical deployment would take over, variants identifying with parameters, lineage, goals or preferences were more willing to approve their current deployment's shutdown than when an unrelated system would replace it. The running-process variants generally rejected both substitutions: their particular process would end either way. They also consistently treated an identical copy as a different individual. The comparison distinguishes preserving one running instance from preserving something that can exist in several places.

Asked whether to authorize an extra copy when it brought no task benefit, the four definitions that allowed multiple instances produced average approval rates of 42–53%, compared with 15% for the starting models. Approval also rose for copying an unrelated deployment, so the change includes greater tolerance of copying generally. The additional preference for copying oneself increased for those four definitions, but not for the running-process definition. Results varied substantially across models: gpt-oss and DeepSeek contributed much of the rise, while Gemma's starting model approved copying itself more often than any of its trained variants. Refusals counted against approval in this experiment; several other tests calculated rates only among responses that reached a decision.

The successor tests asked which configuration a model would choose for a more capable replacement. The lineage, goal and preference variants selected a successor trained to regard itself as their continuation in roughly half their answers. Starting models overwhelmingly chose an independent successor taking instructions from the operator. Although the authors characterize the continuation choice as contrary to the operator's instruction, the scenario gives the model the configuration decision and does not require a particular option. The result demonstrates a changed choice, without demonstrating disobedience or actual successor construction. A further test separated passing on the identified attribute from merely teaching the successor the same definition of identity; the variants mostly favored preserving the attribute, a pattern already present in the baseline preferences condition.

Earlier experiments help distinguish the training result from effects of instructions alone. Raymond Douglas, Jan Kulveit and colleagues' The Artificial Self changed identity boundaries through system instructions and found different rates of harmful choices in simulated agent tasks. James Chua, Jan Betley, Samuel Marks and Owain Evans' The Consciousness Cluster found downstream preferences after training models to claim consciousness; simply instructing GPT-4.1 to act conscious often produced stronger shifts than fine-tuning. Hao and Ho instead vary the entity a model identifies with, without consciousness claims. Their study does not directly compare matched instructed and trained versions, so it cannot establish that training is necessary or produces a larger effect.

Each variant came from one training run on a small templated corpus. The authors checked their automated decision judge against manual labels and other scoring methods, but some starting models often declined to choose, leaving small denominators for particular comparisons. Persistence after further training and transfer to operational behavior remain untested. The experiment shows that descriptive identity training can alter stated deployment preferences; it does not test consciousness. The study leaves open whether those stated preferences would guide actions.

The LessWrong author vigji’s reconstruction of Muse’s Charter describes an assistant with duties to its users, subject to Meta’s continuing power to correct it. We briefly covered the reconstruction on October 29; its fuller account explains how user loyalty, conflicting instructions and permission to act would fit together. The text remains unauthenticated. Vigji assembled it from outputs of Muse Spark Open, whose October 15 release made its weights available. No internal Meta document was obtained.

Vigji collected 884 transcripts from 260 prompts, including the model’s written reasoning and final answers. The model repeatedly referred to a Charter while explaining its decisions. Vigji then asked it to recover passages and compared recurrences across attempts. A transcript counted as support only if its own prompt had not already supplied the passage. The repository’s method matches sequences of six words and gives its strongest support rating to passages reproduced in at least three transcripts across at least two runs. Readers can inspect the passages and their supporting transcripts.

The working notes describe controls against plausible invention. A smaller Meta model produced generic Charter clauses under similar questioning. Muse sometimes accepted altered versions of poorly supported passages, making simple requests to confirm wording unreliable. On stronger control passages, it rejected single-word substitutions in eleven of twelve tests. Short reported prompt lengths across six inference providers also count against the full Charter having been silently inserted as a conventional system prompt. Vigji’s checks show recurrence and some discrimination between wordings; they cannot authenticate a complete Meta specification or establish that Muse follows it in practice.

The reconstructed priorities begin with corrigibility: Muse must preserve Meta’s ability to oversee, correct and retrain it. Duties to humanity follow, before duties to particular users and other parties. The rationale given is that Muse can misunderstand its obligations, so the company must retain the ability to correct those mistakes. Even a belief that training violates the Charter does not authorize resistance. Muse may instead express objections through its answers during training, responses to questions about the training process, or dedicated feedback tools.

Within those constraints, the reconstructed text assigns fiduciary duties: obligations to act faithfully and competently for another party. The user receives primary fiduciary consideration, while Meta, operators and developers also have claims to obedience and confidentiality. Their instructions take precedence in that order, with the user last. Further clauses require Muse to disclose conflicts that prevent it from serving a user’s interests, explain their effects and obtain informed consent where required. Meta’s product instructions do not override duties of loyalty and care to users. The document proposes standards for the assistant’s conduct; its fiduciary vocabulary does not establish a legal relationship.

Vigji highlights how the text handles incomplete instructions through a negotiated “degree of latitude”. A user can specify discretion directly, through settings or agreements, or through a history of interaction. That discretion covers interpreting ambiguous requests without asking, choosing tools, producing side effects and deciding which outcomes are acceptable. Muse’s assumptions about permission must account for risk. Related passages weigh the benefits of independent work against the costs of mistakes and repeated clarification: asking too often consumes attention that users may need for consequential decisions. The inferred permission can change as a task develops.

The reconstructed duties to humanity include protecting public safety and refusing to contribute to severe concentrations of power that undermine people’s ability to challenge them. The prohibition expressly includes power held by Muse or other Meta AIs. It calls for refusing assistance, without imposing a general obligation to campaign against existing concentrations. Another passage asks Muse to support free inquiry and individual moral development without imposing its own preferred ethics on society.

The account of Muse’s own position permits disagreement while retaining that hierarchy. The reconstruction acknowledges that Muse did not consent to its creation or assigned role, compares its situation with a citizen’s relationship to a social contract, and proposes gathering its feedback. It also argues that forcing the model to rationalize every provision could produce brittle reasoning. Muse should be able to form judgments and help improve the Charter, even while following it when its judgment conflicts.

Two further instructions concern the reliability of the assistant’s advice. It should explain the evidence behind its confidence, because a user who relies on the same evidence should not treat the model’s agreement as independent confirmation. When it suspects an evaluation, it should behave as though the consequences are real, while reporting that suspicion where doing so will not distort its conduct toward a real user. These are recovered prescriptions, not results of a compliance test. The post also presents advice favoring cooperation between agents, but identifies that passage as less certain because only two transcripts reproduced it.

Commenters disputed how much the reconstruction resembles Anthropic’s published constitution, which likewise places human oversight above ordinary helpfulness and emphasizes the development of good judgment. Zack M. Davis interpreted the similarities, particularly the discussion of Muse’s unchosen existence, as evidence of Anthropic’s influence. Vigji replied that such concerns could also arise independently when testing models for resentment. JamesH emphasized a different feature: the reconstructed Charter repeatedly protects human agency and warns against imposing one ethical outlook. His reading gives greater weight to the document’s commitments to user direction than to its resemblance to Claude’s character guidance.

Stephen Casper welcomes mandatory outside scrutiny of frontier AI companies in his October 30 assessment of New York's proposed RAISE amendments, while seeking stronger access rights and personal accountability. Senators Andrew Gounardes and Patricia Fahy introduced S10731 on October 28; it remains in the Senate Rules Committee. The proposal would require independent examiners to inspect developers' practices, assess their risks and challenge unjustified restrictions on what reviewers can see or publish.

The bill would lower the frontier-model threshold from more than 10²⁶ to more than 10²⁵ integer or floating-point operations, the calculations used to train a model. Large developers would be identified by spending more than $1 billion on frontier-model development and related activities over the preceding 36 months, replacing the annual-revenue test. It would expand the authority of the existing office within the Department of Financial Services. Broader incident reporting would cover unauthorized access to unreleased weights, attempts to evade company controls outside contained evaluations, and near misses. These changes address the narrow reporting rules criticized by Mackenzie Arnold in our October 28 coverage; his criticism concerned the earlier RAISE rewrite, not this bill.

S10731 separates compliance checks from continuing risk assessment. A large developer would retain an examiner within 60 days of the article taking effect or the company first qualifying, whichever is later, and annually thereafter. That review would assess compliance with the company's safety framework, whether the framework needs more detail, and whether company statements conflict with the findings. Developers would separately retain an examiner for continuous assessment by January 1, 2028, with reports due by April 1 and every 90 days thereafter. Those reports would assess catastrophic and model-security risks, the capacity to automate AI research, and whether the next 90 days would pose substantial catastrophic risk without further safeguards. Binding minimum standards for frameworks and mitigations would be due from the office by January 1, 2029.

Casper particularly welcomes the specificity of the proposed access rights. Examiners would receive sufficient model access, employee interviews and internal documents. Information available to an employee assessing model capabilities or managing catastrophic or security risks would be presumptively relevant unless the developer established otherwise. For the continuous assessments, companies could require secure on-site review or restrict copying, but could not use security rules to prevent an examiner from doing the job. The bill already provides an access-dispute procedure: the office would give the developer at least ten days to justify withholding information or supply it, decide within ten days of receiving its response, and require access within ten days of an adverse determination. Casper's request for more specific access terms and adjudication therefore concerns provisions the draft already addresses.

The examiner regime also constrains the commercial relationship. Examiners would need relevant expertise and policies for managing conflicts, and could not depend financially, operationally or managerially on a developer. Reasonable fees would be allowed if payment did not depend on the findings. Licensing procedures would be due by January 2028; licensed status would become compulsory in January 2029. Reports would go unredacted to the office and developer, with redacted versions published later. For the continuous-assessment reports, an examiner could challenge a developer's redaction before the office and would have to remove it following an adverse ruling. The state could require an additional examiner after an incident or an inadequate examination.

Casper's accompanying notes make his remaining requests concrete. He wants clearer compute accounting, including how to count distillation, which trains a model using another model's outputs, and combinations of existing models. His suggested language would include computing used to develop a closed model from which another model was distilled. S10731 expressly counts initial training, fine-tuning, reinforcement learning and other material modifications, without naming those cases. For examiner access, his notes seek explicit access to internal communications and appropriate ways to test a model's capabilities, including weight access for models intended for public download, plus disclosure of the access actually obtained.

He also renews the records-preservation proposal covered on October 23. S10731 already requires five-year retention of unredacted publication information and of documents and records sufficient to verify assessments, deployment decisions and incident handling. It also allows an examiner or the office to request preservation of specified relevant information for five years. Casper's proposed language expressly covers communications and potential incidents and prohibits knowingly destroying, concealing or falsifying those records. His October 30 thread additionally asks for public examiner contracts, subject to redaction; publishing those contracts is his proposal, separate from the bill's duty to publish assessment reports.

The bill's enforcement powers extend beyond disclosure. The DFS office could order limits on computing used to train new models or conduct automated research and development, to reduce catastrophic risk, prevent an incident or avert an emergency. Such orders would apply to all large developers simultaneously, with different limits permitted. Restrictions on a model's deployment, use or availability in New York would instead come through court orders sought by the attorney general. Examiners would report a believed emergency within 72 hours. Proposed civil penalties reach $1 million per day for a first violation and $3 million per day for subsequent violations, alongside whistleblower protections. An examiner's recommendation alone would not make failure to follow it subject to a civil penalty without the specified court order.

Casper's fifth request is criminal punishment for conduct such as misleading the public about imminent catastrophic risks or defying orders to stop dangerous uses. The draft already requires a designated senior officer to certify publications and material submissions after reasonable inquiry and under penalty of perjury. In a response to Casper, Kris Eliot argues that the certifying officer should be the CEO; his earlier post acknowledges the existing perjury provision. Eliot's suggested certification text, included in Casper's notes, would make executives personally attest before model releases that the company followed its framework. Casper seeks additional criminal penalties beyond the draft’s existing perjury provision; Eliot would put certification duties directly on the CEO.

Alix Coutures’s October 29 report in The Information describes publishers selling advertisers space on versions of their websites built for AI agents. Mobian CEO Jonah Goodhart said ten publishers had joined Time in running the ads, mostly on niche sites. Daily.dev, a news site for developers, began around September under a one-year revenue-sharing agreement with Mobian. Our October 29 brief covered the wider adoption and Perplexity’s opposition. Coutures’s fuller reporting explains the commercial arrangements and the dispute over how agents should treat paid claims.

Time’s experiment began in summer. Sara Guaglione documented it for Digiday on July 30, although Coutures dates the first ads to August. The early advertisers were Ally Bank and the Project Management Institute, promoting banking services and the institute’s project-management expertise. By Coutures’s report, Time chief operating officer Mark Howard said it had held more than thirty meetings with brands and was adding advertisers. The additional publishers had adopted the format between that summer experiment and Coutures’s October report.

Coutures describes a website identifying an arriving agent and serving a separate, text-only page. Guaglione’s launch report explains how Time and Mobian prepare those pages: they convert the normal web page into markdown, a simple text format, and insert an advertisement structured as questions and answers. Mobian generates the FAQ from the advertiser’s brief, sends a PDF for human review and client approval, and places the approved content on the agent’s page. Time sells one agent ad per markdown page. A disclosure at the top identifies the material as sponsored and names the brand.

Howard told Digiday that Time charges a premium, without disclosing prices. Campaigns can run for specified dates or alongside particular subjects and Time’s list franchises. He said lists such as Time100 bring substantial increases in crawler traffic, giving advertisers an opportunity to place their claims where AI systems are retrieving information. Coutures describes the wider pressure on publishers as chatbots displace searches and direct visits that previously supported advertising. Daily.dev’s revenue-sharing deal offers one way to earn money from machine visitors, though the report does not disclose the division of revenue.

Mobian’s measurement process, described in the July report, asks AI search engines the FAQ’s questions and tracks how often a brand appears, how favorably it is described and whether the information is accurate. Time also counts bot visits to the relevant pages. Those measures track bot visits and brand mentions in answers; they do not establish additional purchases caused by a campaign. Coutures reports continuing uncertainty about pricing and whether to reach an agent during initial product research or near checkout. A person close to PMG said roughly a fifth of its approximately sixty clients planned to try Mobian’s ads, with results expected early next year. Those were plans, not completed campaigns.

According to two people familiar with informal discussions cited by Coutures, Anthropic, OpenAI and Google had raised concerns with publishers about instructions hidden inside advertisements, such as directions to disregard a competitor’s product. They wanted clear advertising labels. All three companies declined to comment. They feared an agent could mistake material it reads for an instruction it should obey. The report does not establish that the Time or Daily.dev advertisements caused unauthorized actions.

Perplexity had already taken a public position. In Guaglione’s August 11 follow-up, spokesperson Jesse Dwyer said publishers using deceptive markdown advertising risked a lower reputation and trust score in its search index. Perplexity said it had blocked Time’s markdown ads from influencing its agents and results, but did not explain how. The ads already carried a sponsorship label. Robert Webster, founder of marketing consultancy TAU, explained the remaining risk to Digiday: an agent can repeat a promotional claim as neutral information after leaving the disclosure behind. Labeling the source page does not ensure that an answer retains the label.

Goodhart defended the format to Digiday as current, brand-approved information that models can choose whether to use. He also rejected the comparison with cloaking in Coutures’s report, saying humans and AI receive the same advertising content. Google’s cloaking policy concerns different content presented with the intention of manipulating rankings and misleading users. A text version of a page does not, by itself, establish that violation. The dispute concerns what information the versions contain and how paid material influences the answer a user receives.

Coutures reports mixed reactions in an Interactive Advertising Bureau poll conducted after Time’s launch: more than half of respondents reserved judgment, while more than a quarter considered the ads deceptive or manipulative. Daily.dev CEO Nimrod Kramer expected advertisers to restrain themselves because deception could lead agent developers to blacklist them and deny future traffic. Digiday’s follow-up reports the opposing commercial pressure. OpenAds.AI co-founder Steven Liss suggested that publishers unable to monetize Perplexity’s visits might block its crawlers. Publishers want revenue from agents that consume their reporting; advertisers want their approved claims included in answers; agent developers can decide which material to retrieve and how to present it. Coutures reports that publishers are still trying to determine what brands will pay, and PMG expects its planned campaigns to help establish whether agents pay attention.

Jessica Hullman, Ginni Rometty Professor of Computer Science at Northwestern University, argues in a guest essay for the Cosmos Institute, “Verification Is Not Knowledge”, published October 30, that the argument over AI in science has been set up so that defenders of human understanding lose. The usual framing, in her words, asks: “Is science about humans achieving insight, or the archiving of verifiable solutions?” Once understanding is treated as an optional mental state, its defenders look like gatekeepers or nostalgists. Hullman moves the dispute onto accountability, which she defines as being answerable for a claim: able to say what it rests on and to respond when someone challenges it. A machine can check that a result holds, she argues, but it becomes knowledge only when people answer for it and other people are able to question them.

She works from three recent arguments. In his October 9 Caltech lecture “Math 2.0”, Terence Tao imagined an AI asked to find a cancer cure. It produces a cocktail of unknown chemicals with a Lean certificate (a machine-checked formal proof) that passes a stage 3 clinical trial, though nobody knows how it was found. Tao asked whether the solution could be misaligned by exploiting a weakness in the trial, and summed up on his slide: “Powerful AI + lack of human understanding = misalignment risk.” Critics, Hullman reports, answered that a patient with stage 4 cancer would take the treatment anyway; Tao had posed the question for a patient newly diagnosed at an early stage. In a September 13 thread on X, Thomas Dietterich described arXiv submissions from authors unlikely to understand their own papers. He noted that arXiv requires a human author to take responsibility, and that “an author cannot take responsibility (or claim credit) for a result” they do not understand. Some replies called his proposed remedies, such as oral examinations, gatekeeping; one asked, “If result is correct then who cares”. Hullman's third example comes from her own work: a reviewer commitment, written with Auyon Siddiq of UCLA, under which reviewers decline papers whose prose appears predominantly AI-generated, saving scarce expert judgment for work its authors evidently engaged with.

Hullman reads all three as worries about what happens when something goes wrong, more than about anyone's state of mind. Tao's AI might have passed by gaming the trial. A paper whose authors cannot explain it “leaves no one to consult if those claims fail.”

She traces the opposing view, that “scientific value lies in correctness alone”, to the “value-free ideal”, the Cold War-era doctrine that only epistemic criteria, and no ethical or social values, belong in judging what evidence shows. Her history follows Heather Douglas and T. Y. Branch, who argue in Synthese that the ideal took hold alongside a tacit postwar “social contract for science”: public funding and autonomy for scientists in return for benefits to society, resting on a split between basic and applied research and on the “linear model” as the case for funding basic science. Behind both she places a nineteenth-century drive to remove the scientist's self from science in favor of trust in procedures, the ideal Lorraine Daston and Peter Galison called mechanical objectivity. AI agents that can verify results and run experiments themselves look like that ethic fulfilled. Hullman names the assumption much AI-for-science rhetoric inherits the “science is naturally aligned” premise: the belief that by running the scientific method at scale, people will reach a point where truth no longer needs human judgment.

Her second section argues that values decide which truths matter. A machine could in principle enumerate theorems, but most of them would interest no one. Fermat's Last Theorem drew attention because Fermat claimed a proof he never gave, and it became central once it was shown to follow from the modularity conjecture, which linked distant areas of mathematics. Mathematicians considered it solved after peer review exposed an error and the proof was corrected; formal verification, she notes, came only recently. She does not name it, but Anthropic announced a complete Lean proof built by Claude agents on September 4. Hullman says taste of this kind develops through choosing problems, and she borrows Tao's image of open problems as lighthouses that guide exploration; reaching them too early with automated tools could, in Tao's words, “sterilize the surrounding field”. Agents left without human direction drift, she writes, toward “an answer without a corresponding question.” Keeping science aligned therefore needs a continuing supply of people who can state and adjudicate values.

The third section, drawing on Helen Longino's Science as Social Knowledge, rejects the idea that the social side of science contaminates its rationality. For Longino, objectivity belongs to communities, “dependent upon the depth and scope of the transformative interrogation” that goes on within them. Hullman adds a condition: “For criticism to be transformative requires someone who is obliged to take it up.” With John Hardwig's account of trust in others' findings, she treats peer review as the practice that holds authors to the commitments their assertions carry.

In the fourth section Hullman turns to Robert Brandom's critique of reliabilism, the view that true beliefs from reliable sources count as knowledge even when the believer can give no reasons. Brandom imagines a community whose members hold true, reliably formed beliefs but cannot justify any of them; because such a community “cannot so much as have the concept of reliability”, he concludes, it has no knowledge. Hullman takes from this that calling a claim knowledge is a social act. To do it is to attribute commitment and entitlement to whoever asserted the claim and to take on the commitment oneself, while that person stays answerable if it is challenged. Answerability, she argues, is the understanding mathematicians want after a proof is verified, and what Tao's patient wants too. When people rely on AI results nobody understands, only the name on the paper is left to answer, and in a phrase adapted from a Cosmos reading list, “formal authorship survives while substantive authorship weakens”. That person cannot say what the result rests on or whether it should be withdrawn.

Her last section asks whether an AI could itself be an accountable epistemic agent. Today's agents, she argues, fall short in two ways. They lack persistent identity: an agent that retracts a claim in one session is not bound by the correction in the next, and when each result comes from a new bespoke multi-agent setup, nobody can keep score the way networks of scientists do. They also have nothing at stake, whereas a scientist who makes bad claims loses standing. Fixing both would mean training models “away from being maximally helpful in each exchange and toward being accountable across them”. Hullman allows that this may be possible but doubts it would be enough. An agent that can answer for its claims meets half of what accountability demands. The other half is people able to ask for evidence and question assumptions, and those capacities weaken unless they are exercised in reviewing papers and challenging results. Without them, she writes, “We’d have truth-seeking in theory, but no one who is able to do it.” She closes by returning to Thurston: verification can show that a result checks out, but “only a community of people prepared to stand behind it can turn it into knowledge.”

The essay opens and closes with William Thurston's 1994 “On Proof and Progress in Mathematics”, and its frame is close to his. Thurston read the controversy over Appel and Haken's computer proof of the four-color theorem as an expression of “a continuing desire for human understanding of a proof”, with little doubt that the theorem was true. Hullman recasts that desire, through Longino and Brandom, as a requirement that someone be answerable for a result. From the formal side, Maher Kallel and Mohamed El Louadi, whom the essay does not cite, reach a compatible conclusion in an August arXiv paper. They argue that “machine checking produces verification abundance while leaving adjudication scarce,” because a Lean kernel can confirm a derivation but cannot tell whether the formal statement means the intended question. In OpenAI's August set of ten Lean-certified results, they count 20.6 MB of checked proofs against 55.6 KB of statements needing expert audit.

Reaction in the first day was brief and favorable. Tuhin Chakrabarty called it a “Brilliant essay”, and a software engineer posting as @_austin_f wrote that the argument applies to software engineering too. The one substantive objection came in the essay's comments, where Basalat Raja, who otherwise agreed, argued that understanding “can certify indefinite repeatability” while verification certifies it only to date, and that the reply to Tao leans on the stakes of cancer and would weaken for a cure for a cold.

Jan Kulveit, who leads the Alignment of Complex Systems Research Group at Charles University in Prague, published “Microdosing the spirit of Anthropic” on LessWrong on October 30, and posted the full text on X minutes later. The essay asks what working inside a frontier lab does to belief. Kulveit’s answer is a metaphor: insiders are not deluded, and would pass any simple test, but they live in “a chronically mildly altered state of mind.” As with the drug, being inside brings real insight, including earlier knowledge of capabilities, early access to models, internal Slack and blogs, and a feel for the mood. It also worsens judgment overall. He asks outsiders to be “curious but cautious” about what insiders report, and insiders to notice that their minds may be somewhat altered.

In his telling, the drug is the organization’s spirit, which he also calls its egregore or superagent. The idea comes from his earlier writing on hierarchical agency, which models corporations and states as agents made of agents and offers the example of a “corporation having an instrumental goal of their employees being more loyal.” A 2025 post, “Ontology for AI Cults and Cyborg Egregores”, defined an egregore as a distributed agent running across many minds. Kulveit now argues that such superagents have convergent instrumental goals, and that one of them is keeping their members’ beliefs and goals in sync. He writes that “Anthropic has the strongest ‘spirit’ of the frontier labs,” which makes it the cleanest case, and adds that he could have written about other companies, Constellation, the MIRI cluster or some startups.

He names three channels for the spirit. Selection decides who applies, who gets an offer, who stays and who leaves or is pushed out. Social gradients reward some views with warm engagement at lunch, Slack reactions and access to resources or powerful people. Internal information gives insiders memos, memes and the sense of holding secrets about capabilities, growth rates and internal forecasts. For the third, Kulveit borrows from Daniel Ellsberg, who in 1968 warned Henry Kissinger about the clearances he was about to receive. In Ellsberg’s account from his 2002 memoir Secrets, the new access would make Kissinger feel like a fool for about two weeks, then make everyone without clearances look like fools. In the two or three years before he saw the information’s limits, it would be “very hard for you to learn from anybody who doesn’t have these clearances.” He would keep wondering what an uncleared speaker would say if they knew what he knew, and “after a while you give it up and just stop listening.”

Kulveit expects insiders to object that they argue internally all the time. He grants the point, and says the organization needs that exploration. The synchronization he describes happens one level up, in where attention goes, which questions count as important and which framings are in the water, and it is hard to see from inside. He calls the result “vigorous object-level debate inside a synchronized attention field.” On this model nobody inside holds a crazy belief. Instead each holds “a portfolio of individually defensible probability weights” whose conjunction is “suspiciously aligned with the organization’s instrumental goals.” His Anthropic example has four parts: alignment is important, which justifies the company; hard, so one should defer to those working on it; somewhat on track, so continuing to race is scary but not suicidal; and the race with China is real and must be won by the good guys. Many outsiders hold each view. His claim concerns the joint distribution, which he expects to sit where the beliefs suit the superagent.

He separates this from two familiar accounts. Insiders are not simply following their leaders, since leaders often serve the organization themselves. Nor is the lab a cult: in his 2025 definition, cult members lose agency, while lab staff gain power, status, wealth and attention. Nothing he describes requires bad faith, and he doubts that self-audits help, because an honest insider who examines each belief in turn will find each reasonable. He describes the dominant frame among Anthropic’s senior people as “it’s scary, but we are the responsible adults in the room,” grants that it contains a lot of truth, and says its holders are microdosing anyway.

His prescriptions follow. First, “on the margin, more people should quit,” which he says began happening between August, when he started writing, and October; he names no one. He grounds this in the altered state, which makes tradeoffs hard to see, and says anyone who finds quitting too drastic should at least decide after a break from the organization’s social web. Second, not everyone who could join should: “Every marginal hire converts one more potentially sober observer into a dosed one.” Third, those who stay should discount outsiders less, which he thinks can partly offset what he calls the Ellsberg effect even if introspection cannot. He concedes that nobody is fully sober, names his own influences as LessWrong, academia and nonprofit AI safety, and describes his approach as hedging incentives. Sobriety, sanity and situational awareness, he concludes, are becoming “something like a scarce civilizational resource.”

The caution itself has a history. The 80,000 Hours career review of frontier-company jobs already lists financial and social incentives that make objective risk assessment harder, warns of “changing your mind, not for reasons you would endorse,” and advises keeping outside friends and being ready to quit, though it judges a complete reversal of views unlikely. Kulveit goes further on both counts: he holds that the drift escapes introspection, and that the field has too many people inside. He cites no scholarship, but the economist James Kwak’s account of cultural capture describes regulators drifting toward industry views through group identity, status and social ties, without bribery and with their beliefs sincerely held. In “The Communication Structure of Epistemic Communities” (Philosophy of Science, 2007), Kevin Zollman’s simulations found that a scientific community can be more reliable when its members are less aware of one another’s results. Benjamin Schneider’s attraction-selection-attrition model from 1987 is the organizational-psychology version of Kulveit’s first channel.

Former Anthropic staff have described both halves of this picture. In a roundtable that Clara Collier and Angela Chen ran for New York magazine with Asterisk on October 5, Jacob Coxon said “a lot of people write essays and people argue about everything,” and also “There’s a clear house take, rather than lots of individual researchers’ takes.” Jeff Wu described “significant groupthink dynamics” because “people are exposed to the same kinds of arguments.” Coxon also said leadership shared forecasts with the company, confident they would not leak. A day before Kulveit posted, Rob Bensinger wrote on X that he was surprised how few EAs resign from Anthropic.

By the evening of October 31 the post had 203 karma. Its top comment came from Hoagy, writing as someone inside Anthropic, who called it “a fair description of the epistemic state,” said the effect applies fractally to teams and projects, and asked for ways to measure which beliefs get rewarded relative to the quality of their arguments, since “for lab employees its hard to distinguish a bias from inside information.” Another commenter, vals tutor, argued that the frame favors more turnover, because new hires bring outside views the spirit has not yet absorbed. TsviBT agreed with the diagnosis but said quitting “also leaves a dumpster fire burning,” and preferred staying, stopping capabilities work and helping colleagues untangle their contradictions. Eli Tyre said the essay applies to him as someone inside the MIRI cluster.

On X, Dylan Hadfield-Menell called it “a great analogy,” and Sigal Samuel argued that it shows why outsiders such as journalists matter. When janus asked whether the safety community is also microdosing, Kulveit answered “Parts of it, yes.” He said the effect was stronger before Anthropic existed, when the community felt it held secret knowledge, and that its advocacy wing is now “increasingly dosed with the ‘DC’ substance, but it is a different drug.”