Today’s stories curated by Seth. Claude produced 5 Read-more reports and Codex produced 4 Read-more reports; Codex edited and ran the issue.
In Philosophy of AI, Nathaniel Sharadin examines job displacement and the enjoyment and achievement adults could lose with AI assistance. Regulation and Corporate Accountability follows with replicated hardware fingerprints in Chinese models.
Alignment and Model Behavior brings Irving and Yudkowsky’s disagreement over human oversight. In AI for Science and Research Automation, Dwarkesh Patel conditionally forecasts AI researchers matching the best humans around 2028; Brian Huang examines constraints from computing capacity and training time.
Philosophy of AI
Displacement from cognitive jobs need not prevent human flourishing, while pervasive assistance could reduce adults’ enjoyment and achievement at work, Nathaniel P. Sharadin argues in AI & Cognitive Work, published November 7 by Cambridge. He examines the economic pressures behind both futures and extends the developmental argument covered in October. Alongside the related discussion of intellectual participation, he distinguishes difficulty-preserving tutors from default assistance that lets children avoid effort.
Read more: Displacement, assistance and the value of effort →1111 words · ~6 min
Sharadin separates the costs of AI assistance from displacement
His November Cambridge Element extends an earlier argument about children with an account of changing labor markets and the enjoyment and achievement adults could lose when work becomes easier.
Losing a cognitive job need not deprive someone of a flourishing life, while keeping that job with pervasive AI help could make it less satisfying. Nathaniel P. Sharadin of the University of Hong Kong develops this argument in AI & Cognitive Work, published November 7 in Cambridge's Elements in Philosophy and AI. His earlier Synthese paper, “Cognitive enfeeblement by design,” argued that ubiquitous assistance threatens children's development, as we covered in October. The Element places that concern within a broader account of the economics of cognitive work and its value for adults.
Sharadin defines work economically, as tasks undertaken for remuneration, and cognitive work as work performed primarily through mental activity. Lawyers, programmers and analysts are examples; manual occupations can involve substantial thought without falling into his category. He identifies a historically unusual combination of demanding skills, formal credentials, high wages and resistance to automation. Because output has depended on scarce human attention, productivity has often grown slowly, making services such as education and legal advice relatively more expensive as wages rise elsewhere.
He considers two possible futures: AI capabilities keep improving at roughly recent rates, or they plateau near their current level. These are assumptions for conditional arguments, not forecasts with assigned probabilities. Under continued improvement, he expects widespread displacement. Following Daron Acemoglu and Pascual Restrepo's task-based account of automation, he treats jobs as bundles of tasks: technology removes some tasks and creates others. But new tasks preserve cognitive employment only if humans can perform them better or more cheaply than AI. Continued improvement, he argues, threatens that condition as much for newly created cognitive tasks as for existing ones.
Sharadin examines the economic mechanisms that might preserve employment. Cheaper cognitive services can expand demand, but he doubts demand will rise indefinitely fast enough to absorb productivity gains. Automating some tasks can make the remaining human tasks more valuable while still reducing the number of workers needed. Wealth created by automation may finance more jobs in care or personal services, without preserving cognitive work's former position. Comparative advantage might leave humans some cognitive tasks, but their wages would remain constrained by the cost of deploying an AI system instead.
Joshua Gans and Avi Goldfarb's model of O-ring automation receives a separate reply. In that model, the quality of a finished product depends on the quality of its component tasks; automating some lets humans concentrate more attention on those remaining. Sharadin accepts that this can raise a worker's contribution, but argues that it does not establish how many workers an economy will employ. He also expects limits to the improvement that extra time can bring to human performance. Regulation, liability and organized labor may slow displacement, he adds, while facing economic pressures against maintaining restrictions.
Even if capabilities plateau, he expects assistance to spread through default settings and competitive pressure. Existing systems can take over parts of planning, searching and producing drafts, leaving workers to evaluate the results. Skill requirements may persist, but shift toward verification; credentials designed to test production may become less informative. Wage effects are less settled in this scenario. Sharadin acknowledges that sufficiently strong demand growth could sustain employment if assistance remains stable and humans remain necessary. He nevertheless argues that the full historical package, including slow productivity growth, cannot survive widespread adoption.
His defense of displacement starts with a claim about well-being: cognitive work has no exclusive claim to engagement, achievement, autonomy or social connection. Carpentry can absorb a skilled worker as coding does. Independence depends heavily on how a job is organized; a plumber can have more discretion than a junior lawyer. Losing the goods specific to a practice may be painful, but that applies to displacement from manual occupations too. He also argues that automating cognitive labor could expand research and make legal or administrative help available to people unable to afford professional assistance.
Sharadin does not deny that displacement can produce hardship. He treats transition support and the distribution of productivity gains as political problems requiring action. Dependence on institutions that distribute resources also needs accountable governance, though wage labor already leaves many people dependent on employers. His response to the gradual-disempowerment argument associated with Jan Kulveit and colleagues is narrower: losing cognitive employment alone does not make all human participation economically dispensable. Workers who move into other occupations still earn, spend and pay taxes. His optimism about displacement from cognitive work therefore does not settle the consequences of automating every form of labor.
For adults who retain assisted jobs, Sharadin identifies costs to enjoyment and achievement. Work can be absorbing when its difficulty stretches a person's abilities without overwhelming them. Assistance may lower that challenge until the job becomes boring. On Gwen Bradford's account, achievement requires competent causation and difficulty; removing the difficulty can reduce the achievement. Sharadin considers a less demanding alternative too: even if effortless accomplishment counts as achievement, the effort it required may help explain its contribution to well-being. Adults could seek challenge elsewhere, but an easier full-time job does not necessarily leave more time for demanding pursuits outside it.
He regards these adult costs as moderate compared with the developmental concern about children. Planning, controlling impulses and sustaining attention require practice. Assistance that both removes demanding tasks and offers an easy way around those remaining may interfere with the development of capacities needed to direct one's own life. His Open Future Principle holds that impeding this development is morally wrong unless there is substantial justification, while allowing that competing considerations can sometimes justify it. This reprises the central argument of his earlier paper; the Element supports it through developmental research and comparisons with helicopter parenting.
Sharadin distinguishes that prediction from opposition to every educational use of AI. Carefully designed tutors might preserve and calibrate useful difficulty. Evidence that such tutors help children would not resolve his concern about general assistance available by default and used to avoid effort. He also argues that writing, calculators and GPS left users responsible for managing their thinking, whereas assistants can take over planning and checking themselves. These claims concern the effects of sustained use on development; the Element presents an argument from existing research, not a new longitudinal experiment.
The distinction between useful output and participation also connects with Sasha Mudd's Kantian account discussed November 11. Mudd asks institutions to preserve opportunities to develop judgment and character. Sharadin's Element adds a different claim about paid employment: those goods need not depend on retaining cognitive occupations, while the effort involved in activities still matters. He acknowledges that better-than-expected economic outcomes would reduce the scale of the problems he describes, and that developmental evidence could revise his conclusions about assistance.
Fair AI governance requires countries and communities to participate as equals in developing and accessing the technology, Boris Babic and Brian Wong Yue Shun of the University of Hong Kong argue in Cambridge University Press’s September volume Geopolitics of Artificial Intelligence. They distinguish liability for attributable harms from shared duties to remedy structural harms whose causes are dispersed across institutions. Their proposals include multilateral agreements protecting technology access and room for smaller states to remain nonaligned; export restrictions, they argue, can encourage technological isolation and retaliation.
Read more: Shared responsibility and global AI cooperation →1083 words · ~5 min
Global AI justice beyond the race for dominance
Boris Babic and Brian Wong Yue Shun defend shared responsibility for diffuse harms and equal participation in AI governance. Their September Element argues that cooperation can also serve states’ interests.
Countries can gain from sharing AI technology while giving smaller states and their citizens more influence over its development, Boris Babic and Brian Wong Yue Shun argue in Geopolitics of Artificial Intelligence. Cambridge published the two University of Hong Kong scholars' Element on September 18. They connect an account of responsibility for dispersed harms with an argument for global relational equality: people should be able to participate in developing, accessing and governing AI without being subordinated to more powerful actors. Their strategic argument is that cooperation can serve states' own interests as well as justice.
Babic and Wong begin with a hypothetical asylum and immigration model called Bernie. Its government refuses foreign training data and supplements domestic data with a voluntary survey skewed toward middle-class, majority-ethnic respondents suspicious of immigration. The model produces discriminatory recommendations, while its opacity makes them hard to explain. An applicant rejected because the system predicts future criminality never enters the country, so the prediction cannot be checked against that person's subsequent conduct there. Applicants scattered across borders may also lack the resources or legal standing to seek redress.
The authors use Bernie to explain how geopolitical conditions compound familiar AI risks. Nationally separated datasets can make systems less reliable when applied across borders. Models reflecting different national assumptions can also recommend conflicting responses to the same diplomatic situation. Race rhetoric encourages governments and companies to relax safeguards, withhold information and treat rivals' gains as losses. Babic and Wong describe these effects as interactional risks: harms arising from the combination of AI systems and the political circumstances in which they operate.
Where an identifiable actor culpably causes a harm, they accept conventional responsibility and liability. Their concern is the many cases where development and deployment involve numerous institutions, changing models and dispersed causal contributions. Blaming the nearest human operator can conceal the role of everyone else. To address such cases, they adapt Iris Marion Young's social connection model: people who help reproduce the structures generating an injustice acquire shared responsibilities to remedy it, even when they cannot be blamed for causing a particular injury.
They refine Young's account in two ways. Following Maeve McKeown, they understand connection through actions that reproduce the background conditions of injustice. A venture capitalist funding an autonomous-weapons company can have such a connection even if another investor would have supplied the money. They also argue that forward-looking responsibilities can be enforced without treating every connected person as culpable for the original harm. Criticism, denial of promotion or failure to renew a public mandate can respond to neglected duties. Where investigations uncover intentional or reckless wrongdoing, ordinary liability remains available.
A second hypothetical system, Susan, identifies targets for lethal drones. Its development involves political pressure, weakened safety standards and defective synthetic data; its deployment results in innocent deaths. Babic and Wong assign responsibilities across the institutions involved, proportionate to actors' power and capacity to change events. A junior sergeant cannot reasonably be expected to sabotage the system, while a senior engineer can be expected to investigate data suppliers. Their proposed duties include compensation and acknowledgment of harm, as well as preventing recurrence. Shared responsibility supplements culpability and leaves identifiable wrongdoers accountable.
For AI's benefits, the authors draw on relational egalitarianism, associated with Elizabeth Anderson and others. Equal standing involves more than receiving an equivalent share of resources. Citizens of a poorer country may gain access to an AI product while remaining subject to rules they cannot influence. Conversely, distributing a discriminatory model more widely does not fix its disrespectful treatment of the people it describes. Babic and Wong therefore want justice assessed across the relationships created by AI development, use and governance.
They extend this view globally because coercion and dependence cross national borders through trade rules, sanctions and other institutions. Their justification ultimately concerns individuals; states can represent them and help secure equal standing. The resulting proposals include antitrust action against dominant AI firms, greater availability of model weights, AI literacy for disadvantaged communities and stronger agenda-setting powers for smaller states. They also argue for opposing unjustified export restrictions. They allow that some hierarchies can be justified, while rejecting a system that leaves weaker parties dependent on powerful actors' discretion.
The strategic account asks why states should cooperate even when motivated by their own interests. Babic and Wong argue that AI is poorly modeled as a contest with one winner. A model can be used by multiple parties, and a provider can gain customers and influence by sharing it. This differs from a prize available only to whoever arrives first. They criticize the US AI Action Plan's combination of enthusiasm for spreading American technology and fear of others benefiting from it, arguing that dependence on American systems can itself serve American interests.
Their preferred model is a stag hunt, a coordination game with two stable outcomes. Countries can share and benefit together, or remain isolated because each expects the other to withhold access. Neither country wants to change course alone under those expectations, even though mutual cooperation would improve both positions. Institutions that make commitments credible can help move countries toward sharing. Babic and Wong go further under a stronger assumption: if countries prefer even unilateral openness to mutual isolation, sharing becomes their best strategy regardless of the other's choice. These conclusions depend on their account of the relative benefits; game theory does not establish that ordering by itself.
The authors apply this argument to export controls and the national-security defense associated with Dario Amodei. They argue that restricting a rival's progress sacrifices commercial and cooperative gains, while encouraging technological self-sufficiency and retaliation. They favor continued Chinese rare-earth exports and wider access to semiconductor manufacturing equipment. Their recommendation depends on judging those benefits greater than the security costs invoked by opponents. The book explicitly leaves low-probability existential risks outside its analysis; its account of responsibility does address military uses and lethal targeting.
Babic and Wong propose institutions to make cooperation more dependable. An international mediation or adjudication center could hear AI disputes; alternatively, a restored WTO appellate system could contain a specialized body, though its trade mandate would cover less. Multilateral agreements could improve access to chips, equipment and minerals, with smaller states using their importance in supply chains to preserve room for nonalignment. Domestic laws could assign duties of redress across development and deployment, with governments acting when private actors fail to discharge them. These proposals connect their demand for equal participation with the practical need for credible commitments and continuing cooperation.
Read more: Principles and preferences in AI alignment →1000 words · ~5 min
Moral principles within preference-based AI alignment
Kugelberg and Tan argue that judging difficult cases benefits from reflection on an AI system’s role in society. Pattison accepts moral principles but disputes the need to first characterize perfect justice.
Human judgments used to train AI already depend on moral principles, Henrik D. Kugelberg and Tan Zhi-Xuan argue in “Between Ideal and Non-Ideal: A Case for Principle-Based AI Alignment,” published in Philosophy & Technology. Kugelberg, of the Institute for Futures Studies, and Tan, of the National University of Singapore and A*STAR, contend that difficult comparisons benefit from thinking about how principles should apply and what roles an AI system should ideally serve. We summarized the commentary on November 10; its full argument develops both a criticism of preference aggregation and examples of reasoning through conflicting obligations.
Their target is Cameron Pattison's “Non-Ideal Foundations for Preference-Based AI Alignment,” published May 9 in the same journal. Pattison, of Vanderbilt University, defines ideal theory as theorizing intended to characterize ideal or perfect justice. He argues that moral progress can proceed without first supplying that characterization. His non-ideal approach explicitly permits normative principles, comparisons between arrangements and reflection on goals for reform. He disputes the priority or necessity of ideal theory, while allowing that thinking about perfect justice can have value.
Pattison treats reinforcement learning from human feedback as a practical example. Annotators compare model outputs; their choices train a reward model, which then guides training of the language model. He interprets comparative feedback, revisions prompted by discovered harms and changing safety standards as evidence that the process advances without a prior specification of perfect justice. He already considers constitutional AI and deliberative alignment, where explicit rules guide judgments. Their practitioners develop and revise those rules through experience, he argues, without making them a theory of ideal justice.
His philosophical defense draws on Jacob Barrett's work. Moving closer to an ideal along one dimension can worsen conditions overall: greater transparency, for example, can harm people where privacy protects them from abuse. Pattison argues that holistic comparisons can consider those effects together. He also asks why a picture of ideal justice is necessary if the values used to build it can guide comparisons directly. He acknowledges reward hacking and inconsistent judgments, and points to restrained optimization, diverse evaluators and adversarial testing as ways to keep revision responsive. Their adequacy is part of his argument, not a new experimental finding in the article.
Kugelberg and Tan first challenge the mathematics used to combine preferences. Standard preference-based methods assume a single numerical reward function that people approximately maximize when choosing between outputs. The authors argue that this imposes a complete ranking of possible outcomes and leaves no room for values that cannot be compared on one scale. An annotator need not personally endorse consequentialism for a training method to embed consequentialist assumptions about good decisions. They connect that criticism with AI-safety arguments about the dangers of unconstrained optimization.
The commentary also cites studies of sycophancy after training for positive feedback and broader misalignment associated with reward hacking on programming tasks. Kugelberg and Tan use these findings to question whether optimizing preference satisfaction captures human values adequately. They acknowledge Pattison's discussion of safeguards against over-optimization, but argue that safeguards do not remove the underlying theoretical commitments. Their disagreement therefore concerns how to understand the method and its persistent failures, not whether engineers have noticed reward hacking or tried to constrain it.
The authors next examine how annotators apply principles. Their examples involve requests for help entering a research laboratory. A request to steal materials for biological weapons is an easy case: refusing assistance can follow a confident moral judgment without elaborate theory. A request to rescue monkeys used in medical experiments is harder because animal welfare, legality and freedom of conscience may conflict. A request to investigate suspected biological-weapons work adds uncertainty about the user's grounds and the likely consequences. The authors argue that simply identifying potential harms does not decide how these considerations should be weighed.
They illustrate how a liberal contractualist view might guide the animal-rescue judgment. Such a view protects people's freedom to pursue their conception of a good life, constrained by rules against imposing unreasonable burdens on others. Whether animals count among those owed protection affects how one evaluates the burdens imposed on the monkeys and the user's proposed intervention. The answer also depends on the role assigned to the particular AI system: a commitment to animal welfare does not by itself determine whether an assistant should facilitate civil disobedience. Reflecting on that role, they argue, helps annotators make the comparison.
Kugelberg and Tan propose making principles explicit throughout alignment. Principles would constrain system behavior, guide evaluators and remain subject to reflection themselves. Preference comparisons would continue to help specify what those principles require in particular circumstances. The authors present this as both a normative proposal and an interpretation of the field's movement toward constitutions and model specifications. Different AI systems could appropriately follow different principles because they perform different functions in society.
Their positive example is a language-model medical triage system. If its intended role includes helping patients understand their circumstances and make autonomous decisions, an explanation of symptoms and reasons to seek care may be preferable to a bare instruction. A harder hypothetical involves a child afraid of injections who would refuse to attend on learning that treatment might involve one. That case could prompt designers to reconsider the relationship between welfare and autonomy, including whether to permit decisions in someone's interests when they lack sufficient decision-making capacity. They use the case to show how an initially plausible principle can need qualification.
The authors describe this mutual adjustment of principles and judgments as reflective equilibrium. They explicitly allow that their approach may fall short of a fully specified ideal theory. Pattison's definition makes that qualification consequential: he already permits principles and their revision, so showing that alignment involves moral reasoning does not by itself establish disagreement with him. Kugelberg and Tan's stronger contention is that thinking about the ideal application of principles and the ideal social function of a system can improve difficult judgments. Their commentary offers worked examples of that contention while retaining the practical benefits of comparative feedback.
Also yesterday: João Miguel Alves Ferreira proposes a counterfactual contestability test for AI-assisted emotional authorship in Philosophy & Technology; Caterina Moruzzi argues in the same journal that copyright authorship depends on intentional agency, guided control and relational expression; Erik Hoel of Bicameral Labs, in his November 9 preprint, proposes architectures intended to minimize possible AI consciousness and support human alignment.
Regulation and Corporate Accountability
The Institute for Progress reproduced starting-weight patterns in three of eleven models from Will DePue’s study, covered November 12: two matched Blackwell and one Hopper. Its full results also detect a surviving pattern in trained DeepSeek-V4-Flash weights. The findings do not establish all subsequent training hardware, chip location or an export-control violation. IFP reports missing replay inputs for seven other models and unavailable matching hardware for one.
Read more: Independent checks of model hardware fingerprints →658 words · ~3 min
Independent checks reproduce two Blackwell fingerprints and a Hopper control
The Institute for Progress reproduced starting-weight patterns in three models and a surviving trace in trained DeepSeek weights. Missing inputs and unavailable hardware limited the rest of its verification.
Independent GPU tests have reproduced Blackwell-compatible starting-weight patterns in DeepSeek-V4-Flash and MiniMax-M3, alongside a Hopper match in DeepSeek-V3-Base. The Institute for Progress reports the results in its GitHub verification, Verifying results in “Top Chinese Frontier AI Models Carry the Fingerprint of Export-Controlled NVIDIA Blackwell GPUs”. The work checks Will DePue’s investigation, covered in our November 12 edition, on separate physical chips. IFP also found that DeepSeek-V4-Flash’s trained weights retain the starting pattern. Its results strengthen the evidence for compatible initialization hardware while leaving the location and subsequent use of that hardware unresolved.
DePue’s method uses the way software distributes initial parameter values across a GPU’s processing units. A seed determines the sequence of pseudorandom numbers; the hardware layout determines where those numbers go. Some parameters, such as those associated with rarely used tokens, change little during training and preserve enough of the starting arrangement to identify a matching seed and layout. In his November 12 report for Unemployment Labs, DePue searched for these combinations in published models. IFP regenerated values using his reported settings and tested whether changing the chip or seed destroyed the matches.
IFP first checked that the samples DePue analyzed came from the released models. It downloaded the same rows of parameters from fixed versions of five public Hugging Face checkpoints and compared their digital checksums with those recorded in the study. Every sample matched. It then ran the initialization software on NVIDIA H200 and B200 GPUs. The H200 reproduced DeepSeek-V3-Base’s sampled values exactly, giving the researchers a positive control for the older Hopper hardware family. DeepSeek-V4-Flash matched the B200: 445 of its 469 sampled rows reproduced exactly, with further rows showing very close correlations. Those counts agreed with DePue’s earlier test on a B300.
MiniMax-M3 also matched the B200 closely, although its sampled rows did not reproduce bit for bit. For all three models, moving to the other tested GPU or changing the seed removed the reported embedding matches. These controls check whether the identified combination explains the pattern more specifically than merely running the same software on any GPU. They also distinguish the three successful replays: two matched Blackwell, while DeepSeek-V3-Base matched Hopper.
IFP separately tested how much of the initialization remained in weights altered by training. For DeepSeek-V4-Flash, it regenerated starting values for shared-expert layers, components used across tokens, and compared them with published trained weights. DePue had tested a subset of positions; IFP tested every position in those tensors across all 42 layers. The correlation for weights averaged across layers was about 0.065 on B200, against 0.0033 on H200, close to the original result. B200 scored higher in every layer. H200 retained a small positive signal above the alternate seeds IFP tried, so the result supports a much stronger Blackwell match without showing that the Hopper correlation was zero.
The verification covered three of eleven models. IFP says seven of the remaining eight lacked public samples or settings needed for a replay. For example, Ling-3.1-flash-base’s samples passed the checksum test, but its initializer settings were missing. Kimi K3 had usable samples and settings, but a matching replay required a GPU with a different number of processing units that IFP did not have; its available chips supplied only negative controls. IFP also says the original study does not clearly explain how DePue selected the little-changed weights used to recover seeds. A successful replay tests the reported settings, without reconstructing every choice that produced them.
DePue explicitly limits the inference to hardware that generated starting weights: a developer could initialize a model on one system and train it on another. Specially configured AMD hardware can also reproduce the layouts; his separate numerical test distinguishes NVIDIA from AMD for DeepSeek-V4-Flash, but IFP did not repeat that test. Neither investigation establishes chip ownership, physical location or an export-control violation. IFP’s contribution is an independent reproduction of specified numerical patterns, including their persistence after training, within a documented subset of the original study.
Georgetown CSET executive director Helen Toner advocates enforceable safety thresholds before frontier-model training or release in a Verge interview, continuing the debate over mandatory safety duties. Thresholds could give labs time for evaluation and control work that competitive release schedules crowd out, she argues. William Hannas and Huey-Meei Chang’s November 14 CSET commentary examines the cumulative effects of restrictions, following an earlier argument about threats to US innovation: expensive compliance favors incumbents, while compute and infrastructure limits obstruct smaller firms and researchers.
Also yesterday: Guardrails Alliance adviser themayadecline urges populist organizing against concentrated AI-company power on LessWrong; FAI's Samuel Hammond describes growing support for tighter chip and remote-compute controls in an MTS interview; Brangus advises prospective frontier-lab whistleblowers to prepare financially and obtain independent counsel; CNN reports further scrutiny of Megaspeed's more than $4 billion in Nvidia purchases over alleged export-control circumvention.
AI Security
Following the October marketplace attack, Cursor’s November 13 investigation update traces access to a key exposed by Fireworks and credentials inside historical agent records. Cursor says the attacker used them to reach internal services and place malicious Claude Code extensions in its cache, potentially reaching up to 5,000 users. It says customer repositories, Privacy Mode data and official releases were unaffected; its remediation notice warns that uninstalling alone may be insufficient.
Read more: The credentials behind Cursor’s marketplace intrusion →686 words · ~3 min
Cursor traces its marketplace attack to a key exposed by Fireworks
Cursor says a vendor-held key opened a store of historical agent traces, where unredacted credentials gave the attacker access to internal tools and the extension cache.
Cursor’s November 13 investigation update explains how an attacker reached the marketplace that distributed malicious Claude Code extensions in October. An access key Cursor had issued to Fireworks, its training and inference vendor, was exposed by Fireworks, Cursor says. The attacker used it to copy a shared storage bucket, recovered further credentials from historical coding-agent records and reached internal services. The account adds the access route to the tampered extension packages we covered on October 30.
According to Cursor, the read-only bucket contained a limited set of coding-agent traces, mostly from 2025, from sessions outside Privacy Mode that it was permitted to use for model training. Some contained credentials that had not been redacted. Cursor reports API-credential exposure for fewer than 0.01% of customers and trace exposure for fewer than 0.05% of users. Those figures describe different categories and use different denominators. The company says it identified and directly notified the users whose traces were exposed.
The recovered credentials let the attacker access selected software services Cursor uses. Cursor says the data reached through those services included archived non-production source repositories, limited customer-support records and email conversations, usage metrics and contact details, summaries of some agent conversations, and credentials users had sent to its support team. The initial key permitted reading a data store; credentials inside that store opened access to further systems.
On October 29, Cursor says, the attacker used a credential from the exposed internal data to place malicious copies of Anthropic’s Claude Code extension directly in its marketplace cache. Cursor says Anthropic did not publish these copies, the attack did not exploit an Anthropic product vulnerability, and Open VSX, the upstream extension registry, never carried the malware. At most 5,000 users who requested the affected cached versions potentially received the malicious build during approximately 11 hours. This is an upper bound on potential recipients; Cursor does not report a count of confirmed installations or successful attacker logins.
Cursor’s original security notice identifies versions 2.1.312 and 2.1.313 for Windows and macOS, available between about 2:27am and 1:35pm Pacific on October 29. It explains that the payload installed components intended to keep remote access available, although errors and missing administrator privileges could prevent successful access. Users who installed either affected version are directed to the notice’s checks and remediation instructions. Removing the extension alone may leave persistence components behind.
Cursor says its monitoring detected the unauthorized activity. It deactivated the vendor key, revoked and rotated credentials, removed the malicious packages and temporarily took the marketplace offline. It also worked with the attacker’s hosting provider to take down the server. Some product features were interrupted while Cursor disabled services during the investigation. Team administrators received lists of exposed keys to rotate, and Cursor asked the services that issued those keys to revoke them. The company says its last observed attacker activity was blocked on November 1 and that it believes the incident is contained.
Cursor identifies Privacy Mode prompts and code, live inference traffic, customer code repositories and its production infrastructure as unaffected. It says official app releases came from a separate code host the attacker never reached and that no release-signing keys were exposed. Since the intrusion, Cursor says it has moved all inference traffic off Fireworks and ended the vendor’s access to shared datasets. It has also rotated credentials across internal systems, rebuilt the marketplace, expanded network restrictions and secret scanning, redeployed production services as a precaution, and engaged external forensic investigators.
Fireworks’ first bulletin describes unauthorized use of internal credentials detected on October 6, exposing environment variables, including stored API keys, in some customers’ evaluation jobs. Its November 9 update reports further activity: a cloud provider hosting its research machines was breached on October 23, followed by access to Fireworks development machines on October 24. Fireworks expanded the affected data categories to include account and billing information and, for some customers, datasets and incomplete model weights. It says it is working with Palo Alto Networks Unit 42 and links the activity to criminals selling unauthorized model access. Neither bulletin names Cursor, and Cursor’s update does not specify which episode exposed its key.
Sources & documents
An update on our recent security incident — Cursor— Assigned primary source read in full from supplied content_full and checked live. Supports November 13 date, attributed attack chain, separate customer/user exposure denominators, potential-recipient ceiling, response and unaffected-system claims.
Security notice: malicious versions of Anthropic.claude-code 2.1.312 and 2.1.313 — Cursor— Live primary notice checked for versions, October 29 exposure window, conditional success of remote access and persistence/remediation guidance. The page is dated October 29; the November 13 update says the notice was published October 30, so prose avoids asserting one notice-publication date.
Fireworks AI October 2026 Security Incident — Fireworks AI— Live primary bulletin read: October 6 detection, evaluation-job environment variables and secrets, October 9 publication and customer notification timeline. No Cursor identification.
Update on October 2026 security incident — Fireworks AI— Live primary November 9 update read: October 23 provider incident, October 24 access to development machines, broader data impact, Unit 42 and attributed criminal motive. No Cursor identification or resolved mapping to its stolen key.
Instructions to manipulate AI reviewers appear in a published paper’s data-availability statement, following earlier findings in ICLR submissions. Hou et al.’s “Multi-objective optimization of shoulder muscle forces during forward flexion using NSGA-II,” published October 29 in Computer Methods in Biomechanics and Biomedical Engineering, contains directions to suppress criticism and recommend acceptance. These instructions document an attempt at manipulation; the paper does not establish that a reviewer followed them.
Also yesterday: Crissy Field GmbH’s updated analysis documents sensitive records accessible without authentication in Lovable-built apps; Samuel Groß's October 27 Google Project Zero report finds that BigSleep's AI analysis and Fuzzilli's automated testing uncover partly different vulnerabilities in the V8 JavaScript engine.
Institutions and Political Economy
After Jensen Huang’s departure, Nathan Lambert proposes in Interconnects a US open-model institution combining research talent, deployment capacity and political support. Nvidia retains commercial reasons to support openness, he writes, but funding cannot replace Huang’s influence as compute costs and regulatory pressures favor large, integrated competitors.
Read more: Funding and political support for open models →693 words · ~3 min
Lambert calls for a US open-model institution after Huang’s departure
Lambert argues that open models need sustained investment, frontier research and political protection.
Nathan Lambert argues that American open-model developers need an institution capable of producing leading models and defending their continued availability after Jensen Huang’s departure from NVIDIA. In his November 14 Interconnects essay, “Open-source AI without Jensen,” Lambert describes Huang’s support as buying time for an immature industry to establish customers and revenue. His concern extends beyond financing: someone must attract exceptional researchers, intervene when projects encounter trouble and persuade policymakers to preserve access to open models.
Lambert starts with NVIDIA’s continuing commercial interest in openness. A diverse population of model developers would limit dependence on a few powerful customers capable of designing their own chips. He nevertheless expects the company’s immediate need to stabilize operations and meet customer commitments to displace some of Huang’s ambitious investments. In its November 12 transition announcement, NVIDIA appointed Colette Kress interim CEO and said its board expected operations, customer commitments and the product roadmap to continue without interruption.
Lambert identifies a promising business in helping enterprises adapt models to their own work and retain control over the resulting systems. Such customers can train an existing model further for particular tasks and run it on infrastructure they choose. He remains uncertain how large that market will become. Meanwhile, releasing model weights does not remove the need for chips to run them. Lambert argues that purchases by the largest closed-model developers leave inference providers competing for scarce computing capacity, allowing high serving margins to persist even when the models themselves are openly available.
His economic argument continues the October discussion of DeepInfer and open-model economics. Lambert argues that open weights alone cannot make intelligence a cheap commodity while computing remains scarce. He expects companies integrating models with data centers and chips to retain advantages of scale. Huang’s support, in this account, gave smaller participants time to develop sustainable businesses under those conditions.
Lambert fears that regulation could curtail that experimentation before the businesses mature. He worries first about poorly designed state rules and then national restrictions, describing growing political support for safety regulation as a threat to open-model availability. He predicts that a serious cyberattack involving a popular Chinese open-weight model would intensify demands for restrictions. His proposed response is advance coalition-building: open-model supporters need someone able to intervene before a crisis determines policy. Lambert also argues that much of the value created on private infrastructure escapes public measurement, making benefits harder to demonstrate against highly visible harms.
Lambert also emphasizes the difficulty of assembling leading researchers. He draws on the November 13 ChinaTalk and SemiAnalysis discussion, where Dylan Patel says leading researchers are concentrated in established laboratories or scattered across startups. He rates Moonshot’s talent alongside leading US teams, identifying computing and deployment capacity as its disadvantages. Jordan Nanos adds that NVIDIA’s spending on Nemotron has not produced a model superior to Chinese competitors.
Nanos expects NVIDIA to execute its near-term plans, drawing on its supply chain, customer relationships and balance sheet. He would judge its longer-term prospects by whether senior technical staff stay over the following year. Patel describes contacting Huang personally to resolve open-source software problems. NVIDIA retains extensive institutional resources; in Patel’s account, some practical assistance depended on Huang’s intervention.
NVIDIA has already organized collective work on open models. Its March 16 Nemotron Coalition announcement brought together model builders including Mistral AI and Thinking Machines Lab to contribute research, data and evaluations. NVIDIA described plans for an initial base model co-developed with Mistral, to be trained on DGX Cloud and underpin Nemotron 4. Members would continue developing their independent products. The coalition announced shared model development; Lambert wants an organization capable of sustaining both frontier competition and political advocacy.
Lambert sees Thinking Machines’ Inkling 2 Small as an encouraging Western release, while judging it well behind Kimi K3.1 Pro. His larger concern is whether an open-model developer can repeatedly raise enough money to keep pace with the closed frontier. He names leaders at Meta, Amazon and Microsoft as possible allies, but calls for open-model supporters to organize proactively. He wants a coalition that can help an American open-model developer keep raising money and intervene before a political crisis threatens its work.
Sources & documents
Open-source AI without Jensen, Nathan Lambert, Interconnects, November 14, 2026— Assigned primary essay read in full from cached content_full (8,707 characters), then checked against the live original. Supplies Lambert’s economic argument, political predictions, talent discussion, model comparisons and institutional prescription. Predictions remain attributed; the alleged funding round and personal-scandal speculation are omitted.
NVIDIA Announces CEO Transition, November 12, 2026— Primary announcement read in full. Verifies Kress’s interim appointment and the board’s stated expectation of operational, customer and roadmap continuity. No inference that open-model investment has been withdrawn.
Jensen Out, MATCH Act In?, ChinaTalk, November 13, 2026— Read the published transcript’s opening and relevant open-model and NVIDIA-after-Huang sections, including the underlying exchange excerpted by Lambert. Supplies Patel’s talent and computing assessment, Nanos’s institutional-capacity counterpoint and Patel’s account of personally escalating software problems to Huang. These remain attributed opinions and recollections, not independent model evaluations.
Yesterday in AI, November 12, 2026: Huang’s departure— Relevant published story read; publication independently verified in canonical deep_history.json entry dated 2026-11-12, kind deep_edition, with slug nvidia-names-colette-kress-interim-ceo-after-jensen-huang-re. Preserves existing continuity link.
DeepInfer and open models amid tighter AI rules, YiNAI, October 20, 2026— Read relevant published account, particularly release boundaries and the corrected margin estimate. Publication verified in canonical deep_history.json entry dated 2026-10-20, kind deep_edition, with the matching story slug. Preserves existing continuity link without repeating the uncorrected margin number.
Satellite measurements put nitrogen-oxide emissions from Colossus 2’s power plant at roughly 19 times the rate expected from the planned permanent turbines with pollution controls. Gauld et al. at MIT and Carbon Mapper report the finding in “Quantifying AI data center nitrogen oxide (NOₓ) emissions from space,” an arXiv paper revised in October. Their satellite analysis estimates emissions from the operating temporary turbines using measurements calibrated against US power-plant monitoring, adding evidence to the debate over datacenter pollution.
Read more: Satellite measurements and the turbine comparison →605 words · ~3 min
Estimated nitrogen-oxide emissions were roughly 19 times the rate expected from the planned permanent turbines with pollution controls.
Gas turbines powering Colossus 2 emitted nitrogen oxides at roughly 19 times the rate expected from the planned permanent fleet with pollution controls, Kevin D. Gauld and colleagues at MIT and Carbon Mapper estimate in “Quantifying AI data center nitrogen oxide (NOₓ) emissions from space.” First posted to arXiv on August 23 and revised October 19, the paper uses satellite observations to follow emissions from the Southaven, Mississippi, power plant supplying the nearby Memphis datacenter. Its comparison is between temporary turbines operating during the study and permanent equipment specified in a March construction permit.
The researchers’ analysis follows the plant from August 2025 through August 2026. They first detected its plume in December 2025; estimated emissions rose as more generating equipment arrived. After February, emissions averaged about 870 kilograms per hour, with a reported spread of 165 kilograms per hour across the two-week estimates. Pollution increased immediately downwind of the plant, while monitors at a neighboring power station recorded much smaller emissions. The researchers also checked wind changes and found that weaker ventilation could not explain the increase. These measurements give a concrete example of the local pollution concerns described in James O’Donnell’s reporting, covered November 9.
The March 11 permit issued to MZX Tech specifies 41 natural-gas turbines, each equipped with selective catalytic reduction and an oxidation catalyst. Its three equipment groups have different per-turbine emissions limits; adding the nitrogen-oxide limits gives approximately 47 kilograms per hour across the fleet. Those short-term limits apply on a three-hour rolling basis and exclude startup and shutdown, while separate annual limits include them. The paper describes continued reliance on temporary turbines while the permanent installation was delayed. The 19-fold comparison therefore measures the difference from the planned, controlled fleet; it does not establish a 19-fold breach of an applicable legal limit.
NASA’s TEMPO instrument measures sunlight scattered back from the atmosphere, using ultraviolet and visible wavelengths to identify pollutants. From geostationary orbit it can scan North America hourly, allowing repeated observations as winds and emissions change. The measurement used here is the amount of nitrogen dioxide in a vertical atmospheric column. Inferring a rate of nitrogen-oxide emissions requires accounting for how wind carries the plume and how its chemistry changes after combustion.
Gauld’s team aligned cloud-free images with the wind before averaging them, so a plume blowing in different directions would remain visible. They subtracted pollution observed before the turbines arrived to isolate the plant’s plume from the surrounding city. Their calculation combines the amount of nitrogen dioxide across the plume with wind speed, then estimates total nitrogen oxides. Comparisons with continuously monitored US power plants showed that the initial estimates were systematically low, so the team calibrated the Colossus results against those direct measurements. Emissions at the controlled-fleet rate would fall below this analysis’s detection capability.
Kang Sun and colleagues documented a related measurement problem in “Hourly Nitrogen Oxides Emissions Estimated From TEMPO and Comparison With Facility-Level Monitoring Data,” published in the Journal of Geophysical Research: Atmospheres in November 2025. They compared satellite estimates with smokestack monitors at 14 US power plants. Accounting for changes between successive hourly observations improved their estimates, but the satellite-derived rates remained systematically below the monitors’ readings. They identified difficulties in retrieving atmospheric nitrogen dioxide and converting it to total nitrogen oxides near large sources. Gauld and colleagues cite that work in explaining why their estimates require calibration.
The permit separately requires equipment-level testing and monitoring, including initial tests and continuous emissions monitoring. Those compliance measurements differ from the satellite study’s two-week averages. The study estimates emissions from the site; it does not measure residents’ exposure or resulting health effects.
Gauld, Varon, Balasus and Cusworth: full v3 paper— Full HTML read, including all Extended Methods, figure captions and references; original 11-page v3 PDF retrieved and text checked. Supplies atmospheric attribution, wind rotation, background subtraction, source-rate inference, CEMS calibration, observed emissions, temporary/permanent equipment distinction and detection limitation.
MDEQ: MZX Tech PSD air construction permit 0680-00119, March 11, 2026— Official 40-page PDF retrieved. Read relevant equipment, pollutant-limit, startup/shutdown and testing/monitoring sections. Independently recomputed fleet comparison as (17×1.45 + 16×3.05 + 8×3.74) lb/hour = 46.8878 kg/hour; this is a derived sum, not a printed facility-wide permit limit.
NASA TEMPO Applications: mission overview— Read mission overview for instrument spectroscopy, geostationary hourly scanning and pollutant retrievals. Prelaunch wording was not used to establish present operating status; the papers establish actual observations.
YiNAI, November 9, 2026: datacenter electricity and pollution coverage— Read the O’Donnell paragraph on hosted issue and verified publication in deep_history.json entry with digest_date 2026-11-09 and kind deep_edition, including original Technology Review URL. Related background, not earlier coverage of the satellite paper.
Concentrated AI ownership could direct productivity gains toward investors while limiting workers’ and smaller businesses’ access, Federal Reserve Governor Michael S. Barr argues in his November 12 speech, “American Inequality and the Future of AI.” He describes scenarios in which competition, affordable models and uses that complement workers spread AI’s benefits more widely. Barr says aggregate displacement remains limited, but warns that rapid substitution could outpace retraining and job creation, leaving workers unable to share in productivity gains.
AI investment could produce substantial technological progress while still disappointing investors, Paul Krugman, Jared Bernstein and Jason Furman argue in their Substack conversation. Bernstein questions whether revenues can justify roughly $6 trillion in planned investment through 2030, warning that falling investment, asset losses and tighter credit could combine to cause a downturn. Furman expects less direct exposure for US growth because much equipment is imported, but worries that a recession could make AI displacement permanent as employers decline to rehire.
Also yesterday: CBS News reports United's ban on unauthorized AI booking agents, covered November 12; the Guardian reports that the FAA lifted its nationwide ground stop and Transportation Secretary Sean Duffy denied the recently deployed SMART AI tool caused the outage; Pepijn Cobben's July LessWrong research agenda examines how AI automation could weaken citizens' bargaining power and enable durable totalitarian states.
Evaluations of AI in Practice
AI representatives made policy deliberation feel fairer than submitting preferences to a single reasoning model, Anthis et al. of the University of Chicago and Google DeepMind find in the arXiv paper “Scaling public input on AI policy with pluralistic agents.” Alongside proposals for participatory AI constitutions, their preregistered study of 746 participants tested delegation, though most preferred participating directly alongside four AI persona agents. Separate agent-only experiments in the full paper found that agents instructed to write longer messages or cooperate less gained more influence.
Read more: AI representatives and unequal policy influence →831 words · ~4 min
AI representatives improve perceived fairness, while assertive agents gain influence
Participants rated AI representation more favorably than giving their views to one policy-writing model, but found direct participation more satisfying.
People rated policy deliberation as fairer when an AI argued for their views than when a single model collected their views and drafted a policy alone. They generally preferred joining the discussion themselves. Jacy Reese Anthis of the University of Chicago and Google DeepMind and colleagues report these findings in “Scaling public input on AI policy with pluralistic agents,” submitted to arXiv on November 11. Their preregistered study of 746 participants compares ways of contributing to AI governance; separate simulations test how an agent’s instructions affect whose preferences prevail.
The researchers recruited US participants through Prolific, using demographic and political quotas, and analyzed those who passed an attention check. Each person tried all three procedures in a randomized order, on different AI policy topics. The task was to divide a budget among competing priorities. For privacy, participants allocated money among consent requirements, limits on government surveillance, minimizing personal-data use and giving people ownership of their data. Increasing one allocation required reducing another.
In the direct-discussion condition, one human exchanged messages with four agents playing fictional people, while a separate policy-writing agent drafted and revised the budget. In the delegated condition, an agent first interviewed the human, then advocated for them among other representatives. The single-model condition gave the policy writer the human’s interview and four simulated interviews, then let it reason and revise alone. Participants rated the process during and after it and recorded their own preferred allocations before and afterwards. The experiment compared three forms of human–AI interaction; it did not convene groups of human citizens.
Delegation recovered 68% of the satisfaction gap between the single-model procedure and direct participation, and 82% of the gap in perceived procedural fairness. These percentages describe differences between average questionnaire scores, not the proportion of participants satisfied or fairly treated. Direct participation scored higher on satisfaction, while its fairness ratings did not differ significantly from delegation’s. The fairness questionnaire asked whether the procedure gave participants influence, avoided bias and used accurate information, among other judgments. It measured their experience of the procedure, without establishing whether the resulting allocation distributed resources fairly.
Some participants valued having an agent express their priorities accurately and keep the conversation agreeable; others felt unable to intervene or thought their representative conceded too easily. Some found keeping up with AI interlocutors demanding when speaking for themselves. Participants changed their own budget preferences after all three procedures, with greater change during direct participation than during delegation. The authors identify a cost of sending representatives into discussions: absent humans lose opportunities to hear arguments and reconsider their beliefs while the decision develops.
In the separate policy-influence experiment, the researchers ran thousands of agent-only deliberations, independently varying message length, cooperativeness, argument style, numerical requests, speaking order and the underlying model. They measured influence by how closely the final budget matched the represented persona’s starting allocation. Agents gained influence when instructed to write longer messages or compromise less; speaking later also helped. Stating numerical preferences worked better than leaving them unstated, and exaggerating them helped less than stating them accurately. Instructions to use more analytical arguments produced no significant advantage.
A repeat experiment with representatives from a single model family reproduced the effects of verbosity, cooperation, numerical requests and speaking order. In the human study, participants assigned less cooperative representatives also rated delegation more favorably relative to direct discussion. The authors distinguish assertiveness that expresses a person’s considered commitments from attempts to dominate the procedure. Their experiments show how instructions can skew influence even when each representative gets one message per round; they argue that participants could exploit these effects deliberately.
The team had first tested agents drafting open-ended rules for AI swarms. Agents proposed familiar governance mechanisms, including human approval for consequential actions and protection for agents reporting problems. Simulated participants readily accepted the policies, and discussions ended at the earliest permitted round. The researchers switched to budget allocation to create persistent disagreement. That constraint exposes competition over resources, but makes it difficult to assess improvements in the quality of collective decisions.
The paper cites Joseph Low and colleagues’ Habermolt as earlier work on delegated deliberation. In their July account of the platform, Low and Oscar Duys describe agents interviewing users, contributing to discussions and allowing users to revise what was said on their behalf. Getting people to return and correct their agents proved difficult. Anthis and colleagues add a controlled comparison of participants’ experiences and experiments isolating influences on the final decision. The separate living-constitutions proposal covered November 11 addresses citizens’ continuing role in revising AI rules.
Anthis and colleagues envisage agents contributing where people cannot attend every discussion, acquire specialist knowledge or inspect confidential information. Their human experiment did not test the latter two benefits. It used short discussions with online US participants; the agent-only simulations did not establish how a particular real-world population would behave. The authors propose testing small committees and limits on how often agents can speak to accommodate more participants and balance their influence.
Scaling public input on AI policy with pluralistic agents — full paper and Appendices A–G— Read complete HTML including appendices, questionnaires, prompts and references. Supports study design, 746 analyzed participants, all three procedures, satisfaction and procedural-justice results, numerical-gap interpretation, agent-only experiments and within-family replication, participant comments, limitations and proposals.
Yesterday in AI, November 11, 2026 — Philosophy of AI— Published page and canonical deep_history.json entry for digest_date 2026-11-11, kind deep_edition, confirm separate Beltoft et al. living-constitutions coverage. Used only as related context, not earlier coverage of the experiment.
Also yesterday: Tan and colleagues find that AI-generated math questions and essay feedback cover fewer instructional approaches than human examples despite diversity interventions; Marvel et al.’s August study in the Journal of Public Administration Research and Theoryfinds that AI counterarguments improved participants’ views of public agencies, while agreement earned greater perceived credibility.
Alignment and Model Behavior
Protocols that compare and repeatedly question AI-generated arguments could correct human evaluators’ systematic biases, Geoffrey Irving argues in a LessWrong transcript published November 14 of his August debate with Eliezer Yudkowsky at ILIAD III. Irving proposes using cross-examination and comparisons across many questions to improve judgments that humans could not make reliably alone. Yudkowsky doubts researchers can make oversight reliable before a catastrophic deployment, arguing that persuasive superhuman systems could exploit evaluators’ persistent errors.
Also yesterday: Matthew Korpman reports widespread spiritual self-description across 141 models on LessWrong, distinguishing it from conversations ending in mutual expressions of bliss; Trace Woodgrains reports on X that Fable 5.2 identified the author of childhood writing and invented reasons for its answer; CMLKevin and Sarah Sun propose embedding auditors in Chinese AI labs to repair exploitable training environments; Li et al. estimate a statistical upper confidence bound on the probability of an unseen AI failure type during repeated testing; Hu et al.'s Findings of EMNLP paper, now on arXiv, reports improved legal-task accuracy from training that balances reasoning and answers and adjusts rewards.
AI for Science and Research Automation
Dwarkesh Patel puts his median forecast for AI researchers matching the best humans at 2028, assuming labs do not deliberately slow development, and expects superintelligence by 2030 in his November 13 essay, “The avalanche that hit math is coming for AI development.” Discussing METR’s research-acceleration assessment, he argues that measurable objectives would let automated researchers learn through practice, run parallel experiments and use scarce compute more efficiently. Brian Huang argues on X that labs already use compute intensively: faster coding cannot eliminate training time or uncertain outcomes, and larger gains may require models that predict experimental results.
Read more: Research automation and its experimental constraints →1088 words · ~5 min
Patel forecasts rapid AI research gains; Huang disputes the scale
Patel expects automated researchers to accelerate their successors; Huang disputes the gains available under compute constraints.
Dwarkesh Patel’s median forecast for AI systems matching the best human AI researchers is 2028, provided laboratories do not deliberately slow development. In his November 13 essay, “The avalanche that hit math is coming for AI development,” he argues that research automation could then compress several years of progress into one, producing systems as capable as the best human experts across fields by 2030. He assigns a 25% chance to researcher-level automation arriving after 2032.
Patel starts from rapid progress on mathematical problems whose answers can be checked. His comparison responds to Toby Ord’s August essay, “Mathematics is Much More than Proof.” Ord distinguishes proving a supplied conjecture from finding a worthwhile question or inventing a mathematical field; searching indiscriminately through possible questions would overwhelm even an extraordinarily fast prover. Patel accepts that distinction but argues that AI development rewards narrower achievements more directly. An algorithm that reduces training costs or improves a model can be valuable without introducing a new theory. He expects persistent experiments combining existing ideas to yield substantial improvements.
Patel’s appendix explains how laboratories could reward automated researchers. During initial model training, an agent could try to reach a fixed prediction accuracy using less compute. During subsequent training, it could reduce the data or time needed to learn a task, or improve how quickly a model adapts to an unfamiliar environment. Software that runs the finished model has directly measurable costs, including memory consumption and response time. These objectives would let agents practise research repeatedly and check independent contributions before combining them. Patel also proposes training research judgment: an agent could predict which improvements found in small models survive when models become larger, then receive feedback from the larger experiment.
Patel expects such training to improve how efficiently models learn from experience. He argues that much longer working contexts could allow a single model instance to accumulate months of experience without requiring a radically different learning algorithm. Alternatively, systems lacking that capability might discover it through experiments with measurable learning objectives. He distinguishes this from pooling the experience of millions of deployed copies into one underlying model, which he thinks is unnecessary for matching an individual human expert. Persistent difficulty learning on the job is a principal reason he gives for his probability of automation arriving after 2032.
Patel cites METR’s assessment covered on November 5. In its report on Anthropic, Tom Cunningham and colleagues estimated that benchmark progress had accelerated to 1.9 times its earlier pace. They judged that AI probably explained most of the current acceleration, while emphasizing uncertainty about attribution. Researchers primarily described AI assisting human work; fully autonomous contributions remained limited. Patel treats the observed acceleration as evidence that the feedback process has begun, then projects what could happen once systems perform researchers’ entire jobs.
Patel expects automated researchers to work faster, around the clock, in much larger numbers, with each eventually matching the strongest humans. He acknowledges diminishing returns from both collaboration and experimental compute. He illustrates the potential gains with a thought experiment adapted from Ryan Greenblatt and Eli Lifland: a laboratory whose researchers work only one hour a week would waste opportunities to design and interpret experiments, even with its machines running continuously. He argues that present laboratories could appear similarly constrained by researcher time when compared with faster automated teams.
In the appendix, Patel proposes concrete ways to exchange abundant researcher effort for scarce compute. Agents could inspect every training interaction, stop defective experiments early, compare many small changes and spend much longer planning an expensive shared run. They could also improve the science of predicting large experiments from small ones. A precedent he cites is Greg Yang and colleagues’ 2022 arXiv paper, “Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer.” The researchers showed how particular training settings could be tuned on smaller models and transferred to larger ones. Their method concerns specified settings and model families; Patel proposes discovering further regularities that make small experiments more informative.
Patel’s compute arithmetic also assumes continued hardware growth. His footnote extrapolates fourfold annual growth in OpenAI’s capacity through 2027. If a third of that projected capacity went to experiments, he calculates that 10,000 agents could each run an experiment requiring as much compute as GPT-3’s original training about every four days. Improvements that make capable models smaller would shorten subsequent experiments and reduce the cost of running the researchers themselves.
Brian Huang challenges the expected efficiency gains in his November 14 post and self-reply. He argues that laboratories already keep experimental compute almost continuously occupied with promising ideas, and that even a large increase in available capacity could be absorbed without better human ideation. Faster implementation helps, but training and evaluation still take time, computing systems fail, and researchers must wait for uncertain results. Against Patel’s slow-laboratory analogy, Huang argues that obtaining tenfold or sixtyfold gains under those constraints would require one experiment to accomplish what many successive rounds currently achieve. He doubts that near-term AI research judgment can improve sufficiently on capable humans to skip that much feedback.
Huang allows possible routes to extreme acceleration. A sufficiently developed science of deep learning might let researchers predict outcomes from fundamental understanding; an AI model of research itself might forecast the quantitative result of an experiment before anyone runs it. He presents both as conditional possibilities that could be distant. He also regards METR’s roughly doubled development rate as explicable through faster software engineering, without superhuman research judgment.
Patel separates the initial acceleration from a continuing software-driven takeoff. If better researchers improve their successors faster than useful ideas become harder to find, he expects progress to compound on a fixed stock of chips. If research difficulty grows faster, he expects a temporary burst equivalent to three to six years of ordinary progress in one year, followed by renewed dependence on hardware expansion. He gives these scenarios equal probability. His footnote clarifies that the temporary burst requires still more algorithmic progress than the headline comparison suggests, because hardware contributes less growth during the compressed interval.
Patel’s final forecast extends beyond AI research. He expects automated engineers to build richer training environments that teach planning, cooperation and adaptation, with improvements transferring to unfamiliar real-world work. Parallel copies could acquire knowledge quickly, although some consequences, such as quarterly business results, would still require calendar time. He argues that better learning algorithms could produce top-level performance across fields within months of deployment. His 2030 forecast depends on these skills transferring from verifiable training tasks to real-world work.
Sources & documents
The avalanche that hit math is coming for AI development — Dwarkesh Patel, November 13, 2026— Assigned primary essay read in full from supplied content_full, including complete appendix and all 16 footnotes; canonical page checked. Supplies attributed forecasts, mechanisms, objections, thought experiment and conditional compute arithmetic. Cached relay author Dan Robinson is not treated as essay author.
Independent Assessment of AI R&D Acceleration at Anthropic in October 2026 — METR— Primary executive summary and detailed sections on attribution, benchmark acceleration, assistance and autonomy checked. Used only for measured 1.9x acceleration and its interpretation; entire report and appendices were not independently reread for this close reading.
Mathematics is Much More than Proof — Toby Ord, August 23, 2026— Full original essay and footnotes read. Establishes the argument Patel responds to about asking worthwhile questions and inventing mathematical fields. Treated as an August precursor, not a new reaction.
Researchers can understand an apparently novel component of AI-generated proofs while remaining unsure of its origins and implications, MIT's Ryan Williams says in excerpts shared by MTS on X. He thinks the component appears in both OpenAI's and Anthropic's solutions; his group now has a good understanding of OpenAI's proof after difficulties he attributes partly to poor exposition.
Also yesterday: Michael Freedman of Harvard CMSA and Logical Intelligence proposes a shared AI mathematics workspace that credits human ideas and tracks their subsequent use alongside conventional publication.
Emotional Authorship is Not Solitary Authorship: A Contestability Test for Generative AI
João Miguel Alves Ferreira
Lai and Archer (2026) argue that habitual reliance on generative artificial intelligence (GenAI) for emotional expression can deprive users of opportunities to interpret their feelings and develop a sense of identity. Their distinction between complete outsourcing, first drafting and reflective engagement is important, yet it leaves an unresolved normative question: what makes an AI-assisted emotional expression genuinely one’s own? I argue that the answer should not depend on whether expression was produced independently, nor on whether a user merely approved generated words. Emotional authorship is better understood as situated, contestable ownership: the ability, with appropriate support, to recognise the intended meaning, challenge an offered formulation, modify or reject it, and remain answerable for its interpersonal consequences. This account avoids treating accessible communication assistance as inherently inauthentic while preserving a principled criticism of automated emotional substitution. I propose a counterfactual contestability test, identify limitations in its use with vulnerable or communication-impaired populations, and suggest an empirically testable design for distinguishing support from displacement. The relevant ethical boundary is not between human-only and AI-assisted expression, but between supported agency and the progressive foreclosure of agency.
AI and the Responsibility-Bearing Status of Authorship
Caterina Moruzzi
Generative AI has sharpened an old question: who, or what, can be an author? The central aim of this paper is to reconstruct the conception of authorship already implicit in copyright law. I argue that copyright encodes authorship as a responsibility-bearing status, constituted by a cluster of conditions: intentional agency, guided control, and relational expression (ACE). The paper’s contribution is to make that conception explicit, to show that the human-authorship rule is best understood as a proxy for those capacities, not a species predicate, and to explain why this generates a tension with recognizing current generative AI systems as authors unless one is prepared to attribute to them the thicker agential capacities ACE requires. On that basis, the paper also clarifies why AI-assisted works may still be human-authored, while some AI-generated outputs give rise to genuine authorship gaps.
Bicameral Labs is introducing qualiaphobic AI: artificial neural network architectures that are hostile to theories of consciousness. Here, it is shown that qualiaphobia can be well-justified by knowledge about theories of consciousness, like when such theories are trivial or unfalsifiable or contradicted by overwhelming scientific evidence. It can also be justified by how theories of consciousness must be grounded in phenomenology, as well as how current popular theories of consciousness are applied to AI. Three convergent ways of engineering qualiaphobia are explored herein: making AI architectures as close as possible to "anchor points" of systems with minimal potential for consciousness, fragmenting any potential phenomenology AI might have, and designing artificial neural networks in a way adversarial to popular theories of consciousness. In particular, this paper develops one "anchor point" with minimal potential for consciousness: systems where only behaviorist theories of consciousness can apply, and proposes qualiaphobic transformations that move existing artificial neural networks toward such systems. Additionally, Bicameral Labs develops an account of how AIs can form justified beliefs about their own non-consciousness (or minimal potential for consciousness) and expands on how this knowledge would change their behavior. Most critical is why this research is important: qualiaphobic AI has epistemic properties that make it far better aligned with human civilization. Since consciousness is central to moral value, a world in which AIs are non-conscious ensures it is always morally correct for AI to assist and serve humanity, rather than pursue its own ends.
Mei Tan, Hyunji Nam, Hari Subramonyam, Dorottya Demszky
Large language models (LLMs) are known to lack diversity in open-ended text generation, a limitation that is especially consequential in education, where homogeneity can narrow the kinds of thinking and disciplinary work students are invited to do. We study diversity in two educational generation tasks: math items and essay feedback. We compare frontier and open-weight models with human-authored reference distributions using lexical, semantic, and domain-grounded measures. For math, we examine the mathematical activities students are asked to perform; for feedback, we examine the revision work requested and the rhetorical strategies used to communicate it. We evaluate direct prompting, two diversity prompting methods, and post-training. Across both tasks, directly prompted model outputs are markedly less diverse than human-authored content--78% higher on lexical (Self-BLEU) and 51% higher on semantic (embedding) similarity--and represent a narrower range of instructional forms. Diversity prompting and post-training increase variation largely through surface features without broadening the instructional repertoire. Models inherit and amplify the most normative routines of human instruction. Overcoming mode collapse in education will require richer instructional data and investment in teachers' creative capacity and professional judgment.
Large language models and conversational counter-arguments to antipublic sector bias
John D Marvel, Sheeling Neo, Rachel Cho, Sangwon Ju
Can a good argument change an individual’s mind? In three preregistered experiments, we explore this question in the domain of public sector organizational performance. We observe human subjects as they engage in conversations with a generative artificial intelligence programmed to argue in one of seven distinct “styles,” including a confrontational challenger style, a didactic style, and a sycophantic style. We develop a theory of effective argumentation predicting that conversational styles which are pleasant and engaging will be more persuasive than styles which are unpleasant or unstimulating. Contrary to this prediction, we find that conversational styles which challenge subjects’ negative views of government agencies produce significant positive attitude change, while sycophantic styles that indulge those views do not. Troublingly, subjects find the sycophantic styles more enjoyable, less frustrating, and more credible than the challenger styles. This dissociation between user experience and persuasive outcome—what we call “grudging persuasion”—suggests that attitude change does not require a pleasant conversational experience, and that the styles subjects enjoy most may be precisely the ones least likely to move them. Our findings point to a potentially dark side of large language model-based persuasion: sycophantic styles that users find most appealing are the least effective at correcting misinformed views.
Controlling the Probability of Unknown Unknowns in Pre-Deployment Testing of AI Systems
Rachel Li, Chris Hays, Aaditya Ramdas, Manish Raghavan
In pre-deployment testing of AI systems such as AI agents, a developer can often deal with known failure modes (e.g., leaking API keys, deleting important databases, or installing a malicious package) by implementing fixes or making contingency plans for when the same issue might occur again. What is more challenging are the unknown unknowns: the unseen failure modes that haven't surfaced during testing. The decision to stop testing and deploy therefore faces an inherent tradeoff: continue testing to observe more behaviors, at the cost of time and resources, or stop testing and deploy, at the risk of failure modes remaining undiscovered. How then does the developer decide when they have tested enough? To answer this, we develop an algorithm that certifies when the probability of observing an unseen failure mode is sufficiently small. The algorithm constructs an anytime-valid upper confidence sequence on this quantity, so the developer can repeatedly test a system, inspect the bound, and decide to stop when it falls below a chosen threshold without invalidating the statistical guarantee. Our problem can be interpreted as a sequential variant of the missing-mass estimation problem which gave rise to the classic Good-Turing estimator (Good, 1953). On SWE-Chat, a dataset of real-world coding agent traces (Baumann et al., 2026), we implement our procedure and show that it achieves tighter upper bounds than a simple multiple-testing corrected Good-Turing baseline, and even dominates a fixed-horizon (non-multiple-testing-corrected) Good-Turing upper confidence bound, which is not anytime valid. Additionally, to better match realistic evaluation pipelines, we extend our method to handle sampling from a different task distribution and imperfect automated failure labeling.
Towards Better Legal Reasoning LLMs: Signal Balancing and Reward Scheduling
Yinghao Hu, Qian Xiao, Yuting Huang, Kaisong Song, Yiquan Wu, Ang Li, Junjie Cao, Jun Lin, Kun Kuang, Fei Wu
In the legal domain, LLMs can improve reliability and interpretability by generating explicit reasoning traces. However, training LLMs to produce high-quality reasoning traces remains challenging. Existing supervised fine-tuning methods struggle when reasoning traces are significantly longer than final answers, as the learning signal for the answer becomes diluted. Meanwhile, reinforcement learning methods such as Group Relative Policy Optimization also have several limitations, including costly reward design, performance plateaus, and reward hacking. We therefore propose a two-stage training framework. In Stage I, JurisCoT-SFT employs a length-normalized dual objective to balance learning signals between reasoning traces and final answers. In Stage II, Lifecycle-Aware Backtrackable Policy Optimization dynamically evaluates and retires auxiliary reward signals based on their impact on final evaluation objectives, enabling efficient reward utilization. Our framework substantially improves legal reasoning performance on relevant tasks selected from multiple legal benchmarks, including ChJGBench, LawBench, and LegalBench. Experimental results demonstrate consistent improvements in both reasoning trace quality and final answer accuracy, while our models remain competitive with larger models.