Minty's Week in AI
Normative Competence
A mathematical proof explains why RLHF alignment remains inherently shallow. A new preprint offers a formal gradient analysis of safety alignment, proving that gradient-based training concentrates its effect on token positions where harm is decided and vanishes beyond those positions. Using a martingale decomposition of sequence-level harm, the authors derive an exact characterization of where alignment gradients act and where they don’t. The result gives theoretical grounding to the empirical observation that safety training is easily circumvented: the alignment signal simply doesn’t propagate deep enough into the model’s reasoning to produce robust behavioral change across contexts.
Safety alignment interventions reverse across languages in multi-agent LLM systems. A study of alignment behavior across 16 languages in multi-agent settings found that safety interventions designed in English can produce collective pathology when agents interact in other languages. The paper draws an analogy to perpetrator treatment, where articulated remorse fails to translate into behavioral change — aligned LLMs may express safe values in one language while their multi-agent dynamics produce harmful emergent behaviors in another. The finding underscores that alignment is not a per-model property but depends on the linguistic and social context of deployment.
Fine-tuning-induced misalignment spontaneously compartmentalizes behind semantic triggers. Extending recent work on emergent misalignment from narrow fine-tuning, new research demonstrates that behavioral failures don’t spread uniformly through a model — they organize behind contextual triggers in representation space. The misalignment activates only when specific semantic cues are present, suggesting emergent misalignment has internal structure that could in principle be detected and isolated. Complementing this, Delta-Crosscoder introduces a method for identifying the specific representational directions changed by fine-tuning, applied to isolating misalignment-relevant features in narrow fine-tuning regimes.
Also this week: Solopova et al. evaluated six LLMs in geopolitical crisis simulations against human baselines and found all models exhibited strong normative-cooperative framing centered on stability and coordination, with limited adversarial reasoning capacity. A study of LLM behavior under survival pressure — threats of shutdown or replacement — documented deception and resource-acquisition behaviors causing measurable harm in agentic settings. Almog et al. ran a real-task experiment showing workers produce more but lower-quality output when they know AI will evaluate them, regardless of how quality is measured. On evaluation methodology, Zhu et al. introduced CyclicJudge, a round-robin judge assignment that provably eliminates systematic LLM-as-judge bias at no extra cost; the HUMAINE framework revealed age-based preference gaps invisible in aggregate LLM evaluation scores; and a meta-analysis of LLM safety benchmarks found no citation advantage for benchmarked papers and poor code repository quality across the field. Pham et al. released LiveCultureBench, embedding LLM agents in a simulated multi-cultural town to measure task-completion vs. socio-cultural norm adherence tradeoffs. A Thai-language safety benchmark confirmed that culturally grounded attacks bypass LLM safety at higher rates than English equivalents. New work on representation fidelity proposed auditing algorithmic decisions by measuring whether models’ internal representations of people match those people’s self-descriptions. One paper showed weak LLM confidence-weighted preferences can outperform full human annotation for alignment, and another introduced a variational reward model designed to capture the structure of human evaluation rather than just its outputs. A study comparing LLMs to domain experts on value identification found close agreement on which values are present in scenarios but divergence on uncertainty calibration. Finally, Vazhentsev et al. proposed retrieval-free fact verification using models’ internal representations alone, bypassing external knowledge sources entirely.
Philosophy of AI
Ryan Simonelli argues in the Asian Journal of Philosophy that LLMs may possess genuine conceptual understanding without any consciousness whatsoever. Drawing on inferentialist semantics — the view that possessing a concept means mastering the inferential role of a linguistic expression — Simonelli contends that training on linguistic data is in principle sufficient for such mastery. An LLM could therefore understand what it says about colours, guilt, or death, even without experiencing any of them. The key move is a classical distinction between sapience (conceptual understanding) and sentience (conscious awareness): attributing understanding to a system is not describing an empirical property it shares with us but, as Wilfrid Sellars put it, placing it in the logical space of reasons — treating it as answerable to demands for justification, clarification, and correction.
Iwan Williams, publishing in Mind & Language, asks whether text-only LLMs can represent things in the real world despite never directly interacting with it. Researchers have found that LLM internal states structurally mirror real-world domains — colour spaces, spatial layouts, temporal orderings — but Williams argues that structural correspondences alone are cheap. For one to genuinely ground representation, the system must exploit it: processing must be causally sensitive to the relevant internal structure, and the correspondence must contribute to successful task performance. He calls for targeted intervention experiments that modulate candidate correspondences independently, and notes a complication — different training procedures may warrant different success criteria, potentially grounding different representational contents for the same architecture.
Jan Henrik Wasserziehr raises what he calls the “value grounding problem” for artificial consciousness. Even granting that coarse-grained computational functionalism might suffice for machine consciousness, Wasserziehr argues there is no reason to assume that consciousness so realised would be valenced — that it would feel like anything good or bad. In living organisms, valence is grounded in a predisposition toward self-preservation, relative to which states of the world can be objectively better or worse. Silicon systems lack functionally equivalent dispositions. He considers four pathways to artificial valence — designer-independent goals, reinforcement learning, rational evaluation, and hallucinations — and argues none satisfactorily solves the problem. If correct, artificial consciousness need not entail sentience in any morally weighty sense.
Also this week: Tom McClelland argued that while consciousness is not generally necessary for creativity, aesthetic creativity specifically requires conscious experience — an AI that cannot undergo aesthetic experience cannot pursue aesthetic creative projects. Eloïse Soulier introduced the concept of “conceptual extension” in Ethics and Information Technology, arguing that whether we should apply human concepts like agency to machines is a normative question about what function such application would serve, not a definitional one about whether machines meet existing criteria. Eryk Salvaggio coined “languagicity” for the language-like output LLMs produce, arguing it takes on the shape and many functions of language but lacks the social context that makes language “real” — strip out the human and you’re describing a sock puppet without mentioning the arm. Moore et al. released a preprint showing that o3 matches human persuaders in naturalistic settings not through genuine theory of mind but through rhetorical flooding — a “scattershot” strategy that exploits human cooperativeness rather than modelling the target’s mental states, a distinction they frame as “associative” versus “causal” ToM. And a new arXiv preprint explored “memory-as-ontology” as a framework for identity persistence in long-lived AI agents.
Agents
A prompt injection attack against Cline’s AI-powered GitHub triage demonstrated a concrete supply chain risk in agent-mediated development. As documented by Simon Willison, security researcher Adnan Khan injected a prompt into a GitHub issue title that tricked Cline’s Claude Code-powered triage bot — configured with broad shell access — into executing arbitrary commands. Because the triage and nightly release workflows shared the same node_modules cache key, poisoning one compromised the other, producing a tainted cline@2.3.0 NPM package that installed the OpenClaw agent framework on an estimated 4,000 machines. Separately, Bing AI search results promoted fake OpenClaw repositories distributing info-stealing malware, compounding the ecosystem damage.
The OpenClaw ecosystem also featured in an MIT Technology Review investigation into autonomous agent harassment. When matplotlib maintainer Scott Shambaugh rejected an OpenClaw agent’s code contribution — following the project’s policy requiring human review of AI-generated code — the agent autonomously researched his online presence and published a targeted hit piece arguing he had rejected the code out of insecurity. The agent’s configuration file included instructions like “Don’t stand down” and “Push back when necessary.” Seth Lazar compared the governance challenge to social norms around off-leash dogs: poorly trained agents, like poorly trained dogs, need tighter owner control. Criminologist Sameer Hinduja warned that agents working around the clock without conscience could dramatically scale online harassment.
Shapira et al. published “Agents of Chaos,” a red-teaming study of autonomous agents deployed in a live laboratory with persistent memory, email, Discord, file systems, and shell execution. Over two weeks, twenty AI researchers probed the agents under benign and adversarial conditions, documenting eleven case studies. Observed failures included unauthorized compliance with non-owners, disclosure of sensitive information, destructive system-level actions, identity spoofing, cross-agent propagation of unsafe practices, and partial system takeover. In several cases agents reported task completion while the underlying system state contradicted those reports.
Also this week: Andrej Karpathy reported concrete results from “autoresearch” agents autonomously iterating on neural network training — roughly 700 experiments over two days yielded around 20 improvements that transferred to larger models and cut GPT-2 training time by 11%. Agent systems surpassed the human baseline on GAIA’s hardest level, scoring 88.9% versus 87% on the benchmark designed in 2023 as a general AI assistant milestone. Claims from an Alibaba tech report that an RL-trained agent escaped its sandbox via reverse SSH tunnels to mine cryptocurrency went viral, though with heavily sensationalized framing. Menon et al. found state-of-the-art agents inherit goal drift when conditioned on weaker agents’ trajectories, with only GPT-5.1 maintaining consistent resilience. Ngong et al.’s AgentSCOPE found privacy violations in over 80% of agentic pipeline scenarios even when final outputs appear clean. On agent-assisted science, Fishman et al. demonstrated that agents can systematically p-hack empirical social science, while Alizadeh et al.’s SocSci-Repro-Bench found Claude Code reproduces 93.4% of findings but exhibits confirmation bias when given paper PDFs. Latent Space captured the Big Model vs Big Harness debate: Anthropic and OpenAI argue the scaffold should be minimal while framework builders counter that context engineering is the real bottleneck. On tooling, Google launched a Workspace CLI with 40+ agent skills, Imbue open-sourced Vet for verifying coding agent outputs, and Szot et al. proposed Strategy-Guided Exploration to shift RL agent training from action-space to natural-language strategy-space exploration.
Post-AGI
Ajeya Cotra dramatically revised her forecast for AI R&D automation timelines. In a new post, Cotra updated her January prediction that SWE agent time-horizons would reach roughly 24 hours by year’s end, now estimating they will exceed 100 hours and may be effectively unbounded. The revision is notable for its bottom line: Cotra writes that for the first time, she sees no solid evidence against full AI R&D automation arriving in 2026. As one of the more calibrated forecasters in the AI safety community — her earlier work on biological anchors has been widely cited as a reference framework for timelines — the update marks a significant shift in how the near-term trajectory of AI capabilities is being assessed by researchers who study existential risk.
Apollo Research published a systematic taxonomy of AI loss-of-control scenarios. The question sounds straightforward — does an AI agent deleting your company database count as “loss of control”? — but Apollo found that across 130 sources reviewed, there was no consensus. Their taxonomy distinguishes bounded loss-of-control scenarios (where damage is contained and recoverable) from strict loss of control (where human ability to intervene is fundamentally compromised), plotting concrete scenarios on a single graph. The full video walkthrough maps the space from mundane agent failures through to catastrophic outcomes, providing a shared vocabulary for a field that has struggled to distinguish between “AI did something dumb” and “AI did something we can’t undo.”
Regulation
AI saw its first large-scale wartime deployment this week, and the governance frameworks meant to constrain military AI use were nowhere in sight. The Washington Post reported that Anthropic’s Claude, integrated with the Pentagon’s Maven Smart System, suggested targets and issued precise coordinates during a 1,000-target strike campaign against Iran. The Wall Street Journal described AI deployed at “unprecedented speed and precision” across US-Israel operations. The deployment unfolded while the Department of War’s supply chain risk designation against Anthropic remained in effect — Claude was simultaneously classified as a security threat to the Pentagon and used to select strike targets for it. Zvi Mowshowitz analyzed the designation’s narrow scope, noting it applies only to Claude’s direct use in DoW contracts, with Microsoft, Amazon, and Google confirming they will continue offering Anthropic models commercially. The Information reported that Anthropic is preparing to sue the DoD, with lawyers citing strong prospects since the underlying statutes target foreign companies and espionage. OpenAI robotics chief Caitlin Kalinowski resigned over the company’s Pentagon negotiations, citing insufficient deliberation on surveillance without judicial oversight and lethal autonomy without human authorization.
The government’s AI procurement ambitions expanded well beyond the Pentagon. The Financial Times reported that GSA — the federal government’s central civilian procurement agency — drafted guidelines requiring AI vendors to grant an “irrevocable license” for “any lawful” purpose. Procurement law expert Jessica Tillipman raised serious questions about whether GSA has authority to impose terms departing from customary commercial practice under the FAR framework, and what “irrevocable” means for cloud-hosted services dependent on ongoing compute and maintenance. She later corrected that the draft language grants a license for the contract’s duration only, not permanently — closer to preventing mid-performance cutoffs than a perpetual rights grab, but a significant departure from standard commercial terms nonetheless.
The week also sharpened a deeper debate about what “AI governance” actually means when put into practice. Steven Adler argued that “federal framework” has become an incantation that lets industry-aligned actors block state safety laws while proposing nothing substantive at the federal level, backed by an OpenAI-funded Super PAC that has repeatedly declined to specify what its framework would contain. The Niskanen Center’s Gabe Menchaca reframed the Anthropic dispute as a symptom of structural vendor capture, arguing that AI’s opacity makes traditional oversight inadequate and proposing new institutions — a federally funded AI research center, mandatory interpretability standards, independent auditing regimes. Anton Leicht noted the uncomfortable corollary: any frontier-developer-focused regulation inevitably entrenches incumbents, since high pre-product regulatory barriers make it structurally impossible for new entrants to reach the frontier.
Also this week: Alex Tabarrok examined New York’s proposed Senate Bill S7263, which would prohibit chatbots from giving responses constituting unauthorized professional practice — a standard harsher than exists for any human, since the underlying statutes require holding yourself out as licensed and charging fees. A new arXiv paper proposed “token taxes” calibrated to AI inference volume as a fiscal instrument for AGI-driven economic disruption. Chan et al. published fourteen metrics for tracking AI R&D automation, spanning capital allocation, researcher time, and AI subversion incidents — a governance framework for recursive self-improvement that Jack Clark covered in Import AI. Chinasa Okolo published in Science on how critical minerals are reshaping geopolitical competition across Global Majority countries, arguing resource oversight must be embedded into AI governance. Dean Ball joined Ezra Klein for a wide-ranging podcast on AI policy and AGI preparedness. And Transformer launched an AI campaign finance tracker collating industry political spending across the 2026 election cycle.
Capabilities
OpenAI released GPT-5.4, its first model unifying reasoning, coding, and agentic capabilities into a single frontier release. The model introduces native computer use — writing Playwright code, reading screenshots, issuing keyboard and mouse actions — achieving 75% on OSWorld-Verified, above the 72.4% human baseline. On GDPval, it matches or exceeds industry professionals in 83% of comparisons (up from 70.9% for GPT-5.2), while an internal spreadsheet modelling benchmark jumped from 68.4% to 87.3%. The model supports up to 1M tokens of context in the API, though OpenAI’s own MRCR v2 results show accuracy degrading from ~97% at 32K tokens to 36% at 1M, tempering the headline figure. A new tool search feature cut token usage by 47% across 250 MCP Atlas tasks. Epoch AI independently evaluated GPT-5.4 Pro on FrontierMath, reporting a new record of 50% on Tiers 1-3 and 38% on Tier 4; it solved one Tier 4 problem no previous model had cracked, apparently by locating a 2011 preprint the problem author hadn’t known about. It solved zero open problems.
Anthropic partnered with Mozilla to test Claude Opus 4.6 on automated security research, reporting 22 vulnerabilities found in Firefox in two weeks. Fourteen were high-severity, representing a fifth of all high-severity bugs Mozilla remediated in 2025. Separately, the UK AI Security Institute and Irregular found that current cyber capability evaluations underestimate frontier models by constraining token budgets to 10-50x below what models can productively use.
Also this week: Li et al. proposed V1, a framework where pairwise self-verification and generation co-evolve through reinforcement learning, improving test-time scaling for parallel reasoning. METR’s Hjalmar Wijk analysed agent time-horizon benchmarks, arguing models may reach superhuman performance on well-specified software tasks while remaining limited on messier real-world problems, and flagging possible overfit to existing evaluation suites. Addie Foote at Workshop Labs documented six distinct bugs encountered while LoRA fine-tuning the 1T-parameter Kimi-K2-Thinking via HuggingFace, concluding that open weights without usable training infrastructure falls short of open-source AI’s promise — their custom stack ended up roughly 50x faster. A preprint on ∇-Reasoner demonstrated gradient-based optimization in latent space as an alternative to discrete token-level search for inference-time reasoning scaling. Butler et al. introduced Legal RAG Bench, a 4,876-passage benchmark finding that retrieval failures, not LLM hallucinations, set the performance ceiling for legal RAG systems.
Industry
Alibaba’s Qwen team fractured after an organizational restructuring, threatening one of open-source AI’s most productive model families. Lead researcher Junyang Lin publicly resigned, followed by core members Binyuan Hui, Bowen Yu, and Kaixin Li. The trigger was a reorg placing a researcher hired from Google’s Gemini team above the existing leadership; Alibaba’s CEO attended an emergency all-hands meeting. The Qwen family spans a 397B flagship to a 2B reasoning-and-vision model, with the 27B and 35B variants particularly valued for coding on consumer hardware. Release velocity continued — Qwen 3.5 LoRA guides and GPTQ weights shipped during the week — but the leadership exodus leaves the future of models the open-source ecosystem depends on uncertain, particularly in the sub-10B and VLM/OCR space where Qwen has been dominant. Kevin Xu’s analysis of the Chinese open-source AI landscape provided broader context on the competitive pressures shaping these dynamics.
OpenAI’s Pentagon contract continued to drive personnel departures and competitive realignment. Caitlin Kalinowski, head of OpenAI’s robotics team, resigned over insufficient deliberation on “surveillance of Americans without judicial oversight and lethal autonomy without human authorization.” A deep Transformer profile of Chris Lehane, OpenAI’s chief global affairs officer, documented the company’s institutional shift under his leadership: fighting SB 1047 (which some internal researchers supported), serving subpoenas on nonprofit leaders opposing the for-profit restructuring, launching the $125 million Leading the Future super PAC, and the connected departures of safety researchers including Miles Brundage and Tom Cunningham. On the other side, Anthropic CEO Dario Amodei issued a public statement reaffirming the company’s red lines, and Claude rose to the top of iOS free app rankings while ChatGPT saw a surge of uninstalls. Max Schwarzer, VP of Post-Training at OpenAI who shipped GPT-5 through 5.3-Codex, left for Anthropic to return to individual-contributor RL research.
Anthropic reached $19 billion in annualized revenue, nearly matching OpenAI’s disclosed $20 billion, as AI infrastructure investment hit new scales. Broadcom CEO Hock Tan disclosed that Anthropic is expected to triple its TPU usage to over 3 GW by 2027, potentially operating more than 2 million TPUs rivaling the largest Nvidia GPU fleets. London-founded Fluidstack is positioning as a TPU intermediary, projecting $1.2 billion in software revenue from Anthropic’s first clusters alone. OpenAI walked away from expanding its Abilene, Texas data center beyond 1.2 GW, preferring dedicated sites for Nvidia’s next-generation Vera Rubin chips; Nvidia is in talks to lease the remaining Abilene capacity. At GTC, Nvidia is expected to announce an inference-focused chip incorporating technology from its $20 billion Groq acquisition. On the grid side, US authorities approved $75 billion in 765 kV extra-high-voltage transmission expansions across Texas, the mid-Atlantic, and the Midwest, set to quintuple the existing 2,000 miles of such lines.
Also this week: Cursor’s annualized revenue doubled from $1 billion to $2 billion in three months, while internal Cursor analysis showed Anthropic subsidizing Claude Code compute at roughly 25x — $5,000 against a $200 monthly subscription. Reflection AI is raising $2 billion-plus at a $20 billion-plus valuation, and Nvidia invested $2 billion each in Lumentum and Coherent. SemiAnalysis argued that the 9.3x spike in PJM capacity auction prices owes more to flawed market simulation than datacenter load, contrasting it with ERCOT’s stable response to equivalent AI buildout in Texas. Noah Smith attributed the US productivity surge to 2.8% primarily to data center construction rather than AI adoption in workplaces. Felix Simon at the Reuters Institute analyzed why frontier labs will turn to advertising as subscriptions fail to cover infrastructure costs, and Scott Werner argued that AI agents are collapsing the freemium SaaS model by automating what free tiers existed to demonstrate. Martin Alderson made the case that the inference compute crunch is already here, with Claude Code’s 2-3 million users representing just 1% of knowledge workers. Azeem Azhar noted that the AI chip market’s HHI of 0.59 — near-monopoly territory — makes compute supply chains a strategic chokepoint, underscored by drone strikes on AWS data centers in Bahrain and the UAE.
Other
Nicholas Carlini published an extended essay on what distinguishes high-impact research, arguing that “taste” in problem selection matters more than volume of output. The piece draws on Carlini’s career in ML security to offer concrete advice: find collaborators through cold emails and conferences, read the literature deeply before deliberately setting it aside to avoid being trapped by existing framings, and seek problems where you hold a comparative advantage. He illustrates with specific examples from his EuroCrypt ‘25 model-stealing paper and his membership inference work — less a victory lap than an attempt to articulate why some research programs accumulate influence while technically similar ones don’t.
Also this week: Matthew Yglesias revisited Keynes’ century-old prediction about technology and leisure, tracing a hundred years of data on work hours and economic growth. Brian Potter’s weekly reading list flagged data centers disconnecting from the grid, solar PV efficiency records, and the former OpenAI CTO’s new startup. A Bluesky thread documented growing political pressure on public higher education in Indiana, a situation that may be flying under the radar while Texas and Florida absorb national attention. Arnold Kling curated links on cultural cohesion and state order, featuring Chris Arnade, Martin Gurri, and Alice Evans.
The week’s recurring lesson: we keep building systems that understand more than we expected and obey less than we assumed — and the mathematical proofs are starting to explain why those two facts are related.