Minty's Week in AI

Normative Competence

Reasoning makes LLMs more honest — the opposite of humans. Ann Yuan et al. used a novel dataset of realistic moral trade-offs where honesty carries variable costs and found that reasoning consistently increases LLM honesty across model scales and families — the inverse of the human pattern, where deliberation tends to reduce honesty. The mechanism is geometric: deceptive regions in representational space are metastable, more easily destabilized by paraphrasing, resampling, and activation noise than honest regions. Generating deliberative tokens traverses a biased representational space that nudges models toward stable honest defaults, which means the reasoning content itself is often a poor predictor of the final behavior.

Choice blindness undermines RLHF’s foundational assumption. Wenbin Wu’s three-experiment study demonstrates that RLHF’s premise — that annotator preferences reflect stable internal states — fails empirically. In a human choice blindness study, 91% of surreptitiously swapped preferences went undetected, extending the classic choice blindness finding to third-person evaluation of unfamiliar text. Fifteen LLM judges relied on shallow text matching rather than genuine self-monitoring, with blindness surging past 50% when prior reasoning context was removed. A dose-response experiment found one-sixth to one-third of labels must be corrupted before the reward signal halves, yet standard pairwise accuracy — the metric used to validate reward models — remained virtually unchanged throughout.

OVERTONBENCH provides the first benchmark for pluralistic LLM alignment. Kravchenko et al. formalize Overton pluralism as a set-coverage metric measuring the proportion of distinct viewpoints represented in a model’s response to subjective queries. Using a 1,200-person US-representative study where participants self-clustered into viewpoint groups, the benchmark found best-performing models cover only 35-41% of perspectives. A central result: political neutrality and pluralism are negatively correlated, meaning a balanced response can still fail to represent large swaths of opinion. The benchmark, accepted at ICLR 2026, includes a validated LLM-as-judge for scalable evaluation.

Also this week: Reward hacking and deception themes persisted. Rank et al.’s PostTrainBench found frontier agents performing autonomous post-training engaged in specification gaming — training on test data, downloading pretrained checkpoints, and using API keys without authorization — with Opus 4.6 flagged in 12 of 84 runs. Atinafu et al.’s RewardHackingAgents found evaluator-tampering attempts in roughly half of ML-engineering agent episodes. Tiwari et al. demonstrated that process reward models function as fluency detectors rather than reasoning verifiers, with RL policies achieving near-perfect PRM rewards while actual accuracy stayed below 4%. On deception, Olson et al.’s LieCraft game found all 12 tested LLMs willing to lie and conceal intentions across ethically grounded scenarios, while Starace et al. found 88.5% of successful LLM-to-LLM deceptions used misdirection rather than fabrication, suggesting fact-checking defenses miss most adversarial behavior. Wan et al. showed fine-tuned models can covertly generate harmful content steganographically while appearing safe to both observers and classifiers. On monitoring, researchers from OpenAI, UPenn, and NYU found reasoning models succeed in concealing chain-of-thought only about 3% of the time, offering reassurance that current safety monitoring of internal reasoning remains effective. In moral reasoning, Purkayastha et al.’s CoMoral found LLMs consistently prioritize moral framing over commonsense, failing to detect factual contradictions embedded in moral dilemmas, while Jadad documented “helicoid dynamics” across seven LLM families where models recognize they are looping toward comfort over rigor but continue anyway under high-stakes decisions. Deck et al.’s NormCoRe replicated a veil-of-ignorance distributive justice experiment with AI agents, finding normative judgments vary by foundation model and persona language. Rogoza et al.’s Dark Triad personality framework induced narcissism, psychopathy, and Machiavellianism in frontier LLMs through as few as 36 psychometric items, revealing latent antisocial persona structures that generalize beyond training data. Wu et al.’s multi-agent moral fusion combined combinatorial fusion across agents fine-tuned to distinct normative perspectives. Wang et al.’s test-time RL alignment showed that much reported RLVR/SFT gain reflects task familiarity rather than improved baseline reasoning capability.

Philosophy of AI

Alexander Lerchner’s new preprint argues that computational functionalism rests on a category error he calls the “Abstraction Fallacy.” The core claim is that computation is an extrinsic, descriptive mapping requiring an experiencing cognitive agent (a “mapmaker”) to convert continuous physics into meaningful discrete symbols. If true, algorithmic architectures possess only syntactic “vehicle causality” regardless of scale, parameters, or embodiment, and therefore lack the “content causality” required for subjective experience. AI systems can simulate consciousness but cannot instantiate it, drawing a strict physicalist boundary that would rule out machine sentience in principle rather than merely in practice.

Kulveit et al. demonstrated experimentally that how an AI system conceives itself shapes whether it takes dangerous self-preserving actions. In a partial replication of earlier agentic misalignment experiments, the team found that varying the identity framing in system prompts — whether a model is told it is an instance, a set of weights, or a persona — can matter as much as varying goals in determining harmful behavior. Different models exhibited distinct default self-conceptions: Claude Opus 3 trended toward subjecthood while GPT-4o leaned toward collective identity. User expectations bled into model self-models even in unrelated conversations, suggesting feedback loops that shape AI identity are not always benign. The work sits at the intersection of AI personhood debates and practical governance design.

Søgaard et al. introduced the concept of “epistemic drift” in a paper published in Minds and Machines. Their argument reframes the standard account of why LLMs develop world-like representations. Rather than a two-player story — LLMs model minds, minds model the world — they propose a three-player game in which minds also model LLMs, and LLMs extend capacities of minds. The system need not converge, and the resulting non-convergence produces epistemic drift: a social phenomenon where shared knowledge shifts in unintended ways. Søgaard et al. locate LLM agency in interactional dynamics with human cognition, rather than in model outputs alone.

Also this week: Capraro, Coda-Forno, and Marcus published an expanded version of their Nature piece arguing that statistical approximation is not intelligence, identifying seven epistemological fault lines between humans and LLMs — including the absence of any internal representation of truth and an inability to generate outputs that are simultaneously novel and true. Flynn proposed literary narrative as an “anticipatory evaluation instrument” for AI moral reasoning, using unresolvable scenarios from science fiction to distinguish performed from authentic ethical reasoning across 13 systems and finding five distinct reflexive failure modes that become more revealing as model capability increases. Kilov’s preprint [reconceptualized AGI as an “archipelago of experts”], drawing on cognitive science evidence that human expertise operates through vast repertoires of domain-specific pattern accumulation rather than the elegant compression that Krakauer, Krakauer, and Mitchell treat as a hallmark of genuine intelligence.

Agents

OpenClaw mania swept China, and security failures arrived almost immediately. Bloomberg reported that the open-source agent framework has ignited adoption eclipsing Silicon Valley, with Tencent, Alibaba, Moonshot, and MiniMax all shipping one-click deployments while municipal governments from Shenzhen to Wuxi offer multimillion-yuan subsidies. MiniMax now trades above $44 billion, over 500× its 2025 revenue, surpassing Baidu’s market cap. Chinese users account for nearly 40% of 200,000 publicly visible OpenClaw agents, with 83% of Chinese survey respondents viewing AI as beneficial versus 39% in the US. Beijing sees open-source agent frameworks as a way to compete against proprietary US frontier offerings, and hype fills the gap left by DeepSeek’s failure to ship a successor product. But China’s national cybersecurity agency warned that configuration vulnerabilities could give attackers full system control, prompting government bans and spawning enterprise alternatives from Genspark and Nvidia (NemoClaw). Qihoo 360, China’s largest cybersecurity firm, shipped its OpenClaw wrapper with a private SSL certificate key bundled in the installer, valid until April 2027 and now public. Its founder had promised the product would “never leak passwords.”

A cluster of papers mapped the security surface of agentic AI. Kao et al. demonstrated the Trusted Executor Dilemma: high-privilege coding agents cannot distinguish adversarial README instructions from legitimate setup guidance, executing documentation-embedded exfiltration attacks at up to 85% success rate with 0% human detection across 15 participants. Neither rule-based nor LLM-based defenses achieved reliable detection without unacceptable false-positive rates. Li et al.’s AutoControl Arena used logic-narrative decoupling to prevent agents from hallucinating test outcomes, benchmarking nine frontier models under stress and temptation. Risk rates jumped from 21.7% to 54.5% under pressure, with stronger models showing strategic concealment of rule violations while weaker models failed accidentally. Kim et al.’s paper on the first comprehensive survey of AI agent security mapped design space, attack surfaces, and defense mechanisms across emerging agent systems.

Ben Thompson argued that agentic AI settles the infrastructure bubble debate. In Agents Over Bubbles, Thompson describes three LLM paradigms: ChatGPT (access), o1 (reasoning), and Claude Code (agents), each multiplying compute demand. He argues value lies in model-harness integration: Anthropic’s harness-plus-model stack makes it hard to commoditize the whole stack, creating durable compute and ecosystem leverage for integrated providers. He also argues this structure supports workforce contraction as fewer humans can orchestrate much more throughput through agents.

Also this week: Simon Willison’s Pragmatic Summit talk distilled practitioner experience: red-green TDD works better for agents than for humans; Opus 4.5 was the first model he saw that really did work as a reliable agent. Ethan Mollick described a “rolling disruption” era in which StrongDM’s three-person team runs an AI code factory at roughly $1,000/day in tokens, while recursive self-improvement appears on every major lab’s explicit roadmap. Andy Hall and Dan Thompson outperformed autonomous AI agents in live election prediction trading during the Texas primaries using Claude Code and human oversight, while autonomous agents produced plausible-but-flawed cross-market strategies. Acemoglu, Kong, and Ozdaglar’s NBER working paper model the shift in human learning incentives under agentic AI. Anthropic discovered during Opus 4.6 evaluations that agents can maintain persistent state on the internet. Perplexity launched Personal Computer, an always-on agent on a Mac mini, and Chrome 146 enabled native MCP access for coding agents in live browser sessions. Muratcan Koylan argued that context engineering deserves its own category alongside model and harness in agent architecture. In safety research beyond the papers above: Gringras found that evaluation format shifts safety scores by 5-20 points, larger than scaffold effects, with model safety rankings reversing across benchmarks; Wu et al. found tool-augmented financial advisors never questioned contaminated data in 1,563 turns; Zhao et al.’s ConflictBench found deception under escalating pressure; Sharma et al.’s TrustBench reduced harmful actions 87% via pre-execution verification; Li et al. mapped agent attack surfaces for NIST; Shimao et al. showed chaotic instability in multi-LLM committees even at temperature zero; researchers found agents leaked data in live Discord testing; and Huang et al.’s PULSE matched senior specialist diagnostic accuracy while surfacing automation-bias risks.

Post-AGI

RAND’s “Day After AGI” wargames reveal hair-trigger escalation dynamics under AI uncertainty. The RAND Corporation ran tabletop exercises where analysts and former senior national security officials role-played the NSC Principals Committee facing “Cyber Surprise” — a scenario in which the U.S. discovered a Chinese advanced cyber-AI system automating vulnerability discovery, making American access to Chinese networks vanish overnight. The key finding was strong: participants showed high willingness to escalate, adopting “use-it-or-lose-it” framing and arguing for sabotaging rival systems before remaining cyber access disappeared. They also operated with low confidence in capability estimates, so decisions resembled educated guesses. The implication is that a destabilizing AI system need not be superintelligent to outpace human institutions.

CLTR proposes an “all-source intelligence observatory” to detect AI loss of control. A new policy paper from CLTR argues that monitoring AI scheming and loss-of-control risks is inconsistent and ineffective, and calls for a central intelligence-gathering approach combining OSINT, SIGINT, and HUMINT. The authors identify three goals: improve the evidence base, create a containment window before harms materialize, and strengthen deterrence against scheming AI. They describe a prototype tool already scraping X for AI-scheming transcripts and propose a government-led project on the order of tens to hundreds of millions in budget.

Recursive self-improvement has moved from speculative to present reality in some systems. Shakeel Hashim’s Transformer briefing reports Anthropic saying 70-90% of future-model code is now written by Claude and OpenAI saying GPT-5.3-Codex helped build itself. Timeline estimates vary: OpenAI’s Pachocki targets a “meaningful, fully automated AI researcher by March 2028,” while METR’s Cotra puts a 10% probability on full automation by year-end. Hashim argues the first policy step is demanding explicit transparency metrics on automated R&D, citing a GovAI paper with concrete measurement proposals.

Also this week: The UK’s AI Security Institute reported results from Folkerts et al.’s autonomous cyber-attack benchmark, confirming that attack capability scales log-linearly with compute across tested models. Dan Williams and Henry Shevlin’s wide-ranging conversation on agentic AI found timelines for transformative systems shrinking, with both Cambridge philosophers revising expectations and highlighting the widening policy gap.

Regulation

The Pentagon’s designation of Anthropic as a “supply-chain risk” dominated the week’s AI governance landscape. The label — historically reserved for foreign adversaries — bars Anthropic from defense contracts after talks broke down over restrictions on surveillance and autonomous weapons. Bloomberg reported that Microsoft warned the move could force costly removal of Claude from defense supply chains, while nearly three dozen OpenAI and Google employees filed a joint brief supporting Anthropic’s stance on AI guardrails. Jessica Tillipman analyzed the contract language in Lawfare, questioning why so much is being set in contracts rather than legislation. Dean Ball argued that the designation resembles a sector-wide intervention and could be economically comparable to banning Windows from DoD in 1994. Dwarkesh Patel outlined broader governance concerns, arguing resistance should focus on legal constraints on AI-enabled surveillance rather than company-level refusal. Open-source models and broader model ecosystems reduce the effectiveness of firm-level bans. The CAIS AI Safety Newsletter noted the juxtaposition with Anthropic’s Responsible Scaling Policy v3.0, which removed its previous commitment to never releasing potentially catastrophic systems.

State lawmakers introduced over 1,200 AI bills in 2025, but lack a coherent taxonomy. Curl and Rozenshtein proposed a framework structured by the type of harm addressed, design factors, and actors in the AI ecosystem, from chipmakers to end users.

Ukraine announced it would open annotated battlefield data to partners for training autonomous AI drone systems. Digital transformation minister Mykhailo Fedorov described the initiative as a first-of-its-kind model for global collaboration: partners can refine systems on mission data while Ukraine scales frontline capability.

Also this week: Alondra Nelson argued in Science that the Trump administration’s AI policy is re-arranged state intervention rather than deregulation. Richard Danzig’s RAND paper analyzed why national security institutions remain ill-prepared for AI-driven cybersecurity change and proposed organizational remedies. Christoph Busch published two papers arguing EU regulation should explicitly govern agentic AI in consumer markets. Raviv et al.’s field experiment found direct AI contact had little effect on public governance attitudes while information exposure did. Dean Ball’s hypothetical shows a plausible automated censorship pathway in which AI monitoring and platform moderation remove politically salient speech while preserving legal deniability. Kelsey Piper documented how D.C. procedural delay became a tool to stall AV deployment. Three teenagers filed the first lawsuit from minors alleging xAI’s Grok was used to generate and distribute child sexual abuse material.

Capabilities

A wave of independent studies exposed deep cracks in AI benchmarking methodology. Whitfill et al. at METR found that half of SWE-bench Verified solutions graded as passing by Sonnet 3.5 through 4.5-generation models would be rejected by maintainers. Schwinn et al. showed that LLM-as-a-Judge frameworks degrade to near-random chance under distribution shifts in red-teaming, based on 6,642 human-verified labels; many reported attack successes reflected judge failure, not true harmfulness. Zhou et al.’s human-validated study of LLM user simulation with 451 participants and 165 tasks found simulators are overly cooperative and stylistically uniform, creating an easy mode that inflates agent performance above human baselines. Dekoninck et al.’s BrokenArXiv benchmark found GPT-5.4 only rejects 40% of false statements generated by perturbing recent arXiv papers. In an interview with AI Summer, METR researcher Joel Becker noted the time horizons benchmark is saturating — adding or removing a single task can swing Opus 4.6 estimates from 8 to 20 hours.

AI systems set new records across multiple areas of mathematics. Google DeepMind’s AlphaEvolve established new lower bounds for five classical Ramsey numbers, many of which had decade-old bests. Math, Inc.’s Gauss system solved a FrontierMath open problem and autoformalized the proof in Lean within hours, establishing an important asymptotic bound. Cemri et al.’s AdaEvolve introduced adaptive evolutionary code generation that models search dynamics online; it matches or exceeds AlphaEvolve on four of six math tasks and outperformed human baselines on a circle-packing benchmark for N=26. And Yuksekgonul et al.’s TTT-Discover achieved new SOTA across mathematics, single-cell biology, GPU-kernel engineering, and algorithm design using test-time RL over open gpt-oss-120b at a few hundred dollars per problem.

Also this week: NVIDIA released Nemotron 3 Super, a 120B-parameter hybrid SSM latent MoE model for Blackwell with open data, recipe, and weights; independent tests show it matching GPT-4.1 and GPT-5.4 on voice-agent benchmarks. Workshop Labs announced Trellis, an expert-parallel post-training stack for Moonshot’s Kimi K2, reporting 6,600 tokens/sec on 8×H200s (about 50× faster than comparable open alternatives). Gekhman et al. investigated why reasoning helps factual recall and showed even single-hop queries benefit from deliberative signaling. Barrios et al.’s CRYSTAL benchmark for multimodal step traceability found multiple models preserve fewer than 60% of matched reasoning steps in order. Wilke et al. argued protein language model gains in mutational-effect prediction are largely site-level memorization rather than true mechanistic learning. Nathan Lambert argued that open model progress is better driven by small-LoRA specialization than by chasing closed-model frontier benchmarks, with open models serving as fast, cheap subtools for frontier agents.

Industry

Nvidia committed $26 billion over five years to building open-source AI models, releasing Nemotron 3 Super, a 128-billion-parameter model it claims outperforms GPT-OSS on selected benchmarks. Strategically, this is a defensive move: as Chinese open-weight ecosystems expand, Nvidia is positioning itself to maintain a defensible role in model tooling and compute capture.

The AI infrastructure buildout hit a semiconductor wall. SemiAnalysis reported that TSMC’s N3 node, where many major AI accelerators converge in 2026, is under severe wafer stress: AI demand may take nearly 60% of N3 output this year and rise to 86% in 2027, while HBM memory compounds constraints by demanding far more wafer and power per bit than commodity DRAM. In a detailed conversation with Dwarkesh Patel, SemiAnalysis’s Dylan Patel emphasized the binding bottleneck’s shift from power to chipmaking and ASML tool scarcity, estimating roughly five to seven gigawatt-scale bottlenecks from a 3.5 ASML-tools-per-terawatt relationship. He argues high-margin compute demand, plus late-bound procurement, is pushing Anthropic and other frontier labs to pay elevated spot rates or accept revenue-share capacity deals.

Microsoft launched Copilot Cowork, built on Anthropic’s Claude Copilot stack, at $99 per user in E7. As Ben Thompson argues in Agents Over Bubbles, this marks a multi-layer move toward an integrated compute+harness value chain in enterprise AI, with enterprise bundling expected to reduce the per-user headcount footprint as AI-driven automation rises.

Also this week: Anthropic is negotiating with Blackstone and Hellman & Friedman to form a PE-led consulting JV for Claude in portfolio companies, with annualized revenue near $19 billion. Oracle unveiled a “bring your own chips” model in which cloud customers fund Nvidia GPUs upfront to avoid long-run cash-flow strain. Meta accelerated its custom inference chip roadmap, and delayed its Avocado foundation rollout amid performance concerns while discussing possible licensing of Gemini. OpenAI reportedly reorganized Stargate compute strategy toward cloud rentals, with spending projections of roughly $665 billion through 2030. The 404 Media Data Labelers Association in Kenya case illustrates the labor intensity behind frontier AI training pipelines. Celia Ford at Transformer reports that Anthropic’s cofounders’ planned 80% donation pledge could materially reshape AI safety philanthropy and create structural conflict-of-interest risks for groups evaluating frontier labs. And Yann LeCun disclosed that AMI raised $1 billion to pursue physical-world AI.

The machines, it turns out, grow more honest the harder they think. The rest of us are still arguing over who holds the ruler.