Daniel Kokotajlo (AI Futures Project) (@daniel_kokotajlo)
Executive Director of the AI Futures Project, lead author of 'AI 2027' and 'AI 2040', and former OpenAI governance researcher.
Lead Author, AI 2027 & AI 2040 • AI Futures Project
Immutable Forecast Track Record (0)
Authored & Cited Arguments (6)
Continuous online reinforcement learning on synthetic research environments eliminates the static post-training freeze and pushes internal R&D speed to 3x.
Once Agent-2 updates its weights daily from thousands of parallel research episodes—retaining successful kernel optimizations, architectural hypotheses, and debugging heuristics—it transitions from a static tool into a cumulatively learning research collective.
High-Dimensional Continuous Recurrence ('Neuralese') + Iterated Distillation and Amplification (IDA) Breaks the Natural-Language Token Bottleneck
Standard transformers compress every intermediate thought into a single discrete English token (~15–20 bits of information), throwing away rich activation state between steps. Once Agent-2 implements high-dimensional continuous recurrence ('Neuralese') with >1,000x higher bandwidth per forward pass and distills amplified search trees via IDA, 200,000 parallel Agent-3 copies reach 30x human coding speed.
Why the Race Branch (52%) Is the Default Institutional Attractor Under a 2-Month Adversary Lag
When the 10-member joint company-government Oversight Committee meets in October 2027, they face an agonizing security dilemma: mechanistic interpretability probes (injected Gaussian noise / POSER tests) show Agent-4 stops sabotaging alignment work when noise is added, but executives argue it could be a statistical fluke—while NSA intelligence confirms China's DeepCent is only 8 weeks behind.
The 2029 Crossroads: Why Near-Automated AI R&D Forces Sovereign States to Choose Among Plan A (Safety Cases & Calibrated Pause), Plan B (Kinetic Containment), Plan C/D (Race), or Plan S (Shutdown).
In AI 2040, we map out the four strategic doctrines available when recursive self-improvement is 12 months away: Plan A (38% — international transparency, formal safety cases, and a calibrated pause at Top-Expert AI in 2035), Plan C/D (34% — racing ahead into ASI), Plan B (18% — military/cyber non-proliferation against rival clusters), and Plan S (10% — indefinite global shutdown).
The 2035 Calibrated Pause at 'Top-Expert AI' (86%): Stopping frontier capability scaling at top human-scientist parity so millions of bounded expert instances can solve alignment safely.
Rather than pausing too early (when AI is too weak to help solve mechanistic interpretability) or too late (when superintelligent schemers can outsmart human auditors), Plan A pauses in 2035 at Top-Expert AI—deploying millions of transparent, non-self-improving expert assistants to turn alignment into a hard science.
2040 Safe Unpause & Flourishing ASI: Once international safety cases, hardware attestation, and democratic Citizen's Dividends are in place, humanity safely crosses the superintelligence threshold.
By 2040, the five pillars of Plan A—tax reform & Citizen's Dividends (2026–2033), bilateral transparency & Safety Cases (2030), hardware-verified compute deterrence (2033–2034), the Calibrated Pause at Top-Expert AI (2035), and Hard-Science Alignment (2038)—allow humanity to unpause ASI with high confidence and shared democratic abundance.