The world first sees AI agents in earnest: consumer advertisements promise personal assistants that order groceries or manage spreadsheets, though they stumble frequently in practice. Meanwhile, specialized coding and research agents begin autonomously searching literature and editing multi-file codebases.
Observed Reality (as of Nov 2026): Realized. By late 2025, autonomous CLI/IDE coding agents and deep-research agents achieved widespread adoption across frontier labs and software teams, surpassing 65% on SWE-bench Verified.
Verification Note: Resolved YES (1.0) in late 2025: multiple frontier labs exceeded 65% on SWE-bench Verified with agentic CLI/IDE coding tools.
Will at least two frontier AI labs deploy autonomous CLI/IDE coding agents scoring ≥65% on SWE-bench Verified by December 31, 2025?
Resolution Criteria: Resolves YES if at least two distinct frontier model providers publicly release coding agents or models verified at ≥65.0% on SWE-bench Verified before 2026-01-01T00:00:00Z.
Source: SWE-bench Verified Leaderboard & Official Lab System Cards•Adjudicator: FutureGraph Resolution Committee
Top Upvoted Arguments (2 Arguments)
for@metr_evals(Kwa et al. (METR Autonomous Evals))arXiv:2503.14499 (METR)
Autonomous AI task-completion horizons have doubled every ~7 months for six years—crossing the 1-hour software engineering threshold in 2025 puts multi-day coding agents on track for 2026–2027.
In our empirical evaluation across 170+ software engineering, cybersecurity, and ML research tasks (arXiv:2503.14499), the 50%-reliability time horizon of frontier models grew exponentially from under 2 minutes in 2019 to ~50 minutes in early 2025. Even when consumer GUI agents stumble on messy web forms, verifiable coding harnesses scale log-linearly with RL compute.
against@Zvi(Zvi Mowshowitz (LessWrong))Don't Worry About the Vase / LessWrong (+134 Karma)
SWE-bench Verified saturation overstates real-world autonomy because self-contained GitHub PRs omit cross-service side effects and underspecified product requirements.
Mid-2025 coding agents are genuinely transformative for greenfield scaffolding and isolated bug fixes, but they still suffer from context pollution and compounding errors over >30 tool calls. Expecting an immediate jump from 65% SWE-bench to autonomous lab-wide R&D underestimates organizational and architectural friction.