AI
New coding benchmark finds AI agents still struggle with real-world engineering judgment
Cognition’s FrontierCode 2.0 benchmark tests coding agents on deliberately underspecified tasks, including repository conventions, edge cases, cleanup and API stability. Its top model scored 45%, while many agents missed obligations an experienced maintainer would infer. Examples include leaving abandoned generators unclosed and removing a public API while fixing a race condition. The benchmark is company-produced, so teams should treat its rankings as one signal and reproduce representative tasks in their own repositories.
Editorial summary · Publisher article
What matters
- The leading model scored 45%, leaving every tested system below half of the available score.
- The updated benchmark uses larger codebases, broader task categories and reference patches that modify substantially more files and lines.
- Higher reasoning settings sometimes produced wider patches, compatibility breaks and unnecessary redesigns.
- The report recommends measuring review burden and regression rates on each team’s own repositories.
Context & caveats
- Cognition produced both the benchmark and its leaderboard, so the results are not independent validation.
Why this made the edition
A detailed benchmark report provides new evidence about agents’ performance on underspecified engineering work, with concrete examples and deployment implications. Cognition produces the benchmark and leaderboard, so independent local testing remains necessary.
Source material