AI
OpenAI reports major gains in tests for COBOL migration and legacy code
OpenAI reports large improvements by its newer models on low-resource programming languages, including COBOL, Fortran and Visual Basic. Its COBOL migration test requires byte-for-byte behavioral parity, catching errors that ordinary compilation or limited visible tests could miss. The results suggest AI may be useful for analyzing and drafting migrations of aging systems, but the company says production deployments still require extensive differential testing, domain review and staged rollout. Benchmark performance does not measure full repository maintenance or operational integration.
Editorial summary · Publisher article
What matters
- GPT-6 Astra scored 72.1% on the reported COBOL task, while GPT-6.1 Sol reached 97.9% at high reasoning effort.
- The evaluation checks exact observable behavior, including fixed-point arithmetic and hidden tests.
- OpenAI released MultiLangBench and the migration test materials so teams can inspect and rerun the methodology.
- The report recommends shadow runs, reconciliation and rollback controls because benchmark successes can still miss production edge cases.
Why legacy migrations are difficult
Older programs often encode financial rules through language-specific behavior, data layouts, dependencies and undocumented operational assumptions. A translation can compile and pass incomplete tests while still changing interest, settlement or reporting outcomes.
Near-term use is assistive
The strongest practical applications described are code analysis, test generation, business-rule recovery and migration drafts. Engineers with domain expertise remain responsible for validation, release decisions and handling surrounding systems.
Context & caveats
- The performance figures are OpenAI’s reported results, although the evaluation materials are available for inspection.
- The article’s cited industry estimates about legacy-system usage are not independently established in the supplied material.
Why this made the edition
The report includes concrete benchmark methodology, released evaluation materials, important semantic failure modes and explicit limits on applying model results to production legacy systems.
Source material