Evaluating Agents Beyond the First Prompt

This article introduces EvoCode-Bench, a novel multi-turn coding benchmark designed to evaluate AI agents beyond single-turn tasks like SWE-bench. It features 26 tasks spanning 227 sequential rounds across five domains: ML/MLOps, build systems, data engineering, cloud/security, and scientific computing. Key aspects include a persistent workspace for code evolution, evolving specifications that extend or conflict with prior requirements, and cumulative testing that checks all past requirements after each turn. An example details building a CLI tool in Go over 8 rounds, showing how agents handle new instructions while maintaining previous functionality. The benchmark aims to reflect real-world software development cycles where requirements shift and codebases grow.