Google has unveiled Android Bench 2.0, a significant upgrade to its benchmark for evaluating AI coding agents. This new version expands the scope of testing beyond simple bug fixes and local code adjustments, now encompassing long-horizon tasks that can take a human engineer several days or even weeks to complete. Initial results, however, reveal a stark contrast in performance; the best pass rate for these extended tasks hovers around 28%, a notable drop from the approximately 91% success rate observed in the previous iteration.
Android Bench 2.0 measures partial progress, not just failure
Launched on September 16, Android Bench 2.0 introduces 30 long-horizon tasks that cover a wide array of challenges, including app creation, migrations, feature additions, and app conversions. Among these tasks are the development of a multi-screen food delivery app from design mocks, the migration of libraries and architecture, and the integration of features like widgets and Picture-in-Picture. Additionally, it assesses the conversion of Flutter or React Native applications to native Android.
This updated benchmark now provides insights into both pass rates and completion rates. A task that successfully compiles, meets most functional requirements, and achieves a significant portion of the user interface can still be deemed a failure overall; however, its partial accomplishments are acknowledged rather than disregarded. Google’s methodology for scoring combines functionality, regression checks, requirements, and visual fidelity, incorporating runtime checks, database inspections, accessibility-tree analyses, and a multimodal visual judge. This comprehensive approach aligns more closely with the complexities of mobile software development than merely confirming whether a code patch compiles successfully.
Where AI coding agents still stall
Despite advancements, Google notes that AI models perform better with new code than with modifications to intricate existing systems. Repetitive migrations can effectively scale across more than 125 files and 8,000 lines of code, yet challenges arise with runtime validation, significant framework changes, and unreleased libraries. Cross-platform conversions, in particular, present formidable obstacles, with leading models achieving a maximum completion rate of 80% and none attaining a flawless pass rate.
The benchmark leaderboard pairs various models with their respective coding agents. For instance, Google tested GPT 5.6 Sol alongside Codex and Gemini 3.8 Flash with Antigravity, emphasizing that the surrounding framework influences token utilization and developer outcomes. OpenAI’s GPT-6 Astra currently tops the long-horizon leaderboard with a 28% pass rate, while Gemini 3.8 Flash trails at 8%.
Developers opting for an AI agent must remain cognizant of the incomplete work that may result. An AI model might adeptly generate the initial 80% of an Android application, yet the subsequent architectural refinements, handling of edge cases, and visual enhancements will still require a human touch.
The benchmark is accessible via Google’s Android Bench leaderboard, which also details the methodology behind the continuous completion rate calculations. For further insights, readers may explore how OpenAI integrated the Codex harness into its developer API and the ambitions of a new AI model aiming to supplant senior developers.
