Google’s ongoing enhancements to Android Bench have culminated in the release of version 2.0, a significant leap that showcases the capabilities of artificial intelligence in managing intricate development tasks.
Advancements in Task Complexity
The initial iteration of Android Bench primarily concentrated on minor adjustments within existing repositories, such as bug fixes and smaller feature requests. In contrast, Android Bench 2.0 is designed to tackle “tasks of great complexity,” which may require engineers several days or even weeks to complete. These tasks include adding new features, building applications from the ground up, and adapting cross-platform apps for the Android ecosystem.
This shift in focus necessitated a more sophisticated approach to evaluation and scoring, moving away from a simple pass or fail system. Google is now implementing a continuous scoring method:
“We calculate this completion rate through a combination of factors like functionality, visual fidelity, and avoiding regressions. We also apply objective scoring penalties for deviations from evaluation instructions or structural constraints.”
In this new framework, Google has assessed various AI models, including Gemini 3.7/3.8 Flash, OpenAI GPT-6, Anthropic Fable 5.1, Kimi K3, and Qwen 3.8 Max. Notably, GPT-6 Astra leads the benchmark with a pass rate of 28%, a stark contrast to the previous approach, where scores hovered around the 90% mark.
Challenges in Porting and Refactoring
Google has also provided insights into the ongoing challenges of porting cross-platform applications to Android, noting that no model has achieved a 100% pass rate, with the best-performing models reaching a maximum completion rate of 80%.
- “AI does a better job at writing new code rather than refactoring existing code. Refactors and migrations get trickier because success depends on architectural complexity rather than code volume.”
- “Models show strong capabilities on well-established, deterministic transformations, such as converting Java to Kotlin, swapping Retrofit for Ktor, or introducing a ViewModel layer.”
- “Models struggle when tasks require runtime validation (like missing dependency injection graphs), involve breaking framework changes, or run into knowledge gaps with unreleased libraries.”
In its evaluations, Android Bench utilized “agents from the corresponding model provider,” such as Gemini 3.8 Flash on Google Antigravity and GPT-5.6 Sol with Codex. Google asserts that “harness design positively impacts developer outcomes,” and there are plans to incorporate a variety of model and agent combinations in future iterations of Android Bench.
