Google is intensifying its efforts to enhance the capabilities of its Gemini chatbot and large language model, integrating it seamlessly across its product ecosystem. With Gemini now serving as the default assistant on numerous Android devices, its functionality expands with each update. Currently, while Gemini can interact with various external services, its ability to control Android applications remains limited. However, the upcoming release of Android 16 is poised to change this landscape significantly by introducing a new API that will empower services like Gemini to perform actions on behalf of users within applications.
Gemini Extensions and Their Limitations
Presently, Gemini utilizes Extensions to communicate with external services, allowing it to access platforms such as Google Flights and Google Hotels. These Extensions enable Gemini to retrieve data when users pose relevant inquiries. Additionally, there are Extensions for popular apps like Google Maps, Google Home, and YouTube. However, these Extensions primarily facilitate data access rather than direct control over Android applications. While some Extensions, such as Utilities, allow for basic actions through well-defined intents, they fall short of providing comprehensive control.
The scalability of Gemini Extensions poses a challenge, as the vast number of Android applications makes it impractical for Google to create specific Extensions for each one. Furthermore, many applications lack public APIs that Gemini could utilize. Although a combination of technologies such as screen reading and multimodal AI could theoretically enable users to control any Android app via natural language, the results may lack effectiveness due to insufficient context. A more viable solution appears to be the introduction of an API that allows applications to interact directly with Gemini, a feature that Android 16 aims to deliver.
Introducing App Functions in Android 16
In the recent release of Android 16 DP1, developers discovered a new set of APIs related to a feature termed “app functions.” According to Google’s documentation, an app function represents a specific capability offered by an application to the system, which can be integrated into various system features. Although the description remains somewhat ambiguous, it provides an example where a restaurant app could define a function identified as ‘orderFood’ to facilitate food ordering.
Details about the implementation of these app functions are still emerging. It appears that applications will create functions by defining a service that can only be accessed by a system process. These app functions will be made available to Android’s App Search framework, which powers the universal search experience in the Pixel Launcher, among other functionalities. To execute these app functions, applications must hold either the EXECUTEAPPFUNCTIONS or EXECUTEAPPFUNCTIONS_TRUSTED permission in Android 16.
Both permissions are restricted to system applications, with the former currently granted to system apps that hold the ASSISTANT role (such as the Google app), while the latter is reserved for those with the SYSTEMUIINTELLIGENCE role (like Android System Intelligence). These permissions will enable applications to perform actions on behalf of users within other applications, although app developers can choose to limit access based on the permissions held by the calling applications.
The Potential of Gemini as an AI Agent
While specifics are still unfolding, the new app functions feature in Android 16 could allow Gemini to control applications in a manner that Google Assistant has not achieved in the past. In 2019, Google hinted at the potential for its “new Google Assistant” to orchestrate tasks across different apps, enabling users to multitask and execute complex actions seamlessly. The promise of AI agents today aligns closely with what Google envisioned back then, and given the advancements made over the past year, it seems plausible that Google is revisiting this concept.
As Android 16 approaches its release, the success of Gemini as a true AI agent for Android devices will largely depend on the willingness of app developers to embrace this new functionality. The integration of app functions could mark a significant step forward in transforming how users interact with their devices, paving the way for a more intuitive and efficient mobile experience.
