In an ongoing effort to advance Android capabilities, Google is developing a groundbreaking feature known as Computer Control, which promises to automate Android apps efficiently. This innovation stems from prior initiatives in AI-powered devices, which, despite falling short due to issues like expense and practicality, have paved the way for persistent exploration into agentic AI—tools designed to autonomously execute tasks on behalf of users.
Earlier endeavors, like Project Astra, showcased an AI agent capable of autonomously retrieving online documents, diving into specific sections, and locating relevant videos on YouTube, all without manual intervention. These demonstrations, however, were hampered by constraints, notably owing to reliance on existing Android APIs for screen capture and input, which resulted in sluggish performance and interruptions from notifications and alarms.
The Framework of Computer Control
To surmount these challenges, Google has been diligently developing the Computer Control framework, which facilitates automated control over Android apps discreetly in the background. This framework is layered upon the Virtual Display Manager—a service introduced with Android 13—that enables the creation of separate virtual displays apart from the primary, visible screen. This manager already supports functionalities like app streaming to other devices and virtual cameras.
Through Computer Control, applications can operate in a trusted virtual display setting, utilizing virtual input devices to manage touch and key inputs effectively. These client apps delineate the virtual display’s parameters, such as its name, dimensions, and density, and decide if the display should continue its interactive state when the host device is locked, though unlocking is necessary to initiate a session.
A pivotal component of this framework is its ability to mirror the trusted virtual display onto another interactive virtual display with differing dimensions. This ensures that external interactions or resizing do not trigger changes that might restart apps or stifle automation, thus allowing users to monitor or intervene without derailing the automated sequences.
Computer Control is secured tightly, with access limited to apps possessing the freshly introduced ACCESS_COMPUTER_CONTROL permission, granted only to apps with OS-allied certificates. Post-permission, user consent is required to activate Computer Control sessions, ensuring user oversight in the process either on a per-session basis or for consecutive sessions.
Despite its structured rules, many aspects of Computer Control remain ambiguous. It is uncertain whether this automation will function remotely—sending streams to a PC or server for management—or locally via inherent on-device multimodal models. While remote engagement aligns seamlessly with the framework's core architecture, local automation would inherently be more secure yet demanding on device resources.
Nevertheless, Computer Control harbors the potential to revolutionize automation and enhance accessibility, as Google persists in refining its performance and security features. This description is grounded on insights gathered from recent Android build codes, with no official release timeline announced by Google yet.
