In a significant move for augmented reality development, Google has confirmed that Android XR will allow applications to access the passthrough camera view. This development marks a shift in how third-party developers can interact with the camera systems of headsets, such as the Quest 3 and Apple Vision Pro, which currently restrict access to system software only.
Unlocking New Possibilities for Developers
While existing headsets utilize cameras to provide users with a glimpse of their surroundings, developers have been limited to higher-level data derived from these cameras. This includes skeletal coordinates for hands and bodies, a 3D mesh of the environment, and basic object tracking capabilities. The inability to implement custom computer vision models has constrained the potential for augmented reality experiences.
Apple has made strides in this area by granting enterprise companies raw access to the Vision Pro’s passthrough cameras for internal applications, albeit under strict licensing conditions and limited to business settings. However, the landscape is evolving as Meta has announced plans to provide Quest developers with access to passthrough cameras in early 2025, further expanding the toolkit available for immersive experiences.
Google’s recent announcement to veteran VR developer Antony Vitillo outlines the specifics of accessing the passthrough camera streams. Developers will be able to request camera permissions similar to those used in standard Android applications, utilizing the same APIs. This functionality includes:
- Camera Access: Developers can request cameraid=0 for the main world-facing camera stream, akin to the rear camera on smartphones, and cameraid=1 for the selfie camera stream.
- Permission Requirements: To access the world-facing camera feed, apps must request camera permissions, mirroring the process for standard camera access on mobile devices.
- Avatar Stream: When accessing the selfie camera, developers will receive a video stream of the user’s avatar, generated by Avatar provider apps based on user tracking data from OpenXR APIs.
This tracking data is sourced from the inward-facing cameras that monitor user movements and facial expressions, allowing for a more immersive interaction within augmented reality environments.
As the industry gears up for these advancements, the potential for innovative applications in augmented reality is set to expand, paving the way for a new era of immersive experiences.
