New Augmented Reality APIs for Multi-User Experiences

Building augmented reality experiences just got a lot easier. Software frameworks like ARKit and ARCore enable any developer to build single user experiences fairly easily. The same could not be said about multi-user experiences…until very recently. With Twilio’s new multi-user WebRTC-based augmented reality APIs, engaging apps will be that much easier to build. Developers rejoice!

Twilio has released a set of tools under their Programmable Video feature set that enables users to sync up multiple AR and VR sessions. This is a major step towards the uptake of AR in productivity, industrial, medical and communications-focused apps. Developers can build experiences like a multi-user video chat where they can all manipulate 3D objects in real time. That’s only the beginning of what could be built. Streem is developing their augmented reality handyman app which is powered by Twilio Programmable Video.

The challenges of developing multi-user experiences include audio and video routing as well as data synchronization across devices. Twilio’s abstracts all of these challenges away with their Data Track and Media Sync APIs. These are the first multi-user Augmented Reality APIs that we are aware of.

“Augmented reality is completely transforming the way we understand and interact with the world around us, especially in terms of remote collaboration,” said Rob Brazier, director of product for Twilio’s Programmable Video platform. “Imagine if the next time you called your cable company, you could simply show them a video of the flashing lights on your cable box and they could instantly know what’s wrong, and then guide you through the process of wiring it up correctly by overlaying the correct cabling on-screen in your physical space. The prospects for improving remote support and sales conversations is incredible.”

Twilio’s tools include:

• Global media server infrastructure
  Twilio’s Programmable Video platform provides globally-distributed low-latency media servers that offer complete client-side control over audio and video layout and spatialization, down to the individual client device. Twilio’s approach offers vastly lower latency, improved quality, and superior mobile integration relative to legacy multipoint control unit (MCU) media server technology, because it avoids transcoding or mixing audio and video and lets the client control layout. This gives developers maximum flexibility to control layout and spatialization in their applications.
• DataTrack API
  The new Twilio DataTrack API shares important metadata between endpoints, without needing to setup a separate communications channel. The DataTrack can transport events about AR objects, details for audio and video spatialization, and more. Metadata is shared in real-time so the AR experience can react to changes immediately and provide a more immersive environment for the user.
• Media Sync API
  Media Sync APIs let developers synchronize AR metadata with real-time media, enabling accurate playback of video and augmented reality objects. Media Sync is coming soon to Twilio’s SDKs for iOS and Android. Developers can sign up to learn more here.

We are expecting to see a lot more tooling released to support the development of Augmented Reality Apps. Because it is such a new and rapidly developing space, there are many opportunities to build open-source augmented reality APIs to support the development community. If you’re building one let us know!