Apple’s entrance into the XR space via their announcement of the Apple Vision Pro was one of the most anticipated events in recent XR history. After years of hype build-up and frequent delays and rumors, the device itself left a lot of questions to be answered. I mulled over those questions with Neso Brands CTO Paulo Almeida in the 10th episode of the XR Talks podcast.
Years of Watching Apple
Neso Brands is an investment company specializing in tech-augmented eyewear. Naturally, their Chief Technical Officer has been carefully watching the Apple Vision Pro rumors for longer than many companies have been involved in XR at all.
“Somehow I’ve been following it for the last seven years,” said Almeida. “The job descriptions first showed up across the internet and that started the rumors, I’m going to say back in 2014, 2015.”
However, Almeida said that he had been waiting for something like an Apple headset for even longer than that – ever since he was, like so many others, disenchanted by the Google Glass device that launched in 2013. While this device first piqued Almeida’s interest in the future of XR wearables, he doesn’t think that it’s the real spiritual predecessor of consumer XR.
“The real pioneers of this are definitely Magic Leap and their waveguide lens is the future that I was expecting Apple to actually achieve some kind of breakthrough on,” said Almeida. “Magic Leap have been the true pioneers of trying to miniaturize all of these components and to make it into something that we could call smart glasses… not over-the-head displays.”
A Difference of Displays
Waveguide displays use a light engine and specialized lenses to project virtual content in front of the eye. Depending on the application, the wearer then looks at that content for a virtual screen application or through the lens to see their physical surroundings augmented by the virtual elements. It’s currently the dominant approach in lighter-weight, lower-cost AR devices.
As far as something like smart glasses go, Magic Leap is still pretty big and pretty expensive – and a fairly exclusive enterprise device. Campfire, also an enterprise-focused company, offers a smaller and lightweight headset that can offer VR as well as MR via the addition of a magnetic plate over the normally transparent lenses.
Micro-OLED displays essentially just put a screen in front of the wearer’s eyes. This approach has clear benefits in terms of image quality, brightness, field-of-view, and some other considerations at the trade-off of being more expensive and much heavier than waveguide displays. This is currently the dominant approach in VR devices – including the Apple Vision Pro.
Because the wearer can’t see through an OLED screen, AR and MR experiences are enabled through passthrough – which displays a video feed of the wearer’s surroundings which can then be augmented with virtual content. This is how virtually all AR/VR headsets – including the Apple Vision Pro – are able to offer both of those experiences on the same opaque screen.
An Extra Screen
There is one place where the benefits of a Micro-OLED display just don’t seem to justify the tradeoffs in terms of weight, cost, and power usage. That’s the giant screen on the front of the Apple Vision Pro that shows a virtual reproduction of the wearer’s eyes.
“I think it’s quite a useless feature, per se. Let’s put it this way: I think Apple just needed to put something there that would make the wow effect,” said Almeida. “They’re taking what I would call ‘The Tesla Approach’ of showing what they’re capable of and then saying ‘now wait a few more years, we’re now going to go into mass production, design something simple.’”
The fact that the headset is called “Pro” has led many to the conclusion that Apple is going to release a standard version of the headset, which might well do without some bells and whistles – like the virtual eyes.
“I definitely think that they should offer options with and without the front feature and I can almost guarantee most people would go without because it would probably be a good $500 or $600 cheaper, the battery would probably last a little bit longer,” said Almeida. “I wouldn’t be surprised if there was a version called the ‘Apple Vision Mini’ or ‘Apple Vision S’.”
Controller-Free Design
While the Apple Vision Pro has more screens than most VR headsets, it has fewer controllers. In fact, Apple is adamant about the Pro not having or needing controllers but getting along with eye and hand tracking as its only inputs.
“That’s one of the points where they’re definitely innovating,” said Almeida. “If there are a few hidden gems on the Apple Vision Pro, eye tracking and hand tracking are among them.”
I specifically brought up gaming as one use case where I feel that a controller is still necessary. Almeida had his perspective but toward the end of the episode, I also invited 3lbXR and 3lb Games founder and CEO Robin Moulder up from the audience knowing that she has an interesting perspective on game input.
Hardware Integrations and Third-Party Companies
A recurring point in the conversation was the room around the Apple Vision Pro itself for accessories, whether from Apple or from 3rd party developers. Almeida sees gaming and input in general as one of these opportunities.
“To play a game, having the feeling of a real-life controller in your hand is something that you need,” said Almeida. “I also think that they’re opening a path for haptic gloves.”
Almeida envisions a whole collection of different controllers for different kinds of games and interactions similar to that for headsets like VIVE. Earlier in the conversation, he had also mentioned the Apple Vision Pro’s battery life as one area that could be expanded through partnerships.
“If Apple is smart, they’re going to open the market to third-party companies for the existing hardware to expand over the existing options,” said Almeida. “In order to achieve more market and to grow as a company, Apple needs to invest in breakthrough technology and for that, they need to let third-party partners come to complement the ecosystem.”
Calling on Developers
Moulder had a different perspective on the Apple Vision Pro and seemed eager for the opportunity to develop controller-free applications.
“I am super stoked about Apple and from my perspective, there’s a whole lot to unpack,” said Moulder. “On the input side, the thing that I keep bringing up to people is that video showed the woman with her hand in her lap. (…) I’m looking at that thinking ‘If I could move my hands around in that kind of field-of-view, hand-tracking works a lot better.’”
Here we’re not talking about field-of-view in terms of what the wearer sees in the headset, we’re talking about what the headset sees around the wearer. Headsets have the ability to track the location of controllers no matter where they are but can only track the hands when the hands are in view of the cameras, which puts huge limitations on how effective hand-tracking can be.
“I don’t have to predict the location of where the hands are going to be in a bunch of nonsensical math just to make up for that limitation of the technology now,” said Moulder. “That’s really nice for us from the gaming perspective because that means we can lean into hand-tracking even more than we’re doing right now.”
Moulder said that this would mean “working with the user to train them” on more nuanced hand interactions, but it also sounds very optimistic for a controller-free headset that doesn’t feel like the 3DoF models of yesteryear. The wide range of the Apple Vision Pro does require a huge number of cameras, so it’s likely that other makers won’t emulate the move any time soon.
Parting Thoughts on the Apple Vision Pro
The Apple Vision Pro still remains something of a mystery to me. But, I have another perspective on where Apple might be going with a potential future product line built around it, and that’s exciting. I also have a whole new perspective on controller-free headsets in general, and that’s very exciting.
You can listen to the whole “XR Talks with ARPost: Episode 10 – Another Take on Apple’s Entrance” below, or on Spotify.
