Mixed Reality Has to Feel Real

Screen Shot 2018-05-14 at 12.41.53 PM.png
3D bathroom model in a workspace

3D bathroom model in a workspace

I used to have fights with my cofounder Dr. Julien about the quality of our technology. We'd set an extremely difficult goal, and his R&D team would meet it, but upon internal user testing, I'd fly into an unfair rage as to why it wasn't good enough for commercialization.

"It WORKS.... We did everything you told us to do", Dr. Julien would argue.

"It works, but it doesn't WORK WELL. How am I supposed to sell this? Have you considered the user experience? The user would never accept this", I'd holler back.

"What does WORK WELL, mean? Give me the specs! By the way, what we gave you is the state of the art in computer vision. What you're asking for has never been done."

He was right.

I couldn't give him a proper spec, and he was building bleeding edge tech right out of his academic research. I'd express my loss of confidence until Sonal, our other co-founder, would take me aside and tell me how unfair this was. I'd oblige and apologize. However, this cycle repeated itself too many times over the past four years to no resolution.

The Big Revelation

Last week, I attended a fireside chat between Dr. Barry Sandrew and Dr. Phil Lelyveld on augmented reality storytelling, what the mind will believe, and the state of the technology. Dr. Lelyveld explained that if you were trying to grab a mug off the table, but even 10% of the time your hands would grab nothing, you'd get extremely frustrated. Your expectation is 100%.


Just like that, he just told us what "work well" meant. This was the technical challenge for mixed reality to overcome to make the mind believe.

Here’s an old Shapetrace promo video starting at 1:00 that showed our technology working to align the real and virtual world. It’s a video so it looks perfect. Of course, it wasn’t.

Barry Sandrew on the left and Phil Lelyveld on the right.

Barry Sandrew on the left and Phil Lelyveld on the right.


It's not Easy to Trick the Mind

If you were trying to grab a mug off the table, but even 10% of the time your hands would grab nothing, you’d get extremely frustrated.
— Phil Lelyveld


For mixed reality to be believable, the digital assets need to act like real objects that interact with your physical surroundings. The objects need to be persistent in their location and take into account other people, objects, and interactions in the space. And this persistence must be 99.9% reliable, or the mind will say, "Oh that's not real" and the experience is disturbed.

If you've ever placed a digital asset in mixed reality using ARKit or ARCore, you will notice that its location may jump around slightly depending on the lighting conditions and features in the environment. Even then, slight frustration sets in, and your mind needs to build trust in the system.

For Shapetrace, our specification for the alignment between the 3D architectural plans and real-life was roughly 5-10 cm horizontal and vertical accuracy, only restricted by the quality of the sensors on the tablet. While this was an amazing technical feat, the great challenge was the user could still see the visual misalignment, and their mind would stop believing because it wanted better than 1 cm accuracy all the time. It's almost as if mixed/augmented reality has its own version of the "uncanny valley".

Design Experiences that Guarantee Persistence

Looking back, Shapetrace was designing and engineering a solution that couldn't guarantee object persistence at the scale of a building. So our advice is that in order to monetize to the mass market, mixed reality designers and engineers need to focus on delivering experiences that can pass the "uncanny valley of mixed reality". This usually means something simpler, no matter how much this hurts my stupid engineering ego.

It’s almost as if mixed/augmented reality has its own version of the uncanny valley.

It makes sense why so many augmented reality experiences continue to be rather short, simple, and more often than not, gimmicky. I'm thinking of marker-based triggers like the experience we made below for HBO Canada or even Snapchat's Face Lenses. For the most part these experiences are not overly complex, the object or interaction is persistent with its location, and it has limited interaction with the environment. Yes, the user may run into trouble identifying the marker or their face at times, but for the most part, the experience is smooth, anchored, and believable. We can guarantee a reliability of the experience so that a story is properly told. 

So if we were to make a logical prediction, it seems the applications will follow tracking technology as it evolves. So it starts with marker-based experiences, then move to simple feature tracking, tabletop, room-scale, multi-room scale, building scale, then world scale.

I guess Dr. Julien and I can finally have some resolution.

Here’s a marker-based experience we built for HBO Canada as a subcontractor to Patio Interactive. It’s simple, but at least it’s pretty reliable.

Created for HBO Canada More information at www.WestworldAR.com