Creating Magic

Working in AR/VR/MR

CLIENT: Magic Leap YEAR: 2016-2018 URL: https://www.magicleap.com/ ROLE: UX/UI LEAD

Magic Leap is a technology company within Mixed Reality/Augmented Reality and the biggest competitor to Microsoft HoloLens. In August 2018 they released Magic Leap One, their first edition hardware, and operating system (LuminOS), and apps that I had the opportunity and privilege to work on during my time at Artefact Group.

Designing without 'guardrails' is the hardest and most satisfying thing you can do as a designer.

What I learned working in Augmented, Virtual and Mixed Reality

OVERALL DESIGN CHALLENGES

• Create design experiences and solutions for OS parallel to hardware and architecture being built.
• Figure out and prototype within VR something that will eventually be MR/AR. (Big shout out to Artefact team who built and designed the prototyping tool we ended up using)

INDIVIDUAL DESIGN CHALLENGES

• Design an App store for applications and games, from the ground up.
• Design a Media streaming platform from the ground up.
• Contribute with design thinking and solutions for Lumin OS.

CLIENT

ARTEFACT TEAM

MAGIC LEAP TEAM

INDIVIDUAL CONTRIBUTION

Magic Leap

Jon Mann, Paul Hoover, Craig Erickson, Scott Ichikawa, Jon Garn, Daryl Tanghe, Mei Heish, Sabella Flag, Sam Baker, Spencer Davies, Patrick Mankins, Ryan Moeck, Jason Wiggin, Jason Stokes, Me (Andreas Kleiner)

James Powderly, Coleman Shelton, Tim Stutts, Alysha Naples, Isi Azu, Megan Geiman, Greg Tran, Lorena Pazmino, Tannen Campbell, Josh Rynne, Artur Adib, Chris Nesladek, Tim Zurmöhle ...

• Lead design of App Store on Artefact Side
• UX/Visual Design for Screens app
• Created MR/AR prototypes
• Created MR/AR hardware interaction design and flowcharts.

magic-leap-gives-world-peek-lumin-os.w1456

Renders and in-product shots of UI and UX I worked on for Magic Leap One

“Artefact shares our vision for the future of spatial computing. They provided the frameworks and prototyping tools that enabled our designers to iterate within a medium that didn’t yet exist. And their talented, multifunctional team helped us bring delight and a focus on the user to all phases of the design process.”
- James Powderly Director of UI/UX, Magic Leap

PROJECT SUMMARY

In 2016-18 I had the opportunity and privilege to work with the Magic Leap Design and Dev team to help create their Operating System LuminOS, along with a few of their core Applications. Unfortunately for this case study, I still can't show anything more than a few concepts, mock-ups, and explorations. However, I thought I would provide you with some of my personal best practices for designing in a Mixed Reality or AR environment.

If you want to read more about this project, here is a Case-Study from Artefact Group
https://www.artefactgroup.com/case-studies/magic-leap/

App Store interaction flowchart+site map for hardware controller.

Icon and Shortcut exploration for Magic Leap made in Adobe AE

Redlining of "interactive cells" and determination of Sticky UI for ML 'App-Store'.

Spacial UX / z-alignment with the Application Screens

“Magic Leap was a big sandbox of possibilities. You constantly had to take two steps forward and one steps back with a lot of prototyping. Designing without 'guardrails' is the hardest and most satisfying thing you can do as a designer, and I am very grateful to have got the opportunity to do so with Artefact and the Magic Leap Team”
- Andreas Kleiner(Me) when designing for Magic Leap

What I learned working in MR/AR

DESIGN FUNDAMENTALS

1. Where in the room is the user?

2. Due to hardware restrictions; How do you create an immersive experience with limited FOV (Field of View)?

3. How will the graphics be projected? Since black can't be used, as it is simply transparent since you are projecting with light. How do you get around that?

4. Is the content following the user (user-centric content), or is it placed within the space?

5. Accessibility in MR/AR is extra hard since you can't account for what will be in the background, make sure to use high contrast colors, and if you want to make sure text and UI always will be visible you can put it on a white background.

6. If you do use a lot of white backgrounds to tackle legibility and accessibility issues see bullet above, make sure to not overdo it, since too many surfaces in on the UI takes away from what's so special with Augmented and Mixed Reality, the open space.

7. Shadows are hands down the one thing that makes augmented items/CG look 'real' in space, and can help the user understand where an object is in z-space.

PROTOTYPE AS OFTEN AS POSSIBLE

To better understand how the end-product would look we needed to find a way for placement in z-depth, sizes, and even how colors looked when projected on different surfaces. Since there was no great way to prototype directly in Hololens, VIVE or Oculus, our team created a Unity-based mockup tool called StoryBoardVR. It enabled us to bring in PNGs, place them in x, y and z space and custom HDRI to set the scene we wanted. Then build a step-by-step, non-interactive prototype. This was crucial, as it was time-consuming to continuously build prototypes directly in Unity. StoryboardVR allowed us to create quick and dirty prototypes that were easy to iterate on and incorporate feedback.

Unity prototype with animations

UX Prototpye of POV using Adobe After Effects

StoryboardVR prototype in VIVE

DESIGNING ICONS & SHORTCUTS IN MIXED REALITY

I wanted to create an application icon and shortcut that felt part of the environment and came to life when targeted. The main inspiration for my icon and shortcuts was Nintendo Amiibos where physical figurines can be used to unlock characters in the game and special features.