In film, we’ve seen Ferris Bueller and others break the fourth wall by talking to their audience; in mixed reality, the walls disappear altogether.
Humans have always been storytellers. From the days of cave paintings and early spoken word to theater, cinema, and the internet, the experience has only become richer and more compelling. With each evolution, our experience as an audience has become more visceral, more personal, and increasingly sensory. From old-school movie theaters to fully immersive virtual reality (VR) experiences, our content has only become more convincing.
However, our interaction with this content is hamstrung by the limitations of screens, keyboards, and controllers. With mixed reality (MR) and its ability to bring digital content into the physical world, we are able to leverage gesture and voice to redefine the way that we interact with media in a more natural manner.
Mixed reality is the ability to contextually blend digital elements into your real-world environment. When visualizing mixed reality, think of a digital, interactive, 3D object that exists not on your computer screen, but as a hologram on your coffee table—one that you can interact with as if it is really there. While the technology is related, it differs from virtual reality, where the user immerses themselves in an entirely digital world that cuts them off from their surroundings via a wrap-around headset. It is also different from augmented reality (AR), where a layer of digital information overlays the user’s vision, such as through text notifications—or Pokémon. To break it down clearly, MR places virtual objects you can interact with into reality, AR overlays virtual information into reality that you interact with digitally, and VR supplants reality entirely.
The MR experience is currently delivered via a clear-visored headset that, from the wearer’s perspective, projects holograms into their environments. Today, the technology is mostly used in commercial scenarios: For instance, NASA’s Jet Propulsion Laboratory uses it to visualize the surface of Mars in true 3D using imagery captured by the Curiosity Mars rover, while Case Western Reserve University teaches anatomy to medical students with interactive, animated 3D holograms.
The innovations in this technology represent a much-needed paradigm shift in computing from a flat 2D screen to something far more immersive and tactile. Interaction with content is as simple as selecting digital objects with your hands and positioning them in your environment as desired, just as you would with physical objects.
Mixed reality is about making computing more personal, and many companies and industries have identified this space as the next big, mainstream computing opportunity. Magic Leap and Meta are currently developing their own mixed-reality offerings and, while the existing technology is cost-prohibitive for most companies and users, the goal is to make this technology affordable and broadly available in the very near future. In October, Microsoft shared its commitment to making 3D content—including mixed reality—available to everyone, and the company is actively working to reduce the barriers to entry, both financial and technical.
I work on the team developing Actiongram, which is a mixed-reality application for Microsoft HoloLens that enables creators without professional 3D or visual-effects experience to create computer-generated (CG) videos that convincingly blend the digital and the physical worlds. Users can easily incorporate the holograms into their environment and customize them in clever ways to create unique and sharable video content. Using traditional 3D methods, this could require years of visual-effects experience and proficiency with expensive software and equipment. Mixed-reality tools—replacing screens and menus with integrated digital content and a more natural interaction model—will empower users of all skill levels to easily create compelling and convincing scenes in an afternoon. Just as touchscreens enabled more intuitive and natural navigation of the smartphone user interface, MR will pave the way for the most user-friendly computing experience yet.
Mixed reality can also enrich our personal experience in truly innovative ways. Fragments is a mixed-reality game experience that not only brings digital characters into your living room, but enables them to sit on your couch as they speak with you. The game adapts to your individual environment and incorporates it into the story as it unfolds. RoboRaid is a game where the player must repel robotic invaders that barge through their living-room walls. As eye-tracking tech continues to evolve, companies will be able to create characters that can maintain eye contact with the user as they move about the room. And as mixed reality eventually intersects with artificial intelligence (AI), we can expect holographic content that reacts to users and its surroundings in truly remarkable ways. All of these developments work to personalize our stories, bringing them contextually into our lives.
While the technology is developing rapidly, we’re still in the very early stages of learning its full potential: We also need to learn to think and design differently in order to optimize the impact of the mixed world. We believe that the best way to unlock and discover this potential is to make it accessible, and get it into the hands of creators.
Working directly with our audience at an early phase of development is a great opportunity to design an effective tool. We provided Actiongram in beta to several social media influencers and filmmakers on platforms such as YouTube, Instagram, and Vine and have worked closely with them as they’ve harnessed the power of mixed reality in ways that we hadn’t previously imagined: They’ve held a dance party with Nyan Cat; they’ve had fun as a family, leveraging their children as acting talent to interact with a holographic T-Rex in a short film; and they’ve even used the technology to film a music video for rock band Miniature Tigers.
Mixed-reality technology is the platform that will drive the next creative renaissance in storytelling. Our hope is that in the very near future, it will empower anyone to help shape the future of creativity and storytelling.