Augmented Reality Guide

(2019 Edition)

What does AR stand for?

AR stands for Augmented Reality. If we go to the Merrian-Webster dictionary looking for a definition, this is what they return. “An enhanced version of reality created by the use of technology to overlay digital information on an image of something being viewed through a device (such as a smartphone camera)”. For us to gain a better understanding, we can dissect said definition on its core parts. It is an enriched version of reality first. It is enabled by technology second. In most of the cases, by a Mobile device, what we call context-aware technologies. This enhanced version of reality is usually interactive.

Through their devices, the users are empowered to interact with the extra layer of information. It is embedded within the physical environment.   A slightly different way of understanding the meaning of AR would be to approach it as an alternative user interface paradigm. Instead of using a device to directly interact with information. The device generates data (objects, sounds, visuals, haptics). It places them in our environment, for us to interact with them. It does alter the perception of the environment. Either adding elements to it or masking it, for the user to interact and experience it in an alternative way.

Augmented Reality VS Virtual Reality

We want to take our previous definition of Augmented Reality as the starting point. From this point of view, we can find the differences in the nature of both experiences. While Augmented Reality adds or partially overlaps a different reality in the user’s environment or reality. Virtual Reality completely substitutes the user real environment, with a computer-generated one. Augmented Reality is doing it differently. It is not trying to replace users reality.  It is helping him understand or enjoy it with a distinct layer of information. We have seen that said layer of information can be composed of visuals, sounds or even haptics. 

On the contrary, Virtual Reality implies a complete immersion in a different reality. The physical world disappears, replaced with another reality. The goal is to convince the user’s senses, as realistically as possible, that the virtual world is its new reality. It has different implications and points to consider before deciding if VR is worth it or not. We will review those further down. 

Milgram's Reality Virtuality continuum schema approaches the difference between Augmented Reality and Virtual Reality

Augmented reality VS Virtual Reality, what are the differences?

It is also possible to set apart from each other by the way they are delivered. Right now both interactive experiences differ in the devices used. Virtual Reality requires headsets, powered by mobile devices or not. Mounted in the head of the user and usually accompanied by handheld devices to let the user interact with the virtual world. Augmented Reality on the other side, is widely delivered by mobile devices. 

The fact that Augmented Reality is powered in the vast majority of cases by mobile devices is one of the leading causes of the growth of the AR industry and its adoption. Be it a smartphone, a laptop or a tablet, every home already has several devices. Most of them likely ready to deliver an interactive experience right away. The user only needs to receive a piece of software, paired or not with an AR ready object, to interact with that experience.

There is exciting progress in the industry to take it one step forward. Most notable one would be Microsoft’s Hololens or Magic Leap glasses, delivering an Augmented Reality through the projection of holograms that can be interacted with through a new device. The generation of this Mixed Reality experience attempts to create a new category. We will go through the differences and advantages later.

How does Augmented Reality work?

We have previously defined what AR stands for. So how does it work? Using a camera, the device needs to be able to “understand” its environment and aggregate digital content to the video feed. It needs to be able to do it seamlessly. The computer has to join the camera feed and the virtual elements as smoothly as possible, to generate the illusion of those elements being actually there.  For our perception to be tricked into this expansion of the physical world, those are the basic problems the computer has to deal with. It is a Computer-Vision problem. We need to split the problem and the process to solve it in smaller steps to understand better how AR works. Let’s break it down, in a very simplified way.

  1. Camera(s) feed to ingest the surroundings and transform it into digital information. The camera and sensor(s) (GPS, gyroscope, compass,…) provide the device with information of the real world, the physical surroundings of the user. 
  2. The computer process that data feed. Most usually, a feed of raw images. It does use its CPU, sometimes a GPU, flash memory, RAM or whatever configuration the AR device has. All to “make sense” of the data. The goal of this process is for it to precisely understand what are the user surroundings, location, orientation, physical objects existing in such space. 
  3. With all that processed information, it does generate a 3D map of the space captured in the feed. Now the device is “aware” of its surroundings. It has a 3D mesh of the physical world.
  4. Using this information map, the AR device will place and overlap virtual computer-generated imagery in the scenery

It all happens live. Every change or movement is registered and the virtual addition is adjusted accordingly. The augmentation takes place in real or almost real-time (some devices introduce a really small delay of 50 seconds to give it time to process the data and form the new overlapped image). This would be a very roughly summarized version of how an AR device works. To be more precise, a mobile device. If we are talking about more advanced interfaces, like Virtual Retina Displays (VRD) or Smart lenses, they work in slightly different ways, dealing with different sets of problems. But the basics remain the same. Remember this next time someone asks you how AR works!

History and a brief timeline of Augmented Reality

What are the main milestones that have taken us from the origin of Augmented Reality to the actual state of the art solutions? We are going to trace a brief timeline of events for those of you out there willing to know. It might be an opinionated view of a not so linear process, but we hope you do not take it into account. We have also mixed the most important events with some less significant ones, but somehow meaningful. Frank Baum portrait, AR History1901 (yes, that old) Author L. Frank Baum mentions in his fantasy novel “The Master Key”, a device called a “Character maker” that the protagonist receives as a gift. Once he wears it, essentially a pair of hyped glasses, they overlap over people an indicator of their personality or characteristics. Sounds familiar?

 

 

 

1962 Filmmaker Morton Helig files a patent for the Sensorama Simulator. Morton was a really interesting personality of the time, with a totally not conventional life (you can read more about him in this article, “Forgotten genius: the man who made a working VR machine in 1957” or this fantastic piece from Engadget on the Sensorama Simulator). He began researching machines able to simulate how human experience surroundings years before filing the patent. The machine played a 3D film along with stereo sound, aromas and wind to create an immersive sensory environment. The main idea was layering artificial stimuli for the senses to augment a simple cinema presentation. Here you can watch a video of Morton Heilig’s Sensorama Promo, from the late 1980s. 

Myron Krueger Videoplace, first AR interactive environment1974 Myron Krueger creates Videoplace while working at the University of Wisconsin-Madison and the University of Connecticut. American computer artist, he was pioneering the creation of full-body interactivity and virtual environments. He pushed forward to create environments the user could engage with. For the very first time, even if in a very rudimentary way, the users can interact with an artificial virtual environment. He coins the term “Artificial Reality” to describe what he was trying to achieve. Here you can watch a video of Myron Krueger’s Video Place:

“The Videoplace used projectors, video cameras, special-purpose hardware, and onscreen silhouettes of the users to place the users within an interactive environment. Users in separate rooms in the lab were able to interact with one another through this technology. The movements of the users recorded on video were analyzed and transferred to the silhouette representations of the users in the Artificial RealityOffsite Link environment”   Tom Caudell develops AR head mounted device1992 Researchers Tom Caudell and David Mizell coin the term “Augmented Reality”. While working at Boeing, they were assigned the task of finding a viable alternative to costly diagrams and marking devices the workers used to guide the process on the factory floor. They proposed to replace the expensive boards used back then to the assembly of an aircraft. A head-mounted device would display the needed plane schematics and project them on multipurpose, reusable boards. The worker would “receive” or wear the instructions and they could be modified efficiently by a computer system.

Tom Caudell, DavidMizell. Boeing's head-mounted Augmented Reality device

Boeing’s head-mounted Augmented Reality device

1994 Ronald Azuma (Ph. D. at UC. Berkeley, team lead at Intel now) published a paper describing the first AR system that generated a convincing illusion of real and virtual coexisting in the same spot even when the viewer rotated and translated his or her head. Exploring outdoor AR systems, it opens the door to a myriad of applications. Previous approaches to AR had limited applications due to portability constraints. Proving the potential to overcome said constraints served as the origin for the AR technology we have now.

“AR connects users to the people, locations, and objects around them, rather than cutting them off from the surrounding environment”, Ronald T. Azuma

 

1999 Hirokazu Kato developed ARtoolKit. Working for NAIST (Nara Institute of Science and Technology ) and releasing it under the University of Washington HIT Lab. It is one of the first SDK, libraries oriented towards the creation of Augmented Reality applications. Its biggest contribution to the community is solving two hard to solve issues. One is the viewpoint tracking. Calculating the real camera position relative to the environment via markers (squares that ARToolKit recognises and tracks in a video stream) or natural features in the feed. The second is the interaction with virtual objects. Positioning 3D objects or elements, in real-time, over the environment is possible now. Without this accurate registration, the illusion that the virtual objects exist in the real environment was severely compromised. It was released as open-source, later acquired, and re-released as an open source again. 

 

2009 Squire Magazine starts experimenting with AR in their printed magazine. Robert Downey Jr. introduces you to the issue. They add AR content, interactive, changing depending on the time of the day, as part of their efforts to engage their readers with an enriched experience. A half-dozen pages featured AR technology. It was smart and one of the first AR experiences applied to the marketing sector.

 

 

 

Volkswagen MARTA app, AR history2013 Volkswagen releases MARTA, an AR app that helps mechanics view parts of Volkswagen’s XL1 through augmented reality. Developed as part of Volkswagen research on Virtual technology, It offered instructions on the hybrid diesel’s mechanical operations. The app simplified working on the car and gave users more information through specific labels of the individual parts (to speed up the repair and service was the ultimate goal of the app).

 

 

 

Nintendo's Pokemon AR game, Augmented reality history2016 Pokemon Go. Based on Nintendo’s mid-90’s video game, the release of the AR game was an unprecedented phenomenon. It probably deserves a further in-deep essay only devoted to it. Declared The World most important game, with the public taken by surprise, wondering about what would exactly be the secret behind its worldwide popularity. The game impacted public space, social interaction and bringing to the masses a new form of experiencing the world. The underlining simplicity of the gameplay, great execution and a “fitness side” wired into it, made the game an instant hit. By the way, Pokemon was already the most profitable media franchise. Ever. $96 Billion in revenue since its inception.

 

2017 ARkit, part of AR historyApple launches ARKit Released with iOS 11 at Apple’s Worldwide Developer Conference. It is proof of Apple’s big bet on AR, as the new kit enables developers to bring AR to iOS devices. Since the release, it has been included in new iOS versions. There are over 14 million augmented reality apps built using Apple’s ARKit.

 

 

 

IKEA's AR place App2017 IKEA launches Place the IKEA Place app lets customers see exactly how furniture items would look – and fit – in their homes. The AR app enables the experience, it automatically scans the environment and resizes the 3D models based on the dimensions of the room.

“IKEA Place makes it easier to make buying decisions in your own place, to get inspired and try many different products, styles and colours in real-life settings with a swipe of your finger. Augmented reality and virtual reality will be a total game-changer for retail in the same way as the internet. Only this time, much faster.” Michael Valdsgaard, Leader Digital Transformation at Inter IKEA Systems.

 

google launches AR live instructions on google maps app2019 Google adds Live Augmented Reality to Google Maps. Adding it to its mapping service to guide directions and signalling makes it one of the first widespread uses of AR.

“With a beta feature called Live View, you can use augmented reality (AR) to better see which way to walk. Arrows and directions are placed in the real world to guide your way. ”

 

 

Augmented Reality VS Mixed Reality

There is a lot of confusion around the terminology the industry uses. From the consumer point of view, it is not easy to discern the differences between VR, AR and Mixed Reality. For good reason, we are talking about a technology field that has improved by leaps and bounds. It is hard to catch up, even for early adopters. 

We have described before what are the main differences among these technologies from diverse angles. The most obvious difference would be in the way MR delivers the experience. You need a headset, while Augmented reality mostly delivers by using a mobile device. Through further examination, we can point out another difference. Although both technologies are aware of the environment, and both use the geometry of the surroundings as a canvas. 

 

Mixed reality promise is to create an even more immersive experience. On one side it does a better job synchronizing the virtual and physical world. By wearing the headset, the sensorial experience is coupled with the virtual one. Movement is translated, and experienced by the user. 

If you have not been around the industry for a while, it does not seem so radical. Previously there was a fair amount of challenges limiting a development like that. Being the main one the latency between the virtual image and the physical perception. When latency is introduced, over a certain threshold, it does mess with our brain’s perception. Inducing an undesired response, dizziness. The advance in processing capacity (among others) enabled MR. 

 

Another distinction, derived from the nature of the MR experience, is the level of interactivity. The headset frees our hands to interact with space and its elements. It also has lead developers and creatives to envision experiences where moving and interacting with the physical elements present in the user’s environment becomes key,  translating into real-time to the virtual layer  

Who are the main contenders in the industry of Mixed Reality now? There are two distinct competitors getting most of the media attention. The first one is Microsoft’s Window Mixed Reality project, Hololens. Hololens is Industry oriented. The Microsoft company has already launched with partners like ThyssenKrupp or Renault. While the Windows Mixed Reality is oriented to serve the consumer market. 

Microsoft Hololens, a promise on Mixed Reality

Microsoft Hololens, a promise on Mixed Reality-ready for Business

The second one is Magic Leap. The company is presented as the very cutting edge technology development in the Mixed Reality space. Backed up by giants like Axel Springer Digital Ventures, Google or the Alibaba Group, they are bringing innovation to the software and hardware needed to take Mixed Reality one step forward. They become the first private Unicorn in the field. Its promise is an even more immersive experience without the total substitution of the user’s reality. They are moving away from the “simple” concept of Mixed Reality, positioning themselves as the company behind the first “spatial computer”. A term they have coined to describe a computer that lets you see and interact with virtual objects around you. 

Magic Leap one, creators edition. Mixed reality for developers and designers.

Magic Leap one, creators edition. Mixed reality for developers and designers.

Although we will not lose sight of their developments in the field, we highly advise keeping an eye on any other strong MR competitors that might come up in the following six to twelve months. It is a growing industry and there are a bunch of future giants flying in stealth mode now. Interesting times to be in this industry


Existing AR Platforms

Who is winning the Augmented Reality wars? Actually, that is a much more difficult question to answer than it seems. When started writing this guide to AR & VR, we needed first to think who would benefit from reading it. What is our goal? We want our users to understand first what is the core of our product, Augmented Reality technology. So we have decided to talk about what are the existing AR platforms. Not from a particularly deep technical point of view, but what are the available options on the market. If you are a developer looking for a comparative of Augmented Reality platforms, or an investor trying to get a grasp of the field, you would probably end up disappointed. But if you are someone interested in the topic and want a guide to know where to start in AR/ VR / MR technology, then this is for you. 

 

The second question that came up was, what is an AR platform? We have seen that Augmented Reality is the denomination we use for a particular technology. There are several implementations or companies developing customer-facing AR solutions. So it is going to be important for us to figure out first what is it and what is the point of it. We can describe an AR platform as a set of tools that help create that kind of singular experience. It is important? in the technology sector, the one defining the standards and able to attract the most creators does actually have an advantage and a saying on how the technology evolves. Moreover, when we are talking about the early days of an increasingly fast-paced market. 

 

Let’s say you want to start digging deep into Augmented Reality, there are several forthcoming options for you. In no particular order; Facebook Spark AR studio is Mark Zuckerberg’s company bet.  It promises to help you create and distribute Augmented Reality experiences for the FB ecosystem (Facebook camera, Instagram, somehow limiting if I dare to say it). It was presented in the F8 conference and it is part of a ten-year roadmap plan (including a patent on AR glasses using technology similar to Hololens, targeting the transition to Mixed Reality at some point down the road). It has a growing community.

Facebook presenting Spark AR at F8

Facebook presenting Spark AR at F8

A great contender is Google ARCore, the Search company answer to the market demands and part of their moonshots (although lately, they have moved part of its initiatives from their moonshot branch to the core company, as proof that the google glasses project is far from being dead ). Their promise? To “help build new augmented reality experiences that seamlessly blend the digital and physical worlds. Transform the way people play, shop, learn, create, and experience the world together—at Google scale”. Since its launch during the Google I/O, with every edition come updates and new features. Some are pretty unique and leveraging the strengths of the giant, like “Environmental HDR”. A complex process that enables ML to recognize the lighting origin in the live feed scene, adding better lightning to the virtual objects. 

ARCore, the AR platform by Google

Example of what can be done using ARCore, the AR platform by Google

The stealth company, Magic Leap’s has Lumin. Magic Leap had some awful critics to their launching offering. But that only applies to their first demos, last iterations have been received somehow more optimistically. Someone has described it to me like “a very well-implemented AR SDK, with lots of features”. Although we have yet to approach it, it looks like a solid alternative, particularly if you are looking to create something for Magic Leap devices.

 

Magic Leap’s Lumin OS, AR platform

Microsoft’s Mixed Reality Toolkit is the last we will mention.  we just want to give an overview of some of the available options. Like the others, it includes its own particularities and features. It does support not only experiences for the Hololens but also other devices like the Oculus. Honestly, there are a lot of platforms out there. The key takeaway from this part would be that there are a diverse set of solutions and toolkits our there. They are helpful and they came with signature advantages and logical tradeoffs. Do your due diligence before embarking yourself in the AR creation world.

Microsoft Mixed Reality Toolkit, Unity, machu Pichu example of application

Microsoft Mixed Reality Toolkit, Unity

What is the future of Augmented Reality?

Saying that no one knows would be an understatement. But we can guess what is coming. As a market, there is little doubt of fast growth and potential. No matter if it is Augmented Reality technology-oriented to the consumer or B2B AR. The question is more about the timing. When will we see widespread adoption?

There are multiple reports, providing different figures, making it hard to pinpoint a precise timeline for the industry progression. Mordor Intelligence, a technology research company, declared the AR market had an estimated worth of 0.35 billion dollars by the end of 2018. It forecast an annual growth rate of over 150% in the bracket between this year and 2024. Statista has an AR/VR market size projection slightly different, 160 billion U.S. dollars by 2023, being the consumer market the major driver (way ahead of the manufacturing market). 

Augmented Reality Market Size Analysis

Projections for AR market size

CCS Insight forecasts that worldwide shipments of virtual and augmented reality devices will reach 14.7 million units in 2019, with a sell-in value of $2.9 billion — an increase of 76%. By 2023, the research firm expects 75 million virtual and augmented devices to be sold worldwide, hitting a market value of $13 billion.

A more quantitative approach to the question, yet actually interesting is the one from this Wired Magazine’s article. The authors decided to survey a bunch of developers in the AR/VR industry, exploring what are their perceptions, what are the trends in the field, etc. One key takeaway from the article is that the main drivers for the industry (at least last year) are Gaming, education and training. 

Another important read is that regardless of what would be the particular device that will get the most market share, there is a shift in the tech involved. Devs are not concerned anymore with making the users sick. Their focus is moving slowly away from “how can we provide the experience” to the experience itself (what content can be created and what are the applications of AR/VR/MR).

FastCompany took a similar path with this article, “Augmented Reality when? We asked Magic Leap, Facebook, Google, and more”. Asking well-known names in the AR, VR and MR where the technology is moving towards (even XR, or Extended Reality. The industry is so young that it is hard to grasp all the changes and new designations). The “visions” they share are obviously biased towards their companies roadmaps, but still appealing for someone willing to know more. 

As a summary, we could say that the market is definitely growing at an incredible pace. There are proofs in the number of companies generating AR experiences, the number of applications, the number of sectors that are introducing Augmented Reality as a way of getting a competitive advantage. As the growth is really fast, forecasting where the ceiling is at is almost impossible. 

We know we have not yet arrived at a tipping point that will lead to mass adoption. But we know we might be not too far away from that. We do not even know what will be the next leader in the AR field. Although we know that the main technology giants have started moving towards taking the lead position. The introduction of AR in most sectors will happen eventually. We have seen a solid progression in multiple fields. No matter if we are talking about Consumer or B2B, it seems there are a lot of opportunities for disruption by the right introduction of Augmented Reality. 

The AR market for the consumer (or the MR market) still needs to resolve some questions. Some are obvious, like what company will be able to deliver the most compelling AR to the market. Not the most flamboyant, but the one able to deliver higher value, with adaptable functionality and lower friction. Other questions are less obvious. What are the challenges on privacy that are inherently attached to this technology? What would be the impact on human behaviour in a society where AR is extended and part of our everyday life? (for those interested in this particular aspect, there are already some compelling research around the matter)

We would like for you to tell us about what else to write about

This is the end of our complete guide to Augmented Reality (for not experts). We hope it helped you get a better understanding of the AR technology advantages, endless potential, market and challenges. We are preparing more in deep articles around our industry, in particular about Augmented Reality in the education field. We create amazing AR educational products for space enthusiasts. It is part of our mission and core values to educate about the technology as well. We would like to know also what would be appealing for our users to read about. We have created this form (click to vote & comment). Please, let us know what AR / VR questions or topics you’d like to read about here!