Augmented reality often loses the battle for the spotlight to its cousin, virtual reality. While both technologies make us think of science fiction movies, I’d like to show you just how accessible AR really is.

Not only accessible, AR is becoming an integral technology in more and more industries. Maybe the most uplifting fact is that the tech is still evolving at a fast pace. It’s in a “wild west” phase of trial and innovation.

I don’t want to just gloss over such an interesting subject, so I’m going to give you now a quick rundown of what this article is about:

What is augmented reality (AR)?
Augmented reality on mobile

The weak points of mobile AR
TappTips for mobile augmented reality

The evolution and expansion of AR
Building a better reality

What is augmented reality (AR)?

Augmented reality is the technology that superimposes different types of data onto the real world, mixing real life objects with virtual ones. Imagined over a century ago and prototyped more than two decades ago, AR has long been hailed as the way of the future.

Unfortunately, its evolution was slow going at times and things seemed bleak in those first years of development. One of the first ever augmented reality systems was developed in 1992 for the US Air Force. It needed a special helmet and a full upper-body exoskeleton to work. It was cumbersome, expensive and it couldn’t run Pokemon Go.

Robocop - What is augmented reality and how to apply it to mobile by tapptitude
What augmented reality felt like in 1992

Things have changed dramatically in the last decade as technology caught up to our imagination. Augmented reality was embraced by multiple platforms, including mobile.

Among the first new devices were the wearable augmented reality glasses. This device had its ancestor feature – they were worn on the head, like normal glasses. Through their lenses the user could see virtual objects that looked pretty real for that time.

Even more noteworthy is the great software evolution that augmented reality was undergoing: the announcement of the first frameworks dedicated to the technology. Among them was Qualcomm’s Vuforia, released in 2011, which was coming to multiple platforms. The Vuforia developers proved the power of mobile augmented reality. That acted as the starting point to develop native AR libraries for mobile OS’s.

A gif of AR software built with Vuforia

Seen as a thriving technology, Augmented reality started to gain traction and the tech giants scrambled to develop native frameworks for multiple platforms. Apple unveiled in 2017 their new and powerful AR framework, ARKit for the iOS. Google founded ARCore, the Android rival of ARKit. Wikitude continued its development, and EasyAR, Maxst, DeepAR became other powerful AR competitors for bringing the best AR experience.

Augmented reality on mobile

Both ARKit and ARCore had humble beginnings. You’d run into small bugs and the apps were huge power hogs. With time, the frameworks became more stable and helped developers deliver amazing experiences with just a few lines of code.

Now more and more businesses worldwide are enhancing user experience by implementing AR-powered interactions. We’ve had the pleasure of working on such projects ourselves:

A case study: Artfare

Simply put, Artfare is an eCommerce platform that connects artists and collectors, facilitating the buying and selling of artwork.

Wireframes for the Artfare app - What is augmented reality and how to apply it to mobile by tapptitude

Right now, the app is localized to New York City and that geographical focus means that we were able to integrate more features without bloating the app. One of those is, of course, augmented reality.

The problem

Art is highly subjective. Artwork has personal meaning to us, it gains emotional value. That’s why buying a painting, for example, is a lot more difficult than buying a nightstand.

You don’t always know where art galleries are and even if you do, who’s to tell if you’ll find something that speaks to you on a deeper level?

Maybe you want to know a thing or two about the artist before making a purchase. Maybe you’re not sure if a certain painting goes well with your curtains because you can’t see them together without buying. In that case, what are you to do?

Larry David looking confused

The solution

You open Artfare. The app has a map with all art galleries in New York and, more importantly, it shows you the paintings displayed at each artist studio or gallery. So you know where you’re going and what you’ll see.

Once there, you can scan the paintings with the app and get more info right on your phone. No tags or plaques required.

Lastly, you can use Artfare to browse artworks and superimpose them on any wall, anywhere. No more second guessing if the piece will be a good fit for your home.

How artfare uses AR

As you might have guessed, placing virtual artwork on your walls means augmented reality in action.

Besides getting an idea about the esthetical fit, another big advantage is the fact that you don’t have to strain to visualize the size of the painting in relation to your home. Gone are the days of reading measurement and then busting out the ruler to see them in real space.

The weak points of mobile AR

Augmented reality as a whole is making great strides, and that includes mobile AR but things aren’t perfect. They never are, are they? When building an AR experience around a mobile device, you run into three major hurdles:

1. Accuracy 

The sensors used for detecting real-world anchors are liable to bias (as in “this table is actually the floor”). This means that the precision of placed virtual objects can decrease in real-time. Even the newer, sharper mobile device sensors are not without flaw.

2. Battery

The most energy-consuming hardware on any mobile device are:

  • orientation and direction sensors
  • image recognition system,
  • camera
  • GPS
  • display

Every single part of this list is an indispensable part of AR. In other words, keep a charger handy when playing around with AR.

BMO from Adventure Time shutting down

3. Accessibility

When the AR frameworks for both OSes were released, people celebrated. Then, they learned the catch-22 of these frameworks: their device coverage. The need for a powerful processor makes AR apps available only on a limited number of devices, usually those released after 2015.

TappTips for mobile augmented reality

Well, you have to take the bad with the good. Those problems are at least partially hardware-related, but there are plenty of opportunities (and threats) that lie in UX and software development.

Let’s focus on what you DO have control over and cover the best practices of mobile augmented reality development:

Plan for the mobile devices’ constraints

An easy mistake to make is to focus on a single scenario and to build around that. For example, tablets have bigger screens so they offer a better AR experience, but a “tablet first” mentality can lead to a lackluster smartphone experience.

When designing the user interface, focus on flexibility. Consider the different screen sizes, then model and scale the UI accordingly. Here are to common aspects:

Virtual object size – one size does not fit all! When designing an object (or button) think in percentages. The subject should occupy X% of the screen, regardless of screen size.

An example of problems caused by screen differences when using augmented reality - What is augmented reality and how to apply it to mobile by tapptitude

Methods of interaction – are you supposed to hold the device in one hand, both or none? A phone might fit nicely into one hand but a tablet won’t. Buttons may become hard to reach on large screens or hard to see on small ones, so build an UI that offers the same ease of use for all types of devices.

Counter the counterintuitive

On the one hand, augmented reality apps thrive when the user is in motion, looking at virtual sprites from different angles and places. On the other hand, people are used to using two-dimensional apps, i. e. the kind that usually don’t care if you’re moving or not.

This disconnect can result in users not using AR to its full capability and coming away with a subpar experience. It’s not their fault for not knowing, it’s your fault for not telling them. Your objective is to bridge the gap between functionality and habits.

A straightforward way is to simply tell the user. Have tips pop up on the screen and explain how to use the app. Yet, this method lacks subtlety, it could hinder immersion, progression and even come across as condescending.

No, think of intuitive cues to guide users in the right direction, literally. For example, your virtual object could go on the move, making the user follow. Other visual effects, like a “glow” coming from a direction offscreen or even an arrow, can help. Another cool idea is to use audio signals like sound effects from the right/left, but that requires the use of headphones so it’s a less common scenario.

An example of problems caused by lack of intuitive design when using augmented reality - What is augmented reality and how to apply it to mobile by tapptitude

Get creative! Find ways to guide your audience in a way that would work only for you. That’s an excellent way of making a memorable augmented reality experience.

Plan for the space constraints

At the moment, diving into augmented reality isn’t seamless. The first, unavoidable, step is scanning the room in order to define the “action space”. That is also known as world discovery.

Not all users know about this step and you shouldn’t expect them to. Offer a clear guide on world discovery (how to do it, why it’s necessary). The advantages are that you make sure the user doesn’t abandon the app before the experience really starts and that you can more easily manage their expectations.

Another constraint could be the size of the space available to the user. Depending on the type of experience you’re developing, you might want a large open space, or close proximity to a wall, or neither. Generally speaking, tight spaces are the enemy.

Austin Powers - What is augmented reality and how to apply it to mobile by tapptitude
Suboptimal augmented reality experience

Tailor the app to work well in as many conditions as possible. When you run into limits, and you most likely will, explain to your users what is required of them to improve performance.

Some of the worst conditions your AR app will run into are: low-light environments, tight spaces, rooms where the colors of the walls, floors or furniture are very similar. These types of factors take their toll on the sensors of mobile devices.

An example of problems caused by an inappropriate space when using augmented reality - What is augmented reality and how to apply it to mobile by tapptitude
Why do you even have a white, unfurnished room?

As I said, you may know all this, but your audience might not. Communicate these facts to them and minimise the risk of them giving up in frustration.

Show and tell

Sometimes digital objects can seem so real, we can’t tell what’s really there or not. That sounds great for mobile AR apps but it’s a double-edged sword. If the objects you create don’t stand out in a meaningful way, they fade into the boring, real-life background. Then, users ignore them.

Or the opposite could happen: they think something real is actually part of the augmented reality experience.

Infomercial woman hitting wall with hammer on accident - What is augmented reality and how to apply it to mobile by tapptitude
Oh, the wall’s real…

Other times, the app and the users don’t agree on what constitutes a viable plane. That happens when the user places an object on a plane (a wall, for example), but the app doesn’t recognize that plane as such. As a result, the object hovers without being anchored and is liable to move in unintended ways or simply disappear.

Technically, these two negative scenarios are different. But they have the same solution!

Use contrasts and highlights strategically. When placing an object, lighten recognized planes and darken everything else. If you want the user to interact with the object, give it a highlight, an animation, something to grab attention. You can also do all that through text pop-ups at the right moments.

An example of problems caused not differentiating virtual objects from real ones when using augmented reality - What is augmented reality and how to apply it to mobile by tapptitude

There are millions of ways of telling the user to look at something. Brainstorm ideas, try them out and see what works and what doesn’t.

Evolution and expansion of AR

Augmented reality is making waves in industries like manufacturing, telecommunication, and energy – where the workforce is usually thinly spread over remote areas. In this kind of situation AR is used for communication and learning.

In addition, educational and healthcare organizations rely on AR and VR as the most engaging and easy-to-use technologies for final users. As an example, healthcare companies are creating AR tools for the operating room, while students experience interactive classes using AR.

Example of using augmented reality to teach concepts of chemistry - What is augmented reality and how to apply it to mobile by tapptitude
Much easier to render than realistic water physics

Moving forward, AR development will bring out many new features, which will lead to:

  • Developers creating even more complex interactions;
  • Businesses finding new ways to engage with their clients through augmented reality.

Building a better reality

The tech world we live in is always changing and innovating. Lately, we’ve been getting new technologies so good that they become a must-have on the market. AR is one of those.

We constantly see things like “Is AR the next big thing in…?”. That applies to gaming, health tech, education, sports, manufacturing, commerce and many more industries. Competition is extraordinarily fierce on the cutting edge but AR’s star is still rising and I’m sure we’ll see more amazing developments in the future.

The appeal of AR on the mobile platform is so strong, many users try apps just to experience the AR features.

Marty McFly being attacked by hologram shark - What is augmented reality and how to apply it to mobile by tapptitude
Wholesome augmented reality fun

Our need to discover and learn has never been as strong as it is today. AR helps fulfill this need, in an immersive way. It brings both fantastic and realistic representations of real world objects in a half-virtual world.

 

A special thank you to Alex Iftode and Erika Kramarik for helping me write this piece and to Sabina Stanescu for making the awesome visuals!

 

Doru Cojocaru
Doru Cojocaru iOS Developer

In love with innovation and technology.

Read other articles