HCIAs promised, this is the second installation of my series on my two most exciting emerging technologies showcased at AWE 2014.  In Part 1, we discussed RGBD cameras and their potential impact in the world of enterprise augmented reality.  Today, we’ll be looking at wearable displays through the same lens.

To avoid duplicating much of Barry Po’s wonderful post on the “Five things you need to know about ‘smart glasses’ in the enterprise”, I’m going to pick a single topic and do a deeper dive.  Namely, how do we make applications intuitive and easy to use when displayed on a wearable device?

Human Computer Interaction

One of the big advantages of the tablet form factor in enterprise AR is that it is mobile and that the interface is both obvious and intuitive.  The downside is that you’re left holding the display in your hand leaving you unable to perform many tasks without putting the device down.

Adopting a wearable display and going “hands-free” does appear to be the logical next step, but it raises some interesting questions.  How does the user interact with the system?  They no longer have a keyboard and mouse, or touch screen in their hands.  If you still need to use your hand to perform an interaction, can we really call it “hands-free”?  There is a distinction between hands-free viewing and hands-free interacting.

The current wearable devices provide various modes of interaction, touch pads, pairing with another device, voice commands, and gesture controls.


The most popular means of interaction for wearable displays comes in the form of a small touchpad, either on a separate handheld device (BT-200) or on the arm of the glasses (Glass, Ora).

Having a separate handheld device is maybe the most familiar method for someone who has never used a wearable display before.  The BT-200s have such a device which is often held in the palm of one hand using the thumb on the touchpad surface.  This will move a cursor about the screen and allow you to tap, long press, and swipe.  This is awfully similar to the touchpad on many current laptops.

The issue here is that you’re not actually hands-free, even for viewing.  To perform any task requiring two hands, you will have to set down this touchpad.  One potential solution is to fix it to the users forearm and use the opposing hand to interact with it.

The other popular touchpad option is generally located on the arm of the glasses.  This has the luxury of already being in a fixed position and doesn’t need to be set down for two handed tasks.  However, it tends to be a long and narrow touch area, limiting the users to a few simple gestures such as a tap, long press, or a swipe to navigate the application.

This is a severely limited set of interaction possibilities which puts the onus on the application developers to ensure all the features are easily discovered and invoked.  In fact, this is why Glass developed the Timeline and Card system over trying to use a standard desktop-like interface to launch applications.

In either case, these interactions are not hands-free.  If you are in the middle of a maintenance procedure you may have to set down the tool or part you are holding to tap the touchpad to proceed to the next instruction.  The touchpad may eventually get dirty or greasy and eventually cease working until cleaned.  In other scenarios, the user will be wearing gloves and will not be able to interact with the touchpad at all.

Connect to another device

Another option is just to connect your smartphone or keyboard via Bluetooth, Wi-Fi, or even wire directly to the glasses themselves.  The BT-200s wired touchpad is an example of this, but it doesn’t have to be a touchpad.  It could be touch screen, keyboard, mouse, or any kind of customized hardware.

There is a large advantage when your wearable display is controlled by a secondary touch screen, like a smartphone.  It means that that wearable display doesn’t need to show any of the interaction UI.  All the buttons and dialogs and navigation items can be displayed on the connected touch screen.  Your wearable view only shows information you need at that very moment.

Of course, this still isn’t hands-free interaction and carries most of the same pitfalls as the touchpads.  It will still get dirty and you’ll either have to fix it to yourself or set it down when you need both hands for your task.

Hand Gestures

Most wearable displays also provide a built in video camera.  This allows for the use of gesture interactions as inputs to the system.  Similar to the gestures on a tablet’s touch screen, but instead you move your hands in the space in front of the glasses.

meta-001Although I didn’t actually get a chance to play with them at the conference, Meta’s offerings appear to be the only wearable display that come with hand gesture support out of the box.  That said, software packages like OnTheGo’s Ari will do much of the heavy lifting for you on any other Android device.

Since your hand no longer needs to touch the device, this solves the problem of your touchpad getting dirty and ceasing to work.  It may also work while wearing safety gloves as long as they are fairly form fitting.  However this still isn’t hands free interaction.  You will still need at least one hand free to perform the gesture.

Further problems arise since there are only a handful (pun intended) of gestures possible.  These include things like, a thumbs-up, open hand, closed hand, pointing, and swipes.  Like the touchpad on the arm of the glasses, it is a severely limited set of interactions but compounds the problem since the interactions are obscure.  Do I use a thumbs up here, or closed hand?  This non-intuitive gesture vocabulary will require either significant up front training on how to use the system or will require a lot of in application prompts reminding the user what interactions to perform.

There is also a problem with false-positives.  Consider you want to perform a left swipe.  So you move you hand from in front of you towards the right, and a right swipe is registered!  Some of this can be mitigated by selectively choosing which gestures to support at what times, but that’s even further limiting the number of ways the user can interact with the system.

Voice Command

Since that famous BMW AR maintenance video, voice command has been considered a good means to interact with wearable displays.  Most of the devices I’ve seen thus far have microphone input so they could potentially handle them.  Unlike the things we’ve looked at thus far, this is a fully hands free interaction solution.  Both of your hands can be occupied and you can still interact with the system.

Natural language processing has come a long way, but it’s still far from perfect.  Users will still be limited to a small vocabulary of phrases – things like “next step” to proceed.  A lot of thought will need to go into which phrases to use.  Too short or too common, and the user may accidentally say them without meaning to.  Too long, and they become clumsy to say and hard to remember.  Also, like the hand gestures, there isn’t always an intuitive phrase for the action, which may require up front training or require in application prompting.  Further, what happens in loud environments?  Yelling commands over the background noise of a factory floor is certainly not ideal.

Head Gestures

Most of the devices have internal gyroscopes and accelerometers built in.  These are exactly the same as the ones in tablets and smart phones that can register how you tilt and shake your device.  It means that you can nod or shake your head to provide input to the system.

The nice thing here is that it is completely hands free, it will work in loud environments, and has a very intuitive interaction vocabulary – nod for yes, shake for no.  The problem is, that everything needs to boil down to a yes or no question.  This is very limiting to the types of applications that can be used.  Also, as with the hand gestures, there is the possibility of false positives where the user’s natural movement inadvertently registers as an interaction.

Bringing it all together

Clearly there is no silver bullet for interaction with wearable displays.  Surely the best solution for most applications will involve many, if not all of the modes of interaction mentioned above.  Perhaps you use a paired smart phone to select and start the procedure, then use head gestures to proceed through the steps.  Perhaps you use hand gestures to show information on a specific part and a voice command to start a phone call with an expert off-site.

A few years ago, we published a paper discussing the available form factors for the use of AR for Maintenance, Repair, and Overhaul (MRO) performance.  The conclusion at the time was that tablets make sense for now, but perhaps wearable displays will in the future. Wearable technology has evolved in leaps and bounds since then, is now the time to adopt?  Honestly, I’m not about to say “Absolutely!” but we are certainly getting closer.

In the meantime, there are still a lot of problems to solve. These are some of the many things that I and the rest of the team at NGRAIN spend our days thinking about.  Keep checking back to see what we come up with!