Point, Click, Tap, Swipe, Wave and Speak

A proliferation of screens doesn’t just mean you have to design one digital experience flexible enough to work across all of them… each introduces a new way of interacting that requires consideration of different actions and gestures.

(Thanks to Kayla Knight from Smashing Magazine for the below diagram.)

Augmented Reality Check

With new technologies like MS Kinetic X-Box 360 that detects movement and can sense when you are in a room, pointing and clicking with a mouse may go the way of the dinosaurs.  By integrating the MS operating system with gesture-based analysis Microsoft opens up a whole avenue of opportunities for the impaired to utilize the web and native applications.  By identifying sign language and translating it into text or the opposite, taking speech and utilizing it to navigate and complete tasks, usability issues could become a thing of the past for handicapped users.  We see real value for voice-navigated applications, and this won’t be the first time we encounter them.  Companies already use voice recognition to route calls to call centers and Siri on the iPhone is doing more and more as users define what they want and demand it from Apple.

The next big trend we are seeing is augmented reality.  Users have already become familiar with it in mobile applications where mobile and tablet devices give information to the user on anything from directions to food, ratings and nearby restaurants.  There is some research being done to display real time data on the lenses of your glasses, panels in the subway or kiosks in the mall.  There is ongoing research to show data on your contact lenses eliminating the use of hand held devices and screens entirely.  Imagine having reminders of your meetings for the day display on your glasses or your contact lenses when you walk into your office. Or having the history of the Bay Bridge pop up as you walk past it. This is the promise of augmented reality, a technology that overlays information from the Web and other sources on the real world.  This would totally remove size limitations, the user could use gestures to shrink or expand the size of the data field, point or wave to retrieve or dismiss the data.  Pattie Maes’ lab at MIT, spearheaded by Pranav Mistry is working on a wearable device with a projector that paves the way for profound augmented interaction with our environment.  “SixthSense” technology would overlay the world with interactive data also making it possible for the user to control the size of what they are looking at with hand gestures, retrieve information with a gesture and interact with users like never before.

How consumers are influenced to use applications, and how technology changes the way consumers interact with various interfaces will impact design.  Hot spots on a page take on a whole new role when body gestures are used to access application points.  Not to mention voice activation and voice navigation.  Intel has just announced it’s intention to release new software that is shifting to “one based on perceptual computing where devices will take on human-like senses to perceive the user’s intentions” utilizing gesture integration, facial and voice recognition, and bringing “augmented reality to life.”

Tap, Talk and Share

Tap to share images with Samsung Phones, and ask Siri on your iPhone how to get to the closest Starbucks.  Instead of unlocking a mobile device, going to the browser and typing in Starbucks, users can simply navigate with their voices.  This accessibility will help many of the impaired and the impact is staggering.  However we have to understand how navigating by voice the user doesn’t land on the Starbucks site for locations, Siri bypasses everything and goes directly to Google or the Apple Map application and plots the locations.  How can Usability counter ease of use, when applications are built to bypass anything you design and get the information in another manner entirely? We have to build a roadmap for functionality that can commandeer or work with or around applications that can potentially hijack the user’s experience.  We strive to stay ahead of trends and think of new ways to treat information to lead consumers down the happy path to buying a particular product or perceiving a brand a certain way.

(An example of a responsive website, The Boston Globe, displayed on multiple devices)

As a User Architect I  work to put Brands ahead of the eight ball building applications that are first of all “responsive” to show a mobile version of your website and identify when to build apps that complement the online brand without overwhelming it.  I work diligently with partners to identify trends so your company’s brand won’t fall flat on its HTML5 face. I work to identify the need to change partners and make alliances to go to market socially or even for mass email campaigns.  I can plan your company’s digital strategy in the social space, and can identify when to pull back the reigns when too many entities are “cluttering” up your online image.  With so many partners we have to utilize analytic applications that help us identify things about your customers to make your web presence more nimble than ever.  Identify how your users are utilizing your application and making it smarter and therefore less clicks to get to the information they seek.


Touch, Swipe and Zoom

Before we go too far down the road of alternate navigation we must address the very real concern of the many varied new touch enabled tablets and mobile web devices being delivered in November for the Christmas buying season. IFA, one of the worlds largest consumer electronics trade shows, attracting 150,000 people this year, highlighted over 60 new devices made to replace laptops and embrace Windows 8 and touch screen technology.  For this reason alone we have to reconsider how your website or online application works and how a native touch screen environment is going to effect it. Apple is unchaining the iPad to keep up with the plethora of devices planned to launch in November designed specifically for Windows 8.  Touch screen devices will be in everyone’s hands by the end of 2013, and a lot of these devices are cheap and will be snapped up by the mobile users looking for bigger screens but portable devices.  Barnes & Noble’s “Nook” now boasts Internet connectivity with a 7” and 9” horizontal display topping out at $300.  So it’s very likely going to be a popular Christmas gift.

Information Overload

Building a design that takes interaction in mind is often considered mind boggling to the Brands wanting to compete in the web and mobile space.  How many screen sizes are out there?  What devices and operating systems are being used to access the web and now my website?  Will my website work on that device or operating system?  We make it our jobs to know what’s emerging and try to build and anticipate the user needs.  By anticipating these trends and not limiting your website to one fixed design for one particular platform your audience grows, therefore your customer base can access your services.

Trends are showing a downturn in PC and laptop sales, while tablet sales are trending higher as employers realize they may not need anything more robust than a tablet or an iPad for their employees.  Some employees are simply bringing in their tablets and iPads and insist on using them in the workplace.  More users save data up to “the cloud” and forgo buying PCs and laptops.  MS has tackled all these new security issues with Server 2013 and a large contingent of security applications for tablets and mobile devices, but this too affects what users access and can see.  Network policies that restrict access to websites still sit in place, email is still monitored, and security features still have an effect even on Mac desktops, laptops and mobile devices.  So the business world is adapting, and they are in a place never before documented where processor speed isn’t king, ease of use and how small and light the tablet can be is key.  Looking ahead at 2013 we see more device display sizes than ever before, so we want to be sure we build applications with the knowledge that touch, swipe, zoom and scale will be the norm and not the outliers.  Menus that users can touch, applications where they can zoom, swipe and even talk to are looming in the future.

So what three things do you take away from this article?  PCs are dying, mobile and tablet sales are up, your users want touch screen and mobile apps, I say we give it to them.  Plan ahead and get your digital strategy in place for touch screen and mobile applications, be ready for the crowd that will come online in 2013 with a web application ready to do business and your sights set on Augmented Reality.


  1. Microsoft hopes new tablet will cure falling PC sales by Charles Arthur The Observer, Saturday 28 July 2012
  2. Smart Customers Abandon Stupid Companies by Bruce Kasanoff Posted on Bloomberg BusinessWeek on April 13, 2012
  3. OEMs Introduce Innovative New PC Designs for Windows 8 by Nick Parker – Windows Team Blog Aug 31, 2012
  4. The Next Big Thing(s) in Tech by David Daw of PCWorld
  5. Computer Sales Statistics – Statistic Brain *Source Gartner, International Data Corporation
  6. Smart Contact Lenses Could Bring Augmented Reality to the Next Level – by Jason Kennedy of PC World January 14, 2011
  7. Intel: Low-Power Processors to Fuel Future of Mobile Computing Innovation posted by Intel PR in News Stories on intel.com
  8. Microsoft hopes new tablet will cure falling PC sales – by Charles Arthur – The Observer July 28, 2012
  9. Responsive Web Design, A List Apart
  10. Responsive Web Design: What It Is and How to Use It Ref. Smashing Magazine – By Kayla Knight
  11. The Big Web Show #9: Responsive Web Design, 5by5 Studios
  12. How to Use CSS3 Media Queries to Create a Mobile Version of Your Website, Smashing Magazine
  13. Pattie Maes and Pranav Mistry demo SixthSense – TED Filmed February 2009