Mixed-Method Approach to User Test Virtual Assistants

Mixed-Method Approach to User Test Virtual Assistants

What does the rise of the Digital Assistant mean for the businesses and industries that are now forced to compete in this Internet of Things (IoT) environment?

In 2008, Marvel’s Iron Man showed us Tony Stark’s house of the future, run by his virtual assistant, J.A.R.V.I.S. The ever-present voice was able to answer queries, complete tedious tasks, and streamline the superhero’s life. !

In 2019, we still don’t have Iron Man suits, but the virtual assistant has become a mainstay in many homes. Amazon’s Alexa is widespread. Contenders such as Google Home and Apple’s Siri, are growing more and more capable. Not limited to the speaker in your home or the device in your pocket, the digital assistant is expanding to be ever-present. “Cars and trucks and bicycles, sure; all your home appliances, switches, bulbs, and fixtures; even your clothes, shoes, and jewelry. They’re all coming online, and Amazon wants Alexa in all of them.” According to Priya Abani, director of Amazon’s Alexa voice services, “You should be able to talk to Alexa no matter where you’re located or what device you’re talking to. We basically envision a world where Alexa is everywhere.”

Consumers love this. Amazon notes that Alexa is in the homes of “tens of millions” of people. But what does it mean for the businesses and industries that are now forced to compete in this Internet of Things (IoT) environment? Consumers are looking to control more and more with their voice, from kitchen appliances, hardware, starting their car, opening garages and door locks. This technology is becoming a differentiator and significant value-add, and companies that ignore this trend are more likely to get left behind.

“We basically envision a world where Alexa is everywhere.”

But it’s not enough to slap a couple of shoddy voice commands and hope for the best. One of the key reasons these digital assistants are so successful is because of how intuitive they are to the user. First and foremost, products have to be usable to be successful; success drives everything.

Here’s how we helped a company validate and improve their entry to this marketplace:

We were contacted by a company that manufactures and sells kitchen appliances. Their internal market researchers did a sophisticated amount of data analysis to determine there was a substantial market for the addition of voice controls to their line of products. “But,” they said, “we’re not product researchers.” So they turned to us for help.

This research was part concept validation and part usability. Though demand for voice control was apparent, it wasn’t clear what the implementation would look like. The team had a list of “user stories” they assumed customers would want to accomplish with the various appliances in the kitchen, but needed confirmation. Through our research, were were able to help identify which areas needed to be developed, and which were less important. This contribution to the product roadmap helped focus development efforts and allowed the client to prioritize issues and remove unwanted features from the roadmap. There were some features that customers expected as “nice-to-haves” but were not critical. Since the team couldn’t deliver at a threshold of quality they were comfortable with; these features were scheduled for version 2.0.

Our research consisted of some one-on-one sessions, as well as research pairs. Conducting usability sessions with two people (in this case, mostly married couples), allowed for rich, dynamic sessions. The participants played off each other, and it helped to keep a reality check of what people say they do vs. what they really do.

Because this was a concept validation, the products with voice control had not been fully developed. As such, we had to go “old-school” and use a person-behind-the-curtain approach to simulate voice control. When the user spoke a command (e.g., “Set the oven to 450°”) then a researcher in the back would affect this (e.g., manually changing the temperature display). This simulation proved incredibly cost-effective, as it allowed our client to identify potential changes before development, saving both time and money in updates and fixes.

Rather than deep dives into process flows, we found it useful to present rapid-fire tasks, to help ensure that users were “in the moment” and not second-guessing themselves. (Having a spouse or other person in the session as well, also helped people “be themselves.”) This was crucially important, as the off-the-cuff commands and requests were more representative of actual user behavior. Furthermore, many of these tasks were repeated, sometimes with slight variations, to see if and how their sentence structure would adapt over time. By conducting this research, our client was able to understand and prioritize the development needs. Early-stage research helps ensure that decisions are not made in a “data vacuum” and ideas have been adequately validated by actual consumers. We also have a second round of research planned to test actual appliances in a practical setting, when everything is further developed. This second round helps catch any oversights, ensure that development has addressed the learnings from round one, and validate that the product is ready to go to market.

Your products and services have to be usable to be successful. Again, success drives everything. But success can’t be measured only by, “Is it available via Alexa?” it has to be, “Are users successful via Alexa?” And if your customers are successfully able to interact and engage seamlessly, they get to feel a little bit more like Iron Man.