Research Needs Qualified Participants
In usability research, the qualifications of the participants are just as important as the qualifications of the researchers. Qualified participants represent the target users of the website or application by matching important demographical and motivational criteria. For example, ecommerce websites such as Amazon or eBay are used by shoppers of practically any age, gender, education or income level, and with virtually any interest. On the other hand, many sites target customers with specific interests or age groups, from those planning a wedding or a family trip to a theme park destination, to those researching insurance or needing to buy office supplies. And many applications are used by people with very specific needs and roles, from users of technical equipment or patients needing information on a drug to treat their condition, to professional tax preparers and shoppers wanting to locate products they researched online in a brick and mortar store.
The applicable criteria for participants in a usability study represent a range of age and gender, household income brackets, education levels, recent activities pertinent to the study, use of mobile devices, and other specifics. This is approved by clients and documented in a screener used by recruiters. Usability Sciences puts as much effort into finding participants who are representative of the target market or user group as we put into developing the tasks and questions that participants will be asked during the study. Quantitative research using online tools can capture the characteristics of actual users as part of an intercept survey while the user is clicking through the site. The captured data informs the criteria used to develop user personas and market segments, which in turn informs the screener used to recruit participants for usability studies.
We recently learned about research being done through Amazon’s MechanicalTurk.com site. As reported on NPR a few weeks ago, basic survey research is being done cheaply and quickly by people who earn a few cents answering survey questions or doing basic tasks such as locating and ranking URLs, tagging images or transcribing items from a digital image of a receipt. Workers, as they are called on Mechanical Turk, select HITs (Human Intelligence Tasks) when they have time and want to earn some money. They are paid for work considered satisfactory by the requester, often only a few cents per HIT. Some HITs require “qualified” workers who have previously shown that they can produce “high quality answers.” In other words, workers are qualified by their performance, not by their demographic or personal data, as is the case in usability studies. Although the NPR article says that the “population of Turkers is pretty representative,” they are self-selecting in that the worker chooses the HIT they want to do. The Mechanical Turk site doesn’t ask for personal information except for country of residence and certification that workers are over 18 years old. In fact, asking any personal identifying information about the worker is a violation of the Mechanical Turk policy.
This kind of research is clearly not about usability since it collects data without any information about the worker or their emotional state, other than that they are motivated to do a task for payment. Although participants in formal usability studies are also paid for their time—whether they give high quality answers or not, their participation is qualified by having met demographic and other personal criteria deemed important to the purpose of the study. They are humans, and their emotional reactions to websites and application tasks are of value as much as, if not more than, their mechanical actions of clicking links and rating search results.
-Judy Kistler-Robinson, Senior User Experience Specialist, Usability Sciences