Why Recruiting Can Make or Break a Research Study

The most frequently overheard comment during any usability study is: “Huh. I would never have guessed that!” You might think that is the most common quote because we test a lot of sites that are confusing, but you’d be wrong. That quote comes from clients in the observation room. No matter the type of business or website, we hear that quote at least once per usability study. Why? Because no matter how long you’ve worked on the website or how much you know about the product or business: you are not your target audience. Neither are your children, your co-worker, nor your neighbor.

Over the years, we have had several well-meaning clients seek out “user feedback” from those around them and then be frustrated or confused that the feedback is conflicting, irrelevant, or flat out wrong. The truth of the matter when it comes to user research is: everyone’s opinion is not equal, or even valid. Asking someone who has and will continue to have their health insurance plan options selected by their employer to look at a health insurance exchange website and “weigh in” isn’t going to be very insightful.

Everyone has opinions, but not every opinion is a source of insight for how to meaningfully assess or improve a user experience.

User research only works when you’ve identified the right users and asked them the right questions. Otherwise, you’re wasting your time and money. The single most important step toward a successful research project is recruiting the right participants for your study. Talking to the wrong people will lead to a lot of useless data points. Talking to the right audience will lead to meaningful insights.

So how do you make sure you get the right people to participate in your research study?

1. Start with a solid definition of “who”

A common answer to the question ‘who is your audience?’ is: ‘everyone’. This answer is easy, but it’s also wrong. Even a site that serves a really broad audience -like Amazon.com, for example- is not for everyone. Amazon.com is not a site that is intended to be accessed and useful to young children or low-income adults who do not have a bank account or credit card.

Why does your site exist? Who is coming to the site? What do you want them to do? There might be multiple answers to these questions. But in order for a research study to be effective, the answers need to be specific and prioritized.

When defining ‘who’- follow this pattern:

Our site provides <description of the service, information or product your site offers or promotes> for <audience>.

Even with multiple audience segments, you’ll want to clearly define who they are and what makes them unique. And you’ll want to prioritize the audiences that are most helpful to answer your particular research questions.

When it comes to what you want to know about your website, you need to understand who can best inform you. And chances are that you have a specific audience: people who need to sign up for a service you offer, people looking to complete a specific task, people at a certain life stage who need to learn about a topic. Your ‘who’ might not be easy to find, but they exist and they are not ‘everyone’.

2. Create a research screener

The research screener is a document that includes your research goals or questions, your audience requirements and the ‘disqualifiers’. The screener will be used to recruit the right participants.

Start with the basics like gender, age and income and move to more complex factors like who is the primary decision maker in the household or, in the case of testing on specific devices, someone who primarily uses their smartphone to access and use a site like yours. Starting with the basics allows the recruiting firm to weed out unqualified participants quickly and to spend their time talking to and narrowing in on the right recruits.

Get input on the screener from people who know the business and the target audiences. We always review and refine our screeners with our clients. This allows us to validate that our criteria are going to result in the right research participants. In this process, we are able to figure out if we’ve missed any key criteria, like significant age ranges or income levels.

3. Find the right recruiting partner

Once you’ve got a good handle on your audience criteria and characteristics, you have to find the right partner to help you find those people. Recruiting is definitely one of those tasks where DIY is a DON’T. Getting the wrong participants is a huge waste of time, effort, and budget, so it is absolutely worth the money to use a recruiting firm to make sure this step is done right. We always partner with a recruiting firm because they are experts at what they do; and have the right processes and protocols in place to make sure that the characteristics defined in the screener match the characteristics of the people who show up on the day(s) of the study.

The only cases where we would not engage a recruiting firm are if we are testing an Intranet or specific B2B portal, where the users are the client’s employees or when the client wishes to do research with existing customers or members. In these cases, the client is best suited to find the appropriate research participants.

4. What if the audience is hard to find?

There are several challenges that come into play when you’re asked to test with a particularly narrow or specialized group of people. How do you find them? Most research recruiting firms have databases of possible participants who represent the population at large. Narrowing in on a particularly small slice of that population poses a few challenges and requires a few accommodations.

• First, you’ll need to allow more time to recruit. When your recruiting criteria are narrow, you’ll need to talk to more possible participants in order to find those people who meet your narrow suite of criteria.
• Second, you’ll need to spend more to recruit them. Since we all know time is money, expect to pay more to recruit these participants. If you’re recruiting highly skilled individuals, like physicians or engineers, plan to offer a larger incentive than normal.
• Finally, you’ll need to get creative with scheduling. You may need to schedule research sessions during evenings or weekends to ensure that certain professionals will be available to attend.

Want help with recruiting? Talk to the Crux Collaborative team —we’re happy to help you understand, define, and prioritize your audiences for your next user research study.