Did you know that a whopping 53% of dating app users lie on their profiles about what they do for a living and how much they earn? In other words, people aren’t just filtering photos, they’re filtering who they really are. And dating is not the only area where this happens.
When filling in a screener survey, eager applicants might exaggerate or, in the odd case, lie to secure a spot in your research. Or leading questions might prompt them to answer in a certain way. Or they simply get confused and skew the results by picking the wrong option – only for you to realize the mistake in the middle of the interview.
Don’t worry though: screener questions are not a minefield to navigate. But there are a couple of things you should keep in mind to make sure that you capture the right participants. Here’s a checklist to help you avoid the most common pitfalls.
Recap: what’s a screener and why is it key to UX research success?
A screener is the last step that stands between you and your ideal research participants. It’s a list of questions to winnow potential testers down to those who can add real value to your research by giving feedback on the quality, performance and flaws of your product. Meaning it takes the guesswork out of the product development process and puts users, along with their struggles, needs and goals, at the heart of it. The result? A boost in user satisfaction and loyalty, a stronger brand, a cut in support costs and less resources spent on redesign. Asking the wrong questions – or asking the right questions in the wrong way – puts all that at risk.
1. Demographics vs. psychographics: know the difference
As a general rule, questions about testers’ employment status, job title, gender or age have no place in a screener. In PingPong, demographic and geographic filters are applied in an earlier step so your survey only pops up in the inbox of users with the right demographic parameters.
Screeners are about mapping out users’ behaviours, attitudes, interests, opinions and beliefs, aka psychographic traits. This type of segmentation not only tells you how candidates live and what’s important to them but also gives you an idea of how they relate to your product.
That being said, remember that certain user characteristics like marital status, country of residence and educational background might change over time. If these filters are highly important to your UX research, you might want to add them to your screener questionnaire.
2. Easy does it: keep screeners short and answers clear
As a rule of thumb, try to limit your screener to five essential questions. A longer survey can easily leave you with a narrow participant pool or fatigue respondents to the point of dropping out. It’s also a good idea to strip your survey of acronyms, abbreviations and jargon, even if they’re widely used in your industry. Plus, get rid of unnecessary embellishments that mislead instead of inform.
Whenever you can, go for single-choice questions. Instead of "Do you commute a lot on an average day?" or "Do you live far from your parents”, ask “How far do you live from your parents?”. Then give options such as: “I live with my parents”, “0-20 km”, “21-100 km”, “101-500 km”, “501+ km” and “Other”. Never skip the last one – or its equivalent. Otherwise candidates might be left to select the next best answer and mess up the results.
3. Don’t shoot yourself in the foot: CATA questions
A word to the wise: if you’re asking candidates to choose multiple answers in a check-all-that-apply type of question, don’t just give “Required” and “Incorrect” options. If testers are expected to meet, say, three qualification criteria out of the ten listed, only those who’ve successfully picked the right ones will be shortlisted for your research. That’s ambitious at best and near impossible at worst. Use “Optional” if there are multiple acceptable responses but not all of them are required.
4. Leading questions: don’t create an echo chamber
Mask the desired answer as much as possible. Obvious “right” answers will tempt respondents to tell you what you want to hear instead of what they really think or do or who they really are. Use neutral language, avoid yes-no questions and don’t be afraid to add dummy questions, if needed. For example, rephrase “Do you drink more than two cups of coffee a day?” as “How many coffees do you drink on an average day?” and provide answers such as “One cup (disqualify)”, “Two cups (disqualify)”, “Three or more cups (qualify)” and “I don't drink coffee (disqualify)”.
5. Screening the screener: four questions to ask yourself before finalizing your survey
- Does any of the questions or how they’re worded reflect your own preferences or opinion?
- Does the survey take into account both positive and negative feedback testers might have?
- Did you set the right qualifying criteria for every answer?
- If your research involves critical and confidential information, did you ask participants if they’re willing to sign an NDA?
In sum, putting together a screener is not a step to save time or thought on, no matter how eager you are to get to the actual research. Bonus tip: before you start, make sure you spend 10-15 minutes visualizing your ideal test participant and listing 5-7 of their key characteristics to stay focused. The good news is that by keeping to these pointers, you won’t miss out on a single one of them.
Need more advice or real-life examples? Check our blog or our video guide to creating research in PingPong. Have more questions? Shoot.
