Over the years, we have conducted numerous usability tests for a variety of clients and partners. A question that consistently appears is: “How many users are enough for testing?” or “What is an adequate sample size?”.
Long story short, a good baseline is 5 – 10 users. However, to set the right number relevant to the context of your usability test, we need to consider several factors.
In this article, we shall explore briefly about usability testing before we get into how to determine the right sample size.
In this Article
What is Usability Testing?
Usability testing, also known as user testing or UX testing, is a crucial aspect of user experience (UX) design. It involves evaluating a product’s usability by testing it with representative users.
The key here is representative users, these are real people who represent your target audience. Usability testing focuses on how real users interact with a product, even at a micro-interaction level.
Usability testing provides a deeper understanding of users and helps identify opportunities to improve user experience and enhance conversion. It also complements quantitative research methods by providing context to data analytics, A/B testing, and market surveys.
To learn more, here are some articles for further reading:
So How Many Users Are Enough for Usability Testing?
While we did say 5 – 10 users is a good baseline, the truth is, there’s no one-size-fits-all answer to the question of how many users are “enough” for usability testing.
The idea of testing with five users, popularized by Nielsen and Landauer (1993), has had a major impact on how we approach testing. The logic is that you get diminishing returns: each additional user gives you less new information, and testing costs more.
As the number of users increases, the number of new findings decreases after testing with the first five users.
Source: Why You Only Need to Test with 5 Users | NN/g
In fact, research shows you can uncover a large chunk (average around 85%) of usability problems by testing with just those first five people.
However, that said research also found out that the minimum percentage of usability problems found with five people was 55%. This highlights that while five users often yields good results, the outcome can vary.
While five is a solid starting point, especially for early-stage testing, it’s not a hard-and-fast rule. A more reliable approach is to aim for 5-10 users, as testing with 10 users will help you discover even more (at least 82%) of the problems.
| Number of Participants | Minimum % Problems Found | Mean % Problems Found |
| 5 | 55 | 85.55 |
| 10 | 82 | 94.686 |
| 15 | 90 | 97.05 |
| 20 | 95 | 98.4 |
| 30 | 97 | 99 |
| 40 | 98 | 99.6 |
| 50 | 98 | 100 |
The table above shows the number of problems found relative to number of participants
Source: Beyond the five-user assumption: Benefits of increased sample sizes in usability testing
What More Should I Consider?
Now that you know 5-10 users is a good starting point, you’re probably wondering, “Will it be enough for my situation?” To figure that out, ask yourself the 4Ws and 1H when considering the appropriate sample size for your usability testing:
Why?
- Tactical vs strategic goals:
If it’s for tactical reasons, aiming for quick fixes and immediate improvements (like addressing urgent bugs), smaller, focused studies might be enough.
If it’s for strategic reasons, informing long-term direction and understanding broader user behavior, you’ll likely need larger samples to get a more comprehensive view.
When?
- Formative vs. Summative Evaluation:
For early-stage formative testing (like checking your work), smaller samples work for quick iterations.
For later-stage summative testing (like a final exam), larger samples may be better for a complete evaluation.
How?
- Moderated vs. Unmoderated Testing:
Moderated testing (like a guided interview) gives you rich data, potentially needing fewer participants.
Unmoderated testing is more hands-off, so you might need more participants to get enough data.
- Number of Testing Rounds:
To gather enough data with limited time/budget, you might test more users per round.
But iterative testing (multiple small tests) is often a more efficient approach. It allows for early issue detection, fix validation, and in-depth user behavior exploration.
Who?
- Distinct User Groups:
When testing products with different user groups (e.g., buyers and sellers), test with representatives from each. Each group has some shared behaviors but also unique needs. Testing 3-4 users per group is generally enough to capture the key differences, but your total sample size increases accordingly.
What?
- Impact of Features:
High-impact features (like payment flows or core functions) need larger sample size for thorough testing to avoid major problems.
Low-impact features (like a “contact us” link) can be tested with fewer users. Think: major detour vs. sidewalk crack.
- Likelihood of Occurrence of Issues:
For major problems (potholes), a smaller sample may suffice.
For subtle/rare issues (needles in a haystack), a larger sample is needed.
- Complexity of the Product:
Complex products with many features and user paths have more potential problems. To find a good chunk of those problems, you need more testers to try out different parts of the product. Imagine testing a simple app vs. a complicated enterprise software.
- Testing Scope:
Keep your sessions to around 60 minutes as users get tired. To test everything you need, prioritize features, test a small set at a time with 5 users, and repeat. This focused testing approach gives you better results than trying to test everything superficially.
Consider the context of your usability testing when determining sample size.
Conclusion
So, there’s no single right answer to the “how many users?” question in usability testing. The five-user guideline can be helpful to get you going, but don’t stick to it blindly. To make sure your testing is effective and gives you useful results, consider the factors we’ve discussed and use iterative testing. This approach will help you get the user insights you need.
If you are interested in running usability testing or if you have any questions, contact Netizen Experience or read more about our user testing services for more information.
