We have conducted numerous usability testing for various clients and partners over many years, and to this day, we always get asked the same, and very important question: “How many users will be enough for testing?” or “What sample size is enough?”
100% of the time the answer to this question surprises them and it may come as a surprise to you too.
So how many test users is enough for usability testing?
The answer is five.
Yes, the answer really is that five test users sample size is enough for usability testing. To elaborate, let’s look into what usability testing is and why five users is enough.
What is Usability Testing?
Usability testing, also sometimes referred to as user testing or UX testing, plays a vital role in user experience (UX) design. This is the evaluation of the usability of a product by testing it with representative users.
The key here is representative users, these are real people who represent your target audience. Usability testing focuses on real users, and we study their interactions with the product, down to the micro-interactions.
For a deeper dive into the different methods and best practices of usability testing, check out this article on Complete Guide About Usability Testing.
Benefits of Usability Testing
Usability testing doesn’t just provide us with a deeper understanding of users, but it also allows us to discover opportunities in identifying problems, to improve the user experience and enhance conversion.
In addition, this form of qualitative research complements many forms of quantitative research such as A/B testing, data analytics, market survey, etc. It is helpful for providing context when we can see what is happening in the data analytics but we don’t know why, usability testing tells us why.
To learn more, we recommend these articles:
- Usability Testing 101: What, Why, How?
- Remote Usability Testing 101: What, Why, How?
- Best Practices for Remote Usability Testing
I can go on about usability testing but let’s focus on the question at hand.
How Many Test Users (Sample Size) Is Enough?
The answer is really five, and I’ll tell you why.
The main justification for the seemingly small amount of sample size of users is quite simply: ROI. Yes, the Return On Investment. The costs increase with each additional test user, this we already know, recruitment costs can often be more expensive than the actual research component.
However, the number of findings yield diminishing returns after five test users. This means that from the 6th test user onwards, individuals will start telling you what the first five test users already told you. The ROI drops after five test users, which is why five test users is enough for each round of usability testing.
You may be wondering how five test users could possibly be sufficient for a website that caters to millions or how to provide a suitable justification to top management when they question why only five test users are used instead of 500. Fret not, let’s address these questions individually:
“Our website caters to millions, how is five users enough?”
Answer:
In user testing, the focus is on website functionality and design elements to determine whether it’s easy or difficult to use. Therefore, the evaluation of the design element’s quality doesn’t depend on how many people use it. If there’s a pothole in front of you, you don’t need millions of people to tell you it’s there.
On the other hand, when deciding whether to fix a design/usability issue, it should depend on whether the feature is being used by a handful of users or majority of the users, so that design and development effort are better invest on delivering the highest positive impact to millions of users.
“But we have so many features, how would 5 test users be sufficient?”
Answer:
Covering all the features in a single usability test is impossible, regardless of how many users. We can’t expect users to sit there for hours upon hours to go through all the features. In fact, we recommend that each usability testing session takes up to 60 minutes, a fatigued user will not be able to focus and provide valuable insights.
In order to cover all the features, start by prioritizing the features you want to test in order of importance. Then select a handful of features that can be covered in an hour, and test with five users. Repeat again with another handful of features, with another five users.
Conducting several different tests and focusing on a smaller set of features each time will reap more benefits and insights compared to glossing over countless features and gaining only surface-level insights.
“But we have various target groups, how do we compile them in five test users?”
Answer:
It’s important to point out that five test users should be representative of each target group instead of the overall target audience. This, however, holds true if the various target groups are different to each other, such as an e-commerce site where there are sellers and buyers. The two different target groups use the product differently and it makes sense to test with two target groups such as five sellers and five buyers.
You could essentially get away with 3-4 users per group because certain aspects of the user experience tend to overlap between the groups.
“How are we doing to cover all the usability problems?”
Answer:
In time. The key is to iterate, select the most important features and conduct usability testing, make improvements and iterate. Whatever isn’t fixed right now will be fixed later and you’ll end up with higher quality & higher business value because of the iterations of testing.
Why You Only Need 5 Users
Credit: nngroup
Looking at this graph, it’s evident that zero users provides zero insights but we see that once we test with the 1st user, we have already learnt about almost a third of the usability problems.
With the 2nd user, a lot of what they do overlaps with the 1st user so the findings are supportive but because people are different, there will be new discoveries that we can observe from the 2nd user. This provides us with a bit more insight and supporting evidence.
Now the 3rd user will do a lot of things that we have observed from the 1st and 2nd user, they will provide a small amount of new data but it won’t be as much as the 1st and 2nd user.
As we keep going with more and more users, we actually learn significantly less each time because the same things keep repeating themselves. So the learning curve flattens out, there’s no need to test with the 20th user when they are going to repeat what the first few users have already pointed out. (You certainly can test with even more than 20 users if you have the budget, but we are taking into account practical business considerations and there is usually a limit of UX budget one can spend).
At which point, we can go back to the drawing board with the insights, make improvements to eliminate the usability problems and test again.
Why Not 15 Test Users Sample Size?
The curve does show that at least 15 users are required to discover all the usability problems in the design. So why do we recommend testing with five instead?
The answer is because of budget, instead of spending the entire budget on 15 test users at a time, it’s better to distribute it across several small tests. Conducting three studies with five users each will reap significantly more valuable insights compared to a single study with 15 users.
In the first study with five users, we identify 85% of the usability problems. We then take these insights to the drawing board and make the fixes. Afterwhich, we can test again with another five users (different to the 1st five users) to identify whether the fixes actually work, sort of a Quality Assurance check.
Furthermore, the 2nd round of testing serves to identify the remaining 15% of the original usability problems that weren’t found in the 1st round of testing. It also allows for a more in-depth study going into task flow and other fundamental structures instead of being blocked by surface-level usability problems.
When To Test With More Users
If you have various distinct groups of users, it would be beneficial to test with more users.
Taking the example above of an e-commerce site with 2 distinct groups of sellers and buyers, they have sufficiently different behaviour and so it’s important to test with people from both groups.
Bear in mind that all the users are human so there will be a certain extent of similarities when we observe the different groups. In addition, a large number of usability problems stem from the fundamental way humans interact with digital products, regardless of groups.
When it comes to testing with distinct groups, because of the overlap in observation, we can test with 3-4 users per group to cover the diversity of behaviour within the group.
If you are interested in running usability testing or if you have any questions, contact Netizen Experience or read more about our user testing services for more information.
