After conducting thousands of usability test sessions with clients and partners in the past, we still find people asking the same questions, for example:
- Is usability testing the same as UAT (user acceptance testing)?
- Do I need 100 or 500 users for usability testing?
Hence, we wish to clarify the doubts that most people have about usability testing by answering few key questions in this article:
- What is Usability Testing?
- Why Usability Testing?
- Advantage of Usability Testing
- Disadvantages of Usability testing
- How to conduct Usability Testing?
- How Much Does Usability Testing Cost?
- Who? How many users? How many testing cycles?
What is Usability Testing?
Usability Testing (some prefer to call it User Testing or UX Testing) is a crucial element of UX (User Experience) design. As the name suggests, usability testing is a common technique used by usability professionals and UX practitioners to evaluate the usability of a product by testing it with real users.
It is totally different from the User Acceptance Test (UAT). UAT is the final phase of the software testing process. Typically, software users are tasked to test the software to make sure it can handle required tasks in real-world scenarios, and according to specifications.
The key goals of usability testing are:
- Determine how smoothly (or if) the users could complete the tasks successfully.
- To check whether the users could complete the tasks independently.
- To evaluate if the users enjoyed using it.
- Check how well the design works and its impact on the mental state of the users.
- Identify problems
- Find solutions
Although usability testing is favoured by developers and designers, such methodology & skillset is now widely adopted by a greater range of professionals and individuals from different fields such as:
- marketing
- research & development
- digital strategy
- customer service
Test it with real users
When we say, ‘test it with real users’, the message is clear – conducting usability testing is totally different from conducting technical testing which is another crucial process in software development. Usability testing focuses on the real user, to study their interaction with the product such as digital touch points, websites, mobile apps and even tangible products.
To put it simply, Usability Testing is one great tool to evaluate the user experience of your products. It provides input to the question, “How user-friendly is the product”. Nonetheless, usability testing allows you to discover opportunities to improve user experience as well as conversions.
We’re now living in the era of digital products and information. With new mobile applications and websites being launched every day, delivering a user-friendly product that emphasizes a user’s experience puts you ahead of the competition.
Complements quantitative research
Usability Testing is a qualitative type of research which serves as a great complement to quantitative research & testing tools like market survey, web traffic analytics, A/B testing & multivariate testing.
Ensures a positive user experience
Testing helps to evaluate and ensure that you’re delivering the right experience to users. Through usability testing, we are able to track not only usability issues, but also many other aspects including content requirement, user journey and preferences.
Increases your confidence
Testing not only helps you to track what is wrong with your product. It also increases your confidence level when a feature or a great element of your product receives approval or is commended.
Study user behavior
As mentioned above, usability testing is no longer exclusive to only designers and developers. We have seen marketing agencies that utilize usability testing to study user behaviour and preference on their own products, a competitors’ product or any other products that they’re interested to venture in.
To a certain extent, individuals (e.g., bloggers) also conduct testing to test the UX of their website to study how blog visitors behave while browsing content and advertisement.
We often hear marketing teams say that they conduct ‘similar’ testing on their digital products, whereby they expose their products to customers or potential customers to enquire their impression of the products without getting them to actually use the products.
Here are some of the metrics that are used to assess the user behaviour:
Qualitative metrics-
- User stress response through facial reactions and body language
- Perceived level of difficulty
- Subjective satisfaction
Quantitative metrics-
- Time taken by users to complete a task
- Effort required (number of clicks, etc)
- Success and failure rate
Accurate feedback and evaluation
Impression tests are great but can be misleading when it comes to evaluating the utility and user friendliness of products. Consider, for example, when a market research firm provide reports for a proposed product that receives a high score or rating from users based on their impression of the products:
The customers were exposed to the product at a glance and were asked questions like, “Do you think you can subscribe to this service easily?”
“Sure thing, I saw the subscription button right here”.– The end.
Following this, when it comes to getting the users to actually go through the subscription process after clicking the ‘subscription’ button, the registration and navigation process turns out to be rather torturous.
This is another time when usability testing can fit in to clear doubts.
Usability testing advantages
Usability testing provides many advantages, including:
- Direct feedback from target audience, giving clarity to the project team on issues to focus on
- Testing resolves internal debates on different options that are available.
- Highlights potential problems and other issues before the product launch.
- Increases the probability of product being used as well as repeat usage by consumers.
- Reduces risk of failure
- Ensures that user experience is good.
Usability testing disadvantages
While usability testing has many benefits, there are also several drawbacks that should be kept in mind.
- Testing is not a 100% representation of real-life scenarios.
- Qualitative testing methods means that sample size is small and thus, does not provide a large amount of data, such as that of a questionnaire. However, feedback for qualitative testing is usually higher in accuracy and insight.
The common way of conducting usability testing is through laboratory tests where users are invited to a lab to test a product. Generally, there are 3 main roles in a lab test session:
- the user (or tester),
- the moderator,
- and the observer.
The user will be the one to test the product by performing certain tasks that are laid out for him/her.
The moderator will be the one who moderates the test session by providing task guidance to the user and will ask questions about the product during the test. One precautionary measure for moderators is to ensure that questions asked are non-leading questions, for example, instead of directing the testers to the checkout process step-by-step, it would be more appropriate to ask them to figure it out by themselves.
The observer usually remains silent and observes how the user interacts with the product, taking notes.
The skillset of asking the right questions in the right way at the right time is not easily mastered; but one will always get better when practicing more in moderating the test session.
Similarly, note-taking & analysis can be challenging for those who are not familiar with the process. Observers should focus on recording down as much detail as possible to avoid missing any insight for subsequent analysis.
Sometimes, it involves 2 or more observers that might comprise of project managers, product owners, designers or other stakeholders. To prevent losing any insight from the testing session, the testing session can be recorded with a video camera so that the team can always review the testing session.
Types of usability testing
There are several types of usability testing methods.
The think-aloud protocol
The key requirement to propel an insightful usability testing session is the existence of a ‘think-aloud protocol’. Users are encouraged to express verbally what they are thinking throughout the whole session, so that observers and moderators can get greater understanding of a user’s experience when interacting with the product.
From here, moderators can also probe for further comprehension should there be any uncertainty. Setting up a full-fledged usability lab can be very costly as it requires certain advanced equipment, and hence, some practitioners still opt to conduct usability testing with minimal equipment such as a laptop & a recording device.
The Guerrilla Test
There are many opting to conduct Guerrilla Test. Guerrilla Testing totally opposes traditional lab tests as it is ‘hunting in the wild’ at public places such as parks & cafes where you can easily access the mass public.
However, there are also many who are reluctant to conduct both lab tests and guerrilla tests because these can be very costly & time consuming, considering that it might not be easy getting the right candidates in a short time. Another good alternative is to conduct remote usability testing, which is proved to be quick and cost-saving.
Remote testing
For a remote testing session, a moderator is not required to be physically present beside the tester. Remote usability testing can be moderated, or un-moderated: which means that there is no one moderating the test session.
Check out an article on Different Types Of Usability Testing Methods to learn more.
How much does usability testing cost?
Factors that affect your usability testing cost include:
- The type of testing you are conducting
- The number of people on the UX research team needed to run the testing
- The number of participants needed for the testing
- The number of days the testing will be carried out.
Simple studies won’t cost much, although you do have to consider paying a few hundred to the participants as an ‘incentive’.
Testing sessions can be carried out in a conference room. Assuming that you already have participants and know how to carry out usability testing, the most simple study can be completed in three days.
- 1st day: Plan the study
- 2nd day: Carry out testing with the participants
- 3rd day: Analyze findings
While the simple studies are usually fast and price effective, more costly research is at times required.
Elaborate studies can amount to a few hundred thousand dollars. Items that will add to the usability testing costs include:
- Testing of several designs
- International testing and any professional translation cost
- Multiple user groups testing
- Using equipment like eye trackers
- Using a physical usability lab/focus group room which enables observation
- A detailed analysis and report of the study results.
As such, it is important to consider the following when you’re coming up with the budget for testing:
- Time – Budget time for test preparations, running the tests, analyzing data, compiling the report and presenting the results. You will also need time for a pilot test.
- Recruitment costs – Think about where and how you will be recruiting your participants. You may need time to recruit on your own (another time factor) or decide to use a respondent recruiting firm to help you.
- Compensation – Factor in how you will be compensating participants for time and travel.
- Rental costs – If you do not have your own, you will have to rent a location as well as other equipment for the testing process.
Do remember that usability testing is not just another thing to cross off from your user list. Your team should have a clear understanding and goal as to why it is being carried out and then make the necessary adjustments according to the findings. Also, it is important to budget for more than one round of usability testing.
You should not use your own team members or your friends & family to test your products, as there is a high possibility of getting false results. Your friends and family might tell you ‘sweet’ comments to fawn you instead of telling the exact truth whilst your team members are usually as passionate as you in believing in the product.
Whereas for users who have no direct relationship to you have the freedom to speak out their mind and they react exactly how an ordinary user would without any conflict of interest.
A more challenging situation is that it might be difficult to get users who fit your criteria to test your product. For example, if you’re an education service platform provider in Malaysia aiming to provide services to children in Indonesia; it can be costly and time consuming to fly in to recruit the right participants for the test.
The next best alternative would be getting the local users (in your country) to do testing, provided that they must fit the crucial criteria of your target users such as age group, gender, education level and computer skill.
How many users do I need for UX testing?
The most frequently asked question would be “how many testers do I need to test? 50? 100?”. Believe it or not, you need only 5 testers (to start or try).
To further explain the logic on that, here’s a good explanation by experts on why you only need to user test with 5 users. The below graphs illustrates that by using 5 users, you are likely to obtain 85% of the usability problems of your product.
A Research Graph by Nielsen Norman Group
Just like the law of diminishing marginal utility in economics, the first tester generally yields more (newly-identified) usability problems than the second & subsequent testers. Less or minor usability problems would be found after the sixth tester. Our experience in conducting hundreds of usability test session has proved this point too.
How many testing cycles do I need?
If you have the budget for conducting a test with 30 users, it is recommended separating the test to two (15 users each) or three sessions (10 users each) depending on your development phase and requirement.
To summarize, it is a qualitative research method which does not require huge number of sample size as conducting a market survey research. You do not need 100 users to tell you that a button or link is not working.
In order to ensure a smooth and user-friendly experience for your products, consider utilizing professional usability testing services. Usability testing services offer expertise in evaluating task completion, user independence, user satisfaction, design effectiveness, and problem identification. They can be beneficial not only for developers and designers but also for professionals in marketing, research and development, digital strategy, customer service, and various other fields.
Image Credit: http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
