How to conduct remote usability testing

This blog has been updated! Originally published 14/04/21. Amended 24/05/23

Remote user testing brings both benefits and challenges. It means you can include people from across the world, however this makes it more difficult to prevent technology mishaps such as software problems or poor internet connection. 

Set of 8 post it notes with one in the foreground reading 'Run a usability test'
Raphael Clegg-Vinell, former Senior Accessibility and Usability Consultant at AbilityNet was a panellist on the UX/Usability Testing Panel: Personas, User Experience, and Research webinar from the International Association of Accessibility Professionals (IAAP) in 2021.

The main issues with user testing 

The panel discussed three main issues: 

  • how to use personas to incorporate accessibility from the beginning of solution's design; 
  • visualizing user experiences as a way to break down customer needs in context; and 
  • outlining a range of research methods you can use to focus on user testing and inclusion. 

Raphael responded to some of the key questions about user testing from the audience. 

What considerations should we be aware of when conducting remote research as opposed to research in a lab? 

Man and woman looking at computer screen, wearing masks

There are both advantages and disadvantages to lab-based and remote research. In a lab setting, you’re likely to have far more control over the technology. For example, internet reliability, up-to-date browsers and so on. 

Remote participants may live in an area where there’s poor internet coverage, have problems installing and using the software and some of the features such as sharing the system audio, for example.   

It’s important to make sure you provide participants with good set-up instructions beforehand and in some cases, I’ve had to run quick test-runs with people before research sessions to make sure they’re comfortable using the platform.    

Another key thing to be aware of is that if you have a client watching research sessions in a lab, you can easily engage with them afterward each session. If they’ve not quite understood an insight or you need to demo something to show them a problem which occurred, it’s usually quite straightforward when you’re in the same space.  

If doing research remotely, you might have to organise meetings to do this separately to make clients feel involved and engaged with the project.

Learn how to conduct moderated usability studies using participants with a diverse range of access needs, remotely and lab-based! Book AbilityNet's online training course: How to do inclusive usability testing 

When conducting usability testing, how would you approach choosing the number and range of participants? 

The number of participants you need for research is a tricky one!  

A 2000 study by the Nielsen Norman Group found that when using more than 5 participants, they weren’t tending to find any new key issues which hadn’t already been identified – the curve plateaued. The use of 5 people for usability studies is now often referred to as ‘the rule of 5’.  Group of people sitting in a circle holding papers. A woman in a wheelchair is the focal point of the picture

However, the study didn’t seem to take into account people with diverse access needs. For example, key issues identified by someone reliant on a screen reader are often going to be different from those identified by a non-screen reader user. So the rule of 5 goes out the window here, as you can imagine! 

Depending on your project, you might be thinking, surely just include 5 screen reader users, 5 people with a motor disability and so on. The difficulty here is that most clients you work with won’t have the budget to test their product or service with lots of people, so there will need to be some compromising in selecting a smaller number of people.  

When deciding on the range of participants to include, it’s important to first look at the product or service you’re designing. If it’s a virtual assistant such as Siri, it’s likely to be more important you include people with speech and language impairments.

If it’s a website for a video-streaming service, there may be more value in ensuring you have people with hearing impairments included in the research so you can get feedback from people reliant on aspects such as closed captioning.

How do you present your findings and insights in an engaging way for clients? 

Try and read your audience as much as possible and find out what’s likely to motivate them to take accessibility seriously.  

Some clients may be very data-focused and for these types of customers, it can be more important to try and back up insights you’ve found with metrics you can quantify.  

Others may be more responsive to qualitative findings, such as insightful quotes from participants. For example, if carrying out usability testing and someone comments on why X product is completely inaccessible for keyboard-only users, a quote from them could have a greater impact than just explaining the problem they encountered. 

If your report or presentation is going to be for a team of web developers, you can use more technical jargon and they may have a better understanding of things such as ARIA. If it’s for a UX team, you may want to tailor it slightly differently. 

Personas and research findings

Alongside Raphael, the panel included Lisa Vissichelli, Head of Visual Design, Research Manager, AnswerLab; Kathryn Weber-Hottleman, IT Accessibility Coordinator, University of Connecticut. 

Lisa talked about how to engage clients with research findings and how to visualize insights using methods, such as illustrations, to try and paint a better picture of feedback/issues identified and to try and convey people's lived experiences in a more engaging format. 

Kathryn shared her views and experiences about personas - what they are, how to use them in projects, how to ensure people include personas with disabilities and how they can be used alongside other research methods such as usability testing.

Learn how we can help with Diverse User Testing

Further resources