How to conduct remote usability testing

"Carrying out research remotely means you can include people from pretty much anywhere, but they might live in an area where there’s poor internet coverage," says Raphael Clegg-Vinell, Senior Accessibility and Usability Consultant at AbilityNet, talking about the challenges associated with different types of usability testing (more on this below).

Set of 8 post it notes with one in the foreground reading 'Run a usability test'

Raphael was speaking as one of the panellists on the UX/Usability Testing Panel: Personas, User Experience, and Research webinar from the International Association of Accessibility Professionals (IAAP).

Alongside Raphael, Lisa Vissichelli, Head of Visual Design, Research Manager, AnswerLab; Kathryn Weber-Hottleman, IT Accessibility Coordinator, University of Connecticut and Bradley Held, Head of Digital Accessibility, UNC - Chapel Hill, discussed three main issues:

  1. how to use personas to incorporate accessibility from the beginning of solution's design;
  2. visualizing user experiences as a way to break down customer needs in context; and
  3. outlining a range of research methods you can use to focus on user testing and inclusion.

Here is Raphael's response to some of the key questions from the audience.

Are there any differences or challenges we should be aware of when conducting remote research as opposed to research in a lab? 

Man and woman looking at computer screen, wearing masksYes, I think there are certainly some differences to be mindful of and there are some advantages and disadvantages to lab-based and remote research. One thing which is perhaps obvious is that in a lab, you’re likely to have far more control over the technology. For example, internet reliability, up-to-date browsers and so on. 

Carrying out research remotely means you can include people from pretty much anywhere, but they might live in an area where there’s poor internet coverage.   

Over the last year or so, we’ve been working a lot more with platforms like Zoom and in many cases, research participants have problems installing and using the software and some of the features such as sharing the system audio. 

If it’s in a lab, this is of course done for them. It’s quite important to make sure you provide participants with good set-up instructions beforehand and in some cases, I’ve had to run quick test-runs with people before research sessions to make sure they’re comfortable using the platform.   

I think another other key thing to be aware of is that if you have a client watching research sessions in a lab, you can easily engage with them afterward each session. If they’ve not quite understood an insight or you need to demo something to show them a problem which occurred, it’s usually quite straightforward when you’re in the same space. 

If doing research remotely, you might have to organise meetings to do this separately to make clients feel involved and engaged with the project. 

Find out more about usability testing
Sign up for AbilityNet's online training course: How to do inclusive usability testing on Thursday 6 May 2021.

Disabilities cover a huge spectrum. When conducting research such as usability testing, how would you go about choosing the number and range of participants? 

The number of participants you need for research is a tricky one! 

Back in 2000, the Nielsen Norman Group wrote an article about an interesting study they’d run which looked at the number of participants needed and they did this by documenting the number of usability issues found per participant. They found that when using more than 5 participants, they weren’t tending to find any new key issues which hadn’t already been identified – the curve plateaued. The use of 5 people for usability studies is now often referred to as ‘the rule of 5’. 

What the study didn’t seem to take into account, though, was people with diverse access needs. For example, key issues identified by someone reliant on a screen reader are often going to be different from those identified by a non-screen reader user. So the rule of 5 goes out the window here, as you can imagine! Two people working on computers at a desk

Depending on your project, you might be thinking, surely just include 5 screen reader users, 5 people with low vision who may use screen magnification software, 5 people with a motor disability and so on. The difficulty here is that most clients you work with won’t have the budget to test their product or service with lots of people. 

A company like Amazon should have the budget, but a small start-up or charity most likely won’t. It’s therefore likely to be case of compromising and selecting a smaller number of people. 

When it comes to working out the range of participants to include, it’s important to first look at the product or service you’re designing. If it’s a virtual assistant such as Siri that you’re working on, it’s likely to be more important you include people with speech and language impairments. 

If it’s a website for a video-streaming service, there may be more value in ensuring you have people with hearing impairments included in the research so you can get feedback from people reliant on aspects such as closed captioning.   

How do you ensure you present the findings and insights from your research in a way which engages clients? 

I think the first step is try and gauge your likely audience for your findings. For example, if your report or presentation is going to be for a team of web developers, you can use more technical jargon and they may have a better understanding of things such as ARIA. If it’s for a UX team, you may want to tailor it slightly differently.   

Most people aren’t likely to be keen to read through lengthy technical reports so I think it can be good idea to try and humanize them wherever possible. One way of doing this is to include insightful quotes from participants. For example, if carrying out usability testing and someone comments on why X product is completely inaccessible for keyboard-only users, a quote from them could have a greater impact than just explaining the problem they encountered.   

Some clients may be very data-focused and for these types of customers, it can be more important to try and back up insights you’ve found with metrics you can quantify. This comes back to trying to read your audience as much as possible – try and find out what’s likely to motivate them to take accessibility seriously.   

Personas and research findings

During the IAAP webinar, Lisa also talked about how to engage clients with research findings and how to visualize insights using methods, such as illustrations, to try and paint a better picture of feedback/issues identified and to try and convey people's lived experiences in a more engaging format.

Kathryn shared her views and experiences about personas - what they are, how to use them in projects, how to ensure people include personas with disabilities and how they can be used alongside other research methods such as usability testing.  


Learn how we can help with Diverse User Testing

Further resources