The nature of software, app, cross-channel experiences and web-development has been subjected to constant shifts and evolutions with time. Because of these shifts, the market has become increasingly global.

With this globalization, the reliance on in-person usability testing and user research has become a heavy financial burden for businesses. Flights, hotel bookings, and time, are all factors which create costs that many organizations find difficult to meet.

Here comes remote usability testing to the rescue…or maybe not?

Remote usability testing, the methodology that uses an online software to record a screen or voice of a test population as they use your site or application from a distance (in their natural environment), has been the option of choice for many businesses, and has, no doubt, been the go-to method for reducing costs and time considerably.

However, what level of reliability can we attribute to remote usability testing that can be a reference point for serious business decisions? As a UX professional that has had extensive experience in remote and in-person usability studies, I’ve found that the sole reliance in remote usability testing as a tool to provide quality data and insights can prove to be misleading.

Here are three reasons why the trustworthiness of remote usability testing is declining.

1. Low Reliability in the Advanced Analytics Engine

While research that is conducted in a population’s natural environment is conducive to more realistic insights driven by honest reactions and opinions, no contact with the users, whatsoever, yields a very low level of reliability.

We live in a world full of interruptions, random conversations, text messages, email updates and messages. When we pursue remote usability testing, we have no way of knowing how the user is interacting with the product – whether they find something difficult to complete or whether the way they are using the product is their preferred method.

Even setting up moderated studies with stationary webcams will provide restrictions on UX researchers that may need to attend to a user’s particular physical movements.

We certainly can set “testing without interference” and request for participants to leave their mobile device outside the test session, but, is this a realistic request? Let’s face it. A handful of us are actually capable of spending more than five minutes away from our phones, let alone capable of turning them off. And, apart from that, do we really want to contaminate results?

Usability testing that does not let UX researchers see the user’s body results in low reliability.

2. Lack of Understanding of the Population’s Culture

We’ve seen remote usability apps allowing for understanding user interactions, but useful usability research, for businesses across the board, is the product of not only ergonomics but of culture.

When designing products, businesses aim for users all over the world to be able to access and interact with those products.

Remote usability testing, largely quantitive in this case, does not allow for physical interaction with users from all over the world.

remote usability testing

Without a true understanding of the cultural behaviors and subtleties of all user populations, you may lead to censorship of the results. When you pursue further usability tests, you may not understand the cultural context, thus leading you to an unclear picture of your results.

As usability researchers, it is encouraged to pursue ethnographic research to get the context for your product, service or content and go beyond simply validating if your product works or needs improvement.

3. No Real Verification of the Persona

In many usability studies, we frequently tend to confirm the persona with the product manager or marketing department – and we leave it at that.

That’s a big “no no” if you ask me.

Meeting with users one-on-one should give us equally important outcomes from the study itself, but verifying the persona – checking whether the participant meets the persona defined at the beginning of the process – is crucial.

remote usability testing

If we’ve missed an important feature of our persona – it’s time figure out the error and fix that mistake.

With remote usability testing, we miss the opportunity to do this.

What’s the best usability testing for you?

While remote usability testing has its benefits, from saving costs to quick insights taken from a user population’s natural environment, in my time as a UX professional, I’ve found remote usability tests to provide lower data quality. This can create huge risks in the reliability of the entire study, which in the long-term can turn to be more time-consuming and costly for businesses.

I’m not a proponent of choosing one method over the other, but I do encourage that UX researchers and stakeholders, alike, take a more comprehensive approach, as opposed to relying strictly on remote usability testing. This lets you apply findings from one study and add it to another in order to get the highest quality data.

Subscribe to the Weekly User Experience Newsletter

Join the growing list of thousands of UX practitioners who are meeting the need for better UX'd products.