Mobile App User Testing: Methods, Tools, Best Practice

Konstantin Kalinin

Head of Content

December 2, 2020

We love mobile usability testing and do it on all projects. That’s how we figure out the strengths and weaknesses of a mobile product and verify app owners’ assumptions. That’s how we’ve helped our partners raise over $165 million. And that’s precisely what you need to build an app that gets real traction in the market.

So if you’re wondering whether you need mobile UX testing and want to make sure you’re covering all the bases, come along. You’re in the right place. We’ll go over everything a mobile app owner needs to know about mobile app user testing.

1. What is Mobile UX Testing
2. Who Takes Part in Mobile UX Testing?
3. What Takes Part in User Testing?
4. When Should Mobile Usability Testing Take Place?
5. What Do You Measure During User Testing?
6. What Are the Benefits of Mobile User Testing?
7. Mobile UX Testing Methods
8. 3 Steps of User Testing: How to Do Usability Testing for Mobile Application
9. Top 3 Mobile App Usability Testing Tools
10. Challenges in Usability Testing of Mobile Application
11. What Does User Testing Cost?
12. Topflight Best Practices: How We Conduct User Testing

What is Mobile UX Testing?

User testing itself goes by many names, for instance, usability testing or UX testing. We’ll be using all these terms here interchangeably.

Definition

If we check a few reputable resources for the definition of user testing, we’ll get these variants:

Usability testing refers to evaluating a product or service by testing it with representative users.

Usability.gov

Usability testing is a popular UX observational methodology used by UX researchers to uncover problems and opportunities in designs.

Nielsen Norman Group

Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users.

Wikipedia

Simple enough, right? So when we talk about mobile UX user testing, we mean examining mobile apps to see if people can effectively use them to address their needs.

Of course, we test the mobile product, not the person using it during usability tests.

Who Takes Part in Mobile UX Testing?

Even though the answer to this question may seem obvious, let’s quickly list everybody who usually participates in UX testing:

• test users (also known as participants)

Someone from the app’s target audience — the most essential member of any UX testing session. They represent the people who will be using your app once it’s out. So pick them carefully.

• facilitator

The facilitator is often a UX expert who instructs and observes participants while (or before) they work on predefined tasks in a test app.

Obviously, this role needs to remain “constant” as much as possible to avoid skewing test results by influencing test users.

• other stakeholders

Other stakeholders may include QA engineers, developers, or anyone on your team with a strong opinion about certain app aspects who think their questions and observations during the test session will result in more product insights.

Related: Hiring App Developers: Finding the right fit 

Please note that the test user should not be aware of other stakeholders taking part in the test.

What Takes Part in User Testing?

Besides the facilitator, representative test users, and (sometimes) other team members, we need the custom built mobile app itself and tasks. What do we mean by tasks?

Well, your app serves a particular purpose, right? Now, you need to offer participants a realistic objective that they can achieve in the app.

For example, suppose it’s a remote patient monitoring app. In that case, you may ask test users to rate their emotions and overall well-being. Therefore, tasks — real-life in-app scenarios (or test cases) are crucial for validating an app’s usability.

When Should Mobile Usability Testing Take Place?

Based on where you are with your mobile app at the moment, we can talk about:

• Prototype user testing

This happens when you start fleshing out your app idea by converting product requirements and everything you know about the target audience into wireframes. The beauty of this approach is cost savings because running tests and amending wireframes is way less expensive than straight coding.

• MVP usability testing

MVP is the initial version of your product that will need to verify all your assumptions in the real market. To maximize this effect, you might want to run UX tests right before you’re ready to release the app.

As a result, you’ll be able to adjust onboarding and other minor details that add up to a huge effect in a live mobile app.

• Post-launch, live app tests

Once we launch the app, we usually start gathering its usage data, and if the data shows potential roadblocks in user flows, we need to verify our hypotheses and remove them.

Ensuring usability is critical, and so is working with a team that understands custom mobile software development for user-centric solutions.

What Do You Measure During User Testing?

Depending on your focus during app examination, you may be after numbers or stories, correspondingly running quantitative or qualitative tests.

Quantitative UX tests are all about numbers and app metrics, for instance:

• How long does it take a new user to onboard a fintech tech?
• How often do test users fail to complete a task?
• How difficult was the task on a scale of 1 to 10?

You run these UX tests to compute approximate cost savings from updating the application UX design, compare with other design versions, or set the future benchmark to compare to.

Qualitative user testing is more common and has to do with discovering inefficiencies in the mobile UX:

• Do test users struggle with finding a contact in this doctor’s appointment app?
• Will they use this new menu as we’ve envisioned?

Qualitative tests are perfect for uncovering bottlenecks in mobile user flows.

What Are the Benefits of Mobile User Testing?

Why would we run a remote mobile usability test on a mobile app? First comes the cost.

• Mobile UX testing is the least expensive option to verify your ideas at any stage: prototyping, pre-release, or post-release.

Of course, when we initiate application usability testing before coding, we save ridiculously. However, if we engage in mobile user interface testing when the app is already live, we can still save the budget because we won’t be experimenting with the existing user base.

Otherwise, if our assumptions are wrong, we could cause higher user churn.

• The second most significant benefit of mobile user testing is quicker time to market.

Getting feedback from test users faster also means that we can release the app into app stores sooner. In a startup world, that could mean the difference between success and failure.

Mobile UX Testing Methods

If you think about it, there are not so many options for arranging mobile app usability tests:

• You can run them remotely or in-person
• You can choose to moderate or not moderate them

If you combine all these options, you get this matrix of mobile usability testing methods:

Remote vs. in-person user tests

Even if we weren’t in the middle of the pandemic, I’d suggest that you ditch in-person user tests. Simply because no one behaves the same when they are watched or left alone.

And that has nothing to do with letting participants relax. The sheer presence of a facilitator nearby will skew test results. In-person app analysis is OK:

• when you need to spend a UX budget for a huge corporation
• when you want to impress your management with all the fuss about organizing the procedure

In real life, especially if you’re a startup, you’re better off running user tests remotely by using special tools. Fortunately, COVID-19 favors this user testing method, too — there’s no need to risk an infection when the whole thing can be easily done remotely.

We’ll speak about the tools you can use for that in a bit, but here’s a brief explanation of how these remote methods really work for mobile app user testing.

Remote Moderated

Remote moderated user testing is like being on a Skype call with a participant and seeing their screen and interactions in real-time.

During such mobile user tests, besides watching users complete tasks, you also get to talk with them, ask additional questions, or set new tasks. In addition, you can get feedback from other stakeholders who, depending on the tool, may request further clarifications from test users.

Remote Unmoderated

Remote unmoderated user testing is the same but without you attending tests. Therefore, such tests can elicit any spontaneous feedback from participants. What you get is a recording and some metrics that participants have generated while completing tasks.

3 Steps of User Testing: How to Do Usability Testing for Mobile Application

So the chances are you’ll be running remote UX tests on your mobile app. How do you organize that? What steps do you take to run a successful user test that gets you better user traction with your mobile app?

Step 1: Plan

Regardless of where you’re with your mobile app — putting together wireframes or getting downloads from mobile stores — you plan to run mobile usability testing for a reason. Therefore, goal setting is critical.

In other words, you need to know beforehand what you’re looking for when you prepare tasks for users. When creating such tasks for usability testing on mobile phones, we often target:

• key functionality
• navigation
• any potential pitfall areas

And of course, setting tasks that follow real-life scenarios for using your app certainly helps to get mobile user experience testing right.

One other thing to note when planning user interface testing for mobile applications is participant selection. We should carefully pick test users based on demographics criteria that we can glean from the app owner: their age, income, gender, location.

The more sophisticated the tool you’re using for remote UX tests is, the more fine-grained filters you can get to pick the right audience for testing the mobile app’s UX/UI.

Step 2: Test

Testing is obviously a crucial part of usability testing for apps, and the best part about it — you don’t have to attend. As we’ve already established, remote mobile user testing can be moderated and unmoderated.

As you go through the second step, you get to see their pros and cons.

As you can see, with remote moderated UX tests, you get higher interactivity with testers, but this method is more expensive and requires the coordination of test sessions.

As for remote unmoderated usability tests, you can run them faster and on a more tangible budget. However, you won’t be able to ask follow-up questions and therefore wouldn’t be able to glimpse behind a test’s face value.

What’s reassuring is we get to analyze participants’ body language, which is a dead giveaway about their experience with our mobile app. I would argue that unmoderated monitoring works best because there’s no additional pressure on test users.

Step 3: Analyze

Finally, once tests are completed, you need to analyze the results. What’s there to analyze, really? Well, you’ll be getting one of these or a mixture of these artifacts:

• written feedback from participants
• complete/incomplete tasks
• video recording

When analyzing a recording, we’re looking for visual cues. Did the participant run into any issues? Did they spend the same kind of time on tasks we expected? As you can see, analyzing user testing feedback is akin to answering questions that stem from your mobile app research goals.

What’s important is that all this analysis needs to transform into a mobile usability testing checklist with actionable insights. We can then apply these to our mobile app’s UX/UI to improve its overall usability.

Top 3 Mobile App Usability Testing Tools

Since I’m trying to convince you of the effectiveness of remote mobile user testing, let’s review the most popular usability testing tools for mobile apps that serve the purpose.

We’d want to pick a platform that allows us to cherry-pick test users based on our preferences and collect quantitative and qualitative data. With that in mind, let’s dive in.

Related: Test Automation Strategy Guide

Userlytics

Founded: 2009.

Description: Userlytics lets you run remote user experience (UX) tests for any type of applications, including mobile apps.

Highlights: annotations, highlight reels, audio transcriptions, a branching logic for tasks.

UXCam

Founded: 2013

Description: UXCam is a remote check-up tool that provides video playbacks, screen UX analysis, and user behavior analysis.

Highlights: besides offering session heatmaps and session replays, UXCam
also uses machine learning to identify users that were frustrated with your app during the test.

UserTesting

Founded: 2007

Description: We love and use UserTesting all the time, so we might be biased here. The tool provides us with everything we need to uncover potential UX issues in a mobile app and create a winning UX/UI strategy for your product.

Highlights: We appreciate their massive user base that allows us to drill down to perfect representatives of an app’s target audience, the ease of use, and they have everything else you’d expect from a remote user testing service like video recording with notes, etc.

UXtweak

Founded: 2019

Description: UXtweak offers powerful research tools for improving usability of websites and mobile apps, from prototypes to production. Their offering includes content organization, various tests and surveys, and a number of recruiting options.

Highlights: The platforms aims to automate and streamline the whole process of user testing, including research and reporting.

Invision / UXPin / Proto.io

It’s worth noting that when you do user testing prior to coding the app, you’ll work with such tools as Invision, UXPin, Proto.io, or similar. Their purpose is to quickly put together app designs and turn that into an interactive prototype of an app.

Once you have such a prototype ready, test users can click it through in a desktop browser or right on mobile devices.

We prefer Invision because of its advanced features:

• Collaboration options for designers (commenting, version control, etc.)
• Rich collection of animation effects
• Integrations with design tools like Sketch, etc.

EXPLORE OUR RAPID PROTOTYPING SERVICES

UXPin also has its advantages. They offer prototyping that is accompanied by coding. As a result, we get prototypes that look closer to real mobile apps, but we also have to sacrifice the cost and speed. Coding the logic layer (in UXPin) takes time and costs more compared to designing screens and interactions in Invision.

What if Mobile Web is Part of your Mobile App Experience?

When we shift our attention to mobile sites and Progressive Web Apps (PWAs), it’s important to note that the landscape of testing tools has been changing. Unfortunately, Google discontinued its Mobile-Friendly Test tool as of December 2023, which was a blow to many developers.

However, all is not lost. Developers can still utilize Google’s Lighthouse tool, an open-source automated tool aimed at enhancing the quality of web pages, especially when building PWAs. The value of Lighthouse lies in its ability to provide actionable insights about your PWA’s performance, accessibility, and more.

Simultaneously, another useful resource that developers can leverage for bulk mobile-friendly testing is offered by EXPERTE. This free alternative tool provides comprehensive reports, helpful for identifying and addressing any potential usability issues.

So, while Google’s Mobile-Friendly Test tool may be out of the picture, there are still robust options available to developers. Remember, consistent usability testing is a critical part of mobile web and PWA development, helping ensure a seamless and satisfying user experience.

Advice for bootstrapped app owners

If you’re absolutely bootstrapped, you can ask your friends and relatives to use iPhone’s built-in screen recording option. That way, you’ll get videos of them testing your app. Alternatively, buy Mr. Tappy’s toolkit for $349 and set up a local user testing lab using this equipment. Finally, you can try screen sharing via Skype or Zoom on a mobile device.

Challenges in Usability Testing of Mobile Application

Mobile usability user testing is still relatively new, and not every company does that. So don’t be surprised if the process and tools look a little intimidating. That’s exactly why many app owners miss out on user testing. Still, if you follow the procedure, just know how to address the most challenging parts.

Provide link in an appropriate format

We want our app prototype to appear on the participant’s device without cut-off parts. For instance, to get that right with an Invision clickable prototype, the participant would need to install their app. The key is to avoid testing a prototype in a mobile browser.

Long testing sessions

Remember that you don’t need to attend remote mobile UX testing and instead review video recordings after tests have been completed.

Some tools would even add timestamps for the most interesting interactions, e.g., when the user encountered a problem in the app.

Quick access to the right audience and test devices

Many of the health and fintech apps we build target specific, niche audiences. And we’ve discovered that one of the best solutions to quickly ramp up a necessary test audience is to use tools like usertesting.com.

What Does User Testing Cost?

We include mobile UX user testing as part of our Rapid Prototyping Package, ranging between $15,000 – $20,000. However, the outcome is a little more than a thoroughly user-tested mobile app prototype — it’s something tangible you can take straight to investors.

If you’re conducting user testing on your own, reach out to one of the platforms we’ve outlined for pricing. Usertesting is around $20K per year. On the other hand, Userlytics allows you to start with $49 per a 15-minute test session. However that quickly becomes a mess if you want more features, longer sessions, etc. Investing in user testing will be key to lower app development costs in the long run since you’ll know exactly what features your users want, and wouldn’t go overboard only to realize you invested in a feature they’d never use. 

Topflight Best Practices: How We Conduct User Testing

After helping startups raise over $165 million, we’ve collected some best practices for arranging user testing for mobile apps. I’d like to share a few of them here with you:

Replace in-person user testing with remote testing

Even if there was no pandemic, testing mobile apps in-person requires too much preparation (read costs more). And you can’t deny the fact that the sheer presence of a facilitator changes participants’ behavior.

Ditch think-out-loud protocols

Some companies tout special protocols that stimulate participants to voice all their actions in real-time as they navigate around a mobile app.

If you try this technique, you’ll discover it’s nearly impossible to concentrate on a task and, at the same time, explain what runs through your mind at the moment.

Have a second say on interpreting feedback

Once the tests are over, and it’s time to analyze findings, remember to get a second opinion from your colleagues to make sure you aren’t making false assumptions. Otherwise, you might end up with unnecessary app modifications, which brings you back to another round of user testing.

Overrecruit to compensate for no-shows

Book one-two extra meetings to compensate for potential no-shows. That way, you still get the results in a timely manner, even if some participants couldn’t make it.

How many testers to recruit?

Nielsen Norman Group recommends recruiting 5 users, which should cover most mobile app owners’ needs. However, if your mobile product is robust (over 30-40 screens), you will benefit from running usability tests on 15 participants in 3 successive batches. That’s enough to uncover 99% of the UX issues in the mobile app.

Use a tool that doesn’t slow down the app

Most remote user testing tools will require SDK installation. So pick the one that won’t slow down your app. We haven’t tested them all, but we can vouch for UserTesting.

All participants should work on the same tasks

You’re kind of already addressing this when you run mobile UX tests remotely because all participants get the same task description. Still, if you’re participating and have spontaneous follow-up questions, make sure you voice them with all the remaining participants.

A word of advice about tasks themselves: try to make them as detailed as possible, including all “presets” that users need to follow, e.g., device orientation, sound on/mute, cellular connectivity, etc.

If you absolutely have to run in-person UX testing

Avoid giving participants additional time at the start to familiarize themselves with the app. That’s a perfect scenario for ruining their first impression of the app, including onboarding and figuring out the app’s basics.

Test your tests

Run a sample with 1-2 users to ensure your instructions are clear and everything works as expected. You might find out that better definitions will help you speed up the user testing application process.

If you have more questions about mobile app user testing, reach out to speak with our consultants.

Related Articles:

1. Mobile App Design Guide 
2. Tips on using customer feedback to iterate your mobile app