How to remotely test prototypes of mobile apps

The HubSpot usability team loves to get feedback on apps we’re developing to make our customer's marketing lives easier. But testing in the mobile environment presents a unique set of challenges. We recently devised a system that allows us to get much more insight from our testing sprints with real mobile users, using real apps in their natural environment -- in situ -- and discovered a whole new set of questions we were able to ask as a result.

The Mobile Marketer

Unless you’ve been living under a rock (or using a Blackberry), you’ve noticed the massive strides that social media and mobile marketing have made over the past year. And the HubSpot iPhone application has been keeping pace. With over 17,000 downloads to date, it’s clear that HubSpot customers are a mobile set. It makes sense that they’d start looking for us to develop more in the way of a social mobile app. After all, social is by its very nature a mobile game and HubSpot customers want to be able to see if social activity is or is not driving their success. So we here on the HubSpot usability team decided to look at how HubSpot customers are using social to help us decide exactly how to pursue developing the mobile social media publishing tool.

The Problems with Most Prototyping Tools

In the past, we used mobile prototypes on desktop computers for mobile testing. We’ve found that this strategy gives us a fairly decent understanding of how our customers use our mobile app. But even though mobile prototyping tools are relatively sophisticated, they were not entirely realistic. When you test with a mobile prototype, you compromise on a lot of important aspects of the user experience.

Prototypes can't:

• Be tapped, swiped or zoomed, because there’s no finger interaction at all
• Scroll, which means you can’t see more than a screen’s worth of content
• Explore the whole in-phone context of the app, such as where the app lives, and how the user accesses it
• Recreate usage on a physical handheld device
• Be installed from the Apple App Store

In addition, we wanted the least intrusive way to actually see how people were using these prototypes, and doing in-person testing simply would not scale.

Trick 1: The Remote Hack

Our mobile product manager Anand Rajaram discovered the first hack from the folks at Mailchimp, who shared a method to use the camera on your laptop or iPad to record what was happening on your iPhone. This simple hack allows you to see someone's phone from nearly the same perspective as they themselves see, and mimics real use amazingly well. (see picture) This solved part one of our dilemma, and would allow us to test with anybody who used the HubSpot mobile app no matter where they were. 

Trick 2: Use Proto.io

The next step was figuring out how to test with a mobile app that didn’t exist yet. Clickable mockups are reasonably useful for desktop testing, but presented an obstacle in the mobile world. After trying out several different options, Anand decided to use an interactive mobile prototype through the tool Proto.io because it did the best job of mimicking a fully functional mobile app that can be downloaded and used on a testing subject’s own phone. It made it easy for Anand and the designers to create and customize prototypes for the varying devices and screen sizes of the testers, which made the testing experience the most realistic.

Unexpected Insights

Once we had a system in place that would allow us to test mobile users using prototypes with some degree of accuracy and authenticity, we were off and running. Aside from the more general usability insights we gained from these in situ sessions, we also discovered that testing the users on actual phones made a huge difference in the scope of what we were able to discover during testing. We documented findings that would never have arisen in a less realistic testing environment.

For instance, we learned:

• What other apps people use: Seeing what other apps HubSpot users have on their home screens helped us understand what marketers are doing on a day-to-day basis on their phones, what other apps they’re trying and using, and what they like and don’t like about them.
• How people organize their apps: We got to see how our users group the HubSpot app with their other apps, whether in folders or just on separate screens. The folder names and the app with which they grouped us with were especially insightful. 
• How people navigate between apps: Looking at how HubSpot users interact with their phones (such as using status bar to scroll to top, if they use the multi-tasking bar and so on), how they switch between apps with bookmarklets, copying and pasting etc. gave us a look into their basic actions with their phones.

The Future of Remote Social and Mobile Testing

Even though we've learned a lot about mobile testing so far, we plan to keep on refining our remote testing techniques. The role of the mobile marketer keeps changing every day, so our usability testing needs to keep changing with it. We’ll continue to conduct research on social media and mobile marketing, but more importantly, we’ll keep learning how to test for the best possible insights no matter what we’re testing for. In situ mobile testing is only the beginning.

The HubSpot usability team has developed a process in which we do one-day testing sprints. We conduct testing in a fast-paced development environment, working to improve the usability and interfaces of the software. For us, it’s just not realistic to ask our customers into the office to test with us for an hour, due to location, time, and energy constraints. You can learn more about the testing sprint process here.