Customer-Driven Product Development: 6 Steps to Using Live Chat For User Testing

The use case for live chat on a website seems obvious at first glance. It’s a support tool that allows users to more quickly and painlessly get help from your team, rather than having to fill out a ticket or call. Here at the Signals team, however, we use live chat for one other very distinct use case: user testing.

The beauty of live chat is that it creates an avenue for you to speak with users when they’re in the thick of their experience with your product. As I explained in my previous post, it helps you get to the real problem. In addition, you can actually use it for gathering both qualitative and quantitative -- if executed under the right circumstances.

Why live chat for user testing?

While it doesn’t replace the value of traditional user testing, using live chat for feedback is a quick way to accomplish a few things.

You can:

• Get feedback on a wide or targeted range of the product

• Get engineers engaging with users and customers

• Keep the product team close to the end user

• Gather lots of data in a short period of time

• Unearth pain points or confusion

• Drive UX & UI product decisions with the user in mind

Step 1 - Get the engineers accounts

Everybody on the team should be involved. After all, it’s their product too and it’s important that everyone has an understanding of the ins and outs of explaining the product. Get all of your team members accounts on your live chat service (at Signals, we use olark). Have them log in and leave it open in a new tab. Ask them to turn on desktop notifications or connect it with their chat client so it pings the corner of the desktop when a user messages the chat room.

Step 2 - Pick a focus

Choose which part of the product you’d like to focus on and open olark chats to display on all the pages relevant to that part of your web app.

Making live chat available on all pages of your app can help you identify where the most pain is, where the most improvements can be made relative to the rest of the product, and allows for easier access to quantifiable data.

If you want to get really targeted, create some javascript snippets that only display live chat for certain users and on certain pages. This could be greatly beneficial when you’re releasing a new feature, implementing a redesign, or just wondering how a certain page in your web app is perceived.

Step 3 - Unleash the live chat

Flip the switch and let the chats start pouring in. Make sure the olark settings are set so that one person, presumably the Product Manager or owner of user testing on the team, gets the all of the transcripts sent to one email account. This will allow you to make sure nothing falls through the cracks and get a feel for what types of things are coming in, overall.

Step 4 - Hold about 200-250 chat conversations, then turn it off

Now that you’ve collect a bunch of data points, you likely have enough to create a backlog for months. Unless you have the human resources to leave chat on, it’s best to turn off the beast and send people through your normal support flows until next time.

Step 5 - Do some data entry

Read all of the transcripts to get the true nugget of where the root of the problem is. Organize it in a spreadsheet. Break down the spreadsheet by section of the product, as seen in the screenshot below. Tally the frequency and take notes so that anybody on the team can reference the sheet and know what’s going on. Make sure the topics are short, concise, and to-the-point. We used conditional formatting so that any topic with a high frequency was colored, which helped us create a heat-map of the product.

Step 6 - Prioritize and take action

Outline the top few things; see what can be solved with a product change, what can be solved with a copy change, and what can be solved with a documentation change. Prioritize those changes and execute.

The beauty of having the entire team on these live chats is that convincing anybody of anything is incredibly easy as you have the quantitative data to back it, quotes to serve as qualitative data, and they’ve all likely experienced each pain point themselves.

Rinse, repeat

When you conduct live chat through the lens of user testing, you can flip the switch anytime and gather data for your backlog that can span months of work. The goal should be to extract the highest-priority issues, act on those, ship fixes, and move on the next round of feature building and testing. 

Warning: Avoid creating negative experiences

With live chat services, you're able to log in often to your web browser to start accepting chats. Once chats die out for a few minutes, it's often easy to forget you had it opened as you turn to do other work. This could spell bad news for customers if you have your chat client set up to automatically accept chats and don't reply promptly. If somebody sees a chat box, messages you, and doesn't get a response within about 10 seconds, you've immediately frustrated your user and given them good cause to feel neglected. Don't do that.

 --

Want to learn more about customer-driven product development?Follow my personal blog at http://bilotti.org