Assessing the Usability of Your Online Community (Part 2)

In the first part of this blog post, I talked about creating personas for your community. Although the title of that post included the word usability, it turns out that I didn’t really have much to say about usability per se in that post. That was intentional. People often think that usability is an absolute measure; either a site is usable or it is not. But that is false. Usability is contextual. You cannot properly assess usability without knowing who the user is and what they are trying to achieve. Personas and the goals/tasks that are associated with them, are a means to capture that contextual information and that’s why I focused on them in the first post.

Now, however, let’s assume that you have your personas in hand.  What do you do with them? There are actually two somewhat different scenarios you could find yourself in. If you already have a community site in place, then usability testing is a means to determine what usability issues exist with the site, and how seriously they impair peoples’ participation in the community. If you don’t have a community site already, then you can carry out usability tests against prototypes or existing platforms as a means to help determine what your site should be like.

I’m going to deal primarily with the first type and leave it as an exercise for the reader to sort out any of the salient differences for the second.

Usability testing all about having real users interact with your site in realistic ways, and then observing how they go about doing that. If at all possible, you want to create a video recording of these sessions because it will be incredibly useful to come back and review the sessions later on.

For each person you bring in, you want to give them a set of tasks to accomplish based on the persona that they most closely match. You need to make sure that these tasks are something that the user would try to do in real life, otherwise their tactics in carrying out these tasks may not be reflective of what will happen outside of the testing environment. Typically, you do not want to assist/interfere with the user while they are carrying out the tasks — in real life, your users won’t have someone sitting beside to lend a hand when they get stuck. Finally, you want to encourage your testers to talk about what they are doing. This is a little unnatural for most people — after all most of us have learned to keep our interior monologues actually interior. Nonetheless, this is a situation in which you really want to know why people are interacting with the system in the particular ways that they have chosen, and having them describe that process is probably the most straightforward way to get this information.

This sounds like a pretty intensive process, so a perfectly reasonable question is: “How many people do I need to use in order to get meaningful results?” Again, you’ll find a lot of consideration given to this in the literature. My own experience is that you can start to see issues after watching 3 or 4 people. By the time you get up to 7 or more, some issues seem blindingly obvious. That is, once you’ve seen everyone stumble while carrying out some particular task or get stuck on some particular screen, you know you’ve got problems. If those tasks are part of the primary mission for your site, then you know you’ve got to fix those things or risk having the site fail.

Other issues will be more subtle. They may only affect certain people, or they may be functionality that you don’t consider critical. As with almost anything of this nature, the reasonable response is to record it all, prioritize it, and figure out what you have the resources to deal with.

Of course, I’ve just scratched the surface of usability testing. There are a huge number of books on the topic, and I would definitely recommend picking one or two up and giving them a read. One of my favourites is Rocket Surgery Made Easy, but your mileage may vary.