Practice

User Research 102 (Moderating, Note-taking, and Other Important Stuff)

Last month, we ran a quick half-day workshop for one of our clients. They had a great internal research group, and were such big believers in the value of research (!) that there were more requests than the internal staff could support.

They wanted to provide some research guidelines, tools and support to their Product Managers, Designers and Developers who were interested in doing research themselves, or in understanding and leveraging it better. This was essentially a “Research 102” workshop, for people who were mostly familiar but not experienced doing research. We didn’t try to cover everything you’d need to know or all types of research – in fact, we only addressed early interviews and usability studies. We talked about conducting the research as the moderator, since many best practices applied to both types of research. We also talked about stuff around research; important things to get right besides moderating.

You can also listen to Example #1, Example #2, and Example #3

We’ve included the entire deck, but wanted to call out some highlights, including things we brought to the workshop, and things we learned from participants while we were there:

Five Moderation Skills to Learn First

  1. Be quiet and listen!
  2. Be neutral and avoid value words. Some strategies for being neutral include using incomplete sentences (“Now that you’ve completed that, would you say that the process was…?”), parroting to probe and avoid answering participants questions when you need to (“You’re wondering what this screen is for?”), use participants’ language (see Steve Portigal’s awesome Tivo story)
  3. Be calm – you set the tone, and if you’re nervous, they will be too. If you’re feeling rushed or anxious, engage in a little small talk about the weather until you’re both feeling comfortable.
  4. Observations ≠ recommendations. Remember the difference between noticing an issue and solving it! There are many potential solutions, and so many people get these mixed up and miss out on opportunities to improve their products.
  5. Note hypotheses up front — and consider, is success proving them right or wrong? Too often, people consider it a victory when they’re proved right about some idea, versus considering it a victory when they uncover something they can improve. Being clear on your own agenda can help preserve the value of your research (see the Third Simple Trick).
  6. (Bonus points) Behavior first, then attitudes and motivation, not the other way around. Anchoring people in a specific example of their own behavior helps orient them to their actual attitudes and motivations, versus letting them tell you about what they wish their attitudes and motivations were. In my favorite example of this, I was helping a team build up their own internal research skills. I had moderated a few sessions, and then was listening and giving feedback on their moderators. The moderator asked the participant whether she ever reviewed products she bought online. She chatted for some time about how she considered their site a community and it was important to contribute to that community and not just take, etc. etc. At the end of the session, when people could ask questions, I asked her to tell me the story of the last time she wrote a review of a product – she couldn’t think of one. Had we started with a question about her actual behavior and then moved into attitudes, I think we would have gotten different, and more accurate information. Having said all that, there is a BUT: if you’ve asked lots of questions about behavior and still don’t have a good sense of attitude because their behavior doesn’t seem consistent or clear, it’s ok to ask a general question and come at it the other way.

Three simple tricks for avoiding common mistakes in moderating research:

  1. Stop talking. Be quiet. Listen. This avoids all kinds of common mistakes like leading, interrupting, and missing the opportunity to hear more about underlying motivations and attitudes. People are uncomfortable with silence, and if you can bite your tongue longer, they will fill it.
  2. Ask about others. If you just can’t resist the urge to ask someone how they’d behave in the future (and there is TONS of research to show that people are completely undependable when answering that question), try asking them how they think others would behave. Data shows that answers are still not great, but considerably more reliable.
  3. Don’t do it. If you feel passionately about a product or topic, your goals, prejudices and inclinations will color your questions and your interpretations of the responses. Just as developers shouldn’t QA their own code, but instead have someone else do it, consider swapping with a colleague and conducting their research while they conduct yours. You can (and should) still watch every session as a note-taker (see that list below) and have a chance to contribute to the questions, both in the protocol itself and at the end of each session.

The single best way to improve your moderation skills:

  1. Watch or listen to recordings of yourself moderating. You can always improve, and it’s impossible to do this enough. You’ll be shocked, horrified, and amused, but you’ll always learn something.

Stuff Besides Moderating That is Critical to Good Research:

  1. Good note taking (see below).
  2. The ability to give feedback to moderators. Stanford is doing some interesting work on giving feedback (pdf, sorry), but generally be direct, specific, and respectful. And if you have something nice to say, then please do so!
  3. The ability to take feedback when moderating. If you can’t take feedback, or you can’t think of any ways you could improve, then moderating just isn’t for you. Some people are so naturally gregarious and confident that they lack the self-consciousness that’s so helpful to moderating well. Take it as a compliment and let someone else moderate.
  4. Effectively synthesizing and applying what you’ve learned. (The master, Steve Portigal, has some great material covering this as a podcast, virtual seminar, and slideshare.)

How to Be a Good Note-Taker

  1. Take notes. Seriously. You’d be surprised how many people get caught up in a session (or bored) and don’t take notes.
  2. Be discrete. In person, sit behind and slightly to one side of the moderator. Remotely, sit off screen or in another room. The goal is not to be part of the conversation, distract from the moderator, or contribute to any feeling of self-consciousness for the participant. I once was working on-site with a note-taker from my client, who in fact was a Product Manager who turned out to be uninterested in research generally. At the participants’ house, the participant sat at his table, I sat beside the table to his right, and the note-taker sat across the table from me, to the left of the participant. He found it so distracting to have someone sitting to his left that he kept turning to try to include the note-taker, and finally said, “I’m sorry, I feel like I’m being rude leaving you out of the conversation.” Don’t do that.
  3. Do not react too much. Be polite, attentive, and moderately interested. Note takers who make a big deal out of typing or writing because they aren’t doing it the whole time, or because they have a visible or audible reaction are giving the participant feedback (see above, be neutral!). Those participants may start trying to say “useful” stuff that you think is worth writing down.
  4. Remember, observations before ideas (see moderation skills).
  5. Save your questions for the end. Some ninja researchers can handle sticky notes or text messages flying at them during the session, but remember that a moderator may have a line of questioning they are pursuing. To handle your question, they have to think about how to work it in, how to ask it neutrally, and how to prioritize it against what they’re hearing from other observers, all while carrying on a conversation. Jot down your questions and then at the end of the session have the moderator ask if you have any. At that point you can do your own prioritization and practice asking neutral questions without detracting from the moderator’s work.
  6. Note your top ~5 takeaways immediately. You’ll be amazed how fast these things can run together. Jot down your takeaways so that you can remember which participant is which, and contribute to the research synthesis later.
  7. Don’t take your own notes when moderating. See #3. Taking notes is reacting. It’s one thing if a note-taker is writing or typing the whole time, but when a moderator does it, they are giving the participant feedback. Furthermore, it’s hard to gauge a participant’s reaction because you’ll be looking down at your notebook or laptop. I’ve seen participants try to read what’s being written and very obviously respond to that feedback. Again, some ninja moderators do this, but it’s better to have someone else do it, and especially if you’re new at moderating. Personally, even if I can’t have a note-taker, I’ll take my own notes later from my recording or get a transcription down.

Five Best Things to Bring to a Workshop

  1. Short lists
  2. An opportunity to practice
  3. Examples
  4. Details
  5. Cookies

Comments

Add a comment…

Note: You can use basic HTML in your comments. Your email address will never be published.