More on Guerilla Usability? Really???

Recently, a colleague contacted me to help back him up on a usability argument he was having with engineering. The engineering lead on his project maintained that usability was much too expensive and slow for them to use with the product they were developing, especially as they’re a small startup that needs to get products out the door quickly.

With all the work that Steve Krug: “Having a small budget and someone on the project with clout who really cares about whether users have a good experience – which is often the case with an amateur site – will often get you much farther than a big budget and no one guiding the whole thing.”

Marty Cagan: “But if you’re like most companies, you have few resources available, and even less money. But you can’t let that stop you. It is absolutely essential that you test your ideas out with real users. It is arguably the single most important part of your job.”

Jared Spool: “Usability problems can cost organizations more than they save by not building a usability process.”

Jakob Nielsen: “The perception that anybody touching usability will come down with a bad case of budget overruns is keeping many software projects from achieving the level of usability their users deserve.” … and others have done in this space, I can’t believe these conversations are still happening. But they are, and my friend needed help. In particular, he didn’t need theories or a list of experts quoted as saying such-and-so. He needed hard data, something an engineer could appreciate. So this is what we worked out:

According to a 1994 Jakob Nielsen essay, Communications of the ACM estimated that the “costs required to add human factors elements to the development of software” was $128,330 [Mantei and Teorey 1988]. Nielsen hypothesized $65,330 in savings, for a total of $63,000. However our more modest and more recent estimates of a “guerilla” study broke down to much more manageable costs. Even our “expensive” version is much easier to swallow than $63k:

"Expensive" vs. "Guerilla" Usability Costs

The expensive version, with a lab, professional moderator, using a recruiter to find participants, 10 participants and edited video as an output (especially useful when making go/no-go decisions or to resolve disagreements) could easily cost $17,000 and take 80-100 hours of internal time. The “guerilla” version, in a coffee shop using friends and family, 5 participants and skipping the video could cost as little as $250 and take 20 internal hours. If you’re running them regularly, you can get it down to 3 participants and 5 hours (because a lot of the legwork will be done).

Most companies will need to conduct multiple studies and will many want to spread the net a bit more broadly than friends and family. However putting together a quick, productive and inexpensive study will go miles towards getting executives or engineers to buy in to making a little more of an investment.

The other piece of the puzzle of course is ROI. There is the cost of the study, but what is the cost of not doing the research? How many customers are they missing or loosing in the interim between launching the product and figuring out what its shortcomings are? Or where too much energy was sunk in a feature that’s never used?

Finally, my colleague was having this conversation about a product they were developing. The later in the game you’re conducting usability, of course, the higher the price in friction (resistance to change) and the real cost when you have to re-do something. That’s not to say that usability shouldn’t happen during and after development, but that research is most successful if it’s refinement and tweaks to a solid product that’s already had plenty of usability research done and changes made before development.

No word yet on whether my friends arguments were successful, but for the sake of his customers, developers, and business, I’m keeping my fingers crossed.