UX nerd

One of the toughest recurrent moments in my job is the “qualitative research moment”. The moment when I have to convince someone to do some in-depth user study with a few participants to produce a list of qualitative results and derive design recommendations. Whether I suggest observing users or interviewing them, the moment I stray outside A/B testing there it comes… the disbelieving look™, like I needed three degrees to become a fortune teller :S

I want to write a full post about prototype fidelity and testing methods later, and some (approximate) guidelines about when to do which thing… so I’ll try to keep digression to a minimum ;) Now, without further ado… my humble best attempt at explaining why qualitative research can be objective, reliable and produce useful insights about how users experience systems and products (and take that look off your face already, I can see it in my crystal ball and I’m not liking it ;)

First I’ll go with my favorite argument for qualitative research: all that glitters is not gold. So, if your product is on the early stages of development, it is reasonable to expect that new insights on how users interact with it will result in substantial changes. Leaving aside the 0% of the cases in which you got it right from the start (ask Mr. Gall), you *want* to be in for substantial changes in the beginning, you want to experiment and get it wrong, learn form it and bake this wisdom into your successive iterations. They key here is the word “substantial” and my assumption that you don’t have 3.7 billion years to throw away in evolution Darwinian-style. What I mean, is that you could always arbitrarily branch your project, do some quantitative research to find what works better, stick to that branch and from there repeat endlessly… 100 million species stand as evidence that this method works. But wouldn’t it be nice to know *why* and *how* something is better than some other thing and do a bit of selective breeding? (at this point I would expect your disbelief to have turned into sporadic nods) The thing is that, quantitative research, as many hard numbers as it can provide, and statistical significance and alphas and betas, can’t say anything about why something happens. It’s power is limited to who did what, where and when (ok, I’ll admit to *procedural* how too, after all it’s just a sequence of whats). Why someone did something or how he/she reached the conclusion that this is what had to be done is out of the scope of quantitative research. This doesn’t mean that we can’t get the hard numbers and then make some educated guesses about why and how things happened. But guess what? We would always be guessing ;) Qualitative analysis, which has become the black sheep of methods for actually coming out on the fact that we do interpret what we see, actually does more observing to back up this why and how interpretation than quantitative research does.

So, how do we cope with the fact that yes, we are interpreting and we’re smearing our preconceptions, our desires, our imperfections onto the facts from which we want to draw objective conclusions? Most of the criticism on qualitative research comes from bad bad stuff that happened in the 70s, when some social scientists presumably tired of abusing hard drugs moved into abusing research methods. But that doesn’t mean that now we don’t have standards to ensure that whatever influence a scientist has, it’s counterbalanced by other scientists, weeded out by the use of common classification methods (coding schemes, ontologies) or at least reflected upon, acknowledged and noted as possible weakness of the research process. Of course no one is perfect, but this also applies to quantitative research as well, and today there are countless journals and conferences that accept qualitative research papers.

Below, there are some of the methods scientists can use to safeguard the quality of their qualitative results:

• During the data collection process, log data on video/audio/etc to make sure that it’s accessible to multiple scientists for later analysis.
• If you’re going to use one, decide on a coding scheme to classify your observations and stick to it.
• Have multiple scientists collect data and if possible have multiple scientists poll the same test sources to correct any systematic bias.
• To analyze the data, always recruit more than one scientist and try to get people with differing positions.
• If in doubt, find some knowledgeable outsider to oversee your process.
• Always support your conclusions with excerpts from your original data.
• Never cite any numerical results. There are no “4 people out of 5″ in qualitative analysis. Things like “most of the people we observed” are OK because they can serve as honest leads for future direction, I trust you’ll know the difference.
• Be honest :)