I want to write a full post about prototype fidelity and testing methods later, and some (approximate) guidelines about when to do which thing… so I’ll try to keep digression to a minimum ;) Now, without further ado… my humble best attempt at explaining why qualitative research can be objective, reliable and produce useful insights about how users experience systems and products (and take that look off your face already, I can see it in my crystal ball and I’m not liking it ;)

First I’ll go with my favorite argument for qualitative research: all that glitters is not gold. So, if your product is on the early stages of development, it is reasonable to expect that new insights on how users interact with it will result in substantial changes. Leaving aside the 0% of the cases in which you got it right from the start (ask Mr. Gall), you *want* to be in for substantial changes in the beginning, you want to experiment and get it wrong, learn form it and bake this wisdom into your successive iterations. They key here is the word “substantial” and my assumption that you don’t have 3.7 billion years to throw away in evolution Darwinian-style. What I mean, is that you could always arbitrarily branch your project, do some quantitative research to find what works better, stick to that branch and from there repeat endlessly… 100 million species stand as evidence that this method works. But wouldn’t it be nice to know *why* and *how* something is better than some other thing and do a bit of selective breeding? (at this point I would expect your disbelief to have turned into sporadic nods) The thing is that, quantitative research, as many hard numbers as it can provide, and statistical significance and alphas and betas, can’t say anything about why something happens. It’s power is limited to who did what, where and when (ok, I’ll admit to *procedural* how too, after all it’s just a sequence of whats). Why someone did something or how he/she reached the conclusion that this is what had to be done is out of the scope of quantitative research. This doesn’t mean that we can’t get the hard numbers and then make some educated guesses about why and how things happened. But guess what? We would always be guessing ;) Qualitative analysis, which has become the black sheep of methods for actually coming out on the fact that we do interpret what we see, actually does more observing to back up this why and how interpretation than quantitative research does.

So, how do we cope with the fact that yes, we are interpreting and we’re smearing our preconceptions, our desires, our imperfections onto the facts from which we want to draw objective conclusions? Most of the criticism on qualitative research comes from bad bad stuff that happened in the 70s, when some social scientists presumably tired of abusing hard drugs moved into abusing research methods. But that doesn’t mean that now we don’t have standards to ensure that whatever influence a scientist has, it’s counterbalanced by other scientists, weeded out by the use of common classification methods (coding schemes, ontologies) or at least reflected upon, acknowledged and noted as possible weakness of the research process. Of course no one is perfect, but this also applies to quantitative research as well, and today there are countless journals and conferences that accept qualitative research papers.

Below, there are some of the methods scientists can use to safeguard the quality of their qualitative results:

• During the data collection process, log data on video/audio/etc to make sure that it’s accessible to multiple scientists for later analysis.
• If you’re going to use one, decide on a coding scheme to classify your observations and stick to it.
• Have multiple scientists collect data and if possible have multiple scientists poll the same test sources to correct any systematic bias.
• To analyze the data, always recruit more than one scientist and try to get people with differing positions.
• If in doubt, find some knowledgeable outsider to oversee your process.
• Always support your conclusions with excerpts from your original data.
• Never cite any numerical results. There are no “4 people out of 5″ in qualitative analysis. Things like “most of the people we observed” are OK because they can serve as honest leads for future direction, I trust you’ll know the difference.
• Be honest :)

I’m working on a project that involves some ergonomics, more specifically it requires or would benefit from the label “comfortable”. Like we always do, we designed a test, collected participants, drafted consent forms, prepared the facilities… and then… the unexpected. To my embarrassment, we had to repeat our whole biomechanics experiment because we had gathered our results in a manner that didn’t afford any meaningful analysis. This is the brave account of what went wrong and how we solved it, which I send into the world hoping that at least one less designer will stumble against this cheeky stone ;)

Like I said, our goal was to determine if a particular physical interaction we were designing was “comfortable”. So, what did we do? I’m not going to explain exactly in what the experiment consisted, but the idea was have people try it and then use some validated questionnaires to tell us if they had felt physical discomfort during or after the tasks we proposed. The questionnaire had a scale with a few ordinal values (uncomfortable, moderately uncomfortable, slightly uncomfortable, comfortable). What can go wrong? Well, in the first place, we were surprised to see that our participants had go through considerable more discomfort than expected during our experiment. We were confused because we had tried the experiment while designing it and none of us had had *any* discomfort whatsoever. Still it could be that overall the system was comfortable enough, we could still do some kind of analysis: we had our ordinal variables… Here was where our lucky misfortune saved us from dataitis (dataitis is what you get when you forget that information is data+meaning, and I can’t emphasize that “+meaning” enough). The crazy discomfort outcome made us uneasy, clearly something had happened there, maybe Hawthorne but also maybe something else. We started questioning our method and the someone said

and what if it actually is uncomfortable? what if all [interactions of this kind] are uncomfortable? what if for [this kind of interaction] comfortable just means less uncomfortable than average?

This made sense, maybe when we tried the experiment we hadn’t found it uncomfortable because we knew in which context the interaction belonged and our users didn’t. But this also opened a whole new set of questions:

• What is comfortable? How should we measure it?
• If all interactions of this kind are uncomfortable, and we measure with our ordinate categorical scale and aggregate the results, we’re going to find that our interaction is indeed uncomfortable to some degree but does it mean that it’s not comfortable enough?
• And even worse, if all interactions of this kind (including ours) are comfortable, and we determine that our interaction is indeed comfortable through our experiment, will it still be comfortable out there in the market is someone does it better?

We were lucky, if the results had been positive, we would have never reflected on this: if you’re going to measure a subjective quality you have to do it in comparison to something else. In other words: if there’s no scale, you have to create your own. Users’ pronouncements on subjective qualities can measure the improvement of a product over time because there’s a past to which new results can be compared, and they can only measure how good a product is if there are other products to compare with.

So what we did was: we repeated the same test with two additional alternatives for the kind of interaction that we wanted to test. We did a within-user randomized test (with 12 participants), and asked users to rank all three interactions in the comfort scale. But there’s another tricky bit yet to come… how do we analyze the data? In these cases, one can be tempted to do the following things (all examples of things I’ve seen done, and even published!)

• Convert the ordinal variables to numerical scores and use a Wilcoxon signed-rank test. This would be wrong because… the fact that you express an ordinal scale in a way that looks more like an interval scale does NOT turn you data into interval variables!! A scale that goes from uncomfortable to comfortable is not, and will never be, an interval scale because the *difference* between a value and the one immediately following is undefined. The only thing we know is that for each individual, “slightly uncomfortable” means more comfortable than “moderately uncomfortable” and this is it, we don’t know and there is no way to know how much more.
• A t-test would not only be wrong on the same grounds as the Wilcoxon signed-rank, but also because you can’t assume the distribution to be normal. Using dependent t-tests in cases like this is something I’ve seen done and published many times :-(
• The Mann-Whitney U test. Mann-Whitney is at least a non-parametric test. This means it works for ordinal data. However Mann-Whitney requires *mutual independence within and between samples*, which is not the case here. As the results where gathered in a within-user test, the way participants used the scale depended on their appreciation of the range of comfort provided by the three interactions and the rank they gave each interaction was definitely affected by its comparison to the other two. So Mann-Whitney is not a choice.

Maybe there are many right choices (and maybe you can think of more possible wrong choices), but this is what we did: a Friedman test. The Friedman test is a non-parametric test used to compare observations repeated on the same subjects. The Friedman test is probably the littlest-known piece of math by Nobel prize winner economist Milton Friedman, you just have to have a look at the newspapers to see why people cared more about his demonstration of the complexity of stabilization policy… but for User Experience the Friedman test is key. A test that can answer a simple but powerful question:

N users rate k different products. Are any products ranked consistently higher or lower than the others?

Some months ago, I asked you to complete a questionnaire about your tabbed browsing habits. Thanks! I got 99 responses. What follows is a brief summary of how I analyzed the data, the results I got and some analysis. Later we used the results to design an alternative to tabs for the Mozilla Labs Design Challenge summer 09 (with Maria and some help from Nico), but I’ll write about that later.

First things first: where did the data come from?

• I posted a link to the questionnaire on reddit
• The link was sent to the mailing list of a software company
• Some of my colleagues also answered the questionnaire

So you can expect my data to be biased towards the savvy user’s experience, which is also something I appreciate given the nature of my research. I would like to use these results to guide me through a redesign of the current browsing paradigm (tabs). As far as I could read, there is (or there was) nothing wrong with tabs themselves; however, as people start to simultaneously open an increasingly large number of tabs and their browsing becomes more complex, tabs become simply not enough. It’s savvy users who are pushing the envelope here, and it’s them whom I mostly want to listen to this time.

How I analized the data:

To analyze the data I used Weka.  Weka is an open source software for data mining tasks, which I used in it’s GUI-like form. It contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. With it, I tried to find some patterns in my data. I don’t really believe in averaging user data, so I clustered my users/participants to find several user profiles. The idea is then to try to come up with a browsing paradigm that is flexible enough to cater, ideally, to all groups.

I used  k-means and got 4 clusters, representing 4 kinds of users (39%, 26%, 22% and 12% of participants in each respective cluster).

This is what my clusters look like:

Or, into words:

• Cluster 0: users that have mostly a limited number of tabs, 6 to 12, of which some are permanently open. Some of the organize tabs, some don’t, but overall there is no preferred organizing method for this group (methods vary across users). They mostly rely on visual cues to locate tabs, which makes sense given that the limited number of tabs enhances title legibility and increases the chance that favicons will be easily discriminable (less chance that any two or more favicons are alike).
• Cluster 1: users that have mostly more than 12 open tabs, of which some are permanently open. Some of them organize their tabs, some don’t. The ones who do seem to have a combination of methods, although they can be singled out for their preference for read/unread organization (is this because opening new tabs on the right end is a default in many browsers?). They largely rely on visual cues and trial and error to locate tabs, and (as opposed to the other group with a large number of tabs) the lack of a single/fixed/permanent semantic structure in their organization seems to hinder their ability to use associative memory for retrieval.
• Cluster 2: they use a very limited number of tabs (up to 6) and don’t have permanently opened tabs. They don’t organize their tabs and retrieve them using various methods (mostly visual cues, sequential search and localized trial and error). They don’t rely on memory, but probably they don’t need it (searching through 6 items can’t take so long).
• Cluster 3: users with more than 12 open tabs, of which some are permanently open. They all organize tabs and create a semantic structure: by subject and by same parent are very popular choices. They also organize their tabs according to frequency of use. Their use of arbitrary and associative memory is a lot higher than that of the other groups, which seems to correlate to their fixed/permanent structuring (frequency of use) and semantic structuring (by subject, by parent).  However, they also rely heavily on visual cues. Going back to the original data, everyone who listed using visual cues in this group also listed some other method, so it looks like visual has a supporting function helping the user refine the search (first letters of the title, thumbnail, favicon, etc) once memory has made the first approach.

Finally: What do I think this means?

There’s an interesting 1983 paper called “How do people organize their desks?” by Thomas Malone from the Xerox Palo Alto Research Center. This paper explores how people organize their documents (not only in their desks but also tables, shelves, cabinets, etc.) and the strategies they use to find them. Two extremes were identified. Some people just had stacks of things to do arranged in ill-defined groups, their main criteria was things that require an action of their part (and the priority of this action) vs. other things. Other people relied on information organizing and stored documents in clearly defined and titled files. This is what Malone called, respectively, piles (groups of elements arranged in no particular order) and files (groups of elements arranged with systematic order). Files are titled and things are found by looking for them in the category where they belong. Pile identification is aided by their spatial location.

Now I’m going a bit into my own interpretation of some of the things he found (like doing some unorthodox rephrasing), I think they’re definitely interesting but some of the notions need updating to fit digital platforms:

• People organize their desks so they can find things, but an equally important function of the organization is reminding. If I see it I’ll attend to it. People with messy offices do indeed have more problems finding information and remembering tasks.
• Items related to pending tasks are singled out.
• Classifying information is cognitively taxing and this difficulty is one of the forces leading the creation of loosely defined piles.
• Thinking of the context in which you last saw something helps remembering where it is. Context can be many things: who was I working with, when, what was I also working on at the time?
• Even if you organized stuff really well, that doesn’t mean that you’ll be able to find it right away: the criteria used to file it somewhere may not be the same criteria you may want to use to retrieve it (let’s say you filed according to the project the document belongs to but the want to find everything one particular person worked on). Malone found this occurred in 2/3 of the cases he analyzed. This means that organizing things doesn’t always pay, and this is why Google changed our lives at the time Yahoo was a web directory.

So, the takeaways are, basically: organizing into categories (even tagging) is hard and doesn’t always pay, users should not be forced to do it; visibility, visualization and spatial arrangeability are the killer features providing reminding power, context awareness and findability; items with pending actions should be somehow singled out; and, finally, a flexible search that does not rely on tags, labels or categories is a must.

Looking closely at clusters 1 and 3, the ones with an amount of tags that merits some organization, there’s a strange effect. The tab bar is is an unfriendly environment to organize stuff because of it’s unidimensionality, and because it does not facilitate absolute spatial arrangements at all: some wrap around creating a second/third/fourth line of tabs drastically changing a tabs position as you add more and, even worse, some extend infinitely into an imaginary space on the right outside the browser window hiding some tabs and then you navigate this macabre strip back and forth never knowing where you are. So, why am I saying this? Because it looks to me, that if technology allowed it, cluster 1 would be pilers and cluster 3 would be filers, and this, sprinkled with the joy of a flexible search engine, could be as close to browser Disneyland as it gets for tab management. Why do I think this? Well, cluster 1 people have a loose organization, based mostly on read/unread and a clearly visual in their location strategy preferences, however piles are not afforded by the current interfase so there’s no explicit grouping and no way to spatially arrange tabs so they don’t really meet the exact definiton of pilers (yet). On the other hand, cluster 2 people do organize tabs into categories in meaningful ways but this organization is not currently supported by the browser so all the labelling and category delimitation is just kept in their minds so they are not real filers (yet). And the Disneyland bit? Well, not being a cognitive psychologist or qualified in any formal way, I’m just guessing that this piler/filer stuff and the preference to spend time puting things where they belong so you know where to find them vs. to spend time searching for them is just something wired into people’s brains. I have no hard evidence, but my intuition and the fact that my results pretty much match Malone’s (in a different domain) point this way. And, wouldn’t the best interfase be the one that affords the users’ natural preferences and behavior? And, wouldn’t this be the case more so when we’re talking about preferences and behaviors that are wired into the user’s brain?

So, for the Mozilla Labs Design Challenge, our goals are:

• To build an interfase that caters to pilers and filers alike: a flexible interfase, just like Malone’s observed offices, that will afford the strategies of pilers, filers and everyone in between.
• To use the digital world’s capabilities to solve the shortcomings that these strategies have in real life.
• To use the information gathered in this study to intoduce features relevant to user preferences (permanently opened tabs, visual search, etc.)

THIS SURVEY IS NO LONGER ACTIVE