A long time ago I attended a workshop with Alan Dix that applied this principle to leverage ancient (failed) wisdom for design inspiration: how to produce a good idea out of a bad idea. How does it work? You can have a look at the example Tomaso, Ting, Maria, Valentina and I worked on.

Objective:

To create a good idea starting from a bad idea; by analyzing the bad idea’s positive and negative aspects, identifying the forcings behind them, and using these forcings in a context in which they constitute positive aspects of a good idea.

The process:

We started by brainstorming for bad ideas. From these ideas we chose one, and traded the rest with other teams, receiving in turn two borrowed bad ideas. Then, we considered their positive, negative and neutral aspects, what causes the systems to behave in these ways, and how the context within the system affects our valuation (positive, neutral or negative) of these aspects.

The bad ideas:

1. Dinner payment chain: A payment system in a restaurant in which a customer pays the bill of the previous customer and cannot leave until he/she have found someone that pays his/hers.

• Pay-it-backward. As opposed to pay-it-forward, which usually requires honesty and proactivity on the part of the participant, pay-it-backward ensures that the chain will be maintained (positive). However, in this case it does not add any benefit compared to the mainstream alternative, in which each customer pays their own bill (neutral).
• Captivity within the chain. On the other hand, in this case, pay-it-backward is inconvenient for the customers who have to stay until someone pays their bill because they may have to spend time and effort looking for the next link in the chain (negative).
• Blindness about cost. Customers are also blind as to how much their meal will cost, because instead of paying the items they ordered, they will be paying another customer’s. This can cause fear to end up losing (negative).
• Snowball effect. Customers will consume a lot because they do not want to end up losing money, and for this they need to consume more than the customer for whom they are paying. In general terms, this means that as people feel they have already paid for something, they will make full use of it to feel they got their money’s worth. This is a positive for the restaurant owner that will see the business grow but a negative for the customers that will end up paying increasing amounts with each iteration.

2. Managing system for football fans (borrowed idea): A system which allows supporters to decide opponent’s team formation through voting.

• Public involvement. People participate in the decision making. They feel more engaged because the results they see are the consequence of their involvement (positive).
• People act on a narrow vision and inadvertently modify the system. People seek to satisfy immediate and individualistic goals, so they choose the worst players for the opposing team and as a result the quality of the whole league degrades (negative).

3. Beeper for restaurant kitchens (borrowed idea): A communication system for a kitchen in which all natural language communication is replaced with messages coded in “beeps”.

• More structured system. In a carefully engineered system it is easier to predict what others will do and cooperation becomes more smooth (positive), on the other hand people may feel trapped or unable to improvise (negative).
• Simplicity, limited communication and behavior. What can be said through beeps is much more limited than what can be said through natural language. This limits people’s freedom to communicate, and in turn it limits their freedom to act when they are working in a team and have to coordinate their actions (negative).
• Overly oriented to goals (avoiding socializing). This does not foster a very agreeable work environment (negative).
• Cognitive effort to remember codes. People have to learn a new language, which requires effort and raises the entry barrier (negative).
• Beeps are annoying (negative).

After the analysis, we combined some of the aspects mentioned above to develop a good idea. Below, we present the idea, and list the aspects of the bad ideas it includes. In the context of this idea, these aspects are all positive.

The good idea: A pay-it-backward vaccination campaign

Background: Vaccines do not work on all the individuals in a population. To be sure that a disease does not spread and all individuals (even those in whom the vaccine does not work) are safe, a population has to achieve herd immunity. This occurs when vaccination is widespread enough so that if one person becomes sick with an infectious disease the likelihood that he/she will be able to pass it on to another person is very low because virtually all the people around this person will be immune to the disease. This means that for an individual to be safe from infectious disease, his/her own behavior is not enough, he/she needs the collaborative behavior of the community.

The idea: We though of a system in which person-A gets vaccinated and pays for the vaccine. In turn, if person-A refers person-B to go and get the vaccine, person-A will get a refund for the cost of the vaccine. Each person wants to get a refund, what works to create a chain reaction that results in most people getting vaccinated and the population achieving herd immunity.

Below, we explain how the aspects of the bad ideas are integrated into this good idea.

• Pay-it-backward. The motivation to get a refund for something that has already been paid ensures that the chain will be maintained.
• Captivity within the chain. People have to spend time and effort looking for the next link in the chain, but this is compensated by the fact that they are sure to earn something in return (the refund).
• Blindness about cost. People pay for their vaccine and finally get a refund, so they perceive that the vaccine has a cost but that they get it for free. This has the double effect of attributing value to the vaccine and making the person feel they have drawn a benefit by not paying the cost. In reality, because all people entitled to a refund and finally the State (through a tax-funded program) ends up paying for all the vaccines, there is an unknown cost (the cost of the vaccine) paid indirectly by each person. But in this case, blindness about the real cost could be used by the people engineering the vaccination campaign to set an arbitrary “price” to the vaccine, maximizing the vaccine’s value perception and people’s motivation to get the refund.
• Public involvement. People can see in the whole of society good results that are the consequence of their involvement, this makes people feel empowered and predisposes them better to new initiatives.
• People act on a narrow vision and inadvertently modify the system. People have an immediate individual incentive (to get a refund) and the acting in pursuit of this immediate individual incentive creates a much larger collective effect: herd immunity. This occurs in this case because the individual and collective goals are aligned.
• More structured system. By creating a system in which individual and collective goals are aligned, the system can be expected to reach the desired equilibrium state without a huge external investment. The system just needs to be started and then each actor will perform its part until the whole system changes state. In this case, for example, although the State would pay for the vaccines (which we assume it would have done anyway) it would save the costs of public health campaigns, advertising, education, etc. to persuade the people to get vaccinated because each individual can be trusted to have an incentive to pass the message along for free.

I want to write a full post about prototype fidelity and testing methods later, and some (approximate) guidelines about when to do which thing… so I’ll try to keep digression to a minimum ;) Now, without further ado… my humble best attempt at explaining why qualitative research can be objective, reliable and produce useful insights about how users experience systems and products (and take that look off your face already, I can see it in my crystal ball and I’m not liking it ;)

First I’ll go with my favorite argument for qualitative research: all that glitters is not gold. So, if your product is on the early stages of development, it is reasonable to expect that new insights on how users interact with it will result in substantial changes. Leaving aside the 0% of the cases in which you got it right from the start (ask Mr. Gall), you *want* to be in for substantial changes in the beginning, you want to experiment and get it wrong, learn form it and bake this wisdom into your successive iterations. They key here is the word “substantial” and my assumption that you don’t have 3.7 billion years to throw away in evolution Darwinian-style. What I mean, is that you could always arbitrarily branch your project, do some quantitative research to find what works better, stick to that branch and from there repeat endlessly… 100 million species stand as evidence that this method works. But wouldn’t it be nice to know *why* and *how* something is better than some other thing and do a bit of selective breeding? (at this point I would expect your disbelief to have turned into sporadic nods) The thing is that, quantitative research, as many hard numbers as it can provide, and statistical significance and alphas and betas, can’t say anything about why something happens. It’s power is limited to who did what, where and when (ok, I’ll admit to *procedural* how too, after all it’s just a sequence of whats). Why someone did something or how he/she reached the conclusion that this is what had to be done is out of the scope of quantitative research. This doesn’t mean that we can’t get the hard numbers and then make some educated guesses about why and how things happened. But guess what? We would always be guessing ;) Qualitative analysis, which has become the black sheep of methods for actually coming out on the fact that we do interpret what we see, actually does more observing to back up this why and how interpretation than quantitative research does.

So, how do we cope with the fact that yes, we are interpreting and we’re smearing our preconceptions, our desires, our imperfections onto the facts from which we want to draw objective conclusions? Most of the criticism on qualitative research comes from bad bad stuff that happened in the 70s, when some social scientists presumably tired of abusing hard drugs moved into abusing research methods. But that doesn’t mean that now we don’t have standards to ensure that whatever influence a scientist has, it’s counterbalanced by other scientists, weeded out by the use of common classification methods (coding schemes, ontologies) or at least reflected upon, acknowledged and noted as possible weakness of the research process. Of course no one is perfect, but this also applies to quantitative research as well, and today there are countless journals and conferences that accept qualitative research papers.

Below, there are some of the methods scientists can use to safeguard the quality of their qualitative results:

• During the data collection process, log data on video/audio/etc to make sure that it’s accessible to multiple scientists for later analysis.
• If you’re going to use one, decide on a coding scheme to classify your observations and stick to it.
• Have multiple scientists collect data and if possible have multiple scientists poll the same test sources to correct any systematic bias.
• To analyze the data, always recruit more than one scientist and try to get people with differing positions.
• If in doubt, find some knowledgeable outsider to oversee your process.
• Always support your conclusions with excerpts from your original data.
• Never cite any numerical results. There are no “4 people out of 5″ in qualitative analysis. Things like “most of the people we observed” are OK because they can serve as honest leads for future direction, I trust you’ll know the difference.
• Be honest :)

I’m working on a project that involves some ergonomics, more specifically it requires or would benefit from the label “comfortable”. Like we always do, we designed a test, collected participants, drafted consent forms, prepared the facilities… and then… the unexpected. To my embarrassment, we had to repeat our whole biomechanics experiment because we had gathered our results in a manner that didn’t afford any meaningful analysis. This is the brave account of what went wrong and how we solved it, which I send into the world hoping that at least one less designer will stumble against this cheeky stone ;)

Like I said, our goal was to determine if a particular physical interaction we were designing was “comfortable”. So, what did we do? I’m not going to explain exactly in what the experiment consisted, but the idea was have people try it and then use some validated questionnaires to tell us if they had felt physical discomfort during or after the tasks we proposed. The questionnaire had a scale with a few ordinal values (uncomfortable, moderately uncomfortable, slightly uncomfortable, comfortable). What can go wrong? Well, in the first place, we were surprised to see that our participants had go through considerable more discomfort than expected during our experiment. We were confused because we had tried the experiment while designing it and none of us had had *any* discomfort whatsoever. Still it could be that overall the system was comfortable enough, we could still do some kind of analysis: we had our ordinal variables… Here was where our lucky misfortune saved us from dataitis (dataitis is what you get when you forget that information is data+meaning, and I can’t emphasize that “+meaning” enough). The crazy discomfort outcome made us uneasy, clearly something had happened there, maybe Hawthorne but also maybe something else. We started questioning our method and the someone said

and what if it actually is uncomfortable? what if all [interactions of this kind] are uncomfortable? what if for [this kind of interaction] comfortable just means less uncomfortable than average?

This made sense, maybe when we tried the experiment we hadn’t found it uncomfortable because we knew in which context the interaction belonged and our users didn’t. But this also opened a whole new set of questions:

• What is comfortable? How should we measure it?
• If all interactions of this kind are uncomfortable, and we measure with our ordinate categorical scale and aggregate the results, we’re going to find that our interaction is indeed uncomfortable to some degree but does it mean that it’s not comfortable enough?
• And even worse, if all interactions of this kind (including ours) are comfortable, and we determine that our interaction is indeed comfortable through our experiment, will it still be comfortable out there in the market is someone does it better?

We were lucky, if the results had been positive, we would have never reflected on this: if you’re going to measure a subjective quality you have to do it in comparison to something else. In other words: if there’s no scale, you have to create your own. Users’ pronouncements on subjective qualities can measure the improvement of a product over time because there’s a past to which new results can be compared, and they can only measure how good a product is if there are other products to compare with.

So what we did was: we repeated the same test with two additional alternatives for the kind of interaction that we wanted to test. We did a within-user randomized test (with 12 participants), and asked users to rank all three interactions in the comfort scale. But there’s another tricky bit yet to come… how do we analyze the data? In these cases, one can be tempted to do the following things (all examples of things I’ve seen done, and even published!)

• Convert the ordinal variables to numerical scores and use a Wilcoxon signed-rank test. This would be wrong because… the fact that you express an ordinal scale in a way that looks more like an interval scale does NOT turn you data into interval variables!! A scale that goes from uncomfortable to comfortable is not, and will never be, an interval scale because the *difference* between a value and the one immediately following is undefined. The only thing we know is that for each individual, “slightly uncomfortable” means more comfortable than “moderately uncomfortable” and this is it, we don’t know and there is no way to know how much more.
• A t-test would not only be wrong on the same grounds as the Wilcoxon signed-rank, but also because you can’t assume the distribution to be normal. Using dependent t-tests in cases like this is something I’ve seen done and published many times :-(
• The Mann-Whitney U test. Mann-Whitney is at least a non-parametric test. This means it works for ordinal data. However Mann-Whitney requires *mutual independence within and between samples*, which is not the case here. As the results where gathered in a within-user test, the way participants used the scale depended on their appreciation of the range of comfort provided by the three interactions and the rank they gave each interaction was definitely affected by its comparison to the other two. So Mann-Whitney is not a choice.

Maybe there are many right choices (and maybe you can think of more possible wrong choices), but this is what we did: a Friedman test. The Friedman test is a non-parametric test used to compare observations repeated on the same subjects. The Friedman test is probably the littlest-known piece of math by Nobel prize winner economist Milton Friedman, you just have to have a look at the newspapers to see why people cared more about his demonstration of the complexity of stabilization policy… but for User Experience the Friedman test is key. A test that can answer a simple but powerful question:

N users rate k different products. Are any products ranked consistently higher or lower than the others?

When I looked for information on device parsing on the Internet, I found several slightly different definitions (for example, this one). I think that the idea is still evolving, and Bert Bongers says that he’s still working into validating it, which is why it’s probably still changing. Anyway, I’ll just refer to what we did in the lecture.

Device parsing is a method that consists on taking apart some device in order to understand its functionality (and potential functionality) through the analysis of its components. The goal of this is to eventually, once you have understood it, build it up again taking the user into account.

So, we started by thinking about what people may want to use the device for (its functionality):

In the case of the alarm clock, primarily:

• See what time it is
• Wake up to it (it is an alarm clock)

But also:

• Check the date (it has that functionality)
• Check the temperature (it has a thermometer)
• Use the light  to see in the darkness of their rooms

You probably note that some of these are weird, like checking the temperature. At least I can’t imagine why I would like to check the temperature in my own room, I mean I’m not relying on temperature measurements to see if I want to put on a sweater or not, for me it’s just a matter of am I warm or cold. However, for some reason someone felt it should be there. Others, like using the clock’s light to look for something in the dark, are weird because they were probably not intended by the manufacturer. But I’m sure I’m not the only one who clicks on the light button to help me look for stuff when I don’t want to turn the lights on.

Then, we took the clock apart to take a look at the components, finding (no surprises here) that alarm clocks are in their electronic, and more precisely digital, technological stage. Why is this relevant? Because every technological stage has it’s own interaction constraints and capabilities. The interaction that you can realistically introduce at each stage is different and, even if you decide to leap on to the next one, you may want to know why you’re doing it.

Next, we took a look at the input and output devices (in the clock, mainly buttons, but also the screen, the buzzer, etc.) and the different components. We analized how the input and output devices relate to the components inside and the functionality, determining the way in which the tasks associated with each function are performed. The idea is that you get to know why things behave the way they behave. Was something the designer’s choice or was it a technological constraint? Could it have been built otherwise? Why does the same button serve as snooze and light on button? What would it take to make things differently?

Finally we put it together again and took a look at the feel: how the interaction is affected by the components. In the case of the clock, the light/snooze button (on top) had haptic feedback, but the others (on the front, used to change the settings) made a noise when touched, for example.

So, this was it. Now we know why this clock behaves the way it does, what are its technological capabilities and what would take to make it behave differently and we’re ready to undertake it’s redesign putting it together with the user in mind. I agree that it doesn’t make much sense with a clock, probably you can tell what’s inside without opening it up. However the concept looks interesting and I’m looking forward to using it if our suggestion to redesign a mobile phone’s interface goes through (fingers crossed :)

• The book, or part I: The Design of Everyday Things
• The clock, part II: an intro to device parsing, and the parsing of a clock as example
• The toaster, part III: our redesign of the toaster, using a different combination of sensors and actuators, following Norman’s principles

(I’ll leave talking especifically about sensors/actuators for some other day, I have a rather ambicious project on them, but it’ll have to wait until I finish with the emoticons and the pagination :)

So, today: the book.

The Design of Everyday Things is a book, written by Don Norman in 1988, about how some designers are bad designers and create things that give some people trouble when they have to use them. Some models and principles are also presented in the book. I guess that you can read as many positive as negative reviews, its current relevance is also praised as well as disputed. I didn’t generally like the book and I wonder how scientific the main models presented in the book are. Even Norman admits that his 7 Stages of Action model is not validated and that the study of how people act is an “unexplored area”. The book seems to be based mainly on the cherry picking of anecdotes, some folk wisdom about how people act, a rather large bibliography list that is never cited in the book  and lots of hope that you’ll identify with some of the episodes described there in which people are frustrated and/or embarrased by their inability to make some device work and you’ll find designers guilty by association. Although just a reshuffle of common sense and other people’s papers, the book is kind of successful in bringing some concepts together, it would actually be useful if it weren’t so long and obfuscated. I don’t know, this is just my opinion. In any case, I summarized the concepts in the book, scraping off Norman’s rants, the pseudoscientific explanations and the stuff which is too old to matter, and I got the summary that follows, which is reasonably ok as very basic advice for  a designer.

First there is the 7 stages of action this is the model suggested by Norman on how people act (chapter 2). If you then look back into the principles of chapter 1 and combine them into it and then add the contents of chapters 3,4 and 5, you get something like this (download as PDF here):

Some other concepts are described in the book, but frankly most of them are obvious or redundant: i.e. what people can do after physical and cultural constraints are taken into account is called by Norman an affordance, like in glass affords seeing through. Or, the notion that tasks described by wide and deep decision trees are complex vs. tasks with shallow or narrow trees, which are the simple ones…

The chapter on errors is kind of weird, the explanation about why and how people tend to overregularize the commonplace and at the same time overemphasize the discrepant is lacking, especially regarding the part about the discrepant stuff. However, I think that his categorization of the nature of slips (mishaps, or errors arising from acting incorrectly when one has the correct intention) may be useful as a checklist I may like to use to test my designs for possible causes for this kind of errors:

• Capture errors: a frequently done activity replaces a more infrequent one if the early steps for both are similar (i.e. I start to dial the dentist’s number and end up calling my friend because the first 4 digits in both phone numbers coincided).
• Description errors: the intended action is incompletely described in the user’s formulation of his/her goal, so it gets replaced by another one that also fulfills the incomplete description (i.e. put the lid of the sugar bowl onto the coffee cup, in this case the goal description would have been something like “put the lid on”).
• Data driven errors: sensory data of a similar nature replaces data that originally was involved in the action (i.e. you try to call a person but you see someone else and you call the latter’s name instead of the former’s).
• Associative activation errors: same as data driven but the “wrong data” comes from the person’s own associations instead of from the external world, the Freudian slip.
• Loss of activation errors: forgetting the goal, as a consequence the sequence of actions is truncated because the next step is unknown (i.e. I walk to the kitchen but once I get there I can’t remember why I went there or what I wanted to do there).
• Mode errors: a device has more than one mode and the user performs an action as if the device were in one mode when it is on another (i.e. trying to write text in the command mode of Vim)

And finally, after a chapter 6 that is absolutely not worth reading, in chapter 7 Norman gives his definition of User Centered Design, which consists of adhering to 7 principles (some of the ones presented in chapter 1 are there, some are missing, the others are new):

1. Use knowledge in the world (cues, labels, etc.) and in the head (user’s mental models, associations and memory).
2. Simplify the structure of the tasks (provide mental aids, improve visibility and feedback, automate parts of the task, replace the task by a simpler one)
3. Make things visible (bridge the gulfs of execution and evaluation)
4. Get the mappings right
5. Exploit the power of constraints, both physical and cultural
6. Design for error (minimize causes of error, make actions reversible, make it easy to discover that an error ocurred, take errors as an approximation through which the user gets to the goal by an alternative course of action)
7. When all else fails standardize (so arbitrary things have to be learn only once).

What can I say about these 7 principles..? For sure that reading through them was puzzling. Why didn’t he include some of the advice that he mentions on the rest of the book? Where are conceptual models and where is feedback? They looked good enough to be included as principles in chapter 1, and surely they are part of User Centered Design: providing a good conceptual model through which your user can undertand the capabilities and functionality of your design. Somehow, some things got left out and others just appeared out of the blue: when reading chapter 2 I had the impression that constraints were part of the knowledge in the world, they are in fact a subsection of that chapter, why are they being mentioned here as a separate entity? At best, chapter 7 looks sloppy. It reinforces the impression that there is no structure behind the book; which, together with the fact that in my edition (Basic Books, 2002) you can’t even tell the hyerarchy of subtitles, makes me wonder if Norman wouldn’t have forgotten to include the part about providing a good conceptual model for your users if he had paid a little bit more attention to it…

In any case, would I advice myself to read it if I ever go back in time? The answer is, unfortunately, no. Not completely useless, but I have better things to do with the time it takes to read 235 pages that can be summarized into one A4 sheet.

I think, because that’s how the process goes in my head, I’ll divide considerations into practical and statistical. I just want to list the factors that need to be taken into consideration and how to reach the best compromise, I guess I’ll leave worrying about the validity of introspective task analysis for some other day :P

I’ll start with the practical factors: money and time. The things that need to be taken into account are:

• Duration of the actual experiment for each participant
• How long does it take to analyze data for each participant? (dependant on kind of media, coding method and type of analysis)
• How much does each participant cost? (test costs + experimenter’s hours + monetary rewards for participants if any)

These may seem obvious but, for example, the time for data analysis is very often underestimated and rarely (seriously) considered a function of the number of participants. Something these factors have in common is that they are all limiting factors. Practical factors are going to tell you how many participants you can (or, more accurately, can’t) have, but how many do you need? Hopefully, that’s what statistical considerations should tell you, or in other words: how good you need your results to be.

So, let’s say that you want to test new interface against an old one, you may want to know if the mean number of errors that people make using the new interface (μ2) is smaller than for the old interface (μ1). I’m going to use this example because I think it’s pretty common, the same principles apply to other kinds of tests. So your hypotheses look like this:

H0: μ1 =< μ2

H1: μ1 > μ2

And you test n users with the old interface and n users with the new interface.

Generally, you’d choose what you want your type I error (α, or probability of rejecting H0 if H0 is true) to be.   The idea behind this is that if you found H1 to be true you’d be 100(1-α)% sure that you’re right, so that’s why α is the kind of thing that you’d like to choose arbitrarily. So once that you choose an α, you end up with a β (probability of type II error, or of failing to reject H0 if it’s false). I don’t think that an extensive explanation of why β is determined by your choice of α would be relevant here, this is the quick one.  If you get too meticulous about the evidence you need to say that the new interface is better (low α) it’s more likely that you’ll end up discarding the claim even if it’s true (high β) just because you’re being too picky. On the other hand, if you definitely don’t want to miss the new interface in the case it’s better (low β), you will adopt it even when the evidence is not so strong, and it’s more likely that you adopt it even if it’s not really better (high α). I hope this makes sense to you, if not you can read more about it here.

So β depends on the α you choose but what other things does it depend on? If we’re considering the scenario in which you had to reject H0 but didn’t (this is what type II error, β, is all about), then μ1 > μ2. This means that there is actually a δ = μ1 – μ2. The image on the below shows a possible distribution of the errors made by the users in the old interface (left) and the new interface (right) when there is a δ, the vertical line . If you look at the image you can easily see how as the difference the means, δ,  increases, (i.e. the curve in the right moves further right) the type II error (which is proportional to the purple area) decreases. So β will not only depend on our choice of α but also on how big the effect that the experimenter is trying to detect is (to which extent the new interface is better than the old one).  Simply put: big effects are easier to detect than small effects.

The image above also shows how β depends on the population variances. It’s easy to see that “flattening” the curves (increasing variance) would increase the purple area.

However, given our choice of α and the fact that the difference in the means and variances are inherent characteristics of the  populations, it would seem that there is little we can do to further improve our experiment (i.e. decrease β), that is until we take a closer look at what happens when we change sample size. To do this, I’ll assume that you agree with me that, if you have an experiment like this, you probably want to do a two-sample t-test. A t-test is what you would use when your populations have a t-distribution, which means that you have a population that ideally should have a normal distribution but you sample is too small to accurately estimate the variance (when the sample is infinte, the t-distribution matches the normal distribution because you know the exact variance). The image below shows how the t distribution changes with sample size (k).

When the sample size (k) increases, the curve rises and gets slimmer, the same effect we would get with a decreasing variance, making the type II error smaller (the purple area shrinks). So sample size is the variable you can manipulate to get results you’re more confident with.

So, how do you do it? How much is enough? Luckily, some nice people came up with something called Operating Characteristics Curves. OC curves show the probability of failing to reject Ho as a function of the difference between means ( δ) and the variance (σ) for different numbers of participants. You can see an example in the image below.

The example in the image shows the probability of accepting Ho as a function of d = (μ1 – μ2)/σ, for different sample sizes (different curves) for the two-sided t-test with an α of 0,05. You can see there that if increasing sample size from 5 to 15 may be a good idea, you may want to think twice about going from 15 to 20 (you may want to if you want to detect very small effects) even if the amount of time and money you’re adding to you your original investment is the same in both cases.

So once you have your α and you have estimated the variance (which you have to do for your test anyway), you decide what is the minimum difference in means ( δ) that you would like to detect and with which confidence you want to be sure that you’ll indeed detect it (β). When you have this information, the procedure goes like this:

• Look for the right OC curve set for your α
• Calculate d
• Find the point in the graph whith coordinates (d, β)
• Move horizontally to the left until you meet the closest OC curve, that curve will give you the number of participants you need (n)
• Then move down to look up the d that will give the β you want for the curve you just chose. This d is the minimum difference in means that you can be (1-β)100% confident to detect

Finally, check that the sample size you found you need according to statistical considerations is within the limits you set due to practical considerations (if not, adjust your budget or expectations and repeat ;-)