In any case, last month, Toon, Dominika, Valentina, Maria and I designed and run an experiment to test if emoticons are actually the facial expressions of IM. Are they used in the same context a real facial expression would be appropriate? Is there a correlation between use of emoticons and real facial expressions produced during an IM conversation? This is how we did it and our results.

We had these two research questions:

• If the number and type of emoticons is correlated to facial expressivity (the number of instances of facial expressions) for a given instant messaging conversation.
• If emoticons aid instant messaging text communication by a mechanism equivalent to facial expressions during face to face communication (i.e. if they appear in a context when the facial expression they depict would be appropriate, natural and spontaneous).

We had the following setup. In the experiment, 10 people participate in an instant messaging conversation with an experimenter. Their faces are recorded and merged with a screen cast of the chat window, so facial expressions and the use of emoticons are recorded for each of them.

The topic of the conversations was fixed to “childhood memories”, because we thought that a topic that  involved remembrances would trigger the expression of emotions. Participants were chosen among acquaintances of the experimenters to facilitate the expression of emotions and provide a more natural setting for chatting (we read that social context is highly determinant of expressivity in computer mediated communication). Participants also had different nationalities and cultural backgrounds. And they were chosen so their ages were between 20 and 30 years old, to increase the likelihood that they are frequent users of emoticons (statistics say this age bracket is where most frequent emoticon users are).

Then we analyzed the data. We had planned to use the synchronicity of the video and the chat to see if emoticons and facial expressions were produced simultaneously. However, the software had been badly configured at the usability lab-not by us-and the videos were out of sync with the chats (we found out by chance, it was very hard to tell). On a side note, I wonder what happened with all the people who used the faulty software in the lab but never found out, what scientific breakthroughs may we be missing :P Later we found these tips for screen recording that I recommend you take a look at if you’re ever going to use Camtasia. In any case, we didn’t have synchronized videos so we had to figure out if there was anything we could do with the data we had collected.

The first research question was easy, we just had to count the number of emoticons and instances of facial expressions and see if there was a correlation. We did that, comparing the number of facial expressions with the number of emoticons sent by the participants, received by the participants (or sent to them by the experimenters) and then the total number of emoticons. Then we compared the number of emoticons in each category (smiling/joy, laughing, thinking, irony/joking, confusion, surprise, sadness, anger, disgust, fear, embarrassment) with the total number of facial expressions.  There was no correlation whatsoever: neither the number of sent/received/total emoticons nor the number of any of the different types of emoticons correlates to the facial expressivity of people during an IM conversation. Strike one.

So, fine, maybe the number or type of emoticons we send and receive has nothing to do with our general emotional expressivity during one conversation (in terms of the number of facial expressions). But it may still be that when they are used emoticons work in the same way as facial expressions (if this is not clear, you may want to pay a visit to my friend M.P. ;) This would have been very easy to prove if the videos and the chats had been synchronized (an emoticon occurrs at the same time that a facial expression or not), but because they weren’t we had to get creative.

What we did was test for a correlation between the number of emoticons in each emoticon category and the number of facial expressions that this emoticon category depicts (smile emoticons against actual smiles, etc.) and then we also counted the number of fillers and onomatopoeias in each of the conversations (you’ll see why later). If emoticons act as facial expressions, we can expect all emoticons of one type to be linked to one instance of the facial expression they depict. Also, there may be extra facial expressions of that category: the noise. You can take a look at our paper and see why we expect this noise to be randomly distributed across participants. If this noise is randomly distributed accross participants then we can still spot a trend if there is one. But again, we found no correlation at all. Although this is not as conclusive as our original experiment (the synchronous one) would have been, it seems that emoticons are not used in the same context than the facial expressions they depict. Strike two.

Finally, we found something that may not be statistically significant at all but is worth exploring in a next experiment. It looked like its possible that when people use less emoticons they use more fillers and onomatopoeias. We can’t be sure because we didn’t control the experiment for this. By visual inspection it seems that people are replacing emoticons with fillers and onomatopoeias, but when less emoticons were used the experimenters also used more fillers. The problem is that apparently people are wired to empathize through mimicking. Mimicking is an important part of making a connection during communication and we can’t help adopting some of the non-verbal quirks of our interlocutors (you can look up these people for further information on this: Buck, 1984; Burgoon et al., 1995; Chartrand & Bargh, 1999). And because we didn’t control for this variable, it’s impossible for us to say if our participants tended to use more fillers and onomatopoeias when not using so many emoticons because they are interchangeable and can thus replace one another or because they were just mimicking the experimenter :( But we think it’s worth exploring. Fillers and onomatopoeias are voicings and hence belong to the sound realm. If emoticons are interchangeable with them, it would be strike two-and-a-half.

Although the synchronous experiment is still pending, and this would be the only way to get conclusive evidence to discard that emoticons are equivalent to facial expressions, all our evidence points to emoticons and facial expressions being different things. There’s even research that suggests that people don’t even use their face-recognition machinery in the brain to decode emoticons. But, are they punctuation marks? I don’t know, some research would be needed to determine this. What would be the consequences? I don’t know either, I’m not much into affective stuff. But it looks interesting. Not only from a linguistic perspective, but also because if this is truly something new (like a new necessary disambiguation method, a new dimension to punctuation that changes the way we communicate, etc.) then undertanding how it works can inspire portability to other areas, and new features and products. Maybe the Nokia glow is just the beginning, maybe mediated communication is just the beginning of emoticons, or maybe saying “lol” out loud is just the beginning… only time will tell… how freakish our communication can get.

In the meantime, if you have some time to kill, you can download our paper form here :)