Of all human expressions, a smile is the most universal. It has tremendous power and can even transform your own subjective state. For example, when not in a good mood originally, smiling can increase subjective-rated happiness¹. Pretty cool, eh?

But when is a smile not a smile? When it’s literally not that. But when and how can a smile not be a smile? Simple: when it’s false.

Automated facial expression analysis

In the early 80s, Paul Ekman and Wallace Friesen² proposed that there are two types of smiles: real, or ‘felt’ smiles, and false ones. The real ones are also called Duchenne smiles, named for Guillaume Duchenne, who first characterized the facial muscles needed to make the expression happen. According to their research, felt smiles activate muscles in the eyes and false smiles do not³.

In addition, false smiles have been shown to be more unilateral, meaning the mouth curls more to one side. In the classic image below⁴, can you tell the difference between the two smiling Pauls?

Automated facial expression analysis tools, such as FaceReader⁵, can not only characterize expressions like happiness, but they can also determine the individual muscles used. The muscles, called Action Units⁶, are expressed on a categorical scale (ranging from A to E) based on the amount of muscle needed.

FREE WHITE PAPER: FaceReader methodology

Download the free FaceReader methodology note to learn more about facial expression analysis theory.

• How FaceReader works
• More about the calibration
• Insight in quality of analysis & output

Download

FACS Action Units

There are twenty eight main Action Units in the face⁷, of which FaceReader can classify twenty. Three of those, AU6, AU7, and AU12 are the three muscles needed to express happiness (see here for video). FaceReader detects the presence of a unilateral smile with AU14. Thus, using FaceReader, I wanted to test expressions from those who were really smiling or just faking it.

A video⁸ was sent to volunteers via the online platform, FaceReader Online⁹. The video, which was found via YouTube, was a series of clips of ‘fails’, which is to say, situations of physical comedy. The results were exported to FaceReader 7.1 in order to classify the Action Units.

Finally, data were exported to The Observer XT10 for segmentation and detailed analysis. Each respondent video was subsequently classified as ‘neutral’, ‘real’, or ‘false’ smiling. Data were segmented blindly, which is to say that I had no idea what muscles were active. I simply coded ‘real’ and ‘false’ based on my own expertise and knowledge of faces. Finally, the Action Unit data was exported as a percentage of time in which the AUs were active.

Real of false smiles?

The hypothesis was simple: real smiles will show more AU6 and AU7, whereas false smiles will show more AU14. In addition, the categorical expression of the AUs will be stronger in the eyes of real smiles.

Not surprisingly, that’s what I found. Real smiles showed more AU6 and AU7, whereas false smiles showed more AU14. In addition, the AUs themselves were stronger with their respective smile.

It can truly be said that you smile with your eyes. What does that mean for you and your business? Can your interviewer tell the difference between that felt or false smile when the respondent is describing her experience with your product? Maybe your respondent isn’t sure if she liked that sample, so she’s just telling you what she thinks you want to hear.

Furthermore, what about your facial expression analysis tool: Can it tell the difference? Smiling Paul was a warm-up. Let’s try another example. In the image below, can you tell which of Dominic’s smiles is real and which is false?

What if I made it even easier and showed you the AUs? Remember, AU6 and AU7 vs AU14:

You guessed it. It was the one on the right. Interestingly, FaceReader reads Dominic as happy in both images; however, only one smile was real.

So when you ask a respondent about your brand, product, or service, be sure to check their eyes. Can’t tell the difference or don’t want to have someone trained to do it? Try FaceReader with Action Unit Module. The Eagles may have said it best when they wrote, “You can’t hide your lying eyes.”

I struggled with this for many years. I have a very hard time showing a real smile in pictures. It has gotten better, but only after my friends finally told me that I looked bored and angry in pictures. The breaking point was one person commented that my face was ruining the shot! It was a joke, but many a truth is said in jest.

To fix the issue, and to keep my friends, I spent many years practicing in front of the mirror. The issue was simple: I couldn’t fake it. It seems that in order to express happiness, I must actually be happy! The practicing paid off. Now, whenever I’m forced into having my picture taken, I remember my mirror training. But now, like Harry Potter with his stag, I conjure my Petronus¹¹ and you get a smile.

See for yourself; progress:

Oh: Still wondering about which Smiling Paul was the correct one? It was the Paul on the left.

References

1. https://www.psychologytoday.com/us/blog/prefrontal-nudity/201208/smile-powerful-tool
2. Ekman, P. & Friesen, W.V. J Nonverbal Behav (1982) 6: 238. https://doi.org/10.1007/BF00987191
3. https://www.psychologytoday.com/us/blog/thriving101/201001/what-science-has-say-about-genuine-vs-fake-smiles
4. Retrieved from https://sonnemann.files.wordpress.com/2012/05/paul-ekman1.png
5. https://www.noldus.com/human-behavior-research/products/facereader
6. https://www.noldus.com/facereader/facial-action-units
7. https://en.wikipedia.org/wiki/Facial_Action_Coding_System
8. https://www.youtube.com/watch?v=yRB_UBef-sA&t=124s
9. https://facereader-online.com/Default.aspx
10. https://www.noldus.com/human-behavior-research/products/the-observer-xt
11. https://www.pottermore.com/writing-by-jk-rowling/patronus-charm