The face is probably the first thing we look at when we meet each other. We use this part of the body daily for identifying and interacting with the world around us. One of the social functions of a face is communicating non-verbally. However, this becomes much more difficult, or in a lot of cases impossible, when the face is severely deformed. 

Last January, my colleague told me about a LinkedIn post of one of our customers, Dr. Miguel I. Dorante. He is a plastic surgery resident at Lahey Hospital & Medical Center in Burlington, MA who wrote about the study he and the team led by Dr. Bohdan Pomahac, Director of Plastic Surgery Transplantation, conducted at the Brigham and Women’s Hospital in Boston, MA, USA. The research team found that face transplant can partially restore emotional expression to patients.

The post intrigued me and I continued to explore what a face transplant actually means. I watched several stories of patients and their relatives on YouTube, which impressed me deeply. I wanted to learn more. Therefore, I read Dorante's article and reached out to him to ask some questions about his study.

Social functions of the face

A face transplant shows promising results in improved functionality of the face (e.g. ability to breath, eat, and speak) and quality of life for these patients. Nevertheless, Dorante wondered, “Would undergoing a face transplant have any value to it if somebody across the room couldn’t tell you were happy by just looking at you?” For him the answer was undoubtedly no. 

That is why the focus of his research turned to social functions of the face like having a human aesthetic and the ability to produce non-verbal communication. Dorante states that, “Research into emotional expressions has determined that there exists a universal language with cultural-specific dialects. This is suggestive of the significant role emotional expressions play in an individual’s ability to integrate into their society. 

After face transplant, the goal is societal reintegration. Up to now, this has been achieved in all cases of face transplant and has become an ethical pillar justifying the continued practice of this operation. The challenge for our group then became how to convert a subjective evaluation of a person’s facial expression to an objective measure of outcome after face transplant.” 

Measuring emotional expressions

Dorante and his team began searching for video analysis software that would be able to provide them with information on the expressive state of the video subject in an objective, noninvasive, and nonobtrusive way. The emotion expression analysis should serve as an indicator of functional outcomes and rehabilitation after face transplant. Using facial recognition technology paired with a deep learning algorithm trained to pick up and detect emotional expressions would be an ideal tool.

Indirect and direct evaluation of emotional expressions

The study included videos of six patients with face transplants (representing 15% of patients with face transplants worldwide) and six healthy controls. Two different protocols were used to either indirectly or directly evaluate emotional expressions. 

Dorante explained the difference between these protocols by using the analogy of computer software and hardware: “The transplanted face is the hardware that is composed of bones, nerves, blood vessels, muscle and overlying skin. Appropriate function of the hardware is necessary to run software. The software is the code that animates the face and allows it to run its functions appropriately. We wanted to explore if both the hardware was functional (indirect evaluation) and if the code was functional (direct evaluation).”

Direct evaluation was performed by asking patients to show the face they would make when feeling happy, sad, angry, surprised, fearful, and scared. This test would show the researchers whether face transplant could restore emotional expression but was limited by individual and cultural biases in expression. This is why they also tested their potential to perform emotional expressions indirectly by asking them to perform facial movements like a smile or a frown. 

Detect the facial expression happiness

FaceReader was used to detect and track faces, extract facial features, and analyze facial expressions. The results showed that for this group of patients, only the emotion of happiness could be reliably detected in comparison to the healthy controls. Dorante thinks that the reason for this is that it takes the fewest number of individual facial movements. ‘A smile is predominantly the only movement needed to denote happiness whereas a complex emotion like anger could involve wrinkling the nose, tightening the lips, closing the eyes and furrowing the brow.’

The research team found that expression of happiness in the first five years after face transplant was restored to a mean of 43%, compared to healthy controls. This means that you could tell every other patient with a  face transplant was happy from across the room. 

FREE WHITE PAPER: FaceReader methodology

Download the free FaceReader methodology note to learn more about facial expression analysis theory.

• How FaceReader works
• More about the calibration
• Insight in quality of analysis & output

 Download

The benefits of using FaceReader software

Using software-based video analysis to measure underlying muscle recovery indirectly via emotional expression could provide novel information about a patient’s recovery. In essence, this tool measures the return of recognizable human aesthetic and nonverbal communication in an unbiased manner. 

It can objectively provide data helpful to determine whether face transplant meets its ethical goal of restoring functions necessary for societal reintegration. Moreover, Dorante points out that the software is able to detect changes over time that were not evident to their clinical team. The ability to detect and track changes will help them determine progress in rehabilitation efforts as well as overall progress in the recovery after face transplant.

Maximize the viability and increase the longevity of face transplants 

To my question, what kind of current or future challenges are important, Dorante answers: “It is very important to continue asking more questions now that we have proven objective data can be extracted from digitizing the profile of patients with face transplants.”

He continues: “Personally, the additional value of measuring social functions after face transplant stimulates me to want to maximize the viability and increase the longevity of face transplants. Rejection of the donor allograft by the host body is the primordial challenge of transplants. Looking for markers of rejection that are not invasive or require blood work would be beneficial for efficient management that could increase the lifetime of the graft. Using image analysis software to evaluate changes to the allograft during rejection episodes and overtime would be an area worth exploring in the field of face transplantation.”

Reference

Dorante, M.I.; Kollar, B.; Obed, D.; Haug, V.; Fischer, S. & Pomahac, B. (2020). Recognizing Emotional Expression as an Outcome Measure After Face Transplant. JAMA Network Open, 3(1) https://doi.org/10.1001/jamanetworkopen.2019.19247