This blog post is written by Mayuko Kanemura. During her internship at Noldus, she performed a validation study of the use of Remote PPG in FaceReader.  She compared three heart rate measurements: RPPG, PPG, and the gold standard, ECG. 

In this blog post, she explains what RPPG is and gives recommendations on how to best measure this.

What is RPPG?

Tools for automated analysis of facial expressions provide an objective assessment of a person’s emotional expressions in consumer behavior and sensory research. FaceReader also offers other video-based functionalities that are useful for sensory and consumer research such as automated registration of the frequency and variability of heart rate.

Heart rate and heart rate variability

Heart rate (HR), the measure of how often the heart beats per minute, is influenced by the autonomic nervous system (ANS) which consists of two primary branches, the sympathetic and the parasympathetic nervous system.

The former is active when we are in an exciting situation (threat, fear, stress, and exercise) and increases heart rate. The latter is active when we are in a relaxed situation (feelings of love, compassion, and calm) and decreases heart rate. As you can see, the responses are influenced by mental and physical conditions.

Heart rate variability (HRV)

Heart rate variability (HRV) is the indicator of the balance of those conditions, being a marker of health and stress. When the sympathetic nervous system is predominant like when people feel stressed, HRV decreases. When the parasympathetic nervous system is predominant like when people are relaxed, HRV increases.

There has been growing attention for HRV as an indicator of cardiovascular disease, and internal emotions such as depression and work stress, etc. [1].

Measuring HR and HRV remotely

FaceReader can measure heart rate and heart rate variability remotely using remote photoplethysmography (RPPG), a tool to measure heart rate remotely without attaching sensors. It only requires video recording with a high-resolution camera. It can be beneficial in various kinds of physical, health and emotional monitoring, such as monitoring of drivers, consumers, the elderly, and infants [2].

Those kinds of monitoring are now available with less discomfort, using remote PPG. As part of my research, I looked at HR and HRV while participants consumed different kinds of foods, because I wanted to observe the physiological mechanism changes during food consumption.

The principles of PPG

The principles of PPG are the following: Using the example of a wearable, such as a heartrate-sensing watch, light emits to the skin from one sensor and a second sensor detects how much light is returned to the device. This forms the contrast between emitted light and reflected light. The amount of reflected light changes according to the blood volume, which is caused by capillary dilation and constriction, hence it can estimate the heart rate.

Figure 1. Optical heart rate sensing. Left: lower pressure preceding the pulse wave means narrower arteries and less absorption (higher reflectivity) of the green light source. Right: a higher blood pressure pulse causes wider arteries and more light absorption (lower reflectivity) [3]. Credit: The Conversation.

Remote PPG

How does remote PPG detect heart rate? It is the same principle but it is a contactless measurement. It measures the variance of red, green, and blue light reflection changes from the skin, as the contrast between specular reflection and diffused reflection.

Specular reflection is the pure light reflection from the skin. Diffused reflection is the reflection that remains from the absorption and scattering in skin tissue, which varies by blood volume changes.

Figure 2. The skin reflection model that contains specular and diffuse reflections, where only the diffuse reflection contains pulsatile information [4]. Credit: Wang et al. (2017). Algorithmic principles of remote-PPG. IEEE Transactions on Biomedical Engineering, 64(7), 1479–1491. DOI: 10.1109/TBME.2016.2609282

The approach to remote detection is the following:

• [Skin pixel selection] The face in the captured webcam image is detected and modeled in order to determine facial landmarks and head orientation. Subsequently, approximately the top two-thirds of the face, where most of the blood vessels are concentrated, is selected as the region of interest.
• [Signal extraction] The average of each pixel colors (red, green, blue) of the region is measured over time (both specular + diffuse reflections).
• [Signal filtering] The noise from the head motions is detected by fitting the facial model and then noise-free heart rate is produced.
• [Output calculations] By detecting peaks, inter-beat intervals are measured and then the heart rates and heart rate variability are estimated.

Figure 3. An overview of the proposed heart rate and heart rate variability estimation pipeline (left to right). The face in captured webcam images is detected and modelled to track the skin pixels in the region of interest. The average value of RGB signals is extracted over time. In parallel, the head movements are tracked and used to suppress motion noise. The process of filtering the wide and narrow band signals produces a clean pulse waveform from which peaks are detected. The inter-beat intervals obtained from these peaks are then used to compute heart rate and heart rate variability. The full analysis can be performed in real time on a computer [5]. 

FaceReader validation study

In the validation study, 25 participants were asked to watch a video (still condition) and to consume different food stimuli, ranging from liquid to solids (water, yoghurt, chicken, sausage), and the other way around. To validate the RPPG by FaceReader, BIOPAC ECG and BIOPAC PPG were used.

Within the still condition, the measurement of RPPG was successful. However, during food consumption, the differences between measurements from RPPG and ECG were bigger during consumption, indicating that RPPG underestimated higher heart rates. The movement of the face during food consumption disturbed the RPPG measurement. On the other hand, BIOPAC ECG and PPG showed high agreements.

FREE WHITE PAPER: FaceReader methodology

Download the free FaceReader methodology note to learn more about facial expression analysis theory.

• How FaceReader works
• More about the calibration
• Insight in quality of analysis & output

 Download

Recommended conditions

As a result of my research, the following conditions are recommended for FaceReader RPPG:

• Use of high resolution and uncompressed cameras, since both lead to a more accurate heart rate detection.
• Use of good light conditions, solely from a front lamp. However, avoid frontal lighting that is too strong, to prevent overexposure. A ceiling lamp causes overexposure, which interrupts the skin color detection.
• Avoid video compression. Video compression decreases the resolution and video quality.                                                                                                                                                           

Moreover, RPPG measurement is sensitive to motion of the head. It is recommended to use a setting where participants are sitting and not moving too much. A high correlation between the gold standard and RPPG in these situations has been observed in many studies. 

Because of the increasing attention to remote PPG, these drawbacks are expected to be solved, at least partly, with the improvement of algorithms in the near future.

References

1. Thayer, J. F.; Yamamoto, S. S. & Brosschot, J. F. (2010). The relationship of autonomic imbalance, heart rate variability and cardiovascular disease risk factors. International journal of cardiology, 141 (2), 122-131.
2. Wang, W.; Den Brinker, A. C. & De Haan, G. (2018). Single Element Remote-PPG. IEEE Transactions on Biomedical Engineering.
3. NeuroData Lab. (2019, March). Every beat counts. Comparing Remote Webcam Heart Rate Detector to Wearables. https://medium.com/@neurodatalab/every-beat-counts-comparing-remote-heart-rate-webcam-detector-to-wearables-d8d59aab863c
4. Wang, W.; den Brinker, A. C.; Stuijk, S. & de Haan, G. (2016). Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering, 64 (7), 1479-1491.
5. Gudi, A.; Bittner, M.; Lochmans, R. & Van Gemert, J. (2019) Efficient Real-Time Camera Based Estimation of Heart Rate and Its Variability. International Conference on Computer Vision (ICCV) Workshop.