Rose Faghih

Listening to music and drinking coffee are the sorts of everyday pleasures that can impact a person’s brain activity in ways that improve cognitive performance, including in tasks requiring concentration and memory.

That’s a finding of a new NYU Tandon School of Engineering study involving MINDWATCH, a groundbreaking brain-monitoring technology.

Developed over the past six years by NYU Tandon's Biomedical Engineering Associate Professor Rose Faghih, MINDWATCH is an algorithm that analyzes a person's brain activity from data collected via any wearable device that can monitor electrodermal activity (EDA). This activity reflects changes in electrical conductance triggered by emotional stress, linked to sweat responses.

In this recent MINDWATCH study, published in Nature Scientific Reports, subjects wearing  skin-monitoring wristbands and brain monitoring headbands completed cognitive tests while listening to music, drinking coffee and sniffing perfumes reflecting their individual preferences. They also completed those tests without any of those stimulants. 

The MINDWATCH algorithm revealed that music and coffee measurably altered subjects’ brain arousal, essentially putting them in a physiological “state of mind” that could modulate their performance in the working memory tasks they were performing. 

Specifically, MINDWATCH determined the stimulants triggered increased “beta band” brain wave activity, a state associated with peak cognitive performance. Perfume had a modest positive effect as well, suggesting the need for further study. 

“The pandemic has impacted the mental well-being of many people across the globe and now more than ever, there is a need to seamlessly monitor the negative impact of everyday stressors on one's cognitive function,” said Faghih. “Right now MINDWATCH is still under development, but our eventual goal is that it will contribute to technology that could allow any person to monitor his or her own brain cognitive arousal in real time, detecting moments of acute stress or cognitive disengagement, for example. At those times, MINDWATCH could ‘nudge’ a person towards simple and safe interventions — perhaps listening to music  — so they could get themselves into a brain state in which they feel better and perform job or school tasks more successfully.”

The specific cognitive test used in this study — a working memory task, called the n-back test — involves presenting a sequence of stimuli (in this case, images or sounds) one by one and asking the subject to indicate whether the current stimulus matches the one presented "n" items back in the sequence. This study employed a 1-back test — the participant responded "yes" when the current stimulus is the same as the one presented one item back — and a more challenging 3-back test, asking the same for three items back.

Researchers tested three types of music - energetic and relaxing music familiar to the subject, as well as novel AI-generated music that reflected the subject’s tastes.  Consistent with prior MINDWATCH research, familiar energetic music delivered bigger performance gains — as measured by reaction times and correct answers — than relaxing music. While AI-generated music produced the biggest gains among all three, further research is needed to confirm those results.

Drinking coffee led to notable but less-pronounced performance gains than music, and perfume had the most modest gains.

Performance gains under all stimulations tended to be higher on the 3-back tests, suggesting interventions may have the most profound effect when “cognitive load” is higher.

Ongoing experimentation by the MINDWATCH team will confirm the efficacy of the technology’s ability to monitor brain activity consistently, and the general success of various interventions in modulating that brain activity. Determining a category of generally successful interventions does not mean that any individual person will find it works for them.

The research was performed as a part of Faghih’s National Science Foundation CAREER award on the Multimodal Intelligent Noninvasive brain state Decoder for Wearable AdapTive Closed-loop arcHitectures (MINDWATCH) project.  The study's diverse dataset is available to researchers, allowing additional research on the use of the safe interventions in this study to modulate brain cognitive states.

Faghih served as the senior author for this paper. Its first author is Hamid Fekri Azgomi, who earned his Ph.D. under Faghih and is now a postdoctoral scholar of neurological surgery at the University of California San Francisco School of Medicine.

Wearable monitoring is likely to play a key role in the future of healthcare. In many cases, wearable devices may monitor our physiological signals that can indicate mental states, such as emotions. The lab of Rose Faghih has been developing a system called MINDWATCH, algorithms and methods for wearable sensors that collect information from electrical signals in the skin to make inferences about mental activity. While their lab has been successful in translating these physiological signals quickly and effectively, they didn't incorporate direct feedback from the individual’s subjective experiences.

Now, the researchers are incorporating feedback and labels from the users, enhanced with machine-learning, and combining it with their existing model for a fuller, more accurate picture of mental states.

At a high level, the human body can be viewed as a complex dynamical system. It is a complex conglomeration of control systems that each works in turn to manage different variables or states.  Unfortunately, a number of the states researchers are interested in — particularly the ones that are more abstract — cannot be measured directly. These include states of emotion, cognition and consciousness. Nevertheless, changes in these unobserved states do give rise to corresponding changes in different physiological signals that can be measured more easily. For instance, we may not be able to directly observe or measure a person’s emotional state, but we can measure subtle changes in a person’s heart rate, breathing or sweat secretions (which in turn affects the conductivity of the skin). These signals can then be used to estimate the states we wish to. 

However, the signals researchers use to estimate these unobserved states are “spikey” or “pulsatile” in nature. These spikey signals can be used to estimate the various states of the human body and brain without direct observation.  With already-existing methods, you could obtain state estimates, but still wouldn’t have any means to have those estimates agree with the more direct state-related information in possession. This is especially important in experiments involving human subjects where the subjects can indeed provide information related to the unobserved state. This type of more-direct state-related information can be called a “label.”

Incorporating direct feedback from users offers information that can’t be gleaned from biological data alone. For instance, a person with PTSD could have their skin conductance continuously monitored to provide an emotion estimate, but ideally, the final estimate should rely both on these signals and information perhaps obtained on rating scales or through regular questionnaires. It is likewise the case for patients with hormone disorders. Hormone measurements do provide valuable information, but should likely be combined with personal feedback (e.g. regarding feelings of energy/lethargy) to obtain a single complete picture. The authors met this need through a proof-of-principle work on a hybrid type of estimator. 

Performing estimation on some data such that what is predicted agrees with available labels falls within the domain of supervised machine learning. This work adapted an existing neural network method for state estimation by adding a penalization term for not agreeing with the labels to enable a hybrid estimator. The proposed hybrid estimator was utilized to determine an aspect of emotion tied to changes in skin conductance (through changes in sweat secretions) and to determine energy states within the body based on pulsatile hormone secretions. A wearable monitoring system that incorporates verbal feedback from the user with physiological signals for hybrid estimation can eventually provide a more complete picture of the user to eventually provide more comprehensive closed-loop care. 
 

Stress has a negative impact on physical health, reduces work productivity, and results in significant annual costs for industries and healthcare. While high stress is known to raise the risk of cardiovascular disease and have negative effects on mental health, It also has key effects on the ability of one to complete tasks by both excessively high or excessively low stress. There has been growing research interest on understanding how real-world stress impacts our body and performance, at work and across life activities

Unfortunately, attempts to simulate their impact in the laboratory or elsewhere are less useful than datasets gathered in real-world circumstances. As a result, researchers have access to fewer real-world stress datasets. Even rarer indeed are such datasets used in longitudinal investigations on the same subjects over time.

Real-world situations are also unrestricted environments. Research-grade equipment is frequently inaccessible, and motion artifact contamination is pervasive. These continue to be some of the biggest barriers to automated emotion decoders outside of the research labs in daily life.

To address the above-mentioned gap, Rose Faghih and her former PhD students Md. Rafiul Amin and Dilranjan Wickramasuriya performed an experiment, in which a set of students' physiological data was gathered over the course of three exams. They used a smartwatch-like wearable device and collected multimodal physiological data. The use of the smartwatch-like wearable device was to provide a seamless data collection experience for the students participating in the experiment. 

The investigation shows that it is possible to link the variations in the physiological signals to the exam performance. More details about this study can be found in the corresponding publication titled "A Wearable Exam Stress Dataset for Predicting Grades using Physiological Signals."

To enable other researchers, use this dataset for additional investigations, the research team has made the de-identified data publicly available on the PhysioNet platform. A Wearable Exam Stress Dataset for Predicting Cognitive Performance in Real-World Settings is available at: physionet.org/content/wearable-exam-stress

Ultimately, the researchers believe it would be extremely beneficial to consider how exam performance and the stress that goes along with it interact. It will allow for a wide range of potential applications with the aim of enhancing personal performance. This may, for instance, assist scientists  in developing effective interventions to improve each person's performance and increase productivity within a company. Additionally, the knowledge may be used in online and remote learning contexts to connect with students effectively and improve learning outcomes.

Emotions directly influence the way we think and interact with others in different situations. Additionally, when it interferes with rationality in our decision-making or perception, emotions play an undeniable role. 

From another perspective, studies from different fields indicate that human emotions could be impacted by any changes in the environment. Thus, having a solid understanding of the dynamics of emotions could benefit individuals in multiple ways: from the enhancement of therapeutic solutions and maintaining well-being to the prevention of unpleasant emotions and inducing favorable ones. Decoding emotions has been an ongoing research among researchers, dictating joint efforts from behavioral, physiological, and computational angles.

According to Russell's circumplex model, emotions are defined in a two-dimensional arousal-valence circular space. Compared to the arousal index, which designates the severity of the emotional state, internal valence levels determine the quality of emotions. In a high arousal mood, feeling excited or happy is opposed to nervousness and feeling stressed. On the flip side and in low arousal conditions, staying calm stands on the opposite side of feeling bored. Hence, feeling pleasant, which refers to the positive valence, is one of the key factors in one’s health condition and could result in a better quality of life. 

In a new piece of research, a team led by Rose Faghih, Associate Professor of Biomedical Engineering, follows the goal of establishing a mechanism to track an individual's emotional valence level and regulate it in a closed-loop framework. Since the emotional valence state is hidden and cannot be measured directly, the authors need to measure it indirectly. 

The use of wearable devices and designing decoders to gain insight into the internal brain states provides a good alternative to directly measuring brain activity. Inspired by recent advances in wearable technologies which could measure various physiological signals from humans in the loop, there exists possibilities to track emotional valence state indirectly and in a more practical manner. Among different physiological signals, it has been shown that facial electromyography (EMG) carries valuable information about the internal valence state. According to various research, facial EMG measurements of the zygomaticus muscle (zEMG) give the most distinct indicator of emotional valence state. In fact, changes in the zygomaticus muscle, which could be captured via wearables, could provide researchers with valuable information regarding one’s valence level.

In this research, the authors employ a mathematical framework to relate changes in zEMG signal for inferring the internal valence state. They extract corresponding features from the zEMG signal and estimate the internal valence state. Taking advantage of tracking of emotional valence state in a systematic way, they then propose a knowledge-based control architecture to close the loop and keep the estimated valence state within a favorable range. Within a closed-loop approach, biomarkers are collected to provide information regarding the individual's valence state to then automatically adjust the actuators in response to the current emotional state.

In silico results verify that, by making use of physiological data, the proposed controller could effectively regulate the estimated valence state. Ultimately, it is envisioned that the future outcomes of this research will include supporting alternative forms of self-therapy by using wearable machine interface architectures capable of mitigating periods of pervasive emotions and maintaining daily well-being and welfare. This study is the very first closed-loop control framework for regulating emotional valence state using facial muscle activity. This research suggests an important new step toward new clinical applications and the self-management of mental health. In the practical implementation of the proposed architecture, a wearable device would collect zEMG signal, and the appropriate practices (e.g., listening to music, breathing exercises) would be suggested to maintain the emotional valence levels within the favorable range.

Faghih, who joined NYU earlier in 2022, has been integrating research into her classroom teaching. The research covered in this Frontiers in Computational Neuroscience paper started as a course project in Faghih’s “State- Space Estimation with Physiological Applications” course at the University of Houston. In this class students had the opportunity to use advanced methods learned in the course to analyze physiological data in hands-on biomedical engineering projects designed by Faghih. Student authors Luciano Branco and Arian Ehteshami were graduate students in the course, and Hamid Fekri Azgomi, who recently completed his PhD under the supervision of Faghih, was a PhD student mentor for the project.