A CNN-LSTM BASED DEEP NEURAL NETWORKS FOR FACIAL EMOTION DETECTION IN VIDEOS

Abstract

Human beings while communicating use emotions as a medium to understand the other person. Face being the primary source of contact while communicating and being the most communicative component of the body for exhibiting emotions, facial emotion detection in videos has been a challenging and an interesting problem to be addressed. The Facial expressions fall under the category of non-verbal type of communication and understanding Emotional state of a person through Facial Expressions has many use cases such as in the field of marketing research – understanding the customers responses for various products, Virtual classroom – understanding the comprehension level of the students, Job Interview – in understanding the changes in emotional state of the Interviewee, etc. This research paper proposes a CNN- LSTM based Neural Network which has been trained on CREMA-D dataset and tested on RAVDEES dataset for six basic emotions i.e. Angry, Happy, Sad, Fear, Disgust, and Neutral. The Faces in the videos were masked using Open Face software which gets the attention on the Face ignoring the background, which were further fed to the Convolutional Neural Network. The research focuses on using LSTM networks which have the capability of using the series of data which will aid in the final prediction of emotions in a video. We achieved an accuracy of 78.52% on CREMA-D dataset and further also tested the model on RAVDEES dataset and achieved an accuracy of 63.35%. This research work will help in making machines understand emotions, can help systems make better decisions and respond accordingly to the user.