IRJET - Emotion Recognition from Facial Expression using Deep Learning

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 08 Issue: 08 | Aug 2021

p-ISSN: 2395-0072

www.irjet.net

Emotion Recognition from Facial Expression Using Deep Learning Deby Sarker1, Tania Sultana2, A.S. Tariq3 12B.Sc.

Students, Dept. of Computer Science and Engineering, Bangladesh Army University of Engineering & Technology (BAUET), Natore-6431, Bangladesh 3Senior Lecturer, Dept. of Computer Science and Engineering, Bangladesh Army University of Engineering & Technology (BAUET), Natore-6431, Bangladesh ---------------------------------------------------------------------***----------------------------------------------------------------------

Abstract - Deep learning based emotion detection gives

performance better than traditional methods with image processing. It discusses about the procedure of emotion detection, which includes basically three main steps: face detection, feature extraction, emotion classification. This paper presents the design of an artificial intelligence (AI) system capable of emotion through facial expression. In the deep Convolutional Neural Network (CNN) fully-connected layers we employ a regularization method called “dropout” that proved to be very effective to reduce over-fitting. The system has been successfully classified seven basic emotion classes. Thus, the proposed method is proven to be effective for emotion recognition.

Fig. 1: Facial Expression Recognition System Flowchart.

The next step after detecting a face from a given image is the feature extraction process that helps isolate the important facial areas. Two methods can be used for feature extraction: • Analytic approach. • Holistic approach.

Key Words: Convolutional Neural Network; Deep Learning; Artificial Intelligence; Emotion Recognition.

The holistic approach uses raw facial image as input, while the analytic approach focuses on some of the important facial features are been detected and extracted from face. The analytic approach is being used in this paper, where we the extracted selected features that were obtained using edge detection from the image are sent as input to a classifier. The holistic approach takes the global properties of the patterns being obtained into consideration while the analytic approach we used computes a set of geometrical features based on feature vectors.

1. INTRODUCTION Human beings communicate with each other in the form of speech, gestures and emotions. As such systems that can recognize the same are in great demand in many fields. With respect to artificial intelligence, a computer will be able to interact with humans much more naturally if they are capable of understanding human emotion [1]. It would also help during counseling and other health care related fields.

Emotion is a mental state connected with the nervous system linked with feeling, perceptions, behavioral reactions, and a degree of pleasure or irritation. One of the present applications of artificial intelligence (AI) using neural networks is the recognition of faces in images [2]. Most techniques process visual data and search for normal pattern present in human faces in images. Face detection can be used for surveillance purposes by law enacts as well as in crowd administration [3]. In this paper, we present a method for identifying seven emotions such as neutral, anger, sad, disgust, fear, happy and surprise using facial images.

Previous used deep-learning technology to create models of facial expressions based on emotions to identify emotions.

3. BACKGROUND STUDY 3.1 Related Work On the following paragraphs, approaches involving the use of feature engineering are introduced. While the approaches on feature extraction and classification are different; all of them involved Cohn-Kanade dataset, as part of its work. It is worth to mention that Cohn-Kanade dataset was used in this research, so the results give a valuable comparison.

Kotsia et al. focused on the effect of occlusion when classifying 6 facial emotion expressions. In order to achieve this, several feature engineering techniques and classification models were combined. Gabor features, which is a linear filter used for edge detection, and Discriminant Nonnegative Matrix Factorization (DNMF), which focuses on the nonnegativity of the data to be handled, are the feature extractors techniques. To classify these features multiclass support vector machine (SVM) and multi-layer perceptron (MLP) were used. The results over Cohn-Kanade are the following: Using a MLP with Gabor 91.6% and with DNMF: 86.7%. While using SVM it was 91.4%. Another corpus used was JAFFE:

2. FACIAL EXPRESSION RECOGNITION Identifying or verifying a person expression is the main task of facial expression recognition which is Combined with some important steps. Figure 1 demonstrates the basic process of facial expression recognition.

© 2021, IRJET

|

Impact Factor value: 7.529

|

ISO 9001:2008 Certified Journal

|

Page 701

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 08 Issue: 08 | Aug 2021

p-ISSN: 2395-0072

www.irjet.net

Gabor combined with MLP achieved 88.1% and when using it with DNMF, it resulted on 85.2% classification accuracy [7], [8].

Wang and Yin examined how the distortion of detected face region and the different intensities of facial expressions affect robustness on their model. Topographic context (TC) expression descriptors were selected in order to perform feature extraction. This technique performs a topographic analysis. In this analysis, the image is treated as a 3D surface.

Each pixel is labeled taking into account its terrain features.

The use of several classifiers was reported: quadratic discriminant classifier (QDC), linear discriminant classifier (LDC), support vector classifier (SVC) and Naive Bayes (NB).

Results using a Cohn-Kanade subset (53 subjects, 4 images per subject for each expression, 864 images): with QDC: 81.96%, with LDA: 82.68%, with NB: 76.12%, with SVC: 77.68%. Results in MMI facial expression dataset (5 subjects, 6 images per subject for each expression. 180 images) were also reported: with QDC: 92.78%, with LDA: 93.33%, and with NB: 85.56% [5], [6].

Fig. 2: Rectified Linear Unit (ReLU).

3.3 Convolutional Neural Network A Convolutional neural network is a neural network comprised of convolution layers which does computational heavy lifting by performing convolution. Convolution is a mathematical operation on two functions to produce a third function. It is to be noted that the image is not represented as pixels, but as numbers representing the pixel value. In terms of what the computer sees, there will simply just be a matrix of numbers. The convolution operation takes place on these numbers. We utilize both fully-connected layers as well as convolutional layers. In a fully connected layer, every node is connected to every other neuron. They are the layers used in standard feed forward neural networks.

3.2 Deep Learning DL has achieved success on fields such as computer vision, natural language processing, and automatic speech recognition. One of the main strengths of using DL techniques is that there is no need for feature engineering. The algorithms are able to learn features by themselves over basic representations. Then, the algorithm will determine if certain pixel combination represents any particular feature, that is repeated through the image. As the data is processed through the layers, the features will go from very abstract forms to meaningful representation of objects. Finally, a deep learning technique that is relevant for this project is presented: Convolutional Neural Networks (CNN).

They help recognize patterns in the data. For each filter, there are two main properties to consider - padding and stride.

Stride represents the step of the convolution operation, that is, the number of pixels the window moves across. Padding is the addition of null pixels to increase the size of an image.

Null pixels here refer to pixels with value of 0. If we have a 5x5 image and a window with a 3x3 filter, a stride of 1 and no padding, the output of the convolutional layer will be a 3x3 image.

This condensation of a feature map is known as pooling. In this case, “max pooling” is utilized. Here, the maximum value is taken from each sliding window and is placed in the output matrix.

3.2.1 Rectified Linear Unit The rectified linear unit function (ReLU) provided benefits compared to previous common activation functions: its computational cost was cheaper, it provided a good error approximation and it did not suffer from the gradient vanishing problem. ReLU is displayed on Figure 2 defined as: f(x) = max (0, x).

However, a major drawback when using ReLU is its fragility when the input distribution is below zero. This happens when the neuron reaches a point when it will not be activated by any data point again during training.

3.2.2 Use of GPU The use of GPU for training has become fundamental for training deep networks because of practical reasons. The main reason is the reduction of the training time compared to CPU training While different speedups are reported depending on the network topology, it is common to have around 10 times speed when using GPU.

© 2021, IRJET

|

Impact Factor value: 7.529

Fig. 3: Convolutional Neural Network Architecture.

3.3.1 Convolution The process by which adding each element of the image with its local neighbors, weighted by the kernel is called Convolution. This is related to a form of mathematical convolution. It should be noted that the matrix operation

|

ISO 9001:2008 Certified Journal

|

Page 702

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 08 Issue: 08 | Aug 2021

p-ISSN: 2395-0072

www.irjet.net

being performed in convolution but it is not traditional matrix multiplication and need a kernel for multiplying. It is simply denoted by *. The representation is, ( , ) ( , )=

third convolution layer use 4×4 kernel rather than 5×5 which gives non-linear capabilities and better result. Weights getting from convolution layer are trained using backpropagation algorithm. ReLU function is an activation function which derives the outputs of Convolutional Neural Network neurons. For faster and deeper training ReLU works better than any other traditional activation function.

(𝑠,) ∗ 𝑓(x− , 𝑦 − ) . . . . . . . . . . . (i)

3.3.2 Pooling The process of combine the outputs of neuron clusters at one layer into a single neuron in the next layer is called pooling.

In image classification pooling is an important method to achieve high performance. Figure shows some pooling operation.

4.1.2 Pooling Layer Pooling layer is situated after convolutional layer and reducing the spatial dimension of input which is needed in next convolutional layer. The technique which is performed by pooling layer is called ‘down-sampling’ and for the reduction of input size some information may loss from the data but this loss is beneficial for CNN to create easy computation process for next layer and work against overfitting. As like as convolution operation pooling operation is performed. Some pooling operations are max pooling, min pooling and average pooling. For effective performance characteristics max pooling has a great favor compare with others. Pooling is an optional process in convolutional neural network, many architecture ignore this operation. This operation is express by,

Fig. 4: Max Pooling & Min Pooling.

𝑥𝑗 𝑙-1 = (𝛽𝑗 (𝑥𝑗 𝑙-1 ) + 𝑏𝑗 ) . . . . . . . . . . . . . . . (ii)

The fundamental terminologies and ingredients of two most successful algorithms of face recognition which are describe above. It can be helpful for the reader to give a clear understanding about those algorithms.

4. METHODOLOGY

Max pool layer will perform a down sampling operation along the spatial dimensions (width, height), resulting in volume.

Max-Pooling with 3×3 filter and stride 2 with 64 filters, and size 5×5, stride 1.

4.1 Facial Expression Recognition Using CNN

4.1.3 Fully Connected Layer

For large amount of data, Convolutional Neural Network (CNN) works well. Again it is a deep learning algorithm. In a Convolutional Neural Network, multiple layer is used for feature extraction.

At the end of Convolutional neural network fully connected layer is placed and combines multiple logical regression public cells. As likes as its name it is in the last and connected with the output of previous layer. To produce desire output and establish connection between output layers is main work of fully connected layer.

For this the result of facial expression recognition is more effective and efficient. Steps of Convolutional Neural Network algorithm are mentioned below -

4.2 Proposed CNN Architecture The networks are program on top of keras, operating on Python, using the keras learn library. This environment reduces the code’s complexity, since only the neuron layers need to be formed, rather than any neuron. The software also provides real-time feedback on training progress and performance, and makes the model after training easy to save and reuse. In CNN architecture initially we have to extract input image of 48*48*1. The network begins with an input layer of 48 by 48 which matches the input data size processed in parallel manner through two similar models that is functionality in deep learning, and then concatenated for better accuracy and getting features of images perfectly as shown in Figure 5 which is our proposed model, Model-A. There are two sub models for the extraction of CNN features which share this input and both have same kernel size. The outputs from these feature extraction sub-models are flattened into vectors and linked to one long vector matrix and transmitted to a fully connected layer for analysis before a final output layer allows for classification. This models

Step 1: At first input the data for training dataset.

Step 2: Pre-process the data for effective result in face detection.

Step 3: Face is detecting in this step.

Step 4: After face detection data is training using CNN model.

Step 5: After training dataset face class is predicted by probability.

4.1.1 Convolutional Layer In one module convolutional neural network combine segmentation, classification and feature extraction which made this method unique. Convolutional layer calculates the output of all neurons that are associated to local regions in the input layer, each calculating a dot product among their weights and a small region they are associated to in Convolution with 64 filters, size 5×5, stride 1.The second and

© 2021, IRJET

|

Impact Factor value: 7.529

|

ISO 9001:2008 Certified Journal

|

Page 703

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 08 Issue: 08 | Aug 2021

p-ISSN: 2395-0072

www.irjet.net

contains convolutional layer with 64 filters each with size of [3*3], followed by a local contrast normalization layer, maxpooling layer, followed by one more convolutional layer, max pooling, flatten respectively. After that we concatenate two similar models and linked to a soft max output layer which can classify seven emotions. We use dropout of 0.2 for reducing over-fitting. It has been applied to the fully connected layer and all layers contain units of rectified linear units (ReLu) activation function.

Fig. 6: Various Emotional Reactions.

5.2 Dropout Dropout minimizes the impact of units that have a strong activation. This method shuts down units during training, so other unit can learn features by itself. Providing with more independence to all units reduces the strong unit bias leading to strong regularization and better generalization.

Fig. 5: Proposed CNN Architecture.

5.3 Network Loss & Learning Rate

First we are passing our input image to convolutional layer which consists of 64 filters each of size 3 by 3, after that it passes through local contrast normalization can remove average from neighborhood pixels leads to get quality of feature maps, followed by ReLu activation function.

The loss chart presented in Figure 6.1 shows how the cost function is minimized over training steps. During the first 450 steps, the loss is reduced smoothly. After that point, the curve starts to converge. On the step 800, the loss value is 0.79; while on step 900, loss value is 0.77. It is just a reduction of 0.02 on a hundred steps. Given this behavior, it can be said that the model has converged. It means that a longer training won’t reduce the networks loss substantially. For instance, using a smaller learning rate.

Here, we use batch normalization, dropout, ReLu activation function, categorical cross entropy loss, Adam optimizer, soft max activation function in output layer for seven emotion classification. Input image size is adjusted to that 128*128*3.

The network starts with an input layer of 128 by 128 which matches the input data size parallel processed through two similar models as shown in Figure 5.

Furthermore, it is concatenated and passed through one more soft max layer for emotion classification and all procedure is same as above. In Model-B, previously proposed

by Correa et al, the network starts with a 48 by 48 input layer, which matches the size of the input data. This layer is preceded by one convolutional layer, a local contrast normalization layer, and one layer of max pooling, respectively.

Fig. 7: Training Accuracy Loss & Validation Accuracy Loss for the Proposed CNN Architecture While Training Period.

5. IMPLEMENTATION & RESULT 5.1 Dataset Description

Loss value implies how poorly or well a model behaves after each iteration of optimization. An accuracy metric is used to measure the algorithm's performance in an interpretable way. It is the measure of how accurate your model's prediction is compared to the true data.

In general, neural networks, especially deep neural networks, tend to perform better when larger amounts of training data set is present. With this in mind, the more popular converted to a 48*48 matrix. Each number represented a pixel value. A one-hot encoding scheme was used for the labels rather than classifying emotions with numbers from 0-6.

In this experiment; 7 reactions have been used:

© 2021, IRJET

|

Impact Factor value: 7.529

|

ISO 9001:2008 Certified Journal

|

Page 704

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 08 Issue: 08 | Aug 2021

p-ISSN: 2395-0072

www.irjet.net

10 11 12 13 14 15 16 17 18 19 20

Fig. 8: Training Accuracy & Validation Accuracy for the Proposed CNN Architecture While Training Period.

This is when your model fits the training data well, but it is not able to generalize and make accurate predictions for data it has not seen before. The training set is used to train the model, while the validation set is only used to evaluate the model's performance. In Figure 8, the blue colored curve shows the training accuracy. The red colored curve shows validation accuracy.

Further improvement on the network OS accuracy and generalization can be achieved through the following practices. The first one is to use the whole dataset during the Optimization. Using batch optimization is more suitable for larger datasets. Another technique is to evaluate emotions one by one. This can lead to detect which emotions are more difficult to classify. Finally, using a larger dataset for training seems beneficial.

Nowadays, optimizers work on a stochastic basis because of large datasets (millions of examples). However, this was true for our project. Given a limited dataset, trying on the whole dataset could have led to a better feature learning. Also, the use of some optimizers reported on this research would have had a different behavior. This behavior can be displayed on the loss curve having a smoother shape or by avoiding an early convergence.

Value 200 20 3x3 1 same Max Pooling 3x3 2 384 192 0.25 ReLU 7 cross entropy momentum 0.9 0.01

Finally, using full dataset for training, pre-training on each emotion, and using a larger dataset seem to have the possibility to improve the networks performance. Thus, they should be addressed in future research on this topic.

REFERENCES [1]

V. D. Bharate et. al., “Human Emotions Recognition using Adaptive Sublayer Compensation and various Feature Extraction Mechanism”, IEEE WiSPNET, 2016.

[2]

Zhiding Yu et. al., “Image based Static Facial Expression Recognition with Multiple Deep Network Learning”, ACM, 2015.

[3]

S. M. Banu et. al., “A Novel Approach for Face Expressions Recognition”, IEEE 10th Jubilee International Symposium on Intelligent Systems and Informatics, 2012.

[4]

D.Drume and A.S.Jalal , “A Multi-level Classification Approach for Facial Emotion Recognition”, IEEE International Conference on Computational Intelligence and Computing Research, 2012.

Table-2: Measurement of Loss & Accuracy.

Epoch 1 2 3 4 5 6 7 8 9 © 2021, IRJET

loss 2.81 1.96 1.90 1.87 1.84 1.84 1.79 1.79 1.77 |

Accuracy 0.18 0.22 0.27 0.26 0.27 0.28 0.30 0.30 0.32

Impact Factor value: 7.529

0.30 0.35 0.38 0.40 0.42 0.42 0.45 0.45 0.52 0.51 0.52

6. CONCLUSIONS

Table-1: Summary of Phase Experiment.

Parameter Batch size Number of epoch Kernel size Kernel stride Padding Pooling Pool size Pool stride FC1 neurons FC2 neurons Dropout Activation function Number of classes Cost function Optimizer Momentum Learning rate

1.78 1.68 1.62 1.58 1.53 1.52 1.45 1.38 1.27 1.27 1.22

|

ISO 9001:2008 Certified Journal

|

Page 705

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 08 Issue: 08 | Aug 2021

p-ISSN: 2395-0072

www.irjet.net

[5]

Aleksic, P. S., and Katsaggelos, A. K. (2006). Automatic facial expression recognition using facial animation parameters and multistream hmms. IEEE Trans. Inform.

Forens. Sec. 1, 3–11. doi: 10.1109/TIFS.2005.863510

[6]

Bartlett, M. S., Hager, J. C., Ekman, P., and Sejnowski, T. J. (1999). Measuring facial expressions by computer image analysis. Psychophysiology 36, 253–263. doi: 10.1017/S0048577299971664

[7]

Pantic, M. and Rothkrantz, L., Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE. v91 i9. 1370-1390.

[8]

Pentland, A. and Choudhury, T., Face recognition for smart environments. IEEE Computer. v33 i2. 50-55.

[9]

Abhinav Dhall, Roland Goecke, Simon Lucey, and Tom Gedeon. Static fa-cial expression analysis in tough conditions: Data, evaluation protocol and benchmark. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, IEEE, 2011.

© 2021, IRJET

|

Impact Factor value: 7.529

|

ISO 9001:2008 Certified Journal

|

Page 706