Emotion Based Music Recommendation System by IJRASET

2, 3

Abstract: Our world today is driven by machines of various complexities. From a basic one like a computer to a highly complex humanoid robot, everything is a product of human intelligence. A lot of industries are being benefited from such new technologies. Facial Expression Recognition is one of these technologies. It has a wide range of applications and is an area that is constantly evolving. The analogy behind it is, when we gaze at someone, the eyes send signals to the brain. The face patterns of that specific person are carried by these messages. These patterns are then compared to those stored in the brain's memory. Inspired by such innovations, our research collects human expressions and analyses their emotions using our vast dataset, offering some necessary strategies to change their facial expressions. Due to the competitive environment, the youth of our generation has been inclined to a lot of mental health problems such as anxiety and depression. Our generation's youth has been predisposed to a variety of mental health issues. Our idea attempts to provide a relaxing atmosphere to a person based on his or her facial expressions. Keywords: Facial Expression, Face Recognition, Python, PyWhatkit, OpenCV. I. INTRODUCTION From the time we are born, we spend an inordinate amount of time studying each other’s behavioural traits because that’s what makes us human. This is something that renders feelings into us and is a way we express them. The face takes primacy for us. We have behaviours associated with our facial expressions and thus decoding this facial expression is a part of communication. Even the first expression we learn from the world when we are born can be of any type; that is a smile can be social, interested, curious, or secretive. Catching these human expressions and subconsciously assessing the meanings of these expressions from our memory lane is the task we need to bring in for our computer/smart devices to perform. With machine learning, we tend to give a set of images/video/webcam and expect the computer in return to detect the expression, emotion, and sentiments. It all started from text recognition and now with the advancement of technologies, we promoted ourselves towards facial recognition. After we catch the accurate emotion or expression the main target is to help the user or to suffice them to change their feelings. So, this research paper demonstrates how a model can recognize the facial expression of a person and assist in changing the user’s mood.

Fig. 1 Facial Expressions Facial expression is one of the most important features of human emotion recognition. Images are high-dimensional objects. Facial features over high dimensional spaces can be counter-intuitive. Words may not match emotions and the face can act as a window and reveal what someone is actually thinking and feeling. Facial expressions are the same regardless of race or culture all around the world. This is what stands out and makes it important for us to use it for our machine learning objectives.

©IJRASET: All Rights are Reserved

1674

International Journal for Research in Applied Science & Engineering Technology (IJRASET) ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.429 Volume 9 Issue X Oct 2021- Available at www.ijraset.com II. METHODOLOGY The main ideology of the research revolves around the concepts of identifying the most suitable facial expression from a dataset by classification using a machine learning model with a python framework. These results would be exceedingly beneficial for the psychology of expression analysis. The dataset has been taken via Kaggle. The dataset contains training pictures of 7 different emotions. We have applied deep learning algorithm on the dataset. Once the model is trained, it is saved and is used whenever needed. For real time face detection, Haar Cascade features are used. It is one of the oldest known Object Detection Algorithm used to identify faces in images or real time videos. According to Faizan Ahmed from the department of computer science and engineering, Beijing university quoted in his work “state of art” that in ‘current system Haar-Like features reported relatively well but it has much false detection than LBP which could be consider being a future work in surveillance to reduce false detection in Haar-like feature and for the recognition part Gabor is reported well as its qualities overcomes datasets complexity.’ A. Research Approach We all have been well acquainted with Facebook, right after we upload a picture on it, it uses an auto-attack to tag people which is an interdisciplinary field that deals with how computers can be made to gain high level understanding from digital images or videos. So, the idea is to automate tasks that the human visual systems can do. Neuropsychiatric symptoms appear to be common in people due to covid-19. The humans are found to have an increased chance of developing a new mental health illness within 90 days when impaired with other illnesses. In such difficult circumstances, it became necessary for us to devise a system that would assist the people in expressing what they truly feel. Since facial expression is the superior non-verbal way of communication that can lay forward the sentiments of an individual, inspired from this, we developed a platform that recognizes the human facial expression then suggests a YouTube playlist to the person after decoding their mood. B. OpenCV OpenCV library is used for computer vision. It supports a wide variety of programming languages such as C++, python, java etc. OpenCV python a python wrapper for the original OpenCV C++ implementation. The images are converted from or to NumPy arrays. This makes it easier to integrate with other libraries. The computer will first convert the image into a matrix; in this case it’ll convert it into a NumPy matrix or an array. Loading, resizing and displaying the image are some basic operations that can be easily performed using it. It uses the coordinates of the face for understanding the image. The video is captured in a similar way. This library supports a wide range of complex image processing models and includes a number of image modification methods. C. Haar Cascade Features Voila and Jones introduced the first image detection method, known as Haar Cascades, in 2001. As the model has been working exceptionally well, the project also incorporates the usage of the same algorithm. The algorithm returns the coordinates which makes a rectangular box around the face. All the contours of the face are detected. A single Haar feature would traverse pixel by pixel and change its size throughout the image frame and detect edges in it. D. Optimizers The optimizers are functions utilized to alter the characteristics of our neural network such as hyperparameters of weights and learning rate of the model in order to minimize the losses and achieve results quicker. The two optimizers are used in our research. 1. ReLU Activation Function. 2. Adam Optimizer The ReLU optimizer or the Rectified Linear Activation Function is used as an activation function in the project. It is a basic function that returns the same input as output, if positive; else, it returns 0. The ReLU activation function enables models to learn quicker and perform better. It has been widely used for multilayer models and Convolutional Neural Networks. It is used to overcome the vanishing gradient problem.

Fig. 2 Graph of ReLU activation function

©IJRASET: All Rights are Reserved

1675

International Journal for Research in Applied Science & Engineering Technology (IJRASET) ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.429 Volume 9 Issue X Oct 2021- Available at www.ijraset.com The Adam optimizer, which was presented by Diederik Kingma and Jimmy Ba in their 2015 ICLR work titled "Adam: A Method for Stochastic Optimization," is also employed. The Adam Optimizer combines a hybrid of two gradient descent algorithms: Gradient Descent with Momentum Algorithm (AdaGrad) and Root Mean Square Propagation Algorithm (RMSP). Adam Optimizer employs a new formula derived from the two approaches. The optimizer surpasses the previous two models and delivers better results at a lower cost.

Fig. 3 Comparison of Adam Optimizer with other Optimizers E. Convolutional Neural Network We employed a neural network to identify the emotion. A neural network learns from the examples that are fed into it. The weights associated with the neurons are adjusted by the network. Consecutive adjustments tend to teach the model more and help it perform better. Convolutional Neural Networks (CNN) are used in this case. Since, the image is a multi-dimensional array, Artificial Neural Networks (ANN) cannot be employed because ANN requires 1-dimensional vectors. Thus, increasing the number of training parameters massively and not providing an efficient result. CNN, on the other hand, is a feed-forward network that is used to analyse and recognise picture input. The project detects emotions of 7 categories - fear, surprise, angry, happy, sad, disgust, neutral. The dataset “Facial expression recognition dataset” is being used, available on Kaggle. It contains 48x48 pixel grayscale images grouped according to emotions in 7 different folders.

Fig. 4 Visualization of a Convolutional Neural Netowrk We fit the training set into the model. Then the performance of the model is evaluated by using the accuracy metrics to determine the evaluation of the loss functions and the characteristics of the negation in heatmap.

©IJRASET: All Rights are Reserved

1676

International Journal for Research in Applied Science & Engineering Technology (IJRASET) ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.429 Volume 9 Issue X Oct 2021- Available at www.ijraset.com F. Pywhatkit Pywhatkit is a useful package which deals with automating the processes. It is a very easy to use library that can perform a large number of functions. One can send images, WhatsApp messages, search on web, many more tasks could be accomplished using this library. We have used one of the methods of the Pywhatkit library which is to play songs on YouTube. The method searches for the inputted keyword on YouTube and then plays the first match. In our case, we pass the emotion to uplift the mood of the user as one of the keywords and get the playlists played accordingly. III. ALGORITHM This section goes through the modules and methods that are utilized to implement facial expression analysis integrating them with the pivotal libraries to attain our aspired process. A. Image Capturing and Face Detection This section describes how the user's picture is acquired and how the face is extracted from it. The user's image is captured using one of the Python's libraries, OpenCV. The command cv2.CascadeClassifier(file name) is used to read the xml file for “haarcascadefrontal-face-default”. The file contains the features for detecting a person’s face. The command cv2.videoCapture() is used to get the object from the device's camera. An infinite loop is created where the read command is used to get the frames of the created object. After capturing the user’s picture in a frame with OpenCV, the following step is to detect the user’s face from a frame captured through the webcam. The frame holding the image of the user is converted to Grayscale as the face is more defined by the composition of the set of pixels and not by the colours. So, the whole dataset of images is converted to grayscale by using the command cv2.COLOR_BGR2GRAY (image is initially present in BGR format). The image is modified (blurred, sharpened, or masked as required). The image undergoes segmentation. The segmentation algorithm divides the image into a set of pixels by assigning each pixel a label. The pixels having the same labels fall under a category that helps identify the contours in any given image. The Haar Cascades image detection method is used here. The algorithm returns the coordinates which make a rectangular box around the face. The algorithm uses features as shown below.

Fig. 5 Haar Cascade Features As an image is represented as a NumPy array, it is not hard to imagine the dark part of the rectangle as a group of pixels with value 1 and the other as pixels with value 0. Considering feature (a), it detects the features with a horizontal edge, having darker pixels towards the bottom and lighter pixels towards the top. Haar value is calculated by the formula ii(x, y) =

( ′, ′) ,

Fig. 6 Haar Cascade Mathematical Equation A method of Haar algorithm "detectMultiScale" is used to have the faces of different sizes in the input image. The method returns a tuple which is then unpacked to get the rectangular coordinates of the face. Then by using the method cv2.rectangle(a, b, c, d), where a-d are the coordinates, the face is highlighted.

©IJRASET: All Rights are Reserved

1677

International Journal for Research in Applied Science & Engineering Technology (IJRASET) ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.429 Volume 9 Issue X Oct 2021- Available at www.ijraset.com B. Training the Model After detecting the face, the next step is to detect the emotion. For doing that, we require the use of Convolutional Neural Networks (CNN). To visualize an image stored as an array, we use the Matplotlib library. The method imshow() is used to see an image. The dataset is split into training and test sets. Then a neural network model is created by using the Keras library from Tensorflow. The model is a multilayered neural network. The layers are added by using the “add” method to the model initialized. It contains a convolutional 2D layer with ReLU as the activation function. The numbers of filters are constantly increased after every layer. The filter size determining the dimensions of the kernel are taken as inputs. The input shape of the image is needed to be defined for the first layer which is (48, 48, 1) in this case. The layers are MaxPooled and are given a small pool size of (2, 2). To avoid overfitting the data, a dropout layer is also introduced. The neurons from the current layer are randomly disconnected in the following layer with a probability p, forcing the network to rely on the existing connections. The Adam optimizer is used to optimize gradient descent; it takes a learning rate as the input. Finally, the model is flattened to a 1 Dimensional array format.

Fig. 7 Summary of the Convolutional Neural Network The features detected in the model are saved in a JSON file to avoid retraining the model every time to save the operation and time complexity to train and attain similar accuracy for proceeding to neural network training. C. Redirecting to Songs This is the final step of our project. The emotion of the user has been captured, the emotion is used as a keyword to play the song on YouTube accordingly. As the model detects 7 expressions, it also includes seven playlists. The model uses the pywhatkit function "pywhatkit.playonyt (keyword)" to redirect to the playlists on YouTube. The define keywork is entitled with a corresponding playlist linked to the expression to elevate the individual emotional anxiety.

1678

International Journal for Research in Applied Science & Engineering Technology (IJRASET) ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.429 Volume 9 Issue X Oct 2021- Available at www.ijraset.com IV. RESULTS AND DISCUSSION The Haar Features are successfully able to detect the face during real-time video with a good precision. The model gives an accuracy of 64.52% to detect the right emotion. Detecting emotions has always been tricky even for humans. Performing the same task with a better accuracy by the help of machines has always been the part where new innovations takes place. The emotions are detected by the model stored in JSON format and can also be used as an API for other projects. This model then is reiterated to visualize the facial characteristic to depict the song playlist in correlation to the neural network and the emotional sentiment of the individual. We were able to effectively show an intriguing use of facial emotion detection. The model can be quite useful for people having a hectic lifestyle. Thus, the model accomplishes the goal of proving a stress-free atmosphere to the people who need it the most.

Fig. 8 Highest accuracy achieved of 64.52%

Fig. 9 Model in JSON format

1679

Fig. 10 Playing songs through Pywhatkit library V. FUTURE SCOPE In conclusion, facial recognition not only recognizes the state of emotion but also the identity. It is an unavoidable conclusion that will pervade much of our lives in most of the planet. The following are few areas of future research: A. Facial expression analysis can be employed extensively in interrogations. As one cannot mimic all of the distinctive qualities all at once, the same subtle cues that a person takes while interacting with others may be utilized with criminals to know their state of mind as technology advances. B. Our face is both the most important and the most expressive aspect of us. This can be used in instances where people are unable to communicate, such as at a rehabilitation center or when they are mentally ill. C. If carried out with a better accuracy, the expression analysis can also be integrated in phones, doorbells, or locks in our homes, automobiles, virtual reality, and so on. REFERENCES [1] [2] [3] [4] [5] [6] [7] [8]

R. Padilla, C. F. F. Costa Filho and M. G. F. Costa, “Evaluation of Haar Cascade Classifiers Designed for Face Detection”, International Journal of Computer, Electrical, Automation, Control and Information Engineering Vol:6, No:4, 2012. K .H. Wanjale, Amit Bhoomkar, Ajay Kulkarni, Somnath Gosavi, “Use Of Haar Cascade Classifier For Face Tracking System In Real Time Video”, International Journal of Engineering Research & Technology (IJERT). Mingjie Wang, Pengcheng Tan, Xin Zhang, Yu Kang, Canguo Jin, Jianying Cao1, “Facial expression recognition based on CNN”. Alpika Gupta, Dr. Rajdev Tiwari, “Face Detection Using Modified Viola Jones Algorithm”, International Journal of Recent Research in Mathematics Computer Science and Information Technology Vol. 1, Issue 2, pp: (59-66), Month: October 2014 – March 2015. Tania Akter Setu, Dr. Md. Mijanur Rahman, “Human Face Detection and Segmentation of Facial Feature Region”. Paul Viola, Michael J. Jones , “Robust Real-Time Face Detection”, International Journal of Computer Vision 57(2), 137–154, 2004. Karen L. Schmidt and Jeffrey F. Cohn, “Human Facial Expressions as Adaptations:Evolutionary Questions in Facial Expression Research”. Sakshi Indolia, Anil Kumar, Goswami, S.P.Mishra, Pooja Asopa, “Conceptual Understanding of Convolutional Neural Network- A Deep Learning Approach”.

1680