Synopsis of Facial Emotion Recognition to Emoji Conversion

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 10 Issue: 12 | Dec 2023

p-ISSN: 2395-0072

www.irjet.net

Synopsis of Facial Emotion Recognition to Emoji Conversion Rohit Karmakar1, Subhadip Das2, Angshuman Rana3, Kunal Roy4 123 Students, Dept. of Computer Science and Engineering, Durgapur Institute of Advanced Technology and

Management, Durgapur, West Bengal, India

4 Assistant Professor, Durgapur Institute of Advanced Technology and Management, Durgapur, West Bengal, India

---------------------------------------------------------------------***---------------------------------------------------------------------

Abstract - This project presents a real-time Facial Emotion Recognition (FER) to Emoji Conversion system developed using OpenCV, TensorFlow, and NumPy. The system aims to identify and classify human emotions from facial expressions captured in real-time through a webcam. The FER system leverages the powerful image processing capabilities of OpenCV, the computational efficiency of NumPy, and the machine learning techniques provided by TensorFlow. A model is trained on a large dataset of facial images labeled with their corresponding emotions. The trained model is then used to predict emotions from facial expressions captured by a webcam in real-time and the conversion of recognized emotions into corresponding emojis, providing a visual representation of the detected emotion.

machine learning, enables computers to interpret and classify human emotions from facial cues.

This project dives into the creation of a real-time FER system utilizing OpenCV, TensorFlow, and NumPy. OpenCV provides a robust foundation for image and video analysis, while TensorFlow offers a comprehensive platform for machine learning. NumPy, with its efficient handling of large, multidimensional arrays, proves to be an ideal complement to these tools.

The project’s target is to develop a system capable of accurately identifying and categorizing human emotions in real-time based on facial expressions. Employing machine learning techniques and leveraging OpenCV’s adept image processing capabilities, a model will be trained to recognize diverse emotions from a vast dataset of labelled facial images. Once trained, the model will be deployed to analyze facial expressions captured by a webcam, providing realtime emotion predictions.

Key Words: Facial Emotion Recognition (FER), OpenCV, TensorFlow, NumPy, Computer Vision

1.INTRODUCTION Emotion recognition constitutes a captivating facet of human behaviour and assumes a pivotal role in our daily interpersonal exchanges. With the progression of technology, the capability to discern emotions extends beyond the realm of humans, encompassing the training of computers for such recognition. Facial Emotion Recognition (FER) emerges as a subset within computer vision and AI, dedicated to formulating algorithms and methodologies for identifying human emotions through facial expressions and transformation of recognized emotions into corresponding emojis. This will amplify user engagement and interaction, making the system more user-friendly and appealing.

This initiative marks a significant step toward empowering machines with a deeper understanding of human emotions, paving the way for the realization of truly intelligent and empathetic artificial intelligence.

3. Project Objectives The primary objectives of this project on Facial Emotion Recognition to Emoji Conversion using OpenCV, TensorFlow, and NumPy are as follows: 

Develop a Real-Time Facial Emotion Recognition to Emoji Conversion System:

This undertaking centers on the creation of a FER system utilizing OpenCV, TensorFlow, and NumPy. OpenCV furnishes a robust framework for analysing images and videos, TensorFlow provides a comprehensive platform for machine learning, and NumPy facilitates efficient computation with expansive, multi-dimensional arrays and matrices. The selection of these tools is grounded in their suitability for the project's objectives.

The main objective is to create a system that can accurately identify and classify human emotions from facial expressions in real-time and will display the emotion. This involves developing an algorithm that can process facial images, extract relevant features, and classify the emotion expressed.

2. Motivation



The ability to decipher human emotions from facial expressions to emoji conversion provides a window into human behavior. FER, a convergence of computer vision and

© 2023, IRJET

|

Impact Factor value: 8.226

Utilize OpenCV for Robust Image and Video Analysis:

OpenCV will serve as the cornerstone for image and video analysis, providing a robust framework to handle real-time video streams from the webcam. The system

|

ISO 9001:2008 Certified Journal

|

Page 206

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 10 Issue: 12 | Dec 2023

p-ISSN: 2395-0072

www.irjet.net

will employ OpenCV functionalities for face detection, tracking, and feature extraction to ensure precise and efficient processing of facial expressions. 



In addition to robust functionality, the project aims to deliver a user-friendly interface. The system should provide a seamless and intuitive experience for users interacting with the real-time Facial Emotion Recognition, promoting accessibility and ease of use.

Employ TensorFlow for Machine Learning:

The project aims to integrate TensorFlow, a comprehensive machine learning platform, to develop a model capable of recognizing diverse emotions. Through machine learning techniques, the model will be trained on a substantial dataset of labeled facial images, enabling it to make accurate predictions in real-time. 



Use NumPy for Efficient Computation:



Train the Model on a Diverse Dataset:

Lastly, implementing the above-mentioned objectives will help seamlessly integrate OpenCV, TensorFlow, and NumPy to develop a real-time Facial Emotion Recognition system. Utilizing webcam input, the system will accurately identify and classify human emotions, enhancing user experience through a user-friendly interface. Thorough testing and documentation will ensure the system’s reliability, contributing to the advancement of emotion recognition technology.

Train the Model on a Diverse Dataset: To enhance the system’s accuracy and versatility, the model will undergo training using a diverse dataset of facial images labeled with corresponding emotions. This comprehensive training approach will equip the model to recognize a wide spectrum of human emotions under various conditions.



4. Literature Review This list is not exhaustive but represents a selection of key sources that informed our understanding of the topic.

Implement Emotion to Emoji Conversion:

An integral part of this project is to convert the recognized emotions into corresponding emojis. This feature aims to provide a more intuitive and engaging way for users to understand the output of the emotion recognition system. 



“Computer Vision and Image Processing” by Victor Wiley, Thomas Lucas [1] gives an overview of computer vision and its relation to image processing and pattern. It also describes the basic stages of image analysis and the common frameworks used in computer vision. It highlights the benefits and limitations of computer vision and image processing, and suggests some future directions for research and development. It also emphasizes the importance of machine learning and artificial intelligence for improving the performance and accuracy of computer vision systems.



“A Brief Review of Facial Emotion Recognition Based on Visual Information” by Myoung Chul Ko [2] reviews the conventional and deep-learning-

Real-time Emotion Detection using webcam input:

The system will leverage the webcam as the input device, capturing real-time video data for instantaneous emotion recognition. The integration of OpenCV and TensorFlow will enable the system to process each frame efficiently, providing a continuous and dynamic assessment of the user’s emotional state.

© 2023, IRJET

|

Impact Factor value: 8.226

Enhance User Interaction and Experience:

By recognizing the user’s emotions, the system can potentially enhance user interaction and experience in various applications. This could include adapting the system’s responses based on the user’s emotional state or providing feedback to the user about their emotional state.

To enhance the system’s accuracy and versatility, the model will undergo training using a diverse dataset of facial images labeled with corresponding emotions. This comprehensive training approach will equip the model to recognize a wide spectrum of human emotions under various conditions. 

Evaluate System Performance:

Thorough testing and evaluation will be conducted to assess the performance, accuracy, and responsiveness of the Facial Emotion Recognition system. Real-world scenarios and a range of emotions will be simulated to ensure the system’s reliability and effectiveness.

NumPy will have a crucial role in optimizing computational efficiency, particularly in handling large, multi-dimensional arrays and matrices involved in the machine learning process. This integration will ensure that the Facial Emotion Recognition system operates seamlessly, even with the complexities of real-time video data. 

Creating a User-Friendly Interface:

|

ISO 9001:2008 Certified Journal

|

Page 207

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 10 Issue: 12 | Dec 2023

p-ISSN: 2395-0072

www.irjet.net

based approaches for FER, which is an important topic in computer vision and artificial intelligence. It introduces the terminology, databases, evaluation metrics, and challenges related to FER, and compares the advantages and disadvantages of different methods. It also focuses on the recent advances in FER using CNN and LSTM models, which can learn spatial and temporal features from facial images or videos. The article also suggests some future directions and applications for FER, such as using multi-modal data, improving robustness, and enhancing human-computer interaction. 



rate of the facial detector, and the processing speed of the facial expression recognizer. The journal suggests some future improvements, such as using a larger and more diverse dataset, making the CNN model deeper and wider, adding eye and mouth detectors, and making the system more compatible for different platforms and applications.

“Face Detection and Recognition Using OpenCV” by Ramadan TH. Hasan, Amira Bibo [3] is about Face Detection and Recognition Using OpenCV, a free and open source library for computer vision applications. The article introduces the main OpenCV modules, features, and algorithms that are used for face detection and recognition, such as Haar Cascade, LBP, EigenFaces, FisherFaces, LBPH, YOLO, Faster R-CNN, and SSD. This journal also explains how to use OpenCV based on Python, a popular and easy-to-use programming language for computer vision projects. It also reviews some recent literature that use OpenCV for various face detection and recognition tasks, such as emotion recognition, attendance system, security system, mask detection, and personal identifier. It compares the accuracy, performance, and results of different techniques and classifiers of OpenCV used in the reviewed studies. The article concludes that OpenCV can be used in different fields and scenarios for face detection and recognition, and that it can be combined with other libraries and frameworks to improve its efficiency and functionality.

“REALTIME FACIAL EMOTIONS RECOGNITION SYSTEM (RFERS)” by Mohit Verma, Arpit Sharma, Mayank Verma, Tushar Rawat, and Teena Verma [4] aims to design a Python-based system that can detect and recognize the facial expressions of six universal emotions plus neutral using a Convolutional Neural Network (CNN) model and TensorFlow library. The system has two components: a pre-train program that uses a dataset of labeled facial images to train the CNN model, and a recognizer that uses a camera to capture real-time video and apply facial detection and recognition on the extracted facial images. The journal reports that the system achieved 73.8percent accuracy on fixed user and was able to train images, detect faces, and recognize facial expressions in real time. The journal also discusses the challenges and limitations of the system, such as the size and quality of the training images, the error

© 2023, IRJET

|

Impact Factor value: 8.226

|



“Dynamic Emotion Recognition and Emoji Generation” by Pradnya Bohr, Ashwini Raut, Sachin Kate, Anurag Salve, Chandan Prasad [5] proposes a system that can detect human emotions from facial expressions and generate corresponding emojis in real time. The system uses machine learning techniques to process the input from the camera and create emojis that can be shared or saved across different platforms. The paper also reviews the existing systems and the literature on emoji and emotion recognition. Emoji are pictographic forms of facial expressions, objects, and symbols that originated from Japan and have evolved into various forms and styles with different colors and meanings. Emoji became popular worldwide and are used in various mobile messaging applications and social media platforms. Emoji can express the feelings and emotions of the sender and help the receiver to understand the message better. Emoji can also demonstrate interpersonal functions such as personal expressions and mood boost.



“Emoji: A Deep Learning Approach For Custom Emoji Creation And Recognition” by Venkata Ravi, Kiran Kolla [6] the paper describes a software program that can create and recognize personalized emojis based on human facial expressions. This is the main method used in the paper, which involves building and training a convolutional neural network (CNN) to classify facial emotions into five categories: sad, surprised, scared, happy, and neutral. The process of creating a graphical user interface (GUI) that can capture the user’s face from a webcam and display the corresponding emoji or avatar based on the CNN’s prediction. The paper concludes that the project is a novel and fun way of communicating with customizable emoticons.



A Systematic Review of Emoji: Current Research and Future Perspectives” by Qiyu Bai, Qi Dan, Zhe Mu and Maokun Yang [7] the article delves into the intricate world of emojis and their impact on computer-mediated communication. Emojis have evolved from humble beginnings as smileys and emoticons into a diverse set of symbols capable of representing facial expressions, emotions, objects, concepts, and activities. Their popularity has led to widespread integration into online communication, but their usage is subject to the influence of

ISO 9001:2008 Certified Journal

|

Page 208

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 10 Issue: 12 | Dec 2023

p-ISSN: 2395-0072

www.irjet.net

individual preferences, cultural nuances, and platform-specific factors. Emojis function as nonverbal cues, facilitating the conveyance of meanings, fostering interactions, and expressing emotions.

Their significance extends across various fields, attracting the attention of researchers in computer science, communication, marketing, psychology, and education. Research endeavors span a spectrum of topics, reflecting the multidisciplinary nature of emoji studies.

TensorFlow for machine learning, and NumPy for efficient computation. The system will capture real-time video data through a webcam, identify and classify human emotions from facial expressions, and enhance user interaction and experience in various applications.

2.

Research:

Gathering information for this project involves a thorough exploration of the strengths and weaknesses of OpenCV, TensorFlow, and NumPy, adopting effective integration strategies, and delving into the realm of accessible full-stack technologies. Image Processing Techniques. Various algorithms are utilized to develop software and hardware that recognizes the human face.

[8] The algorithm will compare the various pictures to pre-defined or learnt images, as well as real video images.

5. Methodology

3.

Data Collection:

This chapter details the process of collecting and preparing the dataset used for training the emotion recognition model. It includes the sources of the data, the types of data collected, and the preprocessing steps undertaken to prepare the data for model training.

4.

Model Development:

It mentions all the intricate details of constructing the emotion recognition model. It meticulously outlines the machine learning techniques employed, unravels the intricate architecture of the model. Also, the advanced deep expression recognition algorithms in the field, the performance of these algorithms on common expression databases is compared, and the strengths and weaknesses of these algorithms are analyzed in detail.

[9] This comprehensive exploration provides a profound understanding of the model’s inner workings, enabling readers to replicate its success and expand upon its capabilities [10] 5.

The Facial Emotion Recognition (FER) project utilizes a pretrained model provided by the FER2013 dataset to streamline the model training process. [11] This robust TensorFlow-based model has been meticulously trained on a vast collection of facial images and their corresponding emotion labels, empowering it to accurately detect emotions in real-time scenarios.

During the training phase, the model is systematically fed with a diverse range of facial images, each associated with its corresponding emotion label. This process enables the model to establish intricate associations between facial features and the emotions they convey, laying the foundation for its real-time emotion recognition capabilities. This approach not 12 only saves

Fig. 1. Flowchart of proposed Methodology The project consists of seven chapters, and the organization of the project is as follows: 1.

Project Objectives:

The project’s scope is to create a Facial Emotion Recognition to Emoji Conversion system that operates in real-time. It will use OpenCV for image processing,

© 2023, IRJET

|

Impact Factor value: 8.226

Model Training:

|

ISO 9001:2008 Certified Journal

|

Page 209

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 10 Issue: 12 | Dec 2023

p-ISSN: 2395-0072

www.irjet.net

time and computational resources but also ensures the model’s robustness and generalizability across various facial expressions and individuals. [12] 6.

emotion-based human-computer interaction. We believe that the FER to emoji conversion feature of our project will pave the way for more empathetic and interactive AI systems in the future.

Implementation:

7. Future Work

The project implements real-time Facial Emotion Recognition (FER) using Open-CV. [13] By harnessing the capabilities of Open-CV, the project establishes a real-time facial emotion recognition system, effectively capturing live video from a webcam and accurately identifying the emotions of each detected face. These emotions are then instantaneously superimposed onto the video frames, creating an engaging and responsive interface.

7.

Facial emotion recognition (FER) is a rapidly growing field with a wide range of potential applications. As FER technology continues to develop, it is important to consider the ethical implications of this technology and to develop safeguards to prevent its misuse.

Improved accuracy and robustness: One of the most important areas of future work is to improve the accuracy and robustness of FER systems. This includes developing algorithms that can better recognize emotions in real-world conditions, such as in low-light or noisy environments.

2.

Increased diversity: Another important area of future work is to increase the diversity of FER datasets. Current datasets are often biased towards certain demographics, such as young, white males.

This can lead to FER systems that are less accurate for people from other demographics.

3.

Multimodal FER: Multimodal FER systems combine information from multiple sources, such as facial expressions, voice, and body language, to improve emotion recognition accuracy. Multimodal FER is a promising area of future research, as it can potentially overcome some of the limitations of unimodal FER systems.

4.

Ethical considerations: The use of FER technology raises a number of ethical concerns. For example, FER could be used to unfairly discriminate against individuals or to invade their privacy. It is important to develop ethical guidelines for the use of FER technology and to ensure that it is used responsibly.

5.

Commercial applications: FER technology has the potential to be used in a wide range of commercial applications. For example, it could be used to improve customer service by detecting customer emotions and tailoring responses accordingly. FER could also be used to develop personalized learning systems that adapt to the student’s emotional state.

6.

Mental health applications: FER technology could also be used to develop mental health applications.

For example, it could be used to detect signs of depression or anxiety and to provide early intervention. FER could also be used to develop personalized mental health treatments.

System Testing and Evaluation:

The important step in ensuring the accuracy and reliability of the facial emotion recognition (FER) system. To evaluate the system’s performance, a dataset of facial images with corresponding emotion labels can be employed. The system’s ability to correctly classify emotions should be assessed using metrics such as precision, recall [14]. Additionally, real-time performance should be evaluated by measuring the system’s latency and frame rate.

8.

1.

Final Deployment:

The project’s final deployment marks the end of the development journey and its transition into the hands of analysts. This critical phase ensures that the project meets its scalability and performance requirements.

Streamlining the deployment process and integrating continuous integration facilitates seamless updates and adjustments in response to user feedback and evolving requirements.

6. Conclusion One of the most innovative aspects of our project is the FER to Emoji Conversion. This feature takes the Facial Emotion Recognition (FER) technology to the next level by not only recognizing emotions but also translating them into universally understood emojis. This conversion process involves mapping each recognized emotion to a corresponding emoji. For instance, happiness could be represented by a smiling face emoji, sadness by a crying face emoji, and so on. This provides a more intuitive and engaging way for users to understand the output of the FER system. Moreover, this feature significantly enhances the user experience, especially in applications like social media, gaming, and virtual meetings, where emojis are widely used to express emotions. It’s like having a personal emotion translator that speaks the universal language of emojis! Our project has not only achieved its goal of real-time emotion recognition but has also opened up new possibilities for

© 2023, IRJET

|

Impact Factor value: 8.226

|

ISO 9001:2008 Certified Journal

|

Page 210

7.

8.

9.

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 10 Issue: 12 | Dec 2023

p-ISSN: 2395-0072

www.irjet.net

Educational applications: FER technology could be used to develop educational applications that adapt to the student’s emotional state. For example, an FER system could detect when a student is frustrated and provide additional support.

recognition system. In 2015 21st Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV), pages 1–4. IEEE, 2015.

[10] D Mary Prasanna and Ch Ganapathy Reddy.

Development of real time face recogni tion system using opencv. Development, 4(12):791, 2017.

Human-computer interaction: FER technology could be used to improve human-computer interaction.

For example, an FER system could detect when a user is confused and provide additional help.

[11] Panagiotis Giannopoulos, Isidoros Perikos, and Ioannis Hatzilygeroudis. Deep learning approaches for facial emotion recognition: A case study on fer-2013.

Advances in 18 Hybridization of Intelligent Methods: Models, Systems and Applications, pages 1–16, 2018.

Safety and security: FER technology could be used to improve safety and security. For example, it could be used to detect when a person is angry or upset and to take steps to prevent violence.

[12] VS Amal, Sanjay Suresh, and G Deepa. Real-time emotion recognition from facial expressions using convolutional neural network with fer2013 dataset. In Ubiquitous Intelligent Systems: Proceedings of ICUIS 2021, pages 541–551. Springer, 2022.

These are just a few of the exciting areas of future work in FER. As FER technology continues to develop, we can expect to see even more innovative and impact applications emerge.

[13] Lyron Foster. Recognizing human emotions with ai. (tensorflow, keras, opencv), Mar 2023.

REFERENCES [1] Victor Wiley and Thomas Lucas. Computer vision and image processing: a paper review. International Journal of Artificial Intelligence Research, 2(1):29–36, 2018.

[14] Daniel E Ho, Emily Black, Maneesh Agrawala, and Li FeiFei. Evaluating facial recognition technology: a protocol for performance assessment in new domains. Denv. L. Rev., 98:753, 2020.

[2] Byoung Ko. A brief review of facial emotion recognition based on visual information. Sensors, 18(2):401, January 2018.

[3] Ramadan TH Hasan and Amira Bibo Sallow. Face detection and recognition using opencv. Journal of Soft Computing and Data Mining, 2(2):86–97, 2021.

[4] Mohit Verma, Arpit Sharma, Mayank Verma, Tushar Rawat, and Ms Teena Verma. Realtime facial emotions recognition system (rfers).

[5] Pradnya Bhor, Ashwini Raut, Sachin Kate, Anurag Salve, and Chandan Prasad. Dynamic emotion recognition and emoji generation. International Research Journal of Engineering and Technology, 6(9):1891–1894, 2019.

[6] Venkata Ravi Kiran Kolla. Emojify: A deep learning approach for custom emoji creation and recognition.

International Journal of Creative Research Thoughts, 2021.

[7] Qiyu Bai, Qi Dan, Zhe Mu, and Maokun Yang. A systematic review of emoji: Current research and future perspectives. Frontiers in Psychology, 10:2221, 2019.

[8] Heera Mohan KV, Pranav PV, Swetha Pai, et al. A synopsis on intelligent face discovery frameworks.

Journal of Applied Information Science, 10(1), 2022.

[9] Heechul Jung, Sihaeng Lee, Sunjeong Park, Byungju Kim, Junmo Kim, Injae Lee, and Chunghyun Ahn. Development of deep learning-based facial expression

© 2023, IRJET

|

Impact Factor value: 8.226

|

ISO 9001:2008 Certified Journal

|

Page 211