Issuu

https://doi.org/10.22214/ijraset.2022.44296

June 2022

Gesture Vocalizer Using Flex Sensor and Software Visualization Katikaneni Nitish Kumar1, Thota Guna Prashant2, Pendli Rishith Reddy3, Mrs. B. Priyanka4 1, 2, 3

B.E. Scholars, Dept. of Electronics and Communications Engineering, Sreenidhi Institute of Science and Technology, Hyderabad, Telangana 4 Associate Professor, Dept. of Electronics and Communications Engineering, Sreenidhi Institute of Science and Technology, Hyderabad, Telangana Abstract: Hardware version of the project: Gesture Vocalizer is a multi-microcontroller-based framework intended to work with the correspondence among the imbecilic, hard of hearing and visually impaired networks and their correspondence with the ordinary individuals. This system can be flexibly changed to function as a "smart device.". In this project we showcased a gesture vocalizer based on a microcontroller and sensors. Gesture vocalizer design is basically a hand glove and a microcontroller system. A hand glove can detect practically all hand gestures, and a microcontroller-based system can translate a few of those gestures into identifiable speech. The hand glove has two types of sensors: flex sensors that act as bend sensors and accelerometers that act as tilt sensors. This device is advantageous to deaf people because it allows their hands to speak once they have worn the gesture vocalizer glove. In this project we use flex sensors and Arduino Mega microcontroller which are the backbone of the project. The flex sensor otherwise called the twist sensor is a sort of sensor that measures the measure of bowing or avoidance or flexing as the name proposes. As the sensors are bowed the sensors give us an obstruction esteem, the sensor gives so the more the sensors are twisted, more opposition esteem. When estimated with a multimeter, the flex sensor at a level and consistent position gives an opposition esteem close to 25k ohm while when the sensors are completely bowed, they give an obstruction worth of 72k ohm. The flex sensors are typically of a slim construction and are entirely agreeable to utilize, and can be twisted without utilizing a lot of power. Software version of the project: We describe a revolutionary real-time approach for hand gesture identification in this paper. The background subtraction method is used in our framework to remove the hand region from the background. The palm and fingers are then subdivided so that the fingers may be detected and recognised. Finally, to predict hand gesture labels, a rule classifier is used. Experiments on a 1300 image data set reveal that our solution works well and is quite efficient. Furthermore, on another data set of hand movements, our method outperforms a state-of-the- art method. The background subtraction approach is used to detect the hand region from the backdrop. The palm and fingers are segmented after that. The fingers in the hand image are discovered and recognised based on the segmentation. Hand gesture recognition is performed using a simple rule classifier. On a data set of 1300 hand photographs, the performance of our technique is tested. The results of the experiments suggest that our method works effectively and is suitable for real-time applications. Furthermore, on an image collection of hand movements, the suggested technique beats the state-of-the-art FEMD. The proposed method's performance is significantly dependent on the results of hand detection. If there are moving objects with a color that is similar to that of the skin, the items are detected as a result of the hand detection and reduce the hand gesture recognition performance. Machine learning algorithms, on the other hand, can distinguish the hand from the backdrop. ToF cameras provide depth information that can help enhance hand detection performance. To address the complicated background problem and improve the robustness of hand identification, ML algorithms and TOF cameras may be applied in future research. I.

INTRODUCTION

A. Deaf-Dumb Hand Gestures There are 22 official languages in India, as well as 415 living languages.When it comes to communicating between villages, groups, and states, such variation in languages poses a barrier. The Deaf and Mute communities in India utilize Indian Sign Language (ISL) as one of their living languages. The total percentage of population of India that is handicapped with hearing cumulates to nearly 1.3 million people out of the total 21.9 million people with disabilities. Previously, India's educational system was based on the oralaural approach.

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 |

2294

International Journal for Research in Applied Science & Engineering Technology (IJRASET) ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538 Volume 10 Issue VI June 2022- Available at www.ijraset.com As more Indian Sign Language is employed, the situation is improving. In India, there are 22 official languages and 415 additional languages that are still spoken. When it comes to communicating between villages, groups, and states, such variation in languages poses a barrier. Indian Sign Language (ISL) is a live language in India that is spoken by Deaf and Mute people. Previously, the Indian educational system was based on the oral-aural approach. The situation is improving as more Indian Sign Language is used Sign languages are languages that use the visual-manual modality to convey the meaning or messages. This language is expressed by a combination of manual signs and non- manual features. Sign languages are distinct natural languages with their own syntax and lexicon. This indicates that sign languages are not universal or mutually comprehensible, despite the fact that there are obvious parallels between them. It's uncertain how many sign languages are now in use around the world. Each country has its own native sign language, and some countries have many sign languages. The Ethnologue 2021 edition counts 150 sign languages, but the SIGN-HUB Atlas of Sign Language Structures lists over 200, with the caveat that there are likely to be more that have yet to be recorded or discovered. Some sign languages have received legal recognition in some manner. Natural sign languages are distinguished from other systems that are antecedents to or derived from them, such as devised manual codes for spoken languages, home sign, and "baby sign," according to linguists, and signs learned by non-human primates. There are striking similarities among the signs in the sign language such as: 1) Direction of the Sign: When the same sign is interpreted in multiple ways, it might have completely distinct meanings. When the sign "give" is directed toward the one signing, it indicates "give to me," but when it is directed away from the one signing, it indicates "give to you.". 2) Hand positioning: The palm of your hand should face the person(s) you're speaking with. Maintain a comfortable position for your arm. Your hand should not be in the way of your mouth. 3) Flow and Rhythm: With a little break between words, each letter should be signed clearly, distinctly, and crisply. A pause is indicated by holding the last letter of the word for several seconds. 4) Vocalization: You should speak the words you finger spell at the same moment. The letters themselves should not be uttered aloud. 5) Double letters in a word: Between the first and second letters, the hand is slightly opened. 6) Understanding Signs: It's not uncommon for novice or inexperienced signers to have trouble comprehending what others are saying to them. Deaf or speech-impaired people who use sign language will be patient with you if you don't understand them Sign Language is divided into three groups such as:  Everyday Terms: This class incorporates terms that are utilized in regular correspondence like Sad, Happy, Good, How Are You, Sleepy, Angry, etc.  Basic terms: This classification incorporates the terms that are utilized for our fundamental necessities like Alphabets (A, B, C, ...), Numbers (1, 2, 3, ...), Days (Sunday, Monday, ...), Months (January, February, ...), Etc.  Emergency terms: This class incorporates the terms that are utilized for a critical need or to make mindful of certain perils like Thirsty, Emergency, Discharge, Call Ambulance, Etc. B. Gesture Vocalizer People have the voice capacity for cooperation and correspondence among one another. Lamentably, not every person has the capacity of talking and hearing. Communication via gestures utilized among the local area of individuals who can't talk or hear as the methods for correspondence. Communication through signing is a motion portrayal that includes at the same time joining hand shapes, direction and development of the hands, arms or body, and looks to communicate fluidly with a speaker's musings. Individuals who can't talk utilizes the communications via gestures to speak with other individual vocally weakened individual and even with other typical individuals who knows the implications of gesture based communications or a translator is expected to decipher the implications of gesture based communications to others who can talk and don't have the foggiest idea about the implications of gesture based communications. The communication gap between -special person and a normal person is one of the main obstacles that this unique person faces. Deaf and dumb persons have a hard time communicating with normal people. This enormous challenge makes them uneasy, and they believe they are being discriminated against in society. It's anything but consistently workable for a person to associate with constantly to decipher the gesture-based communications and not every person can get familiar with the gesture-based communications.

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 |

2295

International Journal for Research in Applied Science & Engineering Technology (IJRASET) ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538 Volume 10 Issue VI June 2022- Available at www.ijraset.com On account of a breakdown in correspondence Deaf and moronic individuals accept they can't impart, and therefore, they can't pass on their feelings. In this manner, another option is that we can utilize a PC or an advanced mobile phone as an arbiter. The HGRVC (Hand Gesture Recognition and Voice Conversion) innovation finds and tracks the hand movements of those who are hard of hearing and unable to speak so they can speak with others. The PC or a PDA could take a contribution from the vocally disabled individual and give its text based just as and sound type of yield. II. BACKGROUND 1) “Real-Time Hand Gesture Recognition Using Finger Segmentation” paper written by Zhi- hua Chen, Jung-Tae Kim, Jianning Liang, Jing Zhang, and Yu-Bo Yuan has mentioned that for human-computer interaction, hand gesture recognition is critical. We describe a revolutionary real-time approach for hand gesture identification in this paper. The hand locale is separated from the setting in our system utilizing the foundation deduction technique. The palm and fingers are then partitioned so the fingers might be identified and perceived. Finally, to predict hand gesture labels, a rule classifier is used. Experiments on a 1300 image data set reveal that our solution works well and is quite efficient. Furthermore, on another data set of hand movements, our method outperforms a state-of- the-art method. 2) “Gesture Based Vocalizer for Deaf and Dumb” paper written by Supriya Shevate , Nikita Chorage, Siddhee Walunj, Moresh M. Mukhedkar have pointed out that gesture vocalizer is a socially conscious project. According to the results of the survey, deaf persons find it extremely difficult to communicate with others. Hard of hearing people typically speak with hand signals, making it hard for others to appreciate their communication through signing. We intend to execute a motion-based vocalizer that will perceive all hard of hearing individuals' motions, make an interpretation of them to discourse, and show them on a LCD screen. To do as such, we're utilizing an ARM 7 regulator to associate the entirety of the sensors and the discourse synthesizer. The information glove comprises two kinds of sensors: a flex sensor and an accelerometer that fills in as a slant sensor. Remote information gloves, which are standard material driving gloves with flex sensors along the length of each finger and the thumb, are utilized. Dumb persons can wear the gloves to make hand gestures, which are then translated into speech so that others may understand them. This technique allows persons who are deaf to communicate with others. 3) “Hand Gesture Recognition and Voice Conversion for Deaf and Dumb” paper written by Cheshta has told that Communication is the primary means by which people communicate with one another. Birth defects, accidents, and oral disorders have all contributed to a rise in the number of deaf and dumb people in recent years. Because deaf and dumb persons are unable to speak with normal people, they must rely on visual communication. People sometimes misinterpret these communications, whether through sign language, lip reading, or lip sync. This project is designed to enable these persons with special needs participate in society on an equal footing. 4) “Orientation Histograms for Hand Gesture Recognition” paper written by William T. Freeman and Michal Roth were informative and expressed that they offer a method for recognizing hand gestures that is based on McConnell's” pattern recognition methodology that uses histograms of local orientation. For gesture categorization and interpolation, we employ the orientation histogram as a feature vector. This approach is simple and quick to compute, and it provides some resistance to fluctuations in scene light. We created a real- time version that can discriminate between a small set of hand movements (around ten). The entire computation takes place on a workstation, with separate hardware only being utilized to digitize the image. A user can use hand gestures to control a computer graphic crane or play a game. We go over the method's drawbacks. The histogram of the spatio- temporal gradients of picture intensity forms the comparable feature vector for moving or "dynamic gestures," and may be beneficial for dynamic gesture detection. 5) “Hand Gesture recognition using a real-time tracking method and hidden Markov models” paper written by Feng-Sheng Chen, Chih-MingFu, Chung-LinHuang said that They present a technique for recognising continuous gestures in front of a stationary background. Ongoing hand following and extraction, including extraction, covered up Markov model (HMM) preparation, and motion acknowledgment are the four segments that make up the framework. To follow the moving hand and concentrate the hand locale, we first utilize a continuous hand following and extraction strategy, then, at that point we utilize the Fourier descriptor (FD) to describe spatial highlights and movement investigation to portray fleeting qualities. As our feature vector, we combine the spatial and temporal properties of the incoming image sequence. We use HMMs to recognise the input gesture after extracting the feature vectors. Different HMMs are used to score the gesture to be recognised. The gesture is shown by the model with the greatest score. We tried our system to recognise 20 various gestures in the experiments, and the recognition rate was above 90%.

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 |

2296

International Journal for Research in Applied Science & Engineering Technology (IJRASET) ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538 Volume 10 Issue VI June 2022- Available at www.ijraset.com 6) “Microcontroller and sensor based gesture vocalizer” by Ata-Ur-Rehman Ata-Ur-Rehman, Salman Afghani, Muhammad Akmal, Raheel Yousaf have given insight that Gesture Vocalizer is a large-scale multi-microcontroller-based system that aims to improve communication between the dumb, deaf, and blind groups, as well as between them and the general public.This system may be configured to function as a mobile "smart gadget". A gesture vocalizer based on a microcontroller and sensors is presented in this work. The gesture vocalizer under discussion is essentially a data glove with a microprocessor. A data glove can detect practically all hand movements, and a microcontroller-based system can translate some of those movements into identifiable human speech. The data glove has two types of sensors: flex sensors that serve as bend sensors and accelerometers that serve as tilt sensors. This device is advantageous to deaf people because it allows their hands to speak once “they have worn the gesture vocalizer data glove. 7) “Hand Gesture Vocalizer for Dumb and Deaf People” paper Written by Sanish Manandhar, Sushana Bajracharya, Sanjeev Karki, Ashish Kumar Jha have told that The significant objective of this work is to introduce a framework that changes over an offered hint utilized by a handicapped individual into its appropriate composed, hear-able, and graphical structure using normal segments like Arduino Mega, Flex sensors, and Accelerometer. A wearable glove regulator is planned with flex sensors on each finger that permit the framework to recognize finger developments, just as a Gy-61 accelerometer that identifies the debilitated individual's hand development. The gathered input signal is sent to the system for processing by the wearable input glove controller. On the present training model, the system employs the Random Forest calculation to anticipate the right yield with a precision of 85%. III. PROPOSED METHODOLOGY A. Random forest Algorithm Random forests, also known as random decision forests, are an ensemble learning method or supervised classification algorithm for classification, regression, and other tasks that work by training a large number of decision trees and then outputting the class that is the mode of the classes (classification) or the mean prediction (regression) of the individual trees. This algorithm generates a forest with a large number of trees, and the more trees in the forest, the more resilient it seems. The decision trees' characteristic of overfitting to their training set is corrected by random decision forests. Ensemble algorithms are algorithms that integrate many methods of the same or different types to categorize things. For example, perform a prediction using Naive Bayes, SVM, and Decision Tree, and then vote on the final class for the test item. Similarly, with the random forest classifier, a larger number of trees in the forest means that the outputs are more accurate. The random forest classifier produces a collection of decision trees from a randomly selected subset of the training set, then combines the votes from the individual decision trees to determine the test object's final class. Finally, predictions are formed based on the majority of votes from each of the decision trees. This approach works effectively because a single decision tree can be susceptible to noise, but aggregating numerous decision trees reduces the influence of noise, resulting in more accurate results for various input sets. The random forest may use the weight notion to assess the influence of each decision tree's output. When a tree's mistake rate is high, it has a low weight value, and vice versa. This algorithm has two stages: the first is to generate a random forest classifier, and the second is to produce a prediction using the random forest classifier built in the first step. The entire procedure is detailed here There are four major steps of the DCP-based image de-hazing, which are atmospheric light estimation, transmission map estimation, transmission map refinement, and image reconstruction. B. Random Forest Prediction Pseudo Code Takes the test characteristics, predicts the outcome using the rules of each randomly generated decision tree, and saves the projected result (target). Determine the number of votes for each forecasted goal. Consider the random forest algorithm's final forecast to be the target with the most votes. When we utilize the random forest technique to solve any classification issue, the problem of overfitting will never arise. Both classification and regression tasks may be solved using the same random forest approach. For feature engineering, the random forest method might be utilized. It entails selecting the most essential characteristics from the training dataset's available feature

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 |

2297

Training set

Test Set

Sample 1

Sample 2

Decision Tree 1

Decision Tree 2

Sample 3

Decision Tree 3

Voting

prediction

Random forest is regarded as an approach that is both accurate and reliable. This approach is free of the overfitting problem and may be used to solve both classification and regression problems. Random forest may also manage missing values in two ways: by replacing continuous variables with median values or by computing the proximity-weighted average of missing values. Because it has several decision trees, the random forest takes a long time to provide predictions. This algorithm takes a long time to complete. In comparison to a decision tree, where you can readily make a decision by following the route in the tree, the model is harder to comprehend. When a user makes a hand motion, the glove controller sends eight user inputs to the system, including a three-axis accelerometer signal and five flex sensor signals. The characteristics for detecting a given gesture are three axis values and five flex sensor values that have been acquired from the user. These gesture characteristics were sent into a random forest classifier, which then categorized the motions based on the features, and the output was recognised. For the real-time application of the trained machine, the flex sensor and accelerometer data from the glove movement were gathered and processed by the Arduino, turning those raw data to useful data. These values aid in the extraction of features. Those data were delivered from the Arduino to the serial port, collected by Python via the serial interface, and recorded in CSV format as a temporary dataset. Finally, the computer ran the temporary dataset through the machine, which extracted the dataset's characteristics and predicted the proper output for that gesture using the Random Forest Classifier model. Because the model identified the output, the output was displayed to the user on the laptop screen and over the speaker IV. CONCLUSION This examination presents another technique for hand signal distinguishing proof in this investigation. The foundation deduction approach is utilized to recognize the hand area from the scenery. The palm and fingers are portioned after that. The fingers in the hand picture are found and perceived dependent on the division. Hand motion acknowledgment is performed utilizing a straightforward guideline classifier. On an informational collection of 1300 hand photos, the exhibition of our strategy is tried. The consequences of the investigations propose that our strategy works adequately and is appropriate for continuous applications. Besides, on a picture assortment of hand developments, the recommended procedure beats the cutting edge FEMD.

2298

International Journal for Research in Applied Science & Engineering Technology (IJRASET) ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538 Volume 10 Issue VI June 2022- Available at www.ijraset.com The proposed technique's presentation is altogether reliant "on the consequences of hand location. On the off chance that there are moving articles with a shading that is like that of the skin, the things are distinguished because of the hand identification and decrease the hand signal acknowledgment execution.However, AI computations can distinguish the hand from the setting. ToF cameras give profundity data that can help upgrade hand discovery execution. To address the confounded foundation issue and improve the heartiness of hand distinguishing proof, AI calculations and ToF cameras might be applied in future exploration. Currently, research efforts have primarily concentrated on recognizing static ISL indications from pictures or video sequences captured under controlled conditions. The dimensionality of the sign recognition process will be minimized by employing the LDA method. Noise will be decreased and with excellent accuracy as a result of dimensionality reduction. This project will be improved in the future by determining the numbers that will be Displayed in words. We sought to build this system by integrating a variety of image processing techniques and basic picture attributes. The recognition of gestures has been accomplished using LDA algorithms. Remembering that every God creature has value in society, let us endeavor to incorporate hearing challenged persons in our daily lives and live together. V. FUTURE SCOPE This research article discusses the design and operation of a machine that allows dumb, deaf, and blind people to communicate with each other and with regular people. The stupid people utilise a common signal language that is difficult to understand by common humans, and blind humans are unable to perceive their motions. This computer translates the signal language into a voice that is easily understood by both blind and non-blind people. Sign language is also translated into a few written content formats to help deaf people. VI. ACKNOWLEDGEMENT It gives us great pleasure in presenting the preliminary project report on ‘GESTURE VOCALIZER USING FLEX SENSOR AND SOFTWARE VISUALIZATION’. REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]

[15]

Sanish Manandhar, Sushana Bajracharya, Sanjeev Karki, Ashish Kumar Jha, “Hand Gesture Vocalizer for dumb and deaf people”, SCITECH Nepal,2019. www.elprocus.com en.wikipedia.org www.hindawi.com www.ijarcce.com Y C Venkatesh,A. S Srikantappa, P Dinesh. “Smart Brake Monitoring System with Brake Failure Indication for Automobile Vehicles”, IOP Conference Series:Materials Science and Engineering,2020. Circuitdigest.com Sawant, Shreyashi Narayan, and M.S.Kumbhar.”Realtime sign language Recognition using PCA”,2014 IEEE International Conference on Advanced Communications Control and Computing Technologies,2014. Yikai Fang, Kongqiao Wang,Jian Cheng, Haning Lu.”A Real-Time Hand Gesture Recognition mafiadoc.com link.springer.com dl.acm.org A. D. Bagdanov, A. Del Bimbo, L. Seidenari, and L. Usai, “Real-time hand status recognition from RGB-D imagery,” in Proceedings of the 21st International Conference on Pattern Recognition (ICPR '12), pp. 2456–2459, November 2012.View at: Google Scholar M. Elmezain, A. Al-Hamadi, and B. Michaelis, “A robust method for hand gesture segmentation and recognition using forward spotting scheme in conditional random fields,” in Proceedings of the 20th International Conference on Pattern Recognition (ICPR '10), pp. 3850–3853, August 2010.View at: Publisher Site | Google Scholar C.-S. Lee, S. Y. Chun, and S. W. Park, “Articulated hand configuration and rotation estimation using extended torus manifold embedding,” in Proceedings of the 21st International Conference on Pattern Recognition (ICPR '12), pp. 441–444, November 2012.View at: Google Scholar

2299