OpenCV-based Gesture Controlled Robot

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

OpenCV-based Gesture Controlled Robot Shreyas Paimode1, Shreyash Girge2, Dr. Mugdha Kango3, Dr. Anil Shirsat 4 1,2,3,4Electronics & Tele. Dept., PES Modern College of Engineering, Shivajinagar, Pune

---------------------------------------------------------------------***---------------------------------------------------------------------

Abstract - The current project focuses on developing a

equipped with accelerometers and flex sensors to record hand movements, which are then wirelessly transmitted to the robot device through NRF24L01 transceivers.

Although such an arrangement is feasible and effective, it has limitations that can impair its operational performance, such as signal interference and sensor noise.

hand gesture-controlled robot that functions in real time, utilizing the OpenCV library for the detection and interpretation of hand gestures. A camera captures visual data, which is subsequently processed to identify and analyze hand movements. The robot's actions are controlled by precise commands that are based on the movement that was detected. The main goal is to make an interface that is easy to use and doesn't require hands. This will improve the interaction between people and robots, which is especially helpful for people with physical disabilities or in situations where traditional input methods would be impractical. This paper examines pertinent research on gesture recognition and control, outlining key challenges and results.

A Virtual Blackboard Operating Through Hand Gestures. Soroni has developed a new virtual blackboard system in which users can write on a virtual screen with their hand gesture alone, which is recorded using a webcam. It recognizes and processes the gesture using OpenCV and therefore is quite suited to virtual classrooms or virtual learning environments. But it is highly sensitive to light, which can limit its use in other environments of contexts.

Key Words: Gesture recognition, Human-robot interaction, OpenCV, Real-time control, Robotics.

A control mechanism where a robot tracks hand movements on the palm was proposed by Floersch and Li. The system uses accelerometers and infrared cameras to detect hand movements in real time. Even though the system is simple and remarkably accurate, it has problems when used with low-power processors, particularly when more computation is needed.

1. INTRODUCTION Robotics has made a lot of progress in the last few decades, especially with the use of smarter and more creative control systems like gesture recognition. Gesture control is an even more natural way to interact with machines than traditional methods that involve direct physical contact or remote controls. You can do it with simple hand gestures. This not only improves the overall user experience, but it also makes the technology easier to use, especially for people with disabilities who can't touch the devices. So, this gives people more freedom and makes it easier to use.

Using a Raspberry Pi 3 and a PiCamera, Emma Kaufman and Chloe Kuo developed a robot that recognizes hand gestures. The robot has an accelerometer to improve movement accuracy, pressure sensors to improve interaction, and an ultrasonic sensor to prevent collisions.

It is also fed gesture commands via Bluetooth. Although the design is usable, its accuracy in detecting gestures is affected by lighting variations, making it unreliable.

We were able to make a robot that can be controlled by hand gestures in this project. The system is built on the powerful OpenCV library, which is the main part that recognizes and interprets hand gestures through a live video feed. The main goal was to teach the robot how to read gestures correctly so that it could move in different directions based on the signals it saw. This technology has a lot of potential for many uses, from helping patients in hospitals to making smart home automation and industrial processes possible with touchless, easy-to-use controls.

In order to enable movement control, Harish Kumar Kaura and his colleagues created a robotic system that could recognize hand gestures through a webcam and process them using OpenCV. The system can effectively translate gestures into commands and exhibits outstanding real-time response. But like the majority of vision-based systems, it struggles to recognize more complex gestures and is affected by changes in lighting.

A head-controlled wheelchair was created by Pathan et al. to help people with physical disabilities. The system makes use of an RF module for wireless communication and a number of sensors, including the MPU6050 sensor for movement detection. The innovation increases the system's operational flexibility by enabling it to switch between head gesture control and joystick control.

2. LITERATURE REVIEW A robot arm system was created by Chong et al. with the purpose of handling hazardous materials in facilities such as nuclear power plants and poisonous waste treatment facilities. The system makes use of a glove

© 2025, IRJET

|

Impact Factor value: 8.315

|

ISO 9001:2008 Certified Journal

|

Page 540

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

Although head gestures are the primary focus of the paper, it also shows how gesture control and wireless technology can be used in a variety of contexts. This approach aligns with the goals of our hand gesture-controlled robot system, particularly in assistive technology.

moving backward. The ESP32 microcontroller receives the command and uses it to regulate the robot's motion.

D. System Architecture The entire framework comprises a series of connected elements each with a certain role:

3. METHODOLOGY

• Gesture Detection Module: Utilizes the webcam to take a steady stream of photographs, which are analyzed employing OpenCV for identifying certain hand gestures.

A. Overview of the System Our system is a hand-gesture-controlled robot that can follow certain movement instructions and react to hand gestures. There are three main parts to it:

• Command Processing Module: Converts the identified gestures into a direct control command that the robot can read and follow.

• Webcam: Continuous live video capture of hand movements.

• Movement Control Module: Passes the instructions after processing to the Cytron Motor Driver (MD30C series), which subsequently drives the motors to execute the intended movement.

• OpenCV-based Gesture Recognition: identifies various hand gestures from video streams by using image processing techniques made possible by the OpenCV library.

4. BLOCK DIAGRAM AND DESCRIPTION

The ESP32 microcontroller: it receives the recognized gesture data and transforms it into the appropriate movement commands for the robot.

B. Components of Hardware The following are the system's fundamental hardware components: • ESP32: The robot's brain, this component takes input from the gesture recognition module and uses it to control the appropriate motors.

• Cytron Motor Driver (MD30C series): Regulates the robot's DC motors' speed and direction.

• DC motors: Give the robot mechanical movement so it can move in response to commands.

Fig -1: Block Diagram A webcam module or a camera module must first capture video frames.

• The webcam provides input for the visual processing system by continuously recording the user's hand movements.

Gesture Motion Capture: Hand gestures are recognized by processing the captured images. Included are hand detection and user tracking.

C. Software Design It is intended for accurate command execution and efficient gesture recognition:

Hand Gesture Detection: OpenCV allows the system to recognize the hand region in each picture. Features such as palm orientation, hand shape, and finger locations are used.

• OpenCV Gesture Recognition: OpenCV helps the system process the webcam images. Hand shapes are detected and recognized using methods such as contour detection, skin color filtering, and background subtraction.

Command Identification Using a Designated Methodology: Certain gestures (such as the thumbs up, closed fist, and open hand) are associated with the features that have been extracted. The robot can be controlled using these gestures as instructions.

• Command Mapping and Execution: After a gesture is identified, it is mapped to a specific robot command, which may include turning left and right, moving forward, or

© 2025, IRJET

|

Impact Factor value: 8.315

|

ISO 9001:2008 Certified Journal

|

Page 541

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

Create Individual Signals: The system creates individual control signals after detecting the gestures. These signals tell the robot what to do (e.g., turn left, move forward).

Transmit directive to microcontroller: The robot's microcontroller receives the appropriate instruction from the system.

Bluetooth Communication: The ESP32 receives the control signals remotely via Bluetooth technology. The ESP32 serves as an interface between the robotic system and the computer.

Execute robot movement (arm movement): The robot follows the gesture by carrying out the appropriate movement.

End: The process is over.

The ESP 32 microcontroller decodes incoming gesture commands and translates them into motor control actions.

6. RESULTS AND DISCUSSIONS

The MD30C Series Cytron motor driver is a control signalaccepting module.

It receives inputs from the ESP32 and uses the recognized gestures to control the robot's DC motors.

DC Motors: Finally, the robot's DC motors respond to the control signals. The operator's hand movements are followed by the robot

5.SOFTWARE DESIGN

Fig -1: OpenCV Based Gesture Controlled Robot The system's functionality and performance were evaluated using a range of hand movements and lighting conditions.

A. Accuracy of gesture recognition In good lighting, its mean gesture recognition accuracy was about 92%; however, when the robot's gesture response is taken into account, it performs up to 30% worse than mean in low or excessively bright lighting.

Fig -1: Software Design

B. Instant Reaction

Start: The procedure gets started.

According to the results, the average gesture to robotic movement reaction time was 150 milliseconds, which is usually adequate for most real-world uses.

However, the detection and execution processes were delayed by more complex gestures that involved finger overlap.

Start setting up the robotic apparatus, sensors, and camera: Every piece of hardware is turned on by the system.

The camera records the user's hand gesture.

Image processing to find gesture: The gesture is found by processing the image.

C. Robustness of the System Because the system is sensitive to variations in lighting, it performed well indoors but poorly outdoors. The system may become sensitive to variations in lighting and environmental conditions with further research.

Map gesture to corresponding action: An action is assigned to the gesture that has been identified.

© 2025, IRJET

|

Impact Factor value: 8.315

|

ISO 9001:2008 Certified Journal

|

Page 542

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

7. APPLICATIONS

9. REFERENCES

It has numerous applications in gesture-controlled robots.

[1] F. Soroni et al., “Hand gesture based virtual blackboard using webcam,” Proc. IEEE 12th Ann. IEMCON, 2021. researchgate.net+3smartquantai.com+3

Medical care: During a pandemic like COVID-19, a gesture-controlled robot can be very helpful. Its touch-free operation reduces the risk of viruses spreading through direct contact with surfaces and eliminates the need for physical contact altogether. To reduce virus exposure, medical personnel can use gesture-controlled robots to deliver supplies, food, or medication to patients without going into their rooms. Additionally, the robot can communicate with and care for patients remotely thanks to gesture navigation, making sure safer medical treatment without violating social distancing.

[2] B. Zain, S. Hassan, B. Mir, and R. H. Dar, “Robotic hand control using hand gesture recognition,” Int. Res. J. Eng. Technol., vol. 6, no. 2, Feb. 2019. studylib.net+1irjet.net+1 [3] A. Budhiraja, “Gesture controlled robotic arm for social distancing,” Int. J. Res. Anal. Rev., 2023.

[4] S. Dasgupta and A. Ghosh, “Bluetooth controlled Arduino based robotic arm,” IRJET, 2020.

Industrial Automation: To minimize human interaction with hazardous machinery, gesture-controlled robots can be used in factories or other hazardous settings.

[5] R. R. Prabhu and R. Sreevidya, “Design of robotic arm based on hand gesture control system using wireless sensor networks,” Int. Res. J. Eng. Technol., 2017.

8. CHALLENGES AND FUTURE WORK

[6] N. Anughna, V. Ranjitha, and G. Tanuja, “Wireless robotic arm using flex and gyroscopic sensors,” Int. J. Recent Technol. Eng., vol. 8, no. 5, Jan. 2020.

The most critical issues encountered during the project are:

[7] S. M. K. Pathan et al., “Wireless head gesture controlled robotic wheelchair system,” J. Sensor Technol., vol. 10, pp. 47–59, 2020.

Lighting Sensitivity: A lot of light affects how well a system works. This significantly affects gesture recognition accuracy. Future designs should therefore take into account algorithmic adaptation to varying light intensities. Complex Gesture Recognition: When fingers overlap, the system is unable to recognize complex gestures. CNN and other machine learning algorithms can significantly increase the precision of gesture recognition.

[8] H. K. Kaura et al., “Gesture controlled robot using image processing,” Int. J. Adv. Res. Artif. Intell., vol. 2, no. 5, 2013 [9] C. K. Wen, L. C. Yong, and C. Nataraj, “Gesture controlled robotic arm for handling hazardous chemicals,” J. Appl.

Technol. Innov., vol. 6, no. 3, pp. 47–54, 2022.

Environmental Robustness: The system has undergone a great deal of indoor testing. To test the system as a whole, it needs to be tested in industrial settings or outdoors. In the future, the system will be strengthened by adding depth sensors such as Kinect or Intel RealSense, enhancing the gesture recognition algorithm, and expanding the number of gestures to accommodate increasingly complex commands.

[10] J. Paterson and A. Aldabbagh, “Gesture-controlled robotic arm utilizing OpenCV,” Proc. IEEE IEMCON, 2021.

[11] J. Qi et al., “Computer vision–based hand gesture recognition for HRI,” Int. J. Mach. Learn. Comput., 2023 [12] S. A. Hande and N. R. Chopde, “Automating industrial applications using gesture-controlled robotic arms,” Int. J. Innov. Res. Sci. Technol., 2020.

9. CONCLUSION

[13] N. Chaudhari et al., “Hand gesture control for robotic arms using OpenCV,” 2023.

This paper presents real-time hand gesture recognition using OpenCV and a gesture-controlled robot. It has been difficult for the system to manage intricate gestures and lighting changes because of the desire to showcase these systems to the world in an attempt to develop gesturebased control for applications like home automation, healthcare, and industrial automation. These limitations will be addressed in future studies to produce a reliable and flexible gesture-controlled robot.

© 2025, IRJET

|

Impact Factor value: 8.315

[14] A. Nale et al., “Gesture recognition system for robotic cars,” 2015.

[15] E. Kaufman and C. Kuo, “Gesture-controlled educational robot prototype,” 2020.

[16] J. Floersch and J. Li, “Palm-steered point-and-follow control system using accelerometers and IR cameras,” 2019.

|

ISO 9001:2008 Certified Journal

|

Page 543