Multimodal Touchless Control System Using Eye, Hand, and Voice Interaction

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 13 Issue: 06 | Jun 2026

p-ISSN: 2395-0072

www.irjet.net

Multimodal Touchless Control System Using Eye, Hand, and Voice Interaction Mr. S. Ananthakoothan1, Kiran Kumar A2 1Professor of Department of Computer Science and Engineering, Meenakshi College of Engineering, Chennai,

Tamil Nadu, India

2Student of Department of Computer Science and Engineering, Meenakshi College of Engineering, Chennai,

Tamil Nadu, India ---------------------------------------------------------------------***--------------------------------------------------------------------1.2 Problem Statement Abstract - Human–Computer Interaction (HCI) plays a vital role in improving accessibility for individuals with physical disabilities. This paper presents a Multimodal Touchless Control System Using Eye, Hand, and Voice Interaction, which enables users to operate a computer without physical contact. The proposed system combines eye tracking and hand gesture for cursor control, and voice commands for performing system actions. Developed using Python, OpenCV, MediaPipe, and speech recognition technologies, the system utilizes a standard webcam and microphone, making it costeffective and easy to deploy. By integrating multiple interaction modes, the system provides greater flexibility, accessibility, and user comfort compared to traditional input devices. The proposed solution offers an effective assistive technology for differently-abled users and demonstrates the potential of multimodal interfaces in modern Human– Computer Interaction applications.

Traditional computer interaction methods depend heavily on physical devices such as keyboards and mice, which may not be suitable for users with motor impairments. Although several touchless solutions have been developed, most focus on a single input modality such as eye tracking, hand gestures, or voice commands. This limitation can reduce flexibility and user comfort. Hence, there is a need for a multimodal touchless control system that integrates eye, hand, and voice interactions to provide accessible, accurate, and efficient computer control.

2. EXISTING SYSTEM Several touchless human–computer interaction systems have been developed to assist users in controlling computers without relying on traditional input devices such as a mouse and keyboard. Existing research primarily focuses on singlemodal interaction techniques, including head movement tracking, eye-tracking systems, hand gesture recognition, and voice-based control. Head movement-based systems use computer vision and facial landmark detection algorithms to estimate head orientation and translate it into cursor movements. Similarly, eye-tracking systems detect pupil movements and gaze direction to control cursor navigation and perform click operations. Hand gesture recognition systems employ image processing and machine learning techniques to identify predefined gestures and map them to computer commands, while voice-controlled systems use speech recognition algorithms to execute user instructions through spoken commands.

Key Words: Human–Computer Interaction, Eye Tracking, Hand Gesture Recognition, Voice Control, Touchless Interface, Assistive Technology.

1. INTRODUCTION Human–Computer Interaction (HCI) has become an important area of research for improving the way users interact with computers. Traditional input devices such as keyboards and mice require physical contact and precise hand movements, which can be challenging for individuals with physical disabilities or motor impairments. Recent advancements in computer vision and speech recognition have enabled the development of touchless interaction systems that provide a more natural and accessible way of controlling digital devices.

Although these systems provide an alternative means of interaction, they have several limitations. Most existing solutions depend on a single mode of input, reducing flexibility and making the system less adaptable to different user needs. Continuous use of head or eye movements may cause fatigue and discomfort, while hand gesture systems can be affected by lighting conditions, background noise, and camera positioning. Voice-controlled systems may experience reduced accuracy in noisy environments and often cannot independently perform precise cursor control tasks. In addition, many existing systems require frequent calibration and are unable to effectively combine multiple interaction methods. These limitations highlight the need for a

1.1 Motivation People with disabilities, hand injuries, or limited mobility often face difficulties while using traditional computer input devices. Existing touchless systems usually rely on a single interaction method, which may limit usability and accuracy.

Therefore, there is a need for an integrated system that combines multiple control methods to provide a more efficient and user-friendly interaction experience.

© 2026, IRJET

|

Impact Factor value: 8.315

|

ISO 9001:2008 Certified Journal

|

Page 755

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 13 Issue: 06 | Jun 2026

p-ISSN: 2395-0072

www.irjet.net

multimodal touchless control system that integrates eye tracking, hand gesture recognition, and voice interaction to provide a more accessible, accurate, and user-friendly computing experience.

1.

Start the system and initialize the webcam, microphone, and required libraries.

2. Capture video frames and audio input continuously.

3. Allow the user to select either Eye + Voice mode or Hand + Voice mode.

4. Preprocess the captured visual and audio data to remove noise and improve quality.

5. If Eye + Voice mode is selected: o Detect facial landmarks and track eye movements. o Map eye gaze direction to cursor movement. o Detect eye blinks for click operations.

6. If Hand + Voice mode is selected: o Detect hand landmarks and recognize predefined gestures. o Map hand gestures to cursor movement and control actions.

7. Process voice input using a speech recognition engine.

8. Identify spoken commands such as scrolling, zooming, opening applications, or system operations.

9. Convert the detected eye movement, hand gesture, or voice command into the corresponding computer action.

10. Execute the action and provide real-time system response.

11. Repeat the process until the user exits the application.

12. Release all resources and terminate the system.

3. PROPOSED SYSTEM The proposed Multimodal Touchless Control System Using Eye, Hand, and Voice Interaction is designed to enable users to operate a computer without physical contact by combining visual and speech-based interaction techniques. The system provides two modes of operation: Eye + Voice and Hand + Voice, allowing users to choose the most suitable method based on their comfort and physical abilities. In the Eye + Voice mode, eye movements are used for cursor navigation and click operations, while voice commands are utilized for tasks such as scrolling, zooming, and system control. In the Hand + Voice mode, hand gestures are employed for cursor movement and interaction, whereas voice commands assist in executing additional system functions.

The proposed system utilizes a webcam and microphone to capture visual and audio inputs in real time. Computer vision techniques are used to detect eye movements and hand gestures, while speech recognition algorithms process voice commands. The extracted features are analyzed and translated into corresponding computer actions through a decision-making module, enabling smooth and efficient interaction.

Compared with existing single-modal systems, the proposed approach offers greater flexibility and accessibility by providing multiple control options. Users can switch between Eye + Voice and Hand + Voice modes according to their needs, reducing fatigue and improving usability. The system is particularly beneficial for individuals with physical disabilities, motor impairments, or temporary injuries, offering a cost-effective and user-friendly solution for touchless human–computer interaction.

End Algorithm

5. IMPLEMENTATION

Algorithm: Multimodal Touchless Control System

The proposed Multimodal Touchless Control System was implemented using Python and various computer vision and speech processing libraries. A standard webcam is used to capture eye movements and hand gestures, while a microphone is used to receive voice commands. MediaPipe and OpenCV are utilized for real-time detection and tracking of facial and hand landmarks, and speech recognition techniques are employed to process voice inputs. The system operates in two modes, namely Eye + Voice and Hand + Voice, allowing users to perform cursor control, clicking, scrolling, zooming, and other system operations without physical contact. The implementation was carried out on a Windows-based environment and demonstrated smooth and responsive touchless interaction.

Input: Webcam video stream and microphone audio input

6. MODEL ARCHITECTURE

Output: Cursor movement, click actions, and system command execution

The architecture of the proposed Multimodal Touchless Control System illustrates the flow of visual and audio inputs through eye, hand, and voice processing modules. The

4. ALGORITHM The proposed multimodal touchless control system follows a structured process to convert eye movements, hand gestures, and voice commands into computer actions. The algorithm ensures real-time interaction, accurate command execution, and seamless operation through Eye + Voice or Hand + Voice control modes.

© 2026, IRJET

|

Impact Factor value: 8.315

|

ISO 9001:2008 Certified Journal

|

Page 756

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 13 Issue: 06 | Jun 2026

p-ISSN: 2395-0072

www.irjet.net

system integrates these inputs through a centralized control mapping unit, enabling users to interact with computers using either Eye + Voice or Hand + Voice modes. This architecture ensures efficient, accurate, and touchless human–computer interaction while improving accessibility and user convenience.

cost-effective solution for touchless human–computer interaction.

8. CONCLUSION The proposed Multimodal Touchless Control System Using Eye, Hand, and Voice Interaction provides an effective and user-friendly approach for touchless human– computer interaction. By integrating Eye + Voice and Hand + Voice control modes, the system enables users to perform cursor navigation, command execution, and system operations without relying on traditional input devices. The use of computer vision and speech recognition technologies ensures real-time performance, accessibility, and ease of use while requiring only a standard webcam and microphone.

The developed system is particularly beneficial for individuals with physical disabilities, motor impairments, or temporary injuries, as it offers a flexible and cost-effective alternative to conventional computer interaction methods.

The experimental results demonstrate that the system can perform touchless operations accurately and efficiently under normal conditions. In the future, the system can be enhanced with advanced artificial intelligence techniques, improved gesture recognition, multilingual voice support, and integration with smart devices to further improve its functionality and user experience.

Fig -1: Architecture for proposed system

7. RESULTS AND DISCUSSION

REFERENCES

The proposed Multimodal Touchless Control System was successfully implemented using Python, OpenCV, MediaPipe, and speech recognition technologies. The system effectively enabled touchless computer interaction through two operating modes: Eye + Voice and Hand + Voice. In the Eye + Voice mode, eye movements were accurately tracked to control cursor navigation, while voice commands were used to perform actions such as scrolling, zooming, and executing system functions. Similarly, the Hand + Voice mode allowed users to interact with the computer through hand gestures combined with voice commands, providing a flexible and intuitive user experience. The system operated in real time using a standard webcam and microphone, eliminating the need for specialized hardware.

[1] Touchless Head-Control (THC): Head Gesture Recognition for Cursor and Orientation Control, IEEE Transactions on Neural Systems and Rehabilitation Engineering, June 30, 2022.

[2] K. S. Varun, I. Puneeth, and T. P. Jacob, “Virtual Mouse Implementation Using OpenCV,” 2022.

[3] V. S. Vasisht, S. Joshi, S. Shreedhar, and C. Gururaj, “Human–Computer Interaction Based Eye Controlled Mouse,” in Proceedings of the Third International Conference on Electronics Communication and Aerospace Technology, 2021.

[4] V. Khare, S. G. Krishna, and S. K. Sanisetty, “Cursor Control Using Eye Ball Movement,” 2023.

The experimental results demonstrated that the proposed system provides smooth and responsive interaction with minimal latency under normal lighting and environmental conditions. The integration of visual and speech-based inputs improved usability and reduced dependence on traditional input devices such as keyboards and mice.

Compared to existing single-modal systems, the proposed approach offers greater accessibility, flexibility, and user comfort. The system was found to be particularly beneficial for individuals with physical disabilities, motor impairments, or temporary injuries, highlighting its potential as an effective assistive technology. Overall, the results confirm that the proposed multimodal framework is a practical and

© 2026, IRJET

|

Impact Factor value: 8.315

[5] K. Meena, M. Kumar, and M. Jangra, “Controlling Mouse Motions Using Eye Tracking Using Computer Vision,” in Proceedings of the International Conference on Intelligent Computing and Control Systems, 2021.

[6] M. A. Haq, S.-J. Ruan, M.-E. Shao, Q. M. U. Haq, P.-J. Liang, and D.-Q. Gao, “One Stage Monocular 3D Object Detection Utilizing Discrete Depth and Orientation Representation,” IEEE Transactions on Intelligent Transportation Systems, May 2022, doi: 10.1109/TITS.2022.3175198.

|

ISO 9001:2008 Certified Journal

|

Page 757

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 13 Issue: 06 | Jun 2026

p-ISSN: 2395-0072

www.irjet.net

[7] J. K. Muguro et al., “Development of Surface EMG Game Control Interface for Persons with Upper Limb Functional Impairments,” Signals, vol. 2, no. 4, pp. 834–851, Nov. 2021.

[8] J. M. Vojtech, S. Hablani, G. J. Cler, and C. E. Stepp, “Integrated Head Tilt and Electromyographic Cursor Control,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 28, no. 6, pp. 1442–1451, Jun. 2020.

[9] W. Rahmaniar, Q. M. U. Haq, and T.-L. Lin, “Wide Range Head Pose Estimation Using a Single RGB Camera for Intelligent Surveillance,” IEEE Sensors Journal, vol. 22, no.

11, pp. 11112–11121, Jun. 2022.

[10] R. H. Abiyev and M. Arslan, “Eye Mouse Control System for People with Disabilities,” Expert Systems, vol. 37, no. 1, Art. no. e12398, Feb. 2020.

BIOGRAPHIES Mr. S. Ananthakoothan, Professor, Dept of Computer Science and Engineering, Meenakshi College of Engineering, Chennai, Tamil Nadu, India

Kiran Kumar A, Student, Dept of Computer Science and Engineering, Meenakshi College of Engineering, Chennai, Tamil Nadu, India

© 2026, IRJET

|

Impact Factor value: 8.315

|

ISO 9001:2008 Certified Journal

|

Page 758