Virtual Mouse Control Using Hand Gesture Recognition
G N Srinivas1, S Sanjay Pratap2 , V S Subrahmanyam 3 , K G Nagapriya4, A Venkata Srinivasa Rao55 Department of ECE, Sasi Institute of Technology & Engineering, Tadepalligudem, W.G.Dist, India.
1,2,3,4UG Students, Department of ECE, Sasi Institute of Technology & Engineering, Tadepalligudem, W.G.Dist, India ***
Abstract - The mouse is an excellent device for humancomputer interaction. We currently use three types of mice: wired, wireless, and Bluetooth. We need power to connect a dongle to a PC in all of these scenarios. This work employs cutting-edge machine learning and computer vision algorithms to recognize hand gestures, which work flawlessly without the use of any hardware. It is compatible with CNN models implemented by mediapipe. In this paper, we work on S.Shriram’s algorithm that [11] propose a method for controlling the cursor's location using only one's hands andno mouse. Some actions, like clicking and dragging items, will necessitate a variety of hand movements. The proposedsystem will only require the use of a single computer. A camera is used as an input device. The following programs will be used: The proposed system necessitates the use of Python and OpenCV. The output of the camera will be displayed on the system's display so that the end-user can fine-tune it.
Key Words: Camera, Machine Learning, CNN Model, Mediapipe, Virtual Mouse, etc
1. INTRODUCTION
Hand gestures are universally recognized as the most expressive and effective form of human communication. Hand signals, thumbs up, and thumbs down have always existed.Gesturesareregardedasthemostnaturalwayfor people to communicate with one another. It has a lot of personality. It was written in such a way that it could be understoodbythedeafanddumb.Sowhynotputittouse on our machines? In this work, we present actual hand gestures. The initial setup includes a low-cost USB web cameraforsysteminput.
Thispaperproposesareal-timehandgesturesystem.The experimental setupof thesystemmakesuseof a low-cost webcamerawithhigh-definitionrecordingcapabilitythatis installed in a fixed position. A camera mounted on a computer monitor is used to photograph a laptop. This project proposes an effective hand gesture segmentation techniquebasedonpre-processing,backgroundsubtraction, andedgedetectionapproaches.
ThePythonprogramminglanguageandOpenCV,acomputer vision library, were used to create the AI virtual mouse system.TheMediaPipepackageisusedtotrackhandsand fingers, as well as the Pynput, Autopy, and PyAutoGUI packagesfornavigatingthecomputer'swindowscreenand
performing operations like left click, right click, and scrolling.
2. LITERATURE SURVEY
Chen-Chiung Hsieh et al. [1] proposed "A Real Time Hand GestureRecognitionSystemUsingMotionHistoryImage"to controlthemousecursor.Theproposedmethodemploysan adaptive skin colour detection model to reduce misclassifications. To develop these methodologies, they used a C++ software platform with the image processing libraryopencvinstalled.KamranNiyazietal[2]proposed "MouseSimulationUsingTwoColoredTapes,"whichused theBackgroundSubtractionmethod,SkinDetectionmethod, and HSV Color Model to control the cursor and perform clickingoperations.Thedistancebetweenthetapecolours wasusedtoguidetheclickingoperations.Thismodelwas created using Java software. Abhik Banerjee et al. [3] proposeda"MouseControlUsingaWebCameraBasedon Color Detection" to control cursor movements and click eventsbydetectingcameracolour.Eachcolourrepresentsa differentcursorcontrol,andclickingactionsareperformed bysimultaneouslydetectingthecolours.Thismethodwas createdwiththehelpofMATLABsoftwareandtheMATLAB imageprocessingtoolbox."Vision-basedComputerMouse ControlUsingHandGestures"[4]wasproposedbySandeep Thakuretal.Toimprovetheefficiencyandreliabilityofthe interaction,thismethodemploysavision-basedsystemto control various mouse activities such as left and right clicking with hand gestures. To improve the system's efficiencyandperformance,differentcolourcapsareusedon fingers to recognise hand gestures. The MATLAB environmentwasusedtoimplementthismethod.Tocontrol the mouse cursor, Horatiu-stefan Grif et al [5] proposed "MouseCursorControlBasedonHandGesture".Theyused anexternalcameraattachedtoahandpadandcolourstrips attached to the fingers in the proposed method. To implement this methodology, they used C programming software along with an image processing library called OpenCV.PoojaKumari etal.[6]proposed"CursorControl UsingHandGestures"forcontrollingamousewithcameracapturedhandgestures.Thecameraactsasasensorinthis method,capturingandrecognisingcolourtipsattached to thehand.Becauseitrequirestheusertohavecolourtipson hishandinordertocontrolthemouse,thismethodisalso known as the marker-based approach method. To implement this methodology, they used the MATLAB environment,theMATLABImageProcessingToolbox,and
the OpenCV library. Using an Extreme Learning Machine Method, Danling Lu et al. [7] proposed a "Gesture Recognition Using Data Glove" method for controlling the mouse cursor and performing clicking operations. In this method,theycollecteddataforgesturerecognitionusinga novel data glove. This glove is inexpensive. Advanced machine learning was used to create this method. Alisha Pradhana et al. [8] proposed the "Design of Intangible Interface for Mouseless Computer Handling Using Hand Gestures"tocontrolthemousecursorandclickoperations. Thismethod,which employsa convexhull algorithm,was implemented using Microsoft virtual studio, a Microsoft integrateddevelopmentenvironment.Todetecttheuser's handandreducenoise,aredgloveisused.AashniHariaaet al. [9] proposed "Hand Gesture Recognition for Human ComputerInteraction,"whichusesabackgroundextraction and contours detection system to improve user-computer interaction.Humaninteractioncanbeaccomplishedthrough avarietyofsensormodessuchasgesture,speech,facialand body expressions. In their paper, Horatiu-Stefan et al. proposeda"HumanHandGestureBasedSystemforMouse Cursor Control". This model was created using a bluecolored hand pad, a webcam, and a computer. Specific operations were carried out based on hand pad postures. This model was created using Visual C++ 2008 and the OpenCV library. For controlling the mouse cursor and performingclickingoperations,Abhilashetal.[10]proposed the "Virtual Mouse Using Hand Gesture" technique. They madea mask out ofa redobject in thiscase. This method wasimplementedusingPythonsoftware,whichincludeda varietyofmodulesandfunctions.
3. METHODOLOGY
We presented the block diagramand flow chartofVirtual Mouse Control Using Hand Gesture Recognition in this section.Wealsoprovidedabriefdescriptionofthesystem's operationandcomponents.
Fig-3.1depictsafunctionalblockdiagramoftheproposed system,whichdemonstrateshowthesystemoperates.We mustholdourhanduptothewebcam.Thewebcamstarts the video and captures the frames. The input image was subjectedtopre-processing.Thefunctionoftheimagepreprimaryprocessoristostandardizetheimage.Theprocess of scaling and pre-processing an image to have similar heightsandwidthsisknownasstandardization.Toimprove thequalityofthestandardimage,itisnowprocessedusing animageprocessingtechnique.Thecameramovesitshand afterimageprocessing,andthefingertipsaredetectedusing MediaPipe and openCV. After recognizing the hand and fingertips,itbeginstodraw.
Therearehandlandmarksandaboxaroundthehandonthe screen.Onthewindowpc,drawarectangularboxtohold themouse.Itwilldeterminewhichofyourfingersisupand which is down. Based on the finger detections, the mouse actionisperformed,andtheprogramreturnstotheframes toperformthenextoperation.Thisishowtheentiresystem operates.
3.1. ALGORITHM
Step1:Start
Step 2: Start the webcam video capture and initialize the system.
Step3: Framecapturewithawebcam.
Step4:UsingMediaPipeandOpenCV,detecthandsandhand tipsanddrawhandlandmarksandaboxaroundthehand.
Step5:Draw arectanglearoundthecomputerwindowarea wherewe'llbeusingthemouse.
Step6:Determinewhichfingerisraised.
Step6.1: Thegestureisneutralifallfivefingersareup,and thenextstepistaken.
Step6.2: Thecursormovestostep2ifboththemiddleand indexfingersareraised.
Step6.3: Adoubleclickisperformedwhenbothindexand middle fingers are joined side by side, and step 2 is performed.
Step6.4: Ifbothindexandmiddlefingersaredown,perform aleftclickandproceedtostep2.
Step6.5:Ifthemiddlefingerisdownandtheindexfingeris up,therightclickisperformedandtheprocessproceedsto step2.
Step6.6:Volumeupanddownareaccomplishedbyjoining thethumbandindexfingersandmovingthemupanddown.
Step7:Toexit,presstheEXITkey.
The preceding algorithm shows how to simulate virtual mousecontrolwithhandgesturesstepbystep.Thismakesit easiertoswapaphysicalmouseforavirtualmouse.Thiswill aid in the conversion of input hand gestures into mouse clickingfunctions.
The preceding procedure, i.e. the algorithm, is depicted diagrammaticallyinflowchart Fig-3.2.
•TheAIVirtualMouseSystemusesacamera.Theproposed AIvirtualmousesystemisbased onframescapturedbya laptoporPC'swebcam.Thevideocaptureobjectiscreated usingthePythoncomputervisionlibraryOpenCV,asshown inFig-3.1,andthewebcamerabeginscapturingvideo.The webcameracapturesimages,whicharethensenttotheAI virtualsystem.
•Thevideoiscurrentlybeingrecordedandprocessed.The AI virtual mouse system uses a webcam to capture each frameuntiltheprogramisterminated.
• Detecting which finger is up and carrying out mouse functions.Atthispoint,weusethetipIdoftherespective fingerdiscoveredusingtheMediaPipeandtherespectivecoordinatesoftheupfingers,asshowninFig-3.3,todetermine which finger is up, and then we perform the appropriate mousefunction.
• Mouse functions based on hand gestures and hand tip detectionusingcomputervision.
4. RESULTS & DISCUSSIONS
FortestingthisalgorithmComputerissetnottoperformany mouseactionsonthescreen.
• ToperformNoActionontheScreen,asshowninFig-4.1,if all of the fingers are up with tip Id = 0, 1, 2, 3, and 4, the computer is set to not perform any mouse events on the screen.
Fornavigatingthecomputerwindowwiththemousecursor. The mouse cursor is made to move around the computer window using Python'sAutoPy package if theindex finger withtipId=1andthemiddlefingerwithtipId=2areup,as showninFig-4.2
•To use the mouse's right-button click. The computer is programmedtoperformtherightmousebuttonclickifboth theindexfingerwithtipId=1andthemiddlefingerwithtip Id=2areupandthedistancebetweenthetwofingersisless than40px,asshowninFig-4.4
• To perform a left-button click with the mouse. The computer is made to perform the left mouse button click usingthepynputPythonpackageifboththeindexfingerwith tipId=1andthemiddlefingerwithtipId=2areupandthe distancebetweenthetwofingersislessthan30px,asshown inFig-4.3.
•Ifboththeindexandmiddlefingersareclosed,themouse will perform a drag operation. As shown in Fig-4.5, the computer is programmed to perform the drag operation buttonclickusingthepynputPythonpackage.
Finally, the images above show the various mouse operations that can be carried out with hand gestures. Recognizing different fingertip ids allows you to perform various mouse operations. The aforementioned mouse functionscanbeperformedeveninlow-lightconditions.
5. CONCLUSIONS
TheprimarygoaloftheAIvirtualmousesystemistocontrol mouse cursor functions with hand gestures rather than a physical mouse. The proposed system can be realized by using a webcam or an in-built camera that detects hand gestures and hand tips and processes these frames to performthespecificmousefunctions.
Basedonthemodel'sresults,concludethattheproposedAI virtual mouse system performed well and has higher accuracy than existing models, and that the model
overcomesmostofthelimitationsofexistingsystems.TheAI virtual mouse can be used for real-world applications because the proposed model is more accurate and can be usedvirtuallyusinghandgesturesratherthanthetraditional physicalmouse.
REFERENCES
[1] Chen-ChiungHsieh,Dung-HuaLiouandDavidLee“A real time hand gesture recognition system usingmotionhistoryimage”Proc.IEEE, 2010,pp. V2-394-V2-398
[2] Kamran Niyazi, Vikram Kumar , Swapnil Mahe and SwapnilVyawahare“MouseSimulationUsingTwo Coloured Tapes”,IJIST 2012, Vol.2, No.2, DOI : 10.5121.
[3] Abhik Banerjee , Abhirup Ghosh , Koustuvmoni Bharadwaj,HemantaSaikia“MouseControlusinga Web Camera based on Colour Detection”IJCTT ,March2014,volume9number1,ISSN:2231-2803
[4] S. Thakur, R. Mehra and B. Prakash, "Vision based computer mouse control using hand gestures," International Conference on Soft Computing TechniquesandImplementations(ICSCTI), 2015, pp.85-89
[5] Horatiu-stefanGrif,Trainturc“Humanhandgesture based system for mouse cursor control” INTERENG,October2017.
[6] PoojaKumari,GhaziabadSaurabhSingh,Ghaziabad Vinay and Kr. Pasi “Cursor Control using Hand Gestures” International Journal of Computer Applications(0975–8887),2016.
[7] DanlingLu,YuanlongYu,andHuapingLiu“Gesture Recognition Using Data Glove: An Extreme Learning Machine Method”Proc.IEEE,December 2016, pp.1349-1354.
[8] Alisha Pradhan , B.B.V.L. Deepak “Design of Intangible Interface for Mouseless Computer Handling using Hand Gestures” ICCCV(International Conference on Communication, Computing and Virtualization), 2016,oi:10.1016/j.procs.2016.03.037.
[9] Aashni Hariaa , Archanasri Subramaniana , Nivedhitha Asokkumara , Shristi Poddara and Jyothi S Nayak “Hand Gesture Recognition for Human Computer Interaction” Proc.ICACC(InternationalConferenceonAdvances in Computing & Communications), 2017 August,pp367–374.
[10] Abhilash S , Lisho Thomas, Naveen Wilson, and Chaithanya “VIRTUAL MOUSE USING HAND GESTURE”Proc.InternationalResearchJournalof Engineering and Technology (IRJET), e-ISSN: 2395-0056p-ISSN:2395-0072,Apr-2018,Volume: 05Issue:04
[11] S.Shriram,B.Nagaraj,J Jaya,SShankarandPAjay “DeepLearning-BasedReal-TimeAIVirtualMouse SystemUsingComputerVisiontoavoidCovid-19 Spread” Journal of Health Care Engineering, Volume:2021,ArticleID:8133076.