Virtual Mouse Control Using Hand Gestures

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume: 10 Issue: 01 | Jan 2023 www.irjet.net p-ISSN:2395-0072

Virtual Mouse Control Using Hand Gestures

LY B. Tech Computer Engineering, Science & Technology, Vishwakarma University, Pune, India – 411048 5Assistant Professor, Dept. of Computer Engineering, Vishwakarma University, Pune, India - 411048*** -

ABSTRACT

TherehavebeenmanydevelopmentstowardstheHuman-ComputerInteraction(HCI).Manymoduleshavebeendevelopedto helpthephysicalworldinteractwiththedigitalworld.Mostexistingapproachesinvolvechangingmousepartssuchasadding morebuttonsorchangingthepositionofthetrackingball.Ourproposedsystemservesanewapproachforcontrollingmouse movement using Colored object, marker motion tracking and hand tracking. The project mainly aims at mouse cursor movementsandclickeventsbasedontheobjectdetectionandmarkeridentification. Wewillbeusingacameraforcomputer visiontechnology,suchasimagesegmentationandgesturerecognition,tooverseemousetasks(leftandrightclicking,doubleclicking,andscrolling)andshowhowitcanperformeverythingthatthecurrentmousedevices can.Thesoftwarewillbe developedinPythonLanguage,OpenCVandPyAutoGUIformouse functions.Identifyinggesturesisadifficulttaskwhich involves many aspects such as motion modeling, motion analysis, pattern recognition and machine learning. We will use coloredobjectstoperformactionssuchasthemousemovementandclickevents.Thismethodmainlyfocusesontheuseofa WebCameratodevelopavirtualHuman-ComputerInteractiondevice.

Keepingalltheimportantfactorsinmindasystemhasbeencreatedwhichidentifiesthemovementofhandanditsvarious gestures.Henceafterrecognizingthegesturesvariousmouseactivitiescanbeperformed.

Keywords:Human-ComputerInteraction(HCI),Motiontracking,Handtracking.

1. INTRODUCTION

1.1 INTRODUCTION

Computertechnologyhastremendouslygrownupoverthepastyearsandhasbecomeanecessarypartofeverydaylife.The firstComputerpartthathuman’saccessforHumanComputerInteraction(HCI)isthemouse.Themouseislessappropriatefor HCIinsomerealitythings,likewithHumanMachineInteraction.Therehavebeenmanyexplorationsondifferentwaysto controlthecomputermouseforHCI.ThefirstnaturalandintuitivetechniqueforHCI,thatisaviablereplacementforthe computermousecanbetheintroductionofhandgesturesintocomputersystems.Thisprojectisthusmainlyfocusedtowards developingacomputermanagementsystemusinghandgestures.Mostlaptopsandsystemswhicharehomethesedaysare equippedwithwebcamsthathavegreatlyhelptodeployinsecurityapplicationsutilizingfacerecognitionsoastogetthe extendeduseofadigitalcamera,itmaybeutilizedforvisionbasedmostlyCC,whichmightconstructivelyhelpsustoeliminate thenecessityforamouseormousepad.TheutilizationofadigitalcameramaybegreatlyextendedtodifferentHCI(Human ComputerInteraction)applicationslikeasignallanguagedatabaseormotioncontroller.Overthepastdecadestherehavebeen vitaladvancementsinHCItechnologiesforgamingfunctions.Wearelookingforwardtoapplyingtechniquestouseacamera andcomputertechnology,suchasimagesegmentationandgesturerecognitiontomanagetasksthatareperformedbyMouse andshowhowitcanperformeverythingthatthecurrentmousedevicescan.

1.2 MOTIVATION

Themainpurposeofimplementingthissystemistohelptheambitiousphysicallychallengedpeoplewhowanttoworkona computer.Therearemanyambitiousphysicallychallengedpeopleinsocietywhowanttoworkonacomputer.Thissystem wouldhelpthem.Therearemanypeoplewhodon'tfeelcomfortablewithatouchpad.Thissystemwouldprovideflexibilityto peopleandwouldbeconvenientforuserswhoarenotcomfortablewithatouchpad.Weallwanttomirrorourexperiencesin therealworldintoacomputer.Thissystemwouldallowthat.

1.3 AIM AND SCOPE OBJECTIVE

Thisprojectattemptstousegesturestocontrolthecomputercursorratherthanpointingandclickingamouseordirectly touchingthedisplay,loweringhardwarecostsandincreasingthequalityoflifeforphysicallychallengedpersons.Oneofthe

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume: 10 Issue: 01 | Jan 2023 www.irjet.net p-ISSN:2395-0072

project'sgoalsistoassistpeoplewhoaresufferingfromCarpalTunnelSyndrome.CarpalTunnelSyndromeaffectsmorethan 10millionpeopleinIndiaeachyear.Thiswouldbebeneficialtothem.

1.4 OUR CONTRIBUTION

Wedevelopedanalgorithmtocontrolmousemovementandothermouseactivitiesbyusinghandgesturestoeliminatethe usageofmouseasfaraspossible.Tosummarize,thispapermakesthefollowingcontributions:

● ThisprojectprovidesamajorpreventiontotheCarpalTunnelSyndrome.Recentlyalltheworkhasbeenrunningon PCs.Duetotheheavyusageofmouse,manypeopleareaffectedwithCarpalTunnelSyndrome.Asperthestatistics, therearemorethan10millioncasesperyearinIndia.Carpaltunnelsyndromeisacommonconditionthatcauses pain,numbness,andtinglinginthehandandarm.Theconditionoccurswhenoneofthemajornervestothehand,the mediannerveissqueezedorcompressedasittravelsthroughthewrist.

● The algorithm works in such a way that people with physical challenges could use the PCs easily through hand gestures.

● WehaveimplementedthisprojectinourhomessothattheelderlypeoplefinditmucheasiertointeractwiththePCs.

2. LITERATURE SURVEY

a. Description of Related Theory:

TherearetwomainapproachesforHandGestureRecognitionforHCI(Human-ComputerInteraction);thefirstishardware basedandthesecondisvision-based.Oneofthehardware-basedapproachesproposedbyQuam(1990)usesadatagloveto achieve gesture recognition. This method needs the user to wear a sophisticated data glove which makes some gestures difficulttoperform.Vision-basedHCIcanbeclassifiedintotwocategories,Colormarker-basedapproach,andcolormarkerlessapproach.Thecolormarker-basedapproachrequirestheusertowearcolormarkersorgloves,whilethecolormarkerless approachdoesn’trequirethat.

b. Microsoft applied for patent based on Gesture and Voice recognition system:

TheengineersatMicrosofthaveappliedfortwopatentsdetailingtheArchitecturetocontrolthepersonalcomputer usinghandgesturesandvoiceinput.EarlierMicrosoftshowedcontrollingdevicesbymusclemovement,andnowthey areinterestedingestureandvoice-basedsystems.Theirbasicideaistousevoice,gesture,mouse,orkeyboardasinputs toacomputer.Theytalkaboutusingbothvoiceandgesturetogethertocontrolvariousactionsonthecomputer.To ascertainwhentoconsidermovementasinput,itdependsonthevolumeareaaroundit.Thecorrespondencewiththe computerhastobewithinthevolumeareabeforethesystemstartsconsideringthemovementasinput.Whenagesture ismade,thescreenwillidentifyitandshowyouoptionsonthescreenlikemove,close,scroll,andflick.

c. Kinect for Xbox 360 will be using gesture and voice commands:

Kinectisacontroller-freegamingexperiencefortheXbox360videogameplatform.Microsoftsaysthatmaybelaterit willbesupportedbypersonalcomputersviaWindows8.Itisbasedonawebcam-styledevicefortheXbox360console. ItallowsuserstocontrolandinteractwiththeXbox360withoutthenecessitytophysicallytouchthecontrollerthrough auserinterfaceusinggestures,spokeninstructions,orgivenobjectsandpictures.Theprojectisaimedtowardwidening theXbox360'saudiencebeyonditstypicalgamerbase.

d. A Gesture-Based control for handheld devices using Accelerometers:

TheKoreanDepartmentofInformationandCommunicationpresentedapapertodemonstratehowthesignalsfroman accelerometercanbeprocessedtoaccuratelyidentifyusergesturesafterapplyingaslightmovementtoahandheld device.Forgesture-basedcontroltobeefficaciousinhandhelddevices,theoverheadsinvolvedinrecognizinggestures shouldbeminimal,andthemotionshouldbeaccuratelyidentifiedinrealoperationalenvironments.

3. PROBLEM STATEMENT .

Aswelookatthetopic,CursorMovementonobjectmotion,itisanapplicationthathelpsinacknowledginghandgesturesor movementofthehandwhichisbeingusedbypeople.Itcanbeattainedbytheutilizationofimageprocessingtechniques.Itis

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume: 10 Issue: 01 | Jan 2023 www.irjet.net p-ISSN:2395-0072

themostinterestingapplicationthatonecanworkonandimplementinthisphysicalworldinacomfortableway.Itisoneof theapplicationswhichwillworkonhandgesturesthatarebeingshownbytheuserthroughthesystemusingthisprojectina comfortablemanner.

CursorMovementonObjectMotionisasystemthatusesnophysicaldeviceforcontrollingthemousecursorofthecomputer. Basicallywewillbeusingourpalm.Thevideoofthemotionofthehandwillbecapturedbythewebcamerawhichwillbe actingasasensor.Thehandistrackedandbylookingatitsmotion,themousecursorisbeingcontrolled.Inorderforthe cursortomove,wewilluseourhandwithintheviewingareaofthecamera.There'saRegionofInterestinwhichthehand shouldbeplaced.Thevideogeneratedbythecameraisdetectedandanalyzedusingimageprocessingandthecomputercursor movesordisplaysitsclickeventsaccordingtothehandmovementanditsgestures.

4. PROPOSED SYSTEM ARCHITECTURE .

Theproposedsystemisvision-based,anditemploysimageprocessingtechniquesandinputsfromacomputerorwebcamera. The input frame would be obtained from the web camera, and the system would be split down into four stages, object detection,handcontourextraction,andtheskinregionwouldbedetectedvictimizationskindetection.Thehandshapewould thenbediscoveredandusedforgesturedetectionandhandtrailing.Handmovementswouldbeusedtoaccomplishmouse actions such as right click, left click, scroll up and scroll down, and hand trailing would be used to navigate the Personal Computerpointer.Asaresult,theproject'sgoalwouldbetocreateavision-basedCCsystemthatcandoavarietyoftasks.The suggestedsystem'sarchitectureisdepictedinthediagrambelow.

Figure

4.1 System Architecture for Cursor Movement on Object Motion.

5. METHODOLOGY

a. Hand Contour Extraction

Afterobtainingtheskinsegmentedbinaryimage,edgedetectionisusedtoobtainthehandcontourinsidetheimage.Many edgedetectionalgorithmsareavailable,includingLaplacianedgedetection,cannyedgedetection,andboundarydiscovery.The OpenCVfunctioncvFindContours()searchesthepictureforcontoursusinganorderfindingedgedetectionalgorithm.Thekey benefitoftheborderfindingedgedetectiontechniqueisthateachcontouridentifiedintheimageissavedinanarray. This meansthatwecandeterminethehandshapebyanalyzingeachcontourintheimageindividually.ThecannyandLaplacian edgedetectorscanrecognizethecontoursinapicture,buttheydon'tgiveusaccesstoeverysingleone.Asaresult,inthe planned design, the boundary finding edge detection technique was adopted. We have a propensity to be interested in

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume: 10 Issue: 01 | Jan 2023 www.irjet.net p-ISSN:2395-0072

extracting the hand contour as part of the contour extraction approach so that form analysis may be performed on it to determinethehandgesture.Thelittlecontoursarelikelytobenoiseandshouldbeignored.Theassumptionwasmadethatthe handcontouristhelargestcontourintheimage,therebyignoringallnoisecontours.Ifthefaceshapeislargerthanthehand contour,thisassumptionmaybefalse.Toaddressthisflaw,thefacesectionoftheframeshouldberemoved.Theassumption wasthatthehandwastheonlymovingiteminthephotograph,andthatthefacewasrelativelymotionlessincomparisontothe hand.Thismeansthatbackgroundsubtractioncanbeusedtoremoveridoftheimage'sstationarypixels,aswellastheface region.ThisisfrequentlydoneusingtheOpenCVfunction"BackgroundSubtractorMOG2."

b. Hand Tracking

Handgesturesareusedtocontrolthepointer'smovement.Thecenterofthepalmshouldbediscoveredinitiallyinorderto locatethehand.Theapproachfordeterminingthehandcenterformhasthebenefitofbeingsimpleandstraightforwardto perform.Theshortestdistancebetweeneachpointinsidetheinscribedcircleandthecontourwasmeasured,withthepoint withthegreatestdistancebeingrecordedasthecenter.Theradiusofthehandwascalculatedasthedistancebetweenthe centerofthehandandthehandcontour.Thehandcenterwasdeterminedforeachsuccessiveframe,andthetipofthefinger wasknownandusedforhandtrackingutilizingthehandcenter.

c. Gesture Recognition

Theproposeddesign'sgesturerecognitiontechnologyisahybridoftwotechnologiesproposedbyYeoandBalazs.Thehand contour'sconvexityfaultsmustfirstbecomputed.Thehandcontour'sconvexitydefectsweredeterminedusingtheOpenCV built-infunction"cvConvexityDefects."Theconvexitydefect'sparameters(startpoint,endpoint,anddepthpoint)arestoredin aseriesofarrays.

d. Cursor Control

Oncethehandgestureshavebeenidentified,mappingcompletelyseparatehandgesturestospecificmouseoperationswillbea breeze.

6. RELEVANT MATHEMATICAL MODEL ASSOCIATED WITH THE PROJECT

.

Figure 6.1 Mathematical model

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume: 10 Issue: 01 | Jan 2023 www.irjet.net p-ISSN:2395-0072

Thepreprocesseddatagatheredfromcomputerperipherals(i.e.camera,sensors,etc.)isshownintheabovediagramgoing throughtheTensorFlowalgorithm.Itdividestheworkloadamongstthecomputerprocessors.Oncealloftheprocesseshave beencompleted,eachprocessortransferstheprocesstothedevelopmentmodel,whichthenshowstheresultaccordingtothe variousservices.

7. CONCLUSION

Thisprojectwilldoawaywiththerequirementforamouseoranyotherphysicaldevicetocontrolcursormovement.Forthe implementationofourproposedtask,wewilluseOpenCVforobjectdetection.Themousecanbemovedwithagreatlevelof precisionandaccuracy.BetterHuman-ComputerInteraction(HCI)isachieved.ItisalsousefulinAugmentedReality,current gaming,andcomputergraphics.Inthecontextofcomputergraphicsandgaming,thistechnologyhasbeenimplementedin moderngamingconsolestoproduceinteractivegamesthatdetectandinterpretaperson'smovementsasorders.Themajor goalistoeliminatetheusageofanyphysicaldevice,suchasamouse,andinsteadrelyonawebcam,whichisreadilyavailable withthelaptop.Althoughthisinitiativeoffersnumerous benefits,italsohassignificantdisadvantages.Ifthebackground pictureclasheswiththegivenimage,itmayofferanincorrectresultandmaynotperformproperly.Asaresult,it'sbesttouse this technique when the background light and the colour of the object don't combine. On PCs with poor resolution and computationalcapability,thesystemmayrunslower.Ifthecamerahasahighresolution,thesystemmaybeslow,butthis problemcanberectifiedbyloweringtheimageresolution.

8. REFERENCES

[1]A.Erdem,E.Yardimci,Y.Atalay,V.Cetin,A.E,“ComputerVisionBasedMouse”,Acoustics,Speech,andSignalProcessing, Proceedings.(ICASS).IEEEInternationalConference,2002.

[2] K. Pendke, P.Khuje, S. Narnaware, S. Thool, S. Nimje, “Computer Cursor Control Mechanism by Using Hand Gesture Recognition”,InternationalJournalofComputerScienceandMobileComputing,Vol.4Issue.3,March-2015,pg.293-300.

[3] A Banerjee, A. Ghosh, K. Bharadwaj, H. Saikia, “Mouse Control Using a Web Camera Based on Colour Detection”, InternationalJournalofComputerTrendsandTechnology(IJCTT)–volume9number1–Mar2014.

[4]Yadav,A.Pandey,A.Singh,A.Kumar,“ComputerMouseImplementationUsingObjectDetectionandImageProcessing”, InternationalJournalofComputerApplications.(IJCA),Volume69-Number21,2013.

[5]D.Verma,etal.,“VisionBasedComputerMouseControllingUsingHandGestures”,InternationalJournalofEngineering Sciences&ResearchTechnology.(IJESRT),7(6):June,2018.

[6]HojoonPark,“AMethodforControllingtheMouseMovementusingaRealTimeCamera”,BrownUniversity,Providence,RI, USA,Departmentofcomputerscience,2013.

[7]Chu-FengLien,“PortableVision-BasedHCI–ARealtimeHandMouseSystemonHandheldDevices”,NationalTaiwan University,ComputerScienceandInformationEngineeringDepartment.

[8]OnkarYadav,SagarMakwana,PandhariYadav,Prof.LeenaRaut,“InternationalJournalofEngineeringSciences&Research Technology(IJESRT)”,2017.