Virtual Mouse Using Hand Gesture Recognition by IRJET Journal

Virtual Mouse Using Hand Gesture Recognition

The Mouse is a great invention of technology. In recent years, different types of the mouse have been invented. There are many types of mouse available Optical Mouse, Wireless Mouse, and Bluetooth Mouse. However, hardwaredevicesareusedinthismouse,andalsoexpensivebecausesomemousetheyusedsensors.Thisproposedsystem isbasedonthelatesttechnologywhichusedhandgestures.Handgesturesarecapturedwiththehelpofacameraandafter that hand landmark key pointsaredetected andaccording toa particular gesture is recognized and it performs various operationsofthemousecursor.Userscanperformvariousoperationsofthemousewithoutusinganyhardwaredevicesor sensors only required your hand with the help of your fingers you can perform mouse operations. and it is user friendly also,cost effective.

Keywords: Hand gesture, hand landmark key points, user friendly. Cost effective

INTRODUCTION

a virtual mouse is a device used for to perform the operations of mouse cursor , using hand gesture recognition. Hand gestures are the most effortless and natural way ofcommunication. performthe various operation of the mouse cursor. Instead of using more expensive sensors, a simple web camera can identify the hand gesture and perform the action. It helps the user to interact with a computer without any physical hardware device to control mouse operation. The Convolutional Neural Network is used in this model to recognize various features of an image or video frame. It accepts video data as input, and according to the gesture recognition performs the particular task of the mouse cursor. The utilizationofvirtualmouseappearsinspace savingsituationsormovementsituations.

2. Literature Survey

Sr.no Researchpaper Technology Advantage

1 VirtualMouseControl UsingColoredFingertips andHandGesture Recognition11 12Sep 2020

V.ReddyT.Dhyanchan d,G.Krishn,S. Maheshwaram

2 VirtualMouse ImplementationUsing Opencv,2019SaiVarun. I Puneeth, Dr.T.Prem Jacobi

3 HandGesture

VirtualMouseFor Human Computer Interaction ,2018 S.Mohammed ,V.HPreetha

HandGesture Recognition,IP, NeuralNetwork, Python,OpenCV, pyautogui

usedneuralnetwork forhandgesture recognition.

Limitations

LimitationofThis Mousetheyused ColoredFingertips

Opencv,IP, DeepLearning, Pyautogui,NumPy, Anaconda

usedpythonforhand gesturerecognition.

Matlabs/w,Two cameras, Get90%Correct Detectionforboth views,under enoughLight Condition

Inthis,theyused colorvariation techniques ,it'sverycomplexto use

usedtwocameras that’swhyitsexpansive andcostly.

International Research Journal of Engineering and Technology (IRJET) e ISSN:2395 0056 Volume: 09 Issue: 04 | Apr 2022 www.irjet.net p ISSN:2395 0072 © 2022, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page3452

Mukti Kalsekar1, Shubhangi Shirke1, Kumud Wasnik2 1Students of Department of Computer Science and Technology, 2Professor.Kumud Wasnik, 1,2 Usha Mittal Institute of Technology, SNDT Women’s University Mumbai *** Abstract

4 VirtualMouse12th March

A.mhetar,B.KSrioop, KavyaAGS,Ramanath Nayak,RaviKumar

IR Camera,USB HID,IR Pen,Teensy(ARMM4)

Mouse

10thJuly

M.Shetty,C.Daniel, M.BhatkarOfrinLopes

6 Designand Developmentof

K.Shibly,SamratDey, A.Islam,S.Showrav

3. Existing System

HSV Technique,Python, Opencv,Ip

itsworksasa virtualmarker,it's effectivetouse whileteaching

Accuracyofthis SystemisinPlain Background:95%

TheyusedHardware devices,andtheCostof thisisMore,

Non PlainBackground: 40%

HCI Technology useofthelatest technology.

InthisSystembarrier isthelightning Condition that’swhysystemstill notenoughtoreplace themouse

It consists of the simple mouse operations Using the colored fingertips. This system divided into two methods finger detectionusingcoloredfingertipsandgesturerecognition,Asshowninbelowfigure.

It used video interface processing the image by background substraction. They used colored finger caps for finger detection, after that color identification will be done then circle identification will be formed on number of fingers.it will recognizethegestureaccordingtoitwillperformvariousoperationsofthemouse.

2015.

Virtual

UsingObject Tracking

2020

Hand GesturebasedVirtual Mouse

Dec2019

To design a virtual

weusedthewebcameraofourdevice,

instead of using more expansive extra

thelivefeedthroughthewebcamera.

Flow of the virtual mouse system is firstly video interface will start using the web camera. Then it recognize the hand gesturesandcontrolthemouseinteractionthenperformthevariousoperations ofthemousecursor.

Thisvirtualmouseisaportable,userfriendly,andcostisless.

mouse without any hardware devices and colored fingertips.

camera,

forcapturing

Fig2:Proposedsystemofvirtualmouseusinghandgesturerecognition 4. Proposed system

© 2022, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page3456 Fig 4:BlockDiagramofVirtualMouseusinghandgesturerecognition 4. Methodology steps : 1. Itwilldetectthecamera,videointerfacewillbestarted. 2. Thecameracanextractandrecognizehumanhandgesturesfromavideointerface. 3. Hand tracking functionality is done by usingmediapipe. 4. Aftertherecognitionthecursormovesaccordingly,toperformvariousoperations. Fig4.1HandKeyPoints[7] 4.1 Classification Model Fig 4.1.1:ConvolutionNeuralNetwork[8] • TorecognizevariousfeaturesofanimageorvideoframeCNNmakesuseoflayers. • Cnnusedlayerstodetectthefeaturesoftheinputimage. • Andapplyconvolutiontoeachfeaturedetection,andgettherecognizedoutputimage. International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056 Volume: 09 Issue: 04 | Apr 2022 www.irjet.net p-ISSN:2395-0072

e-ISSN:2395-0056

4.2 Libraries are Used :

1. Mediapipe:HandTrackingfunctionalityisdonebyusingamediapipe.

2. Numpy:Itprovidesamultidimensionalarrayobjectandtoolsforworkingwiththesearrays

3. OpenCV:thisisusedmainlyforreal timecomputervision.

4. Pycaw:itisusedforaudioutilities.

5. Pyautogui:Pyautoguikeepsthepythonscriptsandcontrolsthemouseandkeyboardtoautomateinteractionswith otherapplications.

6. ScreenBrightnessControl:itisusedocontrolsystembrightnessofyoursystem.

5. Implementation

We used one hand fingers to do this mouse operation. Hand gestures are input data and performing various mouse operationsaccordingtothehandgestureareoutputdata.handlandmarkdetectionisdonebymediapipe.Usingpyautogui weperformthemouseoperationslikeRightClick,LeftClick,DoubleClick,Move,ScrollUp,ScrollDown. thecameracapturedthelivefeeddatausingopencvanditsinputdata.weusedpyautoguiwhichallowsthemousecursor to move and controls the mouse operation interaction with hardware. Pycaw is used for audio utilities. We performed volumecontrolforthatweusedthePycawlibrary.toperformvolumecontrolwecalculatethedistancebetweentheindex tipandthumbtip.Forsystembrightnesscontrolweusedthescreenbrightness controllibrary.Inthisoperation,wetrack allpointsonthehandanddetectshandlandmark,calculatingthedistancebetweentheindextipandmiddlefingertipwith brightnessrange.

Wedeclarevariablesinthatwecanmaketheconnectionbetweeneachfinger.Wecreatedthefunctionofhandlandmarks andotherfunctionsforfingers.ifthefingerisupthenit's1,ifthefingerisdownthenit's0.Usingthiswegiveconditionsto performmouseoperations.Weuseddifferentcombinationsoffingerstoperformmouseoperations.

Fingersuseformouseoperationsare:

1. HandLandmarkdetection:Itdetectsthehandwith21handkeypoints.Itprintswhichfingersareup(1)anddown(0)

2. Move:performmoveoperationusingIndexFinger.

3. LeftClick:performleft clickoperationusingThumbFinger.

4. RightClick:perform right clickoperationusingMiddleFinger.

5. DoubleClick:performadoubleclickoperationusingthreefingers,fingersareindexfinger,middlefinger,ringfinger.

6. Volumecontrol:performvolumecontrolusingthethumbandindexfinger.

Scrollup:performscrolloperationusingtheindexfinger,middlefinger,littlefinger,ringfinger.

Scrolldown:performascrolldownoperationusingallfingers.

Brightness

adjust

9. Screen

Control:Weusedtheindexandmiddlefingersto

thesystembrightness 5.1 Result and Outputs: Fig 5.1.1:HandLandmarkDetection(Performingwhichfingersareupanddown) International Research Journal of Engineering and Technology (IRJET)

Volume: 09 Issue: 04 | Apr 2022 www.irjet.net p-ISSN:2395-0072

© 2022, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page3458 Fig 5.1.3:RightClick(Right clickperformedbymiddlefinger) Fig 5.1.4:LeftClick(LeftClickperformedbythumbfinger) Fig 5.1.5DoubleClick(Doubleclickperformedbyindex,middleandringfingers) International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056 Volume: 09 Issue: 04 | Apr 2022 www.irjet.net p-ISSN:2395-0072 Fig-5.1.2:CursorMove(Movingthecursorusingindexfinger)

© 2022, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page3459 Fig 5.1.6:ScrollDown(scrolldownimplementedbythethumbtothelittlefinger) Fig 5.1.7:ScrollUp(Scrollupperformedbytheindexfinger,middlefinger,littlefinger,ringfinger) Fig 5.1.8BrightnessControl(SystemBrightnessControldonebyindexandmiddle Fig 5.1.9VolumeControl(volumecontroldonebythumbfingerandindexfinger) International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056 Volume: 09 Issue: 04 | Apr 2022 www.irjet.net p-ISSN:2395-0072

e-ISSN:

6. CONCLUSION

TheVirtualMouse

ofdeveloping

anyother

7. Acknowledgement

8. REFERENCES

the

real

Thissystemiscost effectiveandeasytouse.

functionsbyusinghandgesturesinsteadof

canbeusedtoreducethespreadof COVID

usingthetraditionalphysicalmouse.wecan

anyhardwaredevices.Thisvirtualmousesystem

thecourseofthis

1. Vantukul Reddy , Thumma Dhyanchand, Galla Vamsi Krishna , Satish Maheshwaram “Virtual Mouse Control using ColoredFingertips

Hand

2. KolliparaSaiVarun,IPuneeth,Dr.T.Prem

”VirtualMouse

”,11 12Sep

UsingOpenCv”,

3. SherinMohommed,VHPreetha“HandGesture

MouseForHumanComputerInteraction”,2018

Ashish Mhetar , B K Srioop, Kavya AGS, Ramanath Nayak, Ravikumar Javali, Suma K V “Virtual Mouse” 12th March

MonaliShetty,ChristinaDaniel,ManthanBhatkar,OfrinLopes”

MouseUsingObjectTracking”,10th July2020

systemdevelopmentisuser friendlyandusedin

time.

Themainobjective

avirtualmouseistocontrol

mousecursor

using

physicalmouseand

hardwaredevices. Thevirtualmousewillbeusedinreal worldapplications,andalso,it

19.sincethemousesystemcanbeusedvirtuallyusinghandgestureswithout

concludethatthevirtualmousesystemhasperformedverywellwithout

overcomemostofthelimitationsoftheexistingsystems

WeWouldliketoshowourgratitudetotheprofessorforsharingtheirpearlsofwisdomwith

during

research,andwethankherforinsightsandforhercommentsthatgreatlyimprovedthereport.

and

GesturesRecognition

2020

Jacobi

Implementation

2019

Virtual

2015 5.

Virtual

6. KabidShibly,SamratDey,AminulIslam, ShahriarShowrav”DesignandDevelopmentofHandGesturebasedVirtualMouse”,

Dec2019 7. https://google.github.io/mediapipe/solutions/hands 8. https://www.analyticsvidhya.com/blog/2021/05/convolutional neural networks cnn/ International Research Journal of Engineering and Technology (IRJET)

2395-0056 Volume: 09 Issue: 04 | Apr 2022 www.irjet.net p-ISSN:2395-0072