International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056 Volume: 08 Issue: 12 | Dec 2021 www.irjet.net p ISSN: 2395 0072 © 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page1 An Extensive Survey on 3D Face Reconstruction Based on Passive Method Manasa Sandeep1, C. Nandini2 1Research Scholar, Assistant Professor, Department of Computer Science and Engineering, Dayananda Sagar Academy of Technology and Management, Bengaluru, Visvesvaraya Technological University, Belgaum, Karnataka, India 590018. 2Professor and Head, Department of Computer Science and Engineering, Dayananda Sagar Academy of Technology and Management, Bengaluru, Karnataka, India 560082 *** Abstract - 3D Reconstruction of a face image is gaining more attention from the last few years due to its vast applications. 3D reconstruction of an image provides a realistic image for analysis. Facial Image Reconstruction of a 2D image gives details on face geometry, texture, depth, color etc. which are used in diverse industries ranging from simple gaming and animation applications to complex medical imaging and biometrics applications. This work attempts to compare the work done in recent years based on passive methods. This paper concentrates majorly on the models that use 3DMM, SFS, SfM, deep learning approaches or the combination of these methods. The paper also discusses some of the available 3D face databases. Key Words: 3D face Reconstruction, Passive Methods, 3DMM,DeepLearning 1.INTRODUCTION whereInteractionimaging,forensics,applicationsworkproblems.alsoadditionemotions,ascomputers.technologyandpresenceanyprovidesFaceisaveryimportantpartofthehumanbodythatanindividualidentity.Humanbeingscanperceiveobjectintheirsurroundingsasa3Dmodelduetotheofacomplexopticnervesystem.ComputerVisionMachineLearningfieldhasadvancedsomuchinthatwecanrepresentanimagein3DinButfacialImageanalysisfacesmanychallengesthehumanfacekeepschangingwithexpressions,age,facialhairandotherexternalfactors.Intothesechallenges,3Dfacereconstructionshoulddealwith,pose,illuminationandocclusionrelatedThisgivesawidescopefortheresearcherstointhisfield.Thereconstructed3DfacesfinditsinabroadrangeofapplicationsincludingVirtualreality,animation,movies,medicaltelecommunication,biometrics,HumanComputeretc.TableIdetailsthedifferentapplicationareas3Dreconstructedfacesareused . 1.1 Applications Field PossibleApplications Advantages csForensi ➢ 3D Modeling of suspects or culprits facesfrom ○ 2Dimages ○ Sketchesavailable ○ Imagesobtained from footageCCTV ➢ Faces reconstructed deadfromcranialremainsofbodies ProvidesindividualidentificationofapersonandhelpsinsolvingCrime icsBiometr To provide Access control for ➢ Homes and Offices ➢ Mobiles and Laptops ProvidesSecurity Medical Used in Plastic Surgery planning o orSurgeryReconstructiveafterburnanaccident o Cosmetic Surgery to change facial features Helps doctors and Patients in outcomeofunderstandingbetterprovidingthesurgery nmentEntertai ➢ Interactive Virtual RealitySystems ➢ Animation in Gaming andMovies expression,humanIdentifiesand actions Table 1: 3DReconstructedFacesApplicationAreas
Reconstruction Oncethefaceisidentifiedandlocalizedinthegiven inputimage,nextstepistoreconstructthefacebyusing3D facereconstructionapproach.Severalsuchapproachesexist. 3.2 Challenges Involved Severalchallengesareinvolvedinreconstructingan imagefrom2Dinputimage.Thechallengesareduetomany reasons. Thefollowingare few reasonsthat make3D face reconstructionadifficulttask. ChangeinFacialExpressionofanindividual Problemsarisesduetochangeinposeoftheneck, faceandhead Lighting Problem,Noiseand other environmental factorsthataffectstheinputimage OcclusionProblem whenpartofafaceiscovered fromsomeobstacles 3.3 General Approaches There are different approaches available for 3D reconstruction of face. By referring to the literature, this papercategorizesthemintofourtypesnamely: The 3 D Morphable Model Approach which is popularlyknownas3DMM ReconstructionfromShading ReconstructionfromVideo DeepLearningMethodsknownasDLMethods
in 3D bythereconstructionasthedepthfeaturegivestherealisticlooktoreconstructedimage.Depthestimationcanbeperformedbuildingthedepthmap.
International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056 Volume: 08 Issue: 12 | Dec 2021 www.irjet.net p ISSN: 2395 0072 © 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page2 2. 3D FACE RECONSTRUCTION Existing Methods Reconstruction.MethodActiveReconstructionMethodandPassiveReconstructionarethetwomethodsavailablefor3DFace 2.1 Active Method Active method involves projection of electromagnetic waves like X rays, laser, visible light, ultrasound or andexample.environment.constructednoise,tobasedreacheswavesmicrowavesontheobjecttobereconstructed.Theprojectedarereflectedbackbytheseobjects.Thereflectedwavebacktothesourceandthe3Dmodelisconstructedonphasechangeandtimetakenbythereflectedwavereachthesourceback.Externalfactorssuchasscattering,absorptionbythemediumetc.mayaffectthemodel.HenceitrequiresacontrolledCyberware3DlaserScannerisonesuchRGBDepthcameraslikeKinectareeasiertousecheaperbutlackinqualityofthescan. 2.2 Passive Method asThecapturedobstructingradiation.expensiveThoughActivemethodsgivehighqualityscans,theyneedmachineryandtheobjectsareexposedtoPassivemethodsconstructthe3Dmodelwithoutdirectlywiththetargetobjects.Ituses2Dimagesfromdigitalcamerastoestimatethefacegeometry.algorithmsusedwillsingleimageorsequenceofimagesinput. 3. PASSIVE METHOD BASED 3 DIMENSIONAL FACE RECONSTRUCTION3.1GeneralSteps in Facial Reconstruction Figure 1 represents the general steps of the 3D face inreconstructionandthenextsectionexplainseachofthestepsdetail Fig- 1: Generalstepsofthe3Dface 3.1.1. Input Image 2DImageistakenasinput.InputImagecanSingle 2DImageormorethanoneimagelikefrontalandsideview etc.orasequenceofImagestakenfromvideolikeCCTVetc. basedontheapproachandthealgorithmused. 3.1.2 Preprocessing Once the input image is obtained, it is always preprocessed.Presenceofnoise,shadows,Occlusion,poor iscontraststeplightingmayaffectaccuracyofthereconstructedimage.Thisremovesanynoisepresentintheinputimage,performsandintensityadjustment,repairsiftheinputimagecorruptedfromshadow,poorlightingorocclusion. 3.1.3 Face Localization The process of identifying the face in the given pictureiscalledfacelocalization.Thisinvolvesremovalof nosebackgroundandidentifyingthefacialpartslikeeyes,mouth,usingdifferentfacealignmentalgorithms.
3.1.4 3D Face
3.1.4 plays a major role
Depth Estimation Depth finding
International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056 Volume: 08 Issue: 12 | Dec 2021 www.irjet.net p ISSN: 2395 0072 © 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page3 4. THE 3 D MORPHABLE MODEL APPROACH Thismethodusesageneric3Dfacemodel.Itfitsthe imageproblem.shaperepresentis3DMMproposed(BFM)genericasface.giveninputimagetothatgenericmodelandreconstructstheThisisapassivemethodthattakesoneormoreimagesinput.Thefacemodelusedcouldbe3DMMoranyothermodel.FewotherfacialmodelsareBaselFaceModel[1],SurreyFaceModel(SFM)[2]etc.BlanzandVetter3DMorphableModelwhichiscommonlyknownasisthewidelyusedmodel[3].Theideabehind3DMMthatthelinearsumofbasicfunctionscanbeusedtothe3Dfacialshape.ItallowsreconstructingtheandalbedobysolvinganonlinearoptimizationThisapproachgivesthebestresultswhentheinputissimilartothatofthegeneralimageused. 5. RECONSTRUCTION FROM SHADING Basically, Reconstruction from Shading is the processofusingoneinputimageoftheobjecttocompute improvescombiningeyesareaswellsmoothLambertianK.reconstructSFSthe3Dshapeofthatobject.3Dfacereconstructionusingtheapproachusesintensityvariationoftheimageinputtothe3Dface.ThisconceptwasinitiallygivenbyB.P.Hornin1989[4].Thismethodisbasedonthemodel.SFSobtainsgood3Dinformationfromsurfaces.Whenusedwithfaceimages,itworksveryinestimatingthedepthfeature,butfailsinhandlingthewherethetexturechangesarenotsmoothlikenose,etc.Researchershandlethisindifferentwayslikethisapproachwithotherapproaches.Thistheaccuracyoftheresultingmodeletc. 6. RECONSTRUCTION FROM VIDEO In this method, continuous frames from a single source like video input is considered as the input. 3D thethetrackedImage.theestiobtainedreconstructionoffaceisdonebyusingthisinput.Theimagesbytheinputvideosequenceareanalyzedtomatethe3Dfaceoutput.ThecorrespondencebetweenavailabledifferentinputimagesisusedtofindtheoutputFeaturesfromthefaceareselectedandtheyarefromoneimagetothenext,whichhelpsinfindingcorrespondencethatexistsbetweenthemandconstructs3Dface. 7. DEEP LEARNING METHODS ThismethodfollowstheMachinelearningapproach Reconstructionphasetypemethodthesystemweightsthetrainingreconstructiononeandthesolutionisnormallydevelopedintwophases,phaseisthelearningphaseandthesecondbeingthephase.Themodelisrigorouslytrainedinthephase.Thepriorinformationknownisusedtotrainmodel.Priorknowledgeisencodedintheformofinthetrainednetwork.Usingthistraining,thelearnstofindthemapping.Itlearnstomapbetween2Dinputimageandthe3Dface.Thesuccessofthisdependsonthenumberoftrainingdatasetused,ofthenetworkused,numberoflayersusedintrainingetc.Table2comparestheworkdoneon3Dfaceinrecentyearsbasedonpassivemethods NoSl. ermbNunceereRef Year UsedchApproa No. deredConsisImageInputof d)ontrolle(Wild/CmentEnvironImageInput Methodology Used GapResearch AccuracyRecognition 1 [5] 3200 3DMM Single meEnvironedControllnt Computer Graphics simulationiscombinedwith Deformable 3D model. Projection and illumination areused.Shapeandtexture are used as intrinsic parameters. Both Intrinsic independentandextrinsicparametersareofeachother. dependentHighly on the database used Hit CMURatioPIEdatabase: FERET77.5%database: False87.9%AlarmRsate:1% 2 [6] 7201 VideofromructionReconst Single wild Structure from motion is used which is named as “Priorconstrainedstructure from motion (PCSfM)”. It videoofonConcentratessequenceframes/input. 3DRootMeanSquare WithError:outconsidering poseandExpression:
International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056 Volume: 08 Issue: 12 | Dec 2021 www.irjet.net p ISSN: 2395 0072 © 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page4 usesthe3DMMapproachto find the prior knowledge of shapes. Video input is used tofindthelandmarksofthe input face and fitting the prior knowledgetoestimate thefaceshapes By2.09consideringpose andExpression:1.92 3 [7] 8201 Shading+3DMM ImageSingle wild Theapproachusesdifferent steps, first step gives a 3D facewhichissmooth.Thisis generated using “Example based bilinear face model”. Then medium face shape is method.byrecoveredgeometricconstraints.photometricfaceobtainedbyrefiningbilinearmodelusingconsistencyFinally,finedetailsarefrommediumfaceusingshapefromshading Gives good result only when coarse face model is similar to the ground truth shapeoverall 3DRootMeanSquare Error: Disgust:SurpriseHappy:Expression1:71±0:34:2:05±0:491:98±0:42 4 [8] 9201 gLearninDeep Single Wild A MobileNet based fusion network is proposed. This method has encoder that estimates the following parametersofafacemodel: Pose Identity coefficientsExpression Coefficients of a shape correction model Next, the final output face shape is given by the decodermodule. The resultssatisfactorygive.reconstructifinedatabasdifferentlandmwithnotalgorithmproposedistestedthearksofesanddetailfaceonItdoesn'twhen we don’t fix the position ontheface Average Normalized mean error (NME) 3.37correctionused:whencorrectionisnot3.41Whenisused: 5 [9] 9201 gLearnin+3DMMDeep ImageSingle Wild 3DMM fitting approach is identicaldeepusedincombinationwiththefeaturesthatarebytrainingaGAN “Generative Adversarial Network”onalargescale3D texturedataset strongfacialocclusionsSelfpresencethereconstructsuccessfullyThoughsfaceinofselfandhairand ShapeEstimation MeanandStandard Condition:CooperativeGANFit++Deviation0.94± Indoor0.17Condition: 0.92±0.14
International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056 Volume: 08 Issue: 12 | Dec 2021 www.irjet.net p ISSN: 2395 0072 © 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page5 absoluteimprovingscopethereilluminations,isaforerror OutdoorCondition: 0.94±0.19 6 ][10 9201 gLearninDeep sImageple/MultiSingle Wild The proposed architecture reconstructs the 3D face based on siamese neural network using one or multipleimages inputfrominformation3DmergingextractionefficientlyperforcapableItarchitecture.dedicatedGivesisnotofmingtheandofthemultipleviews Average Root Mean Square Error of the twomodelsproposed MVAddModule:2.43 MV Concat Module: 2.33 7 ][11 0202 3DMM sImageleMultip isfunctionobonCorrelatiGradientWildjectiveused Landmark Features are estimated. Setting up of Interval for basis coefficients. Sparse landmarkdetectorsareused The efficiency of the “Inequality minimizationconstrained algorithms” is notevaluated Normalized mean error for Synthesized dataset for 68 SingleLandmarksframe:0.036 Multi frame:0.032 8 ][12 0202 Shading Single Wild Basel face model is used. Facialfeaturesextractedare subjected to scale invariant feature transform. Multivariate Gaussian distribution is used to calculatedepth Tested only for Wild Home LFW database and it uses faces that are labelled. Can’t handle when an image input y.givenrandomlis RMSerrorfor Image1:0.8866 Image2:0.8689 Image3:1.6468 9 ][13 0202 gLearnin+3DMMDeep sImleMultipage Wild accordingencodedinputtwoSystemusesoneencoderanddecodernetworks.The2Dfaceimageisbyencodernetworkto Identity Expression Poseetc. The 3D face shape is calledThestagedependsfacereconstructedTheshapeon1output.stage1is “foundation Mean error Outdoor:Indoor:1.16±0.29Cooperative:overcomparisonofaveragethreeview:1.24±0.301.22±0.28
International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056 Volume: 08 Issue: 12 | Dec 2021 www.irjet.net p ISSN: 2395 0072 © 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page6 regressed by the first decoder network and the second regress the albedo. Thenthesystemrendersthe facial image back and matchestheinputimage. face shape”. This stage uses linear 3DMM which doesn’t handles the lightingsexpressions,occlusions,poses,diversityinputwhenwildconditionthecontainsofetc. 10 ][14 0202 gLearnin+3DMMDeep Single Wild 3DMM encoder which is coarseisusedforregressing the parameters of 3DMM, thenfaceoftheinputimage alignedspace.isandthetextureofthe3DMMunwrappedintotheUVHerethefacesareprecisely. includestheFailstohandleinputthat nexpressiolarge occlusion pose.large POINT TO PLANE IndoorERROR RMSE:Outdoor:1:35±0:31Cooperative:1:25±0:211:81±0:43 TABLE -2.Comparisonofworkdoneon3Dface Reconstruction 8. 3D FACE DATABASE databases.commercialFaceScape,likeincreasedMany3Dfacedatabasesareavailable.Theavailabilityhasalotduringlastfewyears.SomeofthedatabasesBFM,FLAMEetc.aremadepublicandfewotherslikeFlorenceetc.canbeaccessedonrequestfornonpurposes.TableIIIlistsafewavailable3Dface
[6] M.Hernandez,T.Hassner,J.Choi,G.Medioni,‘Accurate 3Dfacereconstructionviapriorconstrainedstructure frommotion’,ComputerGraphics.66pp.14 22,2017.
[5] V.BlanzandT.Vetter,‘Facerecognitionbasedonfitting a3Dmorphablemodel’inIEEETransactionsonPattern Analysis and Machine Intelligence, vol. 25, no. 9, pp. 1063 1074,Sept.2003.
reconstruction
[4] B. K. P. Horn, ‘Obtaining shape from shading information’inB.K.P.HornandM.J.Brooks,MITPress, ShapefromShading,pages121 171,1989.
International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056 Volume: 08 Issue: 12 | Dec 2021 www.irjet.net p ISSN: 2395 0072 © 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page7 TABLE 3 LISTOF3DFACEDATABASES 9. CONCLUSION AND DISCUSSIONS Thetimereconstructing(cooperative(neutral,ofisReconstruction3Dreconstructionofthefaceisachallengingtask.3Dfacefromoneinputimagetakeninwildconditionmorechallengingthanusingmultipleimagesasinput.Mosttheworkconcentratesontwotothreeexpressionshappy,sadetc.)andtwotothreeconditionsindoor,outdoorconditionetc.)whilethe3D.Alsoonlyfewworksconsidersthecomplexityandthroughputoftheproposedmethods.futureworkscanalsoconcentrateonimproving throughput, efficiency, time complexity. Through critical toreconstructimproveocclusion.thethisdiscussionontheresultsofthemanymodelsasdiscussedinpaper,stillthereisscopetoimprovetheefficiencywheninputscenarioscontainwideexpression,ethnicityandThereisneedforthemultimodalapproachtoonvariousparametersasdiscussedabovetotheimagestakeninamorechallengingsituationserveforrealtimeapplications REFERENCES [1] P.Paysan,R.Knothe,B.Amberg,S.Romdhani,T.Vetter, ‘A3Dfacemodelforposeandilluminationinvariantface recognition’,ProcofIEEEAVSS,pp.296 301, 2009.
fidelity NoSl. Database No sSubjectof ImageSource yearsCoverageAge Ethnicity Expression dlocalizerkLandma 1 eFaceWarehous[15] 150 Kinect 7to80 WideRange expressionneutraland19 otherexpressions Yes 2 FaceScape [16] 847 multi viewsystem 68 Cameras 16to68 20expressions Yes 3 BFM2009[17] 200 Two projectors and three cameras and SLRcamera 8to62 Europeans ExpressionsNeutral Yes 4 BFM2017[18] 200 Two projectors and three cameras and SLRcamera 16to70 UnionEuropean 6expression Yes 5 FLAME[19] 3800 multicamera active stereo system. It consistsof3pairsof Stereocameras Colorcameras projectorsSpeckle White light LED panels. widerange widerange expressionsIncludes Yes
[2] P.Huber,G.Hu,J.R.Tena,P.Mortazavian,P.Koppen,W. J.Christmas,M.Ratsch,J.Kittler,’Amultiresolution3D morphable face model and tting framework’, Proc. of VISAPP,pp.79 86,2016.
[7] L. Jiang, J. Zhang, B. Deng, H. Li, L. Liu, ‘3D face reconstruction with geometry details from a single image’,IEEETransactiononImageProcessing.274756 4770,2018. [8] P.Wang,Y.Tian,W.Che,B.Xu,‘Efficientandaccurate face shape by fusion of multiple landmarkdatabases’,publishedinProc.ofICIP,pp.335 339,2019. [9] B. Gecer, S. Ploumpis, I. Kotsia, S. Zafeiriou, ‘GANFIT: Generativeadversarialnetworkfittingforhigh
[3] V.Blanz,T.Vetter,‘Amorphablemodelforthesynthesis of3Dfaces’,Proc.ofACMSIGGRAPH,pp 187 194,1999.
International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056 Volume: 08 Issue: 12 | Dec 2021 www.irjet.net p ISSN: 2395 0072
[10] E.Ramon, J.Escur, X.Giro i Nieto, ‘Multi view3D face reconstruction in the wild using siamese networks’, published in Proc. of IEEE ICCV Workshops, pp.3096 3100,2019.
[14] Y. Chen, F. Wu, Z. Wang, Y. Song, Y. Ling, L. Bao, ‘Selfsupervised learning of detailed 3D face 29reconstruction,IEEETransactionsonImageProcessing,pp.86968705,2020.
[15] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, Kun Zhou,’FaceWarehouse:A3DFacialExpressionDatabase for Visual Computing’, IEEE Transactions on Visualization and Computer Graphics, Volume: 20, Issue:3,2014.
[16] HaotianYang,HaoZhu,YanruWang,MingkaiHuang,Qiu Shen,RuigangYang,XunCao,‘FaceScape:aLarge scale HighQuality3DFaceDatasetandDetailedRiggable3D FacePrediction’,CVPR2020.
[17] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, Thomas Vetter, ‘A 3D Face Model for Pose andIlluminationInvariantFaceRecognition’,2009Sixth IEEEInternationalConferenceonAdvancedVideoand SignalBasedSurveillance,2009
[12] H.M.RehanAfzal,SuhuaiLuo,M.KamranAfzal,Gopal Chaudhary,ManjuKhari,SathishA.P.Kumar,‘3DFace ReconstructionFromSingle2DImageUsingDistinctive Features’, IEEE Access 8:180681 180689, October 2020
[19] T.Li,T.Bolkart,M.J.Black,H.Li,J.Romero,‘Learninga model of facial shape and expression from 4D scans’, ACMTransaction.Graphics.36,2017.
[13] Z. Gao, J. Zhang, Y. Guo, C. Ma, G. Zhai, X. Yang, ‘Semisupervised3Dfacerepresentationlearningfrom unconstrainedphotocollections’, Proceedingsof IEEE pp.348 349,IEEECVPRWorkshops,2020.
[18] T.Gerig,A.Morel Forster,C.Blumer,B.Egger,M.Luthi, S. Schoenborn, T. Vetter, ‘Morphable face models An openframework’,Proc.ofIEEEFG,pp.75 82,2018.
3D face reconstruction’, in: Proc. of IEEE CVPR, pp. 1155 1164,2019.
[11] E. Sariyanidi, C. J. Zampella, R. T. Schultz, B. Tunc, ‘Inequality constrained and robust 3D face model fitting’,in:Proc.ofECCV,2020.
© 2021, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page8