International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056
Volume: 11 Issue: 06 | Jun 2024 www.irjet.net p-ISSN:2395-0072
ADVANCEMENTS IN AUTOMATED VALIDATION TESTING FOR AUTONOMOUS VEHICLES
Sree Ramya Yendluri Zoox.Inc, USA
ABSTRACT:
Themarketforself-drivingcarshasgrowna lotin recentyears.By2026,theworldmarketis expectedto be worth $556.67 billion. It is hard to make sure that these cars are safe and reliable through validation testing because they are getting more complicated and AI and EV technologies are being added to them. The change from manual testing to automated validation testing is talked about in this piece, along with the pros, cons, and possible futures of this method. This article looks at case studies,researchresults,andindustryeffortstogiveafullpictureofwhereautomatedvalidationtestingisnowandwhere it's going in theautonomousvehicleindustry.The results makeitclear how importantitisfor people in business andpeoplein academiatoworktogethertodriveinnovationandstandardizationintestingandvalidationprocedures.
Keywords: AutomatedValidationTesting,AutonomousVehicles,Self-DrivingCarSafety,SimulationTools,Industry-Academia Collaboration
I. INTRODUCTION
The market for self-driving cars has grown by leaps and bounds in the past few years. It's expected to hit $556.67 billion by 2026[1].Thishugegrowthisduetomorepeoplewantingtransportationoptionsthataresafer,moreefficient,andbetterfor theearth.Therewere24.10billiondollarsworthofautonomousvehiclesonthemarketaroundtheworldin2020.Themarket is projected to grow at a rate of 18.06% per year over the next few years [2]. This fast growth is due to improvements in artificialintelligence(AI),sensortechnology,andhigh-performancecomputing,whichhavemadeitpossibleformorecomplex self-drivingcarsystemstobemade[3].
Asthesecarsgetmorecomplicated,itbecomesmoreandmoreimportanttohavefastandaccuratevalidationtesting.Alotof sensors,cameras,andsoftwaresystemsworktogethertohelpself-drivingcarsfindtheirwayandmakedecisionsinreal-time.
International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056
Volume: 11 Issue: 06 | Jun 2024 www.irjet.net p-ISSN:2395-0072
A study by the National Highway Traffic Safety Administration (NHTSA) says that self-driving cars could cut down on car accidents by up to 90% [4]. But making sure that these vehicles are safe and reliable takes a lot of testing and approval processes.AccidentsinvolvingTesla'sAutopilotsystemthatkilledpeopleshowhowasinglesoftwarebugorsensingproblem canhaveterribleresults[5].
AutomakersarehavinganevenhardertimemakingsuretheirgoodsaresafeandworkwellnowthatAIand EVtechnologies arebeingusedtogether[6].AIsystems,likedeeplearningandcomputervision,letself-drivingcarsseeandunderstandwhat's goingonaroundthem.Butthesealgorithmsneedtobetaughtandtestedalottomakesuretheycanhandlealotofdifferent drivingsituations.EV(electriccar)technology,ontheotherhand,hasitsownproblems,likehowtohandlebatteriesandplace chargers[7].Whenyouputthesetechnologiestogether,yougetacomplicatedsystemthatneedsalotoftestingandproofto makesureitworkssafelyandreliably.
TheWorldEconomicForumdidasurveyandfoundthat58%ofautomotiveexecutivesthinkthatmakingsurethatself-driving cars are safe and reliable is the biggest challenge to their growth [8]. This shows how important validation testing is in the businessofself-drivingcars.It'snolongerpossibletokeepupwiththefastprogressinautonomousvehicletechnologyusing the old-fashioned way of testing vehicles on public roads with human drivers. Automated validation testing has become a hopefulwaytodealwiththeseproblemsbecauseitallowstestingtobedonefaster,moreefficiently,andmorethoroughly.
Usingtheold-fashionedwayoftestingbyhandtakesalotoftimeandwork.Usually,ittakesseveralweekstofinishonetest run[9].AcasestudybyVolvoCarsshowedthatwith50engineersworkingaroundtheclock,themanualtestingprocessfora singletypeofcar tookanaverageof 8 weeks[10].Thisshowsthatmanual testing methodsaren't able tokeepup withhow quickly modern autonomous cars are being developed. Also, human limitations like tiredness, reaction time, and personal opinioncanmakephysicaltestinglessreliableandmorelikelytomakemistakes[11].
Fig.1:ProjectedExpansionoftheAutonomousVehicleIndustry[1–2]
II. THE AUTOMATION IMPERATIVE
International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056
Volume: 11 Issue: 06 | Jun 2024 www.irjet.net p-ISSN:2395-0072
It'sgettingharderandhardertotesttheperformanceofAI-integratedandelectriccarsusingtheseoldmethodsbecausethey have more complex features. Complex algorithms and machine learning models help self-driving cars understand and find their way around the world. It would take a huge amount of time and money to test these systems by hand. A report by McKinsey&Companysaysthatusingmanualtestingmethodstoprovethatanautonomouscarworkswouldmeandrivingit for over 8.8 billion miles, which is the same as driving it nonstop for 400 years [12]. Because of how quickly the industry is changingandhowquicklynewcarsneedtobeonthemarket,thisisnotpossible.
The National Highway Traffic Safety Administration (NHTSA) conducted a study that demonstrated how using automatic testing techniques could reduce the time required for validation by as much as 80% [13]. This research looked at how different automakers use human and automated testing methods and compared them. The results showed that automatic testingnotonlycutdownonthe timeneededforvalidation, butitalsomadethetestingprocessmoreaccurate andreliable. Automatedtestingsystemscanworknonstop,24hoursaday,sevendaysaweek,withnohelpfromaperson.Sincethisisthe case,alotmoretestingcanbedoneinlesstime,whichspeedsupthedevelopmentanduseofself-drivingcars.
Withautomated testing,a lotofdifferentdrivingsituationscanbesimulated, even onesthat would be hardor impossibleto testbyhand.Edgecasesare veryrarebutveryimportantsituationsthathappenveryrarelybutcanhaveverybadresultsif the autonomous driving system doesn't handle them correctly. Some examples are sudden obstacles in the road, extreme weather, and complicated traffic situations. In a study from the University of Michigan, it was found that automated testing couldcreateupto100,000newdrivingscenarioseveryday,whilehumans couldonlytryafewhundred[14].Withthismuch testing, possible problems can be found and fixed before they happen in real-life driving situations, which is important for makingsurethatself-drivingcarsaresafeandreliable.
Additionally, automated testing is what makes continuous integration and continuous deployment (CI/CD) techniques possible. This means that feedback loops are faster and software updates happen more often. CI/CD is a way of making softwarethatautomaticallybuilds,tests,anddeployschangestocode.Thisletsyoumakechangesmorequicklyandwithmore confidence.Becauseithasadvancedautomatedtestingtools[15],Tesla,forexample,hasbeenabletosendweeklyover-theairsoftwareupdatestoitscars.ThishasmadeitpossibleforTeslatokeepimprovingthespeedandfeaturesofitscarswhile alsoquicklyfixinganysafetyissuesthatmightcomeup.
Automatedtestinghasmanyperksthatgobeyondjustmakingsurethatself-drivingcarswork.Otherimportantpartsofselfdriving cars, like sensors, hardware, and communication systems, can also be tested automatically. For instance, automated testing can be used to make sure that LiDAR sensors work well in different weather situations, like rain, fog, and snow [16]. This makes sure that the sensors can find and follow objects around the car accurately, even when conditions are bad. Similarly, automatic testing can be used to make sure that vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communicationsystemsworkproperly.Thesesystemsarenecessaryforlettingdriversworktogether,whichimprovessafety andtrafficflowoverall[17].
Table1:ComparativeAnalysisofManualandAutomatedTestingMethodsinAutonomousVehicleValidation[9–17]
III. OVERCOMING HURDLES WITH INNOVATION
There have been some problems with the switch to automatic testing. The variable data that self-driving cars produce has beenoneofthemainproblems,asitcanmakeithardtorepeattest scenarios[18].Thesensorsonautonomouscars,suchas
International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056
Volume: 11 Issue: 06 | Jun 2024 www.irjet.net p-ISSN:2395-0072
cameras, LiDAR, and radar, gather a huge amount of data. This information can be very different based on things like the weather,thetypeofroad,andhowpeopleuseit.Theinformationthatself-drivingcarsgivecanbeoffbyupto20%between testruns,evenifthesamerouteistaken[19].ExpertsattheUniversityofTorontofoundthis.Becauseofthis,itcanbe hardto make testing scenarios that are reliable and consistent since the data used to train and confirm the autonomous driving systemmightnotbetypicalofdrivingconditionsintherealworld.
Tosolve thisproblem, businesseshavecreatedhigh-techsimulationtoolsthatcan createaccuratevirtual worldsfortesting. Developers can test their cars in a lot of different situations with these tools because they let them make very detailed and flexibletestsetups.Forinstance,NVIDIA'sDRIVESimtoolcanmakea digital copyofa real-worldscene,whichletsdifferent driving situations be simulated [20]. This platform can create accurate sensor data, such as camera images, LiDAR point clouds, and radar signatures. This lets developers test their decision-making and perception algorithms in a safe and controlled setting. Companies can make a lot of fake data with modeling tools that can be used to train and test their selfdrivingsystems,sotheydon'thavetorelyondatafromtherealworld.
Getting the right tools and infrastructure has been another problem with the switch to automated testing. To handle and analyze the huge amounts of data that self-driving cars produce, automated testing needs powerful computers and fast communication networks. To deal with this problem, businesses have spent money to build testing labs and cloud-based systems.Waymo,forexample,hasbuiltatestingcenterinCaliforniathatcoversmorethan100acresandhasdifferenttypes of roads and obstacles [21]. Waymo can test its cars in a controlled setting and gather useful information for teaching its machine-learning models at this facility. In the same way, companies like Tesla and Nvidia have spent money to build data centersandgroupsofhigh-performancecomputerstohelpwiththeirautomatedtesting[22].
Companieshavedevelopedadvancedsoftwaretoolstospeedupthetestingprocessinadditiontosimulationtoolsandtesting centers.Someofthesetoolsautomaticallycreatetestcases,runtests,andlookattheresults.Forinstance,Cruise,acompany owned by General Motors, has developed a tool called Octopus that can make test cases automatically based on a list of predefined situations [23]. Cruise has been able to test its self-driving cars with a lot less time and work thanks to this tool. Anotherexampleishowmachinelearningmethodsareusedtolookattestresultsandfindproblemsthatmightbehappening. Basedon pasttest data,researchersatMIThavecreateda machinelearningsystemthatcanguesshowlikely itisthata car will face a certain type of driving situation [24]. This lets developers focus their testing on the most important cases, which cutsdownonthetimeandmoneyneededfortestinggenerally.
It's very important for businesses and universities to work together to solve the problems that come up with automatic testing. Researchers from universities have a lot of useful knowledge in areas like computer vision, machine learning, and robots, which can be used to create new testing methods and algorithms. For instance, experts at Stanford University have come up with a new way to test self-driving cars that they call "adversarial testing" [40]. The idea behind this method is to create test scenarios that are meant to push the vehicle's perception and decision-making systems. This lets edge cases and possibleweaknessesbefound.Companiescangetaccesstocutting-edgeresearchandexpertknowledgethatcanhelpspeed upthedevelopmentanduseofself-drivingcarsbyworkingwithuniversityresearchers.
Challenge Solution Example Improvement
DataVariability Advanced Simulation Tools NVIDIADRIVESim 20% reduction in data variability
Infrastructure Requirements Dedicated Testing Facilities Waymo's California Facility 100 acres of controlled testingenvironment
Testing Process Efficiency Automated Software Tools Cruise'sOctopus 50%reductionintesting timeandeffort
Table2:InnovativeSolutionstoAutomatedTestingChallengesintheAutonomousVehicleIndustry[18–24,40]
International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056
Volume: 11 Issue: 06 | Jun 2024 www.irjet.net p-ISSN:2395-0072
IV. ADVANTAGES OF AUTOMATED TESTING
Themovetowardautomationhasbeengoodfortheautonomousvehiclebusinessinmanyways.Waymodidacasestudythat showedtheirautomatedtestingsystemcouldmodelmorethan10millionmilesofdrivingeveryday,whichisthesameas300 years of human driving experience [26]. The accuracy and consistency of car performance have gotten a lot better thanks to this level of testing. A human driver, on the other hand, would have to drive for over 1,500 years to get the same amount of experience[4].Thisreallyshowshowbigandusefulautomatictestingis;itcancutyearsoftestingdowntojustafewdaysor weeks.
Faster validation processes made possible by automated testing have also reduced the time it takes to get new cars on the market.Becauseithasadvancedautomatedtestingtools[28],Tesla,forexample,hasbeenabletosendsoftwareupdatestoits carsevery week.Thisisa bigimprovementover thetraditional autobusiness,wheresoftwareupdatesareusuallyonlysent out once or twice a year. McKinsey & Company did a study that showed automated testing could cut the time needed to validatesoftwarebyupto90%[29].Thiswouldallowcompaniestoreleasenewfeaturesandchangesmorequickly.Thisnot onlymakestheexperiencebetterforeveryone,butitalsoletscompaniesfixanybugsorsafetyissuesintheirsoftwarequickly.
Autonomous vehicle businesses have saved a lot of money by automating testing. This is because it speeds up the validation process.Inastudy,Accentureclaimedthatautomatedtestingcouldreducetestingcostsbyasmuchas80%whencomparedto manual testing [30]. This is because automatic testing doesn't need as many people and can be done 24 hours a day, seven days a week. By finding and fixing bugs early in the development process, automated testing can also help companies avoid expensive refunds and legal problems. A study by the NHTSA found that in the last ten years, recalls of cars because of software problems have grown by 30% [31]. These problems can be avoided with automated testing, which can save businessesmillionsofdollarsinpossiblerefundcostsandlegalfees.
Companies can also test their cars in more situations and odd cases thanks to automated testing. Edge cases are rare but importantsituationsthatarehardtotestbyhand,likeapersonsteppingoutintotheroadwithoutwarningoracarrunninga redlight.Companiescanmakesurethattheirvehiclescanhandlethesesituationssafelyandconsistentlybysimulatingthem inavirtualworld.AstudyfromtheUniversityofWaterloofoundthatautomatedtestingcouldcoverupto95%ofallpossible drivingsituations,whilehumantestingcouldonlycover60%[32].Becauseitletspossibleproblemsbefoundandfixedbefore theyhappeninreal-lifedrivingsituations, thiswide rangeoftesting is necessaryto makesurethat self-driving carsaresafe andreliable.
Lastly,automatictestinghasmadeitpossibleforbusinessestogatherhugeamountsofusefuldata thattheycanusetotrain their machine-learning models. Machine learning is an important part of systems that let cars drive themselves. It lets cars learnfromtheirmistakesandgetbetterovertime.Buttotrainthesemodels,youneedalotoftaggeddata,whichcanbehard togetandcostalotofmoneytodobyhand.Companiescan makefakedatathatcanbeusedtotraintheirmachine-learning modelsbycollectingdatafromsimulateddrivingevents.AstudyfromtheUniversityofCalifornia,Berkeley,foundthatusing fake data from automated testing to train machine-learning models could make them up to 20% more accurate than models learnedwithonlyreal-worlddata[33].Byprovidingalotoftrainingdataformachinelearningalgorithms,automatedtesting hastheabilitytospeedupthedevelopmentanddeploymentofself-drivingcars.
V. THE FUTURE OF VALIDATION TESTING
The autonomous vehicle business is always changing, and it's very important for people in the industry and people in academiatoworktogethertodriveinnovation.PartnershipforTransportationInnovationand Opportunity(PTIO)programs havebroughttogetherpeoplefromdifferentfieldstoworkonproblemsinthetransportationbusiness[34].ThePTIOismade upofsomeofthebiggestnamesintheself-drivingcarbusiness,likeWaymo,Uber,andtheAmericanAutomobileAssociation (AAA).Throughstudy,education,andpublicpolicyefforts,thepartnershipwantstoencouragethesafeandresponsibleuseof self-drivingcars.
Standardized testing and validation processesfor self-driving carsare oneofthe mainthingsthat the PTIO workson. Atthe moment, there isn't a single industry standard for testing and approving self-driving cars. This can cause problems with consistency and safety. The PTIO is working on making a set of standards and best practices for testing and validation that
International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056
Volume: 11 Issue: 06 | Jun 2024 www.irjet.net p-ISSN:2395-0072
companiesinallfieldscanuse.Onewaytodothisistocomeupwithastandardsetofmetricsandkeyperformanceindicators (KPIs)forcheckinghowsafeandreliableself-drivingcarsare[35].
Thecreationofadvancedsimulationtoolsandvirtualtestingsettingsisanotherimportantareawherepeopleworktogether. Aswealreadysaid,simulationtoolslikeNVIDIA'sDRIVESimplatformarenownecessarytotryandproveself-drivingcarsin a safe and controlled setting. But we still need more advanced and lifelike simulation tools that can perfectly copy driving situationsthathappeninreallife.UniversityofMichiganresearchersmadeasimulationplatformcalledmcity.Ithasa32-acre testsitewithdifferenttypesofroads,trafficlights,andobstacles[36].Theplatformalsohasacloud-basedsimulationsetting thatcancreatethousandsofreal-lifedrivingsituationsthatcanbeusedfortestingandapproval.
Thereisaneedforbothsimulationtoolsandmoreadvancedtestingcentersandinfrastructureintherealworld.Eventhough companieslikeWaymohavebuilttestingsites,thewholeindustrystilldoesn'thaveastandardwaytotestthings.TheNHTSA wantstocreatea nationwidesystemoftesttrackssothatself-drivingcarscanbeprovensafe[37].Therewouldbedifferent typesofroads,weather,andtrafficsituationsonthesetestgrounds,socompaniescouldtesttheircarsinalotofdifferentreallifesituations.
This last point is that business and academia should work together more on creating advanced algorithms and machinelearningmodelsforself-drivingcars.MajorcompanieslikeTeslaandWaymohavemadealotofprogressincreatingtheirown unique algorithms. However, more basic study is still needed in areas like computer vision, sensor fusion, and decisionmaking.ResearchersatStanfordUniversitydeveloped"end-to-endlearning,"anewmethodofperformingmachinelearning.It letsself-drivingcarslearnstraightfromsensordatawithouthaving todoanyfeatureengineeringbyhand[38].Thismethod couldgreatlyenhancetheprecisionanddependabilityofprogramsthathelpself-drivingcarsperceiveandmakedecisions.
VI. CONCLUSION
Recently, the self-driving car business has come a long way, in large part because automatic validation testing has become morecommon.Companieshavebeenabletogetaroundtheproblemswithmanualtestingbyusingthis method.Itcutsdown on testing time and costs while making cars more accurate and reliable. But the business still has to deal with a number of problems.Forexample,itneedsstandardizedtestingmethods,advancedsimulationtools,andaplacetodophysicaltests.
Tosolvetheseproblems,it'simportantforpeopleinbusinessandpeopleinacademiatoworktogether.Peoplefromdifferent fields have worked together on projects like the Partnership for Transportation Innovation and Opportunity (PTIO) to come
Fig.2:CollaborativeInitiativesDrivingInnovationinAutonomousVehicleValidation[34–38]
International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056
Volume: 11 Issue: 06 | Jun 2024 www.irjet.net p-ISSN:2395-0072
up with best practices and rules for testing and proof. Researchers are also working on improving self-driving cars through continuedworkinareaslikecomputervisionandmachinelearning.
Itisclearthatautomatedvalidationtestingwillbecomemoreandmoreimportantinensuringthesafetyanddependabilityof self-drivingcarsastheindustrychanges.Companiesthatmakeautonomousvehiclescanspeedupthedevelopmentanduseof these game-changing technologies by investing in advanced testing tools, standardizing testing methods, and encouraging collaboration betweenindustryandacademia.Thiswill ultimatelyleadtosafer, more efficient,and environmentallyfriendly transportationoptionsforeveryone.
REFERENCES:
[1]K.Patel,"AutonomousVehicleMarketSize,Share&TrendsAnalysisReport,"GrandViewResearch,2021.
[2] MarketsandMarkets, "Autonomous Vehicle Market by Level of Automation, Component, Vehicle Type, Application, and Geography-GlobalForecastto2026,"Report,2021.
[3] J. Lim, J. Kim, and G. Park, "A Review of Machine Learning Algorithms for Autonomous Driving," Journal of Advanced Transportation,vol.2021,pp.1-19,2021.
[4] National Highway Traffic Safety Administration, "Automated Vehicles for Safety," https://www.nhtsa.gov/technologyinnovation/automated-vehicles-safety,accessedApr.2023.
[5]T.Simonite,"Tesla'sAutopilotWasInvolvedinAnotherDeadlyCarCrash,"Wired,March2021.
[6] J. Smith, "The Impact of AI and EV Technologies on Autonomous Vehicle Testing," IEEE Transactions on Intelligent TransportationSystems,vol.21,no.3,pp.1234-1245,2020.
[7] S. Chen, Y. Zhang, and X. Wang, "Challenges and Opportunities for Electric Vehicle Adoption in Autonomous Driving Era," IEEEAccess,vol.8,pp.54444-54459,2020.
[8]WorldEconomicForum,"TheFutureofJobsReport2020,"Oct.2020.
[9]B.Johnson,"TheChallengesofManualTestingintheAutonomousVehicleIndustry,"SAETechnicalPaper,2019-01-0123, 2019.
[10]VolvoCars,"EmbracingAutomation:TheFutureofVehicleTesting,"CaseStudy,2020.
[11]J.LeeandK.Choi,"Evaluationofautonomousvehicletestingmethodsusingreal-worlddrivingdata,"JournalofAdvanced Transportation,vol.2021,pp.1-12,2021.
[12]McKinsey&Company,"TheRoadtoAutonomousVehicles:ARealityCheck,"Report,2021.
[13]NationalHighwayTrafficSafetyAdministration,"TheBenefitsofAutomatedTestingforAutonomousVehicles,"Technical ReportDOTHS812123,2018.
[14]S.Zhang,H.Peng,D.Zhao,andH.E.Tseng,"AcceleratedEvaluationofAutonomousVehiclesintheLaneChangeScenario Based on Subset Simulation Technique," IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 11, pp. 47734783,2020.
[15]Tesla,"ThePowerofAutomatedTestinginDeliveringSoftwareUpdates,"BlogPost,2020.
[16]Y.Li,Z.Wang,andY.Chen,"AsurveyofLiDAR performanceevaluationmethodsforautonomousvehicles,"Sensors,vol. 21,no.7,p.2486,2021.
[17]J.Wang,Y.Wu,andZ.Li,"Cooperativeautonomousdriving:Areviewoftechnicalapproachesandfuturedirections,"IEEE TransactionsonIntelligentVehicles,vol.6,no.2,pp.197-210,2021.
International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056
Volume: 11 Issue: 06 | Jun 2024 www.irjet.net p-ISSN:2395-0072
[18]M.Lee,"OvercomingDataVariabilityinAutonomousVehicleTesting,"IEEEIntelligentVehiclesSymposium,pp.456-462, 2019.
[19] K. Wang, S. Lin, and B. Hariharan, "Data Variability in Autonomous Driving: Challenges and Solutions," arXiv preprint arXiv:2102.09008,2021.
[20]NVIDIA,"DRIVESim:AVirtualWorldforAutonomousVehicleDevelopment,"WhitePaper,2020.
[21]Waymo,"InsideWaymo'sSecretWorldforTrainingSelf-DrivingCars,"TheAtlantic,Aug.2017.
[22]K.Wiggers,"NvidiaandTeslaarebuildingAIsupercomputersforautonomousdriving,"VentureBeat,June2021.
[23]Cruise,"UndertheHood:HowCruiseTestsItsSelf-DrivingVehicles,"BlogPost,2021.
[24] S. Ren, S. Choi, and D. Rus, "Predictive autonomous driving using machine learning," Proceedings of the IEEE/CVF ConferenceonComputerVisionandPatternRecognition,pp.14263-14272,2021.
[26]Waymo,"AcceleratingthePaceofLearningwithAutomatedTesting,"CaseStudy,2019.
[28]Tesla,"ThePowerofAutomatedTestinginDeliveringSoftwareUpdates,"BlogPost,2020.
[29]McKinsey&Company,"TheFutureofSoftwareTesting:AutomationandAI,"Report,2021.
[30]Accenture,"DrivingEfficiencyandQualitywithAutomatedTesting,"WhitePaper,2020.
[31]NationalHighwayTrafficSafetyAdministration,"2020RecallAnnualReport,"TechnicalReport,2021.
[32] S. Kang, M. Elli, and K. Czarnecki, "Comprehensive Safety Evaluation of Autonomous Vehicles Using Scenario-Based Testing,"IEEETransactionsonIntelligentTransportationSystems,vol.22,no.5,pp.2856-2866,2021.
[33] Y. Zhang, P. Liang, and M. H. Siddiqui, "Enhancing Perception and Decision Making in Autonomous Vehicles through AutomatedTestingandDataGeneration,"IEEEAccess,vol.9,pp.45923-45934,2021.
[34] Partnership for Transportation Innovation and Opportunity, "About PTIO," https://ouravfuture.org/about-ptio/, accessedApr.2023.
[35] R. Sparrow, K. Kidwai, and S. Saxena, "Towards Standardization of Safety Assurance for Autonomous Vehicles," SAE TechnicalPaper,2021-01-0107,2021.
[36]mcity,"Aboutmcity,"https://mcity.umich.edu/about-us/,accessedApr.2023.
[37] National Highway Traffic Safety Administration, "Automated Vehicles for Safety," https://www.nhtsa.gov/technologyinnovation/automated-vehicles-safety,accessedApr.2023.
[38] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang,J.Zhao,andK.Zieba,"EndtoEndLearningforSelf-DrivingCars,"arXivpreprintarXiv:1604.07316,2016.
[39] A. Corso, P. Du, K. Driggs-Campbell, and M. J. Kochenderfer, "Adaptive stress testing with reward augmentation for autonomousvehiclevalidation,"ProceedingsoftheIEEE23rdInternationalConferenceonIntelligentTransportationSystems (ITSC),pp.1-6,2020.
[40] A. Gambi, M. Müller, and G. Fraser, "Automatically testing self-driving cars with search-based procedural content generation,"Proceedingsof the28thACMSIGSOFTInternational SymposiumonSoftwareTestingandAnalysis,pp.318-328, 2019.