Journal of Automation, Mobile Robotics and Intelligent Systems, vol. 20, no. 1 (2026) by Ł-PIAP

WWW.JAMRIS.ORG pISSN 1897-8649 (PRINT)/eISSN 2080-2145 (ONLINE) VOLUME 20, N° 1, 2026

Indexed in SCOPUS

AI-generated illustration

Journal of Automation, Mobile Robotics and Intelligent Systems

A peer-reviewed quarterly focusing on new achievements in the following fields: • automation • systems and control • autonomous systems • multiagent systems • decision-making and decision support • • robotics • mechatronics • data sciences • new computing paradigms •

Editor-in-Chief

Janusz Kacprzyk (Polish Academy of Sciences, Łukasiewicz-PIAP, Poland)

Advisory Board

Dimitar Filev (Research & Advenced Engineering, Ford Motor Company, USA)

Kaoru Hirota (Tokyo Institute of Technology, Japan)

Witold Pedrycz (ECERF, University of Alberta, Canada)

Co-Editors

Roman Szewczyk (Łukasiewicz-PIAP, Warsaw University of Technology, Poland)

Oscar Castillo (Tijuana Institute of Technology, Mexico)

Marek Zaremba (University of Quebec, Canada)

Executive Editor

Katarzyna Rzeplinska-Rykała, e-mail: office@jamris.org (Łukasiewicz-PIAP, Poland)

Associate Editor

Piotr Skrzypczyński (Poznań University of Technology, Poland)

Statistical Editor

Małgorzata Kaliczyńska (Łukasiewicz-PIAP, Poland)

Editorial Board:

Chairman – Janusz Kacprzyk (Polish Academy of Sciences, Łukasiewicz-PIAP, Poland)

Plamen Angelov (Lancaster University, UK)

Adam Borkowski (Polish Academy of Sciences, Poland)

Wolfgang Borutzky (Fachhochschule Bonn-Rhein-Sieg, Germany)

Bice Cavallo (University of Naples Federico II, Italy)

Chin Chen Chang (Feng Chia University, Taiwan)

Jorge Manuel Miranda Dias (University of Coimbra, Portugal)

Andries Engelbrecht ( University of Stellenbosch, Republic of South Africa)

Pablo Estévez (University of Chile)

Bogdan Gabrys (Bournemouth University, UK)

Fernando Gomide (University of Campinas, Brazil)

Aboul Ella Hassanien (Cairo University, Egypt)

Joachim Hertzberg (Osnabrück University, Germany)

Tadeusz Kaczorek (Białystok University of Technology, Poland)

Nikola Kasabov (Auckland University of Technology, New Zealand)

Marian P. Kaźmierkowski (Warsaw University of Technology, Poland)

Laszlo T. Kóczy (Szechenyi Istvan University, Gyor and Budapest University of Technology and Economics, Hungary)

Józef Korbicz (University of Zielona Góra, Poland)

Eckart Kramer (Fachhochschule Eberswalde, Germany)

Rudolf Kruse (Otto-von-Guericke-Universität, Germany)

Ching-Teng Lin (National Chiao-Tung University, Taiwan)

Piotr Kulczycki (AGH University of Science and Technology, Poland)

Andrew Kusiak (University of Iowa, USA)

Mark Last (Ben-Gurion University, Israel)

Anthony Maciejewski (Colorado State University, USA)

Typesetting

SCIENDO, www.sciendo.com

Webmaster TOMP, www.tomp.pl

Editorial Office

ŁUKASIEWICZ Research Network

– Industrial Research Institute for Automation and Measurements PIAP

Al. Jerozolimskie 202, 02-486 Warsaw, Poland (www.jamris.org) tel. +48-22-8740109, e-mail: office@jamris.org

The reference version of the journal is e-version. Printed in 100 copies.

Articles are reviewed, excluding advertisements and descriptions of products.

Papers published currently are available for non-commercial use under the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 (CC BY-NC-ND 4.0) license. Details are available at: https://www.jamris.org/index.php/JAMRIS/ LicenseToPublish

Open Access.

Krzysztof Malinowski (Warsaw University of Technology, Poland)

Andrzej Masłowski (Warsaw University of Technology, Poland)

Patricia Melin (Tijuana Institute of Technology, Mexico)

Fazel Naghdy (University of Wollongong, Australia)

Zbigniew Nahorski (Polish Academy of Sciences, Poland)

Nadia Nedjah (State University of Rio de Janeiro, Brazil)

Dmitry A. Novikov (Institute of Control Sciences, Russian Academy of Sciences, Russia)

Duc Truong Pham (Birmingham University, UK)

Lech Polkowski (University of Warmia and Mazury, Poland)

Alain Pruski (University of Metz, France)

Rita Ribeiro (UNINOVA, Instituto de Desenvolvimento de Novas Tecnologias, Portugal)

Imre Rudas (Óbuda University, Hungary)

Leszek Rutkowski (Czestochowa University of Technology, Poland)

Alessandro Saffiotti (Örebro University, Sweden)

Klaus Schilling (Julius-Maximilians-University Wuerzburg, Germany)

Vassil Sgurev (Bulgarian Academy of Sciences, Department of Intelligent Systems, Bulgaria)

Helena Szczerbicka (Leibniz Universität, Germany)

Ryszard Tadeusiewicz (AGH University of Science and Technology, Poland)

Stanisław Tarasiewicz (University of Laval, Canada)

Piotr Tatjewski (Warsaw University of Technology, Poland)

Rene Wamkeue (University of Quebec, Canada)

Sławomir Wierzchon (Polish Academy of Sciences, Poland)

Janusz Zalewski (Florida Gulf Coast University, USA)

Teresa Zielińska (Warsaw University of Technology, Poland)

Publisher:

Journal of Automation, Mobile Robotics and Intelligent Systems

VOLUME 20, N˚1, 2026 Contents

Research on a Robust Adaptive Controller with Disturbance Observer for Wheeled Mobile Robot

Trong Tai Nguyen, Doan Phuc An Nguyen, Dai Nghia

Tran, Phuc Bao Nguyen Nguyen, Thanh Dat Mai

DOI: 10.14313/jamris‐2026‐001

A Flexible Mobile Manipulator Architecture: A Case Study on Plasterboard Wall Preparation

Łukasz Granat, Michał Bryła, Kuba Kamiński, Filip Jędrzejczyk, Sławomir Puchalski, Ingmar Kessler, Alexander Perzylo, Ángel Soriano

DOI: 10.14313/jamris‐2026‐002

Integration of Trailer and Tractor Mobile Robot Considering Towing Dynamics

Ünal Dana, Levent Çetin

DOI: 10.14313/jamris‐2026‐003

Adaptive Upper Limb Robot‐Assisted Rehabilitation: Learn‐From‐Therapist Demonstrations

Ismail Auta, Ahmed Fares, Hiroyasu Iwata, Haitham El‑Hussieny

DOI: 10.14313/jamris‐2026‐004

Towards Explainable Graph Spectral Clustering for BERT Embeddings

Mieczysław A. Kłopotek, Sławomir T. Wierzchoń, Bartłomiej Starosta, Piotr Borkowski, Dariusz Czerski

DOI: 10.14313/jamris‐2026‐005

Application of AI Using U‐Net in Skin Lesion Segmentation

Serra Aksoy

DOI: 10.14313/jamris‐2026‐006

Analysis of Histogramasymmetry Forwaste Recognition

Janusz Bobulski, Kamila Pasternak

DOI: 10.14313/jamris‐2026‐007

Explainability of a Deep Neural Network Model for Prediction of Solar Panels Generation: Comparative Study

Rosalís Amador García, María Matilde García Lorenzo, Rafael E. Bello Pérez

DOI: 10.14313/jamris‐2026‐008

A Simple Approach to Designing and Implement‐ing a Fault Detection Device for Large Quantities of LED Boards

Van Anh Pham, Minh Tien Do

DOI: 10.14313/jamris‐2026‐009 85

Design, Implementation, and Performance Opti‐mization of a ROS Based Autonomous Mobile Robot for Intralogistics in Manufacturing Facilities

Neslihan Demir, Pinar Demircioglu, Ismail Bogrekci

DOI: 10.14313/jamris‐2026‐010 93

Semantic‐Aware Trajectory Planning for UAV in Dynamic Environments

Van Hung Nguyen, The Tien Nguyen, Tran Thang Le, Viet Hong Le

DOI: 10.14313/jamris‐2026‐011

113

Evaluating Dijkstra and A* Pathfinding Algorithms for Mobile Robots in Warehouse Environments Using Coppeliasim

Prabin Kumar Jha, Shambo Roy Chowdhury

DOI: 10.14313/jamris‐2026‐012

Energy‐Aware Cluster‐Based Routing with Federated Learning Integration for Scalable IoT Environments

Ankur Sisodia, Swati Vishnoi, Shivshanker Singh, Nandini Sharma, Ajay Kumar Yadav

DOI: 10.14313/jamris‐2026‐013

131

Comparative Analysis of Effective Ai‐Based 3D Multi‐Object Detection and Tracking Methods for Autonomous Driving

Dheepika P.S., Umadevi V.

DOI: 10.14313/jamris‐2026‐014

RESEARCHONAROBUSTADAPTIVECONTROLLERWITHDISTURBANCEOBSERVER FORWHEELEDMOBILEROBOT RESEARCHONAROBUSTADAPTIVECONTROLLERWITHDISTURBANCEOBSERVER FORWHEELEDMOBILEROBOT

Submitted:16th September2024;accepted:4th November2025

TrongTaiNguyen,DoanPhucAnNguyen,DaiNghiaTran,PhucBaoNguyenNguyen,ThanhDatMai DOI:10.14313/jamris‐2026‐001

Abstract:

Thispaperpresentsarobustadaptivecontrollerfor wheeledmobilerobots(WMRs)designedtoeffectively compensatefordisturbancesandsystemuncertainties. Theproposedcontrolschemeincludesaninnerloopwith aPIDcontrollerforwheelsspeedcontrolandanouter loopwithadisturbanceobservertotrackthetrajec‐toryandminimizepositionerrors.Theouterloopcon‐trolsignalisderivedbasedontheWMR’sdynamicand kinematicmodels,whilethedisturbanceobserveradapts touncertaintiesandexternaldisturbances.Thesystem’s stabilityisproveninsenseofLyapunovtheory.Theper‐formanceoftheproposedcontrollerisvalidatedthrough simulations,whichshowsignificantimprovementsintra‐jectorytrackingcomparedtoexistingmethods.Further‐more,experimentalresultsconfirmthatthecontroller maintainsstabilityandrobustnessundervaryingload conditionsanddifferenttrajectorypaths,demonstrating itseffectivenessinreal‐worldapplications.

Keywords: WheeledMobileRobot(WMR),Lyapunov Stability,DisturbanceObserver,AdaptiveControl,WMR Dynamic

1.Introduction

WheeledMobileRobot(WMR)referstorobots thatnavigateonthegroundusingmotorizedwheels topropelthemselves.Amongthebranchesofmobile robots,WMRsaresimplerandmoreef icientthan otherdesigns,suchasthosethatutilizegroovesor leggedrobots.Theyareeasytodesign,produce,and programforlinearandangularmotiononahori‑ zontalplane.ThismakestheWMRsanactive ield ofresearchinordertoful illtheneedsofsurveys, patrols,emergencysavings,identi ication,petrochem‑ icalsinautomation,buildingstructures,transporta‑ tion,andhealthcare.Withnumerousfeatures,WMRs arebeingappliedinvarious ieldssuchasagriculture servicesandotherindustries[1–7].Sincethen,the requirementsforaccuratelycontrollingthemovement trajectoryofmobilerobotswithwheelshavealso increased.

Experimentsshowthatbetweenthetwomain typesofWMR,nonholonomicrobotsaremoredif icult tocontrolthanholonomicrobots[8].Hence,thiskind ofsystemposesmoreproblemsforresearchers.

Thetrajectoryaccuracyofholonomicrobotsis affectedbyvariousuncertainfactorsfromtherobot itselfandexternalforces,makingitmorechallenging tocontrolthesystem[9].

Toaddressthesechallenges,varioussolutions havebeenproposedtoenhancetheaccuracyand trajectorytrackingofnonholonomicwheeledmobile robots(WMRs)underuncertaininterferencefactors. Thesesolutionsincludefuzzycontrol[10–12],back‑ steppingbasedcontrol[13–15],andslidingmodecon‑ trol(SMC)[16–18],amongothers[5, 12, 19–21]. However,controllersbasedonthe“no‑slipwheel” hypothesis,suchasSMC[16,18,22],adaptivecontrol [23–26],andbacksteppingcontrol[13,14],failtomeet thesystem’srequirementsforstabilityandprecise trajectorytracking.

Inrecentyears,severalmethodshavebeenpro‑ posedtosolveinterferenceproblemsinWMRcon‑ trol.Oneofthemisrobustadaptivetrackingcontrol [26–28],atrackingcontrollerthatcombinesdistur‑ banceobserverandaresponsivecompensationmech‑ anismtodealwithunstablesystems.Therefore,the errorsandresponsesarelimited.In[29],theprob‑ lemoftrajectorytrackingforWMRunderconditions ofsystemuncertaintyandexternaldisturbanceswas solved.Byusingafuzzylogicsystem(FLS)formodel estimationandanadaptivefuzzyobserverforesti‑ matingunmeasuredvelocities,thepositionerrorsare minimizedandsystemresponseisfast.

WheelslipsinWMRarealsoaninterestingprob‑ lemformanyresearchers.Inordertoovercomethe challengesinaconventionalbacksteppingcontroller, adynamicsurfacecontroller(DSC)isdeveloped[5, 30],whichincorporatesskiddingandslidingmech‑ anisms.In[17],acontrolschemecombiningsliding modecontrolwithabacksteppingobserverispro‑ posedforWMRswithunknownslidingandmodel uncertainty.Thisalgorithmmitigatestheeffectsof systemuncertainties,disturbances,andvibrations inherentinslidingmodecontrol;however,theillus‑ trativeresultsarenotsatisfactory.Acontroltechnique utilizingafuzzyPIDcontroller,combinedwithIMU andfeedbackcurrent,isintroducedin[31]toreduce slippagecausedbyenvironmentalfactors.Whilethe resultsindicatethatthiscontrolschemeiseffective, systemstabilityisnotaddressed.In[17,32],acon‑ trollerbasedonadisturbanceobserverthataccounts forskiddingandslidingispresented,howeverthe errorinthisalgorithmremainssigni icantlyhigh.

Therobustadaptivecontrollerdevelopedin[33] accountsfordisturbancesandwheelslips,withthe WMRdynamicsexplicitlyconsideringwheelslipinthe trackingcontroller’sdesign.Adisturbanceobserveris utilizedtoestimaterobotuncertaintiesandslip.How‑ ever,thecontrollerhasonlybeenveri iedthrough simulations,andthediscontinuoustransitioninthe controllawmayleadtoinstability.

Thispaperaimstodealwiththeshortcomingsof theapproachpresentedin[33]fortrackingcontrol ofWheeledMobileRobots(WMRs).Theproposed controlschemeconsistsoftwoloops:aninnerloop, whichusesaPIDcontrollerforprecisewheelspeed regulation,andanouterloop,whichemploysadis‑ turbanceobservertoaccuratelytrackthereference trajectoryincaseofdisturbancesanduncertainties. Thekeycontributionsofthispaperareasfollows:

• Improvementof[33]: Theouterlooptrackingcon‑ trollerisre‑derivedusingLyapunovstabilitythe‑ ory,andanovelmethodforhandlingnoninvertible matricesisintroduced.

• SimulationValidation: Theproposedcontrol structureisthoroughlytestedinsimulations, demonstratingsigni icantimprovementsinsystem responsecomparedtothemethodin[33].

• ExperimentalVeri ication: Theeffectivenessof theproposedschemeisfurthervalidatedthrough realworldexperimentsonaWMR,withtheresults con irmingitssuperiorperformance.

2.WheeledMobileRobotModel

Inthisresearch,theWMRmodelisreferred from[33],thekinematicsanddynamicsmodelof WMR,whichtakesaccountofwheelslips,andissum‑ marizedasfollows:

3.DevelopmentofController

Similartotheapproachin[33],thecontrolscheme fortheWMRconsistsoftwoloopsarrangedasacas‑ cadecontrolsystem,asshowninFig. 2.Theouter loopisresponsibleforcontrollingtherobot’sposition. Thiscontrollerincorporatesadisturbanceobserver tocompensatefordisturbances,systemuncertainties, andmodelerrors.Meanwhile,theinnerloopman‑ ageswheelspeedcontrol,ensuringthatthewheels followthespeedcommandsgeneratedbytheouter loop.

ThePIDcontroller,knownforitswideapplication duetoitsstability,robustness,andreliablecontrol performance,isusedintheinnerloop.Asaresult, theoverallcontrolsystemheavilydependsonthe effectivenessoftheouterloopcontroller.Therefore, thissectionfocusesontheouterloopcontrollerand theintegrationofthedisturbanceobservertoenhance systemperformance.

3.1.Robotpositiontrackingcontrol

IntheMXYcoordinate,theerrorsbetweenthe robotposition ��(��,��) anditsdesiredpoint ��(��,��)aredeterminedas:

where��= cos(��) sin(��) sin(��) cos(��)

Takingderivativeof(4)withrespecttotime,itcan beobtainedas[33]:

where��,��aretheWMRforwardvelocityandangular velocityatmidpointofdrivingwheels��,respectively, whichisdeterminedas:

WMRincoordinatesanditsdynamicfactors

Therobotparametersarede inedandshownin Fig.1.Inwhich,��(��,��)isthecenterofmassofthe WMRplatform,��(��,��)isthemidpointofthedriv‑ ingwheels,��(��,��)isthedesiredposition;��isthe orientationofWMR;2��isthewidthofWMR;��isthe wheelradius;��1,��2 arethepositionerrorsbetween �� and �� intheMXYcoordinate; �� isthedistance between��and��;��,�� arelongitudinalslipfactorsof therightandleftdrivingwheels,respectively;��isthe lateralslipfactoralongthewheelshaft;��,��,��,��and�� areweightingmatricesandvectorsinWMRdynamic, whicharede inedindetailsin[33]:

Overallstructureofproposedcontrolscheme

Figure1.

Figure2.

=��̇�� +ℎ(.)��+��(.) (6)

where ��= �� isthevectorrotational speedsofrightandleftdrivingwheels;

�� = [ �� ]�� ;ℎ(.)��(.) aretwo matrices,thatare determinedasfollows:

Thenthederivativeofthiserrorcanbedetermined as:

(16)

From(14)and(16),itiseasytoobtain:

(17)

Itisassumedthatthedisturbanceislowvariation, then��≈0and

becomes:

3.4.Systemstabilitywithdisturbanceobserver

(18)

Withthedisturbanceobserverin(14),thecontrol signalin(12)willbecome:

Then,theobjectiveoftheWMRcontrolisto ind thecontrolrulefor��sothat�� convergestozero.

3.2.Trackingcontrolsignal

WhentheWMRfollowsthereference,theposi‑ tionerrorsalong��andydirectionsconvergetozero. Hence,to indthecontrolruleforpositioncontroller, theLyapunovcandidatefunctionischosenas:

From(6),itcanbeobtained:

From(11),thecontrollaw��isselectedas:

Then(11)becomes:

Thus,thesystemisasymptoticstablewiththecon‑ trollawin(12)and �� convergestozerointermof Lyapunovstablecriteria.

From(11),theterm��(.)isanunknownfactor,thus thecontrolsignalcannotbecalculateddirectly.Inthis researchthedisturbanceobserverin[34]isemployed toestimatethisterm.

3.3.Disturbanceobserver

Considerthesystem(6),thedisturbanceobserver [34]isdesignedas:

Thentheequation(6)willbecome:

Thedisturbanceestimationerrorisde inedas:

(15)

where�� =��−��isdisturbanceestimationerror TheLyapunovcandidatefunctionischosenas:

Thenthederivativeofthisfunctionwillbe:

If �� and �� arechosenas ��>1/2 and ��>1/2, then��<0.Thismeansthatthesystemisasymptotic stableaccordingtoLyapunovtheorem. �� and �� will convergetozerowiththecontrollawin(19)andthe disturbanceobserver(14).

In(7),when

��1 ��

�� →0,then

Henceℎ(.)isnotinvertible,thecalculationofthe controlsignal �� cannotbeimplemented.Toresolve thatproblem,insteadofusinglawas[33]inthis research,thevalueof ��1 in(7)isintroducedtobe replacedby ��1 =��sign ��1 as ��1 ≤��.This solutionisconsistentbecausethesignof��1 directly affectsthecontrolsignal��;thevalueof�� isselectso thatthewheelrotationalspeedsreachtheminimum speedincase��1 ≤��

Thedetaildiagramofthetrackingcontrollerwith disturbanceobserverisdepictedinFig.3.

4.DescriptionoftheHardwareandtheExper‐imentalProcedure

TheblockdiagramforaWMRexperimentalcon‑ trolsetupisshowninFig. 4.Inthissetup,theouter loopcontrollerisexecutedontheembeddedcom‑ puter.Inwhich,theWMRpositionandorientationis determinedbyanArUcomarkerandvisionmethod. Basedonthesemeasurements,theadaptivecontroller withdisturbanceobserverdescribedin(19)isimple‑ mentedto indthereferencesignalsfortheinnerloop controller.Theinnerloopcontrollerisexecutedonthe microcontrollertocontroltherotationalspeedofthe rightandleftdrivingwheelofrobot.

TheapparatusofWMRhardware,componentsand experimentalcontrolsetupareshowninFig. 5 to Fig. 7.Inthissetup,theIPcameraismountedon thetoppositionsothatitcancovertherobotand itsexperimentaltrajectory.Therobotpositionand

orientationisdetectedbytheArUcomarkerthatis placedatmidpointofdrivingwheels M.

Theexperimentalprocedureinvolvesthefollow‑ ingsteps:

(1) DeploythePIDcontrollerfortherobot’sinnerloop onthemicrocontroller.

(2) SetupthemountingframefortheIPcameraand performcalibrationusingacheckerboardpattern.

Figure3. Diagramofouter‐looptrackingcontroller

Figure4. BlockdiagramofWMRexperimentalcontrol implementation

Figure5. ApparatusofWMRwithanArUcomarkerand embeddedcomputer

Figure6. SetupofelectriccomponentsinWMR

(3) Deploytheposeestimationprogramonthe embeddedcomputertogatherpositiondata(x,y, ��)fromthecamera.

(4) Deploytheadaptivecontrollerwithadisturbance observerontheembeddedcomputer.

(5) Positiontherobotatitsinitialpointfortrajectory tracking.

(6) ExecutetheprogramsthroughROS.

(7) Collectthedataandprinttheresults.

(8) Stoptheobservationprocessbyterminatingthe program.

5.SimulationandExperiment

Toevaluatetheproposedcontroller,itwastested inbothsimulationandexperimentalsettings. Inthesimulation,theWMRmodelandthe proposedcontrollerwereimplementedinMATLAB Simulink.Twotrajectorycaseswereusedtoverify thesimulationresults:aneight‑shapedtrajectoryand acirculartrajectory.Themathematicalexpression fortheeightshapedtrajectoryispresentedinEqua‑ tion(23),whilethecirculartrajectoryisdescribedin

Table1. Robotsimulationparameters

Equation(24):

ThesimulationparametersfortheWMRaresetin theconditionofourexperimentrobotasTable1;and thecontrollerparametersaresetas ��= 20 02 , ��= 1000 0100 .Theloaddisturbancesanduncer‑ taintyfactorsaresetas:

= 1+sin(0.2��) 1+cos(0.2��) ;

SimulationresultsarepresentedinFig.8through Fig. 23,demonstratingtheperformanceofthepro‑ posedcontrolleroncircularandeight‑liketrajecto‑ ries.Additionally,thealgorithmfrom[33]isevalu‑ atedforcomparisonwiththeproposedcontroller.The simulationresultsindicatethatbothcontrollerscan effectivelytrackthereferencetrajectories.Tracking

Figure8. Circulartrajectorytrackingresultbyusing proposedcontroller

Figure7. Overallimageofexperimentalcontrolsetup

Figure9. Circulartrajectorytrackingerrorsbyusing proposedcontroller

Figure10. Referencespeedsofdrivingwheelswith respecttocirculartrajectorygeneratedbyproposed controller errorsforbothcontrollersareshowninFig.9,Fig.13, Fig.17andFig.21

ItisevidentthatthetrackingerrorsinFig. 9 andFig.17convergesmoothlytosmallvalues,while thecontrollerfrom[33]exhibitssigni icantsudden changeswhentheerrorissmall,asshowninFig.13 andFig.21.Thisbehaviorisduetothediscontinuous translationinthematrixℎ(.)when�� →0inequation (19)changes.

Theproposedcontroller’ssigni icantadvantages overtheonein[33]areevidentinthewheelspeed controlsignals(Fig.10,Fig.14,Fig.18,andFig.22)and disturbanceestimationresults(Fig.11,Fig.15,Fig.19 andFig.23).

Itcanbeobservedthatthewheelvelocitiesand estimateddisturbancesarestableandsmoothwith theproposedcontroller,whereasthecontrollerin [33]exhibitslargevariations,particularlyfromthe 47thsecondonward,asseeninFig. 15,andFig. 23 Thesevariationsmayleadtosysteminstability.

Thesimulationresultscon irmthattheproposed controllerperformseffectively,exhibitingaccurate‑ tracking,smoothresponses,preciseestimations,and overallsystemstability.

Figure11. Disturbancesestimationwithrespectto circulartrajectoryfromproposedcontroller

Figure12. Circulartrajectorytrackingresultby controllerfrom[33]

Figure13. Circulartrajectorytrackingerrorbycontroller fromreference[33]

Followingthesuccessfulsimulations,thepro‑ posedcontrollerwasimplementedandtestedunder real‑worldconditionsusingaphysicalrobot.Inthis

Figure14. Referencespeedsofdrivingwheelswith respecttocirculartrajectorygeneratedbycontroller from[33]

Figure15. Disturbancesestimationbycontroller from[33]

Figure16. Eight‐liketrajectorytrackingresultbyusing proposedcontroller experimentalsetup,therobotwastaskedwithtrack‑ ingcircularandsquaretrajectories,bothunderno‑ loadand5kgloadconditions.

Figure17. Eight‐liketrajectorytrackingerrorwith respecttoeight‐likebyproposedcontroller

Figure18. Referencespeedsofdrivingwheelswith respecttoeight‐liketrajectorygeneratedbyproposed controller

Figure19. Disturbancesestimationwithrespectto eight‐liketrajectoryfromproposedcontroller

Fig.24throughFig.27presentthecontrolresults obtainedfromthereal‑worldexperiments.Theresults demonstratethatthecontrollersuccessfullyguides therobotalongthecirculartrajectory(Fig. 24)with asmallsteady‑stateerroroflessthan3cm(Fig.25). Thecontroller’soutputsignals,representedbythe

Figure20. Eight‐liketrajectorytrackingresultby controllerfrom[33]

Figure21. Eight‐liketrajectorytrackingerrorby controllerfromreference[33]

Figure22. Referencespeedsofdrivingwheelswith respecttoEight‐liketrajectorygeneratedbycontroller from[33]

Figure23. Disturbancesestimationwithrespectto eight‐likebycontrollerfrom[33]

Figure24. Experimentaltrackingresultwithrespectto circulartrajectory

Figure25. Experimentaltrackingerrorswithrespectto circulartrajectory

Thedisturbanceestimationresultsareillustratedin Fig.27

wheelvelocities,areshowninFig. 26.Forthecircu‑ lartrajectory,therightwheelappearstomaintaina constantspeedof0.06m/s,whiletheleftwheelspeed adjuststofollowthetrajectorywithsmallerspeed.

Similarly,thecontrolresultsforthesquaretrajec‑ toryareshowninFig.28throughFig.31.Fig.28illus‑ tratesthetrackingperformanceforthesquaretrajec‑ tory.Itcanbeobservedthatthereisanovershoot

Figure26. Experimentalwheelsvelocitywithrespectto circulartrajectory

Figure27. Experimentaldisturbancesestimationwith respecttocirculartrajectory

Figure28. Experimentaltrackingresultwithrespectto squaretrajectory

whenthereferencechangesitsorientation,which occursduetotherobot’sinertiaandreferencespeed, causingittocontinuemovingforwarduntilitadaptsto theneworientation.Fig.30depictsthewheelspeeds, showingthattheleftwheelspeeddecreasesrapidly

Figure29. Experimentaltrackingerrorwithrespectto squaretrajectory

Figure30. Experimentalwheelsvelocitywithrespectto squaretrajectory

Figure31. Experimentaldisturbancesestimationwith respecttosquaretrajectory

toaccommodatethesuddenchangeinthereference trajectory.Thedisturbanceestimationresultsarepre‑ sentedinFig.31

Finally,thecontrollerisveri iedinlong‑timeand loadconditions.Inthistest,therobotcarriesa5kg

Figure32. Experimentaltrackingresultwithrespectto 4loopscirculartrajectoryunder5kgload

Figure33. Experimentaltrackingerrorswithrespectto 4loopscirculartrajectoryunder5kgload

Figure34. Experimentalwheelsvelocitywithrespectto 4loopscirculartrajectoryunder5kgload

loadandtracktocircularandsquaretrajectoriesin4‑ loops.ThetrackingresultsareshowninFig. 32 and Fig.35.Itcanbeseenthatthecontrollersuccessfully adaptstovaryingloadconditions.Furthermore,the controlperformancesarekeptstableinallloops,this demonstratesthatthecontrollerisstableandrobust

Figure35. Experimentaltrackingresultwithrespectto 4loopssquaretrajectoryunder5kgload

Figure36. Experimentaltrackingerrorswithrespectto 4loopssquaretrajectoryunder5kgload

Figure37. Experimentalwheelsvelocitywithrespectto 4loopscirculartrajectoryunder5kgload

overtime.Thecontrolsignalswithrespecttotra‑ jectories,aredisplayedinFig. 34 andFig. 37.They showthatthewheels’speedsaresimilaraftereach cycle.Thisprovestherobustnessandstabilityofthe controller.

Finally,theproposedcontrolleristestedunder longdurationandloadconditions.Inthisexperiment, therobotcarrieda5kgloadwhiletrackingcircular andsquaretrajectoriesoverfourloops.Thetrack‑ ingresultsareshowninFig. 32 andFig. 35.The resultsindicatethatthecontrolleradaptswellunder loadconditions,maintainingstablecontrolperfor‑ mancethroughoutallloops.Thisdemonstratesthe controller’sstabilityandrobustnessovertime.Fur‑ thermore,thecontrolsignalscorrespondingtothetra‑ jectoriesaredisplayedinFig.33andFig.35,showing thatthewheelspeedsremainconsistentaftereach cycle.Thisconsistencyfurtherprovestherobustness andstabilityofthecontroller.

6.Conclusion

Inthisresearch,arobustadaptivecontrollerwith adisturbanceobserverwasinvestigatedfornonholo‑ nomicrobots,basedonthekinematicanddynamic modelsofWheeledMobileRobots(WMRs).Thedis‑ turbanceobserverwasemployedtohandlesystem uncertaintiesandexternaldisturbances.Thestability oftheproposedcontrollerwasprovenusingbothHur‑ witzandLyapunovstabilitycriteria.

Thestudyprimarilyfocusesontheouterloop trackingcontrolscheme,whichimprovestheperfor‑ manceofapreviouslyproposedmethodbyincorpo‑ ratingadisturbancehandlerandaddressingremain‑ ingissues,suchasprovidingaclearmathematical interpretationfornon‑invertiblematricesandimple‑ mentingthisdirectly.

Moreover,extensivesimulationsandexperimental resultswereconducted,demonstratingtheproposed controller’sef iciency,robustness,andstability.

ACKNOWLEDGEMENTS

ThisresearchisfundedbyVietnamNationalUniver‑ sityHoChiMinhCity(VNU‑HCM)undergrant num‑ berC2021‑20‑15 andbyHoChiMinhCityUniver‑ sityofTechnology‑VNU‑HCMundergrant number SVOISP‑2023‑ĐDT‑75

Weacknowledgethesupportoftimeandfacili‑ tiesfromHoChiMinhCityUniversityofTechnology (HCMUT),VNU‑HCMforthisstudy.

Authors’Contributions TrongTaiNguyenconceived ofthepresentedidea,developedthetheoryand performedthecomputations,preparedfacilitiesfor experimental,veri iedtheanalyticalmethodsand results.Othersconductedtheexperimentalresults, anddatacollection.Allauthorsdiscussedtheresults andcontributedtothe inalmanuscript.

Declarations

DataAvailability Theauthorscon irmthatthedata supportingthe indingsofthisstudyareavailable withinthearticleanditsSupplementarymaterial.Raw datathatsupport indingsofthisstudyareavailable fromthecorrespondingauthor,uponthereasonable request.

Competinginterests Notapplicable

AUTHORS

TrongTaiNguyen∗ –FacultyofElectrical&Elec‑ tronicsEngineering,HoChiMinhCityUniversity ofTechnology(HCMUT),700000,Vietnam,e‑mail: nttai@hcmut.edu.vn.

DoanPhucAnNguyen –FacultyofElectrical&Elec‑ tronicsEngineering,UndergraduateSchool,HoChi MinhCityUniversityofTechnology(HCMUT),Vietnam NationalUniversityHoChiMinhCity,700000,Viet‑ nam,e‑mail:an.nguyen23112001@hcmut.edu.vn.

DaiNghiaTran –FacultyofElectrical&Electronics Engineering,UndergraduateSchool,HoChiMinhCity UniversityofTechnology(HCMUT),VietnamNational UniversityHoChiMinhCity,700000,Vietnam,e‑mail: nghia.tran.sherlock@hcmut.edu.vn.

PhucBaoNguyenNguyen –FacultyofElectrical& ElectronicsEngineering,UndergraduateSchool, HoChiMinhCityUniversityofTechnology (HCMUT),VietnamNationalUniversityHo ChiMinhCity,700000,Vietnam,e‑mail: nguyen.nguyenbao311003@hcmut.edu.vn.

ThanhDatMai –FacultyofElectrical&Electronics Engineering,UndergraduateSchool,HoChiMinhCity UniversityofTechnology(HCMUT),VietnamNational UniversityHoChiMinhCity,700000,Vietnam,e‑mail: dat.maibk021@hcmut.edu.vn.

∗Correspondingauthor

References

[1] R.RajandA.Kos,”AComprehensiveStudy ofMobileRobot:History,Developments, Applications,andFutureResearchPerspectives”, AppliedSciences.,vol.12,no.14,2022.DOI: 10.3390/app12146951

[2] I.DorofteiandF.Adăscăliței,”PRACTICAL APPLICATIONSFORMOBILEROBOTSBASED ONMECANUMWHEELS‑ASYSTEMATIC SURVEY”,TheRomanianReviewofPrecision Mechanics,OpticsandMechatronics.,no.40, 2011,21–29.

[3] C.Mireles,M.Sanchez,D.Cruz‑Ortiz,I.Salgado, andI.Chairez,”Home‑carenursingcontrolled mobilerobotwithvitalsignalmonitoring”,Med‑ ical&BiologicalEngineering&Computing.,vol. 61,no.2,2023,399–420.DOI:10.1007/s11517‑ 022‑02712‑y

[4] O.Buckmann,M.Krömker,andU.Berger,”An ApplicationPlatformfortheDevelopment andExperimentalValidationofMobile RobotsforHealthCarePurposes”,Journal ofIntelligentandRoboticSystems., vol.22,no.3,1998,331–350.DOI: https://doi.org/10.1023/A:1007945702881

[5] R.Galati,G.Mantriota,andG.Reina,”Adaptive headingcorrectionforanindustrialheavy‑duty omnidirectionalrobot”,(ineng),SciRep.,vol.12,

no.1,2022,19608.DOI:10.1038/s41598‑022‑ 24270‑x

[6] C.Patruno,V.Renò,M.Nitti,N.Mosca,M.di Summa,andE.Stella,”Vision‑basedomnidirec‑ tionalindoorrobotsforautonomousnaviga‑ tionandlocalizationinmanufacturingindustry”, Heliyon.,vol.10,no.4,p.e26042,2024.DOI: 10.1117/12.2595511

[7] J.P.Trevelyan,S.‑C.Kang,andW.R.Hamel, ”RoboticsinHazardousApplications”, inSpringerHandbookofRobotics.,B. SicilianoandO.Khatib,Heidelberg:Springer BerlinHeidelberg,2008,1101–1126.DOI: 10.1007/978‑3‑319‑32552‑1_58

[8] H.TaheriandC.X.Zhao,”Omnidirectional mobilerobots,mechanismsandnavigation approaches,”MechanismandMachine Theory.,vol.153,2020,103958.DOI: 10.1016/j.mechmachtheory.2020.103958

[9] L.Wijayathunga,A.Rassau,andD.Chai, ”ChallengesandSolutionsforAutonomous GroundRobotSceneUnderstandingand NavigationinUnstructuredOutdoor Environments:AReview”, AppliedSciences., vol.13,no.17.DOI:10.3390/app13179877

[10] S.A.AhmedandM.G.Petrov,”Trajectory ControlofMobileRobotsusingType‑ 2Fuzzy‑NeuralPIDController”,IFAC‑ PapersOnLine.,vol.48,no.24,2015.DOI: https://doi.org/10.1177/09596518251322228

[11] K.Al‑MutibandF.Abdessemed,”Indoormobile robotnavigationinunknownenvironmentusing fuzzylogicbasedbehaviors”,Adv.Sci.Technol. Eng.Syst.J.,vol.2,no.3,327–337,2017.DOI: https://dx.doi.org/10.25046/aj020342

[12] Y.‑H.ChenandY.‑Y.Chen,”NonlinearAdaptive FuzzyControlDesignforWheeledMobile RobotswithUsingtheSkewSymmetrical Property”,Symmetry.,vol.15,no.1.DOI: 10.3390/sym15010221

[13] G.Zidani,S.Drid,L.Chri i‑Alaoui,A.Ben‑ makhlouf,andS.Chaouch,”Backsteppingcon‑ trollerforawheeledmobilerobot”,in20154th InternationalConferenceonSystemsandCon‑ trol(ICSC),2015,443–448.DOI:10.1109/ICoSC. 2015.7153286

[14] G.Yanfeng,Z.Hua,andY.Yanhui,”Back‑ SteppingandNeuralNetworkControlofa MobileRobotforCurvedWeldSeamTracking”, ProcediaEngineering.,vol.15,2011,38–44.DOI: https://doi.org/10.1016/j.proeng.2011.08.009

[15] M.A.MoqbelObaid,A.R.Husain,andA.A. MohammedAl‑kubati,”RobustBackstepping TrackingControlofMobileRobotBased onNonlinearDisturbanceObserver”, InternationalJournalofElectricalandComputer Engineering(IJECE).,vol.6,no.2,2016.DOI: 10.11591/ijece.v6i2.pp.901–908

[16] S.A.Tchenderli‑Baham,F.Hamerlain,andN. Saadia,”Adaptiveslidingmodeforthecon‑ trolofawheeledmobilerobot”,in201515th InternationalConferenceonControl,Automa‑ tionandSystems(ICCAS),2015,699–703.DOI: https://doi.org/10.1109/ICCAS.2015.73650

[17] Y.Jinhua,Y.Suzhen,andJ.Xiao,”Trajectory TrackingControlofWMRBasedonSlidingMode DisturbanceObserverwithUnknownSkidding andSlipping”,in20172ndInternationalConfer‑ enceonCybernetics,RoboticsandControl(CRC), 2017,18–22.DOI:10.1109/CRC.2017.40

[18] A.Filipescu,V.Minzu,B.Dumitrascu,and E.Minca,”Trajectory‑trackinganddiscrete‑time sliding‑modecontrolofwheeledmobilerobots”, in2011IEEEInternationalConferenceon InformationandAutomation,2011,27–32.DOI: 10.1109/ICINFA.2011.5948958

[19] J.Meng,H.Xiao,L.Jiang,Z.Hu,L.Jiang, andN.Jiang,”AdaptiveModelPredictive ControlforMobileRobotswithLocalization FluctuationEstimation”,(ineng),Sensors (Basel).,vol.23,no.5,2023.DOI: https://doi.org/10.3390/s23052501

[20] L.Jiang,S.Wang,Y.Xie,S.Q.Xie,S.Zheng,and J.Meng,”Fractionalrobust initetimecontrol offour‑wheel‑steeringmobilerobotssubjectto serioustime‑varyingperturbations”,Mechanism andMachineTheory.,vol.169,2022,104634. DOI:10.1016/j.mechmachtheory.2021.104634

[21] I.Unal,O.Kabaş,O.Eceoğlu,andG.Moiceanu, ”AdaptiveMulti‑RobotCommunication SystemandCollisionAvoidanceAlgorithm forPrecisionAgriculture”,Applied Sciences.,vol.13,no.15,2023,8602.DOI: https://doi.org/10.3390/app13158602

[22] Z.Cao,J.SongandY.Niu,”Finite‑timesliding modecontrolofMarkovianjumpsystemssub‑ jecttoactuatornonlinearitiesanditsapplication towheeledmobilemanipulator”,Journalofthe FranklinInstitute.,vol.355,no.16,2018,7865‑ 7894.DOI:10.1109/TAC.2019.2926156

[23] L.Ding,S.Li,Y.J.Liu,H.Gao,C.Chen,and Z.Deng,”AdaptiveNeuralNetwork‑Based TrackingControlforFull‑StateConstrained WheeledMobileRoboticSystem”,IEEE TransactionsonSystems,Man,andCybernetics: Systems.,vol.47,no.8,2017,2410–2419.DOI: 10.1109/TSMC.2017.2677472

[24] K.Shojaei,”Neuraladaptiveoutputfeedback formationcontroloftype(m,s)wheeled mobilerobots,”IETControlTheory& Applications.,vol.11,no.4,2017,504–515.DOI: https://doi.org/10.1049/iet‑cta.2016.0952

[25] Z.Peng,S.Yang,G.Wen,A.Rahmani,and Y.Yu,”Adaptivedistributedformation controlformultiplenonholonomic wheeledmobilerobots”,Neurocomputing.,

vol.173,2016,1485–1494.DOI: https://doi.org/10.1016/j.neucom.2015.09.022

[26] M.Begnini,D.W.Bertol,andN.A.Martins, ”Arobustadaptivefuzzyvariablestructure trackingcontrolforthewheeledmobilerobot: Simulationandexperimentalresults”,Control EngineeringPractice,vol.64,2017,27–43.DOI: https://doi.org/10.1016/j.conengprac.2017.04. 006

[27] L.Xin,Q.Wang,J.She,andY.Li,”Robust adaptivetrackingcontrolofwheeled mobilerobot”,RoboticsandAutonomous Systems.,vol.78,2016,36–48.DOI: https://doi.org/10.1016/j.robot.2016.01.002

[28] H.Fang,Y.Zhu,S.Dian,G.Xiang,R.Guo,andS.Li, ”Robusttrackingcontrolformagneticwheeled mobilerobotsusingadaptivedynamicprogram‑ ming”,ISATransactions.,vol.128,2022,123–132.DOI:10.1016/j.isatra.2021.10.017

[29] S.PengandW.Shi,”AdaptiveFuzzyOutput FeedbackControlofaNonholonomicWheeled MobileRobot”,IEEEAccess.,vol.6,2018,. DOI:10.1109/ACCESS.2018.2862163

[30] J.T.HuangandY.L.Sung,”Adaptivebeck‑ steppingdynamicsurfacetrackingcontrolof nonholonomicmobilerobots”,in2016IEEE 11thConferenceonIndustrialElectronicsand Applications(ICIEA),2016,1026–1031.DOI: 10.1109/ICIEA.2016.7603733

[31] D.E.Kim,H.N.Yoon,K.S.Kim,M.S.Sreejith,andJ. M.Lee,”Usingcurrentsensingmethodandfuzzy

PIDcontrollerforslipphenomenaestimation andcompensationofmobilerobot,”in201714th InternationalConferenceonUbiquitousRobots andAmbientIntelligence(URAI),2017,397–401.DOI:10.5772/intechopen.110497

[32] M.Chen,”DisturbanceAttenuationTracking ControlforWheeledMobileRobotsWithSkid‑ dingandSlipping”,IEEETransactionsonIndus‑ trialElectronics.,vol.64,no.4,2017,pp.3359–3368.DOI:10.1109/TIE.2016.2613839

[33] N.T.‑T.Vu,L.X.Ong,N.H.Trinh,andS.T.Huong Pham,”Robustadaptivecontrollerforwheel mobilerobotwithdisturbancesandwheelslips”, InternationalJournalofElectricalandComputer Engineering(IJECE).,vol.11,no.1,2021,11.DOI: 10.11591/ijece.v11i1.pp336–346

[34] W.H.Chen,J.Yang,L.Guo,andS.Li,”Disturbance‑ Observer‑BasedControlandRelatedMethods–AnOverview”,IEEETransactionsonIndustrial Electronics.,vol.63,no.2,2016,1083–1095. DOI:10.1109/TIE.2015.2478397

AFLEXIBLEMOBILEMANIPULATORARCHITECTURE: ACASESTUDYONPLASTERBOARDWALLPREPARATION ACASESTUDYONPLASTERBOARDWALLPREPARATION

AFLEXIBLEMOBILEMANIPULATORARCHITECTURE: ACASESTUDYONPLASTERBOARDWALLPREPARATION

Submitted:6th November2024;accepted:24th February2025

ŁukaszGranat,MichałBryła,KubaKamiński,FilipJędrzejczyk,SławomirPuchalski,IngmarKessler,AlexanderPerzylo, ÁngelSoriano

DOI:10.14313/jamris‐2026‐002

Abstract:

Skilledlaborshortageandacompetitivemarketare challengesformanybusinesses.Hence,automationand robotizationarewell‐establishedinthelarge‐scale,well‐structuredindustrialenvironmentsofmanyfactories. Moreover,thereisapushforsaferandmoreeasily programmablerobots,so‐calledcobots,thatwouldbe moresuitableforsmaller‐scaleapplications.However, theimplementationandcustomizationofcomplexrobot systemswithdifferentsensors,tools,externalsystems, etc.inparticularfortheusecasesofsmallandmedium‐sizedenterprises(SMEs)arestillofteninfeasible.There‐fore,thispaperdescribesthedesign,implementation, andtestingofaflexiblemobilemanipulatorarchitec‐turewiththefollowingthreeabstractionlayers.Ahigh‐levelsemanticlayermodelsandutilizesOWLontologies ofabstractmanufacturingprocesses,specificworkcell environments,linkedcontextknowledge,andthecurrent semanticworldstate.Anintermediatetranslationlayer collectsandexchangesdatafrommultiplesystemcom‐ponentsviaaunifiedinternaldatabase,whichisused, e.g.,incombinationwithbehaviortreestoconvertsym‐bolic,high‐levelactionsintomultipleparameter‐driven low‐levelcommands.Alow‐levelcontrollayerisimple‐mentedlocallyonamobilerobottoprovideaunified interfaceofitspotentiallycustomizedsubsystemstothe higherlayers.Thisarchitecturalapproachfacilitatesthe implementationofnewusecases,e.g.,byasysteminte‐grator,viaflexiblyadaptingsemanticrepresentations andbehaviortreesonthemodellevelwithoutchanging sourcecode.Theimplementedsystemwasdeployedin fivedifferentreal‐worldusecasesofindustrialpartners intheVOJEXTproject.Thispaperfocusesontheusecase ofplasterboardwallpreparationfromtheconstruction domain,whichincludestaping,spraying,andsanding operations,toevaluatetheparticularrequirementsof thisapplicationandhowtheycanbemetbytheproposed robotcontrolarchitecture.

Keywords: Knowledge‐basedsystem,Interfaceand translationlayer,Unifiedrobotcontrol,Semantic processdescriptions,Reusablemodels,Usecase adaption,Hardwareandsoftwareabstraction,Human workingenvironments,Buildingconstructiondomain, Mobilemanipulation

1.Introduction

IntheeraoftheIndustry4.0,automationofthe productionandhandlingofgoodsisanecessity,as thecompetitivemarketrequiresconstantcostopti‑ mizationsandef iciencyimprovements.Moreover,the labormarkethasbeenexperiencingashortageof skilledlaborworkers[6].Thissituationencourages anincreaseinautomationevenforenterpriseswhere robotizationisnotpresentorpoorlydevelopedto haverobotsorothermachinesperformatleastparts ofworkers’activities.However,robotsareperceived ascomplicatedandpotentiallyexpensivetointroduce tobusinesses,andforsomesectors,thepresenceof quali iedworkersisstillrequiredduringproduction processes,asitisverydif iculttofullyautomatethem. Insomesectors,likefooddeliveryandlogistics, thereisamovementtowardsaffordableautoma‑ tion[21],butmanyothers,suchastheconstruction domain,donothaveeasilyavailablesolutionsfortheir typeofoperation.Multiplefundingopportunitiesare availabletomodernizethoseenterprises,butwithout well‑establishedsystemdesignsandparadigms,mod‑ ernizationprogressesslowly.

Totacklethisproblem,highly lexibleandmodular solutionsareneeded,thatareabletoadjusttospeci ic needsindifferentindustrialsectors.Thisissuewas addressedbythe ValueOfJointEXperimentationindig‑ italTechnologiesformanufacturingandconstruction (VOJEXT)project(https://vojext.eu),whichaimedat providingasolutionthatusesrobots,avarietyofsen‑ sors,perceptionmodels,andadvancedcontrolsys‑ temstoenabledynamicdecisionsmadebycognitive subsystemsandinteractionwithhumanworkers.The maintargetsofthecreatedsolutionarenon‑robotic companies,includingbothsmallartisanworkshops andlargemultidisciplinaryenterprises,asitallowsto supportquali iedworkersinperformingtheirtasksby takingpartinmultipledifferentprocesses.Through boostingtheworkers’productivity,itmayhelptomiti‑ gatethelaborgapandmakebusinessesmorecompet‑ itiveinthemarket.Additionally,useofmobilemanip‑ ulatorsprovidesmoreversatilityinhumanwork‑ ingenvironmentsthanimmobilesolutions,suchas roboticworkcellsorstationarymachines.

Therearemultiplecommercialsolutionsthataim tofacilitatethedeploymentofroboticsystemsby providingauni iedinterfaceforcontrollingcompat‑ iblehardwareandplanningtheexecutionoftasks byde iningsequencesofatomicactions,suchas SiemensSIMATICRobotIntegratorwithSIMATIC RobotLibrary[22],ArtiMindsRPS[3],orIntrinsic Flowstate[11].Thesekindsoftoolsusuallyprovide awiderangeofready‑to‑usesolutionsformostcom‑ monscenarios,whichcanbeadaptedtoaparticular scenariousingdedicatedGUIs.

However,theuseofsuchtoolsisusuallylimitedto compatiblehardware,otherdevicescannotbeeasily integrated.Moreover,mostcommercialsolutionsare basedonproprietary,monolithicclosedsoftwarethat cannotbeeasilyextendedbyendusers.Anotherissue isthelackofsupportforreasoning,i.e.,ratherthan settingagoalforthesystemalongwithprovidinga setofknowledgeaboutaprocess,anenduserhasto de inealloftheactivitiesperformedbythesystem inastep‑by‑stepmanner.Thisapproachforceshighly specializedsolutionsthatfocusonspeci icprocesses andcannotbeeasilytransferredtodifferentusecases. Moreover,itmayrequirespecializedknowledgeand becomeoverwhelminglycomplexinasystemwith multipleassets.

Asnoneoftheexistingsolutions itallofthe requirementsoftheVOJEXTproject,i.e.,a lexible roboticsystemthatisabletocollaboratewithahuman operator,capableofreasoning,independentofahard‑ warevendor,anddeployableinvariousscenarios,a novelsolutionwasdeveloped,basedontheproject partners’experienceintheirrespective ields.Itcon‑ sistsofthreemainlayers:aknowledge‑augmented high‑levelsemanticlayer,anintermediatecontroland translationlayer,andalow‑levelmobilerobotcon‑ trollayer.Theyenabletheplanningandexecution ofmobilemanipulationtasksnexttohumanwork‑ ersbasedonformallyrepresentedautomationknowl‑ edge,sensordata,andoperatorinstructions,e.g., viahandgesturesorinteractionswithaweb‑based human‑machineinterface(HMI).

Theproposedcontrolarchitecturewasimple‑ mented,integratedwiththerestoftheVOJEXTsystem, andtestedinmultipledistinctusecases:

1) Handlingandqualityassuranceoffoampillows (https://youtu.be/tyyFJBAa44Q)

2) Transportandcollaborativehandoverofobjects forlogisticswhileconnectedtoafactorysystem (https://youtu.be/‑Ppr0rwjU0A)

3) Sensordatacollectionforqualityassuranceinan automotiveOEMfactory (https://youtu.be/PLMMWsvYgJc)

4) Plasterboardwallpreparationinthebuildingcon‑ structiondomain (https://youtu.be/PLMMWsvYgJc)

5) Supportofworkersinartisan loortilesproduction (https://youtu.be/pdlQuxJiEbM)

Moreover,someofthecontrolsystemmodules weretestedforotheractivitiessuchasweldingmetal plates,pick‑and‑placeoperationsinacarpentershop, andmosaicassembly.

Thispaperpresentsthemulti‑layerarchitecture ofthedevelopedcontrolsystemanditsbene its,and assessesitsapplicabilitytotheusecaseofrobot‑based plasterboardwallpreparation.

Itisstructuredasfollows.Section 2 providesan overviewofthesystemarchitectureandthefollow‑ ingSections 3, 4,and 5 describethethreeessential abstractionlayersofourdesigninmoredetail.

Section 6 discussesthecharacteristicsofrobots compatiblewiththerequirementsoftheguidinguse caseofthiswork,explainstherequiredfunctionali‑ tiesofrobotstoworkeffectivelywiththepresented solution,andprovidesseveralexamples.Section 7 describestheexperimentscarriedoutaspartofthe plasterboardwallpreparationusecase,anddiscusses theintricaciesofadeploymentofourproposedsolu‑ tioninthisdomain.Finally,Section 8 concludesthis work.

2.SystemArchitecture

Thisworkaimstoenablecomplexautonomous robotsystemsformanufacturingandconstruction. Theproposedapproachwasrealizedbasedonthe systemarchitecturedepictedinFig.1,whichconsists ofthreeabstractionlayerstoenableseparationof concerns.Theselayersfocus,respectively,onseman‑ ticknowledgemodelingandreasoning,intermediate controlandtranslation,anduni iedinterfacestoboth generalhardwarecomponentsaswellascustomones developedforspecializedusecases.Eachlayerhasa primarymodulewithitsownscopedareaofactivities, encapsulatedlevelofdetails,andgeneralizedinter‑ facesofinteraction.

high-level semantic layer intermediate translation layer low-level control layer

high-levelactions andtheirsymbolic parametervalues

updatestocurrent high-levelsystem situation

HICEM

low-levelcommands andtheirnumeric parametervalues

low-level results

Otherhigh-level modules e.g.,aremotetriplestore oraKPIcalculation

Actionrequests e.g.,fromaGUIor externalsystems

Environmentdata e.g.,perception(object orgesturedetection)

Non-robotlow-level modules e.g.,sensors,actuators

Figure1. Overviewofthesystemarchitecturewithits threeabstractionlayersandtheirprimarymodules PEMKRE,HICEM,andLOCEM

Thehigh‑levelsemanticlayercontainsthe PEMKRE(Planning,ExecutionandManagement KnowledgeReasoningEngine),whichusesSemantic WebtechnologiessuchasOWLontologies[24]and SPARQLrequests[25].

Itmodelsmanufacturingandconstructionpro‑ cessesbycombiningandparameterizinghigh‑level actionssimilarlyto lowchartsaswellasthecurrent semanticworldstateoftherobotanditsworkarea.On request,itprovidesthenexthigh‑levelactionandits symbolicparametervaluestotheHICEM(High‑level ControlEngineModule),whilebeingagnostictohow theyareimplementedinthelowerlayers,andenables themodelingofnewusecasesintheOWLontolo‑ gieswithouthavingtochangeanyofthePEMKRE’s sourcecode.Forexample,inthesemanticlayer,new instancesofexistinghigh‑levelactiontypescouldbe combinedinanontology‑encodedprocesstosolvea requestedusecasewithouthavingtochangethelower layers.Conversely,thelowerlayersmayprovidean alternativeimplementationofanactiontype,aslong asitadherestothesameinterfacetypeandachieves thesamehigh‑leveleffect.Thislayermayalsocontain additionalmodules,suchasanembeddedorremote triplestore,i.e.,anRDFgraphdatabase[5],topersis‑ tentlystore,query,andupdatetheOWLontologies. Anotherhigh‑levelmodulemayautomaticallycalcu‑ latekeyperformanceindicators(KPIs)basedonan executedproductionrun’ssemanticallyloggedpro‑ cessdatainthetriplestore.

Theintermediatetranslationlayercontainsthe HICEM,whichusesonebehaviortreeperhigh‑level actionincombinationwithaninternaldatabaseto translateonehigh‑levelactionintomultiplelow‑level commandstodifferentlow‑levelmodules.Duringthis procedure,ittranslatesthesymbolicparameterspro‑ videdbythePEMKREmoduletonumericones,based ondatafrommultiplesources,e.g.,userrequestsfrom aGUI,detectionsfromadvancedperceptionmodules, ordatafromexternalsystemssuchasacompany’s database.

Thelow‑levelcontrollayercontainstheLOCEM (Low‑levelControlEngineModule),whichinternally usesdifferentprogramminglanguages,frameworks, andmiddlewarestoprovidehardwareabstractionand aconsistentcontrolinterfacetothehigherlayer,i.e., theHICEM.TheLOCEMisabletofocusononesubsys‑ tem,i.e.,therobotwithitsmobileplatform,articulated arm,gripperactuators,etc.,andimplementsboth atomicfunctionsaswellasspecializedonesfornew usecases.Hence,theLOCEMmayinternallyinclude additionalrobot‑relatedhardwareandsoftwarecom‑ ponentsthatarecloselyintegratedintooneofthese functions.Examplesincludevisualperception,e.g.,of dockingmarkersforthemobileplatform,orspecial‑ izedhardwareintegration,e.g.,ofacompressorfora sprayinggun.TheHICEMmaycommunicatedirectly withnon‑robotlow‑levelmodulesinthislayer,for instance,toreceivebasicsensorreadingsfroman industrialmachineortotriggerproductionofthenext partafterthepreviousonewasremovedbytherobot.

Thefollowingsectionsprovidedetailsaboutthe primarymoduleoneachindividuallayer.

3.PEMKRE

ThePEMKREisdirectlyconnectedtoatriple‑ store,whichpersistentlystoresOWLontologiesthat semanticallymodelarobotsystem’shigh‑levelpro‑ cess low,thecurrentworldstateinasymbolicman‑ ner,andrelevantcontextknowledge.Theseseman‑ ticdescriptionmodelsformallyrepresenttarget products,productionprocesses,andmanufacturing resourcesrelatedtoadomainandusecaseaccord‑ ingtothePPRparadigm[2].Thisprovidesauni ied andreusableknowledgerepresentationoftheiroften heterogeneousdatatoenableautonomousrobotsys‑ tems[4].

TheformalsemanticsofOWLontologiesenable thebuilt‑inreasonerofthetriplestoreemployedinthe systemtoautomaticallyextendtheresultingknowl‑ edgegraph’ssetofassertedstatementswithasetof inferredstatements.Byworkingonthismodel‑centric semanticlevelintheOWLontologies,newusecases canbeimplementedwithoutchangingthePEMKRE’s sourcecode.

CN=CondionalNode

FN=FinalNode

∃ =existenalquanﬁer

Figure2. Visualizationofanontology‐encodedabstract processwithtasks,controlnodes,andSPARQL‐based semanticconditionsforaplasterboardtapingprocess. Parametersandsemanticeffectsoftasksarenotshown

Fig. 2 showsthetask‑levelinstancesandtheir relationsinanabstractprocessontologyforthe irst ofthethreeparts(i.e.,taping,spraying,sanding)in theplasterboardwallpreparationusecase(seeSec‑ tion7).Suchabstractprocessesmodelthework lowof arobotsystemsimilarlyto lowcharts,whicharealso intuitivelyunderstandabletonon‑roboticsexperts.

Abstractprocessesareabstractionsthataredis‑ tinctfromboththecurrentworldstateandthe multipleexecutionsofasinglework lowovertime. Theseabstractprocessesorothersemanticmodels canbegeneratedautomatically[15]orde inedin anintuitiveGUI[16],butthisisout‑of‑scopefor thiswork.Byusingthesemanticquerylanguage SPARQL,branchingconditionsaremodeledwithin theontologyasrequeststothecurrentsemantic worldstateoftherobotsystem[13].ThePEMKRE evaluatesthemdynamicallyatruntime,whilealso mappingobject‑levelabstracttaskparametersto theactualinstancesinthecurrentsemanticworld state,togenerateanexecutablespeci icprocess ontology.

TheabstractprocessinFig. 2 isabouttheauto‑ matedtapingofplasterboardwalljointlines.The otherabstractprocessesinthisusecaseforthespray‑ ingandsandingofthecompoundonthewallareas followthesamestructure,onlywithdifferenttaskand parametertypes.Therobot’s irsttaskintheabstract processistoscantheenvironmentforlinesandareas toaddthesenewwallfeatureinstancestothecurrent semanticworldstate.Iftherearestillanyunprocessed wallfeaturesleftinthesemanticworldstatefrompre‑ viousprocessexecutions,therobotcanskipthistask forthemoment.Iftherearenoneatallafterthescan‑ ning,themobilerobotmovestoahomelocationand theprocessendsearly.Otherwise,therobotsystem getsthetapingtool.Fromtheperspectiveofthehigh‑ levelsemanticlayer,theabstractprocessisagnostic inregardtohowthisisdone.Duringinitialdevelop‑ ment,therobotsystemmayrequesttheassistanceof ahumanoperatorforthis,butinthefutureitmay insteadbeequippedwithanautomatictoolchanger. Aftergettingthetool,theabstractprocessentersits mainloop.There,ifthe validateLines optionissetto true,therobot irstperformsa“dryrun”ofthetaping forthenextwalljointlinesothatahumanoperator cancheckitbeforehand.Iftheoperatorindicatesto therobotthatthelinewasvalid,itactuallytapesthe line,andifnot,pausessothattheoperatorcantape thisparticularlinemanually.Oncealllinesaretaped, therobotmovestoahomelocationandtheprocess ends.

Duringprocessexecution,thePEMKREprovides severalservicestotheHICEM.Theyallowto,e.g., parameterizeandevaluateprede inedSPARQL requestsstoredwithintheOWLontologies,getthe nexthigh‑levelactionanditsparametervaluesby evaluatingtheontology‑encodedprocess,orsetanew statusforthecurrenttask.ThePEMKREautomatically derivesahigh‑levelaction’sparameterizationfrom theobject‑leveltaskparametersinthecorresponding speci icprocessdescriptionandlinkedcontext knowledge.

HDH(1)

Read/Writedata (non-volatile)

Goals,resultsofactions

HEMcore (5) HDICT(3) HDM(2)

HEM(5)

Project-specific adapter

HHLC(4) Project-specificadapter

HHLCcore (4)

Figure3. InternalarchitectureoftheHICEMandthe dataflowbetweenitscomponents

ItistransmittedasanN‑Triples‑formattedstring, whichalsoenablestheHICEMandthePEMKRE’s semanticdescriptionmodelstoimplementnewaction typeswithoutchangingtheircommoninterface.

4.HICEM

4.1.HICEMConcept

ThemainconceptoftheHICEMistointerfacetwo conceptuallyindependentpartsofasystem:thehigh‑ levelsemanticlayerandthelow‑levelcontrollayer (i.e.,therobotsubsystemandotherlow‑levelmodules suchasexternalsensorsoractuators)byproviding twomaincapabilities.

First,basedonhigh‑leveldecisionsandfactors thatin luencethesystem’sstate,themodulecanexe‑ cutealow‑levelplanusingbehaviortrees[9, 12], whicharemodelsusedtoplanandexecutedetermin‑ isticsequencesofactions.

Theyarerepresentedinahierarchicaltree,where eachnodede inesaspeci icbehavior.Thetreeistra‑ versedfromtheroottotheleaves,withthecurrent stateofthesystemguidingthe lowthroughthenodes.

Second,themodulecollectsimportantdatafor bothlayersandmaintainsitstranslatability.The semanticlayerhastodealwithonlythedatathatis relevanttoit,suchasidenti iersofobjectsdetected byasensororhuman‑readabletagsforrobotposes. Basedonahigh‑levelactionreceivedfromit,the HICEMselectsallthedatathatisrelevanttothelow‑ levelcontrollayerofthesystem,suchastheposeof adetectedobject,andexecutesacon iguredsequence oflow‑levelcommands.

4.2.HICEMInternalArchitecture

TheHICEMisaROS‑based[14]implementationof acontroller,whichisbuiltontopofagenericlibrary

calledtheHICEMCore.Itissplitintofunctionalparts, asshowninFig.3,thatprovideuniversalmechanisms forlow‑levelcommandsplanning,actionexecution, monitoringofsystemhealth,datacollection,anddata storage.Thesecomponentsaredescribedinthefol‑ lowingparagraphs.

HDH,HICEMDataHandler,(1)inFig. 3 Itconsistsof theHICEMDataHandlerCoreanditsadapter,which providesusecasespeci icinterfaces(e.g.,ROStopics andservices).Itcollectsdatafromothersystemmod‑ ulesandconvertsitintoaformatthatallowstostore itintheHDMandHDICT.Italsocollectssystemhealth dataandsuppliesittotheHEM.

HDM,HICEMDataManager,(2)inFig. 3 Itisaninter‑ nalcomponentthathandlesinternaldatastorage.It implementsasimplevolatiledatabase,whichstores datauntiltheHICEMisclosedandmanagesthenon‑ volatiledatabase(HDICT)byprovidinganinterfaceto itfortheotherpartsoftheHICEMandorganizingthe datainit,e.g.,byremovingoutdatedentries.

HDICT,HICEMDictionary,(3)inFig. 3 Itisaninternal componentthatimplementsanon‑volatiledatabase tostoredatarelevanttotheHICEMandisusedto preservedatabetweenitsruns.Itprovidesaninter‑ facefortheHDMtoadd,remove,update,andread data,independentlyfromthedatabaseimplementa‑ tion.Itmaystore,e.g.,systemcon iguration,environ‑ mentdescription,stateofthesystem,ahigh‑levelgoal, orafeedbackfromasafetymodule.

HHLC,HICEMHighLevelController,(4)inFig. 3 It consistsoftheHICEMHighLevelControllerCoreand itsadapter,whichprovidesusecasespeci icinter‑ faces(e.g.,ROStopicsandservices).Itimplements aninterfacetoahigh‑levellayerandalow‑levelcon‑ troller.TheHHLCcanaccesstheinternaldatabases (eitherthevolatileoneintheHDMorthenon‑volatile oneintheHDICT)viatheHDM.Italsocontainsa BehaviorTreeEnginethatisanimplementationofthe BehaviorTree.CPPlibrary[7].Thisengineprovidesa mechanismoflow‑levelactionsplanningusingbehav‑ iortrees.Thebehaviortreemodelsarecomposed ofatreeofnodes,whichcorrespondtogenericor usecasespeci icfunctions,eachresponsibleforan atomicaction,e.g.,executingalow‑levelcommand. Thebehaviortreenodesoftenusedatastoredinthe HDMtosettheargumentsofato‑be‑calledlow‑level command,changethetypeofcommand,orchangea sequenceofexecutednodes.

HEM,HICEMEnvironmentMonitor,(5)inFig. 3 Itis aprojectspeci icadapteroftheHICEMEnvironment Module.Itcontinuouslyreadsstatusesofallthemod‑ ulespresentinthesystemfromtheHDM,processes them,and,ifneeded,callstheHHLCtostopthecur‑ rentlyrunningbehaviortree.

Figure4. BehaviortreeoftheTAPE_LINEhigh‐level action

4.3.ExampleInteractionwiththePEMKREandLOCEM AsanexampleoftheHICEM’sinteractionwith theothertwoprimarymodulesinthiswork,theline tapingprocessfromoneoftheusecasesintheVOJEXT projectisdescribedinthefollowing.

Tostartaprocess,theHICEMneedsanexternal input,e.g.,fromanoperatorusingaGUI.Theinput mustcontaintheIRI(InternationalizedResource Identi ier)oftheprocessthatwillbestarted.Using thedatainput,theHICEMrequeststhePEMKREto initializetheprocessinthesemanticlayer,andthen requeststhenextactionforthesystemtoperform.The responsefromthePEMKRE,i.e.,ahigh‑levelaction withtherequiredparametervalues,ismatchedtoan existingbehaviortree,whichisinthisexamplethe onecorrespondingtothelinetaping,thatisloadedup afterwards.

Theresponsealsocontainsadditionalarguments, e.g.,theIDoftherobottobeusedandtheIDofthe linetobetaped,whichareusedasinputsforthe speci icbehaviortreenodes.Next,theHICEMnoti ies thePEMKREthatthereceivedactionisinprogressand executesthebehaviortree.

Notallofthenodesresultintheexecutionoflow‑ levelcommandsintheLOCEM,assomeofthemhandle internalHICEMdataprocessing.ForexampleinFig.4, “CheckHealth”checksthesystemhealthandifacru‑ cialmodulereportsanunrecoverableerror,theaction isaborted.“GetTapingPosition”mapsthearguments receivedfromthePEMKREtothenumericalvalues requiredbytheLOCEM,by(inthiscase)calculating theapproachpointfortherobottotapethelinebased onitsreceivedID.“TagIdNameToObjectId”loadsthe tapingargumentsrequiredbytheLOCEMfromthe HDM.

Usingthedatafromtheinitialsteps,theHICEM employstheactionclienttosendacommandtothe LOCEMthatcontainsthecommandstringandaddi‑ tionaldata.Afterwards,theHICEMwaitsuntilthe LOCEMreturnsaresponse.Onasuccessresponse,the behaviortree inishesassuccessful,andtheHICEM sendstothePEMKREthatthisparticularactionhas inishedasasuccess.Otherwise,thebehaviortree in‑ ishesasafailureandtheHICEMsendsthisresulttothe PEMKRE.Whileexecutingabehaviortree,HICEM’s internal lagsareconstantlymonitored,aswellas externalsignalsfromaGUI,e.g.,tostopthecurrent process.

5.LOCEM

TheLOCEMisanabstractionlayerofsoftwarethat actsasaninterfacebetweenlow‑levelrobotsystems andhigher‑levelcontrolsystems.TheLOCEMallows tointeractwiththecomplexcontrolstructureofrobot subsystemsinasimplemanner.

ThosesubsystemscanbedescribedasaPhysical InterfaceModule(PIM)andconsistof,e.g.,mobile basenavigationandlocalization,aroboticarmmove‑ mentsplanner,orasafetymodule.

Anexampleofthedata lowsintheLOCEMcanbe seeninFig.5.CommandsreceivedbytheLOCEMare sentasactionsviaoneoftheinterfacessupportedby theLOCEMandusedbytherobotsubsystems,which havemultipleinternalprotocols,messagestypes,and formats.TheatomicactionsshowninFig. 5 canbe, e.g.asimplemovementfrompointAtoB,changing adigitaloutputstate,orrequestinginformationfrom asensor.Thislayeralsohasthecapabilitytointer‑ actwithmultipleatomicactionsusingasingleROS actioncall.Thisfeatureisnotaprimaryfunctionof theLOCEM,butitmaybeusedtosimplifytasksthat arealwayssupposedtousethesamesetofatomic actions.

TheLOCEMconsistsofseveralsubcomponents thatcanworkindependently.The irstoneisaser‑ viceinterfacethatisdesignedforquickactionsthat donotrequirealongresponsetime.Thesecond oneisanactioninterfacethatisdesignedforlong‑ lastingactionsthatrequirestatusmonitoring,e.g.,for mobilebaseorroboticarmmovements.TheLOCEM alsoincludesthreemoresubcomponents:aprocedure serverinterfacethathassimilarfunctionalitytothe ROSactionserverinterfacebutisdesignedforlow bandwidthandslower‑in‑responseatomicactions,as wellasasubscriberinterfaceandpublisherinterface thatareusedtorespectfullylistentoandbroadcast informationwithoutcon irmation.

TheLOCEMacceptscommandssentbytheHICEM asstringswithargumentsthatareconvertedbythe subcomponentstoROSmessagetypesusedbythecor‑ respondingrobotsystemcomponents.Suchamessage correspondstoanamespeci iedatthebeginningof thestring.Theorderofthegivenargumentsandavail‑ ableoptionsarespeci iedinanApplicationProgram‑ mingInterface(API)documentation.AROStopic‑ basedLOCEMnodeprovidesconstantstatusfeedback duringexecutionandinformationabouttheresultofa inishedcommand.

Figure5. Invocationandmonitoringofatomicactions bytheLOCEMforonethehigher‐levelactionssentby theHICEM

AROSservice‑basedLOCEMnodeprovidesjust theresult.Thementionedsubcomponentsweredevel‑ oped,validated,anddemonstratedduringtheVOJEXT project.

TheLOCEMisprimarilydesignedtoworkwith RobotOperatingSystem(ROS)[15]basedrobotenvi‑ ronments.IntheROSecosystem,thetypesofthe transferreddataareessentialforcommunication betweenmicroservices.Frequently,thosetypesare custom,whichcreatesanecessitytoshareandmain‑ tainonbothendsup‑to‑datecustommessagepack‑ agesforeachROSnodethatanexternalsystemwants tointeractwith.Moreover,thisapproachforcesto exposepotentiallysensitivepartsofasystemtothe public.Tosolvethoseproblems,LOCEMnodesactas translatorforROSinterfaceswithonede inedmes‑ sagetype.Thisinterfaceuni icationisoneofthemain bene itsoftheLOCEMasitprovidessimplicityof usagewhilepreservingthesafetyofrobotsystems.

InformationreceivedbytheLOCEMcanbedirectly passedasROSmessagesforfurtherprocessing,such asanavigationgoalforamobileplatform.However, messagescanalsobetranslatedtoothercommunica‑ tionprotocols,suchasMODBUSforasafetysystemor TCP/IPforcommunicationwitharoboticarm.

Thismeansthatatargetatomicactiondoesnot needtobewritteninaspeci icprogramminglan‑ guage,framework,orenvironment.Hence,higher‑ levelcontrollayersalsoneedonlyonemiddleware regardlessofthetechnologiesusedinsideofarobot software.

6.CharacteristicsofCompatibleRobots

WithintheVOJEXTproject,thepresentedarchitec‑ turewasproventoworkcorrectlywithrobotsprovid‑ ingthefollowingprimaryfeatures:amobileplatform

withthecapabilitytoautonomouslyplanapathto agiventarget,aroboticarmwithsuf icientsizeand payloadforcompletingtheneededtasks,andsafety sensors.AnexampleofthisistheRB‑KAIROS+[19], whichisamobilemanipulatordesignedforindoor materialhandling.TherobotconsistsofanRB‑ KAIROS[18]mobileplatformwithamountedUni‑ versalRobotse‑Seriesroboticarm.Roboticarmsare usuallymountedstationarynexttomachinesorwork‑ stations,whichlimitstheirworkspacetoonlythat place.TheRB‑KAIROS+allowsfortheexpansionof therobotworkspaceastheroboticarmcanmove tomultiplelocationsthankstoitsmobilebase.This enablestherobotto,e.g.,operateonlargeparts orperformpick‑and‑placeoperationsinlargeareas. Thisautonomousmobilerobot(AMR)utilizesomni‑ directionalwheelsthatmakereachingtightspaces possible.Itisequippedwithsafetyscannersand othersafetymeasurestocomplywithrecentindus‑ trialstandards(EN60204‑1:2006/A1:2019,ENISO 12100:2010‑11,ENISO13849‑1:2015,ENISO13849‑ 2:2012,ENISO13850:2015,EN60204‑1:2006‑6,EN 1175‑1:1998+A1:2010,2006/42/EC).

Scalingsoftwaretodifferenthardwarecon igu‑ rationsischallengingandrequiressigni icanteffort. DuringthecourseoftheVOJEXTproject,thepresented architecture,includingtheLOCEM,wasdeployedon sevendifferentrobotsthatusedthreedifferenttypes ofsafetyscanners.Therewerealsotwodifferenttypes ofroboticarms(UR5eandUR10e)infourdifferent con igurations:singleroboticarminthemiddleof mobileplatform,singleroboticarminthemiddleof mobileplatformonpedestal,singleroboticarmin frontofmobileplatform,andtworoboticarmsin frontofmobileplatform.Allrobotswerebasedon onemobileplatformmodelwithmultiplehardware adjustmentstospeci icusecases.

Forallofthementionedrobots,thesystemarchi‑ tecture(seeSection2)withitsPEMKRE,HICEM,and LOCEMmodulesoperatedinthesameway.Moreover, thisarchitecturecanbetransferredtootherrobot modelswithlimitedeffortrequired,aslongastheir internalsubcomponentsprovidefunctionalitiessim‑ ilartoalreadyimplementedones.

AgoodexampleofthiscouldbetheRB‑ VOGUI+[20]robotthatisdesignedforoutdoor operation.Itusestypicalwheelswithoutdoortires mountedonasuspensionthatcanrotateeachwheel separately.ThismakesRB‑VOGUI+mobileplatform navigationcontrolsigni icantlydistinctfromthe RB‑KAIROS+.However,thesameLOCEMinterfaces couldberunontheRB‑VOGUI+robotasinprinciple itscontrollersprovidethesameatomicactions.

7.PlasterboardWallPreparationCaseStudy

Thissectiondescribesthedeploymentofthe implementedrobotsysteminoneoftheusecasesdur‑ ingtheVOJEXTproject,i.e.,thepreparationofplaster‑ boardwallsintheconstructiondomain.Thisusecase consistsofcoveringjointsbetweenplasterboardwalls withtape,sprayingcompoundonwalls,andsanding

thesurfaceoftheappliedcompoundtosmoothit afterithasdried.Alloftheseprocessesalsoinclude human‑robotcollaborationforchangingthetoolsof theroboticarm.

Therequirementsofthiswallpreparationusecase werede inedbyoneoftheVOJEXTproject’sindustrial partner—Acciona[1].Successfuloperationrequired thefollowingprocessgoalstobemet:

1) Propertoolselection (basedonprocesschosenbyworker)

2) Tapeapplicationonplasterboardwalljoints (requiredworkerevaluation)

3) Sprayingjointsandtapewithcompound

4) Sprayingwallwithcompound (requiredworkerevaluation)

5) Smoothingofdriedcompoundtoachieveauni‑ formlayerthickness

Toachievethegivengoals,theRB‑KAIROS+robot presentedintheprevioussectionwasequippedwith threedifferenttypesoftools.Compoundsprayingwas performedusingaGraco26C624sprayinggun[10]. Smoothingofappliedmaterialwasperformedusing anOnRobotsolution[13]thatwasavailableonthe market.Tapingofplasterboardwalljointsisnotcov‑ eredwellinresearchandmarket‑readysolutions (FerRobotics[8],StraubDesignCompany[23],Robo‑ Tape[17])intermsofroboticoperation,asthosetools

Figure6. RB‐KAIROS+robotanditscustomized equipmentforaplasterboardwallpreparationusecase duringtheVOJEXTproject

Figure7. Simplifiedexcerptsfromtheworkflowand dataflowwithinandbetweenthelayersforthetaping process.Thesprayingandsandingprocessesare performedanalogously

supportonlytapeswithawidthsmallerthan50mm orweredesignedforthick,morestifftapetypesthan theoneusedbyAccionaattheirconstructionsites. Therefore,availablerobot‑ready,tape‑applyingsolu‑ tionswerenotdesignedtoworkintheconditions speci iedforthisusecasebyAcciona.Fortheserea‑ sons,acustomtapingsolutionwasdevelopedforthis usecase.

Additionally,averticalpedestalwasmountedon themobileplatformtoincreasethereachofthe roboticarmtomeettherequirementsofthisusecase. Thecustomizedrobotanditsequipmentcanbeseen inFig.6

Themainobjectiveoftheperformedtestsisthe successfulcompletionofthetasks.Toachievethis goal,thesystemneedstobeabletocompletetasksin amostlyautonomousmanner.Thesystemwasrunin twodifferentenvironments:atanAccionaworkshop facilityandinaRobotniktestarea.Thisallowedto testtheperformanceoftheproposedsolutionwith varyingenvironmentalfactors.

Chronologicalsequenceofactionsrelatedto demonstrationrunsoftherobotperformingthe plasterboardwallpreparationprocesses

Eachofthetaskswasperformedinthewaypre‑ sentedinFig. 7 andFig. 8.Thelatter igurepresents alloperationsthatarepartoftheplasterboardwall preparation.Atthebeginningofitsindividualtaping, spraying,andsandingprocesses,therobotscans(if currentlyneeded)theenvironmentforjointlinesand areasonthesurroundingwallsandthepropertoolis selectedforthecurrentprocess.Next,thecorrespond‑ ingtoolismountedbyaworkerontherobot’send‑ effector(sub igure1).Thenthemobileplatformdrives toapositionwherethetapingcanstart(sub igure 2).Thesystemisabletoapplytapeonwalljoints automatically,butbeforeeachofthetapingoperations aworkermayoptionallyevaluatewhetherthejoint wouldbeproperlycoveredandwhethertherobotis wellpositionedtocompletethetask(sub igure3). Therobotcoversthejointwithtape(sub igure4)and spraysthecompoundonthewall(sub igure5).The lastpartoftheusecaseissandingthewalltomakeits surfaceeven(sub igure6).

Fig.7showsthelinevalidationtaskwithinthetap‑ ingprocessasanexample.ThePEMKREmoduleuses aprocessontologythatissimilartoa lowchartand combinesasetofrequiredtaskstoachievethepro‑ cess.ThePEMKREsendsa“VALIDATE_TAPE_LINE” high‑levelactionwiththeIDofthenextline(here“1”) fromitssemanticworldstate,whichisalsomodeled asanontology.Thebehaviortreede inedforthis actiontypeintheHICEMisexecuted,whichcallslow‑ levelcommandsintheLOCEM,whenrobotoperations arerequired.InthecasepresentedinFig. 7,those arethe“GOTO”and“UC4_APPLY_TAPE_START”com‑ mands.The irstcommandsendsthemobileplatform tocoordinatesinfrontofline“1”.Thosecoordinates werederivedbasedonapreviousenvironmentalscan andstoredintheHICEM.Thesecondcommandmoves theroboticarmtoapositionforevaluationbythe worker.Incaseofsuccessfulvalidation,theHICEM passesthepositiveresulttothePEMKRE,whichthen returnsthe“TAPE_LINE”actiontoproceedwiththe actualtaping.

Figure8.

Incaseoffailure,theworkermayperform thetapingofthislinemanuallyandthePEMKRE returnsthenext“VALIDATE_TAPE_LINE”action withtheIDofanotherline.TheLOCEMinterprets inputsfromtheHICEMandcallscorresponding interfaces,i.e.,thenavigationinterfaceforthe“GOTO” commandandthemanipulationinterfacesforthe “UC4_APPLY_TAPE_START”command.

Duringthetestsofthesystem,theusecasepart‑ nerAccionadecidedthatsprayingcompoundacross thewholewallsurfaceprovidessuf icientqualityto notrepeatthistaskseparatelyforboththejoints andthewholewallsurface.Usingthedesignedsys‑ temarchitecture,therobotmanagedtoalsosmooth thewallsurfacetoachievetherequiredthickness. ForthetestsperformedattheRobotniksite,a videofromanassociatedwebinarisavailableonline (https://youtu.be/PLMMWsvYgJc).Thesamehierar‑ chicalarchitecturewasalsosuccessfullyusedinthe otherusecasesde inedintheVOJEXTproject.Those usecasespresenteddifferentchallenges,suchas compatibilitywithanalreadyexistingfactoryman‑ agementsystem,integrationofexternaldeviceslike punching,injection,andmoldingmachines,aswellas workerstatedetection.

Successfulimplementationinavarietyofdifferent usecasescon irmsthecorrectoperationofthecontrol systemarchitecture.

8.Conclusion

Theproposedsolutionwassuccessfullyimple‑ mentedandtestedinabuildingconstructionuse case,whosedomainisparticularlyaffectedbythe skilledlaborshortageandlackofavailableautomation solutions.Thesolutionwasalsodeployedinother usecasesthatwereprovidedbyotherendusersin theVOJEXTprojectandachievedsuccessfulopera‑ tionsineachone.Theimplementationoftheseuse cases,whichusuallyinvolvedmultipleexternalsys‑ tems,advancedperception,andhumaninteraction, wasmadefeasiblebythe lexibleabstractionlayers oftherobotsystemarchitecture.Whileitwasnot demonstrated,inprinciple,thatthesolutionisscal‑ able,itcanbeeasilyappliedtoalargenumberofdif‑ ferentscenarios.Thepresentedarchitectureistrans‑ ferableandmaybeimplementedformultiplemobile manipulatorrobotswithahighlevelofhardwareand softwarecustomization.Thiswouldrequirefurther research,work,andinvolvementofwell‑established roboticsolutionprovidersandintegrators.

AUTHORS

ŁukaszGranat∗ –RobotnikAutomationS.L.,Ronda AugusteyLouisLumière8,46980ParqueTecnológico Paterna,Valencia,Spain,e‑mail:lgranat@robotnik.es. MichałBryła –ŁukasiewiczResearch Network–IndustrialResearchInstitutefor AutomationandMeasurementsPIAP,Al. Jerozolimskie202,02‑486Warsaw,Poland,e‑mail: michal.bryla@piap.lukasiewicz.gov.pl.

KubaKamiński –ŁukasiewiczResearch Network–IndustrialResearchInstitutefor AutomationandMeasurementsPIAP,Al. Jerozolimskie202,02‑486Warsaw,Poland,e‑mail: kuba.kaminski@piap.lukasiewicz.gov.pl.

FilipJędrzejczyk –ŁukasiewiczResearch Network–IndustrialResearchInstitutefor AutomationandMeasurementsPIAP,Al. Jerozolimskie202,02‑486Warsaw,Poland,e‑mail: ilip.jedrzejczyk@piap.lukasiewicz.gov.pl.

SławomirPuchalski –ŁukasiewiczResearch Network–IndustrialResearchInstitutefor AutomationandMeasurementsPIAP,Al. Jerozolimskie202,02‑486Warsaw,Poland,e‑mail: slawomir.puchalski@piap.lukasiewicz.gov.pl.

IngmarKessler –fortiss,ResearchInstituteofthe FreeStateofBavariaassociatedwithTechnicalUniver‑ sityofMunich,Guerickestraße25,80805München, Germany,e‑mail:ikessler@fortiss.org.

AlexanderPerzylo –fortiss,ResearchInstituteofthe FreeStateofBavariaassociatedwithTechnicalUniver‑ sityofMunich,Guerickestraße25,80805München, Germany,e‑mail:perzylo@fortiss.org.

ÁngelSoriano –RobotnikAutomationS.L., RondaAugusteyLouisLumière8,46980Parque TecnológicoPaterna,Valencia,Spain,e‑mail: asoriano@robotnik.es.

∗Correspondingauthor

ACKNOWLEDGEMENTS

Theresearchleadingtotheseresultshasreceived fundingfromtheEuropeanUnion’sHorizon2020 researchandinnovationprogram(H2020‑DT‑2019‑2) undergrantagreement952197(VOJEXT).

References

[1] Acciona.“ACCIONA|Businessasunusual”.https: //www.acciona.com.Accessed2024‑07‑23.

[2] A.F.Cutting‑Decelle,R.I.M.Young,J.J.Michel, R.Grangel,J.LeCardinal,andJ.P.Bourey,“ISO 15531MANDATE:AProduct‑process‑resource basedApproachforManagingModularity inProductionManagement”, Concurrent Engineering,vol.15,no.2,2007,217–235, 10.1177/1063293X07079329.

[3] ArtiMinds.“ArtiMindsRPS‑IntuitiveRobot SoftwareforAdvancedRobotics”. https://ww w.artiminds.com/robotics‑software‑and‑ services/robot‑programming‑suite‑basic. Accessed2024‑08‑01.

[4] A.Björkelund,H.Bruyninckx,J.Malec,K.Nils‑ son,andP.Nugues,“KnowledgeforIntelligent IndustrialRobots”.In: AAAISpringSymposium onDesigningIntelligentRobots:ReintegratingAI, vol.SS‑12‑02,2012,1–6,https://lup.lub.lu.se/ record/4679237

[5] R.Cyganiak,D.Wood,andM.Lanthaler.“RDF1.1 ConceptsandAbstractSyntax”.W3CRecommen‑ dation,W3C,2014. https://www.w3.org/TR/rd f11‑concepts.

[6] DirectorateGeneralforCommunicationofthe EuropeanCommission.“Tacklinglabourand skillsshortagesintheEU”. https://commissi on.europa.eu/news/tackling‑labour‑and‑skills‑ shortages‑eu‑2024‑03‑20_en.Accessed2024‑ 07‑23.

[7] D.Faconti.“BehaviorTree.CPP4.6.Frameworkto createBehaviorTrees”. https://github.com/Beh aviorTree/BehaviorTree.CPP/blob/2a8a226f bbd99f524f0796e0d8a3145773c61c06/REA DME.md.Accessed2024‑07‑23.

[8] FerRoboticsCompliantRobotTechnologyGmbH. “ActiveTapingKitATK,roboticmasking,tape applicator,endeffector”. https://www.ferrob otics.com/en/services/products/active‑taping‑ kit‑atk‑robotic‑masking‑tape‑applicator‑end‑ effector.Accessed2024‑07‑23.

[9] R.Ghzouli,T.Berger,E.B.Johnsen,S.Dragule, andA.Wąsowski,“Behaviortreesinaction:a studyofroboticsapplications”.In: Proceedingsof the13thACMSIGPLANInternationalConference onSoftwareLanguageEngineering,vol.1,2020, 196–209,10.1145/3426425.3426942.

[10] Graco. 311053L,Manual,AutomaticAirlessSpray Guns,Instructions,English,2020. https://www. graco.com/content/dam/graco/tech_documen ts/manuals/311/311053/historic/311053EN‑ L.pdf

[11] Intrinsic.“IntrinsicFlowstate”. https://www.in trinsic.ai/flowstate.Accessed2024‑08‑01.

[12] M.Iovino,E.Scukins,J.Styrud,P.Ogren,and C.Smith,“AsurveyofBehaviorTreesinrobotics andAI”, RoboticsandAutonomousSystems,vol. 154,2022,1–18,10.1016/j.robot.2022.104096.

[13] I.KesslerandA.Perzylo,“FlexibleModelingand ExecutionofSemanticManufacturingProcesses forRobotSystems”.In: 2024IEEE29thInterna‑ tionalConferenceonEmergingTechnologiesand FactoryAutomation(ETFA),vol.1,2024,1–8, 10.1109/ETFA61755.2024.10710791.

[14] OnRobot.“SanderToolforCollaborativeRobots |Sanding&Polishing|EOAT”. https://onrobo t.com/en/products/onrobot‑sander.Accessed 2024‑07‑23.

[15] OpenRobotics.“ROS‑RobotOperatingSystem”. https://www.ros.org.Accessed2024‑07‑23.

[16] A.Perzylo,I.Kessler,S.Profanter,andM.Rick‑ ert,“TowardaKnowledge‑BasedDataBack‑ boneforSeamlessDigitalEngineeringinSmart Factories”.In: 2020IEEE25thInternational ConferenceonEmergingTechnologiesandFac‑

toryAutomation(ETFA),vol.1,2020,164–171, 10.1109/ETFA46521.2020.9211943.

[17] A.Perzylo,N.Somani,S.Profanter,I.Kessler, M.Rickert,andA.Knoll,“Intuitiveinstructionof industrialrobots:Semanticprocessdescriptions forsmalllotproduction”.In: 2016IEEE/RSJ InternationalConferenceonIntelligentRobots andSystems(IROS),vol.1,2016,2293–2300, 10.1109/IROS.2016.7759358.

[18] RoboTape.“RoboTape:TapeBetter,Faster, Smarter”.https://robotape.com.Accessed2024‑ 07‑23.

[19] Robotnik.“RB‑KAIROSAMRwith250Kgof payloadandomnidirectionalwheelsforindoor applications”. https://robotnik.eu/products/m obile‑robots/rb‑kairos‑2/#rb‑kairos.Accessed 2024‑07‑23.

[20] Robotnik.“RB‑KAIROS+UR10eAutonomous MobileManipulatorforindoorapplicationswith a12,5kgpayloadonitsarm”. https://robotnik .eu/products/mobile‑robots/rb‑kairos‑2/#rb‑ kairos‑plus‑ur‑ten.Accessed2024‑07‑23.

[21] Robotnik.“RB‑VOGUI+MobileManipulatorfor R&Dapplicationsinoutdoorenvironments”. ht tps://robotnik.eu/products/mobile‑robots/rb‑ vogui‑en/#rb‑vogui‑plus.Accessed2024‑07‑ 23.

[22] J.Romeo.“RoboticsfortheRestofUs:Automa‑ tioninSmallandMidsizeBusinesses”. https: //www.robotics247.com/article/robotics_f or_the_rest_of_us_automation_in_small_and_mi dsize_businesses.Accessed2024‑07‑23.

[23] Siemens.“EasyrobotintegrationwithSIMATIC”. https://www.siemens.com/global/en/produc ts/automation/industry‑software/automation‑ software/tia‑portal/highlights/robot‑integrat ion.html.Accessed2024‑08‑01.

[24] StraubDesignCompany.“SD‑610Robot‑ MountedTapeApplicator”. https://straubdesi gn.com/new‑products/sd‑610‑robot‑mounted Accessed2024‑07‑23.

[25] W3COWLWorkingGroup.“OWL2WebOntol‑ ogyLanguageDocumentOverview(SecondEdi‑ tion)”.W3CRecommendation,W3C,December 2012. https://www.w3.org/TR/owl2‑overvi ew.

[26] W3CSPARQLWorkingGroup.“SPARQL1.1 Overview”.W3CRecommendation,W3C,2013. https://www.w3.org/TR/sparql11‑overview

DYNAMICS

Submitted:28th February2025;accepted:25th April2025

ÜnalDana,LeventÇetin

DOI:10.14313/jamris‐2026‐003

Abstract:

Theindustrialrobotsareusedindifferentareas,oneof theseareasistowingtrailer’s.Thisresearchproposesthe useofanomniwheelmobilerobot,designedspecifically fortowingpurposes,toenhancethetowingcapabilities. Thestudyfocusesonanalyzing,controlling,andtesting thedynamicsofatrailerundertheinfluenceofexternal towingforcesappliedbyanomniwheelmobilerobot. Thestudyincludesdynamicalanalysis,pathplanning, trackingcontrol,prototypedesign,tuningofselected actuators,andsimulationresultsoftheproposedomni‐wheeltractorrobot.Thedynamicequations,thepath planningalgorithm,thepathtrackingmethods,andthe PIDtuningofactuatorsaretestedinaMATLABSimulink Simscapeenvironmentusingthedesigned3Dmodels. Thesimulationresultsareusedtofine‐tunethesystem parametersandminimizeerrors.Improvedtuningof theseparametersleadstobettersimulationoutcomes. Thefinalsimulationresultsshowthatpositionalaccuracy reaches4.0cmRMSprecision(peakdeviations:7.6cm) overa6.63‐metercourse,andspeeddeviationsremain below%4.3.Angularcontrolmaintainsstabilitybycap‐pingerrorsat0.117rad(average:0.008rad)andlimiting transientovershootto%17.17foraggressivemaneuvers.

Keywords: omniwheelmobilerobot,dynamicalanalysis, trackingcontrol,pathplanning,PIDcontrol

1.Introduction

Theadvancementsinwarehouseroboticsover thepastdecadehavebeensigni icant[1],driving increasedattentiontowardscontrollingandevaluat‑ ingvariousmobilerobotdesignsforwarehouseenvi‑ ronments[2,3].Thisresearchfocusesprimarilyon keyareassuchasenergyef iciency,pathplanning [4,5],pathtracking[6,7],dynamicbehavioranalysis [4,8–10]andmechanicaldesignconsiderations[2,11]. Thesetopicsformthefoundationformobilerobotics, speci icallyforautonomoustowingoftrailersexploit‑ ingintransferringproductsbetweenwarehouseand manufacturingfacilitiesorviceversa.

Previousstudiesinmobileroboticshaveexamined trailersystems[12–14],whichsharemechanicalsimi‑ laritieswithnon‑holonomictwo‑wheelmobilerobots. Asigni icantportionoftheliteraturefocusesontrailer dynamicsandcontrolforpathtracking,oftenemploy‑ ingLyapunov‑basedapproachestoaddresstracking controlfornon‑holonomicwheeledtrailer’s[15,16].

Whilethesemethodsofferusefulinsights,theyelabo‑ ratelycalculaterequiredtorquesforactuatedwheels byimplementingdifferentialdrivemodelsconcerning onlytrailersandcombineddynamicsoftractorand trailer.Nevertheless,incontextofmobilizationand autonomyofalreadydeployedtrailers,thesestrate‑ giesbecomeinvalidbecauseofthefactthatonlytow‑ ingofunactuatedtrailersisfeasibleintheircontrols. Totowformotioncontrolofthetrailerwithnon‑ holonomicdynamics,atractorrobotshouldhavefully controlledactuationforplanarmotion.

Omniwheelrobots,knownfortheirmaneuverabil‑ ityandholonomickinematics,havebeenasigni icant focusofresearch.However,theircomplexkinematic anddynamicanalysespresentadditionalchallenges [8,11,17].Controlstrategiesforomniwheelrobots havebeensuccessfullytested,employingaPIDcon‑ trolmethodtoadjustwheelaccelerations[17],imple‑ mentingadaptivecontrollers[9].Thisapproachcal‑ culatesrequiredmotortorquesusinginversedynamic analysisandadjuststherobot’svelocitybasedonposi‑ tionerrors.Thedesiredvelocityandplannedpathare providedasinputbytheuser,withthesystemcon‑ trollingtherobotaccordingly.Althoughthismethod iseffective,thearticledoesnotspecifytheexactpath trackingorplanningalgorithmstestedalongsidethe controlsystem.

Pathplanningisacriticalaspectthatenables autonomousmovementinmobilerobotsandrequires anaccompanyingpathtrackingalgorithm[4,7].Dur‑ ingpathplanning,amobilerobottypicallyrelieson mapdata,whichmaybepreexistingoracquiredin real‑timeusingsensors.Thesedataareessentialfor computinganoptimalroutefromtheinitialpositionto thedestinationthroughpathplanningalgorithms.The resultingpathcanberepresentedasasetofdiscrete posesorasamathematicalfunction,bothofwhich requireatrackingalgorithmtofollowtheplanned route[5,6].Pathtrackingalgorithms,suchasthePure Pursuitmethod,determinethenextdestinationbased onalookaheaddistance.Thisapproachcontinuously adjuststherobot’strajectorybyselectingtheclosest pointalongthepaththatlieswithinthelookahead distance,ensuringsmoothandef icientnavigation.

Thisstudyaddressesthechallengeoftowinga trailerusinganomniwheelmobilerobot,offeringa uniqueapproachtothiscommonprobleminmobile robotics.

Whilepreviousresearchhasexaminedthedynam‑ icsofomniwheelrobots[8,11,17–19]andtrailer’s [4,7,10,13,19–22],thisstudyre‑examinesthemdueto changesintheproblem’srequirements.Itderivesthe dynamicsoftowingandtheforcesinvolved,providing newinsightsintotheirinteraction.

Thisstudyintroducesamethodusingcubic Bezierfunctionswithpurepursuittracking.This approachreducestheneedforexcessivemaneuvers andcorrectsrotationalerrors,makingitsuitablefor alltraileroperations.Thecontrolstrategyemploys dynamicanalysiswithLyapunovstabilitymethods. Thoughcommonlyusedfornon‑holonomicrobots, thismethodisadaptedherefortowingapplica‑ tions,withexperimentsconductedtoassessitseffec‑ tiveness.Thisresearchcontributessigni icantlyby enhancingwarehouseroboticsef iciencyintowing operations.Additionally,the indingsbene itanysys‑ temutilizingtowedtrailer’s,offeringvaluableinsights andmethodologiesforimprovedperformance.

2.DynamicalAnalysis

Theproposeddesignofthetowedtrailermobile robotconsistsoftwoparts:thenon‑holonomictrailer partandtheholonomicomniwheeltractorpart.The twopartsareconnectedtoeachotherusingarevolute joint,thisjointallowstherotationsonZdirectionand transmitsonlytheappliedforceswithoutde lection butnotthetorque.Toachievethedesiredposeofthe trailer,therequiredtowingforcesontherevolutejoint mustbecalculated.Then,thecorrespondingmotor torquesormotorspeedsofthetowingrobotmustbe determinedbasedonthecalculatedexternalforcesof thetrailer.

Thefollowingassumptionsandproblem‑based constraintswereconsideredthroughoutthewhole research.

Thetrailerhas3degreesoffreedom.

Slipbetweenthewheelsandthegroundisneglected.

Thecenterofmassofthetowingrobotisatthe geometricalcenterofitsbody,equidistantfromall threewheels,asshowninFigure1a.

Thecenterofmassofthetrailerislocatedbetween themiddleofthetwowheels,asshowninFigure1b.

2.1.TowingDynamics

Tomodelthedynamicsofthetrailer’smotionthe geometricalandmechanicalrelationshipbetweenthe trailer’swheelsandbodyframeshouldbeknown.All ofthenecessaryparametersarepresentedinFigure2.

Thetrailer’slinearvelocity �� isdeterminedas theaverageoftheleftandrightwheelvelocities �� and��,whiletheangularvelocity�� dependsonthe differencebetweenthesevelocitiesandtherobot’s wheelbase��.

Theserelationshipsareessentialfordetermin‑ inghowtherobot’swheelsmustmovetoachieve adesiredtrajectory.Speci ically,therelationship betweenthelinearandangularvelocitiesisgivenby Equation(1).

(1)

Furthermore,thereversecalculationallowsto determinetherequiredwheelvelocitiesforagivenlin‑ earandangularvelocity.Byrearrangingtheequations inEquation(1)thefollowingexpressionsfortheleft andrightwheelvelocitiesareobtainedasshownin Equation(2).

(2)

Thevelocitiesoftheleftandrightwheelsare relatedtotheangularvelocities�� and�� throughthe wheelradius��,asshowninEquation(3).

(3)

Toexpresstheangularvelocitiesintermsofthe trailer’slinearvelocity�� andangularvelocity��,the matrixequationinEquation(4)isderivedbyusing Equation(2)andEquation(3).

Thisequationallowstodeterminetheangular velocitiesoftheleftandrightwheelsbasedonthe trailer’slinearandangularvelocities.Therelation‑ shipsfortheangularaccelerationsofthewheelsin termsofthelinearandangularaccelerationsofthe trailerisexpressedinEquation(5)bytakingthe derivativeofEquation(4).

Figure1. 2Dillustrationof(a)omniwheelmobilerobot and(b)trailer

Figure2. Forces(a),torques(b)ontrailer

Theserelationshipshelpstounderstandhow changesinthetrailer’svelocityaffecttheangular accelerationsofthewheelsandbytakinginverseof thisequationwecanunderstandtheinverserelation‑ shipshownonEquation(6).

(6)

Theforcesexertedonthewheelsarerelatedto theangularaccelerationsofthewheelsthroughtheir momentsofinertia.Speci ically,theforces �� and �� arerelatedtotheangularaccelerations�� and�� by thesystemofequationsshowninEquation(7).

(7)

Equation(8)representsthebalanceofforcesand torquesactingonthetrailer,where

⃗ �� represents theexternaltorque,andthetermsinvolving ⃗ �� and ⃗ �� describethetorquecontributionsfromthewheels. Thisbalanceisfundamentalforcontrollingtherobot’s rotationalmotion.

(8)

Analysisoftheforcesalongthe ��‑axis,whichare in luencedbythewheelforcesandthetrailer’saccel‑ erationisgivenbyEquation(9).

(9)

Thisequationshowsthatthenetforcealongthe ��‑axisisthesumoftheforcesfromtheleftandright wheels,minustheexternalforce ��,whichispro‑ portionaltotherobot’schangeinvelocity.Theforce �� actingalongthe ��‑axiscanbederivedfromthe momentsofinertiaoftherobotandthegeometryof thesystem.ItisexpressedinEquation(10).

(10)

Further,theforce �� canalsobedescribedin termsoftheangularaccelerationsoftheleftandright wheels,asshowninEquation(11).

(11)

Thisshowshowthewheelinertiaandangular accelerationsin luencetheexternalforceactingon thetrailer.Theforcesalongthe��‑axisisasshownin Equation(12).

(12)

Thisequationshowsthatthenetforcealongthe�� axisdependsontheforcesfrombothwheelsandthe externalforce.Thetotalforcealongthe��‑axiscanalso beexpressedasinEquation(13).

(13)

Thisshowsthatthetotalforcealongthe��‑axisis thesumoftheforcesfromtheleftandrightwheels, plusthecontributionfromthetrailer’schangein velocity.Theforce�� canbeexpressedintermsofthe angularaccelerationsoftheleftandrightwheels,as showninEquation(14).

Theforce

Thisequationgivesamoredetailedrelationship betweentheforcesandtheangularaccelerationsof theleftandrightwheels.The inalequationofmodel isamatrixrepresentationoftheexternalforcesalong the ��‑and ��‑axesintermsoftheangularaccelera‑ tionsoftheleftandrightwheels.Thismatrixformis usefulfornumericalcomputationandsimulationof thetrailer’sdynamics.Thematrixequationisgivenin Equation(16).

2.2.OmniwheelRobotDynamics

Thedesignedomniwheelmobilerobot,usedfor towingthetrailer,calculatestheexternalforcesaffect‑ ingthetrailertodeterminetheneededequivalent motorangularvelocities.Inthiscontext,thegeometri‑ calrelationshipsbetweenthemotorvectorsresulting fromthemechanicalconnectionstothemobilerobot bodyplayacrucialrole.Theserelationshipsarefun‑ damentaltocalculatingthenecessaryvelocities.All requiredgeometricalandmechanicalparametersare speci iedintheconceptualdesigninFigure3. Thevelocityvectoroftheomniwheelrobot’scen‑ terofmass,denotedby ��,isgivenbythefollowing equation.

Equation(17)de inesthevelocityvector �� ofthe omniwheelrobot’scenterofmass,where

and

Figure3. Forces(a),torques(b)onomniwheelmobile robot

representthelinearvelocitiesalongthe �� and �� axes,and�� istheangularvelocityoftheomniwheel robot.

Tocalculatethelinearvelocityalongthe�� direc‑ tion��,weusethefollowingequation.

(18)

Equation(18)combinesthecontributionsofthe individualmotorvelocities ��1, ��2,and ��3,wherethe motorsarearrangedinatriangularcon iguration.The equationshowshowtheomniwheelrobot’s�� direc‑ tionvelocityisin luencedbythesemotorvelocities.

Similarly,thelinearvelocityalongthe�� direction �� iscomputedas:

(19)

Equation(19)showsthatthe�� directionvelocity �� dependsonthemotorvelocities��1 and��3,witha factorof√3fromthegeometryoftherobot’swheels.

(20)

Equation(20)providestheangularvelocityatthe robot’scenterofmassbasedonthesumofthemotor velocitiesandthedistance�� fromthecenterofmass tothewheels.

Theindividualmotorvelocities ��1, ��2,and ��3 are relatedtothemotorangularvelocities��1,��2,and��3 bythefollowingrelations.

(21)

Inequation(21), �� representstheradiusofthe omniwheels,andeachomniwheel’svelocityispropor‑ tionaltoitsangularvelocity.

Rearrangingthesystemofequationsforlinearand angularvelocitiestowritethematrixequationthat transformsthemotorangularvelocitiesintothelinear velocitiesoftheomniwheelrobot’scenterofmass.

Thetotaltorque �� actingontheomniwheel robotduetotheforcesexertedbythemotorscanbe expressedas:

(24)

Equation(24)describesthetotaltorqueacting ontherobot.Itincludesthecontributionsfromthe individualmotorforces

,and

,aswellasthe rotationalinertiaterm

(25)

Equations(25)and(26)calculatetheforcesalong the�� and�� axes.

Theforcesexertedbyeachmotorcanbecalculated as:

Inequation(27),theforcesgeneratedbytheindi‑ vidualmotorsarerelatedtothemotorangularaccel‑ erations��1,��2,and��3,anddependonthemomentof inertia�� andtheradius��.

Toexpresstheforces �� and �� intermsofthe motorangularaccelerations,weusethefollowing equations.

Equations(28)(29)and(30)areexpressedwith coef icientstoshortentheequations.

Theserelationshipscanbewritteninmatrixform:

Equation(22)expressestherelationshipbetween themotorangularvelocitiesandthelinearvelocities oftherobot’scenterofmassinmatrixform.This matrixallowsforthedirectcalculationoftherobot’s velocitiesfromthemotorangularvelocities.

Theinverseofthisrelationshipisfoundtocalcu‑ latethemotorangularvelocities ��1, ��2,and ��3 from thelinearvelocities��,��,and��

(23)

Equation(23)providestheinversematrixthat calculatesthemotorangularvelocitiesfromtheomni‑ wheelrobot’slinearvelocitiesandangularvelocity.

Equation(31)expressestheforcesandtorquesin termsofthemotorangularaccelerationsinmatrix form.

Thecoef icients�� arede inedasfollows:

Equations(32)through(36)de inethecoef icients ��.Theangularaccelerationsofthemotorscanbe expressedasafunctionoftheforcesandtorques.

Equation(37)istheinverseofEquation(31).This inverseequationcalculates ��1, ��2,and ��3 basedon theequivalentforcesandtorquesontherobot.The coef icients�� arede inedasfollows:

Equations(38)through(41)de inethecoef icients of��

2.3.MobileRobotIntegration

Theintegrationoftwosystems,thetrailerandthe omniwheelmobilerobot,canbemodeledusingthe dynamicanalysisofbothsystems.Whenthesesys‑ temsareintegratedasshownintheFigure4,theexter‑ nalforcesactingonthetrailer,denoted�� and��,are equaltothetractor’sresultingtrackingforces

and ��,asexpressedinEquation(28)andEquation(29).

BymultiplyingtheJacobianfromthedynamic modelsofboththetrailerandtheomniwheelmobile robot,wecanestablishtherelationshipbetweenthe wheelaccelerationsoftheomniwheelmobilerobot andthetrailerwheelvelocities.Theresultingequation isprovidedbelow,whichmodelsthisrelationship.

Equation(43)providesthematrix��,whichtakes intoaccountthephysicalpropertiesofthetrailer, includingmomentsofinertia �� and ��,thewheel radius��,andthedistance��betweenthewheels.This matrixprovidestherelationshipbetweenthetrailer’s wheelaccelerationsanditsdynamics.

Thematrix��,whichmodelsthedynamicproper‑ tiesoftheomniwheelmobilerobot,isde inedas:

Equation(42)de ineshowthewheelaccelera‑ tionsoftheomniwheelmobilerobotarerelatedto thetrailerwheelvelocitiesthroughtheproductofthe Jacobianmatrices�� and��

Thematrix ��,representingthetrailer’sdynamic model,isgivenby:

Equation(44)showsthematrix��,wheretheele‑ ments �� representcoef icientsthataccountforthe robot’sgeometryanddynamics,includingtheeffects offorcesandtorquesonthewheelaccelerations.This matrixisessentialintransformingthetrailer’swheel velocitiesintothemobilerobot’smotoraccelerations.

3.ControlMethod

Asitisshownindynamicalanalysis.Thecontrol taskformotioncontroloftrailerrealizedbyapplying thetowingforceswhicharegeneratedviatheomni‑ wheeltractorrobot.Therefore,controlrequirements aregivenas:pathplanningbetweenwaypointscon‑ cerningnon‑holonomictrailerkinematics,derivation ofrequiredforcesusingitsdifferentialkinematics basedcontrolstrategyandactuatorlevelcontrolof omniwheelrobotmotionforcontrolledtowing.

3.1.PathPlanning

Pathplanningintrailer‑tractorrobotsareessen‑ tialinordertoachieveautonomousmovement[4,5, 20].Thisautonomousmovementcanbeplannedin differentways,theycanaimchoosingthemostef i‑ cient,feasibleorwantedsetofmovementsforthis reasonthepathplanningofmobilerobotsisacom‑ plicatedandimportanttask.Pathplanningofmobile robotscanbedoneusingbothmathematicalfunctions andalgorithmsdependingonthepathplanningstrat‑ egy.Apathplanningstrategyforamobilerobotis oftendeterminedbyconsideringaplannedscenario orasetofmovements,theplannedscenarioincludes theworkenvironmentandalloftheconstraintsofthe mobilerobot.Anexampleforthisisifamobilerobotis inaclosedandblockedenvironment,themovementof themobilerobotisunconstrainedandfullautonomy ofthemobilerobotisdesiredfromthepathplanner, themostcommonpathplanningstrategiesuseslocal andglobalmaps[23,24]todeterminethemostef i‑ cientmovementsetsbyhelpofmanydifferentalgo‑ rithms[25–28].Thesepathplanningstrategieswill aimtobethemostbene icentforthatcertainhypo‑ theticalscenarios.Inthisresearchapathplanning methodisrequiredinordertotestthepathtracking successofthetrailertractorrobot.

Figure4. Mobilerobotintegratedtrailerrepresentation

Duetothisneed,ahypotheticalscenarioandall ofthede inedconstraintsofthetrailertractorrobot areidenti iedandbasedonthisthedesignedpath planningissimulatedandtestedinMATLABSimulink.

Inanidealenvironmentwithoutobstacles,the simplestmethodtoachievethedesiredposeisto movealongtheshortestlinearpath.However,this straightpathisnotsuitableforthegivenproblem becauseitwouldresultinrotationsaroundthecen‑ terofmassofthetrailer.Theserotationsaroundthe centerofmassofthetrailerishardertoapplybecause ofthenon‑holonomicstructureofthetrailer.Tosolve this,acubicBéziercurve(with��=3)isemployedto generateapaththatavoidstheserotations.Thegen‑ eralformoftheBéziercurve[29]isgivenbyEquation (45).

(45)

ThisequationexpressestheBéziercurveasa weightedsumofbasisfunctions �� (��),where �� are thecontrolpoints,and�� istheparameterthatvaries between0and1.Thissumproducesacurvethat interpolatesbetweenthecontrolpoints.

Thebasisfunctions��(��)fortheBéziercurveare de inedinEquation(46).

��(��)= �� (1−��)��−��,��=0,…,�� (46)

Thesefunctionsdeterminethein luenceofeach controlpointonthe inalcurve.Thecoef icientsare computedusingbinomialexpansions,ensuringthat thecurvesmoothlyinterpolatesbetweenthegiven controlpoints.

Algorithm1 PathGenerationProcedure

1: procedure PATHGENERATED(��,��)

2: ��0 ←[��(1);��(2)]

3: ��3 ← ��(1);��(2)

4: ��←norm(��3 −��0)

5: ��1 ←(��0 +0.5��)[cos(��(3)); sin(��(3))]

6: ��2 ←��3 −0.5�� cos(��(3)); sin(��(3))

7: ��←50

8: ��←linspace(0,1,��)

9: ��ℎ ��←zeros(2,��)

10: for ��=1to�� do

11: ��ℎ ��(∶,��)←(1−��(��))3��0 + 3(1−��(��))2��(��)��1 +3(1−��(��))��(��)2��2 +��(��)3��3

12: endfor

13: endprocedure

ForcubicBéziercurves,where��=3,thecurveis determinedbyfourcontrolpoints.Thepathplanning algorithmutilizestwocontrolpoints��1 and��2 along withtheinitialanddesiredposesofthetrailer.Control point��1 ispositionedtowardsthefrontofthetrailer’s initialpose,whilecontrolpoint ��2 isplacedinthe oppositedirection,towardsthefrontofthetrailer’s desiredposeasshownontheFigure5.

ThecubicBéziercurveensuresthatthetrailerfol‑ lowsasmoothpathwithoutexcessiverotationaround itscenterofmass,providinganef icientsolutionfor pathplanning.

Thisapproachavoidsthecomplexitiesofmore advancedalgorithmswhilestillachievingthedesired motion.Thegeneratedpathiscalculatedusingthe pathgenerationprocedure,describedinAlgorithm1.

3.2.PathTracking

Pathtrackingistheprocessthatallows autonomoussystemstotravelalongadesired trajectory.Theprocesstypicallyconsistsofseveral setsofrules;hence,thesearereferredtoas algorithms.

Thepathtrackingalgorithmsareresponsiblefor determiningthenextwaypointontheplannedpath andcalculatingthedesiredvelocitiestoreachtheway‑ point.Manydifferentpathtrackingalgorithmsexist, butoneofthemostsuitablealgorithmsforthetrailer tractorsystemisthepurepursuitalgorithm[6].

Thepurepursuitalgorithm,implementedinMAT‑ LABSimulink,usesaspeci icdistancecalledthelooka‑ headpointdistancetodecidethenextwaypoint,as showninFigure6.Thewaypointsaregeneratedfrom thepathplannerbydividingthegeneratedpathinto anumberofwaypointsandlistingthemfromstart to inish.Eachwaypointconsistsofthepositionand orientationofthevehicleintaskspace.Thenextway‑ pointisthefarthestpointwithintheradiusofthe lookaheaddistance.

Figure5. CubicBéziercurve

Figure6. Purepursuitalgorithm

Thewaypointsandtheactualposeofthevehicle areusedtocalculatethesteeringangle.Thealgorithm cangenerateaconstantlinearvelocity,andtheangu‑ larvelocitycanbeboundedtothedesiredlimits.

Thesuccessoftrajectorytrackingdependsonthe followingfactors:

Thenumberofwaypointsinthepathandthelooka‑ headdistanceareimportant.Withinthelookahead distanceradius,theremustalwaysbeatleastone waypoint.

Thevehicle’sdesiredlinearvelocityandangular velocityshouldbeproportionaltothelookahead distance.Alowerlookaheaddistancecausesthe vehicletooscillatearoundtheplannedpath,while ahigherlookaheaddistancecausesthevehicleto convergetotheplannedpathoveralongertime. Theparametersshouldbetunedwithaconstant distancebetweeneachwaypointforthealgorithm toworkregardlessoftheplannedpath.

Thealgorithmalwayshasaslighterrorduetothe lookaheaddistance;therefore,thetuningofparam‑ etersisdonebyconsideringoscillationsonthe plannedpath.

Thepurepursuitalgorithmdoesnotstopthevehicle atthelastwaypoint;therefore,astoppingalgorithm isneeded.

Algorithm2 PurePursuitPathTrackingAlgorithm

1: procedure PUREPURSUIT(��,��,��,��) 2: �� ←��

3: ��ℎ ←None

4: for each�� in�� do 5: ��←Dist(��,��)

6: if ��≥��and��<�� then 7: �� ←�� 8: ��ℎ ←�� 9: endif

10: endfor

11: ��←��ℎ(1)−��(1)

12: ��←��ℎ(2)−��(2)

13: ��←atan2(��,��)−��

14: ��← 2⋅sin(��) ��

15: ��←atan(��)

16: return ��

17: endprocedure

3.3.DynamicTrackingControl

Theproblemde initionstatesthatoneoftheobjec‑ tivesofthisstudyistocontrolthetrailerpose�� and velocity��,�� inthetaskspace.Thismeansthattrailer dynamicsneedtobecontrolledwithatrackingcontrol method.Thedynamictrackingcontrolmethodshould maintainthevelocityatthereferencedvaluewhile trackingcontinues,bycalculatingtheforcesortorques thatneedtobeapplied.

Thedynamictrackingcontrolequationsforthe traileraregivenby:

(47)

(48)

ThedynamictrackingcontrolbasedontheLya‑ punovfunction[15, 16]workswellforthenon‑ holonomicwheeledmobilerobot.Asshownabove Towingatrailerisanon‑holonomicmotioncon‑ trolproblemwhichresultsinnonlinearmathemat‑ icalmodel.Hence,Lyapunovcontroldesignmethod resultsindeployablecontrollawscompatiblewith kinematics,itisadoptedinourproblemasacomple‑ mentarypartofmotionplanning[15,16].InFigure8, thesimulationsetupfordynamictrackingcontrolis shown.

Where �� isthetrailer’sacceleration, �� isthe tangentialvelocitycontrolcoef icient,�� isthetrailer’s desiredtangentialvelocity,̇�� isthetrailer’sangular acceleration,�� isthetrailer’sdesiredangularveloc‑ ity,and �� istheangularvelocitycontrolcoef icient. TheEquations(47)and(48)describetheforcesthat needtobeappliedtomaintainthedesiredvelocity

Figure7. Purepursuitalgorithmexamplefor(a)low,(b) highlookaheaddistance

Figure8. Dynamictrackingcontrol

andorientationofthetrailer.Thecontrolcoef icients �� and�� aretunedtoensurethatthepathtracking isperformedsuccessfully.Themethodisbasedon Lyapunovstability,whichguaranteesthatthesystem willremainstablewhilefollowingthedesiredpath.

3.4.MotorTuning

TheactuatoristheselectedmotorPololu37Dx73L, 12V131:1gearmotorwithaninternal64CPRtwo‑ channelencoder.Thismotorhasagearboxinfront ofitandhasanencoderattheback,whichcauses theinaccuraciesintheencoderreadingstobeless effectiveonthemotortorqueorposition.

Thismotorisagrayboxsystemmodel,where systemmodelsarecalledgrayboxwhenthetheo‑ reticalstructureisknown,buttheparametersare unknown.Grayboxmodelsystemsaretheoretical whiteboxmodelsbecause,withproperdata,these systemparameterscanbeidenti ied,andthisprocess iscalledsystemidenti ication.

Thesystemidenti icationstartswithdataacqui‑ sition.TocollectdatafromthethreeDCgearmotors, asetupcontaininganNImyDAQcard,ArduinoMega, andL298Nmotordriverismade.Withthissetup, differentPWMvaluesaregiventothemotor,andtheir encoderdataarecollectedfromonechannel.With thehelpofamyDAQcardusedasaswitchanddata collector,thestartofthemotoriscoordinatedwiththe giveninputvoltage.Theencoderdataiscollectedwith asamplingrateof50kHz.Allofthecollectedsamples aregatheredfor5seconds.ThePWMvoltageisgiven inthe irstsecondandcutoffattheexactthirdsecond. AlloftheencoderdatacollectedisstoredinanExcel ile.Thiscollecteddatamustbeprocessedtobeused insystemidenti ication.

Thetheoreticalstructureofthissystemismost similartothetransferfunctionofabasicarmature‑ controlledDCmotor.Thearmature‑controlledDC motortransferfunctionhastheinputasvoltageand theoutputasangularvelocity.Systemidenti ication datacollectedfromencoderchannelsneedtobe convertedtomotorangularvelocities,andtheinput PWMshouldbeconvertedtotheequivalentvoltages. TheseconversionsaremadeinMATLABusingMAT‑ LABscripts.Theconversionforequivalentvoltageof PWMissimple:multiplyingthe12Vwiththeduty percentageworks.Theencoderdatacollectedpro‑ videstherotationinformation,witheachrisingpulse counted,and16countsequalsonerotation.Withthe timedatacollected,wecan indtheRPMvaluesby dividingthetotalrotationcountedoverseconds.Dis‑ turbedencoderdataiscleanedand ilteredwithalow‑ pass ilter.Thechangeofmotorangularvelocityand inputvoltageovertimeisgiveninFigure9

Thesedataconversionsaremadeforevery ive inputvoltagedutypercentages,andencoderdataare acquiredfromallthreemotors.Whentheconver‑ sionsaredone,thedataareusedintheMATLAB systemidenti icationtoolbox.Thetransferfunctionis obtainedusinganiterativecurve‑ ittingmethod.

TheSystemIdenti icationToolboxinMATLAB requirestheusertoknowthenumberofpolesand

Curvefittingofthetransferfunction

zerosofthetransferfunction.Therequiredpiecesof informationareknownsincethesystemisagraybox model,thesystemhastwopolesandnozerosbecause itworkssimilarlytobasicarmature‑controlledDC motors.

Figure 10 showsthecurve ittingofthetrans‑ ferfunction.Thecreatedtransferfunction’ssystem responseis%83.63accurateaccordingtothepro‑ cessedencoderdata.Themotortransferfunctionis known,whichmeanswecanuseotherMATLABtools to indtheoptimalPIDcoef icients.FindingthePID coef icientscanbedonewithanothertoolboxcalled PIDTuner.WhiletuningthePID,thecontroleffort shouldbesaturated.Thissaturationisnecessaryto applythesePIDvaluesinreal‑lifeapplicationsandto seethesameresultsinsimulation.Thecontroleffort isthevoltageinputofthissystem,whichmeansitcan onlybebetween12Vand‑12V.Theoutputcanonly bebetween1050rad/sand‑1050rad/s.Thesesatu‑ rationvaluescomefromthespecsofthecomponents used.Thesampletimeofthesimulationmustbethe sameasthesampletimeoftheusedhardware,and sincethehardwareisdigital,a ixed‑stepsolvershould beused.

Themotortransferfunctioniscontrolledusingthe tunedPIDparametersinFigure 11 12.Thecontrol systemreachesasteadystatein0.5seconds.Steady‑ stateerrorandovershootofthesystemresponseare zero.

Figure9. Inputvoltageandoutputvelocityagainsttime

Figure10.

Figure11. TestedmotortransferfunctionandPID parameters

Figure12. PIDcontrolledmotortransferfunction

Thismethodisappliedtoallthreemotors,and differentPIDparametersaretuned.Table1showsthe PIDparameterstunedforallthreemotors.Equation (50)describestheeffectofPIDcontrolparameterson thecontroleffort.

Table1. TunedPIDParameters

4.SimulationProcedureandResults

Theproposedmobilerobotisdrawnina3D CADprogram,andthemobilerobotisexportedto MATLABSimulinkusingSimscape.Simscapeisa Simulinkadd‑onthatcangeneratemechanicaland electromechanicalmodelsinsideasimulationworld, workingtogetherwithMATLAB.Thistoolallowsthis researchtobetestedinanidealandknownsimu‑ lationsetting.Thus,thealloweddegreesoffreedom oftheworld,thefrictioncoef icient,andmanyother variablescanbeadjusted.Additionally,simulation resultsoranyvariablecanbemeasured,viewed,and logged.

Beforesimulatingthefulltowingrobot,thetrailer dynamics,dynamictrackingcontrol,pathplanning, andpathtrackingaretestedusingasimpli iedSim‑ scapemodeldisplayedonFigure13 14thatincludes

onlythetrailerportionofthemechanicalsystem shownonFigure19.Thisapproachallowsforamore manageableanalysisbydividingtheproblemintotwo parts,facilitatingtheevaluationofthecontrolmeth‑ odsandalgorithmseffectiveness[30].

ThepathgeneratorsubsystemshowninFigure 15 usesthepseudocodeinAlgorithm 1 andcreates thepathofthecubicBézierfunction.Thepathis dividedintowaypointsandalookaheaddistanceis calculated.Thewaypointsarethenusedinthepath trackingalgorithmtocalculatethedesiredveloci‑ ties.Thedesiredoutputvelocitiesarethenusedin thedynamictrackingcontrolsubsystemshownin Figure 16.Thissubsystemusesthecontrolequa‑ tionsandproducesthedesiredexternaltowing forces.

TheSimscapeMultibodyLinkPluginexportsthe trailer’sgeometricalandphysicalattributestoMAT‑ LABSimulinkfromSolidWorksassemblyandsaves

Simulationblockdiagramoftrailercontrol

Figure14. TrailercontrolsimulationinMATLABSimulink

Figure15. Subsystemofpathgenerationandpath trackingalgorithm

Figure16. Dynamictrackingcontrolsubsystem

Figure13.

Figure17. SubsystemofthetrailermechanicalSimscape model

Figure18. Stoppingalgorithmofthetrailer

attributesinaMATLAB ile,Simscapeschematicin aSimulink ile,andmodelappearanceinSTLor STEPformat.Aftertheexport,somevariablescanbe changedtoadjustfortheassumptionsmadeforthe convenienceoftheanalysisinthe irstchapter.The Simscapemechanicalmodeliscreatedinanempty worldwhen irstexported.Themovementplane,main bodyconstraintswiththeworldframe,wheelcontact parameters,bodyframes,andsensorsareaddedtothe subsystemofthemechanicalSimscapemodelshown inFigure17.Thegeneratedpathisshowninsimula‑ tionFigures23 24withaspline.Theframecreatedon thetowingpointisusedforapplyingexternaltowing forces,andtheframeonthecenterofmassisusedfor positionandvelocitymeasurements.

Thepurepursuitalgorithmisinsuf icientwhenthe mobilerobotneedstostop.Therefore,thecurrentand inalpositionsofthetrailerareusedtocreateastop‑ pingalgorithm.Thedistancebetweenthecenterof massofthetrailerandthe inalwaypointiscalculated.

Figure19. Simulationstart,thetrailerisatthe beginningofthepath(a)isometricview,(b)frontview, (c)topview,(d)bottomview

Figure21. Subsystemofdynamictrackingcontrolof trailertowedbyomniwheelmobilerobot

Figure22. Simulationstart,themobilerobotandtrailer areatthebeginningofthepath(a)isometricview,(b) leftview,(c)topview,(d)bottomview

Whenthetraileriswithinaradiusof0.01mof the inalposition,thetrailerstopsmovingandthe simulationends.Figure 19 illustratesthesimulation settingsfromisometric,front,top,andbottomviews.

Afterthetrailercontrolsimulationapprovesthe successofthecontrolmethodsandalgorithms,the omniwheelmobilerobotisaddedtotheSimscape mechanicalmodel,andtheexternaltowingforces arereplacedwiththeinversedynamicsoftheomni‑ wheelmobilerobotasshowninFigure20.Thestop‑ ping,pathtracking,andpathplanningalgorithms remainthesame;therefore,theimplementationof themechanicalmodelandinversedynamicanalysisis simpler.

Figure21illustratesthesamemethodofdynamic trackingcontrolusedinFigure 16,buttheoverall dynamicsofthesystemhaschanged,sothecoef i‑ cients�� and�� areadjustedagain.

ThesimulationisshownonFigure22.Theanalysis ofthesimulationsaredonebyinvestigatingthegraphs generatedfromthesimulationdata.Thedynamic

Figure20. SimulationBlockDiagramofthetractorrobot

Figure23. BézierCurvepathanalysiswithtuned parameters

trackingcontrolparameters,pathtrackingparame‑ ters,implementationofselectedmotorstothesimula‑ tionandPIDtuningofthemotorsaredonebyanalyz‑ ingtheresults.Analysisofthesuccessofpathtracking requiresthesimulationpathtobegeneratedindif‑ ferentformssuchassquare,circleandarchimedian curvedpathsratherthanusingthepathgeneration algorithmtogenerateBéziercurvedpaths.

4.1.SimulationResultsforTunedDynamicTracking ControlParameters

Thetuningofthedynamictrackingcontrolparam‑ eters(��,��)aredonebyanalyzingthesettlingtime, risetime,overshootandsteadystateerrordatagained fromsimulations.Theparametervaluesarekeptwhen adesiredresultisreached.Inthiscasethecontrol parametersare�� =200and�� =300.

Figure 23 demonstratesthepathtakenbytrailer whenthetuneddynamictrackingcontrolparameters areused.Thepathtakenbythetrailerremainedcor‑ rect.ThetrailercontrolsimulationhasanRMSerror of29.7mm,andthetractorrobotssimulationhasa 3.23mmRMSerror.Althoughtheerrorseemslower inthetractorrobotsimulation,thepathtakenbythe trailerisclosertothedesiredpathinthetrailercontrol simulation.Thelookaheaddistanceinthetracking algorithmofbothsimulationswasthesame;therefore, thedifferencemustbeduetothedynamictracking controlcoef icients.Thereasonfortheerroronway‑ points49and50isthatinbothofthesimulations, thetrailerstartstoshiftitsorientationirrelevantto thepathplanningwhenthelookaheaddistanceradius cannotdetectanymorewaypointsahead.Thepath trackingalgorithmneedstobeadjustedsothatthe lastpointcanbedetected.Thesolutiontothepath trackingalgorithmistoadjustthelookaheaddistance accordingtothestoppingradiusinthestoppingalgo‑ rithm.

Thedynamictrackingcontrolmethodsuccessfully controlsthetrailerwiththedesiredvelocitiesinthe trailercontrolsimulationshowninFigures 24 and 25.Thetractorrobotsimulationiscontrolledwith thesamedynamictrackingcontrolmethod,butthe

Angularvelocityoftowedtrailer

resultshavehighsteady‑stateerrors.Thissteady‑state errorcanresultfromthehand‑tuneddynamictrack‑ ingcontrolparameters.Thesettlingtimeofthetrailer controlsimulationisfasterthanthetractorrobotsim‑ ulation.Thetrailercontrolsimulation’slinearveloc‑ itycontrolsareslowandrobustwithnoovershoot, whilethetractorrobotsimulation’slinearvelocity controlisfastandaggressiveandhasovershoots.The simulatedangularvelocitiesofbothsimulationshave dif icultysettling,andthedesiredangularvelocityis alwayschanging.Therapidchangeofthedesiredvalue causesthesimulatedvaluestohaveerrors.Thetractor robotrequiresafasterresponsetorapidchanges,or anothersolutionistoreducetherateofrapidchanges. Therefore,bettertuningofthedynamictrackingcon‑ trolcoef icientsisneededtoachievefastersettling times.

Figures 24 25 showsthemeasuredand desiredvelocitiesofthetrailerwhensimulation isstartedwiththetuneddynamictrackingcontrol parameters.

TheuseofcubicBéziercurvesforpathgeneration andtheimplementationofthepathtrackingalgorithm bothfunctionseffectively.Theresultingsimulation pathalignscloselywiththedesiredtrajectoryandthe trailersuccessfullyreachesitstargetposewithmini‑ maldiscrepancies.Bothsimulationsachievethesame

Figure24. Linearvelocityofthetowedtrailer

Figure25.

desiredposebyemployingidenticalpathplanningand trackingalgorithms.However,inthetrailercontrol simulation,thetrailerreachesthedesiredposemore quicklycomparedtothetractorrobot’ssimulation. Thisdifferencearisesfromvariationsinthetrailer’s averagespeed;speci ically,thesloweraveragespeed observedinthetractorrobotisattributedtosteady‑ stateerrorsandovershootswithinthecontrolmech‑ anism.AsshowninFigure 23,atthecoordinate[1, 1.6],thesimulationofthetractorrobotrevealsthatthe trailerdeviatesfromitsintendedpathby0.05meters. Thisdeviationpersistsuntilthetargetpointis inally reached.

Aftertuningthedynamictrackingcontrolcoef i‑ cients,thesimulationsarerepeated,andthedesired linearvelocityisreachedwitha3.96%steady‑state error,asshowninFigure24,whiletheerroroccurring inthedesiredangularvelocityisdecreased(Figure 25).Thetuneddynamictrackingcontrolcoef icients areusedinotherpaths,andthetrajectorycontrolis analyzed.

4.2.SimulationResultsforDifferentPaths

Thesuccessofthepathtrackingisfurtheranalyzed bygeneratingnon‑optimalpathsforpathtrackingin Simulink.Thetuneddynamictrackingcontrolparam‑ etersareusedandtheeffectoflookaheaddistance isanalyzedonasquarepath.Thelookaheaddistance isinitially0.25m,forthesquarepathanalysistwo lookaheaddistancesareexamined,0.75mand1.25m.

Figures 26 28 aretheresultofsimulationswhen thelookaheaddistanceis0.75m.

ThesquarepathanalysisinFigures 26 31 shows theimportanceofthelookaheaddistance.Errorson thetrackappearwhenasharpturnisaheadanda higherlookaheaddistanceisneededtotrackthepath. Thedesiredangularvelocitieschangewiththelooka‑ headdistanceinbothofthelookaheaddistances.

ThedesiredvelocitiesareachievedinFigures 27 and30,therefore,thepatherroronlyhappensbecause thedesiredangularvelocitiesareincorrect.Figures 29 31 aretheresultofsimulationswhenthelooka‑ headdistanceis1.25mandthetrailerisfollowinga squarepath.

Circularpathanalysisusesthesametunedparam‑ etersandanalyzesthebehaviorofthetrackingalgo‑ rithmincircularmotion.Figure32showsthecircular pathtakenbythetowedtrailerinsimulation.

ThecircularpathinFigure32istrackedwiththe sameparametersastheBéziercurvepathplanning simulations.Thecircularpathshouldhavea ixed

Figure26. Squarepathanalysis(D=0.75m)

Figure27. Linearvelocityoftowedtraileronsquare path

Figure28. Angularvelocityoftowedtraileronsquare path

Figure29. Squarepathanalysis(D=1.25m)

desiredangularvelocity,butthesmalloscillationsin Figure34areduetothelowresolutionofthepath.The desiredlinearvelocityisreachedwithinanacceptable marginoferror.

Thecircularpathandthearchimediancurvepath areanalyzedwiththetuneddynamictrackingcontrol parameters(�� =200and �� =300.)andtheinitial lookaheaddistance(0.25m).

TheArchimedeanspiralpathinFigure 35 is trackedwithanerroratthebeginningofthepath. Thecurveatthebeginningofthepathissharp,sothe maximumvalueoftheangularvelocityisdesiredin Figure37.Itisobservedthatthelimitangularvelocity of0.1rad/sisnotenoughtotaketheturnwithout errors.Evenwithalargercurvedturn,thetrailer inds itstrackandreachesthedesiredvelocityasshownin Figure36

4.3.SimulationResultsforImplementationofPIDCon‐trolledDCMotors

Thethreemotorsselectedforthetowingrobotare integratedinthesimulationinthissection.Inthesim‑ ulationtheestimatedtransferfunctionforthethree motorsareusedtosimulatetheelectromechanical

Linearvelocityoftowedtraileroncircular path

Figure34. Angularvelocityoftowedtraileroncircular path

effectmotors.Inordertosimulatetheelectromechan‑ icallimitsofthemotor,stalltorquesarede inedusing saturationblocks.PIDcontrolofthemotorscannow beimplementedinthesimulationwiththefoundPID parameters.

TheimplementedonSimulinksubsystemisshown inFigure39.

Figure30. Linearvelocityoftowedtraileronsquare path

Figure31. Angularvelocityoftowedtraileronsquare path

Figure32. Circularpathanalysis

Figure33.

Archimedeanspiralpathanalysis

Figure36. Linearvelocityoftowedtraileron Archimedeanspiralpath

Thecharacteristicsoftheselecteddcmotorsand thePIDcontrolparametersdeterminedwithtestsare integratedintothesimulationtoapplyafeasiblesim‑ ulationsettinginreallifeapplication,asshownin

Figures38 39.TheintegrationofthemotorsandPID controlinthesimulationcausedthetrailertohave smalloscillationsaroundthedesiredpathshownin

Figure 40.Thelinearandangularvelocitiesofthe trailerinFigures41 42areclosertothedesiredinputs whenthePIDcontrolisapplied.

Figure37. Simulatedvsdesiredangularvelocityof towedtraileronArchimedeanspiralpath

Figure38. SimulationblockdiagramincludingPID control

Figure39. Simulatingtheselectedmotors

4.4.ComparisonofResults

Theperformanceresultsofthesimulationsare presentedinTable2foreachstepofdifferenttuning andimplementedcontrolsystemfortractortrailer mobilerobot.Theperformancecriteriaforthesim‑ ulationresultsare;steadystatelinearvelocityerror (��ess,%)toanalyzetheclosenessofthe inallinear velocitytothedesiredvaluetheinordertotunethe

Figure35.

Table2. PerformanceComparisonofNon‐Tuned,Tuned,andMotorIntegratedPIDTunedControllers

Figure40. Béziercurvepathanalysiswith implementationofPIDcontrolleddcmotors

Figure41. LinearvelocityoftowedtraileronBézier curvepathwithimplementationofPIDcontrolleddc motors

controllers,linearvelocityovershoot(%)toanalyze theinitialmaximumlinearvelocityerrorofthemobile robotforthedeterminationofsuccessofthetuned

Figure42. AngularvelocityoftowedtraileronBézier curvepathwithimplementationofPIDcontrolleddc motors

controllers,steadystatetime(��ss,s)inordertodeter‑ minethetimethatthecontrollertakestobeinthe %5rangeofthedesiredoutput,maximumandaverage angularvelocityerror(��emax,��eavg,rad/s),execution time(s),trackingerror(Max,RMS,m)andtracked andplannedpathlength(m)todecideifthetracking controlalgorithmsandpathplanningareeffective. Eachperformancecriteriahasapurposeandeffects thetuningofcontrolparameter.Alloftheresults intheTable 2 aresimulatedusingthesamepath planningandpathtrackingalgorithmstoachievethe sameposefromthesameinitialconditionsandwith equalconditions.Thedifferencebetweennon‑tuned, tunedandmotorintegratedPIDtunedresultsis,non‑ tunedresultsrefertotheresultswhenthereisno PIDcontroller,dynamictrackingparameters�� is50 and �� is120andtheyarenottunedforintegrated trailertractormobilerobot,tunedresultsarefromthe Figures23 25whendynamictrackingparameters�� is200and �� is300withoutPIDcontrolandmotor integration,themotorintegratedPIDtunedresultsare fromtheFigures38 42whenthemotorstransferfunc‑ tionandthetunedPIDcontrollerisaddedtothecon‑ trolsystem,workingtogetherwiththetuneddynamic tracking.Accordingtotheseresultscommentsabout successofthemethodscanbemade.

Inordertocomparethesuccessofthesimu‑ latedcontrolmethod,resultsregardingcontrolper‑ formancecollectedfromreferencesthatusesdiffer‑ entmethodsforpathtrackingandcontroloftractor‑ trailermobilerobots[21,31,32].Themethodssuch asadaptivecontrol,backwardsmotioncontrol,LQR (LinearQuadraticRegulator)algorithm,modelpre‑ dictivecontrolandPIDcontrolareusedtocontrol thetrajectorysuccessfully.Thesemethodsarejust aseffectiveasthedynamictrackingcontrolandPID controlusedinthisstudy.Tomakethisclaim,the simulationandtestresultsofthesemethodsshownin different iguresarecomparedandanalyzed.

5.Conclusion

Thesimulationresultsshowthetrailercontrolled byexternalforcesonthejointsuccessfullytracks thegeneratedpath.Thesimulationresultsindicate thatthedesignedpathplanningalgorithmfortrailer dynamicsisworkingasintended.Pathplanningcan nowbedevelopedwhenthetrailerdetectsobstacles andthepathplanningisadjustedaccordingtothemap data.Thepurepursuittrackingalgorithmworkswell withtrailerdynamics.Thealgorithmshouldhavea built‑instopperwhenthelastwaypointisthenext targetdestinationstoppershouldactivate.Whenthe trailerstartstomoveawayfromthelastwaypoint, itmeansthatthetrailerhaspassedby,thestopper algorithmshouldactivateandstopthetrailerbyforce.

Thetrailer’sdynamictrackingcontrolisper‑ formedeffectivelyandasfortractorrobot,thecontrol systemisfastandaccurate.Steadystateerrorand settlingtimeofthetractorrobotcanbeimprovedby thebettertuningofthedynamictracking,PIDandpath

trackingcontrolparameters.Oscillationsofthesim‑ ulatedangularvelocitiescanbeweakenedbyfurther tuningofthedynamictrackingcontrollercoef icients orbyincreasingtheresolutionofthepathgeneration. Theoveralltrajectorycontroloftheomniwheeltrac‑ torrobotperformssuccessfully.

AUTHORS

ÜnalDana∗ –IzmirKâtipCelebiUniversity‑ BalatçıkMah.,Izmir,35620Türkiye,e‑mail: unal.dana@ikc.edu.tr,www.ikc.edu.tr. LeventÇetin –IzmirKâtipCelebiUniversi‑ tyBalatçıkMah.,Izmir,35620Türkiye,e‑mail: levent.cetin@ikc.edu.tr,www.ikc.edu.tr.

∗Correspondingauthor

ACKNOWLEDGEMENTS

ThisworkwassupportedbyIzmirKâtipÇelebiUni‑ versity(projectno:2024‑TYL‑FEBE‑0008)

References

[1] R.RajandA.Kos“Acomprehensivestudy ofmobilerobot:History,developments, applications,andfutureresearchperspectives” AppliedSciences vol.12,no.142022 doi:10.3390/app12146951.

[2] R.J.Barros,J.L.SilvaFilho,J.V.Neto,and T.P.Nascimento“Anopen‑designwarehouse mobilerobot”2020LatinAmericanRoboticsSym‑ posium(LARS),2020BrazilianSymposiumon Robotics(SBR)and2020WorkshoponRoboticsin Education(WRE) 2020pp.1–6.

[3] D.JainandM.Y.Sharma.“Adoptionofnext generationrobotics:Acasestudyonamazon” 2017.

[4] P.M.KolahiandM.Nazemizadeh“Nonlin‑ eardynamicmodelingoftractor‑trailermobile robotswithconsiderationofwheelsinertia andtheiroptimalpoint‑topointpathplanning” Meccanica vol.58,no.12023pp.245–253 doi:10.1007/s11012‑022‑01578‑6.

[5] G.Tian,F.Zhou,andB.Song“Béziercurvebased smoothpathplanningformobilerobot” Journal ofInformationandComputationalScience vol.8 2011.

[6] R.C.Coulter.“Implementationofthepurepur‑ suitpathtrackingalgorithm”1992.

[7] M.Sampei,T.Tamura,T.Itoh,andM.Nakamichi “Pathtrackingcontroloftrailer‑likemobile robot”ProceedingsIROS’91:IEEE/RSJInterna‑ tionalWorkshoponIntelligentRobotsandSys‑ tems 1991pp.193–198.

[8] Z.HendzelandM.Rykała“Modellingofdynamics ofawheeledmobilerobotwithmecanumwheels withtheuseoflagrangeequationsofthesecond kind” InternationalJournalofAppliedMechanics

andEngineering vol.22,no.12017pp.81–99 doi:10.1515/ijame‑2017‑0005.

[9] R.D.AhmadAbuHatab“Dynamicmodellingof differential‑drivemobilerobotsusinglagrange andnewton‑eulermethodologies:Auni ied framework” AdvancesinRobotics&Automation vol.02,no.022013doi:10.4172/2168‑ 9695.1000107.

[10] S.M.Yeşiloǧlu,H.Temeltaş,andO.Kaynak “Dynamicalmodelingoftwo‑wheeledcart: Cooperatingmanipulatorapproach” Conference Proceedings‑IEEEInternationalConferenceon Systems,ManandCybernetics vol.32005pp. 2008–2012doi:10.1109/icsmc.2005.1571443.

[11] C.A.Cáceres,J.M.Rosário,andD.Amaya“Design, simulation,andcontrolofanomnidirectional mobilerobot” InternationalReviewofMechani‑ calEngineering vol.12,no.42018pp.382–389 doi:10.15866/ireme.v12i4.13974.

[12] J.DavidandP.Manivannan“Controloftruck‑ trailermobilerobots:asurvey” IntelligentSer‑ viceRobotics vol.7,no.42014pp.245–258.

[13] W.Chung,M.Park,K.Yoo,J.I.Roh,andJ.Choi “Backward‑motioncontrolofamobilerobot withnpassiveoff‑hookedtrailers” Journalof MechanicalScienceandTechnology vol.25,no.11 2011pp.2895–2905doi:10.1007/s12206‑011‑ 0909‑7.

[14] F.LamirauxandJ.‑P.Laumond“Apracti‑ calapproachtofeedbackcontrolforamobile robotwithtrailer”Proceedings.1998IEEEinter‑ nationalconferenceonroboticsandautomation (Cat.No.98CH36146) vol.41998pp.3291–3296.

[15] R.Fareh,M.Saad,S.Khadraoui,andT.Rabie “Lyapunov‑basedtrackingcontrolfornonholo‑ nomicwheeledmobilerobot” InternationalJour‑ nalofElectrical,Computer,Energetic,Electronic andCommunicationEngineering vol.10,no.8 2016pp.1021–1026.

[16] S.Tzafestas Introductiontomobilerobotcontrol Elsevier2014.

[17] K.Watanabe,Y.Shiraishi,J.Tang,andS.G.Tzafes‑ tas“Autonomouscontrolforanomnidirectional mobilerobotwithfeedbackcontrolsystem” Pro‑ ceedingsofthe1999IEEEInternationalConfer‑ enceonRoboticsandAutomation 1999.

[18] A.Khadr“Multibodydynamicsmodelingand simulationofathree‑wheeledmobilerobot usingaroboticapproach” InternationalJournal ofIntelligentUnmannedSystems 2024.

[19] G.Campion,G.Bastin,andB.Dandrea‑Novel “Structuralpropertiesandclassi icationof kinematicanddynamicmodelsofwheeled mobilerobots” IEEETransactionsonRobotics andAutomation vol.12,no.11996pp.47–62 doi:10.1109/70.481750.

[20] A.B.Robat,K.Arezoo,K.Alipour,and B.Tarvirdizadeh“Dynamicsmodelingand

pathfollowingcontrolleroftractor‑trailer‑ wheeledrobotsconsideringwheelsslip.” ISA transactions 2024.

[21] N.T.Binh,N.A.Tung,D.P.Nam,andN.H.Quang “Anadaptivebacksteppingtrajectorytracking controlofatractortrailerwheeledmobilerobot” InternationalJournalofControl,Automationand Systems vol.172019pp.465–473.

[22] Z.Jin,C.Wang,D.Liang,S.Wang,andZ.Ding “Fixed‑timeconsensusformultipletractor‑ trailervehicleswithdynamicscontrol:A distributedinternalmodelapproach” IEEE TransactionsonIntelligentVehicles vol.92024 pp.656–669.

[23] J.Zhang,S.Singh,etal.“Loam:Lidarodometry andmappinginreal‑time.”Robotics:Scienceand systems(RSS) 2014pp.1–9.

[24] P.Marin‑Plaza,A.Hussein,D.Martin,and A.d.l.Escalera“Globalandlocalpathplanning studyinaros‑basedresearchplatformfor autonomousvehicles” JournalofAdvanced Transportation vol.2018,no.12018Art.no. 6392697.

[25] L.Liu,X.Wang,X.Yang,H.Liu,J.Li,andP.Wang “Pathplanningtechniquesformobilerobots: Reviewandprospect” ExpertSystemswithAppli‑ cations vol.2272023Art.no.120254.

[26] P.RajaandS.Pugazhenthi“Optimalpathplan‑ ningofmobilerobots:Areview” International journalofphysicalsciences vol.7,no.92012pp. 1314–1320.

[27] Z.TangandH.Ma“Anoverviewofpathplan‑ ningalgorithms”IOPconferenceseries:earthand environmentalscience vol.804,no.22021Art. no.022024.

[28] H.Qin,S.Shao,T.Wang,X.Yu,Y.Jiang,and Z.Cao“Reviewofautonomouspathplanning algorithmsformobilerobots” Drones vol.7,no. 32023211.

[29] S.R.Buss BézierCurves CambridgeUniversity Press2003pp.155–199.

[30] U.DanaandL.Çetin.“Omni‑wheeltractor robot”.Master’sthesis IzmirKatipCelebiUniver‑ sity Izmir,Turkey2024.

[31] G.LeiandY.Zheng“Researchoncooperative trajectoryplanningalgorithmbasedontractor‑ trailerwheeledrobot” IEEEAccess vol.102021 pp.64209–64221.

[32] E.Kayacan,E.Kayacan,H.Ramon,andW.Saeys “Robusttube‑baseddecentralizednonlinear modelpredictivecontrolofanautonomous tractor‑trailersystem” IEEE/ASMETransactions onMechatronics vol.20,no.12014pp.447–456.

ADAPTIVEUPPERLIMBROBOT‐ASSISTEDREHABILITATION: LEARN‐FROM‐THERAPISTDEMONSTRATIONS LEARN‐FROM‐THERAPISTDEMONSTRATIONS

ADAPTIVEUPPERLIMBROBOT‐ASSISTEDREHABILITATION: LEARN‐FROM‐THERAPISTDEMONSTRATIONS

Submitted:18th May2024;accepted:7th November2024

IsmailAuta,AhmedFares,HiroyasuIwata,HaithamEl‑Hussieny

DOI:10.14313/jamris‐2026‐004

Abstract:

Robotic‐assistedrehabilitationisapromisingmethodfor improvingmotorfunctioninindividualswithupperlimb impairments.However,generationofpersonalizedand adaptiveassistancepatternsremainsachallenge.Our studyintroducesaLearn‐from‐TherapistDemonstration (LfTD)framework,whichemploysDynamicMovement Primitives(DMP)totrainarobotarmtolearnfromthera‐pistskills.Therapistmovementswerecapturedviavisual tracking,andtheDMPaccuratelylearnedandreplicated thesemotionsusingaroboticarmtoassistpatients. Thesemovementsweretheneffectivelygeneralizedto newgoalswhilepreservingtheoriginalmotionpat‐terns.Meanwhile,aModelReferenceAdaptiveController (MRAC)hasbeenutilizedtorefinetherobot’sadaptive performancewhileensuringdemonstrationtracking.We assessedtheefficacyofLfTDwithasimulatedtwo‐link robot,whichdemonstratedpromisinglearning,adapta‐tion,andabilitytoperformcomplexrehabilitationtasks withprecisetrajectorytracking.Furthertestsevaluated therobustnessoftheMRACagainstintroduceddistur‐bancesthatmimickedpatientdeviations,demonstrating resilienceandadaptability.Thesefindingssuggestthat LfTDcouldenhanceupperlimbrobot‐assistedrehabil‐itationthroughprecise,adaptablemotionreplication, withfuturevalidation,includingtrialswithactualrobots, neededtosupporttheseresults.

Keywords: Personalizedrehabilitation,Learningfrom demonstrations,DynamicMovementPrimitives,Model adaptivecontrol

1.Introduction

Globally,strokeisamajorhealthproblemand isamongtheleadingcausesofdeathandlong‑term disability[1, 2].Anestimatein2019indicatesthat about101millionpeoplelivewiththeeffectsofstroke, andabout12.2millionnewcasesarereportedyearly [3].Withtheagingofpopulation,thenumberof expectedincidencesisprojectedtorise[4,5].After astroke,morethantwo‑thirdsofpatientsexperience impairedmotorfunctionintheirupperlimbs[6],and atsixmonthspost‑stroke,50%ofthepatientscon‑ tinuetoexperienceasigni icantlossoffunctionality [7].Thisimpairmentpresentsachallengingconse‑ quenceforstrokesurvivors,itparticularlyinterferes withtheirabilitytoperformdailytasksindependently [8].Therefore,asthenumberofsurvivorsincreases,

thedemandforeffectiverehabilitationservicesgrows signi icantly[9,10].Theseservicesareessentialfor assessingandimprovingmotorfunctionsbyproviding targetedtrainingduringthevitalearlyrecoveryphase, whichcanlastuptothreemonthspost‑stroke[11,12]. Additionally,thisarearemainsatopresearchpriority instrokerehabilitation[13,14].

Conventionalrehabilitationtechniques,including physicalandoccupationaltherapy,andvariousforms ofelectricalandsensorystimulation,oftenfailto restorefullmotorfunctionintheupperlimbof strokepatients[15].Thisisoftenattributedtoinsuf‑ icienttherapydoses,poorpatientengagement,and lackofobjectivefeedback[16].Theseapproaches donotinducetheneuroplasticchangesnecessary formaximummotorrecovery,suggestinganeedfor moreintensiverehabilitationstrategies[17].Robotic technologyintegratedintorehabilitationhasshown remarkablepotentialinenhancingrecoveryfollow‑ ingstroke[18].Forinstance,technologiessuchas exoskeletonsandend‑effectorrobotsaddressthese limitationsbyofferingaccuratesupportandprecise movementofthelimbs,signi icantlyimprovingfunc‑ tionaloutcomes[7,19].Systematicreviewsandmeta‑ analysesverifythatrobot‑assistedtherapyenhances armfunction,musclestrength,andactivitiesofdaily livingbypromotingintensive,repetitiveexercises [20–22].

ProblemStatement: Despitetheadvancementin roboticrehabilitation,somecriticalareasstillrequire furtherexplorationtoenhancetheef icacyofthese interventions.Currently,robotic‑assistedrehabilita‑ tionfacessigni icantchallenges,includingtheneed forpersonalizationtoaddressindividualdifferences amongpatients[23],andthenecessityforadaptabil‑ itytoensurethattherapiescanadjusttopatients’ progressandspeci icneedsinreal‑time[24].This adaptabilityprovidesassistancewhenitismost neededtooptimizerecoveryandtofacilitatepatient engagementduringtherapysessions,particularlydur‑ ingthecritical irstthreemonthspost‑stroke,which areimportantforrecovery[6,25].

Acommonlimitationofcurrentrehabilitation robotsistheirlimitedgeneralizabilitytoabroader patientpopulation[26].Thisisbecausestandardized treatmentsarenotoptimallyeffectiveforeveryone, andeachpatientmayrequiredistinctlevelsofassis‑ tanceandtypesofmovementtherapy.

Thismakesitdif icultforasingleroboticdesign toeffectivelyserveallpotentialusers.Additionally,a therapistisrequiredtoprovidedemonstrationsfor eachtask,whichmaynotalwaysbefeasiblewhere therapisttimeisscarce.Thisdependencycanlimit thepracticalityandscalabilityofroboticrehabilita‑ tionsystemswhereconsistenttherapistavailability maynotbefeasible.Finally,patient’sfeedbackisnot takenintoaccount,whichmaylimititsabilitytoadapt tothepatient’sneeds.Addressingthesechallenges requiresthedevelopmentofrehabilitationrobotsthat arecapableofblendingpatientneedsandrobotfunc‑ tionalitiestoprovideeffectiveandpromptresponses totherapeuticinterventions.

Avarietyofmethodsandtechnologieshavebeen developedtoenhancethefunctionalityoftheupper limb.Amongtheseareexoskeletonrobots,designedto mimictheskeletalstructureofpatient’slimb,andend‑ effectorrobots,whichinteractdirectlywiththelimb throughthemovementofitsend‑effector.

Theserobotictechnologiesarecategorizedbased onthetypesofassistancetheyprovide,whichinclude active,passive,andhaptic[27].Activeassistanceis employedtosupportpatientswhoretainsomemotor functionality;inthismode,patientsactivelyengagein rehabilitationexercisesalongsidetherobot.Incon‑ trast,passiveassistancedoesnotrequireanyeffort fromthepatient,astherobotperformsthemove‑ mentsindependently.Hapticassistance,meanwhile, involvessensorystimulation,oftenthroughtouch, thatcanbeeitheractiveorpassive.Recently,haptic deviceshavebeenintegratedwithvirtualreality(VR) tocreaterehabilitationscenariosthatprovidesen‑ soryfeedback,enhancingthetherapeuticexperience [1,10,15].

Atypicalapproachtomodelingrehabilitation exercisesinvolvestheuseofprobabilisticmodels.For instance,[28]and[29]presentarehabilitationtech‑ niquethatincorporatesbothforceandimpedance‑ basedbehavioursfromthepatient,theyutilizeGaus‑ sianMixtureModels(GMM)withGaussianMixture Regression(GMR)todevelopageneralizedmodel basedonreal‑timepatientresponses.In[28],a dynamicbicyclecrankingmodelisusedtoadjust thelevelofassistanceprovidedaccordingtother‑ apistperformanceindemonstratingdifferentsub‑ tasksatapatient‑speci icbasis.[30]employedGMM andGMRforroboticassistanceinplayactivitiesfor childrenwithcerebralpalsy.Theyapplieditina two‑dimensionalpick‑and‑placetaskutilizingmaster‑ slaveteleoperationsystem.Theirmajordrawbackis theinabilitytoadapttopatientvariabilityandto externaldisturbances.

Recentdevelopmentshaveincorporatedmachine learningtechniques.Intheirstudy,[31]developedan IntelligentAssistantforRoboticTherapy(iART)using LongShort‑TermMemory(LSTM)networksthathelps inthereplicationoftherapistbehavioursinassisting tasks,involvingtrackingcomplexthree‑dimensional trajectory.

In[32],afeedforwardneuralnetworkcombined withVR‑basedhapticsisusedtomodelandregenerate therapeuticrehabilitationstrategiesinitiallydemon‑ stratedbytherapists,enablingthemtoremotely demonstrateandmanagerehabilitationthroughtele‑ operationandfacilitatingongoingtherapyevenwhen theyarenotphysicallypresent.However,adapting thesesystemstothewidevariabilityinpatientcon‑ ditionsposesanotherchallenge.

Movingbeyondthesetechniques,DynamicMove‑ mentPrimitives(DMP)offeranotherapproachin roboticrehabilitation,widelyrecognizedfortheirabil‑ itytomodelcomplexmovementpatterns.Theyhave beenappliedacrossvarious ields,showingsigni icant resultsinenhancingroboticfunctionalityandinterac‑ tion.[33]hasinvestigatedtheuseofDMPbyproviding amethodtoteachrobotshowtoreplicateobserved actionsandadapttonewgoalssimplybyadjusting startandgoalparameters,makingthesystemhighly adaptabletodifferentscenarios.Theresultsshow‑ casedincludesuccessfulimplementationonaSarcos robotarm,wheretherobotperformedtaskslikepick‑ and‑placeandwater‑serving,demonstratingsigni i‑ cantgeneralizationcapabilitiesinnovelsituations.

[34]proposesanupperlimbrehabilitationrobot thatperformstask‑orientedexercisesbyrecognizing objectsandgeneratingtrajectoriesforreachingthem. TheDMP‑basedmotionplannerusedinthissystem successfullyreplicatesthemotionstyleofhealthy subjectsandachievestargetpositionswithminimal error.Similarly,[35]developedamotionplanning systembasedontheDMPandvalidateditusingthe KukaLWR4+roboticarm.Eighthealthysubjectswere recruitedtoperformaseriesoftaskssuchas:pouring, drinking,andeatingwhiletheirwristswereattached totherobot’sendeffector.Thesystemaccurately mirrorsandgeneralizesthepersonalmotionstyleof usersacrossdifferenttaskscenarios.Comparedto previousworkby[36],thisstudyachievedimprove‑ ments,includingminimizedtrajectoryerrors,aug‑ mentedcapacity,andoptimizedmemoryutilization fortheDMPdatabase.However,ithasadrawbackof executingeachmovementseparately,whichrestricts itsapplicationinscenariosthatrequireintegration ofseveralmovements.[37]addressesthisproblem byusinghierarchicaldeepreinforcementlearning tointegrateDynamicMovementPrimitives(DMPs) withactor‑criticalgorithms.Thisapproachenables robotstoperformsequencesofmovementprimitives, therebyenhancingef iciencyintaskexecution.Itwas successfullyevaluatedona6DOFrobotarmthrough apick‑and‑placeoperation.

Thesestudieshavedemonstratedthecapabilityof DMPstoperformcomplex,real‑worldtasksbydynam‑ icallyadaptingtochangesandensuringprecisetra‑ jectoryplanning.Buildingonthis,anovelapproach involveslearningdirectlyfromtherapistdemonstra‑ tions.Inthismethod,atherapistperformsrehabilita‑ tionexerciseswhichareobservedbyarobot,which thenreplicatesthesemovementswithapatient.

Thistechniqueeliminatestheneedfordeeptech‑ nicalknowledgeinroboticsfromthetherapist’sside, asasimpledemonstrationtotherobotsuf ices.While variousalgorithmscanbeusedforlearningfromther‑ apistdemonstrations,theintegrationofDMPswith LearningfromDemonstrations(LfD)presentsapar‑ ticularlyeffectivesolutionforrepresentingcomplex humanmovements[38].

InthisresearchweproposeaLearn‑from‑ TherapistDemonstrationsframeworkthatintegrates DMPwithModelReferenceAdaptiveControl(MRAC) toenhanceroboticassistedrehabilitation.TheDMP componentcapturesdemonstrationsbyformulating themasanon‑lineardifferentialequation.Theseare thenmodelledusingLocallyWeightedRegression (LWR),whichlearnsadynamicforcingterm, therebygivingtheDMPtheabilitytoadaptthe learnedtrajectoriestovariousscenarios.TheMRAC componentisusedtodynamicallyadaptthelevel ofassistanceprovidedtopatients.Itemploysa Jacobian‑basedcontrollertoconverttaskvelocities intojointvelocities,coupledwithanadaptivecontrol mechanismthatadjuststheassistanceinreal‑ time,basedonongoingpatientperformance.This frameworkprovidesthe lexibilitytogeneralizethe observeddemonstrationbyadjustingtheparameters oftheDMP.Italsoenhancestheadaptationto patientperformancethroughtheMRAC,thereby implementingan’assist‑as‑needed’rehabilitation strategy,whichgivethepatientsthefreedomto activelyparticipateinrehabilitation.

Thus,themainobjectiveofthisresearchis todevelopaLearn‑from‑TherapistDemonstrations frameworkthataddressesseveralaspectsofrobotic‑ assistedrehabilitation.Theframeworkaimsto:(i) enabletherapistswithnotechnicalskillstotrain thesystemingeneratingrehabilitationexercises(ii) learnandgeneralizethemotionstylefromthera‑ pistdemonstrations(iii)accuratelyreplicatethese exerciseswitharobotforpatienttreatmentand(iv) dynamicallyadaptthelevelofroboticassistancebased onpatientdeviations,promotingan’assist‑as‑needed’ approach.Therefore,wehypothesizethattheLfTD frameworkwillsigni icantlyenhancemotorfunction rehabilitationbyfacilitatingmoreaccuratereplica‑ tionoftherapist‑ledmovements,therebyincreasing theadaptabilityandpersonalizationoftherapyses‑ sions.Additionally,wealsoexpectthisframeworkto improvepatientoutcomeswithfasterrecoverytimes andgreaterimprovementinmotorfunctioncompared toconventionalrobotic‑assistedtherapies.

TheLfTDframeworkisexpectedtosigni icantly improvetheeffectivenessandpersonalizationof rehabilitationtherapiesbymergingtheexpertiseof humantherapistswiththeprecisionandadaptabil‑ ityofroboticsystems.Suchanapproachbridgesthe gapbetweensophisticatedroboticprogrammingand practicaltherapeuticinterventions,makingadvanced rehabilitationtechnologyaccessibleandeffectivefor abroaderrangeofpatientsbyprovidingtailoredand responsivetherapy.

Figure1. AblockdiagramoftheproposedLfTDsystem, whichtrainsarobotwithtwodegreesoffreedomto performrehabilitationexercises.Theseexercisesare learnedfromtherapistdemonstrationsusingDMP

Theexpectedbene itsareextensive:therapistscan leveragethistechnologytoenhancetheircapabilities, offeringinterventionsthatwerepreviouslyunfeasible duetophysicalortimeconstraints.

Therestofthispaperisorganizedasfollows:After theintroduction,SectionIIdetailstheLfTDframe‑ work.Thissectiondescribestheprocessofdemon‑ strationcollection,DMPcomputation,exercisegen‑ eration,andfeedbackcontrolimplementation.Based onthisframework,SectionIIIpresentsitsapplica‑ tiononasimulatedroboticmanipulator,alongwitha discussionoftheresultsobtained.Lastly,SectionIV concludeswithourmain indingsanddirectionsfor futureresearch.

2.Learn‐from‐TherapistDemonstrations

2.1.SystemOverview

TheproposedLfTDframeworkisdesignedto enableatwo‑linkroboticmanipulatortomimicupper limbrehabilitationexercisesbylearningfromdemon‑ strationsprovidedbytherapists.Thisprocessisstruc‑ turedintofourmainsteps,asoutlinedinFigure1:

1) Collectingdemonstrationsfromtherapists,

2) LearningthemovementprimitivesusingDynamic MovementPrimitives(DMP),

3) Generatingpersonalizedexerciseroutines,

4) Implementingadaptivefeedbackcontroltoensure preciseexecutionoftheexercises.

Thesubsequentsectionswillprovideadetailed explanationofeachofthesesteps.

2.2.DemonstrationsCollection

Theprocessofcollectingrehabilitationdemon‑ strationsinvolvescapturingthetherapist’sCartesian motionusingavisualtrackingsystem.Inthissetup, ahealthyhumansubjectsimulatesthetherapistby demonstratingtherehabilitativeexerciseintendedfor thepatient,withthetrackingsystemcapturingthis demonstrationforlaterreplicationinrobotic‑assisted rehabilitation.Thesubject’strajectoryisrecorded throughacoloredmarkerplacedontheirwrist,which istrackedbyanRGBcameramountedonanL‑shaped

standaimedatthewrist,asshowninthecollectionof demonstrationsblockinFigure.1

Thisdemonstrationestablishesthedesired motionpattern,whichisthenutilizedwithinourLfTD frameworkinasimulatedenvironment.Tosimplify datacollection,thecaptureddataisrepresentedin two‑dimensionalCartesiancoordinateswithinthe imagespace.Thedecisiontousevisualtrackingwas drivenbytheneedforastraightforwardyeteffective methodtopreciselycapturedynamicmovements.

Themotioniscapturedasaseriesofpixelcoordi‑ nates, p ∈ R2,andthenconvertedintocoordinates, y ∈ R2,thattherobotcanuseinitsoperationalspace. Theconversionprocessismathematicallyexpressed byEquation(1),whichmapsthecamera’s ieldof view,de inedbytherange[��min,��max],totherobot’s workspace,de inedbytherange[��min,��max]:

y = (p −��min)×(��max −��min) (��max −��min) +��min (1)

Thisconversionisessentialtoensurethatthecap‑ turedtrajectoriesalignwiththeconstraintsandcapa‑ bilitiesoftherobot.Itaccuratelyscalesthemotion datato itwithinthedimensionsoftherobot’soper‑ ationalarea.Aftercapturingthemotion,thenextstep istocalculatethevelocityandacceleration,which arecriticalinputsfortheDMPalgorithm.Duetothe discretenatureofthedata,thesecalculationsareper‑ formedusingthe initedifferencemethod,asdetailed inEquations(2)and(3):

v(��)= y(��+1)− y(��) Δ�� (2) a(��)= v(��+1)− v(��) Δ�� (3)

Inthiscontext, y(��), v(��),and a(��) representthe position,velocity,andaccelerationattime ��,respec‑ tively.Thetimeintervalbetweensamplesisdenoted by Δ��.Toreducenoiseinthecapturedmotiondata, amovingaverage ilterisappliedtotheposition databeforecalculatingvelocityandacceleration.This smoothingprocessisde inedbyEquation(4),where themovingaverageattime��iscalculatedas: ̄y (��)= 1 �� −1 ��=0 y(��−��) (4) where �� isthenumberofsamplesusedinthemov‑ ingaverage.This iltersmoothsthedatabyaveraging adjacentpoints,effectivelyminimizingrandom luctu‑ ationsandensuringthatsubsequentcalculationsfor velocityandaccelerationarebasedoncleanerinput. Oncecapturedandcalculated,thedataisnormalized, preparingitforfurtherprocessing.

2.3.DynamicMovementPrimitives(DMP)

ThefundamentalprincipleofDynamicMovement Primitives(DMP)istomodelcomplexmotionsusing differentialequationsthatdescribethetemporalevo‑ lutionofmotion[38, 39].Eachdegreeoffreedom

(DoF)inthedemonstratedtrajectoryismodeledby asecond‑orderdifferentialequation.Thisequationis analogoustoapointmassattachedtoaspring‑damper mechanism,combinedwithanon‑linearforcingterm f,withanacceleration ̈y = a,andisexpressedas[40]:

��y =��(��(g y) y)+ f (5)

Inthismodel, y representsthepositionand y = v thevelocityofthetherapist’swristcollectedduring demonstrations.Thevariable g denotestheend‑target positionofthemovement.Theconstants ��, ��,and �� arepositiveparametersthatadjustthespatialand temporalscalesofthedemonstration.Meanwhile, f isthenon‑linearforcingtermthatencapsulatesthe uniquemovementpatternsobservedinthedemon‑ strations.

TheprimarygoaloftheDMPistolearntheforc‑ ingterm f,asspeci iedinEquation(5),basedonthe demonstrationsthatprovidedatafor y, y,and y [41]. Thisforcingterm f isexpressedasaweightedsumof ��Gaussianbasisfunctions,asfollows:

y0) (6)

Inthisformulation, �� representstheweights learnedfromdemonstrations,while �� denotesthe Gaussianbasisfunctions.Thevariable g istheend‑ targetposition,and y0 isthestartingpositionofthe demonstration.Thephasevariable �� modulatesthe in luenceoftheterm��(g y0),whichactsasbotha diminishingandascalingfactor.Thisensuresthatthe systemconvergestowardsthegoal,aidingtheforcing terminstabilizingthesystemtoasteadystateof rest.Theweights �� aredeterminedthroughlocally weightedregression[42],allowingfortheaccurate reproductionoftherequiredtrajectorywhenneces‑ sary.

Giventhetime‑dependentnatureoftheforcing term,autonomyisachievedbyintroducingaphase variablethroughacanonicalsystem,whichensures themotion’stemporalalignmentbystartingatan arbitraryvalueusually1anddecayingto0overthe courseofthemotion,makingthemotiontosmoothly startsandendsatthedesiredpositions[43].This phasevariabletypicallyutilizesasimple,time‑based decayingexponentialfunction,expressedintermsof irst‑orderdynamicsasfollows:

̇��=−�� (7) where �� isaconstantthatdetermineshowfastthe systemdecays.Meanwhile,theGaussianbasisfunc‑ tionsarerepresentedas:

�� =exp(−ℎ�� (��−��)2) (8) where ℎ�� and �� representthemeanandvarianceof eachGaussianfunctionrespectivelythatde inehow thein luenceofeachbasisfunctionvarieswiththe phasevariable��

AsdepictedinFigure 2,totailortheDMPto ef icientlylearnthetherapistdemonstrations y =

[��,��],weusethecaptureddatafromthedemon‑ stratedtrajectorytocomputetheforcingterm f = [��1,��2]inEquation.(5).Thecanonicalsystemoutlined inequation(7)thenundergoesintegration,withaspe‑ ci ictemporalscaling,toaccuratelyderivethephase variable.Foreachbasisfunctionin f,locallyweighted regressiontechniquesareappliedtodeterminethe weights�� thatminimizetheerrorbetweentheforce inferredfromdemonstrationsinEquation.(5)andthe estimatedforceinEquation.((6)).

Canonical System

assessedusingMeanAbsoluteError(MAE),asde ined byEquation(9):

(9)

ThisequationprovidesacomprehensiveMAE valueoverthetotalnumberoftimeinstants(N)by takingtheabsolutevalueofthedifferencebetween demonstrated(y�� (��)) andreplayedtrajectories (y�� (��))ateachtimeinstant(��)alongthe ��‑th Cartesianaxis.

weights( )

positions( )

system1

system2

Figure2. Schematicdiagramofatwo‐dimensionalDMP. Thisdiagramillustrateshowlearnedweights(��)model trajectories,adjustingthemotionfromthestart(y0)to thegoal(g)positions

2.4.GenerationofRehabilitationExercises

Inrobot‑assistedrehabilitation,thegeneralization capabilitiesofDMPareparticularlyvaluable[33]. Rehabilitationrobotsoftenneedtoassistpatients withavarietyofmovementexercisesthatmaychange asthepatient’sconditionevolves.Bylearningthe fundamentalpatternsofmovementexercisesthrough therapist‑leddemonstrations,DMPscangenerate new,patient‑speci ictrajectories[39].Thisadaptive qualityallowstherehabilitationrobottocatertothe uniquerecoveryneedsandprogressofeachpatient. Forexample,asapatientregainsmoremotorcontrol, theDMPcanadjustthetrajectoriestobemorechal‑ lengingortargetdifferentaspectsofmovement.

DMPsofferthe lexibilitytoadaptlearneddemon‑ strationstonewgoalsorconditionsbymodifying theirtemporalandspatialattributes.Oncetheweights fortheforcingterm(asshowninEquation(5))are established,newmotionscanbegeneratedinterms ofposition(y),velocity(y),andacceleration(̈y )[36]. Thisadaptationinvolvesadjustingtheinitialposition (y0),goalposition(g),orthetemporalscalingfactor (��)asdetailedinEquation(5).Theseadjustments canbemadewithouttheneedtoretrainthemodel fromscratch.Afterdeterminingthenewacceleration fromEquation(5),integrationisusedtocomputethe correspondingvelocityandpositionpro ilesovertime forthenewlygeneratedtrajectories.

Thiscapabilityisimportantfortherapistsasitcan enablethemtoprogressivelyincreasethecomplex‑ ityandrangeofexercisesaccordingtothepatient’s improvingcondition,helpingtoensurethatthereha‑ bilitationprocessremainsbothappropriatelychal‑ lengingandengagingthroughout[44].

TheabilityoftheproposedLfTDframework toaccuratelyreproducethetherapist’smotionis

2.5.ModelReferenceAdaptiveControl(MRAC)for KinematicControl

Aftergeneratingnewrehabilitationexercises learnedfromtherapistdemonstrations,atwolink manipulatorwillbeusedtoensurethatthepatient’s hand,whichisattachedtotherobot’send‑effector, followsthedesiredtrajectoryaccurately.Thissetup allowsforprecisecontroloftherobot’smovements, aligningthemwiththespeci icrehabilitationgoals setbasedonthetherapist’sinput.

DMPframeworkgeneratesmotionbyde ininga sequenceofposition,velocity,andaccelerationsam‑ plesthatoutlinethe desiredtrajectory fortherobot’s endeffector.Totranslatethistrajectoryfromthetask spacetotherobot’sjointspace,itisessentialtosolve therobot’sinversekinematics.Thisresearchadopts aJacobian‑basedkinematiccontrolmethodtoitera‑ tivelycalculatetherequiredjointcon igurations(q)to guidetherobotalongthespeci iedtrajectory.These calculatedcon igurationsthenactasreferenceinputs withintherobot’sjointspace,toensurepreciseadher‑ encetothetrajectories[45].

Consequently,amechanismforadaptationis requiredtodynamicallyadjusttheassistancepro‑ videdtothepatientwhilefollowingthetrajectory, ensuringthattheassistanceisprovidedbasedon theperformanceofthepatient.Thedesignedcon‑ trolschemeaddressestheserequirementsandis explainedinrelationtotheschematicinFigure3,as follows,

1)Jacobian‐basedcontroller: Thenon‑linearrela‑ tionshipbetweenend‑effectorvelocitiesandjoint velocitiesisgivenby:

y�� = J(q)q (10) where y�� isthevelocityoftherobot’send‑effector, arethejointvelocities,and��(��)istherobot’sanalyti‑ calJacobianatcon iguration��de inedas��y/��q [46].

Thisrelationshipallowsforthedeterminationof jointvelocitiesbycomputingtheinverseoftheJaco‑ bianmatrixasfollows[47],

q = J−1(q)y�� (11)

Thejointcon iguration q canthenbecomputed byintegratingthesevariablesovertime.Itisworth notingthatinourrobot’scase,theJacobianmatrixis

Figure3. ConceptualblockdiagramoftheMRACillustratingtheadaptivecontrolstrategybyhighlightingthefeedback loopthatdynamicallyadjustscontrollergains(K, ��)toenhancedisturbancerobustness

square,whichallowsforthedirectcomputationofits inverse.Toavoidnumericaldriftduringthetracking ofthedesiredtrajectory y��,thejointvelocitiesare rede inedasfollows,

q = J−1 �� (q)(y�� + Ke) (12)

Here, y�� representsthetimederivativeofour desiredtrajectory,whiletheerror e representsthe differencebetweenthedesiredandactualendeffector positions (y�� y��).Additionally,adiagonalpositive de initematrix K ∈ R2×2 isintroducedtoensure stabilityandconvergence.

Analternativeapproachutilizesthetransposeof theJacobianmatrix,expressedas:

q = J�� (q)Ke (13)

Thisalternativerepresentationnotonlyreduces trackingerrorsbutalsoeliminatessteady‑stateerrors [48].

2)AdaptationMechanismforPersonalizedAssistance: Toencourageactiveinvolvementofpatientsinthe rehabilitationprocess,wehavedesignedthegain K tobeadaptive,allowingthelevelofassistancetobe modi iedaccordingtothepatient’sperformance.The newadaptedgaindenotedas K iscomputedbasedon thecurrenterror e,theoriginalgain K,andaprede‑ terminedrateparameter��,asfollows:

Thisgainisdesignedtobeproportionaltoboththe magnitudeoftheerrorandtherateparameter,ensur‑ ingthataseitherincreases,theassistanceprovided tothepatientisappropriatelyadjusted,withhigher ratevaluesleadingtomoresubstantialassistance.The adaptabilityofthegainbasedonerrormeasurement allowsthesystemtorespondtochangesinapatient’s performancelevels.Thismeansthattheassistance providedbytheroboticsystemisdirectlyin luenced bythepatient’sdeviationfromthedesiredtrajectory. Bycontinuouslymonitoringtheerror,thesystemcan adjustthelevelofsupporttomeeteachpatient’s uniqueneeds,ensuringthatthosewhorequiremore assistancereceiveitwhileallowingmorecapable patientstoengagewithreducedsupport.Tofurther re inethelevelofassistance,anadditionalcondition isapplied:iftheerrormagnitude |e| fallsbelowa smallthreshold ��,therateparameter �� isreduced signi icantly,indicatedas:If|e|<��,⇒��≪�� Thisconditionaladjustmentprovidesamechanismto encouragepatientstotakemorecontrolovertheir movementsastheyimprove,therebyfosteringactive participationintherehabilitationprocess.

3.ResultsandDiscussion

TheproposedLfTDframeworkisimplemented usingtheparametersprovidedinTable 1 toteacha two‑linkrobothowtolearnandadapttherapeutic motionsforupperlimbrehabilitation,withexperi‑ mentsconductedinbothrealandsimulatedenviron‑ ments.

Table1. ParametersfortheLfTDModel

No.ofgaussians 60

DMPparameter 8.5

DMPparameter 4 �� Canonicalsystemconstant 0.8

�� Timescalingconstant 0.218

r Controlleradaptationrate 100

�� Adaptivecontrollergain [1016]

�� Robotlinklengths [50,50]cm

Intheinitialphase,wesuccessfullycapturedthe rehabilitationmotionsthroughvisualtracking,by usinganRGBcameramountedonanL‑shapedtri‑ podasexplainedintheLfTDframework.Twodiffer‑ entdemonstrationswererecordedasillustratedin Figure 4,witheachinvolvingahumansubjectmov‑ inghiswrist(withalabel)inthecamera’s ieldof viewtogeneratethetherapeuticmotion,whileatthe sametimeconvertingthecapturedmotionfrompixel coordinatestotherobot’scartesiancoordinatesusing equation 1.Wechoosemotionssuitableforatwo‑ linkmanipulatorbyconsideringmovementslikethe igure’S’illustratedinFigure4a.Thisexerciseinvolves movingthearmtotraceanS‑shapedpatterninthe horizontalplane.Theanalysisinthispaperwillfocus onthisdemonstration.

Aftercapturingthetrajectory,thegenerated motioninCartesiancoordinateswasfedtotheDMP forcalculatingthetargetforcingterm ��,withthe aimoflearning,reproducing,andgeneralizingthe capturedmotion.Sixtybasisfunctionswereused tobalanceaccuracyandcomputationalef iciency, allowingthesystemtoeffectivelycapturemotion detailswithoutover itting.Thiscon igurationwas re inedthroughcarefulevaluation.Utilizinglocally weightedregressionandthesixtyequallyspaced basisfunctions,asdepictedinFigure 5,thesystem successfullylearnedtheweightsforeachdegreeof freedom(DOF),whicharecrucialforreproducingthe capturedmotion.ParticleSwarmOptimization(PSO) wasusedtooptimizetheDMPparameters �� and ��. Aswarmof16particleswasrunfor40iterations, resultinginconvergencewithin51.61secondsand anobjectivefunctionvalueof0.0041.Thesearch spacefortheparameterswaslimitedto[0,0]and [30,30].Theoptimalvaluesfoundwere ��=22.23 and��=6.64,whichimprovedthemodel’strajectory accuracy.Figure 6 illustratestheTrajectoryError Function,showingthereductioninerrorovertime. Byef icientlylearningtheweightsandoptimizingthe parameters,theDMPwasabletoaccuratelylearnand reproducethetrajectorywithgoodprecision.

Oneexampleofthereproduceddemonstrationis depictedinFigure 7 whereitiscomparedwiththe originaldemonstrationwhichhavestartingand inal pointsat 00 �� and −26.5−22.6 �� cmrespec‑ tively.TheDMPsuccessfullyreproducedthemotion withnegligibleerrorinboth��and��positionswiththe errorvariationdepictedinFigure8.Thiserror,though

negligible,isprimarilyattributedtotheapproxima‑ tionofthenon‑linearforcingtermandthetuningof theDMPparameters.

DetailedMAEvaluesarepresentedinTable 2, highlightingtheaccuracyofthereproducedtrajectory, withatotalpositionerroracrossbothaxesof0.51cm. Theselowervaluesindicateahigherdegreeofsimilar‑ ity.

Table2. MeanAbsoluteError(MAE)metrics

Position(cm)

TheabilityoftheDMPtogeneralizetonewgoals wasexaminedbymodifyingboththeinitialand inal pointsofthedemonstratedtrajectorywhilemain‑ tainingthelearnedweights,asillustratedinFig‑ ure 9.Theoriginaltrajectory,withinitialand inal pointsat 00 �� and −26.5−22.6 �� cm,respec‑ tively,wasadaptedtoanewtrajectorywithinitialand inalpointsat 2.82.0 �� and −32.3−29.6 �� cm, respectively.

Thegeneratedtrajectorysuccessfullyadaptedto thenewgoalwhilemaintainingtheexactshapeofthe demonstratedtrajectorypatternwithoutalteringthe fundamentalmotionstyle.Thiscapabilityisadvanta‑ geousforperformingrepetitivetherapeuticexercises withvaryinggoals,asiteliminatestheneedtoretrain therobotforeachdistinctgoal.Forinstance,thesame trajectorypatterncanbeappliedtobothadultsand children,accommodatingthedifferentsizesoftheir movements.Additionally,manyactivitiesofdailyliv‑ ing,suchaspickingandplacingobjects,donothave ixedinitialand inalpoints;thus,bydemonstrating themotionpatternonce,theDMPcangeneralizethe movementtosuitvariousapplications[30,33,34].

Inadaptingthetrajectorytonewinitialand inal points,theparametersoftheDMPde inedinTable 1 werekeptconstant.Thisensuredthattheshape anddynamicsoftheoriginalmotionpatternwere preserved.

Figure4. Twodistinctdemonstrationsperformedby humansubject,depictingvariationsinmovement patternstracedduringdemonstrations.(a)S‐shaped pattern;(b)’3’‐shapedpattern

Figure5. SixtyequallyspacedGaussiandistributions illustratingthebasisfunctionsused

x-axis

Figure6. PerformanceoptimizationusingPSO

x-axis

Figure7. Reproduceddemonstratedtrajectoryusingthe proposedLfTDtechnique

x and y position error (mm)

Figure8. TotalpositionreproductionerrorinDMP modeling

A)RobotSimulationandDesignConsiderations: The simulationprocess,illustratedinFigure 3,utilized MATLAB’sSimscapewithakinematicmodelofthe

Figure9. Adaptationofthedemonstratedtrajectoryto changesinstartingPointsandgoals.Figure(a)shows thetrajectoryadaptedtoanewgoal,maintainingthe originalstartingpoint.Figure(b)demonstrateshow boththestartingpointandgoalarechanged

two‑linkmanipulator,focusingsolelyontheposi‑ tionsandtrajectoriesoftherobot.Thedeveloped LfTDperformanceintrackingthegeneratedtrajec‑ torywasanalysedbycon iguringtherobottofollow trajectoriesde inedbytheDMP.Thisisachievedby integratingthereproducedtrajectoryintoourMRAC, whichfacilitatedtheconversionofthesetrajectories intojoint‑spacecommandsfortherobottoexecute themovements.Thisintegrationshowcasestheseam‑ lesstransitionfromtask‑spacemodellingtoactual roboticmotion.At irst,itispresumedthatthepatient requiresassistancetocompleteagiventask.

Thedeviationbetweentheactualanddesiredend‑ effectortrajectoriesexecutedbytherobotisillus‑ tratedinFigure 10.This igureshowsthattherobot accuratelytracksthedesiredtrajectorythroughout itsmotion.Thesimulationresultsfurtherhighlight therobot’scapabilitytomaintainprecisetrajectory trackingina2Dsetting,providinganevaluationofthe MRAC’sperformance.

B)TestingAdaptabilityandFutureProjections: To assesstheadaptabilityoftheMRAC,disturbances

wereintroducedtotheend‑effector’spositionduring therobot’smotion,asillustratedinFigure 11.These disturbancessimulatepatientimpairmentbehaviors, suchassuddenchangesinmovementordeviations fromtheintendedpath[49].Disturbancesaregener‑ atedaswaveformswithanamplitudeof2cmanda durationof20sforthex‑axis,andanamplitudeof3 cmwiththesamedurationforthey‑axis.Eachwave‑ formis ilteredthrougha irst‑ordertransferfunction, 1 ��+1,wheretheparameter��issetto6,basedoncrite‑ riasuchasovershootandsettlingtime,tosmooththe signalandensurethedisturbancesexhibitarealistic, gradualeffect.Thecontroller’sadaptationmechanism adjuststhegaininreal‑timebasedonfeedbackfrom thepatient’sperformance.Thisfeatureallowsthe patienttoactivelyparticipateintherehabilitationpro‑ cess,asthecontroller’sassistancedynamicallycor‑ respondstothepatient’sdeviationfromthedesired trajectory.Thecontroller’sresponsetothesedistur‑ bancesasillustratedinFigure 11 demonstratesits robustnessandadaptabilitytochangingconditions. ByexaminingtheMRAC’sabilitytomanagethese il‑ tereddisturbances,wegaininsightintoitspotential (a)

Figure10. LfTDframeworkintrajectorytracking performancebetweendesiredandactualtrajectories. (a)TrackingforeachDOF.(b)Trackingforthe demonstratedtrajectoryexecutedbytherobot (a)

Figure11. Trackingthegeneratedtrajectorywithadded disturbancesmimickingpatientdeviationfor(a)each DOFand(b)forthedemonstratedtrajectoryas executedbytherobot

tohandlepatient‑inducedvariations,supportingthe effectivenessoftherapeuticexercisesdespiteunpre‑ dictablemovementpatterns.Theseresponseswere recordedaftertuningdifferentgainvaluesandrate parameters,andtheireffectonsystembehaviourwas analyzed.Thecon igurationsthatyieldedthemost favourableoutcomesaredetailedinTable1.

Inevaluatingthesystem’sperformance,threekey metricswereprioritized:theaccuracyofmotion reproduction,thegeneralizability,andadaptabilityto dynamicchanges.Ourresultsindicateasigni icant improvementinmotionreproductionaccuracywith MAEvaluesof0.34cmand0.17cminthexandyposi‑ tionalcoordinatesrespectively,highlightingthesys‑ tem’sabilitytocloselymimicandgeneralizedesired movements.Furthermore,testsundervaryingcondi‑ tionswithdisturbancesmimickinghumandeviation demonstratedthesystem’sadaptabilitytodynamic changes.These indingssupporttheef icacyofour approachinaddressingthecriticalaspectsofper‑ formance,underscoringitspotentialforreal‑world applications.Finally,ourtechniqueischaracterizedby itsabilitytocapturedemonstrationthroughone‑shot

learningandtohavetheabilitytogeneralizeandadapt thisdemonstration.

4.Conclusion

Inthispaper,wepresentedaninnovative approachthatutilizeslearningfromtherapist demonstrationstomodeltherapeuticexercisesforthe upperlimb.ALearn‑from‑TherapistDemonstration frameworkwasdevelopedthroughthisapproach whichemploysDynamicMovementPrimitives (DMP)formodellingcomplexmotorfunctions,and aModelReferenceAdaptiveController(MRAC)for adaptivelearning.Thisframeworkenablesa2‑link robottoreplicaterehabilitationexerciseswithina simulated2Denvironment.Additionally,thesystem’s adaptabilityhasbeenfurthervalidatedthrough disturbancetests,underscoringthepotentialofour roboticrehabilitationsolutiontoofferpersonalized andeffectivetherapyandforfuturedevelopmentsin robotic‑assistedrehabilitation.

Bymodelingandgeneralizingtherapeuticexer‑ cisesandprovidingassistancebasedonthesub‑ jectsperformance,wehavedemonstratedthepoten‑ tialfeasibilityofourapproach.Althoughtheresults arepromising,therelianceonsimulationspresents somelimitations.Futureworkwillfocusonadvanc‑ ingtopracticalexperimentswithactualrobotsto assesstheeffectivenessofthisapproachinimprov‑ ingpatientoutcomes.Additionally,incorporatinga dynamicmodelcouldbeexploredtoevaluatethe in luenceofdynamicparametersinscenarioswhere dynamicresponsesplayamoreprominentrole.

AUTHORS

IsmailAuta∗ –DepartmentofMechatronicsand RoboticsEngineering,Egypt‑JapanUniversityofSci‑ enceandTechnologyNewBorgAlArabCity,Alexan‑ dria,Egypt,e‑mail:ismail.auta@ejust.edu.eg. AhmedFares –DepartmentofComputerScienceand Engineering,Egypt‑JapanUniversityofScienceand Technology,Alexandria,Egypt on‑leave:ElectricalEngineeringDepartment,Faculty ofEngineeringatShoubra,Benhauniversity,Cairo, Egypt,e‑mail:ahmed.fares@ejust.edu.eg.

HiroyasuIwata –FacultyofScienceandEngi‑ neering,WasedaUniversity,Tokyo,Japan,e‑mail: jubi@waseda.jp.

HaithamEl‑Hussieny –DepartmentofMechatronics andRoboticsEngineering,Egypt‑JapanUniversityof ScienceandTechnologyNewBorgAlArabCity,Alexan‑ dria,Egypt,e‑mail:haitham.elhussieny@ejust.edu.eg.

∗Correspondingauthor

References

[1] A.Demeco,L.Zola,A.Frizziero,C.Martini, A.Palumbo,R.Foresti,G.Buccino,and C.Costantino,“Immersivevirtualrealityin post‑strokerehabilitation:asystematicreview,” Sensors,vol.23,no.3,p.1712,2023.

[2] N.Marotta,A.Ammendolia,C.Marinaro, A.Demeco,L.Moggio,andC.Costantino, “Internationalclassi icationoffunctioning, disabilityandhealth(icf)andcorrelation betweendisabilityand inanceassetsinchronic strokepatients,” ActaBioMedica:Atenei Parmensis,vol.91,no.3,p.e2020064,2020.

[3] G.Collaborators,V.L.Feigin,B.A.Stark,C.O. Johnson,G.A.Roth,C.Bisignano,G.G.Abady, M.Abbasifard,M.Abbasi‑Kangevari,F.Abd‑ Allah, etal.,“Global,regional,andnationalbur‑ denofstrokeanditsriskfactors,1990–2019:a systematicanalysisfortheglobalburdenofdis‑ easestudy2019,” TheLancetNeurology,vol.20, no.10,pp.795–820,2021.

[4] A.Demeco,R.Foresti,A.Frizziero,N.Darac‑ chi,F.Renzi,M.Rovellini,A.Salerno,C.Martini, L.Pelizzari,andC.Costantino,“Theupperlimb orthosisintherehabilitationofstrokepatients: Theroleof3dprinting,” Bioengineering,vol.10, no.11,p.1256,2023.

[5] C.M.Stinear,C.E.Lang,S.Zeiler,andW.D. Byblow,“Advancesandchallengesinstroke rehabilitation,” TheLancetNeurology,vol.19, no.4,pp.348–360,2020.

[6] K.S.Hayward,S.F.Kramer,E.J.Dalton,G.R. Hughes,A.Brodtmann,L.Churilov,G.Cloud, D.Corbett,L.Jolliffe,T.Kaffenberger, etal.,“Tim‑ inganddoseofupperlimbmotorintervention afterstroke:asystematicreview,” Stroke,vol.52, no.11,pp.3706–3717,2021.

[7] R.Bertani,C.Melegari,M.C.DeCola,A.Bramanti, P.Bramanti,andR.S.Calabrò,“Effectsof robot‑assistedupperlimbrehabilitationin strokepatients:asystematicreviewwith meta‑analysis,” NeurologicalSciences,vol.38, pp.1561–1569,2017.

[8] L.A.Simpson,K.S.Hayward,M.McPeake,T.S. Field,andJ.J.Eng,“Challengesofestimating accurateprevalenceofarmweaknessearlyafter stroke,” NeurorehabilitationandNeuralRepair, vol.35,no.10,pp.871–879,2021.

[9] A.G.Doğan,“Theeffectofoccupationalther‑ apyonupperextremityfunctionandactivities ofdailylivinginhemiplegicpatients,” Journal ofMedicineandPalliativeCare,vol.4,no.4, pp.350–354,2023.

[10] B.Zhang,K.‑P.Wong,andJ.Qin,“Effectsofvirtual realityonthelimbmotorfunction,balance,gait, anddailyfunctionofpatientswithstroke:Sys‑ tematicreview,” Medicina,vol.59,no.4,p.813, 2023.

[11] J.Huang,J.‑R.Ji,C.Liang,Y.‑Z.Zhang,H.‑C. Sun,Y.‑H.Yan,andX.‑B.Xing,“Effectsofphysi‑ caltherapy‑basedrehabilitationonrecoveryof upperlimbmotorfunctionafterstrokeinadults: asystematicreviewandmeta‑analysisofran‑ domizedcontrolledtrials,” AnnalsofPalliative Medicine,vol.11,no.2,pp.52131–52531,2022.

[12] M.Franceschini,S.Mazzoleni,M.Goffredo, S.Pournajaf,D.Galafate,S.Criscuolo,M.Agosti, andF.Posteraro,“Upperlimbrobot‑assisted rehabilitationversusphysicaltherapyon subacutestrokepatients:Afollow‑upstudy,” JournalofBodyworkandMovementTherapies, vol.24,no.1,pp.194–198,2020.

[13] M.Coscia,M.J.Wessel,U.Chaudary,J.d.R. Millán,S.Micera,A.Guggisberg,P.Vuadens, J.Donoghue,N.Birbaumer,andF.C.Hummel, “Neurotechnology‑aidedinterventionsforupper limbmotorrehabilitationinseverechronic stroke,” Brain,vol.142,no.8,pp.2182–2197, 2019.

[14] J.Bernhardt,K.S.Hayward,N.Dancause, N.A.Lannin,N.S.Ward,R.J.Nudo,A.Farrin, L.Churilov,L.A.Boyd,T.A.Jones, etal.,“A strokerecoverytrialdevelopmentframework: consensus‑basedcorerecommendationsfrom thesecondstrokerecoveryandrehabilitation roundtable,” InternationalJournalofStroke, vol.14,no.8,pp.792–802,2019.

[15] R.M.Al‑Whaibi,M.S.Al‑Jadid,H.R.ElSerougy, andW.M.Badawy,“Effectivenessofvirtual reality‑basedrehabilitationversusconventional therapyonupperlimbmotorfunctionofchronic strokepatients:asystematicreviewandmeta‑ analysisofrandomizedcontrolledtrials,” Phys‑ iotherapyTheoryandPractice,vol.38,no.13, pp.2402–2416,2022.

[16] A.N.Malik,H.Tariq,A.Afridi,and F.AzamRathore,“Technologicaladvancements instrokerehabilitation,” JournalofPakistan MedicalAssociation,vol.72,no.8,pp.1672–1674,2022.

[17] M.Kyrarini,F.Lygerakis,A.Rajavenkata‑ narayanan,C.Sevastopoulos,H.R.Nambiappan, K.K.Chaitanya,A.R.Babu,J.Mathew,and F.Makedon,“Asurveyofrobotsinhealthcare,” Technologies,vol.9,no.1,p.8,2021.

[18] M.Germanotta,L.Cortellini,S.Insalaco,and I.Aprile,“Effectsofupperlimbrobot‑assisted rehabilitationcomparedwithconventionalther‑ apyinpatientswithstroke:Preliminaryresults onadailytaskassessedusingmotionanalysis,” Sensors,vol.23,no.6,p.3089,2023.

[19] S.Giovannini,C.Iacovelli,F.Brau,C.Loreti, A.Fusco,P.Caliandro,L.Biscotti,L.Padua, R.Bernabei,andL.Castelli,“Robotic‑assisted rehabilitationforbalanceandgaitinstroke patients(roar‑s):studyprotocolforaprelimi‑ naryrandomizedcontrolledtrial,” Trials,vol.23, no.1,p.872,2022.

[20] S.B.Reis,W.M.Bernardo,C.A.Oshiro,H.I.Krebs, andA.B.Conforto,“Effectsofrobotictherapy associatedwithnoninvasivebrainstimulation onupper‑limbrehabilitationafterstroke:sys‑ tematicreviewandmeta‑analysisofrandomized

clinicaltrials,” NeurorehabilitationandNeural Repair,vol.35,no.3,pp.256–266,2021.

[21] I.Aprile,G.Guardati,V.Cipollini, D.Papadopoulou,A.Mastrorosa,L.Castelli, S.Monteleone,A.Redol i,S.Galeri,and M.Germanotta,“Roboticrehabilitation:an opportunitytoimprovecognitivefunctions insubjectswithstroke.anexplorativestudy,” FrontiersinNeurology,vol.11,p.588285,2020.

[22] R.Iandolo,F.Marini,M.Semprini,M.Laffranchi, M.Mugnosso,A.Cherif,L.DeMichieli,M.Chi‑ appalone,andJ.Zenzeri,“Perspectivesandchal‑ lengesinroboticneurorehabilitation,” Applied Sciences,vol.9,no.15,p.3183,2019.

[23] F.Yakub,A.Z.M.Khudzari,andY.Mori,“Recent trendsforpracticalrehabilitationrobotics,cur‑ rentchallengesandthefuture,” International JournalofRehabilitationResearch,vol.37,no.1, pp.9–21,2014.

[24] T.Takebayashi,K.Takahashi,Y.Okita,H.Kubo, K.Hachisuka,andK.Domen,“Impact oftherobotic‑assistancelevelonupper extremityfunctioninstrokepatientsreceiving adjunctroboticrehabilitation:sub‑analysis ofarandomizedclinicaltrial,” Journalof NeuroEngineeringandRehabilitation,vol.19, no.1,p.25,2022.

[25] G.A.Albanese,A.Bucchieri,J.Podda,A.Tacchino, S.Buccelli,E.DeMomi,M.Laffranchi,K.Man‑ nella,M.W.Holmes,J.Zenzeri, etal.,“Roboticsys‑ temsforupper‑limbrehabilitationinmultiple sclerosis:aswotanalysisandthesynergieswith virtualandaugmentedenvironments,” Frontiers inRoboticsandAI,vol.11,p.1335147,2024.

[26] F.Romero‑Sánchez,L.LuporiniMenegaldo,J.M. Font‑Llagunes,andM.Sartori,“Rehabilitation robotics:Challengesindesign,control,andreal applications,” Frontiersinneurorobotics,vol.16, p.957905,2022.

[27] P.Maciejasz,J.Eschweiler,K.Gerlach‑Hahn, A.Jansen‑Troy,andS.Leonhardt,“Asurveyon roboticdevicesforupperlimbrehabilitation,” JournalofNeuroengineeringandRehabilitation, vol.11,pp.1–29,2014.

[28] M.Maaref,A.Rezazadeh,K.Shamaei,and M.Tavakoli,“Agaussianmixtureframework forco‑operativerehabilitationtherapyin assistiveimpedance‑basedtasks,” IEEEjournal ofselectedtopicsinsignalprocessing,vol.10, no.5,pp.904–913,2016.

[29] Y.Meng,C.Munroe,Y.‑N.Wu,andM.Begum,“A learningfromdemonstrationframeworktopro‑ motehome‑basedneuromotorrehabilitation,” in 201625thIEEEInternationalSymposiumon RobotandHumanInteractiveCommunication (RO‑MAN),pp.1126–1131,IEEE,2016.

[30] M.Maaref,A.Rezazadeh,K.Shamaei,R.Ocampo, andT.Mahdi,“Abicyclecrankingmodelfor

assist‑as‑neededroboticrehabilitationtherapy usinglearningfromdemonstration,” IEEE RoboticsandAutomationLetters,vol.1,no.2, pp.653–660,2016.

[31] S.PareekandT.Kesavadas,“iart:Learningfrom demonstrationforassistedrobotictherapyusing lstm,” IEEERoboticsandAutomationLetters, vol.5,no.2,pp.477–484,2019.

[32] S.F.Atashzar,M.Shahbazi,M.Tavakoli,andR.V. Patel,“Acomputational‑model‑basedstudyof supervisedhaptics‑enabledtherapist‑in‑the‑ looptrainingforupper‑limbpoststrokerobotic rehabilitation,” IEEE/ASMETransactionson Mechatronics,vol.23,no.2,pp.563–574,2018.

[33] P.Pastor,H.Hoffmann,T.Asfour,andS.Schaal, “Learningandgeneralizationofmotorskills bylearningfromdemonstration,”in 2009 IEEEInternationalConferenceonRoboticsand Automation,pp.763–768,2009.

[34] C.Tamantini,M.Lapresa,F.Cordella,F.Scottodi Luzio,C.Lauretti,andL.Zollo,“Combineduseof dmpandrealobjectsinrobot‑aidedrehabilita‑ tion,”in 2020ItalianConferenceonRoboticsand IntelligentMachines(I‑RIM),012021.

[35] C.Lauretti,F.Cordella,E.Guglielmelli,and L.Zollo,“Learningbydemonstrationforplan‑ ningactivitiesofdailylivinginrehabilitationand assistiverobotics,” IEEERoboticsandAutomation Letters,vol.2,no.3,pp.1375–1382,2017.

[36] A.J.Ijspeert,J.Nakanishi,H.Hoffmann,P.Pastor, andS.Schaal,“Dynamicalmovementprimitives: learningattractormodelsformotorbehaviors,” Neuralcomputation,vol.25,no.2,pp.328–373, 2013.

[37] W.Kim,C.Lee,andH.J.Kim,“Learningandgen‑ eralizationofdynamicmovementprimitivesby hierarchicaldeepreinforcementlearningfrom demonstration,”in 2018IEEE/RSJInternational ConferenceonIntelligentRobotsandSystems (IROS),pp.3117–3123,IEEE,2018.

[38] M.Saveriano,F.Abu‑Dakka,A.Kramberger,and L.Peternel,“Dynamicmovementprimitivesin robotics:Atutorialsurvey,” TheInternational JournalofRoboticsResearch,vol.42,092023.

[39] I.A.Seleem,H.El‑Hussieny,andS.F.M.Assal, “Motionplanningforcontinuumrobots:Alearn‑ ingfromdemonstrationapproach,”in 201827th IEEEInternationalSymposiumonRobotand

HumanInteractiveCommunication(RO‑MAN), pp.868–873,IEEE,2018.

[40] A.Li,Z.Liu,W.Wang,M.Zhu,Y.Li,Q.Huo,and M.Dai,“Reinforcementlearningwithdynamic movementprimitivesforobstacleavoidance,” AppliedSciences,vol.11,no.23,2021.

[41] G.Li,Z.Jin,M.Volpp,F.Otto,R.Lioutikov,and G.Neumann,“Prodmp:Auni iedperspective ondynamicandprobabilisticmovementprim‑ itives,” IEEERoboticsandAutomationLetters, vol.8,no.4,pp.2325–2332,2023.

[42] S.Shaw,D.K.Jha,A.Raghunathan,R.Corcodel, D.Romeres,G.Konidaris,andD.Nikovski,“Con‑ straineddynamicmovementprimitivesforsafe learningofmotorskills,”2022.

[43] A.S.Anand,A.Østvik,E.I.Grøtli,M.Vagia,and J.T.Gravdahl,“Real‑timetemporaladaptationof dynamicmovementprimitivesformovingtar‑ gets,”in 202120th InternationalConferenceon AdvancedRobotics(ICAR),pp.261–268,2021.

[44] J.‑C.Metzger,O.Lambercy,A.Calif i,D.Dinacci, C.Petrillo,P.Rossi,F.M.Conti,andR.Gassert, “Assessment‑drivenselectionandadaptationof exercisedif icultyinrobot‑assistedtherapy:a pilotstudywithahandrehabilitationrobot,” Journalofneuroengineeringandrehabilitation, vol.11,pp.1–14,2014.

[45] B.Siciliano,“Kinematiccontrolofredundant robotmanipulators:Atutorial,” JournalofIntel‑ ligentandRoboticSystems,vol.3,pp.201–212, 091990.

[46] L.SciaviccoandB.Siciliano,“Modellingandcon‑ trolofrobotmanipulators,” MeasurementScience andTechnology,vol.11,p.1828,dec2000.

[47] D.E.Whitney,“Resolvedmotionratecontrol ofmanipulatorsandhumanprostheses,” IEEE TransactionsonMan‑MachineSystems,vol.10, no.2,pp.47–53,1969.

[48] P.ChiacchioandB.Siciliano,“Aclosed‑loopjaco‑ biantransposeschemeforsolvingtheinverse kinematicsofnonredundantandredundant wrists,” JournalofRoboticSystems,vol.6,no.5, pp.601–630,1989.

[49] M.L.LatashandJ.G.Anson,“Whatare“normal movements”inatypicalpopulations?,” Behav‑ ioralandBrainSciences,vol.19,no.1,p.55–68, 1996.

TOWARDSEXPLAINABLEGRAPHSPECTRALCLUSTERINGFORBERTEMBEDDINGS

Submitted:10th July2025;accepted:10th August2025

MieczysławA.Kłopotek,SławomirT.Wierzchoń,BartłomiejStarosta,PiotrBorkowski,DariuszCzerski

DOI:10.14313/jamris‐2026‐005

Abstract:

ArtificialIntelligencealgorithmsareincreasinglyapplied totasksinNaturalLanguageProcessing,includingdoc‐umentclustering.Asthesealgorithmsbecomeincreas‐inglycomplex(suchastransformer‐basedembeddings, likeBERT)and/orareofa“black‐box”nature,suchas GraphSpectralClustering(GSC)algorithms,thedemand forexplainingtheresultsofsuchalgorithmsisbecom‐ingincreasinglyurgent.Inthispaper,weproposea model‐awaremethodtoexplaintheresultsofGSCinthe contextofBERT‐basedembeddings.Wepresentanovel theoreticalmethodologyforexplanation,basedonthe premisethatdocumentsimilarityinGSCiscomputedas cosinesimilarityofBERTembeddingsofdocuments.We demonstratethevalidityofthismethodologybypresent‐ingstrongGSCclusteringresults,restoringthehuman‐madeassignmentofhashtagstotweets.Weshowthat GSCbasedonBERTembeddingsoutperformsapproaches usingTermVectorSpaceandGloVeembeddings.There‐fore,theresultingexplanationsarealsoexpectedtobeof higherquality.

Keywords: ExplainableMachineLearning,NaturalLan‐guageProcessing,GraphSpectralClustering,Document EmbeddingversusExplainability,BERTandGloVeand TVSEmbedding

1.Introduction

Asnotedin[1],accessingtextualinformationhas neverbeeneasyinhumanhistory.Therewasatime whentextdocumentswerescarce;today,wefacean overwhelmingstreamofdiversematerials.However, ineachcase,creatingoverviewsofdocumentcollec‑ tionsisvital.Forcenturies,subjectindexingandclassi‑ icationsystemshaveservedthispurpose.Inthepast, itwaspossibletopreparealistofcategoriesandthen manuallyadddocumentstothatlist.However,today thereisagrowingurgencytoautomatebothtasks.

Althoughthereexistelaboratehuman‑madecat‑ egorydictionariesforgeneralpurposes,specialized collectionsofdocumentsrequirespecializedlocal methodsforsplittingthedocumentsintoreasonable groupsandlabelingthemappropriately.Thelatter taskseemstorequiretheinvolvementofautomated methodologies.

Clusteringmethodologiesandaccompanyingtech‑ nologiesforexplaininggroupmembershipcanprove tobehelpfulsupportingtoolshere.

Methodsofclusteringtextdocumentshavealready foundnumerousapplications.Theycanhelpuncover connectionsbetweendifferenttexts,grouptheminto “natural”categories,orpinpointthemostrelevant topicsintheircontentandexpressthemindistinct terms.Theyareusedforrapidinformationretrieval and iltering,topicextraction,andautomaticdocu‑ mentorganization.

Clusteringrequiresthedevelopmentofasimilar‑ itymeasurebetweendocuments.Thesimplestway wouldbetoembedthedocumentsinanEuclidean spaceand(1)tousethedistancebetweentheembed‑ dingpointsandapplycommonalgorithms,suchas�� means[2]forclusteringinsuchaspace,or(2)touse, forexample,thecosinesimilaritybetweenembed‑ dingvectorsandthenusesimilarity‑basedalgorithms likethoseoftheGraphSpectralClusteringclass[3]. Althoughamultitudeofsimilaritymeasureshavebeen proposed[4],cosinesimilarityseemstore lectseman‑ ticsimilarityquitewell[5].1 Thelatterisembracedin thispaper.

Manytypesofembeddingspaceshavebeendevel‑ opedsincethevery irstproposaloftermvectorspace (TVS)[6],[7].Despiteitsstraightforwardness,TVShas twonotabledrawbacks:(1)thedocumentisregarded asacollection(or“bag”)ofwords,whichleadstothe lossofcontextandtherelationshipsamongterms,and (2)thedimensionalitycanbeashighasdozensof thousands,evenforamoderate‑sizecollectionofsev‑ eralthousanddocuments.Therefore,newembedding approaches,suchasWord2vec[8],Doc2Vec[9],GloVe [10],orBERT[11],andmanyothers(see[12–14], or https://huggingface.co/spaces/mteb/leade rboard),weredevelopedtoaccommodaterelation‑ shipsbetweenterms,andtoreducedimensionality, althoughdimensionalitycanstillbehigh,reaching hundredsofdimensions.

Fortunately,anembeddingspacewithasmall numberofdimensionscanbesuf icientifGSCisused forclustering.

Usually,wedonotneedonlytheclustersoftextual documentsbutalsoacharacterizationoftheircontent intermsofkeywords,forexample.Theusualapproach tothistaskwastogettheclustering irst,andthento seekdifferences/similaritiesinthewordsetsrelated toeachoftheclusters.Thisapproachcanbetermed the“characterizationofclusters.”Itidenti iesthefea‑ turesoftheobtainedclustersbutdoesnotexplain whyadocumentbelongstooneclusterratherthan

another.Yetoften,wewanttounderstandwhyadoc‑ umentispartofagroupandwhythealgorithmputit inthatgroup.Weshalltermthis“clustermembership explanation”.Itseemslikeasimpleprocessifweclus‑ terthedocumentsdirectlyintheTVSvia,e.g.,��‑means algorithm.Thisisbecausethecentercoordinatesof theclustersshowhowimportantthewords/termsare fortheclustermembership.

However,bothwhenclusteringinthemodern embeddingspaces,likeword2vec,doc2vec,GloVe, BERTandothertransformermethods,andwhenusing GSCbasedclustering,wegetresultsthatarehard toexplainastherelationshipbetweentheembed‑ dingspacecoordinatesanddocumentwords/terms getslostwhichiscounterproductivewithrespectto thelibrarian’sgoaltoassignsomeautomatedsubject index.Forexample,inGSC,theclusteringprocessis completelydetachedfromthewords,asonlysimilar‑ itiesareused.BERTembedding,ontheotherhand, makestherelationshipbetweenwordsanddocument embeddingcompletelyunrecoverable.

Inthispaper,wemakeanattempttoovercomethe mentionedproblemswithexplainability.Weoutlinea methodologyfortheexplanationofGSCresultsper‑ formedinaBERTembedding,wherebycosinesimi‑ larityisusedassimilaritymethod.Beforedoingso insection 4,we irstrecallsomerelatedwork.In section2weprovideabriefoverviewofthemethod‑ ologiesbehindBERTembeddingsofdocuments.In section3,wepresentanintroductiontoGSCmethod‑ ology.InSection5wepresentexperimentssupporting thevalidityofourapproach.Section6concludesthe paper.

2.BERTEmbeddingExtractions

BERT(BidirectionalEncoderRepresentations fromTransformers),availablesince2018,isamethod ofpretraininglanguagerepresentationsofnatural languagetextmodeling[11].Onecaneitherusethese modelstoextracthigh‑qualitylanguagefeatures fromone’stextdata,oronecan ine‑tunethese modelsonaspeci icmachinelearning(ML)task, suchasclassi ication,entityrecognition,orquestion answering,basedonone’sdataset.Thereexist multipleBERTmodelsnowadays,startingwiththe so‑called bert-base-uncased,a110Mparameter modelcontaining12layers(blocks)andworkingwith lowercasetext.2

Toextractthecontextualembeddingofeach wordinthesentence,oneneedstotokenizethe documentandfeedthetokens,togetherwithposi‑ tionalandtokentypeinformation,tothepre‑trained BERT,whichwillreturntheembeddingsforeach ofthetokens.Apartfromobtainingthetoken‑level (word‑level)representation,onecanalsoobtain thedocumentlevelrepresentation.Theresultis obtainedbypassingtheinputthroughmultiple(12in bert-base-uncased)neurallayers.Eachlayer(called atransformerblock)consistsoftwosublayers.The irstsub‑layerimplementsamulti‑headself‑attention mechanism,andthesecondoneisaposition‑wisefully

connectedfeed‑forwardnetwork.Theoutputofeach sublayerisaddedtotheinputthatbypassesthesub‑ layerthrougharesidualconnection,andtheresulting signalendsinlayernormalizationthatispassedtothe nextlayer.Atthe inallayer,onegetsarepresentation ofeachtokenateachposition.Notethatgenerally, BERTmodelsreturntokenembeddingsrelativetoa givensentence/document.Thismeansthatthesame wordcanhavedifferentembeddingsindifferentsen‑ tences,andeveninthesamesentencewhenitisused severaltimes.

Asexplainedby[15],therearemultiplewaysof extractingembeddingsofwords,sentences,ordocu‑ mentsfrompre‑trainedBERTmodels.

Aswordsareconcerned,thereexistmultiplepos‑ sibilitiestoobtainastaticwordembedding.Forexam‑ ple,the“AveragedBERT”methodconsistsofaver‑ agingtherepresentationsofthesamewordinits differentcontextstoacquireastaticembedding.The “AveragedplusRegularBERT”approachmeansthat therepresentationofawordinagivencontextisthe sumoftheoriginalBERTrepresentationplusthemen‑ tionedaveragedvalue.Onemaynotrestrictoneself tothelastlayeronly,butuse,e.g.,last4layers.Other variantsarealsodiscussedintheliterature.

Thesimplestrepresentationoftheentiretextis tousetheaverageoftheembeddingsofindividual words.Moreadvancedistheweightingofwordswith thelogarithmofinverteddocumentfrequency.

Theobtainedvectorsmaybenormalizedtothe unitlength,orviadividingbytheirstandarddevia‑ tions,andinmanyotherways.

BERTwasusedinMLapplicationsrequiringexpla‑ nation.Forinstance,[16]utilizedtheBERTmodel forthepurposeofmedicalclassi ication,usingatwo‑ stagemodel.The irststagewasusedforpurposes ofdocumentembedding,whilethesecondstagewas trainedtopredicttheclassi ication.Explanationswere word‑based,wherethewordimportanceforclassi‑ icationdecisionwasproducedbasedonagradient‑ basedmethodcalledintegratedgradients[17,18].

[19]discussestheexplainabilityofBERTmodel resultsinthetaskofsentimentanalysis.

3.AShortIntroductiontoGSC

Letusnowexplainhowtoexploittheproperties oftheGraphSpectralClusteringmethodologywhen clusteringdocumentsinBERTembedding.Usageof GSCforclusteringinsteadofdirectclusteringinthe BERTembeddinghastheadvantageofreducingthe dimensionalityoftheclusteringproblembyanorder ofmagnitude.Aswewillshowsubsequently,theresult ofGSCapproximatestheresultofdirectclustering inBERTembedding.Thisallowsclusteringinalow‑ dimensionalspace.Inaddition,itispossibletobuild anexplanatorybridgetotheGSC/BERTcombination.

LetuscharacterizetheGSCapproachtoclustering brie ly.Amoredetaileddescriptioncanbefound,for example,in[3].

Let��bea(symmetric)similaritymatrixbetween pairsofitems(documents,liketweetsinourcase).

Itinducesagraphwhosenodescorrespondtothe entities(documents),hencethe“Graph”partofthe GSCname.Let��denotethenumberofitemsforwhich �� hasbeencomputed.Let �� bethediagonalmatrix with�� =∑�� =1 �� foreach��∈[��]

Inthedomainoftextmining,thementionedsimi‑ laritymatrixisusuallybasedoneitheragraphrepre‑ sentationofrelationships(links,hyperlinks)between items,orsuchagraphisinducedby(cosine)simi‑ laritymeasuresbetweentheseitems.However,mixed objectrepresentations(textandlinks)havealsobeen studied,forexampleby[20].Inthispaper,weuse thecosinesimilaritybetweenBERTembeddingvec‑ tors.Hence,theelementsof �� areoftheform �� = ��(��)��(��),where��(��)standsfortheBERTembed‑ dingfordocument��.Weassumethatalltheseembed‑ dingsarevectorsofunitlength.Additionally,by GSCconvention,alldiagonalelementsof �� arezero. (Unnormalizedor)combinatorialLaplacian �� corre‑ spondingtothismatrixisde inedas

(1)

A normalizedLaplacian ℒofthegraphrepresented by��isde inedas

ℒ=��

The rationormalizedLaplacian3 takestheform

ℒℛ =��′−1/2��′−1/2 =��−��′−1/2��′��′−1/2 (3)

where��′ =��+��and��′ =��+��

Thesplitinto �� disjointclustersisachievedas follows.Onecomputestheeigen‑decompositionofthe respectiveLaplacian(e.g.,��,ℒ,orℒℛ),getting��eigen‑ values��1 ≤…≤�� (always��1 =0,duetomathemat‑ icalpropertiesofthementionedLaplacian’s)andcor‑ respondingeigenvectors v1,…, v��.Thenoneembeds thedocumentsintothe��‑dimensionalspacespanned bythe��eigenvectorscorrespondingto��lowestposi‑ tiveeigenvalues.Thatis,��‑thdocumentisrepresented bythevector�� =[��,2,…,��,��+1]��.Thisshallbecalled ��‑embedding,resp.��‑embedding,or��‑embeddingif theeigenvectorsaredeterminedfromthecombinato‑ rial(resp.normalizedorrationormalized)Laplacian ��, ℒ,or ℒℛ.Mathematicalpropertiesimplythatthe eigenvector v1 ofthecombinatorialLaplacianisacon‑ stantvector;hence,togetaninformativeembedding, weusetheeigenvectors v2,…, v��+1.Wealsoapplythis conventiontotheotherembeddings,thatis�� and�� forconsistency.Onceaproperembeddinghasbeen determined,itispossibletoclusterthedocumentsin achosenembeddingspaceusing,forexample,the�� meansalgorithm.See[3]or[21]fordetails.��‑means clusteringinthe��‑embeddingshallbecalled��‑based clustering.��‑basedclusteringand��‑basedclustering bede inedanalogously.Notealso,thatthe ��‑based clusteringallowsustoapproximatetheclustering optimizingso‑calledRCutcriterion.Incontrast,the ��‑basedclusteringapproximatestheclusteringopti‑ mizingNCutcriterion,while��‑basedclusteringallows

theapproximationoftheNRCutclusteringcriterion,a mixtureofboth.

Note,thatitispossibletoformalizethethreeclus‑ teringcriteriaoftheform

where Γ={��1,…,��} isapartitionofthesetof documents,and�� isavectorwithcomponents�� (��∈ 1∶��)beingthe��‑thentriesofthe irst��eigenvectors oftheappropriatematrix��,ℒ,or��.Thatisif v�� isan eigenvector,then�� =��,��

The��‑thclustercenterisde ined

(6)

where �� representsembeddingof ��‑thdocumentas describedabove.

Theformula(4),ispreciselythelossfunctionof the ��‑meansalgorithmintherespective(Euclidean) embeddingspace.So,itisquitenaturalthattheirmin‑ imaaresoughtbyapplyingthetraditional ��‑means algorithm.

Graphclusteringwasprimarilyassociatedwiththe quantity

whichrepresentstheaggregatesimilarityofnodes thatareneighborsinagivengraphbutbelongto differentclusters.Properlynormalizingthisquantity weobtainthreedifferentcriteria,i.e.,

ThelastformulaallowsustoconsidertheNRCutclus‑ teringcriterionasamixtureofNCutandRCut.

Notethesymmetrybetweenthe��‑meanscluster‑ ingcriterionandthecuttingcriterion.In(4),theaver‑ ageddifference(measuredbytheEuclideandistance)

betweenthemembersofthe��‑thclusterisminimized, whilein(8),theaveragedsimilaritybetweenneigh‑ boringnodesassignedtodifferentclustersismini‑ mized,respectively.

4.OurApproach

Ourapproachtoexplanationdiffersfromthatpre‑ sented,e.g.,by[18]astheirmethodologyis(1)tar‑ gettingexplanationforclassi icationwhileweaimat explanationofclusteringresults,(2)theclassi ication mechanismtheyuseisablack‑boxitself,whileweaim atamethodologylinkingtheclusteringresultcleanly tothetextualcontentofdocuments,withoutreferring tomysteriouscoef icients.

4.1.PreparingBERTEmbeddingforExplanation

Asrecalledinsection2,onecangetanembedding ofanentiresentenceaswellasembeddingsofeach wordoccurrenceinthesentenceinmultipleways.Let ustrytogetakindofuniformrepresentationforthem. Assumewehaveacollection��ofdocuments��1,…,�� andadictionary �� consistingofwords ��1,…,��.A word �� occursindocument �� (��,��) times.Letus de ineafunctionℰ(��)returningtheBERTembedding inthespace ℝ�� ofthedocument �� andafunction ℰ(��,��,��)returningtheBERTembeddingofthe��ℎ occurrenceinthespaceℝ�� oftheword�� inthedoc‑ ument�� [11].Thedimensions�� and�� areassumed tobeidentical.

Furthermore,aswewanttocomputecosinesim‑ ilarities,assumethat ℰ∗(��)=ℰ(��)/‖ℰ(��)‖,and ℰ∗(��,��,��)=ℰ(��,��,��)/‖ℰ(��,��,��)‖

Frequently,theembeddingextractiontechnologies inducethatℰ(��)isalinearcombinationofalloccur‑ rences��,��,butitdoesnotneedtobeso[15].

However,thedimensionality �� ishigherthanthe maximumnumberofwordoccurrencesinadocu‑ ment,wecanassumethatthereexistsasetofnon‑ negativecoef icients �� foreachdocument �� such that‖∑��∈�� ∑��(��,��) ��=1 ��,��,��ℰ∗(��,��,��)−ℰ∗(��)‖ismin‑ imized(constrainedminimization). Inotherwords

ℰ∗(��)≈ ��∈�� (��,��) ��=1 ��,��,��ℰ∗

Letusde inetheimportance(orcontribution)of awordforthedocumentinsuchawaythatthesum oftheembeddingsofalloccurringwordsisequalto 1(approximately),whileclosenessofawordembed‑ dingvectortothedocumentembeddingvectoris re lectedbyhighervalues.

Letusclusterthedocumentsinthisembedding using��‑means.Thatis,weminimizethelossfunction

(Γ)= ��

where Γ={��1,…,��} isapartitionofthesetof documents.

Eachcluster�� willhaveaclustercenter(orproto‑ type)��(��)suchthat

). (13)

Byanalogy,theimportanceofawordfortheclus‑ termaybede inedas

(14)

Bysortingthewordsbytheirdecreasingimpor‑ tance,wegettheknowledgewhichwordexplainsbest clustermembership.

Note,that ��(��,��)= 1 |��| ��

4.2.WeightedClusteringinBERTEmbedding Considernowamodi iedvectorintheBERT embedding.Let��

where gi isavectorofdimension ��,equaltozero everywhereexceptthe��thelement,�� = ��−1

.With thisnotation,letuscomputethesquareddistance betweentwodocuments:(fordifferent��,��)

Clearly,ifthedocumentembeddingisa linearcombinationoftokenembeddings,then

Thedissimilarityisgreaterwhenthevectorsare longer,butsmallerifthedotproductisbigger.Under thisassumption,letusperformweighted ��‑means clustering(forweightedkernel‑��‑meanssee,e.g., [22]).Thatis,weminimizethelossfunction

where Γ={��1,…,��} isapartitionofthesetof documents.Eachcluster�� willhaveaclustercenter (orprototype)��(��)suchthat

Note,thatthe��

(��

)vectorsareofhigherdimen‑ sion.Thereisnoneedtousetheminpracticeduring theclusteringprocess.Oneusesthemonly“mentally” forthesakeofexplanation.Asprovenin[23],clusters resultingfromclusteringthe

(��

) vectorsarethe sameasclustersresultingfromclusteringusingGSC basedonnormalizedLaplacians.

Asforexplanationpurposesnote,that

whichmeansthatthewordimportancecanbecom‑ putedinanalogytotheunweightedcase,butwiththe distinctionthatpotentiallytheclustersaredifferent duetoweightedclustering.

Theweightedimportancecanbede inedas

Thende ine ��=��−1(��+��−2��)��−1 (30) with ��,�� beingde inedaspreviously.Let ℳ bethe matrixoftheform:

ℳ=− 1 2(��− 1 �� 11��)��(��− 1 �� 11��). (31)

Note,that 1 isaneigenvectorof ℳ,withthecorre‑ spondingeigenvalueequalto0.Alltheothereigen‑ vectorsmustbeorthogonaltoitas ℳ isrealand symmetric,soforanyothereigenvector v of ℳ we have: 1��v =0. LetΛbethediagonalmatrixofeigenvaluesofℳ, and �� thematrixwherecolumnsarecorresponding (unitlength)eigenvectorsofℳ.Thenℳ=��Λ��.Let �� =Λ1/2�� ,where�� standsforthe��‑throwof��.Let ��,�� betheembeddingsofthedocuments ��,��,resp. Thisembeddingshallbecalledℳ‑embedding.Then ‖�� −��‖2 =�� =(�� +�� −2��)/(��) (32) for��≠��,andzerootherwise.Letusnowdiscussper‑ formingweighted ��‑meansclusteringonthevectors �� withweightsamountingto�� respectively.

Letususethefollowingweightingofdocuments: �� =��.Clusteringviaweighted ��‑meanswith weights �� inthe ℳ embeddingwilloptimizethe followingcriterion ��[ℳ��](Γ;��)= ��

(��)‖2 (33) whereby

(34)

In[23]ithasbeenproventhat ��[ℳ��](Γ;��)=��−2��+��(Γ) (35)

Recall,that��(Γ)wasde inedinequation(8c). Since ��−2�� isaconstant,minimizingonecriterion minimizestheother.As ��‑basedClustering(cluster‑ ingusingthenormalizedLaplacian ��)hasthesame targetasNCutclustering,theyareequivalent(seesec. 3).

In[23],proofcanbefoundthat ��[ℳ��](Γ;��)= ��

4.3.AProposalofDouble‐centered“Normalized”Doc‐umentSimilarityMatrixBasedEmbedding(ForUse WithWeighted ��‐Means)

In[23],ithasbeensuggestedtousethe��matrix ofthefollowingform. �� beamatrixofthefollowing form ��= 11�� −��. (29)

). (36)

Itiseasilyseenthatitisidenticalwiththeequation (24)showingequivalenceof ��[ℳ��](Γ;��) with ��[��](Γ;��)

Thiscompletestheexplanationbridge:youcan clusterwithnormalizedLaplacianbasedGSCand explainwithweightedBERTexplanationmethod.The advantageisthattheclusteringisperformedinamuch lowerdimensionalspace(��dimensionswhenseeking ��clusters),butwithoutthedisadvantageofbasicGSC ofnon‑explainability.Explanationisreachedinthe BERTspace.

4.4.AlgorithmicDescriptionofClusteringwithExplana‐tionUnderBERTEmbedding

Theoutlinedmethodologycanbesummarizedas follows:

1) EmbedthedocumentcollectionintoBERT.

2) Foreachdocumentℰ(��)computethenormalized embeddingvectorℰ∗(��)

3) Computecosinesimilaritybetweendocuments��,�� ascosinebetweentheembeddingvectorsofdoc‑ uments ��ℰ∗(��)��ℰ∗(��) formingthesimilarity matrix��

4) Forthesimilaritymatrix ��,applynormalized Laplacianbasedclustering,eq.(4).

5) Foreachtoken ℰ(��,��,��) ofthedocument computethenormalizedembeddingvector ℰ∗(��,��,��)

6) Foreachdocument,computethelinearapproxima‑ tionofthenormalizeddocumentembeddingvec‑ torbasedonnormalizedtokenembeddingvectors eq.(10).

7) Foreachcluster,withcentereq.(21),calculatethe importanceofwordsassociatedwiththecluster eq.(28).

8) Take��mostimportantwordsastheclusterexpla‑ nationforeachcluster.

5.Experiments

In[23],ithasbeenshownthatexplainabilitycan beachievedwhenperformingGSCforsimilarities obtainedintheTermVectorSpace.Therefore,the questioncanberaised:Whatisgainedwhenconsider‑ ingtheBERTembedding?Experimentshavebeenper‑ formedshowingtheimprovedclusteringperformance ofthelatter.

Forthispurpose,westudiedtheeffectivenessof BERT‑basedclusteringversusGloveandtraditional TVSembeddings.ThesBERTandBERTembedding modelsweredownloadedautomatically(inthe background)bylibrariesusedforembedding oftextualdocuments,wherebymodelnames neededtobeprovided.Themodelsareavailable, forexample,inthehuggingfacerepository (https://huggingface.co/models).Alternatively, manualdownloadisalsopossible.Pythonlibraryused forBERTembeddingswas transformers.Models usedinexperimentswere: bert-base-uncased, vinai/bertweet-base, distilbert-base-uncased, cardiffnlp/twitter-roberta-base. Python libraryusedforsBERTembeddingsiscalled sentence_transformers.ThenamesofsBERT andBERTembeddingsvisibleinthetables 1 5, consistoftwoandthreeparts,respectively,separated withand#.The irstpart(sBERT,BERT)indicatesthe typeofembedding.Thesecondpartisthecommon nameoftherespectiveembeddingmodel.Thelast part,forBERTembeddings,following#,indicates theversionofembeddingextraction.Twodifferent documentembeddingextractorswereconsidered:

thosemarkedwith#[CLS]meanembeddingstaken fromthe[CLS]token,while#T_AVGindicatethatthe documentembeddingwastakenastheplainaverage of(theother)tokenembeddings.

GloVe‑basedembeddingsaretheonestrained onTwitterdata(GloVe@twitter)andtrainedon Wikipediadata(Glove@wiki),downloadedfromthe pagesindicatedin[10].Theembeddingswithtra‑ ditionalTVS(tf,t idf)arebasedonthePython scikit-learn library.

Theclusteringmethodoveralltheembeddings wasN‑basedclustering.sBERTembeddingsdiffer fromBERTembeddingsinthatforBERTembeddings, wegetbothdocumentembeddingandembeddings ofindividualtokens.Thisenablesexplanationstobe derived.However,sBERTprovidesdocumentembed‑ dingonly.Inthisstudyofclusteringef iciency,we comparesBERTandBERTembeddingstoseewhether ornotsBERT‑basedclusteringsperformsigni icantly betterthanBERTembeddings.Iftheydonotperform signi icantlybetter,thenwecanuseBERTembed‑ dingsforclusteringandcanbesatis iedwiththe explanationmethodologyprovidedinthispaper.If sBERTweresigni icantlybetter,separateexplanation methodsneedtobesoughtforsBERT.

Theclusteringexperimentswereperformedwith popularPythonlibraries: numpy [24], scipy [25], scikit-learn [26]and soyclustering [27]which isanimplementationofspherical ��‑means[28].In particular,weused SpectralClustering classfrom scikit‑learnwiththe affinity parametersetto: precomputed (af inityfromsimilaritymatrix)asa representativeofthe��‑embeddingbasedclustering.

TheHungarianmethod[29]isappliedtomatch hashtagsandclusters,aimingtoachievethebest agreement(lowesterrorrate).

Wemadethecomparisonbasedon5datasets drawnfromTwitter.Eachdatasetconsistedofdocu‑ mentsrelatedtooneof ivehashtagsonly.Below,a shortcharacterizationofeachdatasetisgiven.

DataSet0:8050documentswith23855distinct terms.Itconsistsof5hashtags:mentalhealth(1003 docs),brexit(1607),ukraine(1687),covid_19 (1696),writingcommunity(2057).

DataSet1:8159documentswith21188distinct terms.Itconsistsof5hashtags:bbcqt(1025docs), sidnaaz(1153),lufc(1470),100daysofcode(1718), tejran(2793).

Dataset2:8256documentswith20217distinct terms,and5hashtags:r4today(1036docs),rhobh (1298),bb22(1622),blm(1849),cdnpoli(2451).

Dataset3:8218documentswith21061distinct terms,and5hashtags:smackdown(1063docs), rhop(1154),btc(1487),maga(2079),nufc(2435).

Dataset4:8440docswith26939distinctterms, and5hashtags:dnd(1031docs),robostopia(1323), browns(1493),aewdynamite(1821),2(2772).

Tomeasurethequalityofclustering,wecompared theclusteringresultswiththehuman‑madeclustering intermsofthehashtagsassignedtodocuments.Out

Table1. N‐basedclusteringofDataset0usingvarious embeddings(vectorizers)

Vectorizer F1‑avg F1‑stdev

CountVectorizer 0.255152 0.000049

TfVectorizer 0.255146 0.000102

T idfVectorizer 0.396431 0.000360

GloVe@wiki 0.363963 0.000401

GloVe@twitter 0.355927 0.001543

sBERT@all‑MiniLM‑L6‑v2 0.974979 0.000092

sBERT@all‑distilroberta‑ v1 0.951857 0.000382

sBERT@multi‑qa‑mpnet‑ base‑dot‑v1

0.942263 0.000090

BERT@bert‑base‑ uncased#[CLS] 0.472723 0.000262

BERT@vinai/bertweet‑ base#[CLS]

BERT@distilbert‑base‑ uncased#[CLS]

BERT@cardiffnlp/twitter‑ roberta‑base#[CLS]

BERT@bert‑base‑ uncased#T_AVG

0.675733 0.000439

0.592854 0.000481

0.831508 0.000091

0.602534 0.000423

BERT@vinai/bertweet‑ base#T_AVG 0.618835 0.000151

BERT@distilbert‑base‑ uncased#T_AVG 0.558688 0.000899

BERT@cardiffnlp/twitter‑ roberta‑base#T_AVG

0.476953 0.002215

Table2. N‐basedclusteringofDataset1usingvarious embeddings(vectorizers)

ofthemultitudeofpossibilities(e.g.,[30]or[31]),as aqualitymeasureforclustering,wetookthepopular externalqualityF1measureasitre lectsbothpreci‑ sionandrecall.BasedontheHungarianmethod,we associateeachclusterwithasinglehashtag,calculate F1foreachhashtagagainsttherest.Thencomputethe averageforallhashtags.

Wehadtorefrainfromevaluationoftheexpla‑ nationslimitingourselvestovisualinspection(see anexamplebelow)asnoappropriatereferencedata setsforthetweetsweconsideredareavailable.We arecurrentlydevelopinganevaluationmethodthat doesnotrequirehumaninterventionandistherefore objective.Thispaperestablishesonlythetheoretical basisfortheexplanationprocessitself.Werefrain fromaddingunnecessarycomplexitytothispaper. Thereexistmultipleworks[32–36]andsurveyson assessingthequalityofexplanationslike[37–40]. Mostqualityevaluationmethodsrequireeitherdirect interactionwithhumanuserorsomeprede inedsets with”groundtruth”explanations.Wechoseinthis paperadifferentpathwayandprovidedananalytical justi icationwhytheexplanationsaretrustworthy.

Tables1 5presentclusteringresultsfordatasets 0‑4,respectively.TheF1‑scorepresentedthereinis anaverageover30runs.Note,thatinsomecases,GSC clustering(asbasedonnormalizedLaplacian)failed duetonegativesimilarities.Mostfrequently,sBERT

GloVe@twitter 0.594311 0.001918

sBERT@multi‑qa‑mpnet‑ base‑dot‑v1

BERT@bert‑base‑ uncased#[CLS]

BERT@vinai/bertweet‑ base#[CLS]

BERT@distilbert‑base‑ uncased#[CLS]

BERT@cardiffnlp/twitter‑ roberta‑base#[CLS]

BERT@bert‑base‑ uncased#T_AVG

BERT@vinai/bertweet‑ base#T_AVG

BERT@distilbert‑base‑ uncased#T_AVG

BERT@cardiffnlp/twitter‑ roberta‑base#T_AVG

0.000166

wasaffected,butalsoinsomecasesGloVeandBERT embedding.

Inallcasesexceptfordataset4,sBERTembedding methodsyieldthebestclusteringresults.Theyare, however,notthebestoptionforexplanationpurposes asanapproximationequation.(10)constitutesapoor approximationofdocumentembeddingsviatoken embeddings.

BERTembeddingsarenotasgoodassBERT embeddings.However,ifwecompare#[CLS]embed‑ dingswith#T_AVGinthecaseofBERTmethods,they donotdifferverymuch.Hence,theusageoftheaver‑ agingmethodfordocumentembeddingiswelljusti‑ iedandpavesthewayfortheapplicationoftheeq. (10)approximation,andhencetheproposedexplana‑ tionmethodisjusti iable.And,asalreadymentioned, sBERTsaremoreproblematicthanBERTsduetofail‑ urescausedbyfrequentnegativesimilarities.

BERTmethodsaregenerallybetterthantheTVS andGloVe‑basedembeddings,butnotinallcases(as statedalsoin,e.g.,[41]).

ExplanationsofclustersunderBERTembeddings canbeessentiallyusedinameaningfulwayonlywhen theembeddingofthedocumentistheaverageover tokenembeddings(#T_AVGvariants).Onefacesasim‑ ilarproblemasinothersettings,thatis,thestopwords playanon‑negligiblerole.Seeanexampleofacluster descriptionwithtop50tokens: thetoof#mental‑ healthandaisincovidwritingcommunityforyouare thisthatwe@_ion19yourbeall‑withithavenot& peoplehealthourancanfromukrainementalwillneed thoseataboutmytheirsowhobyas.

Table3. N‐basedclusteringofDataset2usingvarious embeddings(vectorizers)

Vectorizer F1‑avg F1‑stdev

CountVectorizer 0.277486 0.000189

TfVectorizer 0.277534 0.000169

T idfVectorizer

GloVe@twitter

sBERT@all‑MiniLM‑L12‑ v2

sBERT@multi‑qa‑ distilbert‑cos‑v1

sBERT@multi‑qa‑mpnet‑ base‑dot‑v1

BERT@bert‑base‑ uncased#[CLS]

BERT@vinai/bertweet‑ base#[CLS]

BERT@distilbert‑base‑ uncased#[CLS]

BERT@cardiffnlp/twitter‑ roberta‑base#[CLS]

BERT@bert‑base‑ uncased#T_AVG

BERT@vinai/bertweet‑ base#T_AVG

0.575343 0.000483

0.385300 0.000959

0.834372 0.000242

0.820115 0.000252

0.854759 0.000197

0.488475 0.000369

0.780850 0.000328

0.464572 0.000172

0.632636 0.000423

0.478280 0.000458

0.540016 0.003658

BERT@distilbert‑base‑ uncased#T_AVG 0.444183 0.000424

BERT@cardiffnlp/twitter‑ roberta‑base#T_AVG

0.551978 0.000632

Table4. N‐basedclusteringofDataset3usingvarious embeddings(vectorizers)

Sameclusterdescribedafterremovingthestop‑ wordslookslikethis: mentalhealthcovidwriting‑ communitypeoplehealthukrainementalsupporttime pandemiclovecaresocialdaydistancingsafelifecrisis issuesfeellivesremembercomplacencycurrentdepres‑ sionmedicalzelenskyyuafuturellinzigraywipfeeling decisionprojectgovernmentbasedschoolbanffacade‑ mybxaviruschildrentalkpsychosocialpublicequip‑ mentretweetinguncivilizedpower lyfofaaviationfood‑ safetygovmoneybrexitwriting,whichseemstobe moretopical.

Althoughremovalofstopwordsimprovesthequal‑ ityofthedescription,oneissueremains.BERTlike modelscreateembeddingsbasedontheentiredoc‑ umentandthereisnoobviousjusti icationfordis‑ cardingstopwordsfromthetextwhenknowingthe speci icwayhowthemodelsaretrained(transformer method).Theeffectsofremovingstopwordsunder thesesettingswouldrequireanin‑depthinvestiga‑ tion.WhereasembeddingslikeGloVe,TVSthatare performedonaword‑by‑wordbasisarefreefrom thisambiguity–wecansimplyignorethestopwords underembeddings.

6.Conclusions

Inthispaper,westudiedtheproblemsbehind explainabilityoftheresultsofGraphSpectralClus‑ tering(GSC)methodsappliedunderdiversetypesof

sBERT@multi‑qa‑MiniLM‑ L6‑cos‑v1

sBERT@multi‑qa‑mpnet‑ base‑dot‑v1

BERT@bert‑base‑ uncased#[CLS]

BERT@vinai/bertweet‑ base#[CLS]

BERT@distilbert‑base‑ uncased#[CLS]

BERT@cardiffnlp/twitter‑ roberta‑base#[CLS]

BERT@bert‑base‑ uncased#T_AVG

BERT@vinai/bertweet‑ base#T_AVG

BERT@cardiffnlp/twitter‑ roberta‑base#T_AVG

documentembeddings,belongingtotheTVSgroup, GloVegroupandBERTgroup.

Theessentialproblemwithexplainabilityunder GSCisthattheGSCembeddingsoftheresultsofclus‑ teringhavenothingtodowiththedocumentcon‑ tent.Therefore,[23]suggestedamultistageexplana‑ tionprocesswhichisapplicabletoTVSembedding types.In[42],anextensionwasdiscussedtoGloVe typeembeddingswhichismorecomplexthanthatfor TVSembeddings.

Thementionedpapersconstituteakindofbreak‑ throughinthedomainofGSA.GSAwaspreviouslycon‑ sideredasablack‑boxmethod,whilethementioned articlesmakeitexplainable.Hereby,thereisagrading ofcomplexityonthesideoftheNLembeddings.TVS representsarelativelysimplecasewheretheweightof explainingwordisdirectlytheTVScoordinate.GloVe embeddingsaremorecomplicatedbecauseofinter‑ sectionofwordvectorsinthisspace.Thoughthisis stillasimplercasethanBERT,aseachwordhasa ixed embedding.BERTfamilyhasadifferentembedding ofeachwordnotonlyindifferentdocuments,but alsowithinthesamedocument.Theembeddingof theentiredocumentmaynotcorrespondtoalinear combinationofwordtokens.

Asdemonstratedinthispaper,astillmorecomplex extensionoftheexplanationconceptisalsopossible forBERTembeddings.

Table5. N‐basedclusteringofDataset4usingvarious embeddings(vectorizers)

Vectorizer F1‑avg F1‑stdev

CountVectorizer 0.238978 0.000000

TfVectorizer 0.238978 0.000000

T idfVectorizer 0.239120 0.000000

GloVe@wiki 0.292245 0.000981

GloVe@twitter 0.327367 0.059270

sBERT@multi‑qa‑mpnet‑ base‑dot‑v1

BERT@bert‑base‑ uncased#[CLS]

0.670961 0.000124

0.532433 0.000220

BERT@vinai/bertweet‑ base#[CLS] 0.450124 0.000068

BERT@distilbert‑base‑ uncased#[CLS]

BERT@cardiffnlp/twitter‑ roberta‑base#[CLS]

BERT@bert‑base‑ uncased#T_AVG

BERT@vinai/bertweet‑ base#T_AVG

BERT@distilbert‑base‑ uncased#T_AVG

BERT@cardiffnlp/twitter‑ roberta‑base#T_AVG

0.644404 0.000593

0.769169 0.000094

0.614746 0.000177

0.592477 0.000221

0.663272 0.000055

0.672432 0.000529

differently.InterpretabilityisonesideofMLusage pointingatthepossibilityto”makemoney”outofthe MLresults.However,theothersideoftheresultsis theirtrustfulness–investmentrisk.Onehastounder‑ standhowtheresultscameabout.Theresultsmustbe henceexplainable.

WerestrictedourselvestotheBERT‑typedocu‑ mentembeddingmodels,althoughnewonesarecon‑ stantlybeingdeveloped,suchasthoselistedonthe leaderboardat https://huggingface.co/space s/mteb/leaderboard.Wechose,however,BERT duetoitsavailabilityandbroadusage,andduetoits opensourcenatureandrelativelylowcomputational demands.WelookatthepathTVS‑GloVe‑BERTasa pathwaytowardsdevelopmentofmoregeneralexpla‑ nationmethodsforthesteadilyevolvingLLMmodels.

Whiletherearemultiplewaysofevaluatingclus‑ ters,weconcentrateoncomparisonwith”intrin‑ sic”clusteringthatishashtags.BasedonHungarian method,weassociateeachclusterwithasinglehash‑ tag,calculateF1foreachhashtagagainsttherest.Then computetheaverageforallhashtags.

Notes

1Semanticsimilarityisneverthelessanongoingsubjectof research;see,e.g.,[45].

2 ForalistofsomeBERTmodelvariantsavailablefromHugging‑ faceseehttps://www.kaggle.com/datasets/sauravmaheshkar/hu ggingface‑bert‑variants.Foralistofothertransformermodels,see, e.g.,https://github.com/abacaj/awesome‑transformers

Wefacedtwochallenges:(1)derivingtheoretical formulasofextractingwordimportanceunderthe assumptionoflinearcombinationoftokenembed‑ dings(2)experimentalcheckingifusageoflinear wordtokencombinationdeterioratesperformance comparedtothedocumentembedding(CLStoken). Managingthesetwopointsmakesthesigni icantdif‑ ferencetothepreviousresearchandallowstosay: clusteringwithGSAunderBERTisalsoexplainable.

Thequestionwasalsoraisedastowhetherornotit isworththeextraworktousetheBERT‑basedembed‑ dings.Theaccuracyofclusteringforembeddingsof thesedifferentclassesofdocembeddingswerestud‑ ied.BERTembeddingsappeartoyieldbestresults.

However,anissuewasobservedthatneedsa futureinvestigations.SimilarlytoGloVeembeddings, itmayhappenthatBERTembeddingsproducenega‑ tivecosinesimilaritybetweendocuments.Aninves‑ tigationsimilartothatdescribedin[42]isneeded toensurecompletenessofapplicabilityofexplnations underBERTembeddings.

Also,adeeperinvestigationisneededwithrespect totheroleofstopwordsinBERT,bothontheirimpact ofclusteringqualityandexplanationquality.The researchquestionis:shallweremovestopwords(1) beforestartingtheprocessofBERTembedding,(2) afteritbutbeforecomputingtheembeddingofthe documentbasedontokensforclusteringor(3)after theclusteringbutbeforeexplaining.

Note,thatanumberofresearchersinsistthatnot theexplainabilitybutrathertheinterpretabilityofML resultsismoreimportant,see[43,44],butweseeit

3 OtherLaplaciansarealsoused,e.g.,therandomwalkLaplacian ��ofagraph,de inedas ��=��−1 =��−��−1 (37) OtherLaplacianswerealsostudied[21].

AUTHORS

MieczysławA.Kłopotek∗ –InstytutPodstaw InformatykiPolskiejAkademiiNauk,ul. JanaKazimierza5,01‑248Warszawa, Poland,e‑mail:klopotek@ipipan.waw.pl, https://home.ipipan.waw.pl/m.klopotek/. SławomirT.Wierzchoń –Instytut PodstawInformatykiPolskiejAkademii Nauk,ul.JanaKazimierza5,01‑248 Warszawa,Poland,e‑mail:stw@ipipan.waw.pl, https://home.ipipan.waw.pl/s.wierzchon/.

BartłomiejStarosta –InstytutPodstawInformatyki PolskiejAkademiiNauk,ul.JanaKazimierza5,01‑248 Warszawa,Poland,e‑mail:barstar@ipipan.waw.pl, https://home.ipipan.waw.pl/b.starosta/.

PiotrBorkowski –InstytutPodstaw InformatykiPolskiejAkademiiNauk,ul. JanaKazimierza5,01‑248Warszawa, Poland,e‑mail:p.borkowski@ipipan.waw.pl, https://home.ipipan.waw.pl/p.borkowski/.

DariuszCzerski –InstytutPodstawInformatyki PolskiejAkademiiNauk,ul.JanaKazimierza5,01‑ 248Warszawa,Poland,e‑mail:dcz@ipipan.waw.pl, https://home.ipipan.waw.pl/d.czerski/.

∗Correspondingauthor

References

[1] S.A.P.Murray, TheLibrary:AnIllustratedHistory, SkyhorsePub.:NewYork,NY,2009.

[2] S.Lloyd,“Leastsquaresquantizationin PCM”, IEEETransactionsonInformation Theory,vol.28,no.2,1982,129–137, 10.1109/TIT.1982.1056489

[3] U.Luxburgvon,“Atutorialonspectralcluster‑ ing”, StatisticsandComputing,vol.17,no.4,2007, 395–416, https://doi.org/10.1007/s11222‑ 007‑9033‑z.

[4] J.WangandY.Dong,“Measurementoftextsimi‑ larity:Asurvey”, Information,vol.11,no.9,2020, 10.3390/info11090421

[5] A.Ittoo,L.M.Nguyen,andA.vandenBosch, “Textanalyticsinindustry”, Computers inIndustry,vol.78,no.C,2016,96–107, 10.1016/j.compind.2015.12.001

[6] G.Salton,A.Wong,andC.S.Yang,“Avector spacemodelforautomaticindexing”, Communi‑ cationsoftheACM,vol.18,no.11,1975,613–620, 10.1145/361219.361220

[7] A.P.SunitaBisht,“Documentclustering:A review”, InternationalJournalofComputer Applications,vol.73,no.11,2013,26–33, 10.5120/12787‑0024.

[8] X.Rong.“word2vecparameterlearning explained”,2014.arXiv:1411.2738[cs.CL].

[9] J.H.LauandT.Baldwin,“Anempiricalevaluation ofdoc2vecwithpracticalinsightsintodocument embeddinggeneration”.In: Proceedingsofthe1st WorkshoponRepresentationLearningforNLP, Berlin,Germany,2016,78–86,https://doi.org/ 10.18653/v1/W16‑1609

[10] J.Pennington,R.Socher,andC.Manning,“GloVe: Globalvectorsforwordrepresentation”.In: A.Moschitti,B.Pang,andW.Daelemans,eds., Proceedingsofthe2014ConferenceonEmpirical MethodsinNaturalLanguageProcessing (EMNLP),Doha,Qatar,2014,1532–1543, 10.3115/v1/D14‑1162.

[11] J.Devlin,M.‑W.Chang,K.Lee,andK.Toutanova, “BERT:Pre‑trainingofdeepbidirectionaltrans‑ formersforlanguageunderstanding”, Computing ResearchRepository,vol.abs/1810.04805,2018, https://doi.org/10.48550/arXiv.1810.04805.

[12] E.SezererandS.Tekir,“Asurveyonneural wordembeddings”, ComputingResearchReposi‑ tory,vol.abs/2110.01804,2021.

[13] Y.LiandT.Yang.“Wordembeddingforunder‑ standingnaturallanguage:Asurvey”.In:S.Srini‑ vasan,ed., GuidetoBigDataApplications,83–104.SpringerInternationalPublishing,Cham, 2018.

[14] L.daCosta,I.Oliveira,andR.Fileto,“Textclassi‑ icationusingembeddings:asurvey”, Knowledge

andInformationSystems,vol.65,2023,2761–2803,https://doi.org/10.1007/s10115‑023‑ 01856‑z.

[15] L.StankevičiusandM.Lukoševičius,“Extract‑ ingsentenceembeddingsfrompretrainedtrans‑ formermodels”, ComputingResearchRepository, vol.abs/2408.08073,2024,https://doi.org/10 .3390/app14198887

[16] S.Talebi,E.Tong,A.Li,andetal.,“Exploring theperformanceandexplainabilityof ine‑tuned bertmodelsforneuroradiologyprotocolassign‑ ment”, BMCMedicalInformaticsandDecision Making,vol.24,no.40,2024,10.1186/s12911‑ 024‑02444‑z

[17] N.Kokhlikyan,V.Miglani,M.Martin, E.Wang,B.Alsallakh,J.Reynolds, A.Melnikov,N.Kliushkina,C.Araya,S.Yan, andO.Reblitz‑Richardson,“Captum:A uni iedandgenericmodelinterpretability libraryforpytorch”, ComputingResearch Repository,vol.abs/2009.07896,2020, https://doi.org/10.48550/arXiv.2009.07896

[18] M.Sundararajan,A.Taly,andQ.Yan,“Axiomatic attributionfordeepnetworks”.In: Proc.34th InternationalConferenceonMachineLearning, vol.70,2017,3319–28.

[19] N.O.Tan,J.Bensemann,D.Benavides‑Prado, Y.Chen,M.Gahegan,L.Lee,A.Y.Peng,P.Riddle, andM.Witbrock,“Anexplainabilityanalysisofa sentimentpredictiontaskusingatransformer‑ basedattention ilter”.In: Proceedingsofthe NinthAnnualConferenceonAdvancesinCognitive Systems,2021.

[20] Z.XuandY.Ke,“Effectiveandef icientspectral clusteringontextandlinkdata”.In: CIKM’16: Proceedingsofthe25thACMInternationalon ConferenceonInformationandKnowledgeMan‑ agement,2016,357–366,https://doi.org/10.1 145/2983323.2983708.

[21] S.WierzchońandM.Kłopotek, ModernClustering Algorithms,volume34of StudiesinBigData, SpringerVerlag,2018,421.

[22] I.S.Dhillon,Y.Guan,andB.Kulis,“Kernelk‑ means:Spectralclusteringandnormalizedcuts”. In: ProceedingsoftheTenthACMSIGKDDInter‑ nationalConferenceonKnowledgeDiscoveryand DataMining,NewYork,NY,USA,2004,551–556, 10.1145/1014052.1014118

[23] B.Starosta,M.A.Kłopotek,S.T.Wierzchoń, D.Czerski,M.Sydow,andP.Borkowski,“Explain‑ ablegraphspectralclusteringoftextdocu‑ ments”, PLoSOne,vol.20(2):e0313238,2025, 10.1371/journal.pone.0313238

[24] C.R.Harris,K.J.Millman,S.J.vanderWalt, etal.,“ArrayprogrammingwithNumPy”, Nature, vol.585,2020,357–362,10.1038/s41586‑020‑ 2649‑2.

[25] P.Virtanen,R.Gommers,T.E.Oliphant,etal., “SciPy1.0:Fundamentalalgorithmsforscienti ic computinginpython”, NatureMethods,vol.17, 2020,261–272,10.1038/s41592‑019‑0686‑2.

[26] L.Buitinck,G.Louppe,M.Blondel,etal.,“API designformachinelearningsoftware:experi‑ encesfromthescikit‑learnproject”, Computing ResearchRepository,vol.abs/1309.0238,2013, https://doi.org/10.48550/arXiv.1309.0238

[27] H.KimandH.K.Kim.“clustering4docsgithub repository”,2020. https://pypi.org/project/soy clustering/.

[28] H.Kim,H.K.Kim,andS.Cho,“Improvingspher‑ icalk‑meansfordocumentclustering:Fastini‑ tialization,sparsecentroidprojection,andef i‑ cientclusterlabeling”, ExpertSystemswithAppli‑ cations,vol.150,2020,113288,https://doi.org/ 10.1016/j.eswa.2020.113288

[29] H.W.Kuhn,“Thehungarianmethodforthe assignmentproblem”, NavalResearchLogistics Quarterly,vol.2,1955,83–97.

[30] D.P itzner,R.Leibbrandt,andD.Powers,“Char‑ acterizationandevaluationofsimilaritymea‑ suresforpairsofclusterings”, Knowledgeand InformationSystems,vol.19,no.3,2009,361–394, https://doi.org/10.1007/s10115‑008‑ 0150‑6

[31] E.Achtert,S.Goldhofer,H.‑P.Kriegel,E.Schubert, andA.Zimek,“Evaluationofclusterings–met‑ ricsandvisualsupport”.In: 2012IEEE28thInter‑ nationalConferenceonDataEngineering,2012, 1285–1288,10.1109/ICDE.2012.128.

[32] A.Balagopalan,H.Zhang,K.Hamidieh, T.Hartvigsen,F.Rudzicz,andM.Ghassemi, “Theroadtoexplainabilityispavedwithbias: Measuringthefairnessofexplanations”.In: 2022ACMConferenceonFairnessAccountability andTransparency,2022,1194–1206, 10.1145/3531146.3533179

[33] R.R.Hoffman,S.T.Mueller,G.Klein,andJ.Lit‑ man,“MeasuresforexplainableAI:Explana‑ tiongoodness,usersatisfaction,mentalmod‑ els,curiosity,trust,andhuman‑aiperformance”, FrontiersofComputerScience,Sec.Theoretical ComputerScience,06February2023,vol.5,2023, https://doi.org/10.3389/fcomp.2023.1096257

[34] A.Holzinger,A.M.Carrington,andH.Müller, “Measuringthequalityofexplanations:Thesys‑ temcausabilityscale(SCS).comparinghuman andmachineexplanations”, ComputingResearch Repository,vol.abs/1912.09024,2019.

[35] F.SovranoandF.Vitali,“Anobjectivemetricfor explainableAI:howandwhytoestimatethe degreeofexplainability”, ComputingResearch Repository,vol.abs/2109.05327,2021.

[36] C.Rudin,“Stopexplainingblackboxmachine learningmodelsforhighstakesdecisionsand

useinterpretablemodelsinstead”, Computing ResearchRepository,vol.abs/1811.10154,2019.

[37] H.Löfström,K.Hammar,andU.Johansson. AMetaSurveyof QualityEvaluationCriteria in ExplanationMethods,55–63.SpringerInter‑ nationalPublishing,2022.

[38] J.Zhou,A.H.Gandomi,F.Chen,andA.Holzinger, “Evaluatingthequalityofmachinelearning explanations:Asurveyonmethodsandmetrics”, Electronics,vol.10,no.5,2021, 10.3390/elec‑ tronics10050593.

[39] M.Tamajka.“Howtomeasurethequalityof explanationsofAIpredictions”. https://kinit.sk /how‑to‑measure‑the‑quality‑of‑explanations‑ of‑ai‑predictions/,2022

[40] P.Linardatos,V.Papastefanopoulos,andS.Kot‑ siantis,“ExplainableAI:Areviewofmachine learninginterpretabilitymethods”, Entropy,vol. 23,no.1,2021,10.3390/e23010018.

[41] M.Baroni,G.Dinu,andG.Kruszewski,“Don‘t count,predict!asystematiccomparison ofcontext‑countingvs.context‑predicting semanticvectors”.In:K.ToutanovaandH.Wu, eds., Proceedingsofthe52ndAnnualMeeting oftheAssociationforComputationalLinguistics (Volume1:LongPapers),Baltimore,Maryland, 2014,238–247,10.3115/v1/P14‑1023

[42] M.A.Kłopotek,S.T.Wierzchoń,B.Starosta, D.Czerski,andP.Borkowski,“Amethodforhan‑ dlingnegativesimilaritiesinexplainablegraph spectralclusteringoftextdocuments–Extended Version”, ComputingResearchRepository,vol. abs/2504.12360,2025,https://doi.org/10.485 50/arXiv.2504.12360

[43] C.Rudin,“Stopexplainingblackboxmachine learningmodelsforhighstakesdecisions anduseinterpretablemodelsinstead”, Nature MachineIntelligence,vol.1,no.5,2019,206–215, 10.1038/S42256‑019‑0048‑X

[44] P.Linardatos,V.Papastefanopoulos,and S.Kotsiantis,“ExplainableAI:Areview ofmachinelearninginterpretability methods”, Entropy2021,vol.23,no.1,2021, https://doi.org/10.3390/e23010018.

[45] S.Farquhar,J.Kossen,L.Kuhn,etal.,“Detecting hallucinationsinlargelanguagemodelsusing semanticentropy”, Nature,vol.630,2024,625–630, https://doi.org/10.1038/s41586‑024‑ 07421‑0

A.ResultsofClusteringUsing ��‐Means

Forthecompletenessofourresearch,weposedthe questionofwhetheritmakessensetouseaclustering methoddifferentfromthestraightforwardusageof thepopular ��‑meansalgorithm,thatis,whetherit makessensetoapplyGraphSpectralClustering.In table6weshowclusteringresultsofdataset0using ��‑meansdirectlyintherespectiveembeddingspace.If wecomparethiswithTable1,whichshowsclustering resultsusingGSCclustering,weseethatGSCoffers animprovementinclusteringquality.Hence,itmakes sensetobotheraboutexplainabilityinGSCinspiteof thefactthatfortheplain ��‑means,theexplanations arestraightforward.

Tables7 10presentclusteringresultsfordatasets 1‑4respectivelyfor ��‑meansclustering.Acompar‑ isonwithtables 2 5 allowstodrawsimilarconclu‑ sions:GSCoffersgenerallyanimprovementinclus‑ teringqualityespeciallyforBERTtypeembeddings. However,adrawbackforsomeBERTimplemnenta‑ tionsisvisible.UndersomeembeddingsGSCcluster‑ ingcannotbecomputedduetomassivenegativesim‑ ilaritieswhile ��‑meansclusteringispossibleinsuch cases.Thisissueissubjectofourfurtherinvestiga‑ tions.Asshownin[42],thisissuecanbesuccessfully mitigatedforGlovetypeembeddingsandwearecur‑ rentlyexperimentingwithsimilarmethodsforBERT typeembeddings,withpromisingresults.

Table6. ��‐meansclusteringofDataset0usingvarious embeddings(vectorizers)

Vectorizer F1‑avg

CountVectorizer 0.249608

TfVectorizer 0.249584

T idfVectorizer

0.348829

GloVe@wiki 0.285086

GloVe@twitter 0.239426

sBERT@all‑MiniLM‑L6‑v2 0.961594

sBERT@all‑MiniLM‑L12‑v2 0.937874

sBERT@all‑mpnet‑base‑v2 0.925444

sBERT@all‑distilroberta‑v1 0.929055

sBERT@multi‑qa‑MiniLM‑L6‑cos‑v1 0.941966

sBERT@multi‑qa‑distilbert‑cos‑v1 0.952534

sBERT@multi‑qa‑mpnet‑base‑dot‑v1 0.762779

BERT@bert‑base‑uncased#[CLS] 0.346147

BERT@vinai/bertweet‑base#[CLS] 0.492142

BERT@distilbert‑base‑uncased#[CLS] 0.445644

BERT@cardiffnlp/twitter‑roberta‑ base#[CLS] 0.540962

BERT@bert‑base‑uncased#T_AVG 0.401162

BERT@vinai/bertweet‑base#T_AVG 0.369402

BERT@distilbert‑base‑uncased#T_AVG 0.400333

BERT@cardiffnlp/twitter‑roberta‑ base#T_AVG 0.362900

Table7. ��‐meansclusteringofDataset1usingvarious embeddings(vectorizers)

sBERT@all‑MiniLM‑L12‑v2

sBERT@all‑distilroberta‑v1

Table8. ��‐meansclusteringofDataset2usingvarious embeddings(vectorizers)

Table9. ��‐meansclusteringofDataset3usingvarious embeddings(vectorizers)

Table10. ��‐meansclusteringofDataset4usingvarious embeddings(vectorizers)

GloVe@wiki 0.187456

GloVe@twitter 0.192742

sBERT@all‑MiniLM‑L6‑v2 0.984532

sBERT@all‑MiniLM‑L12‑v2 0.986152

sBERT@all‑mpnet‑base‑v2 0.990348

sBERT@all‑distilroberta‑v1 0.974033

sBERT@multi‑qa‑MiniLM‑L6‑cos‑v1 0.945448

sBERT@multi‑qa‑distilbert‑cos‑v1 0.957767

sBERT@multi‑qa‑mpnet‑base‑dot‑v1 0.972517

BERT@bert‑base‑uncased#[CLS] 0.329817

BERT@vinai/bertweet‑base#[CLS] 0.365470

BERT@distilbert‑base‑uncased#[CLS] 0.453654

BERT@cardiffnlp/twitter‑roberta‑ base#[CLS] 0.672232

BERT@bert‑base‑uncased#T_AVG 0.442688

BERT@vinai/bertweet‑base#T_AVG 0.508783

BERT@distilbert‑base‑uncased#T_AVG 0.438658

BERT@cardiffnlp/twitter‑roberta‑ base#T_AVG 0.572831

sBERT@all‑MiniLM‑L12‑v2

sBERT@all‑mpnet‑base‑v2

BERT@distilbert‑base‑uncased#[CLS] 0.516114

BERT@cardiffnlp/twitter‑roberta‑ base#[CLS]

BERT@bert‑base‑uncased#T_AVG 0.531288

BERT@vinai/bertweet‑base#T_AVG 0.432609

BERT@distilbert‑base‑uncased#T_AVG 0.523886

BERT@cardiffnlp/twitter‑roberta‑ base#T_AVG 0.505151

Submitted:15th July2024;accepted:22nd August2024

SerraAksoy

DOI:10.14313/jamris‐2026‐006

Abstract:

Skinlesionsegmentationisacriticaltaskindermatology, essentialfortheaccuratediagnosisandtreatmentof variousskinconditions,includingskincancer.Theprecise identificationoflesionboundariesinmedicalimagessig‐nificantlyhelpsinearlydetectionandeffectivemanage‐mentoftheseconditions.Inthisstudy,aU‐Netmodel wasemployedtoperformsegmentationofskinlesions, usingitsadvancedencoder‐decoderarchitectureandskip connectionstocapturefinedetailsandspatialhierarchies withintheimages.Themodelyieldedanoverallaccuracy of95.47%,aprecisionof97.21%,arecallof84.04%, andanF1scoreof90.15%.TheseresultsaffirmtheU‐Netmodel’sproficiencyinaccuratelysegmentingskin lesions.Thepreciseboundarydelineationprovidedbythe modelcanhelphealthcareprofessionalsdetectmalig‐nantlesions,whichoftenexhibitirregularboundaries, therebyimprovingdiagnosticaccuracy.Theintegration ofthismodelintoclinicalpracticecanenhancediagnostic accuracyandefficiency,reducetheworkloadonhealth‐careprofessionals,andimprovepatientoutcomes.The promisingperformanceoftheU‐Netmodelemphasizes itspotentialtorevolutionizedermatologicaldiagnostics andsupporthealthcareprofessionalsindeliveringtimely andprecisepatientcare.

Keywords: skinlesion,segmentation,U‐Net,deeplearn‐ing,dermatology,convolutionalneuralnetworks(CNNs)

1.Introduction

Skinlesionsegmentationisthecreationofa detailedmapoftheskin’slandscapetoguideder‑ matologistsinunderstandingandtreatingconditions likemelanomaandbasalcellcarcinomaaccurately. Justasacartographercarefullymarksboundaries andlandmarksonamap,segmentationidenti iesand outlineslesionswithinskinimages.Thishelpsder‑ matologistsgaugetheseriousnessofdiseases,mon‑ itorhowtheychangeovertime,anddetermineif treatmentsareworkingeffectively.Theyuseamixof traditionalmethodslikesettingthresholdsand ind‑ ingedges,andcutting‑edgetechniques,suchasdeep learningwithCNNs,U‑Net,andMaskR‑CNN,which actlikeadvancedGPSsystemsformedicalimages. Challengesincludethediverseappearancesoflesions, noisyimagedata,andtheneedforalgorithmsthatcan handlelargeamountsofdataef iciently.

Successfullysegmentinglesionsnotonlyaidsin makinginformedmedicaldecisionsbutalsodrives forwardresearchandimprovesremotemedicalcon‑ sultationsbyprovidingclearandreliableinformation foranalysis.

2.RelatedWorks

Theskin,ourbody’slargestorgan,playsavital roleasourmainbarrieragainsttheoutsideworld, coveringabout16%ofourtotalbodyweight.Because it’sconstantlyexposedtotheenvironment,theskinis susceptibletoarangeofissues,fromcommonabnor‑ malitieslikelesionstovariousappendages[1,2].Skin cancer,includingtypeslikesquamouscellcarcinoma, basalcellcarcinoma,andmalignantmelanoma,poses aseriousglobalhealththreat.Amongthese,malignant melanomaisparticularlyaggressiveandisaleading causeofcancer‑relateddeathsworldwide.In2020 alone,estimatesfromtheAmericanCancerSociety indicatedaround100,350newcasesofskincancer andmorethan6,500deathsattributedtothisdis‑ ease.Thesenumbersunderscoretheurgentneedfor effectiveprevention,earlydetection,andtreatment strategiestocombatthissigni icanthealthissue[1,3].

Skinlesionsegmentationiscrucialinmedicaldiag‑ nostics,playingaroleindetectingandtreatingcan‑ cerearly.However,accuratelydelineatingskinlesions presentschallengesduetotheirdiverseappear‑ ancesandsometimesambiguousborders,especially inbenignornevuslesions.Thiscomplexitymakes distinguishingthemfromhealthyskintissueschal‑ lenging,underscoringthenecessityforadvancedseg‑ mentationtechniquesthatcanpreciselyidentifyand differentiatedifferenttypesofskinlesions.

Inrecentyears,deeplearning‑basedalgorithms forimagesegmentationhavemadesigni icant advancementsovertraditionalmethods,aiming toenhancemedicaldiagnosticsbyreducing subjectivefactorsandimprovingef iciencyin clinicalsettings[4–7].Researchershavefocused ondevelopingef icientandstablesegmentation algorithmstoimproveperformance.Theseeffortsare crucialforaccuratelyidentifyinganddistinguishing variousmedicalconditionsfromimagingdata[8–10].

Theprimaryframeworksformedicalimageseg‑ mentationtodayincludedecoder‑encoderstructures likeU‑NetandDeepLab,whichleveragedilatedcon‑ volutionsforhandlingdifferentscalesandimproving objectboundaryaccuracy[11,12].

U‑Net,knownforitssimplicityandutility,serves asafoundationinthe ieldbuthaslimitations incomplexityandcomputationaldemands,particu‑ larlywhenintegratingmorecomplexstructureslike UNet++ andResUNet++ [13–15].Modelssuchas DeepLabV3andDeepLabV3++,basedondilatedcon‑ volutions,aimtoaddresslarger‑scaletargetsbut oftenrequiresubstantialdataandcomputational resourcesforeffectivetraining[16, 17].Multiscale featurefusioniscrucialforenhancingsegmenta‑ tionaccuracy,blendinglocalandglobalinforma‑ tionthroughstructureslikeFPN[18],MANet[19], LinkNet[20],andPSPNet[21],whichintegrateinfor‑ mationacrossdifferentlevelstoimproveoverallseg‑ mentationquality.Despitetheseadvancements,chal‑ lengesremaininoptimizingfeatureextractionalgo‑ rithms,balancingshallowhigh‑resolutioninformation loss,andaddressingbottlenecksinencoderlayers wherevitalcontextualdetailsareaggregated.Thisis criticalforadvancingsegmentationperformancein medicalimaging.

Researchersareusingtraditionalmachinelearn‑ ingtechniquestoimprovehowmedicalimagesare segmented,focusingonmakingthesemethodsmore accurate.Jaisakthietal.[22]havedevelopedamethod thatcombinesGrab‑cutandK‑meansclusteringtoout‑ lineskinlesionslikemelanoma.They’vealsoadded stepstoenhanceimagequality,likesmoothingout irregularitiesandstandardizinglightingbeforeana‑ lyzingtheimages.Meanwhile,MohanadAljanabiet al.[23]havecomeupwithatechniquethatusesarti‑ icialbeecoloniesto indthebestwaytodistinguish betweenhealthyskinandlesions.Thismethoduses fewerstepsandcanpinpointlesionswithalotof precision,whichmakesitapowerfultoolfordiag‑ nosingskinconditions.Bothapproachesshowhow researchersare indingnewwaystousemachine learningtohelpdoctorsgetmoreaccurateresults whenthey’relookingatmedicalimages.Bersethet al.[24]createdaspecialtypeofcomputerprogram calledaU‑Nettohelpdoctors indskinproblemsin picturesofskin.Theyusedamethodcalledten‑fold crossvalidationtomakesuretheirprogramwasaccu‑ rate.Meanwhile,Mishraetal.[25]cameupwitha differentwaytousecomputersto indskinproblems inpicturesofskin.

Properlydiagnosingandscreeningforskindis‑ ordersreliesheavilyonaccuratelypinpointingthe affectedareas.Segmentationofskinlesionsistricky duetofactorsliketheirdiverseshapes,theirclose‑ nesstonormalskin,andthepresenceofhair.To tacklethesechallengeseffectively,weintroducea cutting‑edgemethodthatusesdeeplearningtoauto‑ matethesegmentationofskinlesions.Thisinnovative approachaimstosimplifyandimprovetheaccuracy ofdiagnosingskinconditions,ensuringbettercare andoutcomesforpatientsdealingwithdermatologi‑ calissues.

Table1. TrainandValidationData

3.MaterialandMethod

3.1.DataPreparationandAugmentation

TheHAM10000dataset,consistingof10,015der‑ moscopicimagesofskinlesionsalongwiththeircor‑ respondingsegmentationmasks,isusedinthisstudy.

Thisdatasetincludesacomprehensivecollection ofcriticaldiagnosticcategoriesofpigmentedlesions, suchasactinickeratosesandintraepithelialcarci‑ noma(AKIEC),basalcellcarcinoma(BCC),benign keratosis‑likelesions(BKL),dermato ibroma(DF), melanoma(MEL),melanocyticnevi(NV),andvascular lesions(VASC).

Foreffectivemodeltrainingandvalidation,the datasetisdividedintotwosubsets:atrainingsetand avalidationset.Thetrainingsetconsistsof8,016 imagesandtheirrespectivemasks,whilethevalida‑ tionsetcomprisestheremaining1,999imagesand masks.Thissplitensuresarobusttrainingofthe model,givingthemodelthechancetoseealotofdata andevaluationofthemodel’sperformanceonunseen data(Table1).

Theimagesandtheircorrespondingmasksare loadedandpreprocessedusingacustomdatasetclass, ‘SkinLesionDataset‘.Thisclassisdesignedtofacili‑ tatetheretrievalofimage‑maskpairsfromspeci ied directories.Withintheclass,the‘__init__‘methodini‑ tializesthedatasetbyspecifyingthedirectoriesfor imagesandmasks,alongwithanytransformationsto beapplied.The‘__len__‘methodreturnsthetotalnum‑ berofimages,andthe‘__getitem__‘methodretrieves theimageandmaskatthespeci iedindex.Themasks areconvertedtobinaryformat,wherethepixelvalue of255isreplacedwith1forcompatibilitywiththe segmentationtask.

Toenhancethemodel,dataaugmentationtech‑ niquesareemployedonthetrainingimages.The transformationsappliedtothetrainingsetinclude resizingtheimagesto224x224,rotationwithaprob‑ abilityof1.0,horizontal lippingwithaprobability of0.5,vertical lippingwithaprobabilityof0.1,and normalizationwithmeanandstandarddeviationset to[0.0,0.0,0.0]and[1.0,1.0,1.0]respectively,with themaximumpixelvaluesetto255.0.

Forthevalidationset,asimpleraugmentation pipelineisemployedtomaintainconsistencyduring modelevaluation.Thispipelineinvolvesresizingthe imagestothesamedimensionsasthetrainingimages andnormalizingthepixelvaluesusingthesamemean, standarddeviation,andmaximumpixelvaluesettings asthetrainingset.Thesepreprocessingandaugmen‑ tationstepsarecrucialforimprovingthemodel’sgen‑ eralizationcapabilityandperformanceinaccurately segmentingskinlesions.

3.2.ProposedModel

TheU‑Netarchitecturewaschosenforitswell‑ establishedeffectivenessinbiomedicalimageseg‑ mentation.Themodelfeaturesasymmetricencoder‑ decoderstructurethatexcelsinpreciselocalization throughitsskipconnections.

Theseconnectionsenabletheintegrationof detailed,high‑resolutionfeaturesfromtheencoder withupsampledfeaturesinthedecoder,whichis crucialforaccurateskinlesionsegmentation.While FullyConvolutionalNetworks(FCNs)offerend‑to‑ endsegmentation,theymaylose ine‑graineddetails becausetheylackskipconnections.TheDeepLab modelsuseatrousconvolutionsandspatialpyramid poolingtocapturemulti‑scalecontextbutare computationallyintensiveandrequirelargedatasets. SegNetisef icientinmemoryusagethroughitsuseof max‑poolingindices,thoughitmightcompromiseon localizationaccuracy.AttentionU‑Net,anadvanced versionofU‑Net,incorporatesattentionmechanisms tohighlightrelevantfeaturesbutaddscomplexity. Despitethesealternatives,U‑Netremainstheoptimal choiceforitsprovenperformanceandsuitabilityin managinghigh‑resolutionmedicalimages,ensuring robustandaccuratesegmentationcrucialforearly diagnosisandtreatmentofskinconditions.

Theproposedmodelforsegmentingskinlesions istheU‑Netarchitecture,ahighlyeffectiveconvo‑ lutionalneuralnetwork(CNN)designedspeci ically forbiomedicalimagesegmentationtasks.TheU‑ Netarchitectureischaracterizedbyitssymmetric encoder‑decoderstructure,whichfacilitatesprecise localizationrequiredforaccuratesegmentation.The

encoder,orcontractingpath,isresponsibleforcap‑ turingthecontextoftheinputimagebyprogressively downsamplingitthroughaseriesoflayers.

Eachlayercomprisestwo3x3convolutionaloper‑ ationsfollowedbybatchnormalizationandReLUacti‑ vationsandconcludeswitha2x2max‑poolingopera‑ tionthathalvesthespatialdimensionswhiledoubling thenumberoffeaturechannels.Thisdownsampling processallowsthenetworktocaptureincreasingly abstractandhigh‑levelfeaturesoftheimage.

Ontheotherhand,thedecoder,alsocalledthe expansivepath,strivestorecreatethespatialinfor‑ mationoftheinputimage.Thisinvolvesupsampling layersusingtransposedconvolutionallayers,followed bytheconcatenationofthefeaturesfromtheencoder networkusingskipconnections.Theseskipconnec‑ tionsaresigni icantastheyallowthemodeltoretain thehigh‑resolutioninformationthatisdiscardeddur‑ ingthedownsamplingprocess.Afterconcatenation, eachupsamplingstepincludestwo3x3convolutional operationswithbatchnormalizationandReLUacti‑ vations,similartotheencoder.Thisprocessreduces thenumberoffeaturechannelswhileincreasingthe spatialdimensions,effectivelyrecoveringtheoriginal imagesize(Figure1).

Atthedeepestpartofthenetwork,thebottleneck layerprocessesthesmallestspatialrepresentation withthehighestnumberoffeaturechannels,imple‑ mentedusingthesamedoubleconvolutionapproach. The inallayeroftheU‑Netarchitectureemploysa1x1 convolutiontomapthefeaturemapstoasingleoutput channel,whichisessentialforbinarysegmentation tasks.Theoutputisthenpassedthroughasigmoid

Figure1. U‐NetArchitecture

activationfunction,whichsquashesthevaluestothe range[0,1],producingaprobabilitymapthatindi‑ catesthelikelihoodofeachpixelbelongingtothe lesion.

Forthissegmentationtask,themasksarecon‑ vertedtoabinaryformatwherepixelvaluesof255 arereplacedwith1,ensuringcompatibilitywiththe binaryclassi icationrequiredbythemodel.Thesig‑ moidactivationfunction’soutputisthresholdedat 0.5toproduceabinarymask,wherepixelswitha probabilitygreaterthan0.5areclassi iedaspartof thelesion(assignedavalueof1),andthosewitha probabilityof0.5orlessareclassi iedasbackground (assignedavalueof0).

ByusingtheU‑Netarchitectureandincorporat‑ ingappropriatedataaugmentationtechniques,the proposedmodelaimstoachievehighaccuracyand robustnessinsegmentingskinlesions.Thisrobust segmentationcapabilityiscrucialforfacilitatingearly diagnosisandtreatmentofvariousskinconditions, ultimatelycontributingtoimprovedpatientoutcomes.

3.3.ExperimentalSetup

Theexperimentalsetupwasconductedonahigh‑ performancecomputingenvironmentequippedwith anIntelCorei9processorandanNVIDIAGeForce RTXGPU.Thissetupensuredtheef icienthandling ofthecomputationaldemandsofdeeplearningtasks, particularlytheextensivematrixoperationsrequired fortrainingconvolutionalneuralnetworks.Theexper‑ imentswereprogrammedusingPyTorch,awidely useddeeplearningframeworkknownforitsdynamic computationalgraphand lexibility,whichallowedthe implementationandexperimentationwiththeU‑Net architecture.

DatapreparationbeganwiththeHAM10000 dataset,whichconsistsof10,015dermatoscopic imagesofvariousskinlesionsandtheircorresponding segmentationmasks.Thedatasetwasdividedinto trainingandvalidationsets,with8,016imagesallo‑ catedfortrainingand1,999imagesforvalidation.

Acustomdatasetclass,‘SkinLesionDataset‘ retrievestheimage‑maskpairs,convertingthemasks toabinaryformatwherepixelvaluesof255are replacedwith1toensurecompatibilitywiththe segmentationtask.Thetrainingsetwasaugmented usingrotation,verticalandhorizontal lips,and transformedsimilarlytothevalidationset,whilethe validationsetwasonlyresized,convertedtotensors, andnormalized.Thetrainandvalidationdataloaders werecreatedwithabatchsizeof32.TheU‑Net architecturewasemployedforthesegmentationtask. Duringmodeltraining,thebinarycross‑entropyloss functionwasusedtooptimizethemodelparameters, minimizingthedifferencebetweenthepredicted probabilitymapandthegroundtruthmask.The optimizationprocessusedtheAdamoptimizerwitha learningrateof0.001.Thetrainingprocessinvolved iteratingoverbatchesoftrainingdata,wherethe imagesandmaskswereloaded,augmented,and passedthroughthemodeltogeneratepredictions. Thepredictionswerethresholdedat0.5toproduce

binarymasks,whichwerethencomparedwiththe groundtruthmaskstocomputeevaluationmetrics (Figure2).

Validationoftheresultsinvolvedcarefullyclassi‑ fyingthemodel’spredictionsintocategoriessuchas TruePositives(TP),FalsePositives(FP),TrueNega‑ tives(TN),andFalseNegatives(FN).Thiswasdoneby comparingeachpixelinthepredictedbinarymasks withthecorrespondingpixelinthegroundtruth masks.TPreferredtopixelscorrectlyidenti iedas partofthelesion,FPtopixelsmistakenlyidenti ied aspartofthelesion,TNtocorrectlyidenti iedback‑ groundpixels,andFNtopixelswherethelesionwas missed.Theinitialclassi icationwashandledauto‑ maticallythroughPythonscripts,whichcompared everypixel’spredictiontoitstruelabel.

Toensureaccuracy,thisautomatedprocesswas supplementedwithmanualcheckswhereexperts reviewedselectedresultstocon irmtheclassi ica‑ tionsandensuretheygenuinelyre lectedthemodel’s performance.Thisthoroughapproachensuredthat metricslikeaccuracyandtheDicecoef icientaccu‑ ratelyrepresentedthemodel’ssegmentationabilities andprovidedreliableinsightsintoitsperformance.

Modelperformancewasassessedusingaccuracy andtheDicecoef icient.Accuracymeasuredthepro‑ portionofcorrectlypredictedpixels(bothlesionand background)outofthetotalnumberofpixels,provid‑ ingabasicmeasureofoverallcorrectness.TheDice coef icient,alsoknownastheF1score,waspartic‑ ularlyusefulforevaluatingsegmentationtasks,asit measuredtheoverlapbetweenthepredictedmask andthegroundtruthmask.TheDicecoef icientwas calculatedastwicetheoverlapbetweenthepredicted andgroundtruthmasks,dividedbythetotalnumber ofpixelsinbothmasks(Equation1).Furthermore,the confusionmatrixwasvisualizedasaheatmap,with groundtruthlabelsontherowsandpredictedlabels onthecolumns,whichhighlightedwherethemodel succeeded,whichmeanshighTPandTNandwhere itstruggles,whichishighFPandFNinaccurately segmentingobjects.Unliketraditionalclassi ication taskswherepredictionsaremadeforentireinstances, segmentationinvolvespredictinglabelsforeachpixel. Theconfusionmatrixforsegmentation,knownasa pixel‑wiseconfusionmatrix,categorizespredictions intofourgroups:TruePositives(correctlyidenti ied objectpixels),FalsePositives(incorrectlyidenti ied asobjectpixels),TrueNegatives(correctlyidenti‑ iedbackgroundpixels),andFalseNegatives(incor‑ rectlyidenti iedasbackgroundpixels).Inaddition totheconfusionmatrix,plotsdepictingmetricslike Dicescore,trainingloss,accuracy,andvalidationloss andaccuracyareessential.Together,thesevisualiza‑ tionsofferacomprehensiveviewofasegmentation model’sprogression,helpingtore ineandoptimize itformorepreciseanddependablesegmentationsin applications.

4.ResultsandDiscussion

4.1.Results

TheperformanceoftheU‑Netmodelforskin lesionsegmentationwasevaluatedover20epochs, employingacomprehensivesetofmetricstomonitor andanalyzeitseffectiveness.Throughoutthetrain‑ ingprocess,aconsistentandsubstantialdecreasein traininglosswasobserved,startingfromaninitial valueof0.2788inthe irstepochandprogressively reducingto0.1232bythetwentiethepoch.Thissteady declineindicatesthemodel’sincreasingabilityto minimizepredictionerrorseffectively.Concurrently, thetrainingDicescoreexhibitedcontinuousimprove‑ ment,beginningat0.8032andrisingto0.9102by the inalepoch,emphasizingthemodel’sincreasing performanceinaccuratelysegmentingskinlesions. Validationmetricsdemonstratedremarkablestability, withvalidationaccuracyremainingconsistentlyhigh andvalidationDicescoresshowingminimal luctu‑ ationthroughoutthetrainingepochs.Thisstability

suggeststhatthemodelmaintainedrobustgeneral‑ izationcapabilitiesandeffectivelyavoidedover itting (Figure3).

Theconfusionmatrixfurtherillustratedthe model’sperformance,showingahighnumberof truepositives(20,781,788)andtruenegatives (74,978,631),alongwithrelativelylowercounts offalsepositives(595,742)andfalsenegatives (3,945,663).Fromthisconfusionmatrix,several keyclassi icationmetricswerederived:anoverall accuracyof95.47%,aprecisionof97.21%,arecall of84.04%,andanF1scoreof90.15%.Thesemetrics collectivelyindicatethatthemodelachievedahigh levelofaccuracy,demonstratingitsreliabilityin correctlyidentifyingbothlesionandnon‑lesion pixels.Thehighprecisionvalueunderscoresthe model’sabilitytominimizefalsepositives,thereby ensuringthatmostoftheidenti iedlesionsare indeedtruelesions.Therecallvalue,re lectingthe model’seffectivenessinidentifyingtruepositive cases,suggeststhatwhilesomefalsenegativeswere

Figure2. ExperimentalSetup

Figure3. TrainingLoss,DiceScore,andValidationMetricsover20Epochs

present,themajorityofactuallesionswereaccurately detected.TheF1score,whichbalancesprecision andrecall,furthercon irmsthemodel’srobust performanceanditspro iciencyinhandlingthetask ofskinlesionsegmentation(Figure4).

Generally,theresultsaf irmtheU‑Netmodel’shigh performanceinmedicalimagesegmentation,demon‑ stratingbothaccuracyandreliabilityinidentifying skinlesions.The indingsshowthemodel’spotential asavaluabletoolinclinicalsettings,whereaccurate andef icientlesionsegmentationiscriticalfordiag‑ nosisandtreatmentplanning.

4.2.Discussion

Tojudgethemodel’saccuracy,predictionswere comparedtogroundtruthmaskspixelbypixel,using metricslikeaccuracyandtheDicecoef icient.While thesenumbersprovideaclearpictureofhowwellthe modelperformsinacontrolledenvironment,applying themtoreal‑worldsettingsrequiresmoreconsider‑ ation.Real‑worldeffectivenessdependsonhowwell themodeladaptstothevarietyandcomplexityof skinconditionsseenineverydayclinicalpractice.High accuracyandDicescoresintestingarepromising,but it’sessentialtoevaluatehowthemodelperformswith differentpatientpopulationsandclinicalscenarios.

Therearesomelimitationstotheproposed approach.TherelianceontheHAM10000dataset meansthemodelmightnotcoverthefullrangeofskin conditionsseeninreallife,whichcouldaffectitsper‑ formancewithlesscommonoratypicalcases.While theU‑Netarchitectureiseffectiveforsegmentation, itmightmisssubtledetailsorstrugglewithextreme variationsinlesionappearance,potentiallylimitingits robustnessoutsideofthedatasetused.

IntegratingAIintomedicalpracticerequirescare‑ fulconsiderationofvariousfactors.Errortoleranceis crucial;falsepositivesornegativescanimpactpatient care,sotheAIshouldactasasupportivetoolrather thanareplacementforclinicaljudgment.Itshould

provideanadditionallayerofanalysis,allowingdoc‑ torstomakemoreinformeddecisionswhilemain‑ tainingcompassionateandpatient‑centeredcare.The AI’sroleistosupportandenhancemedicalprac‑ tice,nottoreplacethehumantouch.Explainability iskeyforbothpatientsandmedicalprofessionals. TheU‑Netmodel’sstructure,withitsclearencoder‑ decoderframework,helpsmakeitsoperationssome‑ whattransparent.However,makingtheAI’sdecision‑ makingprocessmoreunderstandablethroughvisual aidslikeheatmapsorattentionmapscanhelpbridge thegapbetweencomplextechnologyandpractical use.Thistransparencyhelpsbuildtrustandensures thatbothpatientsanddoctorscaneffectivelyusethe model.

Thebene itsofthisapproacharesubstantial.By improvingtheaccuracyandef iciencyofskinlesion segmentation,themodelcansupportearlierdiagno‑ sisandbettertreatmentoutcomes.Itenablesmed‑ icalprofessionalstohandlelargeamountsofdata moreeffectively,reducingtheriskofdiagnosticerrors andsupportingconsistentevaluations.Themodel’s impactcanbefurtherenhancedthroughongoing improvements,real‑worldtesting,andcollaboration withclinicianstoensureitmeetspracticalneedsand improvespatientcare.

5.Conclusion

Inconclusion,theU‑Netmodeldemonstrated excellentperformanceinthetaskofskinlesionseg‑ mentation,asevidencedbytheconsistentlydecreas‑ ingtrainingloss,increasingDicescores,stablevalida‑ tionmetrics,andhighclassi icationmetricsderived fromtheconfusionmatrix.Theseresultssupportthe model’shighperformanceinaccuratelysegmenting medicalimagesandidentifyingskinlesions.

Lookingahead,thepotentialapplicationsofthis modelinhealthcarearevastandpromising.The deploymentofsuchanadvancedsegmentationtool cansigni icantlyhelphealthcareprofessionalsinsev‑ eralways.Firstly,itcanenhancetheaccuracyandef i‑ ciencyofdiagnosingskinconditions,enablingquicker andmorepreciseidenti icationoflesions.Thiscan beparticularlybene icialinearlydetectionofskin cancers,wheretimelydiagnosisiscrucialforeffective treatmentandimprovedpatientoutcomes.

Additionally,theuseofU‑Netinautomateddiag‑ nosticsystemscanhelpreducetheworkloadonder‑ matologists,allowingthemtofocusonmorecomplex casesandpatientcare.Byprovidingreliablesegmen‑ tationresults,themodelcanserveasavaluablesec‑ ondopinion,assistingcliniciansinmakinginformed decisions.Thiscanbeespeciallyusefulinremoteor under‑resourcedareaswhereaccesstospecialized medicalexpertiseislimited.

Furthermore,integratingtheU‑Netmodelinto telemedicineplatformscanfacilitateremoteconsul‑ tationsandcontinuousmonitoringofpatientswith chronicskinconditions.Patientscancaptureimages oftheirskinlesionsusingmobiledevices,andthe

Figure4. ConfusionMatrixforSkinLesionSegmentation

modelcanprovideimmediateanalysis,aidinginongo‑ ingdiseasemanagementandfollow‑up. Futureresearchanddevelopmentshouldfocuson re iningthemodel’sperformanceandexpandingits applicationtoothertypesofmedicalimaging.Col‑ laborativeeffortsbetweendatascientists,clinicians, andhealthcareprofessionalsareessentialtoensure themodel’srobustness,reliability,andintegration intoclinicalwork lows.Ethicalconsiderations,such aspatientprivacyanddatasecurity,mustalsobe addressedtobuildtrustandacceptanceamongusers. Overall,theimplementationofadvancedmodelslike U‑Netinclinicalpracticeholdsthepotentialtorevolu‑ tionizedermatologyandimprovepatientcarethrough enhanceddiagnosticaccuracy,ef iciency,andaccessi‑ bility.

AuthorContribution SAhelpedinconceptualiza‑ tion,software,validation,writing—originaldraft, resourcesanddatacuration.SAhelpedinmethodol‑ ogy,writing—review&editing,supervision,revision, fundingacquisitionandformalanalysis.

DataAvailability Allthedatasetsusedinexperimen‑ tationareavailableonlineontheinternet.

ConflictofInterest Authorsdeclarethattheyhave noknowncompeting inancialinterestsorpersonal relationshipsthatcouldhaveappearedtoin luence theworksubmittedforpublication.

AUTHOR

SerraAksoy∗ –InstituteofComputerScience, LudwigMaximilianUniversityofMunich (LMU),Oettingenstrasse67,80538Munich, Germany,e‑mail:serra.aksoy@campus.lmu.de, https://orcid.org/0009‑0001‑2391‑9437.

∗Correspondingauthor

References

[1] J.Lietal.,“DSEUNet:AlightweightU‑Netfor dynamicspacegroupingenhancementforskin lesionsegmentation,” ExpertSystemswithAppli‑ cations,vol.255,Art.no.124544,Dec.2024;doi: 10.1016/j.eswa.2024.124544.

[2] R.J.Hayetal.,“Theglobalburdenofskindis‑ easein2010:Ananalysisoftheprevalenceand impactofskinconditions,” JournalofInvestiga‑ tiveDermatology,vol.134,no.6,pp.1527–1534, Jun.2014;doi:10.1038/jid.2013.446.

[3] P.Mathuretal.,“Cancerstatistics,2020:Report fromnationalcancerregistryprogramme,India,” JCOGlobalOncology,no.6,pp.1063–1075,Nov. 2020;doi:10.1200/GO.20.00122.

[5] P.Tangetal.,“Ef icientskinlesionsegmentation usingseparableU‑Netwithstochasticweight averaging,” ComputerMethodsandProgramsin Biomedicine,vol.178,pp.289–301,Sep.2019; doi:10.1016/j.cmpb.2019.07.005.

[6] R.Iranpoor,A.S.Mahboob,S.Shahbandegan,and N.Baniasadi,“Skinlesionsegmentationusing convolutionalneuralnetworkswithimprovedU‑ Netarchitecture,” Proc.6thIranianConf.Signal ProcessingandIntelligentSystems(ICSPIS),2020, pp.1–5;doi:10.1109/ICSPIS51611.2020.93495 77.

[7] L.Liu,L.Mou,X.X.Zhu,andM.Mandal,“Skin lesionsegmentationbasedonimprovedU‑Net,” Proc.IEEECan.Conf.ofElectricalandComputer Engineering(CCECE),2019,pp.1–4;doi:10.110 9/CCECE.2019.8861848.

[8] G.Litjensetal.,“Asurveyondeeplearningin medicalimageanalysis,” MedicalImageAnalysis, vol.42,pp.60–88,Dec.2017;doi:10.1016/j.me dia.2017.07.005.

[9] L.Qian,H.Huang,X.Xia,Y.Li,andX.Zhou, “AutomaticsegmentationmethodusingFCN withmulti‑scaledilatedconvolutionformedical ultrasoundimage,” Vis.Comput.,vol.39,no.11, pp.5953–5969,Nov.2023;doi:10.1007/s003 71‑022‑02705‑w.

[10] G.Zhan,W.Wang,H.Sun,Y.Hou,andL. Feng,“Auto‑CSC:Atransferlearningbasedauto‑ maticcellsegmentationandcountframework,” CyborgBionicSystems,vol.2022,2022,Art.no. 9842349;doi:10.34133/2022/9842349.

[11] X.Li,H.Huang,H.Zhao,Y.Wang,andM.Hu, “Learningaconvolutionalneuralnetworkfor propagation‑basedstereoimagesegmentation,” Vis.Comput.,vol.36,no.1,pp.39–52,Jan.2020; doi:10.1007/s00371‑018‑1582‑y.

[12] O.Ronneberger,P.Fischer,andT.Brox, “U‑Net:Convolutionalnetworksfor biomedicalimagesegmentation” arXiv,2015; http://arxiv.org/abs/1505.04597

[13] Z.Zhou,M.M.RahmanSiddiquee,N.Tajbakhsh, andJ.Liang,“U‑Net++:AnestedU‑Netarchitec‑ tureformedicalimagesegmentation,” inDeep LearninginMedicalImageAnalysisandMulti‑ modalLearningforClinicalDecisionSupport,D. Stoyanovetal.,Eds., LectureNotesinComputer Science,vol.11045,Springer,2018,pp.3–11;doi: 10.1007/978‑3‑030‑00889‑5_1.

[14] Z.Zhang,Q.Liu,andY.Wang,“Roadextractionby deepresidualU‑Net,” IEEEGeosci.RemoteSens. Lett.,vol.15,no.5,pp.749–753,May2018;doi: 10.1109/LGRS.2018.2802944.

[15] D.Jhaetal.,“ResUNet++:Anadvancedarchi‑ tectureformedicalimagesegmentation.” arXiv, 2019;http://arxiv.org/abs/1911.07067

[4] J.Wu,E.Z.Chen,R.Rong,X.Li,D.Xu,andH. Jiang,“SkinlesionsegmentationwithC‑UNet,” Proc.41stAnnu.Int’lConf.IEEEEng.Med.Bio.Soc. (EMBC),2019,pp.2785–2788;doi:10.1109/EM BC.2019.8857773.

[16] L.C.Chen,G.Papandreou,F.Schroff,andH. Adam,“Rethinkingatrousconvolutionfor semanticImagesegmentation.” arXiv,2017; http://arxiv.org/abs/1706.05587

[17] L.C.Chen,Y.Zhu,G.Papandreou,F. Schroff,andH.Adam,“Encoder‑decoder withatrousseparableconvolutionfor semanticimagesegmentation.” arXiv,2018; http://arxiv.org/abs/1802.02611

[18] T.Y.Lin,P.Dollár,R.Girshick,K.He,B. Hariharan,andS.Belongie,“Featurepyramid networksforobjectdetection.” arXiv,2017; http://arxiv.org/abs/1612.03144

[19] T.Fan,G.Wang,Y.Li,andH.Wang,“MA‑Net: Amulti‑scaleattentionnetworkforliverand tumorsegmentation,” IEEEAccess,vol.8,pp. 179656–179665,2020;doi:10.1109/ACCESS .2020.3025372.

[20] A.ChaurasiaandE.Culurciello,“LinkNet: Exploitingencoderrepresentationsfor ef icientsemanticsegmentation,” Proc. IEEEVisualCommunicationsandImage Processing(VCIP),2017,pp.1–4;doi: 10.1109/VCIP.2017.8305148.

[21] H.Zhao,J.Shi,X.Qi,X.Wang,andJ.Jia, “Pyramidsceneparsingnetwork.” arXiv,2017; http://arxiv.org/abs/1612.01105

[22] S.M.Jaisakthi,P.Mirunalini,andC.Aravindan, “Automatedskinlesionsegmentationofdermo‑

scopicimagesusingGrabCutandk‑meansalgo‑ rithms,” IETComputerVision,vol.12,no.8,pp. 1088–1095,Dec.2018;doi:10.1049/iet‑cvi.20 18.5289.

[23] M.Aljanabi,Y.E.Ozok,J.Rahebi,andA.S.Abdul‑ lah,“Skinlesionsegmentationmethodforder‑ moscopyimagesusingarti icialbeecolonyalgo‑ rithm,” Symmetry,vol.10,no.8,p.347,Aug. 2018;doi:10.3390/sym10080347.

[24] M.Berseth,“ISIC2017‑Skinlesionanalysis towardsmelanomadetection.” arXiv,2017;doi: 10.48550/ARXIV.1703.00523.

[25] R.MishraandO.Daescu,“Deeplearningforskin lesionsegmentation,” Proc.IEEEInt’lConf.Bioin‑ formaticsBiomedicine(BIBM),2017,pp.1189–1194;doi:10.1109/BIBM.2017.8217826.

ANALYSISOFHISTOGRAMASYMMETRYFORWASTERECOGNITION

Submitted:22nd May2022;accepted:25th July2024

JanuszBobulski,KamilaPasternak

DOI:10.14313/jamris‐2026‐007

Abstract:

Despitemanyyearsofeffortandresearch,thecurrent wastemanagementproblemremains.Sofar,nofully effectivewastemanagementsystemhasbeendevel‐oped.Manyprogramsandprojectsimprovestatisticson thepercentageofwasterecycledeveryyear.Modern computervisiontechniquessupportedbyartificialintel‐ligenceareworthusingintheseefforts.Inthearticle, wepresentamethodofidentifyingplasticwastebased ontheasymmetryanalysisoftheimage’shistogram containingthewaste.Themethodissimplebuteffective ( 94% ),allowingittobeimplementedondeviceswith lowcomputingpower,particularlymicrocomputers.Such deviceswillbeusedbothathomeandinwaste‐sorting plants.

Keywords: environmentalprotection,imageprocessing, computervision,wastemanagement

1.Introduction

Nowadays,environmentalprotectionisavery importantissue.Recyclingisoneofthemostcru‑ cialmethodsusedtoprotecttheenvironment,and involvesrecoveringrawmaterialsbytransforming substancesormaterialscontainedinwasteinthepro‑ ductionprocesstoobtainthesubstanceormaterialfor thefateofprimaryorotherpurposes.Itsmaingoal istoreducethewastestoredinland illsandconserve naturalresources.InmanyEuropeancountries,waste segregationisdoneinhouseholds,i.e.atthebeginning oftherecyclingpipeline,andinvolvesdividingrubbish intogroupssuchasmetal,glass,plastic,paper,and organicwaste.Thisapproachmakesusingselective automatictechniquesmucheasierthanformunicipal solidwaste.However,mostwasteisstillcollectedas mixedwaste.Therefore,itisreasonabletostriveto reprocesswastematerialsmoreeffectively,andan alternativetoamanual‑automaticsortingprocessis highlysought‑after.Withthedevelopmentofarti i‑ cialintelligence,deeplearning,andotherintelligent technologies,itispossibletoreducethemanpower andmaterialresourcesrequiredforthewastesorting process.Therefore,themaingoalofthispaperisto proposeanef icientsystemforwasteclassi ication. goalofthispaperistoproposeanef icientsystemfor wasteclassi ication.

2.RelatedWorks

Twodifferentcategoriesofresearchonwaste classi icationmethodscanbefoundinthelitera‑ ture:traditionalmethodsandneuralnetworkmeth‑ ods.Anexemplarytraditionalapproachisappliedin [1],whichpresentsaBayesiancomputationalframe‑ workformaterialcategoryrecognition;theproposed augmentedLatentDirichletAllocation(aLDA)model achievesa 44.6% recognitionrate.Anexistingman‑ ualengineeringmodel,animprovedconventional machinelearningalgorithm,andarandomforestclas‑ si ierareusedin[2]toobtainthebesteffectand improvepredictionqualityforemptyingrecycling containers.In[3],amathematicalstatisticsmethod isproposedtoexpressindividualboundedrationality andusethespeci icgraphstructureofascale‑free networktorepresentthegroupstructure.

Theresultspresentedinthispapershouldhave apositiveeffectonwasteclassi ication.Itshouldbe notedthattraditionalmachinelearningmethodsneed thecalibrationofalargeamountoftrainingdata.Algo‑ rithmssuchask‑NearestNeighbor(kNN)andrandom forest(RF)performahugeamountofcalculations, andthuscannot itthedataandbalancesamples well.Therefore,itcanappearthattraditionalmachine learningtechnologiesarenotasuitablechoicefor wasteclassi ication.However,theadvantageofneu‑ ralnetworkmethods(speci icallytheconvolutional neuralnetwork)overthetraditionalmachinelearning approachisshownin[4].Accuracylevelsobtained usingkNN,SupportVectorMachine(SVM)andRF were 88%,85%,and 80%,respectively.Bycompari‑ son,testaccuraciesof93%and91%wereachieved using,apre‑trainedVGG‑16CNNandAlexNetCNN respectively.Thecomparisonofresultsobtainedwith traditionalandneuralnetworkapproachescanalsobe seenin[5].

Therearemanyresearchworksinthewastesort‑ ingliteraturethatuseneuralnetworkmethods.In[6], publishedin2016,the irstimportantresultsinwaste sortingusingdeeplearningwereobtained,leading tothedevelopmentofTrashNet,amunicipalwaste database.databasewasusedbyauthorstotraintwo classi iers,SVM(SupportVectorMachine)andCNN (ConvolutionalNeuralNetwork),toclassifyimagesof wasteintosixcategories:metal,paper,glass,plastic, trash,andcardboard.Theformerachievedanaccu‑ racyof63%,whilethelatterdidnotlearnwellbecause ofthehyper‑parametersetup,andonly22%accuracy wasachieved.

Followingtheresultsof[6],thesamedatasetwas augmentedin[7]andusedtotrainFasterR‑CNN, whichobtainedabettermeanaverageprecisionof 68.3%.

FurtherresearchontheTrashNet(orTrashNet withsomeaugmentation)datasethasprovidedbetter results.Forexample,avalidationaccuracyof88.42% wasachievedin[8]withVGG‑19CNN.Theauthors performedsomeadjustmentstothehyperparameters, architecture,andclassi icationonthefullyconnected layers.Aprecisionof 84.2% andarecallof 87.8% wereobtainedin[9]usingaFasterR‑CNNbasedon InceptionV2andpre‑trainedontheMSCOCOdataset. Thestudy[10]experimentedwithseveraldifferent deepCNNarchitectures—forexample,DenseNet121, withatestaccuracyof95%,andInception‑ResNetV2, withatestaccuracyof87%.Thesamestudyproposed anovelarchitecturespeci ictotherecyclingmaterial dataset,RecycleNet,whichobtainedatestaccuracyof 81%.

In[11],theresultsshowedatestaccuracyof87% afterusinga50‑layerresidualnetwork(ResNet50) astheextractorwithanSVMclassi ier.Averyhigh accuracy( 98.7% )wasachievedin[5]byusing MobileNetV2forfeatureextractionandanSVMclassi‑ ier.In[12],severaltypesofCNNareappliedtomunic‑ ipalwasteidenti ication.Twotypesofobjectdetec‑ torsarestudiedinthispaper:SingleShotDetectors (SSD),whicharefastandabletodetectlargeobjects, andRegionalProposalNetwork(RPN),whichisvery goodatidentifyingsmallobjects,butisslowerthan SSDnetworks.Thehighestaccuracy(97.63%)was obtainedwithSSDMobileNetV2.TheRPNmodel— FasterR‑CNNarchitecturebasedonInception‑ResNet —achieved95.76%accuracy.

TheliteratureshowsthattheTrashNetdataset (and/oritsaugmentations)arewidelyused[4–12]. However,therearealsoauthorswhousedtheirown datasetfortheirresearch.In[13],forexample,the LabelledWasteintheWilddatasetisproposedand usedfortrainingtheFasterR‑CNN,whichobtained 86%ofthemeanaverageprecision.Acustomgarbage datasetfortrainingamultilayerhybriddeeplearn‑ ingmodel(MLH)forwasteclassi icationwasdevel‑ opedin[14],demonstratingthattheMLHapproach canachievehigherclassi icationperformancethanthe CNN‑onlymodel.Thisapproachyieldedaccuraciesof 98.2%and91.6%undertwodifferenttestingscenar‑ ios.

Amultilayerhybridconvolutionalneuralnetwork wasalsoproposedasawasteclassi icationmethodin [15]—anotherstudybasedontheTrashNetdataset ‑andachievedanaccuracyof 92.6%.Anotherinter‑ estingstudycanbefoundin[16],whichproposeda deepneuralnetworkbasedonFasterR‑CNNtodetect coastalwaste;theauthorscreatedanewwasteobject datasetnamedIST‑Waste,andpresentedamodelthat obtained83%ofthemeanaverageprecision.

3.MaterialsandMethods

Inmanycountries—butnotall—pre‑sortingof garbagealreadyoccursathome.

Therefore,insomesortingplants,itisnecessary tosortwasteintoindividualfractions.Thisisatime‑ consumingandcostlyjob,andthatiswhyautomatic sortingsystemsareappearingmoreandmoreoften. Inthisstudy,weproposeasimplemethodbasedon theanalysisofthehistogramsofphotosofwaste.A camerawillbeplacedonaconveyorbelt,andthe capturedphotowillbesenttoacomputerforanalysis anddecision‑making.Then,eachtypeofrubbishis directedtotheappropriatecontainerwiththehelpof amechanicalarm.Anotherwaytousetheproposed methodiswithaportablemicrocomputerthatan employeecanusetoclassifytypesofwaste.Thebasic prerequisitewhendevelopingthisalgorithmwasthat itshouldbesimpleandfast,sothatitcouldbeusedin thesortingplantinrealtime.

First,weloadtheimage,andacascadingobject detectorusestheViola‑Jonesalgorithmtodetectplas‑ ticwasteinthedigitalimage.Inthepreliminarytests, weadaptedthedetectortoourtask,teachingithowto detectgarbageusingimagesfromthedatabaseusedin theexperiment.Afterdetectingtheobject,theregion ofinterest(Rol)isextractedfromtheRGBimage.In thenextstep,wecomputeahistogramforeachR,G andBcomponentoftheRoI.Thehistogramisthen analysedbycomparingthesumsoftherangesofthe starting(A)andending(B)partsofthehistograms. Forexample,wemightaddthe irst100andlast100 elementsofthehistogramtogetherandcomparethe twosums.Inthecaseofplastic,the irstsumwillbe higher,whileinthecaseofotheropaquematerials, thesecondsumwillbehigher(Figures2and3).Inthe lastphase,adecisionismadetoclassifythefacilityas ”Plastic”or”NotPlastic.”

Algorithm:

loadthephotoI; detectionoftheobjectDonI; selecttheareaI2fromIcontainsobjectD; calculatethehistogramofI2foreachRGBcompo‑ nentseparately; selectrangesAandB; calculatethesumofelementsrangeAandB; comparesums; makedecision:Plastic/notPlastic

InFigures1and2,wecanseethecalculatedhis‑ togramsforanon‑plasticobject(Fig.1)andaplastic object(Fig.2).Weusetheequation:

3.1.TrashBoxDataset

WeusedtheTrashBoxdatasetforwasteclassi i‑ cationintheexperiment[17],whichcontains17,785 wasteobjectimagesscrapedfromthewebsite.Images don’tcontaindetectionannotationsprovidedinthe repository.Inthisstudy,weuse5,000randomimages fromallcategories.Imageparametersareasfollows:

Figure1. Histogramofanon‐plasticobject

Figure2. Histogramofaplasticobject

Figure3. Histogramofanimageofaplasticobject

• Size:512×384pixels

• Colordepth:24bits

• Resolution:96dpi

• Format:.jpg

Wastecategoriesareasfollows:

• Trashwaste:random;thenumberofimages = 2,010

• Plastic:bags,bottles,containers,cups;numberof images=2,669.

• Paper:TetraPaks,newspapers,papercups,paper tissues;thenumberofimages=2,695

• Metal:beveragecans,scrap,spraycans,food‑grade cans;numberofimages=2,586

• Glass:bottles;numberofimages=2,528

• Cardboard:numberofimages=2,414.

Hardwareusedinexperiment:Processor:Intel Corei7‑10700F‑8‑core;RAM:16GB;NVIDIA GeForceRTX2080Ti‑8GBGDDR6197;HDD:SSD 1TB.

4.ResultsandDiscussion

Table 1 presentstheresultsofthemainexperi‑ ment.Theobjectrecognitiontaskwastestedbasedon therangesofthehistogram(Fig. 2).Whenanalyzing theresults,wecanseethattheselectionoftheelement rangesfromthehistogramhasasigni icantimpact ontherecognitionresults.Asimplesymmetricalsplit inhalfproducesweakerresults,asdoselecting100 extremeelementsateachendofthehistogram.The bestresultswereobtainedforasymmetricsizesof ranges �� and �� andtheirasymmetricalposition.In addition,itisalsorecommendedtoselecttherange fromtheso‑calledoverlap,thatis,onethatpartially overlaps.

Table2showstheresultsofthesecondstageofthe experiment,inwhichwetestedthemethod’seffective‑ nessdependingonthetypeofmaterialfromwhichthe objectinthegarbagephotoismade.Weobtainedthe bestresultsformixedwasteandplastic.Wegotthe worstlevelofidenti icationformetal.Thereasonfor thismaybethepropertiesofthemetalintheformof lightre lections.Regardless,weachievedanaverage recognitionrateof94%.

Table1. Resultsofexperiment

RangeA RangeB Accuracy[%]

1‑100 155‑255 74

1‑100 101‑255 91

1‑150 155‑255 54

1‑100 101‑200 88

50‑150 151‑200 51

50‑100 101‑200 89

50‑150 151‑255 70

1‑120 151‑255 69

1‑120 121‑255 83

1‑180 121‑255 94

Table2. TheresultsofrecognitionusingtheTrashnet database

Table3. Comparisontoothermethods

Study Year

[6]

[7] 2017 TrashNet FasterR‑CNN

[8] 2018 TrashNet VGG‑19CNN

[9]

[10]

[4]

[5]

[13] 2019 LWW FasterR‑CNN 86(mAP) [14] 2018 Customdataset

[15] 2021 TrashNet MultilayerHybridCNN(MLHCNN)

[16] 2021 ISTWaste FasterRCNN

[18] 2021 Wadaba CNN

Ownwork

Table 3 showsacomparisonofourproposed methodwithotherknownmethods.Comparedtothe methodsthatusearti icialneuralnetworks,partic‑ ularlyconvolutionalnetworks(CNN),theproposed methodislesseffectiveduetolowercomputational complexity.Thisisanadvantagewhenwewanttouse amethodonamobiledeviceorinreal‑time;however, comparedtoothermethodsusingKNN,SVMorRan‑ domForest,asymmetrichistogramanalysisprovides betterresults.

Theanalysisoftheresultsweobtainedallowsus toconcludethattheideaofapplyingtheasymmetric histogramanalysisturnedouttobecorrect,andthat theobtainedresultsallowforitsimplementationin realconditions.

5.Conclusion

Thepaperpresentsamethodofrecognizing domesticwasteusingcomputervisiontechniques.We usedasimpleschemetoanalyzetheasymmetryofthe histogramofadigitalimageofagarbageobject.The conductedresearchcon irmsthattheuseofsimple imageanalysistechniquesallowsfortheconstruc‑ tionofeffectivemethodsforidentifyingorclassifying objects.Themethodprovedtobe94%effective,which isasatisfactoryresult,andallowstheprocesstobe usedinrealsystems,particularlyonmobilemicro‑ computers.Thisimplementationcallsforitswider applicationandfurtherdevelopmentintheareaof wastemanagement.

Despitethemanyyearsofstrugglewiththisprob‑ lem,itremainscurrent.Workoncomprehensivewaste managementsystemsisstillongoing.Newprojects sponsoredbyglobalconcernsarebeinglaunchedto reducethescaleoftheproblem,butthereisstilla

lotofworktobedone.Therefore,researchshould stillbeconductedtodevelopeffectivemethodsfor automatingtherecyclingprocesses.

AuthorContributions: Conceptualisation,J.B.; methodology,J.B.;software,J.B.;validation,K.P.; formalanalysis,K.P.;investigation,J.B.;resources, K.P.;datacuration,K.P.;writing—originaldraft preparation,K.P.;writing—reviewandediting, J.B.;visualisation,J.B.;supervision,J.B.;funding acquisition,K.P.Allauthorshavereadandagreedto thepublishedversionofthemanuscript.

Funding: ThisresearchwasfundedbytheMinis‑ terofScienceandHigherEducationunderthename ”RegionalInitiativeofExcellence”intheyears2019–2022,projectnumber020/RID/2018/19anamount of inancingof12,000,000PLN.

ConflictsofInterest: Theauthorsdeclarenocon lict ofinterest.Thefundershadnoroleinthedesignof thestudy;inthecollection,analyses,orinterpretation ofdata;inthewritingofthemanuscript,orinthe decisiontopublishtheresults.

AUTHORS

JanuszBobulski∗ –CzestochowaUniversity ofTechnology;Czestochowa,Poland,e‑mail: januszb@icis.pcz.pl. KamilaPasternak –CzestochowaUniversity ofTechnology;Czestochowa,Poland,e‑mail: kamila.bartlomijczyk@icis.pcz.pl.

∗Correspondingauthor

References

[1] C.Luietal.,”ExploringFeaturesinaBayesian FrameworkForMaterialRecognition,” Proc. 2010IEEEComputerSocietyConferenceon ComputerVisionandPatternRecognition,2010, pp.239–246.

[2] D.Rutqvistetal.,”AnAutomatedMachineLearn‑ ingApproachforSmartWasteManagementSys‑ tems,” IEEETransactionsonIndustrialInformat‑ ics,vol.16,2020,pp.384392.

[3] J.Zhengetal.,”ModelingGroupBehaviourTo StudyInnovationDiffusionBasedonCognition andNetwork:AnAnalysisfortheGarbageClas‑ si icationSysteminShanghai,China,” Interna‑ tionalJournalofEnvironmentalResearchand PublicHealth,vol.16,no.18,2019,3349.

[4] B.S.Costaetal.,”Arti icialintelligenceinauto‑ matedsortingintrashrecycling,” Proc.Anaisdo XVEncontroNacionaldeInteligênciaArti iciale Computacional,2018,pp.198–205.

[5] X.Xu,X.Qi,andX.Diao,”ReachonWasteClassi‑ icationandIdenti icationbyTransferLearning andLightweightNeuralNetwork,”Preprints,vol. 2,2020,327.

[6] M.YangandG.Thung,”Classi icationof TrashforRecyclabilityStatus,”CS229Project Report,StanfordUniversity:Stanford,CA,USA, 2016.

[7] 0.Awe,R.Mengistu,andV.Sreedhar,”Smart TrashNet:WasteLocalisationandClassi ica‑ tion.” arXivpreprint,2017.

[8] T.Kennedy,”OscarNet:UsingTransferLearn‑ ingtoClassifyDisposableWaste,”CS230Report: DeepLearning,StanfordUniversity:Stanford, CA,USA,2018.

[9] H.N.KulkarniandN.K.S.Raman,”WasteObject DetectionandClassi ication,”CS230Report: DeepLearning;StanfordUniversity:Stanford, CA,USA,2018.

[10] C.Bircanogluetal.,”RecycleNet:Intelligent WasteSortingUsingDeepNeuralNetworks,” Proc.2018InnovationsinIntelligentSystemsand Applications (INISTA),2018.

[11] 0.AdedejiandZ.Wang,”IntelligentWasteClas‑ si icationSystemUsingDeepLearningConvolu‑ tionalNeuralNetwork,” ProcediaManufacturing, 2019,vol.35,pp.607–612.

[12] D.O.Melinte,A‑MTravediu,andD.N.Dumitriu, ”DeepConvolutionalNeuralNetworksObject DetectorforReal‑TimeWasteIdenti ication,” AppliedSciences,vol.10,no.20,2020,7301.

[13] J.Sousa,A.Rebelo,andJ.SCardoso,”Automation ofWasteSortingwithDeepLearning,” Proc.2019 XVWorkshopdeVisãoComputacional (WVC), 2019,pp.43–48.

[14] Y.Chu,etal.,MultilayerHybridDeepLearning MethodforWasteClassi icationandRecycling. ComputationalIntelligenceandNeuroscence, 2018,5060857.

[15] C.Shietal.,”AWasteClassi icationMethodBased onaMultilayerHybridConvolutionNeuralNet‑ work,” AppliedSciences,2021,vol.11,no.18, 8572.

[16] C.Renetal.,”CoastalWasteDetectionBasedon DeepConvolutionalNeuralNetworks,” Sensors 21,2021,7269.

[17] N.KumsettyandA.Nekkare,TrashBoxdatabase, IEEEDataport,15Nov.2022,doi:10.21227/csg 6‑h017

[18] J.BobulskiandM.Kubanek,”DeepLearning forPlasticWasteClassi icationSystem,” Applied ComputationalIntelligenceandSoftComputing, 2021,6626948.

EXPLAINABILITYOFADEEPNEURALNETWORKMODELFORPREDICTIONOFSOLAR

Submitted:13th April2024;accepted:24th January2025

RosalísAmadorGarcía,MaríaMatildeGarcíaLorenzo,RafaelE.BelloPérez

DOI:10.14313/jamris‐2026‐008

Abstract:

TheexplainabilitymethodsLIME,SHAP,IntegratedGra‐dientsandTimeSeriesSaliencywerecomparedtoexplain thepredictionsofadeeplearningmodeltrainedtopre‐dicttheelectricitygenerationofaphotovoltaicpark. Thesemethodsallowanalyzingtherelativeimportance oftheinputfeaturesforeachprediction.Thequality oftheexplanationsgeneratedwasevaluatedusingthe fidelityandcontinuitymetrics,beforeandafterapplying perturbationstothedata.

Keywords: LIME,SHAP,IntegratedGradients,TimeSeries Saliency

1.Introduction

Drivenbythedevelopmentofdeepneuralnet‑ works,notableadvancesinArti icialIntelligence(AI) havebeenevidentinrecentyears,revolutionizing ieldssuchasimageprocessing,voicerecognitionand naturallanguageprocessing;makingthesesystems increasinglypresentindifferentdomainsandassuch indailylife[1,2].

Despitebeingincreasinglypresent,thesemodels actas“blackboxes”,whichmakesthemincomprehen‑ sibletohumans,makingitimpossibletounderstand theirinternalreasoningandthecausesthatledthe systemtoitsprediction.Giventheneedtointerpret andunderstandAImodels,the ieldofexplainable arti icialintelligence(XAI)isdeveloped[3].

XAIseekstocreatetechniquestomakeAImodels moreinterpretableandunderstandabletohumans, generatingexplanationsforhowtheyreachtheircon‑ clusionsandpredictions.Forexample,youcanshow themostin luentialinputfeaturesordesignmore transparenthybridmodels.Althoughthereisatrade‑ offbetweenperformanceandtransparency,abet‑ terunderstandingofhowthesystemworkscancor‑ rectitsshortcomings.XAIisoneofthekeyrequire‑ mentsforimplementingresponsibleAI,amethodol‑ ogyforlarge‑scaleimplementationofAImethodsin realorganizationswithfairness,modelexplainability, andaccountability[3].

Inrecentyears,severaltechniqueshaveemerged toexplainthepredictionsofthesesystems,highlight‑ ingLIME[4],SHAP[5],IntegratedGradients[6]and TimeSeriesSaliency(TSSaliency)[6].

Thispaperpresentsaneuralnetworkforpredict‑ ingelectricitygenerationfromaphotovoltaicpark

andalsomakesacomparisonbetweendifferentmeth‑ odsforexplainingregressionnetworks,withtheaim ofofferingaguideontheoptionsavailablewhenit comestoimplementingexplainablesolutionsinarti‑ icialintelligenceprojectsforthepredictionofneural networks[1].

2.Content

Explainabilityisthebasisforhumanstotrustand understandthedecisionsmadebymachinelearning models,asitoffersclarityandunderstandingabout theinternalworkingsofthesemodels,facilitating theirresponsibleuseandincreasingtheireffective‑ nessinvariousscenarios.

Therearetwoimportantaspectsregarding explainability.The irstaspect,understandingthe causeofthedecision,allowsustounderstandthe speci icfactorsthatledthemodeltomakeaspeci ic prediction.Thisusuallyinvolvesbeingabletoidentify themostrelevantcharacteristicsandtheirimpacton thedecision.Thesecondaspect,theabilitytopredict futureresults,impliesthatbyunderstandingthe causesofpreviousdecisions,itispossibletoreason andpredictthefutureresultsofthemodelmore accurately[1,7].

InthelasthalfdecadetheXAIhasbeguntogain somestrengthasaresearch ieldandconsequently moretoolsarebeingdevelopedforthispurpose, someofthemostimportantare:AIX360[8],inter‑ pretML[9],SHAP[10],AlibiExlain[11],Captum[12], iNNvestigate[13],explAIner[14],LIME[4],OmniXAI, amongmanymore.Eachonehasitsownapproachto differenttypesofdataandnetworkarchitectures.

2.1.ApproachestoExplainability

Therearetwomainapproachestoexplainability: modelsinterpretablebynatureandexplanationmeth‑ ods.Modelsinterpretablebynaturearebasedonthe designofmodelsthat,duetotheirownstructure, areeasyforhumanstounderstand.Someexamples are:decisiontreemodels,simplelinearregression, k‑nearestneighbors(k‑NN),andNaiveBayes.They haveasimplefunctionalformandlittlecomplexity thatallowsthepredictionlogictobeunderstoodintu‑ itively.

Explanationmethodsaretechniquesdeveloped speci icallytogenerateexplanationsaboutthefunc‑ tioningofblackboxmodelssuchasneuralnet‑ works,SupportVectorMachines,RandomForest,

etal.Examplesinclude:variableimportanceanalysis, roughruleestimation,heatmaps,andselectingrep‑ resentativeexamples.Eachmethodtakesadifferent approachtoopeningtheblackboxandexplainingthe model’sdecisionmaking.

Bothinterpretablemodelsbydesignandsub‑ sequentexplanationmethodsareusefulandcom‑ plementeachothertoimprovethetransparency, trustandadoptionofarti icialintelligencesystems. Ahybridapproachusingbothapproachesisidealfor manyreal‑worldapplications[15].

3.LocalandGlobalExplanationMethods

Explanationmethodsaredividedintotwocate‑ gories:localmethodsandglobalmethods.Thesediffer inwhattheyexplain(individualpredictionsorthe modelasawhole.)

Globalmethodsaregenerallyusedtodescribehow amodelworkswiththeinspectionofthemodel’scon‑ cepts.Thisreferstotheabilitytoask,“Whatcommon featuresweregenerallyassociatedwiththeimages assignedtothisparticularclass?”Butitisdif icult todirectly indaglobalexplanationforablackbox model.

Localmethodsexplainindividualdecisionsmade byamodel,providinginformationaboutwhythe modelselectedaspeci icoptionforaparticular case.Thesemethodsarebasedonthecontextofthe inputexampleandhelpunderstandtherelationship betweenthespeci iccharacteristicsofthatexample andtheprediction[16].

Thechoicebetweenthesemethodsdependsonthe natureofthequestionyouseektoanswer.However, bothglobalandlocalareessentialtoensurefairness andpreventbiasinAImodels,thusofferingabroader andmoredetailedunderstandingofhowthesemodels maketheirdecisions.

Post‑hocexplainabilitytakesatrainedmodelas inputandextractstheunderlyingrelationshipsthat themodellearnedbybuildingawhiteboxsurrogate model.Bydoingsotheyonlygenerateanapproxi‑ mationofhowtheblackboxmodelworks.Although thisroughexplanationisnotanexactmatch,it iscloseenoughtobeusefulinunderstandingthe logicoftheblackboxmodelwithoutaffectingthe prediction.

Post‑hocexplainabilitycanbeappliedintwoways. The irstismodel‑speci icexplainability,whichrefers toexplanationsexclusivetoaparticulartypeofmodel. Thesecondreferstomodel‑independent(agnostic) explanatorymethods.Theseapproximatethebehav‑ iorofthemodelstogenerateexplanationsforthe enduser,independentlyoftheinternallogicusedto generatepredictions,andarestandardized.Dueto theirpotentialtobeappliedtomoremodels,theyhave large‑scaleutility[17].

4.CharacterizationofExplainabilityMethods

Asmentionedabove,explanatorymethods developtechniquestoexplainthedecisionsofblack boxmodels.Thereisawidevarietyofmethodsbased

onpatterns,gradientandrelevance.Forthisresearch, post‑hocagnosticlocalexplanationmethodswere taken:LIME,SHAP,IntegratedGradientsandTime SeriesSaliency.

LIME(LocalInterpretableModel‑agnosticExpla‑ nations)allowsyoutounderstandtheimportance ofinputfeaturesforthepredictionofamachine learningmodel.Itisbasedondisturbingtheseinput datatoamodelandseeinghowtheyin luenceits predictions.

LIMEconsiderswhetherafeatureisimportantfor themodelprediction,inwhichcaseitsperturbation shouldcauseasigni icantchangeintheprediction, asconverselyanunimportantoneshouldcausean insigni icantchangeintheprediction.

Thismethodoperateseachfeatureoverasmall range,thencalculatesthedifferencebetweenthe model’spredictionsbeforeandaftertheperturbation, knownasthefeature’scontributiontotheprediction.

Finally,theimportanceofafeatureiscalculated, aswellasitscontributionstotheprediction.A featurewithahighcontributionisimportantfor prediction,whileafeaturewithalowcontributionis unimportant[4].

SHAP(SHapleyAdditiveexPlanation)isbasedon gametheoryandisusedtoassigneachmodelinput featureanimportancevalueforaparticularpredic‑ tion.

Gametheoryisabranchofmathematicsthatstud‑ iesthebehaviorofagentsthatinteractwitheach other.InthecaseoftheSHAPmethod,theinputdataof amodelareconsideredagentsthatinteractwitheach othertoproducetheprediction.

Itsoperationisbasedontheimportanceofthe characteristicfacebeingequaltothecontributionit hastotheprediction,calculatingthiscontribution usingtheShapleyvalue.

TheShapleyvalueofaninputfeatureistheaverage contributionthatthisfeaturemakestothemodel’s prediction,whenplayedagainstallotherfeatures. Thatis,theShapleyvaluerepresentstheimportance ofafeaturetothemodel’sprediction,onaverage, acrossallpossiblefeaturecombinations.

Moresimply,theShapleyvalueofafeatureisa measurebyhowmuchthemodel’spredictionchanges whenthatfeatureisincludedorexcluded[5].

IntegratedGradientisamethodusedtounder‑ standandexplainthepredictionsmadebymachine learningmodels.Itsgoalistoassignrelativeimpor‑ tancetotheinputfeaturesofamodelforaspeci ic prediction.

Thismethodisbasedontheconceptof“linearityin featurespace”,withtheideaofcomputingtheintegral alongapathbetweenareferencepointandthecurrent inputpointtoobtaintheimportanceofeachfeature, tothenintegratetheslopestoobtainrelativeimpor‑ tancesandthenscaleandassignthoseimportancesto eachcharacteristic.

The inalresultisafeaturerelevancemapthat indicatestherelativecontributionofeachfeatureto thepredictionmadebythemodel[6].

Asaliencymapisusedtohighlighttheregionsof animagethataremostrelevanttoadatatensorfora speci ictask.

ThegoaloftheTimeSeriesSaliencyistoidentify thepartsthathavethegreatestimpactonthepredic‑ tionmadebyamachinelearningmodel.Thesemaps areusefulforunderstandingwhichfeaturesaremost importanttothemodelindecisionmaking.

Theresultingsaliencymapprovidesavisualrep‑ resentationormeasureofimportanceforeachele‑ mentinthedatatensor,helpingtounderstandwhich elementsofthetensoraremostrelevanttoagiven modeloutputandprovidinganexplanationforthe predictionsmade.

Itisimportanttoknowthattheprocessofgener‑ atingasaliencymapvariesdependingonthetaskand themodelused,andthesaliencymapcanbeapplied todifferenttypesofdatatensors,suchasimages,text andaudiosignals[6].

5.EvaluationoftheQualityoftheExplanation

Itisimportanttoevaluatetheexplanationsgener‑ atedbytheexplainabilitymethodstoguaranteethat theseexplanationsaretrustworthy.Inthisresearch theyplayaveryimportantrole,giventhatthere aremethodsthatcon lictintheirexplanations.At thesametime,theyallowforbetteridenti icationof opportunitiestoimproveamodel.

FidelityMetrics

Itiscrucialtoevaluatehowaccuratelytheexpla‑ nationsre lecttheinternalworkingsofthemodel. Fidelityensuresthattheexplanationsprovidedare representativeofthetruebehaviorofthemodel, offeringabasisfortrustingtheinterpretationsgen‑ erated.Itisimportanttoverifytheauthenticityof theexplanationsanddifferencesfromotherformsof evaluationthatfocusonmorespeci icaspectsofthe characteristics.

ContinuityMetric

Itprovidesanadditionaldimensionbyevaluating theconsistencyofthegeneratedexplanations.Based ontheprinciplethatminorchangesintheinputdata shouldnotcauselargevariationsintheexplanations, thismetricisessentialtoguaranteethestabilityand reliabilityoftheexplanations.Itservesasakeyindi‑ catorofconsistencyintheinterpretationsgenerated, complementingthe idelitymetric,whichfocuseson howwellexplanationsre lecttheinternalworkingsof themode[11].

6.LearningModelandCaseBase

Forthisresearch,aconvolutionalneuralnetwork modelbasedontimeseriesisusedtopredictthe electricitygenerationofaphotovoltaicplant[17].

ThisnetworkwastrainedwithdatafromCuban plants.ForthisresearchthedatafromtheUCLVplant isused.Whereafterpreprocessing8featuresare usedforthenetworkinput:“Irradiancia”,“Tambiente”, “Tmodulo”,“Potencia”,“Daysin”,“Daycos”,“Yearsin”,

“Yearcos”;aswellas14daysinthepast;sothenet‑ workreceivesatensorwithdimensionsof(None,14, 8).“None”representsthenumberofinstancesthat youwanttopasstothemodel.

7.ExplanationsObtained

Torepresenttheexplanations,whenusingtwo‑ dimensionaltensors(14,8),itwasdecidedtouse tablestorepresenttheimportanceofeachinputchar‑ acteristic.

Greenrepresentsapositiveimportance(Positive Contribution)totheprediction.

Redrepresentsanegativeimportance(Negative Contribution)fortheprediction.

WithLIMEitisshownhoweachcharacteristic contributestotheprediction,whereitcanbeseen thatthecharacteristicthatmostcontributestothe predictionisthe“Iradiancia”inallthedaysthatare lookedback,andinthesamewaywecanseethatthe variable“Tmodulo”hasnopositivecontributionsto theprediction.TheFigure2showstheresultsbyusing LIMEexplanation.

SHAP(Figure 3)yieldsresultsverysimilarto thoseobtainedwithLIME.Inbothcases,“Irradian‑ cia”acrosstheentirelook‑backwindowisthefeature

Figure1. Modelarchitecture

Figure2. ExplanationobtainedwithLIME

Figure4. ExplanationobtainedwithIntegrated Gradients

thatcontributesmosttotheprediction,whereas “Tmodulo”showsnopositivecontribution.

IntegratedGradientoffersadifferentexplanation fromthatobtainedbytheLIMEandSHAPmethods (Figure4),butonthecontrary,similartothatobtained withtheTimeSeriesSaliency,where“Irradiancia”is avariablethatgreatlyin luencestheprediction,but dependingonthedayinthepast,canhaveanegative contributiontotheprediction.Itshouldbenotedthat forthismethodthevariable“Tmodulo”hasthegreat‑ estimportanceforprediction.

Thismethodobtainedanexplanationthatissimi‑ lartoeachofthepreviousmethods,sinceonceagain wecanseethatthevariable“Irradiancia”hasgreater importancethantherestofthevariables,depend‑ ingontheday.Inthepast,itpresentsnegativecon‑ tributions,closelyresemblingthepreviousmethods explained.Itshouldbenotedthatinthiscase,like LIMEandSHAP,therestofthevariablesdonot havesigni icanceexcept“Tmodulo”which,similarto IntegratedGradient,hasapositiveimportanceinthe explanation.ThisisshowninFigure5.

8.ComparisonofExplanations

Tocomparetheexplanationsobtainedwiththese methods,theformsofevaluationofexplanationsmen‑ tionedinsection5areused.

Fidelity

Pre_perturbated Post_perturbated

Figure6. Fidelityvaluesofmethods

Thecomparisonofthe idelityoftheLIME,SHAP, IntegratedGradientsandTimeSeriesSaliencymeth‑ odsbeforeandafterperturbations,highlightsthesta‑ bilityofLIMEagainstchangesinthedatamaintaining aperfect idelityof1.0,indicatingthatthismethod offersconsistentinterpretationsandisrobusttodis‑ turbances.

Ontheotherhand,SHAPexperiencesasubstantial dropin idelityfrom1.0to0.55uponperturbations, highlightingitssensitivitytochangesinthedata.Such SHAPvariabilitycouldbeusefulforexploringhow differentconditionsaffectthemodel,althoughthis mayimplyreducedreliabilityinscenarioswherecon‑ sistencyisrequired.

Finally,theIntegratedGradientandTimeSeries Saliencymethodspresentamediumstabilityinthe faceofchangesinthedata,experiencingadropof1.2 and1.1respectively.

Theresultssuggestthat,dependingontheneed forconsistencyininterpretations,LIMEcouldbe preferredovertherestofthemethodsincontexts whereinterpretivestabilityiscrucial,althoughInte‑ gratedGradientandTimeSeriesSaliencycanalsobe considered.

Theseminimumcontinuityvaluesoffercrucial insightintothereliabilityofexplanationmethods whenfacedwithvariationsinthedata.

Figure3. ExplanationobtainedwithSHAP

Figure5. ExplanationobtainedwithSaliencyMaps

Figure7. MethodContinuityValues

9.Conclusion

LIMEprovedtobethemostrobustandconsis‑ tentmethod,maintainingperfect idelityintheface ofchangesinthedata.Itissuitablewhenstabilityis crucial.

SHAPhadthegreatestvariabilityinexplanations, althoughitcouldbeusefultoexplorehowdifferent conditionsaffectthemodel.

IntegratedGradientsandTimeSeriesSaliency showedintermediatestability.

Intermsofcontinuity,SHAPwasthemethodwith theleastvariabilityinexplanations,followedbyLIME. Quantitativeevaluationisimportanttoobjectively comparethequalityofthegeneratedexplanationsand selectthemostappropriatemethodaccordingtothe requirements.

AUTHORS

RosalísAmadorGarcía∗ –CII,UCLV,SantaClara, Cuba,e‑mail:rosalisamadorgarcia@gmail.com. MaríaMatildeGarcíaLorenzo –CII,UCLV,Santa Clara,Cuba,e‑mail:mmgarcia@uclv.edu.cu. RafaelE.BelloPérez –CII,UCLV,SantaClara,Cuba, e‑mail:rbellop@uclv.edu.cu.

∗Correspondingauthor

References

[1] D.V.Carvalho,E.M.Pereira,andJ.S.Cardoso, “MachineLearningInterpretability:ASurveyon MethodsandMetrics”,Electronics.2019.DOI: 10.3390/ELECTRONICS8080832.

[2] C.Molnar,2.2“WhatIsMachineLearning?| InterpretableMachineLearning”,Accessed:Dec. 17,2023.[Online].Available: https://christop hm.github.io/interpretable‑ml‑book/what‑is‑ machine‑learning.html

[3] A.B.Arrietaetal.,“ExplainableArti icialIntel‑ ligence(XAI):Concepts,Taxonomies,Opportu‑ nitiesandChallengestowardResponsibleAI”, arXiv.Dec.26,2019.Accessed:Jan.21,2024. [Online].Available:http://arxiv.org/abs/1910 .10045

[4] M.T.Ribeiro,S.Singh,andC.Guestrin,“‘Why ShouldITrustYou?’:ExplainingthePredictions ofAnyClassi ier”,arXiv.Aug.09,2016.Accessed: Dec.18,2023.[Online].Available:http://arxiv. org/abs/1602.04938

[5] S.M.LundbergandS.‑I.Lee,“AUni iedApproach toInterpretingModelPredictions”,inAdvances inNeuralInformationProcessingSystems.Cur‑ ranAssociates,Inc.,2017.Accessed:Dec.18, 2023.[Online].Available:https://papers.nips. cc/paper_files/paper/2017/hash/8a20a8621 978632d76c43dfd28b67767‑Abstract.html

[6] M.Sundararajan,A.Taly,andQ.Yan,“Axiomatic AttributionforDeepNetworks”,arXiv.Jun.12, 2017.Accessed:Jan.29,2024.[Online].Avail‑ able:http://arxiv.org/abs/1703.01365

[7] T.Miller,“ExplanationinArti icialIntelligence: InsightsfromtheSocialSciences”,arXiv.Aug.14, 2018.Accessed:Dec.17,2023.[Online].Avail‑ able:http://arxiv.org/abs/1706.07269

[8] “Trusted‑AI/AIX360”,Trusted‑AI,Jan.21,2024. Accessed:Jan.21,2024.[Online].Available:http s://github.com/Trusted‑AI/AIX360

[9] “interpretml/interpret”,InterpretML.Jan.21, 2024.Accessed:Jan.21,2024.[Online].Avail‑ able:https://github.com/interpretml/interpret

[10] “shap/shap”,shap.Jan.21,2024.Accessed:Jan. 21,2024.[Online].Available: https://github.c om/shap/shap

[11] J.Klaise,A.VanLooveren,G.Vacanti,andA. Coca,“AlibiExplain:AlgorithmsforExplaining MachineLearningModels”,JournalofMachine LearningResearch.vol.22,no.181.pp.1–7,Jun. 2021.Accessed:Jan.21,2024.[Online].Avail‑ able:http://jmlr.org/papers/v22/21‑0017.html

[12] “pytorch/captum”,pytorch.Jan.21,2024. Accessed:Jan.21,2024.[Online].Available: https://github.com/pytorch/captum

[13] M.Alber,“albermax/innvestigate”,Jan.18,2024. Accessed:Jan.21,2024.[Online].Available:http s://github.com/albermax/innvestigate

[14] “dbvis‑ukon/explainer”,DataAnalysis&Visual‑ izationGroup.Nov.21,2023.Accessed:Jan.21, 2024.[Online].Available:https://github.com/d bvis‑ukon/explainer

[15] W.J.Murdoch,C.Singh,K.Kumbier,R.Abbasi‑ Asl,andB.Yu,“Interpretablemachinelearning: de initions,methods,andapplications”, Proc.Natl.Acad.Sci.U.S.A.,vol.116,no. 44,pp.22071–22080,Oct.2019,DOI: 10.1073/pnas.1900654116.

[16] C.Gomes,L.Natraj,S.Liu,andA.Datta,“ASur‑ veyofExplainableAIandProposalforaDisci‑ plineofExplanationEngineering”,arXiv.May20,

2023.Accessed:Dec.17,2023.[Online].Avail‑ able:http://arxiv.org/abs/2306.01750

[17] R.HerreraCasanova,R.H.Casanova,B.B.Gar‑ cia,J.G.Tamayo,L.L.Viltres,andF.H.Fernán‑

dez,“Propuestademodelopredictivointeligente paraunaplantafotovoltaica”,Rev.Cuba.Cienc. Inform.,vol.16,no.1,Mar.2022,[Online].Avail‑ able:https://rcci.uci.cu/?journal=rcci&page=ar ticle&op=view&path[]=2419

Submitted:6th April2025;accepted:13th May2025

VanAnhPham,MinhTienDo

DOI:10.14313/jamris‐2026‐009

Abstract:

Light‐emittingdiodes(LEDs)playacrucialroleinvari‐ouspracticalapplications.Thispaperpresentsasimple andcost‐effectiveapproachfordetectingfaultsinLED boardswithalargenumber.Thetypesoffaultsdetected includeopencircuitsandshortcircuits.Theproposed solutionisbasedonavoltagethresholdandastatistical algorithmtofilteroutnoise.Thepaperfirstoutlines thedesignconcept,followedbythedevelopmentand fabricationofafaultclassificationmodule.Subsequently, acontrolcenterisestablishedtocollectsignalsfromall faultclassificationmodulesconnectedtoLEDboardsin pairs.Thereceivedsignalsareprocessedandstoredona memorycard.Finally,theresultingdatacanbeaccessed eitherfromthememorycardordirectlyviaanexternal computerthroughaUSBport.Theexperimentalresults demonstratethattheproposeddesignandfabricated devicecanaccuratelydetectandclassifyfaultsonthe testedboards.Thedataobtainedcanbeusedforsta‐tisticalanalysisandrootcauseinvestigationtoenhance productquality.

Keywords: LEDboards,LFDmodule,controlcenter, ArduinoMega2560,faultdetectionmodel

1.Introduction

Light‑emittingdiode(LED)circuitboardsare increasinglyutilizedinlighting,display,andindustrial applicationsduetotheirhighef iciency,longlifes‑ pan,andlowenergyconsumption[1–4].However, duringproductionandoperation,LEDcircuitboards mayencountervariousfaults,suchaspowersup‑ plyfailures,circuitbreaks,semiconductorcomponent damage,ordegradationinlightquality.Accuratefault detectionanddiagnosisplayacrucialroleinmaintain‑ ingoperationalperformance,reducingrepaircosts, andextendingsystemlongevity.

Somepreviousstudieshavefocusedonfaultdetec‑ tionbasedonimageprocessingtechniquescombined withanautomatedopticalinspectionsystem[5], achievinganaccuracyofupto95%andaprocess‑ ingspeedof0.3secondsperimage.Thestudy[6] concentratedontheelectromechanicalcharacteris‑ ticsofLEDsthatmayleadtodefectswhenoperat‑ ingin lammableenvironments.In[7],theauthors employedabackpropagationneuralnetworkalong withthesupportvectormachine(SVM)algorithm forfaultdiagnosis.Alow‑costmethodutilizingfault

detectiononaseriesofthreeLEDswasalsoconsid‑ ered[8],however,thisstudyonlyaddressedauto‑ matedinspectionforasmallnumberofLEDs.Another investigation[9]employedanon‑destructivetesting methodbasedonaconfocallaserscanningmicro‑ scopesystemtoidentifyfaultsundervaryinginspec‑ tiondurations.Amachinelearningapproachwaspre‑ sentedin[10]topredictfaultsinLEDs.In[11],the authorsfocusedonthefrequency‑timecharacteristics ofLEDlightemissiontodetectfaultsthatconventional measurementmethodsfailtoidentify,achievingan accuracyofupto92%anddemonstratingapplicability forlarge‑scaleLEDmaintenance.Additionally,astudy ondetectingshort‑circuitfaultsinLEDsbasedondif‑ ferencesinvoltagethresholdsbycombiningamathe‑ maticalmodelwithneuralnetworkswasmentioned in[12].Amongthesemethods,mostwereconcen‑ tratedonfaultdetectionforarelativelysmallnumber ofLEDs.

Unliketheaforementionedstudies,thispaper investigatesafaultdiagnosismethodforLEDcircuit boardsbyanalyzingabnormalfeaturesintheLED shapesontheboardandtheconnectingcircuitry. Basedontheseanalyses,anautomatedfaultanaly‑ sisanddetectionmodelisproposed,employingsig‑ nalprocessingtechniquestoenhancetheaccuracy ofidentifyingtherootcausesoffaults.Ourapproach isstraightforward,cost‑effective,andapplicabletoa largenumberofLEDboards.

Thisstudyutilizeselectronicmeasurementtech‑ niquesandX‑rayimagingtoconstructafaultdetection model.CommonfaultsonLEDcircuitboardssuchas opencircuits,shortcircuits,andpowerdegradation willbeanalyzedtodeterminefaultpatternsandpro‑ posedesignsolutionsforfaultdetectionsystems.

Moreover,thisstudyidenti iestheprimarycauses offailuresinLEDboards,includingindividualLEDs, basedonananalysisofdefectreportsreceivedfrom userpartners.Theresultsrevealedthatdamage occurredtotheLEDcomponentsaftertheyweresol‑ deredontoaninsulatedmetalsubstrate(IMS),which servedasaheatsink.Figure. 1 illustratesadam‑ agedLEDmodule(coded2583‑164).X‑rayimages revealednodefectsintheelectricalcircuitonthealu‑ minumheatsinksubstrate(seeFigure. 2).However, thedefectswereobservedintheLEDchips.Perpendic‑ ularandtiltedX‑rayimagesoftheLEDchip(Figures.3 and 4)identi iedanomaliesinthedefectivechips. Incontrast,side‑viewimagesofthechipsdidnot

Amoduleof3‐LED(SMD2583‐164)

X‐rayimageoftheelectricalcircuitoftheLED board

Figure3. X‐rayimageshowingadefectintheLEDchip

Figure4. LateralX‐rayimageshowingadefectonthe LEDchip

exhibitanydistinguishabledifferencesbetweenthe functionalanddefectiveLEDs,asshowninFigure.5

InadditiontotheX‑raymethod,manualmeasure‑ mentswereperformedonbothfunctionalandfaulty LEDsusingadigitalmultimeter(Fluke87V).Anexam‑ pleofthemanualmeasurementresultsispresented asfollows:Thelightemittedisbrightforfunctional LEDs,withtheforwardresistancemeasuring37.9MΩ Incontrast,adamagedLEDthatdoesnotemitlight exhibitsaresistancevalueofonly38.2Ω.Theresults areillustratedinFigure.6.

DefectiveLEDFunctionalLED

Figure5. X‐rayimageofthesideviewofthedefective andfunctionalLEDs

Figure6. Demonstrationofmanualresistance measurementsforfunctionalanddefectiveLEDs

Basedontheseobservations,thefollowinggeneral designrequirementsareproposed:atestingduration of120minutes,withacycleconsistingof5minutes ONand1minuteOFF.Aconstantelectricalcurrentof 300to350mAthatisappliedtoeach3‑LEDboard, ensuringthecurrentdoesnotexceed350mA.The systemiscapableofautomaticallydetectingshortand opencircuitsinindividualLEDs.

Thispaperproposesanapproachfordesigning testingequipmentthatsimultaneouslyidenti ieserror typesonsurfacemounteddevice(SMD)LEDboards. Thesystemsupportstestingofupto2003‑LEDboards or6001‑LEDboards.Thedeviceoperatesbasedon avoltagethresholdtodetectopencircuits,shortcir‑ cuits,ornormaloperationalstatusesofLEDboards. Thecontrolcenter,faultcollection,anddataexportare managedbyanArduinoMega2560.Thiscentralunit isinterfacedwith74HC165modulestofullyexpand thedatacaptureports.Thedesign,manufacturing,and testingprocesseshavedemonstratedthatthedevice meetstheproposeddesignrequirementsandensures testingaccuracy.

Theremainderofthispaperisorganizedas follows:SectionIIpresentstheconceptualdesign, SectionIIIdetailsthedesignofthereportingsystem, SectionIVdiscussestheexperimentalfabrication process,andSectionVpresentstheresultsandtheir subsequentdiscussion.Finally,theconclusionsare drawninthelastsection

Figure1.

Figure2.

2.ConceptualDesign

Thissectionintroducesthedevice’scoreconcept andtheessentialcomponentsforaccuratefaultdiag‑ nostics.Aproposeddesignincludesblocksshownin Figure.7toimplementtherequirementscheckfor200 boardswith3‑LED.ThecontrolcenterisanArduino MegaKit.Thisboardwasselectedduetothemany digitalI/OpinsessentialforhugeLEDboards(54I/O pins).Thesecondimportantmoduleistheexpansion board.Onethousandtwohundredinputpinsarenec‑ essarytodeterminethestatesof600LEDunits.It shouldbenotedthataLEDhasthreestates:short, open,andregularcircuit.Itneedstwopinstoknow theLEDstate.ForthemaximumcheckedLEDoneach boardcase,thecalculationisconductedwith3‑LED boards.Therefore,150unitsof74HC165chipcan appreciatetotalLEDsoncheckedboards.

Thefollowingimportantpartistheerrorclassi‑ icationmodule.Thismoduleoutputsthreepossible statusesforeachLED:short,open,andregularcircuit. Acomparisonstructureofvoltagelevelcanbeused toidentifytheerrorofasingleLED.However,itis necessarytodeterminetheresistancevalueoffalse LEDs.Anotherequallyimportantpartisthepower supplymodule.Insteadofusingrelaystoturnonoroff theLEDpowersupplysource,themodulesthatapply theMOSFETtransistorareusedtorestrictnoisethat affectfeedbacksignals.

Furthermore,amicroSDcardisappliedtowrite theresultdata.Thismoduleisdirectlyconnectedto thecentercontrolmodule.Similarly,anaccuratetimer isaddedtowritethe inaltimeofthetestingpro‑ cess.Theinputcommandsfromthecontrolhumanare typedtosetuptheparameterofthetestingprocess throughajoystickattachedtoaswitch.Thisjoystickis connectedtoArduinoMegathroughtwoanalogpins andadigitalinputpin.

3.SystemDesign

3.1.MechanicalDesign

Next,amechanicalstructuresupportingtheelec‑ tricsystemisdesignedandpresentedinFigure. 8 Thisusespro ilealuminumbars40×40,combined withpowder‑coatedsteelsheetsasanexternalcover. Totaldimensionsare1100,1000,and350mm,cor‑ respondingtoheight,length,andwidth.Thegrooved insulatingpanelscontainingtheLEDboardsactas asupport loor,separatingthe loors.Furthermore, anindicationlampisaddedtoidentifythedevice’s operationstatus.

A0-JOYSTICK

A1-JOYSTICK

10-LED-MODULE-19

10-LED-MODULE-18

10-LED-MODULE-17

10-LED-MODULE-15

10-LED-MODULE-11

SDA

P0-CS-0

P1_SCK_0

P3_SCK_0

green_lamp

10-LED-MODULE-1

10-LED-MODULE-3

10-LED-MODULE-7

10-LED-MODULE-9

P0-SCK-0

P2_CS_0 SDA

P1_CS_0

P2_MISO_0

P3_MISO_0 SCL

yellow_lamp

10-LED-MODULE-0

10-LED-MODULE-5 MISO-SDCARD

MOSI-SDCARD

CS-SDCARD

SCK-SDCARD P0-MISO-0

10-LED-MODULE-2

SCK-SDCARD

3.2.ElectricalDesign

Thissectionpresentsthenecessarystepsofthe electroniccircuitdesignprocess.First,theapproach forthecontrolschematicsisintroduced,asshown inFigure. 9.Thecontrolcenterisopticallyisolated fromthepowersourceoftheLEDmodulesandthe relayscontrollingthealarmlampsinordertoreduce theimpactofsignalnoise.Themodules,includingthe 3‑signalpinjoystick,the20‑column × 04‑rowLCD, thereal‑timeclock(RTC3231),andthemicroSDcard, aredirectlyconnectedtotheArduinoMegaKIT(see Figure. 9).Two5Vpowersupplymoduletypesare employed:onetopowerthemicrocontroller,input expansionmodules,LCD,RTC,andSDcard,andthe othertosupplypowertothehigh‑powerLEDmodules andfaultclassi icationmodules.Thisdesignensures thestableoperationofthesystemduringextended testingperiods.

Additionally,afour‑channelrelaymoduleisincor‑ poratedtooperatealarmlampsthatindicatethe device’soperationalstatus.Thisrelaymoduleis

Figure7. Blockschematicofcontrolsystem

Figure8. Conceptmodelofmechanicalstructure

Figure9. Schematicdiagramofcontrolcenter

Figure10. Schematicsoftheopticallyisolatedrelay module

P0-EDM-5

Figure12. FaultdetectioncircuitofanLED

Figure13. (a)Schematicsofa3‐LEDmodule(SMD 2583‐164),(b)theDCequivalentcircuitofanLED

P0-EDM-5 GND-MCU

P0-EDM-0 P0-EDM-17 P0-EDM-14

P0-EDM-17

GND-MCU P0-EDM-23

P0-EDM-22

Figure11. Connectiondiagramofperipheraldevices (74HC165andLFDmodule)

opticallyisolatedfromtheArduinoMega,asdemon‑ stratedinFigure.10

Thenextsectionofthedesigndiagramillustrates theconnectionbetweentheLEDfaultdetection(LFD) modulesandtheinputexpansionmodulesbasedon the74HC165shiftregister.EachLFDmoduleiscapa‑ bleofidentifyingthefailuretypesofathree‑LED board(open,short,andregularcircuits).Apinexpan‑ sionboardconsistsofthree74HC165chips,providing atotalof24digitalinputs,andisconnectedtofourLFD modules(seeFigure. 11).Therefore,fortesting200 LEDboards,atotalof600LFDmodulesand50pin expansionmodulesarerequired,whichcorresponds to1200digitaldatapinstransferredtotheArduino MegaKIT.

Inthisdesign,duetotherelianceonthemechani‑ calframestructure,theportexpansionsystemisdis‑ tributedacrossfourmainblocks,eachutilizingthree pins:CLK,CS,andMISO,whichareconnectedtothe ArduinoMega2560.The irstthreeblockscontain13 modulesof74HC165,whilethefourthblockcontains 11modules

AnLFDmoduleconsistsofthreetestingclusters, witheachclustermonitoringasingleLEDchip.The

schematicdiagramoftheclusterisshowninFig‑ ure.12.Itisimportanttonotethatanop‑ampLM324 isemployedtodeterminewhethertheLEDisexperi‑ encingashortcircuitorisinnormaloperation.The classi icationthresholdissetviathevoltagedivider networkcomprisingresistors R8 and R9.Transistors Q4,resistors R6 and R7,areusedtodetectopen‑circuit LEDfaults.Theresultsarecommunicatedthroughtwo pins, O1 and S1,whichindicatethefollowingstatuses: opencircuit(if O1 ishighandS1islow),shortcircuit (if S1 ishighand O1 islow),ornormaloperation(if both O1 and S1 arelow).TheISO1andISO2opto‑ couplersprovideelectricalisolationfortheteststatus signalbeforetransmittingthe S1 and O1 signalstothe ArduinoMegaKIT.Thisisolationensuresthestability ofthecentralcontroller’soperationduringprolonged testingperiods.

TheschematicdiagramoftheLEDmoduletobe testedisshowninFigure.13a.TheLEDsareconnected toacommonanode,withthepowerpins(L1, L2,and L3)directlyconnectedtotheLFDmodule.Theequiva‑ lentschematicdiagramforaregular,forward‑biased LEDispresentedinFigure. 13b,wheretheforward voltage V�� = R��.I�� +V��,with V�� representingthebar‑ riervoltageoftheLEDjunction,and Rb denotingthe internalresistanceoftheLED.

3.3.ControlFlowChart

Theprimarycontrolalgorithm,asillustratedin Figure. 14,providesaclearerunderstandingofthe device’soperation.Therearethreemainoperating modes:SetupMode(i),CalibrationMode(ii),andTest‑ ingMode(iii).InMode(i),userscancon igureinput parameterssuchasthenumberofboardstobetested, theLED”on”time(t��),theLED”off”time(t��), andthetotaltestingtime(t��).Itshouldbenotedthat thenumberofLEDtestingcycles(n��)iscalculatedas follows: n�� =ROUNDDOWN(t�� /(t�� + t��)),where ROUNDDOWN(.)denotesthe loorfunction.InMode

Start

Initiate LCD screen, SD card, IO expansion

Show testing mode on LCD screen

Setup mode = 1

Calibration mode = 1

Testing mode= 1

Test LED on setup cycles

Display the parameters of the test process

Check if the number of cycles is enough?

Filter noise using averaging algorithm

Calculate the faultprobability of each LED

Setup number of boards, time of turn on/off, total time of testing

Calibrate IO pins

Figure15. Illustrationofthemeasurementofcurrent andvoltageparameterson1‐Ledmodel:SMD2583‐426 REVB(a),3LEDboardmodel:SMD2583‐164PCBREVB (b),andfaultclassificationmodule(c)

4.Manufacturingprocess

Figure14. Schematicsofcontrolflowchart

(ii),userscanperformdiagnosticsontheextended I/OportswithoutconnectingtotheLEDmodules. Thismodeenablestheveri icationofthestatusof theextendedportsandtheidenti icationofpotential faultsintheassociatedinternalmodules.Mode(iii)is thetestingoperationtodeterminewhetherLEDsare faultyoroperationalwhenconnectedtothedevice. Inthismode,theLEDmodulesareswitchedonand offcyclically.Duringthe”on”state,thefaultdetection signalissampledcontinuouslyatafrequencyof1Hz throughoutthe”on”period(ton),whichminimizes theimpactofsignalnoiseduringtesting.Aparame‑ ter,referredtoastheactualfaultreturnfrequency (��),isintroducedandcalculatedasfollows: �� = ��/��,where �� isthenumberofdetectedfaults, �� denotesthetotalnumberofsamplingacquisitions duringthetestingprocess,and��=1,2,...,1200.It shouldbenotedthat�� iscomputedasfollows:�� = ∑�� =1 �� [��],where, �� representsthelogicvalueof theoutput��atposition��,if��isanevennumber,then ��=�� else ��=��; �� and �� aresymbolsrelatedto theoutputstatepinoftheLFDboard.Theparameter �� iscrucialforevaluatingthereliabilityofthetesting process,especiallyinindustrialenvironments.Afault iscon irmedtohaveoccurredwhen �� ≥��, ��is thethresholdvalue.Beyondaveraging‑basednoise iltering,sequentialswitchingof20LR7843power modulesisemployedtominimizesysteminterference.

Insummary,integratingnoise iltering algorithms,rationalpowerswitchingcontrol,and microcontroller–powerstageisolationenhances systemstabilityandreliability.

Thefabricationprocessofthedeviceconsistsof twoprimarystages:theexperimentalphase(i)and themanufacturingphase(ii).Inthe irststage(i), thefaultclassi icationmoduleforasingleLEDunit istested.Thisprocessinvolvesconductingmanual measurementstoidentifyfaulttypesandtestinga prototypeofthefaultclassi icationmodule,whichis connectedtoa3‑LEDboardusingamultimeterand adigitalpowersupply(Topward3303D).Thisproce‑ dureisillustratedinFigure.15.Itisimportanttonote thattheSMD2583‑426LEDmodulecontainsasin‑ gleLED,whereastheSMD2583‑164modulecontains threeLEDs.Thesemodulesarenotequippedwith current‑limitingresistors(refertoFigure. 13).The secondstage(ii)entailsthemassproductionofLFD modulesandtheirintegrationwithextendedI/Omod‑ ules,MosfetLR7843‑basedPWMcontrollermodules (HW‑532),andthecentralcontroller(ArduinoMega 2560).TheLEDboardstobetestedaresystematically arrangedacrosseightlevelsofastoragecabinet,with eachlevelaccommodating25LEDboards.EachHW‑ 532moduleisemployedtopower10to12LFDmod‑ ules.Thismodulewasselectedduetoitsresistance characteristics,with R��(��)�� =3.3mΩ,leadingtoa voltagedropacrossthepowermoduleislow, V��(��) =R��(��)�� × I�� ≈ 0.043V.Four5VDC,20Apower suppliesareevenlydistributedtosupplypowertothe eightlevelsofLEDboards(2003LED‑boards)andthe 200LFDboards.Thepowersupply,with�� =5��,was selectedduetoitswidespreadindustrialadoptionand compatibilitywiththeforwardvoltageoftheLEDchip. Thecurrentperthree‑LEDboard,��3��max,islimitedto 350mA.Consequently,themaximumaveragecurrent perLEDisgivenby��max =��3��max/3.Thetotalpeak powerofthedevice,��max =600��max�� =350��, remainswithinsafeoperationallimits.Althoughthe chosen400Wtotalpowerisnotfullyoptimized,it ensuresareliablesafetymarginforthesystem.

(a) (b) (c)

Todemonstratetheresultsofthemanufacturing process,Figures.16and 17illustratetheassemblyof modulesintothemechanicalcabinet,alongwiththe testingofeachpartbeforethe inaldeviceinspection. Additionally,thisprocessincludesthecalibrationof I/Oexpansionmodulesto ixanyerrors.

5.ResultsandDiscussion

Tofurtherevaluatethedevicebeforecompletion, wetestedsomesamplessentbycustomers.Thenum‑ berofsampleboardsusedfortestingthedeviceis5 (for3‑ledboards),namedNCR1toNCR5andillus‑ tratedinFigure. 18.First,thesesampleLEDboards manuallytestedeachLEDbyusingthemultimeter forresistormeasurementandthedigitalpowersup‑ plytotrackconsumedcurrent,todeterminethefault types.Then,theyareconnectedtoonlyoneindividual LFDmoduleandthedigitalpowersourcetocon irm. Thisprocessistoclaimthatthefaultykindofsample boardsareexact.ResultsaresummarizedinTable1. ItisnotedthatthereturnedoutputvaluesoftheLFD moduleareshowninthistable,whereOiandSiindi‑ cate“openstate”and“shortstate”correspondingly (��= 1, 2, 3).Next, ivesampleboardsareattachedto themanufacturedtestingdevice,andthepositionof theconnectorjacksismarked.Thedeviceisactivated, andthetestingresultsarecomparedtothesample set.Figure. 19 illustratesthediagramofthetesting process.Thetestresultsareperformed40timeswith ashortenedtimeof2minutesONtimeand1minute OFFtime.Thistestisbecausethenumberofboards withbrokenLEDsislimited.Thetesttimeisshort‑ enedcomparedtoreality,butthenumberoftestsper‑ formedhasincreased(upto40times).Thepositions notconnectedtotheLEDboardsareconsideredopen circuitfaultboardoneduringthetest.

Testingboardsamples(2583‐164‐PCB‐REVB, 2583‐426‐REVB)

Theresultsoftheindividualtestsarepresentedin Figure. 20.Thekeyparametersincludethetestdate andtime,thecircuitboardposition,theLEDlocation, theteststatus,andtheprobabilityofmeetingthe requiredcriteria.Itisimportanttonotethattheprob‑ abilityofmeetingthecriteriaiscalculatedastheratio ofsuccessfultrialstothetotalnumberoftestscon‑ ducted.Thisvaluecorrespondsto �� asmentioned inSection3.3.Theteststatusisdeterminedduring theperiodwhentheLEDisilluminated.Duetomea‑ surementnoise,thisparametermaynotreachavalue of1,butitre lectsthereliabilityofthetestresults. Experimentalresultsdemonstratethatthedevice’s trialrunsaccuratelyidenti iederrorsonthesample LEDboards,includingLEDlocation,boardposition, andfaulttype.Atotalof40testswereperformed,all ofwhichmettherequiredcriteria.

Tofurtherassesstheaccuracyandreliabilityofthe device,astatisticalanalysiswasconductedwith40 differenttrialsusing ive3‑LEDboardsamplesperrun atvaryingpositions,i.e.,theevaluationused15LEDs, including3open‑circuit,2short‑circuit,and10func‑ tioningLEDs(seeTable1)foreachtrial.Thisarrange‑ mentensuresthatall600positionsweretestedonce. Thedeviceaccuratelyidenti iedthecorrectstatusof eachLEDinalltrials.Therefore,theobservedaccuracy is100%.UsingtheWilsonscoreintervalmethod,the 95%con idenceintervalforthetrueaccuracyofthe devicewasestimatedtobenolessthan99.37%.These resultsindicatethatthedeviceishighlyreliable.

InadditiontotheinspectionstatusoftheLEDs derivedfromthedatastoredontheSDcard,thedevice

Figure16. LEDfaulttesterduringassemblyandtesting

Figure17. Displayscreenofparameters

Figure18.

Table1. SampleoffaultyLEDboards(2583‐164‐PCB‐REVB).

Start the process

Mount the 5 LED boards at the first position of the machine

Set the number of tests to 40

Run the machine to test LED boards

Record positions of the LED boards and the test results

The number of test = 0

Decrease the number of tests

Move the 5 LED boards to a new position on the machine Yes No

Stop the machine test process

Figure19. Illustrationofthedevicetestingprocesswith asmallnumberoffaultyLEDboards

Figure20. Areportsampleofthetestingresult

wasalsotestedforcurrentandvoltagevaluesdirectly acrosstheLEDsusingamanualrandommeasurement methodwithamultimeter.Theresultsareasfollows:

theaveragecurrentthrougheachLEDrangedfrom96 mAto100mA,andtheforwardvoltagerangedfrom 2.7Vto3.0V.Thesevaluesconformtotherequired speci ications.Aslightdeviationbetweenthecalcu‑ latedandmeasuredvalueswasobserved,attributable tothecableresistanceconnectingtheLEDboardsand LFDmodules.Theaverageactualpowerconsumption ofafunctionalLEDiscalculatedas ��1 = �� ≈ 0.28��,whiletheaveragepowerconsumptionofan LEDbranchandthetestingcircuitis ��2 = �� ≈ 0.49��,wheretheaveragecurrentthroughasin‑ gleLEDis �� =0.098�� andtheaveragevolt‑ agedropacrosstheLEDis �� =2.85��.Conse‑ quently,thecalculatedenergyef iciencyofthetesting deviceis ��=��1/��2 =0.57,whichremainssubopti‑ mal.Toenhanceef iciency,futuredesignadjustments maybenecessary,particularlybyloweringthesupply voltage.

Incomparisonwithpreviousresearchmethods thatemployedimageprocessingtechniques[5],neu‑ ralnetworkscombinedwithSVM[7],andlearn‑ ingmachines[10],whichachievedhighaccuracy forsmall‑scaleinspectionsbutincurredhighcom‑ putationalcosts,ourpresentedapproachissimpler andcomputationallyef icientonlow‑endprocessors. Moreover,itcanbeeasilycalibratedandscaledfor high‑throughputinspection.

Insummary,thedevicetrialmettheessential requirements,effectivelydistinguishingthestates oftheLEDboards.However,thepracticaltesting processrevealedsomelimitations:thesamplesize wasinsuf icient,withonly iveboardstestedacross 40trials;thedevicewastestedsolelyon3‑LED boards,withoutcomprehensivetestingon1‑or2‑LED boards.

Additionally,long‑cyclesequentialtestingwasnot performedduetothesigni icanttimerequired.One reasonforthislimitationwastheinsuf icientnumber of1‑LEDand2‑LEDboardsprovidedbythecustomer. Nonetheless,the1‑LEDand2‑LEDboardssharethe sameLEDcomponentsasthe3‑LEDboardsandlack current‑limitingresistors(seeFigure13a).Therefore, the3‑LEDboardswereconsideredrepresentative andusedassubstitutesfortheothertypesduring testing.Finally,thedevicehasbeentransferredtothe operationalunitwithoutreportedissues,exceptfor arequesttoupdatethecontrolprogramtoenablethe displayofinspectiondataforLEDboards.TheArduino

programsourcecodeisaccessibleat:https://github. com/anhcdt/ArduinoMega‑code/tree/main

6.Conclusion

Inthispaper,adesignproposalandimplementa‑ tionofatestingdevicefordetectingLEDfailureson circuitboardshavebeencarriedout.Thedesignsolu‑ tionisbasedonclassifyingtheoperatingconditionsof eachLED:opencircuit,shortcircuit,andregular.The classi icationcircuitreliesontheworkingresistance thresholdoftheLEDtoidentifyfaults.Toenablesimul‑ taneoustestingofmultiplecircuitboards,thecentral controlcircuitutilizesanArduinoMegaand74HC165 inputexpansionmodules,allowingforanextension ofupto1200inputs.Thecontrolalgorithmoperates basedoncalculatingcontinuoussamplingfrequency throughouttheoperatingperiodtoaccuratelydeter‑ minethestatusofeachLEDandgeneratetheresult report,whichcanbeaccesseddirectlyviaaUSB‑ Comportconnectionorindirectlyviaamemorycard. Thetestresultsindicatethatthedesigneddevicecan accuratelydetectthefaultspresentinLEDboards.In thefuture,severalworkscanbeidenti iedasfollows:

(i) Acomprehensiveanddiversesetofcircuitboard samplesfromthefactorywillbecollectedfor testingandoverallmachineevaluationinacon‑ tinuousimprovementcycle.

(ii) Amachinelearning‑basedapproachwillbe incorporatedtoanalyzefaultpatternsusing time‑seriesdatastorage,particularlyindesign enhancementsforexpandingRGBLEDs,high‑ powerCOBLEDs,ormulti‑channelLEDsystems.

(iii) ThesystemwillbeconnectedtoInternetof Thing(IoT)platformsviaBluetoothandWi‑Fi, enablingreal‑timeremotemonitoringandpre‑ dictivemaintenance.

(iv) AGUI‑basedinterfacewillbedevelopedtodis‑ playresultswithinanindustrialsetting.

(v) Optimizationofthedevice’senergyef iciencywill alsobeconsideredinmachinedesignadjust‑ ments.

AUTHORS

VanAnhPham∗ –DepartmentofMechatronics, FacultyofTechnologyEngineering,PhamVanDong University,CamThanhward,53129,QuangNgai province,Vietnam,e‑mail:pvanh@pdu.edu.vn, https://orcid.org/0000‑0002‑6793‑5260.

MinhTienDo –DepartmentofMechanicalEngi‑ neering,FacultyofTechnologyEngineering,PhamVan DongUniversity,CamThanhward,53129,QuangNgai province,Vietnam,e‑mail:dmtien@pdu.edu.vn.

∗Correspondingauthor

ACKNOWLEDGEMENTS

Wewouldliketothanktheengineer,CatLenLe,from DKMECCompanyfordesigningthechassisframeand providingvaluabledesigninputforthisproject.

References

[1] X.Longetal.,“AReviewonLight‑EmittingDiode BasedAutomotiveHeadlamps,” Renewableand SustainableEnergyReviews,vol.41,2015,pp. 29–41.

[2] D.BusekandK.Dusek,“OverviewofVarious FailuresofLuminairesAssembledwithSMDLED Chips,” 202245thInternationalSpringSeminar onElectronicsTechnology(ISSE),2022,pp.1–5.

[3] M.Estekietal.,“LEDSystemsApplicationsand LEDDriverTopologies:AReview,” IEEEAccess, vol.11,2023,pp.38324–38358.

[4] R.Sananda,C.Tejaswi,andB.Rishaya,“Trends andChallengesinAutomotiveHeadlamps,” Proc.SPIE,vol.12638,2023,p.126380R.

[5] D.‑B.Perng,H.‑W.Liu,andC.‑C.Chang, “AutomatedSMDLEDInspectionusingMachine Vision,” TheInternationalJournalofAdvanced ManufacturingTechnology,vol.57,no.9,2011, pp.1065–1077.

[6] K.Fumagalli,R.Faranda,andL.Farnè,“Analysis ofPossibleLEDFailureMode,” 2014Petroleum andChemicalIndustryConferenceEurope,2014, pp.1–8.

[7] H.Jiangetal.,“LEDDeviceFaultDiagnosis BaseonNeuralNetworkandSVMModelAnal‑ ysis,” 201714thChinaInternationalForumon SolidStateLighting:InternationalForumon WideBandgapSemiconductorsChina(SSLChina: IFWS),2017,pp.45–47.

[8] A.AroraandV.Goel,“RealTimeFaultAnal‑ ysisandAcknowledgementSystemforLED String,” 2018InternationalConferenceonCom‑ puting,PowerandCommunicationTechnologies (GUCON),2018,pp.457–461.

[9] H.K.Fuetal.,“AcceleratedLifeTestingandFault AnalysisofHigh‑PowerLED,” IEEETransactions onElectronDevices,vol.65,no.3,2018,pp. 1036–1042.

[10] M.S.Ibrahimetal.,“MachineLearningand DigitalTwinDrivenDiagnosticsandPrognos‑ ticsofLight‑EmittingDiodes,” Laser&Photonics Reviews,vol.14,no.12,2020,p.2000254.

[11] Y.Shangetal.,“ANovelFaultDiagnosisStrategy forLEDLampsviaLightOutputTime‑Frequency CharacteristicsAnalysisandMachineLearning,” Heliyon,vol.9,no.9,2023.

[12] J.R.Martı́nez‑Pérezetal.,“AdvancedDetection ofFailedLEDsinaShortCircuitforAutomotive LightingApplications,” Sensors,vol.24,no.9;doi: 10.3390/s24092802.

DESIGN,IMPLEMENTATION,ANDPERFORMANCEOPTIMIZATIONOFAROSBASED

Submitted:6th November2024;accepted:24th February2025

NeslihanDemir,PinarDemircioglu,IsmailBogrekci

DOI:10.14313/jamris‐2026‐010

Abstract:

Thisstudydealswiththedevelopmentandoptimization ofanROS‐basedAutonomousMobileRobot(AMR)for intralogisticsinproductionenvironments.Witharobust mechanicaldesign,high‐levelsensorfusion,andHybrid A*pathplanning,thisAMRoffersconsiderableflexibil‐ity,safety,andefficiencygainsovertraditionalauto‐matedguidedvehicles(AGVs).TheproposedAMRsystem allowsfordynamic,autonomouslocomotioninintricate industrialsettings,withadaptiveobstacleresponseand variablepayloads.Verificationinbothvirtualandreal‐lifesettingsdemonstratethesystem’seffectivenessin navigationprecision,energyefficiency,andload‐carrying capacity,highlightingitspotentialtoimprovelogistics operationsandreduceoperationalcostsintheIndustry 4.0context.

Keywords: AutonomousMobileRobots(AMR),Robot OperatingSystem(ROS),Intralogistics,HybridA*Algo‐rithm.

1.Introduction

Industry4.0isaneracharacterizedbytrans‑ formativeevolutioninthe ieldofmanufacturing, facilitatedbythehighspeedatwhichdigitaliza‑ tion,automation,andtheintegrationofadvanced technologiesareevolving.Itischaracterizedbythe combinationofrobots,theInternetofThings(IoT), andbigdataanalytics,bringingthesetechnologies togetherinanintegratedwhole[1].Theindustrial revolutionhasintroducedtheideaofLogistics4.0, whichfavorsinternallogisticsthroughautomation anddata‑orientedstrategiesthatconsequentlyreduce thedemandforhumanlaborinmaterialhandling. AutonomousMobileRobots(AMRs)havebecomea centraltechnologywithinthisrevolution,providing the lexibilityandversatilityrequiredforintricate anddynamicmanufacturingsetups[2].Incontrastto AutomatedGuidedVehicles(AGVs),whichrunalong pre‑programmedroutes,AMRsutilizesensors,com‑ plexsystems,andalgorithmsforautonomousnavi‑ gation,whichenablethemtoresponddynamicallyto changesintheirenvironment[3].

AMRsandAGVsarebothextensivelyutilized forintralogisticstasks,buttheygreatlydifferin navigationmethodsandfunctionality.

AGVsrequirephysicalinfrastructuresuchasmag‑ neticstrips,wires,ormarkersonthe loortolead themonsetpaths.Thissetupfacilitatesreliabilityand precisioninfacilitieswhererepetitive,organizedpro‑ cessesarethenorm.However,AGVslackthe lexibil‑ ityrequiredtonavigateunstructuredsettingswhere pathsundergofrequentchange.Hence,theyareless suitableforenvironmentsthatcallfor lexibilityand adaptabilitytonewlayouts,e.g.,modernmanufactur‑ ingplants[4,5].

AMRstransformintralogistics.Unlikeconven‑ tionalsolutions,theydon’tneedpre‑de inedroutes; rather,theyutilizeadvancedsensorslikeLiDAR, cameras,andinertialmeasurementunits(IMUs)to dynamicallyandintelligentlymapandnavigatetheir surroundings.AMRsusemethodslikesimultaneous localizationandmapping(SLAM)inconjunctionwith path‑planningalgorithmstobuildtheirownmapof theroute,regardlessofanyalterationintheenviron‑ mentinreal‑time.SuchmobilityallowsAMRstomove overandaroundobstaclesef iciently,re‑routethem‑ selvesautomatically,andadapttonewconditions, makingthemparticularlysuitableforcomplexman‑ ufacturingenvironmentswhereoperationalrespon‑ sivenessandef iciencyareoftheessence[6, 7].In addition,AMRsareengineeredwithsafetyfeatures thatallowthemtooperateinpartnershipwithhuman workers,allowingthemtomovearoundworkingenvi‑ ronmentstogetherwithoutcausinganyharm.Thisis afeaturethatAGVscannotoffertothesamedegree, sincetheymovealongprede inedpaths[8].

AMRsarehighlyvaluableinfactories,wherethey optimizeproductivity,conservelabor,andminimize downtime.Bycarryingoutrepetitiveandmanual transporttasksontheirown,AMRsenablehuman labortofocusonmoreadvancedtasksandthusboost itsproductivity.Suchinternallogisticsautomation alignswiththeprinciplesofLogistics4.0,whichpro‑ motespeed,accuracy,anddataintegrationinmate‑ rialhandling[2,9].Incontrasttotraditionalautoma‑ tion,whichtypicallyrequireslargeinfrastructuresand ixedpathways,AMRsallowformore lexibilityand scalabilityinamoreadaptivesolutionthataddresses evolvingproductiondemandandwork lowcon igura‑ tionswithoutrequiringsigni icantalterationstofac‑ torylayouts[10].

Intermsofstate‑of‑the‑artrobotics,AMRsareat theintersectionofrobotics,arti icialintelligence,and InternetofThings(IoT)technology.Thisintersection makesitpossibleforAMRstogather,process,and respondtovastvolumesofinformationinrealtime, therebyfacilitatingtheireasyoperationinsophis‑ ticatedenvironments.TheRobotOperatingSystem (ROS)platformhasbeenkeytoadvancingtheevo‑ lutionofAMRsthroughthefacilitationofamodular, open‑sourcestructureforintegratingmultiplesen‑ sors,softwarealgorithms,andcontrolsystems.ROS enablesreal‑timecontrol,communication,andsensor fusion,andishenceanidealframeworkforAMRsin dynamicmanufacturingsystems[11,12].

Evidencecollectedthroughresearchcarriedout in[12]and[13]demonstratestheef icacyofROS‑ basedAMRs(includingsoftwaresuchasGazeboand MoveIt)inthesimulation,development,andoptimiza‑ tionphasesofroboticsystems.

ThedesignandadvancementofAutonomous MobileRobots,alsoreferredtoasAMRs,requires cross‑disciplinarycollaborationbetweenmechanical design,electronics,andsoftwaresystemsintegration. Onthemechanicalfront,itisnecessarythatAMRsbe madehighlyreliable,stable,andversatileregarding thevarietyofpayloadstheycanaccommodateand theevolvingphysicalrequirementsoffactory loors. A2014study[15]emphasizestheneedfordevel‑ opingcooperativemobilerobotswithliftingdevices, therebyincreasingtherobots’load‑carryingability. Thisisafeaturevitaltomaterialhandlingtasksin productionenvironments[14].Ef icientmechanical designoftenemployslightweightmaterialsandmodu‑ larcomponentstosupportef iciencyandaffordability withoutcompromisingonstrongperformance.

Withinthedomainofelectronics,anautonomous mobilerobot(AMR)employsaseriesofsensorsto acquirethevitalenvironmentalinformationneeded formovement,collisiondetection,andbalancing.In general,AMRshaveemployedaseriesofLiDARsen‑ sors,cameras,andanIMUtoconstantlymonitortheir surroundingsandreacttoenvironmentalchanges.In particular,asensorysystememployingLiDARand stereocamerasandutilizingaRobotOperatingSys‑ tem(ROS)hasbeenpresentedin[7],highlightingthe impactoflarge‑scalesensorarrangementsonenhanc‑ ingrobotnavigationandcollisionavoidancecapa‑ bilities.Additionally,[16]exploresstabilizationtech‑ niquesusingsensorfusion,highlightingthesigni i‑ canceofintegratedsensorsinenhancingAMRperfor‑ manceagainstvaryingloadconditions.Thesedevel‑ opmentsenableAMRstooperateautonomouslywhile adjustingtoenvironmentalandoperationalvariables.

TheROSframeworkhasbeeninstrumentalin advancingAMRdevelopmentbyprovidingrobust toolsforsoftwareintegrationandreal‑timesystem communication.ROS’sopen‑sourcestructureallows developerstoimplementadvancedfunctionalitieslike SLAM,pathplanning,andobstacledetection,makingit anidealchoiceforAMRsthatmustoperateindynamic, unstructuredenvironments.

Quigleyetal.introducedROSasaversatile frameworkthatenablesseamlesscommunication betweenvarioussensors,actuators,andcontrol algorithms,furtherenablingAMRstohandle complexnavigationanddecision‑making processes[11].

ThedemandforAMRsworldwidehasrisen greatly.ThemarketforindustrialAMRswasvalued atapproximately$1.97billionUSDin2021,andis anticipatedtocontinuetogrowasindustriesembrace moreautomation[10].MajorcompanieslikeAmazon haverealizedthepotentialofutilizingAMRsfor large‑scaleuse.OneexampleisAmazon’sProteus robot,whichmovesbyitselfaroundwarehouseareas alongsidehumanworkers.Proteusdemonstratesthe abilityofAMRstoseamlesslyandsafelyintegrateinto collaborativeworkspaces,therebyillustratingtheir signi icanceinlarge‑scalelogistics[10].BeyondAma‑ zon, irmslikeMobileIndustrialRobots(MiR),Boston Dynamics,andGeekPlushaveledthedevelopmentof AMRsolutionsinsectorslikemanufacturing,health‑ care,andwarehousing,illustratingthe lexibilityand broadapplicationsofAMRtechnology[8,9].

AsthedemandforAutonomousMobileRobots,or AMRs,continuestoexperienceadrasticupwardtrend, bothresearchandcommercialinnovationareplaying theirrespectiverolesinfuelingtheinnovationrateof thisrapidly‑evolvingindustry.Thepresentresearch projecttakesacuefromthisestablishedbodyof workbyproposinganintegratedandcomplete systembasedontheRobotOperatingSystem,or ROS,speci icallydesignedforAMRs.Theproposed systementailstheintegrationofmodernmechanical designprinciplesandsensortechnologydesignedfor enhancedperformance,aswellastheapplicationof advancednavigationalgorithmsthatareintelligent innature.Alltheseelementsareparticularlytailored toaddressandeffectivelysolvethecomplex intralogisticsproblemsthatarecommonlyfaced bymodernmanufacturingplantstoday.

Table 1 showsthecomprehensivedifferences betweenAMRsandAGVs,withtheirrespective navigationsystems, lexibilitylevels,adaptability,and operationalcosts,supportedbypertinentscholarly literature.

2.DesignandMethodology

TheROS‑basedautonomousmobilerobot(AMR) usedforintralogisticspurposesinamanufacturing factoryisanintegrativesystemthatcomprises mechanical,electronic,andsoftwaresystem components.Inthissection,weelaborateonthe considerationsthatguidedthedevelopmentofthe ROS‑basedautonomousmobilerobot(AMR),the developmentandtestingprocesses,andmeasures thatensureitsoptimalperformance.

2.1.MechanicalDesign

TheAMR’smechanicaldesignprocessplaced ahighpriorityonseveralkeyaspects—including durability,loadcapacity,andmaneuverability— inordertoaddressandful illingthediverse requirementsofvariousmanufacturingsettings.

Table1. ComparisonofAMRsandAGVs

Feature AMRs AGVs

NavigationTechnology

PathDependency

EnvironmentalAdaptability

AI‑drivensensor‑basednavigation (LiDAR,cameras,millimeter‑wave sensing)[17]

Noprede inedpaths;dynamically plansroutesinreal‑time[17]

Highlyadaptabletounstructuredand dynamicenvironments[17,19]

ObstacleDetection AdvancedAI‑basedobstacle detectionwithreal‑timepath adjustments[20]

OperationalFlexibility

ImplementationCost

ApplicationSuitability

Path‑PlanningAlgorithms

High lexibility;cannavigatenew environmentswithoutpre‑set guides[21]

Higherinitialinvestmentdueto advancedsensingandAI[22]

Smartfactories,adaptivelogistics, andwarehouses[17,18]

HybridA*,RRT,D*,and reinforcementlearning‑based methods[23]

Toachieveoveralldesignstrengthanddurability, aprocessknownas initeelementanalysis(FEA)was thoroughlyutilizedasanessentialandinvaluabletool toanalyzeandexaminethestructuralstrengthofthe AMR.Testsincludedvariousoperationalloadingcon‑ ditionstowhichtherobotwouldinevitablybesub‑ jectedinitsreal‑worldapplicationsandoperations. ThroughthehelpofFEA,thechassiswasdesigned tobestrongenoughtocarrychangingloadsandbe lightinweight,usingaluminum5754‑Osheetmetal toachievetheoptimalstrength‑to‑weightratioforthe bestperformanceandconsequentlymaximummobil‑ ityandenergyef iciency.Fig.1displaysthechassisof theAMR,establishingthelayoutofload‑carryingseg‑ mentsandmotorbeddingnecessaryforstablefunc‑ tionality.

Inordertoimproveits lexibility,theAMRhas beendesignedwithamodularstructurethatfacili‑ tateseasyassembly,maintenance,andreplacementof itsmodules.

Follow ixedpathsusingmagneticstrips, beacons,orQRcodes[18]

Fixedpathswithminimaldeviationfrom prede inedroutes[18]

Limitedtostructuredenvironments withprede inedroutes[18]

Basicobstacledetection;usuallystops whenencounteringobstacles[18]

Low lexibility;requiresinfrastructure modi icationforroutechanges[21]

Lowerinitialinvestmentbuthighercost forinfrastructuresetup[22]

Manufacturinglines,repetitivelogistics, andcontrolledenvironments[17]

Mostlyrule‑basedor ixed path‑followingalgorithms[23]

Figure2showsthemodularstructureoftheAMR system,designedasa3Dmodelwitheasilydistin‑ guishablepartsthatmakeiteasytocustomizethesys‑ temfordifferentoperationalrequirements.Indesign‑ ingtheAMRsystem,kinematicanalysishasbeenused forthewheelsandactuatorsofthesystemtoenable theAMRsystemtopassthroughmanufacturingenvi‑ ronmentswithcomplexandnarrowpaths.

2.2.ElectronicSystemDesign

TheAMR’selectronicarchitecturehasbeen designedforaccuratelocalization,dataprocessing, andcommunication.Acomprehensivesetofsensors, comprisingaLiDAR,astereocamera,andaninertial measurementunit,arethecoreoftheelectronic architectureoftheAMR.Thissetofsensorsequipsthe AMRwiththecapabilityofsimultaneouslocalization

Figure1. Exploded3Dmodelofmobiletransportation robot

Figure2. 3Dmodelviewofmobiletransportationrobot

Table2. DCmotorgeneralspecifications

GeneralSpeci ications

Ratedvoltage 12V

Size 37D×70L mm

Shaftdiameter 6 mm

Gearratio 70:1

Speedwithoutload 150rpm

Speedatmax.ef iciency 130

Currentwithoutload 0.2 A

Currentatmax.ef iciency 0.68 A

Stalltorque 27kg/cm

Torqueatmax.ef iciency 32kg/cm

Encoderresolution 64CPR

andmapping,aswellastheabilitytodetectobjects onthenavigationroute,thusenablingtheAMRto createamapoftheenvironmentinrealtime.

Inordertoachieveaccurateandstabledatafrom thelocalizationsystem,necessaryforthesafenavi‑ gationoftheAMR,thedatafromtheLiDAR,camera, andIMUarefusedtogetherusingKalman iltermeth‑ ods[16].

Themainpowersourceutilizedbythe AutonomousMobileRobot(AMR)isalithium‑ ionbatterypack.Thisiseffectivelyutilizedthrough abatterymanagementsystem.Thesystemensures continuousmonitoringofthepowerlevelspresent inthebatteries.Moreover,itoptimizestheutilized powertoattainthemaximumlevelsofef iciency. Thecon igurationensureseffectiveutilizationof thebatteriestocatertosituationswherehigher levelsofexpenditurearerequired,likeliftingheavier loadsandmovingoverlargerdistances.Forprecise andef icientcontrolofthemovementoftheAMR,DC motorshavebeenchosenasthemostef icient.Forthis purpose,controlalgorithmshavebeenprogrammed toadjustthespeedanddirectionofmovementbased onthechangingconditionsprovidedbytheload. Table 2 presentsadetailedanalysisoftheutilized motors,focusingparticularlyonthecon iguration thatensuresef icientmovement.

2.3.ROS‐BasedSoftwareIntegration

ROSoffersanunderlyingsoftwareinfrastructure, supportingmodularity, lexibility,andreal‑timepro‑ cessingofthedata.ROS’sgmappingpackagewas usedtoprovideSLAMcapability,whichallowedthe AMRtobuildandupdateitsenvironmentalmapwhile navigatingaroundthefacility.Thisfunctionalityis depictedasaCreationMapbyslam_gmappingNode, whichshowsasamplemapbeingbuiltbytheAMR whileitnavigatesthroughvariouszones.

Forpathplanningandobstacleavoidance,ROS’s movebasepackageisused,whichincludesaHybrid A*algorithmthataidsinoptimizingpaths.Thisisben‑ e icialforanAMRbecauseobstaclescanbeavoided inreal‑time,thusfacilitatingsafemovementaround otherequipmentandpeople.BasedonFigure 4,the HybridA*algorithmisbene icialforanAMRbecause itaidsinef icientmovementthroughcomplexand sharedareas,thusemphasizingits lexibilityforuse inactivemanufacturing.

Creationofmapbyslam_gmappingnode accordingtoreal‐worldenvironment

Amulti‑threadedROSnodecon igurationwas employedtohandleconcurrenttasks,suchaslocaliza‑ tion,sensormonitoring,andmovementcontrol.This structureallowstheAMRtoprocessreal‑timedata andadaptcontinuouslyduringoperation,providing responsiveandautonomousnavigation.Fig. 5 illus‑ tratesthedataexchangeessentialforsmoothopera‑ tionindynamicmanufacturingsettings.

2.4.TestingandValidation

TheAMRunderwentrigoroustestinginbothsim‑ ulatedandreal‑worldenvironmentstovalidateits performanceinnavigation,loadhandling,obstacle avoidance,andenergyef iciency.Simulationtesting was irstconductedintheGazeboenvironment,which allowedforiterativeadjustmentsandvalidationof SLAMandpath‑planningalgorithms.Thisstagepro‑ videdacontrolledsettingforoptimizingtheAMR’s navigationandobstacle‑avoidancefunctions.

Followingthesuccessofthissimulation,theAMR wasdeployedinacontrolledsectionofamanufactur‑ ingfacilitytoassessitsperformanceinloadhandling andreal‑worldnavigation.Inthisphase,theAMR demonstrateditsabilitytoautonomouslytraversethe facilityandadjusttobothstaticanddynamicobsta‑ cles,effectivelyhandlingvariousloads.Table 3 pro‑ videsaclearexplanationoftheoptimizedforceand torquecon igurationsthatplayafundamentalrole inguaranteeingstabilityandachievingenergyef i‑ ciencyintheprocessofloadtransport.Thevaluespro‑ videdinthetablewerecalculatedmeticulously,taking intoconsiderationalargenumberofslopeanglesin ordertofullyinvestigatetheeffectsoftheseangleson thetractionalforce,totaldrivingforce,andtractional torqueinvolvedintheoperation.

Adetailedstatisticalanalysisoftheresultscanbe obtainedbyreferringtotable 4,whereitispossible toseetheessentialparameters:themean,standard deviation,minimum,maximum,andrange.Thiswill becomecrucialfortheinterpretationofhowbothcon‑ sistencyandvariationarerepresentedintheresults forforceandtorqueinthevariouscircumstances.

Figure3.

Table3. Resultsofanalyticalcalculationsforeachwheelofmobiletransportationrobot

Table4. Resultsofanalyticalcalculationsforeachwheelofthemobiletransportationrobot

Table5. Comparisonofimplementedpathplanningalgorithms

HybridA*Algorithm

•Demonstrateshigherprecisionindynamic environments.

•Continuouslyadjuststhepathbasedonreal‑time sensordata.

•Ef icientlynavigatesaroundobstacles,ensuring collision‑freemovement.

•Particularlyeffectiveinenvironmentswherefrequent adjustmentsarenecessaryduetochangingconditions.

•Showsrobustnessinmaintainingtherobot’sposewith theaidoftheAMCLalgorithm.

Toobtainawiderrangeofcon irmatoryresults, theperformanceofAMRwasalsotestedwithvarious performancemetrics,includingnavigationvelocity, obstacledetectionprecision,loadstability,andenergy consumption.Table 5 showsthat,whenvarious path‑planningalgorithmswerecompared,HybridA* provedtobethemosteffectivealgorithmfora dynamicenvironment,whileMoveBaseprovedtobe highlyeffectiveforenvironmentswithamoreroutine pattern.Theseresultscon irmthattheproposedAMR iseffectiveforuseinanintralogisticsroleinamanu‑ facturingenvironment.

3.TechnicalContribution

TheROS‑basedAMRintralogisticsdesignsignif‑ icantlyenhancesproductionmaterialhandling.It combinesrobustmechanicaldesign,advancedsensor fusion,andtheHybridA*path‑planningalgorithm toprovidemore lexibility,precision,andsafetythan traditionalAGVs.Thepaperpresentsnoveldevelop‑ mentsinthedesign,navigation,andpath‑planningof AMRsformanufacturingintralogistics.Eachcontri‑ butionutilizesthelatestandbesttechniquesinthe industry,andincludeswell‑designedmechanismsto rendertheAMRef icient, lexible,anddependablein dynamicmanufacturingsettings.

MoveBase(A*Algorithm)

Showsstrongperformanceinreal‑timepath adjustmentsandobstacleavoidance.

•Therobot’splannedpathisdynamicallyupdated baseditsposition,aswellasnewlydetectedobstacles.

•Maintainsaccuratelocalizationandadjustsits trajectoryef iciently.

•Suitableforenvironmentswithwell‑de inedobstacles andpredictablechanges.

•ReliesheavilyontheAMCLalgorithmforconsistent andaccurateposeestimation.

ThemechanicalstructureoftheAMRisload‑ bearing,optimizedthroughdiligentFEAanalysis offorcesandtorquesforvariedconditions.High‑ strengthaluminumallowsthechassistoendureheavy loadswhileremaininglightinweight,whichisnec‑ essaryformaneuverabilityandoperationalef iciency. Fig.6isagraphoftorquevaluesagainstpayloadmass andslopeangle,aswellasthemotorrequirements undermaximumload.Thisequilibriumensuresreli‑ ablematerialtransportfortheAMRacrossslopeswith varyingconditionswithinafactorytominimizethe useofmanualeffortandincreaseef iciency.Analyzing structuresledtotheselectionofaccuratemotorswith speci iedtorqueperformanceforpreciseimprove‑ mentoftherobot’sload‑handlingcapabilities.

Forpreciseandautonomousnavigation,the AMRemploysaSLAMsystem,implementedthrough theslam_gmappingROSpackage.Thissystemuses LiDARandodometrydatatogeneratereal‑time maps,allowingtherobottonavigatedynamiclayouts withouthumanintervention.Fig. 7 illustratesthe detailedmapscreatedduringGazebosimulations, whichenabledhigh‑accuracynavigationacross variousenvironments.TheSLAMprocessachieved localizationaccuracywithin2cmduringreal‑world testing,ashighlightedbyodometrydatainFig.8.This precisioniscrucialforsafeoperationsinmanufactur‑ ingenvironmentswithconstantlychanginglayouts andmobileobstacleslikeequipmentandpersonnel.

(c)

Figure4. PathplanningwithHybridA*node

Oneofthebiggestcontributionstotheperfor‑ manceoftheAMRisitsintegrationofaHybrid A*algorithm,whichcreatesdynamicpathplanning basedonglobalandlocalenvironmentaldata.Differ‑ entfromconventionalalgorithms,HybridA*dynam‑ icallychangestherouteinrealtimesothattheAMR canadjustthepathaccordingtoobstaclesperceived bysensors.Fig. 4 illustratestherobot’smovement throughanenvironment,modifyingitsrouteasthe HybridA*algorithmreactstoenvironmentalchanges. ComparativeperformancedatainTable 3 highlights theef iciencyofthealgorithminguidingtherobot throughcomplicatedindustrialareascomparedto otheralgorithmssuchasMoveBase.Such lexibil‑ ityguaranteesseamless,collision‑freefunctionality

Figure5. Publishedandsubscribedtopicsduring localizationprocess

Figure6. Thetorqueischangingintermsofmassand slopeangle

inenvironmentswithobstaclesarisingunpredictably andimprovesthesafetyandef iciencyofAMR.

Thecombinationofoptimizedmechanicaldesign, SLAM‑basedlocalization,andHybridA*path‑ planningallowstheAMRtooperateautonomously withlittlesupervision.Theload‑carryingstructure providesstablematerialhandlingwithenergy savings,theSLAM‑basedmappingallowsforongoing localizationindynamiclayouts,andtheHybrid A*algorithmdynamicallyoptimizesroutes.These featurestogetherallowtheAMRtoful illthedemands

Figure7. Createdmapsbyslam_gmappingnode accordingtogazeboworldsonROS

Figure8. ExampleoutputoftheROSodomtopic obtainedduringexperimentalvalidationtests performedduringthedevelopmentphaseoftheAMR systemgeneratedduringreal‐worldAMRoperation ofmodernmanufacturing,demonstratinganadaptive androbustintralogisticssolution.

Theworkpresentedhereinclearlyillustrates AMR’simpressivecapacityforindependentmove‑ mentandeffectiveoperationwithinintricateandmul‑ tifacetedenvironments.Thisfunctionalityoffersasig‑ ni icantbasisuponwhichtheinvestigationanddevel‑ opmentofwiderapplicationsforautonomousmobile robotscanbepursued,particularlyintherealmsof intralogisticsandotherassociateddomains.

4.PerformanceEvaluation

TheAMRnamedProdigiMoverwascarefully testedwithanarrayofintensiveprocessesapplied

bothinsimulationenvironmentssuchasGazeboand inreal‑worldapplications.

Theassessmentfocusedonawiderangeofper‑ formancemetricsthatareessentialtotheAMR’sfunc‑ tionalapplicationwithinthedomainofintralogistics. Theseincludedeterminingtransportspeed,localiza‑ tionaccuracy,payloadhandlingability,batterylifes‑ pan,andadaptabilityacrossvariedscenarios.Perti‑ nent iguresandtablesfromthedissertationareeffec‑ tiveinrepresentingthehigheffectivenessandperfor‑ manceoftheAMRwithinthesecriticalparameters. Thesegraphicalrepresentationsofferclearandin‑ depthinsightintoProdigiMover’sfunctionalitywhen comparedtotraditionalAGVs,depictingitsbene its andpotentialuseinthesector.

AMRdemonstratedahighlevelofconsistency intransportspeedandpathaccuracyinalltesting phases.InvirtualtestingwithintheadvancedGazebo environment,theAMRperformedexcellently,record‑ inganimpressivemeantransportspeedof0.5meters persecondwhilemakinglittleornodeviationin itspathfromitsplannedroute.Thisperformance levelledtoanoutstandingachievementwherebyit recordedameandeviationinaccuracyoflessthan2 centimeters,anindicatoroftheprecisionofitsnav‑ igationsystem.Withregardtorealtestsperformed outsidethevirtualworld,theresultwasnodiffer‑ entfromthesimulations,anindicatorthatevenin morecomplexanddynamicenvironments,theAMR wasstillcapableofanoutstandingpathdeviation ofnomorethan3centimeters.Theef icacyofthe HybridA*path‑planningalgorithmisdemonstrated inFig. 4,whichshowsthetrajectoriesbywhichthe AMRdynamicallymovesaroundobstacles.Thepaths ofnavigationdemonstratetheAMR’sabilitytomain‑ tainhighaccuracy,evenincomplexenvironments. Fig. 7 presentsSLAM‑generatedmapsfromGazebo simulations,whichshowtheadherenceoftheAMRto plannedtrajectoriesundervaryingconditions.

Highaccuracylocalizationwascriticallyimpor‑ tantfortheAMR’sautonomousnavigationcapabil‑ ities.ByutilizingtheROSgmappingSLAMpack‑ age,theAMRwasabletolocalizewithhighpreci‑ sion,withsimulationtestresultsindicatingthatits errormarginremainedwithinamere1cm.However, whenconductingreal‑worldtestingincomplicated anddynamicenvironments,theerrormarginwas observedtoincreaseuptoamaximumof2cm.Fig.7 alsodisplaysthehigh‑qualitymapsthatweregener‑ atedduringtheGazebosimulations,portrayingthe mappingcapabilitiesoftheAMR.Thesecapabilities arefundamentallyrequiredforsuccessfulnavigation throughtheintricatelayoutsthattheAMRencounters. SLAM‑generatedmapfromreal‑worldexperiments canalsobefoundinFig3,highlightingtheaccuracyof theAMRinmappingenvironmentaldatanecessaryfor safenavigationinrapidlychangingindustriallayouts.

TheabilityoftheAutonomousMobileRobot,or AMR,tosupportloadsisakeyperformanceaspect

thatiscentraltoitsfunctionality.TheAMR’compli‑ catedmechanicaldesignensuresthatitiscapableof carryingpayloadsofupto100kgwithoutsacri icing stabilityintheprocess.

Inaddition,detailedforceandtorquesimulations wereconductedontheAMRusingsophisticatedsoft‑ waretoolssuchasMATLABandANSYS,whichveri ied andvalidatedtheAMR’sstructuralintegrityunder thesevariousloads.Fieldtestingandanalysisdemon‑ stratedthattheAMRoperatedreliablyandef iciently evenwhenitwasrunningatfullcapacity,thusprov‑ ingitsruggednessunderpracticalconditions.Table3 providesforceandtorquevaluesforeachwheel,con‑ irmingtheAMR’sstructuralintegritywhensubjected toloads.Fig.6alsoshowstorquevalueswithdifferent slopeanglesandpayloadsituations,illustratingthe AMR’scapabilitytowithstanddifferentpayloadsand inclines.

BatterylifewasalsotestedsothattheAMRwould bepreparedforthecontinuousoperatingdemands foundinindustrialenvironments.Utilizingalithium‑ ionbatterypackandbatterymanagementsystem,the AMRcouldrunforaslongaseighthoursonone charge,whichiscomparabletoastandardindustrial laborshift.Thislongerbatterylifemeanssigni icantly lessdowntime.Additionally,amodulardesignfeature allowsforsimplebatteryexchanges,whichallowsfor longeroperatinghoursifnecessary.

TheAMR’sreal‑timenavigationcapabilities,pow‑ eredbytheadvancedHybridA*algorithm,provide itwithadecisiveadvantageovertheolderAGVs. UnlikeAGVs,whichrelyonpre‑programmedroutes orexternalfeedbackmechanismstonavigatetheir environment,theAMRhastheabilitytoalteritsroutes dynamicallyinreal‑time,continuouslyrespondingto changingsensorinputsandenvironmentalrealities. Table 5 indicateshowHybridA*exhibitssuperior performancecomparedtoitscounterparts,i.e.,Move Base,inthecontextofverydynamicenvironments wheresophisticatedcollisionavoidanceand lexibil‑ ityarerequired.Thistypeofnavigation lexibility enablestheef icientmovementoftheAMRincommon industrialsettingswithregularobstaclesandlayout changes.

5.DiscussionandImplications

Theresultsofthisresearchhighlightthe signi icantadvantagesofAMRsformanufacturing productivityandlaboroptimization.Withreliable, autonomousguidanceandaccuratematerialhandling, AMRsoptimizeintralogisticsbyminimizingtheuse ofmanuallabor,maximizingproductivity,and reducingoperationaldowntime.Byintegratinga robustmechanicaldesignwithsophisticatedsensor fusionandaHybridA*path‑planningalgorithm, theAMRismore lexible,accurate,andsafethan conventionalAGVs.Testinginbothsimulationsand real‑worldenvironmentshasdemonstratedthe AMR’ssuperiorityinnavigationaccuracy,payload

handling,andenergyef iciency,withthepotentialto optimizeprocessesandreduceoperationalexpenses inIndustry4.0settings.Thesebene itsdemonstrate thein luenceofintelligentsystemsonmanufacturing andsuggestgreaterpotentialinarangeofindustries.

Therobotizationofmaterialtransportactivities inmanufacturingareasiskeytoenhancingtheir overallef iciencylevels.AutomatedMobileRobots,or AMRs,effectivelyrelievethehumanworkers’burden incarryingoutmundaneandlessskill‑intensiveactiv‑ ities,transportingmaterialsfromoneworkstation toanother.Insodoing,theserobotsenablehuman workerstodevotetheirtimeandenergytoworking onmorecomplicatedandhigh‑valueactivitiesthat requiregreaterskillsandcriticalthinking.Thiscon‑ siderablealterationintheutilizationoflabornotonly raisesproductivitylevelsacrosstheboard,butalso leadstoenhancedjobsatisfactionforemployeesby reducingtheneedforthemtocarryoutroutineand repetitivework,thusprovidingthemwiththeoppor‑ tunitytoworkonmorerewardingandful illingtasks.

Furthermore,thestreamlineddesignofAMR, togetherwithitsadvancedbatterymanagementsys‑ temandpower‑savingnavigation,translatesinto extendedoperationperiodswithreducedrecharging requirements.Thepowercycle’ssynchronizationwith industrialshiftlengthsreducesdowntimeandensures non‑stopproduction low.Modularconstructionalso makesquickandeasymaintenancepossible,with moreuptimeandlessdowntime.Allofthesetrans‑ lateintomeaningfultimeandlaborcostsavingsthat justifyAMR’svalueinhigh‑demand,reliability‑centric manufacturingsetups.

WhileAMRtechnologywascreatedwithmanufac‑ turinginmind,itisalsoversatile,withcross‑industry applicationsinhealthcare,warehousing,retail,and agriculture.Inhealthcare,medicalsuppliesorfood couldbedeliveredautomaticallybyAMRs,freeingup healthcarestafftospendtheirtimeonpatientcare. TheAMRs’precisenavigationandobstacleavoidance wouldbeespeciallyusefulinhectichospitalsettings, wheretheycouldmaneuveraroundpatientsandstaff withoutissue.

Inlogisticsandwarehousing,AMRsofferadapt‑ abilityand lexibilityduetotheirlackof ixedinfras‑ tructurerequirements.Beingabletoadapttochang‑ inglayoutsorseasonal luctuationsindemandmakes themmoreuser‑friendlyindynamicstoragefacilities. Inretail,AMRshavethepotentialtoautomateopera‑ tionssuchasrenewalofshelvesandorderful illment, makingthesupplychainmoreef icientwithinstores. AgriculturewouldalsohaveausecaseforAMR technologyinthearenaofprecisionagriculture.Sen‑ sorandmappingalgorithm‑basedAMRscould indan applicationinmonitoringcrops,detectingchangesin plantorsoilhealth,andeveninseedingorharvest‑ ingprocesses.Byreducingtheutilizationofhuman resourceswhileoptimizingtheutilizationofother resources,AMRscouldbringaboutmoresustainable methodologiesandimprovedcropyield.

The indingshereindiscussedareinkeepingwith theincreasinginterestandattentionaffordedtointel‑ ligentsystems’wider‑rangingimplicationsforvarious industries.

TheeffectiveuseoftechnologiessuchasSLAM, HybridA*pathplanning,andamodulardesign approachinthisparticularAMRmodelgoesalong wayindemonstratingtheincreasingabilityofintel‑ ligentsystemstooperateeffectivelyandwithhardly anysetupprocesstospeakofinenvironmentsthat areunstructuredandhumanactivity‑oriented.Such remarkable lexibilityisincreasinglypertinentin thesemoderntimes,especiallyconsideringthatvar‑ iousindustriesareactivelyseekingscalablesolutions withtheabilitytoexpandanddevelopintandemwith theevolvingneedsofthemarketplaceandadvancesin technology.

WhenAMRsareconnectedtoIoTnetworks,they cancommunicatewithotherrobots,machines,and centralsystems,automatingandoptimizingprocesses forentirefacilities.Integrationcanfacilitatepredictive maintenance,real‑timeanalytics,andcontinuousopti‑ mization.

6.Conclusion

AMRsaredesignedtoadapttovariousproduction setupsbyusingmodulardesigns,navigationbasedon arti icialintelligence,andscalablesystemintegration. Theirversatilitycomesfromsensorfusion,dynamic mapping,andindependentdecision‑making,which enablethemtomovearoundinstructuredspaces likeassemblylines,orhighlydynamicspaceslike warehouses.AMRs’modularityenablesadaptation tospeci icindustryneeds,withswappablepayloads, con igurablesoftwareapplications,andscalable leet managementsystems.IntegrationmustbeIndustry 4.0principle‑compliant,withIoTconnectivity,cloud analytics,andRPAforinteroperabilitywithexisting infrastructure.AMRscanbeutilizedinassemblylines forjust‑in‑timematerialdelivery,andinwarehouses formaintaininginventorylevelsusingreal‑timedata sharing.Inordertodeployef iciently,AMRsrequire leetmanagementsoftwareinthemiddle,smartdock‑ ingstations,andintegrationintoexistingERPsystems toachieveoptimalef iciencyacrossavarietyofman‑ ufacturingsettings.

Thisresearchoffersvitalinnovationsinthedesign andrealizationofAMRmechanicsforenhancing industrialautomationsolutions,withaspecialfocus onmanufacturingintralogistics.Throughitsef icient load‑carryingmechanics,accurateSLAM‑basednav‑ igation,andanovelHybridA*path‑planningalgo‑ rithm,thisAMRhasexcellentprospectsforenhancing operationalef iciency,minimizinglaborexpenditures, andenhancing lexibilityindynamicsituations.

TheAMR’sself‑navigatingabilityincomplexlay‑ outswithhighlocalizationaccuracyandoptimum obstacleavoidanceenablesittomanagerepetitive materialtransportoperations,reducingthenecessity

forhumaninterventionanddowntime.Therobot’s modulardesignandbatterymanagementsystemalso allowforeasyextensionofitsworkinghours.

Inhuman‑orientedspaces,AMRscannavigate dynamicareaswherehumanmovementcreates uncertainty.Unlike ixed‑pathAGVs,theyutilize real‑timesensorinputfromLiDAR,cameras,and millimeter‑wavesensorstodetectobstaclesandalter course.

Sensorfailureandunpredictablehumanmotion posesafetyhazards;thus,strictsafetyprecautionsare implemented,includingspeedlimits,emergencystop‑ ping,anddesignatedsafezones.AIpredictivemod‑ elsimprovehuman‑robotcollaborationbypredicting movement,minimizingthepossibilityofcollisions, andenablingwork lowintegration.Theseactionsare vitalforAMRstosafelycollaboratewithhumanwork‑ ersinIndustry4.0workplaces.

Futurestudiescanextendthesecontributionsby investigatingthepotentialofAMRsinotherindustries beyondmanufacturing.WhenintegratedwithIoTnet‑ worksandcentralcontrolsystems,AMRscanmake evengreatercontributionstointelligentautomation bydeliveringreal‑timefeedback,predictivemainte‑ nance,andongoingprocessoptimization.

Thisworkbringsthepotentialandpromiseof AMRsinthe ieldofindustrialautomationtoawhole newlevel.Itprovidesabasisfortheinnovationsimmi‑ nentonthehorizon—innovationswiththepoten‑ tialtocompletelyrevolutionizework lows,maximize productivitylevels,anddrivesustainabilityefforts acrossarangeofindustries.Withintelligentrobots settocontinuedevelopingandadvancingevenfur‑ ther,AMRsliketheonecreatedinthisprojectpresent anextremelyscalableandversatilesolution,ideally positionedtoaddresstheincreasingdemandsand complexitiesofmodernindustry.

AUTHORS

NeslihanDemir –Dept.ofIndustrialEngineering, IstanbulAydinUniversity,Istanbul,Turkey,e‑mail: neslihan.demir@adu.edu.tr.

PinarDemircioglu∗ –Dept.ofMechanicalEng,Fac‑ ultyofEng,AydinAdnanMenderesUniversity,Aydin, Turkey,e‑mail:pinar.demircioglu@adu.edu.tr.

IsmailBogrekci –Dept.ofMechanicalEng,Facultyof Eng,AydinAdnanMenderesUniversity,Aydin,Turkey, e‑mail:ibogrekci@adu.edu.tr.

∗Correspondingauthor

ACKNOWLEDGEMENTS

Thisstudy,fundedbytheScienti icResearchProjects ofAyd��nAdnanMenderesUniversity(MF‑22005),has beensupportedbytheScienti icResearchProjects CommissionofADU.

References

[1] J.Smitetal.,“Industry4.0:PolicyDepartmentA‑ EconomicandScienti icPolicy,”EuropeanParlia‑ ment,Directorate‑GeneralforInternalPolicies, February2016.

[2] M.B.AlatiseandG.P.Hancke,“AReviewonChal‑ lengesofAutonomousMobileRobotandSensor FusionMethods,” IEEEAccess,vol.8,2020,pp. 39830–39846.doi:10.1109/ACCESS.2020.29 75643.

[3] G.Fragapaneetal.,“PlanningandControlof AutonomousMobileRobotsforIntralogistics: LiteratureReviewandResearchAgenda,” Euro‑ peanJournalofOperationalResearch,vol.294, no.2,2021,pp.405–426.doi:10.1016/j.ejor.2 021.01.019.

[4] K.Katona,H.A.Neamah,andP.Korondi,“Obsta‑ cleAvoidanceandPathPlanningMethodsfor AutonomousNavigationofMobileRobot,” Sen‑ sors,vol.24,no.11,2024,3573.doi:10.3390/ s24113573.

[5] J.MorgensternandH.Zadek,“AMR:In luencing FactorsandPotentialsofCloud‑Robotics,” Proc. 16thInternationalDoctoralStudentsWorkshops onLogistics,SupplyChainandProductionMan‑ agement,2023,pp.81–88.

[6] J.R.Sánchez‑Ibáñez,C.J.Pérez‑del‑Pulgar,andA. Garcı́a‑Cerezo,“PathPlanningforAutonomous MobileRobots:AReview,” Sensors,vol.21,no.23, 2021,7898.doi:10.3390/s21237898.

[7] S.Santoro,“DesignandImplementationofaSen‑ sorySystemforanAutonomousMobileRobotin aConnectedIndustrialEnvironment,”master’s thesis,Dept.MechatronicEngineering,Politec‑ nicodiTorino,2021.

[8] GeekPlusTechnologies,“AutonomousMobile RobotSolutions,”Available:https://www.geek plus.com/en/(accessedSep.2024).

[9] ClearpathRobotics,“AutonomousMobile RobotsandRoboticsPlatforms,”Available: https://clearpathrobotics.com/(accessedSep. 2024).

[10] ROBOTS(IEEESpectrum),“Proteus–AutonomousMobileRobot,”2022.Available: https://robotsguide.com/robots/proteus (accessedSep.2024).

[11] M.Quigleyetal.,“ROS:AnOpen‑SourceRobot OperatingSystem,” Proc.IEEEInternationalCon‑ ferenceonRoboticsandEngineering,2009.

[12] A.Alessandro,“RoboticArmPick‑and‑Place Tasks:ImplementationandComparisonof ApproacheswithandwithoutMachineLearning Techniques,”master’sthesis,Dept.Mechatronic Engineering,PolitecnicodiTorino,2020.

[13] D.Tarantaetal.,“AnAutonomousMobileRobot NavigationArchitectureforDynamicIntralogis‑ tics,” Proc.2021IEEE19thInternationalConfer‑ enceonIndustrialInformatics(INDIN),2021,pp. 1–6.

[14] B.Hichrietal.,“LiftingMechanismforPayload TransportbyCollaborativeMobileRobots,” New TrendsinMechanismandMachineScience,vol. 24,P.Flores,F.Viadero,eds.,Springer,Cham, 2015,pp.157–165.

[15] D.Heath,“DesignofaCompetitionCapable HobbyRocket,”OxfordBrookesUniversity,2018.

[16] D.P.Sichkaretal.,“MovingElementsofMobile RobotsStabilizationModelling,” Proc.2019Sys‑ temsofSignalsGeneratingandProcessingin theFieldofonBoardCommunications,2019, pp.1–5.

[17] C.Wangetal.,“AutonomousMobileRobotNavi‑ gationinUnevenandUnstructuredIndoorEnvi‑ ronments,” Proc.2017IEEE/RSJInternational ConferenceonIntelligentRobotsandSystems (IROS),2017,pp.109–116.

[18] A.K.GroverandM.H.Ashraf,“Leveraging AutonomousMobileRobotsforIndustry4.0 Warehouses:AMultipleCaseStudyAnalysis,” InternationalJournalLogisticsManagement, vol.35,no.4,2024,pp.1168–1199.doi: 10.1108/IJLM‑09‑2022‑0362.

[19] R.Bostelmanetal.,“3DRangeImagingforUrban SearchandRescueRoboticsResearch,” Proc. IEEEInternationalSafety,SecurityandRescue RoboticsWorkshop,2005,pp.145–150.

[20] A.Vemula,K.Muelling,andJ.Oh,“PathPlanning inDynamicEnvironmentswithAdaptiveDimen‑ sionality,” arXiv preprint,22May2016.

[21] M.Zeng,“PI(t)D(t)ControlandMotionPro il‑ ingforOmnidirectionalMobileRobots,” arXiv preprint,19Oct.2021.

[22] P.Hart,N.Nilsson,andB.Raphael,“AFormal BasisfortheHeuristicDeterminationofMini‑ mumCostPaths,” IEEETransactionsonSystems ScienceandCybernetics,vol.4,no.2,pp.100–107,1968.doi:10.1109/TSSC.1968.300136.

[23] J.Petereitetal.,“ApplicationofHybridA*to anAutonomousMobileRobotforPathPlanning inUnstructuredOutdoorEnvironments,” Proc. 7th GermanConferenceonRobotics (ROBOTIK 2012),2012.

SEMANTIC‐AWARETRAJECTORYPLANNINGFORUAVINDYNAMICENVIRONMENTS

Submitted:07th July2025;accepted:1st October2025

VanHungNguyen,TheTienNguyen,TranThangLe,VietHongLe

DOI:10.14313/jamris‐2026‐011

Abstract:

Generatingtrajectoriesthatleveragesemanticinforma‐tiontoguideaUAVsafelyandaccuratelytoitsdestina‐tioninadynamicenvironmentremainsanopenproblem. Intheexistingliterature,semanticshavebeenusedto prioritizecertainareas–eithertoguidetheUAVthrough ortoavoidthem–forspecificobjectives,suchasreducing errorsinvisual‐inertialSLAM(VI‐SLAM).However,prior worktypicallyassumesastaticenvironmentwhenper‐formingcollisionchecking,eveninclutteredanddynamic settings.

Weproposeatwo‐stageworkflow:Thefirststage performssemantic‐awarepathfinding.Thesecondstage optimizestheresultingpath,incorporatingkinematic constraintsandperformingcollisioncheckingthat accountsforobstaclemotion,whilestillensuring real‐timeperformance.

Tothebestofourknowledge,thisisthefirstapproach thatgeneratesUAVtrajectoriesbysimultaneouslylever‐agingsemanticinformationandaccountingforcluttered, dynamicenvironments.Asummaryvideoisavailableat https://youtu.be/I5w6AP7HThU.

Keywords: UnmannedAerialVehicles,TrajectoryPlan‐ning,DynamicObstacleAvoidance,Semantic‐Aware, Dynamicenvironment

1.Introduction

Ahigherlevelofautonomyinunmannedaerial vehicles(UAVs)expandstheirpotentialfordeploy‑ mentacrossvariousreal‑worldapplications.This autonomyreliesheavilyonsimultaneouslocalization andmapping(SLAM)andthecapabilitytogenerate safeandprecisetrajectoriestowardatarget.

VI‑SLAMsystemsarewidelyusedinUAVsdue totheirhighaccuracy,real‑timeperformance,and autonomy,especiallyinGPS‑deniedenvironments suchasindoorspacesorobstructedareas[1–5].Addi‑ tionally,forquadrotorswithlimitedpayloadcapacity andbatterylife,camerasserveasidealonboardsen‑ sorsfornavigation.However,akeydrawbackofVI‑ SLAMisitsrapiddeclineinaccuracywhenencoun‑ teringtexture‑lessregions.Acommonapproachto improvingaccuracyinvolveskeepingspeci icfeatures orlandmarkswithinthe ieldofview(FOV)[6–8].

Nowadays,theadvancementsinarti icialintel‑ ligence(AI),particularlydeeplearningappliedto semanticsegmentation[9]andobjectdetection[10], haveachievedhighaccuracyandperformance.These techniquesenabletolabeltheregionswithdifferent

Figure1. Workflowofsemantic‐awaretrajectory planningincludingtwoconsecutivesteps.Theyare semantic‐awaresearchandoptimization,describedin section 4 and 5,respectively characteristicssemantically.Semanticinformationis oftenincorporatedasatermorconstraintinoptimiza‑ tionframeworkstohelpavoidtexturelessorproblem‑ aticregions,suchaslakesandoceans,whichcancause signi icantdriftorfailuresinposeestimation[11,12]. Additionally,ithelpsprioritizehigh‑texturedregions, therebyimprovingthequalityofposeestimation[13–19].Moreover,semanticshavealsobeenemployedin themulti‑robotplanningproblem[20].

Ensuringsafearrivalatthedestinationalso requireseffectiveobstacleavoidance.However,many existingstudiesassumetheenvironmentisstaticdur‑ ingcollisionchecking.Thislimitstheirdeploymentin real‑worldscenarios.Becausetherealworldisexactly theclutteredanddynamicenvironment.

Commoncollision‑checkingmethodsinvolve decomposingfreespaceintoconvexregionssuchas sequencesofaxis‑alignedcubes[21],convexregions fromseeding[22,23],orcreatingsafe lightcorridors (SFC)byin latingpre‑existingtrajectories(oftenthe globaltrajectory)[24–28].Collision‑checkingcanbe performedusingeitherdiscretizingthetrajectoryinto pointsorouterpolyhedralrepresentations.While discretizedpointsarecomputationallyintensive anddonotguaranteecollision‑freepathsbetween sampledpoints,increasingthenumberofsamplesto improveaccuracyfurtheraddstothecomputational burden[29–32].

Toreducetheburdenofcomputation,outerrep‑ resentationtechniquesenclosethetrajectorywithina polyhedron.Ifthispolyhedronremainsinsidethefree space,theentiretrajectoryisconsideredcollision‑ free.Forexample,inpolynomialtrajectoryoptimiza‑ tion[33, 34],itisveri iedwhethertheouterpoly‑ hedralrepresentationofeachtrajectorysegmentis containedwithinthefreespace.

Acommonapproachtoobtainingthispolyhedral representationisbyusingtheconvexhullofthecon‑ trolpointsfromtheBernsteinorB‑Splinebasis[35–37].However,inclutteredanddynamicenvironments, thefreespaceissigni icantlyreduced.Amorecom‑ pactouterrepresentationimprovesthelikelihoodof successfultrajectorygenerationwhilereducingcom‑ putationaltime.

Gettingtheideafromthepriorpublications[38–40],thisworkusestheMINVObasis[41]insteadofthe BernsteinorB‑Splinebasis.Dependingonthepoly‑ nomialdegree��,MINVO[41]canyieldasigni icantly smallervolume.

Decompositionisespeciallychallenginginclut‑ teredanddynamicenvironments.Indenseenviron‑ ments,itisdif iculttoconstructatightrepresen‑ tationoffreespace.Indynamicsettings,anaddi‑ tionaldimensionoftimemakesthedecomposition muchmorecomplicated,andsometimesitisinfea‑ sible.Toeliminatetheneedfordecomposition,this workimposesaconstraintthatveri iestheexistence ofaseparatingplanebetweentheUAV’strajectoryand obstacletrajectories.Thisplaneconstraintisincorpo‑ ratedintotheoptimizationprocess[38,40].

Thisstudypresentsanovelwork lowthatguides UAVstoprioritizehigh‑textureareaswhileavoiding texture‑lessandhazardousregionsindynamicenvi‑ ronments.Theproposedapproachconsistsoftwo mainstages:(i)Semantic‑AwareTrajectoryInitial‑ ization,called semantic‑awareA*search,prioritizes safeandhigh‑textureareaswhileavoidingtexture‑ lessandhazardousregions.Theoutputofthisstep servesastheinitialguessforthesecondstage.(ii) Dynamic‑AwareOptimizationaccountsforenviron‑ mentaldynamicsbycombining:(a)Eliminatingfree spacedecompositionandreplacingitwithaseparat‑ ingplaneconstraint.(b)UtilizingtheMINVObasis.At thesametime,thetrajectoryisalsoenergy‑optimal andsatis iesdynamicconstraints.

2.Problemdeclarationandsolvingapproach

TheUAVismodeledbythegeometricshapeand stateat ��.Itsshapeisasetofverticesin3Dspace �� =[V0, V1,…]⊂ℝ3.Andthestatevector s��(��)= [x�� , x�� , x��]=[x�� , v�� , a��],where x, v and a arethe position,velocityandacceleration,respectively.

TheenvironmentinwhichtheUAVoperatesis aclutteredanddynamicenvironment.Itismod‑ eledbyametric‑semanticmap ℳ.Itconsistsof unknownregions ℳ�� andknownregions ℳ��.Theseknownregionscontainstaticobsta‑ cles ��,dynamicobstacles �� andtheregions whichsemantically‑labelledastexture‑high ℳ�� or hazardous/texture‑less ℳ��.So ℳ=ℳ�� ∪ �� ∪�� ∪ℳ�� ∪ℳ��

Atthistime,theproblemishowtogenerateafeasi‑ bletrajectorythatguidestheUAVfromtheinitialstate s0 tothegoalstate s�� safelyandaccuratelywithinthe environment ℳ,whileensuringtheminimizationof controlenergybyleveragingthesemanticinformation availableinthatsemanticmap.

Tosolvethisproblem,severalkeysubproblems needtobeaddressedasfollows: subproblem1:De iningthetrajectory.

subproblem2:HOWTOcheckcollisionincluttered anddynamicenvironmentevenduringtrajectory generation.Detailsarepresentedinsection3below.

subproblem3:HOWTOleveragesemanticinforma‑ tiontoimprovetheprocessoftrajectorygeneration foracertainpurpose,moredetailedinsection4. subproblem4:Formulatingandsolvingthepro‑ grammingoptimization.Itisdescribedinmore detailinsection5

Weusethemethodofpolynomialtrajectoryplan‑ ning[28, 33]with clampeduniform B‑Splines.So, theUAV’strajectory x(��)∶=[x(��), y(��), z(��)]�� = ∑�� =0 ��,��(��)q�� isde inedby ��+1 controlpoints {q0,…, q��} and ��+1 knots {��0,��1,...,��}.Itseach segmentisa ��‑degreeB‑splinefunctionandindexed by��(��∈��,��isthetotalnumberofintervals)starting from0(AsdescribedinFig.2,��=0,…,��−2��−1). Intotal,ithas ��−2��−1 intervals.Itis clamped to ensurethatitpassesthrough s0 (the irst��+1knots areidentical)and s�� (thelast��+1knotsareidentical). Theknotsbetweenthe irst��+1andthelast��+1 knotsarecalledinternalknots.The uniform means thattheinternalknotsareequallyspaced.

Inthispaper,weusethecubicsplines(i.e.,��=3). ThisbalancesthedynamicfeasibilityofaUAVand computationalef iciency[40].So,theinputcontrol u(��)isjerk j(��)anditisconstantatthesameinterval j(��)=��

Toensurethereal‑timeperformanceandfeasibil‑ ity,theconsumingtimeoftrajectorygenerationneeds tobelimitedwithinatimeintervalof ��(��).Thisis achievedbygeneratingonlytheportionofthetra‑ jectory(illustratedbyFig. 2,itisthegolden‑brown segment)thatlieswithinasphere �� withtheradius ��.Duringre‑planning,��remains ixed.Thetrajectory generationstartsatatimewhenUAVisstayingat s�� andatthemomentbeforethetimeofcompletingthe executionofthepreviousportionofthetrajectoryby ��(��)(illustratedinFig.2).

Figure2. TheproblemofUAVtrajectoryplanningina clutteredanddynamicenvironment.Thetrajectoryis generatedportionbyportiontoensurereal‐time performanceandfeasibility

Thus,atthispoint,thestartingpointisthemoment whentheexecutionofthepreviousportionofthetra‑ jectoryiscompleted�� ,correspondingtostate s�� Thegoalofthetrajectoryisnolonger s�� butinstead atemporarytarget s�� .Thistemporarytargetis obtainedbytheintersectionbetweensphere �� and apiece‑wiselinearpaththatgoesfrom (s�� ,�� ) thatavoidsthestaticobstacles.The inaltargetofthe resultingportionis s�� attime�� ,whichdoesnot necessarilycoincidewith s�� .Next,wedelveinto thedetailsofsolvingtheremainingsubproblems.

3.Collision‐checkingindynamicenvironment

Asdescribedinsection1,toalleviatetheburdenof computing,Allofthetrajectoriesarerepresentedby outerpolyhedrals.Andthencheckingwhetherornot theintersectionofthemforcollision.

3.1.UAVrepresentationandtheboundingpolyhedron

ofitstrajectory

Theboundingpolyhedron(orouterrepresenta‑ tion)ofUAV’strajectory,whichisde inedinsection 2,canbegeneratedfromitscontrolpoints.Theyare indexedusingthesymbol ��.Thenumberofcontrol pointsforeachsegmentisonemorethanthedegreeof theB‑spline,��+1.However,weuseMINVObecause, asdemonstratedin[41],itprovidesamuchtighter representation.Indetail,withdegree ��=3 shows that,thevolumereduces2.36and254.9timessmaller thantheonesobtainedbytheBernsteinandB‑Spline bases,respectively.Whenn=7,theseratiosincrease to902.7and2.997.1021,respectively.

Thus,fromeachintervaloftheB‑spline,thecontrol B‑splinepoints �� arecomputed,formingtheset Q�� .Fromthere,theMINVOcontrolpointsaredeter‑ minedaccordingto[42],resultingintheMINVOcon‑ trolpoints�� =�� (�� )andtheircorresponding sets Q�� .Asstatedinsection 2,weusetheB‑spline basistobecubic(degree��=3).Thus,eachinterval �� isguaranteedtoliewithintheconvexhullofits4 controlpoints{q��, q��+1, q��+2, q��+3}∈�� . 3.2.Representationofobstacleandtheboundingpoly‐hedronofitstrajectory

Intheenvironment,thereare��obstaclesincluding staticanddynamicones.The i‑th obstacleissymbol‑ ledby �� (��∈��).Anobstacleischaracterizedbyits trajectory ��(��)∶=[��(��),��(��),��(��)]�� anddimen‑ sion �� ={v�� 1, v�� 2,…, v�� }⊂ℝ3.Obviously,for staticobstacles, ��(��)=��.Itisin latedbythe sizeofUAV(thatisMinkowskisum,themathemati‑ calnotationis⊕)andtheninferringtheconvexhull (mathematicalnotionis��(.))ofin latedobstacle, ��(�� ⊕��),asdescribedinFig.4a

Fordynamicobstacles,itstrajectoryispredicted segmentbysegmentwiththepredictionerror �� (illustratedbyFig.4b).Eachsegmentofi‑thobstacle’s trajectory, ��(��),iscorrespondingtoaj‑thinterval timeΔ�� oftheUAV’strajectory.Inthiscase,theconvex hulliscalculatedasfollows:First,itisin latedby theUAV’ssize,similartoastaticobstacle.Next,itis expandedwiththepredictionerror��.Then,itisslid alongitspredictedtrajectorywithasamplingtimeof ��.Theentireoccupiedspaceoftheobstacle, �� = �� ⊕��⊕2��⊕2��,isnowtheunionofalloccupied regionsateachtimestep��.Finally,theconvexhull isgeneratedforthisoccupiedspace.Thesetofall verticesofthisconvexhull,�� =��(��).Allthese stepsareillustratedinFig.4b

Theproblemnowishowtopredicttheobstacle’s motiontrajectory.Inthescopeofthispaper,thetra‑ jectorypredictionfunctionisassumedtobeprecom‑ puted.Someparticularobstacle’strajectoriesareused forsimulation,withdetailsprovidedinsection6.

3.3.CollisionChecking

Insubsection3.2,theconvexhulloftheobstacles hasalreadyaccountedfortheUAV’ssize,sotheUAV isnowtreatedasapointofmass.Thismeansthat theUAVandtheobstacledonotcollideiftheMINVO convexhull�� MV �� (computedinsubsection 3.1)ofthe UAV’strajectorydoesnotintersectwiththeconvex hulloftheobstacle�� (computedinsubsection3.2).

where:

n�� isthenormalvectorde iningtheseparating hyperplane.

�� isabiastermshiftingthehyperplane.

�� istheconvexregionrepresentingtheobstacle.

�� MV �� istheconvexregionrepresentingtheUAVtra‑ jectory(usingtheMINVObasis).

��isthetotalnumberofobstacles.

��isthesetofUAVtrajectoryintervals.

Inotherwords,theydonotcollideifthereexistsa separatinghyperplane�� (characterizedbythenor‑ malvector n�� andbias ��)betweentheirconvex hulls,describedbyEq.1.The irstinequalityofEq.1 ensuresthatallpoints c intheobstacleset�� lieon onesideoftheplane.Thesecondoneensuresthatall points q intheUAV’strajectoryset��MV �� lieontheother side.

Anillustrationofanouterpolyhedralrepresen‑ tationandcollision‑checkingthatincludesstaticand dynamicobstacles,aswellasUAVisshowninFig.3 Fig.3aillustratesalloftheconvexhullsandcollision checkingatthe irstandsecondintervals.Fig.3band 3c illustratetheconvexhullandcollisionchecking atthethirdandfourthintervals,respectively.This problemissolvedusingGLPK[43]orGurobi[44].

4.Semantic‐awareA*search

ThisalgorithmisinspiredbytheoriginalA*[45] forpath‑ indingbasedonnotonlytraditionalcostsbut alsothesemanticinformationabouttheenvironment. Inparticular,itsresultingpathistoprioritizetherich‑ informativeregionswhileavoidinghazardousorlow‑ informativeareas.Soitiscalledassemantic‑awareA*. Thisisachievedbyintroducingadditionalcostvalues intothetotalcostfunction ��.EachMINVOcontrol pointservesasanodeinthesearch.Allopennodes aremaintainedinapriorityqueue ��,where elementsareorderedinascendingorderof��

Withsemantic‑labeledmap ℳ,theregionswith highinformationtendtoattracttheUAV,modeled by ��att.Whereasthehazardousorlow‑informative regionstendtorepeltheUAV,modeledby ��rep (describedinFig.5).

Figure3. OuterrepresentationoftrajectoriesofUAV,dynamicobstacles(forinstance,obstacle 0 and 1),staticobstacle (forexample,obstacle ��=2)andCollision‐checkingbyseparatingplanes ��:(a)Collision‐checkingatthe 1�� and 2�� intervalofUAVtrajectory.(b)Collision‐checkingatthe 3�� intervalofUAVtrajectory.(c)Collision‐checkingatthe 4��ℎ intervalofUAVtrajectory

Figure5. Thetotalcostofsemantic‐awareA*includes traditionalcosts,alongwithrepulsiveandattractive costsarisingfromtexture‐less/hazardousregions(vivid pinkish‐magentacolor)andhigh‐textureareas(fresh greencolor),respectively.AlongwithHOWTOcalculate theEuclideandistancein3Dspaceusedasprimitivefor calculatingthecosts

i-thobstacle trajectory

Obstacle inflatedbyUAV Inflatedobstaclewith errorprediction Convexhullofsegment ofi-thdynamicobstacle

(b)

Figure4. Outerrepresentationoftheobstacles:(a)For staticobstacle.(b)Fordynamicobstacle

So,atthistime,thetotalcostfunction��(.)includes fourterms,where ��(.) isthesumofthedistances (betweensuccessivecontrolpoints)from q0 tothe currentnode q�� (cost‑to‑come), ℎ(.) isthedistance fromthecurrentnode q�� tothegoal s�� (heuristicsof thecost‑to‑go),��rep(.)isforrepellingUAVawayfrom thelow‑textureregions,whileconversely,��att(.)isfor attractingtowardsinformativeareas,asmodeledin Eq.2.

Thepositionofeachvoxelinthemapℳ is v�� = (�� ,�� ,�� ),so v�� ∈ℳ.Let ℳ��,ℳ�� ⊆ℳ bethesetofallvoxelsintheattractiveandrepul‑ siveregions,respectively.Thecostvalueofthesetwo regions(ℳ�� and ℳ��)iscalculatedasthesumof thedistancesfromthecurrentcontrolpoint q�� toall thevoxelsinthatregion.

However,theirin luenceonthetotalcost (accordingtoEq. 2)isopposite.Byincreasing thetotalcost, ��att prioritizesregionswithsmaller distances.Incontrast, ��rep decreasesthetotalcost, therebyprioritizingregionswithlargerdistances, meaningittendstopushthetrajectoryawayfrom thoseregions.TheyarecalculatedbyEq.3,wherethe sign“_”indicates��or�� (.)∶= 1 |ℳ | v��∈ℳ ��(q��, v��) (3)

Theweights ��,��ℎ,��att,��rep ∈ℝ+ inEq. 2 deter‑ minethein luenceofeachcomponentonthetotalcost ��(⋅).Ifaweightisgreaterthan1,thecorresponding componenthasastrongerin luence(forexample,if ��att >1,thetrajectoryismorestronglyguidedtoward theattractiveregion).Ontheotherhand,whenthe weightislessthan1(0<��<1),thecorresponding componenthasaweakerin luencecomparedtothe others.Iftheweightequals1,ithasabalancedin lu‑ enceaccordingtoitsoriginalvalue.

inflated byUAV

Thealgorithmisrepresentedasapseudo‑codein Alg. 1.Firstly,controlpoints q0, q1, q2 isdetermined fromtheinitialstate s�� atthemoment�� (line1).

Atthestartingmoment ��0,wehave �� =��0 and s�� = s0.Andthenitinitializesthe�� queue, ��and��(line2to5).

Algorithm1 Semantic‑AwareA*

1: (q0, q1, q2)←CALCONTROLPOINT(s�� )

2: ��← q2

3: ��,��←∞

4: ��[q0]←0

5: Calculating��[q0]byEq.2and3

6: while (�� isnot empty) or timeout do

7: q�� ←Firstitemof��

8: if ‖q�� s�� ‖2 <�� and ��=��−2 then

9: {q��}��−2 ��=0 ←GETSCPS(q��)

10: q��−1 ← q��−2

11: q�� ← q��−1

12: return [{q��}�� =0,��]

13: endif

14: ��.REMOVE(q��)

15: [��,��]←CHECKCOLLISION(q��)

16: if ‖q�� s�� ‖2 >�� or ‖q�� q��‖∞ ≤�� or �� then

17: continue

18: endif

19: for Δ��∈UNIFORMSAMPLING(v��, a��) do

20: q��+1 ← q�� + Δ��.Δv ��

21: �� ←��[q��]+DIST(q��, q��+1)

22: if �� <��[q��+1] then

23: S��[q��+1]← q��

24: ��[q��+1]←��

25: Calculating��[q��+1]byEq.2and3

26: if ‖q��+1 q��‖∞ >�� then

27: ��← q��+1

28: endif

29: endif

30: endfor

31: endwhile

32: return [GETBESTSCPS(S��),��]

Theloopofsemantic‑awareA*searchisrununtil ��queueisemptyoroutoftime:The irstitem, whichiswith��valueislowest,ispopped(line7)and removeit(line14)ifitsimultaneouslydoesnottouch thetarget(line8)andsatis iessomeconditions(line 16).Thesearchprocessisconsideredcompleteif q�� is withinaprede ineddistance��fromtheintermediate goal s�� andit’sindex ��=��−2 (line 8).Since thevelocity v andacceleration a oftheUAVarezero whenitreachesthe inaltarget s��,itfollowsthat q��−2 = q��−1 = q��.Theresultwillbeasequence ofcontrolpoints {q��}�� =0,whichcomefrominferring thesequencefrom q0 to q��−2 (line9)backwardand appendingtwoendingpoints(line 10 and 11),and hyperplanes�� thatseparatethemfromobstacles. For q�� nodetobeaccepted,itmustsatisfyseveral conditions(line 16):itmustbeinsidethesphere �� (‖q�� s�� ‖2 ≤��)asstatedinsection2,notbetoo

closetoanother q�� alreadyinthe �� (‖q�� q��‖∞ >��)toalleviatetheburdenofcomputing,and obviouslynotcollide(��isfalse).Thecollisionis checkedbydeterminingwhich�� MV �� contains q��,then solvingEq.1(detailsinsubsection3.3).

Theresultisahyperplaneaddedto �� anda binaryvariable �� indicatingwhetherornota collisionoccurs(line15).

Next,thebestneighborof q�� isexpandedand addedto��queue:ForeachvalueofΔ��,which isuniform‑sampledensuringthelimits vmax and amax Theneighboriscomputed(line20)usingatimestep ofΔ��/��,whereΔ��= ‖s�� s�� ‖2 vmax .Thebestneighbor isselected,andifitisnotalreadyinthe��,itis added(lines 22 to 27),whileits �� valueiseval‑ uated(line 25).Moreover,ifthesearchtimeexceeds theprede inedlimit(thatis ��),itwillreturn thesequenceofcontrolpoints,whichiswiththelast pointbeingtheclosesttotheintermediategoal s�� , andit’scorrespondinghyperplanes�� (line32).This algorithmistestedinsubsection6.1below.

5.Optimization

Basedontheinitialtrajectory(convertedfrom [{q

�� =0,��0 ��])gettingfromsemantic‑awareA*.We needtosmoothandmakeitfeasiblebyprogram‑ mingoptimizationwiththegoalofminimizingenergy consumptionandreachingthetargetascloselyas possible.Thisproblemisparameterizedbycontrol points QBS �� andplanesvariables ��(n��,��).Itmin‑ imizestheenergyconsumptionthroughcontrolinput ∫

Andthetargetisreachedascloselyaspossiblein termsofdistance‖q�� s�� ‖2.Becauseof q��−2 = q��−1 = q��,itshouldbe ‖q��−2 s�� ‖2.Thisis modelledbyEq.4.

subjectsto:

)= s�� , (ii) v(�� )= 0, a(�� )= 0, (iii) n�� c +�� >0,∀c ∈�� ,∀��,��,

Thisproblemsubjectstosomeconstraints:

(i) Thestartingpointofthisinvervalistheend‑ pointofpreviousone.Thatis, s(�� )= s�� asdescribledindetailinsection 2.Obviously s��(0)= s0.

(ii) When s�� isinsidethesphere ��,itmeansthat thelastsegmentoftrajectoryisoptimized(that

is, ��=��−2��−1 or s(�� )= s��).The UAV’svelocity v(�� )= 0 andacceleration a(�� )= 0.

(iii) Ensuringthesafety(collision‑avoiding) inadynamicenvironment,detailedin subsection3.3.

(iv) Guaranteeingthatthegeneratedsegmentoftra‑ jectoryremainsinsidesphere��,detailedinsec‑ tion2

(v) Thetrajectorymustcomplywithkinematiccon‑ straints.Speci ically,velocityandacceleration mustnotexceedUAV’sphysicallimits v�� and a��,respectively.Becausethetrajectory isrepresentedbyB‑Splines,whichisacontinu‑ ousfunctionoftime.Sodirectlyimposingthese constraints(v��, a��)ateverysinglepoint intimealongthiscontinuoustrajectorywould leadtoanin initenumberofconstraints,mak‑ ingtheoptimizationproblemcomputationally intractable.

Moreover,thecontrolpoints q parameterizethe entiretrajectorysegment.Therefore,byplacingcon‑ straintsonthesecontrolpoints(velocity v andaccel‑ eration a),wecanindirectlyin luenceandbound thephysicalvelocityandaccelerationthroughoutthe interval.Theboundofthevelocity vmax andacceler‑ ation amax ofcontrolpointsareinferredthephysical ones,respectively.Thisproblem(Eq. 4)issolvedby Gurobi[44].

6.Experiments

Intheexperiments,weusethecon igurationofthe systemasfollowing:

Hardware:16coresIntelCorei7‑10875H@ 2.30GHz;GPUNvidiaTU117GLM[QuadroT1000 Mobile];RAMmemoryof32GB.

Software:Ubuntu20.04LTS;ROSnoetic[46]serves asthemiddlewareframeworkforcommunica‑ tionbetweentheplanner,simulator,andvisualiza‑ tion/loggingtools.Itprovidesstandardizedmes‑ sagepassingandmodularintegrationofdifferent softwarecomponents.

Forallplanningandcollisionchecking,theUAVis modeledasasphereofradius�� =0.1(��)which upper‑boundsthevehiclebodyandrotorsweep. Whilerotordisksareomittedinthe iguresforvisual clarity,thisin latedmodelguaranteessafeclearance inallexperiments.Toguaranteereal‑timeperfor‑ manceandfeasibility,theplanninghorizonislimited toasphereofradius��=4.0m,asdetailedinSection2 Moreover,tofocusonthetrajectoryplanningalgo‑ rithms,itisassumedthattheUAVcanperfectlytrack thetrajectoriesgeneratedbytheplanner.

6.1.Testingsemantic‐awareA*solely

Thissectionevaluatesthesemantic‑awareA*algo‑ rithm(presentedinsection 4)intwoaspects:the in luenceofsemanticinformationandtheavoidance ofdynamicobstacleswiththefollowingcon iguration: ��g =1.0, ��h =1.0,runtime=0.1second,degreeof

B‑spline=3(cubicb‑spline),numberofsegments=6 andonetrefoil‑knot‑baseddynamicobstacle. Thetrajectoryofthedynamicobstacle, ��(��),is modeledasatrefoilknot[47,48].Althoughthetrefoil knotdoesnotre lectrealisticobstaclemotion,itpro‑ videsseveraladvantages:

(i) Challengingyetstructured–its3D,non‑trivial trajectoryismoredemandingthanlinearorcir‑ cularpaths,makingitastrongtestforcollision avoidance.

(ii) Repeatable–itsmathematicalde inition ensuresidentical,reproducibleruns.

(iii) Controlledcomplexity–thetrajectoryiswell understood,enablingsystematicevaluationof algorithmperformance.

(iv) Visuallydistinct–itsclearshapefacilitates observationandvalidationinsimulations.

The irstcaseconsidersnoin luenceofsemantic informationwith ��att =0,��rep =0.Inthiscase,the searchprocesstendstowardthegoalwhileavoiding obstacles(Fig. 6a).Inthesecondcase,testingthe algorithminanenvironmentcontainingatexture‑high region(thegreenareainFig. 6c and 6d)with ��att = 2.0,��rep =0.TheexperimentalresultsinFig. 6c showthatthesearchprocesstendstoshifttoward thetexture‑highregioncomparedtothe irstcase.The thirdcaseexaminesthealgorithminanenvironment withatexture‑lessorunsaferegion(theredareain Fig. 6e and 6f)with ��att =0,��rep =0.5.Theresults indicatethatthesearchprocesstendstoavoidthis region.

Inallofthesecases,theresultsin(Fig. 6b, 6d and 6f)showthatdynamicobstacleavoidanceis fullyensuredateachsegment(from 1�� to 6��ℎ one). Becausetheobstaclesarestillconsideredtobein motion(notemporarily‑static)duringthecollision‑ checkingprocess.Theexperimentalresults(Figure 6)demonstratethattheproposedsemantic‑awareA* algorithmbehavesasexpected,showingatendencyto movetowardregionsrichininformationwhileavoid‑ inginformation‑poorareas.

6.2.Experimentwithclutteredanddynamicenviron‐ment

Inthissection,weevaluatethesystem’scapability tosimultaneouslyleveragethesemanticinformation –bothtoavoidand/ortoprioritizetraversingcertain regions–whileoperatinginadynamicenvironment, bycomparingitwiththeMADERsystem[38],which doesnotutilizesemanticinformation.Theevaluation isconductedinadynamicenvironmentmeasuring70 x4.0x4.0meters(Fig. 7a and 7d),where65%of theobstaclesaredynamic–representedbyredcubes, eachsized0.8x0.8x0.8(m)–whiletheremaining 35%arestaticobstacles,depictedasbluerectangular boxes,eachmeasuring0.4x0.4x8.0(m).

Figure6. Theexperimentalresultsofsemantic‐awareA*withonedynamicobstacleinthreecases(Withoutsemantics, ConsideringtheinfluenceoftheattractiveregionandConsideringtheinfluenceoftherepulsiveregion):(a),(c),and(e) showthecompletetrajectories,while(b),(d),and(f)displaythecorrespondingindividualtrajectorysegments(1–6)for eachcase

Twosimulationscenariosareconducted.The irst featuresadynamicenvironmentcontainingahigh‑ textureregion,visuallyindicatedbyagreenrectan‑ gle(Fig. 7a, 7b and 7c).Thesecondincludesapoor‑ textureorhazardousregion,representedbyared rectangle.BothscenariossharethesameUAVdynamic constraints,withamaximumvelocityof vmax = [6.06.06.0]��/�� andamaximumaccelerationof amax =[202010]��/��2,correspondingtothe velocityandaccelerationlimits,respectively.Addi‑ tionally,inthe irstscenario,thegoalpositionisset to (75.0,−10.0,1.0)��,whereasinthesecond scenario,itissetto(75.0,−1.0,1.0)��.Theruntime oftheMILPphaseisboundedbetween0.05and0.35 second.

Inthe irstscenario,simulationsareconducted byMADER[38]andoursuggession(�� =2.0), followedbyacomparisonoftheresultingtrajectories.

ThesimulationresultsshowthattheUAViscapa‑ bleofnavigatingsafelyinadynamicenvironment (Fig. 7b),whilealsoexhibitingatendencytopass throughthehigh‑textureregion(thedarkbluetrajec‑ toryinFig.7c).

Similarly,inthesecondscenario,consideringthe low‑textureorhazardousareacorrespondsto�� = 2.0.TheresultingtrajectoryshowsthattheUAVtends toavoidtheseregions(thedarkbluetrajectoryin Fig. 7f),whilestillsuccessfullyreachingthegoal withinthedynamicenvironment(Fig.7e).

7.Conclusion

Thisworkintroducesacompletework lowfor semantic‑awaretrajectoryplanningthatenablesUAVs tonavigateautonomouslyincluttered,dynamicenvi‑ ronmentswhilesimultaneouslyexploitingsemantic informationtoprioritizeoravoidspeci icregions

(a) (b) (c)

(e)

(f)

Figure7. Experimentsindynamiccorridorenvironment:(a)Theenvironmentincludesdynamicandstaticobstacles(the redcubeandthebluerectangularbox,respectively),aswellasarich‐informationregion(thegreenrectangularbox);(b) Thequadrotorfliesinadynamicenvironment,withouterpolyhedra(redboxes)surroundingobstacles;(c)Thedarkblue anddarkredtrajectoriescomefromoursuggestionandMADER[38],respectively;(d‐f)Thecorrespondingsetup,flight, andtrajectoryresultsforascenarioincludingalow‐textureorhazardousregion(redrectangularbox) dependingontaskobjectives.Comparedtonon‑ semanticplanners(suchasMADER),whichdonot leveragesemantics,ourapproachachievesmoreef i‑ cientandcontext‑awarenavigation,withtrajectories thattendtoavoidhazardouszonesandapproach information‑richregions.

Theproposedapproachcombinesasemantic‑ awareA*initialization,whichbiasestrajectories towardsafeandinformativeregions,withadynamic‑ awareoptimizationthatre inesthepathusingsepa‑ ratingplaneconstraintsandtheMINVObasiswhile ensuringenergyef iciencyanddynamicfeasibility.

Nevertheless,severallimitationsremain.Theopti‑ mizationcurrentlyconsidersonlytheUAV’sposi‑ tionandnotitsorientation,whichmayconstrain applicabilityintasksrequiringviewpointcontrol. Oursimulationsalsodonotyetreportquantitative state‑estimationerrorsundersemanticversusnon‑ semanticsettings,anddynamicobstaclesareassumed tofollowknowntrajectoriesratherthanstochastic, uncertainmotions.

Concretedirectionsforfutureworkinclude extendingtheformulationtoexplicitlyhandle UAVorientationandperception‑awareobjectives, incorporatingonlineestimationofdynamicobstacle motionwithuncertainty,andvalidatingtheapproach inhardwareexperimentswithrealUAVs.Another promisingdirectionistoadaptorlearnthesemantic weights(��att,��rep)onlineusingreinforcementor imitationlearning,therebytailoringbehaviorto speci icmissionswhilemaintainingrobustness.

Overall,thisworkrepresentsa irststeptoward bridginghigh‑levelsemanticunderstandingwithlow‑ leveldynamicfeasibility,pavingthewayforsaferand moreintelligentUAVautonomyincomplexreal‑world environments.

AUTHORS

VanHungNguyen∗ –Control,automationinpro‑ ductionandimprovementoftechnologyinstitute (CAPITI),AcademyofMilitaryScienceandTechnol‑ ogy89LyNamDeStreet,Hanoicity,Vietnam,e‑mail: nvhung.v2k@gmail.com.

TheTienNguyen –LeQuyDonTechnicalUniversity 236HoangQuocViet,Hanoicity,Vietnam,e‑mail: tiennt.isi@lqdtu.edu.vn.

TranThangLe –Control,automationinproduction andimprovementoftechnologyinstitute(CAPITI), AcademyofMilitaryScienceandTechnology89Ly NamDeStreet,Hanoicity,Vietnam,e‑mail:ltran‑ thang@gmail.com.

VietHongLe –Control,automationinproduction andimprovementoftechnologyinstitute(CAPITI), AcademyofMilitaryScienceandTechnology89 LyNamDeStreet,Hanoicity,Vietnam,e‑mail: lehong0174@gmail.com.

∗Correspondingauthor

References

[1] R.Mur‑Artal,J.M.M.Montiel,andJ.D.Tardos, “ORB‑SLAM:aversatileandaccuratemonocular SLAMsystem”, IEEEtransactionsonrobotics,vol. 31,no.5,2015,1147–1163,Publisher:IEEE.

(a)

(c)

(d)

(f)

[2] R.Mur‑ArtalandJ.D.Tardós,“Orb‑slam2: Anopen‑sourceslamsystemformonocular, stereo,andrgb‑dcameras”, IEEEtransactionson robotics,vol.33,no.5,2017,1255–1262,Pub‑ lisher:IEEE.

[3] C.Campos,R.Elvira,J.J.G.Rodriguez,J.M. M.Montiel,andJ.D.Tardos,“ORB‑SLAM3:An AccurateOpen‑SourceLibraryforVisual,Visual–Inertial,andMultimapSLAM”, IEEETransactions onRobotics,vol.37,no.6,2021,1874–1890, 10.1109/tro.2021.3075644,Publisher:Institute ofElectricalandElectronicsEngineers(IEEE).

[4] D.ScaramuzzaandZ.Zhang,“Visual‑ inertialodometryofaerialrobots”, arXivpreprintarXiv:1906.03289,2019, https://doi.org/10.48550/arXiv.1906.03289.

[5] C.DebeunneandD.Vivet,“AReviewofVisual‑ LiDARFusionbasedSimultaneousLocalization andMapping”, Sensors,vol.20,no.7,2020, 10.3390/s20072068.

[6] I.Spasojevic,V.Murali,andS.Karaman, “Perception‑awaretimeoptimalpath parameterizationforquadrotors”.In: 2020 IEEEInternationalConferenceonRoboticsand Automation(ICRA),2020,3213–3219.

[7] V.Murali,I.Spasojevic,W.Guerra,and S.Karaman,“Perception‑awaretrajectory generationforaggressivequadrotor light usingdifferential latness”,2019,3936–3943, 10.23919/ACC.2019.8814697.

[8] D.Falanga,P.Foehn,P.Lu,andD.Scara‑ muzza,“PAMPC:Perception‑awaremodelpre‑ dictivecontrolforquadrotors”.In: 2018IEEE/RSJ InternationalConferenceonIntelligentRobots andSystems(IROS),2018,1–8.

[9] A.Garcia‑Garcia,S.Orts‑Escolano,S.Oprea, V.Villena‑Martinez,andJ.Garcia‑Rodriguez, “Areviewondeeplearningtechniques appliedtosemanticsegmentation”, arXivpreprintarXiv:1704.06857,2017, https://doi.org/10.48550/arXiv.1704.06857.

[10] JosephRedmon,S.Divvala,R.Girshick,and A.Farhadi,“YouOnlyLookOnce:Uni ied,Real‑ TimeObjectDetection”.In: 2016IEEEConfer‑ enceonComputerVisionandPatternRecognition (CVPR),2016,779–788,10.1109/CVPR.2016.91.

[11] L.Bartolomei,L.Teixeira,andM.Chli, “Perception‑awarePathPlanningforUAVs usingSemanticSegmentation”.In: 2020 IEEE/RSJInternationalConferenceonIntelligent RobotsandSystems(IROS),2020,5808–5815, 10.1109/IROS45743.2020.9341347.

[12] S.Achat,J.Marzat,andJ.Moras,“PathPlanning IncorporatingSemanticInformationfor AutonomousRobotNavigation”.In: 19th InternationalConferenceonInformaticsin Control,AutomationandRobotics(ICINCO) 2022,Lisbonne,Portugal,2022,285–295, 10.5220/0011134300003271.

[13] I.Spasojevic,V.Murali,andS.Karaman,“Joint featureselectionandtimeoptimalpath parametrizationforhighspeedvision‑aided navigation”.In: 2020IEEE/RSJInternational ConferenceonIntelligentRobotsandSystems (IROS),2020,5931–5938.

[14] K.Lee,J.Gibson,andE.A.Theodorou, “Aggressiveperception‑awarenavigation usingdeepoptical lowdynamicsandpixelmpc”, IEEERoboticsandAutomationLetters,vol.5,no. 2,2020,1207–1214,Publisher:IEEE.

[15] Z.ZhangandD.Scaramuzza,“Perception‑aware RecedingHorizonNavigationforMAVs”.In: 2018IEEEInternationalConferenceonRobotics andAutomation(ICRA),2018,2534–2541, 10.1109/ICRA.2018.8461133.

[16] G.Costante,C.Forster,J.Delmerico,P.Valigi,and D.Scaramuzza,“Perception‑awarepathplan‑ ning”, arXivpreprintarXiv:1605.04151,2016, https://doi.org/10.48550/arXiv.1605.04151.

[17] K.M.Frey,T.J.Steiner,andJ.P.How, “Towardsonlineobservability‑awaretrajectory optimizationforlandmark‑basedestimators”, arXivpreprintarXiv:1908.03790,2019, https://doi.org/10.48550/arXiv.1908.03790.

[18] P.Salaris,M.Cognetti,R.Spica,andP.R.Gior‑ dano,“Onlineoptimalperception‑awaretrajec‑ torygeneration”, IEEETransactionsonRobotics, vol.35,no.6,2019,1307–1322,Publisher:IEEE.

[19] B.Zhou,J.Pan,F.Gao,andS.Shen,“Raptor: Robustandperception‑awaretrajectoryreplan‑ ningforquadrotorfast light”, IEEETransactions onRobotics,vol.37,no.6,2021,1992–2009,Pub‑ lisher:IEEE.

[20] S.Kalluraya,G.J.Pappas,andY.Kantaros, “Multi‑RobotMissionPlanninginDynamic SemanticEnvironments”.In: 2023IEEE InternationalConferenceonRoboticsand Automation(ICRA),2023,1630–1637, 10.1109/ICRA48891.2023.10160344.

[21] J.Chen,T.Liu,andS.Shen,“Onlinegeneration ofcollision‑freetrajectoriesforquadrotor lightinunknownclutteredenvironments”.In: 2016IEEEInternationalConferenceonRobotics andAutomation(ICRA),2016,1476–1483, 10.1109/ICRA.2016.7487283.

[22] R.DeitsandR.Tedrake,“Computinglargeconvex regionsofobstacle‑freespacethroughsemidef‑ initeprogramming”.In: AlgorithmicFounda‑ tionsofRoboticsXI:SelectedContributionsofthe EleventhInternationalWorkshopontheAlgorith‑ micFoundationsofRobotics,2015,109–124.

[23] B.Landry,R.Deits,P.R.Florence,andR.Tedrake, “Aggressivequadrotor lightthroughcluttered environmentsusingmixedintegerprogram‑ ming”.In: 2016IEEEInternationalConference onRoboticsandAutomation(ICRA),2016,1469–1475,10.1109/ICRA.2016.7487282.

[24] S.Liu,M.Watterson,K.Mohta,K.Sun,S.Bhat‑ tacharya,C.J.Taylor,andV.Kumar,“Planning DynamicallyFeasibleTrajectoriesforQuadro‑ torsUsingSafeFlightCorridorsin3‑DCom‑ plexEnvironments”, IEEERoboticsandAutoma‑ tionLetters,vol.2,no.3,2017,1688–1695, 10.1109/LRA.2017.2663526.

[25] C.Liu,C.‑Y.Lin,andM.Tomizuka,“Theconvex feasiblesetalgorithmforrealtimeoptimization inmotionplanning”, SIAMJournalonControland optimization,vol.56,no.4,2018,2712–2733, Publisher:SIAM.

[26] F.Gao,W.Wu,W.Gao,andS.Shen,“Flyingon pointclouds:Onlinetrajectorygenerationand autonomousnavigationforquadrotorsinclut‑ teredenvironments”, JournalofFieldRobotics, vol.36,2018,10.1002/rob.21842.

[27] S.‑p.Lai,M.‑l.Lan,Y.‑x.Li,andB.M.Chen,“Safe navigationofquadrotorswithjerklimitedtra‑ jectory”, FrontiersofInformationTechnology& ElectronicEngineering,vol.20,no.1,2019,107–119,10.1631/FITEE.1800719.

[28] G.Rousseau,C.S.Maniu,S.Tebbani,M.Babel,and N.Martin,“Minimum‑timeB‑splinetrajectories withcorridorconstraints.Applicationto cinematographicquadrotor lightplans”, Control EngineeringPractice,vol.89,2019,190–203, https://doi.org/10.1016/j.conengprac.2019.05.022.

[29] M.Szmuk,C.A.Pascucci,andB.AÇikmeşe,“Real‑ timequad‑rotorpathplanningformobileobsta‑ cleavoidanceusingconvexoptimization”.In: 2018IEEE/RSJInternationalConferenceonIntel‑ ligentRobotsandSystems(IROS),2018,1–9.

[30] H.ZhuandJ.Alonso‑Mora,“Chance‑constrained collisionavoidanceformavsindynamicenviron‑ ments”, IEEERoboticsandAutomationLetters, vol.4,no.2,2019,776–783,Publisher:IEEE.

[31] N.D.Potdar,G.C.deCroon,andJ.Alonso‑ Mora,“Onlinetrajectoryplanningandcontrol ofaMAVpayloadsystemindynamicenviron‑ ments”, AutonomousRobots,vol.44,no.6,2020, 1065–1089,Publisher:Springer.

[32] D.Mellinger,A.Kushleyev,andV.Kumar,“Mixed‑ integerquadraticprogramtrajectorygeneration forheterogeneousquadrotorteams”.In: 2012 IEEEinternationalconferenceonroboticsand automation,2012,477–483.

[33] C.Richter,A.Bry,andN.Roy.“PolynomialTra‑ jectoryPlanningforAggressiveQuadrotorFlight inDenseIndoorEnvironments”.649–666.April 2016.

[34] J.Park,J.Kim,I.Jang,andH.J.Kim,“Ef icient multi‑agenttrajectoryplanningwithfeasibil‑ ityguaranteeusingrelativebernsteinpolyno‑ mial”.In: 2020IEEEInternationalConferenceon RoboticsandAutomation(ICRA),2020,434–440.

[36] J.Tordesillas,B.T.Lopez,M.Everett,andJ.P. How,“Faster:Fastandsafetrajectoryplanner fornavigationinunknownenvironments”, IEEE TransactionsonRobotics,vol.38,no.2,2021, 922–938,Publisher:IEEE.

[37] J.A.Preiss,K.Hausman,G.S.Sukhatme,and S.Weiss,“TrajectoryOptimizationforSelf‑ CalibrationandNavigation.”.In: Robotics: ScienceandSystems,vol.13,2017.

[38] J.TordesillasandJ.P.How,“MADER:Trajec‑ toryplannerinmultiagentanddynamicenviron‑ ments”, IEEETransactionsonRobotics,vol.38,no. 1,2021,463–476,Publisher:IEEE.

[39] J.TordesillasandJ.P.How,“PANTHER: Perception‑AwareTrajectoryPlanner inDynamicEnvironments”, IEEE Access,vol.10,2022,22662–22677, 10.1109/access.2022.3154037,Publisher: InstituteofElectricalandElectronicsEngineers (IEEE).

[40] K.Kondo,R.Figueroa,J.Rached,J.Tordesillas, P.C.Lusk,andJ.P.How,“Robustmader:Decen‑ tralizedmultiagenttrajectoryplannerrobustto communicationdelayindynamicenvironments”, IEEERoboticsandAutomationLetters,vol.9,no. 2,2023,1476–1483,Publisher:IEEE.

[41] J.TordesillasandJ.P.How,“MINVObasis:Find‑ ingsimplexeswithminimumvolumeenclosing polynomialcurves”, Computer‑AidedDesign,vol. 151,2022,103341,Publisher:Elsevier.

[42] K.Qin,“GeneralmatrixrepresentationsforB‑ splines”.In: ProceedingsPaci icGraphics’98. SixthPaci icConferenceonComputerGraphics andApplications(Cat.No.98EX208),1998,37–43.

[43] GNUProject,FreeSoftwareFoundation, GLPK –GNULinearProgrammingKit,FreeSoftware Foundation,2025.

[44] GurobiOptimizationLLC.“GurobiOptimizer ReferenceManual”,2020.

[45] P.E.Hart,N.J.Nilsson,andB.Raphael,“AFormal BasisfortheHeuristicDeterminationofMini‑ mumCostPaths”, IEEETransactionsonSystems ScienceandCybernetics,vol.4,no.2,1968,100–107,10.1109/TSSC.1968.300136.

[46] M.Quigley,K.Conley,B.Gerkey,J.Faust,T.Foote, J.Leibs,R.Wheeler,andA.Ng,“ROS:anopen‑ sourceRobotOperatingSystem”.In: ICRAWork‑ shoponOpenSourceSoftware,vol.3,2009.

[47] M.C.Escher,F.Bool,J.Locher,andB.Ernst,“MC Escher:hislifeandcompletegraphicwork:with afullyillustratedcatalogue”, HarryN.Abrams, 1982.

[48] E.W.Weisstein, TrefoilKnot,MathWorld,2025.

[35] B.Zhou,F.Gao,L.Wang,C.Liu,andS.Shen, “Robustandef icientquadrotortrajectorygen‑ erationforfastautonomous light”, IEEERobotics andAutomationLetters,vol.4,no.4,2019,3529–3536.

EVALUATINGDIJKSTRAANDA*PATHFINDINGALGORITHMSFORMOBILEROBOTS

EVALUATINGDIJKSTRAANDA*PATHFINDINGALGORITHMSFORMOBILEROBOTS EVALUATINGDIJKSTRAANDA*PATHFINDINGALGORITHMSFORMOBILEROBOTS

Submitted:26th August2025;accepted:27th August2025

PrabinKumarJha,ShamboRoyChowdhury

DOI:10.14313/jamris‐2026‐012

Abstract:

Inmodernwarehouseautomation,mobilerobots areessentialforenhancingoperationalefficiencyby autonomouslynavigatingtopickandtransportitems. Effectivepathplanningiscrucialfortheserobotsto movethroughcomplexenvironments,avoidobstacles, andminimizetraveltime.Thisstudyevaluatestwo prominentpath‐planningalgorithms,Dijkstraand ��∗ , implementedonamobilerobotwithinasimulated 3DwarehouseenvironmentusingCoppeliaSim.Three distinctracklocationswereanalyzedtoassessthe performanceofbothalgorithmsconcerningpath optimality,computationalefficiency,andreal‐time applicability.Simulationresultsindicatethatwhileboth algorithmssuccessfullygeneratedsafeandaccurate paths, ��∗ outperformedDijkstraintermsofspeedand pathefficiency. ��∗′ ’sheuristic‐drivenapproachresulted inlowercomputationalloadandfasterexecution time,makingitmoresuitableforreal‐timewarehouse operations,whereresponsivenessiscritical.Theinsights gainedprovidevaluableguidanceforroboticsengineers anddevelopersinselectingappropriatepath‐planning strategiesforautonomousnavigationinindustrial settings.

Keywords: DijkstraAlgorithm;A*Algorithm;Warehouse; MobileRobot;CoppeliaSim.

1.Introduction

Pathplanningofmobilerobotinvolvesdetermin‑ ingaroutefromastartingpositiontoadestina‑ tionwhileavoidingobstacles.Thisprocessisguided byoptimizationcriteriasuchasminimizingdistance, time,orenergyconsumption.

Tobeginpathplanning,therobot’soperatingenvi‑ ronmentmustbemodeled,aprocessknownasmap construction.Therearetwoprimarymethodsforthis: theroadmarkingmethodandthegridmethod.The roadmarkingmethodcreatesafeasiblepathbycon‑ nectingspeci icmarkers,usingtechniqueslikethe visibilitygraphandthetangentmethod.However,this approachiscomplex,haslowaccuracy,andisnot wellsuitedforlarge‑scaleenvironmentswithmany obstacles.Thegridmethoddividesthespaceinto small,uniformcells,makingmapconstructionmore straightforward.

Despiteitsadvantages,thismethodstruggleswith real‑timeandhigh‑precisionpathplanningwhen dealingwithlargegridsizesanddenselypacked obstacles[1].

Usinggrid‑basedenvironmentmodeling,various techniqueshavebeendevelopedformobilerobotpath planning.Traditionalgraphsearchalgorithms,suchas Dijkstra’salgorithmandtheA∗ algorithm,arewidely used.

Thesealgorithmsprovideaccurateresultsand can indtheoptimalpath.However,asthegridsize increases,theircomputationalef iciencydecreases, makingthemlesspracticalforlarge‑scaleenviron‑ ments.TheA∗ algorithmisamoremodernversion ofDijkstra’salgorithm,designedtoimproveef iciency byintroducingaheuristicfunctiontoprioritizenodes thatareclosertothegoal.ThisheuristicmakesA∗ moreef icientthanDijkstra’s,especiallyinlargeor complexsearchspaces.

TheA∗ algorithm,integratedwithagreedy approach,isutilizedformultiobjectivepointplanning tonavigate ivetargetnodeswithinawarehouse setting,allowingforacomparisonofpathlength, turningangles,andadditionalmetrics.Simulation resultsindicatethattheproposedstrategyproducesa smootherpath[2].AnimprovedversionofDijkstra’s algorithm,incorporatingamultilayerdictionary approach,wasutilizedtoenablemultiplerobotsto autonomouslyandsimultaneouslynavigateacrossan indoorenvironment[3].

Apheromone‑basedenhancementtothetradi‑ tionalDijkstraalgorithmhasbeenintroducedto reduceredundantin lectionpointsandimprovepath ef iciencyinmobilerobotnavigation[4].Theproximal policy‑Dijkstra(PP‑D)methodintegratesproximal policyoptimization(PPO)withDijkstra’salgorithm toenableef icientstrategylearningandreal‑time decision‑makingincomplexwarehouseenvironments [5].AnintegrationofDijkstra’salgorithmwithgenetic algorithmstodevelopapath‑planningstrategyfor robotsoperatingin irescenarios,focusingonmini‑ mizingtraversaltimeandthermalexposure[6].An improvedpath‑planningapproachwasdevelopedby combiningasegmentedpath‑planningmethodwith Dijkstra’salgorithmtoenhanceef iciencyindynamic environments[7].

Designandimplementationofanoptimized, collision‑freepath indingalgorithmbasedon anenhancedDijkstra’salgorithm,tailoredfor practicalapplicationsinobstacle‑richenvironments isreportedin[8].AnenhancedDijkstraalgorithm wasintegratedwithparticleswarmoptimization forglobalpathplanninginmobilerobots,aimingto reducecomputationaltimeandimproveef iciency in[9].Anovelmethodforautonomousmobilerobot pathplanningthatre inesroutesgeneratedbyboth conventionalandsampling‑basedalgorithmsthrough Beziercurveoptimization,designedforenvironments whereallobstaclesarefullyknownandaccountedfor hasbeenshownin[10].

Anintelligentpath‑planningframeworkfor assistive‑carerobots,integratingDijkstra’salgorithm withprobabilisticmodelcheckingtodynamically navigateenvironmentsbasedonpredictedhuman movementisshownin[11].

AhybridofanimprovedA∗ algorithmforoptimal globalpathplanningandanenhancedDWAalgorithm forlocalpathplanningwasproposedfortheforklift automatedguidedvehicle(FAGV).TheimprovedA* ensuresaglobalpathbettersuitedtoFAGVnavigation, whilethemodi iedDWAevaluationfunctionguides thelocalpathtoalignmorecloselywiththeglobal trajectory[12].

Localpathplanningintroducedanenvironment‑ awarestrategyfordynamicallyadjustingparame‑ ters,integratingevaluationfactorsfordeviationand dynamicobstacleavoidance.Thisapproachhelps overcomelocaloptimaandensurestimelyresponses tomovingobstacles[13].

AcomparativeanalysisofDijkstraandA*for mobilerobotsnavigatingthroughurbanenviron‑ mentswithobstacleshasbeenstudiedinextensively. Bothalgorithmsarecapableofproducingoptimal paths.A*outperformsDijkstraintermsofcomputa‑ tionalef iciencyduetoitsheuristic‑basedapproach, especiallyinenvironmentswherethegoalwasfar fromthestart.However,thestudyconcludedthat A*performancedegradedindynamicenvironments wheretheheuristicfunctionwasnotupdatedquickly enough[14].Anothercomparisonwasdonebetween Dijkstra’sandA*algorithmsformultirobotsystemsin acrowdedenvironment.ItwasnotedthatA*provides fasterpath indingforindividualrobotsinnondynamic environmentsbutbecamelessef icientwhenusedin real‑timemultirobotscenarioswhererobotshaveto avoideachother.Dijkstra’salgorithm,ontheother hand,wasmorepredictableandconsistentinmul‑ tirobotpath indingbutrequiredsigni icantlyhigher computationtime[15].

AnanalysiswasdoneforbothDijkstraandA*for real‑timerobotnavigation,particularlyinthecontext ofindoornavigationandobstacleavoidance.Itwas foundthatA∗ wasmoresuitedforreal‑timeappli‑ cations,asitcouldproducepathsmorequicklyby usingasuitableheuristic.However,inmorecomplex indoorenvironmentswhereobstaclesarenonstatic, Dijkstra’salgorithm,whencombinedwithobstacle detectionanddynamicplanning,showedbetterreli‑ abilityinensuringoptimalpaths[16].

Ahybridalgorithm irstusesA*to indaquickpath inastaticenvironmentandswitchestoDijkstrawhen theenvironmentbecomesdynamicorwhenthepath needstobeadjustedfrequently.Atestwasperformed onmobilerobotsindynamicandpartiallyobserv‑ ableenvironments;resultsshowedthathybridiza‑ tionimprovedef iciencyandrobustness[17].Acom‑ binedDijkstra’salgorithmwithA*forsolvingmulti‑ objectivepath indingproblemsinmobilerobotscon‑ sideredmultiplefactorssuchassafety,time,and energyconsumption.Aconclusionwasdrawnthat thehybridapproachcouldmoreeffectivelyaddress complexobjectiveswhilestillensuringoptimalpaths, combiningtheguaranteedoptimalityofDijkstrawith theef iciencyofA*algorithm[18].

Dijkstrademonstratedmarginallysuperiorpath qualityatthecostofincreasedcomputationtime whiletestedonTurtleBot3acrossbothsymmetrical andasymmetricaltestsetups[19].Adetailedcase studyhasbeendiscussedforA*algorithmadvan‑ tageinhandlingdynamicscenarioswhileemphasiz‑ ingDijkstra’srobustnessinobstacle‑densestaticset‑ tings[20].Theexperiments,doneundernonholo‑ nomicconstraints,reinforcedA*’sstrengthsinspeed andadaptability[21].Random‑sampling‑basedrobot pathplanningisreportedin[22]forautonomousfruit harvestingrobots.AmongDijkstra,A*,antcolonyopti‑ mization(ACO),andrapidlyexploringrandomtree (RRT)algorithmsevaluatedingridmazeenviron‑ ments,A*provedtobethemostef icient,delivering thefastestsearchtimesandthemostaccuratepaths underthegiventestconditions[23].Aself‑directed telepresencerobot(TR)capableoffollowingahuman subjecthasbeenintroduced.Itleveragesasingle‑ lens(monocular)cameraandtheyou‑only‑look‑once (YOLO)deep‑learningframeworkforidentifyingindi‑ vidualsandestimatingtheirproximity[24].

Insummary,pathplanningofmobilerobotsbegins withenvironmentmodeling,commonlydoneusing eithertheroad‑markingorgrid‑basedmethods.While roadmarkingoffersprecisegeometricpaths,itlacks scalabilityandaccuracyincomplexenvironments. Thegridmethodissimplerandmoreadaptablebut canfaceperformanceissuesindenseorlarge‑scale scenarios.

RecentadvancementsincludecombiningDijkstra orA*withalgorithmslikeRRT,DWA,andBeziercurve optimizationtogeneratesmoother,moreef icient, anddynamicallyfeasiblepaths.Comparativestud‑ iesshowA∗ generallyoutperformsDijkstrainspeed, especiallywithreal‑timeandheuristic‑guidedplan‑ ning,whileDijkstraismoreconsistentinstaticand multirobotsettings.Hybridstrategiesleveragingboth algorithmshaveshownpromisingresultsinadapting todynamic,uncertain,ordenselyobstructedenviron‑ ments.OtherAImethodslikeconsideringconstraint satisfactionproblemshavealsobeenusedforsolving robotpathplanning[25].Traditionalalgorithmslike Dijkstra’sandA*arewidelyusedforgrid‑basedpath planning.

Dijkstraensuresoptimalpathsbutiscomputa‑ tionallyintensive,whileA∗ improvesef iciencyusing heuristics.Bothhavebeenenhancedthroughhybrid models,integrationwithmachinelearning(e.g.,PPO), swarmintelligence(e.g.,PSO,ACO),andmultiobjec‑ tiveoptimization(e.g.,geneticalgorithms),toimprove navigationindynamicandobstacle‑richenviron‑ ments.

ThisstudypresentsacomparativeanalysisofDijk‑ straandA∗ algorithmsimplementedinautonomous mobilerobotsforwarehousenavigationusingthe CoppeliaSimsimulationin3Denvironment.Bothalgo‑ rithmsareevaluatedbasedonpathoptimalityand computationalef iciencytothewarehouselayout.

2.ModelingofGrid

Thegridmethod,introducedbyW.E.Howdenin 1968,isbasedonrepresentingamobilerobot’senvi‑ ronmentasabinarymatrix.Eachcellholdseithera0 ora1,where0indicatesafreespace,and1indicates anobstacle.Thisapproachsimpli iesenvironmental modelingforroboticpathplanning[1].

Algorithm1: MakeGraph(Grid)

Objective: TogenerateanundirectedgraphG=(V,E)froma2D occupancygrid,whereeachfreecellistreatedasavertex,and edgesareaddedbetweenadjacentfreecellsinthe four‑connectedneighborhood.

Description:

1. InitializeanemptyvertexsetVandanemptyedgesetE.

2. Iterateovereachcell(i,j)inthegrid:Ifthecellismarkedas free(i.e.,Grid[i][j]=0),dothefollowing:

a) CreateanentryinGraphwithkey(i,j)andinitializeits valuetoanemptylist.

b) Foreachofthefourneighbors (i+1,j),(i−1,j),(i,j+1),(i,j−1): IftheneighboriswithinboundsandGrid i′ j′ =0: Append(i′ ,j′)toGraph[(i,j)].

3. ReturntheconstructedGraph.

IntheCoppeliaSim,thisgrid‑basedrepresenta‑ tionoffersaclearervisualunderstandingofthe workspace.’S’denotesthestartinglocationand’ G’indicatesthedestination.Theenvironmentisdis‑ cretizedintoa2Dgrid.Eachcellinthegridisconsid‑ eredasanodebythesearchalgorithm.

To indacollision‑freepathfrompointStopoint G,speci icsearchrulesmustbefollowed.Thealgo‑ rithmusedinthisstudyinitiallyappliesafourneigh‑ borhoodsearchpattern,wherethemovementcost fromthestartingpoint �� inanydirectionissetto 1.ThealgorithmforgridisdescribedinAlgorithm 1andisimplementedwithaPythonAPI.Toenable ef icientpathplanning,weuseanadjacencylistrep‑ resentation,wherethegraphisstoredasadictionary, mappingeachnodetoalistofitsdirectlyreachable neighbors.

3.ImplementationofDijkstraAlgorithmina MobileRobot

Dijkstra’salgorithmisusedtosearchthe shortestpathbetweentwopointsbyminimizing thetotalcostoreffortrequiredtotravelfrom thestartingpointtothedestination.Itworksby evaluatingnodesinagraphandkeepingtrackof theoneswiththelowesttravelcost.Startingfrom thesource,thealgorithmexploresallpossible routes,continuallyupdatingtheshortestknown distances.Routeswithhighercostsaredisregarded infavorofmoreef icientpaths,ensuringthe inal resultisthemostcosteffectivepathtothegoal. TheDijkstra’salgorithmisexplainedinAlgorithm2.

Algorithm2: Dijkstra’sAlgorithmforGridBasedPathPlanning

Objective:

GivenagraphintheformatGraph[node]=[listofneighbor nodes],computetheshortestpathfromtheStartnodetothe GoalnodeusingDijkstra’salgorithm.

Description: Let:

• Graphbeadictionaryrepresentingtheadjacencylist,

• StartandGoalbenodesoftheform(x,y)

• distbeadictionarymappingeachnodetoitscurrent shortestdistanceestimate,

• prevbeadictionaryforbacktrackingtheoptimalpath.

Steps:

1. Initializedist[node]=∞forallnodesinthegraph;set dist[Start]=0

2. Initializeprev[node]=Noneforallnodes.

3. CreateapriorityqueueQandinsertStartwithpriority0.

4. WhileQisnotempty:Extractthenode��withthelowest dist[��].If��==Goal,terminatetheloop. ForeachneighborvinGraph[u]: Computealternativepathcost:alt=dist[u]+1 Ifalt<dist[v],then: Updatedist[v]=alt Updateprev[v]=u InsertorupdatevinQwithpriorityalt

5. ReconstructthepathfromGoaltoStartbyfollowingprev[v] links.

6. Returnthepathanditstotalcost(dist[Goal]).IfGoalis unreachable,returnfailure.

4.ImplementationofA*AlgorithminaMobile Robot

TheA*algorithm,introducedin1968,isan informedsearchtechniquethatcombineselementsof uniform‑costsearchandgreedybest‑ irstsearch.It evaluatespathsbasedonboththecostincurredandan estimatedcosttoreachthegoalfromthatnode.This estimationistypicallyderivedfromaheuristicfunc‑ tion,oftenrepresentingtheremainingdistancetothe goal.Byintegratingthisheuristic,A*ef icientlydirects itssearchtowardthemostpromisingpaths,aiming tosearchtheoptimalsolutionbyexploringafewer numberofnodes.TheA*algorithm,withsimilar2D occupancyGridasinput,isexpressedinAlgorithm3.

AnevaluationfunctionF(n)=G(n)+H(n),which combinestwokeyelements:G(n),theknowncost fromthestartingpointtothecurrentnode,andH(n), anestimatedcostfromthecurrentnodetothe goal.Thesecomponentsworktogetherandneedtobe properlybalancedtoensureoptimalperformance[2]. TheH(n) isaheuristicfunctionthathelpssteerthe searchintherightdirection.

Algorithm3: A*Algorithm

Step1:Initialization

1. Createanopenset(priorityqueue)tostorenodestobe evaluated,sortedbylowestf(n)

2. Createaclosedset(visitednodes).

3. Initializeg(start)=0

4. Calculatef(start)=g(start)+h(start)usingManhattan distance.

5. Pushthestartnodeintotheopenset.

Step2:MainLoop

6. Whiletheopensetisnotempty:

a. Popthenodecurrentwiththelowestf(n)fromthe openset.

b. Ifcurrentisthegoalnode,reconstructandreturnthe path.

c. Addcurrenttotheclosedset.

d. Foreachvalidneighborofcurrent(up,down,left, right):

i. Skipiftheneighborisanobstacleoralreadyinthe closedset.

ii. Calculate:

tentative=g(current)1

h=∣x_goal‑x_neighbor|+|y_goal‑y_neighbor|

f=tentative_g+h

Ifneighborisnotinopensetorhasabettergvalue:

i. Updateg(neighbor),f(neighbor) ii. SetcameFrom[neighbor]=current iii. Addneighbortoopenset(orupdatepriority)

Step3:PathReconstruction

1. Oncethegoalisreached,backtrackfromthegoalusing camefrom[]maptobuildtheshortestpath.

2. Returnthepathasasequenceofgridcoordinates.

However,ifH(n) istooloworunderestimated, thealgorithmreliesmoreheavilyonG(n),causingit tobehavemorelikeDijkstra’salgorithm.Thisresults inexaminingmorenodesandincreasescomputa‑ tionaltime,ultimatelyreducingtheef iciencyofthe A∗ search.Thus,selectinganappropriateandaccurate heuristicisessentialtomaintainA*’seffectivenessand

speed.Forourcase,wehadconsideredtheManhattan distancebetweenthecurrentnodeandthegoalnode astheheuristics.

Algorithm4: HeuristicFunction

Step1:Initialization

1. h(n)=|x.goal‑xn|+|y.goal‑yn|=abs(x.goalcurrent)+abs(y. goal‑current)

2. Pseudo‑code

3. functionA_Star(start,goal,grid):

4. openSet←priorityqueuewith(start,f(start))

5. cameFrom←emptymap

6. gScore[start]←0

7. fScore[start]←h(start,goal)

8. whileopenSetisnotempty:

9. current←nodewithlowestfScoreinopenSe

10. ifcurrent==goal:returnreconstructpath(cameFrom, current)removecurrentfromopenSetforeachneighborin get_4_neighbors(current,grid):ifneighborisobstacle: continuetentative_gScore←gScore[current]+1//move costis1ifneighbornotingScoreortentative_gScore< gScore[neighbor]:cameFrom[neighbor]←current gScore[neighbor]←tentative_gScorefScore[neighbor] ←gScore[neighbor]+h(neighbor,goal)ifneighbornotin openSet:addneighbortoopenSetreturnfailure//nopath found

11. functionh(node,goal):returnabs(node.x‑goal.x)+abs (node.ygoal.y)

12. functionget_4_neighbors(node,grid):neighbors←[]for directionin[(0,1),(1,0),(0,−1),(−1,0)]:nx←node.x+ direction.xny←node.y+direction.yifwithingrid(nx,ny) andnotgrid[nx][ny].isobstacle:neighbors.Append((nx, ny))returnneighbors

13. functionreconstructpath(cameFrom,current):path ←[current]

5.SimulationEnvironmentinCoppeliaSim

ThePioneerP3DXmobilerobotmodelistaken fromthelibraryofCoppeliaSim.Itisadifferential driverobothaving16sonars(ultrasonicsensors).Itis 485mmlong,381mmwide,and217mmhigh[25,26]. PioneerP3DXissimulatedforthenavigatingfrom astartpoint(S)toagoalpoint(G)usingDijkstra’s algorithmandA∗ algorithminCoppeliaSimversion 4.7.0[24].Itisfreesoftwareforacademicresearch purposes.ThesoftwareoffersaremoteAPIinterface withPython.TheODEphysicsengineisselectedfor thesimulationofrobotwitha0.001stimestep.

TheworkframeofCoppeliaSimisdesignedasa warehouselayout,asshowninFigure 1.Eightracks areimportedinthesimulationenvironment,andthe mobilerobotreachestothetargetrack.

Thecompletecontrolalgorithmsaredevelopedin aPythonAPI.Thedimensionsoftherackis6mlong, 0.6mwide,and4mhighandhassixracks.Thegrid sizeforbothalgorithms(DijkstraandA*)istakenas 10×10.Eachcellis1×1m2

Nodescorrespondtokeylocationslikeintersec‑ tionsandstorageareas,whileedgesdenotethecon‑ nectingroutesbetweenthem.Edgesaregivenweights accordingtoeitherthedistancebetweennodesorthe estimatedtraveltimealongthepath.

Foroursimulation,tomaintainuniformity,we havekepttheweightas1foreachneighboringgrid. ThestartpositionistheinitialpositionofthePioneer P3DX.Thestartpositionandthegoalpositionare (0,0)and(9,9)forinitialsimulations.Inaddition,two morestudieshavebeendonefortargetracksat(6,4) and(4,6),respectively.

Themobilerobotisplacedattheoriginofthe framelayout.Thepathisdeterminedbyfollowing theorderedlistofnodes.Further,thecontrollogicis designedinPythontofollowthepathtoreachthegoal rack.Aseparatesimulationisrunfortheimplementa‑ tionofDijkstraalgorithmandA∗ algorithminmobile roboticstosearchtheshortestpath.

6.Result

Pathplanningisacrucialaspectofrobotnaviga‑ tion.Itensuresthatarobotcan indthemostef icient routefromastartpositiontoagoalwhileavoiding obstacles.

TheDijkstraalgorithmwassuccessfully implementedinamobileroboticsystemwithin asimulatedenvironmentusingCoppeliaSim. TheredlineinFigure 2 showstheshortest pathtoreachthegoalrack.Thepathwas achievedthroughbothalgorithmsis:[(0,0),(0,1), (0,2),(0,3),(0,4),(1,4),(2,4),(2,5),(2,6),(2,7),(2,8),(3,8), (4,8),(5,8),(6,8),(6,9),(7,9),(8,9),(9,9)].Theobstacles were: = [(5,9),(3,9),(1,5),(7,5),(4,2),(8,2),(4,7),(0,0), (0,6),(2,3)].Therobotwasabletofollowtheshortest pathfromthede inedstartlocationtothegoalwhile avoidingobstaclesplacedintheenvironment.The Pioneer3PDXhasultrasonicsensorstodetectthe obstacles.Thresholdvalueforthesensorsarewritten inthecontrollogictomovetherobotproperly.

Theenvironmentwasdiscretizedintoanuniform 2Dgrid,whereeachcellrepresentedapossiblerobot location.Nodesweregeneratedforalltraversable gridcells,andedgeswerecreatedbasedonadjacency (four‑connectedneighbors)forthesearchgraph.

Thetrajectoryofthemobilerobot(green)followed theshortestpath.Itwasnotedthatthereisavariation inthetrajectoryofthemobilerobot.Thedistance betweenthestartpointtothegoalpointis24m, whichwasdeterminedbytheDijkstraalgorithm.The robottraveledthesamedistancein58s.Aseparate simulationwasperformedwiththeA∗algorithminthe sameenvironment.

Thesimulationenvironmentincludedprede ined startandgoalpositions,alongwithagrid‑basedrep‑ resentationoftheworkspace,whereeachgridcellwas assignedacostvaluebasedonitsdistancefromthe goalandanyobstaclepresent,asshowninFigure4.

Thesimulationresultsvalidatetheeffectivenessof theA*algorithmforgrid‑basedpathplanninginstruc‑ turedenvironments.Theheuristicfunctionplayeda signi icantroleinoptimizingperformanceandensur‑ inggoal‑orientedsearchbehavior.Thecomputational timeforthemobilerobottocoverthesamedistance was47storeachthetarget.A*algorithmwithan admissibleheuristicguaranteesoptimality.

Targetrackhasbeenplacedintwodifferentnodes (6,4)and(4,6),respectively.Thesimulationsare runtostudytheperformanceofbothalgorithms. Figure3showstheresultofbothalgorithms(Dijkstra atleftandA∗ atright).Thepathforthispurposeis [(0,0),(0,1),(0,2),(0,3),(0,4),(1,4),(2,4),(3,4),(4,4),(5,4), (6,4)].

Figure2. SimulationofDijkstraalgorithminCoppeliaSim

Figure3. Simulationof ��∗ algorithminCoppeliaSim

Figure4. Implementationofalgorithmsinmobilerobot fortargetrack (6,4)

Figure5. Implementationofalgorithmsinmobilerobot fortargetrack (4,6)

Theshortestdistancecoveredbythemobilerobot toreachthetargetrackinthewarehouseis15min 43swiththeDijkstraalgorithm,whereasintheA*star algorithm,thesamedistancewascoveredin39s.

Theresultofbothalgorithms(Dijkstraon theleftandA∗ iontheright)atthetargetnode (4,6),asshowninFigure5.Thepathforthis studyis[(0,0),(0,1),(0,2),(0,3),(0,4),(1,4),(2,4),(2,5), (2,6),(3,6),(4,6)].Theshortestdistancetraveled bythemobilerobottoreachthetargetrackinthe warehouseis19min48swiththeDijkstraalgorithm and41swiththeA∗ algorithm.

Dijkstra’salgorithmisaclassicapproachfor ind‑ ingtheshortestpathinaweightedgraphwithnon‑ negativeweights.A*,ontheotherhand,enhances Dijkstrabyusingheuristicstoprioritizewhichpaths toexplore,makingitmoreef icientinmanycases.

7.Discussion

ThecomparativeimplementationoftheDijkstra andA∗ algorithmsformobilerobotpathplanningina warehouseenvironmentusingCoppeliaSimrevealed signi icantdifferencesinperformance,computational ef iciency,andpracticalsuitabilityforstructured spaceslikewarehouses.

Threedifferentcaseshavebeenstudiedinthe warehouseenvironment.Thetargetrackhasbeen placedinnodes(9,9),(6,4),and(4,6),respectively. ItwasnotedfromTable1thatthecomputationtime forA*staralgorithmislowerinallthesecasesforthe robottoarriveatthetargetpointwhileavoidingthe obstacles.

Table1. Computationaltimetoreachthetargetrack

Target Node ComputationalTimes(s) Dijkstra algorithm A∗ algorithm %difference intime (9,9) 58 47 23 (6,4) 43 39 10 (4,6) 48 41 17

Table1alsoshowsthatthedifferenceincompu‑ tationaltimeisproportionaltothedistancebetween thestartnodeandthegoalnode.Bothalgorithms successfullygeneratedcollision‑freepathsfromthe starttothegoalpoint,avoidingstaticobstacles.

Dijkstra’salgorithm,whilereliable,showed increasedlatencyinlargermaps,asitcomputes theshortestpathtoallnodesbeforeidentifyingthe optimalroutetothegoal.ThismakesA*moresuitable forreal‑timeapplications,wherequickresponseand adaptabilityarecrucial.

Bothalgorithmswereeffectivelyimplemented usingembeddedscriptsinCoppeliaSim,leveraging integrationwithexternalPythonAPIsviatheremote API.Thesimulationenvironmentallowedvisualiza‑ tionofrobotpaths,nodeexploration,andobstacle avoidance.A*requiredtheadditionofaheuristicfunc‑ tion,whichaddedminorcomplexitybutsigni icantly improvedperformance.

Table2. Summaryofresearchwork

Criteria DistanceTraveled(m)

Dijkstraalgorithm A*algorithm

Pathoptimality Good Better

Speed Slower Moderate

Heuristicuse No Yes

Computationalcost Higher Lower

Implementationease Easy Moderate

Whilebothalgorithmsfunctionedwellinsimula‑ tion,A*provesmorepracticalforreal‑worldware‑ housescenarios.Itoffersbetteradaptabilityforstatic environments,especiallywhencombinedwithreal‑ timereplanningorsensorfeedback.Dijkstra,although deterministicandoptimal,lackstheresponsiveness requiredforsuchconditionsunlessheavilymodi ied, assummarizedinTable2.

8.Conclusion

Intherealmofautonomousmobile‑robotnavi‑ gationwithinwarehouseenvironments,theA*The algorithmhasgarneredsigni icantattentiondueto itsef iciencyandoptimalityinpathplanning.Recent literatureunderscoresA*’ssuperiorityovertradi‑ tionalalgorithmslikeDijkstra,particularlyinscenar‑ iosdemandingrealtimedecision‑makingandenergy ef iciency.Forinstance,studieshavedemonstrated thatA*notonlyreducescomputationtimebutalso maintainspathoptimality,makingitapreferred choiceformobilerobotsoperatingincomplexware‑ housesettings.

Traditionally,manyresearchershavefocusedon 2Drepresentationsofwarehouselayoutsforpath planning.However,thedynamicnatureofmod‑ ernwarehousesnecessitatesamorecomprehensive approach.Addressingthis,recentresearchhasintro‑ duced3Denvironmentalmodeling,whereracksand otherobstaclesareconsideredinthreedimensions, providingamorerealisticsimulationofwarehouse conditions.SimulationsconductedusingCoppeliaSim havehighlightedtheeffectivenessofgraph‑based pathplanningalgorithms,particularlyA*andDijkstra, innavigatingthePioneerP3DXrobotthroughcomplex warehouseenvironments.Byselectingthreedistinct targetnodeswithinthewarehouseracks,researchers havebeenabletoassesstheaccuracyandef iciency ofbothalgorithms.WhilebothA*andDijkstraare capableof indingoptimalpaths,��∗ hasdemonstrated asigni icantadvantageintermsofcomputationalef i‑ ciency,primarilyduetoitsheuristic‑drivenapproach. Thisef iciencyiscrucialforreal‑timeroboticapplica‑ tionswhererapiddecision‑makingisessential.

Theimplementationofthesesimulationsunder‑ scorestheutilityofplatformslikeCoppeliaSimin prototypingandvalidatingroboticsalgorithmsbefore deployingtheminrealworldscenarios.Suchsimu‑ lationenvironmentsallowforthoroughtestingand re inement,ensuringthatthealgorithmsperformreli‑ ablyundervariousconditions.Moreover,theuseof 3Dmodelinginthesesimulationsprovidesamore

accuraterepresentationofactualwarehouseenviron‑ ments,leadingtomorerobustandapplicableresults. Lookingahead,futureresearchcanexplorethe developmentofhybridalgorithmsthatcombine thestrengthsofvariouspath‑planningmethods toenhanceperformancefurther.Additionally, integratingdynamicreplanningcapabilitiesand incorporatingsimultaneouslocalizationandmapping techniques,alongwithreal‑timesensordata,can signi icantlyimprovetheautonomyandadaptability ofwarehouserobots.Suchadvancementswillenable robotstonavigatemoreef icientlyinever‑changing warehouseenvironments,ultimatelyleadingto increasedproductivityandoperationalef iciency.

AUTHORS

PrabinKumarJha∗ –DepartmentofElectronicsand CommincationEngineering,ASET,AmityUniversity, Bengaluru,India,e‑mail:pkjha@blr.amity.edu.

ShamboRoyChowdhury∗ –DepartmentofRobotics andAutomationEngineering,TheNeotiaUniversity, Kolkata,India,e‑mail:shambo.roychowdhury@tnu.in.

∗Correspondingauthor

References

[1] S.Alshammrei,S.Boubaker,L.Kolsi”Improved DijkstraAlgorithmforMobileRobotPathPlan‑ ningandObstacleAvoidance,” Computers,Mate‑ rialsandContinua,”vol.3,no.72,2022,pp.5939‑ 5954;https://doi.org/10.32604/cmc.2022.02 8165

[2] D.Xiang,H.Lin,J.Ouyang,J.etal.,”Combined ImprovedA*andGreedyAlgorithmforPath PlanningofMulti‑objectiveMobileRobot,” Scien‑ ti icReports,vol.12,2022,13273;https://doi.or g/10.1038/s41598‑022‑17684‑0.

[3] S.A.M.AbgenahandA.J.Azrul,”Multi‑Robot PathPlanningUsingDijkstra’sAlgorithmwith MultilayerDictionaries,” InternationalJournalof AdvancedResearchinComputerandCommunica‑ tionEngineering,vol.10,2021.

[4] Q.Ma,etal.,”PathPlanningforMobileRobots BasedonImprovingDijkstraAlgorithm,” Proc. SPIE12712,InternationalConferenceonCloud Computing,PerformanceComputing,andDeep Learning (CCPCDL2023),127121F,2023;https: //doi.org/10.1117/12.2678830

[5] K.Li,etal.,”ResearchonReinforcementLearning BasedWarehouseRobotNavigationAlgorithmin ComplexWarehouseLayout,”arXiv,2024;https: //arxiv.org/abs/2411.06128

[6] X.Xing,etal.,”ResearchonGlobalPathPlanning AlgorithmforMobileRobotsBasedonImproved A*,” ExpertSystemswithApplications,vol.243, 2024; https://doi.org/10.1016/j.eswa.2023. 122922.

[7] S.Vaish,Shreyam,andS.Singhal,”Segmented ApproachtoPathPlanning,” 2020SixthInter‑ nationalConferenceonParallel,Distributedand GridComputing”(PDGC),2020,pp.266‑271; ht tps://doi:10.1109/PDGC50313.2020.9315788.

[8] A.UbaidillahandH.Sukri,„ApplicationofOdom‑ etryandDijkstraAlgorithmasNavigationand ShortestPathDeterminationSystemofWare‑ houseMobileRobot,” JournalofRoboticsandCon‑ trol,vol.4,no.3,2023,pp.413‑423;https://doi. org/10.18196/jrc.v4i3.18489

[9] Y.Tang,M.A.Zakaria,andM.Younas,„PathPlan‑ ningTrendsforAutonomousMobileRobotNav‑ igation:”AReview,” Sensors,vol.25,no.4,2025, 1206;https://doi.org/10.3390/s25041206.

[10] Z.DuraklıandV.Nabiyev,”ANewApproach BasedonBezierCurvestoSolvePathPlanning ProblemsforMobileRobots,” JournalofCompu‑ tationalScience,vol.58,2022,101540; https: //doi.org/10.1016/j.jocs.2021.101540.

[11] J.Lietal.,”FusionofImprovedA*Algorithm andDynamicWindowApproachfor PathPlanning,” 2022ChinaAutomation Congress (CAC),2022,pp.390‑394;doi: 10.1109/CAC57257.2022.10056090.

[12] B.Wuetal.,”DynamicPathPlanningforForklift AGVBasedonSmoothingA*andImprovedDWA HybridAlgorithm,”, Sensors,vol.22,2022,7079; https://doi.org/10.3390/s22187079

[13] Y.Shi,S.HuangandM.Li,”AnImprovedGlobal andLocalFusionPath‑PlanningAlgorithmfor MobileRobots,” Sensors,vol.24,2024,7950; ht tps://doi.org/10.3390/s24247950

[14] Y.ZhouandN.Huang,”AirportAGVPathOpti‑ mizationModelBasedonAntColonyAlgorithm toOptimizeDijkstraAlgorithminUrbanSys‑ tems,” SustainableComputing,Informaticsand Systems,vol.35,2022,100716; https://doi.or g/10.1016/j.suscom.2022.100716

[15] R.Kumaretal.,”PerformanceComparisonofA* SearchAlgorithmandHill‑ClimbSearchAlgo‑ rithm:ACaseStudy,” Multifacetedapproachesfor DataAcquisition,Processing&Communication, 1sted.CRCPress,2024,pp.185194,; https:// doi.org/10.1201/9781003470939

[16] C.Chengetal.,„PathPlanningandObstacle AvoidanceforAUV:AReview,” OceanEngineer‑ ing,vol.235,2021,109355;https://doi.org/10 .1016/j.oceaneng.2021.10935

[17] H.Duetal.,”Multi‑ObjectiveLooselySynchro‑ nizedSearchforMulti‑ObjectiveMultiAgentPath FindingwithAsynchronousActions,” Journalof ShanghaiJiaotongUniversity(Science),vol.29, 2024,pp.667‑677; https://doi.org/10.1007/ s12204‑024‑2744‑x

[18] L.AitBenMouhetal.,(2023).”AHybrid ApproachofDijkstra’sAlgorithmandA*Search, withanOptionalAdaptiveThresholdHeuristic,”

BusinessIntelligence,R.ElAyachi,M.Fakir,and M.Baslam,eds.Springer,2023;https://doi.org/ 10.1007/978‑3‑031‑37872‑0_9.

[19] P.M.deA.Brasiletal.,”DijkstraandA*Algorithms forGlobalTrajectoryPlanningintheTurtleBot 3MobileRobot,” AdvancesinIntelligentSystems andComputing,Springer,2020;https://doi.org/ 10.1007/978‑3‑030‑711870_32

[20] J.S.S.Pavithraetal.,”ContrastiveAnalysisof PathPlanningAlgorithmsforMobileRobots:A CaseStudy,” 2023InternationalConferenceon Energy,MaterialsandCommunicationEngineer‑ ing (ICEMCE),2023,pp.1‑6;doi:10.1109/ICEM CE57940.2023.10434168

[21] A.R.Warrieretal.,”ImplementationofClassical PathPlanningAlgorithmsforMobileRobotNav‑ igation:AComprehensiveComparison,” 2022 InternationalConferenceonElectrical,Computer, CommunicationsandMechatronicsEngineering (ICECCME),2022;https://doi.org/10.1109/ic eccme55909.2022.99.88092

[22] T.Yoshidaetal.„AutomatedHarvestingbya Dual‑armFruitHarvestingRobot,” Robomech Journal,vol.9,no.19,2022;https://doi.org/10 .1186/s40648‑022‑00233‑9.

[23] H.S.Yang,”AnalysisandStudyonPathPlanning AlgorithmsintheFurtherMobileAction,” Journal ofPhysics,vol.2824,no.1,2024,012006;https: //doi.org/10.1088/17426596/2824/1/0120 06.

[24] A.A.F.Sakrietal.„AutonomousPerson‑Following TelepresenceRobotUsingMonocularCamera andDeepLearningYOLO,” ApplicationsofMod‑ ellingandSimulation,Vol.8,2024,pp.101109.

[25] RobotSimulatorCoppeliaSim:Create,Compose, Simulate,AnyRobot,CoppeliaRobotics.

[26] WebotsDocumentation:Adept’sPioneer3‑DX.

Abstract:

ENERGY‐AWARECLUSTER‐BASEDROUTINGWITHFEDERATEDLEARNING INTEGRATIONFORSCALABLEIOTENVIRONMENTS

Submitted:16th July2025;accepted:1st October2025

AnkurSisodia,SwatiVishnoi,ShivshankerSingh,NandiniSharma,AjayKumarYadav

DOI:10.14313/jamris‐2026‐013

AsaresultoftheInternetofThings’(IoT)explosive growthsecurerouting,energyoptimization,andpri‐vacypreservationinresource‐constrainedenvironments havebecomemajorchallenges.Highoverhead,static decision‐makingandsusceptibilitytomalevolentattacks arecommonproblemswithtraditionalroutingprotocols. FederatedLearning‐AssistedEncryptedRoutingbasedon CostFunction(FL‐ERCF),animprovedroutingprotocol thatcombinesencryptedtransmissionwithintelligent, privacy‐preservingclusterhead(CH)selection,ispro‐posedinthispapertoaddresstheseissues.Theproposed protocolconsistsofthreekeyoperations:linkquality assessmentbasedonReceivedSignalStrengthIndica‐tor(RSS)I,SNR,andvariancemeasurementstrustbased clusteringledbyfederatedlearning(FL)thathasbeen trainedusingdistributedIoTnodestodynamicallyselect themostsuitableCHsandsecuredatatransmissionvia alightweightsymmetricencryptionalgorithmthathas beenimprovedwithdigitalcertificates(DCs).FLcanpre‐servetheprivacyofnodesatthelocallevelandpro‐videstrongrobustnesstomodelflexibilityandnetwork dynamics.TheproposedFL‐ERCFprotocolisimplemented intheNetSimsimulatorandcomparedwithstronger protocolssuchasEnergyEfficientSecureRouting(EESR) andHybridSecureRouting(HSR).Theperformanceeval‐uationdemonstratesanimprovedpacketdeliveryratio (PDR)andreducedroutingoverheadandthroughput evenagainstonadversaryattack.Inaddition,theadap‐tiveandsecurenatureofFL‐ERCFalsomakesitsuitable forothermobileroboticnetworkslikedroneswarms andindustrialrobots,wheretrust,energyefficiency,and mobilityareessential.

Keywords: FederatedLearning(FL),SecureIoTRout‐ing,ClusterHead(CH)Selection,TrustScoreEvaluation, Energy‐EfficientCommunication.

1.Introduction

TheInternetofThings(IoT)hasrapidlyevolved intorevolutionaryadvancementinaworldpopu‑ latedwithbillionsofsmartthingsthatgenerate andexchangedataacrossdifferentdomains,such ashealth,smartcities,smartagricultureandtrans‑ portation,andindustrialautomation(precisewiring). Morethan75billiondeviceswillbeconnectedworld‑ wideby2025,creatinganecosystemthatisboth

dynamicandcomplex,andwhichmandatesintelli‑ gent,secure,andreliablecommunicationprotocols. DespiteadvancesintheadoptionofIoT,severalchal‑ lengescontinuetopreventproperfunctioning,par‑ ticularlyinsecurity,energyandroutingaspects.In routesetupandclusterhead(CH)selection,conven‑ tionalroutingprotocolsoftenusestaticmetricorpre‑ determinedthresholds,whichmakeitlessef icient indynamicnetwork.Moreover,mostofIoTdevices haveconstrainedmemory,computationalcapacity, andenergyresourceswhichalsomakethemhighly susceptibleagainstpacketloss,networkcongestion, andothercyberthreats,i.e.,spoo ing,Sybilattacks, anddataforgery.Securityandprivacyconcerns areevenexacerbatedinthecaseofdistributedIoT setups.Although,thiskindoftraditionalcentralized orcryptographic‑basedschemes,arenotsuitableand oftensufferfromthehighcommunicationoverhead andnon‑scalabilityissuesespeciallyforlarge‑scale andheterogeneousIoTnetworks.Alsowhenrawdata cannotbesharedbetweennodesandprivacycon‑ straintscanbecompromised,existingsecurerouting solutionsarelikelytostruggletomeettherequire‑ mentofreal‑timeadaptabilityorprivacypreserving managementofdata.PastandrecentIoTandrobot‑ relatedchallengesalsopointtotherequirementfor novelroutingsolutions.Forexample,mobility‑related latencyandcoordinationinroboticswarmsoflarge sizesrequireprotocolsthateffectivelytradesecurity, lexibility,andlowoverheadagainstprivacy.Resilient communicationinfrastructureisessentialinindus‑ trialautomationandintelligentmanufacturingfor real‑timecontrolandanomalydetection.

Inthispaper,weproposeanaugmentedrouting protocolnamedFederatedLearning‑assisted EncryptedRoutingusingCostFunction(Fl‑ERCF). Federatedlearning(FL)isadecentralizedmachine learning(ML)techniquetojointlytrainaglobal modelviacooperationoflocalIoTnodeswithout sharingtherawdata,whichistheprincipleideafor introducingtheintelligenceandprivacyawareness intothedeploymentoftherouting.Thisensures routingdecisions,especiallytheselectionofCH, arealsoresilienttodataleakageandadaptive towardsnetworkchanges.Theoperationofthe FL‑ERCFprotocolconsistsofthreeindependentyet interrelatedphases:

LinkQualityAssessment–AQualityIndicatorfor Link(QIL)isderivedbasedonconsiderationofkey

transmissionparameterssuchasRSSI,SNR,quality variance.Thesemetricshelpto indreliablecommu‑ nicationpathswithlittle‑energyoverheadandlow interference.

FederatedLearning‑GuidedClusterFormation–A post‑trainedFLmodelpredictingtheoptimalCHsis supplementarytostandardcost‑basedCHselection, basedonhistoricalinformation,residualenergy,trust ratingsandthenumberofnodes.Integratinglocal learningupdatesofeachnodeintoaglobalmodel,it avoidsrawdataexchangeandenhancessecurity.

SecureEncryptedDataTransmission–Symmetric keyencryption(SKE)andDCbasedauthentication arethetypesoflightweightcryptographictechniques employed[4].Theseschemesensurecomputational ef iciencyondeviceswithconstrainedresourcesas wellasensuresafe,low‑latencydataforwarding betweenCHsandthesink.

WeusetheNetSimsimulatortovalidatethepro‑ posedprotocolunderdifferentcommunitytypesand networksettings.Theperformanceoftheproposed schemeisanalyzedintermsofthroughput,routing overheadandpacketdeliveryratio(PDR)against twoexistingsecureprotocols,namelyEnergyEf icient SecureRouting(EESR)andHybridSecureRouting (HSR).Accordingtosimulationresults,FLERCFper‑ formsbetterthanthecurrentmethodsinbothtypical andattack‑proneenvironments,exhibitingimproved securitycompliance,scalability,andenergyef iciency. Thispaper’smaincontributionsareasfollows:

1. Asecureroutingframeworkthatcombines linkqualitymetrics,trust‑basedclustering,and lightweightencryption;

2. FLintegrationforintelligentandprivacypreserv‑ ingCHselectioninIoTnetworks.

3. Athoroughassessmentofperformanceusingsim‑ ulationandcomparisonwithmodernprotocols.

4. Therestofthepaperisorganizedasfollows:The relevantliteratureonFLstrategiesandsecureIoT routingisreviewedinSection 2.Thesuggested FLERCFprotocol’soperationisdescribedindetail inSection 3.Thesimulationsetupandperfor‑ mancemetricsaredescribedinSection4.Results arepresentedanddiscussedinSection5.Section6 bringsthestudytoacloseandidenti iesareasfor furtherresearch.

2.LiteratureReview

Thedynamicnatureofwirelesssensorenviron‑ mentsandtheexponentialgrowthofIoTdevices havemadesecureandeffectiveroutinganessential researchtopic.Numerousstudieshaveconcentrated onenhancingdatatransmissionusingcryptographic schemes,energy‑awarerouting,andclustering.Exist‑ ingmethodsarestillconstrainedbyissueslikereal‑ timeadaptability,privacy‑preservingintelligence,and attackresilience.Inordertogetaroundtheselimita‑ tions,thereisalsogrowinginterestinincorporating FLandMLintoIoTprotocols,accordingtorecent

literature.Inordertoensureintegrityandauthentica‑ tioninIoTcommunications,secureroutingprotocols havedeveloped.Cluster‑basedroutingisanemerg‑ ingtechnologythatadoptstraditionalenergy‑ef icient protocolssuchasLEACH,HEED,andSEP,however theseprotocolsdonothaveasoundsecuritymecha‑ nism.Eventhoughitisachallengetohavethe ixed selectionofselectionindynamicenvironmentsisa bottleneck,inauthor[1]presentedatrustawareclus‑ teringtechniqueinwhichselectionofclusterheads iscarriedoutbasedonlinkqualityandenergymet‑ rics.Itisworthnotingthatin[2]authorspresenteda protocolcombiningdigitalsignaturesandencrypted routingtoprovideahigherlevelofresiliencetoattacks despitebeingbothenergyef icientandavoidingintru‑ sion.Inthesamelinewecanalso indawork[3], whereanHSRmechanismisproposedand,where traf icmonitoringformaliciousnodedetectionis combinedwithlightweightcryptographicfunctions. However,thesemechanismsdependoncentralized decisionand ixedthresholds,whichmaynotbefea‑ sibleinmobileorheterogeneousIoTnetworks.In resource‑constrainedenvironments,trust‑basedtech‑ niqueshavegainedpopularityindetectingmalicious behavior.Afuzzyclusteringprotocoldevelopedbythe author[4]toimproveenergyusageandnodeselection trust.Accordingtotheauthor[5],itislesspossibleto havedataspoo ingandselectiveforwardingattacksif weincorporatetrustvaluescalculatedthroughphys‑ ical,bandwidth,andcongestionscores.Despitetheir effectiveness,thesemethodsfrequentlylack lexibility andrelyonpresetformulas.Thereisstillaresearch gapintheintegrationofadaptive,data‑driventrust evaluation.Furthermore,distributedintelligenceand real‑timelearningbasedonnodebehaviorevolu‑ tionarerarelysupportedbythetrustmodelsinuse today.FLisadecentralizedMLparadigmthathas gainedpopularityrecently.Itisperfectforenviron‑ mentsliketheIoTthatarebandwidth‑constrained andprivacysensitive.ToprotectdataprivacyinFL, eachnodetrainsalocalmodelusingitsowndata andonlycommunicatesmodelupdatestoacentral aggregator.ThefoundationofFLwaslaidbyGoogle’s groundbreakingworkby[6],whichhassincebeen appliedto ieldslikesmarttransportationandmobile health.FLhasbeenusedforpredictivemaintenance, anomalydetection,anddynamicresourceallocation inthecontextoftheIoT.AsafeFL‑basedintrusion detectionsystemforedgedeviceswaspresentedby [7],whoachievedpreciseresultswithoutjeopardiz‑ ingdataprivacy.Inasimilarvein[8]usedFLin wirelessnetworkstofacilitategroupdecisionmaking withoutexchangingrawdata.Nevertheless,littleis knownabouttheapplicationofFLinCHselection. Currentclusteringalgorithmsdon’ttakeadvantage oflocalintelligenceoradjusttoquicklyshiftingnet‑ workconditions.Thispaperattemptsto illthegap bypredictingoptimalnodesbasedonenergy,trust, andlinkqualitybyincorporatingFLintoCHselec‑ tion.Hybridmodelsthatcombineintelligenceand securityhavebeenstudiedrecently.Forlightweight

datasecurityintheIoT[9]suggestedacrypto‑ graphicmethodbasedonUnicode.Bycombiningsev‑ erallevelsoftrustandauthentication,[10]high‑ lightedthenecessityofsecureIoTdesign.Although fewofthesestudiesuseFLoraddressrealtimeintel‑ ligence,theydohighlightthesigni icanceofhybrid designs.

Newcontributionsalsopointoutweaknesses inresiliencetoadversariallearningconditions.For instance,[11]addressedtheissueofnon‑IIDdataand poisoninginFLbyemployingadversarialsynthetic data,whereas[13]introducedgradientscalemoni‑ toringasasecuritymechanismforFLsystems.These developmentsshowthatFLinIoThastoconsidernot justcommunicationef iciencybutalsorobustnessto poisoningattacks.

Concurrently,IoTenergyef iciencyisalsoofcrit‑ icalconcern.In[12],acomprehensiveevaluation ofmulti‑hopmesh‑basedIoUTnetworks’protocols emphasizedthecompromisesbetweenenergycon‑ sumptionandsecuretransmission.Likewise,[15]pre‑ sentedanumericalIoT‑bigdataintegrationmodel forreducingenergyconsumptioninsmartbuild‑ ingsinaccordancewithsustainabilityneeds.Stud‑ ieslike[16]and[18]usedIoTdatacompression algorithms(e.g.,LZW)toimproveperformanceover land,and[14]showedimprovedreliabilityforunder‑ waterWSNsinSociety5.0applications.Collectively, thesestudieshighlightthatsecurityandenergyef i‑ ciencyneedtoco‑evolvetogetherindiverseIoT settings.

Hybridandroboticsystemsalsoshedlighton howintelligenceandcommunicationareintegrated. In[17],RGB‑Dperception‑basedmultimodalrobot programminginterface,and[19]designedaninverse kinematicsmodelforan18‑degreeoffreedomrobot. Likewise,[20]showedahybridnavigationmethod thatallowsrobotstodriveelevatorsautonomously. Althoughtheseroboticsapplicationsaredistinct fromIoTrouting,theyre lecttheincreasinguse ofdistributedintelligence,adaptivealgorithms,and realtimedecision‑making,whicharesimilartothe requirementsforIoTclusteringmechanisms.

Upontheseobservations,thenewFL‑ERCFproto‑ colimprovesuponexistingcontributionsinthefol‑ lowingways:

Incontrasttostaticclusteringmethods[1–5,12,14–16,18],FL‑ERCFinvolvesfederatedlearning‑based CHselection,renderingadaptive,data‑informed decisionshighlyresponsivetodynamicnetwork conditions.

Security‑orientedmodels[2,3,9–11,13]lackincor‑ poratingsymmetricencryptionwithlowcomputa‑ tionalcomplexityandDCcheckingforstrongtrust estimation.

UniquecomparedtoearlierFLdeploymentsinIoT [6–8, 11, 13],suchasanomalydetectionordata exchange,FL‑ERCFexplicitlyextendsFLtoCHselec‑ tionandroutingandtherebyfacilitatesreal‑time privacy‑preservingintelligence.

Withthesynergisticuseoflinkqualityestimation, trustscoring,andFL‑basedclustering,FL‑ERCFpro‑ videsimprovedadversarialrobustnessatthecostof noscalabilityorenergyef iciency[12,15,16].

Bybeingbasedonrobotics‑inspireddistributed intelligence[17,19,20],theprotocolisscalableand adaptableinheterogeneousIoTdeployments. Consequently,FL‑ERCFoffersitselfasaparadigm forscalableIoTsystemstocombineprivacypreserving learningwithsecure,real‑time,and lexiblerouting schemes, illingtheloopholesleftopenbyprevious research.

3.Methodology

ThesuggestedFL‑ERCFprotocoldividestherout‑ ingprocessintothreemethodicalandinterconnected stagesinordertoprovidesecure,effective,andintel‑ ligentroutinginIoTnetworks:

1. LinkQualityAssessment

2. ClusterFormationAssistedbyFL

3. LightweightDCsforEncryptedTransmission

LinkQualityEvaluation

SNR RSSI QIL

Federated LearningAssisted Cluster Information

Cost Function

Federated Learning

ECDH Digital Certificates

Encrypted Transmission

Figure1. BlockdiagramofFL‐ERCFprotocolshowing threecorephases:LinkQualityEvaluation,Federated Learning‐AssistedClusterFormation,andEncrypted Transmission

Figures1and2displayablockdiagramofFLERCF. Belowisanexplanationofeachelement.

Start

Equidistant Initial

Computecost function onclusters

Detectand mitigate attacks Stop Cost function minimized Yes No

Figure2. Flowofsecureclusterformationandattack detectionusingcostfunctionandtrustscoresinFLERCF

3.1.PhaseI‐LinkQualityEvaluation

Choosingrouteswithminimalinterferenceand highlinkstabilityisnecessaryforreliabledatarout‑ ing.Threeprimaryparametersareusedinthisphase tocalculatetheQIL:

• TheRSSI,orReceivedSignalStrengthIndicator

• SNR,orsignal‑to‑noiseratio

• LinkVariance

ThesemetricsareessentialforsubsequentCH selectionandarecomputedusingreal‑timedatagath‑ eredfromeachnode’stransceiver.

Step1:ComputeRSSI

AsignalmetriccalledRSSIindicateshowstrong thesignalwaswhenitwasreceived.FornodeI,the RSSIisprovidedbyEq.1andEq.2.

• RIvalue:registerreading

• RSSIoffset:ahardware‑speci icvalue(e.g.,‑45dBm)

• Bn:backgroundnoise

Step2:ComputeSignal‐to‐NoiseRatio(SNR) SNRcanbeevaluatedusingEq.3

Where:

Step3:CalculateQualityIndicatorforLink(QIL) QILcanbeevaluatedusingEq.4.

Where:

• C:hardwarecorrelationcoef icient(range:50100)

• x,y:empiricallyderivedconstants

Eachnode’scommunicationstabilityisre lectedin theQILscorethatisproducedbythisphase.Forintel‑ ligentclustering,theQILmovesontothefollowing stage.

3.2.PhaseII‐FederatedLearning‐AssistedClusterFor‐mation

ConventionalCHselectionthatreliesondirect metricsorstaticthresholdsfrequentlyresultsinless‑ than‑idealroutingchoices.WepresentaFLframe‑ workthatusesdistributedtrainingtochooseCHsand ClusterGateways(CGs)inanadaptiveandintelligent manner.

3.2.1.CostFunctionComputation

Eachnodecalculatesalocalcostfunctionthat weighsenergyef iciency,linkquality,andsinkdis‑ tanceasinEq.5:

Where:

• Eresidual:residualenergy

• QILi:computedlinkquality

• Di:distancetosinknode

• ��,��,��:tunableconstants(e.g.,0.4,0.4,0.2)

3.2.2.TrustScoreCalculation

ThreefactorsareusedtocalculateaTrustScore (TS),whichprotectsCHsfrommaliciousactivity:

• BandwidthTrustScore(BTS): BTSisusedtoevaluatethereliabilityofpacket forwardingasshowninEq.6.

• CongestionTrustScore(CTS):

CTSisusedtoevaluatehownodesarehandlingthe overallloadshowninEq.7

• PhysicalTrustScore(PTS):

PTSisusedtoevaluatehowtheresidualenergy combinespacketsuccessratioshowninEq.8

The inalTSiscalculatedwiththehelpofweighted combinationasshowninEq.9.

TS=a∗BTS+b∗CTS+c∗PTSwherea+b+c=1 (…Eq.9) NodeswithTS<0.35areexcludedfromclustering.

3.2.3.FederatedLearning(FL)ModelforCHSelection

EverynodetrainsalocalCHpredictionmodel,such asalightweightneuralnetordecisiontree,usinglocal values(QIL,TS,Cost,andEnergy).

• Nodessharemodelweightsorgradientsratherthan sendingdata.

• AglobalCHpredictionmodeliscreatedbycombin‑ ingupdatesfromacentralaggregator(ordecentral‑ izedaggregatorsviasecureaggregation).

• Allnodesreceivetheupdatedmodelbackforinfer‑ ence.

• EverynodepredictsifitshouldfunctionasaCH usingtheglobalmodel.

Toadjusttochangesintopologyandenergy dynamics,thisprocedureisrepeatedonaregular basis(e.g.,every5‑10rounds).

3.2.4.FinalClusterFormation

Thenetworkcreatesclusterswithone‑hopor twohopcoverageafterCHsandCGsarechosenusing theFLmodelandtrustvalidation.Nodesjointheclos‑ estCHwiththehighestQILvalueandtrust.

3.3.PhaseIII‐EncryptedTransmissionUsing LightweightCertificates

• EllipticCurveDif ie‑Hellman(ECDH)forkey exchangeisoneofthehybridtechniquesusedto securedatatransmission.

• DCESC,orDC‑basedauthentication

• Forrealmessagetransfer,SKE Everynodekeepsapairofprivateandpublickeys:

• PKi∈[1,n−1],and

• PUKi = PKi ⋅ B,whereBisthebasepointonthe ellipticcurve.

DigitalCertificate(DC)Exchange:

• IoTnodesaskthegatewayforaDC.

• GatewayissuesasignedDCaftercon irmingthe identityofthenode.

• Tomutuallycon irmidentities,nodesuseDCs.

EncryptionProcess(Algorithm2):

• DeterminethesharedkeySH=PKs⋅PUKd.

• UseasymmetrickeygeneratedfromSHtoencrypt themessage.

• Delivertheencryptedmessagetotherecipientalong withaveri icationsignature.

DecryptionProcess(Algorithm3):

• VerifysenderusingDC

• ComputeSH=PKd⋅PUKs

• Decryptmessageusingderivedkey

Thisguardsagainstimpersonationandeavesdrop‑ pingattacksandguaranteeslow‑energy,securetrans‑ missionappropriateforIoTdevices.

3.4.ComputationalConsiderations

WhileFLallowsfor lexibilityandprivacy,its applicationtoIoTnetworksandrobotswarmsbrings withitsomecomputationalconstraints.Localtraining oneverynodedemandsextraCPUcyclesandtransi‑ torymemoryallocationforupdateofgradients,and periodicmodelaggregationcontributestocommuni‑ cationoverhead.Suchoverheadsmayleadtoraised energyconsumption,whichisespeciallyimportantfor low‑resourcedevices.Tocounterthis,thesuggested FL‑ERCFframeworkisarchitectedwithlight‑weight MLmodelslikeshallowneuralnetworksratherthan computationallydemandingdeeparchitectures.Addi‑ tionally,thelearningprocessisconductedperiodi‑ callyinsteadofinreal‑time,thuscuttingdownonthe updatefrequencyandsavingenergyandbandwidth. Tofurtherenhancecommunicationef iciency,updates tomodelsarecompressedpriortosending,makingFL possibleforevenlow‑powerIoTdevices.Thismanner, FL‑ERCFstrikeabalanceamongdistributedintelli‑ gence,privacyprotection,andtherealisticresource constraintspresentinactualIoTandroboticsettings.

4.PerformanceEvaluationandExperimental Setup

WeusedtheNetSimsimulatorwithextensionsthat supportFLcomponentsthroughPythonintegration tocreateacomprehensivesimulationenvironmentin ordertoassesstheperformanceofthesuggestedFL‑ ERCFprotocolanditssimulationenvironmentparam‑ etersareshowninTable 1.Thesimulation’smain objectiveistoevaluatehowFLassistedCHselection andencryptedroutingaffectimportantnetworkper‑ formancemetricsincontrasttotwopopularsecure routingprotocols:EESRandHSR.

Theseprotocolswereselectedastheyexhibited strongpropertiesofenergy‑awaresecuredroutingas wellasclusteringprotocols.

Thepurposeofthesimulationusedinthis studyistothoroughlyevaluatehowwellthe proposed(FLERCFprotocolamelioratesissues ofsafe,costeffective,and lexibleroutingforIoT environments.Aprimaryaimoftheevaluationisto validatetheeffectofselectingtheCHdrivenbyFLon

Parameter Value/Range

IoTNodes 50,100,200,300,400,500

SimulationArea 500m×500m

InitialEnergyperNode 0.6Joules

SimulationTime 200seconds

CommunicationRange 50m

NodeDeployment RandomUniform

Mobility Static

RoutingProtocolsEvaluated FL‑ERCF,EESR,HSR

Attackers(MaliciousNodes) 10%and20%oftotalnodes

FLRoundInterval Every10simulationseconds

FLModel DecisionTreeClassi ier(Scikit‑learn)

FLOptimizer FedAvg(FederatedAveraging)

SecurityTechnique ECDHwithDigitalCerti icates(DCESC)

theapplicationsinfrastructureoverallperformance: especiallyconcerningroutingeffectivenessand robustness.Toevaluatetheprotocol’srelative meritinscalability,energypreservation,and trustmanagement,wecompareitagainsttwo popularsecureroutingstrategies:EESRandHSR. Furthermore,thesimulationextendsintoanalysisof theprotocol’slevelofresistancetomaliciousattacks, namelybyobservingtrendsinthethroughput, PDR,androutingoverhead,accordingtoincreasing adversarialbehavior.Avarietyofmetricswillbe usedtoensureathoroughperformanceevaluation. Throughputorsuccessfuldatareceptionpersecond, yieldsinsightintohowwellaprotocolmaintains communicationinmeasureofregardtotherelative circumstances.Thenetwork’sreliabilityismeasured viathePDR,astheratioofpacketssuccessfully received,relativetosentpacketsfromsourcenodes. Routingoverhead,i.e.numberofcontrolpackets generatedduringroutediscoveryandmaintenance, isindicativeofhowwelltheprotocolcommunicates. Inaddition,networklifetimewillbecaptured usingthreeindicators:FirstNodeDies(FND),Half NodesDead(HND),andLastNodeDies(LND), whichcapturesthefullperspectiveoftheenergy sustainabilityrateacrosstime.Thenextstepisto evaluatetheenergyef iciencyoftheprotocolby calculatingtheaverageenergyconsumptionbynode oneachsimulationround.Finally,bycalculatingthe percentagedegradationinthroughputandPDRasthe numberofmaliciousnodesincreases,therobustness oftheprotocoltoattacksisassessed.Toevaluate theFL‑ERCFprotocolundervaryingoperational conditions,threesimulationscenarioswerecreated. Inthe irstscenario,10%ofthenodesaremalicious andtheprotocolisevaluatedforscalabilityand performanceatincreasingnodedensitiesbyvarying thenumberofIoTnodesfrom50to500.Inthe secondsimulationscenario,thenumberofnodesis heldconstant(300)andthepercentageofmalicious nodes(simulatingaformofSybilandselective packet‑dropping)wheremaliciousnodesarechanged from10%to20%,isvariedinordertoassessindetail howtheprotocoloperatedunderincreasinglevelsof securityrisks.Thethirdsimulationscenarioaimedto testthespeci icimpactsofFLbysimulatingthesame

networkineachscenariowithonenetworkusingFL withCHpredictionandtheotherusingstandardCH selectioncondition.The indingsfromthecomparison revealhowtheintegrationofdistributedintelligence enablesPDRtobeenhancedwhilereducingrouting overheadandimprovingenergyef iciency.Toinitiate thesimulationprocess,IoTnodesare irstdeployed inaspeci iedareaofsize 500m ×500m,with eachnodepresetwithaninitialenergylevel.Nodes evaluatethequalityoftheircommunicationlink (i.e.RSSI,SNR,andQIL)toquantifyasetofvalues aspartofthecommunicationlinkqualityphaseof PDR.Afterestablishingacommunicationlink,nodes derivetrustscorevaluesandlocalcostfunctions usingthesemetrics.Valuesderivedfromthese metricsserveasinputsforalightweightMLmodel whichistrainedlocallyateachnode.Insteadof transmittingrawdatatoacentralaggregator,nodes transmitonlythemodelparameterssecurely,and thecentralaggregatorcalculatestheglobalmodel usingFedAvg.Theglobalmodelisthensentbackto thenodesinordertopredictCHeligibilitythatis privacy‑consciousandadaptive.Duringsimulation, thisFLcycleisrepeatedatregulartimeintervals tofactorinnodeenergystatesandchangesto networktopology.Atthe inalstage,secureroutingis establishedbyusinganECDHbasedkeyexchangeand lightweightDC‑basedauthentication(DCESC).Once communicationshavebeenenabledwithencryption, sensornodestransferdatatothesinknodesafely viaCHs.Throughoutthesimulationprocessmetrics suchasthroughput,energyconsumption,routing overhead,andsuccessfuldeliverycharacteristicsare recordedcontinuously.Thedeploymentofmalicious nodesduringthesimulationandevaluationprocess enhancesourevaluationoftheprotocol’sresiliency, robustness,andperformanceconcerningother attacksbysimulatingreal‑worldsecuritythreats.

5.ResultsandDiscussion

ResultsofthesuggestedFL‑ERCFprotocolanda discussionofthoseresultsarepresentedinthissec‑ tion.Amongtheperformancemetricsconsideredto comparetheresultstotwobaselineprotocols,EESR andHSRarethroughput,PDR,routingoverhead,net‑ worklifetime,andenergyconsumption.Alsoevalu‑ atedinthesimulationsaretheeffectsofintegrating FLintheCHselectionprocessandhowresilientthe protocolistoadversarialthreats.

Figures3,4,and5illustratetheperformanceofFL‑ ERCF,EESR,andHSRundervaryingnodedensitiesof 50,150,250,350,450,and500.InFigure3,FLERCF consistentlyproducedgreaterthroughputthanEESR andHSR.Althoughallprotocolshadtheirthroughput curvesloweredbyincreasingnodedensity,FL‑ERCF keptitsthroughputcurverelativelysteady,andthese lowerthroughputrateswerecausedbyincreased channelcontentionandinterference.Amajorityofthe dropinthroughputcanbeattributedtotheintelligent clusterformationbasedoffoftheFLwhichprovided adaptiveroutingandoptimalloadbalancing.

Table1. SimulationEnvironment

Figure3. ThroughputofIoTnetworkwithdifferent countofIoTnodes

Figure4. PacketDeliveryRatio(PDR)(%)ofIoTnetwork withdifferentcountofloTnodes

FL‑ERCFmaintainsahigherPDRatallnodeden‑ sities,asshowninFigure 4.FL‑ERCFmaintainsa PDRofroughly 92% whenthenetworkreaches500 nodes,whereasHSRandEESRdecreaseto 83% and 78%,respectively.TheFLmodel,whichadjustsrout‑ ingpathsinresponsetochangingnetworkconditions, andthetrust‑awareCHselection,whichsteersclear ofunreliablenodes,isthetwomaindriversofthis improvement.

Figure 5 illustrateshowroutingoverheadraises withnodedensityacrossallprotocolsasaresultof morecontrolmessageexchanges.Incontrasttothe otherprotocols,FL‑ERCFshowsanoticeablylower overhead.ThisisbecausetheFLmodelmakespredic‑ tiveCHdecisions,whichreducestheneedforfrequent controlmessageexchangesandeliminatesneedless re‑clusteringandroutediscoveryprocesses.

Table 2 presentstheaveragethroughputtrend overvariednodedensities.FL‑ERCFperforms1825%

Figure5. RoutingOverheadofIoTnetworkwith differentcountofloTnodes

betterthanEESRand12−20%betterthanHSR,vali‑ datingthestrengthofitsadaptiveroutingmechanism.

Table2. AverageThroughput(%)underdifferentnode densities

Nodes FL‑ERCF HSR EESR p‑value (FL‑ERCF vsHSR) p‑value (FL‑ERCF vsEESR) 50 96.2 84.1 79.3 <0.01 <0.01 250 91.8 82.5 76.4 <0.01 <0.01

<0.01

Figure6. ThroughputofIoTnetworkwithdifferent countofattacks

Figures 6, 7,and 8 showtheperformancewhen theproportionofmaliciousnodesrisesfrom10%to

20%inordertoassessthesecurityandrobustnessof theprotocols.Thesenodesmimicselectiveforwarding andSybilattacks.Underattackconditions,allproto‑ cols’throughputdeteriorates,asshowninFigure 6. However,FL‑ERCFseestheleastamountofreduc‑ tion,about12%,whileHSRandEESRseereductions of 20% and 28%,respectively.TheTScomputation, whichsuccessfullyeliminateslow‑trustnodesfrom theCHselectionprocess,andtheDCbasedauthen‑ tication,whichstopsunwanteddataexchange,are responsibleforFL‑ERCF’ssuperiorperformance.

Figure7. RoutingoverheadofIoTnetworkwith differentcountofattacks

Figure8. PacketDeliveryRatio(PDR)(%)ofIoTnetwork withdifferentcountofattacks

Evenwhentheintensityoftheattackincreases,FL‑ ERCFmaintainsalowerroutingoverhead,asseenin

Figure7.ThereasonforthisisthatduringtheTSeval‑ uationphase,compromisednodesarequicklylocated andisolated,negatingtheneedforreroutingorre‑ clustering.Ontheotherhand,frequentpathrepairs broughtonbypacketdropsandtrustfailuresresultin signi icantoverheadincreasesforbothEESRandHSR. TherobustnessofFL‑ERCFisfurthercon irmedin Figure8,whichdisplaysaPDRreductionoflessthan 9%when20%ofnodesaremalicious.PDRdeclinesof 15% and 21%,respectively,areexperiencedbyHSR andEESRincontrast.CHscanadaptivelyrecon igure basedonlocalobservationsthankstoFL‑ERCF’sFL framework,improvingfaulttoleranceandminimizing dataloss.

Basedontheresults,theFL‑ERCFprotocolpro‑ videsanotableincreaseinnetworklifetimeatevery stageofoperation.TheFNDismuchlongercompared tothebenchmarkprotocolsEESRandHSRabout 28%longercomparedtoEESRand20%longercom‑ paredtoHSR.Theenergy‑awareCHselectionpro‑ cess,whichaimstoensurebalancedenergyuseof nodes,largelycontributedtothisresult.Furthermore, thefairloaddistributionviafederatedlearning,and loweroverheadduetocontrolmessageexchanges, worktoalsoextendoperation.Theenergyoptimiza‑ tionaspectofFL‑ERCFisparticularlyconsequential whentakingintoconsiderationlong‑termdeployment inlargescaleIoTnetworkswherebatteryreplacement isnotanoption.

Table 3 presentstheattack‑basedperformance trends,whereinFL‑ERCFoutperformsbaselinesinall caseswithstatisticalsigni icance.

Table3. Performanceunderattackconditions(20% maliciousnodes)

Thereisalsoconsiderableperformanceimprove‑ mentsassociatedwithFLintotheclusterformation procedure.TheFL‑assistedmethodproducesa 12% improvementinthePDRandreducesroutingover‑ headby 18% comparedtoavariantoftheprotocol thatusesstaticthreshold‑basedCHselection.Thisis largelyduetothepredictiveabilitiesoftheglobal federatedmodeltoavoidunnecessaryre‑clustering andselectingreliableandenergyef icientCHs.These resultsclearlydemonstratethatFLisveryeffective fordynamicandlarge‑scaleIoTscenariosbypreserv‑ ingnode‑levelprivacyandsimultaneouslyimproving routingef iciencyandenergymanagement.

Metric

OutsideofstaticIoTenvironments,thenew FLERCFprotocolishighlyviableinmobileroboticnet‑ workslikedroneswarmsandindustrialautonomous robots.Thesenetworkshaveanumberofthesame characteristicsasIoTdeployments,includingdynamic topologies,resourcelimitation,andtherequirement forlow‑latency,securecommunication.TheFLaspect ofFL‑ERCFcanbeusedtoforecaststableclusterheads inthemostmobileenvironmentsbyusingmobility attributeslikerelativevelocityandlinklifetimeas inputstothecostandtrustfunctions.Thisprovides adaptiveCHchangeoverevenwithhigh‑frequency topologychanges.Additionally,FL‑ERCF’slightweight cryptographicprimitivesenableittobeapplicablefor battery‑restrictedrobotsanddrones,andthetrust‑ basedsecuritylayercanresistattackslikecompro‑ misedorspoofedroboticnodes.Hence,FLERCFisnot restrictedtotraditionalIoTapplicationsbutcanalso improvemissionsuccess,energysavings,androbust‑ nessinroboticnetworksdeployedinindustrialor outdoorsettings.

6.ConclusionandFutureWork

Inthiswork,wepresentFL‑ERCF,anovelrouting protocoldevelopedtoaddressthefewbutessential requirementsofsafe,low‑cost,and lexiblecommu‑ nicationinIoTenvironments.InFL‑ERCF,thecon‑ ceptofathree‑phaseprocess,foridentifyingreliable communication,capturesourkeynotions.First,the evaluationofthequalityoflinks(linkquality),use ofmetricsbasedonRSSI,andSNRisperformedas a irstphaseforlinkqualityassessment.Thesecond phaseinvolvestheactualintelligentformationofclus‑ ters,guidedbyaFLmodelwhichis lexible,privacy‑ preserving,andallowsforCHselectionbasedonTSs, costsfunction,linkquality,andresidualenergy.The lastphaseguaranteessecuredatatransmissionlever‑ agingencryptionwithlightweightalgorithmssuchas ellipticcurvecryptologyandaDC‑basedauthenti‑ cationscheme.Aspartofacomprehensivesimula‑ tioncapabilityinsupportofthiswork,weundertook anexaminationofFL‑ERCFusingsimulationsbased ontheNetSimsimulator.TheFL‑ERCFperformance wascomparedagainsttheothertwoprotocolsintwo establishedareas,HSRandEESR,usingseveraldiffer‑ entsizednetworksandwithseveraldifferentattacks. TheresultsrevealedthatFLERCFoutperformingthe existingprotocolswasconsistentacrossallkeyperfor‑ manceindicatorsthroughput,PDR,routingoverhead, networklifetime,etc.Furthermore,byapplyingtheFL modelinCHselection,energyconsumptionwasbet‑ terbalanced,re‑clusteringoverheadwaslower,and the lexibilityofthenetworktoadapttothevarying conditionswasimprovedwithoutexposingrawnode data,andthereforeensuringprivacy.Theresistanceto Sybilandselectiveforwardingattackswasenhanced withtheinclusionofatrust‑basedattackdetection algorithm,andthelightweightcryptographicdesign allowedfordataintegrityandauthenticityatarela‑ tivelylowcomputationalcost.

Therearemanypossibilitiesforfollowingworkin thisareaofresearch.TheFL‑ERCFcouldbedeployed inactualphysicalIoTtestbedstoperformreal‑time validationinapplicationssuchassmartagriculture, smarthealthcare,orindustrialautomationinsubse‑ quentstudies.Inaddition,theFLaspectofFLERCF couldalsobeadvancedbymeansofmorecomplex approaches,suchasdeeplearningorreinforcement learningformoresophisticateddecision‑makingas therearestillchallengesrelatedtomobilenetworks anddelay‑tolerantnetworks.Additionally,theFLpro‑ cesscouldincorporateconceptslikesecureaggrega‑ tionprinciplesanddifferentialprivacymechanisms tostrengthendataprivacyguarantees.Overall,FL‑ ERCFcombinesarobustIoTmodelwhichincludes thefeaturesofdecentralizedintelligence,strongtrust management,morelightweightformsofencryption, whichprovidesanimplementable,scalableandsecure approachfornext‑generationIoTnetworks.

Apartfrommassive‑scaleIoTsettings,FL‑ERCF alsopromisestobedeployedinmobilerobotics likedronenetworksandautonomousfactoryrobots. Suchsystemsnecessitateadaptiveroutingbecauseof perpetualmobilityandneedsecuresecurityagainst spoo ingattacksaswellasselectiveforwarding attacks.Theadaptiveselectionofclusterheadsusing FL‑drivenadaptationinFL‑ERCFcanbeexpanded byconsideringmobilitymetrics,makingtheproto‑ colmoreeffectiveinsupportingdynamictopologies. FutureresearchcaninvolveapplyingFLframework tomobilityprediction,reducinghandoverlatency, andassessingperformanceoverreal‑worldrobotic testbeds.SuchanextensionmaymakeFL‑ERCFa general‑purposeprotocolthatcanbeusedwithboth stationaryIoTinfrastructuresandintenselydynamic roboticsettings.

AUTHORS

AnkurSisodia∗ –DepartmentofComputerScience EngineeringandIoT,NoidaInstituteofEngineering andTechnology,GreaterNoida,UttarPradesh,India, e‑mail:ankur22887@gmail.com.

SwatiVishnoi –DepartmentofComputerScienceand Engineering,NoidaInstituteofEngineeringandTech‑ nology,GreaterNoida,UttarPradesh,India,e‑mail: swativishnoi1@gmail.com.

ShivshankerSingh –DepartmentofComputerSci‑ enceandEngineering,GLBajajGroupofInstitu‑ tions,Mathura,UttarPradesh,India,e‑mail:shiv‑ shanker.singh@gmail.com.

NandiniSharma –DepartmentofComputerScience andEngineering,AnandSchoolofEngineering&Tech‑ nology,ShardaUniversity,Agra,UttarPradesh,India, e‑mail:nandini72@gmail.com.

AjayKumarYadav –comSchoolofCyberSecu‑ rityandDigitalForensics,NationalForensicSciences University,Bhopal,MadhyaPradesh,India,e‑mail: ajay.iitdhn@gmail.com.

∗Correspondingauthor

References

[1] Y.Lietal,”Energy‑awareEdgeAssociationfor Cluster‑BasedPersonalizedFederatedLearn‑ ing”,IEEETransactionsonVehicularTechnology, vol.71no.6,2022,pp.6756–6761;doi:10.110 9/TVT.2022.3161503.

[2] S.Sureshetal,”IntelligentDataRoutingStrategy BasedonFederatedDeepReinforcementLearn‑ ingforIOT‑EnabledWirelessSensorNetworks”, Measurement:Sensors,vol.31,no.101012, 2024.Doi:10.1016/j.measen.2023.101012.

[3] S.Lietal.,”TowardsEnhancedEnergyAware ResourceOptimizationforEdgeDevices ThroughMulti‑clusterCommunicationSystems”, JournalofGridComputing,vol.22,no.2,2024, p.56;doi:10.1007/s10723‑024‑09773‑3.

[4] N.Prabakaran,”OptimizedAdaptiveMulti‑Scale DualAnforMulti‑ObjectiveCHSandEner‑ gyAwareRoutingin6GWC”,IETEJournalof Research,vol.70,no.12,2024pp.8692–8701; doi:10.1080/03772063.2024.2387288.

[5] A.Dasetal.,”EnergyAwareDBSCANandMobil‑ ityAwareBalancedq‑LearningBasedOppor‑ tunisticRoutingProtocolinMANET”,Peer‑to‑ PeerNetworkingandApplications,vol.18,no. 4,2025,pp.1–19;doi:10.1007/s12083‑025‑ 02004‑w

[6] R.Alkanhel,”Dedg:Cluster‑basedDelayAnd Energy‑AwareDataGatheringin3d‑UwsnWith OptimalMovementOfMulti‑Auv”,Drones,vol.6, no.10,2022,p.283;doi:10.3390/drones6100 283.

[7] E.DritsasandM.Trigeka,”FederatedLearning forIoT:ASurveyofTechniques,Challenges,and Applications”,JournalofSensorandActuator Networks,vol.14,no.1,2025,p.9;doi:10.339 0/jsan14010009.

[8] E.C.PintoNetoetal.,”FederatedReinforcement LearninginIot:Applications,Opportunitiesand OpenChallenges”,AppliedSciences,vol.13,no. 11,2023,p.6497;doi:10.3390/app13116497.

[9] D.RupanettiandN.Kaabouch,”CombiningEdge Computing‑AssistedInternetofThingsSecurity withArti icialIntelligence:Applications,Chal‑ lenges,andOpportunities”,AppliedSciences,vol. 14,no.16,2024,p.7104;doi:10.3390/app141 67104.

[10] R.Kumaretal.,”FromEf iciencytoSustainabil‑ ity:ExploringthePotentialof6GforaGreener Future”,Sustainability,vol.15,no.23,2023,p. 16387;doi:10.3390/su152316387.

[11] A.Danilenka,”TacklingNon‑IIDDataAndData PoisoninginFederatedLearningusingAdversar‑ ialSyntheticData”,JournalofAutomationMobile

RoboticsandIntelligentSystems,vol. 18,2024; doi:10.14313/JAMRIS/3‑2024/17.

[12] A.Sisodiaetal,”Enhancingenergyef iciency:a protocolassessmentinmulti‑hopmesh‑based IOUTnetworks”,MultimediaToolsandApplica‑ tions,vol. 83,no.37,2024,p.p.8499985026;doi: 10.1007/s11042‑024‑19345‑y.

[13] K.Bogackaetal.,”GradientScaleMonitoringfor FederatedLearningSystems”,JournalofAutoma‑ tionMobileRoboticsandIntelligentSystems, vol. 18,2024;doi:10.14313/JAMRIS/32024/18.

[14] A.Sisodiaetal.,”ToBraceSociety5.0:Enhanced ReliabilitywithaCost‑EffectiveProtocolfor UnderwaterWirelessSensorNetwork”,Sustain‑ ableComputing:TransformingIndustry4.0to Society5.0,SpringerInternationalPublishing, 2023,pp.171–185;doi:10.1007/978‑3‑031‑ 13577‑4_10.

[15] A.Zouhri,”ANumericalAnalysisBasedInternet ofThings(IOT)andBigDataAnalyticstoMin‑ imizeEnergyConsumptioninSmartBuildings”, JournalofAutomationMobileRoboticsandIntel‑ ligentSystems,vol. 18,2024;doi:10.14313/JAM RIS/2‑2024/12.

[16] A.SisodiaandA.K.Yadav,”PerformanceAnaly‑ sisofIoTNetworksinTerrestrialEnvironment utilizingLZWDataCompressionTechnique”, ReviewofComputerEngineeringResearch,vol. 10,no.4,2023,pp.165–181;doi:10.18488/76. v10i4.3550..

[17] B.Kulecki,”MultimodalRobotProgramming InterfaceBasedonRGB‑DPerceptionandNeu‑ ralSceneUnderstandingModules”,Journalof Automation,MobileRoboticsandIntelligentSys‑ tems,2023,pp.29–37;doi:10.14313/JAMRIS/ 3‑2023/20.

[18] A.SisodiaandA.K.Yadav,”InternetofThings:A ComparativeAnalysisofNetworkTerminologies withandwithoutDataCompressionTechniques” 2022 11thInternationalConferenceonSystem Modeling&AdvancementinResearchTrends (SMART),2022,pp.236–242;doi:10.1109/SM ART55829.2022.10047640.

[19] M.A.Ortega‑Palacios,A.D.Palomino‑Merino,and F.Reyes‑Cortes,”InverseKinematicsModelfor a18DegreesofFreedomRobot”,Journalof Automation,MobileRoboticsandIntelligentSys‑ tems,2023,pp.22–29;doi:10.14313/t4yf9254.

[20] P.L.Wuetal.,”HybridNavigationofan AutonomousMobileRobottoDepressan ElevatorButton”,JournalofAutomation,Mobile RoboticsandIntelligentSystems,2022,pp. 2535;doi:10.14313/JAMRIS/4‑2022/30.

Submitted:22nd November2023;accepted:17th September2024

DheepikaP.S.,UmadeviV.

DOI:10.14313/jamris‐2026‐014

Abstract:

Objectdetectionisacrucialtaskforautonomousdriv‐ing,anddifferentautonomousvehicleshavevaryingper‐ceptions.Theadvancementsofobjectdetectionpaved thewayfor3Dobjectdetection,whichisconsideredto bethecentralcomponentofperceptionsystemsthat predictobstacles,vehicles,pedestriansandotherkey featuresoftheenvironmentalbackgorund.Generally, varioussensorsandcamerasareusedinautonomous drivingproducingaccurateperedictionofobjects.Several algorithmshavebeenemployedinobjectdetection,but theyhavenotproducedeffectiveoutcomes.

Thus,thepresentstudyimplementsHDL‐MODT (hybriddeeplearningbasedmulti‐objectdetectionand tracking)usingasensorfusionapproach.Itusessolid‐stateLiDAR,pseudo‐LiDARandanRGBcameratocapture objectsandprovideeffectivetrackingabilities.Initially, thepre‐processingmethodsinvolvednoiseremovalusing anA‐Fuzzy(adaptivefuzzy)filter.Contrastenhancement isthenperformedusingtheMSO(mothswarmopti‐mization)algorithm,andfeaturesegmentationisdone byLGAN(lightweightgeneraladversialnetworks),where bothchannelandpositionattentionmechanismsprovide precisesegmentation.TheYOLOv4approachisdeployed fordetectionofobjectssuchasground,vehicles,pedstri‐ansandobstacles.Finally,thetrackingofobjectsisper‐formedusingIUKF(improvedunscentedKalmanfilter). Thesimulationoftheproposedmethodisdemonstrated byusingMATLABR2020simulationtool;theperformance oftheproposedmethodisalsopredictedbycomparing theresultswithexisitngalgorithms.

Keywords: AutonomousDriving,ObjectDetection,Track‐ing,OneStageObjectDetector,TwoStageObjectDetec‐tor

1.Introduction

Withthedevelopmentofautonomousdriving, accurateobjectdetectionisahugechallengein decision‑makingcontrolstoensuresafedriving.

Operatingavehiclewithlittleornoeffortfrom humaninvolvementiscomplex,andthenumberof accidentshasrapidlyincreasedinrecentyears.This canbemitigatedusingobjectdetectiontechniques, whichtendtoperceivesurroundingpassengers,traf‑ iclights,vehiclesandevenunknownobjects.Object detectionisconsideredoneofthemostimportant tasksforavoidingtraf icaccidentsandavarietyof

otherhumanerrors[1].Automaticdrivingtechniques canimprovetheirsafetybyaccountingforhuman manipulationofthevehicleandpredictionofroad conditions.Obstaclesinautonomousdrivingmight includemotorcycles,bicycles,trucks,cars,pedestri‑ ansorotherobjectswithinvisualrange.Severalobject detectionapproacheshavebeenutilizedinthis ield ofimageprocessing,buttheyhavenotproduced ef icientfeaturerepresentationmethods.Moreover, objectdetectionalgorithmshavemostlyreliedon thedevelopmentofmanualfeaturesandtraditional approachesinvolvingDPM(deformablepartsmodel), HOG(histogramoforientedgradients)andothers[2]. Consequently,3Dobjectdetectionisconsideredtobe thedominantcomponentofperceptionsystemsinthe caseofautonomousdrivingthatprojectspointsonto asingleprescribedassessmentoffeaturelearning. Generally,3Dobjectdetectiontechniquessufferfrom inaccuratedepthestimation,whichcausesreduced accuracy.Inrecentyears,3Dobjectdetectionbased onstereomatching[3],LiDAR[4]andradarper‑ ception[5,6]haveachievedmoreimpressiveperfor‑ manceindetectionaccuracy.Tomitigatethelimita‑ tionsof3Dobjectdetection,aneffectivemethodology forautonomousdrivingshouldthusbedeveloped.

Concurrently,rapidadvancementsinAI(arti icial intelligence)computervisionhaveledtothedevelop‑ mentofef icientoutcomes.Hazarikaetal.[7]imple‑ mentedamulti‑camerasolutioninwhich3Dbounding boxesandweighted‑boxselectionmethodsareused forobjectdetection.ADCNN(deepconvolutionalneu‑ ralnetwork)isemployedtomap2Dboundingboxesto their3Dcounterpartsinordertoidentifyanobject’s dimensionandorientation.Additionally,aViT(Vision Transformer)isaddressedforaccuratelydetecting depthandocclusionusingaself‑attentivemodule.

Thecomprehensivesimulationisperformedafter fusingoutcomesfrommultiplecameras.KITTIstan‑ darddataisanalyzedusingexistingsimilarcamera‑ basedtechniques.

Similarly,MTL(multi‑tasklearning)playsasignif‑ icantroleinthegrowing ieldofautonomousvehi‑ cles.Orchestratingmultipletaskswithsensordatahas proventobeacomplextask,andMTLmeetsthese challengesbytrainingasinglemodeltodomultiple taskssimultaneously.Thestudyin[8]involvedascal‑ ableMTLforobjectdetection,whichisusedtodevelop anMTLnetworkwithvariedshapesandscales.

ItalsodeploystheextendedversionofMask‑RCNN (Region‑CNN)toovercomethelimitationoflearn‑ ingseveralobjectsinmulti‑labellearning.Perfor‑ mancewasevaluatedusingtheBerkeleyDeepDrive 100KBDD100kdataset,andtheoutcomesproduced bythestudyachievedamAP(meanaverageprecision) of50.

However,single‑stageapproachesexempli ied objectdetectionwithbetteraccuracyandpossessed theabilitytopredictboundingboxcoordinates andobjectclassesfromsingleevaluationofthe network.Thehybridapproachin[9]approach incorporatedbothFasterR‑CNNandYOLO,combining theboundary‑boxassortmentabilityofYOLOwith theRoI(regionofinterest)poolingofFasterR‑CNN. Analysisfoundthatthestudyproducedanimproved accuracyof74.3%,withaprocessingtimeof52ms. Despiteseveralbene its,imageresolutionandobject detectionofvaryingsizesunderchallengingdriving conditionsisanimportanttaskforsingle‑stage detectionalgorithms.

Toaddresstheissuespresentedbysingle‑stage approachesandlow‑resolutionimages,theproposed methodinvolves3Dmulti‑objectdetectionandtrack‑ ingforautonomousdriving.Thisapproachisintended toleveragethestrengthsofthesingle‑stageparadigm. TheYOLOv4approachisusedforfasterandmore accurate3Dmulti‑objectdetection.Theproposed methodincorporatesYOLOv4’seffectiveobjectdetec‑ tionforboundingboxassortmentforclassi ication. Initially,theimagesfromthe3DLiDARandstereo RGBcamerasareusedinpre‑processing,amethod thatincludestheremovalofnoiseusinganA‑Fuzzy ilter,followedbycontrastenhancementonnois‑ eremovedimagesusingMSO.Voxelizationisused enhancetheperceptivenessofsolidLiDARpoints. Bothcontrast‑enhancedimagesandvoxelizedpoint cloudsareintegratedtogeneratehighqualityimages. ThesegmentationprocesscombinesLGANwithpre‑ processedfusedimagesinordertoreducethecom‑ plexityofclassi icationandtracking.Thechanneland positionattentionmechanismareappliedfor,ensur‑ ingimprovedaccuracyinsegmentation.Inorderto reducecomplexity,VGG‑16isdeployedforbetterfea‑ tureextraction,inwhichfeaturevectorsareformed. Thesefeaturevectorsareselectedforobjectdetec‑ tionusingYOLOv4,whichcreatesfourclasses:ground, vehicles,pedestriansandobstacles.Afterclassi ica‑ tion,thedetectedobjectsaretrackedbyusingIUKF. Time‑basedmappingisperformedforvehiclesby consideringtheRFID,velocityandlocationofthe detectedobjects.TheMATLABR2020asimulationtool isusedtosimulatetheproposedmethodology.Fur‑ therassessmentoftheproposedalgorithmisper‑ formedusingexistingapproacheswithdifferentper‑ formancemetricstoexaminetheef icacyofthepro‑ posedsystem.

Themaincontributionsofthepaperareasfollows:

• Toapplypre‑processingtoinputimages,usingA‑ fuzzyfornoiseremovalandMSOalgorithmforcon‑ trastenhancement;

• ToprocesssegmentationusingLGANandfeature extractionusingVGG‑16algorithm,extractingfea‑ turevectorswhileavoidingsystemcomplexity;

• ToimplementobjectdetectionusingYOLOv4to classifyobjectssuchasground,vehicles,pedestri‑ ans,andobstacles;and

• Toassesstheef iciencyoftheproposedmodelby comparingitsoutcomeswiththosefromexisting methods.

1.1.PaperOrganization

Thepaperisorganizedbasedonthemostef i‑ cientmethodsdeployedintheobjectdetectionofvehi‑ cles.Itanalyzesconventionalmethodsusedinsimilar applicationswiththevaryingapproachesdiscussedin Section2.Theelaboratedprocedureoftheproposed methodisshowninSection 3.Theresultsobtained bytheimplementedapproacharethendeliberatedin Section 4.Finally,Section 5 presentstheconclusion, withsuggestionsforfutureworkusingtheproposed method.

2.RelatedWorks

Vision‑baseddrivingassistancesystemsmainly relyontheconceptofobjectdetection,whichhas becomeincreasinglyattractiveinsmarttransporta‑ tionsystems.However,itiscomplextoproduce energy‑ef icientandcost‑savingautonomousvehicle systems.So,[10]implementededge‑cloudassistance basedobjectdetection,alongwithareconstructive CNNknownasedgeYOLO.Thisstudyavoidedthe extremerelianceoncomputationalpowerandirreg‑ ularspreadofCC(cloudcomputing)assets,projecting alightweightobject‑detectionsystemthatcombined acompressionfeaturefusionnetworkwithapruning featureextractionnetworktoimprovetheef iciencyof multi‑scaleidenti ication.Anautomaticdrivingplat‑ formwithNVIDIAJetsonwasalsoinvolvedforsystem‑ levelevaluation.Ouranalysisfoundthatthestudypro‑ ducedmAP(accuracy)of47.3%ataspeedof26.6FPS (framespersecond).Thoughthestudygeneratedsat‑ isfactoryresults,thesystemreducesdownprocessing attheedge,andthetransmissionpressureproduced byCCincreaseslatency..Edgecloudssigni icantly reducecomputationalburdenandlatency;however, theirperformanceisaffectedbydynamicchannelcon‑ ditions.

Toaddresstheissueofedgeclouds,[11]deployed anEODF(edgenetwork‑assistedreal‑timeobject detectionframework).WhenusingtheEODFalgo‑ rithm,anautonomousvehiclemustcapturetheimage andextracttheRoIwhenthechannelqualityisnot goodenoughforreal‑timeobjectdetection.Forthis reason,theextractedimagestransferthecompressed RoItoedgeclouds,thusreducingthetransmission latencyofthesystem.Theresultsin[11]showed anaccuracyvalueof 82% forallcategoriessuchas tram,truck,vanandcar,andamAPof84%.However, increasesincompressionratiocauseddecreasesin informationlossanddegradationinobjectdetection performance.

Theautonomousdrivingsystemusesthesur‑ roundingenvironmenttomakedrivingdecisionsby modellingscenerydatagainedfromthesensors.The FPN(featurepyramidnetwork)generateshigh‑level semanticfeaturepyramidsandthusidenti iesobjects ofvaryingscales.Hence,[12]appliedenhancedFPN (EFPN)todevelopanadaptiveparalleldetection subnetandanimprovedfeatureextractionsubnet. TheadaptiveparalleldetectionsubnetemployedPDB (paralleldetectionbranch)andACE(adaptivecon‑ textexpansion).Meanwhile,toenhancethepyra‑ midfeatures,theenhancedfeatureextractionsub‑ netemployedanFWM(featureweightmodule).The studyevaluatedthesystemoncityscapes’datasets alongwiththeKITTIdataset,andproducedimproved objectdetection.However,theaverageprecisionof cityscapeshasbeenfoundtobelackingduetoan increaseinsmallandoccludedobjects.Smallobjects atthepixellevelhaveevenvanishedafterdifferent down‑samplings.Thecomplexityoftraf icenviron‑ ments,lightingsituationsandvariancesinimagequal‑ ityhavealsodecreasedprecision.

Similarly,theautomaticdrivingcanbeenabledby identifyingmissedandfalsedetectionsofsmalland occludedobjects.Hence,Zhouetal.[13]employed anAutomaticDrive‑Faster‑RCNNalgorithmforissues relatedtosmallscaleobjectdetection,densenessand occlusion.TheResnet‑50andpartialattentionmech‑ anismwereusedtoimprovethefeatureextraction ofsmallobjects,andthefeaturepyramidstructure hasalsobeenemployedtodecreasefeaturelossin thefusionprocess.Moreover,threecascadedetectors ‑namely,sideawareboundarylocalization,threshold mismatchandIOU(intersection‑over‑union)‑have beenadaptedtoperformframeregression.Thestudy producedbetteroutcomes,butwithshortcomings; thepositioninginformationisnottransmitted,and thesematicinformaticscontainedweredilutedduring informationfusion.

Thelowaccuracyandinterferencespeedinrecog‑ nitionofobjectsisconsideredasthemainhindrance inthedevelopmentofautomatedvehiclesystems. Forthisreason,[14]deployedMCS‑YOLO(multi‑scale smallobjectdetection),alongwithacoordinateatten‑ tionmechanism,todetectmulti‑scaledensesmall objects.Theattentionmechanismwasusedtoaggre‑ gatethecross‑channelinformationandfeaturemap’s spatialcoordinates.Additionally,aSwimTransformer structurewasutilizedtoincreasethefocusofthe networkoncontextualspatialinformation.Thestudy producedbetteroutcomes,butwasnotabletosolve issuesrelatedtoMOT(multipleobjecttracking).The end‑to‑enddelaysinobjectdetectioninreal‑time circumstancesalsopresentothersafetyconcernsin autonomousdriving.

In[15],threeoptimizationapproacheswere implemented:on‑demandcapture,zero‑slackpipeline andcontention‑freepipeline.TheDarknetYOLOwas usedtooptimizeobjectdetectiondependenton OpenCvlibraryforcapturingrealtimeimagesand achieveaqueueforbufferingimageframes.

ThestudywasexecutedandevaluatedinNvidia AGXXavierCPU‑GPUheterogeneouscomputingplat‑ form.Analysisindicatesthatthestudyhasgener‑ atedbetterresultswithlowerdetectionquality,but thesystemdidnotfocusonasensorfusionsystem forcomplexapplicationscenarios.Multi‑scaleobject detectionforautonomousdrivinghasbeenemployed usingaYOLOX‑basednetworkmodelundercom‑ plexscenarios.ACBAM‑G(channel‑basedattention module‑grouping)approachhasbeenintegratedto altertheheightandwidthoftheconvolutionalker‑ nelofthespatialattentionmodule[16].Anobject context‑basedfeaturefusionmodulehasbeenutilized toproduceadditionalsemanticdataandincreasethe observationofmulti‑scaleobjects.Theseexperiments, conductedonBDD100kandKITTIdatasets,produced bettermAPvalues.

However,thestudydidnotconcentrateonthe lightweightmulti‑scaleobjectdetectionthatmustbe appliedunderpracticalapplications.

Inautonomousdrivingapplications,Lidar‑point cloud‑based3Dobjectdetectionplaysasigni icant part,andhasprovenchallengingduetouneven distributionofdatapoints.Hence,[17]proposed atransformedapproachknownasTCT(tempo‑ ralchanneltransformer)todevelopthespatialtem‑ poralandchannel‑domainrelationships.Theinfor‑ mationencodedintheencoderwasvariedwith thedecoder,andthespatialdecoderofthetrans‑ formerdecodedtheinformationforeachlocationof thepresentframe.Incontrast,thetemporal‑channel encoderofthetransformerhasbeenspeci icallymod‑ elledtoencodethedataoftheframes,aswellas severalchannels.Thisstudyhasproducedbetterout‑ comes.

Toincreasethedetectionaccuracyandtherobust‑ nessoftheperceptionsysteminautonomousdriving systems,[18]imposedDMIQADNN(dual‑modalimage qualityawaredeepneuralnetwork).Ananalysisof theearly,middle,lateandscorestagesoffusionarchi‑ tectureswasperformedtopredictdetectionaccu‑ racyandspeed.AnIQAN(imagequalityassessment network)wasalsousedintheanalysisoftheRGB imagequalityscore.ThefusionweightsfortheLiDAR andRGBsub‑networkwereallocatedbyapplyinga fusionweightassignmentfunction.Further,thestudy foundscoresof27onamodi iedKITTIbenchmark and39.1onanAP(averageprecision)benchmark. LiDAR‑camera‑based3Dobjectdetectorswereused toextractthespeci icfeaturesandadjacent3Ddata knownaspointclouds.Thecameracapturedhigh‑ resolutionRGBimagesandcombinedthefeaturesof theRGBimagesandpointcloud.

Anearlyfusionmodulewasemployedtoexploit cameraandLiDARforfaster3Dobjectdetection[19]. Afeaturefusionmodelhasbeenutilizedtoextract point‑wisefeaturesfromrawRGBimagesandthen fusethemtotheequivalentpointclouds.Initially,the systemvoxelizesapointcloudinto3Dvoxelgrid,and thenusestwomethodstodecreaseofinformationloss whileperformingvoxelization.

TheresultshavebeenappliedinaKITTIbench‑ markdatasettoevaluatetheirspeedandaccuracy.

2.1.ProblemIdentification

Themainconcernsidenti iedthroughtheanalysis ofexistingalgorithmsareconferredinthissection.

• AhybridapproachincorporatingFasterRCNNand YOLO[9].Thisstudycouldbeimprovedbyutilizing largerandmorediversedatasetstopredicttheef i‑ cacyoftheapproachinvariousself‑directedvehi‑ cleplatforms.Enhancingtheexecutionanddiscov‑ eringinteractionswithotherAI‑basedtechniques canhelpoptimizetheef iciencyandsafetyofself‑ drivingvehicles.

• UsingEODFforobjectdetection,withcompressed imagestransmittedtoedgeclouds[11].However, thecompressionratioledtoinformationlossand reductioninobjectdetectionperformance.Thus,the compressionratioshouldbesettothemaximum valuetoevaluatetheaverageprecisionofthesys‑ tem.

3.SystemFramework

Increasedpotentialofprovidingroadsafety,effec‑ tivedecisionmakinganddecreasedtraf iccongestion canbeachievedbyusingautonomousdrivingsystems. Theymustobserve,recognize,design,adoptandper‑ formdecisionswithinanuncontrolled,complicated realworld.Thisisamorechallenginggoal,asasmall faultinunderstandingthesurroundingbackground anddecisionmayleadtomortaleffects.Areliableand effectiveautonomoussystemisthusrequiredtoelim‑ inateerrorsandmakecorrectdecisionsaccordingto changingsituations.Therecognitionsystem‑namely, the3Dobjectdetectionmethod‑supportsautomatic interpretationthevehicle’sdriving.The3Dsensor, knownasLiDAR(lightdetectionandranging),works withRGB‑Dcamerastoproduce3Dinformationabout theenvironment,suchasspeedanddistanceestima‑ tions.TheLiDARsensorisef icientunderavarietyof weatherconditions,butitsmajordrawbackisthatit tendstostruggleindetectingclose‑andfar‑distance objects.Differentapproacheshavebeendeveloped toimplementtheLiDARpointclouddata,butthey providelesstextureandcolorinformation.Toover‑ comethelimitationsofexistingmethods,thisstudy’s methodinvolvestheenhancementofinputimagesto provideeffectiveoutcomesin3Dobjectdetection.

3.1.DataCollection

Intheconceptof3Dobjectdetection,thedatasets arecategorizedas”indoors”and”outdoors”based ontheirapplications.Differentstudiesthatused 3Dobjectdetectionforintelligentdrivingsystems mostlyrelyonoutdoordatasets.Themostwellknown datasetsuseLiDARandRGBcamerastopredictvar‑ ioustypesofdata.Moreover,thesedatasetscontain amassof3Dannotationboundingboxes,multiple objectclassi ications,andselfdrivingscenes.Table 1 showsasummaryof3Dobjectdetectiondatasetsfor autonomousdrivingsystems.

ThestudyusestheKITTI(KarlsruheInstituteof TechnologyandToyotaTechnologicalInstitute)3D objectdetectionbenchmarkdatasetfortrainingand effectiveautonomousdrivingsystem.Itisconsidered asalargestcomputervisionalgorithmevaluation datasetforautomaticdrivingscenarios.Thedataset comprises52,979labelledobjects,ofwhich7,481 aretrainingand7,518testingimages.Alltheimages presentinthedatasetarecoloredandsavedin.png format.OneimageintheKITTIdatasetmightconsist ofupto30pedestriansand15cars.The3Dobject detectionsystemhaslabelsforninecategories:car, pedestrian,cyclist,van,truck,sittingperson,tram, miscellaneous,and”don’tcare”objects.

3.2.DataPre‐Processing

Datapre‑processingisthemethodofchangingthe rawdataintoacleandataset.Thisimportantpro‑ cesspreparesthedatainthemostmeaningfuland understandablewayforthemodeltoabletoeasily analyze.Dataarecleaned,transformedandintegrated inordertomakethemreadyforanalysis.Theinput datasetisalsopre‑processedtoverifymissingvalues, noisydataandotherinconsistenciesbeforeexecut‑ ingthealgorithm.Noisydataaretheresultofdata entryerrorsandfaultydatacollectionthatarerep‑ resentedasmeaninglessdata.Generally,noisydata canbehandledbyusingmethodssuchasclustering, regressionandbinning.Clusteringisatechniquein whichthesimilardatapointsaregroupedtogetherto form”clusters.”Themainobjectiveofclusteringisto predictpatternspresentinthedataandgroupbased onsimilardatapoints,andtoseparatedissimilardata pointsintodifferentgroups.Regression,ontheother hand,istheprocessofmakingdatasmoothinorder to itittoaregressionfunction.Theregressionused canbeeitherlinear,withoneindependentvariable,or multiple,withseveralindependentvariables.Binning isatechniquethatworksonsorteddatainwhichthe wholedatasetiscategorizedintosegmentsofequal size.Eachsegmentishandledseparatelyandcanbe replacedbyitsboundaryormeanvaluestocomplete aspeci ictask.

NoiseRemoval

ThisstudyusesRGB‑DimagesandLiDARcloud pointsforobjectdetection.Theseimagestendtogen‑ eratemorenoise,whichreducesthequalityofthe image.Here,pre‑processinginvolvesnoiseremoval frominputdatausingtheA‑Fuzzy(AdaptiveFuzzy ilter)performedundertwostages.TheA‑Fuzzyrules andmemberfunctionsareusedtodeterminewhether thepixelunderconsiderationisnoisy.Todifferentiate thenoisypixels,thedifferencebetweenthegradients arecalculated,anditisveri iedwhethertheyaresmall orlarge.Thea1,a2,a3,a4 arethefouradaptivefuzzy rulesandarede inedas:

��1 =Small(��1,��1,��2) Small(��2,��1,��2)

a2 =Small(F1,��1,��2).Large(F2,��1,��2)

��3 =Small(��1,��1,��2) Small(��2,��1,��2)

a4 =Small(F1,��1,��2).Large(F2,��1,��2)

Table1. Summaryof3DObjectDetectionDatasets

Thesetermsarerepresentedasadaptivefuzzysets. Theadaptivefuzzymembershipfunctions”Small”and ”Large”aregivenbytheequations(1)and(2):

Where ��1 and ��2 arerepresentedasthethreshold parameters.Afterapplyingtheadaptivefuzzyrules, theadaptivemembershipdegreesisdenotedasin equation(3),

��degree =Maximum(a1,a2,a3,a4) (3)

Themembershipdegreeplaysasigni icantroleinthe adaptive ilteringphase,inwhichpixelswithmember‑ shipdegree( ��degree =��1 arenoisypixels.When ��degree =��2 or ��3,itistreatedasanoisypixel thatis ilteredusingtheA‑fuzzytechnique.Pixelswith ��degree =��4 signifynoise‑freepixels.

ContrastEnhancement

Meta‑heuristic‑basedalgorithmsusedforcontrast enhancementyielddifferentpixelintensityredistri‑ butionpatternscomparedtotraditionalhistogram equalization(HE).Someimagecontrastenhancement techniquesincludeFFA( ire lyalgorithm),CS(cuckoo searchalgorithm),GA(geneticalgorithm),ABC(arti i‑ cialbeecolony)andothers.Thoughthesealgorithms haveproducedbetterresults,theyalsohavecertain laws,suchasaninabilitytomaintainpopulation diversityandprematureconvergencetolocaloptima. Theseeffectscannotbeimprovedunlessnoise,irrel‑ evantvisualinformationandsmallsetsofpixelsare removed.

TheproposedworkthusemploysanMSO(moth swarmoptimization)algorithmtoovercomethese impactsandtogeneratemoreeffectiveimagequality. TheMSOisameta‑heuristicalgorithminspiredby thealignmentofmothstowardsthemoonlightthat analysesthreeexplicitsubpopulations.Dependingon thesub‑population,theindividualperformsbyimple‑ mentingvaryingevolutionaryoperations.

Thisprocessisinspiredbythereal‑lifebehaviorof moths,andtheintegrationofoperatorsinthesearch methodtendstomitigatecriticalissueslikeprema‑ tureconvergenceandimproperexplorationexploita‑ tionbalance.Themothswarmalgorithminthestudy isusedtopredictthebestredistributiontoassemble theimprovedimage.Here,thesearchfeaturesofthe algorithmallowfordiscoveryofthesolutionspace.In MSO,theswarmpopulationissegregatedintothree parts:path inders,prospectorsandonlookers.The stepsinvolvedareasfollows:

Step1:Initialization

Intheinitializationprocess,therandompositions areassumedbysearchagentsbasedonthemodel giveninequation(3.2),

Where�� and�� signifythe lowerandupperlimitsofthesearchspace.

Step2:PredictionPhase

Twooperators‑crossoverandLevyperturbation ‑areemployedforeachindividualsolution.Thefor‑ mulaforthecrossoverpointisgivenbytheequa‑ tion(5),

Where �� isrepresentedasthetotalnumberof path indermoths.Thus,thenewsolutionisshownas inequation(6):

TheMSOalgorithmimpliesLevyperturbationsin ordertogeneraterandomsteps. �� istherandom samplegenerated.andiscalculatedasinequation(7),

Wherescaledenotesthedispersionsize,��[0,2], entry‑wisemultiplicationisperformedwith*,andthe twonormaldistributionsareu = M(0,��)anda = M(0,��).

Further,inadaptivecrossover,eachpath inder updatesitspositionbyintegratingthemutatedvari‑ ablesandcrossoveroperators.Aftercompletingadap‑ tivecrossover,the itnessvalueforentiretrailsolution isevaluatedandisequatedwithitscorresponding hostsolution.Additionally,asetofsolutionsarecho‑ senbasedontherouletteapproach.

Step3:TransversalFlight

Forthenextiterationlevel,aclusterofelements comprisingthebestluminescenceconcentrationsare representedasprospectors.Thenumberofprospec‑ torsisreducedduringtheoptimizationprocess.

Step4:CelestialNavigation

Withthereductioninthenumberofprospectors, theonlookersnumberincreases,whichcausesalarge increaseintheconvergencerate.Onlookersarecon‑ sideredtobethemothsproducinglessluminescent causesintheswarm.Inthecelestialnavigationstep, theonlookerssearch,followingtheprospectors.Then theonlookersarefurtherseparatedintotwophases, namelyGaussianwalksandALIM(associativelearn‑ ingmechanismswithimmediatememory).Then,the itnessoftheonlookersisevaluatedandglobalbest isupdateduntilthecriteriaarereached.Therefore, incontrastimprovementoftheinputimage,theMSO algorithmisdeployedtoadjustthepixelconcentra‑ tions,andthusthequalityoftheimageisenhanced.

Voxelizationisperformedtoenhancethepercep‑ tivenessoftheimage.Owingtotheincreasedvariable densityofLiDARpointclouds,itisacomplextask topredicttheinformationlossandperceptiveness, andachievebalanceabetweentheseandthespeedof pointcloudprocessing.Voxelizationisthetechnique inwhichthegroupingofpointsintovoxelsisper‑ formedbasedontheircorrespondingspatialproxim‑ ity.Thedepthofinformationintheresultingvoxelgrid isdetermineddependingonthevoxelsize.Withthe contrast‑improvedimage,voxelpointcloudsarethus integratedtogeneratehigh‑qualityimages.Afterthe pre‑processedfusedimagesareobtained,theinstance segmentationusingisperformedanLGANapproach toreducecomplexityofthetrackingandclassi ication.

3.3.Segmentation

InthecaseofLGAN‑basedinstancesegmentation, pre‑processedRGB‑Dimagesarecombinedwiththe voxelizedLiDARpointclouds.Theanglesoftheimages arevariedfromeachother,reducingdetectionaccu‑ racy;thus,theyarechangedto10∘ ,90∘,180∘ and270∘ toincreasetheirprecision.Afteralteringtheangles oftheimages,instancesegmentationisperformedfor fusedimagesusingtheLGANmethod.Thissegmenta‑ tionisamethodofinterpretingvisualdataassociated withanentitywhilealsoconsideringspatialinforma‑ tion.

Intheproposedstudy,LGANisusedtoperform instancesegmentation,anditcomprisestwomodels ‑ageneratorandadiscriminator.Here,thegenerator capturesthedistributionofinputdataandgenerates fakesamplesofdata.Thestudyimplementsthree losses:adversarialloss,L1 lossandJaccardloss.The adversariallossslowsdownthelearningprocess;L1 lossstorestheobjectboundaries;andJaccardloss enhancesthecorrelationbetweentheoriginaland segmentedimages.Thegeneratoristrainedwhilethe discriminatorisstable,andthepartsthattrainsthe generatorareasfollows:

• Inputwithnoisyvector

• Generatornetworkthattransformstherandom inputintodatainstances

• Discriminatornetworkthatclassi iesproduceddata

• Generatorloss

Ontheotherhand,thediscriminatorisanNN(neu‑ ralnetwork),whichisusedtopredictrealdatausing thefakedataproducedbythegenerator.Thediscrimi‑ natorconsistsoffourlayers:aconvolutional,position attention,channelattention,andanactivationlayer. Thetrainingdataforthediscriminatorisdoneusing:

• Realdatainstancesutilizedbythediscriminatoras positivesamplesduringtraining

• Fakedatainstancesproducedbythegenerator,sig‑ ni iedasnegativesamples

Theattentionlayersintheencoderanddecoder areusedtolearnbothlow‑andhigh‑levelfeatures.In thisway,thediscriminatorcategorizesrealandfake dataproducedfromthegeneratorbyeffectivelyusing thepositionandchannelattentionlayers.Thus,both generatoranddiscriminatoroperatesimultaneously tolearnandtraincomplexdata.

3.4.FeatureExtraction

ThesegmentedimageproducedfromtheLGAN isprocessedforfeatureextractiontoincreasedetec‑ tionaccuracy.Here,featureextractionisperformed usingtheVGG‑16approach.TheVGG16architectureis composedof41layers,ofwhich13areconvolutional layers,16areweightedlayers,andthreearefully‑ connectedlayers.Inputat224×224isprovidedwith RGBchannels.Toimproveoutcomes,thesizeofinput imageisreducedforeachpixel.Thealgorithmshows thestepsinvolvedinextractingfeaturesusingVGG‑16. UsingVGG‑16,high‑levelfeaturesareextracted, allowingformoreaccurateclassi icationofthe images.

3.5.ObjectClassificationandTracking

Afterthefeatureextractionphase,objectclas‑ si icationisperformedbyimplementingYOLOv4,a one‑stageobjectdetectionnetwork.YOLOv4uses anchorboxestodetectclassesofobjectsintheinput imageandidenti iesthreeattributes:(Intersection overUnion),anchorboxoffsets,andclassprobability. TheYOLOv4iscomposedofthreepartsbackbone, neckandhead‑asshowninFigure1.

AlgorithmI:FeatureExtractionusingVGG‑16

Input Trainingimages(T),Corresponding labels(L)pre‑trainedVGG16and Output Featuresareextractedfrominput images

ArrangetheVGG‑16toperformextractionof featuresfrominputimagebyeliminatingthe fullyconnectedlayers,module‑VGG‑16=VGG‑ 16‑FC

Forj=1toT: Readimagej

Theimagejisresizedto224×224×3Features extracted:(j)=moduleVGG16(j) Flatten(j)

Transformthefeaturesextracted(j)from3D featurestackto1DarrayFlatten(j)=Fea‑ tures(j)

Endfor

Figure1. ArchitectureofYOLOv4[20]

• Thebackboneispre‑trainedusingCSPDarkNet53 (CrossScalePartialDarkNetwork‑53)andactsas afeatureextractionnetwork,calculatingfeature mapsfrominputimages.Thebackboneconsists of iveresidualblockmodules,andthefeature mapoutcomesfromresidualblocksarecombined togetherintheneckpart.

• Theneckpartinterlinksthebackbonewiththehead ofthenetwork.ItconsistsofPAN(PathAggregation Network)andSPP(SpatialPyramidPooling).Itcon‑ catenatesthefeaturemapsfromseverallayersof backboneandtransfersthemasaninputtothehead network.

• Theheadoperatestheaccumulatedfeatures,and thusidenti iestheboundingboxesandobjectness scores,alongwithclassi icationscores.

ThelossofYOLOv4suchasobjectclassi ica‑ tion,localizationandoffsetloss,areevaluated,and lossfunctionispredicted.Hence,theuseofYOLOv4 impliesthatthelearningabilityofthenetworkis increasedandthustheclassi icationaccuracyhas beenoptimized.Inthisway,thetrackingofmov‑ ingobjectsisperformedafterclassi icationbasedon RFID,uniqueID,dimensionandorientation.TheIUKF (improvedunscentedKalman ilter)isusedtotrack thepositionandvelocityofthetargetandobjects.

Itisarecursivealgorithmusedtoestimatethe evolvingstateofprocesswhenmeasurementsare

done.Time‑basedmappingisemployedbyconsider‑ ingtheprecedingandpresenttimeandlocationfrom RFIDtoimprovetrackingreliability.

4.ComparativeAnalysisofAI‐BasedTech‐niquesInvolvedinObjectDetection

Visualobjectrecognitionisapplicableto autonomousvehicles,whichareabletosense andnavigatethesurroundingenvironmentwithout humaninvolvement.Predictionofobstaclesforsafe ridingisasigni icantchallengeinautomaticdriving systems.Theonlywaytoavoidroadaccidentsisto consciouslyrecognizeobstaclesandtraf iclights.

Thus,DL(deeplearning)‑basedobjectrecognition techniquesareusedtodetectobjectswithbetteraccu‑ racy.Theyareclassi iedintotwomajorclasses:single‑ stagearchitectureandtwo‑stagearchitecture.Two‑ stageobjectdetectorsusuallyachievebetterdetec‑ tionperformance,whilethesingle‑stagedetectorsare moreef icientandaresuitablefordetectingobjects withlimitedresources.Figure 2 showsthearchitec‑ tureofsingle‑stageandtwo‑stageobjectdetectors.

4.1.ObjectDetectionUsingSingle‐StageArchitecture (ThreeAlgorithms)

Thesingle‑stageobjectdetectorsrequireonlya singlepassthroughtheneuralnetwork(NN)and predictentireboundingboxesinoneattempt.This makestheirapproachtoworkfasterandincreases theirspeedofdetection.LiDAR‑based3Dobjectdetec‑ tionplaysacrucialroleinautonomousdriving;how‑ ever,itsperformancedegradesunderhighlysparse pointcloudconditions.In[21],a3Dobjectdetec‑ torwasimplementedbasedonvoxels,usingPV‑ SSD(projectiondoublebranchfeatureextraction) toreduceinformationloss.Voxelfeatureswerefed asinputcontaininglocalsemanticfeatures.Infea‑ tureextractionphase,thesesemanticfeatureswere combinedwithprojectedfeaturestodecreasethe localinformationlossfrompointcloudprojection.A featurepoint‑samplingalgorithmwithweightsam‑ plingwasdeployedto indthefeaturepointsthat weremostbene icialforthedetectionprocess.The MSSFA(multi‑layerspatialsemanticfeatureaggrega‑ tion)methodwasappliedtoprovidebetterdetection accuracy.

Figure2. ArchitectureofObjectDetectors

Theoutcomesgeneratedbythestudywereevalu‑ atedinKITTIdatasetandwerefoundtoproducebet‑ terresults.WhencomparedwiththeproposedHDL‑ MODTapproach,thismethodisadvantageousinterms ofdetectionaccuracyandtrackingreliability.The studydidnot,however,involvethetrackingofmoving targetobjects,whichwouldshowtheimprovements inourproposedmodel.

AnothermethodofobjectdetectionusingYOLOv4 wasappliedusingthebackbonenetworkCSPDark‑ net53_dcn(P)[22].Featurefusionwasperformed usingPAN++,alongwith ivescaledetectionlayersto enhancedetectionaccuracyforsmallobjects.Thelast layerofCSPDarknet53wasreplacedwithdeformable convolution,andapruningapproachwasintroduced inthestudytoresolveissuescreatedduringthereal‑ timeperformanceofthesystem.Asparsescalingfac‑ torisimpliedinordertoovercometheseissues,as theparametersigni icanceevaluationtechniquecan‑ notdifferentiatetheimportanceoftheconvolution kernelinalarge‑redundancyconvolutionallayer.The BN(batchnormalization)layeravoidsthe”internal covariateshift”issue,inwhichtheactivationinput valueisalwaysmaintainedwithaspeci icdistance fromthederivativesaturationareaandinsensitive areas.Thisresolvesthevanishinggradientproblem occurringinthebackpropagation.

4.2.ObjectDetectionUsingTwo‐StageArchitecture (ThreeAlgorithms)

Distantobjectpredictioninautonomousdriving canbeaddressedbyusingFasterRCNNalgorithm [23].WiththedevelopmentofDCNN,vision‑based vehicledetectionhasachievedsigni icantenhance‑ mentscomparedtotraditionalmethods.However, heavyocclusionandlarge‑scalevehiclevariationhas ledtolimitationsinDCNN’sperformance.Theemer‑ genceofFasterRCNNthushasallowedforfaster vehicledetectionwithanimprovedframework.Ini‑ tially,theMobileNetstructurewasdevelopedtocon‑ structthe irstleveloftheconvolutionalnetworkin FasterRCNN.TheNMS(non‑maximumsuppression) approachisthenappliedsubsequentlytotheregion proposalnetwork,andtheFasterRCNNissubstituted usingSoft‑NMSinordertoresolvefakeproposals. Then,thecontext‑awareRolpoolinglayerisconsid‑ eredtoregulatetheproposalsintoitsspeci icsize withouteliminatingtheessentialcontextualdata.To constructthe inalphaseofFasterRCNN,theassembly ofdepth‑wiseseparableconvolutionintheMobileNet designisemployedtodeveloptheclassi ier.Then,pro‑ posalsareclassi iedandtheboundingboxisadjusted foreachofthedetectedvehicles.Theexperimental outcomesproducedintheLSVHdatasetandKITTI datasetprovethatthismethodattainedbetterperfor‑ manceintermsofbothinferencetimeanddetection accuracy.

BothADS(advanceddriverassistance)andthe ADAS(advanceddriverassistancesystem)require effectivedetectionoftraf icsigns.ThoughtheFPN

(featurepyramidnetwork)hasobtainedbetterout‑ comes,[24]employedplug‑and‑playnecknetwork IFA‑FPN(integratedFPNwithfeatureaggregation).

At irst,thelightoperationisappliedtoef iciently usethesystemandenhancetheinferencespeedof thesystem.ThenIO(integratedoperation)isdeployed toresolvetheimbalanceissueswiththeRoIinpyra‑ midstages.FAisthenintroducedtoimprovethefea‑ turerepresentationcapabilityofthefeaturemaps, andthisoptimizestherobustnessofthenetwork. Tosignifydatawithlargevariancesinsize,theFA structureisapplied,aggregatingthemulti‑scalefea‑ turestoproducefeatureswithhighrepresentational ability.Theexperimentalresultsofthestudyhave projectedthatthesystemwillproducebetterout‑ comeswhenappliedintheTT100k(Tsinghua‑Tencent 100k),STSD(SwedishTraf icSignDataset)andGTSDB (GermanTraf icSignDetectionBenchmark)datasets, butwouldlackimprovementindetectionaccuracy andef iciency.

Thecomparisonofone‑stageandtwo‑stagedetec‑ torsdenotesthatthetwo‑stagedetectorsusethe proposalgeneratortogenerateasparsesetofpro‑ posals,thenextractthefeaturesfromeachproposal. Thisisfollowedbyregionclassi ierspredictingthe categoryoftheproposalregion.One‑stagedetectors directlyperformcategoricalpredictionofobjectson eachlocationoffeaturemapswithouttheuseofa cascadedregionclassi icationphase.Thisshowsthat theone‑stageobjectdetectorsareeffectiveforvisu‑ alizingobjectswithoptimizeddetectionaccuracyand reducedcomputationaltime.

5.ResultsandDiscussion

Todeterminetheperformanceoftheproposed approach,theoutcomesproducedbytheproposed studyarecomparedwiththosefromexisting approaches.Table 1 representstheassessmentof proposedmethodalongsideconventionalalgorithms.

TheconventionalstudyemployedaYOLONMS fuzzyalgorithmtosimulatethedriver’sreactionto obstacleswithimprovedspeedandaccuracy.Object detectionandtrackingwasaccomplishedusinga hybridofthefuzzyandNMSalgorithms.Theperfor‑ manceofthesystemisexaminedusingKITTIdataset. Accordingtothecomparisonofthealgorithmsshown inTable 1,however,abetteroutcomeisgained throughtheproposedYOLOv4model,withanAPof 98% andanMAPof 95%.Thisindicatesincreased speedanddetectionaccuracyofobjectswithopti‑ mizedtimeef iciency.

TheexistingmethodusedaDLalgorithmforactive sensorfusionofthevisiblecamerawithcorrespond‑ ingsensorsforautonomousdriving.Askipconnection allowedafeature‑levelsensorfusionapproachtobe applied,alongwithathermalcameraandmillimeter‑ waveradar.TwonetworkscalledTV‑NetandRV‑Net wereemployedforperformingsensorobjectdetec‑ tion,featurelevelfusionandspeci icfeatureextrac‑ tion.

Table2. ComparisonofProposedModelwithExistingModels[25]

Table3. ComparisonofAveragePrecisionofProposed ApproachwithConventionalAlgorithms[26]

ofdifferentobjectstopromotesaferautonomousdriv‑ ing.

7.Declaration

• Con lictofInterest: Theauthorreportsthereisno con lictofinterest.

• Funding: None

• Acknowledgement: None

Table 2 showstheaverageprecisionlevelspro‑ ducedbytheexistingalgorithmstinyYOLOv3,late fusionandRVNet,whichare0.40,0.40and0.56,with computationaltimesof10,14and17ms,respectively. Theproposedmethod,ontheotherhand,achievedan averageprecisionof0.74withacomputationaltimeof 14.85ms.Thissubstantiatesthebetterperformance ofproposedapproachwhencomparedwithconven‑ tionalmethods.

6.ConclusionandReflectionsonFutureWork

Objectdetectionapproacheswithimprovedspeed andaccuracyareessentialforreal‑timecontrolof automaticvehicles.Variousapproacheshaveinves‑ tigatedobjectdetectionusingdifferenttechniques, buttheywereunabletorectifytheissueofbal‑ ancingspeedwithaccuracyofdetection.Toaddress theseproblems,thepresentstudyimplementedHDL‑ MODTtoeffectivelydetectandclassifyobjectssuch asground,vehicles,pedestrians,andobstacles.Noise removalandcontrastenhancementwasperformed byA‑FuzzyandMSOtechniques,whichreducedthe complexityofthesystem.Thesegmentationprocess involvedininstancesegmentationofthefeatureswas thenusedtoincreasethedetectionaccuracyofthesys‑ tem.VGG‑16wasusedforfeatureextraction,andthe extractedfeaturevectorswereusedforobjectdetec‑ tionwiththeYOLOv4algorithm.Detectedobjects weretrackedusingIUKF,andthemappingofvehicles wasexhibited,takingintoaccounttheRFID,velocity andlocationofobjects.Thisworkwasappliedinthe MATLABR2020simulationtooltoobtaintheresults, whichshowedimprovedoutcomes.Toevaluateper‑ formance,wealsomadeacomparativeanalysisof existingstudieswithproposedmethod.Theresults projectedthattheproposedmethodoutperformed otherexistingalgorithmsinbothaccuracyandspeed. Inthefuture,thisproposedworkcouldbeextended byusinganimprovedversionofYOLOwithdetection

AUTHORS

DheepikaP.S.∗ –DepartmentofComputerScience, TheAmericanCollege,MaduraiKamarajUniversity, Madurai,India,e‑mail:psdheepika@gmail.com. UmadeviV. –DepartmentofComputerScience, NehruMemorialCollege,BharathidasanUniversity, Tiruchirapalli,India,e‑mail:umadevi@gmail.com.

∗Correspondingauthor

References

[1] J.Mao,S.Shi,X.Wang,andH.Li,”3DObjectDetec‑ tionforAutonomousDriving:AComprehensive Survey,” InternationalJournalofComputerVision, pp.1–55,2023.

[2] Z.Wei,etal.,”MmwaveRadarAndVisionFusion ForObjectDetectionInAutonomousDriving:A Review,” Sensors,vol.22,p.2542,2022.

[3] B.Pan,L.Zhang,andH.Wang,”Multi‑Stage FeaturePyramidStereoNetwork‑BasedDispar‑ ityEstimationApproachForTwoToThree‑ DimensionalVideoConversion,” IEEETransac‑ tionsonCircuitsandSystemsforVideoTechnol‑ ogy,vol.31,pp.1862–1875,2020.

[4] Y.Zhang,etal.,”PMPF:Point‑CloudMultiple‑ PixelFusion‑Based3DObjectDetectionfor AutonomousDriving,” RemoteSensing,vol.15,p. 1580,2023.

[5] D.Feng,etal.,”DeepMulti‑ModalObject DetectionAndSemanticSegmentationfor AutonomousDriving:Datasets,Methods,and Challenges,” IEEETransactionsonIntelligent TransportationSystems,vol.22,pp.1341–1360, 2020.

[6] S.Yao,etal.,”Radar‑CameraFusionForObject DetectionandSemanticSegmentationin AutonomousDriving:AComprehensiveReview,” arXivpreprintarXiv:2304.10410,2023.

[7] A.Hazarika,etal.,”Multi‑camera3DObject DetectionforAutonomousDrivingUsingDeep LearningandSelf‑AttentionMechanism,” IEEE Access,2023.

[8] S.Rinchen,B.Vaidya,andH.T.Mouftah,”Scalable Multi‑TaskLearningR‑CNNforObjectDetection inAutonomousDriving.” In2023International WirelessCommunicationsandMobileComputing (IWCMC),Marrakesh,19‑23June2023;pp.518–523.

[9] S.A.Khan,H.J.Lee,andH.Lim,”Enhancing ObjectDetectioninSelf‑DrivingCarsUsinga HybridApproach,” Electronics,vol.12,p.2768, 2023.

[10] S.Liang,etal.,”EdgeYOLO:Real‑TimeIntelli‑ gentObjectDetectionSystemBasedonEdge‑ CloudCooperationinAutonomousVehicles,” IEEETransactionsonIntelligentTransportation Systems,vol.23,pp.25345–25360,2022.

[11] S.‑W.Kim,etal.,”Edge‑Network‑Assisted Real‑TimeObjectDetectionFrameworkfor AutonomousDriving,” IEEENetwork,vol.35,pp. 177–183,2021.

[12] Y.Wu,etal.,”AnEnhancedFeaturePyramid ObjectDetectionNetworkforAutonomousDriv‑ ing,” AppliedSciences,vol.9,p.4363,2019.

[13] Y.Zhou,etal.,”ObjectDetectioninAutonomous DrivingScenariosBasedOnAnImprovedFaster‑ RCNN,” AppliedSciences,vol.11,p.11630,2021.

[14] Y.Cao,C.Li,Y.Peng,andH.Ru,”MCSYOLO:AMul‑ tiscaleObjectDetectionMethodforAutonomous DrivingRoadEnvironmentRecognition,” IEEE Access,vol.11,pp.22342–22354,2023.

[15] W.Jang,etal.,”R‑TOD:Real‑timeObject DetectorWithMinimizedEnd‑To‑EndDelayfor AutonomousDriving,” in2020IEEEReal‑Time SystemsSymposium(RTSS),1‑4th December 2020,pp.191–204.

[16] S.Wu,Y.Yan,andW.Wang,”CF‑YOLOX:An AutonomousDrivingDetectionModelforMulti‑ ScaleObjectDetection,” Sensors,vol.23,p.3794, 2023.

[17] Z.Yuan,etal.,”Temporal‑ChannelTransformer for3DLidar‑BasedVideoObjectDetectionfor AutonomousDriving,” IEEETransactionsonCir‑

cuitsandSystemsforVideoTechnology,vol.32, pp.2068–2078,2021.

[18] K.Geng,G.Dong,andW.Huang,”RobustDual‑ ModalImageQualityAssessmentAwareDeep LearningNetworkforTraf icTargetsDetection OfAutonomousVehicles,” MultimediaToolsand Applications,vol.81,pp.6801–6826,2022.

[19] L.‑H.WenandK.‑H.Jo,”FastandAccurate 3DObjectDetectionForLidar‑CameraBased AutonomousVehiclesUsingOneSharedVoxel‑ BasedBackbone,” IEEEAccess,vol.9,pp.22080–22089,2021.

[20] K.Roszyk,M.R.Nowicki,andP.Skrzypczyński, ”AdoptingtheYolov4ArchitectureforLow‑ LatencyMultispectralPedestrianDetectionin AutonomousDriving,” Sensors,vol.22,p.1082, 2022.

[21] Y.Shao,etal.,”PV‑SSD:AProjectionandVoxel‑ basedDoubleBranchSingle‑Stage3DObject Detector,” arXivpreprintarXiv:2308.06791, 2023.

[22] Y.Cai,etal.,”YOLOv4‑5D:AnEffectiveAndEf i‑ cientObjectDetectorforAutonomousDriving,” IEEETransactionsonInstrumentationandMea‑ surement,vol.70,pp.1–13,2021.

[23] H.Nguyen,”ImprovingFasterR‑CNNFrame‑ workforFastVehicleDetection,” Mathematical ProblemsinEngineering,vol.2019,pp.1–11, 2019.

[24] Q.Tang,G.Cao,andK.‑H.Jo,”IntegratedFeature PyramidNetworkwithFeatureAggregationfor Traf icSignDetection,” IEEEAccess,vol.9,pp. 117784–117794,2021.

[25] N.Zaghari,etal.,”TheImprovementinObsta‑ cleDetectioninAutonomousVehiclesUsing YOLONon‑MaximumSuppressionFuzzyAlgo‑ rithm,” TheJournalofSupercomputing,vol.77, pp.13421–13446,2021.

[26] V.JohnandS.Mita,”Deepfeature‑levelSensor FusionUsingSkipConnectionsforReal‑Time ObjectDetectioninAutonomousDriving,” Elec‑ tronics,vol.10,p.424,2021.