HandbookofHuman-MachineSystems
Editedby
GiancarloFortino UniversityofCalabria Italy
DavidKaber UniversityofFlorida USA
AndreasNürnberger
Otto-von-Guericke-UniversitätMagdeburg Germany
DavidMendonça MITRECorporation USA
Copyright©2023byTheInstituteofElectricalandElectronicsEngineers,Inc.Allrightsreserved.
PublishedbyJohnWiley&Sons,Inc.,Hoboken,NewJersey. PublishedsimultaneouslyinCanada.
Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyany means,electronic,mechanical,photocopying,recording,scanning,orotherwise,exceptaspermittedunderSection 107or108ofthe1976UnitedStatesCopyrightAct,withouteitherthepriorwrittenpermissionofthePublisher,or authorizationthroughpaymentoftheappropriateper-copyfeetotheCopyrightClearanceCenter,Inc.,222 RosewoodDrive,Danvers,MA01923,(978)750-8400,fax(978)750-4470,oronthewebatwww.copyright.com. RequeststothePublisherforpermissionshouldbeaddressedtothePermissionsDepartment,JohnWiley&Sons, Inc.,111RiverStreet,Hoboken,NJ07030,(201)748-6011,fax(201)748-6008,oronlineathttp://www.wiley.com/ go/permission.
Trademarks:WileyandtheWileylogoaretrademarksorregisteredtrademarksofJohnWiley&Sons,Inc.and/or itsaffiliatesintheUnitedStatesandothercountriesandmaynotbeusedwithoutwrittenpermission.Allother trademarksarethepropertyoftheirrespectiveowners.JohnWiley&Sons,Inc.isnotassociatedwithanyproduct orvendormentionedinthisbook.
LimitofLiability/DisclaimerofWarranty:Whilethepublisherandauthorhaveusedtheirbesteffortsinpreparing thisbook,theymakenorepresentationsorwarrantieswithrespecttotheaccuracyorcompletenessofthecontents ofthisbookandspecificallydisclaimanyimpliedwarrantiesofmerchantabilityorfitnessforaparticularpurpose. Nowarrantymaybecreatedorextendedbysalesrepresentativesorwrittensalesmaterials.Theadviceandstrategies containedhereinmaynotbesuitableforyoursituation.Youshouldconsultwithaprofessionalwhereappropriate. Further,readersshouldbeawarethatwebsiteslistedinthisworkmayhavechangedordisappearedbetweenwhen thisworkwaswrittenandwhenitisread.Neitherthepublishernorauthorsshallbeliableforanylossofprofitor anyothercommercialdamages,includingbutnotlimitedtospecial,incidental,consequential,orotherdamages.
Forgeneralinformationonourotherproductsandservicesorfortechnicalsupport,pleasecontactourCustomer CareDepartmentwithintheUnitedStatesat(800)762-2974,outsidetheUnitedStatesat(317)572-3993orfax(317) 572-4002.
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprintmaynotbe availableinelectronicformats.FormoreinformationaboutWileyproducts,visitourwebsiteatwww.wiley.com.
LibraryofCongressCataloging-in-PublicationDataappliedfor:
HardbackISBN:9781119863632
CoverDesign:Wiley
CoverImages:©bergamont/Shutterstock;SviatlanaLaza/Shutterstock;barleyman/GettyImages;3alexd/Getty Images;Olemedia/GettyImages;Westend61/GettyImages;3alexd/GettyImages;MontyRakusen/GettyImages; WitthayaPrasongsin/GettyImages;LucaSage/GettyImages;NisianHughes/GettyImages;SerhiiBrovko/ GettyImages
Setin9.5/12.5ptSTIXTwoTextbyStraive,Chennai,India
Contents
EditorsBiography xxi
ListofContributors xxiii
Preface xxxiii
1Introduction 1 GiancarloFortino,DavidKaber,AndreasNürnberger,andDavidMendonça
1.1BookRationale 1
1.2ChaptersOverview 2 Acknowledgments 8 References 8
2Brain–ComputerInterfaces:RecentAdvances,Challenges,andFuture Directions 11 TiagoH.Falk,ChristophGuger,andIvanVolosyak
2.1Introduction 11
2.2Background 12
2.2.1Active/ReactiveBCIs 13
2.2.2PassiveBCIs 14
2.2.3HybridBCIs 15
2.3RecentAdvancesandApplications 15
2.3.1Active/ReactiveBCIs 15
2.3.2PassiveBCIs 16
2.3.3HybridBCIs 16
2.4FutureResearchChallenges 16
2.4.1CurrentResearchIssues 17
2.4.2FutureResearchDirections 17
2.5Conclusions 18 References 18
3Brain–ComputerInterfacesforAffectiveNeurofeedbackApplications 23 LucasR.TrambaiolliandTiagoH.Falk
3.1Introduction 23
3.2Background 23
3.3State-of-the-Art 24
3.3.1DepressiveDisorder 25
3.3.2PosttraumaticStressDisorder,PTSD 26
3.4FutureResearchChallenges 27
3.4.1OpenChallenges 27
3.4.2FutureDirections 28
3.5Conclusion 28
References 29
4PediatricBrain–ComputerInterfaces:AnUnmetNeed 35
EliKinney-Lang,EricaD.Floreani,NiloufaralsadatHashemi,DionKelly,StefanieS.Bradley, ChristineHorner,BrianIrvine,ZeannaJadavji,DanetteRowley,IlyasSadybekov,SiLong JennyTou,EphremZewdie,TomChau,andAdamKirton
4.1Introduction 35
4.1.1Motivation 36
4.2Background 36
4.2.1ComponentsofaBCI 36
4.2.1.1SignalAcquisition 36
4.2.1.2SignalProcessing 36
4.2.1.3Feedback 36
4.2.1.4Paradigms 37
4.2.2BrainAnatomyandPhysiology 37
4.2.3DevelopmentalNeurophysiology 38
4.2.4ClinicalTranslationofBCI 38
4.2.4.1AssistiveTechnology(AT) 38
4.2.4.2ClinicalAssessment 39
4.3CurrentBodyofKnowledge 39
4.4ConsiderationsforPediatricBCI 40
4.4.1DevelopmentalImpactonEEG-basedBCI 40
4.4.2HardwareforPediatricBCI 41
4.4.3SignalProcessingforPediatricBCI 41
4.4.3.1FeatureExtraction,SelectionandClassification 42
4.4.3.2EmergingTechniques 42
4.4.4DesigningExperimentsforPediatricBCI 43
4.4.5MeaningfulApplicationsforPediatricBCI 43
4.4.6ClinicalTranslationofPediatricBCI 44
4.5Conclusions 44 References 45
5Brain–ComputerInterface-basedPredator–PreyDroneInteractions 49 AbdelkaderNasreddineBelkacemandAbderrahmaneLakas
5.1Introduction 49
5.2RelatedWork 50
5.3Predator–PreyDroneInteraction 51
5.4ConclusionandFutureChallenges 57 References 58
6LevelsofCooperationinHuman–MachineSystems:AHuman–BCI–Robot Example 61 Marie-PierrePacaux-Lemoine,LydiaHabib,andTomCarlson
6.1Introduction 61
6.2LevelsofCooperation 61
6.3ApplicationtotheControlofaRobotbyThought 63
6.3.1DesigningtheSystem 64
6.3.2ExperimentsandResults 66
6.4ResultsfromtheMethodologicalPointofView 67
6.5ConclusionandPerspectives 68 References 69
7Human–MachineSocialSystems:TestandValidationviaMilitary UseCases 71
CharleneK.Stokes,MonikaLohani,ArwenH.DeCostanza,andElliotLoh
7.1Introduction 71
7.2BackgroundSummary:FromToolstoTeammates 72
7.2.1TwoSidesoftheEquation 72
7.2.2MovingBeyondtheCognitiveRevolution 73
7.2.2.1ARediscoveryoftheUnconscious 74
7.3FutureResearchDirections 75
7.3.1Machine:FunctionalDesigns 75
7.3.2Human:GroundTruth 76
7.3.2.1PhysiologicalComputing 76
7.3.3Context:TyingItAllTogether 77
7.3.3.1TrainingandTeamModels 77
7.4Conclusion 79 References 79
8TheRoleofMultimodalDataforModelingCommunicationinArtificial SocialAgents 83
StephanieGrossandBrigitteKrenn
8.1Introduction 83
8.2Background 84
8.2.1Context 84
8.2.2BasicDefinitions 84
8.3RelatedWork 84
8.3.1HHIData 85
8.3.2HRIData 85
8.3.2.1JointAttentionandRobotTurn-TakingCapabilities 85
8.3.3PublicAvailabilityoftheData 87
8.4DatasetsandResultingImplications 87
8.4.1HumanCommunicativeSignals 87
8.4.1.1ExperimentalSetup 87
8.4.1.2DataAnalysisandResults 88
8.4.2HumansReactingtoRobotSignals 89
8.4.2.1ComparingDifferentRoboticTurn-GivingSignals 89
8.4.2.2ComparingDifferentTransparencyMechanisms 90
8.5Conclusions 91
8.6FutureResearchChallenges 91 References 91
9ModelingInteractionsHappeninginPeople-DrivenCollaborative Processes 95
MaximilianoCanche,SergioF.Ochoa,DanielPerovich,andRodrigoSantos
9.1Introduction 95
9.2Background 97
9.3State-of-the-ArtinInteractionModelingLanguagesandNotations 98
9.3.1VisualLanguagesandNotations 99
9.3.2ComparisonofInteractionModelingLanguagesandNotations 100
9.4ChallengesandFutureResearchDirections 101 References 102
10TransparentCommunicationsforHuman–MachineTeaming 105
JessieY.C.Chen
10.1Introduction 105
10.2DefinitionsandFrameworks 105
10.3ImplementationofTransparentHuman–MachineInterfacesinIntelligent Systems 106
10.3.1Human–RobotInteraction 106
10.3.2MultiagentSystemsandHuman–SwarmInteraction 108
10.3.3Automated/AutonomousDriving 109
10.3.4ExplainableAI-BasedSystems 109
10.3.5GuidelinesandAssessmentMethods 109
10.4FutureResearchDirections 110 References 111
11ConversationalHuman–MachineInterfaces 115
MaríaJesúsRodríguez-Sánchez,KawtarBenghazi,DavidGriol,andZoraidaCallejas
11.1Introduction 115
11.2Background 115
11.2.1HistoryoftheDevelopmentoftheField 116
11.2.2BasicDefinitions 117
11.3State-of-the-Art 117
11.3.1DiscussionoftheMostImportantScientific/TechnicalContributions 117
11.3.2ComparisonTable 119
11.4FutureResearchChallenges 121
11.4.1CurrentResearchIssues 121
11.4.2FutureResearchDirectionsDealingwiththeCurrentIssues 121 References 122
12Interaction-CenteredDesign:AnEnduringStrategyandMethodologyfor SociotechnicalSystems 125 MingHou,ScottFang,WenbiWang,andPhilipS.E.Farrell
12.1Introduction 125
12.2EvolutionofHMSDesignStrategy 126
12.2.1AHMSTechnology:IntelligentAdaptiveSystem 126
12.2.2EvolutionofIASDesignStrategy 128
12.3State-of-the-Art:Interaction-CenteredDesign 130
12.3.1AGenericAgent-basedICDFramework 130
12.3.2IMPACTS:AnHuman–MachineTeamingTrustModel 132
12.3.3ICDRoadmapforIASDesignandDevelopment 133
12.3.4ICDValidation,Adoption,andContributions 134
12.4IASDesignChallengesandFutureWork 135
12.4.1ChallengesofHMSTechnology 136
12.4.2FutureWorkinIASDesignandValidation 136 References 137
13Human–MachineComputing:Paradigm,Challenges,andPractices 141
ZhiwenYu,QingyangLi,andBinGuo
13.1Introduction 141
13.2Background 142
13.2.1HistoryoftheDevelopment 142
13.2.2BasicDefinitions 143
13.3StateoftheArt 144
13.3.1TechnicalContributions 144
13.3.2ComparisonTable 148
13.4FutureResearchChallenges 150
13.4.1CurrentResearchIssues 150
13.4.2FutureResearchDirections 151 References 152
14CompanionTechnology 155
AndreasWendemuth
14.1Introduction 155
14.2Background 155
14.2.1History 156
14.2.2BasicDefinitions 157
14.3State-of-the-Art 158
14.3.1DiscussionoftheMostImportantScientific/TechnicalContributions 159
14.4FutureResearchChallenges 159
14.4.1CurrentResearchIssues 159
14.4.2FutureResearchDirectionsDealingwiththeCurrentIssues 160 References 161
15ASurveyonRollator-TypeMobilityAssistanceRobots 165
MiladGeravand,ChristianWerner,KlausHauer,andAngelikaPeer
15.1Introduction 165
15.2MobilityAssistancePlatforms 165
15.2.1Actuation 166
15.2.2Kinematics 166
15.2.2.1LocomotionSupport 166
15.2.2.2STSSupport 166
15.2.3Sensors 168
15.2.4Human–MachineInterfaces 168
15.3Functionalities 168
x Contents
15.3.1STSAssistance 169
15.3.2WalkingAssistance 169
15.3.2.1ManeuverabilityImprovement 169
15.3.2.2GravityCompensation 170
15.3.2.3ObstacleAvoidance 170
15.3.2.4FallsRiskPredictionandFallPrevention 170
15.3.3LocalizationandNavigation 170
15.3.3.1MapBuildingandLocalization 171
15.3.3.2PathPlanning 171
15.3.3.3AssistedLocalization 171
15.3.3.4AssistedNavigation 171
15.3.4FurtherFunctionalities 171
15.3.4.1ReminderSystems 171
15.3.4.2HealthMonitoring 171
15.3.4.3Communication,Information,Entertainment,andTraining 172
15.4Conclusion 172 References 173
16AWearableAffectiveRobot 181
JiaLiu,JinfengXu,MinChen,andIztokHumar
16.1Introduction 181
16.2ArchitectureDesignandCharacteristics 183
16.2.1ArchitectureofaWearableAffectiveRobot 183
16.2.2CharacteristicsofaWearableAffectiveRobot 184
16.3DesignoftheWearable,AffectiveRobot’sHardware 185
16.3.1AIWACBoxHardwareDesign 185
16.3.2HardwareDesignoftheEEGAcquisition 185
16.3.3AIWACSmartTactileDevice 185
16.3.4PrototypeoftheWearableAffectiveRobot 186
16.4AlgorithmfortheWearableAffectiveRobot 186
16.4.1AlgorithmforAffectiveRecognition 186
16.4.2User-BehaviorPerceptionbasedonaBrain-WearableDevice 186
16.5LifeModelingoftheWearableAffectiveRobot 187
16.5.1DataSetLabelingandProcessing 188
16.5.2MultidimensionalDataIntegration 188
16.5.3ModelingofAssociatedScenarios 188
16.6ChallengesandProspects 189
16.6.1ResearchChallengesoftheWearableAffectiveRobot 189
16.6.2ApplicationScenariosfortheWearableAffectiveRobot 189
16.7Conclusions 190
References 190
17VisualHuman–ComputerInteractionsforIntelligentVehicles 193 XumengWang,WeiChen,andFei-YueWang
17.1Introduction 193
17.2Background 193
17.3State-of-the-Art 194
17.3.1VHCIinVehicles 194
17.3.1.1InformationFeedbackfromIntelligentVehicles 195
17.3.1.2Human-GuidedDriving 195
17.3.2VHCIAmongVehicles 195
17.3.3VHCIBeyondVehicles 195
17.4FutureResearchChallenges 196
17.4.1VHCIforIntelligentVehicles 196
17.4.1.1VehicleDevelopment 196
17.4.1.2VehicleManufacture 197
17.4.1.3PreferenceRecording 197
17.4.1.4VehicleUsage 197
17.4.2VHCIforIntelligentTransportationSystems 198
17.4.2.1ParallelWorld 198
17.4.2.2TheFrameworkofIntelligentTransportationSystems 198 References 199
18IntelligentCollaborationBetweenHumansandRobots 203 AndreaMariaZanchettin
18.1Introduction 203
18.2Background 203
18.2.1Context 203
18.2.2BasicDefinitions 204
18.3RelatedWork 205
18.4ValidationCases 206
18.4.1ASimpleVerificationScenario 207
18.4.2ActivityRecognitionBasedonSemanticHand-ObjectInteraction 208
18.5Conclusions 210
18.6FutureResearchChallenges 210 References 210
19ToBeTrustworthyandToTrust:TheNewFrontierofIntelligent Systems 213 RinoFalcone,AlessandroSapienza,FilippoCantucci,andCristianoCastelfranchi
19.1Introduction 213
19.2Background 214
19.3BasicDefinitions 214
19.4State-of-the-Art 215
19.4.1TrustinDifferentDomains 215
19.4.2SelectedArticles 215
19.4.3DifferencesintheUseofTrust 216
19.4.4ApproachestoModelTrust 217
19.4.5SourcesofTrust 218
19.4.6DifferentComputationalModelsofTrust 218
19.5FutureResearchChallenges 220 References 221
20DecodingHumans’andVirtualAgents’EmotionalExpressions 225 TerryAmorese,GennaroCordasco,MarialuciaCuciniello,OlgaShevaleva,StefanoMarrone, CarlVogel,andAnnaEsposito
20.1Introduction 225
20.2RelatedWork 226
20.3MaterialsandMethodology 227
20.3.1Participants 227
20.3.2Stimuli 228
20.3.3ToolsandProcedures 228
20.4DescriptiveStatistics 229
20.5DataAnalysisandResults 230
20.5.1ComparisonSyntheticvs.NaturalisticExperiment 234
20.6DiscussionandConclusions 235
Acknowledgment 238
References 238
21IntelligentComputationalEdge:FromPervasiveComputingandInternetof ThingstoComputingContinuum 241 RadmilaJuric
21.1Introduction 241
21.2TheJourneyofPervasiveComputing 242
21.3ThePoweroftheIoT 243
21.3.1InherentProblemswiththeIoT 244
21.4IoT:TheJourneyfromCloudtoEdge 245
21.5TowardIntelligentComputationalEdge 246
21.6IsComputingContinuumtheAnswer? 247
21.7DoWeHaveMoreQuestionsthanAnswers? 248
21.8WhatWouldourVisionBe? 249
References 251
22ImplementingContextAwarenessinAutonomousVehicles 257 FedericoFaruffini,AlessandroCorrea-Victorino,andMarie-HélèneAbel
22.1Introduction 257
22.2Background 258
22.2.1Ontologies 258
22.2.2AutonomousDriving 258
22.2.3BasicDefinitions 259
22.3RelatedWorks 260
22.4ImplementationandTests 261
22.4.1ImplementingtheContextofNavigation 261
22.4.2ControlLoopRule 262
22.4.3Simulations 263
22.5Conclusions 264
22.6FutureResearchChallenges 264 References 264
23TheAugmentedWorkforce:ASystematicReviewofOperatorAssistance Systems 267
ElisaRoth,MircoMoencks,andThomasBohné
23.1Introduction 267
23.2Background 268
23.2.1Definitions 268
23.3StateoftheArt 269
23.3.1EmpiricalConsiderations 270
23.3.1.1ApplicationAreas 270
23.3.2AssistanceCapabilities 270
23.3.2.1TaskGuidance 271
23.3.2.2KnowledgeManagement 271
23.3.2.3Monitoring 273
23.3.2.4Communication 273
23.3.2.5Decision-Making 273
23.3.3Meta-capabilities 274
23.3.3.1ConfigurationFlexibility 274
23.3.3.2Interoperability 274
23.3.3.3ContentAuthoring 274
23.3.3.4Initiation 274
23.3.3.5Hardware 275
23.3.3.6UserInterfaces 275
23.4FutureResearchDirections 275
23.4.1EmpiricalEvidence 275
23.4.2CollaborativeResearch 277
23.4.3SystemicApproaches 277
23.4.4Technology-MediatedLearning 277
23.5Conclusion 277 References 278
24CognitivePerformanceModeling 281
MaryamZahabiandJunhoPark
24.1Introduction 281
24.2Background 281
24.3State-of-the-Art 282
24.4CurrentResearchIssues 286
24.5FutureResearchDirectionsDealingwiththeCurrentIssues 286 References 287
25AdvancedDriverAssistanceSystems:TransparencyandDriverPerformance Effects 291 YulinDengandDavidB.Kaber
25.1Introduction 291
25.2Background 292
25.2.1Context 292
25.2.2BasicDefinition 292
25.3RelatedWork 293
25.4Method 294
25.4.1Apparatus 295
25.4.2Participants 296
25.4.3ExperimentDesign 296
25.4.4Tasks 297
25.4.5DependentVariables 297
25.4.5.1HazardNegotiationPerformance 297
25.4.5.2VehicleControlPerformance 298
25.4.6Procedure 298
25.5Results 299
25.5.1HazardReactionPerformance 299
25.5.2PosthazardManualDrivingPerformance 299
25.5.3PosttestingUsabilityQuestionnaire 301
25.6Discussion 302
25.7Conclusion 303
25.8FutureResearch 304 References 304
26RGB-DBasedHumanActionRecognition:FromHandcraftedtoDeep Learning 307 BangliLiuandHonghaiLiu
26.1Introduction 307
26.2RGB-DSensorsand3DData 307
26.3HumanActionRecognitionviaHandcraftedMethods 308
26.3.1Skeleton-BasedMethods 308
26.3.2Depth-BasedMethods 309
26.3.3HybridFeature-BasedMethods 309
26.4HumanActionRecognitionviaDeepLearningMethods 310
26.4.1CNN-BasedMethods 310
26.4.2RNN-BasedMethods 311
26.4.3GCN-BasedMethods 313
26.5Discussion 314
26.6RGB-DDatasets 314
26.7ConclusionandFutureDirections 315 References 316
27HybridIntelligence:AugmentingEmployees’Decision-MakingwithAI-Based Applications 321 InaHeine,ThomasHellebrandt,LouisHuebser,andMarcosPadrón
27.1Introduction 321
27.2Background 321
27.2.1Context 321
27.2.2BasicDefinitions 322
27.3RelatedWork 323
27.4TechnicalPartoftheChapter 324
27.4.1DescriptionoftheUseCase 324
27.4.1.1BusinessModel 324
27.4.1.2Process 324
27.4.1.3UseCaseObjectives 325
27.4.2DescriptionoftheEnvisionedSolution 325
27.4.3DevelopmentApproachofAIApplication 326
27.4.3.1DevelopmentProcess 326
27.4.3.2ProcessAnalysisandTimeStudy 326
27.4.3.3DevelopmentandDeploymentData 327
27.4.3.4SystemTestingandDeployment 327
27.4.3.5DevelopmentInfrastructureandDevelopmentCostMonitoring 327
27.5Conclusions 330
27.6FutureResearchChallenges 330 References 330
28HumanFactorsinDriving 333
BirsenDonmez,DengboHe,andHollandM.Vasquez
28.1Introduction 333
28.2ResearchMethodologies 334
28.3In-VehicleElectronicDevices 335
28.3.1Distraction 335
28.3.2InteractionModality 336
28.3.2.1VisualandManualModalities 336
28.3.2.2AuditoryandVocalModalities 337
28.3.2.3HapticModality 338
28.3.3WearableDevices 338
28.4VehicleAutomation 339
28.4.1DriverSupportFeatures 339
28.4.2AutomatedDrivingFeatures 341
28.5DriverMonitoringSystems 342
28.6Conclusion 343 References 343
29WearableComputingSystems:State-of-the-ArtandResearch Challenges 349
GiancarloFortinoandRaffaeleGravina
29.1Introduction 349
29.2WearableDevices 350
29.2.1AHistoryofWearables 350
29.2.2SensorTypes 351
29.2.2.1PhysiologicalSensors 352
29.2.2.2InertialSensors 352
29.2.2.3VisualSensors 352
29.2.2.4AudioSensors 355
29.2.2.5OtherSensors 355
29.3BodySensorNetworks-basedWearableComputingSystems 355
29.3.1BodySensorNetworks 355
29.3.2TheSPINEBody-of-Knowledge 357
29.3.2.1TheSPINEFramework 357
29.3.2.2TheBodyCloudFramework 359
29.4ApplicationsofWearableDevicesandBSNs 360
29.4.1Healthcare 360
29.4.1.1CardiovascularDisease 362
29.4.1.2Parkinson’sDisease 362
29.4.1.3RespiratoryDisease 362
29.4.1.4Diabetes 363
29.4.1.5Rehabilitation 363
29.4.2Fitness 363
29.4.2.1DietMonitoring 363
29.4.2.2Activity/FitnessTracker 363
29.4.3Sports 364
29.4.4Entertainment 364
29.5ChallengesandProspects 364
29.5.1MaterialsandWearability 364
29.5.2PowerSupply 365
29.5.3SecurityandPrivacy 365
29.5.4Communication 365
29.5.5EmbeddedComputing,DevelopmentMethodologies,andEdgeAI 365
29.6Conclusions 365 Acknowledgment 366 References 366
30MultisensorWearableDeviceforMonitoringVitalSignsandPhysical Activity 373
JoshuaDiTocco,LuigiRaiano,DanielaloPresti,CarloMassaroni,DomenicoFormica,and EmilianoSchena
30.1Introduction 373
30.2Background 373
30.2.1Context 373
30.2.2BasicDefinitions 374
30.3RelatedWork 375
30.4CaseStudy:MultisensorWearableDeviceforMonitoringRRandPhysical Activity 376
30.4.1WearableDeviceDescription 376
30.4.1.1ModulefortheEstimationofRR 377
30.4.1.2ModulefortheEstimationofPhysicalActivity 377
30.4.2ExperimentalSetupandProtocol 378
30.4.2.1ExperimentalSetup 378
30.4.2.2ExperimentalProtocol 378
30.4.3DataAnalysis 378
30.4.4Results 378
30.5Conclusions 379
30.6FutureResearchChallenges 380 References 380
31IntegrationofMachineLearningwithWearableTechnologies 383
DariusNahavandi,RoohallahAlizadehsani,andAbbasKhosravi
31.1Introduction 383
31.2Background 384
31.2.1HistoryofWearables 384
31.2.2SupervisedLearning 384
31.2.3UnsupervisedLearning 386
31.2.4DeepLearning 386
31.2.5DeepDeterministicPolicyGradient 387
31.2.6CloudComputing 388
31.2.7EdgeComputing 388
31.3StateoftheArt 389
31.4FutureResearchChallenges 392 References 393
32Gesture-BasedComputing 397
GennaroCostagliola,MattiaDeRosa,andVittorioFuccella
32.1Introduction 397
32.2Background 398
32.2.1HistoryoftheDevelopmentofGesture-BasedComputing 398
32.2.2BasicDefinitions 399
32.3StateoftheArt 399
32.4FutureResearchChallenges 402
32.4.1CurrentResearchIssues 402
32.4.2FutureResearchDirectionsDealingwiththeCurrentIssues 403 Acknowledgment 403 References 403
33EEG-basedAffectiveComputing 409 XueliangQuanandDongruiWu
33.1Introduction 409
33.2Background 409
33.2.1BriefHistory 409
33.2.2EmotionTheory 410
33.2.3EmotionRepresentation 410
33.2.4EEG 410
33.2.5EEG-BasedEmotionRecognition 411
33.3State-of-the-Art 411
33.3.1PublicDatasets 411
33.3.2EEGFeatureExtraction 411
33.3.3FeatureFusion 412
33.3.4AffectiveComputingAlgorithms 413
33.3.4.1TransferLearning 413
33.3.4.2ActiveLearning 413
33.3.4.3DeepLearning 413
33.4ChallengesandFutureDirections 414 Acknowledgment 415 References 415
34SecurityofHumanMachineSystems 419
FrancescoFlammini,EmanueleBellini,MariaStelladeBiase,andStefanoMarrone
34.1Introduction 419
34.2Background 420
34.2.1AnHistoricalRetrospective 420
34.2.2FoundationsofSecurityTheory 421
34.2.3AReferenceModel 421
34.3StateoftheArt 422
34.3.1SurveyMethodology 422
34.3.2ResearchTrends 425
34.4ConclusionsandFutureResearch 426 References 428
35IntegratingInnovation:TheRoleofStandardsinPromotingResponsible DevelopmentofHuman–MachineSystems 431
ZachMcKinney,MartijndeNeeling,LuigiBianchi,andRicardoChavarriaga
35.1IntroductiontoStandardsinHuman–MachineSystems 431
35.1.1WhatAreStandards? 431
35.1.2StandardsinContext:TechnologyGovernance,BestPractice,andSoftLaw 432
35.1.3TheNeedforStandardsinHMS 433
35.1.4BenefitsofStandards 433
35.1.5WhatMakesanEffectiveStandard? 434
35.2TheHMSStandardsLandscape 435
35.2.1StandardsinNeuroscienceandNeurotechnologyforBrain–MachineInterfaces 435
35.2.2IEEEP2731–UnifiedTerminologyforBCI 435
35.2.2.1TheBCIGlossary 439
35.2.2.2TheBCIFunctionalModel 439
35.2.2.3BCIDataStorage 439
35.2.3IEEEP2794–ReportingStandardfor invivo NeuralInterfaceResearch(RSNIR) 441
35.3StandardsDevelopmentProcess 443
35.3.1WhoCanParticipateinStandardsDevelopment? 443
35.3.2WhyShouldIParticipateinStandardsDevelopment? 444
35.3.3HowCanIgetInvolvedinStandardsDevelopment? 444
35.4StrategicConsiderationsandDiscussion 444
35.4.1ChallengestoDevelopmentandBarrierstoAdoptionofStandards 444
35.4.2StrategiestoPromoteStandardsDevelopmentandAdoption 445
35.4.3FinalPerspective:OnInnovation 445 Acknowledgments 446 References 446
36SituationAwarenessinHuman-MachineSystems 451 GiuseppeD’AnielloandMatteoGaeta
36.1Introduction 451
36.2Background 452
36.3State-of-the-Art 453
36.3.1SituationIdentificationTechniquesinHMS 454
36.3.2SituationEvolutioninHMS 455
36.3.3Situation-AwareHumanMachine-Systems 455
36.4DiscussionandResearchChallenges 456
36.5Conclusion 458 References 458
37Modeling,Analyzing,andFosteringtheAdoptionofNewTechnologies:The CaseofElectricVehicles 463
ValentinaBreschi,ChiaraRavazzi,SilviaStrada,FabrizioDabbene,andMaraTanelli
37.1Introduction 463
37.2Background 464
37.2.1AnAgent-basedModelforEVTransition 464
37.2.2CalibrationBasedonRealMobilityPatterns 466
37.3FosteringtheEVTransitionviaControloverNetworks 468
37.3.1RelatedWork:APerspectiveAnalysis 468
37.3.2ANewModelforEVTransitionwithIncentivePolicies 469
37.3.2.1ModelingTime-varyingThresholds 469
37.3.2.2CalibrationoftheModel 470
37.4BoostingEVAdoptionwithFeedback 470
37.4.1FormulationoftheOptimalControlProblem 470
37.4.2DerivationoftheOptimalPolicies 471
37.4.3ARecedingHorizonStrategytoBoostEVAdoption 472
37.5ExperimentalResults 473
37.6Conclusions 476
37.7FutureResearchChallenges 477
Acknowledgments 477 References 477
Index 479
EditorsBiography
GiancarloFortino(IEEEFellow’22)isfullprofessorofComputerEngineeringintheDepartment ofInformatics,Modeling,Electronics,andSystemsattheUniversityofCalabria(Unical),Italy. HehasaPhDinSystemsandComputerEngineeringfromUniversityofCalabriain2000.His researchinterestsincludewearablecomputingsystems,InternetofThings,andcybersecurity.He isnamedHighlyCitedResearcher2002–2022byClarivateinComputerScience.Hehasauthored morethan650papersininternationaljournals,conferences,andbooks.Heis(founding)series editoroftheIEEEPressBookSeriesonHuman–MachineSystemsandoftheSpringerInternet ofThingsseries,andisAssociateEditorofpremierIEEETransactions.HeiscofounderandCEO ofSenSysCalS.r.l.,aUnicalspinofffocusedoninnovativeIoTsystems,andcofounderofBigTech S.r.l.,astartupfocusedonAI-drivensystemsandBigData.Fortinoiscurrentlyamemberofthe IEEESMCSBoGandco-chairoftheSMCSTConIWCD.
DavidKaberiscurrentlytheDean’sLeadershipProfessorandChairoftheDepartmentofIndustrialandSystemsEngineeringattheUniversityofFlorida(UF).PriortojoiningUF,Kaberwasa distinguishedprofessorofindustrialengineeringatNorthCarolinaStateUniversitywherehealso servedastheDirectorofResearchfortheErgonomicsCenterofNorthCarolina.Kaber’sprimary areaofresearchinterestishuman-systemsengineeringwithafocusonhuman–automaton interaction,includingdesignandanalysisforsituationawarenessincomplexhumanin-the-loop systems.Domainsofstudyforhisresearchhaveincludedphysicalworksystems,industrialsafety systems,roboticsystems,transportationsystems,andhealthcare.KaberisafellowofIEEEand previouseditor-in-chiefofthe IEEETransactionsonHuman–MachineSystems.Heisafellowof InstituteofIndustrialEngineersandafellowoftheHumanFactorsandErgonomicsSociety.Kaber isalsoaCertifiedHumanFactorsProfessional(BCPE)andaCertifiedSafetyProfessional(BCSP).
AndreasNürnbergerisprofessorofDataandKnowledgeEngineeringattheOtto-von-Guericke UniversitätMagdeburg(OVGU),Germany.Hisresearchfocusesonadaptivityinhuman–machine systems,consideringaspectssuchasuserbehavioranalysisandintelligentuserassistance.He wasinvolvedintheorganizationofmanyconferencesandworkshopsinrelatedareasandthe developmentofnewscientificevents,amongothers,theIEEESMCSsponsoredinternational conferenceseriesonHuman–MachineSystems(IEEEICHMS).Andreaswasvisitingresearcher attheUniversityofMelbourne,Australia;postdocatUCBerkeley,UnitedStates;andvisiting professoratUniversitéPierreetMarieCurie,Paris.AndreasisanEmmyNoetherFellowofthe GermanScienceFoundation(DFG).
DavidMendonça(SeniorMember,2012)isSeniorPrincipalDecisionScientistatMITRECorporation.HepreviouslyheldtherankofprofessorintheDepartmentofIndustrialandSystems EngineeringandintheDepartmentofCognitiveScienceatRensselaerPolytechnicInstitute. HeservedasaProgramDirectorattheNationalScienceFoundationfrom2015to2017.Hewas avisitingscholarattheUniversityofLisbon(Portugal)andatDelftUniversityofTechnology (TheNetherlands).HeiscurrentlyamemberoftheBoardofGovernorsoftheIEEESystems, ManandCyberneticsSociety,andoftheIEEEBoston(Massachusetts)Section.HeholdsaPhD inDecisionSciencesandEngineeringSystemsfromRensselaerPolytechnicInstitute,anMSfrom CarnegieMellonUniversity,andaBAfromUniversityofMassachusetts/Amherst.
ListofContributors
Marie-HélèneAbel HeudiasycLaboratory UniversityofTechnologyofCompiègne Compiègne
France
RoohallahAlizadehsani InstituteforIntelligentSystemsResearchand Innovation(IISRI) DeakinUniversity WaurnPonds,VIC Australia
TerryAmorese DepartmentofPsychology UniversitàdellaCampania“L.Vanvitelli” Caserta
Italy
AbdelkaderNasreddineBelkacem DepartmentofComputerandNetwork Engineering CollegeofInformationTechnology UnitedArabEmiratesUniversity AlAin
UAE
EmanueleBellini DipartimentodiMatematicaeFisica UniversitádegliStudidellaCampania “LuigiVanvitelli” Caserta
Italy
KawtarBenghazi UniversidaddeGranada Granada
Spain
LuigiBianchi DepartmentofCivilEngineeringand ComputerScienceEngineering “TorVergata”UniversityofRome
Rome
Italy
MariaStellaDeBiase DipartimentodiMatematicaeFisica UniversitádegliStudidellaCampania “LuigiVanvitelli”
Caserta
Italy
ThomasBohné DepartmentofEngineering InstituteforManufacturing UniversityofCambridge
Cambridge UK
StefanieS.Bradley IIIBloorviewResearchInstitute HollandBloorviewKidsRehabilitation Hospital Toronto Canada
ValentinaBreschi
DepartmentofElectricalEngineering EindhovenUniversityofTechnology Eindhoven
Netherlands
ZoraidaCallejas UniversidaddeGranada Granada
Spain
MaximilianoCanche FacultyofMathematics UniversidadAutónomadeYucatán Mérida,Yucatán
Mexico
FilippoCantucci
TrustTheoryandTechnologyGroup InstituteofCognitiveSciencesand Technologies NationalResearchCouncilofItaly
Rome Italy
TomCarlson AspireCreate UniversityCollegeLondon Stanmore,Middlesex UK
CristianoCastelfranchi
TrustTheoryandTechnologyGroup InstituteofCognitiveSciencesand Technologies NationalResearchCouncilofItaly
Rome Italy
TomChau IIIBloorviewResearchInstitute HollandBloorviewKidsRehabilitation Hospital
Toronto Canada
RicardoChavarriaga CentreforArtificialIntelligence SchoolofEngineering ZurichUniversityofAppliedSciencesZHAW Winterthur
Switzerland and CLAIREOfficeSwitzerland,ZHAWdigital ZurichUniversityofAppliedSciences Winterthur
Zürich Switzerland
JessieY.C.Chen U.S.ArmyResearchLaboratory AberdeenProvingGround,Maryland USA
MinChen SchoolofComputerScienceandTechnology HuazhongUniversityofScienceand Technology
Wuhan China
WeiChen StateKeyLabofCAD&CG ZhejiangUniversity Hangzhou
China
GennaroCordasco DepartmentofPsychology UniversitàdellaCampania“L.Vanvitelli” Caserta
Italy
AlessandroCorrea-Victorino HeudiasycLaboratory UniversityofTechnologyofCompiègne Compiègne
France
GennaroCostagliola DipartimentodiInformatica UniversitàdiSalerno ViaGiovanniPaoloII,132,84084 Fisciano(SA)
Italy
MarialuciaCuciniello DepartmentofPsychology UniversitàdellaCampania“L.Vanvitelli” Caserta Italy
GiuseppeD’Aniello DepartmentofInformationandElectrical EngineeringandAppliedMathematics UniversityofSalerno Fisciano(SA)
Italy
FabrizioDabbene IstitutodiElettronicaeIngegneria dell’Informazioneedelle Telecomunicazioni–IEIIT CentroNazionaledelleRicerche Torino
Italy
ArwenH.DeCostanza U.S.DEVCOM ArmyResearchLaboratory AberdeenProvingGround,MD USA
YulinDeng CepheidHumanFactorsandEngineering Team Sunnyvale,CA USA
BirsenDonmez DepartmentofMechanicalandIndustrial Engineering UniversityofToronto Toronto,ON Canada
AnnaEsposito DepartmentofPsychology UniversitàdellaCampania“L.Vanvitelli” Caserta
Italy
RinoFalcone TrustTheoryandTechnologyGroup InstituteofCognitiveSciencesand Technologies NationalResearchCouncilofItaly
Rome
Italy
TiagoH.Falk Institutnationaldelarecherchescientifique UniversityofQuebec Montreal,Quebec Canada ScottFang TorontoResearchCenter DefenceResearchandDevelopmentCanada Toronto Canada
PhilipS.E.Farrell TorontoResearchCenter DefenceResearchandDevelopmentCanada Toronto Canada
FedericoFaruffini HeudiasycLaboratory UniversityofTechnologyofCompiègne Compiègne France and DIBRIS UniversityofGenoa
Genoa
Italy
ListofContributors
FrancescoFlammini SchoolofInnovation,Design,andEngineering MälardalenUniversity Eskilstuna
Sweden
EricaD.Floreani IBCI4Kids UniversityofCalgary Calgary Canada and IIDepartmentofPediatrics AlbertaChildren’sHospital Calgary Canada
DomenicoFormica SchoolofEngineering NewcastleUniversity UK
GiancarloFortino DepartmentofInformatics,Modeling ElectronicsandSystems UniversityofCalabria Rende Italy
VittorioFuccella DipartimentodiInformatica UniversitàdiSalerno Fisciano(SA)
Italy
MatteoGaeta DepartmentofInformationandElectrical EngineeringandAppliedMathematics UniversityofSalerno Fisciano(SA)
Italy
MiladGeravand DeepCareGmbH Waiblingen Germany
RaffaeleGravina DepartmentofInformatics,Modeling ElectronicsandSystems UniversityofCalabria Rende
Italy
DavidGriol UniversidaddeGranada
Granada
Spain
StephanieGross AustrianResearchInstituteforArtificial
Intelligence
Vienna
Austria
ChristophGuger g.tecmedicalengineeringGmbH g.tec
Austria
BinGuo Departmentofintelligentcomputingsystem SchoolofComputerScience
NorthwesternPolytechnicalUniversity Xi’an
China
LydiaHabib DepartmentofAutomationandControl Univ.PolytechniqueHauts-de-France,CNRS UMR8201–LAMIH Valenciennes
France
NiloufaralsadatHashemi IIIBloorviewResearchInstitute HollandBloorviewKidsRehabilitation Hospital Toronto Canada
KlausHauer AgaplesionBethanien-Hospital,Geriatric Centre
UniversityofHeidelberg Heidelberg Germany
DengboHe IntelligentTransportationThrust,SystemsHub HongKongUniversityofScienceand Technology(Guangzhou) Guangzhou China and DepartmentofCivilandEnvironmental Engineering
HongKongUniversityofScienceand Technology
HongKongSAR China
InaHeine OrganizationalDevelopment LaboratoryforMachineToolsandProduction Engineering
RWTHAachenUniversity Aachen,NRW Germany
ThomasHellebrandt OrganizationalDevelopment LaboratoryforMachineToolsandProduction Engineering
RWTHAachenUniversity Aachen,NRW Germany
ChristineHorner IIIBloorviewResearchInstitute HollandBloorviewKidsRehabilitation Hospital Toronto Canada
MingHou TorontoResearchCenter DefenceResearchandDevelopmentCanada Toronto Canada
LouisHuebser OrganizationalDevelopment LaboratoryforMachineToolsandProduction Engineering
RWTHAachenUniversity Aachen,NRW Germany
IztokHumar FacultyofElectricalEngineering UniversityofLjubljana Ljubljana Slovenia
BrianIrvine IIDepartmentofPediatrics AlbertaChildren’sHospital Calgary Canada and IIIBloorviewResearchInstitute HollandBloorviewKidsRehabilitation Hospital Toronto Canada ZeannaJadavji IBCI4Kids UniversityofCalgary Calgary Canada
RadmilaJuric IndependentResearcherLondon UK
DavidB.Kaber
UniversityofFlorida DepartmentofIndustrialandSystems Engineering
Gainesville,FL
USA
DionKelly IBCI4Kids UniversityofCalgary
Calgary Canada and IIDepartmentofPediatrics
AlbertaChildren’sHospital Calgary Canada
AbbasKhosravi InstituteforIntelligentSystemsResearchand Innovation(IISRI)
DeakinUniversity WaurnPonds,VIC Australia
EliKinney-Lang IBCI4Kids
UniversityofCalgary
Calgary Canada and IIDepartmentofPediatrics
AlbertaChildren’sHospital
Calgary Canada
AdamKirton IBCI4Kids UniversityofCalgary
Calgary Canada and IIDepartmentofPediatrics AlbertaChildren’sHospital
Calgary Canada
BrigitteKrenn AustrianResearchInstituteforArtificial Intelligence Vienna Austria
AbderrahmaneLakas DepartmentofComputerandNetwork Engineering CollegeofInformationTechnology UnitedArabEmiratesUniversity
AlAin UAE
QingyangLi Departmentofintelligentcomputingsystem SchoolofComputerScience NorthwesternPolytechnicalUniversity Xi’an China
BangliLiu SchoolofComputerScienceandInformatics FacultyofComputing,Engineering,andMedia DeMontfortUniversity
Leicester,England
UK
HonghaiLiu SchoolofComputing,FacultyofTechnology UniversityofPortsmouth
Portsmouth,England
UK
