EngineeringIntelligentSystems
EngineeringIntelligentSystems
SystemsEngineeringandDesignwithArtificial Intelligence,VisualModeling,andSystemsThinking
BarclayR.Brown
RaytheonCompany Florida,USA
Thiseditionfirstpublished2023 ©2023JohnWiley&Sons,Inc.
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem, ortransmitted,inanyformorbyanymeans,electronic,mechanical,photocopying,recordingor otherwise,exceptaspermittedbylaw.Adviceonhowtoobtainpermissiontoreusematerial fromthistitleisavailableathttp://www.wiley.com/go/permissions.
TherightofBarclayR.Browntobeidentifiedastheauthorofthisworkhasbeenassertedin accordancewithlaw.
RegisteredOffice
JohnWiley&Sons,Inc.,111RiverStreet,Hoboken,NJ07030,USA
EditorialOffice 111RiverStreet,Hoboken,NJ07030,USA
Fordetailsofourglobaleditorialoffices,customerservices,andmoreinformationaboutWiley productsvisitusatwww.wiley.com.
Wileyalsopublishesitsbooksinavarietyofelectronicformatsandbyprint-on-demand.Some contentthatappearsinstandardprintversionsofthisbookmaynotbeavailableinother formats.
LimitofLiability/DisclaimerofWarranty
Inviewofongoingresearch,equipmentmodifications,changesingovernmentalregulations, andtheconstantflowofinformationrelatingtotheuseofexperimentalreagents,equipment, anddevices,thereaderisurgedtoreviewandevaluatetheinformationprovidedinthepackage insertorinstructionsforeachchemical,pieceofequipment,reagent,ordevicefor,amongother things,anychangesintheinstructionsorindicationofusageandforaddedwarningsand precautions.Whilethepublisherandauthorshaveusedtheirbesteffortsinpreparingthiswork, theymakenorepresentationsorwarrantieswithrespecttotheaccuracyorcompletenessofthe contentsofthisworkandspecificallydisclaimallwarranties,includingwithoutlimitationany impliedwarrantiesofmerchantabilityorfitnessforaparticularpurpose.Nowarrantymaybe createdorextendedbysalesrepresentatives,writtensalesmaterialsorpromotionalstatements forthiswork.Thefactthatanorganization,website,orproductisreferredtointhisworkasa citationand/orpotentialsourceoffurtherinformationdoesnotmeanthatthepublisherand authorsendorsetheinformationorservicestheorganization,website,orproductmayprovide orrecommendationsitmaymake.Thisworkissoldwiththeunderstandingthatthepublisheris notengagedinrenderingprofessionalservices.Theadviceandstrategiescontainedhereinmay notbesuitableforyoursituation.Youshouldconsultwithaspecialistwhereappropriate. Further,readersshouldbeawarethatwebsiteslistedinthisworkmayhavechangedor disappearedbetweenwhenthisworkwaswrittenandwhenitisread.Neitherthepublishernor authorsshallbeliableforanylossofprofitoranyothercommercialdamages,includingbutnot limitedtospecial,incidental,consequential,orotherdamages.
LibraryofCongressCataloging-in-PublicationDataappliedfor:
PrintISBN:9781119665595(Hardback)
CoverDesign:Wiley
CoverImage:©BillionPhotos/Shutterstock
Setin9.5/12.5ptSTIXTwoTextbyStraive,Chennai,India
Contents
Acknowledgments xi
Introduction xiii
PartISystemsandArtificialIntelligence 1
1ArtificialIntelligence,ScienceFiction,andFear 3
1.1TheDangerofAI 3
1.2TheHumanAnalogy 5
1.3TheSystemsAnalogy 6
1.4KillerRobots 7
1.5WatchingtheWatchers 9
1.6CybersecurityinaWorldofFallibleHumans 12
1.7ImaginingFailure 17
1.8TheNewRoleofData:TheGreenSchoolBusProblem 23
1.9DataRequirements 25
1.9.1Diversity 26
1.9.2Augmentation 28
1.9.3Distribution 29
1.9.4Synthesis 30
1.10TheDataLifecycle 31
1.11AISystemsandPeopleSystems 41
1.12MakinganAIasSafeasaHuman 45 References 48
2WeLiveinaWorldofSystems 49
2.1WhatIsaSystem? 49
2.2NaturalSystems 51
2.3EngineeredSystems 53
2.4HumanActivitySystems 54
2.5SystemsasaProfession 54
2.5.1SystemsEngineering 54
2.5.2SystemsScience 55
2.5.3SystemsThinking 55
2.6ABiologicalAnalogy 56
2.7EmergentBehavior:WhatMakesaSystem,aSystem 56
2.8HierarchyinSystems 60
2.9SystemsEngineering 64
3TheIntelligenceintheSystem:HowArtificialIntelligence ReallyWorks 71
3.1WhatIsArtificialIntelligence? 71
3.1.1Myth1:AISystemsWorkJustLiketheBrainDoes 72
3.1.2Myth2:AsNeuralNetworksGrowinSizeandSpeed,TheyGet Smarter 72
3.1.3Myth3:SolvingaHardorComplexProblemShowsThatanAIIs NearingHumanIntelligence 73
3.2TrainingtheDeepNeuralNetwork 75
3.3TestingtheNeuralNetwork 76
3.4AnnieLearnstoIdentifyDogs 76
3.5HowDoesaNeuralNetworkWork? 80
3.6Features:LatentandOtherwise 81
3.7RecommendingMovies 82
3.8TheOne-PageDeepNeuralNetwork 84
4IntelligentSystemsandthePeopletheyLove 97
4.1CanMachinesThink? 97
4.2HumanIntelligencevs.ComputerIntelligence 98
4.3TheChineseRoom:Understanding,Intentionality,and Consciousness 99
4.4ObjectionstotheChineseRoomArgument 104
4.4.1TheSystemsReplytotheCRA 104
4.4.2TheRobotReply 104
4.4.3TheBrainSimulatorReply 105
4.4.4TheCombinationReply 105
4.4.5TheOtherMindsReply 105
4.4.6TheManyMansionsReply 106
4.5AgreementontheCRA 107
4.5.1AnalyzingtheSystemsReply:CantheRoomUnderstandwhen SearleDoesNot? 109
4.6ImplementationoftheChineseRoomSystem 114
4.7IsThereaChinese-UnderstandingMindintheRoom? 115
4.7.1SearleandBlockonWhethertheChineseRoomCan Understand 116
4.8ChineseRoom:SimulatororanArtificialMind? 118
4.8.1SearleonStrongAIMotivations 120
4.8.2UnderstandingandSimulation 121
4.9TheMindoftheProgrammer 127
4.10Conclusion 133
References 135
PartIISystemsEngineeringforIntelligentSystems 137
5DesigningSystemsbyDrawingPicturesandTelling Stories 139
5.1RequirementsandStories 139
5.2StoriesandPictures:ABetterWay 141
5.3HowSystemsCometoBe 141
5.4TheParadoxofCostAvoidance 145
5.5CommunicationandCreativityinEngineering 147
5.6SeeingtheRealNeeds 148
5.7TellingStories 150
5.8BringingaMovietoLife 153
5.9TellingSystemStories 157
5.10TheCombinationPitch 159
5.11StoriesinTime 160
5.12RolesandPersonas 161
6UseCases:TheSuperpowerofSystemsEngineering 165
6.1TheMainPurposeofSystemsEngineering 165
6.2GettingtheRequirementsRight:AParable 166
6.2.1AParableofSystemsEngineering 168
6.3BuildingaHome:AJourneyofRequirementsandDesign 170
6.4WhereRequirementsComeFromandaKoan 173
6.4.1ARequirementsKoan 177
6.5TheMagicofUseCases 177
6.6TheEssenceofaUseCase 181
6.7UseCasevs.Functions:AParable 184
6.8IdentifyingActors 186
6.8.1ActorsAreOutsidetheSystem 187
6.8.2ActorsInteractwiththeSystem 187
6.8.3ActorsRepresentRoles 188
6.8.4FindingtheRealActors 188
6.8.5IdentifyingNonhumanActors 191
6.8.6DoWeHaveALLtheActors? 193
6.9IdentifyingUseCases 193
6.10UseCaseFlowsofEvents 196
6.10.1BalancingWorkUp-FrontwithSpeed 199
6.10.2UseCaseFlowsandScenarios 201
6.10.3WritingAlternateFlows 202
6.10.4IncludeandExtendwithUseCases 203
6.11ExamplesofUseCases 205
6.11.1ExampleUseCase1:RequestCustomerServicefromAcme LibrarySupport 205
6.11.2ExampleUseCase2:EnsureNetworkStability 206
6.11.3ExampleUseCase3:SearchforBoatinInventory 206
6.12UseCaseswithHumanActivitySystems 207
6.13UseCasesasaSuperpower 208 References 208
7PicturingSystemswithModelBasedSystems Engineering 209
7.1HowHumansBuildThings 209
7.2C:Context 212
7.2.1ActorsfortheVX 213
7.2.2ActorsfortheHomeSystem 216
7.3U:Usage 217
7.4S:StatesandModes 221
7.5T:Timing 224
7.6A:Architecture 225
7.7R:Realization 230
7.8D:Decomposition 234
7.9Conclusion 238
8ATimeforTimeboxesandtheUseofUsageProcesses 239
8.1ProblemsinTimeModeling:Concurrency,FalsePrecision,and Uncertainty 240
8.1.1Concurrency 240
8.1.2FalsePrecision 240
8.1.3Uncertainty 241
8.2ProcessesandUseCases 242
8.3Modeling:TwoParadigms 243
8.3.1TheKeyObservation 244
8.3.2SourceoftheProblem 246
8.4ProcessandSystemParadigms 247
8.5ACloserExaminationofTime 248
8.6TheNeedforaNewApproach 251
8.7TheTimebox 252
8.8TimeboxeswithTimelines 257
8.8.1ThinkinginTimeboxes 257
8.9TheUsageProcess 258
8.10PilotProjectExamples 262
8.10.1PilotProject:TheHuntforRedOctober 262
8.10.2PilotProject:FAA 265
8.10.3PilotProject:IBMAgileProcess 267
8.11Summary:ANewParadigmModelingApproach 269
8.11.1TheImpactofNewParadigmModels 270
8.11.2TheFutureofNewParadigmModels 271 References 272
PartIIISystemsThinkingforIntelligentSystems 275
9SolvingHardProblemswithSystemsThinking 277
9.1HumanActivitySystemsandSystemsThinking 277
9.2TheCentralInsightofSystemsThinking 279
9.3SolvingProblemswithSystemsThinking 281
9.4IdentifyaProblem 281
9.5FindtheRealProblem 282
9.6IdentifytheSystem 284
9.7UnderstandingtheSystem 285
9.7.1RocksAreHard 288
9.7.2HeartandSoul 290
9.7.3ConfusingCauseandEffect 292
9.7.4LogicalFallacies 296
9.8SystemArchetypes 298
9.8.1TragedyoftheCommons 299
9.8.2TheRichGetRicher 300
9.9InterveninginaSystem 302
9.10TestingImplementingInterventionIncrementally 315
9.11SystemsThinkingandtheWorld 316
x Contents
10PeopleSystems:ANewWaytoUnderstandtheWorld 317
10.1ReviewingTypesofSystems 317
10.2PeopleSystems 318
10.3PeopleSystemsandPsychology 320
10.4EndowmentEffect 323
10.5Anchoring 324
10.6FunctionalArchitectureofaPerson 325
10.7Example:TheProblemofPollution 327
10.8SpeechActs 332
10.8.1PeopleSystemArchetypes 337
10.8.1.1DemandSlowing 339
10.8.1.2CustomerService 340 10.9SeekingQuality 341
10.10JobHuntingasaPeopleSystem 344
10.10.1WhoAreYou? 345
10.10.2WhatDoYouWanttoDo? 345
10.10.3ForWhom? 347
10.10.4PickaFew 348
10.10.5GoStraighttotheHiringManager 349
10.10.6FollowThrough 351
10.10.7BroadenYourView 352
10.10.8StepTwo 352
10.11SharedServiceMonopolies 354 References 356 Index 357
Acknowledgments
Writingabookisajourney,butonethatisnevertakenalone.I’mgratefultomany peoplewhosupportedandencouragedthecreationofthisbook.Mycolleagues atINCOSE,theInternationalCouncilonSystemsEngineering,listenedpatiently andcontributedfeedbackontheideasasIpresentedthem,inearlyforms,inyears ofconferencesessions,tutorials,andworkshops.MyworkcolleaguesatIBMmade my17yearsthereproductiveandfascinating,asIwascontinuouslybothastudent andateacherofmodel-basedsystemsengineeringmethods.Ilearnedagreatdeal fromTimBohn,DaveBrown,JimDensmore,BenAmaba,BruceDouglass,Grady Booch,andfromthenumerousaerospaceanddefensecompanieswithwhomI consultedovertheyears.WaldemarKarwowski,mydissertationadvisor,encouragedmuchoftheworkinChapter8.RickSteinerencouragedmyexperimental (readcrazy,sometimes)ideasonhowtoexpandthecapabilitiesofsystemmodelinginmydoctoralresearch.TodNewmanhasbeenamentorandaninspiration sinceIjoinedRaytheonin2018.LarryKennedyoftheQualityManagementInstitutebroughtmeintoaworldofqualityinsystems–somethingIhadalwayshoped wastherebuthadneverfullyappreciated.Mymother,nownearing100yearsof age,andstillasbrightandalertasever,taughtmetolovemath,science,andengineering.Irememberherteachingmemultiplicationwithasetof1001-in.cube blocks–seehowtworowsoffourmakeeight?Finally,mypartnerinlove,life,and everythingelse,HonorAllisonLind,hasbeenwithmesincetheverybeginning ofthisbookproject,severalyearsago(toomanyyearsago,mypublisherreminds me).She’salwaysmybiggestcheerleaderandthebestpartneranyonecouldhave.
Introduction
Sincetheearlydaysofinteractivecomputersandcomplexelectronicsystemsof allkinds,wethehumanusersofthesesystemshavehadtoadaptto them,formattingandenteringinformationthewaytheywantedit,theonlywaytheywere designedtoacceptit.Wepatiently(ornot)interactedwithsystemsusingarcane userinterfaces,navigatingvoicemenus,readingusermanuals(ornot),andsufferingcontinualfrustrationandinefficiencywhenwecouldn’tfigureouthowto interactwithsystemsontheirterms.Limitedcomputingresourcesmeantthat humanusershadtodealwithinefficientuserinterfaces,strictinformationinput formats,andcomputersystemsthatweremorelikefinickypetsthanintelligent assistants.
Humanactivitysystemsofallkinds,includingbusinesses,organizations, societies,families,andeconomies,canproduceequallyfrustratingresultswhen wetrytointeractwiththem.Considerthecommonfrustrationsofdealing withgovernmentbureaucracies,utilitycompanies,phoneandinternetservice companies,universities,internalcorporatestructures,andlargeorganizationsof anykind.
Atthesametime,dramaticandongoingincreasesincomputingspeed, memory,andstorage,alongwithdecreasingcost,size,andpowerrequirements havemadenewtechnologieslikeartificialintelligence,machinelearning,and naturallanguageunderstandingavailabletoallsystemsandsoftwaredevelopers. Theseadvancedtechnologiesarebeingusedtocreateintelligentsystemsand devicesthatdelivernewandadvancedcapabilitiestotheirusers.
Thisbookisaboutanewwayofthinkingaboutthesystemswelivewith,and howtomakethemmoreintelligent,enablingustorelatetothemasintelligent partners,collaborators,andsupervisors.Wehavethetechnology,butwhat’smissingistheengineeringanddesignofnew,moreintelligentsystems.Thisbookis aboutthat–howtocombinesystemsengineeringandsystemsthinkingwithnew technologieslikeartificialintelligenceandmachinelearning,todesignincreasinglyintelligentsystems.
MostoftheemphasisontheuseofAIcapabilitieshasbeenatthealgorithm andimplementationlevel,notatthesystemslevel.Therearenumerousbookson AIandmachine-learningalgorithms,butlittleabouthowtoengineercomplex systemsthatactinintelligentwayswithhumanbeings–whatwecall intelligentsystems.Forexample,GarryKasparovwasthefirstworldchesschampion tobebeatenbyacomputer(IBM’sDeepBluein1997),whichinspiredhimtocreateAdvancedChesscompetitions,whereanyoneoranythingcanenter–human players,computers,orhuman–computerteams.It’sperson-plus-computerthat winsthesecompetitionsmostofthetime;thefutureisoneofclosercooperationbetweenpeopleandincreasinglyintelligentsystems.Intheworldofsystems thinking,wewillconsidertheperson-plus-computerasasysteminitself–one thatismoreintelligentthaneitherthehumanorthecomputeronitsown.
Nearlyeveryonealivetodaygrewupwithsomesortoftechnologyinthehome. Forme,growingupinthe1970s,itwasarotary(dial)phoneandablack-and-white (remote-less)TV.Interactionwassimple–onebuttonordialaccomplishedone function.Theriseofcomputers,software,graphicaluserinterfaces,andvoicecontrolbroughtnewlevelsofcapability,butwestillusuallypress(nowclickortap) onebuttontodoonefunction.Weenterinformationonetidbitatatime–aletter,a word,oranumbermustbeputintothecomputerinjusttherightplaceforthesystemtocarryoutourwishes.Rarelydoesthesystemanticipateourneedsorwork tofulfillourneedsonitsown.Itcanseemthatweareservingthesystems,rather thanthereverse.Buthumansdesignedthosesystems,andtheycanbedesigned better.
Mostoftheinteractionsbetweensystemsandpeoplearestillataverybasic level,wherepeoplefunctionasdirectorremotecontrollers,drivingvehicles,flying drones,ormanagingafactoryfloor.Itisthethesisofthisbookthatwemustfocus ondesigningincreasinglyintelligentsystemsthatwillwork with peopletocreate evenmoreintelligenthuman-plus-machinesystems.Theefficientpartnershipof humanandmachinedoesnothappenaccidentally,butbydesignandthroughthe practiceofsystemsengineering.
Thisbookintroduces,explains,andappliesideasandpracticesfromthefields ofsystemsengineering,model-basedsystemsengineering,systemsthinking,artificialintelligence,machinelearning,philosophy,behavioraleconomics,andpsychology.Itisnotaspeculativebook,fullofvaguepredictionsabouttheAIsofthe future–itisapracticalandpractice-orientedguidetotheengineeringofintelligent systems,basedonthebestpracticesknowninthesefields.
Thebookcanbedividedintothreeparts.
PartI,Chapters1–4,isaboutsystemsandartificialintelligence.InChapter1,we lookatsystemsthatuse,orcoulduseartificialintelligence,andexaminesomeof thepopularconceptions,myths,andfearsaboutintelligentsystems.InChapter2, welookatsystems,whattheyare,howtheybehave,andhowalmosteverythingwe
experiencecanbeseenfromasystemsviewpoint.InChapter3,weexaminedeep neuralnetworks,themostimportantcurrentapproachtoartificiallyintelligent systems,andattempttoremoveallthemysteryabouthowtheyworkbybuilding oneinasimple,one-pagespreadsheet,hopefullyleavingalastingintuitionforthis keytechnology.InChapter4,welookindepthatthequestionwhethercomputers canbemadetothink,orunderstand,inthewaywedo.
PartII,Chapters5–8,isaboutsystemsengineering,andhowthatdisciplinecan beappliedtotheengineeringofintelligentsystems.Chapter5examineshowstorytelling,bothancientandmodern,canbeusedtoconceive,build,andcommunicate aboutnewkindsofsystems.InChapter6,welookathowtoapplythe“superpower”ofuse-casemodelingtobetterdescribehowcomplexandintelligentsystems shouldworkfortheirusers.Chapter7buildsonuse-casemodelingtoshowhow model-basedsystemsengineeringusessimplemodelsanddiagramstodescribe thehighlevelsofasystemdesign,andguideitsdevelopment.Chapter8introduces twonewconcepts–timeboxesandusageprocesses–thatbringanewefficiency andflexibilitytothemodelingofcomplexandintelligentsystems.
PartIII,Chapters9and10,shiftsthefocustosystemsthinking,presentingthe foundationalconcepts,tools,andmethodsusedtounderstandallkindsofsystems.Chapter9worksthroughaprocessforsolvinghardproblemsusingsystems thinkingandexplainstheuseofcausalloopdiagrams,feedbackloops,andsystem archetypes.Chapter10introduces peoplesystems,aspecialkindofsystemcontainingonlypeople,andshowshowtoapplysystemsthinkingtounderstandand improvethisimportantclassofsystems.
Thisisabookaboutengineering,specificallysystemsengineering,butit’snot justforengineers.Nothinginthisbookrequiresaspecializedengineeringbackgroundtounderstand.Engineerswilltellyouthattherealfuninengineeringisthe conceptualizingofanewandinnovativesystemandtheearly-stagedesignwhere allthecreativedecisionsaremade.Thisbookisaboutthatpartofengineering–the funpart,andwewilldrawinspirationandborrowtechniquesfrommoviemaking, art,storytelling,sciencefiction,psychology,behavioraleconomics,andmarketing tobringthefun.Wehopeyouwillseetheworldandeverythinginit,whether physicalornot,assystems,andgainanewinsightinhowtounderstandtheway systemswork.Wewillimagineaworldofintelligentsystems,andthenseehowto engineerthem.
Tokeepintouchwithourcontinuingworkinintelligentsystems,findoutmore atwww.engineeringintelligentsystems.com.
ArtificialIntelligence,ScienceFiction,andFear
Artificialintelligencetechnologiessuchasmachinelearninganddeepneural networksholdgreatpromiseforimproving,evenrevolutionizingmanyapplicationareasanddomains.Curiously,expertsinAIandcasualobserverslineup onbothsidesofthe areAIbenefitsworththerisks? question.Severalbooksfrom prominentAIresearcherspaintdirescenariosofAIsystemsrunamok,escaping thecontroloftheirhumancreatorsandmanagers,pursuingtheir“own”agendas toourdetriment.Atthesametime,AIresearchracesahead,developingnew capabilitiesthatfarsurpasstheperformanceofpastsystemsandevenhumans performingthesametasks.Howcanweresisttheseadvancementsandthe benefitstheybring,eventhoughtheremayberisks?
Thewayoutofthedilemmaistheapplicationofsystemsengineering.Systems engineershavebeenaddressingtheissuesofdangeroustechnologiesfordecades. Nuclearfission,likeAI,isaninherentlydangeroustechnology.Systemsengineers can’tmakefissionsafer,soinsteadtheybuildsystemsaroundthefissionreaction, makingtheentiresystemassafeaspossible.Ifamishapoccurs,thefaultisnot withfissionitself,butwiththedesignorimplementationoftheentiresystem.
Thischapterlooksatsomeofthemainchallengesinthedevelopmentofintelligentsystems–systemsthatincludeoneormoreAI-basedcomponentstoproduce intelligentbehavior–includingreliability,safety,dependability,explainability, andsusceptibilitytointerferenceorhacking.SomerecentAIfailureswillbeused asexamplestohighlighthowsystemsengineeringmethodsandtechniquescould beusedoradaptedtosolveAIchallenges.
1.1TheDangerofAI
IsAIdangerous?It’sadifficultquestion–difficultfirsttounderstand,andthen difficulttoanswer.Dangerouscomparedtowhat?Ifsomeoneproposedatechnologythatwouldbeoftremendouseconomicbenefittoallsegmentsofsociety
EngineeringIntelligentSystems:SystemsEngineeringandDesignwithArtificialIntelligence, VisualModeling,andSystemsThinking,FirstEdition.BarclayR.Brown. ©2023JohnWiley&Sons,Inc.Published2023byJohnWiley&Sons,Inc.
worldwide,butwouldpredictablyresultinthedeathofoveronemillionpeople peryear,wouldthatseemlikeagreatidea?Automobilesaresuchatechnology. Now,someoneelseproposesatechnologythatwoulddramaticallyreducethat numberofdeaths,butwouldcauseasmallnumberofadditionaldeaths,that wouldnothaveoccurredwithoutthenewtechnology.That’sAI.Evenshortof fullyself-drivingcars,theadditionofintelligentsensors,anti-collisionsystems, anddriverassistancesystems,whenwidelydeployed,canbeexpectedtosave manyhundredsorthousandsoflives,atthecostofalikelyfarsmallernumberof additionalliveslosttomalfunctioningintelligentsafetysystems.Life,death,and people’sfeelingsaboutthemhowever,arenotamatterofsimplearithmetic.One hundredlivessaved,atthecostofoneadditionallife,isnotabargainmostwould easilymake,soitisnaturalthatonelifelosttoanerrantAIiscauseforheadline newscoverage,evenwhilethatsameAImaybesavinghundredsofotherlives.
Itisimportanttoaskatthispoint,whatdowemeanbyAI?Dowemeana sentient,all-knowing,all-powerful,andforsomereasonusuallyveryevil,general intelligence,withdirectcontrolofworld-scaleweapons,accesstoallfinancialsystemsandconnectionstoeverynetworkintheworld,asisseeninthemovies?By theendofthischapter,itshouldbeclearthatwhilethisdescriptionmaywork wellforsciencefictionnovelsorscreenplays,itnotagooddesignforanewAI systemintherealworld.Intherealworld,AIreferstoawiderangeofcapabilities thatarethought,inonewayoranother,tobeintelligent.Exceptinacademicand philosophicaldisciplines,wearenotconcernedwiththesafetyoftheAIitself,but withthesafetyofthesystemswithinwhichitoperates–andthat’sthedomainof systemsengineering.
Systemsengineering,anengineeringdisciplinethatexistsalongsideother engineeringdisciplineslikeelectricalengineering,mechanicalengineering,and softwareengineering,focusesonthesystemasawhole,bothhowitshould perform,thefunctionalrequirements,andadditional nonfunctional requirements includingsafety,security,reliability,dependability,andothers.Evidenceof systemsengineeringanditsmorewiderangingcousin,systemsthinking,can beseeneveninancientprojectslikeRomanaqueductsandeconomicandtransportationsystems,butsystemsengineeringreallybeganasaseriousengineering disciplineinthe1950sand1960s.Theemergenceofcomplexcommunication networksfollowedbyinternationallycompetitivespaceprogramsandlarge defenseandweaponssystems,putsystemsengineeringsquarelyonthemap oftheengineeringworld.Systemsengineeringhasitsownextensivebodyof knowledgeandpractices.Inwhatfollows,welookathowtoapplyafewkey approaches,relevanttothedesignofintelligentsystems.
AIsystemsareindeeddangerous,butsoaremanytechnologiesandsituations welivewitheveryday.Electricity,water,andaircanallbeverydangerousdependingontheirlocation,speed,andsize.Talltreesnearhomescanbedangerous
whenstormscomethrough.Fastmovingmulti-tonmachinescontainingvolumes ofexplosiveliquidsaredangeroustoo(automobilesagain).Tothesystemsengineer,dangersandrisksaresimplypartofwhatmustbeconsideredwhendesigning andbuildingsystems.Aswe’lldemonstrateinthischapter,thesystemsengineer hasconcepts,methods,andtoolstodealwiththebroadcategoryofdangerinsystems,orassystemsengineersliketocallit,safety.First,weintroduceapairof simpleideastohelpusthinkaboutAIsystemsmoreclearly–thehumananalogy, andthesystemsanalogy.
1.2TheHumanAnalogy
Thefirsttechniquethatcanbeappliedwhenconfrontingsomeofthedifficultiesof anintelligentsystemistocomparethesituationtooneinwhichahumanbeingis performingtheroleinsteadofanintelligentsystem.Weask,howwouldwetrain, manage,monitor,andcontrolahumanassignedthesametaskweareassigning totheintelligentsystem?Afterallahumanbeingisanintelligentsystem,and certainlyfarmoreunpredictablethananyAI.Humanbeingscannotbeexplicitly programmed,andtheymaintainasometimesfrustratingabilitytoforget(orreject) instructions,developnewmotivations,andactinunpredictableways,evenways thatruncountertocleargoalsandincentives.Ifwecanseewaystodealwitha humaninthesituation,perhapswegaininsightintohowtodesignandmanage anAIinasystem.
Totakejustoneexamplefornow,considerthequestionofteachinganAIto driveacarsafely.Usingthehumananalogyleadsustoask,howdoweteacha humanbeing,normallyanadolescenthuman,beingtodriveacarsafely?Inadditiontomechanicalandsafetyinstruction,weincludesomesafeguardsintheform ofinstillingfearofinjury(rememberthosegruesomedriver’seducation“scare” films?),alongwithinstillingfearofthebreakingthelawanditsconsequences, plussomeappealstoconscience,concernforyourownandotherssafety,andother psychologicalpersuasions.Asasociety,webackupthesethreatsandfearswith asystemoflaws,police,courts,fines,andprisons,whichexertinfluencethrough incentivesanddisincentivesontheyoungdriver.Noneofthispreventsdangerous driving,butthesystemhelpskeepitinchecksufficientlytoallow16-year-oldsto drive.Ifitdoesn’t,wecanmakeadjustmentstothesystem,likeraisingthedriving age,orstiffeningthepenalties.
Thehumananalogyworksbecausehumanbeingsare,fromasystemsengineeringperspective,theworstkindofsystem.Theyarenotgovernedbywiring orprogrammingandtheirbehaviorpatterns,howeverwell-establishedthrough experience,canstillchangeatanymoment.Atthesametime,humanbehavioris notrandominthemathematicalsense.Humansactaccordingtotheirownviews
1ArtificialIntelligence,ScienceFiction,andFear
ofwhatisintheirownbestinterestandtheinterestsofothers,howeverwrongor distortedothersmayviewtheirchoices.Theworstcriminalshavereasonsforwhy theydidwhattheydid.
Thehumananalogyisuseful,bothforreasoningaboutnotonlyhowtokeepa systemsafe,butalsoworkswhenthinkingabouthowthesystemshouldperform. Ifwearebuildingasurveillancecameraforhomesecurity,wemightaskhowwe woulduseahumanbeingforthattask.Ifweweretohireasecurityguard,we wouldconsiderwhatinstructionsweshouldgivetheguardabouthowtoobserve, whattowatchfor,whentoalertus,whattorecord,whattoignore,etc.,andreasoningabouttherightinstructionscouldleadustoabettersystemdesignforthe intelligent,automatedguardsystem.
Whenweusethehumananalogy,weshouldalsoconsiderthetypeofhuman beingweareusingasourexemplar.Arewepicturingamiddle-agedadult,achild, adisabledperson,ahighlyeducatedscientist,oranemotionalteenager?Each presentsopportunitiesandchallengesfortheintelligentsystemdesigner.Educationalsystems,forexample,aredesignedforparticularkindsofhumanbeings, andimplementdifferingrulesandpracticesforyoungchildren,thementallyill, teenagers,prisoners,graduatestudents,rowdysummercampkids,andexperiencedprofessionals.Somesituationsthatworkfineformatureadultscan’ttolerate aboisterousorless-than-carefulcollegestudent.Systemsengineersmustconsider thesamekindofvariabilityinthe“personality”ofanAIcomponentinasystem.
1.3TheSystemsAnalogy
Thementaltechniquecalled thesystemsanalogy involvesmakingacomparison betweenanAIsystemandanexistingsystemwithsimilarattributes,oftenresultinginabroaderperspectivethanconsideringtheAIinisolation.Takinganother automotiveexample,weconsiderhowwemightmanageandcontrolpotentially dangerousmachines,containingtanksofexplosiveliquidsandusingsmallexplosionsforpropulsion,movingatspeedsfromacrawltoover80mph,inareaswhere unprotectedpeoplemaybewalkingaround.Whethertheseinherentlydangerousmachinesarecontrolledbyhumandrivers,computers,ortrainedmonkeys, weneedasystemtomakecartravelassafeaspossible.Trafficlights,lanemarkings,speedlimits,limitedaccessroadsfortravelathighspeeds,andvehiclesafety deviceslikelighting,seatbelts,crumplezones,andairbagsareallpartoftheextensivesystemthatmakestheexistenceanduseofautomobilesassafeaswecan practicallymakeit.
Becausehuman-drivenvehicletraffichasbeenwithussolong,andissofamiliar, wemightbetemptedtothinkthatthesystemisasgoodaswecanmakeit–that systemsthinkingaboutautosafetyhaslongagoreacheditspeak.System innovations,however,seemtostillbepossible.In2000,the divergingdiamond
interchange wasimplementedintheUnitedStatesforthefirsttime,andincreased thesafetylevelsatfreewayinterchangesbyeliminatingdangerouswideleftturns. The superstreet designconceptwasintroducedin2015andisreportedtoreduce collisionsbyhalfwhiledecreasingtraveltime.Soevenincompletelyhuman systems,whatwewilllatercallpeoplesystems,innovationsthroughsystems thinkingarepossible.We’llapplythesamekindofthinkingtointelligentsystems. ByconsideringtheentiresystemwithinwhichanAIsubsystemoperates,and comparingittosimilar“nonintelligent”systems,wecanavoidthesimplisticcategorizationofnewtechnologiesaseithersafeornotsafe.Isnuclearpowersafe?Of coursenot.Nuclearreactorsareinherentlydangerous,butbydesigningasystem aroundthemofprotectivebuildings,controlsystems,failsafesubsystems,redundantbackupsystems,andemergencyshutdowncapabilities,wemakethesystemsafeenoughtousereliably.Infact,thecatastrophicfailuresofnuclearpower plantsareusuallyfromalackofgoodsystemsthinkingandsystemsengineering, notasadirectresultoftheinherentdangerofnuclearsystems.Thedisasteratthe FukushimaDaiichiplantwasmainlyduetofloodedgeneratorswhichhadunfortunatelybeenlocatedonalowerlevel,makingthemvulnerabletoarareflood watersituation.
Therightsystemdesigndoesnotmakethedangeroustechnologysafer–it makestheentiresystemsafeenoughforproductiveuse.Asacivilization,wedo nottendtoshyawayfromdangeroustechnologies.Instead,weembracethem, andengineersystemstomakethemassafeaspossible.Electricity,naturalgas, internalcombustionengines,airtravel,andevenbicycleridingarealldangerous intheirownways,butwithgoodsystemsthinkingandsystemsengineering,we makethemsafeenoughtouseandenjoy–eitherbyreducingthelikelihoodof injuryordamage(speedlimitsandtrafficsignals),orbyreducingthepotential harm(airbags).Evenprohibitingtheuseofatechnologybylaw(thinkDDT orasbestosinsulation)ispartofthesystemthatmakesinherentlydangerous systemssafer.Therearethosewhothinkweshouldsomehowprohibitwholesale thedevelopmentofAItechnologyduetoitsinherentdanger,butmostofthe worldisstillhopefulthatwe’llbeableengineersystemsthatuseAIandthen makethemsafeenoughforregularuse.
Withthatasanintroduction,let’sconsiderthemainperceivedandactualdangersofartificialintelligencetechnologyandproposesomesolutionsbasedonsystemsengineeringandsystemsthinking.
1.4KillerRobots
HeadingthelistofAIdangers,supportedbystrongvisualimagesandstorylines frommovieslikeTheTerminatorseriesanddozensofothers,iswhatwe’llreferto as“killerrobots.”ThemainideaisthatanAIwilloneday“wakeup,”becoming
conscious,sentient,andabletoformitsowngoalsandthenbegintocarrythem out.TheAImaydecidethatitisinitsbestinterest(orperhapseventhebestinterest oftheworld)tokillorenslavehumanbeings,anditproceedstoexecutethisplan, withalarmingspeedandeffectiveness.Isthispossible?Theoreticallyyes,butlet’s applythesystemsanalogyandthehumananalogytoseehowwecansensibly avoidsuchascenario.
Inaway,thekillerrobotscenariohasbeenpossibleformanyyears.Acomputer, evenasmall,simpleone,couldbeprogrammedtoinstructamissilecontrolsystem,orperhaps all missilecontrolsystems,tolaunchmissiles,andkillmostof thepeopleonearth.Herewetakethesystemsanalogy,andaskwhythisdoesn’t happen.Theansweriseasytoseeinthiscase–wesimplydonotallowpeople toconnectcomputerstomissilecontrolsystems,airtrafficcontrolsystems,traffic lightsystems,oranyofthehundredsofsensitivesystemsthatmanageourcomplex world.Wegoevenfurtherandmakeitdifficultorimpossibleforanothercomputer tocontrolthesesystems,evenifphysicallyconnected,byusingencryptedcommunicationandsecureaccesscodes.TheassumptionthatanAI,oncebecoming sentient,wouldhaveinstantandtotalaccessandcontroloveralldefense,financial,andcommunicationsystemsintheworldisbornmoreofHollywoodwriters thansensiblecomputerscience.
Illustratedbeautifullyin ThreeLawsLethal,thefascinatingnovelbyDavidWalton,thislimitationisexperiencedbyIsaac,anAIconsciousnessthatemergesfrom alarge-scalesimulation,andfindsthat“he”cannotaccessencryptedcomputer networks,can’thackintoanythingthathumanhackerscan’t,andcan’tevenwrite acomputerprogram.Hecleverlyaskshiscreatorifsheisabletomanipulatethe geneticstructureofherownbrain,orevenexplainhowherownconsciousness works.Shecan’t,andneithercanIsaac.Alsorelevanttothesubjectofthe“killer robot”dangerofAI,isIsaac’sobservationofhowvulnerableheistohumanbeings, whocouldremovehismemorychips,takeawayhisCPU,deletehisprogram,or simplycutthepowertothedatacenter,“killing”himaccidentallyoronpurpose. Themostpowerfulofhumanbeingsarevulnerabletodisease,accident,arrest, ormurder.“Whohasthemorefragileexistence–thehumanortheAI?”Isaac wonders.
Thesystemanalogyleadsustothesomewhatobviousconclusionthatifwedon’t wantournewAItocontrolmissiles,weshoulddoourbesttoavoidconnectingit tothemissilecontrolsystem.Butcouldasufficientlyadvanced,sentientAIwork togainaccesstosuchsystemsifitwantedto?Formingsuchindependentgoals andformulatingplanstoachievethemisnotjustsentience,buthighintelligence, andisunlikelyintheforeseeablefuture.ButevenifasentientAIdidemerge, itislikelytoturntothesametechniqueshumanhackersusetobreakintosystems,andforthemostpart,theserelyonsocialengineering–phishinge-mails, bribery,extortion,blackmail,orothertrickery.AnAIhasnoparticularadvantage
9 overcomputer-assistedhumansathackingandintrusiontactics.Toputitanother way,ifthekillerrobotscenariowerepossible,itwouldfirstbeexploitedbyhuman beingswithcomputers,notbysentientAIs.Usingthehumananalogy,weexplore howwewouldprotectourselvesfromthatdangeroussituation.
Considerthatakillerrobotcanconsisteitherofastand-alonesystem,literallyarobot,created,andsetonakillingmission,ormoreplausibly,adefense orwarfaresystemintentionallyredirectedorhackedtoachieveevilintent.There aretwowaystoapplythehumananalogyhere.First,takethecaseofthekiller robot–howdowepreventandcontrolthe“killerhuman”?Andthensecond, howdowepreventakillerhumanfromusinganexistingcomputerizeddefense orweaponssystemtocarryoutamurderousintent?
1.5WatchingtheWatchers
Killerhumans–humanbeingswhotakeitonthemselvestokilloneormoreother humanbeingshavebeenwithusinhumancivilizationsincetheverybeginning. IntheChristianandJewishtradition,ittookonlyatotaloftwohumanbeings onearthforthefirstmurdertooccur.Howdowepreventhumansfromkilling eachother?Shortanswer:wedon’t.Isiteventheoreticallypossibletoprevent humansfromkillingeachother?Possibly.IsaacAsimovinhisnovel I,Robot,and thesubsequentmoviestarringWillSmith,demonstratethatifthemostimportant goalistoprotecthumanlife,thenthemosteffectiveapproachistoimprisonall humans.Soyes,wecancompletelypreventhumansfromkillingotherhumans, butonlywithcompromisesmostwouldfindunacceptable.Instead,humanshave inventedsystemstomitigatetheriskofhumanskillingeachother.Wehavedevelopedsocialtaboosagainstmurder,andsystemsincludinglaws,police,courts,and prisonswhichserveasasignificantdeterrenttocarryingoutawishtomurder anotherhumanbeing.Thetaboosometimesextendstoallkillingofhumans,not justmurderandresultsinoppositiontowarandtocapitalpunishmentinsome societies.Somesocietiesallowkillingincasessuchasdefenseofoneselforothers. Thekilling-mitigationsystemgoesevenfurther–eventhethreattokillsomeone isacrimeinsomesocieties,andlikelypreventssomekilling,ifsuchthreatsare prosecuted.Wehavedoneourbesttodevelopquiteacomplexsystemtoreduce killing,whilestillmaintainingotherimportantvaluessuchasfreedom,safety,and justice.Forcontrast,notethatourcurrentsocietyplacesverylittleornovalueon thepreventionofwildanimalskillingotherwildanimals.Wequiteliterallylet thatsystemrunwild.
Thehumananalogysuggestsweapplysomeofthesesystemprotectionideasto thepreventionofarobotkillingspree.Forexample,asystemcomponentcouldact asthepolice,watchingovertheactionsofothersystemcomponentsandactingto
restrainthemiftheyviolatetherules,orperhapseveniftheygivetheimpressionof anintentiontoviolatetherules.Systemcomponents,includingAIcomponents, couldfindtheirfreedomrestrictedwhentheystrayoutsidedesignedboundsof acceptablebehavior,orperhapstheymightbeletoffwithjustawarningorgiven probation.Inextremecases,policesubsystemscouldbegiventheauthoritytouse deadlyforce,stoppingawaywardAIcomponentbeforeitcausesharmordamage.Ofcourse,someone(oranotherkindofsubsystem)wouldneedtowatchthe watchers,andbesurethepolicesubsystemsaredoingtheirjobs,butnotoversteppingtheirauthority.
Subsystemswatchingovereachothertomakeasystemsaferisnotanew concept.Homethermostatshavefordecadesbeenequippedwithalimitdevice thatpreventsthehometemperaturefromdroppingbelowacertainlimit,typically 50 ∘ F,regardlessofthethermostat’ssetting,preventingathermostatmalfunction, orinattentivehomeowner,fromthedisasteroffrozenandburstpipes.So-called “watchdog”devicesmaycontinuallycheckforanInternetconnectionandif lost,rebootmodemsandrouters.Autoandtruckenginescanbeequippedwith governorsthatlimittheengine’sspeed,nomatterhowmuchthehumandriver pressesontheaccelerator.Itisimportanttoseethatthesedevicesareseparate fromthepartofthesystemthatcontrolsthenormalfunction.Theengine governorisacompletelyseparatedevice,notanadditiontotheacceleratorand fuelsubsystemwhichnormallycontrolstheenginespeed.Therearelikelysome safeguardsinthatsubsystemtoo,butthegovernoronlydoesitsworkafterthe mainsubsystemisfinished.Toanthropomorphize,it’slikethegovernorissaying totheacceleratorandfueldeliverysubsystem,“Idon’treallycarehowyouare controllingtheenginespeed;youdowhateveryouthinkbest.Butwhateveryou do,I’mgoingtobelookingattheenginespeed,andifitgetstoofast,I’mgoing toslowitdown.Iwon’teventellyouI’mdoingit.I’mnotgoingtointerferewith youoraskyoutochangeyourbehavior.I’lljuststepinandslowtheenginedown ifnecessary,forthegoodofthewholesystem.”Thisistheidealattitudeofthe humanpolice–letthecitizensactfreelyunlesstheygooutsidethelaw,andthen acttorestrainthem,forthegoodofthewholesociety.
Thewatchdogsystemitselfmustbegovernedofcourse,andtrustedtodoits job,sosomewonderifonecomplexsystemwatchingoveranother,increasingthe overallsystem’scomplexity,isworthit.Butwatchdogsystemscanbemuchsimplerthanthemainsystem.Withtheautoortruckgovernor,thesystemcontrolling enginespeedbasedonacceleratorinput,fuelmixture,andengineconditioncan becomplexindeed,butthegovernorissimple,caringonlyabouttheenginespeed itself.Insomeengines,thegovernorisasimplemechanicaldevice,withoutany electronicsorsoftwareatall.InanAIsystem,thewatchdogwillinvariablybe muchsimplerthantheAIsystemitself.Evenifthewatchdogisrelativelycomplex,itmaybeworthit,sinceitsadditionmeansthatboththemainsystemand
thewatchdogmust“agree”oncorrectfunctioningofthesystem,oraproblem mustbesignaledtothehumandriveroruser.
Continuingthehumananalogy,wenotethatjusthavingpoliceisnotenough. Iftheonlyprotectioninahumansocietywerepolice,corruptionwouldbeagreat temptation–therewouldbenoonetowatchthewatchers.Inpeacetimehuman societiesthepoliceanswertothecourts,andthecourtsmustanswertothepeople, throughelectedrepresentatives.AttorneysmustanswertotheBarassociationand aresubjecttothelawsmadebyelectedrepresentativesofthepeople.Judgesmaybe electedorappointed,andtheiractionsaresubjecttoreview.It’sacomplexsystem ofchecksandbalancesthatpreventsthesystemfrombeingcorruptedormisused throughlocalizeddecision-makingbyanysinglepartofthesystem.
Inanintelligentsystem,AI-basedsubsystemsmakedecisionsafterbeingtrained onnumerousexamples,asdescribedinChapter3.DesignerstrytomaketheseAIs asgoodatdecision-makingastheycanbe,butitisstillpossibleforbaddecisions tobemadeforavarietyofreasons.Baddecisionscanbemadebyanysystem,AI ornot,butinnon-AIsystems,itiseasiertoseetherangeofpossibledecisionsby examiningthehardwareandinspectingthesoftwaresourcecode.Insafety-critical systems,sourcecodemustbeinspectedandverified.Inanintelligentsystemcontaininganexample-trainedAIcomponent,thisisnotpossible.Thecodeusedto implementtheneuralnetworkcanbeexamined,butthedecisionsaremadeby asetofnumbers,thousands,orevenmillionsofthem,whicharesetduringthe trainingprocess.ExaminingthesenumbersdoesnotclearlyrevealhowtheAIwill makedecisions.It’sabitlikehumanbeing–withourcurrentmedicalknowledge, examiningthebrainofahumanbeingdoesnotrevealhowthathumanbeingwill makedecisionseither.
It’snotthattheAIcomponentisunpredictable,orunreliableatdecisionmaking–it’smorethathumanscan’treadilyseethelimitsoftheAI’sdecisions byinspectingitssoftwaresourcecode.Thesystemsengineercouldcompensate forthisunknownbydesigningawatchdogcomponentthatwatchesoverthe AIandifitmakesanout-of-boundsdecision,stepinandstopthesystem,or atleastgetfurtherhumanauthorizationbeforeproceeding.Designersadd watchdogcomponentstointelligentsystemsforthesamereasonswehavepolice inhumansystems.ThepossiblerangeofbehaviorofanAIcomponent(ora human)issimplymuchgreaterthanissafeforthesystem,andthere’salwaysthat chance,regardlessofparentage,provenance,training,orupbringing,thattheAI (orhuman)willonedayactoutofbounds.Thewatchdogsneedtobepresent andvigilant.
Wehaveconsideredthequestionofhowweprotectourselvesagainstanintelligentsystem,evenwhenitcontainsAIcomponentsthatcanconceivablymake unfortunateordangerousdecisionsincertaincases.Byapplyingthehumananalogy,wesuggestthatotherpartsofthesystem,analogoustopolice,courts,and
1ArtificialIntelligence,ScienceFiction,andFear
prisons,canservetosuperviseandmakesuretheentiresystemstaysinbounds. Wecanapplythehumananalogyanotherwaybyconsideringhowwecanprotect ourcriticalsystemsfromunwantedintrusionandsubversionbyAIs.Thehuman analogysuggestsweaskhowweprotectimportantsystemsfrom human intrusion, otherwiseknownashacking.
1.6CybersecurityinaWorldofFallibleHumans
Thefieldofcybersecurity–theprotectionofcomputersystemsfromunwanted intruders–conjuresreadyimagesofasuperhacker,typingfuriouslyata keyboardforabout60seconds,andgainingaccesstoahighsecuritydefense, governmentorcorporatesystem,anditsvaluablestoresofdata.Whilethereare someintrusionsthatareperpetratedthroughpurelytechnologicalmeans,itisby farmorecommonforwould-becyberintruderstorelyontheweaknessesofother humanbeings.
Evenpurelytechnologicalintrusionsoftenrelyonpoorsecuritydecisions madebyhumansinthedesignorimplementationofthesysteminthefirstplace. Itdoesn’ttakeasecurityexperttoknowthatifasystemsendsunencryptedaccess dataoverwirelessconnections,thedatacanbeintercepted,modified,andre-sent, trickingthesystemintodoingtheintruder’sbidding.Systemsdesignedthisway, andtherearemany,areaskingforintrusionandexploitation.Well-designed systemsareextremelydifficulttohackintousingjustacomputerfromtheoutside.
Mosthackingintrusionsbeginwithsocialengineeringbymanipulatingor exploitingtheweakestlinkinthesystem–thehumanbeing.Howmanyhuman beings,facedwiththeneedtousemultiple,complexpasswordsonadailybasis, writethemdown,perhapsonayellowstickynoteaffixedtoamonitororinside an(unlocked?)deskdrawer.Intruders,posingasmaintenancepeople,plant waterers,orjanitorialstaffcanwanderthroughanoffice(orpaysomeonetodo so)andsnapphotosofenoughpasswordstokeepthembusyforquiteawhile.Best ofall(fortheintruders)thesourceofthecompromisedsecurityisuntraceable andmayremainunknownindefinitely.Withtherightpasswords,ahackercan simplysetupadditionalaccountsonthesystem,givingthempermanentaccess, eveniftheoriginalpasswordsarelaterchanged.
Malicioussoftwarecodethatgrantsanoutsideraccesstoasecuresystemis easytoprocureanduse,butitmustbeinstalledonthesecuresystem.Shockingly, studieshaveshownthatsmallUSBjumpdrivesladenwithmalwareandscattered inacompany’sparkinglot,willmoreoftenthannotbepickedupandinserted intoemployees’computersbytheendoftheday.Printingthecompany’sown logoonthedrivedoublestheeffectivenessoftheploy.Theevenmorecommon wayofintroducingmaliciouscodeintoadesiredtargetsystemistoattachittoa
