LISTOFCONTRIBUTORS
AdityaAcharya UniversityofBirmingham,UK
LeifAzzopardi StrathclydeUniversity,UK
NikolaBanovic CarnegieMellonUniversity,USA
XiaojunBi StonyBrookUniversity,USA
DuncanP.Brumby UniversityCollegeLondon,UK
PaulCairns UniversityofYork,UK
XiuliChen UniversityofBirmingham,UK
JessieChin UniversityofIllinois,USA
AnindK.Dey CarnegieMellonUniversity,USA
AlanDix UniversityofBirmingham,UK
HyoJinDo UniversityofIllinois,USA
CamilleFayollas UniversitédeToulouse,France
Wai-TatFu UniversityofIllinois,USA
MingkunGao UniversityofIllinois,USA
OtmarHilliges AdvancedInteractiveTechnologiesLab,ETHZürich,Switzerland
AndrewHowes UniversityofBirmingham,UK
TakeoIgarashi UniversityofTokyo,Japan
ChristianP.Janssen UtrechtUniversity,TheNetherlands
AndreasKarrenbauer Max-Planck-InstitutfürInformatik,Germany
YukiKoyama NationalInstituteofAdvancedIndustrialScienceandTechnology,Japan
PerOlaKristensson UniversityofCambridge,UK
TuomoKujala UniversityofJyväskylä,Finland
RichardL.Lewis UniversityofMichigan,USA
Q.VeraLiao IBMThomasJ.WatsonResearchCenter,NewYork,USA
JenniferMankoff CarnegieMellonUniversity,USA
CéliaMartinie UniversitédeToulouse,France
RoderickMurray-Smith UniversityofGlasgow,UK
AnttiOulasvirta AaltoUniversity,Finland
TomOuyang Google,Inc.,USA
PhilippePalanque UniversitédeToulouse,France
DarioD.Salvucci DrexelUniversity,USA
BrianSmith ColumbiaUniversity,USA
HaroldThimbleby SwanseaUniversity,UK
JohnWilliamson UniversityofGlasgow,UK
ShuminZhai Google,Inc.,USA
GuidoZuccon QueenslandUniversityofTechnology,Australia
Introduction xiaojunbi, andrewhowes, perolakristensson, anttioulasvirta, johnwilliamson
Thisbookisconcernedwiththedesignofinteractivetechnologyforhumanuse.Itpromotesanapproach,calledcomputationalinteraction,thatfocusesontheuseofalgorithms andmathematicalmodelstoexplainandenhanceinteraction.Computationalinteraction involves,forexample,researchthatseekstoformallyrepresentadesignspaceinorderto understanditsstructureandidentifysolutionswithdesirableproperties.Itinvolvesbuilding evaluativemodelsthatcanestimatetheexpectedvalueofadesigneitherforadesigner,or forasystemthatcontinuouslyadaptsitsuserinterfaceaccordingly.Itinvolvesconstructing computationalmodelsthatcanpredict,explain,andevenshapeuserbehaviour.
Whileinteractionmaybeapproachedfromauserorsystemperspective,allexamples ofcomputationalinteractionshareacommitmenttodefiningcomputationalmodelsthat gaininsightintothenatureandprocessesofinteractionitself.Thesemodelscanthenbe usedtodrivedesignanddecisionmaking.Here,computationalinteractiondrawsona longtraditionofresearchonhumaninteractionwithtechnologyapplyinghumanfactors engineering(Fisher,1993;HollnagelandWoods,2005;SandersandMcCormick,1987; Wickensetal.,2015),cognitivemodelling(PayneandHowes,2013;Anderson,2014; KierasandHornof,2017;Card,Newell,andMoran1983;GrayandBoehm-Davis,2000; Newell,1994;Kieras,Wood,andMeyer,1997;PirolliandCard,1999),artificialintelligence andmachinelearning(SuttonandBarto,1998;BrusilovskyandMillan,2007;Fisher,1993; Horvitzetal.,1998;Picard,1997;Shahriarietal.,2016),informationtheory(Fittsand Peterson,1964;Seow,2005;Zhai,2004),designoptimization(LightandAnderson,1993; Eisenstein,Vanderdonckt,andPuerta,2001;GajosandWeld,2004;Zhai,Hunterand Smith,2002),formalmethods(Thimbleby,2010;Dix,1991;HarrisonandThimbleby, 1990;Navarreetal.,2009),andcontroltheory(Craik,1947;Kleinman,Baron,andLevison, 1970;JagacinskiandFlach,2003;SheridanandFerrell,1974).
ComputationalInteraction.AnttiOulasvirta,PerOlaKristensson,XiaojunBi,AndrewHowes(Eds). ©OxfordUniversityPress2018.Published2018byOxfordUniversityPress.
Computationalinteractionisascienceoftheartificial(Simon,1996),wheretheobjectof researchistheconstructionofartefactsforstatedgoalsandfunction.HerbertSimontook upconstructionasthedistinctivefeatureofdisciplineslikemedicine,computerscience, andengineering,distinguishingthemfromnaturalsciences,wherethesubjectisnatural phenomena,andfromarts,whichmaynotshareinterestinattaininggoals:‘Engineering, medicine,business,architecture,andpaintingareconcernednotwiththenecessarybutwith thecontingent,notwithhowthingsarebutwithhowtheymightbe,inshort,withdesign’ (Simon,1996,p.xiii).
Simonmadethreeimportantobservations,whichweshare,aboutsciencesoftheartificial.First,hedistinguishedthe‘innerenvironment’oftheartefact,suchasanalgorithm controllingadisplay,andthe‘outerenvironment’,inwhichtheartefactservesitspurpose. Whenwedesignauserinterface,wearedesigninghowtheouterandtheinnerenvironmentsaremediated.Boththehumanandthetechnologyalsohaveinvariantproperties thattogetherwiththedesignedvariantpropertiesshapetheprocessandoutcomesof interaction(seeFigureI.1).Thefinalartefactreflectsadesigner’simplicittheoryofthis interplay(CarrollandCampbell,1989).Whenwedesigncomputationalinteraction,we aredesigningoradaptinganinnerenvironmentoftheartefactsuchthatitcanproactively, andappropriately,relateitsfunctioningtotheuserandhercontext.Incomputational interaction,thetheoryormodel,isexplicitandexpressedincodeormathematics.Second, Simonelevatedsimulationasaprimemethodofconstruction.Simulations,ormodelsin general,allow‘imitating’realityinordertopredictfuturebehaviour.Simulationssupportthe discoveryofconsequencesofpremiseswhichwouldbedifficult,oftenimpossible,toobtain withintuition.Incomputationalinteraction,modelsareusedofflinetostudytheconditions ofinteraction,andtheymaybeparameterizedinareal-timesystemwithdatatoinferuser’s intentionsandadaptinteraction.Third,Simonpointedoutthatsincetheend-productsare artefactsthataresituatedinsome‘outerenvironments’,theycanandshouldbesubjectedto
FigureI.1 Interactionisanemergentconsequenceofbothvariantandinvariantaspectsoftechnologyandpeople.Technologyinvariantsinclude,forexample,currentoperatingsystemsthatprovide anecosystemfordesign.Technologyvariantsincludeprogrammedinterfacesthatgiverisetoapps, visualizations,gesturalcontrol,etc.Aperson’sinvariantsincludebiologicallyconstrainedabilities, suchastheacuityfunctionofhumanvision.Aperson’svariantsconsistoftheadaptivebehavioural strategies.Todesigninteractionistoexplainandenhancethevariantaspectsgiventheinvariants (bothhumanandtechnology).Incomputationalinteraction,variantaspectsarechangedthrough appropriatealgorithmicmethodsandmodels.
Variant Invariant
empiricalresearch.Designrequiresrigorousempiricalvalidationofnotonlytheartefact,but alsoofthemodelsandtheoriesthatcreatedit.Sincemodelscontainverifiableinformation abouttheworld,theyhavefacevaliditywhichisexposedthroughempiricalobservations. Theydonotexistasmeresourcesofinspirationfordesigners,butsystematicallyassociate variablesofinteresttoeventsofinteraction.
However,computationalinteractionisnotintendedtoreplacedesignersbuttocomplementandboosttheveryhumanandessentialactivityofinteractiondesign,whichinvolves activitiessuchascreativity,sensemaking,reflection,criticalthinking,andproblemsolving (Cross,2011).Weholdthatevenartisticandcreativeeffortscangreatlybenefitfrom awell-articulatedunderstandingofinteractionandmasteryofphenomenacapturedvia mathematicalmethods.Byadvancingthescientificstudyofinteractivecomputingartefacts, computationalinteractioncancreatenewopportunitiesandcapabilitiesincreativeand artisticendeavors.Weanticipatethattheapproachesdescribedinthisbookareuseful intheinteractiondesignprocessesofideating,sketching,andevaluating,forexample. Computationalinteractioncanalsobestudiedanddevelopedinbroadercontextsofsociotechnicalsystems,wherepower,labour,andhistoricalsettingsshapeinteraction.
Thefundamentalideasofcomputationalinteractionhavebeenpresentformanyyears. However,thereisnowstrongmotivationtocollect,rationalize,andextendthem.Intheearly daysofinteractiondesign,designspaceswererelativelysimple;inputdevices,forexample, wereexplicitlydesignedtobesimplemappingsofphysicalactiontodigitalstate.However, computersystemsarenowvastlymorecomplex.Inthelasttwodecades,interactionhasbrokenoutfromthelimiteddomainsofworkstationsforofficeworkerstopervadeeveryaspect ofhumanactivity.Inthecurrentmobileandpost-PCcomputingera,newtechnologieshave emergedthatbringnewchallenges,forwhichthetraditionalhand-tunedapproachtodesign isnotwellequipped.Forexample,wearablecomputing,augmentedandvirtualreality,and customizableinteractivedevices,poseincreasinglywickedchallenges,wheredesignersmust consideramultiplicityofproblemsfromlow-levelhardware,throughsoftware,alltheway tohumanfactors.Intheseincreasinglycomplexdesignspaces,computationalabstraction andalgorithmicsolutionsarelikelytobecomevital.
Increasingdesigncomplexitymotivatesthesearchforascalableinteractionengineering thatcancomplementandexceedthecapabilitiesofhumandesigners.Acontentionofthis bookisthatacombinationofmethodsfrommodelling,automation,optimization,and machinelearningcanofferapathtoaddressthesechallenges.Analgorithmicapproach tointeractionoffersthehopeofscalability;anopportunitytomakesenseofcomplex dataandsystematicallyandefficientlyderivedesignsfromoverwhelminglycomplexdesign spaces.Thisrequirespreciseexpressionofinteractionproblemsinaformamenableto computationalanalysis.
Thisbookispartofanargumentthat,embeddedinaniterativedesignprocess,computationalinteractiondesignhasthepotentialtocomplementhumanstrengthsandprovide ameanstogenerateinspiringandelegantdesigns.Computationalinteractiondoesnot excludethemessy,complicated,anduncertainbehaviourofhumans.Neitherdoesitseek toreduceuserstomechanisticcaricaturesforeaseofanalysis.Instead,acomputational approachrecognizesthattherearemanyaspectsofinteractionthatcanbeaugmentedbyan algorithmicapproach,evenifalgorithmsarefallibleattimesandbasedonapproximationsof humanlife.Furthermore,computationalinteractionhasthepotentialtoreducethedesign
errorsthatplaguecurrentinteractivedevices,someofwhichcanbelifeending.Itcan dramaticallyreducetheiterationscyclesrequiredtocreatehigh-qualityinteractionsthat otherapproachesmightwriteoffasimpossibletouse.Itcanspecializeandtailorgeneral interactiontechniquessuchassearchfornichetasks,forinstance,patentsearchesand systematicreviews.Itcanexpandthedesignspaceandenablepeopletointeractmore naturallywithcomplexmachinesandartificialintelligence.
Inparallelwiththeincreasedcomplexityoftheinteractionproblem,therehavebeen rapidadvancesincomputationaltechniques.Atthefundamentallevel,rapidstridesin machinelearning,datascience,andassociateddisciplineshavetransformedtheproblem spacethatcanbeattacked,generatingalgorithmsandmodelsoftransformativepower. Atthepracticallevel,affordablehigh-performancecomputing,networkedsystems,and availabilityofoff-the-shelflibraries,datasets,andsoftwareforcomputationalmethods, makeitfeasibletoanalyseandenhancethechallengingproblemsofinteractiondesign. Thesearedevelopmentsthatshouldnotbeignored.
Thisbooksetsoutavisionofhumanuseofinteractivetechnologyempoweredby computation.Itpromotesananalytical,algorithmic,andmodel-ledapproachthatseeks toexploittherapidexpansionincomputationaltechniquestodealwiththediversification andsophisticationoftheinteractiondesignspace.AsnotedbyJohnCarroll(1997),andby othersmanytimesafter,researchonhuman-computerinteractionhashadproblemsassimilatingthevarietyofmethodologies,theories,problems,andpeople.Theimplementation oftheoreticalideasincodecouldbecomeasubstrateandanexusforcomputerscientists, behaviouralandsocialscientists,anddesignersalike.Ifthispromiseistobefulfilled,then seriouseffortmustbemadetoconnectmainstreamsearchinhuman-computerinteraction tocomputationalfoundations.
I.1Definition
Definingafieldisfraughtwithdangers.Yettherearetworeasonstoattemptonehere:(i)to provideasetofcoreobjectivesandideasforlike-mindedresearchersinotherwisedisparate fieldstocoalesce;(ii)tohelpthosenotyetusingcomputationalmethodstoidentifythe attributesthatenablethosemethodstobesuccessfulandunderstandhowthiswayof thinkingcanbebroughttobearinenhancingtheirwork.Whileanydefinitionisnecessarily incomplete,thereisastrongsensethatthereisanucleusofdistinctiveandexcitingideas thatdistinguishcomputationalfromtraditionalhuman-computerinteractionresearch.The goalofourdefinitionistoarticulatethatnucleus.
Definition Computationalinteractionappliescomputationalthinking—thatis,abstraction,automation,andanalysis—toexplainandenhancetheinteractionbetweenauser (orusers)andasystem.Itisunderpinnedbymodellingwhichadmitsformalreasoningand involvesatleastoneofthefollowing:
•awayofupdatingamodelwithobserveddatafromusers;
•analgorithmicelementthat,usingamodel,candirectlysynthesizeoradaptthe design;
•awayofautomatingandinstrumentingthemodellinganddesignprocess;
•theabilitytosimulateorsynthesizeelementsoftheexpecteduser-systembehaviour.
Forexample,thedesignofacontrolpanellayoutmightinvolvefirstproposinganabstract representationofthedesignspaceandanobjectivefunction(forexample,visualsearch performanceandselectiontime)forchoosingbetweenvariants.Itmighttheninvolveusing anoptimizationmethodtosearchthespaceofdesignsandanalysingthepropertiesofthe proposeddesignusingformalmethods.Alternatively,explainingobserveduserbehaviour givenaspecificdesignmightbeexplainedbyfirstproposingabstractrepresentationsofthe informationprocessingcapacitiesofthemind(forexample,modelsofhumanmemoryfor interferenceandforgetting),thenbuildingcomputermodelsthatautomatethecalculation oftheimplicationsofthesecapacitiesforeachparticularvariantacrossthedistributionof possibleusers,andthenanalysingtheresultsandcomparingwithhumandata.
Asinengineeringandcomputerscience,thehallmarkofcomputationalinteraction ismathematical—andoftenexecutable—modellingthatconnectswithdata.However, whilecomputationalinteractionisfoundedonfundamentaltheoreticalconsiderations,it isconstructiveinitsaimsratherthandescriptive.Itcallsforempiricalrigourinevaluation andmodelconstruction,butfocusesonusingcomputationallypoweredmodelstodowhat couldnotbedonebefore,ratherthandescribingwhathasbeendonebefore.Tothisend, itemphasizesgeneratingconstructiveconceptualfoundationsandrobust,replicableand durablemethodsthatgobeyondpointsamplingoftheinteractionspace.
I.2Vision
Theoverarchingobjectiveofcomputationalinteractionistoincreaseourcapacitytoreliably achieveinteractionswithdesirablequalities.Agenericcapacityispreferableoverpoint solutions,becauseweneedtodealwithdiverseusers,contexts,andtechnologies.By seekingshared,transferablesolutionprinciples,thelong-termaimshallbetoeffectthe runaway‘ratcheting’ofabodyofresearchthatbuildsupconstructivelyandcompositionally;somethingthatresearchonhuman-computerinteractionhasstruggledtoachieve inthepast.
Thismotivatesseveraldefiningqualitiesofcomputationalinteraction:verifiable,mathematicaltheorythatallowresultstogeneralize;scalable,theory-ledengineeringthatdoesnot requireempiricaltestingofeveryvariant;transparencyandreproducibilityinresearch;and theconcomitantrequirementforreusablecomputationalconcepts,algorithms,datasets, challenges,andcode.Inmachinelearning,forexample,thepivottoopen-sourcelibraries, largestate-of-the-artbenchmarkdatasets,andrapidpublicationcycleshasfacilitatedboth theuptakeofdevelopmentsandprogressrateofresearch.
Thisvisionopensupseveralpossibilitiestoimprovethewayweunderstand,design,and useinteractivetechnology.
Increaseefficiency,robustness,andenjoyabilityofinteraction Modelsofinteraction, informedbyoftenlarge-scaledatasets,canenablethedesignofbetterinteractions.For
instance,interactionscandemandlesstime;reducethenumberoferrorsmade;decrease frustration;increasesatisfaction,andsoon.Inparticular,computationalapproachescan helpquantifytheseabstractgoals,inferthesequalitiesfromobservabledata,andcreatemechanismsthroughwhichinterfacescanbeoptimizedtomaximizethesequalities. Acomputationalapproachaimstocreateanalgorithmicpathfromobserveduser-system interactiondatatoquantifiablyimprovedinteraction.
Proofsandguarantees Aclearbenefitofsomeapproaches,suchasformal,probabilistic, andoptimizationmethodsisthattheycanofferguaranteesandevenproofsforsomeaspects ofsolutionquality.Thismaybevaluablefordesignerswhoseektoconvincecustomers,or foradaptiveuserinterfaces,thatcanmorereliablyachievedesiredeffects.
Developuser-centreddesign Computationalinteractionnotonlysupportsbutcanreenvisionuser-centreddesign,wheretechniquessuchasparametricinterfaces,data-driven optimization,andmachine-learnedinputrecognitioncreatedirectdatapathsfromusage andobservationstodesignandinteraction.Computationallowsinterfacestobeprecisely tailoredforusers,contexts,anddevices.Structureandcontentcanbelearnedfromobservation,potentiallyonamassscale,ratherthandictatedinadvance.
Reducetheempiricalburden Modelscanpredictmuchofexpecteduser-system behaviour.Interactionproblemscanbedefinedformally,whichincreasesourabilityto reasonandavoidblindexperimentation.Computationalmethodsshouldreducethe relianceonempiricalstudiesandfocusexperimentalresultsonvalidatingdesignsbased onsoundmodels.Strongtheoreticalbasesshouldmove‘shot-in-the-dark’pointsampling ofdesignstoinformedanddataefficientexperimentalwork.
Reducedesigntimeofinterfaces Automation,data,andmodelscansupplanthandtweakinginthedesignofinterfaces.Itshouldbequicker,andlessexpensive,toengineer complexinteractionsiftheminutiaeofdesigndecisionscanbedelegatedtoalgorithmic approaches.
Freeupdesignerstobecreative Inthesamevein,algorithmicdesigncansupport designersintedioustasks.Fromtuningposerecognizersatpublicinstallationstochoosing colourschemesformedicalinstrumentpanels,computationalthinkingcanletdesigners focusonthebigpicturecreativedecisionsandsynthesizewell-designedinteractions.
Harnessnewtechnologiesmorequickly Hardwareengineeringisrapidlyexpanding thespaceofinputdevicesanddisplaysforuserinteraction.However,traditionalresearch onhuman-computerinteraction,reliantontraditionaldesign-evaluatecycles,struggles totacklethesechallengesquickly.Parametricinterfaces,predictive,simulatablemodels, crowdsourcing,andmachinelearningofferthepotentialtodramaticallyreducethetime ittakestobringeffectiveinteractionstonewengineeringdevelopments.
I.3Elements
Computationalinteractiondrawsonalonghistoryofmodellingandalgorithmicconstructioninengineering,computerscience,andthebehaviouralsciences.Despitethe apparentdiversityofthesedisciplines,eachprovidesabodyofworkthatsupportsexplainingandenhancingcomplexinteractionproblems.Theydosobyprovidinganabstract,
oftenmathematical,basisforrepresentinginteractionproblems,algorithmsforautomating thecalculationofpredictions,andformalmethodsforanalysingandreasoningaboutthe implicationsoftheresults.
Thesemodelsdifferinthewaytheyrepresentelementsofinteraction:thatis,howthe properties,events,processes,andrelationshipsofthehumanandtechnologyareformally expressedandreasonedwith.Theymayrepresentthestructureofdesign,humanbehaviour, orinteraction,andtheymayrepresentsometendenciesorvaluesinthosespaces.They allowthecomputertoreasonandtestalternativehypotheseswiththemodel,to‘speculate’ withpossibilities,andtogobeyondobservations.Thiscanbeachievedanalytically,suchas whenfindingtheminimaofafunction.However,inmostcases,themodelsarecomplexand permitnoanalyticalsolution,andanalgorithmicsolutionisneeded,inwhichcasemethods likeoptimizationorprobabilisticreasoningcanbeused.Suchcomputationsmaygiveriseto predictionsandinterpretationsthatareemergentinthesensethattheyarenotobviousgiven lower-leveldata.Anotherpropertysharedbytheseapproachesisthattheinvolvedmodels canbeparameterizedbysomeobservations(data)thatrepresenttheproblemathand.
Wehavecollectedafewkeyconceptsofinteractionanddesign,whichcanbeidentified behindthemanyformalismspresentedinthisbook.Thisis not meanttoexhaustivelylist allconceptsrelatedtocomputationalinteractionandweinvitethereadertoidentify,define, andelaborateadditionalconceptscentraltocomputationalinteraction,suchasdialogues, dynamics,gametheory,biomechanics,andmanyothers.
Information Informationtheory,drawingfromShannon’scommunicationtheory,modelsinteractionastransmissionofinformation,ormessages,betweentheuserandthe computer.Transmissionoccursasselectionofamessagefromasetofpossiblemessagesand transferringoveradegraded(noisy,delayed,intermittent)channel.Therateofsuccessful transmission,andthecomplexityofthemessagesthatarebeingpassed,isacriterionfor betterinteraction.Forinstance,touseakeyboard,theusercommunicatestothecomputer systemviathehumanneuromuscularsystem.Ausercanbeseenasasourcecommunicating messagesinsomemessagespaceoveranoisychannelinordertotransmitinformationfrom theuser’sbrainintothecomputersystem.Importantly,throughputandsimilarinformation theoreticalconstructscanbetakenasobjectivesfordesign,modelledusinghumanperformancemodelssuchasFitt’slawandtheHick–Hymanlaw,andoperationalizedincodeto solveproblems.Applicationsincludeinputmethods,interactiontechniques,andsensing systems.
Probability TheBayesianapproachtointeractionisexplicitinitsrepresentationof uncertainty,representedbyprobabilitydistributionsoverrandomvariables.Underthis viewpoint,interactioncanbeseen,forexample,astheproblemofinferringintentionvia evidenceobservedfromsensors.Intentionisconsideredtobeanunobservedrandom variableexistinginauser’smind,aboutwhichthesystemmaintainsandupdatesaprobabilisticbelief.Aneffectiveinteractionisonethatcausesthebeliefdistributiontoconverge totheuser’strueintention.Thisapproachallowsinformativepriorstobespecifiedacross potentialactionsandrigorouslycombinedwithobservations.Uncertaintyaboutintention canbepropagatedthroughtheinterfaceandcombinedwithutilityfunctionstoengage functionalityusingdecision-theoreticprinciples.Asaconsequenceofproperrepresentation ofuncertainty,Bayesianapproachesofferbenefitsintermsofrobustnessininteraction.
Onemajoradvantageoftheprobabilisticviewisthatmanycomponentsatmultiplelevels ofinteractioncanbeintegratedonasoundbasis,becauseprobabilitytheoryservesasa unifyingframework;forexample,linkingprobabilisticgesturerecognizerstoprobabilistic textentrysystems.ToolssuchasrecursiveBayesianfiltersandGaussianprocessregression canbeuseddirectlyininferringuserintention.
Learning Machinelearningequipsacomputerwiththeabilitytousedatatomake increasinglybetterpredictions.Itcantransformstaticdataintoexecutablefunctionality, typicallybyoptimizingparametersofamodelgivenasetofobservationstominimize someloss.Incomputationalinteraction,machinelearningisusedtobothpredictlikelyuser behaviourandtolearnmappingsfromsensingtoestimatedintentionalstates.Supervised machinelearningformulatestheproblemaslearningafunction y = f(x) thattransforms observedfeaturevectorstoanoutputspaceforwhichsometrainingsetofobservations areavailable.Afteratrainingphase,predictionsof y canbemadeforunseen x.Estimation ofcontinuousvaluessuchas,forinstance,intendedcursorposition,isregression,and estimationofdiscretevaluessuchas,forexample,distinctgestureclasses,isclassification. Supervisedmachinelearningcanreplacehand-tweakingofparameterswithdata-driven modelling.Therearemanyhigh-performingandversatiletoolsforsupervisedlearning, includingsupportvectormachines,deepneuralnetworks,randomforests,andmanyothers. Unsupervisedlearninglearnsstructurefromdatawithoutasetofmatchingtargetvalues. Techniquessuchasmanifoldlearning(learningasimplesmoothlow-dimensionalspace thatexplainscomplexobservations)andclustering(inferringasetofdiscreteclasses)have potentialinexploringandelicitinginteractions.Unsupervisedlearningiswidelyusedin recommendersystemsanduser-modellingingeneral,oftenwithanassumptionthatusers fallintodistinctclustersofbehaviourandcharacteristics.
Optimization Optimizationreferstotheprocessofobtainingthebestsolutionfora definedproblem.Forexample,adesigntaskcanbemodelledasacombinationofelementary decisions,suchaswhichfunctionalitytoinclude,whichwidgettypetouse,wheretoplace anelement,andsoon.Optimizationcanalsouseconstraintstoruleoutinfeasibledesigns. Severalapproachesexisttomodellingdesignobjectives,rangingfromheuristicstodetailed behavioural,neural,cognitive,orbiomechanicalmodels.Thebenefitofformulatingadesign problemlikethisisthatpowerfulsolutionmethodscanbeexploitedtofindbestdesigns automatically,rootedondecadesofresearchonoptimizationalgorithmsforbothoffline andonlinesetups.
States Statemachinesareapowerfulformalismforrepresentingstatesandtransitions withinaninterface.Thismodelofinteractioncapturesthediscreteelementsofinterfaces andrepresentsstates(andtheirgroupings)andeventsthatcausetransitionsbetween states.Formalismssuchasfinite-statemachines(FSMs),andspecificationlanguagessuch asstatecharts,allowforpreciseanalysisoftheinternalconfigurationsofinterfaces.Explicit modellingpermitsrigorousanalysisoftheinterfaceproperties,suchasreachabilityof functionality,criticalpaths,bottlenecks,andunnecessarysteps.GraphpropertiesofFSMs canbeusedtomodeloroptimizeinterfaces;forexample,outdegreecanbeusedtostudythe numberofdiscretecontrolsrequiredforaninterface.Statemachinesofferbothconstructive approachesinrigorouslydesigningandsynthesizinginterfacesandinanalysingandcharacterizingexistinginterfacestoobtainquantifiablemetricsofusability.
Control Withrootsincyberneticsandcontrolengineering,controltheoryprovidesa powerfulformalismforreasoningaboutcontinuoussystems.Inapplicationstohumantechnologyinteraction,theuserismodelledasacontrolleraimingtochangeacontrolsignal toadesiredlevel(thereference)byupdatingitsbehaviouraccordingtofeedbackabout thesystemstate.Thedesignofthesystemaffectshowwelltheusercanachievethegoal givenitsowncharacteristics.Controltheoryviewsinteractionascontinuous,althougha computermayregisteruserbehaviourasadiscreteevent.Modellinginteractionasacontrol systemwithafeedbackloopovercomesafundamentallimitationofstimulus–response basedapproaches,whichdisregardfeedback.Thecontrolparadigmpermitsmulti-level analysis,tracingtheprogressionofuser-systembehaviourovertime(asaprocess)toexplain eventualoutcomestouser.Itallowsinsightintoconsequencesofchangingpropertiesofthe userinterface,ortheuser.
Rationality Rationalanalysisisatheoryofhumandecision-makingoriginatingin behaviouraleconomicsandpsychologyofdecision-making.Theassumptionisthatpeople strivetomaximizeutilityintheirbehaviour.Boundedrationalityistheideathatrational behaviourisconstrainedbycapacityandresourcelimitations.Wheninteracting,users pursuegoalsorutilityfunctionstothebestoftheircapabilitywithinconstraintsposedby userinterfaces,environments,andtasks.Theideaofboundedrationalityisexploredin informationforagingtheoryandeconomicmodelsofsearch.
Agents Boundedagentsaremodelsofusersthattakeactionandoptimallyadapttheir behaviourtogivenconstraints:environmentsandcapabilities.Theboundsincludenot onlythoseposedbytheenvironment,whichincludestheinterface,butlimitationson theobservationandcognitivefunctionsandontheactionsoftheagent.Thesebounds defineaspaceofpossiblepolicies.Thehypothesisisthatinteractivebehaviourisrationally adaptedtotheecologicalstructureofinteraction,cognitiveandperceptualcapacities,and theintrinsicobjectivesoftheuser.Theinteractiveproblemcanbespecified,forexample,as areinforcementlearningproblem,oragame,andbehaviouremergesbyfindingtheoptimal behaviouralpolicyorprogramtotheutilitymaximizationproblem.Therecentinterestin computationallyimplementedagentsisduetothebenefitthat,whencomparedwithclassic cognitivemodels,theyrequirenopredefinedspecificationoftheuser’stasksolution,only theobjectives.Increasinglypowerfulrepresentationsandsolutionmethodshaveemerged forboundedagentsinmachinelearningandartificialintelligence.
Thechaptersofthisbookpresentfurtherdetailsontheassumptions,implementation, applications,aswellaslimitationsoftheseelementaryconcepts.
I.4Outlook
Thechaptersinthisbookmanifestintellectualprogressinthestudyofcomputational principlesofinteraction,demonstratedindiverseandchallengingapplicationsareassuch asinputmethods,interactiontechniques,graphicaluserinterfaces,informationretrieval, informationvisualization,andgraphicdesign.Muchofthisprogressmayhavegone unnoticedinmainstreamhuman-computerinteractionbecauseresearchhasbeenpublished
indisconnectedfields.Tocoalesceeffortsandexpandthescopeofcomputationallysolvable interactionproblems,anexcitingvistaopensupforfutureresearch.
Boththepotentialofandthegreatestchallengeincomputationalinteractionliesin mathematicsandalgorithms.Asharedobjectiveforresearchismathematicalformulation ofphenomenainhumanuseoftechnology.Onlybyexpandingformulationscanone devisenewmethodsandapproachesfornewproblems.Ontheonehand,mathematicsand algorithmsarethemostpowerfulknownrepresentationsforcapturingcomplexity.Onthe otherhand,thecomplexityofapresentationmustmatchthecomplexityofthebehaviour ittriestocaptureandcontrol.Thismeansthattheonlywayoutofnarrowapplicationsis viaincreasinglycomplexmodels.Thechallengeishowtoobtainandupdatethemwithout losingcontrolandinterpretability.
Asignificantfrontier,therefore,istotrytocapturethoseaspectsofhumanbehaviorand experiencethatareessentialforgooddesign.Whetherusersaresatisfiedwithadesignis determinednotonlybyhowmuchtimetheyspendtoaccomplishthetask,butalsoby itsaestheticaspect,theeaseoflearning,whetheritfitsthecultureofcertainregions,etc. Interactionisalsooften coupled withtheenvironmentandthesituatedcontextsoftheuser. Forexample,auser’stypingbehaviourisdependentonwhetherauseriswalking,ifthe userisencumbered,andthewaytheuserisholdingthedevice.Thesystemitselfisalso oftencoupledtotheenvironment.Also,computationalinteractionshouldnotberestricted tomodelasingleuser’sinteractionwithasingleacomputersystem.Asingleusermay interactwithmanydevices,manyusersmayinteractwithasinglecomputersystem,or manyusersmayinteractwithmanycomputersystemsinordertoforinstancecarryouta sharedtaskobjective.Thesechallengescallforcollaborationwithbehaviouralandsocial scientists.
Thisincreasingscaleandcomplexitywillposeachallengealsoforalgorithms.Algorithms underpincomputationalinteractionandforsystemsimplementingprinciplesofcomputationalinteractionitisimportantthattheunderlyingalgorithmsscalewithincreasing complexity.Forexample,naiveoptimizationisofteninfeasibleduetothecomplexityof theoptimizationproblem.However,itisoftenrelativelystraight-forwardtosearchforan approximatelyoptimalsolution.Complexityismultifacetedincomputationalinteraction andmay,forinstance,concerntheexpressivenessofasystem(forinstance,thenumber ofgesturesrecognizedbythesystem),theabilityofasolutiontosatisfymulti-objective criteria,ormanageacomplexandconstantlychangingenvironment.Toincreaseourability todealwithcomplexreal-worldphenomenaimpliesthatweneedtosearchformore efficientwaystoupdatemodelswithdata.Somecomputationalmodelsareheavilyrelianton representativetrainingdata.Achallengeindatacollectionistoensurethedataisaccurately reflectingrealisticinteractioncontexts,whichmaybedynamicandconstantlychanging, andthatitcapturesthevariabilityofdifferentusergroups.Moreover,datamaynotbe availableuntiltheparticularuserhasstartedadoptingthedesign.Sucha‘chickenandthe egg’dilemmahaslongbeenaproblemincomputationalinteraction:theinteractiondatais neededfordesigninginterfaceorinteractivesystems;yetthedatawillnotbeavailableuntil thedesignorsystemisavailableandtheuserstartsadoptingit.Thesechallengescallfor collaborationwithcomputerandcomputationalscientists.
Finally,there’sthehuman.Computationalinteractioncanbeviewedasacontinuation ofthelonghistoryofautomation,whichhasundergoneaseriesofvictoriesandsetbacks duetocomplexitycausingthelossofagency,deskilling,demotivation,andsoon.Oncea computationalinteractiontechniqueisoperatingthereisarisktheuserislosingagencyas aconsequenceofanoverlycomplexsystemaidingoroverridingtheuser.Suchproblems areexacerbatedwhenthesystemfailstocorrectlyinfertheuser’sintention,inparticular, ifthesystemfailsinanunexpectedway,orifitfailstooffersuitablecontrolsandinterpretability.Computationalinteractionshouldofferappropriatedegreeoftransparencythat allowsuserstoatunderstandthemechanismsleadingtoaparticularsystemprediction orsuggestionatsuchalevelthattheycanachievetheirgoals.Todosoeffectivelyeither requiresunderstandingusers’existingworkflowsandpractices,userstoadapttonewways ofinteractiontailedforcomputationalinteractiondesign,oracombinationofboth.From theperspectiveofalgorithms,eventhedesignproblemiscenteredaroundmathematics: thecentralproblemforuserinterfacedesignforalgorithmicsystemsistoassistusers inunderstandingandshapingcontrol,learning,andoptimizationfunctionsandguiding asystem-informedexplorationofthedecisionspace.Whatisthebestuserinterfaceto mathematics?Whenthisproblemissuccessfullysolved,computerscansupportusers’ creativitybyassistingthemineffectivelyexploringhigh-dimensionaldecisionspaces.By modellingadomain-specificcreativeprocess,itispossibletooptimizethecreativeprocess itselfandhelpidentifysuitablesolutionsthatarebetteraccordingtosomecriteria,suchas speed,aesthetics,novelty,etc.Thesechallengescallforcollaborationwithdesignersand designresearchers. ....................................................................................................
references
Anderson,J.R.,2014. Rulesofthemind.Hove,UK:PsychologyPress. Brusilovsky,P.,andMillan,E.,2007.Usermodelsforadaptivehypermediaandadaptiveeducational systems.In:P.Brusilovsky,A.Kobsa,andW.Nejdl,eds. TheAdaptiveWeb:MethodsandStrategies ofWebPersonalization.Berlin:Springer,pp.3–53.
Card,S.K.,Newell,A.,andMoran,T.P.,1983.Thepsychologyofhuman-computerinteraction.New York,NY:LawrenceErlbaum.
Carroll,J.M.,1997.Human-computerinteraction:psychologyasascienceofdesign. AnnualReview ofPsychology,48(1),pp.61–83.
Carroll,J.M.,andCampbell,R.L.,1989.Artifactsaspsychologicaltheories:Thecaseofhumancomputerinteraction. BehaviourandInformationTechnology,8,pp.247–56. Craik,K.J.W.,1947.Theoryofthehumanoperatorincontrolsystems:1.theoperatorasan engineeringsystem. BritishJournalofPsychologyGeneralSection,38(2),pp.56–61. Cross,N.,2011. Designthinking:Understandinghowdesignersthinkandwork.Oxford:Berg. Dix,A.J.,1991.Formalmethodsforinteractivesystems.Volume16.London:AcademicPress. Eisenstein,J.,Vanderdonckt,J.,andPuerta,A.,2001.Applyingmodel-basedtechniquestothedevelopmentofUIsformobilecomputers.In: Proceedingsofthe6thInternationalConferenceonIntelligent UserInterfaces.NewYork,NY:ACM,pp.69–76.
Fisher,D.L.,1993.Optimalperformanceengineering:Good,better,best. HumanFactors,35(1),pp. 115–39.
Fitts,P.M.,andPeterson,J.R.,1964.Informationcapacityofdiscretemotorresponses. Journalof ExperimentalPsychology,67(2),pp.103–12.
Gajos,K.,andWeld,D.S.,2004.SUPPLE:automaticallygeneratinguserinterfaces.In: Proceedingsof the9thInternationalConferenceonIntelligentUserInterfaces.NewYork,NY:ACM,pp.93–100.
Gray,W.D.,andBoehm-Davis,D.A.,2000.Millisecondsmatter:Anintroductiontomicrostrategies andtotheiruseindescribingandpredictinginteractivebehavior. JournalofExperimentalPsychology:Applied,6(4),pp.322–35.
Harrison,M.,andThimbleby,H.,eds.,1990. Formalmethodsinhuman-computerinteraction.Volume2. Cambridge:CambridgeUniversityPress.
Hollnagel,E.,andWoods,D.D.,2005. Jointcognitivesystems:Foundationsofcognitivesystemsengineering .Columbus,OH:CRCPress.
Horvitz,E.,Breese,J.,Heckerman,D.,Hovel,D.,andRommelse,K.,1998.TheLumiereproject: Bayesianusermodelingforinferringthegoalsandneedsofsoftwareusers.In: UAI‘98:Proceedings oftheFourteenthConferenceonUncertaintyinArtificialIntelligence.SanFrancisco,CA:Morgan Kaufmann,pp.256–65.
Jagacinski,R.J.,andFlach,J.M.,2003. ControlTheoryforHumans:Quantitativeapproachestomodeling performance.Mahwah,NJ:LawrenceErlbaum.
Kieras,D.E.,andHornof,A.,2017.Cognitivearchitectureenablescomprehensivepredictive modelsofvisualsearch. BehavioralandBrainSciences,40.DOI:https://doi.org/10.1017/ S0140525X16000121
Kieras,D.E.,Wood,S.D.,andMeyer,D.E.,1997.Predictiveengineeringmodelsbasedonthe EPICarchitectureforamultimodalhigh-performancehuman-computerinteractiontask. ACM TransactionsonComputer-HumanInteraction(TOCHI),4(3),pp.230–75.
Kleinman,D.L.,Baron,S.,andLevison,W.H.,1970.Anoptimalcontrolmodelofhumanresponse partI:Theoryandvalidation. Automatica 6(3),pp.357–69.
Light,L.,andAnderson,P.,1993.Designingbetterkeyboardsviasimulatedannealing. AIExpert ,8(9). Availableat:http://scholarworks.rit.edu/article/727/.
Navarre,D.,Palanque,P.,Ladry,J.F.,andBarboni,E.,2009.ICOs:Amodel-baseduser interfacedescriptiontechniquededicatedtointeractivesystemsaddressingusability,reliabilityandscalability. ACMTransactionsonComputer-HumanInteraction(TOCHI),16(4). <doi:10.1145/1614390.1614393>.
Newell,A.,1994. UnifiedTheoriesofCognition.Cambridge,MA:HarvardUniversityPress. Payne,S.J.,andHowes,A.,2013.Adaptiveinteraction:Autilitymaximizationapproachtounderstandinghumaninteractionwithtechnology. SynthesisLecturesonHuman-CenteredInformatics, 6(1):pp.1–111.
Picard,R.W.,1997. AffectiveComputing .Volume252.Cambridge,MA:MITPress. Pirolli,P.,andCard,S.K.,1999.InformationForaging. PsychologicalReview,106(4),pp.643–75. Sanders,M.S.,andMcCormick,E.J.,1987. HumanFactorsinEngineeringandDesign.Columbus,OH: McGraw-Hill.
Seow,S.C.,2005.InformationtheoreticmodelsofHCI:acomparisonoftheHick-Hymanlawand Fitt’slaw. Human-ComputerInteraction,20(3),pp.315–52.
Shahriari,B.,Swersky,K.,Wang,Z.,Adams,R.P.,anddeFreitas,N.,2016.Takingthehumanoutof theloop:AreviewofBayesianoptimization. ProceedingsoftheIEEE,104(1),pp.148–75.
Sheridan,T.B.,andFerrell,W.R.,1974. Man-machineSystems:Information,Control,andDecision ModelsofHumanPerformance.Cambridge,MA:MITPress. Simon,H.A.,1996. TheSciencesoftheArtificial.Cambridge,MA:MITpress.
Sutton,R.S.,andBarto,A.G.,1998. Reinforcementlearning:Anintroduction.Volume1,Number1. Cambridge,MA:MITPress.
Wickens,C.D.,Hollands,J.G.,Banbury,S.,andParasuraman,R.,2015. EngineeringPsychology& HumanPerformance.Hove,UK:PsychologyPress. Thimbleby,H.,2010. Presson:PrinciplesofInteractionProgramming .Cambridge,MA:MITPress. Zhai,S.,2004.CharacterizingcomputerinputwithFitts’lawparameters—theinformationand non-informationaspectsofpointing. InternationalJournalofHuman-ComputerStudies,61(6), pp.791–809.
Zhai,S.,Hunter,M.,andSmith,B.A.,2002.Performanceoptimizationofvirtualkeyboards. Human–ComputerInteraction,17(2–3),pp.229–69.
