Interfacing with multiple robots using environmental awareness from a multi-modal HRI

-R¶R ;DYLHU DQG 8UEDQR 1XQHV ,QVWLWXWH RI 6\VWHPV DQG 5RERWLFV 'HSDUWPHQW RI (OHFWULFDO DQG &RPSXWHU (QJLQHHULQJ 8QLYHUVLW\ RI &RLPEUD 3RUWXJDO VPRJ]HU#JPDLO FRP XUEDQR#LVU XF SW

$57,*2 7®&1,&2

,17(5)$&,1* :,7+ 08/7,3/( 52%276 86,1* (19,5210(17$/ $:$5(1(66 )520 $ 08/7, 02'$/ +5, $%675$&7

7KLV DUWLFOH GHVFULEHV WKH GHYHORSPHQW RI D +XPDQ 5RERW ,QWHUIDFH WKDW LV DGDSWDEOH WR PXOWLSOH VHUYLFH SURILOHV 7KH V\VWHP SURYLGHV DQ LQIRUPDWLRQ VHUYLFH IURP ZKLFK LV SRVVLEOH WR FRRUGLQDWH XVHU HQYLURQPHQW GDWD 7DVN &RQWH[WV DQG $ZDUHQHVV ERWK VSDWLDO DQG VLWXDWLRQDO

DUH SURYLGHG VR WKDW WKH V\VWHP LV UHDFWLYH WR HYHQWV LQ WKH SHUFHLYHG ZRUOG &RPSRQHQWV WKHQ XVH WKLV FRPPRQ LQIRUPDWLRQ VWUXFWXUH WR SHUIRUP DFWLRQV RQ WKH VXUURXQGLQJ LQWHUDFWLYH HQYLURQPHQW LQFOXGLQJ WKH URERW V 7KH V\QHUJ\ REWDLQHG IURP GLIIHUHQW V\VWHP FRPSRQHQWV LV XVHG IRU LQWHUDFWLQJ ZLWK NH\ DVVLVWLYH WHFKQRORJLHV VXFK DV GRPRWLFV HPHUJHQF\ PDQDJHPHQW GULYHU VXSSRUW DLGV ' ZDONWKURXJKV :H H[SODLQ WKH DUFKLWHFWXUH RI WKH V\VWHP DQG GHPRQVWUDWH LWV XVDJH ZLWK DQ H[DPSOH RI YRLFH LQWHUIDFLQJ ZLWK PXOWLSOH URERWV DQG HQYLURQPHQW DZDUHQHVV 7KH GHYHORSHG V\VWHP LV GLVWULEXWHG IUHHO\ VR WKDW RWKHUV FDQ EHQHILW DQG FRQWULEXWH WR LW DV ZHOO

, ,1752'8&7,21 0RELOLW\ LPSDLUHG XVHUV EHQHILW IURP XVLQJ VRFLDOO\ DVVLVWLYH WHFKQRORJLHV IRU DLGLQJ WKHP ZLWK WKHLU GDLO\ WDVNV 7KH LQWHUIDFHV XVHG LQ +XPDQ 5RERW ,QWHUIDFHV OLNH URERWLF ZKHHOFKDLUV RU RWKHU DVVLVWLYH +XPDQ 0DFKLQH ,QWHUIDFHV KDYH QRW \HW DFKLHYHG WKH GHVLUHG LQWHJUDWLRQ WKDW FRXOG EH SRVVLEOH ZLWK WRGD\ V WHFKQRORJ\ 7KH SURSRVHG +5, VROXWLRQ H[SORUHV PRUH LQ GHSWK WKH SRVVLELOLWLHV RI LQWHUDFWLRQ RI D URERW IHDWXUH ULFK DQG G\QDPLF HQYLURQPHQW :H GHVFULEH KRZ RXU +5, LQWHJUDWHV HQYLURQPHQWDO LQIRUPDWLRQ DFTXLUHG IURP VHQVRUV LQWR D FRJQLWLYH OD\HU DQG KRZ WKLV FRJQLWLYH GDWDEDVH LV XVHG IRU LQWHUDFWLRQ LQ D PXOWL PRGDO PXOWL URERW LQWHUIDFH :H SUHVHQW D 6SHHFK 5HFRJQLWLRQ ,QWHUIDFH WKDW G\QDPLFDOO\ DGDSWV WR WKH URERW HQYLURQPHQW ZKHUH WKH OH[LFRQ RI WKH FRPPDQGV UHIOHFWV WKH DELOLWLHV RI LQWHUDFWLRQ ZLWK WKH VXUURXQGLQJ HQYLURQPHQW DQG WKDW LV QRW OLPLWHG WR FRQWURO D VLQJOH URERW 7KH 3OD\HU PLGGOHZDUH GHVFULEHG LQ > @ LV XVHG WR DFFHVV URERWV DQG VHQVRUV IURP WKH 6WDJH DQG *D]HER VLPXODWRUV DORQJ ZLWK WKH UHDO RQHV IURP RXU 5REFKDLU URERW GHVFULEHG LQ GHWDLO LQ > @ 7KLV ZRUN LV DYDLODEOH RQ OLQH LQ > @

$ $UWLFOH 6WUXFWXUH 7KH UHVW RI WKLV DUWLFOH LV RUJDQL]HG DV IROORZV 7KH UHODWHG ZRUN LV GHVFULEHG LQ 6HFWLRQ ,, 7KH PRGXODU LQIUDVWUXFWXUH RI RXU V\VWHP LV GHVFULEHG LQ 6HFWLRQ ,,, 7KH FRJQLWLYH OD\HU LV GHVFULEHG LQ 6HFWLRQ ,9 'HWDLOV DERXW WKH PXOWL PRGDOLW\ DQG PXOWL URERW LQWHUIDFLQJ DUH GHVFULEHG LQ 6HFWLRQ 9 $Q H[DPSOH RI YRLFH LQWHUIDFLQJ WR PXOWLSOH URERWV ZLWK HQYLURQPHQW DXJPHQWDWLRQ LV GHVFULEHG LQ 6HFWLRQ 9, 7KH FRQFOXVLRQ LV LQ 6HFWLRQ 9,,

,, 5(/$7(' :25.

2Q WKH IROORZLQJ VXEVHFWLRQV VRPH JRRG SUDFWLFHV DQG DSSOLFDWLRQV DUH GHVFULEHG UHODWHG WR WKH DUHDV RI LQQRYDWLRQ LQ RXU ZRUN

$ 0XOWL URERW LQWHUIDFLQJ $ WD[RQRP\ IRU FDWHJRUL]LQJ KXPDQ URERW LQWHUDFWLRQ LV SUHVHQWHG LQ

> @

UREËWLFD

> @ 2I WKH PRGHOV DYDLODEOH LQ WKLV WD[RQRP\ RXU PRGHO ZRXOG EH LQ WKH |RQH RSHUDWRU WR PXOWLSOH URERWV} 7KH WDVN RI LQWHUIDFLQJ ZLWK VZDUPV RI URERWV LV GLVFXVVHG LQ > @ ZKHUH WKH DXWKRU LQWURGXFHV WKH FRQFHSW RI URERW VRFLHW\ %RWK > @ > @ KDYH GHYHORSHG ' +5, IRU LQWHUIDFLQJ RI D VLQJOH XVHU WR PXOWLSOH URERWV ZLWK VXSSRUW IRU PRWLRQ SODQQLQJ DQG URERW WUDMHFWRU\ JHQHUDWLRQ IRU WDUJHW WUDFNLQJ ,Q > @ DUH VWXGLHG WKH FRQGLWLRQV XQGHU ZKLFK WKH FROODERUDWLYH KXPDQ LQYROYHPHQW LQ VKDUHG +5, ZLOO QRW MHRSDUGL]H VFDODELOLW\ RI WKH QHWZRUN RI URERWV 7KH PXOWL DJHQW V\VWHP 0$6 LQIUDVWUXFWXUH WKDW FRPELQHV +5, ZLWK D VLPXODWLRQ HQYLURQPHQW IRU +XPDQ 6HDUFK DQG 5HVFXH +65 RSHUDWLRQV LV GHVFULEHG LQ > @ $OVR IURP WKH +65 UHVHDUFK DUHD FRPHV WKH WRSLF RI DGMXVWDEOH DXWRQRP\ GHVFULEHG LQ > @ $GMXVWDEOH $XWRQRP\ GHILQHV OD\HUV RI DXWRQRP\ IRU LQWHUDFWLRQ ZLWK WKH URERWV ZKHUH WKH KLJKHU OD\HUV PHDQ PRUH GHFLVLRQV E\ WKH URERW H J LQ WKH ERWWRP OD\HU WKH XVHU PRGLILHV WKH URERW YHORFLWLHV ZKLOH LQ WKH DERYH OD\HU WKH XVHU MXVW JLYHV D WDUJHW GHVWLQDWLRQ DQG VR RQ

% &RQWH[W DQG $ZDUHQHVV &RQWH[W $FTXLVLWLRQ LV WKH PHFKDQLVP XVHG WR XQGHUVWDQG WKH URERWV VLWXDWLRQ 7KH FRQFHSW ZDV ILUVW GHVFULEHG LQ > @ EXW VLPLODU IXQFWLRQDOLW\ FDQ EH LGHQWLILHG LQ > @ > @ WKDW LQWURGXFHV WKH FRQFHSW RI 6LWXDWLRQDO $ZDUHQHVV $Q H[DPSOH RI D SURMHFW WR JHQHUDOO\ LPSURYH WKH DFFHVVLELOLW\ IRU LPSDLUHG SHRSOH LV > @ 2XU V\VWHP LPSOHPHQWV WKH IXQFWLRQDOLW\ RI WKH IROORZLQJ OD\HUV RI WKH $VN LW SURMHFW DPELHQW LQWHOOLJHQFH SODWIRUP VHOI FRQILJXUDEOH XVHU LQWHUIDFHV LQWXLWLYH VHPDQWLFV 6RPH V\VWHPV OLNH 1DYFKDLU GHVFULEHG LQ > @ KDYH GLIIHUHQW QDYLJDWLRQ PRGHV GHSHQGLQJ RI WKH VLWXDWLRQ WKH\ DUH LQ 2EVWDFOH DYRLGDQFH GRRU SDVVDJH DQG DXWRPDWLF ZDOO IROORZLQJ DUH WKH FRQVLGHUHG PRGHV ZKLFK DUH VHOHFWHG E\ D %D\HVLDQ QHWZRUN GHFLVLRQ IUDPHZRUN EDVHG RQ WKH ZKHHOFKDLU VLWXDWLRQ $ URERWLF ZKHHOFKDLU ZLWK D VSHHFK UHFRJQLWLRQ LQWHUIDFH WKDW FDQ XQGHUVWDQG WKH XVHU LQWHQWLRQ XVLQJ WKH HQYLURQPHQWDO LQIRUPDWLRQ REWDLQHG IURP UDQJH VHQVRUV LV SUHVHQWHG LQ > @ 7KH DXWKRU QRWHV WKDW HYHQ LI WKH XVHU GRHV QRW VD\ GHWDLOV DQG LVVXHV D VLPSOH YRLFH FRPPDQG WKH ZKHHOFKDLU WDNHV DQ DSSURSULDWH DFWLRQ WKDW WKH XVHU H[SHFWV ,Q SURMHFW &R\RWH > @ LV SUHVHQWHG D PRELOH URERW WKDW FDQ XQGHUVWDQG VSDWLDO FRPPDQGV DQG JHQHUDWH VSDWLDO OLQJXLVWLF GHVFULSWLRQV RI LWV HQYLURQPHQW ,W

> @ + $ <DQFR DQG - 'UXU\ |&ODVVLI\LQJ KXPDQ URERW LQWHUDFWLRQ $Q XSGDWHG WD[RQRP\ } LQ ,((( &RQIHUHQFH RQ 6\VWHPV 0DQ DQG &\EHUQHWLFV 2FWREHU >2QOLQH@ $YDLODEOH ZZZ FV XPO HGX aKROO\ SDSHUV > @ $ +DOPH 3 -DNXELN 7 6FKRQEHUJ DQG 0 9DLQLR |&RQWUROOLQJ WKH RSHUDWLRQ RI D URERW VRFLHW\ WKURXJK GLVWULEXWHG HQYLURQPHQW VHQVLQJ } LQ ,)$& :RUNVKRS RQ

+XPDQ 2ULHQWHG 'HVLJQ RI $GYDQFHG 5RERWLFV 6\VWHPV 9LHQQD $XVWULD > @ & &ODUN DQG ( )UHZ |$Q LQWHJUDWHG V\VWHP IRU FRPPDQG DQG FRQWURO RI FRRSHUDWLYH URERWLF V\VWHPV } LQ ,QWHUQDWLRQDO &RQIHUHQFH RQ $GYDQFHG 5RERWLFV &RLPEUD 3RUWXJDO -XQH > @ + -RQHV DQG 3 +LQGV |([WUHPH ZRUN JURXSV 8VLQJ VZDW WHDPV DV D PRGHO IRU FRRUGLQDWLQJ GLVWULEXWHG URERWV } LQ &RQIHUHQFH RQ &RPSXWHU 6XSSRUWHG

&RRSHUDWLYH :RUN 1RYHPEHU > @ $ 0DNDUHQNR 7 .DXSS % *URFKROVN\ DQG + 'XUUDQW :K\WH |+XPDQ URERW LQWHUDFWLRQV LQ DFWLYH VHQVRU QHWZRUNV } .REH -DSDQ SS z -XO\ > @ , 1RXUEDNKVK . 6\FDUD 0 .RHV 0 <RXQJ 0 /HZLV DQG 6 %XULRQ |+XPDQ URERW WHDPLQJ IRU VHDUFK DQG UHVFXH } >2QOLQH@ $YDLODEOH FLWHVHHU LVW SVX HGX QRXUEDNKVK KXPDQURERW KWPO > @ $ %LUN DQG 0 3ILQJVWKRUQ |$ KXPDQ PDFKLQH LQWHUIDFH VXSSRUWLQJ DGMXVWDEOH DXWRQRP\ RI UHVFXH URERWV } LQ 5RER&XS 5RERW 6RFFHU :RUOG &XS ,; VHU /HFWXUH 1RWHV LQ $UWLILFLDO ,QWHOOLJHQFH /1$, , 1RGD $ -DFRII $ %UHGHQIHOG DQG < 7DNDKDVKL (GV 6SULQJHU YRO SS z > @ 7 )RQJ ' .DEHU DQG HW DO |&RPPRQ PHWULFV IRU KXPDQ URERW LQWHUDFWLRQ } 6HQGDL -DSDQ >2QOLQH@ $YDLODEOH FLWHVHHU LVW SVX HGX IRQJ FRPPRQ KWPO > @ % *UDYHV |$ JHQHUDOL]HG WHOHDXWRQRPRXV DUFKLWHFWXUH XVLQJ VLWXDWLRQ EDVHG DFWLRQ VHOHFWLRQ } 7H[DV $ 0 8QLYHUVLW\ &ROOHJH 6WDWLRQ > @ - 6FKROW] % $QWRQLVKHN DQG - <RXQJ |(YDOXDWLRQ RI D KXPDQ URERW LQWHUIDFH 'HYHORSPHQW RI D VLWXDWLRQDO DZDUHQHVV PHWKRGRORJ\ } +DZDLL ,QWHUQDWLRQDO

&RQIHUHQFH RQ 6\VWHP 6FLHQFHV +,&66 YRO > @ |$VN LW SURMHFW $PELHQW LQWHOOLJHQFH V\VWHP RI DJHQWV IRU NQRZOHGJH EDVHG )LJXUH $ VWDWH IORZ GLDJUDP UHSUHVHQWLQJ WKH VWDWH WUDQVLWLRQV RI D FRPPRQ VHWXS IRU VSHHFK EDVHG LQWHUIDFLQJ ZLWK D PRELOH URERW

DQG LQWHJUDWHG VHUYLFHV IRU PRELOLW\ LPSDLUHG XVHUV } >2QOLQH@ $YDLODEOH ZZZ DVN LW RUJ > @ 6 /HYLQH ' %HOO / -DURV 5 6LPSVRQ DQG . .RUHQ |7KH QDYFKDLU DVVLVWLYH ZKHHOFKDLU QDYLJDWLRQ V\VWHP } ,((( 7UDQVDFWLRQV RQ 5HKDELOLWDWLRQ (QJLQHHULQJ

9,, &21&/86,21 ,Q WKLV ZRUN D SODWIRUP IRU PXOWLSOH URERW LQWHUDFWLRQ LV LQWURGXFHG ZKLFK VXSSRUWVPXOWLSOH LQWHUIDFHV PXOWLPRGDO DQG WKDW PDNHV XVH RI OLJKWZHLJKW FRJQLWLYH OD\HU WKDW LV VKDUHG EHWZHHQ DOO WKH URERWV RU FRPSRQHQWV WKDW WKH XVHU LQWHUIDFHV ZLWK :H H[SODLQ WKH GHVLJQ RI WKH DZDUHQHVV FRQWH[WV WKDW DUH HVVHQWLDO IRU WKH VSDWLDO DQG WKH VLWXDWLRQDO DZDUHQHVV RI WKH SODWIRUP 7KH DSSURDFK RI XVLQJ D G\QDPLF OH[LFRQ IRU VSHHFK UHFRJQLWLRQ LQWHUIDFH DQG LWV DGYDQWDJHV RYHU WUDGLWLRQDO VWDWLF OH[LFRQ LV GLVFXVVHG ,W LV H[SODLQHG KRZ DOO WKHVH IHDWXUHV DUH FRPELQHG VHDPOHVVO\ WRJHWKHU LQ D FRPSRQHQW EDVHG V\VWHP WKDW ZRUNV WR FUHDWH UREXVW G\QDPLF FRPSRVHG +XPDQ 5RERW ,QWHUIDFHV 7KH GHVFULEHG V\VWHP LV D SDUW RI D ZKROH IUDPHZRUN WKDW LV DYDLDEOH IRU GRZQORDG IURP WKH VLWH > @

> @ $ 1 7 ,ZDVH DQG < .XQR |5RERWLF ZKHHOFKDLU XQGHUVWDQGLQJ WKH XVHUiV LQWHQWLRQ LQ VSHHFK XVLQJ WKH HQYLURQPHQWDO LQIRUPDWLRQ } LQ $GYDQFHV LQ &RPSXWHU 6FLHQFH

DQG 7HFKQRORJ\ > @ 0 6NXELF ' 3HU]DQRZVNL 6 %OLVDUG $ 6FKXOW] : $GDPV 0 %XJDMVND DQG ' %URFN |6SDWLDO ODQJXDJH IRU KXPDQ URERW GLDORJV } ,((( 7UDQVDFWLRQV RQ 6\VWHPV

0DQ DQG &\EHUQHWLFV > @ % .XLSHUV |7KH VSDWLDO VHPDQWLF KLHUDUFK\ } $UWLI ,QWHOO YRO QR SS z >2QOLQH@ $YDLODEOH KWWS SRUWDO DFP RUJ FLWDWLRQ FIP"LG > @ + 6 0RVDUX ,VKLL |$ VWHS WRZDUG D KXPDQ URERW FRRSHUDWLYH V\VWHP } $UWLILFLDO

/LIH DQG 5RERWLFV > @ + ;X + 9 %UXVVHO DQG 5 0RUHDV |'HVLJQLQJ D XVHU LQWHUIDFH IRU VHUYLFH RSHUDWLRQV RI DQ LQWHOOLJHQW PRELOH PDQLSXODWRU } 7HOHPDQLSXODWRU DQG 7HOH

SUHVHQFH 7HFKQRORJLHV ,9 YRO QR SS z >2QOLQH@ $YDLODEOH OLQN DLS RUJ OLQN "36, > @ 6 7HOOH[ DQG ' 5R\ |6SDWLDO URXWLQHV IRU D VLPXODWHG VSHHFK FRQWUROOHG YHKLFOH } 5()(5(1&(6

LQ +5, 3URFHHGLQJ RI WKH VW $&0 6,*&+, 6,*$57 FRQIHUHQFH RQ +XPDQ URERW

> @ % *HUNH\ 5 7 9DXJKDQ DQG $ +RZDUG |7KH SOD\HU VWDJH SURMHFW 7RROV IRU

LQWHUDFWLRQ 1HZ <RUN 1< 86$ $&0 3UHVV SS z

PXOWL URERW DQG GLVWULEXWHG VHQVRU V\VWHPV } LQ 3URF WK ,QWHUQDWLRQDO &RQIHUHQFH

RQ $GYDQFHG 5RERWLFV &RLPEUD 3RUWXJDO SS z >2QOLQH@ $YDLODEOH SOD\HUVWDJH VI QHW > @ * 3LUHV DQG 8 1XQHV |$ ZKHHOFKDLU VWHHUHG WKURXJK YRLFH FRPPDQGV DQG DVVLVWHG E\ D UHDFWLYH IX]]\ ORJLF FRQWUROOHU } LQ ,QWHOOLJHQW DQG 5RERWLF 6\VWHPV YRO QR -XO\ SS z > @ - ;DYLHU |0RGXOHV IRU LQWHOOLJHQW DXWRQRPRXV URERW QDYLJDWLRQ } >2QOLQH@ $YDLODEOH PLDUQ VI QHW

> @

UREÃ&#x2039;WLFD

> @ ) )OLSSR |$ QDWXUDO KXPDQ FRPSXWHU LQWHUIDFH IRU FRQWUROOLQJ ZKHHOHG URERWLF YHKLFOHV } >2QOLQH@ $YDLODEOH FLWHVHHU LVW SVX HGX IOLSSR QDWXUDO KWPO > @ &08 |&PX VSKLQ[ VSHHFK UHFRJQLWLRQ HQJLQH } >2QOLQH@ $YDLODEOH FPXVSKLQ[ VRXUFHIRUJH QHW > @ . .RQROLJH |$ JUDGLHQW PHWKRG IRU UHDOWLPH URERW FRQWURO } LQ ,((( 56-

,QWHUQDWLRQDO &RQIHUHQFH RQ ,QWHOOLJHQW 5RERWV DQG 6\VWHPV

,17(5)$&,1* :,7+ 08/7,3/( 52%276 86,1* (19,5210(17$/ $:$5(1(66 )520 $ 08/7, 02'$/ +5,

%\ WKDW WLPH WKH URERW ZLWK FRQWH[W |7:2} DOUHDG\ DUULYHG DW WKH |5220} ]RQH DQG QHZ VXE ]RQHV |7$%/(} DQG |%('} DUH QRZ DYDLODEOH 7KHVH QHZ ]RQHV ZHUH FUHDWHG E\ WKH =RQH &UHDWRU FRPSRQHQW WKDW ZDV ORDGHG IURP GLVN ZKHQ WKH 6SDWLDO $ZDUHQHVV FRPSRQHQW QRWLFHG WKDW WKH XVHU HQWHUHG D |5220} ]RQH 7KH RSHUDWRU WKHQ FRPPDQGV WKH URERW WR JR WR WKH QHZO\ DGGHG |7$%/(} ]RQH

)LQDOO\ WKH XVHU VHOHFWV ERWK URERWV DQG WHOOV WKHP WR MRLQ DW WKH |+$//} 7KH H[SHULPHQW HQGV 7KH RUGHU RI YRLFH FRPPDQGV DQG WKH ILQDO YLVXDO ORJ RI WKH H[SHULPHQW DUH GHSLFWHG LQ )LJ 7KLV H[DPSOH ZDV SHUIRUPHG RQ WKH 6WDJH > @ PXOWLSOH URERW VLPXODWRU DQG DV VXFK FDQ EH UHSURGXFHG LQ RWKHU FRPSXWHUV 7KH URERW QDYLJDWLRQ DOJRULWKP LV WKH 9)+ DQG WKH SDWK SODQQHU LV WKH :DYHIURQW ERWK DYDLODEOH LQ 3OD\HU > @