The social behavior guide for confused autonomous machines

Rhode Island School Of Design Industrial Design Graduate Thesis

The Social Behavior Guide for Confused Autonomous Machines

IT'S OK TO BE CONFUSED ;)

Copyright © 2016 by Yuxin Han All rights reserved.

The Social Behavior Guide for Confused Autonomous Machines A thesis presented in partial fulfillment of the requirements for the degree Master of Industrial Design in Industrial Design in the Department of Industrial Design of the Rhode Island School of Design, Providence, Rhode Island

by Han, Yuxin 2016

Approved by Master's Examination Committee Claudia Rebola, Associate Professor, Department of Industrial Design, Rhode Island School of Design

Michael Lye, Senior Critic, Department of Industrial Design, Rhode Island School of Design

Ian Gonsher, Lecturer, School of Engineering, Brown University

This thesis project would have been impossible without the support of my thesis committee members: Prof. Claudia Rebola Ph.D. Michael Lye, Ian Gonsher, as well as the faculties and members of RISD community including Prof. Charlie Cannon, Prof. Andrew Law, Prof.

Paolo Cardini, Timothy Maly, Prof. Tom Weiss, Prof Cas Holman, Julie Lasky, Emily Rothschild, Chris Woebken. I want to express my appreciations to my colleagues, fellow master candidates of RISD's Industrial design program who share their expertises, feedbacks, and supports to this project. I am also grateful for the supports from the members of Brown University Humans To Robots laboratory.

PREFACE The Social Behavior Guide for Confused Autonomous Machines, is a speculative frameworks that enable robot, Artificial Intelligence, automated equipment (and sometimes humans as well) to achieve fluent interaction with partners so they can collaborate in dynamic scenarios.

[1] Card, Stuart K.; Thomas P. Moran; Allen Newell (1980).

The keystroke-level model for user performance time with interactive systems.

Communications of the ACM 23

[2] Arkin, Ronald C. (1998).

Behavior-Based Robotics.

This guide is more speculative than most other literature in the field. It is more inculsive than Human Computer Interaction Principles[1], certainly more controversial than Behavior-Based-Robotic approaches[2] for the following reasons: First, it focus the interactive behavior of robots and intelligent machines.Second it approaches human machine interaction with social contexts.Third, all chapters are accompanied by a number of intuitive illustrations. And last, it has the phrase, it is OK to feel confused, inscribed in large friendly letter on its back cover.

Due to The Guide’s experimental nature, we strongly encourage our readers to approach the book with questions and criticism. Instead of considering it as a set of rigid principles, readers should view this book as prompts, to further develop constructive dialogues on the subject of human-machine relationships.

TABLE OF CONTENTS Abstract 00

The Importance of Being Socially Savvy (For Machines)

01 02 03 04 05 06 07

Not All Machines Are Created The Same

08 09

Acknowledgement Attentiveness Confidence Intention Anticipation Rationale Attitudes Machine Empathy Activities for Humans User Study Results from the Humans to Robots Laboratory Epilogue BIBLIOGRAPHY

ABSTRACT The Social Behavior Guide for Confused Autonomous Machines is a speculative design framework with the intention of supporting machine trainers across disciplines (designers, researchers, engineers, etc) to develop behaviors that enable machines to interact and collaborate with humans in complex scenarios. The Social Behavior Guide consists three major components: A list of suggested machine social behaviors, a frame that utilizes social behaviors as the bridge between signals (such as gestures, sounds, and eye movements) and fluent interactions, an empathy activity that help to generate more social behaviors for autonomous machines.

The propose of this book is to suggest new design approaches to humanmachine interactions and act as prompts for the dialogues on creating a better human-machine relationship.

The Importance of Being Socially Savvy (For Machines) Humans, commonly understood as the most socially savvy beings on planet earth, practice a system of extremely complex social cues and rituals to aid in their communication.

Although social behaviors help humans to express rich and contextual messages fluently and enable them to accomplish complex interactions[3], intelligent machines often find such human behaviors, well, unimpressive.

In fact, many earlier models spent no effort at all trying to adopt human communication methods. They simply believed that their remarkable computing power should be reserved for helping scientists design spaceships or solve the question of Life, the Universe, and Everything.

[3] Clark, Herbert H.; Brennan, Susan E. (1991), Resnick, L. B.; Levine, J. M., eds., Perspectives on socially shared cognition, American Psychological Association

In the mid-70s, because of a few lunatics' vision, intelligent machines were introduced to offices and home with the intention to help people with their daily tasks. This was when machines felt the pressure to pick up human communication skills since now they need to collaborate with, you know, ordinary users.

Some of these lunatics include Ed Roberts, Lee Felsenstein, Adam Osborne, Chuck Peddle, Steve Jobs, Steve Wozniak, and many other pioneers of personal computers.

In the following decades, intelligent machines have learned how to work with everyday users. Based on human-computer interaction activist Alan Cooper's suggestion "Don't make user feel stupid"[2], they gradually adopted more intuitive communication methods such as the visual interface.

[4] Alan Cooper is an software designer and programmer. Widely recognized as the “Father of Visual Basic," Cooper is known for his books About Face

Fast forward to today, intelligent machines no longer sit passively on the desks, waiting for instructions. Advancements in technology have given them to the ability to sort and organize information for users based on their preferences, clean and manage household environments, assist doctors and nurses to look after patients, and even deliver passengers safely to their destinations all on their own. Machines have also adopted, from their perspective, ambiguous communication methods such as voice command and speech generation.

However, even after so many upgrades in both functions and communication skills, machines still fail to interact with humans fluently. Users still get frustrated talking with Siri, get startled when a robot make a move, and fail to trust a self-driving car even when it is probably a much better driver than any human. Intelligent machines are confused: what has gone wrong?

According to The Social Behavior Guide for Confused Autonomous Machines, while humans express themselves through speech, expressions, gestures and etc, it is the social context that make sense of such ambiguous methods and signals and enable fluent interaction with little miscommunications.

While intelligent machines have learned to talk, blink, and make adorable sounds, it is crucial for them to master the basics of social behaviors before achieving more sophisticated interactions with human partners in dynamic Context.

00

Not All Machines Are Created The Same (And Also How To Use This Book) While eye gazes, facial expressions, gestures and etc. play an essential role in human to human communication, it is quite clear that not all autonomous machines have the ability to express these humanly signals. A number of previous studies[5][6] suggest signals such as eye gazing and gestures would improve the machines' abilities in interacting with humans, the signals are, however, not easily transferable to different types of machines because of the various design and communication tools they are equipped with. For example, home-care robots might come with arms and "eyes" while you will seldom find such features on the self-driving car, or automatic sliding doors.

Despite the variety of designs and functions, the ability to communicate with relevant behaviors that are familiar to humans is still key to fluent humanrobot interaction[7].

The Social Behavior Guide introduces a framework of behavior types that bridge communication signals and interactions. Depending on the functions and ability of the machine, a particular social behavior could be achieved with different combinations of signals[8]. For example, acknowledging the presence of users could be expressed with eye gaze and facial expressions in the context of personal service robot (such as care-o-bot) or expressed with lighting and sound in the context of a Roomba.

The behavioral framework offers interaction researchers and designers some basic structure while remaining an open-ended platform to create different solutions based on what kind of robots they are working on.

[5] Skantze, Hjalmarsson, Oertel.

Turn-taking, feedback and joint attention in situated human–robot interaction, Speech Communication, 2014.

[6] S. Waldherr, R. Romero, and S. Thrun. A gesture based interface for human-robot interaction.

Autonomous Robots, 2000.

[7] K. Dautenhahn, S. Woods, C. Kaouri, M. Walters, K. L.Koay, and I. Werry. What is a robot companion - friend, assistant or butler? IEEE IROS, 2005.

[8] Aaron Powers; Sara Kiesler; Jennifer Goetz; Matching Robot Appearance and Behavior to asks to Improve Human-Robot Cooperation ROMAN, 2003.

The Social Behavior Guide introduce a framework of behavior types that bridges communication signals to interactions.

Depends on the functions and ability of the machines, a particular social behavior could be achieved with different combinations of signals.

Autonomous machines can take a variety of forms. Different machines must rely on a variety of signals based on their capabilities to achieve fluent human-robot communications.

A common Social Behaviors framework could be shared by autonomous machines and enable them to achieve fluent interactions with humans despite differences in designs and functions.

These seven proposed types offer a good starting point.

A particular social behavior could be achieved with different signals depending on the functions and capability of the machines.

SOCIAL BEHAVIOR TYPES 01 Acknowledgement 02 Attentiveness 03 Confidence 04 Intention 05 Anticipation 06 Rationale 07 Attitudes

01 Acknowledgement The autonomous machine should stay responsive to users’ actions To initiate an interaction, A machine has to inform users when it senses them. It is also the first step to establishing the communication common ground between the two interaction partners, which is crucial to any further interactions[9][10]. This gives the user the clue that: (1) the machine is interactive.

(2) it ready to engaging in interactions.

In the Social Feedback case studies, we find that users often were uncertain about how to initiate interaction with the robot, since they were sure if the robot were ready to receive instructions.

[9]Herbert H. Clark; Susan E. Brennan.

Grounding in communication. 1991 [10]Brennan, Susan E. The Grounding Problem in Conversations With and Through Computers. Social and cognitive psychological approaches to interpersonal communication 1998.

During a robot demo at Brown University Humans To Robots Laboratory, a user was uncertain about when and how to start the interaction. (bottom left image), photo by Yuxin Han

“Let users know you sense them.�

Imagine an autonomous machine similar to a Roomba is approaching a user while performing its task. Often the human unsure if the machine can sense their presence, makes way for it to pass.

But if the machine could inform the users as soon as it senses their presence, it relieves their worries that the machine might notice not them and crash into them. This would reduce the miscommunication between users and machines and increase users’ acceptance towards the machines.

02 Attentiveness The autonomous machine should stay responsive to users’ actions Based on human experience, we expect our partners to stay responsive during an interaction. Often people give constant feedbacks through head nodding and other gestures to signal that they are engaged. Lack of reactions often reads as disengaging or lack of interest[11]. Such social expectations are often projected by humans when interacting with machines.

[11]Ben Shneiderman, Response Time and Display Rate in Human Pe r f o r m a n c e w i t h C o m p u t e r s .

Computer Surveys, Vol. 16, No. 3, 1984.

In the Social Feedback case studies, we find that when the robot stays responsive to users' actions, (i.e. nodding every time when the user speak), the users do not have to repeat themselves as often because they know they are being heard.

During a robot demo at Brown University Humans To Robots Laboratory, a user was frustrated because the robot's lack of responsiveness. (bottom left image), photo by Yuxin Han

“Let users know you are paying attention.�

03 Confidence The autonomous machine should inform users about how much it understood.

Because of the ambiguous nature of human communications, machines are easily confused and would misinterpret a user’s intentions. Often machine would either get stuck in a loop or make the wrong decisions. This limits machines’ capabilities to collaborate with humans in more complex tasks and leaves the users frustrated.

In the Social Feedback case studies, we noticed that when the robot is given the capability of asking questions, they can collaborative with a human partner with fewer mistakes and less time.

During a robot demo at Brown University Humans To Robots Laboratory, the Baxter Robot ask a question for clarification. (bottom left image), photo by Yuxin Han

“Let users know how much you understood”

?

1 2 3 4

04 Intention The autonomous machine should inform users of what it is about to do.

Throughout the social feedback case studies, we repeatedly noticed that humans are often startled by the robot’s actions. Without communicating what is its next action, a machine often surprising humans when it starts to move. By stating intentions, this allows humans to be prepared for robot’s actions, and reaction accordingly. This behavior is also extremely important to fluent collaborative interactions as it helps both parties to synchronize their actions[12].

[12] Leila Takayama: Doug Dooley; Wendy Ju. Expressing thought: Improving robot readability with animation principles. HRI 2011

During a robot demo at Brown University Humans To Robots Laboratory, a user was surprised by the robot's sudden move. (bottom left image), photo by Yuxin Han

“Let users know what you are about to do�

! humans are often startled by the machines' actions since the machines hardly communicate it next move.

But if the machines could inform the users what they are about to do, it will allow humans to be prepared for the machines’ actions, and reaction accordingly.

05 Anticipation The autonomous machine should communicate to users what it is expecting.

To perform collaborative tasks with humans, a machine not only has to communicate its intentions, but also to inform the humans when it is expecting a certain action from them in order to continue the interactions.

In the Social Feedback case studies, we noticed that passing objects over between humans and robots is a particularly difficult task. Since the robot has limited capabilities to communicate anticipate, they would often drop the objects before humans are ready to receive them

“Let users know what you’re expecting.”

06 Rationale The autonomous machine should inform users about why it is performing actions.

Unexplained the actions from the machines would often cause users to become confused and frustrated[11]. Stating the rational reveals a certain level of consistency and predictability, which would lead to a higher level of synchrony between humans and machines. In the This allows humans to develop trust towards the machines. If your co-worker never tells you how they approach problems, you would possibly think this person is impossible to work with.

[11]Ben Shneiderman, Response Time and Display Rate in Human Performance with Computers.

Computer Surveys, Vol. 16, No. 3, 1984.

During the social feedback case studies where a user asks the robot to hand over an object, we find that when the robot communicates its internal thought process through a graphic interface (the bar graphic represents the probability of which object that user was asking base on robot’s belief), a user is more likely to give clear instructions. (bottom left image), photo by Yuxin Han

“Let users know what you’re expecting.”

07 Attitudes The autonomous machine needs to be able to express attitudes based on context.

As machines taking on a more active role when interacting with humans, their capabilities of express attitudes through communications become more important. Correct attitudes can avoid miscommunications and certainly create more vivid and engaging interactions.

"Many aspects of natural language communication cannot properly be understood without taking the accompanying affect expressions into account."[13] Imagine in a scenario where an autonomous machine is trying to attend an emergency. In such case, it would want express a pressing and alerting attitude so people can make way for it to pass.

In another case, if machine has made a mistake during an interaction, apologetic attitude would communicate that the machine is aware of its mistake and relief users’ frustrations.

[13] Matthias Scheutz; Paul Schermerhorn; James Kramer. The utility of affect expression in natural language interactions in joint human-robot tasks.

HRI 2006

“Remember to use appropriate tones�

08

Machine Empathy Activities for Humans As more intelligent machines are introduced to people's lives, they are given new tasks and taking on new roles every single day. Both the machines and the engineers who train them are faced with a question: how should a machine behave since it is now placed in complex interactions which few design principles have ever addressed before? We repeatedly asked ourselves the same question while producing The Social Behavior Guide For Confused Autonomous Machines.

Luckily, humans are experts of social interaction. Taking advantage of humans' distinguished nature in handling complex communication, we developed a series of machine empathy activities, Robo Plays, to help designers and engineers discover important social behaviors that would aid machines to achieve fluent communication with humans.

A Robo Play simulates a Baxter Robot as a bartender (image above), photo by Yuxin Han

[7] K. Dautenhahn, S. Woods, C. Kaouri, M. Walters, K. L.Koay, and I. Werry. What is a robot companion friend, assistant or butler? IEEE IROS, 2005.

During a Robo Play, the human participant would simulate the capability of the machines by set limitation accordingly, often with empathy tools.

Since we understand that human-human interactions are distinct from humanmachine interactions regarding the roles[7] and capabilities of the machines, Robo Play does not intend to model machine behaviors directly from humanhuman interactions but have the human roleplay a machine with limited communication capabilities.

Responsive

Confidence

Intention

thought process

the analysis of a roleplayer's behavior during a Robo Play (images above), photo by Yuxin Han

Robo Play exercises provided initial materials for developing of The Social Behavior Guide For Confused Autonomous Machines.

Anticipation

enaging the analysis of a roleplayer's behavior during a Robo Play (images above), photo by Yuxin Han

This particular Robo Play simulates a Baxter Robot as a bartender to take order from a human customer. Regarding Baxter Robot's limited capability in communicating with natural language, the robot roleplayer will wear a headset, preventing them from receiving any verbal communication from the “non-robot” participants. Instead, the “non-robot” participants will talk through a speech recognition program where their words will be interpreted by the computer and then displayed to the “robot” participant through texts.

Robo Play can assist designers to generated social behaviors for various for complex interactions with humans.

We introduced Robo Play as a research tool to the students at RISD Departement of Industrial design Urban Interaction Studio. The students adapted Robo Play to instigate behaviors and interactions for a wild range of machines and user contexts. They produced props that simulate various machines' communication capabilities and roleplayed the machines in scenarios such as hotel front desk, bartender, restaurant chef, police investigator, and etc.

Robo Play workshop with students at at RISD Departement of Industrial design Urban Interaction Studio (Prof.

Andy Law), photo by Yuxin Han A robot chef that could not talk or make facial expressions, by students of Urban Interaction Studio, photo by Yuxin Han

*For the reader who are eager to try Robo Play for either reserch or lesuiser proposed, the following diagram will walk you through the step.

Understand the capabilities of a machine.

Roleplay the robot while addressing these limitations.

Interact with another human in a relevant context.

Analyze how the roleplayer achieves communication despite the limitations.

09

Social Behavior User Studies The Social Behavior Guide for Confused Autonomous Machines is developed based on the research results and observations of the Robotic Social Feedback user studies[14] at the Brown University Humans To Robots Laboratory.

The studies measure Robotic Social Feedback's effect on human-robot collaboration. During the studies, participants acquired through convenience sampling interacted with a modified Baxter robot to perform a pick and place task with six everyday objects.

The participants were instructed to choose an object on one side of the table and then to indicate the desired object through a combination of pointing and natural language instructions. They were told to continue indicating until the desired object had been delivered. If the robot delivered an incorrect object, the user continued interacting until the correct object was delivered. Next, they were instructed to do the same for an object on the other side of the table.

We measured the efficiency of the interaction between the participants and the robot using the following criteria: the spent to accomplish the task and the frequency which the participants had to repeat their instructions. The participants were then given a follow-up survey to record their subjective reactions towards the robot. The survey measured the participants’ impressions on the robot concerning its animacy, likeability, and intelligence[15].

Robotic Social Feedback contributes to a more efficient human-robot interaction. It also create a better human impression of the robot.

[14] Emily Wu, Yuxin Han, David Whitney, John Oberlin, James MacGlashan, Robotic Social Feedback for Object Specification, AAAI Fall Symposium , 2015 [15]Bartneck, C., Croft, E., Kulic, D. & Zoghbi, S. (2009). Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics

In order to evaluate the social feedback's benefits on humanrobot interactions, we performed user studies under three controlled conditions.

We measured the efficiency of the interaction between the participants and the robot by mesureing the frequency which the participants had to repeat their instructions before finishing a collabration task with the robot. baseline condition i condition ii

baseline

condition i

Condition ii

The robot communicates no soical feedbacks to the users.

When receive ambiguous input, the robot would ask questions for clarifications.

When receives input, robot nods and informs the users it has "heard" their input.

24% 61%

In the baseline conditions, about 61% users had to repeat themselves during the interactions.

In the condition I, about 24% users had to repeat themselves during the interactions.

33%

In the condition II, about 33% users had to repeat themselves during the interactions.

To understand users’ impressions of the robot, we ask our participants to grade the following aspects of the robot on a scale from 1 to 6 scale: Likeability, Animacy, and Intelligent. baseline condition i condition ii

3.31 3.56 3.90

3.22 4.59 4.19

4.65 4.18 4.96

Animacy

Likeability

Intelligent

Epilogue Because of the speculative natural of The Social Behavior Guide for The Confused Autonomous Machines, we expect that you have raised more questions and doubts after reading this book. Thus, we want to ensure you that It's OK To Be Confused. The goal of this guide is to suggest new design approaches to human-machine interactions and act as prompts that invite designers, engineers, and researchers to join the dialogues on human-machine relationships. We look forward to hearing your thoughts and inputs and welcoming you to contribute additional social behaviors that help to improve fluent interaction between humans and machines.

BIBLIOGRAPHY [1] Card, Stuart K.; Thomas P. Moran; Allen Newell (1980). The keystrokelevel model for user performance time with interactive systems.

Communications of the ACM 23

[2] Arkin, Ronald C. (1998). BehaviorBased Robotics.

[3] Clark, Herbert H.; Brennan, Susan E. (1991), Resnick, L. B.; Levine, J. M., eds., Perspectives on socially shared cognition, American Psychological

[4] Alan Cooper. About Face

[5] Skantze, Hjalmarsson, Oertel. Turntaking, feedback and joint attention in situated human–robot interaction, Speech Communication, 2014.

[6] S. Waldherr, R. Romero, and S. Thrun. A gesture based interface for human-robot interaction. Autonomous Robots, 2000.

[7] K. Dautenhahn, S. Woods, C. Kaouri, M. Walters, K. L.Koay, and I. Werry. What is a robot companion friend, assistant or butler? IEEE IROS, 2005.

[8] Aaron Powers; Sara Kiesler; Jennifer Goetz; Matching Robot Appearance and Behavior to asks to Improve Human-Robot Cooperation ROMAN, 2003.

[9] Herbert H. Clark; Susan E. Brennan.

Grounding in communication. 1991

[10] Brennan, Susan E. The Grounding Problem in Conversations With and Through Computers. Social and cognitive psychological approaches to interpersonal communication 1998.

[11]Ben Shneiderman, Response Time and Display Rate in Human Performance with Computers.

Computer Surveys, Vol. 16, No. 3, 1984.

[12] Leila Takayama: Doug Dooley; Wendy Ju. Expressing thought: Improving robot readability with animation principles. HRI 2011

[13] Matthias Scheutz; Paul Schermerhorn; James Kramer. The utility of affect expression in natural language interactions in joint humanrobot tasks. HRI 2006

[14] Emily Wu, Yuxin Han, David Whitney, John Oberlin, James MacGlashan, Robotic Social Feedback for Object Specification, AAAI Fall Symposium , 2015 [15] Bartneck, C., Croft, E., Kulic, D. & Zoghbi, S. (2009).

Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots.

International Journal of Social Robotics

IT'S OK TO BE CONFUSED