Non-verbal affective communication in mobile media - with whom matters Ruben van Dijk
Pepijn Verburg
Shigeru Yamada
r.j.j.v.dijk@student.tue.nl
p.verburg@student.tue.nl
s.yamada@student.tue.nl
Yang Yu y.yu@student.tue.nl
Department of Industrial Design, Eindhoven University of Technology ABSTRACT
The purpose of this research is to describe how people experience non-verbal affective (NVA) communication in mobile media. Ethnographic interviews (Spradley, 1979) are conducted with aspects of Cultural Probes (Gaver et al., 1999) to find out when, why, to whom and how people use NVA communication. A variety of different cultures were included in order to find similarities across cultures. In summary, we found relation between intensity and type of the emotion are there (Schachter & Singer, 1962) and these four questions. More intense emotions tend to be communicated with closer circles with the exception of happiness (in the context of special events, fun or beauty). A hypothetical model is presented. Finally, non-verbal communication is seen as a way to set the tone and to leave a message open for interpretation. The results can be used as guidelines for future media platforms or creating a platform supported across cultures. INTRODUCTION
Non-verbal communication is a significant component of our everyday communication (Argyle & Hinde, 1972). With the rise of mobile media, shifts can be seen in nonverbal communication towards the form of emoticons, pictures, music or other abstract representations to support the main medium: the written word. In this research we dove into these phenomena to explore how mobile media is used for non-verbal affective (NVA) communication across different cultures. The aim was not to find cultural differences, but to find shared patterns. The ultimate goal was to define recommendations for existing media to open up for other international markets or give directions for a true world-wide media platform supported by different cultures. We saw this as a great opportunity, because somehow there is still a clear distinction between what media is used around the world. In summary, our question is: How is mobile media used for non-verbal affective communication across different cultures? Approach
In this research we used the study of ethnography with ethnographic interviews (Spradley, 1979). Our intention is to provide a hands-on semi-structured setup to get people to be more interactive during the interview. We got inspiration from Cultural Probes (Gaver et al., 1999). The informants
received small tasks to perform during the interview. The results of these tasks are used as data and as input in further discussions. With laddering we went more into depth on topics of choice during the interview (Hawley, 2009). Participants are sampled with maximum variation sampling on cultural background. The contents of the interviews are based on four sub questions: (1) When do people show NVA communication in mobile media? (2) How do people describe the motivation for NVA communication in mobile media? (3) Who do people communicate with in a non-verbal affective way through mobile media? (4) And how do people express non-verbal affect in mobile media? The goal is that the answers to these four questions (when, why, who and how) would create a clear image of how NVA communication is used in mobile media across different cultures. Definitions
Non-verbal affective (NVA) communication Non-spoken and non-written information where emotions and feelings within a certain context are communicated with. Mobile media Mobile media in this paper are digital social communication platforms that are available on a phone and/or tablet. ITERATION 1: PILOT INTERVIEW
For the initial version of the interview we designed 7 handson tasks for stimulating various aspects of use of mobile media and of non-verbal affective communication (see Appendix B). Procedures
A pilot interview was conducted between the team members. After getting a concrete episode with the diary method, we inquired the possible posting behaviors. Then we used the Sheets of Emoticon, Contact Group, and Sample Article for inquiry.
Findings
We experienced that every hands-on task worked well, but from the time and research question appropriateness perspective, we found that we needed to eliminate the tasks into half and to add open-ended questions derived from each research sub questions. The new open questions can be read in appendix D.
times and finally clustered. In table 1 you can see which groups and clusters emerged. In appendix H you can see all the chunks grouped accordingly. Cluster
Groups
Non-verbal Emoji positive, emoji negative, non-verbal, affective extensive emoticon usage, images, abstraction. communication
ITERATION 2: INTERVIEWS
Embarrassment, types of media, share with whom, Emotion happiness and beauty, face-to-face stronger emotion, fun.
Procedures
Attitude Oversharing, positive, negative, attention, bragging.
The hands-on tasks are considered as a warm-up and the outcome is used as reference material for the open-ended questions. In appendix E you can find the setup used during the interview itself. Many of the hands-on tasks are related to the open-ended questions. In this way it could serve as back-up input when the open-ended questions would not result in a rich qualitative answers. Sampling
An important aspect of the research direction is to discover what people from all over the world using mobile media have in common. Therefore, as said before, we choose for a maximum variety sampling: we interviewed a total of 8 people with 5 different cultural backgrounds: Chinese, Singaporean, Australian, Dutch, and Lithuanian. In order to find multicultural informants the “common room” of the foreign student association “Cosmos” of University of Technology Eindhoven was used as the recruitment location. Five females and three males were recruited. Interviews
The informants were briefed about the subject and procedures of the research and were offered a soda and chocolate to thank them for the cooperation. Once they agreed on the interview, their age, gender and cultural background were documented. After the hands-on tasks were given photos were made for the documentation. Also, the hands-on tasks served as a tool to give the informant an idea of what the subject and research is about, and what kind of answers are welcome on the open questions that followed. At the end of the interviews, which each approximately had duration of 30 minutes, the students had the opportunity to provide their email addresses in order to keep them up to date of the research. After each interview, the information and photos collected during the interview were documented digitally. Analysis
Affinity Diagramming (AD) (Pyzdek, 2003) is used to analyze the transcriptions from the interviews (see appendix F for the transcriptions). The research team reviewed all the digital content several times, and added comments to mark useful chunks. Both the chunk and the comment were printed out. First, the research team grouped them according to their contents. The groups were revised several
Gender, interpretation, differences of people, Personality culture, privacy, face-to-face, the change of people on mobile media. Food, serious things, different experiences, choice of media and groups, interests. Table 1: The affinity-diagram consisted of 5 clusters, each containing 5-7 groups that relate. Events
Findings
With the addition of the open-ended questions the interviews were immediately experienced more open. The data was richer in a sense that people were thinking consciously about their experiences and decision making in mobile media. Now some conclusions will follow, related to each of the four sub questions. (1) When do people show communication in mobile media?
non-verbal
affective
From the sample it became clear that at the origin of each post containing non-verbal affective content like a picture on mobile media there was emotion. The group Happiness and Beauty within the cluster Emotion of the AD contains write-ups like: “For example, I would express my feelings in mobile media at my graduation ceremony, because it is a happy situation.” People do not share activities that feel normal. Each subject of the post gave the person a feeling of happiness in the context of a special event, fun or beauty. People impulsively post and use non-verbal affective content as a result of this feeling. Often special situations or events trigger these feelings, such as special food, interesting topics or group related situations. Once emotions become very strong, they will try to communicate this in a as real as possible way, such as using video calling. The group Face to Face Stronger Emotion, also within the cluster Emotion consists of reasoning like: “I find Skype way more suitable for these types of communicating. Text causes a lot of misunderstandings.” In these cases they do not use non-verbal affective communication tools such as emoji or pictures anymore. Private situations often bring emotion, but people consider their privacy before posting. If the situation more emotional, it is likely that they will share their experience non-verbally. People are also selective on showing emotion. They think about what image that the viewer will get of them.
(2) How do people describe the motivation for non-verbal affective communication in mobile media?
Sharing is always motivated by emotion. Almost every post of non-verbal affective communication is about sharing. However, people are not consciously thinking about the emotional relation with all posts. For example, people post a picture of a dish and comment: “It is so delicious!”. Behind this you can also find an emotion. In addition, people tend to share positive emotions rather than negative ones. They like to show happiness (in the context of special events, fun and beauty) and some embarrassing but interesting experiences. For negative emotions, people have target groups to share with. With more intense emotions they prefers face to face communications. Furthermore, people changed their motivation of sending which kind of non-verbal affective communication over time. “In the beginning of Facebook, I used it as a trash-bin to just put anything on it. Now, I mainly use it for educational purposes and useful cases.” From the words we could see, people shared a lot when social networks were just launched, but after they used it for a long time, they are a little tired of it. People also change the platform to share their emotions or feelings over time. Another typical phenomenon is that some people would post negative things before, but they do not do it anymore. Finally, we could see from the data that people use nonverbal affective communication as they need to update with others on mobile media. People need to keep their social relationship with others. “With two good friends (one in Spain and one in Australia), this is general about happy emotions, but possibly also sad or angry.” Normally, people only communicate strong feelings with close friends. (3) Who do people communicate with in a non-verbal affective way through mobile media?
The third question dives into whom people communicate with non-verbally and affect through mobile media. Many of our informants put forward that the intensity and type of an emotion is important aspect in their actions on mobile media. The type was already put forward in the first sub question about when people show NVA communication. The intensity however, comes forward when talking about who to communicate with and how they describe the motivation (second sub question). It makes a big difference when you are sad, because you missed the train or when your grandmother died. These two properties relate strongly to Schachter’s Theory (Schachter & Singer, 1962) about emotion where type and intensity determine the perception, bodily arousal and thereby future decision making. It looks like similar processes are taking place within the context of mobile media. Peculiar are statements as “I think I was so happy I needed to ‘get rid’ of my emotions” are convincing for such phenomena. When going more into depth in this decision making we see that not all intense emotions are triggering NVA
communication to a bigger public. In the first sub question three were defined that do: happiness, perception of beauty and fun. When asking people with whom they communicate with on emotions as sadness or anger they quickly refine their selection to “close friends”, “family” and “parents”. As mentioned before, some add to this that “face to face” contact is then preferred (see AD group: Face To Face Stronger Emotion). (4) How do people express non-verbal affect in mobile media?
We picked up two interesting findings about the expression. (1) People prefer to express their affect in an “abstract” way rather than in a verbal way. People prefer to convey their emotion or feeling with emoticons or emoji and images, and sometimes with music. Some informants said: “non-verbal content can be understood easily and contains more emotion”. Others even said: “text is confusing”. One of the Asian informants explained the reason of using Emoji by saying: “to avoid being impolite”. (2) Secondly, non-verbal communication is open for interpretation. People prefer the room of interpretation. Another Asian informant said “non-verbal content tells more, a picture says more than a thousand words. Also it leaves room for interpretation.” The answers to the sub-questions were used to answer the main research question: How is mobile media used for non-verbal affective communication across different cultures?
Overall we see several important processes. (1) Non-verbal communication is mainly used to set the tone and (2) sometimes to open up a message for the reader’s interpretation. (3) There is a strong tendency to use nonverbal communication when an emotion is involved. (4) Informative posts tend to include text. This could be explained by the abstraction level an emotion is on. People try to communicate this on a similar abstraction level by not using text. (5) As said before, the intensity and type of the emotion determines if it is communicated and who it is communicated with. Especially the impulsive emotion of happiness (in the context of special event, fun or beauty) is put out in the world quickly. We noticed there is some kind of relation that can be represented in a model (see table 2).
Type
Who
How
Impulsive
Many people
Shared broadly
Negative
Only close
Face to face
Strong Only close Privately shared Table 2: relations between type of emotion, the target group and the format of sharing
DISCUSSION Model
The model presented in table 2 could be more precise and more into-depth, because the different dimensions are now very limited. The model also wasn’t one of our focus points but emerged from the data. In our opinion future quantitative research would support it in a better way. Analyzing higher quantities would in this case lead to more solid conclusions. Some existing research even contradicts this model through quantitative research. They pose that anger spreads faster and more broadly than joy (Fan et al., 2013). Our data doesn’t show this at all. One important difference is that Fan was interested in the spreading component. This research dives into the origin. Cultural differences
We had an impression that Asians tend to regard other’s faces and feelings more than Westerns. But we cannot draw conclusions on that based on our data. More interviews with Asian and Western informants should be conducted in the future in order to confirm this. Recommendations
Google+ [1] is an example where the people are invited to create groups (so called ‘circles’) of people to select when sharing information. It is interesting to see that our findings also relate to these differentiations in groups. However, somehow Google+ doesn’t work and it seems that Google is lowering its support (DeMers, 2014). In our research impulsiveness was an important aspect of mobile media. What Google did with introducing circles was to let this impulsiveness die. People are invited to consciously think about whom to share their messages with. In reality however, people almost automatically determine who to share it with and pick the appropriate media. A conscious representation of all your different circles forces people to think about it resulting in doubtful situations not knowing whom to decide on. We experienced some of this doubt during the interviews, but it was not recorded properly. This
could be an interesting topic for future research to focus on. People are generally well aware of the social situations and appropriate actions: is making this conscious really a good decision? [1] https://plus.google.com, visited on 21 November 2014 REFERENCES
1. Argyle, M. & Hinde, R. A. , (1972). Non-verbal communication. ,Oxford, England: Cambridge U. Press, xiii, 443 pp. 2. Brown, B.A.T., Sellen, A.J. and O’Hara, K.P. (2000). A Diary Study of Information Capture in Working Life. Proceedings of ACM CHI’00 Conference on Human Factors in Computing Systems, 438-445 3. DeMers, J., (2014). Visited on 21 November 2014. http://www.forbes.com/sites/jaysondemers/2014/04/28/i s-google-really-walking-dead/ 4. Fan, R., Zhao, J., Chen, Y., Xu, K., (2013). Anger is More Influential Than Joy: Sentiment Correlation in Weibo 5. Gaver, W, Dunne, A., & Pacenti, E (1999). Design: Cultural probes, Interactions, Vol 6, Issue 1. 6. Hawley, M., (2009). Visited on 20 November 2014. http://www.uxmatters.com/mt/archives/2009/07/ladderi ng-a-research-interview-technique-for-uncovering-corevalues.php 7. Pyzdek, T., (2003). The Six Sigma Handbook. 8. Schachter, S., & Singer, J. (1962). Cognitive, Social, and Physiological Determinants of Emotional State. Psychological Review, 69, pp. 379–399. 9. Spradley, J. P., (1979). The Ethnographic Interview, Wadsworth-Cengage Learning