New Frontiers in Human Emotion Recognition, Capture, Display and Transmission

New Frontiers In Human Emotion Recognition, Capture, Display and Transmission SHI YIHANG / B9178806, Supervised by David Hall

#IDAS #DigitalMediaDesign #Thesis2020 #humanconnection #socialmedia #emotionrecognition #emotiondisplay

1. Introduction 1.1 Background and Problem 1.2 Purpose 1.3 Methodology

1.1 Background With the development of social media sites, human connection through online technology is already dominant. Since Facebook, WhatsApp, Twitter, Instagram.etc were launched, for many of us, we keep checking email, Facebook or whatever platforms where we can communicate with others online, which is like an endless loop.

Our need for social interaction is almost an instinct. Online social media makes us think that we can always be heard and get attention. However the more you use virtual online social media, the lonelier you would be.

1.1 Problem Definition In modern cities, we may feel less human connection due to the rapid development of social media. It sounds like a paradox but a truth. Individuals waste amounts of time on internet, especially on social media. Social media addiction has been a concern among people.

1.2 Purpose Digital art can change the relationship among human beings. If the artworks can create beautiful changes among people, then the existence of people themselves is a positive element. While the viewers become part of the artwork itself, it can not only create a new type of relationship among human but also explore a new way to strengthen the human connection.

1.3 Methodology I did the empirical study in the beginning, generated a few optimized concepts and made some experimental projects which is related to human connection. I started with initial ideas first, and quickly jump to execution to see the effect of rough output. That’s because the first idea may or may not be the right one.

After going through the ideation phase, I converged the concepts to help insure that the best idea will result.

Background

Background

Latest data shows that 4.14 billion people across the planet are using social media in October 2020, equating to 53 percent of the total global population.

Our needs for social connection is almost an instinct.

Background If we look through the iteration of modern social communication media, it seems llike we all expected users would require high resolutions.

However this technology seems to be lower both in resolution and complexity,

Communication among people are extremely simplified.

Target Audience - Who need emotion display?

There’s no specific target audience.

It's about human emotional connection without formal language.

❏

Those who’re separate in two countries/regions there’s not much things to talk about everyday, but we’d like to know their emotion while taking a rest without bothering each other

❏

Those who’re not willing to let others see their real face online but need to convey emotion/facial expression like some feedbacks for online class (feedbacks about if you understand or not)

❏

… ...

Case Study Polygram ďźˆApp With just a minimal 20 milliseconds of lag, Polygram shows a wireframe mirror of your face that recognizes a wide variety of emotions corresponding to different emoji. It relies on an artificial intelligence convolutional neural network to detect how your face moves and map it to a specific emotion. As I smiled, frowned and raised my eyebrows, Polygram accurately surfaced the related emoji.

Case Study Smile2Vote （Installation in Subway Station） The SAW 2020 ‘Smile2Vote’ interactive digital screen is located exclusively at City Hall MRT station till 15 January 2020. In addition, on 10 and 11 January, 5pm to 8pm, participants can bring home exclusive SAW merchandise when they smile to vote for their favourite artwork.

Case Study Neutralité : can’t and won’t. facial expression recognition system embedded clothing https://vimeo.com/169623995 Two dresses, named “Can’t” and “Won’t”, displaying an aesthetic and motion reminiscent of microbial life, which react according to a facial expression recognition system and stop moving as soon as the on-looker begins to emote.

They exist at a very real tension between humanity and machines, between self and a new-other.

Digital technology is a great facilitator to enrich audience experience.

2. Empirical Study / Experimental Projects 2.1 2.2 2.3 2.4

Infinite Loop —— Endless Checking on Social Media Red Thread —— Invisible Connection Visualization Digital Shadow —— Mood Visualization Emotion Recognition

*Separate projects which are related to human connections

2.1 Infinite Loop

*mirror effect / kaleidoscope effect to represent the infinite loop

2.1 Infinite Loop Concept - There is an invisible loop in the modern society, which I would called, mobius band. For many of us, that loop is simply checking in on your smartphone, which quickly becomes a loop of also checking email, Facebook, Instagram, or whatever else.

Nevertheless, most of people have already feel sick of this endless news checking, but cannot stop from it. In this case, I was trying to represent this invisible loop in a new media way.

Method - In this case I took videos in JejuDo and Hongdae, then added mirror effects and kaleidoscope effects in AfterFX to show this endless loop.

2.1 Infinite Loop

2.1 Infinite Loop Research Part - Bilateral Symmetry - Most of the animals are bilateral symmetric, which can help them move forward and support the streamlining.

Symmetry provides our brains, which are always looking for structure and patterns in the world, with a sense of harmony.

Symmetry is also aesthetically pleasing because of the balance it provides in an image. The creatures with the symmetric body can adapt the environment much better than those who are asymmetry. Normally, asymmetry is a sign of danger in the nature.

2.2 Red Thread Concept - I deem that there must have some connections between individuals even though we cannot see it. Six Degrees of Separation said that that all people on average are six, or fewer, social connections away from each other, which means, even strangers have connection with each other. But the connections among human beings are not stable. It’s hard to maintain a permanent connection between one and another. Normally you will have new friends in different phases of your life, meanwhile, you'll lose some friends too. Depending on your status, your social network is keep updating.

*took videos near hongdae, then added plexus effect in Adobe AfterFX

2.2 Red Thread Variation of Connections Among Individuals I also did a 2d interaction in Processing3 to represent the relationship between us. It's a mouse pressed event. White lines are others' social networks. When you click/press the mouse, you can connect with others (red lines). In the first version, you can connect with everyone. And in the second version, you can only connect with parts of ppl. Also when you get into another social group, you'll lose some previous connections.

Research Part - Generative Art One overly simple but useful definition is that generative art is art programmed using a computer that intentionally introduces randomness as part of its creation process. The most appealing point of generative art is controlled randomness. I create a frame for this project by coding, but I cannot control the specific result. Unlike analog art, where complexity and scale require exponentially more effort and time, computers excel at repeating processes near endlessly without exhaustion.

As we will see, the ease with which computers can generate complex images contributes greatly to the aesthetic of generative art.

2.2 Red Thread

2.3 Digital Shadow Concept - I hope there’s something which can express our emotions. Most of Asians cannot express themselves well——it’s a shame to express their real feelings and emotion, thus they need the medium to help them express. I chose the shadow as the visible medium. The shadow is too subtle to be noticed, which is . Still I believe that if you care about someone, you will notice every subtle things on that person. That’s what introvert people need I guess.

The more moleculars the shadow has, the more sociable the people are. (Smooth Shape = Talkative; Fuzzy = Leave me alone)

*deleted real shadow in Photoshop, rendered fake shaodw with photo in Maya

2.3 Digital Shadow Way of Interaction v1

v2

v3

v4

2.4 Facial Expression Recognition Concept - Since we have 44 muscles on our face, we can express a wealth of information through different kinds of facial expressions. When we were an infant, we were not able to say anything. However infants are old enough to know the meaning of their facial expressions, they have already known how to use facial expressions to telegraph our feelings and intentions, which is the efficient way of communication. Facial expressions may have evolved as efficient ways to telegraph feelings and intentions. And it is possible that forcing your face to express happiness, sadness or anger may help you feel those emotions.

Input - Facial Expression(detected by faceOSC) OSC data ●

Pose ○ ○ ○

center position scale orientation

●

Gestures ○ mouth width ○ mouth height ○ left eyebrow height ○ right eyebrow height ○ left eye openness ○ right eye openness ○ jaw openness ○ nostril flate

●

Raw ○

raw points (66 xy-pairs): /raw

2.4 Facial Expression Recognition Outcomes - Digital - Physical / RealWorld (Simulation)

Physical Outcome (Simulation) V1 - Petal Breathe The abstract paper flowers can be hang on the ceiling or just put on the ground. Then the changing of your emotion can control the breathing rate.

Research Part - Scale and Amount in Art Large Scale While we experience the scale of artworks, we tend to compare to the size of our bodies.

Oversized things = powerful; giving a sense of grandeur; like a part of nature; immersive Large Amount Immersive - Audiences can walk through/into them and be surrounded by them.

Repetition in Art Helps to create a rhythm and a sense of movement & tension & order The surface of work can be enhanced.

Repetition can transform simple things into complexity and holistic.

2.4 Facial Expression Recognition Digital Outcome V1 - Emotion Morphing

2.4 Facial Expression Recognition Digital Outcome V1 - Emotion Morphing Research Part - Relationship among Color, Shape and Emotion

Color: Happiness - Yellow/Linear, Sadness - Blue, Anger - Red, Fear - Purple, Disgust - Green

Smoothness： Negative Emotion - Turbulence Positive Emotion - Smooth

Wax & Wane： Strong Feeling - Wax Weak Feeling - Wane

2.4 Facial Expression Recognition Digital Outcome V2 - 3D Tranformation (Realtime Interaction) *Spiky & Red = Angry; Smooth & Blue = Sadness Made 3D transformation in Maya, and then exported as a video.

Imported into Processing3 and connected to FaceOSC. Let facial expressions control the animation in Processing3.

In this case, only mouth width can be detected.

2.4 Facial Expression Recognition Physical Outcome V2 - Light and Shadow of Stained Glass (Realtime Interaction)

Rendered by Arnold

2.4 Facial Expression Recognition Physical Outcome V2 - LIght and Shadow of Stained Glass (Realtime Interaction) Research Part I - Beauty of Basic Shapes The simple shapes are the square, rectangle, circle, ellipse, and triangle.

These are the basic forms that are used as the foundation for all other shapes.Squares and rectangles are familiar safe and comfortable.

Squares are considered to be one of the most honest shapes, even more than other types of rectangles, because of their mathematical and visual simplicity.

Simple shapes work beautifully, they are memorable, easy on the eye, and instantly recognisable.

Research Part II - Stained Glass Before modern media apart from print (China 220AD) the only place people where exposed to image changing media was through glass. More specifically stained glass and glass objects. Looking at the world through coloured glass you could call the beginning of real time media. The world looked different and enchanting when looking through coloured or tinted glass. Stained glass and sunlight through coloured glass has always held a fascination for humans as it seems to be a magical experience. This is the reason ancient churches had stained glass windows. The original media was a church with stained glass windows. Since then it has held a special status for people.

Rendered by Arnold

2.4 Facial Expression Recognition Physical Outcome V2 - LIght and Shadow of Stained Glass (Realtime Interaction) Research Part III - Primary Colors Color is an utterly transcendent language of sorts, a way to examine the universal aesthetic. Since we were a baby, the reason why primary colors are so appealing is that we can only see bright primary colors. Also that’s why we’re so familiar with primary colors.

Happiness equals bright and shorter shadow Sadness equals dark and longer shadow

2.4 Facial Expression Recognition Input - Facial Expression(detected by faceOSC); Outcomes - Digital & Physical

2.4 Facial Expression Recognition Physical Outcome V2 - LIght and Shadow of Stained Glass (Realtime Interaction) Simulation by using acetate sheets

2.4 Facial Expression Recognition Physical Outcome V2 - LIght and Shadow of Stained Glass (Realtime Interaction) Motor Controling

ToolsArduino; Motor; Light; Acetate Sheets; Glass; Water It’s a easy simulation of stained glass. I tried to use a cup of water and acetate sheets to simulate the shadow of colored stained glass. And also figured out how to control the motors by using arduino.

3. Development / Final Project 3.1 3.2 3.3 3.4

Conceptual Design 3D Development Technique Deployment

3.1 Conceptual Design Theme of Outcome - Neuron / Synapse In the nervous system, a synapse is a structure that permits a neuron (or nerve cell) to pass an electrical or chemical signal to another neuron or to the target effector cell. Santiago Ramรณn y Cajal proposed that neurons are not continuous throughout the body, yet still communicate with each other, an idea known as the neuron doctrine.

Due to brain imaging techniques, we've already known that the function of the cerebral cortex in general. The frontal lobe is the part of the brain that controls important cognitive skills in humans, such as emotional expression, problem solving, memory, language, judgement and sexual behaviors. It is, in essence, the "control panel" of our personality and our ability to communicate. The connection of human society cannot be separated from emotional communication and expression between individuals, especially facial expressions are one the most important means. Therefore, the brain allocates more areas to these "important departments".

3.1 Conceptual Design Deployment - Hollow synapse model(which can tranfsfer air) in the fish tank -Facial expression controls the air pump -While air pump works, synapse model can generate bubbles in the tank Process analogue mental synaptic activity = emotion = face muscular activity muscular facial activity = digitised = 0101001 digital data to analogue object = installed servo bubble pump analogue

3.1 Conceptual Design Deployment v1 (Physical)

Glass Tank Water Inside Synapse

Laptop with FaceOSC

Air Pump

Latex Tube

Stand

3.1 Conceptual Design Deployment v2(Digital)

LCD Screen Spotlight Laptop with FaceOSC

3.2 3D Development Synapse 3D

3.2 3D Development Other types of Synapse

3.2 3D Development 2 Ways of Assembling

3.3 Technique - 3D Printing

3.3 Technique - Facial Expression Controlling FaceOSC + Processing

import facial data from faceOSC to Processing3

mouth width controlling

3.4 Deployment v1

3.4 Deployment (Simulation)

3.4 Deployment v1 (Air Pump & 3D Model Test)

3.4 Deployment v2 (digital)

LCD Screen

Spotlight Laptop with FaceOSC Power

3.4 Deployment v2 (Exported Videos Test)

5. Conclusion 5.1 Limitation of Study 5.2 Future Opportunity

5.1 Limitation of Study Emotion recognition cannot actually assess an individual’s internal emotions and experience.

People always hide their real emotion and even, facial expressions can be trained as well. We can only estimate obvious emotions approximately.

“No serious researcher would claim that you can analyze action units in the face and then you actually know what people are thinking.”

5.1 Future of Opportunities For now, we can only detect if you’re smile or not. And most of the displays i did are digital.

In the future, we’d like to ‘unlock’ more facial expressions and let all the displays (both digital and physical) can be sold to everyone who’s interested.

This project would be refined and developed as well.