TiCC - The Human and the Artificial - e-Humanities Research

Colofon his brochure is meant for special-interest groups T about research at Tilburg Center for Cognition and Communication, Tilburg University, Netherlands.

Publisher Tilburg University, School of Humanities Editor in chief Jaap van den Herik Editors Leon Heuts, Annemeike Tan Contributing editors Mieke Fiers, Leon Heuts Photo editors Beelenkamp Ontwerpers Tilburg, Annemeike Tan Photography Johner Royalty Free/Hollandse Hoogte (cover), Jean Claude Moschetti/Hollandse Hoogte (p.6), Flip Franssen/Hollandse Hoogte (p.12), Richard Jones/Hollandse Hoogte (p.18), Seunghan Son/ Hollandse Hoogte (p.25) Translation Taalcentrum VU Amsterdam Lay out and graphic design Beelenkamp Ontwerpers Tilburg Printer OCC De Hoog, Oosterhout Š 2011 Tilburg Univerisity, School of Humanities

Contact iCC T Tilburg University, School of Humanities PO BOX 90153 5000 LE Tilburg Phone +31 (0)13 466 8118 E-mail ticc@tilburguniversity.edu Website: www.tilburguniversity.edu/ticc

The Human and the Artificial

e-Humanities Research

Tilburg Center for Cognition and Communication

e-Humanities Will we ever be able to interact with computers in a totally familiar and natural way? Could we communicate with them in exactly the same way as we do with other people? That is what TiCC’s research into creative and intelligent systems is working towards.

Under the slogan, “Make the computer adapt to us, not the other way round.”

16 NON VERBAL COMMUNICATION Interview: A sensitive avatar Research portrait: Hand gestures and computers

3

20

Preface Jaap van den Herik e-Humanites

SOCIAL MEDIA Is a Facebook friend a real friend?

Research portrait: The social signals in human voice

Language, Communiation and Cognition led by Emiel Krahmer and Marc Swerts

4 GAMES Interview: Fighting fires with serious games Research portrait: Serious game, psychological tool

24 HUMAN COMPUTER INTERACTION Interview: The computer must adapt to us, not us to it Student projects: User interface design for a better life

Creative Computing

led by Antal van den Bosch and Eric Postma

8

ART AND VISION Interview: Fake Van Goghs, suspect shipping and real tumours Research portrait: Archaeology and computers

12 CULTURE AND LANGUAGE Interview: High-speed heritage trawl Research portrait: History and computers

As such, our work is exploring both intelligent systems and human intelligence. To find out how computers can communicate with us in a natural fashion, we need to know more about how we do that with each other. How do we use language? And what about non-verbal means of communication, such as hand gestures?

TiCC brings together a wide spectrum of academic research disciplines: computer science, artificial intelligence, psychology, neurosciences, linguistics and sociology. In our work on AI, for example, we are trying to train computers to generate their own rules or algorithms from large quantities of data. A computer learns by measuring and checking, by recognizing patterns and by applying intelligent combinations and recombinations.

Amongst the resources being developed by TiCC with the help of machine learning are intelligent databases. Another line of research is developing algorithms that can be used to tell the difference between a genuine Van Gogh, say, and a forgery. And there are more questions we hope to answer. For example, can a computer identify potentially dangerous situations by analysing the movements of a group of people? Or differentiate a tumour from a healthy tissue?

Then there is our research into human communication. Even though this is something we are all intimately familiar with, it is difficult to formalize in terms of hard and fast rules. We use deictic adverbs like “here” and “there” effortlessly in conversation, and when talking we often gesticulate without even thinking about it. But when exactly?

And how? Our work is identifying conversational situations that are enhancing our knowledge of human communication, with a view to enabling computers to reproduce them in a naturalistic manner. By, for example, recognizing human emotions. Through this research we are entering a new phase in the development of artificial intelligence, with intuitive computers that can “sense” situations and so respond appropriately to unexpected circumstances. Eventually, we will be able to communicate with computers on an equal footing.

By understanding the computer better, we are learning to understand people better.

That is why I call our field “e-humanities”.

Jaap van den Herik, professor Computer Science and director of TiCC

Preface

3

Games

Lifelike games help in high-risk jobs

Interview

Jaap van den Herik and Pieter Spronck

Fighting fires with serious gaming Computer games are now more than just entertainment. People in dangerous professions, like the fire and ambulance services, are using them to train in virtual environments. But how do you make them as realistic as possible?

Deep in concentration, the soldier scans the hostile landscape, alert to any suspicious movement or potential danger. The turret of his tank obscures part of his view, and his radio is crackling in his ear. He is not in Iraq or Afghanistan, though, but in a virtual world in the Netherlands. There, troops are trained by a sensemaking system in “managing the unexpected”. Serious gaming is one of the research topics being addressed by Professor of Computer Science Jaap van den Herik and researcher Pieter Spronck. In this field, Tilburg University works closely with the Netherlands Defence Academy (NLDA).

“When we talk about the intelligent use of computer systems,” says Spronck, “we can divide that into three distinct forms.” First there is the computer as a repository of knowledge. A tool, in other words. “You look something up – on your iPad, for example. Then you, the user, interpret what you find.” The second form brings the computer into the human world. This covers everything from “intelligent” garden furniture (“Where do I position myself so that my owner is sitting in the sun?”) to chess computers, a subject Jaap van den Herik has studied

in depth and made a major contribution to as a researcher. Finally, you can take people into the computer’s world so that they experience a total virtual environment.

There is already a flourishing industry in intelligent computer worlds and entertainment games, like World of Warcraft and Second Life. But gaming is now also rapidly gaining ground in the professional domain. Firefighters, for instance, can practise how to enter burning buildings in situations which would be too dangerous for real-life training. Police officers and other front-line officials can use simulations to learn how to deal with difficult members of the public. Managers can improve their specific skills and ambulance crews can brush up their patient communication abilities.

For the training to be effective, it is vital that the “games” – in the form of simulations – be as realistic as possible. That is still not always the case, says Spronck. “In the training game for ambulance crews, for example, actors are often employed to play out the emotions involved. A computer is simply unable

to express them realistically.” That degree of realism is also a “Holy Grail” for entertainment games, since the enjoyment of immersing yourself in an entirely different world is spoilt every time the computer “shines through”.

“There are games and training environments that look perfect,” says Spronck. “Photorealistic, you might say.

But you can still see immediately that they’re not real. If a robot here in the room walks into a wall, we laugh. If a virtual person in a game does it, we say ‘how stupid’.” So the computer world not only has to look real, the people in it

5 Games

4

Games

also have to behave realistically. A good deal of the research being conducted by Van den Herik and Spronck is homing in on this factor.

For a computer to imitate a person convincingly, you have to teach it what typical human behaviour is. And that is difficult to pin down. “People and computers are so different,” explains Van den Herik. “They reason in such different ways. Chess computers may be able to beat even the world champion, but when you really think about it you realize that they’re not actually playing the same game. If you feed a computer with examples of ‘well-mannered’ drinking, say, then it analysed what constitutes ‘well-mannered drinking’. At the moment, that learning process is still quite laborious.” But Spronck predicts that it will become steadily easier for the machine. “We have to reach the point where a computer is able simply to observe its environment and pick up from that any mannerisms it is capable of learning.”

“Managers can improve their specific skills and ambulance crews can brush up their patient communication abilities.”

Research portrait

Giel van Lankveld

Serious game, psychological tool How you respond in a particular situation. Your way of thinking. What motivates you... All are related to your personality. And not just in “real life”. Research by PhD student Giel van Lankveld has revealed that a person’s game play in the virtual world is also influenced by their personality, in a variety of ways.

Van Lankveld spent several years studying Computer Science, earned a degree in Psychology and now combines the two fields in his work at TiCC. “I originally began my research to make computer games better suit their players,” he explains.

To do that, he drew upon ideas from psychology. For example, he applied an incongruence model to games. This predicts that new users of software like Word or visitors to a website form a picture of what they can do with it. If the experience turns out to be more complicated than expected, they become frustrated. More experienced users, on the other hand, can cope better with the software but eventually start to find it boring. “Applied to games,” says Van Lankveld, “you can conclude from this that they have to be flexible in order to retain their appeal and to interest everyone.”

“Recognizing behaviour and non-verbal communication is becoming increasingly important,” he continues. “If we start using robots to help us move house, it’s not such a problem if they behave like computers. But in the not too distant future we’re also going to need them in the healthcare sector. And for that we’ll need a model of a human being.” Those computers will have to learn to recognize emotions and to respond accordingly.

Van den Herik’s research group is interested in every possible form of human behaviour. For instance, two PhD students from the National Police Services Agency (KLPD) are working on the fight against terrorism. Nick Koeman is studying the characteristics of attacks like the one targeting the Dutch royal family on Queen’s Day 2009, whilst Peter Berenschot is looking into factors whereby you can identify somebody as a terrorist. “The result of his research will be a realistic training module,” says Van den Herik. “And together,” adds Spronck, “the two studies should enable a computer to identify any weak points in security systems at an early stage.”

Another psychological concept he has applied to the world of games is personality theory. Again, the original aim was to adapt games to their users. But he has now shown that personality plays a key part in determining player behaviour, whether they are on something like World of Warcraft or taking part in one of the serious training games used by, say, the police. This raises another interesting question: can a serious game help in classifying an individual’s personality? Psychologists conduct a personality test to do this, but that has a number of drawbacks. In particular, the test currently in use is not suitable for children. Van Lankveld is now investigating whether a game might deliver comparable results, or perhaps even better ones. In this way he hopes to develop a useful tool for psychologists.

Newstweet Robots will be able to do everything better than people in the future #TiCC

Want to know more?

TED Talk: Peter Singer on military robots and the future of war

http://www.ted.com/talks/pw_ singer_on_robots_of_war.html

7 Games

6

Art and Vision

The endless possibilities of image recognition

Interview

Eric Postma and Jeroen Janssens

Fake Van Goghs, suspect shipping and real tumours A technology that recognises painters from their brushstrokes, spots ships acting suspiciously and identifies possible tumours at an early stage – image recognition does it all.

Got a fake Van Gogh in your attic? Don’t get excited, because art historians have recently added a new weapon to their armoury of tools for determining the authenticity – or otherwise – of the artist’s work. It is a computer system which analyses putative Van Goghs by comparing their visual structure with that of his known paintings.

The man behind the system is Eric Postma, Professor of Artificial Intelligence at TiCC. The Van Gogh project is just one of many he is working on in the field of visual perception. We humans can tell the difference between, say, a spoon and a book at a glance. But the underlying brain processes involved in drawing that distinction are actually

very complex. Postma and his colleagues want to teach computers to do much the same thing with visual images. A technology that could be used to help track down criminals, for example, but also one with the potential to improve search engines like Google Images.

Postma is developing image-recognition algorithms, from which a new image can be identified or classified based upon a collection of known examples. “The algorithm translates the image into a series of numbers,” he explains, “in such a way that visual representations which resemble one another produce similar numerical sequences.” The scientist does need to give the computer a helping hand, though. “We know, for instance, that people tend to concentrate upon the transitions from light to dark. That’s something we can describe mathematically and tell the computer about.”

Inspired by a visit to an exhibition of Impressionist paintings, Postma and his colleagues decide to test one of their algorithms on those works. That was the beginning of a highly successful project. They ‘fed’ a computer with digital reproductions of paintings, together with the names of the artists. From that input the system was able to automatically identify particular characteristics associated with each painter. “Van Gogh used a very particular brushstroke,” explains Postma.

“His paintings from certain periods are composed of a huge number of very short strokes.” Given a new, unidentified work to analyse, the computer would try to attribute it based upon the characteristics it knew. And it was successful in 95 percent of the cases. The group’s follow-up project homed in on works by Van Gogh.

Converting the numerical sequences into a graph produces a scatter plot in which the genuine Van Goghs are grouped closely together. If a new painting falls outside that “cloud”, the art historian needs to take a really good look at it. “This is not a process that replaces people,” Postma stresses, “but a tool they can utilize.”

Jeroen Janssens, a PhD student working at TiCC under the supervision of Eric Postma and Jaap van den Herik, is using the same principle to tackle a very different issue. For defence technology firm Thales Netherlands, he is designing a computer system to assist the coastguard. “At any given time,” he explains, “there are 600 or so ships off the coast of Rotterdam. The coastguard keeps an eye on them all on big screens in its control rooms.” Janssens’ system picks out those which are behaving in an unusual fashion, and so require extra monitoring. “That could be a fishing boat which is stationary in an area where fishing is prohibited, a tanker entering the

An analysis of the brush-strokes in Van Gogh’s “The Sower”

9 Art and Vision

8

11

Research portrait

Art and Vision

port too fast or a vessel sailing in the wrong direction – even out at sea there are shipping lanes you have to follow.”

“The computer has to learn what constitutes strange behaviour on the part of ships,” adds Postma. “But what does that mean? Is it standing still? Coming too close to another vessel? And how close is that?” Because it has proven difficult to define hard and fast rules for unusual behaviour, Janssens has been exploiting the computer’s ability to recognize patterns in the data it is supplied with. From a huge number of examples, he has abstracted “normal behaviour”. Not so much by defining a norm as by identifying deviations from it.

The techniques being developed by Janssens, Postma and their colleagues are abstract. “It doesn’t matter to the computer whether it’s dealing with ships or paintings or apples,” Postma explains. But the theoretical systems are always linked and applied to real-life domains, be they art or maritime security. “The two sides of the process go hand in hand. By testing it constantly in practice, you know whether your algorithm is working and what direction you need to take it in.”

Postma is also looking forward to his next project. That centres on non-verbal communication, a topic which is already engrossing several researchers at TiCC. “Right now,” he says, “they still have to record manually how often, say, a particular television newsreader raises her eyebrows on air. But we can develop an algorithm to recognize that.” That knowledge could then be used to teach computers to identify and reproduce facial expressions. “In the future our computer screens will show a virtual face which can communicate with the user both verbally and non-verbally.”

It is a tough challenge, but the possibilities seem endless. “We’ve recently teamed up with Marius Nap, a pathologist at the Atrium Medical Centre in Heerlen and research affiliate of TiCC,” Postma continues. “He works with cross-sections of cancerous tissue. It should be possible for a computer program to identify the anomalies in them.”

This photograph illustrates the use of a filter that is sensitive to the visual contours of brush-strokes. The higher the intensity, the more visible the brush-strokes.

Laurens van der Maaten

Archaeology and computers Archaeologists can garner all kinds of information from the shape of the glass found in mediaeval rubbish pits. Its form tells them something about the time in which it was used; the Middle Ages, after all, were as fashion-conscious as our own times. To catalogue their finds, the archaeologists draw them in vertical cross-section to produce a “profile”.

Researcher Laurens van der Maaten has developed a technique for comparing these sections. Various properties of each are analysed, enabling the computer to calculate how closely its parameters match other pieces of glass in the collection. The greater the correlation, the more similar the profiles are. This technique also allows the computer to present similarities and differences as a scatter plot.

Newstweet The better computers become at detecting counterfeits, the better counterfeiters will get at making them. #TiCC In 20 years, computers and computerized analysis methods will be a standard part of the art historian’s toolbox. #TiCC

“Take a map of the Netherlands,” Van der Maaten explains. “Using that map, you can determine the distances between various towns and cities. What we are doing is the same process in reverse: we know the distance between the profiles, and we try to draw the map that corresponds to those distances.”

Very similar profiles are positioned close together, whilst one which is quite distant from the rest may indicate that the find dates are from a completely different period.

Want to know more?

Article: Eric Postma et al., ‘Image Processing for Artist Identification - Computerized Analysis of Vincent van Gogh’s Painting Brushstrokes’ http://people.ece.cornell.edu/ johnson/spm.pdf

“Seven kinds of mediaeval forget-me-nots” Van der Maaten was awarded his PhD for his work on the new technique at TiCC in 2009, and is now a research affiliate at the centre. Application of the procedure is not confined to illustrations: it can be used with any data where the parameters are convertible into distances on a chart.

“Our colleagues working on speech recognition could use it to compare audio files,” claims Van der Maaten.

This might enable them to tell whether the same person is talking in two different extracts, for example.

Van der Maaten recently performed an analysis of seeds recovered during archaeological digs. Those found in ancient pots can help archaeologists to determine what people ate during a particular period, and what crops were grown at the time. “We applied the technique to the seeds of forgetme-nots,” he says. “Thanks to the resulting chart, we were able to distinguish seven different varieties very clearly.”

Art and Vision

10

Interview

Culture and Language

Antal van den Bosch and Maarten van Gompel

13 Smart database saves scholars more than just time

High-speed heritage trawl Historical research can sometimes take many years. But thanks to a smart database being developed at TiCC, the whole process could soon be very much faster. And the computer can reveal surprising facts about the past, without even “knowing” what it is doing.

Anyone interested in delving into the cultural heritage of the Netherlands can turn to a whole variety of institutions, such as the National Library, the National Museum of Natural History and the International Institute of Social History. These have collected and preserved mile after mile of cultural documentation for our enlightenment, and that of future generations. Until recently, though, finding the answer to a particular research question or aggregating information from all these texts was a formidable challenge. One historian, for example, ventured to catalogue every strike ever held in the Netherlands. That became the work of a lifetime.

To make their archives more accessible, these institutions are currently digitizing as much of their collections as they can. The National Library alone is in the process of scanning eight million newspaper pages, and making them available online, so that anyone can search through them with the help of tools like Google.

“But that’s not enough for serious scientific research,” says Antal van den Bosch, Professor of Memory, Language and Meaning at TiCC. “A search query reveals what you’ve actually looked for. But you don’t know what you might have missed because, say, there are various synonyms for your search term. And there’s also a lot of socalled ‘noise’: hits that are irrelevant to your query. When using a search engine in everyday life, we solve that problem mentally. We usually realize which results are worth clicking on and which are not. But that approach is inadequate for serious searches.”

took place, and similar key pieces of information from raw text and place them in easily searchable databases.”

The team is also trying to teach the computer to recognize more complex patterns. One example might be the string of events surrounding a general election: not just the vote itself, but also the campaign preceding it and the process of forming a government afterwards.

TiCC has already enriched the database of strikes originally compiled so meticulously by hand for the IISH, says Van den Bosch. “We are able to add information to it, such as the names of strikers or strike breakers, or details of events in the immediate run-up to the dispute. That sometimes reveals historical connections that no-one had noticed before.”

The basic idea behind the programs being designed by Van den Bosch and his colleagues is that the computer learns to recognize patterns. For instance, there are certain words which have a particular meaning in a particular context. The computer does not know what they mean, but it does come to realize that how they are positioned is significant. It is taught to do that by feeding it with examples. “We believe that we can derive predictive models from data,” says Van den Bosch. “In other words, we have a radical, datadriven perspective.”

To overcome this, Van den Bosch and his research group are working with the institutions mentioned above. “At TiCC we are developing technology that can His group is not confining its work to ‘retrieve’ information on individuals, events, the place and time these events historical texts. It is also applying the

Culture and Language

12

15

Research portrait

Culture and Language

same idea to language processing by computers in general. “Some people believe that language is a system governed by rules,” he says. “But we view it as a means of communication containing patterns within which you can discover statistical regularities. Language is not only about words but also about groups of words – constructions – which convey meaning.” This is a perspective shared by a number of linguists working, like Van den Bosch, at Tilburg’s Faculty of Humanities. In his approach they recognize the way in which people learn a language. “We had

a Hungarian researcher who spoke perfect Dutch. That, so she said, was because she had read a great deal in Dutch and also watched a lot of TV here. She had not learned a construction like ‘Ik belde hem gisteren op’ [I called him yesterday] in the way you would learn individual words, but she had seen and heard it so many times that she knew the choice of words and their order was correct.”

Starting from that idea, TiCC is developing several models with a variety of applications. Some to identify errors, some to paraphrase, some to translate. And some which can complete a sentence. As Van den Bosch explains, “If you type ‘It’s a cold w…’, our model will suggest ‘winter’ because it has frequently seen ‘cold’ and ‘winter’ used together in this way.”

Matje van de Camp

“It’s not easy charting Domela Nieuwenhuis’ network” With whom did Ferdinand Domela Nieuwenhuis associate? Who were his friends? Who did he argue with? PhD student Matje van de Camp is currently researching the relationships maintained by the great politician Domela Nieuwenhuis (1846-1919), one of the founders of the socialist movement in the Netherlands. Her social network analysis could thus shed new light on the early years of Dutch socialism. As she explains, “If two people start out as friends but later on they are enemies, you can safely assume that something has happened between them. You can then look for what that was.”

Crucial to the effective functioning of these models is the number of sample texts from which the computer can abstract constructions. Scientific programmer Maarten van Gompel is working on one such ‘corpus’. In his case, the key aspect is the meaning of a term. “Words like ‘image’ and ‘bank’ have multiple definitions,” he says. “I’m developing a model whereby I can teach the computer to choose the correct meaning from the context.” The definitions of a huge number of words in given contexts are being entered, with some of the input by hand. This model is an important component in a translation engine on which TiCC is working.

Van de Camp has been able to draw upon the huge historical archive held by the International Institute of Social History (IISG) in Amsterdam.

“It contains a lot of material about Domela Nieuwenhuis,” she says. “He appears frequently in biographies of other people. We have manually classified a portion of those texts to produce the input for the system.”

From that material, the computer can then be trained to retrieve similar information from the remaining texts and classify it in order to produce an overall picture of Domela Nieuwenhuis’ relationships. But there are plenty of hurdles. In many cases, for example, the relationships are described by third parties and thus are rather subjective.

Van Gompel and his fellow scientific programmers have another important task within the research group: making the systems it has developed ready for actual use.

“If you promise something, you have to deliver it. So once we’ve developed a system – an alternative spellchecker, for example, or a translation engine – we want to put a demo version online.” This makes the results of research available immediately to everyone. And in many cases a model like this becomes a tool for use in subsequent scientific research.

Want to know more?

Maarten van Gompel, Antal van den Bosch, and Peter Berck, ‘Extending Memory-Based Machine Translation to Phrases’ http://lotos.library.uu.nl/publish/articles/000362/bookpart.pdf

Implicit Linguistics.

Machine Learning of Text-to-Text Processing http://ilk.uvt.nl/il/

Newstweet Thanks to user-friendly online resources, nonexperts will be able to find their way through an enormous mountain of documents much more easily in the future. #TiCC

Van de Camp’s research forms part of HiTiME – short for Historical Timeline Mining and Extraction – a joint project by TiCC and the IISG to make historical data accessible and comprehensible.

For Van de Camp herself, the collaboration this involves has been inspiring. “Once every two or three months,” she explains, “we all sit down together: people from the IISG and us researchers from TiCC. They let us know what information they are interested in finding, which makes us look at our research from a different angle. And when we show them just what the technology can do, that inspires them anew.”

Culture and Language

14

Nonverbal Communication

A sensitive avatar

Interview

The computer of tomorrow will recognize emotions and non-verbal expression

If we are to communicate with computers in a natural way, it is vital that they be able to recognize and reproduce non-verbal signals. In the near future we will be able to talk to a “sensitive” avatar that knows what a raised eyebrow means.

Marc Swerts and Martijn Goudbeek

but it’s extremely difficult to teach a computer to replicate exactly when and how they do that. For example, raising the eyebrows too often makes the face look neurotic.”

As Swerts succinctly puts it, “Non-verbal expression is not what you say but how you say it. That can be visually – through hand gestures, a frown, posture and so on. But it can also be audible: style of speech, intonation, variations in tempo and loudness… These signals are vital to good communication. Imagine that someone tells you his girlfriend is pregnant. It matters quite a lot how he puts the message across. Is he furious? Happy? Surprised? If you fail to pick up on that, it can leave you in an embarrassing situation even though the meaning of the words is perfectly clear.

Communications between people and computers are no different. Take a call to a travel helpline that uses speechrecognition technology. It can save you a lot of frustration if the computer ‘recognizes’ from your intonation that you are annoyed and responds accordingly.”

To convey a story convincingly, you need to make sure that your body language matches your message. So wrote the great Roman rhetorician Quintilian in his Institutio Oratoria. Good politicians, actors and other ‘performers’ understand the need to master that art: they know that good communication is not just about the spoken word, but also about non-verbal expression. But what exactly is the secret of these masters? “Despite the fact that we’re so familiar with body language,” says Marc Swerts, “we still lack a good deal of explicit knowledge on how it functions in human interactions.” As Professor of Discourse studies, Swerts has been conducting research on non-verbal communication. During a working visit to animation studio Pixar, he once saw at first hand how weeks of work went into creating a single expression on the face of cowboy Woody from the film Toy Story. “That’s how difficult it is. And artistic creation is very different from scientific research. One thing we want to do at TiCC is to create a dataset or corpus from which a computer can both generate various forms of non-verbal communication and recognize them in people. What we are actually doing is coding non-verbal signals as instructions a computer can carry out. There are many potential applications. For example, we could create a synthetic face through which the “It is already relatively easy to identify machine can communicate in a ‘natural’ fashion. But it’s laborious work. Just take clear-cut emotions like anger or joy,” the eyebrows. People use them all the time to add expression to their spoken words, continues Swerts. “But everyday

non-verbal communication is far more subtle than that, and goes beyond emotions alone. When we’re holding a conversation, for instance, how do we know so well when it’s the right moment for us to speak? We do it instinctively, but it’s really difficult to understand why. There are many variables involved: intonation, eye contact...

“And even with emotions, things are much subtler,” adds Martijn Goudbeek, a lecturer who is carrying out research into their production and perception.

“It’s not difficult to recognize the basic

emotions: happiness, anger, fear and disgust. But what about a secondary one, like relief? Or tenderness? Yet they are actually much more common. So if we’re going to compile a good corpus, we’ll have to focus upon those subtle feelings.”

An affectionate computer, is that the goal? “Call it an empathetic computer,” says Swerts. “What we want is that, instead of us having to learn the computer’s language, it adapts itself to us.” Goudbeek proffers an example: “Computergenerated faces or avatars. They’re a much more pleasant means of communication.

In fact, they exist already. There was ‘Clippy’, for instance, the Windows paperclip, although that was really primitive.” “But,” continues Swerts, “imagine an avatar in an educational program for children – one which not only recognizes that the user has difficulty with sums, but can also offer them encouragement in a naturalistic way.” Goudbeek: “An empathetic intelligent system could be useful in other areas, too. Like crowd control, for the early identification of aggression within a group.

Or in product evaluations. At the moment manufacturers use questionnaires to

17 Nonverbal Communication

16

19

Research portrait

Nonverbal Communication

Lisette Mol

measure consumer satisfaction, but they are often quite unsophisticated and respondents are inclined to provide answers they think are socially acceptable. A computer with emotionrecognition capability, on the other hand, could ‘read’ the subject’s face and speech directly.”

Although Quintilian wrote his advice on the subject two millennia ago, scientific research into non-verbal expression is still in its infancy. “Only in the past couple of years,” says Swerts, “have new digital techniques made it possible to manipulate images of facial expressions or recordings of speech with very great precision, and also made it fairly easy to present the results to a large group of assessors.” A slight difference in the position of an eyebrow, higher or lower, in two otherwise identical portraits can have a huge effect upon how the subject’s expression is perceived.

“What we notice time and again,” says Goudbeek, “is how subtle and versatile non-verbal communication is. Take adaptation, for example – the habit of adjusting our behaviour according to who we are with – which is something we have been doing a lot of research on recently. We find that there are personal and cultural factors involved.” Swerts: “A person can laugh at a funeral. It takes a refined form of recognition to interpret that properly. We are not just learning a lot about artificial intelligence, but also about people themselves.”

Hand gestures and computers “When people talk, their hands are usually all over the place, depicting many things.

Are these hand gestures part of their communication? And what do gestures tell us about the way people think? These are central questions in my research.”

“When communicating through gesture, it is of course necessary to see one another. In day-to-day communication, this is often the case. Yet what happens when you cannot see the person you are talking to, whereas they can see you? Or vice versa? Our research shows that people adapt their gesturing to what their interlocutor sees. This shows that some gestures are meant to be seen!”

“Our hands show that we really do understand the message”

“When communicating, people adapt their behavior to one another in many ways. Their gesturing too starts looking more and more alike. Yet what exactly are interlocutors aligning here? Is it their movement, or their way of thinking? We tested this by showing people gestures that almost, but not quite, fitted a particular situation: pointing out a route on a map. In response, they produced gestures that were not exactly like the ones they had seen, but entirely like one would point on a map. This shows that people did not so much copy the movements of their interlocutor, but rather the meaning of these movements. Therefore, we now know that gestures aid mutual understanding.”

“This knowledge of the use and significance of gestures is useful in developing ‘natural’ forms of distance communication, such as videoconferencing.”

Newstweet You can tell whether a person is dominant or not, not only from how loudly they speak, but also from their tone of voice. #TiCC Some emotions can be recognized better in the voice than in the face, and vice versa. This often depends on the evolutionary role of the emotion. #TiCC

Want to know more?

Juslin, P. N., & Scherer, K. R. (2005). Vocal expression of affect. In J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.), The new handbook of methods in nonverbal behavior research (pp.

65-135). New York: Oxford University Press

Expressive Synthetic Speech: a collection of examples of synthetic affective speech conveying an emotion or natural expression

http://www.affective-sciences.org/publication/684

http://emosamples.syntheticspeech.de/

Nonverbal Communication

18

Interview

Fons Maes and Marjolijn Antheunis

Commu n

icating

via com

puter: n

onverba

l inform

ation is

visible.

Social Media

How social media are changing the world

Is a Facebook friend a real friend?

21

Social media have become an integral part of our lives, but we still know almost nothing about their impact upon friendships and “real-life” social contacts. Longterm research at TiCC is set to change that.

Not so long ago, parents worried if their kids spent too much time hanging around on street corners. “Now they’re concerned about the amount of time they spend on Facebook or other social network sites,” says Marjolijn Antheunis, who is investigating the effect of social media upon happiness and the formation of friendships. “And even I don’t know the answer yet. We are on the threshold of a massive social change. A lot of research is needed before we’ll be able to properly assess the impact of the network society. At this stage, though, the scientific studies seem to indicate that parents have no need to worry.”

At first sight, social scientist Antheunis looks rather odd amongst all the cognition experts at TiCC. But times like ours require a multidisciplinary approach, says Professor of Corporate Communications and Digital Media Fons Maes. “Today’s social upheavals are occurring hand in hand with a digital revolution. New technical possibilities in communications are changing social relationships, whilst at the same time those relationships are creating a demand for new technologies.”

Antheunis cites an example: “The way social media are changing the meaning of the term ‘friendship’. We studied whether the number of friends young people have on the web bears any relation to their actual popularity. That turns out not to be the case. But what does that actually mean? Theories about friendship all pertain to the ‘real’ world, and there the number of friends you have is significant. It’s clear that different rules apply online. We have now begun a long-term study asking how social network sites influence young people’s sense of well-being, measured over two years. That has not been studied before.”

For that research, Antheunis is focusing upon adolescents aged 11-14, since that is a period when friendship is essential. It is also when their parents have moral concerns; they are worried about their children’s welfare. But there is something else, too. This generation – now extending to those in their early twenties – is the first not to know what it is like to live without a digital network. The social upheaval is most apparent in this group, with technology and internet use developing accordingly. “For example,” Maes explains, “you can see how social media are supplanting e-mail and the blog. What was innovative yesterday is obsolete today. Yet there is one constant: digital communications are becoming steadily more infused with ‘cues’. There is an ongoing shift from asynchronous to synchronous conversation, as described in media synchronicity theory (MST). In other words, more simultaneous interaction – and with a number of advantages over face-to-face communication:

Social Media

20

Social Media

it is not confined to one place, and there are opportunities to gather more information during the conversation.

“Social network sites are evolving into a ‘café plus’,” continues Antheunis. If you fall into conversation with someone in a café, you receive extra information.

For example, the group of friends they are with says a lot about them. Social media go even further. They provide the dynamics of a ‘real’ conversation plus references to personal websites or holiday photos. And there is plenty more to come.” Maes suggests an example.

“The eyecatcher, a device that makes it possible to look into one another’s eyes when chatting with a webcam.

Recent research by our department has revealed that digital communication now very much resembles its face-to-face equivalent.”

The new social relationships are also going to change marketing. Here lies another unexplored area of research: how does marketing work on social media? That social aspect, after all, makes them particularly well-suited to marketing activities.

As Antheunis explains, “Brands like Apple and Coca-Cola know that it’s beneficial to have consumers attach intangible values to your product, like ‘friendship’ or ‘coolness’. That creates what‘s known as brand loyalty or brand engagement. In partnership with [Dutch networking website] Hyves and others, we are investigating whether social media can play a part in that. For example, how important is the relationship with the sender of the message? Does a strong tie affect the value attached to a product or a campaign? What contribution do opportunities for interaction make? And do consumers mind if companies make use of private information on their personal pages? Say you rave about a band online – do you then want to be presented with ads for its new CD?”

Research portrait

Marie Nilsenova

23

The social signals in human voice “When a company in a crisis situation selects a spokesperson to inform the general public, what should the person sound like? Our study has shown that a low-pitched female voice and a high-pitched male voice are the most appropriate. A low-pitched male voice comes across as too dominant – and is considered unfriendly.”

“My research focuses primarily on auditory prosody, that is, audible non-verbal cues such as intonation, timbre, tempo and volume. The use of these vocal properties has a number of social functions. For example, people adapt their voice to each other to show that they belong together or that they want to cooperate with each other.

Vocal behavior also enables us to deduce the social hierarchy of participants in a conversation and how the speakers are feeling.

“Increasingly,” concludes Maes, “research into communications mediated by digital networks is focusing upon the subtleties involved. Little things, which are also of interest to the marketing industry. You can investigate whether such communications address the big issues, too, the things you discuss with friends or loved ones in real life. Love and death, good and evil. But that’s pretty crude.

The research at TiCC is far more refined. Because of that, it lays bare things that are much more common in everyday life. Subtle emotions, gestures, eye contact…

Therein lie the secrets of communication.”

Another line of my investigation involves individual differences in the perception of speech. There is a significant individual variation in how people perceive complex sounds such as speech or music. Due to discrepancies in our neuro-cognitive architecture, we may respond differently to the same musical piece and give preference to different musical instruments. Both the left and the right hemisphere process auditory signals, but the way in which they are involved varies from one person to another. A greater right-hemisphere dominance in pitch processing leads us to focus more on timbre, whereas a left-hemisphere dominance results in attention to basic melody and rhythm. Studies have shown that listeners of the first type are better able to recognize other people’s emotions.”

“And that’s interesting for me as a social scientist,” says Antheunis. “What are the implications of this kind of more naturalistic communication? It is known, for instance, that people who meet online tend to like one another more than when they meet offline. But will that still apply when they can make eye contact?” “Or,” asks Maes, “if computers can provide additional information such as details of the other person’s mood?

From the research carried out at TiCC, that’s not unthinkable.”

Newstweet Want to know more?

Antheunis, M.L., (2010). Getting Acquainted through Social Networking Sites: Testing a Model of Online Uncertainty Reduction and Social Attraction. Computers in Human Behavior, 26, 100-109. http://www.sciencedirect.com/science

search for author: Antheunis

The number of friends on social network sites is no indication of popularity. #TiCC

People get to like each other more quickly online than offline. #TiCC

Social Media

22

Interview

Human Computer Interaction

Emiel Krahmer and Suleman Shahid

25

Computers that are easy to use, help improve our quality of life and can understand the subtleties of human communications – that is the way we need to go.

What do we want from a computer? Emiel Krahmer can sum it up: “A machine that adapts to us, not one that we have to adapt to. The ‘Holy Grail’ is a computer that understands our language, rather than us having to learn computer speak.”

Suleman Shahid has even higher expectations. “We want a computer to be valuable not just in enabling interpersonal communication, but valuable to society in its own right. For example, intelligent computer systems could help the elderly and disabled to live better lives or support all of us in leading a more sustainable existence.”

properly and healthy? “In the near future,” he says. “this so-called user interface design is going to become an increasingly common part of our everyday lives. But to do that it has to satisfy a number of preconditions. The system must be fast and easy to use, and not too prominent. It should sense the user and their needs, as it were.”

And therein lies the problem, continues Krahmer. “Take automatic language generation, for example. In other words, technologies that translate information into speech. They certainly make the computer user-friendlier, as in the

Krahmer and Shahid are researchers working on human-computer interaction, a discipline dedicated to creating a friendly, helpful machine that communicates with us in as “natural” a way as possible. A computer with human traits, in other words, and one capable of performing all kinds of useful tasks for us. One possible solution to that challenge lies in the further development of avatars, computerized characters with whom you can communicate in the same way as with a real person.

According to Krahmer, they represent a promising line of investigation from a technical point of view. “Computing power has increased so enormously in the past few years that it is now becoming theoretically possible for machines to understand and imitate the finer points of human communication. The problem is this: what exactly are those points? And how do we explain them to a computer?”

Able to understanding the finer points of language, and easy to use too

The computer must adapt to us, not us to it

In fact, two separate approaches are required. The first is for us to actually understand all the secrets and subtleties of human communication. That is Krahmer’s specialization. “It’s called psycholinguistics,” he explains. “A combination of psychology and communication science. The research focuses upon how people produce and comprehend language, both verbal and non-verbal.”

Meanwhile, Shahid’s work is homing in on the computer itself. At its heart is the creation of user-friendly hardware and software. For example, how can an application remind a forgetful elderly person to switch out a light, say, or to eat

case of a device that gives you spoken directions as you walk through a city. But such a system is very difficult to develop.

It requires a lot of research, especially if you want it to respond naturally to what the user sees and says. Let me give you an example. Every day we describe things almost without thinking about them. If someone asks us the way, for instance, we might say, ‘Turn right at the building on the corner’. But how do we get the computer to reproduce that kind of instruction in a natural fashion?

One way, you would think, is to teach it to give an exact description. One with no superfluous information, but just the facts needed to distinguish that one building from all the others in the vicinity. In practice, though, people tend

Human Computer Interaction

24

27

Human Computer Interaction

User Interface Design for a better life

to provide far more detail than is strictly necessary. We don’t actually say ‘the building on the corner’, but ‘the old house on the corner, with the big windows’. In other words, we overspecify. And that’s something it’s difficult to teach a computer to replicate.

The Bachelor’s course in User Interface Design (UID), taught by Suleman Shahid, gives students an introduction to the underlying theory of interface design, and provides them with the skills to put the theory into practice. During this course, students develop apps designed to make everyday life easier and more enjoyable.

Here are two examples.

“Another issue, as highlighted in research we completed only very recently, is paraphrasing. That’s the human ability to express exactly the same thing in quite different ways. Compare these two sentences: ‘Turn right at the church and walk down to the bridge.’ And: ‘When you get to St Mary’s, take a right and then carry straight on until you reach the river.’ Both give exactly the same directions, but using totally different terminology.

We’ve just released a dataset containing more than two million words, to help automatically identify and generate paraphrases.

This is the only dataset of its kind, and should help to produce more varied and more human descriptions of routes.”

“Research shows that people are very apt to ascribe computers with all kinds of human qualities,” continues Shahid. “A machine may not respond in the same way as a person, but often we rather expect it to. That’s why research into a so-called ‘user-centred’ intelligent system is so important.

Moreover, I’m convinced that this work is going to help make the world a better place. It’s partly for this reason that it fits so well into the activities of a school of Humanities: it’s showing how computers can play their part in putting human values into effect. We see a good example of this in the applications students have to develop for final assessment of the undergraduate course in User Interface Design. These must either support a sustainable lifestyle, take over simple care tasks, offer instructive entertainment or provide personal assistance. Results have included an app to measure domestic energy consumption, an electronic diary for autistic children and a set of games to make kids more environmentally aware. They’re all small things, but they show how technology can contribute to a better world.”

The Augenda

The Green Life Guide

The Augenda is a digital diary for young people with autism, which was developed by five students. The prototype can help these young people to structure their lives by giving them checklists and appropriate reminders. It also uses emoticons because previous research has shown that those with autism respond much more easily to visual stimuli. Parents and carers can also be given feedback. The students were invited to present their project at the renowned Assets conference in Orlando.

The Green Life Guide, which was also developed by five students, is a smartphone application which helps people to have a more sustainable lifestyle. The application has four green functions: location-based services, cooking, news, and transport. Users receive immediate feedback on the number of trees that have been saved by making use of more sustainable services.

“And even if the trend towards making computers more human were to stall tomorrow,” adds Krahmer, “then it would still all have been worthwhile. After all, building them is perhaps the best

way of understanding people better.”

Want to know more?

Project Bridging the gap between psycholinguistics and computational linguistics: the case of referring expressions. http://bridging.uvt.nl/

Newstweet Why do I have to learn how a computer works?

Why can’t it learn how I work? #TiCC

Human Computer Interaction

26