Interaction Technology

Interaction Technology

Interaction Technology

12 stories, 12 perspectives If you’re wondering just what the University of Twente’s Master’s programme in Interaction Technology – I-Tech for short – is all about, this booklet will help you find real answers. It offers 12 stories in which students and staff describe current or recent I-Tech projects. Colourful and refreshingly personal, these stories will open your eyes to the ins and outs of I-Tech, the disciplines it involves, the type of people it attracts and the exciting results it can lead to.

INTERACTION TECHNOLOGY FROM R3D3 TO THINKING PALLETS For example, meet Daan and Max, who are helping a robot called R3D3 improve its communication skills. Or Janwillem, who hand-built the world’s first giant drawbot to create a Mona Lisa reproduction. David created a ground-breaking Virtual Reality experience to help the car industry explore how humans will interact with autonomous vehicles. Emiel brought virtual one step closer to reality.

You’re concerned I-Tech is a man’s world? Silke devised an interactive storytelling system for kids. Carmen is developing a game that will help us understand the feelings of dementia patients. Marlène launched a startup to save businesses and jobs.

More interested in controls? Kees is equipping telerobots to do complex work in hazardous conditions. Data analytics? Monique is using crowdsensing to study student stress. Logistics? Thijs is busy developing thinking pallets. Smart environments? Tobias has pushed this field forward with an innovative design and prototyping system, while Jelle is building a playground that can self-adapt to players’ behaviour.

Find out more about Interaction Technology and other Master’s programmes at the University of Twente on our website at utwente.nl/en/education/master/.

4 | I-Tech

SIX I-TECH CHARACTERISTICS If you’re trying to catch the thread connecting these stories, here are six pointers: R Working on systems that understand people and can adapt to their requirements R Exploring new research areas where tech meets human behaviour and impacts society R In-depth technology courses with a broad, cross-disciplinary outlook R Project teams that bring together different disciplines, countries and cultures R Opportunities abroad, for instance with the Human Computer Interaction and Design double degree R One of the few engineering programmes to include entrepreneurship and design COULD YOUR STORY BE NEXT? Interaction Technology is a two-year, English-taught Master of Science degree that evolved from our Human Media Interaction (HMI) programme. If you’re as excited about computer science or electrical engineering as you are about analysing human behaviour, and you’re eager to work on some of the world’s smartest and best interactive technologies while also engaging with customers and end users – you could soon be adding your story to this colourful collection. We look forward to hearing from you!

Alma Schaafstal, Programme Director Dirk Heylen, Professor of Socially Intelligent Computing

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 5

STRESS BUSTER ‘STRESS AFFECTS US ALL. IT WOULD BE GREAT IF MY RESEARCH COULD HELP US DEVELOP EFFECTIVE SOLUTIONS’ MONIQUE TE RIETSTAP 6 | I-Tech

MONIQUE TE RIETSTAP IS USING SMART PHONES TO EXPLORE AND EXPLAIN STUDENT STRESS LEVELS In today’s restless age, stress is a major problem, sometimes even a health threat.

Monique te Rietstap believes data generated by smart phones can help her track down stress factors – and make life a little easier for us all. Her target group: stressed-out students on the University of Twente campus.

‘Stress has been researched extensively, with lab tests focussing on physiological factors, and questionnaires covering people’s personal perceptions. What’s new in my research is that we’re using mobile phones to collect data,’ says Monique, who hopes the project will get her a Master’s degree in Biomedical Engineering by the end of 2017.

Assistant Professor of Creative Technology and Senior Telemedicine Researcher Oresti Baños explains that Monique’s work is part of a broader project called HoliBehave: Holistic Behaviour Inference based on Mobile Sensing. ‘Our hypothesis is that mobile sensing, or crowdsensing, has huge potential for charting and explaining behaviour and promoting healthier behaviour and lifestyles. Mobile phones are everywhere and we use them to share a lot of data daily, so they represent a huge potential source of behavioural data.

We think we can use them to detect physical, social, emotional and cognitive behavioural patterns. The advantages of this research over lab tests or interviews are that mobile sensing is unobtrusive, objective and continuous.’

THE FIRST EXPERIMENT Monique’s first experiment will involve 20 to 30 Bachelor’s students engaged in different UT programmes. These volunteers will be requested to download an app called Aware (awareframework.com). The app will use their smartphone sensors and additional software plugins to anonymously collect data on factors such as physical activities, phone and battery power usage, and use of text messaging, apps and social media. Monique will also track participants’ locations and scan their surroundings for Bluetooth-enabled devices to get an idea of how many people they are with. ‘By comparing the data with university schedules I hope to develop a model that will help us understand student behaviour and stress levels.’

IMPACTING UNIVERSITY PLANNING Oresti believes Monique’s work could seriously impact how courses, assignments, deadlines, projects and exams are planned at the university. ‘Ideally, we will identify root causes of stress and expose the influence of scheduling. Because mobile sensing is an emerging field, we’d also be very pleased if the research teaches us how best to organize data collection campaigns and how to develop the analytics. The longer-term potential of this technology is huge, especially in the medical domain. Think, for example, of researching and combatting diseases or lifestyle issues, such as obesity. It’s an exciting development.’

GRADUATION STRESS As her graduation project unfolds, Monique certainly expects to see her own stress levels rising. ‘Stress affects us all,’ she muses. ‘It would be great if my research could help us to understand it better and to develop effective solutions.’

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 7

SILKE TER STAL’S INTERACTIVE CONCEPT AIMS TO HELP GET TECHNOLOGY INTO THE CLASSROOM

STORYTELLING ROBOT Not long from now, smart toys will be a regular classroom feature. Silke ter Stal’s prototype of an interactive storytelling system, involving a robot and a tablet application, has brought us one step closer to a future in which technology will play a fun and vital role in educating kids.

‘My passion is combining technology and design in a way that will get right inside the minds of users,’ Silke says. The prototype she built was part of the thesis that earned her a Master’s degree in Human Media Interaction* in 2017. Providing kids aged 6 to 10 with a lot of fun, the system also challenges and enables them to develop their creativity. It consists of a robot, Cozmo – available in shops – and a self-built tablet application. Cozmo is the lead character. The action begins when Cozmo asks you to help him go to the moon, which you can do in the app by selecting story actions, such as going somewhere, moving an object or meeting another character. You then have to explain – in typing – why that event matters. Cozmo plays out the actions on a play mat with marked locations. Silke fitted his Playmobil co-actors and the various props and locations with markers Cozmo can identify. Emoticons on his display show when he is angry, happy, sad, afraid or surprised.

Kids then have to explain why he feels that way. Meanwhile, a storyline on the tablet stays in view to keep the kids aware of how the story is progressing.

GETTING KIDS TO REFLECT ‘Silke found a clever way of structuring the creative process and getting kids to reflect on their choices,’ says PostDoc researcher Alejandro Catala Bolos. ‘That’s what makes her system stand out from existing robot-app combinations.’ Says Silke, ‘Kids tend to just play around with a robot. I had to figure out how to get them to work on developing a story line. There’s hardly any literature on this yet. I learned a lot about user-centric design methods and interaction with artificial characters.’

CLASSROOM TECHNOLOGY Silke worked under the flag of the CoBOTnity Project, which aims to deliver affordable robots and virtual agents that can be used to foster creative thinking and social awareness in children. Explains Alejandro, ‘Silke’s thesis focussed on storytelling, because it is a specific creative skill with a social context and a very rich learning potential. The prototype she built and the insights she gained represent important steps forward. As technologies mature, smart toys like this will become an important part of education – not as a replacement of teachers, but as flexible tools enabling teachers to challenge kids in new ways. Part of Silke’s achievement is that she got teachers to come on board.’ Adds Silke, ‘The use of robots in class is very incidental so far. CoBOTnity’s aim is to help them embrace robots and virtual agents as co-designers of classroom activities.’

8 | I-Tech

‘IN DEVELOPING TECHNOLOGY, I LIKE GETTING AS CLOSE TO THE USER AS I CAN’ SILKE TER STAL

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 9

‘I’M A COLLECTOR OF SKILLS. THAT’S MY HOBBY.’ JANWILLEM TE VOORTWIS

10 | I-Tech

JANWILLEM’S GRADUATION PIECE, A GIANT DRAWBOT, LIVENS UP A FESTIVAL AND SPARKS SOCIAL DEBATE

Picture a 15-foot tall square tower used for posting large posters or artwork. One of its sides is boarded up and plastered with a large canvas. From the top hangs a rudimentary V-plotter built with two black steering wheels that look as they’ve been stripped from an ancient farm vehicle, a few clunky bicycle chains and a primitive looking printer head the size of a small cartwheel, fitted with spray paint cans in the colours of the rainbow. Keep watching as the plotter lurches into motion and the bike chains jerk the print head into position over the canvas. The head whirrs round, stops. Somehow a 3D-printed lever automatically releases the pressure on the spray mechanism of one the paint cans and – pssssst! – the first dab of brownish paint spills onto the canvas. While a few blobs of shiny brown fluid drip downward, the plotter twitches sideways and the next spray can is jolted into place to do its job. Keep watching for about two hours and before your eyes, wondrously, will appear the world’s first giant drawbot Mona Lisa reproduction. You’re also looking at the graduation project of Creative Technology Bachelor’s student Janwillem te Voortwis.

TECH MEETS ART Leonardo da Vinci’s famous Mona Lisa portrait has been copied and covered in countless ways, but never – until 2016 – by a giant drawbot built with a do-ityourself V-plotter, a pair of bicycle chains and some spray cans from the local paint shop. When tech meets art, everything is possible.

RAISING QUESTIONS ‘I guess it’s a mixture of technology, art and performance,’ Janwillem muses. ‘We could have tried to build the perfect drawbot, but the imperfections and clumsiness make it art.’

His supervisor, Assistant Professor of Robotics and Mechatronics and part-time tech artist, Edwin Dertien, comments, ‘The idea originated when a festival organizer asked us to come up with a way of decorating festival towers. I turned that into a thesis assignment.

Janwillem agreed to make something big that would raise questions and spark conversation about the place of robots in society by exposing the nuts and bolts behind the technology. Will robots take over creative jobs as well as mundane ones? Could a robot produce art? At the festival where it was displayed, it did in fact make people laugh, wonder, join the conversation. Which is one thing good art – like good science – is supposed to do.’

A COLLECTOR OF SKILLS The drawbot is just one of Janwillem’s many surprising handiworks involving electronics, wood, metal, data science, performing arts, music. Says Janwillem, ‘Some people collect certain objects. I collect skills. It’s my hobby to get as good as I can in as many possible disciplines.’

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 11

SPARKING DEBATE, BOOSTING BUSINESSES ‘THE UNIVERSITY HAS REALLY ENABLED ME TO COMBINE GRADUATING WITH BUILDING MY OWN FIRM’ MARLÈNE HOL 12 | I-Tech

Marlène Hol does credit to the University of Twente’s core value of entrepreneurship. Having launched a business in 2015, she is now devoting her thesis to improving the company’s software usability. In 2018 she hopes to contract the first paying customers – and earn a Master’s degree in Computer Science.

MARLÈNE HOL IS USING HER MASTERS’ THESIS TO DEVELOP HER COMPANY AND HELP OTHER ENTREPRENEURS

Ask Marlène how she’s managing to follow an intensive Master’s programme while running a business on the side, and she’ll be quick to correct you. ‘It’s actually the reverse of that,’ she says wryly. ‘I’m running a business and studying on the side.’ Far from frowning on this academic neglect, Associate Professor Maurice van Keulen expresses unconditional support. ‘We do all we can to tailor our education to the needs and potential of individual students. It’s what we’re here for. An international review committee gave our Computer Science research the highest praise in 2016, partly for its number of successful spinoff companies.’ Adds Marlène, ‘The university does so much to encourage, activate and challenge you in your pursuits. In my case, they’ve been flexible with deadlines and schedules, given me access to facilities – friends of mine at other universities are totally amazed.’

HELPING SMALL BUSINESSES It does help, of course, that Marlène’s company, Mobina (mobina-it.com), and her graduation research are intertwined. ‘The goal of Mobina is to help small to medium-sized enterprises in the manufacturing industry to achieve operational excellence and to prepare themselves for the digital future. My research is centred on analysing anonymous data derived from our web application in order to improve the usability of our software.’

The application offers extensive knowledge on innovations, organizational change and information systems. Subscribing businesses can access the content and, more importantly, discuss it online with team members, colleagues or other subscribers.

Comments Maurice, ‘It’s all about using data science technology to encourage and improve interaction between professionals from different layers of an organization.’ The result is a living knowledge base, where manufacturers can get real answers to real questions, says Marlène. ‘In my research I’m using process mining to find out how users use the software, navigate the information landscape and interact with each other,’ says Marlène. ‘What I hope is that my research will help us see how we can fine-tune Mobina’s services for optimal usability and customer results.’

COMPANIES, JOBS, FAMILIES Behind Marlène’s high-tech skills is a strong personal drive. ‘I enjoy the Computer Science element of Interaction Technology, because it’s always new and surprising. Using that knowledge to make manufacturing companies futureproof, and to help them retain jobs for the people and families that rely on them, is very satisfying.’

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 13

‘IT’S EXCITING TO CREATE AN IMMERSIVE EXPERIENCE AND SEE HOW PEOPLE REACT TO IT’ DAVID GOEDICKE

14 | I-Tech

DAVID GOEDICKE USES VIRTUAL REALITY TO STUDY HOW WE WILL INTERACT WITH AUTONOMOUS VEHICLES

FEEL THE WHEELS As a teenager, David Goedicke planned to study music composition. ‘I wanted to do art that would interact with people.’ Then he stumbled on Creative Technology. He loved it – and is now completing his Master’s in Human Media Interaction*.

Next stop: a PhD programme at Cornell Tech, New York.

David’s thesis hinges on the same desire ‘to craft a story that will trigger a response’, as he puts it – only now he’s doing it for the car industry. ‘The idea is to use Virtual Reality for evaluating how human drivers will interact with autonomous vehicles,’ he explains. ‘It’s a new research method. My goal was to use VR for building a perfectly safe, highly immersive environment. It involved measuring how the car moves and feeding that into a game engine to copy the motion of the physical car to the virtual car the participant is sitting in. The goal is to make the participant feel he or she is really sitting in an autonomous vehicle. So, the houses and the traffic have to look real – but the really essential thing is to make sure what they see matches what they feel through so-called motion matching. For example, if another car rushes by, you want the test subject to experience that sensation of danger – otherwise you won’t get the gut reaction you need to make a reliable study of human behaviour in this situation.’

BETTER AND CHEAPER David built the technology at Stanford University’s Centre for Design Research in the United States, then returned to Twente for his validation studies. His supervisor, Vanessa Evers, Professor of Human Media Interaction, points out that the system David built is ‘more realistic and immersive, yet cheaper’ than the big simulators currently available on the market. ‘When his publication comes out, it is sure to be picked up and developed further.’

The broader backdrop, she says, is that ‘we need to understand how autonomous systems are going to work in the real world. The car industry is an obvious place to begin, because it has access to funding and a short time to market. David has essentially created a new platform for this kind of research. Demand for Creative and Interaction Technology experts like him, who are trained to operate on the cutting edge of technology, creativity and user experience, is on the rise.’

PREMIERE NIGHT To David, the graduation project experience is reminiscent of a part-time job he had. ‘I was on the technical support staff at a theatre and I came to love the thrill of premiere nights when you see and feel the audience’s emotion in response to a play you helped create.

My thesis work was really the same kind of thing, creating an experience and seeing how people react to it. I like making things.’

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 15

An intelligent playground uses cameras, lighting, tracking sensors, audio and other components to invite and stimulate children or other users to move and play together.

Master’s student Jelle is taking the concept to a higher level by making smart playgrounds adaptive. ‘An adaptive system can come up with its own changes or interventions on the basis of what is happening in its surroundings,’ he explains. ‘An adaptive playground, for example, could respond to users’ behaviour by deciding to change the pace, start a new game, or make things easier or more difficult.’

ENGINEER OF FUN JELLE PINGEN’S ADAPTIVE, INTELLIGENT PLAYGROUND WILL KNOW WHAT KEEPS KIDS GOING Intelligent playgrounds began as a gimmick for amusing museum or trade fair visitors. In five or 10 years, almost every school, gym, physical therapy centre or waiting lounge could have one – with a suite of games capable of self-adapting to players needs and wants. Jelle Pingen is one of the people making it happen.

MEASURING KIDS’ ENGAGEMENT Jelle’s focus is on using adaptivity to influence the level of engagement of children playing a game of tag. ‘In most games or exercises, players lose interest after a while. My goal is to figure out how to measure engagement and to design interventions that will reactivate them.’

Robby van Delden, who teaches in the Creative Technology Bachelor’s programme as well as the Interaction Technology Master’s, says Jelle’s work will help to make intelligent playgrounds commercially more interesting. It will also open up avenues for other, more serious applications. ‘Think, for example, of sports and fitness exercises, revalidation, or learning environments. As we develop more games, more functionalities and more interventions, the concept will really spread.’ Adds Dennis Reidsma, another UT researcher and lecturer in this area, ‘Any domain involving care, play or learning will hugely benefit from the next generation of adaptive intelligent playgrounds.’

USING YOUR INTUITION Using the facilities installed at the University of Twente’s DesignLab playground as well as wrist-worn fitness trackers, Jelle can measure game players’ heart rates, positioning, speed of movement and sound. ‘Those are good indications of engagement levels, but I have to analyse the data to figure out exactly where the levels drop.’ The next steps are to build an algorithm that will enable the system to respond and to design effective interventions. ‘Building the technology is not the biggest challenge here,’ he says. ‘The tough part is combining the research and data with your creative intuition to implement interventions that actually work. I’m thinking of using power-ups or achievements, or adding new games.’

FUN AND WONDER The fun of it all, to Jelle, is using technology to let people experience fun and wonder – even if they have no idea about the technology behind it. ‘My dream job? That would be to work on the technology and design of really cool theme park attractions.’

16 | I-Tech

‘I REALLY ENJOY USING TECHNOLOGY TO GIVE PEOPLE AN EXPERIENCE OF WONDER AND AMAZEMENT’ JELLE PINGEN

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 17

KEES VAN TEEFFELEN’S THESIS WILL HELP EQUIP ROBOTS TO DO COMPLEX WORK IN HIGH-RISK CONDITIONS

Kees has a Bachelor’s degree in Mechanical Engineering, then switched to a Master’s in Systems and Control. ‘During the Bachelor’s I realized the field I enjoy most is control,’ he says. A highly multidisciplinary programme, Systems and Control teaches students all about the control of different kinds of technical systems, combining in-depth mathematics with knowledge of hardware and mechatronics. It offers students like Kees access to the University of Twente’s world-class facilities, such as the Robotics lab.

ROBOT BOOSTER Robots are muscling up – and not just for sci-fi movies and bot fighting arenas. In the real world, they are increasingly taking on tough jobs in hazardous conditions: deep sea and deep space, military action, medical and industrial tasks. Kees van Teeffelen is one of their enhancers.

REMOTELY CONTROLLED ROBOTS Robotics are Kees’ current focus. With his graduation thesis he hopes to contribute to scientific and technical advances in telerobotics and exoskeletons. ‘Telerobotics is all about using remotely controlled robots,’ he explains, ‘for example, to do jobs that are too dangerous for people, or to go places people can’t easily go. I’m researching the problem of time delay: if the distance between the robot and the operator is too big, there is a time delay in the transmission of the data between the two that affects the robot’s controllability and performance. For example, if your robot is about to bump into something and you don’t find out until a few seconds later, you could be too late.

Specifically, I’m developing a force feedback concept. Force feedback allows you to feel – and not just see – what your robot arm is doing. In my research, I’m adding measurements of the operator’s muscle activation to improve performance and safety. It will give operators more control over the robot arm’s behaviour and help compensate for the time lag.’

I-BOTICS INNOVATION CENTRE Kees is working under the umbrella of i-Botics, an innovation centre founded by the UT and research organization TNO. Its aim is to develop practical robotic innovations and applications with a relative short time-to-market period. Explains Douwer Dresscher, who is Kees’ supervisor as well as a Creative Technology teacher and researcher, ‘With i-Botics we want to achieve more comprehensive coverage of the entire robot development chain, especially bridging the gap between UT’s more fundamental research and TNO’s delivery of ready-to-go industrial solutions. The two main research lines are exoskeletons and telerobotics, especially for inspection and maintenance applications. Time delays and stability are two key problem areas in this field.’

18 | I-Tech

ALL THOSE COOL TOYS A part-time intern at TNO, Kees is enjoying the learning curve. ‘I had very little experience with controlling hardware before I started on my thesis. Also, a lot of components and disciplines come together in this field. The biggest challenge? Putting down all those cool toys I get to play with and getting on with the science.’

‘MY BIGGEST CHALLENGE? PUTTING DOWN ALL THOSE COOL TOYS AND GETTING ON WITH THE SCIENCE’ KEES VAN TEEFFELEN

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 19

THINKING PALLETS ‘THIS IS A FUN WAY OF EXPLORING NEW FIELDS LIKE THE INTERNET OF THINGS AND WIRELESS SENSOR SYSTEMS’ THIJS DE HAAN

20 | I-Tech

THIJS DE HAAN IS STREAMLINING SUPPLY CHAIN LOGISTICS If you want to work in a field where lots of things can go wrong, try logistics. Lost or damaged goods, late deliveries, weight and cost miscalculations, unsafe storage – you name it, it happens daily. Thijs de Haan is developing smart pallets he hopes will help make logistical mess-ups a thing of the past.

Thijs’ work is part of Project Countdown, a three-year research programme of the University of Twente and several business partners. The project, which began in March, 2017, aims at optimizing supply chains. Explains Associate Professor Nirvana Meratnia, ‘By using innovative data collection and ICT technology, we want to provide businesses involved in delivering goods with an end-to-end overview – from factory to customer – of all of the logistical processes involved. The goal is to make supply chains safer, more efficient and more sustainable. The Netherlands is one of the global leaders in logistics.

We believe this programme’s unique and comprehensive reach can push the sector further ahead.’

A NIGHTMARE Getting a product from A to B may sound pretty straightforward, but the reality of it can be a nightmare. Says Nirvana, ‘Supply chain logistics involve a huge array of products and services, companies, people, trucks, containers, pallets, warehouses, distribution centres and more. A lot can and does go wrong.’ Adds Thijs, his eyes alight, ‘A typical distribution centre or trucking company serves many different customers. A pallet used for getting fresh flowers to the florist today may be used for a hazardous substance tomorrow.

Different companies have different procedures, systems and software. Mapping all of that is a challenge in itself – never mind harmonizing it into one, perfectly streamlined process.’

THINKING PALLETS In all this delightful complexity, Thijs has found his Master’s thesis subject. ‘Developing intelligent, communicative pallets is my main focus. I want to enable pallets not only to know what they carry, to know where they are located and being able to be traced. I also want them to think for themselves so they can monitor wrong behaviour and send out warning signals. This will create an extra layer of distributed control over the logistic process. A lot of waste and losses in logistics result from the domino effect: if one thing goes wrong, everything following it does, too. Early, effective interventions can have a huge impact.’

Adds Nirvana, ‘By the end of Project Countdown we hope to have an end-to-end pallet localization system suitable for every supply chain; the tools and dashboard instruments for controlling the logistical process; and a number of use cases, for example for Thijs’ smart pallets.’

EXPLORING NEW FIELDS Delving into sensor and wireless technology, data analytics, planning, business logistics, localization techniques and electronics, while also communicating with managers, operators and truck drivers is the kind of cocktail Thijs likes best. ‘Fields like the Internet of Things and wireless sensor systems are new. And this is a fun and exciting way of exploring them.’

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 21

‘I’M QUITE AMBITIOUS. NARROWING DOWN IS ALWAYS A CHALLENGE.’ TOBIAS UEBBING

22 | I-Tech

TOBIAS UEBBING HAS MADE THE PROTOTYPING OF USER-FRIENDLY SMART ENVIRONMENTS A LOT EASIER

Tobias wants to make technology more accessible to us all. In his graduation project for the Master’s programme Human Media Interaction*, he explored how users experience and interact with smart environments. ‘Environments and applications that can adapt to user needs and wants have huge potential,’ he says. ‘Smart offices can help us work more efficiently and comfortably. Smart learning environments can help kids have more fun and learn more. Smart homes can add to the quality of life. Unfortunately, a lot of products out there today are not yet user-friendly enough to be easily picked up by consumers.’ In his research, Tobias found out why – and proposed a solution.

USER’S BEST FRIEND As the Internet of Things takes shape all around us, we will be seeing more smart offices, smart learning environments, smart homes. Tobias Uebbing found out why many smart environment solutions are still too geeky to be picked up by consumers – and made the prototyping process a lot easier.

NO TIME FOR USER TESTING The main problem, Tobias explains, lies in the design and prototyping processes commonly used to develop smart environment solutions. ‘Smart environments are complex. And prototypes have to have high-fidelity: they have to be very close to the real thing, or you won’t be able to examine usability questions. This is a huge challenge for researchers and developers, because of the time constraints they always face. My research shows that most of the available time goes into the technical development, with little or no time left over for user experience research. The result: geeky products and slow uptake by consumers.’

Tobias came up with an alternative he calls Functional Experience Prototyping. ‘The main difference is that in my approach technical and user experience research are integrated by means of an iterative process: you develop an idea, realize it in simple form, test it on users, go back and develop it further, test it again, and so on. It means you can launch a more mature product in less time.’

QUITE AMBITIOUS Tobias even managed to create a list of components that can make up a standard toolbox for Functional Experience Prototyping. ‘I wanted to create the toolbox itself, but there was not enough time,’ he says. ‘Narrowing down is always a challenge. I’m quite ambitious.’

Angelika Mader, Tobias’ supervisor and Assistant Professor at the University of Twente’s Human Media Interaction research group, calls that an understatement. ‘Realizing the actual toolbox would have been worthy of a 4-year PhD project. Tobias has delivered half a PhD in about six months. It is an important contribution to the industry.’ In other words, if smart environments start popping up faster than you expected, you’ll know who’s responsible.

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 23

‘WORKING IN A MULTIDISCIPLINARY TEAM ADDS SO MUCH TO THE WHOLE LEARNING EXPERIENCE’ CARMEN BURGHARDT

24 | I-Tech

CARMEN BURGHARDT’S GAME TRIGGERS CONVERSATION AND REMINISCING WITH DEMENTIA PATIENTS

REMEMBER, REMEMBER On visits to her grandmother, who suffers from dementia, Carmen Burghardt has noticed that some visitors and carers feel awkward around dementia patients. Now she is designing a game that will make things easier for both sides, while also providing insights into how dementia patients express emotion.

‘We’re developing a gamified solution aimed at fostering conversation and memory recall for dementia patients,’ explains Carmen on behalf of her project team of first-year Master’s students in Human Media Interaction*. ‘It’s a board game combined with an app. Visitors or caregivers will be able to play it with patients. The idea is that you get to ask each other questions in different categories, such as Hobby, Family, Teenage Years, or Love – for example, ‘Share a happy memory of father’. It will liven up interaction, while also evoking memories. And it’s competitive, too.’

FIRST, THE DATA Khiet Truong, Assistant Professor of Human Media Interaction and Carmen’s supervisor, says the original project assignment was ‘to come up with some form of multisensory stimulation that would evoke memories in dementia patients and enable them to relive those memories as fully as possible.’ The project is part of a broader 4-year research programme launched in 2017, with the aim of exploring how these patients give facial and vocal expression to emotions. Says Khiet, ‘Very little is known about this. The first step is to develop a method for collecting data; that’s where the game comes in. Once we have that as an accepted working method, we can start to collect data, for instance by video recording game sessions with elderly dementia patients. Then we want to develop software that can recognize and analyse those emotions. Current software is not capable of doing that.’

VERY DIFFERENT BACKGROUNDS Carmen’s team mates have very different academic and cultural backgrounds. ‘I’ve learned as much from being on a multidisciplinary team like this as from the project itself.

For example, our first brainstorm was prepared by a team member with a background in Business & IT who did a minor in ideation in Denmark. It was a full day of interaction, creative mind-setting, games – very professional. Then we applied rapid prototyping, which means you draw up an easy prototype, test it with users, go back to develop more, then test again. It leads to really user-centric solutions in a short time. ’

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 25

Emiel built the system during his graduation project for the Human Media Interaction* Master’s programme and has called it openIMPRESS (for open source IMmersive PRESence System). ‘With openIMPRESS two people who are in different places can be together virtually,’ he says. ‘You could use it for long-distance visits with family and friends, or for offering or receiving remote assistance, say if your car has broken down.

Microsoft has demonstrated a similar system, but it’s not available publicly – I made mine with consumer-grade components you can buy anywhere.’

VIRTUAL VISIT

EMIEL HARMSEN’S VIRTUAL AND AUGMENTED REALITY SYSTEM COMES PRETTY CLOSE TO TELEPORTATION From Star Trek to Harry Potter, almost every self-respecting sci-fi or fantasy story uses it: that magical fiction called teleportation. Emiel Harmsen, who spent part of his graduation period in San Francisco, has built a Virtual and Augmented Reality system that brings it one step closer to reality.

26 | I-Tech

SITTING TOGETHER VIRTUALLY To use the system, you put on a pair of VR glasses, while your faraway friend wears AR (Augmented Reality) glasses. Using gaming software, Emiel’s system generates a 3D model of your friend’s environment and streams the data to your VR set. Open your eyes and – presto – you’re right there with your friend at his place. You can talk, admire his surroundings, or virtually sit down next to him. Using the AR glasses, your friend sees a hologram of you coming over and seating yourself beside him. ‘Actually, at this point, I can only give the wearer of the AR glasses a hologram of your hands and head, thanks to trackers on the VR set,’ says Emiel. ‘Your friend will see a pair of virtual hands moving around in his environment. So you could virtually give him a high five. Or if you were a car mechanic and his car was broken, you could give him very precise instructions on what to do.’

A FASCINATING EXPERIENCE Professor of Socially Intelligent Computing Dirk Heylen is one of several people who have tried Emiel’s system. ‘It was a fascinating experience,’ he says. ‘It really is like teleportation. You’re in two different places, but you come together in a shared virtual space. And this time you’re not represented by an avatar, like you are in applications such as Second Life, but it’s a realistic representation.’

Dirk’s main interest in openIMPRESS is that it opens up a rich potential of research possibilities. ‘It provides a means of exchanging and manipulating all kinds of data, from audio and video to motion sensing. That makes it an entirely new framework for research into human and human-system interaction. It also means we can experiment with connecting other shared, mixed realities, for example, by adding Internet of Things or virtual humans. Emiel’s work is going to keep the Interaction Technology community busy for some time to come.’

‘WITH THE VR GLASSES, YOU’RE RIGHT THERE WITH THE OTHER PERSON IN THEIR SURROUNDINGS’ EMIEL HARMSEN

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 27

’TAKING R3D3 TO THE NEMO SCIENCE MUSEUM TO INTERACT WITH KIDS WAS A GREAT EXPERIENCE’ DAAN WILTENBURG

‘FUTURE RECEPTIONIST ROBOTS WILL PERFORM THEIR TASKS SO EASILY, WE’LL HARDLY NOTICE THEY’RE ROBOTS’ MAX BODE 28 | I-Tech

DAAN WILTENBURG AND MAX BODE ARE HELPING A ROBOT IMPROVE ITS COMMUNICATION SKILLS

R3D3, an acronym for Rolling Receptionist Robot (three R’s) with Double Dutch Dialogue (three D’s), is a recently completed research project named after its star robot. The aim of the 18-month project was to address scientific ICT challenges in the areas of computer vision, spoken language interaction, and data processing, as well as practical challenges in robot development. The robot is coupled to an avatar, or virtual human, shown on a screen placed in R3D3’s hands. ‘R3D3 can’t talk, but he’s capable of non-verbal communication, such as nodding. The virtual human can talk,’ says Daan. ‘Together they will be able to communicate with several people at once, while also picking up basic social cues. The technology is much more socially intelligent than the question-and-answer functionalities you get from services like Siri and Alexa, the Apple and Amazon voice assistants.’

R3D3 LEARNS MANNERS Remember R2D2, the cute droid in Star Wars? He has a younger cousin at the University of Twente. His name: R3D3.

Master’s students Daan Wiltenburg and Max Bode have been busy teaching R3D3 how to behave in public.

LOOK AT PEOPLE Daan and Max, Master’s students of Human Media Interaction* and Human Computer Interaction and Design, have spent several project periods and a lot of free time with R3D3. ‘The main focus has been to teach him certain behaviour by combining his computer vision capabilities with new programming,’ says Daan. ‘Specifically, we wanted to teach him to look at individuals approaching him; we’ve found that people interact more easily with a robot if it looks at them. Then we wanted him to recognize when a person starts or stops speaking. In our most advanced experiment so far, we took him to the NEMO Science Museum in Amsterdam for a day and had him interact with children.

His job was to help kids take turns speaking by managing the turns with looks. It was great, the kids really reacted!’ Comments Max, ‘Our main finding is that it is possible to give a robot a role in a multiparty conversation. We also made other discoveries, for example, that kids will interact more easily with the robot if the virtual human involves it in the conversation.’

TEN YEARS FROM NOW Ten or so years from now, Max and Daan believe robots like R3D3 will be performing many public roles. ‘And they’ll do it so easily, we’ll hardly notice they’re robots,’ says Max. Mariët Theune, who teaches Conversational Agents and Natural Language Processing in the HMI programme, is pleased with R3D3’s development. ‘On a low budget and in a short time, Max and Daan have turned R3D3 into a useful robot. The combination of robot and virtual human with group interaction makes their work quite unique. The next steps will be to add new functionalities.’

INTERACTIVE TECHNOLOGY | UNIVERSITY OF TWENTE | 29

* In 2017, various new developments in the field and in our educational programme led us to the decision to carry forward the HMI Master’s under a new name as of the 2018-2019 academic year: Interaction Technology.

You can find out more about this exciting 2-year science programme and the different tracks and degree options we offer on our website at utwente.nl/ en/education/master/.

COLOPHON Text: Stephen Teeuwen (More) Design: Station Noord Photography: Eric Brinkhorst Thanks to all participants with whom we shared the Interaction Technology experience Publication University of Twente, 2018

Campus address University of Twente Drienerlolaan 5 NL-7522NB Enschede The Netherlands Postal address University of Twente P.O. Box 217 NL-7500AE Enschede The Netherlands T +31 (0)53489 5489 Study@utwente.nl www.facebook.com/utwente www.utwente.nl/go/itech