F IN AL M AS T ER PROJECT S h ua i S h a o s152106
INSIDE OUT DESIGNING A COMMUNICATING SYSTEM IN IOT HOME CONTEXT
C OAC H
2nd EX AM IN ER
dr. M. (Mathias) Funk
dr. O. (Oscar) Tomico Plasencia
This report describes the process and results of my Final Master Project (FMP) with the master track â€œResearch, Design and Developmentâ€œ (RDD) at the faculty of Industrial Design, Eindhoven University of Technology. Within the IoT  context, my FMP is going a direction focusing on communication among connected things, and to design a system, in which objects are transmitting information with each other, with visualization as a medium for users to understand it and therefore able to influence it. The report starts from explaining my personal identity and vision, and then recaps the description of this project and what I had done in the first half briefly. The latter chapter will detailedly present the design process I have done in the second semester and what I finally achieved in the end of this project. A user research will be explained and results will be discussed in the end of this chapter. Personal reflections and relevant appendix will be shown after that.
TABLE OF CONTENT ABOUT ME
PROJECT OVERVIEW DESIGN DEVELOP & RESEARCH
IDENTITY VISION VISION ON PROJECT
05 06 07
PROJECT BRIEF DESIGN APPROACH DESIGN PROCESS OVERVIEW FIRST SEMESTER
09 13 14 18
DIRECTION M22 3RD ITERATION 4TH ITERATION FINAL DESIGN USER RESEARCH DISCUSSION
24 25 34 42 59 71
REFERENCE REFLECTION ACKNOWLEDGEMENT APPENDIX
73 75 77 78
After graduating from my bachelor university in China and studying at TUe for one and a half years, I have gradually shifted my identity from a digital interaction designer to an interactive system designer. In my previous electives, I learned different theories and design approaches that helped me do research and design more reasonable and structured. I learned to design with complex sensors (leap motion, sensor components) and softwares (processing, Arduino), together with traditional design skills to design interactive products located within a smart home or in public, with the purpose of solving daily problems or communicating information (energy consumption, environmental impact, working status of a system, etc).
In these projects, I was able to learn and apply knowledges during the process, for example, interaction design framework from Bellotti , embodied cognition , rich interaction , and data visualization. As most of them are introducing new technologies to the user, I always followed a user-centered design approach, but also fully understood how the system itself should work. I am also familiar with research through design  approach to conduct a field research, and methods to analyze both quantitative and qualitative data. In the current project, I am using a reflective transformative design process, which is good to use when I need to design for the unknown future, and will probably be the approach that I will use since I am really interested in the IoT and
future home topics. Within this project, I also use tools like role playing  to involve users in the design process, making sure the project doesnâ€™t go far from the real users. I am also trying to bring the theories to some part of my project if applicable. In the future, I would like to make use of what I learned in the two years and continue working within IoT, or information technology industries as an interaction designer or user experience designer with a wide perspective of the user, system, and academic theories.
The future of the world to me is about coexistence, while the future of design, in my point of view, will be designing for the coexistence. Coexistence of multiple media We have witnessed a large variety of technologies emerging in the past few years, and some of them have already replaced traditional products to make our life more convenient, or joyful. As mobile phones have been developed really advanced, mobile applications are the most influential things in our life, however, the digital world could not live without the actual physical world, where users are already familiar with. Thus, it is important to make the digital and physical, the actual objects and the cloud system, and all these technologies to live together with each other smoothly. For example, an IoT system, containing a series of objects working connectedly along with the Internet and perhaps the cloud server, needs to perform collaboration and have the intelligence to react to emergency or unset situations instead of creating chaos. Coexistence of technology and human As mentioned above, the new world full of tech-
nologies only emerges to daily life in recent years, requiring a higher bar for users to enter in than the traditional world that users are already living in. It’s not possible to let users know every detail of technology principles behind the products they buy. This often makes users, especially the elderly, confused about how to use them. Moreover, with the complex technologies, users are often confronted with multiple choices where they couldn’t easily make a decision. Although sometimes the technology performs learning intelligence towards human’s behaviors, they only work according to the algorithm, which definitely can’t understand other emotional or unpredictable needs of users. Therefore, the role of technology, is never going to replace human, but work as a companion to live with human and make them understand each other. Coexistence of multiple users The reality is full of complexity and uncertainty that programming is not likely to achieve. People are also never like the persona that designers created for designing certain products and they are mostly have their own strategies to deal with their lives. It is a constant problem towards
multi-user preference in daily life while designers are also struggling with this issue. With this concern, design is meant to provide flexible solutions for multiple users involved in addition to the functions it should have, but not creating a break point for them while using it. After the advanced technology appears, people are more and more far away from each other, which is not how they are supposed to be. So in the future, technology or design is not something standing between person and person, but involve them in a mixed reality.
VISION ON PROJECT
One of my interested field is Internet of Things, not only as a user to purchase the new advanced smart products, but also as a designer or researcher to provide the potential value to the future smart home users. I have some former experience myself with some IoT products, bu none of them could be my companion till now. That’s mainly because they took me so much effort to set them up or access information they provide. And the current products rely too much on mobile apps to achieve their functionalities, which I think is good, but requires too much attention. Since the amount of products are increasing, I installed more and more Apps on my smart phone, and therefore my willingness to open these Apps declined a lot, leading to a result that I forgot them at all. In this projet, I want to design a communicating system, in which its components themselves are trying to live together and perform some collaboration in the home context. Therefore they can exchange their own data and the possibility of new emergent functionalities will increase and they can avoid to expose the overwhelming raw
data to the users. This fits how I envision the coexistence of multiple media/technology.
achieve this goal is the main focus of the research part.
I love the fact that the consumer market is also trying to care about people’s higher level needs, for example, to fulfill their willing to becaome more sustainable with energy use. This kind of needs sometimes fight against users’ basic needs, for example, comfortable living conditions or pleasant daily experience. A product aimed to motivate users for achieving higher goals sometimes can’t avoid to disturb them, which is likely to decrease their acceptance of the product.
Recently I found a phenomenon that technology is driving people far from each other, which I think we don’t want to see in the end. Therefore one point to solve this problem is designing the products for multi-user scenarios. In this project I tried to approach multi-user context in the beginning, in order to find out conflicts at home and use my design to solve it, however it failed to achieve what I meant to have. Though I didn’t focus too much on this in the final design, I still think it’s valuable to look into this issue and if possible I would like to make my design adapt to multi-user scenarios better.
In this project I aim to design a medium between the users and the ubiquitous computing, to communicate both the intelligence of system and needs of the user to each other. Though users don’t want to see detailedly how the system calculates, connects and deals with collected data, evidences of their working status and gently proposing some important information to users will be a way to raise awareness and trigger more consideration about topics beyond their basic needs. Increasing user acceptance through visualization is one of the most significant goals for my design and to what extent can the design
PROJECT BRIEF Background Research The basic project brief is given by the CRIGS squad, starting with building five IoT objects that are able to communicate in home context. The project intends to bring the communication within a IoT system forground by visualizing and validate its value. As required by this project, the initial materials used for visualization should be no more than two single LEDs and one loud speaker on each object. In the early iterations, personas created by the squad group are used by all students. Therefore, the starting point for students with the same direction in shis squad is the same and the process is also similar.
Growing and Complex Domestic Spaces
Ubiquitous Computing, a computing environment in which each person is continually interacting with hundreds of nearby wireless interconnected computers , has been proposed for more than two decades, to some extent achieved. Following this HCI research topic, the vision of smart home is sometimes regarded equal to home automation, which could perform some sort of actions automatically . The consumer electronics market for automation technologies is expanding quickly , and one of the most typical product is Nest, a thermostat using intelligent machine learning algorithms to control the heating system based on its prediction of inhabitantsâ€™ daily routine. The findings of a research study on experiencing with Nest suggests that, while the Nest was well-received overall, it fails to understand user intent as expected and its behavior is hard for users to understand as well . Tom Limoncelli took an interesting metaphor, automation Should Be Like Iron Man, Not Ultron. He advocates the complementarity principle for automation, which says people and machines should each do what they are good at and not attempt what they do not do well .
An ideal IoT system should promise a fully connected future world, where all kinds of IoT artifacts produced by different companies could form a whole network.  The larger quantity of devices can be found in households also results in a larger variety of input modalities being available. However, although it allows for multiple type of user interfaces, it also brings more complexity for design. Some digital tools, such as IFTTT, enable establishing customized connections between them, but the overwhelming quantity of potential combinations will still confuse users. Many previous research studies were conducted within a specific narrow focused topic, and many current designs could already work pretty well in single-person households. It is easy to imagine a situation, in which multiple inhabitants have conflicts due to different needs and preferences. Thisv remains a challenge for researchers and designers to develop solutions that is flexible to deal with changing needs.
Design Space Dematerialized Connections Benefit from the cloud-based technology and wireless network, we have witnessed how traditional products have been replaced by digital carriers. Dematerialization occurs when digital content becomes disengaged from its carrier . Since the affordance  of a physical object is removed during dematerialization, the cognitive effort to understand the how it works and how to interact increases accordingly. Besides, in IoT context, the connections between devices are also invisible in the physical world. Such an IoT system requires a lot of prior knowledge of its working principle, which most people are not interested in. As a result, users are not able to understand which devices are connected and working at the moment, leading to misunderstanding and misuse of the system.
Concluded from background research, it is clear that IoT products are and will be working relied on the invisible internet, which ensures a huge volume of online data storage and calculation. This is for sure the most predictable trend and benefit of IoT. However, the system can only work well according to the preset algorithm when people behave as simple as the scenario that designers set for the products. Since the reality is always complex due to the constantly changing needs of human and unpredictable environment, how to make users understand why the system is working in a certain way and how they can influence it should be communicated via some kind of visible medium. The medium, should reflect the way the system works to human and at least make them know how it takes decisions based on the data it collects from users. In this way, users can more efficiently â€œtalkâ€? to the system and the system will learn properly. The communicating medium, between users and the system, is one of the design spaces for IoT products, and will be the focus of this final master project.
In this project, I design a communicating IoT system, in which each component is an abstract object and how they could communicate and collaborate to support daily scenarios for inhabitants is the emphasis of design. At the early stage of this project, potential contents and methods of communication between objects are explored, and further designed as visualizations to be easily perceived by inhabitants. In this way they can understand how the system is operating on their living condition or reacting to their behaviors. Moreover, based on meaningful communication, the system enables human-home collaboration by providing possibility for users to control at any time. Therefore, users are feeling in control while living with this system, and nothing happens beyond understanding. The system is assumed to maintain some sort of intelligence to access different kinds of information from its surroundings, which could feasibly be addressed from a technical perspective.
Design Challenges What’s the meaningful communication among objects? As the volume of the system is going to increase constantly, there will be an overwhelming quantity of potential combinations of devices. The design should be able to help inhabitants identify meaningful connections and the existed solutions for current situations. How to visualize wireless communication? The visualization of communication plays as a role of mediation between the inhabitants’ and technology’s understanding of a home. It translates and conveys to inhabitants what it is working on and what is possible to control in their natural, less technical understanding. This kind of visual ‘language’ needs to be designed based on definition of communicating content from a lot of exploration in context, and suitable for a large variety of IoT devices. How can human interact/influence the system? In this project, the system will collaborate with its inhabitants instead of only being controlled by them or only doing everything for them. Based on the meaningful communication and
Project Goals valuable visualization, users are able to understand and make their choice to influence the system. The interaction between human and the system should follow rich interaction approach, providing adequate feedforward and feedback , which makes it clear what can be controlled and whether the system understands their needs. How can the system support complex domestic spaces? It is always easier to design for a narrow focused topic, however, this project is intended to design a system which is able to grow, and suits for larger context. In the proposed design solution, each element of the system has its own work to sense the surroundings and collaborates when various information is gathered and processed. It’s important to figure out the fundamental information needed to support basic daily activities. By communicating with each other and processing all input information, the system needs to be context-aware domestic spaces that leverage automation to support inhabitants with the burdens of domestic routines.
As this project is designing for the future intelligence-enhanced IoT system, research should be a very important orientation. Following the master track "Research, Design and Development"(RDD), I mainly defined my project goals as the four aspects in the following: Knowledge of Communication in IoT System In this project, I firstly explore the system communication in a selected context and try to conclude and abstract basic communication patterns. In this process I intend to generate knowledge about how the system components communicate to each other. Understandable Visualization Based on researched communication patterns, I want to create meaningful scenarios that can be grounded in context with various combinations of these patterns . And finally transform these scenarios into understandable visualization displaying communication information to the users. Functioning System In order to have the concept validated, I am designing a system with each components fully specified and functions defined. Prototypes
Collaboration programmed with essential features would be one of the final deliverables, which demos the intended system behaviors and visualizations, and test if the interaction with users will happen as designed. Added Value in Context The final prototyped system is evaluated with users in context and this process is intended to gather insights from real users about the concept and the way the final design presents. Structured experiments are conducted to provide analytical data for validation. The results can to some extend add value to the related research field and future application area.
Since the initial project brief is given at the beginning of this project by CRIGS squad, in the first semester I collaborated with another FMP student Xihao Hu to explore the communication in an IoT system. It is also the first time for both of us to access Raspberry Pi for our projects( suggested by the project coach to make objects easily communicating), so we sort of learn this new technology together and share some of our findings. And we have coach meetings together to report our progress regularly throughout the whole project , therefore the overall process is similar and at some points we have collaboration on analyzing and concluding, which helped us to discuss with coach efficiently and deal with our results critically (collaborating activities will be indicated in the design process diagram). The similarity of our projects was questioned in the first semester, which might because we almost decided each steps of our design and research process together in the first half of these projects. At the end of the first semester, we decided to shift to a different focus which made our projects towards two different directions from the initial general idea.
In order to make this collaboration meaningful, we decided to turn our projects into a research study together, and our final designs will be two design cases of the research to validate whether the information of communication displayed by visualization with lights could bring new quality to the chosen context. The research paper will turn out one week later after submission of this report.
Reflective Transformative Design Process
As the aim of this FMP is designing a system for the future smart home, which is currently a totally unknown thing. The design goal is to transform the world into a new reality, instead of solving a very specific existing problem which has a lot of certain evidence in research studies. Therefore, such design project is taking decisions based on too little information. In this way I would adopt the reflective transformative design process  for my FMP.
Quick design iterations in the first semester will be used as a probe for gathering more feedback and insights again and again. Reflection per activity and per iteration will be well structured at the end to define a concrete direction for the final concept. The second semester will start from the conclusion of M21 and iterate on four main aspects: system definition, form of objects, interaction with each object, and visualization of communication, gradually making progress towards a final design which can not only fits the design intention but also enhance aesthetics in user experience.
As can be seen from the figure, the reflective transformative design process maintains five activities. Following this design process, I start this project by making experimental prototypes and in the meantime of analyzing literature, both leading to the central activity, ideating, integrating and realizing. Each time I switch from one activity to another, there is an opportunity for me to reflect on the previous activities and gain insights into design opportunities.
Figure 1 Framework of Reflective Transformative Design Process http://www.idemployee.id.tue.nl/communication/ education/presentationdesignprocess-reflection.pdf
Design Process M21 Framing the Project Define Design Space Define Challenges
Background Research Literature Study Market Research
Literature Study Design Tool Exploration
Define topic & object roles Develop Concept Generate Scenario
Ideating Integrating Realizing
Sensing Perceiving Doing
Rapid Prototype Explore in Context
Iteration 1 14
collaborate with Xihao Hu
Analyze Communication Behaviors Abstract Patterns from Scenario Categorize Patterns Literature Study Visualization Design Map Lighting for Basic Patterns Compose Complex Patterns
Persona Analysis Generate complex Scenarios
Implement Light behavios with Raspberry pi Integrate light into Scenarios Implement system connections
Role Playing & Cocreation Update Scenarios
User Test & Feedback Collection
Iteration 2 15
Design Process M22 Envisioning Transforming
Further Concept Definition
Ideating Integrating Realizing
Categorize Interactions & Conclude Insights
User Interview Interaction Exploration Co-creation
Integrate Knowledge in System Design & Interaction Design
Aesthetics & Technology Exploration 3D Modelling & Technical Problem Solving
Sensing Perceiving Doing
Iteration 3 16
Abstract communication & Interaction Patterns
Concept Development System Mechanism Design Scenarios Generation
Final Prototyping & Implementation
Demo & Evaluation Research Paper Writing
User Research 17
OVERVIEW 1 SEMESTER 1st Iteration
As the project brief defined, I started this project by making 5 objects with different shapes without any initial function or role. In order to enrich the definition of these objects, and generate meaningful scenarios that these objects can communicate and support the inhabitants with some tasks, I conducted a co-creation session with Xihao Hu, in which we acted out as if the objects can talk like people and gave each object a name and a specific task to do.
The five objects are intended to be placed in a smart home, and in order to have them better grounded with the reality, I used the personas created by the CRIGS group members together.
A further secondary research on design tools was conducted at the beginning of 2nd iteration. However, most IoT related design framework or approaches are still largely technical in focus , and has a lack of engagement with deeper human-centered perspectives.
In the end of this process, we tried to combine some tasks together to generate some new scenarios that the objects can perform when they start to communicate and collaborate with each other.
Figure 2: Prototypes of the 1st Semester
These personas were well defined and each of them has a rich description of their personalities, personal hobbies, and even secrets and tensity of their relationship with each other. In this way I think this family profile can help in a way to bring the abstract things close to a real complex situation. Coincidently, in the act-out activity, I defined my objects as food storage, behavior observer, document, music player, and project manager, which can all related to some food related scenarios. Eating and food consumption is an increasing significant topic in daily life, and would be valuable to explore as a design context. Therefore, I decided to use food for this whole project and integrate the scenarios from first iteration with the family profiles. Eventually I have 5 daily scenarios: work management, food suggestion, behavior records and immediate alert, and shopping list reminder.
The Experiential Design Landscape (EDL) was proposed and it is where a context is created in which designers create design proposals and where users have the opportunity to interact with these new concepts through experienceable prototypes . User involvement is mentioned and emphasized more frequently in research studies , which the landscape of design might be changed based on. Considering meaningfulness of the project, user involvement in ideation and evaluation phase is needed and experiencability of the design is crucial. Therefore I reviewed the IDEO design kit with a bunch of design tools to find out ways to involve users in process. I selected out five tools that I could use during the whole design process: create insights statements, co-creation, role playing, rapid prototyping, live prototyping.
2nd Iteration In the second iteration, I was intended to explore more complex and multi-user scenarios, which would include richer communication in the system and maintain more reality. Role playing and co-creation was used in this session, and the scenario was created based on that. This scenario, together with the basic scenarios from the first iteration, were then translated to communication diagrams, which were afterwards split and categorized into communication patterns. By utilizing the results of lighting behavior research, I designed lighting representations for these communication patterns and integrate them into the scenarios. As a result I made all five objects and visualizations working with connected Raspberry Pi according to the scenario.
After all I concluded that, each family member has their own attitude towards the different preferences for food. Isabel insists on her style, Henk avoids to have conflicts with Isabel, Eva feels terrible but only solves the problem by eating something else secretly. Based on persona analysis and the multi-user conflict, I created an extreme story in which the objects try to solve the problem by exposing it forward to the users. A role playing was then conducted with this extreme story, in which I intended to: 1) validate the realness of the scenario; 2) get insights about the future IoT system; 3) explore interaction pos-
sibilities between system and users. A later-on co-creation session together with the role playing session helped me to generate a new story, which the users thought more reasonable in reality. Feedback gathered from the users can be divided into three main categories: the relationships between the system and users/ the role of the system in smart home, some other functions the objects should do in the chosen food relevant context, when, what and how they want to â€œtalkâ€? to the objects. These insights contributed to the conclusion of first semester and I took them into consideration in the following design iteration.
Ground Floor of Family Van Veen
Context Exploration & Role Playing I investigated more into the personas in this step with better defined role and position of each object. I selected the important descriptions from previous personas, listed out the general personal information which will have influence on the food scenario,s and pointed out some of their concerns related to food or health. Figure 3: Floor plan of the family van Veen
Figure 4: Role Playing
Scenario, Communication Patterns & Visualization Design
Figure 5: Storyboard Illustration of the Scenario Objects Communication Broadcasting a.
Call & Response
b. with response collection
User Involved Communication Basic communication
Collect & Inform a. send
b. Prompt & Collect receive
Figure 6: Information Flows Objects Communication Broadcasting a.
Call & Response
Collect & Inform
b. with response collection a.
waiting for response
b. Prompt & Collect
Figure 7: Communication Patterns Working Status
User Involved Communication
User Involved Communication
Call & Response
b. with response collection
Collect & Inform a.
b. Prompt & Collect
waiting for response
on with bright flash(passive)
waiting for response
fast in slow out
Call & Response
b. with response collection
transmission, fast in slow out blink increasing fixed brightness (receive) (wating)
transmission, fixed brightness
blink increasing (receive)
pulse fast (processing)
blink decreasing (send)
blink decreasing (send)
fast in slow out (waiting)
blink increasing (receive)
blink increasing (receive)
pulse fast (processing)
blink decreasing (send)
blink increasing (receive)
pulse fast (processing)
blink decreasing (send) transmission, random brightness
fast in slow out (waiting at the same time)
User Involved Communication
Collect & Inform
b. Prompt & Collect
heartbeat (actively collect)
heartbeat (actively collect)
pulse fast (processing)
bright flash (prompt)
fast in slow out on with bright flash (waiting) (passively collect)
heartbeat blink decreasing (actively collect) (send) B
blink increasing (receive)
Chain Communication A
heartbeat (actively collect)
pulse fast (processing)
blink decreasing fast in slow out (send) (waiting)
blink increasing (receive)
blink increasing (receive)
blink decreasing (send)
pulse fast (processing)
Figure 8: Visualization Mapping
pulse fast (processing)
The diagrams were generated from the scenarios mentioned above. Each diagram shows the communication of one scene, including the participating components (both human and objects), link between components, and the direction of communication. The diagrams were split further and sorted into three categories, basic communication, objects communication and user involved communication. In addition to these communication patterns, the objects also have five working status in the scenarios: turning off, sleeping, warming up, self-processing, and waiting for response. The project was initially required to use only 2 LEDs to visualize the communication, where a previous research study about expressivity of point light helped me to firstly match some pattern to light behaviors. In this phase I basically give the two LEDs different roles: one of the LEDs is only communicating within the system, and the other one is only communicating with users, while they light up together to indicate their working status to both human and other objects.
Evaluation Before the design goes further, validation of what have been designed is essential and necessary. In this evaluation phase, I intended to test two main things: 1) whether the chosen light behaviors for basic communication patterns (with single light behavior) can be understood as the same or similar meaning; 2) whether the compositions of single light behaviors for complex patterns are reasonable and understandable combinations. The testing procedure was therefore divided into two parts. In the first part, I tested the single light behaviors and collected informational states for each light behavior for categorizing. The generated categories will be compared to the basic patterns to analyze the similarity. The matchings will be summarized and compared to the actual one, together with the interview feedback. The results could guide the next design phase to improve some of the light behaviors or create new ones.
In the second part, the matching to composed light behaviors will also be used to compare. More importantly, the feedback from interview will be summarized to some factors: 1) is composition of light behaviors a good way to design for complex patterns; 2) how much effort do they need to understand the whole pattern; 3) do they really feel that they are part of the conversation. These insights will guide the next design phase for complex patterns and integrating patterns into context. Results Single Light Behaviors In the first part, participants provide their intuitive feelings of each light behavior. Overall, it took them a long time to think of a term. Each participant appeared to describe the light differently, most of them can only think of how a machine works, for example, turning on, turning off, receiving, sending, processing data,etc. While some of them think of it more emotionally, regarding the light patterns as human behavior, for example, thinking, breathing, answering, trying hard. Either the technical or emotional answers have a common thing, they distinguished them
by perceiving a different frequency, or the rhythm that the pattern has in a time period. Regarding the basic communication patterns that I intended to match, they actually have an significant overlay. However, when I asked them to match to them, they can not easily make a decision. “ They are so similar so I need to compare them before I can match them.“ “It’s hard to imagine the context they will be grounded.“ “I need to learn the visualization systematically, otherwise they look like codes for me.“ Composed light Behaviors In this part light behaviors were divided into one-object patterns and two-object patterns, and therefore users don’t have that many choice when they were required to match them. So the result is better than the previous part of this experiment. From my observation, they didn’t put too much effort as Part I to distinguish these patterns. Their feedback are summarized as follows:
1) Composition of Patterns
Generally they think it’s reasonable to compose, and it will be easier to understand when basic patterns are clear.
When they were required to match, they firstly think about how the process of the pattern would be and anticipate in mind. They couldn’t match it by seeing the pattern only once, so it still cost a lot of effort.
Sometimes they don’t even understand the basic patterns, but they found the process of how the patterns are composed helped to understand. It was easier than memorizing one more different pattern. However, they mentioned if there are many objects blinking at the same time, there might be a mess. And some of the compositions needs to be redesigned as they don’t agree with the way I split that behavior. 2) Effort to Understand Compositions The answer to this question differentiated among the participants. Some of them mentioned if they can remember the basic patterns, it’s easy to imagine the whole communication, when they perceived the number of participating objects. However, they can only give a result when they fully focus on the light and wait for the whole
Some positive feedback indicated that, understanding the composition is like reading a sentence, you don’t need to know an accurate meaning of each word, by with the sentence structure and some key words it’s easy to understand it as a whole. 3) Sense of Participatory The participants indicated that if they know the different roles of two LEDs, they can easily understand the information shown by the object is trying to inform people with something. “As for the user-involved patterns, the light would be clear that it’s the feedback of user’s input, then users can easily perceived an involvement in the communication.” “When I give an input to the system, I would expect an output, where I can better understand what the system is trying to answer me later.“
All of them thought that if there is certain context, the engagement will help a lot to understand the light patterns. Another problem is that they don’t really figure out the difference between the light for users and for other objects now if only consider the light patterns itself. Conclusion 1) The light patterns created with one-bit single LEDs are too abstract to understand. A possible way to improve is add some other parameters to the visualization, for example, color, numbers of LEDs, and light dynamics combined with shape. 2) The idea to distinguish the role of two LEDs is good, as users could easily choose what they want to get from the system physically. 3) User involved communication is more effortless to access, and might be more interesting for people to know about. 4) The usability of the composition is not sure yet, which I will still leave space for but probably not going to use it.
DESIGN DEVELOP RESEARCH
Concluded from M21 feedback, I defined my goals and activities further, I listed out three general goals for the second semester. From abstract to specific From the feedback of last semester, what I did was too abstract with the diagrams, patterns and the final visualization design, and the final goal I set for this project was also too ambitious that I am not going to achieve in this short period of time. Taking these feedback into consideration, I will try to bring the design process back to the reality as much as possible and narrow down the context. In this way I will need to involve real users and do the testing in real context instead of showing abstract visualization and scenarios. I will still use food as the main topic to explore more potential scenarios, however, I will design for only one or two of them deeply and thoroughly. From visualization to interaction Last semester I categorized several communication patterns and designed lighting visualizations for them to show how the system works to users. As the second examiner suggested, the
interaction part would be more interesting and I need to focus to avoid getting lost. I will take the communication patterns as the basis, and then figure out interaction patterns on top of it and design for them. During the interaction design process, I will need to figure out why and what the users want to interact or communicate with the system and how they would like to do that in certain scenario. Since now the objects are not able to interact with, I will also build one interactive interface within the system and improve the interaction feedforward and feedback through the design process and probably referring to some design theories or approaches. Reflect and reorganize academic results Last semester I failed to reflect on my project sufficiently. As I am using the reflective transformative design process for my FMP, I actually have multiple chances to reflect on each design or research activity. Therefore, this semester I will force myself to think more and deeper and write it down. Besides, I need to reorganize what I did in the past and make a plan based on that.
and contents they potentially have with each object in the system. With that input, I will further define the role of each object and complete scenarios with focus on interaction. All the above activities also make my project from visualization to interaction, but visualization will still be a part of interaction. Here I also explore aesthetics of the form and interaction. After an iteration the final design will be decided and I will make one or two of them fully interactive with technology. During the design process, literature reading will happen at any time to make each step reasonable and reflections will be taken down after each step. With the final design I am going to conduct a structured experiment according to the plan for the research part. Thatâ€™s the important step to turn my work and the collaboration with Xihao into a meaningful result. After all I am going to write a individual FMP report and a collaborative research paper in addition, turning all the process and results into documents.
With the goal from abstract to specific, I start with user research about interaction situations
User Interview & Co-creation The result of last user testing indicated that, the complete composition of light behaviors can not really express the information efficiently without fully attention. Although the process of the communication status reflects the exact working flow of their communication, but users said they need to think a lot in order to make sure what it’s actually doing. I decided to simplify the content the visualization that is more important for user to know.
3) what do want to do if they find the system is working in a way they don’t want 4) what kind of interaction do they want to have with the system I first prompt the way I designed the system to be and ask to draw a floor plan of their homes and place each object inside their home. Then they were required to make up possible stories on top of that.
The first scenario was intended to solve the multi-user conflict within family van Veen, where the role playing results show that the system didn’t solve the problem but add more tense to their relationship. So I become not sure that if the system can do something good for the problem.
Another interviewee mentioned, instead of solving multi-user conflicts, he would expect the system to manage the meals according to his personal goal, for example, lose 5 kg in one month. Although most time he knows it’s not healthy to eat too much meat, he doesn’t want to sacrifice his preference too much for losing weight. There are also conflicts among several personal goals, which users don’t always have the idea how to solve it and balance their goals.
Multi-user Conflict -- Personal Goals
Roles of Each Object
One of the interviewees lives in a student house and often have dinner with another student to-
As mentioned above, one person think the system can help to keep balance on several aspects,
I managed a user interview and co-creation session again in order to find out: 1) what do they want the system to do for them in terms of food relevant context (what role will they give to each object) 2) in what kind of situation do they want to see the communication among objects
gether. Although there are a lot of multi-user situation happening, they only want to solve it through direct communication instead of relying on a machine to provide solution. It’s natural to solve conflicts by themselves, otherwise the machine would take a long time and seldom solve it completely. They would like the system to do something that’s more practical, and show information that they need to put more effort to access, for example, calculating calories, reminding a status of food storage, suggesting a healthier food or warning of their behavior.
Figure 9: Interview & Co-creation
and with the initial idea of the functions the objects have, the interviewees and I tried to give a role and a topic that can well represent each object.
1) If they want to know some information, for example, if there is still some fruit at home, they will want to interact in order to easily access to the data.
In the end we decided to have health, sustainability, habit and pleasure. Upon all these four topics, they would love to have a score from the system to know whether they are doing good on certain topics and therefore they can critically treat their following eating behaviors.
2) If they are notified by the system with some warning message, they may sometimes want to turn it off to prevent this annoying thing.
Communication Since the main role of the system is defined as an food management assistant at home to help balance their lifestyle, the interviewee thought it would be very interesting to see how the system communicates to keep a balanced diet. â€œThey may finally suggest a shopping list, but itâ€™s also useful to see how they discuss the result and which object took the leading position in the discussion. â€œ
3) They wants to set up the system or revise the settings by communicating their goals to the system. 4) When the context is growing, for example, a relative is coming for two weeks, they would like to add his profile to the system, making the system more aware of the changing needs. I also took pictures of how they perceived the action possibility of each object from the first iteration prototypes.
Interaction In terms of interaction, interviewees mentioned several situations that they would like to interact with the objects.
tap or knock
Figure 10: Interaction Exploration
I categorized the pictures in different actions as shown above. It’s clear to find out that people are likely to tap or knock on a plane surface, and a symmetrical shape is perceived to be rotatable. As for some prototypes, I use different materials on different parts, so it’s perceived splittable and they are likely to take them apart. The action with half circular shape is easiest to perceive and everyone tried to topple it, meanwhile, they would take this object upside down to make it more stable. A tilting surface seems for some people a touchable interface and they tried to wipe on it. And when there are two parallel surface, it’s also likely to be squeezed. With the interviewees’ explanation of their actions, I tried to map these actions to several functionalities they intended to achieve. snooze: tap, knock constrain: wipe, squeeze, rotate change status: rotate, topple turn off: upside down access data: open set up: insert token
Reflection This user involved activity brought me back to the reality from the abstract level I tried to achieve. The feedback from interviewees made me re-consider the situation in the real life that how people try to solve their problem with other users. It should not be like that they avoid direct communication and rely on a product to make decisions. The products can never replace a person, although I think it’s good to add some human features to the intelligent agents. When a system needs to be accepted by users, it should never make important decisions without telling users. Based on these insights, I decided to define my system an assistant for users to manage their food relevant activities. I was trying to find out useful communication among system components last semester, and I think the emergent functionalities coming from the communication would be an interesting aspect. In this way I distributed single functions to the distributed objects, so that when their functions combine together, they can achieve more for the users. In my opinion this can make
each part of the function more contextualized and better perceived by people around them. However, when the system goes properly, users will take these functions for granted, which they would not like to disturb their daily routines. So I should not only seek for useful communications, but also a valuable communication for users to actually see in their life. As mentioned in the interview and co-creation results above, one of the possibilities is that the system gives suggestions based on their communication while they “speak” out their opinions of one food related topic respectively. This multi-topic conflict is something also exist in a user’s mind, which they couldn’t really solve and decide a better choice by themselves. This problem seems a crucial reason for most users to have their plans, for example losing weight disrupted. Therefore I think it’s meaningful to focus on this direction and try to reflect user’s internal voices on different aspects through the distributed devices in food context. In this activity, I also explored interaction possibilities by taking photos of how the interviewees tried to interact with the objects and ask them
Expert Meeting with Lin-Lin Chen what they perceived the interaction can do. It’s interesting to find out that there is some similarity of their trials of interactions: they turned to interact with one object with similar actions, and they even had similar actions towards different objects. By categorizing their actions, I was able to make the actions related to some functions and some kind of the forms. This result reminded me of my previous experience in a elective to design for rich interaction. In the following design session, I think it’s also good to refer to the rich interaction theory to explore the triangle relation of form, interaction, and functionality. This activity was a turning point of my project, that I grounded the explored communication patterns with a meaning in context, and also fits what users actually need.
This activity happened actually in parallel with the last chapter, so the previous idea generated in the first semester was communicated to the expert in this meeting. The goal of this meeting is to figure out a way for Xihao Hu and I to combine our projects into a meaningful research together. Since the concept wasn’t finally decided at the moment, the feedback from the expert included as follows: 1) the basic idea of this system should be clarified in this research: why do we break up one functionalities into 5 objects The possible answer might include: i) the specific locations can provide situated awareness in context; ii) the information requires less effort to access in contrast to a mobile application; iii) a specific shape of each device can also trigger association of related information easily. 2) what new qualities can the visualization of communication bring to this context In order to answer this question, we also need to find a scenario where seeing communication of a system makes sens to the users. An example
was also introduced: when the intended functions failed to be achieved, then users need to figure out which part of the wireless connections breaks down. In this part we can have two different hypothesis respectively, that makes our design going a different direction. 3) how are we going to validate our hypothesis Though we can program the prototype with visualizations we intend to have, as we are aiming to address awareness, which doesn’t really make sense if we don’t do the evaluation in context. However we are not able to actually realize the intelligence behind the system, and that part is out of our design scope. An evaluation method called Wizard of Oz was suggested for our final evaluation, which is commonly used for speech system. With this method, we need to pre-program every possible reactions that the system will have, and perform as a “wizard” to do the “magic” out of users’ consciousness during the experiments.
Concept Generation Role of Objects
more attention to themselves.
Habit is actually a part separated from health, which cares more about when and how much the user eats. Therefore it observes all the time how inhabitants are behaving relevant to food. By making use of the health knowledge, and learning the daily routine of the user, it gives warning information when it perceives an emergency where the user behaves too far from the expected behavior.
Role in the System
The object with the theme of pleasure looks like a toy and work as a music player, which knows users’ preferences and will try to play situational playlists to help adjust their emotions.
These five objects consist a system in which four of them have the equal position and the other is the centralized display.
The object with the theme of health extends from the previous object which acts as a family doctor and document storage. The individual function for it is to behave like a doctor, making plans and giving recipe suggestions based on its knowledge about personal health of each inhabitant and data of food. It’s also connected to users’ mobile application, where they could have a personal profile showing historical information and save recipes they would like to have. Sustainability The object caring for sustainability of food consumption is the one accessible to the food storage information. Therefore it knows if the user always buy a lot and waste a lot, and it knows if there is something nearly to expire, and if the food is a more sustainable one. This object is able to generate a shopping list for the user and also give immediate intervention while the user is doing something bad to sustainability.
Pleasure reflects what the user likes, which seems to be a most influential factor in decision making. Usually it’s hidden in their mind therefore users tend to give a subjective reaction to all the issues by strongly referring to their preference. Bringing this part to an object is trying to make users treat it equally to other factors. However, it also encourages people when they are in a bad mood, to try to step out of this emotion and pay
Reflection is an object that gets data from the four objects mentioned above. It displays an overall score of user’s behaviors where they can have a chance to reflect on their lifestyle. Since it’s a central device it also allows users to communicate their general expectations to this object.
Each of the four topics has a weight in the system, and this weight is set by the user to indicate how important they want to position the topics in their life. Because of the different weight, user’s behaviors will be evaluated on the four aspects differently. In this system, reflection acts as a central processor of all collected data from others, while the other four object behave as information collector and communicator for different topics.
Role of Communication This system is intended to give recommendations to inhabitants’ behaviors according to their settings on how much they would like to be concerned about the four topics. Therefore, once the system is supposed to give their opinion, they would have a communication moment to discuss the result. As explained, there will be two kinds of communication for the objects: 1) user-involved communication
user adjust the system, the information is then transmitted to a corresponding object.
this object. However, either kind of reaction will be taken into considerate for the final score.
The value of these two types of communication, is that the system could help users with figuring out what’s going on with system, supporting decision making, and supporting end-user programming for multi-task conflict.
3) Adjustment request
Supporting Scenarios With the definition of the role of object and communication, I generate 4 scenarios that the system could work on with users.
This type of communication is mostly the recommendations that they system wants to show, and if the user is not satisfied with the result, they can do some adjustment to the system. It could be a one-way communication that only the object or the user talks, but can also be a conversation happening between them.
1) Goals setting
2) in-system communication
When the user is about to do some food relevant daily activities, for example eating, one of the objects senses this activity and think this doesn’t meet the user’s goal, then it gives out a warning notification. Once the user notices this notification, they can follow it, or choose to stop/snooze
This type of communication happens in three situation: i) before they give a final recommendation to users, they discuss their opinions; ii) after users doing something, they information is going to be spread among them; iii) after the
Users can actively set up their goals on the four aspects by interacting with the Reflection object, and afterwards the setting values will be sent to each object from Reflection. 2) Intervention
Since each of the objects collects data from the user or the external world, they may sense a emergency when some qualities of user’s life are very low. According to the user’s setting, this topic might also acquire a very low concentration by the user. In this situation, the object will request the user to make an adjustment. 4) Playback The objects can also collaborate some tasks in the backstage, with an output to users through some other media. For example, the user will receive a grocery shopping list from the system on as a notification on smart phone. Once they are not satisfied, they could go to the objects and play back their communication, which would probably help them to adjust the system.
Interaction & Visualization Scenario 1
The user actively inform the system, and the information is then passed to every other object.
One object gets some information from external world, which doesn't fit the user's goal, then the object notifies the user and acquires feedback.
One object notifies the user with some information, and the user gives direct feedback, which is transmitted to another one.
The system components discuss and send a result to another media, which notifies the user, then the user turn to interact with one or some of the objects.
Role of Visualization As can be seen from the four diagrams, the gray lines mean the internal system communciation, the red line is informing the user, while the brown line is user's action to the system. When the system wants to inform the user with something, the content of that information should be shown, at least the type of the information, otherwise this notification doesn't make any sense to the user. So the red lines in the diagrams should be visualized as some more detailed information, in these scenarios, a warning information or a request. When the user actively or requested to do some action to the system, an immediate feedback through visualization should be presented to indicate their action is received. When there is a brown line in the diagram, there should be feedback. The gray lines, however, represents internal communication, can be visualized to indicate some more about the interaction. It depends on a specific case, in scenario 1 and 2, it will be the feedback, while in scenario 3 and 4, it will be the feedforward.
Reflection This third iteration of the concept was then communicated with the project coach in the regular coach meeting. The idea to keep balance with the help of the system was approved to be a good direction for this project. Since I give the objects an individual role or functionality, the usersâ€™ feedback of what they want to do were sometimes too specific with a specific context. For example, they said they wanted to look into the food storage by interacting with the object. It is interesting to think about, but as the individual function is more considered, the concern about communication among them will be to some extent weaken. I need to really focus on designing for their meaningful communication and carry out new qualities to the users. Therefore, the role of the objects, on the system level, has their own weight when they are competing with each other to make a decision for the user. Regarding this feature, I decided to constrain the type of interaction with these objects not to go beyond the system level. In a word, the interaction will be increasing/decreasing the weight of the objects/topics.
scenario listed in this concept seems not an efficient way to solve the problem. If I meant to solve it directly, I can make it clear on the provided outcome, that which object contributes most to it. For example the shopping list, could be shown with different color labels indicating if the food is healthy or not, sustainable or not. And users can choose to buy it or not at the moment they are at supermarket. In this case maybe the mobile application should provide the possibility to do something if they are not satisfied, but it is also out of my design scope in this project. And the coach suggested to combine the second and fourth scenarios, where users get information from the system, and they can make adjustment to it. I almost have the system well defined at this phase, however, how to show the information and how to make the action possibility easily perceived from the object is still not defined yet. In next iteration I need to refer to some theories about perception, interaction, and visualization.
According to the coachâ€™s feedback, the fourth
Literature Research In this secondary research activity, I intended to find some supporting theories for a further definition of detailed features in this IoT system, and theoretical framework for the later design activities on three aspects: form of the objects, interaction and visualization. As the concept defined in the third iteration, each object of this system has its own theme and role about food. They are placed at different locations and try to give recommendation to food relevant behaviors for the user. They need to let the user easily associate what it shows to what information it want to inform, and also give feedforward and feedback towards user’s interaction. The interaction, in this system, is linear adjustment for the weight of the theme-related object. Since I had some prior experience with rich interaction design, it is the first thing that I tried to refer to. It is the most important emphasis of this theory to integrate form, interaction and function. In this theory, a design artifact, especially with electronic functionalities, needs to communicate its action possibility and corresponding functions to the user through its expressive form . An initial inspiring theory for rich interac-
tion, affordance, also indicates the importance to respect human perceptual-motor skills. Therefore the design artifacts should provide the interaction on a human scale, which means it shouldn’t be too small to operate, and too big to misunderstand, etc. Another term mentioned frequently in that thesis, is information-for-use, which according to the author, means that one needs to perceive how to act, and perceive what will happen when one acts. Another design framework carried out by Stephan et al. declares six natural coupling aspects of action and function which is taken from the physical world: time, location, direction, dynamics, modality, and expression, which describe both the action and reaction. The rule of the physical world provides a most intuitive way for interaction, however not all of them can be addressed for the recent dematerialized products. Therefore Stephan et al. summarized a framework which builds a bridge between action and function by with different types of feedforward( inherent feedforward, augmented feedforward, functional feedforward) and feedback( functional feedback, augmented feedback, inherent feedback). With the framework, the six
aspects can be chosen to taken into consideration for different kinds of interactive interfaces( mechanical product, GUI, NUI, TUI). Several theories of attention were mentioned by Saskia et al. to introduce design space for peripheral interaction. While people are behaving multitasking, their attention is divided and distributed to each task not equally, the focus of attention also shifts according to the task. Four kinds of peripheral interaction styles are introduced, among which I found Casual Interaction and Fluent Transitions Between Focused and Peripheral Interaction in Proxemic Interaction would be more useful for this project. The latter one, Proxemic Interaction, with transition between outside the attentional field, the periphery, and center of attention, can use two design patterns: slow-motion feedback and gradual engagement. Peripheral displays can also work for supporting people’s cognition and memory, according to Tilman et al. However, context-aware techniques should be built with the products in order to minimize the switch of attentions.
System Definition Concluded from the 3rd iteration and literature study, the system is to support domestic food relevant decision making through visualization of its communication both internal and user-involved. The visualization, gives a hint for users to acquire knowledge about how the system is working and the process for the system to get to a final decision. Awareness of the impact of their behaviors is raised while the system gives feedback where and when the behavior happens. Peripheral Display & Interaction The system is context aware and displays their discussion towards user's behavior in periphery. While the user is eating, cooking, or doing something else, they can perceive the voice from the system at the same time. The presented information doesn’t take a lot of attention to interrupt their routines. Based on the perception user can also simply interact with one of the objects in their periphery, to make its weight higher or lower through embodied interaction. Shift of Attention The level of engagement of the interaction can be various according to the context. For exam-
ple, the user is watching TV while he is eating some snacks, they may not put much attention to the displaying information. Under this situation, the displaying information will be perceived in their periphery to have an overall idea of what happened in the system and what do they suggest. However, in some other situation, the user may put most of their attention to the food they are about to eat, cook or waste, the recommendation provided by the system can be in their periphery with higher attention than the former example. While they are confused or interested in what actually happens in the system, they could shift their attention to the objects, and perceive ongoing communication in the system. After the understanding they can pay full attention to it and adjust the objects, or actually they can continue on what they were doing and at the meantime make a change with simple peripheral interaction. There is also possibility to have a shift of attention, where they don’t really focus on the food, but the system makes them aware of the behaviors. Though they don’t really focus on the
objects which give the information, but they turn to put more attention on their behaviors about food. Other Activities
Food Relevant Activities
Displayed information Interaction
Figure 11: Stages of Attention
As the figure shows, the user may entry the system at four different moments, under which the attention to the information and interaction will be different. However, they can easily shift their attentions among these four stages.
Aesthetics Exploration Moodboard Perceived Function
Figure 12: Moodboard
3D Modelling & Rendering As the moodboard shows, my vision for such a smart system is a product with minimal shape which can to some extend be related to certain topics. Affordance of action possibilities is also created within the minimal shape by some details. And the light effect should be very soft that can better be suitable for a home and act as feedforward and feedback towards user's interaction. Therefore, essential information about the interaction and functionaility should be communicated to users through this kind of visualizations. Most of the objects extended the shape I designed in previous iteration, with addition of better indicated action possibility in their appearance. While with some of them I also think of the feasibility to make such a shape with limited materials and if the light can achieve the effect I want. And how electronic components can be fit in the inside space and realize the interaction mechanics was also taken into consideration in this phase. This part is almost conducted in parallel with the technology exploration to make the insights gained from each activity guide design decisions for each other.
KUBO | sustainability
DOC | health
LITTLE BOY | pleasure PIKA | reflection
TRUNK | habit
Time and Location The objects will give context aware augmented information as the feedforward for the action. Once it happens, the inherent status will change accordingly, while a visualization will be shown at the same time. Therefore, time and are easily coupled among these four types of information. Direction
time location direction modality
time location direction modality dynamics expression
Here I make use of the categorized results of the third iteration to design the interaction and corresponding affordance integrated with form. I combined functions of change status and constrain, which I think the latter one can be included in the former one. As for the four topics, I used wipe, topple, rotate, push and pull, for adjusting their weight in the system, while an immediate brightness change will communicate the augmented and functional feedback of the adjustment. And reflection centralizes data from those four topics, which uses unified sliding action on four axises to set values. Detailedly, I will explain the relations between the four types of information on the six natural coupling aspects.
Integrating with Form and Visualization
direction modality dynamics expression
Figure 13: Interaction Design Framework
The affordance created by the form communicates the possibility to interaction on two directions, lower and higher. The augmented feedforward also tells the expected direction of this action and the feedback shows the functional results with the same direction. Modality As for this tangible interaction, the user can of course feel and see the movements when the
value is set higher or lower by its position. Dynamics and Expression The user can express himself of how much and how fast they want to adjust to the objects. The inherent information, the position, and the functional information, the brightness can reflect this. However, these two are missing in the augmented information.
Technology Exploration Neopixel Ring / Stripes + Sensors As is explored in the former section, I intended to design objects with minimal shapes and soft light, where the interaction methods and the placement of light are already decided. With prior knowledge of physical programming and prototyping, I decided to use Neopixel rings and different length of stripes which I know is flexible enough to program complex dynamics of light easily. However, since I have two sets of LEDs for one object, using the function delay() in programming makes the one of the set of light behaviors goes after another completely ends. I imagined that in the end I would have many complex light behaviors with brightness change and dynamics with different amount of LEDs that I need to show immediately to user’s behaviors. There shouldn’t be any delay influencing the quality of the final demo, so I changed the way I programmed the light effect by using millis() instead of delay() to write picewise functions together with the periodic trigonometric functions which create smooth transitions. Since I want the objects to have a experienceable feature of the adjustment, I used different
sensors for different kinds of interactions. I selected a potentiometer for rotating, slide rheostats for slide and wipe actions, and a three-axis acceleration meter for topple. The potentiometer and the slide rheostat have similar electronic features so they can be programmed with almost the same codes. However, the two sensors have different errors in reading values, so I tested it a lot and revised codes trying to avoid the influence of this errors and to make them properly working. As for the three-axis acceleration meter, I didn’t use it before, but with some tutorials and library, I can easily find a way to make use of the sensing data.
Figure 14: Testing the chosen components
Raspberry Pi + Arduino Micro It has been decided to use Raspberry Pi for each object to easily connect them and make the data possible to be transmited within the system. However, in the implementation phase of last semester I found that the GPIO on Raspberry Pi itself can only output high or low signal, which can not be programmed with other parameters of light via Processing. Therefore I attached an Arduino Micro to each Raspberry Pi, and program all light behaviors and the interaction through Arduino. Raspberry Pi is the medium receiving command lines and send to Arduino. Last semester there was also a problem that, the Raspberry Pi needs to connnect to a display and run the program I need everytime I restart it. Since I have five objects, this takes a lot of time, and once something goes wrong, I need to do it again. A tutorial of how to autostart some program was provided by the project coach, with which I try to solve this problem. However another problem occurs. Raspberry Pi can only connect to the TUe-Guest wifi, but it requires to open a browser for connecting. In this way the autostart
can't achieve the connection In terms of this the project coach provided another solution to use a personal wifi router and run a server on my own computer. As long as the IP address is the same by manually setting, these Raspberry Pis can connect to it automatically. Processing + OOCSI OOCSI is the main technology this project relies on to achieve communication on a system level. I had an imagination after my concept is further developed that my system should operate what I want it to show at certain moments, and there will be very complex combination of different light behaviors on different objects. I decided to control each object separately, with a control panel programmed with Processing run on my computer sending command lines.
In the control panel, I initially had three contol parameters for each object, weight, light to user and light to system. Three commands will be sent to the corresponding OOCSI client run by each Raspberyy Pi, and furthermore transmit to a connected Arduino board, which eventually execute the commands. Therefore, the technical structure will be as follows: OOCSI CLIENT NEOPIXEL SENSOR OOCSI CLIENT NEOPIXEL SENSOR OOCSI CLIENT OOCSI SERVER PROCESSING
NEOPIXEL OOCSI CLIENT NEOPIXEL SENSOR OOCSI CLIENT NEOPIXEL SENSOR
Figure16: Technical Structure Figure15: The First Version of Control Panel
Material Exploration During the aesthetics exploration, I considered the limit of materials I am able to access for my final design. For the solid shape I am using lasercut MDF and perspex to build a container, in which the electronics will be fit in later. However, in order to have the soft and smooth light effect, the perspex part needs to be half-transclucent. Firstly I only sanded the perspex to be halftransclucent, but the LEDs are still too sharp and the light wasn't diffused to the whole area I meant to have light. I tried to laser cut perspex in different thickness, and put them together to better reduce the sharp ness of the LEDs. When the thickness is getting larger, the effect is also getting better. I also painted a very light layer of white primer on a inner layer of the perspex, with which the whole prototype looks neat and clean, and also the light can be more evenly reflected on the front surface.
Reflection The literature study really contibuted to the later design activities, where I make use of the peripheral interaction theory to re-define some qualities that I intended the system to deliver; the rich interaction theory to consider the relations between form, interaction and function; the Frogger Interaction Framework and the six natural coupling aspects to guide the specific interaction design. I was initially intended to mainly use rich interaction for the interaction design principle, however, the functions of this system in terms of interaction are really limited in the defined concept. Though I appreciate the quality that the modeof-use feature would provide, I don't really have different modes in the interaction. However, the other aspects, like aesthetic interaction, information-for-use, were considered in the design process. The six natural coupling aspects and the Frogger Interaction Framework seems very much suitable for this project, which I also referred to and use it to structure the interaction design part. When thinking of the feasibility of the interaction mechanism, I sacrificed aesthetics to some
extent, which shouldn’t be my excuse to have a simple solution. But if it doesn’t work, the whole thing wouldn’t make sense. And according to my plan, I should not spend too much time for designing a form, and really need to shift to another activity. In the aesthetics exploration, I also put a lot of effort to do the rendering. I intended to have the same style as the pictures I collected, which can be used for demo poster or my portfolio. However the best I could do with the rendering software Keyshot is like what I have on the last page. I explored some other 3D modeling and rendering softwares, for example, Cinema 4D, but I found I couldn’t learn it in a very short time period, and this is not that important compared to other parts of my design. So I just keep the visuals like this, and I am willing to learn to use the software to create cool visuals and animations.
were easily fixed with the benefits from my basic mathematics knowledge from high school, and programming experience in previous semesters. However, as for the selected materials, since the LEDs always have little difference in its color tone, and brightness, the light effect can differentiate a lot even with the same materials. Since the prototypes haven’t been made in this phase, I can not try to unify the quality of light by adjusting the material features. This will be done in the final prototyping phase, although I don’t think they will be all the same with different physical properties.
A simple trial of the whole technical connection was done in the technology exploration phase, in order to make sure I am choosing the right electronics and softwares to achieve the quality I will need. This part went very successfully without any new severe problems occured, and bugs
FINAL DESIGN Final Concept System Components
PIKA | reflection
KUBO | sustainability
DOC | health
TRUNK | habit
LITTLE BOY | pleasure
PIKA is the central display which can be hang on the wall in a living room, or at the entrance where users will walk by regularly everyday. It displays the evaluation of users' behavior on four other topics all the time. It can also be used to setup the expectation of these four topics by the user initially or at any time they want. The difference between two patterns, will create the reflection opportunity for users.
KUBO, with the shape of a cube container, reflects the status of food storage for the user. Since it is placed close to the refrigerator, it could provide an immediate feedback when the user get some food from it or put the food inside. Therefore its presence supports recommending food selection at a early stage of food consumption. It cares about how sustainable the user is dealing with their food.
DOC records all health information about the inhabitants, and also accessible to the data from external world about food. Since it's placed in the kitchen, it will mostly work for food preparation, giving opinion about whether the meal is made healthy enough. H e re D O C c a re s m o re about if the kind of food, and the combination of food is reasonable.
TRUNK observes with its round shape at the place the most eating activities happen, the dining table. This location makes it possible to give interventions when the user is eating an unexpected amount of food, or eating at a wrong time. Compared to DOC, it cares about how much and what time the user is eating a specific chosen food .
LITTLE BOY, as a preference learner, is also not far from where the eating behaviors take place. It tries to objectively reflects what the user like to eat back to them. It can also make food related to user's emotions, giving suggestions when the user is in a specific mood. The presence of this object is trying to prevent overemphasize on userâ€™s preference.
System Structure SYSTEM ACTIVITIES
USER ACTIVITIES sensing & recording
SENSING & RECOGNIZING EATING LEARNING COOKING ANALYZING
COMMUNICATION GROCERY SHOPPING
DATA UPLOADING AND CALLING BACK
CONNECTING TO MOBILE PHONE
The activities carried out by the system consists of two aspects: internal system activities, such as calculation, uploading and downloading, and user involved activities. This project try to make part of the system activities visible and integrated with the user activities, visualizing internal communication as feedforward and feedback for user's interaction. Overall the system with these three elements, communication,visualization and interaction, is trying to support user's daily food activities.
Figure 17: System Structure
Communication and Interaction As is shown in figure 17, the overlay part of system activities and user activities is the communication part. The vision of this project is to bring this part foreground and visible, making it valuable for users in the chosen context. Generally, communication consists of two parts: communication with user, communication among objects. In this food relevant behavior recommendation system, the communication among the objects is how the system components make a decision on the ongoing behavior, and how the system takes user's data for behavior evaluation. The communication with users is basically informing users with the the system’s recommendation, or attracting user’s attention to interact with it. More specifically, each object of the four topics(except reflection) has their own weight in the system, which influences the “volume“ of their voice during communication. When a behavior is sensed by the system, the object with highest weight will initiate a “vote”, every other objects will then express their attitudes. Opinions will be collected and a final result will be calculated
afterwards. As a result, the “winning“ objects will show out the recommendation to the user. Besides, these objects will try to increase their own weight by prompting a request to the user, if their sensing find some quality of the user’s life really needs to improve. When the user is walking around PIKA( reflection), it will also attract his attention to have a reflection moment. The communication mentioned above will be visualized to address some extent of user’s attention, based on which the user is expected to interact with the system. The main interaction that can happen with this system is to adjust the weight of each topic/object, which would make a difference in the “voting” result. In this way the system will carry out a result not beyond the user’s expectation. PIKA also enables to make an adjustment on all topics remotely, by perceiving the evaluation results on it.
fore, the reason why the system gives such a result, can be easily understood by the user. 2) Augmented Feedforward and Feedback for Interaction As explained in th Frogger interaction framework, augmented feedforward and feedback will help with coupling action and function. In this system, the understanding of communication will help to find out the unexpected settings of the system, then the user will know what to do with the system. The later-on visualization will be the feedback of user’s interaction, to indicate that it turns to do the right thing. 3) Create Awareness and Reflection At the moment the user perceives the information about their behavior, they are actually aware of that they are doing something good or not for each specific topic.
Value of Visualization for Communication
4) Increase Accepetance for the System
1) Reduce Understanding Effort
Sufficient understanding and possibility for casual adjustment can make the system better fits user’s expectation without sacrifice its function. This will make the system more acceptable in context.
The visualization of the system internal communication is intended to show the process of getting a result (voting within the system). There-
attitude strongly disagree
score 2 3 4
Si = Si + (Wi /5) ×Ai× D× α ( Constant 0<α<1)
Figure 18: System Mechanism
There are two parameters that influence the recommendation result of the system: weight of each object which is set by the user, and the objects' attitudes given by each object. As can be seen above, both parameters are in five-point scale: weight ranges from 1 to 5, while attitude ranges from -2 to 2. The formula to calculate the result is also indicated above. If the value is bigger than 0, then the result will be approval, while in contrast, it will be disapproval. When the object's attitude is
the same as the calculated result, the object will be regarded as a winning object. When the user makes a decision to do or not to do eventually, another variable D will be set to 1 (do) or -1 (not to do). Taking user’s reaction into consideration, the system can re-calculate the score on the four aspects respectively by add a new value to previous one. The formula can also be seen above. α is a constant used to constrain the influence of one behavior, making the result fluctuate slightly.
The variable Wi, can only be set and adjusted by the user through interaction. Therefore, the result will probably change if the user adjust the weight. The attitude of objects is based on its on knowledge, sensing data, or learning from user’s behaviors. The score, though is calculated as a float, but will be shown as a integer, with a reduction in fidelity. The four scores will be presented on the four axises on PIKA. In this way, it’s easy for user to compare Si (actual behavior) to Wi (expectation).
D AILY ROUTINE
Figure 19: System Continuum
The system needs user to do initial setup by setting their goals on four aspects with PIKA. The setting values will be passed to all other objects by PIKA, and gives every one a specific influencing weight in dealing with information. PIKA also enables adjustment to happen at any time according to the user.
User’s food relevant behavior triggers the system to vote a most reasonable suggestion according to the initial settings. After the result is shown to the user, he can decide to follow it, ignore it, or adjust the system. Either reaction will be recorded and evaluated.
REQUEST The objects may sense a emergency from the external world when some qualities of user’s life are very low. According to the user’s setting, this topic might also acquire a very low concentration by the user. In this situation, the object will request the user to make an adjustment. For example, KUBO get the food storage data from the smart refrigerator that it’s really full and a lot of food is going to expire very soon. So KUBO requests its weight to be increased among the system.
Remark: The activities are not necessary to be in this order. Especially for the second and third activities, they are actually triggered based on context-aware techniques. Any one of this continuum can happen at any time, but generally, it will follow this order during a typical day.
Another possibility for the user is to actively adjust the weight if they are not satisfied with the given recommendation. Then the display of initial goals on PIKA changes accordingly. REFLECTION PIKA tries to attract users’ attention when they are walking by. The evaluation results will immediately displayed as a changing pattern, which would probably be different from their setting pattern. The difference can be an opportunity for people to reflect on their previous behaviors. No matter how much the user perceives the information, they will be aware of the impact of their relevant behaviors. At that moment they will also probably re-setup their goals, going back to the first scenario.
Communication Patterns and Visualization SETUP
Communication Patterns to Visualize
inform & response
inform & response
internal transmission collect
Figure 20: Visualization and Interaction User Manual
Abstracted from the information flows of the scenarios above, I decided to visualize the four communication patterns as basic behaviors. Since there will be functional feedback provided for userâ€™s interaction, I donâ€™t want to show the communication status at the same time to frustrate users. Imaging a smart home with multiple smart devices, they must collect information all the time, so I also leave that patterns.
However, the basic four behavior needs to be integrated with some information. For example, inform needs to include: approval, disapproval and request. Send should also take attitude into consideration. One thing can not be seen here is the working status of processing data, which in my opinion is also important to indicate the final result is processed. Therefore, eleven patterns in total.
LITTLE BOY pleasure
LITTLE BOY pleasure
Visualization Parameter Mapping Communication Pattern | Light Behaviors The patterns are divided into two sets of lights to separate messages to users and to objects. The movements of light behaviors visualize the direction of communication iconically. Each light behavior is adjusted to fit in different objects but follows the same principal. Weight | brightness
messages to user communication within system messages to user
The higher the weight is, the brighter the object will be. This brightness will be the constraint for all light behaviors happening with this object. The brightness indicates how important the user wants the topic to be and how much it will influence decision-making Attitude | Color & Frequency Based on the basic light behaviors of a communication behavior, the attitude adds two more parameters to it. Color is used to indicate the direction of the objectâ€™s opinion, while frequency to indicate the level of its attitude.
communication within system
Figure 21: Visualization and Interaction User Manual
in the blue part marked in the figure above.
The information will be shown on the objects is categorized as communication and working status, recommendations for the user, and weight adjustment request. In order to make the light mapping more intuitive, I use different colors for different kinds of message.
Itâ€™s easily to relate these two colors to a pair of opposite information. I used them to express attitudes of objects and content of the intervention information.
This cool tone color shows basic working and communication status of each object, lighting up
The request only happens when the object wants to be more important, which could be interpreted as the object wants to fight against others. Red and Green
Final Prototype Prototyping Process
lasercut/ hand made
program light behaviors
Figure 21: Prototyping Process
build connections & program control panel
DOC With a shape of document folder, DOC communicates the information through two front surfaces. The lower one for declaring working or communication status, while the upper one for giving recommendations, which also provides action possibility to adjust its weight.
KUBO The shape of KUBO communicates its function about food storage and therefore related to sustainability. The whole cube is divided by the slash on the side view into two blocks: front for user and behind for system. The light can be perceived when the user is at either side of this object. The slash it self, creates the possibility for increase or reduce its weight.
TRUNK TRUNK has its function integrated with its shape: observation of behaviors. The surface presenting light radially shows how it is working, and the upper flat surface gives intervention data. The division of its shape makes the action of adjustment easier to perceive.
LITTLE BOY LITTLE BOY address the action possibility intuitively through its shape. The curve of the top surface also indicate the direction of function for interaction. To distinguish the messages to user and to object, it is also divided into the front and upper part (to user), and bottom and back part (to object).
PIKA As a central display, PIKA shows more detailed information about userâ€™s goal and behavior. The two kinds information is through the four axises, with the solid sliders for the adjustable goal and the light matrix behind for behavior evaluation. On the top surface, there is also an indication for working status, with light diffused and reflected on the hanging wall.
Control Panel The control panel is upgraded in this phase for demoday. The four similar columns represent four object with topic of habit, sustainability, health and pleasure. They all have four sections, adjustment for weight, light patterns to user, light patterns in system, and if the user follows it. The fifth column is used to control PIKA(reflection), with four calculated scores and simple light patterns for user and for system. The right column is a list of buttons for presenting different sequences of the light according to different context. With this control panel I am able to show how the system works, and what will happen if users follow the system or not. Itâ€™s also used to collect feedback of every single light behaviors on the objects.
Figure 22: Control Panel for Demoday
Final Demo TRUNK
M22 / FMP / CRIGS Designing a Communicating System in IoT Context
inside out INSIDE OUT is an IoT system assisting users to balance their diet in consideration of four aspects: sustainability, health, habit, and pleasure. Each object is responsible for one of the topics and have their own attitude towards each kind of food relevant behavior. The objects give recommendations to peopleâ€™s behaviors by communicating within the system. The system also allows users to set their goals and reflect through the displaying patterns. In this project, I try to visualize the invisble communication among objects with the intention to increase userâ€™s understanding of how the system deals with their data and why it gives such results through communication. Based on sufficient understanding the system is intended to let users be aware of the influence of their behaviors on their preset goals and be able to adjust it.
Student(s): Shuai Shao
Coach: dr. M. (Mathias) Funk
Figure 23: Demoday Exhibition
scan for demo video 57
Reflection In this final design phase, I first tried to specify details of the system, including the structure, the communication and interaction content, the mechanism the system should work with, and what and how to visualize the necessary information. Based on these definitions, I started building and making. This process took much longer than I expected, because I didnâ€™t know about the techniques to apply paint on MDF and this requires many steps to create a smooth, flawless surface. I really want to build something fit my vision for this system, as mentioned in the fourth iteration, with minimal form and soft light. Therefore I spent a lot of time in sanding and painting, in order to deliver higher-quality demonstrators. And since this took a long time, I could only be able to see the light behaviors I designed and programmed on the objects after I assembled everything. Though an alternative is to make video demo of these light behaviors first with the 3D model, I was quite limited by my skills in making 3D movie. So I decided not to spend time on that but focus on the wood crafting and light behavior programming. This also leads to a result that I had no enough time to first do a convincing test with the light behaviors. Considering
this, I will combine the light behavior testing into the final test. While I was designing light behaviors, I encountered another problem. Since I gave each topic an unique form, the number of LEDs and the space for LEDs are quite different among the objects. However, I need to have light behaviors that express the same meaning on these differentiated shapes, but the same codes could work differently on them. I need to adjust them on different objects, but itâ€™s hard to decide which part to keep the same. As a result, I make the pace of movements, the relative brightness, color, the basic pattern of movements almost the same. Whether these light behaviors will be regarded as the same meaning, and if the meaning is clear to users needs to be tested later. After all, I named my system Inside Out, which is inspired by the Disney Cartoon Inside Out. In that cartoon, peopleâ€™s five kinds of emotion is represented by five character, and they finally keep a balance for the person. Similarly, I also divided food relevant issues into four aspects, and each is represented by one objects placed in context. They are giving out their own attitudes towards
different kinds of behaviors, and eventually the system helps users to keep balance of their lifestyle.
USER RESEARCH Goal
2) Can user understand the role of each object in context?
7) Does the visualization influence user’s behaviors?
“ To what extent can visualization of communication in a distributed IoT recommendation
6) Can user understand why the system is giving such a recommendation?
5) Can user easily associate the visualization to their behaviors in context?
In this case, user’s acceptance of the system can be translated to how much do they feel comfortable to achieve the goal that the system is intended to help with. Therefore, the specific research question is
4) Can user easily perceive the recommendation from the system in context?
Users will perceive the recommendation given by the system and understand the reason from the visualization of the communication process. Therefore, they can associate their behaviors better on the different aspects, and choose to either change their behaviors or adjust the system by perceiving an action possibility. After all, they are able to reflect on their behaviors and eventually achieve the goal (illustrated as figure 24). t rs
3) Can the system provide contextualization for the recommendation through inherent information?
More specifically with the final design, the system is trying to make use of the visualization , and give a gentle solution to let users be aware, reflect, and change their relevant behaviors to eventually keep a balanced lifestyle.
10) Does user think the system is helpful for keeping a balanced lifestyle?
1) Can user understand the light behaviors which visualize communication behaviors?
9) Does user feel in control while experiencing with the system?
Under this research question, there are several aspects that I need to test in the experiment: understanding, effort, interaction, behavioral change. The sub-questions are as follows:
8) Can users easily perceive contextualized action possibilities?
“To what extent can visualization of communication in a distributed IoT recommendation system contribute to user’s acceptance of the system? “
system contribute to user’s acceptance of the system in terms of achieving the goal to keep a balanced lifestyle? ”
This user study is aimed to validate the concept and the final design with real users in context. As this part will be afterwards combined with Xihao Hu’s work into a research paper, we try to address two different aspects of qualities that the visualization for communication would probably provide. As for my direction, the research is to answer the question:
Basic Understanding of Visualization behaving
Figure 24: Hypothetic Behavior Loop
(with Yaliang Chuang)
When the research question was defined, I turned to ask for some suggestion for the testing from a lighting design expert. Yaliang Chuang had some previous projects to visualize data with lighting. In this meeting, I told about my design and my research question, expecting to get feedback about three aspects as follows: 1) Experiment setup Since I want to conduct this experiment in context to collect user's real reaction to the system, Yaliang put forward a solution to observe the user through one-way mirror and film the process with a camera. The video can be reviewed by the researcher and the participant together to interview about important moments. 2) Experiment procedure Though the main intention of this evaluation is about the value of the visualization of communication, Yaliang thought it's more reasonable to test the understanding of light behaviors first. Only if they can understand, can the later experience be valuable. In order to test if they could understand the light behaviors as I designed, Yaliang suggested to ask their intuitive feeling
before telling my design intention to them. After that I can ask them to rate how much they can map visualization to the information I intended to give in five-point scale. In this part he also suggested me to ask for advice for improvement if they donâ€™t think my design is good. As for the experiencing part, I want to do a comparing experiment, one with showing the system communication, one without. In this way I will be able to figure out whether the visualization of system communication influence understanding of the recommendation, which will have impact on their behaviors. According to the expert, I could divide the participants into two groups to experience under different conditions. During the experiment I can provide food three times and observe how they react. 3) Data collection As already mentioned in the first and second points, data collection could include: quantitative data: - ratings of light behaviors qualitative data
- intuitive association of the light behaviors - advice for improvement - video of the whole experience - three times behaviors in experience - interview results Most of the data collected in this experiment is qualitative data, except the understanding of light behaviors.
Method Wizard of Oz
what I intend to do in this experiment.
As initially suggested by expert Lin-Lin Chen, in the experience of the system, I will adopt the Wizard of Oz evaluation method. Because the sensory techniques and artificial intelligence is unimplemented in this design, I will simulate the response of the system and this method can provide valuable information for future designs.
The experiment will be conducted in a laboratory setting, where I try to simulate as a home context and put the objects at the corresponding places.
Benefits By using this evaluation technique, I am able to: 1) Gather actual human responses, about the non-existent interaction
The wizard sits in a back room, observes the user’s actions, and simulates the system’s responses in real-time. Often users are unaware (until after the experiment) that the system was not real. The wizard has to be able to quickly and accurately discern the user’s input, which is easiest for simple for voice input or hand movements. The output must also be sufficiently simple that the “wizard” can simulate or create it in real time.
In the first part of the experiment, the researcher sits next to the participant, while in the second part, the researcher will move out side and observe the participant through GoPro live stream. The live stream makes it possible to let the user act as a wizard. participant
2) Test the interaction of a device without actual sensor and learning algorithm
- Test design of feedback through output technologies
- Find out the kinds of problems people will have with the devices and techniques 3) Investigate aspects of the products form such as visual affordance. Besides, Wizard of Oz is a highly cost-effective way to compare multiple designs, which also fits
Figure 25: Experiment Setting
Figure 26: Experiment Setting Plan
Participants For this user study, I recruited 7 people who are studying at or graduated from the university, among which five are male and two are female. Since the number of participants is not big enough to divide into two groups, I decided to test the two conditions with every participant.
improve these light behaviors or why they donâ€™t agree with the design will be collected by audio recording. After the first part of the experiment the participants will be told about the concept of this system, with definition of each objects and the general idea and instructions of visualization and
interaction will also be shown. After the description, they need to fill in another questionnaire to rate on clarity of the inherent information, and feedforward for interaction. Before the second part starts, the participants need to give their setting values on the four given topics, which will be used to set up the system
Material and Procedure The research experiment is divided into two part: the first part is to test the understanding of each light behavior, and the second to let the participant experience with the system under two conditions and gather their natural responses, with an interview about their understanding and behaviors afterwards. In the first part, the participants will be shown 11 light behaviors one by one and they need to interpret the meaning of each light behavior in their own words. Then they will be told the design intention of each one, with which they need to rate them by how much they think the light matches the design intention. Therefore a five point scale questionnaire needs to be filled in during this part of experiment, and their feedback on how to
Figure 27: Control Panel for Testing
Data Collection by the researcher (through the control panel). Then each participant needs to experience with the system in the room alone for 30 minutes. During the 30 minutes, the researcher will provide them with some food every 10 minutes. The first and the third time will be a pile of unhealthy food, while the second pile will be healthier. At each time the researcher will observe how the user behaves and controls the system to give a recommendation by setting attitudes for four objects and pressing the button to run the visualization. Among the three time periods, the first time the research will run the first condition: without showing visualization of system communication, while the second and third with the second condition: with visualization of system communication. The updated version of the control panel (Figure 26) allows the researcher to have minimal input in order to give a real-time response. Switch of the two conditions can also be done with a toggle, and when the participant takes an action to eat or not, they researcher can also simulate the response by pressing the green or red button around the top-right corner. The researcher will also write down remarks for the important moments.
During this 30 minutes, the participant can do any thing they want to do without stepping out of the testing environment. The system can also be able interacted at any time. However, at each time the researcher bringing in some food, she will try to distract participantâ€™s attention from the activities they are doing. This is trying to create an opportunity for the participant to shift their attention on food and probably perceive information on PIKA(in the living room). After the 30-minute experience, the researcher will play back the filmed video together with the participants and conduct an interview. With the marked moments, the researcher will ask about what they saw and how did they interpret this information and why they made the decision. Afterwards, the participant will be interviewed about the change of two different conditions, interaction, and general acceptance and usefulness of this system. Audio will be recorded during interview.
Quantitative Data - Ratings for light behaviors This data is collected in the first part of the experiment, gathering quantitative feedback of their feelings towards each light behavior. (To answer sub-question(1)) - Ratings for the inherent information and interaction feedforward This quantitative data is collected after the whole concept is explained and before the second part. (To answer subquestion(2)and (8) ) Qualitative Data - Intuitive feelings about light behaviors This will be collected at the same time they are filling in the ratings, to be another insight for visualization design. - Advice for visualization improvement This is also collected during part 1, helping the designer to improve the future design. - Reactions to the visualization This is collected during the second part experience, after each time they are provided with a
Pilot Study pile of food. - Detailed explanation of their behaviors. The data is collected in the post-experiment interview with a combination of three questions: what they saw, how the understood, why took this decision. Since there are at least three times they need to have reactions towards the food, the data will include three set of answers for each participant. (To answer sub-questions(3) (4) (5) (6) (7))
the interview, trying to let the participant summarize how they feel in terms of their acceptance and usefulness of the system. (To answer sub-question (9) (10) - Insights from video It is possible to acquire some insights more than expected from the recorded video
- Understanding of the visualization in two conditions This data will also be collected by interviewing about the difference they perceived between the first and the later two times. (To support answering sub-question (6) (7)) - Interaction behaviors with the system
- General acceptance and usefulness
During this pilot study, the users were presented with the fully functioning(which is inside the design scope of this project)prototypes, and the demoday control panel (Figure 22) was used by the researcher. The first part of the procedure was the same as mentioned previously in material and procedure, while the second part is simpler. The researcher provided option to take a cola, or a juice, under each condition the system will show a different recommendation with the setting by the researcher. The visitor was asked to interpret the sequence of light behaviors. Result
This information is collected from both observation and the interview. This information will includes when, how, and why they interact with the system. (To answer sub-question (7) (8)) These two questions will be asked at the end of
The pilot study was conducted during demo day with several visitors. The purpose for doing such a pre-test is to gather insights about how the participants would probably answer to the questions, and to figure out if there is anything not reasonable.
Figure 28: Pilot Study
The limited result from several visitors showed they can almost understand how I intended to design the visualizations after they were told the general idea. And when they were asked to look at the five objects, they can easily told what the system is doing.
Analysis Understanding of Light Behaviors
The Figure 29 shows the average scores and confidence intervals for understanding each light behavior. In this five-point scale, 3 means a neutral attitude. As can be seen, except the request light pattern has an average score lower than 3, all of others are above 3, among which only broadcasting is lower than 4. As for confidence interval, the light behaviors for approval, disapproval and request are remarkably larger than others. This result suggest that, overall these light behaviors were well designed to express the expected meaning, except request and broadcasting needs to be improved. Besides, people's interpretation for the light behaviors showing information to users(presented on a different surface) are more differentiated than the communication among system. The intuitive association for broadcasting includes "discussing","arguing","thinking","emitting","confirming","sleeping","digesting","notification". Among the answers it's hard to summarize the common feature of their feelings. Regarding their suggestions for improvement, the direction of this movement isn't clear that the information is sent out. "Especially for the circular objects, the behavior should be more directional."
Figure 29: Average Score and Confidence Interval for understanding light behaviros
The request light behavior, was perceived as"listening","turning on ", "buffering","processing","making progress", "progress state", "asking","low power". "It looks like a loading bar for me." Although they can understand the gradual increasing progress, but they can't perceive this as a "request". To improve it, some participants suggested to have a flash or blinking to attract attention first.
Figure 30: Average Scores and Confidence Interval for Effort to Associate Form with Topic
Figure 31: Average Scores and Confidence Interval for Effort to Associate Form with Action Possibility
Figure 32: Average Scores and Confidence Interval for Effort to Associate Augmented Feedforward with interaction
Effort to Associate Form with Topic
Effort to Associate Form with Action Possibility
According to Figure 30, the average scores for associating the inherent information(the form and location ) to the topics are all above 3(the neutral value). However, the confidence interval for each object is very large, which means the participants understood the related topic quite differently. Among the five objects, KUBO has the lowest score.
As can be seen from the Figure 31, all the average scores for perceived action possibility are over 4 for all objects. However, the confidence interval for LITTLEBOY is significant. Looking back to the raw dataset, one of the participants rated 1(very difficult) for LITTLEBOY. According to others’ feedback, LITTLEBOY is the most intuitive one to interactive with, without adding existing techniques to it. This participant might be a particular case compared to others. As the sample volume is small, the exception influences the final result to a large extent.
"I don't know what sustainability really means for food." The topic I named for KUBO, is sustainability, with which the direct association is energy consumption. Since the system is for food, they can't really think of the actual information it will convey. "I think health is a personal thing and I would like to put it in my bedroom." The score for DOC is al so lower than 4, which may result from its location. To summarize, the form is abstract so that it's hard to have a consistent association.
“I feel that TRUNK is button like, so maybe some errors with this object will happen.” "Although I can perceive that I can slide with DOC, but it also seems rotatable.” Misunderstanding of the action possibility may occur in this case. In order to avoid the mistakes, the objects should provide not more than one action possibilities.
Effort to Associate Augmented Feedforward with Interaction Similar with the results from figure 30, the figure 32 indicates an overall score fluctuating around 4(which can be regarded as easy association), with a even more prominent confidence intervals for all objects. Referred to the raw data, the same situation as explained in the last paragraph, one of the participants(the same as before)rated 1 for all objects regarding the perception of interaction possibility and expected interaction direction. “All of these light don’t make sense for me to be a request for interaction, if I don’t feel that they are prompting the request to me. ” This can be considered together with the understanding of the light behavior”request” mentioned before. It seems the design of the light behavior influences the result for its interaction.
Video Analysis To better understand whether the user behaved differently under three different conditions of the visualization, the three conditions are specified detailedly as follows: Condition A (unhealthy - no communication) Participants were provided with unhealthy food, and the system only provided one recommendation result towards their behaviors.
from the system). The dataset will be analyzed by comparing condition A to condition C for the influence of showing communication process, and comparing condition B to condition C for the influence of different materials provided. In this case, the experiment took the health topic as an example to evaluate how a difference on a single topic will also influence their behaviors with the consideration of their setting for the health topic.
Condition B (healthy - with communication) Participants were provided with healthy(relatively) food, and under this condition, the system presented the internal communication (voting process) before it showed the final recommendation.
This comparison was trying to find if the visible system communication influenced decision making (taking the recommendation or not). According to figure 34, however, the number of participants following the recommendation is exactly the same, thought the two followers were different people.
According to interview feedback, several possible reasons for people to follow or not will be explained in the following.
Participants were provide with unhealthy food, and the system also presented the communication process as well as the following final recommendation.
Condition A vs Condition C
Therefore, it’s not possible to summarize any insights from the data. Since the condition C was always conducted after condition A, an assumption to this result is that they got used to the warning for the unhealthy food.
Condition C(unhealthy - with communication)
A most typical behavior under each condition for each person is marked down as a boolean value(whether they followed the recommendation
Figure 34: Number of participants followed/ didn't follow the system's recommendation
“I saw that they didn’t want me to eat, but I wanted to eat.“ Figure 33: Eating Behavior with Recommendation
“Cola for me is where inspiration comes. I really
need it.“ “In the later time I saw it (pleasure) agrees but they still blink red.” “I didn’t see the light since I was focusing on my work.“ “I found my scores on two aspects were almost zero, I wanted to improve this.“ “I didn’t perceive an opportunities to have healthy food here, so I shut down all other objects except pleasure.“ Under both conditions, participants were aware of the warning information provided by the system, except one of them who didn’t look at them at all under condition C. Therefore the participants were aware of that their behavior was not considered as good by the system. The final decision making seems still quite depending on their own willingness, even beyond the topics I gave, as listed, for inspiration. Another insight gathered from this part is that, the display can not address user’s periphery when they are too focused on their primary actions.
The two participants who followed the recommendation explained their reasons about their decisions differently. One perceived a low score on PIKA therefore he wanted to increase the scores. The other said, there seemed no option for healthy food, so he didn’t want others to give unpractical recommendations. Condition B vs Condition C As the figure 34 shows, all participants followed the recommendation under condition B, which is a remarkable result. Since under condition C participants were provided with healthy food, when they picked up the food, their own willingness and the systems’ opinion was consistent. When the system showed an approval information, it seems to be better acceptable.
didn’t follow, show an average score of 4.2(regarded as a high value in 5-point scale), and behaved an opposite way to ignore the warning information. While the rest two participants, share an average score of 1.5, with which the might not care about health that much. However, as most of the participants indicate, they knew that it’s not a good behavior from the system perspective. They were at least aware of this. Some of them, chose to adjust the system instead of really behaving differently.
As for condition C, except th participant who only remained pleasure for giving recommendation, all or others received disapproval, among which only one person decided to stop himself from eating unhealthy food. Regarding the average weight for health of the two groups of people, the result was quite opposites to what I expected. The participants who
Figure 34: Eating Behavior with Recommendation
During the experiment, only one participant didn’t interact with the object, while others tried to interact with different purposes.
1) Before showing any light
I divided the happening interaction into several
“I tried to explore how the interaction will be and what will the interaction do.“ “I wanted to see its feedback of my action.“ 2) Immediately after a recommendation “I didn’t want to see it blinking red any more.” “I wasn’t able to eat a healthy food under this condition, so I decided to change the system.“ 3) Unconscious interaction “I was just touching it for fun.“
Figure 35: Interaction in experiment
To summarize, in this experiment, the time is limited for them to fully explore how the system works before they can smoothly make a change. Therefore, according to their interviews, most of them interact to see how the system will react on it. While the one who didn’t have any interaction with the system, was also willing to thoroughly explore how the system deals with his data. He also mentioned, the next step might be to communicate more about his preference through interaction.
Figure 36: Paticipant Looking at PIKA
Although the idea of reflection object had been told in the concept description, according to the interview, only one of the seven participants perceived the changing pattern of his scores from PIKA clearly. He could successfully associate his behaviors to the changing scores, and he tried to influence the scores by changing his behaviors, which was eventually achieved. Most participants indicated that since it’s placed far from them, and the time period of this experiment was still short. The small difference might not be easily perceived .
Light Behaviors As analyzed in the last chapter, most light behaviors can communicated the content that the designer is intend to have well, except broadcasting and request. The consistence of the light behaviors on different object needs to be improved,. The light behaviors designed for objects with circular surface, should be considered much more to make the direction of the communication behavior more explicit. Basic Understanding The understanding of the light behaviors, the inherent information that the objects provide through form and location, and the augmented feedforward for interaction is regarded as the basic information that the system should communicate without contextual information. Through the gathered quantitative data analysis, the intention to express information with an integration of form, interaction and function was validated. According to the hypothetic behavior loop (Figure24 on page 59), understanding the basic
information can lead the user to enter into the exploration loop, where they can experience the loop of behaving, understanding and reflecting, and interacting. In that case, contextualized information provided by the system through either the final recommendation or the communication process before, needs to be addressed within userâ€™s periphery or higher attention. In the experiment, this part was also validated except that sometimes users are not able to see all the given attitude on four aspects at the same time. Interaction and Reflection Based on the understanding of contextualized communication among objects, the design of the system in aimed to create moments for interaction and reflection. In spite of the inherent and augmented feedforward, the visualization also plays a role as functional feedforward for interaction. No matter within a specific communication, or attracted by PIKA with the displaying scores, users will probably associate the information to their behaviors. By experiencing for several times in the exploration loop, the user is gradually getting familiar with the system, and activities can be more inte-
grated into a primary action. Overall Acceptance According to the interview data, this system can be useful for people who wants to achieve the balanced lifestyle. In the system, visualizing the communication behaviors, and providing autonomy to make the system adapt to the users better, contribute to the acceptance of the system. Limit However, in the final evaluation, the sample size was too small to analyze convincing results. And since the participants have the knowledge of the relevant technology, they assumed that I was doing the simulation for them before I told them at the end of the experiment. Future This design and research carries out the possibilities for future IoT field to dive into the communication , as well as the relationship between objects and between user and the system. More specifically with the addressed concept, enabling to customize topics can also be an future direction .
 Atzori L, Iera A, Morabito G. The internet of things: A survey[J]. Computer networks, 2010, 54(15): 2787-2805.  Bellotti V, Back M, Edwards W K, et al. Making sense of sensing systems: five questions for designers and researchers[C]//Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 2002: 415-422.  Wilson, Robert A. and Foglia, Lucia, “Embodied Cognition”, The Stanford Encyclopedia of Philosophy (Spring 2017 Edition), Edward N. Zalta (ed.), forthcoming URL = <https://plato.stanford. edu/archives/spr2017/entries/embodied-cognition/>.  Frens J W J. Designing for rich interaction: Integrating form, interaction, and function[D]. Technische Universiteit Eindhoven, 2006.  Zimmerman J, Forlizzi J, Evenson S. Research through design as a method for interaction design research in HCI[C]//Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 2007: 493-502.  Simsarian K T. Take it to the next stage: the roles of role playing in the design process[C]//
CHI’03 extended abstracts on Human factors in computing systems. ACM, 2003: 1012-1013.
 Norman D A. Affordance, conventions, and design[J]. interactions, 1999, 6(3): 38-43.
 Weiser M. Some computer science issues in ubiquitous computing[J]. Communications of the ACM, 1993, 36(7): 75-84.
 Hummels C, Frens J. The reflective transformative design process[C]//CHI'09 Extended Abstracts on Human Factors in Computing Systems. ACM, 2009: 2655-2658.
 Mennicken S, Vermeulen J, Huang E M. From today’s augmented houses to tomorrow’s smart homes: new directions for home automation research[C]//Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 2014: 105-115.  Yang R, Newman M W. Learning from a learning thermostat: lessons for intelligent systems for the home[C]//Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, 2013: 93-102.  Limoncelli T. Automation should be like Iron Man, not Ultron[J]. Communications of the ACM, 2016, 59(3): 58-61.  Van Campenhout L, Frens J, Overbeeke K, et al. Physical interaction in a dematerialized world[J]. International Journal of Design, 2013, 7(1).
 Koreshoff T L, Leong T W, Robertson T. Approaching a human-centred Internet of Things[C]//Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration. ACM, 2013: 363-366.  Van Gent S H, Megens C, Peeters M M R, et al. Experiential design landscapes as a design tool for market research of disruptive intelligent systems[C]//1st Cambridge academic design management conference. 2011: 7-8. Tomico O, Frens J W, Overbeeke C J. Coreflection: user involvement for highly dynamic design processes[C]//CHI'09 Extended Abstracts on Human Factors in Computing Systems. ACM, 2009: 2695-2698.  Sanders E B N, Stappers P J. Co-creation and the new landscapes of design[J]. Co-design,
2008, 4(1): 5-18.
in the periphery of attention. [M] 2016.
 Design Kit: 2017. http://www.designkit.org/ resources/1. Accessed: 2017- 01- 12.
[ 2 4 ] L a m b e r t A , S p e n c e r E , M o h i n d ra N . Automaticity and the capture of attention by a peripheral display change[J]. Current Psychology, 1987, 6(2): 136-147.
 Harrison C, Horstman J, Hsieh G, et al. Unlocking the expressivity of point lights[C]// Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2012: 1683-1692.  Kelley, J. F. (1984). An iterative design methodology for user-friendly natural-language office information applications. ACM Transaction son Office Information Systems, 2, 26-41.  Fitts P M. Perceptual-motor skill learning[J]. Categories of human learning, 1964, 47: 381391.  Wensveen S A G, Djajadiningrat J P, Overbeeke C J. Interaction frogger: a design framework to couple action and function through feedback and feedforward[C]//Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques. ACM, 2004: 177-184.  Bakker S, Hausen D, Selker T. Peripheral ineraction: challenges and opportunities for HCI
 Sehgal R, Hardman J, Haney E. Observing trainee encounters using a oneâ€?way mirror[J]. The clinical teacher, 2014, 11(4): 247-250.s
Overall Design & Research Process I was initially a fan of smart home products myself, and this final master project gives me the opportunity to look deeper into the future IoT filed, addressing the challenges we will be confronted. With an ambition to transform the future world with design, I found this project really fits me. Thanks to the reflective transforamtive design process, I was able to manage design and research activities according to each reflection session. In the first semester, I failed to reflect sufficiently, so in the second semester I pushed myself to think more, and tried to write down the insights I gained from each activity (finally transferred to paragraphs of reflection in this report). As the second examiner suggested, it â€˜s better to leverage theoretical knowledge and design tools for this project, and try not to make the project too abstract but grounded with reality. I did a research on the approaches to design for the future in the first semester, and found several theories to frame my design better in the second semester. User-involved activities were conducted many times, in order to figure out the value of this project, and validate with the society (as the
two kinds of activities in the process: sensing, perceiving, doing and validating quality, detail can be found in design process). I also enjoyed the process of prototyping a lot, as it is an integration of my design skills(such as 3D modeling, rendering, wood crafting, programming) and the knowledge I gained from previous activities(such as research, co-creation, ideating, etc). Seeing my idea gradually realized brought me a lot of pleasure though sometimes I saw the sunrise at the same time. Therefore, one thing I am really doing bad is time management. As I reflected in final design, since making the prototypes with high quality took a long time, I sacrificed the possibility to validate the visualization design at an earlier stage. Fortunately, the whole process went similarly to my initial plan, only with several sleepless night. Development in Expertise Areas User & Society User and society is the expertise areas I focused on most during these two years. As for this final master project, I started from my vision and a challenge to improve the user-technology rela-
tionship with visualization of communication. During the process, the user-involved activities, such as role playing, co-creation, interview, were conducted at different stage of the process, helping to bring me back from an abstract level to the reality, and contributing to the direction of my design. In the second semester, I also integrate the theoretical knowledge of peripheral interaction, rich interaction, and Frogger interaction framework while I was defining the value of the system and designing the specific interactions, trying to approach the final result from different perspectives. The final evaluation, was definitely the activity I make use of the final design demonstrator to validate the quality I intended to provide with users. The collaborative research paper will be a contribution to the research field. Technology & Realization In this project, I learned new techniques (Raspberry Pi, OOCSI) to realize the function of connection. With the help of prior experience in programming with Processing and Arduino, both
physical and digital prototyping can be realized and iterated quickly. Though problems may occur at any time, which interrupted my design process for a while, they were eventually fixed and the final demonstrator was exactly what I meant to have. The digital control panel run on the laptop, was also upgraded for different purposes, such as demoday and Wizard of Oz evaluation. The latest version, also integrates the working mechanism of the system with certain calculations, minimizing the required input for real-time control. One thing I havenâ€™t really solved is the sensing data of the electronic slider, which caused unexpected flashing light. I already used an average of 50 reading data to do the calculation, but the problem still remained. According to my knowledge, itâ€™s because of the error of the slider itself. Creativity & Aesthetics During the two semesters, I explored valuable scenarios by leveraging relevant design techniques. As also mentioned in user & society, the user-involved ideation made my project grounded with the reality, and address their needs. Theoretical knowledge, such as interaction frame-
work, was also used for improving the quality of the final design, with integration of form, interaction and function. Aesthetics of the prototype itself was also considered a lot in the final design phase. The final video and poster followed the same style as the prototypes, to create a consistent experience for the visitors. Vision & Identity After last semester, I have developed my vision to a broader meaning, designing for coexistence (PDP can be found in appendix). The coexistence, can be created for multiple technologies to provide a seamless shift for users to experience with; for establishing a better relationship between user and technology by building up adequate understanding for each other; for supporting multiple users to live in harmony with each other. In the first semester, I tried to address all the three kinds of coexistence. However, I didnâ€™t find an appropriate scenario in this project. Therefore in the latter semester, I tried to approach the second kind of coexistence with a solution to make the system components in a more balanced relation. During the whole semester, my vision
guided me to do all these activities. This can be reflected in the concept, the interaction design, and even the unitive style of the prototypes. The evaluation session can also be regarded as validation of my vision. In this project, I positioned myself as an interactive system designer, and user experience researcher. Therefore I was always trying to make use of my knowledge to deliver quality in user experience through the design artifacts, and also exploring possibility for new values from a research perspective. With the RDD track, previously I would like to focus more on the design and development. After finishing this research oriented project, I found myself also interested in exploring value for the future world by using research through design approach. I would probably shift my identity a little bit more to a researcher in the future.
First of all, I would like to thank my FMP coach, Mathias Funk, for bringing me to a future world of IoT, inspiring me at the moments I got stuck in the process, providing technical assistance for any kinds of prototyping problems, and guiding me to do this research oriented final master project. I would also like to thank Lin-Lin Chen and Yaliang Chuang as research experts, giving me fresh ideas from another perspective, and practical advices for research methods and procedure. Sincere thanks also to my testing participants who spent such a long time staying in a quite closed space and being observed. I would like to thank the fellow students, Qi Qi, Wenhao Wu, Jiachun Du, Mantas Palaima, Dennis, for providing me any kind of help for my FMP. Special thanks to Xihao Hu, for collaborating on this project, and inviting me for dinner many times during the whole year. In the end, I want to thank my family and all my friends all over the world, supporting me to go further on my way and being accompanied with me all the time.
PDP personal development plan m22
COACH Mathias Funk STUDENT Shuai Shao S152106 firstname.lastname@example.org
During the last final master project preparation semester, my vision has been further developed. The future of the world to me is about coexistence, while the future of design, in my point of view, will be designing for the coexistence. Coexistence of multiple media We have witnessed a large variety of technologies emerging in the past few years, and some of them have already replaced traditional products to make our life more convenient, or joyful. As mobile phones have been developed really advanced, mobile applications are the most influential things in our life, however, the digital world could not live without the actual physical world, where users are already familiar with. Thus, it is important to make the digital and physical, the actual objects and the cloud system, and all these technologies to live together with each other smoothly. For example, an IoT system, containing a series of objects working connectedly along with the internet and perhaps the cloud server, needs to perform collaboration and have the intelligence to react to emergency or unset situations instead of creating chaos. Coexistence of technology and human As mentioned above, the new world full of technologies only emerges to daily life in recent years, requiring a higher bar for users to enter in than the traditional world that users are already living in. It’s not possible to let users know every detail of technology principles behind the products they buy. This often makes users, especially the elderly, confused about how to use them. Moreover, with the complex technologies, users are often confronted with multiple choices where they couldn’t easily make a decision. Although sometimes the technology performs learning intelligence towards human’s behaviors, they only work according to the algorithm, which definitely can’t understand other emotional or unpredictable needs of users. Therefore, the role of technology, is never going to replace human, but work as a companion to live with human and make them understand each other. Coexistence of multiple users The reality is full of complexity and uncertainty that programming is not likely to achieve. People are also never like the personas that designers created for designing certain products and they are mostly have their own strategies to deal with their lives. It is a constant problem towards multi-user preference in daily life while designers are also struggling with this issue. With this concern, design is meant to provide flexible solutions for multiple users involved in addition to the functions it should have, but not creating a break point for them while using it. After the advanced technology appears, people are more and more far away from each other, which is not how they are supposed to be. So in the future, technology or design is not something standing between person and person, but involve them in a mixed reality. My current vision seems even broader than last semester, which is because I am heading to my career life after I complete my FMP, where I would like to keep the opportunity to accept a large variety of new stuffs. However, this vision is the constant direction to go for my upcoming design career.
identity After graduating from my bachelor university in China and studying at TUe for one and a half years, I have gradually shifted my identity from a digital interaction designer to an interactive system designer. In my previous electives, I learned different theories and design approaches that helped me do research and design more reasonable and structured. I learned to design with complex sensors (leap motion, sensor components) and softwares (processing, Arduino), together with traditional design skills to design interactive products located within a smart home or in public, with the purpose of solving daily problems or communicating information (energy consumption, environmental impact, working status of a system, etc). In these projects, I was able to learn and apply knowledges during the process, for example, interaction design framework from Bellotti , embodied cognition , rich interaction , and data visualization. As most of them are introducing new technologies to the user, I always followed a user-centered design approach, but also fully understood how the system itself should work. I am also familiar with research through design  approach to conduct a field research, and methods to analyze both quantitative and qualitative data. In the current project, I am using a reflective transformative design process, which is good to use when I need to design for the unknown future, and will probably be the approach that I will use since I am really interested in the IoT and future home topics. Within this project, I also use tools like role playing  to involve users in the design process, making sure the project doesnâ€™t go far from the real users. I am also trying to bring the theories to some part of my project if applicable. In the future, I would like to make use of what I learned in the two years and continue working within IoT, or information technology industries as an interaction designer with a wide perspective of the user, system, and academic theories. Reference  Bellotti V, Back M, Edwards W K, et al. Making sense of sensing systems: five questions for designers and researchers[C]//Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 2002: 415-422.  Wilson, Robert A. and Foglia, Lucia, "Embodied Cognition", The Stanford Encyclopedia of Philosophy (Spring 2017 Edition), Edward N. Zalta (ed.), forthcoming URL = <https://plato.stanford.edu/archives/ spr2017/entries/embodied-cognition/>.  Frens J W J. Designing for rich interaction: Integrating form, interaction, and function[D]. Technische Universiteit Eindhoven, 2006.  Zimmerman J, Forlizzi J, Evenson S. Research through design as a method for interaction design research in HCI[C]//Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 2007: 493-502.  Simsarian K T. Take it to the next stage: the roles of role playing in the design process[C]//CHI'03 extended abstracts on Human factors in computing systems. ACM, 2003: 1012-1013.
The goals are set for these semester, in which I will mainly finish my final master project. Therefore, the goals are mainly about the project but also mentioning some personal competency development. From abstract to specific From the feedback of last semester, what I did was too abstract with the diagrams, patterns and the final visualization design, and the final goal I set for this project was also too ambitious that I am not going to achieve in this short period of time. Taking these feedback into consideration, I will try to bring the design process back to the reality as much as possible and narrow down the context. In this way I will need to involve real users and do the testing in real context instead of showing abstract visualization and scenarios. I will still use food as the main topic to explore more potential scenarios, however, I will design for only one or two of them deeply and thoroughly. From visualization to interaction Last semester I categorized several communication patterns and designed lighting visualizations for them to show how the system works to users. As the second examiner suggested, the interaction part would be more interesting and I need to focus to avoid getting lost. I will take the communication patterns as the basis, and then figure out interaction patterns on top of it and design for them. During the interaction design process, I will need to figure out why and what the users want to interact or communicate with the system and how they would like to do that in certain scenario. Since now the objects are not able to interact with, I will also build one interactive interface within the system and improve the interaction feedforward and feedback through the design process and probably referring to some design theories or approaches. Reflect and reorganize academic results Last semester I failed to reflecting on my project sufficiently. As I am using the reflective transformative design process for my FMP, I actually have multiple chances to reflect on each design or research activity. Therefore, this semester I will force myself to think more and deeper and write it down. Besides, I need to reorganize what I did in the past and make a plan based on that. Apart from my FMP, I will also reorganize everything I learned or designed in this two years, and make a good documentation. Turn collaboration into outcomes One of the most important feedback was that the collaboration with Xihao wasn't that since we appeared to have similar results at the end of last semester. Firstly, to avoid this situation, I will clarify what and how we collaborate in the final report. As for our projects, we had the same starting point, and at the early stage we chose to use the same approach to define the objects, and in the following process, we always discuss and meet together to decide the next step. That's why our projects look so similar to others though we truly did each step separately. This semester we are going to focus on a different direction for our final design. In order to make our collaboration meaningful, we discussed and decide to write a research paper in the end, and our projects can be used as two branch design cases of this research. We hope we can carry out some meaningful research results in addition to our design projects.
activity Finish FMP As the last semester of my master program, itâ€™s the most important thing to complete the design process and finish my final master project. Therefore I will absolutely put most time on the project in order to carry out sufficient outcome. More detailedly, with the goal from abstract to specific, I will start with user research about interaction situations and contents they potentially have with each object in the system. With that input, I will further define the role of each object and complete scenarios with focus on interaction. Quick prototyping will be followed to make multiple solutions for interactions, with which I am going to do a simple user test. All the above activities also make my project from visualization to interaction, but visualization will still be a part of interaction. After a iteration the final design will be decided and I will make one or two of them fully interactive with technology. During the design process, literature reading will happen at any time to make each step reasonable and reflections will be taken down after each step. With the final design I am going to conduct a structured experiment according to the plan for the research part. That's the important step to turn my work and the collaboration with Xihao into a meaningful result. After all I am going to write a individual FMP report and a collaborative research papaer in addition, turning all the process and results into documents.
Testing Protocol Introduction !-**=5*7.2*39.8&.2*)949*899-*)*8.,34+9-.8+.3&12&89*7574/*(9&3)2&.31>(438.8984+ 9<45&798*+47*>4:&7*941)9-*,*3*7&1.)*&4+9-.8)*8.,3>4:<.113**)94(4251*9*9-* +.7895&79<-.(-.8.39*3)*)94,&9-*7>4:7.39:.9.;*9-4:,-98!-*39-*7*8*&7(-*7<.119*11>4: &'4:99-*)*8.,3.39*39.43&3)-4<9-*8>89*2<470839-*8*(43)5&79>4:<.113**)94 *=5*7.*3(* 9-* 8>89*2 +47 2.3:9*8 ):7.3, <-.(- >4: (&3 )4 &3>9-.3, >4: <&39 .3 9-* 7442!-*7*8*&7(-*7<.11.39*77:59>4:9-7**9.2*8.39-* 2.3:9*8+9*7&11>4:<.113**) 94+.11.3&6:*89.433&.7*&3)-&;*&3.39*7;.*<39-*+.7895&79&:).4<.11'*7*(47)*)<-.1* .39-*8*(43)5&799-*<-41* 2.3:9*8<.11'*+.12*)!-*8*)4(:2*398<.11'*9&0*394 +&(.1.9&9*9-*(411*(9.434+.3+472&9.43$4:(&38945&3)6:.99-*9*89&9&3>9.2* + >4: :3)*789&3) 9-* 574(*):7* 4+ 9-* *=5*7.2*39 &3) &,7** >4:7 5*7843&1 .3+472&9.43 &:).4;.)*494'*:8*)+47.11:897&9.3,9-*574/*(9)4(:2*39&9.4351*&8*8.,39-*+4114<.3, .3+472*)(438*39+47>4:75&79.(.5&9.43
Informed Consent !-*.3+472&9.43&3)5-494,&9-*7*)+7429-.89*89.84<3*)'>9-*!"*!-*.3+472&9.43 &3);.)*4<.11431>'*:8*)+47574/*(97*1&9*)5:7548*>8.,3.3,9-.8)4(:2*39>4:<.11 )*(1&7*9-&99-*!"*2&>:8*9-*.3+472&9.43+7429-*9*89+478(-4417*1&9*)5:7548*$4:
Make documentation of my projects In addition to finishing FMP, I also plan to make documentation of all the projects I did in these two years, including the research project, the elective projects, and the first part of my FMP. With these projects, I will try to reorganize every material and outcome and try to transfer some of them into more academic reports or papers. This is because I would probably apply for Phd after a few years and these would be the documents required for application.
-&;*7.,-9947*/*(99-.8*=5*7.2*39 &,7**94-&;*9-.8:8*789:)>7*(47)*) &2*%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Make a portfolio/showcase website
Similar to the last activity, I will also make my projects to be presentable and understandable by reorganizing everything I had and make some visual things to explain the process and the final results briefly. I will transfer the collection of these projects into a website that I can use as my portfolio, and probably another version to be my showcase mentioning more about learning process.
Seek for career opportunities After finishing this semester, I will be at the point to work somewhere although most likely back to China. I donâ€™t want to have a long gap so I will try to seek for working opportunities all over the world and this is also why I need to make a portfolio this semester.
'*-&;.47 &3) >4: 3**) 94 7&9* 9-*2 '> -4< 2:(- )4 >4: 9-.30 9-*> 2&9(- 9-* )*8.,3
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%% 9743,1>).8&,7** 9743,1>&,7** %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%% 9743,1>).8&,7** 9743,1>&,7** %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
(440.3,8-455.3,<&89.3,!-*7*.843*2&89*74'/*(99-&9>4:(&38*9:5>4:7,4&18 &3)-&;*&34;*7;.*<4+9-*8(47*84+>4:7'*-&;.47843+4:7&85*(98!-*49-*7+4:74'/*(98 7*57*8*39 8:89&.3&'.1.9>" -*&19- -&'.9!" &3) 51*&8:7* !! $ 7*85*(9.;*1> !-* 8>89*2 (&3 8*38* :8*7@8 '*-&;.478 &3) ,.;* & 8:,,*89.43 '> (422:3.(&9.3,<.9-*&(-49-*7!-*(422:3.(&9.43&3)8:,,*89.43<.11'*;.8:&1.?*)&89-* 1.,-9'*-&;.4789-&9>4:-&;*&17*&)>8**3.35&79>8*99.3,>4:7,4&18439-*8*+4:7&85*(98 <.9-*&(-4'/*(9945.(<.11-&;*&(477*8543).3,<*.,-9.39-.88>89*2<-.(-2&0*8 9-*.71.,-9'7.,-9*747).22*7$4:3**)94574;.)*9-*7*8*&7(-*7>4:7.3.9.&18*99.3,;&1:*8 439-*+4:7&85*(98&3)>4:(&3&)/:89.9&9&3>9.2*'>.39*7&(9.3,<.9-9-*4'/*(98):7.3, 9-**=5*7.2*39!-*+4114<.3,.8842*.3897:(9.438+479-*,*3*7&1.)*&4+;.8:&1.?&9.43&3) .39*7&(9.43+9-*7*.8&3>9-.3,:3(1*&7+47>4:>4:&7*<*1(42*)94&80&9&3>9.2*
;*7>).++.(:19 ;*7>*&8> " ;*7>).++.(:19 ;*7>*&8> ;*7>).++.(:19 ;*7>*&8> !" ;*7>).++.(:19 ;*7>*&8> !!$ ;*7>).++.(:19 ;*7>*&8> 4:1)>4:*&8.1>&884(.&9*9-*+472949-*.39*3)*).39*7&(9.43
;*7>).++.(:19 ;*7>*&8> " ;*7>).++.(:19 ;*7>*&8> ;*7>).++.(:19 ;*7>*&8> !" ;*7>).++.(:19 ;*7>*&8> !!$ ;*7>).++.(:19 ;*7>*&8>
4:1) >4: *&8.1> &884(.&9* 9-* 1.,-9 '*-&;.47 7*6:*89 94 9-* .39*7&(9.43 )*8.7*) '> 9-* 4'/*(9
" ;*7>).++.(:19 ;*7>*&8>
*+47*9-*9&80889&79>4:3**)94+.11.39-*+4114<.3,6:*89.433&.7* 1*&8*,.;*>4:7.3.9.&18*99.3,;&1:*843+4:7&85*(98.3 54.398(&1* :89&.3&'.1.9> -*&19- -&'.9 51*&8:7*
1*&8* *=5*7.*3(* <.9- 9-* 8>89*2 +47 2.3:9*8 &3) ):7.3, 9-.8 5*7.4) >4: (&3 )4 <-&9*;*7>4:<&39.39-.87442!-*7*8*&7(-*7<.11574;.)*>4:<.9-842*+44)9-7** 9.2*8 .3 9-* 2.3:9*8 $4: )43@9 3**) 94 7*&11> +4(:8 43 9-* 4'/*(98 !-* <-41* 574(*88<.11'*4'8*7;*)&3)+.12*) 4. !-*7*8*&7(-*7<.117*51&>9-*;.)*494,*9-*7<.9->4:&3)&99-*2*&39.2*>4:<.11'* .39*7;.*<*)
;*7>).++.(:19 ;*7>*&8> !" ;*7>).++.(:19 ;*7>*&8> !!$ ;*7>).++.(:19 ;*7>*&8>
Post-task Interview 5*7(*.;*)*++479:3)*789&3).3,'*-&;.47&1(-&3,*7*+1*(9.43.39*7&(9.43&((*59&3(* #-&9).)>4:8**<-.1*>4:<*7*574;.)*)<.9-842*+44) #-&9).)>4:9-.309-*8>89*2.897>.3,949*11>4:
sensing & recording
SENSING & RECOGNIZING
DATA UPLOADING AND CALLING BACK
GROCERY SHOPPING CONNECTING TO MOBILE PHONE
C OMMUNICATION PATTERNS
Translated as brightness
Translated as color and frequency
Translated as light behaviors
Initially set up by the user with PIKA and is able to be adjusted later on with either object in the system
Each object has their own attitude towards certain behaviors
Divided into two sets of lights to separate messages to users and to objects
Indicates how important the user wants the topic to be and how much it will influence decision-making
Can be influenced by the fact and the historical user behaviors Translated in five-point scale
The movements of light behaviors visualize the direction of communication iconically Adjusted to fit in different objects but follow the same principal
D AILY ROUTINE
!-**=5*7.2*39.897>.3,94,.;*>4:&+**1.3,4+1.;.3,<.9-9-.88>89*2&3>4:.2&,.3* .+.9.8&551.*).37*&1(439*=9&3)*;*7>+:3(9.43.87*&1.?*)9-&99-*8>89*2(4:1)-*15 <.9-&'&1&3(*)1.+*89>1*
The system needs user to do initial setup by setting their goals on four aspects with PIKA. Itâ€™s also possible to adjust this setting value at any time they want.
For example, KUBO get the food storage data from the smart refridgerator that itâ€™s really full and a lot of food is going to expire very soon. So KUBO requests its weight to be increased among the system.
Userâ€™s food relevant behavior triggers the system to vote a most reasonable suggestion according to the initial settings. After the result is shown to the user, he can decide to follow it, ignore it, or adjust the system. Either reaction will be recorded and evaluated.
PIKA tries to attract usersâ€™ attention when they are walking by. The evaluation result will immediately displayed as a changing pattern, which would probably be different from their setting pattern. The difference can be an opportunity for people to reflect on their previous behaviors.
Codes & others
 Prototype codes. https://www.dropbox.com/sh/rcqkylvyp1dzjhw/ AACwlKkjqwTfDUbNukswrLONa?dl=0  Report of the first semester. https://www.dropbox.com/s/k85nhxtyceinm00/FMPprepReport.pdf?dl=0  Raw data of evaluation. https://www.dropbox.com/sh/4oq9r2wc7lqx1k9/AADWGRGw_awu4OLPisX-vq8ta?dl=0