D3.2 - Models for human/machine symbiosis: final results

SmartSociety Hybrid and Diversity-Aware Collective Adaptive Systems When People Meet Machines to Build a Smarter Society Grant Agreement No. 600854

Deliverable D3.2 (Work package WP3)

Models for human/machine symbiosis: final results

Dissemination level (Confidentiality)1:

PU

Delivery date in Annex I:

30. Aug 2014

Actual delivery date:

13. Oct 2014

Status2:

Final

Total number of pages:

24 (excluding pages before the Table of Contents, annexes and references)

Keywords:

Collective Adaptive Systems, Human-Machine Communication

1

PU: Public; RE: Restricted to Group; PP: Restricted to Programme; CO: Consortium Confidential as specified in the Grant Agreement 2 F: Final; D: Draft; RD: Revised Draft

Deliverable D3.2

Š SmartSociety Consortium 2013 - 2017

Disclaimer This document contains material, which is the copyright of SmartSociety Consortium parties, and no copying or distributing, in any form or by any means, is allowed without the prior written agreement of the owner of the property rights. The commercial use of any information contained in this document may require a license from the proprietor of that information.

Neither the SmartSociety Consortium as a whole, nor a certain party of the SmartSociety Consortium warrant that the information contained in this document is suitable for use, nor that the use of the information is free from risk, and accepts no liability for loss or damage suffered by any person using this information.

This document reflects only the authors’ view. The European Community is not liable for any use that may be made of the information contained herein.

Full project title:

SmartSociety - Hybrid and Diversity-Aware Collective Adaptive Systems: When People Meet Machines to Build a Smarter Society

Project Acronym:

SmartSociety

Grant Agreement Number:

600854

Number and title of work package: Document title:

[WP3], Human/Machine Symbiosis Models for human/machine symbiosis: final results

Work-package leader:

Paul Lukowicz, DFKI

Deliverable owner: Quality Assessor:

George Kampis, DFKI

Page 2 of (29)

Kobi Gal, BGU

http://www.smart-society-project.eu/

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

List of contributors Partner Acronym DFKI UNITN UOXF KAU

Contributor George Kampis, Agnes Grünerbl Mattia Zeni, Enrico Bignotti Mark Hartswood, Marina Jirotka Leonardo Martucci, Simone Fischer-Hübner

© SmartSociety Consortium 2013 - 2017

Page 3 of (29)

© SmartSociety Consortium 2013 - 2017

Deliverable D3.2

Executive summary This Deliverable D3.2 of WP3 continues and concludes D3.1 of PY1. In WP3 the active task bw. M1-M18 was T3.1, which is about “Models of Human-Machine Symbiosis”.

This is the final report on T3.1, presenting actual achievements in terms of models, in particular of computational models developed and tested in the form of agent based simulations. These works, ones on collaborative location and knowledge fusion have been printed in multiple publications including a journal paper (see references).

D3.1 has provided an overview and outlined important historical approaches as well as our own methodology. This methodology was identified as having two pillars: (i) the “semantic gap” problem and (ii) the problem of seamless, new kinds of collaboration in man-machine systems.

Accordingly, the current summary document D3.2 presents these parts together, in Sections 2-6 and 7, respectively. We note that the proportions do not necessarily reflect the significance attached to the respective issues. For example, self-organizing interactions leading to emergent coordination and “smart” behaviour is considered the basis of a possible new paradigm for human-machine systems and CASs in particular. Nevertheless, research on this topic is experimental and (at the moment) purely theoretical, whereas treatment of the semantic gap bears on practical applications including Smart Society scenarios such as the RideSharing scenario or the emerging Hospital use case.

We proceed by first recalling results from D3.1 and defining the Semantic Gap problem, then dealing with its social and ethical issues as well as subsequently the technical approach developed by UNITN and DFKI together, applied to the RideSharing use case. In the final Section of the text we provide an overview of work done on the collaborative localization and collaborative fusion problem in WP3.

Follow up works will next continue on T3.2: Context and Intention Interpretation [M19-M30] as well as on T3.3. Human-Machine Composition [M25-M36] and T3.4 Interaction Patterns and Persuasive Technology [M25-M36] which will bear upon the results of T3.1 and work commenced on T3.2.

Page 4 of (29)

http://www.smart-society-project.eu/

Deliverable D3.2

Š SmartSociety Consortium 2013 - 2017

Table of Contents Table of Contents .................................................................................................................................................5 1. Introduction ...................................................................................................................................................6 2. The Semantic Gap Problem ...........................................................................................................................6 3. Activity recognition, semantics and ethics ....................................................................................................7 3.1 Social values and activity recognition .........................................................................................................7 3.1.1 Commodity infrastructure components and personal digital ecosystems .............................................7 3.1.2 The Activity Recognition (AR) pipeline...............................................................................................8 3.1.3 Activity recognition phases ..................................................................................................................9 3.1.4 Application to the SmartShare scenario ................................................................................................9 3.1.5 Autonomy and control ........................................................................................................................10 3.1.6 Philosophical debates ..........................................................................................................................10 4. The Hospital Use Case.................................................................................................................................11 5. Bridging the Semantic Gap..........................................................................................................................11 5.1 The System Architecture ...........................................................................................................................12 5.1.1. The Front-end ....................................................................................................................................12 5.1.2 Back-end .............................................................................................................................................15 5.1.3 Cassandra-based persistence storage ..................................................................................................15 5.1.4 The Knowledge Base ..........................................................................................................................16 5.2 A 3-layer approach for semantic-driven sensor fusion ..............................................................................17 5.3 High Level Models ....................................................................................................................................17 5.3.1 The Mainkofen Hospital Use Case .....................................................................................................17 5.3.1.1. Room ...........................................................................................................................................18 5.3.1.2. Person ..........................................................................................................................................18 5.3.1.3. Activities and Events ...................................................................................................................18 5.3.2 The i-Log Use Case ............................................................................................................................19 6. The RideShare Use Case .............................................................................................................................19 7. Incremental Knowledge Fusion ...................................................................................................................20 7.1 Incremental fusion .....................................................................................................................................20 7.2 How the model works ................................................................................................................................20 7.3 Two Model Variants and Their Results.....................................................................................................21 7.3.1 Model version 1: the averaging of knowledge upon meetings ...........................................................21 7.3.1.1 Explanation of observed behavior ...................................................................................................21 7.3.1.2 Parameter sweep results ...................................................................................................................22 7.3.2 Model version 2: experience as reputation .........................................................................................23 7.3.2.1 Explanation of observed behavior ...................................................................................................23 7.3.2.2 Parameter sweep results ...................................................................................................................23 7.4 Current work..............................................................................................................................................23 References ...........................................................................................................................................................25 Annex A High-level modelled entities: Mainkofen scenario .......................................................................26 Room............................................................................................................................................................26 Person ..........................................................................................................................................................26 Event ............................................................................................................................................................26 Activity ........................................................................................................................................................27 Step ..............................................................................................................................................................27 Annex B High-level modelled entities: Ride-share scenario ......................................................................28 Device ..........................................................................................................................................................28 Accelerometer ..............................................................................................................................................28 GPS ..............................................................................................................................................................28 Bluetooth......................................................................................................................................................29

Š SmartSociety Consortium 2013 - 2017

Page 5 of (29)

© SmartSociety Consortium 2013 - 2017

Deliverable D3.2

1. Introduction This document summarizes effort towards a comprehensive theoretical understanding of symbiotic humanmachine systems. In the preceding Deliverable D3.1 we have outlined the essentials of our approach, which has two main pillars.

One of the key problems as identified in D3.1, and approached in the current document, is that of the “semantic gap”, which is the “no-man’s-land” between the fast, reliable, and low-level machine tagging and the slow, sometimes error-prone, high-level human meaning. Our results on this segment will be seamlessly integrated with other WPs of the project later, serving them from the sensor end.

The other problem of considerable interest – especially in the CAS context where the current project is located – is that of collective interaction and decision-making in heterogeneous human-machine systems where emergent, dynamic coordination is expected. Tackling this problem constitutes a “theoretical leg” of the project and explores a “high end” where notions of central planning, allocation and coordination are abandoned (notions which are demonstrated in the Scenarios, in particular the Ride Sharing Scenario, which DFKI is supporting with a front-end, as well with user studies using TU KL students). Of interest is what kind of coordinated intelligent (or even “smart”) behaviour can be achieved in systems without prior coordination, that is, from “scratch” during the dynamic execution of ad hoc interactions between human and machine agents.

Our report presents work done in these two aspects of the study of symbiotic human-machine systems.

2. The Semantic Gap Problem An excerpt from D3.1 introduces the Semantic Gap as follows. <We begin with a large amount of raw sensing information collected by field sensors deployed in local systems, such as embedded in smartphones or other portable devices (recording physical, physiological, environmental of other kinds of information). Raw information is converted to useable data typically offline and after the conversion, often forming a database.

Data items may already come structured (i.e. equipped with metadata) but typically require additional interpretation and aggregation at various levels, using event identification and reasoning. This is leading finally to a “high” level semantic/knowledge representation in terms of meaningful events and human behaviour>.

How to link or “bridge” these levels transparently, how far “up” on this hierarchy can we go using machine tools? These questions constitute the semantic gap problem, which is thus closely linked to sensing, activity recognition, automatic annotation/tagging, machine learning, computational semantics, etc.

Figure 1. A hierarchy of activities as illustrated on the Mainkofen Hospital example (taken from D3.1). The semantic gap problem is constituted by the question as to how to proceed from one level to another.

Page 6 of (29)

http://www.smart-society-project.eu/

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

Altogether 6 layers of the hierarchy have been identified in (Bahle et al. 2011) which reflect the progression from sensing to semantics, explicated in Figure 1. The semantic gap is now the gap between these levels or layers, and problem is how to “link” them, i.e. how to proceed from the one to the other.

3. Activity recognition, semantics and ethics High level categorizing, tagging, and labelling, which are all essential to the Semantic Gap problem, generate a broader context, the world of social values and ethics problems. On Sept 20 we held a WP1-WP3 joint meeting in Frankfurt to discuss these issues and embed them in a “design for ethics” methodological framework. The current Section emerged from these discussions, and brings excerpts form the resulting documents. Other issues were left to subsequent developments involving further WPs. As a general summary of this Section: although there may be differing perspectives on what is the right way forward for privacy principles and legislation in handling semantic labels (such as those generated when tackling the Semantic Gap), this does not alter the regulatory environment that we are bound by now. So we need to follow existing rules and best practices. However, given the special social responsibility and resources of SmartSociety, we may also ambition to provide recommendations. Work is currently under way in this respect and indeed the text below indicates some of the details.

3.1 Social values and activity recognition 3.1.1 Commodity infrastructure components and personal digital ecosystems We first consider or even question the role of commodity infrastructure components such as SmartPhones, GoogleGlass, wearables etc. to provide sensor data for SmartSociety applications, particularly how these might create unwanted data flows to Google, Apple etc. and in general the hosts or hubs of the relevant applications using the sensors and related components.

In an institutional context, these issues may need to be handled explicitly, for example, to make sure systems comply with policies in highly regulated settings such as hospitals. Care is also needed here in capture of ambient information that may include others and also personal care information.

We suggest that, importantly, in personal settings the items such as phones, wearables etc. all form part an extended “personal digital ecosystem” (PDE) - an assemblage of devices (phones, wearables, laptops, pcs, settop boxes, augmented 'things' etc.) and services (social media, media services, cloud services etc) centred upon individuals (but with complexities where elements of these are shared) and comprised of a continually evolving suite of components and configurations. This is very close to Steve Sawyer's notion of 'digital assemblages' (Sawyer, 2014), or Carroll's notion of 'Personal Information Infrastructures' (Carroll, 2006).

PDEs are heterogeneous systems with varying levels of control possible over components and emergent properties.

This leads us to the following issues:  SmartSociety needs to consider how to interface with PDE's rather than with individual devices (e.g. phones). From a WP3 perspective, one might think about a SmartSociety driver (thinking of the RideSharing Scenario) that sits within a person's PDE giving access to his 'virtual sensors'.  The new kinds of entanglements created when SmartSociety applications enter a person's PDE pose a series of interesting questions. While SmartSociety applications might take advantage of a person's use of certain devices or services - then exactly what responsibilities does SmartSociety assume if those elements of PDE may not always operate in that person's interests? o Privacy principles of informed consent and data subject rights need to be enforced – both for users and uses (non-users). Data may relate to several persons with different control interests in regard to the (entangled) data. o On the one hand, it seems unreasonable that SmartSociety takes responsibility for the myriad of risks attached to use of Google, Facebook etc. SmartSociety can't fix the world. But we do know that people do access digital services without always being fully aware of the consequences and risks, which may become an issue particularly when the use of those services by SmartSociety applications may be seen as a tacit endorsement, or tacit approval – particularly problematic if the SmartSociety itself claims high ethical standards. On the one hand, one wants to respect the individual autonomy - yet on the other hand, not hide behind this as a way of absolving one's own responsibilities. © SmartSociety Consortium 2013 - 2017

Page 7 of (29)

© SmartSociety Consortium 2013 - 2017

Deliverable D3.2

o A clearer case is where SmartSociety applications more directly influence a person's choices about their PDE (e.g. "you need to turn on location services to get these features...") - particularly where there are strong incentives to use a SmartSociety application. In this case then there is a clear case to inform people about the risks entailed by introducing a new element to their PDE. o This is all exceptionally interesting from a responsibility perspective as it provides a powerful example of entanglement between responsibilities and the difficulties of resolving these entanglements. o In response to the privacy principle of minimising data held, one might be tempted to argue that elements of a person's PDE already capture and store vast amounts of data, including data SmartSociety applications are interested in, so data minimisation has little value for the user. But one has to be careful about this argument for more than one reasons:  Some components may collect data covertly or in a way that the owner of the PDE cannot prevent.

Data generated by in-car systems might be one such example. Thus just because data is collected from within a PDE does not necessarily imply that the owner of the PDE controls or consents that data collection.  That the owner of a PDE is exposed to risks due to data collected and stored does not imply this exhausts the risks of collecting and storing further data.  If SmartSociety were to be publically criticised in the media for excessive capture and retention of data, then it would not relieve the situation to claim that this is OK because others do likewise.

3.1.2 The Activity Recognition (AR) pipeline Important ethical and privacy questions for Activity Recognition stem from how AR is implemented, in particular what the data flows are and where processing occurs. The meeting revealed that AR might consist of several steps or iterations involving data at different levels of granularity / semantic abstraction. The initial step involves making deductions from raw sensor data and storing those abstractions via a process of “temporal aggregation”. This involves a “sliding window” of data which is processed and discarded as sensor readings are converted from a time series of measurements to 'higher level' representations. Further rounds of analysis might then process this data into even higher semantic categories according to this type of hierarchy: 1) Sensor data 2) Processed sensor data The above steps may well happen within the PDE (e.g. on a device, or via a web service) - and then passing data 'out'3 into the/a SmartSociety platform – e.g. into the Knowledge Base that lies behind the Peer Manager.

This can be thought of as a “virtual sensor” with differing degrees of processing capability. Retaining this transformation within the PDE has advantages for SmartSociety applications in that raw data is never stored, but only processed.

3) Action (compound movements) 4) Activity (e.g. brushing teeth) 5) Habits (e.g. routines of brushing teeth, travel, etc.)

Issues can be raised about the different risks of having data in these different forms.  From one perspective then, throwing away lower level data in favour of higher semantic categories is privacy enhancing, as information is lost at this stage. This is because retained sensor data could be reanalysed to reveal new things about a person. The capability to perform this sort of analysis is now quite widespread. Throwing the sensor data away prevents this type of reanalysis.  From a different perspective, however, the derived higher-level semantic categories may also be a source of additional risk for the individual. This is because it does provide a more visible or accessible account of what the person was doing - i.e. one that can be read without needing to do an analysis of low level data. It may also be produced in combination with other data held by SmartSociety and reveal habits not easily available from raw data alone. Moreover there is a strong risk of this analysis being read out of context without the caveats that apply with an understanding of the limitations of a machine interpretation4.

3

Depending on where the boundary of the PDE is drawn. Perhaps SmartSociety may view itself as connected to but separate from a PDE - connected through a well-defined API. Yet someone in Google may think the boundaries are drawn differently - that SmartSociety is within the PDE, and Google is connected through a series of interfaces... How should we think about these boundaries?

4 In the workshop an example was used to illustrate. Sensors may record that a person is accelerating, yet that person may be standing still, if for example they are on an aircraft.

Page 8 of (29)

http://www.smart-society-project.eu/

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

 However, the raw sensor data allow different types of possibly unforeseen analyses for different purposes.

For the data subjects it may not be clear what types of information may be derived. (e.g. location data reveals not only geographic traces, but also information about the user’s activities, social contacts.

Moreover, it may even show whether drivers are speeding – or if someone drives very slowly through a red-light district, which may be interpreted in a certain sense.  For these reasons, raw sensor data including location data are always quite sensitive. It will therefore be best if they can be kept on a user’s personal device under his/her control.

3.1.3 Activity recognition phases Different sorts of hierarchy or schemas should be discussed to understand the different capabilities of activity recognition.

Across a semantic spectrum these are: Sense data (time series data), Action, Activity, Agency (values, intentions, etc.), essentially as in Fig 1.

Across a functional spectrum: Sensor data. Activity: Activity recognition is not so good at identifying a specific activity, and always does so only statistically, i.e. with some accuracy and reliability. Thus the requirement on the nurses to “swipe away” misrecognised activities in the care case-study prior to their inclusion in the medical record. A social computation is needed for a more accurate reconstruction of activity?

Habits: Coupling data from activity recognition with deep mining of other profile data helps create a semantic representation in the computer of patterns of activity or habits. This can be used as a powerful tool for understanding preferences to drive more accurate recommender systems. Thus understanding that a journey is for a work-related meeting, and not just from geographical point x to geographical point y - enables contextappropriate suggestions, e.g. based in this case perhaps around punctuality being important component of the trip.

A further elaboration to this spectrum is provided below.

3.1.4 Application to the SmartShare scenario To make our discussion more concrete we next explore how AR might be made relevant to the SmartShare application / scenario. The following possibilities can be considered.  Accepted or completed the ride. That the lift was on time. Route appropriate. Initially it was felt that this was a less interesting and fairly trivial challenge for AR, but with further thought, making the inferences that patterns of movement were in fact due to the ride are more challenging that they first seem. There may be reasons for false positives - e.g. making the journey but not using the proffered ride - or false negatives e.g. the ride was hastily rescheduled or the route changed, or the mobile device left behind or switched off.  Driving skill or ride quality (e.g. Via accelerometers) + preferences – semantic information. This posed some interesting ethical questions - see below.  Long term behavioural patterns – automatically deriving preferences – taking or giving – way that they otherwise travel. Time or comfort penalty... Data mining / activity recognition / - deriving preferences. o e.g. giving or taking a ride at weekdays at 9.00am – between a and b – bring your kid to the kindergarten – translating time and space into semantically meaningful preferences or patterns.  Driving skill or ride quality. Automated assessments of ride quality raised a number of contentious issues.

Firstly, storing data could in principle be incriminating if it revealed enough detail to be able to demonstrate traffic laws were broken. If this sensitive low-level data were not retained, then higher-level semantic categories ('sudden braking' - 'sudden acceleration') although not direct evidence of culpability, could still serve as a 'character witness' should someone be prosecuted for traffic violations.

Elaborating on the semantic spectrum outlined above, we consider AR producing the following: 1) Physical description (more than sense data e.g. 'fast') Here we had a discussion as to the extent that these 'semantic' labels could be contestable or value-free. Noncontestable terms were preferred - e.g. 'walking' or 'running' (leaving out liminal cases) are activities that are hard to dispute. Their meaning is not particularly socially contentious. This does not mean that they are value free terms, though, as walking has obvious value-laden connotations, any of which may be brought to the fore with a little more contextual detail - negative: 'you walked to rescue that person?'; positive - 'you walked rather than taking the car?'. Whether appraised positively or negatively, it would be hard to dispute that walking took place if the act was observed because this is a very stable social category that has a high level of intersubjective agreement. Other terms are perhaps more inherently comparative. Sudden braking or sudden © SmartSociety Consortium 2013 - 2017 Page 9 of (29)

© SmartSociety Consortium 2013 - 2017

Deliverable D3.2

acceleration requires a threshold to be set to partition activities into categories like 'sudden'. Actually, 'walking' as an activity recognised by a machine requires similar (and probably multiple) threshold type judgements to be embedded into the algorithms. Thus whilst people may rarely contest if someone is 'walking' or not, a machine's pronouncement that walking took place may be more contestable5 2) General human characterisation (e.g. 'dangerous') We note that here rules in regard to profiling and the user’s rights (as suggested in the EU General Data Protection Regulation (GDPTR) may apply (consider D4.1) Representing a 'higher' level of semantic analysis labels are attached to the ride that correspond with a 'average' or 'everyperson' judgement as to what the ride was like. In this way the AR reflects what might be a normative judgement about the ride. As an interesting exercise we considered how preference systems 'reflect back' normative judgements by typing the work 'beauty' to retrieve images from a search engine. Google and Ixquick both returned almost entirely images of women conforming to a contemporary Anglo-American media-centric aesthetic of female beauty. Demonstrates how a very partial norm has become embedded within a complex filter system, and suggests the possibility of feedback and reinforcement of that norm (as well as a trigger for dissent and protest).

3) Situated personal evaluation (e.g. 'uncomfortable') This differs from the normative perspective in that it is a situated and personal (or subjective) appraisal of the ride. Thus a normatively labelled 'dangerous' ride might be 'exhilarating' for some - or acceptable in certain circumstances. This perspective takes together context and preference to make an appraisal as to whether, for example, this is going to be the right ride for this person in these circumstances. A comparison might be made with identity - the ascription of traits and encumbancies that identifies you as a particular sort of person in a socially relevant way.

3.1.5 Autonomy and control Questions naturally arise as to the degree of control individuals have in the types of system discussed - how far unseen algorithmic hands (or emergent behaviours) act to give the 'illusion of control'. What happens to autonomy in chimerical systems where decisions are made in a hybrid environment involving devices connecting the user to deep wells of data via value-laden algorithms with the intent of shaping their behaviour? The recent Facebook saga over “emotional control” has brought this issue into the forefront of peoples' minds.

In particular, user interfaces for obtaining informed consent will need special attention (informing users about the consequences in an unbiased fashion).

Also we note that decentralisation is also usually more privacy-friendly, as there is no central point that can extensively profile all activities of users – instead information is more spread.

There is much current work on decentralised social networks (Diaspora, Peerson, Safebook) as more privacyfriendly alternatives to e.g. Facebook, where Facebook as a provider can profile everybody. Usually decentralisation is also exploited for anonymous communication systems, such as Tor.

3.1.6 Philosophical debates Finally we want to indicate that there are a few of philosophical debates over privacy principles and their relevance to modern circumstances. E.g.

1) Principle of limiting data use to specific purposes specified a-priori. Should privacy legislation seek to limit what data can be used for (tying data collection to a very highly specified purpose) - or should freer rein be given to use of data, but prosecutions made on those occasions where data use has materially disadvantaged someone? The issue of suveillance arose - as opposed the surveillance - that the ability undertake surveillance is no longer solely in the hands of governments. Data protection principles can be seen as dedemocratising data use.

5

Charles Goodwin's paper on Professional Vision (Goodwin, 1994) is useful here as it tells how intersubjectivity over people's description of the world are interactionally achieved. His analysis of the Rodney King trial reveals how the polices' actions on the video evidence could be made visible as an enactment of standard police procedures of “escalating” and “deescalating”, restraining force in response to the suspects actions, i.e. it makes visible the social procedures for producing accounts of activity and how these can be contestable.

Page 10 of (29)

http://www.smart-society-project.eu/

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

2) Principle of minimising data. Combining seemingly innocuous data from multiple sources already available means that we can deduce many detail about people in surprising ways. Also, correlating across multiple anonymised datasets quickly allows us to de-anonymise those data. On the one hand it may be argued that privacy is already broken - on the other that continuing to restrain how much data is collected or released at least makes it effortful to undertake these activities. This gives people some control (... or the illusion of control)...

4. The Hospital Use Case UNITN and DFKI started a targeted collaboration at the end of 2013 with the final goal of providing an additional case study to WP3 in order to define new methodologies for activity recognition that accounts both low-level and high-level information. The low-level dataset used has been provided by DFKI and consists of a set of data collected by smartphones carried by nurses during their shifts in the Mainkofen Psychiatric Hospital in Germany (http://www.mainkofen.de). The objective of this experiment, conducted some years ago, was to recognize the activities performed by the nurses using non-invasive logging devices.

Moreover, they manually annotated labels for training during the entire length of the experiment.

What we are attempting to do with this project now is to demonstrate that using a semantic approach in combination with low-level sensor data we can obtain higher recognition accuracy with respect to a solely classifier-based method. During the last months, the two groups agreed on and followed this methodology:  Analyse together the considered scenario, observing data and labels. This part of the project has been performed in Kaiserslautern in the period of 24-28 February 2014 where 2 DFKI researches and 2 UNITN project members met on a hands-on session. The experimental data were transferred to UNITN for a first analysis on the labels.  UNITN creates a semantic model describing the entities in the scenario and their relations (see Sec. 5.4).

The model has been discussed and approved by both parties in a meeting in Trento on 18/08/2014.  Fill up the model with the instances of the entities using an automatic algorithm that extracts semantic information from the labelled data. UNITN has created this script and populated the Knowledge Base.  Agree on a common format for the feeding of the low-level sensor data into the UNITN systems through an API. In the August meetings in Trento the two parties agreed on two possible formats with the idea of testing both of the two and see which one performs best: o Receive a single result merged from all the classifiers o Receive the result of each single classifier (movement, audio, location) and merge them according to the semantic knowledge The low-level sensor data will be stored in a Cassandra Based system as explained in Section 5.  Create software and algorithms to interpret low-level data using the semantic knowledge. This part of the experiment is the “Bridging” part that is expected to create the interface between the two levels.  Collect the results and perform final tests.

We are summarizing our methodology as well as findings and plan to submit a first version of the results to the Journal “Personal and Ubiquitous Computing”6 by November 15th.

5. Bridging the Semantic Gap In this section, in particular, we illustrate how we address the issue of bridging the semantic gap with automated methodology. As stated before, we should try to semantically tie the levels together. Our proposal consists of a three-level schema where each level accounts for different degrees of abstractions: the first one represents sensors, the third represents models and the middle layer fuses the sensor information semantically, i.e., following the model specifications, to provide enriched information about the user. This schema is implemented in a system with a front-end, i.e., mobile devices, and a back-end, i.e., databases for handling sensor streams and models. We now dedicate a subsection each for describing the system architecture (5.1),

6

http://www.springer.com/computer/hci/journal/779

© SmartSociety Consortium 2013 - 2017

Page 11 of (29)

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

the three layers schema itself (5.2) and the formal models (5.3) for our two use cases, the Mainkofen Hospital and i-Log (to be outlined next).

5.1 The System Architecture Partner UNITN has designed and implemented a system capable of dealing with both low-level and high-level information with the final goal of bridging the Semantic Gap between the two levels; the final system architecture is presented in this Section.

The starting point of the process of bridging the semantic gap is data, both low-level and high-level. Real time data is fundamental in the specific scenarios of a Smart Society to obtain information about the on-going tasks and to provide tailored services. For this reason, we have built an infrastructure called i-Log capable of collecting users’ personal information at different levels of abstraction. Its two main elements are shown in Fig. 2.  a Front-end system for mobile devices  a Back-end, aimed at storing and processing user personal information BACK-END FRONT-END

STREAMS’ PERSISTENCE SYSTEM

KNOWLEDGE BASE

APIs

Figure 2. The i-Log architecture The front-end has been created within the Smart Society project and will be released as a Open Source compontent7, while the back-end system, both a Cassandra-based persistence system and the Knowledge Base, are prior developments of our research group. We will arrange these systems to deal with live-streaming data from smartphones and wearables in addition to provide dedicated APIs to work with them.

5.1.1. The Front-end In order to collect users’ personal information, ad hoc logging devices are needed. Today, the market offers different solutions for this purpose, i.e. dedicated logging devices. These devices are becoming popular because of their power to process data, their lightness and increasing unobtrusiveness, etc. but they still face an important bottleneck from the user’s perspective, i.e., the user needs to always carry them, or otherwise they are useless. Consequently, we decided to concentrate our effort on an embedded solution in users’ smartphone and attached wearable devices. As several statistics show, smartphones are the most widespread electronic device on the market, with a penetration rate of 56% worldwide8, up to 97% in the age range 18-299. Almost every user has one and carries it everywhere for the whole day, since he or she needs to keep in touch with the rest of the world. We believe that a logging device embedded in the user’s smartphone can collect more truthful data since it would not alter the user’s usual behaviour like a dedicated logging device would.

7

Released under AGPL 3.0 license: http://www.gnu.org/licenses/agpl-3.0.html http://supermonitoring.com/ 9 http://www.compendium.com/ 8

Page 12 of (29)

http://www.smart-society-project.eu/

Deliverable D3.2

Š SmartSociety Consortium 2013 - 2017

Basing on this assumption, the partner at UNITN has now designed and implemented an infrastructure with logging purposes from smartphones and attached wearable devices. The front-end part consists of a set of mobile applications, distributed across each device and allowing the user to have full control of his logs. In a totally transparent, unobtrusive and safe way the application collects information from both the environment and the user through internal deviceâ&#x20AC;&#x2122;s sensors and generate real-time streams of data that can be synchronized with the back-end infrastructure for further analysis. The application has a dedicated user interface allowing the user to select between different logging modalities, manage his log and visualize the collected information as shown in Figure 3. The i-Log user interface.

Figure 3. The i-Log user interface Once the user chooses the most suitable logging modality, the application starts an Android Service that runs in the background, and the user interface can be closed to return to the normal use of the smartphone; the application is now working in a completely transparent way except for the icon in the Notification Center. The user can return to the application to modify any aspect of the logging at any moment.

Š SmartSociety Consortium 2013 - 2017

Page 13 of (29)

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

It is well known that some smartphone sensors have high energy consumption in terms of battery, e.g., GPS, and they can drain the battery in few hours; thus we introduced smart and battery life friendly logging modalities. For instance, consider the Location of the user: on today’s mobile devices the location is normally provided from the GPS sensor, characterized by a high accuracy (<5m) but also by a high energy demand or by other sources like WIFI (accuracy < 50m) or cellular (accuracy <1000m) networks. In the last two cases the accuracy is lower but the additional battery consumption is almost zero since the WIFI and cellular are always turned on. If we also consider the fact that WIFI is used almost exclusively in indoor scenarios, it is sensible to assume that while WIFI is on, GPS is useless and then it can be turned off while recognizing the location from WiFi only. This and other smart energy saving processes were implemented in order to make i-Log as unobtrusive as possible.

This is also true for the wearable devices where i-Log is an application Figure 4. with a dedicated user interface as well or an add-on (Zeni et al. 2014).

Figure 4. The i-Log wearable user interface Let us now describe in more detail the kind of information i-Log can collect. At this point we must distinguish between two types of sensors: physical sensors and virtual sensors.

Physical sensors are the “normal” sensors used to sense the environment and the user. They collect primitive data, corresponding to physical measures like acceleration, speed, and light intensity, among others. The physical sensors the application can sense are related to the devices; we used a Samsung Galaxy S4 smartphone and a Samsung Galaxy Gear Smartwatch in our experiments. Table 1. shows the available physical sensors for our experiments.

Physical Sensors Accelerometer Gravity Gyroscope Linear Acceleration Rotation Vector Magnetic Field Orientation Proximity Temperature

Light Pressure Humidity GPS Network Location Audio Accelerometer (wearable) Gyroscope (wearable)

Table 1. The i-Log physical sensors On the other hand, virtual sensors are derived variables computed by software algorithms, e.g., the call log or and SMS list. Table 2. shows a list of all streams that i-Log collects. The general idea is to have a dataset as complete as possible in order to study the user behaviours and to provide services accordingly.

Virtual Sensors WIFI networks Page 14 of (29)

Outgoing calls http://www.smart-society-project.eu/

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

Bluetooth Proximity Bluetooth LE Proximity Calendar events Incoming calls

Incoming sms Outgoing sms Running applications

Table 2. The i-Log virtual sensors

5.1.2 Back-end The collected data from i-Log are temporarily stored in an SQLite database on the device but they need to be synchronized periodically or on a real-time basis. They cannot be stored on the smartphone permanently because of their size, which soon grows too big for a normal smartphone. In fact, mobile devices are characterized by limited performances in terms of computational power and storage (up to 64GB on the newest models). Consider that a day of logging with i-Log at the fastest speed generates up to 100 millions of value-pairs or a total of 6GB of raw data. At this rate, many smartphones cannot even store a day’s worth of data. Thus, we needed to develop also a back-end component for this system, aimed at collecting the logs from the user and to process them offline. We decided to deploy an existing system we are using at UNITN, conveniently modified for the specific purpose; in the context of this project, this system counts as previous knowledge. It consists of two components: a Cassandra10 based Persistence System and a Knowledge Base,

5.1.3 Cassandra-based persistence storage At UNITN we have an internal Cassandra Based persistence system storing a huge amount data in forms of streams. We use an adapted version of this back-end for the Smart Society project and will provide access to dedicated APIs for posting and getting data.

Cassandra is a distributed database system crated by Apache, whose main advantages are stability and scalability (thanks to “sharding”). We decided to use this system to allow for future extensions of the work, especially in terms of number of users and services. In order to access the database, we arranged a set of dedicated APIs (Fig. 5.) for the Smart Society project that can be called from the mobile devices to POST data and GET back analysed replies. Moreover, we created two different upload modalities: a real-time one that allows to upload one value per stream at the time and a batch modality that allows to synchronize many entries in one single POST request in a second time. The format of the request is showed in Figure 6. The i-Log POST request format. The batch modality is designed for uploading data when the user is not connected to the Internet.

Figure 5. The i-Log back-end API

10

http://cassandra.apache.org

© SmartSociety Consortium 2013 - 2017

Page 15 of (29)

Deliverable D3.2

Š SmartSociety Consortium 2013 - 2017

Figure 6. The i-Log POST request format Data is processed and stored in the database based on its event type, in the form of key-value pairs.

5.1.4 The Knowledge Base The other component of the back-end and available at UNITN, and to be used for this project, is the Knowledge Base. Our semantic annotation system has to consider previous knowledge as well, and we will allow access to a Knowledge Base through dedicated APIs similar to those of the Cassandra-based system.

The Knowledge Base is aimed at storing high-level representations of the specific scenarios and allowing for high-level semantic queries. By a high-level representation we mean here a model that can be broadly defined as a representation of the world and the relations among its objects. In our case, the model will represent activities, locations, people and objects from which data were collected. In addition, it will account for the interactions among them, especially which activities happens in which locations and whether they follow any pattern, i.e., an actual flow of activities. Thus, the purpose of our model is to provide an effective representation of the world, and to integrate low level sensor data by recognizing and accounting for meaningful patterns of activities. In other words, the model aims at bridging as much as possible the semantic gap between real world data and their high level representation by attempting to find a middle way between a bottom-up and top-down approach.

The Knowledge Base system is ideal for creating a reusable representation of a given domain and providing tools for reasoning and making inferences on data. However, the typical shortcoming of ontologies is the inability to fully accommodate for dynamic data; indeed, this is especially important in the case of activity recognition. In fact, sensor data is by default highly dynamic, thus rendering reasoning computationally inefficient. Furthermore, without considering the kind of data expected in the case study, the granularity of the representation may be lost. For this reason, a real-time approach to update the current status of the scenario with sensor data appears to be not the right solution, altering the reusability of the system. The solution proposed is to create a dedicated version of the semantic model for the considered scenarios as explained in details in Section 0.

Page 16 of (29)

http://www.smart-society-project.eu/

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

5.2 A 3-layer approach for semantic-driven sensor fusion In a real-time system such as a Smart Society application, streams of data are fundamental. As shown above, iLog can generate and efficiently manage the streams of low-level data from internal smartphone sensors.

However, relying solely on low-level streams is not sufficient to bridge the semantic gap while higher levels streams need to be generated, e.g., to train the system for personal everyday life patterns. Starting from this assumption, we propose for this project an existing solution we are using at UNITN based on a 3-layer (or level) approach capable of generating three different layers of information at different abstraction levels:  Level 1, is the low-level one that contains only sensor data collected with the i-Log front-end application. This level has no semantic meaning itself, except for the link to the Knowledge Base.  Level 2, contains streams with an additional level of abstraction from Level 1. Computed at the serverside, these data are derived using a semantic driven sensor fusion approach using the semantic model to create meaningful information through algorithms.  Level 3, contains only semantic information. We can assimilate it as a stream of the contents of the Knowledge Base that can be queried in different moments of time (t0, t1, …, tN). On the contrary, the Knowledge Base system contains only information referred to the present time.

Figure 7. The 3-layers approach shows a schema of how this approach is structured. Thanks to its information at different levels of abstractions, this schema is the factor that we are using to design solutions for bridging the Semantic Gap between low-level sensor data and high-level semantic knowledge.

Figure 7. The 3-layers approach Within the Smart Society project we will provide an Open Source11 version of the algorithms we will produce to bridge the semantic gap at level 2 in our 3-layer approach proposed solution. We start working on this part in the Fall.

5.3 High Level Models In accordance with our 3-layer approach, we had to define the third layer as a semantic model capturing the entities, attributes, and their relations among them. We are currently working on two case studies: the Mainkofen Hospital illustrated in Sec. 4 and the Ride-Sharing use-case in Sec. 6. We dedicate a subsection for both of them, illustrating the general model together with issues we had to address during the modeling part.

5.3.1 The Mainkofen Hospital Use Case In the case of the Mainkofen Hospital case study, the main entities we had to model were people, activities and the wardrooms; see Annex A for the top level e-types’ structure.

Note that the attribute Probability is used for representing the probability that the system is currently recognizing either activities or location; in this case, sensors can give us a recognition probability for either 11

Released under AGPL 3.0 license: http://www.gnu.org/licenses/agpl-3.0.html

© SmartSociety Consortium 2013 - 2017

Page 17 of (29)

Deliverable D3.2

Š SmartSociety Consortium 2013 - 2017

rooms or activities. We now analyze rooms, people and activities to illustrate our though process in modeling them in Sec. 5.3.1.1, 5.3.1.2, and 5.3.1.3, respectively.

5.3.1.1.

Room

As the Figure 8. The Mainkofen Hospital floor plan shows, there are essentially two kind of rooms there: patient rooms (called Z1, Z2, etc...) and other rooms, ranging from hallways to break rooms.

Figure 8. The Mainkofen Hospital floor plan Some of the patient rooms, namely all except Z5, have a private bathroom to clean the patient(s); there is also a bathroom, called SB that can be used if the bathroom is occupied/not present. To capture this distinction, we developed two e-types: Room and Patient Room, where the former is the parent of the latter, structured in Annex A. Initially, objects that are usually in rooms, e.g., stethoscopes, sinks, trolleys and so on, were considered as possible entities in the model. However, since all sensors are on the smartphones carried by the nurses and very few (if no) objects could be recognized, we opted for not modeling them to keep the model simple.

5.3.1.2.

Person

In this experiment there are two types of people: nurses and patients. Since all participants are anonymized by design, they can be recognized either by the room they are staying in (patient) or simply by the area they take care of (nurses). In addition, nurses carry the smartphones providing low-level sensor data about their position and their activities, e.g., washing patient Z1-T in Room Z1. See Annex A for the structure of the Person etype.

As one can see, we did not represent nurses as a more specific kind of person since it is a role of a person, and roles are anti-rigid12, even though it would be more complex than patients who only have IDs. The additional complexity is due to the category representing activities and locations where the nurse is performing, performed or will perform a set of activities, whose representation is explained later below.

5.3.1.3.

Activities and Events

The main issue with this case study was to represent activities via ontologies. The ontological approach, while adopted in an increasing number of works1314 to represent ADL (Activities of Daily Life), still suffers limitations due mainly to the static nature of ontologies and the way the handle uncertain or incorrect information. In our case we needed to represent both activities and the path or sequence among them in

12

Guarino, Nicola, and Christopher Welty. "A formal ontology of properties." Knowledge Engineering and Knowledge Management Methods, Models, and Tools. Springer Berlin Heidelberg, 2000. 97-112.

13 Chen, Liming, and Chris Nugent. "Ontology-based activity recognition in intelligent pervasive environments."

International Journal of Web Information Systems 5.4 (2009): 410-430.

14 Riboni, Daniele, and Claudio Bettini. "COSAR: hybrid reasoning for context-aware activity recognition." Personal and Ubiquitous Computing 15.3 (2011): 271-289.

Page 18 of (29)

http://www.smart-society-project.eu/

Deliverable D3.2

Š SmartSociety Consortium 2013 - 2017

addition to localizing them, i.e., identifying and representing the sequence of rooms where activities take place.

We chose an approach similar to Finite State Machine (FSM), where nodes are activities or rooms and link between them are probabilities of one activity or room to be the next one; these steps are then modeled as complex attributes coupling both the temporal and spatial information, i.e., activities and rooms, respectively.

Then these steps are semantically tied together by the e-type Event. By semantically tie we mean that events group different activities and location by combining the flow of spatiotemporal information and defining it without explicitly constraining the sequence of activities or rooms in it. In fact, the same activity may belong to different events, e.g., cleaning may be in a washing event or in a laundry event. Refer to Annex A for the modeling of the complex attributes Activity and Step, and the e-type Event.

As we can see, activities have a start and end plus duration, together with a probability attribute equivalent to the one of the e-type Room. The reason why Step has exactly two different relational attributes is two-folded.

Firstly, an activity does not repeat itself, since we assume one recognizable activity at the time, and if we a one-to-many or even many-to-many approach we would not be able to represent and link each single probability value for the step.

5.3.2 The i-Log Use Case The model for i-Log is simpler for this first implementation compared to the one used for the Mainkofen Hospital. First of all, there are only two e-types (with their corresponding structured attributes): these are Device and Location, whose structure is showed in the tables in Annex B. The small amount of e-types is due to the simplicity of the experiment. In fact, the experiment will consist in recording and recognizing a small amount of locations and the change of position of a user. While there is not much to discuss concerning locations, here abstracted to a simple position with a name, the actual range of possible activities is limited to movement like walking, running and so on. Nonetheless, they are still assumed to be complex attributes of the device carried by the user. Note that experiment uses both a smartphone and a smartwatch that we treated as clusters of sensors. Therefore we did not further specify them, addressing this distinction via the Class attribute, although smartwatches (at least the one used in the experiment) do not have GPS receivers, while smartphones do.

6. The RideShare Use Case In the Smart Society Ride-Sharing use case, i-Log can be used to send real-time information about the user and the ride to the Peer Manager, according to the 3-layer approach, as explained above and showed in Figure 9. The 3-layers approach in the Ride-Sharing use case: - Level 1: i-Log provides sensor data request by the Peer Manager to fill up the e-type fields like Latitude, Longitude, Speed, among others.

- Level 2: adds an additional level of abstraction to the information collected at level 1. For instance, it can generate a list of peers close to the main peer in order to check the list of participants, or in can understand the quality of the drive from accelerometer data, among others.

- Level 3: provides purely semantic information, e.g., whether the ride is actually taking place or is on time.

Š SmartSociety Consortium 2013 - 2017

Page 19 of (29)

© SmartSociety Consortium 2013 - 2017

Deliverable D3.2

Figure 9. The 3-layers approach in the Ride-Sharing use case This concludes our discussion of the Semantic Gap and our approach to elucidate it. Future work will continue to integrate and coordinate all steps of the methodology and to perform the planned experiments with real data including streaming data directly relevant for SmartSociety applications.

7. Incremental Knowledge Fusion The second leg of our approach is model based, focusing on the modelling of self-organizing collectives with a low degree of prior coordination and with the aim to produce emergent, high-level “smart” collectives from the interactions.

7.1 Incremental fusion In D3.1 we have presented a collaborative localization model developed in DFKI for collaborative localization (subjective positional information merged iteratively upon meetings), as a baseline case for knowledge fusion.

The work was published in a recent journal paper (Kampis et al. 2014) as well as a conference paper (Kampis and Lukowicz, 2014).

To introduce the current work, which goes beyond that, we again quote from D3.1 first: “In the case of collaborative localization the functionality is simple and local (i.e. knowledge of a good estimate of current position by an agent, despite the lack of a central localization service). In the case of collaborative knowledge fusion, the goal is more ambitious and involves a self-organized emergence of common knowledge in a population of agents.” “Collaborative localization is a baseline case for collaborative knowledge fusion (and both are a ground case for CAS) because here essentially the agents exchange knowledge to achieve an advanced knowledge status collectively that is not achievable individually, and to achieve it without planning or design of the agent to agent information flow.” It is this picture that we want to elaborate now in the more general case of incremental (collaborative) knowledge fusion.

7.2 How the model works The basis is the meet-and-locate model of collaborative localization. In the current simple model for incremental fusion, N agents move randomly and meet in 2D space and upon meeting they exchange information. Each agent is a binary classifier, and takes decision 0 or 1 with a probability p (and 1-p, respectively).

Page 20 of (29) http://www.smart-society-project.eu/

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

The current model is static, and there is always a fixed “correct decision” known to the experimenter but not known to the agents, one that stays unchanged within a single run. To be able to speak of the “quality of the knowledge state” of a population of agents, we take each agent’s decision and compare the average of these decisions for the population with the right decision (here we assume that an agent’s decision is 0, if its p value is larger than 0.5 - remember that by construction p is the probability of the zero decision). The quality of knowledge of the population is a real number between 0 and 1.

Further we assume that some of the agents are “well-informed”, and remain well informed all along (i.e., these agents also undergo meetings, but their knowledge is never updated – their meeting partners’ knowledge however is). What happens in the population? For instance, can a correct decision emerge from an incorrect one? If so, under what conditions?

7.3 Two Model Variants and Their Results 7.3.1 Model version 1: the averaging of knowledge upon meetings When meeting (similarly to the case of collaborative localization discussed in D3.1) the agents compute and share an average of their knowledge, i.e. in this case p_new = (p1 + p2) / 2.

Clearly, without well-informed agents (WIAs) the population undergoes a random process towards a convergent end state. Whereas the naive expected value of the population knowledge quality (or its complementary, the error) is 0.5, we note that the exact value is never realised in an individual run. Every individual run rather ends with 0 or 1 being the entire population’s homogeneous knowledge state reached after a time – but the two values are equally probable w/o well-informed agents. To speak of the other extreme, is also obvious that, if all agents are WIAs, then the population starts from and remains in the perfect knowledge state. Interesting things happen between these extremes.

How many WIAs does it take to have a “directed process” so that the end value of the population error is zero, independently from the initial knowledge quality? It turns out, that for this, one single WIA is enough. It is easy to understand why, as there is no equilibrium situation until all agents converge to the knowledge state of the WIA. The “zero crossing time”, to use a statistical physics terminology grows however dramatically with a decreasing value of the WIA proportion, and we have studied this effect using parameter sweeps.

We also note that there is a long-standing effect of the initial knowledge state of the population in the process.

If the initial knowledge state is entirely random (i.e. the expected value of p is 0.5 at t_0) then the process in the presence of WIAs, converging to the completely informed knowledge state is fast. If the initial population is extremely badly informed (p=0 for the 0 decision, when the right decision is 0) then the convergence to the right decision is significantly slower, which is also easy to understand. Yet in all cases tested, the process must necessarily converge to a perfect knowledge state in the presence of WIAs.

7.3.1.1 Explanation of observed behavior The “trick” that makes the convergence happen in the presence of WIAs is that agents no doubt sometime meet with these WIAs, and from this meeting they gain valid knowledge – because of knowledge averaging, after meeting with a WIA the meeting partner has improved knowledge. Then they meet again with the WIAs and with the other “updated” agents and so on – so they improve more and more. In fact, there is a speedup process here, and the convergence looks exponentially accelerating (to be tested yet).

In this “quantitative democracy” (the name is proper because p_new = p1 + p2 / 2), without WIAs the naive expectation is a stable convergence to p=0.5. This is exactly what happens at the detailed level but we should also consider the fact that, as described above, it is ultimately not the p value but the resulting decision what matters when we calculate the quality (or error) of the knowledge state. Plotting the p histogram shows that a sharp end value is indeed always very close to 0.5 but due to random effects never exactly that (to which we could even assign an undecided value – but it is rounded off to one global decision anyway). For the convergent state, we have measured average p values from 0.46 to 0.53 in individual runs (also to be tested: they probably form a Gaussian, if the experiment is repeated many times). Coarse graining of p (e.g. by 5%) would improve the end result (yielding many middle values the can be labeled undecided, but of course with a decreasing probability the system would nevertheless end up in a random homogeneous decision state).

If WIAs are introduced, a dynamic histogram visually shows how different things now are, and shows the WIAs’ role as “attractive wall” at one end of the p spectrum, that ultimately “sucks in” the entire distribution © SmartSociety Consortium 2013 - 2017

Page 21 of (29)

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

to yield a homogeneous correct decision. We also understand why there is no lower limit for the WIAs but that their number determines the speed of convergence via the frequency of meetings. Also visible is that when the entire p histogram is on the “one side” of the 0.5 value already, then the entire population decides for one value, yet it takes another long time for the p values to actually “reach the wall” in an internal dynamics of p yet not reflected in different decisions any more, as these are already frozen when p is smaller (or larger) than 0.5.

7.3.1.2 Parameter sweep results Below some figures showing the effect of WIAs (as their proportion increases) and the effect of the quality of the initial knowledge state of the population. What we see is that these two have a similar effect and that in combination they lead to a fast convergence of the population to a high quality emergent knowledge state.

(Zero crossing time = time until the error disappears, i.e. the imperfectness of knowledge quality goes to zero.)

4000 3000 0

1000

2000

zero crossing time

3000 2000 0

1000

zero crossing time

4000

5000

Convergence of knowledge versus initial right decision probability

5000

Convergence of knowledge versus well-informed agents

2

4

6

8

10

percent of well-informed

0.0

0.2

0.4

0.6

0.8

1.0

initial right decision probability

Below, two summary plots in 3D (left, color is the 3rd variable, another plot with xyz): Aligning of population knowledge 5000

zero.crossing.time

4000

percent.well.informed 10.0

3000

7.5 5.0 2.5

2000

1000

0 0.00

0.25

0.50

0.75

1.00

initial.decision.p.null

All these show the quantitative (as well as qualitative) trade-off between WIA proportions, initial correct decisions and zero crossing time.

Page 22 of (29)

http://www.smart-society-project.eu/

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

7.3.2 Model version 2: experience as reputation As a first step towards more realistic variants, we introduced an “experience wins” version in which a simple reputation system is realized. Here (in the first series of tests) reputation is simply understood as experience of prior meetings.

The idea is that, when two agents meet, the naive agent takes the p value of the experienced agent. (Later it can be proportions etc. also, or, what probably makes a bigger difference, the reputation may be made transitive, that is, taken over by the less experienced, together with the knowledge value.) In the current form, however, the agent with a lower number of prior meetings takes the p value from the one with the more meeting experience, and that’s all there is. (Based on the discussion of Model version 1 above, we can say: with more meeting experience, there will be more chance to be “smarter”, i.e. improved by the meetings with WIAs, so the more experienced can be better – or anyway that is the rationale behind the choice).

Tests first of all show the obvious. If the WIAs have no bias added (no added initial “experience” — i-e. trust, reputation), then “nothing interesting happens”, albeit typically the system now converges to 0.5 and not to one single coherent decision (the reason is that other than in Model version 1 here we have the pullback effect of the badly informed yet highly experienced agents – informally we can call them false prophets).

Similarly, in a less obvious case: if an initial WIA bias is added, then, even at higher values of this bias, the “head advantage” of these WIAs can gradually disappear and compensated by other, however badly informed agents, that have happened to have just more meetings and thus have high experience and so continue to spread a wrong decision. Applying more WIAs and/or more bias can, of course, drive the system to the right decision, or bring it closer.

7.3.2.1 Explanation of observed behavior The behavior is very different now. In this reputation based case case when meeting the agent with a higher reputation “takes it all” (its p value is copied to the other, and there will be no kickback effect).

As expected in this case, several sharp p values rapidly emerge as “attractive centers” around spontaneously emerging random experienced agents in the system, and this is true with or without WIAs, because the process is identical: there is no difference. (The difference, if any, is encoded at time zero in the initial reputation of the WIAs). All this is completely ad hoc, of course, and as a consequence, some agents build up reputation although they do not have better knowledge, they do not know any better.

These “false prophets” tend to compete with the WIAs next, and such “false prophets” usually do not dissolve even after a long time. The reason is that even if WIAs can retain a certain head advantage of their initial experience bias, a WIA may perhaps never actually meet with the “false” prophets” that can therefore remain strong locally (almost) forever. As a result, the system can maintain a mixed knowledge state for a very long time. But ultimately in the very long run the system must collapse to a single homogeneous knowledge state again, for equilibrium reasons, since the random effects drive it around and the extreme values of p act as sinks (absorbing walls) - however usually the time needed is not on the practical scale.

7.3.2.2 Parameter sweep results Since the relevant events appear to be on a plane, below we show them in 3D only (the cross-sections are not informative).

7.4 Current work Current work has the ambition to extend the models in several directions, some of which have been outlined above. How to get rid of the “false prophets”? Building reputation on merits is a possibility, and it takes rounds, to be considered in a next major version of the model. For this we currently do not see a completely local solution but the use of a “central scrutinizer” that assigns reputation values on ground of success in earlier rounds, e.g. if an agent was at some predefined final time step at the “right side of a decision”. That would increase decision convergence but not immediately p-convergence, since interim values on the “right side” of 0.5 could still be held by less “false” but still “imperfect prophets” (that have the right decision already but for the wrong reason - the wrong p). This however may be a feature and not a bug on the other hand, yielding a population that decides correctly but still maintains heterogeneity and variation.

© SmartSociety Consortium 2013 - 2017

Page 23 of (29)

© SmartSociety Consortium 2013 - 2017

Deliverable D3.2

Further questions to be considered: Is there an obvious way to model and modify the binary classifiers directly? Currently it is just a p value, and not a full probabilistic mapping from an input vector (of the Nspace to be classified) to [0,1]. Introducing more complexity towards “multi-agent” models may help experimentation also towards the cognitive dimensions.

Page 24 of (29)

http://www.smart-society-project.eu/

Deliverable D3.2

© SmartSociety Consortium 2013 - 2017

References Kampis, G., Kantelhardt, J.W, Kloch, K., and Lukowicz, P. (2014): Analytical and Simulation Models for Collaborative Localization, J. Computational Science, accepted.

Kampis, G. and Lukowicz, P. (2014): Collaborative Localization as a Paradigm for Incremental Knowledge Fusion, 5th IEEE CogInfoCom 2014 Conference, accepted.

Zeni, M., Giunchiglia, F. and Zaihrayeu, I. (2014), Multi-device Activity Logging, “Ubicomp - ACM International Joint Conference on Pervasive and Ubiquitous Computing”, Seattle, USA, September 1317.

Carroll, J. (2006). Personal Information Infrastructures for Everyday Lived Experience: A Challenge for the Future, Proceedings of Helsinki Mobility Roundtable. Sprouts: Working Papers on Information Systems, 6(39). http://sprouts.aisnet.org/6-39 Goodwin, C. (1994). Professional vision. American Anthropologist, 96(3), 606-633.

Sawyer, S, Wigand, R. and Crowston, K. (2014) “Digital Assemblages: Evidence and Theorizing from a Study of Residential Real Estate,” New Technology, Work, and Employment, in press.

© SmartSociety Consortium 2013 - 2017

Page 25 of (29)

Deliverable D3.2

Š SmartSociety Consortium 2013 - 2017

Annex A

High-level modelled entities: Mainkofen scenario

These etypes and structured attributes extend the ones showed in the WP4 MS8 document, namely those of Person, and Event.

Room ID

1081

Description

A room in the Mainkofen Hospital.

EC

Room

PT

Entity

Attributes ID

Name

Data Type

Description

10810000

Probability

Float

Probability of sensors recognizing this room.

Patient Room ID

10811

Description

A patient room in the Mainkofen Hospital.

EC

Patient Room

PT

Room

Attributes ID

Name

Data Type

Description

108110000

Patient

<Person>[]

The room patient

108110001

Nurse

<Person>

The nurse taking care of the room

108110002

Bathroom

<Room>

The bathroom in the room

Person ID

103

Description

A person is an individual human being.

EC

Person

PT

Entity

Attributes ID

Name

Data Type

Description

1030000

ID

String

The person ID.

1030001

Role

<Concept> [ ]

The person role

1030002

Activity

<Activity>

The activity the person is performing

1030003

Event

<Event>

The event the person is in

1030004

Room

<Room>

The room the person in in

Event ID

130

Description

Something that happens at a given place and time.

EC

Event

PT

Entity

Attributes ID

Name

Data Type

Description

1300002

Step

<Step> [ ]

The sequence of activity steps

Page 26 of (29)

http://www.smart-society-project.eu/

Deliverable D3.2

Š SmartSociety Consortium 2013 - 2017

belonging to this event.

1300003

Movement

<Step> [ ]

The sequence of room changes belonging to this event

Activity ID

700

Description

An action performed by a nurse

EC

Activity

PT

-

Attributes ID

Name

Data Type

Description

7000000

Class

Concept

Class of the activity

7000001

Duration

Moment

The sequence of activity steps belonging to this event.

7000002

Start

Date

The sequence of room changes belonging to this event

7000003

End

Date

End of the activity

7000004

Probability

Float

Probability of sensor detecting an activity

7000005

Patient

<Agent>

Patient involved in the activity

7000006

Direction

Float

Direction towards which the device is moving

7000007

Speed

Float

The speed of the device.

Step ID

800

Description

The step with a certain probability between two entities in a sequence, e.g., drying after washing

EC

Step

PT

-

Attributes ID

Name

Data Type

Description

8000001

Previous Activity

<Activity>

The first activity of the step.

8000002

Subsequent Activity

<Activity>

The second and final activity of the step

8000003

Probability

Float

Probability of this step

Š SmartSociety Consortium 2013 - 2017

Page 27 of (29)

Deliverable D3.2

Š SmartSociety Consortium 2013 - 2017

Annex B

High-level modelled entities: Ride-share scenario

These etypes and structured attributes are the same of those showed in the WP4 MS8 document (Entity and Location) while Activity is taken the structured attribute in Annex A and adding information about speed and direction.

Device ID

132

EC Electronic Device

Entity

Description

The device recording userâ&#x20AC;&#x2122;s locations and activities.

PT

Attributes ID

Name

Data Type

Description

1320001

ID

Float

The device id

1320002

GPS

<GPS>

The GPS receiver of the device

1320003

Accelerometer

<Accelerometer> The accelerometer of the device

1320004

Bluetooth

<Bluetooth>

The Bluetooth of the device

1320005

Activity

<Activity>

The activities recognized by the device

Accelerometer ID

900

EC Accelerometer

Description

An instrument for measuring the acceleration of an entity

PT

-

Attributes ID

Name

Data Type

Description

9000001

X-Axis

Float

The horizontal axis in a plane coordinate system

9000002

Y-Axis

Float

The vertical axis in a plane coordinate system

9000003

Z-Axis

Float

The third axis in a 3-dimensional coordinate system

9000004

Status

<Concept>

The current status of the sensor (e.g. on)

GPS ID

1000

EC -

Description

A sensor for detecting the position of a device

PT

-

Attributes ID

Name

Data Type

Description

10000001

Latitude

Float

10000002

Longitude

Float

10000003

Altitude

Float

The constant coordinate for the latitude (in WGS84 decimal format) The constant coordinate for the longitude (in WGS84 decimal format) Elevation especially above sea level or above the earth's surface

10000004

Speed

Float

The speed of the device

10000005

Bearing

Float

the direction or path along which something moves or

Page 28 of (29)

http://www.smart-society-project.eu/

Deliverable D3.2

Š SmartSociety Consortium 2013 - 2017

along which it lies 0406

Status

<Concept>

The current status of the sensor (e.g. on)

Bluetooth ID

1100

EC -

Description

A sensor for detecting bluetooth devices

PT

-

Attributes ID

Name

Data Type

Description

11000001

Bondstate

String

The state of the connection process where the link key and connection information are stored in non-volatile memory

11000002

Name

String

The name of the device indicating where the bluetooth is

11000003

Address

String

A unique identifier assigned to network interfaces for communications on the physical network segment

11000004

Status

<Concept>

The current status of the sensor (e.g. on)

Š SmartSociety Consortium 2013 - 2017

Page 29 of (29)