Apollon - Evaluation and Impact Assessment Report

APOLLON – D1.6 Evaluation and Impact Assessment Report

DELIVERABLE Project Acronym:

APOLLON

Grant Agreement number:

250516

Project Title:

Advanced Pilots of Living Labs Operating in Networks

D1.6 Evaluation and Impact Assessment Report APOLLON

Revision: Final

Authors: Hans Schaffers (AAL) Pieter Ballon (IBBT) Hendrik Hielkema (AAL) Petra Hochstein (SAP) Anand Raju (IBBT) Mari Runardotter (LTU) Anna Stahlbrost (LTU Sampo Tukiainen (AAL)

Project co-funded by the European Commission within the ICT Policy Support Programme Dissemination Level P

Public

C

Confidential, only for members of the consortium and the Commission Services

x

APOLLON – D1.6 Evaluation and Impact Assessment Report

Revision History Revision Date

Author

Organisation

Description

0.1

14.12.2011

HS

AAL

Report set up and approach

0.3

12.02.2012

HS

AAL

Introduction and framework sections

0.2 0.4 0.5

09.02.2012

04.03.2012 16.03.2012

HS HS, ST, HH, MR AR,PB

AAL AAL, LTU IBB

0.6

20.03.2012

MR

CDT

0.7

28.03.2012

HS

AAL

0.8

07.05.2012

PH

SAP

1.0

10.05.2012

HS, AR, PB

AAL, IBB

0.9

07.05.2012

HS

AAL

Chapter structure changes

Several contributions, editing

Business Impact Assessment and Success stories Conclusions from applying the evaluation framework

Over-all editing, including findings from pilot evaluations Review, eManufacturing inputs Final chapter and final editing

References, editing and review

The information in this document is provided as is and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability.

Statement of Originality This deliverable contains original unpublished work except where clearly indicated otherwise. Acknowledgement of previously published material and of the work of others has been made through appropriate citation, quotation or both.

ICT PSP Project Reporting Template

2

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

Table of Contents 1.

2.

3.

1.1 1.2 1.3 1.4

Introduction..................................................................................................................... 5 Role of D1.6 within the APOLLON project ....................................................................... 5 Evaluation based on APOLLON objectives framework .............................................. 6 Evaluation and impact assessment sources................................................................... 9 Overview of this report ...................................................................................................... 10

Project Evaluation and Impact Assessment Framework .............................. 11

2.1 Introduction ........................................................................................................................... 11 2.2 Evaluation and impact assessment logical framework .......................................... 11 2.3 Types of evaluations and their implementation ....................................................... 14 2.3.1 Evaluation of achieving the objectives ............................................................................... 14 2.3.2 Evaluation at pilot and project level .................................................................................... 14 2.3.3 Description of the different evaluation source materials ........................................... 15

Evaluation and Impact Assessment at Pilot Level ........................................... 18

3.1 Homecare and independent living pilot....................................................................... 18 3.1.1 Approach ......................................................................................................................................... 18 3.1.2 Pilot objective ............................................................................................................................... 18 3.1.3 Methodology evaluation and impact ................................................................................... 18 3.1.4 Value added of cross border living labs networking .................................................... 19 3.1.5 Sustainability................................................................................................................................. 20 3.1.6 Harmonization .............................................................................................................................. 20 3.1.7 Collaboration ................................................................................................................................. 21 3.2 Energy efficiency pilot ........................................................................................................ 22 3.2.1 Approach ......................................................................................................................................... 22 3.2.2 Pilot objective ............................................................................................................................... 22 3.2.3 Methodology evaluation and impact ................................................................................... 23 3.2.4 Value added of cross border living labs networking .................................................... 24 3.2.5 Sustainability................................................................................................................................. 25 3.2.6 Harmonization .............................................................................................................................. 26 3.2.7 Collaboration ................................................................................................................................. 26 3.3 eManufacturing pilot ........................................................................................................... 27 3.3.1 Approach ......................................................................................................................................... 27 3.3.2 Pilot objective ............................................................................................................................... 27 3.3.3 Methodology evaluation and impact ................................................................................... 28 3.3.4 Value added of cross border living labs networking .................................................... 28 3.3.5 Sustainability................................................................................................................................. 29 3.3.6 Harmonization .............................................................................................................................. 30 3.3.7 Collaboration ................................................................................................................................. 31 3.4 eParticipation pilot .............................................................................................................. 31 3.4.1 Approach ......................................................................................................................................... 31 3.4.2 Pilot objective ............................................................................................................................... 31 3.4.3 Methodology evaluation and impact ................................................................................... 32 3.4.4 Value added of cross border living labs networking .................................................... 32 3.4.5 Sustainability................................................................................................................................. 34 3.4.6 Harmonization .............................................................................................................................. 34 3.4.7 Collaboration ................................................................................................................................. 35 3.5 Evaluation and impact profiles of the pilots ............................................................... 37

ICT PSP Project Reporting Template

3

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

4.

5.

6.

Business Impact Assessment .................................................................................. 46

4.1 Business impact assessment of SMEs ............................................................................ 46 4.1.1 Business Profile of APOLLON SMEs ..................................................................................... 46 4.1.2 SMEs Internationalization ....................................................................................................... 49 4.1.3 APOLLON Contributions and Impact on SMEs ................................................................ 52 4.1.4 End of APOLLON: Directions and Exploitation Strategy SMEs ................................. 53 4.1.5 Living Lab Experience of SMEs .............................................................................................. 57 4.2 Corporate business impact assessment ....................................................................... 58 4.2.1 Business profile of APOLLON Corporate Partners ........................................................ 58 4.2.2 Corporate Internationalization.............................................................................................. 61 4.2.3 APOLLON Contributions and Impact .................................................................................. 62 4.2.4 End of APOLLON: Directions and Exploitation Strategy ............................................. 63 4.2.5 Living Lab Experience of corporate partners .................................................................. 65 4.3 Conclusions ............................................................................................................................. 66

Evaluation of Central APOLLON Objectives ....................................................... 68

5.1 Quantitative evaluation of the pilots ............................................................................. 68 5.2 Contribution of methodology development ............................................................... 71 5.2.1 Process of methodology development and validation ................................................. 71 5.2.2 Methodology framework and specific methods and tools ......................................... 71 5.2.3 Methodology validation ............................................................................................................ 72 5.2.4 Main suggestions for improvement ..................................................................................... 72 5.3 Value added of cross border living labs networking ............................................... 73 5.3.1 Cross-border collaboration focus ......................................................................................... 73 5.3.2 Benefits of cross-border networking .................................................................................. 73 5.3.3 Challenges of cross-border networking ............................................................................. 74 5.4 Sustainability of cross-border living labs networks ................................................ 78 5.5 Collaboration in cross-border networks ..................................................................... 78

Conclusions and Outlook ......................................................................................... 81

6.1 Introduction ........................................................................................................................... 81 6.2 Discussion of vertical pilot evaluation results ........................................................... 81 6.3 Summary of main recommendations derived from the vertical pilots............. 87 6.4 Final remarks on objectives and results achieved ................................................... 89 6.5 Apollon Success Stories ...................................................................................................... 90 6.5.1 New product Lines ...................................................................................................................... 90 6.5.2 New Business Partnerships .................................................................................................... 91 6.5.3 New Mergers and Acquisitions .............................................................................................. 91 6.5.4 New Spin Offs and Markets ..................................................................................................... 91

References ................................................................................................................................ 93 Appendix 1: SME Profile Overview .................................................................................. 95 Appendix 2: Corporate Profile Overview ...................................................................... 99 Appendix 3: Evaluation and Impact Assessment Framework ............................. 102

ICT PSP Project Reporting Template

4

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

1. Introduction 1.1 Role of D1.6 within the APOLLON project The deliverable D1.6 comprises the Evaluation and Impact Assessment report of APOLLON. Following the Description of Work (DoW) regarding T1.5 and correspondingly D1.6, the evaluation and impact assessment activity within APOLLON addresses the following aspects: •

•

•

•

Methodology. Evaluation of methodology should address the structure, completeness, cohesiveness and relevance of the developed methodology, relevance and multi-contextual extension of the network, development of sector specific specializations and sustainability of the network.

Value for Stakeholders. Evaluation addressing the stakeholders will assess the impact on various identified user groups or beneficiaries of the cross-border networking of living labs. Stakeholders include SMEs, Living Labs, end users, researchers, and other local ecosystem stakeholders. On the high level, evaluation is performed on the based of identified key performance indicators (KPI’s) defined in the D1.3 [1] and on the applied level, in-depth business impact assessment is performed to demonstrate the value add of APOLLON and crossborder networking on individual firms and corporations.

Sustainability. Evaluation will present the sustainable results of APOLLON in terms of best practices, lessons learned, methodology & toolset and impact it has generated.

Lessons learned. Evaluation at project level includes a multi-criteria impact assessment of the evaluation of the APOLLON project, including lessons learned and ideas for further research.

D1.6 presents a coherent and integrated set of results and impacts. It is based on applying the framework for APOLLON Evaluation and Impact Assessment including the definition and measurement of Key performance Indicators (KPIs), which allows a systematic assessment at pilot level of the APOLLON methodology and the added value of cross border Living Lab networking in terms of specific KPIs. In order not to duplicate the evaluation reports of the four pilots, we will reference to the underlying pilot evaluation deliverables.

The APOLLON project evaluation is based on the evaluation and impact assessment results of the four pilots, and using these evaluation results it draws conclusions at the APOLLON project as a whole. The evaluation therefore thus covers two “layers”:

•

Evaluation of the project as a whole. Evaluation at project level includes the analysis of impact and value added of the project, including the impact on stakeholders, and a benchmark against original objectives. Furthermore evaluation addresses the role and contribution of methodology, and the way

ICT PSP Project Reporting Template

5

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

•

harmonization and interoperability issues are resolved. Project evaluation results into lessons learned at project level.

Evaluation of the four pilots. Outcomes and results from the pilots are evaluated with respect to issues such as: achievement of stated goals, identification of key factors determining the pilot results, resulting impacts and added value of cross border living lab networking for stakeholders, and the sustainability of the living labs networks created.

Resulting from the evaluation and impact assessment activities, this deliverable includes various best practices, concrete examples of impacts APOLLON has generated, lessons learned, and suggestions for follow-up research activities.

APOLLON project evaluation is based on comparing achieved results against objectives set in the beginning. Therefore, systematic evaluation and impact assessment starts with defining a logical framework of the APOLLON objectives and how their achievement can be determined.

1.2 Evaluation based on APOLLON objectives framework

As a starting point and reference framework for APOLLON project evaluation and impact assessment, we first summarize and formulate the objectives that APOLLON aims to achieve: The general objective of APOLLON is to pilot and assess cross-border domainspecific networks of Living Labs, to deliver validated methodologies and tools for this, and to ensure that such networks can offer benefit to all relevant stakeholders, in particular to SMEs, in a sustainable way (Description of Work).

Therefore three central objectives are distinguished which specify the desired APOLLON achievements. For each central objective, specific objectives or target outcomes are identified which represent the specific challenges and target outcomes described in the Description of Work. Evaluating the achievement of the objectives is determined on basis of a set of information categories and indicators.

The APOLLON central objectives are:

1) Demonstrate the value added of cross-border living lab networking; 2) Develop a methodology for cross-border living lab networking; 3) Ensure long-term sustainability of cross-border living labs networking.

These three central objectives are closely related. Sound methodology enables the realization of value added and also contributes to achieving sustainability. Achieving sustainability can be determined by the delivering value to the engaging stakeholders. While we recognize the overlap between the objectives, they represent important achievements in their own and are designated as key APOLLON objectives.

Besides, the DoW also emphasizes the importance of addressing a set of challenges, such as resolving interoperability and harmonization issues for enhancing cross

ICT PSP Project Reporting Template

6

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

border living labs collaboration. These issues are context dependent and are explored in the pilot environments, where different harmonization and interoperability approaches are implemented and experimented. We understand them as important conditions for achieving stakeholder added value and sustainability. Other challenges include methodology, sustainability, ecosystem rules and roles, scalability and integration. The challenge categories are explicitly included in the framework of objectives and indicators. Fig. 1-1 presents a logical framework of objectives and their interrelations.

Fig. 1-1: Logical Framework of APOLLON objectives

APOLLON also formulates a number of specific target outcomes, which clearly are linked to the objectives mentioned. Where appropriate we see them as “indicators� to be used to measure the achievement of objectives. An example is the setting up of thematic networks. This serves the higher-level objective of creating sustainable cross-border living labs networks.

The issue of cross-border collaboration is directly included in the central objective of demonstrating value added of cross-border networking. On the methodology side collaboration guidelines are proposed, covering the over-all process of setting up, planning and running cross-border networks of living labs. We now first shortly comment on the central APOLLON objectives and how they relate to underlying targets or specific objectives.

1) Value added of cross-border living labs networking.

Cross-border networks of Living Labs, in the four domains considered, aims at creating benefits and value added for all the stakeholders involved in the ecosystem.

For example, the cross-border network aims to generate and exploit innovation and ICT PSP Project Reporting Template

7

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

market creation possibilities, to ease the transfer of existing technologies and solutions, and to contribute to a more efficient and effective take up of ICT services and applications. In this respect, best practices are to be identified, demonstrated and shared.

Of particular attention is the value added of cross border living lab networking for SMEs. Main objective here is to provide SMEs with an entry point to larger European markets in cooperation with more established, larger companies; to act as a nucleus for sharing innovative RDI activities within the thematic experiment as well as to find different parties that can contribute to these activities; to act as a gateway for SMEs, through the networks of Living Labs, towards the different EU Member States and regions to explore possible markets; and to construct an open environment in which all types of stakeholders, with a special focus on SMEs, can enter and participate. Besides for SMEs, the cross-border living labs networking should create value added for living labs and other parties.

Creating value added would be dependent on realizing conditions and underlying goals, such as resolving the challenges with respect to the interoperability and harmonization of cross-border networking.

2) Effectiveness of the methodology for cross-border Living Lab networking.

The APOLLON methodology includes the processes, guidelines and toolsets to enable the creation, planning and operation of cross-border networks of living labs.

It also includes, as a specific dimension, the development of guidelines for collaboration. Cross border networking and creating value added of networking will benefit from and even be dependent on the availability of appropriate processes, tools and methods. The objective of APOLLON is to validate the methodology and its elements, and to formulate recommendations concerning such set of methods, tools and guidelines and their application.

We see a direct relation between methodological approach and the issue of resolving interoperability issues and harmonization challenges between the different Living Labs forming a network (creating a common ecosystem, a common benchmark approach, a common platform, a common integration framework). For this purpose, it will be identified for the different domains what the determining local contextual factors are, how they can be dealt with and how the interoperability and harmonization issues have been tackled.

Methodology evaluation includes: what has been the use and value added of the methodology for that pilot, what are factors determining success or failure, and efficiency and effectiveness of the process of methodology introduction and adoption. It also includes general qualities such as structure, completeness, cohesiveness and relevance of the methodology.

3) Sustainability of cross-border living labs networks.

The third central objective of APOLLON is to ensure sustainability of cross-border living labs networks. This objective includes different aspects such as proposing

ICT PSP Project Reporting Template

8

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

governance models for living labs networks, and to embed these networks in wider thematic cross-border domain networks. Such thematic cross-border networks will act as “breeding ground� but also enable distribution and dissemination of information on new technologies and services that are developed and tested in the different Living Labs.

1.3 Evaluation and impact assessment sources

As was mentioned in section 1.1, evaluation and impact assessment regarding the objectives mentioned covers two different aggregation levels: 1) project over-all, 2) pilots (and within pilots even the level of specific experiments).

1) The APOLLON project evaluation as a whole focuses on the impact and benefits of the over-all project including its combined pilots results in terms of value added of cross border living labs networking, methodology support and sustainability of the network.

2) The four APOLLON pilot evaluations address the evaluation of the demonstrated impact and benefits of cross border living labs networking and collaboration created for stakeholders, including for SMEs and Living Labs, focuses on understanding of the local context factors and harmonization challenges.

These levels of aggregation imply that evaluation starts at pilot level, and through analysis of pilot level evaluation results we will be able to draw conclusions at the APOLLON project level over-all. This deliverable makes use of the evaluation-related activities already carried out in the vertical pilots and also of evaluation activities within other WP1 tasks related to methodology and to collaboration analysis. We shortly elaborate on these sources:

Pilot evaluation. For the purpose of pilot evaluations as well as to identify the added value of cross border Living Lab networking, an evaluation and impact assessment framework has been developed within the task T1.3 and reported in D1.3 [1]. This framework defines and develops key indicators and mechanisms for measuring the key performance indicators. The framework has been applied in the four different vertical domain work packages (WP2 to WP5) to evaluate the different vertical experiments in relation to the general objectives and the overall APOLLON methodology. This results into evaluation reports of the four APOLLON pilot experiments prepared by the vertical work packages.

Methodology validation. Within T1.2, a framework has been developed addressing the validation of APOLLON methodology, which is presented in D1.5[3] along with methodology validation results. Although this validation concentrates on the development, adoption and use of methodology, the validation results are useful for evaluation of the impact and effectiveness of the methodology and the value added of cross-border networking as well.

Collaboration and networking analysis and guidelines. Within WP1 (D1.4[2]) we specifically and in detail addressed collaboration in the context of setting up and ICT PSP Project Reporting Template

9

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

running the cross border living labs networks. Within all pilots collaboration was analysed, best practices extracted and further collaboration guidelines proposed.

Project level indicators evaluation. APOLLON partners are reporting on several aspects of project impact, such as number of living labs collaborating within the living labs networks, number of RDI projects in living labs and many more. These performance indicators will be taken into account in this evaluation report. Also, results from dissemination and exploitation activities, for which WP6 (D6.3 [14], D6.5 [15]is responsible, will be included in our analysis.

1.4 Overview of this report

Chapter 2 presents the over-all approach to APOLLON evaluation and impact assessment activities. It elaborates the evaluation and impact assessment framework and its implementation. This framework is based on the identification of objectives, targeted outcomes and indicators.

Chapter 3 presents the evaluation results at pilot level, for the four APOLLON pilots.

For each pilot the chapter addresses the methodology impact, value added of the cross-border networking, resolving the harmonization challenges, and collaboration within the cross-border network.

Chapter 4 specifically addresses the business impact of APOLLON, from the perspective of SMEs and larger companies participating to the APOLLON consortium.

Chapter 5 takes a cross-pilot viewpoint in addressing each of the central objectives of APOLLON based on cross-analysis of pilot level evaluations for these objectives.

This chapter thus presents the main evaluation results.

Chapter 6 takes an over-all project level view to summarize the main conclusions, lessons learned and recommendations.

ICT PSP Project Reporting Template

10

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

2. Project Evaluation and Impact Assessment Framework 2.1 Introduction As explained, a systematic evaluation and impact assessment takes its point of departure in a logical framework of objectives which serves as reference towards measuring the achievements. The actual measuring and assessment then takes place at two aggregation levels: pilot level, and the project as a whole.

APOLLON Pilot level evaluation and impact

APOLLON Project level evaluation and impact

Objective 1: Value added of crossborder living labs networking

What is the value added of cross border living lab networking within the pilot

What is the value added of cross border living labs networking and how to optimize it given local factors

Objective 3: Sustainability of the cross-border network

What is the sustainability of cross border living lab networks achieved in the pilots

What is the sustainability of cross border living labs networks and how to optimize it given local context

Objective 2: Methodology development

Challenges: Addressing harmonization and interoperability

What has been the contribution of methodology development to cross border living lab networking at pilot level

To what extent have harmonization and interoperability challenges been resolved within the pilot Lessons learned and conclusions at pilot level, taking into account local context

What is the contribution of methodology development to cross-border living labs networking in general and how to optimize it

How has APOLLON resolved harmonization and interoperability challenges and what are the local factors Lessons learned and conclusions at project level, taking into account local factors

Table 2-1: Evaluation and impact assessment matrix

This results in the “evaluation and impact assessment matrix� as depicted in Table 2-1. The different evaluation and impact analysis concerns must be addressed at the two aggregation levels identified.

2.2 Evaluation and impact assessment logical framework

The experiments conducted in the four pilots generate the basic information to evaluate the realisation of the APOLLON objectives and target impacts. Additionally, cross-pilot evaluation analysis allows us to draw generic conclusions, taking into account the specific context factors in the pilot environments. In the following we take these objectives as starting point to evaluate in how far these objectives have been achieved and which lessons can be learned.

ICT PSP Project Reporting Template

11

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report Table 2-2: Objectives framework for APOLLON evaluation and impact analysis APOLLON over-all objective The general objective of APOLLON is to pilot and assess cross-border domain-specific networks of Living Labs, to deliver validated methodologies and tools for this, and to ensure that such networks can offer benefit to all relevant stakeholders, in particular to SMEs, in a sustainable way (DoW, 2011)

APOLLON Central Objectives (DoW) 1.

Demonstrate the value added of cross-border living lab networking

APOLLON Targeted Achievements and Outcomes (DoW) 1.1 Create specific added value in terms of results / benefits as well as operational efficiencies of the cross-border approach (for all partners in total, and / or from perspective of specific targets) 1.2 Create value added for specific stakeholders (for specific partners: SMEs, Living Labs)

2.

Develop a methodology for cross border living labs networks

1.3 Realize the harmonization and interoperability of Living Lab approaches and platforms between networks of exemplary European Living Labs

2.1 Develop a set of validated methodologies to set up, plan and conduct cross-border Living Lab networks

Ensure sustainability of cross-border living labs networking

• • • • • • • • • •

• • • •

2.2 Develop and share a recommended toolset and collaboration guidelines for facilitating cross-border research and innovation activities 3.

Indicators and information sources available for evaluation and impact assessment

2.3 Implement successfully the different harmonization and interoperability approaches needed for cross-border living labs networking

3.1 Inform and network stakeholders in Europe for specific thematic domains

•

•

• • • •

3.2 Develop governance and business models to support sustainable networks of living labs

• •

Pilots of cross-border living labs networking demonstrate the value added and benefits, given the domain context.

Improvements in ICT product and service innovation Operational efficiencies of cross border approach Exchange of processes, methods, approaches, results Viability of the cross-border living lab networking activity SMEs participation in Living Lab networking Value added for SMEs (business case) Value added to Living Labs (business case) Benefits of collaboration in the network for individual partners and for the whole Have the pilots successfully implemented the contextual harmonization / interoperability approaches, to what efficiencies, benefits and impacts

Methods that are used, adopted; level of adoption Impact of methods and tools on the creation and operation of crossborder living labs networks Contribution of methodologies to effectiveness of collaboration for cross-border living labs networks Effectiveness of collaboration processes and tools

Set of recommended methods, tools, processes, guidelines as part of a “Knowledge Center” Pilots to develop and validate approaches for harmonization and interoperability: Common ecosystem, Common benchmark, Common platform, Common integration framework

Set up of thematic networks of living labs across Europe Ensure participation to thematic networks Contribution of thematic networks to sustainability of cross border living labs networks Dissemination activities to inform and network stakeholders Develop governance and business models Validation of proposed governance and business models

APOLLON – D1.6 Evaluation and Impact Assessment Report Item

Measures

Indicators (examples)

Approach; methods and tools for crossborder collaboration

Methodologies, know-how actually used in the pilot experiments

Which methods and tools were used, which impact

Cross border collaboration and communication tools actually used to support cross-border collaboration

Which tools were used Impact on collaboration

Technology and infrastructure’s role to facilitate collaboration

Experiences, benefits, challenges in using the methods

Benefits and added value of using cross border collaboration tools

Users role and benefits from cross border collaboration

End user involvement, involvement in crossborder network

SMEs role and benefits from cross-border collaboration

Involvement of SMEs, new international partners

Living labs role and benefits from crossborder collaboration Research related benefits of cross border living labs

Ideas, implementations, products, services emerging from cross-border collaboration

Users’ role in cross-border collaboration activities Business impact on SMEs

Impact on skills, knowledge, image

Impact on testing products, services

Involvement of living labs in the cross-border collaboration network Benefits and value added of living labs involvement Impacts on living labs role, maturity

Insights from research on cross-border living labs collaboration

Bottlenecks hindering the use of methods and tools

Co-creating activities Collaboration effectiveness Access to information Transfer of technology and solutions Involvement activities Redesigned products

User-generated ideas and services Involvement activities Involvement activities Business generated, proposals, customers Skills enhancement Testing activities

Network created by living labs

Collaboration initiatives Network members new international partners Business generated Management quality

Publications New methods and tools based on research findings Lessons learned disseminated

Table 2-3: Pilot Evaluation and Impact Assessment (summary of items and measures)

For the purpose of gathering evaluation and assessment information we are using the logical objectives framework depicted in Table 2-2. This framework disaggregates the APOLLON objectives and distinguishes between: 1. APOLLON over-all objective

2. The three central objectives

3. For each central objective, a number of targeted achievements and outcomes

APOLLON – D1.6 Evaluation and Impact Assessment Report

4. For each of these targeted outcomes, a checklist of indicators and information sources.

For the purpose of gathering data and observations at pilot level, we use the Pilot Evaluation and Impact Assessment framework which was developed as a tool for pilot evaluations (D1.3[1]). The aspects that are covered by this tool are summarized in the Table 2-3. The tool addresses activities, context factors and objectives.

2.3 Types of evaluations and their implementation

For the different types of evaluation identified, we must distinguish between different purposes of the evaluation, different frameworks needed, different information sources and different ways of reporting. Our goal is to create a consistent, reusable framework which enables the use of different types of partial evaluation results from pilot evaluations covering the full set of objectives into the overall APOLLON project evaluation. At the same time, the framework and its underlying information base will allow specific “views”, focusing on different evaluation concerns and aggregation levels. A first step towards an integrated evaluation framework is to recognize the interactions and relations between the different evaluation types.

2.3.1 Evaluation of achieving the objectives

Value added of cross border living labs networking: emphasis is on the value added for particular stakeholder groups: end-users, SMEs, living labs as well as large companies and also research. Actual measurement of value added is in terms of successful technologies and solutions transfer, new business generated, new partners engaged, etc.

Methodology – the set of methods, approaches and tools for setting up, planning and operating the living labs network - forms part of the piloting infrastructure.

Methodology supports the process of creating and operating cross border living lab networks. Therefore methodology evaluation is “immersed” in pilot evaluation and pilot results are affected by the methodology as developed and used. In turn, overall project evaluation compares and synthesizes findings from all pilots.

Sustainability of the cross border living labs network: this aspect has received less attention in the pilot evaluations; however it is a topic in the D6.6[16].

Sustainability mainly depends on the strength of the collaboration network including the wider thematic network at European level as well as the existence of agreements and governance or business models.

2.3.2 Evaluation at pilot and project level

For evaluation, it is important to assess the achievement of objectives at pilot as well as at project level.

ICT PSP Project Reporting Template

14

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

At project level, the objective is to demonstrate the impact or value added of crossborder living labs networking. Beneficiaries include living labs organisations, SMEs, large companies, and also governments at local and regional level.

At pilot level, the objective is equally to demonstrate the impact or value added of cross-border networking, for different types of beneficiaries. Here, the specific needs, circumstances and challenges of the pilot environment must be taken into account. This leads to a more balanced evaluation at project level.

Table 2-4 presents the basic characteristics of the mentioned evaluation aggregation levels in terms of focus and purpose, data needs and frameworks used.

Level

Project level evaluation

Pilot level evaluation

Aspects Evaluation focus and purpose

Data needs

Evaluation Frameworks Deliverables

Impact of APOLLON project as a whole, on its beneficiaries at project level

Pilot impacts in terms of success or failure to achieve results useful for beneficiaries

Value added of cross border networking, for its beneficiaries at project level

Value added of cross-border living labs networking at pilot level

Impact indicators at project level

Impact indicators at pilot level

Role of project specific context, conditions, other factors determining impact and added value Value added indicators at project level Project specific factors and their influence

Evaluation and Impact Assessment framework (Table 2-1) D1.6

Role of pilot specific context, conditions, specific factors determining impact and value added Value added indicators at pilot level

Pilot specific factors and their influence Pilot Evaluation and Impact Assessment framework of D1.3 [1] D2.4[5]; D3.5[8]; D4.4[10]; D5.5 [12]

Table 2-4 Characterization of evaluations

The diversity of these aspects highlights the importance of efficient data collection and reporting, careful structuring and planning the evaluations. This document provides the structure and planning of APOLLON evaluations in WP1, as well as responsibilities for implementation and reporting. Actual evaluation and methods templates used are described in other documents.

2.3.3 Description of the different evaluation source materials Evaluation of pilots The pilot level evaluation addresses the four living labs networking pilots as experiments whose aim is to demonstrate and investigate the opportunities and ICT PSP Project Reporting Template

15

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

enabling conditions of cross border living labs networking for SME innovation. As such the evaluation activity targets two different aspects: •

•

Has the specific purpose of a pilot been achieved? This evaluation includes the analysis of contextual factors in determining how specific harmonization and interoperability challenges have been addressed.

Has the generic purpose of a pilot been achieved? This evaluation addresses the demonstration of the value added of cross-border living labs, the methodology, and the sustainability and investigates the factors determining success and failure in this respect.

Specific purpose: resolving the harmonization challenges

As regards the specific purpose, evaluation focus is on each pilot’s specific results, approach, and factors determining the pilot outcomes. Each of the pilots has identified specific contextual aspects that play a role in the evaluation: •

• •

•

Pilot on Homecare and Assisted Living (WP2) aims to investigate different contextual factors that impact Homecare and Independent Living in order to be able to evaluate to what extent the network can be used for exploring new and emerging markets, For example, the WP2 explores the role of market structures, local value network constellations, interoperability issues and other aspects.

Pilot on Energy Efficiency (WP3) has the purpose to test the impact of real time data with consumers’ energy behavior.

Pilot on eManufacturing (WP4) has a twofold purpose, including to evaluate the MDI platform and its service development capabilities (functional evaluation) as well as to evaluate collaboration capabilities (human centric evaluation) such as the effectiveness of the collaboration environment.

Pilot on eParticipation (WP5) has the purpose to assess the impact of the Living lab activities on eParticipation using eMedia and recommendations for improving the specific methodology for eParticipation. The impact will be measured with success indicators for SMEs and Living Labs.

The specific purpose evaluation has been carried out by each vertical pilot respectively, and has been reported in pilot evaluation deliverables. The WP1 (T1.3) has provided support for harmonizing the specific purpose evaluations.

Generic purpose: achieving the central APOLLON objectives

As regards the generic purpose, the evaluation focuses on the central objectives of APOLLON: 1) the generated added value of cross-border networking based on the experiences from each experiment (or experiments within an overarching experiment); 2) the methodology; 3) the sustainability of the living labs network.

These evaluations are supported by an evaluation and assessment framework which has been developed in T1.3. The evaluation framework provides key performance indicators such as new collaborations, new products and access to new markets that ICT PSP Project Reporting Template

16

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

stem from the collaboration. This evaluation is carried out by the Living Labs in the experiments and supported by the liaisons in WP1, together with the other partners involved depending on the experiment.

The methodology validation as presented in D1.5[3] addresses the cyclic, learning based process of identifying, developing, introducing, adopting or rejecting, and eventually using the methods and tools, and joint learning. Methodology evaluation focuses on the impact of methodology for the pilots in creating living labs networks.

Methodology evaluation questions are:

•

• •

How have methodologies been used, found useful and have created value in supporting cross border living labs networking, including collaboration?

What are the factors affecting value added and impact of the methodologies?

Has the methodology introduction and adoption process been efficient and effective?

For methodology validation, data has been gathered concerning the cross-border networking process, the methods and tools used to support this process and the experiences from this usage. A detailed template has been developed in order to monitor and support the methodology life cycle process from a bottom up approach.

The methodology validation is carried out at pilot level. Findings from pilots will be compared and analyzed, and generalized, using the results from both pilot evaluations and methodology validations.

ICT PSP Project Reporting Template

17

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

3. Evaluation and Impact Assessment at Pilot Level 3.1 Homecare and independent living pilot 3.1.1 Approach Evaluation and impact assessment for this pilot is based on the WP2 pilot evaluation report D2.4[5] as well as on inputs from the WP1-WP2 liaison persons. The Pilot Evaluation and Impact Assessment template, offered by WP1 (see D1.3[1]) has been applied. The evaluation and impact assessment is summarized in a “pilot evaluation profile” at the end of this chapter.

3.1.2 Pilot objective

The Homecare and Independent Living pilot sets out to investigate the different contextual factors that impact Homecare and Independent Living RDI, and to understand specific Homecare and Independent Living market structures as well as to generate interoperability protocols for transferring Homecare and Independent Living experiments between different Living labs. These objectives were addressed by means of conducting several specific experiments in a cross-border Living Lab setting. Within the over-all pilot following activities have been undertaken: • • • •

Defined and compared the different Homecare and Independent Living Lab approaches.

Developed a transfer strategy and working methods

Validated the used methods and tools for cross-border pilots in the Homecare and Independent Living network

Mapped the different contextual factors, including the regulatory and legal issues as well as the service/product value-chain and eco-systems

Within the pilot it was also evaluated to what extent the network can be used for exploring new and emerging markets within the Homecare and Independent Living domain.

3.1.3 Methodology evaluation and impact Use of methods and tools

The actual use and value of methods and tools has been mixed. The most relevant and useful methods and tools proved to be the Research Framework [4] which helped to understand and focus the research elements of the experiments - the Ecosystems analysis, Requirements analysis, Business model design, and the normal web-based communication and collaboration tools. Several of these methods and tools are made available through the Knowledge Center [17].

ICT PSP Project Reporting Template

18

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

Within the pilot project team, there has been extensive collaboration across borders between all the parties involved, using online tools such as monthly Skype sessions, E-mail, Doodle [18]. The shared workspace MyBBT [19] has been used for document sharing.

The Research Framework [4] (consisting of structured questions that help to define what the actual research is about and how the data collection and analysis will be structured) was utilized and found useful. The impact was found mostly relevant as regards the relevance and significance and reliability of data collected.

Table 3-1 provides an evaluation summary of the methods and tools that were used in this pilot, in terms of impact.

1 No impact

2 Limited impact

3 Moderate impact

4 Favourable impact

5 Considerable impact

Ecosystems analysis

Research framework Requirements analysis

Business model design

Collaboration / communication tools

Table 3-1: Evaluation of methods and tools value

Unused methods and tools The methodology elements directed to setting up and planning the cross border network (connect, plan and engage phases) have not been used as the status of the pilot was already beyond these phases.

Pilot experimentation methodology as such has not received explicit attention however it must be underlined that the careful use of such methods, which are also related to project management, is highly critical for success.

3.1.4 Value added of cross border living labs networking

Cross border living labs networking has created value added at several levels. The most significant benefit is the continuous knowledge and information sharing taking place within and between the projects. The structure of the project and the corresponding structure in the pilot experiments enhanced the collaboration and facilitated comparison of different aspects of their experiment, for example ecosystem analysis, technical setup, user selection and other items of common interest.

Value added has been created for different types of partners: •

For Living Labs: the cross border aspect of the collaboration offered experiences and new insights into the dynamics of international collaboration.

ICT PSP Project Reporting Template

19

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

•

•

•

SMEs: participation within the collaborating network of living labs has provided SMEs with experience in internationalization of products and services and better understanding of the product opportunities. Access to foreign ecosystems was found to be most valuable. For SMEs the value of the experience in cross border collaboration is in the learning of how to accomplish such cross border activities.

For an SME operating in small and open economy, such as Finland, The Netherlands or Belgium, an international expansion is a natural process of development, gaining experience in cross-border collaboration is deemed essential for its success in such domains.

Large enterprises: the collaboration offers opportunities for expanding the knowledge base, in terms of technology and markets conditions outside their local ecosystem.

Research and dissemination: insights from the collaboration have been disseminated, mainly by the Living Labs, on high profile events such as ICT 2010, ISPIM, the Living Labs Summer School and the PICNIC event. Presenting at such events extends the network and reputation of the living Labs. Details of speaking engagements, papers and other dissemination activities are documented in WP6 (D 6.3[14]).

3.1.5 Sustainability

The current pilot network, set up within the APOLLON project context, is project based. For the purpose of this project, agreements have been made between the pilot partners.

As part of the WP6 activity (D6.6 [16]) to develop a thematic domain network were initiated for the area of Health and Wellbeing. Potentially this network could act as breeding ground for projects and business in the domain. A set of domain network services is proposed, offered to living labs and SMEs.

Given the embryonic nature of network, the viability of such an initiative is hard to predict and to determine. However, the network is an example of the potentially sustainable nature of the collaboration between the living labs in the project. The four living labs participating in the pilot have expressed their interest in continuing their collaboration, and there is also interest in joining the group from at least 8 other parties. Most partners agree to the need for continuing the collaboration under the flag of the ENoLL [26] in such domain specific network.

3.1.6 Harmonization

Harmonization specifically focused on the development of a common ecosystem for eHealth Living labs. Already in an early stage this was proven to be not feasible.

Healthcare is a strictly regulated business that has many national aspects, in terms of laws, regulation, practices, financial streams and cultural norms. As such it is not possible to build or transfer a specific ecosystem in to an alien ecosystem, instead it was decided to describe the identifiable aspects that are common in every ecosystem. As a next step, key business actors and their roles in country-specific ICT PSP Project Reporting Template

20

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

(Belgium, Netherlands, Finland and Spain) ecosystems were identified and elaborated by the pilot.

For other harmonization aspects of the pilot the focus has been mainly on collaboration via the use of comparable tools and techniques. Although there were considerable differences between the receiving ecosystems, the technologies transferred and other issues, the methods used in setting up the pilots and executing the experiments were highly harmonized.

3.1.7 Collaboration

Facilitating the transfer process, the collaboration between Living Labs both at the home and remote locations played a critical role. Not only these living labs acted as a “central hub” for the SMEs connecting them with appropriate partners, they also performed most of the set-up and execution activities of the cross-border project.

However, establishing cross-border collaboration in this pilot encountered the following main issues: •

•

•

A cross-border collaboration Living Lab pilot in the domain of health is not just collaboration between partners. Such collaboration must be embedded in the existing ecosystem, bringing in stakeholders that already occupy a key-role in the market.

The objective of the Living Lab pilot is not the experiment in itself. It must be driven by a clear market objective and goal by every stakeholder. This requires the creation of a shared vision and expectation of all the involved partners based on their needs, interests and motives. However, in the healthcare domain the visions of the societal and economical stakeholders are often diverging. The challenge is to translate all this in the project set-up and to safeguard the expected added value for each partner involved.

Creating the required eco-system and effectively facilitating technology transfer also challenges the role of the Living Lab as a central hub. Living Labs are being confronted with new roles, and the new set of activities to be performed often goes beyond the initial objectives, activities and competencies of the Living Labs.

During the set-up and deployment of the three experiments, a number of important issues must be taken into account. It is necessary to start with a clear objective and identified added value for all participants that participate. Moreover, these goals and needs should be aligned with and framed within the daily operations or a specific roadmap of the SMEs and additional partners. If not, the motivation and willingness to collaborate and to adjust during the pilot will be limited.

Subsequently this willingness must also be reflected in identified business opportunities after completion of the project resulting in a continuation, based on the results and within a normal commercial market environment of the eco-system as set-up in the pilot.

To conclude, the homecare and independent living pilot is structured as a set of project-like experiments in which a technology solution is transferred from one ICT PSP Project Reporting Template

21

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

country to another. The pilot collaboration needs are largely determined by the need to set up a local ecosystem (preferably similar as the initial market or region, but adjusted to the local context), engaging local stakeholders that fit within that ecosystem, and the adaptation of the solution to the local context. For this no ready to use methodologies exist apart from usual approaches such as requirements definition and normal standards of project management and stakeholder involvement. These however, often lack elements that focus on (1) the contextualisation, (2) the domain where a pilot is performed, and (3) the specific ecosystem in which it has to be deployed. During the APOLLON project we therefore created various methods, such as requirement and value analysis, which aim to capture the specifics of such cross-border collaboration of living labs.

3.2 Energy efficiency pilot 3.2.1 Approach

Evaluation and impact assessment for this pilot is based on the WP3 pilot evaluation report as well as on observations from the WP1-WP3 liaison persons. The Evaluation and Impact Assessment template, offered by WP1 (see D1.3[1]) has been applied. The evaluation and impact assessment is summarized in a “pilot evaluation profile” at the end of this chapter.

3.2.2 Pilot objective

The Energy Efficiency pilot has set out the following concrete objectives: •

•

•

•

•

To create a sustainable network of innovative lead market Living Labs in Europe to address common challenges related to regulatory issues in the so-called ’free last energy mile market’.

To enable household and citizen level validation and empowerment for active role in energy saving, innovative distribution and even areas of local energy productions.

To promote strongly local level SME innovation and create European level synergies to these companies in scaling their market reach in ICT enhanced energy efficiency domain.

To project new emergent value constructions and business models in the liberated energy market through these pilots from the pilot user, community or local SME perspective, and to highlight the related regulatory issues.

To pilot a common benchmark framework, and derive general guidelines related to this. This framework will assess the scalability of the network services and the comparability of research data within cross-border projects.

Each of the Living Labs used its existing project. Instead of transferring these systems, the Energy Efficiency experiment focused on creating a common research benchmark. This research benchmark should centrally steer local research activities ICT PSP Project Reporting Template

22

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

in order to facilitate 1) comparison of different results of each of individual projects and 2) experimentation with successful outcomes of one Living Lab in another.

3.2.3 Methodology evaluation and impact Overview The most frequently used methods and tools in the cross-border experiments in this pilot are summarized in Table 3-2. These include the Research Framework, Project Plan, Road shows, and the Behavioural Transformation approach.

Methods & Tools

Impact

Research framework

Research framework was used for each experiment. This was useful especially for structuring the project. It was fundamental for organizing the pilot research and evaluation work.

Project plan Road shows

Behavioural Transformation approach (including Test Storyline)

Project plan enabled and facilitated project management and monitoring.

Road shows have been useful for the companies in knowledge and business transfer, as companies (incl. SMEs) involved in the local ecosystem have been attending them.

Behavioral transformation approach has enabled transferring best practices within the pilot, structuring the experiments, as well as increased the competencies among the pilot partners.

Table 3-2: Methods used in the pilot

Methods and tools having direct impact Most visible direct impact of using the methods and tools was the following: •

• • • • •

User behaviour transformation methodology: this methodology was supporting the testing of energy monitoring and enabled the tracking of users’ energy consumption behaviour.

Test storyline: this method provided structure for the test where tasks are given and questions related to that task are described in detail; Workshops: these were enabling knowledge sharing activities related to methodology for use transformation;

Knowledge transfer across pilots: these were enabling sharing experiences from processes among the partners in the pilot; Sharing best/good/worst practices between Partners (mainly companies).

Overall, knowledge of the socio-technical context has been increased. This has resulted in better definition of product requirements and understanding of markets and business environments.

ICT PSP Project Reporting Template

23

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

Methods having an enabling role Several methods have been used in the Energy Efficiency pilot that had an enabling role for the cross border living lab pilot. These were, for example, e-mail, shared website, teleconferences, and participating in APOLLON events, fairs, and seminars.

Overall the added value of the methods and tools were in increasing efficiency and savings in the pilots as well as for enabling follow-up. The added value of transferring technology in the cross-border experiment has been in enabling testing foreign technology and understanding the need for local service/maintenance.

Table 3-2 provides an evaluation summary of the methods and tools that were used in this pilot, in terms of impact.

Table 3-2: Evaluation summary of methods and tools, Energy Efficiency pilot 1 No impact

2 Limited impact

3 Moderate impact

4 Favorable impact

5 Considerable impact

Research Framework Project Plan Road shows

Behavioral Transformation

Unused methods and tools Business models design methodology “CANVAS” was demonstrated, but not used actively.

3.2.4 Value added of cross border living labs networking

The value added achieved through cross border networking of living labs includes several aspects.

• • •

•

For local governments (municipalities) the pilot has increased understanding of other environments where energy efficiency is piloted in Europe.

For living labs, the pilot has led to increased cross-border knowledge exchange between the four living labs.

For SMEs this has provided increased understanding of the complexities of creating new business in new markets between SMEs and larger pool of companies.

For large enterprises this has increased understanding of diverse best practice models and understanding the challenges involved in scaling in four different legacy markets (no single European market).

Participation in APOLLON has enabled the signing of a number of letters of intent between partners and customers, as well, as generated new businesses in other countries in terms of business proposals and proposal preparations. For SMEs and

ICT PSP Project Reporting Template

24

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

Large Enterprises this has enabled to evaluate their maturity to enter new markets.

It has increased knowledge on working in a big international project as well as the understanding of the European energy market.

Best practice and recommendations from Lulea were adopted within the pilot network. In addition, the road shows enhanced the understanding of the different businesses and markets. As a concrete example of the value added of the crossborder activities, SME in Portugal (ISA) adopted the business model of the Finnish Process Vision.

Knowledge of working in large international projects has increased as well as the understanding of the European energy market.

Understanding on customers’ needs and requirements of functions of products visualizing the energy consumption has increased.

The added value of the cross-border Living Lab network has been the increased knowledge base in the area of energy saving. This has lead to new project initiatives that in turn has increased collaboration with partners around Europe.

Another aspect of added value has been a strengthened collaboration with a local SME who had not been involved in similar project before. This has also lead to a better collaboration with them as well as with the supporting partner or partner SMEs in each country.

3.2.5 Sustainability

The current pilot network is project based, it was initiated through APOLLON. For the purpose of the project, agreements were reached between the partners. The pilot has attracted some attention from outside. Partners from outside the network have been included as business partners (at least 1 case.)

In the context of dissemination (WP6), activities to develop a thematic domain network were initiated for the area of Energy and Efficiency. Potentially this network could act as breeding ground for projects and business in the domain. A set of domain network services are proposed, offered to living labs and SMEs. It is too early to report about the viability of this network.

The pilot results indicate that cross-border networking can contribute to the promotion of new energy management business models, fostering SME’s and energy service companies’ participation in local and decentralized energy production in partnership with local authorities.

It has been found that SME’s are too small for cross-border networking. Entering the European market is still not their primary objective; instead their main focus lies in engaging with local and national stakeholders.

The pilot has showed that agencies can act as a bridge, or a link, between local authorities, SME’s and universities. A possible explanation for this is that they are perceived as having a neutral role. Moreover, results from cross-border networking can be used by local authorities, as foundation for new policies and regulations.

ICT PSP Project Reporting Template

25

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

As a result of the pilot, prospects for sustainability have emerged. New RDI initiatives between Living Labs have been developed at the national level and on the European level. Knowledge on cross border collaboration among Living Labs has increased. New proposals between Living Labs and the desire to continue APOLLON after 2012 have been expressed. Most SMEs are willing to continue with APOLLON type pilots.

3.2.6 Harmonization

In this pilot, the participation in APOLLON has lead to an increased comparability of Living Lab research among LTU, Aalto University, Alfamicro and Liander (all partners of APOLLON). The experiment has led to increased cross-border knowledge exchange between the four living labs.

3.2.7 Collaboration

Overall a good level of networking at each experiment level has been achieved, as well as across all experiments. This has lead to increased collaboration between SMEs and universities. Sharing of best practices, for example, by case demonstrations has succeeded well.

The cross-border collaboration between experiments regarding smart metering experiments did not function well within the pilot, as there was very limited collaboration between the SME’s in Lisbon, Amsterdam, Helsinki and Luleå. Instead, most collaboration has taken place between the living labs and their activities. Also, the four living labs had similar goals but very different technical architectures, and used different user involvement methods. Overall, collaboration has taken place around a diverse set of topics and issues: administrative issues, development of tools (project work), documentation, technological equipment, energy consumption behavior, sharing of methods, tools and knowledge, and workshops and road shows.

Within the pilot, methods have been developed jointly to support knowledge exchange among partners and methods to create business opportunities. Overall it was decided to have three different types of collaboration activities, namely 1.

Business matching and partnerships (e.g. road shows); 2. Technology testing (which required collaboration between many different actors in order to get the tests operational; the living lab designed the tests and SMEs provided technologies and was in direct contact with the user community); 3. Knowledge transfer; this has proved to be difficult to accomplish, but is regarded as vital for success. The challenge is how to expand solutions and lessons learned within one environment, in relation to environmental issues from another environment.

The energy efficiency pilot results into forms of cross-border interaction focusing mostly on dissemination of knowledge and experiences gathered in the experiments. The cross-border collaboration aspect is rather weak and actually SMEs involved did not have much interest to test their technologies abroad. One reason for this is the fragmented national European energy market, which is not supportive of cross-border activities. The pilots were using various methods such as ICT PSP Project Reporting Template

26

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

user behavior transformation approaches, and the contextualization of such approaches might constitute a benefit for cross-border collaboration. However the original purpose of establishing a benchmark framework of use within a cross border setting of comparable experiments was not realized. Within a setting of more or less independent local experiments on smart metering, for truly exploiting the opportunities of international collaboration this pilot needs an agreed project definition and project plan reflecting the different commitments and business cases involved.

3.3 eManufacturing pilot 3.3.1 Approach

Evaluation and impact assessment for this pilot is based on the WP4 pilot evaluation report (D4.4[10]) as well as on observations from the WP1-WP4 liaison persons.

The Evaluation and Impact Assessment template, offered by WP1 (see D1.3[1]) has been applied. The evaluation and impact assessment is summarized in a “pilot evaluation profile” at the end of this chapter.

3.3.2 Pilot objective

This pilot of cross border networking of Living Labs is situated in the manufacturing domain. It assessed the use of a unified platform between the collaborating Living Labs: two SAP Research platforms came into play, the middleware device integration platform and the service trading platform. Thus, this experiment strived for: •

•

•

Advancement of the state of the art by adapting the MDI (Middleware Device Integration) platform to support cross-border development of services communicating with machines and devices;

Improve collaboration amongst local SMEs (those that are connected to an individual Living Lab) and help them build services that are useful to relevant manufacturing use cases of interest to all Living Labs;

Initiate and improve collaboration amongst selected trans-national SMEs such that they can produce as well as consume each other’s' services using a platform as a basis. Moreover, potential inter-SME activities will be stimulated that are of benefit to all stakeholders (SMEs, Living Labs, and the local economy).

For this it was regarded as crucial to characterize, enhance and adapt the researchlevel integration and MDI service platform to one which is capable of real-world device integration and service provisioning. Within this task the existing MDI integration and a service platform of SAP Research were adjusted for the eManufacturing pilot in order to facilitate integration, interoperability and innovation between the participating Living Labs. Together with SMEs, the eManufacturing pilot also focused on the development of additional, relevant services in the domain. This was intended to enable assessment of the methods on involving SMEs as well as the inter-Living Lab collaboration.

ICT PSP Project Reporting Template

27

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

3.3.3 Methodology evaluation and impact Methods and tools The main methodological approach in this pilot has been of organisational nature, to implement a project management approach based on developing, implementing and testing specific use cases on top of the MDI platform (asset viewing, energy monitoring). Additionally, most attention in this pilot has been on traditional communication and collaboration tools such as web conferencing and application sharing.

In a second phase of the pilot a service trading platform was tested to find out how much of the activities of collaboration that were required to set-up the use cases could be done better by using this service trading platform and thus scale business beyond borders.

Impact of methods and tools

The eManufacturing pilot has used several collaboration and communication tools having a direct impact. In order to ensure a systematic and regular collaboration between the Living Labs and the connected SMEs the methods and tools used included: regular phone conferences on a weekly basis, technical support via email and phone, documentation exchange on the prototype including information on installation, usage and configuration, the use of the APOLLON-internal portal (MyBBT[19]) to share documents and to upload minutes, and SAPconnect (including web-based application sharing and conferencing), LinkedIn closed group, and SAPmats (download container tool for "large file" sharing).

The Remote Living Lab Tour (using a webcam and the SAPconnect session) firstly provided the necessary understanding of each other’s environments and second the potential for technology integration and demonstration to a future customer base in the remote Living Lab. Additional methods and tools included the set-up of service level agreements to ensure reliable technical support, software development and licensing agreement as a base for a trusted environment while no consortium agreement was in place and for the supporting partners who were outside of the consortium agreement; as it would be the case for future partners and customers.

The scenario method offered by WP1 was used to visualize and to better understand the usage of the platforms and the activities that had to be performed in setting up the business and the new services and acquire a first idea of the potential of joining such a cross-border network or project.

3.3.4 Value added of cross border living labs networking

In the eManufacturing pilot, the value added by the cross border activities and the APOLLON pilots have been manifold. APOLLON has made it more efficient to set up cross border activities. For example, it became relatively early clear in WP4 that it would be very helpful to put a collaboration environment in place which facilitates the execution of the cross border activities. Many experiences and insights how ICT PSP Project Reporting Template

28

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

business relationships and finally partnerships between SMEs coming from different countries are actually built and how they can be improved were gained.

As a consequence, for example, with the Energy Monitoring use case as well as with the Asset Viewing use case there has already been a business partnership created between a Living Lab (Future Factory) in Germany and an SME (Ydreams) in Portugal where the SME took advantage from the provisioning of a software technology - the SAP Research MDI platform prototype. The Portuguese SME combined that with their knowledge and expertise to solutions that helped other SMEs such as Imeguisa with their demand for energy monitoring and CENI with their demand for an asset viewing solution to get the right and best product and created two new business partnerships.

Through the establishment of cross-border business partnerships (B2B), using the APOLLON eManufacturing Living Lab approach, the collaboration also accelerated the process of innovation and dissemination of new products / services and the exchange of knowledge. That is specifically proven by the fact that the second use case went live faster due to experiences already built up at YDreams (learning curve).

The applicability of the service platform to better support the B2B business initiation and service negotiation was successfully tested by translating the business processes, required in the first phase of the pilot to set-up the energy monitoring service from partner, via customer search and finding until agreement, and thus proved to be a powerful means for cross-border business collaboration.

This collaborative approach accelerated the establishment of B2B partnerships, thus contributing to the dissemination of new SME solutions and within a broader market. Also, the co-creation of solutions and the collaboration allowed the widespread adoption of best practices.

“The Living Lab methodologies are based on an intensive collaboration between partners to enable the co-creation of services and products. This was already proven during our work on the WP4 use cases by showing a number of advantages such as an effective way to establish trust between SMEs which is essential in B2B relationships and usually not easy to attain: e.g. you could rely on the local Living Lab having the knowledge of and contacts to reliable local partners, having checked them before inviting them into a new business relation. It also supports continuous innovation driven by users which is in the best interest of the collaborative SMEs”.

Three new ideas, three new implementations and three redesigns of products, services, and processes have emerged from the cross-border collaboration with users.

3.3.5 Sustainability

The current pilot network is project-based. For the purpose of the eManufacturing pilot project, agreements have been made between the partners.

ICT PSP Project Reporting Template

29

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

As part of the WP6 activity, it was also initiated a thematic domain network for the area of eManufacturing and Logistics. Potentially this network could act as breeding ground for projects and business in the domain. A set of domain network services is proposed that can be offered to Living Labs and SMEs. These include sales and marketing support, maintenance support, field service management and other. It is too early to report about the viability of this network.

3.3.6 Harmonization

Within the pilot, requirements for a collaboration platform that specifically supports cross border businesses for SMEs in the area of the manufacturing industries were identified. The cross-border collaboration lead to enhanced access to user communities in other countries, to improved value proposition to the stakeholder community, to enhanced learning of Living Lab collaboration in networks, and to improved maturity of Living Lab management.

The interoperability between the Living Labs was realized through organisational and technical methods. Project coordination took place on a regular basis using the project’s MyBBT portal, weekly telephone and Skype conference calls and email communication. The pilot explored, tested and developed tools and methodologies that are expected to drive the next level of cross-border Living Lab collaboration for the manufacturing industries.

Within the SAP Research Future Factory Living Lab, the collaboration is enforced by many projects, use cases, demonstrators and prototypes that were built or are in progress. The various projects are running independently but nevertheless using one and the same infrastructure which is under the responsibility of an SAP Research Future Factory core team. This team assures a consistent management either being project specific or in the overall Future Factory context.

As a result of the agreed use cases of this pilot, SAP Research provided the Middleware for Device Integration prototype (MDI) as the technical basis. The platform was introduced to all the stakeholders of the pilot and shipped to the other Living Labs and respectively to the connected SME partners. This shipment included the executable software and a development kit as well as documentation to help the end users and developers. SAP Research Dresden took care of supporting partners in Portugal setting up and maintaining the MDI platform prototype. The same approach was followed when introducing the service trading platform to serve as the business/interaction basis.

This collaboration generated the exchange of experiences between the partners and lead to a systematic deployment and integration of the MDI platform prototype into the existing systems at Imeguisa and CENI according to the planned expectations.

Due to time restrictions the testing of the service trading platform was only limited but showed enough value to Fiapal, the Living Lab in Portugal, so they will offer it as a regional platform to the local (manufacturing) eco-system for local and later crossborder business and to get ENoLL[26] interested who until the end of the APOLLON

ICT PSP Project Reporting Template

30

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

project will verify its potential for broader use in their eco-system to support business initiation and collaboration.

3.3.7 Collaboration

From the start, the eManufacturing pilot was strictly defined and managed as a project, which highly contributed to the successful collaboration. One observation is that transparency, trust and easy access to information regardless if the SME is a service provider or service consumer is crucial to start business relationships. Here a web cam tour could bridge the gap until a face-to-face meeting could take place.

Most important was, that the Living Labs were able to act as a “trusted hub” through which SMEs became involved in cross-border service innovation. This pilot also demonstrates how a clear and agreed problem definition, timeframes, objectives and expectations regarding results of the experiments combined with direct and transparent communication across the cross-border project teams contributes to success. The scenario method offered a better understanding of the usage of the platforms and the things that have to be done to set-up a cross-border network project.

3.4 eParticipation pilot 3.4.1 Approach

Evaluation and impact assessment for this pilot is based on the WP5 pilot evaluation report as well as on observations from the WP1-WP3 liaison persons. The Evaluation and Impact Assessment template, offered by WP1 (see D1.3[1]) has been applied. The evaluation and impact assessment is summarized in a “pilot evaluation profile” at the end of this chapter.

3.4.2 Pilot objective

Focus for this pilot was twofold: to understand the opportunities and challenges that arise when integrating complementary technologies; and to test if eMedia technologies are an effective way for engaging users to participate in eParticipation and cultural activities. eParticipation was addressed in a wide sense of involving citizens in the city life by providing them innovative digital tools that would facilitate their participation to the city cultural life and history. As far as Living Lab role and missions are concerned, the pilot focused on understanding how Living Labs can: •

• • •

Enable local SMEs and citizens with new ideas and products for using eParticipation for citizen engagement to gain access to local resources which can be used to test and evaluate those new ideas and products;

Take clients through a structured process for evaluating new ideas and products; Facilitate introductions to useful business contacts locally and across Europe; Showcase new ideas and products at a local and European level;

ICT PSP Project Reporting Template

31

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

•

Provide a Single Point of Contact for local stakeholders wishing to use Living Labs methods to improve services.

3.4.3 Methodology evaluation and impact Methods and tools used

Methods and tools in us include Distributed Application development, Observation, Data Collection and Focus Group methods, Scenario Design, and Communication and Collaboration tools. These have been considered to be highly valuable.

Low cost and / or free web and Internet based collaboration tools have helped in the set up and running of the experiments, for example: • • • • •

Dropbox [20] for sharing files without having to email files back and forth.

Skype [21] for video and audio calls.

Google Docs [22] for collaborative document editing.

Vimeo [23] for video sharing.

Delicious [24] and Pinboard [25] for sharing bookmarks.

Unused methods or tools

Based on the experiences in the eParticipation pilot, in the future it would be useful to agree on a range of collaboration tools before the experiment begins so that all participants know what tools to use and what for (e.g. Dropbox to share files). No web-based available project management / 'to-do' list tool was set up for use by the experiment. Though actions were not missed, it usually meant relying on email and individuals project management tools to ensure work was done. In future, it would be useful for the Living Lab and SME to agree on a project management tool to use (such as Basecamp [27]) before the experiment starts. There has been some difficulty in using the above tools with some partners (e.g. Manchester Galleries staff,) as their corporate IT infrastructure was very locked down and is not flexible enough to allow staff to set up and use useful tools, such as Dropbox, without clearance by senior managers and IT policy officers. This has caused some slowdowns in working with those staff during the experiments, as they were not able to directly access resources that were being freely shared between the Living Lab (MDDA) and the SME (IBBT).

3.4.4 Value added of cross border living labs networking Test of technology in different cultural settings Almost all partners could participate to a cross-border experiment that took place outside their home country. AirGraffiti (MuseUs) could experiment its technology in Antwerp (home country), Manchester and Issy. On the other side, Navidis planned to participate to the Manchester City Galleries experiment but could only participate to the French experiment. This example shows that this kind of cross-border testing ICT PSP Project Reporting Template

32

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

is more profitable for “light” technologies using widespread technologies (iOS, Google Maps) with a startup approach that can quickly adapt to different geographical and cultural settings.

Increased understanding of different stakeholders perspectives

The pilot involved the participation of partners with different profiles: SMEs, Living Labs, local administrations, cultural institutions, Startups, NGOs. All partners had to accept some compromise in order to harmonize the expectations and goals of each other. This was useful from the perspective of experience for future projects involving a consortium of European partners.

Increasing income or status

Due to APOLLON every partner could benefit from a positive impact on their status as a European SME or Living Lab. For Living Labs, the main benefit was the opportunity to make their activity visible at both a local and European level. For SMEs, the main benefit was to be able to enter a wider market and promote their technologies as European validated services.

Learning and sharing good practices

APOLLON produced sharing good practices at different levels: Living Lab methodology, application development, data collection, users’ involvement and community building. For all partners this was the first time they could share their experience as a Living Lab and the first time they questioned their Living Lab role as part of a wider European ecosystem.

Attracting EU funding

All partners except Virdual are today engaged in European projects with partners they met directly or indirectly through APOLLON. Enhancing broader cooperation

WP5 partners see APOLLON as a ground for further collaborations in the Living Labs ecosystem. APOLLON succeeded in showing that inside the Living Lab network there are partners that are mature and skilled enough to develop international projects with SMEs. In this cross border pilot, specific benefits achieved through cross border networking of living lab activities are multiple: researchers have facilitated the implementation and user assessment of the game on site, two new services for the inhabitants of Issy (end-users of the Living Lab) have been created, links between the Living Lab and its stakeholders have been reinforced, dissemination activities have allowed the City and the Living Lab to gain national and international visibility through the participation and organization of workshops, international events and in several national competitions, and APOLLON has enabled Issy to increase its Living Lab activities.

ICT PSP Project Reporting Template

33

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

On another level, the visibility gained thanks to APOLLON has opened the door to new foreign partnerships and involvement in new EU funded projects in various domains, to improve existing city services and to create new ones benefiting the inhabitants of Issy. The experiments have given Manchester Galleries an opportunity to test new technologies and new forms of eParticipation. It has brought Galleries staff into contact with developers and given them insight into the development process and the issues developers face when seeking scenarios to test their products in a real world setting. It has also prompted discussion about the value of bespoke mobile content, the role of gaming in gallery spaces and how interpretive content could be used in ways not previously considered.

The two experiments have increased the visibility of the Museum on a local and regional level, offering in the same time the possibility to test beforehand new devices and technologies and experience new forms of citizen Participation. The Living Lab environment has made it possible and has eased the collaboration with a number of partners, local as well as foreign, which in normal conditions would have been difficult or long to achieve.

3.4.5 Sustainability

The current pilot network is project based. For the purpose of the eParticipation pilot project, agreements have been agreed between the partners.

As part of the WP6 activity, activities to develop a thematic domain network were initiated for the area of eParticipation and Media. Potentially this network could act as breeding ground for projects and business in the domain. A set of domain network services is proposed, offered to living labs and SMEs. These include creation of cross-border networks, methodology support, promotion and other. It is too early to report about the viability of this network.

3.4.6 Harmonization

During the pilot two kinds of interoperability challenges were faced: the technical interoperability of partners’ technologies and the harmonization of partners’ goals.

Technical interoperability of the technologies has been reached through the collaboration of SMEs along the definition of the pilot scenario. In all cases this involved some ad-hoc development to reach a level of integration that was sufficient for the pilot and useful to explore the interest of investing further effort to achieve a seamless integration. SMEs went through face-to-face meetings to introduce each other technologies and then worked remotely towards a common solution.

Given the different profiles of SMEs partners, the harmonization of partner’s objectives was a challenge WP5 needed to solve at the beginning of the project.

SMEs like Virdual and Navidis expected to find business opportunities through APOLLON; NGOs like People Voice Media expected an international validation of their activity; Startups like Air Graffiti and the supporting partner Mobexplore wanted to get user traction and visibility through APOLLON. The way in which WP5 managed to harmonize all partners’ objectives was through the process of codesigning the scenarios for each experiment in a way that the goals of each partner

ICT PSP Project Reporting Template

34

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

were addressed if not completely matched. The process of harmonizing these goals involved some level of compromise for each partner. Living Labs willing to realize cross-border activities involving multiple partners will need to take into account these different goals. One solution to reduce the complexity of managing this issue could be to start from one SME goals and try to find partners that have compatible expectations.

3.4.7 Collaboration

Pilot partners are positive about the collaboration experience. All partners could benefit from the cross-border collaboration to improve their technology and better define their solutions. Beyond the results of the pilot, the cross-border collaboration allowed Living Labs and SMEs to start building a European ecosystem that is a fertile ground for future projects. The majority of the partners are already working on new European projects with partners they met through APOLLON.

Pilot partners see APOLLON as a ground for further collaborations in the Living Labs ecosystem. APOLLON succeeded in showing that inside the Living Lab network there are partners that are mature and skilled enough to develop international projects with SMEs.

The cross-border experiment has yielded an interesting application as a consequence of the collaboration between the different parties. The pooling of the expertise of IBBT, Virdual and M HKA resulted in a product that would have been harder to achieve by any one of the involved partners. The cross-border experiment also produced interesting lead contacts in the different markets. For example, the cross-border pilots provided Air Graffiti with access to markets in Belgium, France and England, which we normally would never have achieved. This took place in the following way: •

• •

• •

Air Graffiti gained an insight in the perspectives from different stakeholders that they normally would never have gained, especially from working closely with M HKA. Working with Virdual gave Air Graffiti access to technology which they normally would not have considered nor developed.

The cross-border trials allowed Air Graffiti to test the application in different cultural settings, with different languages and with different museum types. This made the application more generic and therefore useful in different settings. It made Air Graffiti realize the added value of our simple application: the concept is simple, which increased its applicability to different domains.

The cross-border pilots led to a product that Air Graffiti is now developing into a new business.

Because of access though APOLLON to ENoLL, Air Graffiti is now discussing the possibility to test the application in other European countries, like Spain.

ICT PSP Project Reporting Template

35

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

•

•

The living lab perspective has allowed the continuous evolution of the software prototype, based on user feedback. In addition, it has allowed for the formulation of insight concerning the way in which the app should be brought to the market, like for example what business model to use.

In the future, the experiments that are still to be conducted, but that Air Graffiti is currently setting up, will allow them to test the application’s concept in different types of museums. This will make the application more generic and therefore widely available. It will also increase the track record of the application and allow Air Graffiti to present results and testimonials in various markets.

Overall, for this pilot the cross-border collaboration lead to increased access to user communities in other countries, it increased the value proposition to the stakeholder community as well as the learning of living lab collaboration in networks, and finally, it lead to increased maturity of living lab management.

One of the successes of the pilot is that several of the SMEs involved managed to enhance their product and offering and to find new markets in Europe. The pilot demonstrates also several bottlenecks that hinder cross-border collaboration, such as differences in goals and expectations as well as differences in business cases.

SMEs expected to find business opportunities, NGOs expected an international validation of their activity and start ups wanted to get user traction and visibility.

Different entrepreneurial cultures also contributed to creating a bottleneck for collaboration. Harmonization of objectives was a challenge that needed to be resolved at the beginning of the project. The process of harmonizing goal differences involved some level of compromise for each partner. A process of scenario co-design helped to harmonize partners’ objectives and to facilitate the integration of technologies. The co-design phase took the most of effort in all the experiments, namely from the definition of the scenario to identification and involvement of the relevant stakeholders. The early involvement of key stakeholders into the experiment planning is very important.

Also the use of a structured application development framework and project management methods and tools facilitated collaboration. The role of living labs in the collaboration setting was found to be of considerable importance in order to prepare and carry out the experiment successfully. This also requires adequate project management skills and competencies.

Match-making between SMEs and Living Labs was a fundamental step in determining the success of collaboration between Living Lab and SMEs. When the Living Lab ecosystem is not adapted to the SMEs demands, opportunities are easily lost. In the same way, when SMEs solutions are not sufficiently flexible it can be difficult to integrate them. For this reason, agile approaches for prototyping and testing should be encouraged when working with Living Labs. Start ups that can easily pivot if they find unforeseen opportunities benefit more from cross-border pilots than SMEs with a closed solution. On the other hand, the risk of working with ICT PSP Project Reporting Template

36

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

a startup which product is under development is to create delays on the initial schedule.

One of the lessons learned is that the Living Lab should not step back after the SME and local partners and participants have been introduced and the scenario identified, but work with all the participants to ensure that the experiment is carried out successfully, also supporting all the communication between the SME and other participants, to make sure that all involved are aware of their roles and responsibilities during the experiment. From the perspective of overall coherence of the pilot and each experiment there is a need for the supervising role of one partner, having the task of precise project management of all partners and activities with appropriate (possibly web-based) tools.

From a user experience point of view, using new technologies and media attracted especially younger population and resulted in overall positive feedback. However this might be an obstacle for older population and for those who do not have available appropriate equipment for application and service usage. Living Lab should consider these possible obstacles and ensure the needed equipment for the all users, participating in the experiment and pilot. Also the need for training should be considered.

3.5 Evaluation and impact profiles of the pilots

The next tables provide summary “evaluation and impact profiles� of the four pilots.

ICT PSP Project Reporting Template

37

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report Table 3-3a Evaluation and impact profile for the Homecare and Independent Living Pilot Central Objectives

Targeted Achievements and Outcomes

Indicators and information used in evaluation and impact assessment

1. Demonstrate the value added of crossborder living lab networking

1.1 Create specific added value in terms of results / benefits as well as operational efficiencies of the cross-border approach (for all partners in total, and / or from perspective of specific targets)

•

•

• •

1.2 Create value added for specific stakeholders (for specific partners: SMEs, Living Labs)

•

• • •

•

Develop a methodology for

1.3 Realize the harmonization and interoperability of Living Lab approaches and platforms between networks of exemplary European Living Labs

2.1 Develop a set of validated methodologies to

•

•

The healthcare and wellbeing domain has specific dimensions and particularities, such as a high importance of technical reliability, trust, privacy that are demanding on the products tested. By doing cross border collaboration the understanding of the products was greatly increased and gave crucial knowledge to the partners, specifically the companies in the network.

International exposure and collaboration has driven the innovative capacity of the network forward by providing perspective and detailed context sensitive information to the SMEs in the network. For example the new version of the product of Innoviting was tested in the Spanish Pilot rather than the intended original version.

Pilot has demonstrated the underlying reasons for the lack of efficiency in cross-border transfer of technology and market development in the healthcare domain. However the pilot has not yet achieved a sustainable and scalable business solution yet.

The research done in Belgium has formed the basis for the research in Finland. The results of the “telesenior” project have been valuable for the pilot in Finland.

SMEs participation in Living Lab networking has provided the SMEs with experience in internationalization of products and services and a better understanding of the capabilities of the product.

The new version of the product of Innoviting was tested successfully in the Spanish Pilot (product worked).

The real value of using cross-border living labs was seen as getting access to foreign ecosystems, using the contacts and reputation provided by the local living lab.

The Finnish Living Lab Forum Virium is running the extended trial of the Tunstall product pilot until the end of 2012. The Belgium Living Lab IBBT will run the Guardian Angle project in Turnhout for Logical.

Both these Pilots are the result of the collaborations within the APOLLON Project The various partners all have learned extensively how to internationalize and localize for the demanding Healthcare market.

The products have been localized, including interface translation and the formation of the ecosystem needed to run the system in the receiving living labs environment. In the case of the Xtramira the technical hurdle prevented a roll out, in the other three cases the deployment in the receiving environment shows the viability and success of the harmonization methods utilized.

Eco-system Analysis was developed, adopted and utilized in the beginning of several pilots. The results of the eco-system analysis was used as input for the contacting local partners procedure

APOLLON – D1.6 Evaluation and Impact Assessment Report cross border living labs networks

•

set up, plan and conduct cross-border Living Lab networks

•

• • • •

Ensure sustainability of cross-border living labs networking

2.2 Develop and share a recommended toolset and collaboration guidelines for facilitating crossborder research and innovation activities

•

3.1 Inform and network stakeholders in Europe for specific thematic domains

•

3.2 Develop governance and business models to support sustainable networks of living labs

•

2.3 Implement successfully the different harmonization and interoperability approaches needed for cross-border living labs networking

ICT PSP Project Reporting Template

39

•

•

A method for translation of the user interface was developed, tested and adopted by the partners in the pilots.

The requirements analysis procedure was used in the beginning of the pilots; on the basis of this a procedure for pre-deployment testing of the service in a real life environment was developed. The procedure is applied in the Spanish pilot.

The research framework was adopted for developing the user driven research component of the trial.

Business model design methods were used in Helsinki in finding the most logical implementation of the product in the eco-system The Knowledge Center on the Enoll website has been used by an interested third party with ideas on establishing a local pilot in Belgium.

The Use of internet based communication tools and document sharing have greatly increased the ability to collaborate.

A number of tools and methods used and evaluated in the Homecare trials are now available in the knowledge Center: SME partner search and selection; Business model design for cross border living labs network; Profiling Living Labs and SMEs for partner finding; Translating and localization procedure /User interface translation; Eco system analysis for Living Labs; Research framework for designing and evaluating living lab networks; Basic contract between test users and Living Lab

The experiments in the Homecare Pilot have been developing and testing a method that allows for the description of the relevant actors, roles and potential partners in a health care related technology eco system. This method is deployed in the pilots to select partners and also functions as an input for the value proposition analysis of potential services that can be resulting from the pilot. With the results from the pilots the SMEs and the Helsinki based Palmia, (a Large enterprise) and Terke (the homecare provider) have changed their development plans and taken action to further develop their service offering.

The Living Labs in the pilot have agreed to form the nucleus of the new Domain specific network of Living Labs in Healthcare and wellbeing. This network is a long term endeavor on the path to form an EU wide constellation of partner Living Labs all collaborating in various constellations on projects related to the interest area.

The questionnaire that was aimed at collecting information on the possible interest on the Domain Networks has provided the data of a further 8 Living labs that have demonstrated interest in the Domain specific network.

Initial concepts have been developed which may form the basis for future governance and business models.

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report Table 3-4a Evaluation and Impact Assessment profile for Energy Efficiency pilot Central Objectives

Targeted Achievements and Outcomes

Indicators and information used in evaluation and impact assessment

1.

1.1 Create specific added value in terms of results / benefits as well as operational efficiencies of the cross-border approach (for all partners in total, and / or from perspective of specific targets)

•

The added value of the Living Lab has been the increased knowledge base in the area of energy saving.

This has lead to new project initiatives that in turn has increased collaboration with partners around Europe. Another aspect of added value has been a strengthened collaboration with a local SME who had not been involved in similar project before. This has also lead to a better collaboration with them as well as with the supporting partner or partner SMEs in each country.

•

The value added achieved through cross border networking of living labs has several aspects. Firstly, for example, for authorities it has increased understanding of other contexts and environments in Europe.

Secondly, for living labs, the pilot has led to increased cross-border knowledge exchange between the four living labs. Thirdly, for SMEs this has provided increased understanding of the complexities of creating new business in new markets between SMEs and larger pool of companies. And fourthly, for large enterprises this has increased understanding of diverse good practice models and understanding of how difficult it is to scale in four different legacy markets (no single European market).

To be more specific, participation to APOLLON has enabled the signing of a number of letters of intent between partners and customers, as well, as generated new businesses in other countries in terms of business proposals and proposal preparations. For SMEs and Large Enterprises this has enabled to evaluate their maturity to enter new markets. It has increased knowledge on working in a big international project as well as the understanding of the European energy market.

Demonstrate the value added of cross-border living lab networking

1.2 Create value added for specific stakeholders (for specific partners: SMEs, Living Labs)

•

2.

Develop a methodology for cross border living labs networks

1.3 Realize the harmonization and interoperability of Living Lab approaches and platforms between networks of exemplary European Living Labs

2.1 Develop a set of validated methodologies to set up, plan and conduct cross-border Living Lab networks

ICT PSP Project Reporting Template

40

•

•

•

Collaborative network has been established at each experiment level as well as across all experiments.

Sharing of best practices and, for example, case demonstrations succeeded, and consequently, other cases have followed these instructions. Collaboration has increased between SMEs and universities. Best practices and recommendations from Luleå were adopted within the pilot network. The road show enhanced the understanding of business and markets.

In WP3, the participation to APOLLON has lead to increased comparability of Living Lab research among LTU, Aalto University, Alfamicro and Liander. The experiment has led to increased cross-border knowledge exchange between the four living labs.

Several methods and tools with direct impact have been used. These have been: 1) user transformation methodology, a process supporting the test of energy monitoring and enabling tracking users’ energy consumption behaviour; 2) test storyline, a method giving structure for the test where tasks are given and questions related to that task are described in detail; 3) workshops enabling knowledge sharing activities related to methodology for use transformation; 4) knowledge transfer across pilots, enabling Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report

3.

Ensure sustainability of cross-border living labs networking

2.2 Develop and share a recommended toolset and collaboration guidelines for facilitating crossborder research and innovation activities

•

2.3 Support the successful Implementation of the different harmonization and interoperability approaches needed for cross-border living labs networking

•

Inform and network stakeholders in Europe for specific thematic domains

•

Develop governance and business models to support sustainable networks of living labs

•

ICT PSP Project Reporting Template

41

sharing experiences from processes among the partners in the pilot; and 5) sharing best/good/worst practices between Partners (mainly companies).

In WP3, several methods have been used that were in enabling role for the cross border living lab pilot.

These were, for example, mail, IBBT website, telemeetings, and participating in APOLLON events, fairs, and seminars. Overall the added value of the methods and tools were in increasing efficiency and savings in the pilots as well as for enabling follow-up. The added value of transferring technology in the crossborder experiment has been in enabling testing foreign technology and understanding the need for local service/maintenance.

Collaborative network has been established at each experiment level as well as across all experiments, which facilitated the sharing of best practices and case demonstrations. Consequently, other cases have followed these instructions. As a result, collaboration has increased between SMEs and universities. Best practices and recommendations from Luleå were adopted within the pilot network. The road show enhanced the understanding of business and markets.

New RDI proposals between Living Labs have been made nationally and on the European level.

Knowledge of working in a large international project has increased as well as the understanding of the European energy market. Understanding on customers’ needs and requirements of functions of products visualizing the energy consumption has increased. Knowledge on cross border collaboration among Living Labs has increased.

New proposals between Living Labs and the desire to continue APOLLON after 2012 have been expressed. Most SMEs are willing to continue with APOLLON type pilots.

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report Table 3-5a Evaluation and impact assessment profile for eManufacturing pilot APOLLON Central Objectives (DoW)

APOLLON Targeted Achievements and Outcomes (DoW)

Indicators and information used in evaluation and impact assessment

1.

1.1 Create specific added value in terms of results / benefits as well as operational efficiencies of the cross-border approach (for all partners in total, and / or from perspective of specific targets)

•

Demonstrate the value added of cross-border living lab networking

•

•

• 1.2 Create value added for specific stakeholders (for specific partners: SMEs, Living Labs) 1.3 Realize the harmonization and interoperability of Living Lab approaches and platforms between networks of exemplary European Living Labs

•

• • • •

•

ICT PSP Project Reporting Template

42

The pilot evaluation demonstrates the benefits of cross-border collaboration in terms of operational efficiency of the collaboration. Partners evaluated the efficiency as highly satisfactorily.

Business partnerships already existed at the beginning of the pilot. However it was gained many experiences and insights how business relationships and finally partnerships between SMEs coming from different countries are actually built and how they can be improved. Also, the pilot demonstrated how the involved SME took advantage from the provisioning of the MDI and the service trading platform.

In terms of actual added value of the cross border approach in terms of improvements in product or service innovation, and benefits for SMEs, the available information demonstrates that implementation of use cases is possible. However no concrete info on actual SME business benefits is provided.

One benefit was getting access to technology of an LE, while offering testing and application on one side and getting knowledge of future technology on the other According to SAP, over the time, the established network of the Living Labs from Portugal and Germany became very strong and mature. The involved stakeholders setup clear processes and guidelines to execute the identified use cases.

Multiple SMEs were able to work simultaneously through the Living Lab network regardless of the location where they are based.

This pilot addresses the challenge of using a common platform among the participating SAP and FIAPAL Living Labs.

Organisational aspects were covered by using the collaboration and communication tools mentioned under 2.1.

The MDI platform is the basis for technical interoperability. It was agreed on service level agreements and KPI’s in order to understand this technical interoperability and resolve potential problems. The platform was used successfully to implement the use cases. The collaboration generated the exchange of experiences between the partners and lead to a systematic deployment and integration of the MDI platform prototype into the existing systems at Imeguisa and CENI according to the planned expectations.

The service trading platform in the end became the base for future business collaboration, having been tested by “translating” the energy monitoring use case set-up onto this platform and “replay” the collaboration.

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report 2.

Develop a methodology for cross border living labs networks

2.1 Develop a set of validated methodologies to set up, plan and conduct cross-border Living Lab networks

•

•

• •

3.

Ensure sustainability of cross-border living labs networking

2.2 Develop and share a recommended toolset and collaboration guidelines for facilitating crossborder research and innovation activities 2.3 Support the successful implementation of the different harmonization and interoperability approaches needed for cross-border living labs networking

3.1 Inform and network stakeholders in Europe for specific thematic domains 3.2 Develop governance and business models to support sustainable networks of living labs

ICT PSP Project Reporting Template

43

• •

• • •

•

•

The approach to set up, plan and conduct the cross-border living lab network has followed a project management approach based on a project plan and specific use cases, “energy monitoring” and “asset viewing” (scenarios). The research framework and the scenarios (use cases) contributed to giving direction and to implementing the plan.

Validated methods and tools i.e. successfully adopted and used in this pilot, include the set of collaboration and communication tools (e.g. web conferencing, application sharing, shared workspace), the remote living lab tour (dissemination), and software development & licensing agreement.

For larger and more complex projects it is recommended more sophisticated tools facilitating business identification, business generation and business execution and maintenance of solutions.

Such collaboration platform could act as marketplace and also allows exchange of any kind of information helping to understand business partners and acting as service trading mechanism, as the service trading platform, which was introduced towards the end of the project when the need became obvious and other tools proved to be not helpful.

A set of methods and tools have been identified that can be recommended for comparable situations. Apart from the explicitly validated methods and tools, these include others.

The pilot results into a set of lessons learned which is the basis for proposing collaboration guidelines.

The pilot-internal methodology developed within WP4 is based on the use of a common technology platform (MDI) and a service provisioning platform.

At one level, functional evaluation and human factors evaluation were applied to the technical platform.

At a second level, collaboration capabilities have been explored through human-centric evaluation where Living Labs and SMEs provided feedback about the collaboration environment and resulted in the testing of the service trading platform.

The domain network was established during the APOLLON project life time; it is open if and who will take it over to drive it further after the end of the project; will be presented at the final conference, so others can pick it up;

The APOLLON eManufacturing platform will be further applied in the local & regional (manufacturing) eco-system in Portugal, driven by Fiapal and has been prepared for handover in a second – more content-extended version – for ENoLL broader use in their global Living Lab ecosystem.

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report Table 3-6a Evaluation and impact assessment profile for eParticipation pilot Central Objectives

Targeted Achievements and Outcomes

Indicators and information used in evaluation and impact assessment

1.

1.1 Create specific added value in terms of results / benefits as well as operational efficiencies of the cross-border approach (for all partners in total, and / or from perspective of specific targets)

•

1.2 Create value added for specific stakeholders (for specific partners: SMEs, Living Labs)

•

1.3 Realize the harmonization and interoperability of Living Lab approaches and platforms between networks of exemplary European Living Labs

•

2.

Demonstrate the value added of cross-border living lab networking

Develop a methodology for cross border living labs networks

2.1 Develop a set of validated methodologies to set up, plan and conduct cross-border Living Lab networks

• • •

•

• • •

• 2.2 Develop and share a recommended toolset and

ICT PSP Project Reporting Template

44

Mixed results of the pilot in terms of integrating different SMEs, due to divergence of goals of partners (academic vs. business view). T was easier for startups and organizations based on knowledge transfer.

Over-all, participating living labs and SMEs had clear benefits from participating to the pilots e.g. involved in new products, new solutions, access to testing results, sharing of practices.

Experiments reached 120 users, which has been achieved also because of the cross-border networking within the pilot.

Partners felt that they had benefited from the cross-border collaboration in improving their technology and better defining solutions. Cross border networking also allowed Living Labs and SMEs to start building a European network as breeding ground for future projects.

Potential partners were met through APOLLON activities.

Cross border experiments yielded an interesting application as a consequence of the collaboration: pooling of expertise resulted into a product that would have been much more difficult o achieve by individual partner only.

Also promising lead contacts in the different markets were generated. Air Graffiti got access to markets in Belgium, France and the UK. The pilot has successfully realized the integration of eMedia technologies

Favorable experiences in using Distributed Application development methods Method of Living Lab Observation, Data Collection and Focus Groups was successfully used in Issy, and reused after adaptation in other experiment settings.

Scenario design was used successfully by SMEs and Living Labs to elaborate the technology and solution and also as a vehicle to strengthen the collaboration among partners and enhance mutual understanding.

Communication and collaboration tools have been successfully applied; some difficulties were encountered in terms of access to such tools hindered by corporate IT policies.

Final Version, 08/05/2012

APOLLON – D1.6 Evaluation and Impact Assessment Report collaboration guidelines for facilitating cross-border research and innovation activities

3.

Ensure sustainability of cross-border living labs networking

2.3 Implement successfully the different harmonization and interoperability approaches needed for cross-border living labs networking 3.1 Inform and network stakeholders in Europe for specific thematic domains

• •

Challenge in this pilot was to implement a common integration framework as a basis for Living Labs collaboration.

Also other harmonization challenges were addressed, e.g. harmonization of partners’ objectives. Practical experience is that compromise is necessary in many cases.

3.2 Develop governance and business models to support sustainable networks of living labs

ICT PSP Project Reporting Template

45

Final Version, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

4. Business Impact Assessment This chapter provides an over-all overview of business impacts of the APOLLON project based on data and experiences from business partners of APOLLON. The first part of this chapter is devoted to SMEs business impact; the second part of this chapter covers larger companies.

4.1 Business impact assessment of SMEs 4.1.1 Business Profile of APOLLON SMEs

Based on the inputs provided by the respondents, we developed the business profile of an average SME in Table 4-1. Inputs received by various respondents are recorded in Table 4-2 and provides a general overview of their key profile attributes of an SME stakeholders active in APOLLON. It is to be noted that all the data and revenues in our assessment are anonymized for the sake of confidentiality of the partners.

Table 4-1 Business Profile of Average SME respondent Categories

Average

Number of SMEs

11

No. of Employees

90

Years in operation Autonomicity

18 All

Turnover (2009)

6.055.382 €

Turnover (2011)

7.783.660 €

Turnover (2010) YoY increase (2010)

YoY increase (2011)

Turnover/Emp (2010) Turnover/Emp (2011)

6.492.368 € 7%

20%

49.435 € 52.353 €

Apollon – D1.6 Evaluation and Impact Assessment Report Table 4-2 Overview of SME inputs Indicators

1

2

3

4

5

6

7

8

9

10

11

Year of Inception

1983

2003

2009

2005

1990

2003

1993

1995

1946

2007

2001

No. of Employees

25

26

9

9

200

50

250

9

200

9

200

Turnover (2009)

1.200.000 €

1.000.000 €

Turnover (2011)

1.200.000 €

1.800.000 €

Years in operation Autonomicity

Turnover (2010) YoY (2010) YoY (2011)

Turnover/Empl (2010) Turnover/Emp (2011)

29 0

1.100.000 € -8% 9%

44.000 € 48.000 €

9 0

3 0

0

17 0

66 0

5 0

11 0

79.000 €

4.703.587 €

17.641 €

100.000 € 5.450.146 € 470.000 € 25.000.000 € 401.781 €

47.728.000 €

79.000 €

3.373.687 €

638%

Figure 4-1: SME partners: Years of Operations

0

19

42.611.000 €

80%

69.231 €

0

9

120.000 € 4.081.717 € 230.000 € 12.000.000 € 581.198 €

19.882 €

69.231 €

0

22

2.695 €

1.800.000 € 0%

7

-11%

2.209 € 1.960 €

100.000 € 3.876.076 € 430.000 € 13.000.000 € 691.893 € -17% 0%

11.111 € 11.111 €

-5%

87%

19.380 €

8.600 €

41%

27.251 €

9%

9.400 €

8%

92%

52.000 €

100.000 €

19%

-42%

76.877 € 44.642 €

47.728.000 € 12% 0%

238.640 € 238.640 €

79.000 € 0% 0%

8.778 € 8.778 €

2.591.192 € -45% 30%

12.956 € 16.868 €

Apollon – D1.6 Evaluation and Impact Assessment Report

Years in Operation Average time spent in active operation for each SME is around 18 years (12th Row), as shown in the Figure 4-1, some SMEs are only 3 to 5 years old in the market whereas others are older than 15 years in terms of operations.

Annual Revenues (Y2009-Y2011)

Deriving inputs from Table 4-2, annual revenues per SME partner is calculated in the Figure 4-2. Column 12 indicates the harmonized average revenues for all the SMEs active in APOLLON.

Figure 4-2: Annual Revenues per partner [Y2009, Y2010, Y2011]

Year on Year Revenue Increase during APOLLON [Y2009 - Y2011] In order to assess the year on year change in SME revenues during the lifetime of APOLLON, we used inputs from respondents in the Table 4-2 to generate the growth profile during the year 2010 and 2011. The average growth was recorded around 7% and 20% for year 2010 and 2011 consecutively (see column 12).

Figure 4-3: Year on Year Revenue Increase

Apollon – D1.6 Evaluation and Impact Assessment Report

Revenue per Employee during APOLLON [Y2009 -Y2011] Another key indicator to assess the growth is estimating growth in revenues per employee. However this measure holds a caveat that the number of employees during Y2010 and Y2011 remain the same.

Figure 4-4: Revenue per Employee Increase

4.1.2 SMEs Internationalization In order to gauge the cross-border presence and its importance to each SME partner in APOLLON, efforts were made to understand their existing business model and dependence on international businesses and trade.

Among the respondents 64% have presence in international markets and among them 63% of SMEs have 1 or more years of experience operating in International Markets. Upon further research it was revealed that as of 2011 nearly 55% of SMEs are deriving only 0-5% of their annual revenues from international trade and businesses.

Figure 4-5: SME internationalization and revenues from internationalization activities [Y2011]

ICT PSP Project Reporting Template

49

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

SME Revenues due to Internationalization In order to investigate the impacts of APOLLON on international revenues generated by the SMEs, data from years 2008, 2011 and 2012 were extracted from Table 4-2 and analyzed for variations.

Before APOLLON was initiated in Y2008, 64% of SMEs indicated a meager 0-5% dependence on international business and trade as a source of revenue. A small number of SMEs (9%) indicated 56-60% reliance on international businesses and trade for annual income (Figure 4-6).

Thanks to APOLLON, by Y2011, the original 64% of SMEs that indicated only 0-5% dependence on internationalization shrank to 46% Figure 4-2. Also the number of SMEs heavily exposed to international markets doubled from 9% in Y2008 to 18% in Y2011. This gain can be clearly attributed to the cross-border activities carried out the SME partners in APOLLON project.

Figure 4-1: International Revenue Distribution [Y2008]

ICT PSP Project Reporting Template

Figure 4-2: International Revenue Distribution [Y2011]

50

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

Towards the end of APOLLON, when asked almost 73% of SME partners indicated their intentions/reliance over the international markets. The percentage of SMEs still cautious regarding venturing into foreign markets shrank further to 27%, this is one-third of the number recorded in Y2008 (Figure 4-8).

Figure 4-3: International Revenue Distribution [Y2012]

Overall Impact on International Revenues In order to present an overview of impact on international revenues we cross compared all the individual responses for all the years (Y2008, Y2011, and Y2012). As indicated in Figure 4-4, the red quarter in the figure which represents 0-5% share of international revenues decreases as we move from Y2008 to Y2012. In the same time the green quarter of the figure categorically expands by nearly doubling its original share.

Figure 4-4: Overall Impact on International Revenues

Modes of Internationalization Every firm adopts one or the other modes to venture into new markets, establish new business contacts etc. APOLLON as a platform provided a fresh start to these SME partners to re-invent their internationalization strategy and consolidate the existing international markets.

Apollon – D1.6 Evaluation and Impact Assessment Report

Figure 4-5: Modes of Internationalization

As shown in Figure 4-5 among several alternatives outlined by us, 45% of stakeholders preferred the APOLLON Network to Distributorship, Franchise and Export type models. APOLLON Network was considered complementary to the successful “business as usual” modes of internationalization like – Parent company presence, JV and International Offices.

Barriers to Internationalization

Among others, it was the recession and lack of human capital/funds that made the SME partners more cautious towards international expansion and operation (73%). While 45% of respondents agreed upon “lack of knowledge of foreign markets” as one of the barriers they also agreed upon their “lack of focus on internationalization” as a barrier. In order to address these barriers encountered

by SMEs, APOLLON focused on rendering support for the SMEs by providing insights and knowledge on international markets, regulations, customerrequirements etc. Key contributions of APOLLON towards overcoming the barriers addressed by SMEs are elaborated and rated in Figure 4-7.

Figure 4-6: Barriers to Internationalization

4.1.3 APOLLON Contributions and Impact on SMEs As a key metric for assessing impact of APOLLON on SME Partners, each of the partners were required to identify and evaluate the contributions made by APOLLON and its consortium. Among several key contributions made by APOLLON, nearly 70% of respondents valued the Testing and Piloting

ICT PSP Project Reporting Template

52

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

Experience as most crucial to them. Second most important contribution of APOLLON (59%) was equally attributed to the understanding strategy and improved vision of doing business abroad.

Figure 4-7: Contributions of APOLLON and Impact

4.1.4 End of APOLLON: Directions and Exploitation Strategy SMEs All the SMEs were required to provide a snapshot of what could be considered as “APOLLON Success Stories and Achievements”. Given below are excerpts (unedited and in their own wording) from each SMEs highlighting the key economic and strategic goals achieved during the lifetime of APOLLON. 1) VIR • • • •

Issy: the Issy pilot was a real success and received several prices. We have present this solution in many exhibition : Futur en Seine (June 2011), we were in the exhibition hall for 4 days of exhibition meeting professionals and public.

MIPCOM (October 2011), international TV program market, 4 days on European Commission Media Program Space

MIFA (June 2011), international Animation and Digital market, 3 days on Ile de France’s Space.

Anvers: The test result was: • • •

There no market directly by selling this to museum The product have to be only one app

There is maybe a business model by creating a web service in which museum can customized free an app. This app could be sold through an app store.

Revenues of this selling would be shared between the service provider and the museum.

We are currently using the pilots to try to sell programs to international broadcast.

2) ALF

The APOLLON cross-border activities in the domain of Energy Efficiency have been implemented in the context of the EU-Brazil cooperation for Living Labs by Alfamicro. This activity has been developed with a bottom-up approach ICT PSP Project Reporting Template

53

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

involving the main stakeholders of Living Labs – universities, research institutes, enterprises, funding organizations, public administration: • • • • •

Information dissemination and awareness. Fostering the interest to the ENoLL Community membership.

Dissemination Workshops in several Brazilian cities.

Workshops to support the definition of the Living Lab concept: partnership creation, strategic objectives, Federal and State policies opportunities, etc.

Mentoring aiming at the preparation of the Living Lab eco-systems.

Workshops, thematic and institutional meetings.

Meetings with the main stakeholders at academic, research, enterprise, federal and state governments, municipalities, associations and NGO’s, etc.

This environment allowed us to identify the Energy Efficiency (Smart Metering) and the more general Smart Grid as two strong areas of interest in Brazil: looking for the European scientific and technological partnerships in order to implement in Brazil pilot projects of different dimensions which can support policy recommendations adapted to the Brazilian environment and opening the opportunity for commercial agreements to implement large scale Energy Efficient applications.

The driver of this initiative was our understanding that the Living Lab environment can reinforce the level of trust between cross-border stakeholders, thus accelerating the knowledge and the technology transfer activities. The dynamics of the Living Lab creation was used to involve the APOLLON SMEs, either directly as it was the case of ISA (Portugal) and Process Vision (Finland) and indirectly through Alfamicro. This activity led to the following results: • • •

Market evaluation for the European SMEs.

Pilot projects with the involvement of European SMEs. ISA Sul America created by ISA, based in São Paulo.

The enthusiasm and the business potential of Brazil for the Energy Efficiency knowledge and technology led ISA in Portugal to invite other SMEs in order to create a Living Lab embracing the complete value chain of the Energy Efficiency domain. The partnerships in this domain led to an initial number of 30 partners that created ISaLL (Intelligent Sensing and Smart Services Living Lab).

However the main business driver of this collaboration has been the knowledge and experience brought by the Portuguese Living Labs in terms of using ICT to achieve user behaviour transformation. This knowledge has been acquired mainly in the context of SAVE ENERGY pilot projects (a CIP project) and APOLLON. 3) HAE

The course of the economy during the APOLLON project has forced us to focus on our local activities in the Netherlands. Whereas it has strengthened our relationship with the Amsterdam Living Lab, business reality has led us to ICT PSP Project Reporting Template

54

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

review our hardware platform multiple times during the past years, and first realize market penetration in the Netherlands with our products, before expanding beyond.

4) HVE

Our participation in the project shrink to a lower level as originally planned. Due to several reasons, which will be presented in an additional document, our organization couldn't move many efforts as expected. However the international co-operations and some local activities help our members/partners to participate in fruitful project in the field of urban electric vehicle development.

The vehicle has been exhibited proudly by the customer in Geneva Motor Show 2011.

The interesting fact was that during the co-operation about 30 SMEs succeeded us to create a functioning prototype from zero within 6 months. http://www.technologicvehicles.com/en/green-transportationnews/1389/video-micron-by-exid-the-proof-in-motion

5) INN

We had a limited and organic transfer. We learned a lot of practical things like translating issues or the maturity your application has to be in. How to make good documentation that saves time on the long run. We learned that the end users are not so different in the two test countries (NL and E). The real trouble to get a business up and running would be the rules and regulation. Finding a place in the existing ecosystem is the hardest part. We have not decided yet if there is a place for the ADL system in Spain.

6) ISA

We have engaged this project with very high expectation about the output and information we could gather from participating in APOLLON, and also very excited to share our know-how with other SMEs, Living Labs and Institutions from other countries.

We couldn’t ignore the magnitude and positive impact of this kind of project involving so many Institutions and more than 100,000 citizens around Europe.

The main output we expect was to better understand and know leveraging investments in foreign markets, high profile cross-border partnerships, projects and pilots in consistent Living Lab networks and ecosystems with the possibility of understand users, their trends and changing their habits to reach a more sustainable society.

APOLLON allowed us to work closely with markets, methodologies and networks beyond Portugal’s ecosystem. This interaction was a fundamental contribution to access and search for future applications, services and structures that could help in internationalization process as well as access new partnerships and pilot activities in foreign countries. It was very interesting because help us to understand how Living Labs and other Institutions around Europe work, their expertise and methodologies, enabling valuable ideas to implement in Portugal's ICT PSP Project Reporting Template

55

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

innovative ecosystem. Besides APOLLON was an important step towards the understanding international market trends. The participation in Lisbon Pilot, a partnership between several institutions using the best cross-border domain specific methodologies in order to monitor and change users’ behaviour, was a very rewarding experience.

APOLLON helped us to understand international market needs in order to create new, better and more suitable solutions with user’s cooperation and guidance.

This prior know-how about our future clients will largely facilitate, and could be a critical factor in future internationalization processes and market approach.

7) NAV

Our goal was to provide our 3D technology developed mainly for cities and to share our experience in the service of e-participation for communities. We have carried out experiments, well undertaken, mainly by improving Urbadeus and our 3DCity application, our technical solution. Both they allow citizens and endusers to post and share their experiences of the city by posting photos, videos, comments and ratings on the 3D model of the city available online. Our technology has also been interfaced with another product by creating a protocol so that other applications can push content on the 3D model without using "Urbadeus". In building an urban game, it was demonstrated the interest of our participatory approach allowing users to share information with each other through the website of the city. This experiment was a success.

It has unfortunately not been possible to repeat this experience with Manchester City. The 3D Digital Model of the city was not available. It would have been necessary to have a larger budget for doing it and make our application available for them. Nevertheless a model was established to demonstrate the interest and feasibility of the operation in any European city.

Beyond the technological interest, Navidis has enjoyed opportunities to meet within the consortium but also by the many events organized by ENoLL. It allowed Navidis to make known its products at different opportunities and to engage its first international relationships and enable it to expand its market to more short term at least 8) PVI

Process Vision Oy has successfully piloted user-centric energy efficiency improvement concept in the Helsinki Living Lab. With the versatile online reporting and living lab oriented end-user commitment in the office building environment we reached remarkable results in saving energy, minimizing CO2 and minimizing energy bill. The several communication technologies and metering devices provide by many SME companies were integrated together enabling a smooth real-time measurement and reporting of energy consumption in different floors and spaces of the office building. The end user competition to promote energy efficiency was interesting and promising.

ICT PSP Project Reporting Template

56

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

9) PVM We would regard the work we did with APOLLON as crucial in helping us to develop our products across Europe. The experience has given us the confidence to develop the European part of our business; it has forced us to think about licensing our products in order to sell it to partners and linked us up with a number of organisations around technical platforms giving us a greater understanding of the diversity of the market. We felt that there may be barriers to or work due to language restrictions however our work with Issy Le Moulineaux proved that are ways around this. As a result of the APOLLON work we have developed our materials into a clearer training pack, developed our social licence model and have sold licences to 12 organisations across Europe as part of the Isabel project. We also have additional enquires from Stockholm who are likely to take a licence from us. As a result of this work we will be launching on April 19th the first European community reporter editorial board at Media City in Salford, Manchester. With the launch of this we are hoping that this will continue to grow our market across Europe.

10) TLV

The participation in the project for piloting Xtramira across borders yielded valuable lessons learned. The importance of local requirements became very clear, not only in a technical sense, but also towards eco-systems and business models. Televic not only improved their awareness of these important aspects, but this also resulted in improved processes based on the experiences in this project.

11) YDR

YDreams is a global SME that develops proprietary technology in field of new interfaces and interactive installations which through the APOLLON project had experienced the advantages that Living Lab can provide to SME’s when accessing international markets. YDreams had been responsible for the cross-border development of a proof-of-concept that an international partner wanted to develop from scratch with this international approach. This provided the field experience of being part of local company that helps the international contractor to specify and develop technology that, from the scratch, complies with local constrains, combining multi-national developers.

YDreams is a global company, and by participating in APOLLON project had gained the knowledge of a new methodology (cross-border pilot development) and new facility (Living Lab) that could be helpful in future international market initiatives.

4.1.5 Living Lab Experience of SMEs

Every partner in APOLLON had ample opportunity to work with local and cross border living labs. Nearly 91% of SMEs indicated their interaction with Living Labs as “satisfactory”. 73% of the partners recognized the importance of Living Labs as a critical component of their internationalization strategy (Figure 4-13).

ICT PSP Project Reporting Template

57

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

Figure 4-8: APOLLON Living Lab Experience

Among the key recommendations, all (82%) the APOLLON Partners agreed for the need for living labs to provide greater insights to existing and possible funding mechanisms for cross-border activities.

Figure 4-9: Recommendations for Living labs

4.2 Corporate business impact assessment 4.2.1 Business profile of APOLLON Corporate Partners Table 4-4-3 provides a general overview of their key profile attributes of corporate stakeholders active in APOLLON. Table 4-3: Business Profile of APOLLON Corporate Partners Profiles

1

2

3

4

Year of Inception

1896

2009

1972

1969

Number of Employees

200

5000

5000

5000

Years in operation Autonomicity

ICT PSP Project Reporting Template

116 No

3

No

58

40

Yes

43

Yes

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

It is to be noted that although size (No of Employee <250) of Profile-1 corporate partner may qualify this partner as an SME, an average revenue more than â&#x201A;Ź50M per year (see Figure 4-10) and lack of autonomicity allowed us to consider it as a corporate partner and not an average SME.

Overview of Revenue Generated (Y2009-Y2011)

It is to be noted that all the data and revenues in our assessment are anonymized for the sake of confidentiality with the partners. Figure 4-10 represents the annual revenues of each APOLLON corporate partners for year 2009, 2010 and 2011.

Figure 4-10: Corporate Revenues for years 2009, 2010, 2011

ICT PSP Project Reporting Template

59

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

Years of Experience and Employees in payroll

Figure 4-11: Experience and Employees in payroll

Shown in Figure 4-11 all except one corporate partner has more than 5000 employees in their payroll. It can also be seen from Figure 4-11 that most of the firms are fairly new in market (less than 40 years of operating experience).

Cross-Border Presence and Dependence

In order to gauge the cross-border presence and its importance to each corporate partner in APOLLON, efforts were made to understand their existing business model and dependence on international businesses.

Figure 4-12: Cross Border Activities and Impacts

As shown in Figure 4-12, 75% of the respondents identified their presence in international markets. Aligned with their size and maturity, 75% of the firms recognized their active involvement in the international markets for more than 3 years. However, only 50% of stakeholders indicated their direct dependency on international revenues, thereby further downplaying the need for extensive internationalization before consolidating local (and national) markets.

ICT PSP Project Reporting Template

60

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

4.2.2 Corporate Internationalization Modes of Internationalization Every firm adopts one or the other modes to venture into new markets, establish new business contacts etc. APOLLON as a platform provided a fresh start to these corporate partners to re-invent their internationalization strategy and consolidate the existing international markets.

Figure 4-13: Modes of Internationalization

As shown in Figure 4-13 among several alternatives outlined by us, 50% of stakeholders preferred the APOLLON Network to Distributorship, Franchise and Export type models. APOLLON Network was considered complementary to the successful “business as usual” modes of internationalization like – Parent company presence, JV and International Offices.

Barriers to Internationalization

Among others, it was the recession and lack of human capital/funds that made the corporate partners more cautious towards international expansion and operation. While 50% of respondents agreed upon lack of knowledge of foreign markets as one of the barriers they also agreed upon their lack of focus on internationalization as a barrier. In order to address these barriers, various cross border activities and experiments we triggered by APOLLON resulting in greater understanding of international markets, businesses and end-user requirements. More APOLLON contributions are elaborated in the Figure 4-19.

ICT PSP Project Reporting Template

61

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

Figure 4-14: Barriers to Internationalization

4.2.3 APOLLON Contributions and Impact As a key metric for assessing impact of APOLLON on Corporate Partners, each of the partners were required to identify and evaluate the contributions made by APOLLON and its consortium. Among several key contributions made by APOLLON, nearly 70% of respondents valued the cross-border living lab experience as most crucial to them. Second most important contribution of APOLLON was equally attributed to the understanding of technology platforms, testing and piloting experience in APOLLON, International business contacts and exposure during the entire project lifetime.

Figure 4-15: Contributions of APOLLON and Impact

Figure below presents individual ranking of APOLLON Contributions.

ICT PSP Project Reporting Template

62

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

Figure 4-16: Contributions of APOLLON [individual partners]

4.2.4 End of APOLLON: Directions and Exploitation Strategy 75% of the respondents agreed to maintain their international presence, while doing so, 50% of the corporate partners agreed to exploit the business contacts, venturing opportunities and lesson learned while working with APOLLON partners across Europe.

Figure 4-17: Post APOLLON Exploitation

All the Corporate partners were required to provide a snapshot of what could be considered as “APOLLON Success Stories and Achievements”. Given below are excerpts (un-edited) from each stakeholder highlighting the key economic and strategic goals achieved during the lifetime of APOLLON. ICT PSP Project Reporting Template

63

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

1) LEN We had big hopes in the beginning of the project to find a partner abroad that could provide us with a product that could visualize the energy usage in private households. Unfortunately some technical obstacles obstructed this. Now we only test international products but I hope that our evaluation of our pilot will contribute to APOLLON in many ways, such as behaviour change, thoughts about different techniques and input about what the customers wants.

2) SAP

The SAP Research Future Factory LL as a corporate LL built its own strategy based on the existing technology provided by SAP in the area of manufacturing industries.

From planning to execution we followed the red line of building prototypes consisting of the following setup: 1. Leading edge partner technology (hardware and/or software 2. Existing SAP solutions

3. SAP Research prototypes

which were/are combined to proof and deliver new solutions to existing and potential customers.

Those types of prototypes help SAP to position its product portfolio in a tangible way and the same does of course also apply to our more than 25 partners that we're working with under the so called Future Factory InitiativeError!

Reference source not found..

3) LIA

As a member of WP3 Energy Efficiency, Liander has contributed to APOLLON through research into Customer Engagement in the frame of pilot projects. We assessed the acquisition, retention and contribution of pilot project participants, through in-depth interviews with participants, cross-border interviews with project leads and questionnaires. This has resulted in a research report and presentation shared in WP3 Energy Efficiency. Liander has also actively shared knowledge from Liander's previous pilot project experience concerning energy management. We have presented and discussed pilot results at several international APOLLON work package and general meetings, where participants of other EU project were also present. Liander has contributed to cross-border exchange of data from energy efficiency pilots between WP3-members and cross-border liaising between SME's that focus on energy management.

4) LNB

Logica joined APOLLON for experimenting and piloting potential business propositions to the clients which are being developed or were developed in house. We participate in WP3: energy efficiency and WP2: homecare. The pilots we presented in APOLLON were communicated with potential clients and we knew there is an interest. We mainly communicated those pilots within Netherlandsâ&#x20AC;&#x2122; potential clients. As APOLLON is about cross-border Living Labs ICT PSP Project Reporting Template

64

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

collaboration, we had the opportunity to explore the needs of the international/EU market needs. Within APOLLON the pilots are being experimented, improved and customised based on the cross-border needs, which are not the same as the local Netherlands needs. The pilots are still ongoing and as a result the propositions are being improved. Especially considering the users' engagement and participation it gives more room for improvement of services. The engagement in cross-border experiments with Living Labs is valuable for Logica pilots and propositions, because we have an external view and contribution on our services we propose: (1) From Living Labs from another country,

(2) From other users than we considered initially

(3) From other potential clients which could be interested in new propositions as potential spin-offs from APOLLON. We invested in APOLLON with our own products/services, but we are gaining the experience of experiment which has a potential to widen our market.

We are also preparing demonstrators for the services we experiment within APOLLON, which will be installed in our SPARK innovation centre for as a showcase success story.

4.2.5 Living Lab Experience of corporate partners

Every partner in APOLLON had ample opportunity to work with local and cross border living labs. Thanks to the Living Labs active in APOLLON, all (100%) of the corporate partners indicated “satisfactory” experiences while working and interacting with the living. However only 50% of the partners noted the importance of Living Labs as a critical component of their internationalization strategy.

Figure 4-18: APOLLON Living Lab Experience

ICT PSP Project Reporting Template

65

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

Among the key recommendations, all (100%) the APOLLON Partners agreed for the need for living labs to liaise between the corporate and SMEs active in crossborder activities.

Figure 4-19: Recommendations for Living labs

4.3 Conclusions The business impact assessment presented in this chapter is based on a survey among the business partners of APOLLON, SMEs and corporate partners. As the number of partners is limited and as no control group was identified, we need to be careful in interpreting and generalizing the findings. These findings merely represent the opinions of the APOLLON partners and as such they still are valuable.

The available data on internationalization reveal that SMEs that were involved in APOLLON intensified their internationalization activities. Inspection of evaluation reports learns that the increase in internationalization activities can be attributed to the role of APOLLON.

An interesting finding is the role of barriers to internationalization. Besides the lack of resources it is also the lack of knowledge of foreign markets and lack of focus on internationalization that act as barriers. The contribution of APOLLON has been, to a large extent, to overcome these barriers. The survey demonstrates that SME partners evaluate as key contribution from APOLLON the international business contacts and exposure to and knowledge on an international customer base, and access to cross-border living lab environments.

The survey also shows that a key contribution from APOLLON is the testing and piloting experience offered to SMEs (70 %). The exploitation strategies of SMEs also confirm the benefits SMEs received from being part of a cross-border piloting environment.

About 73 % of SMEs recognized the importance of living labs as a critical component of internationalization strategy.

Regarding the four corporate partners, three mentioned lack of resources as a barrier to internationalization, and two also mentioned the lack of understanding of international markets and their lack of focus on internationalization. The contribution made by APOLLON rated highest was the cross border living lab experience. Besides, also international business contacts and exposure, understanding of platforms and standards and testing and piloting ICT PSP Project Reporting Template

66

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

experience were highly valued. Not surprisingly, the main exploitation directions for corporate partners are to widen the business prospects.

Two out of four corporate partners noted the importance of Living Labs as a critical component of their internationalization strategy. All partners recommended for living labs to liaise between the corporate partners and SMEs active in crossâ&#x20AC;&#x201D;border activities.

ICT PSP Project Reporting Template

67

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

5. Evaluation of Central APOLLON Objectives 5.1 Quantitative evaluation of the pilots Table 5-1 below presents quantitative data from the pilots (not included Homecare and Independent living). The cross-border activities have not been that numerous as intended at the APOLLON project start. The reasons for this are diverse, and dependent on pilot context. For example, in the health care area each countries eco-system around the matter plays a significant role, and local (national) adaptation needs to be done. Another reason is that SME’s do not show any need for cross-border collaboration.

Transferring intellectual products as a well as technology has also shown up to be rather difficult. For intellectual products to be transferred there is need of well-thought out and precise processes and routines for this to work out. When it comes to technology, comprehensive analyses must be done in advance, to make sure that the transferred technology function in the receiving environment.

Hence, the numbers are a bit disappointing.

Regarding user involvement, the APOLLON project show good results, 2260 users have participated in 57 activities, and resulted in 17 new ideas, 10 implementation and 8 redesigns of products, services, and processes.

Considering the difficulties faced in the pilots, this is an interesting result, which indicates that there are potential for improving these figures in well-functioning projects/pilots.

The 11 SME’s have gained good insights and participated well, despite the obstacles identified in the APOLLON project, for example SME’s more day-to-day, business-making focus. Even so, the APOLLON project can present benefits of cross-border collaboration for SME’s, e.g. there are possibilities to get new business proposals (10), even if the SME’s current situation did not change.

The 4 large enterprises succeeded to create new international partners contacts, and generate 2 new business proposals. Here were of course expectations for a better result, with new businesses, and new customers, in other countries. However, this can depend upon the areas covered in the APOLLON pilots, e.g. both health care and energy are rather context dependent, and as such not easily transferred cross borders.

When it comes to local authorities, the APOLLON results are rather good since, for local authorities, cross-border collaboration mainly concern knowledge matters. Hence, local authorities can learn a lot, but business is not their main objective.

Concerning products and tools for collaboration used in the APOLLON project, it is obvious that ordinary, well-established ICT tools works out fine. No new ones have been tried, but instead did the “old” ones fulfil the needs for 2 of 3 pilots, leading to increased access to relevant information, increased effectiveness in communication and co-creation of innovations.

Table 5-1: Quantitative results from the pilots ICT PSP Project Reporting Template

68

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report Key Indicator

Sum

No of cross-border activities (e.g. joint workshops, technology transfer, knowledge transfer, business transfer)

47

No of technology transfer activities (e.g. technology being transferred across a border)

9

No of intellectual products (e.g. methodologies, know-how etc) transferred in the experiment No of Users that has been involved in the experiments

No of user involvement activities (e.g. tests, experiments, workshops, interviews) No of new ideas that emerged from the cross-border collaboration with users (e.g. ideas on how to develop the product, service, methodology further) No of implementations of e.g. new functions as a result from the cross-border collaboration with users No of redesign of products, services, processes as a result from the crossborder collaboration with users

8

2260 57 17 10 8

No of SMEs involved in the experiment

11

No of signed letter of intent between partners and/or customers for the SMEs

3

No SME engagement activities (e.g. workshops, seminars, conferences) No of new international partners for the SMEs involved

No of new businesses generated in other countries for the SMEs involved No of new business proposals for the SMEs involved

No of new customers in other countries for the SMEs involved

Did the cross-border collaboration lead to increased turnover for the SMEs involved

Did the cross-border collaboration lead to increased customer retention for the SMEs involved No of Large Enterprises (LE) involved in the experiment

No LE engagement activities (e.g. workshops, seminars, conferences) No of new international partners for the LEs involved

No of signed letter of intent between partners and/or customers for the LEs involved for the LEs involved No of new businesses generated in other countries for the LEs involved No of new business proposals for the LEs involved

No of new customers in other countries for the LEs involved

45 11 2

10 1

No

A bit /n.a 4

10 6 0

0 /n.a 2 0

No of local authorities/public organizations (LA) involved in the experiment

6

No LA engagement activities (e.g. workshops, seminars, conferences)

7

No of new international partners for the LAs involved

3 /n.a

No of new businesses generated in other countries for the LAs involved

0 /n.a

No of signed letter of intent between partners and/or customers for the LAs involved No of new business proposals for the LAs involved

No of new customers in other countries for the LAs involved

Did the cross-border collaboration lead to increased turnover for the LAs ICT PSP Project Reporting Template

69

0

1 /n.a 0 /n.a

No /n.a

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report involved

Did the cross-border collaboration lead to increased customer retention for the LAs involved No of products that has been transferred in the experiment

No /n.a 7

No of cross-border collaboration tools that has been used the experiment

13

No of distributed cross-border collaboration activities (e.g. on-line conferences)

25

No of NEW (for the stakeholders) ICT-tools that has been used in the experiment

8

Did the cross-border collaboration tools you used lead to increased access to relevant information

No, yes, yes

Did the cross-border collaboration tools you used lead to increased cocreation of innovations among stakeholders

No, yes, yes

Did the cross-border collaboration tools you used lead to increased effectiveness in communication No of research activities that has been performed during the experiment

No, yes, yes 12

No of authored journal articles

5

No of authored conference papers

8

No of research conference presentations

22

No of new research projects initiated

6

Did the cross-border collaboration lead to increased comparability of Living Lab research

Yes, yes, n.a

No of Living Labs that has been involved in the experiment

10

No of new collaboration initiatives between Living Lab network members (planned, prepared or submitted)

4

No of new Living Lab network members

19

Did the cross-border collaboration lead to increased access to user communities in other countries?

No, yes, yes

Did the cross-border collaboration lead to increased learning of Living Lab collaboration in networks

Yes, yes, yes

Did the cross-border collaboration lead to increased value proposition to the stakeholder community

No, yes, yes

Did the cross-border collaboration lead to increased maturity of Living Lab management

Yes, yes, yes

Research shows good results in the APOLLON project, with 12 research activities, 5 journal articles, 8 conference papers and 22 conference presentations, numbers that most likely will increase since more of this is on its way. All the key performance indicators for APOLLON research were correctly met, and the dissemination activities are documented in details in WP6 (D6.3[14]). The APOLLON project also shows that cross-border collaboration leads to increased comparability of Living Lab research.

Table 5-2 : Overview of Research and Dissemination Activities (inputs from WP6) Key Performance indicators

ICT PSP Project Reporting Template

M12 70

M24

M30 Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

EU Countries reached

nr of organizations reached with the dissemination material

27/9

27/18

27/27

4/4

6/5

â&#x2C6;ź300/250

14/8

~500/400 ~150/100

Enoll+29/150 Enoll+40/300 ENoLL+500/800

Nr APOLLON Workshop Organized (general) 2/2 Nr APOLLON Workshop Organized (specific) 3/3

9/6

Nr Participants in APOLLON events

â&#x2C6;ź 200/120 60/20

60/50

Dissemination Material Produced

6/1

6+5/2

Participation of SMEs in thematic workshops and events

4+4/3

Living labs, finally, gain benefits from cross-border collaboration, since it generates both new collaboration initiatives and new network members. To get access to user communities in other countries, together with increased learning and also increased maturity indicates that it is advantageous for Living labs to participate in cross-border collaboration networks.

5.2 Contribution of methodology development

5.2.1 Process of methodology development and validation Methodology development has been taken up both by WP1 and by the pilots themselves. As the WP1 has worked on providing support to development of methods and tools relatively late during the project, the pilots have been on the forefront in this respect. However as the pilots had the priority to prepare, set-up and later to execute and evaluate the pilots, methodology development as such was not a key priority. WP1 has tried to overcome the initial lack of focused attention to concrete methods and tools by assigning liaisons that would interact between WP1 and the pilots and would support a cyclic process of joint methodology development, adoption, enhancement and validation. The liaison approach has worked reasonably well in maintaining links between WP1 and the pilots, however overall the collaboration and interaction between WP1 and the pilots was less than should be expected. For next projects it can be recommended to explicitly allocate methodology resources to pilots and establish collaboration with the over-all methodology activity within the work plan.

5.2.2 Methodology framework and specific methods and tools

Some of the main influential contributions on methodology development have been the Research Framework and the phased approach to creating, planning and running cross-border networks of living labs. A number of specific methods and tools has successfully been developed and used within the pilots. The ICT PSP Project Reporting Template

71

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

Homecare and independent living pilot has elaborated and used requirements identification, ecosystems analysis and value analysis. Within the energy efficiency pilot, the user behaviour transformation approach was used along with methods for knowledge and experience sharing such as road shows. The eManufacturing pilot has used a web cam based approach for living lab tour. The eParticipation pilot has adopted an application development framework.

A number of methods and tools that can be considered as crucial for the preparation and planning phase of cross border networks have not been used as APOLLON had these phases already passed when the project started.

All pilots have been using traditional communication tools to support crossborder collaboration processes, such as telephone, Skype, web conferencing, application sharing and e-mail, besides the important face-to-face meetings. To some extent collaboration platforms have been used such as MyBBT and Google Docs. Surprisingly, these tools are only scarcely used in the normal day-to-day project work environment. This indicates that there are a number of tools available that are suitable for cross-border collaboration and networking.

However the pilots do not report detailed experiences in using these tools nor do they report the use of more sophisticated tools.

The eManufacturing pilot identifies the need for more sophisticated marketplace tools that facilitates business identification, business generation, business execution and maintenance of the solutions that has been developed during the cross-border collaboration. This kind of trading platform could provide support for matching requests and offers of services, finding partners and other.

The collection of methods and tools and elaboration of the methodology framework and making it available at the Knowledge Center [17] can be considered as beneficial for next projects aiming to set up cross-border living labs networks.

5.2.3 Methodology validation

The actual validation of the methodology including specific methods and tools at pilot level has been limited. The three-monthly cyclic process of validation was conceived as including both supporting the pilots and evaluating the adoption and use of methods and tools, in such a way that an “action research” approach would have been implemented which would benefit both methodology and pilots. Key reasons for limited validation results was that the methodology validation process started relatively late, definition of the validation approach remained unclear at several points, and liaisons were not sufficiently immersed in both methodology development and piloting work which hindered the implementation of this approach.

5.2.4 Main suggestions for improvement •

•

Explicitly allocate methodology resources to pilot activities, set goals, and establish collaboration with the over-all methodology activity within the work plan.

At the beginning of the project work, organise joint methodology-pilot meetings to agree on the methodological approach and provide training.

ICT PSP Project Reporting Template

72

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

5.3 Value added of cross border living labs networking 5.3.1 Cross-border collaboration focus Within the four pilots, cross border collaboration served different goals: •

•

•

•

In the Homecare and independent living pilot the cross-border collaboration focused on the transfer of technology solutions from one country to another, to establish the ecosystem facilitating such cross-border transfer, transfer, and to investigate the contextual factors affecting the success of this transfer.

For the Energy efficiency pilot, the cross-border collaboration takes a less strict form and focused on the knowledge and data transfer among participating partners and stakeholders in different experiments participating to the pilot.

In eManufacturing pilot the cross-border activities circled around establishing collaboration between living labs, SMEs and a large company to jointly develop and test software services.

For eParticipation pilot the cross-border activities consisted on creating a collaborationship between parties aiming to integrate technologies or components developed elsewhere into an application.

5.3.2 Benefits of cross-border networking

Participating in cross-border networks provides several benefits to SMEs and other stakeholders. These networks enable to test technology in different cultural settings, and get an increased understanding of different stakeholder’s perspectives. In this way it is possible to learn about each other, and share knowledge and best practices. It provides the possibility to test technologies in different environments, with different circumstances, and different contexts.

Further, getting involved in a cross-border network facilitates making new partnerships and acquiring EU funding for new projects.

The energy efficiency pilot indicates that cross-border networking can contribute to the promotion of new energy efficiency management business models, fostering SME’s and energy service companies’ participation in local and decentralized energy production in partnership with local authorities. The eParticipation pilot demonstrates that cross-border networking has a positive impact on all partners’ status, Living Labs becoming more visible, enhancing the capability of SME’s to enter a wider market.

Participating in a cross-border network also contributes to increased understanding of different stakeholders perspectives. This is something that constitutes a good foundation when looking for new partners in future projects, pilots or for networking in general.

The APOLLON project has found results and recommendations that indicate that a network of Living Labs will benefit all parties involved, even if this did not happen during the APOLLON project period (D3.4[7]). However, the pilot evaluations witness that strong cross-border partnerships have been built (D4.4[10]). In these cross-border networks, contextual factors play a significant ICT PSP Project Reporting Template

73

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

role, and hence must be carefully considered (D2.4[5]). Of importance here is to determine what kind of eco-systems are involved, what value network will reign, and to decide on a common research approach.

During the pilots the cross-border networks have generated innovation and market creation opportunities, although not to the extent expected. Results also show that the Living Lab network contributes to a better take up of ICT services and applications, for example did the energy efficiency pilot demonstrate the end-users engagement and changed behaviour when it comes to energy saving.

Thus, several lessons are learnt, which lead to the APOLLON project being able to contribute with solid guidelines and recommendations for this kind of networks.

When it comes to the Living Lab network enabling transfer of applications and services from one Living Lab to another, this was most successful in the eManufacturing pilot, and the e-Media and eParticipation pilot (D4.4[10] and D5.5[12]). The reason for this might be due to the themes; eManufacturing and eMedia and eParticipation are probably not as context and environment dependant as eHealth and energy efficiency.

To involve SMEs into cross-border Living Lab activities has been a challenge in all the pilots. The lessons learned are that for SME’s to participate, they need allocated budget, clear and obvious time frames and solid incentives for their contributions. “What’s in it for me” is a good question to answer during start up, and also ask that question for each partner, when setting up roles and responsibilities. The gains of participating in cross-border networks must be explicitly declared. One advantage for SMEs is that they are offered an entry point to larger European markets in cooperation with more established, larger companies. The fundamental point is that entering European markets must be considered of value by the SME. Living labs have an important role to fulfil in cross-border networks. The APOLLON project research have shown that it is wished for Living labs acting as a nucleus for sharing innovative RDI activities within experiments as well as providing support in finding different parties that can contribute to these activities. Another role that Living labs could play is as the part having the overall view of the experiments, ensuring that things are done according to plans. This, however, should be discussed and sorted out in the start-up of new experiments, just like in any project.

In a wider perspective, the APOLLON project show that Living lab networks also act as a gateway for SMEs towards the different EU Member States and regions, making it possible for SMEs to explore possible new markets. In general Living labs contributes by constructing an open environment in which all types of stakeholders, with a special focus on SMEs, can enter and participate.

5.3.3 Challenges of cross-border networking Preconditions for networking Regardless of all the different possibilities available today to carry out projects cross-border by the aid of technology, all the APOLLON pilots witness of the importance to meet face-to-face. This is regarded as absolutely necessary, not ICT PSP Project Reporting Template

74

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

least in the beginning of cross-border networking, in order to get to know others and create a feeling of belonging, trust and a firm ground on which to build the continued networking. These face-to-face meetings should take place at least once in each participating country, and in such a way provide knowledge and understandings of each country’s environment and circumstances. Overall, transparency, trust and easy access to information are crucial when starting up cross-border networking.

A webcam tour can to some extent replace a physical meeting. The eManufacturing pilot succeeded to provide a detailed insight into capabilities and specific scenarios by a webcam tour with a camera that could offer 360o view of the SAP Research Future Factory Living Lab. Setting up the network

Naturally, what also should be in place is a clear structure of the cross-border activities that are to be conducted during the networking.

To establish a common, fixed eco-system for cross-border operations is not feasible, according to the Homecare and Independent living pilot, since a common eco-system must build on a common environment. In the health sector, as well as in the energy sector, country-specific ways of organizing, countryspecific laws, regulations, and rules and country-specific business processes does not make up a solid foundation for a common eco-system. The challenge then, is to match the ecosystems and elucidate and harmonize rules and roles between them, and to determine what kind of ecosystem, value network and common approach needs to be in place for cross-border activities or pilots. This finding is confirmed by the Homecare and independent living and by the energy efficiency pilot.

An important issue is to create commitment and engagement among partners.

One part of this is the clarification of each partner’s goals and expectations, since these can be very diverse. It is therefore important to create clear goals, to communicate objectives and to early on map demands and barriers the crossborder networking will face. This involves, as pointed out by the eParticipation pilot, harmonization of all partners objectives, which should lead to shared objectives for all project partners, and in turn also embrace meaningful reporting. To be noted is, that objectives must be realistic, too optimistic objectives might hinder collaboration. Once the network has reached an agreement of their cross-border collaboration, a partner’s agreement should be established. The eManufacturing pilot suggests that this agreement include key performance indicators (KPI), such as e.g. personnel availability; message priority and response time; and fix rate (how often information flows back and forth until the problem is solved).

Another part of this is to clarify each partner’s time frame. Living Labs and their environment does not change as rapidly as SME’s and other corporations.

In the energy efficiency pilot it was concluded that SME’s are too small for crossborder networking. Entering the European market is not their main aim, they are happy being active on the local and national level. This is confirmed by the homecare and independent living pilot, where the SME’s declined to formally ICT PSP Project Reporting Template

75

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

participate as associated partners due to financial constraints. SME’s are active in a context where the daily business and making money for survival is the primary concern. Long-term networking becomes less prioritised, probably since it does not render any immediate profit.

An important issue, therefore, is the need for firm considerations on how to allocate budgets. SME’s that not get any budget for their participation are likely to leave the network.

Another responsibility for the Living Labs can be to help overcoming barriers such as language, terminology, and cultural differences, and this is something that is mentioned by all pilots. The Living Lab should also be responsible for ensuring that communication keeps on flowing in the network.

Roles and responsibilities,

In cross-border networking all the partners of course get different roles and responsibilities, which need to be clarified. It is beneficial if this can be sorted out in the start-up phase of the collaboration. The Homecare and independent living pilot suggest that Living Labs could act as mediators, facilitating for stakeholders from different countries to come together, can identify common business interests and contribute to the creation of partnerships. In the eManufacturing pilot they identify Living Labs as an instrumental network that through cross-border collaboration with SME’s can help the latter to identify measures, methods and best practices for cross-border interactions. The eParticipation pilot sees Living Labs as supervisors, in line with the above.

In the Energy efficiency pilot it was demonstrated that agencies could act as a bridge, or a link, between local authorities, SME’s and universities. A possible explanation for this is that they are perceived as having a neutral role.

Moreover, results from cross-border networking can be used by local authorities, as foundation for new policies and regulations.

In general, when it comes to tasks and responsibilities these could very well be matched one-to-one, but not to specific actors. There is need for having access to a network of actors that can be activated and involved on a project base, during the ongoing pilot.

A prerequisite for cross-border networking is communication, and this communication should be close, and happen in reoccurring intervals. It is also possible, as shown by WP4, to agree on on-demand availability, i.e. partners agree to be available during office hours. This is especially important when it comes to technical aspects. Coordination should also happen on a regular basis, and the eManufacturing pilot suggests the use of a collaboration platform, and a common infrastructure for this. Such a platform makes it easy for partners to get in touch with each other.

How to deal with eventual changes in the group of project partners is an issue that needs to be considered. For the energy efficiency pilot it got consequences when a large enterprise, responsible for transferring systems, left the project.

But during a three-year project many thing can happen, e.g. mergers, drop-outs ICT PSP Project Reporting Template

76

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

etc. The challenge for the cross-border network is to quickly find innovative ways to adapt and accommodate to the changed situations.

Bringing in commercial players puts leads to the need to pay specific attention to results that stems from activities that are part of the networking. Questions such as; who own the results; as well as when and by whom the results are possible to use, needs to be clarified and agreed upon. The energy efficiency pilot findings show that SME’s are less open to share their solutions. Trust is a key word and vital for success, and the challenge for e.g. Energy Efficiency Living Labs is to create a stronger network, and harmonize services for bringing business benefits to SME’s, and being able to demonstrate the benefits crossborder collaboration could bring along.

Involving end-users

Cross-border networking that uses the Living Lab methodology and involves end-users adds significantly to increased knowledge of user needs and desires.

Here must be understood though, that users’ commitment is highly important.

When to involve the users is also of concern, this should be done as close to the actual activity or pilot as possible, to avoid users dropping out.

When users are expected to test technology in their home not only a technician should be present when the technical equipment or system is installed, but also a person able to explain every aspect of the pilot, as well as the philosophy behind the service. Moreover, the technology has to be embedded in the existing ecosystem, and in this way reflect the context in which the service or product under development will be commercialized.

Carry out experiments

First and foremost, it is beneficial to do some pre-tests before setting up a largescale experiment, something that might be especially true when testing technologies cross-border. The energy efficiency pilot demonstrates a case when metering solutions were not as mature as wished for; hence they had to postpone a project phase. Nevertheless, the APOLLON pilots have shown that all partners could benefit from cross-border collaboration in order to improve their technologies and better define their solutions. One lesson learned is that it was easier for startups and organizations based on knowledge transfer to adapt their technologies to cross-border testing, while it was harder for SMEs that had to develop specific software to ensure cross-compatibility as no budget was attributed to this task, according to the eParticipation pilot.

Conducting cross-border networking and experiments involves being active in different contexts. Hence, to collect contextual information and to share this among all projects partners is of great value, so that country- or region specific circumstances are understood by those involved. The homecare and independent living pilot emphasizes the importance of focussing on and distinguish between requirements tied to the local or home market, and requirements which are common for different markets.

In order to be able to analyse and compare cross-border experiences, and through this share knowledge, it is vital to use the same methods and ICT PSP Project Reporting Template

77

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

methodologies for setting up experiments and data gathering. The energy efficiency pilot has shown that when local situations and conditions differ too much, cross-border activities such as knowledge sharing becomes harder to carry out.

5.4 Sustainability of cross-border living labs networks

The issue of sustainability of the cross-border networks cannot be answered in a definite way, as the APOLLON pilots should be considered as projects in their own, in principle ending with APOLLON itself. In order to ensure long-term sustainability, APOLLON has undertaken the creation of domain networks for the pilot areas, acting as long-term viable breeding ground and community of innovation. As Chapter 3 explains, the prospects regarding sustainability are somewhat different across these domains.

5.5 Collaboration in cross-border networks

The four APOLLON pilots demonstrate that in each of the pilot environments the situational context of living labs collaboration was different. The four pilots should be seen as cases which we can learn from in making general conclusions for next activities. Table 5-2 summarizes the main situational characteristics.

Based on the analysis of the four pilots we propose the following recommendations.

Ensure collaboration agreement. One of the characteristics of the APOLLON project is that before the pilots started the project proposal had to be approved, and this led to the lack of a clear commitment of SMEs and living labs to agreed results. Actually this means that the connect phase (and partly also the plan phase) was taken for granted. However during the connect phase important agreements to be made are e.g. regarding business model, IPR, business proposition, and contractual agreements.

Define clear roles and responsibilities of living labs and SMEs and other partners. Roles definition, in particular as it comes to the role of Living Labs, is highly important. In case there is a transfer situation such as in the Homecare and independent living pilot and in the eParticipation pilot, the receiving living lab should assume the coordinating role. To ensure that this role can be fulfilled, the living lab should have the necessary competencies, expertise and skills.

Ensure an agreed common business case before starting. Most of the pilots experienced difficulties in engaging the partners and ensuring commitment.

Objectives, results to be achieved, time frames, needs and expectations of partners must be clearly defined and aligned to the project goals before the pilot starts. A win-win for all parties involved should be negotiated before the actual start. The pilot should be part of the roadmap of SMEâ&#x20AC;&#x2122;s and other parties involved. Also, the pilot should target clear business opportunities after projectâ&#x20AC;&#x2122;s end.

Ensure adequate project planning and project management. Pilots and experiments within the pilots are, after all, projects in their own. Sound project definition, project management and the use of project management tools is a precondition for success. Actually, this has been a problem in several pilots.

ICT PSP Project Reporting Template

78

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

Table 5-3 Cross-pilot analysis of collaboration in living lab networks Homecare and independent living

Energy efficiency

eManufacturing

eParticipation

Main collaboration issues

Resolving technical and legal issues Agreeing on project definition Living labs acting as “hub” for SMEs Creating receiving ecosystem

How to support B2B collaboration on testing of technologies How to create a marketplace for services

Creating a strong collaboration between living labs, and between living labs and SMEs, NGOs and start-ups

Collaboration bottlenecks

Benefiting from experiment outcomes elsewhere Creating a common benchmark for experiments, using it to improve technologies

Factors determining collaboration bottlenecks

Quality of project plan and project management

Financial resources of SMEs

Collaboration methods and tools successfully used

Project planning and management Value Analysis Communication tools

Quality of project planning and management Business interests of local partners Involvement of SMEs and their interest for internationalisation

Lacking expertise in Living Labs Inflexibility of SMEs technology solutions Financial constraints of SMEs

Need for engagement of partners involved Need to clarify the business case of cross border collaboration for all partners Focus on methods that can be used in different contexts e.g. behavioural change methods

Importance of a clear project plan and project management

Situational characteristics determining collaboration

Main learned

lessons

Different ecosystems sending receiving end

at and

the

Clear business case (win-win) for all partners needed Roadmaps of partners should be aligned Importance of value analysis for mapping partner objectives Importance of sound project management

Different Smart Meter experiments in several countries, interrelation mostly lacking

Lack of cross-border collaboration

Dissemination and exchange approaches e.g. Road shows

Middleware platform made available Pilot was organised as a project; clear use cases from the beginning

Difficulty for SMEs to participate without financial resources

Project planning and management Communication tools

Integration of technologies from elsewhere into applications for use in another context

Technical problems of integration of technologies Role of the living lab to coordinate the experiment Divergence of partner goals, skills and expectations

Low cost webbased communication tools

Clear expression of expectations by SME Agile approaches to succeed in integrating a technology into another application Matchmaking between SMEs and Living Labs is fundamental

Use adequate collaborative workspaces and communication tools. Most pilots are using a combination of usual communication tools such as ICT PSP Project Reporting Template

79

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

teleconferences, Skype and other tools. Use of shared workspaces is limited even where it seems that it could have contributed to better project planning and management.

Make sure that technologies to be tested or used in other contexts are compatible. Several pilots have been coping with the problem of technologies that have been developed in one context and not being compatible in another environment. This â&#x20AC;&#x153;localizationâ&#x20AC;? issue deserves more attention in terms of pilot preparation, technology analysis, testing procedures, and local situation analysis.

Also legal, cultural, social and organizational issues may hinder the adoption of a technology solution in another context.

ICT PSP Project Reporting Template

80

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

6. Conclusions and Outlook 6.1 Introduction This D1.6 Evaluation and Impact Assessment Report has presented achievements regarding the main objectives of the APOLLON project in terms of 1. Demonstrating the value added of cross-border living lab networking; 2.

Developing a methodology for cross-border living lab networking; 3. Ensuring long-term sustainability of cross-border living labs networking. Additionally we evaluated the approaches towards the specific harmonization and interoperability challenges within the vertical cross-border pilots. The key conclusions have been presented in Chapter 5.

The four pilot experiment environments must be understood as environments to explore the conditions for cross-border living lab networking in different situational circumstances. The vertical pilots are characterized by differences in terms of situational context variables (such as the particular innovation challenge, the sector and its characteristics, the partners and their interests).

Situational context also includes the harmonization and interoperability challenge addressed (common ecosystem, common benchmark, common platform, common service framework). The experimental setup therefore is highly case-based and explorative in nature. To a large extent the lessons learned are case-specific, adding to our understanding of what to do and what not to do regarding the setting up and operation of living lab networks in specific contexts.

Based on this, the vertical pilots have resulted in enhanced strategies for setting up cross-border living lab pilots (D2.5[6], D3.7[9], D5.6[13]).

6.2 Discussion of vertical pilot evaluation results Homecare and assisted living

The three experiments organized in this pilot (see D2.4 [5]) enabled the comparison of different living lab approaches and to develop a transfer strategy and working methods. One of the main findings is the importance of carefully establishing an ecosystem for living labs innovation within the health and wellbeing pilot. One of the conclusions from the pilot is that establishing a common, fixed ecosystem to benefit cross border collaboration of living labs was not feasible, due to the fact that different healthcare systems exist in different countries. However, the pilots also indicated that organizing simplified forms of such ecosystems built around actors and their roles and responsibilities is a necessary condition. In the set-up phase of the cross-border living labs collaboration project it is important to define the added value within the ecosystem. In case this added value is insufficiently clear and explicit, actors are less motivated to engage in collaboration.

The pilot results also show that the Living Lab has clearly been identified as a neutral and valuable partner to facilitate the innovation process from start to end. On the other hand this role â&#x20AC;&#x201C; and in particular the ambition to play a more active role in the deployment of the experiments - is challenged as it is expected ICT PSP Project Reporting Template

81

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

that the living labs also act as “ambassador” of the service that is experimented, which is often beyond what living labs are offering as a service.

In terms of methodology to support the setting up and operation of the living labs network, the four phases (connect, plan and engage, support and govern, manage and track) proved to be very useful stepping stones. Already across the three experiments within this vertical pilot the implementation was different, which demonstrates the need for contextualization of methods and tools. More specific methods and tools that were found valuable, included requirements analysis, value analysis, and contextualization and friendly-user testing of a service.

A final conclusion from this pilot is that each of the different actors involved (Living Labs, SMEs, Large Enterprises) benefited from the pilot in terms of gaining insights on how cross-border pilots work and the implications for the own organization. Living Labs enriched their current portfolio with new processes, tools and methods and gained valuable knowledge. SMEs were able to adjust their technology and service to enter foreign markets, as well as their business model and market positioning, but also participation in the pilot resulted in much more effort on domains not clearly foreseen. Large enterprises gained insights in benefits of the living lab, identified new market prospects and were able to explore of new business models.

Over-all the Homecare pilot has resulted in a rich collection of findings, experiences and evaluations that can serve as a help and source of inspiration for next projects in this domain. The topic of this pilot can be considered as exemplary as many SMEs will need to focus on international markets. The pilot also leaves unanswered questions such as the relation of living labs to existing innovation intermediaries oriented towards supporting SMEs. The value added of living labs collaboration has been demonstrated for this particular pilot.

However, the partnership was already decided beforehand so this does not tell us much about the “competitive position” of living labs vs innovation intermediaries.

Energy efficiency

This pilot addressed four single living lab environments, testing energy saving strategies and user behaviours using smart meters. Within the energy efficiency experiment (see D3.5[8], the four countries involved had a different basis for energy consumption because of the different climate conditions and the borders for energy efficiency set by these conditions. Legal entities as well as national energy players set demands to mitigate energy consumption as well as lowering or shifting of consumption peaks.

The cross border piloting as a concept has shifted from technological implementation cross-border into knowledge sharing among the pilot partners.

Metering set-ups cross-border were tested in the beginning of the project but with lack of common interest of partners operating in different countries the interfaces were deemed unprofitable to SMEs and were therefore finished. The living labs have been in close contact of each other through work package

ICT PSP Project Reporting Template

82

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

meeting and roadshows and with common tools and methodologies in use have been able to share lessons learned and common practices instead.

From the four different living labs few key issues have risen above others in terms of user involvement. The most flagrant issue in energy saving is the avenue with which the users are notified of their energy consumption and therefore are incorporated in the process of energy savings also on how to keep them engaged for the long-term. End users are most likely to change their consumption habits to greener ones if they have a good knowledge on what their usage has been before, what this usage means in terms of minutes and euros as well as clear objectives on what the consumption could be and with what means this could be achieved.

Users demonstrate an interest at the start of ICT use and interaction, but interest tends to decrease in time if users are not engaged and challenge on regular intervals. Hence, energy efficiency information workshops are essential to raise user awareness, provided messaging and language are appropriate to the audience involved. With this in mind cross border activities are vital in sense of fresh ideas and common methodologies for interpreting the user behaviour changes.

Real time data is an added value if the presentation is adequate to the audience, namely baseline and real time consumption displays. Real time measurement has gathered new players to the field of energy efficiency that necessarily have no motivation or the knowhow to create interoperable solutions into the emerging industry of smart metering vendors, instead all want to create a full range solution providing full package from measurements into costumer displays. This creates a large number of players in the market and creates interface challenges, as all are compatible with only their own software.

Conventional metering vendors or established energy management software providers can co-operate quite easily on national level because of old partnerships but new SMEs lack the knowledge or network of partners so they could focus on their core knowledge part of the solution.

The challenge for SMEâ&#x20AC;&#x2122;s is to take the advantage of being in permanent touch with up-to-date technologies, from different companies in the field of energy metering, with partners from different European countries, sharing ideas and forming business alliances. The most obvious benefit is for the Living Lab community and network to be able to actively disseminate APOLLON experiences and pilot results, at the local, national and European levels. The greatest challenge is the absence of a single uniform European Energy service market for consumers. There are different industry legacies, regulatory environments, standards and supporting instruments for each individual European country. This emphasizes the prime importance of an instrument like to bring obstacles and challenges in this emergent lead market to the attention of the European Commission.

International cooperation has shown benefits from exchange of experiences and lessons learnt from the partners, allowing the results of an initiative developed elsewhere to be appropriated and worked upon in other projects. This allows a convergence of resources, leveraging European-wide available assets (scientific

ICT PSP Project Reporting Template

83

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

excellence, technologies, methodologies, tools, experimental facilities, Living Labs, user communities) and avoids double work while achieving the same results. The results of the co-operation can be seen in the good results in energy savings of the four living lab pilots.

Overall the pilot demonstrates the potential value added of close exchange and collaboration between comparable experiments in the domain of energy efficiency. eManufacturing

The eManufacturing pilot (see D4.4[10]) focused on technical implementation of several Future Manufacturing use cases based on using a SAP Research middleware platform prototype. Due to the withdrawal of the Hungarian partners the originally planned triple cross-border business could not be explored. Given the then limited number of participants in the dual cross-border setting of Portugal and Germany, a close collaboration was set up and organized in a clearly defined and project. Both for SMEs, Living Labs and the Large Enterprise involved there was a clear value added of participating in the project, which was fundamental for the cross-border project collaboration.

Various means for collaboration, proper project management and technology training and transfer were applied and validated, both from the pilot team’s own selection and based on recommendations from WP1 (see D1.5[3]). A new means for building trust has been identified, when face-to-face meetings are not possible: the webcam tour combined with online application sharing lays a foundation for building trust in the technology capabilities of the partner, which is essential in the manufacturing environment. And the possibility to being exposed through such a means after having successfully completed a project with the Future Factory in Germany offers a valuable means of communication in addition for any SME abroad.

With respect to IPR issues, especially with the supporting partners, who were not bound by the APOLLON consortium agreement, contracts like “Software Development Licensing Agreement” and “Test and evaluation agreement” were generated, which could be of relevance for similar project activities, where e.g. one company wants to work on prototypes prior to product release.

A surprise was that the technical cross-border project went through very well.

The conclusion was, that in IT and B2B environment the language is mostly English, the terminology somehow universal, the processes from idea or concept to product via testing as well, so time could be taken to address the human factors and not make them become a hurdle.

Technology transfer was executed cross-border. Creation of new products and service offerings, which was meant to be done as co-activity between partners in Hungary and Portugal for both customers in Portugal and Hungary, could not be done, thus happened locally.

Whereas the APOLLON project setting initially relieved from partner selection, the ongoing collaboration on the pilot and especially the drop-out of Hungary raised the awareness for means to better support finding the right partners ICT PSP Project Reporting Template

84

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

abroad to start a new business and to find new customers. Whereas in the Portuguese case the local Living Lab proved to be the “trusted hub” for partner and customer identification, the question was how to support partner selection, business initiation and collaboration outside the current eco-system.

This resulted in the new concept of “APOLLON Marketplace” platform initiated

by SAP Research. This platform addresses the issue how partners can be brought together that are offering and seeking any kind of services that are ready for transnational projects regardless at which location they are based.

Interestingly, this Marketplace platform concept matches very well the thematic domain network and represents a potentially attractive business model triggering and supporting transnational business opportunities for SMEs in Europe (See: D4.5[11]).

eParticipation

This pilot focused on integrating software components and technologies into eMedia and participation applications (see D5.5[12], D5.6[13]). Its conclusions are on different levels: contextual factors, ecosystems, technical and organisational interoperability, lead market opportunities, and collaboration.

Regarding contextual factors, the pilot evaluation highlighted the role of access to eMedia and Mobile technology, as well as language barriers and copyright issues. For example, working with partners with different skills implies a pedagogical approach of assisting them in the use of new technologies. Also, future projects involving the usage of social media for sharing user generated content will need to be aware of copyright and IPR regulations. The piloting of eMedia solutions raised the question of IPR for solutions developed in combining two or more existing technologies. For future projects, IPR and related licenses (Copyright, Creative Commons) will need to be defined at the beginning of the project in order to let each partner decide what to put in the project and understand what they take out. Living Labs could assume a role to support hybrid models such as Creative Commons allowing copyright holders to share part of the knowledge generated in the project with a wider community. These factors have partly also been discussed in other pilots (e.g. Homecare and assisted living) and demonstrate that we need to be aware of human factors as well as business and regulatory conditions.

The pilot has identified some models of how to address the issue of maintaining a close connection with local ecosystems, by setting up forms of customer relations management. Much like the homecare pilot, SMEs working with living labs should define clear goals and clear value added of working with living labs on technical innovation. The pilot was also confronted with the issue of enlarging the ecosystem of involved partners., a situation which is expected to be common in this type of cross-border projects. The local living lab had an important role in this process as it succeeded in finding the required partner, thus demonstrating the way SMEs can benefit from working with living labs. However, additional measures must be taken to really get the new partners involved, such as making a budget reservation for new partners or for additional development costs, and encouraging the participation of SMEs that are able to quickly adapt to shifts in ICT PSP Project Reporting Template

85

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

the scenario and are willing to develop solutions such as APIs that facilitate the integration with third party applications.

The pilot also faced interoperability issues: technical interoperability of partner’s technologies and harmonization of partners’ goals. Technical interoperability was achieved through collaboration of SMEs along the definition of the pilot scenario. This involved ad-hoc development to achieve a sufficient level of integration and to be useful to explore the interest of investing further resources to achieve a seamless integration. Future projects will need to evaluate to which extent partner’s technology is interoperable requiring SMEs to provide detailed technology descriptions before starting a pilot and planning early technology assessment meetings among technology partners. SMEs should be encouraged to develop APIs as this facilitates communication with third party technology providers.

Regarding lead market opportunities, no SME increased its business directly because of APOLLON however APOLLON enabled partners to explore foreign markets and some partners benefited from APOLLON to redefine their business model and identify new business targets. Several partners are engaged in new European-level projects due to APOLLON. The pilot also focused on new business models for social media-based applications that require almost no investment by cultural institutions. Practical bottlenecks such as lack of travelling costs hindered SMEs to engage in meetings with clients, putting Living Labs in the position of SMEs “sales representative”. This again points to the crucial role of living labs – also in other pilots – and to the new set of skills and competencies these Living Labs should acquire compared with existing living labs in order to carry out their role in cross-border settings.

As concerns collaboration processes in the cross-border setting, the matchmaking between SMEs and living labs is to be considered as a fundamental step determining the success of collaboration between SMEs and living labs.

When the Living Lab ecosystem is not adapted to the SMEs, opportunities are lost. The pilot presents interesting examples of mismatches which invite us to pay due attention to the problem of living labs skills and competences. Also, SME solutions should be sufficiently flexible to enable the integration in the living lab ecosystem; agile approaches to prototyping and testing should be encouraged while working with a living lab. One of the conclusions of this pilot is that, for SMEs looking for internationalisation or market expansion, Living Labs should not fulfil the role of “salesman” however could act as single point of contact getting in touch with local living lab ecosystems.

Much like other pilots (energy efficiency, homecare and assisted living) the eParticipation pilot stresses the importance of knowledge transfer to address partner’s needs and based on partners’ skills. It is proposed to set up a platform facilitating knowledge exchange and reusing existing methods and tools, and to pay more attention to training sessions as a means to transfer knowledge among partners and to develop sustainable relations.

ICT PSP Project Reporting Template

86

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

6.3 Summary of main recommendations derived from the vertical pilots All pilots have identified recommendations based on their pilot evaluations. The following summarizes and integrates a number of key recommendations targeting cross-border settings and facilitating the set-up and operation of crossborder living labs networks. For details we refer to the corresponding vertical pilot evaluation and recommendation deliverables (D2.4[5], D2.5[6], D3.5[8], D3.7[9], D4.4[10], D4.5[11], D5.5[12], D5.6[13]). For methodology issues we refer to D1.4 [2]:

1) Apply the phasing approach contextualized to the pilot environment.

Most pilots applied the general phasing model (connect, plan and engage, support and govern, manage and track) to establish a framework for research and innovation in the pilot environment. The phasing model is found very valuable to guide experimentation but always needs contextualisation to the demands of the pilot.

2) The initial “connect” phase is critical but more attention is necessary.

Due to that fact that the vertical pilots started after APOLLON was approved, less attention was paid to the “connect” phase. Important decisions are taken in this phase such as partner finding, consortium building and planning of the project. Selection of the right business partners includes discussing the requirements and agreeing on solutions. Some of the difficulties experienced in the pilots later on e.g. collaboration bottlenecks can be attributed to decisions made before the actual pilot started.

3) For successful transfer and adoption, take into account “soft issues”.

Such issues include trust, privacy, liability, ethics, safety, and regulatory issues when transferring a technology. This recommendation especially stems from pilot work in Homecare and independent living. Transferring a technology implies issues of acceptance and adoption in other contexts; especially in healthcare this is a key issue. The eManufacturing pilot demonstrates that transparency, trust and an easy access to any type of information - regardless if the SME is a service provider or a potential service consumer – is crucial to start running new business engagements.

4) Carefully build and maintain the living labs ecosystem. The Living Lab eco system ensures the required conditions for product or service codevelopment. Both Homecare and eParticipation pilot have worked on building and maintaining the living labs ecosystem, including defining the roles and responsibilities of stakeholders and partners. The roles, responsibilities, competencies and skills of living labs need specific attention as living labs fulfil a critical role in the ecosystem. Skills and competences upgrading will be necessary in many cases. The pilots demonstrate new demands to Living labs e.g. active mediating roles in the eParticipation pilot.

The eManufacturing pilot demands relying on the local Living Lab having the knowledge of and contacts to reliable local partners, having checked them before inviting them into a new business relation.

ICT PSP Project Reporting Template

87

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

5) Define and agree on requirements at early stage. The eManufacturing pilot is an example where requirement findings is carried out in a relatively simple cross-border project setting. More complex requirements finding processes can be found in the homecare and independent living pilot which has worked on methods supporting this process. The eParticipation pilot recommends to Living Labs to be clear on capabilities and requirements of the product or service being tested, identify the target audience and adapt the product or service accordingly.

6) Address local adaptation and reliability of technologies as well as technology interoperability. Especially in cross border settings local testing of technologies and applications coming from elsewhere is highly important, as demonstrated by the pilots homecare, energy efficiency and eParticipation.

The Homecare pilot has elaborated a testing procedure. For eParticipation, technical interoperability of partnersâ&#x20AC;&#x2122; technologies was a key issue. Close collaboration of SMEs in defining the pilot scenario is a key condition for success. SMEs should be prepared to open up and jointly evaluate technologies to find common solutions. They should be encouraged to develop APIs to facilitate third party technologies integration.

7) Address the product or service business model. Several pilots indicate that not just costs but the business model (including also the value proposition) for putting a product or service into the market is relevant.

Business model issues play a role already in the early phase of cross-border projects.

8) Use low cost tools and processes to facilitate collaboration. All vertical pilots have worked with low cost communication and collaboration tools.

"On-demand"-availability in order to immediately react to unexpected technical issues is often critical (e.g. for the eManufacturing pilot).

Remarkably, shared collaboration workspaces have not been used much and probably some dedicated training is necessary to be offered by living labs, as part of project management procedures. It should be recommended to avoid fragmented use of tools and offer an open, common collaboration platform which has low cost tools integrated.

9) Define project planning and management procedures dedicated to cross-border settings. Explicit attention is paid to project planning, management procedures and skills in the homecare, eManufacturing and eParticipation pilots. A sound cross-border project plan is a necessity, however it should be transparent and open for adaptation to new circumstances. As the eManufacturing pilot formulates, not just for the business side of a project but also for the technical aspects like the technology knowledge transfer, the use case implementation, the use case execution and support and maintenance - a clear structure of the cross border activities is a "must have". All means of good project management must be applied: clear roles and responsibilities, as well as agreed means and modes for communication, timelines, tracking and other issues, this helps to avoid project failures, wasting resources and disappointment.

ICT PSP Project Reporting Template

88

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

10)Be aware of the potential need to modify the consortium composition.

The eParticipation pilot is a clear example of the need to add new partners during the pilot process. On the other hand the eManufacturing pilot had to facilitate the exit of an existing partner. So in general the consortium will change over time and early stage provisions must be made to facilitate changes in terms of budget, adaptation of IPR arrangements and other conditions.

11)Create cross-border thematic domain networks to support cross-border living labs networking. The role of WP6 together with vertical pilots has been to push development of thematic domain networks that act as wider networks bringing together living labs, SMEs and large enterprises, research institutes, authorities in a given domain. These networks establish a breeding ground of new cross-border initiatives and as a knowledge-sharing environment. They bring together knowledge, insights, practices, methods and tools, skills and potential partners. They stimulate collaboration and offer services to living labs and SMEs. The spectrum of what domain specific Living Labs can cover in order to bring SMEs faster to new markets includes a variety of services starting from governmental / legal related services up to straight technical services such as early prototyping - depending on the nature of the respective Living Lab. 12)Facilitate cross-border value creation. The SMEs - including all subcategories from really small enterprises to mid-size enterprises - should be enabled to run trans-national businesses not just by getting the chance to trade their services to other countries via the platform but also by getting support from the Living Labs network into the direction of "to be made prepared" for establishing businesses in other countries and regions than their "own".

13)Leverage on the unique capabilities of a living labs environment of being a cross-border test bed. The pilots demonstrate the unique potential of small-scale experimentation and testing in a well-defined and â&#x20AC;&#x153;friendlyâ&#x20AC;? cross-border setting, before larger-scale roll-out. Such well-defined and friendly environments still can take different shapes such as the corporate living lab environment (as in Future Factory living lab) or project-based and temporary collaboration between existing living labs (eParticipation, homecare pilots).

6.4 Final remarks on objectives and results achieved Value added of cross-border networking of living labs

Overall the results of the APOLLON experiments in the four vertical pilots demonstrate the potential value of cross-border networking of living labs.

However, the pilot results also point to the conditions that must be established in order to realize this potential value. For example, the skills and competences of Living Labs need enhancement in order to be able to fulfill the envisaged role of living labs in cross border settings. Living Lab domain networks may play a facilitating role to support the cross border networking. It should be considered ICT PSP Project Reporting Template

89

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

that the pilots were not able to test an important phase in the process of setting up and operating cross border living labs networks: the connect phase.

Methodology for cross-border living labs networking

APOLLON has elaborated a comprehensive set of methods, tools and collaboration guidelines aiming to support the full process of setting up and operating cross-border living labs (D1.4[2]). The Knowledge Centre[17], which stores descriptions of these methods and tools, is an important repository of high relevance for next projects and initiatives. Part of the collection of methods and tools have been developed, introduced and validated in the vertical pilot settings. The pilots demonstrate that methods and tools are important, but must be carefully adapted to the pilot context. Therefore methodology development is a combined bottom up and top down process. It should be considered that important methods and tools covering the connect and plan phases could not be elaborated and tested in the pilots, because the APOLLON pilots’ point of departure was an already existing consortium and business case.

Sustainability of cross-border living labs networks

Ensure long-term sustainability of cross-border living labs networking has been one of the key objectives of APOLLON. At the same time it is somewhat early to evaluate this objective as the results are in the future. It is fair to say that conditions have been established aiming to ensure sustainability. The pilots have actively worked on establishing thematic domain networks that are crucially important for sustainability of cross-border living labs networks (D6.5[15], D6.6[16]). Further development of these domain networks, providing breeding grounds of new initiatives and environments of knowledge exchange, and based on appropriate business models, should be stimulated and the role of ENOLL will be crucial. Based on the work in the pilots, several additional signs have emerged to illustrate the promise of sustainability. The D6.5 presents many examples of exploitable results. Some pilots have developed initial business models for the services they have been working on. New projects and spin-offs have emerged from the pilots work. More insight has been generated in viable business models for collaboration between living labs, SMEs and other partners. The concept of “APOLLON Marketplace”, aiming to support matchmaking between service consumers and service providers, also shows a lot of promise, in combination with the thematic domain networks.

6.5 Apollon Success Stories 6.5.1 New product Lines

During the period of 2009 to 2012, APOLLON partners (SMEs and Large Corporations) collaborated closely with each other to develop new product lines and features. Best example that demonstrates this inter-partner exchange of knowledge and technology is the QUBY case in the Netherlands and Tunstall case in the Finland.

QUBY: Home Automation Europe (HAE), WP3 APOLLON partner specializing in Home Energy Management wanted to test and integrate their product QUBY with ICT PSP Project Reporting Template

90

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

stakeholders in Energy and Home Automation Business. Logica used this opportunity to integrate their EV charging (CiMS) product with QUBY display technology to develop an Integrated Home Controlled EV Charging Model for their customers in the Netherlands. This new product now enables residential customers to charge their Electric Vehicles without overloading or offsetting their connection capacity.

Tunstall Healthcare: Tunstall Healthcare worked as a supporting technology partner of the APOLLON Homecare Domain Network. Tunstall tested and evaluated its video presence technology and use cases to commercially roll out two products lines not just in the European Union, but across Tunstall’s footprint. Currently Tunstall has two initiatives in progress where video presence will be used. One focuses on Telehealth at a distance and the other focuses on Telecare with social inclusion.

6.5.2 New Business Partnerships

SAP-Ydreams: As a consequence of APOLLON, working together in the Energy Monitoring and Asset Viewing use case a strategic business partnership was created between a Living Lab (Future Factory) in Germany and an SME (Ydreams) in Portugal. Using this partnership, Ydreams, took advantage from the provisioning of a software technology - the SAP Research MDI platform prototype and combined that with their knowledge and expertise to solutions that helped other SMEs such as Imeguisa with their demand for energy monitoring and CENI with their demand for an asset viewing solution to get the right and best product and created two new business partnerships.

6.5.3 New Mergers and Acquisitions

INNOVITING: As a partner in APOLLON, INNOVITING participated in the Homecare and Independent living pilot where they successfully transferred their ADL sensor system to Spain. During this process an entire redesigned of the web application, lingual interface was changed. Thanks to the visibility provided by APOLLON and cross-border activities, the ADL system was acquired by LIVIND (Owned by Adesys and Bureauvijftig).

Process Vision: Process Vision Oy has successfully piloted user-centric energy efficiency improvement concept in the Helsinki Living Lab. With the versatile online reporting and living lab oriented end-user commitment in the office building environment PVI reached remarkable results in saving energy, minimizing carbon and energy bills. During APOLLON PVI was able to work in 4different sites, 4-different countries under 4-different regulatory environments that lead to an increased understanding of developing new businesses in larger BRIC markets. During the course of APOLLON, Herkules and Elis Holding acquired Process Vision in 2011.

6.5.4 New Spin Offs and Markets

ISA: ISA extensively used the opportunities created by APOLLON cross-border Living Labs to interact directly our counterparts from Finland (APOLLON partner - Process Vision). Working towards a vision to venture in new international markets, ISA created a spinoff named ISA Sul America in São ICT PSP Project Reporting Template

91

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

Paulo, Brazil. The enthusiasm and the business potential of Brazil for the Energy Efficiency knowledge and technology led ISA in Portugal to invite other SMEs in order to create a Living Lab embracing the complete value chain of the Energy Efficiency domain. The partnerships in this domain led to an initial number of 30 partners that created ISaLL (Intelligent Sensing and Smart Services Living Lab).

People’s Voice Media: Working towards their goal to create a «Reuters of the community», PVM participated in the Manchester Pilot of APOLLON project as SME partner. Through their participation in APOLLON, PVM developed its flagship Social License Model® and have sold licenses to 15 organizations - 5 in the UK and 10 across Europe. Now, PVM is looking to develop their market in France (through APOLLON Partner - Issy Media), Italy, Spain, Turkey and Sweden.

ICT PSP Project Reporting Template

92

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

References [1] D1.3: Framework for APOLLON Evaluation and impact assessment, incl.

KPI definition & measurement [M9]. 28.10.2010.

[2] D1.4: Recommended Toolset and Collaboration Guidelines for CrossBorder Living Lab Networks [M29]. 17.04.2012.

[3] D1.5: Validated APOLLON Methodology for Cross Border Living Lab Activities [M29]. May 2012.

[4] D2.2: Common Living Lab approach [M9]. June 2011.

[5] D2.4: Evaluation and Recommendations Report on the Cross Border Experiment [M29]. 30.03.2012.

[6] D2.5: Strategy for Setting Up Cross Border Pilots in Homecare and Independent Living Domain [M29]. April 2012.

[7] D3.4: Setup of the Experiment in the crossborder Living Lab [M12].

[8] D3.5: Evaluation report on Cross Border Collaboration [M24]. 12.04.2012.

[9] D3.7: Strategy for setting up cross-border pilots in Energy Efficiency domain [M29].

[10]D4.4: Pilot Results and Evaluation of the Cross Border Experiment [M24].

31.10.2011.

[11]D4.5: Impact Assessment and Best Practices [M29]. Version Final (undated).

[12]D5.5: Evaluation Report on Cross Border Pilots [M24]. Version 0.8, 24.01.2012.

[13]D5.6/D5.7: Recommendations for a Cross Border Network of Living Labs / Strategy for Setting Up Cross Border Pilots in eParticipation Domain.

Version 1.0, 26.04.2012.

[14] D6.3: Report on Networking Activities & Workshops [M29], 01.05.12 [15] D6.5: Exploitation and Sustainability Report [M29]. Version 0.7, 30.04.2012.

[16]D6.6: Establishment of Sustainable Thematic Domain Living Lab Networks [M24]. Version FINAL, 31.10.2011.

[17]European Living Lab Knowledge Center, Live at http://knowledgecenter.openlivinglabs.eu/

[18]Doodle, Live at http://www.doodle.com/ [19]MyBBT, Live at https://www.mybbt.be

[20]Dropbox, Live at http://www.dropbox.com [21]Skype, Live at http://www.skype.com

[22]Google Docs, Live at http://docs.google.com

ICT PSP Project Reporting Template

93

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

[23]Vimeo, Live at http://www.vimeo.com

[24]Delicious, Live at http://www.delicious.com [25]Pinboard, Live at http://pinboard.in

[26]ENoLL, European Network of Living Labs, Live at http://openlivinglabs.eu/ [27]Basecamp HQ Live at http://basecamphq.com/

[28] SAP Future Factory, Live at http://sap.com/futurefactory

ICT PSP Project Reporting Template

94

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

Appendix 1: SME Profile Overview

1. Name of your company 2. Would you consider your company to be an SME according to the European Definition?

(Employee <250; Turnover< €50Mill) Yes

No

3. What year your company was founded 4. Is your company an autonomous entity or part of larger entity?

Autonomous Entity Part of an Organization 5. Number of employees 51<200 6. What is the major source of funding for your company? (Check all that apply) Owner’s Equity Local Equity Bank Loan Foreign Stakeholder Investment State-funds Other Sources (Please Identify) 7. Turnover (in €) [Any or All] A) 2009 B) 2010

C) 2011

INTERNATIONALIZATION 8. Does your company have any presence abroad between 2008 to 2011? If yes, please indicate the percentage of revenue derived from operations in other countries. (Rough estimations are expected) 2008

Yes

No

Revenues (%):

0-5% (Please Select)

2009

Yes

No

Revenues (%):

0-5% (Please Select)

2010

Yes

No

Revenues (%):

0-5% (Please Select)

2011

Yes

No

Revenues (%):

0-5% (Please Select)

Apollon – D1.6 Evaluation and Impact Assessment Report Note to Respondent: If your answer to Question 8 was “Yes”, please proceed to Question 9, if any of the answer to Question 8 was “No”; please answer Questions 8a) below. a) Please identify the major barrier that prevented your organization from cross border activities? (Check all that apply) Lack of focus on internationalization

Lack of business contacts in international markets Lack of resources (funds/human capital) for venturing into foreign markets Lack of exposure to end-use customer in foreign markets Lack of support from government and local trade authorities Lack of Knowledge of international regulatory and technical environments Other Reasons (Please Specify)

9. How many years has it been since your company established its presence abroad (by any methods indicated in question below? <1 Year

1-2 Years

2-3 Years

>3 Years

10. Please indicate the mode of entry that your company has used to internationalise. (Check all that apply) APOLLON Network Direct Export/Re-Export Franchise/License Ownership Distributorship International Operations/Offices Joint Ventures and Partnerships M&A, Parent Company Presence Others, (Please Specify) 11. Is your company planning to expand its cross-border activities in and after 2012 Yes 12. What is the expected percentage of revenue from your international engagements in 2012?

(Rough estimations are expected) 0-5% (Please Select) 13. Please identify major barriers that you expect when engaging in cross border activities?

(Check all that apply) Lack of focus on internationalization ICT PSP Project Reporting Template

96

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report Lack of understanding of International markets Lack of resources (funds/human capital) for venturing into foreign markets Lack of exposure to end-use customer in foreign markets Lack of support from government and local trade authorities Other reasons (please specify)

14. Does your company plan to exploit the international experiences, business contacts and lessons learned from APOLLON to venture further into newer markets? Yes 15. Rate the major contributions of APOLLON to expand/operate in international markets.

Note to Respondent: Rate your experiences from 1-4 based on level of importance: 1 - Not Important; 2 - Important; 3 - Very Important; 4 - Critical; Access to cross-border living lab environment

Please Select

Exposure to and knowledge of international customer base

Please Select

International business contacts and exposure

Please Select

Testing and piloting experience

Please Select

Improved vision and strategy for doing business abroad

Please Select

Improved knowledge of regulation in foreign markets

Please Select

Understanding of technology platforms and standards

Please Select

16. How many employees were engaged in international business development activities and projects between 2008 to 2011 period.

2008

Employees in International Business Development:

0-5 (Please Select)

2009

Employees in International Business Development:

0-5 (Please Select)

2010

Employees in International Business Development:

0-5 (Please Select)

2011

Employees in International Business Development:

0-5 (Please Select)

LIVING LAB EXPERIENCE 17. Did your interaction with the Local and Cross-border Living Labs through APOLLON proved instrumental in achieving your goals of Internationalization? No

18. What can and should Living Labs do to further help SMEs to achieve success in cross-border activities? (Check all that apply)

ICT PSP Project Reporting Template

97

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report Liaise between SMEs active in similar cross-border activities Provide insights into funding mechanisms from financial institutions Provide advice on cross-border sales and marketing Provide insights on supply-demand side, enabling SMEs to position better Provide advice and insights on standardizations and technical interoperability Provide detailed insights on local market penetration and demand rates 19. Rate your entire Living Lab experience Satisfactory

SME SUCCESS STORY Begin ...

Short summary of your engagement and results during APOLLON in the shaded region (300 words)

ICT PSP Project Reporting Template

98

Version 1.0, 08/05/2012

Apollon – D1.6 Evaluation and Impact Assessment Report

Appendix 2: Corporate Profile Overview

20. Name of your company

21. What year your company was founded 22. Is your company an autonomous entity or part of larger entity?

Autonomous Entity Part of an Organization 23. Number of employees 0<9 24. What is the major source of funding for your company? (Check all that apply) Owner’s Equity Local Equity Bank Loan Foreign Stakeholder Investment State-funds Other Sources (Please Identify) 25. Turnover (in €) [Any or All] A) 2009 B) 2010

C) 2011

INTERNATIONALIZATION 26. How many years has it been since your company established its presence abroad (by any methods indicated in question below? <1 Year

1-2 Years

2-3 Years

>3 Years

27. Please indicate the mode of entry that your company has used to internationalise. (Check all that apply) APOLLON Network Direct Export/Re-Export Franchise/License Ownership Distributorship International Operations/Offices

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report Joint Ventures and Partnerships M&A, Parent Company Presence Others, (Please Specify) 28. Is your company planning to expand its cross-border activities in and after 2012 Yes

29. Please identify major barriers that you expect when engaging in cross border activities?

(Check all that apply) Lack of focus on internationalization Lack of understanding of International markets Lack of resources (Funds/Human Capital) for venturing into foreign markets Lack of exposure to end-use customer in foreign markets Lack of support from government and local trade authorities Other Reasons (Please Specify)

30. Does your company plan to exploit the international experiences, business contacts and lessons learned from APOLLON to venture further into newer markets? Please Choose 31. Rate the major contributions of APOLLON to expand/operate in international markets.

Note to Respondent: Rate your experiences from 1-4 based on level of importance: 1 - Not Important; 2 - Important; 3 - Very Important; 4 - Critical; Access to cross-border living lab environment

Please Select

Exposure to and knowledge of international customer base

Please Select

International business contacts and exposure

Please Select

Testing and piloting experience

Please Select

Improved vision and strategy for doing business abroad

Please Select

Improved knowledge of regulation in foreign markets

Please Select

Understanding of technology platforms and standards

Please Select

LIVING LAB EXPERIENCE 32. Did your interaction with the Local and Cross-border Living Labs through APOLLON proved instrumental in achieving your goals of Internationalization? Please Choose

ICT PSP Project Reporting Template

100

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report 33. What can and should Living Labs do to further help corporate partners to achieve success in cross-border activities? (Check all that apply) Liaise between Corporate and SMEs active in similar cross-border activities Provide insights into funding mechanisms from financial institutions Provide advice on cross-border sales and marketing Provide insights on supply-demand side, enabling Companies to position better Provide advice and insights on standardizations and technical interoperability Provide detailed insights on local market penetration and demand rates 34. Rate your entire Living Lab experience Please Choose

SUCCESS STORY Begin ...

Short summary of your engagement and results during APOLLON in the shaded region (300 words)

ICT PSP Project Reporting Template

101

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

Appendix 3: Evaluation and Impact Assessment Framework The Evaluation and Assessment Framework The D1.3 [M9] documents the framework for APOLLON Evaluation and Impact Assessment and provides definitions and measurement guidelines for KPIâ&#x20AC;&#x2122;s. The framework supports evaluation of the vertical pilot experiments, but is also complemented with specific questions addressing situational aspects such as usability of technology. The framework consists of sets of questions related to the following topics: 1. Background information: information needed to view the experiment in the correct context.

2. Approach: methods and techniques that have been used to support the crossborder collaboration within APOLLON. 3. Partners and Users: the role of the different actors (end-users, SMEs, large enterprises, local authorities) in the pilot and the results of their contribution.

4. Technology and Infrastructure: the role mof new and existing ICT technology to facilitate new ways of cooperation and co-creation.

5. Research: The learning and reflection resulting from the Living Lab and their relevance to theory and practice.

6. Management: Ownership, organisation and policy aspects of the Living Lab within the context of cross-border collaboration.

7. Qualitative questions regarding: cross-border collaboration experiences and evaluation, benefits from cross-border collaboration, challenges, methods and tools, partners, research and Living Lab experience.

8. Pilot-specific evaluation: observations regarding the purpose of each of the pilots to investigate the different contextual factors influencing cross-border collaboration in their setting.

Using the framework in assessing the vertical pilot experiments

The framework has been discussed thoroughly with pilot experiment leaders who have applied the framework in monitoring and assessing the pilot experiments (tasks X.5), as reported in the vertical pilot evaluation deliverables. The framework also has been used in proposing the reporting structure for the evaluation deliverables.

Validation of the Evaluation and Assessment framework

This aim of the evaluation and impact assessment framework was to identify the added value of cross border Living Lab networking. This framework has been validated by its application in the different vertical domain work-packages (WP2 to ICT PSP Project Reporting Template

102

Version 1.0, 08/05/2012

Apollon â&#x20AC;&#x201C; D1.6 Evaluation and Impact Assessment Report

WP5). The assessment framework has been used to evaluate the different vertical experiments in relation to the general objectives. This has lead to an evaluation report of the four APOLLON.

In the APOLLON project, the assessment framework has been validated in an interactive manner where all relevant partners have been invited to participate and provide their feedback. The first step of the validation of the framework was through a presentation of the framework at a General Assembly meeting in September 2010. The aim of this event was to inform the project partners about the assessment framework and also gather their feedback on the framework. The assessment framework was developed further among the partners in the task. The second step of the validation was to send the framework to the partners responsible for the evaluations in the vertical domains to make sure that the assessment framework was answering to their needs and that it was commonly understood among the partners. The input from this session was incorporated in the final design of the assessment framework that constituted the essence of deliverable 1.3 that was delivered in November 2010, and it was approved by the reviewers.

To smooth the process of implementing and understanding the framework and to harmonize the reporting of the assessment of the pilots a template for the deliverable from task x.4 was developed and distributed to the vertical work packages. The template for the deliverable was communicated through the liaison persons so that all partners of the work packages, especially with the persons responsible for task x.4 in the vertical work packages. In addition, to further facilitate the implementation process of the evaluation framework, phone conferences where all the partners were invited was arranged. In this conference, the partners were encouraged to ask questions about the template and the assessment framework, the process of the usage of the framework and the template was discussed and the required material was distributed to the partners involved in the task. The evaluation documents were also presented and the process was discussed in the General Assembly in October 2011.

ICT PSP Project Reporting Template

103

Version 1.0, 08/05/2012