Page 1

DELIVERY MANAGEMENT PLAN AND TESTING SPECIFICATION Human-enhanced time-aware multimedia search

CUbRIK Project ICT-287704 Deliverable D9.2 WP9

Deliverable Version 1.0 – 31 March 2012 Document. ref.: cubrik.D92.ENG.WP9.V1.0


Programme Name: ...................... ICT Project Number: ........................... 287704 Project Title:.................................. CUbRIK Partners:........................................ Coordinator: ENG (IT) Contractors: UNITN, TUD, QMUL, LUH, POLMI, CERTH, NXT, MICT, ATN, FRH, INN, HOM, CVCE, EIPCM Document Number: ..................... cubrik.D92.ENG.WP9.V1.0.doc Work-Package: ............................. WP9 Deliverable Type: ........................ Report Contractual Date of Delivery: ..... 31 March 2012 Actual Date of Delivery: .............. 31 March 2012 Title of Document: ....................... Delivery Management Plan and Testing Specification Author(s): ..................................... Vincenzo Croce (ENG), Paolo Mabboni (ENG) ....................................................... Piero Fraternali (POLIMI) ....................................................... Marco Tagliasacchi (POLIMI) ....................................................... Uladzimir Kharkevich (UNITN) ....................................................... Aliaksandr Autayeu (UNITN) ....................................................... Theodoros Semertzidis (CERTH) ....................................................... Ismail Sengor Altingovde (LUH) ....................................................... Markus Brenner (QMUL) ....................................................... Patrick Aichroth(FRH) ....................................................... Mark Melenhorst (TUD) ....................................................... Martha Larson (TUD) ....................................................... Ralph Traphรถner (ATN) ....................................................... Bjoern Decker (ATN) Approval of this report ............... Executive Committee Summary of this report: .............. Plan for CUbRIk Platform Releases Delivery History: .......................................... Keyword List: ............................... Release Plan, Testing procedure Availability .................................... This report is public

This work is licensed under a Creative Commons Attribution-NonCommercialShareAlike 3.0 Unported License. This work is partially funded by the EU under grant IST-FP7-287704

CUBRIK Delivery Management Plan and Testing Specification

D92 Version 1.0


Disclaimer This document contains confidential information in the form of the CUbRIK project findings, work and products and its use is strictly regulated by the CUbRIK Consortium Agreement and by Contract no. FP7- ICT-287704. Neither the CUbRIK Consortium nor any of its officers, employees or agents shall be responsible or liable in negligence or otherwise howsoever in respect of any inaccuracy or omission herein. The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7-ICT-2011-7) under grant agreement n째 287704. The contents of this document are the sole responsibility of the CUbRIK consortium and can in no way be taken to reflect the views of the European Union.

CUBRIK Delivery Management Plan and Testing Specification

D92 Version 1.0


Table of Contents 1.

INTRODUCTION................................................................................................................ 2 1.1 CUBRIK ARCHITECTURE ............................................................................................... 3 1.2 WHAT IS A CUBRIK PIPELINE? ...................................................................................... 4 1.2.1 CUbRIK Pipeline Synchronization ....................................................................... 5 1.2.2 CUbRIK Pipeline Data ......................................................................................... 6 1.3 WHAT IS A CUBRIK PLATFORM RELEASE? ..................................................................... 6 1.3.1 Release accompanying Document...................................................................... 6 User guideline.............................................................................................................................. 7 Inter components dependencies.................................................................................................. 7

1.3.2 1.3.3 1.3.4 1.3.4.1. 1.3.4.2. 1.3.4.3. 1.3.4.4. 1.3.4.5. 1.3.4.6. 1.3.4.7.

1.3.5 2.

Platform Service ................................................................................................ 27

MAIN RELEASES .......................................................................................................... 29 RELEASES .................................................................................................................. 29

TESTING.......................................................................................................................... 34 3.1 3.2 3.3 3.4

4.

Logo detection ......................................................................................................... 8 News History...........................................................................................................11 People Identification................................................................................................14 Like Lines................................................................................................................15 Media Entity Annotation ..........................................................................................17 Crosswords .............................................................................................................21 Accessibility aware relevance Feedback ................................................................23

RELEASES SCHEDULING AND PACKAGING STRATEGY ........................................ 29 2.1 2.2

3.

Artefacts............................................................................................................... 7 Vertical demos – the CUbRIK Applications ......................................................... 7 Horizontal demos................................................................................................. 7

UNIT TEST ................................................................................................................... 34 CONSUMER-PROVIDER TESTING ................................................................................... 34 INTEGRATION TESTING ................................................................................................. 35 END TO END FUNCTIONAL TESTING ............................................................................... 35

REFERENCES................................................................................................................. 36

CUBRIK Delivery Management Plan and Testing Specification

D92 Version 1.0


Executive Summary This document describes the plan for CUbRIK platform releases delivery. Moreover the testing procedure, introduced as early version in D9.1, is officially defined with some additional detail on respect of the early version. Chapter 1 provides an introduction on the CUbRIK architecture and describes what will constitute each release of the Platform. In doing this the demonstrators exploited for analysis and requirements elicitation are introduced. Chapter 2 provides the timing and goals of the five planned releases, reporting a detailed table of “what will be in� each release. Then, Chapter 3 is specific for testing strategy, introducing the concept of partner role in the procedure for CUbRIK artefact delivery.

CUBRIK Delivery Management Plan and Testing Specification

Page 1

D92 Version 1.0


1.

Introduction

The goal of the CUbRIK architecture design is to provide a light-weight platform for executing pipelines, each pipeline is an arbitrary mix of human and machine tasks. Pipelines use heterogeneous components, mixing open-source and proprietary software components and services realized by different organizations. The architectural concept of CUbRIK differs from the development of a monolithic do-it-all architecture (e.g., multimedia databases), in that: There are no technological assumptions on the nature of the multimedia data processing components that participate to a content or query processing pipeline; There are no assumptions on the structure of the pipelines that realize a piece of CUbRIK functionality. Each pipeline is characterized by its specific workflow The integration mechanism of pipelines and components is a conceptual model of data (specified in D2.1), which expresses the Object Model of CUbRIK data. Each component is responsible of adapting itself to the CUbRIK data model, via appropriate input/output adaptors. In principle, there is no assumption also on the data storage technology and on the location of data. Components could use different data storage and distribution technologies and policies. If, e.g., for performance reasons, multiple components or pipelines need to share ad hoc data structures and formats (e.g., data caches), they remain responsible for inter-component and inter-pipeline communication and data exchange protocols local to such a component / pipeline pool; when they communicate with other generic components and pipelines (e.g., for accessing input data collections, for communicating conflicts to a conflict resolution module, for interacting with general-purpose crowdsourcing platforms), they must adhere to the CUbRIK data model for data representation and on SOAP and REST for multimedia Web service interactions, possibly enhanced with MTOM1 and SOAP Messages with Attachments2. The design of the CUbRIK architecture is an example of differential design. The architecture is based on SMILA3 as the underlying framework for supporting workflow definition and execution. The goal was to exploit SMILA capabilities to enable easy integration of data source connectors, search engines, sophisticated analysis methods and other components by gaining scalability and reliability out-of-the-box. Therefore, the design of the CUbRIK architecture has proceeded by: Identifying the technical requirements of human computation enhanced multimedia processing in CUbRIK; Analyzing the capacities of SMILA related to CUbRIK objectives Identifying gaps between SMILA and CUbRIK. Designing the architectural extensions needed to bridge the identified gaps.

1 World Wide Web Consortium (W3C). SOAP Message Transmission Optimization Mechanism http://www.w3.org/TR/soap12-mtom/ 2 World Wide Web Consortium (W3C). SOAP Messages with Attachments. http://www.w3.org/TR/SOAP-attachment 3 http://www.eclipse.org/smila/ CUBRIK Delivery Management Plan and Testing Specification

Page 2

D92 Version 1.0


1.1

CUbRIK architecture

The original concept of the CUbRIK architecture is specified in the Project DoW, as shown in Figure 1.

Figure 1: Essential aspects of the CUbRIK architecture (Project DoW) This schema highlights that the CUbRIK architecture relies on a framework for executing processes (aka pipelines), consisting of collections of tasks to be executed in a distributed fashion. Each pipeline is described by a workflow of tasks, allocated to executors. Task executors can be software components (e.g., data analysis algorithms, metadata indexing tools, search engines of different nature, result presentation modules, etc.). Tasks can also be allocated to individual human users (e.g., via a gaming interfaces) or to an entire community (e.g., by a crowdsourcing component). Different pipelines are defined for the different processes of a multimedia search application: content analysis and metadata extraction, query processing, and relevance feedback processing. Pipeline descriptions are stored in a process repository, automatic part of the pipelines is encoded using BPEL standard workflow language or as SMILA pipeline, with data exchanges across services supported by a suitable data model in order to cope with the data intensive nature of multimedia content processing and search. The CUbRIK architecture is divided into four main layers or tiers: 1.

External Interfaces

2.

Processes

3.

Components and executors

4.

Platform services.

Layers group the framework artefacts, essentially composed by Interfaces, CUbRIK Apps , Pipelines –including automatic and human tasks-, components –of different kind- and core services. In reference to the SMILA exploitation for the underlying framework supporting workflow definition and execution, it is essentially corresponding to the Platform Services

CUBRIK Delivery Management Plan and Testing Specification

Page 3

D92 Version 1.0


layer. Starting from some existent services –like execution engine, task management, persistency & cache support- SMILA is extended in the curse of the project with the goal of having a full functional coverage of CUbRIK Platform Services layer.

1.2

What is a CUbRIK Pipeline?

Unix-like computer operating systems implements a simple pipeline mechanism for interlocking programs executions; specifically, software pipeline is composed by a set of processes chained by their standard streams, so that the output of each process feeds directly as input to the next one in the chain. Another example of software pipe is the one implemented by Yahoo! Pipe4; these Pipes implement a composition tool to aggregate, manipulate, and mashup content from around the web in order to allow the User the selfcreation of tailored and personalized Yahoo! homepage. Conceptually speaking, CUbRIK Pipeline reflects a similar arrangement; Pipelines are a mixing of automatic operations –JOBs- and human activities that are chained in a sequence. This concept of Pipeline reflects the CUbRIK approach to have the Humans in the loop of generic Search processes. As mentioned, in CUbRIK architecture, SMILA5 was adopted as underlying framework for supporting workflow definition and execution; So each CUbRIK JOB is implemented, in practice, as a SMILA Workflow. The latter is a system for processing synchronous requests (e.g. search requests) by orchestrating easy-to-implement components (Worker or Pipelets) in workflows defined in BPEL(Pipeline).

Figure 2 : CUbRIK Pipeline Structure In this way, CUbRIK Pipeline results in a powerful mechanism to aggregate: •

Job - automatic workflow o

SMILA Workflow

Human activities o

Crowd-Search CUbRIK Applications

o

Game With A Purpose (GWAP) CUbRIK Application.

More in detail, CUbRIK JOB is defined as a SMILA workflows that is further constitute by Actions aggregation. More, depending from the complexity and specific characteristics of 4 http://pipes.yahoo.com/pipes/ 5 http://www.eclipse.org/smila/ CUBRIK Delivery Management Plan and Testing Specification

Page 4

D92 Version 1.0


activity to be performed, the Action can be formalize as a Worker - Single processing component in an asynchronous workflow-,a Pipelet - a reusable component in a BPEL workflow used to process data contained in records – or a Pipeline - synchronous BPEL process (or workflow) that orchestrates pipelets and other BPEL services (e.g. web services). The Figure 2 depicts the overall CUbRIK Pipeline structure, emphasizing Human activities and Automatic activities executed over SMILA.

1.2.1

CUbRIK Pipeline Synchronization

Having a look to the synchronization aspect CUbRIK Pipeline is conceived as a macroprocess including micro-processes of diverse nature articulated in different combination. These micro-processes, represented by the two typologies above listed, are arranged according to specific synchronization; synchronous automatic processes –JOBs- are combined with CrowdSearch and GWAP Application, performed asynchronously. The image below depicts an example of CUbRIK Pipeline:

Figure 3: Example of CUbRIK Pipeline graphic representation •

JOB (Automatic Workflow) represents a workflow fragment that is executed by process execution engine; JOBs embed and exploit the processing components for the actual activities carrying on.

Crowd Search Application are the first kind of application implementing the mechanism of Human in the loop. This Application is targeted to enabling individual and social participation to search processes; the actual Crowd mechanism leverages on a Crowd Search Framework implementing distributed work solutions for multimedia search. The Framework is in charge of design, execution and verification of tasks by a crowd of performers; In particular it manages core aspect including but not limited to Human task design, People to task matching, Task assignment, Task execution, Executor evaluation and Output aggregation.

CUBRIK Delivery Management Plan and Testing Specification

Page 5

D92 Version 1.0


1.2.2

GWAP Application is the other kind of application implementing the Human in the loop mechanism. As for CrowdSearch Application even for GWAP a specific GWAP Framework is exploited. By leveraging on playing games, the Framework, actually outsources certain processes steps to humans, in an entertaining way. Typical steps include labelling images to improve web searching, transcription of ancient text and other activity requiring common sense or human experience. Basic mechanism is the training of the system to solve problems mainly related to media understanding and contents interpretation.

CUbRIK Pipeline Data

So CUbRIK Pipeline is composed by different JOBs and Human activities that are orchestrated all together on respect of their synchronous or asynchronous nature. CUbRIK infrastructure synchronizes the flow of control. In doing that CUbRIK Platform accomplish the complementary function of data managing. In fact availability of data exchange among JOBs and Human Activities is essential for thorough Pipeline execution. The Pipelines mechanism implements the concept of data sharing among operations and human activities belonging CUbRIK Search processes. Beside this, Pipeline Data constitutes the basis of the triggering mechanism to “put in pipe” JOBs and Human Activities. In fact referring back to the analogy with Unix-like pipeline, it can be thought that one program runs after another; Each program is triggered by underlying infrastructure when the previous program is concluded and its output is make available. The output itself is feed as input of the next one. CUbRIK synchronization process, in a similar way, make available the data produced by each JOB or Human activity to the next JOB or Human activity. The data are make persistent in agnostic way.

1.3

What is a CUbRIK Platform release?

As introduced in Section 1.1 CUbRIK architecture , the framework is not a monolithic piece of software. It is structured in four layers: 1.

External Interfaces;

2.

Processes;

3.

Components & executors

4.

Platform services. The Platform service layer is mainly the framework that leverages on SMILA.

Layers group the CUbRIK framework artefacts, that are: Interfaces, CUbRIK Apps , Pipelines – including automatic and human tasks-, components –of different kind- and core services. According to this structure each CUbRIK release will be composed by group of artefacts corresponding to the advancement achieved at the specific time, provided in packaged way. Moreover the artefacts will be delivered in join with an accompanying document.

1.3.1

Release accompanying Document

For it nature of living document the release plan requires to be checked and validated during the whole development life-cycle. In this perspective, each CUbRIK release is issued with an accompanying document - Release checkout - with the goal to check and validate the implementation status of the release itself, and to update and add further details for the next development phase. Release checkout is scheduled to be released in join with

CUBRIK Delivery Management Plan and Testing Specification

the correspondent release

Page 6

D92 Version 1.0


delivery. In particular, Release checkout(s) are provided to: •

Report and evaluate the situation of the Development Process;

Identify deviations from the Release Plan;

Plan and carry out necessary corrective actions;

Refine and improve the Release Plan by refining its milestones.

User guideline Release Checkout documents will also include a specific section – the User Guideline – that will constitute a sort of handbook for release installation configuration and deployment. In particular the section will provide an overview on: •

SVN infrastructure hosting the package;

Technological Infrastructure needed for Project implementation;

Project Architecture and Modules;

Installation and Configuration guidelines;

Functionalities implemented by the released version.

Inter components dependencies A proper technical and legal management of components dependencies is a crucial step of the overall integration and testing process. Under this point of view, a summary of components dependencies will be part of the Release checkout document. Additionally the information related to components and pipelines licences will be reported in order to allow the management of legal aspect of demonstrator.

1.3.2

Artefacts

Differential approach is adopted for design and development process. Requirements elicitation, analysis, gaps identification and architectural extension design are inspired to actual search driven solution developed over CUbRIK Platform. Three main part will bring into releases different kind of artefacts: Vertical Demos, Horizontal demos and Platform Service.

1.3.3

Vertical demos – the CUbRIK Applications

In the DoW, the realization of two CUbRIK applications is planned in accordance to vertical domains identified: for History of Europe and SME Innovation. The CUbRIK applications are structured to be developed in three steps: Proof of concept –developed on initial requirements- released at M14; First Prototype –will constitute the first version of the application- released at M 24; Complete application –fully implementing CUbRIK featureswill be released at M32. According to the specific time-schedule these application will provide components, CUbRIK pipelines and CUbRIK Apps to be exploited in the specific vertical domains. A part what already described in DoW, specific functionalities and requirements for the two demos are going to be specified in the next period and will be included in accompanying documents.

1.3.4

Horizontal demos

In addition to these application, CUbRIK is developing some demonstrators aimed at proofing and demonstrating some general purpose functionalities, these are referred to horizontal domains identified as potential demonstration field. These application are referred as

CUBRIK Delivery Management Plan and Testing Specification

Page 7

D92 Version 1.0


horizontal demonstrator , H-Demos. Seven H-Demos were defined according to the analysis of domain of practice : 1.

Logo detection

2.

News history

3.

People Identification

4.

LikeLines: Time-point specific search via implicit-user derived information

5.

Media Entity Annotation

6.

Crosswords

7.

Accessibility aware Relevance feedback

As for the previously mentioned CUbRIK Application, the H-Demos will provide components, CUbRIK pipelines and CUbRIK Apps; in this case the reference horizontal domain will constitute the natural environment for the exploitation of these artefacts. Time schedule of HDemos -in correspondence of the five released of CUbRIK platform, R1/2/3/4/5- is depicted in the following table; the rows of the table put the releases in relation with the kind of pipeline that the release is part of:

Figure 4 : H-Demos Pipelines and time schedule H-Demos are described in detail and analyzed in specification deliverable D2.2. In the following a short description of the h-Demos is provided:

1.3.4.1.

Logo detection

Problem Solved: o

Brand Logo detection in collections of video from an input textual keyword.

Description of technical contribution: o

Detection of trademark logos in videos, based on an open source implementation of SIFT visual features.

o

Experiment with engagement of a crowd for human computation tasks, with the goal of improving the quality of the result.

o

Definition of content processing human computation tasks at different levels of difficulty

o

Definition of human computation tasks for relevance feedback

o

Preliminary evaluation of the crowd response using an open social network

CUBRIK Delivery Management Plan and Testing Specification

Page 8

D92 Version 1.0


(Facebook) Description of application context: The logo detection demo could fit in application contexts like: o

Marketing firms assessing the relevance of an advertisement campaign (e.g., sport sponsorship)

o

A public authority wishing to track hidden advertisement in video

o

The schema of the demo could be generalized to many other application contexts, if the logo image matching component is replaced by another domainspecific object matching component or trained on a different object recognition task

Success criteria: The success criteria comprise: o

At least 50 responses per tasks collected from the crowd

o

Quantitative demonstration of improvement of precision and/or recall thanks to the crowd contribution

o

Qualitative elicitation of usability problems in interacting with the crowd on open social networks

o

Qualitative elicitation of technical problems in deploying asynchronous crowd tasks on top of SMILA

Description of data set: The provided dataset is Grozi-120 (http://grozi.calit2.net/grozi.html). Grozi-120 is a multimedia database of 120 grocery products. The database contains both a collection of images (representing the products as isolated objects in ideal imaging conditions) and a collection of 29 videos (taken in a shop). The matching phase will be performed against this video collection. The dataset comprises a ground truth, i.e., each video is provided with annotations about the possible occurrences of logos in each frame. Functional Requirements Automatic functionalities (R1:M12) o

Brand names must be mapped to brand logo images

o

Video files must be segmented

o

Video segments must be summarized by the most relevant fragment in the video (key frame)

o

Visual descriptors must be extracted from logo images and key frames

o

Visual descriptors may be indexed

o

The logo images must be matched against the key frames of a video segment

o

A much must be associated with the confidence value of the match

o

Low confidence matches must be detected based upon a threshold

o

The threshold must be configurable

o

The matches must be presented in a GUI, every match is a triple: video, key frame, confidence

o

Matches must be sorted in descending order of confidence

CUBRIK Delivery Management Plan and Testing Specification

Page 9

D92 Version 1.0


o

Crowd votes for correct and incorrect matches may be exploited for suggesting an update of the matching threshold value

Crowdsourcing functionalities (R1:M12) o

The crowd must validate a logo image with respect to its relevance to the brand name

o

The crowd must retrieve a new logo image relevant to the brand name

o

The crowd must validate (yes/no) if a match between a logo image and a key frame of a video is correct

Non functional requirements R1:M12 o

The GUI must permit the access to a video in the time instant where a given match selected by the user appears

o

The system will produce the query results of a query for a brand in less than 1 seconds

o

The system will produce the query results of a query for a new brand in less than 60 seconds after the time a set of candidate logo images are available for matching.

o

The system will segment and index a new video of up to 60 seconds in less than 60 seconds

Requirements to Features Mapping

REQUIREMENT

FEATURE

Brand names must be mapped to brand logo images

Keyword search: Keyword-based image search

Video files must be segmented

Segmentation and summarization: Temporal Segmentation: Shot & scene detection

Video segments must be summarized by the most relevant fragment in the video (key frame)

Segmentation and summarization: Key frame extraction

Local descriptors must be extracted from logo images and key frames

Low-level detectors

features:

Local

feature

Low-level features: descriptors

Local

feature

The logo images must be matched against the key frames of a video segment

High-level features: Object detection

A match must be associated with the confidence value of the match

High-level features: Object detection

Low confidence matches must be detected based upon a threshold

High-level features: Object detection

The threshold must be configurable

High-level features: Object detection

The matches must be presented in a GUI, every

Special Presentation: Time segment

CUBRIK Delivery Management Plan and Testing Specification

Page 10

D92 Version 1.0


REQUIREMENT match is a triple: video, key frame, confidence

FEATURE access

Matches must be sorted in descending order of confidence

High-level features: Object detection

The crowd must validate a logo image with respect to its relevance to the brand name

Conflict resolution via CrowdSearcher: Image selection task

The crowd must retrieve a new logo image relevant to the brand name

Conflict resolution via CrowdSearcher: Image discovery

The crowd must validate (yes/no) if a match between a logo image and a key frame of a video is correct

Conflict resolution via CrowdSearcher: Binary relevance of matching rating

The GUI must permit the access to a video in the time instant where a given match selected by the user appears

Special presentation: Time segment access

1.3.4.2.

News History

Problem Solved: o

Identification of common sequences among a collection of news clips of a specific topic based on key word and or content-based video query (detection of how content has been used / edited by various actors)

Description of technical contribution: o

Granular temporal video matching based on spatial and temporal video fingerprinting technology.

o

Experimental inclusion of human knowledge/computation for validation, annotation and content provision tasks, with the goal of improving the quality of the result.

o

Prototype of a visualization / interaction interface for result presentation and user feedback

Description of application context: The news history demo could fit in application contexts like: o

Broad news topic summary creation of different sources for end users

o

Historical and political research for interested citizens

o

Support for social / political / media research on different methods of news editing/coverage/manipulation

Success criteria: The success criteria comprise: o

Minor success criterion: Find all related news clips (true positives) with minimal wrong (false positive) or missing (false negative) matches.

o

Major success criterion: Meet minor success criterion and establish the correct temporal sequence relation among the clips

Description of data set: A set of video clips, associated metadata and feature fingerprints (extracted from the clips)

CUBRIK Delivery Management Plan and Testing Specification

Page 11

D92 Version 1.0


are present within the system. A clip may have one or several associated clips, i.e. clips which are complete or partial near duplicates. Dependencies between content segments are established using fingerprint-based robust video identification. The temporal order and provenance of the clips is determined and validated using extracted metadata, contextual information and user annotations Moreover, approaches for detecting editing / coding footprints may be applied for this purpose. The data set to be used for demo purposes consists of a collection of news clips from at least 7 different news topics where each topic is at lease covered by 10 clips from different sources. The data set might be updated/refined by H-Demo administrators during the development phase. Important issue: Benchmarking the system performance requires ground truth data. First experiments showed that annotating overlaps of new clips is a hard task even for humans. Therefore, time efforts are quite high and it might be difficult to handle within the project. One solution might be to use clips from a different domain, such as movie trailers. Functional Requirements Automatic functionalities (R3:M24) o

The system must provide a query interface

o

The system must provide file upload and download functionality

o

The system must be able to (temporally) store content

o

The system must use a data base to store fingerprints, metadata and relationship, i.e. the data base contains the model

o

The system must detect perceptual temporal segment duplicate •

The system may employ temporal pre-segmentation of video footage

The system must extract spatial video descriptors

The system must extract temporal video descriptors

The system must create a fingerprint perceptual descriptors

and may use hashing algorithms on

o

The system may extract metadata from video/ image files

o

The system may generate topic proposals based on keywords

o

The system must be able to trigger external search engines

o

The system must be able to process results of external search engines

o

The system must generate, deploy and display result visualization

Crowdsourcing functionalities (R3:M24) o

The user must provide a query

o

The user/crowd may provide new content

o

The user/crowd may validate query results by confirming / establishing or rejecting relationships

o

The user/crowd may annotate query results by •

providing additional topic keyword

providing temporal relation

providing creation date of the news clip

CUBRIK Delivery Management Plan and Testing Specification

Page 12

D92 Version 1.0


Non functional requirements R3:M24 o

The system must be able to process new content items and do the respective model update within an acceptable time .

o

The system must be able to process user queries in an acceptable time(.

Requirements to Features Mapping

REQUIREMENT

SYSTEM FEATURE

The system must provide a query interface

Keyword-based search - videos by keyword Content-based search – videos by videos

The system must provide file upload and download functionality

Content acquisition – Upload of media elements

The system must be able to (temporally) store content

not in list yet [storage/data base] – to be put into “content acquisition” section

The system may employ temporal presegmentation of video footage

Segmentation & summarization – Temporal Segmentation: Shot and scene detection

The system must extract spatial video descriptors

Identification - perceptual video identification / fingerprinting (spatial low level features)

The system must extract temporal video descriptors

Identification - perceptual video identification / fingerprinting (temporal low level features)

The system must create a fingerprints

Identification - perceptual video identification / fingerprinting (descriptor creation)

The system may use hashing algorithms on perceptual descriptors

Identification - Exact identification / integrity

The system may extract metadata from video/ image files

Content acquisition metadata

The system must be able to trigger external search engines

Keyword search - Keyword-based external image/video search

extract

generic

Content-based search - Image/Video-based video search Content acquisition - Crawling videos based on textual keywords The system must generate, deploy and display result visualization

Special Presentation – Result Graph presentation AND Timeline presentation

The user must provide a query

Keyword search - Keyword search, match in text content Keyword search - Keyword-based external image/video search Content-based search - Image/Video-based video search

CUBRIK Delivery Management Plan and Testing Specification

Page 13

D92 Version 1.0


1.3.4.3.

People Identification

Problem Solved: o

Identification of people (based on their faces) in a (personal) photo collection

Description of technical contribution: o

Detection and identification of people in a photo collection based on their: •

Faces (Predominately)

Possibly other contextual cues like time, social semantics and clothing (body patches)

o

Considering semantics of photos with multiple people

o

Relevance feedback mechanism for improved query-retrieval

Description of application context: The people identification demo could fit in application contexts like: o

o

Generally: •

Identification of people in a personal photo collection

Detection and identification of people in social gatherings

Examples of use-cases: •

Identification of people in historical news archives, e.g. association of depicted people in press photos that have accompanying text

Fashion-related pattern mining tasks, e.g. detecting human bodies for the task of analyzing their clothes

Success criteria: The detection and identification rate of people should be over 50% given a training set size of no more than 10%. For example, with 30 different people in a dataset of 800 face instances, the rate should be over 50% compared to chance level of about 3%. Relevance feedback should further improve the identification rate by at least 5%. Description of data set: A dataset in the form of a photo collection (where photos mainly depict people) is to be used to evaluate the pipeline. Generally, the pipeline targets photo collections with few individual people that frequently appear (together). Thus, the dataset should preferably be from a single source (a personal photo collection). The dataset should comprise a ground truth: each photo should be provided with annotations (face markings and labels) with respect to the depicted people. Functional Requirements Automatic functionalities (R2:M18) o

Instances of depicted people (or their faces) must be mapped to distinct individual labels

o

Faces must be detected

o

Face features must be extracted and face descriptors must be generated

o

Given a face descriptor, a nearest/best match among other face descriptors must be found

CUBRIK Delivery Management Plan and Testing Specification

Page 14

D92 Version 1.0


o

Exclusivity (no same people in a single photo) must be considered

o

Other social semantics may be considered

o

Other contextual cues like time or clothing may be considered

o

Confidence values of matches must be computed and exposed

o

The matches must be presented in a GUI

o

Matches must be sorted in descending order of confidence

Relevance feedback (R2:M18) When user selects/clicks on a person in the gallery overview, initial results are presented. Repeating this step refines the presented results by incorporating a relevance feedback-like mechanism. Non functional requirements R2:M18 o

The GUI must permit the access to a photo in the time instant where a given match selected by the user appears

o

The system will produce the query results of a query for a person in less than 10 seconds

Requirements to Features Mapping

REQUIREMENT

FEATURE

Distinct individual people must be mapped to instances (appearances) of people (and their faces)

Clustering/Inference descriptors

People must be detected in photos

Face detection

Comparing faces

Distance-based face discrimination

Considering constraint)

exclusivity

(unique

people

based

on

face

Graph-based clustering/inference

The matches must be presented in a GUI Matches must be sorted in descending order of confidence

1.3.4.4.

Ranking: By match confidence value

Like Lines

Problem Solved: o

Identification of the most interesting/relevant fragments in a video

Description of technical contribution: o

Open tool for collecting large amounts of timecode-lever user-feedback data needed for research in the young area of timecode-level access of video

o

Identification of the most interesting/relevant fragments in a video based on a fusion of: •

Natural implicit user interactions with the video (e.g., play, pause, rewind)

CUBRIK Delivery Management Plan and Testing Specification

Page 15

D92 Version 1.0


Explicit user interactions with the video (i.e., explicitly liking particular time points)

Multimedia content analysis of the video

o

Use of user interactions as feedback to the multimedia content analysis process

o

Use of multimedia content analysis as feedback to collected user interactions

Description of application context: The LikeLines demo could fit in application contexts like: o

General online video websites, providing fragment-level access to videos

o

Providing analytics to content creators, e.g., explicit liking particular time points could prove useful for fashion show videos

o

Summarization of (personal) video collections

Success criteria: The success criteria comprise: o

Main component implemented and publicly available on e.g. GitHub by the end of July, 2012

o

Feedback collected for at least one (to be determined) video dataset by March 2013: •

At least 30 user interaction sessions per video recorded for at least 30 videos in the dataset (at least 900 interaction sessions in total)

Formulated at least 10 hypotheses describing semantics of implicit user interactions based on the collected feedback

Description of data set: A dataset in the form of a video collection. Further details like video type are to be determined. Ground truth: TBD (current suggestion: “cross-validation” of collected user interactions)

Functional Requirements Automatic functionalities (R2:M18) o

The system must support at least one form of multimedia content analysis

o

The system must capture and store implicit and explicit user interactions

o

The system must be able to aggregate user interactions

o

The system must be able to fuse the output of multimedia content analysis and user interactions

o

The output of multimedia content analysis may be used to value the quality of user interactions

User interactions may be used to gauge the performance of multimedia content analysis. Non functional requirements o

The system must be open-source

o

The system must be able to operate on YouTube videos

o

The system may be able to operate on any video supported through HTML5’s video capabilities

CUBRIK Delivery Management Plan and Testing Specification

Page 16

D92 Version 1.0


Requirements to Features Mapping

REQUIREMENT

FEATURE

User interactions must be captured and stored

Temporal data storage, Data base

The system must perform MCA

Key frame extraction; Shot detection; Face detection; Speaker’s turn

1.3.4.5.

Media Entity Annotation

Problem Solved: o

Harvesting representative images of named entities.

Description of technical contribution: Automatic: o

Automatic harvesting of images from the web (e.g., Picasa, Flickr, Panoramio, etc.) for a set of named entities.

o

Automatic ranking of images using social features (like number of likes/dislikes/favourites, comment, etc.) to filter highly relevant and interesting images.

o

Automatic selection of a small subset of representative images that depict diverse aspects of the entities.

o

Automatic classification of harvested images into low, medium, and high confidence levels.

o

Storing and retrieval of entities, relations between entities, entity attributes and multimedia in the specialised entity repository.

Humans in the loop: o

Improving the quality of the result by applying crowdsourcing techniques, e.g. by verifying images with low confidence by humans.

o

Improving the quality of the results by applying GWAP techniques, e.g. automatic generation of crosswords from entity-games framework for images with medium confidence.

Description of application context: The entity media annotation demo could fit in search application contexts like: o

entity→media: Visualising named entities in the entity search results.

o

media→enƟty: Providing additional information about entities as a result of content based media search. For instance, person takes a photo of a monument. Content based search is used to find images of the monument in the entity repository. System provides additional information about monument and its relation with other entities.

Success criteria: The success criteria comprise: o

Quantitative demonstration of improvement of resulting data set quality after applying crowdsourcing and human computation techniques.

o

Quantitative demonstration of improvement of precision and/or recall of entity

CUBRIK Delivery Management Plan and Testing Specification

Page 17

D92 Version 1.0


search algorithms. Description of data set: Input: A set of famous Italian monuments with metadata information was expert-generated. In total, experts collected a set of 100 monuments located in different Italian cities such as Rome, Florence, and Milan. Entities related to the cities or monuments were also collected. Output: Using different attribute values from the 100 monuments, a data set of about 30.000 images (100 monuments * 100 images * from 3 sources) is to be generated. Images will be queried from Panoramio, Picasa and Flickr.

Functional Requirements Automatic functionalities (R1:M12) o

Entities must be associated with unambiguously identifying metadata.

o

Images must be associated with unambiguously identifying metadata.

o

Images and entities must be automatically related by using their metadata. 100 high quality images must be automatically crawled from each of the image search engines (e.g., Picasa, Flickr and Panoramio) by using (GPS) locations and other metadata. Images must be further ranked and filtered using social features to obtain relevant and socially attractive images.

o

Small number (5) of representative images must be automatically selected.

o

Entities, images, their metadata and relationship must be stored in entity repository.

Automatic functionalities (R3:M24) o

Each piece of metadata must be associated with confidence level.

o

Crosswords must be automatically generated for images and metadata with medium level of confidence.

o

Images must be indexed to allow content based entity search

o

Images must be indexed also based on the time information (time of creation, time of index) to support also time queries directly to the multimedia indexer.

o

Entities must be indexed to allow metadata based entity search. Metadata based search may be one of the following: •

keyword based syntactic search,

•

concept based semantic search,

•

entity search which is based on relationship between entities.

o

Search results must be ordered according to their relevance and also confidence. A learning to rank framework must be constructed to obtain the final set of images using different types of image features (i.e., content-based, social, etc).

o

Entities in the search results may be described by their metadata, related media, or both.

o

The GUI must have compact and extended representation for entities.

Crowdsourcing functionalities (R3:M24) o

For low confidence images, the crowd must validate an image with respect to its relevance to the entity. Validity of non-expert annotations must be evaluated.

CUBRIK Delivery Management Plan and Testing Specification

Page 18

D92 Version 1.0


o

The crowd may suggest a new image for an entity.

o

For medium-high confidence images, the crowd must play crossword games to validate metadata and media associated with entities.

o

The crowd may suggest errors in images used for crossword.

o

Images of monuments must focus on monuments and not on other entries, e.g. image of people in front of monument must be filtered out. Any images that doesn’t depict clearly and distinctively the monument must be discarded.

o

Crowdsourcing tasks may be allocated depending on users’ characteristics such as profile and previous performance.

o

Crosswords might use incentives in order to increase variables such as users’ performance and level of engagement.

Non functional requirements R1:M12 o

The entity repository must be able to store at least 1M entities on a single server.

R3:M24 o

The system must produce the results for entity search in less than 1 second.

Requirements to Features Mapping REQUIREMENT

FEATURE

Entities must be associated with unambiguously identifying metadata

Geo entities extraction from structured resources

Images must be associated with unambiguously identifying metadata.

Crawling of images based on textual keywords Crawling of images based on geoposition or area

Images and entities must be automatically related by using their metadata. 100 high quality images must be automatically crawled from each of the image search engines (e.g., Picasa, Flickr and Panoramio) by using (GPS) locations and other metadata.

Crawling of images based on textual keywords

Images must be further ranked and filtered using social features to obtain relevant and socially attractive images.

Association of images to entities

Small number (5) of representative images must be automatically selected.

Association of images to entities

Entities, images, their metadata and relationship must be stored in entity repository.

Entity search in entity repository

Each piece of metadata must be associated with confidence level.

Association of images to entities

Crosswords must be automatically generated for images and metadata with medium level of

Automatic creation of entity games out of entity repository

CUBRIK Delivery Management Plan and Testing Specification

Crawling of images based on geoposition or area Association of images to entities

Page 19

D92 Version 1.0


REQUIREMENT confidence.

FEATURE

Images must be indexed to allow content based entity search.

Image/Video-based video search

Images must be indexed also based on the time information (time of creation, time of index) to support also time queries directly to the multimedia indexer

The crawlers fetch also time information for the retrieved images and push this information in the entity database and multimedia indexer. The time aware multimedia indexer enables time-related queries.

Keyword based syntactic search

Keyword search, repository

Concept based semantic search

Synonym, hyponym, expansion

Entity search which is based on relationship between entities.

Entity search in entity repository

Search results must be ordered according to their relevance and also confidence.

Entity search in entity repository

A learning to rank framework must be constructed to obtain the final set of images using different types of image features (i.e., content-based, social, etc).

Keyword search in entity repository

Entities in the search results may be described by their metadata, related media, or both.

Entity search in entity repository

For low confidence images, the crowd must validate an image with respect to its relevance to the entity. Validity of non-expert annotations must be evaluated.

Mapping images to entities

The crowd may suggest a new image for an entity.

Mapping images to entities

For medium-high confidence images, the crowd must play crossword games to validate metadata and media associated with entities.

Automatic creation of entity games out of entity repository

match and

in

entity

hypernym

Player profile management Automatic feedback cross-validation The crowd may suggest errors in images used for crossword.

Automatic feedback cross-validation

Crowdsourcing tasks may be allocated depending on users’ characteristics such as profile and previous performance.

Player profile management

Crosswords might use incentives in order to increase variables such as users’ performance and level of engagement.

Player profile management

CUBRIK Delivery Management Plan and Testing Specification

Page 20

D92 Version 1.0


1.3.4.6.

Crosswords

Problem Solved: o

Collecting user feedback for improving quality of entity metadata.

Description of technical contribution: o

Creating a game with purpose which implements collecting feedback, relevance and improvements for metadata of entities.

o

Experiment with engagement of a crowd for human computation tasks, with the goal of improving the quality of the result.

o

Design of reusable Game Framework enabling to easily create new word games with purpose.

Description of application context: The crosswords demo could fit in application contexts like: o

Assessing correctness of metadata of acquired entities;

o

Improving metadata of acquired entities;

o

The schema of the demo could be generalized to many other application contexts, for example, to educational context or human testing and evaluation context.

Success criteria: The success criteria comprise: o

Game Framework implemented and available to game developers, which can access the Game Framework API, its documentation and build new games on top of it by the end of the project;

o

At least 1 game implemented on top the framework by the end of the project;

o

Feedback collected for the collection of entities: correction of mistypes in textual metadata (such as name misspellings), correction of relational attributes in metadata (such as an association between an entity and its image; an association between two entities with a relation like “bornIn�);

o

Quantitative demonstration of metadata improvement thanks to the crowd contribution; The quantitative metadata improvement is measured against a random subset of entities selected out of monuments data set and manually checked. If the error rate in the original data set is too low or the errors do not vary widely enough, manual distortions may be introduced for testing and demonstration purposes.

Description of data set: The provided dataset is Entitypedia (http://entitypedia.org), out of which we focus on a collection of Italian monuments. Functional Requirements Automatic functionalities (R3:M24) o

Crosswords must be generated out of entity repository, using entity metadata;

o

Crosswords must be implemented on top of a Game Framework;

o

Feedback evaluation may be automatic;

o

Users expertise may be collected through Game Framework during the game;

Crowdsourcing functionalities (R3:M24) CUBRIK Delivery Management Plan and Testing Specification

Page 21

D92 Version 1.0


o

Users must be able to submit metadata error corrections as feedback;

o

Crosswords must be playable, that is, the task must be crowdsourced and gamified;

o

Users must be able to register or play as guests;

o

Crosswords must have good incentives embedded such as bonus and achievement systems;

Automatic functionalities (R5:M36) o

Crossword generation may be customizable and flexible;

o

Relevance evaluation may be automatic;

o

Users expertise and crossword topic may be automatically suggested;

Crowdsourcing functionalities (R5:M36) o

Users may be able to have a socially integrated experience (e.g. Facebook integration);

Non functional requirements R3 (M24) o

The game should be responsive, e.g. user input should be processed in less than a second;

o

The game should have appealing user interface;

Requirements to Features Mapping

REQUIREMENT

FEATURE

Crosswords must be generated out of entity repository, using entity metadata

Automatic creation of entity games out of entity repository

Users must be able to submit metadata error corrections as feedback

Association of images to entities

Crosswords must be playable, that is, the task must be crowdsourced and gamified

Automatic creation of entity games out of entity repository

Crosswords must be implemented on top of a Game Framework

Player achievement management User signs up explicitly to a GWAP

Feedback evaluation may be automatic

Automatic feedback cross-validation

Users expertise may be collected through Game Framework during the game

Player achievement management

Users must be able to register or play as guests

Player profile management

Crosswords must have good incentives embedded such as bonus and achievement systems

Player achievement management

Crossword generation may be customizable and flexible

Automatic creation of entity games out of entity repository

CUBRIK Delivery Management Plan and Testing Specification

Page 22

D92 Version 1.0


REQUIREMENT

FEATURE

Relevance evaluation may be automatic

Automatic feedback cross-validation

Users expertise and crossword topic may be automatically suggested

Player profile management

Users may be able to have a socially integrated experience (e.g. Facebook integration)

User signs in to a GWAP with a SN account

1.3.4.7.

Accessibility aware relevance Feedback

Problem Solved: The Accessibility aware Relevance feedback H-Demo aims to enhance the process of multimedia content harvesting in such a way that it will be later able to provide an accessibility related indicator (i.e. confidence factor), regarding the level of accessibility and usability it offers to specific groups of users. Initially, the “Accessibility aware Relevance feedback” demo will be exhibited as a standalone horizontal demo, that could handle various types of multimedia data containing webpages, while later on will be incorporated to the “History of Europe” CUbRIK App. Description of technical contribution o

Data evaluation in terms of accessibility and usability of the multimedia content of Web pages (i.e. images, sound, text, header tags and metadata), data, so as to provide accessibility relevance factors by executing extended multimedia analysis (see list of features below).

o

An initial evaluation of the accessibility level of web sites will be based on the usage of existing accessibility tools (mostly open source tools), such as Tools/APIs for creating accessible software and standardization (e.g. JAVA Accessibility API, GNOME, IBM’s IAccessible2, etc.) and also available accessibility standards and methodologies (e.g. WAI- ARIA, ISO, UK Disability Discrimination Act DDA, etc.)

o

Generation of user-specific profiles from information derived during the user’s registration phase and the follow-up updates of these profiles according to the future behaviour of each user.

o

The accessibility evaluation of each web page will be subject/user-specific and it will address the needs and the disabilities of the profile logged-in user (i.e. specific disabilities will be assigned to certain attributes of the multimedia objects of the web page). For this purpose, a list containing actual disabilities will be constructed that will be taken into account both visual, hearing and possibly even motor impairments. Similarly, all possible web-page data attributes, should be studied in depth and collected: Multimedia content characteristics: for (a) text (i.e. Visual information, hard-coded fonts, text background, multi-column formats, styles and stylesheets, etc.), (b) audio (i.e. sound quality, volume control, captions and/or transcripts, etc.), (c) images (i.e. alternate text, image resolution, zoom feature, etc.) and (d) other multimedia content (i.e. captions, transcripts or audio descriptions in videos, audio description, timing of media delivery, etc.).

o

In order to adapt the accessibility distance measure to match the users expectations (disabilities), the users will be given the opportunity to provide relevance feedback information regarding the accessibility level of the web page, via direct interaction with the query results. The relevance feedback they provide will refer either to the whole web-page itself or to specific multimedia

CUBRIK Delivery Management Plan and Testing Specification

Page 23

D92 Version 1.0


objects contained in the web-page. The relevance feedback refering to single multimedia objects contained in the web page will be processed autonomously and will partially (e.g. via weighting factors) contribute to the final accessibility confidence factor of the full web page. o

Advanced crowdsourcing techniques will be utilized for the evaluation of these attributes of multimedia data or multimedia metadata, for which no automatic evaluation can be performed – or is unacceptably expensive in terms of processing resources (e.g. useful navigation options, meaningful link-text, meaningful tagging of tables, etc.).

Functional Requirements R3-M24 o

Input: web pages returned by the search results related to any possible topic, additional to time- space information and multimedia content, as well as the current user’s profile.

o

Output: confidence factor, which will stand as an indicator for the level that the, accessibility level of the content and the usability level of the web-page is influenced for the specific user.

o

Data storage: database, where the •

accessibility related user profile,

the extracted features (see feature list above),

potentially the multimedia data of the web page itself,

the search history,

the relevance feedback input and

the changes over time of the relevance feedback related weighting factors

will be stored. o

Processing: extraction descriptor vectors per web-page is a complex task that will involve image-, sound-, textual- and metadata analysis and incorporation of state-of-the art of relevance feedback techniques for multimedia data. For the simple case of examining if the “alt” attribute is present for every image in a web page, the analysis lasts only a few seconds when running on a core i5@2.8GHz processor lasts a few seconds. For experimental purposes, a set of multiple different descriptors (i.e. header existence, image- and audio analysis) extraction methods will be used and their combination will lead to the final assessment results. In general, the average system response time is expected to be in proportion with the complexity of the objects that are being tested.

o

Timing and synchronization: no timing issues are involved in the current process since each modality (i.e. text, image, audio, etc.) can be processed asynchronously on different processors, while the final result will be based on the fusion of the each of the latter.

R5-M36 o

Input: web pages returned by the search results related to the History of Europe, additional to time- space information and multimedia content, as well as the current user’s profile.

o

Output: accessibility assessment, via an overall accessibility & usability confidence factor for each separate multimedia element (related to the History of

CUBRIK Delivery Management Plan and Testing Specification

Page 24

D92 Version 1.0


Europe search content), in a ranked list based on the profile of the logged-in user. o

Computation: see R3 above. Additionally, transparent crowdsourcing techniques will be utilized. In particular, a pipeline process will be created, that will process the query results in terms of accessibility of the web page and that will support both relevance feedback and crowdsourcing techniques for enhancing the user(disability)-specific query results and the estimation of timeand processing-expensive cognitive multimedia attributes, respectively.

o

Data storage: similar to R3-M24.and crowdsourcing input.

o

Timing and synchronization: timing will be primary based on the requirements of the multimedia search pipeline.

Non functional requirements R3-M24 o

The system will require users to log in, so as to load their accessibility profile (disabilities, elderly and situated disabilities, etc.), before performing the accessibility analysis.

o

The system will adapt the accessibility scores to users corresponding to their profile levels and display the final accessibility relevance factor as final decision score.

o

The system will provide an administrator functionality for setting permissions

o

Where appropriate the module shall be able to generate automated help wizards and error messages in case of system malfunctions and/or users’ mistakes.

o

A help menu shall exist, that will facilitate the user understand the operation of the system, and guide him along the process.

o

Diagnostics messages in case of unsuccessful or uncertain operations.

o

All software tools developed within the project will be released under an open source license.

o

All modules’ components will be implemented in modular, open source system architecture.

o

The module will be based on a layered solution with a high level of encapsulation of tools to assure the maintainability of the infrastructure with future upgrades.

o

The query result page will be able to switch between two functional modes: •

a simple query result displaying mode and

an interactive mode, where the each registered user will be able to provide relevance feedback with respect to the accessibility level of the web-page.

R5-M36 o

All mentioned in R3-M24.

o

The system will require users to log in, so as to load their accessibility profile, before performing the accessibility analysis.

o

The system will adapt the accessibility scores to users corresponding to their

CUBRIK Delivery Management Plan and Testing Specification

Page 25

D92 Version 1.0


profile levels and display the final accessibility relevance factor as final decision score. o

Only registered users will have the permission to provide accessibility relevance feedback.

o

The crowdsourcing functionality will be open for registered users only.

o

The crowdsourcing input will be asked by pop-up windows that will appear when moving the mouse cursor on specific multimedia objects (i.e. tables, for evaluating their meaningful tagging, etc.).

Here it should be noted that the users’ profile will be synthesized from both users’ personal input/feedback (registration) but also from information derived from social media sites. As such, the crowdsourcing functionality will not only be limited to the straightforward input of the users regarding the accessibility of the multimedia content of the web site. Of course, in the final release (R5-M36), the functionality of the current pipeline will be fine tuned for the multimedia content of the History of Europe application.

Requirements to Features Mapping R3-M24 o

Accessibility related descriptors/features concerning the visual objects in a webpage (i.e. images and video streaming media): •

the colour histogram,

the colour layout,

areas with high luminance values,

the image resolution

shape descriptors of the image

the contrast of the image and

its texture,

so as to address for vision-related disabilities (e.g. colour blindness, etc.). o

o

Accessibility related descriptors/features concerning the audio objects in a webpage (i.e. sounds and audio streaming media): •

Frequency spectrum,

phase related features,

percentage of high-, low- frequencies,

duration,

mono/stereo/surround/etc.,

bit rate,

loudness per frequency band (dB),

DC values,

(P)SNR,

Accessibility related features concerning the text (objects) in a web-page: •

Font size,

CUBRIK Delivery Management Plan and Testing Specification

Page 26

D92 Version 1.0


o

Font colour,

Font contrast with respect to the background,

Text alignment,

Indentation/Spacing,

Accessibility related features concerning the metadata (for image/sound/text) in a web-page: •

Any content in audio/visual format should also be available as a text transcript for hearing impaired users

Existence of images’ “alt” attribute.

Existence of proper header tags (i.e. h1, h2, h3, etc.) that make site navigation easier for users using assistive technologies (e.g. screen readers).

Preservation of consistency in layout, color, and terminology for reducing the cognitive load placed on users.

Absence of “hard-coded” text size so as to defeat the use standard browser controls.

R5-M36 o

All features mentioned in R3-M24

o

Examination whether the content is read by screen readers in a logical order (“div-” tags).

o

Crowdsourcing input:

1.3.5

Verification of logical order reading.

Existence of an option to useful navigation options and meaningful link text (e.g. “skip”, “click here”, etc.)

Meaningful tagging of tables for data presentation proper tagging, as derived by crowdsourcing input .

Platform Service

As said, Platform service is in charge to implement services like execution engine, task management, persistency & cache support etc. Starting from some existent services, SMILA, will be extended with the goal of having a full functional coverage of CUbRIK Platform Services functionalities even according to the requirements that will be elicited in the curse of the project. At this stage is possible to identify some preliminary extensions so far risen as needed services, other will be specified in the rest of project time scale: V1.0 o

Self-scaling ETL - dynamic scaling of data import

o

Migration of crawler implementations to self-scaling ETL

V2.0

CUBRIK Delivery Management Plan and Testing Specification

Page 27

D92 Version 1.0


o

Solr 3.5 integration

o

Aperture integration

o

Load balancing helper

o

General configuration management

o

Concept for debugging of BPEL processes

o

Stabilization and initial implementation of scaling up performance mechanism

o

bug fixing

o

Final implementation of mechanisms for Platform performance scaling up – offering Platform service as service in the cloud.

V3.0

V4.0

V5.0

CUBRIK Delivery Management Plan and Testing Specification

Page 28

D92 Version 1.0


2.

Releases Scheduling and packaging strategy

2.1

Main Releases

In respect of the initial workplan schedule, identified in the DoW, the release scheduling was adjusted and extended, anticipating crowdsourced task and even Relevance Feedback and Time & Space functionalities. Five main development phases were defined for the CUbRIK project in the project timescale. The end of each phase corresponded to a CUbRIK project release which collects the progresses –in terms of artefacts- achieved at that time:

Delivery Date

Release version

Objective

M12

1.0

Initial release of the platform: to get a first version of a CUbRIK Platform having initial version of all available components and Hdemos pipelines in place and integrated in a common environment. Main goal is to validate developed technology and data model, and to the test workflow mechanisms at covering the thorough pipelines range .

M18

2.0

Advancement release: enhancements and Bug fixing, possible runtime environment extension. The enhancement will essentially be focused on Pipeline for Feedback acquisition and processing – the relevant feedback and in functionalities implemented in first prototype of “CUbRIK History of Europe Application” and “Search for SME Innovation Application”. Further identification of some feature extensions and improvements.

M24

3.0

Integrated release of platform: release of a second set of functionalities related to H-Demos. Moreover “CUbRIK History of Europe Application” and “Search for SME Innovation Application” first prototypes will extend the Platform with some additional features.

M30

4.0

Advancement release: enhancement and bug fixing in order to consolidate the platform and prepare it for the final version. Some additional features will be provided as part of H-Demo.

M36

5.0

Final release: to improve effectiveness & efficiency to push CUbRIK closer to real business scenarios Table 1: CUbRIK project initial planning

During the life cycle of the project it is not excluded that some deviation could be applied in order to have a progressive path towards a final effectively running project.

2.2

Releases

Figure 5 shows the Gantt diagram, while Table 2 shows the relationship between CUbRIK development phases and the overall release plan, and illustrates the main functional objects of each release. In particular:

CUBRIK Delivery Management Plan and Testing Specification

Page 29

D92 Version 1.0


Figure 5: GANTT of CUbRIK Releases

Delivery Release Date version

What is going to be delivered

M12

Pipelines for multimodal content analysis & Logo Detection H-Demo enrichment Media Entity Annotation H-Demo (space related)

1.0

In detail

Pipelines for query processing

Logo Detection H-Demo (crowd source content tagging and query expansion)

Pipelines for relevance feedback

Logo Detection H-Demo (crowd source for conflict resolution)

Component and pipeline support services

Components belonging Logo Detection and Media Entity Annotation

Space & Time extension

data-set for space domain including the methodology for cleaning the data-set

CUBRIK Delivery Management Plan and Testing Specification

Page 30

D92 Version 1.0


Delivery Release Date version

What is going to be delivered

M18

Social network analysis, trust & people ranking strategies for MM objects that make use of social features that are search techniques obtained from Web 2.0 platforms.

2.0

Pipelines for relevance feedback

In detail

People Identification H-Demo Like Lines H-Demo

Incentive models and algorithms

Incentive models applied to concrete crowd-sourcing scenarios. Entity game framework-crosswords scenario Component for relevance feedback- crosswords scenario

Pipelines for multimodal content analysis & enrichment Pipelines for query processing Component and pipeline support services M24

3.0

Time core services, components and Multimedia similarity indexer for fast multimedia search. pipelines for preliminary Support to Component for analysis -using various algorithms- of the data coming in the "History of Europe" application databases and those coming from the crawlers in order to find popular content and specific trending topics . Component for text query and a geolocation and crawls data from the web.

CUBRIK Delivery Management Plan and Testing Specification

Page 31

D92 Version 1.0


Delivery Release Date version

What is going to be delivered

In detail

Pipelines for multimodal content analysis & News History H-Demo enrichment Pipelines for query processing

News History H-Demo People Identification H-Demo Media Entity Annotation H-Demo (time related query extension)

Pipelines for relevance feedback

Crosswords H-Demo Media Entity Annotation H-Demo (space related extension) Accessibility Aware Relevance feedback

Component and pipeline support services

M30

4.0

Space core services, components and Entity-centric services: entity search, entity pipelines and entity exploration developed on top of the entity repository.

navigation,

entity-centric algorithms shaped as CUbRIK components. Pipelines for multimodal content analysis & enrichment Pipelines for query processing Pipelines for relevance feedback

CUBRIK Delivery Management Plan and Testing Specification

News History H-Demo

Page 32

D92 Version 1.0


Delivery Release Date version

What is going to be delivered

In detail

Component and pipeline support services Design tools

GWAP and techniques

tools for supporting the design and implementation of content and query processes – first version implicit

user

information conflict resolution appears of automatic metadata creation of CUbRIK framework will be resolved by the Game framework

Social network analysis, trust & people enriched set of strategies that make use of social data and human computation search techniques to improve the quality of ranking of MM objects. methods for identifying individuals with appropriate expertise to carry on a certain human-computation task. M36

5.0

Social network analysis, trust & people search techniques Time core services, components and Storage component that will handle incoming data pipelines Pipelines for relevance feedback Design tools

Tools supporting CUbRIK development environment supporting component registration, process design, components and process repository management. Table 2: CUbRIK development phases, releases plan and objectives

CUBRIK Delivery Management Plan and Testing Specification

Page 33

D92 Version 1.0


3.

Testing

The Testing Strategy is officially defined as part of this document. Its main structure was anticipated in a section of D9.1 in order to provide at the begin of the project a comprehensive big picture of Development and Delivery procedure. This section reports on the testing procedure to be performed in order to release CUbRIK platform. Testing is performed over the artefacts constituting the CUbRIK release(s); as introduced in 1.3 each release is essentially composed by Interfaces, CUbRIK Apps , Pipelines – including automatic and human tasks-, components –of different kind- and core services. So the different parts of the release will be tested according to a twofold approach defining: artefact Provider –the partner in charge of develop and release the artefact- and artefact Consumer –the one(s) in charge of use the artefact leveraging on it for the realization of a more complex artefact, including in the latter even the Pipeline. CrowdSearch and GWAP frameworks, in analogue way, are considered environments for CUbRIK artefacts – the CrowdSearch and GWAP application, that are further aggregated in CUbRIK pipelines. A typical scenario involves components providers that develop component that are further aggregated and orchestrated with Crowdsearch and GWAP Application in a Pipeline. In this case the provider realize artefact that are used by a Consumer for the implementation of a specific pipeline. Once the pipelines are assembled and tested these are delivered to the Integrator Partner – ENG- for the packaging, in this sense the Integrator partner is the Consumer of the thorough Pipeline. Integrator partner has also the role of consumer for the CUbRIK core service developed by partner in charge of the development of Platform services. So ,testing procedure has been organised in two main phases: Artefact testing: it has the goal to find the defects of each artefact of the CUbRIK platform – intended as single component, interface and application. It is articulated in Unit test and Consumer Provider tests (described afterwards). Packaging testing: it is essentially a Pipeline testing and has the goal to find defects when artefacts are integrated into a functional chain, the Pipeline. In general, each package includes more than one Pipeline, all of these are subjected to the testing procedure describe in the following. The testing procedure is made by two phases: the first one, named Functional test performed on the CUbRIK pipeline deployed locally by the Pipeline responsible partner. The second one, named Integration test, tests the actual “installability” level of the components developed by the CUbRIK partners; it is in charge to the Integrator partner. All the testing phases are supported by the Bugzilla infrastructure and in particular by the section by the section Bugs, releases and feedbacks tracking system, described in D9.1 INTEGRATION GUIDELINES document (Cap. 6.5.3).

3.1

Unit test

Unit testing tests an artefact before its official release, by means of functional tests performed to check the correct artefact behaviour, following its functional specifications. Unit testing is not standardized within the CUbRIK project, is applied to both single components, interface and application. Unit tests are carried on under producer’s responsibility.

3.2

Consumer-provider testing

Consumer-provider tests an artefact by the point of view of its direct clients - which exploit the CUBRIK Delivery Management Plan and Testing Specification

Page 34

D92 Version 1.0


artefact under testing - by using specifically written applications, named Consumer Provider (C/P) tests. Each C/P test invokes the artefact using its standard input and checks for the correctness of the output (returned values and status code). Consumer-provider testing is under the responsibility of the partner consumer of the artefact, on the basis of the dependency chart of his own module. For example: a partner develops and deploys Artefact X and exposes it as a web service. The wsdl file provides the all necessary information to test the services exposed. Consumer-provider test is performed when a new artefact is released or changes are applied.. It guarantees that artefact behaviour is compliant to the defined interface. Consumer-provider testing is defined by both artefact consumer and provider. Provider defines an initial proposal to be agreed by the Consumer. This kind of testing is not standardized within the CUbRIK project. When applicable dedicated Java clients can be developed for C/P test implementation using soapUI6, an open source tool for Web Service Testing. Additionally, the SMILA APIs could be tested using a JSON REST Client, which are available for most browsers.

3.3

Integration testing

Integration testing tests the integration of a specific kind of artefact, the Pipeline. Installation is checked in agreement with the installation requirements defined in the corresponding Release accompanying document. Integration test is performed as a final test before Pipeline packaging, after end-to-end functional test which checks the Pipeline adhere to functional requirements, to ensure that Pipeline, and all its components, are correct and complete, starting a “from the scratch� installation. Integration testing is under the responsibility of the partner in charge of the Integration.

3.4

End to end functional testing

End to End Functional testing verifies the functionality of the integrated Pipeline deployed on CUbRIK Platform. It has a dependency on the results of artefacts test that are belonging the Pipeline. CUbRIK pipelines, formalized in use cases, are exploited using the User Interface as test sessions. Functional testing is performed mainly by the Integrator partner , supported by all the other partners involved. Any testing session calls one of the entry points of the prototype and checks desired response. Pipeline testing can be in charge of pipeline owner and/or Platform integrator. Initial test is performed by pipeline owner for smooth testing. Then completed functional testing is performed by platform integrators supported by pipeline owner and by components owner for all the components involved. Note that the relationship between use cases and test sessions could not be one-to-one, because a complex use case could be made by several distinct ways to interact with the CUbRIK Platform, and each one must be subject to a specific testing session.

6 http://www.soapui.org/ CUBRIK Delivery Management Plan and Testing Specification

Page 35

D92 Version 1.0


4.

References

[BOZ12] A Framework for Crowdsourced Multimedia Processing and Querying

CUBRIK Delivery Management Plan and Testing Specification

Page 36

D92 Version 1.0


CUbRIK Delivery Mgnt Plan and Testing Specs  

Release delivery plan

Advertisement
Read more
Read more
Similar to
Popular now
Just for you