FIRST CUBRIK APPLICATION DEMONSTRATORS Human-enhanced time-aware multimedia search
CUBRIK Project IST-287704 Deliverable D10.1 WP10
Deliverable Version 1.0 - 30 September 2013 Document. ref.: cubrik.D101.CVCE.WP10.V1.0
Programme Name: ...................... IST Project Number: ........................... 287704 Project Title:.................................. CUBRIK Partners:........................................ Coordinator: ENG (IT) Contractors: UNITN, TUD, QMUL, LUH, POLMI, CERTH, NXT, MICT, FRH, INN, HOM, CVCE, EIPCM, EMP Document Number: ..................... cubrik.D101.CVCE.WP10.V1.0 Work-Package: ............................. WP10 Deliverable Type: ........................ Accompanying Document Contractual Date of Delivery: ..... 30 September 2013 Actual Date of Delivery: .............. 30 September 2013 Title of Document: ....................... First CUbRIK Application Demonstrators Author(s): ..................................... Lars Wieneke, Claudio Massari, Jasminko Novak, Marilena Lazzaro, Vincenzo Croce Approval of this report ............... Summary of this report: .............. History: .......................................... Keyword List: ............................... Availability .................................... This report is public
This work is licensed under a Creative Commons Attribution-NonCommercialShareAlike 3.0 Unported License. This work is partially funded by the EU under grant IST-FP7-287704
First CUbRIK Application Demonstrators
D10.1 Version 1.0
Disclaimer This document contains confidential information in the form of the CUbRIK project findings, work and products and its use is strictly regulated by the CUbRIK Consortium Agreement and by Contract no. FP7- ICT-287704. Neither the CUbRIK Consortium nor any of its officers, employees or agents shall be responsible or liable in negligence or otherwise howsoever in respect of any inaccuracy or omission herein. The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7-ICT-2011-7) under grant agreement n째 287704. The contents of this document are the sole responsibility of the CUbRIK consortium and can in no way be taken to reflect the views of the European Union.
First CUbRIK Application Demonstrators
D10.1 Version 1.0
Table of Contents EXECUTIVE SUMMARY
1
1
2
HISTORY OF EUROPE APPLICATION 1.1 USER STORY 1.2 OVERVIEW 1.3 INTERFACES 1.3.1 Indexation: Manual upload: Process New folder 1.3.2 Social graph and context expander 1.4 STATUS AND OUTLOOK Y3 1.4.1 Use cases 1.4.2 Components 1.4.3 Data sources
2
SEARCH FOR SME INNOVATION APPLICATION 2.1 USER STORY 2.2 OVERVIEW 2.3 INTERFACES 2.3.1 Image crawling from SN 2.3.2 Trend Analysis for category 2.4 STATUS AND OUTLOOK Y3 2.4.1 Use cases 2.4.2 Components
First CUbRIK Application Demonstrators
2 4 5 5 9 13 13 13 16 17 17 18 19 20 20 31 31 32
D10.1 Version 1.0
Executive Summary The following document describes the first release of the demonstrators for the two CUbRIK V-Apps. The document will provide a general overview for each application but will link to the specifications of the applications (in particular D2.2 and D2.3) for further details. The documentation of the components focuses on the practical use of the components. Whenever possible a connection between the implemented feature and its reference in the requirements document will be given. Chapter 1 introduces the user story for the History of Europe app, provides an overview about the different use cases and components and underlines the different crowd-sourcing approaches that were used in the app. After this the interfaces of the app are described. The chapter concludes with a status of the different components in Y2 and an outlook on Y3. The description of the fashion app in chapter 2 follows the structure of the previous chapter and presents the user story, the use cases with their different components, the crowd sourcing approach taken in the app and the different interfaces of the application. The chapter concludes as well with the status of the components and an outlook on their future use.
First CUbRIK Application Demonstrators
Page 1
D10.1 Version 1.0
1 History of Europe application The breadth and scale of multimedia archives provides a tremendous potential for historical research that hasn't been fully tapped up to know. The History of Europe application demonstrates the integration of human and machine computation that combines the power of face recognition technology with two distinctively different crowd-sourcing approaches to compute co-occurrences of persons in historical image sets. These co-occurrences are turned into a social graph that connects persons with each other and positions them, through information about the date and location of recording, in time and space. The resulting visualization of the graph as well as analytical tools can help historians to find new impulses for research and to un-earth previously unknown relationships. As such the integration of human expertise and machine computation enables a new class of applications for the exploration of multimedia archives with significant potential for the digital humanities. This document describes the first HoE demonstrator released in M24. For further details about the specifications and detailed requirements please have a look at D 2.2 or D2.3. The application is accessible at the address: http://cubrik1.eng.it:8084/SMILA/hoe/index.html
1.1 User story
Figure 1: User story "Who is this person? Tell me more" As outlined in D2.2 and D2.3 the HoE demonstrator builds on the user story "Who is this person? Tell me more" A newly scanned photo shows a few people at a dining table. Amidst familiar faces Ibrahim notices one person that is unknown to him. He uploads the image to the History of Europe application to find out who this is. He selects the person’s faces in the scan and provides the First CUbRIK Application Demonstrators
Page 2
D10.1 Version 1.0
names of some of the people he already recognized. After a while, the image analyser returns suggestions for the identity of the missing persons. One of the suggestions proposes that the unknown person might be a Dutch politician. Based on the reference images that the application offers him Ibrahim is able to confirm this identity. Ibrahim wants to find out more details about the photo and starts the exploration of the social graph. There, he finds more photos of the same event. Based on the number of co-occurrences of persons in these and other photos, he is able to identify possible candidates who, to his surprise, could also have known the politician he identified in the initial picture. For every person Ibrahim selects in the graph he sees the results of the context expander, which in addition to photos provides him with a list of media items such as newspaper articles, interviews, commentaries and videos.
First CUbRIK Application Demonstrators
Page 3
D10.1 Version 1.0
1.2 Overview The user story is implemented through four different use cases with dedicated pipelines and components (see figure 2 for an overview). Image Indexation Clickworkers
Acquisition of co-occurrences
Media harvesting and upload
Copyright aware crawler
Provenance checker
Metadata Entity ex traction
License checker
Face detection
Crowd Face position validation
Face identifi cation
Content provider tools
Data Set Creation
Face recognition
Identity reconciliation CROWD prefi ltering
Entity verifi cation & annotation
Entitypedia Integration
Expert Crowd
Indexation of textual information
Connection to the CVCE collection
Entity anntation and ex traction
Ex pert CROWD verifi cation
Entity extraction
Social Graph construction
Entitypedia data provisioning
Social Graph Creation
Entitypedia Int egration
Entity reconciliation
Uncovering who a person is Visualization of the social graph
Expert CROWD Research Inquiry CROWD Research Inquiry
Graph Visualization
Expanding the context [‌] Expert Crowd Ex pansion through docum ents
Ex pansion through im ages
Ex pansion through videos
Ex pansion through related entities
Social Graph Network Analysis Tk Analysis of the social graph
Graph Visualization
Query for entities
Query ex ecution
Graph Visualization
Figure 2: Overview HoE app in M24, grey boxes mark components for Y3 Overall the CUbRIK platform envisages three different kind of crouwdsourcing approaches depending on the peculiarities of the crowd, e.g. through cultural predispositions and common background knowledge and according to the specific kind of interaction mechanism that is put in place for the execution of the task. The crowds that the CUbRIK platform can address to outsource tasks can be classified as: •
Games With A Purpose (GWAPs) Players: persons engaged in playing with the different games supported by the CUbRIK game framework, including Sketchness, for object detection in images, SteamPilot, for image tagging, and WordGames, such as Hangman and Crosswords, for entity refinement.
First CUbRIK Application Demonstrators
Page 4
D10.1 Version 1.0
•
•
Crowd Workers: a crowd of paid workers. The Amazon Mechanical Turk1 and Microtask Crowdsourcing2 markets have been used for a variety of tasks, ranging from face detection to feedback collection in videos playout. Expert Crowd: a crowd of reference domain experts that interact with CUbRIK platform according to a Query & Answer (Q&A) mechanism. In HoE the Q&A interaction mechanism is exploited to post queries and retrieve answers from a crowd of experts to solve the task of people identification; the expert crowd is addressed with multiple mechanisms, including email invitation and message posting on social networks.
Figure 3: CUbrIK Crowd typologies In the context of HoE two of those crowdsourcing approaches were exploited. In particular CrowdWorkers and Expert Crowd were involved as depicted in Figure 1: User story "Who is this person? Tell me more". For the first release of the demonstrator, a subset of components has been implemented. Further components will be integrated in Y3 and the existing components will be refined based on the evaluation of the first demonstrator. In the following we will present the interfaces that where developed for the different pipelines and components.
1.3 Interfaces 1.3.1
Indexation: Manual upload: Process New folder
Overview A user starts the upload of new images through the "Process new Folder" function (see Figure 4: HoE app "Process new folder"). Once the images are uploaded, the Face detection component starts and returns face positions in the different images.
1 https://www.mturk.com/mturk/ 2 http://www.microtask.com/ First CUbRIK Application Demonstrators
Page 5
D10.1 Version 1.0
Figure 4: HoE app "Process new folder" A crowd of clickworkers on the Microtask platform reviews these face positions in two ways: to identify false positives the crowd members evaluate whether the bounding boxes placed on an image actually cover a face or not (see Figure 5: Crowd Face position validation). For false negatives the crowd is able to annotate missing faces in the image. Overall a face position is verified after three independent confirmations from different clickworkers.
Figure 5: Crowd Face position validation The following Face identification component returns a list of potential identities for the faces in the image. Due to the difficult dataset (non-normalized gaze directions, different levels of image quality, persons are in different ages showing therefore significant changes in appearance) the results of the identification process are not reliable enough and require manual confirmation.
First CUbRIK Application Demonstrators
Page 6
D10.1 Version 1.0
Figure 6: CROWD Research Inquiry
Figure 7: Entity verification & Annotation Two levels of confirmation are introduced: in the Expert CROWD Research Inquiry component an external crowd of experts is invited to participate in the confirmation task and either votes on one of the identities already provided in the list or adds a new identity in case it was not available before (see Figure 6: CROWD Research Inquiry. The user who uploaded the images is also able to annotate images, faces and identities and to verify the votes of crowd users (for face position annotation see Figure 7: Entity verification & Annotation).
First CUbRIK Application Demonstrators
Page 7
D10.1 Version 1.0
Components Clickworkers
Identity reconciliation Media harvesting and upload
Crowd Face position validation
Face detection
Entity verifi cation & annotation
Face identifi cation
Entitypedia Integration
Face recognition
Expert CROWD Research Inquiry CROWD Research Inquiry
Expert Crowd
Figure 8: Process New Folder
Name
Illustration
Functionality
Media harvesting and upload
Figure 4: HoE app "Process new folder"
Upload of images, minimal implementation of the data set creation workflow
Face detection
-
Detects face positions in uploaded images
Crowd Face Position validation
Figure 5: Crowd Face position validation
Click-workers validate the detected face positions
Face identification
-
Algorithm identifies persons in verified face positions
CROWD Research Inquiry
Figure 6: CROWD Research Inquiry
Expert users are invited to give their opinion on the identity of the identified faces
Entity verification & annotation
Figure 7: Entity verification & Annotation
App users can add annotations and verify information
Entitypedia Integration
-
Write annotations to Entitypedia
First CUbRIK Application Demonstrators
Page 8
D10.1 Version 1.0
1.3.2
Social graph and context expander
The social graph is visualized by means of a force-directed graph layout that groups nodes with stronger link weights closer together than nodes with weaker links in such a way that a global optimum is reached. The visualization is performed by the JavaScript components Crossfilter and D3.js3 that display the graph in the form of an interactive visual network of persons and connections between them. Thereby, each node represents a person and each edge represents the frequency of their co-occurrences in the given photo collection. The width of an edge represents the number of co-occurrences, in such way that a wider edge means that both persons appear simultaneously in a high number of photos (see Figure 8: Visualization of the social graph). The integration of the social graph on the ENG platform is currently on-going.
Figure 8: Visualization of the social graph Such network visualizations have the potential to highlight unexpected connections and patterns in all kinds of relations, be they social or between other nodes such as documents. In order to further evaluate any such finding it is however important to be able to interactively explore the relationships between the nodes, manipulate the parameters determining their visualization and re-contextualize the graph based on specific data subsets that are of interest to the user. Users can interact with the History of Europe social graph in different ways, e.g. clicking on a node results in an ego-graph of the selected person (see : Ego graph) and clicking on an edge displays documents that relate to the selected relationship between two persons (see Figure 10: Connection between two persons and related persons). As the documents stored in the collection very often come with a date of creation, the graph can be filtered by date with the timeline, displaying only the connections of documents created within this timespan. This timeline also shows the amount of photos per date that are contained in the collection. Another filtering option is the number of connecting documents, which allows the visualization of those relationships that are only included in an interval of a minimum and maximum
3
see http://d3js.org/
First CUbRIK Application Demonstrators
Page 9
D10.1 Version 1.0
number of documents. This feature is useful to highlight highest co-occurrences. Finally, the number of appearances of a person in the processed collection enables users to identify people who appear particularly often in any given time frame (see Figure 9: Ego graph).
Figure 9: Ego graph
Figure 10: Connection between two persons and related persons Filtering of images used for the construction of the graph is also possible by defining the number of persons that were recognized in the images. This filter is mainly used to show/hide those group pictures that include a high number of people on the same picture.
First CUbRIK Application Demonstrators
Page 10
D10.1 Version 1.0
After selecting a person, the graph turns into an ego-graph (see Figure 9: Ego graph), centring on the selected person and showing all relationships for the person. All photos in which the selected person appears are displayed below the graph, sorted by date and in consideration of other filters. Users can review who is tagged in each of these photos by moving the mouse over the pictures, this highlights the different persons that appear in the selected photo in the graph. In addition to these photos, a list of links to text documents that contain references to the selected person is displayed below the photos (see Figure 11: Expansion through documents, display of related documents for specific persons).
Figure 11: Expansion through documents, display of related documents for specific persons Another powerful feature is the inspection of relationships. Whenever a user clicks on an edge between two persons, the selection shows all images and documents where both persons appear in and that therefore constitute the relationship between the persons (see Figure 10: Connection between two persons and related persons).
First CUbRIK Application Demonstrators
Page 11
D10.1 Version 1.0
Components Uncovering who a person is Visualization of the social graph
Expanding the context […] Ex pansion through docum ents
Graph Visualization
Social Graph Network Analysis Tk Analysis of the social graph
Graph Visualization
Figure 12: Visualization, network analysis and context expander Name
Illustration
Functionality
Graph visualization
Figure 8: Visualization of the social graph
The social graph of co-occurrences is visualized.
Analysis of the social graph
Figure 8: Visualization of the social graph, Figure 9: Ego graph, Figure 10: Connection between two persons and related persons
The user is able to analyse the social graph with a set of filters.
Expansion through documents
Figure 11: Expansion through documents, display of related documents for specific persons
Once the user selects a person or the edge between two persons, related documents from the CVCE collection and other websites are shown
First CUbRIK Application Demonstrators
Page 12
D10.1 Version 1.0
1.4 Status and outlook Y3 1.4.1
Use cases
Use case Indexation
Y2 Boot strapping of the social graph using the CVCE dataset (3k images), initial implementation of Copyright aware crawler, provenance and license checker as well as content provider tools. Indexation of Europeana content.
Y3 Integration of the EC Audiovisual library, Integration of all components in the pipeline
Indexation of textual information
-
Co-occurrence analysis in CVCE documents and interview transcripts
Social graph construction
Inital version ready
Refinement and full integration in SMILA
Uncovering who a person is
Visualisation of the social graph, initial set of analytical tools, initial integration of Expert Crowd Research Inquiries Initial implementation of the Expansion through documents component
Refined visualization and analysis, full integration of all components in the SMILA pipeline
Expanding the contex […]
1.4.2
Implementation of all other components and integration in SMILA.
Components
For guidelines on how to access and install the individual components please see D9.4 Component Media harvesting and upload CERTH
Y2 Crawled Europeana content for given queries
CONTENT PROVIDER TOOL FRH
- Reliably bind arbitrary license information to local content
Y3 SMILA pipeline for the Europeana API Integration of the EC Audiovisual library - Various usability improvements related to content preparation and upload (based on feedback from content providers)
- Uses digital signatures
Object Storage FRH
- Implemented as commandline tool - Binary (media content) and json doc storage, based on a MongoDB core
- Integration of License Checker and Provenance Checker functionalities
- Support for user and component authentication, First CUbRIK Application Demonstrators
Page 13
D10.1 Version 1.0
Component
Copyright aware Crawler FRH
Provenance checker FRH
Y2 and access control (prerequisite for many content providers) - REST interface and SMILA client (both SMILA pipelines integration and direct integration with UI / apps possible) - Crawling only of content that complies with specific required permissions (e.g., rights to modify and use in a commercial context) - Supports relevant image sources, including Flickr and Wikimedia commons - Content duplicate and reuse detection using hashes - Keeping track of content source, creation and publication time
License checker FRH
Face detection POLMI
- Retrieval of Creative Commons information, and interpretation into usage permissions - Keeping track of license metadata and derived usage permissions for content Completed
Crowd face position validation POLMI Face identification POLMI
Completed "face validation" and "face add"
CROWD prefiltering L3S
Systematic study to understand annotation difficulties for different target users (generic crowd / expert crowd). Development of candidate features that enable a prediction whether a photo is easy or hard to annotate.
Completed
First CUbRIK Application Demonstrators
Y3
- Support relevant video sources, including news video podcasts and Vimeo
- Including crowd interaction to complete and validate license information - Perceptual duplicate and reuse detection using phash - Integration with object storage (simplifies communication and usage) - Support for Europeana and proprietary license information mapping - Integration with object storage (simplifies communication and usage)and usage) Review of the "Shore" component as an alternative to KeeSquare. Option to extend to video in Y3 -
Integration of co-occurence information (social graph) and textual annotations in the identification process Integration of the full pipeline to distribute tasks to generic crowds and or experts.
Page 14
D10.1 Version 1.0
Component Entity verification & annotation HOMERIA
Entitypedia integration UTN
Connection to the CVCE collection CVCE
Entity annotation & extraction EMPOLIS Expert CROWD verification CVCE Expert crowd entity verification POLMI, CVCE
Social graph creation QMUL Query for entities EMPOLIS Visualisation of the social graph HOMERIA
Social Graph network analysis toolkit CERTH
Y2 UI for showing the results of face detector/identification of images. UI for annotating data related to one image (place, date, event...), and data related to a person (name, date of birth...) Entitypedia entity model extended with HoE entity model. HoE entities are stored in Entitypedia. Basic entity search functionality on top of HoE entities. Provision of around 10.000 historical documents containing persons, places, organisations, events names and periods of time. -
Y3 UI for enabling verification of entities by voting and adding explanation to the possible values of annotations. (Gamified interface)
-
Application for the validation of named entities within documents by an expert crowd Improvements to the conflict manager: - Handle different task dispatching mechanisms (e.g., Twitter); - Keep track of identity of expert workers (to build expert profiles); - Targeted dispatching of tasks depending on worker expertise.
Implementation of conflict manager (CrowdSearcher) to dispatch entity verification (via email - addresses provided by CVCE). Naive computation of relevance score (output of face identification + crowd feedback) Creation of the social graph from information stored in the Entitypedia. Initial version
Visualization of people as nodes of the graph, cooccurrences of people as relationships of the graph and exploration of images related to a selected person or relationship. Basic graph analysis
First CUbRIK Application Demonstrators
Word games on top of HoE entities
Connection to the CVCE back-end (for entities extraction of new documents)
Extraction of entities from CVCE documents and other text inputs also event extraction
Integration with SMILA,
Full integration of search for query, connection with Entitypedia, Auto completion of Entity names Alternative visualization of the graph for events, places, etc.
Graph analysis toolkit, that provides local topological features, global features (e.g. graph diameter) and clustering based on modularity. Spectral clustering, calculation of shortest distance
Page 15
D10.1 Version 1.0
Component Expansion through documents EMPOLIS Expansion through video FRH
Y2 Complex Query document expansion phase 1
Y3 Complex Query document expansion phase 2
Deferred to Y3 in favor of News History Demo
- Expansion of HoE with News Content History (analysis of shared / reused footage within news videos) - Connection via related events / tags
Expansion through images L3S Expansion through related entities L3S
1.4.3
First implementation of the component based on a detailed annotations study -
- Possibly connection via face recognition Refinement of the component
First release of the component
Data sources
Y2: Bootstrapping of the application with the CVCE data collection 3924 images, r.a. 1700 persons Y3: Extension Integration of material from the EC Audiovisual Service 131k images, unknown amount of individual persons Fallback Despite some recent progress, the agreement of the EC Audiovisual Service is still under review. As fallback a large set of images from the Europeana has been crawled that will be used in case no final agreement with the EC Audiovisual Service can be achieved in Y3.
First CUbRIK Application Demonstrators
Page 16
D10.1 Version 1.0
2 Search for SME innovation application The Search for SME Innovation application is a web application focussed on the fashion domain, a sector constantly searching for new ideas, innovation of products and new trends to be launched in the market, and its purpose is supporting SME’s to exploit the potential of the new technology to get feedback from and learn about the needs of fashion consumers. Through the tool, the organizations are able to receive information about the opinions and the feedback of their potential customers, their preferences, what they like about clothes and the current trends. In this application, primarily existing content from social networks is used (e.g. fashion pictures that are crawled from Twitter) in order to make this application an efficient tool for market analysis for SME’s. Each organization involved in the production of new fashion lines and products, from the main producers to distributors and B2C organizations, invest budget in marketing research for a better and deeper understanding of the potential customers‘ preferences. For this reason, the target groups of the tool are both large organizations in the fashion sector and small and medium sized enterprises (SME): Large organizations (great buyers, distributors, clothes producers addressing general public) already make use of market analysis, with their own experts or with the support of external ones; these organizations may benefit of the more affordable results provided by the application or they may use both the traditional market analysis and the new one, having another useful tool to retrieve information they need; • SMEs may not have enough budget to perform market analysis, basing their analysis just on the experience of their personnel; such organizations, through the application, have the opportunity to gain affordable analysis of their potential customers and trends. This document describes the first Search for SME innovation demonstrator released in M24. For further details about the specifications and detailed requirements please refer to D 2.3. The application is accessible at the address: http://cubrik1.eng.it/TrendAnalysis/index.html •
2.1 User story In the deliverable D2.3 two user stories have been presented as overview of the desired functionalities of the demonstrator of the Search for SME Innovation application. In the second year the first demonstrator has implemented the following user story: Margaret, the experienced retail buyer who usually has a notion of consumer preferences but prefers to verify her own ideas before placing an order. “Fashion4You” is a small clothing company. Margaret, one of their retail buyers, is preparing a new order of women’s casual wear for the upcoming season. She’s trying to decide which models of skirts should be bought from the local distributor. Margaret is very experienced in her job and she already has an idea of how to structure the order. But there are many aspects to be considered, such as cut, colours and textures, and decisions need to be based on trend predictions and consumer preferences. Usually, the manager of the company would commission a trimestral market research from an external agency that Margaret can rely on when planning her order. This time however, Margaret must rely only on her intuition and experience since it was decided to dispense with market research due to recent economic shortcomings. Margaret remembers that her friend Kate, who is also a retail buyer, spoke very enthusiastically about CUbRIK Fashion, a new, innovative application she uses to get an overview of current consumer preferences. The application extracts current fashion trends by garment category, e.g. colour and texture preferences, from social network posts. As a result, First CUbRIK Application Demonstrators
Page 17
D10.1 Version 1.0
it provides statistics depicting the trends over time, as well as examples of the most popular trends. Margaret decides to try out the application to verify if the information extracted from the social networks matches her own notion of customers’ likely preferences. Using the Social Network Trends functionality, Margaret selects the category “skirts” and asks the application to provide a trend analysis based on social network posts from the last two months. Exploring the results of the analysis, Margaret can confirm that “turquoise” has been a very popular colour, as she thought. Moreover, the Colour Combination Analysis shows that this colour is most frequently combined with “purple” and “dark green”. Viewing the Print & Graphic Trends, she verifies that the texture preferences haven’t changed much in the past six months, which is also reflected in her own analysis of the sales in her store. With the Popular Photos Gallery, Margaret can browse images of consumers’ most preferred skirt models, compare them over time and as part of outfits. This helps her discard some models from the order and at the same time take into account others she didn’t consider before. Having retrieved the information she needed, Margaret is satisfied: the order she prepared needs only little adjustment, but her notion about the kinds of models and colours to be ordered was almost correct. In this document the first demonstrator is presented and the use cases realised in the year 2 are reported and described. For more details about the development process, the user stories and the derived use cases as well as a functional description of the different components, the D2.2 and D2.3 should be considered.
2.2 Overview For the first release of the demonstrator and the realization of the user story “Margaret, the experienced retail buyer who usually has a notion of consumer preferences but prefers to verify her own ideas before placing an order” two use cases have been implemented and a subset of components has been used: •
Image crawling from SN: this use case aims at crawling fashion related images from social networks and further processing them in order to extract segments representing different garments and their features. Crowdsourcing is used in the detection of the clothing items present in the images extracted by the social networks, improving the possibility and the quality of the segments detection in low quality images or in images where parts of the body cannot be recognised by the software algorithms. The images are annotated relatively to categories of clothing items, while textures and colours are derived from the images automatically. This use case is technical and not user related: it aims at crawling and processing the images so that they can be used in the other use cases. In relation to the full use case as reported in deliverable D2.3, in the first demonstrator the component Feedback filter – Likelines is not yet integrated. It will be part of the demonstrator in the 3rd year. The component is shown in the overview, but it drawn in a different colour.
•
Trend Analysis for category: this use case aims at providing the SMEs users with a trend analysis of clothing preferences based on the clothing category the SME user has selected. The trend analysis functionality analyses the images extracted from the social network and further processed for the features and segments extraction, in order to provide the users with an insight report of the fashion trends based on the user preferences on the social network. This use case uses the images extracted and processed by the use case described above.
First CUbRIK Application Demonstrators
Page 18
D10.1 Version 1.0
In the following figure the overview of the use cases implemented in the second year.
Figure 13: Overview of the first demonstrator Search for SME innovation app at M24, grey boxes mark components for Y3 As depicted in Figure 1: User story "Who is this person? Tell me more" in general three kind of crowd are involved in CUbRIK. In the context of Fashion V-App the task of body part detection is partially outsourced to humans in case of images that are too difficult to be processed, in particular when the confidence of the body part detection is considered to be not sufficient. In this case GWAP players were exploited through the Sketcheness GWAP.
Figure 14: Sketchness GWAP
2.3 Interfaces The Graphic User Interfaces constituting the application is presented in the following paragraphs, where the GUI of the Search for SME Innovation V-App as envisioned for the Fashion SMEs is described both from the user perspective and the technical point of view. The application focuses on the GUI elements required for the “Trend analysis for category� use case.
First CUbRIK Application Demonstrators
Page 19
D10.1 Version 1.0
2.3.1
Image crawling from SN
Overview As already described in the previous section, the “Image crawling from SN� use case is technical and not user related, thus no part of the GUI is associated to it. Components The figure below reports the components implemented in the use case and in the following table the functionality they provide is described.
Figure 15: Image crawling from SN components at M24
Component Name
Illustration
Functionality
Image extraction from SN
-
It crawls the images from the Social Network (Twitter).
Descriptors extraction
-
It provides the extraction oft he features (colours and textures) of the images and the garments segments
Body part detector and router
-
It provides the automatic extraction of the garment segments, by detecting the parts of the body in the images
GWAP Sketchness
-
It provides the manual extraction of garments segments, by involving the humans in the loop
Accessibility annotation
-
It provides the annotation of the images with metadata accessibility-related
2.3.2
Trend Analysis for category
Overview As explained, the application implements the functionalities needed for the realization of the user story selected for the first demonstrator. The SMEs users can dynamically generate trend analyses based on crawled social network data (Twitter), by selecting a garment category and time span (Figure 17). The analysis provided to the SMEs users identifies colour trends, colour combination trends as well as prints & graphics trends for the identified timespan and selected garment category. The Figure 16 shows the main page of the GUI. First CUbRIK Application Demonstrators
Page 20
D10.1 Version 1.0
As shown in the figure, the SMEs users are provided with three main functionalities, selectable through the main menu: The main functionality is the “Social Network Trends” that implements the core of the Trend Analysis for category use case: it presents to the users the trend analysis of the category selected; • The “Dashboard” that offers respectively the possibility to further access to the analysis already retrieved; • The “Popular Photos” that offers a different point of view of the trends analysis, showing the most popular images. The Main page of the GUI is the Dashboard. •
Figure 16: Main page of the GUI. Social Network Trends This functionality is the core one available to the user and developed in the first demonstrator. It provides the user with an insight on trends for the selected category and the time range set up. As shown in Figure 17, the user has firstly to select the category he is interested in receiving the Trend Analysis. The garment categories available are: • Shirt • T-shirt • Trousers • Skirt • Shorts • Suit • Dress The first two categories belong to the upper body garment typology, the next three to the lower body garment typology and the last two are related to the complete body garments. Such choice on one side ensures the coverage of all three basic typologies of garments First CUbRIK Application Demonstrators
Page 21
D10.1 Version 1.0
(related to the lower part of the body, the upper part of the body and the complete body) and on the other side represents the most used list of categories as suggested by an organization working in the fashion sector informally interviewed. In the second release of the demonstrator the user will be able to select also “all categories�, that will provide him a comparison and an analysis of the trend and the preferences of the garment categories implemented. This function in the first release of the demonstrator is disabled.
Figure 17: Social Network Trend – choice of the category After having selected the category garment, the user can select the timespan and then can ask the system to generate the trend analysis according the parameters selected, as shown in Figure 18.
First CUbRIK Application Demonstrators
Page 22
D10.1 Version 1.0
A trend analysis can be run, with timespan and clothing category as filters.
Figure 18: Social Network Trend – setting up the timespan The results of the trend analysis are shown in Figure 19. The trend report provides information about the most preferred colours, presented in a pie chart with an associated percentage, a chart showing the colours trends in the timeline selected, the most preferred combination of colours and the colour trend images for each of the colours reported. Finally, the user can have an insight on the textures of the top images (the most preferred images).
First CUbRIK Application Demonstrators
Page 23
D10.1 Version 1.0
A trend report is provided for colour, colour trends, colour combinations and prints & graphics.
Figure 19: Results of the trend analysis. The user can also view the most popular social network photos used for the analysis in the “Photo Gallery�. For this gallery, as shown in Figure 20, the samples of the most popular photos are reported, with the information of the timeline considered and the most preferred colours. The images reported can be displayed as a gallery (Figure 20), or as thumbnails (Figure 21), accordingly to the view selected by the user.
First CUbRIK Application Demonstrators
Page 24
D10.1 Version 1.0
Most popular pictures can be displayed, based on the no. of Tweets
The images can be displayed as gallery
Figure 20: the “Photo gallery” of the trend analysis – gallery view.
Figure 21: the “Photo gallery” of the trend analysis – thumbnail view.
First CUbRIK Application Demonstrators
Page 25
D10.1 Version 1.0
The dynamically generated analyses can be saved so users can review them later. Saved analyses are listed and previewed in the application dashboard (Figure 23). In the first demonstrator the analysis are automatically saved. Clicking on each image, a pop up comes up and shows its details, as shown in Figure 22.
Figure 22: the “Photo gallery” of the trend analysis – pop up of the image details Dashboard Through the Dashboard, the SMEs users can have an overview of the analysis launched in the past. As shown in Figure 23, The Social Network Trend Analysis performed in the past are listed with the information about when a specific analysis was launched and what garment category it was referred to. The user is provided also with the possibility to delete a specific past analysis by clicking the red “X” at the right of the analysis.
First CUbRIK Application Demonstrators
Page 26
D10.1 Version 1.0
Saved analyses can be deleted by clicking on the red “X”
List of the saved analyses Saved analyses can be re-visisted by click on the white arrow
Figure 23: Dashboard – main page By clicking on the white arrow at the right of a specific analysis, the user can view a resuming of the results of the analysis, as shown in Figure 24. Clicking on “Details”, the Trend Analysis page will be opened and the information of the stored analysis selected shown as when it was launched.
First CUbRIK Application Demonstrators
Page 27
D10.1 Version 1.0
Saved analyses can be re-visisted by clicking on “Details�
Figure 24: Dashboard – selecting analysis retrieved in the past Popular Photos Using the Popular Photos functionality the users can have an overview of the most popular images extracted and analysed in the trend analysis process, performed every day by the technical component. This functionality allows the users to have an overview of the most popular images with the aim of supporting the users to better understand which are the trends and the most preferred looks in the social network. The functionality is shown in Figure 25. As explained in the balloons, the user can interact with the application by selecting the timespan and the garment category he is interested in; moreover, the user can change how the results are reported by choosing among different views, so the results can be displayed as a gallery (Figure 25), grouped by garment type (Figure 26), or as thumbnails, as shown for the Social Network Trend Photo Gallery (Figure 21). The results reported can be filtered by popularity and sorted by date or popularity. According the sorting parameter, the horizontal scroll bar shows the time range or the popularity range. Clicking on each image, a pop up come up and shows its details, as shown in Figure 27.
First CUbRIK Application Demonstrators
Page 28
D10.1 Version 1.0
The timespan can be changed Images can be selected according users’ preferences The garment categories can be selected
Different views of the images are available Images sorted
can
be
Figure 25: popular photos functionalities
Figure 26: popular photos – images grouped by garment type
First CUbRIK Application Demonstrators
Page 29
D10.1 Version 1.0
By clicking on a image, its details are shown
Figure 27: popular photos functionalities – pop up of the image details
Components The figure below reports the components implemented in the use case and in the table the functionality they provided is descripted.
Figure 28: Trend Analysis for category components at M24
Component Name
Illustration
Functionality
Trend Analyser
Figure 28: Trend Analysis for category components at M24
It provides an analysis of the trend in the fashion sector for the category of clotes addressed
First CUbRIK Application Demonstrators
Page 30
D10.1 Version 1.0
2.4 Status and outlook Y3 2.4.1
Use cases
The following list reports the status of the use cases at Y2 and the foreseen improvements and developments in Y3. Component
Y2
Y3
Image crawling from SN
Crawling fashion related images from Twitter and extraction of the garment segments, colour and textures
Crawling fashion related images from Flickr and integration of Likelines for fashion key frame extraction from videos. Refinement of the feature extraction and segment identification
Trend analysis category
for
Trend analysis provided for a set of garment categories. Analysis of the popularity of the images extracted from the Social Network Twitter based on the numbers of twits and analysis of the most popular colours, combination of colours and textures.
Analysis of “all categories” to be produced to the GUI Refinement of the algorithms for improving the quality of the results
Trend analysis sample
for
-
To be implemented. Analysis of trends provided for images similar to a sample uploaded by the user. The analysis is based on the popularity of the images in the Social Network.
Popularity Assessment
-
To be implemented. Assessment of the popularity of images uploaded by the user by gathering the preferences of the user’s network on the social media about the images uploaded.
Fashion Matching
-
To be implemented. Searching and matching images functionalities for allowing the users to select the possible matches and request users’ assessment using their own network in the social media.
Personalised suggestion
-
To be implemented. Suggestion of queries based on the previous users’ behaviours
query
First CUbRIK Application Demonstrators
Page 31
D10.1 Version 1.0
2.4.2
Components
For guidelines on how to access and install the individual components please see D9.4. The following list reports the status of the component at Y2 and the foreseen improvements and developments in Y3. Component
Y2
Y3
Implicit feedback filter- Likelines
Not integrated in the use case. It extracts key frames from videos fashion-related-
To be integrated into the use case
Image extraction from Social networks
Images crawled in Twitter using predefined categories of garments.
Flickr crawling and integration with the copyright and licensing component
Body part detector and router
Segments selection and routing to crowd based on quality score of the images.
Refining of the algorithms for improving the quality of the detection of the garment segments
Descriptor extractor
Colours and textures extraction from complete images and related segments
Refinement of the algorithms for results improving
GWAP Sketchness
Game for the manual segmentation of garments
Refinement of the Game and introduction of annotation verification of the images
Trend Analyzer
Analysis of the popularity of the images extracted from the Social Network Twitter based on the numbers of twits and analysis of the most popular colours, combination of colours and textures.
Analysis of “all categories� to be produced to the GUI Refinement of the algorithms for improving the quality of the results
Image detection
-
Searching in the internal database of the application images similar to the one chosen by the user, accordingly the category, the colour and the texture parameters.
Relevance feedback
-
Reordering the images ranking the similarity through the usage of human perception of similarity
Conflict manager
-
Publication of links of the vertical application in the social network, to reach the network of the demonstrator users
Accessibility
Annotation of the images with the metadata describing the accessibility-related feature of the images
The accessibility filtering functionality will be implemented for reordering of the images according the sight
similarity
First CUbRIK Application Demonstrators
Page 32
D10.1 Version 1.0
Component
Y2
Y3 impairment selected.
Behavior analyzer
-
Analysis of the users’ behaviours for further query suggestions
User profiling
-
Register and login functionality
CMS for Human Computation
-
Gathering of the users’ actions for the further analysis of the Behaviour Analyser to be used in the query suggestion
First CUbRIK Application Demonstrators
Page 33
D10.1 Version 1.0