Transient Narrative Networks for Interpreting Multi-Modal Text

Transient Narrative Networks for Interpreting Multi-Modal Text.? Luc Steels1[0000−0003−0068−6551] Catalan Institute for Advanced Studies (ICREA) - Institut di Biologia Evolutiva (UPF-CSIC) PRBB. Dr. Aiguadar 88. 08003 Barcelona steels@arti.vub.ac.be https://www.icrea.cat/Web/ScientificStaff/luc-steels-539

Abstract. Multi-modal sources consist of written or spoken text or dialogs, images, sounds, videos and sensory-motor datastreams. To understand them requires the integration of different types of pattern recognition and structure extraction processes (visual, speech, linguistic, sound) augmented with information coming from semantic memory, episodic memory and mental simulation. The reconstruction of a narrative, in which all these disparate sources of information fit together, requires a central data structure representing not only the narrative structure itself but also all relevant background knowledge and the visual or other data involved. This paper makes a concrete proposal for such a structure called a Transient Narrative Network (TNN). The paper details the requirements and functions of this network using a concrete example from the narrative interpretation of a painting by the Flemish contemporary painter Luc Tuymans. Keywords: AI, narrative construction, meaning, understanding, transient narrative networks

1

Introduction

1.1

Background

The notion of a narrative[2], originally the province of literature studies and semiotics, is gaining attention in many fields, ranging from economics and medicine to law and sociology. Also in AI, narratives are increasingly understood as playing a central role in human intelligence[10]. For example, in computational linguistics, they are considered for resolving co-referential relations and ambiguities, in image processing they complement purely bottom-up visual processing with top-down interpretation in order to get more robust image recognition. There has already been a flurry of research activity in AI on narratives between the late nineteen-sixties and mid nineteen-seventies, centered around the ?

The research reported here was partly funded by the EU FET Proactive Project MUHAI on ’Meaning and Understanding in Human-centric AI’ and by an EU STARTS project for ‘Scientists in residence of art studios’ organized by Gluon and BOZAR (Brussels).

2

L. Steels

notion of a semantic network, as first proposed by Quillian, and the notion of a frame, as first proposed by Minsky [7]. Another early notion is that of a script proposed by Schank and colleagues [12]. These concepts have underpinned a large number of experimental AI systems for story understanding and story generation and ideas for the representation and functioning of semantic and episodic memory [11]. The past decade has seen a renewed attention on narratives in AI [5], stimulated by incredible advances in extracting information from multimodal sources and by calls for a ’human-centric’ AI [9] that tries to go beyond data-driven machine learning by addressing issues of meaning and understanding [8]. This paper reports on research within the context of a recently started European EU-FET Pro-active research project on ’Meaning and Understanding in Human-centric AI’, called MUHAI, and of activities within the EU HumanE-AINet coordination action (https://www.k4all.org/project/humane-ai-net/). The MUHAI project targets the understanding and operationalization of processes of narrative construction for two application areas: (i) The understanding of recipes, where understanding translates into the ability to execute the recipe, either with a robotic cooking installation or in simulation, and to invent variations that make sense from a culinary, nutritional or allergic point of view. (ii) The understanding and construction of data stories on social issues to support data journalists and social scientists. The MUHAI project also investigates the application of computational narrative intelligence to art works, such as paintings or musical compositions[13] Semantic Memory Language Transient Narrative Network Sensorymotor

Mental simulation

Episodic memory

Fig. 1. Left: The painting ‘Secrets’ by Luc Tuymans, used as example case study for narrative interpretation in this paper. Right: The cognitive architecture of a Narrative Interpreter integrating multi-modal inputs and diverse knowledge sources to build a transient narrative network.

The present paper focuses on the latter application domain. Specifically, it will draw examples from the narrative interpretation of a single painting, called ‘Secrets’, by the contemporary Flemish painter Luc Tuymans (see Figure 1, left). The painting has been shown in a recent solo exhibition of Tuymans at the Palazzo Grassi in Venice (2019-2020). Our goal is not to build an AI system that autonomously comes up with a single definitive narrative interpretation of this painting but to investigate the interpretation process using existing algorithms and hand-coded additions to fill in gaps in the current state of the art. We do

Transient Narrative networks

3

not strive for a single definitive narrative because that does not exist. Different viewers construct different narratives at different levels of depth and depending on their interests, time and available background knowledge. Most people visiting an exhibition may spend only a few seconds on a painting where others spend minutes, come back to the same painting time and again, keep reflecting and reading about it, thus gradually expanding their narratives. Moreover, the state of the art in AI is far from able to achieve the automatic narrative interpretation of multi-modal sources and it is even questionable whether that is a possible or even desirable goal for AI. Human-centric AI emphasizes instead collaboration with humans. Due to space limitations this paper documents only part of our efforts on narrative interpretations of paintings. More extensive documentation is shown in an exhibition at the BOZAR art center in Brussels, taking place in april 2021 (https://www.bozar.be/en/activities/171222-flow).

2

Cognitive Architecture

The overall cognitive architecture of the Narrative Interpreter explored in the MUHAI project is shown in Figure 1, right. We assume that inputs are coming from different modalities, in this case the original image, the title, the text written in the catalog [3], discussions with the artist and other viewers, images of other paintings or of the situation depicted by the painting, etc. All these elements become available in no particular order. Some of these inputs are the starting point for narrative construction, others come from actively searching for information in external repositories and through expansions based on inference, sensory processing and additional visual or auditory input. At the core of this cognitive architecture is a data structure, called a TNN or Transient Narrative Network, which is progressively expanded as the interpretation process unfolds. This data structure collects all information that is selectively considered to be relevant to the current narrative. The TNN does not necessary represent a particular narration, i.e. a particular story, but rather the underlying narrative structure, the relevant background knowledge, and intermediary steps in the visual, auditory or textual analysis of the multi-modal materials telling a story. Many different narrations could be derived from the same narrative network and the same inputs could trigger the construction of many different narrative networks. A TNN is more specific than a knowledge graph, which contains generic information (a semantic memory of general and specific facts). A TNN contextualizes the information by tying it to the current situation. It also incorporates the emotional and moral stance and the viewpoints and perspectives of the agent constructing the narrative. Technically speaking a TNN is a graph. The nodes in the graph are of three types: (i) Individual concepts, i.e. concepts which refer to individual entities that can be persons, physical objects, buildings, images, sounds, events, actions, etc. (ii) Abstract concepts, such as categories and classes, like types and properties of persons, causal and temporal relations, roles of entities in events, etc.

4

L. Steels

(iii) Anchors that ground individual or abstract concepts into physical reality as experienced through a sensory-motor embodiment, e.g. a region in an image or a URI (Uniform Resource identifier) grounding an individual or abstract concept into open linked data and the semantic web. Not all concepts in a TNN are grounded and the grounding may still be unknown at some point in the narrative construction process. In general, there is a lot of indeterminacy and part of the challenges that drive the interpretation process is to selectively diminish this indeterminacy. For example, we may already know that a painting depicts a face but not know yet to which person this face belongs, and maybe we will never know because the painter simply wanted to paint ‘a face’. The links between the nodes are relations: semantic relations, similar to the ones used in knowledge graphs, associative relations, e.g. that a mustache suggests masculinity, military and hierarchy, and constructive relations which are based on algorithms that established ground concepts, for example through edge detection or color clustering algorithms. A TNN quickly gets quite large, having hundreds of nodes and links even for a simple narrative with as input one or two sentences and an accompanying image. We use tools currently available for presenting and browsing through very large graphs to visualize a TNN and its step-wise expansion, more specifically we use Gephi (https://gephi.org/). See the example in Figure 2.

Fig. 2. Fragment of TNN triggered by the painting ‘Secrets’. It is the area of the graph that contains background knowledge and associations triggered by the title of the exhibition ’La Pelle’, which refers to a film and a book by Curzio Malaparte about the moral decay during and after World War II. The person depicted in the painting, Albert Speer, played a major role in this war as a key member of the Nazi party.

The various components of a Narrative Interpreter (Figure 1, right) add nodes and links to the TNN and consult the TNN for their own operation. The TNN can be seen as a kind of blackboard, similar to the central datastructure used in the blackboard architectures developed in the 1980s [4]. The TNN contains not only concepts and their relations but also a great deal of bookkeeping information, such as by what component and knowledge source a particular link was added,

Transient Narrative networks

5

the time it was added, how much it is currrently in the focus of attention, the network properties of a node such as its in- and out-degree and centrality, the way the concept is to be displayed in the visualization of the TNN, etc. These elements are constantly updated as the TNN expands and as attention shifts from one region of the TNN to another one.

3

Examples

To make all this more concrete, we now look at two examples of steps in the build up of a TNN for the painting ‘Secrets’. The interpretation process is guided by attention mechanisms. For example, the Narrative Interpreter could spontaneously be drawn to certain regions of an image like the human vision system does [14], towards features in the image which deviate from what is expected in reality, or towards discrepancies between the source image and the painting. Attention can also shift when there are ambiguities, anomalies or inconsistencies associated with certain nodes. Example 1. What person is depicted? One of the obvious strategies to interpret a painting is to recognize what is depicted. Using the reverse image search algorithms now offered by many search engines (e.g. Google or Bing), which themselves rely on a lot of AI algorithms and statistical models, the Narrative Interpreter can try to find back the original. However we found that none of the images returned with the painting as image key can have been the source. Instead they are sensitive to the mood of the painting in holistic terms or they show other works by the same painter. Another route is to use a hint from the description of ‘Secrets’ in the catalog, which states in the first line that ”... the face is in fact that of Albert Speer...”. Detecting the URI grounding of Albert Speer in the semantic web is now possible with state of the art named entity recognition algorithms. Queries to Wikidata immediately bring up a wealth of information that can be added to the TNN. Invoking a Google image search on Albert Speer returns quite a few images, some of them shown in Figure 3, top. The human eye immediately recognizes the most likely source of the painting as the second image on the left. Moreover we see that this source is a smaller fragment of the rightmost image. Computing the best match and finding a good alignment is actually a quite hard problem for computer vision, where this problem is known as the image registration problem. We have tried several algorithms, including the widely used SURF method [6], but only achieved results with an evolutionary algorithm that crops and stretches the painting to minimize visual distance [1] (Figure 3, bottom). Note that finding the painting source was only possible by a combination of language processing (the text in the catalog), visual analysis (best matching and alignment), and consultations of semantic memory.

6

SURF-mapping-based-on-interest-points

L. Steels

Painting

Align-painting (purple)-on-source(green)

optical-flow Source(green)-aligned-to-painting(purple)

Fig. 3. Top: Images returned by Google Image search with ‘Albert Speer’ as key. Bottom from left to right: (i) Attempts to match and align the second image resulting from Google search with the painting using the SURF algorithm. (ii) Overlay of painting and aligned source. (iii) Optical flow between painting and aligned source showing the micro-stretching of the source with respect to the painting after cropping and macrostretching.

Example 2: Why is Speer closing his eyes? One of the most eye-catching features of ‘Secrets’ is that the eyes are closed, whereas in the source image they are not. It is again easy to see for us that this is the case, but a big challenge for computer vision algorithms. Performing color clustering is helpful to detect patches that are possible signifiers and it eases pattern recognition. As shown in Figure 4 comparing color regions for source and painting highlights regions which are then categorizable, e.g. as open nostrils, thick S-shaped eyebrows, tightened lips, hollow eyes, and closed eye lids. Let us focus further on the latter. With this additional information, on-line language resources such as curated dictionaries, idiom repositories or crowd-sourced dictionaries can be used for finding associations between labeled visual features and their symbolic associations. For example, we find that hollow eyes are associated with a skull (and hence death), fear, exhaustion and illness. Closed eyelids signify a denial of reality, hiding secrets, inward looking, emotional closure, withdrawal, no longer being open or responsive to others, but also peace and calm, resting, and meditation. These different associated concepts can be added to the TNN and their ramifications propagated. The Narrative Interpreter has to find which of these associations make sense, for example, to realize that the interpretation of closing eyelids is not that of a meditating Buddha but of hiding secrets. The interpretation of hiding secrets is enforced by the catalog text, which actually starts with the words ”The closed-off, inward-looking face of Secrets is in fact that of Albert Speer...”, which is compatible with some of the associations of closed eyelids found in knowledge graphs and dictionaries. It is also compatible with the title of the painting, i.e. ‘Secrets’. Additional queries to Wikipedia or similar resources reveal the history of Albert Speer who managed to claim ignorance of the atrocities of the war. As nodes and links expand the TNN, the

Transient Narrative networks closedeyelids

7

hollow eyes

clos edeyel ids

Fig. 4. Top: Left: Recognition of closed eye-lids. Middle: K-means color clustering for painting (left) and source (right) with K=5. Right: recognition of hollow eyes. Bottom: part of the TNN with nodes for interpretation of results of pattern-recognition as found from language and knowledge sources.

closed-eye lids appear as a key signifier of this painting, particularly because the whole exhibition has moral decay and disregard for truth as its major theme.

4

Conclusions

This paper has suggested that a key component for multi-modal narrative interpretation is a data structure called a transient narrative network that is built up by different processes: visual processing and pattern recognition, language processing to transform text into semantic structures, semantic memory contributing with general knowledge, symbolic associations and facts, mental simulation (not discussed in this paper) and episodic memory of past experiences. Much work remains: On the one hand the state of the art in all components involved has to advance significantly to achieve the subfunctions that humans effortlessly carry out, such as recognizing that lips are tight (another sign of hiding secrets). Today’s knowledge graphs, even if they already contain billions of facts, are still largely ignorant of many common sense associations required to interpret every day situations and art works. On top of that, the general management of the interpretation process is largely virgin territory for AI. We need to concentrate on operational models of attention, forgetting, selective inference, constraint propagation, etc. The study of how narratives are constructed while interpreting multi-modal texts, daily life situations, or social phenomena is therefore an important trigger for future fundamental AI research.

8

L. Steels

References 1. Aslan, S., Steels, L.: Identifying centres of interest in paintings using alignment and edge detection. case studies on works by luc tuymans. In: International Workshop on Fine Art Pattern Extraction and Recognition (FAPER 2020). Proceedings of the International Conference on Pattern Recognition (ICPR) Part III. vol. LNCS 12663. Springer Verlag Berlin (2021) 2. Bruner, J.: The narrative construction of reality. Critical Inquiry 18(1), 1–21 (1991) 3. Donnadieu, M.: La Pelle, exhibition guide. Palazzo Grassi, Venice (2019) 4. Englemore, R., Morgan, T.: Blackboard Systems. Addison-Wesley, Englewood Cliffs, NJ (1988) 5. Gervas, P.: Computational approaches to storytelling and creativity. AI Magazine 30(3), 49 (Jul 2009) 6. Goshtasby, A.A.: 2-D and 3-D image registration: for medical, remote sensing, and industrial applications. John Wiley & Sons (2005) 7. Minsky, M.: Semantic information processing. The MIT Press, Cambridge MA (1969) 8. Mitchell, M.: Artificial intelligence hits the barrier of meaning. The New York Times (2018) 9. Nowak, A., Lukowicz, P., Horodeci, P.: Assessing artificial intelligence for humanity. IEEE Technology and Society Magazine. (2018) 10. Riedl, M.O.: Computational narrative intelligence: A human-centered goal for artificial intelligence. CoRR abs/1602.06484 (2016) 11. Schank, R.: Dynamic memory: A theory of reminding and learning in computers and people. Cambridge University Press, Cambridge Eng (1990) 12. Schank, R., Abelson: Scripts, Plans, Goals, and Understanding: An Inquiry into Human Knowledge Structures. L. Erlbaum, Hillsdale, NJ (1977) 13. Steels, L.: From audio signals to musical meaning. In: Miranda, E. (ed.) Handbook of Artificial Intelligence for Music. Springer Verlag, Berlin (2021) 14. Steels, L., Wahle, B.: Perceiving the focal point of a painting with ai. case studies on works of luc tuymans. In: 12th International Conference on Agents and Artificial Intelligence. Scite Press, Setubal, Portugal (2020)