National Gallery Virtual | ETS 5 Thesis | Tom Chan

National Gallery Virtual: Extrapolating Curation, Space and Display in the age of AI

ETS 5 Thesis

AA School of Architecture

2022 - 2023

Tom Chan | Diploma 6

Statement :

The project proposes a new form of museum curation and viewing. It creates an augmented reality journey that traverses the traditional experience, using Machine Learning image generation and sorting.

Technical Project Proposal

Who curates the dataset?

The designer will propose criteria based on his/her observation of the individual work and then delegates the task of automation to the machine. The machine can learn to curate the dataset by training paired or unpaired data.

Individual Work

Criteria 1

Dataset of Work

Criteria 2

Proposed Criteria

Criteria 3

Machine

Criteria 4

Sorted Clusters

Designer

Existing Big Data Curation Methods

Curation Case Study 1: Lev Manovich

Curation Case Study 2: Andrew Kovacs

Curation Case Study 3: Refik Anadol

These curators or artists use different criteria such as formal values of the images to create the rules for sorting the dataset.

Phototrails - “Montage”

Phototrails - “Montage”

Method 3 : Meta-data and Provenance

Metadata and provenance are both important concepts in data management and analysis.

Metadata refers to data that provides information about other data. It can include information such as the data type, format, source, creation date, author, and description of the data. Metadata can be used to help manage and organize data, as well as to provide context and information to users who are analyzing the data.

Method 4 : AI Semnatic Meaning

Semantic meaning is a key concept in artificial intelligence (AI) and refers to the meaning or interpretation of language or other symbols. In AI, semantic meaning is used to enable machines to understand and process natural language and other forms of communication. AI systems can understand semantic meaning is through the use of natural language processing (NLP) techniques. NLP involves analyzing and understanding human language by breaking it down into its component parts, such as words, phrases, and sentences.

Prompt Text-to-Image Image

Stable Diffusion Clip Model Method

“A painting of a lady standing in a landscape, sky in the background”

Stable Diffusion Clip Model to define semantic Meaning

Stable Diffusion Clip Model to define semantic Meaning

Experiment 1 : RGB Sorting

Images are downloaded from National Gallery Online Archive. And then the Images are sorted through the RGB Value of the image, on average. Rhino grasshopper is used to calculate the average RGB value of image and then to sort them according to their XYZ position.

Experiment 1 : RGB Sorting

RGB values can be used as features for clustering images using the K-means algorithm. K-means clustering is an unsupervised machine learning algorithm that partitions a dataset into K clusters based on their similarity. The algorithm works by first randomly selecting K centroids (representative points for each cluster) and then iteratively updating the centroids to minimize the distance between the data points and their assigned centroid.

The process of clustering and converging

Experiment 2 : Curve Complexity Sorting

Curve complexity sorting involves sorting curves based on their complexity, where complexity is a measure of how many points or segments are needed to approximate the curve. Here’s a possible approach to sort curves based on their complexity: Define a measure of complexity for curves. This measure can be based on the number of points, the number of line segments, or some other measure that captures the complexity of the curve. Extract the curves from the image or dataset of curves. Compute the complexity measure for each curve. Sort the curves based on their complexity measure, in ascending or descending order.

Image Image Trace Crv Sort Crv Length

Image Trace

Sorting / Graph Plot

Grasshopper to represent and plot the result.

Tracing Component in Grasshopper

Experiment 2 : Curve Complexity Sorting

Curve complexity sorting involves sorting curves based on their complexity, where complexity is a measure of how many points or segments are needed to approximate the curve. Here’s a possible approach to sort curves based on their complexity: Define a measure of complexity for curves. This measure can be based on the number of points, the number of line segments, or some other measure that captures the complexity of the curve. Extract the curves from the image or dataset of curves. Compute the complexity measure for each curve. Sort the curves based on their complexity measure, in ascending or descending order.

Image

Painting

Graph of curve complexity

Image traced images vs original images

Experiment 3 : Using National Gallery’s Dataset

AI systems can understand semantic meaning is through the use of natural language processing (NLP) techniques. NLP involves analyzing and understanding human language by breaking it down into its component parts, such as words, phrases, and sentences. By analyzing the structure and context of language, NLP algorithms can infer the semantic meaning of the language and translate it into a form that machines can understand.

2600 Images

Stable Diffusion WebUI

Python Script

Grasshopper for Plotting

Interrogate Clip Model

Text Prompt in CSV

Experiment 3 :

Semantic Clustering Algorithm

Natural language processing (NLP) can be combined with K-means clustering to automatically cluster text documents based on the similarity of their content. One approach is to use K-means clustering to group documents into clusters and then extract keywords or tags from the documents in each cluster.

This approach can be used for various NLP tasks, such as topic modeling, sentiment analysis, and document classification. For example, clustering news articles using this approach could result in clusters representing different topics, such as politics, sports, or entertainment. The extracted tags can be used to summarize the content of each cluster and provide a quick overview of the topics covered in the documents.

Experiment 3 : AI Generated Semantic Tags

One potential limitation of this approach is that it relies on the quality of the clustering and the accuracy of the tag extraction. Therefore, it’s important to evaluate the results and adjust the parameters and techniques used as needed to improve the clustering and tag extraction.

Experiment 3 : Representing Semantic Clusters

Once the text data has been clustered and tagged using K-means, we can plot the results as a graph to visualize the relationship between the clusters and the tags.

One way to do this is to create a network graph, where each node represents a tag or a cluster, and the edges represent the connections between them. The size and color of the nodes can be used to indicate their importance or frequency.

Import Image = 600

Import Image = 200

Import Image = 500

Import Image = 100

Import Image = 3500

Import Image = 10

Import CSV File

Grasshopper script for plotting semantic clusters

Grasshopper script for plotting semantic clusters

Experiment 3 : Representing Semantic Clusters

Once the text data has been clustered and tagged using K-means, we can plot the results as a graph to visualize the relationship between the clusters and the tags. One way to do this is to create a network graph, where each node represents a tag or a cluster, and the edges represent the connections between them. The size and color of the nodes can be used to indicate their importance or frequency. and the thickness or color of the edges can be used to indicate the strength of the connection between them.

Cluster Value = 0

Cluster Value = 0.2

Cluster Value = 0.4

Cluster Value = 0.6

Cluster Value = 0.8

Cluster Value = 1.0

Images clustered based on their semantic tags

AI Sorting Criteria

There are two criteria that are related to the formal qualities and one related to the conceptual quality. RGB or colour palette and curve complexity are the formal values of the images and semantic meaning is the conceptual quality of the image.

Human Curation

Designer

Sorted Clusters Narrowed Down Sequence

Testing Methods from 2D to 3D

Converting a 2D image to a 3D image using artificial intelligence (AI) involves a process called depth estimation or depth map prediction. Depth estimation involves predicting the distance between the camera and the objects in the image to create a 3D representation of the scene.

3D inpainting Mesh

Experiment 1 : Pseudo-2D Image Sequence

Converting a 2D image sequence to a 3D model involves a process called “structure from motion” (SfM) which uses computer vision techniques to extract 3D information from 2D images. SfM algorithms are based on the principle of triangulation, which involves calculating the position of a point in 3D space by using the angles of two or more lines of sight from different positions.

Frame 1

Frame 9

Frame 5

Frame 13

Frame 4

Frame 8

Experiment 1 : Pseudo-2D Image Sequence

NeRF (Neural Radiance Fields) is a state-of-the-art method for synthesizing novel views of a scene from a set of input images. However, it is important to note that NeRF is not designed to resolve camera angles solely from AI-generated images. NeRF requires a set of input images that capture the scene from different viewpoints in order to learn the underlying geometry and appearance of the scene. In other words, NeRF is limited by the information contained in the input images, and cannot generate views that were not originally captured.

Proposed workflow for Generating 3D space

NERF fails to resolve camera angle only from AI images

Attempts to translate zoomiing images to 3D scan

Zooming Images NERF 3D .obj model

Neural Radiance Fields for View Synthesis (Source: Paper with codes)

Frame 19 Frame 20

Series of Zooming Images

Frame 17

Experiment 3 : Z-depth Method

Compared to manual 3D projection, using a Z-depth map method can have some advantages in terms of accuracy and efficiency. Z-depth maps allow for more accurate placement of objects and elements in 3D space, as they capture depth information from the original 2D image.

Additionally, Z-depth maps can be used to create more realistic and accurate depth of field effects, where objects in the foreground or background are blurred to simulate the way the human eye focuses on certain areas.

Experiment 4 : 3D Inpainted Mesh Method

The 3D inpainted mesh method is a technique used to generate 3D models from 2D images. The process involves using a neural network to fill in missing parts of an object in a series of 2D images, resulting in a complete 3D model.

The basic principle behind the 3D inpainted mesh method is to take a series of images of an object from different angles, and then use these images to create a 3D model. However, in order to create a complete 3D model, it is often necessary to fill in missing parts of the object that may be obscured in some of the images.

Camera

Camera Camera

Experiment 4 : 3D Inpainted Mesh Method

To accomplish this, a neural network is trained to “inpaint” missing parts of the object by predicting what the missing parts should look like based on the surrounding pixels. The resulting 3D model is then created by combining the complete 2D images with the inpainted parts.

Text-to-Images

Z-depth Map

Experiments from 2D to 3D Summary

Success

Room for Improvement

N/A

It generates seemingly convincing 3D geometry when viewed from a certain angle.

The photogrammetry method failed to generate any 3D mesh.

It only works with simple mesh like boxes and it requires manual effort.

Displaced mesh is automatically generated that describes the depth of the space. It reveals hidden illusion of an image.

It is only convincing when viewed from a certain angle. Otherwise, distortion appears.

It generates seemingly convincing 3D geometry when viewed from multiple view angles.

It is still not a fully 3D mesh, therefore it still appears distorted from view angle far away.

Method 2

Method 3

Method 4

Method 1

Experiment 5 : Generating 2.5D Painting

The 3D inpainted mesh method can be a useful tool for museums and galleries to enhance the exhibition experience for their visitors. By using this method, they can create 3D models of artworks, sculptures, and artifacts, which can provide a more immersive and interactive experience for the visitors. The 3D models can be used to create virtual exhibitions, which can be accessed online, allowing people to view the artwork from anywhere in the world.

Experiment 5 : Limitation of 2.5D Painting

The 3D inpainted mesh method also has some limitations. First, it heavily relies on the accuracy of the initial 2D image and the inpainting algorithm used to fill in missing information. If the image has low resolution or incomplete information, the generated 3D model may not be accurate or complete. Additionally, the 3D model generated through this method may not accurately represent the true physical properties of the object, such as material or texture.

Experiment 5 : Analysis of 2.5D Painting

Sections are cut to understand the missing parts that the machine interpolate. Another limitation is the computational resources required to generate the 3D model. The process can be computationally intensive, especially for complex or large-scale models, which may require specialized hardware or longer processing times.

Diana and Callisto Paul Bril

Rome: The Interior of St Peter’s Giovanni Paolo Panini

A Dutch Ship coming to Anchor Willem van de Velde

Cognoscenti in a Room hung with Pictures Flemish

Peasants driving Cattle and Sheep Jan Wijnants

A Woman Drinking with Two Men Pieter de Hooch

The Story of Papirius Domenico Beccafumi

The Market Place and the Grote Kerk at Haarlem Gerrit Berckheyde

Venice: A Regatta on the Grand Canal Canaletto

Interface Proposal : AR Experience in Museum

Curator will curate the sequence of paintings, based on the sorting AI’s suggestion. And then designer will design the spatial experience of the AR gallery. The experience is through the use of AR portal like glasses or phone.

Curated Sequence

Physical Gallery

AI + Architect

Curator

Painting Collage

National Gallery Interior Room

AR Gallery Portal (AR Glass / Tablet)

AR Space

Geometry of Anamorphosis

In a perspective projection, the representation of a 3D scene is generated by projecting rays from the camera’s viewpoint through each pixel on the image plane and onto the objects in the scene. When a ray intersects an object in the scene, it determines the color and depth of that pixel. The intersection point of the ray with the object’s surface is then projected onto the image plane, producing the final 2D image.

Geometry of Anamorphosis

It is possible to have different geometries while the projection remains the same. This is because the projection only describes how a three-dimensional object is mapped onto a two-dimensional surface, while the geometry of the object itself is independent of the projection.

Geometry of Anamorphosis

Anamorphosis is a type of perspective drawing that involves distorting an image so that it can only be viewed accurately from a specific vantage point or through a specific optical device. It is a form of geometric projection that distorts the image to create a new, unexpected form. The geometry of perception study on Borromini Galleria Spada can be seen as a type of anamorphosis, as it involves the use of geometric projections to create a 3D representation of a 2D image. Both techniques rely on the use of geometry to create distorted or unexpected images.

Proposed Immersive Gallery Room

The design can be broken into 2.5 D painting generated by AI, and designer will add AR virtual wall or floor in the spatial scale, to show depth and rhythm. Physical visitors will also traverse the AR environment. This is possible because the AR can track the observer’s position and project the respective image.

AR Tablet

Camera AR Glasses

2.5D Painting AR Virtual Wall / Floor

Physical Visitor

The Immersive Room (Projection)

Matching Paintings’ Perspective by Anamorphosis

The method is discovered by trial and error. First, the vanishing point is matched in Rhino’s viewport. However, when shifting the horizon, the camera angle changed. There should be a way to maintain the vanishing while shifting the horizontal geometry.

First, the vanishing point is matched in Rhino’s viewport.

However, when shifting the horizon, the camera angle changed.

There should be a way to maintain the vanishing while shifting the horizontal geometry.

The geometry is then shifted on the plane of projection, known as lens shift.

Proposed Virtual Gallery Elements

An immersive gallery room can be proposed that combines AI-generated 3D paintings with a virtual floor and wall, providing visitors with an immersive and interactive art experience. The virtual floor can be designed to create the illusion of depth and enhance the immersive experience, allowing visitors to feel like they are walking through the art.

AR Elements (Floor / Colonnade / Wall / Furniture)

AR Elements (Floor / Colonnade / Wall / Furniture)

3D Painting

3D Painting

Immersive Room

Immersive Room

Proposed AR Gallery Room

The distorted elements match the perspective of the 3D painting, but they sit in the deepened geometry of the painting, creating a sense of depth.

3D Painting

AR Element (Before Projection)

AR Element

Physical Walkway

Camera Plane

Physical Viewing Area

Simulation of AR Gallery Room

Once the virtual room and artwork have been placed, the 3D model could be exported to an AR software platform like Unity or Unreal Engine. Users could then use their smartphone or tablet to view the AR gallery room and interact with the artwork. They could walk around the virtual room and view the paintings from different angles, or zoom in on specific details of the paintings.

AA School - XR Room

AR Immersive Room

Virtual Immersive Room

Technical Workflow for AR Simulation

Polycam scanning, Blender, and AR are three powerful tools that can be used to create immersive experiences for museum and gallery visitors. AR technology allows for the seamless integration of these 3D models into the real world, creating a fully immersive and interactive environment for visitors to explore.

Digital Modeling

Site Mapping / 3D Scan

Blender Polycam 3D Scanning

AR Testing / Tracking Deployment

Unreal Engine

Smartphone AR

Visitor

Polycam

Preparation for AR Simulation

To prepare for an AR simulation, one option is to use a Polycam scanning device to capture high-quality 3D scans of physical objects or spaces. These scans can then be imported into software such as Blender, where they can be further refined and combined with other digital assets to create a complete virtual environment.

Point Cloud Model

Phototrails - “Montage”

Phototrails - “Montage”

Phototrails - “Montage”

AR Experience Generation

Once the virtual room and artwork have been placed, the 3D model could be exported to an AR software platform like Unity or Unreal Engine. These platforms allow developers to create AR applications for smartphones or tablets that can display the virtual gallery room and paintings in a real-world environment.

Tracking Data

Captured Image

Colorkeyed Matte

Render

Rendered Image

3D Model

Masked Image

Tracking AR Trackers

Resulting Image

Simulation 2 : Multi User Scenario

User using AR Smartphone

“A Woman Drinking with Two Men” - Pieter de Hooch

“ Cognoscenti in a Room hung with Pictures” - Flemish

“A Woman Drinking with Two Men” - Pieter de Hooch

Acknowledgement

I would like express my gratitude to :

AA AV Teams for providing space of testing AR simulation

Alan Harries for creative exhibition references

Anna Font-Vacas for constant support in design and technical matters

Camila Rock for presentation and booklet advice

Francesco Anselmo for technical advice regarding AR and Hologram

Giles Bruce for giving advice in interim jury

Javier Castanon for philosophical investigation

Nacho Marti for technological advice and inspiration

Omid Kamvari for setting up a propositional thesis

Patricia Mato-Mora for clarifying design position

Peggy Yu for filming AR simulation

Sho Ito for setting up a propositional thesis

Tom Raymont for advice on musuem spatial design

Verdi Tsui for filming AR simulation

Zach Chan for technical coding support