Portfolio_Real to Virtual

Portfolio

Real to Virtual

To optimize interfaces and interactions between reality and virtuality Selected Works 2017-2021

Apply for XR Interaction Designer Xinyi Zhou

Content Virtual World

is a new form of human civilization, including a new supply-demand virtual content relationship, continuous interactions, and multifunctional users, which carries past memory, recent experience, and future imagination. It is a space that combines virtuality and reality and communicates through terminal equipment. In the past 20 years, 2D interface personal computers (PCs) and mobile terminals have dominated information transmission.

With the development of Extended Reality(XR) technology and head-mounted displays(HMDs), people access Information, learn knowledge, and meet friends in the virtual space through the 3D mobile interfaces or these HMDs. The new interface allows human experience moves from indirect towards direct, static towards dynamic, fragmented towards continuous. Such an innovation motivates users to freely generate(create) and consume(experience) customized content in virtual space. In this process, users have escaped from the traditional role of the viewer and become a hybrid role that integrates content production, experience design, and platform construction. Therefore, such a brand-new system is subverting our traditional self-identity, social networks, and human civilization, giving us unlimited possibilities for future life.

01 Toward a New Era: Virtual Gallery - Including Pingshan Virtual Gallery & Virtual Gallery at Sansar [VR, Metaverse Platform]

02 "Treasure Hunting" [AR, App, Photogrammetry]

03 Remenbering Urban Village -The written Paper has been submitted to CAADRIA 2022 Conference and is under review (As 1st author) [Photogrammetry, Panoramic video, AR, NVIDIA CloudXR technology]

04 VR Offline Entertainment Experience Module [0-1 Product R&D]

05 AR Cube [AR, App, Object Tracking]

06 Interplay between the virtual and physical heritage [Hololens, Arduino]

07 Meta-Emo [Deep Learning, Pix2Pix GAN, App]

08 Taida AR Exploration [AR, App]

09 Shenyang K11 Kaos VR Experience Space Design [VR Offline Experience]

10 Exhibition: 2021 Asia Digital Exhibition Expo [VR Product; Product Display & Sale]

11 YOMOV VR Experience Hall [VR Offline Experience; Spatial Design]

Toward a New Era: Virtual Gallery

New attempts with two virtual gallery including Pingshan Virtual Gallery & Virtual Gallery at Sansar Location: Shenzhen, China Date: 08/2020 - 12/2020 & 06/2021-08/2021 Team members & Role: Each part has a specific description The form of Contemporary art can be diverse. The transformation of art in different spaces will give the audience a different experience and inspire them to think about the meaning of work from another perspective. Technology is one of the most important labels of Shenzhen. This young city has the best soil for new media art due to its advanced high-tech manufacturing abilities in this background. When the COVID-19 ravaged the world in the first half of 2020, the past life patterns of human beings were affected entirely and changed. This project discussed how traditional art could break through the limitation of space and region to continue the dialogue with the audience.

Virtual reality technology and Metaverse platform may be the answer.

Social attention & impact

Physical & virtual art communities

Virtual Exhibitions

Artworks

Digital version preservation

Physical Exhibitions

Conceptual drawing by Mengqi Tu

s

n tio

to bi ct xhi tra l e t a A tu vir

Virtual exhibition open to audiences on an ongoing basis

Attract to physical exhibitions

Physical & virtual art exhibitions Digital Art Trading Physical Art Trading

Research: Research on the history and case studies of virtual gallery on the metaverse platform

Problem caused tbal pandemic of Covid-19 In 2020, The Covid-19 global pandemic led to the closure of offline public places one after another, and galleries are no exception. Activities that people are keen to socialize in art galleries and museums, participate in workshops, and appreciate works of art have also been canceled.

I hope that more people can see our works of art and participate in more exchange activities.

Different types of art galleries

How should our museum community operate due to the closure of the pandemic?

What kind of activities can keep the museum alive?

Main Technologies -Physical Metarial

Features

Pros and Cons

-Construction

-All exhibits are placed in the physical exhibition hall -Monotonous presentation

-Transportation

-Limited information resources

Really need a way to see good artworks

Cons: -Strong sense of communication

-High topic certainty

Head of Art Museum -Audio, video, games, animation, etc.

-VR technology enhanced visual effects -Huge variety of information resources -Thematic diversity

-Gesture Tracking -Image Processing

Artwork collector

-Scene understanding -Input technique -Mobile headset It’s really sad to lose my friends in the museum community. I hope that the museum’s operations can be restored as soon as possible.

Artist

Participants related to the museum put forward their own new ideas, and hoped to find innovative solutions in the constant discussion.

Let me find a way out!

Internet

-Edge computing

-Audio, video, games, animation, etc.

-Various interfaces, various visual effects and new social connections -Tradable content

-5G Network

-Iteratively user-generated content

-Extended Reality -Telepresence

Research: Research on the history and case studies of virtual art museums Mobile APP

Augmented Reality Virtual Reality

Pros: -Uncomfortable Device -High requirement of network Cons: -Unlimited Space -Feeling of Presence

-Automatic digital twin -Blockchain

Designer

Visitor

CD-ROM

Pros: -Content is difficult to keep -Not universal

Pros: -Not yet mature -High requirements for various technologies

Cons: -Strong sense of social connection -Secure virtual trading system

Proposal: Tripartite cooperation of two virtual art museums + 2Proposal Metaverse Financial Support

Artist

A three-dimensinal computer-generated museum comprising an immaterial constellation of rooms and exhibits

A virtual museum of science, humanities, and culture that uses infomation, virtual experiments, and handson activities

1991

1999

App utilizes mobile phone devices in order to enrich the visitors experience in the museum

2013

An "Interpertive Gallery" that allows visitors to explore Magritte's "the paradoxes at work in reality".

2018

Features 74 Dutch and Flemish Old Master paintings from the Dutch Golden Ager in a fantasy space exists only in VR

Virtual artwork presented in VR

2021

Metaverse Platform

Virtual Art Gallery

Collector

Designer

DSL Collection

Timeline Digital museum/gallery

Coperation

Visitor

A place where galleries are able to connect with international audiences in real time.

2018

Museum

Physical Art Gallery

Proposal: Virtual Art Gallery by VR 2D Artworks

Platform: Unity Engine

Proposal: Virtual Art Gallery at Metaverse

3D Experiences

2D Artworks

Platform: Sansar

Interactions

Interactions TATE Modern

National Palace Museum

Modigliani VR: The Ochre Atelier

Mona Lisa: Beyond the Glass

Roblox

Fortnite

Pingshan Gallery

Human

Technology

Virtual Gallery

Multiplayer Muséum national d'Histoire naturelle

Kremer Museum

The Scream

The Spirit of Autobiography

Decenterland

Sansar

Physical

Interface

3D Experiences

Interface

Avatar

Narrative Virtual

Virtual Gallery

Physical

Virtual

Spatial Sequance | Pingshan Virtual Gallery Virtual Gallery Entrance Hall

Pingshan Virtual Art Gallery

Virtual Exhibition

Virtual Exhibition Interaction

6 Choose a showroom

1

7

3

2 Offline VR experience

Experience sequance

5

4

Enter the buffer gallery of the virtual art gallery to give users t i m e t o prepare

The virtual art gallery exhibition hall built at Pingshan Art Museum is equipped with three Meta Quest 2 devices, which can support multiperson experience and is the physical entrance of the virtual art gallery.

Specific exhibition s p a c e , taking the work "Meat" a s a n example

Exhibition h a l l entrance platform: users can choose their favorite exhibition to go

Spatial Sequance | Virtual Gallery at Sansar Sansar Hall

Virtual Exhibition

Virtual Gallery Entrance Hall

Transfer space

7 3

4

Original Artwork

Entrance of single exhibition

5

1 8

2

Exhibition Hall

Entrance of DSL Collection

Sansar Experience C e n t e r, f o r u s e r s to choose their favorite activities and participate

The entrance hall of the virtual art gallery provides space for users to communicate and discuss (Virtual social). Eight exhibition halls of different works are waiting for users to explore

6 Back to Entrance Hall

The eight virtual spaces are all designed according to the same space experience system, including spaces with Original artwork space, Transfer space, View platform, and Main exhibition and other functions

Production of Pingshan Virtual Gallery STEP2: 2D Artworks to 3D Virtual Experience 3D Virtual Experience:

Pingshan Virtual Gallery

Re-creation of eight independent physical works of art into the content of the virtual exhibition hall, creating a virtual art gallery through the production of new visual systems and interactive methods

An Offline Virtual Gallery; Walkable Virtual Experience; New Form of Artwork Exhibited Way Location: Shenzhen, China Date: 08.2020 - 12.2020 Team members: Xinyi Zhou, Liquan Liu, Rufeng Xiang, Mengqi Tu Role: 90% Pingshan Virtual Gallery Design, Group Technology deployment, Group Virtual Exhibition Production

Jump into next work

STEP3: Interaction

Entrence

Interaction: Teleport： Human-interface interaction: For example, in this experience, the user can hit a 3D object to make the object produce a unique movement trajectory

Audio & Text: Virtual Exhibition Unity

VR

Digital Preservation

Offline Virtual Gallery Walk into the mind of artists

Artworks

Audio:Text introduction voice, BGM, etc., rendering a better immersive experience Te x t : G i v e t h e w o r k a b a s i c background introduction

Physical art communities

Physical art Trading

Physical Exhibition: Cooperate with Pingshan Art Museum to build the offline exhibition hall entrance of the virtual museum

STEP4: Communication ta

Production of Pingshan Virtual Gallery

va -A

n

tio

ac

er

nt rI

tar

Human-Avatar Interaction

STEP1 | 2D Artworks: Eight Chinese artworks from DSL Collection

a Av

Wireless VR streaming 4 1

5

6

2D Paintings

7

Related Arkworks:

2D Videos

In the context of the global pandemic of covid-19, collectors of DSL Collection hope to bring back the original collection of artworks (in the form of oil paintings, videos, etc.) to people in a new way.

Technology Support | Wireless VR Streaming Data Cable

Successful Streaming

PC

2

Oculus Quest 2

3 8

The eight virtual spaces are all designed according to the same space experience system. The set experience sequance will be presented one by one with the narrative experience

Production of Pingshan Virtual Gallery

Pingshan Virtual Gallery - Multiplayer experience

In-site photo of Virtual Gallery Meta Quest (wireless streaming) uses positioning cameras to identify the environment and assist in positioning, which leads to the need for visually different designs on the walls around the exhibition hall so that the headset is easier to locate.

Meta Quest 1 / Meta Quest 2

Positioning camera

Layout and technical deployment

VR experience area Cable

WIFI x 2 Headset area

1

2

3

4

Display The server

Data cable

Waiting area

Virtual experience sequence 1

2

3

4

Pingshan Virtual Gallery - Multiplayer experience

Interaction: Animation

Trigger

Virtual Gallery at Sansar A new experiment in the Metaverse platform: exploring new forms of existence and trading methods for artworks Location: Sansar, Metaverse Date: 06/2021-08/2021 Team members: Xinyi Zhou, Liquan Liu, Fukai Chen, Junxiao Ma Role: 65% Production of Virtual gallery at Sansar, 40% Virtual exhibition design at Sansar Virtual Exhibition Sansar

Sansar

Online Virtual Gallery

Digital Preservation

Other virtual events

Artworks

Interactive Target of Entrance

Entering from the exhibition hall to each exhibition sets interactive a n i m a t i o n c o n t e n t , prompting the user to enter a new exhibition

Entrance of Each Exhibition & Interactive Target

UI Design G u i d a n c e system: including UI design of origital artwork, transfer space, main exhibition, viewing space

Virtual art communities

Center Space with UI Guide Virtual art Trading

Single Exhibition With eight recreated artwork experiences i n a r o w, t h i s section takes the shock of time as a example to explain what kind of interaction , text, sound, and animation have integrated in the space.

Production of Virtual Gallery at Sansar Virtual Gallery Entrance Hall

Artworks from DSL Collection

1

Entrance of exhibition hall

2D Videos Artworks

Recreate at Sansar platform

Animation

Recreated Artwork - Take the shock of time as example

2

In the context of the global pandemic of covid-19, collectors of DSL Collection hope to bring back the original collection of artworks (in the form of oil paintings, videos, etc.) to people in a new way.

2D Painting Artworks

Entrance of each exhibition

Teleport Space

Transfer Space with Gates

T h e s e t experience sequance will be presented one by one with the narrative experience

Cases of Virtual Exhibition at Sansar | Responsible for Spatial Architecture Design

Exhibition Hall | Main Designer: Liquan Liu

The Engine | Main Designer: Fukai Chen

Flirt | Main Designer: Xinyi Zhou

The Big Era | Main Designer: Liquan Liu

"Treasure Hunting" Build up two experience systems for architectural heritage - Take Zhou Family Courtyard as example Location: Yongzhou, China Date: 11/2020-03/2021 Individual Work

With the loss of the population and the intensification of aging, the idle and dilapidated indoor and outdoor spaces in the Zhou family Courtyard are increasing yearly.

Precious buildings and non-cultural heritage are gradually declining. Problems such as lack of protection, single transmission channels, insufficient funds for repairs, and lack of protection awareness among local residents have become more apparent.

The purpose of this project is to use the portable and exciting features of AR technology to disseminate and preserve cultural heritage and try to spread the cultural content of the Zhou Family Courtyard in the concept of "4D museums" locally and remotely so that more People participate in the protection of architectural heritage.

Triangulation

HERITAGE DATASET

IN-SITE System

Windows x 3

Photogrammetry

Vuforia

Door x 3 Wall x 3

ROMOTE System

Historical development of Zhou Family Courtyard

Game Procedure

Architectural Heritage Zhou Family Courtyard are in disrepair for a long time

The Old Courtyard, the earliest of the six existing courtyards was built during the Jingtai period in the middle of the Ming Dynasty (1450-1500).

1450

The New courtyard, first built in 1841, Daoguang, the 21st year of Qin Dynasty, covers 4000 square meters in the floorage.

1638

1841

1611

The Four Family Courtyards consist of Central Courtyard, Old Courtyard, Academy and Defensive Tower. It was built from the 30th year of Guangxu to the early years of the Republic of China (1904-1912).

1904 1894

Timeline

Based on the historical timeline, collect data on the three significant elements of doors, windows, and walls in the Zhou Family Courtyard

"Treasure hunt" map

Three levels related to WINDOWS

Unity Engine as a platform for resource integration and AR content production

Three levels related to DOOR

Tourist Ta r g e t u s e r : To u r i s t s .

Design and develop from the tourists' perspective

Three levels related to WALL

Use Vuforia to make AR content presentation

"Treasure hunt"

Solve puzzle

Scan target

Reward

Solved Animation: Structure explosion

The Red-door Mansion, first built in Wanli of Ming Dynasty (1611-1628), was built by Zhouzuo's younger son, Zhou Xisheng.

The Black-door Mansion, first built in Tianqi of Ming Dynasty (1638), was built by Zhou Xisheng's oldest son, Zhou Zisu.

The Ziyan Mansion named by Zhou Chongfu was built in Guangxu of Qing Dynasty (1894-1902) by his sons and nephews.

Current problems in Zhou Family Courtyard

Successfully solve the puzzle, open the next level

Virtual Button

Physical to Virtual (Data Collection) Point Cloud - Model - Targeted by Vuforia

The old houses are in disrepair for a long time, and the architectural heritage does not have enough funds for protection

The house that grandpa left behind can no longer be the owner. The young people in the village no longer have reliable labor after they go out.

As a well-known heritage tourist area, the guide sign system in the village is very old and unclear, making it difficult for tourists to obtain effective information and unable to have a good understanding of the architectural heritage.

Clear all levels

AR

AR APP | UI Design & Two System

Landscape screen

"Hunting Treasure" brings vivid digital heritage to visitors through lightweight and pervasive mobile devices, creating a new concept of "4D Museum". The boring and monotonous knowledge of ancient buildings is also better spread with this AR Game.

Rewards by completing all levels of the game will promote tourists' desire to explore and compete in the game. The setting of two different experience systems enables more users to participate in the reflection and discussion of the heritage proservation.

The final rewarded 360 panoramic video also leads the viewer to further understand the relationship between the three elements of doors, windows and walls in the indoor environment.

“Hunting Treasure” In-SITE System

"Hunting Treasure" begin!

In the doors and windows of Ming and Qing Dynasties, a large number of plant patterns appeared. Trees and flowers are often used in ancient auspicious patterns. The ancients gave life to plants and paid attention to the inherent qualities of various plants.

Hunting

Introcution

Seeking

Animation

Interaction

REMOTE System

In Song Dynasty, it was called waist board, and "Building Fayuan" was called Jiatang board. According to Liang Sicheng, the double ring board was called Jiatang board.

Hunting

Introcution

Environment

Animation

IN-SITE | Game Procedure

IN-SITE | User Experience

STEP1: HUNTING Getting hidden introduction by clicking different site icons on digital map

STEP2: GENARATION Generate the corresponding digital model f r a m e s b y scanning the AR markers found

STEP3: HINT Using virtual button technology to show that how many pieces of the puzzle composed of this object

IN-SITE | User Experience

The support window is a kind of window that can be raised and taken off.

STEP4: PUZZLE

The upper section can be pushed out and the lower section can be taken off.

Use virtual button technology to show the unique information of each structure, thus guiding users to solve puzzles successfully

S T E P 5 : R E WA R D PANORAMA 360

Using camera and smartphone to catch almost 360 degrees' photos for building up 3D models

REMOTE | Game Procedure

REMOTE | User Experience

STEP1: HUNTING The Zhou Family Courtyard is located at the source of the Xianshui River, a tributary of Xiaoshui River, Fujiaqiao Town, Lingling District, Yongzhou City.

Getting hidden introduction by clicking different site icons on digital map

It is generally divided into upper and lower sections.

STEP2: Experience C l i c k t h e " Tr e a s u r e " p o s i t i o n , and the corresponding target will be displayed. By observing the relationship between "treasure" and the surrounding environment, we have a preliminary understanding of In the windows of Ming and Qing Dynasties, a large number of plant patterns appeared. Trees and flowers are often used in ancient auspicious patterns. The ancients gave life to plants and paid attention to the inherent qualities of various plants.

on

STEP3: PUZZLE

The upper section can be pushed out and the lower section can be taken off.

Use virtual button technology to show the unique information of each structure, thus guiding users to solve puzzles successfully

On

ama

Off

STEP4: Animation

360

Use the button to control whether the "treasure" shows an explosion animation, so as to better interact with people around you

S T E P 5 : R E WA R D PANORAMA 360

Using camera and smartphone to catch almost 360 degrees' photos for building up 3D models

U s i n g camera and smartphone to catch almost 360 degrees' photos for building up 3D models

1

2

3

4

5

6

7

8

Remembering Urban Village Written as paper Remembering Urban Village, accepted by CAADRIA 2022 Conference Using CloudXR Technology as an Enhanced Alternative to Visualise Heritage Interactively Location: Shenzhen, China Date: 8/2020-4/2021 Team Member: Xinyi Zhou, Rufeng Xiang, Mengqi Tu, Haida Sun Role： 80% Innovation & Technology Plan; 100% UI Design; 50% Project landing Urban villages are strictly related to urban growth. It reflects the era characteristics and memory of a region in urban growth, which has significant value in heritage documentation and dissemination. Due to the continuous development of urbanization and the shortage of urban land, many urban villages will be replaced by more valuable functions. Therefore, how to better preserve the digitalization of urban villages and make more people understand the value of urban villages is particularly important.

However, the existing technology still has shortcomings in disseminating digital heritage. For urban villages, usually a large-scale and complex environment, the hardware requirements will be very high for high-precision visualization. Most of the existing solutions use large hardware devices, such as the virtual sand table. Unlike hand-held devices, such devices are expensive and not portable, limiting better dissemination of such heritage.

Due to the hardware limitation of hand-held devices, neither the display resolution nor the interaction effect is satisfying. Therefore, this project proposes a new workflow by NVIDIA CloudXR streaming technology to achieve high-precision visualization and a rich interactive experience on hand-held devices. Such heritage can be promoted and popularised more conveniently.

Computational Process

Documentation Photogrammetry Modeling

Digital Reconstruction

Dissemination High-fidelity

NVIDIA CloudXR Technology

Interactivity High Accessibility

Research of Chinese Urban Village

Proposal

Urban villages have become a powerful booster for the urbanization pro ce ss an d p r ov ide aff or dable housing and jobs for the low-income migration for the rapid urbanism of China over the past years.

AR

Interaction

Designer

Sandbox DOCUMENTATION

BaiShiZhou Renewal Project

TECHNOLOGY

Government

Digital Heritage

Text

Contractor

Movement

Animation

COMMUNICATION

Recall

Communication

Citizen

Ray-casting

Interior Models INTERACTION

For urban villages, usually a large-scale and complex environment, the hardware requirements will be very high for high-precision visualization.

Baishizhou Urban Renewal Project

DISSEMINATION

Child: Wow, I can see where my parents lived before, and the food at the roadside stall looks so delicious.

Contractor: It’s great to protect our urban heritage while promoting urban development.

Participants

Delegate Phase I construction Phase II construction Phase III construction Phase IV construction Phase I demolition

Government

Contractor

Win-Win

To promote urban development, carry out urban renewal.

Phase II demolition

Contractor

Phase III demolition

Economic Goals

Connecting the government and citizen, implement urban renewal plans, and obtain economic benefits.

Contractor: Presenting Baishizhou in this way, I believe it will retain more precious memories of the urban village.

Phase IV demolition

Baishizhou Urban Renewal Project

Urban Development

Government

Citizen Urban village provides them with convenient housing and affordable living needs.

Provide Housing

Designer Consider the needs and interests of all parties and seek creative solutions.

Understand needs

Daily needs

Designer

Residents

Preservation and solutions

Social Development

Development and Construction Land Scope Line

Improve the quality of life

Remove scope line

Firstly using tablet to scan target Villager Own real estate and commity network in urban village.

Main Workflow Methodology:

Computational Process:

UI Design and Presentation:

There are many methodologies for architectural heritage preservation. This

Compared with using traditional localized high computing power

NVIDIA ClouxXR technology significantly promotes the accessibility and fidelity of the content. Meanwhile, the

project uses a methodology with documentation-computation-dissemination

equipment, NVIDIA CloudXR technology can remotely control

Unity engine enables visualized information, amusing animation, sound effects, and UIUX design to be linked to

(DCD) workflow, focusing on technology to promote high accessibility, fidelity,

server equipment for data processing. It is innovative digital heritage

the digital heritage to enhance the fun and interactivity of the content.

and interactivity in disseminating digital heritage. This methodology proposes

preservation and can replace localized high computing power equipment.

UI Design:

a new perspective of the preservation of digital architectural heritage.

DCD Workflow:

both sides: the device and the server, respectively.

Documentation

Digital Reconstruction

Photogrammetry & Modeling:

Softwares:

1.UAV Oblique photography 2.Panoramic camera 3.Sketch and modeling

When using NVIDIA CloudXR technology, it needs to use devices at

1.Agisoft Metashape 2.Unity 3.Adobe PR 4.3D MAX

Computational Process

Dissemination

Technology:

Features:

NVIDIA CloudXR Technology

1.High fidelity 2.Interactivity 3.High Accessibility

SERVER

CLIENT Network

Documentation:

Seek & Scan Target

Ray-casting interaction & Swipe up

Get infomation & Swipe up again

Get Model & Move around

Data of Texture:

NVIDIA CloudXR Server

NVIDIA CloudXR Client

A. Activates the device camera's Ray casting algorithm. When the user aligns the device camera at a specific area where will be highlighted.

Outcome of Photogrammetry:

B. Slide the screen upwards: the specified animation and sound effects emerge.

C. Swipe the screen up again.

D. Based on the technical support of bi-directional audio in CloudXR, users can remotely have real-

How many devices can be tested at the same time

time communication and discussion.

Triangulation Modeling:

Click to experience again Presentation:

Test Result: Single Clean Channel

Vi d e o s t r e a m compression to 30Mbps

Single Clean Channel

Vi d e o s t r e a m compression to 30Mbps

10

30

4

12

High-fidelity urban village model

Multiplayer experience

Representation empowered by NVIDIA Cloud XR

Experiment result:

We invited 27 people to be participants in this experiment: Currently living in Baishizhou: 11 people. Previously lived there: 16 people.

Seek Target

Multiplayer experience

Digital Heritage

Interaction

UI design

Animation

Basing on the high-precision Baishizhou data collection and NVIDIA CloudXR technology for the transmission of high-precision content, the Baishizhou AR Sandbox shows excellent performance in the presentation model content, regardless of the richness or the accuracy of the model content. It exceeds the imagination of the viewer significantly. From the high-precision data collection of Baishizhou and the presentation of high-precision content, NVIDIA CloudXR technology has played a vital role in the workflow.

VR Offline Entertainment Experience Module Research & Development R&D from 0 to 1 [VR; Product experience design; Spatial Design] Location: Shenzhen, China Date: 12/2021-01/2022 Team Member: Xinyi Zhou, Liquan Liu Role： Responsible for product experience design; product production; supplier docking; product landing process; product iteration

From 0-1 design, production and construction of VR interactive game experience modules, to create a new social, high-quality and high-return offline entertainment experience. This product can set and control the number of modules according to the size and budget of the physical store.

Modelling

Rendering

Research & Development

Game modules

SALES MODEL | VR offline experience store

VR offline experience store

AR Cube A AR product to protect the lost cultural heritage Date: 10/2021-12/2021 Team Member: Xinyi Zhou, Xing Hao Role: Technology research, Innovation and technology planning, 70% XR content production As one of the oldest city in China, Beijing has the most complicated city water system, which has witnessed the development process of this historic city.

Many culture meme, such as Hutong(name of traditional streets) and Obon(a traditional a traditional Buddhist and Taoist festival), are related to the water system. However, many people do not know these knowledge and cultural significance. This project aims to popularize the knowledge and history of water system of Beijing.

Media technology can build multiple virtual worlds in parallel for different groups of interest, encouraging creation and recreation without the constraints of time, space, and social class. This project abstracted the characteristics of Shichaha, a manmade lake lies in the center of Beijing, and combined augmented reality with paper box in order to allow people experience the ancient activities related to water and arouse people's awareness of protecting the water environment.

Research & Modelling

AR Cube Product Single Object Tracking

PLAY MODE

Unity with Vuforia

Ancient Water Culture: Take Shishahai area as a example

Mutiple Object Tracking Scene Tracking (4Objects + Map)

Preservation

Dissemination Better

Product Sale Better

Biopolitics | Research on Context Beijing | Histroy | Watersystem

Yuan Dynasty [AD 1215] The city was rebuilt around the Bailiantan lake so that it is convenient for grains and materials from all over the country to the capital through water transportation.

Qing Dynasty [AD 1658]

Ming Dynasty [AD 1420]

In order to defend against enemies in the north, the Ming Dynasty moved the city walls south and rebuilt the moat on the original city.

During the Qing Dynasty, the city of Beijing followed the pattern of the Ming Dynasty and began to build gardens when the river was being treated.

Evolution of Reading Media Newspaper

E-Book

Video

1998: a dedicated e-book reading device appeared 1969: The Internet appeared in 1969

VR/AR

2007: Kindle

Metaverse

2020: Oculus quest 2 is released, making HMD more popular

2005: Youtube was founded

Timeline

2021: Immersive novel

Efficiency of Information Transfer

[Reading by AR] Physical Target

Temples Water conservancy facilities

Folk activities

Surrounding structures

Huitong Temple

Yinding Bridge

Fire Temple Dojo Obon Bathing Buddha Play on the ice Chunxu Shishahai Landscape

Making Process of Object Tracking and Cude Mode

Paper Book: 1st century BC Oracle: About 16th to 10th century BC

Guanghua Temple

Fire Temple

Landscape

Jin Dynasty [AD 1153] The governors of Jin Dynasty first made Beijing the political center, and the construction of palace gardens and the use of water became their first considerations.

Water landscape & culture

Lying in the north of the North China Plain, Beijing has abundant water resources brought by five river systems. Since 1153( Jin Dynasty), Beijing has always served as the political and cultural center of the northern regime and even the whole country.

Thus culture about the water system is also thriving.

The water system in the city of Beijing mainly provided water and landscape for the imperial city. At the same time, the flourishing of Buddhism led to the development of the water landscape into royal palaces, temples and suburban

Realm of Beijing Main river

Beijing Water Culture

Content Structure

Interaction

In many cases, augmented reality can effectively replace human imagination and place objects or devices in front of the learner’s eyes at the right moment in the training.

[Content in AR]

[Text]

[Image]

[Video]

[3DModel]

[3DAnimation]

AR to History | Intersection Method

AR Cube Product | Making Process Stone stele (recording the treatment of waters)

Huitong Temple

STEP1: MAKING

Underground sluice

Water beast

Surrounding environment

There is a large water gate under Huitong Te m p l e w i t h m o v a b l e wooden gates embedded in the stone troughs on both sides. The water level and flow rate of the entire Beijing waters are controlled. Therefore, it is called the throat of the capital waters.

X6

Underground Water Gate The production process of AR CUBE: abstract solid building facade x6 - combined into a cube

Huitong Temple

Guanghua Temple

Fire Temple Dojo

Map: Combination play mode of AR CUBE 1 Single Object Target

2 Mutiple Objects Target

STEP2: LINKING

Yinding Bridge

Fire Temple

Fire Temple Dojo

On June 23rd, all fire temples in the capital will hold grand avenues.

Di’anmen Fire Temple is the largest of the fire temples in Beijing. The Taoist priests wore robes and vestments and held law enforcement instruments or musical instruments when doing the dojo.

Single Water-related Construction

Water-related Activity

3 Scene Target

unity Use the Vuforia platform in Unity to link virtual assets and physical objects and divide them into three levels of content for presentation: 1. Single object target 2. Multiple objects target 3. Scene target. To produce a better product experience for consumers and further understand the rich Shichahai water cultural content.

Obon

Play "AR Cube" on Map

Shicha waters culture： Obon

The Buddhist community held a grand Obon Festival on the Lunar New Year's Day on the 15th day of the lunar calendar. There were mass activities such as the lighting of river lanterns and burning of boats.

Shicha Waters Culture The Shicha sea area was the terminal of the Beijing-Hangzhou Grand Canal in the Yuan Dynasty. It was the basis for the planning of the capital city of the Yuan Dynasty.

Product

Huge Folk Activity on Shisha Sea

Dissemination

STEP3: DISSEMINATION Package the designed AR Cube and Map into "AR CUBE" products for sale and dissemination. To publicize and promote the disappearing water culture of Beijing and let more people learn about and protect Beijing water culture through this exciting product.

AR APP | UI Design & Cube Play Mode

[Sanning Individual AR Cube]

This app has two main functions: 'Start' and 'About'. 'Start' is a camera mode UI, through which users could scan the cubes and the map. Then the 3D model of the sites including Huoshen Temple, Guanghua Temple, Yindian Bridge and Huitong Temple will be displayed. 'About' allows users to get the details of the sites directly.

The combination of the cubes and the map will show different folk activities related to Shichahai.

Move cube

Open Collapsed Map

Scan object

Scan Map Scan 1 Cube

By scaning individual cube on the map, users can get detailed knowledge of the architecture and its components related to shichahai and water culture.

Scan 2 Cubes

The combination of two cubes and the map will show the users the relevant cultural activities about this two buildings as well as the landscape around Shichahai.

Adjust the Focus

Scan 4 Cubes + Map

If all of the four cubes and the map are scanned, the Obon Festival, the most celebrated festival during the history of Shichahai, will be displayed.

Manual Focus

[Sanning Multiple AR Cubes]

Prototype | Usage Scenarios

Interplay between the vir tual and physical heritage Using MR technology to protect architectural heritage.

[Technologies: Hololens, Arduino] Location: Shenzhen, China Date: 11/2021-01/2022 Team Member: Xinyi Zhou, Fukai Chen, Zhouyi Li, Chaohe Lin Role： 80% Innovation & technology planning, 50% Virtual Content Production Cultural heritage contains multiple components, including tangible culture (buildings, monuments, etc.) and intangible culture (history, stories, music, etc.). The virtual and physical features together define the local identity and spirit of place. Immersive technology has much accelerated the digital transformation of heritage. Lots of researches in cultural heritage based on VR, AR, and MR have proved the feasibility. MR is defined as combining virtual elements (e.g., holograms or projections) with the real world. Compared with AR, MR can achieve a broader goal of enhancing our virtual and physical environment understanding.

t pu

In

ce

rfa

te

Hololens

Arduino In

In fe tera ed c Vi ba tiv su ck e al is at io n

We proposed a new method of interaction between the virtual and physical heritage in an MR application, aiming to break through the interplay limitations. This method is based on the integration of environmental sensors, MR technology, and cultural heritage. This integration provides an opportunity to utilize immersive and sensing technology to improve the participation and understanding of visitors.

Input Interface

Unity

Technical Test (MR devices include Hololens & Arduino)

Game Proposal Gesture interaction:

AR glass - Hololens: Display virtual contents and provide virtual interaction to users

Unclarity Low participation

Vapidity

Information platform - Unity: Connect virtual and reality and deliver information

Insufficient relationship of the interplay between virtual information and heritage artifacts

Sensing technology - Arduino： Offer physical interaction to track users' behaviours

Improve the participation and understanding of visitors to architectural heritages

Game platform:

A platform that provides promotions for MR games

Virtual heritage:

Kaiyuan Temple, Yanghe Building, Guanghui Temple, Changle Gate

MR technologies

Research

Solution

Prompt to reselect Choose wrong

Info notification x5

Physical interaction Base on on-site environment

RFID_RC522 x 5

Choose right Virtual heritages Game platform x 5 identifications Gesture interaction

Scale and move

Technical Process

RFID_RC522 x 5

Breadboard x 1

Arduino UNO x1

Finished

Game & Display Content displayed by Hololens

User Experience Experience process

Game procedure

Meta-Emo An virtual media with Pix2Pix GAN Date: 7/2020-10/2020 Team Member: Xinyi Zhou, Han Tu, Ziyu Han, Xinyi Yang Role: Conceptual Design 80%, Data mining 80%, Model training 50% Typically, people's reactions to art are considered subjective — For instance, Munch's The Scream might instill foreboding anxiety in you, but do absolutely nothing for your friend.

However, artist Wassily Kandinsky feels that abstract art, what people would perhaps consider the most emotionally subjective form of art, is entirely objective. He suggests that the emotional objectivity of abstract art lies in the characteristics of the colors and their interactions with one another.

Can the emotions expressed in pictures be recognized by computers? What characteristics of emotions in pictures can be well reflected by computers?

How does the computer add emotional processing to the picture?

This project aims to show that machine can recognize human emotions and express them through the image-to-image translation of hand-drawn sketches by conditional adversarial networks, so that the process of humancomputer interaction will treat emotions. People use art to communicate their emotions to each other, and in the same way art also heal people's emotions.

Dataset Mining & Classification WikiArt Emotions Dataset:

Classification:

There are 2800pieces of drawings collected from WikiArt dataset, which include 12

First, we split the data, using 85% of the data as the training set (and labeled

different genres such as Abstract, Impressionism, Pointillism, Post-Impressionism,

it as one of "Happy", "Angry", "Sad", "Neural" and "Fear"). The remaining 15% of the data is used as the test set.

Also, we conducted data mining on google image through tags such as "Painting", "Emotional", and "Western Painting", and finally we got a dataset of 3500 pieces.

WikiArt Dataset

Training Dataset - 85%

Data Mining

Test Dataset15%

01 | Tagging

02 | Classify ID: Test_Img_001

Fauvism, Classicism, Surrealism and Pop Art etc.

Method of Classification Sentiment scoring: ID: Test_Img_001 Anger rating:0.29 Happiness rating:0.57 Sadness rating:0.25 Neutral rating:0.35 Fear rating:0.19

03 | Identify

ID: Test_Img_001 Sentiment: Happiness

3500 Pieces Collected Dataset:

Sentiment Scoring Sentiment scoring:

Neutral

Happiness

ID: Test_Img_027 Anger rating:0.29 Happiness rating:0.57 Sadness rating:0.24 Neutral rating:0.35 Fear rating:0.18 Sentiment scoring:

Sadness

Fear

The purpose of data classification: classify various types of data sets according to their color and style, and finally classify them into five emotion categories: "Happy", "Angry", "Sad", "Neural" and Anger

"Fear".

Methods of RGB Color Extraction: Method1:

ID: Test_Img_025 Anger rating:0.32 Happiness rating:0.49 Sadness rating:0.24 Neutral rating:0.33 Fear rating:0.30 Sentiment scoring: ID: Test_Img_013 Anger rating:0.29 Happiness rating:0.35 Sadness rating:0.21 Neutral rating:0.26 Fear rating:0.46 Sentiment scoring:

Divide the three primary color dimensions (R, G, B) of the color of the picture into 4096 blocks if n=16, each block is recorded as a color block, and the rgb value of the midpoint of the color block represents the color of the color block . You can take and count the number of pixels in each color block of all the pixels of the picture, and then visualize the top 10 color blocks with the most pixels and use a pie chart to visualize them.

ID: Test_Img_0018 Anger rating:0.40 Happiness rating:0.39 Sadness rating:0.24 Neutral rating:0.32 Fear rating:0.29

Method2:

Sentiment scoring:

Each pixel of the picture, as a sample point, each sample point will fall in a three primary color dimension (R, G, B) space, according to the distance, use the method of clustering in machine learning, these The sample points are clustered into 10 categories, and all points in each category will have a centroid (or center, that is, the average value of the coordinates of each point), and then use the color corresponding to the coordinates of the centroid to color the sample points to obtain a three-dimensional space. A visualization of the distribution.

ID: Test_Img_042 Anger rating:0.29 Happiness rating:0.33 Sadness rating:0.41 Neutral rating:0.34 Fear rating:0.25

Color Cluster

3D distribution visualization

Pix2Pix Model Training & UI Design 3D Distribution Visualization Results

Pix2Pix Model Training

Emotion Express App

Each pixel of the picture, as a sample point, will fall in a three primary colour dimension (R, G, B) space, according to the distance, using the method of clustering in machine learning. These sample points are clustered into 10 categories. All topics in each category will have a centroid (or centre, that is, the average value of each point), and then use the colour corresponding to the coordinates of the centroid to colour the sample points to obtain 3D distribution visualization results.

Generate

Sketch Outline of Phone

Training Result of Classification: Sentiment Scoring Sentiment scoring:

Color Cluster

3D distribution visualization

ID: Test_Img_050 Anger rating:0.29 Happiness rating:0.34 Sadness rating:0.43 Neutral rating:0.35 Fear rating:0.26 Sentiment scoring: ID: Test_Img_0055 Anger rating:0.24 Happiness rating:0.39 Sadness rating:0.42 Neutral rating:0.39 Fear rating:0.19 Sentiment scoring: ID: Test_Img_081 Anger rating:0.32 Happiness rating:0.37 Sadness rating:0.27 Neutral rating:0.40 Fear rating:0.28 Sentiment scoring: ID: Test_Img_083 Anger rating:0.25 Happiness rating:0.47 Sadness rating:0.28 Neutral rating:0.50 Fear rating:0.15

Training Result Input

Output

Pix2pix Loss over time Target

Loss: Discriminator

Sentiment scoring: ID: Test_Img_085 Anger rating:0.25 Happiness rating:0.45 Sadness rating:0.25 Neutral rating:0.48 Fear rating:0.19

Loss: Generator

Loss: Generator_GAN

Sentiment scoring: ID: Test_Img_089 Anger rating:0.27 Happiness rating:0.39 Sadness rating:0.30 Neutral rating:0.44 Fear rating:0.23

Different Mood

Taida AR Exploration Enhance the fun and exploreability of a tour.

[AR, App] Location: Tianjin, China Date: 03/2021-07/2021 Team Member: Xinyi Zhou, Rufeng Xiang, Mengqi Tu, Haida Sun Role: Innovation & technology planning, Content Production, UI Design This project is a large-scale commercial project, which includes the AR exploration experience of aircraft carrier park, popularization of aircraft carrier knowledge, and cloud tourism. My job was mainly responsible for the innovation and technology planning of each AR project and the design of UIUX [Unity, C#, 3ds Max, Adobe XR, App development]. It gave me a new understanding of innovation and technology from a business perspective.

Offline Armament AR experience | UIUX Design

Online Armament AR experience | UIUX Design

Live AR presentation | Offline Armament AR experience

Live AR presentation | Offline Armament AR experience

Profassional Work: Shenyang K11 Kaos VR Experience Space Design [VR Offline Experience]

Location: Shenyang, China Date: 12/2020 Role: Experience Design; Spatial Design

Exhibition: 2021 Asia Digital Exhibition Expo Responsible for product experience design; product display Location: Guangzhou, China Date: 05/2021 Role: Experience Design; Product display

Spatial Design: Design of YOMOV VR Experience Hall Space experience transformation, product display Location: Shenzhen, China Date: 03/2021 Role: Experience Design; Spatial Design