MOMENT | Research
Contents 3
Research 4 5 13 17
User interviews Six mindsets when we reminisce Extended Reality technology Memory product anlaysis
23 Ideation
25 Memory aggregator 26 Emotion theory 27 User flow
29 Design & Protyping 31 34 37 41 44
Technical investigation Design process Art direction Technical challenges User testings
50 Final Project 65 Future Potential 73 References
1
MOMENT | Research
MOM ENT An experimental design project exploring how
extended technology could support reminiscing behaviour. Utilizing geolocation-based AR technology, I created this magical experience which helps people to revisit their past memories at the locations where those experiences happened. The adventure-like interaction design transforms the way we reminisce, and allows us to generate positive self-concept and create a strong sense
Demo Video: https://vimeo.com/404385691 Github Repository: https://vimeo.com/404385691
fcontextualized presence.
2
User interviews Literature review Product analysis
3
Behaviour Product Technology
Research
MOMENT | Research
User Interviews
Questions asked I conducted 6 in-depth interviews with people in various occupations and different life stage in order to gain a more comprehensive
understanding of their reminiscence behaviours. Artist/Designer, Engineer, Accountant, Entrepreneur, Manager, Architect Student, mid-20, mid-30, mid-40 and retired people.
Full Interview Script: https://docs.google.com/document/d/1gD4k4mO0epJ_G0juSdfp_STsTR7A2TNTEsY9zxg7bzs/edit?usp=sharing
4
Six mindsets that explain why we reminisce
5
MOMENT | Research
Reminiscing is a universal behaviour that
the future. By recalling life events, people
consists of memory revisiting and
are developing a strong self-concept that
self-narrative generation. It is an important
allows them to understand and remember
human behaviour that helps us to develop
who they think they are, and connect those
a strong self-concept, enhance the social
experiences with the current situation,
bond and improve our future thinking skills.
which ultimately helps them clarify the self-identity, and create a closure that
Memory as self-making narratives In the article Informing augmented memory system design through autobiographical
connects their past self and the present self. Research in autobiographical memory theory shows the importance of developing self-narrative. Thus, I decided to create a feature that allows people to write a short note about their feeling when they revisit the memory to encourage them to articulate abstract thought in a writing form for revisiting in the future.
memory theory, the author describes the function and importance of memory in the context of autobiographical memory theory: construct our self-concept, maintain social relationships and predict
6
MOMENT | Research
Reminiscing can be the consequence of memory sharing which is a common behaviour in social settings. When we meet friends or strangers, we tend to share experiences to prove the argument or build empathy to enhance the social bond. We exchange stories to maintain and deepen the social relationship. In this case, memory becomes the material of social engagement. Therefore, one of the use cases of my project is in social settings: imagine that you have a friend reunion at the place where you used to hang out together, my project allows them to easily share and revisit those moments and facilitate the reminiscing practice. Besides, memory can also stimulate the future, as we constantly reflect on past experiences and try to recognize similar patterns in current and future scenarios. We gain affirmation and denial, and these experiences regulate the way we behave in future scenarios.
Past Present Future 7
MOMENT | Research
We unconsciously develop biases in reminiscing Adaptive memory theory shows that people tend to look back at past experience from an adaptive perspective when revisiting past experience: they are more likely to think positively and unconsciously develop biases by embracing a more positive mindset, a rosy retrospection, and developing fading affect bias.
8
MOMENT | Research
We unconsciously develop biases in
perspective to an adventure-like
reminiscence and tend to look back at
experience: although they are revisiting
past experiences from an adaptive
their memory, the interaction
perspective. According to adaptive
mechanism makes it like a new
memory theory, people are more likely
adventure.
to have a positive mindset to understand what they did to overcome
Rosy retrospection and fading affect
those negative experiences happened
bias are also at play in reminiscing. In
in the past. People choose to distance
the article Technology-mediated
themselves from past events and look
memory: Is technology altering our
back at it from a third-person
memories and interfering with
perspective so that they can make
well-being, the author argues that all of
sense of it, and embrace mistakes they
these unconscious biases reduce the
have made. In my work, the idea of
pain in reminiscing, and can help to
discovering memory footprint and
interacting with the floating memory
bubble transforms people’s
9
maintain well-being in the long run instead of posing threat on mental health.
MOMENT | Research
Memory and Location Memory is contextually affected and can be triggered by contextual information, such as geolocation, smell, sound, and physical state. The AR project Nevermind created by MIT demonstrates how contextual information such as location could help people improve memorize skills. This project proves the power and the effectiveness of utilizing location as a memory trigger. The strong association between location and memory is the major reason I chose to develop a location-based AR memory project, and also the biggest differentiator between my project and other memory products
Image Credit: NeverMind, Oscar Rosello, MIT Media Lab https://www.media.mit.edu/projects/nevermind/overview/
10
MOMENT | Research
A place is a contextualized space Another location-related feature in my project is a memory footprint map that allows people to see their memory located on the city map, and I use different colours to represent the emotion attached to the memory pieces. This feature generates a psycho-geographical map to let people understand how their emotion changes around the city and provide a new way of appreciating life and the city. It can also be seen as a tool to help people learn about themselves, and even uncover their true desires in everyday life. This idea responses to situationist: build a personal understanding of the city via lived experience rather than being limited by the function and physical existence of urban planning.
11
MOMENT | Research
Technology-Mediated Memory We constantly post information on social media, and later on, this content becomes the memory material to support our reminiscing behaviour. Our life is also recorded and mediated by technology, and a huge amount of metadata is generated alongside the organic content. For example, when we take photos, our phone automatically captures the location tag, date, and even recognize the people, which makes contextual understanding possible. Technology mediated memory is more accessible, selective, and can be displayed in a rich media format. If human memory is the raw material that has been collected and stored in our mind, technology facilitates the process of memory encoding and retrieval by providing a huge amount of records and accessible opportunities for self-reflection, and enable content curation on past experience.
12
MOMENT | Research
XR Industry Report
Biggest Barrier to Mass Consumer Adoption of VR
46.2% of respondents said
Price of HMDs
38.1% of respondents said
Current HMD size/design
37.6% of respondents said
of respondents said
Lack of content
Low consumer awareness
Source: XR Industry Survey Report | VR Intelligence, 2019, www.vr-intelligence.com/xr-industry-survey-report.
13
45.6%
36.6% of respondents said
Usability of HMDs
MOMENT | Research
The majority of respondents believe that
“ it will take 4 or more years for AR to move from predominantly occurring on people’s smartphones to become headset-based.
In 4+ years 43%
Never 8%
“ Consumer XR may not have quite the same level of profitability yet but it is showing signs of major potential and the next 18 months should see key developments. From the likely dropping of the next generation of consoles into the consumer market, to falling headset costs, to the explosion of open-source development software, to the rash of AR integration on major platforms, the space is likely to evolve extremely rapidly and for the better. Source: XR Industry Survey Report | VR Intelligence, 2019, www.vr-intelligence.com/xr-industry-survey-report.
(2024 later)
In the next year 5% (2021)
In 2-3 years 44% (2022-2023)
14
MOMENT | Research
Volumetric video
8i, a volumetric video production studio,
The emergence of immersive technology provides a new way of reminiscing.
is building its repository for hologram projects. In 2017, they taped a volumetric video of a mom and her child and created a hologram project that allows people to watch, walk around and even put that digital content in their hands with a head-mounted device. Although Mixed reality and volumetric
Image Credit: 8i studio, The Verge
15
https://www.theverge.com/2017/11/7/16613234/next-level-ar-vrmemories-holograms-8i-actress-shoah-foundation
video capture are still at a very early stage, and not yet commercially accessible, these projects show the possibility of utilizing XR technology in reminiscing practice and demonstrate the advantages of immersive technology in building emotional connections.
MOMENT | Research
Case Study
Miscrosoft Holoportation
Similar research is also being conducted at
daughter interact with each other.
Microsoft. Holoportation visualizes how MR helps
Although due to technical limitations we have
people to communicate and interact with the
nowadays, this project is still at the exploration
digital avatar of people who stay at another
phase, it shows the promising potential of MR
place, which is called virtual 3d teleportation.
technology. The research also explains
This technology allows people in different
technological challenges and how Microsoft is
places to be present in the same environment
solving those problems in order to improve the
and provide a new way for people to interact
experience in MR, such as reducing bandwidth
with each other. As demonstrated in the video,
using innovative compression algorithms as
the researcher in MR office can interact with her
well as challenges in rendering, lighting, and
daughter who is in another place. The rendering
vibration.
can be a bit glitch due to bandwidth issues, but it is quite realistic as you can see the girl walking around in the space. Later on, both the researcher and his daughter are captured in a video and the researcher can re-enter this scene to watch how he and his Image Credit: Holoportation, Miscrosoft Research
“it’s almost like walking into a living memory that I can see for another pair of eyes from any perspectives
https://www.microsoft.com/en-us/ research/vproject/holoportation-v1-images/
16
MOMENT | Research
Product Analysis
Memory Product Tech giants are embracing the memory economy by creating products and features that leverage existing social media content and allows users to easily revisit past experiences. As primary social media platforms, the huge amount of existing data puts those companies in advantageous positions.
Facebook Memories
Instagram Stories
Snapchat Memories
17
MOMENT | Research
Facebook Memory
Lookback and say thanks video Help users automatically generate lookback video and say thanks videos, and use existing content to encourage interaction between users.
Since 2016, organic content sharing has started to decline on Facebook: people no longer share content on Facebook. Facebook has been struggling to reverse a 21% decline in
2014
original sharing or personal updates form its 1.6n billion monthly active users. To improve this situation, Mark Zuckerberg develops the “time well-spent” strategy to focus on the quality engagement on Facebook even if the time on site suffers as a result. Facebook decides to change the focus to social interaction and meaningful community building. Their algorithm will prioritize posts that promote
Decline in organic content sharing & Time well-spent strategy
2016
natural interaction between people over public content such as businesses or publishers.
“ Time well-spent strategy
Standalone memory feature
2018
A standalone feature is created under more tab. An expanded version of “on this day” and friendship anniversary also published to help users engage with their Facebook friends and encourage community interaction. 18
MOMENT | Research
“ Let people generate new things to share even if they’re laying in bed or stuck somewhere. 19
MOMENT | Research
Instagram Story “On This Day” option shows a random feed
especially helpful because not everyone
post you shared on the same calendar
temporality, and may contribute to
does something Stories-worthy every day.
date in the past. Tap the dice button to
increased emotional rewards. The
And given how many #TBT throwbacks get
view a different On This Day post, and
live-story feature released later can be
shared already, there’s a demand for
seen as an expanded version.
once you find one you prefer, you can
sharing nostalgia with new commentary.
share it to Stories as an embedded post people can open.
Not everyone does something Stories-worthy every day
Playfulness of temporality
Users can choose to make it a permanent content by adding to the highlight, a collection of Instagram stories. These collections allow the user and other people to enjoy the glimpses into friends’
Instagram stories are only visible for 24
lives. It also enables users to curate the
hours, after that, the content will be saved
content based on their preference,
in the archive category under the
whether it is about time, or travel, or
hamburger button. The idea of temporary
mood.
People can choose stickers such as tag
posts reduces the self-representational
memory, hashtag, location, or emojis to
concern and encourages the sharing of
easily convert their old impermanent
mundane experience. it emphasizes on
content into fresh content. That could be
the idea of real-timeish sharing, and
20
MOMENT | Research
Snap Memories Snapchat Memories—introduced in
Launch My Story Feature
2016—shifted Snapchat away from being the platform of
2013
self-destructing photos and videos, to be a direct competition for Facebook. Memories are now a key component of Snapchat.
Launch Snap Map
Add Geofilter for quick image location tags
2017 2016
21
2014
Memories Feature & My Eyes only
Ephemeral social interaction Post saved content Content recognition Private Content Snap Map
MOMENT | Research
Share with comfort Memory is now searchable All saved photos and videos can be searched by keywords. Snapchat will identify the content based on captions, emoji, dates, locations and even object to categorize memories. Snapchat will show users keyword suggestions, such as this date in the past, taken nearby. Users can use a distance radius setting to change the distance to home or a whole city.
Snapchat wants users to feel comfortable passing their phone to other people when sharing their memories. Snaps and Stories can be set to My Eyes Only —no awkward peeking, Only share things you want.
Snap Map Snap Map lets users share their current location by creating their virtual avatars on Snap map. It lets users discover stories in a new way, and even dives into a place to see what people are doing on the other side of the earth. They also can see “heat” colors on the Snap Map to see where lots of Snaps are being uploaded and where are people up to. Their posts will only appear 24 hours for non-friends.
22
Ideation Memory aggregator
Emotion theory
User flow
23
Technology is recording and mediating our memories, and we are contributing to our digital identity.
24
MOMENT | Ideation
Our digital footprint becomes the new memory material
25
We constantly post information on social media which generates new memory material and supports our reminiscing behaviour. Our life is mediated by technology with a huge amount of metadata generated alongside. What if we can create a memory aggregator that collects, analyzes, and categorizes our memory pieces based on location, date of created, tones, and object?
MOMENT | Ideation
Emotion Theory
What if we can create a filter that lets people selectively revisit their past experiences and curate their memory content? Emotion is another important aspect of memory, and it can trigger events that are emotionally congruent and incongruent to our current feeling. Plutchik’s emotion theory describes that there are eight major types of emotions (Joy, Acceptance
Surprise, Anticipation, Fear, Ease and Anger), and the combination these emotional ingredients generate new emotion concepts, such as love, contempt, guilt. Therefore, in my project, I use emotion as the attributes to categorize different emotions and created a filter that allows people to selectively revisit their experience and even curate
Plutchik's Wheel of Emotions
memory content.
26
MOMENT | Ideation
User Flow
This is the user flow llustrating the whole user journey: 1. Get users' consent to connect and analyze their social media accounts. 2. Enter the AR portal and reveal the memory. 27
“Much of the world is mundane, but the memories of what happened there are not..
MOMENT | Ideation
Story 1
Story 2
Storyboard Draft Two storyboard drafts I created to help articulate the use case of MOMENT. Story 1: Use Moment at a friend reunion. Story 2: Feel lonely and take random work, being notified to revisit a memory happened at this place in the pat. 28
Design & Prototyping Technical investigation
Design process
Art direction
Technical challenges
User testings 29
Process Overview
MOMENT | Design & Prototyping
30
MOMENT | Design & Prototyping
Technical Investigation
Choosing the right AR prototyping tools is critical for rapid iterations
Being able to identify when to use what tools for rapid prototyping is critical as it can help to save numerous amount of time. I spend some time testing a few markerless AR software, such as Spark AR, Reality Composer, Metaverse Studio, and compare the pros and cons to find the one that can effectively visualize and test the interaction design. Once I made the decision about the placement and the size of those virtual objects, I start mid-fidelity prototyping in Unity.
31
MOMENT | Design & Prototyping
Tools testing
1
1 3
https://vimeo.com/363191454
https://vimeo.com/363191356
2
https://vimeo.com/434893905
4
https://vimeo.com/ 393174918
2 3 4
Placenote SDK
Use Placenote SDK to scan indoor environment and generate point clouds to place objects.
Unity AR+GPS
Test unity AR+GPS plugin using placeholder object
Reality Composer Very stable rendering in AR using built-in assets
Spark AR
Create a world-effect filter using the front camera
32
5 5
33
https://vimeo.com/363191454
Metaverse Studio
I ended up using metaverse studio for rapid prototyping and created the first version interactive mockups for feedback by the end of the Fall term. In the app, I created each screen separately and connected them based on the interaction flow. The file size reduction takes some iterations and loading speed needs to be improved.
MOMENT | Design & Prototyping
Design Process Body Storming Body Storming is an efficient way of designing interaction mechanisms in immersive experience, and it saves a lot of time needed in unity prototyping. I printed a few photos to simulate the physical interaction from the user's perspective and quickly generate ideas.
34
MOMENT | Design & Prototyping
Design Process
Quick sketch helps me brainstorm the interface design before work on digital artboards
35
MOMENT | Design & Prototyping
Wireframes V1
These are the first version wireframes I created for in-class pitch and feedback collection. It illustrates the rough flow from notification to emotion filter. The heat map shows on the left is a memory footprint mockup where I use different colours to represent various types of emotions. Mockup shows on the bottom left are the rough design for views in augmented reality: a colourful floating bubble can be found on top of the real world layer and once users click on it, it will be transformed to a couple of pictures that are captured at this place in the past. An input field is at the bottom of the screen where users could type in a message or their reflection when they revisit the memory. A folded envelop records this message and reminds users that they have a message to look at.
36
Art direction Memory is such a poetic and beautiful object. To find the style and visualize the feeling memories bring to us, I collect images of textures, text, lines and colours to help me decide the overall aesthetics style and visual elements.
37
MOMENT | Design & Prototyping
Memory Bubble I rendered these orb animations in After Effects to represent different types of memories. Different colours represent different types of emotions.
Fluid
Abstract
Emotional
Distorted
Poetic
Organic
38
Different types of memory
39
MOMENT | Design & Prototyping
User interface design V1 The design showing on the left is the first version user interface design created in Adobe XD, including memory footprint, account setting, homepage, emotion filter and delete the page. I used this prototype to gather feedback at the end of the Fall term. The original thought was to make the
interface as an embedded part of the
memory piece, therefore the tone of the
background is changing with the content: when revealing a sad memory, it changes to blue shade, when revisiting a happy moment, it changes to red/pink shade.
Clickthrough Video https://youtu.be/vA2bbPfDqVY
40
MOMENT | Design & Prototyping
41
Technical Challenges
Develop a fully functional AR app is quite challenging for me. It involves multiple aspects, from VFX to animation, from interaction logic to input and content management, no need to say weird bugs come from nowhere. Thanks to online communities and forums, I solved most of the problems and completed the project as planned.
MOMENT | Design & Prototyping
Visual effect and animation in Unity
My goal is to create a stunning visual effect and a compelling
AR experience. The visual effect component in Unity is quite limited and cannot achieve my expectation, so I rendered the orb animations in After Effects and import the animation sequence to Unity as animation sprite to improve the visual effect quality.
Save posted content and render it on 2d UI sprite
The funcation where people write a message to themselves
is a bit complex. By creating a public variable, the written message can be saved as a string and rendered as text on a note image.
42
MOMENT | Design & Prototyping
Outdoor location-based AR experience
Playmaker cannot achieve desired outcome
challenging because it is not a popular functionality
my scripting skill, but later on, I found that playmaker does not support
foundation, and additional scripting is needed. I
outcome. I chose to do scripting by searching tutorials on google and
and the GPS is pretty decent and accurate.
since my app does not require complex inventory or score system.
Creating an outdoor location-based AR experience is
I purchased the Playmaker plugin because I was not very confident about
offered by AR packages such as vuforia AR or AR
some customized effects and requires extra time to achieve the desired
purchased the AR+GPS plugin to create the prototype
adapt existing code available online, which turns out to be more efficient
Xcode app crashed on iOS 13.3.1 At the time of building the app using Xcode, I encounter issues where apps on iOS (13.3.1) device crash on startup, even though no build errors found, the only solution is to downgrade my ios version to 13.3.
HDRP not supported on mobile devices I used High Definition Render Pipeline in Unity to create the visual effect (particle orb) and found that HDRP is not supported on mobile devices but only on high-end hardware. Since the object has already been created, it cannot be Image Credit: AR +GPS Location Unity Plugin, Unity Assets store
https://assetstore.unity.com/packages/tools/integration/ar-gps-location-134882
43
transformed into another render pipeline. I had to rebuild it from scratch.
MOMENT | Design & Prototyping
User testings I did several in-class user testings sessions to gather feedback and observe peple’s reaction to my app. I also asked about their feelings of using my app, including if the copywriting is straightford, and likeability of the visual style. Later on, I analyze the testing result and refined the design of the mobile app and AR experience.
44
MOMENT | Design & Prototyping
Problem Found
Size and placement of content in AR
When the imagery is too big, users need to step back to view the full size which can cause dangers in an outdoor environment. When the imagery is too small, it is hard to find it as the content can be blended in the background. Where the content is placed in AR is also critical. When placing the object behind users, it can cause frustration if the content cannot be found easily. If the content is placed too high or low in the air and not within their reach, users won’t be able to keep holding their AR device and their attention can be distracted.
Image Credit: Comfortable range of motion zone, Mike Alger https://www.youtube.com/watch?v=id86HeV-Vb8&t=931s
Image Credit: AR Design guidelines, Apple AR
https://designguidelines.withgoogle.com/ar-design/user/safety-comfort.html
Reading notes in AR is not accessible
Reading a long paragraph in AR is not accessible
due to the lighting situation and users can only read content that appears on the screen. They need to move a lot to read the whole paragraph.
45
MOMENT | Design & Prototyping
World Anchor vs Screen-based UI The object is anchored in the world space to enhance the immersion and create an adventure-like experience, but when it comes to message reading, it is not accessible enough especially when users are reading a long paragraph. Therefore, I decided to change the design to a screen-based user interface to improve the readability: it appears as regular text on the screen regardless of the device position or rotation.
46
MOMENT | Design & Prototyping
UX writing is not straightforward enough Some users find the UX writing is not clear enough and hard to understand what to do. I made several iterations to improve the visual style and UX writing. The final version clearly communicates the instruction and keeps the style consistent throughout the whole journey. Adding a vignette overlay underneath improves the readability without distracting attention from the memory bubble. Although immersion is important, adding a glowing effect makes the virtual orb more noticeable in the background and easier to be found. The floating animation also encourages users to interact with it.
47
MOMENT | Design & Prototyping
Set clear user expectation The concept of revisiting memory can be quite abstract. A lot of people don’t know what to expect in the app. In the user testing sessions, people constantly asked me to explain what they are looking at and what to do, which means the interaction flow needs to be improved. Although this can create a sense of surprise and satisfaction when the AR object suddenly appears, it can make users feel confused. Therefore, I added an onboarding flow before users enter the AR world, which helps to set up users’ expectations and articulate the concept in a clear way.
The concept of memory revisiting can be hard to explain, an onboarding flow is needed to set users’ expectation.
Prototyping Iterations
48
MOMENT | Design & Prototyping
Refined wireframes
Onboarding Flow AR Flow Message Flow
49
Final Project
50
MOMENT | Final Project
AR Onboarding Experience People tend to distance themselves from past events and look back at them from a third-person perspective so that they can make sense of it, and embrace mistakes they have made.
Find the AR Portal The floating memory bubble flies out of the screen and an arrow appears to ask users to follow the direction. This design helps users to find the object in AR and encourages them to explore the environment.
Onboarding 1
51
Onboarding 2
MOMENT | Final Project
Onboarding 3: Animation sequence
52
MOMENT | Final Project
Revisit the Moment The memory bubble will fade in as a glowing orb with particle effects, then the core will show up. The bubble is flying up and down so
that it is not blended in the background and
can be found easily. After users find the
bubble itself, the animation can trigger a sense of satisfaction and encourages users to tap on it. The idea of interacting with the memory bubble transforms people’s perspective to an adventure-like experience: although people are revisiting their memory, the interaction mechanism makes it like a new adventure, a new moment.
Bubble fades in
53
Bubble appears
MOMENT | Final Project
Revivify the memory Once the user taps on the memory bubble, it will transform into a photo or a video clip captured and shared by the users in the past. Tapping on it will animate the video
sequence from a static image and bring the
memory alive.
This interaction design helps to create a sense of magic and immediate satisfaction, it can also be seen as a metaphor: old memory is alive and all the feelings come back. Taping on the corner of the screen will hide the action buttons so that the AR view will not be blocked. Users can tap on the photo to bring the buttons back and choose other actions.
Tap to animate the photo
More actions
54
MOMENT | Final Project
Bring the memory back: animation sequence
55
MOMENT | Final Project
Write a short note to your future self By recalling life events, people are developing a strong self-concept that allows them to understand and clarify their self-identity. Writing a short note about feeling when revisit the memory encourages people to articulate the abstract thought and generate positive self-concept. This note is saved alongside the memory piece and an envelop will show up next to the photo content for future revisit.
Write and send a note
Type in the note
56
MOMENT | Final Project
Envelop animation
Envelop opening animation is a metaphor
Note saved alongside the memory
unfolding a story that happened in the past. It also smooths the transition to a screen-based coordinate and bring the user to the message reading scene.
57
MOMENT | Final Project
Open the note: animation sequence
58
MOMENT | Final Project
Memory examples
59
Memory examples 60
MOMENT | Final Project
Memory examples
61
MOMENT | Final Project
Homepage design 62
MOMENT | Final Project
Notification Screens: Various memories 63
MOMENT | Final Project
Emotion Filter 1
Emotion Filter 2
Setting 64
Future Potential
65
MOMENT | Future Potential
When 3D capture becomes as easy as taking a picture, what will our photo albums look like? ScandyPro Bellus3D
The invention of photography disrupted
becomes as easy as taking a picture, our
traditional painting and changed the way
photo album will be full of objects
we capture daily life objects. Nowadays,
captured in 3D space, and we can replay
Taking photos is like second nature when
a moment in a more immersive way.
we want to capture a moment for future reminiscing. With the emergence of 3D capture and volumetric videos, we now have a new way of capturing objects and physical environments. Although volumetric video capture still requires advanced equipment in professional digital production studios, 3D scan has become more accessible in the past few years:
Display.land
free mobile apps such as Scandy Pro and
Fyuse
and generate a full colour mesh which
Bellus3D let users quickly scan their head can be exported to 3d software for editing and sharing purpose. The result is quite accurate and efficient. When 3D capture
3D mesh of my head using Bellus3D app
66
MOMENT | Future Potential
3D human digitization Even objects captured in 2D in the first place can also be converted to 3D format by utilizing advanced algorithms in machine learning and computer vision. Research conducted in Facebook Reality Lab has successfully achieved high-resolution 3d reconstruction of humans using simple input such as a photo. Although the performance is lower than professional 3D capture techniques, it is still significant in enabling 3d human digitization. This research demonstrates the possibility of converting 2d content to spatial objects and can unlock numerous possibilities in storytelling, experience rebuilding, and environment augmentation. Image Credit: High-Resolution 3D Human Digitization https://arxiv.org/pdf/2004.00452.pdf
67
MOMENT | Future Potential
Physical environment scan and rebuild The introduction of iPad Pro 2020 with a LIDAR camera makes physical environment scanning accessible and brings new possibilities to Augmented Reality. Its depth-sensing capabilities enable more accurate object and environment recognition. In WWDC 2020, Apple demonstrates point cloud visualization using sense depth technique and allows users to easily digitize physical surroundings and place points in the real-world environment. The advancement in environment scanning and recognition can significantly improve accuracy in raycasting and make the world-anchored object effect more stable.
Image Credit: Explore ARKit 4, Apple
https://developer.apple.com/videos/play/wwdc2020/10611/
68
MOMENT | Future Potential
Contextualized presence WWDC2020 also introduced the built-in
feeling about a space. Utilizing
location anchor feature in ARKit 4 that
augmented reality, my project lets
allows users to easily create AR
people experience a place in a way that
experience in relation to the globe and place objects as an annotation of their
is unique to them, and create a
personal storytelling experience that is
surroundings. Combining high-resolution
related to their life. Augmented Reality
map data and user’s geo data, the
technology allows us to “customize” the
complexity of creating a location-aware
physical environment by placing
project is remarkably reduced.
personalized mixed-media content on top of the real world layer.
Contextualized presence is the feeling of being in a place rather than a location. Space becomes a place because it is meaningful to us. If we say human invented clocks to measure time, we could also say human created maps to make space meaningful. A place is unique and memorable because each of us can have a unique experience and
69
“A place is more than just a space. A place is redolent with meaning...
Image Credit: Explore ARKit 4, Apple
https://developer.apple.com/videos/play/wwdc2020/10611/
State-of-the-art
Re-enter our memories in 3D space
Future version This mockup demonstrates the look of the ultimate version I am visioning when immersive technology and volumetric capture technology becomes fully accessible. Imagine that we can re-enter those moments and reunion with people in the memory. Although there will be ethical issues that needed to be discussed, it can bring lights to industries such as healthcare, education, and marketing.
70
MOMENT | Future Potential
Revivify our feelings in a multi-model experience My thesis project shows the potential of applying XR in experience rebuilding, and the notion of time travel. When multi-sensory capture becomes accessible, creating a hybrid experience where people could re-enter their past memory with an MR headset will not be a dream anymore. the project could be expanded to a multimodal experience, including visual, audio, olfactory, and haptic. The mixed media format will add a new dimension to reminiscence and memory revisiting. Although this whole project might look a bit conceptual and experimental at this point, I believe that something similar will be invented in the near future.
71
MOMENT | Future Potential
Concept teaser video: https://vimeo.com/417008435 72
References
73
MOMENT | References
Bibliography Cosley, Dan, et al. "Experiences with designing tools for everyday reminiscing." Human–Computer Interaction 27.1-2 (2012): 175-198. Accessed 23 September 2019 This article provides a holistic view of designing experiences for memory products and introduces some key design principles from a human-computer interaction perspective. The author mentions that there is a tendency to designing technologies for experiences and human-centered content. To achieve that, research has been conducted in two areas, one is to understand people’s reminiscing behavior, another one is to explore how technology could support it.
Cutler, Ben, et al. “Holoportation.” Microsoft Research, 2019,
www.microsoft.com/enus/research/project/holoportation-3/ . Accessed 3 October 2019.
This is a Mixed reality prototype created by Microsoft. In the video, it visualizes how MR helps people to communicate and interact with the digital avatar of people who stay at another place, which is called virtual 3d teleportation. This technology allows people in different place to present in the same environment and purposes a new way for people to interact with each other. The research also explains technological challenges and how Microsoft is solving those problems in order to improve the experience in MR, as well as challenges in rendering, lighting and vibration. These are the technical barriers XR tech is facing before achieve mainstream adoption point.
Exposito, Marc, and Oscar Rosello. “Project Overview ' NeverMind: Using AR for Memorization.” MIT Media Lab, 2016, www.media.mit.edu/projects/nevermind/overview/. Nevermind is an AR project aiming to explore how to help people memorize more effectively. It combines location-based AR technology and memory palace, a memorize method first developed in ancient Greece and suggests that this type of memory retrieval makes the accuracy even tripled to conventional paper-based memorizing techniques.
Goode, Lauren. “Are Holograms the Future of How We Capture Memories?” The Verge, The Verge, 7 Nov. 2017, www.theverge.com/2017/11/7/16613234/next-level-ar-vr-memories-holograms-8i-actress-shoah-foundation. 8i is a volumetric video production studio building its repository for hologram projects. Their project makes more impact on the way people experience memory and historical events. It also poses a question on the way people interacting with each other: the ethics question around the content itself, trauma, negative experiences… There are also security issues that need to be discussed: the governing of these digital content, and the misusage for inappropriate activities.
Hoven, Elise, and Berry Eggen. "Informing augmented memory system design through autobiographical memory theory." Personal and Ubiquitous Computing 12.6 (2008): 433-443. Accessed 23 September 2019
74
MOMENT | References This article provides an in-depth explanation of autobiographical memory theory and gives an overview of design augmented memory systems. The author first argues the six functions of memory and why it matters, which provides the context of my work: why we need an application to help people obtain their memory. The three major functions of memory are the construction of self-concept, maintain social relationships, and predict the future.
Isaacs, Ellen etal. "Technology-mediated memory: Is technology altering our memories and interfering with well-being?." ACM Transactions on Computer-Human Interaction (TOCHI) 23.4 (2016): 23. Accessed 25 September 2019 The authors argue that nowadays technology provides a rich record of our memory and makes memory retrieval easier and more accessible. This article also compares the differences between organic human memory and technology-mediated memory and evaluates whether the latter can have a negative impact on well-being. Authors believe that when people revisit and reflect on past experiences, they tend to embrace a more positive mindset and help to maintain well-being in the long run.
Konrad, Artie, et al. "Technology and reflection: Mood and memory mechanisms for well-being." Psychology of Well-being 6.1 (2016): 5. Accessed 1 October 2019 In this article, the author argues that mood is an important aspect of memory, and it can trig events that are emotionally congruent and incongruent to our current feeling. The authors identify and summarize some common patterns in memory retrieval and self-reflection: understanding, redemption, and distancing.
Konrad, Artie. “Facebook Memories: the Research behind the Products That Connect You with Your Past.” Facebook Research, 21 Sept. 2017, research.fb.com/blog/2017/09/facebook-memories-the-research-behind-the-products-that-connect-you-with-your-past/. This article shows the product design rationale behind Facebook Memories from a content strategy to a standalone product. Facebook memory feature shows the foundation of utilizing social media content to generate new content, and allow people revisit past experiences,
Donalek, Julie G. “Phenomenology as a Qualitative Research Method.” Urologic Nursing, vol. 24, no. 6, Dec. 2005, doi:10.5539/ass.v10n21p298. Donalek provides an overview of phenomenology and introduces its origins and development process. It describes the basic principle of phenomenology and its impact in understanding people’s behavior as a qualitative research method.
Holm, Tine, and Dorthe Kirkegaard Thomsen. “Self-Event Connections in Life Stories, Self-Concept Clarity, and Dissociation: Examining Their Relations with Symptoms of Psychopathology.” Imagination, Cognition and Personality, vol. 37, no. 3, 2017, pp. 293–317., doi:10.1177/0276236617733839. This article argues that the ability to connect life events and formulate stories is important for mental health, as these narratives help to connect the past self and the current self, and therefore clarify the self-identity and maintain self-unity. It also introduces theories and practices such as self-report.
75
MOMENT | References Neubauer, Brian E., Catherine T. Witkop, and Lara Varpio. "How phenomenology can help us learn from the experiences of others." Perspectives on Medical Education 8.2 (2019): 90-97. Accessed 2 October 2019. This article provides in-depth descriptions on phenomenology and research methods such as user interviews, focus group. It argues that phenomenology compared to other research methodologies such as participatory, focuses on lived experience and can help researchers understand complex phenomena by learning from individual behaviors.
Papagiannis, Helen. Augmented human: How technology is shaping the new reality. " O'Reilly Media, Inc.", 2017. In this book, Papagiannis envisions how AR technology could change people’s life in all aspects, from transportation to film production. It analyze the unique aspect of AR technology, such as interaction model, haptic, relation with 3D space, and argues that XR could be seen as a mixed media format which can provide new opportunities and challenges for storytelling.
Van Den Hoven, Elise, Corina Sas, and Steve Whittaker. "Introduction to this special issue on designing for personal memories: past, present, and future." Human–Computer Interaction 27.1-2 (2012): 1-12. Accessed 23 September 2019 This article argues that digital technology give people access to numerous memory material and people can effortlessly access those content. It also allows people to organize and curate the memory content based on their preference. The author also introduces the taxonomy of memory and provides a list of the most desirable memory content.
VR Intelligence. “ XR Industry Survey Report.” XR Industry Survey Report | VR Intelligence, 2019, www.vr-intelligence.com/xr-industry-survey-report. Accessed 14 September 2019. This report provides an overview of the development status of XR technology in 2019, and combined both qualitative and quantitative approach to discover the current roadblock and articulate the problem XR technology is facing in order to the point of mainstream adoption.
West, David, Aaron Quigley, and Judy Kay. "MEMENTO: a digital-physical scrapbook for memory sharing." Personal and Ubiquitous Computing 11.4 (2007): 313-328. Accessed 30 September 2019 This article evaluates a memory system called MEMENTO, a multimodal digital scrapbook that allows people to collaboratively collect and share memory materials, and examine its effectiveness in reminiscence for both young and senior people. This article summarizes some common practices in reminiscence, and indicates that digital technology has positive impact on reminiscence.
76
MOMENT | References
Credits Image Holoportation by Microsoft Research, https://www.microsoft.com/en-us/research/project/holoportation-3/ 8i studio, The Verge, https://www.theverge.com/2017/11/7/16613234/next-level-ar-vr-memories-holograms-8i-actress-shoah-foundation Comfortable range of motion zone, Mike Alger, https://www.youtube.com/watch?v=id86HeV-Vb8&t=931s AR +GPS Location Unity Plugin, Unity Assets store, https://assetstore.unity.com/packages/tools/integration/ar-gps-location-134882 AR Design guidelines, Apple AR, https://designguidelines.withgoogle.com/ar-design/user/safety-comfort.html NeverMind, Oscar Rosello, MIT Media Lab, https://www.media.mit.edu/projects/nevermind/overview/ High-Resolution 3D Human Digitization, https://arxiv.org/pdf/2004.00452.pdf Explore ARKit 4, Apple, https://developer.apple.com/videos/play/wwdc2020/10611/ Footage 2046 by Wong Kar-wai, Portrait of a Lady on Fire by CĂŠline Sciamma, Vertigo by Alfred Hitchcock, Room in Rome by Julio Medem, Paprika by Tinto Brass, Blade Runner 2049 by Denis Villeneuve Free videos provided by creators on Pexels Taryn Elliott, Life On Super 8, Athena, Pixabay, cottonbro, Stef, Pressmaster, Mapo Dessin, Desmond Etsebeth, Carlos Arribas, Fauxels, Free Videos, Distill, Pixly Videos, Charlie Mounsey, Kelly Lacy, Joseph Redfield, Ricardo Esquivel Music Blasphemy, Otik Beautiful, AShamaluev
77
Afterword Although thesis was supposed to be
always give me magical and mind-
hunting. I am happy about what I have
completed in April 2020, I have been
blowing experience, and even add an
achieved and can’t wait to share it with
working on it until July, Now I think it is time
additional layer to everyday objects. The
more people.
to end this project and document
idea of combining these two things
everything I designed and created in the
makes MOMENT come to real life.
I also wanted to say thank you to people who have helped me, inspired me and
past 11 months. Throughout the whole journey, I
encouraged me while I was working on
"Put users first " is my mentality as a
encountered a lot of challenges. From a
this project:
designer, while for my thesis, I decided to
conceptual level, memory is such a
take this opportunity to create something
common topic and everyone I talked to
meaningful for myself. As a person who
has their own understanding. As the
likes reminiscing a lot, I think there is
designer of this project, I need to hold my
nothing more fascinating than memory: It
opinion strongly and figure out the right
can be both blurry and vivid. It is distorted
path. From a technical standpoint,
but also accurate. it cheers me up when I
creating outdoor AR experience in Unity
feel in blue and inspires me to keep
can be frustrating and takes a lot effort. In
creating. Meanwhile, as a big fan of
the meantime, I also need to strike a
Augmented Reality, the power of AR can
balance between thesis, work and job
Vivian Wong, Alexander Manu,
Job Rutgers, Emma D’Souza,
Setayesh Babaei, Enna Kim,
Vicky Zheng, Fandongge Wu
Anran Zhou July 15, 2020
78