Tie Memories to E-souvenirs: Hybrid Tangible AR Souvenirs in the Museum Yixiao Kang
In Collaboration with
kyx999@sjtu.edu.cn
Shanghai Science & Technology Museum
Zhenglin Zhang lyric_1244@sjtu.edu.cn
Meiqi Zhao
zhaomeiqi0623@sjtu.edu.cn
Xuanhui Yang yxh-v-7@sjtu.edu.cn
Xubo Yang
yangxubo@sjtu.edu.cn
Digital ART Laboratory of Shanghai Jiao Tong University
Introduction
Design
Traditional physical souvenirs in museums have three major limits: monotonous interaction, lack of personalization, and disconnection to the exhibition. To conquer these problems and to make personalized souvenirs a part of the visiting experience, we create a hybrid tangible Augmented Reality(AR) souvenir that combines a physical firework launcher and AR models. An application called AR Firework is designed for customizing the hybrid souvenir as well as interactive learning in an exhibition in the wild. Multiple interaction methods including mobile user interface, hand gestures, and voice are adopted to create a multi-sensory product. As the first research to propose tangible AR souvenirs, we find that they establish a long-lasting connection between visitors and their personal visiting experiences. This paper promotes the understanding of personalization, socialization and tangible AR.
AR Firework system = Application + QR code generator & printer + online library Designed for an exhibition in Shanghai Science and Technology museum.
2.1 The visual experience: AR technology AR supported with ARFoundation SDK is used in various scenarios, including interaction with the exhibits, presentation of the 3D molecular models, simulation of the experiments, fireworks design process. Meanwhile, AR is used to simulate experiments and visualize abstract concepts.
1 The hybrid souvenir with DIY firework The hybrid tangible AR souvenir include both physical objects and digital information.The physical part retains the formal sense of owning a souvenir. The virtual part contains AR models and audio that serve as a reminder of the visit. The virtual part is made by the visitor during his/her visiting process.
1.1 Picture-based fireworks generation algorithm. 20,000 particles are used to simulate each firework. Users can take photos and convert them into part of their artworks. In this way, memories are evoked when users open the souvenirs again.
Story Board
1.2 DIY AR Firework models 1. Download My Firework.
2. Be guided by cartoon character.
3.Interact with exihits using AR object recognition technique.
After making the fireworks, users can decide the places to set the fireworks like around a crystal ball, etc.
2.3 The auditory experience: recording voice
2.2 The tactile experience: physical firework launcher and hand gesture interaction Natural hand gesture detection supported by ManoMotion SDK is used to trigger the launch of firework. ManoMotion support five hand gestures including pick, pinch, click, grab and release. We finally choose release as the trigger gesture, because setting off fireworks on users hands look like magic. We choose the pinch gesture to control the rotation conveniently and precisely. Controlling time span through thumb and forefinger contact is the most agile and natural gesture in the five gestures provided by ManoMotion. The model rotates clockwise when users pinch their fingers. It stops rotating when users release their fingers. Therefore, users can view AR models from multiple directions easily.
Users can record and bind their voices to the souvenir. It can be a blessing for a friend or feelings about the exhibition.
Usage Scenario
1.3 Bind voice to souvenirs 4. Design AR firework and record voice
7. Buy a physical firework launcher from the shop and generate a QR code.
5. Do AR experiments to learn the 6. Save the scene and upload knowledge about fireworks. it to gallary.
8. Users can view the hybrid souvenir(launcher, AR model, audio) by scanning the QR code.
Visitors are encouraged to record their voices and bind them to the souvenirs. It can be their feelings towards the visit or blessing if they decide to give it to others. This voice message will be played along with the firework show.
9. view the artwork in the gallery and share it.
2 Multiple-sensory interaction and augmented reality
Conclusion • We proposed a hybrid tangible AR souvenir that has great potential in creating a personalized and interactive visiting experience.
• We apply different interactive technologies, including AR, hand gestures and voice to make a multi-sensory souvenir. We discuss the design and report on the deployment of the system in the wild.