Music : wearable Sound possibility Connecting music with people with hearing impairmentw
Yue Hao
Page:1
Content
01 02
03
Page:2
Introduction
User-centric design
Data driven design of the experience of music
1.1 Abstract 1.2 Project design motivation 1.3 Project idea 1.4 Project approch 1.5 Project field 2.1 Survey and interview motivation 2.2 Deaf people’s understandings of music 2.2.1 Deaf people’s music experience 2.2.2 Findings of Msic experience 2.2.3 Alternative hearing methods 2.2.4 Findings of alternative methods 2.3 Finding the design direction 2.3.1 Cutaneous and visual senses 2.3.2 Design direction 2.4 Use cases of the design project 2.4.1 current problems 2.4.2 Intended place of use 2.5 User-centric design conclusion
3.1 Data driven design motivation 3.2 Analysis of sound waves 3.2.1 Volume anlysis 3.2.2 Pitch anlysis 3.2.3 Rhythm and beat anlysis 3.3 Translating sound elements to experiences 4.2.1 Translating volume and rhythm to light 4.2.2 Translating pitch to vibration patterns 4.2.3 Using beats in light and vibration 3.3 Testing the experience body perception in different body parts 3.4. Reflections on the data driven desgin
04 05
Speculative design of the product
Design prototype
06 07
Conclusion and outlook
Appendix
4.1 Motivation of the speculative designs 4.2 Design concept “fancy sticker“ 4.3 Design concept “inconspicious wearables” 4.4 Design concept “comfortable shirt” 4.5 Discusion of the speculative designs
5.1 Circuit design 5.2 Used elements 5.3 Process of fabrication 5.4 Final prototype music shirt
6.1 Future work 6.2 Conclusion
7.1 Deaf world 7.1.1 Deaf people taking part in music activities 7.2 Problems and opportunities related to understanding music 7.2.1 society,emotion lack 7.2.2 Existing device limitations 7.3 Design precedents 7.3.1 Music interaction 7.3.2 Facial expressions of sound 7.3.3 Feeling music 7.4 Participants in surveys and interviews 7.5 User experience of the design project 7.6 Research of audio siganl to FFT 7.7 Details about my music processing situation 7.8 Final CPB board 7.9 Reference Page:3
01
Page:4
Introduction
1.1 Abstract Music is enjoyed and shared by people all over the world. While music is meant to be enjoyed by all, people with hearing impairment cannot participate in the experience. The current assisting devices like cochlear implants or hearing aids are not suitable for listening or understanding music, especially in social settings like concerts, clubs or parties. In this project, I seek to find new ways of enjoying music that do not involve hearing but rather other senses to make it more accessible to people with hearing impairments. I designed a wearable device that helps people with hearing impairments process music in a non-traditional way, in particular by feeling it and seeing it. The device uses microphones to record the surrounding sounds live. The sound data is then analysed by an Arduino to identify the pitch, volume and rhythm of the music. I mapped these features of the music into different levels of vibrations, different colours and different levels of illumination, i.e. lights appearing on the device. The different sounds then create different and immersive feelings across different sensors. People with hearing impairments will then be able to sense the music in a live environment with others and share their experience. Wearing the device they will be able to better understand others’ emotions and reactions to the music without a delay and make new social connections. Last, this device could help people with hearing impairments to be more confident in public areas and increase their self-esteem when attending social events.
Page:5
Page:6
Final product
music shirt vedio available: Page:7 https://vimeo.com/429514150
1.2 Design Motivation EMINEM Concert 24.2.2019
I attended the Eminem concert in 2019. He is one of my favorite singers. But at that time my ears became inflamed and I got otitis media. Listening to high notes and loud noises was very uncomfortable. I didn’t want to give up the opportunity, so I took earplugs to the concert. But I could not hear the lyrics or tunes very clearly and it was quite uncomfortable. For the first time, I learned about the world of people with hearing impairments. From that point in time I thought of ways to help people with hearing impairments to understand music through other ways than hearing.
Page:8
Page:9
Page:10
1.3 Project Idea People without hearing impairments enjoy a plehtora of musical activities. Most people have become accustomed to think that music is a normal part of life. They can distinguish songs, experience the emotion of songs and share their feelings with others. But how do listeners experience music if they are hearing impaired? Deafness can affect people from their early ages, i.e. from before five years old in the case of congenital deafness. Because they can’t hear speech, their language center is degraded and they are thus unable to learn a language. This also translates to people with hearing impairments not being able to understand music, because their corresponding region in the brain was never engaged and degraded. They hence can’t understand the feeling of music. However, this region of the brain was shown to be replaced by a heightened sense of touch for deaf people [Shibata, D. Brains of deaf people “hear” music.]. They can better feel vibrations transmitted through the air or other physical media (such as floors, tables, and machines) from sound than others, which provides a possible explanation for how deaf-mute musicians and deaf people perceive music and enjoy music activities. Through investigation and research, the mechanism of physically “feeling” music may provide an experience for the hearing impaired. However, few studies have specifically focused on how to create a music experience specifically designed for people with hearing impairments. Therefore, in this project I designed and developed a system that records and analyzes live music and then translates it into an experience that people with hearing impairments can access through a wearable. Through research, surveys and interviews I learned that vibration and lights are the most suitable sensors for letting people with hearing impairments feel the translated music. I hope that people with hearing impairments that experience music through my designed music wearable will be able to enjoy music and share their enjoyment with others.
Page:11
Sound
Microphone
Computer
Analysis
Pitch
Volume
Rhythm
Microchip
Colors
Light
Vibration
Human
Shirt
Page:12
Stickers
Bracelet
1.4 Project approach
How does my device work: First, the microphone records real-time sound and sends the data to the Arduino. In the Arduino, I analyze the recorded sound data to derive pitch, volume and rhythm of the music. Through interviews and user experience tests, I designed a mapping of volume and rhythm of the music to the color and brightness of a LED stripe. Further, I designed a mapping of different pitch frequency ranges to three vibration motors. Specifically, I consider a frequency range for the bass range of instruments, the melodical range and the harmonical range. Through different vibration modes people can feel the strength of the different music frequencies. It is then possible to understand that different musical instruments are playing. The vibrations, colors and lights will create an immersive experience that will allow people to understand music. Finally, I designed three products, a T-shirt, circuit stickers and bracelets as possible wearables that can present the music experience with the vibration and light sensors. Each designed wearable has its own strengths and weaknesses. Finally, I decided to create the music shirt as the final prototype for this project.
Page:13
design topic
assistive technologies
people with disabilities
design field
Circuit Design(IC)
Data-Driven Deisgn (DDD)
User-centered design(UCD)
electronics engineering
real-world data
iterative de-
particular logic
sign preocess
circuit design techniques integrated circuit
Analysis
Page:14
Design
signal processing
human-centered computing
testing the experiment
auditory feedback
Evaluation
Implementation
1.5 Project field In my project, I want to design an assistive technology for helping the hearing-impaired people to understand music. I take a user-centered approach to find the possibilities and requirements for the assistive technology with the help of a survey and interviews. In addition to the identification of requirements, I also took an iterative design process by testing my design with participants and then changing the system according to their feedback. In particular, I conducted a first test with users and collected their feedback about their experience. Based on their feedback, I changed the product’s system and code. Then I conducted a second test with them to see if their experience improved. This project also takes on a data-driven approach since the presentation of the music experience relies on the analsysis of the recorded sound. The input data is real-time and real world data. I implemented signal processing methods to derive the three features of music, its pitch, volume and rhythm. After continuous testing and improvement, the final results for the three features is in line with the actual music. In order to achieve a successful product design for the wearable device, the project makes use of circuit design. Circuit design combines the knowledge of circuit engineering with programming to achieve complex design solutions. In this project, I designed a circuit that connects a microphone to an Arduino that analyzes the sound with a C++ program. The Arduino then uses a multidriver controller to then display the music experience over 6 vibration motors and 1 led stripe. Throughout the project, my design process is user-centered design. I first analyze the user’s current problems and their music experience. Then, I analyze the current problems and solutions. Based on my findings, I design or adapt my product. Then the user experiences my product and proposes changes. I improve my product and then repeat the steps to improve the design based on user experience. I believe a user-centric design to be the best approach for this project since music is very complex and a very subjective experience o different listeners. By gathering the experiences of different participants, I try to find a common understanding of music that is shared among listeners and is important to be a part of the music experience of music : wearable. Page:15
02
Page:16
User-centric de
esign
Page:17
Survey
User experience
Interview
User story
Page:18
User existing Problems
Define target audience
demands of users
Define product placement location
Design direction
Define user desired outcome of the new functionality
2.1 Survey and interview motivation
In the user-centered design project, I used a survey to understand the people with hearing impairment’s music experience. The survey was designed to discover the current problems, the needs of users and to determine my design direction. Additionally, I used interviews to listen to the personal stories of the participants and to find the different needs of potential users. In particular, I used the interviews to identify my target audience and to determine the product placement, suitable occasions to wear my product and the user’s desired product functions.
Page:19
2.1.1 Music experience of deaf people to understanding what is deaf world,what is deaf people’s music experience
I asked participants to fill out a standardized questionnaire. I received responses from 16 people, aged 12-55 years old, of different genders and suffering from different degrees of hearing impairment. Among them are some deaf and severely deaf participants. All participants have normal vision and sensory feelings. Regarding the question, have you ever participated in a music event? 10 people chose to participate in movie activities, 2 in music festivals, 5 in concerts, 6 people participated in bars, and 4 people chose to participate in music fountain events (fountains with light shows that react to live music).
If you can hear music, you think it will help
mental emotions
100%
73%
job opportunity
20%
20%
social activities
73% 0
2
4
6
8 not helpful
Page:20
10 helpful
12
14
16
18
To the question “if you can hear music, you think it will help social activities, job opportunities or mental emotions�, all participants responded that their mental emotions could improve. 11 people think that understanding music would not be helpful for job opportunities, 3 people think it is helpful. In contrast, 11 people think that if they could listen to music, it would help their social activities.
scale of be disappointment about not being able to enjoy a music activities as mush as they would like to 12%
19%
44%
25%
0% not disappointed
a litter disappointed
very disappointed
extremely disappointed
disappointed
The frequency of listening to music for a week everyday 12% 3-5 times per week 13% less or never 56% 1-3 times per week 19%
everyday
3-5 times per week
In the question “are you disappointed not to hear music?”, the vast majority of participants (6 people) chose extremely disappointed (44%). 4 people were very dissappointed (25%). 3 people were dissappointed (19%). Only 2 people chose not to be disappointed (12%).
1-3 times per week
less or never
In question “do you often listen to music?”, 2 people chose to listen to music every day. The number of people in this selection accounts for 12% of the total number. 2 people chose to listen 3-5 times a week. Three people chose to listen 1-3 times a week. Accounted for 13%. Eight people chose to listen to little or no music. The proportion is based on the total number of people 56%. Studies have shown that the vast majority of people with hearing impairments rarely or never listen to music. There is a serious lack of music in their lives.
Page:21
Page:22
2.2.2 Findings of Music experience
The above survey data shows that the vast majority of people have participated in film activities. But relatively few people have participated in music festivals or concerts and bars. Acitivites linked to music were avoided by people with hearing impairments. In addition, people with hearing impairments would rarely or never listen to music in their daily lives. While music is an essential part of our daily lives, people with hearing impairments cannot access the beauty of music. They can’t understand the reactions of people towards music. This also makes the vast majority of people with hearing impairment feel extremely disappointed not to hear music. Therefore, all people with hearing impairment agreed that if they could listen to music, they would definitely benefit by improving their mental emotions. A large number of people with hearing impairment think that it would also help their interpersonal communication and social relations.
Page:23
2.2.3 Alternative hearing methods to understand the current problems about what products are currently used by deaf people and what useful products are
frequently used assistive assertive device in music activities other graphical display feel the vibration of sound facial expression performer body language subtitle display 0
2
4
6
8
10
12
14
In the question for frequently used assistive devicex in music activities, the graphic display is the most frequently selected device by 12 people. The second most frequent medium is to feel device vibration selected by 10 people. Third, 9 people thought that the body language of the performer could also help to understand the music. 8 people found the the display of subtitles to be helpful. Finally, the least helpful medium to understand music were facial expressions selected by 5 people.
Page:24
the help of different devices in music activities 80% 70% 60% 50% 40% 30% 20% 10% 0%
sound amplification
subtitle display
sign language vibration devices graphical display
The chart above illustrates the type of assistive devices hearing impaired people have used while engaging in a musical activity and whether they were deemed useful. For this question, participants were asked to rank the different devices on a scale from 1 to 5 with 1 being very helpful and 5 being not helpful at all. The results were processed for presentation to a scale from 0 to 100% where 0% indicates a device not being useful. Deaf people think that the most helpful device for understanding music is vibration equipment. The average ranking accounted for a usefulness of 73%. People think that the second most helpful medium are visual displays, which accounted for a usefulness ranking of 70%. The third and fourth most useful mediums are the subtitle display and sign language. They report a usefulness of 60% and 56%, respectively. The most unhelpful medium was the sound amplifier (20%).
Page:25
Page:26
2.2.4 Findings of alternative methods The survey concludes that assistive technologies can have a positive impact on the understanding of music for deaf people. The vast majority of respondents often use a visual display of music and some believe that feeling the vibration of the sound and the body language of performers are helpful to understand music. The most helpful devices for understanding music identified by the participants, however, are vibration and visual instruments. The current choice among these kind of devices is very limited. The survey results show that the current equipment used by deaf people does not meet the needs of deaf people to listen to music. There have been some studies investigating ways to help deaf people experience music, but few have studied the design of new equipment to enhance the music experience for deaf people.
Page:27
2.3 Finding the design direction 2.3.1 Cutaneous and visual senses To understand more specificly about the problems and comprehension when deaf people use vibration and light displays to access music
When using the vibration mode for the phone, set different vibration modes for different incoming songs. Do you think you can distinguish songs by different vibration feelings?
19%
81%
Yes
No
In the question “when using the vibration mode for the phone,set different vibration modes for different incoming songs.do you think you can distinguish songs by different vibration vibration feelings?”, 13 people chose yes (81%). The vast majority of people with hearing impairment think that they can distinguish songs by different vibration displays on the phone. Some additional textual responses were “Can distinguish between songs; Can distinguish intense rhythm rock music from slow light music. Can feel the vibration“. So, the intensity of vibration is one of the keys to distinguish between different songs. Another person wrote, “I would need to learn how to distinguish songs by vibration.” For people with congenital hearing impairments there need to be some explanations and rules for the meaning of different vibrations so that they could understand music.
Page:28
If you have ever seen a music fountain or a light show, do you think you can understand the song through the change of light? 25% 75%
Yes
No
For the question “if you have ever seen a music fountain or a light show, do you think you can understand the song throung the change of light?”, 12 people chose to feel the change of music through the change of light of the music fountain (a fountain that reacts to live music with lights). There are 4 people who didn’t understand. A German participant wrote to why he could not understand: “I think the presentation of music fountains is beautiful, but I do not understand how the lights represent the music. For example, what does it mean for the music if the light changes from a red to a blue color?” The change of lights is a very interesting experience for people with hearing impairments. He thinks that the light is related to the music. But he couldn’t understand how the change of color would be related to the music. Another person wrote: ”I would need to know the meaning of the different colors”. Therefore, we can find that the hearing impaired people’s understanding of the display of light is less intuitive than that of vibration. Many people are not directly connected to changes in music for changes in lighting. There needs to be a certain degree of relevance and explanation to the meaning of the color changes. Page:29
Page:30
2.3.2 Design direction
Based on the above findings, the visual display and physical sensations are the most helpful ways to replace the hearing loss. So, I decided to use lights and vibration as a music experience for the functionality of my product. The visual selection of colors and the brightness of visual lights will be given to the design. The vibration experience can be designed around the vibration frequency and its amplitude. Finally, a last design decision revolves around the positioning of the vibration motors on the human body for different vibration experiences.
Page:31
2.4 user cases of the design project 2.4.1 current problems
interviewee 1 male middle-age German weakened hearing hearing aid user
Among my interviewees, there is a middle-aged man of German nationality. He had weakened hearing caused by cochlear lesions. He is not a patient with congenital deafness, he still retains part of his hearing. However, when using a hearing aid, the sound may be distorted and part of the sound may be lost. He talked about enjoying drinking at the bar. Many songs that used to be familiar to him are now tormenting to him as he needs to listen to a song many times before he can notice its melody or to find its rhythm. He is using a hearing aid all the year round and is hoping that his hearing will not continue to decline. But hearing aids can’t help him to capture low frequency sounds well. He has already struggles while listening to people. Listening to music is almost impossible to understand as it is complex and not clear only with the help of a hearing aid. If there were a device that could help him understand the beats of a song, or follow the melody of his familiar songs, then he felt confident to slowly re-adapt to the songs he used to know. He thinks it would help his entertainment life. In particular, he could be better integrated into the environment, could memorize his favourite songs and talk about them with others.
Page:32
interviewee 2 male young age chinese sensorineural deafness cochlear implant user
Among my interviews, there is a young male interviewee of Chinese nationality. He has congenital deafness since he was a child. The cause of his deafness is sensorineural deafness and the treatment is very difficult. He was implanted a cochlear implant as a child to keep his language center intact. He usually likes to watch movies, because in the movie he could understand the music through the actors’ facial expressions and the visuals of a scene. But listening to radio or pure music activities is much more difficult as he uses an old-fashioned cochlear implant. When he listens to songs the sound of the song is often inseparable from the sound of the environment. He said that today the design of cochlear implants is mainly for speech recognition, but there are still limitations to improving music perception. Since he was implanted a cochlear implant, he has been doing rehabilitation training. In particular, he exercises the ability to recognize different sounds. He said it was difficult for him to recognize changes in syllables (do re mi). Therefore, he hopes that one day he can feel the change of syllables, and that he can really feel the beauty of music.
Page:33
Envy Nightlife Club, Albuquerque, New Mexico ,https://www.flickr.com/photos/i5design/6300653372
Ultra Music Festival 2009, Photo by Robert Giordano, https://www.pxfuel.com/en/search?q=music+festival
Page:34
a girl wear ballet dress.pxfuel https://www.pxfuel.com/en/
2.4.2 Intended place of use Clubs and music bars Clubs and music bars are places for most normal hearing people to relax after they get off work or after school. People will dance, talk and relax with music. But people with hearing impairment rarely go to clubs and music bars, because they can’t understand music. Even if you participate in bar activities, you can’t feel the atmosphere or experience a strong musical experience like normal people. Therefore, deaf people could make use of a set of equipment that can help them understand the intensity of music and the rhythm of music, that can be worn with them. It would help them integrate into this social environment. At the same time, the bar owner could also provide the deaf people with this equipment in order to attract more deaf people to come and be able to join the bar and music activities.
Music festivals A music festival is a place many young people are keen to go to and thousands of music festivals around the world attract tourists from all over the world every year. There are already many music festivals that have hired sign language guides on site. They use sign language to tell the lyrics and use expressions and body language to tell the feelings of the music to the hearing-impaired people. So, a device that allows deaf people to experience music in additional ways through vibration and lights can be very useful and helpful to capture the feeling of music festivals, encourage a participation in dancing activities or exchanging their feelings with other festival participants.
Dance class Dance is a challenge for deaf people and it is a particularly difficult task for deaf people who have auxiliary equipment and want to dance. Hearing aids are there to help amplify the sound, but it may sound fuzzy and some ranges of the sound are not very clear. Cochlear implants are still relatively easy to listen to music but there are other problems. While dancing they also have to grasp the rhythm while listening to music. Once they miss a certain rhythm, they will be disrupted. This is why conductors continuously have to remind students of the rhythm by clapping on their legs. An assistive technology that communicates rhythm and intensity of the music would help dance classes to better engage deaf people in the activity of dancing. Page:35
2.5 User-centric design conclusion
Survey
User existi Problems
User experience
music experience
Psychoemotional needs
music activities locations
user disappointed can not access music
music activities count
Interview
User story
From the survey, I learned about the user’s music experience, music activity venues and the number of music activities. I found out more about the hearing impaired people’s psychological needs and equipment limits for not being able to participate in music activities. From this, the design direction was derived to be an assistive device that combines vision and haptic functions.
Page:36
Define target audience
T m u
ing
demands of users
s
Design direction
Device restrictions
Emotional needs
Assistive device
The traditional methods often used can only help to understand music with limited
Social activities
The most helpful ways are vibration and visual display
Define product placement location
Mental emotion
The new assistive device
Combination vibration and visual display
Define user desired outcome of the new functionality
From the interviews, I listened to the personal stories of the participants and found out more about the different needs of the potential users. From that, I could identify the target audience for my product design, determine the product placement locations and determine the user’s desired product functions.
Page:37
03
Page:38
Data driven design
of the experience of music
Page:39
3.1 Data driven design motivation
Music is a complex experience with a myriad of facets that we enjoy. Some of these facets are the volume, the rhythm and the pitch of the music. Our ears and our brain do small miracles to extract these facets from the sound we hear. In this project, we need to simulate this processing of the sound with a computer, i.e. with an Arduino. I decided to use a microphone to record the surrounding sound and then analyse it with the Arduino. If the product can support live sound over pre-recorded sound it is much more applicable to be worn in settings where the pre-recorded music is nor available, i.e. in bars or in dancing clubs. It can hence give more flexibility to its wearers. Next I present the analysis of the recorded sound to extract desirable features of the music: (1) the volume represents how loud the music is playing, (2) the rhythm tells us how fast the music is moving and (3) the pitch covers the development of the melody within the music. Last, I will present how I mapped the three analysed elements of the music to the lights and the vibrations of the product to represent the music for its weareres. I used an iterative approach to confirm the design decisions with testers and changed the designs where found necessary
Page:40
Page:41
3.2 Anlysis of sound waves How music is recorded physically as a sound wave: Music can be captured by recording sound with a microphone. To be able to analyse the sound for the features of music, it is first necessary to understand the basic properties of the recorded sound. Physically speaking, sound is pressure in the air. It is not constant, however, as it continuously increases and decreases and thus forms a wave. The recorded sound from a microphone is hence a series of pressure recordings (measured in decibel (dB)) over time. A sound wave can be plotted with time on the x-axis and pressure on the y-axis (dB). The figure below shows a recorded sound wave of my Arduino implementation.
Sound wave captured on the Arduino
Page:42
How the features of music are manifesting in a sound wave: In music, we appreciate certain aspects of a sound wave. How far the sound wave spreads from its centre represents how loud a sound is, i.e. it represents the volume of the music. How fast a sound wave completes one cycle, i.e. how fast the pressure rises then falls and then returns to its centre once, represents the frequency of the sound wave. The higher the frequency of a sound wave the higher we perceive the pitch of a sound. We commonly use the pitch to follow the melodies within songs. The rhythm reflects itself as strong impulses in the sound wave. These impulses are the beats of a song that commonly tell us how fast the music is moving. The figure on the right demonstrates how the different features of music can be identified in a sound wave visually.
Page:43
3.2.1 Volume Analysis The Volume can carry several messages of the music and is an important part of a song: First, I focus on analysing the volume of the music. But before that I want to focus on the question why the volume is an important part to understand music. The volume of music expresses how loud we experience sound. Different volumes can represent different feelings or settings of music. Loud music oftentimes signals an exciting, powerful or motivating setting of a song. Quiet music can represent peaceful, emotional or spiritual moments in a song. Dynamic changes in music can further change the meaning of a song. If the music increases in volume, the song is trying to create an uplifting moment. A decrease in volume represents a calming moment. The volume of song can carry many emotional messages of music and can thus greatly contribute to the understanding of music.
Page:44
Arduino Volume recording Beethoven’s 9th Symphony
Arduino Volume recording Harry potter theme
I detect the volume as the root mean square amplitude of a sample of sound recordings: I followed a method described by (Arik Yavilevich, 2017) to determine the volume of music with an Arduino. The volume needs to be calculated with a sample of sound recordings to get a steady and reliable analysis. As discussed before, the volume can be determined by how far the pressure of the sound wave deviates from its centre, i.e. its average value. Determining the volume takes several steps: Take the absolute difference from the mean pressure of the sound wave for each sample recording. The volume can be determined as the root mean square value of all sampled values after step 1. The root mean square needs to be taken relatively to a reference value specified by the microphone specifics. The figure on the left hand side shows how to visually detect the volume. The figures above show the volume in decibel for two songs determined by my implementation on the Arduino.
Page:45
3.2.2 Pitch Analysis and FFT Melodies in music are defined by the pitch. A soundwave contains several pitches at the same time. I use three pitches to present music: Second, I focus on the analysis of the pitch in music. The pitch of a single note describes how high we perceive the note to be played in a melody. The pitch hence captures our melodically understanding of music. It allows us to follow the melody in a song. In a sound wave, the pitch of the music can be linked to the frequency of the sound wave. Usually, music doesn’t only consist of a single note but of several notes from several instruments at the same time. As a result, the sound wave can comprise several frequencies from the different notes being played. I use the Fast Fourier Transform (FFT) method to separate the different frequencies present in the sound wave. However, the FFT can only resolve ranges of frequencies depending on the quality of the microphone and the processing power of the computer. In my Arduino project, I used three different frequency ranges, because they suitably represent distinct instrument ranges: 250-500Hz: Drums, Bass, Piano, low male voice (blue) 500-2000Hz: Most instruments, Vocals (red) 2000-4000Hz: Harmonics, high string instruments (green) The figures below show the amplitude of the three frequency ranges over time recorded from my implementation on the Arduino.
Page:46
I use the Fast Fourier transform (FFT) method to detect the different pitches in the sound wave: The FFT is a measurement method that transforms the sound wave from the time to the frequency dimension. By that it determines which frequency in the sound wave has which strength, i.e. amplitude. The result of an FFT analysis can be plotted with the frequency on the x-axis and amplitude on the y-axis. The figures below show an audio signal and its corresponding FFT analysis. Audio signal
FFT Anlysis
The output of the FFT analysis is the amplitude for each frequency. This can be used to identify the most important frequencies in the sound wave by ordering them according to their amplitude and to determine their relative importance in the sound. Since the frequencies relate to the pitch in music, the FFT enables us to learn which instruments are important in the sound. However, the FFT can not single out single frequencies but only frequency bins that span several frequencies. How many frequencies span one bin depends on the hardware to record and process the sound samples. The figure below shows a sound wave deconstructed into the three frequency ranges. The plot spans across three dimensions: time on the x-axis, amplitude on the y-axis and frequency on the z-axis.
100
10 13 e
16
19
22
25
fre
Series1
Series2
Series3
qu
Series4
en
cy
5
7 tim
S eries
0 1 4
S eries 3
50
Series 1
amplitude
Amplititude of 3 Frequency bands over time
Series5
Page:47
3.2.3 Rhythm and Beat Analysis
Rhythm and Beats are an important part of music: Last, I focused on the analysis of the rhythm of the music. Rhythm in a song expresses the speed of the movement of the song. It signals listeners how quickly the melody changes. An important part of the rhythm are its beats. Beats are emphasized notes that create a common understanding of the rhythm of a song. Beats are commonly used by the listeners to dance to the music and thus are an important part of a song.
Page:48
Arduino Beat detection For a dance song
Arduino Beat detection For a rock song
Beat detection in the sound wave: To detect the beats of the music, I adapted the technique described in (Tzanetakis et al. 2001). In contrast to their approach, however, several steps could not be accomplished on the Arduino due to a low processing power of the Arduino and the microphone being prone to noise in the low frequency ranges. For that reason the beat detection was performed on the original sound wave instead of the different frequency bands. Intead, I followed the following approach inspired by their approach: Beats are strong impulses in the sound wave that are greater than 99% of the previous sample. Beats can then be detected as follows: Take the absolute difference from the mean of the soundwave for each sample recording Differentiate the sample by taking the difference between each recording and its past recording Determine a threshold for detecting beats by finding the amplitude where 99% of the sample haver smaller values. An impulse is a beat in the next sample, if it is larger than 99% of the values of the previous sample. Page:49
3.3.1 Presenting Volume and Rhythm with Light 1st Idea: Use Volume dynamics of music to define a single colour For a quiet volume use three LEDs For loud volumes use five LEDS When a beat happens shortly light up all eight LEDS User Experience Feedback for the 1st idea: Colours flash uncomfortably with quick volume changes Setting all LEDS to light for quiet setting felt too much 2nd Idea: Use a rainbow colour gradient from green to blue to red to green Use volume dynamics to define brightness Use a dynamic number of LEDs based on how the relative level of the volume changes Use dynamic level of the beat to increase 1-3 LEDs User Experience Feedback for the 2nd idea: Rainbow gradient is comfortable Dynamic increases of volume and beat feel more natural With some time it could become possible to understand the music without listening to it.
Page:50
Volum e Dynam ics
1st
Idea
Music Dynami cs
mediu m loud
loud
very loud
Rhythm
55-60
60-65
65-70
>70
detected
orange
blue
yellow
gre en
same
Explan ation
quiet
mediu m quiet
Decibel
<55
Color
red
3
Color
Idea
Beat
piano (p)
mezzo forte (mf)
#LEDS
2nd
forte (f)
forti ssi mo (ff)
mezzo piano (mp)
5
8
rainbow gradient : green-blue-red-green
Brightn ess
30%
#LEDS
3-5: change dynamically based on previous level
40%
1st idea prototype
1st volume ranges with beat
50%
60%
2nd
70 %
+1-3 relative to volume
idea prototype
2nd volume ranges with beat
Page:51
3.3.2 Presenting Pitch with Vibration
Translating pitches to vibration and positioning the motors: I used three pairs of vibration motors for the three frequency ranges: The lower pair of motors represents the Bass instruments in the frequency range 250-500 Hz. The middle pair of motors represents instruments and the melody of a song in the frequency range 500-2,000 Hz. The high pair of motors represents the harmonics of a song in the frequency range 2,000-4,000 Hz. Each frequency range has its own amplitude, i.e. how loud or important is the frequency range in the song. I present the different frequency ranges then by choosing different strength and vibration types according to their loudness. Each frequency range has three different vibration modes: 60% strength sharp fuzzy click for quiet volume (i.e. <55 dB). 80% strength short double click for medium volume (i.e. <65 dB). 100% strength strong pulsing click for loud volume (i.e. >65 dB) . The pairs of the vibration motors are each positioned one on each side of the spine for a total of 6 vibration motors. The low frequency pair of motors is positioned in the waist area. The middle pair of motors is positioned upwards on the back and the high pairs of motors is positioned above the middle set of motors.
Page:52
Page:53
3.4 User experience to achieve the final prototype Setup of the User experience I conducted user experience tests with three participants by letting them wear my prototype and collecting their feedback. I did the tests with the following three participants: Daniel Reissner is a PhD student at the University of Melbourne studying Computer science. He is playing the saxophone for more than ten years and has a good musical understanding. Xin Zhang is also a PhD student at the University of Melbourne also studying Computer science. He has experience in electrical engineering and built several complex circuits before. I also did the experiments with myself. I am Yue Hao, a Master student at RMIT. I have experience with electrical engineering, have experience with hearing problems due to heavy ear infections and have also musical experience. I play the flute for more than 10 years.
Page:54
Results of the User experience: Light experience:
It is possible to follow the songs by just observing the lights. The second light solution was found to be more comfortable. The changes of the lights are intuitive. The beats can be recognized. Participants would be happy to wear the lights to a club or a music festival to attract attention.
Vibration experience:
Having several vibration motors makes the music experience very immersive. Several vibrations along the back are complex to understand, but it suits the complex nature of music. Much more engaging than existing solutions. The vibrations made the participants want to dance. The vibrations are comfortable and not annoying. The different strengths of the vibration motors can be distinguished clearly.
General experiences:
Vibration and light work well together. The music becomes a whole body experience. If given enough time, the patterns of light and vibration can be learned and the music can be understood without listening to it. All participants agreed that the product is very interesting and would be happy to wear it. The transcripts of the user experiences can be found in the Appendix.
Page:55
3.4.1 Testing body parts for sensitivity of vibration Testing the sensitivity of body parts to vibration As shown by the figure on the left, I used the Apple watch to test the sensitivity to vibration for various body parts. The tests were done by two of the participant, Daniel and me. We found that the most pleasant were behind the ears and at the back. However, the part behind the ears is to small to be accessed by vibration motors and was then left out for the final prototype.
The two figures show comfort level of the haptic experience of the two test subjects from the vibration experiments at the different body parts. I identitied the head (at the back of the ear) as the most sensitve spot. The back of the waist is the second most comfortable spot. Other candidates for comfortable vibrations are the ankle, the front of the waist and the back wrists with a medium level of sensetivity. The most unsensitive area is the upper arm. Page:56
a)
e)
b) f)
c) g)
d)
h)
The Apple watch can provide a vibration experience from low to high level within 2 minutes with the breathing app. As shown in Figure 3, the apple-watch vibration was placed on eight different body positions, i.e., (a) the side of the head, (b) the back, (c) the back of waist, (d) the ankle, (d) the ankle, (e) the chest, (f) the side of the waist, (g) the upper arm, and (h) the wirst. The dataset 1 was collected (using apple watch sound &haptic breath) from 2 subjects (1 male and 1 female), aged between 28-30 years.
Page:57
3.4 Testing the vibration experience Results of positioning the prototype on different parts of the body: I tested the vibration experience on different body parts for all participants. The back was universally agreed upon to be the best positioning for the vibration motors. Alternatively, the forearms were found to be pleasureable and also found to be flexible since the motors were easy to remove.
infront
Page:58
back
Page:59
05
Page:60
04
speculat
tive design of product
Page:61
Page:62
4.1 music sticker The design music stickers is my first design concept. The circuit components are designed to be very thin and can be directly connected to various parts of the body through circuit stickers. The benefits of this design idea are that the stickers look fancy like a tattoo and can attract attention. Further, the stickers allow for the maximum flexibility of placing the sensors all over the body. However, this is also the most futuristic design as the circuit sticker are a recent development in circuit technology. All parts of the body can be used to serve my design. This wearable design allows for the most comprehensive body experience because of the flexibility of the stickers. There are six vibrating devices on the back and two on each arm. Further, there are two vibration motors located on each leg and the microphone is connected behind the ear. The lighting equipment is placed at the side of the arm. This design can thus give an experience over the whole body. Music can flow from the feet to the neck. The lighting effect can be seen well on the arm. Below, I describe the design process to create the music sticker.
set up human body model
design vibrations and sensors
check the con- design LED nection in full lights and batbody, and pro- ters, patch all ject curve the curve
design connection copper foil tape
extrude surface 0.01cm, offset mesh, fillet edges Page:63
Page:64
4.2 music bracelet
The music bracelet is my second product design. The benefits of this design idea are that bracelets are inconspicuous every-day items. They allow wearers to enjoy the music without drawing attention. Some people with hearing impairments might prefer an inconspicuous solution to not get a negative attention for being disabled. The LED stripe can be turned off on this wearable and the vibration can be weakend. The second benefit of the music bracelet is its achievable productivity. Bracelets are relatively easy to manufacture and produce. In my design, first of all, this bracelet is detachable, and there is a ring of elastic rings in the inner layer. It can be worn according to different arm sizes. Secondly, you can use USB to charge the battery, like an electronic watch. There is a vibrator and an LED indicator in each bracelet. The user can wear it on the bare arm or leg.
Page:65
Page:66
4.3 music shirt
My third design concept is a music shirt. A T-shirt combines the advantages of an everyday wearable with connecting to a large part of the body. It is more inconspicous than the music stickers, but allows to get some attention from others with an LED stripe in the front of the shirt. In contrast to the music bracelet, the T-shirt reaches a wider part of the body to create a more immersive experience of the music. In particular, it allows me to place vibration motors at the back, which proved to be a more ideal location to feel vibrations comfortably during my sensitivity tests. The music shirt is also more easy to produce than the still futuristic music stickers. The design of the T-shirt contains a LED stripe in the front, which can be seen clearly by the wearers. The supporting battery and micro controller are placed at the back in the waist area and can be easily removed. I placed 6 vibration motors at the back in pairs with a significant distance from each other climbing upwards from the waist to the upper back nearby the spine. It is important to have enough distance between the vibration motors such that each vibration experience can be recognized clearly and that they do not interfere with each other.
Page:67
DEAF HAVANA AT TRUCK FESTIVAL 2017 The photo shows the singer wearing a music shirt singing on the main stage.
Page:68
Antoine Hunter, Deaf Dancer The photo shows deaf dancer dancing with a m
music bracelet
Yue Hao, 2020 Photo shows young man using music stickers on his body
Page:69
1.
2.
Advantages
3.
1.
Disadvantages
Page:70
2.
Circuit sticker
Bracelet
Shirt
Circuit stickers are light, 1. paper-thin and flexible circuit boards cut into small, different shapes with anisotropic conductive adhesive on 2. the back. It can be close to the skin, so you can feel the changes of vibration very clearly. Some deaf people do not want to be recognized as disabled by others. They don't want to have obvious signs in appearance. Therefore, the circuit sticker can be well hidden under the clothes without being found.
1. It is very convenient to wear and can be removed at any time when you do not want to use it. It can be used on limbs and can be used as an 2. auxiliary device for deaf dancers.
Can be used as a daily wearable device. Suitable for activities such as music festivals and bars. Popular with young people. Most of the interviewees expressed their preference for shirt equipment. Very trendy and fashion. When the lights flash with music, they think they can attract other people and can increase communication activities with others. Will help them make more friends. In order to increase social activities and reduce inferiority complex.
1. When designing an electrical system, the safety assessment must include an assessment of the impact of human contact with any metal or energized parts or conductors that may be present. Circuit stickers are a very advanced subject. There are very few precedents and research on circuit stickers. Therefore, a large number of tests need to be done in terms of implement abilityă&#x20AC;&#x201A;
Can't show music well. 1. The use area is small, only one vibration sensor can be put in. Use experience is lower than multiple vibration sensors.
Because the vibration device needs to touch the human body and skin to be felt. Therefore, people with different body shapes and different genders need different designs of clothes. Electric shock prevention materials are also required when using fabrics.
Page:71
05
Page:72
05 Design pr
rototype
Page:73
Page:74
5.1 Circuit design I designed the circuit board. From the beginning of the circuit design, I used a sound sensor to record the sound. Then I started my design by using a single LED light to indicate the sound level. I gave a red light when there was sound volume. I used digital readings to measure sound. These readings could effectively measure when sound was occurring. But when I tried to get more complicated data, the digital readings weren’t sufficient any more. So, I started to use analog readings to detect the sound and measure its Amplitude. I started using two LEDs to show the sound. After changing to analog readings, I could show the music with more sensitivity in sound tests. I realized that all I showed was the volume of the sound. But music contains more elements. Like the pitch and the rhythm. So I started to read more articles to implement frequency measurements and beat detection on the Arduino by using the FFT. Through the article “Realtime musical applications using FFT based resynthesis” I learned that noise is an important factor in the measurement process. To address this issue, I purchased an amplified microphone to improve the clarity of sound measurements. After I successfully measured the volume, frequency and rhythm, I started to add more devices. I used a neopixle led stripe with eight LEDs to show the lighting effects. The neopixle offers a variety of lighting effects to design the volume and the rhythm. Last, I added the multi driver adafruit TCA9848A to control multiple adafruit DRV2605L vibration motor drivers. Finally, I added 6 adafruit DRV2605L vibration motors to show different frequencies of music.
Page:75
5.2 Used elements
Fritzing Bill of Materials
7/6/20, 11)11 pm
Bill of Materials: ďŹ nal dritzing.fzz /Users/eve/Documents/ďŹ nal dritzing.fzz Sunday, June 7 2020, 23:10:19
Assembly List
Page:76
Label
Part Type
Properties
C1
Capacitor Polarized
c; panasonic_c
U1
Vibration Motor- ROB-08449
10mm; vibe-motor-10mm
U2
Vibration Motor- ROB-08449
10mm; vibe-motor-10mm
U3
Vibration Motor- ROB-08449
10mm; vibe-motor-10mm
U4
Vibration Motor- ROB-08449
10mm; vibe-motor-10mm
U5
Vibration Motor- ROB-08449
10mm; vibe-motor-10mm
U6
Vibration Motor- ROB-08449
10mm; vibe-motor-10mm
VCC1
Battery block 9V
9V
1
Adafruit TCA9548A
1
2
Adafruit DRV2605L
variant 1; Adafruit #2305
3
Adafruit DRV2605L
variant 1; Adafruit #2305
4
Adafruit DRV2605L
variant 1; Adafruit #2305
5
Adafruit DRV2605L
variant 1; Adafruit #2305
6
Adafruit DRV2605L
variant 1; Adafruit #2305
7
Adafruit DRV2605L
variant 1; Adafruit #2305
8
NeoPixel Stick
variant 2; 1426
9
Breakout Board for Electret Microphone
14
Arduino Uno (Rev3)
file:///Users/eve/Documents/final%20dritzing_bom.html
Arduino UNO (Rev3)
Page 1 of 2
Page:77
Page:78
5.3 process of fabrication The process of making a music shirt
electric soldering
soldering all the connection
Sewing electric element
building the electric circuit
place and sew all vibrations
power suppler in the back
test all vibration circuit board and and cross the cloth power in the back from inside to out-
Page:79
5.4 Final prototype music shirt
Page:80
Page:81
Page:82
Page:83
6.1 Future work
6.1.1 Device limitation
I use Arduino uno to handle my sound reception and processing. However, due to the Arduinoâ&#x20AC;&#x2122;s sampling rate, low resolution, and small memory, I think that in the future, rraspberry pi 4 or a computer can better distinguish different frequency sample sound. Memory size : The Arduino UNO has only 32K bytes of Flash memory and 2K bytes of SRAM. Sampling rates:16 MHz Arduino the ADC clock is set to 16 MHz/128 = 125 KHz. Each conversion in AVR takes 13 ADC clocks so 125 KHz /13 = 9615 Hz. That is the maximum possible sampling rate, but the actual sampling rate in your application depends on the interval between successive conversions calls.
Device limitation
Sampling rates: the sampling rate of Raspberry Pi 4 is 200khz. Memory size: The Raspberry Pi 4 is available with 2, 4 or 8 GiB of RAM.
Details about my music processing situation in this project are written in the appendix. Page:84
6.2 Testing with deaf people
Another regret is that due to the covid-19 epidemic, my entire masterâ&#x20AC;&#x2122;s thesis is in a lockdown situation. The epidemic is very serious globally. The Australian government requires that people can travel for only three reasons and cannot visit other people. Therefore I cannot visit the deaf community during the pandemic. I hope the epidemic situation will get better soon. In the near future, I will visit deaf people to test my products. But this difficult time, I was very impressed and faced various other difficulties. Thank participants very much for accepting my online interview and survey in difficult times, the questionnaire and the academic discussions and suggestions of the two doctors. Two phd students who tested my product. make my products have achieved and achieved good results.
Page:85
Page:86
Conclusion • I designed a wearable technique for people with hearing impaired • I used survey and interview to determine the current problems • I used survey and interview to finding the opportunity and design solutions • I analysis and research sound data in Arduino and computer in pitch, volume and beat • I designed the color code and brightness of neopixel led light • I designed the vibration wave style and intensity of vibration motor • I designed three concept product design “fancy sticker” “inconspicuous bracelet ” and “comfortable shirt” • I design the circuit and fabrication the electric circuit into shirt • I present the final prototype with model Page:87
02
Page:88
07 Appendix
Page:89
Dancing Out Loud: Antoine Hunter Uses Dance To Express The Deaf Experience CHRISTOPHER EGUSA â&#x20AC;¢ MAR 11, 2020
Lynn Q. Yu -April 9, 2018
Page:90
These Sign Language Interpreters Are Working Hard to Make Live Music More Accessible to All
7.1 Deaf wolrd 7.1.1 Deaf people with music according to Nanayakkara, S. (2009). the research of 41 people (20 male subjects and 21 female subjects; 36 of them aged 15â&#x20AC;&#x201C;30 years and 5 subjects aged 31â&#x20AC;&#x201C;45 years) with various degrees of hearing impairment by asking them to complete a standardised survey form.
e e
The authors asked the respondents whether they took part in musical activities: whether they attend concerts or listen to music at home. Seventy seven percent of subjects with partial hearing reported taking part in musical activities, whereas only 32% of the profoundly deaf subjects reported being involved in musical activities (Table 1). This observation supports the hypothesis that the partially deaf are more likely to have taken part in musical activities than the profoundly deaf.
8
l
Page:91
7.2 problems and opportunities related to understanding of music 7.2.1 society,emotion lack
speak a new language
confidence of speaking
increase risk of depression
dementia.blood pressure.heart conditions
self-esteem
enormous social
communication
economic costs
a mainstream health issue
education difficulty
work restrictions
Page:92
(1) The lack of the ability and opportunity to communicate with the environment leads to mental retardation; when communicating with normal people, seeing others talking and unable to communicate with others, they will have a strong sense of loneliness and lack of confidence to speak. 2) It causes abnormal personality development, which is manifested in behavior withdrawal, cowardice, inferiority, and low self-esteem. 3) Due to the influence of hearing and language barriers, there are obvious defects in the understanding. often observe othersâ&#x20AC;&#x2122; expressions and movements, which makes people feel not normal or very sensitive, incresing communication barriers. Due to hearing impairment, they will feel low in self-image and immature social performance. They are afraid of seeing strangers and will choose other hearing-impaired people as playmates, which further alienates them from normal people.(4) Some people with sensorineural hearing impairment seek medical treatment often, cost a lot of money, and have no obvious effect.(5) The biggest problem faced by hearing-impaired people is education. Due to their hearing and speech barriers, there will be problems such as poor vocabulary, difficulty in grouping sentences, and inaccurate pronunciation. In addition, they need to listen and observe very carefully. Nervous regions
If people can perceive music , these problems can be improved
Page:93
7.2.2 Existent device limitation From a medical point of view, hearing-impaired people are divided into mild to severe. Not only are there many factors that cause deafness, but there are also pre-verbal deafness and post-lingual deafness. However, this does not mean that deaf people must live in a silent world. Deaf people can rely on auxiliary devices to The cochlear implant is an artificial device that perfectly couples electronic technology with the human nervous system.It replaces the diseased inner ear sensory nerve cells, so that patients with severe and extremely severe sensorineural hearing loss can gain or regain hearing ability (Zeng, F,2008). Over the past 20 years, tens of thousands of deaf patients have returned to the world of sound. However, they cannot distinguish sound, understand speech and enjoy music like normal hearing people. The vast majority of patients currently receiving cochlear implants can Achieve good speech recognition in a quiet environment, but its speech intelligibility in a noisy environment and the recognition of language tones are not yet
can’t use cochlear implants
Risk to cochlear implants surgery
Can’t afford cochlear implants
1 Their hearing is “too good” (they can hear some sound and speech with hearing aids). 2 Their hearing loss isn’t due to a problem with the cochlea. 3 They’ve been profoundly deaf for a long time. 4 The auditory nerve is damaged or absent.
1 infection at the surgical site 2 ringing in the ears (tinnitus) 3 dizziness or balance problems (vertigo) numbness around ears 4 weakness in the muscles of the face 5 leakage of cerebrospinal fluid from around the brain 6 the implant doesn’t work 7 infection of the brain (meningitis)
between $30,000 and $50,000 Without insurance, a cochlear implant can cost between $30,000 and $50,000 on average, according to Boys Town National Research Hospital. Most insurance providers cover cochlear implants or a portion of them. The device is also covered by Medicare, Medicaid, and Veterans Affairs.
Page:94
For users who wear hearing aids, listening to speech is very clear, but listening to music is always not ideal. First of all, the hearing aid is an auxiliary device, which uses the residual hearing of the user to listen to the sound of the external environment, and helps protect the residual hearing. Although it can help people with hearing loss improve their hearing, they have the following difficulties to enjoy music. First: Different frequencies Speech sounds come from our voice-forming organs (such as vocal cords, mouth, tongue, teeth, palate, etc.). These organs make the frequency of our language very specific. So no matter what language you speak, their long-term average speech spectrum is very similar. Conversely, different music may come from different musical instruments. It does not have a fixed frequency, such as stringed instruments (violin, viola, cello, and bass), such as clarinet, trumpet, or percussion instruments, so the music can be narrowband, broadband, low frequency, or high frequency. Second: The sound intensity is different. Compared to speech, the music sound is usually louder and louder, and the fluctuations are higher. Even when speaking loudly, the intensity can reach at most 80-92dBSPL (peak sound intensity), and the current A/D converter of digital hearing aids can convert up to 115dBSPL of input sound without distortion, so for a large Speech sound, hearing aids can easily handle it, but music is not, the music played by the instrument is usually 50-100dB-
cochlear impalts hearing aids
The results of this study show that neither device can achieve satisfactory music appreciation.
Page:95
7.3 design precedent Feeling music â&#x20AC;&#x153;The sound shirtâ&#x20AC;? is the most similar project, the purpose is also to feel the music on the skin. But the difference in the project is that they use multiple microphones, and they put the microphones in front of different instruments when the performance starts. In this way, the frequencies of multiple instruments can be input accurately at the same time. It can make users feel the change more clearly, but the disadvantage is that it is impossible for users to place microphones on the stage at every concert or at bars and music festivals. Music interaction This project describes the development and evaluation of installations that explore the close connection between remote strangers. The conceptual meaning of the device is that music can be used as a universal language through which strangers can communicate. To this end, the author has taken several methods to enhance the intimacy among device users. They described the mediated intimacy and related work in the field of music interaction, forming the initial goal of the system. Then, it describes the iterative development process, which includes two smaller prototype tests. The final installation realized two large human-shaped boxes, each of which had a hole for inserting a human head. Inside the box, the user can view the face of the remote stranger. Special settings make users look very close to each other, while being able to see each other in the eyes, thus enhancing the sense of intimacy. Finally, the facial tracking algorithm can detect when the user opens his mouth, thus triggering the voice of the opera singer. Therefore, people (people who do not understand music) can explore opera duos in the form of music exploration and communication. Facial expressions of sound The organizers of this project have many light shows about music, music events and explorations about music. Their organization is in Sydney. This project is about discussing music and sound in another way. They showed that they can use facial expressions to convey sound. The instrument automatically captures the userâ&#x20AC;&#x2122;s facial expressions and then converts them into different sounds. This project provided me with many interesting ideas. Sound can have many manifestations.
Page:96
CuteCircuit’s Sound Shirt Allows Deaf People To “Feel” Music
The project explores mediated intimacy between remote strangers through music
inside vivid sydney
Page:97
H e llo , t h is is Y u e H a o . I a m a p o s t g r a d u a t e s t u d e n t a t t h e R o y a l M e lb o u r n e I n s t it u t e o f T e c h n o lo g y in A u s t r a lia . M y p o s t - g r a d u a t io n t h e s is is a b o u t h e lp in g t h e h e a r in g im p a ir e d p e o p le f e e lin g t h e p o s s ib ilit y o f e x p e r ie n c in g m u s ic . T h e r e fo r e , I h o p e t o c o lle c t s o m e in fo r m a t io n a b o u t t h e m u s ic e x p e r ie n c e o f t h e h e a r in g im p a ir e d a n d id e a s a b o u t a lt e r n a t iv e h e a r in g m e t h o d s ( s u c h a s v is u a l a n d v ib r a t o r y f e e lin g s ) . T h a n k y o u v e r y m u c h f o r y o u r t im e . T h is in f o r m a t io n w ill h e lp m e d e s ig n a w e a r a b le d e v ic e a n d h o p e t o h e lp m o r e p e o p le in t h e f u t u r e . Have you ever participated in a music event? (Multiple choices) ¨ 1 movie
¨ 2 music festival
¨ 3 concert
¨ 4 club
¨ 5 music fountain
Do you often listen to music? ¨ everyday ¨ 3-5 times per week ¨1-3 times per week
¨ less or nerve
Are you disappointed not to hear music? ¨ 1
Not disappointed
¨ 2
¨ 3
¨ 4
¨ 5
Very disappointed
If you can hear music, you think it will help Social activities
¨ Yes | ¨ No
Job opportunity
¨ Yes | ¨ No
Mental emotions
¨ Yes | ¨ No
Frequently used assertive device in music activities (multiple choices) ¨ subtitle display ¨ Performer body language of sound ¨ graphical display ¨other
¨ facial expression
¨ Feel the vibration
The help of different devices in music activities (1 is very helpful-5 is less helpful) ¨ sound amplification display
¨ subtitle display ¨ sign language
¨ vibration device
¨ graphical
When using the vibration mode for the phone, set different vibration modes for different incoming songs. Do you think you can distinguish songs by different vibration feelings? ¨ yes ¨no
If you feel that the different vibration modes of the mobile phone can't distinguish songs, can you list the reasons? (For example, you cannot feel the beat, do not understand the volume, etc.) Page:98
If you have ever seen a music fountain or a light show, do you think you can understand the song through the change of light? ¨ yes
¨ no
）If you can't understand the song through the change of light, can you list the reasons? (If you cannot understand the rhythm, do not understand the color change. etc.) What do you think is the most effective way for hearing-impaired people to understand music?
Page:99
7.4 Participants in surveys and interviews
Netflix (deaf) Australia
Doctor Hoffmann clinic, German
Page:100
Sounds possible music with the deaf Australia
otolaryngology department, Rehabilitation hospital,China
Answers of survey Q7,8 and 9 if you have ever seen a music fountain or a light show, do you think you can understand the song throung the change of light? Can understand the beat Can’t understand the rhythm I think the presentation of music fountains is beautiful, but I do not understand how the lights represent the music. For example, what does it mean for the music if the light changes from red to blue color? no feeling of rhythm,no connection with lyrics i would need to know the meaning of the different colors cannot understand the rhythm,the color changing and the melody
If you feel that the different vibration modes of the mobile phone can’t distinguish songs, can you list the reasons? (For example, you cannot feel the beat, do not understand the volume, etc.) “Cannot get the rhythm” Can distinguish between songs, can be distinguished between intense rhythm rock music or slow light music. Can use beat or touch Can’t feel the change of music melody Need learning process Can’t feel the change of music melody vibration is too subtle to feel I can understand the intensity of the vibration but not the rythm if i knew the song that might can understand
What do you think is the most effective way for hearing-impaired people to understand music? Listening to music is in a good mood. If you can’t hear music, it’s a color, black and monotonous. When listening to music, the colors are diverse and the mood is pleasant. Hope the design is helpful to us. Haptic, touch the vibration Tactile and visual combination. graphics and images, sense through touch I think, I would start with vibration since I am used to it from my phone. But be careful to explain what each vibration means for the music. feeling the vibration and possiblity to read lyrics give vibrations to understand what rhythm is feeling vibrations the light changings and leave a performer body language vibrations,light,art(visual)
Page:101
7.5 User experience of the design project
Page:102
Page:103
Page:104
7.6 research about Audio Signal to FFT Anlysis The “Fast Fourier Transform” (FFT) is an important measurement method in the science of audio and acoustics measurement. It converts a signal into individual spectral components and thereby provides frequency information about the signal. FFTs are used for fault analysis, quality control, and condition monitoring of machines or systems.
amplitude
The FFT-algorithm uses this principle and essentially enables you to see which frequencies are present in any analog signal and also see which of those frequencies that are the most dominating. This is very helpful in a huge amount of applications. Graphs like the blue one is what you typically get after running an FFT:
tim
e
y
nc
ue
q fre
The x-axis is frequency – the higher up on this axis, the higher the frequency. The y-axis is amplitude – the higher up on this axis, the larger the amplitude. graphs are fetched from the Arduino serial plotter after running FFT on a few different signals with 128 Hz sampling rate and 128 samples.
https://www.numerical-tours.com/matlab/audio_1_ processing/
The numbers on the x-axis in the graphs below are not frequency, but element number (aka. bin). In the graphs below, element number 64 is the top bin (~500 Hz). Since our input signal is has a DC offset I get the peak at zero in the results. Page:105
3.2.3 Rhythm and beat anlysis
Below are more examples of the beat detection capabilities of my Arduino implementation:
LPL Summer championship beat detection
Pop song
DJ Sovjet Tiny Love spiritual / Basic Hard Rock Drum Beat 150 BPM animation
Basic Hard rock drum 60 BPM
Page:106
7.7Details about my music processing situation The effects of sampling on the FFT analysis
Sound waves are continuously contracting and deconstructing. Thus it is not enough to analyse sound with a single value. I need to sample the sound with several recorded values to properly analyse the sound wave. For the sampling, two parameters play a major role: The sampling rate specifies how many samples can be collected in one second The block length specifies how many samples are collected Based on these two parameters several parameters can be determined: Nyquist frequency: The highest sound frequency that can be determined is half of the sampling frequency. Measurement duration: The time used to collect all sample values is block length divided by the sampling rate. Frequency resolution: Each frequency bin of the FFT analysis will have a resolution of sampling rate divided by the block length. The Arduino natively supports a maximal sampling rate of 9846 Hz and the best available FFT libraries support a block length of 256 samples. By reading directly from the registers, I managed to increase the sampling rate by a factor of 2 to 19,692 Hz. The Arduino then has the following parameters for the FFT analysis:
Sampling rate
Block length
Nyquist frequency
Measurement duration
Frequency resolution
19,692 Hz
256
9846 Hz
13 ms
77 Hz Page:107
7.8 Final CPB board
7.8.1 Final dritzing_etch_copper_ bottom
Page:108
7.8.2 Final dritzing_etch_silk_top
7.8.3 Arduino uno DRV2605L ERM
Page:109
7.9 reference Steven Gelineck. 2014. OperaBooth - an Installation for Intimate Remote Communication through Music. In Proceedings of the 9th Audio Mostly: A Conference on Interaction With Sound (AM’14). http://media.aau.dk/~stg/operaBooth.html Disney Research http://www.disneyresearch.com/project/ishin-den-shin/ (2013.11.12) Creative Appications Network http://www.creativeapplications.net/sound/ishin-denshin-physicality-and-intimacy-through-sound-and-touch/ (2013.11.12) Resources: cutecircuit.com, reuters.com https://www.behance.net/gallery/71300109/Deaf-Statistics-Infographics Cortical Neuroplasticity in Hearing Loss: Why It Matters in Clinical Decision-Making for Children and Adults Jun 27, 2018 | Auditory Processing Disorders Picture: Beethoven Music Flat Vector Icons https://www.vecteezy.com/vectorart/151087-beethoven-music-flat-vector-icons Bandodkar, A. J., Jia, W. & Wang, J. Tattoo-based wearable electrochemical devices: a review. Electroanalysis 27, 562–572 (2015). Gao, W. et al. Fully integrated wearable sensor arrays for multiplexed in situ perspiration analysis. Nature 529, 509–514 (2016). Mancini, F., Bauleo, A., Cole, J., Lui, F., Porro, C. A., Haggard, P., & Iannetti, G. D. (2014). Whole-body mapping of spatial acuity for pain and touch. Annals of neurology, 75(6), 917–924. https://doi.org/10.1002/ana.24179 Wearable Musical Beat-Teaching Device,Lee McKean,ECE-499: Capstone Design Project Advisor: Professor Helen Hanson, https://cpb-us-w2.wpmucdn.com/muse. union.edu/dist/0/348/files/2016/03/McKean-Thesis-Final-Report.pdf Wang, Y., Qiu, Y., Ameri, S.K. et al. Low-cost, μm-thick, tape-free electronic tattoo sensors with minimized motion and sweat artifacts. npj Flex Electron 2, 6 (2018). https://doi.org/10.1038/s41528-017-0019-4 Tzanetakis, G., Essl, G., & Cook, P. (2001, September). Audio analysis using the discrete wavelet transform. In Proc. Conf. in Acoustics and Music Theory Applications (Vol. 66). Page:110
An Approach to Biometric Verification Based on Human Body Communication in Wearable Devices Jingzhen Li, Yuhang Liu, Zedong Nie *, Wenjian Qin, Zengyao Pang and Lei Wang Shenzhen Institutes of Advanced Technology, Chinese Academy of Science, Shenzhen 518055, China; lijz@siat.ac.cn (J.L.); yh.liu2@siat.ac.cn (Y.L.); wj.qin@ siat.ac.cn (W.Q.); dl.yang@siat.ac.cn (Z.P.); Mandal, Bibhuti & Srivastava, Anup. (2006). Risk from vibration in Indian mines. Indian Journal of Occupational and Environmental Medicine. 10. 10.4103/00195278.27460. Nanayakkara, S., Taylor, E., Wyse, L., & Ong, S. H. (2009, April). An enhanced musical experience for the deaf: design and evaluation of a music display and a haptic chair. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 337-346). Antoine Hunter is the founder of Urban Jazz Dancer Company and the International Deaf Dance Festival., By CHRISTOPHER EGUSA • MAR 11, 2020, https://www. kalw.org/post/dancing-out-loud-antoine-hunter-uses-dance-express-deaf-experience#stream/0 These Sign Language Interpreters Are Working Hard to Make Live Music More Accessible to All, By Lynn Q. Yu -April 9, 2018, https://www.lamag.com/culturefiles/ sign-language-coachella/ Shibata, D. Brains of deaf people “hear” music. International Arts-Medicine Association Newsletter, 16, 4 (2001). http://www.iamaonline.org/Dec01_IAMA_NL.PDF. Zeng, F. G., Rebscher, S., Harrison, W., Sun, X., & Feng, H. (2008). Cochlear implants: system design, integration, and evaluation. IEEE reviews in biomedical engineering, 1, 115-142. Drennan, W. R., & Rubinstein, J. T. (2008). Music perception in cochlear implant users and its relationship with psychophysical capabilities. Journal of rehabilitation research and development, 45(5), 779. Scott, J. (2014). Computational Modeling and Analysis of Multi-timbral Musical Instrument Mixtures. Drexel University. Mitchell, L. D. (1982). Improved methods for the Fast Fourier Transform (FFT) calculation of the frequency response function. Settel, Z., & Lippe, C. (1994). Realtime musical applications using FFT based resynthesis. In Proceedings of the International Computer Music Conference (pp. Page:111
Page:112
Page:113