Visual learning for ASL phrases Salvador Rojas-Murillo, PhD; Angelique Vega; Russell Burghard; Noah Green Fred DeMatteis School of Engineering and Applied Science, Hofstra University
Methods (2)
Introduction
• When learning a new ASL phrase novice ASL interpreters need to observe the ASL instructor to identify the key visual cues performed not only to make an interpretation but also to comprehend how they would have to move their face and body to replicate what they observe.
•
• Previous researchers did not find observational statistically significant differences between novice and expert ASL interpreters when interpreting ASL dialogues; both groups concentrated their observations in the performer’s face
•
•
The expert ASL interpreters translated the 17 phrases while the novice ASL interpreters only watched and performed the phrases, both performed them 5 times each. The experts viewed the video at regular speed while novices viewed the video at half speed. Both experts and novices wore the Tobii ™ eye tracking glasses in order to record their eye movements while observing the certified ASL interpreter performing the phrases.
• Practice affects peoples’ observational selection and observation time when provided with visual instructions and feedback; practice allows an observation transition from the instruction’s key areas to the feedback’s visual key areas.
Figure 1: Participants’ interface for novices. Part A showed a live video streaming of the participant from a front-facing camera displayed horizontally and mirrored to provide visual feedback. Part B, showed the video recording of the 17 phrases performed by the certified ASL interpreter
Aim
• Determine the visual key areas in which would be selected by novices when learning an ASL phrase. • Determine the effect of experience in the selection of these key areas. • Determine the effect of providing visual feedback in the selection of visual key areas.
Results
• •
•
Hypotheses
• The selection of visual key areas by novice ASL signers will be modified by practice. • As a result of the first hypothesis, novice ASL signers will lower number of eye-fixations in the reference area and will increase the number of fixations in the feedback area. • The selection of key visual areas for expert ASL interpreters and novice ASL signers will be different. Expert ASL interpreters will concentrate 80% of their observations on the face while novice ASL signers will observe the face at a much lower rate.
•
• •
For the outside region, novices’ mean fixation duration was 0.417 s and the experts’ mean fixation duration was 0.0711 s, with a statistically significant difference between means t(17) = 6.76, p = 0.000. For the abdominal region, the mean fixation duration for novices was 0.232 s, while the mean fixation duration for experts was 0.0100 s, showing a statistically significant difference between means, t(17) = 4.20, p = 0.001. For the chest region, the novice ASL signers had a mean fixation duration of 3.82 s, while experts mean fixation duration had a value of 0.526 s, with a significant difference between means, t(17) = 7.86, p = 0.000. For the lower face region, the mean fixation duration for the novices was 1.880 s, whereas the mean fixation duration for the experts was 0.352 s, describing a significant difference between means, t(17) = 6.38, p = 0.000. For the upper face region, the mean fixation duration for novices was 0.859 s, whereas it was 3.46 s for the experts, showing a statistically significant difference between means, t(17) = -8.32, p = 0.000. When comparing the average fixation duration for all 17 phrases between novices and experts, novices observed at different regions for a longer fixation duration than the experts. The average fixation duration per phrase for the novices was 1.443 s and for the experts was 0.885 s. The t-test found a statistically significant difference between means, t(17) = -5.46, p = 0.000.
Discussion
• • •
Figure 4: The left side is the participants area and consists of participant’s upper face (1), participant’s lower face (2), participant’s chest (3), and participant’s abdominal (4). The right is the reference area and consists of upper reference face (5), lower reference face (6), reference chest (7), and reference abdominal (8). The remaining outside region (9).
Methods (1)
•
The experiment consisted of 17 ASL phrases each between 5-15 seconds long performed by a certified ASL interpreter and were displayed on a computer monitor 60cm away from the participants’ eyes.
Novices spent 89.3% of their eye-fixation duration on the reference excluding the outside area, while they only observed the participant area 5.2% and the outside area 5.5% of their eye-fixation duration. The results indicate statistically significant observational differences between novices and experts while observing the reference video. Novices were also considered to have no experience in ASL which was crucial for us to find statistically significant results between expert interpreters and novice ASL signers.
Conclusion
•
Novices concentrated their observations at the reference chest as an indication that they were following the performer’s hand, while experts concentrated their observations at the upper face.
Future Work
• Future work should be considered for short but complex phrases that require a grammatical understanding such as performing complex verbs.