THIS CHAPTER EXAMINES HOW THE SENSES INTERACT AND HOW THIS KNOWLEDGE CAN BE USED IN VIRTUAL REALITY THE FOCUS IS ON INTERACTION BETWEEN AUDITION AND VISION AND AUDITION AND TOUCH In simulating realistic multimodal environments, several elements including synchronization need to be taken into consideration. However, technology gives some limitations, especially when the ultimate goal is to simulate systems that react in real time. [107] explains a tradeoff between accuracy and responsiveness, which represents a crucial difference between models for science and models for interaction. Specifically, computations about the physical world are always approximations. In general, it is possible to improve accuracy by constructing more detailed models and performing more precise measurements, but this comes at the cost of latency, that is, the elapsed time before an answer is obtained. For multisensory models it is also essential to ensure synchronization of time between different sensory modalities. [107] groups all of these temporal considerations, such as latency and synchronization, into a single category called responsiveness. The question then becomes how to balance accuracy and responsiveness. The choice between accuracy and responsiveness depends also on the final goal of the multimodal system design. As an example, scientists are generally more concerned with accuracy, so responsiveness is only a soft constraint based on available resources. On the other hand, for interaction designers, responsiveness is an essential parameter that must be satisfied. There are different ways in which the senses can interact. Cross-modal mapping represents the situation where one or more dimensions of a sound are mapped to a visual or tactile feedback [102]. An example of this situation is a beeping sound combined with a flashing light. Intersensory biases represent the situation where audition and another modality provide conflicting cues. When examining specific multimodal examples in the following section, several examples of intersensory biases will be provided. In most of these situations, the user tries to perceptually integrate the conflicting information. This conflict might lead to a bias towards a stronger modality. One classic example is the ventriloquist effect [55], which illustrates the dominance of visual over auditory information. In this effect, spatially discrepant audio and visual cues are experienced as colocalized with the visual cue. This effect is commonly used in cinemas and home theaters where, although the sound physically originates at the speakers, it appears as coming from the moving image on screen, being for example a person speaking or walking. The ventriloquism effect occurs because the visual estimates of location are typically more accurate than the auditory estimates of location, and therefore the overall perception of location is largely determined by vision. This phenomenon is also known as visual capture [176].
Cross-modal enhancement refers to the situation where stimuli from one sensory channel enhance or alter the perceptual interpretation of stimulation from another sensory channel. As an example, three studies presented in [158] show how high-quality auditory displays coupled with high-quality visual displays increase the quality perception of the visual displays relative to the evaluation of the visual display alone. Moreover, low-quality auditory displays coupled with high-quality visual displays decrease the perception of quality of the auditory displays relative to the evaluation of the auditory display alone. These studies were performed by manipulating the pixel resolution of the visual display and Gaussian white-noise level, and by manipulating the sampling frequency of the auditory display and Gaussian white-noise level. Subjects were asked to rate the quality of the visual image of a radio with different pixel qualities, coupled to auditory feedback resembling sounds coming from a radio. These findings strongly suggest that the quality of realism in an audiovisual display must be a function of both auditory and visual display fidelities inclusive of each other. The findings can show obvious applications in virtual reality, where the quality of the visual display is still poor. Here auditory display can help. Cross-modal enhancements can occur even when the extra-modal input does not provide information directly meaningful for the task. A primary example was reported by [152]. Subjects rated the intensity of a visual light higher when it was accompanied by a brief, broadband auditory stimulus than when it was presented alone. Cross-modal transfers or illusions are the situations where stimulation in one sensory channel leads to the illusion of stimulation in another sensory channel. An example of this is synesthesia, which in the audiovisual domain is expressed for example as the ability of seeing a color while hearing a sound. When considering intersensory discrepancies, [176] propose a modality-appropriateness hypothesis. Their model suggests that the various sensory modalities are well suited to the perception of different events. Their model also shows that the dominance of a particular modality is relative to its appropriateness to the situation. Generally, it is supposed that vision is more appropriate for the perception of spatial location than is audition, with touch somewhere in between. Audition is most appropriate for the perception of temporally structured events. Touch is more appropriate than audition for the perception of texture, whereas vision and touch may be about equally appropriate for the perception of textures. The appropriateness is a consequence of the different temporal and spatial resolution of the auditory, tactile, and visual systems.
13