Audio-visual substitutions and illusions

William James made the hypothesis that, if our eyes were connected to the auditory brain areas, and our ears to the visual brain areas, we would Òhear the lightning and see the thunderÓ [1]. Research suggests that modality-specific brain areas, such as the visual cortex, can process auditory stimuli, for instance in the case of brain alteration (e.g., rewired ferret’s brain) or sensory deprivation (e.g., blindness). The study we conducted aimed at investigating behaviourally this question, by using a non-invasive technique of sensory plasticity. The participants learned to use a visual-to-auditory sensory substitution device, which translates visual images recorded by a camera into soundscapes. Both before and after training, they completed a Stroop-like task in which they had to recognize soundscapes while being simultaneously presented with task-irrelevant visual lines. Before training, the visual images did not influence the participants’ responses. However, after training, they disturbed the participants’ response when the auditory soundscape did not correspond to the conversion of the visual image. This visual interference effect reveals that visual imagery can be associated to auditory stimuli. In addition, the participants’ performance during training for localisation and recognition tasks, as well as their associated phenomenology, depended on their auditory abilities, revealing that processing finds its roots in the input sensory modality. Our results bring behavioural evidence to the thesis that experience with sensory substitution devices is neither strictly visual nor auditory, but the functional plasticity at stake is complex, and based on a multisensory architecture [2]. Altogether, they suggest that brain plasticity allows people to see the thunder while still hearing it.   

Hide abstract

Computations within many brain regions (and their perceptual consequences) are independent of the specific sensory information input they receive. This has been demonstrated in visually-impaired and blind individuals using sensory substitution devices (SSDs). Following training, nominally visual processes of face and letter discrimination can be performed within the fusiform face area (FFA) and the visual wordform area, respectively, via exclusively auditory inputs. Such evidence has hitherto come primarily from fMRI, thus leaving the brain dynamics, and so the cognitive mechanisms, of SSD-driven perceptions unresolved. Furthermore, it is unclear if SSDs can achieve computations ascribed to neurons within the primary visual cortex (V1), such as line-orientation discrimination. We recorded 64-channel EEG from nine adult blind individuals (8 congenitally blind, 1 blind from 1 year of age). All participants trained for ~70 hours with the vOICe visual-to-auditory SSD, and none had prior SSD experience. For the EEG experiment, participants were presented passively with soundscapes of letters, line drawings of faces, and their scrambled counterparts, after having behaviourally demonstrated >95% accuracy in discriminating these categories. First, ERPs to face soundscapes differed from those to scrambled faces within the first 400ms post-soundscape onset and involved stronger sources within the FFA and anterior temporal cortices. Crucially, responses to soundscapes of letters with a predominantly horizontal versus vertical line orientation differed first at 270-400ms post-soundscape onset and within V1 and surrounding visual cortices. Orientation discrimination with SSDs is indeed performed in V1. These collective results provide the first demonstration of the dynamics with which highly-tuned Òsensory-specificÓ neural populations perform stimulus discrimination based on inputs from another sense. On the one hand, our findings extend models positing task-contingent brain organization to include V1. On the other hand, our findings impact designs of SSD technologies and the kinds and speed of visual processing that may be achieved.     

Hide abstract

Sensory information about the state of the world is generally ambiguous. Understanding how the nervous system resolves such ambiguities to infer the actual state of the world is a central quest for sensory neuroscience. However, the computational principles of perceptual disambiguation are still poorly understood: What drives perceptual decision-making between multiple equally valid solutions? Here we investigate how humans gather and combine sensory information_within and across modalities_to disambiguate motion perception in an ambiguous audiovisual display, where two moving stimuli could appear as either streaming through, or bouncing off each other. By combining psychophysical classification tasks with reverse correlation analyses, we identified the particular spatiotemporal stimulus patterns that elicit a stream or a bounce percept, respectively. From that, we developed and tested a computational model for uni- and multi-sensory perceptual disambiguation that tightly replicates human performance. Specifically, disambiguation relies on knowledge of prototypical bouncing events that contain characteristic patterns of motion energy in the dynamic visual display. Next, the visual information is linearly integrated with auditory cues and prior knowledge about the history of recent perceptual interpretations. What is more, we demonstrate that perceptual decision-making with ambiguous displays is systematically driven by noise, whose random patterns not only promote alternation, but also provide signal-like information that biases perception in highly predictable fashion.             

Hide abstract

In a typical experiment investigating the McGurk illusion stimuli are chosen in a way that congruent stimuli (template) and incongruent stimuli (competing-incongruent stimulus, CIS) are not competing for the same perceptual category within a single experimental session. This is to avoid that subjects exploit the temporal proximity of template and CIS to tell their difference, thereby degrading the strength or frequency of occurrence of the illusion. Here, we explicitly address the dynamics of the McGurk illusion in a scenario where template and CIS both typically mediate an auditory ‘ta’ perception (e.g., congruent: visual: ‘ta’/auditory: ‘ta’ and incongruent: visual: ‘ka’/auditory: ‘pa’ both yielding reported ‘ta’-percepts). To this end, we presented template and CIS in a single experimental session and randomized order. Post-hoc we sorted the presented trials based on stimulus type and run length (i.e., the number of times the same stimulus occurred in immediate succession). By this analysis we were able to test the frequency of the McGurk illusion as a function of run length of the CIS (which also correlates to elapsed time). Results showed that when the CIS followed the template immediately, the McGurk illusion was indeed less frequent compared to a baseline. This is in line with the (implicit) belief that individuals can exploit the temporal proximity of template and CIS to tell the difference between the congruent and incongruent stimulus. Further analysis also showed that with increasing temporal distance of the CIS to the template the frequency of occurrence of the illusion approached baseline again.       

Hide abstract

The perception of limbs as one’s own is called body ownership (BO). BO requires the integration of signals from multiple sensory modalities including vision, touch and proprioception (Ehrsson, 2012). Yet, the literature lacks a sensitive and rigorous psychophysics method to register BO. To fill this gap we developed a new discrimination task that allows a more precise and direct measurement of BO based on a version of the Rubber Hand Illusion (RHI).

In this paradigm, the participants’ right hand lies hidden beneath a table, while on this table two identical right RHs are placed in parallel to each other (same distance from the real hand, same orientation). Three robot arms repeatedly apply taps to the two RHs and to the participants’ real hand. Each RH can be touched synchronously with the other RH and the participants’ hand or with a systematically variable delay (<200 ms, the longer the delay the weaker the BO for that RH). The participants then have to decide which of the two hands feels more like theirs. We manipulated the real hand position’s relatively to the RHs (Exp1) and the texture congruency between the visual and tactile stimuli (Exp2).

The results show that participants’ perception of BO can be quantitatively described by the mean and variance of the fitted psychometric curves. This fitting does not work under conditions known to abolish the RHI. By analysing the curves’ means under different distance conditions, we reproduced well-known constrains of the RHI and, thus, were able to confirm that our paradigm adequately registers BO (Exp1). Moreover, even small discrepancies between the seen and felt touches leads to a significant change in BO (Exp2), which demonstrates the sensitivity of our method. Taken together, our results suggest that BO constitutes a genuine perceptual multisensory phenomenon that can be quantified with psychophysics.   

Hide abstract

Dynamic changes in the environment require the human perceptual system to flexibly recalibrate the integration of multi-sensory inputs. In the ventriloquism aftereffect (VAE), repeated presentations of spatially discrepant visual and auditory stimuli lead to a perceptual recalibration of auditory space, shifting the perceived location of subsequently presented auditory stimuli in the direction of the visual offset. Yet the timescales over which spatial multisensory recalibration develops remains unclear. Here we characterise the dynamics of the VAE and ask whether they reflect a single adapting mechanism, or multiple mechanisms operating over distinct timescales. We adapted subjects to a sequence of pink noise bursts positioned across a range of azimuths, each presented in synchrony with a Gaussian blob positioned at either the same location, or offset to the left or right by 20 degrees. After each adapting period, subjects reproduced the perceived location of a series of unimodally presented auditory stimuli. We measured the growth and decay of the VAE after adaptation to audio-visual spatial discrepancies for varying durations of adaptation – from 32 up to 256 seconds. Our results showed that the VAE built up and decayed in a manner proportional to the duration of the initial adaptation period. To distinguish recalibration mechanisms operating at unitary or distinct timescale(s), we employed an adapt/de-adapt paradigm. Subjects initially adapted to an audio-visual spatial offset lasting 256 seconds, then immediately de-adapted to the opposite offset for 32 seconds. VAEs elicited by adaptation were initially cancelled by de-adaptation, but subsequently re-appeared with further testing. This suggests that opposing VAEs can be simultaneously stored at different timescales. Taken together, our results support a model in which the VAE is governed by multiple spatial recalibration mechanisms tuned to different timescales.             

Hide abstract

The McGurk Effect has arguably been one of the most influential phenomena in the history of perceptual psychology. In ostensibly demonstrating the automaticity with which the senses combine, it has help motivate a new understanding of perceptual experience and neuroscience as being supremely multisensory. However, the McGurk Effect has also been used as a methodological tool to establish when and how audiovisual integration occurs. The effect has also been used to evaluate how the processes of multisensory integration might differ across individuals, cultures, development, and attentional states.

In this talk, it will be argued that based on what is now known of the McGurk effect, as well as of perceptual and neurophysiological research, the effect fails as a reliable tool for measuring integration. There are clear empirical examples in which the McGurk effect fails, but audiovisual speech integration still occurs (e.g., Brancazio & Miller, 2005; MacDonald, Andersen, & Bachmann, 2000). Other examples show that despite overt responses reflecting no McGurk Effect integration, covert responses show otherwise (e.g., Gentilucci & Cattaneo, 2005; Sato, et al. 2010). It could very well be that given sufficient crossmodal information for a coherent event, integration always occurs, even when McGurk effect responses fail. It will be argued that other measures, including analyses of production responses, as well as simple evaluations of visual enhancement of speech in noise, are more reliable and informative measures of perceptual integration.         

Hide abstract