The Role of Experience in the Development of Multimodal Integration

The everyday objects and events in an infant’s world are often specified by multisensory attributes. For example, whenever infants interact with social partners, they can see and hear them talking. To perceive their social partners as perceptually unitary and psychologically meaningful entities, infants must attend to their multisensory attributes and process them. We and others have found that multisensory processing is rudimentary at birth and that it improves gradually as infants grow and acquire experience. In this talk, I will review evidence from our lab indicating that early experience is critical for the emergence of multisensory processing in infancy. First, I will show that the multisensory world of young infants is relatively undifferentiated because of sensory limitations and lack of experience. Then I will show that infants gradually shed their primitive multisensory processing mechanisms in favor of increasingly more refined and mature ones and I will provide evidence that this developmental process is partly driven by experience-dependent narrowing of responsiveness to multisensory inputs. Narrowing will be illustrated by evidence from studies of infant matching of other-species faces and voices, matching of native and non-native auditory and visual speech, and discrimination of own and other-race faces. Overall, these findings will show that infants initially exhibit paradoxically broad tuning and, thus, detect the multisensory unity of native and non-native multisensory inputs and that, as they acquire mostly native-input experience, they cease detecting the unity of non-native multisensory inputs. Finally, I will review findings from our recent studies of infant selective attention to fluent audiovisual speech and demonstrate that attention undergoes marked, experience-dependent, developmental changes during the first two years of life. I will conclude by highlighting the intricate relationship between selective attention and multisensory processing and the way that early experience affects the development of both processes.

Supported by Eunice Kennedy Shriver National Institute of Child Health and Human Development, Grant number: R01HD057116

Hide abstract

Patients treated for congenital cataract provide a natural experiment for investigating the role of visual input in perceptual development. Here, we considered the effects of early visual deprivation on the development of audiovisual integration in adults treated for unilateral or bilateral congenital cataract at 0.3-28.8 months of age (n=13/grp).

Simultaneity judgments. Controls showed a typical gaussian curve peaking when the flash slightly preceded the sound, a shift usually attributed to plastic adjustment for slower travel of sound than light. The curve for bilateral patients was abnormally wide, but only on the flash-leading side, the condition most plastic in normal adults (Chen et al., 2017). The curve for unilateral patients had a normal peak but was abnormally wide for both sound- and flash-leading sides as is found in typically developing children (Chen et al., 2016).

Fission illusion. Bilateral patients had a smaller illusion than controls, with less influence of the beeps on the perceived number of flashes. This pattern occurred despite evidence of auditory dominance in other tasks: they are faster than controls at detecting a beep and at switching attention from vision to audition (de Heering et al., 2016) and in fMRI images, show auditory activation of visual cortex (Collignon et al., 2015). Unilateral patients behaved like the control group, with a similarly robust illusion, which was normally modulated between centre (smaller) and periphery (larger).

Bilateral patients’ abnormalities suggest that early visual input is necessary both to set up the neural substrate for later visual development and also to allow normal audiovisual integration. When input is missing to both eyes, even for just the first few months of life, auditory input establishes effective connections in “visual” cortical areas and leads to enhanced auditory processing, but at a cost to effective audiovisual integration. When the deprivation is unilateral, early input through just one eye may be sufficient to reduce that re-organization.

Supported by CIHR, NSERC, James S. McDonnell Foundation

Hide abstract

Congenital deafness prevents the normal growth and connectivity needed to form a normally functioning sensory system– resulting in deficits in oral language and cognition in deaf children. Cochlear implants (CI) bypass peripheral cochlear damage, by directly stimulating the auditory nerve, making it possible to avoid many of the harmful effects of auditory deprivation. Patients who receive CI’s provide a platform to examine the characteristics of neuroplasticity in the central auditory system. An important aspect of compensatory plasticity that is evident in deafness is the re-organization and re-purposing of auditory cortex by other sensory modalities known as cross-modal plasticity. We examined cross-modal plasticity from the visual and somatosensory systems in deaf children with bilateral deafness fitted with CIs and in deaf adults and children with single-sided deafness fitted with CIs and related them to communication and cognitive outcomes. High-density 128 channel electroencephalographic (EEG) recordings to auditory, visual and somatosensory stimulation, tests of auditory-visual speech perception in noise and cognition were administered. Patients with cochlear implants showed activation of auditory cortical areas in response to visual and vibrotacticle stimulation, suggestive of cross-modal recruitment by the visual and somatosensory modalities. Cochlear implanted patients who had greater difficulty understanding speech in noise via their cochlear implant showed greater evidence of cross-modal recruitment. In persons with single-sided deafness the (unstimulated) cortex contralateral to the deaf ear showed greater evidence of cross-modal recruitment. Cochlear implantation appeared to reverse cross-modal re-organization in many (but not all) patients with single-sided deafness. Overall, our results show that compensation for degraded auditory input results in greater dependence on other modalities which serves to aid communication in real-world situations. Our results suggest that the alterations in neural circuitry that underlie compensatory cross-modal plasticity may be an important source of variability in outcomes for patients with CIs.

Supported by the US National Institutes of Health.

Hide abstract

Compared to hearing subjects, psychophysical studies have revealed specific superior visual abilities in the early-deaf, as well as enhanced auditory functions in the early-blind. The neural substrate for these superior sensory abilities has been identified to reside in the deprived cerebral cortices that have been reorganized by the remaining sensory modalities through crossmodal plasticity. Furthermore, the cartography of auditory cortex is altered following the loss of auditory input early in life. The current investigation examines how perinatal exposure to brief periods of acoustic stimulation alters the developmental trajectory of auditory cortex.

Compared to hearing animals, movement detection, localization of a flashed stimulus in the visual periphery, and face discrimination learning are superior in congenitally deaf cats. These enhanced functions are localized to specific regions of deaf auditory cortex. To examine the role of acoustic experience in mediating these enhanced visual functions in the deaf, hearing animals were chemically deafened with ototoxic drugs at increasing ages postnatal. The animals had one to twelve weeks of acoustic exposure prior to deafness onset. In adulthood, the cats were trained and tested on the same visual tasks examined in the congenitally deaf cats. Overall, following twelve weeks of acoustic experience, no enhanced visual abilities could be identified. With only four weeks of acoustic exposure, the enhanced motion detection ability was not evident. These reduced levels of enhanced visual functions are correlated with changes in cortical cartography. As acoustic experience increased during development, the overall size of auditory cortex, and the size of individual auditory areas also expanded.

These results demonstrate that increasingly longer periods of perinatal acoustic exposure result in reduced enhanced visual abilities and an increased size of auditory cortex.

Supported by the Canadian Institutes of Health Research.

Hide abstract

Synesthesia is a condition wherein one sense is evoked by another. How this arises during development remains largely unknown. We investigated auditory responses and audiovisual interactions in the primary visual cortex (V1) of several mouse models to estimate the level of mixture across senses. Sound modulation of V1 spiking activity was temporally dynamic dependent upon the level of GABA-mediated inhibition. Acute optogenetic suppression of either parvalbumin- (PV) or somatostatin- (SOM) expressing inhibitory interneuron subtypes suggested their early or late recruitment by sound, respectively. Maturation of GABAergic circuits in V1 further dictates the timing of a critical period (CP), a developmental window of enhanced plasticity for visual receptive field properties. One role of maturing GABA circuit function during the CP was to dampen the net non-visual sensory influence. Orientation selectivity was thus impervious to sound specifically during the CP due to balanced amounts of sound-driven spike enhancement and suppression. Cross-modal activation of V1 persisted in dark-reared mice whose CP fails to close. Recent studies interestingly suggest a higher incidence of synesthesia among people with autism, which typically display a comorbid imbalance of excitatory/inhibitory (E/I) circuit function. Analysis of V1 auditory responses in the inbred BTBR mouse strain with autistic features revealed a robust contralateral bias to sound stimuli. Early spike modulation of V1 by sound was shifted toward enhancement similarly across three different autism models (BTBR, SCN1A R1407X, Neuroligin3 R451C). Taken together, our results suggest that developmental E/I imbalance may be a common CP circuit dysfunction underlying autism and synesthesia.

Hide abstract