The Role of Experience in the Development of Multimodal Integration

Organizer: Daphne Maurer, McMaster University

Abstract: There are major postnatal changes in multimodal integration. This symposium will consider the role of experience in shaping those changes. First, David Lewkowicz will discuss normal human development during infancy. He will describe the transformation the rudimentary abilities present at birth over the first year of life by experience with specific languages and specific types of faces. Second, Daphne Maurer will discuss changes in the development of audiovisual integration in patients treated for dense congenital cataracts in one or both eyes, even when vision was restored during early infancy. Those results suggest that crossmodal re-organization begins to occur near birth but is compromised differentially by early deprivation to one versus both eyes. Third, Anu Sharma will consider a similar issue for patients deaf from an early age in one or both ears. Using high density EEG, she will illustrate responses of auditory cortex to visual and somatosensory stimuli after early deafness, with different patterns across patients that correlate with success with cochlear implants. Fourth, Steve Lomber will report on an animal model of such cortical re-organization after deafness: deafened cats show enhanced visual sensitivity and visual responsiveness in auditory cortex, but the extent of the changes depends on the age of deafening and the amount of acoustic experience preceding it. Finally, Takao Hensch will describe a mouse model showing the role of the factors controlling critical period plasticity in shaping the development of cortical audiovisual interactions. Collectively, the papers will present new insights by considering the role of experience in shaping the development of multimodal integration from multiple perspectives: development in humans and in animal models; the effects of normal input versus input altered by auditory or visual deprivation; and evidence from behavioural and cortical studies.


S4.1 Early Experience Shapes the Development of Selective Attention and Multisensory Processing in Human Infants

David Lewkowicz, Northeastern University

Show abstract

The everyday objects and events in an infant’s world are often specified by multisensory attributes. For example, whenever infants interact with social partners, they can see and hear them talking. To perceive their social partners as perceptually unitary and psychologically meaningful entities, infants must attend to their multisensory attributes and process them. We and others have found that multisensory processing is rudimentary at birth and that it improves gradually as infants grow and acquire experience. In this talk, I will review evidence from our lab indicating that early experience is critical for the emergence of multisensory processing in infancy. First, I will show that the multisensory world of young infants is relatively undifferentiated because of sensory limitations and lack of experience. Then I will show that infants gradually shed their primitive multisensory processing mechanisms in favor of increasingly more refined and mature ones and I will provide evidence that this developmental process is partly driven by experience-dependent narrowing of responsiveness to multisensory inputs. Narrowing will be illustrated by evidence from studies of infant matching of other-species faces and voices, matching of native and non-native auditory and visual speech, and discrimination of own and other-race faces. Overall, these findings will show that infants initially exhibit paradoxically broad tuning and, thus, detect the multisensory unity of native and non-native multisensory inputs and that, as they acquire mostly native-input experience, they cease detecting the unity of non-native multisensory inputs. Finally, I will review findings from our recent studies of infant selective attention to fluent audiovisual speech and demonstrate that attention undergoes marked, experience-dependent, developmental changes during the first two years of life. I will conclude by highlighting the intricate relationship between selective attention and multisensory processing and the way that early experience affects the development of both processes.

Supported by Eunice Kennedy Shriver National Institute of Child Health and Human Development, Grant number: R01HD057116

Hide abstract


S4.2 The Development of Audiovisual Integration: New Insights from Adults treated for Congenital Cataract

Daphne Maurer, Yi-Chuan Chen, David Shore, & Terri L. Lewis, McMaster University and The Hospital for Sick Childen

Show abstract

Patients treated for congenital cataract provide a natural experiment for investigating the role of visual input in perceptual development. Here, we considered the effects of early visual deprivation on the development of audiovisual integration in adults treated for unilateral or bilateral congenital cataract at 0.3-28.8 months of age (n=13/grp).

Simultaneity judgments. Controls showed a typical gaussian curve peaking when the flash slightly preceded the sound, a shift usually attributed to plastic adjustment for slower travel of sound than light. The curve for bilateral patients was abnormally wide, but only on the flash-leading side, the condition most plastic in normal adults (Chen et al., 2017). The curve for unilateral patients had a normal peak but was abnormally wide for both sound- and flash-leading sides as is found in typically developing children (Chen et al., 2016).

Fission illusion. Bilateral patients had a smaller illusion than controls, with less influence of the beeps on the perceived number of flashes. This pattern occurred despite evidence of auditory dominance in other tasks: they are faster than controls at detecting a beep and at switching attention from vision to audition (de Heering et al., 2016) and in fMRI images, show auditory activation of visual cortex (Collignon et al., 2015). Unilateral patients behaved like the control group, with a similarly robust illusion, which was normally modulated between centre (smaller) and periphery (larger).

Bilateral patients’ abnormalities suggest that early visual input is necessary both to set up the neural substrate for later visual development and also to allow normal audiovisual integration. When input is missing to both eyes, even for just the first few months of life, auditory input establishes effective connections in “visual” cortical areas and leads to enhanced auditory processing, but at a cost to effective audiovisual integration. When the deprivation is unilateral, early input through just one eye may be sufficient to reduce that re-organization.

Supported by CIHR, NSERC, James S. McDonnell Foundation

Hide abstract


S4.3 Cross-Modal Plasticity in Deafness: Evidence from Children and Adults Fitted with Cochlear Implants

Anu Sharma, University of Colorado, Boulder

Show abstract

Congenital deafness prevents the normal growth and connectivity needed to form a normally functioning sensory system– resulting in deficits in oral language and cognition in deaf children. Cochlear implants (CI) bypass peripheral cochlear damage, by directly stimulating the auditory nerve, making it possible to avoid many of the harmful effects of auditory deprivation. Patients who receive CI’s provide a platform to examine the characteristics of neuroplasticity in the central auditory system. An important aspect of compensatory plasticity that is evident in deafness is the re-organization and re-purposing of auditory cortex by other sensory modalities known as cross-modal plasticity. We examined cross-modal plasticity from the visual and somatosensory systems in deaf children with bilateral deafness fitted with CIs and in deaf adults and children with single-sided deafness fitted with CIs and related them to communication and cognitive outcomes. High-density 128 channel electroencephalographic (EEG) recordings to auditory, visual and somatosensory stimulation, tests of auditory-visual speech perception in noise and cognition were administered. Patients with cochlear implants showed activation of auditory cortical areas in response to visual and vibrotacticle stimulation, suggestive of cross-modal recruitment by the visual and somatosensory modalities. Cochlear implanted patients who had greater difficulty understanding speech in noise via their cochlear implant showed greater evidence of cross-modal recruitment. In persons with single-sided deafness the (unstimulated) cortex contralateral to the deaf ear showed greater evidence of cross-modal recruitment. Cochlear implantation appeared to reverse cross-modal re-organization in many (but not all) patients with single-sided deafness. Overall, our results show that compensation for degraded auditory input results in greater dependence on other modalities which serves to aid communication in real-world situations. Our results suggest that the alterations in neural circuitry that underlie compensatory cross-modal plasticity may be an important source of variability in outcomes for patients with CIs.

Supported by the US National Institutes of Health.

Hide abstract


S4.4 Short Periods of Perinatal Sensory Experience Change the Structure and Function of Auditory Cortex

Stephen G. Lomber and M. Alex Meredith, University of Western Ontario (Canada) and Virginia Commonwealth University (USA)

Show abstract

Compared to hearing subjects, psychophysical studies have revealed specific superior visual abilities in the early-deaf, as well as enhanced auditory functions in the early-blind. The neural substrate for these superior sensory abilities has been identified to reside in the deprived cerebral cortices that have been reorganized by the remaining sensory modalities through crossmodal plasticity. Furthermore, the cartography of auditory cortex is altered following the loss of auditory input early in life. The current investigation examines how perinatal exposure to brief periods of acoustic stimulation alters the developmental trajectory of auditory cortex.

Compared to hearing animals, movement detection, localization of a flashed stimulus in the visual periphery, and face discrimination learning are superior in congenitally deaf cats. These enhanced functions are localized to specific regions of deaf auditory cortex. To examine the role of acoustic experience in mediating these enhanced visual functions in the deaf, hearing animals were chemically deafened with ototoxic drugs at increasing ages postnatal. The animals had one to twelve weeks of acoustic exposure prior to deafness onset. In adulthood, the cats were trained and tested on the same visual tasks examined in the congenitally deaf cats. Overall, following twelve weeks of acoustic experience, no enhanced visual abilities could be identified. With only four weeks of acoustic exposure, the enhanced motion detection ability was not evident. These reduced levels of enhanced visual functions are correlated with changes in cortical cartography. As acoustic experience increased during development, the overall size of auditory cortex, and the size of individual auditory areas also expanded.

These results demonstrate that increasingly longer periods of perinatal acoustic exposure result in reduced enhanced visual abilities and an increased size of auditory cortex.

Supported by the Canadian Institutes of Health Research.

Hide abstract


S4.5 Enhanced cross-modal auditory response in primary visual cortex with altered critical period timing

Takao Hensch, Harvard University, Boston Children’s Hospital

Show abstract

Synesthesia is a condition wherein one sense is evoked by another. How this arises during development remains largely unknown. We investigated auditory responses and audiovisual interactions in the primary visual cortex (V1) of several mouse models to estimate the level of mixture across senses. Sound modulation of V1 spiking activity was temporally dynamic dependent upon the level of GABA-mediated inhibition. Acute optogenetic suppression of either parvalbumin- (PV) or somatostatin- (SOM) expressing inhibitory interneuron subtypes suggested their early or late recruitment by sound, respectively. Maturation of GABAergic circuits in V1 further dictates the timing of a critical period (CP), a developmental window of enhanced plasticity for visual receptive field properties. One role of maturing GABA circuit function during the CP was to dampen the net non-visual sensory influence. Orientation selectivity was thus impervious to sound specifically during the CP due to balanced amounts of sound-driven spike enhancement and suppression. Cross-modal activation of V1 persisted in dark-reared mice whose CP fails to close. Recent studies interestingly suggest a higher incidence of synesthesia among people with autism, which typically display a comorbid imbalance of excitatory/inhibitory (E/I) circuit function. Analysis of V1 auditory responses in the inbred BTBR mouse strain with autistic features revealed a robust contralateral bias to sound stimuli. Early spike modulation of V1 by sound was shifted toward enhancement similarly across three different autism models (BTBR, SCN1A R1407X, Neuroligin3 R451C). Taken together, our results suggest that developmental E/I imbalance may be a common CP circuit dysfunction underlying autism and synesthesia.

Hide abstract



Event Timeslots (1)

Friday, June 15