Behavioral and Electrophysiological Investigation of Crossmodal Interaction in Object Recognition
Poster Presentation

*Shlomit Yuval Greenberg
Department of Psychology, The Hebrew University of Jerusalem

Leon Y Deouell
Department of Psychology, The Hebrew University of Jerusalem

     Abstract ID Number: 149
     Full text: PDF
     Last modified: July 3, 2005

Abstract
Recognition of objects in the world may depend on information from various modalities. We investigated the mutual effect of auditory and visual information in this process. In a stroop-like paradigm, fourteen subjects were required to identify animals from pictures or from vocalizations, while information from the other modality could be congruent, incongruent or neutral. Compared to congruent conditions, cross-modal incongruence affected auditory recognition significantly more than it affected visual recognition. Further exploration showed that relative to the neutral condition, cross-modal incongruence in fact hampered recognition similarly in both conditions, suggesting that information from the unattended modality is processed, despite its detrimental effect on performance. However, the cross modal facilitation effects were asymmetrical. Cross-modal congruency facilitated behavior significantly more during auditory than during visual recognition. Mirroring this, event related potential recording from 64 electrodes in these subjects showed that congruency affected the response to the stimuli when auditory recognition was required but significantly less so when visual recognition was required. The congruency effects were manifested as bimodal increased negativities with latencies around 200 and 500 ms after stimulus onset. In additional experiments, we investigate whether the asymmetrical effects result from different speeds of auditory and visual processing of complex information.