The relationship of crossmodal correspondences to language

People can make consistent associations between odors and colors, and these associations have been shown to differ depending on how the odors are named. This suggests an interplay between odor-color correspondences and language. In this talk we will present data that suggests associations between odors and colors can support odor naming. We find that odor-color synaesthetes – individuals who have automatic and vivid color experiences when they smell odors – are better at naming odors than control participants who do not have synaesthesia. We also find that odor-color associations are more consistent for odors that are more nameable, for synaesthetes and controls. However, data from another experiment suggests that language does not directly affect odor-color correspondence online. In a second experiment, participants were asked to match colors to odors in three interference conditions: holding a verbal code in mind, holding a visual pattern in mind, or no interference. Although we again observed that more nameable odors had more consistent color associations, the verbal interference condition did not disrupt the odor-color matches. This suggests that odor-color associations do not arise via the explicit naming of odors. Overall, we propose that odor-color associations strengthen conceptual representations, which lead to stronger connections with language.

Hide abstract

Sound symbolism refers to an association between certain language sounds (i.e., phonemes) and particular perceptual and/or semantic elements (e.g., objects of a certain size or shape). The most well known example of this is the maluma/takete effect, in which phonemes like those in maluma are judged as good matches for round shapes, and phonemes like those in takete are judged as good matches for jagged shapes (Köhler, 1929). While the existence of these associations has been well documented, the mechanism underlying them is still not well understood. Here we review five proposed mechanisms for sound symbolism: 1) a statistical co-occurrence in the world between some phonetic feature and associated stimuli, that has been internalized; 2) a shared property among phonemes and associated stimuli, which may be perceptual or conceptual; 3) a common neural mechanism in the processing of phonemes and associated stimuli; 4) an evolved, species-general, association; and 5) patterns extracted from the lexicons of existing language. Importantly, these mechanisms need not be mutually exclusive and likely interact in the creation of sound symbolism. While sound symbolism is often considered an instance of crossmodal correspondence, we note that the involvement of complex linguistic stimuli in sound symbolism is an important distinction that creates additional complexities in the search for its underlying mechanism. Lastly, we report the results of a study that represents our first step in testing these proposed mechanisms. Adult participants rated a sample of nonwords and abstract shapes on 25 semantic differential scales; participants also rated the fit between nonwords and shapes. Results suggest that shared conceptual properties among nonwords and shapes contributes to shape sound symbolism. We consider these results in the context of the five proposed mechanisms.

Hide abstract

Crossmodal correspondences (CCs) occur between a variety of sensory stimuli (Spence, Attention, Perception & Psychophysics, 2011). Sound symbolism refers to the associations between the sounds of words and their meanings. This has often been studied empirically using CCs between auditory psedudowords and visual shapes; e.g., the pseudowords “takete” and “maluma” are associated with spiky and rounded shapes, respectively. Sound-symbolic associations have been suggested to underlie the origins of language and to form a continuum with synesthesia (Ramachandran & Hubbard, Journal of Consciousness Studies, 2001). Relevant to this idea, we reported earlier that synesthetes exhibit stronger sound-symbolic CCs than non-synesthetes (Lacey et al., European Journal of Neuroscience, 2016). Yet, little is known of how sound-symbolic CCs are processed neurally.

We examined the neural basis of sound-symbolic CCs using functional magnetic resonance imaging (fMRI). The auditory pseudowords “lohmoh” and “keekay” were used: these were previously shown to be rated as rounded or pointed, respectively. They were paired with visual shapes that had either rounded or pointed contours, presented simultaneously. When participants attended to the auditory stimuli to make a two-alternative forced-choice, sound symbolically incongruent visual shapes evoked stronger responses compared to congruent ones, in the anterior intraparietal sulcus and supramarginal gyrus bilaterally, and in the right postcentral sulcus, and left middle frontal and superior parietal gyri. Comparing these incongruency-related activations to functional localizers suggested relationships of the underlying neural processes to those involved in multisensory integration, magnitude estimation and phonology. In addition, these findings point to an important role for multisensory attention. Ongoing studies reveal significant correlations between the dissimilarity matrices for the perceptual ratings of pointedness and roundedness between a range of auditory pseudowords and visual shapes. We are currently investigating the relationship between corresponding neural dissimilarity matrices in neocortical regions that process these stimuli.

Supported by NEI

Hide abstract