Acoustical Society of America
133rd Meeting Lay Language Papers

Investigation One - Association Judgment

In a preliminary study, five scenes were selected from the movie Star Trek IV: The Voyage Home. The audio track was erased and was edited directly from a compact disc recording of the musical score, so that the resulting AV combinations consisted only of visual image and musical soundtrack. In other words, all ambient sound (dialogue, background noise, and sound effects) were removed, so that subjects made their decision based solely on the appropriateness of the pairing of music with the visual scene. The results are shown in Figure 2, in which the diagonal (boldface, italicized font) cells represent the composer-intended combinations.

As you can see, in every case, the majority of the sixteen subjects selected the composer-intended combination. In order to determine subjects’ listening strategies, a second experiment was carried out using the same AV combinations. An independent group of fifteen subjects were asked to watch each combination in a random order and rate them on the following ten scales, which were also randomly ordered:

Past research (Osgood, Suci, & Tannenbaum, 1957) has shown that these adjective pairs separate into the following categories: Evaluative (good/bad, beautiful/ugly, interesting/boring, and effective/ineffective), Potency (strong/weak, heavy/light, and tense/relaxed), and Activity (active/passive, fast/slow, and agitated/calm). Mean ratings (average scores) for each AV combination are shown in the graphs below:

[insert graph thumbnails here]

Overall, the results show that music exercises a strong and consistent influence over subject responses to an AV combination, regardless of visual stimulus. In fact, within the context of the present experiment, music exerted a stronger influence over the subject ratings than did the visual image. A careful music-theoretical analysis of the musical soundtracks revealed that several musical parameters consistently influenced subject ratings. These specific musical parameters include:

clarity of tonal center

harmonic complexity

dynamic variation

tempo (speed and consistency)

phrase structure

amount of melodic activity

Investigation Two - Accent Structure Alignment

Past film music research (Bolivar, Cohen, & Fentress, 1994; Bullerjahn & Güldenring, 1994; Iwamiya, 1994; Lipscomb & Kendall, 1994; Marshall & Cohen, 1988; Sirius & Clarke, 1994; Tannenbaum, 1956; Thayer & Levenson, 1984; Thompson, Russo, & Sinclair, 1994) has focused almost exclusively on the associational (or referential) relationship between the visual images and musical sound. The second experiment reported herein (Lipscomb, 1995) breaks with this trend, focusing instead on the relationship between emphasized moments in the visual image and their alignment or misalignment with emphasized points in the musical sound. To illustrate this relationship, the following examples are provided:

[insert Quicktime movies]

consonant out-of-phase dissonant

Consonant AV combinations occur when the audio and visual accent structures are perfectly synchronized, i.e., every visual accent is accompanied by a synchronized musical accent. Accent structures that are out-of-phase share a common temporal interval between consecutive points of emphasis but are misaligned by a perceptually salient amount. A dissonant relationship exists when visual and auditory accents occur at different rates. These relationships can be illustrated visually, as in Figure 3.

Three independent groups of twenty subjects were asked to rate the synchronization and effectiveness of a collection of AV combinations. In each stimulus set, every combination was designed to be either consonant, out-of-phase, or dissonant. The primary distinction between the stimuli viewed by each group of subjects was the complexity of the audio-visual combinations. The first group rated combinations of simple, single-object animations (created by the author) and isochronous pitch sequences. The second group rated combinations excerpted from experimental animations created by Norman McLaren. The third group rated scenes from Obsession, a Brian DePalma film with a musical score composed by Bernard Herrmann. Once again, all ambient sound was removed, so that the AV composites consisted of visual images and musical sound only. An example of each is provided below:

[insert Quicktime movies]

Across all three stimulus sets, subjects reliably distinguished between the three alignment conditions, providing high synchronization ratings for consonant AV combinations, low ratings for dissonant combinations, and moderate ratings for out-of-phase combinations. Ratings for each of the alignment conditions followed this pattern for all excerpts. Therefore, the subject responses were collapsed across alignment condition resulting in the mean scores depicted in Figure 4. Notice, that as the stimuli became more complex (i.e., actual movie excerpts in Experiment 3 rather than single-object animations in Experiment 1) the mean ratings for consonant combinations dropped, while the ratings for dissonant and out-of-phase combinations reflect a higher degree of tolerance. Ratings of effectiveness for these same AV combinations reveal a slightly higher degree of tolerance for out-of-phase and dissonant combinations than the synchronization ratings (Figure 5a). In Experiment 3 (incorporating actual movie excerpts) subjects did not distinguish between the consonant and out-of-phase conditions, suggesting that accent structure alignment plays a less prominent roles in the determination of an effective AV combination when considering complex stimuli. The complex stimuli in Experiment 3 also revealed the only statistically significant difference between highly-trained musicians (those individuals with more than seven years of private study) and subjects with lower levels of musical training. As shown in Figure 5b, highly-trained musicians rated the dissonant combination significantly lower than did the untrained and moderately trained musicians.