Visual Perception and Emotion In Music

December 2012 | Research Study

Survey and Visual Display

For the purposes of this study, I selected the four basic elements of music (time, pitch, timbre and loudness) to accurately represent the selected film score visually. I displayed time on the horizontal axis, with pitch being displayed vertically. I displayed timbre through the use of color by dividing a color wheel into twelve sections that correspond to the twelve musical notes (Figure 1). This became the participant key for interpreting the visual display musically. The brightness of the color was used to represent the loudness. With these design elements in place, I was able to create three different displays for the key target audience.

I used a paper-based survey, administered after the experience of walking through visual display, to collect relevant data. The survey involves the participants’ perception of our five basic emotions (happiness, sadness, fear, love and anger) in response to the visual display. Participants were asked to communicate the emotion through color and draw where and how it is experienced in specific parts of the body (e.g. happiness in chest, etc). A color palette was provided to facilitate participant selection of the color they associate with a specific emotion. Figure 2 provides mode detail on the survey content and structure.

Figure 1: Visual Key
Figure 1
Figure 2: Survey Front
Figure 2
Figure 2: Survey Back
Figure 2

I designed the visual display to connect with students from the three major departments in the College of Fine Arts at the University of South Dakota: fine arts, music and theater. Given that the display was exhibited in the hall of the USD Fine Arts Building, promoting walker-by participation was of importance. The selected music score was the main theme of the film The Godfather, and is linked with the main character’s (Michael) doomed courtship and marriage to his first wife Apollonia. Entitled “Speak Softly, Love,” it evokes emotional associations to nostalgia, sadness and loss and communicates both the beauty and tragedy of this storyline. The score’s sheet music was transcribed utilizing the color key and was reinterpreted visually.

Squares, Spheres, and Stampings

The visual display of the film score was a continuation of the same song and divided into three sections that were relatable to each demographic. For example, the fine arts students could connect to a purely visual display consisting of colored squares adhered to a white cinder block wall. The natural grid provided by the blocks was used to replicate the arrangement of musical notes. These squares referred to particular note from the participant key and consisted of a visual representation of the first part of the score (Figure 3).

The music students were taken into consideration when creating a sensory representation of the second part of the score. The sensory component consisted of hanging paper mache spheres that visually represented notes, placed in relation to each other as they would in sheet music. These spheres were constructed by taking the actual sheet music of the original score that inspired “Speak Softly, Love,” a playful attempt at a clue. The spheres were stained red to evoke the themes of the Godfather film, mainly bloodshed, and were infused with perfume that hinted at a sense of nostalgia. To finish off the display, black wire was selected to hang the spheres from the ceiling of the hallway, alluding to the famous theme in the film where a strangling occurs by wire. Finally, cues to the notes represented in the spheres were offered by the presence of yarn interwoven though the wire, with the color of the yarn representing a specific note on the participant key (Figure 4).

Figure 3: Visual Display
Figure 3
Figure 4: Sensory Display
Figure 4

The physical display representing the last part of the score was designed to relate to the theater students. A set of handprints, printed on colored paper and alternating from left to right, were adhered to a while cinder block wall used as a grid. Representing a form of manual hopscotch, they resembled stampings on a white background and were an invitation to participants to physically interact with the display and literally move to the music (Figure 5).

Figure 5: Physical Display
Figure 5

A 23.5” x 72” poster was used at the start of the visual display representing the participant key, as well as informing them on the purpose of the study. Participants also used the poster to guide them through the exhibit by sticking to specific steps. For example, step one directed them to look at the purely representation of music and see if a pattern emerges. Several smaller posters were also used to connect the three parts of the display and to keep participants engaged and guided through the experience (Figures 6 & 7).

Figure 6: Exhibit Poster
Figure 6
Figure 7: Poster Guide
Figure 7

Findings

Sixteen participants, the majority of whom were female (10) filled out the survey. As expected, the emotions identified by the participants were mainly negative emotions, with 11 of the participants selected either fear (5), sadness (4), and anger (2). The remaining five participants selected a positive emotion, happiness, with the category of love not being selected at all.

Visually communicating the results involved a process of selecting a thematic structure. Given the neurobiological aspect of emotion and the primary role of the brain in making emotional experience possible, the neuron, the cell transmitting information was selected as a design theme. Neurons form neural network as they connect with each other to transfer electrical signals that communicate emotion. The various emotions experienced by the participants were conceptualized as lines originating from the specified body location and ending with a neuron. The neuron was shaded to represent the color the participant associated with the specific emotion. Great detail was given to the form of the emotional line to communicate the emotional state. For example, anger was displayed through sharp, angled lines to convey aggressiveness, while happiness was displayed through a series of dotted lines forming swirls to convey lightness and positivity. The lines of emotion were colored with the specific color selected by the participant from a provided a color palette (Figure 8).

Figure 8: Data Visualization
Figure 8
Figure 8
Figure 8
Figure 8

Four panels, with each panel visually displaying a specific emotion, were printed on transparent film. Each panel included a drawing of a human figure, as well as a different component of a neuron forest. The results were presented together by lining up each transparent panel to graphically connect all the emotions and colors into one visual experience. The different components of the neuron forest lined up to create the larger pattern, much like pieces of a puzzle coming together. A wooden box with two transparent sides made of plexiglass was built to fit each panel, lined up in the following order: happiness, anger, fear, and sadness. Figure 9 provides visual detail on the graphic representation of the findings.

Conclusions

Music plays a unique role in human experience, as indicated by the importance sound has on human communication. Adding to music’s special role is its influence on culture and the emotional component of musical experience. Music has the power to not only communicate emotion through sound, but also transport the listener to a specific emotional state. As indicated by my study, individuals not only experience emotions in relation to music, but they also experience those emotions physically as residing in the body. The chest, head and stomach seem to be prevalent in the physical experience of emotional states inspired by the visual display of music. In addition to experiencing emotion in specific parts of the body, certain visual components were also associated with emotional states. Colors such as red, yellow, orange were frequently identified with various emotional states, but cool colors seemed to be more associated with the specific state of sadness. Participants also indicated direction to their emotions, with anger radiating upwards from the arm, or happiness moving upwards and in circular patterns.

To synthesize the findings from this project, a visual presentation of the results will be shared with the USD community through. The four panels representing the various emotions will be arranged in a wood and plexiglass box, in order to communicate the complexity and distinctiveness of the various emotional states indicated by the participants. It is my hope that the inter-relations between the visual experience of music, emotion, and color as informed by this study will promote a continuation of this discussion and new ways to examine these phenomena.

Figure 9: Display Box
Figure 9