Description

Multimodal Discourse Analysis (MDA) focuses on “understanding and describing semiotic resources and principles of their systems of meaning in order to understand how people use these resources in social contexts for specific purposes” (Jewitt, 2009, p. 33). Meaning is multimodal and can be constructed in many modes, including words, images, sound, gesture, and layout. Modes are forms of communication that operate simultaneously; a viewer of multimodal work experiences several modes together across the senses. By combining modes, meaning-making has greater power and intensity (Jewitt, 2009). Further, elements among different modes can reinforce meanings, complement, or contradict each other to create new meanings not evident in the elements in one of the modes. Meaning is multimodal, and can be constructed in many modes, including words, images, sound, gesture, and layout. Modes are forms of communication that operate simultaneously; a viewer of multimodal work experiences several modes together across the senses. An informed analysis will avoid overlooking or isolating modes. For example, tone of voice, gesture, and gaze all affect the meaning of words used in an exchange or dialogue. Examining the verbal mode alone would deprive the message of its meaning.

Jewitt’s (2006) introductory text outlines three multimodal approaches. Social semiotics is closely associated with scholars Kress and van Leeuwen (2001). The second approach comes from systemic functional grammar linked to O’Halloran (2004, 2005). Finally, Scollon and Scollon (2003) and Norris (2004) inform multimodal interaction analysis. Comparing these approaches, Jewitt (2006) notes that they stress different aspects in relation to their contexts, systems, and sign-makers.

This research guide contains many resources that provide meaningful entry points into MDA.  To begin with scholarly works, the New London Group’s (1996) ground-breaking reimagining of literacy and pedagogy introduces multimodality as a part of multiliteracies. It is a historical starting point and theoretical valorization of this field in education. Works by Kress (2013) and Machin (2016) contribute overviews to MDA. Erickson (2004) presents a more technologically informed view by discussing video interviews in the 1970s. Complicating some …elements among different modes can reinforce meanings, complement, or contradict each other to create new meanings not evident in the elements in one of the modes. An informed analysis will avoid overlooking or isolating modes. of the earliest multimodal work in images, Ravelli and Van Leeuwen (2018) have only recently discussed changes of technology’s agentive affordances in the early 21st century.

In addition to the numerous academic works, other technology resources are useful for assisting with teaching, learning, and research in MDA. Atlas.ti, Dedoose, and VideoAnt are three applications that assist with the analysis of multimodal data. Secondly, a short YouTube media clip of Gunther Kress’ work provides a personal and social understanding of modes. For conference presentations, VoiceThread will help to disseminate findings in an attractive combination of images, motion, and sound. To make the best use of these tools, researchers may explore instructions online (see the “Internet Resources” section below).

Finally, several dissertations have also incorporated MDA to create new empirical studies.  These works represent scholarly ideas combined with technological support. Fiscus (2018) uses multimodal analysis in an original exploration of reflective practice, and Ma (2005) provides a comprehensive examination of signage in charity work.  Lastly, Lewis (2013) examine multimodality in communication on the ground with a study of a mother and child. Without a doubt, multimodality presents a field wide with possibilities. To this end, we hope this research guide will generate fruitful results for all interested parties.

References

Jewitt, C. (2009). Different approaches to multimodality. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (2nd ed., pp. 28-39). New York, NY: Routledge.

Kress, G. R. and T. van Leeuwen. (2001). Multimodal discourse: The modes and media of contemporary communication. London, England: Routledge.

Norris, S. (2004). Analyzing multimodal interaction: A methodological framework. New York, NY: Routledge.

O’Halloran, K.L. (Ed.). (2004). Multimodal discourse analysis: Systemic functional  perspectives. New York, NY: Continuum.

O’Halloran, K. L. (2005). Mathematical discourse: Language, symbolism and visual images. New York, NY: Continuum.

Ravelli, L., and van Leeuwen, T. (2018). Modality in the digital age. Visual Communication, 17(3), 277-297. https://doi.org/10.1177/1470357218764436

Scollon, R. and S. Scollon. (2003). Discourses in place: Language in the material world. London, England: Routledge.

Key Research Books and Articles on Multimodal Discourse Analysis

The New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60-93.

Although most of their work focuses on multiliteracies, the New London Group (1996) noticed new forms, or multiple modes, of communication appearing with new technologies. The New London Group identify modalities as systems of meaning-making, which encapsulate the various modes of meaning. Thus, these forms of communication and meaning-making are achieved through multimodalities. The use of this type of communication increases cultural and linguistic diversity across all societies. In fact, the New London Group (1996) identifies multimodal meanings as “the most significant, as it relates all the other modes in quite remarkably dynamic relationships … relat[ing] the linguistic to the visual and to the gestural in intricately designed ways” (p. 80). With these new communications came new cultural experiences and meanings, furthering impacting discourses and creating new discourses for the next generation. The spaces where meanings are made are referred to as lifeworlds marked by “language, discourse, and register differences” (New London Group, 1996, p. 71). However, differences continue to blur and overlap as individuals identify with and participate in several communities concurrently. Practically speaking, classrooms make visible this variety of differences and communities. It will be important for schools to welcome the range of communication styles and meaning making that students and their families bring to schooling.

Erickson, F. (2004). Origins: A brief intellectual and technological history of the emergence of multimodal discourse analysis. In P. Levine & R. Scollon (Eds.), Discourse and technology: Multimodal discourse analysis (pp. 196-207). Washington, DC: Georgetown University Press.

This book chapter reviews the history of Multimodal Discourse Analysis. It begins with use of discourse analysis, adding audiovisual technologies to it, which created Multimodal Discourse Analysis. Erickson suggests that the possibility of multimodal analysis originated in film. He describes how one group began to look at the communicative significance of all verbal and nonverbal communication within a film. Erickson also uses many examples, including his own doctoral research, to illustrate the advantages of video over audiotape in the rich study of communicative behaviors. Erickson ends with implications for future study of MDA stating that “it is desirable to consider verbal and nonverbal behavior together” (p. 202) and to not focus on single events in the analysis but to “identify patterns of communicative activity” (p. 203). The article and its parent larger work echo a central theme: all discourse is multimodal. Although the text provides a deep historical account, the chapter lacks any explicit guidelines or information for conducting multimodal discourse analysis. Even with these critiques, this is a strong resource for reviewing the history of multimodal discourse analysis.

Kress, G. (2013). Multimodal discourse analysis. In J. P. Gee & M. Handford (Eds.), The Routledge handbook of discourse analysis (pp. 61-76). New York, NY: Routledge.

This work situates Multimodal Discourse Analysis (MDA) inside of the larger topic of discourse analysis. Written for scholars, this reference work gathers key definitions and concepts of MDA and explains how it is positioned within discourse analysis proficiency. Kress, an expert in the field, defines MDA terms and provides overviews for important issues in MDA. This chapter also demonstrates how MDA informs learning and social life. Lastly, Kress connects to the importance of social semiotics by outlining their process within MDA, which aims “to elaborate tools that can provide insight into the relation of the meanings of a community and its semiotic manifestations” (p. 37).

Kress provides useful definitions and clarifications for several terms and differentiates between several analytical approaches, originating from linguistics, sociolinguistics and sociology. He identifies the blurred meaning of text and discourse as problematic. He explains it by discussing his belief that ideology in MDA is present in the weaving and the weaver of the text as well as the coherence shaped by them and remains a significant issue at all times. When working within MDA, the many textual threads such as “gesture, speech, image (still or moving), writing, music (on a website or in a film)” are situated texts in the social realm. This chapter expounds on multimodality as a field of research and social semiotics as a theory with which to approach that field.

Machin, D. (2016). Introduction to multimodal analysis. London, England: Bloomsbury Publishing.

David Machin offers an introduction to multimodal analysis using everyday examples, such as movie posters and advertisements. The book is an accessible way to understand the impact of semiotic modes. Machin’s explanations are light on academic prose but draw on seminal ideas of leaders in the field, such as Halliday’s metafunctions (ideational, interpersonal, textual) and Kress and van Leeuwen’s concepts of visual grammar. The book devotes separate chapters to semiotic elements, including color, typography, gaze, participant, and layout, and traces their historical use and evolution. It provides information useful for immediate application. For example, space in a frame is divided such that the top of the frame represents the ideal and the bottom the real. The left side holds given information, while the right side presents new information. The concluding chapter brings into question the idea of a visual grammar, and highlights distinctions between image and language. For readers seeking to understand how signs communicate idea and feeling, this book is a good starting point; however, the books’ breadth over depth coverage can reduce its academic potency.

Ravelli, L., & van Leeuwen, T. (2018). Modality in the digital age. Visual Communication, 17(3), 277-297. https://doi.org/10.1177/1470357218764436

Ravelli and van Leeuwen’s (2018) article aims at establishing new understandings in semiotic multimodality research. The authors seek “to set the agenda for such adaptations, and more generally, for rethinking visual modality and its impact in the digital age” (p. 277).

Their key argument states that “[m]odality…is about what is represented as real and it recognizes that a variety of reality criteria exist and that different coding orientations are adopted in different social contexts and genres in ways that are subject to historical change” (p. 294). They discuss diverse methods of production, coding orientation standards, and the contributions of technology to sharing and viewing multimodal texts. The authors delineate three new future research directions: an updated examination of multimodality variables in digital tools, guidelines for new coding schemes, and critical inquiry for issues of power and identity.

One detracting feature of the text, a short discussion of critical views for pedagogy and school, lacks an expanded argument with examples or illustrations. Overall, however, scholars who wish to study image as mode will benefit from numerous modal marker definitions. Additionally, the authors’ prose is clear and would make an excellent introductory, non-technical text for new readers to this field.

Recent Dissertations Using Multimodal Discourse Analysis Methodology

Fiscus, J. (2018). Reflection in motion: A case study of reflective practice in the composition classroom. Retrieved from ProQuest Dissertations & Theses Global. (Order Number 10828873)

This dissertation offers scholars insight into rhetorical characteristics of reflective practice. The purpose is to investigate the rhetorical in reflective practice. Yancy’s (2016) reflection-in-presentation inspired reflective practice by showing learners’ writing processes and transfer practices but not their work as a rhetorical argument. This study triangulates context where reflection is in process and deconstructs it for embodied meaning. Using a mixed methods approach of ethnographic observation, Multimodal Discourse Analysis (MDA) of observations, and genre analysis of written artifacts, the author reveals the qualities of rhetorical reflection: it is habitual, embedded, distributed, and entangled. Fiscus’s methodology sought to be responsive to reflection-in-motion (p. 34). Though MDA is not a typical method for reflective research, the author employs it to discover new findings (p. 43). Grounding her work in MDA, scholars such as Levine & Scollon (2004), Norris (2004), and O’Halloran (2004), Fiscus used a GoPro to capture her observations and cross-analyze them with interviews and artifacts to uncover the discursive meanings of these multimodal data collections. She used MDA to account for the body and the ways in which embodied symbolism (like pausing) might indicate reflection. MDA accounts for the embodied and discursive communication work in tandem to express reflection. The author purports that future possibilities, such as coding gestures, using eye tracking software, or mapping physical movement, would all be research areas for reflection and embodiment.

Lewis, T.Y. (2009). Family literacy and digital literacies: A redefined approach to examining social practices of an African-American family. Retrieved from ProQuest Dissertations & Theses Global. (Order Number 3366120)

Lewis uses an ethnographic case study approach to explore digital literacy practices between a mother and son in their home. Participants’ observations were video and audiotaped, and the mother and son were interviewed. MDA was used to explore how unspoken communication can carry meaning. Lewis believes a researcher “cannot fully understand a practice unless one is able to read the signs of how meaning is construed” (2013, p. 51). She created two column notes where column one served as a place for pictures of the action and column two described what meaning was created through the unspoken communication. Lewis used this method of MDA to investigate the son taking apart a computer and playing a video game with his cousin. In contrast, Lewis created multiple column charts with labels such as: scene, time, action/context, verbal discourse, effect on the action. The author then investigated multimodal practices such as touch between a mother and son and instant messaging via telephone and computers. Lewis states that using MDA allowed her to answer her research questions and see more of the relationship of mother and son; however, she fails to explain why she changed chart formats for different types of MDA. This dissertation is useful to imagine different ways to chart and use MDA in qualitative research.

Ma, M. L. [馬美蓮]. (2005). Multimodal discourse analysis of advertisements of Hong Kong charity organizations. (Thesis). University of Hong Kong, Pokfulam, Hong Kong SAR. Retrieved from http://dx.doi.org/10.5353/th_b3178972

Ma examined thirty-three advertisements on the Hong Kong Mass Transit Railway (MTR) from a multimodal perspective to understand how organizations use text and image to appeal to viewers. She focuses on advertisements by charitable organizations, whose marketing approach differs from that of for-profit organizations because the donor receives nothing concrete in return for their money. Ma provides a concise but complete breakdown of prominent taxonomies for describing color. For example, color has eight properties: saturation, differentiation, modulation, contextualization, representation, depth, illumination, and brightness (Kress & van Leeuwen, 1996). Designers can adjust these facets to influence the communicative impact of color on the viewer. The author also uses three levels of analysis: micro, meso, and macro. Micro-level analysis focuses on semiotic elements, such as text. Meso-level analysis considers social and organizational influences on the creation of the semiotic elements, while macro-level analysis attends to social concerns appearing in the advertisement. The paper offers succinct, lucid explanations of the effect of a range of semiotic elements, including image, color, layout, and language choice. Ma applies these concepts in her analysis of charity advertisements, thereby offering a didactic opportunity in multimodal analysis. From her work, readers quickly and easily gain a background in theory and practice of multimodal analysis.

Internet Resources

ATLAS.ti [App]. (2018). Retrieved from https://atlasti.com/

ATLAS.ti is a powerful application for managing qualitative research projects involving multiple researchers and substantial quantities of data. A project in ATLAS.ti comprises documents, codes, memos and networks. Documents include audio, video, text, PDF, and graphics files in a range of formats. Codes can be used quite flexibly to label data. For example, you can attach codes to a document as a whole and then attach different codes to sections within the document. Through memos, researchers can record thoughts related to the project. Networks help researchers visualize notes and codes in various graphical configurations, such as circular, hierarchical, and tree, in order to gain new perspective on data. These features assist researchers with the processes of making sense of their data and keeping track of their analytical process.

The integration of these functions allows researchers to keep their data collections together with all coding iterations, memos, and network connections as part of the interpretation process of data analysis.
ATLAS.ti interfaces well with other research software. For example, codebooks can be imported from Excel. References can be imported from EndNote and Mendeley. A free trial version is available on the company’s website, which also has sample projects and links to video tutorials and a YouTube channel. Because it handles data across several modes, ATLAS.ti is a valuable tool for MDA research.

Dedoose [App]. (2018). Retrieved from https://www.dedoose.com/

For multimodal discourse analysis, Dedoose is helpful to capture, store, organize, and analyze multimodal data. Accessible online, Dedoose—originally EthnoNotes—is a Web app for both qualitative and quantitative research. It is a secure, encrypted data analysis software available on web browsers, desktops, and mobile devices such as smartphones and tablets. This application allows researchers to warehouse collected data in many modes — audio, video, text as well as databases of captured data. Coding features are especially helpful for MDA as memos can be set to connect specific locations in any text document, video, or audio file to each other. Queries can be set based on researcher inquiry aims. Data analysis visualizations give researchers a visual display to identify data patterns. This software is preferred if cloud sharing and video data are important criteria for a study. In addition to the complete online data analysis process, Dedoose lets users create visual charts for incorporation into reports and presentations.

Kelley, T. A. (2016 February 4). Gunther Kress “Multimodal discourse analysis”. University of North Florida: UNF Writes. Retrieved from https://youtu.be/Sd1dlaLzwOo

In this video, Tara Kelley, instructor of English at University of North Florida, explains Gunther Kress’ work on Multimodal Discourse Analysis. Kelley highlights the key components of Kress’ argument, including historical ideas and terms, such as coherence and power. Kress presents his model of MDA by describing it, explaining how to use it, and discussing why it should be conducted. His argument is that MDA is cross-cultural, which reveals meaning making and discourse. A discussion on discourse has many entry points, and Kress presents many of these in his article, such as ideology, social semiotics, text and coherence. Kress believes in meaning-making and that meaning is made through agency, identity, social semiosis, knowledge, and power. When an individual puts all of these facets together, meaning is created by that person. This video is an thorough summary of Kress’ ideas and is a exemplary resource for learning about Multimodal Discourse Analysis.

VideoAnt. [App] (2018). Retrieved from https://ant.umn.edu

VideoAnt is a free mobile and web-based application from the University of Minnesota that allows users to annotate videos hosted on web platforms, such as YouTube. The observer watches the video on the left pane and uses the right pane to make comments at any desired timestamp. Strengths of this technology include: collaboration, simplicity of use, and utility for researching, teaching, and learning.  Additionally, all comments from the host and invited participants can be exported with timestamps for ease of data analysis. Users must consider two important limitations. First, VideoAnt comments must have both a title and content, which can slow down annotation speed. Second, the software only functions when the video exists on the web; thus, there is no support for researchers needing offline access. Individuals who privilege the affordances of recorded video in their data collection will find VideoAnt highly beneficial. VideoAnt’s ability to maintain synchronicity between the multimodal text and comments or codes will aid Multimodal Discourse Analysis.

VoiceThread [App]. (2018). Retrieved from https://voicethread.com

VoiceThread is an online multimodal tool that creates presentations with interactions among viewers and creators. Presentations can include videos, pictures, voice recordings, video recordings, and text on each slide. Multiple users may employ the same tools simultaneously during interactions. The website does require a yearly membership fee for instructors and institutional users. However, an individual can create up to six presentations before requiring any payment. VoiceThread’s website requires a few minutes to learn each feature but has seamless collaboration for all ages. The website also provides many videos to show how each feature works within the presentations. MDA researchers can use VoiceThread to analyze various modes, pictures, video, audio, and text, and discourses with a group of fellow researchers. This multimodal tool brings connection, literacy, and education together in one place.