The function of face gestures and head movements in spontaneous humorous communication


Multimodality and humour

How to Cite

Buján, M. (2019). The function of face gestures and head movements in spontaneous humorous communication. The European Journal of Humour Research, 7(2), 1–29.


This paper assesses the results of a multimodal analysis of humorous instances found in a collection of 14 interviews from The Late Show with Stephen Colbert. The interviews have been imported and annotated in ELAN for head movements and face expressions. Only the interviewees’ remarks are studied in order to ensure the analysis of spontaneous speech, avoiding mostly pre-scripted host’s comments and questions. The results demonstrate that there are no gestures consistently and recurrently paired with humour. Therefore, none of the multimodal cues found in the sample can be considered markers of humour. Furthermore, the functions of gestures identified are consistent with the uses reported in previous multimodal studies of humorous and non-humorous communication found in the literature.


Adami, E. (2016). ‘Multimodality’, in García, O., Flores, N. & Spotti, M. (eds.), Oxford Handbook of Language and Society. Oxford: Oxford University Press.

Altorfer, A., Jossen, S., Würmle, O., Käsermann, M.L., Foppa, K. & Zimmermann, H. (2000). ‘Measurement and meaning of head movements in everyday face-to-face communicative interaction’. Behaviour Research Methods Instruments and Computers 32 (1), pp. 17-32.

Attardo, S. (1994). Linguistic Theories of Humor. Berlin: Walter de Gruyter.

Attardo S. (2000b). ‘Irony markers and functions: Toward a goal-oriented theory of irony and processing’. Rask. Internationalt Tidsskrift for Sprog og Kommunikation, 12, pp. 3-20.

Attardo, S., Eisterhold, J., Hay, J. & Poggi, I. (2003). ‘Multimodal markers of irony and sarcasm’. Humor, 16(2), pp. 243-260.

Attardo S., Pickering, L. & Baker, A. (2011). ‘Prosodic and multimodal markers of humor in conversation’, in Attardo, S., Wagner, M. & Urios-Aparisi, E. (eds.), Prosody and Humor.

Amsterdam / Philadelphia: John Benjamins, pp. 224-247.

Attardo, S., Wagner, M. & Urios-Aparisi, E. (2011). ‘Prosody and humor’. Pragmatics & Cognition, 19(2), pp. 189-201.

Attardo, S., Pickering, L., Lomotey, F. & Menjo, S. (2013). ‘Multimodality in Conversational Humor’. Review of Cognitive Linguistics, 11(2), pp. 402-416.

Brône, G. (2008). ‘Hyper- and misunderstanding in interactional humor’. Journal of Pragmatics, 40(12), pp. 2027-2061.

Castelfranchi, C. & Poggi, I. (1990). ‘Blushing as a discourse: was

Darwin wrong?’, in Crozier, R. (ed.), Shyness and Embarrassment. Perspective from Social Psychology. Cambridge: Cambridge University Press, pp. 230-251.

Cavé, C., Gua, I., Bertrand, R., Santi, S., Harley, F. & Espesser, R. (1996). ‘About the relationship between eyebrow movements and F0 variations’. International Conference on Spoken Language Processing, pp. 2175-2178.

Cienki, A. (2013). ‘Cognitive Linguistics: Spoken language and gesture as expressions of conceptualization’, in Müller, C., Cienki, A.,

Ladewig, S., McNeill, D. & Teßendorf. S. (eds.), Body–Language–Communication: An International Handbook on Multimodality in Human Interaction (Vol. 1). Berlin: de Gruyter Mouton, pp. pp. 182-201.

Cvejic, E., Kim, J. & Davis, C. (2010). ‘Prosody off the top of the head: Prosodic contrasts can be discriminated by head motion’. Speech Communication, 52(6), pp. 555-564.

Darwin, C. (1872). The Expression of the Emotion in Man and Animals. London: John Murray.

Debras, C. & Cienki, Al. (2012). ‘Some uses of head tilts and shoulder shrugs during human interaction, and their relation to stancetaking’. 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, Amsterdam, pp. 932-937. doi: 10.1109/SocialCom-PASSAT.2012.136

Drahota, A., Costall, A. & Reddy, V. (2008). ‘The vocal communication of different kinds of smile. Speech Communication’, 50(4), pp. 278-287.

DsNet Corp. (2017). aTubeCatcher Version 3.8.9325, retrieved 1 July 2017 from

Eisterhold, J., Attardo, S. & Boxer, D. (2006). ‘Reactions to irony in discourse: evidence for the least disruption principle’. Journal of Pragmatics, 38, pp. 1239-1256.

Ekman, P. (1979). ‘About brows—emotional and conversational signals’, in von Cranach, M. K., Foppa, W., Lepenies, & Ploog D., (eds.), Human Ethology. Cambridge: Cambridge University Press, pp169-248.

Ekman, P., (1982). Emotion in the Human Face. Cambridge University Press, New York.

Ekman, P. & Friesen, W.V. (1971). ‘Constants across cultures in the face and emotion’. Journal of Personality and Social Psychology, 17(2), pp. 124-129.

ELAN (Version 5.0.0-beta) [Computer software]. (2017, April 18). Nijmegen: Max Planck Institute for Psycholinguistics. Retrieved from

El Kaliouby, R. & Robinson, P. (2004). ‘Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures’, in Kiskanin, B., Pavlovic, P. & Huang, T. (eds.), Real-Time Vision for Human-Computer Interaction. Boston: Springer, pp. 181-200.

Feyaerts, K. (2013). ‘Tackling the complexity of spontaneous humorous interaction: An integrated classroom-modeled corpus approach’, in Ruiz Gurillo, L. & Alvarado Ortega, M. B. (eds.), Pragmatics & Beyond New Series (Vol. 231). Amsterdam: John Benjamins, Publishing Company, pp. 243-268.

Flamson, T., Bryant, G. & Barret, H. (2011). ‘Prosody in spontaneous humour’. Pragmatics & Cognition, 19(2), pp. 189-201.

Flecha-García, M. L. (2010). ‘Eyebrow raises in dialogue and their relation to discourse structure, utterance function and pitch accents in English’. Speech Communication, 52(6), pp. 542-554.

Forceville, C. (2016). ‘Mixing in pictorial and multimodal metaphors?’, n Gibbs R. W. (ed.), Metaphor in Language, Cognition, and Communication, Vol. 6, Chapter 11. Amsterdam: John Benjamins Publishing Company, pp. 223-240.

Gironzetti, E. (2017). ‘Prosodic and Multimodal Markers of Humor’, in

Attardo S. (ed.), The Routledge Handbook of Language and Humor, Chapter 28. Routledge, pp. 235-254.

Gironzetti, E. & Menjo, S. (2014). ‘Smiling as a discourse marker of humor’. Proceedings from 2nd Conference of the American Pragmatics Association (AMPRA). Los Angeles: University of California.

Gironzetti, E., Attardo, S., Pickering, L., Huang, M., Menjo, S. & Zhang, Y. (2015). ‘Multimodal analysis of humour in conversation’. Proceedings from the 27th Conference of the International Society of Humor Studies (ISHS). Holy Names University: Oakland.

Goodwin, C. (1980). ‘Restarts, pauses, and the achievement of a state of mutual gaze’. Sociological Inquiry, pp. 272-302.

Guaitella, I., Santi, S., Lagrue, B. & Cave, C. (2009). ‘Are Eyebrow Movements Linked to Voice Variations and Turn-taking in Dialogue? An Experimental Investigation’. Language and Speech, 52(2/3), pp. 207-222.

Hadar, U., Steiner, T.J., Grant, E.C. & Clifford Rose, F. (1983). ‘Kinematics of head movements accompanying speech during conversation’. Human Movement Science, 2, pp. 35-46.

Hadar, U., Steiner, T.J. & Grant, E.C. (1984). ‘The timing of shifts of head postures during conversation’. Human Movement Science, 3, pp. 237-245.

Hadar, U., Steiner, T.J. & Clifford Rose, F. (1985). ‘Head movement during listening turns in conversation’. Journal of Nonverbal Behavior, 9, pp. 214-228.

Hay, J. (2001). ‘The pragmatics of humor support’. Humor: International Journal of Humor Research, 14(1), pp. 55-82.

Heylen, D. (2008). ‘Listening heads’, in Wachsmuth, I. & Knoblich, G. (eds.), Modeling communication. Berlin/Heidelberg: Springer-Verlag,

pp. 241-259.

Hoskin, J. (director), Colbert, S., Spyra, J., Stack, B. & Dinello, P. (writers). (2015-). The Late Show with Stephen Colbert [Television show]. Retrieved from

Holt, E. & Glenn, P.J. (2013). Studies of laughter in interaction. London: Bloomsbury Academy.

Ikeda, K. & Bysouth, D. (2013). ‘Laughter and turn-taking: Warranting next speakership in multiparty interactions’, in Holt, E. & Glenn, P. (eds.), Studies of laughter in interaction. London: Bloombsbury Academic, pp. 36-64.

Ishi, C. T., Ishiguro, H. & Hagita, N. (2014). ‘Analysis of relationship between head motion events and speech in dialogue conversations’. Speech Communication, 57, pp. 233-243.

Jefferson, G., Sacks, S. & Schegloff E. A. (1987). ‘Notes on Laughter in the Pursuit of Intimacy’, in Button, G. & Lee, J.R.E. (eds.), Talk and social Organisation. Clevedon: Multilingual Matters, pp.152-205.

Kendon, A. (1980). ‘Gesticulation and speech: two aspects of the process of utterance’, in R. Key M. (ed.), The Relationship of Verbal and Nonverbal Communication. The Hague: Mouton and Co, pp. 207-227.

Kendon, A. (2002). ‘Some uses of the headshake’. Gesture 2(2), pp. 147-182.

Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge, England: Cambridge University Press.

Kim, J., Cvejic, E. & Davis, C. (2014). ‘Tracking eyebrows and head gestures associated with spoken prosody’. Speech Communication, 57, pp. 317-330.

Kousidis, S., Malisz, Z., Wagner, P. & Schlangen, D. (2013). ‘Exploring annotation of head gesture forms in spontaneous human interaction’ in TiGeR 2013, Tilburg Gesture Research Meeting.

Krahmer, E. & Swerts, M. (2007). ‘The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception’. Journal of Memory and Language, 57(3), pp. 396-414.

Kress, G. & van Leeuwen, T. (2001). Multimodal Discourse: The modes and media of contemporary communication. London: Arnold. New York: Oxford University Press.

Lee, J. & Marsella, S. (2006). ‘Nonverbal behavior generator for embodied conversational agents’. Proceedings of Sixth International Conference on Intelligent Virtual Agents (IVA), Marine del Rey, USA, August 21–23, pp. 243–255.

Lee, J. & Marsella, S. (2010). ‘Predicting speaker head nods and the effects of affective information’. IEEE Transactions on Multimedia, 12(6), pp. 552-562.

Maynard, S. K. (1987). ‘Interactional functions of a nonverbal sign: Head movement in Japanese dyadic casual conversation’. Journal of Pragmatics, 11(5), pp. 589-606.

McClave, E. (2000). ‘Linguistic functions of head movements in the context of speech’. Journal of Pragmatics, 32, pp. 855-878.

McNeill, D. (1992). Hand and mind: what gestures reveal about thought. Chicago/London: The University of Chicago Press.

Mondada, L. (2013). ‘Conversation analysis: Talk and bodily resources for the organisation of social interaction’, in Müller, C.,

Cienki, A., Ladewig, S., McNeill, D. & Teßendorf, S. (eds.), Body–Language–Communication: An International Handbook on Multimodality in Human Interaction (Vol. 1). Berlin: de Gruyter Mouton, pp. 228-226.

Morreal, J. (1983). Taking Laughter Seriously. New York: State University of New York Press.

Norrick, N. (1993). Conversational joking. Bloomington: Indiana University Press.

Norris, S. (2013). ‘Multimodal (Inter)action Analysis: An Integrative methodology’, in Müller, C., Cienki, A., Ladewig, S., McNeill, D. & Teßendorf, S. (eds.), Body–Language–Communication: An International Handbook on Multimodality in Human Interaction (Vol. 1). Berlin: de Gruyter Mouton, pp. 275-286.

Parrill, F. (2012). ‘Interactions between discourse status and viewpoint in co-speech gesture’, in Dancygier, B. & Sweetser E. (eds.). Viewpoint in Language: A Multimodal Perspective. Cambridge: Cambridge University Press, pp. 97-112. doi:10.1017/CBO9781139084727.008

Pelachaud, C., Badler, N. & Steedman, M. (1996). ‘Generating facial expressions for speech’. Cognitive Science, 20, pp. 1-46.

Poggi, I. & Pelachaud, C. (1998). ‘Performative faces’. Speech Communication, 26, pp. 5-21.

Poggi, I., D’Errico, F. & Vincze, L. (2010). ‘Types of Nods. The Polysemy of a Social Signal’- LREC. Retrieved August, 15 2018 from

Provine, R. (2000). Laughter. A scientific Investigation. London/New York: Penguin Random House

Smoski, M.J. & Bachorowski, J.A. (2003). ‘Antiphonal laughter

between friends and strangers’. Cognition & Emotion, 17, pp. 327-340

Swerts, M. & Krahmer, E. (2008). ‘Facial expression and prosodic prominence: Effects of modality and facial area’. Journal of Phonetics, 36(2), pp. 19-238.

Swerts, M. & Krahmer, E. (2010). ‘Visual prosody of newsreaders: Effects of information structure, emotional content and intended audience on facial expressions’. Journal of Phonetics, 38, pp. 197-206.

Tabacaru, S. (2014). Humorous implications and meanings: a multi-modal approach to sarcasm in interactional humor. Université Charles de Gaulle - Lille 3, Doctoral dissertation. Retrieved August 15, 2018 from

Tabacaru, S. & Lemmens, M. (2014). ‘Raised eyebrows as gestural triggers in humour: The case of sarcasm and hyper-understanding’. The European Journal of Humour Research, 2(2), pp. 11-31.

Veale, T., Feyaerts, K. & Brône, G. (2006). ‘The cognitive mechanisms of adversarial humor’. Humor: International Journal of Humor Research, 19(3), pp. 305-338. doi: 10.1515/HUMOR.2006.016

Wagner, P., Malisz, Z. & Kopp, S. (2014). ‘Gesture and speech in interaction: An overview’. Speech Communication, 57, pp. 209-232.

Weeks, M. (2016). ‘The enigma of solitary laughter’. The European Journal of Humour Research, 4(3), pp. 76-87.

All authors agree to an Attribution Non-Commercial Non Derivative Creative Commons License on their work.


Download data is not yet available.