Abstract Emojis are ideograms that are becoming ubiquitous in digital communication. However, no research has yet investigated how humans process semantic and pragmatic content of emojis in real time. We investigated neural responses to irony-producing emojis, the question being whether emoji-generated irony is processed similarly to word-generated irony. Previous ERP studies have routinely found P600 effects to verbal irony. Our research sought to identify whether the same neural responses could also be elicited by emoji-induced irony. In three experiments, participants read sentences that ended in either a congruent, incongruent, or ironic (wink) emoji. Results across all three experiments demonstrated clear P600 effects, the amplitudes of which were correlated with participants’ tendency to treat the emoji as a marker of irony, as indicated by behavioral comprehension question responses. These ironic wink emojis also elicited a strong P200 effect, also found in studies of verbal irony processing. Moreover, unexpected emojis (both mismatch and ironic emoji) also elicited late frontal positivities, which have been implicated processing unpredicted words in context. These results are the first to identify how linguistically-relevant ideograms are processed in real-time at the neural level, and specifically draw parallels between the processing of word- and emoji-induced irony.

Citation: Weissman B, Tanner D (2018) A strong wink between verbal and emoji-based irony: How the brain processes ironic emojis during language comprehension. PLoS ONE 13(8): e0201727. https://doi.org/10.1371/journal.pone.0201727 Editor: Cheryl Mary Corcoran, Icahn School of Medicine at Mount Sinai, UNITED STATES Received: October 31, 2017; Accepted: July 20, 2018; Published: August 15, 2018 Copyright: © 2018 Weissman, Tanner. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: All data is available, hosted on Harvard Dataverse at doi:10.7910/DVN/Q0VR8S. Funding: This work was supported by the National Science Foundation, grant BCS-1431324 awarded to DT. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors have declared that no competing interests exist.

Introduction Emojis are graphical symbols easily inserted into Computer Mediated Communication that have recently exploded in use and popularity. Articles about emojis have been published in venues like The New York Times, The New Yorker, and Time magazine [1, 2, 3], and The Emoji Movie debuted in theaters in July 2017, albeit to horrible reviews. Popular media aside, a few scholarly investigations into emojis have shed light on their communicative properties and tendencies, including the ability to fulfill a wide range of linguistic functions. Broadly speaking, the use of emojis in conjunction with “traditional” written language can be seen as a type of multimodal interaction that combines more than one modality systematically into a single communicative utterance (e.g., [4, 5, 6, 7]). Emoticons, the predecessor to newer emojis, were originally regarded as simple markers of emotions (e.g., [8, 9]) but more recent approaches (e.g., [10, 11]) have posited that they can serve many complex pragmatic functions, a claim that is further bolstered by the recent explosion in number and usage frequency of emojis. Some recent research has investigated the ambiguity and potential for miscommunication that emojis carry, even when embedded in linguistic contexts [12, 13, 14]. One conventional–and less ambiguous–usage of a specific emoticon/emoji is the wink-face symbol used to indicate sarcasm or irony; though not the only usage of this symbol, the sarcasm/irony usage has been demonstrated with consistency in both intent and uptake (e.g., [15, 16, 17]). This usage is consistent with Dresner & Herring’s (2010) theory claiming that emoticons can be used to indicate illocutionary force–the speaker’s intended purpose in performing a given utterance [18, 19]. Nonetheless, despite the fact that emoji use is common, even ubiquitous, in current electronic communication, no work has yet investigated how emojis and their communicative functions are processed in language contexts in real time. Here we focus specifically on the processing of emojis used to signal irony/sarcasm and ask 1) how emoji-related irony is processed at the neural level using event-related brain potentials (ERPs), and 2) whether emoji-induced irony elicits brain responses that are qualitatively similar to word-induced irony seen in prior work. Several studies have used ERPs to study word-induced irony, typically by time-locking brain responses to discourse-final words that were either consistent with the prior discourse context or were unexpected in an ironic way [20, 21, 22, 23]. Across studies, results have been consistent, with ironic completions eliciting a biphasic response characterized by an enhanced P200 component followed by a P600. The P200 is typically interpreted as reflecting attention-related processes (e.g., [24, 25]) while the P600 has traditionally been associated with syntactic processing or syntactic integration (e.g., [26, 27]). However, more recent work has shown that the P600 is not a reflection of syntax specifically, but general integration difficulty, enhanced monitoring processes, or general reanalysis (e.g., [28, 29, 30]). Overall, this prior work has argued that irony is not immediately and directly accessed, as has been claimed by some (e.g., [31, 32, 33]), but instead provides support for theories claiming multi-stage processing of irony (e.g., [34, 35, 36]), with the reanalysis P600 taken as a marker of an increased and delayed processing load. The P600 elicited in these studies contrasts with another well-studied component associated with language comprehension, the N400. The N400 component is elicited by all meaningful stimuli, but its amplitude in language studies is well known to be modulated by a number of factors including word predictability, length, frequency, and semantic fit with a prior sentence or discourse context [37]. Extending this line of research to multimodal communication involving emojis, one might expect that, to the extent that emojis are meaningful stimuli, emojis conveying an unexpected meaning (e.g., mismatching in valence with a prior context) might elicit a larger N400 amplitudes than expected emojis. Additionally, a number of ERP studies have also reported a late frontal positivity (henceforth LFP) in response to unexpected words, which were unpredictable based on prior context [38, 39, 40, 41]. An additional prediction is that unexpected emojis might elicit this late frontal positivity in addition to or perhaps instead of the more posteriorly distributed P600 and centrally distributed N400. With theoretical work on the linguistic and pragmatic functions of emojis suggesting the symbols can have similar functions as words in indicating irony (e.g., [10, 15]), it remains unknown how emojis are processed in meaningful contexts in real time, and whether emojis signaling irony elicit neural responses similar to word-induced irony (that is, P200 + P600). Such an isomorphism would be consistent with the hypothesis that a common set of processes underlies the processing of irony signaled both traditionally and ideographically. Additionally, our experimental design allows for the study of emojis that mismatch with their prior sentence context, but which are not explicitly or conventionally ironic; we hypothesize that these will elicit enhanced N400 and/or LFP effects, as the meaning of these non-ironic (but semantically incongruent) emojis would be harder to access and integrate in the surrounding context or generate a still-plausible prediction error. Moreover, as variability and ambiguity have been demonstrated in emoji interpretation (e.g., [12, 13, 14]), we also expected across-participant interpretations of the emojis in this experiment to vary. To this extent we also investigated how individual differences in interpretation mapped on to brain responses elicited during real-time processing.

Experiment 2 In Experiment 1, we showed that irony-induced emojis elicited a P200-P600 complex in participants who used the emoji to override the literal meaning of the words in the sentence. However, a high proportion of participants did not show any evidence of ironic interpretation of the sentences associated with the wink emoji. This is reasonable, as the stimuli were inherently ambiguous: without any sort of background knowledge or additional context, the wink was the only cue to irony, and many participants may have adopted a default strategy of interpreting sentences literally instead of using the emojis to guide interpretation. In Experiment 2 we therefore sought to replicate and extend the primary effects elicited in Experiment 1 by alerting participants to the possible presence of irony (which we referred to as “sarcasm” with participants) in the experiment. This experiment used identical materials to Experiment 1; the only difference was the brief mention of sarcasm during the instructions for the task. Through this mention we intended to elicit a greater number of nonliteral interpretation of sentences. Method Participants. Thirty-eight monolingual English speakers participated in this experiment for either course credit or monetary compensation. All participants were right-handed and reported no history of brain trauma, neurological impairment, or psychoactive medication; in addition, none of the participants participated in Experiment 1. Three datasets were excluded for high trial rejection rate (> 20%), making a final dataset of 35 participants (24 female (self-reported gender); M = 20.49 years, range: 18–25). The same consent and compensation protocol was used as in Experiment 1. Materials. The same materials and lists were used as in Experiment 1. Procedure. The same procedure was used as in Experiment 1. The only difference was an additional instruction at the beginning of the experiment: participants were told, “This study is investigating sarcasm… Some of the emojis you see will be sarcastic.” Participants were not told specifically that wink emojis should be interpreted sarcastically; rather, what emojis and sentences they should treat as sarcastic was left entirely up to them. Recording and data analysis. The same recording and data analysis techniques were used as in Experiment 1. Results Behavioral results. As is evident from Fig 8, the additional instruction achieved the desired effect, as nonliteral response rate in irony condition rose from 21.3% in Experiment 1 to 55.0% in Experiment 2 (SD = 30.4), though there was still considerable between-subject variability in nonliteral response rate. Since winks were not specified as the marker of irony, the nonliteral response rate in mismatch condition rose as well, from 16.4% in Experiment 1 to 53.9% in Experiment 2 (SD = 28.9). Match condition stayed low, increasing slightly from 3.1% in Experiment 1 to 5.4% in Experiment 2 (SD = 9.1). PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Fig 8. Boxplot of nonliteral response rates in the three conditions in Experiment 2. Horizontal lines in each box depict the median; black circles depict the mean. Boxes extend to first and third quartiles; whiskers extend to 1.5 x interquartile range. https://doi.org/10.1371/journal.pone.0201727.g008 Grand Mean ERP results and discussion. There was a significant main effect of condition in the 175–250 ms time window, F(2, 68) = 14.47, MSE = 6.17, p < .001 at midline sites, F(2, 68) = 12.82, MSE = 20.98, p < .001 at lateral sites, as well as reliable interactions between condition and anteriority, F(6, 204) = 6.33, MSE = .68, p < .001 at midline sites, F(12, 408) = 4.10, MSE = 1.09, p < .001 at lateral sites. Follow up contrasts between the match and irony conditions showed a reliable main effect of condition, F(1, 34) = 17.91, MSE = 8.32, p < .001 at midline sites, F(1, 34) = 16.60, MSE = 27.43, p < .001 at lateral sites, and interaction between condition and anteriority, F(3, 102) = 9.43, MSE = 0.73, p < .001 at midline sites, F(6, 204) = 5.70, MSE = 1.31, p = .01 at lateral sites, as well as an interaction between condition and hemisphere over lateral sites, F(1, 34) = 5.96, MSE = 0.93, p = .02. This additional interaction with hemisphere indicates that the positivity had a slightly left hemisphere preponderance, though it should be noted that the P200 still showed a typical fronto-central preponderance. There were no significant effects for the match/mismatch contrast, Fs < 1. In the 450-750ms time window there was a significant main effect of condition, F(2, 68) = 8.37, MSE = 5.49, p < .001 at midline sites, F(2, 68) = 7.71, MSE = 17.17, p < .01 at lateral sites, and a significant condition by anteriority interaction, F(6, 204) = 4.98, MSE = 0.75, p < .001 at midline sites, F(12, 408) = 4.19, MSE = 1.24, p = .001 at lateral sites. Follow-up contrasts for the match/irony comparison showed a main effect of condition, F(1, 34) = 15.96, MSE = 5.48, p < .001 at midline sites, F(1, 34) = 19.27, MSE = 13.54, p < .001, and an interaction between condition and anteriority, F(3, 102) = 6.55, MSE = 0.76, p < .01 at midline sites. As can be seen in Figs 9 and 10, this interaction was driven by the presence of a small P600 effect at parietal electrode sites (e.g., CP1, CP2, P3, Pz, P4, where P600 effects are typically largest). The condition and anteriority interaction was significant as well, but this is due to as the onset of the late frontal positivity over more anterior electrode sites and we discuss this in the following paragraph. Notably the correlation between nonliteral response rate and P600 effect amplitude at Pz was significant, r = 0.39, p = .02, as in Experiment 1. PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Fig 9. Grand mean ERPs in Experiment 2 (n = 35) at 26 representative electrode sites. The vertical calibration bar indicates the temporal onset of the emoji stimulus and extends to indicate 2.5 μV of brain activity with negative voltage plotted up; 200 ms of prestimulus and 1000 ms of poststimulus brain activity are depicted. https://doi.org/10.1371/journal.pone.0201727.g009 PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Fig 10. Topographic plots depicting ERP effects from the irony minus match condition difference waves. The three time windows correspond to the P200 component (left), the P600 component (center), and late frontal positivity (right). Color scale (± 2μV) is held constant across scalp maps. https://doi.org/10.1371/journal.pone.0201727.g010 In the 600–900 ms time window, there was a significant main effect of condition F(2, 68) = 6.77, MSE = 5.41, p < .01 at midline sites, F(2, 68) = 7.97, MSE = 18.05, p < .001) at lateral sites, and interaction between condition and anteriority, F(6, 204) = 10.90, MSE = 0.85, p < .001 at midline sites, F(12, 408) = 8.44, MSE = 1.46, p < .001 at lateral sites. Pair-wise contrasts showed a significant effect of condition for the match/irony contrast, F(1, 34) = 11.99, MSE = 5.97, p = .001 at midline sites, F(1, 34) = 7.97, MSE = 18.05, p < .001 at lateral sites, and it showed a frontal distribution (condition by anteriority interaction: F(3, 102) = 22.02, MSE = 0.71, p < .001 at midline sites, F(6, 204) = 13.99, MSE = 1.32, p < .001 at lateral sites). The effects were qualitatively similar for the match/mismatch contrast (condition by anteriority interaction: F(3, 102) = 10.71, MSE = 0.99, p < .001 at midline sites, F(6, 204) = 10.45, p < .001 at lateral sites). Overall, Experiment 2 replicated and extended the findings from Experiment 1. By mentioning the possibility of sarcasm briefly during the instructions, a greater number of participants showed nonliteral interpretations of the sentences with emojis. As with both groups in Experiment 1, we found a P200 effect, and as in the nonliteral group in Experiment 1, this effect followed by a P600 effect in response to the ironic (wink) emoji in Experiment 2. Importantly, this effect was found not only in a restricted group of participants who allowed nonliteral responses, but indeed in the grand mean. Again, as in Experiment 1, P600 effect amplitudes over posterior sites were correlated with nonliteral response rates. Taken with the grand mean results, this shows that individual variability in nonliteral interpretations was still a significant driver of the P600 effect seen over posterior sites; by boosting the overall nonliteral response rate, our mention of sarcasm to participants in the instructions made this effect systematic enough to surface in the grand mean ERPs. Additionally, we replicated the late frontal positivity elicited by both types of unexpected emojis (mismatch and ironic). Note that this frontal positivity showed some temporal overlap with the posteriorly distributed P600, leading to the broadly distributed topography for the positivity in the 450–750 ms time window compared to the clear frontal distribution in the 600–900 ms time window (center and right scalp plots in Fig 10). Nonetheless, the fact that nonliteral response rates in the 450–750 ms time window were correlated significantly with effect amplitudes at Pz, as mentioned above, more so than at the frontal electrode site Fz (r = .28, p = .10), indicates that nonliteral sentence interpretations associated with emoji-related irony were indexed by this posteriorly distributed P600. Finally, it is important to note that no P200 or P600 effects were found in the mismatch condition, despite a similarly high overall nonliteral comprehension question response rate in this condition, and nonliteral response rates were not correlated with P600 effect amplitudes in this condition (r = .10, p = .58). However, there was still a reliable late frontal positivity in this condition, signaling that the mismatching emojis were nonetheless unexpected and unpredicted. Two non-mutually exclusive explanations for this lack of a P600 effect readily come to mind. First, ERP components like the P600 only index cognitive processes that are time- and phase-locked to the onset of the critical stimulus. It is possible that, given its broad, conventional use to signal irony/sarcasm [15, 16, 17], the reanalysis or reinterpretation processes giving rise to nonliteral interpretations in the mismatch condition are more variable in their time course than those involved in the irony condition. The high conventionality of the wink emoji to signal irony may result in a more consistent time-locking of reanalysis processes to the onset of the emoji stimulus. The second possibility has to do with the informativity of the emojis used within the context of the experiment. The wink emoji was only present in the irony condition, whereas the smiley and frowny faced emojis were counterbalanced across the match and mismatch conditions. Thus, within the experimental context, the wink was consistent in its relation with irony, whereas the smiley and frowny faced emojis were inconsistent: they could occur both in the match condition, or as mismatching (unexpected) stimuli. This direct mapping between the wink and irony and indirect mapping between the smile and frown and irony within the experimental context may be partly responsible for the pattern of effects both in this experiment and Experiment 1. Experiment 3 was designed to follow-up on this possibility.

General discussion Table 1 shows a collapsed summary of which effects were statistically significant for the different conditions across the different experiments. The relevant effects are strongest over midline sites so the results summarized below are those from the midline ANOVAs. PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Table 1. Summary of statistically significant effects across experiments. https://doi.org/10.1371/journal.pone.0201727.t001 Results from three ERP experiments revealed that irony delivered by emojis elicits the same brain response as irony delivered by words, as reported in prior work. Robust P600 effects, as have been found in prior ERP studies of verbal irony, were consistently elicited by the wink emoji in our study in participants, and this effect was correlated with participants’ eventual interpretation of the sentences across all three experiments: participants who showed higher nonliteral (ironic) response rates to sentences followed by a wink emoji showed larger posterior P600 effects. The fact that most participants in Experiment 1 interpreted wink emoji sentences literally points to the previously-discussed ambiguity surrounding emoji use and interpretation and, additionally, the importance of broader context (e.g., background knowledge, speaker-listener common ground, etc.) in shaping the interpretation of both emojis and irony. Even though interpretations varied significantly across participants, those who did treat the wink emoji as a marker of irony reliably showed the P600 irony response. Previous offline studies have demonstrated that the wink emoji can be interpreted as a marker of irony; our neurocognitive evidence suggests that this irony is processed in the same way as verbal irony. The P200 effects surfaced reliably to irony condition (the wink emoji) in every experiment. The P600 effect surfaced reliably to irony condition, but most strongly when the reader interpreted the sentence nonliterally. The LFP surfaced across experiments to both irony and mismatch conditions. The P200 and LFP effects were reliably frontal, the P600 reliably parietal. Recent conceptions of the P600 have highlighted its general role, above and beyond morphosyntactic processing, in the processing of unexpected information from a range of domains (orthography, musical sequences, morphosyntax, event knowledge, etc.), and in reanalysis and reprocessing. This link with reprocessing has recently been highlighted by work employing co-registration of ERPs and eye movements during reading, which showed that P600s were specifically linked to regressive eye movements, signaling reanalysis [30]. Our present work shows important, novel extensions of this reprocessing account in two ways. First, the present study shows that language-related P600 effects can be elicited by non-verbal ideograms that convey meaning. This signals that the P600 is not uniquely associated with word-related anomalies in the language domain but can be elicited by linguistically-relevant ideograms requiring higher-level cognitive processing for integration. Findings from ERP studies of “visual language”–narratives told through comic strips–have found similar P600 effects to both anomalies in narrative constituent structure and incongruities that surface in new panels [47, 48]. Our work provides an important extension to related work on multi-modal communication, and in particular, links semantic and pragmatic aspects of emoji use with the same neural mechanisms used in higher-order language processing. Second, correlation analyses showed that the presence and size of participants’ P600 effects is linked with the likelihood of nonliteral interpretations. This suggests that the P600 may specifically be tied to reanalysis of higher-level sentence meaning: when the literal meaning of a sentence is overridden in response to an ironic emoji, P600 effects ensue, such that the largest P600 effects are found in the participants who most frequently allowed nonliteral interpretations. This again mirrors findings from visual language studies, which found that more experienced comic strip readers demonstrated significantly larger P600 effects to the relevant anomalies [49]. These findings can be situated among other work on multimodal interactions and support theories including emojis and words as part of the same multimodal communicative system producing a single utterance as opposed to a sentence with an image added to it at the end [7]. This and other findings specifically on the P600 suggest this component is not specific to grammatical processing but rather a domain-general component reflecting reprocessing in structured, hierarchically organized systems (e.g., [47, 50, 51]). A reliable P200 effect was found across all groups in all three experiments to the irony condition wink emoji, but generally not to the mismatch condition. This includes Experiment 3, where the mismatch (smile) and irony (wink) emojis were equally informative. This frontal P200 effect has been reported in previous ERP studies of verbal irony (though not always discussed), but the large P200 effect amplitude and the lack of correlation with behavioral responses in our studies complicates the interpretation of this finding. One possibility is that this component indexes early recognition of ironic meaning, which can possibly be followed by later re-processing of the representation of the sentence meaning (P600). Another possibility is that the wink emoji is simply more stimulating to the visual or attention systems than the smile and frown emojis. The first possibility is supported by the presence of the effect in other ERP verbal irony studies; the second is supported by the fact that there was no correlation between P200 effect amplitudes and behavioral response rate. That mismatch emojis generated no enhanced P200 effect, including in Experiment 3 where it was equally as informative as the wink emoji within the experimental context, could be consistent with either possibility–the lack of effect in this condition could be due to either the mismatch condition’s status as generally unconventional use of the smile and frown emojis, or the smile and frown emojis as less visually arousing. Note that these two explanations are not mutually exclusive, and these two potential causes may be working in combination. Further work can tease apart these two possibilities. Emojis in the mismatch and irony conditions both showed enhanced late frontal positivities compared to expected match emojis. This ERP effect has been found in other language ERP studies in contexts where the time-locked word is plausible yet unexpected (e.g., “she pounded the nails with a book” [39], or specifically unpredicted word [38]; it has been hypothesized by some to reflect the processing cost associated with revising expectations and representations when specific lexical predictions are disconfirmed in the input [38]. While the wink emoji is conventionally used and recognized as a marker of irony, this is not the case with the smile or frown. The mismatch sentences and the emojis used therein have a variety of interpretations, such as communicating affect [52], that fall outside of the realm of irony processing. The fact that an enhanced LFP was found in both the mismatch and irony conditions consistently across all three experiments suggests that expectations for emojis may be qualitatively similar to those seen for predictable words, such that violations of specific expectations for both words and emojis elicit similar neural responses. Our work is foundational in that it is the first to investigate the real-time neural correlates of emoji processing in sentence context, and link processing of irony elicited by words to irony elicited by sentence-embedded ideograms. The P600 effect elicited by ironic (wink) emojis is isomorphic to that seen in other studies of word-induced irony, suggesting a common set of processes involved in the comprehension and interpretation of irony, even in multimodal interactions. The late frontal positivity elicited by unexpected mismatching and ironic emojis also suggests that prediction mechanisms for the semantic or pragmatic content of emojis may be qualitative similar to those seen for words in informative contexts. Additionally, the very recent rapid rise of digital communications using ideographic enhancements means that conventional uses for these icons are likely still evolving and may vary between user communities. The present study, however, constitutes an important first piece of evidence of online emoji processing in utterance comprehension.