You can usually tell you’re starting to bore someone because their eyes will shift away from you and they’ll no longer return your gaze. Aligning pupils with someone might seem like an improbable way to signal to each other, but eye contact is one of the most important forms of nonverbal communication. We use eye contact every day to indicate interest while we listen and speak. Human eyes, with their large unpigmented areas, turn out to be great for deducing where someone is looking. This likely makes it all the easier to track someone’s attention and notice when their interest wanes.

However, eye contact takes two, and in addition to the listener, the speaker also needs to maintain the gaze for eye contact to occur. A number of emotional states can cause someone to avoid eye contact while trying to communicate, such as shyness, embarrassment, or guilt. And you may have noticed that thinking deeply can cause someone to drop their eyes. A recent experiment conducted in Japan suggests that eye contact draws on the same mental resources used for complex tasks, so trying to maintain eye contact can impede your reasoning. In this case, the break in eye contact comes not from emotion, but from the need to preserve cognitive resources. Eye contact can deplete your mental bandwidth.

In the experiment, participants watched a screen displaying a person’s face while they performed a verbal task, having been instructed to keep staring at the eyes of the person on the screen. Sometimes the person depicted on the screen had their eyes directed at the viewer and sometimes their eyes looked to the side.

To complete each task, the participants needed to come up with a verb that could be used with a given noun. For example, if they heard the word milk, they could say drink. Importantly, this task could become more difficult in two different ways. First, a noun might work with many possible verbs (soup could be answered with eat, drink, cook). This would add the additional difficulty of selecting one answer. On the other hand, the task could become more difficult if a noun’s corresponding verbs were harder to remember, as with sky. These two kinds of difficulty made it easier for the researchers to toggle the challenge of the task up or down.

As expected, people took longer to answer when prompted with the more difficult nouns. However, direct eye contact with the person on the screen only hindered people’s performance when the prompts were doubly difficult. These were the nouns difficult in both dimensions, allowing several possible answers and bearing weak linguistic associations to their verbs. Only in these most difficult verbal tasks was there a measurable decrease in people’s performance while maintaining eye contact. Otherwise, keeping close eye contact with the face on the screen did not affect performance.

This finding adds another piece of evidence that eye contact makes people struggle to perform certain tasks. Previous studies established this phenomenon, but many of those studies tested visual thinking, like imagining a 3D landscape or naming the color of a word. In those cases, the interference with thinking could be explained simply because maintaining eye contact is also visual. By delivering the test aurally and having people answer verbally, the researchers in this study avoided that complication.

It is still not clear why a conflict occurs between eye contact and some types of thinking. The researchers suggest that only when the verbal association task is most challenging does a “general cognitive system” kick in to assist the verbal system. Eye contact might be managed by the general cognitive system, which then becomes depleted by the simultaneous demands. However, it’s difficult to speculate without tracking the brainware behind the tasks. Besides the cortical areas believed to be most important for complex tasks and cognition, eye contact also activates many emotional limbic areas like the amygdala.

Researchers still have much to understand about how culture affects people’s use of eye contact. While this study took place in Japanese-speaking adults, another experiment found no significant influence of eye contact on heartrate or eye contact time in Japanese and Finnish participants. Still, studies to date have focused on cultural differences in perceiving eye contact as positive or negative or how to act during eye contact, so less is known about culture and the effects of eye contact on thinking. If looking away to think is cross-cultural, then perhaps cultures with less emphasis on eye contact enable deeper thinking during a given conversation, while those using more eye contact might give better social feedback between conversational partners. In our everyday lives, do we talk more fluently about complicated subjects without eye contact, and if so, do we lose something in exchange?

Eye contact is something we prefer from birth, but it is not advantageous in every situation. Back in 1998, researchers theorized that averting the gaze from surroundings aids thinking by disengaging from potential distractions around us. While it remains to be seen which tasks are harmed (and which might be improved) with eye contact, complex verbal tasks seem to be more difficult for people trying to maintain the gaze of another person. So the next time you’re in a polite staring contest with an interviewer, take the time to look out the window while you ponder the hardest questions. They should forgive you the breach of etiquette if you come up with your best answers.