No one can pay attention to everything they encounter. We simply do not have enough time or mental capacity for it. Most of us, though, do make an effort to acknowledge our fellow humans. Wealth, it seems, might change that.

There’s a growing body of research showing how having money changes the way people see—or are oblivious to—others and their problems. The latest is a paper published in the journal Psychological Science in which psychologists at New York University show that wealthy people unconsciously pay less attention to passersby on the street.

In the paper, the researchers describe experiments they conducted to measure the effects of social class on what’s called the “motivational relevance” of other human beings. According to some schools of psychological thought, we’re motivated to pay attention to something when we assign more value to it, whether because it threatens us or offers the potential for some kind of reward.

The NYU team had a group of 61 study participants walk down a city block in Manhattan wearing Google Glass. The pedestrians, who were told they were testing the technology, later filled out surveys asking them to self-identify their social class. Analyzing the Google Glass recordings, the researchers found that those who had self-identified as wealthy didn’t rest their eyes on their fellow humans for as long as those who said they were from lower social classes.

The researchers conducted a pair of similar follow-up studies using an advanced eye-tracking system. This time, students recruited for the study viewed a series of photographs taken from Google Street View on a computer screen, then answered the same survey about social class. Again, the researchers found that students self-identifying as wealthier spent less time looking at people.

In a separate experiment, the NYU researchers tested whether the difference in the amount of time a participant dwelled on a person was the consequence of a conscious decision or a spontaneous cognitive reaction. They recruited nearly 400 participants for an online study and had them look at alternating pairs of pictures, each of which contained an array of various items, always including one face and five objects (like fruit, an appliance, or an article of clothing.) One picture would appear briefly on the screen, and then be replaced by a second picture that was either identical or nearly identical to the first. The two images would keep flickering this way until the participant hit the spacebar to indicate they had detected a change in one of the objects, or the face, in the photo, or that they had decided there had been no change.

People self-identifying as less wealthy were significantly faster than those of a higher social class at noticing change in faces in the photos, a sign, the researchers say, that faces held higher motivational relevance for them.

“Across field, lab, and online studies, our research documents that other humans are more likely to capture the attention of lower-class individuals than the attention of higher-class individuals,” Pia Dietze, a PhD student at New York University and lead author of the study, said in a press release. And the response is pervasive and spontaneous, she added.

Past studies have investigated the myriad ways the rich interact differently with their community, and the results have not been pretty.

For instance, in a series of studies published in 2012, psychologists from University of California, Berkeley, had college students watch two videos—one of a man explaining how to build a patio, and another depicting the lives of children with cancer—and found the wealthier participants were less likely to report feeling compassion for the children and their families in the second video. (The researchers controlled for factors like ethnicity, spiritual beliefs and gender, all of which also influence compassion.) As they watched the videos, all of the participants also wore heart monitors, because research has shown that our heart rate will slow down when we’re tuning into the feelings of another person. This reaction was noted in the less wealthy participants as they watched the second film, but not the wealthier subjects.

An earlier study published in Psychological Sciences, and led by a University of California, San Francisco psychologist, found that people of a higher socioeconomic status are not as adept at reading other people’s emotions accurately, compared to less affluent peers. What’s more, in a 2009 study, college students of a higher socioeconomic status tended to pay less attention to a stranger with whom they were paired to speak for few minutes, even if the conversation partner was equally affluent. The wealthy, psychologists believe, pay less attention to everyone, regardless of status, which may affect their relationships with friends and family.

One reason the rich may be less likely to value others is because they can afford to hire help to serve their needs (like child care and home repairs) rather than depend on a neighbor, according to Dacher Keltner, a professor of psychology at University of California, Berkeley. Writing in the New York Times, psychologist Daniel Goleman explained that Keltner’s and other social psychologists’ work shows that “financial difference ends up creating a behavioral difference. Poor people are better attuned to interpersonal relations…than the rich are, because they have to be.”

This interplay of power, money, and empathy becomes particularly troubling in contemporary economies marked by growing inequality. Goleman and others argue if those who earn more and therefore hold more power do not see (figuratively and literally) those who have less, reversing financial disparity becomes unlikely. As Goleman points out, “Reducing the economic gap may be impossible without also addressing the gap in empathy.”

One percenters of the world may not be terribly concerned about societal income gaps, but they should care about a significant disadvantage to having a bigger stockpile of cash than everyone else: a diminished ability to experience the benefits of strong interpersonal relationships, which may be the most rewarding part of the human experience—even the secret to happiness, according to a 50-year study from Harvard. Humans are built to thrive in a community, and without it we are at increased risk of loneliness, which is harmful to one’s health, and can play a role in heart disease, depression, and even premature death. Privilege comes at a cost.