It may not pay to be nice in the workplace.

A new study finds that agreeable workers earn significantly lower incomes than less agreeable ones. The gap is especially wide for men.

The researchers examined "agreeableness" using self-reported survey data and found that men who measured below average on agreeableness earned about 18% more—or $9,772 more annually in their sample—than nicer guys. Ruder women, meanwhile, earned about 5% or $1,828 more than their agreeable counterparts.

"Nice guys are getting the shaft," says study co-author Beth A. Livingston, an assistant professor of human resource studies at Cornell University's School of Industrial and Labor Relations.

The study "Do Nice Guys—and Gals—Really Finish Last?" by Dr. Livingston, Timothy A. Judge of the University of Notre Dame and Charlice Hurst of the University of Western Ontario, is to be presented on Monday in San Antonio, Texas, at the annual meeting of the Academy of Management, a professional organization for management scholars. The study is also forthcoming in the Journal of Personality and Social Psychology.

The researchers analyzed data collected over nearly 20 years from three different surveys, which sampled roughly 10,000 workers comprising a wide range of professions, salaries and ages. (The three surveys measured the notion of "agreeableness" in different ways.) They also conducted a separate study of 460 business students who were asked to act as human-resource managers for a fictional company and presented with short descriptions for candidates for a consultant position. Men who were described as highly agreeable were less likely to get the job.

For men being agreeable may not conform "to expectations of 'masculine behavior,'" the researchers write in the study. People who are more agreeable may also be less willing to assert themselves in salary negotiations, Dr. Livingston adds.

Other research shows that rudeness may not always benefit employees or their firms. A paper presented earlier this month at the annual meeting of the American Psychological Association found that 86% of 289 workers at three Midwestern firms in the manufacturing and health-care industries reported incivility at work, including public reprimands and making demeaning comments. Incivility was bad for the organizations as a whole, though, increasing employee turnover, found the researchers, Jeannie Trudel, a business professor at Indiana Wesleyan University-Marion, and Thomas Reio, a professor at Florida International University.

"The problem is, many managers often don't realize they reward disagreeableness," says Dr. Livingston. "You can say this is what you value as a company, but your compensation system may not really reflect that, especially if you leave compensation decisions to individual managers."

Lockerz, a 65-person Seattle, Wash., social-commerce company, has what it calls a "no jerks and divas" policy that is stressed in its employee handbook and orientation, says Chief Executive and founder Kathy Savitt. She notes, though, that there is a difference between being respectful and being agreeable. "We are not about being 'nice' or 'agreeable' or 'civil,'" she says. "We have a lot of robust debates about all kinds of things. But we do stress the notion of being respectful."

Paul Purcell, chairman, president and chief executive of Robert W. Baird & Co., a Milwaukee financial-services firm, says that his 2,700-employee company "doesn't hire or tolerate jerks. That's frankly a large percentage of people in our business. They don't get through the interview process." The firm has fired at least 25 offenders of its "no-jerk" policy, he says.

Human-resources consulting firm Development Dimensions International, of Pittsburgh, offers courses in "Interaction Management," covering interpersonal skills such as teamwork, managing conflict and giving and receiving feedback. "They are very trainable skills," says Jim Davis, DDI's vice president of work force and service development, who says that its interaction-training business is up 20% so far this year.

Write to Rachel Emma Silverman at rachel.silverman@wsj.com