It's one thing to keep an eye on workplace rivals. But another type of sabotage can be much harder to spot.

Shawna Biggars's saboteur would deliver dense, creamy slices of homemade carrot cake to her desk, "wanting affirmation that he was a great cook," she says. When she politely declined, he would press, saying, "You can't not have cake for the rest of your life," she says.

Finally, Ms. Biggars, a human-resources director in Wichita, Kan., "had to sit him down and say, 'If I were an alcoholic, you wouldn't say, 'Just take one drink.' " Over 2½ years, Ms. Biggars lost 120 pounds.

The biggest threat to your diet isn't in the refrigerator - it is sitting at the next desk. Nearly a third of people on weight-loss regimes cited colleagues who tease or pressure them to eat. Sue Shellenbarger has details on Lunch Break.

Some 29% of people on diets say colleagues pressure them to eat more, make fun of their diets or order them restaurant food they know isn't on their diets, according to a recent survey of 325 dieters by Survey Sampling International for Medi-Weightloss Clinics, a Tampa, Fla., franchiser of physician-supervised weight-loss clinics.

The approach can seem innocuous, but can result in weight gain over time. A colleague brings in home-baked cookies to celebrate a promotion, a birthday or to rally the team, and who wants to look like they don't appreciate the work of others if they decline?

Patrice Gibson, a sales representative for a medical-supply company, often sparred with her co-worker Michelle Nunemaker while they ate lunch at their desks. As Ms. Gibson laid out small, measured portions, Ms. Nunemaker "would make fun of what I was eating," Ms. Gibson says. She predicted failure, saying, "I know people who did that and the minute they went off it, they gained it all back," says Ms. Gibson, of Tampa, Fla. She let the criticisms "bounce off" her.

Ms. Nunemaker says she regarded Ms. Gibson's strictly controlled diet as unhealthy and prefers "exercise and watching what you're eating." While she and Ms. Gibson kept the conversation light and friendly, "I was saying serious things that I really believed," she says.

Illustrations by John S. Dykes

When Ms. Gibson shed 35 pounds, Ms. Nunemaker congratulated her. But "it didn't change my opinion," she says.

Peers' attitudes and behavior are linked to success in weight loss, according to a study published last month in the journal Obesity. Among 3,330 participants in a team-based weight-loss competition, including many teams of co-workers, those who reported having positive influence from teammates lost a larger percentage of their body weight than others.

"Social contacts can be extremely powerful," says Tricia Leahey, the study's lead author. While peers' encouragement helps, dieting failures or negative attitudes among colleagues can discourage people from sticking to their own weight-loss plans, says Dr. Leahey, an assistant professor of research on obesity at the Miriam Hospital in Providence, R.I. "It cuts both ways."

At the annual "chili and dessert cook-off" competition Ms. Biggars helps organize at her company, co-workers asked again and again, "Why aren't you eating?" she says. "It's a dieter's nightmare."

Deep down, some co-workers may feel abandoned by a dieter who no longer joins them for big lunches or happy hour, says Chelsey Millstone, corporate dietitian for Medi-Weightloss Clinics. Some feel jealous because they aren't losing weight. Or they see a trimmer colleague as a career threat, Ms. Millstone says. Consciously or not, these co-workers may pressure a dieter by "calling them out" with embarrassing personal questions or comments, she says.

Pushing back can cause some "very touchy" conversations, however, says Becky Hand, a registered dietitian with the weight-loss and fitness website SparkPeople, based in Cincinnati. Colleagues often think they're showing appreciation by bringing in food or building harmony in a department, and they can get annoyed or hurt. In her work as a hospital nutritionist, Ms. Hand coaches weight-loss patients to script a response in their heads and practice in front of the mirror, saying such lines as, "I've had your food in the past and it's always delicious. But I'm sorry, at this time in my life, eating those extra whatever isn't benefiting my health."

If co-workers still don't get it, Ms. Hand recommends taking them aside for a walk or a lunch to explain further and ask for their support.

Sarah Onstott, of Arvada, Colo., says she got tired of trying to avoid office snacks and potlucks at the call center where she works. After learning through networking with other dieters on SparkPeople.com to speak up and share healthy-eating tips, she and three co-workers organized a fruit-and-yogurt breakfast potluck. They also published a health newsletter, including recipes and a critique of fad diets, and distributed it to co-workers.

"People kind of grumbled" about the newsletter, she says. But they started adding fruit and vegetables to the potlucks.

Resisting is even harder when it means saying no to a client. More than 1 in 5 participants in the Medi-Weightloss Clinics survey say business lunches are where they feel the most pressure to overeat.

When Cezanne Macary, a hotel sales account director, eats out for work, clients often urge her to have rich dishes or desserts, saying, "Oh, come on, a little bit is not going to ruin your diet," says Ms. Macary, of Boston, who has lost 60 pounds and is determined to maintain it. "You have to be extremely disciplined" to resist.

Earlier this month, she found the perfect strategy at a client lunch: a flight-scheduling glitch. "Just as they were all ordering every dessert on the menu," she says, "I was leaving."

Diet Saboteurs

How do others interfere with your diet?

• Pressure me to eat foods not on my diet: 53%

• Cook and serve food not on my diet: 40%

• Make jokes about my diet: 35%

• Order me restaurant food that is not on my diet: 31%

Who makes you uncomfortable admitting you are on a diet?

• Friends: 28%

• Relatives: 21%

• Colleagues: 18%

• Spouse: 14%

• Boss: 5%

Why did you feel pressured to break your diet?

• Don't want to insult the host, boss, client or family member: 56%

• Want to eat like everyone else and be part of the crowd: 51%

• Don't want to call attention to my diet or weight: 41%

Source: January 2012 survey of 325 women who have dieted or are dieting by Survey Sampling International for Medi-Weightloss Clinics

Write to Sue Shellenbarger at sue.shellenbarger@wsj.com