You may think your swimmers are invincible, but what you eat, drink, and even how you exercise can harm your ability to make a baby.

“The assumption has been that men make so much sperm every day that it doesn’t matter what they do, but we’re beginning to realize that’s not the case,” says Alice Domar, M.D., assistant professor of OB/GYN and Reproductive Biology at Harvard Medical School. Even though you won’t be carrying the baby (well, not until it’s born anyway), your swimmers provide half the genetic code for your heir.

Here are the top sperm killers—avoid them and make these small moves for healthier (and more plentiful) sperm.

Sperm killer #1: Popping hair loss drugs and antidepressants

An active ingredient in Propecia, known as finasteride, has been found to reduce male hormone activity, reports the FDA. It, along with BPH drug Proscar (used to shrink enlarged prostates), are now required to carry a label that states, ‘Propecia and Proscar may cause male infertility and/or poor semen quality. This side effect lessens or goes away after men stop taking the drugs.’

Anti-depressants, including Paxil and Prozac, may also affect your fertility by potentially damaging DNA. Bottom line: Any man trying to make a baby who is on medication needs to ask a specialist about whether they’re safe to take when trying to conceive, says Dr. Domar.

Sperm killer #2: Frequent cycling sessions

The reason testicles hang low is to keep sperm at 96 degrees Fahrenheit (2.8 degrees cooler than you body). “Bicycling presses on them and raises them up into the body, which can cook the little guys,” Domar explains. The prolonged pressure from the bike seat may also potentially damage reproductive arteries and veins.

Other forms of exercise should be okay (moderate, regular workouts can actually boost testosterone)—as long as you don’t push yourself to your limit (excessive exercise actually throws hormone levels off). Also, make sure your boys are “keeping cool and hanging normally,” says Dr. Domar. If you are doing sports such as soccer or basketball, be sure to protect your boys with cups—but just not for an extended period of time.

Sperm killer #3: Hanging out in hot tubs

A soak in a hot tub can ease muscle soreness after a long workout—or put you in the mood for a baby-making session. Yet frequent dips in the hot tub (or hangouts in the steam room or sauna) can squash sperm production by overheating your boys, just as cycling can raise the temperature of your testicles.

Sperm killer #4: Overdosing on coffee

A moderate amount of caffeine may make sperm happy by helping them swim to their target faster, but too much may do more harm than good. Limit your consumption to no more than the amount in two to three cups of coffee a day—and that includes sports drinks and supplements. Additionally, an addiction to coffee may mask another, larger problem. “Anytime you use coffee daily to compensate for low energy levels, you have to look at rest of your life and figure out why you’re tired all the time,” says Dr. Domar.

Sperm killer #5: Adding more candles to your birthday cake

It turns out that guys have a biological clock, too: It’s not just women who have a harder time getting pregnant in their forties. “The age of a man has a very real impact on his ability to get a woman pregnant, as well as on her likelihood of carrying to term,” says Dr. Domar. The age of the father is also now linked to autism and schizophrenia. “Concern rises for guys over 40, and there’s a more significant impact with men over 50,” she says.

Sperm killer #6: Double fisting

“A daily glass of wine or cocktail may help keep heart disease at bay, but it’s bad for sperm,” says Dr. Domar, who recommends no more than five to seven drinks per week when you’re actively trying to conceive. It’s no surprise that drinking and smoking is twice as harmful to your swimmers. “If you look at the impact of alcohol and nicotine together, they can have a profound effect on sperm condition,” she says. That’s because nicotine has been shown to lower sperm count and quality, squashing your odds of making a baby.

Sperm killer #7: Feeling stressed

Men, too, can be affected by the stress of trying to have a baby. “For men, fertility is linked with virility and sexuality,” says Dr. Domar. Yet baby making can trigger anxiety when it starts to become an effort (think: ovulation tests and pressured sex sessions). Indeed, a recent survey sponsored by Merck found that even more men than women—42 percent versus 36 percent—said stress and tension in their relationship has increased while trying to have a baby. If this happens, try explaining to your partner that you’re just as excited as she is about taking this next big step and you want to give it your best team effort, but that stress can hurt your chances of conceiving. Then blow off steam by doing something together that you both enjoy. Once you’ve shared some fun, don’t be surprised if you end up right back in the bedroom.