We’re surrounded by information about the health and nutritional benefits of different food, but a lot of it conflicts—and it’s leaving people more confused than ever about how to make healthy food choices. Should we eat all organic? Does our food need to be natural, and fresh? One recent fad is to avoid genetically modified food.

GM food has negative connotations for many consumers because of general mistrust of the food production industry, but also because anti-biotech activists have been so effective at stoking concerns. It’s led to an sharp increase in non-GMO labels, even on products like salt, which can’t be genetically modified because sodium chloride is an inorganic compound that doesn’t contain genes.

But non-GMO labels do more than placate people concerned about scientists secretly tinkering with their food. They might persuade people to make a poor food choice. That’s because genetically modifying food can actually make it safer, by limiting the need for, say, pesticides. According to Pam Ronald, who studies genetics at the University of California, Davis and whose husband is an organic farmer, farms going non-GMO to meet consumer demand are causing major damage.

“These non-GMO labels have proliferated, and they’re really a problem,” Ronald told Quartz. “Because there’s no regulation, they can just spray anything they want. So what’s happening is… they’re going back to using [far] more toxic compounds. And I think that’s really a disservice to the consumer to market it as somehow being more healthy—when of course, it’s not, and it’s also more harmful to the environment.”

(A representative from the non-GMO Project was not available for an interview.)

Watch the video above for more on how misleading labels confuse consumers, and some expert advice on how to actually make healthier choices. (Hint: it’s not choosing non-GMO.)