When I first heard that a tsunami had damaged Japan’s Fukushima nuclear reactor five years ago—Which was now melting down? Flooding the ocean with god knows how much radioactive material? We didn’t know at first—I did something that I’m embarrassed about now.

I ordered a bottle of potassium iodide.

The idea is that when the radioactive iodine-131 makes its way across the ocean and rains down its cancer dust, you stuff your thyroid with as much regular iodine as it can possibly hold, blocking the bad iodine from getting in and thus potentially protecting you from getting thyroid cancer. But like many decisions made by anxious people in unwarranted freakout situations, I a) way overestimated the potential for danger and b) totally failed to calculate the dosages. The CDC says you need daily doses of about 130 milligrams of potassium iodide in the event of a nuclear emergency. Not only did I not need KI, my dumb drops had 150 micrograms of the stuff per dose. Off by a factor of about 1,000.1 Useless.

Now, some people actually might have had to worry about radioactive iodine being sucked up into their thyroids: the families (especially kids) living near the Fukushima Daiichi plant. And indeed, kids in the region were screened for thyroid cancer in the years following the disaster. A piece in Science last week walks through the history of this screening, and the lessons it offers are instructive—for any human being who ever requires medical care.

On its face, as Dennis Normile describes, the initial finding from screenings in Japan was super alarming. Almost half (half!) of those screened had nodules or cysts (which can potentially be or become cancerous) on their thyroids.

Nuts, right? And a Japanese epidemiologist named Toshihide Tsuda published a paper in 2015 saying that the rate of thyroid cancer in those Fukushima kids was more than 600 per million—way higher than the 1 to 3 cases per million kids that you would expect. But! As Normile writes, that comparison wasn’t quite fair. The Fukushima survey used advanced ultrasound devices that can detect tiny growths, while the older data came from plain old clinical exams. Oops. You have an apples to oranges thing going on there, in terms of your diagnostic instruments.

Indeed, when other scientists screened kids elsewhere in Japan using the fancy ultrasound devices, rates of cancer were anywhere from 300 to 1,300 per million. What the ultrasound devices find, then, is a whole lot of turtles.

Turtles, All the Way Down

Turtles are part of what H. Gilbert Welch, a professor of medicine at the Dartmouth Institute for Health Policy and Clinical Practice, calls the barnyard pen of cancers. The barnyard has three animals, turtles, birds, and rabbits. “The goal of early detection is to fence them in,” he says.

You can’t fence in the birds. They’re the super aggressive lethal cancers that are beyond cure. The rabbits, you can maybe do something about if you can spot them and treat them. (Treatments that, by the way says Welch, have gotten better and better.) “But for the turtles,” he says, “you don’t need fences because they’re not going anywhere anyway. And the thyroid is full of turtles.”

The breast and the prostate are full of turtles too, and just as the thyroid-scanning ultrasound devices are more likely to find little nodules there, an upsurge in mammography has led to a corresponding upsurge in something called ductal carcinoma in situ. Basically cancers that most of the time would just sit around and do nothing if you left well enough alone. In other words, they’re indolent (great word), not malignant. Whether or not we treat them (or even look for them) has been a matter of great debate in recent years.

It's very hard to know when upticks and outbreaks are quite what they appear. Even infectious disease outbreaks can sometimes be attributed to more-sensitive screening methods. The rise in whooping cough cases has multiple causes of course, but one of them is improved screening methods. Gene-based tests called PCR assays can inflate the number of actual diagnoses, according to a piece by epidemiologist James Cherry in the New England Journal of Medicine.

Which is to say, if you start looking for something carefully, and if you use better methods to see what you’re looking for, you will often find it.

Welch points to an example from the '70s, when some employees at Lawrence Livermore National Labs (who deal with nuclear stuff all day long) were diagnosed with melanoma. Cancer cluster! But no—it was something else.

What happened, Welch writes in his book Should I Be Tested for Cancer, is that one person probably got sick. Then, other people in the lab started getting checked for moles. Some were funky, so that leads to biopsies, which leads to, in some cases, an actual diagnosis. Then people start really getting worried. The lab kicks off an awareness campaign, so more people go in for checks, leading to more biopsies. “The whole epidemic looked subsequently like it was a pseudoepidemic,” he says. “It was an epidemic of diagnosis.” The melanomas were mostly turtles.

In South Korea, checking more thoroughly has absolutely led to more diagnoses of thyroid cancer. In the late ‘90s, doctors in South Korea started screening people for thyroid cancer (it was an add-on test to the national cancer screening program), and cancer cases took off. “There was a 15-fold increase,” says Welch. “There was nothing like it in the world!” Now, he says, thyroid cancer is the most common cancer in Korea—more common than breast, and colon, and lung.

Here’s the really pernicious part. People get checked for thyroid cancer, doctor finds a little nodule, does a biopsy, there’s some cancery stuff in there, so they remove the thyroid, and the person—saints be praised!—the person lives. (Because of course they lived, they just had a little thing that would never have been a problem in the first place.) They live (without their thyroid) and now they are a survivor and the survival rates for thyroid cancer in South Korea are now really high. Great, right? No. “Once you understand the problem of turtles, you understand you’re giving credit to finding the cancers that don’t matter,” says Welch.

How did South Korea combat this surge in cancer cases? A group of doctors (including Welch) wrote a letter in 2014 discouraging screening with ultrasonography. Poof. Thyroid operations dropped by 35 percent in a year. Because the best test “isn’t one that finds the most cancer,” he says. “The best test is one that finds the cancers that matter.”

So, will the kids who lived near the Fukishima plant suffer more thyroid cancer than their peers elsewhere? Well, yes. Probably. They are going to be screened more than most other kids, after all, and those screenings will turn up more thyroid cancer, just by virtue of the fact that people are looking for it. And even though a paper in the Journal of the American College of Radiology recently suggested that thyroid nodules below a certain size should be left alone, clinicians might not pay attention. Yes, it would be comforting to think that doctors and patients could resist the temptation to look for problems … or in some cases (when the data supports it) to ignore or merely watch what they find.

But sometimes people just freak out and order stuff on the Internet.

1UPDATE 13:00 ET 3/9/2016: There are 1,000 micrograms in a milligram, not 100.