Trouble is brewing for coffee lovers in California, where a judge ruled that sellers must post scary warnings about cancer risks. But how frightened should we be of a daily cup of joe? Not very, some scientists and available evidence seem to suggest.

Scientific concerns about coffee have eased in recent years, and many studies even suggest it can help health.

"At the minimum, coffee is neutral. If anything, there is fairly good evidence of the benefit of coffee on cancer," said Dr. Edward Giovannucci, a nutrition expert at the Harvard School of Public Health.

Get Breaking News Delivered to Your Inbox

CBS News medical contributor Dr. David Agus, director of the Westside Cancer Center at USC, says he believes it is too early to put this kind of blanket warning on coffee.

"When you put a bold declaration that 'X may cause cancer' when there isn't data to that effect in humans, to me it causes panic rather than informed knowledge," he told "CBS This Morning."

The World Health Organization's cancer agency moved coffee off the "possible carcinogen" list two years ago, though it says evidence is insufficient to rule out any possible role.

The current flap isn't about coffee itself, but a chemical called acrylamide that's made when the beans are roasted. Government agencies call it a probable or likely carcinogen, based on animal research, and a group sued to require coffee sellers to warn of that under a California law passed by voters in 1986.

The problem: No one knows what levels are safe or risky for people. The U.S. Environmental Protection Agency sets acrylamide limits for drinking water, but there aren't any for food.

"A cup of coffee a day, exposure probably is not that high," and probably should not change your habit, said Dr. Bruce Y. Lee of Johns Hopkins Bloomberg School of Public Health. "If you drink a lot of cups a day, this is one of the reasons you might consider cutting that down."

Here's what's known about the risks.

The chemical

Start with the biggest known risk factor for cancer - smoking - which generates acrylamide . In the diet, French fries, potato chips, crackers, cookies, cereal and other high-carbohydrate foods contain it as a byproduct of roasting, baking, toasting or frying.

Food and Drug Administration tests of acrylamide levels found they ranged from 175 to 351 parts per billion (a measure of concentration for a contaminant) for six brands of coffee tested; the highest was for one type of decaf coffee crystals. By comparison, French fries at one fast food chain ranged from 117 to 313 parts per billion, depending on the location tested. Some commercial fries had more than 1,000.

Even some baby foods contain acrylamide, such as teething biscuits and crackers. One brand of organic sweet potatoes tested as having 121 parts per billion.

What's the risk?

The "probable" or "likely" carcinogen label is based on studies of animals given high levels of acrylamide in drinking water. But people and rodents absorb the chemical at different rates and metabolize it differently, so its relevance to human health is unknown.

A group of 23 scientists convened by the WHO's cancer agency in 2016 looked at coffee - not acrylamide directly - and decided coffee was unlikely to cause breast, prostate or pancreatic cancer, and that it seemed to lower the risks for liver and uterine cancers. Evidence was inadequate to determine its effect on dozens of other cancer types.

The California law

Since 1986, businesses have been required to post warnings about chemicals known to cause cancer or other health risks - more than 900 substances are on the state's list today - but what's a "significant" risk is arguable.

Coffee sellers and other defendants in the lawsuit that spurred Thursday's ruling have a couple weeks to challenge it or appeal.

The law "has potential to do much more harm than good to public health," by confusing people into thinking risks from something like coffee are similar to those from smoking, Giovannucci said.

The International Food Information Council and Foundation, an organization funded mostly by the food and beverage industry, says the law is confusing the public because it doesn't note levels of risk, and adds that U.S. dietary guidelines say up to five cups of coffee a day can be part of a healthy diet.

Dr. Otis Brawley, the American Cancer Society's chief medical officer, said, "The issue here is dose, and the amount of acrylamide that would be included in coffee, which is really very small, compared to the amount from smoking tobacco. I don't think we should be worried about a cup of coffee."

Amy Trenton-Dietz, public health specialist at the University of Wisconsin-Madison, said the California ruling contrasts with what science shows.

"Studies in humans suggest that if anything, coffee is protective for some types of cancer," she said. "As long as people are not putting a lot of sugar or sweeteners in, coffee, tea and water are the best things for people to be drinking."