Those costs are so high that they "may tilt the balance to the point where screening appears relatively cost-ineffective," the authors wrote in the conclusion of the report, which looked at expenditure data from more than 702,000 women with health insurance.

False-positive mammograms and overdiagnosis of breast cancer among women ages 40 to 59 cost $4 billion in health-care spending annually, the study , published Monday by the journal Health Affairs, found.

The costs from false diagnoses of breast cancer are "much higher" than previously documented, according to a new study that is sure to add fuel to the often emotional debate over whether women should routinely receive mammograms.

The findings come amid continued debate over the wisdom of routine screening for breast cancer.

The American Cancer Society continues to recommend annual screenings for women beginning at age 40; but a 2009 recommendation by the U.S. Preventive Services Task Force called for screening every two years in women ages 50 to 74.

That task force, composed of a volunteer panel of experts that generates an annual report for Congress, said that for women younger than 50, "The decision to have screening mammography for breast cancer should be an individual one and should take into account a woman's own situation and her values regarding specific benefits and harms."

The task force is in the process of updating its recommendations, which have not changed since 2009.

Breast cancer is the second-leading cause of cancer deaths among American women, and mammograms offer the opportunity to catch the disease in its early stages. More than 62 percent of women ages 40 to 49 get mammogram screens annually, and more than 72 percent of women between the ages of 50 and 59 get such screenings, according to the National Center for Health Statistics.

Read MoreObamacare: You (still) can't fix stupid



But the rate of false positives for mammograms is relatively high, and the use of mammography "is associated with the harm of overtreatment caused by false-positive mammograms and breast cancer diagnosis," the Health Affairs report noted.

Those include exposing patients to "additional diagnostic workup and psychological distress," the authors wrote. The cumulative probability of a woman being asked to undergo additional testing after getting a false-positive mammogram over the course of 10 years of screening has been estimated at 61 percent, the report found.

And breast cancer overdiagnosis is estimated to be between 22 and 31 percent of all diagnosed breast cancers, according to the study.