There's a lot of millennials, but a lot of them aren't having babies—at least not yet. Birth rates among American women ages 20 to 29 years old hit historic lows in the years right before and after the Great Recession, according to a new report that raises the possibility that a major shift in the ages when women tend to have kids is on the horizon. (Tweet this) The birth rates for women in their 20s saw a 15 percent drop from 2007 to 2012, the Urban Institute report released Tuesday found. The decrease contributed to falling birth rates for women overall, after more than three decades of relative stability.

The study pointed to the recession, a "dramatic decline in birth rates among unmarried" African-American and Hispanic women, and a decline in the number of white married women as major factors behind the overall birth bust among women in their 20s.

Hispanic women in the age group saw the biggest declines in birth rates—a 26 percent plunge. That was followed by a 14 percent decrease among African-American women and an 11 percent fall for white women.

Read MoreWellness program rules up for debate

"We calculate that in 2012, women in their twenties had births at a pace that would lead to 948 births per 1,000 women, by far the slowest pace of any generation of young women in U.S. history," the report said. "If these low birth rates to women in their twenties continue, the U.S. might eventually face the type of generational imbalance that currently characterizes Japan and some European countries, but it is too early to predict or worry about that eventuality."

"This is really quite big," said Nan Marie Astone, one of the report's authors. However, because the big plunge in the birthrates coincided with Great Recession and the following years, "it's hard to think that [the economic decline] wasn't the reason," Astone said. Still, while every previous major economic decline has also been followed by a decrease in the birthrate among young women, "it's not been this big" as the one identified by the new study, Astone said.

(Above: Births per year. Source: Urban Institute, Millennial Childbearing and the Recession)

The findings come in the same year that millennials—the generation that started becoming adults around the turn of the century—are projected to overtake the so-called baby boomers as the largest age group in the United States.

The Pew Research Center estimates that the total number of millennials—people now age 18 to 34—will be 75.3 million in 2015, compared with 74.9 million people between ages 51 to 69. Astone said that because there are a huge number of millennials, a short-term decrease in their birthrates among women in their 20s will not, by itself, mean there will be dramatically fewer Americans in coming decades. Read MoreTwo states' surprising Obamacare success

"Even if the birthrates are very, very low, where the population is very, very large, we're still talking about a lot of babies," she said. Also, women in their 20s who aren't having kids now could start having kids in their 30s, Astone said. That in itself would represent a dramatic demographic change.

Until the 1980s, for as long as records were kept, the highest fertility rates worldwide were among women ages 20 to 24, Astone said.

That shifted in the United States and Europe sometime in the 1980s, when women ages 25 to 29 became the most fertile, she said.

"For demographers, a small group of nerdy people, that was pretty momentous," she said.

Read MoreIs Obamacare getting (marginally) popular again?

"It was associated with women going to school for longer, or going into professional occupations," both of which led women to delay having children, Astone said.

If the birth rates among millennial women rebound by the time they reach their 30s, Astone said, the most fertile age group may become women age 30 to 34 for the first time in human history. (Tweet this) "That would be a big deal," she said.

