Economists aren’t holding out hope for another baby boom anytime soon.

The fertility rate was rising before the economic downturn in 2008, but has been falling since and shows few signs of picking up, a new report by the Federal Reserve Bank of Cleveland concluded. The U.S. fertility rate rose from 55 births per 1,000 women of childbearing age to 58 births per 1,000 women between 2006 and 2008, but fell to 52 births per 1,000 women in 2013, according to the report, which cites the American Community Survey from the U.S. Census. “The declines in fertility rates since 2008 have been remarkable in their extent and persistence,” the report concluded.

A closer look at the numbers shows that women higher up on the career ladder have fewer kids: 15.3% of women aged 40 to 44 were childless last year, up from 9.2% in 2008, recent Census Bureau data found. The percentage of childless women aged 35 to 39 was 18.5% last year, up from 9.9% in 2008. It was higher for those women aged 30 to 34 (28.9% were childless last year versus 26.8% in 2008). “Women aged 40 to 50 in 2014 who were in managerial or professional occupations were more likely to be childless than women of similar age in other occupations,” it found.

Why the decline? Many families require two paychecks, says Fran Walfish, a psychotherapist in Beverly Hills, Calif., and must postpone buying a house — and starting a family — while they save for a down payment. Other women are delaying having a family to focus on their career and education and the window of fertility gets smaller, adds Gretchen Livingston, a senior researcher at Pew Research Center. Childcare remains the biggest household expense for parents in many regions, according to Child Care Aware of America, a nonprofit group in Arlington, Va.

Read: 6 of the best countries for new moms

Here is how the U.S. fertility rate is affecting women of different backgrounds:

“Over the last 35 years, America’s fertility rate has reflected, to some extent, the business cycle,” the Federal Reserve stated. However, this decline in the fertility rate “has been widespread across demographic groups and metro areas.” Between 2008 and 2013, the fertility rate of women with college degrees fell from 64 to 57 babies per 1,000 women and the fertility rate of women without college degrees fell from 57 to 50 babies per 1,000 women. What’s more, women who never earn a college degree average more children over their lifetime versus women who complete college.

For a population to replace itself, the fertility rate must exceed approximately 60 babies per 1,000 women a year. The fertility of native-born American women has been below the replacement level since the 1970s. The U.S. population has actually risen in the last four decades mainly because of immigration and relatively high rates of fertility among immigrants. But the fertility of immigrants also declined steeply following the recession, from 76 babies per 1,000 women in 2008 to 62 in 2013. The gap between the fertility rates for immigrants and U.S.-born women has narrowed from 21 to 12.

Metro areas to the left of this chart had the steepest house price declines from the peak of the property boom to the lowest point of the housing bubble, but couples in these regions did not move into foreclosed or heavily discounted homes and have larger families. Why? “Regions with steep house price declines also had large economic contractions and losses of employment,” the report found. Families with a higher demand for children migrate to places such as Texas and Georgia, where unconstrained construction makes larger homes less expensive, it added.

There is only a mild positive relationship between employment growth and changes in fertility, the report concluded. “Assuming families defer having children during times of hardship and uncertainty, the economic challenges related to weak labor demand appear to have overwhelmed the advantage of more affordable housing,” it said. For couples who want more children, but can’t afford it, there are some positive signs on the horizon: Although the fertility rate would need a further 11% increase to return to pre-recession levels, the report said, it may be beginning to rise from its recent low.

Also see: 10 things your kids won’t tell you