A coffee a day isn't what’s preventing you from retiring early. It’s the bigger budget-line items that are major barriers to a comfortable retirement, according to a new NerdWallet survey and analysis.

More than 2 in 5 Americans reported that housing and/or debt kept them from socking away for their golden years, the most-cited obstacles in their lives. Food and utility bills followed at 31%.

“The latte factor is complete B.S.” Douglas Boneparth, owner of Bone Fide Wealth, a financial planning firm, told Yahoo Finance. “Primarily because it’s the larger expenses that are going to move the meter.”

Still, 7 in 10 adults believe that the simple sacrifice of cutting out their lattes everyday would make a difference to their retirement savings, with a quarter saying it would make a major impact.

The NerdWallet analysis found that buying a slightly more affordable home would be four times more effective than skipping a cup of joe over 20 years, resulting in $137,000 of extra retirement savings.

Cutting small

Americans are more likely to focus on the smaller items to save extra nickels and dimes, the survey found. Almost half said they cook more at home instead of eating out or ordering delivery, while 85% of Americans say they use coupons to save money on purchases.

Americans said housing and debt were the biggest essential expenses that kept them from saving for retirement, according to a NerdWallet survey.

Almost 1 in 5 Americans found both of those tips annoying as well, according to the survey.

Overall when soaking in all the financial advice, it’s important to strike a balance between living your life and managing your budget.

“It’s not always about the number. A lot is to be said in finance about behavioral elements,” said Brandon Renfro, assistant professor of finance at East Texas Baptist University. “Be careful not to get in a position where your money is controlling your life.”

How to cut housing costs?

The survey reflects the rising costs for housing. Home prices have grown 45% in the last decade, while rent has increased approximately 40%. Meanwhile, wages have only risen by almost 23% in the last 10 years, according to the Bureau of Labor Statistics.

There are still some ways you can control the costs.

If you can put down 20% of the purchase price as a down payment, you can avoid private mortgage insurance, an extra monthly expense. Lenders also offer better terms and interest rates for borrowers with bigger down payments and high credit scores.

Millennials are taking advantage of lower rates and refinancing at higher degrees than generations.

Also consider compromising on your dream home. Getting a fixer upper, a smaller home or one outside your ideal neighborhood could mean a more affordable price tag.

Current homeowners should keep an eye out on interest rates, so they can refinance when they are low. Today the rate for a 30-year fixed mortgage stands at 3.69% where a year ago it was at 4.94%.

“Millennials are taking advantage of lower rates and refinancing at higher degrees than generations that are older,” says Jeremy Sopko, chief executive officer at Nations Lending, a direct mortgage lenders.

Dhara is a writer for Yahoo Finance. Follow her on Twitter @dsinghx.

Read more:

Follow Yahoo Finance on Twitter, Facebook, Instagram, Flipboard, SmartNews, LinkedIn, YouTube, and reddit.