Jeff Swaney is worried about selling his 5,600-square-foot home one day. In his neighborhood south of Atlanta, demand and prices for large ranch houses like his ave declined over the last decade, as more young professionals move to smaller abodes in hipper areas. He doesn't expect that to change anytime soon.

Janet and Jeff Swaney in front of their Atlanta area home Source: Jeff Swaney

The 51-year-old real estate investor and owner of Swaney Consulting Group has personal reasons to hold on, at least for now. He may eventually move to a condo at the beach, but wants his future grandchildren to enjoy his pool, yard and basement. For these amenities, he spends about $18,000 annually in lawn maintenance, taxes, insurance and utilities alone. The housing market, on the rebound since the Great Recession, is increasingly being driven by millennials and first-time homebuyers who "are hungry for starter homes and efficient layouts," said Javier Vivas, manager of economic research for realtor.com. The trend may leave some older homeowners in a lurch if they want to retire, downsize and cash in their nest egg. Large single family homes — defined as the largest 25 percent of all listings on realtor.com and about 2,900 square feet to 4,000 square feet — receive 12 percent to 45 percent less views on realtor.com than the typical home in each market. This year so far, large, single family homes are selling up to 73 percent (or 50 days) slower on average than the typical home in each market. The often hefty price tags for bigger homes contribute to their lengthier sale times because there is a smaller pool of buyers who can afford them, said Artur Miller, founder and CEO of Miami-based AMLUXE Realty.

'Very difficult sell'

The Swaney home Source: Jeff Swaney

Even Swaney, whose 1994 home appraised for $350,000, thinks he may have a tough time selling. "The McMansions that soon-to-retire people purchased in the 80s and 90s are a very difficult sell right now," said Melissa Rubenstein, a former real estate attorney who now sells luxury properties with Re/Max HomeTowne Realty in Bergen County, New Jersey. Many are outdated and may not include a first floor bedroom and bath suite for aging in place or in-laws. Listings of large homes are also up two percent from last year, suggesting owners are dumping them faster, while listings of all homes are down 10 percent from last year, according to the realtor.com data. "We're finding these homes are an albatross for clients," said Michael E. Chadwick, a financial planner and owner of Chadwick Financial Advisors in Unionville, Connecticut. "We've got several right now who have been trying to sell them and move south, and they've cut the asking price by over 30 percent each and they're still not going anywhere fast," he said. But even for big homeowners selling now, purchasing another home is likely going to be a lot more expensive for them if they recently refinanced to historically low rates, said Tim Manni, mortgage expert at NerdWallet. Inventory from which to choose is also low.

We're finding these homes are an albatross for clients Michael E. Chadwick owner, Chadwick Financial Advisors

A NerdWallet analysis of three years of data from realtor.com for 20 of the largest U.S. metro areas revealed that the smallest quarter of homes appreciated at a faster rate than larger ones. Here's what's more: Homeowners who thought their big house was going to be their ticket to retirement may realize that's not the case once they analyze the numbers.

Profits may vanish

"People look at purchase price and sales price and think they've made a lot, but once you factor in repairs, maintenance, taxes, insurance, upgrades, renovations, they lose money most of the time," Chadwick said. "The taxes and insurance are outlandish. The younger generation doesn't want to own or take care of these homes. It's all about free time not being tied down to a property."

Phillip Spears | Getty Images

Then there's capital gains taxes: The more valuable and the more equity there is, the more taxes are due at the time of sale. Large house owners have the most trouble when they've expanded or improved their home beyond the rest of the neighborhood. Then, they become stressed to learn they can't get the price they want, said Michael Fisher, a real estate agent with Century 21 Beachside Realtors in Orange County, California. "They may have stayed in their small starter home due to personal financial or market conditions, keeping the children in the same schools or just love the area they live in," Fisher said. "Rather than move as their needs changed they just added on a second story or expanded the home's footprint to accommodate their lifestyle," he said. "Now the home is as twice as large or more than any other home in their tract and can't get the same price per square foot as others have."

All of this is not to say there aren't people who will always prefer a large house. Among the wealthy, extra large homes will always sell if designed and priced right, Miller said. A 5,000- square-foot home Miller listed for nearly $2 million took about a year to sell, while others nearby sold in less than a month at lower, more realistic prices.

Strike while market is hot