ASPEN, Colo. (Fortune) -- Palm Beach and New York may have been hardest hit by the Bernard Madoff scandal, but residents in Aspen, Colo., are quietly tallying up massive losses.

The upscale resort community has a population base of under 6,000, but reports over the past week reveal that dozens of residents have been undone by the $50 billion Ponzi scheme allegedly perpetrated by Madoff.

According to local asset managers and lawyers, these families have lost upwards of $1 billion. Some have seen up to 95% of their life savings disappear in the past few days -- both long-time locals and part-timers are affected.

Involved or not, residents of this wealthy community can talk of nothing else. Everyone, it seems, knows multiple families who were crippled.

One very wealthy couple has already put their house on the market and moved in with their grown kids. Then there's the guy who smelled the recession coming, sold his expensive home here at the height of the market frenzy and, yes, invested it all with Madoff. Today -- no home, no money. Others speak with great pity of the widow who invested her husband's life insurance money with Madoff only to find herself left without a penny.

Though no one was willing to divulge their name due to upcoming legal suits and the fear of being forced to return assets made through Madoff in earlier years, some did share their stories.

One couple who has lived in Aspen for 25 years was able to retire early with their small real estate business and their investments with Madoff. Now, at ages 59 and 64, they'll have to go back to work and sell their home.

"It's shocking," admits the wife, who says they don't have a real grip on things yet. "The hardest part is this guilt feeling we can't shake. But we're tough and we've been through a lot in our life. We'll hunker down and protect what we have."

Selling their home will be no easy feat in a housing market that has grinded to a halt. Home sales are down some 40%, reports Chuck Frias of Frias Properties in Aspen.

The realtor says he is shocked by the number of residents who fell victim to the Madoff scandal - though he is hopeful that the fraud will have only a minor impact on local housing prices. "Home prices have not dropped as much as you might think," he said.

Another influential, full-time local family of 15 years, met Madoff on a week-long boat trip some 12 years ago and ultimately decided to invest millions.

The wife, who got skittish, pulled out her money years ago. The husband will say only that he lost a "large sum of money." Fortunately the couple is wealthy enough that it won't change their lifestyle tremendously and they hope to salvage what they can through tax write-offs. The biggest effect it will have on Aspen, they believe, is in the amounts given to local charities, already hit hard by the recession.