The Federal Emergency Management Agency — or FEMA — wants its money back.

The US disaster response agency is now asking Hurricane Sandy victims to return millions that it accidentally handed out following the devastating storm, which in the fall of 2012 affected the entire eastern seaboard of the US, from Florida to Maine, and as far west as Wisconsin.

The agency is hoping to recoup some $5.8 million in aid it disbursed to households affected by the "superstorm" that flooded several communities and killed dozens of people while damaging or destroying tens of thousands of homes.

FEMA shelled out some $1.4 billion in aid following the storms — but it is now looking into some 4,500 households it has found to be ineligible for the funds, and it has already sent out letters to about 850 of them asking for its money back, the Associated Press found in an investigation of the agency's records.

The funds were "improperly" allocated as a result of technical or bureaucratic mishaps rather than actual fraud. Some households received more than they were supposed to. In some instances, this was because the properties were second homes or rentals rather than primary residencies, because they were later covered by private insurance policies, and because more than one household member applied for and received aid.

"Unfortunately, whether through fraud, human or accounting errors, or for other reasons, assistance sometimes goes to individuals who are not eligible for it during the response to any disaster," a FEMA spokesman told VICE News in an emailed statement. "FEMA and other federal agencies are required by law to recoup improper payments."

The spokesman added that the agency strives to explain the recoupment process to disaster survivors, and that it offers them the option to pay off the debt, reach a compromise, or appeal the decision. As of June, the agency had already recovered about $453,000.

But let's face it: it's pretty unlikely the agency is going to be able to get all the money back, especially as it is demanding an average of $6,987 per household from families that are for the most part middle to low income. At least half of the households under scrutiny for having been paid too much reported an annual income under $30,000, according to the AP.

Ann Dibble, director of the storm response unit at New York Legal Assistance Group, a nonprofit that has been assisting low-income Sandy victims, told VICE News that she has worked with 25 clients to whom FEMA has asked for money back.

Dibble helps them challenge FEMA's recoupments attempts, usually trying to get the debt waived on the basis of financial hardship. Homeowners with more means can sometimes work out repayment plans, she added.

FEMA will often condone the debt — but it will rarely admit its own responsibility in the misallocation of funds, Dibble said.

"In the cases that we have seen so far, FEMA has been frankly a little bit ambiguous," Dibble said. "We've been successful in getting some of this debt waived, but FEMA does not give us reasons. In one instance that I can think of they actually acknowledged that the recoupment was wrong and they reversed, but in other instances they'll send a notice that the debt is waived and they won't tell us why."

Long Island resident Gary Silberman got a letter in the mail asking him to repay FEMA some $17,000 he received after losing the home he shared with his father to the hurricane. A FEMA review found that both he and his father received aid — a violation of FEMA's "double dipping" policy — and that they had failed to get flood insurance following an earlier FEMA payment for damages caused by Hurricane Irene, in 2011.

"I lost my home. I lost everything," Silberman told the AP. "I don't have $17,000 to give back."

FEMA told VICE News that it's been getting better at handling the funds.

"Since 2007, FEMA has worked diligently to put protections and processes in place that reduce the instance of improper payments," the agency spokesman said, citing efforts to reduce duplicate payments.

But the agency is not the only one to have screwed up. The New Jersey Department of Consumer Affairs has also asked 51 homeowners to return about $940,000 in aid. And the US Department of Housing and Urban Development (HUD) has told New York and New Jersey officials to review hundreds of millions of dollars they allocated to post-Sandy rebuilding efforts to ensure the money wasn't "unduly enriching homeowners beyond their actual expenses."

That concern is not entirely unjustified, as FEMA has a bit of a history of misallocating funds and then trying, not always successfully, to get them back.

Following Hurricane Katrina, in 2005, FEMA "improperly" disbursed millions in funds, sending out checks to many that weren't eligible for them in an attempt to speed up the recovery from the devastating storm.

When the agency later tried to get the money back, it was taken to court, and eventually authorized by Congress to forgive much of the debt.

Just two years after Katrina, hundreds of people were charged with attempting to defraud the emergency relief system — but only seven were charged for similar crimes in the two years since Sandy, suggesting either less fraud, or weaker investigations.

"After Katrina, FEMA identified 160,000 people that they thought could potentially be subject to recruitment. It was a very, very messy process," Dibble said. "We hope that FEMA learned some lessons and that after Sandy they'll be more rational and more reasoning when they seek recoupment. However, from my experience, we are just seeing the beginning of a wave of recoupment notices that are going to go out."

That effort is challenged by the fact that many people displaced by Sandy have moved, some several times, and never receive their notices.

"But if people don't challenge these notices, they'll have a significant federal debt that they're going to be dealing with for a long time," she added. "People are still struggling to recover, and the last thing they need is FEMA now coming back and saying, while they're still trying to repair their homes and get their lives back in order, 'you have to repay this money.' We want people to know that there's something they can do about it."

Follow Alice Speri on Twitter: @alicesperi