Alexis Patterson and her 11 year-old daughter, Ophelia. Source: Alexis Patterson

As she scanned the letter's black type, Alexis Patterson felt herself go numb. "The U.S. Department of Treasury ... applied all or part of your payment to delinquent debt that you owe," it read. Patterson stopped reading. One thing mattered now. The single mother would not receive her $3,063 tax refund, including her child tax credit. The government would instead apply the money to her past-due student loan account. She and her 11-year-old daughter, Ophelia, have been homeless over the last few months in Portland, Oregon. Her refund, she hoped, would help them secure a place to stay. "Everything just fell out from beneath me," Patterson, 47, said.

Tax refunds are a lifeline for many people. The average check from the IRS in 2019 so far is $2,795, which represents six weeks of income for the typical family in the U.S., and the largest cash infusion 1 in 3 households will see all year. For the increasing number of people who've fallen behind on their student loans, however, that relief never comes. The U.S. government has extraordinary collection powers on federal debts and it can seize borrowers' tax refunds and earned income and child tax credits, as well as their wages and Social Security. This year, the U.S. government has already taken $3.3 billion from 1.4 million student loan borrowers' federal tax refunds, compared with $2.3 billion from 1.2 million borrowers in 2016, according to data provided to CNBC by the U.S. Treasury Department. It has also pulled $48.6 million this year from 91,000 borrowers' state tax refunds, up from $35.2 million from 70,000 borrowers in 2016. Patterson is now scrambling to get her refund back, but she's finding the process close to impossible. She owes around $80,000 in student loans.

Borrowers in financial distress can request that the Education Department review, and potentially return, their seized tax refund. However, refunds are harder to get back than Social Security or wage offsets, said Persis Yu, director of the Student Loan Borrower Assistance Project at the National Consumer Law Center. To do so, borrowers have to prove that they're at risk of eviction or foreclosure. Since Patterson and her daughter have been homeless for months, she likely doesn't qualify. "It's an absurdly strict standard which doesn't begin to capture the number of different ways in which borrowers are facing extreme financial hardship," Yu said.

The Education Department is required to let borrowers know in advance that their tax refund will be taken but many people never see that letter for one reason or another. "You don't have to get it," Yu said. "They only need to send it." (The government also only needs to notify borrowers the first time their refund will be offset, and not in subsequent years). The Education Department did not respond to CNBC's request for comment. Patterson said she never received a warning, although she also has little time to worry about that now. "When they take your tax refund, what's more important: getting it back or proving you didn't get a letter?" she said.

Patterson had been temporarily renting a room in Portland with her daughter when the bad news from the government arrived. Without that more than $3,000 cash infusion, she said, she couldn't afford to continue to stay there. "We had to leave, and we had nowhere to live," she said. They're currently staying at a friend of a friend's apartment for free until the end of the month, but she said her daughter hasn't been able to attend school on many days after what they've gone though. "She's traumatized," she said. More from Personal Finance:

Retirees are flocking to these 3 states

Unhappy with your tax return? Prepare for 2019

Even the wealthy might not be prepared to retire The government doesn't always appreciate the significant role tax refunds and credits play for financially vulnerable Americans, Yu said. "In conversations I've had with officials at the Department of Education, they view these tax refunds as bonus money," Yu said. "Tax credits and refunds for low-income folks are vital funds to their survival." Last year, the National Consumer Law Center published a report, titled "Voices of Despair," which chronicled how poor people's lives can be thrown into chaos when their tax credits are seized. A single father said he became homeless; a veteran and father of four said he expected to lose his car. The average annual income for student loan borrowers in default in 2009 was $28,680, according to Mark Kantrowitz, an expert on student debt.

Akilah McCadney and her three children, Kamryn, Kyndall and Caleb Source: Akilah McCadney