Bruce Berkowitz's Fairholme Capital Management has taken a roughly $500 million preferred equity stake in Fannie Mae and Freddie Mac, CNBC learned Wednesday.

"Taxpayer dollars expended by the government during a time of national crisis will be fully repaid," he wrote in a note sent to CNBC. "And equitable treatment of taxpaying shareholders, including community banks and insurance companies, must be restored.

"The government's ability to fully recoup its investment and restoring value to shareholders are not mutually exclusive. This is the American way."

The mortgage finance provider that was taken into federal receivership with $188 billion in tax funds, turned a profit for the first time since 2006 earlier this year.

(Read More: Fannie Mae: Zombie Stock Turns Cash Cow)

Shares of Fannie Mae traded at $3.18, down 22 percent around 2 p.m. ET. Freddie Mac declined 23 percent, priced at $2.87 per share.

Berkowitz also told CNBC's David Faber that it was time to restructure the mortgage giants.

"The time to restructure Fannie and Freddie is upon us," he wrote. "Sustaining our nation's economic recovery requires it. On behalf of the hundreds of thousands of Fairholme shareholders who helped to rebuild American International Group, Bank of America, CIT Group, General Growth Properties, MBIA Inc., and others after the Great Recession – we stand ready to do our part."

(Read More: Ralph Nader, Wounded Shareholder of Fannie Mae and Freddie Mac)

Trader disclosure: On May 29, 2013, the following stocks and commodities mentioned or intended to be mentioned on CNBC's "Fast Money" were owned by the "Fast Money" traders: Pete Najarian is long AAPL; Pete Najarian is long BAC CALLS; Pete Najarian is long C CALLS; Pete Najarian is long JPM CALLS; Pete Najarian is long XLF CALLS; Pete Najarian is long CSCO CALLS; Pete Najarian is long YHOO CALLS; Pete Najarian is long BBRY ; Pete Najarian is long SBUX; Pete Najarian is long FB; Pete Najarian is long F CALLS; Pete Najarian is long BMY; Jon Najarian is long BAC; Jon Najarian is long GLD; Jon Najarian is long MSFT; Jon Najarian is long SFD; Jon Najarian is long F; Jon Najarian is long JOY; Jon Najarian is long NTAP; Jon Najarian is long OC; Jon Najarian is long TMO; Jon Najarian is long ZNGA; Jon Najarian is long UVXY; Mike Murphy is long BAC; Mike Murphy is long FB; Mike Murphy is long F; Josh Brown is long AAPL; Josh Brown is long GLD; Josh Brown is long TGT; Josh Brown is long KORS; Josh Brown is long F; Josh Brown is long VAW.

