Bernie Madoff says he is frustrated.

Writing to me from the federal prison where he is serving a life sentence for his epic fraud, Madoff said he is not getting credit for what he calls his "instrumental" role in returning money to his victims. Madoff wrote that he is so frustrated, he is having second thoughts about having pleaded guilty four years ago.

The email tirade—six e-mails in all—follows comments this week by court-appointed trustee Irving Picard, who revealed to CNBC Tuesday that he will seek court approval for another distribution of funds to Madoff victims. That would bring the total payout to more than $5 billion, out of more than $9 billion recovered so far. But Picard said his success is no thanks to Madoff.

"In my view, he has not been helpful," Picard said.

Read More: Money to Be Returned to Madoff Investors Tops $5 Billion

That set off Madoff.

"This is a man that keeps making statements that have no facts to back them up," Madoff wrote. "I wish I went to trial and he would have been required to provide the evidence he claims he has. As you can see, I am frustrated."

Madoff claims his efforts behind the scenes are the reason so much money—more than half his clients' initial investments—has been recovered, and why ultimately all of the $17.5 billion in lost principal could ultimately be paid back.

Related:Are you working for minimum wage? We want to hear from you.

"From the day of my arrest I offered to assist in recovering the investment principal of my customers. I stated that I was confident that I would be able to convince those parties that were complicit in creating my financial problems, to return the money they withdrew from the investment advisory side of my firm. Those parties were well aware of the incriminating evidence I possessed about their complicit activity and wisely came forward with settlements. It was my belief that it was more important to use the evidence I had to pressure the complicit parties to settle, rather than to use this information for a lesser prison sentence for myself. As remorseful as I am for the pain and suffering I have shamefully caused, I take some comfort in the fact that my assistance will in fact accomplish what I have originally claimed, that with my assistance all of my customers will recover their original investment principal."

Read More: Scam of the Century: Bernie Maddof's Crime and Punishment

Let our news meet your inbox. The news and stories that matters, delivered weekday mornings. This site is protected by recaptcha

The largest settlement so far—and the lion's share of the funds recovered for investors—has come from one person: Florida billionaire Jeffry Picower—Madoff's largest client—who died of a heart attack soon after the fraud came to life. Picower's estate turned over $7.2 billion to be distributed to Madoff's other investors, and Madoff claims he is the reason why.

"When I spoke to Picower prior to his death," Madoff wrote, "I made it very clear I would testify about the role of him and ... [others]."

He said he demanded Picower return all the money he had withdrawn from his Madoff accounts over the years.

An attorney for the Picower family calls Madoff's allegation "entirely false."

"There were no conversations between Mr. Madoff and Mr. Picower after Madoff's arrest," said Marcy Ressler Harris, a partner at Schulte, Roth & Zabel in New York.

In a lawsuit filed in May, 2009, Picard alleged Picower was "one of a handful of BLMIS clients with special access to information," including Madoff's fictitious rates of return, and that Picower knew Madoff had backdated trades for him.



Picower died five months after the lawsuit was filed, and his wife settled with Picard in 2011. But Picard said Madoff had nothing to do with it, or any of the other money recovered. Picard said even when he interviewed Madoff in prison on two separate occasions, Madoff was not forthcoming.

"The interviews of him were, really, to confirm the information we found, because those interviews took place shortly before some of our litigations began, and we put this case together really from a blank slate," Picard told CNBC.

Madoff wrote that he has repeatedly offered more help to Picard and his team, but they have not taken him up on it.

"You should be aware that I have continued to offer Picard information concerning the complicity of the banks," Madoff wrote. "I was the only person in my firm that dealt with the officers that handled my account. Not Frank (DiPascali, a Madoff lieutenant who has pleaded guilty to fraud charges) nor anyone else. I have little doubt that the information I could provide would clearly demonstrate the vital role the major banks… played, in the carrying out my fraud, including their role in handling the accounts of my major customers. The ball is in his court."

"There's always the suggestion that there's more that he could help us with," said Picard's chief counsel, David Sheehan. "But when we pressed his counsel to give us an example of that before we start getting on an airplane and fly down to see him, we never get anything that's worthwhile. So we have not taken him up on those offers."

Madoff said he has a theory as to why Picard is not giving him credit for his assistance.

"I certainly can understand his reasons for not acknowledging my role. How could he JUSTIFY HIS FEES," Madoff asked.

Picard and his firm, BakerHostetler, have billed more than $600 million in fees and expenses, money that is paid by Wall Street firms through their dues to the Securities Investor Protection Corporation.

Madoff wrote, "Picard and Sheehan constantly demonstrate their lack of knowledge of how the market making and proprietary firms operate."



He said that is particularly clear when they allege, as they did to CNBC this week, that Madoff's son Mark "knew or should have known" about the fraud. Mark Madoff committed suicide in 2010, on the second anniversary of his father's arrest. Mark Madoff and his brother Andrew worked in Madoff's trading operation, which was separate from his investment advisory business.

"The fact is that the SEC requires the various trading desks to have a CHINESE WALL between each Department," Madoff wrote. "This includes separate supervisors as well as separate profit and loss bookkeeping so that there can be no conflict of interest regarding each other's inventory and trading positions. These Chinese wall procedures are strictly examined by the SEC and FINRA during surprise audits at least twice a year.

As they are well aware, the advisory department was located two floors below the market making and proprietary trading departments and were restricted to card key access which was also audited by the regulators. I should add that our firm has NEVER been found to NOT be in compliance EVER."

Madoff also continues to insist his fraud began in 1992, even though Picard and Sheehan say they have records suggesting the fraud could date back to the 1970s.

"In response to the Trustee's past claims in regard to date the fraud began. I completely dispute his claim of it dating back prior to my statement of the early 1990's. After four years and hundreds of millions of dollars of auditing fees he has not produced any evidence to support his claim."