Large companies that received money from the small business loan program are not to blame, according to former TARP watchdog Neil Barofsky.

"You have to go back to the design of the program itself," Barofsky told CNBC on Thursday.

"I mean, Congress went out of its way to carve out businesses that have more than 500 employees but are restaurants, chain restaurants," he said on "Closing Bell." "They wanted the money to go to chain restaurants."

Barofsky, an attorney, previously served as the inspector general for the controversial Troubled Asset Relief Program, which Congress passed to stabilize the financial system amid the 2008 crisis.

The Paycheck Protection Program, created in March as part of the government's $2 trillion aid package, was intended to help small businesses with fewer than 500 employees weather the economic shock from the coronavirus crisis.

But large restaurants and hotel chains received an exemption for locations with less than 500 employees.

Publicly traded companies such as Shake Shack, Potbelly and Ruth's Hospitality Group all received loans through the program but have since announced they were returning the money.

"They participated because they qualified and then there's this tremendous political backlash," Barofsky said, specifically referencing Shake Shack as an example.

Treasury Secretary Steven Mnuchin has criticized large companies that applied for PPP loans and since announced that the Small Business Administration will audit any loan worth more than $2 million.

"This was a program designed for small businesses. It was not a program that was designed for public companies that had liquidity," Mnuchin told CNBC earlier this week.

The SBA also issued guidance last week that rendered it less likely publicly traded companies can tap into the next round of PPP funding. The initial $349 billion was used up, but Congress has since approved an extra $310 billion.

Some people, such as CNBC's Jim Cramer, have criticized banks who made loans to large companies.

But Barofsky, a partner at the law firm Jenner & Block, sought to deflect blame from the large banks, some of which faced scrutiny for reportedly prioritizing applications from wealthier clients.

Banks are just following a "general business sense, where you take care of your bigger and best customers first," argued Barofsky.

"If the government wants banks to carry out its policy, it has to counter those incentives or provide its own incentives if it wants a truly level playing field, otherwise it's not going to happen," he said.

"And that's not the banks' fault," he added. "Ultimately, that's the government's fault."