In a town hall discussion broadcast live on CNBC Monday, President Barack Obama said the country's economy is "moving in the right direction" -- even if it doesn't feel that way.

Responding to questions from, in addition to the host John Harwood, a student, a Wall Streeter, a small business-owner and a self-described member of the middle class, the president acknowledged that "times are tough for everybody," but pointed to his record so far as president and asked the audience to trust in his agenda. "We went through the worst recession since the Great Depression," he said. "Those programs that we put in place worked. So now you've got a financial system that is stable. ...The challenge is that the hole was so deep."

The tough questions came near the beginning. The chief financial officer for a veterans service organization, who called herself a "middle-class American," said she was "exhausted of defending you" and "deeply disappointed with where we are right now."

WATCH a compilation of attendees' questions for President Obama:

<1--3103--hh>1--3103--hh>

"The life you describe, one of responsibility, looking after your family, contributing to your community, that's what we want to reward," Obama said in response. "I understand your frustration." He proceeded to list his administration's reforms on student loans, credit cards and "a whole host of things that we've put in place that do make your life better."

The president deflected criticism from Wall Street as well. He pushed back against the assertion that he has vilified business, saying the angry response to his reforms has been irrational. His citing of historical examples, such as the creation of Medicare, recalled that passage from a 1933 Time article he read in a speech in April -- "Through the great banking houses of Manhattan last week ran wild-eyed alarm. Big bankers stared at one another in anger and astonishment..." -- which refers to the creation of the Federal Deposit Insurance Corporation. When SkyBridge Capital manager Anthony Scaramucci told Obama, in a question, that Wall Street feels like a "pinata," the president said, to applause, "There's a big chunk of the country that thinks that I have been too soft on Wall Street."

WATCH Obama respond to accusations that he's been too tough on Wall Street:

"Me saying you should be taxed more like your secretary, when you're pulling home a billion dollars or a hundred million dollars a year, I don't think is me being extremist or anti-business."

Throughout the speech, Obama insisted he was allied with businesses of all sizes. In response to a question from a small business owner who does monogram glass work, Obama pointed to the "eight tax cuts for small businesses so far" that his administration has passed, calling them "pro-business agendas." He admitted, further, that his own political approach to this legislation hasn't always been ideal, saying that he will be "setting a better tone so that everybody feels like we can start cooperating again, instead of going at loggerheads all the time."

The president's argument, ultimately, was that reforms need to continue. Regarding housing, Obama countered the argument that allowing the market to fall naturally would help it more than continuing to prop it up. "We were very successful in keeping the housing market alive at a time when it had completely shut down," he said. "My job as president is to think about those families that are losing their homes, not as some abstract numbers. I mean, these are real people."

He also touched on the possibility of a payroll tax holiday ("This is something that we've examined") and on China's capital restrictions. "What we've said to them is you need to let your currency rise," he said about China. "We are going to continue to insist that, on this issue and on all trade issues between us and China, it's a two-way street."