Sen. Sherrod Brown, D-Ohio, said Sunday he is open to working with President Donald Trump on trade, including renegotiating NAFTA to help American workers, but added the "executive retreat for Goldman Sachs" in the White House stands in the way of progress and presents a conflict for Trump.

"The White House looks like an executive retreat for Goldman Sachs now," Sen. Brown told NBC's "Meet the Press" with Chuck Todd. "And the president surrounds himself with that."

President Trump has politicked on funding for his border wall – which Brown called "ludicrous" – and presented a torn up and renegotiated North American Free Trade Agreement with Canada and Mexico can help Trump in getting the latter to help fund the wall.

"As you know, Chuck, I wrote a book about trade," Brown told Todd, referring to his book "Myths of Free Trade: Why American Trade Policy Has Failed." "I understand what we need to do for a different trade policy to raise wages, to help those workers."

President Trump's inner-circle conflicts stand to obstruct his own stated goals on trade.

"And in the end, it's whose side are you on?" Brown asked Todd. "This president is on the side of Wall Street and big oil and the drug companies."

Brown did praise President Trump's U.S. Trade Representative Robert Lighthizer, saying an "America first" policy is tenable here and favors the working class.

"[Lighthizer] is the best appointee in the Trump administration," Brown told Todd. I talk to him at least every two weeks. We are talking about NAFTA renegotiation, how we do it. We start off preconditions for negotiations starting with buy America provisions, with anti-outsourcing provisions.

"I want workers' voices at the table. I don't want these trade agreements written by corporate interests then using these trade agreements to shut down production in Newark and Toledo, Ohio and Mansfield and move overseas and sell those products back into the United States. That's what a trade policy's about."

The ball is in the court of Lighthizer and President Trump is working against – in some cases – the goals of his own advisers.

"The president's problem is he's surrounded in the White House with a bunch of people that like those trade agreements, and so Rep. Lighthizer has to figure out how do you get this White House to support him on wanting to renegotiate NAFTA and put workers first?" Brown said. "That's the problem we face. That's what I fight for in the finance committee. That's what I fight for in the Senate."

Brown concluded with a call for more action is lieu of mere talk on trade from the president.

"I work with this president," Brown said. "When he's right on trade, it's mixed.

"I don't know if he's going to be right on trade because he's not really done anything on it yet except make speeches."