RIO DE JANEIRO, Brazil (CNN) -- A New Jersey man whose son is at the center of a five-year international custody fight that has attracted attention from high-level U.S. and Brazilian authorities expressed hope Wednesday that he will get his boy back.

David Goldman has been fighting for custody of his son, Sean, since the boy's mother took him to Brazil in 2004.

"I always keep hope, but I don't really expect it, given the things that always happen," David Goldman told CNN on what he said was his 10th or 11th trip to Brazil to regain custody of 9-year-old Sean Richard Goldman. "Until the wheels are up, I don't expect it and it's tragic."

David Goldman's comments came after a Brazilian Supreme Court judge on Tuesday suspended a lower court's order that would have granted Sean's custody to the U.S. Consulate in Rio de Janeiro, where father and son were to reunite Wednesday.

Judge Marco Aurelio wrote in his decision that taking Sean "in an abrupt manner" from his home could cause the boy psychological harm.

But Goldman said the boy was suffering psychological harm simply by remaining with his Brazilian relatives, whom Goldman accused of turning his son against him.

"According to the expert reports, they've already been doing that for some time and that's very, very sad," said Goldman, who captains boats and works part-time as a model. "The worst is -- he's my son, I'm his dad, and I can't help him. The legal system here right now is preventing me from helping my child." Watch father explain Brazilian court battle »

The incident began in June 2004, when Goldman's wife, Bruna Bianchi Carneiro Ribeiro, took Sean from the family's New Jersey home for what was to have been a two-week vacation in her native Brazil.

But instead of returning, she divorced Goldman, married Joao Paulo Lins e Silva, a Brazilian lawyer, and remained in Brazil. She died there last September giving birth to a daughter.

Sean attends a private school and lives in a sprawling home in Rio with his half-sister and his stepfather.

Recently, after nearly five years without seeing his child, Goldman began visiting him. "He asked me where have I been for this amount of time. How come I never came to visit him?" Goldman said.

On CNN's "Larry King Live" Wednesday, Goldman said that public prosecutors and psychiatrists have declared the boy "emotionally damaged," and added, "He needs to be reunited with me immediately."

In March, a spokesman for the boy's Brazilian relatives said he did not dispute the father's biological rights, but said other matters needed to be weighed. "The fact of the matter is that, in order to be a parent, you have to be more than a DNA donor," said Helvecio Ribeiro. "Fatherhood is not about making home movies and taking pictures. It's about sacrifice; it's about providing support for your child; it's about being there even when you are not there." Watch Goldman describe his fight to get his son back »

He said Goldman had failed to do that, and accused him of having "paid not a dime of child support" and made allegations "all over the place about us not allowing him to visit the child that are completely untrue."

Goldman responded, "Can you take someone's child to another country and then expect the parent to support you in the abduction of the child?"

He said he had been making tireless efforts since his son was "abducted" to have him repatriated, "never, ever stopping."

Don't Miss Brazilian court rules for New Jersey dad

Goldman said the case sends the wrong message to the world about Brazil's legal system. "They're sending this message that anyone can take any child from anywhere, come to Brazil and if they can hide enough or stall enough or keep the child here long enough then they're entitled to that child? That's unacceptable."

Tuesday night's decision, which means the entire Brazilian Supreme Court will hear the case, comes a day after a Superior Court justice ordered Sean to be taken to the U.S. Consulate in Rio and handed over to his father, who had arrived early Tuesday from New Jersey to pick him up. The exchange was to have taken place on Wednesday.

The case has attracted attention from politicians. U.S. Secretary of State Hillary Clinton met with Brazil's foreign minister Wednesday. A senior State Department official said the case was the first thing they talked about.

"We are disappointed by the decision," State Department spokesman Philip Crowley told reporters Wednesday. "But U.S. Embassy officials continue to work with the family, and will meet with Brazilian attorneys and Mr. Goldman's attorney to learn next steps in the legal process."

A lawyer representing the boy's Brazilian relatives said earlier this week, when it appeared that he would be returned to the United States, that doing so would harm him. "The child said many times that he wanted to stay in Brazil," said lawyer Sergio Tostes. "This is not human and it is a cruelty."

Meanwhile, Goldman was not giving up. "My focus is to do everything I can, through every legal means in all matters of law, to reunite with my son and bring him home," he said.

According to the U.S. State Department, 66 American-born children have been taken by a parent to Brazil, in violation of international treaties.

CNN's Kim Segal in Rio de Janeiro and Adam Reiss in New York contributed to this story.

All About Hillary Clinton • Brazil • New Jersey