WASHINGTON (CNN) -- A pair of Republican congressmen ripped Congressional Black Caucus members for ignoring Cuba's "myriad gross human rights abuses" Thursday, saying this week's caucus trip to the island nation ignored the plight of political prisoners under the Castro regime.

Rep. Chris Smith, R-New Jersey, blasts members of the Black Caucus on Thursday for their trip to Cuba.

They also urged the Obama administration to refrain from easing trade embargo or travel restrictions until the Cuban government releases all "prisoners of conscience," shows greater respect for freedom of religion and speech, and holds "free and fair" elections.

The call from Reps. Chris Smith, R-New Jersey, and Frank Wolf, R-Virginia, came three days after the administration signaled that new rules on family travel and remittances to Cuba may be announced before President Obama goes to the Summit of the Americas on April 17.

It also followed statements from several Black Caucus members Tuesday arguing for consideration of an end to the trade embargo and other diplomatic restrictions placed on Cuba for five decades. Watch CNN's Ed Hornick discuss the story »

"Yes, we have history -- we have good history and not-so-good history," said Rep. Laura Richardson, D-California. "But the point is it's history, and we need to move forward."

Three members of the caucus were visiting the Latin American School of Medicine, where students from nations including the United States study, when they received an invitation to Fidel Castro's home for a meeting.

"Former President Fidel Castro is very engaging, very energetic," said CBC Chairwoman Barbara Lee, also a from California Democrat. "Our conclusion is, given the new direction in our foreign policy, that it's time to look at a new direction in our policy toward Cuba. Watch Lee discuss her visit to Cuba »

"The 50-year embargo just hasn't worked," she said.

The caucus members, who also visited current Cuban President Raul Castro, "did nothing to publicly show any concern for the myriad gross human rights abuses perpetrated by the Cuban government or the tragic fate of hundreds of Cuban democracy and human rights activists," Smith said.

"Yet they held press conferences at which they heaped and lavished praise and affection for a government the United States Department of State only six weeks ago called 'totalitarian.' "

A 2008 State Department human rights report cited, among other things, numerous accounts of beatings of Cuban political prisoners, harsh and life- threatening conditions for dissidents, and severe restrictions on freedom of movement and religion, Smith said.

"Over the past 50 years, the Castros and their secret police have been directly responsible for killing thousands of nonviolent, courageous pro-democracy activists and for jailing and torturing tens of thousands of others. And they continue to this day to perpetrate their brutal crimes," he said.

"Before the Obama administration even thinks about permitting further travel to Cuba, or altering the trade embargo on Cuba, both the White House and Congress have a moral obligation, a duty, to ensure that the Cuban dictatorship releases all prisoners of conscience, makes substantial progress in respecting freedom of religion, speech and, press and assembly, and holds free and fair elections."

Lee responded Thursday that it made no sense to continue what she characterized as a failed policy.

"Everyone is entitled to their own opinion, but by any objective standard our current policy toward Cuba just hasn't worked. Simply put, it's time to open dialogue and discussion with Cuba," she said in a statement.

"I am convinced, based on the meetings which were held, that the Cubans do want dialogue, they do want talks, and they do want normal relations with the United States of America. And I believe that it's in the United States' best interest to do that."

On Monday, the White House adviser for the Summit of the Americas, Jeffrey Davidow, refused to be pinned down about when the administration might announce its easing of the restrictions on travel and remittances to Cuba.

"I really could not say. But I would not be surprised if it came before the summit," he said. "They're going to happen. I can't tell you exactly when."

Before he was elected president, Obama promised to ease restrictions on Cuban-Americans traveling to visit family in Cuba and sending them money.

All About U.S. Congressional Black Caucus • Cuba • Fidel Castro