House Majority Leader Steny Hoyer, D-Md., charged Thursday night that GOP Rep. Jason Smith was casting an inappropriate "aspersion on a Hispanic" Democrat by telling him to "go back to Puerto Rico" -- regardless of whether the comment, as Smith repeatedly claimed, was merely a reference to a Democratic congressional delegation's recent trip to the Caribbean island.

"Either way, it was an aspersion on a Hispanic, that he ought to go back to Puerto Rico," Hoyer told Fox News, when asked about the possibility that Smith did not intend racial animus. Hoyer added that he was sure the comment was directed at Democratic California Rep. Tony Cárdenas, who was born in Los Angeles and has Mexican heritage.

Smith, R-Mo., quickly apologized to Cárdenas for his remark, which came after the parties sparred on the House floor over a bill to reopen the government. Cárdenas said initially that he was "shocked" by the comment, although he later suggested in a statement that he was ready to move on.

"I often heard those kinds of comments when I was a kid growing up in Pacoima, California, where I was born and raised,” he said.

DEMS HEAD TO PUERTO RICO, PARTY IT UP WITH 'HAMILTON' CAST AMID SHUTDOWN

"There is a saying that I was taught by my parents, 'De todo lo malo, siempre sale algo bueno,' which in English means, 'From everything bad, something good will come of it,'" Cárdenas added. "I look forward to working with and getting to know Congressman Smith in the months ahead."

But, Smith reiterated that he was referring to a well-publicized trip by approximately 30 Democratic members of Congress to Puerto Rico --- with their families and lobbyists in tow -- for a winter retreat last week, where some reportedly took in a special performance of the hit Broadway show “Hamilton.”

Those attending the Congressional Hispanic Caucus BOLD PAC gathering in San Juan also met with Puerto Rican officials to discuss ongoing cleanup efforts from Hurricane Maria. The trip attracted scrutiny because, back in Washington, Democrats and Republicans remained at an impasse over how to resolve the partial shutdown over border-wall funding.

Smith was “speaking to all the Democrats who were down vacationing in Puerto Rico last weekend during the shutdown, not any individual," his communications director said.

Approximately 800,000 federal workers are affected by the shutdown, with many working without pay for nearly a month.

Earlier Thursday, Hoyer noted that the House had just voted to disapprove of comments about white supremacy by Iowa Republican Rep. Steve King on Tuesday, and that the mood on Capitol Hill was tense.

"We have just been through a very difficult week. ... I would hope that we could refrain from any implications which have any undertones of prejudice or racism or any kind of -ism that would diminish the character or integrity of one of our fellow members.”

CLICK TO GET THE FOX NEWS APP

Hoyer moved on Wednesday to refer Democrats' motions to censure King to the Ethics Committee. The House approved Hoyer’s motion by voice vote, effectively sidelining the resolutions and precluding a floor vote for now.

House Democratic leaders reportedly were intent on tabling the censure motion because they were concerned about potentially opening the door to similar censure motions against their own members. Michigan Rep. Rashida Tlaib, for example, called President Trump a "motherf---er" and was photographed with Linda Sarsour, a proponent of the controversial Nation of Islam leader Louis Farrakhan, who has compared Jewish people to termites and praised Hitler.

Fox News' Sarah Tobianski contributed to this report.