Story highlights House Speaker John Boehner and other GOP leaders say they'll stand by Steve Scalise

Leaders were initially frustrated with Scalise's first response to the story

Scalise called GOP members of the House on Tuesday

House Republican leadership is standing by Rep. Steve Scalise, the No. 3-ranking House Republican, in the wake of a firestorm of criticism surrounding his 2002 speech to a white supremacist group.

Scalise's position as House majority whip has been thrown into doubt by the revelation, and the congressman had been calling members to gauge the level of support he had from his party, according to a senior House Republican source.

But House Speaker John Boehner, in a statement issued Tuesday afternoon, said Scalise has his "full confidence" as whip.

"More than a decade ago, Representative Scalise made an error in judgment, and he was right to acknowledge it was wrong and inappropriate. Like many of my colleagues on both sides of the aisle, I know Steve to be a man of high integrity and good character. He has my full confidence as our Whip, and he will continue to do great and important work for all Americans," he said.

His message came shortly before a similar message of support from House Majority Leader Kevin McCarthy.

The controversy surrounding Scalise's address to a group founded by former grand wizard of the Ku Klux Klan David Duke had become a major distraction for Boehner and his leadership team since the story drew national attention on Monday.

JUST WATCHED House whip likely spoke to supremacists Replay More Videos ... MUST WATCH House whip likely spoke to supremacists 02:39

JUST WATCHED Congressman: CDC has to do a better job Replay More Videos ... MUST WATCH Congressman: CDC has to do a better job 03:43

JUST WATCHED Jones and Scalise debate Obama's fix Replay More Videos ... MUST WATCH Jones and Scalise debate Obama's fix 01:32

JUST WATCHED Obamacare site a 'national embarrassment' Replay More Videos ... MUST WATCH Obamacare site a 'national embarrassment' 05:17

JUST WATCHED Cicilline: Boehner needs Democrats Replay More Videos ... MUST WATCH Cicilline: Boehner needs Democrats 01:29

In a statement issued Tuesday, Scalise said he spoke to the group while he was barnstorming the district to build support for a piece of legislation — and that speaking to them was a "mistake I regret."

"One of the many groups that I spoke to regarding this critical legislation was a group whose views I wholeheartedly condemn. It was a mistake I regret, and I emphatically oppose the divisive racial and religious views groups like these hold," he said in the statement.

"I am very disappointed that anyone would try to infer otherwise for political gain," he added, and decried the group's "hateful bigotry."

The statement did not address how much Scalise knew about the group at the time of the speech.

Duke himself has said he doesn't know Scalise. Duke's former campaign manager, who said he organized the event, also said the congressman likely wasn't aware of the controversial history of his group.

However, a Roll Call report from 1999 suggests Scalise knew Duke well, and was critical of his beliefs. At that time, both were considered potential contenders in a House special election, and Scalise panned him in comments to the paper.

"The novelty of David Duke has worn off," Scalise said. "The voters in this district are smart enough to realize that they need to get behind someone who not only believes in the issues they care about, but also can get elected. Duke has proven that he can't get elected, and that's the first and most important thing."

In a coordinated move Tuesday, Boehner and McCarthy issued statements supporting Scalise within minutes of Scalise's new statement. There was frustration, according to the aides, that Scalise's initial statement, in which he blamed sloppy staff work for his appearance before the group in 2002, made it appear he wasn't taking full responsibility for the incident and moving to get it passed him.

It is unclear at this point how much blowback Scalise will get from rank-and-file Republicans, and that might now be fully visible until lawmakers return to work next week after digesting the unfolding situation. While the only public expressions from Republicans to this point have been supportive, the aides predicted Scalise still needs to work hard in the days ahead to maintain the confidence of his colleagues.

Critics of Scalise have been bipartisan, but so have his defenders. He's received strong support from Louisiana politicians, with African-American Rep. Cedric Richmond — the state's only Democratic House member — defending him from charges of racism.

"I don't think Steve Scalise has a racist bone in his body," Richmond told the New Orleans Times-Picayune . "Steve and I have worked on issues that benefit poor people, black people, white people, Jewish people. I know his character."

But national Democrats have pounced on the episode, and as Boehner remained silent on the issue for most of Tuesday, ramped pressure on House Republican leadership to weigh in

House Minority Leader Nancy Pelosi's spokesman, Drew Hammill, said the news of Scalise's speech is "deeply troubling" for a GOP leader, but also declared that "actions [of the House GOP caucus] speak louder than whatever" Scalise said during the forum.

"Just this year, House Republicans have refused to restore the Voting Rights Act or pass comprehensive immigration reform, and leading Republican members are now actively supporting in the federal courts efforts by another known extremist group, the American Center for Law and Justice, which is seeking to overturn the President's immigration executive actions," he said in a statement.

The Democratic Congressional Campaign Committee issued a scathing statement charging Scalise "chose to cheerlead for a group of KKK members and neo-Nazis at a white supremacist rally" and slamming House leadership for their silence on the development.

"While David Duke defends Scalise, Speaker Boehner and Leader [Kevin] McCarthy are refusing to condemn Scalise's choice of allies," said Democratic Congressional Campaign Committee National Press Secretary Josh Schwerin.

Schwerin said the incident made it "clear their leadership has a history of embracing anti-Semitic, racist hate groups."

According to an agenda for the event and notes attendees posted afterward, Scalise appeared at the National/International EURO Workshop on Civil Rights, a white nationalist organization founded by David Duke, a former grand wizard of the Ku Klux Klan.

The appearance was first reported on Sunday on CenLamar.com, a Louisiana politics blog run by Lamar White Jr.

But Scalise vehemently disavowed the group's beliefs in his interview with the Times-Picayune, and said he "spoke to any group that called" — comparing it, as an example, to the League of Women Voters, a nonpartisan group known for helping to register new voters.

"When you look at the kind of things they stand for, I detest these kinds of views. As a Catholic, I think some of the things they profess target people like me. At lot of their views run contradictory to the way I run my life," Scalise added.

Scalise also suggested the appearance was in part due to staffing issues.

"I had one person that was working for me. When someone called and asked me to speak, I would go. I was, in no way, affiliated with that group or the other groups I was talking to."

CNN has learned that the staffer at the time was Cameron Henry, who currently holds Scalise's former state House seat. Henry rushed CNN off the phone Monday night and declined to discuss the situation, but did not deny his work for the congressman.

Henry's brother, Charles Henry, is Scalise's current chief of staff. Neither responded to requests for comment on Tuesday.

The controversy comes just days before Republicans take full control of Congress with House Majority Whip Scalise poised to play a key role in shepherding through conservatives' legislative priorities.

But even conservatives have expressed frustration with Scalise, who they believe hasn't stood strong on their priorities — pointing most recently to his vote for the government funding measure that drew heavy opposition from conservatives because it didn't address President Obama's immigration executive action.

Some, however, expressed support for him Tuesday. Rob Maness, the former Louisiana Senate candidate backed by the Tea Party, has been named as a potential primary challenger to Scalise but backed him in a statement to CNN.

"As Congressman Scalise has already conceded - attending this event was a mistake. I think we are all currently taking him at his word that this was an isolated incident that happened some 12 years ago," Maness said.

He added that if that's true, "this is clearly an orchestrated attack designed to distract" from the real issues, including "fighting back against President Obama's executive amnesty, correcting a weak and feckless Foreign Policy and stopping the massive expansion of government growth and spending."

More broadly, however, the establishment silence has infuriated conservatives still smarting over their loss in the Mississippi Senate race, when their chosen candidate, state Sen. Chris McDaniel, was narrowly defeated by Sen. Thad Cochran after establishment Republicans lodged racially-charged attacks against him.

McDaniel came under heavy fire from establishment Republicans for being billed as a headliner for a rally alongside a white nationalist, though he never ultimately attended the rally. In an email to CNN, McDaniel said that "of course there is a double standard for the GOP establishment."

He noted that former Mississippi Gov. Haley Barbour "has a long history of despicable comments, some of them overtly racist," pointing to a Politico article outlining some of the more salacious.

"And yet, he receives a free pass from establishment politicians because of his perceived influence. All the while, others are forever tarnished for less serious deeds," McDaniel said.

He added: "The establishment wings of both parties are more hypocritical than fair, seeking to crush anyone -- friend and foe alike -- who would threaten their hold on power."

Politicians in Scalise's home state of Louisiana rallied to his defense on Monday, however, saying they don't believe he agrees with the white supremacist group's ideology.

"I know Congressman Scalise to be a good man who is fair-minded and kindhearted. I'm confident he absolutely rejects racism in all its forms," Louisiana Gov. Bobby Jindal said in a statement.

But Scalise's alleged appearance at the event drew a harsh rebuke from Erick Erickson , the conservative RedState.com blogger and former Louisiana resident who asked of the congressman: "How do you not know? How do you not investigate?"

"How the hell does somebody show up at a David Duke organized event in 2002 and claim ignorance?" Erickson wrote in a post Monday.

He said Trent Lott -- the former Senate majority leader who was driven from his post after praising Strom Thurmond's 1948 segregationist presidential campaign -- lost his gig "for something less than this" in 2001.

And he pointed to Republicans who hit Mississippi Senate candidate Chris McDaniel in 2014 for attending events hosted by the Sons of Confederate Veterans and for making plans to attend a rally where he was billed along with a white nationalist.