UPDATED 08/22/12 2:27 p.m.

YORKVILLE, Ill. (CBS) — U.S. Rep. Joe Walsh (R-Ill.) has criticized the leaders of his party for failing to stand up for embattled Rep. Todd Akin (R-Mo.) during his race for the U.S. Senate.

Akin ignited a firestorm around the country earlier this week, in the wake of his comments that if a woman is a victim of “legitimate rape,” her body will shut itself down so as to prevent a pregnancy.

Speaking at a Multi-County Young Republican Conference in the Kendall County town of Yorkville Tuesday, Walsh joined a long list of others in condemning Akin’s comments. But Walsh did not question Akin’s continued candidacy.

“What he said was offensive, insulting and wrong, but I’m bothered by this rush to pile on,” Walsh was quoted in a report by the Elmhurst Patch. “And I’m bothered by the silence of members of our own party to stand up for him.”

But a Walsh spokesman said Wednesday that Walsh was only speaking “off the cuff” at the event, and emphasized an official statement that Walsh has released.

“Akin’s remarks were not just idiotic and stupid, they were offensive to all women, especially those who have had to endure sexual abuse,” the statement says. “He needs to try to make things right and apologize to those who he has offended.”

Walsh also believes “that only Mr. Akin and his family should decide if he chooses to drop out, not party leadership,” spokesman Justin Roth told CBS 2.

Reps. Randy Hultgren (R-Ill.) and Peter Roskam (R-Ill.) also appeared for the panel discussion and addressed the Akin issue. They both said the main question was whether Akin now stands a chance of beating the Democratic incumbent, Sen. Claire McCaskill (D-Mo.)

Hultgren, as quoted by the Elmhurst Patch, said: “I hope (Akin) does look beyond himself. We’re at a tipping point right now.” He said the GOP needs the Senate and the White House.

Roskam said in the Patch article that no one is questioning whether Akin is fit to serve, and, “”There is no one saying he is immoral or incapable. He’s not; he made a poor decision. The question is: Can he win in November?”

Akin made the remarks about rape on the interview Sunday.

“It seems to me first of all, from what I understand from doctors, that’s really rare,” Akin said of pregnancy for rape victims when asked if he would support abortion rights in such cases. “If it’s a legitimate rape, the female body has ways to try to shut that whole thing down.”

Akin has apologized repeatedly for his remarks, but has refused to step down and has slammed “party bosses” for urging him to do so, CBS News reports. In an interview Wednesday with ABC’s “Good Morning America,” he would not say whether he might consider dropping out of the race at a later time.