Story highlights Panetta was Bill Clinton's chief of staff

The remarks come after Gillibrand said Clinton should have resigned

Washington (CNN) Former Defense Secretary Leon Panetta defended President Bill Clinton on Saturday, telling CNN he "more than paid the price" for his affair with Monica Lewinsky while he was in office.

"The fact that he went through an impeachment process as President of the United States," Panetta told CNN's Ana Cabrera in an interview Saturday. "The House of Representatives voted for articles of impeachment, (but) the Senate did not.

Panetta, who served as Clinton's chief of staff, continued: "But the mere fact that he went through an impeachment process as President means that there will always be a shadow on the legacy of his presidency, so at least from my point of view, he's more than paid the price."

His remarks come after New York Democratic Sen. Kirsten Gillibrand said she believes Clinton should have resigned after the scandal.

She told The New York Times on Friday when asked if Clinton should have stepped down, "Yes, I think that is the appropriate response."

Read More