(CNN) Former Defense Secretary Harold Brown, who served under President Jimmy Carter, and wore many hats as an educator and a nuclear physicist, has died. He was 91.

Brown played a crucial role in the Carter administration's effort to end the Iranian hostage crisis, and described the botched 1980 rescue attempt as the biggest regret of his career.

"The failure to rescue the U.S. hostages still haunts me," Brown wrote in his memoir "Star Spangled Security," according to the RAND Corporation , a think tank where Brown served on the board of trustees for decades.

It announced his passing Saturday and detailed his life and career as a nuclear physicist, weapons designer, California Institute of Technology president, philanthropist and public servant.

Before he became the the 14th defense secretary, Brown served as the Air Force secretary under Lyndon Johnson during a period that included the US bombing of North Vietnam.

Read More