Washington (CNN) -- On the anniversary of the September 11 terrorist attacks, President Barack Obama reiterated that America is not at war with Islam but with al Qaeda's "sorry band of men which perverts religion."

Speaking at a commemoration ceremony Saturday at the Pentagon, where American Airlines Flight 77 struck and killed 184 people and five hijackers, Obama said America would hold strong to the values and virtues that have defined the nation.

"They may wish to drive us apart but we will not give in to their hatred and prejudice," Obama said as controversy swirls around a proposed Islamic center near ground zero in New York and anti-Muslim sentiment is running high among Americans.

It was not religion that attacked the United States on September 11, 2001, Obama said. It was al Qaeda.

"Just as we condemn intolerance and extremism abroad, so will we stay true to our traditions here at home as a diverse and tolerant nation," Obama said. " We champion the rights of every American, including the right to worship as one chooses -- as service members and civilians from many faiths do just steps from here, at the very spot where the terrorists struck this building."

Obama's remarks were tailored for a poignant anniversary that this year is cast under a different light.

It was not just heart-stopping grief that gripped America on Saturday. On this September 11, Americans planned to make their voices heard on Park51, the proposed Islamic center two blocks from where the World Trade Center once stood. Rallies were planned for Saturday afternoon.

Critics of the center say it is an affront to survivors of the September 11 attacks. Supporters cite First Amendment rights and the need to express religious tolerance.

The proposed center has cast the spotlight on America's attitudes toward Islam and Obama addressed the issue at a news conference Friday, where he sounded the same message he delivered at the Pentagon.

"The perpetrators of this evil act didn't simply attack America, they attacked the very idea of America itself," Obama said.

"And so the highest honor we can pay those we lost, indeed our greatest weapon in this ongoing war, is to do what our adversaries fear the most -- to stay true to who we are as Americans, to renew our sense of common purpose, to say that we define the character of our country and we will not let the acts of some small band of murderers who slaughter the innocent and cower in caves distort who we are," he said.

The president spoke after a moment of solemn silence marked the time, 9:37 a.m., when the jet crashed into the Pentagon. In similar fashion, thousands in New York marked the two times that planes crashed into the Twin Towers at the World Trade Center, as they do every September 11.

Obama was joined by top defense officials at the Pentagon who paused to reflect on the sacrifices of America's men and women in uniform.

"We honor them with our presence and certainly with this memorial," said Navy Adm. Mike Mullen, chairman of the Joint Chiefs of Staff.

"Mostly, we honor them with our lives, with what we have done from that day to this, the sacrifices we have borne, the laughter we have shared, the hope we have dared to let back into our hearts," he said.

The United States went to war against extremists in Afghanistan shortly after the 2001 attacks. Nine years later, about 100,000 U.S. troops, some of whom were in grade school then, are still fighting there.