By Dan Merica, Elise Labott and Shirley Henry

Editor's note: This is one in a series of stories and opinion pieces surrounding the Aspen Security Forum currently taking place in Aspen, Colorado. Security Clearance is a media sponsor of the event, which is taking place from July 17 to 20 in Aspen, Colorado.

After the September 11, 2001, terrorist attacks on the United States, a majority of Americans were worried about terrorism directly impacting their lives, according to a number of polls.

More than a decade later, is that still the case?

That was the primary question John Ashcroft, former attorney general under President George W. Bush, and Phillip Mudd, a former senior official at the CIA and FBI, debated at a Friday panel at the Aspen Security Forum.

“I think we are still at war,” Ashcroft said bluntly. “I don’t know if I will be able to be sure to say when we will be able to say we are not at war. But as long as they are continuing to hit us and allege that they are at war, I think we can.”

In response, Mudd directly challenged Ashcroft.

“I don't agree, by the way, that we are at war,” the author said.

Instead, Mudd argued, that we have a dynamic and ever-changing face of terrorism that may prove to be difficult to squash completely.

But because of two wars in Iraq and Afghanistan, he said the threat of terrorism is not nearly what it used to be.

“In 2001, we would have said terrorism is a potential threat to American families,” Mudd said. “And I would say today, that is not true."

Mudd added that he believed it was a false distinction.

“I have 10 nieces and nephews, I don't think I have ever talked to them about terrorism. … The gang problems in the city that I live in, Memphis, Tennessee, are outrageous. People in this country, partly because there is a racial divide in this country, don’t care. But four people die in an attack and this is a national disaster, I don’t get it,” he said.

Ashcroft, turning to Mudd, pushed back.

"I think one of the most dangerous things we can do is to say that it (terrorism) is not a threat,” Ashcroft said. “There were some families in Boston that found it a threat."

According to Gallup polling, the focus on terrorism among American families has changed over the past 12 years.

After the al Qaeda attacks on New York and Washington, nearly six-in-10 people said they were very or somewhat worried that someone in their family would be a victim of terrorism.

Since then, that sentiment has eased. In 2004, the number of those very or somewhat concerned fell to 28% and mainly has hovered around the 40% percent range since.

In May, President Barack Obama recast the U.S. fight against terrorism as no longer a "boundless global war" but a targeted effort to dismantle specific extremist networks.

He said America was at a "crossroads" and should no longer see it as a "self-defeating" perpetual fight, but one that must at some point end, "like all wars."

(Merica reported from Washington, Henry and Labott from the Aspen Security Forum)