WASHINGTON, July 29, 2009 -- Attorney General Eric Holder told ABC News in an exclusive interview today that he is increasingly concerned about Americans becoming radicalized and turning to terrorism.

"I mean, that's one of the things that's particularly troubling: This whole notion of radicalization of Americans," Holder told ABC News during an interview in his SUV as his motorcade brought him from home to work. "Leaving this country and going to different parts of the world and then coming back, all, again, in aim of doing harm to the American people, is a great concern."

Holder said the ever-changing threat of terror and the pressure to keep up with it weighs heavily on his mind as he tries to ensure that the government has done all it can to anticipate the moves of an unpredictable enemy.

"In some ways it's the most sobering part of the day," Holder said of his morning intelligence briefing, in which he gets the latest report on the landscape of "the organizations, the people who are bound and determined to do harm to our nation."

Recent events, such as the arrests of alleged members of a home-grown terror cell in North Carolina, the return of several Somali-American men to their home country under questionable circumstances and the filing of charges against a New York man who allegedly received al Qaeda training in Pakistan and took part in a rocket attack against U.S. forces, bring the threats to national security into sharp focus.

"But, you know, in the hierarchy of things, it's hard to figure out how to prioritize these things in some ways," he said. "The constant scream of threats, the kind of things you have to be aware about, the whole notion of radicalization is something that didn't loom as large a few months ago ... as it does now. And that's the shifting nature of threats that keeps you up at night."

He noted, however, that the Bush administration "left us an infrastructure that I think is very good," and that national security officials are constantly striving to put the country in a safer position.

"The American people would be surprised by the depth of the threat, but also reassured to see the assets that have been deployed around the world," Holder said, adding that the United States interacts closely with its foreign partners.

Critical Decisions on Guantanamo, Investigation Into Bush Administration Top Holder's To-Do List

Yet, terrorism is just one of the major issues Holder has on his plate. ABC News spent the day with Holder, from a breakfast meeting on Capitol Hill, to his daily walk up the five flights of stairs to his office where he works as the nation's top law enforcement official.

He's just shy of six months into his job as the nation's top cop, and has already been tasked with overseeing the closure of the detention facility at Guantanamo Bay, Cuba, dealt with the continuing fallout from some unpopular Bush administration policies and jumped headfirst into the issue of race relations in America.

As the summer races by, Holder faces a series of critical decisions. The investigation into the Bush administration's policy on the detention and interrogation of suspected terrorists, Holder said is still a real possibility -- regardless of the political fallout.

According to a recent ABC News/Washington Post poll, 54 percent of Americans oppose such an investigation.

"We want to move forward. We don't want to look back in that regard," Holder said. "My obligation as the head of the Justice Department is to make sure that the laws are followed and to the extent that we find that laws were broken, to hold people accountable.

"I think the department, at least some of the people who worked, simply lost their way," he said, citing tactics like waterboarding, which he considers torture. "I will follow the facts and the law wherever it takes me."

Another issue weighing heavy on the attorney general is how best to shut down the detention camp at Guantanamo Bay.

"Guantanamo is a place that has served as a recruiting tool for those who want to do us harm," he said.

Holder declined to say how many detainees could face trial in the United States, but said U.S. attorney offices around the country have been reviewing the cases for possible prosecution in those districts.

"At this point, a significant number have been referred to the Justice Department for possible prosecution and those cases have been referred out [to] four or five U.S. attorneys' offices. And they're in the process right now of looking ... at where they can make cases against these individuals. I don't know what the number, the final number is going to be," he said.

Holder also said some of the detainees could be held indefinitely.

"The possibility exists that there could be people who are held in a preventative way under the laws of war," he said. "If that happens, we'd only do so by creating a system that had due process.

"I think that by closing Guantanamo, by prosecuting people, be it in Article III courts, or in military commissions, we will make the American people safer than they are now," Holder continued.

Holder Stresses Justice Department Independence

On Wednesday, Holder met with Senate Judiciary Committee Chairman Patrick Leahy, whose panel has oversight over the Justice Department.

The Vermont senator said Holder's job "is probably the toughest job there is, because it is the one part of the Cabinet where certainly you have to reflect the president, but you've got to be independent."

Leahy was highly critical of one of Holder's predecessors, Alberto Gonzales, who resigned under fire for allowing political ideology infiltrate the Justice Department. Gonzales had long been a friend and ally of George W. Bush, and critics said Gonzales operated the department as an arm of the White House.

Leahy acknowledged Holder's close ties to President Obama, but said his approach to the department is more in line with that of Robert Kennedy, who served in his brother John F. Kennedy's Cabinet. Recalling a time when he met Attorney General Kennedy, Leahy said he was struck by his independence from the White House.

"Of course he was as loyal to the president as anybody could be. He was the president's closest advisor, most trusted person. He was his brother. But he was independent as attorney general, [which] is probably the best model for being attorney general," Leahy said. "I think Eric Holder fits that model."

On the wall in Holder's office at the Justice Department hang portraits of several past attorneys general -- Kennedy is one of them.

Holder told ABC News that Kennedy serves as an "inspiration" for many reasons, but one that hits close to home.

"He demonstrated, I think, real courage, real sense of this place during the early '60s. I think about my late sister-in-law who integrated [the] University of Alabama," Holder said. His late sister-in-law, Vivian Malone Jones, was the first black graduate of the university.

"[Alabama Gov.] George Wallace [was] standing in the door, she was the young black student who was admitted on that day as a result of his [Robert Kennedy's] efforts; he pushed the Kennedy administration, I think, in the area of civil rights, he's a legend in this building," Holder said.

The main Justice Department building bears Robert Kennedy's name.

Holder: 'Race is a Tough Issue'

Holder's remarks in a February speech that the United States was "a nation of cowards" when it comes to race relations, spurred controversy. Though he said he stands by his comments, the attorney general also said he believes the country has made progress along the lines of race.

Holder Says Gates Arrest Was a 'Teachable Moment'

"Race is a tough issue. Wherever it has been found, whether in the United States or even in other countries, it's an issue that has divided us, I think, in the past. It's an issue that if unaddressed, I think, can divide us in the future. And what that speech was about," Holder said.

On the fallout surrounding the recent arrest of Harvard University professor Henry Louis Gates Jr., which has triggered a national uproar about the role of law enforcement and reignited a debate on race relations, Holder said he agreed with Obama, calling it a "teachable moment" for the nation to improve communication between different groups.

"Based on what I know of Professor Gates and what I've heard about Sgt. [James] Crowley, my guess is that they both think that they might have reacted, perhaps, a little differently and this situation might have -- might have been avoided," Holder said.

Though he declined to take sides, Holder, the nation's first African-American attorney general, said that as a young man, he, himself, had been profiled by police during a traffic stop -- an incident that left a lasting impression on him.

"I was a young college student driving from New York to Washington, stopped on a highway and told to open the trunk of my car, because the police officer told me he wanted to search it for weapons. I was at a rest stop. And, you know, I was a college kid. I didn't know quite what my rights were," Holder said.

"People walked by as I was opening this trunk and the officer was looking in there," he said. "And I remember, as I got back in the car and continued on my journey how humiliated I felt, how angry I got."

Holder said that the incident has been an impetus for his work to counter racial profiling at the Justice Department and work toward establishing trust between communities. And while law enforcement is far from being colorblind, he said that agency is pushing for change.

"I think we've made a lot of progress in this country," he said. "We still have a ways to go. But I'm pretty confident that the America of the 21st century can be all that we want it to be."