WASHINGTON (CNN) -- Congress is looking into allegations that National Security Agency linguists have been eavesdropping on Americans abroad.

Government linguists say the U.S. eavesdropped on Americans, including military officers serving in Iraq.

The congressional oversight committees said Thursday that the Americans targeted included military officers in Iraq who called friends and family in the United States.

The allegations were made by two former military intercept operators on a television news report Thursday evening.

A terrorist surveillance program instituted by the Bush administration allows the intelligence community to monitor phone calls between the United States and overseas without a court order -- as long as one party to the call is a terror suspect.

Adrienne Kinne, a former U.S. Army Reserves Arab linguist, told ABC News the NSA was listening to the phone calls of U.S. military officers, journalists and aid workers overseas who were talking about "personal, private things with Americans who are not in any way, shape or form associated with anything to do with terrorism."

David Murfee Faulk, a former U.S. Navy Arab linguist, said in the news report that he and his colleagues were listening to the conversations of military officers in Iraq who were talking with their spouses or girlfriends in the United States.

According to Faulk, they would often share the contents of some of the more salacious calls stored on their computers, listening to what he called "phone sex" and "pillow talk."

Both Kinne and Faulk worked at the NSA listening facility at Fort Gordon, Georgia. They told ABC that when linguists complained to supervisors about eavesdropping on personal conversations, they were ordered to continue transcribing the calls.

NSA spokeswoman Judith Emmel said the agency's Inspector General has investigated some of the allegations and found them "unsubstantiated." Other accusations are still being looked at, she said.

The NSA operates in "strict accordance with U.S. laws and regulations," she said. "Any allegation of wrongdoing by employees is thoroughly investigated" and if misconduct is discovered, "we take swift and certain remedial action."

CIA Director Mike Hayden, who was the head of the NSA when the terrorist surveillance program began, has always maintained that private conversations of Americans are not intercepted and if it should happen inadvertently, the name is removed from the record.

"At NSA, the law was followed assiduously," said Hayden's spokesman, Mark Mansfeld. "The notion that Gen. Hayden sanctioned or tolerated illegalities of any sort is ridiculous on its face."

Author Jim Bamford was the first to interview the two former NSA linguists for his new book, "The Shadow Factory," which will be published next week. Bamford told CNN the accounts from the whistle-blowers demonstrate the NSA was listening to the private conversations of Americans, transcribing them and keeping them.

"They don't delete them," he said.

Bamford has written two other books on the NSA and was a party to an unsuccessful ACLU lawsuit challenging the constitutionality of the Terrorist Surveillance Program.

The ACLU issued a statement on Thursday saying the allegations show that the government "misled the American public about the scope of its surveillance activities."

The ABC report "is an indictment not only of the Bush administration, but of all of those political leaders, Democratic and Republican, who have been saying that the executive branch can be trusted with surveillance powers that are essentially unchecked," said ACLU official Jamell Jaffer.

Senate Intelligence Committee Chairman Jay Rockefeller, D-West Virginia, called the allegations "extremely disturbing."

"Anytime there is an allegation regarding abuse of the privacy and civil liberties of Americans, it is a very serious matter," said Rockefeller, adding that his committee is prepared to take whatever action is necessary to ensure the government is following the strict procedures to protect U.S. citizens against unwarranted surveillance.

A House Intelligence Committee spokesman said the panel has been in contact with the NSA and is awaiting the agency's response.

All About National Security Agency • Central Intelligence Agency