A Transportation Security Administration (TSA) domestic surveillance program is drawing criticism for tracking information from U.S. citizens not suspected of any crimes. The program has drawn criticism within the agency, according to CBS News sources.

The "Quiet Skies" program, as first reported in the Boston Globe, uses a computer algorithm to spot flyers -- including American citizens -- whose travel and activities match with current threat intelligence. Those flyers are reviewed by a human team that vets the information and if something in the flyer's background raises concerns, that flyer may be surveilled by Federal Air Marshals.

The TSA said in a statement to CBS News the program's primary purpose is to "ensure passengers and flight crew are protected during air travel," adding that it doesn't take race and religion into account and "is not intended to surveil ordinary Americans with routine reviews and active management via legal, privacy and civil rights and liberties offices, it is a practical method of keeping another act of terrorism from occurring at 30,000 feet."

Get Breaking News Delivered to Your Inbox

Officials familiar with the program said "Quiet Skies" complies with security constraints and privacy requirements. One source said the "Quiet Skies" team has to be prepared to justify why an individual is being questioned if they are challenged.

If a flyer is on the list for a certain amount of time and travels without incident, they are automatically removed from the list. Some in the security field believe this creates a certain level of risk because if the person being surveilled is a sleeper agent, they may succeed in flying under the radar. But congressional concerns about redress prompted the policy.

The "Quiet Skies" program has been in existence since 2010 and the TSA says Congress and the airlines have been briefed on the program over the last 12 to 18 months.

A U.S. official said there is "very small list" of flyers and is likely smaller than the number on the TSA's "95 list" that tracks flyers who have a pattern of bad behavior at airports. Fewer than 50 people are currently on that list.

At least one Air Marshal has filed a complaint the Department of Homeland Security Inspector General's Office about the program, believing it is a waste of time, money and manpower.

Kris Van Cleave contributed to this report.