Top U.S. intelligence officials gathered in the White House Situation Room in March to debate a controversial proposal. Counterterrorism officials wanted to create a government dragnet, sweeping up millions of records about U.S. citizens—even people suspected of no crime.

Not everyone was on board. "This is a sea change in the way that the government interacts with the general public," Mary Ellen Callahan, chief privacy officer of the Department of Homeland Security, argued in the meeting, according to people familiar with the discussions.

A week later, the attorney general signed the changes into effect.

Through Freedom of Information Act requests and interviews with officials at numerous agencies, The Wall Street Journal has reconstructed the clash over the counterterrorism program within the administration of President Barack Obama. The debate was a confrontation between some who viewed it as a matter of efficiency—how long to keep data, for instance, or where it should be stored—and others who saw it as granting authority for unprecedented government surveillance of U.S. citizens.

The rules now allow the little-known National Counterterrorism Center to examine the government files of U.S. citizens for possible criminal behavior, even if there is no reason to suspect them. That is a departure from past practice, which barred the agency from storing information about ordinary Americans unless a person was a terror suspect or related to an investigation.