Get breaking news alerts and special reports. The news and stories that matter, delivered weekday mornings.

The Secret Service apologized Wednesday to Rep. Jason Chaffetz, a prominent critic of the agency, for violating federal privacy law by improperly accessing sensitive personal information about him dozens of times in little more than a single week.

Chaffetz, R-Utah — who has aggressively pursued allegations of Secret Service misconduct as chairman of the House Committee on Oversight and Government Reform — called the unauthorized retrieval of his unsuccessful application to join the Secret Service in 2003 "a tactic designed to intimidate and embarrass me."

"It's scary to think about all the possible dangers in having your personal information exposed," Chaffetz said in a statement late Wednesday afternoon. "The work of the committee, however, will continue. I remain undeterred in conducting proper and rigorous oversight."

Later, in a short interview with NBC News, the congressman reiterated how chilling the report was.

"It's a bit scary. If they would do this to me, I just, I shuddered to think what they might be doing to other people," he said. "I'd like to tell you how tough I am, but it's scary, and it's intimidating, and I will continue to investigate the Secret Service and others, but this should have never ever happened."

The improper handling of Chaffetz's personal information was confirmed Wednesday in a 29-page report by the inspector general's office of the Department of Homeland Security, which includes the Secret Service.

Related: Read the Entire Inspector General's Report (PDF)

The report discloses an email in which Ed Lowery, the agency's assistant director for training, wrote that "some information that he might be find embarrassing needs to get out."

The report says Roth told investigators he believed at the time that such disclosures would be "inappropriate" and explained that his email was a human reaction by someone frustrated by "stress and ... anger."

Let our news meet your inbox. The news and stories that matters, delivered weekday mornings. This site is protected by recaptcha

The report specifically said Secret Service Director Joseph Clancy, whom President Barack Obama appointed in February to clean up a long series of scandals at the agency, was not aware of the improper behavior.

When these allegations first surfaced in April, Homeland Security Secretary Jeh Johnson referred them to the inspector general and apologized to Chaffetz. The report published Wednesday details the improper behavior and harshly criticizes the Secret Service for violating the Privacy Act of 1974.

On Wednesday, the Secret Service issued a statement apologizing to Chaffetz for "this wholly avoidable and embarrassing misconduct" and promising that those responsible would be held accountable.

The inspector general's report found that more than 40 Secret Service employees accessed Chaffetz's 2003 job application a total of more than 60 times, even though they had "no official need to query Chairman Chaffetz' name."

The first inquiry was made March 24 — just 18 minutes after Chaffetz opened a committee hearing into alleged Secret Service misconduct, the report says.

Related: Long List of Breaches and Scandals for Secret Service Under Obama

Over the next nine days, the job application was accessed about 60 more times, the report says, by employees in the public relations office, the countersurveillance division, the division overseeing protection for former President Bill Clinton, the training division and at least 15 domestic and overseas field offices.

Only four of the inquiries could reasonably be considered justified, it says.

At least one employee acknowledged to inspector general investigators that he disclosed the protected information to a reporter for The Washington Post, according to the report, and in a further violation of the Privacy Act, some of the employees disseminated Chaffetz's data through the Secret Service email system, it says.

"This episode reflects an obvious lack of care on the part of Secret Service personnel as to the sensitivity of the information entrusted to them," the report concluded. "It doesn't take a lawyer explaining the nuances of the Privacy Act to know that the conduct that occurred here — by dozens of agents in every part of the agency — was wrong."

Rep. Jason Chaffetz, R-Utah Mark Wilson / Getty Images

Johnson, the homeland security secretary, said Wednesday that he had repeated his apology to Chaffetz and stressed that "activities like those described in the report must not, and will not, be tolerated."

But the report angered Chaffetz and other members of his committee, Democratic and Republican alike.

"It is intimidating," Chaffetz said. "Certain lines should never be crossed. The unauthorized access and distribution of my personal information crossed that line."

Rep. Mark Meadows, R-North Carolina, chairman of the committee's panel on government operations, said the report revealed "woeful abuse of power by a government agency tasked with one of the most important jobs in the country: protecting our nation's president."

And Rep. Elijah Cummings of Maryland, the committee's top Democrat, said in a statement: "I believe in fundamental fairness, and those who are unwilling or unable to meet the highest of ethical standards should not be a part of the Secret Service."