Consumers have issues with their mortgages, credit reports, credit cards and bank accounts. But they hate their debt collectors the most.

The Consumer Financial Protection Bureau recently published a report on the most common consumer complaints Americans submit. Debt collection and mortgage-related complaints make up about half of the 1.2 million complaints the CFPB has received since it started accepting complaints in July 2011, the Bureau said in the report.

Debt collection was the most-complained about issue; since the CFPB started accepting debt-collection complaints in July 2013, consumers have filed some 316,810 of them. The CFPB started accepting mortgage-related complaints in December 2011, and since then has received 272,153 of those.

The high volume of debt complaints is likely driven by the aggressive and persistent tactics collectors use to get their funds back, said Matt Schulz, a senior industry analyst for the credit cards website CreditCards.com. “What makes it even tougher for consumers is that most of them don’t know their rights and what the rules are in terms of what debt collectors can do, and it makes it a really confusing thing,” he said.

How you teach a computer to drive like a human

Debt collectors must follow the rules of the Fair Debt Collection Practices Act, which was signed into law in the 1970’s and prohibits actions including collecting any amount that is not “expressly” authorized by the agreement that created the debt.

Despite those federal protections, debt collectors are always finding new — and more intrusive — ways of tracking down borrowers. Collectors over time have developed high-tech ways to find consumers and contact them, including using virtual collections agent “avatars” in emails to consumers and by developing programs that analyze calls between debtors and collectors to determine what types of conversations can best lead to payment.

The Supreme Court in June ruled not to expand a federal law that would protect consumers from harassment and threats. The Court decided the class-action suit consumers had brought against a collector did not fall under the court’s purview.

Some 30% of online payday loan borrowers report being threatened by a lender or debt collector, according to Pew Charitable Trusts, a nonprofit based in Philadelphia that researchers consumer issues.

That said, many Americans do owe money to their creditors. Household debt levels totaled $12.73 trillion in March 2017, surpassing the 2008 peak of $12.68 trillion, largely because of student loans, credit-card bills and auto loans.

When the CFPB receives a complaint, it does not necessarily mean the consumer’s debt will be erased. But the Bureau is able to spot problems including debt-collection scams and identify illegal practices. Consumers can submit their complaints to the CFPB online.

For those who are struggling with debt or an aggressive debt collector, Schulz suggested those who can afford to hire lawyers may want to pursue that option. For the many who cannot, he also said credit counselors can provide guidance, as long as they are legitimate and accredited counselors.