What started as a scandal over the IRS's targeting of conservative groups has broadened, with lawmakers and other critics now questioning whether other kinds of organizations were unfairly flagged for additional scrutiny.

Rep. Sam Graves, R-Mo., chairman of the House Small Business Committee, wrote a letter to Acting IRS Commissioner Daniel Werfel on Friday asking a series of questions about the agency's audit practices for small businesses.

He made no specific allegation, but said that lawmakers' investigations to date prompted the letter.

"(Congressional) investigations have only raised more questions as to the extent these practices may have extended beyond conservative groups," Graves wrote.

Indeed, the scope of the IRS' heavy auditing and scrutiny appears to go beyond Tea Party groups.

A report by the IRS' Taxpayer Advocate Service found the IRS improperly targeted adoptive families -- flagging for further review 90 percent of those who claimed the adoption tax credit in 2012. Further, nearly 70 percent of them endured at least a partial audit of their returns.

By contrast, just one percent of all returns are audited.

The report fueled concerns that the IRS is unfairly lumping categories of filers together for additional review, in turn scrutinizing small-fry individuals and groups while ignoring bigger and richer organizations.

The allegations have been cascading in from across the country as Congress investigates. First, religious groups claimed they had been subjected to additional scrutiny as well. In one high-profile letter from a Christian leader, Franklin Graham alleged that the Billy Graham Evangelistic Association and one other group had been flagged in 2012 for additional review. Graham claimed the reviews were not "justifiable."

The Washington Free Beacon also reported Thursday that at least five pro-Israel organizations were audited by the IRS following a campaign by White House-aligned group to challenge their tax-exempt status.

Those pro-Israel groups came under criticism after they challenged the administration's stance against Israeli settlement construction.

The Hill also reported on Friday that aid groups trying to help those caught in the Syrian civil war are facing long delays in their applications for tax-exempt status.

This is similar to the problem faced by Tea Party and other groups, who had their applications tied up for a roughly 18-month stretch.

Current and former IRS officials have said that while the practice of subjecting those groups to additional scrutiny was inappropriate, it was not politically motivated. Rather, they say the criteria were developed to deal with an influx of cases. Others have suggested it is up to Congress to clarify the rules for the IRS.

An IRS official last week defended the scrutiny of adoptive families, arguing that the agency is obligated to make sure claims are accurate.

"The IRS implemented the adoption credit program with an approach that balanced the objective of paying legitimate credits in a timely manner with that of ensuring that claims were accurate," IRS spokeswoman Michelle Eldridge said in a statement to FoxNews.com. "Our experiences and lessons learned from other refundable credits taught us that high dollar credits have high risk and the potential for fraud. We must ensure delivery of the credit to those entitled while protecting the government's interest in minimizing exposure to fraud."

In 2011, more than 51,000 taxpayers claimed $668 million from the adoption tax credit. This was after the health care law increased the maximum credit per child.

However, the extra audits only found $11 million in improper claims from the 2011 year.

The taxpayer advocate report said the practice "caused significant harm to thousands of families who are selflessly trying to improve the lives of vulnerable children."

The IRS will be in the hot seat once again next week to answer questions on its targeting practices, as Congress plans at least two upcoming hearings on the scandal. Meanwhile, a recent poll showed Americans overwhelming want an outside investigator to be assigned to examine the IRS. The Justice Department already has launched its own probe.