Accounting firm Deloitte & Touche LLP agreed Wednesday to pay $149.5 million to settle Justice Department allegations that it failed to head off a huge fraud at a mortgage company that collapsed during the financial crisis.

Deloitte should have detected misconduct at Taylor Bean & Whitaker Mortgage Corp., for which it was the outside auditor, but the firm “knowingly deviated” from auditing standards and thus allowed fraud at the company to continue until Taylor Bean filed for bankruptcy in 2009, the Justice Department...