WASHINGTON—The Justice Department on Wednesday accused the government's largest private security background check contractor of defrauding the country of millions of dollars by methodically filing more than 660,000 flawed background investigations—40% of the cases it sent to the government over a four-year period.

Prosecutors accused former top US Investigations Services LLC executives of directing improper practices that became a subject of internal jokes among company officials who helped secure millions of dollars in bonuses...