When a team of forensic accountants began sifting through refunds issued by a national call center, something didn’t add up: There were too many fours in the data. And it was up to the accountants to figure out why.

Until recently, such a subtle anomaly might have slipped by unnoticed. But with employee fraud costing the country an estimated $300 billion a year, forensic accountants are increasingly wielding mathematical weapons to catch cheats.

“The...