Hedge-fund group SAC Capital Advisors LP and federal prosecutors have agreed in principle on a penalty exceeding $1 billion in a potential criminal settlement that would be the largest ever for an insider-trading case, according to people familiar with the matter.

The payment by SAC, run by star manager Steven A. Cohen, is expected to be roughly $1.2 billion to $1.4 billion, according to these people.

The...