The Securities and Exchange Commission split in its decision to settle its landmark lawsuit against Goldman Sachs, The Wall Street Journal has learned, in a dispute over the agency's move to levy a $550 million fine even after diluting its fraud allegations against the giant bank.

The 3-2 decision on party lines Thursday came after a 30-minute closed-door session where the SEC's two Republican commissioners voted against settling, said people familiar with the matter. Mary Schapiro, the SEC chairman appointed by President...