NEWTON, Mass.—Rep. Barney Frank (D., Mass.), an architect of last year's landmark financial-regulation overhaul, said Monday he wouldn't seek a 17th term, adding to a list of Democratic retirements that could hurt the party's already slim chances of retaking the House next year.

The decision to leave Congress by Mr. Frank, the most high-profile Democrat to do so since 2010, will deprive liberals of one of their most outspoken champions and conservatives of a national political...