BRUSSELS—The European Union has proposed a new international court system that would settle disputes between investors and national governments, and could help defuse tensions over a sweeping trade deal with the U.S.

The plan, anticipated by an EU concept paper in May, would replace an existing dispute-resolution mechanism that has been sharply criticized by top EU officials and threatened to undermine a planned trans-Atlantic free-trade deal. Campaigners claim that the current system constrains governments and leaves policy...