The company that owns the National Enquirer, a backer of Donald Trump, agreed to pay $150,000 to a former Playboy centerfold model for her story of an affair a decade ago with the Republican presidential nominee, but then didn’t publish it, according to documents reviewed by The Wall Street Journal and people familiar with the matter.

The tabloid-newspaper publisher reached an agreement in early August with Karen McDougal, the 1998 Playmate of the Year. American Media Inc., which owns the Enquirer, hasn’t published anything...