VIENNA—A weekend marathon of talks between major oil producers failed to finalize plans to implement an output cut, threatening the viability of an agreement reached last month to reduce production by as much as 2%.

The discussions held at the headquarters of the Organization of the Petroleum Exporting Countries were supposed to pave the way for a detailed proposal on how to cut production by between 200,000 and 700,000 barrels a day, about 1% to 2%.

The...