Saudi Arabia’s deputy crown prince sent his energy minister to an OPEC meeting last month with a difficult mission: Make a deal with rival Iran but don’t compromise the kingdom’s ability to fight for oil-market share, people familiar with the matter said.

The directive was a departure for Deputy Crown Prince Mohammed bin Salman, the powerful 31-year-old son of King Salman who is prosecuting Saudi Arabia’s war in Yemen against Iran-backed rebels. Prince Mohammed scuttled previous attempts at oil-production deals with the Organization...