One sunny afternoon in August on a yacht off the Red Sea coast, Saudi Crown Prince Mohammed bin Salman and Vice Media Executive Chairman Shane Smith discussed an unlikely collaboration.

A Saudi government-controlled company had already hired Vice to produce documentaries on social reforms in the ultraconservative kingdom. The new proposal would elevate relations to a joint venture, similar to Saudi pairings with other Western media outlets, according to people briefed on the meeting.

Prince...