Vice Media Group saw problems ahead for its business in 2017 when the new-media company set out to raise more money.

The priority for co-founder and then-chief executive Shane Smith was to secure an eye-popping valuation, some people close to the company say. He got his wish when private-equity firm TPG invested $450 million, valuing Vice at $5.7 billion, far higher than any Vice rivals.

But...