The European Union hit Alphabet Inc.’s Google with a record $5 billion antitrust fine and ordered changes that could reshape the business model underlying its biggest growth engine: mobile phones.

In another rebuke to one of Silicon Valley’s tech giants, the bloc’s antitrust regulator found Wednesday that Google had abused the dominance of its Android operating system, which runs more than 80% of the world’s smartphones, to promote and entrench the company’s cash-cow search engine.

The...