CANBERRA, Australia—Tech giants Facebook Inc. and Alphabet Inc.’s Google could have their secretive algorithms policed by a beefed-up watchdog, under what Australia describes as world-first limits to the power that they wield over news and advertising markets.

The recommended changes—which include strengthening privacy safeguards with steep penalties of up to 10% of annual domestic turnover for the misuse of data—are listed in a report by the national competition regulator. It conducted a year-and-a-half investigation into...