LUXEMBOURG— Alphabet Inc.’s appeal against a multibillion-dollar fine for alleged anticompetitive behavior by its Google unit risks backfiring after a European Union court floated the prospect of increasing the fine, rather than scrapping it.

In a surprise twist Friday at the end of a three-day hearing, one of five judges on the panel said the EU’s General Court has the power to increase the €2.4 billion ($2.6 billion) fine, levied in 2017, if it finds that the sum was insufficient to deter the company from further anticompetitive behavior.

“The fine of €2.4 billion was described as eye-catching, but it is a small amount of cash in your hands,” Judge Colm Mac Eochaidh said in court. “Did that level of fine deter you from repeating your behavior?” he asked Google’s counsel.

Mr. Mac Eochaidh said that as recently as this year, travel search engines including Expedia and TripAdvisor had complained to the European Commission, the EU’s executive arm, about Google’s alleged anticompetitive behavior.

Christopher Thomas, a counsel for Google, dismissed the idea that the fine was warranted and said the company takes the entire antitrust process “with extreme seriousness.” Google disputes the findings of the commission that it had willingly or negligently squeezed competitors out of its shopping searches.