SEJONG, South Korea/SAN FRANCISCO (Reuters) - South Korea's antitrust regulator fined Qualcomm Inc QCOM.O 1.03 trillion won ($854 million) for what it called unfair business practices in patent licensing and modem chip sales, a decision the U.S. chipmaker said it will challenge in court.

The fine, the largest ever levied in South Korea, marks the latest antitrust setback for Qualcomm’s most profitable business of licensing wireless patents to the mobile industry, at a time when the business is facing headwinds from a cooling smartphone market. Shares in Qualcomm fell 1.7 percent on Nasdaq.

The Korea Fair Trade Commission (KFTC) ruled on Wednesday Qualcomm abused its dominant market position and forced handset makers to pay royalties for an unnecessarily broad set of patents as part of sales of its modem chips.

Qualcomm also restricted competition by refusing or limiting licensing of its standard essential patents related to modem chips to rival chipmakers such as Intel Corp INTC.O, Samsung Electronics Co Ltd 005930.KS and MediaTek Inc 2454.TW, the regulator said, hindering their sales and leaving their products vulnerable to lawsuits.

The regulator ordered Qualcomm to negotiate in good faith with rival chipmakers on patent licensing and renegotiate chip supply agreements with handset makers if requested -- measures that would affect the U.S. firm's dealings with major tech companies including Apple Inc AAPL.O, Intel, Samsung and Huawei Technologies Co Ltd [HWT.UL] if upheld.

The KFTC said it began its investigations into Qualcomm’s practices in 2014 following complaints from industry participants, but did not name specific companies.

Foreign companies including Apple, Intel, MediaTek and Huawei expressed their views during the regulator’s deliberation process, KFTC Secretary General Shin Young-son told a media briefing in the country’s administrative capital.

"We investigated and decided on these actions because Qualcomm's actions limit overall competition," Shin said, adding that the ruling was not about protecting domestic companies such as Samsung and LG Electronics Inc 066570.KS.

Qualcomm said it will file for an immediate stay of the corrective order and appeal the decision to the Seoul High Court. The firm will also appeal the amount of the fine and the method used to calculate it.

A Qualcomm sign is pictured in front of one of its many buildings in San Diego, California November 5, 2014. REUTERS/Mike Blake/File Photo

“Qualcomm strongly disagrees with the KFTC’s announced decision,” it said in a statement.

The decision in Seoul could mean lower costs for mobile phone makers and others in the tech supply chain who source Qualcomm chips, said a Taiwanese industry executive, declining to be named due to the sensitivity of the matter.

IN REGULATORY CROSSHAIRS

The fine is the latest in a series of antitrust rulings and investigations faced by Qualcomm from regulators across the globe. In February 2015, Qualcomm paid a $975 million fine in China following a 14-month probe, while the European Union in December 2015 accused it of abusing its market power to thwart rivals.

Stacy Rasgon, an analyst with AB Bernstein, said the fine itself was large but also said the KFTC’s orders for Qualcomm to alter its business practices have bigger future implications for the chipmaker.

A major challenge for Qualcomm, he said, would emerge if the ruling forces the company to license patents for some of its chips to rivals such as Intel, which has been competing hard to land its modem chips in mobile phones.

The dispute could take years to play out in South Korean courts, he said.

The KFTC fined Qualcomm 273 billion won in 2009 for abusing its dominant position in CDMA modem chips, which were then used in handsets made by Samsung and LG.

Regulators in other jurisdictions, including the United States and Taiwan, are also investigating Qualcomm. The KFTC move could serve as a benchmark for some of them.

“Korea’s decision will serve as an important reference point for us,” Chiu Yung-ho, vice chairman and spokesman for Taiwan’s Fair Trade Commission, told Reuters. He gave no timeline on when Taiwan’s own year-long investigation would be concluded.

Even if the Korean decision sets a precedent for regulatory proceedings in other countries, Qualcomm's move to purchase chipmaker NXP Semiconductors NXPI.O for $38 billion cuts its reliance on mobile phones, said Bill Kreher, an analyst with Edward Jones. NXP takes Qualcomm into the fast-growing market for automotive chips and puts it into the chip manufacturing business.

“The strategy of diversifying the company’s business lines may prove to be the most prudent strategy of all,” Kreher said. “Since Qualcomm has moved its dependence from royalties, further decisions from other countries such as the U.S. in the future would not impact the company much, relative to in the past.”

Shares of Qualcomm were down $1.16 to $66.09 in early afternoon trade.