SEOUL—A South Korean regulator said it would fine Qualcomm Inc. about $853 million for alleged antitrust violations, the highest such penalty handed to an individual company here, as the U.S. chip maker faces global scrutiny over its patent-licensing business.

After a three-year investigation, the Korea Fair Trade Commission said on Wednesday that Qualcomm breached antitrust law by limiting competing chip makers’ access to its patents. The company also forced mobile-phone manufacturers into unfair license agreements by refusing to supply crucial phone chips to those that disagreed with its terms, the regulator said.

The chip supplier used its market position as a leveraging tool in its negotiations with mobile-phone makers to force them to accept unfair conditions, said Shin Young-son, the commission’s secretary-general.

San Diego-based Qualcomm said it would contest the decision, rejecting the suggestion that it had hindered other chip suppliers’ ability to sell. It said its licensing practices have existed in Korea and world-wide for decades, and that the regulator hadn’t questioned them in the past. The practice of licensing at the device level is an industrywide norm, said Don Rosenberg, the firm’s general counsel.

“It’s a false reality they’ve created of what this industry looks like, what its history has been, what our role has been,” Mr. Rosenberg said.