The Federal Communications Commission on Wednesday said it plans to fine AT&T Inc. T -1.55% $100 million for allegedly deceiving millions of smartphone customers about unlimited wireless data plans.

The FCC alleges AT&T sold consumers data plans advertised as unlimited, then capped data speeds for those subscribers after they used five gigabytes of data within a billing cycle. Those capped speeds, the agency said, were more than 20 times slower than the normal network speeds advertised by AT&T, and hurt consumers’ ability to access the Internet or use applications.

AT&T doesn’t dispute that it slowed data speeds for some consumers with unlimited plans, but a spokesman said the FCC previously endorsed the practice as legitimate and pointed to a disclosure on the company’s website as evidence that AT&T had been candid with customers. He said AT&T plans on “vigorously disputing the FCC’s assertions.”

Wireless carriers started to offer unlimited plans during the previous decade to encourage subscribers to surf the Internet on their smartphones. But as smartphones became more popular and users began watching more video on sites like YouTube and posting more photos to social-media sites like Facebook, the economics of unlimited plans soured. The surge in data traffic required expensive network upgrades, and unlimited plans prevented carriers from collecting more money as usage rose.

Carriers responded by shifting to plans where subscribers paid more for bigger buckets of data. AT&T and Verizon Communications Inc. stopped offering unlimited plans to new subscribers years ago. Their smaller national rivals, T-Mobile US Inc. and Sprint Corp., still offer such plans, but their executives have indicated they might not last forever.

Sprint said it had been slowing its heaviest users’ data speeds as necessary to relieve congestion on its network, but stopped Friday when the FCC’s new net-neutrality rules went into effect.

After AT&T stopped offering unlimited data plans in 2010 it allowed millions of customers that already had the plans to keep them.

On Wednesday, senior FCC officials indicated on a conference call that more carriers might face penalties if they advertised unlimited data plans but capped the amount of data consumers could use at full speed.

The FCC skirmished with Verizon last year about throttling speeds for subscribers on unlimited data plans. Verizon had defended its policy as a reasonable effort to manage its network but ultimately dropped its plan to reduce data speeds in congested areas under pressure from the FCC.

The Federal Trade Commission sued AT&T in October over allegations similar to the FCC’s; that lawsuit is still winding its way through federal court.

The actions are part of a new aggressiveness by the agencies in their regulation of wireless carriers, as these services have become increasingly vital in consumers’ lives. Mobile broadband providers can expect this regulatory trend to continue: The FCC reclassified the services as common carriers in February as part of its net-neutrality rules, expanding the agency’s authority over the industry.

The rules expand the FCC’s authority to regulate wireless broadband providers and practices like data capping. They also for the first time ban the providers from blocking, slowing down or speeding up access to specific websites. Wireless carriers are challenging the rules in court, arguing they create too much uncertainty and give the FCC the power to veto new business plans.

“Consumers deserve to get what they pay for,” FCC Chairman Tom Wheeler said in a statement. “Broadband providers must be up front and transparent about the services they provide. The FCC will not stand idly by while consumers are deceived by misleading marketing materials and insufficient disclosure.”

A senior FCC official said the agency cooperated with the FTC on the AT&T investigation, but the agencies acted separately in their enforcement actions. The FTC lawsuit, filed in federal court in San Francisco, seeks compensation for customers and to bar AT&T from misleading consumers with its advertising. The FCC fine, if upheld, would go to the U.S. Treasury.

The FCC says AT&T violated the transparency rule passed as part of its 2010 open-Internet rules by labeling its plans as unlimited. A federal court struck down most of those rules last January, but the transparency rule was upheld.

The senior FCC official said AT&T made billions of dollars from unlimited data plans that weren’t, in fact, unlimited. An AT&T spokesman said that without those plans, customers would have had to pay far more in overage charges based on their data usage.

The proposed fine against AT&T is the FCC’s first enforcement case under the transparency rule. It is the largest proposed fine in FCC history, according to the agency. FCC officials said part of its purpose was to deter future violations.

AT&T plans to contest the penalty, first during an administrative hearing in the FCC and, if necessary, in court.

The fine could be reduced as part of the FCC’s administrative process, according to the senior official. He said the agency will take into account whether AT&T corrects statements the agency believes are misleading or whether it offers consumers the ability to opt out of the unlimited plans without facing any penalties.

The FCC will also order AT&T to explain the measures it will take to make consumers whole.

The FCC said it got thousands of complaints from customers with AT&T’s unlimited data plan who said they were surprised and misled by AT&T throttling their data speeds.

“Unlimited means unlimited,” said FCC Enforcement Bureau Chief Travis LeBlanc. “As today’s action demonstrates, the commission is committed to holding accountable those broadband providers who fail to be fully transparent about data limits.”

Republican FCC Commissioner Ajit Pai argued in a dissenting statement that AT&T informed customers repeatedly that they may face slower speeds after using a certain amount of data. Several AT&T customers confirmed receiving text message alerts when they were close to the five-gigabyte threshold.

“This enforcement action only confirms my concern that the Internet is now governed not by engineers and innovators but by regulators and lawyers,” Mr. Pai said.

Write to Gautham Nagesh at gautham.nagesh@wsj.com and Thomas Gryta at thomas.gryta@wsj.com