TRENTON, N.J. — New Jersey is seeking more than $640 million from Uber in taxes and penalties, saying the ride-hailing company misclassified its drivers as independent contractors.

The decision is the latest setback for Uber and other companies in the so-called “gig economy” that rely heavily on contract labor to deliver the services at the heart of their popular apps.

Worker advocates say that job classification hurts the laborers and the states where they live, which miss out on tax revenues.

New Jersey’s labor department told Uber it, along with its subsidiary Rasier, owes $523 million in overdue taxes form the last four years and is also facing fines and interest of $119 million, according to letters from the department that were first reported Thursday by Bloomberg Law.

Uber disputed the state’s findings. “We are challenging this preliminary but incorrect determination, because drivers are independent contractors in New Jersey and elsewhere,” the San Francisco-based company said in a statement.

The move was hailed as a victory by those pushing for better working conditions for Uber’s drivers. Many of Uber’s workers are part-time, but others work long hours and rely on ride-hailing as their sole source of income.

“I have clients who are Uber drivers that are sleeping in their cars because they cannot afford the basic necessities, they can’t afford a place to live,” said Shannon Liss-Riordan, partner at Lichten & Liss-Riordan, who has represented drivers in the employment classification cases. “That’s not acceptable.”

New Jersey has among the strictest tests for determining whether a worker qualifies as an independent contractor. To be considered an independent contractor requires meeting all three prongs of a test set up under state law, including that the services performed fall outside the employer’s usual course of business.

Worker classification is important because taxpayers foot the bill for unemployment or disability insurance when independent contractors file for benefits, said Robert Asaro-Angelo, New Jersey’s labor commissioner, in a statement. “This defiance of the law puts honest business owners at an unfair disadvantage,” he said.

The New Jersey Department of Labor and Workforce Development declined to discuss the case or say whether it is seeking similar restitution from Lyft or other gig economy companies. Lyft, which is also based in San Francisco, declined to comment.

But Wall Street reacted, with stocks extending their downward spiral.

Uber’s shares closed Thursday at down 72 cents, or 2.7%, at $25.99. Lyft shed $1.38, or 3.2%, to $41.92. Both stocks are 42% below their initial public offering prices from earlier this year.

The threat of classifying workers as employees is the single biggest worry among investors in companies such as Uber and Lyft, said Dan Ives, managing director at Wedbush Securities. Many thought the threat could materialize eventually but weren’t expecting it within months of the companies’ IPO.

“It adds the black clouds over Uber, Lyft and the overall gig economy, as their business models were built on contractors,” Ives said. “It’s an absolute nightmare scenario that continues to plague the gig economy stocks.”

California, which represents Uber and Lyft’s largest source of revenue, passed a law requiring companies such as Uber and Lyft to classify drivers as employees instead of independent contractors, which would give them protections such as minimum wage, health benefits and the ability to unionize. That law is set to go into effect Jan. 1, but Uber and Lyft have been pushing for a carve-out.