HONG KONG—Thousands of Bangladeshis took to the streets Monday to protest a weekend fire at a factory that made clothes for exports in which more than 100 workers were killed.

The factory supplied clothes for Hong Kong-based sourcing giant Li & Fung , which is a buyer for retailers such as Wal-Mart Stores Inc. Late Monday, Wal-Mart said the factory was no longer authorized to make clothes for the retailer, and that it had cut ties to a supplier that subcontracted with the factory without its authorization. The company declined to name the supplier.

Garment-workers' groups said workers were blocking the streets of a Dhaka suburb in a protest that turned violent at times. The workers threw stones at factories, smashed vehicles and blocked a major highway in the area. Some 200 factories were closed for the day after the protest erupted in Ashulia, the industrial zone just north of Dhaka where Saturday's deadly fire occurred.

The fire broke out late Saturday at an eight-story building of Tazreen Fashions Ltd., a subsidiary of Bangladesh's Tuba Group, an apparel factory that exports to the U.S. and Europe. In addition to making clothes for Li & Fung, the Bangladesh group may also have had direct relationships with some retailers.

The fire was the latest in a series of deadly factory fires in Bangladesh, labor-rights groups say.