It wasn't having oral sex with an employee in front of a female journalist that now threatens to undo Dov Charney, founder of American Apparel. Nor was it simulating oral sex with another female member of staff whom he had ordered to pretend to masturbate in front of him. The 41-year-old's professional and personal reputation isn't even on the line because at least three female employees have filed sexual harassment lawsuits against him (all the cases were settled before reaching trial); nor because he walks through his factory in his underpants and conducts meetings wearing just a thong – or a sock. The sock is not, one should add, worn on his foot.

Instead, the fashion empire of this maverick Canadian entrepreneur – who apparently relishes his reputation as a pervert and a libertine – has gone from being the coolest company on the block when it arrived in Britain in 2004 to the brink of bankruptcy because its auditor, Deloitte & Touche, resigned last month after discovering "material weaknesses" in the financial controls of the clothing company dating back to last year.

For the exhibitionist Charney, whose excesses are the stuff of fashion-industry legend, to be finally humbled by accounts that are more street corner than Wall Street is, some might think, a bit like Al Capone finally being brought to book over tax evasion. But whatever the catalyst, it has led to the unravelling of an empire. It is a rags to riches – and back to rags again – story that has the fashion world in its thrall.

Known for its cotton basics – including the infamous Y-fronts so beloved of the founder himself – shares of the LA-based company touched $14 (£9)in 2008. That same year, the Guardian named American Apparel label of the year. In 2009, Charney was a finalist for Time's 100 most influential people in the world.

Yet last week, his company's shares were trading at an all-time low of 75 cents – a drop of almost 50% in the last week, forcing American Apparel to admit that it now has debts of $120m and is losing money at a rate of nearly $30m a year.

With sales down by 16% in its 279 shops globally – 15 in the UK – the company whose clothes were worn by the coolest kids across the world is now at risk of breaching the terms of an $80m loan provided by British company Lion Capital, in March 2009 – a loan advanced to rescue it from another financial crisis. If Lion Capital call in its loan, American Apparel will be forced into receivership.

Admittedly, Lion Capital may not be an entirely grey-suited, sober company of British investors. Just last week, Lyndon Lea, the son of a Lancashire hairdresser and a founding partner of the private equity firm, was reported to have held a no-expense-spared polo party at his £25m mansion in Montecito, California, with sushi being served off scantily clad women rather than plates. But for all that Lion Capital might recognise a kindred spirit in Charney, the repayment of its loan must remain the bottom line. Lion has indicated it will remain supportive and is considering a number of options, including relaxing its loan to give American Apparel some breathing space. But if the investors change their mind and decide to cut their losses, the jobs of 10,000 employees in 20 countries will be at risk.

It is a chaotic final chapter in the story of a bombastic figure whose out-of-control carnality has, at times, overshadowed the fact that Charney is also an old-fashioned captain of industry – an eccentric, erratic, brilliant figure – with a disconcertingly simple concept: to make humble T-shirts, jogging pants and sweatshirts seem exciting.

Charney cut his teeth in retail while still at high school in Montreal, importing Hanes and Fruit of the Loom T-shirts from the US for his Canadian friends. He claims to have shipped around 10,000 shirts at a time in a rented truck before, in 1990, dropping out of Tufts University to pursue his business full-time. Borrowing $10,000 from his father Morris Charney, an architect, the 21-year-old moved to south California and began manufacturing T-shirts under the American Apparel brand.

Six years later, the company was unable to cover its debts and went into administration. Charney, though, was undaunted, and in 2003 founded American Apparel in two stores in downtown LA, employing 1,300 people. Within a year, he was named Ernst & Young's entrepreneur of the year and Apparel magazine's man of the year. The legend was born.

The confusing thing about Charney is his combination of out-of-control carnality and progressive social liberation. One industry insider calls him an "odious character about whom I have heard nothing but bad things, particularly concerning his recruitment techniques and the way he treats female employees". There is, he adds, a "certain over-reliance on oral sex during interviews over assessing their retail experience".

On the other hand, Charney is passionate about liberalising immigration policies, paying a fair wage, refusing to outsource manufacturing, and workers' rights. Virtually all of the company's manufacturing, distribution and retail is done in-house in downtown LA, by employees – the majority of whom are immigrants – who are paid more than twice the minimum wage, offered low-cost, full-family healthcare, and allowed free international phone calls during work hours.

In 2008, Charney – founder of the largest manufacturer producing garments in America – gave $25m of company shares to his employees. American Apparel still owns all its retail stores, rather than franchising them, and Charney – who is obsessive about his brand, throwing tantrums about everything from stock allocation to necklines – still holds, for the time being, a 53% stake. He explicitly resists being labelled as a manufacturing tycoon with a heart of gold, or even a Jekyll and Hyde, happily admitting that he acts not for moral reasons but because it is a better business strategy.

So what has gone wrong? Fashionistas say the brand is still strong. "Just two years ago, American Apparel was the ultra-fashionable shop that you found in every cool location, from Covent Garden to Oxford Street to Shoreditch," says Melanie Rickey, Grazia's fashion editor-at-large. "It became very big, very quickly. Everybody was wearing it, and I mean everyone: high-fashion kids, clubbers, geeks and gay kids across the world. It crossed all genres and tribes."

Interestingly, Rickey insists the brand has not lost its edge: "No one does what they do in the way they do it: not Zara, not Topshop, and not H&M or Next," she says. "I will personally be really upset if American Apparel close their doors because it would be a great loss to our high street. I will be running to stock up on things they do really well, the classic items they do better than anyone else on the high street. They will be hugely missed."

But not all fashion insiders agree, pointing to the brand's failure to develop. The critics says that, for a company who took the humble T-shirt from basic item to fashion icon, to fail to catch on to this year's hot look – the soft jersey cotton T-shirt – is quite a lapse.

"They should have been very successful," says Lauren Stevenson, a director at my-wardrobe.com. "There has been a huge surge in the fashion for T-shirts in the past three years. Our sales of luxury jersey brands are up by 430% against last year, while luxe jerseys have increased by 450%."

So why, in a market for basics growing by that margin, have sales in American Apparel declined by 16%? Robert Johnston, GQ associate editor, has a theory. "I've bought lots from American Apparel in the past, but now it's mainly their underpants I go back for. They do a lovely Y-front," he says. "My problem with the brand is that it's a one-trick offering: you can't see where they can take it. Calvin Klein has built an empire on pants, but he recreates his image and updates the design."

An American Apparel store in West Hollywood, California. Photograph: Mark Ralston/AFP/Getty Images

The other problem is Charney himself. "He comes over as such a sleazeball," Johnston says. "Because their campaigns are slightly grubby, and he's more than slightly grubby, it all conspires to be rather unappealing. The whole image of American Apparel was supposed to be: 'Aren't we good, making everything in the US and not using sweatshop labour?' Yet every story you hear about Dov himself is so sleazy that all the goodwill their ethical values should create is squandered. His reputation would certainly make me think twice before shopping there."

Pearse McCabe, planning director at global design consultancy Fitch, disagrees. "Dov's image is part of the DNA of the brand. It's all about the understated clothing, the sexy communication style and him. Consumers won't refuse to go into the stores because of him. I really don't think that's what this is about."

American Apparel is teetering on the edge of bankruptcy, says McCabe, for a number of reasons – not least its dramatic expansion: 150 stores in its first three years and nearly twice that in the next three. The level of debt that expansion incurred is only now becoming clear.

"The level of expansion has been breathtaking," McCabe says. "That puts a lot of pressure on what their USP was in the first place, which was to be a trendy niche brand. In addition, I feel the brand has stood still. I see exactly the same things in the Oxford Street store today as I did when I walked into their flagship LA branch the week it launched."

The brand was once a neat proposition with a sexy, preppy image that hit the fashion trends head-on and whose ethics were exactly right, McCabe adds. Then came the recession, and the company found itself trapped in the limbo of the high end of the middle market. "Why pay £30 for a plain white T-shirt when you can pay less in Gap or get a higher quality that will last longer by going slightly upmarket?" he asks.

American Apparel's ethical USP has also, it seems, rinsed out in the wash. "People have forgotten the ethics the company was built on, and so have forgotten why they should pay the premium," says McCabe. "Even if you start to hammer that message home again, it's too late: the retail market has been green-ised. Everyone is trying to be good, so it's really hard for American Apparel to stand out."

It is, of course, far from the only clothing company to hit hard times. A number of big, listed fashion retailers have seen similar declines in the past and recovered. Last year, Abercrombie & Fitch recorded a fourth-quarter decrease of 23%, Saks showed a 14.7% decrease, and Niemann Marcus 21.4%.

"It can happen to the best of companies – look at Marks & Spencer in the early 90s," says Bryan Roberts, research director for retail analysts Planet Retail. Yet he admits that the severity of Deloitte's report, and the fact it then resigned as auditors of the clothing company, is a "fairly cataclysmic" sign that, at best, leaves the business with only a "vague form of future".

"It sounds as if someone has blown a whistle about something less than palatable that's going on," Roberts says. "That's not to say anything illegal is taking place, but someone could be trangressing best practice or minimum requirements."

Charney has pinned the blame on a police raid on its LA factory last year that found 1,500 illegal immigrants working there – nearly a third of its workforce. Charney was warned about the raid by the authorities before it took place, and said he did all he could to get papers from all his workers. He could do nothing, he maintained, about the fact that so many turned out to be forged or fake. Despite that, all the illegal workers had to be fired and production suffered, saddling the firm with more debts.

It has left a toxic soup of contradictions. "What makes American Apparel unique is the flamboyant nature of the man and the business," says Roberts. "He flies in the face of business sense, fashion manufacturing and retailing sense. There are plenty of individuals in the US who would not be shedding tears over his individual demise – but those who campaign to improve the working conditions of underprivileged employees and on immigrant rights, or who want to see manufacturing come back to America, would be upset to see Charney's business go down the pan."