Dov Charney is talking. He's been going for something like an hour and there's little sign of him stopping. He speaks without punctuation, rattling off figures, asides, anecdotes and taking phone calls, all at the same breakneck speed. Occasionally the patter is accompanied by the flapping of arms or a jig around his cluttered office. For emphasis - or perhaps because he simply likes the sound of it - he frequently punches a fist into his open palm. At one point he pulls down the waistband of his trousers to show me his electric-pink Y-fronts.

"I have a men's underwear here, a brief, that you would probably like to wear," he says. "A lot of people say" - smacks fist into palm - "that's a perfect underwear. I made 165 versions of it. I was in my underwear for, like, six months. I gave up even putting my pants on."

Charney is in the clothing business. Just over two years ago Charney, who turns 37 this month, took his successful line of wholesale T-shirts and opened his first American Apparel store in downtown Los Angeles. Today there are more than 100 of the bright, simple shops selling the company's bright, simple clothing, which it proudly boasts is made in a sweatshop-free environment. This year he plans to add 50 more. By the time he has finished, he says, there will be 500 American Apparel outlets across the world.

The list of store locations reads like a hipsters' dream vacation: Seoul, Tokyo, Berlin, Mexico City, New York. London currently has two, with two more in the works and one set for Manchester. The UK, he says, is a 20-store country. "We have the fastest growing apparel company in the US," says Charney, who goes by the title Senior Partner. "People say, 'Everywhere I go I see your store. I go to Paris, I go to London, I go to Mexico City.' But I tell them, 'We're following you, bitch'."

Charney deals in basics. His company's biggest seller is its 2001 fine-jersey T-shirt, available in colours ranging from cranberry to asphalt. But it doesn't just cater to the cotton-clad youth masses. Babygros, infant karate pants, even dog T-shirts are part of the mix.

The Jewish Canadian son of an architect and a painter, Charney began hustling at an early age, selling T-shirts while still at school before eventually dropping out of university to go into the clothing industry. Charney, it is safe to say, is not a regular CEO. Nor is American Apparel a regular American company. Located in a seven-storey pink warehouse covering 800,000 square ft in the post-industrial wasteland of downtown Los Angeles, American Apparel is bullish about its presence. The former train depot is decorated with intense and unfathomable slogans: "Legalize LA" declares one enormous banner strung across the outside of the building. "American Apparel es una compañía rebelde," says another.

Rebel company or not, in its brief life the company has achieved a certain notoriety - for its risque advertising, for the antics of its boss and for its labour practices. Eschewing the fashion glossies and the mainstream media, American Apparel takes out ads in the alternative press featuring unpolished models, often friends of Charney, company employees or AA customers, and unpolished photography, sometimes taken by Charney himself.

"Viva Mexico City!" reads the text of a recent offering above a cropped picture of a dark-haired girl wearing gaudy makeup, a green bikini top and a come-hither look. Sofía, the text tells us, likes to sell bikinis. "Next stop for her may be London, where we hear they could use a dose of Latin warmth. Word is that Mexicans are scarce over there and we couldn't think of a lovelier messenger to spread the news about one of our favourite places."

The aesthetic is Polaroid softcore in homage to Larry Clark and Nan Goldin. Boys with bulging Y-fronts stare breathlessly at the camera; girls in thongs and vests look coy. But there is also a hint of iffy personals, of the unvarnished allure of teenage sex. In one set, Lauren W poses for a series of close-ups with her fingertips in her dully parted lips. As photograph after photograph of Lauren W lying with her mouth agape spools past, one jumps out: a male hand has replaced the model's hand. The photographer, reads the credit, is Dov Charney.

"I think people have fake sex," says Alexandra Spunt, sitting at a computer working on the Mexico City ad in AA's in-house advertising department. "We're accustomed to this sanitised version of sex in public, so yes, there's a lot of interest in the photos, a lot of them are sexual and raw. It's the difference between something you can smell and something you can't." The current catalogue features on its cover not an amateur model or a member of staff but all-American porn star Lauren Phoenix, the star of Assploitation 2.

Charney has little time for the critics who allege that his advertisements are akin to teen porn: "It's noise, it's chatter, it's bullshit, it's perceptions, it's stereotypes, it's manipulation, it's garbage," he says. "I never started making any proper money until I decided to make clothing that was geared towards young adults only and not even consider whether it would fit on a boomer," he says. "To meet their sensibility, sexy was job one. Fashion is all about sex and function. You want to look attractive, and attractive is just another way of saying sexy. Even if you don't want the person, you want them to be attracted to you. Even among other men we want to look attractive and dapper. That's still driven by our sex drives."

Unfortunately, Charney's own sex drive has become as much a part of the American Apparel story as the desirability or otherwise of its Cotton Spandex Jersey Boy Brief. Charney preaches the doctrine of the modern, relaxed workplace: there are phones dotted around the pink building for workers to use free of charge; there is a massage area where workers can take advantage of a free massage every two weeks (although if you're on a piece rate, you lose earnings while having your back kneaded); no one bats an eyelid if you have a drink when working late; and the company's sexually charged imagery graces the upper office floors, away from the areas where the mainly Latino workers stitch garments together.

The company is also tolerant of workplace relationships, including the CEO's dating of his employees. What more natural place to meet someone who shares your interests than at work, ask Charney's defenders. But Charney's relationships, and the hyping of the company as a sexually charged place to work, have brought problems. As American Apparel has expanded - it now employs 5,000 people - so the mores and manners of big business have come into conflict with its founder's happy-go-lucky nature.

Three lawsuits alleging sexual harassment have been filed by former employees. Two have subsequently been withdrawn and a third is languishing. But the damage has been done; the image of the sexually liberated, progressive company has been tainted by the suggestion of exploitation. That suggestion was not helped by an interview Charney gave to the magazine Jane in July 2004. Over the course of two months reporter Claudine Ko spent, it seems, a little too much time with Charney, watching as he and a female employee "put on a show" for her (the implication is that the woman performed oral sex while Ko watched), and then interviewing him as he masturbated in front of her. Several times.

"I've got myself in trouble telling people what I like to do for 20 minutes off-site," Charney says. "Believe me - a) it's fun, and b) it only takes 20 minutes. That's it. The rest of it, I like being productive and animated. To me, I can affect people's lives in a happy way by producing garments and being a happy person and making intriguing photoshoots and opening stores in cool neighbourhoods. That turns me on."

Websites have taken great delight in deconstructing Charney: creep or visionary? It's a close run thing. But the fuss has been enough to persuade him to take a more considered approach to his role as the senior partner of a major company. And so he declines to discuss the lawsuits, other than to declare that there is a "false victim culture ... Most of these kinds of cases, it's always bullshit."

The fallout from the sexual harassment allegations has also taken a toll on the company's two other unique selling points - that it is sweatshop free and that it is pioneering "vertically integrated manufacturing". Exactly what this means is open to debate, but its root is in the notion that everything in the company - from design to manufacture to distribution to marketing - takes place under one roof. This, argues Charney, gives American Apparel a unique flexibility, allowing it to outstrip and out-innovate its competitors, both large and small. (We'll leave aside the fact that the weaving and much of the dyeing takes place off-site, some of it contracted out.)

With the pink HQ capable of churning out 1.325 million garments each week, American Apparel is encroaching on the territory of big players such as T-shirt manufacturers Haines and Fruit of the Loom and bulk retailers from Gap to Wal-Mart. And with everything done in-house, Charney claims that he can dream up a product on Monday and have it in the shops with accompanying promotion by the end of the week. That, claims Charney, is the hidden cost of outsourcing. "I might be able to get someone across the street to make me a coffee but it's nice to make coffee in my house."

His unconventional approach made financing the company difficult at first. "The thing that scares the banks the most is manufacturing in the US" - He adopts a big guy banker voice - "'You want to finance sewing and knitting equipment in the USA? You're fucking crazy, we just bankrupted nine companies that were doing that, what the fuck's your problem, kid?'"

But there were bankers, he says, who were willing to give him a shot. In fact, one of them is waiting outside his office as we speak. "You want to meet my bank manager," says Charney, jumping out of his chair, "put him on the fucking spot? Certain bankers say, 'Hey, it's fucking crazy.' But some bankers say, 'You know what, I'm going to put my money in this fucking new thing. I think this is more authentic, this company is more in tune with what the marketplace wants right now, and this is less risky for me than being with a traditional company that is out of touch.'"

He dashes into the corridor where his bank manager is indeed waiting, conspicuous by his age (over 45). We both stand aside as Charney is intercepted by a young girl - possibly 16 or 17 - accompanied by her mother. The girl, apparently, has visited to try out for some modelling work. "I think you're lovely, I have your pictures and we're probably going to work together," Charney tells her. Without skipping a beat he turns to the mother.

"Your hair is gorgeous." She looks like a flustered teen at the senior prom. Charney turns his attention back to her daughter. "You're very lovely, just don't trim your eyebrows."

We return to his office. "I have a $55m line of credit with that guy," Charney says.

Just what is it that you're good at, I ask, and away he canters. "It's not very English for me to tell you what my secret is, but I can be a cavalier arrogant American and tell you the bottom line if you want. OK ... I think I have an ability to spot something that's right. That house is the right house. Or that store is going to work. Or that underwear is so fucking hot, cut 30,000 of them, it's right on. Or that look, the look of that particular girl's face is so strong. Eyebrows, trimmed eyebrows are out. Any asshole that trims her eyebrows, don't even bring them in, you know what I'm saying? It's superficial, but that's fashion and that's style and that's real estate."

It is all some way from American Apparel's initial appeal, that it made its clothes in LA without using sweatshops. American Apparel is now the largest clothing manufacturer in the US, an oasis among the outsourced hulks of industry in Los Angeles. On the shopfloor it looks pretty much like any other mid-size manufacturing operation. Workers organised into small teams huddle furiously over sewing machines, working against the clock to produce as much as they can. They are paid a piece rate for each item the team produces, with a guaranteed minimum slightly above the minimum wage. Handwritten in marker pen on a board above each team is a tally for the day and the earnings per hour for that day.

The teams are the brainchild of chief operating officer Marty Bailey, an industry veteran brought into the company by Charney as he realised that the expansion was running away with him. "We have the highest-paid apparel workers in the world," says Bailey. "Our company will only be successful if these folks are."

Bailey questions the usefulness of the sweatshop-free label. "This is a business," he stresses. "It goes back to making a quality product. Sweatshop-free or ethical manufacturing, that's going to help sell it once, but quality is what brings people back." The reality, though, is somewhat disappointing: a recent trip to one of their shops revealed crooked seams and missed stitches.

While the shopfloor work looks crippling, the conditions buck the stereotype of the sweatshop. There is plenty of light, each floor is freshly painted white, there is a subsidised canteen, the company offers interest-free loans, there are free bicycles, even cash-machine cards to avoid the charges incurred in cashing a wage cheque. Notably, the sexual iconography that dominates its public face is absent from the walls where the product is actually made. There is a year-long waiting list to get a job.

It all seems remarkably progressive, despite the fact that there are almost no white faces to be seen. And there are no unions - a sensitive subject with Charney and Bailey. A recruiting drive three years ago by a garment workers' union failed, thanks either to a lack of interest on the part of the American Apparel workforce or intimidation from the company, depending whose version of recent history you choose to believe.

"Every employee has the right to organise if he chooses to do so," says Bailey. "That's the worker's choice. There was a rally, and it was more to say leave us alone, we don't need you here. If somebody decides they need someone to speak for them, then I've not been available enough."

An employee is produced to pay tribute to their enlightened bosses. "I'm not against a union but I think if you have the right level of respect, you don't need one," says Jeremias Pablo, a 26-year-old supervisor who has been with the company for seven years. "Also the boss, the great man Dov Charney, he goes on the floor and asks us if we are OK. If you want an appointment with him, you can do it. The doors are open."

Yet despite his assurances, there is no system of collective bargaining at American Apparel. And in a large multinational enterprise such as American Apparel, it is surely unrealistic to imagine that labour issues can be resolved on a one-to-one basis.

But Charney's heart seems to be in the right place, his vision of himself as a benevolent capitalist intact despite the growth of the company. "It's on point not to be exploitative right now," he begins. "While the whole industry is ignoring the workplace and outsourcing it, we're actually reintegrating workers in the company. We don't have to exploit them. Maybe I bought some of that socialist crap from socialist Canada, but I Americanised it because it's the company that's doing it, it's private enterprise."

With the company making so much profit, he says, "Guess what? We didn't have to gip off the damn garment workers. It appears to be somewhat extravagant from the outside, but really - who cares? We're making so much money, so who really, who's counting?" He takes great delight in showing me the spreadsheets for his retail operation from the day he opened his first shop in LA in October 2003 (it took $83 on its fourth day).

"Look," he says, "the fact of the matter is I'm not leading the pack in terms of neo-CEOs, OK? But the old guys are going to retire and there will be a pack of new guys like me. This is just waiting for a new wave. It's not going to be that big a deal if the CEO doesn't get married or doesn't wear a tie. Who fucking cares? Things are going to deformalise. It will be easier to carry on a less formal lifestyle."

Which begs the question: where does this burgeoning enterprise go next? How does it retain the informality of both its working practice and its outside image while it aims to become the Starbucks of the T-shirt? Charney seems to relish too much the control and the flexibility guaranteed by the absence of shareholders to go public. And as for retaining the personal touch over the behemoth of American Apparel, he returns to one of his favourite themes: technology.

"There's new technology today," he exclaims, jumping out of his chair. "Video-conferencing, emails, I could talk to 100 people in a minute. I could poll the stores. I have visuality to what's going on in my stores every night, every day of the week. Right here and now. There's ways I can maintain a connection whereas I couldn't before. And because all my products are made in the single country, I can get stuff into other countries quickly."

Whatever happens, the hustler who started off bringing T-shirts across the Canadian border in black bin liners has arrived. Whatever the fate of his company, he has few concerns about his own future.

"Well, you could say so, by virtue of the fact that I own this company," he says when I ask if he is rich. "I'm not personally wealthy but I'm working on it. You don't have to worry about me".