Dov Charney, the man at least as famous for founding American Apparel as he is for being serially accused of sexual harassment, is showing me around his new factory in south central Los Angeles. As usual, he is talking at least as fast as he is walking.

“See this shirt? That was influenced by a 1990s shirt one of our designers found. And over there is our photostudio. That guy in there, he’s like a Gatsby bon vivant,” he says in his loud, raspy voice, pointing to a tall young man who, like all the young people who work here, has a somewhat bewildering job title and looks like a model. Another one follows us around with a mobile on a selfie stick. This, I’m told, is “for content”.

But there’s no time to ask questions because Charney, who was sacked from his old company in 2014 after years of rumoured sexual misconduct, is on the move again, while simultaneously texting on one phone and talking on another. The reason we are here today is because he is launching a new label, Los Angeles Apparel, and if you think that sounds like his old label you should see the clothes: cute pleated skirts and 1980s-style sportswear are modelled by mannequins in the factory, making the place look an awful lot like an American Apparel shopfloor. Hey, why fix something that only broke because of a few allegations of sexual impropriety?

Charney himself is clad head-to-toe in white – white T-shirt, white tracksuit bottoms and white Reeboks. “I look like I’m in a loony bin!” he crows. But what he really looks like is someone’s Uncle Morty from Miami: hipster fashion, which Charney, 48, did so much to popularise, has a cruelly young cut-off age, after which all those tapered trousers and oversized sunglasses just make you look like someone’s aged relative. And then the man the New York Times described as “a barely restrained id” and feminist blog Jezebel called a sexist “troglodyte” turns to me with a grin: “Come!” he barks. I follow him through the door on to the factory floor.

American Apparel started off selling basics wholesale, and was a fashion sensation when it launched into retail in 2003. Its slouchy hoodies, funky sunglasses and high-waisted jeans will be seen to be as much a part of the look of the early 2000s as punk was in the 1970s and grunge in the 1990s. It sold a lifestyle to the masses cheaply and let suburban kids pretend they were, as Charney puts it, “the creative class in urban areas” (hipsters, in other words). But the company itself presented a paradox: on the one hand it was manufactured in the US by workers who were paid well; on the other, its advertising featured young women in absurdly provocative poses. Charney himself appeared in some, lying next to seemingly naked young women.

Charney at the Los Angeles Apparel factory where he lives 24/7, sleeping on a mattress. Photograph: Melissa Lyttle/The Guardian

Unusually – uniquely, even – American Apparel was a high-street store that had a face to it and Charney – whose facial hair, tight T-shirts and vintage glasses suggested a 1970s pornographer – was that all-too-visible face. His reputation as a sexual creep became unshakeable when he masturbated – twice – in front of a young female magazine journalist during an interview in 2004 (“‘Can I?’ he says, adjusting himself in his chair …”), and he was whacked with a seemingly endless series of sexual harassment charges over the next few years. In 2011, five ex-employees filed lawsuits. This increasingly became a problem for consumers: in the early years “hipster” meant someone who wore vintage clothes and read Vice magazine, but as the decade progressed the term denoted someone who cared about ethical values, and Charney’s reputation was overshadowing the company’s record on workers’ rights. By the time Charney was finally sacked by the board of his own company, he’d had one of the most vertiginous rises and falls in the business world, and he went from having over $500m in stock options to bankruptcy.

Charney is desperately counting on Los Angeles Apparel – which, like American Apparel, is starting off in wholesale – to restore his standing. To this end, he is currently living in the factory so he can keep an eye on things 24/7, sleeping on a mattress that everyone carefully walks around. This also saves time in the mornings: instead of commuting he can spend an extra hour dealing with the four lawsuits linked to American Apparel’s implosion that he is still involved in. No one seems to think it is a little ironic for a man who was brought down by accusations of sexual impropriety in the workplace to now keep a mattress in his office.

I’d been warned that I might be a little shocked by the factory. “There might be some people, um, undressed,” Charney’s loyal assistant – and, it turns out, his cousin – Sam said to me on the phone.

“What?” I replied.

“But journalists don’t think it’s strange when athletes are getting undressed in locker rooms after a game, so it’s weird people get freaked out by it,” Sam says. And people do indeed get freaked out by it: one of the many allegations made about Charney at American Apparel was that he wandered through the offices in his underwear.

But the factory does not look like the Roman orgy I’d expected. The 350 largely Hispanic workers on the factory floor are all cutting and sewing while Charney, who seems to know them all by name, and is fully clothed, talks to them intently about the tiniest details. In fact, most of them worked for him at American Apparel and such is their loyalty to a man who always paid them at least minimum wage, plus benefits, that they have bet the little they have on him being able to start up a new company. Charney is not the only one with a lot at stake here.

“Wait,” he says to me, “I gotta show ya this skirt, it’s just like what you’re wearing. It’s so cute!”

Watching him shuffle along in his grubby tracksuit, talking urgently to the workers about how the seam of a T-shirt must sit, babbling to bemused twentysomethings about how his family has always been in the schmatte (Yiddish for “clothing”) business, all I can think is, this is the guy who dictated youth taste for over a decade? This is the guy who was seen as so out there his adverts were banned in Britain? This was one of the most notorious predators in the fashion business? This guy?

But then the two of us go into a small side office and things become a lot clearer very quickly.

People walk past an American Apparel store in Los Angeles in 2016, after a bankruptcy court approved the company’s reorganisation plan. Photograph: Mark Ralston/AFP/Getty Images

In order to understand Charney you have to understand that American Apparel was, from its clothes to its advertising to its ethos, fully an expression of him. His move into hipster fashion was the culmination of a lifelong obsession with American youth style that began when he was a precocious teenager in Montreal. He went to university in the US but dropped out to manufacture and sell T-shirts. He was widely criticised in later years for allegedly hiring employees based on their looks, but he says people misunderstood: a teetotal workaholic, he needs to have trendy young people around him to keep him plugged into the zeitgeist.

“For example, there’s this girl, Jasmine, and she was just an intern but she’s got good taste so boom! She’s in!” he barks, smacking his hands together.

Lots of his heroes stayed in touch with youth culture by hanging out with young people, he adds, citing Andy Warhol, Vivienne Westwood and “that crazy German guy, designer, lost lots of weight, has boyfriends” (Karl Lagerfeld). Also: Woody Allen. As it happens, Charney had legal issues with Allen in 2008 when American Apparel used his image without asking, but now Charney claims two are buddies.

“He came out to see me in LA before I was fired. He toured the factory, we hung out, he’s a mensch,” Charney says.

The image of these two men with a notorious penchant for young women together is certainly a memorable one. Did Woody reach out to him?

“Yes, through a mutual friend. He’s a fine man,” he says.

As he is telling me this story he is fiddling with something on the arm of the sofa he is sitting on. At first I think it’s a light but it turns out to be another selfie stick. He puts in his phone and carefully turns it so it is filming his face, which is where it stays for the next three hours. Does he film his interviews as a precaution, given what’s happened in the past?

“I just think interviews are interesting. They’re fun to watch back and wonder what I was thinking. Shame you’re not in it!” he says.

To Charney, his story is incredibly simple. Like his business hero, Steve Jobs, he is the free-thinking maverick who corporate forces set out to destroy. It’s easy to see why he inspires such loyalty from his employees: he is undeniably charismatic and talks with passion about how a business should be run, with an emphasis on workers’ rights, listening to young people and having no hierarchical divides. His supporters and critics talk about “the cult of Dov” but Charney sees it more simply: “I like young people. I get them. I’m like a young person. The thing about monogamy is it freezes you, so one way to stay young is to never graduate to that conventional situation,” he says.

Does he have a girlfriend who shares his office mattress with him? “I wouldn’t say that, but I have bonds with people that are very intense and important.”

So he doesn’t have any trouble dating now, despite his reputation? He makes a wolfish grin: “No, that is not a problem. The women like an enfant méchant. Also, I look like a warrior because I’m coming back.”

Charney describes Los Angeles Apparel as “a continuum” of American Apparel: “The people aren’t different, the materials aren’t different, the environment isn’t different.”

Does that mean he’s still going to walk around in his underwear? “That [claim] was untrue. Absolutely untrue! I mean, it is true that I was in my underwear in front of employees when I was doing underwear fittings. That happens in fashion companies.”

He grabs hold of a pair of tiny black panties that happen to be on the side of the sofa.

“So take Jasmine –”

Jasmine the intern?

“Yeah, she wore this underwear in front of me,” he says. “It’s not incendiary, it’s not inflammatory, it’s totally normal.”

But it is sexy, presumably.

“It is! I mean, have fun, try on the underwear. I’m not unfit, you know.”

Charney insists he’s too busy at the moment to think about this kind of sexy stuff, although this would be a little more believable if two hours before our interview he hadn’t posted on his Instagram a video of a young female employee in the office bending over in a thong leotard, filmed in the photo studio we just visited. As the camera looms right up to her face she looks around and smiles sexily.

“Look, I’m not going to be a victim of sex-shame tactics,” Charney says when I ask about the film. “This obsession that I should be punished for the advertising is fascistic and anti-woman. I will express myself as I always have done.”

Is he dating the young woman in the interview? “No, no. But there’s always a connection between a filmmaker and subject.”

The story of what actually happened to American Apparel depends on who you ask, Charney or the board members. The shortest answer is that the problems started when the company went public in 2007, and soon enough, all the qualities that Charney saw as his strengths – his unpredictability, his dizzying ambition, his notoriety – were liabilities in the context of Wall Street. American Apparel was also crushed with debt accrued from rapid over-expansion, despite raking in hundreds and hundreds of millions of dollars a year, and Charney himself was costing the company money. All told, the litigations against him cost the brand $8.2m, although most was covered by insurance. In 2014, it was announced that Charney was fired, “citing an ongoing investigation into alleged misconduct”. But without Charney, acting as the head, face and groin of the brand, the company crashed and thousands of jobs were lost. It was chopped up, sold and re-sold, and is currently limping along online, using many of Charney’s images.

That Charney slept with many young women who worked for him has never been up for debate. But it is also true that he was never actually found guilty of sexual harassment, despite the multiple allegations. Of the five suits filed in 2011, for example, three were cleared by a judge and two went to arbitration.

“There had been stories about Dov for years and years, but they had been very hard to pin down because every time an employee made a complaint against him it went to arbitration,” Allan Mayer, former co-chairman of American Apparel’s board, tells me. “But when we were able to conduct a more forensic investigation with an outside investigator we found videos and emails from him on the company server that, well, to call them inappropriate would be an understatement.”

Charney insists this is all bunkum and was just an excuse for the board to take the company from him and make money for themselves. Yes, there were sexual harassment allegations, but these were old by the time he was fired, and in none of the cases was anything found against him, and this is all true. He also insists the business was in great shape financially: “Why else would they want to take it off me?”

But Mayer says that because of Charney’s notoriety no reputable business would lend them money, so they had to borrow “at credit card rates”.

“I’ve known Dov since 2004 and I know he honestly doesn’t believe he sexually harassed anyone,” says Mayer. “But when a 45-year-old CEO is sleeping with 19-year-old sales clerks it doesn’t make it consensual. The imbalance is so vast.”

Mayer admits American Apparel’s policy on workplace relationships “was not as hard and fast as it is at other companies” and Charney seizes on this: “If it was such a problem for them why didn’t they just ask me to sign a non-fraternisation policy?”

Would he have signed it?

He hesitates for a few seconds: “Temporarily, maybe. Sure.”

Many people see an inherent contradiction between Charney’s indefatigable championing of workers’ rights and his equally energetic pursuit of his female employees. But for Charney, the through line is obvious: he is, essentially, a libertarian who thinks there should be no boundaries, national, professional, sexual.

“Look, let’s say this first: I abhor all forms of sexual harassment, period. But it’s unrealistic for the government to interfere with people’s private lives, and that’s it,” he says.

I ask if he’s still sleeping with employees. “That’s private!” he retorts.

Charney talks about his firing with obsessive fury, raging about how his business was “stolen from” him. But does he regret the behaviour that led to his sacking? “Not at all! Sleeping with people you work with is UNAVOIDABLE!”

But “employees” are not people you work with – that’s colleagues. An employee is someone who works for you, I say. “Yeah, but that’s – OK, I’ll say this, I never had a romantic relationship with a factory worker. Ever! It wouldn’t be possible! But a creative equal? Yeah! And if anything, I’ll tell you, I don’t know who was the predator – you know what I’m saying?” he laughs.

“Take yourself,” he continues. “You’re well-spoken, well-educated, you decide to work here. And we develop a romantic interest in each other. We could say, ‘OK, we’re attracted to each other, but it’s better we just work together.’ OK, we could try that. And that may work. But if the attraction is so intense, eventually we’re gonna give up! We’ve tried to avoid it, but we’ve decided that we’re going to get involved.”

But could he really not have changed his behaviour to stay in control of his own company? “Never! Out of the question. It wouldn’t be good for society! It wouldn’t advance the rights of workers.”

But it would have kept your workers employed.

“No, no!” He is exasperated that I’m still not getting the truth here. “You think, I was just supposed to stand up straighter, not allowed to wear [just] my underwear? No! [The board] wanted control! It was all a hoax.”

But even if it was all a hoax, even if the board just wanted to seize the company, didn’t he leave himself vulnerable to it?

“Maybe, a little bit, probably. But I think my real mistake was that I was too trusting. I should have removed some of the board members.”

“I think Dov is irrepressible,” says Mayer. “He is who he is and he sincerely does not see that he did anything wrong, so it’s hard to see why he would change.”

There is no doubt Charney is, when it comes to retail and workers’ rights, something of a visionary. But if you are not willing to keep it zipped to pursue your dreams, you will only run so far before tripping over your trousers. You can insist that this is just about society’s hypocrisies and limitations all you want, but if you’re not willing (or able) to compromise at least on this issue for the greater good, then people will wonder what your priorities actually are. But to Charney, his story exemplifies how hysteria about sex and gender can obscure the real issues.

“Like with Trump, OK? It disgusted me when they made a big deal about the Billy Bush episode. The man’s a terror because he’s anti-worker, anti-immigrant, a nationalist, hostile to environmental ideology and knows nothing about how to bring manufacturing back. He has no ideas! That’s what matters! Liberals lost on ideology!”

And of course, he’s kind of right, and just as I find myself nodding along he adds, “That stuff he said to Billy Bush [about grabbing women by the vagina] – who cares? If you recorded all the things I said about women in the past 10 days it would be no different.”

Interview done, he gives me one last tour of the factory. He is a ball of energy; you’d never guess he’d been talking pretty much non-stop for three hours as he chatters away to suppliers, workers and employees, talking on this phone, texting on that one. I tell him I’m going to call a cab and wait out front. A few minutes later, he suddenly appears next to me. “So are you hanging round in LA for a while?” he asks, and he has a shy smile on his face.

I say I am.

“What are you up to?” he asks.

I tell him I’m doing another interview, I might go check out some museums.

“Uh-huh,” he says, still smiling.

I mention I also want to pick up some American toys for my kids.

“Right,” he says, smile disappearing. “OK, bye.”

And just like that, he disappears, already on the move again.

• This article was amended on 11 and 15 September 2017. An earlier version included an image of a spoof advert featuring Dov Charney. This has been removed. Post publication a spokesperson for Woody Allen denied, but later confirmed that the meeting with Charney took place.