The irony of Elon Musk’s latest legal drama is grim. He wanted to take the car maker Tesla private, because he hates the bureaucracy, red tape, and regulation that comes with being a company owned by shareholders, traded on the stock market. But the ham-fisted way he tweeted about those intentions, and then later walked them back, has now landed him in a battle with the body responsible for enforcing those very regulations, the Securities and Exchange Commission, which is suing him and wants him removed as CEO of Tesla.

But the thing that is really playing badly with investors, who have traditionally given Musk a lot of leeway, is the idea that Musk walked away from a settlement deal that would have seen all this go away and allowed him to get on with running Tesla at this crucial time for the company. Instead, he wants to take on the government, risking not only his own position but the company’s future too.

“This has become ‘I’m Elon Musk, and I’m going to fight the BS charges,’ and that’s not what he needs to be doing. It’s stupid,” says Ross Gerber, CEO and president of Gerber Kawasaki wealth and investment management. He plans to team up with other large investors and write to Tesla’s board, telling them they need to step up, protect the company, and make it less dependent on the Musk roller coaster.

Big corporate shareholders and investment companies have let Musk get away with being unconventional. He’s smoked a blunt live on Joe Rogan’s podcast, he’s called a member of the Thai cave rescue team a pedophile, he’s missed more of his own production targets and deadlines than an arrow in a windstorm. But he’s also created a car company that is building and selling vehicles at an ever-growing rate and that at one point was valued higher than General Motors. Fighting the SEC, though, is not just him being a quirky entrepreneur—it's him being reckless.

The SEC had reportedly worked out a settlement with Musk’s lawyers, which it was ready to announce on Thursday. According to CNBC, Musk wouldn’t have had to admit or deny fraud allegations over the tweet he dispatched on August 7 saying he had funding secured to take Tesla private. The SEC would have fined him and Tesla, and barred him from being company chairman (but not CEO) for two years. It would also have required the company to appoint two new, independent directors. (Musk acts as both chairman of the board of directors and the CEO of Tesla. Technically the CEO reports to the board, so he’s acting as his own boss, which isn’t unusual. The SEC’s settlement proposal to remove him as chairman would have enabled him to keep running the company, but with oversight.)

The conditions may sound harsh, but they’re nothing compared to what he could face if this goes all the way to a trial. But Musk pulled out of the deal at the last minute.

“It has to be that ego that made him walk away from a deal that is so light that I’m surprised the commissioners approved the offer,” says Erik Gordon, a specialist in entrepreneurship at Ross School of Business, University of Michigan.

In its legal document, the SEC is now asking the court to consider five remedies and reprimands against Musk, including making him give up any “ill-gotten gains” he got out of the process, ordering him to pay a fine, and most seriously, ordering that Musk be banned from acting as an officer or director of a publicly traded company. That would effectively remove him as CEO of Tesla and mean he would have no say in the running of the company.