The problems are structural in the industry, but they're not an accident: they're a result of the way the (current) industry emerged. Publishing is a very old business—it's been around in some shape or another for over 500 years, and in much its current shape since the mid-Victorian period—and the way publishers do business is the intersection of the set of all business practices that did not cause one of their antecedents to go bust, over a roughly 250 year period.

Until the 1980s, publishing houses were generally small family-run businesses who owned their own typesetters and printing presses and employed their own production people and warehoused their own stock. And growth in the industry was negligible—it's a mature field, you can't easily "grow" a business in publishing because the amount of time consumers have available for reading (i.e. consuming more product) is an inelastic constant: the best you can run is a zero-sum game against other publishers. And in the large, you're competing with other media. Every computer game a punter buys is 20-100 fewer hours they have for reading, for example.

The biggest change of the 50 years prior to 1995 was that a wave of mergers and take-overs swept through the industry, first in the USA and then in Europe, starting in the early 1980s. This was largely complete by the mid-90s, and utterly changed the landscape. The former family businesses were now "imprints" or trading subsidiaries. Core competencies were merged, with cost savings, while less central stuff like owning warehouses and printing presses was out-sourced. Costs were pared to the bone. A senior editor these days doesn't just 'edit', they're a business manager running acquisitions and supervising product workflow and marketing strategy—jobs which come out of the time available for editing, in some cases consuming all of it.

But it's still a more or less global zero sum game (competing for readers eyeball-hours). And because the rate of individual production is relatively low and the product is still produced artisanally by cottage industries, product lead time is measured in years, time to achieve net positive revenue is also measured in years, and it's important to keep the back list on tap because it can take decades to grow an author's career. Stephen King was an overnight success with "Carrie" after a decade of learning to write, but Terry Pratchett took about 15 years to finally break big. J. K. Rowling took 3 books to really get rolling, and she grew eye-wateringly rapidly by industry standards. And some authors are slow-burn successes: my big breakthrough book was my tenth novel in print ("Halting State"). J. R. R. Tolkien's Lord of the Rings was in print for a decade or more before it really took off in the 1960s. If you practice ruthless commercial Darwinism, weeding out any hopeful mutants that aren't immediately successful, you will miss out on a lot of huge opportunities.

So reforming the publishing industry is a very non-trivial undertaking.

Which is also why Jeff Bezos picked it as his #1 target when he founded Amazon. He set out to disrupt an incumbent mature industry using the internet, and picked publishing because it was obviously the most dysfunctional. After all, if he'd gone after groceries he'd be competing with sharks like Tesco and WalMart.

But the trouble with disruption is that it's dangerously close to detonation. You can end up destroying what you sought to shake up and take over.

PS: there have been attempts to get away from artisan production. Shared universes, media spin-offs, and the likes of the DC and Marvel Comics constitute such attempts. They generally don't work well for the creators (most of them entail work-for-hire and put harsh constraints on the creative freedom available to the authors or other artists involved); collaborative short fiction/novels are much free-er, but it's a general truism that when you collaborate with another author, both of you end up writing 75% of the book. Somehow we've been unable to come up with a better way to do high quality fiction since Thomas Hardy's day, despite having a century filled with progress in other fields.