On my way back to London the other day, I was clawing my way toward the buffet car when I noticed with a shock that more or less the entire train carriage was reading… novels. This cheered me up immensely: partly because I have begun to fear that we are living in some kind of Cowellian nightmare, and partly because I make a good part of my living writing them. Where were the Heats and the Closers, I wondered? The Maxims and the Cosmos? Where the iPads, the iPhones, the Blackberrys and the Game Boys, the Dingoos and the Zunes? Why neither the ding of texts, nor the dong of mail? Barely anyone was even on the phone, for Christ's sake. They were all reading. Quietly, attentively, reading.

My cheer modulated into something, well, less cheerful (but still quite cheerful) when I realised that they were all, in fact, reading the same book. Yes, you've guessed it: The Girl With the Dragon Tattoo who Played With Fire and who, some time later we are led to believe, Kicked the Hornet's Nest. In the next three carriages it was the same story – men, women, toddlers. A glance out of the window revealed that even the cows were at it – nose deep, hay forgotten. And when, finally, I arrived at the buffet car, I was greeted with a sigh and a how-dare-you raise of the eyebrows. Why? Because in order to effectively conjure my cup of lactescent silt into existence, the barrista in question would have to put down his… Stieg Larsson.

In terms of sales, 2010 has been the year of the Larsson. Again. His three books have been the three bestselling fiction titles on Amazon UK. Along with Dan Brown, he has conquered the world. The success of the Millennium trilogy is a tale of unimaginable public appetite, staggering international sales, big-screen boosts, perplexed publishers and (let's face it) not-that-originally-reformulated formula fiction. Not least among the reasons for the bafflement of the industry (and fellow writers) is the amateurishness of the books – something, curiously, that Larsson has in common with Brown. Readers, publishers and writers alike can agree that John Grisham, Robert Harris, Tom Clancy or Danielle Steel build up their massive readerships by knowing precisely what they are doing; they are master practitioners of their highly skilled craft. Conversely, Brown and Larsson – in their different ways – are mesmerisingly bad.

Here is Dan Brown, for example, describing – without flinching – how women find his hero's voice like "chocolate for their ears" before having said hero muse to himself that "he knew what came next" – "some ridiculous line about 'Harrison Ford in Harrison Tweed'". Leaving aside the queasiness of the gender politics (another communality with Larsson is the cod feminism), ridiculous is not the word we're after here. Larsson, meanwhile, opens Part 1 ("Incentive") of his first book with the most tedious acronym-packed exchange – not a conversation, not dialogue – that I have ever read. His two characters sit stranded in harbour because one of them can't start his engine (no joke) and begin "to explore what was ethically satisfactory in certain golden parachute agreements during the 90s". Says character "B": ''The AIA obtained government guarantees for a number of projects… The Swedish Trade Union Confederation, LO, also joined in… [and] Wennerström presented a plan, seemingly backed by interests in Poland, which aimed at establishing an industry for the manufacture of packaging food stuffs." Pause for a line or two to take this in before – again without irony – says character "A" in reply: "This is starting to get interesting." No it isn't.

I realise we are sailing into choppy waters here. With Larsson now dead and so decent a chap, how dare I go up on deck and start explaining – amid the storms of publicity and howl of Hollywood and the relentless sluicing of the sales – that his work is not very good even by the standards of his genre? Well because, in my view, we need urgently to remind ourselves of – for want of better terminology – the difference between literary and genre fiction; because, to misquote the literary essayist Isaac D'Israeli, "it seems to me a wretched national compulsion to be gratified by mediocrity when the excellent lies before us".

We need to be clear-eyed here because although there is much written about this subject, there is also much theatricality to the debate. And this serves to hide (on both sides) a fundamental dishonesty. The proponents of genre fiction are not sincere about the limitations even of the best of what they do while being scathing and disingenuous about literary fiction (there's no story, nothing happens etc). Meanwhile, the (equally insincere) literary proponents say either: "Oh, don't blame us, it's the publisher's fault – they label the books and we really don't see the distinction"; or, worse, they adopt the posture and tone of bad actors delivering Shakespeare and talk of poetry and profundity without meaning a great deal or convincing anyone. Both positions are bogus and indicative of something (also interesting) about the way we talk of literature and culture more widely.

It's worth dealing with the difference again, since everyone seems to have forgotten it or become chary of the articulation. Mainly this: that even good genre (not Larsson or Brown) is by definition a constrained form of writing. There are conventions and these limit the material. That's the way writing works and lots of people who don't write novels don't seem to get this: if you need a detective, if you need your hero to shoot the badass CIA chief, if you need faux-feminist shopping jokes, then great; but the correlative of these decisions is a curtailment in other areas. If you are following conventions, then a significant percentage of the thinking and imagining has been taken out of the exercise. Lots of decisions are already made.

So it follows that genre tends to rely on a simpler reader psychology. If you have a body on the first page, then you raise a question: who killed it and how did it get there? And curiosity will power readers a surprisingly long way. As will, say, a treasure hunt (Brown) or injustice (Grisham) or the locked room mystery format (Larsson). None of this is to say that writing good thrillers is easy. It is still incredibly difficult. But it is easier.

These are the reasons, too, why a bad thriller or detective novel or murder mystery will feel so much better than a bad literary novel – why it might even thrive. Even in a bad genre book, you've still got the curiosity and the reassuring knowledge that the writer will eventually deliver against the conventions. Bad literary fiction, on the other hand, is mostly without such fallback positions and is therefore a whole lot worse.

To enlist a comparison, one might choose to set up a vast and international burger chain and sell millions of burgers. Or one might choose to open a single restaurant selling line-caught eel lasagne one night and hand-fondled quail poached in liquorice the next. We all like burgers – me as much as the next man – and there is absolutely nothing wrong with that. But let's be honest: there is a major difference in both the production and the consumption of the two experiences. Again, we can see why bad literary fiction is so much more annoying than bad genre. We pay more attention to the restaurant that claims to have carefully sourced its ingredients and then used skill and imagination to bring those to the table in a manner that is original, surprising, beautiful, clever and delicious. Failure in this second case, therefore, is far more irritating. But equally, if you are in the burger-selling business, then although your burgers may appear different – you can flip them with bacon or jalapeño or even Stilton – the truth is that they are all fundamentally the same; you are in the burger business or you are not in business at all.

This is why genre writers cannot claim to have everything. They can take the money and the sales and all that goes with that. And we can sincerely admire them for doing so. But they should not be allowed to get away with suggesting that these things tell us anything about the intrinsic value or scope of their work. Here, for example, is Lee Child talking the kind of ersatz machismo bullshit that so confuses the issue: "The thriller concept is why humans invented storytelling, thousands of years ago. [Is it?] The world was perilous and full of misery, so they wanted the vicarious experience of surviving danger. [Did they?] It's the only real genre and all the other stuff has grown on the side of it like barnacles. [Really? Barnacles?] I could easily write a work of literary fiction. [No you couldn't.] It would take me three weeks, [No it wouldn't] sell about 3,000 copies [Doubt it] and be at least as good as the competition. [Absolutely no chance.] But literary authors can't write thrillers. They try sometimes, but they can never do it. [Crime and Punishment.]"

I'd love to end this piece by dealing with the fallacies of relativism, exposing the other misconceptions surrounding both genre and literary fiction (class needs tackling) and then round the whole thing off with a series of extracts from any number of fine contemporary novelists whom I love – Franzen, Coetzee, Hollinghurst, Amis, Mantel, Proux, Ishiguro, Roth – to illustrate again the happy, rich and textured difference. But there's simply not enough space. Our culture is ever more congested – for lots of good reasons as well as bad. There's huge pressure on books, particular pressure on fiction, and the most pressure of all on literary fiction. And yet, the English language, not football, is our greatest gift to the world. So, if we are to save our excellence in this from its slow extinction, then we simply have to find a way to bring the finest writers of the language more often to the attention of the carriages of people up and down the country who are evidently still willing and able to buy novels for the journey. Because right now – as you read this – they are being subjected to an atrociously bad (and translated) exchange between character A and character B on a broken-down Swedish boat about the establishment of a Polish industry for the manufacture of packaging foodstuffs. Surely they deserve better. No?

Edward Docx's latest novel, The Devil's Garden, is published by Picador in April