I'm going to be talking here about the literary establishment. You know who I mean. Don't you? Well, even if you don't, you certainly know what I mean. You don't? Come to think of it, neither do I. It's one of those expressions and concepts whose rhetorical potency and convenience derive, imprecisely, from the fact that no one stops to think whether it means anything at all – any more than a squash player pauses to consider why there's a wall at the back of the court.

Here is a classic example, from a book of criticism published in 1986, in which we learn that "the literary establishment does not see its confusion" – about the status of the writer John Berger – "in these terms". The author of these lines detects a whiff of conspiracy without being able to say with any certainty – as is the case with all the best conspiracies – who the conspirators are or, for that matter, exactly what form the conspiracy takes or what it is intended to achieve. I say this with more than usual certainty because the author of these lines was my 27-year-old self.

Bringing things up to date, Jonathan Coe was one of a number of writers invited to pick their books of the year in these pages. He prefixed his choices by praising the Booker judges who "chose a diverse and challenging shortlist and then, having royally offended the literary establishment by excluding so many of their current favourites … "

Again, who and what, in an admittedly off-the-cuff aside, did Coe have in mind? He didn't mean himself, presumably, because that would have made no sense, though to be asked to contribute to these lists could be a sign of establishment membership. In fairness to Coe, although he won the Samuel Johnson prize for a lobbying biography of BS Johnson (who, as Coe puts it in the entirely on-the-cuff context of the book's introduction, "was always angry, and hurt, and unhappy at his treatment by the literary establishment"), he has since retired from book reviewing and critical reputation-making to concentrate on his own fiction (which could be taken as proof that someone has succeeded in ascending/absconding to the loftiest perches of the literary establishment). The important thing is the unspoken assumption that this literary establishment – whatever it is – is a bad thing.

Reflecting on some of the ideas in his great book Culture and Society, Raymond Williams observed that it was only when he "realised that no one ever used [the word] 'community' in a hostile sense" that he became suspicious of it. On that basis we should be suspicious of "the literary establishment", because it is only used in a negative way – like a squash court wall again, something that exists in order to have stuff hurled against it.

So what are the characteristics of this impregnable wall? At the risk of tautology, the first thing to be said about the establishment is that it is unique to literature. One does not speak of the music, art or film establishment ("Hollywood" is the preferred target for the cinematically disgruntled and marginalised, shorthand for a lack of the cultural snobbery that is assumed to be one of the hallmarks of the literary establishment). It is taken to mean a cabal of aesthetically conservative power-brokers and reputation-makers, preservers and preventers.

If we had to visualise this establishment, it would resemble an Edwardian board of aesthetic censors presided over by a stern TS Eliot–type figure inherently hostile to innovation (the irony, of course, is that Eliot was the greatest revolutionary in modern poetry) and keeping a perpetually wary eye on the likes of a Terry Eagleton or a James Kelman. It is often said to be London-centric, dominated by white males who had the privilege of attending the ancient universities of Oxford and Cambridge. One of these Oxbridge-educated London-dwellers enjoyed the privilege of interviewing Kelman on stage in 1987. The occasion was the publication of Kelman's Greyhound for Breakfast, before the Booker-winning How Late It Was How Late. When I asked Kelman about his literary standing, he claimed he had been "marginalised". But, I responded timidly, his book had been enthusiastically and widely reviewed – and he was speaking to a packed and doting audience at the ICA. Yes, he snapped, "I've been colonised." A superbly balanced answer in the way Linford Christie was famously described as well-balanced by Derek Redmond: because he had a chip on both shoulders.

Unpicking things more closely, the establishment presumably comprises literary agencies and publishing houses, some of whom have more sway than others. In London these publishers would include Jonathan Cape and Faber; in New York, Knopf and FSG. At a human level there are the editors at these houses, and the literary editors of papers and periodicals (some of which wield more of what my late Italian publisher termed "power-clout" than others), who decide which books to review and who to ask to review them. Then there are the people who get asked to sit on prize-giving panels, who decide which books to honour, and academics within the English departments at universities who decide which books to teach and canonise. Hang on, I feel sure I'm forgetting at least one other important category of person. Ah, right, stupid me … the writers!

Clearly, these groups are not entirely distinct and there is a good deal of movement between them. Writers write reviews, serve as judges, even start their own imprints with agents, and send emails to literary editors saying they'd like to review this book, or that another book should be reviewed by someone, or they write to publishers suggesting that a book that has fallen out of print might be considered for reissue. A lot of these people know each other in the same way that bricklayers in a given town will tend to know the other bricklayers (and plumbers and plasterers, and the people who supply building materials). Membership is not fixed, it changes all the time as new voices and minds emerge and old brains lapse into silence and senility. It is not politically aligned, though it tends to be left-liberal leaning and predominantly atheistic.

Beyond that, I don't detect anything monolithic or impregnable about this literary establishment except a belief in the importance of spelling and punctuation. It is more that an accumulation of voices – some of which make themselves heard more effectively and frequently than others – results in a consensus of opinion that is all the time being tested and contested. Most important, this mythic beast, the literary establishment, consists of people lamenting not only the bias, misjudgments and stupidity of other people who are part of it but – as we have seen – its very existence.

Complaints about the poor taste, flawed judgment and provincialism displayed in the literary world are not the war cries of a lone Apache looking in vain for some weakness in the circled wagons of the establishment – they are the quarrelsome sounds emerging from the scattered wagons themselves. To drop the metaphor, they're the sounds of people who care about books talking to and about other people who care about books. There is no such thing as the literary establishment. I know this because I am part of it.