In Geoff Dyer’s sixth book, Out of Sheer Rage, the narrator – who may be a representative of the author, a fictional persona, or a mixture of the two – is preparing to speak at an academic conference in Denmark. More exactly, he is failing to prepare. He has come down with what he calls “a virulent strain of the flu”. When he eventually arrives at the podium, clutching no more than a few scribbled notes, he begins to improvise, deconstructing the subject of his scholarly paper: “D H Lawrence and Englishness”.

“Let’s think about the two parts of this, uh, statement,” he says, alarmed to realise that people have started to take notes. “Do we mean he was a writer who was English? Or that he was an Englishman who wrote? Or both? And even when we concentrate on just one of these two terms we discover that that too is made up of two parts: English and man.” He continues to parse his thesis – that Lawrence is a problematic figure about whom it is difficult to say very much – for another 40 minutes or so, during which time his nose starts bleeding. Eventually he sits down, humiliated, coughing up blood in an attempt to provoke sympathy from a group of people he has spent a great deal of energy deriding and despising – those murderers of literature, academics.

And so I couldn’t help but smile on a recent, drizzly afternoon when an earnest, hyper-intelligent historian from Queen Mary, University of London, concluded a lecture entitled “What Colour Was the 1990s?” – an echo of Dyer’s debut novel, The Colour of Memory (1989) – by putting his head in his hands and confessing: “I suppose I don’t know what I mean by the 1990s. I don’t know what I mean by colour.”

It was a brave and welcome admission. I had no idea what he meant either. The lecture was the eighth in a day-long series of talks being given at the first international conference dedicated to the work of the English writer Geoff Dyer, held on 11 July at Birkbeck, University of London. It was a happy occasion, only complicated slightly by the fact that Dyer himself was seated in the back row throughout the day, taking notes with a yellow pencil on a floppy white pad. The main problem was one of etiquette. “When speaking about the work, use Dyer,” urged Dr Bianca Leggett, the convenor of the conference, in her opening remarks. “When speaking about the man in the room, use Geoff.”

For the next eight hours a series of professors and researchers took their turn in front of the audience to systematically abuse this rule. One speaker, whose paper focused on a great but unfinished opera by the name of Roggerio, felt no need to mention him at all. All the while Geoff sat silently, taking notes. “It’s the only time I’ve ever been in any kind of seminar where I have been unquestionably the leading authority on the subject,” he said later. “I’ve got so much to say, I can hardly wait.”

I first met the man in the room ten days earlier at an art fair in Chelsea, where he was chairing a discussion with the Hackney-based photographer Tom Hunter. There was champagne before the event, which I sipped while reading about Thomas Piketty in a complimentary issue of Vanity Fair. I was glad to see Dyer, who is 56, looking well, after hearing about his ischaemic stroke earlier in the year, which had temporarily impaired his vision. “I’m absolutely fine,” he told me, thanks to statins and “cutting down on pastries”. I asked him how he was feeling about the upcoming conference, especially given his protest in Out of Sheer Rage (1997) that the hallmark of academic criticism is that: “It kills everything it touches.”

“You have to bear in mind that book came out a long time ago,” he said, heading for the exit to make it home in time to watch the World Cup. “They are not ex cathedra truths, they are circumstantial.”

As one of a handful of non-academics attending the conference, I rolled my eyes when, after turning down a name-tag, an academic told me he respected my Dyer-esque refusal to be labelled. “Is it going to be like this all day?” I asked. I rolled them again every time a word like “polymorphous”, “interstitial” or “syllogistic” was deployed in attempt to pin down the genre-smashing nature of the Dyer canon – a total of 14 books made up of memoir, fiction, essays, criticism and travel writing, often within the same volume.

“One of the great privileges of my life was growing up in a house without books,” Dyer told me, three days before the conference in the study of his book-filled flat in Lad­broke Grove, west London (“We’re a no-shoe household, so wear your favourite socks,” he warned me in advance). “When I finally discovered them it was so totally transformative. This is my life’s work here, all these,” he said, pointing to the shelves.

Geoff’s mother, Phyllis “Mary” Dyer, was a school dinner lady; his father, Arthur “John” Dyer, a sheet-metal worker. (Both of them died in 2011.) As a student he progressed without great difficulty through Cheltenham Grammar School to Corpus Christi College, Oxford, where his disillusionment with academia took root. “Contrary to popular belief Oxford has the highest concentration of dull-witted, stupid, narrow-minded people anywhere in the British Isles,” he wrote in Out of Sheer Rage, complaining of the “dimwit academics shovelling away at their research, digging the graves of literature”.

While this Bernhardian rant may have been trotted out for comic value, Dyer’s distaste is genuine. “There’s something awful about Oxford, I think. It’s such a little ghetto. My own trajectory has been so opposed to the standard path of education in this country, which is always moving towards finer and finer specialism. The paradox is that the ability came from that Oxford way of being taught – ie, not being taught but finding out everything for yourself.”

If the ability came from Oxford, the confidence to use it came from John Berger, the painter-turned-polymath – and art critic at the New Statesman in the 1950s – who became a kind of mentor to Dyer and the subject of his first book, Ways of Telling (1987). This combination of the autodidact’s mania for learning coupled with Berger’s insistent refusal to delineate fact from fiction produced books including But Beautiful (1991), a retelling or improvisation upon the lives of jazz legends, and The Missing of the Somme (1994), an extended essay, with all the deviation readers have come to expect from Dyer’s work, on the cultural memory of the First World War.

These examples typify his approach to non-fiction: gorging on a specific subject for a year or two then moving on. “The person doing the learning is the person writing the book as much as the person reading it,” he wrote in his 2005 book about photography, The Ongoing Moment.

That these early books did not sell particularly well may not come as a surprise. “This is exactly how you do not go about establishing a successful literary career, or a brand,” Dyer said in the Q&A that concluded the conference. It did, however, remove any pressure to please an existing audience – “Because there was none!” – allowing him to follow the digressive course of his curiosity and build a reputation over time. “Lawrence said that one sheds one’s sickness in books,” he writes in the final pages of Out of Sheer Rage. “I would say one sheds one’s interest.” The same holds true for his most recently published book, a record of two weeks spent as writer in residence on board a US navy aircraft carrier: the USS George H W Bush.

Another Great Day at Sea (2014) immerses the reader in a disciplined, martial, rubber-and-steel universe where deviation from the chain of command is unthinkable and “lessons are written in blood”. “It’s not necessarily a dangerous business,” explains one pilot (nicknamed “Disney”), “just terribly unforgiving of mistakes.” It is arguably the occupation most opposed to Dyer’s: physical, regimented, with terrible food.

“I was definitely conscious of that when I signed up,” he told me. “You join up to the military and that’s the last free decision you ever have to make, whereas in the writing life all you’re doing is making decisions, about one word or another. As I wrote in the book, ‘It’s a self-discipline which is almost indistinguishable from self-indulgence.’ Those two ways of living mirrored each other quite nicely, I think.”

It’s also the place where his admiration for all things American shows most clearly. The writer in his “cranial” helmet and “float coat” appears every bit as doltish, charming and naive as Louis Theroux in his early documentaries, only without the whiff of contempt. Dyer really does admire Americans – their dependable manners, commitment to liberty and overwhelmingly can-do attitude. If anything highlights the gap that has opened up between Dyer and Berger since they first met for a Marxism Today interview in 1984, it is Dyer’s refusal to argue with anyone on the ship. “I am not a confident debater,” he says in his defence.

Another thing he claims not to be is a reporter. He describes himself as a tourist: “a marine anthropologist whose data was so thoroughly and distortingly mixed up with the means of obtaining it that it probably had no value as data”. He avoids winding up on a British ship, fearing “the accents, the audible symptoms of the top-to-bottom, toff-to-prole hierarchy that is so clearly manifest in the British military”. His love of America, where he and his wife, the Saatchi Art curator Rebecca Wilson, now spend most of their time, away from the toxicity of class and British negativity, is not new.

“A key part of my formation as a writer is that I didn’t read people like Evelyn Waugh until later on,” he said. “I decided as a protest I would never read Kingsley Amis, because he was always saying he was never going to read people like Saul Bellow and Joseph Heller, who I read instead.” Amis is a surprising omission given that his debut novel Lucky Jim (1954) contains, in Jim Dixon’s “Merrie England” speech, the best-known conference parody in literature.

Which brings us to the question that most intrigued delegates at the inaugural Geoff Dyer conference, that of “Geoff Dyer and Englishness”. One participant argued his work was un-English; another, the University of East Anglia’s Jonathan Gibbs, praised him for “putting on page the shivering rage that never comes out at the world”, a symptom of “weapons-grade Englishness”, while a French audience member stated that though she loved his books, her English friend couldn’t stand them. The English friend happened to be sitting next to her and began shaking his head in Dyer’s direction, eyes wide in alarm. They all agreed that the essence, whatever it was, lay in his “vinegary” sense of humour.

Dyer answered with a quote from Lawrence’s letters: “English in the teeth of all the world, even in the teeth of England,” suggesting that hating England was perhaps every bit as English as identifying with it. Maybe more so.

At the end of the day Dyer conceded that he might have misconstrued the masochistic impulses of academia, happily proposing the inauguration of “an annual, weekly, not conference but festival devoted to me”. Afterwards we all disappeared to the pub, which everybody knows is the real purpose of these intellectual get-togethers: to share a room with people who harbour the same obsessions you do, then drink them under the table. So what next for Geoff? Ironically, teaching. In the past few years he has taught non-fiction workshops and reading courses (“This isn’t a course, it’s just a book group,” one student objected) at Iowa and Columbia universities and will move on to Austin, Texas, next year.

“Teaching is great: I enjoy the sort of parasitic, vampiric relationship I have with the students. One problem, though, is they all want to write memoirs,” he says. His own reading these days is mostly “straight-down-the-line history books”, thousand-page tomes by Pulitzer-winners such as Taylor Branch and Richard Rhodes.

“I’m one of the people who seem to have licensed the ‘I’m meant to write about this book but I’m just going to write how I got stoned instead’ essay – but it only works for certain subjects. It has to lead you into a deeper appreciation of the subject than could have been attained in a more direct way. It’s like those legal highs,” he said. “Some of them can get you pretty messed up. Really they ought to be proscribed.”

Geoff Dyer is a judge for the 2014 Goldsmiths Prize for Fiction, in association with the New Statesman

Portrait by Eva Vermandel for New Statesman