Robert Silvers, the editor of the New York Review of Books, once told a journalist rather wistfully that he doesn't have "a very full sense of time". What he meant was that he is one of those rare people for whom the passing of the years is almost an irrelevance; the decades don't pile up behind him, burdensome and melancholy, for the simple reason that he's apt only to look forward. Silvers, who is 84, has been at the helm of the NYRB, the paper he helped to found, for more than 50 years and yet every issue – every article – remains as important to him as the last. His engine is, and has always been, a compulsive curiosity and it just so happens that his job – what an amazing piece of luck – can take him from the West Bank to cutting-edge developments in neuroscience to the Booker prize in the course of a single, head-spinning morning.

But perhaps this quirk of character applies to timekeeping in the more usual sense, too. In New York, people tell me in slightly awed tones that Silvers works such long hours that he has two assistants, one to work the day shift, the other to stay on late into the evening. There is talk of a bed he reputedly keeps in a cupboard, and a newspaper cutting reveals that he has been known to telephone his writers with queries – "On column six of the third galley, there's a dangling modifier" – on Christmas Day.

My encounter with him is to take place at the NYRB's West Village offices at 3pm on a Sunday afternoon, and though this suits me just fine (Silvers knew Auden, Mailer and Mary McCarthy; I would meet him in Lincoln, Nebraska at three o'clock in the morning if this was what he wanted), as I cross the lobby, I can't help but feel a stab of regret. The city is bathed in the soft sunshine of early summer. The last thing I hear before the lift doors close is the laughter coming from the outdoor tables of the restaurant across the street and the stubborn thunk of a basketball as it hits the pavement.

Upstairs, I wonder if I've been forgotten. There is an atmosphere of weekend abandon. After a moment, though, one of the famous assistants arrives at the door, offers me a somewhat frail smile, and leads me into an airy, loft-like space with sofas, a coffee table and a crammed bookcase that belonged, according to its label, to Barbara Epstein, Silvers's co-editor and dear friend (she died in 2006, and, as well as the usual American suspects, seems to have liked George Gissing and Stevie Smith). Soon after this, Silvers appears. He is wearing a shirt and tie and bearing what we might call the sacred texts around which the religion of the NYRB is built: a facsimile of the first edition in which its editors announced their intention not to waste space on reviews of books "which are trivial in their intentions or venal in their effects, except occasionally to reduce a temporarily inflated reputation or to call attention to a fraud", and a copy of the famous piece – The Decline of Book Reviewing, which appeared in Harper's in October 1959 – by Elizabeth Hardwick, a critic and novelist, which was its (the NYRB's) inspiration. Some passages in this essay, I will see later, have been underlined: in particular, the paragraph in which she calls the New York Times Book Review a "provincial literary journal" full of "flat praise and faint dissension" and "light, little article[s]".

If this were anyone else, I'd be insulted. Naturally, I was poring over these very documents only last night; Hardwick's piece, in particular, retains all of its power, not least because what she says is as true now as it was then. "Sweet, bland commendations fall everywhere upon the scene," she wrote of America's newspaper and magazine book pages, apparently oblivious to the chilling effect her words might have on her own reviewing career (the New York literary scene is nothing if not incestuous). "A universal, if somewhat lobotomised, accommodation reigns. A book is born into a puddle of treacle; the brine of hostile criticism is only a memory." Crikey. She would have been horrified by today's book pages, wouldn't she? Silvers smiles. "Yes, she would," he says. "But Lizzie was often appalled. Sometimes, she was appalled at what we did. She would say, 'Honey, you've got to do better than that.'" He weighs her essay in his hand. "I constantly think of her piece. The list of things that depressed her: the 'faint dimension' and the 'light, little article'. Sometimes, I say to myself: is this a light, little article we're doing?"

Silvers, centre, with the Paris Review’s George Plimpton, right, circa 1966. Photograph: PBS

There is surely no danger of that. In the current issue of the NYRB, you will find Claire Messud on Albert Camus, Joyce Carol Oates on Larry McMurtry, John Banville on Edward St Aubyn, Ahmed Rashid on America in Afghanistan and an essay on Ukraine by Anatol Lieven – and this is what I would regard as one of its softer numbers. The offer of such an embarrassment of riches is wholly amazing in a world where print journalism increasingly operates in the most threadbare of circumstances; editors everywhere must wish that their journals were owned by its proprietor, Rea Hederman who, since he bought the NYRB in 1984, has given his journalists absolute freedom (the paper that in the 60s famously ran a cover featuring a diagram showing how to make a Molotov cocktail still reserves the right to question the activities of Israel and to read congressional reports as closely as if they were Victorian novels).

But what truly makes it astonishing are the numbers. Its readership is healthy (around 150,000); it has been in the black since 1966; it has spawned a marvellous literary imprint (NYRB Classics) that has put hundreds of wonderful books back on our shelves. No wonder it is held in such great, almost preposterous, esteem. How great? Well, understand that we are not only talking about Harvard professors and Upper East Side shrinks. Martin Scorsese's next picture, A 50 Year Argument, is an hour-long documentary about the paper. It will receive its UK premiere at the Sheffield Documentary festival tomorrow.

How did this film come about? "It was unexpected," says Silvers, who has delicious old-school manners, but whose voice is surprisingly loud. "When we were facing our 50th year [in 2013], we had a number of people who wanted to make documentaries. We talked to them and they were extremely nice. But the question was: what did they really know about the paper? Then someone told me Marty might be interested. I didn't know him, but I know his wife. So I wrote him a note. He said he was interested. He'd been reading the paper since 1963, when he was at NYU. He knew the articles extremely well and he had a big pile of the paper sitting in some room. Fortuitous! He's a marvellous fellow. His staff did many interviews with our writers, and they came to a party here, and to a conference we had in Oxford on Isaiah Berlin, and they got some footage of Mary McCarthy going to Vietnam [the paper sent her to report from Saigon and Hanoi in 1967] and of Occupy Wall Street, which we covered, and they put together their conception of the Review." Does it match his? "It can only be, for me, a slice of the paper. We have had about 15,000 articles. Philosophy, physics, art, ancient history, poetry: all this could not come into a one-hour film. But I think it is a very interesting take. I don't have any quarrel with it. There's a fairness to it."

I find it hard to picture this film. How to animate the quiet, rather dusty world of book reviewing? Silvers, though, doesn't see it like this at all. "The book review sounds like it might be quiet, but it can be extremely intense and revelatory. It can have a lot of power – or that's what you hope for."

The first issue of the New York Review of Books. Published in 1963, it sold out its 100,000 print run.

The NYRB was born during the New York newspaper strike of 1963; Silvers was then an editor at Harper's. It happened like this. Elizabeth Hardwick and her husband, poet Robert Lowell, were having dinner with Jason Epstein, a publisher at Random House, and his wife, Barbara, a writer and editor. Epstein knew how much the strike was hurting publishing – with the New York Times Book Review closed for business, there was nowhere to advertise, and no way of telling readers about new books – and is said to have turned to his wife and friends and said: "Kids, let's put on a show!"

The group would establish its own ruthlessly highbrow journal, an ongoing counterblast to the soporific blandishments of which Hardwick had complained. They asked Silvers, who had commissioned Hardwick's piece, to edit the journal alongside Barbara Epstein. "Jason had the immediate insight that this was the only time in history you could start a review for no money because the publishers were going crazy for somewhere to advertise. The three of us – Barbara, Lizzie and me – went into the offices of Harper's at night, and picked out books, and sent them the next day to people we really admired, many of whom hadn't written a review for years." The first issue included pieces by Auden, John Berryman, Mailer, Adrienne Rich, Susan Sontag and Gore Vidal.

Why, though, has the NYRB thrived when so many have faltered? "I don't know! That is a big mystery. When we started, our publisher said we should have a survey to find out what readers want. But Barbara and I both said, no, we must pick the subjects and writers we believe in; we won't take dictation. If it's interesting, people will go on subscribing. If it's not, they'll say: to hell with it."

So editing is about instinct? "No, the opposite. It's about reading book after book, and thinking of writer after writer, and then saying, no, no, no… yes. You must ask: who will be the best person in the world to write the best article on this subject? You must be interested in everything. Barbara and I thought from the first that no subject would be excluded. Someone is writing a piece about Nascar racing for us; another is working on Veronese."

His own curiosity about the world has, he insists, only grown with the years – and it's thanks to this alone that he works seven days a week. "It's not a question of love. I see this as a series of problems to be solved. An infinity. In the summer, I get away. But I always have manuscripts coming to me and I call the office every day." What time will he leave this afternoon? "I might leave by midnight – or not, it all depends." Does he ever go to parties? He laughs. "Yes, I do!" And then, unprompted, there comes this tribute: "For many years, my partner and I, Grace Dudley [aka the Countess of Dudley, widow of the third earl], have had a life together. She has meant everything to me: her intelligence, her perceptiveness. We're often together. Next week, I will visit her in Lausanne. But when she's not here, I just keep going."

Silvers was born in December 1929, in Mineola, New York. His father was a businessman who had retired to a farm on Long Island; his mother had been a music critic for the New York Globe. "They were both extremely encouraging, so when the chance came for me to go to the University of Chicago at the age of 15, they thought it was a good idea for me to get right off to college. My father was a man I always thought knew everything, I really did." Having graduated at the age of just 17, he got a job as press secretary to Chester Bowles, the Democrat governor of Connecticut. When Bowles failed to be re-elected in 1950, Silvers was drafted into the army to do his national service.

This, it turned out, would be the first in what he regards as a series of great adventures: sent to work at Nato headquarters in Paris, a new world opened up before him. "A little publishing house in New York, the Noonday Press, said, 'While you're in Paris, why don't you look around French publishing and see if there are any books for us?' So on Wednesday afternoons – my free day – I would go and visit publishers. But I also went to the Paris Review; I'd bought it on the newsstands, and I thought it was an interesting paper. That was how I met George [Plimpton, the Review's celebrated editor]. We sat down and talked for hours. He was the most charming, encouraging, friendly guy. Shortly afterward, he invited me to be the Review's managing editor.

"Marvellously, there was the GI Bill, which paid you enough to live on when you left the army, so long as you were registered at a university. So I kept going to French universities and the Review didn't have to pay me anything. It was a wonderful time. The city was in some ways poor, but it was very beautiful. There was the possibility of making all sorts of friends. At the Review, there was a feeling of adventure working with George. You felt that school was over and you were setting out on vacation. The great thing was living." He stayed in Paris until 1958, and might not have left at all if it hadn't been for the fact that he was offered a job as associate editor of Harper's (the magazine paid his passage back to New York).

Silvers has never married and has no children, though for a long time he believed he was the father of Ivana Lowell, with whose mother, novelist Caroline Blackwood, he had an affair (Blackwood, the daughter of the Marchioness of Dufferin and Ava, was the first wife of Lucien Freud; Ivana Lowell wrote of her sadness at discovering that Silvers was not her father in her 2010 memoir, Why Not Say What Happened?).

But it would be a mistake to depict him as some kind of anchorite, the altar of his desk having kept him from the stuff of real life. For one thing, so much of that time was spent with Barbara Epstein, who was larky and mischievous and always there to make him laugh. (He misses her hugely.) For another, his deep admiration for writers puts him at the centre of a vast intellectual web. He does not want for friends, for he induces a kind of magnificent fondness – not reverence, exactly, but a warm, high-minded regard you find only exceedingly rarely in journalism – in all those lucky enough to have written for him.

Silvers with long-time NYRB co-editor Barbara Epstein in 1993. Photograph: New York Times/Redux/Eyevine

He is, too, so impressively involved in the world, albeit at a distance. While we're together, he roams almost casually from Zadie Smith to America's use of drones to the essays of Montaigne (the book he turns to most often). What was the last thing he read? "Teju Cole's novel," he tells me, in a flash. What does he make of Cole? "He's very interesting," he says. "You want to know what he thinks. I would love him to write about the missing girls in Nigeria for us [Cole is an award-winning Nigerian-American writer]."

Is Silvers a scary editor? He looks almost hurt. "Certainly not! We want the writer to achieve their best. That's all we want. I feel they have done something very generous, agreeing to take weeks out of their life to write for us and you owe the writer every consideration." Aren't they thrilled to be asked? He laughs. "Some are. Some feel quite grudging. 'I don't know if I can fit this in,' they say." How does he operate when so many writers are friends? What does he do when a bad review comes in of someone he knows? (In 1963, for instance, Silvers commissioned Norman Mailer to review Mary McCarthy's novel, The Group, and he savaged it, describing it as squatting on the Grand Avenue of the Novel like a shabby little boutique.)

"You've put your finger on something central. The act of reviewing can have a deep emotional effect. People get hurt and upset. You have to be aware of that, but you can't flinch." Does he spike a lot of pieces? "Not many. But the test is that you do, sometimes. You say, 'No, I'm terribly sorry, I can't visualise that in the paper. I don't think it's adequate to the subject.'" He is careful not to commission a writer whose work he has not already seen elsewhere, though, alas, this doesn't wholly eliminate the possibility that someone will file drivel. "It's unfortunate, but sometimes things have been rewritten elsewhere." How does he feel when he looks at the New York Times Book Review? "It would be wrong of me to be critical of the Times. I feel they've a terrible problem. They want to come close to being comprehensive, which means they're committed to reviewing a great many books that are probably rather second rate."

About the literary future, Silvers is more hopeful than you might expect. He relishes the global feel and reach of novels just now. The digital world often baffles him, and as he "stares" at various websites, he often wonders at their lack of quality. Would William Faulkner have messed with Twitter? He thinks not. But he is upbeat about the NYRB's digital subscriptions and tells me that he often runs longer versions of pieces from its blog in the paper itself.

Only when it comes to the delicate question of who'll succeed him does he seem unwilling fully to engage. There is Glenn Greenwald's new book to be tackled and the issue of "Mr Bezos", king of Amazon and the new owner of the Washington Post – and as if suddenly remembering all this, he gets up to show me out. En route, we detour at my request to take in his desk, which is huge and semi-circular, the work station of a Bond villain if it wasn't for all the books piled at its edges. For a moment, he rests the tips of his fingers lightly on some papers and then he shakes my hand.

• This article was amended on Sunday 8 June 2014 to correct mistakes that crept in during the editing process.