Amid the ambient wails of doom about the publishing industry, I’d like to enter a note of encouragement. The mainstream may be getting dumber by the day, but we are living in what looks like a golden age of publishing for, of all people, the university presses.

At the moment, I don’t think there’s a trade publishing house producing high-calibre, serious non-fiction of the quality and variety of Yale University Press; and snapping at its heels are Harvard, Oxford, Princeton, Cambridge and Chicago. As the literary editor of a middlebrow news magazine I’m finding ever more of the reviews I commission are from such presses.

Where 15 or 20 years ago the big trade publishers were, oddly, swamping the market with sort-of-scholarly micro-histories of salt or longitude, they now seem, with exceptions of course, to be tiptoeing away from specific, knotty, deeply researched and nuanced books about things. The sorts of book on which they tend now to rely are investigations of “big ideas”. Their lodestars or exemplars are the Malcolm Gladwells and Daniel Kahnemans and Nicholas Carrs. I do not mean to denigrate those individual authors, rather to say that they produce a particular type of work.

A ‘big ideas’ man … Malcolm Gladwell. Photograph: Michael Loccisano/Getty Images for HBO

These are talking-point books. They are easily imitated; their headline conclusions tend towards the categorical, and can be summed up in a dinner party one-liner or a 900-word newspaper op-ed; they lean on a vogueish but vague pop-theoretical or neuroscientific framework. They often have in their titles, or imply, a big question answered; or they have a flavour of self-help or how-to. They like to offer us things in the forms of lists. And – did I say this before? – they are easily imitated. Because imitated they are. From William Carlos Williams’ notion of “no ideas but in things”, we’re moving towards “no things but in ideas”.

The imitation books are, unsurprisingly, not as good as the originals. Their argument often seems appropriate to a compelling 10,000-word magazine article, but has been expanded to fill a full-length study. We have a flock of books arguing that the internet is either the answer to all our problems or the cause of them; we have scads of books telling us about the importance of mindfulness, or forgetfulness, or distraction, or stress. We have any number about what one recent press release called the “always topical” debate between science and religion. We have a whole subcategory that concern themselves with “what it means to be human”.

These are not books that always make our understanding of the world deeper and more complex. They are books that argue that one equation will make sense of economics for us, or tell us why human societies prosper or fail; that “the modern world” came about because of six ideas, or one historically traded commodity, or one decisive battle; they tell us that the key to life on Earth is X, Y or Z. The disproportion between the size of the question proposed, and the simplicity of the pretended answer, is the primary marketing hook.

They are, at best, semi-serious books dressed up as marketing propositions, and, at worst, marketing propositions dressed up as semi-serious books. A classic recent example – and a particularly wretched one, at least according to Steven Poole, who reviewed it in the Spectator and Bryan Appleyard, who reviewed it in the Sunday Times – is Clotaire Rapaille and Andres Roemer’s Move Up: Why Some Cultures Advance While Others Don’t, in Poole’s words a“rambling pseudoscientific argument” based on crudely archetypal “national characteristics”. It left a few of us wondering aloud whether Allen Lane’s in-house conscience – the admirably serious-minded Stuart Proffitt – might not even now be wriggling in the stationery cupboard with gaffer tape over his mouth and his limbs secured with climbing rope.

This is, of course, to cherrypick. Academic presses still publish lousy, abstruse, jargon-heavy books; and Allen Lane, Bodley Head, 4th Estate, Picador, Bloomsbury, Profile, Atlantic, Granta and Little, Brown continue to publish great ones, in history, nature, biography – all subjects. But it’s an impression that’s not mine alone, and it might indicate a direction of travel.

These days I’m very seldom excited by a trade non-fiction title, roaring as most of them are down the middle lane of the same motorway, to the degree I’m excited by the original and vital byways that university presses are exploring for the general reader.

Sue Prideaux’s award-winning life of Strindberg was published by Yale.

In natural history and popular science, alone, for instance: Hal Whitehead and Luke Rendell’s amazing book The Cultural Lives of Whales and Dolphins or Brooke Borel’s history of the bedbug, Infested, or Caitlin O’Connell’s book on pachyderm behaviour, Elephant Don, or Christian Sardet’s gorgeous book Plankton? All are published by the University of Chicago. Beth Shapiro’s book on the science of de-extinction, How to Clone a Mammoth? Published by Princeton. In biography, Yale – who gave us Sue Prideaux’s award-winning life of Strindberg a couple of years back – have been quietly churning out the superb Jewish Lives series. Theirs is the new biography of Stalin applauded by one reviewer as “the pinnacle of scholarly knowledge on the subject”, and theirs the much-admired new life of Francis Barber, the freed slave named as Dr Johnson’s heir. Here are chewy, interesting subjects treated by writers of real authority but marketed in a popular way. The university presses are turning towards the public because with the big presses not taking these risks, the stuff’s there for the taking.

It was only when I remarked on this to Toby Mundy, founding publisher of Atlantic Books, that I got a coherent sense of why this might be. Mundy is the newly appointed director of the Samuel Johnson prize; he knows how publishing works. And he explained that the changing economics of the industry are to blame; and that, in a way, my “note of encouragement” may not be all that encouraging after all. “This is part of a bigger story,” he says, “which is a general crisis of non-fiction publishing. With print books, the more copies you print, the lower the unit cost, and in these times sellers are very wary of stock risk.” He describes an arms race between inventory management technology and reprint technology in which the former is winning: Amazon and Waterstones will simply order a small run of books and order more only when they run out.

So the upfront costs of non-fiction – plates, photographs, indexing and subsidising the research – are becoming increasingly out of proportion to the likely payoffs. More than that, ebooks – which publishers love for obvious reasons – are only a very small part of the life of serious non-fiction.

“Enter the university presses stage left,” says Mundy. “Most are non-profits – they don’t require the 15p in the pound margins that the likes of Penguin Random House do. They often operate globally. Academic presses love librarians, and university libraries still buy books. And they are able to be much bolder with higher cover prices. So they are able to take those risks. Big conglomerates are very effective brand management systems – they can make comedians into novelists. And they can sell the big names – the Pinkers and the Schamas and so on. But the nursery slopes – the future stars – are now more and more on the lists of the university presses.”

• Sam Leith is the literary editor of the Spectator and a former judge of the Samuel Johnson prize for non-fiction.