Insomnia is a silent epidemic. Reports suggest that 30% of adults fail to get sufficient rest, more if you look at over-65s, while at the other end of the scale schoolchildren are struggling to sleep through the night. Medical professionals blame our over-stimulated lives for this plague of sleeplessness, calling out our lack of boundaries between work and rest, environmental disturbances such as noise and light pollution and our addiction to screens.

Yet what qualifies as common sense is not always accurate. One sleep expert told me that clinical experiments that exposed people by night to blue light (the kind our mobiles emit) only delayed the onset of sleep by 10 minutes. Still, not sleeping is bad for us. It causes confusion, impairs cognition and increases anxiety; exacerbates underlying gastric, lung and heart problems; makes us fat, stupid and depressed.

My sense that insomniacs are disillusioned with the language of affliction and cure persuaded me that they’re sated with meditation and CBT, “sleep hygiene” and pill-popping, and hungry for something more probing. My book Insomnia is an attempt at a corrective. By portraying my wakefulness from the inside, and recording it in the lived moment without censoring its unsettling, feverish and contrarian detail, I hope I posed difficult questions about why so many of us cannot sleep – questions that led me to ponder the nature of desire and the habits of the human mind.

These are some of the books I reached out for when sleepless, to help me make sense of how being up at night speaks to our innermost fears, but also taps into the wellsprings of creativity and longing.

1. Warm and Snug: The History of the Bed by Lawrence Wright

For insomniacs, sleep possesses the sheen of an unattainable ideal. No longer pedestrian or merely functional, it gets burnished with symbolism. In looking at how we’ve made a fetish of sleep, then, everything points to the bedroom. Wright’s old-fashioned history, chronological and dutiful, describes every kind of bed you can imagine; palette beds, bunks and box beds; beds that are curtained, pillared and hoisted up on mounts, kingly beds, merchant’s beds and pauper’s beds, dormitory beds, hospital beds and modern mattresses. Wright finds it incredible that so simple an invention as the modern mattress was such a long time coming: attendant on the coil spring, invented in 1857. Paradoxically, greater comfort did not engender more rest.

2. At Day’s Close: A History of Nighttime by A Roger Ekirch

This is a book to raid and pillage, rich with stories of wakeful nights, drawn from the diaries and letters of insomniacs past. Samuel Pepys emerges as a star in recording his nocturnal escapades (a lot of banter and disputation; sometimes reading, sometimes getting a haircut, sometimes consuming victuals with a friend). Ekirch’s book is notable for busting the myth of the eight-hour solid stretch of sleep. His discussion of the pre-modern habit to take a first, then a second sleep, bedding down as darkness fell, but then rising at dawn for a couple of industrious hours before a much-needed top-up, swiftly began to look to my tired eyes like some pre-capitalist nirvana.

3. The Third Reich of Dreams: The Nightmares of a Nation 1933-1939 by Charlotte Beradt

Beradt was an unlikely sociologist. A Jewish journalist living and working in Vienna in the 1930s, she suffered endless nightmares of being “shot at, tortured, scalped”. It led her to wonder if her fellow countrymen and women, living in fear of the totalitarian Nazi regime, were similarly troubled in their sleep. So she began collecting their dreams. What emerges in this strange collection are the shared nighttime hauntings of a nation driven to paranoid self-doubt – and self-blame – by rigid intolerance and oppression. Beradt reveals a collective dreamscape of barely suppressed horror that makes for an unforgettable read.

4. Compass by Mathias Énard

Inspired (I’m guessing) by Proust, Énard’s magnificent novel is set on a single sleepless night, when it auto-fictional narrator is beset by lustful longings for a youthful love. Conceptually it’s brilliant. Between the lines, Énard seems to suggest, in language and plot, that insomnia is all about bridge-building: between East and West, land and water, day and night, consciousness and the unconscious, and finally with its dredging up of painful memories, past and present.

Mysteries of the night … Robert De Niro (left) and Robin Williams in the film version of Awakenings (1990). Photograph: Allstar/Columbia Pictures

5. Awakenings by Oliver Sacks

In 1917, a strange epidemic of sleep sickness swept the globe, causing thousands of people to collapse with raging fevers and wild hallucinations before falling into spells of extended sleep. Sacks worked with survivors of this epidemic 40 years later, after neurologists found that administering the drug levodopa could jolt the long-slumbering victims into periods of manic wakefulness. Some of them turned from real-life Rip Van Winkles into insomniacs, becoming enervated and hyper-alert, overly loquacious, overly familiar and, in one case, messianic.

6. The Fall of Sleep by Jean-Luc Nancy

You require a taste for French academic writing, with its circular, self-referential and unremittingly highbrow language to get on with this book. It reads like a reverie – as if the logic and vistas of dreaming had taken up residence in Nancy’s thinking mind. He had me, though, when he wrote that “insomnia is a kind of occupation of night”. It is – a trespass and presumption, an infiltration and colonisation. In fact, Nancy’s book is full of sharp little insights that chimed with me. I’ll give you another: “[In insomnia] I can’t sing or make a noise. So I have to be with myself, endure myself. And that is hard.”

7. Acquainted With the Night: Insomnia Poems, edited by Lisa Russ Spaar

Reading these poems was like popping vitamin pills – each offered the balm of companionability, a salve to my own troubled soul. Because what insomniacs lack is community; twisting awake in our own beds, we confront only ourselves. And what a collection this is. Russ Spaar places Plath next to Pushkin, Auden next to Bishop. Its rather apt conceit is that poets and insomniacs both pride themselves on being the ones who are awake enough to see the truth of things.

8. Night Passages: Philosophy, Literature and Film by Elisabeth Bronfen

Running to nearly 500 pages, this erudite, fact-filled study is a rare treat, rich with every kind of symbolic use of night that writers and artists have availed themselves of. Bronfen is especially interested in the moral dimensions of night: its temptations, what it veils, its reckonings and revelations. In a wonderful chapter on film noir, she examines the way providence manifests itself in our lives (and our stories) – providence being a dark art, because its ways are unknown and cannot be seen, as opposed to the artificially lit world of noir, which illuminates the antihero’s desire to manipulate events. This book delights in the potency of border-crossing and trespass, the wiles of nocturnality.

9. The Poetics of Sleep, from Aristotle to Nancy by Simon Morgan Wortham

The title of this book nods at Gaston Bachelard’s The Poetics of Space – space, like sleep, being a quality we cannot grasp; another absence. Wortham asks how we can speak of a philosophy of sleep, given that philosophy studies consciousness: the very thing sleep forces us to relinquish. The book is full of heavyweights: Kant and Freud, Merleau-Ponty, Levinas, Bergson, Blanchot. It asks what kind of thinking dreaming is, and what kind of absence sleep represents. Is there community in sleep? Reality in dreams?

10. Why We Dream by Alice Robb

Covering the latest breakthroughs in the science of dreaming, Robb argues that REM sleep is valuable, not least because it helps us process emotional trauma. Perhaps, then, insomniacs ought not to fret over sleep but dream loss.

• Insomnia by Marina Benjamin is published by Scribe, priced £9.99. It is available from the Guardian bookshop for £6.99.