When I told people that I was going to write a book on memory, I saw "good luck with that" written on a few faces. Memory is a massive topic. Any intelligent system needs some way of tracking where it is in time, and that means remembering where it has been. No surprise, then, that studying memory proliferates into numerous sub-disciplines. You can specialise in short-term memory (memory traces that persist for a few seconds) or cast your net into memories that stretch back through an entire human lifetime. An essential distinction is between memory for facts (semantic memory) and memory for events (episodic memory). I was interested in a branch of memory research that straddles the two: autobiographical memory, or the memory we have for the events of our own lives.

Roughly four decades of research (with historical precedents that stretch back much further) tell us that this kind of memory is essentially reconstructive. A memory is stitched together in the present moment from several different kinds of information, in a process that's subject to the current beliefs and biases of the person doing the remembering. But surveys tell us that many people remain wedded to a view of memories as immutable, static possessions. Why do we get memory so wrong? One possible reason is that memories are precious to us: they define us in many ways, and so we react with discomfort to the idea that they are the constructions of a story-telling mind.

Although few scientists would quibble with the idea of the reconstructive nature of memory, there have been some hot new developments: in understanding the social dimensions of remembering, particularly in the very young and very old; in working out how memory functions in trauma and extreme emotion; and in linking remembering the past to thinking about the future, to imagination and to creativity. Keeping up with the latest research meant that I stuck mainly to journal articles when writing Pieces of Light. But several books, although sometimes a little out of date, had profound influences on my thinking about memory's slippery charms.

Memory has been a topic of fascination for centuries. For the long view on how humans have gone about studying it, this book by a Dutch historian of psychology is hard to beat. In poised, humorous prose, he ranges from the stories of respondents to an 1899 survey who had "flashbulb" memories of hearing of the death of Abraham Lincoln, to the diary study of psychologist Willem Wagenaar, who for six years wrote out a daily memory so that he could test his own forgetting, to the nineteenth-century French philosopher Jean-Marie Guyau, who asked why the years speed by more quickly as we age.

2. Searching for Memory by Daniel L Schacter

Harvard psychologist Schacter has been a leading figure in the cognitive neuroscience of episodic memory. In this, his first book, he provides a detailed and highly readable account of how memories are encoded, stored and retrieved, how remembering is damaged and preserved in amnesia, and how memories are distorted by trauma. Particularly interesting is his focus on how memory processes are depicted and interrogated by visual artists, although the pictures unfortunately don't reproduce too well in the paperback. While the field of memory has moved on a fair bit in the eighteen years since this was published, its erudition and scientific authority make it unmatched as an introduction to the study of autobiographical memory.

3. The Craft of Thought by Mary Carruthers

Memory was a big thing when books had to be copied out by hand. Building on classical ideas, such as the "method of loci" attributed to Simonides (think of a place and fill it with striking images corresponding to the items you want to remember), our medieval ancestors turned remembering into a developed art. Carruthers provides a brilliant critique of key texts such as Frances Yates's The Art of Memory, showing that medieval memoria was nothing less than a theory of the recombinative power of thought. Carruthers' dense, ambitious analysis of the medieval mind is an extraordinary work of scholarship.

In memory, we narrativise ourselves like novelists. Galgut's unsettling triptych of travel stories (shortlisted for the 2010 Man Booker Prize) play disarming tricks with perspective, as the Damon in the stories flips identities with Damon the narrator. Echoing psychologists' distinction between 'field' (first-person) and 'observer' (third-person) memories, Galgut makes austere, uneasy fiction from the idea that we are both the actors and the witnesses in memory.

British psychologist Baddeley's work transformed the science of short-term memory, or working memory as it is now more typically known. Comprehensive and readable, this popular textbook, co-authored with two other eminent psychologists, is a great resource for those setting out on the academic study of memory.

You can remember too much. In this classic case study, Russian neuropsychologist Luria tells the story of his patient S. (Solomon Shereshevsky) who harnessed his synaesthetic powers to perform preternatural feats of remembering. Aside from his scientific prowess, Luria is a wonderfully humane writer, and brings S.'s intense, troubled imagination, through which he perceived reality 'as though through a haze', vividly to life.

To the extent to which they track selves through time, all novels are about memory. But Barnes's 2011 Man Booker-winning novel thinks more deeply about it than most. Middle-aged protagonist Tony finds himself trying to make sense of past relationships and their painful consequences, questioning the reliability of his own story-telling mind as he explores how memories are charged with and shaped by emotion.

Writers over the centuries have had plenty to say about memory. From Virginia Woolf on the birth of the self to Steven Rose on memory molecules, this endlessly fascinating sourcebook gathers writings from the classical era to the present day, covering territory from the literary to the neuroscientific.

The medium of memory is narrative, and the best writers on the topic can mimic its reconstructive processes. Sebald creates fictions that are like memories themselves: fragments of fact and imagination restlessly reorganised into shifting renditions of the past. Austerlitz's memories of his childhood in Sebald's final novel build to an anxious, enigmatic portrait of a mind trying to place itself in time.

It's not just about the petite madeleine. Proust's name has entered cognitive science as code for the power of involuntary memory, but there is much more to his masterpiece than the redolent taste of a tisane-soaked morceau. As critic Roger Shattuck observed, Marcel's million-word quest to reconstruct his life story shows us how memory orchestrates selves in relation: the person doing the remembering held in vibrant tension with the remembered self from long ago. Both co-exist in a memory, meaning that remembering is about the present almost as much as it's about the past.

• Charles Fernyhough will be speaking about Pieces of Light at the Cambridge Literary Festival on 4 April.