I am moving house. I am moving from the spacious flat I have lived in for 19 years, a corner house, very bright and full of windows, a place of flights of stairs and landings and hallways, no room on the same level as another. There has always been space for more books, you could tuck in a few shelves in all kinds of places. I had some built by a carpenter when I moved here. "These aren't going anywhere," he said, as he applied brackets to the wall.

But however many shelves were built there were still never enough. The books in alphabetical rows were overgrown by piles of new books, doubled in front. Books multiplied, books swarmed; they were a papery population explosion. When they had exhausted the shelves, they started to take over the stairs. You cannot have a taste for minimalist decor if you seriously read books.

For many weeks before I left the building, I sorted them out. The decision about what would stay and what would go, live or die, began with kindness and ended in rage and ruthlessness. I have a pair of library steps I bought in an antique shop in Cornwall and schlepped back to London, which I climbed every afternoon and scanned the shelves. What I saw, swelling with self-important pride was evidence of how I constructed my own intellectual history through reading. Here is Proust. Here is Jean Rhys. Here is Milton. Here isn't Henry James because I have never been able to remember the beginning of his sentences by the time I get to the end.

Here is JK Rowling. Here is Jilly Cooper. This is a library that tells you everything about its owner, that doesn't conceal the shameful reads, the low taste. Here are first editions, bought at abe.com, of my childhood favourites, the ballet and riding books of Lorna Hill, which taught me about ambitious, arty girls from Northumberland who went to London, became prima ballerinas, married conductors and lived in smart flats in a St John's Wood mansion block with a service restaurant.

Here are books that were birthday presents with inscriptions in the front from dear friends, some of whom are no longer living. Here is a copy of the first paperback edition of Tom Stoppard's only novel, Lord Malquist and Mr Moon, published in 1966, which nobody but me has ever heard of. Edmund Spenser's The Faerie Queene, the cover drizzled with spilt candlewax, an accident that took place at university in 1974. An American first edition of Bleak House given to me at the end of the 70s by a man as an apology for going back to his wife.

In the middle of my move I watched a documentary called The Flat. A family was clearing out the Tel Aviv apartment of a 97-year-old woman who had recently died. She had lived there for 70 years, since arriving from Germany in the 30s. The walls of the flat were lined with books published in her native language. Her grandson called in an antiquarian book dealer. He took the volumes off the shelf and hurled them with force to the floor. "No one reads Balzac," he said. "No one reads Shakespeare. Nobody wants Goethe. Know how many books they throw away in Germany?"

Who destroys books? Cities, churches, dictators and fanatics. Their fingers itch to build a pyre and strike the match. On 10 May 1933, students gathered in Berlin to dance around a bonfire of 25,000 "un‑German" books. They burned, among many others, Bertolt Brecht, Otto Dix, Heinrich Heine, Ernest Hemingway, James Joyce and HG Wells. They destroyed them because the contents were too dangerous. Now, in an apartment on the Mediterranean, the same authors were being dumped because no one wanted to read them. They are the detritus not just of the digital revolution but of disposable living and small houses.

And I, too, have committed murder in my library. I have killed my books.

The little girl who lay in bed, a circle of illumination on the sheets from her toadstool nightlight, afraid to go to sleep because her Struwwelpeter picture book lay next to her in the dark confinement of the ottoman with her toys, frightened of the scissorman who cuts off the thumbs of children who suck them – that small person who understood, even before she could read, the power of a book, has just liquidated half her own.

This isn't me. I am the adult outcome of the shy, awkward only child who, instead of running around in the garden or clambering on slides and swings or slapping bats against balls or skipping down muddy lanes, preferred, above all else, as I still do, to stay indoors and read. I found in books my friends and my fantasy lands, and never looked to fiction for social realism, or expected books to tell me about the life I led in suburban Liverpool with immigrant parents who muttered in an obscure tongue and in the kitchen found a use for every part of the chicken.

I was enraptured by what disgruntled readers now refer to as matters "not relevant to my personal experience". I was over-familiar with chairs that flew, with wardrobes that led to snowy woods and holes in the ground with Hobbits in them. In books was life! The great life!

It is more than 50 years since I began to build my library from its earliest foundations in the elementary sentence construction of Enid Blyton. Now, at least half of the thousands of books I have bought are gone. It is one of the worst things I have ever done. I hate myself. But not as much as I have come to hate the books. Hate books! A thought-crime at the very least. Only a philistine, a religious zealot, a Nazi would hate books.

It is not the words I hate, not literature, but their physical manifestation as old, musty, dusty, yellowing, cracked objects, heavy to lug around. When I open the pages swarms of black ants dance on the paper. No one told me. No one said: "In the future you will squint and screw up your face and try to decipher these words you once read so easily." When I look at my books I feel like Alice in the closing pages of Alice in Wonderland, when the cards all rise up and overwhelm her.

When the estate agent came to look at my flat he winced when he saw all those books. What did he see? Clutter. Estate agents do not think that books furnish a room; books make rooms look messy. You would not display the contents of your knicker and sock drawer or your bathroom cabinet with its face creams and cough remedies, so why put off potential buyers with your taste in literature?

In order to market my flat, the books had to be pruned back. At the very least, they would not be permitted to exceed the number of shelves available to house them. So the murder began.

The methodology I used for my cull was very high-minded: I would preserve those books of literary merit, the books I had not yet read but wanted to and the books given as gifts with an inscription on the flyleaf. Judging literary merit at the top of library steps is a beautiful and contemplative activity. I see Catherine Deneuve, half‑lit with the illumination from a Parisian window on a Rive Gauche boulevard below cloudy, pearly autumnal skies, a few streets from Shakespeare & Co. She picks a book out from the shelf, examines the spine. Ah, Mathieu! The much older lover, a grizzled intellectual with whom she spent a summer in Cadaqués when she was 20. Fade … dissolve to Charlotte Gainsbourg in 1967 in the kitchen cutting tomatoes, while out on the terrace Daniel Auteuil is typing the masterpiece that will win the Prix Goncourt and later be filmed by Truffaut.

I sneezed. The shelves were filthy. I wobbled, looked down, got vertigo. How do we assess André Gide's reputation? By we I don't mean the French Academy. Does anyone still read him? If no one still reads him what does that tell me about literary merit? I went down the steps to the internet and looked him up. He won the Nobel prize for literature; he died in the year of my birth. Strait Is the Gate was one of the Penguins I bought in my early teens. I have absolutely no recollection of its subject matter. It gets thrown to the floor and joins the other splayed volumes of rejects.

After a couple of hours the process of deciding literary merit speeds up considerably. When I was young, boys invited in for sex would examine your bookshelves. A collection of novels by Arthur Hailey and Barbara Cartland would not necessarily be enough to prevent them from ripping off their T-shirts, loon pants and desert boots, but if you added to the sexual experience a credible book collection then the move from one-night stand to girlfriend was consolidated. My books had to make up for my LPs. "What's this? I didn't know Tom Stoppard wrote a novel." "But of course, you mean you haven't read it?" Lord Malquist and Mr Moon was the literary equivalent of the Wonderbra for intellectually pretentious students of the 70s.

I no longer need to impress male visitors with the depth of my reading. So what is the nature of this library; what function does it serve other than being a filing system for books? What, to use the phrase beloved of cultural criticism, does it say about me and to whom is it addressing this message? When builders or grocery delivery men come in, they often say: "Blimey, have you read all these books?" In friends' houses I have stopped inspecting their bookshelves for evidence of their literary taste because we have all read, more or less, the same books. My curiosity is limited now to how they store and display them. The former literary editor of a national newspaper told me the other day that most of his books are in paid-for storage and have been for years. Now there is a strong part of me that thinks that if you don't have access to your books you might as well not have them.

I made this point to him, but he argued that he had first editions of the Brit Boys, Amis, McEwan and Barnes. I wonder if he thought they were his pension. A Christmas email from an old friend in Vancouver spoke mourn-fully of a friend whose husband is an antiquarian book dealer. The bottom had dropped right out of the market, she reported, even first editions.

If there is a need it is not a functional one, it is something else. For one thing, I would be ashamed of being a writer whose house had no books. For another, the books, as I have already said, are a library. In a library, you do not read a book to the last page and dispose of it; you return, you return. I go back to Paris in the 30s, to Jean Rhys, the novelist of unmediated longing and yearning and rage and sexual desire, and the need for nice clothes, and the fear of what happens to women when they are old and lose their looks and become the woman alone upstairs, drinking alone, smoking alone, dying alone. Sentence after apparently unremarkable sentence passes until suddenly I feel myself hit in the solar plexus by the accumulated tension. I look back and ask, how did you do that?

I kept Jean Rhys, I kept Anita Brookner, I kept Beryl Bainbridge. These books are personal not only as objects but also for the intense relationship I have with the text.

The flat I moved to has a second bedroom, 12 feet by six feet, a sort of corridor, occupied by a child's cot when I came to view. This would be my office. This flat has no landings, just a narrow entrance hall that its previous owners used to house their only bookcase. The building is on a ridge in north London looking down on the city. Through this study window, in a gap between two sinister mansion blocks opposite – the repository, I hope, of several novels' worth of speculative daydreaming – the sharp pyramid of the Shard is framed, and at the back the kitchen overlooks a row of gardens. Like James Stewart, in Rear Window, his leg raised in a cast, here I sit, hopeful that the mansion blocks will sooner or later yield a murder. So far, all I've seen are people washing up and people staring at the screen of a computer. A sense of light and space and other people's lives is terribly important to me; spending so much time at home I must have something to look at out of the window. That's why I bought the place. I didn't think about my books, my soon-to-be dead children, my murdered soulmates, for which there would not be enough room.

But when the last paintbrush had been put away and the books finally came to rest on the newly painted shelves (not in any order at all, that is for later), there was something immediately wrong. In my fear of not having enough room in my new flat for my books, I had got rid of far too many. The truth was I now had empty shelves. The shame. I am a person who does not have enough books. My library is denuded. It doesn't seem like a library. It feels … like the house of a person who reads, but not the house of a person for whom books have been everything.

I'm not going to reread these books before I die. I am just bequeathing my nephew and his wife the heavy task of removing them at a later date. They will not call in the antiquarian book dealer … It is death we're talking about. Death is the subject. The death of the book, but also my death. Because I am kidding myself if I think that I am going to reread a fraction of the books I have brought with me or a fraction of those I have never got round to reading.

In my youth, I imagined old age and retirement as the time when one sat back, relaxed and read. There would be all the time in the world for reading. Sixty was so far away, and 80 stretching out into a future not imaginable, that you might as well be talking about living forever. Now time gobbles up my life. I don't need most of books on my shelves, those pitifully empty shelves.

The whole business of this move has made me massively insecure, blind-sided everywhere. I'm not writing a book at the moment; I don't have the concentration, that's why I'm writing this. Everything is wrong, abnormal. Three-quarters of the way into my life I've had the ground taken from under my feet. I have damaged my connection to the little girl frightened of her Struwwelpeter book.

When I began to write on a computer, when I abandoned the typewriter, I didn't look back. I don't miss my turn-table or my cassette player. I'm not a luddite, I'm a modernist. But a part of the cliff has fallen into the sea. There are not enough books here. The sight of the bare shelves shames me. What have I done?

• (c) Linda Grant 2014 Extracted from I Murdered My Library, published as a Kindle Single available at www.amazon.co.uk at 99p