I was always a scribbler. As soon as I was taught the alphabet I scribbled – even before I could spell so that I was always harassing everyone in the household (including the cook who knew no English) “How do you spell “house”? How do you spell tree, fire, bird, fish …?” (He responded by making me a magnificent gift on my birthday of an inkwell carved out of soft soapstone, which I unfortunately ruined by pouring real ink into that delicate, decorative object.) I filled notebook after notebook seated on a cane stool at my round green table and was labelled, with an understandably resigned sigh, “The Writer in the Family”.

What was I writing? Consciously, with awareness and intent, very little. I simply had an urge to put everything that I saw, heard and experienced on paper, in ink. I had little awareness of categories – books were books to me, the imposing leather-bound books behind the glass on my parents’ bookshelves, the worn, dog-eared paperbacks on my siblings’ shelves, and the exciting, inviting ones in all their diversity in the bookshop where I spent my pocket money. I can’t remember when I learned to differentiate between the short story and the novel – no, actually, I can: it was when I first decided to send a piece out to be published (publishing was important, I knew writing had to be in print if it was to earn its name), and it was, of course, of a short length to fit into a magazine or journal. But I was also always writing at length with the idea of a book, a proper book, in my mind, and a part of me believed short stories to be failed novels.

But a short story is not a failed novel any more than a novella is an extended short story. Each has an altogether different set of rules and effects. Length is one of them, but lengths vary wildly. As Hortense Calisher said, “How long should a short story be? As long as a piece of string. I mean – to tie up the parcel with.” I like her practical, workmanlike approach, but there is, in addition, the element of chance. How did one piece I scribbled end up a short story, another extend, unwrap itself, wander, digress and venture on to a path, a road to a further destination – novella, or novel?

It is all a matter of instinct, really, and exploration – a conviction that dawns on one, while one works, that one has said what one set out to say: there is no need to go further. It may be just one small episode, stumbled on unexpectedly, a glimpse out of a window, the fall of light on one object while bypassing another, that gives one pause and for some reason is not forgotten. Why has it stayed in the mind when so many other impressions, encounters and experiences have turned into a blur and disappeared? And when one has found the answer to that – the story is done. It can come to one quickly or it may take long, very long, to discover. In the short story, it need not be pursued further. Many writers have commented on its identity being closer to a poem’s than a novel’s.

I have written only a few short stories that have provided me with that sensation that one craves: “Ah, I have done what I set out to do, no more is needed.” The stories that constitute my new collected edition are those that I ended on that note. For the most part, I have taken longer and watched the stone I’d flung into the pond create ripples that extend further, ripple on ripple, arc on arc, struggling to reach the far shore, and wondered: where will this go? How will it end? And that search has turned into a novel.

It is the latter mode that I have mostly chosen. It is the one that offers space both dangerous and forgiving, and lays one open to what may be years of discouragement, dejection, doubt and isolation while one considers options, takes one direction and then another, makes errors, corrects them, picks oneself up and struggles on, only gradually building up the momentum needed for narrative. But while involved in so much that is frustrating and exhausting, one may be granted – briefly and sporadically – that mysterious breath of air that comes up unexpectedly, creating a ripple, a stir, a tide that thrusts one forward and sends one soaring, sailing, flying through space and time.

It is the pursuit of that elusive and mysterious sensation that one undertakes in the short story, so different a form. Instead of those long stretches in which a novelist becomes stranded, the short-story writer must launch forth on what is a high-wire act, refusing to look back or down into the abyss, running the length of it at a sprint so as not to lose balance: quick, quick before you fall! You may go back and start all over again, or change sentences and scenes, but that initial urge must retain its urgency from beginning to end.

Lightning that mocks the night,

Brief even as bright.

Percy Bysshe Shelley

In this, the short story is the more challenging form – as I realised when I had the temerity to “teach” the writing of it to students who came to the creation of fiction as complete novices, simply because it was easier to fit into the space of a class, a term – that “length of string” again. But it was the very brevity and confinement of the form that demanded skill, learning and understanding to make it “work”, ie, to create the desired effect.

But every once in a while, when completing that frantic dash of the short story, even after it is in print, one finds it won’t let go of one. It pursues one – or, rather, one pursues it because there is more to be said, more to be delved into, discovered and exposed. So every once in a while I have found, years and years later, a short story written long ago insisting on becoming a novel.

It is the experience I had when I wrote the short story “The Accompanist”. I felt then that I had put on paper all I knew – very little – about that minor figure of the musical world, the musician in the background, barely noticed, all attention being given to the maestro, the soloist. Was he content for it to be so?

Was he – or not? There was so much in the life and work of that overlooked artist, and I wrote the novel In Custody to give him his due, although I changed the two characters into a poet and scholar. And again, later still, into the novella Translator Translated. One of my earliest short stories, “Scholar and Gypsy”, eventually carried on a whole new life as the novel Journey to Ithaca, something I did not even know until a reader pointed out the development of the theme: the difference between the character who feels the world is all we need and the character for whom the world is limited; beyond it there surely lies more. The search for that other world – physical or spiritual – that compels them on their journeys, had carried on from the short story into the novel as a sketch might lead to a painting. This subterranean element rising to the surface surprised me, I had not been conscious of that development.

Each form requires a different set of abilities, even materials – as an artist might need pencil or pen and ink, or watercolours or oils for one work or another. Brevity and concision will do for one, while the other requires doubt, mystery, mistake and stamina. If one writes both, which gives the greater satisfaction? Now one, now the other – that is the only answer.

• The Complete Stories by Anita Desai is published by Chatto & Windus. To order a copy for £14.44 (RRP £16.99) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.