In a lecture delivered in New York in the spring of 1983, Italo Calvino remarked that "most of the books I have written and those I intend to write originate from the thought that it will be impossible for me to write a book of that kind: when I have convinced myself that such a book is completely beyond my capacities of temperament or skill, I sit down and start writing it".

Like much of Calvino's work, the statement is at once ironic and quite serious, and is justified entirely by his extraordinary bibliography. Between the late 1940s and his relatively early death in 1985, aged 61, he produced as varied and individual a body of work as any writer in the modern era: neorealist stories of the second world war and postwar period (the young Calvino served as a partisan, and was a Communist party member until shortly after the Soviet invasion of Hungary in 1956), fables, political allegories, historical novels, stories inspired by scientific theories, thought experiments, narratives generated using tarot cards and computer processes, and, at either end of the 1970s, two novels – Invisible Cities and If on a Winter's Night a Traveller – that exemplify his method of using fantasy to address reality. In structural terms these books also codify his passion for beginnings and mistrust of endings, positions that perhaps explain why he habitually moved from one obsession to another, looking for the next impossible thing to write. As the narrator of his 1959 novel The Nonexistent Knight mourns, revealing the pessimism that counterbalances the exuberance of his work,

"One starts off writing with a certain zest, but a time comes when the pen merely grates in dusty ink, and not a drop of life flows, and life is all outside, outside the window, outside oneself, and it seems that never more can one escape into a page one is writing, open out another world, leap the gap."

In the late 1940s Calvino wrote a number of Hemingwayesque stories that drew on his own wartime experience. Already, however, a fairytale quality imbued some of them, for example Animal Wood, and this is the direction in which his work tended throughout the 1950s; the neorealist novel of Italian society that readers anticipated did not appear. As he later explained: "Instead of making myself write the book I ought to write, the novel that was expected of me, I conjured up the book I myself would have liked to read, the sort by an unknown writer, from another age and another country, discovered in an attic."

That book, the historical fantasy The Cloven Viscount (1951), was followed by two of similar style, The Baron in the Trees and The Nonexistent Knight. In the same period Calvino worked on his anthology Italian Folktales (1956), a conscious attempt to produce a Grimm-like survey of stories from the Italian peninsula. This project cemented his regard for "the force of reality which bursts forth into fantasy" a term that could be applied to the majority of his subsequent work. With occasional exceptions, such as the fine 1963 story The Watcher, about the 1953 national election, Calvino abandoned realism.

From the 1960s onwards Calvino's work often took the form of a game played between himself and the reader, or himself and a concept. In stories such as t zero and The Night Driver (1967) he goes astonishingly far, stripping out character and plot yet retaining narrative momentum. These stories aligned Calvino with the OULIPO group, the Paris-based circle including Raymond Queneau and Georges Perec, which sought to graft the rigour and constraints of mathematics onto the freedoms and ambiguities of literature. Not everyone was a willing passenger on these particular journeys, however. Gore Vidal, who in a famous 1974 essay in the New York Review of Books effectively launched Calvino in the English-speaking world, grumbled that t zero "could have been written (and rather better) by Borges" and that The Chase reminded him of Robbe-Grillet. "This", he clarified, "is not a compliment".

Those stories appear in a second collection of Cosmicomics, the story sequence that represents one of Calvino's greatest achievements. Written mostly between 1964 and 1968, although he made additions to the collection as late as the 1980s, they are indebted, according to their author, "particularly to Leopardi, the Popeye comics, Samuel Beckett, Giordano Bruno, Lewis Carroll, the paintings of Matta and in some cases the works of Landolfi, Immanuel Kant, Borges, and Grandville's engravings". Mostly narrated by a protean being with the unpronounceable name Qfwfq (I'm happy to go along with Robert Coover's "Kwoofk"), each story uses scientific statements as launch pads for imaginative tours de force, exploring the domestic, the romantic and the existential via astronomy, geology and evolutionary biology. Qfwfq has been around since the big bang, and we follow him across the wastes of deep space, through stands of proto-conifers, into fecund tide pools and beneath the earth's crust. As the translator Martin McLaughlin notes, Calvino's constant aim was "to raise the target which literature sets itself: he challenges literature to describe the indescribable, from macrocosm to microcosm, from the Big Bang to the division of cells". The language he uses to achieve this is striking, delivering concrete descriptions of, for example, cosmological theory, as with this imagining of the now dismissed steady-state theory of the universe's origins:

"Space is curved everywhere, but in some places it's more curved than in others: like pockets or bottlenecks or niches, where the void is crumpled up. These niches are where, every two hundred and fifty million years, there is a slight tinkling sound and a shiny hydrogen atom is formed like a pearl between the valves of an oyster."

More than any other writer I know, to begin a new Calvino story is like embarking on a voyage to unknown lands; there is a joy to the sense of expectation he inspires. Yet at the same time, his work is tempered with an atmosphere of melancholy and increasing pessimism. Images of the end of the world recur with obsessive regularity, and even terror is present in a significant minority of his works. The most remarkable example of this is The Argentine Ant (1953), which describes a man, woman and baby who move to a new village filled with hope – the promise of a new beginning, that core Calvino theme – only to find that their house, indeed the entire locality, is infested with ants. The story really is, as Gore Vidal stated, "as minatory and strange as anything by Kafka", and the way it builds its air of threat and mystery with plain, undemonstrative language recalls Calvino's own description of Kafka "using a language so transparent that it reaches a hallucinatory level".

"My author is Kafka", Calvino once told an interviewer when asked about his influences, and his presence is discernible throughout Calvino's work, from The Argentine Ant to the 1984 story Implosion. Here Calvino links two of literature's most introspective characters, the doomed prince Hamlet (the story begins: "To explode or to implode – said Qfwfq – that is the question") and the mole-like creature from Kafka's death-haunted story The Burrow, in a beautiful but deeply melancholic rumination on black holes and the death of the universe, and an apprehension that obliteration lies at the heart of each individual consciousness:

"Don't distract yourselves fantasizing over the reckless behaviour of hypothetical quasi-stellar objects at the uncertain boundaries of the universe: it is here that you must turn your attention, to the centre of our galaxy, where all our calculations and instruments indicate the presence of a body of enormous mass that nevertheless remains invisible. Webs of radiation and gas, caught there perhaps since the time of the last implosions, show that there in the middle lies one of these so-called holes, spent as an old volcano. All that surrounds it, the wheel of planetary systems and constellations and the branches of the Milky Way, everything in our galaxy rests on the hub of this implosion sunk away into itself."

The last line of this final Cosmicomic – "I go on digging my hole, in my mole's burrow" – forges a link with The Burrow, a story that Kafka was yet to finish at the time of his death, or so it appears: the final page of the manuscript is completely filled, with no terminating punctuation mark. How fitting that in one of his last stories Calvino, whose work is a catalogue of new beginnings (the "collection of fragments that is my oeuvre", he called it), should gesture to the last story of another author that does not end, only stops. Calvino's own life, too, didn't draw to a close but broke off abruptly with a cerebral haemorrhage, the next impossible book, finally, remaining unwritten.

• Translations from the work here are by William Weaver, Archibald Colquhoun and Tim Parks.

Next: Barry Hannah