The Ouvroir de littérature potentielle (Workshop for Potential Literature) was founded in Paris in 1960 by the writer and publisher Raymond Queneau and the scientist and educator François Le Lionnais. Oulipo set out to formally explore the use of mathematical and other rules – known as “constraints” – in the writing of literature.

Its approach was two-pronged. First, what Oulipo called “anoulipism”, discovering constraints used by writers from other ages and cultures, wittily referred to as “plagiarism by anticipation”: they seized on reversible poems in 3rd-century China and acrostics concealed in the Psalms, as well the bifurcating narratives in Jorge Luis Borges’s 1941 “The Garden of Forking Paths”. Second, “synthoulipism”, the invention and demonstration of new constraints for any writer who wished to use them.

The first consciously Oulipian literary work was Queneau’s Cent Mille Milliards de poèmes (A Hundred Thousand Billion Poems) of 1961, a book of 10 sonnets with each line printed on a separate strip. Since all 10 share the same rhyme scheme and rhyme sounds, each line can be replaced by the homologous line in any other sonnet, giving an astonishing one hundred thousand billion (10ⁱ⁴) variations. This is a truly potential work of literature: no one could read the entire collection in a lifetime.

In the 'prisoner’s constraint', letters with ascenders or descenders (b, d, f, g, h, j, k, l, p, q, t, y) can't be used

Oulipians including Italo Calvino, Georges Perec and Harry Mathews went on to produce some of the most extraordinary and compelling literature of the 20th century. Calvino’s dazzling 1973 novel The Castle of Crossed Destinies (published before he was co-opted into the group in 1974) was inspired by two spreads of the Tarot, which he called “a machine for constructing stories”. Perec’s 1969 La Disparition is an avant-garde detective novel written entirely without the letter E. In Oulipian terms it is a “lipogram in E”, where moreover the missing letter is the crux of the novel (as Philip Terry notes, “the missing ‘e’ is a homophone for ‘eux’, those who disappeared in the Holocaust”). When La Disparition was translated into English by Gilbert Adair in 1995 as A Void, Adair honoured Perec’s central constraint throughout.

Mathews’s “35 Variations on a Theme from Shakespeare” functions as an Oulipo sampler in which he demonstrates constraints by applying them, one after another, to Hamlet’s most famous soliloquy. So “To beckon or not beckon: that is the quinsy” is “Transposition W+7”, ie replacing a word with the seventh subsequent dictionary entry; “Nothing and something: this was an answer” is “Antonymy”; and “Two-beer naughty beet shatters equation” is “Homophony”. All of these, and more wonders by all three authors, are sampled in The Penguin Book of Oulipo, a superb selection of 100 Oulipian texts published on the eve of the group’s 60th anniversary.

As well as writings by the group itself, Terry’s survey includes the work of so-called “anticipatory plagiarists”: Jonathan Swift’s writing machine in Gulliver’s Travels, Lewis Carroll’s experiments with scale in Alice’s Adventures in Wonderland, and the painting machine from the “Clinamen” chapter of Alfred Jarry’s Exploits and Opinions of Dr Faustroll, Pataphysician. The name of the chapter the last appeared in, “Clinamen”, became the Oulipian term for an allowable deviation from the strict application of a constraint that can only be used if it is not needed. Avoiding disappearing down a rabbit hole of his own, Terry briefly sketches but doesn’t go too deeply into the historical connections between Oulipo and the Collège de ’Pataphysique, nor too far into the proliferation of official and unoffocial “Ou-X-Po” groups in other fields and genres such as Oulipopo, the study of crime fiction; the extra “po” from policier.

Facebook Twitter Pinterest ‘Georges Perec’s Life, a User’s Manual uses chess moves and other constraints to depict every room in a large Parisian apartment building.’ Photograph: Louis Monier/Gamma-Rapho via Getty Images

Illustrating the group’s legacy and the principle that these constraints are available to anyone, Terry also includes examples of writings “after Oulipo”. There’s an extract from Richard Beard’s arch Donald Barthelme-esque 2011 novel Lazarus Is Dead, the chapters of which shrink then grow again as Lazarus is reborn (schematically reminiscent, in its symmetry, of “anticipatory plagiarist” George Herbert’s 1633 calligram “Easter Wings”). We also find Alice Oswald, whose poignant and beautiful Memorial removes most of the Iliad, to leave only accounts of the deaths of 200 soldiers. which she garlands with choruses drawn from Homer’s extended similes: “Like leaves/Sometimes they light their green flames/And are fed by the earth/And sometimes it snuffs them out.”

So what’s in it for the reader? Lovers of puzzles and word games will relish the many sometimes deceptively simple constraints, such as the “snowball”, in which each successive line is one unit (a letter, syllable or word) longer, or the “prisoner’s constraint”, in which letters with ascenders or descenders (b, d, f, g, h, j, k, l, p, q, t and y) cannot be used. For Mathews, though, interviewed in the Paris Review, “What distinguishes Oulipo from other language games, is that its methods have to be capable of producing valid literary results.”

There’s a debate about whether Oulipian authors need be open about the constraints they’re using. Jacques Jouet is pro-transparency – the “scaffold” is part of the substance of the work. Calvino and Mathews preferred that the reader detect hidden patterns without necessarily knowing what they are; Calvino initially forbade publication of the algorithm behind his most famous novel, If on a Winter’s Night a Traveller. Terry takes Calvino and Mathews’s lead, so as you read through The Penguin Book of Oulipo the writing stands on its own. I’m with Jouet, and like to have ready access to further information. I found the index of constraints challenging and occasionally mystifying, since in trying to discover which were employed in a particular text, one must use it in reverse.

Some Oulipian works far transcend whatever constraints produced them. Perec’s masterpiece, Life, a User’s Manual, uses chess moves and various constraints to depict every room in a large Parisian apartment building, and the myriad stories, texts, objects and lives they contain. Even in excerpt, it is a multilayered and supercharged reading experience. When it first appeared in David Bellos’s masterly English translation in 1987, the UK publisher included no mention whatsoever of Oulipo: readers apprehended that something out of the ordinary was afoot, but could grasp only fleeting impressions, enhanced by the typographic games and acrostics captured or created in English.

In their provocative 2013 book The End of Oulipo?, Lauren Elkin and Veronica Esposito suggest that Oulipo today can’t help but be overshadowed by its illustrious former members; that “the dynamism … has flagged”, and the group is now more concerned with self-promotion, resulting in a kind of Oulipo-lite. Anxieties about live performance – that Oulipians risk “becoming ‘seasoned performers’, working on their stage presence” – feel like a page v stage straw man. But Elkin also robustly dismantles what she calls the “frustratingly macho” output of Oulipo’s current president, Hervé Le Tellier.

Top 10 experimental thrillers Read more

And the real discovery in The Penguin Book of Oulipo are the works by women. Of particular interest is the British author Christine Brooke-Rose (1923–2012), whose virtuosic, typographically volatile 1998 novel Next is an Oulipo-inspired policier set among the homeless, and written without using the verb “to have”. Juliana Spahr of feminist offshoot Foulipo “slenderises” (removes parts of) the text of anti-abortion legislation to produce the oppositional poem, “HR 4881 is a Joke”.

The biggest surprise is Anne Garréta, whose novel Sphinx, first published in 1986, explores DJ culture, gender and sexual politics in Parisian nightclubs. The two main characters are the narrator and their lover A***, the sex or gender of neither of whom is given. Sphinx (available in Emma Ramadan’s 2015 English translation) is ahead of its time, a radical bridge between Kathy Acker, the “chemical generation” authors of the 90s, and emerging innovators such as this year’s Goldsmiths prize-nominated Isabel Waidner.

The Penguin Book of Oulipo already feels indispensable. It’s also a welcome celebration of the contribution of literary translators, including Bellos, Gilbert Adair and William Weaver, to the popularising of this initially reclusive group. Terry’s anthology connects us to a wider world of Oulipian wordplay, and beyond. Perhaps that’s the point. After all, as Calvino reminds us in The Castle of Crossed Destinies, there are certain fountains that “once you drink from them, increase your thirst instead of slaking it”.

• The Penguin Book of Oulipo: Queneau, Perec, Calvino and the Adventure of Form, edited by Philip Terry, is published by Penguin (RRP £25). To order a copy go to guardianbookshop.com. Free UK p&p over £15.