Last week, the British Library’s collection of obscene, sexually explicit and erotic novels and writing went online for the first time. Called the Private Case collection, these 2,500 texts, including fictional confessions of adventurous ladies, compendiums of London courtesans and erotic nun novels, have now been made available to registered scholars across the world. Cue social media jokes about library membership expanding.

In the days of researching my DPhil on 18th-century attitudes, I spent a good amount of time at the “special collections table” in the rare books room of the British Library. This was the table for those using works from the Private Case, as well as other delicate materials (and, apparently, any books bound in with magical texts or spells, because, as one reader told me: “The witches come in and try to steal them”). A librarian watched over us, we had to hand in the material whenever we left our chair and one did rather wonder what everybody else was reading. But I found all kinds of gems (but never any spells). One of my favourites was Venus in the Cloister, a 17th-century bestseller in which a young, pretty novice joins a nunnery and… well, you can imagine.

The earliest book in the Private Case is from 1658, Rare Verities: The Cabinet of Venus Unlocked and Her Secrets Laid Open, a collection of sections of Latin authors “never before in English”, but much of the collection is from the 18th century, the era of the beginning of the novel. They’re fascinating examples of early fiction and give a lot of insight into reading at the time; these books tend not to trade in style and elegance but repeated scenarios and endlessly inventive ways of throwing their heroines into the arms of various partners.

Most great libraries across the world have similar collections, many established in the mid 19th century – the Delta Collection at the Library of Congress or l’Enfer at the Bibiliothèque Nationale in Paris, which contains thousands of volumes, many taken from aristocrats during the French Revolution.

The Private Case dates to the 1850s and at its height had thousands of books, but over the 20th century volumes were transferred into the main collections as attitudes changed.

The biggest bequest, of nearly 700 works, was from the great 19th-century book collector Henry Spencer Ashbee, who produced a three-volume bibliography of erotic literature. Ashbee’s second volume was called A Hundred Books That Should Be Hidden. And hidden they were.

It’s hard to know how widely they were read. Not only are they not the sort of thing that readers would have openly treasured or discussed in letters, as one lecturer of mine put it, they’re also the sort of book that gets worn away… Certainly, these were generally reading matter created by men, for men – and hidden from women.

Harris’s List of Covent-Garden Ladies (1757-95), an annual user’s guide to the courtesans of London, is a fascinating resource for scholars looking into these women, who leave so few records behind, though one wonders quite how they themselves felt about their entries. There are enthusiastic reviews for Miss R–chds–n of Rathbone Place, as creating “momentary delirium” and then “each part combines to raise up the fallen member”, and Miss L–c–s in York Street, of whom it is promised “all who bathe in her Castalian spring will be overwhelmed with a flood of delight”. But though Miss H–rd–y of Newman Street is praised, “not knowing how long this train of perfections may last”, readers are advised to “make hay while the sun shines”.

Now, thanks to the internet, the obscene material created by our society is out there, accessible at a click, not hidden in the slightest. Still, the current political and media fallout over texts by Jeff Bezos is a reminder that when it comes to racy materials written by oneself, it’s perhaps best to lock them in a cabinet.

• Kate Williams is professor of history at the University of Reading. Her book Rival Queens: The Betrayal of Mary, Queen of Scots is out now