The descendants of Charles Darwin, his publisher – and his pigeons – met this week on the 150th anniversary of the publication of On the Origin of Species

"That's the fireplace where Byron's memoirs were burned after his death because they were thought too salacious," my host Virginia Murray casually throws into the conversation as she shows me around the upstairs drawing room of number 50 Albemarle Street, just a few steps from Piccadilly in London's Mayfair.

I am here on something of a scientific pilgrimage: to see the place where Charles Darwin and his publisher, John Murray III, discussed drafts of arguably the most important book in scientific history – On the Origin of Species. I had hoped to get a sense of the heritage of the book on the 150th anniversary of its publication, but I was not prepared for a fascinating all-round history lesson.

"This was the meeting place in England for literary and political types at the beginning of the 19th century," said Murray (the great great grandson of Darwin's publisher). At Murray III's soirees, Darwin rubbed shoulders with his great intellectual influences, the economist Thomas Malthus, the botanist Joseph Hooker and the geologist Charles Lyell.

John Murray III's file copy of the first edition of On the Origin of Species. Photograph: Graeme Robertson

And in its 234-year history, which began 21 years before the French revolution but sadly ended in 2002, the publishing house played host to the likes of Benjamin Disraeli, John Betjeman, Kenneth Clark, David Livingstone and James Watt. Busts of the latter two glower down at us from either side of a massive portrait of Lord Byron that hangs above the fireplace where the poet's diaries met their end.

Tonight's soiree is touched by a more modern twinkling of stardust, with the novelists AS Byatt and Ian McEwan, and broadcaster Andrew Marr on the guest list. We are here to see a unique collection of Darwin artefacts that have been brought together for the anniversary – the climax of a year of Darwin-related events that began with the great naturalist's 200th birthday on 12 February.

On display is the publisher's original "file copy" of On the Origin of Species, one of 1,250 copies in the first print run, which earned Darwin £180. This copy, which would now probably fetch something north of £100,000 at auction, is now part of a collection held by the National Library of Scotland in Edinburgh. The collection includes other treasures such as Darwin's submission letter to Murray and the entry for the Origin in Murray's financial ledger.

Across the room is a collection of six live fancy pigeons – the same varieties that Darwin used as examples in the first chapter of his book to demonstrate the power of selection by human hand. "This is a world first," said Randal Keynes, one of Darwin's great great grandsons, who explains that this is the first time that the six varieties have been brought together since the publication. "If you gave them to an expert ornithologist he would say that they are not only different species but also different genera."

Letter from Darwin to John Murray III, his publisher, dated 31 March 1859. Photograph: Graeme Robertson

And he's right. The tall, proud English pouter with its puffed-out chest looks nothing like the dainty Almond Tumber, while the scandaroon's massive beak and upright posture could hardly be more different from the fantail with its splayed plumage.

These varieties would have been very familiar to people in the 19th century, but it was Darwin's genius to point out that they were all closely related to the rock dove (essentially the same as the pigeons in Trafalgar Square) and had all been created in a few generations by breeders selecting the characteristics they liked. His message: if people can do this, what could nature achieve with plenty of time on her hands?

The most valuable thing in the room, at over £200,000, is a single leaf of Darwin's original manuscript owned by Keynes's father (pictured at the top of this article). The text reads:

"Finally then, the facts too briefly given in this chapter, do not seem to me opposed, but rather to support the view that there is no fundamental difference between species and varieties."

Darwin did not value the manuscript and gave it to his children to use as writing paper. It ended up forgotten in a cupboard at Down House, where Darwin wrote On the Origin of Species, but some of the leaves survive today. Thank goodness they did not end up in the fireplace.