Amid the neon-lit diners and coffee shops of New York’s Upper East Side sits a townhouse that’s a world away from the fast-paced drama of Manhattan. In sight of Central Park, but not as far north as the Metropolitan Museum of Art, it is just one of many such houses on a street full of elite mansions and enviable residences. No sightseeing map would direct you to East 70th Street, and it’s routinely bypassed by cab drivers, commuters and pedestrians, all of whom have somewhere else more important to be.

But beyond the townhouse’s wrought iron doors, under a keystone archway, a world of tightly guarded secrets awaits. For this intriguing six-storey mansion, 109-years-old and a fusion of Jacobean Renaissance and Tudor architecture, is a social club for a clandestine group of travellers who have seen more of the world – and universe – than anyone would think possible.

You may also be interested in:

• The travel guides that charted our world

• The origin of life on Earth?

• A trip most people wouldn’t dare to do

The deepest oceans. The farthest rivers. The highest peaks. Even the moon and outer space itself. All of it has been mapped by the club’s globetrotting members. And on any given day, many can be found in the back room, taking tea while plotting their next extraordinary adventure. Talk is not of the weather, but of moon landings and blow dart encounters.

The people who’ve marched through these doors created pages of history

This is the little-known Explorers Club, the headquarters of one of the world’s most awe-inspiring field science institutions. Its illustrious list of current, historical and honourary members includes Edmund Hillary and Tenzing Norgay, who first summited Mt Everest; aviator Charles Lindbergh, who made the first solo transatlantic airplane flight in 1927; Norwegian explorer Thor Heyerdahl who sailed his hand-built balsawood raft, the Kon-Tiki, from Peru to Polynesia; famed pilot Amelia Earhart who disappeared in the Pacific; Apollo astronauts, including Buzz Aldrin and Neil Armstrong, the first men on the Moon; record-breaking deep-sea diver Sylvia Earle; British paleoanthropologist Mary Leakey, who discovered 15 new species of animal; Amazon CEO Jeff Bezos; Titanic film director and deep-sea explorer James Cameron; and primatologist Dame Jane Goodall, considered the world's foremost expert on chimpanzees. The list is mind-boggling. Small wonder the club’s in-joke involves an astronaut, aquanaut and speleologist walking into a bar.

“There are millions of stories associated with this place, and sometimes I need to catch myself from name-dropping,” said two-time club president Richard Wiese, who himself is an explorer with more than 200 episodes of the Emmy Award-winning TV travel series Born to Explore with Richard Wiese on his resume. “The people who’ve marched through these doors created pages of history. They’re immortalised figures.”

Now the club’s 44th president, Wiese was drawn into this Indiana Jones world by his father, Richard Wiese Sr, who was the first man to solo the Pacific Ocean in a plane. He remembers standing on his front lawn in Connecticut looking at cumulus and contrail clouds wishing he could be just as adventurous. By age 12, he had travelled to Africa and climbed Mt Kilimanjaro.

“I recall the first time I came to the club in the mid-1980s,” Wiese told me, while we sat at a table once owned by former member and US president Theodore Roosevelt in the club’s boardroom. “It was to see a lecture about black bears in northern New Jersey, and straightaway I knew I had found my people.”

Like the other mountain-climbing, polar-exploring, zeitgeist-defining club presidents before him, Wiese maintains the society’s purpose is for knowledge enhancement alone, not self-fulfilment. Its 3,500 members – spread across 32 global chapters, including the New York headquarters – are bound by a bond to push the boundaries of science and education. And these days, membership is predominantly taken up by oceanographers, lepidopterologists, primatologists and conservationists. Not wannabe Shackletons.

A case in point: this past summer, a group of club palaeontologists were in Mongolia’s Gobi Desert hunting for fossilised dinosaur remains using drone scanners. “They found dozens, if not hundreds,” Wiese told me, almost as if he couldn’t quite believe it himself. “Exploration for us is now less a cult of personality and more a cult of data. And because of that we’re getting better at finding the truth.”

It was 1904 when The Explorers Club was founded by historian, journalist and explorer Henry Collins Walsh and like-minded Arctic explorers. At the time, the race to the North Pole had brought the group together with a broader purpose to explore by air, land, sea and space. This saw the first meetings held at its original headquarters in the Studio Building at 23 West 67th Street. But as the club grew in stature, so too did its need to expand to house trophies, books and priceless artefacts.

Enter American writer and broadcaster Lowell Thomas of ‘Lawrence of Arabia’ fame years later. An enthusiastic member in the 1960s, he was instrumental in the club acquiring its current headquarters, once a private family home owned by an heir to the Singer Sewing Machine.

“This place used to be about pushing dragons off the map,” said the club’s archivist and curator of research collections Lacey Flint, leading me on a fascinating tour of the townhouse. “We still push those dragons, but the club has become so much more. What really excites members is that we know more about the volcanoes on Jupiter than we do about the very bottom of our oceans.”

History is alive in the building’s upper galleries like few other places in New York. It isn’t just the taxidermy polar bear guarding the staircase. Or the sledge used by Robert Peary and Matthew Henson on an expedition to the North Pole in 1909 (now placed above a door in the Clark Room). It’s in the indigenous totems found by Michael Rockefeller on a trip to collect primitive art from New Guinea (while several artefacts were airmailed to the US, Rockefeller never returned and rumours persist he was eaten by cannibals). It’s in the series of framed club flags, once folded into spacesuit pockets and carried on every Apollo mission into space. And it’s in the artefacts on the desk of the club’s archivist that still need to be catalogued.

It’s a place that boggles the senses

Moreover, it’s a place that boggles the senses. On the day of my visit, Flint’s desk was taken over by a prized 17th-Century Persian helmet and a pair of Spanish colonial spurs. She oversees some 1,000 objects in the club’s collection, as well as a library brimming with 14,000 volumes, photographs, slides and reports. One recent acquisition is a century-old Akeley Pancake Camera, dating to 1919 and first built for rugged expeditions.

The townhouse is an intriguing architectural marvel in itself. There are wooden beams taken from HMS Daedalus (an 1826 frigate warship). A ceiling bought from a 15th-Century Italian monastery, plus original stained-glass windows inlaid with Tudor roses from Windsor Castle in England. It’s so out-of-this-world, in fact, it feels as if it could have been designed by Jonathan Swift’s fantastical traveller, Lemuel Gulliver.

One floor up, past the Hall of Fame and the Sir Edmund Hillary Map Room, is the extraordinarily detailed Gallery. A drop-in visitor can see trophies of cheetah and lion from Smithsonian expeditions; a yeti scalp and prayer wheels from Tibet; a first edition of Napoleon’s description of Egypt; an Alaskan mammoth tusk, moose heads and stuffed penguins; a pelt from a man-eating Nepali tigress; and the remarkable ivory of a four-tusked elephant, a rare genetic anomaly from Congo. The horde of artefacts is so exact – so remarkable – that, at first, it feels like a film set suspended in time.

They were rock stars of their age

“The risks these explorers took were crazy,” said Flint, pointing to an oil painting of Danish explorer Peter Freuchen that hung above the fireplace. Freuchen, she told me, wore a coat from a polar bear he killed, and once escaped an ice cave using frozen excrement as an improvised dagger. “These were people who would amputate their own foot. They were rock stars of their age and their stories are just as radical. Some unearthed burial grounds of ancient kings, while others travelled to the Arctic with a full tea service or crossed a desert with a camel carrying a full-size writing desk. Unbelievable, isn’t it?”

Members’ adventures are just as inspiring today. Archaeologist Joan Breton Connelly – aka ‘Indiana Joan’ – continues to dig for clues at a Cypriot temple built by Cleopatra that she discovered, while deep-sea explorer Jennifer Arnold’s passion is diving for megalodon teeth.

To visit the club – possible for around $25 during the society’s weekly public lectures – is to experience an expense of spirit most people can only dream of. While the world of the explorer is changing, the club’s president believes it is a golden era for members – particularly in the fields of palaeontology, anthropology and space exploration.

There is still plenty of magic left in the world – and it’s our job to find it

With evolution comes opportunity and time for reflection, Wiese told me, and over the next few months, the club will undertake one of its most ambitious projects to date: to bring together the largest ever gathering of moon walkers and Apollo astronauts to celebrate 50 years since the Moon landing in July 1969.

“Our challenge is to stay relevant,” Wiese said, looking out the window. “In science, if an organism doesn’t evolve it’ll go extinct. Yes, we have an illustrious history, but our members are focussed on the future – on climate change and on animal and human preservation. So the more we can promote and popularise science to people that have curiosity about the world the better. There is still plenty of magic left in the world – and it’s our job to find it.”

CORRECTION: A previous version of this article misspelled the name of the Explorers Club. We regret the error.

Join more than three million BBC Travel fans by liking us on Facebook, or follow us on Twitter and Instagram.

If you liked this story, sign up for the weekly bbc.com features newsletter called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Culture, Capital and Travel, delivered to your inbox every Friday.