Sitting in a glass case, standing out against a backdrop of deep red, there’s an ice axe that still bears a rust mark, the consequence of a bloody fingerprint left on it decades ago. One day in 1940, this axe was hidden inside Ramón Mercader’s suit jacket, suspended by a string, as he walked into the office of Leon Trotsky, the Russian revolutionary living in exile in Mexico, having been sentenced to death as an “enemy of the people” in his home country.

Mercader slipped behind Trotsky’s desk and brought the axe down with tremendous force, penetrating two-and-three-quarter inches into his skull. Trotsky died 26 hours later. Mercader served 20 years in prison then returned to a hero’s welcome in Moscow. On his deathbed in 1978, his last words were: “I hear it always. I hear the scream. I know he’s waiting for me on the other side.”

The Mexican police department kept the axe as evidence, later putting it on display. When its museum director retired in the 1960s, the axe was presented to him as a gift. He passed it on to his daughter who, for 40 years, slept with it under her bed before finally putting it up for sale in 2005. This extraordinary remnant of Stalin’s murderous rule eventually found its way into the International Spy Museum, which has just opened in Washington DC.

‘I hear the scream’ … collector H Keith Melton by the ice axe used to kill Trotsky. Photograph: Saul Loeb/AFP/Getty

The axe is among nearly 10,000 artefacts on display at this sprawling $162m not-for-profit museum. More than half were donated by H Keith Melton, a Florida businessman who spent 45 years travelling the world to collect them. The museum’s previous incarnation, at a venue across town, had about 3,000 artefacts and could only display about 600 at a time. This new home – a giant, upturned wedge of a building not far from the Washington Monument – was designed by Rogers Stirk Harbour and Partners.

The artefacts and installations now have room to breathe, particularly the section about James Lafayette, a Virginia slave who, after claiming he had run away from his plantation, managed to infiltrate British intelligence during the American revolution. As a double agent, Lafayette played a pivotal role in the 18th-century conflict, misleading the British and passing vital information back to his own side. Then, when America celebrated its freedom, he returned to life as a slave, his status as a spy meaning he did not benefit from the Act of 1783, which emancipated slaves who fought for the revolution. After petitioning Congress, he was eventually freed, moving to a 40-acre farm and raising a family.

Gadget heaven … devices containing hidden surveillance equipment including, bottom left, a shoe with a bug in its heel. Photograph: Saul Loeb/AFP/Getty

Which brings us to George Washington, the first president of the United States, who took office in 1789. Having seen the value of espionage in the revolution, Washington was also the country’s first spymaster – and the letter he wrote creating the first US intelligence agency is a star exhibit. It talks of “the advantage of obtaining the earliest and best intelligence of the designs of the enemy” and was addressed to Nathaniel Sackett, a valuable spycatcher, who is offered $50 per month plus $500 to set up an intelligence network. “Washington did not beat us militarily,” one British major said of the revolution. “He simply outspied us!”

There’s Mata Hari, the archetypal femme fatale. The exotic dancer from the Netherlands, who was executed for espionage in 1917, loved celebrity and filled entire scrapbooks with articles and photos of herself. France arrested her for spying for Germany during the first world war. One caption summarises the life of agent H-21 simply: “Seductive dances. Flimsy veils. Invisible ink. Shot by firing squad.”

There’s Sidney Reilly, the “ace of spies”, supposedly the inspiration for James Bond. Born Shlomo Rosenblum in what is now Ukraine, he moved to Britain, travelled the world selling arms, and had three or four wives (“at least two simultaneously”) and up to six mistresses. He staged a doomed counter-coup against Lenin in Russia, where he eventually met his end. His codename was ST1, a reference to the office in Stockholm that handled him, though Bond author Ian Fleming changed this to the much catchier 007. “James Bond is just a piece of nonsense I dreamed up,” Fleming once said. “He’s not a Sidney Reilly, you know.”

Imposing … the International Spy Museum. Photograph: Saul Loeb/AFP/Getty Images

There is Jonna Mendez, who defied gender prejudice to become a CIA agent and “master of disguise”. She even wore a mask for an Oval Office meeting with President George HW Bush and he failed to see through it. The mask, however, is not on display – it remains classified. A specialist in miniature cameras, Mendez went undercover in hostile terrain during the cold war, spying everywhere from Havana to Moscow to Beijing. She is now a photographer and author.

Boasting poison umbrellas and cuffs used to shackle captured agents, the collection is vast, wide-ranging and full of surprises, some stretching the definition of “spy” to breaking point. After being assigned a “secret identity badge” that uses radio-frequency technology, visitors find themselves being “recognised” by interactive exhibits. As you zigzag through the museum, though, it can all feel a little disorderly and unstructured: one minute, you’re reading about the recent capture of Russian agents in the US, the next you’re finding out about the secrets of Venice glass-makers.

One powerful room juxtaposes two of America’s greatest intelligence failures: the attacks on Pearl Harbor in 1941 and on New York in 2001. Among the more startling exhibits is the boarding pass used by Ahmed al-Ghamdi to board United Airlines flight 175 from Boston. One of the “muscle hijackers”, Ghamdi helped to storm the cockpit and keep passengers under control so that Marwan al-Shehhi could fly the plane into the south tower of the World Trade Center.

Weaponising weather … an umbrella similar to the one used by the KGB to kill dissident Georgi Markov in London in 1978. Photograph: Saul Loeb/AFP/Getty Images

Both sections offer lessons on the difficulty of separating signal from noise. A memo from the chief of US naval operations, dated 1 February 1941, is entitled: “Rumored Japanese attack on Pearl Harbor.” It says the Peruvian minister in Tokyo had heard from many sources that “in the event of trouble breaking out between the United States and Japan, the Japanese intend to make a surprise attack against Pearl Harbor with all of their strength”. Meanwhile, a CIA daily briefing, given to President George W Bush the month before the 9/11 attacks, has the heading: “Bin Laden Determined to Strike in US.” It mentions “sensational threat reporting” about Osama bin Laden wanting to hijack US aircraft.

Similarly, eight years after it happened, Bin Laden’s death at the hands of US Navy Seals has become a museum piece, if not an actual video game. A giant screen has the title “The hunt for Osama bin Laden” with icons for “Bin Laden’s courier” and “al-Qaeda leadership”. Visitors sit at touch screens before a model of the compound in Pakistan where Bin Laden was hiding. A simulation shows a figure pacing around. “Who could the pacer be?” a voiceover says. “Remember – the president is deciding to go ahead with a raid on the compound. We need to show him our results.”

Toys and games from movies and TV shows about spies. Photograph: Saul Loeb/AFP/Getty Images

One of the most fascinating sections, about the war on terror, is likely to prove the most controversial. In giant yellow letters stencilled on to a cinderblock wall, this museum – which does not receive government funding – asks: “What is torture?” Visitors can see a waterboarding kit for themselves, or climb inside a replica of a “stress box” that’s too narrow to sit down in and too low to stand up in.

A panel notes that George W Bush signed off on an “unprecedented” enhanced interrogation programme that enabled interrogators to “take off the gloves” and use such techniques as waterboarding, but that Barack Obama ended this in 2009. Another asks provocatively: “Does ‘enhanced interrogation’ work?” A third panel gives this answer: “It depends on who you ask. Many are confident that strong methods are effective. Others, including experienced interrogators, psychologists and people who have endured such techniques, believe that less severe forms of questioning yield better results.” Some visitors are likely to find this less than full-throated denunciation of torture disturbing.

There is a great deal more at the International Spy Museum: the astonishing surveillance techniques of the Stasi in East Germany, with 90,000 agents, many citizen informers and an entire hotel rigged for snooping; a segment of a tunnel built under Berlin in the 1950s to allow the US and Britain to tap into Soviet communication lines; and the story of National Security Agency whistleblower Edward Snowden, including a cutting from the Guardian.

The name’s Martin, Aston Martin … Bond’s famous car. Photograph: Saul Loeb/AFP/Getty Images

Fictional spies are never far away, either. Bond’s Aston Martin, with its rotating licence plate, sits in the entrance hall, while the metal teeth of his giant foe Jaws are displayed in a case. There is a section devoted to TV and film spies, and an exhibit about the real-life suburban couples from Russia who inspired the TV drama The Americans.

Although the narrative is non-linear, you do emerge with a lasting impression of the rapidly changing nature of spycraft, from bugs hidden in shoe heels or tree stumps to the more modern aspects of cyber-espionage. Bringing us bang up to date, Russia’s meddling in the 2016 presidential election also gets a mention. “Details of Russian attacks continue to emerge,” the museum observes, adding ominously: “The story is still unfolding.”