What could be more festive than spending a night locked in an art gallery with a dozen reindeer and a fridge full of psychedelic drugs? Soma, Carsten Höller's current installation in a former railway station in Berlin, purports to be offering exactly that. A pen running the length of the Hamburger Bahnhof, now the city's contemparary art museum, contains 12 reindeer, 24 canaries, eight mice and two flies. Giant toadstool sculptures are planted on a mushroom clock that the reindeer can turn with their antlers, and at the centre is a mushroom-shaped "floating hotel" – a bed on a platform complete with minibar, yours for €1,000 a night. (There's also a raffle giving away free places.)

The twist is that this is meant to be a scientific experiment, in which half the reindeer have been fed "fly agaric" mushrooms, which they consume naturally in the wilds of Siberia. It makes their urine hallucinogenic (some people believe that this is the origin of the story of Santa Claus's sleigh being pulled by flying, red-nosed reindeers).

The urine is collected by handlers and stored in fridges by the walls, which also hold both dried and fresh fly agaric mushrooms. By day they're locked, but at night the fridges are opened, allowing people staying over to sample the contents. However, because only half the reindeer are fed the mushrooms, it's impossible to know which bottles, if any, contain hallucinogenic urine.

Tanja Klein, 28, won a competition to spend the night in the museum with her boyfriend, Sachar Kriwoj, 30. "I wasn't going to go and drink six bottles of reindeer urine to find out," says Klein. "I'm not into drugs, I'm into art."

Höller hasn't tried the urine, but he has tried the mushrooms. "They're very unpleasant," he says, speaking from his home in Stockholm. "And you throw up. The first four times I tried it, I became comatose. Then you wake up, throw up, and you don't know where you are, or how long you've been asleep. The sixth time, I started to chant like a Tibetan monk."

The title Soma comes from the name of the sacred libation drunk by the Indo-Persian followers of the Vedic religion, Hinduism's 5,000-year-old parent. Its ancient text, the Rigveda, contains 114 hymns to "creative juice", supposed to offer immortality. The recipe was lost, but in the 1960s researcher Robert Wasson hypo-thesised that soma was based on the fly agaric mushroom.

Höller's installation sets out to test this hypothesis – and the possibility that art may change perceptions even more effectively than drugs. It takes the form of an experiment set in a playground: from that giant "double mushroom clock" the reindeer move with their antlers, to the "mice square", based on an actual playground in Paris designed by sculptor Pierre Székely.

One side of the hall is the "test", the other the "control". Reindeer on the test side are fed the mushrooms. ("At least in principle," says Höller, helpfully.) On each side, the reindeer urine is spread on the food of the other animals. From observation posts, visitors watch the behaviour of the canaries, mice and houseflies for signs of intoxication and form their own conclusions. "The experiment is completed in the minds of the visitors," says Höller. "It's very unscientific." In other words, it's an open question whether the reindeer are even fed the mushrooms at all: the power of suggestion makes you likely to observe something that may not take place.

Experimentation has been a part of Höller's work since he began his career as an artist while still an agricultural research scientist in the early 1990s. He went on to install 2006's Test Site, in Tate Modern's Turbine Hall, which allowed gallery-goers to throw themselves down double-helix slides.

Overnight visitors to Soma have reported some strange events. Florian Wojnar, a friend of Höller's, spent the night in the museum with his 11-year-old son. "He was really excited, because at some point, there were seven reindeer on one side and five on the other. In the morning, we counted again and there were six on each. I never saw them move."

Dorothée Brill, the museum's lead curator, says: "As far as we can tell, nobody's done anything they shouldn't have." Staff at the restaurant, however, report that some guests "drink the minibar dry".

It's hard to resist the suspicion that the exhibition is intended as a microcosm of society, an allegory for democracy, with extra privileges and more fun for those able to pay. And, if this is an experiment, make no mistake: it's you in the lab. Meanwhile, those tempted to make a Christmas visit should bear in mind that the Hamburger Bahnhof is closed on Christmas Eve. "The reindeer have somewhere else to be that day," the museum explained.

• Soma is at the Hamburger Bahnhof, Berlin, until 6 February. Details: somainberlin.org