Members of the punk group Pussy Riot have been detained by Russian police in Sochi, apparently on suspicion of stealing a woman's handbag.

Ten people were arrested, including Nadezhda Tolokonnikova and Maria Alyokhina, the Pussy Riot duo who were released from prison in December, as well as local activists and journalists.

All were released after several hours in a police station minutes from the coastal venues for the Winter Olympics. On their release, they pulled on the group's trademark balaclavas and began singing on the steps of the police station.

The Pussy Riot women said they had come to Sochi to record their new protest song, but were simply walking on the street in central Sochi when arrested. They were taken to a police station in Adler, not far from the Olympic Park.

"We were simply walking around Sochi when they grabbed us," said Tolokonnikova by telephone from the police station during her detention. "They told us we are suspected of theft. Of course there has been no theft."

Tolokonnikova said she and other members of Pussy Riot had arrived in Sochi on Sunday evening, since when they have been constantly tailed. They had not previously made their presence in Sochi public, but began to live tweet their arrest on Tuesday, bringing a huge crowd of journalists to the police station in Adler.

The women arrived on Sunday night and were arrested immediately – supposedly to determine their identity, said Alexander Popkov, a lawyer for the women: "Then on Monday they were detained again and questioned for several hours. And then today, for supposed theft from the hotel. Which is being investigated by 12 specialist investigators."

Eventually, the police decided to drop the charges, and five women including Tolokonnikova and Alyokhina pulled on brightly coloured balaclavas and ran down the steps, singing: 'Putin will teach you to love the motherland'.

"Our authorities have this amazing ability to organise a scandal," said Popkov. "If they hadn't arrested them, there would have been none of this crazy media attention."

Those arrested included David Khakim, who was yesterday sentenced to 30 hours of community service by a Sochi court for holding a one-man protest in support of Evgeny Vitishko, a jailed environmentalist. Vitishko has been sentenced to three years in prison for vandalising a fence, in a case that activists say is linked to his criticism of environmental violations during Olympic construction.

Tolokonnikova and Alyokhina were released from prison in December, after serving the majority of a two-year sentence for "hooliganism motivated by religious hatred" for their punk performance in Moscow's main cathedral. They fell under a wide-ranging amnesty approved by President Vladimir Putin that was seen as a way to boost Russia's image ahead of the Olympics.