Mick Jagger has pulled out of an event due to be hosted by David Cameron at the Davos summit, complaining of being used as a "political football".

The Rolling Stones singer had been due to attend a tea party at the Swiss resort as part of a campaign to promote Britain alongside others such as the model, Lily Cole, and the inventor Sir Tim Berners-Lee

But he said in a statement on Tuesday: "During my career I have always eschewed party politics and came to Davos as a guest, as I thought it would be stimulating."

He added: "I have always been interested in economics and world events. I now find myself being used as a political football and there has been a lot of comment about my political allegiances which are inaccurate. I think it's best I decline the invitation to the key event and curtail my visit."

Jagger arrived in Zurich on Tuesday morning and will now be flying out of Switzerland on Wednesday, said a spokesperson for the singer.

The withdrawal will come as an embarrassment for the government. The original announcement had sparked a flurry of headlines in the UK such as 'Sympathy for the Dave-il' and 'Pain it Blue'. Coverage stated that the singer had finally "come out" as a "closet Tory" and the prime minister was said to be delighted by Jagger's involvement as the event's "star attraction".

Earlier, a Downing Street source had welcomed Jagger's involvement amid efforts to plug Britain in a year that will see the Olympics in London and the Queen's Diamond Jubilee. But a senior government source later denied that his involvement in the tea party – part of a wider "Great" campaign to encourage investors and tourists – had been politicised.

"Clearly we are disappointed that Sir Mick doesn't feel he can support a non-political event that promotes Britain," the source said.

"At no point was there ever any suggestion that Sir Mick was a Conservative. He understandably wishes to remain apolitical. This is not a political event. This is simply an event that promotes British creativity abroad."

It is not clear to what extent Jagger's withdrawal will come as a disappointment to Cameron, who has talked in the past about his fondness for the music of the Killers and the Smiths, although the prime minister was instructed in 2010 by Johnny Marr, a founding member of the latter band, to stop saying that he liked them.

He is also not the first resident of Downing Street to discover that relations with rockstars can have a prickly side. Oasis's Noel Gallagher – who met with Tony Blair in Number 10 in 1997 as part of New Labour's courtship of Brit Pop – launched a scathing attack on the prime minister 10 years later. He said voters had believed Blair "was going to be John F Kennedy", but he had become "saddled" with the Iraq war.

Damon Albarn replied to a 1997 party invite from Blair, telling the then prime minister: "I'm sorry, I won't be attending, as I am now a Communist. Enjoy the schmooze, comrade!"

A spokeswoman for Boris Johnson, who is also attending the event at the five-day summit, said: "The British delegation at Davos decided having a cup of tea and a biscuit mid-afternoon was the most frugal and efficient way of getting access to the global financial elite and to tell them about investment opportunities in UK, not least in this Olympic year."

"The mayor was asked if his name could be put on the invitation in order to attract the best crowd and be there to bang the drum for London."