David Cameron faced calls last night to remove the Tory whip from the backbench MP Aidan Burley after a report commissioned by the party concluded that Burley acted in a "stupid and offensive way" when he bought a Nazi uniform for a stag party.

Ian Austin, the former Labour minister, criticised the prime minister for standing by Burley, who the report said was naïve for remaining at the stag do during a "Nazi-themed toast".

But Burley won support from the Tory leadership after the report's author, the Conservative peer Lord Gold, concluded that the MP was not a racist or antisemitic.

The Oxford-educated Burley, who had been marked out by No 10 as one of the most impressive Tories elected in 2010, lost his job as a ministerial aide after it emerged he had bought an SS uniform for a stag party at the Alpine resort of Val Thorens in 2011.

He had initially been allowed to keep his job when the Mail on Sunday revealed in December 2011 that he was the best man for his Oxford friend Mark Fournier who wore the SS outfit. But Burley was sacked when it emerged a week later he had hired the costume for Fournier, 33.

Fournier was yesterday fined €1,500 (£1,230) by a court in the French Alps after being charged with "wearing a uniform or insignia of an organisation guilty of crime against humanity".

The court also ordered Fournier to pay €1,000 in damages and interest to an organisation representing families of those who disappeared from France or were deported to the death camps in the second world war.

Gold's report, released after the conclusion of proceedings against Fournier, criticised Burley for buying the costume and for failing to protest when the groom's brother John raised a Nazi toast.

Gold, who accepted Burley's explanations of the evening's events, wrote: "The report concludes that Mr Burley's conduct caused deep offence and that he acted in an unacceptable and offensive way … Mr Burley is not a bad man, still less a racist or anti-Semite. However, his actions were stupid and offensive, and the conclusions and recommendations reflect that."

Gold's findings prompted the Conservative party to release a statement by Michael Ellis, parliamentary private secretary to the party chairman Lord Feldman, declaring that Burley has a bright future in public life.

Ellis said: "Aidan is a hard-working MP who is committed to his constituents in Cannock Chase and I do not believe Aidan is at all anti-Semitic. He has paid a heavy price for his foolish behaviour over two years ago and has rightly apologised repeatedly for his error of judgment. I am confident he will be an effective representative for the people of Cannock Chase and he has a lot to offer in public life."

But the Labour MP Ian Austin said that the Tory whip should be removed from Burley on the grounds that he gave a misleading account of the stag party. In an article for the Jewish Chronicle on 15 December 2011, four days after the revelations about the stag do were published in the Mail on Sunday, Burley issued a strong apology but disassociated himself from the "clearly inappropriate behaviour" of other guests.

He lost his job as a ministerial aide the following June after the Mail on Sunday reported that he had in fact bought the SS uniform for the groom.

Austin said last night: "He claimed at the time that this event was a fancy dress party, and he told his constituents and the Jewish Chronicle that he had not known what was going to happen. The trial in France and this report shows neither of those statements was true and it beggars belief that anyone should believe him now. His behaviour was an absolute disgrace and I can't understand why the Tories haven't removed the whip."

Austin said he was surprised that the Tory leadership was throwing its weight behind Burley. "Why is the Tory party so determined to stand by someone who has admitted to bringing disgrace on his constituency and who many people feel is not fit to be an MP?"

Neither Burley nor any of the other 10 guests who attended the party were prosecuted.