Labour MPs have called for philosopher Sir Roger Scruton to be sacked from his newly appointed position as chair of a government housing commission after it emerged that he had described Jews in Budapest as forming part of a “Soros empire”.

Backbencher Wes Streeting accused Scruton of propagating antisemitic conspiracy theories in a 2014 lecture that referred to the Hungarian-born currency trader and philanthropist George Soros. He called on the communities secretary, James Brokenshire, to dismiss him.

Scruton was last week appointed by Brokenshire to chair the Building Better, Building Beautiful commission, an unpaid job on a new body, which aims to promote a design and style of homes that “reflect what communities want”.

The Conservative philosopher is a frequent visitor to Hungary and says he has known its nationalist prime minister, Viktor Orbán, since 1987. In 2014, he gave a lecture in the country, which referred to Soros, Judaism and Islam.

“Many of the Budapest intelligentsia are Jewish, and form part of the extensive networks around the Soros empire,” Scruton wrote in a lecture entitled The Need for Nations, which is reproduced in Hungarian and English on his website.

Orbán’s ruling Fidesz party has run a number of campaigns against the supposed threat to Hungary and Europe posed by Soros. These have accused him of funding a “Soros plan” to erase national identities and increase migration flows to Europe. Last month, an institution Soros helped found, the Central European University, said it would leave the country.

Scruton’s lecture acknowledged that antisemitism was a problem in modern Hungary. It read: “Moreover, as the world knows, indigenous antisemitism still plays a part in Hungarian society and politics, and presents an obstacle to the emergence of a shared national loyalty among ethnic Hungarians and Jews.”

Nonetheless, Labour MPs said that there were questions for Brokenshire to answer for appointing Scruton. Luciana Berger said: “An individual who peddles antisemitic conspiracy theories has no place advising government about anything,” while the shadow housing minister, John Healey, asked the minister in a tweet: “Can you explain this?”

The lecture, first picked up by the Red Roar website, also argued that the nation-state is a Christian, European construct, which explained why a country such as Pakistan was a “failure”. Focusing on Islam, Scruton wrote: “The same is true of many other countries in which Islam is the dominant faith. Even if such countries do function as states, like Pakistan, they are often failures as nations.”

Scruton’s observations about Soros are not isolated. The philosopher used an article in the Daily Telegraph in February 2018 to discuss sovereignty and Brexit, and whether leave or remain factions “has the nation at heart”.

In the article, he referred to Best for Britain, a pro-remain campaign, which has accepted donations from Soros. It said: “And even if it accepts donations from George Soros, whose behaviour in Hungary, Albania and the Balkan states has been inspired at every stage by a visceral hostility to the national idea, this does not affect the issue.

“It is possible that this man, whose currency speculation once almost bankrupted our country, is as devoted to the British national interest as is Gina Miller.”

When asked about Scruton’s comments about Soros, the department of communities said: “Due diligence was carried out prior to Sir Roger Scruton’s appointment as chair.”

A spokesman added: “Professor Sir Roger Scruton, as a long-standing public intellectual, has strong views on a number of issues. He received a knighthood in 2016 and advised the coalition government on design.

“His commitment to driving quality in the built environment is well known and he has published extensively on architecture and place, which makes him an excellent candidate for the unpaid chairmanship of the Building Better, Building Beautiful commission.”

Scruton did not respond to a request for comment.