One Nation represents “the old racist element” of the Labor vote, the Queensland Labor senator Anthony Chisholm has said on Twitter.

Chisholm made the remark in response to a speech by the One Nation senator Malcolm Roberts in which he claimed his party was “the real successor” to the Labor party of the 1940s, 50s and 60s.

In comments to Guardian Australia, Chisholm has clarified that he was “calling the old 1960s Labor policy of supporting White Australia racist” and accusing Roberts of “yearning for the Labor party to go back to these times”.

In a speech to the Australian Industry Group conference in Canberra on Monday, Roberts argued that since the 1970s Labor has “undergone a constituency inversion, abandoning workers in favour of leftwing elites and taxpayer-funded freeloaders”.

He said the trend was “reflected in modern Labor candidates who are all middle-class lawyers or professional union officials and wouldn’t know a real worker if their chauffeured commonwealth car ran over one”.

When Roberts posted a link to the speech on social media, Chisholm replied with a tweet appearing to concede that a section of people who formerly voted Labor were racists:

Only the old racist element m8 and it's all yours. https://t.co/3t7juyeL4o — Anthony Chisholm (@AnthonyChisholm) May 1, 2017

Chisholm was elected to the Senate in 2016 and is a former state secretary and campaign director of the Queensland Labor branch.

A spokesman for Chisholm confirmed that the account was the senator’s and the tweet was authored by Chisholm personally.

“It is a well-known fact that the White Australia policy was racist,” Chisholm said. “The Labor party has moved on and it was the Whitlam government that formally abolished the policy.

“It is currently One Nation who want to support racist and discriminatory policies and I will absolutely call them out on it every day of the week.”

The White Australia policy was dismantled over a decade from the mid 1960s and ended under the Whitlam Labor government in 1973. It was supported by Labor’s John Curtin, the prime minister during the second world war, and the first immigration minister, Arthur Calwell.

Senator Anthony Chisholm Twitter profile with tweet about ‘old racist’ ex Labor voters Photograph: Twitter

In his speech Roberts had argued that One Nation “is the party of the nationalist working class” in contrast to “the Liberal party, whose origins were as a conservative middle-class party, and the modern Labor party, which has become the party of the government employed middle-class and of professional welfare recipients”.

“What makes us a ‘workers’ party’ is that the membership and support base of One Nation is made up of often poorly paid, hardworking Aussies whose basic decency, quiet patriotism, strong moral compass and fierce work ethic define them as the heart and soul of our nation.”

Roberts pointed to the backgrounds of the four federal One Nation senators, claiming one was a boilermaker (Brian Burston), one an electrician (Peter Georgiou), one a coalminer (Roberts) and one a fish and chip shop owner (Pauline Hanson).

“What makes us a nationalist party is our patriotic commitment to Australia, including our commitment to defend the traditional English speaking, predominantly European makeup of Australian society,” he said.

“Coupled with One Nation’s strong support for traditional family values, this makes us in many ways like the Labor party of Ben Chifley, John Curtin and Arthur Calwell.”