Liberal MP John McCallum says the prime minister should remove New Brunswick MP John Williamson from the Conservative caucus following Williamson's comments about "whities" and "brown people" in the temporary foreign worker program.

In question period Thursday, McCallum accused Stephen Harper of remaining silent on Williamson's comments, which were made last weekend.

He said Harper was giving in to "racist sentiments within his own party."

Williamson told delegates at a conference in Ottawa that it makes no sense to pay "whities" to stay home while companies bring in "brown people" as temporary foreign workers.

New Brunswick Southwest MP John Williamson was in Question Period on Thursday but he did not respond to Liberal MP John McCallum's comments. (Fred Chartrand/Canadian Press) The New Brunswick Southwest MP later apologized "unreservedly" on Twitter for his "offensive and inappropriate language."

Williamson attended question period on Thursday, but Pierre Poilievre, the minister of employment and social development, responded to McCallum's comments instead.

Poilievre ignored McCallum's remarks and referred instead to complaints made by the Centre for Israel and Jewish Affairs about a speech by Justin Trudeau, in which the Liberal leader compared Canada’s immigration policy for Jews during the Holocaust to present-day fear-mongering about Muslims.

Williamson, a former director of communications for Harper, has called his comments about temporary workers the biggest mistake of his political career.

On Wednesday, New Brunswick member of the legislative assembly John Ames, also a Liberal, introduced a motion to condemn Williamson's comments and called them racially charged.