Former deputy prime minister Barnaby Joyce claims he is being stalked but won't deny flicking a man's hat off during a heated encounter in a New England pub.

Fairfax Media can reveal the Nationals candidate was confronted in the Graman Hotel near Inverell on Monday night and questioned about family matters.

When questioned by Fairfax Media on Tuesday morning, Mr Joyce won't deny he flicked the man's hat off his head during the confrontation, but said his "private life is private".

"We've had death threats and now we've got stalkers," Mr Joyce said.