Former high court judge dismisses union applications that he disqualify himself from the inquiry after row over plan to speak at Liberal party fundraiser.