The defence secretary said parties wanting to frustrate Brexit did not make gains in the election.