Former Top Gear presenter Jeremy Clarkson says times were "tricky" in the run-up to the BBC dropping him from the show.