The right-wing United Kingdom Independence Party voted 867 to 500 to oust its leader, Henry Bolton, in a vote of no-confidence on Saturday, making him the fourth party leader to step down in the space of 18 months.

The party, which put its stamp on history by goading a reluctant prime minister into calling for a referendum to leave the European Union, has since drifted to the ideological fringes, struggling to cope with debts and a fractious membership.

Mr. Bolton, a married former army officer, was brought down by a matter of the heart.

In January, he was badly damaged by revelations that his girlfriend, Jo Marney, had used racist language to deride the actress Meghan Markle, who will marry Prince Harry this spring.

In text messages to a friend that were published by The Mail on Sunday, a tabloid, Ms. Marney expressed her distaste for interracial marriage, saying Ms. Markle’s “seed” would “taint our royal family” and lead to a “black king.”