The violent Charlottesville, Va., rally last August energized the alt-right, and many in its ranks predicted their momentum would continue. But a year later, what is billed as a “white civil rights rally” across the street from the White House is expected to draw a smaller crowd, its organizer said.

“It’s been a total fracturing of the right,” said Jason Kessler, who organized the Charlottesville gathering and is planning Sunday’s rally in Washington, D.C.

Infighting,...