A tea-party-backed challenger resoundingly defeated Rep. Eric Cantor of Virginia, the second-ranking House Republican, in a primary election Tuesday, an unexpected assertion of insurgent conservative power sure to reshape Republican policy and leadership.

Mr. Cantor, considered the leading candidate to be the next House speaker, lost to David Brat, a college economics professor, by 11 points, 55.5% to 44.5%.

Mr. Cantor's defeat marked an unexpected and staggering turn in this year's primary-election season, overturning the building narrative that Republican Party leaders and allied business groups had trampled the GOP's tea-party wing, which has fought to push the party to the political right.

The result in the district north and west of Richmond will echo through the ranks of Republican officeholders nationally, showing the risks that even senior leaders can face in crossing the party's activists. Mr. Brat was an underfunded candidate who drew little support from the national groups that have spent millions of dollars in other races to try to oust Republican incumbents that they deem insufficiently conservative.

While other GOP incumbents facing tea-party challengers—such as South Carolina Sen. Lindsey Graham, who won his primary Tuesday night—worked for months to crush their challengers, Mr. Cantor didn't push back hard against his opponent until too late in the campaign.