WASHINGTON—A standoff between Republicans and the Obama administration over a botched gun-trafficking operation escalated Wednesday, with a House committee voting to hold Attorney General Eric Holder in contempt of Congress.

The party-line vote came hours after President Barack Obama, for the first time, asserted executive privilege, aiming to block Republicans from gaining access to Justice Department documents about the operation.

House...