A federal jury on Friday dealt a potentially costly legal blow to Rolling Stone magazine and a reporter in a defamation trial over its discredited 2014 story about an alleged fraternity party gang rape at the University of Virginia.

A 10-person jury in Richmond, Va., found that Rolling Stone defamed a university administrator who was in charge of handling student complaints of sexual misconduct at the school.

The...