A patent-holding company that won a huge court victory against Apple had its victory wiped out today, and its stock plunged by more than 40 percent.

Nevada-based VirnetX won a jury trial against Apple earlier this year. An East Texas jury ruled that Apple must pay $625.6 million to VirnetX for infringing four patents. The patents are said to cover Apple's VPN on-demand feature, as well as FaceTime.

US District Judge Robert Schroeder, who oversaw the trial, published an order (PDF) on Friday that vacates the verdict and orders a new trial to begin in September.

Schroeder said that the complex, consolidated trial, as well as repeated references to an earlier trial in which Apple was found to have infringed VirnetX patents, created "potential for juror confusion" and unfairly prejudiced Apple. The judge quoted a VirnetX attorney reciting an argument he viewed as potentially prejudicial.

"There was a trial in November of 2012 right here in this courthouse and right here in this courtroom," the VirnetX lawyer said. "And at trial Apple said those same remarks about not using the patent. And you know, the jury didn't believe them and agreed with us. And there was a verdict in late 2012, November, that [VOD] and FaceTime infringes."

The earlier trial was "discussed in depth with multiple witnesses" and thus may have had a prejudicial effect, Schroeder wrote—even though the earlier trial is relevant to whether Apple's infringement was "willful" or not.

Schroeder emphasized that the consolidated case, while complicated, "could very likely have been tried successfully, and fairly," if the earlier verdict hadn't been mentioned "repeatedly" and "often unnecessarily" throughout the trial.

"Under the circumstances here, the repeated references to the prior jury verdict in the consolidated case resulted in an unfair trial," Schroeder wrote. He set a date for a new trial, on September 26.

VirnetX CEO and President Kendall Larsen said in a statement today that his company is "disappointed" by the decision and is reviewing its options. "We are confident that we have the resources required for these retrials," he said. "We trust that the jury will again make the right decision in the retrials."

The VirnetX patents (1, 2, 3, 4) originated at a company called Science Applications International Corporation, or SAIC. VirnetX has been saying for years it has plans to market various products, but for the time being, its income comes from licensing patents. It has 14 employees and leases a small office suite in Zephyr Cove, Nevada, according to its most recent yearly financial statement.

Life has gotten a bit easier for patent defendants in recent years, and claims by standalone licensing companies like VirnetX, sometimes called "patent trolls," have dropped. New legal precedent has made it easier for defendants to win on the grounds that patents are too abstract. A process called inter partes reviews, or IPRs, has served to knock out hundreds of software patents at the US Patent and Trademark Office.

Those tools won't help Apple beat VirnetX's already-tested patents, though. The matter may ultimately have to be resolved by an appeals court.