When Netflix and its transit providers fought with cable companies and telcos over who should have to pay for network upgrades, it’s no secret that innocent bystanders were harmed. Companies that sent data over networks that were congested because of money disputes got poorer performance, despite having no direct role in the fight.

One new detail we learned today sheds some light on how customers were affected in that dark period earlier this year. The detail comes from Cogent, an IP transit provider whose paths into the networks of Comcast, Time Warner Cable, Verizon, and AT&T were congested when the ISPs refused to upgrade network connections unless they were paid to do so.

Cogent VP of IP Engineering Hank Kilmer wrote that Cogent deployed a Quality of Service (QoS) system that de-prioritized traffic from wholesale customers so that traffic from retail customers would get through without dropping packets.

Wholesale customers are the ones delivering large-scale services over the Internet. Netflix is the largest but Cogent has other, unnamed wholesale customers who suffered because of the Netflix money disputes. When Cogent had to drop packets because of congestion in its connections to ISPs, the packets of wholesale customers were dropped before those of retail customers. Cogent lists a few customer success stories on its website but a spokesperson declined to identify wholesale customers besides Netflix.

Retail customers are smaller businesses whose use of the Internet would more closely resemble a typical consumer than a giant Web company.

“Retail customers were favored because they tend to use applications, such as VoIP, that are most sensitive to congestion,” Kilmer wrote.

The QoS wasn’t any more granular than that, Cogent officials told Ars. Traffic was either labeled wholesale or retail, with wholesale being given the lower priority regardless of the source. The system was deployed in February and March.

This helps explain data from an M-Lab study we wrote about last week, which showed that connections between Cogent and ISPs were terrible for more than half a year before suddenly improving in February and March. The study measured retail connections, which benefited from the QoS system.

A matter of perspective

The M-Lab study set off something of a philosophical debate. Those who believe Netflix and Cogent should have to pay ISPs to upgrade their networks say the data proves Netflix and Cogent deliberately saturated the networks to force the ISPs into offering a better deal. ISPs that had agreed to provide free access to Netflix suffered no serious congestion, and network operators that delivered only non-Netflix traffic to ISPs did not suffer congestion, either.

Frost & Sullivan analyst Dan Rayburn is in the anti-Netflix and anti-Cogent camp, writing that the study “sheds light on [the] widespread harm caused by Netflix routing decisions.”

Susan Crawford, a law professor and former tech policy advisor to President Obama, wrote that the M-Lab data showed just the opposite, that “Verizon and Comcast, as well as Time Warner Cable, CenturyLink, and AT&T, are intentionally squeezing data coming from some incoming networks—in particular, networks associated with Netflix, which competes with these companies in video entertainment.”

After Cogent’s statement today, Rayburn wrote a post titled, “Cogent now admits they slowed down Netflix’s traffic, creating a fast lane & slow lane.”

“What Cogent did is considered a form of network management and was done without them disclosing it, even though it was the direct cause of many of the earlier published congestion charts and all the current debates,” Rayburn wrote.

Cogent disputes his interpretation in a phone interview. There weren’t “fast” and “slow” lanes as the phrasing is usually meant because services weren’t sped up or slowed down, they said. Packets were dropped when necessary and re-sent as soon as possible.

The system “took a lot of testing, took a lot of engineering time and resources,” Kilmer told Ars. Kilmer wrote that the system is “consistent with recommendations from BITAG (Broadband Internet Technical Advisory Group).” BITAG also recommends disclosing network and congestion management practices. Cogent said it implemented the network management in a "visible" and "transparent" way, but the company didn't discuss it publicly when the system was implemented.

The system was put in place only because Internet service providers refused to upgrade connections to Cogent to meet new capacity needs, Cogent Chief Legal Officer Bob Beury told Ars.

“The problem was, after years of upgrading connections as necessary to accommodate the flow of traffic as necessary at these peering points, Comcast, Time Warner Cable, Verizon, and AT&T refused to do so and let their customers be hurt,” he said.

As Ars readers no doubt remember, Netflix ended up paying those four ISPs for direct access to their networks. This diverted the traffic away from Cogent’s connections to the ISPs, improving performance for other users of the links. Beury said there is still congestion in Cogent’s links to the ISPs, but not as much.

The QoS system is still used during times of congestion. Back in February, Cogent told Ars that packets were being dropped around the clock. Now it’s less, “maybe one hour in a day,” Cogent said.

Beury disagreed with Rayburn’s opinion that Netflix was responsible for causing congestion. “That’s completely untrue,” Beury said. “Netflix wants to get its traffic to the customers of AT&T, Verizon, Time Warner Cable, and Comcast, and those customers want to see it. It’s idiotic to suggest that Netflix would try to harm its customers.”

UPDATE: Rayburn pointed us to a statement Cogent makes on its website claiming that it doesn't prioritize traffic. "Cogent practices net neutrality," Cogent writes. "We do not prioritize packet transmissions on the basis of the content of the packet, the customer or network that is the source of the packet, or the customer or network that is the recipient of the packet." As described in this story, Cogent is prioritizing packets based upon the customer, and has been for most of the year.