Roundtrip time — latency — is up, particularly between Comcast and Time Warner Cable and Level 3. Throughput — the number of packets allowed through the door between the two networks — is down, particularly between Verizon and Level 3. The uniformity of these issues in different geographic areas between the same pairs of companies signals that this is not a technical problem. This is a business dispute that is harming American consumers.

Across many of these connections, the effect on consumers continues. The practical dimensions of these effects vary. If a delayed or dropped data packet is part of an email, then there is probably a short delay in the delivery of that message. If the data packet is carrying information for a performance-sensitive application like a voice phone call or a streaming video, the user may be experiencing much more trouble — garbled calls and degraded video. Applications (like Netflix, Sony, Apple, Google, Amazon, and others) likely are having more trouble working and online streaming services are likely affected (Major League Baseball, for example). Problems are particularly bad at peak usage times; the doorway between the two networks is simply too narrow for the amount of data that needed to be handled. Level 3 is the largest transit provider in the world, but data coming from Level 3 is not being treated well.

Why would Comcast, Verizon, and Time Warner Cable make their users’ experience accessing the online world worse? The obvious answer: money. In the Cogent “cliff” context, Cogent had signed a deal with Netflix, and Comcast and Verizon wanted to make sure that they were getting a slice of Netflix’s consumer revenue. Also, Comcast sells its own transit network services, and would be happy to sell those services to customers — so Comcast is effectively competing with Cogent, Netflix’s transit carrier, and Level 3 — as well as Netflix.

In the Level 3 “slope” context, it is the naked principle of the thing: eyeball networks want payment before they will upgrade their interconnection points to the transit networks.

And the payment will be whatever the eyeball networks ask. “Paying for upgrades,” in the eyeball networks’ view, means paying an invented cost for traffic volume plus whatever equipment costs that Comcast and the others say they need to invest to handle the traffic. Such costs may be minimal—a $300 piece of fiber or other cheap interconnection equipment upgrades. But as last-mile monopolies, the eyeball networks can invent fees and then assess them, and force everyone else in the system to pass those costs on to subscribers.

In this case, there seemed to be no upgrades, despite a clear need for them. Thus, as traffic predictably grew on Cogent’s network from increasing numbers of Netflix subscribers streaming videos, but the width of Cogent’s doorway into Comcast’s and Verizon’s network didn’t change, packets started dropping. And Devan Dewey started getting inexplicable complaints.

Before M-Lab’s data, of course, affected customers and businesses had no way of knowing why the Internet access they had paid for was not being delivered. Dewey scrambled for weeks, trying to figure out why his employees could no longer do their work. Finally, after hiring an Internet “health” monitoring firm, Dewey’s team discovered a chokepoint: the interconnection point in New York City. Packets traveling from NEPC’s offices to NEPC’s workers via Cogent were being lost at the moment Cogent handed those packets to Comcast and Verizon in New York City for delivery over the eyeball networks to the worker. Just two percent of packet loss can make a perceptible difference in the quality of a phone call; at the height of NEPC’s problems as many as 20% of their packets weren’t making it through, which is why the firm’s remote work ground to a halt.

Cogent confirmed there was an issue: “We have a problem in New York,” Cogent told him. Dewey ended up buying a second Internet connection from NEPC to the outside world in addition to his Cogent line, and spent dozens more hours getting that circuit up and running. All told, Dewey estimates that his firm spent about two hundred employee-hours trying to diagnose and fix the problems the firm’s employees were encountering — not counting the time wasted by frustrated NEPC employees who were unable to work without interruption or delay.

In part, the reason why it took Dewey so long is that the answer is counter-intuitive. NEPC had nothing to do with Netflix, so why was it affected by Comcast’s and Verizon’s apparently anti-competitive activity? The answer requires a bit of explanation.

Cogent sells two flavors of services to businesses: Internet access — the connection that NEPC bought for its operations center — and Internet transit, or connections between networks.

Cogent has assembled its network by leasing segments of fiber across the country and it aggressively charges less in order to gain more customers; it connects to 1400 office buildings and would like to have more data centers on its list. NEPC’s problem was that it had bought the first flavor of service from Cogent: NEPC is one of many professional services firms (legal, financial, advertising, consulting) located in multi-tenant office buildings that use Cogent for Internet access.

But Cogent was being systematically disfavored by Verizon, Comcast, and the other providers because of its sale of the second flavor of service — transit — to bandwidth-intensive Netflix. Both flavors of services look the same from the perspective of a fiber network; they’re just packets flowing over Cogent’s leased or owned facilities. When the Cogent transit network seeks to interconnect with the Comcast or Verizon eyeball network to hand off packets, things can evidently go wrong. NEPC’s communications cataclysm was a byproduct of the battle between eyeball and transit networks. In order to make life miserable for Netflix and force that company to share its revenue with the eyeball networks, Comcast and the others had simultaneously made life miserable for many other companies.

Comcast’s response: this is business as usual.

“The M-Lab Interconnection Study only confirms what network engineers have always known: If a network does not obtain sufficient capacity to deliver its content to another network that content runs the risk of being transmitted over congested links which could degrade performance. That is why Comcast reached out [to] transit providers, CDNs, and content providers to negotiate business agreements that would provide them with the capacity they needed, as we explained in our sworn declaration before the FCC. As our declaration shows, Netflix hailed our agreement as great for consumers, and affordable for them.” [The Comcast statement ignores Netflix’s more recent clarification to the FCC on why that agreement occurred; “It is extortion when Comcast fails to provide its own customers the broadband speed they’ve paid for unless Netflix also pays a ransom.”]

This interconnection problem is not confined to the world of telecommuting. Andrew Boegly is the Chief Information Officer of the Colonial School District in Plymouth Meeting, Pennsylvania — a Philadelphia suburb. He oversees a state-of-the-art fiber optic network, a 10 Gbps backbone, that serves the district’s 4,633 students and 718 teachers and staff. Since about May of this year, however, Colonial’s Internet access connection to the outside world has been intermittently experiencing severe packet loss, causing enormous practical problems. Colonial’s payroll data is due to the processing company by 8pm every other week in order for employees to be paid two days later, and Colonial is required to send its payroll information online. But recurring connection issues have meant racing against the clock to re-submit the payroll data multiple times until it goes through. “A lot of times it came down to the last minute with us calling the end [payroll] provider who actually did it manually for us because we couldn’t do it,” he says.

Boegly does not have a large tech staff at his beck and call, but he and his team put in countless hours trying to trace the problem. “It was a lot of wasted time, a lot of impact for us in terms of employee work,” says Boegly. They determined that they were suffering packet loss — a network along the way was ignoring and not forwarding some of the data they were sending — a sure sign of network congestion. Because Boegly uses a Citrix application for payroll data, and “Citrix doesn’t like packet loss,” according to Boegly, the process was failing. After getting in touch with Cogent, with whom Colonial’s aggregating larger school district, the Montgomery County Intermediate Unit or MCIU, contracts for Internet access service, Boegly learned what was happening and why.

After Colonial’s payroll data leaves its network it is transmitted via fiber to the MCIU. From there, the MCIU forwards it on to Cogent, which transports it to an interconnection point with Verizon for delivery to its final destination. And guess what happens next? Just after the handoff from Cogent, Verizon loses many of the packets, often causing the data transfer to fail. “Any time you have packet loss over a Citrix connection, it’s going to dump you,” says Boegly. “And that’s exactly what’s happening in the middle of the process. It has to then be started all over again and then begin the transfer process to get it to connect.”

When Boegly complained, he got a classic runaround: Cogent blamed Verizon for not upgrading its ports, and told Boegly the problem had to do with Netflix. After working with Verizon, going up the ladder from the online tech support team to senior level Verizon customer service, Boegly was told by Verizon that the problem was Cogent’s fault. Meanwhile, Colonial’s problems persist.

Boegly says he feels “helpless.” “You know, I contacted the FCC. I contacted the [state] utilities commission, just for help because you couldn’t make any ground with Verizon, Cogent, or anyone,” he says. “It’s just not a priority for anyone.” Colonial pays quite a lot of money for its connection, and Boegly has spent an enormous number of hours trying to sort out this problem. There has been no progress. As a stopgap, Boegly purchased a wireless card to use when he has connectivity issues. The school district with state-of-the-art fiber will now submit its payroll using the equivalent of a cell phone connection. “It’s the giants that are out there that are not really paying attention to these little incidents, which may be small for them, but they’re large for us,” says Boegly.

M-Lab’s thorough research substantially narrows the likelihood that the problems faced by consumers as a result of these congestion issues are caused by anything other than business decisions about interconnection.

Consider this: The lab’s newly-published study shows that NYC-interconnected traffic from eyeball subscribers of Verizon, Comcast, and Time Warner Cable had no trouble reaching M-Lab nodes hosted on Internap, a different content network — a network that isn’t Cogent or Level 3. Performance between Internap and users was exponentially healthier than that between Cogent and those same users: data traveling from Internap to Comcast users zipped along at download speeds of 12 or more Mbps and experienced far lower retransmission and latency rates.