Speed problems in both the US and Europe

In addition to the Netflix complaints by Verizon customers, Time Warner Cable customers recently complained about poor YouTube service. One reddit thread shows how some technically inclined Time Warner customers used traffic analysis tools to identify the cause of YouTube problems. They were able to improve performance by blocking certain IP addresses, likely ones associated with slow YouTube content servers.

Time Warner Communications Director Jeff Simmermon wrote in a blog post last month that a "small subset of our customers... seem to think we are intentionally degrading their service. If I were to sum up these kinds of complaints, it would go something like this: 'Hulu and YouTube wouldn’t intentionally degrade their services and provide a subpar experience, but for some reason, it makes total sense that Time Warner Cable would by throttling customers’ traffic.'"

Simmermon further noted that "some, but not all, online video providers have the resources to store copies, or caches, of their videos on servers that are a part of a Content Delivery Network, or CDN. When a user clicks on a link to a particular video, the Internet provider can quickly determine where the nearest cached copy of the video is on the CDN and deliver it. More popular videos will have more cached copies, with better performance, while less popular videos may be stored in fewer, further places."

What he didn't comment on is whether Time Warner accepts Google's YouTube caches into its network. But as noted earlier, Time Warner has refused to accept Netflix's caching equipment. Netflix may still be hosting its own caches throughout the country, but if they're not in Time Warner data centers, Time Warner customers get worse performance than they otherwise might.

Europeans are suffering similar problems. "There is a kind of hot/cold thing," said telecom watcher Benoît Felten, the CEO of Diffraction Analysis. Felten lives in France and is a customer of Free, an ISP that has been accused of throttling YouTube, or at least under-investing in the connections that pass YouTube traffic to end users.

"Right now, YouTube doesn't work too bad on Free," Felten said earlier this month. "Three weeks ago it was horrendous. A 30-second video, like an Angry Birds solution video, would take 12 minutes before the first frame moved. Right now, you get some lags but it's acceptable. I suspect whatever they're doing, they're constantly shifting it so it doesn't look like it's constantly horrible."

Instead of intentional discrimination against YouTube, Free may simply not have deployed enough infrastructure to handle all the video traffic its customers request. A few days after Felten spoke with Ars, the French telecom regulator ARCEP announced that it found no evidence of Free using discriminatory practices against YouTube, although it did say traffic is congested during peak hours.

The European market has far more competition for home Internet service than the US does. Felten said he is going to switch to a different ISP and hopes many other Free customers will do the same.

While European ISPs want payments from Google and Netflix, Felten said they should just focus on improving Internet connectivity. "Customers are already paying for it," he said. "You sell a service to the end user which is you can access the Internet. You make a huge margin on that. Why should they get extra revenue for something that's already being paid for?"

Paid peering contracts with fixed prices also don't make sense, he said, because IP transit is "constantly going down in price... At some point you'll be paying more for peering than you would for transit, which is absurd."

To illustrate the absurdity of ISP proposals to charge video providers, Felten noted that "when Orange purchased Dailymotion, that same week a white paper released by A.T. Kearney that Orange had financed basically said, 'this is how much online service providers should pay per gigabit of traffic that is delivered through our networks.' I applied that calculation to Dailymotion's traffic, and basically they would have had to pay twice their annual revenue in charges (nearly €42 million [roughly $55.5 million]). It's absurd. The whole economic model would break down if you were to apply that."

New, improved way of measuring traffic

Concerned about ISPs demanding payment from fellow network peers, Level 3 has come up with a new way of measuring peers' impact on each other's networks. Traditionally, traffic loads have been thought to be "in balance" if each peer sends about as much traffic to the other peer as it receives.

But the direction in which traffic flows has no impact on how much it costs to carry it, Level 3 argues. Because video streaming traffic dominates the Web, so-called "eyeball" networks (ISPs who deliver traffic over the last mile) can never be in balance with the networks that deliver video under the old measurement.

Instead, Level 3 wants to measure via "bit miles," the distance traffic is carried and the number of bits carried, regardless of which direction the traffic flows. So far, tw telecom and XO Communications have entered long-term, settlement-free peering agreements with Level 3 using bit mile measurements.

"This is a new idea. I think it's going to take a little bit to get more people on board," Forster said. "It needs to be simplified. Right now, both us and Level 3 are working out kinks in how this works."

Two networks could still end up with unbalanced traffic under the bit miles approach, but it would be easier to fix. "If you connect in 10, 15, 20, 30 places, it's easy to add connectivity points, it's easy over time to add and subtract those connectivity points to make sure the networks are in balance and you incur equal costs of transmission," Taylor said.

Level 3 objects to most demands that peers pay each other, even if the amounts aren't huge. The amount of money "for Level 3 is not material," Taylor said. "It's the principle of who decides because it's a non-competitive market. We have no choice. There is one party that sets the rate; there is no competitive pressure we can bring to bear to negotiate that rate or get a different way into the network."

This is where a lack of competition in the home Internet market comes into play. When a consumer's second option is slow or non-existent, the dominant Internet provider can maintain a monopoly even while delivering poor service. According to FCC data, 86 percent of American households have a choice of at least two wired Internet providers offering 3Mbps download speed and 768Kbps upload—but that's not even fast enough to qualify as broadband. Only 34 percent of US households have a choice of Internet providers offering at least 6Mbps down and 1.5Mbps up. The FCC defines broadband as 4Mbps down and 1Mbps up.

Netflix recommends 5Mbps for HD streams—and remember that a home Internet connection is often shared across multiple devices and people. Moreover, speeds promised by ISPs may guarantee a certain level of bandwidth between you and your ISP's network, but not between you and the rest of the Internet because of peering disputes and under-provisioned connections.

The argument that payments between network operators are required because streaming video throws traffic loads out of balance "is a complete canard," Crawford said. She explains:

When the commercial Internet was born, the telcos exchanged traffic with each other using handshake peering arrangements for which the paradigm was "ratio-based" payments. Because phone traffic was basically 1:1, or often 1:1, upstream and downstream, the big actors peered with one another for free. Today, we have enormously powerful eyeball network operators who are still taking advantage of this peering paradigm. And so they say, "my users are asking for more traffic than they're sending upstream." Using this rationale, the cable companies have moved from having to pay to get their users' traffic out onto the Internet (which is where they were when the Internet began) to demanding free peering, to, today, charging for interconnection. But the use of the network, and the architecture of the network, is now completely different. Cable is asymmetric; it was built for passive downloads and does not favor uploads. Also, users are doing a lot of downloading of content. It's a user-driven Internet. So of course traffic volumes are greater downstream than upstream. That's what cable is selling. It should want to sell more; it should want to build capacity and connect freely with transit networks; and it would if there was any competition. But there isn't. So cable companies can draw up the drawbridge and charge with impunity, or refuse to add capacity to service interconnecting providers unless they're paid (which is the same thing).

John Bergmayer, senior staff attorney at advocacy group Public Knowledge, said ISPs like Verizon can't credibly claim that their networks are overloaded by Netflix.

"We've seen all the cable advocates and the NCTA (National Cable & Telecommunications Association) back away from the claim that data caps are necessary to manage congestion," Bergmayer said. "Networks really aren't saturated on the last mile like they have been trying to claim for years. Particularly with Verizon—with FiOS, it has huge capacity. Verizon has said again and again that it could upgrade to gigabit; it just doesn't think that consumers really want it. They're always bragging about how great their fiber network is. They can't have their cake and eat it, too, and not really have real capacity problems on the one hand except when it comes to hammering out these deals with the Internet content companies on the other hand."

While European authorities have been demanding data on peering agreements from Internet companies, Bergmayer said the FCC should do the same. "Antitrust people should be aware of potential issues as well as the FCC, and the FCC should be taking steps to make sure it has the data it needs to determine whether it needs to be doing anything to begin with," Bergmayer said.

Levy prefers that government stay out of it. "Any time you get a regulator involved you're pretty much guaranteed for stuff to go downhill," he said, only half joking. "What they don't really understand is that sometimes this is OK. Basically, people need to argue. This is a relationship. No one can actually afford for this stuff not to work. Trust me."

But some IP networks argue with each more than others, he acknowledged. At Hurricane Electric, "We have an open peering policy, which means if someone asks us to peer we have a very limited requirement on them in order to accept them as a peer." Hurricane Electric has so far been conspicuously absent from any of the recent public disputes. "We're boring," Levy said. "By design, of course, because we're not like the people who are in these disputes."

Felten holds out hope that ISPs will become less likely to war with each other as they embrace over-the-top content—meaning the delivery of streaming video to any customer, even those who don't subscribe to the ISP's Internet service. ISPs that sell video outside their own networks will become just like Netflix and need other Internet companies to accept their traffic, the argument goes. They won't be able to throttle third-party traffic if they're worried about those same third parties throttling their own, Felten said. "If you do [over-the-top], you're not serving customers on your network only anymore, you're serving customers on everybody's network," Felten said. "If everybody else's network throttles your traffic, you have no business case. It can't work."

Although Verizon is getting into over-the-top video through its Redbox venture, that's not the strategy most are taking (and it hasn't stopped Verizon from warring with Cogent over Netflix).

Bergmayer is skeptical that Felten's theory will prove true in the US. "I've been hoping for a while that these guys would realize they could enter the over-the-top market," Bergmayer said. "But we've just seen things like TV Everywhere where it's essentially each ISP runs service just for its own customers and they sort of divide up the country like that."