With last month's court decision to kill net neutrality, Internet users are on the lookout for signs of preferential handling of network traffic. Doing the rounds today is a blog post by David Raphael.

Raphael's boss had noticed slow Internet performance at work. Raphael had noticed the same thing at home, and both he and his office were using Verizon's FiOS service. So he performed some testing. His work used Amazon's AWS for hosting, so he used AWS for his testing. He put a file on the S3 storage service and downloaded it at home. The transfer rate was a positively feeble 40 kilobytes per second.

He then connected to his office and downloaded the same file again. This time? 5,000 kilobytes per second.

Though this was somewhat more respectable, it was still a long way short of the 75 megabits per second that a speedtest reported his connection as. At around 4pm each day, Raphael's home Internet performance to AWS fell precipitously.

Update: I originally wrote 500KBps. The correct speed is 5,000KBps. 5,000KBps is very respectable.

Raphael contacted Verizon's customer service, chatting online to a representative to complain about the low speeds. After some amount of to-ing and fro-ing, along with the usual pointless rebooting of routers and patching of systems, Raphael asserted to the rep that something was limiting the bandwidth and asked if Verizon was limiting bandwidth to cloud providers. The customer service rep parroted his question back at him, apparently confirming his suspicions. Raphael asked if this was also behind poor Netflix performance; again the rep said yes.

The blog post was concluded with a pair of traceroutes that Raphael thought significant. They showed the paths taken to AWS from his home connection and his work connection.

Verizon's official line is that the customer service rep was misinformed, and that it treats all traffic equally.

First, a statement of the obvious. Customer service reps are pretty much the last people who would know about such a policy, let alone be able to inform customers of it. Add to this the usual confounding factors—non-native English speakers, a desire to "resolve" the customer's problem as quickly as possible (even if this means simply agreeing to what they say just to make them go away), and insufficient knowledge to actually go off-script—and the statements that Raphael cites as evidence don't really stand up.

As tempting as it may be when frustrated that your Internet connection isn't working as well as it should be, it's simply not fair to ask that kind of question to a low-level support person, much less use a chat transcript to accuse companies of widely unpopular behavior.

ISPs keep their networks opaque

But the bigger issue is this. It's not impossible that Verizon is giving preferential treatment to traffic. The technology exists to do such things, and Verizon was the company that challenged the net neutrality mandate, so it's not as if it isn't interested in giving some traffic better treatment than other traffic. It's just that as an end-user, it's really hard to tell.

We looked last year at the complexities of the relationships and deals between online video streaming and Internet service providers. Some traffic travels across paid links; other traffic goes across free, unpaid links. Sometimes companies get into disputes over what the fair rate to pay is and don't buy as much bandwidth as they need, leading to congestion and poor performance when connecting to certain services.

Even apart from issues with connectivity between providers, there can be bandwidth issues within a company's network. A street of 50 houses might boast 75 megabits per second of optic fibre Internet per household, but that doesn't mean that there's (75 × 50) = 3.75 gigabits per second of bandwidth between those houses and the ISP's core network. Those 50 connections might be aggregated into, say, a 1 gigabit link. If everyone in the street is hammering their Internet connection to download Linux ISOs from BitTorrent, they're simply not going to get the 75Mbps that their connections notionally provide. They'll be limited to an average of 20Mbps.

At one time it was common in the UK for ISPs to cite this provisioning ratio; typically it was 50:1 for consumer-oriented products and 20:1 for business ones. That is to say, for every 50 megabits per second of bandwidth sold to end users, only one megabit of bandwidth on the ISP's core network was available.

Raphael's description of the testing he performed (downloading remotely, rather than via VPN) doesn't rule out this much more mundane problem. The last hop, between Verizon's core network and Raphael's home (or office) could well be the source of the slowdown. If at 4pm each afternoon the kids are back from school, watching cartoons on Netflix and YouTube, watching streaming games on Twitch, and all the other bandwidth intensive things that we know home users tend to do in the afternoons and evenings, we'd expect to see poor performance on residential connections.

Business services, which tend to have lower contention and don't tend to have the same afternoon/evening surge, will be fine, even after 4pm.

To be able to authoritatively pinpoint the sources of slowness requires a lot more knowledge of ISP networks than ISPs are willing to disclose. As end users, we simply can't see which network ports on which switches and routers are running close to capacity. We don't know which links are massively oversubscribed. We also have no insight into who is buying bandwidth from who.

This leaves us little way of distinguishing between poor performance that's a "natural" result of congestion and poor performance that's an "artificial" result of traffic shaping and prioritizing. From an end-user perspective, Verizon throttling connections to AWS to 40kbps would look essentially identical to excessive network usage causing the link between Verizon and AWS to support only an average of 40kbps per connection. It would also look very similar to local congestion causing the same average throughput.

Verizon could, if it wanted to, figure out where the slowdown was coming from. Its network equipment will expose the information the company needs to determine which links are congested and which are not. Verizon may not be altogether interested in making and publicizing that information, however. ISPs seem to regard this kind of detail as a trade secret and are reluctant to disclose a level of overselling that'd make airlines blush.

With ISPs apparently now free to prioritize network traffic in whatever ways they see fit, it's certainly wise to be on the lookout for ISPs providing differentiated services in ways that put their customers at a disadvantage. But that vigilance needs to be tempered with an understanding of how ISPs actually work and much more careful analysis of network performance. Comments from first-tier support personnel aren't enough to draw any conclusions.