This post attempts to describe how Internet traffic with a fairly standard ISP works in New Zealand, from the point of a fictional ISP called “Generic ISP NZ Limited”. I can’t stress enough that this isn’t modelled on any actual ISP in New Zealand!The first thing to note is if you work in an ISP in New Zealand is that I've taken a few creative liberties with a few things. Feel free to point these out in the comments and I’ll tell you if I meant to do that, or I’ll correct the article as required. This is a fairly generic model I'm discussing here, I’m also talking mostly about ADSL services here too, though most things are relevant regardless of access method.I'm also not talking much about the portion of the network from your house to the DSLAM. There are other more knowledgeable people than me that can describe how it all works in detail.Ok, so! We’re based at 101 Fictional Street in Central Auckland.Let's look at our core network. This is where all our traffic has come in from our ADSL/Cable/3G/Dedicated links, across our backhaul connections and now has to route to the final destination. We now have a few different places we could send the traffic.Here's Generic ISP NZ’s Network Diagram (click it for bigger):Generic ISP NZ has 3 distinct different traffic paths out of the network. The following section discusses these paths, where they go and the costs associated with them.The first sort of traffic link we have to buy is International traffic. Our users would be pretty unhappy if they couldn't load Facebook, Twitter and JustinBeiber.com (Why God? Why?)All of our user’s traffic that has to “leave the country” travels over this link.This links is very expensive, so we only buy 100Mb/s of capacity. This is despite the fact we actually have a Gigabit Ethernet connection to our upstream, they rate limit us to 100Mb/s because that’s all we’re paying for.During peak hours, this link tends to run flat out and some of our users see a little bit of packet loss. The loss isn’t bad however and it’s only for 2-3 hours per night, so we’re sticking with it for the moment. We can’t afford to buy 130Mb/s like we’d need to keep the link uncongested and keep our subscriber prices the same currently.The next traffic link we have is our Peering traffic. This is traffic to other ISPs in New Zealand towards sites like TradeMe, local FTP mirrors etc. This is much cheaper than International traffic because most of the traffic is swapped over “free” peering sessions between the ISPs.Peering is great because it benefits both parties; therefore neither party changes the other. If we send all our traffic to Bob ISP Ltd for free, then they will send us all their traffic for free. Because of this, the more traffic we can get to go over our peering links the better for us, because it’s taking traffic that’d otherwise go over other links.The major peering exchange in New Zealand is called APE , the Auckland Peering Exchange. ISPs connect into APE and are able to freely exchange traffic.WIX, the Wellington Internet eXchange is another local peering point for ISPs that are in Wellington. There are other peering exchanges in New Zealand.We can send/receive as much traffic over our Gigabit Ethernet link to APE as we want; all we have to pay for is the physical connectivity to get to APE because it’s hosted in the SkyTower.Another small but interesting thing to note is that when you peer at APE, you don’t actually have to peer with every single ISP at APE. All you do is peer with the CityLink Hosted Route-Servers and they teach everyone everyone else’s routes. CityLink do a really great job of hosting this service, cheers CityLink!There are two major ISPs in New Zealand that don't peer with anyone else, TelstraClear and Telecom New Zealand. They do peer with each other though. What this means for us is that we have to purchase a connection from either TelstraClear or Telecom in order to get access to the resources and traffic within their networks. This cost costs us a fair amount of money, but it’s not as expensive like our International traffic is. So we can afford to keep this uncongested at all times, but we operate it with a tight margin.If we didn’t buy this link, we could still get to TelstraClear and Telecom, but we’d have to send traffic over our International link to get there. That’s an expensive thing for us to do and our smart users would complain about it, so we’re forced into buying a link from TelstraClear or Telecom.A lot of people would say this is a very dirty tactic from TelstraClear and Telecom, forcing us smaller players into buying a connection from them. I’ll leave that discussion for another time.The other side to this argument is that it's not fair for Telecom or TelstraClear to be offering the large amount of content within their network to a small ISP, when that small ISP has very little to offer in return. Or the fact that Telecom/TelstraClear might have to backhaul traffic up from ChristChurch for example to hand it over at peering point in Auckland, if the smaller ISP peers only in Auckland.Peering is a complex beast and there's the technical advantages and the commerical advantages. I didn't present the commercial aspect, but it's important to note that it's there. Personally I think peering is a good thing, but a beancounter would see it otherwise!Some other ISPs in the market might not peer directly at APE like we do, instead they might just buy a single Domestic connection from TelstraClear or Telecom which contains all of TelstraClear’s, Telecom’s and APE’s routes in it. Doing this means they only have to manage two links, an International one and a Domestic one.We find it easier to have 3 links though we have a good idea of where our traffic is flowing.Because International bandwidth is so much more expensive for us, our user's HTTP traffic over this link is subject to caching. When we do it properly, caching is not noticeable to our users and in fact is beneficial to them as content served from the cache is much faster than content served from overseas. It also saves us from having to transmit/receive data over our International link, freeing it up for other user traffic.So caching is actually a good thing, despite what a lot of people think. The reason that most people have the misconception that caching is bad is because when it's not working properly, stale or even just wrong content can be served. This is pretty frustrating for our end users . A lot of sites also don't provide proper information to our caches, causing us even more problems with stale content. We usually have to bypass these sites as we learn about them, so that we don’t attempt to cache any of their content.A lot of people use the SpeedTest.net site as a measure of their ISPs speed and hopefully the above explanation will now help explain why it alone isn’t a great measure. The reason it’s not is because SpeedTest traffic travels over the Peering (or Domestic) connection that your ISP has. As discussed, these links are easy to keep upgraded and uncongested. International links, where most of your day-to-day browsing and downloading will come from, are expensive and ISPs like ourselves experience congestion over it during peak hours.So it’s possible for Generic NZ Ltd customers to a good SpeedTest result, but for them to still have a sub-optimal browsing experience during peak hours.SpeedTest is a good indicator of your access method’s (ADSL/Mobile etc) maximum performance capability, not so much overall Internet experience you’ll actually get.Edit: It's also good to show you if your ISP is experiencing congestion over their backhaul links during peak hours (also quite common) as you'll get poor SpeedTest results showing this. [Thanks kyhwana2]I hope this gives a good overview of how an ISP in New Zealand routes traffic around.I am happy to answer any technical type questions or even post another blog with more detail if you're interested in this.Cheers,MuppetPS: I'm crap with HTML/CSS so apologies for the bland look and horrid formatting.Other related posts: