In Part 1 of this series, we covered DOCSIS 2.0. Now, DOCSIS 2.0 is no slouch, but with the hot breath of fiber-to-the-home, WiMax, and LTE on its neck, it was clear to cable providers early on that DOCSIS needed to up its speed. DOCSIS 2.0 was able to improve upload speeds over DOCSIS 1.0 using denser QAM constellations and CDMA, but unfortunately, there’s a limit to simply cramming even more bits into a single QAM symbol. The problem is noise.

Suppose that we have a channel where our signal and the noise are exactly the same strength, and we can still distinguish the two symbols -1 and +1 to a usable degree. (We’ll let one little detail slide: when the signal is +1 and the noise -1, that would be indistinguishable from a signal of -1 and noise of +1.) If we now increase our signal strength by a factor three, our two symbols are -3 and +3, making our signal stand out from the noise much more. We can then make use of this increase in the signal-to-noise ratio by using a larger number of distinct symbols. Previously, the distance between our two symbols was two. If we keep that distance, we can now have four symbols: -3, -1, +1, +3. Four symbols gives us two bits per symbol, so our three-fold SNR gain gave us a doubling of our bitrate. The relationship between the two is expressed in the Shannon-Hartley theorem:

C is the highest possible channel capacity in bits per second, B the channel bandwidth in Hertz, S the signal, and N the noise. Note that the signal and noise are not expressed in decibels, but as a linear power ratio, or Volt squared.

This little foray into information theory makes it clear that we can’t simply increase the number of distinct symbols to reach higher and higher bitrates, but we’re limited by the available signal-to-noise ratio. The noise floor is dictated by the thermal noise that all electronics produce and the interference from outside the cable system, as well as the other signals carried by the cable. The maximum signal strength is limited by the cable’s properties and the limits on the interference that we can reasonably impose on others—don’t forget, we’re using frequencies inside the coax cable that are used for other purposes over the ether, and in practice no cable is shielded perfectly.

DOCSIS 3.0 gets around these limits using brute force: it bundles multiple channels. By using multiple channels at the same time for the same data stream, the data stream can be bigger than the bandwidth that a single channel allows. By default, in the downstream direction, the CMTS will send individual packets over different channels, but each packet is labeled with a sequence number so if packets arrive out of order, they can be put back into their original order before they’re given to the user. The Internet Protocol (IP) explicitly allows for out-of-order arrival of packets—but if this happens routinely, it does slow down transfers. Alternatively, the CMTS can use different channels for different types of communication, where each channel has its own interleave level that is appropriate for the type of traffic flowing over that channel.

Interleaving is basically slowing down the transmission of packets so that when there is a spike in interference, for instance, from the power grid as an appliance turns on, fewer bits of the packet are lost. This way, the forward error correction (FEC) has a better chance of repairing the damage and packet loss is avoided. By interleaving multiple packets, the aggregate bandwidth remains high, but the latency goes up. The highest interleave level of 128:1 adds a delay of 4 milliseconds to the downstream path with QAM64 and 2.8ms with QAM256. The typical interleave level is 32:4, which adds 1 or 0.7 ms of latency, respectively.

A high level of interleaving is appropriate for video streaming, where the additional latency is inconsequential but lost packets immediately impact quality. For VoIP, losses are also bad, but latency is much worse, so a lower level of interleave is better. The CMTS may thus be configured to send video packets over a channel with a high level of interleave and VoIP packets over a channel with a low level of interleave.

In the upstream direction, cable modems make requests for bandwidth when they have data to transmit. The CMTS then “grants” minislots to the CM for that transmission on any combination of bonded upstream channels, depending on available bandwidth. Unlike downstream channels, where there's just QAM64 and QAM256, there are many different ways an upstream channel can be configured, and the interference in the upstream direction also varies a lot more for different channels, as strong sources of interference tend to be lower in frequency. It's even possible for certain (upstream or downstream) channels to become completely unavailable. In that case, communication continues over the remaining channels in the bonded group until the lost channel(s) can be brought back into service.

The DOCSIS standard doesn’t specify a specific number of channels that must be supported, but it does mandate a minimum capability of four downstream channels. That adds up to 172Mbps of raw bandwidth, before taking various types of overhead into account.

Sharing the bandwidth: statistical multiplexing

So far, we’ve mostly talked about the maximum number of bits per second that a DOCSIS system can carry—more than 100 megabits per second, which is nothing to sneeze at. However, that bandwidth is shared among multiple users.

Interestingly, one of the main reasons the ARPANET, the illustrious forebearer of the Internet, was created was to enjoy the benefits of statistical multiplexing.

WiFi and EDGE 2G mobile work the same way: one user gets to use the capacity at a time, so the speed each user experiences goes down as the number of active users goes up. Some other technologies don’t have this limitation. With today’s Ethernet, where everyone has their own cable to a switch, everyone gets to use the full Ethernet speed regardless of what other users do. ADSL works the same way: every subscriber has their own circuit to the central office. The mobile 3G standards are based on CDMA, which also manages to give different users each the maximum speed they’re capable of at the same time—within limits.

So maybe 1.5Mbps ADSL is a lot slower than cable, but at least you always get to use that 1.5Mbps, right? Unfortunately, no. Suppose the local phone company has 650 ADSL customers in a neighborhood of a medium-size city in one of the flyover states. That’s 650 x 1.5Mbps = 1Gbps. But does the ISP in question reserve 1Gbps of capacity from that neighborhood to one of the big cities, where it connects to the rest of the Internet? Of course not, because 650 users never all use their connection to capacity at the same time and thus never generate 1Gbps of traffic. It’s the same way with all other infrastructure: it’s designed for average usage patterns with some headroom, not for the possibility that every user needs the full capacity at the same time. If everyone picks up their landline phone at the same time, many won’t get a dial tone. Same thing with cell phones, electricity, water, and of course the roads. If every car owner pulls out of their driveway at the same time, good luck reaching the speed limit.

Interestingly, one of the main reasons the ARPANET, the illustrious forebearer of the Internet, was created was to enjoy the benefits of statistical multiplexing. Before the ARPANET, communication networks, such as the phone network, were circuit switched—as the user dials a number, a path is created through the network. That path is then dedicated to that user for the duration of the call in both directions. This is a tremendous waste of resources, as people typically take turns talking, and sometimes there are gaps in the conversation where nobody speaks.

Computer communication is the same way. For instance, when surfing the Web, the browser will send requests when the user clicks a link, and then one or more remote servers send back data. Then the communication link is idle for some time while the user looks at the page. But the Internet is packet-switched rather than circuit-switched, so the unused capacity doesn’t have to go to waste. Packet switching is similar to traffic on the road—there is no need to reserve a path through the city when driving from point A to point B; cars simply enter the road if there is room for them. This approach works even better with packets, because congestion control algorithms make sure new packets are only injected into the network at a rate that the network can accommodate—if the network is overloaded, communication sessions simply slow down.

Through statistical multiplexing, it’s entirely possible to have more than a thousand users share 100Mbps worth of capacity. Many users won’t even be using their network connection at any given time. Of the ones that do, most will generate intermittent traffic (such as in our Web browsing example) or use other applications that generate limited amounts of traffic (such as listening to streaming audio). Only a few users will be engaged in bulk data transfers (uploads or downloads) at any given time. A good deal of the time, downloaders won’t see 100Mbps transfers—that would be a little over 11 megabytes per second. But just in case nobody else is using the network, DOCSIS and other shared networking technologies make it possible for one user to use almost all the bandwidth. That is, as long as cable operators don’t limit their user’s speeds to something much lower to avoid creating unrealistic expectations.

So if you get 107Mbps cable service, which is now available in several places in the US, that probably means that “under the hood” four or probably more channels are bonded for a raw speed much higher than 107Mbps.