Faced with a 4-hour layover today at JetBlue’s Terminal 5 in New York’s John F. Kennedy airport, I planned to catch up on email, work on my to-do list, and generally make the most of the afternoon. JetBlue proudly advertises its free WiFi services on large signs thoroughout T5, and previous trips to/through T5 have allowed me to experience the excellent quality of Internet service available. I was ready to roll.

Or so I thought.

Upon arrival at JFK, I ordered a chicken Stromboli for lunch, found a table, pulled out my laptop, and … waited in vain as minute after minute slipped by without a successful connection reported by the wireless adapter.

Frustrated by the delay, I attempted to connect with my Moto X and with the second laptop I had in my bag – to no avail. After looking around, I realized that no one else in the terminal had laptops open – only phones, tablets, and books were seen. This was unusual – on previous stopovers at T5, there have always been many other travelers engaged with their laptops to pass the time. Something must be wrong with the wireless network.

Awesome.

With my carefully laid plans foiled, I decided to make the most of the time with the tools I had on hand. As a wireless network designer, I always have a handful of tools available any time I travel. Using a mix of software and hardware from my carry-on, I conducted an audit of the T5 wireless network. The results were surprising.

Below follows a brief review of my findings and some recommendations to resolve the issues identified. This review is shorter and shallower than the audits I normally provide my clients, but enough detail is included to convey the problem and some options resolution. Feel free to pass this along if you know any network engineers at JetBlue or the JFK airport — my hope is that the JetBlue / JFK network engineering team will have already come to these conclusions and is taking the necessary steps to rectify these problems so future travelers will not be similarly inconvenienced. I hope to experience blazing fast WiFi the next time I pass through T5.

Data Collection

During this audit, data was collected near Gate 1, Gate 4, Gate 6 Gate 7, and the main Terminal 5 Food Court. This provided a complete sample along the main corridor of T5, so the findings identified are systemic trends rather than isolated incidents.

For readers less familiar with wireless networks, here is a high-level summary of what happens when a user connects to a wireless network:

User selects a network name and clicks “connect” The computer maps the user’s selection to a target access point The computer attempts to send a radio signal to the target access point After the radio signal is acknowledged, the computer asks the access point for an IP address that it can use to talk to other computers/servers After receiving this request, a server on the local network allocates an available IP address for the user’s computer, records which device now has that address, and responds to the user’s computer with this address After the user’s computer receives the address information from the server, it configures its network connection using this information, enabling the user’s computer to browse the web

In each case during this survey, client devices were completely unable to connect to the network, receiving no response to DHCP broadcast requests. This means that client devices were unable to receive IP addresses or other important network information, making them totally unable to connect to the Internet or other networked resources.

Findings

The Open Standards Initiative established a standard 7-layer model for describing the relationship and interactions between different components in networks. For the purposes of this discussion, we’ll focus on layers 1–3 (physical, data link, network). The physical layer includes the actual physical medium over which communication occurs (such as Cat5e Ethernet cable or the 2.4GHz wireless band). The data link and network layers include the basic addressing details necessary to construct a connection (MAC address, IP address, etc).

Layer 1: Physical

When troubleshooting a network, it’s best to start from the bottom and work upward since a failure at a lower layer will always result in failures at higher layers (but not vice versa). When troubleshooting wireless networks, this means looking at the networks (SSIDs) available in the area, confirming the number (and type) of access points in play, and investigating the signals and bands used to determine potential cases of interference.

In T5, Cisco access points (non-Meraki, likely controller-based) are configured with 10 networks (SSIDs) each; the exact models are currently unknown — I was unable to visually locate them (they are most likely hidden in walls or above the ceilings). These access points support dual-band connectivity (both 2.4GHz and 5GHz signals), which is a positive sign — 5GHz support is key to delivering connectivity in high density scenarios like airport terminals.

Of the 10 networks broadcast by the T5 access points, only 4 were delivered on both the 2.5GHz and 5GHz bands, and only one was made available for public use.

The six 2.4GHz networks available included:

JetBlue Hotspot — no encryption (Open)

— no encryption (Open) jb_corp — using WPA2-Enterprise encryption

— using WPA2-Enterprise encryption jb_mobile — using WPA2-Enterprise encryption

— using WPA2-Enterprise encryption jb_trax — using WPA2-Enterprise encryption

— using WPA2-Enterprise encryption <name hidden #1> — using WPA2-Enterprise encryption

using WPA2-Enterprise encryption <name hidden #2> — using WPA2-Enterprise encryption

The four 5GHz networks available included:

jb_corp — using WPA2-Enterprise encryption

— using WPA2-Enterprise encryption jb_mobile — using WPA2-Enterprise encryption

— using WPA2-Enterprise encryption jb_trax — using WPA2-Enterprise encryption

— using WPA2-Enterprise encryption <name hidden #3> — using WPA2-Enterprise encryption

At each data collection point, the total number of networks visible varied a bit depending on proximity to each access point. Due to signal attenuation, not all 10 networks would be visible from an AP that was far away from a test location. Additionally, since the network was almost universally unusable, many travelers chose to turn on the WiFi hotspot feature on their cell phones, adding noise and interference (test locations typically showed ~50 networks broadcast by the nearest ~7 airport infrastructure APs, along with another 10+ networks broadcast by nearby cell phones).

During assessment, it became clear that the airport’s infrastructure APs made use of non-overlapping channels in an effort to avoid destructive adjacent-channel interference. However, the power levels and/or spacing of the access points were inappropriately selected in the context of the 2.4GHz band, leading to overlapping co-channel interference between access points (multiple APs on the same channel). This is a problem because client devices and APs can be confused about which device to respond to, and will be forced to “wait in line” twice as often — WiFi is naturally a shared medium, meaning that the total available bandwidth is divided among the number of connected devices.

Additionally, for some reason the public JetBlue Hotspot signal was not available on the 5GHz band. This is a major drawback because it prevents access points from properly load balancing by spreading users across all of its radios (instead forcing one or two radios on an AP to handle the same load as 4+ radios, significantly reducing the available bandwidth and increasing latency). This is also a drawback because many more non-overlapping channels are available on 5GHz, meaning that more 5GHz APs can be placed closer to each other to provide better coverage in high density deployment scenarios.

Finally, all 10 of the networks provided by all access points appeared to only support a maximum of 54mbps 802.11a/b/g connectivity. This limits the capacity of the network, leading to a decreased user experience.

Layers 2 & 3: Data Link & Network

After the number of nearby users decreased somewhat (~90 minutes after this audit began), a successful connection on the laptop was established, albeit still extremely sluggish, which allowed additional Layer 2/3 testing to be conducted.

A network scan showed 547 clients currently connected with a very large address allocation – a /20 mask allowed up to 4094 clients to be connected on the same subnet. Additionally, it appeared that all access points throughout T5 were assigning users to this one subnet, regardless of which access point they used. This is a problem because broadcast messages(such as DHCP, ARP, Multicast, etc) will be echoed out to all users; these broadcasts are used frequently, such as when a new client connects to the network, so a huge amount of bandwidth will be dedicated to mirroring these broadcasts rather than handling Facebook and email messages (significantly increased latency).

Furthermore, speed test outputs (sample below) summed up the dismal state of the connection – hovering around 70ms latency with ~0.5mbps bandwidth (“slower than 93% of U.S.”).

Recommendations

There is no one-size-fits all solution for network design, but especially in the context of high density wireless network design. Airports are a high density scenario — at approximately 2:1 (devices:people), airports are somewhat higher density than lecture halls (approx. 1:1), but less dense than hackathons (approx. 4:1). The recommendations for this scenario address the issues above in the context of the following goals:

maximize bandwidth available

minimize client latency

maximize number of client devices connected

Below are a few short recommendations that will resolve the problems described above. This list is not inclusive, and should be considered carefully in the context of other variables specific to the network infrastructure — such as location and type of access points, building materials, density of users, etc.

Enable 5GHz support for public WiFi on all existing access points

Deploy additional APs strategically to increase 5GHz coverage density

Disable 2.4GHz radios on 25% — 50% of deployed APs

Enable load balancing and/or band steering

Enable device isolation and/or small VLANs

Disable 802.11b device support

Enable 802.11n support

Take-Away

Wireless network design and implementation is difficult. As discussed above, it looks like the configuration of the T5 network has changed over time, resulting in massively user-impacting issues. Without deeper investigation and collaboration by the JetBlue / T5 network engineering team, it’s hard to suggest the motivation for these changes. The bottom line is that network design/engineering is an art form, and that care must be taken when implementing changes. It’s hard to get right, but it’s not impossible, no matter the scale of the challenge — experience is key to success.