Network isolation is an vital part of any layered security approach. It’s used by all players, from small businesses to large organizations, in different capacities. More demanding networks require a high level of outbound filtering, to prevent data exfiltration and lessen the impact of breaches.

We at SensorFu chose to test AWS, Google Cloud, and Azure, since they’re some of the most popular providers. They also provide a good amount of networking features for larger organizations wanting to mimic their on-prem infrastructure in the cloud.

We created a simple network of two servers, and tried to break out from “vault” (below), using spoofing, tunneling, and network bounces. The 10.10.10.0 network was heavily firewalled, and all routes to other networks were deleted.

SSH tunneling lets us connect to vault via multihome

Common Features

All the providers we looked at have a similar high-level network architecture. There is a metadata server for each machine, which provides information for the OS like SSH keys and configuration data. The meta-server is usually also a DNS resolver. Also, subnets are not traditional L2 broadcast domains. They’re more like a group of addresses with common firewall and routing rules.

What’s interesting, is how each platform has its own way of implementing this “virtual subnet”. On Google Cloud Platform, each NIC uses a 32-bit netmask, and subnets are specified in the routing table. On Azure, there is a normal netmask but all ARP entries point to the network’s gateway with the MAC address 12:34:56:78:9a:bc.

What we found

The good

Because all three providers’ virtual networks are not traditional L2 broadcast domains, this means that all network traffic has to pass through some kind of L3 virtual router. This makes it possible to apply routing and firewall policies at the host-level, which is really nice for network security. There are also anti-spoofing measures in place, something which could not be accomplished (to the same degree) in traditional networks. This is a real-world, practical example of the benefits of micro-segmentation.

None of our basic spoofing or bounce tests were successful. We also had a look at the meta-server used with each platform, but didn’t find any simple way to exploit them. Aside from the glaringly obvious way below :).

The bad

Each cloud provider uses some kind of metadata server to provide information for guest VMs in their platform. This in itself isn’t bad, but the servers implementations did allow us to tunnel out of the network in Google Cloud and Azure.

Google Cloud’s meta-server (169.254.169.254) acts as a DNS server, and there’s no way to configure this functionality. The meta-server is also whitelisted in a way that no firewall or routing rules affect it, so you can’t block DNS traffic. This means that there is always the option to tunnel out of the network over DNS.

Azure is basically the same. One of Azure’s meta-servers (168.63.129.16) also resolves DNS queries, and is whitelisted so that there’s no way to block DNS tunneling out of the network. You can configure a different DNS server to be used at the OS-level, but the meta-server is always available if you just know its address. Azure has a second meta-server in use, located at 169.254.169.254.

AWS is the best of the three. You can disable the built-in DNS server and specify your own. In this case, your own DNS server is subject to the same firewall and routing rules as any other IP address, and you can achieve a reasonable level of network isolation.

Documentation for each platform was reasonably good — at least pertaining to firewalls and routing. A common issue, however, was documentation about the network’s meta-server. We couldn’t find any mentions of the hidden whitelisting that we encountered, and in the worst case (Azure), documentation amounted to a single blog post.

In summary

Aside from the issue with meta-server DNS resolution, all three cloud platforms held up ok. None of our other escapes were successful. But in their current state, only AWS is able to achieve true network isolation.

Coming from a traditional on-premises network, it can be a daunting task to create an isolated network in any of the platforms tested. Because they don’t work like traditional networks, with switches and routers, there are nuances in each platform that must be taken into account. Real-world testing is hugely important. It can shed light on things that you may have overlooked, or things that simply aren’t found in the documentation.