I am working behind a cloud service kintone.com as an infrastructure engineer. My recent work was the replacement of NAT inside our data center with a transparent SOCKS proxy.

In this post, I will describe our motivation for the replacement and how we can do it using Go effectively.

TL; DR

NAT has deficiencies in routing, connectivity, and access control.

We made a transparent SOCKS proxy transocks to replace NAT.

Using iptables is far better than LD_PRELOAD for transparent proxy.

We made our own SOCKS5 server usocksd too.

Deficiencies of NAT

NAT (Network Address Translation) is a technology to allow computers inside a private network, thus having private IP addresses only, to connect to the servers on Internet transparently as if the host has a global IP address.

The transparent nature is good since programs need nothing special to connect to the servers on Internet. However, NAT has some deficiencies:

Routing depends on network peripherals. Servers with tcp_tw_recyle=1 fails to receive NATed packets. No ability for access control based on domain names.

Routing depends on network peripherals.

Fairly large cloud infrastructures often have several network segments facing Internet. In such a network, routers use Policy-Based Routing (PBR) or BGP to route packets to the appropriate NAT server.

Since network configurations are often out of hands for software developers, this means software developers cannot choose which NAT server to be used dynamically in their programs.

Servers with tcp_tw_recyle=1 fails to receive NATed packets.

There is an infamous Linux kernel switch called net.ipv4.tcp_tw_recyle . Read this SO article for details.

Unfortunately, there are servers that enable this switch, and some of our customers need to connect to such servers. Disabling TCP timestamps in our data center could fix the problem, though.

No ability for access control based on domain names.

Since NAT servers can know only the destination IP addresses and port number, NAT cannot control accesses based on destination domain names.

There are many web servers sharing the same IP address with different domains. CDNs are typical case. Controlling access to such servers is difficult or impossible with NAT.

SOCKS proxy

Proxy is another technology to allow computers in a private network to connect to servers on Internet. As its name suggests, a proxy server makes connections to Internet servers on behalf of computers in a private network.

SOCKS is a simple proxy protocol mostly for tunneling TCP connections. With SOCKS5, clients can ask SOCKS server to create tunnels with a IPv4/IPv6 address or a DNS domain name. For this, SOCKS is not as transparent as NAT.

Sacrificing transparency, SOCKS can resolve NAT deficiencies as follows:

Client programs can route packets by choosing which SOCKS server to be used. tcp_tw_recycle=1 does no harm with SOCKS. SOCKS can control access based on domain names supplied by clients.

How transparent SOCKS proxy can be implemented

If we can force programs to use SOCKS without modification, NAT can be replaced with SOCKS unnoticeably.

One way to do that is to override system calls by LD_PRELOAD or by kernel modules. However, this is not a good idea.

First, LD_PRELOAD cannot be used for statically-linked executables or executables not depending on libc. For instance, LD_PRELOAD cannot override Go programs.

Second, it is quite difficult to guarantee that overridden programs work correctly. What and how system calls need to be overridden? Are connect and getpeername enough? For asynchronous connections, maybe select , poll , and epoll too? What about TCP Fast Open?

The better idea is to use the mechanisms exclusively for network packet manipulation; for Linux, it is iptables / ip6tables. By adding the following example rules, any locally-generated TCP packets outgoing to Internet will be redirected to the program listening on the port 1081.

*nat :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] :TRANSOCKS - [0:0] -A OUTPUT -p tcp -j TRANSOCKS -A TRANSOCKS -d 0.0.0.0/8 -j RETURN -A TRANSOCKS -d 10.0.0.0/8 -j RETURN -A TRANSOCKS -d 127.0.0.0/8 -j RETURN -A TRANSOCKS -d 169.254.0.0/16 -j RETURN -A TRANSOCKS -d 172.16.0.0/12 -j RETURN -A TRANSOCKS -d 192.168.0.0/16 -j RETURN -A TRANSOCKS -d 224.0.0.0/4 -j RETURN -A TRANSOCKS -d 240.0.0.0/4 -j RETURN -A TRANSOCKS -p tcp -j REDIRECT --to-ports 1081 COMMIT

REDIRECT target rewrites the destination address in the packets. The program then need to recover the original destination by a undocumented getsockopt option.

transocks: a SOCKS client for transparent proxy

The code below demonstrates how to recover the original destination address:

struct sockaddr_storage addr; socklen_t len = sizeof ( struct sockaddr_in); int level = IPPROTO_IP; int opt = SO_ORIGINAL_DST; if ( getsockopt(sock, level, opt, &addr, &len) == - 1 ) { }

Although there are some existing software that use this technique, they do not meet our requirements, or are not stable under heavy load. So we decided to make our own SOCKS client for transparent proxy.

The result is transocks, a transparent proxy library and executable written in Go. Not only SOCKS, transocks support HTTP proxy with CONNECT method as well. For more information, visit GitHub.

usocksd: a micro SOCKS server

We also created our own SOCKS server on top of github.com/armon/go-socks5 library.

One thing we want to implement is to load balancing between multiple global addresses while keep using the same global address for the same client. This was done by extending go-socks5 to support context, and used it.

Using the consistent global IP address is required for some servers. For example, some FTP servers require the same remote IP address for control and data connections in passive mode.

The result is usocksd, a micro but powerful SOCKS5 server.

Status and summary

Both transocks and usocksd are working smoothly in our production data center. They have replaced NAT already.

What we learned through this project are: