Linux network metrics: why you should use nstat instead of netstat

TL;DR: This article is about the differences between netstat and nstat regarding Linux system network metrics, and why nstat is superior to netstat (at least for this purpose.)

Updates

2016-04-12 - note about ss command

Network metrics with netstat

netstat can be found in the net-tools software collection. Depending on your linux Distribution, it may not be installed by default, like in Archlinux since 2011.

Below is the output of netstat –statistics on my system:

$ netstat --statistics Ip: 34151 total packets received 0 forwarded 0 incoming packets discarded 34108 incoming packets delivered 38436 requests sent out Icmp: 6 ICMP messages received 0 input ICMP message failed. ICMP input histogram: destination unreachable: 6 6 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 6 IcmpMsg: InType3: 6 OutType3: 6 Tcp: 365 active connections openings 0 passive connection openings 17 failed connection attempts 2 connection resets received 14 connections established 35389 segments received 39132 segments send out 83 segments retransmited 1 bad segments received. 156 resets sent Udp: 655 packets received 1 packets to unknown port received. 0 packet receive errors 662 packets sent 0 receive buffer errors 0 send buffer errors IgnoredMulti: 7 UdpLite: TcpExt: 137 TCP sockets finished time wait in fast timer 337 delayed acks sent Quick ack mode was activated 47 times 3 packets directly queued to recvmsg prequeue. 21584 packet headers predicted 7317 acknowledgments not containing data payload received 1128 predicted acknowledgments 2 congestion windows recovered without slow start after partial ack 19 other TCP timeouts TCPLossProbes: 20 TCPLossProbeRecovery: 2 47 DSACKs sent for old packets 8 DSACKs received 46 connections reset due to unexpected data 2 connections reset due to early user close 5 connections aborted due to timeout TCPDSACKIgnoredNoUndo: 6 TCPRcvCoalesce: 6121 TCPOFOQueue: 2421 TCPChallengeACK: 1 TCPSYNChallenge: 1 TCPSpuriousRtxHostQueues: 14 TCPAutoCorking: 1123 TCPSynRetrans: 26 TCPOrigDataSent: 16502 TCPHystartTrainDetect: 1 TCPHystartTrainCwnd: 16 TCPKeepAlive: 1292 IpExt: InMcastPkts: 27 OutMcastPkts: 2 InBcastPkts: 7 InOctets: 28620819 OutOctets: 22032907 InMcastOctets: 864 OutMcastOctets: 64 InBcastOctets: 1202 InNoECTPkts: 34992

Some sections are standardized and based on RFCs MIB:

To match netstat output with RFCs variables names, I did not find another way apart from reading netstat source code, especially statistics.c, where the relation are stored in arrays, extract:

{ "Forwarding" , N_( "Forwarding is %s" ), i_forward | I_STATIC}, { "ForwDatagrams" , N_( "%llu forwarded" ), number}, { "FragCreates" , N_( "%llu fragments created" ), opt_number}, { "FragFails" , N_( "%llu fragments failed" ), opt_number}, { "FragOKs" , N_( "%llu fragments received ok" ), opt_number}, { "InAddrErrors" , N_( "%llu with invalid addresses" ), opt_number}, { "InDelivers" , N_( "%llu incoming packets delivered" ), number},

The remaining sections (TcpExt, IpExt, …) are less rigid, as far as I know they have been added once someone has proven them useful.

net-tools is officially obsolete in favour of iproute2, quote from linuxfoundation.org

Please keep in mind that most net-tools programs are obsolete now

Metrics with nstat

nstat is provided by the iproute2 collection, which is usually also the name of the package in many Linux distributions. This package also provides the most well-known command ip

Extract of non-zero metrics:

$ nstat -a #kernel IpInReceives 69783 0.0 IpInDelivers 69469 0.0 IpOutRequests 68643 0.0 IcmpInMsgs 6 0.0 IcmpInDestUnreachs 6 0.0 IcmpOutMsgs 6 0.0 IcmpOutDestUnreachs 6 0.0 IcmpMsgInType3 6 0.0 IcmpMsgOutType3 6 0.0 TcpActiveOpens 1011 0.0 TcpAttemptFails 37 0.0 TcpEstabResets 27 0.0 TcpInSegs 71580 0.0 TcpOutSegs 71010 0.0 TcpRetransSegs 410 0.0 TcpInErrs 4 0.0 TcpOutRsts 369 0.0 UdpInDatagrams 1348 0.0 UdpNoPorts 1 0.0 UdpOutDatagrams 1366 0.0 UdpIgnoredMulti 47 0.0 Ip6InReceives 5236 0.0 Ip6InAddrErrors 421 0.0 Ip6InDelivers 4693 0.0 Ip6OutRequests 4913 0.0 Ip6InMcastPkts 780 0.0 Ip6OutMcastPkts 200 0.0 Ip6InOctets 3743259 0.0 Ip6OutOctets 710669 0.0 Ip6InMcastOctets 71232 0.0 Ip6OutMcastOctets 14384 0.0 Ip6InNoECTPkts 5725 0.0 Icmp6InMsgs 972 0.0 Icmp6InErrors 6 0.0 Icmp6OutMsgs 709 0.0 Icmp6InDestUnreachs 148 0.0 Icmp6InEchos 102 0.0 Icmp6InRouterAdvertisements 140 0.0 Icmp6InNeighborSolicits 521 0.0 Icmp6InNeighborAdvertisements 61 0.0 Icmp6OutDestUnreachs 148 0.0 Icmp6OutEchoReplies 102 0.0 Icmp6OutRouterSolicits 2 0.0 Icmp6OutNeighborSolicits 240 0.0 Icmp6OutNeighborAdvertisements 205 0.0 Icmp6OutMLDv2Reports 12 0.0 Icmp6InType1 148 0.0 Icmp6InType128 102 0.0 Icmp6InType134 140 0.0 Icmp6InType135 521 0.0 Icmp6InType136 61 0.0 Icmp6OutType1 148 0.0 Icmp6OutType129 102 0.0 Icmp6OutType133 2 0.0 Icmp6OutType135 240 0.0 Icmp6OutType136 205 0.0 Icmp6OutType143 12 0.0 Udp6InDatagrams 51 0.0 Udp6OutDatagrams 53 0.0 TcpExtTW 349 0.0 TcpExtDelayedACKs 811 0.0 TcpExtDelayedACKLost 137 0.0 TcpExtTCPPrequeued 14 0.0 TcpExtTCPHPHits 44384 0.0 TcpExtTCPPureAcks 10490 0.0 TcpExtTCPHPAcks 4460 0.0 TcpExtTCPLossUndo 5 0.0 TcpExtTCPSlowStartRetrans 4 0.0 TcpExtTCPTimeouts 113 0.0 TcpExtTCPLossProbes 46 0.0 TcpExtTCPLossProbeRecovery 2 0.0 TcpExtTCPDSACKOldSent 136 0.0 TcpExtTCPDSACKRecv 12 0.0 TcpExtTCPAbortOnData 101 0.0 TcpExtTCPAbortOnClose 21 0.0 TcpExtTCPAbortOnTimeout 23 0.0 TcpExtTCPDSACKIgnoredNoUndo 10 0.0 TcpExtTCPRcvCoalesce 15084 0.0 TcpExtTCPOFOQueue 5832 0.0 TcpExtTCPChallengeACK 4 0.0 TcpExtTCPSYNChallenge 4 0.0 TcpExtTCPSpuriousRtxHostQueues 224 0.0 TcpExtTCPAutoCorking 1242 0.0 TcpExtTCPSynRetrans 83 0.0 TcpExtTCPOrigDataSent 23610 0.0 TcpExtTCPHystartTrainDetect 3 0.0 TcpExtTCPHystartTrainCwnd 48 0.0 TcpExtTCPKeepAlive 2528 0.0 IpExtInMcastPkts 157 0.0 IpExtOutMcastPkts 2 0.0 IpExtInBcastPkts 47 0.0 IpExtInOctets 67200127 0.0 IpExtOutOctets 24997379 0.0 IpExtInMcastOctets 5024 0.0 IpExtOutMcastOctets 64 0.0 IpExtInBcastOctets 8252 0.0 IpExtInNoECTPkts 74074 0.0

In addition to absolute values of counters given by the -a option, nstat can also provide a delta since its last execution, to ease live system debugging:

$ nstat #kernel IpInReceives 1 0.0 IpInDelivers 1 0.0 IpOutRequests 1 0.0 TcpInSegs 1 0.0 TcpOutSegs 1 0.0 TcpExtTCPOrigDataSent 1 0.0 IpExtInOctets 54 0.0 IpExtOutOctets 58 0.0 IpExtInNoECTPkts 1 0.0 $

All values, even the zero ones with -- zero

$ nstat --zero #kernel IpInReceives 2 0.0 IpInHdrErrors 0 0.0 IpInAddrErrors 0 0.0 IpForwDatagrams 0 0.0 IpInUnknownProtos 0 0.0 IpInDiscards 0 0.0 ( ... )

Finally, metrics can be displayed in JSON format, to ease their processing by all your fancy tools:

$ nstat --json { "kernel" : { "TcpInSegs" :2, "TcpOutSegs" :4, "Ip6InReceives" :2, "Ip6InDelivers" :2, "Ip6OutRequests" :4, "Ip6InOctets" :776, "Ip6OutOctets" :770, "Ip6InNoECTPkts" :2, "TcpExtTCPHPHits" :1, "TcpExtTCPHPAcks" :1, "TcpExtTCPOrigDataSent" :2 }}

Differences

Output

netstat appears more user-friendly by describing some metrics with plain English, while nstat displays raw information.

This can be considered as an advantage to roughly identify the purpose of the metric, but also a drawback if you are interested in the RFC name of the variable, going through netstat source code is hence a mandatory step.

Output comparison of 3 metrics:

# nstat IpInReceives 74923 IpOutRequests 73128 IcmpInMsgs 6

# netstat Ip: 74923 total packets received 73128 requests sent out Icmp: 6 ICMP messages received

Parsing nstat output is also easier, even almost done thanks to the JSON output format option.

Metrics completeness

Both netstat and nstat read the metrics provided by the kernel through the /proc virtual filesystem:

$ strace -e open nstat 2>& 1 > /dev/null|grep /proc open ( "/proc/uptime" , O_RDONLY ) = 4 open ( "/proc/net/netstat" , O_RDONLY ) = 4 open ( "/proc/net/snmp6" , O_RDONLY ) = 4 open ( "/proc/net/snmp" , O_RDONLY ) = 4 $ strace -e open netstat -s 2>& 1 > /dev/null|grep /proc open ( "/proc/net/snmp" , O_RDONLY ) = 3 open ( "/proc/net/netstat" , O_RDONLY ) = 3

However, only nstat retrieves all the metrics provided by the kernel. Netstat seems to skip some of them, breakdown of metrics number per section:

Netstat Nstat Difference Ip 6 17 +11 Ip6 14 32 +18 Icmp 6 29 +23 Icmp6 25 46 +21 Tcp 10 10 0 Udp 7 8 +1 Udp6 4 8 +4 UdpLite 0 15 +15 UdpLite6 0 7 +7 TcpExt 48 116 +68 IpExt 11 17 +6

Why? Just because netstat maintains a static table of metrics entries, while nstat parses the whole /proc files. Since netstat is obsolete, new entries are not taken into account.

Note about ss command

ss is another utility to investigate sockets provided by iproute2 package, like nstat.

Unlike netstat and nstat, ss does not provide system-wide network statistics, but is more oriented towards analysis of established sockets connections from many families (raw, tcp, udp, Unix domain, dccp)

The only overall statistics option -- summary is limited to the opened sockets:

$ ss --summary Total: 433 ( kernel 0 ) TCP: 31 ( estab 17, closed 1, orphaned 0, synrecv 0, timewait 1/0 ) , ports 0 Transport Total IP IPv6 * 0 - - RAW 2 0 2 UDP 22 10 12 TCP 30 18 12 INET 54 28 26 FRAG 0 0 0

However ss is way more comprehensive when it comes to TCP connection internals, by reading /proc/net/tcp.

For instance, for an established TCP connection you can retrieve almost every number that characterize the state of an established TCP connection:

$ ss --info --tcp|tail -1 cubic wscale:7,7 rto:223.333 rtt:22.325/0.746 ato:40 mss:1428 cwnd:2 ssthresh:2 bytes_acked:12935 bytes_received:19093 segs_out:113 segs_in:57 send 1.0Mbps lastsnd:1226 lastrcv:1203 lastack:1203 pacing_rate 1.2Mbps retrans:0/4 rcv_rtt:27.083 rcv_space:28800

Every field will be explained in another blog post, but here you can recognize the congestion control algorithm cubic, various TCP timers rto, rtt, …

Another super feature of ss is its filters based on the states of a connection, more handy than grepping netstat output:

STATE-FILTER STATE-FILTER allows to construct arbitrary set of states to match. Its syntax is sequence of keywords state and exclude followed by identi‐ fier of state.

In addition to all the TCP states, others grouping keywords are possible:

Available identifiers are: All standard TCP states: established, syn-sent, syn-recv, fin-wait-1, fin-wait-2, time-wait, closed, close-wait, last-ack, listen and closing. all - for all the states connected - all the states except for listen and closed synchronized - all the connected states except for syn-sent bucket - states, which are maintained as minisockets, i.e. time-wait and syn-recv big - opposite to bucket

The manpage provides useful examples:

ss -o state established '( dport = :ssh or sport = :ssh )' Display all established ssh connections. ss -x src /tmp/.X11-unix/* Find all local processes connected to X server. ss -o state fin-wait-1 '( sport = :http or sport = :https )' dst 193.233.7/24 List all the tcp sockets in state FIN-WAIT-1 for our apache to network 193.233.7/24 and look at their timers.

Try that with netstat :)

Summary

nstat offers all the linux network metrics provided by the kernel, but without any knowledge of the aforementioned RFCs their names might look more or less cryptic.

offers all the linux network metrics provided by the kernel, but without any knowledge of the aforementioned RFCs their names might look more or less cryptic. netstat is obsolete and does not provide all the available metrics, but many are described with plain English, which is easier to understand when looking for simple metrics.

is obsolete and does not provide all the available metrics, but many are described with plain English, which is easier to understand when looking for simple metrics. If you want to extract every possible information on your established connections, ss is what you are looking for.

Plan

I plan to write another article to describe every metric provided by nstat, if you are interested please leave a comment.

Please enable JavaScript to view the comments powered by Disqus.