24 iostat, vmstat and mpstat Examples for Linux Performance Monitoring

This article provides a total of 24 examples on iostat, vmstat, and mpstat commands.

iostat reports CPU, disk I/O, and NFS statistics.

vmstat reports virtual memory statistics.

mpstat reports processors statictics.

This article is part of our ongoing Linux performance monitoring series.

Please note that iostat and vmstat are part of the sar utility. You should install sysstat package as explained in our sar (sysstat) article to get iostat and vmstat working.



IOSTAT EXAMPLES

1. iostat – Basic example

Iostat without any argument displays information about the CPU usage, and I/O statistics about all the partitions on the system as shown below.

$ iostat Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 avg-cpu: %user %nice %system %iowait %steal %idle 5.68 0.00 0.52 2.03 0.00 91.76 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 194.72 1096.66 1598.70 2719068704 3963827344 sda1 178.20 773.45 1329.09 1917686794 3295354888 sda2 16.51 323.19 269.61 801326686 668472456 sdb 371.31 945.97 1073.33 2345452365 2661206408 sdb1 371.31 945.95 1073.33 2345396901 2661206408 sdc 408.03 207.05 972.42 513364213 2411023092 sdc1 408.03 207.03 972.42 513308749 2411023092

2. iostat – Display only cpu statistics

iostat option -c, displays only the CPU usage statistics as shown below.

$ iostat -c Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 avg-cpu: %user %nice %system %iowait %steal %idle 5.68 0.00 0.52 2.03 0.00 91.76

3. iostat – Display only disk I/O statistics

iostat option -d, displays only the disk I/O statistics as shown below.

$ iostat -d Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 194.71 1096.61 1598.63 2719068720 3963827704 sda1 178.20 773.41 1329.03 1917686810 3295355248 sda2 16.51 323.18 269.60 801326686 668472456 sdb 371.29 945.93 1073.28 2345452365 2661209192 sdb1 371.29 945.91 1073.28 2345396901 2661209192 sdc 408.01 207.04 972.38 513364213 2411024484 sdc1 408.01 207.02 972.38 513308749 2411024484

4. iostat – Display only network statistics

iostat option -n, displays only the device and NFS statistics as shown below.

$ iostat -n Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 avg-cpu: %user %nice %sys %iowait %idle 4.33 0.01 1.16 0.31 94.19 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 2.83 0.35 5.39 29817402 457360056 sda1 3.32 50.18 4.57 4259963994 387641400 sda2 0.20 0.76 0.82 64685128 69718576 sdb 6.59 15.53 42.98 1318931178 3649084113 sdb1 11.80 15.53 42.98 1318713382 3649012985 Device: rBlk_nor/s wBlk_nor/s rBlk_dir/s wBlk_dir/s rBlk_svr/s wBlk_svr/s 192.168.1.4:/home/data 90.67 0.00 0.00 0.00 5.33 0.00 192.168.1.4:/backup 8.74 0.00 0.00 0.00 8.74 0.00 192.168.1.8:/media 0.02 0.00 0.00 0.00 0.01 0.00

5. iostat – Display I/O data in MB/second

By default iostat, displays the device I/O statistics in Blocks. To change it to MB, use -m as shown below.

$ iostat -m Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 avg-cpu: %user %nice %system %iowait %steal %idle 5.68 0.00 0.52 2.03 0.00 91.76 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 194.70 0.54 0.78 1327670 1935463 sda1 178.19 0.38 0.65 936370 1609060 sda2 16.51 0.16 0.13 391272 326402 sdb 371.27 0.46 0.52 1145240 1299425 sdb1 371.27 0.46 0.52 1145213 1299425 sdc 407.99 0.10 0.47 250666 1177259 sdc1 407.99 0.10 0.47 250639 1177259

6. iostat – Display I/O statistics only for a device

By default iostat displays I/O data for all the disks available in the system. To view statistics for a specific device (For example, /dev/sda), use the option -p as shown below.

$ iostat -p sda Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 avg-cpu: %user %nice %system %iowait %steal %idle 5.68 0.00 0.52 2.03 0.00 91.76 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 194.69 1096.51 1598.48 2719069928 3963829584 sda2 336.38 27.17 54.00 67365064 133905080 sda1 821.89 0.69 243.53 1720833 603892838

7. iostat – Display timestamp information

By default iostat displays only the current date. To display the current time, use the option -t as shown below.

$ iostat -t Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 Time: 08:57:52 AM avg-cpu: %user %nice %system %iowait %steal %idle 5.68 0.00 0.52 2.03 0.00 91.76 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 194.69 1096.49 1598.45 2719070384 3963829704 sda1 178.18 773.32 1328.88 1917688474 3295357248 sda2 16.51 323.14 269.57 801326686 668472456 sdb 371.25 945.82 1073.16 2345452741 2661228872 sdb1 371.25 945.80 1073.16 2345397277 2661228872 sdc 407.97 207.02 972.27 513364233 2411030200 sdc1 407.97 207.00 972.27 513308769 2411030200

8. iostat – Display Extended status

Use option -x, which will displays extended disk I/O statistics information as shown below.

$ iostat -x Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 avg-cpu: %user %nice %system %iowait %steal %idle 5.68 0.00 0.52 2.03 0.00 91.76 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 27.86 63.53 61.77 132.91 1096.46 1598.40 13.84 0.21 1.06 2.28 44.45 sda1 0.69 33.22 48.54 129.63 773.30 1328.84 11.80 1.39 7.82 2.28 40.57 sda2 27.16 30.32 13.23 3.28 323.13 269.56 35.90 0.55 32.96 3.44 5.68 sdb 39.15 215.16 202.20 169.04 945.80 1073.13 5.44 1.05 2.78 1.64 60.91 sdb1 39.15 215.16 202.20 169.04 945.77 1073.13 5.44 1.05 2.78 1.64 60.91 sdc 8.90 3.63 356.56 51.40 207.01 972.24 2.89 1.04 2.56 1.55 63.30 sdc1 8.90 3.63 356.55 51.40 206.99 972.24 2.89 1.04 2.56 1.55 63.30

To display extended information for a specific partition (For example, /dev/sda1), do the following.

$ iostat -x sda1 Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 avg-cpu: %user %nice %system %iowait %steal %idle 5.68 0.00 0.52 2.03 0.00 91.76 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda1 0.69 33.21 48.54 129.62 773.23 1328.76 11.80 1.39 7.82 2.28 40.56

9. iostat – Execute Every x seconds (for y number of times)

To execute iostat every 2 seconds (until you press Ctl-C), do the following.

$ iostat 2 Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 avg-cpu: %user %nice %system %iowait %steal %idle 5.68 0.00 0.52 2.03 0.00 91.76 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 194.67 1096.39 1598.33 2719070584 3963891256 sda1 178.16 773.26 1328.79 1917688482 3295418672 sda2 16.51 323.11 269.54 801326878 668472584 sdb 371.22 945.74 1073.08 2345454041 2661251200 sdb1 371.22 945.72 1073.08 2345398577 2661251200 sdc 407.93 207.00 972.19 513366813 2411036564 sdc1 407.93 206.98 972.19 513311349 2411036564 ..

To execute every 2 seconds for a total of 3 times, do the following.

$ iostat 2 3

10. iostat – Display LVM statistic (and version)

To display the LVM statistics use option -N as shown below.

$ iostat -N

To display the version of iostat, use -V. This will really display the version information of sysstat, as iostat is part of sysstat package.

$ iostat -V sysstat version 7.0.2 (C) Sebastien Godard

VMSTAT EXAMPLES

11. vmstat – Basic example

vmstat by default will display the memory usage (including swap) as shown below.

$ vmstat procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 305416 260688 29160 2356920 2 2 4 1 0 0 6 1 92 2 0

vmstat output contains the following fields:

Procs – r: Total number of processes waiting to run

Procs – b: Total number of busy processes

Memory – swpd: Used virtual memory

Memory – free: Free virtual memory

Memory – buff: Memory used as buffers

Memory – cache: Memory used as cache.

Swap – si: Memory swapped from disk (for every second)

Swap – so: Memory swapped to disk (for every second)

IO – bi: Blocks in. i.e blocks received from device (for every second)

IO – bo: Blocks out. i.e blocks sent to the device (for every second)

System – in: Interrupts per second

System – cs: Context switches

CPU – us, sy, id, wa, st: CPU user time, system time, idle time, wait time

12. vmstat – Display active and inactive memory

By default vmstat doesn’t display this information. Use option -a, to display active and inactive memory information as shown below.

$ vmstat -a procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free inact active si so bi bo in cs us sy id wa st 0 0 305416 253820 1052680 2688928 2 2 4 1 0 0 6 1 92 2 0

13. vmstat – Display number of forks since last boot

This displays all the fork system calls made by the system since the last boot. This displays all fork, vfork, and clone system call counts.

$ vmstat -f 81651975 forks

14. vmstat – Execute Every x seconds (for y number of times)

To execute every 2 seconds, do the following. You have to press Ctrl-C to stop this.

$ vmstat 2 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 537144 182736 6789320 0 0 0 0 1 1 0 0 100 0 0 0 0 0 537004 182736 6789320 0 0 0 0 50 32 0 0 100 0 0 ..

To execute every 2 seconds for 10 times, do the following. You don’t need to press Ctrl-C in this case. After executing 10 times, it will stop automatically.

$ vmstat 2 10 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 537144 182736 6789320 0 0 0 0 1 1 0 0 100 0 0 0 0 0 537004 182736 6789320 0 0 0 0 50 32 0 0 100 0 0 ..

15. vmstat – Display timestamp

When you use vmstat to monitor the memory usage repeately, it would be nice to see the timestap along with every line item. Use option -t to display the time stamp as shown below.

$ vmstat -t 1 100 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp--- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 3608728 148368 3898200 0 0 0 0 1 1 0 0 100 0 0 2011-07-09 21:16:28 PDT 0 0 0 3608728 148368 3898200 0 0 0 0 60 15 0 0 100 0 0 2011-07-09 21:16:29 PDT 0 0 0 3608712 148368 3898200 0 0 0 0 32 28 0 0 100 0 0 2011-07-09 21:16:30 PDT

For me, the timestamp option worked in the following version.

$ vmstat -V procps version 3.2.8

Note: If you use a older version of vmstat, option -t might not be available. In that case, use the method we suggested earlier to display timestamp in vmstat output.

16. vmstat – Display slab info

Use option -m, to display the slab info as shown below.

$ vmstat -m Cache Num Total Size Pages fib6_nodes 5 113 32 113 ip6_dst_cache 4 15 256 15 ndisc_cache 1 15 256 15 RAWv6 7 10 768 5 UDPv6 0 0 640 6 tw_sock_TCPv6 0 0 128 30 ...

17. vmstat – Display statistics in a table format

Instead of displays the values in the record format, you can display the output of vmstat in table format using option -s as shown below.

$ vmstat -s 4149928 total memory 3864824 used memory 2606664 active memory 1098180 inactive memory 285104 free memory 19264 buffer memory 2326692 swap cache 4192956 total swap 274872 used swap 3918084 free swap 1032454000 non-nice user cpu ticks 14568 nice user cpu ticks 89482270 system cpu ticks 16674327143 idle cpu ticks 368965706 IO-wait cpu ticks 1180468 IRQ cpu ticks ..

18. vmstat – Display disk statistics

Use option -d to display the disk statistics as shown below. This displays the reads, writes, and I/O statistics of the disk.

$ vmstat -d disk- ------------reads------------ ------------writes----------- -----IO------ total merged sectors ms total merged sectors ms cur sec sda 153189971 69093708 2719150864 737822879 329617713 157559204 3965687592 4068577985 0 1102243 sdb 501426305 97099356 2345472425 731613156 419220973 533565961 2661869460 1825174087 0 1510434 sdc 884213459 22078974 513390701 452540172 127474901 8993357 2411187300 2133226954 0 1569758

19. vmstat – Increase the width of the display

The default output without increasing the width is shown below.

$ vmstat 1 3 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 3608688 148368 3898204 0 0 0 0 1 1 0 0 100 0 0 0 0 0 3608804 148368 3898204 0 0 0 0 72 30 0 0 100 0 0 0 0 0 3608804 148368 3898204 0 0 0 0 60 27 0 0 100 0 0

Use option -w to increase the width of the output columns as shown below. This give better readability.

$ vmstat -w 1 3 procs -------------------memory------------------ ---swap-- -----io---- --system-- -----cpu------- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 3608712 148368 3898204 0 0 0 0 1 1 0 0 100 0 0 0 0 0 3608712 148368 3898204 0 0 0 0 93 23 0 0 100 0 0 0 0 0 3608696 148368 3898204 0 0 0 0 35 34 0 0 100 0 0

20. vmstat – Display statistics for a partition

To display the disk I/O statistics of a specific disk partition use option -p as shown below.

$ vmstat -p sdb1 sdb1 reads read sectors writes requested writes 501423248 2345417917 419221612 2661885948

21. vmstat – Display in MB

By default vmstat displays the memory information in kb. To disply in MB, use the option “-S m” as shown below.

$ vmstat -S m procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 281 288 19 2386 0 0 4 1 0 0 6 1 92 2 0

MPSTAT EXAMPLES

22. mpstat – Display basic info

By default mpstat displays CPU statistics as shown below.

$ mpstat Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 10:25:32 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 10:25:32 PM all 5.68 0.00 0.49 2.03 0.01 0.02 0.00 91.77 146.55

23. mpstat – Display all information

Option -A, displays all the information that can be displayed by the mpstat command as shown below. This is really equalivalent to “mpstat -I ALL -u -P ALL” command.

$ mpstat -A Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_ (4 CPU) 10:26:34 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 10:26:34 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.99 10:26:34 PM 0 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 99.98 10:26:34 PM 1 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98 10:26:34 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 10:26:34 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 10:26:34 PM CPU intr/s 10:26:34 PM all 36.51 10:26:34 PM 0 0.00 10:26:34 PM 1 0.00 10:26:34 PM 2 0.04 10:26:34 PM 3 0.00 10:26:34 PM CPU 0/s 1/s 8/s 9/s 12/s 14/s 15/s 16/s 19/s 20/s 21/s 33/s NMI/s LOC/s SPU/s PMI/s PND/s RES/s CAL/s TLB/s TRM/s THR/s MCE/s MCP/s ERR/s MIS/s 10:26:34 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.47 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:26:34 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.90 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:26:34 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.04 0.00 0.00 0.00 0.00 0.00 3.32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:26:34 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.17 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00

24. mpstat – Display CPU statistics of individual CPU (or) Core

Option -P ALL, displays all the individual CPUs (or Cores) along with its statistics as shown below.

$ mpstat -P ALL Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_ (4 CPU) 10:28:04 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 10:28:04 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.99 10:28:04 PM 0 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 99.98 10:28:04 PM 1 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98 10:28:04 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 10:28:04 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

To display statistics information of a particular CPU (or core), use option -P as shown below.

$ mpstat -P 0 Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_ (8 CPU) 10:28:53 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 10:28:53 PM 0 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 99.98 $ mpstat -P 1 Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_ (8 CPU) 10:28:55 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 10:28:55 PM 1 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98

Finally, as we mentioned earlier mpstat is part of the sysstat package. When you do mpstat -V, it will really display the version number of the systat package as shown below.

$ mpstat -V sysstat version 9.0.4 (C) Sebastien Godard (sysstat orange.fr)

Previous articles in the Linux performance monitoring and tuning series:

If you enjoyed this article, you might also like..