Welcome to /proc

What is /proc? It is a pseudo directory present in all Linux systems which provides information about the system. It was originally designed to convey process information (hence the name), but now conveys a wide array of system information. Note that I referred to /proc as a “pseudo directory”. It doesn’t actually exist (on disk). Instead, the Linux kernel creates this pseudo filesystem in memory, so that we can access it as if it were a directory on disk.

Let’s take a look.

$ ls /proc 1 13401 2 25 34 4478 50 5336 61 674 75 818 88 98 device-tree ioports locks slabinfo tty 10 14 21 26 35 45 5037 5337 62 676 77 844 89 asound devices irq meminfo softirqs uptime 1084 142 22 26222 3611 450 51 54 63 679 774 85 9 buddyinfo diskstats kallsyms misc stat vc-cma 1088 144 223 27 434 46 5136 5464 64 68 775 8580 90 bus driver key-users modules swaps version 11 15 228 28 435 47 5183 55 65 69 78 8588 91 cgroups execdomains keys mounts sys vmallocinfo 1105 152 229 29 438 48 52 56 66 7 79 8593 92 cmdline fb kmsg net sysrq-trigger vmstat 1127 17 23 3 439 481 53 57 67 70 8 86 93 consoles filesystems kpagecgroup pagetypeinfo sysvipc zoneinfo 1163 18 23307 31 44 485 5322 58 670 71 80 87 939 cpu fs kpagecount partitions thread-self 1238 18779 237 32 442 49 5323 59 671 72 801 870 96 cpuinfo interrupts kpageflags sched_debug timer_list 13 19 24 33 4477 5 5332 60 672 73 810 8719 97 crypto iomem loadavg self timer_stats

As you notice, there’s a whole slew of numeric directories. These contain information about the process with the corresponding id. There’s also a wide variety of other information, such as memory information (meminfo), load average statistics (loadavg), kernel version, gcc version, and Linux distribution (version), virtual memory statistics (vmstat), and a wide variety of other kernel and hardware information.

So, what’s in a process? Let’s take a peek.

$ ls /proc/ $( pgrep -f -n tmux ) attr auxv clear_refs comm cpuset environ fd gid_map limits map_files mem mounts net numa_maps oom_score pagemap projid_map sched sessionid smaps stat status task uid_map autogroup cgroup cmdline coredump_filter cwd exe fdinfo io loginuid maps mountinfo mountstats ns oom_adj oom_score_adj personality root schedstat setgroups stack statm syscall timers wchan

You can check out the status of a process in status.

$ cat /proc/ $( pgrep -f -n tmux ) /status Name: tmux State: S ( sleeping ) Tgid: 4143 Ngid: 0 Pid: 4143 PPid: 1 TracerPid: 0 Uid: 1001 1001 1001 1001 Gid: 1001 1001 1001 1001 FDSize: 64 Groups: 10 1001 VmPeak: 130100 kB VmSize: 129840 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 6136 kB VmRSS: 5516 kB RssAnon: 4220 kB RssFile: 1296 kB RssShmem: 0 kB VmData: 4404 kB VmStk: 140 kB VmExe: 448 kB VmLib: 2700 kB VmPTE: 80 kB VmSwap: 160 kB Threads: 1 SigQ: 0/127679 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000081802 SigCgt: 0000000188034201 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000001fffffffff CapAmb: 0000000000000000 Seccomp: 0 Cpus_allowed: ff Cpus_allowed_list: 0-7 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 1246008 nonvoluntary_ctxt_switches: 10550

In status you can see the state (sleeping), the process id (4143), the parent process id (1), ownership (user/group 1001), memory sizes, the number of threads, and many other details. It contains much of what is in stat and mem (status and memory information respectively), but in more human readable form.

You can also see what files the process is referencing in the fd directory. Here are the open file descriptors for my Plex Media Server.

$ cd /proc/ $( pgrep -f -n 'Plex Media Server' ) $ sudo ls -l fd total 0 lr-x------ 1 plex nogroup 64 Apr 28 00:44 0 -> /dev/null l-wx------ 1 plex nogroup 64 Apr 28 00:44 1 -> /dev/null l-wx------ 1 plex nogroup 64 Apr 28 00:44 10 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log lrwx------ 1 plex nogroup 64 Apr 28 00:44 11 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.dlna.db-shm lrwx------ 1 plex nogroup 64 Apr 28 00:44 12 -> socket:[11739] lrwx------ 1 plex nogroup 64 Apr 28 00:44 13 -> socket:[11740] lrwx------ 1 plex nogroup 64 Apr 28 00:44 14 -> socket:[11741] lrwx------ 1 plex nogroup 64 Apr 28 00:44 15 -> socket:[11742] lrwx------ 1 plex nogroup 64 Apr 28 00:44 16 -> socket:[11743] lrwx------ 1 plex nogroup 64 Apr 28 00:44 17 -> socket:[11744] lrwx------ 1 plex nogroup 64 Apr 28 00:44 18 -> socket:[11746] lrwx------ 1 plex nogroup 64 Apr 28 00:44 19 -> socket:[11747] l-wx------ 1 plex nogroup 64 Apr 28 00:44 2 -> /dev/null lrwx------ 1 plex nogroup 64 Apr 28 00:44 20 -> socket:[11748] lrwx------ 1 plex nogroup 64 Apr 28 00:44 21 -> socket:[11749] lrwx------ 1 plex nogroup 64 Apr 28 00:44 22 -> socket:[11750] lrwx------ 1 plex nogroup 64 Apr 28 00:44 23 -> socket:[11751] lrwx------ 1 plex nogroup 64 Apr 28 00:44 24 -> socket:[11752] lrwx------ 1 plex nogroup 64 Apr 28 00:44 25 -> socket:[11753] lrwx------ 1 plex nogroup 64 Apr 28 00:44 26 -> socket:[11754] lrwx------ 1 plex nogroup 64 Apr 28 00:44 27 -> socket:[11755] lrwx------ 1 plex nogroup 64 Apr 28 00:44 28 -> socket:[11756] lrwx------ 1 plex nogroup 64 Apr 28 00:44 29 -> socket:[11757] lrwx------ 1 plex nogroup 64 Apr 28 00:44 3 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs/Plex DLNA Server Neptune.log lrwx------ 1 plex nogroup 64 Apr 28 00:44 30 -> socket:[11758] lrwx------ 1 plex nogroup 64 Apr 28 00:44 31 -> socket:[11759] lrwx------ 1 plex nogroup 64 Apr 28 00:44 32 -> socket:[11760] lrwx------ 1 plex nogroup 64 Apr 28 00:44 33 -> socket:[65881] lrwx------ 1 plex nogroup 64 Apr 28 00:44 34 -> socket:[65882] lrwx------ 1 plex nogroup 64 Apr 28 00:44 35 -> socket:[65883] lrwx------ 1 plex nogroup 64 Apr 28 00:44 4 -> anon_inode:[eventfd] lrwx------ 1 plex nogroup 64 Apr 28 00:44 5 -> anon_inode:[eventpoll] lrwx------ 1 plex nogroup 64 Apr 28 00:44 56 -> socket:[10045] lr-x------ 1 plex nogroup 64 Apr 28 00:44 57 -> pipe:[10038] lrwx------ 1 plex nogroup 64 Apr 28 00:44 58 -> socket:[10044] lrwx------ 1 plex nogroup 64 Apr 28 00:44 59 -> socket:[10050] lrwx------ 1 plex nogroup 64 Apr 28 00:44 6 -> anon_inode:[timerfd] lrwx------ 1 plex nogroup 64 Apr 28 00:44 60 -> socket:[10053] l-wx------ 1 plex nogroup 64 Apr 28 00:44 61 -> pipe:[10039] lrwx------ 1 plex nogroup 64 Apr 28 00:44 62 -> socket:[11730] lrwx------ 1 plex nogroup 64 Apr 28 00:44 63 -> socket:[10058] lrwx------ 1 plex nogroup 64 Apr 28 00:44 64 -> socket:[11731] l-wx------ 1 plex nogroup 64 Apr 28 00:44 7 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs/Plex DLNA Server.log lr-x------ 1 plex nogroup 64 Apr 28 00:44 72 -> /dev/urandom lrwx------ 1 plex nogroup 64 Apr 28 00:44 8 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.dlna.db lrwx------ 1 plex nogroup 64 Apr 28 00:44 82 -> socket:[10072] lrwx------ 1 plex nogroup 64 Apr 28 00:44 9 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.dlna.db-wal

Using the proc filesystem, we can even access and recover files. Let’s get the tail of the log for file descriptor 10 (Plex Media Server.log).

$ sudo cat fd/10 | tail Apr 28, 2017 00:20:19.835 [ 0x731ff400] DEBUG - NAT: UPnP, not an IGD: <http://192.168.0.1:49152/wps_device.xml>. Apr 28, 2017 00:20:19.837 [ 0x731ff400] DEBUG - NAT: UPnP, found device <http://192.168.0.1:49152/wps_device.xml> with private address <192.168.0.101> Apr 28, 2017 00:20:19.838 [ 0x731ff400] DEBUG - NAT: UPnP, not an IGD: <http://192.168.0.1:49152/wps_device.xml>. Apr 28, 2017 00:20:19.840 [ 0x731ff400] DEBUG - NAT: UPnP, found device <http://192.168.0.1:1900/igd.xml> with private address <192.168.0.101> Apr 28, 2017 00:20:19.845 [ 0x731ff400] DEBUG - NAT: UPnP, usable device <http://192.168.0.1:1900/igd.xml> with private address <192.168.0.101>. Apr 28, 2017 00:20:19.847 [ 0x731ff400] DEBUG - NAT: UPnP, public address is XXX.XXX.XXX.XXX Apr 28, 2017 00:20:19.850 [ 0x680fb400] DEBUG - PublicAddressManager: Mapping succeeded for 192.168.0.101:23979. Apr 28, 2017 00:20:19.854 [ 0x731ff400] DEBUG - MyPlex: Last published value didn 't change, we' re done . Apr 28, 2017 00:20:19.855 [ 0x680fb400] DEBUG - MyPlex: Last published value didn 't change, we' re done . Apr 28, 2017 00:50:17.467 [ 0x680fb400] DEBUG - Sync: uploadStatus

We can also check out the environment of the process via environ.

$ sudo cat environ INFINALITY_FT_AUTOHINT_HORIZONTAL_STEM_DARKEN_STRENGTH = 10^@MAIL = /var/mail/plex^@USER = plex^@INFINALITY_FT_FRINGE_FILTER_STRENGTH = 0^@SHLVL = 1^@LD_LIBRARY_PATH = /usr/lib/plexmediaserver^@HOME = /var/lib/plexmediaserver^@PLEX_MEDIA_SERVER_TMPDIR = /tmp^@OLDPWD = /var/lib/plexmediaserver^@INFINALITY_FT_CONTRAST = 0^@INFINALIT Y_FT_USE_VARIOUS_TWEAKS = true ^@INFINALITY_FT_GAMMA_CORRECTION = 0 100^@INFINALITY_FT_WINDOWS_STYLE_SHARPENING_STRENGTH = 10^@INFINALITY_FT_STEM_ALIGNMENT_STRENGTH = 25^@TMPDIR = /tmp^@LOGNAME = plex^@_ = /usr/sbin/start_pms^@XDG_SESSION_ID = c1^@INFINALITY_FT_AUTOHINT_VERTICAL_STEM_DARKEN_STRENGTH = 25^@PATH = /usr/local/sbin:/us r/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games^@XDG_RUNTIME_DIR = /run/user/117^@PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR = /var/lib/plexmediaserver/Library/Application Support^@INFINALITY_FT_GLOBAL_EMBOLDEN_X_VALUE = 0^@LANG = en_GB.UTF-8^@INFINALITY_FT_AUTOHINT_SNAP_STEM_HEIGHT = 100^@INFINALITY _FT_GLOBAL_EMBOLDEN_Y_VALUE = 0^@INFINALITY_FT_CHROMEOS_STYLE_SHARPENING_STRENGTH = 0^@SHELL = /bin/bash^@PLEX_MEDIA_SERVER_MAX_STACK_SIZE = 3000^@INFINALITY_FT_BRIGHTNESS = 0^@INFINALITY_FT_STEM_SNAPPING_SLIDING_SCALE = 40^@INFINALITY_FT_STEM_FITTING_STRENGTH = 25^@INFINALITY_FT_GRAYSCALE_FILTER_STRENGTH = 0^@INFINALITY_FT_US E_KNOWN_SETTINGS_ON_SELECTED_FONTS = true ^@PWD = /usr/lib/plexmediaserver^@INFINALITY_FT_AUTOHINT_INCREASE_GLYPH_HEIGHTS = true ^@PLEX_MEDIA_SERVER_HOME = /usr/lib/plexmediaserver^@INFINALITY_FT_BOLD_EMBOLDEN_X_VALUE = 0^@INFINALITY_FT_FILTER_PARAMS = 11 22 38 22 11^@INFINALITY_FT_BOLD_EMBOLDEN_Y_VALUE = 0^@PLEX_MEDIA_SERVER_ MAX_PLUGIN_PROCS = 6^@

We can inspect the command with which the process was launched via cmdline.

$ sudo cat cmdline ./Plex Media Server

We can also take a look at the various kernel level limits for a process in limits.

$ sudo cat limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 7336 7336 processes Max open files 65536 65536 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 7336 7336 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us

Wrapping up

As you can see, there’s all kinds of cool information in /proc!

Many utilities (ps, top, lspci, etc.) get their information from the /proc filesystem.

Let’s take a look at the files ps opens when it runs on my Raspberry Pi.

$ strace -etrace = open ps open ( "/etc/ld.so.preload" , O_RDONLY|O_CLOEXEC ) = 3 open ( "/usr/lib/arm-linux-gnueabihf/libarmmem.so" , O_RDONLY|O_CLOEXEC ) = 3 open ( "/etc/ld.so.cache" , O_RDONLY|O_CLOEXEC ) = 3 open ( "/lib/arm-linux-gnueabihf/libprocps.so.3" , O_RDONLY|O_CLOEXEC ) = 3 open ( "/lib/arm-linux-gnueabihf/libdl.so.2" , O_RDONLY|O_CLOEXEC ) = 3 open ( "/lib/arm-linux-gnueabihf/libc.so.6" , O_RDONLY|O_CLOEXEC ) = 3 open ( "/sys/devices/system/cpu/online" , O_RDONLY|O_CLOEXEC ) = 3 open ( "/proc/self/stat" , O_RDONLY ) = 3 open ( "/proc/uptime" , O_RDONLY ) = 3 open ( "/proc/sys/kernel/pid_max" , O_RDONLY ) = 4 open ( "/proc/meminfo" , O_RDONLY ) = 4 open ( "/proc/1/stat" , O_RDONLY ) = 6 open ( "/proc/1/status" , O_RDONLY ) = 6 open ( "/proc/2/stat" , O_RDONLY ) = 6 open ( "/proc/2/status" , O_RDONLY ) = 6 open ( "/proc/3/stat" , O_RDONLY ) = 6 open ( "/proc/3/status" , O_RDONLY ) = 6 ...

Nifty, huh? strace is another one of those cool Linux utilites. It can be used to trace system calls (the essential interface between programs and the operating system) in a running program. More on that in a future post. :)