monitoring, file descriptor, linux, python, psutil

What is a file descriptor

What is a file descriptor, and why do I need them? A file descriptor is a handle that a program uses to communicate with operating system resources. The most common case, is a program that is reading from a file.

Each process has a table with numbered file descriptors. Usually, the three first ones are:

0 stdin (Standard input stream)

1 stdout (Standard output stream)

2 stderr (Standard error output stream)

When a text editor, requests access to a resource, for example HOMEWORK.txt, the operating system will return a number that represents a file descriptor. Writing to that file descriptor, or number if you want, will store whatever you type, to a file on your filesystem.

A practical example with file descriptors I use all the time, is stderr redirecting. Let's say i'm looking for a file, and I don't want to see all those pesky permission denied messages from find:

$ find / -xdev -name "*geoip*" 2>/dev/null /usr/share/man/man1/geoiplookup6.1.gz /usr/share/man/man1/geoipupdate.1.gz /usr/share/man/man1/geoiplookup.1.gz /usr/share/licenses/php-pecl-geoip-1.1.1 ... ... (notice the lack of permission denied messages) ... $

Appending 2>/dev/null to the end of a command, will redirect all output from filedescriptor two, to /dev/null, so I won't have to see it.

Most languages, like Perl, PHP, Python, Ruby, etc, doesn't return a number, but a file handle. The file handle is just a abstraction, and will always contain a file descriptor as well. They all use a underlying system call called open() to open a file.

#!/usr/bin/python fh = open('/tmp/what-fd-is-this.txt', 'w+') fh.write("This file has file descriptor number: %d in this process

" % fh.fileno()) fh.close() fh = open('/tmp/what-fd-is-this.txt', 'r') print fh.readline()

Another very common use case for file descriptors, are sockets. Sockets will allow you to write data to other processes on your machine, or processes on other machines across the Atlantic ocean.

Why do I need to know about file descriptors?

If you are a programmer or a system administrator, chances are, you will or have had problems with file descriptor limits. This means that your Redis database won’t accept more connections, your nginx webserver can’t connect to the PHP-FPM backend, or that MySQL can’t open new files for writing. This is especially relevant in 2018, where busy websites have thousands of simulatanious clients. These clients can in turn open tens of file desciptors.

File descriptors can be treacherous because they have limits in several layers. When this limits is reached, software will fail. Sometimes only temporarily. This can lead to reduced performance and service outages, which are hard to track down.

Always check the ulimit of your processes. This is the maximum number of file descriptors processes spawned can use.

$ ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files (-n) 256 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 2032 cpu time (seconds, -t) unlimited max user processes (-u) 256 virtual memory (kbytes, -v) unlimited

Then you have the global operating system limit:

# sysctl fs.nr_open fs.nr_open = 1048576 # sysctl fs.file-max fs.file-max = 2427544 #

Additionally, each service often has their own, internal limit:

Nginx: worker_rlimit_nofile

Redis: maxclients

MySQL: max_connections

Each service has their own way of configuring these max settings, and their unique way of failling when the limits are reached.

I’ve had problems with both redis and kafka meeting these limits in large scale deployments. A distributed system won’t help you if misconfiguration is the part that’s failing.

Why can't I just turn off the limits?

The limits are there, because each allocated file descriptor costs memory. Also, with unlimited connections, any service would slowly grind to a halt. It's better to have sensible limits, which can later be raised, than having the OOM killer on unannounced visits.

What should I do about it?

Correctly configuring services and limits are part of a system administrators job. Increasing these limits is also one of the first things a typical "tune for performance" article will tell you to do.

But since I'm a human, and tend to forget things, I wrote a simple script in python, to remind me of problems before they occur.

#!/usr/bin/python import sys import psutil WARN_PERCENT = 50 WARN_NUM = 1000 onlyWarn = False hasPrintedHeader = False def printHeader(): global hasPrintedHeader if hasPrintedHeader: return hasPrintedHeader = True print " USED TOTAL USED% REMAINING PROCESS" if not onlyWarn: printHeader() for pid in psutil.pids(): try: p = psutil.Process(pid) except psutil._exceptions.NoSuchProcess: continue num_fds = p.num_fds() if not num_fds: continue soft, hard = p.rlimit(psutil.RLIMIT_NOFILE) used_soft_percent = 100 * num_fds / soft remaining = soft - num_fds if onlyWarn: if used_soft_percent < WARN_PERCENT and remaining > WARN_NUM: continue printHeader() print "%8d %8d %3d%% %8d %s (%d)" % ( num_fds, soft, used_soft_percent, remaining, p.name(), pid)

The script requires the psutil package. It can be installed with the following command:

$ pip install psutil

This assumes that you have a working python 2 install with pip and the psutil installation went fine. Save the contents of your script to fd.py, and you should be ready to go.

[root@orange src]# python fd.py | sort -nk1 USED TOTAL USED% REMAINING PROCESS ... 6 1024 0% 1018 xinetd (95) 7 1024 0% 1017 saslauthd (113) 7 1024 0% 1017 saslauthd (114) 9 1024 0% 1015 agent.py (103) 9 1024 0% 1015 php-fpm (4057) 9 1024 0% 1015 php-fpm (7411) 11 1024 1% 1013 polkitd (2998) 11 1024 1% 1013 sshd (18195) 11 1024 1% 1013 sshd (19149) 11 1024 1% 1013 sshd (8699) 12 1024 1% 1012 systemd-udevd (66) 12 4096 0% 4084 pickup (28181) 12 4096 0% 4084 qmgr (387) 13 65536 0% 65523 dbus-daemon (105) 14 16384 0% 16370 systemd-journald (32374) 15 4096 0% 4081 node (115) 15 16384 0% 16369 systemd-logind (93) 17 1024 1% 1007 nginx (352) 18 1024 1% 1006 nginx (1251) 33 65536 0% 65503 systemd (1) 39 1024 3% 985 rsyslogd (94) 52 10240 0% 10188 redis-server (104) 83 16364 0% 16281 mysqld (258) 90 4096 2% 4006 master (373) [root@orange src]#

With small modifications, I could use this script to send me an email, if any of my hosts were about to run out of file descriptors.

if onlyWarn: if used_soft_percent < WARN_PERCENT and remaining > WARN_NUM: continue # Exit with erronous status when error condition is met sys.exit(1)

It can be added to cron like this:

# Periodically check fd limits. Exit status of 1 means that the expression # after || will be executed * * * * * /path/to/fdcheck.py || (echo | mail -s FDERROR myself@mydomain.com)

The script can obviously benefit from some cleanups, but as an illustrative example, it serves it purpose well.

And that's all there is to it! Now you will receive a email once one of your processes exceeds the thresholds you set, which allows you to sleep just slightly better than before.