kernel.org status: hints on how to check your machine for intrusion

From: Greg KH <greg-AT-kroah.com> To: Linux Kernel Mailing List <linux-kernel-AT-vger.kernel.org> Subject: kernel.org status: hints on how to check your machine for intrusion Date: Fri, 30 Sep 2011 16:59:24 -0700 Message-ID: <20110930235924.GA25176@kroah.com> Archive-link: Article, Thread

The compromise of kernel.org and related machines has made it clear that some developers, at least, have had their systems penetrated. As we seek to secure our infrastructure, it is imperative that nobody falls victim to the belief that it cannot happen to them. We all need to check our systems for intrusions. Here are some helpful hints as proposed by a number of developers on how to check to see if your Linux machine might be infected with something: 0. One way to be sure that your system is not compromised is to simply do a clean install; we can all benefit from a new start sometimes. Before reinstalling any systems, though, consider following the steps below to learn if your system has been hit or not. 1. Install the chkrootkit package from your distro repository and see if it reports anything. If your distro doesn't have the chkroot package, download it from: http://www.chkrootkit.org/ Another tool is the ossec-rootcheck tool which can be found at: http://www.ossec.net/main/rootcheck And another one is the rkhunter program: http://www.rootkit.nl/projects/rootkit_hunter.html [Note, this tool has the tendancy to give false-positives on some Debian boxes, please read /usr/share/doc/rkhunter/README.Debian.gz if you run this on a Debian machine] 2. Verify that your package signatures match what your package manager thinks they are. To do this on a rpm-based system, run the following command: rpm --verify --all Please read the rpm man page for information on how to interpret the output of this command. To do this on a Debian based system, run the following bash snippet: dpkg -l \*|while read s n rest; do if [ "$s" == "ii" ]; then echo $n; fi; done > ~/tmp.txt for f in `cat ~/tmp.txt`; do debsums -s -a $f; done If you have a source-based system (Gentoo, LFS, etc.) you presumably know what you are doing already. 3. Verify that your packages are really signed with the distro's keys. Here's a bash snippet that can do this on a rpm based system to verify that the packages are signed with any key, not necessarily your distro's key. That exercise is left for the reader: for package in `rpm -qa`; do sig=`rpm -q --qf '%{SIGPGP:pgpsig}

' $package` if [ -z "$sig" ] ; then # check if there is a GPG key, not a PGP one sig=`rpm -q --qf '%{SIGGPG:pgpsig}

' $package` if [ -z "$sig" ] ; then echo "$package does not have a signature!!!" fi fi done Unfortunately there is no known way of verifying this on Debian-based systems. 4. To replace a package that you find suspect, uninstall it and install it anew from your distro. For example, if you want to reinstall the ssh daemon, you would do: $ /etc/init.d/sshd stop rpm -e openssh zypper install openssh # for openSUSE based systems yum install openssh # for Fedora based systems Ideally do this from a live cdrom boot, using the 'rpm --root' option to point rpm at the correct location. 5. From a liveCD environment, look for traces such as: a. Rogue startup scripts in /etc/rc*.d and equivalent directories. b. Strange directories in /usr/share that do not belong to a package. This can be checked on an rpm system with the following bash snippet: for file in `find /usr/share/`; do package=`rpm -qf -- ${file} | grep "is not owned"` if [ -n "$package" ] ; then echo "weird file ${file}, please check this out" fi done 6. Look for mysterious log messages, such as: a. Unexpected logins in wtmp and /var/log/secure*, quite possibly from legitimate users from unexpected hosts. b. Any program trying to touch /dev/mem. c. References to strange (non-text) ssh version strings in /var/log/secure*. These do not necessarily indicate *successful* breakins, but they indicate *attempted* breakins which means your system or IP address has been targeted. 7. If any of the above steps show possible signs of compromise, you should investigate further and identify the actual cause. If it becomes clear that the system has indeed been compromised, you should certainly reinstall the system from the beginning, and change your credentials on all machines that this machine would have had access to, or which you connected to through this machine. You will need to check your other systems carefully, and you should almost certainly notify the administrators of other systems to which you have access. Finally, please note that these hints are not guaranteed to turn up signs of a compromised systems. There are a lot of attackers out there; some of them are rather more sophisticated than others. You should always be on the alert for any sort of unexpected behavior from the systems you work with. thanks, greg k-h