Crash recovery using debsums

2004-11-08

This document describes the steps I used to recover a disk crash on my laptop. My disc had some problems and most reiserfs filesystems were corrupted.

I hope this helps someone when recovering their system. It's not perfect but it worked for me so it can't be completely useless.

Get your system to boot in single-user mode

Try to boot the system in single user mode by typing

linux s

linux init=/bin/bash

Run fsck

on the LILO prompt. Another way would be to pass LILO:booting from a rescue disk or the Debian installer would also be a good idea if your root filesystem is corrupt.If you have any processes running that use the filesystem (e.g. portmap) kill them off.

Most of my partitions had problems so I had to run reiserfsck :

reiserfsck --check /dev/hda1

hda1

--fix-fixable

--rebuild-tree

--rebuild-sb

mount /dev/hda1 -o remount,ro

Get basic files in place

(substitutewith the partition you want to check)If it doesn't complain the filesystem is ok. Otherwise you may have to passor(reiserfsck tells you which to use).You cannot do this on read-write mounted filesystems though, best is to unmount it but some things may also be possible with:

After you get your filesystems back in order you may be missing some files. I was missing parts of apt and libc .

The tools you want working are apt and debsums .

I got the libc parts from the install cd (don't worry about correct versions, it just needs to be more or less compatible, we'll get to fixing that later).

If you are running from a rescue cd. you may want to do

chroot /target

.deb

cd / (or wherever your root filesystem is) ar x blah_i386.deb tar xvf data.tar.gz

/etc

/var/lib

Restore broken packages

to try out your system before you reboot.I had to restore another file from a deb at a time when dpkg wasn't working properly. To unpack afile without dpkg you can try this:Another useful thing I had was a backup ofand(which of course I restored first).

First you need to know which packages are broken. This is where debsums comes in. First, you need to generate a list of packages debsums knows nothing about. To fetch all packages without stored md5sums and generate them do:

cd /var/cache/apt/archives apt-get --download-only --reinstall install `debsums -l` debsums --generate=keep,nocheck *.deb

`debsums -l | grep -v dbgsym`

debsums -s -a 2> /tmp/broken.log

... debsums: checksum mismatch zsh file bin/zsh4 debsums: no md5sums for kernel-headers-2.4.24-local debsums: can't open libglib1.2-dev file usr/lib/libglib.a (No such file or directory) ...

sed -n 's/^.*\(checksum mismatch\|changed file\) \([^ ]*\) file.*$/\1/p;s/^.*t open \([^ ]*\) file.*$/\1/p' < /tmp/broken.log | sort -u > /tmp/broken.pkgs

apt-get --reinstall install `cat /tmp/broken.pkgs`

cat /tmp/broken.pkgs | xargs -n 10 apt-get --reinstall install

(for Ubuntu you needinstead) You can also configure debsums to automatically generate missing md5sums for newly installed packages (from version 2.0.7).To get a list of broken packages do:This will run quite some time and produce all the packages that have modified or missing files. It should look something like:You should probably review this list manually but you could also do:(all on one line)Now reinstall all the packages with:orif you run into problems.The only thing that might be missing now is stuff that is set up during package installation (symlinks, etc).