Arthur de Jong

Crash recovery using debsums

2004-11-08

This document describes the steps I used to recover a disk crash on my laptop. My disc had some problems and most reiserfs filesystems were corrupted.

I hope this helps someone when recovering their system. It's not perfect but it worked for me so it can't be completely useless.

Get your system to boot in single-user mode

Try to boot the system in single user mode by typing

linux s

on the LILO prompt. Another way would be to pass LILO:

linux init=/bin/bash

booting from a rescue disk or the Debian installer would also be a good idea if your root filesystem is corrupt.
If you have any processes running that use the filesystem (e.g. portmap) kill them off.

Run fsck

Most of my partitions had problems so I had to run reiserfsck:

reiserfsck --check /dev/hda1

(substitute hda1 with the partition you want to check)
If it doesn't complain the filesystem is ok. Otherwise you may have to pass --fix-fixable, --rebuild-tree or --rebuild-sb (reiserfsck tells you which to use).
You cannot do this on read-write mounted filesystems though, best is to unmount it but some things may also be possible with:

mount /dev/hda1 -o remount,ro

Get basic files in place

After you get your filesystems back in order you may be missing some files. I was missing parts of apt and libc.
The tools you want working are apt and debsums.
I got the libc parts from the install cd (don't worry about correct versions, it just needs to be more or less compatible, we'll get to fixing that later).
If you are running from a rescue cd. you may want to do

chroot /target

to try out your system before you reboot.
I had to restore another file from a deb at a time when dpkg wasn't working properly. To unpack a .deb file without dpkg you can try this:

cd /     (or wherever your root filesystem is)
ar x blah_i386.deb
tar xvf data.tar.gz

Another useful thing I had was a backup of /etc and /var/lib (which of course I restored first).

Restore broken packages

First you need to know which packages are broken. This is where debsums comes in. First, you need to generate a list of packages debsums knows nothing about. To fetch all packages without stored md5sums and generate them do:

cd /var/cache/apt/archives
apt-get --download-only --reinstall install `debsums -l`
debsums --generate=keep,nocheck *.deb

(for Ubuntu you need `debsums -l | grep -v dbgsym` instead) You can also configure debsums to automatically generate missing md5sums for newly installed packages (from version 2.0.7).
To get a list of broken packages do:

debsums -s -a 2> /tmp/broken.log

This will run quite some time and produce all the packages that have modified or missing files. It should look something like:

...
debsums: checksum mismatch zsh file bin/zsh4
debsums: no md5sums for kernel-headers-2.4.24-local
debsums: can't open libglib1.2-dev file usr/lib/libglib.a (No such file or directory)
...

You should probably review this list manually but you could also do:

sed -n 's/^.*\(checksum mismatch\|changed file\) \([^ ]*\) file.*$/\1/p;s/^.*t open \([^ ]*\) file.*$/\1/p' < /tmp/broken.log | sort -u > /tmp/broken.pkgs

(all on one line)
Now reinstall all the packages with:

apt-get --reinstall install `cat /tmp/broken.pkgs`

or

cat /tmp/broken.pkgs | xargs -n 10 apt-get --reinstall install

if you run into problems.
The only thing that might be missing now is stuff that is set up during package installation (symlinks, etc).