Homepage of Arthur de Jong

This document describes the steps I used to recover a disk crash on my laptop. My disc had some problems and most reiserfs filesystems were corrupted.

  1. get your system to boot in single-user mode
    Try to boot the system in single user mode by typing
    linux s
    on the LILO prompt. Another way would be to pass LILO:
    linux init=/bin/bash
    booting from a rescue disk or the Debian installer would also be a good idea if your root filesystem is corrupt.
    If you have any processes running that use the filesystem (e.g. portmap) kill them off.
  2. run fsck
    Most of my partitions had problems so I had to run reiserfsck:
    reiserfsck --check /dev/hda1
    (substitute hda1 with the partition you want to check)
    If it doesn't complain the filesystem is ok. Otherwise you may have to pass --fix-fixable, --rebuild-tree or --rebuild-sb (reiserfsck tells you which to use).
    You cannot do this on read-write mounted filesystems though, best is to unmount it but some things may also be possible with:
    mount /dev/hda1 -o remount,ro
  3. get basic files in place
    After you get your filesystems back in order you may be missing some files. I was missing parts of apt and libc.
    The tools you want working are apt and debsums.
    I got the libc parts from the install cd (don't worry about correct versions, it just needs to be more or less compatible, we'll get to fixing that later).
    If you are running from a rescue cd. you may want to do
    chroot /target
    to try out your system before you reboot.
    I had to restore another file from a deb at a time when dpkg wasn't working properly. To unpack a .deb file without dpkg you can try this:
    cd /     (or wherever your root filesystem is)
    ar x blah_i386.deb
    tar xvf data.tar.gz
    Another useful thing I had was a backup of /etc and /var/lib (which of cource I restored first).
  4. restore broken packages
    First you need to know which packages are broken. This is where debsums comes in. First, you need to generate a list of packages debsums knows nothing about. To fetch all packages without stored md5sums and generate them do:
    cd /var/cache/apt/archives
    apt-get --download-only --reinstall install `debsums -l`
    debsums --generate=keep,nocheck *.deb
    (for Ubuntu you need `debsums -l | grep -v dbgsym` instead) You can also configure debsums to automatically generate missing md5sums for newly installed packages (from version 2.0.7).
    To get a list of broken packages do:
    debsums -s -a 2> /tmp/broken.log
    This will run quite some time and produce all the packages that have modified or missing files. It should look something like:
    debsums: checksum mismatch zsh file bin/zsh4
    debsums: no md5sums for kernel-headers-2.4.24-local
    debsums: can't open libglib1.2-dev file usr/lib/libglib.a (No such file or directory)
    You should probably review this list manually but you could also do:
    sed -n 's/^.*\(checksum mismatch\|changed file\) \([^ ]*\) file.*$/\1/p;s/^.*t open \([^ ]*\) file.*$/\1/p' < /tmp/broken.log | sort -u > /tmp/broken.pkgs
    (all on one line)
    Now reinstall all the packages with:
    apt-get --reinstall install `cat /tmp/broken.pkgs`
    cat /tmp/broken.pkgs | xargs -n 10 apt-get --reinstall install
    if you run into problems.
    The only thing that might be missing now is stuff that is set up during package installation (symlinks, etc).

I hope this helps someone when recovering their system. It's not perfect but it worked for me so it can't be completely useless.