red hat logo

So you've screwed-up your kernel:         last update: 10 Oct 2001.

If you've somehow rendered your system inoperable, this page may be able to help to get you back in the air.

Your system says:

       lilo:
       Booting vmlinuz.....................................
and then hangs.

If you have a rescue disk, and you didn't blow away your old kernel, you should be able to rescue your system using RedHat's excellent instructions.

On the other hand, if you blew away your kernel, or you don't have a rescue disk...  You'll need to do some typing.
 

-1.  Interesting tidbits I found out after writing this page...

I was reading a mailing list and I found out the following response to one user having the problem described above, which appears below. I haven't had time to integrate this into this page, but you may try this stuff if you're in this situation...

From Roswell mailing list, 10Oct2001:

     1.  At the boot prompt type
             linux single root=/dev/hda? initrd=
     where /dev/hda? is your / partition (e.g. mine is /dev/hda8).  This will
     boot your system in single-user mode, and you can make any modifications
     that you need, then reboot.

     2.  At the boot prompt type
             linux rescue
     This will ask a few questions (like your language), and then it will tell
     you that your system has been mounted on /mnt/sysimage.  At this point, type
             chroot /mnt/sysimgae
     And do whatever changes are necessary.  After you are done, type
             sync;sync;exit
     twice and reboot.

     That should do it.

I believe that item 1 refers to being able to boot your system and mount the hard drive alone.

I believe the use of chroot would remove the differencing that would be needed in item 3 below, and you might not have to change your mkinitrd file. For example, it might be as easy as:

	# chroot /mnt/root
	# sh mkinitrd /boot/initrd-2.2.19-6.2.7smp 2.2.19-6.2.7smp
	# sh mkinitrd /boot/initrd-2.2.19-6.2.7 2.2.19-6.2.7

Finally, I think the sync;sync;exit twice might solve the dismounting problem mentioned in section 5 and 6 of this page.

0.  Think.

Think about what you did last:
  • Did you delete your kernel?  I think in this case you have to pull out your disk, put it in another system, copy the kernel onto the disk, and then rebuild the initrd and lilo map using the steps below.
  • Did you forget to mkinitrd?  This is probably not important if you don't have a SCSI system, but if you do, then it is very important.    In this case, you'll have to mkinitrd  and then use lilo to build the boot map.
  • Did you forget to use lilo to build a boot map before you turned off your system? If this is what you did, you can probably skip step 3.
  • 1.  Getting a prompt.

    You'll need a copy of the the bootable CDROM that comes with the distribution.   Boot off this disk.  Some text will appear requesting how you would like to boot.  Type 'expert' and press return:
     
         boot: expert
    


    The system will start the install script.  It'll request some drivers off a floppy, you can skip that part.  Next, it'll ask you if you need to load any drivers for special devices.  If you have a SCSI system, then use this to load your SCSI driver.  If you can't remember which one it is, you may have to play around with this a bit until you find something that works.

    Next, answer all the questions as usual until you get to the page that asks about what kind of mouse you have.   At this point, X-Windows should have started, and you should be able to press CTRL-ALT-F2 to get into the virtual terminal.  A pound prompt should appear.

    2.  You're in: evaluating your disks.

    OK so you've successfully managed to get a prompt.   At this point you'll need to find out where your filesystems are.  You'll need to make nodes for them and then mount them so you can access them.

    If you have an IDE drive in your system, your major number is probably 3.  If you have a SCSI drive in your system, you should have loaded a SCSI driver as in step 1, and your major number is probably 8.  So then, for a SCSI system, type:
     

         # mknod /dev/sda b 8 0
         # fdisk /dev/sda
    


    You can query fdisk for your partitions:

         Command (m for help): p
    
         Disk /dev/sda: 255 heads, 63 sectors, 2213 cylinders
         Units = cylinders of 16065 * 512 bytes
    
            Device Boot    Start       End    Blocks   Id  System
         /dev/sda1             1         8     64228+  82  Linux swap
         /dev/sda3             9      2213  17711662+   5  Extended
         /dev/sda5             9       100    738958+  83  Linux
         /dev/sda6           101       500   3212968+  83  Linux
         /dev/sda7           501      1000   4016218+  83  Linux
         /dev/sda8          1001      1200   1606468+  83  Linux
         /dev/sda9          1201      2213   8136891   83  Linux
    
         Command (m for help): q
    

    Looking at this map, it's clear that device 1 is swap and 3 can't be mounted, because it's divided into 5, 6, 7, 8, and 9.  5 is probably /, and 7 is probably /usr, so let's go with that hunch:

         # mknod /dev/sda5 b 8 5
         # mknod /dev/sda7 b 8 7
         # mkdir /mnt/root
         # mount /dev/sda5 /mnt/root
         # ls /mnt/root
         bin   cern      dev  home  lost+found  opt   root  tmp  var
         boot  csc_data  etc  lib   mnt         proc  sbin  usr
         # mount /dev/sda7 /mnt/root/usr
    

    We'll need /usr because we need to get to /usr/bin/cut, which is used by mkinitrd.

    OK so now we have enough of our file system to continue.

    3.  Making initrd.

    This is probably the most complicated part. You should perform this part if you think that you need a RAM file system. For this part, we'll assume that you've mounted your filesystems as in section 2.

    You need to list the contents of /mnt/root/boot. Look at the files that start with vmlinuz-:

    A listing of /mnt/root/boot:

        # ls /mnt/root/boot
        System.map                  kernel.h-2.2.19
        System.map-2.2.19-6.2.7     map
        System.map-2.2.19-6.2.7smp  module-info
        boot.0800                   module-info-2.2.19-6.2.7
        boot.b                      module-info-2.2.19-6.2.7smp
        chain.b                     os2_d.b
        initrd-2.2.14-6.1.1.img     vmlinux-2.2.19-6.2.7
        initrd-2.2.14-6.1.1smp.img  vmlinux-2.2.19-6.2.7smp
        initrd-2.2.19-6.2.7.img     vmlinuz
        initrd-2.2.19-6.2.7smp.img  vmlinuz-2.2.19-6.2.7
        kernel.h                    vmlinuz-2.2.19-6.2.7smp
    

    In this system, you can see that initrd is already made for the vmlinuz's that exist. If they weren't, you'd have to perform the steps in this section.

    If you check out where vmlinuz and System.map point, you can see that this system is ready to boot the smp kernel, not the uniprocessor one.

    So at this point we'd like to build the smp initrd. Further, in this example we'd like to build it with the aic7xxx.o SCSI driver. Now, you can chose one of two procedures. You can either perform the steps below by hand, or you can copy mkinitrd to another place, and then modify it to perform these steps automatically. You'll need to do this by hand if you couldn't get /usr to mount as in step 2.

    To perform the necessary steps by hand, type:

        make a blank image which is the right size
         # cd /mnt/root/tmp
         # dd if=/dev/zero of=initrd.img-0 bs=1k count=1500
    
        find a loop device which is available:
         # /mnt/root/sbin/losetup /mnt/root/dev/loop0 initrd.img-0
        (if no error, then try with loop1, loop2, etc until loop8...
         in the continuation of this example, let's say that we have
         found loop5 to be empty...)
    
        make a file system:
         # echo y | mke2fs /mnt/root/dev/loop5 1500
    
        make a mount point and mount the file system:
         # mkdir -p /mnt/root/tmp/initrd.mnt-0 
         # mount -t ext2 /mnt/root/dev/loop5
    
        get rid of lost+found on our file system, we won't need it:
         # rm -rf /mnt/root/tmp/initrd.mnt-0/lost+found
    
        make a 'work space' to build our image that we need:
         # mkdir -p /mnt/root/tmp/initrd.0
         # mkdir -p /mnt/root/tmp/initrd.0/lib
         # mkdir -p /mnt/root/tmp/initrd.0/bin
         # mkdir -p /mnt/root/tmp/initrd.0/etc
         # mkdir -p /mnt/root/tmp/initrd.0/dev
         # mkdir -p /mnt/root/tmp/initrd.0/loopfs
    
        put the files that we'll need in our file system:
         # cp /mnt/root/sbin/sash /mnt/root/tmp/initrd.0/bin/sash
         # cp /mnt/root/bin/insmod.static /mnt/root/tmp/initrd.0/bin/insmod
         # cp -a /mnt/root/lib/modules/2.2.19-6.2.7smp/scsi/aic7xxx.o \
                  /mnt/root/tmp/initrd.0/lib
         # cp -a /mnt/root/dev/console /mnt/root/tmp/initrd.0/dev
         # cp -a /mnt/root/dev/null /mnt/root/tmp/initrd.0/dev
         # cp -a /mnt/root/dev/ram /mnt/root/tmp/initrd.0/dev
         # cp -a /mnt/root/dev/systty /mnt/root/tmp/initrd.0/dev
         # cp -a /mnt/root/dev/tty1 /mnt/root/tmp/initrd.0/dev
         # cp -a /mnt/root/dev/tty2 /mnt/root/tmp/initrd.0/dev
         # cp -a /mnt/root/dev/tty3 /mnt/root/tmp/initrd.0/dev
         # cp -a /mnt/root/dev/tty4 /mnt/root/tmp/initrd.0/dev
    
        make a linuxrc file by bringing up the vi editor:
         # /mnt/root/bin/vi /mnt/root/tmp/initrd.0/linuxrc
    
        put the following lines in this file:
         #!/bin/sash
    
         aliasall
    
         echo "Loading aic7xxx module"
         insmod /lib/aic7xxx.o
    
        make linuxrc be executable:
         # chmod +x /mnt/root/tmp/initrd.0/linuxrc
    
        copy your new file system into the image:
         # (cd /mnt/root/tmp/initrd.0; /mnt/root/bin/tar cf - . ) | \
               (cd /mnt/root/tmp/initrd.mnt-0; tar xf - )
    
        unmount and compress:
         # umount /mnt/root/tmp/initrd.mnt-0
         # /mnt/root/sbin/losetup -d /mnt/root/dev/loop5
         # gzip -9 < /mnt/root/tmp/initrd.img-0 > \
                 /mnt/root/boot/initrd.-2.2.19-6.2.7smp.img
    
    

    On the other hand, if you'd like to modify mkinitrd to handle this stuff at once, here are the important lines from the diff in the order they occur:

         -PATH=/sbin:$PATH
         +PATH=/mnt/root/sbin:$PATH
    
         -modulefile=/etc/conf.modules
         +modulefile=/mnt/root/etc/conf.modules
    
         -fstab="/etc/fstab"
         +fstab="/mnt/root/etc/fstab"
     
         -    echo "usage: `basename $0` [--version] [-v] [-f] [--ifneeded] [--preload ]" >&2
         +    echo "usage: `/mnt/root/bin/basename $0` [--version] [-v] [-f] [--ifneeded] [--preload ]" >&2
    
         -    echo "       (ex: `basename $0` /boot/initrd-2.2.5-15.img 2.2.5-15)" >&2
         +    echo "       (ex: `/mnt/root/bin/basename $0` /boot/initrd-2.2.5-15.img 2.2.5-15)" >&2
     
         -    if [ $(echo $modName | cut -b1) = "-" ]; then
         +    if [ $(echo $modName | /mnt/root/usr/bin/cut -b1) = "-" ]; then
    
         -       modName=$(echo $modName | cut -b2-)
         +       modName=$(echo $modName | /mnt/root/usr/bin/cut -b2-)
    
         -       fmPath=`(cd /lib/modules/$kernel; echo */$modName.o)`
         +       fmPath=`(cd /mnt/root/lib/modules/$kernel; echo */$modName.o)`
     
         -    if [ ! -f /lib/modules/$kernel/$fmPath ]; then
         +    if [ ! -f /mnt/root/lib/modules/$kernel/$fmPath ]; then
    
         -if [ ! -d /lib/modules/$kernel ]; then
         -    echo "/lib/modules/$kernel is not a directory." >&2
         +if [ ! -d /mnt/root/lib/modules/$kernel ]; then
         +    echo "/mnt/root/lib/modules/$kernel is not a directory." >&2
    
         -        modulefile=/etc/modules.conf
         +        modulefile=/mnt/root/etc/modules.conf
    
         -       scsimodules=`grep scsi_hostadapter $modulefile | grep -v '^[    ]*#' | LC_ALL=C sort -u | awk '{ print $3 }'`
         +       scsimodules=`grep scsi_hostadapter $modulefile | grep -v '^[    ]*#' | LC_ALL=C /mnt/root/bin/sort -u | /mnt/root/bin/awk '{ print $3 }'`
    
         -    if grep '^/dev/md' $fstab | grep -v noauto >/dev/null 2>&1 ; then
         +    if grep '^/mnt/root/dev/md' $fstab | grep -v noauto >/dev/null 2>&1 ; then
    
         -                         awk '{print $2}' | LC_ALL=C sort -u) ; do
         +                         /mnt/root/bin/awk '{print $2}' | LC_ALL=C /mnt/root/bin/sort -u) ; do
    
         -               echo "raid level $number (in /etc/raidtab) not recognized" >&2
         +               echo "raid level $number (in /mnt/root/etc/raidtab) not recognized" >&2
    
         -rootdev=$(awk '{ if ($2 == "/") { print $1; }}' $fstab)
         -if echo $rootdev | cut -d/ -f3 | grep loop >/dev/null; then
         -    key="^# $(echo $rootdev | cut -d/ -f3 | tr '[a-z]' '[A-Z]'):"
         +rootdev=$(/mnt/root/bin/awk '{ if ($2 == "/") { print $1; }}' $fstab)
         +if echo $rootdev | /mnt/root/usr/bin/cut -d/ -f3 | grep loop >/dev/null; then
         +    key="^# $(echo $rootdev | /mnt/root/usr/bin/cut -d/ -f3 | tr '[a-z]' '[A-Z]'):"
    
         -    loopDev=$(echo $line | awk '{print $3}')
         -    loopFs=$(echo $line | awk '{print $4}')
         -    loopFile=$(echo $line | awk '{print $5}')
         +    loopDev=$(echo $line | /mnt/root/bin/awk '{print $3}')
         +    loopFs=$(echo $line | /mnt/root/bin/awk '{print $4}')
         +    loopFile=$(echo $line | /mnt/root/bin/awk '{print $5}')
    
         -MNTIMAGE=/tmp/initrd.$$
         -IMAGE=/tmp/initrd.img-$$
         -MNTPOINT=/tmp/initrd.mnt-$$
         +MNTIMAGE=/mnt/root/tmp/initrd.$$
         +IMAGE=/mnt/root/tmp/initrd.img-$$
         +MNTPOINT=/mnt/root/tmp/initrd.mnt-$$
    
         -    if losetup /dev/loop$devnum $IMAGE 2>/dev/null ; then break; fi
         +    if /mnt/root/sbin/losetup /mnt/root/dev/loop$devnum $IMAGE 2>/dev/null ; then break; fi
     
         -LODEV=/dev/loop$devnum
         +LODEV=/mnt/root/dev/loop$devnum
     
         -inst /sbin/sash "$MNTIMAGE/bin/sash"
         -inst /sbin/insmod.static "$MNTIMAGE/bin/insmod"
         +inst /mnt/root/sbin/sash "$MNTIMAGE/bin/sash"
         +inst /mnt/root/sbin/insmod.static "$MNTIMAGE/bin/insmod"
     
         -    cp $verbose -a /lib/modules/$kernel/$MODULE $MNTIMAGE/lib
         +    cp $verbose -a /mnt/root/lib/modules/$kernel/$MODULE $MNTIMAGE/lib
    
         -    if [ ! -d /initrd ]; then
         -       mkdir /initrd
         +    if [ ! -d /mnt/root/initrd ]; then
         +       mkdir /mnt/root/initrd
    
         -    echo "losetup $rootdev /loopfs$loopFile" >> $RCFILE
         +    echo "/sbin/losetup $rootdev /loopfs$loopFile" >> $RCFILE
    
         -losetup -d $LODEV
         +/mnt/root/sbin/losetup -d $LODEV
    

    In our example, the new mkinitrd_2 (don't overwrite your old one!) will be executed:

         # sh /mnt/root/sbin/mkinitrd_2 \ 
                    /mnt/root/boot/initrd-2.2.19-6.2.7smp.img 2.2.19-6.2.7smp
         # sh /mnt/root/sbin/mkinitrd_2 \
                    /mnt/root/boot/initrd-2.2.19-6.2.7.img 2.2.19-6.2.7
    
    We make both of them because there's less we have to edit out of lilo.conf later. See section 4.

    4.  Rebuilding the boot map with lilo.

    So at this point, we've either got an initrd or we don't need one because we don't have a SCSI system. But we need to reconnect our boot map because our system isn't booting correctly. To do this, we'll need to use lilo.

    Check out what you have in your /mnt/root/boot directory, as shown above. Also, see which kernel vmlinuz and System.map point to. You'll be needing that information.

    Next, edit your lilo.conf file to make sure that it points to the write places. Use the command:

         # /mnt/root/bin/vi /mnt/root/etc/lilo.conf
    
    to perform this editing. For our example, the correct lilo.conf file would be:
         boot=/dev/sda
         map=/boot/System.map
         install=/boot/boot.b
         prompt
         timeout=50
         linear
         default=linux
    
         image=/boot/vmlinuz-2.2.19-6.2.7smp
                  label=linux
                  initrd=/boot/initrd-2.2.19-6.2.7smp.img
                  read-only
                  root=/dev/sda5
    
         image=/boot/vmlinuz-2.2.19-6.2.7
                  label=linux-up
                  initrd=/boot/initrd-2.2.19-6.2.7.img
                  read-only
                  root=/dev/sda5
    

    If you were building mkinitrd by hand, you'll want to remove the five lines which describe the image for the image that you didn't build.

    Next, execute the following command to map everything correctly:

         # /mnt/root/sbin/lilo -v -r /mnt/root
    

    5.  Unmounting the file system.

    At this point, you think you might be able to just dismount your file system and reboot- everything's done, right?

         # cd /
         # umount /mnt/root/usr
         # umount /mnt/root/
         umount: unable to unmount, /dev/sda5 busy
         #
    

    < Newsflash: > Try sync;sync;exit twice, and then try to umount /mnt/root, as mentioned in section -1. It might work. < End Newsflash >

    I haven't figured out a way out of this one. If your system doesn't claim this, then you're OK, go to step 7. In any event, you'll have to try to unmount it using init 0:

         # init 0
    

    but it really won't help. So, just reboot your system from the CD-ROM again, and enter expert mode as in step 1. You'll have to clean your root file system:

    6.  Cleaning the file system.

    Reboot the system using the CDROM, make the node as in step 2, and then clean the file system:

         # fsck /dev/sda5
    

    Just answer all the questions yes, because there's not much else you can do. Hope and pray that your system comes back together, then...

    7.  Reboot the system.

    At this point, the system should boot by itself off the hard drive.
     
     

      Good Luck! If you discover any errors or want to suggest any additions or amplifications, please send me a letter.