GRUB & RAID have me Stumped

[Paul Lussier]

Hi all,

I have set up a 4-way mirror of my OS partions.  I have 4 400GB SATA drives
partitioned thusly:

  # fdisk -l /dev/sda

  Disk /dev/sda: 400.0 GB, 400088457216 bytes
  255 heads, 63 sectors/track, 48641 cylinders
  Units = cylinders of 16065 * 512 = 8225280 bytes

     Device Boot    Start         End      Blocks   Id  System
  /dev/sda1             1         262     2104514+  fd  Linux raid autodetect
  /dev/sda2           263         524     2104515   fd  Linux raid autodetect
  /dev/sda3           525       48641   386499802+   5  Extended
  /dev/sda5           525         626      819314+  fd  Linux raid autodetect
  /dev/sda6           627         888     2104514+  82  Linux swap / Solaris
  /dev/sda7           889         901      104422   fd  Linux raid autodetect
  /dev/sda8           902        1554     5245222   fd  Linux raid autodetect
  /dev/sda9          1555        2729     9438187   fd  Linux raid autodetect
  /dev/sda10         2730        2991     2104514+  83  Linux
  /dev/sda11         2992       48641   366683624+  da  Non-FS data

Everything is RAID-1, and the arrays are configured like this:

  /dev/md0 devices=/dev/sdd1,/dev/sdc1,/dev/sdb1,/dev/sda1
  /dev/md1 devices=/dev/sdd2,/dev/sdc2,/dev/sdb2,/dev/sda2
  /dev/md2 devices=/dev/sdd5,/dev/sdc5,/dev/sdb5,/dev/sda5
  /dev/md3 devices=/dev/sdd7,/dev/sdc7,/dev/sdb7,/dev/sda7
  /dev/md4 devices=/dev/sdd8,/dev/sdc8,/dev/sdb8,/dev/sda8
  /dev/md5 devices=/dev/sdd9,/dev/sdc9,/dev/sdb9,/dev/sda9

All pretty straight forward and simple.  However, I can't seem to get
GRUB to co-operate with this view of the world.

In order to get to this point, I had to follow several steps:

 1. Boot of an existing installed configuration on /dev/sda1
 2. Re-partition /dev/sd[bcd] as shown above
 3. Configure RAID-1 sets across /dev/sd[bcd] and a "missing" drive to
    be added later.
 4. Reboot, telling grub that / is now on /dev/sdb1.
 5. Repartition /dev/sda to be as mentioned above.
 6. Add /dev/sda[125789] into the appropriate RAID-1 sets.

In theory, after I've completed step 6, I should be able to reboot.
Thus far, all attempts to do so have resulted in a cursor blinking
around the screen and finally a small ascii smiley-face in the lower
left corner of the screen, put there, I'm sure, by GRUB or mdadm
specifically to mock me :)

At this point however, I've tried several different things to convince
GRUB that it should really boot off /dev/sda1.  In this process, not
only have I discovered that I seem to not fully understand GRUB, but I
stumbled upon something which I believe to either be really wrong, or
an indication that I also completely misunderstand RAID-1!

  # df
  Filesystem           1K-blocks      Used Available Use% Mounted on
  /dev/md0               2071352   1137960    828172  58% /
  tmpfs                  1030468         0   1030468   0% /dev/shm
  /dev/md4               5162728    155972   5006756   4% /var
  /dev/md5               9288792     46488   9242304   1% /var/log
  /dev/sdb10             2071416     32828   1933364   2% /index1
  /dev/sdc10             2071416     32828   1933364   2% /index2
  /dev/sdd10             2071416     32828   1933364   2% /index3

  # mdadm -D /dev/md0
  /dev/md0:
  [...]
    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1

  # mount /dev/sda1 /mnt
  # df
  Filesystem           1K-blocks      Used Available Use% Mounted on
  /dev/md0               2071352   1137960    828172  58% /
  tmpfs                  1030468         0   1030468   0% /dev/shm
  /dev/md4               5162728    155972   5006756   4% /var
  /dev/md5               9288792     46488   9242304   1% /var/log
  /dev/sdb10             2071416     32828   1933364   2% /index1
  /dev/sdc10             2071416     32828   1933364   2% /index2
  /dev/sdd10             2071416     32828   1933364   2% /index3
  /dev/sda1              2071352   1137952    828180  58% /mnt

  # ls -l /boot/grub/menu.lst /mnt/boot/grub/menu.lst
  -rw-r--r--  1 root root 3705 Jun 15 15:56 /boot/grub/menu.lst
  -rw-r--r--  1 root root 3190 Jun 15 14:52 /mnt/boot/grub/menu.lst

14:52 is likely when /dev/sda1 was added into /dev/md0.  I changed
/boot/grub/menu.lst yesterday at 15:56.  So it appears that the RAID
set has not or does not sync immediately?  Or, for that matter, within
an 18+ hour period?

Can anyone explain what I'm missing here?

Thanks.
-- 

Seeya,
Paul