LVM problem

[Ben Scott]

On Dec 18, 2007 9:36 AM, Dan Coutu <coutu at snowy-owl.com> wrote:
> Well, I have my response from Red Hat ... Make sure you have full backups of
> everything, reinstall Linux, and restore your files from backups.

  "Take off and nuke the site from orbit.  It's the only way to be sure."

  I suspect they're more afraid of the unknowns.  The current system
is a bit of a question mark.  A total reload will provide assurance
the system config is intact and known.  That's the safer route in
their book.  Of course, they're not the ones who have to rebuild the
system.... :-(

> Here is the output of various display commands:

  Which all seem to indicate that the volume group named "VolGroup00"
is not working.

  You did say this system was running, right?

  I'm wondering just how screwed up things are.  If what LVM is
reporting is right, there shouldn't be a system to run...

  Have you rebooted since the borkage occurred?  If not, I suggest
avoiding doing so.  The system may not come back up.

  I'm especially worried that you have "holes" in your LVs that you
just haven't run into yet.

  If you have backups from before the borkage, I'd pull the media out
of the rotation pool, and keep them indefinitely, in case you discover
trouble later.

  At some point, once you're out of the woods, and after making new
backups, I'd suggest scheduling downtime for an fsck, and/or a
database or other application-level consistency check.  I know
downtime can be hard to come by, but... better to run the checks at 4
AM on a Sunday than 1:32 PM on a Thursday after the system crashes.

> You will note that nowhere in there is any mention of the problematic
> uuid. Also there is no mention of the physical volume sdb, only of sda2.

  That may or may not be a good thing.  It all hinges on whether any
extents from the PV that was on sdb got allocated to any LVs at some
point.

  If the LVs have extents mapped to the damaged PV on sdb, then
restoring the metadata of the damaged PV is what you want.

  If the LVs exist entirely on the PV on sda2, you may be better off
restoring the "VolGroup00" metadata from before the trouble happened.
Then just start over -- create the PV on "sda2" as if none of this
ever happened.

  Normally, "lvdisplay -m" will show you the LE-to-PE mapping.  But
according to the "lvdisplay -v" you posted, there isn't any such
mapping.  At all.

  Try the "--partial" switch to the various LVM commands.  According
to the man page, it will not allow modification of metadata, so it
should be safe.  There are also some words in there about re-creating
a missing PV and restoring LVM config that you may want to read.

-- Ben