Admin horror stories (was: Replacing NIS)

[ken at flyingtoasters.net]

Back at UPS, they had departmental sysadmins for departmental servers --
these would be people who'd give, roughly, 3.5 days of the week to their
regular job, and 1.5 days to being a sysadmin (running backups, modifying
permissions -- nothing too rough).  We needed to give them some extra
space on one of their partitions, so we checked with the sysadmin that,
yes, she'd been running backups since time out of mind, and had them
on-hand.  So we wiped the partition, added the disks, re-created it... and
found that she'd been doing backups of the first two partitions, but not
the last one -- the very one we'd just deleted.

I can still remember that sinking feeling, 13 years later...

The whole department had to work overtime, over Christmas, doing data
entry to even attempt to come back to par with the accounting info that
went MIA.

Two different lessons were learned:
1) UPS phased out departmental sysadmins.
2) _I_ learned to never, ever, EVER trust a backup until you've restored
   something off of it.

-Ken

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Wed, Jun 18, 2003 at 04:23:49PM -0400, bscott at ntisys.com wrote:
>>   Anyone care to top that?  (I have no doubt that some could; I'm
>> curious if
>> anyone *will*.  <GRIN>)
>
> I didn't do it, and I didn't work there when it supposedly happend so
> I also didn't see it, but at that same job, someone told me a story
> about someone absent-mindedly leaning on the computer room's panic
> button...
>
> [Bear in mind, this data center contained 20 servers and 4 EMC arrays
> that supported operations for, among other things, half of our 800
> dialysis clinics across the US... and realize that if dialysis
> patients don't get their treatment on time, they could die.]
>
> - --
> Derek D. Martin
> http://www.pizzashack.org/
> GPG Key ID: 0xDFBEAD02
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.1 (GNU/Linux)
>
> iD8DBQE+8M49HEnASN++rQIRAtE8AJ9+ICEJHTPRWqSVr/7kZ5ffK5vySgCfQ9Jj
> z4NmA2WdZ8l/4eJbH6Ejk08=
> =YBvC
> -----END PGP SIGNATURE-----
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
>