[OT] Generator testing

[Ben Scott]

On Tue, Sep 8, 2009 at 8:03 PM, Dan Jenkins<dan at rastech.com> wrote:
> The unusual thing, to me, about it is that they test it in the middle of
> every week during normal work hours. First thing in the morning, the
> power in all the buildings goes off for about 20-60 seconds when they
> switch to the generator, and then does it again 5-10 minutes later when
> they switch back.

  I have little direct experience, but I can describe what I've been told:

  Generators can be tested in two ways, no-load and loaded.
Additionally, loaded testing can be real or simulated.

  A no-load test just tests the engine (prime mover) and spins all the
moving parts; it doesn't actually test power generating capability,
transfer switches, etc.  It is sometimes called "exercising".  It's
still a very useful thing to do, because the moving parts are usually
the most likely to fail, and they'll also tend to stick if you don't
run them regularly.

  A loaded test actually puts a load on the generator (duh).  A
simulated load puts a dummy load on the generator, to make sure it can
actually generate power, and that the wiring to the switch point is
good.  It doesn't test your live transfer capability, nor prove that
everything will work right "for real".  And obviously, a real load
test is what you're doing -- see if the generator can power the real
load under (supposedly) real conditions.

  The do-it-yourself automatic-generator-in-a-box kits I've seen for
home use advertise the capability to exercise themselves
automatically, once per week.  Loaded testing is entirely up to the
home owner.

  A big datacenter I took a tour of once (hi Ted!) had truck-sized
generators.  They did no-load tests every other week, IIRC.  They did
simulated load testing once a year, which involved the generator
service company trucking in a huge resistor bank.  I don't know if
they did any periodic real-load tests, but the trucking in of a dummy
load suggests not.

  I've worked in one or two other buildings that supposedly had backup
generator capability.

  In that limited experience, I've *NEVER* heard of shutting the whole
building down once per week to test the generator.  Talk about missing
the fscking point.  Unless your power utility is so unreliable that
they have outages more than weekly, you're actually better off without
that generator at all.

  Possibly relevant:

  The thing that selects "public utility power" vs "local generator"
is called a transfer switch.

  Cheaper transfer switches do a break-before-make, keeping everything
in isolation, but meaning you drop the load at least briefly during
the switch.  Cheap, safe, easy.

  There are also break-before-make transfer switches, which don't drop
the load, but also have to be sophisticated enough to keep Bad Things
from happening when both sources are connected at once.  You don't
want your generator feeding back into the grid, or the linemen trying
to fix your utility outage can be killed.  You also don't want your
generator trying to fight the utility for control of phase and
frequency, because your generator will lose.  How exactly these
switches accomplish this is pure magic to me.

  I imagine if you have the later kind of transfer switch, real-load
testing is easier.

  DISCLAIMER: Everything I say could be a total lie.

> The servers, all but one running Linux (to bring this
> slightly on-topic), are protected by UPSes, so they aren't directly
> affected.

  For most UPSes once encounters running computers, doing that every
week will significantly shorten the battery lifetime, and possibly
that of the power electronics as well.

> They have noticed it takes the network awhile to operate normally after
> this weekly event. Not surprisingly with a flood of DHCP requests and
> SMB network convergence happening when several hundred computers
> are turned on.

  Indeed.

  If you're not already doing so, set up WINS servers, and configure
your NetBIOS node type to 0x2 (WINS only, AKA P-node, AKA peer node).
This will at least significantly reduce broadcast traffic due to name
resolution.

  You can also disable NetBIOS browsing.  That will reduce traffic
from browser elections and browse list maintenance.  However, you'll
lose the ability to view active NetBIOS nodes -- i.e., "NET VIEW" and
"Network Neighborhood" will be empty.

> It seems wrong to me & to the maintenance director, however,
> the company which installed the equipment says this is normal operation.

  Talk to the company's competitors and see what they say.

> Someone involved with the project implied it is a legal requirement in
> some manner.

  Electrical and safety codes are a maze of twisty passages, all
similar but none alike.  A lot of stuff that someone says you can or
can't do can often be done in a different way to satisfy regulations.
A lot of it is open to interpretation, and so what matters is what you
and the relevant authorities can agree upon.  Quite often both sides
are right, or nobody is, etc.  None of that helps you directly, but
the point is that just because someone can find a way "why not"
doesn't mean that you can't find another way "this is how".

-- Ben