Server-blockages (was: Delayed mail from GNHLUG?)

Joshua Judson Rosen rozzin at geekspace.com
Tue Dec 11 11:04:53 EST 2012


Ben Scott <dragonhawk at gmail.com> writes:
>
> On Tue, Dec 11, 2012 at 8:04 AM, Kyle Smith <askreet at gmail.com> wrote:
> > Anyone else just get roughly a weeks worth of mail from the list at once?
>
>   There was an issue[1] with the GNHLUG Internet server.  Mail was
> being queued for the list server to process, but wasn't going any
> further.  Once the problem was cleared, all the accumulated mail let
> lose at once.
>
> -- Ben
>
> [1] The log partition was full.[2]  This caused the list server to shut down.
> [2] Turns out somebody[3] forgot to set-up log rotation for the web
> sites it hosts.
> [3] That would be me.[4]
> [4] Sorry 'bout that.

Hee. :)

I once had a *bunch* of services on a server get stuck...
because someone had pressed the scroll-lock key on the console
the day before.

Syslog was relaying to the console, and the console being scroll-locked
meant that everything was just getting written to one of the kernel's
internal buffers--which was smartly setup to just make writes block
when it was full (rather than overflowing); blocking console-writes
meant that syslog writes blocked, which meant that any process
that was in any way doing anything that resulted in a call to syslog()
were blocked. Cron was blocked, for example. I couldn't even su or sudo
to investigate the problem, because PAM tried to syslog() the authentication
and session-start events.

I wasn't the one who pressed the scroll-lock key, but I did spend
the better part of a day unable to figure out WTF was going on.
I don't remember how we came to realise that that was the problem,
but we chuckled when we did. And then we un-scroll-locked the damned
console. ;)

-- 
"Don't be afraid to ask (λf.((λx.xx) (λr.f(rr))))."



More information about the gnhlug-discuss mailing list