iptables out of memory?

[Alan Johnson]

On Fri, Jan 30, 2009 at 9:23 PM, Alan Johnson <alan at datdec.com> wrote:

> On Fri, Jan 30, 2009 at 6:23 PM, Kevin D. Clark <kevin_d_clark at comcast.net
> > wrote:
>
>> > I also saw high load average at times of high %si, so I had chaulked it
>> up
>> > to a work-station grade processor not being able to handle a lot of
>> context
>> > switching.  Now, I've just cleared out iptables back to the default
>> handful
>> > of rules, and I see the %si back down to the usual <3%.  So, I'm
>> guessing
>> > that each packet comes in causes a system interrrupt and the more rules
>> in
>> > iptables, the more time it takes to process each interrupt.  I can't be
>> sure
>> > form these observations though, because the disk wait (%wa) also goes
>> > through the roof when I clear out iptables.
>>
>> I find this to be puzzling, because I do not believe that iptables
>> operates in interrupt context.  The interrupts should be serviced as
>> quickly as they always are.  The only way that I could envision %si
>> going up like this would be if there was some strange bug that
>> prevented the interrupt handler from putting the skb on the queue (or
>> dropping them on the floor...) while iptables was processing your
>> gigantic ruleset.
>
>
> Interesting.  Like I said, could just be disk IO coming to the fore-front
> as more log lines are written.  Logs along won't do it, but the system was
> backed up a bit on mail queue (was >170K messages at one point) plus I am
> doing a massive delete of old mail, so disk is going N-V-T-S nuts right
> now.  Starting about now, there is a giant archive process kicking off,
> which will also skew the disk wait for a few days.  Give me a few days,
> maybe a couple weeks, and when things settle down on this machine (if they
> ever do) I'll let you know what happens.
>

Finally getting around to wrapping this up.  Anxiety induced insomnia has
some benefits. ;-)

So, a number of things were at play here.  I have since learned that
reiserfs (not reiser4, which I have yet to use) is a total sloth on write
performance.  So, while unlimited inodes is a good thing for a Maildir FS
with a lot of little files, the trade off is not worth it.  I have since
learned hot to jack up the inodes when creating and ext3 fs, so I'll be
going back to that if I ever get a chance.  All that said, reiserfs
certainly didn't help this situation, but long term monitoring proved that
disk IO was definitely not at fault here.

I am now 100% sure that the high %si is directly tied to the number of rules
in iptables, but I still won't pretend to understand why.  If is see a high
load and a high %si, I clear the rules, and bang, the %si drops to peanuts.

So, I have managed to cobbled together a small set of scripts that appear to
be pretty robust.  They still scan the mail log for spam-haus blocked IPAs
and stuff them into iptables, which does seem to reduce the load on the mail
system, but now they also clear out the rules ever few hours, so the number
rarely exceeds 5 digits.  I'll share if anyone is interseted, but it might
take me a while to put them together in a digestable presentation.  I expect
there are more elegant and smarter solutions out there anyway, given the
overhead of firewall rules.

Thanks to all for your help!

-- 
Alan Johnson
alan at datdec.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.gnhlug.org/mailman/private/gnhlug-discuss/attachments/20090620/f68b157a/attachment.html