Linux, gobs of RAM, RAID and performance suckage...

[Paul Lussier]

This is bizarre.

We've got an NFS server with Dual 3Ghz Xeon CPUs as our NFS server
connected to a Winchester OpenSAN FC-based RAID array.  The array is a
single 1TB partition (unfortunately).

Before yesterday we were noticing lots of NFS drop-outs on the clients
(300+ of them) and we correllated this pretty much to the backups
(amanda).  The theory was that local disk I/O was beating out
nfs-client requests.

We also noticed that our memory utilization was through the roof.  We
had 2GB of PC2300, ECC, DDR, Registered RAM.  That RAM was averaging
the following usage patterns:

 active     - 532M
 inactive   - 1.2G
 unused     -  39M
 cache      - 1.3G
 slab cache - 255M
 swap cache -   6M
 apps       -  78M
 buffers    - 350M
 swap       -  11M

We were topping out our memory usage and occasionally dipping into
swap space on disk.

Yesterday we added 2GB of RAM and our memory utilization now looks like this:

 active     - 793M
 inactive   - 2.3G
 unused     - 213M
 cache      - 2.9G
 slab cache - 194M
 swap cache -   2M
 apps       -  71M
 buffers    - 313M
 swap       - 4.5M

So, it appears we really only succeeded in doubling the cache
available to the system, and a little more than halving the amount of
swap that was getting touched.

However, now when backups are run, the system becomes completely
unresponsive from an NFS client perspective, and the load average
skyrockets (e.g. into the 40s!).

Does anyone have any ideas ?  I'm at a complete loss on this one.

Thanks.
-- 
Seeya,
Paul
--
Key fingerprint = 1660 FECC 5D21 D286 F853  E808 BB07 9239 53F1 28EE

A: Yes.                                                               
> Q: Are you sure?                                                    
>> A: Because it reverses the logical flow of conversation.           
>>> Q: Why is top posting annoying in email?