Tuning NFS (and Gbit Enet ?)

[Mark Komarinski]

Michael ODonnell wrote:
> Using the old config files (which worked OK on the old HW)
> on the new HW, we're seeing some depressing NFS throughput
> numbers.  For example, writes of large files from client to
> server are down around ~450KB/S when /etc/{fstab,exports}
> entries look, respectively, like this on client and server:
>
>    serverBox:/filesysOnServer /mountPointOnClient nfs bg,soft,intr,nodev,nfsvers=3,rsize=32768,wsize=32768,tcp,sync 0 0
>
>    /filesysOnServer *(rw,no_subtree_check,insecure,sync)
>
> Throughput immediately improved to ~10MB/S when I experimentally
> changed them to be thus:
>
>    serverBox:/filesysOnServer /mountPointOnClient nfs defaults 0 0
>
>    /filesysOnServer *(rw,async)
>   
Don't use sync unless it's read-only or you don't otherwise care about
the data.  sync makes sure that data is written to disk before returning
an all clear.  async doesn't.  So there's a possible case where data
gets sent, gets lost, and you never know about it.
> ...which sucks less but, of course, we expect much better from
> Gbit Enet - three times that, or so.
>
> Since it's been a while since I've fiddled with this stuff
> I'll publicly beg for advice (or dope-slaps) here in hopes
> that everyone benefits.
>
> One feature we'll unfortunately not be able to make use of is
> Jumbo Frames since (we're annoyed to discover) it turns out
> that neither these NICs nor this switch can do that....  :-(
>
> It's easy to show that sustained xfer rates to the server's
> local disk are around 100MB/S from RAM and 60MB/S from another
> local disk so that's not the problem.
>
> IIRC there are also tweaks to the networking infrastructure
> itself (independent of NFS) that can come into play, maybe
> TCP buffer size?
>   
Check out an article from Aug 2005 Linux Magazine written by some
chowderhead for more on tuning NFS and GigE.  Free (as in beer)
registration required.

http://www.linux-mag.com/content/view/2191/

-Mark