OT: PC Gigabit Throughput Question

Ric Werme ewerme at comcast.net
Thu Jun 14 20:30:11 EDT 2007


> What can you reasonably expect a pci gigabit card to give you for
> throughput? 

> Anyone have real world answers for that stuff?

When I worked on Tru64 Unix (yeah, I know, not Intel, not AMD, not Linux,
but was using PCI-??), I was able to saturate GbE with NFS traffic,
at least reading from the files cached on the server.  Some customers
could saturate it while reading from disk.

A 1500 byte Ethernet message is 12,000 bits, and hence only 12 usec of
wire time.  If you need full performance and can use "Jumbo Frames"
of 9,000 bytes (72 usec) so much the better.

I generally used UDP, despite some interesting problems with UDP
and fast links, and 48 KB read sizes, so protocol overhead would be
100+ bytes for NFS, 8 for UDP (only one of those per NFS read reply),
20 for IP and 14 for Ethernet (one for each of 6 IP fragments) which
translates into "not much".

If you are looking for that level of performance, one of my mantras
is "no good GbE switch."  Even some of the top-name, top-shelf
switches can only buffer a few milliseconds of data.  Instead of
cheap DRAM, they often use FIFOs, much smaller and more expensive.
I suspect most switches can handle most Windows and desktop Linux
systems, but if you can crank up TCP window sizes (I think I used
512KB) and see lots of retransmits, cast a jaundiced eye at the
switch.

This isn't really supercomputer stuff any more.  I needed 2-3 Alpha
CPUs to saturate the wire, so a single Core Duo ought to be plenty
adequate these days.

	 -Ric Werme


More information about the gnhlug-discuss mailing list