Recommended PCI gigabit ethernet card? OT: PC Gigabit Through put Question

[Jon 'maddog' Hall]

On Thu, 2007-06-14 at 22:16 -0400, Paul Lussier wrote:
> "Tom Buskey" <tom at buskey.name> writes:
> 
> > On top of that, if hdparm says timed disk writes are around 40MB, what
> >> could you see for sustained download speeds? Maybe a static cached
> >> webpage could saturate a gig connection, sustained 5 gig http download
> >> couldn't right?
> >>
> >> Anyone have real world answers for that stuff?
> >
> > What if you're downloading to RAM disk?
> 
> What if you're building a router?  The traffic never hits the disk, so
> drive performance is irrellevent here.


For each individual node, the speed of the bus and how fast the packet
can be transferred to the disk is important, but for the total bandwidth
of the wire, it is nice to have it as fast as possible while still
preserving things like reliability, reasonable physical length of the
wire for that speed, etc.

If I remember correctly, after a packet comes across, the controller is
supposed to wait some period of time before grabbing the wire again.
This allows some other controller time to grab the wire.  So even a
10Gbit wire is not really going to transfer data at a full 10Gbits
continuously.

I remember a case where Sun was beating Digital's (and other vendors)
pants off on ETHERNET transfers.  Then we did a study of a Sun system on
the wire, and found out that they were not waiting this entire "period"
between packet transfers, so would grab the wire a disproportionate
amount of the time.  When they corrected this "problem", their ETHERNET
capability dropped back to normal.

Digital used to talk about "balance" of a system.  Balancing CPU power,
with size of memory, size of cache, speed of disks and bus.  Putting a
super-fast CPU on a bus that could not deliver the data meant that you
were wasting a lot of power.  Likewise a super-fast ETHERNET controller.

AFAIK it was accepted and anticipated that an ETHERNET controller was
never going to effectively move wire speed at any given time onto or off
of the disk.  That is one of the reasons why there are so many "copy to"
moves inside of the kernel, but it did mean that if you sent the bits
over the wire at "wire speed" the controller would have half a chance of
receiving them correctly and making sure they went where they were
supposed to go, then free up the wire for someone else to use.

Maybe engineering expectations have changed since the days of 10Mbit
ETHERNET, but I do remember having those discussions with the
engineering staff.

And I agree that if you want to have the fastest path to your disk,
having a bus that can support it is a place to start, no matter what
ETHERNET controller you have.

md