Classic running out of memory... huh? what? <long>

[Jarod Wilson]

On Jun 11, 2009, at 1:39 PM, Tom Buskey wrote:

>
>
> On Thu, Jun 11, 2009 at 1:27 PM, Thomas Charron <twaffle at gmail.com>  
> wrote:
> On Thu, Jun 11, 2009 at 1:22 PM, <bruce.labitt at autoliv.com> wrote:
> > Thomas Charron <twaffle at gmail.com> wrote on 06/11/2009 12:30:16 PM:
> >>   Another trick I've used when dealing with massive amounts of  
> data is
> >> to use 'fast' media, aka, flash instead of a hard drive.  Memory
> >> mapping to this sort of media worked well for me.
> > I would love to run a fast drive.  However, all I have available  
> is NFS on
> > gbit e-net.  My sustained file write rate is ~45MiB/sec.  (377.5e6
> > bits/sec)
> > That sounds moderately ok until one realizes the file size (one  
> chunk) is
> > ~4.5GiB.
>
>  You don't have physical access to the machine?  Even a USB can give
> better performance then that.
>
> Does it?

Typically, no. USB sucks horribly for disk I/O.

> USB 2.0 is 480 mbits/s which is probably close to 48 MB/s.
> 45 MB/s on gigabit ethernet isn't too bad.

(480 Megabit/second) * (1 Megabyte/8 Megabit) = 60 MB/s. But in  
practice, you'll rarely see much better than about 30MB/s, because all  
bus arbitration is done by the host cpu, which is grossly inefficient.  
FireWire or (even better) eSATA blows USB out of the water for  
external disk I/O performance.

-- 
Jarod Wilson
jarod at wilsonet.com