Classic running out of memory... huh? what? <long>

[bruce.labitt at autoliv.com]

Thomas Charron <twaffle at gmail.com> wrote on 06/11/2009 01:27:01 PM:

> On Thu, Jun 11, 2009 at 1:22 PM, <bruce.labitt at autoliv.com> wrote:
> > Thomas Charron <twaffle at gmail.com> wrote on 06/11/2009 12:30:16 PM:
> >>   Another trick I've used when dealing with massive amounts of data 
is
> >> to use 'fast' media, aka, flash instead of a hard drive.  Memory
> >> mapping to this sort of media worked well for me.
> > I would love to run a fast drive.  However, all I have available is 
NFS on
> > gbit e-net.  My sustained file write rate is ~45MiB/sec.  (377.5e6
> > bits/sec)
> > That sounds moderately ok until one realizes the file size (one chunk) 
is
> > ~4.5GiB.
> 
>   You don't have physical access to the machine?  Even a USB can give
> better performance then that.

I do have physical access, but the system is in a locked rack in 
a limited access area.  It would be a pain to install a physical drive in 
the 
rack because there is no good place for the drive.

I do achieve burst writes of over 100MB/sec on NFS.  But sustained, it is 
around 
45MB/sec.  Will a standard USB drive achieve 480Mbits/sec?  That is USB2 
spec, 
right?  480e6 vs 377e6, hmm.  I bet it won't do 480Mbits sustained. 
(For a 10 GB file write) However, I just might try it for the heck of it!

> 
> 
>   Any ability to preprocess the files?  I don't know what kind of data
> your talking about here, but multipass processing of the data could
> result in less memory usage.
> 

I already multipass the data.  It is possible that I f'd up something... 
Maybe at least a half a dozen times, heh.

-Bruce

******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an <br>electronic signature and/or legally binding agreement in the absence of an <br>express statement or Autoliv policy and/or procedure to the contrary.<br>This E-mail and any attachments hereto are Autoliv property and may contain legally <br>privileged, confidential and/or proprietary information.<br>The recipient of this E-mail is prohibited from distributing, copying, forwarding or in any way <br>disseminating any material contained within this E-mail without prior written <br>permission from the author. If you receive this E-mail in error, please <br>immediately notify the author and delete this E-mail.  Autoliv disclaims all <br>responsibility and liability for the consequences of any person who fails to <br>abide by the terms herein. <br>
******************************