Classic running out of memory... huh? what? <long>

bruce.labitt at autoliv.com bruce.labitt at autoliv.com
Thu Jun 11 13:22:02 EDT 2009


Thomas Charron <twaffle at gmail.com> wrote on 06/11/2009 12:30:16 PM:

>   valgrind
> 
>   It's what for diner.  :-D
> 
> 
>   What tests did you perform using valgrind?  The 'simple' running of
> it will just look at things like memory leaks, however, if your
> cleaning up after you run, it won't always see that as being an error.
>  Have you tried using it in massive mode?

Not yet.  I'll look into it.
> 
>   http://valgrind.org/docs/manual/ms-manual.html
> 
>   The only other suggestion I would have would be to potentially store
> data in memory mapped files.  Not as fast as having them resident, but
> not as slow as having to directly access the file.
> 
>   Another trick I've used when dealing with massive amounts of data is
> to use 'fast' media, aka, flash instead of a hard drive.  Memory
> mapping to this sort of media worked well for me.
> 
I would love to run a fast drive.  However, all I have available is NFS on
gbit e-net.  My sustained file write rate is ~45MiB/sec.  (377.5e6 
bits/sec)
That sounds moderately ok until one realizes the file size (one chunk) is 
~4.5GiB.

>   But I have to say, 32 gigs resident memory sounds like a metric
> arseload of data.  Perhaps it's best to look at what your keeping and
> how.  How big is your data set?
> 

The end dataset is even bigger.  I have to worry about stuff fitting on 
disk.
I just got a 1TB disk.  It will last me about 10 runs.  (~100GB files) 
Good
thing they are cheap.

> -- 
> -- Thomas


******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an <br>electronic signature and/or legally binding agreement in the absence of an <br>express statement or Autoliv policy and/or procedure to the contrary.<br>This E-mail and any attachments hereto are Autoliv property and may contain legally <br>privileged, confidential and/or proprietary information.<br>The recipient of this E-mail is prohibited from distributing, copying, forwarding or in any way <br>disseminating any material contained within this E-mail without prior written <br>permission from the author. If you receive this E-mail in error, please <br>immediately notify the author and delete this E-mail.  Autoliv disclaims all <br>responsibility and liability for the consequences of any person who fails to <br>abide by the terms herein. <br>
******************************



More information about the gnhlug-discuss mailing list