Classic running out of memory... huh? what? <long>
bruce.labitt at autoliv.com
bruce.labitt at autoliv.com
Thu Jun 11 13:22:02 EDT 2009
Thomas Charron <twaffle at gmail.com> wrote on 06/11/2009 12:30:16 PM:
> valgrind
>
> It's what for diner. :-D
>
>
> What tests did you perform using valgrind? The 'simple' running of
> it will just look at things like memory leaks, however, if your
> cleaning up after you run, it won't always see that as being an error.
> Have you tried using it in massive mode?
Not yet. I'll look into it.
>
> http://valgrind.org/docs/manual/ms-manual.html
>
> The only other suggestion I would have would be to potentially store
> data in memory mapped files. Not as fast as having them resident, but
> not as slow as having to directly access the file.
>
> Another trick I've used when dealing with massive amounts of data is
> to use 'fast' media, aka, flash instead of a hard drive. Memory
> mapping to this sort of media worked well for me.
>
I would love to run a fast drive. However, all I have available is NFS on
gbit e-net. My sustained file write rate is ~45MiB/sec. (377.5e6
bits/sec)
That sounds moderately ok until one realizes the file size (one chunk) is
~4.5GiB.
> But I have to say, 32 gigs resident memory sounds like a metric
> arseload of data. Perhaps it's best to look at what your keeping and
> how. How big is your data set?
>
The end dataset is even bigger. I have to worry about stuff fitting on
disk.
I just got a 1TB disk. It will last me about 10 runs. (~100GB files)
Good
thing they are cheap.
> --
> -- Thomas
******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an <br>electronic signature and/or legally binding agreement in the absence of an <br>express statement or Autoliv policy and/or procedure to the contrary.<br>This E-mail and any attachments hereto are Autoliv property and may contain legally <br>privileged, confidential and/or proprietary information.<br>The recipient of this E-mail is prohibited from distributing, copying, forwarding or in any way <br>disseminating any material contained within this E-mail without prior written <br>permission from the author. If you receive this E-mail in error, please <br>immediately notify the author and delete this E-mail. Autoliv disclaims all <br>responsibility and liability for the consequences of any person who fails to <br>abide by the terms herein. <br>
******************************
More information about the gnhlug-discuss
mailing list