Packing/unpacking binary data in C - doubles, 64 bits

[bruce.labitt at autoliv.com]

gnhlug-discuss-bounces at mail.gnhlug.org wrote on 09/09/2009 03:28:04 PM:

> On Wed, Sep 9, 2009 at 2:57 PM, Kevin D. Clark
> <kevin_d_clark at comcast.net> wrote:
> > 4:  Sending binary doubles over the wire can be very complicated.
> 
>   Do tell.  :)
> 
> > 5:  If at all possible, consider encoding your protocol in ASCII text
> > rather than sending binary stuff on the wire.
> 
>   I know from past discussions (maybe on the PySIG list) that Bruce is
> dealing with hundreds of megabytes of data in an HPC scenario.  This
> is one of those situations where overhead matters.
> 

PySIG list - I think I may have been responsible for 90% of the message 
traffic for the last couple of months.  The learning curve was steep :)

>   (His hardware is kind of neat: He's got a big PC to store data, and
> a network node with a multi-core Cell processor (CPU architecture of
> the PlayStation 3) to do calculations.  The network node does the math
> real fast but is I/O poor.)

The multi-core Cell processor (the QS22 has two of them) is especially 
efficient at calculating FFTs using the open source software FFTW.  The 
good folks at FFTW managed to get IBM to contribute software to optimize 
Cell performance.  The Cell is about 20x to 100x times faster than a run 
of the mill 3.2 GHz P4.  It is especially efficient for problems that are 
larger than processor cache.  If the Intel processors actually had good 
memory bandwidth (compared to cache) maybe I wouldn't have used the Cell. 

As for I/O, I have a mere 1Gbit/sec private LAN between my PC and the 
Cell.  Large file transfers seem to run at 80% of maximum, which is very 
good.  The key is to send large blocks of data, not to send line by line. 
Line by line transfer is limited by fprintf.  fwrites of blocks are good. 
For this particular app, I need to transfer 160MB of binary data to the 
FFT server.  It takes <2 sec.  I have the server collapse the FFT data and 
only have to send back ~5K points.  So in effect, the app only takes the 
hit for the upload of data, not the download.

-Bruce


> 
> -- Ben
> 
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/


******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an <br>electronic signature and/or legally binding agreement in the absence of an <br>express statement or Autoliv policy and/or procedure to the contrary.<br>This E-mail and any attachments hereto are Autoliv property and may contain legally <br>privileged, confidential and/or proprietary information.<br>The recipient of this E-mail is prohibited from distributing, copying, forwarding or in any way <br>disseminating any material contained within this E-mail without prior written <br>permission from the author. If you receive this E-mail in error, please <br>immediately notify the author and delete this E-mail.  Autoliv disclaims all <br>responsibility and liability for the consequences of any person who fails to <br>abide by the terms herein. <br>
******************************