FWIW: The bigger picture... Or why I have been asking a lot of questions lately...

[Bruce Labitt]

Drew Van Zandt wrote:
> Have you considered using a fast compression/decompression algorithm 
> before you transmit, one that isn't too computationally intensive for 
> either compression or decompression?  You won't get high compression 
> (factor of 10) as you might with slower ones, but if you get even a 
> factor of 2, you've just doubled the effective network speed.
>
I hadn't thought of this.  Typically a lot of the data is random, so 
compression doesn't help too much.  Or at least, that is what I 
thought.  If I'm wrong, I hope someone will cheerfully correct me :)

I did do an experiment that had curious results.  Instead of sending 
double precision binary data, I sent single precision or 'float'.  I was 
expecting to halve my transmission time, since it is half the size.  
Instead, there was only a 10-15% speed increase, not a 100%.  This 
result is telling me something, although at this time, I'm too brain 
dead to really ascertain what it really means.
> It appears to me that this would only gain you 50% or so, as the 
> obvious fast compression algorithms are only about twice as fast 
> (http://www.quicklz.com/bench.html) as Gigabit Ethernet's theoretical 
> speed, but it's worth consideration.  The decompression is faster than 
> the compression, which would hopefully result in a net win on 
> processing time on remote server.
>
> --DTVZ
>
Thanks for your suggestions.  Compression would certainly help out. 
Bruce
> On Sun, Oct 11, 2009 at 10:29 AM, Joshua Judson Rosen 
> <rozzin at geekspace.com <mailto:rozzin at geekspace.com>> wrote:
>
>     Bruce Labitt <bruce.labitt at myfairpoint.net
>     <mailto:bruce.labitt at myfairpoint.net>> writes:
>     >
>     > What I'm trying to do:  Optimizer for a radar power spectral
>     density problem
>     >
>     > Problem:  FFTs required in optimization loop take too long on
>     current
>     > workstation for the optimizer to even be viable.
>     >
>     > Attempted solution:  FFT engine on remote server to reduce overall
>     > execution time
>     >
>     > Builds client - server app implementing above solution.  Server uses
>     > OpenMP and FFTW to exploit all cores.
>     [...]
>     > Implements better binary packing unpacking in code.  Stuff works
>     >
>     > Nit in solution:  TCP transport time >> FFT execution time,
>     rendering
>     > attempted solution non-viable
>     >
>     > Researches TCP optimization: Reads countless papers on tcp
>     optimization
>     > techniques... Fails to find a robust solutions or methodology for
>     > problem.  Tries most techniques written in papers, only
>     realizing a 10%
>     > gain.  Not good enough.  Still needs to be faster
>     >
>     > Driven to more exotic techniques to reduce transport time.  Explores
>     > parallel sockets, other techniques
>     [...]
>     > Hey, that is my bigger picture...  Any and all suggestions are
>     > appreciated.  Undoubtedly, a few dumb questions will follow.  I
>     appear
>     > to be good at it.  :P  Maybe this context will help list subscribers
>     > frame their answers if they have any, or ask insightful questions.
>
>     Where exactly does NFS fit into this?
>
>     --
>     Don't be afraid to ask (Lf.((Lx.xx) (Lr.f(rr)))).
>     _______________________________________________
>     gnhlug-discuss mailing list
>     gnhlug-discuss at mail.gnhlug.org <mailto:gnhlug-discuss at mail.gnhlug.org>
>     http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>