Advanced shell scripting question :-)

[Bob Bell]

On Thu, Sep 05, 2002 at 01:03:13PM -0400, Bob Bell <bobbell at zk3.dec.com> wrote:
> On Thu, Sep 05, 2002 at 10:04:17AM -0400, Steven W. Orr <steveo at syslang.net> wrote:
> > The problem is this: I expect that when data is written to a pipe, that 
> > the the order is preserved. Using this construct, the order is *not* 
> > preserved. So on a p that produces many kilobytes of spew, I get big 
> > chunks in the BOTH file which are from each seperate channel.
> 
>     I think what you are see is that p is writing to the pipe to "tee
> BOTH", and "tee ERR" is writing to the pipe to "tee BOTH".  Due to
> scheduling between the two processes, buffering, etc., ordering is not
> guaranteed.

    I think the crux of the issue here is that you want to both treat
stderr individually (to log to an error file), and treat stderr and
stdout collectively (to log to another file, and to the screen).  This
means that you need to duplicate the output of stderr (not duplicate
a file descriptor, but take 1 input and send the output two different
places) by itself, but also combine the two outputs while maintaining
order.

    The real problem here is that I don't believe that is possible
without having another program read the data once and reproduce it
twice.  When you do that, you throw ordering of output out the window.
Even using C and select(), I don't see this working.  Using C, it would
still be possible for the parent process to receive data on both stdout
and stderr pipes at the same time.  The C parent process has no way of
knowing in what order the data was written.

    I believe what you need here is kernel support (or possibly libc
support) for "write() on this file descriptor actually means write to
these two file descriptors"*.  Without that, you can never guarantee
proper ordering (unless you have support from the program generating the
output).  And I don't think we have that capability...


* Other possibilities here may include intercepting all write() calls,
  or forcing a context switch to your process so that you can handle
  each write() atomically.  But I believe you wind up in the same
  boat...

-- 
Bob Bell <bobbell at zk3.dec.com>
-------------------------------------------------------------------------
 "If you're going through hell, keep going."
   -- Winston Churchill