Shell tips and tricks (was: cat pipe)

[Ben Scott]

On 10/5/07, Steven W. Orr <steveo at syslang.net> wrote:
> sort -k 9 <(ls -l /bin) <(ls -l /usr/bin) <(ls -l /usr/X11R6/bin)
> # Lists all the files in the 3 main 'bin' directories, and sorts by filename.
> # Note that three (count 'em) distinct commands are fed to 'sort'.

  Hmmm.  Interesting.  Useful, too.

  Hmmm!  This effectively lets you concatenate *processes*!  Or
rather, their output.  In other words, say I wanted to combine the
output of disparate operations into one, so I can sort or grep the
common output.  This is common enough with commands which (either
through arbitrary limitations or complexity of operation) cannot
simply take multiple arguments.  For example, say "foo" can only take
one argument:

	cat <(foo Fred) <(foo Wilma) <(foo Pebbles) | grep ERROR

  Here I'm invoking foo on three Flintstones, but I'm only interested
in the output if there is an error message.  (Note that I'm also using
cat to concatenate.  Wonder of wonders.)

  Without this, I'd have to invoke grep multiple times, or play games
with temp files, right?

> diff <(command1) <(command2)    # Gives difference in command output.

  Ohhhh.  Another useful idiom!

> tar cf >(bzip2 -c > file.tar.bz2) $directory_name
> # Tar up a dir and compress the output (Yeah, I know, just use j opt.)

  Yah, bad example.  Hmmmm.  Perhaps process substitution is less
useful for output than input.  Makes some sense, because after all,
you can easily combine inputs (concatenation), but the same principle
does not apply to output.  And one would normally just use a pipe.

  However, for some command which writes multiple distinct output
streams to named files, it might still be useful for post-processing.
Something like:

	baz >(sort > data_file) >(grep -v blah | sort > activity_log )

  Here, we have "baz", which does something complicated, and writes
two very different output streams to named arguments.  The first is
the actually process data, while the second is a detailed activity
log.  (Standard error is already taken for serious errors.)  We want
to sort the main output before putting it in the file.  For the
activity log, we want to ignore "blah", so we filter it before
sorting.

  Good stuff!

  Do others here have additional shell tips and tricks?  You know, the
kind of thing that you don't see others using much, so you remain
unaware of it, then when you discover it, you find it very useful, and
keep thinking about how much other people should be using it.  :)

  (The distinctions being, it's not just an esoteric trick you never
use in real-life, and it's also not something everybody already uses.)

-- Ben