Bash Stream a file through multiple programs at once


Example

This counts the number of lines in a big file with wc -l while simultaneously compressing it with gzip. Both run concurrently.

tee >(wc -l >&2) < bigfile | gzip > bigfile.gz

Normally tee writes its input to one or more files (and stdout). We can write to commands instead of files with tee >(command).

Here the command wc -l >&2 counts the lines read from tee (which in turn is reading from bigfile). (The line count is sent to stderr (>&2) to avoid mixing with the input to gzip.) The stdout of tee is simultaneously fed into gzip.