Filter stdout and stderr Together
Scott Wegner
I would like to write a script that filters lines from both stdout and
stderr, and output the lines in the same order as they were originally
submitted. I've tried a few times using some complex stream
redirection and "grep -v ...", although I can't seem to get the
results I want *in order*. Has anybody done this before? Essentially
I want a shell script with the following syntax:
myscript command params
where myscript is the script I'm writing. command and params is what
should be executed. Output from the command should come on stdout and
stderr as expected, but with particular lines filtered out. My first
attempt was something along the lines of:
#!/usr/bin/env bash
MYFILTER="uninteresting output"
((($@) | grep -v "$MYFILTER") 3>&1 1>&2 2>&3 | grep -v "$MYFILTER")
2>&1 1>&3 3>&2
However, this was outputting text in the wrong order. Anybody know
what I'm doing wrong?
Scott
Dan Stromberg
If you don't mind losing the distinction between stdout and stderr,
easiest is probably to:
eval "$@" 2>&1 | grep -v "$MYFILTER"
If you do mind losing the distinction, it gets harder. In that case, you
might end up writing in more of a general purpose language like python
(using the powerful subprocess module) or similar, and reading from the
two file descriptors in a select loop or with two threads.
Stephane CHAZELAS
[...]
> ((($@) | grep -v "$MYFILTER") 3>&1 1>&2 2>&3 | grep -v "$MYFILTER")
> 2>&1 1>&3 3>&2
>
> However, this was outputting text in the wrong order. Anybody know
> what I'm doing wrong?
You're misunderstanding how x>&y works.
x>&y makes x point to the same resource as that pointed to by y.
It does a dup2(y, x).
$@ should almost never be left unquoted. You need to ensure that
$MYFILTER is not taken as a grep option.
As someone else alread answered:
"$@" 2>&1 | grep -ve "$MYFILTER"
or
eval "$@" 2>&1 | grep -v -- "$MYFILTER"
for the arguments to be taken as a shell command line.
But note that most commands behavior change with regards to
output buffering when their output is no longer a terminal. So
you might see a different output order from when you run the
command without the filtering (typically, you'd see the stderr
output (not buffered) before the stdout output (line-buffered
when stdout is a terminal but buffered by blocks when not).
--
Stéphane
Scott Wegner
Hmm, so I guess it's going to be more difficult than I expected.
Ideally, I would like to keep stdout and stderr separate.
Unfortunately, I'm not too familiar with python scripting. Does
anybody have a quick python script that would satisfy what I'm looking
for?
Scott
Barry Margolin
<52c49eb5-b57e-4926...@k37g2000hsf.googlegroups.com>,
Scott Wegner <sweg...@gmail.com> wrote:
> Hi All,
>
> I would like to write a script that filters lines from both stdout and
> stderr, and output the lines in the same order as they were originally
> submitted. I've tried a few times using some complex stream
> redirection and "grep -v ...", although I can't seem to get the
> results I want *in order*. Has anybody done this before? Essentially
> I want a shell script with the following syntax:
If the command you're running uses stdio, the problem is likely that the
default is to buffer stdout when it's not writing to a terminal, but
stderr is not buffered. So stderr messages will often jump ahead of
stdout messages, compared to how they look when you use the program
interactively.
--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***