output redirection with process substitution asynchronous?

[pjodrr]

Hello,

how can I prefix every line of output of some command with a
timestamp? I thought like this:

$ exec 3> >(while read line; do echo "$(date): $line"; done)
$ seq 4 >&3
Friday, December 4, 2009 4:20:29 PM MET: 1
$ Friday, December 4, 2009 4:20:29 PM MET: 2
Friday, December 4, 2009 4:20:29 PM MET: 3
Friday, December 4, 2009 4:20:29 PM MET: 4

please note that the prompt ($) returns before the command completes.
Why is the
process substitution asynchronous?
Does anybody know of a better way to accomplish this?

Thanks,

Peter

[DennisW]

This should be in gnu.bash rather than gnu.bash.bug

Would this work for you?

while read line; do echo "$(date): $line $((num++))"; done

[pk]

pjodrr wrote:

What's wrong with

seq 4 | while read line; do echo "$(date): $line"; done

[DennisW]

It works for me. Does it not for you? If you're asking why not do it,
then the answer is "why call an external program unnecessarily?".

Sorry, by the way, I missed what you were doing with the file
descriptor on my first read. What is it that you're trying to
accomplish? Are you doing this only to number the lines or is either
seq or the while loop a stand-in for something else?

[pjodrr]

On Dec 4, 7:46 pm, DennisW <dennistwilliam...@gmail.com> wrote:
>
> This should be in gnu.bash rather than gnu.bash.bug

oh, you are right, it's not a bug yet

> Would this work for you?
>
> while read line; do echo "$(date): $line $((num++))"; done

ah sorry, I used the command "seq" just as an example, it could
be any other command that produces output, should have probably
written like:

$ exec 3> >(while read line; do echo "$(date): $line"; done)

$ <some command that produces some output> >&3

Friday, December 4, 2009 4:20:29 PM MET: 1
$ Friday, December 4, 2009 4:20:29 PM MET: 2
Friday, December 4, 2009 4:20:29 PM MET: 3
Friday, December 4, 2009 4:20:29 PM MET: 4

thanks,

Peter

[pjodrr]

Hi

On Dec 4, 7:58 pm, pk <p...@pk.invalid> wrote:
> What's wrong with
>
> seq 4 | while read line; do echo "$(date): $line"; done

it creates a subshell, "seq" was just an example, sorry for
the confusion, it could be any other command, and it
should run in the current shell.

thanks,

Peter

[pjodrr]

Hello,

On Dec 4, 8:18 pm, DennisW <dennistwilliam...@gmail.com> wrote:
> It works for me. Does it not for you? If you're asking why not do it,
> then the answer is "why call an external program unnecessarily?".
>
> Sorry, by the way, I missed what you were doing with the file
> descriptor on my first read. What is it that you're trying to
> accomplish? Are you doing this only to number the lines or is either
> seq or the while loop a stand-in for something else?

the seq was only an example for demonstration.
here is another example that shows what I mean:

$ exec 3> >(while read line; do echo "tag: $line"; done)
$ seq 4 >&3
tag: 1
tag: 2
tag: 3
tag: 4

$ exec 3> >(while read line; do echo "$(date): $line"; done)
$ seq 4 >&3

$ Sat Dec 5 10:11:25 CET 2009: 1
Sat Dec 5 10:11:25 CET 2009: 2
Sat Dec 5 10:11:25 CET 2009: 3
Sat Dec 5 10:11:25 CET 2009: 4

while in the first example the prompt returns after the
command completes, the prompt returns immediately
in the second example.

thanks for your attention,

Peter

[DennisW]

Your example here:

$ exec 3> >(while read line; do echo "tag: $line"; done)
$ seq 4 >&3

just executes too quickly to exhibit this behavior. Try this and it
will do it, too:

$ exec 3> >(while read line; do for i in {1..10000}; do :; done; echo
'.'; done)
$ seq 4 >&3

I think the thing to remember is that doing this is like running
something in the background with "&". So, yes, it's going to be
asynchronous.

[pk]

pjodrr wrote:

> Hi
>
> On Dec 4, 7:58 pm, pk <p...@pk.invalid> wrote:
>> What's wrong with
>>
>> seq 4 | while read line; do echo "$(date): $line"; done
>
> it creates a subshell

uh...where do you think your original

>(while read line; do echo "$(date): $line"; done)

runs?

[pjodrr]

in my original example the "seq 4" runs in the current shell
while here the command runs in a subshell.

Peter

[pjodrr]

this results in:

malloc: ../bash/subst.c:4198: assertion botched
realloc: start and end chunk sizes differ
last command: exec 3> >(while read line; do for i in {1..10000}; do :;
done; echo '.'; done)
Aborting...Aborted

but I see your point of course.

> $ seq 4 >&3
>
> I think the thing to remember is that doing this is like running
> something in the background with "&". So, yes, it's going to be
> asynchronous.

thanks for your explanation. Is there any way to avoid this
behaviour?
There is probably no way to wait for the completion, is there?
Other than like this:

fifo=$(mktemp -u) || exit
mkfifo $fifo || exit
trap "rm -f $fifo" 0
trap exit 1 2 15
while read line; do echo "$(date): $line"; done < $fifo &
prefix_pid=$!
seq 4 > $fifo
wait $prefix_pid

Or how would you accomplish this?

Peter

[pk]

pjodrr wrote:

> in my original example the "seq 4" runs in the current shell
> while here the command runs in a subshell.

It would be nice if you explained what it is you're attempting to do, rather
than ask for a solution for what you're thinking would do that.

[Marc Herbert]

> pjodrr wrote:
> It would be nice if you explained what it is you're attempting to do, rather
> than ask for a solution for what you're thinking would do that.

To be honest that is the first thing he (tried to) do:

pjodrr wrote:
> how can I prefix every line of output of some command with a
> timestamp?

What is wrong with the following:

prefix_with_date ()
{
while read; do
printf '%s: %s\n' "$(date)" "$REPLY";
done
}

seq 4 | prefix_with_date
ls | prefix_with_date

[Greg Wooledge]

> pjodrr wrote:
> > how can I prefix every line of output of some command with a
> > timestamp?

Mark Herbert wrote:
> What is wrong with the following:
>
> prefix_with_date ()
> {
> while read; do
> printf '%s: %s\n' "$(date)" "$REPLY";
> done
> }
>
> seq 4 | prefix_with_date
> ls | prefix_with_date

On Sun, Dec 06, 2009 at 12:49:44AM -0800, pjodrr wrote:
> fifo=$(mktemp -u) || exit
> mkfifo $fifo || exit
> trap "rm -f $fifo" 0
> trap exit 1 2 15
> while read line; do echo "$(date): $line"; done < $fifo &
> prefix_pid=$!
> seq 4 > $fifo
> wait $prefix_pid
>
> Or how would you accomplish this?

What on earth is "this"? If you just want to prefix every line of some
command's output with a timestamp, and you're not willing to use
multilog (http://cr.yp.to/daemontools.html) or a perl one-liner, then
the bash function Mark gave is quite reasonable (maybe use "read -r"
instead of "read" to preserve backslashes).

The obvious disadvantage of doing this in bash is that bash has to
invoke an external date(1) command for every line it processes.
A perl one-liner could do it with built-in time functions, and of
course multilog is a single program.

It seems you have some other goal in mind, though, besides "prefix every
line of output of some command with a timestamp". What is the point of
all this command substitution and FIFO trickery?

[Marc Herbert]

Marc Herbert wrote:
> What is wrong with the following:
>
> prefix_with_date ()
> {
> while read; do
> printf '%s: %s\n' "$(date)" "$REPLY";
> done
> }
>
> seq 4 | prefix_with_date
> ls | prefix_with_date

Sorry I missed the fact that you want to run your commands in the current shell.

There are no real coroutines in shell. The current shell process does
not know how to schedule two pieces of code. So each time two
pieces of code communicate through a pipe they have to be run in two
*concurrent* processes (typically: subshells). The producer and
consumer of a pipe must run independently of each other. Whether you
are using process substitution or the more usual and portable pipe "|"
does not matter here: you need concurrency.

So at least one of: 1. your prefixer code, 2. your unknown command
has to run in a independent process, "asynchronous" with the current
shell.

Since you absolutely want your unknown commands to run in the current
shell, then it is your "prefixing" code that has to run in a
concurrent process.

Now I do not really see any other way to avoid the ugliness of
concurrently printing on stdout than to "wait" for your concurrent
prefixer to complete, more or less like you did.

A variant is to ask socat to handle the cleanup actions for you like
this:

prefix_with_date()
{
local P=/tmp/dataorig.$$
socat -u PIPE:${P} SYSTEM:'while read; do echo "$(date):\\ $REPLY"; done' &
socatPID=$!
until [ -e ${P} ]; do sleep 1; done
$@ > ${P}
wait $socatPID
}

prefix_with_date seq 5

[DennisW]

Would you care to comment on the coproc command in Bash 4?

[pk]

Marc Herbert wrote:

>> pjodrr wrote:
>> It would be nice if you explained what it is you're attempting to do,
>> rather than ask for a solution for what you're thinking would do that.
>
> To be honest that is the first thing he (tried to) do:
>
> pjodrr wrote:
>> how can I prefix every line of output of some command with a
>> timestamp?

I disagree. All the further changes in the requirements because creating a
subshell or being asynchronous is not acceptable etc. are not a goal in
themselves, but rather the indicators that he's trying to accomplish
something else.

[pjodrr]

Hi Marc,

thank you very much for your explanation! I feel understood and will
think about
using socat and if I want my script to depend on another program. But
at least
it became clear that the builtin process substitution is not the
solution for me.

Regards,

Peter

Peter

[Marc Herbert]

DennisW wrote :

> Would you care to comment on the coproc command in Bash 4?

I wish I could, but I know nothing about it. Anyone else?

pjodrr wrote :

> But at least it became clear that the builtin process substitution is
> not the solution for me.

Wait! Maybe it is. I found a much nicer way to wait on the prefixer
process, if you can live with the "flock" dependency:

prefix_with_date()
{
local somelock=$(tty)
$@ > >(flock $somelock -c 'while read; do printf "%s: %s\n" "$(date)" "$REPLY"; done ')
2>/dev/null flock $somelock :
}

prefix_with_date seq 10

Alternatives to flock might do; you got the idea.

[Marc Herbert]

pk a écrit :

>
> I disagree. All the further changes in the requirements because creating a
> subshell or being asynchronous is not acceptable etc. are not a goal in
> themselves, but rather the indicators that he's trying to accomplish
> something else.
>

I think he just want side-effects like in this recent example from Chris:

<http://thread.gmane.org/gmane.comp.shells.bash.bugs/13863/focus=13907>

Granted: if he was explaining in greater detail which side-effects he wants,
people might be able to suggest better alternatives.

[pjodrr]

I promise I'll do better the next time.

Peter

[pjodrr]

On Dec 8, 10:55 am, Marc Herbert <Marc.Herb...@gmail.com> wrote:
> DennisW wrote :
>
> > Would you care to comment on the coproc command in Bash 4?
>
> I wish I could, but I know nothing about it. Anyone else?

yeah, I tried that:

prefix_timestamp() {

while read line; do
echo "$(date): $line"
done
}

coproc prefix_timestamp
seq 10>&${COPROC[1]}
eval "exec ${COPROC[1]}>&-"
cat <&${COPROC[0]}
wait $COPROC_PID

but am not sure if I like that construct better.

Peter

[pjodrr]

Hello again,

I have to reply to my own post to correct it:

On Dec 8, 2:00 pm, pjodrr <pjo...@gmail.com> wrote:
> coproc prefix_timestamp
> seq 10>&${COPROC[1]}
> eval "exec ${COPROC[1]}>&-"
> cat <&${COPROC[0]}
> wait $COPROC_PID

replace this with:

{ coproc prefix_timestamp >&3 ; } 3>&1

seq 10>&${COPROC[1]}
eval "exec ${COPROC[1]}>&-"

wait $COPROC_PID

this is how i do it now, thanks for the discussion.

Peter