Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
How to source a fifo ?
30 views
Skip to first unread message
Timothy Madden
Nov 24, 2011, 1:23:02 PM11/24/11
to
Hello
I would like to feed input to multiple wget processes from my script, so
that the list of URLs I read from the database can be visited and
downloaded concurrently.
For this, I create the 20 fifos, start the 20 wget processes attached to
the read ends of the fifos, and open 20 output file descriptors in the
current shell process to output to the write end of the fifos.
To open the file descriptors I use
exec 14>"fifo_file_14"
in a loop, with 14 replaced by the current fifo number.
The problem is you can not use parameter substitution to replace 14 with
the current fifo or file descriptor number, because the shell will interpret
exec $no>"fifo_file_$no"
just as it will interpret
eval "exec $no" >"fifo_file_$no"
that is, after expansion it will attempt to `exec 14` instead of
redirecting file descriptor 14 from 14>"fifo_file_14".
So I want to output "exec $no>\"fifo_file_$no\"" to yet some other fifo,
and have my shell source that. Like this
mkfifo ./shell_script_line
echo "exec $no>fifo_file_$no" >/shell_script_line &
source ./shell_script_line.
And here is my problem, as the exec from the fifo is never seen and
executed.
I have seen this strange behavior in the following simple command
mkfifo ./sh_script_fifo
echo ls -l>./sh_script_fifo &
source ./sh_script_fifo
I would then expect the `source´ command to read the "ls -l" line
written in the fifo by the previous echo. Instead, I find that nothing
happens !.
If I use `cat ./sh_script_info´ instead of `source ...´ in the above
example, than everything goes well and I get a line of output reading
"ls -l".
Do you know why this is happening ? What am I missing about fifos ?
Thank you,
Timothy Madden
I would like to feed input to multiple wget processes from my script, so
that the list of URLs I read from the database can be visited and
downloaded concurrently.
For this, I create the 20 fifos, start the 20 wget processes attached to
the read ends of the fifos, and open 20 output file descriptors in the
current shell process to output to the write end of the fifos.
To open the file descriptors I use
exec 14>"fifo_file_14"
in a loop, with 14 replaced by the current fifo number.
The problem is you can not use parameter substitution to replace 14 with
the current fifo or file descriptor number, because the shell will interpret
exec $no>"fifo_file_$no"
just as it will interpret
eval "exec $no" >"fifo_file_$no"
that is, after expansion it will attempt to `exec 14` instead of
redirecting file descriptor 14 from 14>"fifo_file_14".
So I want to output "exec $no>\"fifo_file_$no\"" to yet some other fifo,
and have my shell source that. Like this
mkfifo ./shell_script_line
echo "exec $no>fifo_file_$no" >/shell_script_line &
source ./shell_script_line.
And here is my problem, as the exec from the fifo is never seen and
executed.
I have seen this strange behavior in the following simple command
mkfifo ./sh_script_fifo
echo ls -l>./sh_script_fifo &
source ./sh_script_fifo
I would then expect the `source´ command to read the "ls -l" line
written in the fifo by the previous echo. Instead, I find that nothing
happens !.
If I use `cat ./sh_script_info´ instead of `source ...´ in the above
example, than everything goes well and I get a line of output reading
"ls -l".
Do you know why this is happening ? What am I missing about fifos ?
Thank you,
Timothy Madden
Kaz Kylheku
Nov 24, 2011, 2:17:46 PM11/24/11
to
On 2011-11-24, Timothy Madden <termin...@gmail.com> wrote:
> The problem is you can not use parameter substitution to replace 14 with
> the current fifo or file descriptor number, because the shell will interpret
> exec $no>"fifo_file_$no"
> just as it will interpret
> eval "exec $no" >"fifo_file_$no"
> that is, after expansion it will attempt to `exec 14` instead of
> redirecting file descriptor 14 from 14>"fifo_file_14".
This is due to inadequate quoting. Consider the difference between
> The problem is you can not use parameter substitution to replace 14 with
> the current fifo or file descriptor number, because the shell will interpret
> exec $no>"fifo_file_$no"
> just as it will interpret
> eval "exec $no" >"fifo_file_$no"
> that is, after expansion it will attempt to `exec 14` instead of
> redirecting file descriptor 14 from 14>"fifo_file_14".
# redirection happens inside eval-ed code
eval "echo foo > bar"
and
# redirection happens now, evaled code is just the "echo foo"
eval "echo foo" > "bar"
> So I want to output "exec $no>\"fifo_file_$no\"" to yet some other fifo,
> and have my shell source that. Like this
them in variables.
> mkfifo ./shell_script_line
> echo "exec $no>fifo_file_$no" >/shell_script_line &
> source ./shell_script_line.
So you have a race condition between writing the shell script line and reading
it.
Where in the above logic do you ensure that the "source" command waits
for the echo to finish producing the shell script line?
> And here is my problem, as the exec from the fifo is never seen and
> executed.
> I have seen this strange behavior in the following simple command
> mkfifo ./sh_script_fifo
> echo ls -l>./sh_script_fifo &
> source ./sh_script_fifo
. ./sh_script_fifo
source is a Tcsh compatibility command in Bash.)
> I would then expect the `source´ command to read the "ls -l" line
> written in the fifo by the previous echo.
Generally if you want command C2 to use a file produced by C1, it looks like
this:
C1 > file
C2 file
Not like this:
C1 > file &
C2
Now this would work if you put in a wait:
C1 > file &
wait
C2
But now you have a UUPP: useless use of parallel processing.
Can you explain why you are putting the echo into the background? What do you
hope to achieve by farming out a simple echo to a parallel process?
Did you by chance want to quote the ampersand so it is part of the command
being run?
echo "command with ampersand &" > file
Anyway, eval will do what you want; you don't need to source a file.
Timothy Madden
Nov 24, 2011, 4:11:37 PM11/24/11
to
>
> Anyway, eval will do what you want; you don't need to source a file.
The quoting behavior for file descriptors used with redirection
> Anyway, eval will do what you want; you don't need to source a file.
operators is specified by POSIX, and is not the problem here as the
shell does behave as specified.
So no, there is no way for me to execute
exec $fd_no>>"fifo_file_$no"
and expect to get "fifo_file_$no" opened as file descriptor $fd_no.
Under all circumstances and any quoting, the shell will just try to exec
the $fd_no command, and will try to redirect the standard output for it.
This is why I am trying to first compose the string "exec
$fd_no>>fifo_file_$no", and then to source that string. But to source it
I need a file with the string. An attempt to just eval the string
(instead of sourcing it) will keep the redirection only for the duration
of the eval statement, and then at the next statement I no longer have
the file descriptor opened.
So as a file containing my string, to be source'd, I would like to use a
fifo. Lets call it "sh_script_fifo", and lets suppose I want to use `ls
-l` as the only line to be written/read on the fifo.
An attempt to write to a fifo that is not already open for reading by
some other (or even the same) process will block the write until such a
time that someone opens the same fifo for reading.
In a similar fashion, an attempt to read a fifo that is not already open
for writing by some other (or even the same) process will block the read
operation until such a time that someone opens the same fifo for writing.
Now I want to both read and write the fifo from the same script, which
means that whatever I try first is going to block my script.
This is why the first operation, write, with `echo ls -l
>./sh_script_fifo &` is terminated by the ampersand and put in the
background. This command will have to wait a little more for me to also
read from the pipe, until the `echo´ can write anything to the pipe.
I would expect this to happen really soon, as my next command is
source ./sh_script_fifo
which opens the fifo for reading and reads it.
However, something is wrong because nothing happens and nothing is read
from the fifo by the `source´ command, it just exits as if the fifo has
been closed immediately after open, with no data ever written.
Timothy Madden
Robert Bonomi
Nov 25, 2011, 12:18:14 PM11/25/11
to
In article <4ece8b8e$0$291$1472...@news.sunsite.dk>,
1) use 'lockfile' to protect the critical code,
2) write _all_ the exec commands to a tempfile, *not* a FIFO
3) source that single file.
4) remove the tempfile
5) remove the lockfile
Timothy Madden <termin...@gmail.com> wrote:
>Hello
>
>I would like to feed input to multiple wget processes from my script, so
>that the list of URLs I read from the database can be visited and
>downloaded concurrently.
>
>For this, I create the 20 fifos, start the 20 wget processes attached to
>the read ends of the fifos, and open 20 output file descriptors in the
>current shell process to output to the write end of the fifos.
>
>To open the file descriptors I use
> exec 14>"fifo_file_14"
>in a loop, with 14 replaced by the current fifo number.
>
The 'obvious' solution is to:
>Hello
>
>I would like to feed input to multiple wget processes from my script, so
>that the list of URLs I read from the database can be visited and
>downloaded concurrently.
>
>For this, I create the 20 fifos, start the 20 wget processes attached to
>the read ends of the fifos, and open 20 output file descriptors in the
>current shell process to output to the write end of the fifos.
>
>To open the file descriptors I use
> exec 14>"fifo_file_14"
>in a loop, with 14 replaced by the current fifo number.
>
1) use 'lockfile' to protect the critical code,
2) write _all_ the exec commands to a tempfile, *not* a FIFO
3) source that single file.
4) remove the tempfile
5) remove the lockfile
Timothy Madden
Nov 25, 2011, 4:10:29 PM11/25/11
to
On 25.11.2011 19:18, Robert Bonomi wrote:
> In article<4ece8b8e$0$291$1472...@news.sunsite.dk>,
> Timothy Madden<termin...@gmail.com> wrote:
[...]
> In article<4ece8b8e$0$291$1472...@news.sunsite.dk>,
> Timothy Madden<termin...@gmail.com> wrote:
>> To open the file descriptors I use
>> exec 14>"fifo_file_14"
>> in a loop, with 14 replaced by the current fifo number.
>>
>
> The 'obvious' solution is to:
> 1) use 'lockfile' to protect the critical code,
> 2) write _all_ the exec commands to a tempfile, *not* a FIFO
> 3) source that single file.
> 4) remove the tempfile
> 5) remove the lockfile
Thanks for the tip on lockfile, but what is wrong with a fifo ?
>> exec 14>"fifo_file_14"
>> in a loop, with 14 replaced by the current fifo number.
>>
>
> The 'obvious' solution is to:
> 1) use 'lockfile' to protect the critical code,
> 2) write _all_ the exec commands to a tempfile, *not* a FIFO
> 3) source that single file.
> 4) remove the tempfile
> 5) remove the lockfile
Thank you,
Timothy Madden
Geoff Clare
Nov 28, 2011, 8:52:35 AM11/28/11
to
Timothy Madden wrote:
> So no, there is no way for me to execute
> exec $fd_no>>"fifo_file_$no"
> and expect to get "fifo_file_$no" opened as file descriptor $fd_no.
>
> Under all circumstances and any quoting, the shell will just try to exec
> the $fd_no command, and will try to redirect the standard output for it.
>
> This is why I am trying to first compose the string "exec
> $fd_no>>fifo_file_$no", and then to source that string. But to source it
> I need a file with the string. An attempt to just eval the string
> (instead of sourcing it) will keep the redirection only for the duration
> of the eval statement, and then at the next statement I no longer have
> the file descriptor opened.
$ cat try.sh
> So no, there is no way for me to execute
> exec $fd_no>>"fifo_file_$no"
> and expect to get "fifo_file_$no" opened as file descriptor $fd_no.
>
> Under all circumstances and any quoting, the shell will just try to exec
> the $fd_no command, and will try to redirect the standard output for it.
>
> This is why I am trying to first compose the string "exec
> $fd_no>>fifo_file_$no", and then to source that string. But to source it
> I need a file with the string. An attempt to just eval the string
> (instead of sourcing it) will keep the redirection only for the duration
> of the eval statement, and then at the next statement I no longer have
> the file descriptor opened.
no=9
eval "exec $no>>file$no"
echo foo >&9
$ ls file9
ls: cannot access file9: No such file or directory
$ sh try.sh
$ cat file9
foo
--
Geoff Clare <net...@gclare.org.uk>
Robert Bonomi
Nov 30, 2011, 11:35:19 AM11/30/11
to
In article <4ed00444$0$287$1472...@news.sunsite.dk>,
"Experimental evidence" indicates there are problems using FIFOs in your
situation. <grin>
Writing anything more than a trivial amount of data to a pipe ('named' or
otherwise)is guaranteed to run into blocking problems when the 'writer'
gets 'far enough' (which -is- context-sensitive) ahead of the reader.
Using a tempfile eliminates those possible problems.
It also allows _detection_ of 'write' problems, and possible recovery
from that error.
Timothy Madden <termin...@gmail.com> wrote:
>On 25.11.2011 19:18, Robert Bonomi wrote:
>> In article<4ece8b8e$0$291$1472...@news.sunsite.dk>,
>> Timothy Madden<termin...@gmail.com> wrote:
>[...]
>>> To open the file descriptors I use
>>> exec 14>"fifo_file_14"
>>> in a loop, with 14 replaced by the current fifo number.
>>>
>>
>> The 'obvious' solution is to:
>> 1) use 'lockfile' to protect the critical code,
>> 2) write _all_ the exec commands to a tempfile, *not* a FIFO
>> 3) source that single file.
>> 4) remove the tempfile
>> 5) remove the lockfile
>
>Thanks for the tip on lockfile, but what is wrong with a fifo ?
Do you want 'something that works', or do you insist on 'do it your way'?
>On 25.11.2011 19:18, Robert Bonomi wrote:
>> In article<4ece8b8e$0$291$1472...@news.sunsite.dk>,
>> Timothy Madden<termin...@gmail.com> wrote:
>[...]
>>> To open the file descriptors I use
>>> exec 14>"fifo_file_14"
>>> in a loop, with 14 replaced by the current fifo number.
>>>
>>
>> The 'obvious' solution is to:
>> 1) use 'lockfile' to protect the critical code,
>> 2) write _all_ the exec commands to a tempfile, *not* a FIFO
>> 3) source that single file.
>> 4) remove the tempfile
>> 5) remove the lockfile
>
>Thanks for the tip on lockfile, but what is wrong with a fifo ?
"Experimental evidence" indicates there are problems using FIFOs in your
situation. <grin>
Writing anything more than a trivial amount of data to a pipe ('named' or
otherwise)is guaranteed to run into blocking problems when the 'writer'
gets 'far enough' (which -is- context-sensitive) ahead of the reader.
Using a tempfile eliminates those possible problems.
It also allows _detection_ of 'write' problems, and possible recovery
from that error.
Timothy Madden
Dec 12, 2011, 9:52:42 AM12/12/11
to
On 30.11.2011 18:35, Robert Bonomi wrote:
> In article<4ed00444$0$287$1472...@news.sunsite.dk>,
If the writer gets 'far enough', the system API write request will block
until the reader can consume some more data. I hear the write buffer
size for pipes is usually only 64k (for which reason some people prefer
think of pipes as having 'no buffering at all').
You are right: since 'my way' is for some reason broken, I will have to
get some other way that works, but I would still like to know what is
wrong with my case (like maybe I misunderstood some details and I am
doing something wrong).
I find out after posting in bash group that this is a bug with (really)
old versions of bash:
nntp://news.gmane.org/gmane.comp.shells.bash.bugs/17651
Thank you,
Timothy Madden
> In article<4ed00444$0$287$1472...@news.sunsite.dk>,
until the reader can consume some more data. I hear the write buffer
size for pipes is usually only 64k (for which reason some people prefer
think of pipes as having 'no buffering at all').
You are right: since 'my way' is for some reason broken, I will have to
get some other way that works, but I would still like to know what is
wrong with my case (like maybe I misunderstood some details and I am
doing something wrong).
I find out after posting in bash group that this is a bug with (really)
old versions of bash:
nntp://news.gmane.org/gmane.comp.shells.bash.bugs/17651
Thank you,
Timothy Madden
Robert Bonomi
Dec 16, 2011, 1:24:08 PM12/16/11
to
In article <4ee6153a$0$291$1472...@news.sunsite.dk>,
Timothy Madden <termin...@gmail.com> wrote:
[sneck]
when system memory was typically much smaller,
Like a mere 4k bytes, *or*less*.
One _cannot_ count on a FIFO having any specific amount of buffering.
"In theory", it shouldn't matter, 'if", or "how much" buffering there
is inside the O/S, but some kinds of 'look ahead' functions are well
known to have 'unexpected' and even 'unpredictable' behavior, when the
underlying 'device' is not a filesystem.
Timothy Madden <termin...@gmail.com> wrote:
[sneck]
>
>If the writer gets 'far enough', the system API write request will block
>until the reader can consume some more data. I hear the write buffer
>size for pipes is usually only 64k (for which reason some people prefer
>think of pipes as having 'no buffering at all').
The 'buffer size' for pipes is IMPLEMENTATION DEPENDENT. In 'days of old',
>If the writer gets 'far enough', the system API write request will block
>until the reader can consume some more data. I hear the write buffer
>size for pipes is usually only 64k (for which reason some people prefer
>think of pipes as having 'no buffering at all').
when system memory was typically much smaller,
Like a mere 4k bytes, *or*less*.
One _cannot_ count on a FIFO having any specific amount of buffering.
"In theory", it shouldn't matter, 'if", or "how much" buffering there
is inside the O/S, but some kinds of 'look ahead' functions are well
known to have 'unexpected' and even 'unpredictable' behavior, when the
underlying 'device' is not a filesystem.
Kaz Kylheku
Dec 16, 2011, 2:04:28 PM12/16/11
to
On 2011-12-16, Robert Bonomi <bon...@host122.r-bonomi.com> wrote:
> In article <4ee6153a$0$291$1472...@news.sunsite.dk>,
> Timothy Madden <termin...@gmail.com> wrote:
>
> [sneck]
>>
>>If the writer gets 'far enough', the system API write request will block
>>until the reader can consume some more data. I hear the write buffer
>>size for pipes is usually only 64k (for which reason some people prefer
>>think of pipes as having 'no buffering at all').
>
> The 'buffer size' for pipes is IMPLEMENTATION DEPENDENT. In 'days of old',
> when system memory was typically much smaller,
>
> Like a mere 4k bytes, *or*less*.
>
> One _cannot_ count on a FIFO having any specific amount of buffering.
> "In theory", it shouldn't matter, 'if", or "how much" buffering there
> is inside the O/S, but some kinds of 'look ahead' functions are well
You can't look ahead in a pipe, because it doesn't support seek. This is why
> In article <4ee6153a$0$291$1472...@news.sunsite.dk>,
> Timothy Madden <termin...@gmail.com> wrote:
>
> [sneck]
>>
>>If the writer gets 'far enough', the system API write request will block
>>until the reader can consume some more data. I hear the write buffer
>>size for pipes is usually only 64k (for which reason some people prefer
>>think of pipes as having 'no buffering at all').
>
> The 'buffer size' for pipes is IMPLEMENTATION DEPENDENT. In 'days of old',
> when system memory was typically much smaller,
>
> Like a mere 4k bytes, *or*less*.
>
> One _cannot_ count on a FIFO having any specific amount of buffering.
> "In theory", it shouldn't matter, 'if", or "how much" buffering there
> is inside the O/S, but some kinds of 'look ahead' functions are well
the errno ESPIPE exists.
This is not related to buffer size limitations.
But you can usually count on programs to behave properly with regard
to stdin and stdout being connected to pipes. There is a good
chance they were tested using pipes.
For programs that open files by name, all bets are off if
the input is suddenly connected to a pipe (not only via opening
a fifo, but possibly due to process substitution syntax in the shell.)
Usually, programs that open a file by name, but have the option to
read standard input (e.g. via the special name "-" or some such)
usually are well-behaved also.
The problem posed by the buffering limitations in a pipe is the threat
of deadlock when two pipes are used to communicate with a coprocess.
If you want to read the standard output of a process to which you
are sending output, you can deadlock due to the pipes becoming full
simultaneously.
If the pipe can buffer arbitrary amounts of unread data (and so
never blocks on write), the deadlock goes away.
This applies to anonymous pipes created by pipe() as well as named fifos.
Robert Bonomi
Dec 16, 2011, 2:20:54 PM12/16/11
to
In article <201112161...@kylheku.com>,
>But you can usually count on programs to behave properly with regard
>to stdin and stdout being connected to pipes. There is a good
>chance they were tested using pipes.
"Usually" is the operative word. And the OP's -entire- 'problem'
revolved around a case where things were *NOT* working 'as expected'.
>
[sneck]
>This applies to anonymous pipes created by pipe() as well as named fifos.
You would be advised to read the entire thread, And see what the issue is
that the OP was wrestling with.
Kaz Kylheku <k...@kylheku.com> wrote:
>On 2011-12-16, Robert Bonomi <bon...@host122.r-bonomi.com> wrote:
>> In article <4ee6153a$0$291$1472...@news.sunsite.dk>,
>> Timothy Madden <termin...@gmail.com> wrote:
>>
>> [sneck]
>>>
>>>If the writer gets 'far enough', the system API write request will block
>>>until the reader can consume some more data. I hear the write buffer
>>>size for pipes is usually only 64k (for which reason some people prefer
>>>think of pipes as having 'no buffering at all').
>>
>> The 'buffer size' for pipes is IMPLEMENTATION DEPENDENT. In 'days of old',
>> when system memory was typically much smaller,
>>
>> Like a mere 4k bytes, *or*less*.
>>
>> One _cannot_ count on a FIFO having any specific amount of buffering.
>> "In theory", it shouldn't matter, 'if", or "how much" buffering there
>> is inside the O/S, but some kinds of 'look ahead' functions are well
>
>You can't look ahead in a pipe, because it doesn't support seek. This is why
>the errno ESPIPE exists.
>
>This is not related to buffer size limitations.
"no shit sherlock" applies. I made no claim the behavior _was_ buffer-related.
>On 2011-12-16, Robert Bonomi <bon...@host122.r-bonomi.com> wrote:
>> In article <4ee6153a$0$291$1472...@news.sunsite.dk>,
>> Timothy Madden <termin...@gmail.com> wrote:
>>
>> [sneck]
>>>
>>>If the writer gets 'far enough', the system API write request will block
>>>until the reader can consume some more data. I hear the write buffer
>>>size for pipes is usually only 64k (for which reason some people prefer
>>>think of pipes as having 'no buffering at all').
>>
>> The 'buffer size' for pipes is IMPLEMENTATION DEPENDENT. In 'days of old',
>> when system memory was typically much smaller,
>>
>> Like a mere 4k bytes, *or*less*.
>>
>> One _cannot_ count on a FIFO having any specific amount of buffering.
>> "In theory", it shouldn't matter, 'if", or "how much" buffering there
>> is inside the O/S, but some kinds of 'look ahead' functions are well
>
>You can't look ahead in a pipe, because it doesn't support seek. This is why
>the errno ESPIPE exists.
>
>This is not related to buffer size limitations.
>But you can usually count on programs to behave properly with regard
>to stdin and stdout being connected to pipes. There is a good
>chance they were tested using pipes.
revolved around a case where things were *NOT* working 'as expected'.
>
[sneck]
>This applies to anonymous pipes created by pipe() as well as named fifos.
that the OP was wrestling with.
Geoff Clare
Dec 19, 2011, 8:38:42 AM12/19/11
to
Kaz Kylheku wrote:
> You can't look ahead in a pipe, because it doesn't support seek. This is why
> the errno ESPIPE exists.
On STREAMS-based pipes you can do a "read without consuming" operation
> You can't look ahead in a pipe, because it doesn't support seek. This is why
> the errno ESPIPE exists.
to look ahead (using ioctl() with I_PEEK). No seeking needed.
Of course, STREAMS-based pipes aren't that widely implemented - as far
as I know only SVR4-derived systems such as Unixware and Solaris have
them.
--
Geoff Clare <net...@gclare.org.uk>
0 new messages