fifosophical question ...
lijn...@embl-heidelberg.de
I am puzzled by fifo's, or named pipes. To paraphrase Magritte, cela n'est
pas une pipe, but then what is it? What can you use it for? Is it a just a
neat trick, or sometimes indispensable? Could you use it for
eg. self-decompressing files, or something? Why do they always seem to have
length 0? Do these pipe-thingies reside on the disk, and can you share them
through NFS? comp.unix.questions-4 says something obscure about using a named
pipe as a .plan file, but that isn't quite clear. I am just asking out of
curiosity, just wasting bandwidth and time 8-). And yes, i tried to rtfm,
but tfm does not explain a thing about fifos. Am i asking a faq ? or am i
asking too much ? ( 8 no that's not a smiley 8 question marks so far ...8-)
yes that was one. Should it be a faq ? (9) Anyone out there ? (10: i stop)
Thanks,
Philip
dame...@mmm.com
: Dear netters !
: Philip --
A FIFO is a pipe that acts like a file and a file that acts like a
pipe. Like a file, it has a name, so that it can be used by
non-related processes for Inter Process Communications. But it
acts like a pipe:
1. It must be opened at both ends before it comes open;
2. Reads and writes block until both ends are open;
3. It blocks when full;
It is very useful for scheduling; requests are written (an atomic
operation) to the open fifo, and a second process reads the fifo to
get its work. Lp uses it.
Fifos can be used with vi to provide advanced editing functionality.
For instance, I use a program that supplies boilerplate based upon
a few characters and the language being used (an if in a shell script
expands to if [ test ];then ~;else ~;fi , while an if in a 'C'
program becomes if (!) { ~ } else { ~ } (supply your own newlines).
I developed a program that had drop down menus called from within
vi.
Dan Mercer Applications + Plus
Reply To: dame...@mmm.com "The Mad Pedant"
======================================================================
About a year ago I told the following joke which got a strong positive
reaction from only 1 in 5 (all males my age - 40+) and shrugs from the
rest. I wonder if it is more relevant now?
"What does Mogadishu mean in Somali?" Answer - "Saigon"
Jim Kissel
: Dear netters !
: I am puzzled by fifo's, or named pipes. To paraphrase Magritte, cela n'est
: pas une pipe, but then what is it?
A fifo is a First In First Out file (stack/object etc.) You cannot seak
on it. You can write to it up to a limit of about 5K (may be more now
but this was the limit in Ver 7 & Sys III). The data you write to it
can be read in the byte order that you wrote it.
: What can you use it for?
The named pipe was one of the first IPC (Inter Process Communication)
facilities available on Unix. (I know Ver 7 had the multiplexed file
system but it wasn't very wide spread) It pre-dates Shared memory and Queues
Consider
ls -lt|head
This pipes the output of the ls command into the head command. The
pipe used is not a named pipe but behaves exactly like a named pipe
would. You could get the same result with something like the following
commands.
mknod -p my_named_pipe
head -t < my_named_pipe&
ls -lt > my_named_pipe
The syntax may be incorrect but the ideas are exactly what the shell
does when you use the '|' command/pragma/object .....
: Is it a just a
: neat trick, or sometimes indispensable? Could you use it for
: eg. self-decompressing files, or something?
It depends if you write multi-process applications. If you don't, you
don't need IPC. If you do write multi-process applications, pipes can
be used for communication between processed. Named pipes allow two
non-related (don't have the same parent) processes to have a rendezvous
point. (yes I know you can do it with sockets, but when I started
programming on Unix, local area networks didn't exist). The first process
to open the pipe can attemp to write. A failure indicates that no one
is ready to read. You can use two pipes to provide bi-directional
communications.
: Why do they always seem to have
: length 0?
If you haven't written anything to them, there is not anything in them,
therefore length == 0.
: Do these pipe-thingies reside on the disk, and can you share them
: through NFS?
All pipe-thingies are part of the file system. Named pipes have an
external/viewable mark on the file system.
No, the last time I looked, NFS did not support pipes.
: comp.unix.questions-4 says something obscure about using a named
: pipe as a .plan file, but that isn't quite clear. I am just asking out of
: curiosity, just wasting bandwidth and time 8-). And yes, i tried to rtfm,
: but tfm does not explain a thing about fifos. Am i asking a faq ? or am i
: asking too much ? ( 8 no that's not a smiley 8 question marks so far ...8-)
: yes that was one. Should it be a faq ? (9) Anyone out there ? (10: i stop)
: Thanks,
: Philip
Pipes are the oldest (I think) inter process communication technology in
the Unix operating system. I first seen Named Pipes in Sys III c 1981.
There are other and in many cases better IPC methods available in BSD/
AIX/SYS V.4....... than named pipes.
-------------------------------------------------------------------------------
Jim Kissel Telephone +44 344 863 222
Siemens Nixdorf Information Systems 344 850 461 (Direct line)
Systems Development Group Fax +44 344 850 452
Nixdorf House Domain j...@sni.co.uk
Oldbury, Bracknell, Berkshire UUCP ....{ukc,athen}!sni!jlk
RG12 4FZ Great Britain
Have you noticed that no one talks about Pavlov's cat?
-------------------------------------------------------------------------------
Alvin W. Law
>lijn...@embl-heidelberg.de wrote:
>
>: Is it a just a
>: neat trick, or sometimes indispensable? Could you use it for
>: eg. self-decompressing files, or something?
>
>It depends if you write multi-process applications. If you don't, you
>don't need IPC. If you do write multi-process applications, pipes can
>be used for communication between processed. Named pipes allow two
>non-related (don't have the same parent) processes to have a rendezvous
>point. (yes I know you can do it with sockets, but when I started
>programming on Unix, local area networks didn't exist). The first process
>to open the pipe can attemp to write. A failure indicates that no one
>is ready to read. You can use two pipes to provide bi-directional
>communications.
There are programs out there which does not read from stdin/write to
stdout. They expect a specific filename from the command line. I
find named pipes extremely handy when the file I want to operate on
are hugh and I don't have diskspace to store the uncompressed version.
In that case, I'll simply:
Read from compressed file:
% some_program file=my_named_pipe &
% zcat some_file.Z > my_named_pipe
Write to compressed file:
% compress -c my_named_pipe > some_other_file.Z &
% some_other_program file=my_named_pipe
--
Alvin W. Law .............................................. Oracle Corporation
Senior Software Engineer ...................... 300 Oracle Parkway, Box 659306
Manufacturing Applications .......................... Redwood Shores, CA 94065
Email: al...@oracle.com ........... Voice: 1.415.506.3390 . Fax: 1.415.506.7299
Peter Steele
>A FIFO is a pipe that acts like a file and a file that acts like a
>pipe. Like a file, it has a name, so that it can be used by
>non-related processes for Inter Process Communications. But it
>acts like a pipe:
> 1. It must be opened at both ends before it comes open;
> 2. Reads and writes block until both ends are open;
What happens if two processes try to write to the same fifo
file when there is process in place to read from the pipe?
Both processes would go into a write-blocked state, but which
one gets to write into the fifo first when the processes
become unblocked?
> 3. It blocks when full;
When is a fifo considered to be full?
>Fifos can be used with vi to provide advanced editing functionality.
>For instance, I use a program that supplies boilerplate based upon
>a few characters and the language being used (an if in a shell script
>expands to if [ test ];then ~;else ~;fi , while an if in a 'C'
>program becomes if (!) { ~ } else { ~ } (supply your own newlines).
>I developed a program that had drop down menus called from within
>vi.
Are you saying you developed a program that provides vi with
drop menus? That would be something I'd like to see...
--
Peter Steele Unix Services Manager peter....@acadiau.ca
Acadia Univ., Wolfville, NS, Canada B0P 1X0 902-542-2201 Fax: 902-542-4364
dame...@mmm.com
: dame...@mmm.com writes:
: >A FIFO is a pipe that acts like a file and a file that acts like a
: >pipe. Like a file, it has a name, so that it can be used by
: >non-related processes for Inter Process Communications. But it
: >acts like a pipe:
:
: > 1. It must be opened at both ends before it comes open;
: > 2. Reads and writes block until both ends are open;
:
: What happens if two processes try to write to the same fifo
: file when there is process in place to read from the pipe?
: Both processes would go into a write-blocked state, but which
: one gets to write into the fifo first when the processes
: become unblocked?
Whoever was first on the queue. And his write will be fulfilled, even
if it blocks the queue, guaranteeing the atomicity of the write.
For instance, if the first write was 10K, and the PIPSIZ = 8K, the
firts 8K would be written, the pipe would block until read, then the
next 2K would be written, then the other process would become
unblocked. It is important, when multiple processes write to the
same pipe, that some form of protocol be used, even if it is only
line blocking (which can be very effective).
:
: > 3. It blocks when full;
:
: When is a fifo considered to be full?
From man 2 write:
If the file being written is a pipe (or FIFO), the system-dependent
maximum number of bytes that it can store is given by PIPSIZ (defined
in <sys/inode.h>). The minimum value of PIPSIZ on any HP-UX system is
8192. When writing a pipe, the following conditions apply:
If the O_NDELAY or O_NONBLOCK file status flag is set:
If nbyte is less than or equal to PIPSIZ and sufficient room
exists in the pipe or FIFO, the write() succeeds and returns
the number of bytes written;
If nbyte is less than or equal to PIPSIZ but insufficient
room exists in the pipe or FIFO, the write() returns having
written nothing. If O_NONBLOCK is set, -1 is returned and
errno is set to EAGAIN. If O_NDELAY is set, 0 is returned.
If nbyte is greater than PIPSIZ and the pipe or FIFO is
full, the write returns having written nothing. If
O_NONBLOCK is set, -1 is returned and errno is set to
EAGAIN. If O_NDELAY is set, 0 is returned.
If nbyte is greater than PIPSIZ, and some room exists in the
pipe or FIFO, as much data as fits in the pipe or FIFO is
written, and write() returns the number of bytes actually
written, an amount less than the number of bytes requested.
:
: >Fifos can be used with vi to provide advanced editing functionality.
: >For instance, I use a program that supplies boilerplate based upon
: >a few characters and the language being used (an if in a shell script
: >expands to if [ test ];then ~;else ~;fi , while an if in a 'C'
: >program becomes if (!) { ~ } else { ~ } (supply your own newlines).
:
: >I developed a program that had drop down menus called from within
: >vi.
:
: Are you saying you developed a program that provides vi with
: drop menus? That would be something I'd like to see...
: --
: Peter Steele Unix Services Manager peter....@acadiau.ca
: Acadia Univ., Wolfville, NS, Canada B0P 1X0 902-542-2201 Fax: 902-542-4364
Vi was only used as an editor within the program. The menus could be
used to browse data structures in a coredump and output text in a more
readable fashion. The user would hit a function key to get the menus
which would cause vi to write to an input pipe and then block on a
:r infifo
When the user was done prowling the menus and had selected his
function, the background process would write to info and, voila, it
wound up back in the editor for browsing. Their he user could mangle
the returned data into other commands also accessible through fifos.
Unfortunately, the code isn't mine to share and I doubt if it's even
being used anymore. The business unit it was developed for having
fallen on hard times.
--
Christoph Badura
>For instance, if the first write was 10K, and the PIPSIZ = 8K, the
>firts 8K would be written, the pipe would block until read, then the
>next 2K would be written, then the other process would become
>unblocked.
This isn't guaranteed. In fact, it doesn't work as you describe on
most Unix Versions.
--
Christoph Badura --- b...@flatlin.ka.sub.org --- +49 721 606137
Personally, I don't care whether someone is cool enough to quote Doug
Gwyn--I only care whether Doug Gwyn is cool enough to quote. -- Larry Wall
dame...@mmm.com
I was describing an ordinary write, of course. If you are suggesting
that ordinary writes can be interleaved, that would violate the
atomicity of the write. Different rules do exist if O_NDELAY or
O_NONBLOCK file status flag is set. To excerpt from HP-UX man pages
write (2)...
If the file being written is a pipe (or FIFO), the system-dependent
maximum number of bytes that it can store is given by PIPSIZ (defined
in <sys/inode.h>). The minimum value of PIPSIZ on any HP-UX system is
8192. When writing a pipe, the following conditions apply:
If the O_NDELAY or O_NONBLOCK file status flag is set:
If nbyte is less than or equal to PIPSIZ and sufficient room
exists in the pipe or FIFO, the write() succeeds and returns
the number of bytes written;
If nbyte is less than or equal to PIPSIZ but insufficient
room exists in the pipe or FIFO, the write() returns having
written nothing. If O_NONBLOCK is set, -1 is returned and
errno is set to EAGAIN. If O_NDELAY is set, 0 is returned.
If nbyte is greater than PIPSIZ and the pipe or FIFO is
full, the write returns having written nothing. If
O_NONBLOCK is set, -1 is returned and errno is set to
EAGAIN. If O_NDELAY is set, 0 is returned.
If nbyte is greater than PIPSIZ, and some room exists in the
pipe or FIFO, as much data as fits in the pipe or FIFO is
written, and write() returns the number of bytes actually
written, an amount less than the number of bytes requested.
If the O_NDELAY and O_NONBLOCK file status flags are clear:
The write() always executes correctly (blocking as
necessary), and returns the number of bytes written.
I believe this i sthe same way it ran on the NCR SYS V systems I ran
on in my previous job. We considered using FIFO's in our current
project as an IPC mechanism (used sockets instead) and ran extensive
tests (mega-megabytes under all load conditions) to make sure the
software agreed with the manpage (not always guaranteed :-) ). We
were satisfied that no interleaving could occur.
Alan Watson
>Christoph Badura (b...@flatlin.ka.sub.org) wrote:
>>In <1993Sep9.2...@mmm.mmm.com> dame...@mmm.com writes:
>>>For instance, if the first write was 10K, and the PIPSIZ = 8K, the
>>>firts 8K would be written, the pipe would block until read, then the
>>>next 2K would be written, then the other process would become
>>>unblocked.
>
>>This isn't guaranteed. In fact, it doesn't work as you describe on
>>most Unix Versions.
>
>that ordinary writes can be interleaved, that would violate the
>atomicity of the write. Different rules do exist if O_NDELAY or
>O_NONBLOCK file status flag is set. To excerpt from HP-UX man pages
Welcome to the reality of a million and one versions of UNIX. To
excerpt from the OSF/1 man pages (although it is the same on Ultrix and
also, from my reading, SunOS):
Write requests to a pipe (or FIFO) are handled the same as a regular
file with the following exceptions:
[...]
+ If the size of the write() request is less than or equal to the
value of the PIPE_BUF system variable, the write() function is
guaranteed to be atomic. The data is not interleaved with data
from other processes doing writes on the same pipe. Writes of
greater than PIPE_BUF bytes can have data interleaved, on
arbitrary boundaries, with writes by other processes, whether or
not O_NONBLOCK or O_NDELAY are set.
If you need (portable) atomic operations, don't use FIFOs and don't use
NFS.
dame...@mmm.com
: [...]
Ugh!!!! Must have been the influence of IBM ;-)
Christoph Badura
>Christoph Badura (b...@flatlin.ka.sub.org) wrote:
>: In <1993Sep9.2...@mmm.mmm.com> dame...@mmm.com writes:
>: >For instance, if the first write was 10K, and the PIPSIZ = 8K, the
>: >firts 8K would be written, the pipe would block until read, then the
>: >next 2K would be written, then the other process would become
>: >unblocked.
>: This isn't guaranteed. In fact, it doesn't work as you describe on
>: most Unix Versions.
>I was describing an ordinary write, of course.
I know. I was responding to precisely that case.
> If you are suggesting
>that ordinary writes can be interleaved, that would violate the
>atomicity of the write.
The atomicity of writes to pipes was never guaranteed for arbitrarily
large writes. It's usually only guaranteed for writes of up to PIPSIZ
bytes, but you won't find that explicitly stated in most man pages.
POSIX requires writes of up to PIPE_BUF[*] bytes to be atomic.
The relevant paragraph of you qoute form the write(2) man page is:
> If the O_NDELAY and O_NONBLOCK file status flags are clear:
> The write() always executes correctly (blocking as
> necessary), and returns the number of bytes written.
Note that this doesn't explicitly guarantee atomicity. It just say
that write() will write all the data it is told to write. And the bit
about "blocking as necessary" is a strong hint that the write is not
atomic. (Blocking translates to sleeping in the kernel and if more
than one process is sleeping on the same channel all of them will be
awakened and the order in which they are run is random.)
>I believe this i sthe same way it ran on the NCR SYS V systems I ran
>on in my previous job.
That's right.
> We considered using FIFO's in our current
>project as an IPC mechanism (used sockets instead) and ran extensive
>tests (mega-megabytes under all load conditions) to make sure the
>software agreed with the manpage (not always guaranteed :-) ). We
>were satisfied that no interleaving could occur.
The you were very lucky. To make writes of aribtrary sizes to pipes
atomic would require a major redesign of the relevant kernel modules
in System V.
[*] I can see how patchconf() can return the value "indetermined" for
_PC_PATH_BUF, which would give you atomic write to pipes and FIFOs for
arbitrarily large writes.