How reliable are named pipes?
Peter Olcott
named pipes?
Ian Collins
> Are there any reliability issues or other caveats with using
> named pipes?
In what context?
[c.p.t. removed again]
--
Ian Collins
Peter Olcott
"Ian Collins" <ian-...@hotmail.com> wrote in message
news:82726l...@mid.individual.net...
One respondent in another group said that named pipes are
inherently very unreliable and I think that he switched to
some sort of sockets. In any case he did not choose named
pipes for his IPC because of significant reliability issues
that he encountered.
Chris Friesen
> Are there any reliability issues or other caveats with using
> named pipes?
They work exactly as described.
Chris
Peter Olcott
"Chris Friesen" <cbf...@mail.usask.ca> wrote in message
news:lsOdnZNAxutl_iPW...@posted.sasktel...
The other respondent must have been referring to MS Windows
named pipes then, thanks.
Ian Collins
> "Ian Collins"<ian-...@hotmail.com> wrote in message
> news:82726l...@mid.individual.net...
>> On 04/ 9/10 04:58 AM, Peter Olcott wrote:
>>> Are there any reliability issues or other caveats with
>>> using
>>> named pipes?
>>
>> In what context?
>
> inherently very unreliable and I think that he switched to
> some sort of sockets. In any case he did not choose named
> pipes for his IPC because of significant reliability issues
> that he encountered.
Care to cite a reference?
--
Ian Collins
Peter Olcott
http://groups.google.com/group/microsoft.public.vc.mfc/msg/e1b2fc1b0ff39589?hl=en
Baho Utot
>
> "Ian Collins" <ian-...@hotmail.com> wrote in message
> news:8277c2...@mid.individual.net...
>> On 04/ 9/10 11:12 AM, Peter Olcott wrote:
>>> "Ian Collins"<ian-...@hotmail.com> wrote in message
>>> news:82726l...@mid.individual.net...
>>>> On 04/ 9/10 04:58 AM, Peter Olcott wrote:
>>>>> Are there any reliability issues or other caveats with
>>>>> using
>>>>> named pipes?
>>>>
>>>> In what context?
>>>
>>> One respondent in another group said that named pipes are
>>> inherently very unreliable and I think that he switched
>>> to
>>> some sort of sockets. In any case he did not choose named
>>> pipes for his IPC because of significant reliability
>>> issues
>>> that he encountered.
>>
>> Care to cite a reference?
>>
>> --
>> Ian Collins
>
>
http://groups.google.com/group/microsoft.public.vc.mfc/msg/e1b2fc1b0ff39589?hl=en
I think he means Linux...not microsoft it's well known microsoft is borked.
Barry Margolin
"Peter Olcott" <NoS...@OCR4Screen.com> wrote:
> "Ian Collins" <ian-...@hotmail.com> wrote in message
> news:8277c2...@mid.individual.net...
> > On 04/ 9/10 11:12 AM, Peter Olcott wrote:
> >> "Ian Collins"<ian-...@hotmail.com> wrote in message
> >> news:82726l...@mid.individual.net...
> >>> On 04/ 9/10 04:58 AM, Peter Olcott wrote:
> >>>> Are there any reliability issues or other caveats with
> >>>> using
> >>>> named pipes?
> >>>
> >>> In what context?
> >>
> >> One respondent in another group said that named pipes are
> >> inherently very unreliable and I think that he switched
> >> to
> >> some sort of sockets. In any case he did not choose named
> >> pipes for his IPC because of significant reliability
> >> issues
> >> that he encountered.
> >
> > Care to cite a reference?
> >
> > --
> > Ian Collins
>
> http://groups.google.com/group/microsoft.public.vc.mfc/msg/e1b2fc1b0ff39589
> ?hl=en
That message is almost incomprehensible. I can tell that he's saying
there are reliability problems, but not what they actually are.
--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
Peter Olcott
"Barry Margolin" <bar...@alum.mit.edu> wrote in message
news:barmar-EA6C6B....@62-183-169-81.bb.dnainternet.fi...
Yes, and I still could not get him to confirm that these
problems are in Windows or Linux/Unix, he seems to have
hinted the former.
Rainer Weikusat
A 'named pipe' on Windows is something completely different than a
named pipe on UNIX(*). Specifically a 'named pipe' on Windows is a
service endpoint which can be used for datagram-style or
bytestream-style communication among networked hosts using the
'standard', proprietary Microsoft networking protocols which grew
'organically' (this should really be 'carcinomically' :->) out of the
original DOS file and printer sharing support (OS/2 has/had a similar
facility). Not even Microsoft has (as far as I know) complete and
accurate documentation about that.
In contrast to this, a 'named pipe' on UNIX(*) is just that: A pipe
(unidirectional[*] bytestream-style communication channel for IPC on a
single computer) with a name (in the filesystem namespace). Basically,
this is just a memory buffer in the kernel which can be accessed by
doing an open(2) on the pathname of the pipe and processes can then
sequentially copy data into the buffer with write(2) and read data
sequentially (in FIFO order) with read(2).
[*] Traditionally, pipes were unidirectional. "On many
systems" (Linux being a notable exception), they are actually
bidirectional nowadays.
The main problem of FIFOs is that they are not suitable for anything
except 1:1 communication because there is neither a way to determine
which of possibly many processes wrote a particular sequence of bytes
nor one to send a reply to a specific recipient.
Casper H.S. Dik
>A 'named pipe' on Windows is something completely different than a
>named pipe on UNIX(*). Specifically a 'named pipe' on Windows is a
>service endpoint which can be used for datagram-style or
>bytestream-style communication among networked hosts using the
>'standard', proprietary Microsoft networking protocols which grew
>'organically' (this should really be 'carcinomically' :->) out of the
>original DOS file and printer sharing support (OS/2 has/had a similar
>facility). Not even Microsoft has (as far as I know) complete and
>accurate documentation about that.
Originally, they were actually buffered in the filesystem
>In contrast to this, a 'named pipe' on UNIX(*) is just that: A pipe
>(unidirectional[*] bytestream-style communication channel for IPC on a
>single computer) with a name (in the filesystem namespace). Basically,
>this is just a memory buffer in the kernel which can be accessed by
>doing an open(2) on the pathname of the pipe and processes can then
>sequentially copy data into the buffer with write(2) and read data
>sequentially (in FIFO order) with read(2).
> [*] Traditionally, pipes were unidirectional. "On many
> systems" (Linux being a notable exception), they are actually
> bidirectional nowadays.
>The main problem of FIFOs is that they are not suitable for anything
>except 1:1 communication because there is neither a way to determine
>which of possibly many processes wrote a particular sequence of bytes
>nor one to send a reply to a specific recipient.
For a systems with STREAMS, there's a connld module which creates
a new STREAM everytime the pipe is opened. (You get a pipe
which works like accept)
But without that, multiple messages from multiple processes can be
concatenated and they need to be picked apart.
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
Ersek, Laszlo
> The main problem of FIFOs is that they are not suitable for anything
> except 1:1 communication because there is neither a way to determine
> which of possibly many processes wrote a particular sequence of bytes
> nor [...]
I apologize for splitting hairs and parroting banalities, but if all
producers agree to write only messages not bigger than _POSIX_PIPE_BUF (==
512) bytes, then those messages will be atomic, in a portable way, and a
single consumer can parse the stream. This enables n-to-1 communication.
(Admittedly, not a very general kind.) Messages may carry a fixed length
header, designating the complete (or remaining) length of the message, and
identifying the sender.
lacos
Peter Olcott
"Rainer Weikusat" <rwei...@mssgmbh.com> wrote in message
news:87hbnkx...@fever.mssgmbh.com...
Unless the ProcessID is passed back and forth. In my case I
have many threads of one process that will communicate with
four processes. I was considering named pipes from the
threads to each of the four processes (depending on the
process priority of the job). To communicate back I would
have four more named pipes that communicate with the web
server's many threads. They would pass the ThreadID back and
forth.
Someone else proposed that I use some form of sockets.
Peter Olcott
"Casper H.S. Dik" <Caspe...@Sun.COM> wrote in message
news:4bbf1f6a$0$22940$e4fe...@news.xs4all.nl...
Fixed length messages with ThreadID and/or ProcessID.
Peter Olcott
"Ersek, Laszlo" <la...@caesar.elte.hu> wrote in message
news:Pine.LNX.4.64.10...@login01.caesar.elte.hu...
Yes, exactly what I am proposing.
Would there be any advantage to using some form of sockets
instead of named pipes?
Rainer Weikusat
> "Ersek, Laszlo" <la...@caesar.elte.hu> wrote: in message
>>> The main problem of FIFOs is that they are not suitable
>>> for anything except 1:1 communication because there is
>>> neither a way to determine which of possibly many
>>> processes wrote a particular sequence of bytes nor [...]
>>
>> I apologize for splitting hairs and parroting banalities,
>> but if all producers agree to write only messages not
>> bigger than _POSIX_PIPE_BUF (== 512) bytes, then those
>> messages will be atomic, in a portable way, and a single
>> consumer can parse the stream.
[...]
> Yes, exactly what I am proposing. Would there be any advantage to
> using some form of sockets instead of named pipes?
The disadvantage of using AF_UNIX sockets is that 'opening' would be
more complicated. Apart from that, there would be only advantages
- message size not limited to PIPE_BUF
- bidirectional
- if datagram-based sockets are used, neither framing nor
parsing are necessary to extract messages from a bytestrem
- kernel provides sender and recipient addresses automatically
- replies can use the same socket(s) as requests
Peter Olcott
Can this work for TCP sockets too?
Ian Collins
> "Rainer Weikusat"<rwei...@mssgmbh.com> wrote:
[formatting fixed]
>> The disadvantage of using AF_UNIX sockets is that 'opening' would be
>> more complicated. Apart from that, there would be only advantages
>>
>> - message size not limited to PIPE_BUF
>> - bidirectional
>> - if datagram-based sockets are used, neither framing nor
>> parsing are necessary to extract messages from a bytestrem
>> - kernel provides sender and recipient addresses automatically
>> - replies can use the same socket(s) as requests
>
> Can this work for TCP sockets too?
Can what work? There's quite a list there!
--
Ian Collins
Peter Olcott
Can I use TCP sockets as the kind of sockets that I am using
for IPC?
Ian Collins
[give up on reformatting]
> Can I use TCP sockets as the kind of sockets that I am using
> for IPC?
Well stream sockets are closer to pipes than datagram based ones. But
you do loose the advantages cited in Rainer's now horribly mangled list.
If you are sending small packets of data or using multi cast or a
connectionless setup, datagram based sockets make more sense.
So the answer is probably yes. But datagram based sockets may be a
better solution for your problem, assuming delivery is guaranteed.
--
Ian Collins
Bob Smith
> Are there any reliability issues or other caveats with using
> named pipes?
As one reply says "exactly as described", but there are few
things that are of note.
First, a named pipe protects the reader of the pipe. If the
reader does not keep up, it is the writer to the pipe that
get the EPIPE error. I use a fanout device to get around
this problem.
Second, the writer can write about 4K bytes before getting
blocked if the reader is not reading. This is usually not
a problem but was for some of my robotic control software.
I now use a proxy device for this.
The fanout and proxy drivers are not part of the kernel and
are available here: http://www.linuxtoys.org/usd/usd.tar.gz
I have an article that describes how to use them. Please
let me know if you would like a copy.
Bob Smith
Peter Olcott
"Bob Smith" <bsm...@linuxtoys.org> wrote in message
news:175497-...@mail.linuxtoys.org...
> Peter Olcott wrote:
>> Are there any reliability issues or other caveats with
>> using named pipes?
>
> As one reply says "exactly as described", but there are
> few
> things that are of note.
>
> First, a named pipe protects the reader of the pipe. If
> the
> reader does not keep up, it is the writer to the pipe that
> get the EPIPE error. I use a fanout device to get around
> this problem.
>
> Second, the writer can write about 4K bytes before getting
> blocked if the reader is not reading. This is usually not
> a problem but was for some of my robotic control software.
> I now use a proxy device for this.
That sounds dumb. Why not simply grow the queue length to
some arbitrary pre specified length?
Moi
> "Bob Smith" <bsm...@linuxtoys.org> wrote in message
> news:175497-...@mail.linuxtoys.org...
>> Peter Olcott wrote:
>>> Are there any reliability issues or other caveats with using named
>>> pipes?
>>
>> As one reply says "exactly as described", but there are few
>> things that are of note.
>>
>> First, a named pipe protects the reader of the pipe. If the
>> reader does not keep up, it is the writer to the pipe that get the
>> EPIPE error. I use a fanout device to get around this problem.
>>
>> Second, the writer can write about 4K bytes before getting blocked if
>> the reader is not reading. This is usually not a problem but was for
>> some of my robotic control software. I now use a proxy device for this.
>
> That sounds dumb. Why not simply grow the queue length to some arbitrary
> pre specified length?
Because that would consume precious kernel bufferspace.
In the normal case one *wants* the writer to block. Take for example the
classic chained pipeline of filters used by lp / lpr; the final part of the
chain is the (physical) printer, which is also the slowest part.
Having larger buffers would only result in the final pipe buffer becoming very big.
Better is to shift the congestion upstream: don' produce more output until
the printer can handle it. There is also the chance of the processes in the
pipeline to do some buffering themselves; in userspace, which is cheaper.
BTW: you can always produce a longpipe by creating two pipes with a buffering
process inbetween. The buffer is _again_ in userspace.
HTH,
AvK
Peter Olcott
"Moi" <ro...@invalid.address.org> wrote in message
news:213b3$4bc09722$5350c024$27...@cache110.multikabel.net...
My process can handle 100 transactions per second. I was
envisioning a FIFO at least this long. I guess that I have
to change my vision. It does seem to make sense that it is
done this way if the kernel has limited buffer space. Why
does the kernel have limited buffer space?
Moi
cat /dev/zero | dd | >dev/null
What would happen if the kernel would have unlimited bufferspace ?
Also; you are confusing "throughput" (tps) with chunk-size.
The above dd pipe easily handles 2 * 100 MB data transfer per second.
Every read and every write of these is atomic. (modulo sizeof buffer
or PIPE_MAX, say 512) That is a lot more than 100 tps.
AvK
Peter Olcott
No I am not. Because my process can handle 100 transactions
per second a queue length of 100 transactions is reasonable.
I will just have to adjust my design I guess, possibly use
another form of IPC.
Moi
> "Moi" <ro...@invalid.address.org> wrote in message
>
>>
>> cat /dev/zero | dd | >dev/null
>>
>> What would happen if the kernel would have unlimited bufferspace ?
>>
>> Also; you are confusing "throughput" (tps) with chunk-size.
>
> No I am not. Because my process can handle 100 transactions per second a
> queue length of 100 transactions is reasonable. I will just have to
> adjust my design I guess, possibly use another form of IPC.
Why would you keep 100 "transactions" (actually they are just messages)
stored "in transit" in a system buffer ?
If you want them buffered, then let the reader buffer.
AvK
Peter Olcott
So that I would not have to do the buffering myself.
> If you want them buffered, then let the reader buffer.
I guess that's my only choice.
>
> AvK
Moi
> "Moi" <ro...@invalid.address.org> wrote in message
> news:a56f3$4bc0a145$5350c024$55...@cache100.multikabel.net...
>>
>>
>> Why would you keep 100 "transactions" (actually they are just messages)
>> stored "in transit" in a system buffer ?
>>
>>
> So that I would not have to do the buffering myself.
>
>> If you want them buffered, then let the reader buffer.
>
> I guess that's my only choice.
No, the other choice is to let the writer block.
(which is not as bad as it seems; it makes little sense to
accept work that you cannot handle (yet) )
And there still is the choice of a spooldir, which offers you persistence
and atomicity for free. Plus high-capacity ;-)
AvK
Peter Olcott
> On Sat, 10 Apr 2010 11:08:06 -0500, Peter Olcott wrote:
>
>> "Moi" <ro...@invalid.address.org> wrote in message
>> news:a56f3$4bc0a145$5350c024$55...@cache100.multikabel.net...
>>
>> I guess that's my only choice.
>
> No, the other choice is to let the writer block.
> (which is not as bad as it seems; it makes little sense to
> accept work that you cannot handle (yet) )
>
It would have provided a simpler design.
> And there still is the choice of a spooldir, which offers
> you persistence
> and atomicity for free. Plus high-capacity ;-)
>
> AvK
I made a note of that.
Bob Smith
> "Bob Smith" <bsmith@linux
>> Second, the writer can write about 4K bytes before getting
>> blocked if the reader is not reading. This is usually not
>> a problem but was for some of my robotic control software.
>> I now use a proxy device for this.
>
> That sounds dumb. Why not simply grow the queue length to
> some arbitrary pre specified length?
Sorry I wasn't clear. The problem is that I don't want
any data queued. In my robotic application the data is
from a sensor reading. The problem with reading queued
data is that the reader has no idea how stale the sensor
data is. I wanted something that would block the writer
immediately if no reader was present.
Bob
Peter Olcott
In contrast my use is exactly and precisely like waiting
lines in a grocery store. It does not matter how long the
line gets as long as the jobs are processed in order. The
practical limit is about 100 jobs because I also want to
limit the customers waiting time one second.
Jasen Betts
>
> My process can handle 100 transactions per second. I was
> envisioning a FIFO at least this long.
If your process can handle 0.0001 transactions per microsecond.
how big do you need the buffer to be?
--- news://freenews.netfront.net/ - complaints: ne...@netfront.net ---
Phil Carmody
> Peter Olcott wrote:
>> "Bob Smith" <bsmith@linux
>>> Second, the writer can write about 4K bytes before getting
>>> blocked if the reader is not reading. This is usually not
>>> a problem but was for some of my robotic control software.
>>> I now use a proxy device for this.
>>
>> That sounds dumb. Why not simply grow the queue length to some
>> arbitrary pre specified length?
>
> Sorry I wasn't clear.
I think you were perfectly clear.
> The problem is that I don't want
> any data queued.
Your 'any' there muddies the water. Apparently you want, or are
happy with, a 4k buffer.
> In my robotic application the data is
> from a sensor reading. The problem with reading queued
> data is that the reader has no idea how stale the sensor
> data is. I wanted something that would block the writer
> immediately if no reader was present.
An alternative would be circular-buffer-like - accept new data,
but throw away old data rather than processing it (think input
devices in linux, which can include some 'measurement' devices
like accellerometers and magnetometers, say). That might not be
suitable for your particular application, of course.
Phil
--
I find the easiest thing to do is to k/f myself and just troll away
-- David Melville on r.a.s.f1
Peter Olcott
"Jasen Betts" <ja...@xnet.co.nz> wrote in message
news:hps2v2$nta$1...@reversiblemaps.ath.cx...
You can't make the decision on this basis. The basis that
this decision needs to be decided upon is the unpredictable
pattern of jobs that will be experienced in the future. This
can range anywhere from getting 100,000 jobs all at the same
minute once a day, and no other jobs any other time, to a
completely even flow of maximum processing capability 24/7.
Because of this is would be best to form a FIFO queue of
arbitrary length.
hsantos
> On 04/ 9/10 11:12 AM, Peter Olcott wrote:
>
> >news:82726l...@mid.individual.net...
> >> On 04/ 9/10 04:58 AM, Peter Olcott wrote:
> >>> Are there any reliability issues or other caveats with
> >>> using
> >>> named pipes?
>
>
> > One respondent in another group said that named pipes are
> > inherently very unreliable and I think that he switched to
> > some sort of sockets. In any case he did not choose named
> > pipes for his IPC because of significant reliability issues
> > that he encountered.
>
> Care to cite a reference?
Ian,
As you all know how "Olcott Threads" inevitably become prolonged,
mangled, twisted and lost. The recent Wndows threads he started were
no different. The problem is that he ask questions, and as you know
will always argue the input provided, when in fact, he has shown he
knows nothing about what he is asking or gets it all wrong. There is
no experimentation, exploring on his part. For him, the presumption
is that you are wrong with your input unless he has basic
understanding of what you are talking about. The problem is he often
does not.
Named pipes do work reliable under Windows and IMTE, for certain low
end types of applications. When the loads are higher, which was what
Olcott was wishing with an overly optimistic throughput requirement,
I was reflecting that since he has shown a lack of experience to
program for named pipes which can be very complex, especially what for
he was asking to do, can be very unreliable in high throughput and
network scenarios. For our product line, we considered Named Piped
for the RPC protocol. It proved to not work very well, it didn't
scale, there were lock ups and bottle necks. Switching to a socket
RPC bind, resolved this high end throughput and networking issues.
I suggested, for HIM, that he should look at other ways and cited
various methods to explore, including a simplistic log file because he
was getting crazy with new requirements of having 100% "crash
recovery" and "Fault Tolerance." He didn't want and lost, so I told
him just use disk files with no caching (as much as he can turn of or
flush often). But every idea was throw to him. I even cited
Microsoft's own recommendation (sorry, don't have the link off hand)
to considers sockets, not named pipes, when high throughput and
networking may be important. I also provided a DR DOBBS 2003 article
with a C++ Named Pipe class discussion how easy it can be when you can
implement it right taking into account, error trappings,
sychronization, buffering, etc.
If you follow Olcotts messages/threads of late and all the huge thread
growth with nearly all of them ending the same way, the guy wants
complexity yet simplicity with everything running with perpertual
motion in a resistance-less world. Pure Memory, No Virtual Memory,
yet he wants to load memory beyond the OS effective process limits
with no sharing, No Disk Caching yet he wants 100% crash recovery - no
lost of data and the most important of all, is what nearly a dozen
experts, scientist and engineers have stated his design is crippled
for his desired throughput.
Putting Windows/Linux aside, what he wants is:
Multi-Thread Web Server --> 4 EXE processes each with 1 FIFO
queue
He has trouble designing his own EXE for threads. He only got the
Multi-thread web server because he found Mongoose which he statedd he
would embedded to each EXE. So right there he is confused with
citing a different model:
4 EXEs each Mongoose Web Server
In prolong threads, we tried to get him to understand that he has 4
different web servers now, each with their own URL or IP or whatever.
He might consider using a web proxy perhaps unless he going to use 4
different Web Forms each with their own URL. I don't think he has yet
to understand how 4 EXE Web Server model alters his client side
designs.
But regardless of the layout, he was also not getting that his
suggested rate was telling him otherwise on how many Handlers were
required. I provided a simple formula which is a variation of
Little's Law:
TPS = N * 1000 / WT
where
TPS is transactions per second
N is number of handlers
WT is the worktime to complete a single transaction in msecs.
He indicated:
TPS = 100 jobs per sec
WT = 100 ms Maximum Work Time per job
Therefore, on this basis, for the worst case, he needs a minimum of 10
handlers, regardless of how the handlers are spread; one machine or
multi-machines, one exe with multiple threads or multple exes, etc.
But he was trying to do this with 4 exes, and he wants each EXE to be
totally separated per job task. He cited he wanted no contention
between the four EXE, no data sharing, total autonomous, yet, he kept
talking of needs to do scaling, thread/process priority changles to
help balance the single high priority EXE by putting the others to
sleep.
Which is all fine and good, but based on his confused, changing by the
day design, back and forth, in my experience and others that been
participating in is vapor project of his, is flawed and will not work.
Finally, just like one would expect in a Linux techncal forum, when
anyone post in Windows forum, they should expect to get Windows based
answers, and even then generality was provided to him. I'm not a
Linux expert, but I'm sure Linux also promotes common sense
engineering and the same time of time-based sound design basic
principles just like under Windows. For his case,
- Use Shared Memory
- Use Memory Maps
- Use Thread Worker Pools
- Use IOCP for scaling
and foremost don't re-invent the wheel, use the power of the computer
which he is totally underestimating, especially with thread designs.
Even then, I cited how some vendors are moving to a process-centric
design such as Chrome and IE9 browsers to provide better browser
reliability when many web sites are opened.
So everything was discussed with him. The benefit of the doubt was
given to him with every idea throw out to him, even with a flawed
Many Thread To One Thread FIFO design.
The bottom line he doesn't wish to explore, test and measure on his
own regardless of what is suggested. He is looking for proof and
theory and even when thats provided, he continues to rationalizes the
opposite or incorrect conclusion of whats stated, written and long
published to suit his purpose.
Go Figure
Hector Santos/CTO
http://www.santronics.com