linux, named pipe (fifo) and (while ...)
prhlava
Hello everyone,
I am trying to read (repeatedly) from named pipe (fifo) on linux (the
program will be a long running process...).
The following does not work, but if I remove the (while true , it
does...
(def pipe-name "/tmp/my-pipe")
(def buffer-size (* 1 1024))
(while true
(with-open [pipe (new java.io.FileInputStream pipe-name)]
(print pipe)
(let [buffer (make-array Byte/TYPE buffer-size)]
(. pipe read buffer)
(map print buffer))))
I am lost here...
Vlad
PS: The new java.io.FileInputStream call blocks until the pipe is
closed at the "other" end...
Randall R Schulz
> Hello everyone,
>
> I am trying to read (repeatedly) from named pipe (fifo) on linux (the
> program will be a long running process...).
>
> The following does not work, but if I remove the (while true , it
> does...
Characterize "does not work," if you would.
> (def pipe-name "/tmp/my-pipe")
>
> (def buffer-size (* 1 1024))
>
> (while true
> (with-open [pipe (new java.io.FileInputStream pipe-name)]
> (print pipe)
> (let [buffer (make-array Byte/TYPE buffer-size)]
> (. pipe read buffer)
> (map print buffer))))
>
> I am lost here...
For one thing, you will need to loop on reading from the pipe. Reads
will block while there a writer holds the pipe open but no bytes are in
the pipe. Once there are zero writers (and no buffered / unread data
remains in the pipe), readers will get an end-of-file indication (zero
bytes returned) on every read attempt.
> Vlad
>
> PS: The new java.io.FileInputStream call blocks until the pipe is
> closed at the "other" end...
That is how pipes work. The kernel cannot signal and end-of-file while a
writer has the pipe open, since that writer may at any time decide to
writer more data. The attempt to open a named pipe for reading will
likewise block until there's a writer. And vice versa: an attempt to
open a named pipe for writing will block until there is a reader.
Similarly, write attempts will block when the writer overruns the
reader and operating system kernel's internal pipe high-water mark is
reached. And naturally, read attempts block when there's no available
data.
There are non-blocking modes that the OS's native descriptor can be
placed in. I don't know if there's any support for this in the Java IO
libraries.
All these things happen in the operating system.
Randall Schulz
prhlava
> > does...
>
> Characterize "does not work," if you would.
the
"while" out, and run the remaining code, it does what expected -
prints the content of the buffer (once, of course).
> (explanations snipped)
Thanks for the explanations, the blocking is not a problem. The way I
expected it to work is "open-read-close" on one end and "open-write-
close" on the other...
Kind regards,
Vlad
Randall R Schulz
> > > The following does not work, but if I remove the (while true , it
> > > does...
> >
> > Characterize "does not work," if you would.
>
> Well, nothing happens with "(while" around the code... But if I take
> the "while" out, and run the remaining code, it does what expected
> - prints the content of the buffer (once, of course).
You wrote:
> (while true
> (with-open [pipe (new java.io.FileInputStream pipe-name)]
> (print pipe)
> (let [buffer (make-array Byte/TYPE buffer-size)]
> (. pipe read buffer)
> (map print buffer))))
You're asking for the pipe to be repeatedly opened, one uninterrupted
glob of bytes read and processed and then the pipe closed. Is that
really what you intend?
As written, this suggest a kind of "daemon" that monitors the pipe,
waiting for successive writers, each of which must write everything
they want processed by the far side in a single write call and
furthermore that transmission must not exceed the operating system's
pipe high-water mark. All this seems a bit fragile to me.
But more practically, you should _say_ what you want your code to
accomplish.
> > (explanations snipped)
>
> Thanks for the explanations, the blocking is not a problem. ...
It was not clear whether or not your mention of the blocking behavior
was something you considered unexpected or problematic.
> Vlad
Randall Schulz
prhlava
> You're asking for the pipe to be repeatedly opened, one uninterrupted glob of bytes read and processed and then the pipe closed. Is that really what you intend?
> As written, this suggest a kind of "daemon" that monitors the pipe,
> waiting for successive writers, each of which must write everything
> they want processed by the far side in a single write call and
> furthermore that transmission must not exceed the operating system's
> pipe high-water mark. All this seems a bit fragile to me.
>
> But more practically, you should _say_ what you want your code to
> accomplish.
already does this, as an exercise) + making it work through pipe (as
java start-up is longish) => therefore I will have submitter and
"daemon" receiver...
> > Thanks for the explanations, the blocking is not a problem. ...
> It was not clear whether or not your mention of the blocking behavior
> was something you considered unexpected or problematic.
woking with named pipe, so the explanations were welcome...
But why it does not work with the "while" around the code still
escapes me...
Kind regards,
Vlad
PS: back tomorrow
Randall R Schulz
On Sunday 07 December 2008 16:12, prhlava wrote:
> > You're asking for the pipe to be repeatedly opened, one
> > uninterrupted glob of bytes read and processed and then the pipe
> > closed. Is that really what you intend?
>
> Yes, that was my intention, maybe a rethink is in order...
>
> > As written, this suggest a kind of "daemon" that monitors the pipe,
> > waiting for successive writers, each of which must write everything
> > they want processed by the far side in a single write call and
> > furthermore that transmission must not exceed the operating
> > system's pipe high-water mark. All this seems a bit fragile to me.
> >
> > But more practically, you should _say_ what you want your code to
> > accomplish.
>
> Store e-mail messages in a database (I am porting a program that
> already does this, as an exercise) + making it work through pipe (as
> java start-up is longish) => therefore I will have submitter and
> "daemon" receiver...
I cannot really recommend using named pipes for this. Or, really, for
anything. Naturally, a CGI-style approach that uses Java applications
is going to have a severely limited request processing rate because of
the very high JVM start-up cost. So the desire to create a permanent
server is appropriate. However, the use of a named pipe most likely is
not.
If your application is client/server, you really should just go with an
ordinary TCP connection (or, conceivably, a UDP port), define a proper
protocol and do the whole thing properly.
Realistically, I'd start by looking at the ordinary Servlet /
HttpServlet mechanism. You get so much from existing servlet containers
(Tomcat, Jetty, GlassFish, etc.) that it's very hard to justify
starting from scratch.
> > > Thanks for the explanations, the blocking is not a problem. ...
> >
> > It was not clear whether or not your mention of the blocking
> > behavior was something you considered unexpected or problematic.
>
> No worries, I ment it as a hint, but this is also the 1st time I am
> woking with named pipe, so the explanations were welcome...
Named pipes have peculiar semantics and, of course, do not cross machine
boundaries (unless you're running in one of the now-rare distributed
Unix kernels—I say this as a one-time employee of Locus Computing
Corporation...). I can't say named pipes really useful for much other
than their use by shells for their <( command ) syntax.
> ...
>
> Vlad
Randall Schulz
Samantha Atkins
On Dec 7, 2008, at 4:12 PM, prhlava wrote:
>
>
>> You're asking for the pipe to be repeatedly opened, one
>> uninterrupted glob of bytes read and processed and then the pipe
>> closed. Is that really what you intend?
>
> Yes, that was my intention, maybe a rethink is in order...
>
>> As written, this suggest a kind of "daemon" that monitors the pipe,
>> waiting for successive writers, each of which must write everything
>> they want processed by the far side in a single write call and
>> furthermore that transmission must not exceed the operating system's
>> pipe high-water mark. All this seems a bit fragile to me.
>>
>> But more practically, you should _say_ what you want your code to
>> accomplish.
>
> Store e-mail messages in a database (I am porting a program that
> already does this, as an exercise) + making it work through pipe (as
> java start-up is longish) => therefore I will have submitter and
> "daemon" receiver...
Why not just stream it into a JDBC Blob to your database? Is there
some reason the mail to be stored needs to be read remotely to the
machine storing messages to the database BTW?
- samantha
prhlava
Hello Randall,
> If your application is client/server, you really should just go with an
> ordinary TCP connection (or, conceivably, a UDP port), define a proper
> protocol and do the whole thing properly.
times in parallel and submit
more than one e-mail in parallel to the daemon - and this is needed).
> Realistically, I'd start by looking at the ordinary Servlet /
> HttpServlet mechanism. You get so much from existing servlet containers
> (Tomcat, Jetty, GlassFish, etc.) that it's very hard to justify
> starting from scratch.
container...
> Named pipes have peculiar semantics and, of course, do not cross machine
> boundaries (unless you're running in one of the now-rare distributed
> Unix kernels—I say this as a one-time employee of Locus Computing
> Corporation...). I can't say named pipes really useful for much other
> than their use by shells for their <( command ) syntax.
Thanks for all the info, and I can proceed in other way(s) now...
But would you not expect that the working code still works when
enclosed in the "while" construct?
Kind regards,
Vlad
prhlava
Hello Samantha,
> Why not just stream it into a JDBC Blob to your database?
basicaly a postfix mail filter something like spamassassin in
principle, but does different things with the e-mails).
> Is there some reason the mail to be stored needs to be read remotely to the
> machine storing messages to the database BTW?
(from database) on the machine doing the storing...
Kind regards,
Vlad