popen() and r+

[Mut...@dastardlyhq.com]

Has anyone ever used popen() with the r+ option on MacOS? Writing to the file
descriptor works fine (the output goes to the printer queue) but any attempt to
read the response from CUPS from the file descriptor simply hangs. eg:

if ((pfile = popen("lp","r+")))
{
... write data to printer ...
fgets(str,sizeof(str),pfile); <- hangs here
}

It also hangs if I try while(!feof(pfile)) or read(fileno(pfile),....)

lp is producing output and I can redirect to a file , eg popen("lp > myfile",..)
I assume this is some kind of buffer flush issue but I can't find anything on
google about it.

Thanks for any help.

[Kaz Kylheku]

On 2023-01-09, Mut...@dastardlyhq.com <Mut...@dastardlyhq.com> wrote:
> Has anyone ever used popen() with the r+ option on MacOS? Writing to the file

Coprocesses are tricky; it's easy to get into a deadlock if you just
read and write with one thread, without multiplexing the I/O using
poll or select.

> descriptor works fine (the output goes to the printer queue) but any attempt to
> read the response from CUPS from the file descriptor simply hangs. eg:
>
> if ((pfile = popen("lp","r+")))
> {
> ... write data to printer ...

Did you fflush(pfile) here? Maybe all the data didn't go.
I think popen returns streams that are in fully buffered mode,
so output is only flushed when a buffer fills, the stream is
closed or fflush is called.

You would think that a simple half-duplex situation like this would
avoid deadlock, since lp should read your entire request until the end,
and only then produce a response.

However, if some trailing part of your request is sitting in a buffer,
such that lp is waiting for that last bit before replying, then you have
a deadlock.

(Note that ISO C itself requires a flush operation when switching from
output to input on a bidirectional stream, as a matter of well-defined
behavior. When switching from input to output, a file positioning
operation is required (regardless of whether the stream supports
positioning; seeking zero bytes from the current position with
fseek should do the trick). I think the main reason for this is that
some library implementations share one buffer between input
and output.)

> fgets(str,sizeof(str),pfile); <- hangs here
> }

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazi...@mstdn.ca

[Mut...@dastardlyhq.com]

On Mon, 9 Jan 2023 11:20:09 -0000 (UTC)
Kaz Kylheku <864-11...@kylheku.com> wrote:
>On 2023-01-09, Mut...@dastardlyhq.com <Mut...@dastardlyhq.com> wrote:
>> Has anyone ever used popen() with the r+ option on MacOS? Writing to the file
>
>
>Coprocesses are tricky; it's easy to get into a deadlock if you just
>read and write with one thread, without multiplexing the I/O using
>poll or select.
>
>> descriptor works fine (the output goes to the printer queue) but any attempt
>to
>> read the response from CUPS from the file descriptor simply hangs. eg:
>>
>> if ((pfile = popen("lp","r+")))
>> {
>> ... write data to printer ...
>
>Did you fflush(pfile) here? Maybe all the data didn't go.

No I didn't. Just tried it - didn't make any difference sadly.

>You would think that a simple half-duplex situation like this would
>avoid deadlock, since lp should read your entire request until the end,
>and only then produce a response.
>
>However, if some trailing part of your request is sitting in a buffer,
>such that lp is waiting for that last bit before replying, then you have
>a deadlock.

Oddly if popened in write only mode the output from lp is written to stdout
without any delay and if stdout is redirected to a file, eg:

popen("lp > myfile","w")

that happily writes to the file.

>(Note that ISO C itself requires a flush operation when switching from
>output to input on a bidirectional stream, as a matter of well-defined
>behavior. When switching from input to output, a file positioning
>operation is required (regardless of whether the stream supports
>positioning; seeking zero bytes from the current position with
>fseek should do the trick). I think the main reason for this is that

Unfortunately not:

if (fseek(pfile,0,SEEK_SET) == -1) perror("fseek()");

results in

fseek(): Illegal seek

Perhaps the popen() implementation on MacOS/BSD is just broken. I'll have to
try it on Linux when I get a chance later.

[Nicolas George]

Kaz Kylheku , dans le message <202301090...@kylheku.com>, a
écrit :

> However, if some trailing part of your request is sitting in a buffer,
> such that lp is waiting for that last bit before replying, then you have
> a deadlock.

I would guess it is not enough: lp is probably waiting for EOF, and there is
no way to close only half of a popen stream.

[Mut...@dastardlyhq.com]

On 09 Jan 2023 12:01:35 GMT

Surely if you're going to implement r+ functionality you'd make sure you
flushed the write buffer before the close so this issue doesn't occur?

[Nicolas George]

Mut...@dastardlyhq.com, dans le message <tphdfa$1016$1...@gioia.aioe.org>,
a écrit :

>>I would guess it is not enough: lp is probably waiting for EOF, and there is
>>no way to close only half of a popen stream.
> Surely if you're going to implement r+ functionality you'd make sure you
> flushed the write buffer before the close so this issue doesn't occur?

The issue is not about an unflushed buffer, it is about a stream that is
still open and therefore could receive more data. Consider this:

f = popen("some_command", "r+");
fprintf(f, "Hello.\n");
fflush(f);
fgets(line, f);

How does some_command know you are done talking after "Hello."?

[Scott Lurndal]

The 'p' in "popen" stands for "pipe". Pipes in unix are unidirectional.

From https://pubs.opengroup.org/onlinepubs/9699919799/functions/popen.html


"Since open files are shared, a mode r command can be used as
an input filter and a mode w command as an output filter."

and
"The behavior of popen() is specified for values of mode of r and w."

You can't do what you want to do with popen.

[Mut...@dastardlyhq.com]

From the MacOS popen() man page:

reading or writing, not both. Because popen() is now implemented using a
bidirectional pipe, the mode argument may request a bidirectional data
flow. The mode argument is a pointer to a null-terminated string which
must be `r' for reading, `w' for writing, or `r+' for reading and
writing.

HTH.

[Mut...@dastardlyhq.com]

On 09 Jan 2023 16:09:18 GMT

Fair point. So clearly a 2 way popen isn't going to work for utilities that
don't respond until they get an EOF on their input. Unless there's some way to
spoof that.

[Kenny McCormack]

In article <63bc3c2e$0$3184$426a...@news.free.fr>,

Nicolas George <nicolas$geo...@salle-s.org> wrote:
>Mut...@dastardlyhq.com, dans le message <tphdfa$1016$1...@gioia.aioe.org>,

> a crit:

>>>I would guess it is not enough: lp is probably waiting for EOF, and there is
>>>no way to close only half of a popen stream.
>> Surely if you're going to implement r+ functionality you'd make sure you
>> flushed the write buffer before the close so this issue doesn't occur?
>
>The issue is not about an unflushed buffer, it is about a stream that is
>still open and therefore could receive more data. Consider this:
>
> f = popen("some_command", "r+");
> fprintf(f, "Hello.\n");
> fflush(f);
> fgets(line, f);
>
>How does some_command know you are done talking after "Hello."?

Right. The classic example of this is "sort". sort cannot possibly
generate any output until it is certain that no more input is coming.

I assume "lp" behaves likewise. It looks to me that the best solution is
as given by OP: redirect output to a file (in the shell command passed to
popen()), then parse that file after doing pclose().

--
If Jeb is Charlie Brown kicking a football-pulled-away, Mitt is a '50s
housewife with a black eye who insists to her friends the roast wasn't
dry.

[Scott Lurndal]

gaz...@shell.xmission.com (Kenny McCormack) writes:
>In article <63bc3c2e$0$3184$426a...@news.free.fr>,
>Nicolas George <nicolas$geo...@salle-s.org> wrote:
>>Mut...@dastardlyhq.com, dans le message <tphdfa$1016$1...@gioia.aioe.org>,
>> a crit:
>>>>I would guess it is not enough: lp is probably waiting for EOF, and there is
>>>>no way to close only half of a popen stream.
>>> Surely if you're going to implement r+ functionality you'd make sure you
>>> flushed the write buffer before the close so this issue doesn't occur?
>>
>>The issue is not about an unflushed buffer, it is about a stream that is
>>still open and therefore could receive more data. Consider this:
>>
>> f = popen("some_command", "r+");
>> fprintf(f, "Hello.\n");
>> fflush(f);
>> fgets(line, f);
>>
>>How does some_command know you are done talking after "Hello."?
>
>Right. The classic example of this is "sort". sort cannot possibly
>generate any output until it is certain that no more input is coming.
>
>I assume "lp" behaves likewise. It looks to me that the best solution is
>as given by OP: redirect output to a file (in the shell command passed to
>popen()), then parse that file after doing pclose().

Or bite the bullet and use fork/exec and set up stdin and stdout
for the child appropriately with two stdio streams, one for read
and one for write.

[Kenny McCormack]

In article <YfZuL.38332$0dpc....@fx33.iad>,
Scott Lurndal <sl...@pacbell.net> wrote:
...

>>I assume "lp" behaves likewise. It looks to me that the best solution is
>>as given by OP: redirect output to a file (in the shell command passed to
>>popen()), then parse that file after doing pclose().
>
>Or bite the bullet and use fork/exec and set up stdin and stdout
>for the child appropriately with two stdio streams, one for read
>and one for write.

Of course you can always do that. I took it as a parameter of the problem
that the goal was to stay inside the context of popen().

--
The difference between communism and capitalism?
In capitalism, man exploits man. In communism, it's the other way around.

- Daniel Bell, The End of Ideology (1960) -

[Nicolas George]

Scott Lurndal, dans le message <a8XuL.575788$GNG9....@fx18.iad>, a
écrit :

>>Has anyone ever used popen() with the r+ option on MacOS?

^^^^^^

> The 'p' in "popen" stands for "pipe". Pipes in unix are unidirectional.
>
> From https://pubs.opengroup.org/onlinepubs/9699919799/functions/popen.html
>
>
> "Since open files are shared, a mode r command can be used as
> an input filter and a mode w command as an output filter."
>
> and
> "The behavior of popen() is specified for values of mode of r and w."
>
> You can't do what you want to do with popen.

Unfortunately, they can:
https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/popen.3.html

“or `r+' for reading and writ-ing.”

Blame on Apple for adding this extension without documenting how to
half-close the stream. They could have returned two FILE*, or documented the
use of shutdown(fileno(p)).

Lucky guess: try write(fileno(p), "", 0).

[Kenny McCormack]

In article <tphh1a$q5j$1...@gioia.aioe.org>, <Mut...@dastardlyhq.com> wrote:
...

>> "The behavior of popen() is specified for values of mode of r and w."
>>
>>You can't do what you want to do with popen.
>
>From the MacOS popen() man page:
>
> reading or writing, not both. Because popen() is now implemented using a
> bidirectional pipe, the mode argument may request a bidirectional data
> flow. The mode argument is a pointer to a null-terminated string which
> must be `r' for reading, `w' for writing, or `r+' for reading and
> writing.
>
>HTH.

When answering questions like this, it is important to be clear as to
whether you are answering from a "POSIX standards" POV vs. from the OP's
actual use case situation. The POSIX standards answer is, as stated above,
you can't do that (and now go home and bother us no more). However, OP
stated that he was using a Mac and that Mac documentation explicitly allows
this (with, of course, the caveat that it is an "extension").

Note, BTW, that I don't currently have access to a Mac to check this, but I
am assuming that the reports posted so far in this thread are accurate.

Interestingly, the Linux documentation that just checked makes it clear
that only "r" and "w" are valid. Even though, Linux certainly has
bidirectional pipes available (and, in fact, I think it is common knowledge
that pipe is implemented with socketpair), so you'd think Linux would have
a bidirectional version of popen() available, but apparently such is not
the case.

--
Watching ConservaLoons playing with statistics and facts is like watching a
newborn play with a computer. Endlessly amusing, but totally unproductive.

[Mut...@dastardlyhq.com]

On 09 Jan 2023 21:55:39 GMT

Good idea, didn't work though. It does look like Apples implementation is
broken, or more politely - incomplete. There seems to be no way to close one
half of the pipe to force the child process to complete its stdout.

[Mut...@dastardlyhq.com]

Urgh. Yes I could but thats what one wants to avoid by using popen(). Its such
a faff to setup plus all the error checking whereas popen() is 1 line of code.

[Mut...@dastardlyhq.com]

On Tue, 10 Jan 2023 02:20:15 -0000 (UTC)
gaz...@shell.xmission.com (Kenny McCormack) wrote:
>In article <tphh1a$q5j$1...@gioia.aioe.org>, <Mut...@dastardlyhq.com> wrote:

>....

>>> "The behavior of popen() is specified for values of mode of r and w."
>>>
>>>You can't do what you want to do with popen.
>>
>>From the MacOS popen() man page:
>>
>> reading or writing, not both. Because popen() is now implemented using a
>
>> bidirectional pipe, the mode argument may request a bidirectional data
>> flow. The mode argument is a pointer to a null-terminated string which
>> must be `r' for reading, `w' for writing, or `r+' for reading and
>> writing.
>>
>>HTH.
>
>When answering questions like this, it is important to be clear as to
>whether you are answering from a "POSIX standards" POV vs. from the OP's
>actual use case situation. The POSIX standards answer is, as stated above,
>you can't do that (and now go home and bother us no more). However, OP
>stated that he was using a Mac and that Mac documentation explicitly allows
>this (with, of course, the caveat that it is an "extension").
>
>Note, BTW, that I don't currently have access to a Mac to check this, but I
>am assuming that the reports posted so far in this thread are accurate.
>
>Interestingly, the Linux documentation that just checked makes it clear
>that only "r" and "w" are valid. Even though, Linux certainly has

Yes, I didn't realise that when I started writing the code on MacOS.

>bidirectional pipes available (and, in fact, I think it is common knowledge
>that pipe is implemented with socketpair), so you'd think Linux would have
>a bidirectional version of popen() available, but apparently such is not
>the case.

Perhaps they realised it can't work properly in the current API of popen()
as there's no way to close one half of the pipe without some modification
to pclose().

Anyone know if r+ is a MacOS thing or it was brought over from FreeBSD?

[Geoff Clare]

Muttley wrote:

> It does look like Apples implementation is
> broken, or more politely - incomplete. There seems to be no way to close one
> half of the pipe to force the child process to complete its stdout.

If popen(..., "r+") is only usable with commands that don't wait for
EOF, then execute a command that doesn't wait for EOF.

You were trying to execute "lp". You could instead execute something
like:

sed '/^=EnD=oF=InPuT=$/{d;q;}' | lp

and write that sentinel string (plus a newline) to the stream after
the data to be printed. To reduce the risk of the sentinel string
occurring in the data to be printed, you could include some rarely-used
control characters.

--
Geoff Clare <net...@gclare.org.uk>

[Scott Lurndal]

Does OSX have a printer API that allows one to submit a job (rather than
using system(3) or popen(3))?

[Mut...@dastardlyhq.com]

Sounds like more trouble than its worth and rather flakey.

[Mut...@dastardlyhq.com]

On Tue, 10 Jan 2023 14:37:17 GMT
sc...@slp53.sl.home (Scott Lurndal) wrote:

>Mut...@dastardlyhq.com writes:
>>>Or bite the bullet and use fork/exec and set up stdin and stdout
>>>for the child appropriately with two stdio streams, one for read
>>>and one for write.
>>
>>Urgh. Yes I could but thats what one wants to avoid by using popen(). Its
>such
>>a faff to setup plus all the error checking whereas popen() is 1 line of code.
>
>>
>
>Does OSX have a printer API that allows one to submit a job (rather than
>using system(3) or popen(3))?

Yes, there's the standard CUPS C API which Linux also shares, but its a bit long
winded. However the main reasons for not using it are if the box still uses
the old style lp system CUPS is no use plus the headers and library might not
be installed either so the program won't even compile.

[Kaz Kylheku]

That's a great observation.

Now the "r+" mode is not portable to begin with.

The POSIX model of standard I/O streams is such that a stream has one
underlying descriptor, accessible via fileno.

Suppose an implementation implements a bidirectional popen using only
one descriptor, such that the descriptor returned by fileno is all there
is; that object handles both data transfer directions.

Such an implementation is quite possibly using the socketpair function
to creae the bidirectional descriptor.

And so if socketpair is being used, then it should be possible to use
shutdown on the descriptor to do a half-close.

As in:

// write data to lp ...

// send everything to socket
fflush(pipe_stream);

// close in writing direction
if (shutdown(fileno(pipe_stream), SHUT_WR) != 0)
{
// didn't work: not a socket?
}

It's worth a try.

[Kaz Kylheku]

On 2023-01-09, Mut...@dastardlyhq.com <Mut...@dastardlyhq.com> wrote:

And, so, continuing in the same vein as my previous posting: we have had
bidirectional pipes in Unix for decades. The socketpair function creates
two descriptors and places them into an array, just like pipe. But
they are a pair of connected sockets: writing to one makes data appear
in the other and vice versa.

So there are two possibilities:

1. MacOS invented a new kind of bidirectional pipe that is other than a
socket pair.

2. The bidirectional pipe is actually a socket pair.

If (2) is the case, it would be amazing if, after flushing all the
written data to lp, you couldn't pull out the descriptor using fileno,
and then half-close it using shutdown(fd, SHUT_WR).

[Geoff Clare]

Kaz Kylheku wrote:

> And, so, continuing in the same vein as my previous posting: we have had
> bidirectional pipes in Unix for decades. The socketpair function creates
> two descriptors and places them into an array, just like pipe. But
> they are a pair of connected sockets: writing to one makes data appear
> in the other and vice versa.
>
> So there are two possibilities:
>
> 1. MacOS invented a new kind of bidirectional pipe that is other than a
> socket pair.

I'm fairly sure it's a true bidirectional pipe, not a socket pair.
However, such things have existed since SVR4 (where they were
STREAMS-based). I think in Solaris they are no longer STREAMS-based
but are still bidirectional.

MacOS got them from FreeBSD, which has had them since 2.2.1 according
to their historical man pages:

www.freebsd.org/cgi/man.cgi?query=pipe&sektion=2&manpath=FreeBSD+2.2.1-RELEASE

> 2. The bidirectional pipe is actually a socket pair.

This would not conform to POSIX, as fstat() would show S_ISSOCK(st_mode)
true instead of S_ISFIFO(st_mode).

--
Geoff Clare <net...@gclare.org.uk>

[Nicolas George]

Geoff Clare , dans le message
<p1r29j-...@ID-313840.user.individual.net>, a écrit :

> I'm fairly sure it's a true bidirectional pipe, not a socket pair.

Note that pipes, unidirectional or bidirectional, are as much pairs than
socket pairs. There is little difference between a stream socket pair and a
bidirectional pipe: maybe socket pairs have a few extra features like
half-close through shutdown() or ancillary data, or maybe they are the same
thing internally.

[Mut...@dastardlyhq.com]

Tried, didn't work. feof() was always true and fgets() returned garbage. :(

[Kaz Kylheku]

On 2023-01-11, Geoff Clare <ge...@clare.See-My-Signature.invalid> wrote:

> Kaz Kylheku wrote:
>
>> 2. The bidirectional pipe is actually a socket pair.
>
> This would not conform to POSIX, as fstat() would show S_ISSOCK(st_mode)
> true instead of S_ISFIFO(st_mode).

Sure, but the behavior "r+" in popen is unspecified.