IPC::Run on win32 failures
Jos I. Boumans
we're noticing a few bleadperl smokes failing on win32 because IPC::Run
requires some win32 modules that aren't part of core currently.
The 2 modules in question are Win32API::File and Win32::Process, both
part of the libwin32 distribution on CPAN.
CPANPLUS currently needs IPC::Run to reliably do shelling out on win32
(and cygwin). On all other platoforms (at least that i know of), we can
use IPC::Open3.
Therefor, to make CPANPLUS run on win32 (and cygwin), we need some good
IPC support.
The options I see open are as follows (non mutualy exclusive, in
no-particular order)
1) Fix IPC::Open3 for win32/cygwin (and any other os it currently
doesn't support) -- we may well find we need the same tools for that as
IPC::Run uses
2) Leave bundling of IPC::run (and therefor win32::process and
win32api::file)
to 3rd party vendors like activestate -- we'll have to skip cpanplus
(and a few other modules') tests then on win32/cygwin that involve IPC
as they will undoubtedbly fail
3) add libwin32 to the core (that's 3.6 megs -- just those 2 modules,
which are
standalone with their own makefile would be about 300k)
4) make IPC::Run DTRT on win32 without these 2 modules (or at least
gracefully fail)
5) bundle these 2 modules with IPC::Run distribution, making it
responsible for
building them.
I'm not sure what the best course of action is here, but a solution
needs to be found to support win32 and cygwins IPC in this case.
I'm eager to hear your thoughts.
--
Jos Boumans
"You know you are never more indignant in life than when you're
shopping at a store you feel is beneath you and one of the other
customers mistakes you for one of the employees of that store."
- Dennis Miller
CPANPLUS http://cpanplus.sf.net
Reini Urban
> we're noticing a few bleadperl smokes failing on win32 because IPC::Run
> requires some win32 modules that aren't part of core currently.
> The 2 modules in question are Win32API::File and Win32::Process, both
> part of the libwin32 distribution on CPAN.
If you are really going to add Win32::Process to core, which I would
agree upon, please think about adding support to cygwin_to_winpid /
winpid_to_cygwin translation to Win32::Process, and not in
cygwin/cygwin.c under Proc::, which was posted 2/8/05 by Yitchak
Scott-Thonnes.
Do you need the Win32::Process cygwin patches for those needed helpers?
I posted them to libw...@perl.org 2/7/05.
This does not affect IPC::Run on cygwin, cygwin IPC::Run runs fine with
its internal cygwin pid, but when Win32::Process will be added, we
terribly need those translation funcs.
Win32::Process and Proc::ProcessTable just work with the winpid's, which
doesn't help IPC::Run.
> 3) add libwin32 to the core (that's 3.6 megs -- just those 2 modules,
> which are standalone with their own makefile would be about 300k)
If you add libwin32 to core, please also think about cleaning that MSVC
mess up, and add our cygwin/gcc patches. Unfortunately I cannot test
borland, just MSWin32, cygwin and mingw.
> I'm not sure what the best course of action is here, but a solution
> needs to be found to support win32 and cygwins IPC in this case.
--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/
Steve Hay
>Hi,
>
>we're noticing a few bleadperl smokes failing on win32 because IPC::Run
>requires some win32 modules that aren't part of core currently.
>The 2 modules in question are Win32API::File and Win32::Process, both
>part of the libwin32 distribution on CPAN.
>
>CPANPLUS currently needs IPC::Run to reliably do shelling out on win32
>(and cygwin). On all other platoforms (at least that i know of), we can
>use IPC::Open3.
>
I was just talking to Jan Dubois about this (off list, sorry -- I should
have kept it on the list), and Jan asked what the problem is with
IPC::Open3 on Win32.
Other than the manpage's reference to a "forking open" style behaviour
being unsupported on Win32 I'm not aware of any problems with it. It's
been in the core for ages and all its tests pass (with none skipped).
Can you desribe what the problem is? Fixing IPC::Open3 for Win32 would
be the cleanest solution, though I realise that may not be possible.
- Steve
------------------------------------------------
Radan Computational Ltd.
The information contained in this message and any files transmitted with it are confidential and intended for the addressee(s) only. If you have received this message in error or there are any problems, please notify the sender immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Radan Computational Ltd. The recipient(s) of this message should check it and any attached files for viruses: Radan Computational will accept no liability for any damage caused by any virus transmitted by this email.
Barrie Slaymaker
>
> Can you desribe what the problem is? Fixing IPC::Open3 for Win32 would
> be the cleanest solution, though I realise that may not be possible.
fork & execute's not such a big deal, IPC::Open3 does that ok. Open3()
has got special spawning code with Ilya's (I think) system 1, ... hack
that avoids fork emulation.
The hard part for me was getting I/O pipes to work with select() or to
get TCP sockets to work reliably with subprocesses. I'm not a Win32
guru and at the time I wrote this, I wanted to get it working on Win32
without requiring XS code or an upgrade to the Win32:: modules.
What I had problems with was combining the select()-based polling (which
only works for WinSock sockets even today, as far as I know) technique
used by IPC::Run with the need to guarantee that the child process
properly pushes all data it wrote to STDOUT before exiting.
Near as I can tell without digging in to the gory details, WinSock is a
DLL. So, if you write data to a socket and then exit, that data could
still be in userspace when the DLL is unloaded as a result of your exit.
Your data is lost. Doublecheck that explanation, it's a reasoned guess
based on the behavior I saw. When I redirected a child process to
read/write from/to TCP sockets--as recommended in at least one MS
knowledgebase article--I saw trailing output from the child disappear
more often than not.
However, anonymous pipes don't seem to have this problem on NT-based
platforms; though they don't work with select(). So--grab your inflight
discomfort bags now please--IPC::Run uses "pump" processes so that the
child does I/O to pipes and the parent does I/O to TCP sockets. These
pumps also address line-end conversion details.
Everybody's happy, but performance could be better and it's an ungainly
machine built of barely compatible ducts taped together with blowers and
fans.
IPC::Run3 works around it by only offering batch mode interaction (a la
system()) with redirection, which allows it to spool all stdin, stdout,
and stderr to temp files and everyone is happy, especially if the child
runs so fast that these files never get written to disk. Sucky disk
space implications if you need to move big hunks of data.
A more better approach for IPC::Run would be to cut away from select()
on this platform and use WaitForMultipleObjects() (IIRC) and anonymous
pipes, but looking through the Win32:: space I couldn't find anything
that allowed me to take pipe handles and poll them for I/O to do.
However, if you're doing single-child-process things and the syntactic,
um, sugar of run()'s redirections don't help you, Open2 and Open3 may be
better solutions than IPC::Run or IPC::Run3.
For Open3(), you run the risk of deadlocks unless you are *very*
careful (as I assume CPANPLUS is) or you use polling/nonblocking I/O.
I'm not sure how CPANPLUS avoids deadlocks.
Note that there are several deadlock situations. The two that I
experienced with Open3 are:
1. Parent blocks reading child output while child blocks waiting for
more input.
2. Parent blocks reading child output while child experiences a problem
and dumps lots of output (say a confess() stack dump or a BFD data dump)
to its STDERR and blocks due to a full stderr buffer.
- Barrie
Jos I. Boumans
On Feb 15, 2005, at 8:40 PM, Barrie Slaymaker wrote:
> On Tue, Feb 15, 2005 at 04:40:57PM +0000, Steve Hay wrote:
>>
>> Can you desribe what the problem is? Fixing IPC::Open3 for Win32
>> would
>> be the cleanest solution, though I realise that may not be possible.
>
> fork & execute's not such a big deal, IPC::Open3 does that ok. Open3()
> has got special spawning code with Ilya's (I think) system 1, ... hack
> that avoids fork emulation.
>
> The hard part for me was getting I/O pipes to work with select() or to
> get TCP sockets to work reliably with subprocesses. I'm not a Win32
> guru and at the time I wrote this, I wanted to get it working on Win32
> without requiring XS code or an upgrade to the Win32:: modules.
The problem was using piped commands in the system call, which
IPC::Run3 (according to its manpage) and IPC::Open3 couldn't do (if i'm
wrong please correct me, this assumption has been made a while ago).
Thinking about it, the only pipes used by CPANPLUS are probably those
done in Archive::Extract to grab gzip output (gzip -cd | tar xf - and
plain old gzip -cd). If Archive::Tar is core as well, these could
probably be avoided from CPANPLUS land, eliminating the need for pipes.
At which point, maybe IPC::Run3 or IPC::Open3 can do what we need.
Thoughts?
--
Jos Boumans
From kid's Superman costume for Halloween (stitched into the cape's
tag) -- "Warning: Use of This Device Does Not Enable
Wearer To Fly."
CPANPLUS http://cpanplus.sf.net
Nicholas Clark
> However, anonymous pipes don't seem to have this problem on NT-based
> platforms; though they don't work with select(). So--grab your inflight
> discomfort bags now please--IPC::Run uses "pump" processes so that the
> child does I/O to pipes and the parent does I/O to TCP sockets. These
> pumps also address line-end conversion details.
Aha yes. There ought to be a TODO to solve this. Most of the work is done.
I don't know how Win32 currently does anonymous pipes, but it's possible
to fake up a pipe with local TCP sockets. The plan was to replace pipe()
on Win32 with a call to socketpair(), so that pipes would also be selectable.
The C code is in the core to do this - see Perl_my_socketpair in util.c:
http://public.activestate.com/cgi-bin/perlbrowse?file=util.c&rev=
It would be simple to convert that code to perl, ship it as part of IPC::Run,
use it to make pipes on Win32 (or anywhere, if that's easier) and hence get
select()able pipes.
Meanwhile, is anyone with Win32 and C knowledge able to finish the job on
core pipe() to get it to use socketpair() on Win32 rather than the RTL's
pipe?
Nicholas Clark
Steve Hay
>On Tue, Feb 15, 2005 at 04:40:57PM +0000, Steve Hay wrote:
>
>
>>Can you desribe what the problem is? Fixing IPC::Open3 for Win32 would
>>be the cleanest solution, though I realise that may not be possible.
>>
>>
>
>fork & execute's not such a big deal, IPC::Open3 does that ok. Open3()
>has got special spawning code with Ilya's (I think) system 1, ... hack
>that avoids fork emulation.
>
>The hard part for me was getting I/O pipes to work with select() or to
>get TCP sockets to work reliably with subprocesses. I'm not a Win32
>guru and at the time I wrote this, I wanted to get it working on Win32
>without requiring XS code or an upgrade to the Win32:: modules.
>
>[...]
>Everybody's happy, but performance could be better and it's an ungainly
>machine built of barely compatible ducts taped together with blowers and
>fans.
>
>
So are you saying that IPC::Open3 *can* do piped commands, but just that
it isn't very nice behind the scenes?
Steve Hay
>On Feb 15, 2005, at 8:40 PM, Barrie Slaymaker wrote:
>
>
>
>>On Tue, Feb 15, 2005 at 04:40:57PM +0000, Steve Hay wrote:
>>
>>
>>>Can you desribe what the problem is? Fixing IPC::Open3 for Win32
>>>would
>>>be the cleanest solution, though I realise that may not be possible.
>>>
>>>
>>fork & execute's not such a big deal, IPC::Open3 does that ok. Open3()
>>has got special spawning code with Ilya's (I think) system 1, ... hack
>>that avoids fork emulation.
>>
>>The hard part for me was getting I/O pipes to work with select() or to
>>get TCP sockets to work reliably with subprocesses. I'm not a Win32
>>guru and at the time I wrote this, I wanted to get it working on Win32
>>without requiring XS code or an upgrade to the Win32:: modules.
>>
>>
>
>The problem was using piped commands in the system call, which
>IPC::Run3 (according to its manpage) and IPC::Open3 couldn't do (if i'm
>wrong please correct me, this assumption has been made a while ago).
>
>Thinking about it, the only pipes used by CPANPLUS are probably those
>done in Archive::Extract to grab gzip output (gzip -cd | tar xf - and
>plain old gzip -cd). If Archive::Tar is core as well, these could
>probably be avoided from CPANPLUS land, eliminating the need for pipes.
>At which point, maybe IPC::Run3 or IPC::Open3 can do what we need.
>
>Thoughts?
>
If IPC::Open3 does work then I guess CPANPLUS should use it, and we
don't need IPC::Run in the core.
If not then either someone needs to look at Nicholas' suggestions for
improving pipes on Win32 so that IPC::Run's requirement for Win32::
modules can be avoided, or failing that including Archive::Tar sounds
like the best solution to me, although Rafael said previously that he
wasn't happy about that --
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-01/msg00503.html.
Vadim Konovalov
> an ungainly
> >machine built of barely compatible ducts taped together with
> blowers and
> >fans.
> >
> >
> So are you saying that IPC::Open3 *can* do piped commands,
> but just that
> it isn't very nice behind the scenes?
not only performance.
It starts $^X as subprocess whith some parameters to manage Win32 pipes and
assumes that it starts another instance of Perl (or two) to help processing.
This fails miserably and silently when invoked from application, which
embeds perl, where $^X is something else. It just dont work and no warning
during execution or mentioning of such a limitation.
I don't know how to fix this however.
Best regards,
Vadim.
Nicholas Clark
> > So are you saying that IPC::Open3 *can* do piped commands,
> > but just that
> > it isn't very nice behind the scenes?
>
> not only performance.
>
> It starts $^X as subprocess whith some parameters to manage Win32 pipes and
> assumes that it starts another instance of Perl (or two) to help processing.
> This fails miserably and silently when invoked from application, which
> embeds perl, where $^X is something else. It just dont work and no warning
> during execution or mentioning of such a limitation.
>
> I don't know how to fix this however.
I couldn't find the literal text $^X in Open3.pm, so I'm not sure where the
buggy code is. What did I miss?
Nicholas Clark
Vadim Konovalov
> > > but just that
> > > it isn't very nice behind the scenes?
> >
> > not only performance.
> >
> > It starts $^X as subprocess whith some parameters to manage
> Win32 pipes and
> > assumes that it starts another instance of Perl (or two) to
> help processing.
> > This fails miserably and silently when invoked from
> application, which
> > embeds perl, where $^X is something else. It just dont work
> and no warning
> > during execution or mentioning of such a limitation.
> >
> > I don't know how to fix this however.
>
> I couldn't find the literal text $^X in Open3.pm, so I'm not
> sure where the
> buggy code is. What did I miss?
that is inside IPC::Run, which uses IPC::Run::Win32IO on win32 to do pipes
work, and it suffers this behaviour.
This closely relates in general, but probably does not fit to current
discussion.
Sorry, this probably should be in different topic..
Vadim.
Barrie Slaymaker
>Aha yes. There ought to be a TODO to solve this. Most of the work is done.
>I don't know how Win32 currently does anonymous pipes, but it's possible
>to fake up a pipe with local TCP sockets.
>
The problem I had doing child process redirects with TCP sockets was
that trailing data got lost when the child exited; the sockets don't
seem to be closed gracefully. Does your code experience this problem?
- Barrie
Steve Hay
I think we've got confused here. Vadim previously reported a problem
with $^X in IPC::Run, not IPC::Open3:
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-02/msg00182.html
so I assume that's what he meant above. IPC::Run::Win32IO does indeed
use $^X at least twice. Just one more reason to prefer IPC::Open3 over
IPC::Run, I guess.
Nicholas Clark
I have no idea. I don't have access to any Win32 machines with compilers,
I didn't realise that this was a problem, and didn't test it. I assume that
it would have the same problem, if both it and your code did "nothing special".
Is this data loss a known issue with the Win32 C RTL?
Nicholas Clark
Barrie Slaymaker
Not sure; witnessed it a few years ago on NT. Can't reproduce this
morning on XP, so maybe it's fixed or was driver error back then. Best
to test it thoroughly, I suspect.
- Barrie
- Barrie
Rafael Garcia-Suarez
> If IPC::Open3 does work then I guess CPANPLUS should use it, and we
> don't need IPC::Run in the core.
>
> If not then either someone needs to look at Nicholas' suggestions for
> improving pipes on Win32 so that IPC::Run's requirement for Win32::
> modules can be avoided, or failing that including Archive::Tar sounds
> like the best solution to me, although Rafael said previously that he
> wasn't happy about that --
> http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-01/msg00503.html.
Well, my opinion is to maximise portability first, and secondly minimize size.
The two being not necessarily incompatible.
Rafael Garcia-Suarez
>
> Basically, if we can either get:
> a) pipes to work 'somehow' on win32 (that being ipc::run or a fix on
> ipc::open3 or teaching IPC::Cmd to use IPC::Run3).
I don't know whether we can hope to get a reliable version of this in
the near future, so give me hints, windows hackers :) If we don't, then
I'll remove IPC::Run and began integrating Archive::Tar and its
dependencies (notably Compress::Zlib).
> b) Get archive::tar in the core, so we can eliminate the only need for
> pipes by Archive::Extract. We'd still need to catch output buffers from
> commandline tools, but that might be doable (even on win32) with
> ipc::open3 or ipc::run3.
Jos I. Boumans
On Feb 16, 2005, at 4:26 PM, Rafael Garcia-Suarez wrote:
> Steve Hay wrote:
>> If IPC::Open3 does work then I guess CPANPLUS should use it, and we
>> don't need IPC::Run in the core.
Just to nuanciate; it's the modules that are used by cpanplus that does
this: Archive::Extract supports extracting tar archives via /bin/tar
and needs piping for that (gzip -cd | tar). It uses IPC::Cmd for this
(which is a way to use IPC::Run, IPC::Open3 and system calls, depending
on OS, availability and environment).
Basically, if we can either get:
a) pipes to work 'somehow' on win32 (that being ipc::run or a fix on
ipc::open3 or teaching IPC::Cmd to use IPC::Run3).
or
b) Get archive::tar in the core, so we can eliminate the only need for
pipes by Archive::Extract. We'd still need to catch output buffers from
commandline tools, but that might be doable (even on win32) with
ipc::open3 or ipc::run3.
--
Jos Boumans
'Real programmers use "cat > a.out"'
CPANPLUS http://cpanlus.sf.net
Nicholas Clark
> Basically, if we can either get:
> a) pipes to work 'somehow' on win32 (that being ipc::run or a fix on
> ipc::open3 or teaching IPC::Cmd to use IPC::Run3).
Please remind me, what's not working about pies on win32 here?
Nicholas Clark
Nicholas Clark
> It would be simple to convert that code to perl, ship it as part of IPC::Run,
This seems to be portable OS wise (well OS X, FreeBSD and Solaris), and works
on perl 5.005 as well as 5.8.
Nicholas Clark
#!/usr/bin/perl -w
use strict;
use Socket;
# The C prototype is int family, int type, int protocol, int fd[2]
# We're always going to generate a pair of TCP/IP sockets.
sub stream_socketpair {
# Is this really generating 3 independent anonymous filehandles?
my ($listener, $connector, $acceptor) = map {local *H; *H} 1..3;
my $tcp = getprotobyname('tcp');
socket $listener, AF_INET, SOCK_STREAM, $tcp
or die "First socket failed: $!";
# 0 for port means kernel chooses
bind $listener, sockaddr_in(0, INADDR_LOOPBACK) or die "Bind failed: $!";
listen $listener, 1 or die "Listen failed: $!";
socket $connector, AF_INET, SOCK_STREAM, $tcp
or die "Second socket failed: $!";
my $connect_addr = getsockname($listener);
if (!$connect_addr) {
die "first getsockname failed: $!";
}
connect $connector, $connect_addr or die "connect failed: $!";
accept $acceptor, $listener or die "accept failed: $!";
close $listener or die "close failed: $!";
# Paranoia check - are we really talking to ourselves
my $accepted_from = getpeername($acceptor);
if (!$accepted_from) {
die "getpeername failed: $!";
}
# Note that this is not going to be the same port number as the listening
# socket, so we can't just use the address we connected to above.
my $connected_to = getsockname($connector);
if (!$connected_to) {
die "second getsockname failed: $!";
}
if ($accepted_from ne $connected_to) {
die "Consistency check failed - sockets not connected to each other "
. unpack ("H*", $accepted_from) . ' '
. unpack ("H*", $connected_to);
}
return ($connector, $acceptor);
}
my ($left, $right) = stream_socketpair;
select $left; $| = 1;
select $right; $| = 1;
select STDOUT;
print $left "Hello ";
print $right "World\n";
# Read 6 bytes on each. Else we hang
$/ = \6;
print scalar <$right>;
print scalar <$left>;
Yitzchak Scott-Thoennes
> # Is this really generating 3 independent anonymous filehandles?
> my ($listener, $connector, $acceptor) = map {local *H; *H} 1..3;
Yes. It even makes a weird kind of sense when you think of the GP as
being the value of a GV, just as the IV, NV, PV, RV, or UV is the value
of an SV, and what is returned from the block (and what is localized) is
the value, the GP (with a fresh GV wrapped around it).
It also makes sense that map {local *H; \*H} *doesn't* return refs to
unique globs, since the GV is never actually changed by the localization,
and once outside the block, the GP has been restored.
But I'd use Symbol::gensym anyway.
Steve Hay
>Jos I. Boumans wrote:
>
>
>>Basically, if we can either get:
>>a) pipes to work 'somehow' on win32 (that being ipc::run or a fix on
>>ipc::open3 or teaching IPC::Cmd to use IPC::Run3).
>>
>>
>
>I don't know whether we can hope to get a reliable version of this in
>the near future, so give me hints, windows hackers :) If we don't, then
>I'll remove IPC::Run and began integrating Archive::Tar and its
>dependencies (notably Compress::Zlib).
>
Both Nicholas and I have now asked exactly what the problem with
IPC::Open3 on Win32 is, but have not yet received an answer. Presumably
nobody is quite sure what, if anything, the problem is, so I thought I'd
try and find out.
The attached program contains what I believe is the relevant code,
lifted from CPANPLUS::Tools::Cmd, to try open IPC::Open3 vs IPC::Run.
Running "perl test.pl run" runs "$^X -v" via IPC::Run and produces this
output:
=====
C:\Temp>perl test.pl run
err=[0]
buffer=[
This is perl, v5.8.6 built for MSWin32-x86-perlio
Copyright 1987-2004, Larry Wall
Perl may be copied only under the terms of either the Artistic License
or the
GNU General Public License, which may be found in the Perl 5 source kit.
Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'. If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.
]
buferr=[]
bufout=[
This is perl, v5.8.6 built for MSWin32-x86-perlio
Copyright 1987-2004, Larry Wall
Perl may be copied only under the terms of either the Artistic License
or the
GNU General Public License, which may be found in the Perl 5 source kit.
Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'. If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.
]
=====
Running "perl test.pl open" runs the same $cmd via IPC::Open3 but
produces just this:
=====
C:\Temp>perl test.pl open
err=[0]
buffer=[]
buferr=[]
bufout=[]
=====
And that's a $cmd that doesn't even involve pipes.
Changing the $cmd to qq[$^X -e "open FH, '>C:/Temp/testout'"] reveals
that the $cmd definitely is being executed; we're just losing all its
output.
But IPC::Open3 passes all its tests OK (and the tests include gathering
output from the child), so perhaps the _open3_run() subroutine is doing
something wrong or non-portable?
Adding some debug, I find that $sel->can_read() is returning an empty
list. If I change
while (my @ready = $sel->can_read) {
foreach my $fh (@ready) { # loop through buffered handles
...
}
}
to
foreach my $fh ($outfh, $errfh) {
...
}
then the output is now as expected.
Presumably IO::Select->can_read() doesn't work on Win32 because it uses
a 4-arg select(), which is only implemented for sockets on Win32. Is
the above change safe, or did we need to call can_read() for some
reason? Would the proposed "selectable pipes" change have any impact on
this?
What do I need to do to test out whether IPC::Open3 is working with
pipes or not? (Open3.t doesn't seem to include any such tests.)
- Steve
------------------------------------------------
This email has been scanned for viruses and content by the Radan Computational Webshield Appliances.
Ton Hospel
Steve Hay <stev...@uk.radan.com> writes:
> Adding some debug, I find that $sel->can_read() is returning an empty
> list. If I change
>
> while (my @ready = $sel->can_read) {
> foreach my $fh (@ready) { # loop through buffered handles
> ...
> }
> }
>
> to
>
> foreach my $fh ($outfh, $errfh) {
> ...
> }
>
> then the output is now as expected.
>
> Presumably IO::Select->can_read() doesn't work on Win32 because it uses
> a 4-arg select(), which is only implemented for sockets on Win32. Is
> the above change safe, or did we need to call can_read() for some
> reason? Would the proposed "selectable pipes" change have any impact on
> this?
>
tries to write more than can be buffered in the communication channel to
err_fh and only then bothers to write to $out_fh. In that case you will
deadlock. the source will wait until enough gets drained from err_fh so it
can write again, and you will wait on $out_fh, which hasn't gotten anything
yet and now never will.
So whether that is relevant for your case depends on what you're running
(how it uses its output channels).
Jos I. Boumans
On Feb 18, 2005, at 2:10 PM, Steve Hay wrote:
> Both Nicholas and I have now asked exactly what the problem with
> IPC::Open3 on Win32 is, but have not yet received an answer.
> Presumably nobody is quite sure what, if anything, the problem is, so
> I thought I'd try and find out.
Good initiative :)
> The attached program contains what I believe is the relevant code,
> lifted from CPANPLUS::Tools::Cmd, to try open IPC::Open3 vs IPC::Run.
Best to lift from IPC::Cmd, as that's the library we're actually using
-- cpanplus::tools::cmd is rather obsolete (luckily the ipc::open3
implementation hasn't changed.. the ipc::run one has though)
>
> Running "perl test.pl open" runs the same $cmd via IPC::Open3 but
> produces just this:
>
> =====
> C:\Temp>perl test.pl open
> err=[0]
> buffer=[]
> buferr=[]
> bufout=[]
> =====
>
> And that's a $cmd that doesn't even involve pipes.
Yup, that's pretty much our experience.
> Changing the $cmd to qq[$^X -e "open FH, '>C:/Temp/testout'"] reveals
> that the $cmd definitely is being executed; we're just losing all its
> output.
[...]
> Presumably IO::Select->can_read() doesn't work on Win32 because it
> uses a 4-arg select(), which is only implemented for sockets on Win32.
> Is the above change safe, or did we need to call can_read() for some
> reason? Would the proposed "selectable pipes" change have any impact
> on this?
I have no idea myself, i'm not much of an IPC guru, i just RTFM'd on
this. But if others can assure me it's safe & portable, it's an easy
patch to do.
> What do I need to do to test out whether IPC::Open3 is working with
> pipes or not? (Open3.t doesn't seem to include any such tests.)
That might be a very useful addition to the open3.t tests...
Perhaps something like this would work as a test case:
"$^X -e'print 1' | $^X -e'print 1 + <>'"
--
Jos Boumans
"If superman is so smart, why does he wear underpants over his
trousers?"
CPANPLUS http://cpanplus.sf.net
Steve Hay
I don't know what commands are being run by IPC::Cmd and whether or not
they would be likely to be affected by these problems, but I guess
removing the can_read() check isn't ideal if it *can* cause problems,
even if it doesn't affect us now -- we'd only be storing up problems for
the future.
Does anyone know if the selectable pipes work that Nicholas mentioned
would be likely to make the 4-arg select(), and hence can_read(), work?
I don't really know what would be involved in finishing off the work in
question, but if someone can give me some pointers then I'm willing to
give it a go.
Alternatively, Nicholas also mentioned the possibility of converting the
C code to Perl (and indeed, actually did so) and putting that into
IPC::Run. Again, I've no idea how to fit that Perl code into IPC::Run
(or IPC::Cmd). Can someone help out?
Steve Hay
>On Feb 18, 2005, at 2:10 PM, Steve Hay wrote:
>
>
>
>
>>The attached program contains what I believe is the relevant code,
>>lifted from CPANPLUS::Tools::Cmd, to try open IPC::Open3 vs IPC::Run.
>>
>>
>Best to lift from IPC::Cmd, as that's the library we're actually using
>-- cpanplus::tools::cmd is rather obsolete (luckily the ipc::open3
>implementation hasn't changed.. the ipc::run one has though)
>
>
OK, I've updated the attached program to use code from IPC::Cmd.
>
>
>
>
>
>>Presumably IO::Select->can_read() doesn't work on Win32 because it
>>uses a 4-arg select(), which is only implemented for sockets on Win32.
>> Is the above change safe, or did we need to call can_read() for some
>>reason? Would the proposed "selectable pipes" change have any impact
>>on this?
>>
>>
>I have no idea myself, i'm not much of an IPC guru, i just RTFM'd on
>this. But if others can assure me it's safe & portable, it's an easy
>patch to do.
>
>
Ton says it isn't (always) safe, so I think that's not the way to go :(
>
>
>>What do I need to do to test out whether IPC::Open3 is working with
>>pipes or not? (Open3.t doesn't seem to include any such tests.)
>>
>>
>That might be a very useful addition to the open3.t tests...
>Perhaps something like this would work as a test case:
> "$^X -e'print 1' | $^X -e'print 1 + <>'"
>
I've put that $cmd into the attached test.pl, but something's not
right. The IPC::Run version with this new $cmd doesn't work. It gives
me the error:
'ARRAY' not allowed as a source for input redirection at test.pl line 68
I think the scan for $special_chars has got confused over the "<>" in
the Perl one-liner -- it thinks they are shell redirection characters.
Presumably this would affect other OS's too, and is really a bug in
IPC::Cmd?
Using IPC::Open3 I get the same as "perl -v" gave me before:
err=[0]
buffer=[]
buferr=[]
bufout=[]
and again removing the can_read() calls "fixes" it:
err=[0]
buffer=[2]
buferr=[]
bufout=[2]
That's great news, as it gives us hope that IPC::Open3 + pipes is OK.
So the only issue here is that can_read() doesn't work because pipes
aren't selectable on Win32. I'm therefore interested what work would be
involved (either using C code in the core, or equivalent Perl code in
IPC::Cmd) to fix it.
Jos I. Boumans
On Feb 23, 2005, at 12:11 PM, Steve Hay wrote:
>>> What do I need to do to test out whether IPC::Open3 is working with
>>> pipes or not? (Open3.t doesn't seem to include any such tests.)
>>
>> That might be a very useful addition to the open3.t tests...
>> Perhaps something like this would work as a test case:
>> "$^X -e'print 1' | $^X -e'print 1 + <>'"
>>
> I've put that $cmd into the attached test.pl, but something's not
> right. The IPC::Run version with this new $cmd doesn't work. It
> gives me the error:
>
> 'ARRAY' not allowed as a source for input redirection at test.pl line
> 68
>
> I think the scan for $special_chars has got confused over the "<>" in
> the Perl one-liner -- it thinks they are shell redirection characters.
> Presumably this would affect other OS's too, and is really a bug in
> IPC::Cmd?
Hmm, sure seems like it... dang... we have to do some massaging of
arguments towards ipc::run when it comes to capturing buffers etc...
> Using IPC::Open3 I get the same as "perl -v" gave me before:
>
> err=[0]
> buffer=[]
> buferr=[]
> bufout=[]
>
> and again removing the can_read() calls "fixes" it:
>
> err=[0]
> buffer=[2]
> buferr=[]
> bufout=[2]
>
> That's great news, as it gives us hope that IPC::Open3 + pipes is OK.
That's indeed great news... i'd be much in favour now that if we can
get ipc::open3 to work right (basically fixing the can_read call), to
just favour that over IPC::Run, meaning one less module that needs to
go core.
> So the only issue here is that can_read() doesn't work because pipes
> aren't selectable on Win32. I'm therefore interested what work would
> be involved (either using C code in the core, or equivalent Perl code
> in IPC::Cmd) to fix it.
The fix to IPC::Cmd is to use another call that DTRT rather than
can_read. Ideally of course, can_read is just fixed and IPC::Cmd will
be changed to favour IPC::Open3 (newest version) over IPC::Run.