with non-existing command "asdfasdf" returns perfectly fine handles.
I would expect an exception.
You can even hGetContents on stdout: You just get "".
I find this highly counter-intuitive. Is this intended?
Thanks
Niklas
PS: I checked how some other programming languages do this:
On Sun, Aug 12, 2012 at 6:18 PM, Niklas Hambüchen <m...@nh2.me> wrote: > I just came across the fact that running
> createProcess (proc "asdfasdf" [])
> with non-existing command "asdfasdf" returns perfectly fine handles. > I would expect an exception. > You can even hGetContents on stdout: You just get "".
> I find this highly counter-intuitive. Is this intended?
Yes, I ran into the same thing a while back. The problem is that the subprocess has already been forked off before it runs exec() and finds out the file doesn't exist. The reason python reports the right error is that it sets up a pipe from child to parent to communicate just this error. It's more friendly, but on the other hand the implementation is more complicated.
If you don't want to hack up the whole send-exception-over-the-pipe thing, the easiest thing to do is to wait for the processes's return code. If you don't want to do that, you can at least have the subproces log, e.g.:
loggedProcess :: Process.CreateProcess -> IO (Maybe IO.Handle, Maybe IO.Handle, Maybe IO.Handle, Process.ProcessHandle) loggedProcess create = do r@(_, _, _, pid) <- Process.createProcess create Concurrent.forkIO $ do code <- Process.waitForProcess pid case code of Exit.ExitFailure c -> notice $ "subprocess " ++ show (binaryOf create) ++ " failed: " ++ if c == 127 then "binary not found" else show c _ -> return () return r where binaryOf create = case Process.cmdspec create of Process.RawCommand fn _ -> fn Process.ShellCommand cmd -> fst $ break (==' ') cmd
As an aside, I've had the idea to at some point go look at the latest python's version of the subprocess module and see about porting it to haskell, or at least make sure the haskell version doesn't suffer from problems fixed in the python one. They went through a lot of iterations trying to get it right (and earlier python versions are broken in one way or another) and we might as well build on their work.
On Sun, 2012-08-12 at 23:18 -0700, Evan Laforge wrote:
> Yes, I ran into the same thing a while back. The problem is that the
> subprocess has already been forked off before it runs exec() and finds
> out the file doesn't exist.
Given how astonishingly common it is to pass an invalid executable name
and/or path, wouldn't it be worth doing a quick probe to see if the file
exists before createProcess actually forks?
[It's not like the effort the OS is going to do for the stat is going to
be thrown away; whether that call pulls it up off of disk or the one
after the fork that exec will do doesn't matter]
In Unix, at least, "check, then act" is generally considered unwise:
1. Something can go wrong between checking and acting.
2. You might not be checking the right thing(s). In this case, the fact
that the file exists is not useful if you don't have permission to execute
it. You may not be able to determine whether you have the appropriate
permissions without fairly deep manipulation of ACLs.
3. Even if the OS has the info at hand, making the system call(s) necessary
to get it is not free. Various non-free things happen every time control
passes between user-space and kernel-space.
On Aug 13, 2012 4:17 AM, "Andrew Cowie" <and...@operationaldynamics.com>
wrote:
> On Sun, 2012-08-12 at 23:18 -0700, Evan Laforge wrote:
> > Yes, I ran into the same thing a while back. The problem is that the
> > subprocess has already been forked off before it runs exec() and finds
> > out the file doesn't exist.
> Given how astonishingly common it is to pass an invalid executable name
> and/or path, wouldn't it be worth doing a quick probe to see if the file
> exists before createProcess actually forks?
> [It's not like the effort the OS is going to do for the stat is going to
> be thrown away; whether that call pulls it up off of disk or the one
> after the fork that exec will do doesn't matter]
> In Unix, at least, "check, then act" is generally considered unwise:
> 1. Something can go wrong between checking and acting.
> 2. You might not be checking the right thing(s). In this case, the fact
> that the file exists is not useful if you don't have permission to execute
> it. You may not be able to determine whether you have the appropriate
> permissions without fairly deep manipulation of ACLs.
> 3. Even if the OS has the info at hand, making the system call(s)
> necessary to get it is not free. Various non-free things happen every time
> control passes between user-space and kernel-space.
> On Aug 13, 2012 4:17 AM, "Andrew Cowie" <and...@operationaldynamics.com>
> wrote:
>> On Sun, 2012-08-12 at 23:18 -0700, Evan Laforge wrote:
>> > Yes, I ran into the same thing a while back. The problem is that the
>> > subprocess has already been forked off before it runs exec() and finds
>> > out the file doesn't exist.
>> Given how astonishingly common it is to pass an invalid executable name
>> and/or path, wouldn't it be worth doing a quick probe to see if the file
>> exists before createProcess actually forks?
>> [It's not like the effort the OS is going to do for the stat is going to
>> be thrown away; whether that call pulls it up off of disk or the one
>> after the fork that exec will do doesn't matter]
> ... or at least make sure the haskell version doesn't suffer from
> problems fixed in the python one.
Exactly. This morning I'm reading suggested solutions that would
work only some of the time, or on only some platforms, which wouldn't
be satisfactory in the long run.
Though speaking of platforms, I guess one large headache would be
what to do about Microsoft operating systems. Given the unusual
nature of these functions (I mean, what operating-system-independent
command are you going to invoke, anyway?), maybe it would be OK for
the more elaborate support functions to be POSIX / Windows specific.
At the level where people are redirecting the output FD and not the
error FD, etc.
On Mon, Aug 13, 2012 at 7:26 AM, Alexander Kjeldaas <
alexander.kjeld...@gmail.com> wrote:
> This isn't that hard - a pipe shouldn't be needed anymore. Just require a
> post-2003 glibc.
So, we are desupporting the *BSDs and OS X (and Solaris etc.) now? glibc
is only used on Linux and the Hurd (and debian kfreebsd, if that hasn't
fallen on its face yet).
POSIX has some new spawn-type calls, btw, but I don't know how widely
implemented they are or how buggy they are.
-- brandon s allbery allber...@gmail.com
wandering unix systems administrator (available) (412) 475-9364 vm/sms
On Mon, Aug 13, 2012 at 10:23 AM, Donn Cave <d...@avvanta.com> wrote:
> Though speaking of platforms, I guess one large headache would be
> what to do about Microsoft operating systems. Given the unusual
Microsoft provides APIs that work as is for this, by my understanding; it's
the POSIX fork/exec model that makes life difficult.
-- brandon s allbery allber...@gmail.com
wandering unix systems administrator (available) (412) 475-9364 vm/sms
> On Mon, Aug 13, 2012 at 10:23 AM, Donn Cave <d...@avvanta.com> wrote:
>> Though speaking of platforms, I guess one large headache would be
>> what to do about Microsoft operating systems. Given the unusual
> Microsoft provides APIs that work as is for this, by my understanding; it's
> the POSIX fork/exec model that makes life difficult.
Or interesting, anyway. I wasn't thinking of the `exception in child'
problem here, so much as more generally, how much is a fully cross-platform
API worth, in a situation where the eventual application of the API is
inherently unlikely to be of a cross platform nature. The Python version
goes to some length, but can't fully resolve the inconsistencies. That's
OK if someone wants to go to the trouble, but if I'm right about inherent
platform dependence, it runs the risk of being more irritating than helpful!
On 13/08/2012, at 11:26 PM, Alexander Kjeldaas wrote:
> This isn't that hard - a pipe shouldn't be needed anymore. Just require a post-2003 glibc.
> fexecve is a system call in most BSDs. It is also implemented in glibc using a /proc hack.
fexecve is now in the Single Unix Specification, based on
POSIX as of 2008, I believe. However,
http://www.gnu.org/software/gnulib/manual/html_node/fexecve.html says
Portability problems not fixed by Gnulib:
* This function is missing on many non-glibc platforms: MacOS X 10.5, FreeBSD 6.0,
NetBSD 5.0, OpenBSD 3.8, Minix 3.1.8, AIX 5.1, HP-UX 11, IRIX 6.5, OSF/1 5.1,
Solaris 11 2010-11, Cygwin 1.5.x, mingw, MSVC 9, Interix 3.5, BeOS.
That warning doesn't seem to be fully up to date. I'm using MacOS X 10.6.8
and fexecve() isn't in the manuals or in <unistd.h>.
> On 13/08/2012, at 11:26 PM, Alexander Kjeldaas wrote:
> > This isn't that hard - a pipe shouldn't be needed anymore. Just require
> a post-2003 glibc.
> > fexecve is a system call in most BSDs. It is also implemented in glibc
> using a /proc hack.
> fexecve is now in the Single Unix Specification, based on
> POSIX as of 2008, I believe. However,
> http://www.gnu.org/software/gnulib/manual/html_node/fexecve.html > says
> Portability problems not fixed by Gnulib:
> * This function is missing on many non-glibc platforms: MacOS X 10.5,
> FreeBSD 6.0,
> NetBSD 5.0, OpenBSD 3.8, Minix 3.1.8, AIX 5.1, HP-UX 11, IRIX 6.5,
> OSF/1 5.1,
> Solaris 11 2010-11, Cygwin 1.5.x, mingw, MSVC 9, Interix 3.5, BeOS.
> That warning doesn't seem to be fully up to date. I'm using MacOS X 10.6.8
> and fexecve() isn't in the manuals or in <unistd.h>.
So support is pretty good, I'd say. For non-modern systems, checking the
existence of the file first is possible. The race isn't important, and one
can always upgrade to a modern operating system.
> On 13 August 2012 23:49, Richard O'Keefe <o...@cs.otago.ac.nz> wrote:
>> On 13/08/2012, at 11:26 PM, Alexander Kjeldaas wrote:
>> > This isn't that hard - a pipe shouldn't be needed anymore. Just require
>> > a post-2003 glibc.
>> > fexecve is a system call in most BSDs. It is also implemented in glibc
>> > using a /proc hack.
>> fexecve is now in the Single Unix Specification, based on
>> POSIX as of 2008, I believe. However,
>> http://www.gnu.org/software/gnulib/manual/html_node/fexecve.html >> says
>> Portability problems not fixed by Gnulib:
>> * This function is missing on many non-glibc platforms: MacOS X 10.5,
>> FreeBSD 6.0,
>> NetBSD 5.0, OpenBSD 3.8, Minix 3.1.8, AIX 5.1, HP-UX 11, IRIX 6.5,
>> OSF/1 5.1,
>> Solaris 11 2010-11, Cygwin 1.5.x, mingw, MSVC 9, Interix 3.5, BeOS.
>> That warning doesn't seem to be fully up to date. I'm using MacOS X
>> 10.6.8
>> and fexecve() isn't in the manuals or in <unistd.h>.
> So support is pretty good, I'd say. For non-modern systems, checking the
> existence of the file first is possible. The race isn't important, and one
> can always upgrade to a modern operating system.
The check would be unreliable, the file's existence doesn't imply that
it's executable. Furthermore it would add unnecessary overhead,
createProcess can be run thousands of times in a program and should be
lean and mean.
> 2012/8/14 Alexander Kjeldaas <alexander.kjeld...@gmail.com>:
> > On 13 August 2012 23:49, Richard O'Keefe <o...@cs.otago.ac.nz> wrote:
> >> On 13/08/2012, at 11:26 PM, Alexander Kjeldaas wrote:
> >> > This isn't that hard - a pipe shouldn't be needed anymore. Just
> require
> >> > a post-2003 glibc.
> >> > fexecve is a system call in most BSDs. It is also implemented in
> glibc
> >> > using a /proc hack.
> >> fexecve is now in the Single Unix Specification, based on
> >> POSIX as of 2008, I believe. However,
> >> http://www.gnu.org/software/gnulib/manual/html_node/fexecve.html > >> says
> >> Portability problems not fixed by Gnulib:
> >> * This function is missing on many non-glibc platforms: MacOS X 10.5,
> >> FreeBSD 6.0,
> >> NetBSD 5.0, OpenBSD 3.8, Minix 3.1.8, AIX 5.1, HP-UX 11, IRIX 6.5,
> >> OSF/1 5.1,
> >> Solaris 11 2010-11, Cygwin 1.5.x, mingw, MSVC 9, Interix 3.5, BeOS.
> >> That warning doesn't seem to be fully up to date. I'm using MacOS X
> >> 10.6.8
> >> and fexecve() isn't in the manuals or in <unistd.h>.
> > FreeBSD 8.0 is covered.
> > OpenBSD not covered
> > OS X not covered
> > So support is pretty good, I'd say. For non-modern systems, checking the
> > existence of the file first is possible. The race isn't important, and
> one
> > can always upgrade to a modern operating system.
> The check would be unreliable, the file's existence doesn't imply that
> it's executable.
See access(2)
> Furthermore it would add unnecessary overhead,
> createProcess can be run thousands of times in a program and should be
> lean and mean.
Just to keep the bikeshedding doing, I'm going to state as a fact that
running performance sensitive *server* workload on any unix other than
Linux is purely of theoretical interest. No sane person would do it.
Therefore, from a performance overhead, Linux performance is the only
important performance measure.
But even given the above, the overhead we're talking about is minuscule. A
program like '/bin/echo -n ''' which does exactly *nothing*, requires
35(!) system calls to do its job :-).
A more complex program like 'id' requires 250 system calls!
Also, to see just how minuscule this is, the dynamic linker, ld-linux.so
does a few extra access(2) system calls *to the same file*,
/etc/ld.so.hwcaps, on startup of every dynamically linked executable. 2 in
the 'echo' case, and 8 in the 'id' case above. Even the glibc folks
haven't bothered to optimize those syscalls away.
Quoth Alexander Kjeldaas <alexander.kjeld...@gmail.com>,
> See access(2)
... a classic "code smell" in UNIX programming, for the same reasons.
We can solve this problem in an efficient way that works well, and equally
well, on any POSIX platform that supports F_CLOEXEC on pipes, and I can't
think of anything that doesn't. The appended code is the basic gist of it.
This was not invented by the Python world, but as suggested it's one of
the things that we'd get from a review of their subprocess module.
Donn
spawn file cmd env = do
(e0, e1) <- pipe
fcntlSetFlag e1 F_CLOEXEC
t <- fork (fex e0 e1)
close e1
rx <- readFd e0 256
if null rx
then return t
else ioerr (chrToErrType rx) file
where
fex e0 e1 = do
close e0
catch (execve file cmd env)
(\ e -> writeFd e1 (errToChr e : ioeGetErrorString e))
ioerr (e, s) file = ioError (mkIOError e s Nothing (Just file))
> Quoth Alexander Kjeldaas <alexander.kjeld...@gmail.com>,
>> See access(2)
> ... a classic "code smell" in UNIX programming, for the same reasons.
> We can solve this problem in an efficient way that works well, and equally
> well, on any POSIX platform that supports F_CLOEXEC on pipes, and I can't
> think of anything that doesn't. The appended code is the basic gist of it.
> This was not invented by the Python world, but as suggested it's one of
> the things that we'd get from a review of their subprocess module.
> Donn
> spawn file cmd env = do
> (e0, e1) <- pipe
> fcntlSetFlag e1 F_CLOEXEC
> t <- fork (fex e0 e1)
> close e1
> rx <- readFd e0 256
> if null rx
> then return t
> else ioerr (chrToErrType rx) file
> where
> fex e0 e1 = do
> close e0
> catch (execve file cmd env)
> (\ e -> writeFd e1 (errToChr e : ioeGetErrorString e))
> ioerr (e, s) file = ioError (mkIOError e s Nothing (Just file))