Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
createProcess running non-existent programs
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  15 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Niklas Hambüchen  
View profile  
 More options Aug 12 2012, 9:18 pm
From: Niklas Hambüchen <m...@nh2.me>
Date: Mon, 13 Aug 2012 02:18:56 +0100
Local: Sun, Aug 12 2012 9:18 pm
Subject: [Haskell-cafe] createProcess running non-existent programs
I just came across the fact that running

    createProcess (proc "asdfasdf" [])

with non-existing command "asdfasdf" returns perfectly fine handles.
I would expect an exception.
You can even hGetContents on stdout: You just get "".

I find this highly counter-intuitive. Is this intended?

Thanks
Niklas

PS: I checked how some other programming languages do this:

Python:
    import subprocess; subprocess.call("asdfasdf", shell=False)

    OSError: [Errno 2] No such file or directory

Ruby:
    exec("asdfasdf")

    Errno::ENOENT: No such file or directory - asdfasdf

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Evan Laforge  
View profile  
 More options Aug 13 2012, 2:18 am
From: Evan Laforge <qdun...@gmail.com>
Date: Sun, 12 Aug 2012 23:18:30 -0700
Local: Mon, Aug 13 2012 2:18 am
Subject: Re: [Haskell-cafe] createProcess running non-existent programs

On Sun, Aug 12, 2012 at 6:18 PM, Niklas Hambüchen <m...@nh2.me> wrote:
> I just came across the fact that running

>     createProcess (proc "asdfasdf" [])

> with non-existing command "asdfasdf" returns perfectly fine handles.
> I would expect an exception.
> You can even hGetContents on stdout: You just get "".

> I find this highly counter-intuitive. Is this intended?

Yes, I ran into the same thing a while back.  The problem is that the
subprocess has already been forked off before it runs exec() and finds
out the file doesn't exist.  The reason python reports the right error
is that it sets up a pipe from child to parent to communicate just
this error.  It's more friendly, but on the other hand the
implementation is more complicated.

If you don't want to hack up the whole send-exception-over-the-pipe
thing, the easiest thing to do is to wait for the processes's return
code.  If you don't want to do that, you can at least have the
subproces log, e.g.:

loggedProcess :: Process.CreateProcess -> IO (Maybe IO.Handle,
       Maybe IO.Handle, Maybe IO.Handle, Process.ProcessHandle)
loggedProcess create = do
    r@(_, _, _, pid) <- Process.createProcess create
    Concurrent.forkIO $ do
        code <- Process.waitForProcess pid
        case code of
            Exit.ExitFailure c -> notice $
                "subprocess " ++ show (binaryOf create) ++ " failed: "
                ++ if c == 127 then "binary not found" else show c
            _ -> return ()
    return r
    where
    binaryOf create = case Process.cmdspec create of
        Process.RawCommand fn _ -> fn
        Process.ShellCommand cmd -> fst $ break (==' ') cmd

As an aside, I've had the idea to at some point go look at the latest
python's version of the subprocess module and see about porting it to
haskell, or at least make sure the haskell version doesn't suffer from
problems fixed in the python one.  They went through a lot of
iterations trying to get it right (and earlier python versions are
broken in one way or another) and we might as well build on their
work.

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Andrew Cowie  
View profile  
 More options Aug 13 2012, 4:16 am
From: Andrew Cowie <and...@operationaldynamics.com>
Date: Mon, 13 Aug 2012 18:16:50 +1000
Local: Mon, Aug 13 2012 4:16 am
Subject: Re: [Haskell-cafe] createProcess running non-existent programs

On Sun, 2012-08-12 at 23:18 -0700, Evan Laforge wrote:
> Yes, I ran into the same thing a while back.  The problem is that the
> subprocess has already been forked off before it runs exec() and finds
> out the file doesn't exist.

Given how astonishingly common it is to pass an invalid executable name
and/or path, wouldn't it be worth doing a quick probe to see if the file
exists before createProcess actually forks?

[It's not like the effort the OS is going to do for the stat is going to
be thrown away; whether that call pulls it up off of disk or the one
after the fork that exec will do doesn't matter]

AfC
Sydney

  signature.asc
< 1K Download

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
David Feuer  
View profile  
 More options Aug 13 2012, 5:23 am
From: David Feuer <david.fe...@gmail.com>
Date: Mon, 13 Aug 2012 05:23:05 -0400
Local: Mon, Aug 13 2012 5:23 am
Subject: Re: [Haskell-cafe] createProcess running non-existent programs

In Unix, at least, "check, then act" is generally considered unwise:
1. Something can go wrong between checking and acting.
2. You might not be checking the right thing(s).  In this case, the fact
that the file exists is not useful if you don't have permission to execute
it. You may not be able to determine whether you have the appropriate
permissions without fairly deep manipulation of ACLs.
3. Even if the OS has the info at hand, making the system call(s) necessary
to get it is not free. Various non-free things happen every time control
passes between user-space and kernel-space.
On Aug 13, 2012 4:17 AM, "Andrew Cowie" <and...@operationaldynamics.com>
wrote:

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alexander Kjeldaas  
View profile  
 More options Aug 13 2012, 7:26 am
From: Alexander Kjeldaas <alexander.kjeld...@gmail.com>
Date: Mon, 13 Aug 2012 13:26:28 +0200
Local: Mon, Aug 13 2012 7:26 am
Subject: Re: [Haskell-cafe] createProcess running non-existent programs

This isn't that hard - a pipe shouldn't be needed anymore.  Just require a
post-2003 glibc.

fexecve is a system call in most BSDs.  It is also implemented in glibc
using a /proc hack.

http://www.kernel.org/doc/man-pages/online/pages/man3/fexecve.3.html

Apparently, there are proposals/RFCs to get a system called execveat into
the linux kernel which makes this work properly without /proc.

http://www.gossamer-threads.com/lists/linux/kernel/1574831

Alexander

On 13 August 2012 11:23, David Feuer <david.fe...@gmail.com> wrote:

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Donn Cave  
View profile  
 More options Aug 13 2012, 10:23 am
From: Donn Cave <d...@avvanta.com>
Date: Mon, 13 Aug 2012 07:23:42 -0700 (PDT)
Local: Mon, Aug 13 2012 10:23 am
Subject: Re: [Haskell-cafe] createProcess running non-existent programs
Quoth Evan Laforge <qdun...@gmail.com>,
...

> ... or at least make sure the haskell version doesn't suffer from
> problems fixed in the python one.

Exactly.  This morning I'm reading suggested solutions that would
work only some of the time, or on only some platforms, which wouldn't
be satisfactory in the long run.

Though speaking of platforms, I guess one large headache would be
what to do about Microsoft operating systems.  Given the unusual
nature of these functions (I mean, what operating-system-independent
command are you going to invoke, anyway?), maybe it would be OK for
the more elaborate support functions to be POSIX / Windows specific.
At the level where people are redirecting the output FD and not the
error FD, etc.

        Donn

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Brandon Allbery  
View profile  
 More options Aug 13 2012, 10:28 am
From: Brandon Allbery <allber...@gmail.com>
Date: Mon, 13 Aug 2012 10:28:50 -0400
Local: Mon, Aug 13 2012 10:28 am
Subject: Re: [Haskell-cafe] createProcess running non-existent programs

On Mon, Aug 13, 2012 at 7:26 AM, Alexander Kjeldaas <

alexander.kjeld...@gmail.com> wrote:
> This isn't that hard - a pipe shouldn't be needed anymore.  Just require a
> post-2003 glibc.

So, we are desupporting the *BSDs and OS X (and Solaris etc.) now?  glibc
is only used on Linux and the Hurd (and debian kfreebsd, if that hasn't
fallen on its face yet).

POSIX has some new spawn-type calls, btw, but I don't know how widely
implemented they are or how buggy they are.

--
brandon s allbery                                      allber...@gmail.com
wandering unix systems administrator (available)     (412) 475-9364 vm/sms

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Brandon Allbery  
View profile  
 More options Aug 13 2012, 10:29 am
From: Brandon Allbery <allber...@gmail.com>
Date: Mon, 13 Aug 2012 10:29:58 -0400
Local: Mon, Aug 13 2012 10:29 am
Subject: Re: [Haskell-cafe] createProcess running non-existent programs

On Mon, Aug 13, 2012 at 10:23 AM, Donn Cave <d...@avvanta.com> wrote:
> Though speaking of platforms, I guess one large headache would be
> what to do about Microsoft operating systems.  Given the unusual

Microsoft provides APIs that work as is for this, by my understanding; it's
the POSIX fork/exec model that makes life difficult.

--
brandon s allbery                                      allber...@gmail.com
wandering unix systems administrator (available)     (412) 475-9364 vm/sms

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Donn Cave  
View profile  
 More options Aug 13 2012, 11:14 am
From: Donn Cave <d...@avvanta.com>
Date: Mon, 13 Aug 2012 08:14:33 -0700 (PDT)
Local: Mon, Aug 13 2012 11:14 am
Subject: Re: [Haskell-cafe] createProcess running non-existent programs
Quoth Brandon Allbery <allber...@gmail.com>,

> On Mon, Aug 13, 2012 at 10:23 AM, Donn Cave <d...@avvanta.com> wrote:

>> Though speaking of platforms, I guess one large headache would be
>> what to do about Microsoft operating systems.  Given the unusual

> Microsoft provides APIs that work as is for this, by my understanding; it's
> the POSIX fork/exec model that makes life difficult.

Or interesting, anyway.  I wasn't thinking of the `exception in child'
problem here, so much as more generally, how much is a fully cross-platform
API worth, in a situation where the eventual application of the API is
inherently unlikely to be of a cross platform nature.  The Python version
goes to some length, but can't fully resolve the inconsistencies.  That's
OK if someone wants to go to the trouble, but if I'm right about inherent
platform dependence, it runs the risk of being more irritating than helpful!

        Donn

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Richard O'Keefe  
View profile  
 More options Aug 13 2012, 5:49 pm
From: "Richard O'Keefe" <o...@cs.otago.ac.nz>
Date: Tue, 14 Aug 2012 09:49:52 +1200
Local: Mon, Aug 13 2012 5:49 pm
Subject: Re: [Haskell-cafe] createProcess running non-existent programs

On 13/08/2012, at 11:26 PM, Alexander Kjeldaas wrote:

> This isn't that hard - a pipe shouldn't be needed anymore.  Just require a post-2003 glibc.

> fexecve is a system call in most BSDs.  It is also implemented in glibc using a /proc hack.

fexecve is now in the Single Unix Specification, based on
POSIX as of 2008, I believe.  However,
http://www.gnu.org/software/gnulib/manual/html_node/fexecve.html
says
Portability problems not fixed by Gnulib:
  *  This function is missing on many non-glibc platforms: MacOS X 10.5, FreeBSD 6.0,
     NetBSD 5.0, OpenBSD 3.8, Minix 3.1.8, AIX 5.1, HP-UX 11, IRIX 6.5, OSF/1 5.1,
     Solaris 11 2010-11, Cygwin 1.5.x, mingw, MSVC 9, Interix 3.5, BeOS.

That warning doesn't seem to be fully up to date.  I'm using MacOS X 10.6.8
and fexecve() isn't in the manuals or in <unistd.h>.

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alexander Kjeldaas  
View profile  
 More options Aug 14 2012, 4:42 am
From: Alexander Kjeldaas <alexander.kjeld...@gmail.com>
Date: Tue, 14 Aug 2012 10:42:57 +0200
Local: Tues, Aug 14 2012 4:42 am
Subject: Re: [Haskell-cafe] createProcess running non-existent programs

On 13 August 2012 23:49, Richard O'Keefe <o...@cs.otago.ac.nz> wrote:

FreeBSD 8.0 is covered.
OpenBSD not covered
OS X not covered
http://developer.apple.com/library/mac/#documentation/Darwin/Referenc...
Solaris probably not covered.

So support is pretty good, I'd say. For non-modern systems, checking the
existence of the file first is possible.  The race isn't important, and one
can always upgrade to a modern operating system.

Alexander

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Niklas Larsson  
View profile  
 More options Aug 14 2012, 11:22 am
From: Niklas Larsson <metanik...@gmail.com>
Date: Tue, 14 Aug 2012 17:22:28 +0200
Local: Tues, Aug 14 2012 11:22 am
Subject: Re: [Haskell-cafe] createProcess running non-existent programs
2012/8/14 Alexander Kjeldaas <alexander.kjeld...@gmail.com>:

The check would be unreliable, the file's existence doesn't imply that
it's executable. Furthermore it would add unnecessary overhead,
createProcess can be run thousands of times in a program and should be
lean and mean.

Niklas

> Alexander

> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-C...@haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alexander Kjeldaas  
View profile  
 More options Aug 14 2012, 5:09 pm
From: Alexander Kjeldaas <alexander.kjeld...@gmail.com>
Date: Tue, 14 Aug 2012 23:09:51 +0200
Local: Tues, Aug 14 2012 5:09 pm
Subject: Re: [Haskell-cafe] createProcess running non-existent programs

On 14 August 2012 17:22, Niklas Larsson <metanik...@gmail.com> wrote:

See access(2)

> Furthermore it would add unnecessary overhead,
> createProcess can be run thousands of times in a program and should be
> lean and mean.

Just to keep the bikeshedding doing, I'm going to state as a fact that
running performance sensitive *server* workload on any unix other than
Linux is purely of theoretical interest.  No sane person would do it.
 Therefore, from a performance overhead, Linux performance is the only
important performance measure.

But even given the above, the overhead we're talking about is minuscule.  A
program like '/bin/echo -n '''  which does exactly *nothing*, requires
35(!) system calls to do its job :-).
A more complex program like 'id' requires 250 system calls!

Also, to see just how minuscule this is, the dynamic linker, ld-linux.so
does a few extra access(2) system calls *to the same file*,
/etc/ld.so.hwcaps, on startup of every dynamically linked executable.  2 in
the 'echo' case, and 8 in the 'id' case above.  Even the glibc folks
haven't bothered to optimize those syscalls away.

Alexander

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Donn Cave  
View profile  
 More options Aug 15 2012, 1:25 am
From: Donn Cave <d...@avvanta.com>
Date: Tue, 14 Aug 2012 22:25:41 -0700 (PDT)
Local: Wed, Aug 15 2012 1:25 am
Subject: Re: [Haskell-cafe] createProcess running non-existent programs
Quoth Alexander Kjeldaas <alexander.kjeld...@gmail.com>,

> See access(2)

... a classic "code smell" in UNIX programming, for the same reasons.

We can solve this problem in an efficient way that works well, and equally
well, on any POSIX platform that supports F_CLOEXEC on pipes, and I can't
think of anything that doesn't.  The appended code is the basic gist of it.
This was not invented by the Python world, but as suggested it's one of
the things that we'd get from a review of their subprocess module.

        Donn

spawn file cmd env = do
  (e0, e1) <- pipe
  fcntlSetFlag e1 F_CLOEXEC
  t <- fork (fex e0 e1)
  close e1
  rx <- readFd e0 256
  if null rx
     then return t
     else ioerr (chrToErrType rx) file
  where
     fex e0 e1 = do
       close e0
       catch  (execve file cmd env)
              (\ e -> writeFd e1 (errToChr e : ioeGetErrorString e))
     ioerr (e, s) file = ioError (mkIOError e s Nothing (Just file))

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Niklas Hambüchen  
View profile  
 More options Aug 30 2012, 9:20 pm
From: Niklas Hambüchen <m...@nh2.me>
Date: Fri, 31 Aug 2012 02:19:30 +0100
Local: Thurs, Aug 30 2012 9:19 pm
Subject: Re: [Haskell-cafe] createProcess running non-existent programs
Well, overhead or not, it would be nice to at least have *some* solution.

Currently, it just doesn't work.

I am sure that as soon the functionality is there, somebody will step in
to fake it fast.

On 15/08/12 06:25, Donn Cave wrote:

_______________________________________________
Haskell-Cafe mailing list
Haskell-C...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »