Search for API function to activate a task and return it's PID to the caller

[dma...@cfl.rr.com]

I'm searching for a Linux function that will act like the
"system(pathname)" function but return to the caller, the PID of the
new task.

Thanks
Mark

[Måns Rullgård]

fork + exec

--
Måns Rullgård
ma...@mansr.com

[dma...@cfl.rr.com]

> m...@mansr.com

That's a little heavy weight though. The task doing this is very
large. Wouldn't this mean that until the exec'd task exited there
would be a copy of my large task, that was forked, in memory, no? And
if I need to do this for a few hundreds of programs that stayed alive
for long periods of time, wouldn't that be a bad thing? And how would
the PID of the exec'd task be returned to me. All I would get is the
PID of the forked copy of my original task, no?

Mark

[Måns Rullgård]

"dma...@cfl.rr.com" <dma...@cfl.rr.com> writes:

> On May 19, 9:36 am, Måns Rullgård <m...@mansr.com> wrote:
>> "dma...@cfl.rr.com" <dma...@cfl.rr.com> writes:
>> > I'm searching for a Linux function that will act like the
>> > "system(pathname)" function but return to the caller, the PID of the
>> > new task.
>>
>> fork + exec
>

> That's a little heavy weight though.

It is exactly what the system() function does.

> The task doing this is very large. Wouldn't this mean that until the
> exec'd task exited there would be a copy of my large task, that was
> forked, in memory, no?

No. Physical memory pages are shared with the original. If either of
the processes modifies a page, a copy of that page is made at that
time. This is called copy-on-write.

Typically, you would exec() very shortly after the fork() call, so
only the current stack page is likely to be actually copied.

> And if I need to do this for a few hundreds of programs that stayed
> alive for long periods of time, wouldn't that be a bad thing? And how
> would the PID of the exec'd task be returned to me. All I would get is
> the PID of the forked copy of my original task, no?

exec() does not change the PID.

--
Måns Rullgård
ma...@mansr.com

[dma...@cfl.rr.com]

> m...@mansr.com

So the actual PID of the exec'd task would be that of the fork'd copy
of my task?

Mark

[Lew Pitcher]

On Saturday 19 May 2012 10:35, in comp.unix.programmer, dma...@cfl.rr.com
wrote:

The PID returned to the parent process by fork() will be the PID that the
child process uses, even after it exec()s.

The PID returned to the child process by fork() (which, after the fork(), is
a duplicate of the parent process, with some changes in it's environment)
will be 0.


--
Lew Pitcher

[Barry Margolin]

In article
<a9f265f8-9c81-42e2...@f14g2000yqe.googlegroups.com>,

> So the actual PID of the exec'd task would be that of the fork'd copy
> of my task?

Correct. exec() doesn't create a new process, it replaces the current
process's memory with the specified program and starts running it.

This is the usual way programs are run on Unix: a new process is created
with fork(), then the program is loaded into it with exec(). It's what
the shell does for every program you run (except the last command in a
script may be exec'ed without forking, since the shell is just going to
exit immediately after it finishes).

--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

[Gordon Burditt]

>> > I'm searching for a Linux function that will act like the
>> > "system(pathname)" function but return to the caller, the PID of the
>> > new task.

If the code you want acts like system(), the PID will be pretty useless
since system() and presumably its replacement waits for the process
to exit before returning.

>> fork + exec

> That's a little heavy weight though.

Compared to what?
system(pathname) does *TWO* fork+exec, plus running a shell.

> The task doing this is very
> large. Wouldn't this mean that until the exec'd task exited there
> would be a copy of my large task, that was forked, in memory, no? And

system() is still worse. But copy-on-write means that most of the
task won't be copied.

> if I need to do this for a few hundreds of programs that stayed alive
> for long periods of time, wouldn't that be a bad thing? And how would

If you are having a few hundreds of programs running simultaneously
for any reason, I'd worry about the system paging itself to death,
running out of swap space (possibly invoking the Linux OOM killer),
and response time getting killed.

> the PID of the exec'd task be returned to me. All I would get is the
> PID of the forked copy of my original task, no?

exec() doesn't change the PID, so those two would be the same.

[Casper H.S. Dik]

gordon...@burditt.org (Gordon Burditt) writes:

>system() is still worse. But copy-on-write means that most of the
>task won't be copied.

It should be noticed that even copy-on-write is O(n) (where N is the
size of the address space).

system() may use vfork() but using the shell is a possible issue;
posix_spawn() might be an alternative as the system implementor
should have picked the cheapest way to implement it.

Casper

[dma...@cfl.rr.com]

On May 20, 5:04 am, Casper H.S. Dik <Casper....@OrSPaMcle.COM> wrote:

Thanks all. I currently have it working using fork - exec. I don't yet
have hundreds of these tasks running so I don't yet know what kind of
memory footprint it's going to have. I'll keep posix_spawn in mind and
try it if I need to and if it is actually available to Linux. I have a
man page but not sure if it actually lives here.

Regards
Mark

[Jeff Flinn]

On 5/19/2012 12:33 PM, Barry Margolin wrote:
> In article
> <a9f265f8-9c81-42e2...@f14g2000yqe.googlegroups.com>,
> "dma...@cfl.rr.com"<dma...@cfl.rr.com> wrote:
>
>> On May 19, 10:30 am, Måns Rullgård<m...@mansr.com> wrote:
>>> "dma...@cfl.rr.com"<dma...@cfl.rr.com> writes:
>>>> On May 19, 9:36 am, Måns Rullgård<m...@mansr.com> wrote:
>>>>> "dma...@cfl.rr.com"<dma...@cfl.rr.com> writes:
>>>>>> I'm searching for a Linux function that will act like the
>>>>>> "system(pathname)" function but return to the caller, the PID of the
>>>>>> new task.
>>>
>>>>> fork + exec

...

>>> exec() does not change the PID.
>>
>> So the actual PID of the exec'd task would be that of the fork'd copy
>> of my task?
>
> Correct. exec() doesn't create a new process, it replaces the current
> process's memory with the specified program and starts running it.
>
> This is the usual way programs are run on Unix: a new process is created
> with fork(), then the program is loaded into it with exec(). It's what
> the shell does for every program you run (except the last command in a
> script may be exec'ed without forking, since the shell is just going to
> exit immediately after it finishes).
>

Where can one find a document that describes the design rationale for
the whole fork/exec paradigm? If not where could I find such in depth
description of this architecture.

Thanks, Jeff

[Barry Margolin]

In article <jpdeu0$65p$1...@dont-email.me>,

Jeff Flinn <TriumphS...@hotmail.com> wrote:

> Where can one find a document that describes the design rationale for
> the whole fork/exec paradigm? If not where could I find such in depth
> description of this architecture.

It's basically the same design rationale as most of Unix: provide very
simple primitives, and let the programmers build more complex
applications out of them.

You can probably find discussion of it in the original BLTJ article on
the design of Unix.

[Casper H.S. Dik]

Barry Margolin <bar...@alum.mit.edu> writes:

>In article <jpdeu0$65p$1...@dont-email.me>,
> Jeff Flinn <TriumphS...@hotmail.com> wrote:

>> Where can one find a document that describes the design rationale for
>> the whole fork/exec paradigm? If not where could I find such in depth
>> description of this architecture.

>It's basically the same design rationale as most of Unix: provide very
>simple primitives, and let the programmers build more complex
>applications out of them.

Unix has moved on but compare fork() & exec() and other simple
system calls (open(), dup(), dup2(), close()) to posix_spwan();
then look at all the parameters and flags used and map them to
simple system calls.

Casper

[Rainer Weikusat]

Casper H.S. Dik <Caspe...@OrSPaMcle.COM> writes:
> Barry Margolin <bar...@alum.mit.edu> writes:
>>In article <jpdeu0$65p$1...@dont-email.me>,
>> Jeff Flinn <TriumphS...@hotmail.com> wrote:
>
>>> Where can one find a document that describes the design rationale for
>>> the whole fork/exec paradigm? If not where could I find such in depth
>>> description of this architecture.
>
>>It's basically the same design rationale as most of Unix: provide very
>>simple primitives, and let the programmers build more complex
>>applications out of them.
>
> Unix has moved on but compare fork() & exec() and other simple
> system calls (open(), dup(), dup2(), close()) to posix_spwan();

For as long as people are convinced that they still have to make the
system behave like UNIX(*) v7 did because - clearly - that's how God
wanted it to be, UNIX(*) hasn't really 'moved on' very much (since
this concern dates back to the first virtual memory implementations
for UNIX(*)).

In case someone doesn't understand this: 'posix_spawn' was designed as
a process creation primitives for system lacking a decently working
virtual memory implementation, be it because the hardware is to feeble
to support that (eg, ARMv7) or because of an incurable 'Marshall
McKusick on my Mind!' syndrome (Solaris).

[Måns Rullgård]

Casper H.S. Dik <Caspe...@OrSPaMcle.COM> writes:

posix_spawn() exists mainly because it is easier to implement on a
system without an MMU. The POSIX spec explains the rationale in detail:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_spawn.html#tag_16_371_08

--
Måns Rullgård
ma...@mansr.com

[dma...@cfl.rr.com]

As I said the process of starting a task and obtaining it's PID is
working. The reason I need the PID was for some level of task
managment. In particular, one thing I need to be able to do is pause
and continue these tasks using SIGSTOP/SIGCONT. There are other things
also but this one seems to be giving me problems right now. When I
send SIGSTOP to one of these processes, it EXITS instead of pausing.
That same task pauses and continues just find when I send these
signals from a shell via "kill -s SIGSTOP/SIGCONT pid". In fact it
seems that if I send the signals from a different task (one that
didn't actually create the process) it also? Why is the task that
created the process not able to "correctly" send the SIGSTOP signal to
it? Sample code:

tskmgt.c : task that starts and attempts to pause another task

int32_t main(int argc, char **argv)
{
pid_t pid;
const char *path = "/home/markh/test/tsttsk";
char *arg = 0;

pid = fork();
if (pid == 0) { // then I am the new task
execv(path, &arg);
} else {
printf("PID of new task = %d\n", pid);
}
sleep(5);
printf("Sending SIGSTOP to pid %d\n", pid);
if (kill(pid, SIGSTOP) != 0)
perror("Failed to send SIGSTOP: ");
return 0;
}

tsttsk.c:

int32_t main(int argc, char **argv)
{
FILE *fd;
const char *path = "/var/log/srtl.log";
const char *mode = "w+";
fd = fopen(path, mode);
if (fd < 0) {
perror("fopen of /var/log/srtl.log failed: ");
return 1;
}
while (1) {
sleep(1);
fprintf(fd, "tsttsk is running\n");
fflush(fd);
}
return 0;
}

I have not yet attempted to use posix_spawn BTW. I would like to
understand why this doesn't work first.

Regards
Mark

[Rainer Weikusat]

Casper H.S. Dik <Caspe...@OrSPaMcle.COM> writes:

> gordon...@burditt.org (Gordon Burditt) writes:
>>system() is still worse. But copy-on-write means that most of the
>>task won't be copied.
>
> It should be noticed that even copy-on-write is O(n) (where N is the
> size of the address space).

It should be noted that systems other than NetBSD are in common use.
Consequently, 'empirical evidence' suggests that this isn't really an
issue in practice.

> system() may use vfork() but using the shell is a possible issue;
> posix_spawn() might be an alternative as the system implementor
> should have picked the cheapest way to implement it.

Read: Since the inner workings of posix_spawn are
implementation-defined, there's no way telling how it will be
implemented on any particular system and consequently, it can't be
relied on if 'performance'/ 'cost' is a problem except when the source
code (insofar available) of all posix_spawn implementations that could
conceivably be used is checked first to determine if the
implementation is a suitable one for the given system and the way the
applications intends to use it.

In particular, on Linux (last time I looked), it was a library
implementation wrapping fork and exec and this means the cost of
posix_spawn is actually higher than that of using the traditional way.

[Barry Margolin]

In article <87396su...@sapphire.mobileactivedefense.com>,

Rainer Weikusat <rwei...@mssgmbh.com> wrote:

> In particular, on Linux (last time I looked), it was a library
> implementation wrapping fork and exec and this means the cost of
> posix_spawn is actually higher than that of using the traditional way.

True. But if the impact were high enough, and there were sufficient
demand, someone could replace the library function with a system call,
and everyone using posix_spawn would reap the benefit when they upgrade
the OS.

[Xavier Roche]

On 05/22/2012 04:15 PM, Rainer Weikusat wrote:
> In particular, on Linux (last time I looked), it was a library
> implementation wrapping fork and exec and this means the cost of
> posix_spawn is actually higher than that of using the traditional way.

Indeed, the glibc (latest checked) is at least using vfork() for simple
cases (no file actions, for example).. but is using fork() in other cases.

[Rainer Weikusat]

Which brings us to the next problem with posix_spawn: Depending on the
implementation (maybe you still remember the details from checking the
sources last time), it may or may not suspend the process which
executes it for an indefinite amount of time ...

[Eric Sosman]

Doesn't stat() have the same "problem?"

--
Eric Sosman
eso...@ieee-dot-org.invalid

[Rainer Weikusat]

That's why program which care about such delays do disk accesses in
helper processes. But this seems somewhat besides the point here.

I'm - of course - completely unaware if the Solaris stat doesn't
perhaps call vfork to invoke a helper binary running with the help of
a PDP-11 emulator which has been a binary-only part of every 'remotely
BSD' system ever since someone first didn't chose to avoid 'reinventing the
wheel' in 1977 or so....

[Xavier Roche]

On 05/22/2012 06:38 PM, Rainer Weikusat wrote:
>> Doesn't stat() have the same "problem?"
> That's why program which care about such delays do disk accesses in
> helper processes. But this seems somewhat besides the point here.

The probable worst part is not (this specific implementation of)
posix_spawn() blocking (temporarily) the caller's thread, but IMHO the
side effect of using the same process pages ; ie. between vfork() and
the final call to exec*(), all process pages are marked copy-on-write,
and thus, all other threads currently working are duplicating them,
which can slowdown everyone, unless I am missing something here.

[William Ahern]

vfork() suspends the calling process (or, presumably, thread) as an
_alternative_ to marking the pages. So no copying is taking place.

[Xavier Roche]

Le 23/05/2012 17:42, William Ahern a écrit :
> vfork() suspends the calling process (or, presumably, thread) as an
> _alternative_ to marking the pages. So no copying is taking place.

Humm, so it stops the process as a whole, which is even worse.

[William Ahern]

Probably just the thread, but I haven't confirmed. The kernel suspends the
initiating process (or thread) because both child and parent after vfork()
are sharing the stack. They can't both execute simultaneously without bad
things happening. But I don't see any reason to block other threads in the
initiating processs; they have their own stack.

[Rainer Weikusat]

William Ahern <wil...@wilbur.25thandClement.com> writes:
> Xavier Roche <xro...@free.fr.nospam.invalid> wrote:
>> Le 23/05/2012 17:42, William Ahern a écrit :
>> > vfork() suspends the calling process (or, presumably, thread) as an
>> > _alternative_ to marking the pages. So no copying is taking place.
>
>> Humm, so it stops the process as a whole, which is even worse.
>
> Probably just the thread, but I haven't confirmed. The kernel suspends the
> initiating process (or thread) because both child and parent after vfork()
> are sharing the stack. They can't both execute simultaneously without bad
> things happening.

Since the vfork'd process must not modify any memory of the inherited
address space, as soon as it writes to the stack, bad things are
happening/ have happened because the stack, as seen from the viewpoint
of the suspended process, is now corrupted.

[Barry Margolin]

In article <8762bmk...@sapphire.mobileactivedefense.com>,

That "corruption" will never be noticed in the parent process, because
it's writing to newly pushed stack frames. The parent's stack pointer
doesn't include them.

After vfork, the child must not write to the data segment or to ancestor
stack frames, since those ARE shared with the parent.

[Casper H.S. Dik]

Right; including all threads; the point is that vfork() should be
followed immediately by exit or execve and then the command can run
again.

Casper

[Casper H.S. Dik]

But they all have shared data may be using mutex locks. The numbers
of things you can do after vfork() is limited; e..g, don't even try
using stdio.

Casper

[Casper H.S. Dik]

And the vfork() thread should return unwind the stack.

E.g.,:

int myvfork() { return vfork(); }

will give you issues.

Casper

[Rainer Weikusat]

'stack frame' is not some kind of standardized concept. Also, even
assuming a 'stack frame', there would be the current stack frame the
child could write to, thus possibly changing the values of certain
objects the parent will later use, and the child can also overwrite the
contents of this current stack frame with 'something completely
different' by returning from the currently executing subroutine and
then calling another one.

Also, I strongly suspect the "common use case" for vfork is not
"creating a process and then immediately exec'ing another program in
an otherwise unchanged environment", but rather the udchpd 'programmer
on crack' way to execute an external tool:

1. Write code to record the current state of the process.
2. Write code to change the state of the process such that it
is suitable for the helper script which is going to be
executed.
3. call vfork because that's the Bill![tm] way of doing it
which must surely be Very Superior
4. exec
5. in the parent, restore the state of the process to the one
the parent expects

IOW, it's an excuse to write about three times more code than would
actually be required because This Must Be An Important
Optimization!!1.