fork in LISP
Robert Bralic
Does anybody knows how to make new process
from LISP like fork() in C.
Edi Weitz
> Does anybody knows how to make new process from LISP like fork() in
> C.
That's not part of the ANSI standard but here's an example for CMUCL
provided by Martin Cracauer:
<http://cl-cookbook.sourceforge.net/os.html#fork-cmucl>
Edi.
Christopher C. Stacy
Robert> Does anybody knows how to make new process from LISP
Robert> like fork() in C.
Robert,
You don't fork() in C; you fork() in Unix.
The idea that a new process should be a clone of its parent
process is a very Unix-specific idea, and rather a kludge.
However, Unix programs often use fork() rather than starting
up threads for historical reasons and because thread-safe
programming can be difficult in C/Unix.
In Lisp we usually solve the problem a different way altogether.
All major Lisp implementations since about 1980 have featured threads,
even before the underlying operating systems supported threads.
Thread-safe programming is much easier to do in Lisp.
Traditionally, Lisp has called threads "processes" (because the word
"thread" had not yet been coined). Implementation-wise, they might
correspond to native OS threads, or they might be part of an internal
scheduler in Lisp.
Lisp implementations differ slightly on the exact details of the API,
but there will be a function usually named PROCESS-RUN-FUNCTION which
takes as its argument another Lisp function. It starts a new thread
executing the given function (calling it with the other given arguments)
and returns immediately. There are also some other functions for
manipulating these threads in various ways.
If you really need for some weird reason to do a Unix fork(),
some Lisp implementations support that ability. The ANSI Lisp
standard doesn't define a way to manipulate Unix processes,
because that is of course an OS-dependant feature. The way to do
it is to use the implementation-specific binding to the operating
system calls. (Lisp implementations offer some such bindings,
and also a general "foreign call" interface that allows linking or
interfacing to arbitrary other programs, such as C libraries or COM.)
Someone will be able to give you a more specific answer (and an example)
if you say what Lisp implementation you are actually using.
Assuming you still think that what you want to do is fork().
cheers,
Chris
Erik Naggum
| Does anybody know how to make a new process from LISP like fork() in C.
Yes. But before we give you an answer that you might use to hurt yourself,
what is the problem to which you believe `fork()´ is the answer?
--
Erik Naggum, Oslo, Norway
Act from reason, and failure makes you rethink and study harder.
Act from faith, and failure makes you blame someone and push harder.
Kent M Pitman
Incidentally, the reason this kind of stuff isn't in the standard is that
it would mean the standard woudl be quickly obsoleted.
Lisps over the years have varied a lot in memory models and not all Lisps
would be able to implement this, which makes strong assumptions about the
nature of the host operating system.
Kaz Kylheku
> >>>>> On Thu, 29 Aug 2002 17:53:02 +0200, Robert Bralic ("Robert") writes:
> Robert> Does anybody knows how to make new process from LISP
> Robert> like fork() in C.
>
> Robert,
>
> You don't fork() in C; you fork() in Unix.
>
> The idea that a new process should be a clone of its parent
> process is a very Unix-specific idea, and rather a kludge.
> However, Unix programs often use fork() rather than starting
> up threads for historical reasons and because thread-safe
> programming can be difficult in C/Unix.
Also, standard POSIX threads don't have their own user credentials.
So you have to fork the process if you want to downgrade a root
privilege to multiple different user ID's concurrently. (Of
course in a weak programming language, juggling multiple credentials
in the same process is a potential security hole).
This POSIX stupidity makes it impossible to, for instance,
write a multithreaded file server which will perform accesses on
behalf of actual authenticated UNIX users who are making the
requests. If one thread changes the effective user ID, they
all get it.
There are other stupidities in POSIX threading for which forking
is required as a workaround, such as chdir() having a global
effect in every thread, rather than being a thread-specific
variable.
So if you have some directory traversal algorithm that uses chdir(),
you have to isolate it in its own process. Ditto for chroot().
Another stupidity is that advisory locks are owned by the
process. So one thread can lock a range of bytes that overlaps
with another range that was locked by a different thread.
Thus if you need mutually exclusive access over a file
in the same process, you need an additional layer of locking
over top of the advisory locks.
Ng Pheng Siong
> You don't fork() in C; you fork() in Unix.
>
> The idea that a new process should be a clone of its parent
> process is a very Unix-specific idea, and rather a kludge.
It's not kludgy; it's simple and elegant, IMHO.
Compare and contrast with Win32CreateProcessAndBTWDo27OtherThingsAsWell().
I've never touched a Lisp machine, so maybe there is an even better model
there.
--
Ng Pheng Siong <ng...@netmemetic.com> * http://www.netmemetic.com
Tim Bradshaw
> Also, standard POSIX threads don't have their own user credentials.
> So you have to fork the process if you want to downgrade a root
> privilege to multiple different user ID's concurrently. (Of
> course in a weak programming language, juggling multiple credentials
> in the same process is a potential security hole).
Doing something like this right is *enormously* hard. For instance,
Posix threads all share address space, and you *definitely* want
something better than that if they have different privilege levels -
not being able to write some file is not much protection when you can
poke bits all over the rest of the server.
At minimum you would need to have mandatory privilege support within
the address space of a single process - whether Lisp or C or anything
else. Doing something like this efficiently probably requires
hardware support as well as major user-level work (for instance the
memory allocator must be sure to allocate in different areas (likely
sets of pages) per thread, and the GC must preserve this - it can
never copy information across protection boundaries.)
(Someone is going to suggest at this point that all this can be solved
by having a known secure compiler and suitable language design. I
know that, we are talking about CL and Unix here).
> This POSIX stupidity makes it impossible to, for instance,
> write a multithreaded file server which will perform accesses on
> behalf of actual authenticated UNIX users who are making the
> requests. If one thread changes the effective user ID, they
> all get it.
No this isn't Posix stupidity, this is `I want a system which is
completely different than Unix'. That's a reasonable thing to want,
but don't blame Posix for not providing it.
--tim
Barry Margolin
Ng Pheng Siong <ng...@netmemetic.com> wrote:
>According to Christopher C. Stacy <cst...@dtpq.com>:
>> You don't fork() in C; you fork() in Unix.
>>
>> The idea that a new process should be a clone of its parent
>> process is a very Unix-specific idea, and rather a kludge.
>
>It's not kludgy; it's simple and elegant, IMHO.
>
>Compare and contrast with Win32CreateProcessAndBTWDo27OtherThingsAsWell().
>
>I've never touched a Lisp machine, so maybe there is an even better model
>there.
Since Lisp Machines have a single, shared address space for all processes
and no inter-process protection mechanisms, what they call "processes" are
actually more like Unix threads.
--
Barry Margolin, bar...@genuity.net
Genuity, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Christopher C. Stacy
Ng> According to Christopher C. Stacy <cst...@dtpq.com>:
>> You don't fork() in C; you fork() in Unix.
>>
>> The idea that a new process should be a clone of its parent
>> process is a very Unix-specific idea, and rather a kludge.
Ng> It's not kludgy; it's simple and elegant, IMHO.
The fork() idea is an artifact of the limitations of the Unix
virtual memory and IO system circa 1970.
My goal is to create a new execution context (eg. PC, stack,
and maybe some other environment) that may or may not wish
to share some address space. To do this, I will set a global
variable in my program, make a copy of my entire program
(regardless of whether it's related to what I want to do next),
start executing it, look at the flag, and then decide that I
am running the wrong program, and then go call the function I wanted...
Yes, how elegant, compared to, say, an interface that just runs
the function you wanted in the first place?
Ng> Compare and contrast with Win32CreateProcessAndBTWDo27OtherThingsAsWell().
Your adherence to UNIX dogma is showing through: what made
you suppose that I was comparing it Windows, when all I
wrote about was the abstract interface provided in Lisp?
Ng> I've never touched a Lisp machine, so maybe
Ng> there is an even better model there.
There are a variety of other models, besides.
Tim Bradshaw
> My goal is to create a new execution context (eg. PC, stack, and
> maybe some other environment) that may or may not wish to share some
> address space. To do this, I will set a global variable in my
> program, make a copy of my entire program (regardless of whether
> it's related to what I want to do next), start executing it, look at
> the flag, and then decide that I am running the wrong program, and
> then go call the function I wanted...
Well, if that's your goal then you should obviously use threads of
some kind. However if the goal is to provide a genuinely isolated new
program then fork is not unreasonable, and it also doesn't actually
behave as you said on recent systems - it only makes a copy of
everything in the sense of promising that you can't tell the
difference whether it had or not. In particular it copies almost
nothing but adjusts the VM system to do copy-on-write for writable
pages.
I'm sure you know this, but maybe not everyone reading it does.
--tim
Bit Bucket
>>> "Christopher" == Christopher C Stacy <cst...@dtpq.com> writes:
Christopher> The fork() idea is an artifact of the limitations of
Christopher> the Unix virtual memory and IO system circa 1970.
Christopher> To do this, I will set a global variable in my
Christopher> program, make a copy of my entire program (regardless
Christopher> of whether it's related to what I want to do next),
modern unices generally use some form of copy-on-write with fork(2) --
the entire program does not get copied. but that is orthogonal to your
point...
--
Bit Bucket
bitb...@styx.org
-----= Posted via Newsfeed.Com, Uncensored Usenet News =-----
http://www.newsfeed.com - The #1 Newsgroup Service in the World!
-----== 100,000 Groups! - 19 Servers! - Unlimited Download! =-----
Christopher C. Stacy
Tim> Well, if that's your goal then you should obviously use threads of
Tim> some kind. However if the goal is to provide a genuinely isolated new
Tim> program then fork is not unreasonable, and it also doesn't actually
[...]
Tim> I'm sure you know this, but maybe not everyone reading it does.
Yes, but conceptually, if I want an isolated new program,
why would I do this by "copying" the current program?
Tim Bradshaw
> Yes, but conceptually, if I want an isolated new program,
> why would I do this by "copying" the current program?
Because you have a whole lot of state which it's taken you a long time
to build and which you don't want to have to rebuild, but which you
don't want to share.
Or alternatively because you can build spawn from fork and exec but
you *can't* build fork and exec from spawn, and (nowadays) the fork
overhead is very small.
--tim
Barry Margolin
If you have the primitives "copy current process" and "replace current
process image with new program", you can use them together to implement
"start isolated new program".
If you only have a primitive "start isolated new program", you can't use it
to implement "copy current process". So if the latter is a desirable thing
to do, you'd need that primitive anyway. In fact, this is needed quite a
bit; for instance, when the shell runs commands in a pipeline, it has to do
some manipulation of the pipe's file descriptors in the child process
before it executes the new program. This could be done by adding
parameters to the "start isolated new program" primitive, but then every
time you think of a new thing you need to do when starting a new program
you'd have to change the parameters to that primitive, and you end up with
a complex interface like Windows.
Unix's fork provides a simple primitive that allows you to construct
whatever you want. And when implemented using the VM's copy-on-write,
performance doesn't suffer.
Christopher C. Stacy
Barry> In article <ubs7k3...@dtpq.com>,
Barry> Christopher C. Stacy <cst...@dtpq.com> wrote:
>>>>>>> On 30 Aug 2002 19:44:18 +0100, Tim Bradshaw ("Tim") writes:
Tim> Well, if that's your goal then you should obviously use threads of
Tim> some kind. However if the goal is to provide a genuinely isolated new
Tim> program then fork is not unreasonable, and it also doesn't actually
>> [...]
Tim> I'm sure you know this, but maybe not everyone reading it does.
>>
>> Yes, but conceptually, if I want an isolated new program,
>> why would I do this by "copying" the current program?
Barry> If you have the primitives "copy current process" and "replace current
Barry> process image with new program", you can use them together to implement
Barry> "start isolated new program".
Barry> If you only have a primitive "start isolated new program",
Barry> you can't use it to implement "copy current process"
That's a red herring, though: you don't need to clone the program in
order to share state, because you also have the seperate ability to
share memory across processes by adjusting the page maps.
I still fail to see the conceptual elegence of replicating a bunch
of information, then setting some global flag to indicate that all
you really wanted to do was throw some or all of it away and go do
something else.
I think it's better to just start the program that you intended,
and hand it the shared resource (data channels, memory) containing
the state that you intend to pass it.
Tim Bradshaw
> of information, then setting some global flag to indicate that all
> you really wanted to do was throw some or all of it away and go do
> something else.
There isn't any *if that's what you want to do*. But sometimes it
isn't. Sometimes you want to do something else. If you only have
spawn, then you can only do what spawn can do.
--tim
Kalle Olavi Niemitalo
> This could be done by adding parameters to the "start isolated
> new program" primitive, but then every time you think of a new
> thing you need to do when starting a new program you'd have to
> change the parameters to that primitive, and you end up with a
> complex interface like Windows.
The Spawn option of IEEE Std 1003.1-2001 has is an interface like that:
<URL:http://www.opengroup.org/onlinepubs/007904975/basedefs/spawn.h.html>
GNU libc 2.2 supports this.
Johan Ur Riise
> * Robert Bralic
> | Does anybody know how to make a new process from LISP like fork() in C.
>
> Yes. But before we give you an answer that you might use to hurt yourself,
> what is the problem to which you believe `fork()´ is the answer?
I did this too. The forked image just saves some data structure to a
file, then exits. The parent continues modifying the data. The file is
only used at startup to initialize the data structure.
What I wanted to achive, was to have a snapshot of the data regularly,
and the normal execution thread should not have to spend time saving.
There was complexity, like I had to call wait on the daughter process,
and I had to make sure that one save was completed before the next one
was started.
I could have done it with the multiprocessing package in cmucl, but I
expected there would be a problem with simultaneous updating of the
data structure.
Would this be reasonable, or is it just unnessesary old C thinking?
J.
Tim Bradshaw
> I did this too. The forked image just saves some data structure to a
> file, then exits. The parent continues modifying the data. The file is
> only used at startup to initialize the data structure.
Actually, that's a really good example. We did[1] more-or-less
exactly this. We had an application which some fairly large amount of
state (~a few GB). Every so often we wanted to checkpoint the state
in such a way that we didn't have large downtime. We could (and
originally did) do this by freezing the system, dumping state, and
then unfreezing it. But this fails on the downtime issue. So instead
we can just fork, and one branch of the fork can then carry on
running, while the other saves the application data and then exits.
It's a nice solution to some problems.
--tim
[1] I think we didn't actually implement this but it was talked about
and prototyped.
Martti Halminen
> * Johan Ur Riise wrote:
> > I did this too. The forked image just saves some data structure to a
> > file, then exits. The parent continues modifying the data. The file is
> > only used at startup to initialize the data structure.
> Actually, that's a really good example. We did[1] more-or-less
> exactly this. We had an application which some fairly large amount of
> state (~a few GB). Every so often we wanted to checkpoint the state
> in such a way that we didn't have large downtime. We could (and
> originally did) do this by freezing the system, dumping state, and
> then unfreezing it. But this fails on the downtime issue. So instead
> we can just fork, and one branch of the fork can then carry on
> running, while the other saves the application data and then exits.
> It's a nice solution to some problems.
This requires a sufficiently nice fork implementation to keep the memory
usage bearable. Several years ago I was running CMU-CL 17f on HP-UX 9.05
on a machine with a whopping 32 MB of RAM. That combination merrily
copied the whole process when forking (starting the debugger, IIRC),
thereby immediately killing the whole application when it ran out of
memory.
--
Paul F. Dietz
> > I did this too. The forked image just saves some data structure to a
> > file, then exits. The parent continues modifying the data. The file is
> > only used at startup to initialize the data structure.
>
> Actually, that's a really good example. We did[1] more-or-less
> exactly this.
...
> [1] I think we didn't actually implement this but it was talked about
> and prototyped.
The Fuzzball MUCK server does this. It's not written in Lisp, though.
Paul
Nils Goesche
> Tim Bradshaw wrote:
>
> > * Johan Ur Riise wrote:
> > > I did this too. The forked image just saves some data structure to a
> > > file, then exits.
>
> > Actually, that's a really good example. We did[1] more-or-less
> > exactly this. We had an application which some fairly large amount of
> > state (~a few GB). Every so often we wanted to checkpoint the state
> > in such a way that we didn't have large downtime. We could (and
> > originally did) do this by freezing the system, dumping state, and
> > then unfreezing it. But this fails on the downtime issue. So instead
> > we can just fork, and one branch of the fork can then carry on
> > running, while the other saves the application data and then exits.
> > It's a nice solution to some problems.
>
> This requires a sufficiently nice fork implementation to keep the memory
> usage bearable. Several years ago I was running CMU-CL 17f on HP-UX 9.05
> on a machine with a whopping 32 MB of RAM. That combination merrily
> copied the whole process when forking (starting the debugger, IIRC),
> thereby immediately killing the whole application when it ran out of
> memory.
But on a system with a copy-on-write policy it should be worth a
try, I should think.
Regards,
--
Nils Goesche
Ask not for whom the <CONTROL-G> tolls.
PGP key ID #xD26EF2A0
Christopher C. Stacy
Nils> Martti Halminen <martti....@kolumbus.fi> writes:
>> Tim Bradshaw wrote:
>>
>> > * Johan Ur Riise wrote:
>> > > I did this too. The forked image just saves some data structure to a
>> > > file, then exits.
>>
>> > Actually, that's a really good example. We did[1] more-or-less
>> > exactly this. We had an application which some fairly large amount of
>> > state (~a few GB). Every so often we wanted to checkpoint the state
>> > in such a way that we didn't have large downtime. We could (and
>> > originally did) do this by freezing the system, dumping state, and
>> > then unfreezing it. But this fails on the downtime issue. So instead
>> > we can just fork, and one branch of the fork can then carry on
>> > running, while the other saves the application data and then exits.
>> > It's a nice solution to some problems.
>>
>> This requires a sufficiently nice fork implementation to keep the memory
>> usage bearable. Several years ago I was running CMU-CL 17f on HP-UX 9.05
>> on a machine with a whopping 32 MB of RAM. That combination merrily
>> copied the whole process when forking (starting the debugger, IIRC),
>> thereby immediately killing the whole application when it ran out of
>> memory.
Nils> But on a system with a copy-on-write policy it should be worth
Nils> a try, I should think.
Yes, assuming that the mutating process is not changing everything
around (which would result in the OS copying all the pages for the
forked process). The implementation of the GC is also factor there.
Carl Shapiro
As does LambdaMOO (which is also not written in Lisp).
I know of one Lisp system which is able to perform this sort of world
save while running but without the assistance of "fork". Like the
"fork" approach, there exists a process which traverses an object
hierarchy writing out state information. However, this process is
interleaved with the normal application process(es) and operates on
the same set of live objects. The trick is basically that all object
slot writers first check to see if the system is in the midst of a
world save. If the system is within this state and the application
attempts to mutate an object that has not been saved out to disk, that
object is immediately forced out to disk. I was surprised to see how
in practice, this system turns out to work quite well.
Tim Bradshaw
> This requires a sufficiently nice fork implementation to keep the
> memory usage bearable. Several years ago I was running CMU-CL 17f on
> HP-UX 9.05 on a machine with a whopping 32 MB of RAM. That
> combination merrily copied the whole process when forking (starting
> the debugger, IIRC), thereby immediately killing the whole
> application when it ran out of memory.
Well, yes, I kind of assumed that. The trick wouldn't have worked
very well on 7th edition Unix either.
--tim
Nils Goesche
I wouldn't worry about that too much, but...
> The implementation of the GC is also factor there.
Yes, GC could kill the whole idea. Although... if you're lucky, stuff
in generation 2 or higher won't be touched during the short time it
takes to save everything to disk. Maybe playing with (room t) would
give a better idea of how likely it is that GC causes too much
copying.
Regards,
--
Nils Goesche
"Don't ask for whom the <CTRL-G> tolls."
PGP key ID 0x0655CFA0
Barry Margolin
Christopher C. Stacy <cst...@dtpq.com> wrote:
>>>>>> On Fri, 30 Aug 2002 20:22:03 GMT, Barry Margolin ("Barry") writes:
>
> Barry> In article <ubs7k3...@dtpq.com>,
> Barry> Christopher C. Stacy <cst...@dtpq.com> wrote:
> >>>>>>> On 30 Aug 2002 19:44:18 +0100, Tim Bradshaw ("Tim") writes:
> Tim> Well, if that's your goal then you should obviously use threads of
> Tim> some kind. However if the goal is to provide a genuinely isolated new
> Tim> program then fork is not unreasonable, and it also doesn't actually
> >> [...]
> Tim> I'm sure you know this, but maybe not everyone reading it does.
> >>
> >> Yes, but conceptually, if I want an isolated new program,
> >> why would I do this by "copying" the current program?
>
> Barry> If you have the primitives "copy current process" and "replace current
> Barry> process image with new program", you can use them together to implement
> Barry> "start isolated new program".
>
> Barry> If you only have a primitive "start isolated new program",
> Barry> you can't use it to implement "copy current process"
>
>That's a red herring, though: you don't need to clone the program in
>order to share state, because you also have the seperate ability to
>share memory across processes by adjusting the page maps.
Shared memory wasn't added to Unix until the 80's, in BSD.