>>>>> On Thu, 29 Aug 2002 17:53:02 +0200, Robert Bralic ("Robert") writes:
Robert> Does anybody knows how to make new process from LISP Robert> like fork() in C.
Robert,
You don't fork() in C; you fork() in Unix.
The idea that a new process should be a clone of its parent process is a very Unix-specific idea, and rather a kludge. However, Unix programs often use fork() rather than starting up threads for historical reasons and because thread-safe programming can be difficult in C/Unix.
In Lisp we usually solve the problem a different way altogether.
All major Lisp implementations since about 1980 have featured threads, even before the underlying operating systems supported threads. Thread-safe programming is much easier to do in Lisp.
Traditionally, Lisp has called threads "processes" (because the word "thread" had not yet been coined). Implementation-wise, they might correspond to native OS threads, or they might be part of an internal scheduler in Lisp.
Lisp implementations differ slightly on the exact details of the API, but there will be a function usually named PROCESS-RUN-FUNCTION which takes as its argument another Lisp function. It starts a new thread executing the given function (calling it with the other given arguments) and returns immediately. There are also some other functions for manipulating these threads in various ways.
If you really need for some weird reason to do a Unix fork(), some Lisp implementations support that ability. The ANSI Lisp standard doesn't define a way to manipulate Unix processes, because that is of course an OS-dependant feature. The way to do it is to use the implementation-specific binding to the operating system calls. (Lisp implementations offer some such bindings, and also a general "foreign call" interface that allows linking or interfacing to arbitrary other programs, such as C libraries or COM.)
Someone will be able to give you a more specific answer (and an example) if you say what Lisp implementation you are actually using. Assuming you still think that what you want to do is fork().
Incidentally, the reason this kind of stuff isn't in the standard is that it would mean the standard woudl be quickly obsoleted.
Lisps over the years have varied a lot in memory models and not all Lisps would be able to implement this, which makes strong assumptions about the nature of the host operating system.
> >>>>> On Thu, 29 Aug 2002 17:53:02 +0200, Robert Bralic ("Robert") writes: > Robert> Does anybody knows how to make new process from LISP > Robert> like fork() in C.
> Robert,
> You don't fork() in C; you fork() in Unix.
> The idea that a new process should be a clone of its parent > process is a very Unix-specific idea, and rather a kludge. > However, Unix programs often use fork() rather than starting > up threads for historical reasons and because thread-safe > programming can be difficult in C/Unix.
Also, standard POSIX threads don't have their own user credentials. So you have to fork the process if you want to downgrade a root privilege to multiple different user ID's concurrently. (Of course in a weak programming language, juggling multiple credentials in the same process is a potential security hole).
This POSIX stupidity makes it impossible to, for instance, write a multithreaded file server which will perform accesses on behalf of actual authenticated UNIX users who are making the requests. If one thread changes the effective user ID, they all get it.
There are other stupidities in POSIX threading for which forking is required as a workaround, such as chdir() having a global effect in every thread, rather than being a thread-specific variable.
So if you have some directory traversal algorithm that uses chdir(), you have to isolate it in its own process. Ditto for chroot().
Another stupidity is that advisory locks are owned by the process. So one thread can lock a range of bytes that overlaps with another range that was locked by a different thread. Thus if you need mutually exclusive access over a file in the same process, you need an additional layer of locking over top of the advisory locks.
* Kaz Kylheku wrote: > Also, standard POSIX threads don't have their own user credentials. > So you have to fork the process if you want to downgrade a root > privilege to multiple different user ID's concurrently. (Of > course in a weak programming language, juggling multiple credentials > in the same process is a potential security hole).
Doing something like this right is *enormously* hard. For instance, Posix threads all share address space, and you *definitely* want something better than that if they have different privilege levels - not being able to write some file is not much protection when you can poke bits all over the rest of the server.
At minimum you would need to have mandatory privilege support within the address space of a single process - whether Lisp or C or anything else. Doing something like this efficiently probably requires hardware support as well as major user-level work (for instance the memory allocator must be sure to allocate in different areas (likely sets of pages) per thread, and the GC must preserve this - it can never copy information across protection boundaries.)
(Someone is going to suggest at this point that all this can be solved by having a known secure compiler and suitable language design. I know that, we are talking about CL and Unix here).
> This POSIX stupidity makes it impossible to, for instance, > write a multithreaded file server which will perform accesses on > behalf of actual authenticated UNIX users who are making the > requests. If one thread changes the effective user ID, they > all get it.
No this isn't Posix stupidity, this is `I want a system which is completely different than Unix'. That's a reasonable thing to want, but don't blame Posix for not providing it.
In article <ako4c4$gc...@mawar.singnet.com.sg>, Ng Pheng Siong <n...@netmemetic.com> wrote:
>According to Christopher C. Stacy <cst...@dtpq.com>: >> You don't fork() in C; you fork() in Unix.
>> The idea that a new process should be a clone of its parent >> process is a very Unix-specific idea, and rather a kludge.
>It's not kludgy; it's simple and elegant, IMHO.
>Compare and contrast with Win32CreateProcessAndBTWDo27OtherThingsAsWell().
>I've never touched a Lisp machine, so maybe there is an even better model >there.
Since Lisp Machines have a single, shared address space for all processes and no inter-process protection mechanisms, what they call "processes" are actually more like Unix threads.
-- Barry Margolin, bar...@genuity.net Genuity, Woburn, MA *** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups. Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
>>>>> On 30 Aug 2002 15:52:36 GMT, Ng Pheng Siong ("Ng") writes:
Ng> According to Christopher C. Stacy <cst...@dtpq.com>: >> You don't fork() in C; you fork() in Unix. >> >> The idea that a new process should be a clone of its parent >> process is a very Unix-specific idea, and rather a kludge.
Ng> It's not kludgy; it's simple and elegant, IMHO.
The fork() idea is an artifact of the limitations of the Unix virtual memory and IO system circa 1970.
My goal is to create a new execution context (eg. PC, stack, and maybe some other environment) that may or may not wish to share some address space. To do this, I will set a global variable in my program, make a copy of my entire program (regardless of whether it's related to what I want to do next), start executing it, look at the flag, and then decide that I am running the wrong program, and then go call the function I wanted...
Yes, how elegant, compared to, say, an interface that just runs the function you wanted in the first place?
Ng> Compare and contrast with Win32CreateProcessAndBTWDo27OtherThingsAsWell().
Your adherence to UNIX dogma is showing through: what made you suppose that I was comparing it Windows, when all I wrote about was the abstract interface provided in Lisp?
Ng> I've never touched a Lisp machine, so maybe Ng> there is an even better model there.
> My goal is to create a new execution context (eg. PC, stack, and > maybe some other environment) that may or may not wish to share some > address space. To do this, I will set a global variable in my > program, make a copy of my entire program (regardless of whether > it's related to what I want to do next), start executing it, look at > the flag, and then decide that I am running the wrong program, and > then go call the function I wanted...
Well, if that's your goal then you should obviously use threads of some kind. However if the goal is to provide a genuinely isolated new program then fork is not unreasonable, and it also doesn't actually behave as you said on recent systems - it only makes a copy of everything in the sense of promising that you can't tell the difference whether it had or not. In particular it copies almost nothing but adjusts the VM system to do copy-on-write for writable pages.
I'm sure you know this, but maybe not everyone reading it does.
*** post for FREE via your newsreader at post.newsfeed.com ***
>>> "Christopher" == Christopher C Stacy <cst...@dtpq.com> writes:
Christopher> The fork() idea is an artifact of the limitations of Christopher> the Unix virtual memory and IO system circa 1970.
Christopher> To do this, I will set a global variable in my Christopher> program, make a copy of my entire program (regardless Christopher> of whether it's related to what I want to do next),
modern unices generally use some form of copy-on-write with fork(2) -- the entire program does not get copied. but that is orthogonal to your point...
>>>>> On 30 Aug 2002 19:44:18 +0100, Tim Bradshaw ("Tim") writes:
Tim> Well, if that's your goal then you should obviously use threads of Tim> some kind. However if the goal is to provide a genuinely isolated new Tim> program then fork is not unreasonable, and it also doesn't actually [...] Tim> I'm sure you know this, but maybe not everyone reading it does.
Yes, but conceptually, if I want an isolated new program, why would I do this by "copying" the current program?
> Yes, but conceptually, if I want an isolated new program, > why would I do this by "copying" the current program?
Because you have a whole lot of state which it's taken you a long time to build and which you don't want to have to rebuild, but which you don't want to share.
Or alternatively because you can build spawn from fork and exec but you *can't* build fork and exec from spawn, and (nowadays) the fork overhead is very small.
In article <ubs7k3ywh....@dtpq.com>, Christopher C. Stacy <cst...@dtpq.com> wrote:
>>>>>> On 30 Aug 2002 19:44:18 +0100, Tim Bradshaw ("Tim") writes: > Tim> Well, if that's your goal then you should obviously use threads of > Tim> some kind. However if the goal is to provide a genuinely isolated new > Tim> program then fork is not unreasonable, and it also doesn't actually > [...] > Tim> I'm sure you know this, but maybe not everyone reading it does.
>Yes, but conceptually, if I want an isolated new program, >why would I do this by "copying" the current program?
If you have the primitives "copy current process" and "replace current process image with new program", you can use them together to implement "start isolated new program".
If you only have a primitive "start isolated new program", you can't use it to implement "copy current process". So if the latter is a desirable thing to do, you'd need that primitive anyway. In fact, this is needed quite a bit; for instance, when the shell runs commands in a pipeline, it has to do some manipulation of the pipe's file descriptors in the child process before it executes the new program. This could be done by adding parameters to the "start isolated new program" primitive, but then every time you think of a new thing you need to do when starting a new program you'd have to change the parameters to that primitive, and you end up with a complex interface like Windows.
Unix's fork provides a simple primitive that allows you to construct whatever you want. And when implemented using the VM's copy-on-write, performance doesn't suffer.
-- Barry Margolin, bar...@genuity.net Genuity, Woburn, MA *** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups. Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
>>>>> On Fri, 30 Aug 2002 20:22:03 GMT, Barry Margolin ("Barry") writes:
Barry> In article <ubs7k3ywh....@dtpq.com>, Barry> Christopher C. Stacy <cst...@dtpq.com> wrote: >>>>>>> On 30 Aug 2002 19:44:18 +0100, Tim Bradshaw ("Tim") writes: Tim> Well, if that's your goal then you should obviously use threads of Tim> some kind. However if the goal is to provide a genuinely isolated new Tim> program then fork is not unreasonable, and it also doesn't actually >> [...] Tim> I'm sure you know this, but maybe not everyone reading it does. >> >> Yes, but conceptually, if I want an isolated new program, >> why would I do this by "copying" the current program?
Barry> If you have the primitives "copy current process" and "replace current Barry> process image with new program", you can use them together to implement Barry> "start isolated new program".
Barry> If you only have a primitive "start isolated new program", Barry> you can't use it to implement "copy current process"
That's a red herring, though: you don't need to clone the program in order to share state, because you also have the seperate ability to share memory across processes by adjusting the page maps.
I still fail to see the conceptual elegence of replicating a bunch of information, then setting some global flag to indicate that all you really wanted to do was throw some or all of it away and go do something else.
I think it's better to just start the program that you intended, and hand it the shared resource (data channels, memory) containing the state that you intend to pass it.
> I still fail to see the conceptual elegence of replicating a bunch > of information, then setting some global flag to indicate that all > you really wanted to do was throw some or all of it away and go do > something else.
There isn't any *if that's what you want to do*. But sometimes it isn't. Sometimes you want to do something else. If you only have spawn, then you can only do what spawn can do.
Barry Margolin <bar...@genuity.net> writes: > This could be done by adding parameters to the "start isolated > new program" primitive, but then every time you think of a new > thing you need to do when starting a new program you'd have to > change the parameters to that primitive, and you end up with a > complex interface like Windows.
Erik Naggum <e...@naggum.no> writes: > * Robert Bralic > | Does anybody know how to make a new process from LISP like fork() in C.
> Yes. But before we give you an answer that you might use to hurt yourself, > what is the problem to which you believe `fork()´ is the answer?
I did this too. The forked image just saves some data structure to a file, then exits. The parent continues modifying the data. The file is only used at startup to initialize the data structure.
What I wanted to achive, was to have a snapshot of the data regularly, and the normal execution thread should not have to spend time saving.
There was complexity, like I had to call wait on the daughter process, and I had to make sure that one save was completed before the next one was started.
I could have done it with the multiprocessing package in cmucl, but I expected there would be a problem with simultaneous updating of the data structure.
Would this be reasonable, or is it just unnessesary old C thinking?
> I did this too. The forked image just saves some data structure to a > file, then exits. The parent continues modifying the data. The file is > only used at startup to initialize the data structure.
Actually, that's a really good example. We did[1] more-or-less exactly this. We had an application which some fairly large amount of state (~a few GB). Every so often we wanted to checkpoint the state in such a way that we didn't have large downtime. We could (and originally did) do this by freezing the system, dumping state, and then unfreezing it. But this fails on the downtime issue. So instead we can just fork, and one branch of the fork can then carry on running, while the other saves the application data and then exits. It's a nice solution to some problems.
--tim
[1] I think we didn't actually implement this but it was talked about and prototyped.
Tim Bradshaw wrote: > * Johan Ur Riise wrote: > > I did this too. The forked image just saves some data structure to a > > file, then exits. The parent continues modifying the data. The file is > > only used at startup to initialize the data structure. > Actually, that's a really good example. We did[1] more-or-less > exactly this. We had an application which some fairly large amount of > state (~a few GB). Every so often we wanted to checkpoint the state > in such a way that we didn't have large downtime. We could (and > originally did) do this by freezing the system, dumping state, and > then unfreezing it. But this fails on the downtime issue. So instead > we can just fork, and one branch of the fork can then carry on > running, while the other saves the application data and then exits. > It's a nice solution to some problems.
This requires a sufficiently nice fork implementation to keep the memory usage bearable. Several years ago I was running CMU-CL 17f on HP-UX 9.05 on a machine with a whopping 32 MB of RAM. That combination merrily copied the whole process when forking (starting the debugger, IIRC), thereby immediately killing the whole application when it ran out of memory.
Tim Bradshaw wrote: > > I did this too. The forked image just saves some data structure to a > > file, then exits. The parent continues modifying the data. The file is > > only used at startup to initialize the data structure.
> Actually, that's a really good example. We did[1] more-or-less > exactly this. ... > [1] I think we didn't actually implement this but it was talked about > and prototyped.
The Fuzzball MUCK server does this. It's not written in Lisp, though.
Martti Halminen <martti.halmi...@kolumbus.fi> writes: > Tim Bradshaw wrote:
> > * Johan Ur Riise wrote: > > > I did this too. The forked image just saves some data structure to a > > > file, then exits.
> > Actually, that's a really good example. We did[1] more-or-less > > exactly this. We had an application which some fairly large amount of > > state (~a few GB). Every so often we wanted to checkpoint the state > > in such a way that we didn't have large downtime. We could (and > > originally did) do this by freezing the system, dumping state, and > > then unfreezing it. But this fails on the downtime issue. So instead > > we can just fork, and one branch of the fork can then carry on > > running, while the other saves the application data and then exits. > > It's a nice solution to some problems.
> This requires a sufficiently nice fork implementation to keep the memory > usage bearable. Several years ago I was running CMU-CL 17f on HP-UX 9.05 > on a machine with a whopping 32 MB of RAM. That combination merrily > copied the whole process when forking (starting the debugger, IIRC), > thereby immediately killing the whole application when it ran out of > memory.
But on a system with a copy-on-write policy it should be worth a try, I should think.
Regards, -- Nils Goesche Ask not for whom the <CONTROL-G> tolls.
>>>>> On 03 Sep 2002 03:58:46 +0200, Nils Goesche ("Nils") writes:
Nils> Martti Halminen <martti.halmi...@kolumbus.fi> writes: >> Tim Bradshaw wrote: >> >> > * Johan Ur Riise wrote: >> > > I did this too. The forked image just saves some data structure to a >> > > file, then exits. >> >> > Actually, that's a really good example. We did[1] more-or-less >> > exactly this. We had an application which some fairly large amount of >> > state (~a few GB). Every so often we wanted to checkpoint the state >> > in such a way that we didn't have large downtime. We could (and >> > originally did) do this by freezing the system, dumping state, and >> > then unfreezing it. But this fails on the downtime issue. So instead >> > we can just fork, and one branch of the fork can then carry on >> > running, while the other saves the application data and then exits. >> > It's a nice solution to some problems. >> >> This requires a sufficiently nice fork implementation to keep the memory >> usage bearable. Several years ago I was running CMU-CL 17f on HP-UX 9.05 >> on a machine with a whopping 32 MB of RAM. That combination merrily >> copied the whole process when forking (starting the debugger, IIRC), >> thereby immediately killing the whole application when it ran out of >> memory.
Nils> But on a system with a copy-on-write policy it should be worth Nils> a try, I should think.
Yes, assuming that the mutating process is not changing everything around (which would result in the OS copying all the pages for the forked process). The implementation of the GC is also factor there.
> > > I did this too. The forked image just saves some data structure to a > > > file, then exits. The parent continues modifying the data. The file is > > > only used at startup to initialize the data structure.
> > Actually, that's a really good example. We did[1] more-or-less > > exactly this. > ... > > [1] I think we didn't actually implement this but it was talked about > > and prototyped.
> The Fuzzball MUCK server does this. It's not written in Lisp, though.
As does LambdaMOO (which is also not written in Lisp).
I know of one Lisp system which is able to perform this sort of world save while running but without the assistance of "fork". Like the "fork" approach, there exists a process which traverses an object hierarchy writing out state information. However, this process is interleaved with the normal application process(es) and operates on the same set of live objects. The trick is basically that all object slot writers first check to see if the system is in the midst of a world save. If the system is within this state and the application attempts to mutate an object that has not been saved out to disk, that object is immediately forced out to disk. I was surprised to see how in practice, this system turns out to work quite well.
