I had an idea to play with this long weekend and I want to share some results and get some feedback.
Basically, I was wondering whether it will be feasible to extend the concept of group leaders beyond I/O so that you can build your own primitives around this construct of inherited group leaders. It seemed to me that tasks like implicit configuration, security groups and such can benefit from having a more generalized group leadership mechanism.
The EEP has some examples, and you can play with that PoC implementation (group_leader_scope branch). It's not ready for the prime time, though — it lacks automated tests and the performance implications have not yet been measured (although this should be fixed soon). No optimizations were considered at this moment.
The EEP draft itself has not been sent to EEP editors as I plan to elaborate on motivations and possibly add more examples.
Either way, please let me know if this is of any interest to anybody and if you have any thoughts about it.
I agree this might be quite useful. To me one of the interesting points has to be the possibility to write your own protocols similar to the IO one. I could see something like a web server or any other resource handler of the kind allowing a bunch of workers to pipe their output straight out to some socket, file descriptor or whatever.
The abstractions put in place in front of the IO server could be used across other resources with potentially different semantics (and thus requiring a different protocol). This could be a nice step towards that.
> I can actually see a ton of uses for this. Not the least of which is > security, I just did a deep security piece in an erlang app, and it > turned out well, but I could see where something like this would have > made it much more elegant. The only thing that would concern me (from > looking at your implementation) is how much over head is added to > process creation. I think thats probably the main unknown, there are > probably some other things as well.
> Just out of curiosity does a group leader go away when a group dies? > On Sat, Jul 2, 2011 at 7:38 PM, Yurii Rashkovskii <yra...@gmail.com> wrote: >> Hi,
>> I had an idea to play with this long weekend and I want to share some >> results and get some feedback.
>> Basically, I was wondering whether it will be feasible to extend the >> concept of group leaders beyond I/O so that you can build your own >> primitives around this construct of inherited group leaders. It seemed >> to me that tasks like implicit configuration, security groups and such >> can benefit from having a more generalized group leadership mechanism.
>> The EEP has some examples, and you can play with that PoC >> implementation (group_leader_scope branch). It's not ready for the >> prime time, though — it lacks automated tests and the performance >> implications have not yet been measured (although this should be fixed >> soon). No optimizations were considered at this moment.
>> The EEP draft itself has not been sent to EEP editors as I plan to >> elaborate on motivations and possibly add more examples.
>> Either way, please let me know if this is of any interest to anybody >> and if you have any thoughts about it.
I can actually see a ton of uses for this. Not the least of which is security, I just did a deep security piece in an erlang app, and it turned out well, but I could see where something like this would have made it much more elegant. The only thing that would concern me (from looking at your implementation) is how much over head is added to process creation. I think thats probably the main unknown, there are probably some other things as well.
Just out of curiosity does a group leader go away when a group dies?
On Sat, Jul 2, 2011 at 7:38 PM, Yurii Rashkovskii <yra...@gmail.com> wrote: > Hi,
> I had an idea to play with this long weekend and I want to share some > results and get some feedback.
> Basically, I was wondering whether it will be feasible to extend the > concept of group leaders beyond I/O so that you can build your own > primitives around this construct of inherited group leaders. It seemed > to me that tasks like implicit configuration, security groups and such > can benefit from having a more generalized group leadership mechanism.
> The EEP has some examples, and you can play with that PoC > implementation (group_leader_scope branch). It's not ready for the > prime time, though — it lacks automated tests and the performance > implications have not yet been measured (although this should be fixed > soon). No optimizations were considered at this moment.
> The EEP draft itself has not been sent to EEP editors as I plan to > elaborate on motivations and possibly add more examples.
> Either way, please let me know if this is of any interest to anybody > and if you have any thoughts about it.
> I can actually see a ton of uses for this. Not the least of which is > security, I just did a deep security piece in an erlang app, and it > turned out well, but I could see where something like this would have > made it much more elegant
How, exactly?
The things that are difficult to do today, in terms of security in Erlang, is to *enforce* security in a way that cannot be subverted.
Group leaders are natural for IO, since IO necessarily involves sending a message to some entity. Some other things are overloaded - e.g. the application controller using the group leader to find out which application a process belongs to. That is a clear case where I think this would be useful: an "application group leader". But for security, I don't see it solving any of the problems I consider difficult - perhaps this is my failing: after having implemented reasonably transparent security in ErlHive, I get all kinds of associations to nasty problems that wouldn't be helped by this suggestion. But perhaps you are viewing it from a different angle?
> On Mon, Jul 4, 2011 at 10:15 AM, Ulf Wiger <ulf.wi...@erlang-solutions.com> wrote:
> On 4 Jul 2011, at 16:15, Eric Merritt wrote:
>> I can actually see a ton of uses for this. Not the least of which is >> security, I just did a deep security piece in an erlang app, and it >> turned out well, but I could see where something like this would have >> made it much more elegant
> How, exactly?
In the existing model a context is passed through out the system, identifying the users and their capabilities. In this case, assuming that all threads related to use where spawned at some level by the incoming process, a context would not have had to be passed, it could have been implicit in the group leader. This would have been much less invasive in the system. I made no attempt to protect against developers adding malicious code to the system. That is hard regardless of the model. Perhaps this would have been better called an access control system rather then a security system given the assumptions.
> The things that are difficult to do today, in terms of security in Erlang, is to *enforce* security in a way that cannot be subverted.
> Group leaders are natural for IO, since IO necessarily involves sending a message to some entity. Some other things are overloaded - >e.g. the application controller using the group leader to find out which application a process belongs to. That is a clear case where I think >this would be useful: an "application group leader". But for security, I don't see it solving any of the problems I consider difficult - perhaps >this is my failing: after having implemented reasonably transparent security in ErlHive, I get all kinds of associations to nasty problems >that wouldn't be helped by this suggestion. But perhaps you are viewing it from a different angle?
I am not thinking about this as solving any truly hard problems, not that can't be solved in some other way. It just would have made the code created for this project, more elegant, and the security aspects much less invasive then they currently are, at the cost of slightly more difficult testing and some knowledge about where the process group boundries are.
> The only thing that would concern me (from > looking at your implementation) is how much over head is added to > process creation. I think thats probably the main unknown, there are > probably some other things as well.
Judging by my limited "unscientific" performance testing that I did yesterday, I've got virtually no impact for situations where you do not create any extra group leaders (i.e. running code that can be done today). In a case when you have a hundred extra group leaders, it slows down by about 2.5 times.
Please bear in mind that current implementation is a proof-of-concept level quality. I can already foresee some techniques for speeding up group leader allocation by allocating on process creation in bulk, it's rather simple and will reduce number of allocations to just one. In fact, this is so easy I might update this PoC implementation rather soon! It sounds like a good idea.
> Just out of curiosity does a group leader go away when a group dies?
>> Just out of curiosity does a group leader go away when a group dies?
> Define group death?
When when all processes in a group have terminated, either normally or abnormally. _______________________________________________ erlang-questions mailing list erlang-questi...@erlang.org http://erlang.org/mailman/listinfo/erlang-questions
On Mon, Jul 4, 2011 at 9:18 AM, Eric Merritt <ericbmerr...@gmail.com> wrote:
>>> Just out of curiosity does a group leader go away when a group dies?
>> Define group death?
> When when all processes in a group have terminated, either normally or > abnormally.
I have intentionally decided not to overcomplicate current implementation. Right now nothing will happen. _______________________________________________ erlang-questions mailing list erlang-questi...@erlang.org http://erlang.org/mailman/listinfo/erlang-questions
> >> Just out of curiosity does a group leader go away when a group > dies?
> > Define group death?
> When when all processes in a group have terminated, either normally > or > abnormally.
Today there is nothing in the system which keeps track of the processes in a group, process groups don't really exist at all. A "process group" is those processes which have the same group leader. The group leader has no information about which processes have it as group leader. You can make any process a group leader by setting it as group leader.
Process groups were explicitly added in such a way as NOT to impose any structure or hierarchy on processes, process space is flat.
<robert.vird...@erlang-solutions.com> wrote: > Today there is nothing in the system which keeps track of the processes in a group, process groups don't really exist at all. A "process group" is those processes which have the same group leader. The group leader has no information about which processes have it as group leader. You can make any process a group leader by setting it as group leader.
> Process groups were explicitly added in such a way as NOT to impose any structure or hierarchy on processes, process space is flat.
That's right. I personally understand them as "ephemeral" notions, just like you defined. That's why I don't do anything if a group leader dies. It's a duty of individual process to decide what to do with this.
I understand and that makes sense, but it does raise a potential problem. That is group leaders are created but never destroyed. In the case of the process itself, this is probably ok. The developer will have to create some mechanism to know when his group leader is no longer useful and have it die on its own. However, I didn't see any mechanisms to unregister group leaders and reclaim the memory alloced when they are created. That could be me missing something when I was looking at the code, if so then the point is moot, however if not that's something that would probably need to be added to make this workable. Of course, that still leaves the problem that if a group leader dies unexpectedly, or the user mis-codes his application and forgets to unregister a group leader when it dies, then he has a memory leak. You could say that its a users problem and go from there, but leaving around unreferenced memory just doesn't smell right. (I know, we do that already with the atom table, but it doesn't make it a good thing)
>> >> Just out of curiosity does a group leader go away when a group >> dies?
>> > Define group death?
>> When when all processes in a group have terminated, either normally >> or >> abnormally.
> Today there is nothing in the system which keeps track of the processes in a group, process groups don't really exist at all. A "process group" is those processes which have the same group leader. The group leader has no information about which processes have it as group leader. You can make any process a group leader by setting it as group leader.
> Process groups were explicitly added in such a way as NOT to impose any structure or hierarchy on processes, process space is flat.
On Tue, Jul 5, 2011 at 10:31 AM, Yurii Rashkovskii <yra...@gmail.com> wrote: > Right now there is no way to remove a group leader from process, but > it is not inherently impossible to do so, just was out of scope of > this prototype.
> On Tue, Jul 5, 2011 at 8:05 AM, Eric Merritt <ericbmerr...@gmail.com> wrote: >> I understand and that makes sense, but it does raise a potential >> problem. That is group leaders are created but never destroyed. In the >> case of the process itself, this is probably ok. The developer will >> have to create some mechanism to know when his group leader is no >> longer useful and have it die on its own. However, I didn't see any >> mechanisms to unregister group leaders and reclaim the memory alloced >> when they are created. That could be me missing something when I was >> looking at the code, if so then the point is moot, however if not >> that's something that would probably need to be added to make this >> workable. Of course, that still leaves the problem that if a group >> leader dies unexpectedly, or the user mis-codes his application and >> forgets to unregister a group leader when it dies, then he has a >> memory leak. You could say that its a users problem and go from there, >> but leaving around unreferenced memory just doesn't smell right. (I >> know, we do that already with the atom table, but it doesn't make it a >> good thing)
>> On Mon, Jul 4, 2011 at 6:57 PM, Robert Virding >> <robert.vird...@erlang-solutions.com> wrote: >>> ----- "Eric Merritt" <ericbmerr...@gmail.com> wrote:
>>>> >> Just out of curiosity does a group leader go away when a group >>>> dies?
>>>> > Define group death?
>>>> When when all processes in a group have terminated, either normally >>>> or >>>> abnormally.
>>> Today there is nothing in the system which keeps track of the processes in a group, process groups don't really exist at all. A "process group" is those processes which have the same group leader. The group leader has no information about which processes have it as group leader. You can make any process a group leader by setting it as group leader.
>>> Process groups were explicitly added in such a way as NOT to impose any structure or hierarchy on processes, process space is flat.
Right now there is no way to remove a group leader from process, but it is not inherently impossible to do so, just was out of scope of this prototype.
On Tue, Jul 5, 2011 at 8:05 AM, Eric Merritt <ericbmerr...@gmail.com> wrote: > I understand and that makes sense, but it does raise a potential > problem. That is group leaders are created but never destroyed. In the > case of the process itself, this is probably ok. The developer will > have to create some mechanism to know when his group leader is no > longer useful and have it die on its own. However, I didn't see any > mechanisms to unregister group leaders and reclaim the memory alloced > when they are created. That could be me missing something when I was > looking at the code, if so then the point is moot, however if not > that's something that would probably need to be added to make this > workable. Of course, that still leaves the problem that if a group > leader dies unexpectedly, or the user mis-codes his application and > forgets to unregister a group leader when it dies, then he has a > memory leak. You could say that its a users problem and go from there, > but leaving around unreferenced memory just doesn't smell right. (I > know, we do that already with the atom table, but it doesn't make it a > good thing)
> On Mon, Jul 4, 2011 at 6:57 PM, Robert Virding > <robert.vird...@erlang-solutions.com> wrote: >> ----- "Eric Merritt" <ericbmerr...@gmail.com> wrote:
>>> >> Just out of curiosity does a group leader go away when a group >>> dies?
>>> > Define group death?
>>> When when all processes in a group have terminated, either normally >>> or >>> abnormally.
>> Today there is nothing in the system which keeps track of the processes in a group, process groups don't really exist at all. A "process group" is those processes which have the same group leader. The group leader has no information about which processes have it as group leader. You can make any process a group leader by setting it as group leader.
>> Process groups were explicitly added in such a way as NOT to impose any structure or hierarchy on processes, process space is flat.
I am talking about the situation where no process that where once part of the group exist any longer. When a group has no processes, the group no longer exists, however the memory alloced by this will continue to exist, thats the problem.
<robert.vird...@erlang-solutions.com> wrote: > Conceptually a process is always the member of a group so the very idea of removing a group leader from a process does not make sense. All you can do is assign it another group leader.
>> I wouldn't expect in in the prototype. I am more thinking of the end >> result.
>> On Tue, Jul 5, 2011 at 10:31 AM, Yurii Rashkovskii <yra...@gmail.com> >> wrote: >> > Right now there is no way to remove a group leader from process, >> but >> > it is not inherently impossible to do so, just was out of scope of >> > this prototype.
>> > On Tue, Jul 5, 2011 at 8:05 AM, Eric Merritt >> <ericbmerr...@gmail.com> wrote: >> >> I understand and that makes sense, but it does raise a potential >> >> problem. That is group leaders are created but never destroyed. In >> the >> >> case of the process itself, this is probably ok. The developer >> will >> >> have to create some mechanism to know when his group leader is no >> >> longer useful and have it die on its own. However, I didn't see >> any >> >> mechanisms to unregister group leaders and reclaim the memory >> alloced >> >> when they are created. That could be me missing something when I >> was >> >> looking at the code, if so then the point is moot, however if not >> >> that's something that would probably need to be added to make this >> >> workable. Of course, that still leaves the problem that if a group >> >> leader dies unexpectedly, or the user mis-codes his application >> and >> >> forgets to unregister a group leader when it dies, then he has a >> >> memory leak. You could say that its a users problem and go from >> there, >> >> but leaving around unreferenced memory just doesn't smell right. >> (I >> >> know, we do that already with the atom table, but it doesn't make >> it a >> >> good thing)
>> >> On Mon, Jul 4, 2011 at 6:57 PM, Robert Virding >> >> <robert.vird...@erlang-solutions.com> wrote: >> >>> ----- "Eric Merritt" <ericbmerr...@gmail.com> wrote:
>> >>>> >> Just out of curiosity does a group leader go away when a >> group >> >>>> dies?
>> >>>> > Define group death?
>> >>>> When when all processes in a group have terminated, either >> normally >> >>>> or >> >>>> abnormally.
>> >>> Today there is nothing in the system which keeps track of the >> processes in a group, process groups don't really exist at all. A >> "process group" is those processes which have the same group leader. >> The group leader has no information about which processes have it as >> group leader. You can make any process a group leader by setting it as >> group leader.
>> >>> Process groups were explicitly added in such a way as NOT to >> impose any structure or hierarchy on processes, process space is >> flat.
> I am talking about the situation where no process that where once > part > of the group exist any longer. When a group has no processes, the > group no longer exists, however the memory alloced by this will > continue to exist, thats the problem.
> On Tue, Jul 5, 2011 at 6:04 PM, Robert Virding > <robert.vird...@erlang-solutions.com> wrote: > > Conceptually a process is always the member of a group so the very > idea of removing a group leader from a process does not make sense. > All you can do is assign it another group leader.
> >> I wouldn't expect in in the prototype. I am more thinking of the > end > >> result.
> >> On Tue, Jul 5, 2011 at 10:31 AM, Yurii Rashkovskii > <yra...@gmail.com> > >> wrote: > >> > Right now there is no way to remove a group leader from process, > >> but > >> > it is not inherently impossible to do so, just was out of scope > of > >> > this prototype.
> >> > On Tue, Jul 5, 2011 at 8:05 AM, Eric Merritt > >> <ericbmerr...@gmail.com> wrote: > >> >> I understand and that makes sense, but it does raise a > potential > >> >> problem. That is group leaders are created but never destroyed. > In > >> the > >> >> case of the process itself, this is probably ok. The developer > >> will > >> >> have to create some mechanism to know when his group leader is > no > >> >> longer useful and have it die on its own. However, I didn't see > >> any > >> >> mechanisms to unregister group leaders and reclaim the memory > >> alloced > >> >> when they are created. That could be me missing something when > I > >> was > >> >> looking at the code, if so then the point is moot, however if > not > >> >> that's something that would probably need to be added to make > this > >> >> workable. Of course, that still leaves the problem that if a > group > >> >> leader dies unexpectedly, or the user mis-codes his application > >> and > >> >> forgets to unregister a group leader when it dies, then he has > a > >> >> memory leak. You could say that its a users problem and go from > >> there, > >> >> but leaving around unreferenced memory just doesn't smell > right. > >> (I > >> >> know, we do that already with the atom table, but it doesn't > make > >> it a > >> >> good thing)
> >> >> On Mon, Jul 4, 2011 at 6:57 PM, Robert Virding > >> >> <robert.vird...@erlang-solutions.com> wrote: > >> >>> ----- "Eric Merritt" <ericbmerr...@gmail.com> wrote:
> >> >>>> >> Just out of curiosity does a group leader go away when a > >> group > >> >>>> dies?
> >> >>>> > Define group death?
> >> >>>> When when all processes in a group have terminated, either > >> normally > >> >>>> or > >> >>>> abnormally.
> >> >>> Today there is nothing in the system which keeps track of the > >> processes in a group, process groups don't really exist at all. A > >> "process group" is those processes which have the same group > leader. > >> The group leader has no information about which processes have it > as > >> group leader. You can make any process a group leader by setting it > as > >> group leader.
> >> >>> Process groups were explicitly added in such a way as NOT to > >> impose any structure or hierarchy on processes, process space is > >> flat.
Conceptually a process is always the member of a group so the very idea of removing a group leader from a process does not make sense. All you can do is assign it another group leader.
> I wouldn't expect in in the prototype. I am more thinking of the end > result.
> On Tue, Jul 5, 2011 at 10:31 AM, Yurii Rashkovskii <yra...@gmail.com> > wrote: > > Right now there is no way to remove a group leader from process, > but > > it is not inherently impossible to do so, just was out of scope of > > this prototype.
> > On Tue, Jul 5, 2011 at 8:05 AM, Eric Merritt > <ericbmerr...@gmail.com> wrote: > >> I understand and that makes sense, but it does raise a potential > >> problem. That is group leaders are created but never destroyed. In > the > >> case of the process itself, this is probably ok. The developer > will > >> have to create some mechanism to know when his group leader is no > >> longer useful and have it die on its own. However, I didn't see > any > >> mechanisms to unregister group leaders and reclaim the memory > alloced > >> when they are created. That could be me missing something when I > was > >> looking at the code, if so then the point is moot, however if not > >> that's something that would probably need to be added to make this > >> workable. Of course, that still leaves the problem that if a group > >> leader dies unexpectedly, or the user mis-codes his application > and > >> forgets to unregister a group leader when it dies, then he has a > >> memory leak. You could say that its a users problem and go from > there, > >> but leaving around unreferenced memory just doesn't smell right. > (I > >> know, we do that already with the atom table, but it doesn't make > it a > >> good thing)
> >> On Mon, Jul 4, 2011 at 6:57 PM, Robert Virding > >> <robert.vird...@erlang-solutions.com> wrote: > >>> ----- "Eric Merritt" <ericbmerr...@gmail.com> wrote:
> >>>> >> Just out of curiosity does a group leader go away when a > group > >>>> dies?
> >>>> > Define group death?
> >>>> When when all processes in a group have terminated, either > normally > >>>> or > >>>> abnormally.
> >>> Today there is nothing in the system which keeps track of the > processes in a group, process groups don't really exist at all. A > "process group" is those processes which have the same group leader. > The group leader has no information about which processes have it as > group leader. You can make any process a group leader by setting it as > group leader.
> >>> Process groups were explicitly added in such a way as NOT to > impose any structure or hierarchy on processes, process space is > flat.
