^^ that would then wrap the binary, enforcing a maximum address space of 256M.
As you gave the option --limit-as to uwsgi, which can apply to any python webapp, I was wondering if it's possible for us to do the same, but without compiling the original binary into uwsgi.
Let me know if the above doesn't make any sense lol.
> ^^ that would then wrap the binary, enforcing a maximum address space of 256M.
> As you gave the option --limit-as to uwsgi, which can apply to any > python webapp, I was wondering if it's possible for us to do the same, > but without compiling the original binary into uwsgi.
> Let me know if the above doesn't make any sense lol.
Using ulimit is going to be an issue as it relies on the host allowing users to modify their ulimit (some aren't allowed). It also then applies that rule to any other processes within that user, which is bad as different processes may need different limits.
I have seen other examples in the past of binaries being wrapped, which then applied a memory limit, but I can't for the life of me remember what it was called. :(
On Fri, Jul 8, 2011 at 2:33 PM, Tom Evans <tevans...@googlemail.com> wrote: > On Fri, Jul 8, 2011 at 2:23 PM, Cal Leeming [Simplicity Media Ltd] > <cal.leem...@simplicitymedialtd.co.uk> wrote: >> Hi Roberto,
>> Wonder if you might be able to give me some advice..
>> Is there is a way to limit the address space of any other binary, >> without relying on /etc/security/limits.conf??
>> For example, is it possible to do something like this:
>> ^^ that would then wrap the binary, enforcing a maximum address space of 256M.
>> As you gave the option --limit-as to uwsgi, which can apply to any >> python webapp, I was wondering if it's possible for us to do the same, >> but without compiling the original binary into uwsgi.
>> Let me know if the above doesn't make any sense lol.
>> Cal
> ulimit -a
> Eg: ulimit -s 4096
> Cheers
> Tom
> -- > You received this message because you are subscribed to the Google Groups "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
On Fri, Jul 8, 2011 at 2:33 PM, Tom Evans <tevans...@googlemail.com> wrote: > On Fri, Jul 8, 2011 at 2:23 PM, Cal Leeming [Simplicity Media Ltd] > <cal.leem...@simplicitymedialtd.co.uk> wrote: >> Hi Roberto,
>> Wonder if you might be able to give me some advice..
>> Is there is a way to limit the address space of any other binary, >> without relying on /etc/security/limits.conf??
>> For example, is it possible to do something like this:
>> ^^ that would then wrap the binary, enforcing a maximum address space of 256M.
>> As you gave the option --limit-as to uwsgi, which can apply to any >> python webapp, I was wondering if it's possible for us to do the same, >> but without compiling the original binary into uwsgi.
>> Let me know if the above doesn't make any sense lol.
>> Cal
> ulimit -a
> Eg: ulimit -s 4096
> Cheers
> Tom
> -- > You received this message because you are subscribed to the Google Groups "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
On Fri, Jul 8, 2011 at 2:43 PM, Cal Leeming [Simplicity Media Ltd]
<cal.leem...@simplicitymedialtd.co.uk> wrote: > Sorry, I should have given a bit more detail.
> Using ulimit is going to be an issue as it relies on the host allowing > users to modify their ulimit (some aren't allowed). It also then > applies that rule to any other processes within that user, which is > bad as different processes may need different limits.
ulimit doesn't do that. ulimit as a user only allows a user to lower their hard limits to something in the range (0, hard limit). Only super user can increase limits.
It also only applies to processes created after ulimit has run. Therefore, your server start script could (probably should) apply limits to the server processes, whilst not affecting the users limits in their shell or other processes.
> I have seen other examples in the past of binaries being wrapped, > which then applied a memory limit, but I can't for the life of me > remember what it was called. :(
It's called ulimit. This is the designed use for it.
Hmm, I should have posted this to a C mailing list, as it's extremely off topic for Django.
I'll move this discussion to a different list.. but here's what I've found so far (for anyone interested).
Within C code, setting 'rlim_max' allows you to set a floor limit for a process (http://linux.die.net/man/2/setrlimit). However, I am yet to find out if this limit is inherited after spawning/forking.
On Fri, Jul 8, 2011 at 2:43 PM, Cal Leeming [Simplicity Media Ltd]
<cal.leem...@simplicitymedialtd.co.uk> wrote: > Thank you for taking the time to reply though!
> On Fri, Jul 8, 2011 at 2:33 PM, Tom Evans <tevans...@googlemail.com> wrote: >> On Fri, Jul 8, 2011 at 2:23 PM, Cal Leeming [Simplicity Media Ltd] >> <cal.leem...@simplicitymedialtd.co.uk> wrote: >>> Hi Roberto,
>>> Wonder if you might be able to give me some advice..
>>> Is there is a way to limit the address space of any other binary, >>> without relying on /etc/security/limits.conf??
>>> For example, is it possible to do something like this:
>>> ^^ that would then wrap the binary, enforcing a maximum address space of 256M.
>>> As you gave the option --limit-as to uwsgi, which can apply to any >>> python webapp, I was wondering if it's possible for us to do the same, >>> but without compiling the original binary into uwsgi.
>>> Let me know if the above doesn't make any sense lol.
>>> Cal
>> ulimit -a
>> Eg: ulimit -s 4096
>> Cheers
>> Tom
>> -- >> You received this message because you are subscribed to the Google Groups "Django users" group. >> To post to this group, send email to django-users@googlegroups.com. >> To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
On Fri, Jul 08, 2011 at 02:43:11PM +0100, Cal Leeming [Simplicity Media Ltd] wrote:
> Sorry, I should have given a bit more detail.
> Using ulimit is going to be an issue as it relies on the host allowing > users to modify their ulimit (some aren't allowed). It also then > applies that rule to any other processes within that user, which is > bad as different processes may need different limits.
My bash(1) man page says the following: "Provides control over the resources available to the shell and to processes started by it, on systems that allow such control."
Also, setrlimit(3) states this: "A child process created via fork(2) inherits its parent's resource limits. Resource limits are preserved across execve(2)."
Which means, if you set a ulimit in a process, only the process itself and its children are affected. At least that's how I understand it.
My understanding is backed by the fact that online judges used in ACM ICPC-like contest use setrlimit in their sandboxes to limit the memory for submitted solutions. (-;
> I have seen other examples in the past of binaries being wrapped, > which then applied a memory limit, but I can't for the life of me > remember what it was called. :(
Just running it as bash -c 'ulimit -S <set your limits here>; command' should be enough. Or if you want, you can create your own wrapper in C which calls setrlimit and then execs the binary.
On Fri, Jul 8, 2011 at 2:55 PM, Michal Petrucha <michal.petru...@ksp.sk> wrote: > On Fri, Jul 08, 2011 at 02:43:11PM +0100, Cal Leeming [Simplicity Media Ltd] wrote: >> Sorry, I should have given a bit more detail.
>> Using ulimit is going to be an issue as it relies on the host allowing >> users to modify their ulimit (some aren't allowed). It also then >> applies that rule to any other processes within that user, which is >> bad as different processes may need different limits.
> My bash(1) man page says the following: > "Provides control over the resources available to the shell and to > processes started by it, on systems that allow such control."
> Also, setrlimit(3) states this: > "A child process created via fork(2) inherits its parent's resource > limits. Resource limits are preserved across execve(2)."
Ah, I found setrlimit earlier, but I somehow missed the section about fork() inheriting the parent's resource limits (guess I skimmed through the man too quickly), so many thanks for clarifying this.
> Which means, if you set a ulimit in a process, only the process itself > and its children are affected. At least that's how I understand it.
> My understanding is backed by the fact that online judges used in ACM > ICPC-like contest use setrlimit in their sandboxes to limit the memory > for submitted solutions. (-;
Ack.
>> I have seen other examples in the past of binaries being wrapped, >> which then applied a memory limit, but I can't for the life of me >> remember what it was called. :(
> Just running it as > bash -c 'ulimit -S <set your limits here>; command' should be enough. > Or if you want, you can create your own wrapper in C which calls > setrlimit and then execs the binary.
Although I'm sure both methods would work, would you recommend any particular preference? (as my preference is merely on the fact that having a wrapper seems a lot cleaner).
I guess the custom binary I was trying to think of earlier used the setrlimit/fork method mentioned above.
On Fri, Jul 8, 2011 at 3:04 PM, Cal Leeming [Simplicity Media Ltd]
<cal.leem...@simplicitymedialtd.co.uk> wrote: > Although I'm sure both methods would work, would you recommend any > particular preference? (as my preference is merely on the fact that > having a wrapper seems a lot cleaner).
> I guess the custom binary I was trying to think of earlier used the > setrlimit/fork method mentioned above.
There is only one way to modify limits, and that is setrlimit, which is how ulimit is implemented. Given ulimit and setrlimit are both in POSIX, I don't see why anyone would write their own wrapper to do this.
The way I typically add limits is to edit the rc (startup) script, and add a call to ulimit there. This applies the limit to the process which launches the daemon, its a simple clean one liner, and doesn't require massaging command line arguments into a sh -c '' line (or bash).
On Fri, Jul 8, 2011 at 3:13 PM, Tom Evans <tevans...@googlemail.com> wrote: > On Fri, Jul 8, 2011 at 3:04 PM, Cal Leeming [Simplicity Media Ltd] > <cal.leem...@simplicitymedialtd.co.uk> wrote: >> Although I'm sure both methods would work, would you recommend any >> particular preference? (as my preference is merely on the fact that >> having a wrapper seems a lot cleaner).
>> I guess the custom binary I was trying to think of earlier used the >> setrlimit/fork method mentioned above.
> There is only one way to modify limits, and that is setrlimit, which > is how ulimit is implemented. Given ulimit and setrlimit are both in > POSIX, I don't see why anyone would write their own wrapper to do > this.
My worries were that in the event of children processes spawning some a single parent, both executing ulimit, there could potentially be a race condition.
In the above scenario, ulimit would apply that limit to everything within that forked supervisord instance, correct?
Therefore, if nginx-wrapper calls ulimit at almost the exact same point as other-wrapper, there could be a race condition between the time it takes to jump from ulimit to the binary, where the wrong address space is inherited.
> The way I typically add limits is to edit the rc (startup) script, and > add a call to ulimit there. This applies the limit to the process > which launches the daemon, its a simple clean one liner, and doesn't > require massaging command line arguments into a sh -c '' line (or > bash).
> Cheers
> Tom
> -- > You received this message because you are subscribed to the Google Groups "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
<cal.leem...@simplicitymedialtd.co.uk> wrote: > On Fri, Jul 8, 2011 at 3:13 PM, Tom Evans <tevans...@googlemail.com> wrote: >> On Fri, Jul 8, 2011 at 3:04 PM, Cal Leeming [Simplicity Media Ltd] >> <cal.leem...@simplicitymedialtd.co.uk> wrote: >>> Although I'm sure both methods would work, would you recommend any >>> particular preference? (as my preference is merely on the fact that >>> having a wrapper seems a lot cleaner).
>>> I guess the custom binary I was trying to think of earlier used the >>> setrlimit/fork method mentioned above.
>> There is only one way to modify limits, and that is setrlimit, which >> is how ulimit is implemented. Given ulimit and setrlimit are both in >> POSIX, I don't see why anyone would write their own wrapper to do >> this.
> My worries were that in the event of children processes spawning some
> In the above scenario, ulimit would apply that limit to everything > within that forked supervisord instance, correct?
> Therefore, if nginx-wrapper calls ulimit at almost the exact same > point as other-wrapper, there could be a race condition between the > time it takes to jump from ulimit to the binary, where the wrong > address space is inherited.
> Or have I understood that wrong??
>> The way I typically add limits is to edit the rc (startup) script, and >> add a call to ulimit there. This applies the limit to the process >> which launches the daemon, its a simple clean one liner, and doesn't >> require massaging command line arguments into a sh -c '' line (or >> bash).
>> Cheers
>> Tom
>> -- >> You received this message because you are subscribed to the Google Groups "Django users" group. >> To post to this group, send email to django-users@googlegroups.com. >> To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
On Fri, Jul 8, 2011 at 9:18 AM, Cal Leeming [Simplicity Media Ltd]
<cal.leem...@simplicitymedialtd.co.uk> wrote: > In the above scenario, ulimit would apply that limit to everything > within that forked supervisord instance, correct?
> Therefore, if nginx-wrapper calls ulimit at almost the exact same > point as other-wrapper, there could be a race condition between the > time it takes to jump from ulimit to the binary, where the wrong > address space is inherited.
if nginx-wrapper and other-wrapper is each one a different process, then ulimit would affect only to that process and descendants. it doesn't propagate to brother-processes
besides, this is posix we're talking about; there are no race conditions in calling executables or atomic system calls
On Fri, Jul 8, 2011 at 3:26 PM, Javier Guerra Giraldez <jav...@guerrag.com>wrote:
> On Fri, Jul 8, 2011 at 9:18 AM, Cal Leeming [Simplicity Media Ltd] > <cal.leem...@simplicitymedialtd.co.uk> wrote: > > In the above scenario, ulimit would apply that limit to everything > > within that forked supervisord instance, correct?
> > Therefore, if nginx-wrapper calls ulimit at almost the exact same > > point as other-wrapper, there could be a race condition between the > > time it takes to jump from ulimit to the binary, where the wrong > > address space is inherited.
> if nginx-wrapper and other-wrapper is each one a different process, > then ulimit would affect only to that process and descendants. it > doesn't propagate to brother-processes
> besides, this is posix we're talking about; there are no race > conditions in calling executables or atomic system calls > -- > Javier
> -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en.
> ^^ that would then wrap the binary, enforcing a maximum address space of > 256M.
> As you gave the option --limit-as to uwsgi, which can apply to any > python webapp, I was wondering if it's possible for us to do the same, > but without compiling the original binary into uwsgi.
> Let me know if the above doesn't make any sense lol.
> Cal
you can use the --worker-exec option.
This is used to run php-fastcgi (or other software requiring specific environment) under the uWSGI environment
uwsgi --limit-as 256 --worker-exec /bin/ls
As others have already said to you, remember that limits set by a parent are inherited by all of the children. So running apps via fork+exec or via attach-daemon in uWSGI will set their limit automatically.
> > ^^ that would then wrap the binary, enforcing a maximum address space of > > 256M.
> > As you gave the option --limit-as to uwsgi, which can apply to any > > python webapp, I was wondering if it's possible for us to do the same, > > but without compiling the original binary into uwsgi.
> > Let me know if the above doesn't make any sense lol.
> > Cal
> you can use the --worker-exec option.
> This is used to run php-fastcgi (or other software requiring specific > environment) under the uWSGI environment
> uwsgi --limit-as 256 --worker-exec /bin/ls
> As others have already said to you, remember that limits set by a parent > are inherited by all of the children. So running apps via fork+exec or via > attach-daemon in uWSGI will set their limit automatically.
> -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en.
