Limiting resources used by scripts on web server
graeme....@gmail.com
to limit resources (especially memory) used by a script?
So far I have:
1) FastCGI and use a process supervisor and/or OS limits
2) Apache pre-fork with websh (or Rivet or mod_tcl?) and a similar
approach
As far as I can see there is no way of doing this with Aolserver,
TClHttpd or Wub. I have used TCLHttpd once before and liked it a lot,
Wub and AOlserver look good too (although Wub seems to lack
documentation). Is there a way I could use any of these?
Bauk seems very nice, but hardly anyone seems to be using it. It does
what I want, but may not be well tested and there does not seem to be
much of a community.
Are there any other options I have missed?
Gerald W. Lester
> What are my options for a webserver that will support TCL and allow me
> to limit resources (especially memory) used by a script?
>
> So far I have:
>
> 1) FastCGI and use a process supervisor and/or OS limits
> 2) Apache pre-fork with websh (or Rivet or mod_tcl?) and a similar
> approach
>
> As far as I can see there is no way of doing this with Aolserver,
> TClHttpd or Wub. I have used TCLHttpd once before and liked it a lot,
> Wub and AOlserver look good too (although Wub seems to lack
> documentation). Is there a way I could use any of these?
TclHttpd supports cgi as well as threads, thus you could use the same
limiting approach as #1.
Also, if you use TclHTtpd with Tcl 8.5, 8.5 give you several ways to control
execution resource usage in a slave interp.
>
> Bauk seems very nice, but hardly anyone seems to be using it. It does
> what I want, but may not be well tested and there does not seem to be
> much of a community.
>
> Are there any other options I have missed?
--
+--------------------------------+---------------------------------------+
| Gerald W. Lester |
|"The man who fights for his ideals is the man who is alive." - Cervantes|
+------------------------------------------------------------------------+
graeme....@gmail.com
> graeme.piete...@gmail.com wrote:
> > What are my options for a webserver that will support TCL and allow me
> > to limit resources (especially memory) used by a script?
>
> > So far I have:
>
> > 1) FastCGI and use a process supervisor and/or OS limits
> > 2) Apache pre-fork with websh (or Rivet or mod_tcl?) and a similar
> > approach
>
> > As far as I can see there is no way of doing this with Aolserver,
> > TClHttpd or Wub. I have used TCLHttpd once before and liked it a lot,
> > Wub and AOlserver look good too (although Wub seems to lack
> > documentation). Is there a way I could use any of these?
>
> TclHttpd supports cgi as well as threads, thus you could use the same
> limiting approach as #1.
I had forgotten that TCLHttpd supports CGI, but it does rather negate
the advantages of using a server with TCL support.
>
> Also, if you use TclHTtpd with Tcl 8.5, 8.5 give you several ways to control
> execution resource usage in a slave interp.
If I understand the documentation, there are limits on time but on
memory. If I am wrong about it would solve the problem very nicely.
Alexandre Ferrieux
<graeme.piete...@gmail.com> wrote:
>
> I had forgotten that TCLHttpd supports CGI, but it does rather negate
> the advantages of using a server with TCL support.
>
> memory. If I am wrong about it would solve the problem very nicely.
Well, you're asking too much I'm afraid. As soon as you insist on
servicing all requests from a single process (which is what you're
doing by rejecting CGI), pre-request memory accounting becomes
extremely hard.
More precisely, accounting by the OS becomes impossible (since the
heap is shared among all threads of a process), so must be done at the
application level. And when the application is Tcl, it turns out it is
hard because (1) Tcl_Objs are allocated globally, and not bound to a
thread/interp, (2) to my knowledge there is no instrumentation to keep
track of how many Tcl_Objs were created by a given thread/interp.
Advice: if you're worried by memory consumption of a single request,
my bet is that the granularity will be large enough for CGI to be an
acceptable vehicle. And it will give you 'ulimit' for free.
-Alex
graeme....@gmail.com
wrote:
> On Jun 30, 8:59 am, "graeme.piete...@gmail.com"
>
> <graeme.piete...@gmail.com> wrote:
>
> > I had forgotten that TCLHttpd supports CGI, but it does rather negate
> > the advantages of using a server with TCL support.
>
> > If I understand the documentation, there are limits on time but on
> > memory. If I am wrong about it would solve the problem very nicely.
>
> Well, you're asking too much I'm afraid. As soon as you insist on
> servicing all requests from a single process (which is what you're
> doing by rejecting CGI), pre-request memory accounting becomes
> extremely hard.
What I meant by that is that if I use CGI I might as well use Apache
as TclHttpd.
If I do use Apache then I might as well use FastCGI or an Apache
module like Websh rather than CGI.
> More precisely, accounting by the OS becomes impossible (since the
> heap is shared among all threads of a process), so must be done at the
> application level. And when the application is Tcl, it turns out it is
> hard because (1) Tcl_Objs are allocated globally, and not bound to a
> thread/interp, (2) to my knowledge there is no instrumentation to keep
> track of how many Tcl_Objs were created by a given thread/interp.
I had a vague idea it was difficult, but your explanation makes it
clearer. Thanks.
Donal K. Fellows
> [...] it turns out it is
> hard because (1) Tcl_Objs are allocated globally, and not bound to a
> thread/interp,
Actually, they're thread-bound (there's a per-thread memory pool).
> (2) to my knowledge there is no instrumentation to keep
> track of how many Tcl_Objs were created by a given thread/interp.
On a thread level, if there isn't (I'd need to check the source to see
if there is when memory debugging is turned on) then it would be easy to
add. The real problems are that all other accounting is per-interpreter
(memory is often allocated in contexts where there is no interpreter to
account it against) and it is very difficult to recover from memory
allocation failures — almost the whole of Tcl assumes that Tcl_Alloc()
never ever fails, which is true right now and also massively simplifies
a lot of code — and everything is made more complex by the fact that any
recovery strategy will almost certainly need to allocate more memory
while it runs...
Donal.
Alexandre Ferrieux
<donal.k.fell...@manchester.ac.uk> wrote:
> Alexandre Ferrieux wrote:
> > [...] it turns out it is
> > hard because (1) Tcl_Objs are allocated globally, and not bound to a
> > thread/interp,
>
> Actually, they're thread-bound (there's a per-thread memory pool).
Well, it was my understanding that this private part was just a cache
of bounded size, and that beyond this threshold everything was in
anonymous malloc()s...
-Alex
Donal K. Fellows
> Well, it was my understanding that this private part was just a cache
> of bounded size, and that beyond this threshold everything was in
> anonymous malloc()s...
That's possibly true. I'd need to read the source to be sure. But in any
case, it's still pretty easy to do thread-level accounting of memory;
there's always a current thread available when you're doing a malloc().
Donal.
Alexandre Ferrieux
OK. The question then, is would we put this accounting inside or
outside an #ifdef TCL_SOME_DEBUG, because of the possible perf hit to
the vast majority (1-1e-5 is an underestimate) of scripts uninterested
in it ;-)
-Alex
David Gravereaux
> What are my options for a webserver that will support TCL and allow me
> to limit resources (especially memory) used by a script?
Is this for development/test purposes or for use in production? What is
the dropback behavior when a limit is reached, return a '500 server too
busy' and wait for the current activity to complete?
graeme....@gmail.com
For production.
A 500 is OK, because it should only happen when something is wrong or
the user is malicious.