ANNOUNCE: WebStack 1.2.6
Paul Boddie
Improved Jython plus Servlet API support - I've given Jython and
different Java runtimes some exercise on my platform, and I've
improved some of the tools supplied. In addition, I've been working on
a Jython-oriented WebStack package for Ubuntu to match the Python-
oriented one, and this should help to make the web of Java
dependencies more explicit. In response to earlier requests, support
for forwarding to JSPs and other resources in the same container is
now supported, although a better API may be developed to apply to the
other frameworks as well.
Some small but important fixes for CGI, WSGI (environment variables),
Webware (support for 0.9.4) and Zope (paths).
Support for wsgiref has been added, and there is now a choice of WSGI
deployment functions instead of the single deploy function previously
exposed. It is now necessary to import the right function, although
this can be aliased to "deploy" in order to preserve the illusion.
The software is available, as usual, from the package index:
http://www.python.org/pypi/WebStack
Paul
Kevin Glynn
Hi Paul,
Paul Boddie writes:
>
> Announcing WebStack 1.2.6 with the following changes:
>
Firstly, (too long delayed) thanks for this latest release. We are
having a lot of fun using it for our application. We have a mixed
environment with one component using jython/JavaServlet and the rest
using ModPython.
I have some small comments, probably the only one worth mentioning is
that it would be nice to be able to add pythonpath entries on the
commandline when using webstack_xxxx_build.py. Its not so important
for the apache route, because you can just edit the webapp config file
one time, but for the javaservlet route which regenerates the web.xml
each time.
I have been investigating some headaches we have with
mutlipart/form-data request containing an embedded zip file. The
ModPython route handles this OK, except I think Filecontent should be
wrapping the file object rather than the content. Otherwise we will
lose all our RAM :-(.
The situation is much worse on the JavaServlet side where the call to
get_fields is taking over four minutes for an upload of !2MB :(. It
also seems to be leaking memory. I am trying to track it down, but
if you have any ideas or can recreate it I would love to hear them.
We have used 'eric' successfully to debug the modpython route, do you
have any suggestions for the javaservlet route?
thanks
k
Paul Boddie
>
> Firstly, (too long delayed) thanks for this latest release. We are
> having a lot of fun using it for our application. We have a mixed
> environment with one component using jython/JavaServlet and the rest
> using ModPython.
Interesting stuff!
> I have some small comments, probably the only one worth mentioning is
> that it would be nice to be able to add pythonpath entries on the
> commandline when using webstack_xxxx_build.py. Its not so important
> for the apache route, because you can just edit the webapp config file
> one time, but for the javaservlet route which regenerates the web.xml
> each time.
I have to think a bit more about deploying Java-based applications.
The webstack_java_build.py script is very monolithic and just gathers
everything together, which works reasonably well for the examples but
probably isn't a scalable solution. Of course, in the Java world,
everyone I knew used ant for deploying applications, but I don't have
any inclination to revisit that particular tool. Ideas for
improvements or alternatives would be most welcome!
> I have been investigating some headaches we have with
> mutlipart/form-data request containing an embedded zip file. The
> ModPython route handles this OK, except I think Filecontent should be
> wrapping the file object rather than the content. Otherwise we will
> lose all our RAM :-(.
Yes, I think this needs improving, too. The cgi.FieldStorage class
uses a temporary file, and that class is the basis of WebStack's
support for a number of frameworks (and mod_python provides a
compatible implementation), so some kind of a solution for WebStack
might be to not "materialise" the content of the file upload (ie. read
the contents into memory) but to make the FileContent objects a bit
smarter, as you say. I've made a few edits to my sources, and I'll try
and test them out tomorrow.
> The situation is much worse on the JavaServlet side where the call to
> get_fields is taking over four minutes for an upload of !2MB :(. It
> also seems to be leaking memory. I am trying to track it down, but
> if you have any ideas or can recreate it I would love to hear them.
I think that providing access to streams/files might help a lot. I
should also review my code to deal with multipart forms since it may
need fixing and/or extending. The Java Servlet API didn't provide
convenient support for multipart forms once upon a time, which was a
major complaint everyone had, if I remember correctly.
> We have used 'eric' successfully to debug the modpython route, do you
> have any suggestions for the javaservlet route?
I'm not much of a debugger person, I'm afraid. However, a few changes
from my side might help you without any need for a debugger. I'll try
to test and suggest some changes tomorrow.
Paul
Kevin Glynn
Paul Boddie writes:
>
> On 22 Okt, 21:33, Kevin Glynn <kevin.gl...@gmail.com> wrote:
> >
> > I have some small comments, probably the only one worth mentioning is
> > that it would be nice to be able to add pythonpath entries on the
> > commandline when using webstack_xxxx_build.py. Its not so important
> > for the apache route, because you can just edit the webapp config file
> > one time, but for the javaservlet route which regenerates the web.xml
> > each time.
>
> I have to think a bit more about deploying Java-based applications.
> The webstack_java_build.py script is very monolithic and just gathers
> everything together, which works reasonably well for the examples but
> probably isn't a scalable solution. Of course, in the Java world,
> everyone I knew used ant for deploying applications, but I don't have
> any inclination to revisit that particular tool. Ideas for
> improvements or alternatives would be most welcome!
>
Sorry, I think I only have criticism ;-).
> > I have been investigating some headaches we have with
> > mutlipart/form-data request containing an embedded zip file. The
> > ModPython route handles this OK, except I think Filecontent should be
> > wrapping the file object rather than the content. Otherwise we will
> > lose all our RAM :-(.
>
> Yes, I think this needs improving, too. The cgi.FieldStorage class
> uses a temporary file, and that class is the basis of WebStack's
> support for a number of frameworks (and mod_python provides a
> compatible implementation), so some kind of a solution for WebStack
> might be to not "materialise" the content of the file upload (ie. read
> the contents into memory) but to make the FileContent objects a bit
> smarter, as you say. I've made a few edits to my sources, and I'll try
> and test them out tomorrow.
>
The modPython stuff looks nice and efficient. For JavaServlets it
looks like this is a well-known solution:
I don't know how you can combine it with your code, given your
differing licenses ....
> > The situation is much worse on the JavaServlet side where the call to
> > get_fields is taking over four minutes for an upload of !2MB :(. It
> > also seems to be leaking memory. I am trying to track it down, but
> > if you have any ideas or can recreate it I would love to hear them.
>
> I think that providing access to streams/files might help a lot. I
> should also review my code to deal with multipart forms since it may
> need fixing and/or extending. The Java Servlet API didn't provide
> convenient support for multipart forms once upon a time, which was a
> major complaint everyone had, if I remember correctly.
>
> > We have used 'eric' successfully to debug the modpython route, do you
> > have any suggestions for the javaservlet route?
>
> I'm not much of a debugger person, I'm afraid. However, a few changes
> from my side might help you without any need for a debugger. I'll try
> to test and suggest some changes tomorrow.
>
No problems, it was really a profiler I wanted anyway. However a few
time stamped print statements later and I can see the problem ....
cheers
k
Paul Boddie
>
> Sorry, I think I only have criticism ;-).
Well, it's an area which needs more work, I'll admit that. :-)
[File uploads]
> The modPython stuff looks nice and efficient. For JavaServlets it
> looks like this is a well-known solution:
>
> http://www.servlets.com/cos/
>
> I don't know how you can combine it with your code, given your
> differing licenses ....
Yes, such classic 1996-vintage Java developer licensing that one gets
a real Netscape Communicator feeling just from reading the page. I
think I'd rather do the work to fix up the classes I have.
[...]
> No problems, it was really a profiler I wanted anyway. However a few
> time stamped print statements later and I can see the problem ....
I'll look into some improvements and get back to you.
Paul
Kevin Glynn
Paul Boddie writes:
>
> On 23 Okt, 01:00, Kevin Glynn <kevin.gl...@gmail.com> wrote:
> >
> > Sorry, I think I only have criticism ;-).
>
> Well, it's an area which needs more work, I'll admit that. :-)
>
[install scripts]
Sorry that I can't be more helpful. It is pretty ugly deploying the
java servlet, but its all hidden in a shell script so we don't suffer.
It might be nice if we could just copy new files over because jython
seems to take a hit from re-processing the jar files, but probably it
is actually just jython starting up and it wouldn't help ...
> [File uploads]
>
> > The modPython stuff looks nice and efficient. For JavaServlets it
> > looks like this is a well-known solution:
> >
> > http://www.servlets.com/cos/
> >
> > I don't know how you can combine it with your code, given your
> > differing licenses ....
>
> Yes, such classic 1996-vintage Java developer licensing that one gets
> a real Netscape Communicator feeling just from reading the page. I
> think I'd rather do the work to fix up the classes I have.
For our purposes (and only considering modpython and javaservlet) I
changed it so that file uploads are always written to disk and
FileContent stores the name of the local file, rather than its
contents.
For the java servlet I used the apache commons-fileupload/io jars
which have a much easier license to work with.
Now the file uploading for a 12MB file has gone from 4 minutes to a
small number of seconds :) (it is actually processed twice, once by a
modpython component and then by a javaservlet component)
If you want to see what I have done I can send more details / patches,
it works for us so far, but isn't pretty and I haven't considered any
of the other environments you support.
cheers
k
Paul Boddie
>
> [install scripts]
>
> Sorry that I can't be more helpful. It is pretty ugly deploying the
> java servlet, but its all hidden in a shell script so we don't suffer.
>
> It might be nice if we could just copy new files over because jython
> seems to take a hit from re-processing the jar files, but probably it
> is actually just jython starting up and it wouldn't help ...
I think a workable solution might be to have some script
(webstack_java_build.py, for example) used to set up the general
skeleton of a deployable servlet-based application, which is really
what it does now, and then provide a means to update the skeleton
without copying all the files around again. I know that others in the
Web frameworks scene have devised their own ways of deploying
applications, but I think that this is more of a general build/deploy
issue: it's like being able to run "make" and have only the modified
files processed.
[File uploads]
> For our purposes (and only considering modpython and javaservlet) I
> changed it so that file uploads are always written to disk and
> FileContent stores the name of the local file, rather than its
> contents.
Right.
> For the java servlet I used the apache commons-fileupload/io jars
> which have a much easier license to work with.
I might take a look at this. If we end up using the Apache commons
libraries and the licences aren't quite compatible with LGPLv2 or
later, WebStack will probably become LGPLv3 or later since the FSF
worked to achieve compatibility with the Apache licences in the GPLv3
effort.
> Now the file uploading for a 12MB file has gone from 4 minutes to a
> small number of seconds :) (it is actually processed twice, once by a
> modpython component and then by a javaservlet component)
Great! Generally it's probably best not to load large files into
memory, especially in Java environments which already use a lot of it.
> If you want to see what I have done I can send more details / patches,
> it works for us so far, but isn't pretty and I haven't considered any
> of the other environments you support.
I think that the other environments are mostly dealt with. I'm not
quite sure about Django's file upload support and whether it provides
the same temporary file tricks (or just loads everything into memory),
but as long as uploads are exposed as FileContent objects and there's
an easy way to read them, we shouldn't shy away from doing the
necessary things internally.
Paul