editing models in admin fails on Windows

[Jacob Fenwick]

I started from scratch and installed the latest released Jython, Django and Django-Jython on Windows XP to start the sanity checks all over again.

Once I finished setting up an app similar to the pollsite app from the Jython book, I tried going into the admin to add a poll, and then edited that poll.

This worked find on the Django test server.

Then I tried deploying a war to Tomcat.

Once I was on Tomcat and tried to edit an existing poll, I got this error:

Environment:

Request Method: POST

Request URL: http://localhost:8080/foo/admin/bar/poll/1/

Django Version: 1.1.1

Python Version: 2.5.1

Installed Applications:

['django.contrib.auth',

 'django.contrib.contenttypes',

 'django.contrib.sessions',

 'django.contrib.sites',

 'foo.bar',

 'django.contrib.admin',

 'doj']

Installed Middleware:

('django.middleware.common.CommonMiddleware',

 'django.contrib.sessions.middleware.SessionMiddleware',

 'django.contrib.auth.middleware.AuthenticationMiddleware')

Traceback:

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\core\handlers\base.py" in get_response

  92.                 response = callback(request, *callback_args, **callback_kwargs)

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\contrib\admin\sites.py" in root

  490.                 return self.model_page(request, *url.split('/', 2))

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\views\decorators\cache.py" in _wrapped_view_func

  44.         response = view_func(request, *args, **kwargs)

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\contrib\admin\sites.py" in model_page

  509.         return admin_obj(request, rest_of_url)

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\contrib\admin\options.py" in __call__

  1098.             return self.change_view(request, unquote(url))

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\db\transaction.py" in _commit_on_success

  240.                 res = func(*args, **kw)

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\contrib\admin\options.py" in change_view

  825.                 formset = FormSet(request.POST, request.FILES,

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\forms\models.py" in __init__

  723.         super(BaseInlineFormSet, self).__init__(data, files, prefix=prefix,

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\forms\models.py" in __init__

  459.         super(BaseModelFormSet, self).__init__(**defaults)

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\forms\formsets.py" in __init__

  44.         self._construct_forms()

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\forms\formsets.py" in _construct_forms

  88.             self.forms.append(self._construct_form(i))

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\forms\models.py" in _construct_form

  737.         form = super(BaseInlineFormSet, self)._construct_form(i, **kwargs)

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\forms\models.py" in _construct_form

  475.             pk = self.data[pk_key]

File "C:\tomcat6\webapps\foo\WEB-INF\lib-python\django\utils\datastructures.py" in __getitem__

  203.             raise MultiValueDictKeyError, "Key %r not found in %r" % (key, self)

Exception Type: MultiValueDictKeyError at /admin/bar/poll/1/

Exception Value: "Key 'choice_set-0-id' not found in <QueryDict: {}>"

I tried this with both postgres and mysql and got the same result.

I've also tried adding a poll in the admin and it gives me back a blank form, which of course didn't submit because required fields were blank.

I haven't gotten far with debugging. All that I know so far is that the self.data in the BaseModelFormSet is a QueryDict, and it's blank, when there should be a bunch of data.

I'm going to keep digging and see if I can find what is wrong but I thought I would throw this out there in case someone has seen something like this before.

Jacob

[Josh Juneau]

Jacob-

I have also found this to be the case.  I am using the Oracle backend for my apps.  I've had similar experiences using both Tomcat and Glassfish...the admin does not work when deployed to a Java servlet container.  What's more, if you create editable views then they do not work either.  I'm currently deploying to the django built-in test server for my development and product testing in hopes that this issue is resolved so I can deploy to the Java Servlet containers soon.  I had made mention of this issue on the list a while back and nobody else had run into it before.

Could this have something to do with modjy since everything works okay with the django test server?  Leo, have you run into this issue before when deployed to Tomcat or Glassfish?

Josh Juneau
june...@gmail.com
http://jj-blogger.blogspot.com
http://www.jythonpodcast.com
Twitter ID:  javajuneau

--
You received this message because you are subscribed to the Google Groups "django-jython-dev" group.
To post to this group, send email to django-j...@googlegroups.com.
To unsubscribe from this group, send email to django-jython-...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-jython-dev?hl=en.

[Leo Soto M.]

On Thu, Jan 7, 2010 at 5:51 PM, Josh Juneau <june...@gmail.com> wrote:
> Jacob-
> I have also found this to be the case.

Also on Windows or on other OSs?

Deploying on top of Tomcat works fine for me on Mac OS X and Linux
(haven't tested on Linux the last month or so though) with Tomcat 6.

--
Leo Soto M.
http://blog.leosoto.com

[Josh Juneau]

I have only seen this problem on windows machines.  I use OS X for all of my development, but unfortunately we use Windows 2003 servers at my shop.  Everything (tomcat, glassfish) works fine on OS X, just not on Windows.

Best

Josh Juneau
june...@gmail.com
http://jj-blogger.blogspot.com
http://www.jythonpodcast.com
Twitter ID:  javajuneau

[Frank Wierzbicki]

This might be related: http://bugs.jython.org/issue1489

-Frank

[Leo Soto M.]

On Thu, Jan 7, 2010 at 6:57 PM, Josh Juneau <june...@gmail.com> wrote:
> I have only seen this problem on windows machines.  I use OS X for all of my
> development, but unfortunately we use Windows 2003 servers at my shop.
>  Everything (tomcat, glassfish) works fine on OS X, just not on Windows.

Crap, so it's a Windows-related issue (probably on Jython itself?).

I won't be able to look at it this weekend but hopefully next weekend
should be able to get a VM running to help with the problem. Any
progress you folks can do on the meantime would be really appreciated,
of course.

[Josh Juneau]

I've been meaning to poke around on this one a bit more anyways since I've got to deploy on Windows.  If I get a chance then I'll take a look at it this weekend.

Josh Juneau
june...@gmail.com
http://jj-blogger.blogspot.com
http://www.jythonpodcast.com
Twitter ID:  javajuneau

[Jacob Fenwick]

Josh, sorry I missed your older email, during the time I'm in school I become a sort of hermit who can't read long, technical emails.

The ticket Frank linked to says that the problem only occurs if you use Tomcat as a service. I tried Tomcat from the command line and I still have the issue. Also toward the end of the ticket Alan says a workaround is to move the jython.jar file from WEB-INF/lib to tomcat/lib. I have tried this and it has also not worked. My guess is that the issue is not related to this ticket, but I could be wrong.

I doubt I will be all that helpful with this issue since I know very little about the Jython internals, but I'm still going to poke around the code and see what I can do.

Jacob

On Thu, Jan 7, 2010 at 4:59 PM, Frank Wierzbicki <fwier...@gmail.com> wrote:

This might be related: http://bugs.jython.org/issue1489

-Frank

[Leo Soto M.]

On Thu, Jan 7, 2010 at 7:16 PM, Jacob Fenwick <jacob....@gmail.com> wrote:
> Josh, sorry I missed your older email, during the time I'm in school I
> become a sort of hermit who can't read long, technical emails.
> The ticket Frank linked to says that the problem only occurs if you use
> Tomcat as a service. I tried Tomcat from the command line and I still have
> the issue. Also toward the end of the ticket Alan says a workaround is to
> move the jython.jar file from WEB-INF/lib to tomcat/lib. I have tried this
> and it has also not worked. My guess is that the issue is not related to
> this ticket, but I could be wrong.

My 2 cents: I bet you are right. The bug linked by Frank tends to
manifest itself as a "maximum recursion depth" exception. I'd rather
continue debugging it from the Django side but sure it help to have
this other bug in mind in case we find something that looks like
related to it.

> I doubt I will be all that helpful with this issue since I know very little
> about the Jython internals, but I'm still going to poke around the code and
> see what I can do.

I can help with the Jython internals. Hopefully you will be able to
advance a lot (and probably find the actual bug) just poking at the
Python level.

[Jacob Fenwick]

I haven't found the deeper cause but I've traced back to WSGIHandler.

On test server:

request.POST in WSGIHandler:

<QueryDict: {}>

*****In models.py

[07/Jan/2010 21:29:11] "GET /admin/bar/poll/1/ HTTP/1.1" 200 9334

request.POST in WSGIHandler:

<QueryDict: {}>

[07/Jan/2010 21:29:11] "GET /admin/jsi18n/ HTTP/1.1" 200 803

request.POST in WSGIHandler:

<QueryDict: {u'choice_set-2-votes': [u''], u'choice_set-0-poll': [u'1'], u'choic

e_set-3-poll': [u'1'], u'choice_set-4-id': [u''], u'choice_set-3-id': [u''], u'c

hoice_set-2-choice': [u''], u'choice_set-4-poll': [u'1'], u'choice_set-INITIAL_F

ORMS': [u'2'], u'choice_set-0-id': [u'1'], u'choice_set-1-poll': [u'1'], u'choic

e_set-3-choice': [u''], u'choice_set-0-votes': [u'16'], u'pub_date_1': [u'12:42:

41'], u'choice_set-4-choice': [u''], u'choice_set-2-poll': [u'1'], u'choice_set-

0-choice': [u'Yes'], u'choice_set-1-id': [u'2'], u'choice_set-4-votes': [u''], u

'_save': [u'Save'], u'choice_set-3-votes': [u''], u'choice_set-1-votes': [u'1'],

 u'choice_set-1-choice': [u'No'], u'choice_set-TOTAL_FORMS': [u'5'], u'question'

: [u'Does this work?'], u'pub_date_0': [u'2010-01-12'], u'choice_set-2-id': [u''

]}>

*****In models.py

[07/Jan/2010 21:29:23] "POST /admin/bar/poll/1/ HTTP/1.1" 302 0

request.POST in WSGIHandler:

<QueryDict: {}>

[07/Jan/2010 21:29:29] "GET /admin/bar/poll/ HTTP/1.1" 200 3269

On Tomcat:

request.POST in WSGIHandler:

<QueryDict: {}>

*****In models.py

request.POST in WSGIHandler:

<QueryDict: {}>

request.POST in WSGIHandler:

<QueryDict: {}>

*****In models.py

The request.POST does not pass any data when the WSGIHandler is invoked from the servlet.

[Jacob Fenwick]

I've gone all the way to the root of modjy.py. When I grep for the service function in the modjy directory I don't find anything except for the function itself, so I assume that is the function the servlet container calls every time a request/response cycle is initiated.

The two parameters are req and resp. req is of type org.apache.catalina.connector.RequestFacade.

What I want is the request in a format where I can see that the POST data so I'll know it's at least getting that far. I'm guessing that the data I POST will show up in that req object, so then it's just a matter of tracking down which function between service calls on its way to call_application function that is causing the problem.

Of course, I could be wrong, and there won't be any POST data in there, in which case I have no idea where to go next.

So far I've been unsuccessful. I guessed that RequestFacade.getParameterNames() might give me what I want. Unfortunately it just tells me it's a java.util.Hashtable$EmptyEnumerator@5074de, and I don't know what to do with that object.

Does anyone have some insight into how to get the right data out of the RequestFacade or maybe the HashTable$EmptyEnumerator?

Jacob

[Jacob Fenwick]

Ok, this line in modjy_wsgi.py is revealing:

dict["wsgi.input"] = create_py_file(req.getInputStream())

The type of the wsgi.input is:

<open file '<Java InputStream 'org.apache.catalina.connector.CoyoteInputStream@121f956' as file>', mode 'r' at 0xec>

I'm stuck at this point as I don't know much about dealing with InputStreams in Jython.

If we assume there is data in the CoyoteInputStream, then I'm guessing there is either something wrong with FileUtil.wrap or PyFile as create_py_file can be either function. 

Jacob

[Josh Juneau]

Nice work Jacob.  I'll see if I dig into this part of the code a bit and figure out what is in the CoyoteInputStream.

Best

Josh Juneau
june...@gmail.com
http://jj-blogger.blogspot.com
http://www.jythonpodcast.com
Twitter ID:  javajuneau

[Alan Kennedy]

[Jacob]

> Ok, this line in modjy_wsgi.py is revealing:
> dict["wsgi.input"] = create_py_file(req.getInputStream())
> The type of the wsgi.input is:
> <open file '<Java InputStream
> 'org.apache.catalina.connector.CoyoteInputStream@121f956' as file>', mode
> 'r' at 0xec>
> I'm stuck at this point as I don't know much about dealing with InputStreams
> in Jython.
> If we assume there is data in the CoyoteInputStream, then I'm guessing there
> is either something wrong with FileUtil.wrap or PyFile as create_py_file can
> be either function.

Couple of things.

1. That definition of create_py_file (in modjy_wsgi.py) i.e.

try:
from org.python.core.util import FileUtil
create_py_file = FileUtil.wrap
except ImportError:
from org.python.core import PyFile
create_py_file = PyFile

was put in because PyFile creation changed in revision 5514

http://fisheye3.atlassian.com/changelog/jython/trunk/?cs=5514

Since FileUtil.wrap was introduced, the old method of creating a
PyFile from a java.io.InputStream, i.e. the
PyFile(java.io.InputStream) constructor, is no longer valid. So, since
revision 5514, the above definition should really be replaced by

create_py_file = FileUtil.wrap

The old definition given above was so that the same code would work
pre and post revision 5514.

Looking closer at the source for
org.apache.catalina.connector.CoyoteInputStream, I see the following
code in the read method.

if (SecurityUtil.isPackageProtectionEnabled()){
try{
Integer result =
(Integer)AccessController.doPrivileged(
new PrivilegedExceptionAction(){
public Object run() throws IOException{
Integer integer =
new Integer(ib.read(b, off, len));
return integer;
}
});
return result.intValue();

The definition of Security.isPackageProtectionEnabled depends (in
part) on the definition of two system properties, package.definition
and package.access. These are set in catalina.properties file in
$TOMCAT_HOME/conf. Their meaning can be read here

http://tomcat.apache.org/tomcat-6.0-doc/security-manager-howto.html

So, what I'd like for you to try is to set the package.definition and
package.access properties to null, and restarting your container. You
can do this by simply commenting out the two properties in the
catalina.properties file, like so

# package.access=sun.,org.apache.catalina.,org.apache.coyote.,org.apache.tomcat.,org.apache.jasper.,sun.beans.
# package.definition=sun.,java.,org.apache.catalina.,org.apache.coyote.,org.apache.tomcat.,org.apache.jasper.

If that changes the behaviour you're seeing, then we might be onto something.

Regards,

Alan.

[Jacob Fenwick]

Commented out the lines, restarted Tomcat, tried the edit operation again.

Same error.

Jacob

[Alan Kennedy]

[Jacob]

> Commented out the lines, restarted Tomcat, tried the edit operation again.
> Same error.

OK.

Another suggestion: try reading the java ServletInputStream directly,
by accessing it through environ['j2ee.request'].getInputStream()

The return value should be a ServletInputStream

http://java.sun.com/j2ee/1.4/docs/api/javax/servlet/ServletInputStream.html

Try using the methods of that directly to see if you get any data.

If you succeed in getting data from it, then we're looking at a jython
problem with wrapping ServletInputStream in a PyFile.

Alan.

[Jacob Fenwick]

environ['j2ee.request'].getInputStream():

org.apache.catalina.connector.CoyoteInputStream@daa9f1

It's true that CoyoteInputStream inherits from ServletInputStream, but this doesn't get me any farther than I already was since I don't know how to communicate with an InputStream in Jython.

I was trying to convert this Java code to Jython last night as I figured knowing how to do this would help:

import java.io.File;

import java.io.FileInputStream;

import java.io.BufferedInputStream;

import java.io.ByteArrayOutputStream;

import java.io.InputStream;

import java.io.IOException; 

class ReadInputStreamToString {

public static String readInputStreamAsString(InputStream in) 

throws IOException {

BufferedInputStream bis = new BufferedInputStream(in);

ByteArrayOutputStream buf = new ByteArrayOutputStream();

int result = bis.read();

while(result != -1) {

 byte b = (byte)result;

 buf.write(b);

 result = bis.read();

}        

return buf.toString();

}

public static void main(String[] args) throws Exception {

InputStream is = new FileInputStream(new File("foo.txt"));

String output = readInputStreamAsString(is);

System.out.println(output);

output = readInputStreamAsString(is);

System.out.println(output);

is.close();

}

}

This is where I got to in the conversion, but it still doesn't work:

from java.lang import System

from java.io import File

from java.io import FileInputStream

from java.io import InputStream

from java.io import BufferedInputStream

from java.io import ByteArrayOutputStream

from java.io import IOException

indata = new FileInputStream(new File("foo.txt"))

bis = new BufferedInputStream(indata)

buf = new ByteArrayOutputStream()

result = bis.read()

while result not equal -1:

byte b = (byte)result

buf.write(b)

result = bis.read()

System.out.println('POST data:')

System.out.println(buf.toString())

except IOException, iox:

System.out.println("Got IOException")

This code gives me the error:

  File "bar.jy", line 9

    indata = new FileInputStream(new File("foo.txt"))

                ^

SyntaxError: mismatched input 'FileInputStream' expecting NEWLINE

If someone can show me how to do something like that then maybe I can read from the InputStream.

I think if I even knew how to create a byte and a byte array in Jython then I could probably at least do a readLine.

Jacob

[Josh Juneau]

Jacob-

I've adjusted your code a bit and it now works.  Your code ad several unnecessary "new" statements, and some other Java mixed in.  I got the following to run...does this help you at all?

from java.lang import System

from java.io import File

from java.io import FileInputStream

from java.io import InputStream

from java.io import BufferedInputStream

from java.io import ByteArrayOutputStream

from java.io import IOException

indata = FileInputStream(File("foo.txt"))

bis = BufferedInputStream(indata)

buf = ByteArrayOutputStream()

result = bis.read()

while result != -1:

    try:

b = result

        buf.write(b)

        result = bis.read()

        print 'POST data:'

        print buf.toString()

    except IOException:

        print "Got IOException"

Thanks

Josh Juneau
june...@gmail.com
http://jj-blogger.blogspot.com
http://www.jythonpodcast.com
Twitter ID:  javajuneau

[Jacob Fenwick]

Yep, that helped.

I modified the code and stuck it in at the very top of service in modjy.py so I could print out req.getInputStream() and this is what I got:

POST data:

-----------------------------114782935826962

Content-Disposition: form-data; name="question"

Does this work?

-----------------------------114782935826962

Content-Disposition: form-data; name="pub_date_0"

2010-01-12

-----------------------------114782935826962

Content-Disposition: form-data; name="pub_date_1"

12:42:41

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-TOTAL_FORMS"

5

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-INITIAL_FORMS"

2

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-0-choice"

Yes

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-0-votes"

18

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-0-id"

1

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-0-poll"

1

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-1-choice"

No

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-1-votes"

1

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-1-id"

2

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-1-poll"

1

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-2-choice"

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-2-votes"

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-2-id"

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-2-poll"

1

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-3-choice"

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-3-votes"

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-3-id"

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-3-poll"

1

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-4-choice"

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-4-votes"

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-4-id"

-----------------------------114782935826962

Content-Disposition: form-data; name="choice_set-4-poll"

1

-----------------------------114782935826962

Content-Disposition: form-data; name="_save"

Save

-----------------------------114782935826962--

The POST data is at least getting into modjy.py.

I think this narrows it down to something between service and call_application in modjy.py.

Jacob

[Josh Juneau]

Glad this helped you, and thanks for working on this issue.

Josh Juneau
june...@gmail.com
http://jj-blogger.blogspot.com
http://www.jythonpodcast.com
Twitter ID:  javajuneau

[Jacob Fenwick]

Alan seemed to want to know about environ['j2ee.request'].getInputStream(). I was able to print the data out in call_application just before WSGIHandler is called with app_callable.__call__ and all the POST data showed up fine.

I also printed out the data from environ['wsgi.input'] from inside call_application and it also had the same data.

So now I'm not sure what to do. It seems that the data is there and it can be read using Java methods, but somehow the Python isn't able to read it.

I'm wondering if django.http.multipartparser is capable of reading from a file of type <open file '<Java InputStream 'org.apache.catalina.connector.CoyoteInputStream@db1c5' as file>', mode 'r' at 0xeb>. I would assume that's just a pointer to a location in memory but I'm not too clear on what an InputStream actually entails at a low level, and I'm not sure what MultiPartParser is expecting.

Jacob

[Alan Kennedy]

[Jacob]

> Alan seemed to want to know about environ['j2ee.request'].getInputStream().
> I was able to print the data out in call_application just before WSGIHandler
> is called with app_callable.__call__ and all the POST data showed up fine.
>
> I also printed out the data from environ['wsgi.input'] from inside
> call_application and it also had the same data.

OK, so both environ['wsgi.input'] and
environ['j2ee.request'].getInputStream() both return the same data.

So wsgi.input correctly provides the data, i.e. if you call
environ['wsgi.input'].read(), then wsgi.input is working correctly.

> So now I'm not sure what to do. It seems that the data is there and it can
> be read using Java methods, but somehow the Python isn't able to read it.

But that doesn't agree with what you've seen, except maybe if Django
is reading the post data in some peculiar way.

What code does Django use to to read wsgi.input?

Alan.

[Jacob Fenwick]

I'll briefly describe what I've found out about where the code goes when it leaves modjy.

The jumping off point from modjy.py to Django is in a method called call_application. This method calls another method called app_callable.__call__(environ, response_callable). Through some sort of magic, app_callable has been assigned earlier in the program in the get_app_object method to the WSGIHandler class in django.core.handlers.

One of the two objects passed to the WSGIHandler __call__ method is environ, which contains all the request information in the WSGI format. But Django has its own idea of what a request should look like, so environ is passed to another method named request_class, which is really the class WSGIRequest.

WSGIRequest is what is commonly called request in Django. It has an attribute we're very interested, the POST attribute. This is assigned using a property, since we want to access POST as an attribute, even those its getter and setter are both methods. The getter method is _get_post, which goes to _load_post_and_files, which decides whether it is multipart or not. In our case it is multipart, so we go to the method parse_file_upload. That method is stored in WSGIRequest's parent class, http.HttpRequest, found in django.http.__init__.

I'm pretty sure where it goes from here is django.http.multipartparser.MultiPartParser. According to the doc, it is:

A rfc2388 multipart/form-data parser.

``MultiValueDict.parse()`` reads the input stream in ``chunk_size`` chunks and returns a tuple of ``(MultiValueDict(POST), MultiValueDict(FILES))``. If

    """

http://code.djangoproject.com/browser/django/trunk/django/http/multipartparser.py

This is the code that ultimately processes the data stored in wsgi.input. I think the variable that should be the pointer to the file containing the data is called input_data.

MultiPartParser seems to really kick off once the parse function is called. This is a large piece of custom code built to parse and Input Stream. The entire file is about 700 lines and I've skimmed some of it. If I have some time later on I'll look into it more, otherwise maybe someone else can print out what is happening with input_data and stream to see where it's failing to read.

Jacob

[Jacob Fenwick]

I have tracked the problem down to one line:

In django.http.multipartparser.py:

http://code.djangoproject.com/browser/django/trunk/django/http/multipartparser.py

In the parse_boundary_stream function, this line fails:

header_end = chunk.find('\r\n\r\n')

This line incorrectly returns a -1 every time.

Jacob

[Jacob Fenwick]

Another tidbit, the type of the chunk object is:

class org.python.core.PyString

[Jacob Fenwick]

I compared the POST data being generated by Python and Jython at the place in the code that I pointed out. Indeed, the Python generated POST has the \r\n\r\n pattern. In the version generated in Jython it comes up as \n\n.

I went all the way back to modjy.py and checked the data in the req variable of the service method and this also has \n\n instead of \r\n\r\n.

I think there are three possible points that could be causing failure:

1. The Windows version of the servlet container is passing the wrong data

2. modjy.py is ignoring the \r characters

3. jython is ignoring the \r character

There's also a small possibility that there is some obscure line of Django code that is supposed to do add in those extra \r characters, but I think that is unlikely.

I would appreciate if someone who is more of an expert on jython or the servlet container took a look at this.

If anyone needs some of the code I used to print to a file I would be happy to share it with you.

Jacob

[Jacob Fenwick]

Sorry if this is way too many emails, but I decided to read some of RFC 2616 and found some interesting pieces:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1

"3.7.1 Canonicalization and Text Defaults

Internet media types are registered with a canonical form. An entity-body transferred via HTTP messages MUST be represented in the appropriate canonical form prior to its transmission except for "text" types, as defined in the next paragraph.

When in canonical form, media subtypes of the "text" type use CRLF as the text line break. HTTP relaxes this requirement and allows the transport of text media with plain CR or LF alone representing a line break when it is done consistently for an entire entity-body. HTTP applications MUST accept CRLF, bare CR, and bare LF as being representative of a line break in text media received via HTTP."

According to RFC 2616, it appears to be perfectly acceptable for a servlet container to send all LF's. I bet that's how it's laid out in JSR 53.

I think this means Django is not compliant with RFC 2616 since it will only accept CRLF.

Is this something I should file as a ticket with them?

Jacob

[Jacob Fenwick]

Nevermind, I already filed it. If they don't like it they can close it and not be standards compliant:

http://code.djangoproject.com/ticket/12578

Jacob

[Alan Kennedy]

[Jacob]

> I compared the POST data being generated by Python and Jython at the place
> in the code that I pointed out. Indeed, the Python generated POST has the
> \r\n\r\n pattern. In the version generated in Jython it comes up as \n\n.
> I went all the way back to modjy.py and checked the data in the req variable
> of the service method and this also has \n\n instead of \r\n\r\n.

Well done, Jacob, for narrowing this one down.

So this looks like a simple line-endings translation problem. So

> I think there are three possible points that could be causing failure:
> 1. The Windows version of the servlet container is passing the wrong data

I don't think this is it. I expect that if you look at the data you
read from the java ServletInputStream, the correct "\r\n\r\n" will be
there.

> 2. modjy.py is ignoring the \r characters

This is not the case, since modjy does no processing of the wsgi.input data.

> 3. jython is ignoring the \r character

I think this is the most likely.

> There's also a small possibility that there is some obscure line of Django
> code that is supposed to do add in those extra \r characters, but I think
> that is unlikely.

I also think this is unlikely.

> I think this means Django is not compliant with RFC 2616 since it will only accept CRLF.
> Is this something I should file as a ticket with them?

Strictly speaking, I think your interpretration of RFC 2616 is
correct, and if django is not accepting "\n\n" as a separator, then it
is *strictly* not compliant. This clause was put in the spec for
precisely the reason of varying line-ending representation between
windows( "\r\n"), *nix ("\n") and mac ("\r").

But I think that you'll find it extremely rare, in the wild, for
"\n\n" to appear as a separator in POST data: that's why the django
code has been working correctly in pretty much every scenario in
existence, except this one. I wouldn't be surprised if the django
folks reject your bug report.

I'll be looking into this one today: more later.

Alan.

[Jacob Fenwick]

On Mon, Jan 11, 2010 at 5:39 AM, Alan Kennedy <alan.kennedy.name@gmail.com> wrote:

> I think there are three possible points that could be causing failure:
> 1. The Windows version of the servlet container is passing the wrong data

I don't think this is it. I expect that if you look at the data you
read from the java ServletInputStream, the correct "\r\n\r\n" will be
there.

Is there an easy way we can test this? 

 

> 3. jython is ignoring the \r character

I think this is the most likely.

I agree. I would guess that the notorious variation of how Windows ends lines with CRLF and UNIX ends lines with LF is to blame, and Jython is suffering as a result.

I can't help but wonder if Microsoft was the one who squeezed the CRLF standard into the RFC since they were on the list of people who wrote it.

Where could we even begin looking in the Jython source to figure this out? I imagine this requires looking deep inside the Jython parser.

Is there code that branches in the parser based on Windows or UNIX? If not, could that be possible?

 

> I think this means Django is not compliant with RFC 2616 since it will only accept CRLF.
> Is this something I should file as a ticket with them?

I wouldn't be surprised if the django

folks reject your bug report.

I think that's probably what will happen, which is why I think fixing Jyhon is the best course. That is, if the servlet container is really communicating in CRLF (which it probably is).

Although I think we could probably also hack it at the modjy level. I imagine this could potentially be affecting any other Python framework trying to talk to Java over the WSGI gateway. I may try Pylons on Jython and see how it fairs.

Jacob

[Jacob Fenwick]

Oops, I missed this part:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.2

It looks like they are compliant. Now I think we definitely need to start digging into the parser to fix this, or write a platform dependent hack.

Jacob

[Leo Soto M.]

On Mon, Jan 11, 2010 at 11:35 AM, Jacob Fenwick <jacob....@gmail.com> wrote:
> Oops, I missed this part:
> http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.2
> It looks like they are compliant. Now I think we definitely need to start
> digging into the parser to fix this, or write a platform dependent hack.

If you mean the Python language parser, then not. This problem
shouldn't have anything to do with the parser.

But there may be something wrong when translating Java strings to
Python strings on Windows. Maybe you could try invoking a simple Java
method returning CRLF and seeing if Jython translates it as just LF on
the Python side?

Regards,

[Jacob Fenwick]

I tried this:

from java.lang import System

from java.io import File

from java.io import FileInputStream

from java.io import InputStream

from java.io import BufferedInputStream

from java.io import ByteArrayOutputStream

from java.io import IOException

from java.io import FileWriter

from java.lang import Integer

try:

    # file that contains CRLF characters

    indata = FileInputStream(File("out_python.txt"))

    bis = BufferedInputStream(indata)

    # output file

    buf = FileWriter(File("foo.txt"))

    result = bis.read()

    while result != -1:

        result_str = Integer.toString(result)

        result = Integer.parseInt(result_str)

        buf.write(result)

        result = bis.read()

    System.out.println(buf.toString())

    buf.close()

    bis.close()

except IOException:

    System.out.println("Got IOException")

Some of the data in the input file had CRLF data.

I figured if I convert to String, then convert back to int, the issue might come up.

But maybe not. I'm not sure if this is the right approach.

How can I make a String turn into a "Python string"?

Jacob

[Jacob Fenwick]

Nevermind, this is probably a more accurate test:

from java.lang import String

s = String("Hi\r\n")

print 'Before Carriage Return'

print ord(str(s.charAt(2)))

print 'After Carriage Return'

print 'Before Line Feed'

print ord(str(s.charAt(3)))

print 'After Line Feed'

output:

Before Carriage Return

13

After Carriage Return

Before Line Feed

10

After Line Feed

Hmmm, it sure looks like it's working.

So what the heck is going on?

Jacob

[Leo Soto M.]

Try something like this on a windows machine:

Create foo/Foo.java:

package foo;
public class Foo {
public static String crlf() {
return "\r\n";
}
}

Compile it:

> javac foo/Foo.java

and then fire up Jython (staying on the same directory) and invoke the
Java method:

>>> from foo import Foo
>>> Foo.crlf()
u'\r\n'

[Jacob Fenwick]

I got the correct result.

Jacob

[Jacob Fenwick]

I decided to try something else. I scanned the packets getting sent when I POST and they contain CRLF in the multipart portion. So I've verified the problem is happening after the packets are received by the servlet container.

Jacob

[Jacob Fenwick]

I'm did some more research into this. I decided to change my methods by using Python based printing and file IO instead of Java based and I've noticed the data comes out very differently.

I posted what I've learned here:

http://code.djangoproject.com/ticket/12578#comment:8

I don't know if that's the right way to go about it since the ticket is closed. I'm not sure if anyone will see it.

Any suggestions on how I can raise this issue correctly with Django?

Or does anyone have some insight into why I'm seeing what I'm seeing?

Jacob

[Jacob Fenwick]

kmtracy responded to my post on the ticket. This is the most useful idea I got out of it:

"Note I believe it is the reverse of this problem that is causing the failure with Jython/Windows.  Some code somewhere is reading the data as a text file, instead of a binary file, and the existing CRLFs are being turned into just plain LFs."

Inside Lib/modjy/modjy_wsgi.py I found this line:

dict["wsgi.input"] = create_py_file(req.getInputStream()

If you follow code it eventually goes to org.python.core.PyFile at this line:

PyFile(InputStream istream, int bufsize)

This constructor calls another constructor and passes "r" as the mode. I believe this could be where the problem is, as it may need to pass "rb" to read the stream as binary instead of text.

However, I cannot call this constructor directly as it is not a public method:

PyFile(InputStream istream, String name, String mode, int bufsize, boolean closefd)

So now I'm at a point where I need some help.

I checked out Jython 2.5.1 from the tags section of SVN. For a while, I've been able to compile it and use it. But now I'm getting a weird error when I try to load any page in Django on Tomcat:

javax.servlet.ServletException: Exception creating modjy servlet: ImportError: No module named site

com.xhaus.modjy.ModjyJServlet.init(ModjyJServlet.java:124)

javax.servlet.GenericServlet.init(GenericServlet.java:212)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)

org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)

org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)

java.lang.Thread.run(Unknown Source)

This error is preventing me from further testing since I have to be able to recompile PyFile to do my test.

So I'm hoping that either:

1. Someone can help me figure out what their weird runtime error about site is

or

2. Someone can do the test for me.

   a. Download the source for the Jython 2.5.1 release

   b. Make the constructor in PyFile public

   c. change the line in modjy_wsgi to something like this:

   dict["wsgi.input"] = PyFile(istream, "<Java InputStream 'xxx' as file>", "rb", -1, True )

   d. Test a Django POST on Tomcat and see if it works.

I certainly hope this is what's wrong with it, that would be great if it was that simple.

Jacob

[Alan Kennedy]

[Jacob]

> kmtracy responded to my post on the ticket. This is the most useful idea I
> got out of it:
> "Note I believe it is the reverse of this problem that is causing the
> failure with Jython/Windows.  Some code somewhere is reading the data as a
> text file, instead of a binary file, and the existing CRLFs are being turned
> into just plain LFs."
> Inside Lib/modjy/modjy_wsgi.py I found this line:
> dict["wsgi.input"] = create_py_file(req.getInputStream()
> If you follow code it eventually goes to org.python.core.PyFile at this
> line:
> PyFile(InputStream istream, int bufsize)
> This constructor calls another constructor and passes "r" as the mode. I
> believe this could be where the problem is, as it may need to pass "rb" to
> read the stream as binary instead of text.

Problem reported on the jython bug tracker.

Wrapping an InputStream with a PyFile wrongly carries out line-ending
translation.
http://bugs.jython.org/issue1549

Alan.

[Jacob Fenwick]

Thanks Alan.

Jacob

--

[Jacob Fenwick]

Just as an update, I added wrap method to FileUtil that has a mode, had it call the Jython method that lets you change the mode, and then I called this wrap method from modjy_wsgi.py and it did indeed fix the problem.

It's only a few lines of code to change, how can we get these changes into trunk?

Jacob

[Josh Juneau]

Great work Jacob!  Looking forward to the fix.  Sorry I haven't been able to help out with this one...

Best

Josh Juneau
june...@gmail.com
http://jj-blogger.blogspot.com
http://www.jythonpodcast.com
Twitter ID:  javajuneau

[Jacob Fenwick]

I have attached the svn diffs of the files I have changed.

Note 1: These diffs are again the version of Jython at https://jython.svn.sourceforge.net/svnroot/jython/tags/Release_2_5_1

This is because I cannot get Django-Jython to run on jython-trunk! I get this error:

javax.servlet.ServletException: Exception creating modjy servlet: ImportError: No module named site

com.xhaus.modjy.ModjyJServlet.init(ModjyJServlet.java:124)

Note 2: My solution is far from elegant, I think there should really be a one-to-one relationship between the constructors of PyFile and the wrappers in FileUtil. However, this does fix the immediate problem.

Jacob

[Alan Kennedy]

[Jacob]

> Just as an update, I added wrap method to FileUtil that has a mode, had it
> call the Jython method that lets you change the mode, and then I called this
> wrap method from modjy_wsgi.py and it did indeed fix the problem.

Glad to hear it.

> It's only a few lines of code to change, how can we get these changes into
> trunk?

Attach your patches to the bug I reported.

Regards,

Alan.

[Jacob Fenwick]

I added my patches to the bug report.

Jacob

Alan.

[Josh Juneau]

I've patched my Jython 2.5.1 with your code changes to PyFile.java, FileUtil.java, and modjy_wsgi.py and deployment to Windows 2003 server works as expected now.  Jacob, I also confirm that I was unable to patch the Jython trunk and compile.  I had to pull down the 2.5.1 branch in order to compile successfully with the patches.

Has anyone looked into including these patches in the trunk?  While patching the trunk and compiling I received errors with some jruby dependencies.  Including the latest jruby.jar did not do the trick.  I didn't investigate any further, but rather pulled down the 2.5.1 branch as Jacob suggested and it worked.  I can investigate further if nobody else has looked into this yet.  I think these patches should go in 2.5.2 if possible.

  
Josh Juneau
june...@gmail.com
http://jj-blogger.blogspot.com
http://www.jythonpodcast.com
Twitter ID:  javajuneau