Blobstore upload forms and character encoding

Showing 1-10 of 10 messages
Blobstore upload forms and character encoding Guillaume Laforge 9/14/10 1:59 AM
Hi all,

I'm wondering if someone has faced the following.

I have a form for uploading a picture to be stored in the blobstore,
with a file input field.
But I also have text fields, so users can give a title to the picture
they're uploading.

My form is something along the lines of:

<form id="upload-form" action="/_ah/upload/xxxxxxxxx"
        method="post" enctype="multipart/form-data" accept-charset="UTF-8">

I've even specified the accept-charset to be UTF-8, to deal with
things like accentuated characters, but it seems there's an encoding
problem somewhere, as the characters I get are interpreted as iso
latin (as far as I can see).

However, everything is configured for UTF-8 everywhere:
- my html file is encoded in UTF-8
- it specifies a meta attribute: <meta http-equiv="content-type"
content="text/html; charset=utf-8" />
- jetty does send the page encoded as UTF-8

But... the input fields I get are showing weird characters as soon as
non-ASCII characters are used in my input field.

Has anybody faced a problem like this when uploading files to the
blobstore with accompanying text fields?

Thanks in advance for your lights on this.

--
Guillaume Laforge
Groovy Project Manager
Head of Groovy Development at SpringSource
http://www.springsource.com/g2one

Re: Blobstore upload forms and character encoding Guillaume Laforge 9/14/10 2:17 AM
Just wanted to add that I have a workaround for the problem, as I'm doing:
new String(titleParam.getBytes(('ISO-8859-1')), 'UTF-8')

But it is *just* a workaround, and I'd much rather have a clean solution :-)

Guillaume

Re: [appengine-java] Re: Blobstore upload forms and character encoding Pieter Coucke 9/14/10 2:24 AM
This is something I have problems with every time I start a new project.

Maybe this helps (not sure since you say Jetty already sends it correctly): http://www.onthoo.com/blog/programming/2005/07/characterencodingfilter.html

I also have
<%@ tag pageEncoding="UTF-8"%>
and
<%@ page pageEncoding="UTF-8" contentType="text/html;charset=UTF-8"%>
in every tag and page

And make sure your pages (jsp, tags) are also encoded in UTF-8.

Hope this helps!
Re: [appengine-java] Re: Blobstore upload forms and character encoding Guillaume Laforge 9/14/10 2:40 AM
Hi Pieter,

Thanks a lot for the pointer!
Using a specific filter for that purpose seems like a good idea.

As you mentioned it, I've also added the <%@ page pageEncoding="UTF-8"
contentType="text/html;charset=UTF-8"%> directive too, but it didn't
change anything.

I've not tried the filter idea yet, as I had a workaround.
But I've got the feeling I'm missing something somewhere.
All the knobs I could think of seem to indicate UTF-8 (even what I see
through the response Jetty sends), so this is really weird.

Perhaps it could be a problem with App Engine's /_ah/upload servlet
which does something weird with the fields before returning them to my
own controller.

Guillaume

--

Guillaume Laforge
Groovy Project Manager
Head of Groovy Development at SpringSource
http://www.springsource.com/g2one

Re: [appengine-java] Re: Blobstore upload forms and character encoding Guillaume Laforge 9/26/10 7:02 AM
Hi all,

Coming back to this issue...

Interestingly... it illustrates yet another difference between the dev mode and deployed mode.

On the local development server, I have this encoding issue, that when I upload a form containing a file for the blobstore and a text field, the field is wrongly encoded, so I have to use a hack to decode as ISO-8951 and re-encode as UTF-8.

BUT when I deploy my application on Google App Engine, the problem isn't present there, and my hack wrongly decode/re-encode the text fields.

So locally, I need my hack, and remotely, I don't.

I'm going to have to do some local branching like if (localMode) { ... } else { ... }

This would be good if we could remove such differences between local and prod environments.

Guillaume
Re: [appengine-java] Re: Blobstore upload forms and character encoding Pieter Coucke 9/27/10 12:22 AM
Re: [appengine-java] Re: Blobstore upload forms and character encoding Guillaume Laforge 9/27/10 1:46 AM
Hi Pieter,

Although happening in a different context, this seems to be a similar issue:

1) there is a difference between the dev server environment, and the production environment

2) it's again a problem with encoding / decoding parameters

This would be great if Google could investigate that one :-)
These little differences and issues are quite annoying and can be quite time consuming when trying to figure out what's the problem.

Thanks for the pointer, Pieter.

Guillaume

On Mon, Sep 27, 2010 at 09:22, Pieter Coucke <pieter...@onthoo.com> wrote:
Maybe this is related to issue 2097, I ended up using the same fix in Deferred.java as described here:
http://code.google.com/p/googleappengine/issues/detail?can=2&q=2097&colspec=ID%20Type%20Status%20Priority%20Stars%20Owner%20Summary%20Log%20Component&id=2097#c13


--
Guillaume Laforge
Groovy Project Manager
Head of Groovy Development at SpringSource
http://www.springsource.com/g2one
Re: [appengine-java] Re: Blobstore upload forms and character encoding Mikhail Zemlyanukha 9/29/10 12:18 AM
Well, Google could open source at least their implementations of local
dev servers...

Thanks guys for sharing this, I new with GAE and surely would face the
this issue, since I also started using Blobstore.

Regards,
Mike

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to google-app...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine-java+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>

Re: [appengine-java] Re: Blobstore upload forms and character encoding Guillaume Laforge 9/29/10 12:48 AM
Yup, that would be handy to be able to look at the sources of the local dev server.
I mean, even if it's not "open source" in the sense we can fork it or anything, but just being able to view the sources may help sometimes understand the differences between dev and prod.
Re: [appengine-java] Re: Blobstore upload forms and character encoding Don Schwarz 11/12/10 9:46 AM
Hi Guillaume,

Is it possible for you to use WireShark or a logging proxy to capture
the actual HTTP session?

When parsing the multipart/form-data response from the browser we look
at the charset attribute on each individual mime part, and if it is
missing we assume ISO-8859-1.  I suspect that's biting you here, but
it's not clear to me where we should be getting the fallback charset
if the browser isn't setting it on each part.

As for seeing the relevant code, we have a mechanism in place to begin
open-sourcing parts of the SDK.  We will be doing this gradually over
several releases, but I will see if we can start with this code.

Thanks,
Don