Service delivers ISO-8859-1 as UTF-8

[Martin Egri]

I am using Dropwizard 0.6.1, having upgraded from an earlier version.

My service was working until recently when it suddenly started delivering ISO-8859-1 encoded responses as UTF-8, which throws web browsers off guard. I'm not sure why this is, everything seems to be in order.

This is my resource:

@Path("/test")

@Consumes(MediaType.APPLICATION_JSON + "; charset=utf-8")

@Produces(MediaType.APPLICATION_XHTML_XML + "; charset=utf-8")

public class TestResource {

    @GET

    public PdfView get() {

        try {

            FrontpageModel mockModel = null; // new FrontpageModel();

            mockModel = fromJson(jsonFixture("assets/model.json"), FrontpageModel.class);

            return new PdfView(mockModel, layout);

        } catch (Exception e) {

            throw new RuntimeException("Couldn't generate view. Exception message: " + e.getMessage());

        }

    }

Both model.json and the ftl-template are neturally encoded (UTF-8 without BOM)

When trying to access the resource with Chrome-browser, I get 

This page contains the following errors:

error on line 40 at column 32: Encoding error

Below is a rendering of the page up to the first error.

It goes on to display everything until the first character that's supposed to be multi byte but for some reason isn't.

[Nick Telford]

Can you make the request with cURL (or equivalent tool) that displays the response headers to take the browser out of the equation?

It sounds like your content is in UTF-8 but your browser is trying to display it as ISO-8859-1? Your initial description suggests the reverse of this, which should work fine as ISO-8859-1 is a strict subset of UTF-8.

Regards,

Nick Telford

[Martin Egri]

The topic may be a bit confusing. What I mean to say is that I want DW to deliver a response in UTF-8, but generates it in ISO-8859-1.

curl confirms this:

HTTP/1.1 200 OK

Date: Tue, 15 Jan 2013 12:53:11 GMT

Content-Type: application/xhtml+xml; charset=utf-8

In ISO-8859-1, the letter 'ü' is encoded with one byte, while it takes 2 in UTF-8. So when my web browser encounters these single-byte encoded letters it tries to parse them as one single multi-byte character but fails. Since the page is being delivered as XHTML+XML, this is a critical error for the XML parser and so it stops (if I pass it as plain HTML I get a full page with garbled characters)

The content type is what I set it to be with the @Produces-annotation, but the response is still in ISO-8859-1. I traced the issue to FreemarkerViewRenderer's render-method, which passes a Locale to the template object which in turn grabs an encoding from a map, the result being ISO-8859-1.

I could work around the issue by replacing the renderer with a subclass that explicitly passes an encoding to Freemarker's configuration. I'll do this but I'm surprised no one else has encountered the same issue.

The view renders properly if I change this encoding to UTF-8 during runtime with a debugger.

[Martin Egri]

I did some more digging around and found the culprit!

A change that was recently added to FreemarkerViewRenderer, namely the line:

configuration.loadBuiltInEncodingMap();

Prior to this, Freemarker's encodingMap was left uninitialized which made it return a default charset initialized to the system property "file.encoding".

[Adam Jordens]

Should be set an appropriate charset on your @Produces annotation.

I was the one who submitted the loadBuiltInEncodingMap() and we've successfully toggled back & forth between UTF8 and ISO8859-1 for testing & production without issue now.

Cheers,

Adam

[Mar]

I am using a @Produces annotation with a charset. This is my freshly deployed test service:

@Path("/hello-world")

@Produces(MediaType.APPLICATION_XHTML_XML + ";charset=utf-8")

public class MyResource {

    @GET

    @Timed

    public MyView sayHello() {

        return new MyView("Testing encoding: åäöÅÄÖ");

    }

}

This yields the following in-browser error:

This page contains the following errors:

error on line 9 at column 4: Encoding error

Below is a rendering of the page up to the first error.

Changing mediatype to TEXT_HTML+";charset=utf-8" gives

Hello, Testing encoding: ������

However, if I debug and clear the encodingMap, everything is good again (since Freemarker is using my default encoding, which is utf-8)

My template is of course encoded in utf-8

[Mar]

(actually, freemarker is using configuration.setDefaultEncoding(Charsets.UTF_8.name());)

I'll see if I can whip together something.

[Adam Jordens]

I went back and reviewed our testing and configuration around this.

Our use-case was actually more around reading UTF-8 templates but injecting locale-specific data into them.  

What we might need to do in this case, is to update FreemarkerViewRenderer.render() to do something like {

    try {

        final Configuration configuration = configurationCache.getUnchecked(view.getClass());

        final String targetEncoding = view.getEncoding() != null ? view.getEncoding() : configuration.getEncoding(locale);

        final Template template = view.getEncoding() != null ? view.configuration.getTemplate(view.getTemplateName(), locale, targetEncoding);

        template.process(view, new OutputStreamWriter(output, template.getEncoding()));

    } catch (TemplateException e) {

        throw new ContainerException(e);

    }

}

(I haven't tried this code yet, but from looking at the stack, it should override the defaults)

To test whether passing the encoding does the trick, you can create:

src/main/resources/META-INF/services/com.yammer.dropwizard.views.ViewRenderer w/ the contents:

com.x.y.MyViewRenderer

MyViewRenderer is a clone of FreemarkerViewRenderer with the requisite changes.  

I did something very similar to this a couple releases back when I needed to roll out the modified view renderer without it being in dropwizard core yet. Of course locally you could just compile against a SNAPSHOT and give it a quick test.

Cheers,

Adam

[Adam Jordens]

final Template template = view.getEncoding() != null ? view.configuration.getTemplate(view.getTemplateName(), locale, targetEncoding);

My bad. Remove that extraneous view.getEncoding() != null bit, that's what happens when u refactor pseudo code mid reply.

Cheers.

[Adam Jordens]

https://github.com/ajordens/dropwizard/commit/1d846c4356a5c59e653c98c858e900cc33b7f232 is sort of what I had in mind for configurable view encodings.

I still to create an example project off 0.6.2-SNAPSHOT and test it out with actual mustache and freemarker templates.

Cheers,

Adam

[Martin Egri]

I got around to tidying up some code in my project and noticed you had committed a patch for dealing with encoding from the resource-end of things.

Everything seems to be working well with the 0.6.2 release. Thanks :)