FreeMarker template encoding not working right

2,141 views
Skip to first unread message

Ramon Sistermans

unread,
Aug 25, 2015, 6:14:51 AM8/25/15
to Hippo Community
Hey all,

This might be a very simple issue but I'm not sure why I can't solve it.

I want to use UTF-8 encoding in my FTL templates but characters like é and è are showing as Ã© and è

- My browser meta charset is set properly and the response header shows this. I can use these characters in my content but not when using plain text in templates.
- I have checked my editor (IntelliJ) and verified it's saving files as UTF-8.
- In my web.xml I have the default:
<param-name>ContentType</param-name>
      <param-value>text/html; charset=UTF-8</param-value>
- I have tried overriding the charset with <#ftl encoding="UTF-8">

Unfortunately nothing works. Should I expect to escape all text I use in my FTL templates? What am I missing here?

Thanks!
Ramon

Jeroen Reijn

unread,
Aug 25, 2015, 6:54:47 AM8/25/15
to hippo-c...@googlegroups.com
Hi Ramon,

In which case doesn't it work?

- Showing normal content from a document?
- Show the text after a query string?

This should work locally without any extra effort.

Jeroen


--
Hippo Community Group: The place for all discussions and announcements about Hippo CMS (and HST, repository etc. etc.)
 
To post to this group, send email to hippo-c...@googlegroups.com
RSS: https://groups.google.com/group/hippo-community/feed/rss_v2_0_msgs.xml?num=50
---
You received this message because you are subscribed to the Google Groups "Hippo Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hippo-communi...@googlegroups.com.
Visit this group at http://groups.google.com/group/hippo-community.
For more options, visit https://groups.google.com/d/optout.



--
Jeroen Reijn
Hippo

Amsterdam office - Oosteinde 11, 1017 WT Amsterdam
Boston office - 745 Atlantic Ave, Eight Floor, Boston MA 02111, United states of America.

US +1 877 414 4776 (toll free)
Europe +31(0)20 522 4466
www.onehippo.com

http://blog.jeroenreijn.com | @jreijn | http://about.me/jeroenreijn

Woonsan Ko

unread,
Aug 25, 2015, 7:58:12 AM8/25/15
to hippo-c...@googlegroups.com
Hi Ramon,

Can you try setting *input encoding* [1] in your .ftl file?

<#ftl encoding="utf-8">

Regards,

Woonsan

[1] http://freemarker.org/docs/pgui_misc_charset.html
> --
> Hippo Community Group: The place for all discussions and announcements
> about Hippo CMS (and HST, repository etc. etc.)
>
> To post to this group, send email to hippo-c...@googlegroups.com
> RSS:
> https://groups.google.com/group/hippo-community/feed/rss_v2_0_msgs.xml?num=50
> ---
> You received this message because you are subscribed to the Google
> Groups "Hippo Community" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to hippo-communi...@googlegroups.com
> <mailto:hippo-communi...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/hippo-community.
> For more options, visit https://groups.google.com/d/optout.


--
w....@onehippo.com www.onehippo.com
Boston - 745 Atlantic Ave, 8th Floor, Boston MA 02111
Amsterdam - Oosteinde 11, 1017 WT Amsterdam

Ramon Sistermans

unread,
Aug 27, 2015, 7:47:59 AM8/27/15
to hippo-c...@googlegroups.com
Hi,

Jeroen solved this problem for us. :) 

The Freemarker servlet parses the .ftl using the platform's default Java encoding which for me (and most Windows users) is Cp1282.

What worked was changing the file encoding system property in the root pom by adding it to the cargo-maven2-plugin systemProperties: <file.encoding>UTF-8</file.encoding>. Change to this setting can be seen in cms > Admin > System properties.

Thanks!
Ramon

Ramon Sistermans
Consultant online

e:  ramon.si...@incentro.com
t:  +31204090444 
m:  +31615886436
w:  www.incentro.com

amsterdam office | mediahaven
moermanskkade 113  |  1013 bc   |  amsterdam

incentro

incentro news


To unsubscribe from this group and stop receiving emails from it, send an email to hippo-communi...@googlegroups.com.

Ard Schrijvers

unread,
Sep 3, 2015, 10:50:54 AM9/3/15
to hippo-c...@googlegroups.com
Hey Ramon and Reijn,

apologies for my late response. Seem like you hit a nasty issue. Nice
catch Reijn. Is there already a JIRA issue for this because we need to
improve this (perhaps by just having <file.encoding> by default in our
mvn profile like you did) however I'd also like to know what happens
if on a windows platform an ftl file gets pushed to the repository.
The fix you had implies I think that it gets stored in the repository
UTF-8 encoded, but I have no clue how this gets done for ftl files (by
jackrabbit filevault)

Regards Ard

On Thu, Aug 27, 2015 at 1:47 PM, Ramon Sistermans
<ramon.si...@incentro.com> wrote:
> Hi,
>
> Jeroen solved this problem for us. :)
>
> The Freemarker servlet parses the .ftl using the platform's default Java
> encoding which for me (and most Windows users) is Cp1282.
>
> What worked was changing the file encoding system property in the root pom
> by adding it to the cargo-maven2-plugin systemProperties:
> <file.encoding>UTF-8</file.encoding>. Change to this setting can be seen in
> cms > Admin > System properties.
>
> Thanks!
> Ramon
>
> Ramon Sistermans
> Consultant online
>
> e: ramon.si...@incentro.com
> t: +31204090444
> m: +31615886436
> w: www.incentro.com
>
> amsterdam office | mediahaven
> moermanskkade 113 | 1013 bc | amsterdam
>
>
--
Hippo Netherlands, Oosteinde 11, 1017 WT Amsterdam, Netherlands
Hippo USA, Inc.- 745 Atlantic Ave, Eight Floor, Boston MA 02111,
United states of America.

US +1 877 414 4776 (toll free)
Europe +31(0)20 522 4466
www.onehippo.com

Jeroen Reijn

unread,
Sep 4, 2015, 3:13:39 AM9/4/15
to hippo-c...@googlegroups.com
Hi Ard,

we've not created an issue yet. What we did to figure this out was the following:

The FTL was in UTF-8 encoding on Ramons filesystem. Both IntelliJ and any other editor showed this. We had to make the assumption that the synchronization was keeping the correct encoding. 

We noticed that the jcr:content node did contain a jcr:mimeType property (set to application/octet-stream), but is lacking the jcr:encoding property.

Now because it did work on my machine we assumed that it wasn't in the code that synchronized this, but it had to be somewhere else.

Setting the file.encoding was a hunch, but later I tracked it down to:

org.hippoecm.hst.freemarker.jcr.TemplateLoadingCache#createRepositorySourceFromBinary

Where the inputstream of the file is copied to a String. We use 

String template = IOUtils.toString(stream); 

Now looking at the Javadoc explains what happens:

Get the contents of an <code>InputStream</code> as a String
     * using the default character encoding of the platform.

So on windows that's not UTF-8 hence it broke.

I'll create a JIRA issue for it. Where do you think it would suite best? REPO, HST or archetype?

Cheers,

Jeroen

Jeroen Reijn
Hippo

Amsterdam office - Oosteinde 11, 1017 WT Amsterdam
Boston office - 745 Atlantic Ave, Eight Floor, Boston MA 02111, United states of America.

hippo_connect_600w_amsterdam_boston.png

Ard Schrijvers

unread,
Sep 4, 2015, 3:28:33 AM9/4/15
to hippo-c...@googlegroups.com
Hey Jeroen,

On Fri, Sep 4, 2015 at 9:13 AM, Jeroen Reijn <j.r...@onehippo.com> wrote:
>
> Hi Ard,
>
> we've not created an issue yet. What we did to figure this out was the following:
>
> The FTL was in UTF-8 encoding on Ramons filesystem. Both IntelliJ and any other editor showed this. We had to make the assumption that the synchronization was keeping the correct encoding.
>
> We noticed that the jcr:content node did contain a jcr:mimeType property (set to application/octet-stream), but is lacking the jcr:encoding property.
>
> Now because it did work on my machine we assumed that it wasn't in the code that synchronized this, but it had to be somewhere else.
>
> Setting the file.encoding was a hunch, but later I tracked it down to:
>
> org.hippoecm.hst.freemarker.jcr.TemplateLoadingCache#createRepositorySourceFromBinary
>
> Where the inputstream of the file is copied to a String. We use
>
> String template = IOUtils.toString(stream);
>
> Now looking at the Javadoc explains what happens:
>
> Get the contents of an <code>InputStream</code> as a String
> * using the default character encoding of the platform.
>
> So on windows that's not UTF-8 hence it broke.
>
> I'll create a JIRA issue for it. Where do you think it would suite best? REPO, HST or archetype?

Thank you so much for tracing this all the way down, perfect! AFAICS
we only need to change

String template = IOUtils.toString(stream);

into

String template = IOUtils.toString(stream, "UTF-8");


Great job! You can create an HST issue for it. If you like you can fix
it yourself, otherwise I'll pick it up.

Regards Ard

Ard Schrijvers

unread,
Sep 7, 2015, 2:00:22 PM9/7/15
to hippo-c...@googlegroups.com
FYI : I fixed the issue according Reijn's findings in
https://issues.onehippo.com/browse/HSTTWO-3393 and backported it to
HST 3.0.1 (CMS 10.0.1). Thanks for spotting and tracing down

Regards Ard

On Fri, Sep 4, 2015 at 9:28 AM, Ard Schrijvers

Ate Douma

unread,
Sep 7, 2015, 2:29:57 PM9/7/15
to hippo-c...@googlegroups.com
On Mon, Sep 7, 2015 at 8:00 PM, Ard Schrijvers <a.schr...@onehippo.com> wrote:
FYI : I fixed the issue according Reijn's findings in
https://issues.onehippo.com/browse/HSTTWO-3393 and backported it to
HST 3.0.1  (CMS 10.0.1). Thanks for spotting and tracing down

I think you mean CMS 10.0.3 (yet to be released). 



--
Hippo B.V. The Netherlands  Oosteinde 11, 1017 WT Amsterdam  +31 (0)20 522 4466
Hippo USA Inc.  745 Atlantic Ave, Eigth Floor, Boston MA 02111  +1 877 414 4776
_______________________________________________________________________________
This e-mail may be privileged and/or confidential, and the sender does not
waive any related rights and obligations. Any distribution, use or copying of
this e-mail or the information it contains by other than an intended recipient
is unauthorized. If you received this e-mail in error, please advise me (by
return e-mail or otherwise) immediately.

Ard Schrijvers

unread,
Sep 7, 2015, 3:24:25 PM9/7/15
to hippo-c...@googlegroups.com


On Sep 7, 2015 8:29 PM, "Ate Douma" <a.d...@onehippo.com> wrote:
>
>
>
> On Mon, Sep 7, 2015 at 8:00 PM, Ard Schrijvers <a.schr...@onehippo.com> wrote:
>>
>> FYI : I fixed the issue according Reijn's findings in
>> https://issues.onehippo.com/browse/HSTTWO-3393 and backported it to
>> HST 3.0.1  (CMS 10.0.1). Thanks for spotting and tracing down
>
>
> I think you mean CMS 10.0.3 (yet to be released). 

Ah apologies, indeed. I confused the HST 3.0.1 to think the next cms patch version to be 10.0.1 but GA was already 10.0.2 of course. Thanks for spotting

Reply all
Reply to author
Forward
0 new messages