[play 2.6] charset removed with application/json content-type

[Yann Simon]

Hi,

When sending a request with the content type "application/json;charset=UTF-8", request.mediaType is "Some(application/json)"

When sending a request with the content type "text/plain;charset=UTF-8", request.mediaType is "Some(text/plain; charset=UTF-8)" (and the charset is then "UTF-8")

Is it something expected or a bug?

Yann

[Greg Methvin]

Hi Yann,

The Akka HTTP server actually tries to parse the Content-Type intelligently. Since application/json does not define a charset parameter (https://www.iana.org/assignments/media-types/application/json), it omits that parameter when we render the Akka HTTP content type as a string. Play's JSON body parsers use Jackson to parse the bytes of the JSON, which detects the encoding according to the spec and does not look at the charset parameter (this was the case in 2.5 as well).

In your case, it should make no difference, since UTF-8 is a valid charset for JSON, and that will be autodetected by the parser.

So it's not a bug in the sense that it shouldn't affect spec-compliant recipients, but might affect certain non-spec-compliant users. There's some debate about it here: https://github.com/playframework/playframework/pull/7498.

Greg

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framework+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/CAFfBxfEiWFV8GSmpsxxy8PR6i_N5ywZVdgndJAv3o2LgBj_bmg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Greg Methvin

Tech Lead - Play Framework

Lightbend, Inc.

[Yann Simon]

Hi Greg, thanks for the answer.

But we can also parse the json payload as text (tolerant text for example). In that case, the payload is wrongly decoded as this parser does not default to utf-8 and cannot use the charset send by the client.

This is what happened to one of our application where we do not use the default json parser.

Yann

On Wed, Jun 28, 2017, 21:05 Greg Methvin <gr...@lightbend.com> wrote:

Hi Yann,

The Akka HTTP server actually tries to parse the Content-Type intelligently. Since application/json does not define a charset parameter (https://www.iana.org/assignments/media-types/application/json), it omits that parameter when we render the Akka HTTP content type as a string. Play's JSON body parsers use Jackson to parse the bytes of the JSON, which detects the encoding according to the spec and does not look at the charset parameter (this was the case in 2.5 as well).

In your case, it should make no difference, since UTF-8 is a valid charset for JSON, and that will be autodetected by the parser.

So it's not a bug in the sense that it shouldn't affect spec-compliant recipients, but might affect certain non-spec-compliant users. There's some debate about it here: https://github.com/playframework/playframework/pull/7498.

Greg

On Wed, Jun 28, 2017 at 9:06 AM, Yann Simon <yann.s...@gmail.com> wrote:

Hi,

When sending a request with the content type "application/json;charset=UTF-8", request.mediaType is "Some(application/json)"

When sending a request with the content type "text/plain;charset=UTF-8", request.mediaType is "Some(text/plain; charset=UTF-8)" (and the charset is then "UTF-8")

Is it something expected or a bug?

Yann

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/CAFfBxfEiWFV8GSmpsxxy8PR6i_N5ywZVdgndJAv3o2LgBj_bmg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Greg Methvin

Tech Lead - Play Framework

Lightbend, Inc.

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/CAA%3D11HyUESPa7%2B2rHYYMcvMGyAOVq%2BC-LdcPZjnYa52z5Hqtew%40mail.gmail.com.

[Igmar Palsenberg]

 

But we can also parse the json payload as text (tolerant text for example). In that case, the payload is wrongly decoded as this parser does not default to utf-8 and cannot use the charset send by the client.

Officially, you can't. JSON is UTF-8, UTF-16, or UTF-32. Anything else is invalid. If it doesn't default to UTF-8, it's broken according to the RFC.

Igmar

[Greg Methvin]

Just like the other tolerant body parsers, since "text" parses "text/plain", "tolerantText" assumes "text/plain" and tries to parse using the same rules, without validating the content type. This means it happens to work with other text content types that use ISO-8859-1 as the default. It's not meant to parse application/json, application/xml, etc.

The issue of using tolerantText on JSON has also been discussed in the past: https://github.com/playframework/playframework/issues/5334

--

You received this message because you are subscribed to the Google Groups "Play Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to play-framework+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/2ae9ccd8-d7a2-4d09-9a68-8e95db81084b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Yann Simon]

thanks for all the information.

It was very instructive.

To recap:

- Play 2.6.0 with akka-http removes the charset for application/json content type (as the parser should rely on the first bytes of the payload to detect the encoding)

- tolerantText then uses ISO-8859-1 as default, as the Charset data send by the client was removed.

As far as I understand, it does not make any sense to send a charset when sending a application/json payload.

Thanks for the quick responses.

Yann

Le jeu. 29 juin 2017 à 00:31, Greg Methvin <gr...@lightbend.com> a écrit :

Just like the other tolerant body parsers, since "text" parses "text/plain", "tolerantText" assumes "text/plain" and tries to parse using the same rules, without validating the content type. This means it happens to work with other text content types that use ISO-8859-1 as the default. It's not meant to parse application/json, application/xml, etc.

The issue of using tolerantText on JSON has also been discussed in the past: https://github.com/playframework/playframework/issues/5334

On Wed, Jun 28, 2017 at 2:29 PM, Igmar Palsenberg <ig...@palsenberg.com> wrote:

 

But we can also parse the json payload as text (tolerant text for example). In that case, the payload is wrongly decoded as this parser does not default to utf-8 and cannot use the charset send by the client.

Officially, you can't. JSON is UTF-8, UTF-16, or UTF-32. Anything else is invalid. If it doesn't default to UTF-8, it's broken according to the RFC.

Igmar

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/2ae9ccd8-d7a2-4d09-9a68-8e95db81084b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Greg Methvin

Tech Lead - Play Framework

Lightbend, Inc.

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/CAA%3D11HzpC6_67Bk-%3D4yXPr%3D%2BZ_PmWKsMn8dp%3DiGmxQw6AbGtaQ%40mail.gmail.com.

[malli...@gmail.com]

Hi,

Is it possible to force a Content-Type of "application/json; charset=utf-8" in play 2.6.x with akka-http? I am not interested in RFCs, specs, or whether it makes sense or not. I want to serve JSON with that Content-Type.

Thanks!

Michael