Intent to Implement and Ship: Accept-Language Headers language expansion

[Claudio Magni]

Contact emails

claudi...@google.com, yyus...@google.com

Spec

See design doc.

The HTTP headers spec is here.

Summary

We want to change how Chrome generates the Accept-Language HTTP headers. As websites tend to only accept languages without region (i.e. “en” vs “en-AU”), the user could receive websites in an unexpected language. We plan to add the base language in the correct position so that users receive webpages in their preferred language.

Motivation

Some users have a bad user experience when they receive a website that is not in their preferred language, despite having set their language preferences correctly.

Interoperability and Compatibility Risk

Each browser has a different behavior regarding Accept-Language headers.

• Safari: only the top language is used for the header. The header always contains a single language and it does not fallback to the base language from a regional one. However the set of languages that can be sent in the header is limited and I wasn't able to figure out the exact logic that governs it. For example, English languages are sent as "en", while Spanish-Argentina is passed as "es-es".
  
• Firefox: the header matches exactly the language list in the user preferences. Nothing is added or moved, thus they do not perform the expansion that we are suggesting in this design.
  
• Edge: given that in the user preferences they do not allow a base language without at least a region associated to it, they add the base language after the last occurrence of any corresponding regional language. Out of all the browsers, their behavior is the closest to the one we want to implement, with just small differences in a few edge cases.
  
• Opera: did not test.
  

This page has a good summary of the different behaviors.

Our change would be mostly in line with Edge browser.

Ongoing technical constraints

None.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Is this feature fully tested by web-platform-tests?

No.

These tests do not apply since the spec is implemented differently across browsers. Refer to the Interoperability section above.

OWP launch tracking bug

http://crbug/737232

Feature dashboard

N/A

Requesting approval to ship?

Yes

Note:

I have previously sent a similar "Intent to Implement" email for this change. You can view the discussion here.

[Anne van Kesteren]

On Fri, Aug 25, 2017 at 7:04 AM, 'Claudio Magni' via blink-dev
<blin...@chromium.org> wrote:
> Each browser has a different behavior regarding Accept-Language headers.

Maybe file bugs on the other browsers to drive toward some
convergence? Maybe if everyone did the same thing sites would be more
likely to use this as opposed to IP sniffing. We could maybe also add
some more specific requirements in Fetch if the HTTP specification is
not specific enough.


--
https://annevankesteren.nl/

[Rick Byers]

Yeah, the web compat data you've accumulated in particular is probably priceless for helping the other engines decide on the best choice.  Can you at least please share your specific data (examples of sites that are broken today in Chrome) on a Firefox bug?  Given the same information it seems pretty likely that Firefox, Chrome and Edge would converge on the same design, which we could then spec somewhere and have web-platform-tests to prevent regression and assist other implementors.

Have you done any sort of compat analysis?  Do you expect some sites may break / regress as a result of this change?

--
https://annevankesteren.nl/

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADnb78iPmFJND%2Bf-hhqEK1HF4Eg02ac9Xdv-%2ByCcY6C0Uz%3Dcaw%40mail.gmail.com.

[Claudio Magni]

That sounds good, I have filed a bug on Firefox.

Should I provide more info about our intentions there? Maybe a link to our doc?

Regarding compat data, we are not aware of any website that would be broken by our proposal. Given the structure of the Accept-Language header, it's actually difficult to see how that would be possible.

We have not performed any large-scale analysis of websites to find out how many are currently broken. That would require running headless chrome in batch over a large portion of the web and it might not be worth it.

[Philip Jägenstedt]

To shed some light on the compat situation, can you share what the Accept-Language header is by default in various browser with different OS languages? I assume that only a tiny fraction of users change the language settings of the OS or browser, so that the defaults are what sites have come to depend on.

What assumptions have sites come to make that don't hold true in Chromium? Whatever they are, if the constraints don't overly limit the usefulness of the header, then I too think it might make sense to just standardize those constraints, and have shared tests for that.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BQCODpxsX6rEYfEO0omqq%2Bxan0bkfyZckN5FjpRyL_A-_VM7Q%40mail.gmail.com.

[Yana Yushkina]

We're actually not changing accept-language "defaults" for users who have not changed their language settings. What we're changing is how we're interpreting those settings when a user changes them to correctly reflect the will of the user in the accept-language headers.

We think that right now sites can't actually work around what we do to give the user the correct language. In the design doc we provide a canonical example of this:

"There can be a situation like the following: “en-US, fr, en”. The corresponding header would be: “en-US,fr;q=0.8,en;q=0.6”. Given that many websites do not recognize the first language+region code “en-US”, they perceive French as the first preference and return webpages in French language instead of English, if available."

[Claudio Magni]

More info has been added to the Firefox bug (thanks Anne for help).

[Philip Jägenstedt]

That the defaults are changing makes the risk of this lower, and in fact since you're trying to fix sites that are known to be broken, it's supposed to be a net win for site compat.

So, IIUC from the design doc, what is currently “en-US,fr;q=0.8,en;q=0.6” will become “en-US,en;fr;q=0.8”, is that right? (In my mind: "Insert region-free languages right after the first matching one with a region, drop later duplicates.")

This isn't the place to debate the design, but the fix makes sense to me. Can you also file bugs for WebKit and Edge?

I suspect that testing this using web-platform-tests won't be possible without WebDriver APIs to set the user preferences, and it's a case where I wonder if that investment would actually make sense, compared to each engine having unit tests for it. Nevertheless, do file bugs under https://github.com/w3c/web-platform-tests/labels/type%3Auntestable if you think it's worth pursuing.

LGTM1

[Rick Byers]

LGTM2

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYdSP7P6qbRqXpRLZkeOAvjyMneFk-mNxGFWNb8fBrk65w%40mail.gmail.com.

[Philip Jägenstedt]

On Wed, Aug 30, 2017 at 3:35 PM Philip Jägenstedt <foo...@chromium.org> wrote:

That the defaults are changing makes the risk of this lower

Typo, the defaults aren't changing :)

[Don London]

[Don London]

On Friday, August 25, 2017 at 7:05:15 AM UTC+2, Claudio Magni wrote:

[Dimitri Glazkov]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY_EdbwsUO%3DvjyAXbt0eYzVF8t_cShDFteCBe9-ZDd2sfA%40mail.gmail.com.

[Claudio Magni]

Thanks for the LGTMs.

I've added Dimitri as a review for the changelist. If you can't do it, please choose an appropriate reviewer for it. Thank you.

@Philip:

"what is currently “en-US,fr;q=0.8,en;q=0.6” will become “en-US,en;fr;q=0.8”, is that right? (In my mind: "Insert region-free languages right after the first matching one with a region, drop later duplicates.")".

That's is not right, but close. It would become "en-US,en;q=0.8,fr;q=0.6". We still decrement the qvalue for each language, in order to avoid any case (since it's not specified by the spec) in which a website sees both "en-US" and "en" with the same score so they decide to send "en" instead of "en-US".

Separately, we could drop later "duplicates" as you suggest, but it will have no effect on the behavior of websites, so I believe it's a metter of preference. My take is that the least we change the header with respect to the user preferences, the least weirdness we might have down the road.