Subject: Possible issue: Accept-language priority based on language order

From: Harald Alvestrand <[email-masked]>
Date: Wed, 23 Nov 2011 16:41:36 +0100

I ran across this in a discussion, and went to check it up; it might 
need fixing.

At the moment, section 6.4 of draft-ietf-httpbis-p3-payload-17.txt says:

6.4.  Accept-Language

    The "Accept-Language" header field can be used by user agents to
    indicate the set of natural languages that are preferred in the
    response.  Language tags are defined in Section 2.4.

      Accept-Language =
                        1#( language-range [ OWS ";" OWS "q=" qvalue ] )
      language-range  =
<language-range, defined in [RFC4647], Section 2.1>

    Each language-range can be given an associated quality value which
    represents an estimate of the user's preference for the languages
    specified by that range.

Way back in ancient history, when I wrote RFC 3282, I inserted the 
following clarification:

    The syntax and semantics of language-range is defined in [TAGS].  The
    Accept-Language header may list several language-ranges in a comma-
    separated list, and each may include a quality value Q.  If no Q
    values are given, the language-ranges are given in priority order,
    with the leftmost language-range being the most preferred language;
    this is an extension to the HTTP/1.1 rules, but matches current
    practice.

    If Q values are given, refer to HTTP/1.1 [RFC 2616] for the details
    on how to evaluate it.

Now, it seems still to be fairly normal to give a sequence of 
language-ranges in this header without any q= values, and expect the 
result to be deterministic.

Should this clarification be inserted into the updated HTTP document?
(The status of RFC 3282 is, btw, DRAFT STANDARD according to the RFC 
Editor's list)

                Harald

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Nov 2011 16:55:25 +0100

On 2011-11-23 16:41, Harald Alvestrand wrote:
> ...

I'm not sure that "fixing" is the right term; HTTP doesn't assign 
semantics to the ordering, and AFAIU never has.

How is this a problem in practice?

Best regards, Julian

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 23 Nov 2011 17:22:29 +0100

> How is this a problem in practice?

Well... either the semantics of the header

Accept-language: en, no

is that the languages are ordered by preference, or it does not imply an 
ordering.

As of the time I wrote the RFC (2002), I observed multiple browsers 
offering an UI that let people rank their preferences for language, and 
observed the relevant Accept-Language: headers being sent without q= values.

When I inquired, the response was that "leftmost language wins".
If this is still "what people do", it should be documented.

Note: Firefox and Chrome seem to send q= values in my current versions. 
The world might have changed.
But digging around in server-side code, I find that some code (which I 
hope is not used) actually ignores q values totally and just picks 
languages starting at left. Perhaps it hasn't.

                         Harald

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 24 Nov 2011 11:29:28 +1300

On Wed, 23 Nov 2011 17:22:29 +0100, Harald Alvestrand wrote:
> is that the languages are ordered by preference, or it does not imply
> an ordering.
>
> offering an UI that let people rank their preferences for language,
> and observed the relevant Accept-Language: headers being sent without
> q= values.
>
> Note: Firefox and Chrome seem to send q= values in my current
> versions. The world might have changed.
> But digging around in server-side code, I find that some code (which
> I hope is not used) actually ignores q values totally and just picks

 I've been keeping an eye on this since implementing language 
 negotiation in Squid.

 It appears that nearly all agents are sending the language codes sorted 
 by q value anyway. Whether they send the q value or not it is still 
 possible to optimize by using the left-most wins assumption.

 If anyone is interested in doing a deeper analysis I have a dataset 
 available covering the last year on several networks linking the 
 Accept-Language and User-Agent header pair.

 Amos

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Nov 2011 23:41:22 +0100

On 2011-11-23 23:29, Amos Jeffries wrote:
> ...
> I've been keeping an eye on this since implementing language negotiation
> in Squid.
>

Analysis would be good.

I'm skeptical because we're not really allowed to make changes breaking 
previously compliant implementations without *very* good reasons.

I'm also not too enthusiastic having to consider whether this would be 
*specific* to Accept-Language, or apply to all Accept headers.

Best regards, Julian

---

From: Dale Anderson <[email-masked]>
Date: Wed, 23 Nov 2011 18:18:04 -0800

Hi Julian, can you post the low-down on that dataset or email me on the
side I'm on paid time-off (not that HTTP isn't a pure pleasure, I assure
you I'm very fond especially of transmitting those texts bearing
Hyperlinks) but I will shoot the facts of the matter over to my time-on
colleages at F5 etc. Where's the data,

Thanks, (from the top of the world. Home office).

Dale Anderson

:)

2011/11/23 Julian Reschke <[email-masked]>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 24 Nov 2011 15:41:13 +0900

Hello Julian,

>> Accept-Language and User-Agent header pair.

I agree with Harald's analysis. I think it's up to people who want to 
claim the contrary to do some footwork.

I would definitely NOT go as far as Dale and say "ignore the q= values, 
they will be in order". That would explicitly be against the current 
spec. But saying that if there are no q values, then the leftmost 
matching should win will definitely bring the spec and reality closer 
together.

> previously compliant implementations without *very* good reasons.

Would the fact that widespread practice is different from the spec, and 
changing the spec seriously increases interoperability, not be a *very* 
good reason?

> *specific* to Accept-Language, or apply to all Accept headers.

I don't know much about the other headers. There is a strong difference 
between Accept-Language (which can be set by the user through 
Options/Preferences in most browsers) and the others (which are mostly 
just baked into the browser).

Regards,    Martin.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Nov 2011 09:43:12 +0100

> matching should win will definitely bring the spec and reality closer

That's not clear at all. A conforming implementation might handle things 
with the same q value (or missing values) as a set (unordered) and would 
become non-compliant with this change.

> just baked into the browser).

Indeed. But they share a similar syntax, and I'm really REALLY unhappy 
to add more special cases when they are not really needed.

Best regards, Julian

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 25 Nov 2011 00:15:19 +0100

Thanks for the datasets, Amos!

Quick analysis of the 1742 different Accept-Language header:

156 multiple languages, none with q values
247 single language with no q value
43 all languages with q value
1255 all languages but one with q value
41 multiple languages without q value, some with q value

I didn't check whether the values were always sorted; there were some 
like this one:

th-th,th;q=0.8,en-us;q=0.6,en-gb;q=0.4,en;q=0.2,x-ns1rW_REX3VNhu,x-ns2p1c0Nnym7b6

where it certainly looks as if the accept-language header was used to 
communicate something that isn't a standard language, but strictly 
speaking, those rightmost values sort before #2 from the left, because 
the default q value is 1.0.

So there are 197 examples of headers whose interpretation according to 
the standard might be affected by the proposed interpretation (or 
integration of information from another specification).

If practice out there is consistent with regard to this interpretation, 
I think it is good to document it, so that we might reduce the chance of 
future practice from diverging from current practice.

              Harald

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Nov 2011 16:39:34 +0100

> integration of information from another specification).

Could you by any chance check what UAs are sending these?

I just tried FF8/IE9/Chrome15/Opera11, and they all send q values. (For 
Safari, I couldn't figure out how to get multiple languages into the 
header).

> future practice from diverging from current practice.

I'm still unconvinced that there is a problem here:

- we have no proof that the order is intentional and that it matters in 
these cases

- there may be implementations written to what HTTP always said that 
treat values with same q value (or missing q value) as having the same 
preference, and a change to the spec would make those non-compliant

- changing Accept-Language would make it inconsistent with the other 
Accept header fields (or do we want to change them all?)

We *could* add a note that the interpretation is different here to the 
MIME variant, and point out that some senders may rely on it (if we 
really believe that).

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 26 Nov 2011 15:42:14 +0100

> integration of information from another specification).

The data set that we're looking out has only one entry per unique 
charset/UA value. This makes it impossible to understand how frequent 
these combinations actually appear. For instance, out of the 45000 
records I see roughly ~100 for Firefox 7 and Firefox 8 combined.

Maybe the dataset could be regenerated with information about the 
frequency of each of these combinations?

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 27 Nov 2011 16:38:42 +1300

> frequency of each of these combinations?

Sorry. It is generated from anonymixed contributions by our client ISP. 
Due to the intended purpose and size of the raw data frequency is not 
available. All I get is what you see.

AYJ

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 27 Nov 2011 07:44:19 +0100

On 11/25/2011 04:39 PM, Julian Reschke wrote:
> I just tried FF8/IE9/Chrome15/Opera11, and they all send q values. 
> (For Safari, I couldn't figure out how to get multiple languages into 
> the header).
They are all over the map, it seems.
Some Android versions, some Chrome versions, some MSIE Media Center 
edition, some Safari versions, some crawlers ...... it seems that these 
are the long tail of the browser market, and stuff that uses HTTP but 
isn't really browsers.

Example:

en-us,en Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; Valve Steam 
Client/1672; ) AppleWebKit/534.1 (KHTML, like Gecko) Chrome/6.0.444.0 
Safari/534.1

That's likely not a Chrome or a Safari, but an embedded browser in Valve 
Steam.
If practice out there is consistent with regard to this interpretation,
> - we have no proof that the order is intentional and that it matters 
> in these cases
We have code examples that pick the leftmost language at the server.
We have no code examples that do anything else (so far).
> preference, and a change to the spec would make those non-compliant
Hm. The Apache 1.3 documentation:
http://httpd.apache.org/docs/1.3/content-negotiation.html
seems to indicate that Apache treats them like a set.

The Apache 2.2 documentation:

http://httpd.apache.org/docs/2.2/content-negotiation.html
says "Select the variants with the best language match, using either the 
order of languages in the Accept-Language header (if present), or else 
the order of languages in the LanguagePriority directive (if present)."

(this step is applied only if two or more variants have the same preference)

So it seems that Apache 1.3 did not document the RFC 3282 behaviour, but 
Apache 2.2 documents that it behaves according to RFC 3282.

> Accept header fields (or do we want to change them all?)
What other fields do we have?
The four Accept headers are Accept, Accept-Charset, Accept-Encoding and 
Accept-Language.
> MIME variant, and point out that some senders may rely on it (if we 
I don't believe this header is in significant use outside HTTP. The 
writeup in RFC 3282 was intended specifically to capture HTTP usage. If 
you really have evidence that common HTTP usage is not consistent with 
RFC 3282, an errata should be filed against RFC 3282.

I have so far seen no such evidence.

                      Harald

---

From: Julian Reschke <[email-masked]>
Date: Sun, 27 Nov 2011 10:01:59 +0100

> That's likely not a Chrome or a Safari, but an embedded browser in Valve

Staying with the example: in all cases, this browser sent either no A_L 
header, or

   en-us, en

That really doesn't look like a sufficient reason to make a change.

> Apache 2.2 documents that it behaves according to RFC 3282.

No. What it says is "or else the order of languages in the 
LanguagePriority directive (if present)." - that's the order in the 
httpd config entry, not the header field.

> RFC 3282, an errata should be filed against RFC 3282.

Hm, no. So far, the header type registry refers to RFC 2616 as normative 
specification for Accept-Language. It only mentions RFC 3283 for mail.

> I have so far seen no such evidence.

What I see is that all major desktop browsers send q values. Sure, there 
are many non-browser UAs, but so far we don't have any statistics that 
could help making a decision.

Best regards, Julian

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 27 Nov 2011 11:09:00 +0100

Trimming to one specific point:

> httpd config entry, not the header field.
I don't see how you can read the text that way. It says

EITHER the order of languages in the Accept-Language header (if present)
OR ELSE the order of languages in the LanguagePriority directive (if 
present)

So if the Accept-Language header is present, the order of languages in 
the LanguagePriority is NOT consulted.

That's how I read it. But the code rules.

                  Harald

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 28 Nov 2011 10:21:29 +1100

My .02 -

I'm reluctant to have *two* ways to indicate priority in the same header; that's asking for bugs.

Yes, the code rules, but picking out a few (or even few hundred) examples is not "the code." The implementation footprint of HTTP is massive.

Cheers,

On 27/11/2011, at 1:42 AM, Julian Reschke wrote:

> The data set that we're looking out has only one entry per unique charset/UA value. This makes it impossible to understand how frequent these combinations actually appear. For instance, out of the 45000 records I see roughly ~100 for Firefox 7 and Firefox 8 combined.
> 
> Maybe the dataset could be regenerated with information about the frequency of each of these combinations?
> 
> Best regards, Julian
> 

--
Mark Nottingham   http://www.mnot.net/