Subject: #428 Accept-Language ordering for identical qvalues

From: Julian Reschke <[email-masked]>
Date: Wed, 16 Jan 2013 16:56:08 +0100

Hi there,

with <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2119#file1>, 
the spec now says:

"If no quality values are assigned or multiple language tags have been 
assigned the same quality, the same-weighted languages are listed in 
descending order of priority."

This is a change from both RFC 2068 and RFC 2616 which we *did* discuss 
back in the thread starting with <​ 
http://lists.w3.org/Archives/Public/ietf-http-wg/2011OctDec/0223.html>; 
back then we decided not to make this change because we know of 
implementations ignoring the ordering, and no convincing argument was 
given for making the ordering significant.

I believe this change should be backed out.

Best regards, Julian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 17 Jan 2013 17:26:25 +0900

Hello Julian,

> given for making the ordering significant.

We also know of implementations, both on the sender and on the receiver 
side (as far as I remember) that use ordering. So maybe something like:

"If no quality values are assigned or multiple language tags have been
assigned the same quality, the same-weighted languages may be listed in
descending order of priority."

I'm sure somebody can come up with better wording, but I hope you get 
the idea.

Regards,   Martin.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Jan 2013 10:00:12 +0100

> I'm sure somebody can come up with better wording, but I hope you get

The key question is whether the order of same-weighted values matters. 
The proposed text doesn't answer that.

It hasn't mattered according in RFC 2068 and 2616, it doesn't matter for 
other Accept-* header fields, and we know of recipients ignoring the 
order. I think this is sufficient reason to stick with what we had in 
2616, meaning reverting this change.

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 17 Jan 2013 00:59:39 -0800

On Jan 16, 2013, at 7:56 AM, Julian Reschke wrote:

> Hi there,
> 
> with <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2119#file1>, the spec now says:
> 
> "If no quality values are assigned or multiple language tags have been assigned the same quality, the same-weighted languages are listed in descending order of priority."
> 
> This is a change from both RFC 2068 and RFC 2616 which we *did* discuss back in the thread starting with <​ http://lists.w3.org/Archives/Public/ietf-http-wg/2011OctDec/0223.html>; back then we decided not to make this change because we know of implementations ignoring the ordering, and no convincing argument was given for making the ordering significant.

The change is to improve interoperability when the preferences sent
result in a tie or contain no qvalues.

http://www.w3.org/International/questions/qa-lang-priorities.en.php

Firefox and Chrome have an ordered language UI that takes whatever
list the user comes up with and creates q-values to associate with
each language tag after the first.  The languages are always listed
in order of preference.  I've heard that Opera and MSIE do the same,
but haven't tested.  Chrome has a bug with the ordering of tags to
match the UI, but that's orthogonal to this issue.

Safari does not support languages other than the list set in the
user's system settings, and it only provides the first of those,
which is wrong for both functionality and privacy reasons.
The system settings define the core UI in OS X, so the user can't
turn it off.

Older browsers did not send qvalues. Hence, server implementations
of language negotiation do use the ordering provided as I described
in the change.

> I believe this change should be backed out.

The change has no impact on user agents that send distinct
qvalues.  At most, it would change the interpretation for those
few requests that still rely on ordered language tags, for which
the prior specs had no interpretation at all.

http://forums.thedailywtf.com/forums/t/15895.aspx

What I added is how Apache httpd has implemented it since before
any of the HTTP specs were RFCs, which was compatible with how
CERN httpd implemented it before that (no qvalues at all).
The change has no impact on conformance because language
negotiation is optional and the change is not expressed as a
requirement.  It also resolves an inconsistency with RFC4647.

There are far more examples on the Web where applications
incorrectly assume the list is ordered

  http://pic.dhe.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=%2Fcom.ibm.itame.doc_6.1%2Fam61_webseal_admin134.htm

  http://www.developershome.com/wap/detection/detection.asp?page=acceptLanguageHeader

than there are servers that implement language negotiation and
actually want to resolve ties at random.

As Harald said,

> it seems still to be fairly normal to give a sequence of 
> language-ranges in this header without any q= values, and expect the 
> result to be deterministic.

because that's how it works in practice for the vast majority
of systems that implement content negotiation.  The spec should
reflect what is most interoperable for the users.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Jan 2013 10:14:04 +0100

On 2013-01-17 09:59, Roy T. Fielding wrote:
> in order of preference.  I've heard that Opera and MSIE do the same

But they have different qvalues, so these UAs do *not* rely on ordering.

> ...

Some, apparently. But not all. Servers have been written according to 
the spec, ignoring ordering. If we make ordering significant, these will 
not interoperate anymore.

> the prior specs had no interpretation at all.

Well, it had an interpretation ("same weight").

It's good that there are only few requests on relying on this. Do you 
want to encourage more, potentially breaking servers that ignore ordering?

> http://forums.thedailywtf.com/forums/t/15895.aspx

Yes, I'm aware that this is a FAQ. But we're not starting from scratch here.

> requirement.  It also resolves an inconsistency with RFC4647.

It does have an impact, because servers that previously implemented the 
optional feature now do not anymore.

> incorrectly assume the list is ordered

That'll be hard to count. :-)

> actually want to resolve ties at random.

They do not "want" to resolve at random; they do so because they have 
implemented what the spec says. There's no reason to create an ordered 
list structure when the spec says that an unordered list is sufficient.

>> result to be deterministic.

I'd like so see evidence of that, not hearsay. Browsers do *not* rely on 
this.

> reflect what is most interoperable for the users.

As far as I can tell, what's interoperable is the exact opposite: not 
relying on ordering, but sending qvalues.

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 17 Jan 2013 02:17:06 -0800

> But they have different qvalues, so these UAs do *not* rely on ordering.

They order them *and* they send qvalues, because of broken sites like

   http://wiki.nginx.org/AcceptLanguageModule

and the change I made has no effect on UAs that send qvalues.
It does, however, improve the lot for users of other user agents
that either do not send qvalues or allow the user to specify the
value by hand, either of which can result in same-valued tags.

> Some, apparently. But not all. Servers have been written according to the spec, ignoring ordering. If we make ordering significant, these will not interoperate anymore.

In what way do they interoperate now, and in what way does that
change?  As far as I can tell, the only effect this change has
is a suggestion that they not respond randomly.

> It's good that there are only few requests on relying on this. Do you want to encourage more, potentially breaking servers that ignore ordering?

It isn't possible to break those servers any more than they
are already broken by responding in a random fashion.

> It does have an impact, because servers that previously implemented the optional feature now do not anymore.

A server is not required to obey Accept-Language.

> They do not "want" to resolve at random; they do so because they have implemented what the spec says. There's no reason to create an ordered list structure when the spec says that an unordered list is sufficient.

Yes, which is why we are changing the spec so that they can eventually
improve their implementation (or not).  An unordered list is not
sufficient because users don't send unordered lists, not even when
they are hand-crafted.  Everyone assumes they are ordered or they
send them with correct qvalues.

> I'd like so see evidence of that, not hearsay. Browsers do *not* rely on this.

ENOCARE.

> As far as I can tell, what's interoperable is the exact opposite: not relying on ordering, but sending qvalues.

One has nothing to do with the other.  The added text does not
change the ordering for UAs that send proper, non-identical
qvalues on language tags, even if they choose not to sort them
by order (which the latest browsers do for a good reason).

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Jan 2013 22:03:53 +0100

>     http://wiki.nginx.org/AcceptLanguageModule

...so they do not rely on recipients treating the list as ordered.

> and the change I made has no effect on UAs that send qvalues.

Agreed.

> value by hand, either of which can result in same-valued tags.

Well, it's doing that at the cost of making recipients non-compliant 
that implement RFC 2068/2616. So this is a change that would need to be 
listed in the Changes section.

> is a suggestion that they not respond randomly.

Right now they interoperate as specified by the spec. If we change the 
spec, they do not anymore (or only some of the time).

> are already broken by responding in a random fashion.

They are not broken. They do what the spec says.

> A server is not required to obey Accept-Language.

Yes, but a server that *does* implement it according to the *current* 
spec (literally) won't be conforming to the new spec anymore.

> send them with correct qvalues.

If people stick to what 2068 and 2616 says, there is no problem.

>> I'd like so see evidence of that, not hearsay. Browsers do *not* rely on this.

Not helpful.

> by order (which the latest browsers do for a good reason).

Yes. But it makes recipients that are currently conforming potentially 
non-conforming. Yes, for an optional feature, but still.

It seems there are two logical approaches:

(1) Leave things as they are, but explain that HTTP is not totally 
consistent with RFC4647, and that there may be broken senders that rely 
on ordering, and also recipients that do support this; thus it might be 
a good idea to mirror that behavior.

OR

(2) Change the spec to what it says no, noting that this is a change 
from 2068/2616, and also inconsistent with other Accept header fields 
(or do you want to change those, too???).

Note that introducing an inconsistency here may be a bad idea; either it 
defeats code re-use, or the "new old" semantics may bleed over 
unintentionally into the other header fields.

My preference is (1). I also don't think we have not even rough 
consensus for a change from 2616 here.

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 18 Jan 2013 21:46:39 +1300

On 18/01/2013 10:03 a.m., Julian Reschke wrote:
> that implement RFC 2068/2616. So this is a change that would need to 
> be listed in the Changes section.
>

I'm with Roy on this one. It's not adding any new requirement about 
interpretation, simply stating that the list is ordered, as is actually 
the case from most senders.
There is no requirement added/removed about server interpretation so 
those servers implementing random selection out of the ordered set are 
still compliant. Those servers implementing ordered interpretation are 
now compliant - where before with the list defined as un-ordered they 
would be non-compliant due to mis-interpreting an un-ordered list as 
ordered.

>>> Some, apparently. But not all. Servers have been written according 
>>> to the spec, ignoring ordering. If we make ordering significant, 
>>> these will not interoperate anymore.
>>

The new spec does not forbid random selection. Merely states that the 
client *wants* it to be interpreted non-randomly. Obeying that client 
preference is still optional.

Amos

---

From: Julian Reschke <[email-masked]>
Date: Fri, 18 Jan 2013 10:07:20 +0100

> still compliant. Those servers implementing ordered interpretation are

They are? How so?

If the client sends

	Accept-Language: en, de

and the server returns German text, although English would have been 
available, is it still compliant?

> would be non-compliant due to mis-interpreting an un-ordered list as

That doesn't make sense, sorry.

If the list ordering is defined to be irrelevant it's totally ok to pick 
the first match.

> preference is still optional.

Again, that doesn't make any sense at all.

If we say that the list is ordered by preference (in absence of 
qvalues), this implies that a recipient should pick the *first* matching 
language. If it does not, it's not interpreting the message as defined.

Best regards, Julian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 18 Jan 2013 18:57:02 +0900

> available, is it still compliant?

One way to claim compliance is to claim that the German text is of 
higher quality. If there are no (or the same) q-values, then higher 
quality on the server side wins.

So I'm with Roy (and now Amos) on this.

Regards,    Martin.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 18 Jan 2013 11:13:22 +0100

> quality on the server side wins.

Let's assume for this argument that they are same quality.

> So I'm with Roy (and now Amos) on this.

I don't see how that follows from what you just said...

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 18 Jan 2013 23:49:20 +1300

> I don't see how that follows from what you just said...

The server has potentially some idea of the translated % or quality of 
trust in each phrases translation.

Even putting that aside though:

Server A responds with German. Server B responds with German.
Which one was selecting randomly? which was responding with sequential 
search and missing English? you can't tell.

Likewise, if they return German and English respectively you still can't 
tell unless you are maintaining state and considering requests statefully.

Then you need to consider a server which does object translation based 
on the Accept-Language header. If it *were* to respond with random 
selection of language between those two and host an entire site 
translated into both. Users would end up getting a random language 
change on every page load as they navigated, potentially each of several 
components on a page in different languages. The random selection is 
*not* a good algorithm to use from this point of view and effecitively 
mandates stateful sessions just for the translation to work in a 
user-friendly way.

FWIW: Squid is able to do language negotiation on its generated 
responses fast and statelessly only by ignoring q-values and handling 
the list as ordered. Managing q-values or comparing the entire list 
against available languages increases the response processing costs by a 
factor of 6.

Amos

---

From: Julian Reschke <[email-masked]>
Date: Fri, 18 Jan 2013 11:57:55 +0100

> mandates stateful sessions just for the translation to work in a

Um, no. That would be the case if common browsers indeed would send 
multiple values without qvalues, which they do not.

> against available languages increases the response processing costs by a

That may be true but doesn't change the arguments. We are not designing 
something new. If Squid is ignoring qvalues then it's not compliant to 
RFC 2068, RFC 2616, *nor* the new text.

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 18 Jan 2013 23:59:37 +1300

> available, is it still compliant?

Where is the text saying the server MUST or even SHOULD return the 
clients first preference?
I could not find any in RFC 2068, 2616 or the current Draft.

The server gets to decide what it returns, the text is lacking in 
normative requirements, so we are quibbling over 
compliance/non-compliance with textual guidance on best interoperable 
practice.

> If the list ordering is defined to be irrelevant it's totally ok to 
> pick the first match.
>

If the ordering is defined as relevant to the clients preference, why is 
it non-compliant to pick one of the *available* ones randomly? a bit 
daft, but as you say that used to be prescribed so tose who opted to 
follow the old text see no harm in it. They have the option of 
optimizing a bit or ignoring the new text.

> qvalues), this implies that a recipient should pick the *first* 
> matching language. If it does not, it's not interpreting the message 
> as defined.

Sure it is both unfriendly to the client and requires sub-optimal code 
to actually perform. But there is still no requirement to return the 
*first* matching language after the change. Just the implication that 
more optimal code may be used.

Amos

---

From: Julian Reschke <[email-masked]>
Date: Fri, 18 Jan 2013 12:09:59 +0100

> I could not find any in RFC 2068, 2616 or the current Draft.

How is it relevant whether there's a SHOULD or MUST?

If the specification describes the meaning of the message, and the 
server ignores some part of that, it's not working per spec.

> compliance/non-compliance with textual guidance on best interoperable

The text describes the semantics of the header field. It doesn't need to 
invoke MUSTs or SHOULDs for that.

> optimizing a bit or ignoring the new text.

If the client says "I take A and B but prefer A", and the server has 
both, then returning B is not the right thing to do.

> more optimal code may be used.

We seem to have a fundamental disagreement about the way the spec 
defines things. This is not the only place where we define things by 
simply stating facts; it doesn't mean a server can simply ignore those 
things and still be considered doing the right thing.

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 18 Jan 2013 04:11:21 -0800

On Jan 18, 2013, at 1:07 AM, Julian Reschke wrote:

> and the server returns German text, although English would have been available, is it still compliant?

Yes.  It would also be conformant to send Mäori text.

> If the list ordering is defined to be irrelevant it's totally ok to pick the first match.

Yes.

> If we say that the list is ordered by preference (in absence of qvalues), this implies that a recipient should pick the *first* matching language. If it does not, it's not interpreting the message as defined.

Ignoring the preferences sent in Accept-Language is conforming behavior.

Conformance is not a relevant issue here.  What matters is what the
user actually prefers. It is my opinion that when a user sets an
Accept-Language header to

  Accept-Language: en, de

what they are actually saying is that they accept both languages
but would prefer en if the de representation is no better.

The reason I believe this is because user agents that allow a
user to send such a header field have explicitly instructed the
user that the field is ordered (or based the value on some other
ordered list for the host UI, as is the case for some cell phones).
Servers that wish to implement content negotiation according to the
user's desires (not just the IETF's desires) will, in fact, treat
the order as significant; failing to do so results in bug reports.

This is a change to the specified semantics for Accept-Language,
but that is well within the scope of what we are chartered to do.

....Roy

---

From: Nicholas Shanks <[email-masked]>
Date: Fri, 18 Jan 2013 12:50:29 +0000

>>> I'm with Roy on this one. It's not adding any new requirement about

I feel I concur with Julian the most.

> Yes.  It would also be conformant to send Mäori text.

Use a macron or leave it off ;-)
[Option-a] [a] on a Mac with one of the "Extended" keyboard layouts

> but would prefer en if the de representation is no better.

You cannot assume that. They are either using a broken client, or both
are acceptable. Please don't change the standard to accomodate broken
clients, especially as these are going to become fewer in number as
time progresses and machines get upgraded.

> ordered list for the host UI, as is the case for some cell phones).

All UAs I know of that allow users to set an ordered list of
languages, also send auto-generated q-values.

Do you actually have any statistics to back up your belief, or is it
just a gut feeling?
Some numbers to say that "versions x and earlier of so-and-so browser
on X-series phones allow users to define an ordered list but do not
send q-values; those browsers currently have a worldwide market share
of 0.0001%" would be useful to know whether it's worth ignoring such
broken UAs to pandering to them.

FWIW, my usual AL string, in browsers that let you set one, is:
"en-GB, en-IE, en-AU, en-US;q=0, en;q=0.95, fr;q=0.5, de;q=0.5,
zh-Hant;q=0.1, *;q=0.2"
My goals should be self-evident from the q-values, specifically to get
english, french or german, to demote 'complicated' Han script and fall
back to anything else. The US thing is to see if sites are actually
obeying my preferences (I get many more "y'all"s than 406's sadly!)

-- 
Nicholas.

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 19 Jan 2013 17:29:20 +1300

> obeying my preferences (I get many more "y'all"s than 406's sadly!)

According to the 2616 spec we are quibbling over you would get en-GB, 
en-IE, or en-AU if any of them were available. These being assumed to be 
q=1 and all equal valued; one is supposed to be selected *randomly*.

Now suppose you had a blog page with each comment loaded by XHR as a 
separate GET request using that AL header and auto-translation of 
comments - your page looks like a group of multi-cultural responses some 
possibly with pidgin-English style wording despite probably all actually 
being en-US text to begin with. AND if you refresh the page everybodies 
language changes from what it was last load.

Versus a server which assumed en-GB, en-IE, en-AU were equal q=1 and 
ordered by preference would supply you with the en-GB for each response 
part of the page.

Which is the better outcome for web developers to rely on?
Which one is easier for servers to write fast code for?

Amos

---

From: Nico Williams <[email-masked]>
Date: Sat, 19 Jan 2013 01:03:22 -0600

On Jan 17, 2013 3:05 PM, "Julian Reschke" <[email-masked]> wrote:
> Well, it's doing that at the cost of making recipients non-compliant that
implement RFC 2068/2616. So this is a change that would need to be listed
in the Changes section.

So what?  Lots of implementations of lots of Internet protocols are
non-compliant with the relevant specs.  It's not like there's an IETF
Police.  Also, this squeamishness about making improvements like this means
we're stuck with mistakes forever, even when fixing them breaks nothing.

Nico
--

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Sat, 19 Jan 2013 07:09:52 +0000

RFC 2616 already strongly implies (in prose not ABNF) order preference

>From 14.4

"The quality value defaults to "q=1". For example,

Accept-Language: da, en-gb;q=0.8, en;q=0.7

would mean: "I prefer Danish, but will accept British English and
other types of English."

The prose states a preference of Danish over the following languages.

Actually the non-requirement for a server to take into account any 
indicated preferences in Accept-* leads to some interesting conundra for 
caching.

For instance,

GET /something HTTP/1.1
Host: someserver.org
Accept-Language: fr
...

200 OK Document follows
Content-Language: en
Vary: Accept-Language
...

leads to problems when the cache sees a request

GET /something HTTP/1.1
Host: someserver.org
Accept-Language: en
...

Even though it's obvious to send the cached resource, it's not correct 
since the selecting header does not match between requests.  This could 
be ameliorated if there were ETags since at least the cached version 
could be checked?

Adrien

------ Original Message ------
From: "Amos Jeffries" <[email-masked]>
To: "Nicholas Shanks" <[email-masked]>
Cc: "Roy T. Fielding" <[email-masked]>; "Julian Reschke" 
<[email-masked]>; "[email-masked]" <[email-masked]>
Sent: 19/01/2013 5:29:20 p.m.
Subject: Re: #428 Accept-Language ordering for identical qvalues
>en-IE, or en-AU if any of them were available. These being assumed to 
>be q=1 and all equal valued; one is supposed to be selected *randomly*.
>
>comments - your page looks like a group of multi-cultural responses 
>some possibly with pidgin-English style wording despite probably all 
>actually being en-US text to begin with. AND if you refresh the page 
>everybodies language changes from what it was last load.
>

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Sat, 19 Jan 2013 07:12:14 +0000

sorry, missed that ;q=0.8

ignore that comment about order pref

------ Original Message ------
From: "Adrien W. de Croy" <[email-masked]>
To: "Amos Jeffries" <[email-masked]>; "Nicholas Shanks" 
<[email-masked]>
Cc: "Roy T. Fielding" <[email-masked]>; "Julian Reschke" 
<[email-masked]>; "[email-masked]" <[email-masked]>
Sent: 19/01/2013 8:09:52 p.m.
Subject: Re: #428 Accept-Language ordering for identical qvalues
>indicated preferences in Accept-* leads to some interesting conundra 
>for caching.
>
>For instance,
>
>GET /something HTTP/1.1
>Host: someserver.org
>Accept-Language: fr
>...
>
>200 OK Document follows
>Content-Language: en
>Vary: Accept-Language
>...
>
>>ordered by preference would supply you with the en-GB for each 
>>response part of the page.
>>

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 19 Jan 2013 22:02:44 +1300

> ignore that comment about order pref

But your message is another great example of why stating order will 
improve interoperability.

The explicit statement that no q-value defaults to q=1, AND discussion 
about ordering by q-value leads many to interpret the old specs as the 
header being ordered exactly like you did there. Some large deployed 
implementations are doing this; correctly or not according to 2616.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Jan 2013 13:34:39 +1100

Julian et al,

I think the important bit here is the context that we're talking about the semantics of an expressed preference -- which can be freely ignored, or selectively applied, without affecting conformance. The important thing is that the preference itself have clear semantics, which I think Roy's change does (especially in concert with changes elsewhere).

As such, I think the relevant question is whether this is specific to A-L, or all A-* that take qvalues. Roy, thoughts?

--
Mark Nottingham   http://www.mnot.net/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sun, 20 Jan 2013 04:52:03 -0800

> As such, I think the relevant question is whether this is specific to A-L, or all A-* that take qvalues. Roy, thoughts?

I am pretty sure it is specific to languages.  Accept has never been
treated as an ordered list, Accept-Encoding was originally designed
to prefer the smallest representation (changing that to qvalues was
unfortunate), and Accept-Charset is almost deprecated at this point.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 21 Jan 2013 08:51:40 +1100

> unfortunate), and Accept-Charset is almost deprecated at this point.

So, wouldn't the same arguments (minus the implementation status) apply to Accept?

I.e., if it's just a preference, and the server is free to choose among the preferences anyway (or even ignore them), why *not* say Accept is ordered?

---

From: James M Snell <[email-masked]>
Date: Sun, 20 Jan 2013 14:50:43 -0800

+1.. in fact, for 2.0, I'd very much like to get rid of q-values entirely
and depend entirely on order.
On Jan 20, 2013 1:54 PM, "Mark Nottingham" <[email-masked]> wrote:

> >> I think the important bit here is the context that we're talking about
> the semantics of an expressed preference -- which can be freely ignored, or
> selectively applied, without affecting conformance. The important thing is
> that the preference itself have clear semantics, which I think Roy's change
> does (especially in concert with changes elsewhere).
> >>
> >> As such, I think the relevant question is whether this is specific to
> A-L, or all A-* that take qvalues. Roy, thoughts?
> >
> I.e., if it's just a preference, and the server is free to choose among
> the preferences anyway (or even ignore them), why *not* say Accept is
> ordered?
>
>
>
>
> --

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 21 Jan 2013 09:59:00 +1100

On 21/01/2013, at 9:50 AM, James M Snell <[email-masked]> wrote:

> +1.. in fact, for 2.0, I'd very much like to get rid of q-values entirely and depend entirely on order.

Doing that would lose information; e.g., there's no way to say q=0 in a purely ordered set. 

That may or may not be acceptable, but we need to talk through it.

Cheers,

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Sun, 20 Jan 2013 23:30:45 +0000

------ Original Message ------
From: "James M Snell" <[email-masked]>
>entirely and depend entirely on order.
same here.

The idea may have been laudable in 1998, but really, how can a web 
server tell if some resource is 80% better than another?  A human needs 
to tell it, and humans have enough trouble with other things.

the q=0 option would need to be turned into a Naccept-* header or 
something.   But does anyone even use it outside of testing for 406 
responses which never come?

>> >> I think the important bit here is the context that we're talking 
>>about the semantics of an expressed preference -- which can be freely 
>>ignored, or selectively applied, without affecting conformance. The 
>>important thing is that the preference itself have clear semantics, 
>>which I think Roy's change does (especially in concert with changes 
>>elsewhere).
>> >>
>> >> As such, I think the relevant question is whether this is specific 
>>to A-L, or all A-* that take qvalues. Roy, thoughts?
>> >
>>I.e., if it's just a preference, and the server is free to choose 
>>among the preferences anyway (or even ignore them), why *not* say 
>>Accept is ordered?
>>
>>
>>
>>
>>--

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 21 Jan 2013 14:56:23 +1300

On 21/01/2013 12:30 p.m., Adrien W. de Croy wrote:
> From: "James M Snell" <[email-masked] <mailto:[email-masked]>>
>>

My collection of 2 years worth of language headers says no.

Of 2018 unique Accept-Language header field-values;
   1532 are using q-values in a strictly sorted list
   491 are not using q-values
   14 are using "q=0.0".
   5 are using q-values and non-qvalues without ordering the sent list 
(1 looks otherwise normal, teh others are using puny-codes)

The 14 are also unique in being very long and having multiple entries 
with equal q-values. They are still without exception strictly ordered 
with the entries having no q-value entries first (as if q=1.0 was used 
for sort but omitted sending). They are also containing a number of 
oddities such as multiple entries for language codes with differing 
q-values.

NP: Of those 14 odd A-L headers noted above I have UA details on 8 of 
them. All claim to be Firefox but the Gecko dates do not line up with 
other info on those versions (the 11.0 was released some years before 
3.5.9 on the same OS) so the whole input is a bit suspect.

The 5 cases un-ordered list have puny-code values with no q-value being 
listed after an otherwise normal series of languages. Like so:
  "en-us,en;q=0.5,x-ns1qHkbtrt8Nhv,x-ns2E1e0Nnym7b6"

I have a few cases of q-value ordered list followed by wildcard "*" with 
no q-value. Sender obviously assuming the list is ordered.

Broken down by UA, which I started ~6 months ago at Juliens suggestion I 
have 54289 distinct UA visiting, of which;
   21756 are not sending A-L header at all
   19621 unique UA are using a single language code with no q-value
   12495 unique UA are using q-values as above.
   8 are sending only wildcard "*" or "*/*"

The remainder ~400 roughly match up with the 491 AL field-values not 
using q-values. Are older agents (Windows 98, NT, 2k stand out), agents 
sending the same language multiple times (VoilaBot variants and Safari 
there), or sending sub-language variants with the generic form last eg 
"en-GB,en", "en-US,en", "en-US,en,*" (Tablets and Mobile Safari mostly). 
Obviously assuming sorted lists even back into the Windows 98 ones.

There are also a few bots sending exactly 2 puny-code entries.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 21 Jan 2013 13:06:47 +1100

That's interesting, thanks. 

One thing to add; even if the client includes a q=0, the server can still ignore it. 

Cheers,

P.S. If you are able (considering privacy issues, etc.) and want to dump such data in a useable format, feel free to ask for a repository on the github account.

On 21/01/2013, at 12:56 PM, Amos Jeffries <[email-masked]> wrote:

>> The idea may have been laudable in 1998, but really, how can a web server tell if some resource is 80% better than another? A human needs to tell it, and humans have enough trouble with other things.
>> the q=0 option would need to be turned into a Naccept-* header or something.   But does anyone even use it outside of testing for 406 responses which never come?
> 
>  5 are using q-values and non-qvalues without ordering the sent list (1 looks otherwise normal, teh others are using puny-codes)
> 
> The 14 are also unique in being very long and having multiple entries with equal q-values. They are still without exception strictly ordered with the entries having no q-value entries first (as if q=1.0 was used for sort but omitted sending). They are also containing a number of oddities such as multiple entries for language codes with differing q-values.
> 
> NP: Of those 14 odd A-L headers noted above I have UA details on 8 of them. All claim to be Firefox but the Gecko dates do not line up with other info on those versions (the 11.0 was released some years before 3.5.9 on the same OS) so the whole input is a bit suspect.
> 
> 
> The 5 cases un-ordered list have puny-code values with no q-value being listed after an otherwise normal series of languages. Like so:
> I have a few cases of q-value ordered list followed by wildcard "*" with no q-value. Sender obviously assuming the list is ordered.
> 
> 
> 
> Broken down by UA, which I started ~6 months ago at Juliens suggestion I have 54289 distinct UA visiting, of which;
> The remainder ~400 roughly match up with the 491 AL field-values not using q-values. Are older agents (Windows 98, NT, 2k stand out), agents sending the same language multiple times (VoilaBot variants and Safari there), or sending sub-language variants with the generic form last eg "en-GB,en", "en-US,en", "en-US,en,*" (Tablets and Mobile Safari mostly). Obviously assuming sorted lists even back into the Windows 98 ones.
>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 21 Jan 2013 00:37:12 -0800

> I.e., if it's just a preference, and the server is free to choose among the preferences anyway (or even ignore them), why *not* say Accept is ordered?

I don't see any value in that given the lack of users setting the
values for Accept directly (outside of command-line tool usage)
and no assumption among UAs that Accept is ordered.

Apache httpd resolves ties in type negotiation using the
internal ordering of representation variants.  That is unlike
languages, for which the code deliberately checks the order received
in Accept-Language for resolving ties.

Regarding proactive negotiation in HTTP/2, I'll note that Waka
strips all negotiation fields.  I find the entire feature revolting,
from every architectural perspective, and would take the opportunity
of 2.x to remove it entirely.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 21 Jan 2013 21:41:42 +1100

> of 2.x to remove it entirely.

The value is that Accept-* headers all use qvalues the same way, which is much less confusing for pretty much everybody, and paves the way for a potentially simpler approach in 2.0 (your feelings about having proactive negotiation at all noted).

Again, implementations are free to ignore it -- this is merely the semantics of the preference, not constraints on how it's followed.

---

From: Nicholas Shanks <[email-masked]>
Date: Mon, 21 Jan 2013 14:22:50 +0000

I have a 111 MB access log beginning in 2004 that records UA, AL & AC headers which I don't mind posting publicly. I have not analysed it so do not know what fraction of this log is bots, and it's only from an obscure 'test your accept language/charset headers' page on my personal site, so could be full of weird junk.

"wc -l" says there are 1.8 million lines in there, and the head of the file looks fine, but when I run tail, I see "n" where my script should have written a new line, so I probably edited the script at some point and missed a backslash. that will need cleaning up.

― Nicholas.

---

From: James M Snell <[email-masked]>
Date: Mon, 21 Jan 2013 21:58:35 -0800

+1 on this... in the end, it's far better to have all the similar things
acting in the same way. Not only does that help developers, it's going to
help us make improvements in 2.0 and beyond.

On Mon, Jan 21, 2013 at 2:41 AM, Mark Nottingham <[email-masked]> wrote:

> important thing is that the preference itself have clear semantics, which I
> think Roy's change does (especially in concert with changes elsewhere).
> >>>>
> The value is that Accept-* headers all use qvalues the same way, which is
> much less confusing for pretty much everybody, and paves the way for a
> potentially simpler approach in 2.0 (your feelings about having proactive
> negotiation at all noted).
>
> Again, implementations are free to ignore it -- this is merely the
> semantics of the preference, not constraints on how it's followed.
>
>
> --

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 22 Jan 2013 09:30:54 +0000

Just wondering if there is anything to be discussed about 
Accept-Encoding.

There are some cases where it can cause issues for a cache.

for example

GET /something HTTP/1.1
Host: somewhere.com
Accept-Encoding: gzip, deflate

200 OK HTTP/1.1
Content-Encoding: gzip
Cache-Control: max-age=2000000
Vary: Accept-Encoding

GET /something HTTP/1.1
Host: somewhere.com
Accept-Encoding: deflate, gzip

cache MUST NOT select 1st response due to difference in Accept-Encoding 
header between requests.  Even though there's arguably no semantic 
difference.

I've never seen a q value on Accept-Encoding, and the spec talks about 
transforming headers as part of matching, but it only allows adding / 
removal of LWS and combining multiples.  Not re-ordering.

I know there's the pathological case about something returning the 
headers of the request, but it seems like a big price to pay for this.

Maybe all browser vendors should always order Accept-Encoding tokens in 
the same order.

Adrien

------ Original Message ------
From: "James M Snell" <[email-masked]>
To: "Mark Nottingham" <[email-masked]>
Cc: "Roy T. Fielding" <[email-masked]>; "Julian F. Reschke" 
<[email-masked]>; "HTTP Working Group" <[email-masked]>
Sent: 22/01/2013 6:58:35 p.m.
Subject: Re: #428 Accept-Language ordering for identical qvalues
>+1 on this... in the end, it's far better to have all the similar 
>things acting in the same way. Not only does that help developers, it's 
>going to help us make improvements in 2.0 and beyond.
>
>
>> >>>> As such, I think the relevant question is whether this is 
>>specific to A-L, or all A-* that take qvalues. Roy, thoughts?
>> >>>
>>The value is that Accept-* headers all use qvalues the same way, which 
>>is much less confusing for pretty much everybody, and paves the way 
>>for a potentially simpler approach in 2.0 (your feelings about having 
>>proactive negotiation at all noted).
>>

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 22 Jan 2013 23:24:52 +1300

On 22/01/2013 10:30 p.m., Adrien W. de Croy wrote:
> Just wondering if there is anything to be discussed about Accept-Encoding.
> cache MUST NOT select 1st response due to difference in 
> Accept-Encoding header between requests.  Even though there's arguably 
> no semantic difference.

I'm interested in where that "MUST NOT" comes from. My understanding was 
that in the cache role we could take the Content-Encoding header on the 
stored reply as the variant key on the vary response to match against 
the new client Accept-Encoding entries and serve up the HIT if we really 
wanted to.

> I've never seen a q value on Accept-Encoding, and the spec talks 
> about transforming headers as part of matching, but it only allows 
> adding / removal of LWS and combining multiples. Not re-ordering.
> Maybe all browser vendors should always order Accept-Encoding tokens 
> in the same order.
>

But this is a case of a metric where the browser agent *can* usually 
determine ordering automatically.

For example; deflate, gzip, and friends have different CPU consumption 
and bandwidth saving profiles. The browser can get access to the clients 
own CPU and uplink properties to determine whether it needs to proritize 
deflate,gzip or gzip,deflate. Having both indicated shows that either 
format response is acceptible, but the more responses this client gets 
in the first-listed entry the better the user experience.

Amos

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 22 Jan 2013 11:13:37 +0000

13.6 para 3 & 4

"When the cache receives a subsequent request whose Request-URI
specifies one or more cache entries including a Vary header field,
the cache MUST NOT use such a cache entry to construct a response to
the new request unless all of the selecting request-headers present
in the new request match the corresponding stored request-headers in
the original request.

The selecting request-headers from two requests are defined to match
if and only if the selecting request-headers in the first request can
be transformed to the selecting request-headers in the second request
by adding or removing linear white space (LWS) at places where this
is allowed by the corresponding BNF, and/or combining multiple
message-header fields with the same field name following the rules
about message headers in section 4.2. "

Cheers

Adrien

------ Original Message ------
From: "Amos Jeffries" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 22/01/2013 11:24:52 p.m.
Subject: Re: #428 Accept-Language ordering for identical qvalues
>I'm interested in where that "MUST NOT" comes from. My understanding 
>was that in the cache role we could take the Content-Encoding header on 
>the stored reply as the variant key on the vary response to match 
>against the new client Accept-Encoding entries and serve up the HIT if 
>we really wanted to.
>
>and bandwidth saving profiles. The browser can get access to the 
>clients own CPU and uplink properties to determine whether it needs to 
>proritize deflate,gzip or gzip,deflate. Having both indicated shows 
>that either format response is acceptible, but the more responses this 
>client gets in the first-listed entry the better the user experience.
>
>Amos
>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 22 Jan 2013 05:22:09 -0800

...
> 13.6 para 3 & 4

Why are you quoting from RFC2616?  See section 4.3 in p6 and
you will note that it has already been addressed.

....Roy

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 22 Jan 2013 13:30:56 +0000

------ Original Message ------
From: "Roy T. Fielding" <[email-masked]>
>you will note that it has already been addressed.

so it has... sorry about that.  It's been a while since I read that part 
of P6.  Obviously too long!

Thanks

Adrien

---

From: Nicholas Shanks <[email-masked]>
Date: Tue, 22 Jan 2013 13:40:02 +0000

On 17 January 2013 09:14, Julian Reschke <[email-masked]> wrote:
> implemented what the spec says. There's no reason to create an ordered list
> structure when the spec says that an unordered list is sufficient.

I think no implication of randomness should be permitted by the specifications.
They should instead require that a deterministic process be used, and
that, other than requests to services which explicitly exist to
provide random results (e.g. Wikipedia's "Random Page" link), the same
request should generate the same result providing nothing pertinent to
the resource has changed on the server.

Someone, I don't recall who, gave the example of a home page loading
blog posts via AJAX, where the blog posts are available in two
languages. Random selection between the variants, where (q * qs)
values are equal for both languages, or are being ignored, would
result in a mixture of languages being presented in the resultant web
page, and a reload of the page would generate a different mixture. I
agree that such behaviour is undesirable in all cases I can think of,
(excepting the case of a 'randomizer' service URI) and should be
discouraged or better, prohibited in the spec text.

I believe Apache uses file size in it's TCN mechanism to resolve equal
(q*qs) results. It then falls back to a non-exhaustive ordered list in
it's config in the highly unlikely case that two static files varying
by language have the same byte length. Apache seems to try, therefore,
to be wholly deterministic. (Roy may know whether this is accurate for
A-L in Apache.)

At it's most basal extreme, the deterministic process could be an
ASCII value comparison of the language tags' bytes using strcmp()
implemented by the httpd software.

This should apply to all Accept-* headers.

-- 
Nicholas.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 22 Jan 2013 14:53:39 +0100

> values are equal for both languages, or are being ignored, would

Can you please give an example of clients sending these kind of header 
field values?

Clients that care can provide different qvalues, and as a matter of 
fact, they do.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 24 Jan 2013 12:17:28 +1100

So, does anyone have an issue with making ordering significant when there's no qvalue for *all* headers that use qvalues?

Roy, I'm interpreting your answer as "we don't do anything with this information today," but as per below I don't think this stops us from defining it that way.

Regards

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 23 Jan 2013 21:40:16 -0800

> Roy, I'm interpreting your answer as "we don't do anything with this information today," but as per below I don't think this stops us from defining it that way.

Sorry, I wasn't clear.  There is no code out there today that would
correspond to such a change.  I don't like making changes to HTTP
just for the sake of imaginary consistency of definitions.
Making them for the sake of consistency with implementations is fine.

If it is a choice, I'd rather remove the line from Accept-Language
than introduce new (unproven) things to Accept.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 24 Jan 2013 16:56:10 +1100

What about reworking the text to turn it into a note about implementation behavior?

As an aside, looking at those sections again made it apparent we're re-specifying preference selection a number of times in the accept-* headers, each slightly different. 

I realise that they *are* slightly different, but it's not very apparent to readers; it reminds me of the repetition we got rid of in the caching section. 

Would it be possible to explain it once, reference it from each section, and then point out the differences?

Sent from my iPhone

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Jan 2013 09:26:32 +0100

On 2013-01-24 02:17, Mark Nottingham wrote:
> ..

I still do, and I'd prefer we go back to what the spec has been saying 
for well over a decade.

What *real* problem are we solving with this change that justifies 
making current implementations broken?

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 24 Jan 2013 21:37:24 +1300

On 23/01/2013 2:53 a.m., Julian Reschke wrote:
>> On 17 January 2013 09:14, Julian Reschke wrote:

That would be me. Take a note of the Androids below...

> Clients that care can provide different qvalues, and as a matter of 

Uhm. Lets see..... where shall I start ?
  I think an overview of what happens what agents "care" enough to send 
q-values.
  Followed by a small sample of the 513 agents I have on record with no 
q-values at all.
  Judge for yourself which ones are interpreted better as sorted lists.

For starters I would like to say, that to be completely fair the 
majority of agents that I have on record (~54% of unique language:agent 
pair entries) *do* send q-values properly in accordance with the 
specification - and that same 54% of unique agent entries is all 
'voting' for the list to be ordered. I am presenting this sub-set as 
what types of complexity/confusion issues we are introducing when we 
rely solely on q-values to provide ordering semantics in the list.

WebKit ...

cs, en-us; 0.9, de-de; 0.8, ru-ru; 0.7
  - Mozilla/5.0 (X11; U; Linux; cs-CZ) AppleWebKit/532.4 (KHTML, like 
Gecko) Arora/0.10.1 Safari/532.4
  + do we consider that a list with q-values or not?
  + notice also how it is a much more "up to date" version the the 
following...

en;q=1.0, en;q=0.5, zh-cn, zh;q=0.5, en;q=0.5
  - Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaE71-1/300.21.012; 
Profile/MIDP-2.0 Configuration/CLDC-1.1 ) AppleWebKit/413 (KHTML, like 
Gecko) Safari/413
  + Nokia Symbian and SonyEricsson WebKit/ 4XX-532 derived agents across 
the board seem to have 1 primary language set at q=1.0 followed by a 
list of others all sharing q=0.5 or no q-value at all as seen above.

cs-CZ, en-US
  - Mozilla/5.0 (Linux; U; Android 2.2; cs-cz; HTC Legend Build/FRF91) 
AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1
  + Starting with WebKit/533 all the mobiles seem to have moved to this 
2-language model with something then "en-US"

da-DK, en-US
  - Mozilla/5.0 (Linux; U; Android 4.0.4; da-dk; GT-P5110 Build/IMM76D) 
AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30

en-us,en
  - Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; Valve Steam Client; 
) AppleWebKit/534.1 (KHTML, like Gecko) Chrome/6.0.444.0 Safari/534.1

th-TH, en-US
  - Mozilla/5.0 (Linux; U; Android 4.0.3; th-th; A1 Build/IML74K) 
AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

... and then we have iTunes. A massive "WTF?" going out to the iTunes 
developers if anyone is reading.

en;q=1.0,fr;q=1.0,de;q=0.9,ja;q=0.9,nl;q=0.9,it;q=0.9,es;q=0.8,pt;q=0.8,pt-PT;q=0.8,da;q=0.7,fi;q=0.7,nb;q=0.7,sv;q=0.7,ko;q=0.6,zh-Hans;q=0.6,zh-Hant;q=0.6,ru;q
=0.5,pl;q=0.5,tr;q=0.5,uk;q=0.5,ar;q=0.4,hr;q=0.4,cs;q=0.4,el;q=0.3,he;q=0.3,ro;q=0.3,sk;q=0.3,th;q=0.2,id;q=0.2,ms;q=0.2,en-GB;q=0.1,ca;q=0.1,hu;q=0.1,vi;q=0.1
  -  iTunes-iPad/5.1.1 (2; 32GB; dt:74)

en;q=1.0,fr;q=1.0,de;q=0.9,ja;q=0.9,nl;q=0.9,it;q=0.9,es;q=0.8,pt;q=0.8,pt-PT;q=0.8,da;q=0.7,fi;q=0.7,nb;q=0.7,sv;q=0.7,ko;q=0.6,zh-Hans;q=0.6,zh-Hant;q=0.6,ru;q
=0.5,pl;q=0.5,tr;q=0.5,uk;q=0.5,ar;q=0.4,hr;q=0.4,cs;q=0.4,el;q=0.3,he;q=0.3,ro;q=0.3,sk;q=0.3,th;q=0.2,id;q=0.2,ms;q=0.2,en-GB;q=0.1,ca;q=0.1,hu;q=0.1,vi;q=0.1
  - iTunes-iPhone/5.0 (4; 16GB)

en;q=1.0,fr;q=1.0,de;q=0.9,ja;q=0.9,nl;q=0.9,it;q=0.9,es;q=0.8,pt;q=0.8,pt-PT;q=0.8,da;q=0.7,fi;q=0.7,nb;q=0.7,sv;q=0.7,ko;q=0.6,zh-Hans;q=0.6,zh-Hant;q=0.6,ru;q
=0.5,pl;q=0.5,tr;q=0.5,uk;q=0.5,ar;q=0.4,hr;q=0.4,cs;q=0.4,el;q=0.3,he;q=0.3,ro;q=0.3,sk;q=0.3,th;q=0.2,id;q=0.2,ms;q=0.2,en-GB;q=0.1,ca;q=0.1,hu;q=0.1,vi;q=0.1
  - iTunes-iPhone/4.3.5 (3; 16GB)

... spiders are mostly doing a remarkably good job. At least it looks 
that way until the q-values get involved.

ja-JP,ja
  - Baiduspider+(+http://www.baidu.jp/spider/)

ja,en
  - Mozilla/5.0 (compatible; Steeler/3.5; 
http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)

  ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
  - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  + q=0.8 - Ukranian or Belarusian ?

  en-us,en-gb,en;q=0.99,*;q=0.01
  - TosCrawler/Nutch-1.5.1 
(http://www.toshiba.co.jp/rdc/about/crawl_info.htm; <dc-crawler at ml 
dot toshiba dot co dot jp>)
  + q=1.0 - English US or British? (no so much trouble for humans but 
for a search engine it might cause indexing trouble).

Don't know if you would call some of the major search engine bots 
popular or even "fixable problem"?

I host a translation server so it is likely that these below are from 
actual users working on text translation. You know, the kind of person 
who *really* objects to getting a randomly-wrong language displayed. 
Also these people are highly knowledgeable about language codes and what 
they mean, so if they entered these manually it was for a specific 
reason according to how they or their tools author interpreted the 
Accept-Language specs.

Note how the first entries have no q-value and are *sorted* as if they 
were q=1.0, which is what the spec says to do when no q-value is 
supplied remember ... Treat it as q=1.0.

  ca,ca-ES,es-es;q=0.9,es;q=0.9,en-US;q=0.9,en;q=0.9,es-419;q=0.8,ca-AD;q=0.8,en-gb;q=0.8,de-de;q=0.7,de;q=0.7,ca-CA;q=0.7,cs-CZ;q=0.6,cs;q=0.6,it-it;q=0.6,it;q=0.6,es-CL;q=0.5,en-au;q=0.5,fr-FR;q=0.5,fr;q=0.4,ru-ru;q=0.4,ru;q=0.4,es-x-mtfrom-en;q=0.4,es-ar;q=0.3,ja-JP;q=0.3,ja;q=0.3,pt-PT;q=0.2,pt;q=0.2,do-es;q=0.2,do;q=0.1,es-x-mtfrom-it;q=0.1,nl-nl;q=0.1,nl;q=0.1,en-en;q=0.0
  - Mozilla/5.0 (X11; Linux x86_64; rv:10.0.6) Gecko/20100101 
Firefox/10.0.6 Iceweasel/10.0.6
  + q=1.0 - Catalan Valencian or Spanish Catalan?
  + q=0.9 - Spanish or English? Generic or nationalized grammar?
  + q=0.8 - Spanish or Catalan Andoran or English or German or Catalan 
Valencian?
  + q=0.6 - want to try again with German or Catalan Generic?
  + q=0.5 - Spanish or Australian English or French?
  + q=0.4 - what about French or Russian?
  + q=0.3 - Argentine Spanish or Japanese?
  + q=0.1 - Spanish or Dutch?

de,de-DE,en-US;q=0.9,en;q=0.9,nl-nl;q=0.8,nl;q=0.8,en-gb;q=0.8,ro-RO;q=0.7,ro;q=0.7,fr-FR;q=0.6,fr;q=0.6,de-DE-1901;q=0.5,tr-TR;q=0.5,tr;q=0.5,pl-PL;q=0.4,pl;q=0.4,nl-NL;q=0.3,de-de;q=0.3,de-at;q=0.3,en-us;q=0.2,pl-pl;q=0.2,de;q=0.1,en-us;q=0.1,en;q=0.0
  - Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.15) 
Gecko/20110303 Firefox/3.6.15
  + q=0.9 - English Generic or US-centric ?
  + q=0.8 - Dutch or English?
  + q=0.5 - German or Turkish?
  + q=0.3 - Dutch or German?
  + q=0.2 - English or Polish?
  + q=0.1 - German or English?
  + q=0.1 - oops Cancel that q=0.9 US English option.
  + q=0.0 - oops Cancel that q=0.9 generic English option.

  + I skip q=1.0 (none), q=0.7, q=0.6 and q=0.4 because these, while 
being alternatives sharing a q-value, are in the ISO definitions 
semantically equivalent aliases for the same language. So any selection 
algorithm other than if-it-exists is a waste of CPU cycles but not a 
user problem.

We have only a few agents sending "q=1.0", by my interpretation of 2616 
these few are the "correct" users of q-values when q=1:

  en;q=1.0
  - w3m/0.5.2
  also the YoudaoBot spider with a mix of language codes. It seems to be 
trying to fetch different translations specifically for some reason.

en-us;q=1.0, es-ve;q=0.5
  - Mozilla/4.1 (U; BREW 3.1.5; en-US; Teleca/Q05A/INT)
  - NetFront/3.5.1 (BREW 5.0.1.2; U; en-us; LG; NetFront/3.5.1/AMB) 
Sprint LN510 MMP/2.0 Profile/MIDP-2.1 Configuration/CLDC-1.1
  there are a few other variations of this "NetFront/" framework from 
Samsung and LG mobile devices.

The rest (~50 unique agent:language pairs) using q=1.0 somewhere in the 
A-L header are all WebKit derived agents. We already covered how well 
they handle q-values.

Still a fair few browser few browser agents around with no q-values.

zh-cn,zh-tw
  - Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.0.1) 
Gecko/2008070208 Firefox/3.0.1

zh-cn,zh-tw
- Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.3) 
Gecko/20100401 Firefox/3.6.3

en,zh,fr,de,it
  - Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.20) Gecko/20081217 
Firefox/2.0.0.20 Novarra-Vision/8.0

ru, en-US, en
  - Mozilla/5.0 (compatible; Konqueror/4.4; Linux) KHTML/4.4.5 (like Gecko)

ru, uk, en-US, en
- Mozilla/5.0 (compatible; Konqueror/4.4; FreeBSD) KHTML/4.4.3 (like Gecko)

HTH
Amos

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 24 Jan 2013 21:54:36 +1300

On 24/01/2013 9:26 p.m., Julian Reschke wrote:
>> So, does anyone have an issue with making ordering significant when 
>> there's no qvalue for *all* headers that use qvalues?
>> ..
>

Problem 1) a lot of agents (~57% by unique U-A string) are using 
q-values to specify ordering where the spec says "unordered".

Problem 2) a majority of the remaining agents appear to be treating the 
field-value as an ordered list of preferences even without q-values.

Problem 3)  ~1% of agents are incorectly implementing q-values. (see my 
earlier post responding to your request for examples).

Problem 4) q-values being mandatory when preference order is wanted adds 
complexity on both ends of the transaction, causing unnecessary CPU 
burden on the recipient. Misunderstandings and a host of needless 
mistakes by end-users and developers alike. (again see my earlier post 
for examples).

Problem 5) On the Accept-Language header the bandwidth required to 
transmit q-values inflates the header size by at least 55% (ISO code: 
2-5 bytes, q-v component: 6 bytes).

Amos

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Jan 2013 10:50:33 +0100

> q-values to specify ordering where the spec says "unordered".

Not sure what you're trying to say here. If they send q-values there is 
no doubt about the semantics, right?

Also, counting unique UA strings generates a totally distorted statistic.

> field-value as an ordered list of preferences even without q-values.

Recipients, I assume? How is that a problem? They choose one plausible 
interpretation where the spec doesn't define one.

> earlier post responding to your request for examples).

Are these agents widely used? Can they be fixed? Did you report bugs 
against them?

> mistakes by end-users and developers alike. (again see my earlier post

That is true, but we can't remove q values at this point. Note we are 
discussing HTTP/1.1.

> 2-5 bytes, q-v component: 6 bytes).

For that field value, yes.

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 24 Jan 2013 23:17:45 +1300

On 24/01/2013 10:50 p.m., Julian Reschke wrote:
> Not sure what you're trying to say here. If they send q-values there 
> is no doubt about the semantics, right?
>

Somewhat yes. However IME it distorts the numbers to be more inline with 
total traffic load %'s each agent places on the network overall.

> interpretation where the spec doesn't define one.

Senders.

The pattern is clearly a country-specific code, a generic language code 
and possibly followed by a few similar languages in decreasing order of 
similarity, possibly followed by wildcards.
The pattern is strong and almost identical for both agents sending 
ordered q-values and agents sending no q-values at all.
The errors all appear to be at the interface between mechanisms with 
confusion evident when they are mixed in one request header.

Problem is that the original spec wording you are arguing to keep says 
it is a wrong interpretation. One which leads to all these being problems.

> That is true, but we can't remove q values at this point. Note we are 

But we can make the primary need for them decrease by making the header 
ordered by default.

> For that field value, yes.

That is the field-value we are discussing though.

Amos

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Jan 2013 11:29:53 +0100

> total traffic load %'s each agent places on the network overall.

I don't understand that sentence.

>> interpretation where the spec doesn't define one.

Then you need to explain the problem better.

> it is a wrong interpretation. One which leads to all these being problems.

I still don't understand what the problem is. Can you give a concrete 
example?

> But we can make the primary need for them decrease by making the header

Recipients will still need to implement qvalues, so I don't see a big 
gain here.

> That is the field-value we are discussing though.

Understood, but making A-L more compact will not fix HTTP/1.1's overhead 
significantly.

I think our priorities should be:

1) Get HTTP/1.1 out of the door. To get that done, it would be helpful 
not to make any new changes after WGLC (and this is the case here).

Then:

2) Think about compact header field serialization in HTTP/2.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Jan 2013 15:50:19 +0100

> than introduce new (unproven) things to Accept.

+1

In the meantime I had another look at RFC 4647:

> 2.3. The Language Priority List
>
>
>    A user's language preferences will often need to specify more than
>    one language range, and thus users often need to specify a
>    prioritized list of language ranges in order to best reflect their
>    language preferences.  This is especially true for speakers of
>    minority languages.  A speaker of Breton in France, for example, can
>    specify "br" followed by "fr", meaning that if Breton is available,
>    it is preferred, but otherwise French is the best alternative.  It
>    can get more complex: a different user might want to fall back from
>    Skolt Sami to Northern Sami to Finnish.
>
>    A "language priority list" is a prioritized or weighted list of
>    language ranges.  One well-known example of such a list is the
>    "Accept-Language" header defined in RFC 2616 [RFC2616] (see Section
>    14.4) and RFC 3282 [RFC3282].
>
>    The various matching operations described in this document include
>    considerations for using a language priority list.  This document
>    does not define the syntax for a language priority list; defining
>    such a syntax is the responsibility of the protocol, application, or
>    specification that uses it.  When given as examples in this document,
>    language priority lists will be shown as a quoted sequence of ranges
>    separated by commas, like this: "en, fr, zh-Hant" (which is read
>    "English before French before Chinese as written in the Traditional
>    script").
>
>    A simple list of ranges is considered to be in descending order of
>    priority.  Other language priority lists provide "quality weights"
>    for the language ranges in order to specify the relative priority of
>    the user's language preferences.  An example of this is the use of
>    "q" values in the syntax of the "Accept-Language" header (defined in
>    [RFC2616], Section 14.4, and [RFC3282]).

So in fact what the spec used to say is indeed consistent with 4647; 
there is no consistency problem that needs to be solved.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 25 Jan 2013 10:37:34 +1100

Removing the text does seem like the most expedient path forward. 

That said, I don't find it particularly satisfying; our job is to improve interop, and when there are latent semantics that aren't documented, we have to consider whether we're doing it well.

I propose:

"""
Note that some recipients treat language tags that have the same quality values (including when they are both missing) to be listed in descending order of priority. However, this behaviour cannot be relied upon, and if their relative priority is important, it ought to be communicated by using different quality values.
"""

... because I think it best captures where we're at.

Roy and Julian? And, especially, anyone else?

On 25/01/2013, at 1:50 AM, Julian Reschke <[email-masked]> wrote:

> So in fact what the spec used to say is indeed consistent with 4647; there is no consistency problem that needs to be solved.
> 
> Best regards, Julian
> 
>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 25 Jan 2013 14:31:07 +0900

> ... because I think it best captures where we're at.

Maybe I'm getting this wrong, but it sounds to me that Julian is 
insisting that it's okay to send arbitrary replies (e.g. once French, 
once English at random) if there are no q-values. It has been very 
clearly explained that this is highly confusing (in other words, bad for 
interoperability). Even if the current spec allows this, it would be 
good to have some text in the new spec that says that's a bad idea.

Otherwise, I'm fine with the above Note, except for a small nit:
Please change "including when they are both missing" to "including when 
they are missing", because there may be more than two missing (or equal) 
q-values.

Regards,   Martin.

> On 25/01/2013, at 1:50 AM, Julian Reschke<[email-masked]>  wrote:
>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Jan 2013 07:14:03 +0100

> Roy and Julian? And, especially, anyone else?

That works for me.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Jan 2013 07:16:27 +0100

On 2013-01-25 06:31, "Martin J. Dürst" wrote:
>> That said, I don't find it particularly satisfying; our job is to
>> improve interop, and when there are latent semantics that aren't
>> documented, we have to consider whether we're doing it well.
>>
>> I propose:
>>
>> """
>> Note that some recipients treat language tags that have the same
>> quality values (including when they are both missing) to be listed in
>> descending order of priority. However, this behaviour cannot be relied
>> upon, and if their relative priority is important, it ought to be
>> communicated by using different quality values.
>> """
>>

It is, according to the spec. If it hurts, don't do it (thus add qvalues).

> good to have some text in the new spec that says that's a bad idea.

We could also say that leaving the choice to the server might lead to 
different languages being picked in subsequent requests.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Jan 2013 14:02:01 +0100

>> they are missing", because there may be more than two missing (or equal)

Proposed change: 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/428/428.diff>.

This removes the new text about ordering, and adds the note below:

 >       Note: Some recipients treat language tags that have the same
 >       quality values (including when they are missing) to be listed in
 >       descending order of priority.  However, this behavior cannot be
 >       relied upon, and if their relative priority is important -- such
 >       as for consistent results for a sequence of requests -- it ought
 >       to be communicated by using different quality values.

Feedback appreciated, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Jan 2013 14:55:48 +0100

> Feedback appreciated, Julian

In the meantime, Roy resolved this in 
<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2163>, which works 
for me as well:

"Note that some recipients treat the order in which language tags are 
listed as an indication of descending priority, particularly for tags 
that are assigned equal quality values (no value is the same as q=1). 
However, this behavior cannot be relied upon. For consistency and to 
maximize interoperability, many user agents assign each language tag a 
unique quality value while also listing them in order of decreasing 
quality. Additional discussion of language priority lists can be found 
in Section 2.3 of [RFC4647]."

Best regards, Julian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 28 Jan 2013 11:51:01 +0900

> in Section 2.3 of [RFC4647]."

Sorry, but I'm not yet happy with this. It doesn't mention the random 
return problem at all, and puts all responsibility on the client.

So I propose adding the following:

Note that it would be allowed for servers to return a version at random 
if they receive language tags with equal quality values. However, this 
can be very confusing for human users. A more deterministic behavior, 
e.g. treating the order in which language tags are listed as an 
indication of descending priority for tags that are assigned equal 
quality values, is preferable.

Regards,   Martin.

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 28 Jan 2013 12:12:44 +0900

-1 I don't think that's necessary; it already states that ordering is a good idea. 

On 28/01/2013, at 11:51 AM, "Martin J. Dürst" <[email-masked]> wrote:

> Sorry, but I'm not yet happy with this. It doesn't mention the random return problem at all, and puts all responsibility on the client.
> 
> Note that it would be allowed for servers to return a version at random if they receive language tags with equal quality values. However, this can be very confusing for human users. A more deterministic behavior, e.g. treating the order in which language tags are listed as an indication of descending priority for tags that are assigned equal quality values, is preferable.
> 
> Regards,   Martin.