Subject: Re: Proposed resolution for Issue 13 (language tags)

From: Julian Reschke <[email-masked]>
Date: Sat, 18 Jul 2009 13:57:43 +0200

Julian Reschke wrote:
> 
> OK,
> 
> thanks for all the feedback so far. I (hopefully) have addressed many of 
> the issues; here's the new proposed text for 3.5:
> ...

We stopped to discuss this 
(<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/13>) 15 months ago; 
in the meantime RFC4646bis went through many revisions, and now is 
approved and in the RFC Editor queue.

I have updated the proposed change for HTTPbis Part 3 accordingly; see 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/13/i13.4.diff>.

The full text now would read:

--
2.4.  Language Tags

    A language tag, as defined in [RFC4646bis], identifies a natural
    language spoken, written, or otherwise conveyed by human beings for
    communication of information to other human beings.  Computer
    languages are explicitly excluded.  HTTP uses language tags within
    the Accept-Language and Content-Language fields.

    In summary, a language tag is composed of one or more parts: A
    primary language subtag followed by a possibly empty series of
    subtags:

      language-tag = <Language-Tag, defined in [RFC4646bis], Section 2.1>

    White space is not allowed within the tag and all tags are case-
    insensitive.  The name space of language subtags is administered by
    the IANA (see
    <http://www.iana.org/assignments/language-subtag-registry>).

    Example tags include:

      en, en-US, es-419, az-Arab, x-pig-latin, man-Nkoo-GN

    See [RFC4646bis] for further information.
--

I understand that back in April 2008 we still discussed various details, 
such as

1) The exact wording of the "summary",
2) whether we're referring the right ABNF production (does it need to be 
  "obs-language-tag" instead, or both), and
3) the examples

(the discussion is archived around 
<http://lists.w3.org/Archives/Public/ietf-http-wg/2008AprJun/0210.html>)

I'd really like to close this one finally, so feedback from the language 
tag experts would be appreciated.

BR, Julian

---

From: "Phillips, Addison" <[email-masked]>
Date: Sat, 18 Jul 2009 08:45:07 -0700

Hello Julian,

> 1) The exact wording of the "summary",

I think your wording is generally good. There are a couple of minor points to make.

> HTTP uses language tags within the Accept-Language and Content-Language fields.

Not quite. It uses language tags in Content-Language. Accept-Language uses language *ranges*, which are currently defined by RFC 4647, although I would tend not to change this text here to note that fact. Section 5.4 can cover that.

> 2) whether we're referring the right ABNF production (does it need
> to be "obs-language-tag" instead, or both), and

I don't think it should be both. That would be confusing and lead to interoperability issues. 

I think that, ideally, you would use the new production rather than obs-language-tag. While obs-language-tag is more permissive, language tags that match it (but not language-tag) have never been valid. And many of the most common invalid values happen to match both productions.

> 3) the examples

There is nothing wrong with the set of examples you have, although "x-pig-latin" is suspect :-) and the list is somewhat eclectic. The list in RFC 2616 was:

  en, en-US, en-cockney, i-cherokee, x-pig-latin

This list had the advantage of being somewhat obvious to English speakers without additional annotation. (Note that the Cherokee tag was never actually valid!!) Perhaps some values from 4646bis Appendix B would be suitable. I suggest a carefully constructed list, such as:

====
   Example tags include:

   en (English)
   en-US (English, United States)
   en-US-x-pig-latin (English, United States, private use subtags)
   hy-Latn-IT-arevela (Armenian, Latin script, Italy, eastern variant)
   es-419 (Spanish, Latin America)
====

One additional issue related to language tags (you don't appear to be tracking it separately). I don't think that the current incarnation of section 5.4 (Accept-Language) is quite right. In particular, the Basic Filtering algorithm is made normative as the language negotiation strategy. I tend to find the Lookup algorithm a better/more common choice, personally, and others might make different choices, depending on the application. I think this text should be made more dependent on the text in 4647, rather than trying to recreate it in a shorter form. If HTTP-WG feels that an algorithm must be made normative, then my personal opinion is that it should be Lookup, not Filtering.

Addison

PS> I have blind-copied the [email-masked] list on this message because that group has an interest in this topic but I don't want to expand the cross-posting of the thread.

Addison Phillips
Globalization Architect -- Lab126
Chair -- W3C Internationalization WG

Internationalization is not a feature.
It is an architecture.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Julian Reschke
> Sent: Saturday, July 18, 2009 4:58 AM
> To: HTTP Working Group; LTRU Working Group
> Subject: Re: [Ltru] Proposed resolution for Issue 13 (language tags)
> 
> Julian Reschke wrote:
> >
> > OK,
> >
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ltru

---

From: Julian Reschke <[email-masked]>
Date: Sat, 18 Jul 2009 18:18:10 +0200

> One additional issue related to language tags (you don't appear to be tracking it separately). I don't think that the current incarnation of section 5.4 (Accept-Language) is quite right. In particu…

Hi Addison,

actually we *do* track this separately; the issue is 
<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/113>, and we thought 
we were done with it almost a year ago. See discussion around 
<http://lists.w3.org/Archives/Public/ietf-http-wg/2008JulSep/0214.html>.

The intention was to normatively refer to that matching algorithm that 
actually is equivalent to what RFC2616 used to define (remember, we're 
not changing the protocol here). Did we pick the wrong one?

BR, Julian

---

From: John Cowan <[email-masked]>
Date: Sat, 18 Jul 2009 13:59:18 -0400

> not changing the protocol here). Did we pick the wrong one?

No, basic filtering is the RFC 2616 algorithm all right.  You might
consider allowing HTTP servers to do lookup if basic filtering
produces no results: Apache already does this.

Is there some reason why you aren't referring to BCP 47?

-- 
John Cowan  [email-masked]    http://ccil.org/~cowan
No man is an island, entire of itself; every man is a piece of the
continent, a part of the main.  If a clod be washed away by the sea,
Europe is the less, as well as if a promontory were, as well as if a
manor of thy friends or of thine own were: any man's death diminishes me,
because I am involved in mankind, and therefore never send to know for
whom the bell tolls; it tolls for thee.  --John Donne

---

From: Julian Reschke <[email-masked]>
Date: Sat, 18 Jul 2009 20:17:16 +0200

> Is there some reason why you aren't referring to BCP 47?

The spec does refer to BCP 47: 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p3-payload-07.html#RFC4647>.

BR, Julian

---

From: "Phillips, Addison" <[email-masked]>
Date: Sat, 18 Jul 2009 12:58:23 -0700

> produces no results: Apache already does this.

Basic filtering is "pretty much" the 2616 algorithm--in fact, I think we went out of our way to make them compatible. But I think our understanding of language negotiation has evolved somewhat since 1999. There are a number of recommendations and statements in 2616 that suggest that both servers and client applications might tailor their matching behavior, etc.

I think it would be useful and wise to allow for both sorts of behavior. Many application servers use lookup (which is the locale/resource loading protocol for Windows, Java, etc.)

> Is there some reason why you aren't referring to BCP 47?

I think they are trying to be specific about their reference to avoid having <Language-Tag> change underneath them. 

> <http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p3-payload-07.html#RFC4647>.

Yes, but not in the same way as it refers to BCP 97.

Addison

Addison Phillips

---

From: "Phillips, Addison" <[email-masked]>
Date: Sat, 18 Jul 2009 13:04:40 -0700

Hi,
> wg/2008JulSep/0214.html>.

Thanks for this. I meant as an open issue. I scanned the list but didn't see this one.

> not changing the protocol here). Did we pick the wrong one?

I responded in my response to John Cowan. I think I would want to reopen this issue. Compatibility is a Good Thing, but as I said, I think language negotiation has evolved somewhat and you could incorporate more of 4647 rather than strictly requiring Basic Filtering.

Addison

Addison Phillips

---

From: Julian Reschke <[email-masked]>
Date: Sat, 18 Jul 2009 22:11:53 +0200

> I think they are trying to be specific about their reference to avoid having <Language-Tag> change underneath them. 

Exactly.

We need a precise reference to an ABNF production and definition here. 
Citing the BCP instead of the concrete document would make it impossible 
to refer to a specific section (or even the name of a production).

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 19 Jul 2009 14:59:35 +0200

> produces no results: Apache already does this.

Now tracked as <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/181>.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 19 Jul 2009 15:13:13 +0200

>    es-419 (Spanish, Latin America)

OK, I have updated the text as proposed (with 
<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/613>), but will 
leave the issue (<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/13>) 
open for more review, and potentially fine-tuning the examples.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 27 Jul 2009 13:46:30 +0200

> I responded in my response to John Cowan. I think I would want to reopen this issue. Compatibility is a Good Thing, but as I said, I think language negotiation has evolved somewhat and you could in…

Addison,

so I had a look at RFC 4647 and I'm totally open to allow more than 
Basic Filtering; but I'm not sure about what exactly we want to say...

- require to try Basic Filtering First, but allow to fall back to Lookup 
when nothing is returned?

- just stay point to Section 3.1 and leave it to the implementer?

Best regards, Julian

---

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 27 Jul 2009 09:17:33 -0700

Hello Julian,
> - require to try Basic Filtering First, but allow to fall back to
> Lookup when nothing is returned?

I think this would actually be a Bad Thing. Filtering and Lookup work very differently.

> - just stay point to Section 3.1 and leave it to the implementer?

I think that's probably a the best choice. There are different purposes to the matching schemes. 

I tend to think that HTTP's requirements are most like what the Lookup algorithm provides. That is, you can (and must) return exactly one result for a given request. It also works most like the resource-and-localization mechanisms in programming languages and (some) Web infrastructure. While filtering can return "exactly one" result, it isn't always clear what you will get and it works less well when a wide variety of content is available with closely related language tags.

On the other hand, there are implementations based on filtering (Basic Filtering *is* the algorithm in 2616) and these can be made to work. There are many applications where filtering is a good choice (this is especially true, for example, when aggregating content). So, although my personal preference would be to require Lookup, I don't think that choice can be the only one permitted.

I would suggest some text, but want to see other's reactions first.

Best Regards,

Addison

Addison Phillips

---

From: John Cowan <[email-masked]>
Date: Mon, 27 Jul 2009 13:32:22 -0400

Phillips, Addison scripsit:

> I tend to think that HTTP's requirements are most like what the
> Lookup algorithm provides. That is, you can (and must) return
> exactly one result for a given request. 

Actually, no; that's an oversimplification of HTTP.  The whole point of
the error code 300 Multiple Choices is that the server has determined
that more than one resource satisfies the prescriptions of the Accept-*
headers as to the media types, charsets, content-encodings, and languages
specified as allowable by the client.  The body in that case will contain
hyperlinks to the original resources.

While this mechanism is not widely used in the Web today, it does serve
the intended purpose: servers can make different variants of pages
available, and clients see only the ones that they specify as meeting
their minimum requirements.

The question is, then, what to do if there is no resource that specifies
those minimum requirements.  Apache in this case applies the lookup algorithm
to loosen the client requirement in hopes of finding something usable.

-- 
Values of beeta will give rise to dom!          John Cowan
(5th/6th edition 'mv' said this if you tried    http://www.ccil.org/~cowan
to rename '.' or '..' entries; see              [email-masked]
http://cm.bell-labs.com/cm/cs/who/dmr/odd.html)

---

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 27 Jul 2009 11:15:13 -0700

> Actually, no; that's an oversimplification of HTTP.  

Well... what I was trying to say was "HTTP is most commonly used in a way that I think works better with Lookup". That is, most typically, HTTP is used to return a single resource at the end of a URI. I'm well aware that there are other cases, including the 300 case, which is why I went on to say the rest of what I said :-). 

This is also why I didn't suggest that we merely replace filtering with lookup. I do think that the most common use of HTTP involves returning a single information object for a single request and, in the case where language negotiation is done at all, these typically fit Lookup more closely than Basic Filtering. A significant subset fit Basic Filtering better. And a different significant subset happen to use Basic Filtering (even if Lookup would have been a better choice) simply because 2616 said to.

> to loosen the client requirement in hopes of finding something

Yes, and this is neither "Basic Filtering" nor "Lookup". Similarly, implementations sometimes make use of outside information (Mark Davis's example of Breton falling back to French, for example). And so forth. The problem, as I see it, is that over-specificity in HTTPbis might lead implementers astray and we really need a more comprehensive treatment of language negotiation so that folks can choose wisely.

Addison