Subject: [Ltru] matching: basic range considers extentions and private use

From: "Addison Phillips" <[email-masked]>
Date: Wed, 28 Sep 2005 10:28:31 -0700

I think that the numbered list in Section 2.1 about handling extensions and private use is of dubious value in basic range matching. The positioning of these subtags at the end of tags means that they don't interfere with prefix matching as described in this section. I think we should move this list to Section 2.2 (Extended Language Range) and substitute this instead:

--
When working with tags and ranges, note that extensions and most private use subtags are generally orthogonal to language tag fallback and users SHOULD avoid using these subtags in language ranges, since they will often interfere with the selection of available language content. Since these subtags are always at the end of the sequence of subtags, they don't normally interfere with the use of prefixes for matching in the schemes described below.
--

Addison

Addison P. Phillips
Globalization Architect, Quest Software
http://www.quest.com

Chair, W3C Internationalization Core Working Group
http://www.w3.org/International

Internationalization is not a feature.
It is an architecture. 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "John.Cowan" <[email-masked]>
Date: Wed, 28 Sep 2005 14:14:23 -0400

Addison Phillips scripsit:

> I think that the numbered list in Section 2.1 about handling extensions
> and private use is of dubious value in basic range matching. The
> positioning of these subtags at the end of tags means that they don't
> interfere with prefix matching as described in this section. I think
> we should move this list to Section 2.2 (Extended Language Range)
> and substitute this instead:

+1
-- 
Verbogeny is one of the pleasurettes    John Cowan <[email-masked]>
of a creatific thinkerizer.             http://www.reutershealth.com
   -- Peter da Silva                    http://www.ccil.org/~cowan

---

From: Mark Davis <[email-masked]>
Date: Wed, 28 Sep 2005 12:01:39 -0700

http://www.inter-locale.com/ID/draft-ietf-ltru-matching-04.html#length

A. Parts of this section are holdovers from the language tag document. 
In particular, this section should not repeat the discussions of 
limitations on length in storage. Thus the following should be removed:

In practice, most tags do not require additional subtags or 
substantially more characters. Additional subtags sometimes add useful 
distinguishing information, but extraneous subtags interfere with the 
meaning, understanding, and especially matching of language tags. Since 
language tags or ranges MAY be truncated by an application or protocol 
that limits storage, when choosing language tags or ranges users and 
applications SHOULD avoid adding subtags that add no distinguishing 
value. In particular, users and implementations SHOULD follow the 
'Prefix' and 'Suppress-Script' fields in the registry (defined in 
Section 3.6 of [draft-registry] (Phillips, A., Ed. and M. Davis, Ed., 
“Tags for the Identification of Languages (work in progress),” 
August 2005.) 
<http://www.inter-locale.com/ID/draft-ietf-ltru-matching-04.html#draft-registry>): 
these fields provide guidance on when specific additional subtags SHOULD 
(and SHOULD NOT) be used.

[then down 2 paragraphs]

Applications or protocols which have to truncate a tag MUST do so by 
progressively removing subtags along with their preceding "-" from the 
right side of the language tag until the tag is short enough for the 
given buffer. If the resulting tag ends with a single-character subtag, 
that subtag and its preceding "-" MUST also be removed. For example:

------------------------------------------------------------------------

Tag to truncate: zh-Hant-CN-variant1-a-extend1-x-wadegile-private1
1. zh-Hant-CN-variant1-a-extend1-x-wadegile
2. zh-Hant-CN-variant1-a-extend1
3. zh-Hant-CN-variant1
4. zh-Hant-CN
5. zh-Hant
6. zh

* Figure 6: Example of Tag Truncation *
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Mark Davis <[email-masked]>
Date: Wed, 28 Sep 2005 12:31:40 -0700

The main issue I have with the document as it stands is that it
neglects a very important case. The general case of language matching
involves having a *list* of language ranges. You take a list like
"en-US en fr it" and use it for either matching or lookup. So I would
suggest adding the following text. (It probably needs fleshing out a
bit, but this is a start.).

2.3 Language Range Lists

A language range list is a space-delimited list of zero or more
language ranges. An extended language range list contains extended
language ranges; a basic language range list is limited to containing
basic language ranges. In matching, anything that matches any element
of the list according to the above options is included. (With distance
metrics, the shortest distance from any element in the list is the one
used for the overall distance).

In lookup, the list is examined from first to last for a match,
according to the above options. For example, starting with the list
range "zh-Hant-CN en-US fr", and using Lookup with truncation as in
2.1.1, the lookup would progressively search for content as shown
below:

List Range to match: zh-Hant-CN en-US fr
1. zh-Hant-CN
2. zh-Hant
3. zh
4. en-US
5. en
6. fr
7. (default content or the empty tag)

2.3 Meaning of Language Tags and Ranges [RENUMBER to 2.4...]

---

From: "John.Cowan" <[email-masked]>
Date: Wed, 28 Sep 2005 17:19:31 -0400

+1

-- 
John Cowan  www.ccil.org/~cowan  www.reutershealth.com  [email-masked]
SAXParserFactory [is] a hideous, evil monstrosity of a class that should
be hung, shot, beheaded, drawn and quartered, burned at the stake,
buried in unconsecrated ground, dug up, cremated, and the ashes tossed
in the Tiber while the complete cast of Wicked sings "Ding dong, the
witch is dead."  --Elliotte Rusty Harold on xml-dev

---

From: Martin Duerst <[email-masked]>
Date: Thu, 29 Sep 2005 16:23:33 +0900

as a chair: Please use 'matching:' in the subject of all mails
concerning the matching draft. Thanks!
[I have done this in this subject, and also made the subject
to be closer to the actual topic.]

At 04:31 05/09/29, Mark Davis wrote:
>The main issue I have with the document as it stands is that it neglects a 
>very important case. The general case of language matching involves having 
>a *list* of language ranges. You take a list like "en-US en fr it" and use 
>it for either matching or lookup. So I would suggest adding the following 
>text. (It probably needs fleshing out a bit, but this is a start.).

without my chair hat:

This is already defined, in quite a bit more detail and
with some slight differences (i.e. allowing a q value) in
the HTTP 1.1 spec. See e.g.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4

Do we need anything more?

Please also note that according to the HTTP rules,
zh-Hant-CN

in the 'language range list' does NOT match zh or zh-Hant.
(whereas the other way round, a document of zh-Hant-CN IS
matched by a request for zh or zh-Hant).
This was done to avoid false positives. In the example belows,
zh is in third place, but if the user explicitly specifies
zh-Hant-CN, this probably means (or should be reserved to
mean) that she doesn't read simplified,... In that case,
it seems that en-US or fr should have higher priority,
because the user actually understands that.

Regards,    Martin.
>2.3 Language Range Lists
>
>A language range list is a space-delimited list of zero or more language 
>ranges. An extended language range list contains extended language ranges; 
>a basic language range list is limited to containing basic language 
>ranges. In matching, anything that matches any element of the list 
>according to the above options is included. (With distance metrics, the 
>shortest distance from any element in the list is the one used for the 
>overall distance).
>
>In lookup, the list is examined from first to last for a match, according 
>to the above options. For example, starting with the list range 
>"zh-Hant-CN en-US fr", and using Lookup with truncation as in 2.1.1, the 
>lookup would progressively search for content as shown below:
>
>
>2.3ツ Meaning of Language Tags and Ranges [RENUMBER to 2.4...]
>

---

From: "Jukka K. Korpela" <[email-masked]>
Date: Thu, 29 Sep 2005 15:03:10 +0300 (EEST)

On Thu, 29 Sep 2005, Martin Duerst wrote:

> the HTTP 1.1 spec.

Sorry if I'm missing something obvious, but isn't the scope of 
applicability of the HTTP 1.1 spec the HTTP protocol only?
It deals with a specific mechanism, content negotiation, which lets an 
HTTP server and a client negotiate on the choice of a language version. 
(This has, sadly enough, been used rather on multilingual sites.)

So it seems to be rather specific, rather than generally about language 
codes and ranges, which have many potential applications outside the HTTP 
protocol too. (The data might be carried over HTTP, but that's a different 
issue.)

If the HTTP definitions on language negotiation are regarded as suitable 
for more general use, this should be based on a conscious and explicit 
generalization.

-- 
Jukka "Yucca" Korpela, http://www.cs.tut.fi/~jkorpela/

---

From: Mark Davis <[email-masked]>
Date: Thu, 29 Sep 2005 06:15:21 -0700

It should also refer to the Accept Language usage. I had meant to do 
that, but forgot. However, it cannot be replaced by Accept-Lange, 
because Accept-Language doesn't extend to the different types of 
matching and lookup used in this document.

There are indeed different kinds of lookup. Matching zh in a pattern 
against zh-Hant-CN can sometimes be as much of a "false positive" as 
matching zh-Hant-CN in a pattern to zh.

Mark

Note: To my knowledge, the q parameter is only ever used to get an 
ordering of the list, which is then simply walked through. And of course 
the phrase "If no Accept-Language header is present in the request, the 
server SHOULD assume that all languages are equally acceptable." is, in 
practice, pretty useless. Implementations that I know of will still do 
some kind of fallback, because after seeing the list "en-US fr", and not 
finding en-US(-.*)? or fr(-.*), it is a bit absurd to conclude that hi 
or haw would be just as good a match as en would be. Thus the 
oddly-worded add-on phrase that contradicts the SHOULD:

 Note: When making the choice of linguistic preference available to
      the user, we remind implementors of  the fact that users are not
      familiar with the details of language matching as described above,
      and should provide appropriate guidance. As an example, users
      might assume that on selecting "en-gb", they will be served any
      kind of English document if British English is not available. A
      user agent might suggest in such a case to add "en" to get the
      best matching behavior.

Martin Duerst wrote:

>> of the list according to the above options is included. (With 
>> distance metrics, the shortest distance from any element in the list 
>> is the one used for the overall distance).
>>
>> 2.1.1, the lookup would progressively search for content as shown below:
>>
>>

---

From: r&d afrac <[email-masked]>
Date: Thu, 29 Sep 2005 15:35:32 +0200

At 14:03 29/09/2005, Jukka K. Korpela wrote:
>>>neglects a very important case. The general case of language 
>>>matching involves having a *list* of language ranges. You take a 
>>>list like "en-US en fr it" and use it for either matching or 
>>>lookup. So I would suggest adding the following text. (It probably 
>>>needs fleshing out a bit, but this is a start.).
>>
>>without my chair hat:
>>
>It deals with a specific mechanism, content negotiation, which lets 
>an HTTP server and a client negotiate on the choice of a language 
>version. (This has, sadly enough, been used rather on multilingual sites.)
>
>So it seems to be rather specific, rather than generally about 
>language codes and ranges, which have many potential applications 
>outside the HTTP protocol too. (The data might be carried over HTTP, 
>but that's a different issue.)
>
>If the HTTP definitions on language negotiation are regarded as 
>suitable for more general use, this should be based on a conscious 
>and explicit generalization.

Your remark is general. But a good illustration is OPES. The initial 
work has been carried on HTTP. The current work is about SMTP. I am 
interested in OPES behaviour as I see them as a strategic place for 
multilingual filtering and routing. We initially had a long debate 
concerning OCP (the OPES call-out protocol) to know if would be over 
TCP/IP or over IP. A TCP replacement has been retained as a possible 
further option. This means that language options are to be portable 
and scalable and worded as such.

However the constrains imposed by the RFC 3066 Bis draft on "private 
use" single equation with two variables and this reference to HTTP 
are perfectly acceptable if the document is PS and its scope is 
limited to XML, HTML, CLDR (cf. Charter) and possible documented extensions.
jfc

---

From: Martin Duerst <[email-masked]>
Date: Fri, 30 Sep 2005 15:53:15 +0900

Sorry I wasn't explicit enough. As you say, this is a specific
mechanism. I wanted to bring it up because:
- We could decide that this was the right thing (or good enough at least),
   and simply point to it (other specs can easily reference that part of
   the HTTP spec if they want).
- We could decide that this was the right thing (or good enough at least),
   and copy the text for one-stop-shopping by other standards.
- We could look at it, try to understand why it is the way it is,
   and come away with the conclusion that we need something different.

Well, yes, it's a specific thing, but it's not too difficult to see
how this could be used in other cases.

 >(This has, sadly enough, been used rather on multilingual sites.)

Not sure I get this. It was designed for multilingual sites.
Did you want to say "it is used rather rarely on multilingual sites"?
Or is there something else that you don't like in the way it is used?

I agree.     Regards,    Martin.