Subject: RE: [Ltru] Summary of comments, draft-ietf-ltru-matching-07

From: "Addison Phillips" <[email-masked]>
Date: Tue, 6 Dec 2005 08:42:38 -0800

Hello Harald,

The markup I have not looked over yet. Some comments below.

Addison

Addison P. Phillips
Globalization Architect, Quest Software

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Harald Tveit Alvestrand
> Sent: Monday, December 05, 2005 11:51 PM
> To: LTRU Working Group
> Subject: [Ltru] Summary of comments, draft-ietf-ltru-matching-07
> 
> At long last, I have managed to read through the whole of
> draft-ietf-ltru-matching-07 and mark it up with comments.
> 
> I am not happy with the current state of this document.
[Addison Phillips] 

I don't think it is fully baked yet. Since there has been a lot of churn in the text, it has been difficult to get a sense of whether it hangs together yet.
> 
> Most important points:
> 
> - The writing is inconsistent about what the inputs and outputs of a
> matching function is. Sometimes it talks about language ranges, sometimes
> it talks about language priority lists. I think it needs to always
> describe
> the function in terms of language priority lists.
[Addison Phillips] 

I completely agree. Actually, the LPL bits come from my efforts to change the text in that direction, having recognized that this is how each algorithm will be implemented.

> 
> - The lookup function, as designed, is broken. It introduces truncation of
> language tags outside of the user's control, and it prefers truncated
> versions of early tags in a LPL to full versions of later ones.
[Addison Phillips] 

I'm not sure I understand what you think is broken here. Lookup searches for content based on the idea that prefixes are mutually intelligible. It merely reverses which piece (the tag or the range) is looked at as the prefix.

This may be "broken" from your point of view, but it is, de facto, how many systems actually work. My "quarrel", if you can call it that, with Mark is that the text is starting to diverge from the way that these mechanisms are actually implemented, which makes me nervous.

> A much better lookup function is simply "use matching and pick the
> highest-scoring result; if the result set is empty, use default". User
> interfaces can do the expansion IF they decide it's useful to do so.
[Addison Phillips] 

Locale-based resource/message catalog systems do not work this way: they use lookup as described by the draft. Any number of Web technologies (both J2EE and ASP.Net come instantly to mind) do the same with static or semi-static content. There is a deterministic search though the available material, exhausting each range before moving on to the next, that is undertaken in the language negotiation.

> 
> - The spec needs to make up its mind about whether it's specifying an
> algorithm for the IETF standards track or giving a bunch of useful advice
> to people writing language tag matching functions. If it's the former,
> EVERY SINGLE INSTANCE of "an implementation MAY decide to...", where that
> can affect the returned result, needs to be removed from the specification.
[Addison Phillips] 

I agree kind-of. If this document were to go on the STD track, then I completely agree. But as an editor I received instructions immediately following the draft-registry period (off-list) that we could not make that determination. If we are to change to the STD track (which I have no objection to) then various document changes would need to be incorporated to support that.

> 
> A more detailed markup is enclosed, including notes of the five (?) places
> where "zh" has turned into "z.", but these are the 3 high points; I
> believe
[Addison Phillips] 

That was a bad search-and-replace, repaired some days ago in my copy of draft-08. I'll go through your other comments and incorporate/comment as appropriate. I agree that the WG needs to come to consensus on the high points (at least).

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Scott Hollenbeck" <[email-masked]>
Date: Tue, 6 Dec 2005 11:54:58 -0500

> From: Addison Phillips [mailto:[email-masked]] 
> Sent: Tuesday, December 06, 2005 11:43 AM
> To: Harald Tveit Alvestrand; LTRU Working Group
> Subject: RE: [Ltru] Summary of comments, draft-ietf-ltru-matching-07
> 

[snip]

> I agree kind-of. If this document were to go on the STD 
> track, then I completely agree. But as an editor I received 
> instructions immediately following the draft-registry period 
> (off-list) that we could not make that determination. If we 
> are to change to the STD track (which I have no objection to) 
> then various document changes would need to be incorporated 
> to support that.

The group is going to have to think about this and produce a recommendation
before the document is submitted for AD and IESG evaluation.  The IESG will
ultimately decide if that recommendation is acceptable, but the requested
publication status usually comes from the working group.

-Scott-

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 06 Dec 2005 18:16:10 +0100

(breaking the stuff apart into separate threads - we seem to agree on the 
first one, so ignoring that)

--On tirsdag, desember 06, 2005 08:42:38 -0800 Addison Phillips 
<[email-masked]> wrote:

>> - The lookup function, as designed, is broken. It introduces truncation
>> of language tags outside of the user's control, and it prefers truncated
> I'm not sure I understand what you think is broken here. Lookup searches
> for content based on the idea that prefixes are mutually intelligible. It
> merely reverses which piece (the tag or the range) is looked at as the
> prefix.
>

[Addison Phillips]

I'm not sure I understand what you think is broken here. Lookup searches 
for content based on the idea that prefixes are mutually intelligible. It 
merely reverses which piece (the tag or the range) is looked at as the 
prefix.

This may be "broken" from your point of view, but it is, de facto, how many 
systems actually work. My "quarrel", if you can call it that, with Mark is 
that the text is starting to diverge from the way that these mechanisms are 
actually implemented, which makes me nervous.
de facto, how
> many systems actually work. My "quarrel", if you can call it that, with
> Mark is that the text is starting to diverge from the way that these
> mechanisms are actually implemented, which makes me nervous.

I think that it's broken in two ways.

First:
If the language range in a lookup is:

 zh-Hant, en-GB

and the available objects for matching have tags:

 zh-Hant-TW en-GB zh-Hans-CN zh

then returning an object labelled "zh-Hant-TW" is consistent with user 
expectations.
Matching an object labelled "zh" is not - and that is what the current 
algorithm of "truncate first, move on later" will produce.

Second:
If the language range in a lookup is:

  zh-Hant, en, zh-Hans

and the available objects for matching have tags:

  en-GB zh-Hans

then returning an object labelled "en-GB" is consistent with user 
expectations, and returning an object labelled "zh-Hans" is not - and that 
is what the current algorithm of "truncate from the language range, not the 
language tag" will produce.

WRT running code:
If you have reference to source-available implementations, I'd like to look 
at what they actually do - but it would be no surprise to find that they 
behave in ways that I would never WANT software to behave in certain 
situations. That's why we discuss....

                         Harald

---

From: "John.Cowan" <[email-masked]>
Date: Tue, 6 Dec 2005 12:49:21 -0500

> publication status usually comes from the working group.

My understanding (which may be out of date) was that the whole point of
splitting RFC 3066 into registration and matching drafts was that
matching was unsuitable for a BCP.  I didn't see why, since the matching
parts of 3066 basically said "This is how 2616 does it", but I wasn't
going to argue with the PTB.  Then there was talk of registration and
matching jointly representing BCP 47.

As I read the matching draft, it says what protocols/implementations
MUST do.  That smells like a standards-track RFC to me.

Comments?

-- 
Yes, chili in the eye is bad, but so is your    John Cowan
ear.  However, I would suggest you wash your    [email-masked]
hands thoroughly before going to the toilet.    http://www.reutershealth.com
        --gadicath                              http://www.ccil.org/~cowan

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 07 Dec 2005 06:48:19 +0100

John.Cowan wrote:

> That smells like a standards-track RFC to me.  Comments?

+1

---

From: Martin Duerst <[email-masked]>
Date: Wed, 07 Dec 2005 20:32:35 +0900

<[email-masked]> wrote:
 >This may be "broken" from your point of view, but it is, de facto, how 
expectations.

The problem with the above example is that the content is labeled
with too much detail. The principle (currently used) for lookup
is to be as specific as possible for ranges, but to be general
on the content side. So the conclusion may simply be that the above
content is labeled too specificly. Changing the content labels to
    zh-Hant en-GB zh-Hans
would make this example work. Please note that this refers only
to the labels that the content is looked up with, which do not
necessarily have to be the same as those the content is actually
labeled with (e.g. we could have an extension corresponding to
zh-Hant on the filename, while internally, it would say
xml:lang='zh-Hant-TW'). Of course, we should consider the
repercussions this has for content management.

There is another aspect, however, than can also hurt this scheme.
Take again a language priority list of:
    zh-Hant, en-GB
and assume we have
    zh, en
as content, where zh is actually zh-Hans. Then we get simplified
Chinese back to somebody who may not be able to read it. Again,
the problem can be solved by fixing the content labeling: Change
zh to zh-Hans if it is indeed simplified.

But thinking this through, we get into a series of recommendations
for both sides:
- Recommendation for user (1): Be as specific as you like. You'll
   get the more general stuff anyway as a fallback.
- Recommendation for content provider (1): Do not be more specific
   than necessary. Some users may use more generic ranges, in which
   case your content won't be used.
- Recommendation for content provider (2): Do not be too general.
   Do not use tags that leave out information that makes or breaks
   intellegibility for users.
- Recommendation for users (2): Do not be too general. Do not use
   ranges that leave out information that makes or breaks
   intellegibility for other users, because servers won't use
   them, and your ranges won't match. If you understand more than
   one 'version' of a language that in generall are not considered
   mutually intellegible, list them both, in your preference.

This probably still leaves out some corner cases. The whole
exercise shows that there is a balance, and maybe in some
cases even a tug-of-war between tags and ranges used by
clients and servers.

For the zh case, this would lead to servers using zh-Hans and
zh-Hant, only something more specific if they e.g. have
several country alternatives (but in that case also serving
one of them under a generic tag), and not using zh only
because its variants are not mutually intellegible in
the general public. Users would use zh-Hans and zh-Hant,
as appropriate, or something more specific, but not simply
zh, because that wouldn't match.

 From the thread between Harald and Addison:

[Harald]

[Addison Phillips]
 >I'm not sure I understand what you think is broken here. Lookup searches 
for content >based on the idea that prefixes are mutually intelligible.

Which we know isn't true (unless we avoid content labeled with such prefixes).

 >It merely reverses which >piece (the tag or the range) is looked at as 
the prefix.

 >This may be "broken" from your point of view, but it is, de facto, how 
many systems >actually work. My "quarrel", if you can call it that, with 
Mark is that the text is >starting to diverge from the way that these 
mechanisms are actually implemented, which >makes me nervous.

[Harald]

[Addison Phillips]
 >Locale-based resource/message catalog systems do not work this way: they 
use lookup as >described by the draft. Any number of Web technologies (both 
J2EE and ASP.Net come >instantly to mind) do the same with static or 
semi-static content. There is a >deterministic search though the available 
material, exhausting each range before moving >on to the next, that is 
undertaken in the language negotiation.

I think one additional reason for doing this is providing a *consistent*
user experience. Assume a user has a preference of

en-gb, fr

Falling back to prefixes assures that e.g. en is choosen when en-gb is not
available. In cases where a user navigates a Web site, it is a more
consistent experience to go through pages that are all in English
(even if slightly different style) than to flip between English and
French pages. Even more so for cases where a single page is patched
together from different content pieces.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 07 Dec 2005 20:09:34 +0900

>> That smells like a standards-track RFC to me.  Comments?

[as a technical contributor]

It seems to make sense to me, but I agree with Harald
that in that case, we have to be considerably more
precise (many less "MAY"s, even though I don't agree
with Harald that we have to completely eliminate them).

Regards,   Martin.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 07 Dec 2005 13:24:31 +0100

>    mutually intellegible, list them both, in your preference.

My worry is that if the recommendations for the user and the content 
provider are *opposite* for the lookup case and the matching case, this 
creates a huge amount of headache when people are trying to use both within 
a single application, or with a single datastore.

I still haven't seen the convincing argument why they need to be opposite.

                    Harald