Subject: RE: [Ltru] Re: matching for non-RFC3066bis langtags

From: "Addison Phillips" <[email-masked]>
Date: Mon, 22 Aug 2005 09:14:25 -0700

FWIW, a reminder: the matching draft hasn't had a lot of close scrutiny to its specific language. It was split from draft-registry back at the start of the working group's activity. The three current drafts have mostly focused on keeping it in sync with draft-registry. As a result, the text still needs some editing of this nature.

I modified the introduction and began some other minor editing before the weekend, but didn't have a chance to post the results until just now. I believe these changes address JFC's comments about the introduction.

The content section that received the most attention was the one on scored matching (2.2.3). Previously that section was stripped from some emails by John Cowan. I have rewritten this section using the same general tone as the rest of the document, adding an example of quadruples. There is one bad word choice in the result currently posted ("Simple applications..." should read "Some applications...")

Best Regards,

Addison

Addison P. Phillips
Globalization Architect, Quest Software
Chair, W3C Internationalization Core Working Group

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Doug Ewell
> Sent: 2005年8月22日 8:17
> To: LTRU Working Group
> Subject: [Ltru] Re: matching for non-RFC3066bis langtags
> 
> If the matching schemes described in draft-matching were specifically
> designed for use with language tags as defined in draft-registry, then I
> think it would be appropriate to say so in the Introduction.
> 
> It is often the case that a good engineering solution can be applied, in
> whole or part, to situations for which it was not specifically designed.
> I don't see any need to call special attention to this phenomenon in
> draft-matching.
> 
> --
> Doug Ewell
> Fullerton, California
> http://users.adelphia.net/~dewell/
> 
> 
> 
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ltru

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Mark Davis <[email-masked]>
Date: Mon, 22 Aug 2005 18:17:06 -0700

I think the scored matching would be better framed as a 'distance' 
metric. Here is suggested replacement text:
=====

In the "scored matching" scheme, the extended language range ...
...
... SHOULD probably allow users to choose the most appropriate selection 
value.

=>

In the "distance metric" scheme, a numeric value representing a 
'distance' between a language tags and a language range is computed. A 
distance of zero means that they are identical, a small distance 
indicates that they are very similar, and a large distance indicated 
that they are very different. Given a distance, implementations can 
allow users to select a threshold distance for a match to be successful 
or a filter to be applied.

The first step in the process is to normalize the extended language 
range and the language tags by canonicalizing both, and mapping 
grandfathered and obsolete tags into modern equivalents. If there is no 
script, then the Suppress-Script value from the registry is added as the 
script.

The language range and the language tags are then transformed into 
quadruples of elements of the form (language, script, country, variant). 
Any extended languages are considered part of the language element and 
any extension subtag sequences are considered  part of the variant 
element; an x private use subtag sequences is considered part of the 
language element if it is initial and part of the variant element if 
not. Missing components in the language range are set to "*"; thus a "*" 
pattern becomes the quadruple ("*", "*", "*", "*"). Subtags 'und' and 
'Zyyy' are converted to "*".

Each language tag being matched or filtered is assigned a distance, 
whereby small values indicate better matches and large values indicate 
worse ones. The distance between the language tags is the sum of the 
distances for each of the corresponding elements of the quadruple. If 
the elements are identical or one is '*', then the distance between them 
is zero. Otherwise, it is given by the following table:

  128    language mismatch
   32    script mismatch
    4    region mismatch
    1    variant mismatch(es)

Examples of distances from en-US:

         fr: 132
      en-GB:   4
 en-Latn-US:   0
    en-Kata:  36

Implementations may want to use more sophisticated weights that depend 
on the values of the corresponding elements. For example, depending on 
the domain, an implemenation might give a small distance to the 
difference between 'no' and 'nb' or 'nn'; or between 'Kata' and 'Hira'; 
or between 'US' and 'UM'.

Mark

Addison Phillips wrote:

>------------------------------------------------------------------------
>
[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Tue, 23 Aug 2005 11:45:36 +0900

Resent, with more appropriate Subject:.

PLEASE everybody make sure you change the Subject: when changing
the subject!

Regards,    Martin.

At 10:17 05/08/23, Mark Davis wrote:
>I think the scored matching would be better framed as a 'distance' metric. 
>Here is suggested replacement text:
>=====
>
>... SHOULD probably allow users to choose the most appropriate selection value.
>
>=>
>
>distance of zero means that they are identical, a small distance indicates 
>that they are very similar, and a large distance indicated that they are 
>very different. Given a distance, implementations can allow users to 
>select a threshold distance for a match to be successful or a filter to be 
>applied.
>
>The first step in the process is to normalize the extended language range 
>and the language tags by canonicalizing both, and mapping grandfathered 
>and obsolete tags into modern equivalents. If there is no script, then the 
>Suppress-Script value from the registry is added as the script.
>
>Any extended languages are considered part of the language element and any 
>extension subtag sequences are consideredツ  part of the variant element; 
>an x private use subtag sequences is considered part of the language 
>element if it is initial and part of the variant element if not. Missing 
>components in the language range are set to "*"; thus a "*" pattern 
>becomes the quadruple ("*", "*", "*", "*"). Subtags 'und' and 'Zyyy' are 
>converted to "*".
>
>distances for each of the corresponding elements of the quadruple. If the 
>elements are identical or one is '*', then the distance between them is 
>zero. Otherwise, it is given by the following table:
>
>ツ  128 ツ ツ  language mismatch
>ツ ツ  32 ツ ツ  script mismatch
>ツ ツ ツ  4 ツ ツ  region mismatch
>ツ ツ ツ  1ツ ツ ツ  variant mismatch(es)
>
>ツ ツ ツ ツ ツ ツ ツ ツ  fr: 132
>ツ ツ ツ ツ ツ  en-GB:ツ ツ  4
>ツ en-Latn-US:ツ ツ  0
>ツ ツ ツ  en-Kata:ツ  36
>Implementations may want to use more sophisticated weights that depend on 
>the values of the corresponding elements. For example, depending on the 
>domain, an implemenation might give a small distance to the difference 
>between 'no' and 'nb' or 'nn'; or between 'Kata' and 'Hira'; or between 
>'US' and 'UM'.
>
>Mark
>Addison Phillips wrote:
>>
>>FWIW, a reminder: the matching draft hasn't had a lot of close scrutiny 
>>to its specific language. It was split from draft-registry back at the 
>>start of the working group's activity. The three current drafts have 
>>mostly focused on keeping it in sync with draft-registry. As a result, 
>>the text still needs some editing of this nature.
>>
>>I modified the introduction and began some other minor editing before 
>>the weekend, but didn't have a chance to post the results until just now. 
>>I believe these changes address JFC's comments about the introduction.
>>
>>The content section that received the most attention was the one on 
>>scored matching (2.2.3). Previously that section was stripped from some 
>>emails by John Cowan. I have rewritten this section using the same 
>>general tone as the rest of the document, adding an example of 
>>quadruples. There is one bad word choice in the result currently posted 
>>("Simple applications..." should read "Some applications...")
>>
>>Best Regards,
>>
>>Addison
>>
>>Addison P. Phillips
>>>From: <mailto:[email-masked]>[email-masked] 
>>>[mailto:[email-masked]] On
>>>Behalf Of Doug Ewell
>>>Sent: 2005蟷エ8譛�2譌・ 8:17
>>>To: LTRU Working Group
>>><http://users.adelphia.net/~dewell/>http://users.adelphia.net/~dewell/
>>>
>>>
>>>
>>><mailto:[email-masked]>[email-masked]
>><mailto:[email-masked]>[email-masked]

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 25 Aug 2005 21:36:36 -0700

Hi -

...
...
>Implementations may want to use more sophisticated weights that depend on

editorial: you mean "Implementations MAY use more sophisticated..." right?

>domain, an implemenation might give a small distance to the difference

might -> MAY?

...

There are obviously lots of *possible* distance metrics for use in scored
matching algorithms.  However, as a technical contributor, I think it's
helpful to keep in mind what our charter asks us to do:

| The second document will describe matching algorithms for use with
| language tags. Language tags are used in a broad variety of contexts
| and it is not expected that any single matching algorithm will fit all
| needs. Developing a small set of common matching algorithms does seem
| likely to contribute to interoperability, however, as it seems likely
| that using protocols could reference these well-known algorithms in
| their specifications.

I think the phrase "a small set of common matching algorithms" is important.
Consequently, I think the discussion about what distance metrics *might* be
used, while perhaps of academic interest, doesn't belong in this document
unless we plan to use in one of the algorithms in our "small set" in order to
foster interoperability.

Randy

---

From: Mark Davis <[email-masked]>
Date: Fri, 26 Aug 2005 08:35:26 -0700

>>domain, an implemenation might give a small distance to the difference

For both of these, I think the normal English meaning is being used. All 
  implementations are free to match in different ways. What is being 
describe here is not a standardized option on the method being 
described, but rather pointing the user to an alternative matching that 
could be employed. (see below).

> unless we plan to use in one of the algorithms in our "small set" in order to

I agree that the goal of the document is to present a small list of 
well-defined matching algorithms. However, I think it is also not 
outside the scope to at least point out ways in which implementations 
may choose to reasonably extend the methods that we define.

And it is also useful during the development state, to get people 
thinking about various options.

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 26 Aug 2005 18:05:54 -0700

Hi -

As a technical contributor...

> From: "Mark Davis" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Friday, August 26, 2005 8:35 AM
> Subject: Re: [Ltru] Re: Distance Metric
...
> may choose to reasonably extend the methods that we define.

That sort of stuff generally doesn't belong in a standards-track document.
When it does, it's because there's a need to explain, for example, an extension
mechanism.

...

I agree that it's useful to think about it, and discuss the possibilities
on the mailing list if someone actually has requirements for them,
but I really don't like seeing that kind of stuff in standards-track
documents, since, at least in this form, it does nothing to help
interoperability.

Randy