Subject: RE: [Ltru] Re: matching for non-RFC3066bis langtags

From: "Addison Phillips" <[email-masked]>
Date: Fri, 26 Aug 2005 08:44:03 -0700

Comments follow.

Addison P. Phillips
Globalization Architect, Quest Software
Chair, W3C Internationalization Core Working Group

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: Mark Davis [mailto:[email-masked]]
> Sent: 2005年8月22日 18:17
> To: Addison Phillips
> Cc: Doug Ewell; LTRU Working Group
> Subject: Re: [Ltru] Re: matching for non-RFC3066bis langtags
> 
> I think the scored matching would be better framed as a 'distance' metric.
[Addison Phillips] 

I like a distance metric better.

> Here is suggested replacement text:
> =====
> 
> 
> In the "scored matching" scheme, the extended language range ...
> ...
> ... SHOULD probably allow users to choose the most appropriate selection
> value.
> 
> 
> =>
> 
> 
> In the "distance metric" scheme, a numeric value representing a 'distance'
> between a language tags and a language range is computed. A distance of
> zero means that they are identical, a small distance indicates that they
> are very similar, and a large distance indicated that they are very
> different. Given a distance, implementations can allow users to select a
> threshold distance for a match to be successful or a filter to be applied.
> 
> The first step in the process is to normalize the extended language range
> and the language tags by canonicalizing both, and mapping grandfathered
> and obsolete tags into modern equivalents. If there is no script, then the
> Suppress-Script value from the registry is added as the script.
[Addison Phillips] 

This is wrong for two reasons. 

First, some languages do not have Suppress-Scripts and these languages would be at a disadvantage or difficult to implement for. No other matching scheme requires access to the registry to do its work. This one requires some (for the canonicalization just mentioned), but this strikes me as a leap of complexity that may not be necessary.

Second, a missing value in a quadruple may be missing for a reason. I think only two values might potentially be added for a missing value: the empty value or "*". I have to question why script would be different than other subtags here...
> 
> The language range and the language tags are then transformed into
> quadruples of elements of the form (language, script, country, variant).
> Any extended languages are considered part of the language element and any
> extension subtag sequences are considered  part of the variant element; an
[Addison Phillips] 

Including extensions in variant will skew the scores in an inappropriate manner, at least potentially. Perhaps extensions should be scored separately and at a lower value:

Range=sl-IT-rozaj-r-foo

Sl-IT: 1
Sl-IT-rozaj: 1 // hm...
Sl-IT-r-foo: 1
Sl-rozaj: 5 // hm...
Sl-rozaj-r-foo: 4

> x private use subtag sequences is considered part of the language element
> if it is initial and part of the variant element if not. Missing
> components in the language range are set to "*"; thus a "*" pattern
> becomes the quadruple ("*", "*", "*", "*"). Subtags 'und' and 'Zyyy' are
> converted to "*".
[Addison Phillips] 

Where did 'Zyyy' come from? 

Also: What about 'mul'? Does it match "*"? Or just 'mul'?
> 
> 
> Each language tag being matched or filtered is assigned a distance,
> whereby small values indicate better matches and large values indicate
> worse ones. The distance between the language tags is the sum of the
> distances for each of the corresponding elements of the quadruple. If the
> elements are identical or one is '*', then the distance between them is
> zero. Otherwise, it is given by the following table:
> 
>   128    language mismatch
>    32    script mismatch
>     4    region mismatch
>     1    variant mismatch(es)
[Addison Phillips] 

Is this last meant to indicate that a score of "2" is possible for two variant mismatches? Or just that multiple subtags are possible there? I read it as the latter.

Also: note that subtag order is significant in variant matching if one assumes simple string matching.

I'm curious why the values are each an additional power of two apart. If multiple scoring on a level isn't possible, then all this does is magnify the apparent distance. 
> 
> Examples of distances from en-US:
> 
> 
>          fr: 132
>       en-GB:   4
>  en-Latn-US:   0
>     en-Kata:  36
> 
> 
> Implementations may want to use more sophisticated weights that depend on
> the values of the corresponding elements. For example, depending on the
> domain, an implemenation might give a small distance to the difference
> between 'no' and 'nb' or 'nn'; or between 'Kata' and 'Hira'; or between
> 'US' and 'UM'.
> 
Or we should just specify the weights to use. I see scored matching being used mostly to present a list of results ranked by relevance to the request. The other matching schemes produce binary matches and are more suited to returning an individual document or piece of content.

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Mark Davis <[email-masked]>
Date: Fri, 26 Aug 2005 09:55:23 -0700

>>I think the scored matching would be better framed as a 'distance' metric.

good.

> Second, a missing value in a quadruple may be missing for a reason. I think only two values might potentially be added for a missing value: the empty value or "*". I have to question why script wou…

That's reasonable; if a script is missing it gets '*', so matching will 
still work fine between en-Latn-US and en-US

> Including extensions in variant will skew the scores in an inappropriate manner, at least potentially. Perhaps extensions should be scored separately and at a lower value:

I agree; let's make it a quintuple.

> Also: What about 'mul'? Does it match "*"? Or just 'mul'?

Probably '*' is the best.

Zyyy means undetermined script. We should also say 'Language subtags 
'und' and 'mul', and the script subtag 'Zyyy' are converted to "*".' 
since und could at some point be a region code.

> Is this last meant to indicate that a score of "2" is possible for two variant mismatches? Or just that multiple subtags are possible there? I read it as the latter.

Yes, I meant the latter.

> Also: note that subtag order is significant in variant matching if one assumes simple string matching.

Yes.

> I'm curious why the values are each an additional power of two apart. If multiple scoring on a level isn't possible, then all this does is magnify the apparent distance. 

Gaps were left just to allow for intermediate weights. For example, when 
639-3 comes along, one might want to allow for weight differences 
between dialects that are between 128 and 64.

Logically, it would be cleaner to just have them be floating point 
numbers, but I didn't want to run afoul of any integral purists ;-).

> Or we should just specify the weights to use. I see scored matching being used mostly to present a list of results ranked by relevance to the request. The other matching schemes produce binary matc…
We could do that, and there are some obvious cases like the examples 
given. As long as people didn't expect completeness, that would be ok. 
But I'm sure that if we supply such weightings there will always be 
people who argue that X language or region or script is being 
disadvantaged by not being weighted specially, so that's why I didn't 
want to go there at first.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 26 Aug 2005 17:39:17 -0700

(Yes, I'm still around....)
I like distance metrics..... since the subject has changed, I've changed 
the subject....

>> Or we should just specify the weights to use. I see scored matching
>> being used mostly to present a list of results ranked by relevance to
>> the request. The other matching schemes produce binary matches and are
>> more suited to returning an individual document or piece of content.
>>

Stuff that people will want to tweak endlessly shouldn't be specified in a 
standard, IMHO.
And there's some interesting special cases; I thnk it was Mark who told me 
about a case where the user wanted "xx-Aaaa" and "xx-Bbbb" to match with a 
score of +INFINITY, so that when the user asked for "xx-Aaaa", "en" would 
match better than "xx-Bbbb".

(script names and language codes changed to protect the guilty..)

But as a general rule, it may be a Good Idea to specify a sentence such as

   The standard's weights give a maximal metric of distance
   between two language tags. Local knowledge may assign a lower
   distance between specific pairs (such as between "nn" and "no").
   Assigning such lower distances is outside of the scope of
   this standard.

IE allow tweaking, but specify that normal tweaks bring things closer 
together, it does not drive them further apart.

(Historical parenthesis: the original HTTP "accept-language" specification 
used floating point preference values between 0 and 1, so if specifiers 
want floating point, they're following precedent....)

                  Harald

---

From: r&d afrac <[email-masked]>
Date: Sat, 27 Aug 2005 11:21:49 +0200

At 02:39 27/08/2005, Harald Tveit Alvestrand wrote:
>And there's some interesting special cases; I think it was Mark who 
>told me about a case where the user wanted "xx-Aaaa" and "xx-Bbbb" 
>to match with a score of +INFINITY, so that when the user asked for 
>"xx-Aaaa", "en" would match better than "xx-Bbbb".

This has a known name: "shaping the world".
It took time before someone eventually acknowledges it.
Thank you.
jfc

---

From: Mark Davis <[email-masked]>
Date: Sat, 27 Aug 2005 09:59:10 -0700

good points.

Harald Tveit Alvestrand wrote:
> Stuff that people will want to tweak endlessly shouldn't be specified in 
> a standard, IMHO.
> And there's some interesting special cases; I thnk it was Mark who told 
> me about a case where the user wanted "xx-Aaaa" and "xx-Bbbb" to match 
> with a score of +INFINITY, so that when the user asked for "xx-Aaaa", 
> "en" would match better than "xx-Bbbb".
> 
> (Historical parenthesis: the original HTTP "accept-language" 
> specification used floating point preference values between 0 and 1, so 
> if specifiers want floating point, they're following precedent....)
> 
>                  Harald