Subject: RE: [Ltru] Re: matching for non-RFC3066bis langtags

From: "Addison Phillips" <[email-masked]>
Date: Mon, 22 Aug 2005 10:24:38 -0700

Jefsey: 

Do you have text to propose? Or is this just another example of your wasting our time with vacuous rants?

I believe the changes I have made in the draft should address your original question: these matching schemes are for matching RFC 3066bis language tags. If you have some other suggestion, you will need to propose text or risk being ignored as irrelevant.

Addison

Addison P. Phillips
Globalization Architect, Quest Software
Chair, W3C Internationalization Core Working Group

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of r&d afrac
> Sent: 2005?8?22? 9:16
> To: Doug Ewell; LTRU Working Group
> Subject: Re: [Ltru] Re: matching for non-RFC3066bis langtags
> 
> At 17:16 22/08/2005, Doug Ewell wrote:
> >If the matching schemes described in draft-matching were specifically
> >designed for use with language tags as defined in draft-registry, then I
> >think it would be appropriate to say so in the Introduction.
> 
> The point is what is a language tag not defined in the RFC 3066 bis.
> A chance we could agree, I unfortunately do not beleive in.
> jfc
> 
> 
> 
> >It is often the case that a good engineering solution can be applied, in
> >whole or part, to situations for which it was not specifically designed.
> >I don't see any need to call special attention to this phenomenon in
> >draft-matching.
> >
> >--
> >Doug Ewell
> >Fullerton, California
> >http://users.adelphia.net/~dewell/
> >
> >
> >
> >_______________________________________________
> >Ltru mailing list
> >[email-masked]
> >https://www1.ietf.org/mailman/listinfo/ltru
> 
> 
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ltru

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Addison Phillips" <[email-masked]>
Date: Fri, 26 Aug 2005 15:52:36 -0700

Okay, in incorporating this text, I found a small potential glitch in Mark's algorithm. Consider the range "en-US" with the following tags and their scores:

"fr"             256 (language mismatch, region match)
"en-GB"          384 (language, region mismatch)
"en-Latn-US"       0 (all fields match)
"en-Brai"         32 (region mismatch)
"en-US-x-foo"      0 (variant mismatch)
"en-US-r-wadegile" 0 (extension mismatch)

These last two are not logical results: there is (or should be) a distance between the last two tags and the range "en-US", if one follows the text about extended language ranges just above it.

Therefore I changed the expansion of ranges and tags to have separate rules. Tags expand missing fields to "*". Ranges expand missing fields up to the last concrete field to "*" and to the empty string beyond that. Thus range "en-US" has the quintuple:

"en","*","US","",""

Now the scores are a more logical:

"fr"             256 (language mismatch, region match)
"en-GB"          384 (language, region mismatch)
"en-Latn-US"       0 (all fields match)
"en-Brai"         32 (region mismatch)
"en-US-x-foo"      4 (variant mismatch: range is the empty string)
"en-US-r-wadegile" 1 (extension mismatch: range is the empty string)

Mark's text thus modified is incorporated on inter-locale. See:

http://www.inter-locale.com/ID/draft-ietf-ltru-matching-04.txt
http://www.inter-locale.com/ID/draft-ietf-ltru-matching-04.html

Addison

Addison P. Phillips

> From: Mark Davis [mailto:[email-masked]]
> Sent: 2005年8月26日 9:55
> To: Addison Phillips
> Cc: Doug Ewell; LTRU Working Group
> Addison Phillips wrote:
> > Comments follow.
> >>From: Mark Davis [mailto:[email-masked]]
> >>Sent: 2005年8月22日 18:17
> >>To: Addison Phillips
> >>Cc: Doug Ewell; LTRU Working Group
> >>I think the scored matching would be better framed as a 'distance'
> metric.
> 
> good.
> 
> >
> > [Addison Phillips]
> >
> > I like a distance metric better.
> >
> >
> >>Here is suggested replacement text:
> >>=====
> >>
> >>
> >>In the "scored matching" scheme, the extended language range ...
> >>...
> >>... SHOULD probably allow users to choose the most appropriate selection
> >>value.
> >>
> >>
> >>=>
> >>
> >>
> >>In the "distance metric" scheme, a numeric value representing a
> 'distance'
> >>between a language tags and a language range is computed. A distance of
> >>zero means that they are identical, a small distance indicates that they
> >>are very similar, and a large distance indicated that they are very
> >>different. Given a distance, implementations can allow users to select a
> >>threshold distance for a match to be successful or a filter to be
> applied.
> >>
> >>The first step in the process is to normalize the extended language
> range
> >>and the language tags by canonicalizing both, and mapping grandfathered
> >>and obsolete tags into modern equivalents. If there is no script, then
> the
> >>Suppress-Script value from the registry is added as the script.
> >
> > [Addison Phillips]
> >
> > This is wrong for two reasons.
> >
> > First, some languages do not have Suppress-Scripts and these languages
> would be at a disadvantage or difficult to implement for. No other
> matching scheme requires access to the registry to do its work. This one
> requires some (for the canonicalization just mentioned), but this strikes
> me as a leap of complexity that may not be necessary.
> >
> > Second, a missing value in a quadruple may be missing for a reason. I
> think only two values might potentially be added for a missing value: the
> empty value or "*". I have to question why script would be different than
> other subtags here...
> 
> That's reasonable; if a script is missing it gets '*', so matching will
> still work fine between en-Latn-US and en-US
> 
> >
> >>The language range and the language tags are then transformed into
> >>quadruples of elements of the form (language, script, country, variant).
> >>Any extended languages are considered part of the language element and
> any
> >>extension subtag sequences are considered  part of the variant element;
> an
> >
> > [Addison Phillips]
> >
> > Including extensions in variant will skew the scores in an inappropriate
> manner, at least potentially. Perhaps extensions should be scored
> separately and at a lower value:
> >
> > Range=sl-IT-rozaj-r-foo
> >
> > Sl-IT: 1
> > Sl-IT-rozaj: 1 // hm...
> > Sl-IT-r-foo: 1
> > Sl-rozaj: 5 // hm...
> > Sl-rozaj-r-foo: 4
> 
> I agree; let's make it a quintuple.
> 
> >
> >
> >
> >>x private use subtag sequences is considered part of the language
> element
> >>if it is initial and part of the variant element if not. Missing
> >>components in the language range are set to "*"; thus a "*" pattern
> >>becomes the quadruple ("*", "*", "*", "*"). Subtags 'und' and 'Zyyy' are
> >>converted to "*".
> >
> > [Addison Phillips]
> >
> > Where did 'Zyyy' come from?
> 
> >
> > Also: What about 'mul'? Does it match "*"? Or just 'mul'?
> 
> Probably '*' is the best.
> 
> Zyyy means undetermined script. We should also say 'Language subtags
> 'und' and 'mul', and the script subtag 'Zyyy' are converted to "*".'
> since und could at some point be a region code.
> 
> 
> >
> >>
> >>Each language tag being matched or filtered is assigned a distance,
> >>whereby small values indicate better matches and large values indicate
> >>worse ones. The distance between the language tags is the sum of the
> >>distances for each of the corresponding elements of the quadruple. If
> the
> >>elements are identical or one is '*', then the distance between them is
> >>zero. Otherwise, it is given by the following table:
> >>
> >>  128    language mismatch
> >>   32    script mismatch
> >>    4    region mismatch
> >>    1    variant mismatch(es)
> >
> > [Addison Phillips]
> >
> > Is this last meant to indicate that a score of "2" is possible for two
> variant mismatches? Or just that multiple subtags are possible there? I
> read it as the latter.
> 
> Yes, I meant the latter.
> 
> >
> > Also: note that subtag order is significant in variant matching if one
> assumes simple string matching.
> 
> Yes.
> 
> >
> > I'm curious why the values are each an additional power of two apart. If
> multiple scoring on a level isn't possible, then all this does is magnify
> the apparent distance.
> 
> Gaps were left just to allow for intermediate weights. For example, when
> 639-3 comes along, one might want to allow for weight differences
> between dialects that are between 128 and 64.
> 
> 
> Logically, it would be cleaner to just have them be floating point
> numbers, but I didn't want to run afoul of any integral purists ;-).
> 
> >
> >>Examples of distances from en-US:
> >>
> >>
> >>         fr: 132
> >>      en-GB:   4
> >> en-Latn-US:   0
> >>    en-Kata:  36
> >>
> >>
> >>Implementations may want to use more sophisticated weights that depend
> on
> >>the values of the corresponding elements. For example, depending on the
> >>domain, an implemenation might give a small distance to the difference
> >>between 'no' and 'nb' or 'nn'; or between 'Kata' and 'Hira'; or between
> >>'US' and 'UM'.
> >>
> >
> > Or we should just specify the weights to use. I see scored matching
> being used mostly to present a list of results ranked by relevance to the
> request. The other matching schemes produce binary matches and are more
> suited to returning an individual document or piece of content.
> >
> We could do that, and there are some obvious cases like the examples
> given. As long as people didn't expect completeness, that would be ok.
> But I'm sure that if we supply such weightings there will always be
> people who argue that X language or region or script is being
> disadvantaged by not being weighted specially, so that's why I didn't
> want to go there at first.

[email-masked]