Subject: [Ltru] mapping ext-lang-range to basic in lookup

From: "Addison Phillips" <[email-masked]>
Date: Fri, 18 Nov 2005 14:55:47 -0800

> >>----
> >>
> >>Wildcard values in an extended language range are supposed to match any
> >>value
> >>=>
> >>Wildcard values in an extended language range match any value
> >>
> >>
> >[Addison Phillips]
> >
> >No, they are *supposed* to match, but in this case they don't. They match
> the empty value.
> >
> >
> I read that differently. When you say "supposed to match", it sounds
> like there is a choice, that there are some values that they don't
> match. Like the US is "supposed to" disallow torture.

When considered in isolation, I suppose one could have that reading, but I think the text is actually pretty clear here. Perhaps it could be clearer. The missing point that I was trying to convey is that extended language ranges do not produce the expected results based on their meaning. The range "en-*-US" will never pass through content labeled "en-Brai" (or any other content with a script subtag), even though this matches the meaning of the extended language range. From "en-US" you skip immediately to "en". It is supposed to match... and it doesn't.

I think we might make a version of the change you suggest and then spell out my example.

The text currently:

--
In some cases, the language priority list might contain one or more extended language ranges (as, for example, when the same language priority list is used as input for both lookup and filtering operations). Wildcard values in an extended language range are supposed to match any value that occurs in that position in a language tag. Since only one item can be returned for any given lookup request, the wildcards must be processed in a predictable manner (or the same request might produce widely varying results). Thus, for each range in the language priority list, the following rules must be applied to produce a basic language range for use in the fallback mechanism:
--

Proposed change:

--
...Wildcard values in an extended language range normally match any value that occurs in that position in a language tag... 
--

And then add (after the example):

--
The resulting basic language ranges will produce different results that the user might expect. For example, a request for "en-*-US" will never find content labeled "en-Brai", even though this tag matches the extended language range.
--

Addison P. Phillips
Globalization Architect, Quest Software

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: Mark Davis [mailto:[email-masked]]
> Sent: 2005年11月18日 13:42
> To: Addison Phillips
> Cc: [email-masked]
> Subject: Re: [Ltru] draft-06 comments: choice of range
> 
> 
> 
> Addison Phillips wrote:
> 
> >Comments interlinearly below. I have some strong reservations about some
> of these comments.
> >
> >Addison
> >
> >Addison P. Phillips
> >Globalization Architect, Quest Software
> >
> >Internationalization is not a feature.
> >It is an architecture.
> >
> >
> >
> >>-----Original Message-----
> >>From: Mark Davis [mailto:[email-masked]]
> >>Sent: 2005年11月18日 10:15
> >>To: Addison Phillips
> >>Cc: [email-masked]
> >>Subject: Re: [Ltru] draft-06 comments: choice of range
> >>
> >>
> >>
> >// text removed
> >
> >
> // just keep the text, makes it harder to tell what you are commenting
> on ;-)
> 
> >>[The "this also means" only applies when the implemenation does the MAY.]
> >>
> >>
> >[Addison Phillips]
> >
> >Agreed, although I tend to think we should go the other direction and say
> MUST.
> >
> >
> If we say must, then we are really saying that any input range x-y-z-w
> acts like x-*-y-*-z-*-w, whenever something could go between. Since we
> say the algorithms work in the absence of parsing the language tag, that
> means putting them in all the time. Seems like a bit much, but I'll mull
> it over.
> 
> >>----
> >>
> >>measurment
> >>[spell-check]
> >>
> >>
> >[Addison Phillips]
> >
> >Thanks.
> >
> >
> >>----
> >>
> >>[General Observation.
> >>
> >>We need to give a mechanism that will always give a precise, determinant
> >>answer. We then let protocols specify alternatives (if they need to) by
> >>additional specification. In particular, in each section we have to make
> >>clear what exactly happens with a language range, and what happens with
> >>a language priority list.
> >>
> >>For filtering, this can be done by moving the discussion of the language
> >>priority list up into 3.2. ]
> >>
> >>
> >[Addison Phillips]
> >
> >Actually, I'd go further and suggest that all of the matching schemes be
> defined in terms of language priority lists (a language range by itself
> being the collapsed case of a language priority list with only one item--
> inside the specific scheme we describe how to match a single range to a
> single tag, but the scheme overall matches the user's language
> preferences--a list--against content). Lookup now reads this way.
> >
> That works too. Then we just say at the very top that all the mechanisms
> treat a single language range as a single element language priority list.
> 
> >This section 3.2 text:
> >
> >--
> >Filtering is used to select the set of content that matches a given
> prefix.
> >--
> >
> >Should be modified to say:
> >
> >--
> >Filtering is used to select the set of content that matches the user's
> language preferences (see Section 2.1, Language Priority Lists).
> >--
> >
> >Before objecting to this change because it removes the critical point
> about the range being a prefix, consider that the entire second paragraph
> of this section covers that.
> >
> >
> >
> >>3.2.
> >>If the language priority list (see Section 2.1 (Lists of Language
> >>Ranges)
> >><http://www.inter-locale.com/ID/draft-ietf-ltru-matching-07.html#lists>)
> >>contains more than one range, the content returned is typically ordered
> >>in descending level of preference.
> >>=>
> >>If the language priority list (see Section 2.1 (Lists of Language
> >>Ranges)
> >><http://www.inter-locale.com/ID/draft-ietf-ltru-matching-07.html#lists>)
> >>contains more than one range, the contents that match that range are
> >>added to the set of results.
> >>
> >>
> >[Addison Phillips]
> >
> >The word content in this sentence is already plural. "Contents" is wrong.
> If we want to be explicit about it, we should use the previously quoted
> formulation "set of content".
> >
> >
> set of content works.
> 
> >
> >
> >>[At the end of the section, after we've introduced the discussion of an
> >>ordered list and distance metric, add]
> >>
> >>When the set of matches is returned, the items are in descending order
> >>where the weight for each item is the priority in the language priority
> >>list divided by the distance metric. (If the filtering method does not
> >>supply a distance metric, then the distance metric is 1.) If the item
> >>matches multiple ranges, then its weight is the square root of the sum
> >>of squares of the individual weights for each range. If two items have
> >>the same weight, then their ordering is based on a case-insensitive
> >>ordering of the language tags.
> >>
> >>
> >[Addison Phillips]
> >
> >Ugh. Icky. Many items will have the same weight even in spite of this
> complex scheme.
> >
> >
> Can't get around that. If priorities are equal and there is no distance
> metric, then you essentially have an unordered list.
> 
> >I note that q weights in Accept-Language are always less than or equal to
> one and greater than or equal to zero. Your algorithm won't work correctly
> with those values (instead multiplication of the score works better).
> >
> >
> 
> You're right that there is a numeric problem, but you can't multiply,
> because smaller distances are better, but lower priorities are *worse*.
> So you have to divide one way or another. Look at an example.
> 
> X, q=0.5; Y, q=0.3;
> 
> Z matches X with a distance of, say, 2. W matches Y with a distance of
> 10. If you multiply, you get
> 
> Z -> (0.5 * 2) = 1
> W -> (0.3) * 10) =  3
> 
> That would rank Y better than Z! What you want is a division.
> 
> Now, as I think about it, the simplest way is to still think of the
> results as distances, with the smallest distance being the best. Then
> the algorithm is
> 
> a) toss out any q weights that are zero. After all, that means that you
> give them *no* priority.
> b) divide the distance by the priority.
> c) order in ascending order
> 
> Z -> (2 / 0.5) = 1
> W -> (10 / 0.3) = 33.33
> 
> which ranks them correctly.
> 
> The square rooty stuff was just because distance metrics with multiple
> fields weight better when you treat them as dimensional vectors.
> However, in retrospect it is probably just simpler to take the minimum
> of the possible weights. So if Q matches X with a distance of, say, 2. Q
> matches Y with a distance of 10. Then the result would be the minimum, or
> 1.
> 
> >>A protocol which uses language priority lists MAY specify an alternative
> >>mechanism for computing the ordering.
> >>
> >>----
> >>
> >>A protocol which uses language ranges MAY specify additional rules about
> >>the semantics of "*";
> >>=>
> >>A protocol which uses language ranges MAY specify additional rules about
> >>the semantics of "*" or other tags;
> >>[no reason to restrict it]
> >>
> >>----
> >>
> >>Implementations or protocols sometimes might wish to use more
> >>sophisticated weights that
> >>=>
> >>Implementations or protocols sometimes MAY specify the use of
> >>alternative weights that
> >>
> >>
> >[Addison Phillips]
> >
> >Either "...protocols MAY..." or "... protocols sometimes might..."
> >
> >
> sometimes MAY => MAY
> 
> >I chose non-normative language initially to allow things without turning
> the whole section into RFC 2119 soup.
> >
> >
> >>----
> >>[Make subsections in Lookup. First language range lookup, then language
> >>priority list lookup, then extended language range lookup.]
> >>
> >>
> >[Addison Phillips]
> >
> >I think that's an awful idea.
> >
> >And make up your mind: we REMOVED extended language range lookup because
> it doesn’t make sense. Lookup is now cast as I suggested above: it is
> strictly a language priority list beast, but it starts by defined the
> match between a tag and a range.
> >
> >
> Lemme relook at it, but it seems to me that you are doing it, but *via*
> transforming it. (but more in a separate message anyway).
> 
> >>----
> >>Other examples of lookup might include:
> >>=>
> >>Other examples of lookup include:
> >>[the examples *do* include. no need to be tentative ;-)]
> >>
> >>
> >[Addison Phillips]
> >
> >Okay.
> >
> >
> >>----
> >>
> >>If this language range is the only one in the language priority list, it
> >>matches the default content. If this language range is followed by other
> >>language ranges, it should be skipped.
> >>[doesn't address  *;*]
> >>
> >>
> >[Addison Phillips]
> >
> >Yes, it does... iteratively.
> >
> >
> You're right.
> 
> >
> >
> >>=>
> >>All "*" language tags are removed from language priority list. If there
> >>is nothing left, then it matches the default content.
> >>
> >>
> >[Addison Phillips]
> >
> >That's "ranges" in your first sentence.
> >
> >
> >>----
> >>
> >>5. (default content or the empty tag)
> >>=>
> >>5. (default)
> >>[explain below what this is. it is also otherwise odd that 1-4 are tags,
> >>but then this is 'content']
> >>
> >>
> >[Addison Phillips]
> >
> >Okay.
> >
> >
> >>----
> >>
> >>When performing lookup using a language priority list, the progressive
> >>search MUST proceed to consider each language range before finding the
> >>default content or empty tag. The default content might be content with
> >>no language tag (or with an empty value, as with xml:lang in the XML
> >>specification), or it might be a particular language designated for that
> >>bit of content.
> >>
> >>One common way to provide for default content is to allow a specific
> >>language range to be set as the default for a specific type of request.
> >>This language range is then treated as if it were appended to the end of
> >>the language priority list, rather than after each item in the language
> >>priority list.
> >>
> >>=>
> >>[reword for clarity. Funny thing: when I first read 'might be content
> >>with no language tag' I read it as the verb "be content with", like "be
> >>happy with". The empty tag is not a valid language tag, so we should say
> >>"und", then say that a system can use alternatives. Here is the text.]
> >>
> >>
> >[Addison Phillips]
> >
> >No, not 'und' (and I *hate* 'und'). 'Und' would be an example of an
> implementation specific decision. Personally I would prefer never to see
> 'und' content being used as a default.
> >
> >
> Well, "" is illegal as a language-tag, so I was just using it as a
> stand-in because we don't have a language tag for "root".
> 
> >>When performing lookup using a language priority list, the progressive
> >>search MUST consider each language range before finding the default. The
> >>default might be specified as "und" (or be indicated with an empty
> >>value, as with xml:lang in the XML specification, or with "root" as in
> >>some systems), or it might be specified as a particular language range.
> >>This language range is thus treated as if it were appended to the end of
> >>the language priority list, rather than after each item in the language
> >>priority list.
> >>
> >>
> >[Addison Phillips]
> >
> >I prefer not to make this change. I think it delves too closely into the
> implementation specific.
> >
> >
> It can't be left as it is. Other alternatives?
> 
> >>----
> >>
> >>5. (return default content)
> >>a. ja-JP
> >>b. ja
> >>c. (empty tag or other default content)
> >>
> >>5. (default)
> >>a. ja-JP
> >>b. ja
> >>c. und
> >>
> >>
> >[Addison Phillips]
> >
> >How about:
> >
> >5. (default content lookup)
> >   a. ja-JP
> >   b. ja
> >   c. (implementation defined)
> >
> >
> That works.
> 
> >>----
> >>
> >>Add
> >>
> >>
> >>      3.3.x. Distance Metric Lookup
> >>
> >>Distance metric lookup is performed by producing an ordered list of
> >>filtered results as specified in 3.2.2, then returning the first item.
> >>
> >>
> >[Addison Phillips]
> >
> >I think we'd best not go down that path. Among other things: it won't
> work. Distance metric filtering can produce items with identical scores...
> >
> >
> Ok, drop it.
> 
> >
> >
> >>----
> >>
> >>Mark
> >>
> >>P.S. I still think that the extended range lookup doesn't quite work,
> >>but I'll send that in a separate message.
> >>
> >>
> >[Addison Phillips]
> >
> >No, it doesn't work and we eliminated it. We've mapped extended ranges
> onto basic ranges, which do work, producing different results than the
> language priority list might lead one to expect.
> >
> >
> Let me look this over again.

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Mark Davis <[email-masked]>
Date: Fri, 18 Nov 2005 17:38:28 -0800

>I think we might make a version of the change you suggest and then spell out my example.
ok

---

From: Martin Duerst <[email-masked]>
Date: Sun, 20 Nov 2005 16:16:40 +0900

[co-chair hat on]

Please use quoting carefully. Not enough quoting is confusing, but
quoting too much should also be avoided.

[I'm on purpose quoting a full mail below.]

Thanks,      Martin.

At 10:38 05/11/19, Mark Davis wrote:
 >>When considered in isolation, I suppose one could have that reading, but 
I think the text is actually pretty clear here. Perhaps it could be 
clearer. The missing point that I was trying to convey is that extended 
language ranges do not produce the expected results based on their meaning. 
The range "en-*-US" will never pass through content labeled "en-Brai" (or 
any other content with a script subtag), even though this matches the 
meaning of the extended language range. From "en-US" you skip immediately 
to "en". It is supposed to match... and it doesn't.
 >>In some cases, the language priority list might contain one or more 
extended language ranges (as, for example, when the same language priority 
list is used as input for both lookup and filtering operations). Wildcard 
values in an extended language range are supposed to match any value that 
occurs in that position in a language tag. Since only one item can be 
returned for any given lookup request, the wildcards must be processed in a 
predictable manner (or the same request might produce widely varying 
results). Thus, for each range in the language priority list, the following 
rules must be applied to produce a basic language range for use in the 
fallback mechanism:
 >>--
 >>
 >>Proposed change:
 >>
 >>--
 >>...Wildcard values in an extended language range normally match any 
value that occurs in that position in a language tag... --
 >>The resulting basic language ranges will produce different results that 
the user might expect. For example, a request for "en-*-US" will never find 
content labeled "en-Brai", even though this tag matches the extended 
language range.
 >>>Sent: 2005$Bj;%((B11$Bk|/(B8$Bkw!&(B 13:42
 >>>To: Addison Phillips
 >>>Cc: [email-masked]
 >>>>>Sent: 2005$Bj;%((B11$Bk|/(B8$Bkw!&(B 10:15
 >>>>>To: Addison Phillips
 >>>>>Cc: [email-masked]
 >>>it doesn$Bc`QU(B make sense. Lookup is now cast as I suggested above: it is