Subject: RE: [Ltru] draft-06 comments: choice of range

From: "Addison Phillips" <[email-masked]>
Date: Fri, 18 Nov 2005 11:00:49 -0800

Comments interlinearly below. I have some strong reservations about some of these comments.

Addison

Addison P. Phillips
Globalization Architect, Quest Software

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: Mark Davis [mailto:[email-masked]]
> Sent: 2005年11月18日 10:15
> To: Addison Phillips
> Cc: [email-masked]
> Subject: Re: [Ltru] draft-06 comments: choice of range
> 
// text removed
> 
> [The "this also means" only applies when the implemenation does the MAY.]
[Addison Phillips] 

Agreed, although I tend to think we should go the other direction and say MUST.
> ----
> measurment
> [spell-check]
[Addison Phillips] 

Thanks.
> 
> ----
> 
> [General Observation.
> 
> We need to give a mechanism that will always give a precise, determinant
> answer. We then let protocols specify alternatives (if they need to) by
> additional specification. In particular, in each section we have to make
> clear what exactly happens with a language range, and what happens with
> a language priority list.
> 
> For filtering, this can be done by moving the discussion of the language
> priority list up into 3.2. ]
[Addison Phillips] 

Actually, I'd go further and suggest that all of the matching schemes be defined in terms of language priority lists (a language range by itself being the collapsed case of a language priority list with only one item--inside the specific scheme we describe how to match a single range to a single tag, but the scheme overall matches the user's language preferences--a list--against content). Lookup now reads this way. This section 3.2 text:

--
Filtering is used to select the set of content that matches a given prefix.
--

Should be modified to say:

--
Filtering is used to select the set of content that matches the user's language preferences (see Section 2.1, Language Priority Lists).
--

Before objecting to this change because it removes the critical point about the range being a prefix, consider that the entire second paragraph of this section covers that.

> 
> 3.2.
> If the language priority list (see Section 2.1 (Lists of Language
> Ranges)
> <http://www.inter-locale.com/ID/draft-ietf-ltru-matching-07.html#lists>)
> contains more than one range, the content returned is typically ordered
> in descending level of preference.
> =>
> If the language priority list (see Section 2.1 (Lists of Language
> Ranges)
> <http://www.inter-locale.com/ID/draft-ietf-ltru-matching-07.html#lists>)
> contains more than one range, the contents that match that range are
> added to the set of results.
[Addison Phillips] 

The word content in this sentence is already plural. "Contents" is wrong. If we want to be explicit about it, we should use the previously quoted formulation "set of content".

> 
> [At the end of the section, after we've introduced the discussion of an
> ordered list and distance metric, add]
> 
> When the set of matches is returned, the items are in descending order
> where the weight for each item is the priority in the language priority
> list divided by the distance metric. (If the filtering method does not
> supply a distance metric, then the distance metric is 1.) If the item
> matches multiple ranges, then its weight is the square root of the sum
> of squares of the individual weights for each range. If two items have
> the same weight, then their ordering is based on a case-insensitive
> ordering of the language tags.
[Addison Phillips] 

Ugh. Icky. Many items will have the same weight even in spite of this complex scheme.

I note that q weights in Accept-Language are always less than or equal to one and greater than or equal to zero. Your algorithm won't work correctly with those values (instead multiplication of the score works better).
> 
> A protocol which uses language priority lists MAY specify an alternative
> mechanism for computing the ordering.
> 
> ----
> 
> A protocol which uses language ranges MAY specify additional rules about
> the semantics of "*";
> =>
> A protocol which uses language ranges MAY specify additional rules about
> the semantics of "*" or other tags;
> [no reason to restrict it]
> 
> ----
> 
> Implementations or protocols sometimes might wish to use more
> sophisticated weights that
> =>
> Implementations or protocols sometimes MAY specify the use of
> alternative weights that
[Addison Phillips] 

Either "...protocols MAY..." or "... protocols sometimes might..."

I chose non-normative language initially to allow things without turning the whole section into RFC 2119 soup.
> 
> ----
> [Make subsections in Lookup. First language range lookup, then language
> priority list lookup, then extended language range lookup.]
[Addison Phillips] 

I think that's an awful idea.

And make up your mind: we REMOVED extended language range lookup because it doesn’t make sense. Lookup is now cast as I suggested above: it is strictly a language priority list beast, but it starts by defined the match between a tag and a range.
> 
> ----
> Other examples of lookup might include:
> =>
> Other examples of lookup include:
> [the examples *do* include. no need to be tentative ;-)]
[Addison Phillips] 

Okay.
> ----
> 
> If this language range is the only one in the language priority list, it
> matches the default content. If this language range is followed by other
> language ranges, it should be skipped.
> [doesn't address  *;*]
[Addison Phillips] 

Yes, it does... iteratively.

> =>
> All "*" language tags are removed from language priority list. If there
> is nothing left, then it matches the default content.
[Addison Phillips] 

That's "ranges" in your first sentence.
> 
> ----
> 
> 5. (default content or the empty tag)
> =>
> 5. (default)
> [explain below what this is. it is also otherwise odd that 1-4 are tags,
> but then this is 'content']
[Addison Phillips] 

Okay.
> 
> ----
> 
> When performing lookup using a language priority list, the progressive
> search MUST proceed to consider each language range before finding the
> default content or empty tag. The default content might be content with
> no language tag (or with an empty value, as with xml:lang in the XML
> specification), or it might be a particular language designated for that
> bit of content.
> 
> One common way to provide for default content is to allow a specific
> language range to be set as the default for a specific type of request.
> This language range is then treated as if it were appended to the end of
> the language priority list, rather than after each item in the language
> priority list.
> 
> =>
> [reword for clarity. Funny thing: when I first read 'might be content
> with no language tag' I read it as the verb "be content with", like "be
> happy with". The empty tag is not a valid language tag, so we should say
> "und", then say that a system can use alternatives. Here is the text.]
[Addison Phillips] 

No, not 'und' (and I *hate* 'und'). 'Und' would be an example of an implementation specific decision. Personally I would prefer never to see 'und' content being used as a default.
> 
> When performing lookup using a language priority list, the progressive
> search MUST consider each language range before finding the default. The
> default might be specified as "und" (or be indicated with an empty
> value, as with xml:lang in the XML specification, or with "root" as in
> some systems), or it might be specified as a particular language range.
> This language range is thus treated as if it were appended to the end of
> the language priority list, rather than after each item in the language
> priority list.
[Addison Phillips] 

I prefer not to make this change. I think it delves too closely into the implementation specific.
> 
> ----
> 
> 5. (return default content)
> a. ja-JP
> b. ja
> c. (empty tag or other default content)
> 
> 5. (default)
> a. ja-JP
> b. ja
> c. und
[Addison Phillips] 

How about:

5. (default content lookup)
   a. ja-JP
   b. ja
   c. (implementation defined)
> 
> ----
> 
> Wildcard values in an extended language range are supposed to match any
> value
> =>
> Wildcard values in an extended language range match any value
[Addison Phillips] 

No, they are *supposed* to match, but in this case they don't. They match the empty value.
> 
> ----
> 
> Add
> 
> 
>       3.3.x. Distance Metric Lookup
> 
> Distance metric lookup is performed by producing an ordered list of
> filtered results as specified in 3.2.2, then returning the first item.
[Addison Phillips] 

I think we'd best not go down that path. Among other things: it won't work. Distance metric filtering can produce items with identical scores... 

> 
> ----
> 
> Mark
> 
> P.S. I still think that the extended range lookup doesn't quite work,
> but I'll send that in a separate message.
[Addison Phillips] 

No, it doesn't work and we eliminated it. We've mapped extended ranges onto basic ranges, which do work, producing different results than the language priority list might lead one to expect.

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Mark Davis <[email-masked]>
Date: Fri, 18 Nov 2005 13:42:16 -0800

>>Subject: Re: [Ltru] draft-06 comments: choice of range
// just keep the text, makes it harder to tell what you are commenting 
on ;-)

>Agreed, although I tend to think we should go the other direction and say MUST.
If we say must, then we are really saying that any input range x-y-z-w 
acts like x-*-y-*-z-*-w, whenever something could go between. Since we 
say the algorithms work in the absence of parsing the language tag, that 
means putting them in all the time. Seems like a bit much, but I'll mull 
it over.

>Actually, I'd go further and suggest that all of the matching schemes be defined in terms of language priority lists (a language range by itself being the collapsed case of a language priority list with only one item--inside the specific scheme we describe how to match a single range to a single tag, but the scheme overall matches the user''s language preferences--a list--against content). Lookup now reads this way. 
>
That works too. Then we just say at the very top that all the mechanisms 
treat a single language range as a single element language priority list.

>This section 3.2 text:
set of content works.

>Ugh. Icky. Many items will have the same weight even in spite of this complex scheme.
Can't get around that. If priorities are equal and there is no distance 
metric, then you essentially have an unordered list.

>I note that q weights in Accept-Language are always less than or equal to one and greater than or equal to zero. Your algorithm won't work correctly with those values (instead multiplication of the…

You're right that there is a numeric problem, but you can't multiply, 
because smaller distances are better, but lower priorities are *worse*. 
So you have to divide one way or another. Look at an example.

X, q=0.5; Y, q=0.3;

Z matches X with a distance of, say, 2. W matches Y with a distance of 
10. If you multiply, you get

Z -> (0.5 * 2) = 1
W -> (0.3) * 10) =  3

That would rank Y better than Z! What you want is a division.

Now, as I think about it, the simplest way is to still think of the 
results as distances, with the smallest distance being the best. Then 
the algorithm is

a) toss out any q weights that are zero. After all, that means that you 
give them *no* priority.
b) divide the distance by the priority.
c) order in ascending order

Z -> (2 / 0.5) = 1
W -> (10 / 0.3) = 33.33

which ranks them correctly.

The square rooty stuff was just because distance metrics with multiple 
fields weight better when you treat them as dimensional vectors. 
However, in retrospect it is probably just simpler to take the minimum 
of the possible weights. So if Q matches X with a distance of, say, 2. Q 
matches Y with a distance of 10. Then the result would be the minimum, or 1.

>Either "...protocols MAY..." or "... protocols sometimes might..."
sometimes MAY => MAY

>And make up your mind: we REMOVED extended language range lookup because it doesn’t make sense. Lookup is now cast as I suggested above: it is strictly a language priority list beast, but it starts…
Lemme relook at it, but it seems to me that you are doing it, but *via* 
transforming it. (but more in a separate message anyway).

>Yes, it does... iteratively.
You're right.

>No, not 'und' (and I *hate* 'und'). 'Und' would be an example of an implementation specific decision. Personally I would prefer never to see 'und' content being used as a default.
Well, "" is illegal as a language-tag, so I was just using it as a 
stand-in because we don't have a language tag for "root".

>I prefer not to make this change. I think it delves too closely into the implementation specific.
It can't be left as it is. Other alternatives?

>   c. (implementation defined)
That works.

>No, they are *supposed* to match, but in this case they don't. They match the empty value.
I read that differently. When you say "supposed to match", it sounds 
like there is a choice, that there are some values that they don't 
match. Like the US is "supposed to" disallow torture.

>I think we'd best not go down that path. Among other things: it won't work. Distance metric filtering can produce items with identical scores... 
Ok, drop it.

>No, it doesn't work and we eliminated it. We've mapped extended ranges onto basic ranges, which do work, producing different results than the language priority list might lead one to expect.
Let me look this over again.

---

From: Mark Davis <[email-masked]>
Date: Sat, 19 Nov 2005 13:25:13 -0800

Here is the problem I have with the current formulation of "*" in 
filtering. There is a conflict between the user expectations and the 
need for a determinant answer.

A. If a user has "sr-*-BA", and the actual content is one of:

sr
sr-Cyrl
sr-Cyrl-BA
sr-Latn

Then I believe the user's perfectly reasonable expectations would be 
that "sr-Cyrl-BA" would match. That is an exact, single match against 
the pattern.

B. However, if the content is

sr
sr-Cyrl
sr-Cyrl-BA
sr-Latn
sr-Latn-BA

then we can't get a determinant answer from that pattern: both 
sr-Cyrl-BA and sr-Latn-BA are perfectly reasonable matches.

Your proposal has the advantage of getting a determinant answer, but not 
matching the user expectations. You transform the sr-*-BA into sr-BA, 
then do the lookup. In both cases A and B above, you'd return "sr", 
which is not what the user would expect.

My view is that there are two possible alternatives.

=====

1. Continue the fallback unless there is a *single* match, adding text 
to the effect that:

    As usual, a wildcard matches against a subtag in that position. When
    filtering, however, the fallback process returns contents if and
    only if there is exactly one match against the language priority
    list. If there are two possible matches against the wildcard (or
    zero) then the fallback continues. Thus suppose we are filtering
    with the language priority list "sr-Cyrl-BA", and content for the
    following tags:

    sr
    sr-Cyrl
    sr-Cyrl-BA
    sr-Latn

    The result is the content for "sr-Cyrl-BA". When filtering with the
    language priority list "sr-*", and the same possible content, the
    result is the content for "sr".

=====

2. Leave out wildcards entirely, replacing the discussion of wildcards 
(in two places) with text to the effect that:

    Wildcards are not permitted in filtered lookup, because they allow
    for the possiblity of indeterminant answers. If, for example, the
    language priority list is "sr-*" and the content is the following,
    then the results would be indeterminant.

    sr
    sr-Cyrl
    sr-Cyrl-BA
    sr-Latn

====
In both cases, add:

    However, a protocol MAY add additional specifications to filtered
    lookup to provide for for resolution of cases where there are
    multiple matches.

---

From: Mark Davis <[email-masked]>
Date: Sat, 19 Nov 2005 16:19:20 -0800

I'm sorry,  I really jumbled my message. What I meant was *lookup*; 
replace the word "filter" by that (mutatis mutandis) and see if it makes 
more sense. -- Mark

Mark Davis wrote:

> Your proposal has the advantage of getting a determinant answer, but 
> not matching the user expectations. You transform the sr-*-BA into 
> sr-BA, then do the lookup. In both cases A and B above, you'd return 
> "sr", which is not what the user would expect.
>
>

---

From: Martin Duerst <[email-masked]>
Date: Sun, 20 Nov 2005 16:31:10 +0900

I don't really see the point of throwing out two (or more) choices
and go for something that easily could be seen as matching less just
to avoid having to throw a dice. Or did I misunderstand something?

 >> Here is the problem I have with the current formulation of "*" in 
filtering. There is a conflict between the user expectations and the need 
for a determinant answer.
 >> Then I believe the user's perfectly reasonable expectations would be 
that "sr-Cyrl-BA" would match. That is an exact, single match against the 
pattern.

Yes, agreed.

So can't we just choose one of them, at random?

 >> Your proposal has the advantage of getting a determinant answer, but 
not matching the user expectations. You transform the sr-*-BA into sr-BA, 
then do the lookup. In both cases A and B above, you'd return "sr", which 
is not what the user would expect.

I'm not sure that there would be that many variants, anyway.
If there is sr-Cyrl and sr-Latn, then sr is most probably just
a second mention of the content labeled by either sr-Cyrl or sr-Latn.
Of course, in the general case, that may be different.

 >>    result is the content for "sr".

As I said above, I wouldn't think that makes sense. If there are two
matches that are 'better' than anything else, but can't be distinguished,
let's just return one of them at random or in an implementation dependent
fashion.

 >>    lookup to provide for for resolution of cases where there are

detail: "for for" -> "for"

Two concerns: 1) Adding wildcards back in will need a lot of careful work
which we shouldn't expect the users of our draft to need. Forbidding it
for cases that don't want them may be a lot easier for users, although
a bit more work for us.
2) It is easily possible that in a system (say database,...), there
are multiple variants of content that are tagged exactly the same.
So the whole effort of avoiding ties because of slightly differently
tagged content may be fruitless anyway.

Regards,    Martin.

---

From: Mark Davis <[email-masked]>
Date: Sun, 20 Nov 2005 23:22:19 -0800

Martin Duerst wrote:

> replace the word "filter" by that (mutatis mutandis) and see if it 
> makes more sense. -- Mark
>
> >> Then I believe the user's perfectly reasonable expectations would 
> be that "sr-Cyrl-BA" would match. That is an exact, single match 
> against the pattern.
>
> Yes, agreed.
>

At random really won't do. If we want to do anything, we should have a 
determinant rule, such as "return the content for the language tag that 
is alphabetically less (case-insensitive)."

It already has (in my formulation) an escape clause at the end, so 
someone can override this behavior.

>
> >> Your proposal has the advantage of getting a determinant answer, 
> but not matching the user expectations. You transform the sr-*-BA into 

In the general case, especially with en-*, es-*, *-CA, etc, there will 
be many possible matches.

> >> 1. Continue the fallback unless there is a *single* match, adding 
> text to the effect that:
> >>

as above.

>
> >> =====
> >>
> >> 2. Leave out wildcards entirely, replacing the discussion of 
> wildcards (in two places) with text to the effect that:
> >>

I'm not really adding the ability to have wildcards in lookup back in. 
On *input* the current draft handles them; it just handles them in a way 
that I don't think works (by removing then doing a lookup, and getting 
suboptimal results).

> tagged content may be fruitless anyway.

Remember, we are talking lookup, not filtering. In that case it is 
pointless to have two pieces of content (such as a ResourceBundle) with 
the same tag. You don't want the choice of return to be random!

---

From: Martin Duerst <[email-masked]>
Date: Mon, 21 Nov 2005 18:09:15 +0900

At 16:22 05/11/21, Mark Davis wrote:

[co-chair hat on] Mark, please try to cite a bit less. Thanks.

[co-chair hat off] See below.

 >Remember, we are talking lookup, not filtering. In that case it is 
pointless to have two pieces of content (such as a ResourceBundle) with the 
same tag. You don't want the choice of return to be random!

Ok, I see that true 'random' doesn't make sense in such a case.
But we could easily say that the returned piece is
'implementation defined'. That would avoid us having to make
an arbitrary choice (and getting into complaints like that we
preferred English over French or Japanese because we proposed
alphabetic order) while still allowing implementations to be
predictive.

Regards,    Martin.

---

From: Mark Davis <[email-masked]>
Date: Mon, 21 Nov 2005 08:53:49 -0800

When something has that big a hole in it, then we are then probably best 
off saying that *anything* with wildcards in lookup is 
implementation-defined, and not trying to specify it any further than 
that. (That has no impact on the use of wildcards in filtering, of course.)

Mark

Martin Duerst wrote:

> pointless to have two pieces of content (such as a ResourceBundle) 
> with the same tag. You don't want the choice of return to be random!
>