Subject: Re: [Ltru] Re: weights in scored filtering

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 20 Dec 2005 10:15:05 +0100

--On fredag, november 18, 2005 17:37:05 -0800 Mark Davis 
<[email-masked]> wrote:

> We can do an integer only approach. Convert the q's into round(100/q).
>
> For filtering, we can't assume anything about the order of items in the
> list that have the same q values. I have en, de, fr, it, nl. But my
> English is significantly better than my German, which is somewhat better
> than my French, which is significantly better than my Italian, etc.
>
> For lookup, we and everybody I know just uses the q values to reorder the
> list. Then throws them away. So for lookup, ordering is very important.

If we want to have ordering and q values both "work", we might have to bite 
the bullet and define them.... I'd suggest something like

If an item in the list does not have a q value, its q value is assumed to 
be 0.75 times the q value of the preceding item in the list.
If the first item in the list does not have a q value, its q value is 
assumed to be 1.

So the two lists

  en;q=1.0, fr;q=0.75, de

  en, fr, de;q=0.5625

would be completely identical.
That at least gives us a consistent, and mostly intuitive, interpretation 
of such lists.

                   Harald

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Mark Davis <[email-masked]>
Date: Tue, 20 Dec 2005 07:32:44 -0800

I suspect that would break a lot of people currently unless it were 
optional; as I said, we and at least some others just use the q values 
to re-sort.

Harald Tveit Alvestrand wrote:

> If we want to have ordering and q values both "work", we might have to 
> bite the bullet and define them.... I'd suggest something like
>
> If an item in the list does not have a q value, its q value is assumed 
> to be 0.75 times the q value of the preceding item in the list.
> That at least gives us a consistent, and mostly intuitive, 
> interpretation of such lists.
>
>                   Harald
>
>
>
>
>
>
>

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 20 Dec 2005 16:50:05 +0100

--On tirsdag, desember 20, 2005 07:32:44 -0800 Mark Davis 
<[email-masked]> wrote:

> optional; as I said, we and at least some others just use the q values to
> re-sort.

My sneaky suspcion is that *all* implementations will behave bizarrely on a 
list that mixes sequences and Q values.

So I don't think we have current implementations....

---

From: "Jukka K. Korpela" <[email-masked]>
Date: Wed, 21 Dec 2005 10:44:47 +0200 (EET)

> <[email-masked]> wrote:
- -
> the bullet and define them.... I'd suggest something like

This is getting very confusing. How did we get here? Maybe it's just me,
but it seems that a fairly complicated issue is getting unnecessarily
complicated. Please excuse me if I have missed the reasons that make
such complication unavoidable.

> If an item in the list does not have a q value, its q value is assumed to be 
> 0.75 times the q value of the preceding item in the list.

That sounds both arbitrary and incompatible with e.g. the Accept-Language
header in HTTP, where the order is not significant (though the client 
might conceivable use the order as the last criterion in resolution)
and the q value defaults to 1.

Regarding what Mark Davis wrote, in the user side, it's rather the 
opposite. A user interface for setting language preferences in a 
web browser typically lets the user select a set of languages and put 
them into some order. This order will, invisibly to the user, dictate the 
q values that the browser generates into an Accept-Language header.
Browsers use different algorithms in computing the values. Thus, the user 
who selects, say, French and English in that order does not really know 
(and cannot control) how much better her French is than her English
according to the preference settings.

However, although this is mostly just Greek to most users, the principle 
that preferences are not a simple ordered list is an important one.
Some quantitative indicators are needed. I don't know about anything
essentially better than the q range from 0 to 1. It would seem natural
to keep things that way, even though we know that the actual q values are, 
at present, rather arbitrary even if the user knows what he is doing.

This means that the order in which the languages or locales are listed
somewhere should be completely immaterial; it should be treated as a 
linearized set, not as an ordered list. This allows ties, i.e. languages 
with exactly the same q value, and I think that's _good_. If the user is 
allowed to express language preferences, then a choice that makes two 
languages _equally_ preferred should be allowed too (without raising any
theoretical objections about the impossibility of knowing two languages
exactly equally well).

Naturally, this does not prevent anyone from doing whatever he wants
with the preferences and q values, such as using them _only_ to set up
an ordered list, or to filter out everything below a specific q value.

-- 
Jukka "Yucca" Korpela, http://www.cs.tut.fi/~jkorpela/

---

From: Mark Davis <[email-masked]>
Date: Wed, 21 Dec 2005 18:51:53 -0800

For those who might be interested, I just updated CLDR to use the data 
in http://www.ietf.org/internet-drafts/draft-ietf-ltru-initial-06.txt 
for the language tag part of locale ids. As it turns out, we then only 
have the following few extensions:

        {"language", "root", "Description", "Root", "CLDR", "True"},
        {"variant", "POLYTONI", "Description", "Polytonic Greek", 
"CLDR", "True"},
        {"variant", "REVISED", "Description", "Revised Orthography", 
"CLDR", "True"},
        {"variant", "SAAHO", "Description", "Dialect", "CLDR", 
"True"},           
        {"region", "QE", "Description", "European Union", "CLDR", 
"True"},           

The code for root is just a special case for CLDR, and I don't see a 
good alternative in 3066bis. Possibly we could have used "und", but by 
now it is pretty baked.

POLYTONI we probably want to recast as a private use script code, eg. 
Qaap. That way inheritance will work better.

REVISED is only used for Armenian (hy-AM-REVISED)
(cf http://dev.icu-project.org/cgi-bin/locale-bugs/confirm?id=938)

Question: according to 
http://www.ethnologue.com/show_language.asp?code=hye there is no code 
for Eastern vs Western Armenian. Will ISO be adding this as part of the 
next revision? It appears that there are distinct differences in 
orthography that we need to account for, at least according to 
http://en.wikipedia.org/wiki/Armenian_language

SAAHO is only used in ar-ER-SAAHO in CLDR
(http://unicode.org/cldr/data/common/main/aa_ER_SAAHO.xml)
According to SIL, we could be using ssy for that.
http://www.ethnologue.com/show_language.asp?code=aar
http://www.ethnologue.com/show_language.asp?code=ssy

Although not now in the registry, it looks like it will come in with the 
next version of ISO 639, so we could deprecate it at that point.

QE is also a special case for CLDR, needed internally for other 
purposes, and is not used in composing language tags.

Mark

Cc'ing Michel, since he is my best Armenian contact.