Subject: RE: [Ltru] Re: weights in scored filtering

From: "Addison Phillips" <[email-masked]>
Date: Tue, 20 Dec 2005 08:57:28 -0800

As far as I know, the only implementation of scored filtering--an algorithm proposed by John Cowan on this list as a thought experiment--is the one on my website. Every filtering implementation that I'm aware of is some form of the basic or extended algorithms. So we won't be fighting a large installed base with this particular algorithm.

I'm not sure that the algorithm currently makes a lot of sense. We added text in draft-08 that says:

--
As with the other forms of filtering, the process considers each language range in the language priority list in order of priority.
--

But it doesn't go on to say anything about how to consider each range compared to the others. Would the assumption be something like:

--
As with the other forms of filtering, the process considers each language range in the language priority list in order of priority. Each range in the language priority list is compared to each language tag. The lowest weighted score for the language priority list is the closest match and is the score for that tag. Scores are weighted by dividing the score by the quality weight of the language range in the language priority list. 
--

This would mean, as Harald points out, providing a "rule for thumb" for weighting. Perhaps we could add to section 2.3:

--
Some algorithms require that the language priority list be prioritized or "quality weighted". If a particular form of language priority list does not assign weights to one or more of its items, then a weight of 0.75 times the previous item in the list is suggested (with the first item being assumed to have a weight of 1.0).
--

Addison P. Phillips
Globalization Architect, Quest Software

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: Harald Tveit Alvestrand [mailto:[email-masked]]
> Sent: 2005年12月20日 7:50
> To: Mark Davis
> Cc: Addison Phillips; [email-masked]
> Subject: Re: [Ltru] Re: weights in scored filtering
> 
> 
> 
> --On tirsdag, desember 20, 2005 07:32:44 -0800 Mark Davis
> <[email-masked]> wrote:
> 
> > I suspect that would break a lot of people currently unless it were
> > optional; as I said, we and at least some others just use the q values
> to
> > re-sort.
> 
> My sneaky suspcion is that *all* implementations will behave bizarrely on
> a
> list that mixes sequences and Q values.
> 
> So I don't think we have current implementations....
> 
> 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 21 Dec 2005 12:51:01 +0100

Addison Phillips wrote:

> As far as I know, the only implementation of scored
> filtering--an algorithm proposed by John Cowan on this
> list as a thought experiment--is the one on my website.

I've compared his original idea 8-4-2-1 with other weights.

Admittedly I'm a bit lost since you went to five dimensions
and a distance metric.  IIRC it was Mark who said that any
scheme involving "divide by Q" has a problem with Q = 0.

You can avoid this problem by scoring, with highest score =
ideal match, followed by "multiply with Q".

To switch from some kind of distance metric (0 = ideal) to
scoring (0 = worst) you only need the maximal distance M.

If that exists.  Since you have in theory unlimited numbers
of variants and/or extensions it could be M = infinity, and
you lose.

Otherwise you can define score := M - distance to get some
scores in the range 0 .. M, multiply it with any given Q
in the range 0.00 .. 1.00, and that's your final score for
the language-range in question.

Implementations can of course scale it down to 0..100, 100
for M (best score), 50 for M/2, 0 for 0, etc., but we don't
need to discuss such implementation details in an RFC.

The language-range with the highest final score wins.  It's
also unnecessary to discuss draws:  Implementations can be
free to handle this in any way they wish, not restricted to
"reboot", "offer checkbox with 1,000,000 possible choices",
or similar stunts... ;-)

> If a particular form of language priority list does not
> assign weights to one or more of its items, then a weight
> of 0.75 times the previous item in the list is suggested
> (with the first item being assumed to have a weight of 1.0)

With 0.75 you get about 19 steps until you drop below 0.005.
With 0.9 you get more steps.  That's irrelevant, isn't it ?
Any number greater than 0 and smaller than 1 will do.

Again an implementation detail, implementations may offer
to transform prioritized lists of language-ranges into some
kind of weighted lists by assigning appropriate weigths:

The Q as we know it, any sequence 1 = q(1) > q(2) etc. will
do, if 1 > q(n) > q(n+1) > 0 for all n > 1.  Is it really
desirable to discuss such details in an Internet standard ?

                          Bye, Frank