Subject: [Ltru] Proposal for -matching: "scored filtering" > "scoring"

From: John Cowan <[email-masked]>
Date: Tue, 21 Feb 2006 14:34:43 -0500

I am proposing that we change "scored filtering" to "scoring" throughout.
Filtering is about taking a bunch of tags and removing some while keeping
others.  Scoring takes a bunch of tags and puts them in an order.  These are
fundamentally different operations.

This would involve renumbering 3.2.3 as 3.3 and 3.3 as 3.4, since filtering,
scoring, and lookup would now have equal status.  Otherwise,
there would be a handful of local editorial changes.

-- 
Kill Gorgûn!  Kill orc-folk!            John Cowan
No other words please Wild Men.         [email-masked]
Drive away bad air and darkness         http://www.ap.org
with brig ht iron!  --Ghân-buri-Ghân    http://www.ccil.org/~cowan

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 21 Feb 2006 12:02:13 -0800

I agree. If none opposed I will make the change in my proto-draft-10 later
today. 

Note: I am trying to implement my previous suggestions regarding extended
filtering at the same time. In order to do that I am splitting out the
original text, in case we want to include both Kent's proposal ("extended
patterns"?!?) and my own.

Addison

Addison Phillips
Internationalization Architect - Yahoo! Inc.

Internationalization is an architecture.
It is not a feature. 

> -----Original Message-----
> From: John Cowan [mailto:[email-masked]]
> Sent: Tuesday, February 21, 2006 11:35 AM
> To: [email-masked]
> Subject: [Ltru] Proposal for -matching: "scored filtering" > "scoring"
>

---

From: Mark Davis <[email-masked]>
Date: Tue, 21 Feb 2006 14:27:17 -0800

It could be separated, but since the main reason for using scoring is 
for filtering, I don't think it is really worth it.

And frankly, at this point, I'm starting to think that we'd be better 
off without Scored Filtering at all in this version; that we should just 
get this out so that it doesn't delay publication of the registry 
document, and work on Scored Filtering in a future version.

Mark

---

From: John Cowan <[email-masked]>
Date: Tue, 21 Feb 2006 17:28:50 -0500

> document, and work on Scored Filtering in a future version.

I wouldn't die if that happened.

-- 
John Cowan  [email-masked]  www.ccil.org/~cowan  www.ap.org
If I have seen farther than others, it is because I am surrounded by dwarves.
        --Murray Gell-Mann

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 21 Feb 2006 15:08:06 -0800

It turns out to be a bit painful, but not quite impossible. 

In an attempt to placate Kent I did as described in my previous note and implemented a full language pattern syntax alongside my proposal for extended filtering. It required that I invent a new negation wildcard. Not sure I'm overly fond of the results. At the same time, I married it to "scoring". 

The resulting document is viewable on inter-locale:

  http://www.inter-locale.com/ID/draft-ietf-ltru-matching-10.txt
  http://www.inter-locale.com/ID/draft-ietf-ltru-matching-10.html
  http://www.inter-locale.com/ID/draft-ietf-ltru-matching-10.xml

We should decide:

A) Do we need a full pattern syntax or was my little proposal "enough"?

B) Do we need scoring?

If the answer to both of the above is "NO", then we should remove all mention of both and publish. I nominate that we answer both "NO".

Extended reason: there is nothing in this document that says these are the *only* matching schemes for language tags. Anyone may publish an informational or individual submission RFC containing an additional scheme, range/pattern syntax, and set of use cases. In fact, draft-matching should encourage this as the pattern for future matching scheme documentation. I will happily contribute the current text for language-pattern and scored filtering, along with an implementation of same, to anyone who wishes to make such an RFC out of that scheme.

Addison

Addison Phillips

> From: Mark Davis [mailto:[email-masked]]
> Sent: Tuesday, February 21, 2006 2:27 PM
> To: Addison Phillips
> Cc: 'John Cowan'; [email-masked]
> Subject: Re: [Ltru] Proposal for -matching: "scored filtering" > "scoring"
> 
> >> others.  Scoring takes a bunch of tags and puts them in an order.
> These
> >> are

---

From: John Cowan <[email-masked]>
Date: Tue, 21 Feb 2006 18:33:37 -0500

Addison Phillips scripsit:

> In an attempt to placate Kent I did as described in my previous note
> and implemented a full language pattern syntax alongside my proposal
> for extended filtering. It required that I invent a new negation
> wildcard. Not sure I'm overly fond of the results. At the same time,
> I married it to "scoring".

Bah.  My little squib, which was just intended to show that there are
other possibilities besides matching and lookup, has now grown completely
out of the draft of control.  Rip it out and hang it up to dry.

> If the answer to both of the above is "NO", then we should remove all
> mention of both and publish. I nominate that we answer both "NO".

+2

-- 
A witness cannot give evidence of his           John Cowan
age unless he can remember being born.          [email-masked]
  --Judge Blagden                               http://www.ccil.org/~cowan

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 22 Feb 2006 01:59:04 +0100

> B) Do we need scoring?

John's idea to use three steps for up to three <extlang>s with
2048, 1024, 521, 256 (3rd extlang) is fine,  128 for <script>
is also okay,  You omitted 64, why ?  And why is a region more
important (32) than a variant (4) or an extension (1) ?

Why not simply 1 for region / variant / extension ?  If there
are more than 128 overruling the <script> they might actually
mean something.  How about 160, 80, 40, 20, 10 for the first
five dimensions, and 1 for the zoo region / variant / extension
- admitting that scoring is merely a hack to be used with a
Language-Priority-List as specified for HTTP with q-values ?

The scoring chapter shouldn't be normative, it's an example how
to implement 2.4 (priority list) together with 3.4 (lookup).
3.3 is an example for 3.4, maybe join them into 3.3 plus 3.3.1.

                           Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 22 Feb 2006 01:32:42 +0100

> http://www.inter-locale.com/ID/draft-ietf-ltru-matching-10.txt

The 2.1 ABNF is still _very_ wrong, fixed ABNF posted here
about three or four times.

> A) Do we need a full pattern syntax or was my little proposal

The 2.2 ABNF is too minimalistic, build on top of 3066bis as
discussed (e.g. there is no tag or range starting with a subtag
containing digits, same issue as in 2.1).

2.3 is completely different from anything discussed here in the
past weeks.  We agreed that more than one <variant-pat> is not
a good idea.  What is a <privateuse-pat>, nobody ever proposed
it here, and why doesn't it have a "!" like all other *-pat ?

A syntax allowing (in theory) unlimited numbers of "*" or "!"
is utter dubious, you have this bug/feature with <variant-pat>
and <privateuse-pat>.

The syntax for <extension-pat> and <privateuse-pat> should be
almost identical, if you want the new <privateuse-pat> at all.

Just take the last version of Kent's proposal and add "!" in
all places where you see "*", as posted / validated / discussed
here.  Okay, not in <primary-pat>, that's okay.  OTOH you have
unreferenced <subtag> and <extlang-pat> in 2.3, and a missing
<extlang>.

Why do you need "!" at all ?  With embedded stars you can just
say that unspecified means NOT, and no "*" at the end means no
additional subtags, as in your "de" example.  The "!" is not
necessary with explicit embedded / trailing stars.

Please use something in the style of Kent's syntax _plus_ fixes
as discussed / validated here.  Last known 2.1 + 2.2 states in
<http://article.gmane.org/gmane.ietf.ltru/4431> replacing
-   basic-range   = basic-range / "*"
+   basic-range   = basic-tag / "*"
(that basic part is of course only for 2.1)

The second ABNF variant is what you need for 2.2.  The first
variant is a start for 2.3, but that's far from clear, do we
need "!" at all with explicit stars everywhere ?

                         Bye, Frank

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 21 Feb 2006 17:56:47 -0800

Some small comments follow. Please note that I have updated the editor's
copy on inter-locale to have a correct algorithm description for extended
matching. I have also placed a version of same with patterns/scoring ripped
out at this different URI:

   http://www.inter-locale.com/ID/draft-ietf-ltru-matching-10-var1.txt
   http://www.inter-locale.com/ID/draft-ietf-ltru-matching-10-var1.html
   http://www.inter-locale.com/ID/draft-ietf-ltru-matching-10-var1.xml

See below...

Addison

Addison Phillips

> From: Frank Ellermann [mailto:[email-masked]]
> Sent: Tuesday, February 21, 2006 4:33 PM
> To: [email-masked]
> Subject: [Ltru] Too many ABNF issues (was: Proposal for -matching:
> "scoredfiltering" > "scoring")
> 
> Addison Phillips wrote:
> 

Yep. My bad. I did want a general reaction to the form of the document,
tho'.
> containing digits, same issue as in 2.1).

Why? My proposal was to base it on basic range and allow Bad Ranges to exist
which do not match anything valid. I'm happy to build a more 3066bis-like
syntax, but only if it adds to understanding of the document. Would extended
ranges have to validate? With the new algorithm, they do not need the
extra-fussy syntax except to provide it for algorithms we might not include.
Only scoring depends to any great depth on well-formed ranges. Perhaps I am
trying to do too much with the scoring algorithm, tho'.
> it here, and why doesn't it have a "!" like all other *-pat ?

If you specify a private use pattern, you cannot negate what follows (if you
don't want to match "x" at all, I don't know what you do). What would this
pattern mean:  "foo-x-!"

It means "foo followed by an illegal 'x' singleton with no subtag
following..."

The mistake is allowing negation in the extension field.
> is utter dubious, you have this bug/feature with <variant-pat>

Agreed. Fixed in the ABNF below.
> necessary with explicit embedded / trailing stars.

No, I can't say that and have the pattern work correctly. What does
"de-*-1996" represent? It may need to say "de-!-*-1996" (no script, yes
region) or the converse "de-*-!-1996" (yes script, no region). And that
*ignores* extlangs. In order for the pattern syntax to be fully expressive
in the way that Kent suggested, I have to allow for negation of a field
*explicitly*. Otherwise we're just playing with stars, cf. Dr. Seuss's
Sneeches. Kent didn't go that far with his ABNF but did go that far in his
correspondence. The only way to have "unspecified" specified is to have
"de--*" (for example), which is also icky.

> need "!" at all with explicit stars everywhere ?

Do we need "pattern" at all if we change the algorithm as proposed? I don't
think we do.

Note a subtlety in variant: you do not need a "*" at the end of a variant
sequence. You only need it if there are no variants at all. The same goes
with negation, although one might want to say "de-1996-!" to mean "exactly
de-1996, no de-1996-x-foo"

Here is a corrected ABNF (minus terminology changes), unposted as yet:

---
language-pattern  = (primary-pat
                       script-pat
                       region-pat
                       variant-pat
                       *(extension-pat)
                       ["-" privateuse-pat])
                    / privateuse-pat 
                    / grandfathered

primary-pat   = (2*3ALPHA [ extlang ]) ; shortest ISO 639 code
              / 4ALPHA                 ; reserved for future use
              / 5*8ALPHA               ; registered language subtag
              / "*"                    ; or wildcard; negation not permitted

extlang-pat   = *2("-" 3ALPHA) ("-" ( 3ALPHA / "*" / "!"))
                                       ; reserved for future use
                                       ; wildcards can only appear
                                       ;   at the end

script-pat    = "-" 4ALPHA             ; ISO 15924 code
              / "-*"                   ; wildcard
              / "-!"                   ; or negation

region-pat    = "-" 2ALPHA             ; ISO 3166 code
              / "-" 3DIGIT             ; UN M.49 code
              / "-*"                   ; wildcard
              / "-!"                   ; or negation

variant-pat   = 1*("-" (5*8alphanum / DIGIT 3alphanum))
              / "-*"                   ; wildcard
              / "-!"                   ; or negation

extension-pat = "-" singleton *("-" (2*8alphanum))
                [ "-*" ]               ; extension sequence
                                       ; wildcards can only appear
                                       ;   at the end

singleton     = %x41-57 / %x59-5A / %x61-77 / %x79-7A / DIGIT
              ; single letters (except for "x") or digits

privateuse-pat = "x" 1*("-" subtag / "*")
                                       ; negation not permitted

subtag         = 1*8alphanum

grandfathered  = 1*3ALPHA 1*2("-" (2*8alphanum))
                ; grandfathered registration
                ; wildcard not permitted here

alphanum       = (ALPHA / DIGIT)
                ; letters and numbers
---

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 21 Feb 2006 18:10:08 -0800

I saw a lot of +1 for using a simple ABNF and associated matching scheme.
Only Kent appeared to me to demur. I put some of his proposals into a
parallel set of text to give us a chance to see them side-by-side. I think
we should abandon the original extended language range in favor of the
simpler proposal. If there is a need for the more 3066bis-like syntax, we
should work on it, but I think we should have an algorithm in mind at the
same time. As it sits, extended filtering doesn't need the extra syntax. I
can imagine processes that do.

I'll review the ABNF's posted previously tomorrow and incorp as necessary.
In the meantime... I take it you do not concur with my two "NO" votes,
Frank?

Addison

Addison Phillips

> From: Addison Phillips [mailto:[email-masked]]
> Sent: Tuesday, February 21, 2006 5:57 PM
> To: 'Frank Ellermann'; [email-masked]
> Subject: RE: [Ltru] Too many ABNF issues (was: Proposal for -
> matching:"scoredfiltering" > "scoring")
>

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 22 Feb 2006 04:03:57 +0100

> See below...

Okay.  I just tried for two hours to create a syntax where
it's clear (by counting positions) what a "*" or "!" stand
for:  It fails miserably.  We'd need -*-*-* to bridge the
gap of potential extlangs for all alpha2 / alpha3 languages.

Not impossible, but too horrible to consider it seriously.

>> The 2.2 ABNF is too minimalistic, build on top of 3066bis
>> as discussed (e.g. there is no tag or range starting with
>> a subtag containing digits, same issue as in 2.1).

> Why? My proposal was to base it on basic range and allow
> Bad Ranges to exist which do not match anything valid. I'm
> happy to build a more 3066bis-like syntax, but only if it
> adds to understanding of the document.

The simple "Texas"-syntax is ready, no need to build anything,
just rename what you don't like avoiding all 3066bis names.

If you think that bogus "Texas"-ranges matching nothing are no
problem you'd get this derived from <basic-range>:

   language-range = ( 1*8ALPHA / "*" ) *( "-" 1*8alphanum )

> What would this pattern mean:  "foo-x-!"

Maybe it means "no privateuse", same idea as "foo-y-!".

> The mistake is allowing negation in the extension field.

Okay, no "!" for both x and y, that's something I understand.

> Do we need "pattern" at all if we change the algorithm as
> proposed? I don't think we do.

Well, we don't need to quibble about 2.3 if we don't need it.

I guess we're trapped by the <extlang>s, that was the final
point of no return killing any positional stars.  Unless you
try **** for any script, ** for any region, etc., different
kinds of wildcards.

> Note a subtlety in variant: you do not need a "*" at the end
> of a variant sequence. You only need it if there are no
> variants at all.

If there are pidgin-north and pidgin-south, why no pidgin-* ?

> primary-pat   = (2*3ALPHA [ extlang ]) ; shortest ISO 639
                              ^^^^^^^
It's extlang-pat here.

> extlang-pat   = *2("-" 3ALPHA) ("-" ( 3ALPHA / "*" / "!"))

But what is de-!-CH ?  Is that no <extlang> or no <script> ?
We're doomed, the <extlang> kills all those cute shorthands.

>                 [ "-*" ]               ; extension sequence

What is the difference between y-* and no y at all ?  That can
be fixed by demanding at least one real y-subtag.  Dito for a
privateuse-pat.
                         Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 22 Feb 2006 04:22:55 +0100

Addison Phillips wrote:

> I take it you do not concur with my two "NO" votes, Frank?

For "B" I'd say "scoring is at best an example for lookup
together with q-priorities".  Your idea to kill it is a bit
harder, but why not, we can insert it again later if we need
an example for some "numerical priority-lookup".

For "A" I didn't get your revised 2.2 idea, with consistent
ABNF in 2.1 and 2.2 it's no problem:

2.1 is either * or tag.  2.2 is either * or primary-tag,
followed by a subtag sequence, two one-liners:

 basic-range    = "*" / ( 1*8ALPHA *( "-" 1*8alphanum ) ) ; 2.1
 language-range = ( "*" / 1*8ALPHA ) *( "-" 1*8alphanum ) ; 2.2

Without new <language-tag>, that's already defined in 3066bis.

                      Bye, Frank

---

From: John Cowan <[email-masked]>
Date: Tue, 21 Feb 2006 23:52:57 -0500

> trying to do too much with the scoring algorithm, tho'.

I think these !s are grotesque, and I agree with Frank that in order to
handle extlang subtags properly, you need to do things like ar-!-!-! to
specify just ar with no extlang subtags.  Which is bletcherous.

Let's call the whole thing off.

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 21 Feb 2006 21:15:40 -0800

Yes, the !s are grotesque. But once you open the world to negation of any
kind you're stuck with (some form of) them. Interior wildcards introduce all
manner of problem if we don't consider them and extlangs make a mess of an
otherwise neatly ordered world.

I think it wise to not document algorithms we don't actually need. It would
be nice to include text inviting individual contributions as necessary. In
the meantime, consider the "-var1" text.

Addison

Addison Phillips

> Sent: Tuesday, February 21, 2006 8:53 PM
> To: Addison Phillips
> Cc: 'Frank Ellermann'; [email-masked]
> Subject: Re: [Ltru] Too many ABNF issues (was: Proposal for -matching:

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 21 Feb 2006 21:29:27 -0800

> gap of potential extlangs for all alpha2 / alpha3 languages.

I'm not insane :-)!

The alternative is to make extlangs "default ignored" or some such.
Originally I thought we could get away with *-expansion only and, until this
afternoon, thought that we could absorb "no star in the middle equals no
tags in the middle", but this would mean that interior stars match
*everything* in a greedy manner, producing odd matches.
> Not impossible, but too horrible to consider it seriously.

Precisely.
> just rename what you don't like avoiding all 3066bis names.

Done (although you won't see it until the morning)
> Maybe it means "no privateuse", same idea as "foo-y-!".

Not the way the algorithm is written. It would require 'x' followed by
nothing (illegal). We'd have to invent a special meaning or special syntax
like "foo-!x" (double-ick).
> try **** for any script, ** for any region, etc., different

We could number the stars or do other foolishness... a syntax is possible,
but ugly. If you need it, you need a more complex algorithm.
> If there are pidgin-north and pidgin-south, why no pidgin-* ?

"pidgin" matches "pidgin-*" by the prefix rule in both basic and extended
matching. Read the new extended algorithm carefully: if you run out of range
subtags first, it is a match.
> > primary-pat   = (2*3ALPHA [ extlang ]) ; shortest ISO 639

Yes, thanks.
> We're doomed, the <extlang> kills all those cute shorthands.

Missing fields are negated fields. "de-!-CH" is "de-!-!-!-!-CH" too.
> be fixed by demanding at least one real y-subtag.  Dito for a

I forgot: we need text that says: if you see a singleton and no singleton in
the range, stop (no match if you have subtags left over, otherwise a match).
If the range contains a singleton, it MUST be present in the tag (or no
match). The syntax permits requiring a specific singleton (including 'x')
and matching a specific starting sequence. It should be noted that this may
violate extensions in which position matters in some way (something
extension authors have already been warned not to use).

Best Regards,

Addison