Subject: [Ltru] Re: Preparations for smart matching

From: "Doug Ewell" <[email-masked]>
Date: Sat, 19 Mar 2005 09:59:26 -0800

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

> Hi, so far en-Latn-US-boont not matching en-boont is one of my
> standard examples why I have a problem with 3066bis as it is.
>
> This could be solved later by a smarter matching strategy.  Or
> by subtags used in the order of their importance, something
> like en-boont-Latn-US would match en-boont with the normal RTL
> strategy.

Have you read the new draft on "Matching Language Identifiers"?  It
explains that the "default" matching algorithm is not the only
possibility.

> But it would require complex restrictions allowing to identify
> the class of some subtags independent of their position.  Most
> probably that was discussed years ago before the first of the
> old drafts, and it simply doesn't work.

This is not correct.  First, the registry draft sets out specific rules
for the position and length of subtags, for the exact reason you state:
so that parsers can identify the pieces algorithmically.  It does work,
and it's not all that complex.

For instance, given a fictitious tag
"pu-rud-Blaz-FU-scrog-f-vlad-x-druz", I can tell you exactly what each
of the subtags "means" (that is, its "class"), even though none of them
is assigned in the registry.  The tag is well-formed but not valid.  I
can also tell you that the tags "a1" and "en-US-Fullerton" and
"zh-CN-Hant" and "de-a-b-frank" are not valid, because one or more of
the subtags breaks the rules or because something is wrong with the
order or other syntax of the tag.

Try it yourself.  Send me any tag and I will tell you whether it's
valid, well-formed, or neither.  I promise to do it visually first,
before running it through the validating parser I wrote for this
purpose.  :-)

> So the last chance is smart matching.  To be smart the matching
> algorithm needs some input.  For variants this input could be
> the corresponding default language.   If boont has a pointer to
> en, or if 1996 has a pointer to de, a smart matching algorithm
> could identify fr-1996 or sl-boont as "garbage" and stop any
> attempt to be smart.

Variant subtags already have a "recommended prefix," which is not
restricted to language alone, although all the entries in the draft
registry happen to be primary languages.  Take a look at the entry for
'boont':

variant | boont | Boontling | 2003-02-14 | | en

The "en" at the end is the recommended prefix.  This tells a process
that "en-boont" is an example of using 'boont' for its intended purpose,
while "sl-boont" is an example of abusing it.  The registry draft allows
you to write "sl-boont" but warns that "this is probably a poor choice"
(Section 3.3).

(N.B.  Note the difference between "registry draft" and "draft
registry."  We need better terminology for this, don't we?)

> For laguages the registry could list a default script, with a
> "SHOULD NOT be used" somewhere in the draft.  For de / en / fr
> that would be Latn, and the SHOULD NOT then reduces a dubious
> en-Latn-boont to en-boont.  Or a de-Latn-1996 to de-1996.

This is a more slippery slope, as I mentioned in my response on
ietf-languages.  Section 2.3 of the registry draft says:

"Avoid using subtags that are not important for distinguishing content
in an application. For example, including the script subtag in
"en-Latn-US" is generally unnecessary, since nearly all English texts
are written in the Latin script and it is generally not important to
filter out those few that are not."

This leaves it up to the generator to decide when a script tag is
necessary and when it is not.  This is a good thing.  People and
processes should tag content wisely.

> The smart matching algorithm knows this and reduces all kinds
> of en-Latn-* to en-* in a second matching attempt.  It also
> knows that the US in any en-US-* is "only" a region code, so
> it could start a third pass with en-*.  The longest match after
> up to three attempts (default script removed, region removed)
> wins.  Then en-Latn-US-boont finally matches en-boont.

I claim, as I think you are doing here, that a fully RFC 3066bis-aware
matching algorithm should be able to take apart the entire tag (using
the rules explained above, which again are not all that complex) and
perform matching based on the pieces.  Thus if you want to match
"en-Latn-US-boont" with "en-boont", it is enough to disassemble both
tags and observe that all the subtags in the second tag are also present
in the first.  If this is what you call a "match," then you have found
one.  Other needs might call for different kinds of "matching," and the
matching draft accounts for this.

> And it would not match en-other-boont or en-other-US-boont with
> en-boont.

I don't see why not.  Nothing in the registry definition of 'boont'
defines it as being specifically Latn or specifically US, although
historically that is what Boontling was.  If "en-Latn-US-boont" is a
match for "en-boont", then so is "en-Ogam-MV-boont".

> For de-Latn-*-boont-* it would give up after the
> reduction to de-boont-* shows that it's not a known en-boont-*.

Likewise, even though "de-*-boont" is silly, matching does not depend on
considerations of silliness.  John Cowan likes to point out that "nv-DK"
is silly but perfectly valid.  The same applies to combinations like the
one you are describing.

> So de-Latn-US-boot won't match de-boont, because boont is only
> registered for use with en.  Does that make any sense ?  If yes
> we need optional default scripts for languages, and maybe also
> default languages for all variants in the registry.

As stated, we already have recommended prefixes for variants, but they
are not normative.  And the project of determining default scripts for
*all* languages (not just the 80% or 90% for which this task is easy) is
not something that anyone on this list, or perhaps anywhere else, is
qualified to tackle.

-Doug Ewell
 Fullerton, California
 http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 19 Mar 2005 10:48:10 -0800

Hi -

> From: "Doug Ewell" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
> Sent: Saturday, March 19, 2005 9:59 AM
> Subject: [Ltru] Re: Preparations for smart matching
...
> explains that the "default" matching algorithm is not the only
...

Has the i-d been posted?

Randy

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 20 Mar 2005 05:09:11 +0100

Doug Ewell wrote:

> Have you read the new draft on "Matching Language
> Identifiers"?

If you're talking about the split of dradt 9 into draft 10 and
a separate draft with a matching strategy - not very carefully:
I only saw that it's essentially the same as in draft 9.

> It explains that the "default" matching algorithm is not the
> only possibility.

IIRC that's also the same as in draft 9 and earlier.

> "pu-rud-Blaz-FU-scrog-f-vlad-x-druz",

You promised that you won't use a parser if I find an example,
where it isn't obvious, therefore I have to take this beast
without looking in one of the drafts.

Language pu, no script (assuming that rud is too short), no
region (assuming that rud is too long), so the rest could be
variants rud, Blaz, FU, scrog, f-vlad, and something private
in x-druz.

> I can also tell you that the tags "a1" and "en-US-Fullerton"
> and "zh-CN-Hant" and "de-a-b-frank" are not valid

That tells me that I got it wrong for FU in your first example.
Looking into the ABNF, yes, I completely forgot the reserved
"extlang", and FU is too short for a variant.

Okay, that allows to identify all pieces in well-formed tags.

> variant | boont | Boontling | 2003-02-14 | | en

Another error on my side, I always thought that the last (6th)
column is a comment and irrelevant for parsers.  But it has a
structure recomended_prefix "#" comment.

Please ignore my stupid idea to get rid of all "#" in another
article, the 6th column is not only a comment.  Comments have
nothing to do with the column, they are simply at the end of a
row introduced by "#".

So you always have a 6th colum, and for all variants it's the
pointer to a language - as I wanted it, I only didn't see that
you already have it.

Could we rename "recommended_prefix" to "associated_tag", or
something in this direction ?  Then the 6th column of languages
would be free for a pointer to a default script.

At the moment the draft says:

| The "recommended_prefix" field is empty, except where the
| type is "variant"

That could be changed to e.g.

  The "recommended_prefix" is generally empty.  In the case of
  a "variant" it's often the recommended "language" prefix, but
  could be also a language-script, language-script-region, or
  language-region prefix.

  For the type "language" this field can be used to define a
  default "script" for the correspoding "laguage", and then the
  resulting tag or prefix language-sript is NOT RECOMMENDED.

> (N.B.  Note the difference between "registry draft" and
> "draft registry."  We need better terminology for this, don't
> we?)

As long as <http://users.adelphia.net/~dewell/lstreg.html> is
the state of the art I don't care about the terminology.  If
you'll submit it as an I-D it would be the "draft registry".

BTW, the initial mail header registry now got its own RfC 4021
<http://purl.net/net/rfc/4021> - so that wasn't lost somewhere
in limbo between the IESG and IANA.

> Thus if you want to match "en-Latn-US-boont" with "en-boont",
> it is enough to disassemble both tags and observe that all
> the subtags in the second tag are also present in the first.

That's IMFO not good enough.  If a client offers en-boont and
en-misc (script "misc"), and the server has en-misc-US-boont,
then the client should try en-misc, but not en-boont.

Because "misc" is probably very important, if it's not "Latn"
as in the default for en.

> If "en-Latn-US-boont" is a match for "en-boont", then so is
> "en-Ogam-MV-boont".

Unless you have en-Ogam, then you need to know the default
script for en (which isn't Ogam).

> the project of determining default scripts for *all*
> languages (not just the 80% or 90% for which this task is
> easy) is not something that anyone on this list, or perhaps
> anywhere else, is qualified to tackle.

*all* is not what I proposed.  I'm only interested in the 80%
where it's easy for humans, but not for parsers.  Bye, Frank