Subject: [Ltru] filtering and extended filtering...

From: "Addison Phillips" <[email-masked]>
Date: Wed, 15 Feb 2006 09:56:52 -0800

I have followed the current debate with some interest, but this is the first time I've been able to craft a complete response before more messages have rendered it obsolete.

I have a few observations.

1. Frank is correct that the "*" is redundant in all but the first position in an extended language range (since "missing" or unspecified subtags expand to star). I think we ought to remove them because it simplifies implementation and interoperability with basic ranges.

2. There seems to be an assumption that extended filtering should require registry access. This is based on the thought that a range such as "en-Latn" should match all "en-*" content that has obeyed the "suppression" of the 'Latn' subtag. This is, I think, wrong for three reasons:

   i. There is no other reason to require registry access or a validating processor in order to do extended filtering.

   ii. Operations in which the user specifies something should produce recognizable matches. If I wish to select all content in my document that is labeled with the Latin script subtag (perhaps I am going through the content to remove the subtag!), the matching set from the filter should look maximally like my range.

   iii. The use cases for extended filtering does not include language negotiation (see below).

3. The only valid use cases for extended filtering deal with selecting content in which the user is interested in specific subtag values, not with the semantic meaning of the resulting tag. The example I use most often is applying fonts to Chinese texts in CSS:

   :lang(*-Hant)  # use a trad font
   :lang(*-Hans)  # use a simpl font

I believe that all examples of this sort produce sets of content that are not required to be (and usually actually are not) linguistically related or mutually understandable at all. This suggests that the extended filtering is mostly to do with document or content processing. Thus, we might wish to do selections like "*-CH" (find all the content that is purported to be in a Swiss dialect) or "de-1996" (find any content that is labeled as using the 1996 orthography, regardless of script or region). Extended filtering is simply a tool used by higher level protocols to select specific language tags based on their attributes and not directly on the language itself.

4. Frank's points about variants, while technically correct, I don't believe are cause for worry in practice. Variants can appear in unlimited multiples and they can appear in any order in theory. However, variants *do* have Prefix fields and the text of 3066bis is quite clear about when variants may be used together. To wit:

--
Most variants that share a prefix are mutually exclusive. For example, the German orthographic variations '1996' and '1901' SHOULD NOT be used in the same tag, as they represent the dates of different spelling reforms. A variant that can meaningfully be used in combination with another variant SHOULD include a 'Prefix' field in its registry record that lists that other variant. For example, if another German variant 'example' were created that made sense to use with '1996', then 'example' should include two Prefix fields: "de" and "de-1996".
--

I would suggest that very few if any variants will ever be used *correctly* together in any sequence and I'd be willing to wager that we won't seen a "correct" three-variant sequence at any time in the foreseeable future. 

5. I wrote the original extended filtering grammar based on the idea that the wildcard must come at the end of a subtag type. This complicates the code for variants quite a bit (if the variant is expressed in the range, you have to check that no variant subtags appear before that subtag). Given the use cases above, I think now that this is actually wrong. Extended range filtering should work like regexp, I think: there is a match if and only if all subtags in the range appear (in the same order) in the tag. But all tags that contain the same subtags in the same order match. Implementation of this scheme is easy and requires *no* access to the registry at all. It *also* means we can get rid of the complicated ABNF and move to:

Ext-range = ("*" / Subtag) *["-" Subtag]
Subtag = 1*8alphanum

An implementation of this is *very* simple, simple enough to explain to actual users, which I think is also a benefit.

So I've written some opinions down. I think my main proposals would be:

A. Include some text in draft-matching pointing out that an explicit script subtag in the range does not match a missing implied one in the tag and that the reason why is so that users can select content that has the subtag visible.

B. Change the text and grammar to suppress wildcards except in the initial position.

C. Spell out usage scenarios for extended filtering.

D. Use the simple ABNF and treat extended filtering as a form of regexp.

Addison

Addison Phillips
Internationalization Architect - Yahoo! Inc.

Internationalization is an architecture.
It is not a feature. 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 16 Feb 2006 00:14:50 +0100

> D. Use the simple ABNF and treat extended filtering as a

Sounds good so far.  Maybe add a hint that there is no NOT
operator in the syntax, but that extended filtering allows
to implement this on top of it.

I'm not yet sure about the Suppress-Script.  It turns out to
be a bit different from what the subject of one of the first 
threads here indicated ("prepare for smart matching" IIRC).

Well, Randy proposed MUST NOT for Suppress-Script, if I now
suddenly think that MUST NOT might be cleaner than SHOULD NOT
it's about nine months too late.

                           Bye, Frank

---

From: Mark Davis <[email-masked]>
Date: Wed, 15 Feb 2006 16:20:12 -0800

This looks really good. +1

---

From: r&d afrac <[email-masked]>
Date: Thu, 16 Feb 2006 01:36:29 +0100

>This looks really good. +1

good to me. For the implied purpose.
jfc

>Addison Phillips wrote:
>>I have followed the current debate with some interest, but this is 
>>the first time I've been able to craft a complete response before 
>>more messages have rendered it obsolete.
>>
>>1. Frank is correct that the "*" is redundant in all but the first 
>>position in an extended language range (since "missing" or 
>>unspecified subtags expand to star). I think we ought to remove 
>>them because it simplifies implementation and interoperability with 
>>basic ranges.
>>
>>2. There seems to be an assumption that extended filtering should 
>>require registry access. This is based on the thought that a range 
>>such as "en-Latn" should match all "en-*" content that has obeyed 
>>the "suppression" of the 'Latn' subtag. This is, I think, wrong for 
>>three reasons:
>>
>>    i. There is no other reason to require registry access or a 
>> validating processor in order to do extended filtering.
>>
>>    ii. Operations in which the user specifies something should 
>> produce recognizable matches. If I wish to select all content in 
>> my document that is labeled with the Latin script subtag (perhaps 
>> I am going through the content to remove the subtag!), the 
>> matching set from the filter should look maximally like my range.
>>
>>3. The only valid use cases for extended filtering deal with 
>>selecting content in which the user is interested in specific 
>>subtag values, not with the semantic meaning of the resulting tag. 
>>The example I use most often is applying fonts to Chinese texts in CSS:
>>
>>I believe that all examples of this sort produce sets of content 
>>that are not required to be (and usually actually are not) 
>>linguistically related or mutually understandable at all. This 
>>suggests that the extended filtering is mostly to do with document 
>>or content processing. Thus, we might wish to do selections like 
>>"*-CH" (find all the content that is purported to be in a Swiss 
>>dialect) or "de-1996" (find any content that is labeled as using 
>>the 1996 orthography, regardless of script or region). Extended 
>>filtering is simply a tool used by higher level protocols to select 
>>specific language tags based on their attributes and not directly 
>>on the language itself.
>>
>>4. Frank's points about variants, while technically correct, I 
>>don't believe are cause for worry in practice. Variants can appear 
>>in unlimited multiples and they can appear in any order in theory. 
>>However, variants *do* have Prefix fields and the text of 3066bis 
>>is quite clear about when variants may be used together. To wit:
>>
>>--
>>Most variants that share a prefix are mutually exclusive. For 
>>example, the German orthographic variations '1996' and '1901' 
>>SHOULD NOT be used in the same tag, as they represent the dates of 
>>different spelling reforms. A variant that can meaningfully be used 
>>in combination with another variant SHOULD include a 'Prefix' field 
>>in its registry record that lists that other variant. For example, 
>>if another German variant 'example' were created that made sense to 
>>use with '1996', then 'example' should include two Prefix fields: 
>>"de" and "de-1996".
>>--
>>
>>I would suggest that very few if any variants will ever be used 
>>*correctly* together in any sequence and I'd be willing to wager 
>>that we won't seen a "correct" three-variant sequence at any time 
>>in the foreseeable future.
>>5. I wrote the original extended filtering grammar based on the 
>>idea that the wildcard must come at the end of a subtag type. This 
>>complicates the code for variants quite a bit (if the variant is 
>>expressed in the range, you have to check that no variant subtags 
>>appear before that subtag). Given the use cases above, I think now 
>>that this is actually wrong. Extended range filtering should work 
>>like regexp, I think: there is a match if and only if all subtags 
>>in the range appear (in the same order) in the tag. But all tags 
>>that contain the same subtags in the same order match. 
>>Implementation of this scheme is easy and requires *no* access to 
>>the registry at all. It *also* means we can get rid of the 
>>complicated ABNF and move to:
>>
>>An implementation of this is *very* simple, simple enough to 
>>explain to actual users, which I think is also a benefit.
>>
>>A. Include some text in draft-matching pointing out that an 
>>explicit script subtag in the range does not match a missing 
>>implied one in the tag and that the reason why is so that users can 
>>select content that has the subtag visible.
>>

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 16 Feb 2006 13:43:04 +0100 (CET)

> 1. Frank is correct that the "*" is redundant [...]

That's a CHOICE, and I don't see why you've made that particular choice.

> 2. There seems to be an assumption that extended filtering should require
> registry access. This is based on the thought that a range such as
> "en-Latn" should match all "en-*" content that has obeyed the
> "suppression" of the 'Latn' subtag. This is, I think, wrong for three
> reasons:
...

Ok. But Karen's reason is a better one.

> Extended filtering is simply a tool used by higher level protocols to
> select specific language tags based on their attributes and not directly
> on the language itself.

Yes.

> Extended range filtering should work like regexp, I think:

I think you mean "wildcard patterns" rather than regexp [regular
expression (patterns)] (and they are still patterns, not ranges).
For a regexp approach, one would be able to have patterns like
en-(GB|EI) and zh-?*. I think that may be overkill.

> there is a
> match if and only if all subtags in the range appear (in the same order)
> in the tag. But all tags that contain the same subtags in the same order
> match. Implementation of this scheme is easy and requires *no* access to
> the registry at all. It *also* means we can get rid of the complicated
> ABNF and move to:
>

If you really want something similar to wildcard patterns, then in
general the empty string (here: of subtags) match ONLY the empty
string (here: of subtags). To match "anything" (here: any string
of zero or more subtags), there has to be an explicit wildcard.

That's why I've been suggesting that wildcards must be explicit,
not implicit, also in this case.

A simple syntax could be:
  languagetagpattern = subtagpattern *["-" subtagpattern]
  subtagpattern = (1*8alphanum / "*")

> explain to actual users, which I think is also a benefit.

Yes, but then make it actually look like something familiar, like
wildcard patterns, where wildcards generally *are* explicit. Other
than for -matching, I cannot recall seeing another example where they
were not explicit (except at the end; cmp the prefix matching, but
then there cannot be wildcards anywhere else but the end).

(And yes, the semantics for this would still be based on a
partial ordering that CANNOT be seen as a tree; the latter
would restrict (implicit or explicit) wildcards to be at the
end, and never to be at the beginning or in the middle.)

     /kent k

---

From: r&d afrac <[email-masked]>
Date: Thu, 16 Feb 2006 15:30:42 +0100

At 13:43 16/02/2006, Kent Karlsson wrote:
>en-(GB|EI) and zh-?*. I think that may be over kill.

Value domains are an interesting idea in this way. Could be of 
interest in Gateway protocols and in interfacing ISO 11179 systems.

>not implicit, also in this case.

This is not the idea behind the existing RFC 3066 Bis. But you right, 
the langtag being inclusive or exclusive, normative or descriptive, 
etc. are among the things we need to know before filtering. I thought 
we should have discussed (IMHO in a framework document). But as far 
as I understand the consensus ruled against this. These aspects have 
to be defined somewhere. I propose (cf. appeal) to use RFC 4151, the 
language space defined by the IRI-tag can provide this information 
and more. But there are less general other ways. You are correct, 
another way is to precisely define what the current discussion is 
about. This is why I approved Addisson's proposition: I understand it 
in line with an inclusive description of a document.

>end, and never to be at the beginning or in the middle.)

Let say that these are default format equivalences for the proposed 
type of applications. Where I am confused is about the way extensions 
will be supported.

jfc

---

From: John Cowan <[email-masked]>
Date: Tue, 21 Feb 2006 15:27:05 -0500

Addison Phillips scripsit:

> position in an extended language range (since "missing" or unspecified
> subtags expand to star). I think we ought to remove them because it
> simplifies implementation and interoperability with basic ranges.

This turns out not to be the case for scoring purposes as the scoring
algorithm is currently written (3.2.3):

	Any remaining missing components in the language tag are set to
	"*"; thus an empty language tag becomes the quintuple ("*",
	"*", "*", "*", "*"). Missing components in the language range
	are handled similarly to extended range lookup: missing internal
	subtags are expanded to "*". Missing end subtags are expanded as
	the empty string. Thus a pattern "en-US" becomes the quintuple
	("en","*","US","","").

Currently, therefore, the ranges "en-us", "en-us-*", and "en-us-*-*"
expand to different quintuples and produce different scores, because
"" in a quintuple is equal only to itself or to "*".  I am not sure why
this is considered desirable, but it is so.

(If this is to be preserved, the third sentence should be merged into
the second, so that both are subordinated to the clause ending in a colon.)

> A. Include some text in draft-matching pointing out that an explicit
> script subtag in the range does not match a missing implied one in
> the tag and that the reason why is so that users can select content
> that has the subtag visible.

+1

> B. Change the text and grammar to suppress wildcards except in the

I agree, but only if the anomaly over scoring is removed.

> C. Spell out usage scenarios for extended filtering.

+1

> D. Use the simple ABNF and treat extended filtering as a form of regexp.

+1

-- 
John Cowan  [email-masked]  www.ccil.org/~cowan  www.ap.org
The penguin geeks is happy / As under the waves they lark
The closed-source geeks ain't happy / They sad cause they in the dark
But geeks in the dark is lucky / They in for a worser treat
One day when the Borg go belly-up / Guess who wind up on the street.

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 21 Feb 2006 12:55:50 -0800

> I agree, but only if the anomaly over scoring is removed 

I had noted the discrepancy in the scoring section and am working to reword
it.

I am not sure why
> this is considered desirable, but it is so.

The reason for the odd expansion is due to my experimental implementation:
it seems to produce the "best" results that way. Cf.
http://thread.gmane.org/gmane.ietf.ltru/4151

Addison

Addison Phillips