Subject: [Ltru] comments on draft-10

From: "Addison Phillips" <[email-masked]>
Date: Wed, 1 Mar 2006 10:10:07 -0800

I have just a few substantive observations:

1. For some reason the formulation "applications, protocols, and specifications" was rendered as "applications, protocols, and implementations" in a few places. I fixed this.

2. In the penultimate paragraph in Section 3.1 (choosing types of matching), I think we should include this normative text directly after stating that the schemes don't require the registry or validation:

--
An implementation MAY choose to check if either the language ranges or language tags being matched are "well-formed" or "valid" (see <xref target="RFC3066bis" />, Section 2.2.9) and MAY choose not to process invalid ranges.
--

This will make it clear that, while checking is not required, it is permitted.

3. I think we need to change one aspect of extended range matching: I don't think the match should proceed past singletons unless the singleton is "named" in the range. Currently the range "de-DE" would match "de-x-DE", which I find problematic. Thus I'd propose changing the third paragraph in Section 3.2.2 from:

--
Take each non-wildcard subtag in the language range and compare it to the next subtag in the language tag in turn until a matching subtag is found or the langauge tag is exhausted. If the end of the language tag is found first, the match fails. If a match is found, this step is repeated with the next non-wildcard subtag in the language range (and beginning with the next subtag in the language tag) until the list of subtags in the language range is exhausted or the match fails.
--

To read:

<t>Take each non-wildcard subtag in the language range and compare it in a case-insensitive manner to the next subtag in the language tag. If the range's subtag exactly matches the tag's subtag, proceed to the next non-wildcard subtag in the language range (and beginning with the next subtag in the language tag) until the list of subtags in the language range is exhausted or the match fails. If the tag's subtag is a "singleton" (a single letter or digit, which, in this case, includes the private-use subtag 'x') and the range's subtag does not match or if the language tag's list of subtags is exhausted, the match fails. If the language range's list of subtags is exhausted, the match succeeds.</t>

3a. I think we should include counter examples to the ones that match:

--
<t>The same range does not match any of the following tags for the reasons shown:<list>
   <t>de (missing 'DE')</t>
   <t>de-x-DE (singleton 'x' occurs before 'DE')</t>
   <t>de-Deva ('Deva' not equal to 'DE')</t>

</list></t>
--

4. I think it would also be useful to point out in this same section the fact that 3066bis tag structure makes ranges quite specific:

<t>Note: The structure of language tags defined by <xref target="RFC3066bis" /> defines each subtag according to position, size, and content. This means that the subtags in a language range can only match specific types of subtags in a language tag. For example, a subtag such as 'Latn' is always a script subtag (unless it follows a singleton) while a subtag such as 'nedis' can only match the equivalent variant subtag (unless it follows a singleton).</t>

5. In Section 3.3 (lookup) we show but say nothing about removing singletons along with their just-following subtag, but we demonstrate it. I.e.:

Zh-Hant-CN-x-private
Zh-Hant-CN

It doesn't actually matter whether one does this or not, but should we say anything?

6. Section 4.1 (Choosing language ranges). Should we mention that users SHOULD choose well-formed, valid, canonicalized ranges? Also, should we again mention the ability for implementations to check for any of those three attributes?

Addison

Addison Phillips
Internationalization Architect - Yahoo! Inc.

Internationalization is an architecture.
It is not a feature. 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Mark Davis <[email-masked]>
Date: Wed, 01 Mar 2006 14:26:42 -0800

> 1. For some reason the formulation "applications, protocols, and specifications" was rendered as "applications, protocols, and implementations" in a few places. I fixed this.
good
> 2. In the penultimate paragraph in Section 3.1 (choosing types of matching), I think we should include this normative text directly after stating that the schemes don't require the registry or vali…
good
I don't think this quite works, since that would have de-x-DE not 
matching de-x-DE. What I suggest is instead changing it to add instead:

Extension and private use subtag sequences are treated as a single unit 
for the purpose of matching. Thus in matching de-DE against de-x-DE, the 
x-DE is treated as treated as a single unit, which does not match DE.
agreed
> It doesn't actually matter whether one does this or not, but should we say anything?
see above
> 6. Section 4.1 (Choosing language ranges). Should we mention that users SHOULD choose well-formed, valid, canonicalized ranges? Also, should we again mention the ability for implementations to chec…
Not necessary, but doesn't hurt.

---

From: "Addison Phillips" <[email-masked]>
Date: Wed, 1 Mar 2006 15:44:53 -0800

> x-DE is treated as treated as a single unit, which does not match DE.

No, because that prevents doing matching of private use or extension sequences. There is no reason to break those matches. All my text means is that you must EITHER:

a. Specify the private use or extension in the range
b. Run out of range subtags before you get to the private use or extension singleton. 

Maybe my text wasn't clear enough? Here is some perl code that implements the algorithm proposed (not pretty, but I wrote it in five minutes):

sub match {
      @range = split(/-/, shift);
      @tag    = split(/-/, shift);
      my $count = 0;
      
RANGE:    foreach my $rsubtag (@range) {
            # skip wildcards, if initial, increment tag's subtag count
	      if ($rsubtag eq "*") {
		      if ($count == 0) {
		            $count++;
			}
			next RANGE;
		}
	      while ($count <= $#tag) { # look for matching subtag
		      if (uc($rsubtag) eq uc($tag[$count])) {
			      $count++;
			      next RANGE;
		      }
                  # subtags don't match, check for singleton
		      if (length($tag[$count++]) == 1) {
			      return 0;
		      }
	      }
            # ran out of tag subtags before finding last range subtag
            # note: "next RANGE" in perl is not like 'break' in C and 
            #   skips this line!
	      return 0;
      }
      # exhausted the range first
      return 1;

}

This correctly returns a "match" result for "de-x-DE" with "de-Latn-x-DE" and a "non-match" for range "de-DE" with "de-x-DE".

Addison Phillips

---

From: Martin Duerst <[email-masked]>
Date: Thu, 02 Mar 2006 18:44:52 +0900

At 03:10 06/03/02, Addison Phillips wrote:
 >1. For some reason the formulation "applications, protocols, and
 >specifications" was rendered as "applications, protocols, and
 >implementations" in a few places. I fixed this.
 >
 >2. In the penultimate paragraph in Section 3.1 (choosing types of
 >matching), I think we should include this normative text directly after
 >stating that the schemes don't require the registry or validation:
 >
 >--
 >An implementation MAY choose to check if either the language ranges or
 >language tags being matched are "well-formed" or "valid" (see <xref
 >target="RFC3066bis" />, Section 2.2.9) and MAY choose not to process
 >invalid ranges.
 >--
 >
 >3. I think we need to change one aspect of extended range matching: I don't
 >think the match should proceed past singletons unless the singleton is
 >"named" in the range. Currently the range "de-DE" would match "de-x-DE",
 >which I find problematic. Thus I'd propose changing the third paragraph in
 >Section 3.2.2 from:
 >
 >--
 >Take each non-wildcard subtag in the language range and compare it to the
 >next subtag in the language tag in turn until a matching subtag is found or
 >the langauge tag is exhausted. If the end of the language tag is found
 >first, the match fails. If a match is found, this step is repeated with the
 >next non-wildcard subtag in the language range (and beginning with the next
 >subtag in the language tag) until the list of subtags in the language range
 >is exhausted or the match fails.
 >--
 >
 >To read:
 >
 ><t>Take each non-wildcard subtag in the language range and compare it in a
 >case-insensitive manner to the next subtag in the language tag. If the
 >range's subtag exactly matches the tag's subtag, proceed to the next
 >non-wildcard subtag in the language range (and beginning with the next
 >subtag in the language tag) until the list of subtags in the language range
 >is exhausted or the match fails. If the tag's subtag is a "singleton" (a
 >single letter or digit, which, in this case, includes the private-use
 >subtag 'x') and the range's subtag does not match or if the language tag's
 >list of subtags is exhausted, the match fails. If the language range's list
 >of subtags is exhausted, the match succeeds.</t>

This text looks like it's way too complex. I'm not yet sure whether that's
a problem of the wording, or of the thing we are trying to describe.

 >3a. I think we should include counter examples to the ones that match:

Yes.

But I'm also confused about de-DE. Would it match the same stuff as
de-*-DE? If yes, why do we have wildcards (except in first place)?
If this is indeed what we want (which I'm not sure about), please
at least say so.

 >4. I think it would also be useful to point out in this same section the
 >fact that 3066bis tag structure makes ranges quite specific:
 >
 ><t>Note: The structure of language tags defined by <xref
 >target="RFC3066bis" /> defines each subtag according to position, size, and
 >content. This means that the subtags in a language range can only match
 >specific types of subtags in a language tag. For example, a subtag such as
 >'Latn' is always a script subtag (unless it follows a singleton) while a
 >subtag such as 'nedis' can only match the equivalent variant subtag (unless
 >it follows a singleton).</t>

makes sense

 >5. In Section 3.3 (lookup) we show but say nothing about removing
 >singletons along with their just-following subtag, but we demonstrate it. I.e.:
 >
 >Zh-Hant-CN-x-private
 >Zh-Hant-CN
 >

Probably yes.

 >6. Section 4.1 (Choosing language ranges). Should we mention that users
 >SHOULD choose well-formed, valid, canonicalized ranges?

Yes.

 >Also, should we
 >again mention the ability for implementations to check for any of those
 >three attributes?

Probably no.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 02 Mar 2006 19:25:50 +0900

At 08:44 06/03/02, Addison Phillips wrote:
 >No, because that prevents doing matching of private use or extension
 >sequences. There is no reason to break those matches. All my text means is
 >that you must EITHER:
 >
 >Maybe my text wasn't clear enough? Here is some perl code that implements
 >the algorithm proposed (not pretty, but I wrote it in five minutes):

I'm rewriting this, to try to get a better understanding
(and get rid of the double loop). [This code is removing
subtags from the arrays of subtags (@range, @tag) with shift,
and making use of the fact that if there is nothing to remove
anymore, the 'undef' value (false) is returned.]

sub match {

       $rsubtag = shift @range;
       $subtag = shift @tag;
       if ($rsubtag eq "*") {  # '*' in first position is special
           $rsubtag = shift @range;
           $subtag = shift @tag;
       }
       while ($rsubtag) {
           if ($rsubtag eq "*") {  # ingnore non-initial '*'s
               $rsubtag = shift @range;
           }
           else if (!$subtag) { # ran out of subtags
               return 0;
           }
           else if (uc($rsubtag) eq uc($subtag)) {  # check for equality
               $rsubtag = shift @range;
               $subtag = shift @tag;
           }
           else if (length($subtag) == 1) {  # check for singletons
               return 0;
           }
       }
       return 1;  # exhausted range first, match found

 >This correctly returns a "match" result for "de-x-DE" with "de-Latn-x-DE"
 >and a "non-match" for range "de-DE" with "de-x-DE".

It's very asymetric, but that seems to be a result of the fact that
singleton-started extensions always are at the end, yes?

Regards,    Martin.

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 2 Mar 2006 09:57:29 -0800

Much nicer code. I can tell you spent at least 6 minutes on it. :-)

Yes.

Addison

Addison Phillips

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 2 Mar 2006 10:07:48 -0800

> a problem of the wording, or of the thing we are trying to describe.

The wording most likely. Ideally this would be broken up into steps.

>  ><t>The same range does not match any of the following tags for the
> reasons
>  >shown:<list>
>  >   <t>de (missing 'DE')</t>

"de-DE" and "de-*-DE" both match the same stuff. This is what we want. The
subtle bit is: we define the extended language range (XLR) separately from
the algorithm. Other schemes that people come up with in the future (such as
scoring) might assign different semantics to "de-DE" and "de-*-DE". Thus the
XLR syntax allows them even though extended filtering makes no use of them.

>  >singletons along with their just-following subtag, but we demonstrate
it.
> I.e.:
>  >
>  >Zh-Hant-CN-x-private
>  >Zh-Hant-CN
>  >
>  >It doesn't actually matter whether one does this or not, but should we
> say
>  >anything?
> 
> Probably yes.

Okay. Done.
>  >SHOULD choose well-formed, valid, canonicalized ranges?

Done per last mail.

---

From: Martin Duerst <[email-masked]>
Date: Fri, 03 Mar 2006 11:50:46 +0900

>Much nicer code. I can tell you spent at least 6 minutes on it. :-)

I spent at least 20 or 30 minutes on it, but most of it was to try
and figure out what your code did, or what the draft said (or both).

Regards,    Martin.