Subject: [Ltru] two identical singleton extension tags issue

From: "Mark Davis" <[email-masked]>
Date: Thu, 14 Dec 2006 09:05:03 -0800

I recommend that we change the restriction on two identical singleton
extension tags to be a criterion for validity instead of well-formedness.
This allows programs to test for well-formedness with a single regular
expression, without any other other processing, significantly increasing the
probability that they will do it correctly.

We originally had the restriction under well-formedness because it can be
tested without access to the database. But in practice, and especially
considering that there should be extremely few tags that illustrate the
problem, it is operationally much simpler to have it under validity.

Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Thu, 14 Dec 2006 12:13:30 -0500

Mark Davis scripsit:

> I recommend that we change the restriction on two identical
> singleton extension tags to be a criterion for validity instead of
> well-formedness.  This allows programs to test for well-formedness
> with a single regular expression, without any other other processing,
> significantly increasing the probability that they will do it correctly.
> 
> We originally had the restriction under well-formedness because it
> can be tested without access to the database. But in practice, and
> especially considering that there should be extremely few tags that
> illustrate the problem, it is operationally much simpler to have it
> under validity.

I'm inclined to agree, since I am now writing a Perl langtag checker
for 4646bis and would be happy to scrap that part of my code.  :-)

I would say the only reason *not* to do this is the desire to make as
few changes as possible, but I think this change is small enough and in
the correct direction, and we should go with it.

-- 
So that's the tune they play on                 John Cowan
their fascist banjos, is it?                    [email-masked]
        --Great-Souled Sam                      http://www.ccil.org/~cowan

---

From: Addison Phillips <[email-masked]>
Date: Thu, 14 Dec 2006 09:45:31 -0800

Hmm... that's a good point.

+1

Mark Davis wrote:
> ------------------------------------------------------------------------
> 

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: Addison Phillips <[email-masked]>
Date: Thu, 14 Dec 2006 10:02:42 -0800

I went to make some proposed edits to cover this... and suddenly find 
some doubts.

The problem here is in Section 2.2.6 (Extensions) where we find this text:

--
# Each singleton subtag MUST appear at most one time in each tag (other 
than as a private use subtag). That is, singleton subtags MUST NOT be 
repeated. For example, the tag "en-a-bbb-a-ccc" is invalid because the 
subtag 'a' appears twice. Note that the tag "en-a-bbb-x-a-ccc" is valid 
because the second appearance of the singleton 'a' is in a private use 
sequence.
--

With a MUST, I don't see how we can relax the well-formedness 
requirement. A tag is not well-formed if it violates this requirement. 
Validity rises to a higher level.

I think we could relax the well-formedness requirement on checking 
non-repetition so that we have:

well-formed MUST check for ABNF validity
well-formed SHOULD check for non-repetition of singletons
valid MUST check non-repetition
valid MUST check subtags

This is what I'm implementing in draft-02.

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 14 Dec 2006 10:27:28 -0800

+1

---

From: John Cowan <[email-masked]>
Date: Thu, 14 Dec 2006 17:29:06 -0500

+1

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
I come from under the hill, and under the hills and over the hills my paths
led. And through the air. I am he that walks unseen.  I am the clue-finder,
the web-cutter, the stinging fly. I was chosen for the lucky number.  --Bilbo

---

From: Martin Duerst <[email-masked]>
Date: Fri, 15 Dec 2006 16:50:38 +0900

At 03:02 06/12/15, Addison Phillips wrote:
>I went to make some proposed edits to cover this... and suddenly find some doubts.
>
># Each singleton subtag MUST appear at most one time in each tag (other than as a private use subtag). That is, singleton subtags MUST NOT be repeated. For example, the tag "en-a-bbb-a-ccc" is invalid because the subtag 'a' appears twice. Note that the tag "en-a-bbb-x-a-ccc" is valid because the second appearance of the singleton 'a' is in a private use sequence.
>--

This text uses '(in)valid' twice, and never (non)well-formed.
Is this new text, or from RFC 4646?

>With a MUST, I don't see how we can relax the well-formedness requirement. A tag is not well-formed if it violates this requirement. Validity rises to a higher level.
>
>I think we could relax the well-formedness requirement on checking non-repetition so that we have:
>

This looks weird. A tag is either well-formed or not.
With the above SHOULD, things become unclear.
It is much more straightforward if you have:

well-formed means:
- check for ABNF match
valid means:
- check for non-repetition of singletons
- check for actually existing subtags,...

I'm ambivalent on the change itself, but if we follow Mark and
John, let's not use SHOULD in the definition of well-formedness
or validity.

Regards,     Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Addison Phillips <[email-masked]>
Date: Fri, 15 Dec 2006 11:22:04 -0800

Hello Martin,

> Is this new text, or from RFC 4646?

This is the RFC 4646 text.

> This looks weird. A tag is either well-formed or not.

Ah... but these are requirements for an *implementation* claiming to 
check well-formedness. The SHOULD means that a tag is "well-formed" with 
a repeating singleton (which bothers me), but, basically, that all 
implementations really ought to check for it.

I agree that checking for non-repetition is expensive and annoying. But 
there is a very clear MUST in the text. A tag cannot be well-formed if 
it has a singleton repeat.

What we could do is add a new conformance layer:

-  An implementation is "minimally conformant" if it checks for 
adherence to the ABNF.
-  An implementation is "well-formed" if it also checks for 
non-repetition of singletons.
-  An implementation is "validating" if it checks the subtags using a 
copy of the registry.

Best Regards,

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 15 Dec 2006 13:03:56 -0800

I agree that *currently* the well-formedness criteria require that contains
a singleton repeat. What I requested was that we move this requirement to
the validity criteria, for reasons cited earlier. I don't think we need
three levels. And I think trying to distinguish between what a tag is and
what an implementation does is messy (you may not be proposing that, but
just to be sure).

So what I propose is:

-  A tag is "well-formed" if it adheres to ABNF.
-  A tag is "valid" if it contains no repeat singletons, and the subtags
check out according to the registry.

Mark

---

From: John Cowan <[email-masked]>
Date: Fri, 15 Dec 2006 16:18:09 -0500

Mark Davis scripsit:

> three levels. 

I agree.

> I think trying to distinguish between what a tag is and what an
> implementation does is messy (you may not be proposing that, but just
> to be sure).

I think it's quite reasonable to distinguish here.

>    check out according to the registry.

I think that's correct, and would add these:

- A well-formedness-checker MUST reject a tag that is not well-formed,
   and SHOULD reject one that contains repeat singletons.
- A validity-checker MUST reject a tag that is not valid.

I think this captures both your ideas and Addison's.

-- 
What has four pairs of pants, lives             John Cowan
in Philadelphia, and it never rains             http://www.ccil.org/~cowan
but it pours?                                   [email-masked]
        --Rufus T. Firefly

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 15 Dec 2006 13:31:18 -0800

Good point. Ok, I agree with that approach, and it should make Addison
happy....

Mark

---

From: Addison Phillips <[email-masked]>
Date: Fri, 15 Dec 2006 13:49:37 -0800

Mark Davis wrote:

 > What I requested was that we move this requirement to
> the validity criteria, for reasons cited earlier. 

Yes, and I think those reasons are valid. The problem I'm having is that 
the non-repetition check really does speak to well-formedness of the tags.

It's annoying to have a syntax that cannot be checked with a single 
expression.

 > I don't think we need

I note Section 2.2.9 already makes the distinction: it speaks about 
implementations and their claims of conformance. It says nothing about 
tags being "well-formed" or "valid" (even if we refer to tags that way 
from time-to-time).

Admittedly, it does no harm to existing implementations if we relax the 
rules (presumably implementations are allowed to do more than the 
minimum). But I am leery of this change because it seems contrary to the 
name to have a tag that isn't properly formed be "well-formed".

A good analogy is checking UTF-8 validity. Checking if some text follows 
the encoding scheme is far simpler than checking that it also avoids 
non-shortest forms. But the latter is still required (of course, it can 
still be done in a single regexp, so maybe the analogy is incomplete).

Still, I support making the change. Perhaps if we note that a tag that 
is "well-formed" might still not be, uh, well-formed? Inserting a health 
warning might address this. Perhaps, instead of my proposed SHOULD text 
in the current draft-02, we could say:

--
Note that an implementation that implements "well-formed" checking 
according to the ABNF might not identify tags that contain multiple 
occurrences of the same singleton subtag. Such tags are invalid 
according to the rules in Section 2.2.6 (Extension Subtags). Extensions 
are relatively rare and their use is generally orthogonal to that of the 
main body of the language tag. So, implementing a check for 
non-repetition of these subtags is not required, because that would 
force implementations to use more than one regular expression or to be 
more complicated than necessary.
--

Addison

---

From: Addison Phillips <[email-masked]>
Date: Fri, 15 Dec 2006 14:22:01 -0800

Go back and look at draft-02. It talks only about implementation 
requirements. There is no definition of "well-formed" or "valid" tags. 
This is not a change from RFC 4646 either.

Addison

---

From: John Cowan <[email-masked]>
Date: Fri, 15 Dec 2006 17:24:24 -0500

> This is not a change from RFC 4646 either.

Oopsie.  I am going to review -02 de novo shortly.

-- 
I marvel at the creature: so secret and         John Cowan
so sly as he is, to come sporting in the pool   [email-masked]
before our very window.  Does he think that     http://www.ccil.org/~cowan
Men sleep without watch all night?

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 15 Dec 2006 14:26:30 -0800

Hi -

As a technical contributor...

> From: "Addison Phillips" <[email-masked]>
> To: "Mark Davis" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Friday, December 15, 2006 1:49 PM
> Subject: Re: [Ltru] two identical singleton extension tags issue
>
> Mark Davis wrote:
> 

Agreed.  Consequently, I do not support "relaxing" the requirement.

> It's annoying to have a syntax that cannot be checked with a single 

This is a problem with a specific choice of implementation technology.
Though I have a little sympathy, I don't find this persuasive.

...
> name to have a tag that isn't properly formed be "well-formed".

Yup.

...
> in the current draft-02, we could say:
...

I disagree.  A "might not" describing the behaviour of a conformant
implementation is a red flag for me as a reader of a specification.
Furthermore, talking about the use of regular expressions as an
implementation strategy really doesn't belong in this document,
unless explicitly as an example.  But for me the complaint carries
as much weight as "string manipulation is awkward in FORTRAN II".

Patient: It hurts when I do this.
Doctor: Then don't do it.

Summary: I think we should leave this part of the material
as it was in RFC 4646.

Randy

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 16 Dec 2006 12:19:06 +0100

> It's annoying to have a syntax that cannot be checked with a single

For 4646 John proposed to sort extensions (excl. X) lexicographically,
with that we'd get what, 35**2 cases ?  For 35! my REXX calculator
claims that we would need about 1E40 lines, or about 2E38 RFC pages.

ABNF can't do everything.  Look at the syntax for the head-element in
XHTML, it's at the border where saying "at most one <base>, and one
and only one <title>, please" in prose would be clearer.

> A good analogy is checking UTF-8 validity. Checking if some text
> follows the encoding scheme is far simpler than checking that it
> also avoids non-shortest forms. But the latter is still required

After some lousy approximations I finally found that it's trivial -
especially the new UTF-8 with only 15 cases for a decoder rejecting
as many trailing bytes as plausible for invalid encodings.  The old
UTF-8 is dead until the aliens in need of 2**31 code points arrive.

Frank

-- 
      select                     /* for CESU remove both LB = 237 */
         when  LB < 192             then  LEN = -0 /* trail bytes */
         when  LB < 194             then  LEN = -1 /* bad C0 + C1 */
         when  LB < 224             then  LEN = +1
         when  LB = 224 & TOP =  8  then  LEN = -2 /* E08x is bad */
         when  LB = 224 & TOP =  9  then  LEN = -2 /* E09x is bad */
         when  LB = 237 & TOP = 10  then  LEN = -2 /* EDAx is bad */
         when  LB = 237 & TOP = 11  then  LEN = -2 /* EDBx is bad */
         when  LB < 240             then  LEN = +2
         when  LB = 240 & TOP =  8  then  LEN = -3 /* F08x is bad */
         when  LB < 244             then  LEN = +3
         when  LB = 244 & TOP =  8  then  LEN = +3 /* F48x is ok. */
         when  LB < 248             then  LEN = -3 /* bad F4 - F7 */
         when  LB < 252             then  LEN = -4 /* bad F8 - FB */
         when  LB < 254             then  LEN = -5 /* bad FC + FD */
         otherwise                        LEN = -0 /* bad FE + FF */
      end

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 20 Dec 2006 10:45:10 +0100

On Thu, Dec 14, 2006 at 09:05:03AM -0800,
 Mark Davis <[email-masked]> wrote 
 a message of 47 lines which said:

> with a single regular expression,

Sorry for not replying earlier, but I wanted to think about the
problem before giving an implementer's opinion.

I disagree with the suggestion. Being able to parse tags "with a
single regular expression" does not seem a worthy goal to me,
specially since this regexp would be awfully complex
(http://unicode.org/cldr/data/tools/java/org/unicode/cldr/util/data/langtagRegex.txt).

Testing that no singleton is repeated is quite simple (two lines in my
program).

> can be tested without access to the database.

Yes, it seems to me that drawing the border between well-formedness
and validity like that is simple to understand and of clear practical
use. I vote for keeping RFC 4646 here.