Subject: [Ltru] RFC 4646 production "grandfathered" considered harmful

From: John Cowan <[email-masked]>
Date: Sun, 17 Sep 2006 02:19:12 -0400

Section 2.2.9 of RFC 4646 says:

   An implementation that claims to check for well-formed language tags
   MUST:

   o  Check that the tag and all of its subtags, including extension and
      private use subtags, conform to the ABNF OR that the tag is on the
      list of grandfathered tags.

   o  Check that singleton subtags that identify extensions do not
      repeat.  For example, the tag "en-a-xx-b-yy-a-zz" is not well-
      formed.

(I have emphasized the word OR in the first bullet point.)

Unfortunately, this wording allows too much.  For example, the invalid tag
"ra-bb-it" matches the "grandfathered" rule in the ABNF.  Therefore it
winds up being well-formed even though it cannot be analyzed as a sequence
of subtags and is not on the grandfathered list either.

To avoid this, we can take one of two actions:

1) Remove the "grandfathered" production in the ABNF altogether, and use
the "OR" in the conformance clause to allow the irregular grandfathered
tags (that is, those which don't match the "langtag" or "privateuse"
productions) to be well-formed.  The danger here is that people will
implement the ABNF only, and the grandfathered tags will become outright
unusable rather than merely discouraged.

2) Add an explicit "irregular" production in place of the "grandfathered"
production which explicitly enumerates the 17 irregular grandfathered
tags, thus:

irregular = "en-GB-oed" / "i-ami" / "i-bnn" / "i-default"
            / "i-enochian" / "i-hak" / "i-klingon" / "i-lux" / "i-mingo"
            / "i-navajo" / "i-pwn" / "i-tao" / "i-tay" / "i-tsu"
            / "sgn-BE-fr" / "sgn-BE-nl" / "sgn-CH-de"

It is safe to enumerate this list explicitly, as it can neither grow
nor shrink.  It's true that all the tags except "i-default" can become
deprecated, but that makes no difference to well-formed processors.
The other grandfathered tags in the registry are all well-formed already
and do not need to be in this list.

In this case the conformance clause can be simplified by omitting the
second part of the "OR".

I favor choice 2.

-- 
John Cowan                                   [email-masked]
        "You need a change: try Canada"  "You need a change: try China"
                --fortune cookies opened by a couple that I know

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 17 Sep 2006 12:23:30 +0200

John Cowan wrote:
 
> Add an explicit "irregular" production

Interesting idea.  Maybe add "zh-hakka" as "almost irregular",
or let's _define_ hakka as deprecated zh variant.  For zh-cmn
I hope 4646bis will solve a "missing cmn" issue as side effect.

Frank

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 17 Sep 2006 18:42:03 +0200

On Sun, Sep 17, 2006 at 02:19:12AM -0400,
 John Cowan <[email-masked]> wrote 

> production which explicitly enumerates the 17 irregular grandfathered

+1

(When I wrote a parser, I suppressed that production from my grammar,
replacing it with a list of grandfathered tags, extracted from the
registry. I prefer when the parser's gramamr is as close as possible
from the RFC's gramamr.)

> nor shrink.

Yes, that's the big reason for me.

---

From: Addison Phillips <[email-masked]>
Date: Sun, 17 Sep 2006 10:11:51 -0700

Good point. I like the idea of the irregular list.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 17 Sep 2006 15:08:58 -0700

I do as well. Note that the BNF provides both checks of well-formedness
(partial, because of repeating extensions), but also a division into
semantic units. So we'd have to do something like:

grandfathered = irregular
            / 1*3ALPHA 1*2("-" (2*8alphanum)) ; registered
grandfathered codes that are well-formed, but would not otherwise be
valid

We could also break them down into other divisions ;-)

grandfathered = "i-" irregular
            / irregularEverson
            / 1*3ALPHA 1*2("-" (2*8alphanum)) ; registered
grandfathered codes that are well-formed, but would not otherwise be
valid

irregulari = "ami" / "bnn" / "default" / "enochian" / "hak" / "klingon" /
"lux" / "mingo" / "navajo" / "pwn" / "tao" / "tay" / "tsu"
irregularEverson = "en-GB-oed" / "sgn-BE-fr" / "sgn-BE-nl" / "sgn-CH-de"

Mark

---

From: John Cowan <[email-masked]>
Date: Sun, 17 Sep 2006 18:42:21 -0400

Mark Davis scripsit:

> semantic units. 

Well, it does and it doesn't.  It's true that "grandfathered" matches
all the grandfathered tags (and much else as well); but "langtag"
matches a good many grandfathered tags.  So I'd like to make a
firm distinction between syntactic irregularity (tags which cannot
be decomposed into subtags) and semantic grandfathering (tags which,
if they can be decomposed, produce the wrong meaning).

> "lux" / "mingo" / "navajo" / "pwn" / "tao" / "tay" / "tsu"

This would save a little vertical space, but not much else.

-- 
"The serene chaos that is Courage, and the phenomenon   [email-masked]
of Unopened Consciousness have been known to the        John Cowan
Great World eons longer than Extaboulism."
"Why is that?" the woman inquired.
"Because I just made that word up", the Master said wisely.
        --Kehlog Albran, The Profit             http://www.ccil.org/~cowan

---

From: Martin Duerst <[email-masked]>
Date: Sun, 17 Sep 2006 16:32:12 +0900

Like John, I favor choice 2.    Regards,   Martin.

>In this case the conformance clause can be simplified by omitting the

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Mon, 18 Sep 2006 12:05:54 +0900

[chair hat on]
I have seen at least four people approving the idea to list
grandfathered/irregular tags explicitly, and nobody opposed.
I think that in principle, we seem to have consensus to go this
way, although some details may need to be worked out (see below).
If anybody disagrees with this assessment, please state so soon.

[chair hat off]
I clearly thing that it is a bad idea to include somebody's name
in a nonterminal of an ABNF. This is true both in general and also
in this specific case. When Michael proposed and approved the
tags listed below, they were not in any way illegal or irregular.

Regards,    Martin.

At 07:08 06/09/18, Mark Davis wrote:
>I do as well. Note that the BNF provides both checks of well-formedness (partial, because of repeating extensions), but also a division into semantic units. So we'd have to do something like:
>
>
>            / 1*3ALPHA 1*2("-" (2*8alphanum)) ; registered grandfathered codes that are well-formed, but would not otherwise be valid
>
>            / 1*3ALPHA 1*2("-" (2*8alphanum)) ; registered grandfathered codes that are well-formed, but would not otherwise be valid
>irregulari = "ami" / "bnn" / "default" / "enochian" / "hak" / "klingon" / "lux" / "mingo" / "navajo" / "pwn" / "tao" / "tay" / "tsu"

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 18 Sep 2006 05:52:34 +0200

> irregulari = "ami" / "bnn" / "default" / "enochian" / "hak" /
> "klingon" / "lux" / "mingo" / "navajo" / "pwn" / "tao" / "tay" / "tsu"

> irregularEverson = "en-GB-oed" / "sgn-BE-fr" / "sgn-BE-nl" /

Splitting i-anything from the rest is fine, IMO unnecessary to
enumerate them in the ABNF:  Any decent parser could handle
x-anything, it should be also able to handle i-anything.  With
that you'd get:

grandfathered = i-legacy / irregular / ( 2*3ALPHA "-" 3*8ALPHA )
i-legacy      = "i-" 3*8ALPHA 
irregular     = sgn-legacy / "en-GB-oed" / "zh-min-nan"
sgn-legacy    = "sgn-" ( "BE-fr" / "BE-nl" / "CH-de" )

Cheating, zh-min-nan isn't really irregular, but together with
en-GB-oed and <sgn-legacy> it's the list of grandfathered tags
with three parts. 

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 17 Sep 2006 21:11:19 -0700

> I clearly thing that it is a bad idea to include somebody's name

That was not intended seriously -- sorry for the confusion.

Mark

On 9/17/06, Martin Duerst <[email-masked]> wrote:
> >            / 1*3ALPHA 1*2("-" (2*8alphanum)) ; registered grandfathered
> codes that are well-formed, but would not otherwise be valid
> >
> >            / 1*3ALPHA 1*2("-" (2*8alphanum)) ; registered grandfathered
> codes that are well-formed, but would not otherwise be valid

---

From: John Cowan <[email-masked]>
Date: Mon, 18 Sep 2006 00:50:46 -0400

Frank Ellermann scripsit:

> x-anything, it should be also able to handle i-anything.  

There's no reason to allow i-bogon to be well-formed.

> en-GB-oed and <sgn-legacy> it's the list of grandfathered tags

Just so.  The ABNF is syntactic, so I listed only the elements
that do not match the "langtag" production.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
The present impossibility of giving a scientific explanation is no proof
that there is no scientific explanation. The unexplained is not to be
identified with the unexplainable, and the strange and extraordinary
nature of a fact is not a justification for attributing it to powers
above nature.  --The Catholic Encyclopedia, s.v. "telepathy" (1913)

---

From: Addison Phillips <[email-masked]>
Date: Mon, 18 Sep 2006 07:55:21 -0700

> If anybody disagrees with this assessment, please state so soon.

[editor hat on]

I'll prepare an updated ABNF shortly.

> tags listed below, they were not in any way illegal or irregular.

[contributor hat on]

I agree. I think that proposal was made tongue-in-cheek. I intend to 
start with the production "irregular" shown below

Addison

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 19 Sep 2006 00:38:23 +0200

Addison Phillips wrote:

> I intend to start with the production "irregular" shown below

>>>            / 1*3ALPHA 1*2("-" (2*8alphanum)) ; [...]

In that version s/1*3/2*3/.  If you've a good idea to split
"two parts" (the 1 in 1*2) from "three parts" (the 2 in 1*2)
please propose it.

>>>           / "i-enochian" / "i-hak" / "i-klingon" / "i-lux"
[...]
>>>           / "sgn-BE-fr" / "sgn-BE-nl" / "sgn-CH-de"

If i-anything clearly belongs to <irregular> I'm not thrilled
by enumerating the 13 i-anything explicitly.  10 of the 13 are
deprecated in 4645bis, only i-default, i-enochian, and i-mingo
will survive.  There's a registry with a museum for this cruft.

Talking about unnecessary details in ABNF could backfire, e.g.
if somebody decides to use i-klingon "because it's legal - as
seen in ABNF".

Unfortunately deprecating the complete zoo won't fly, i-default
is too important.  And the opposite proposal "no individually
registered language subtags, register i-anything for temporary
workarounds" was rejected.  Are you still sure that that's as
it should be ?

Frank

---

From: John Cowan <[email-masked]>
Date: Mon, 18 Sep 2006 21:50:01 -0400

Addison Phillips scripsit:

> >>           grandfathered codes that are well-formed, but would not 
> >>           otherwise be valid

I still don't understand the purpose of the second alternative.
It matches a random subset of well-formed tags which happen to
include all the grandfathered/redundant tags, but they also
match "langtag", and it allows all sorts of cruft to be well-formed
that will never be needed or useful.

-- 
A rabbi whose congregation doesn't want         John Cowan
to drive him out of town isn't a rabbi,         http://www.ccil.org/~cowan
and a rabbi who lets them do it                 [email-masked]
isn't a man.    --Jewish saying

---

From: Addison Phillips <[email-masked]>
Date: Mon, 18 Sep 2006 19:38:36 -0700

(laughing)

In the actual editor's copy, I replaced it with 'langtag' for that reason.

Addison

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 19 Sep 2006 05:39:51 +0200

John Cowan wrote:

>>>> 1*3ALPHA 1*2("-" (2*8alphanum))

> I still don't understand the purpose of the second

Right, now I also see it.  An enumeration is useful if
it could be mistaken as well-formed "regular" tag.  For
a parser it's important to match some "irregular" tags
_before_ it starts to match the "regular" tags.

 From that POV x-anything and i-anything are harmless,
they're never ambiguous.  And sgn-BE-nl or en-GB-oed
are also not critical, they never match "regular" tags,
because they're not well-formed.

At some point (not necesarily a.s.a.p.) they have to
match, after all they're valid.  Therefore these tags
could be enumerated as part of <irregular>.

The critical tags are well-formed grandfathered tags
like zh-xiang, because there is no registered variant
xiang (at the moment).

You're trying to clean it up for boont and scouse, if
that's accepted we end up with nine critical tags:

art-lobjan, cel-gaulish, no-nyn, no-bok, zh-guoyu,
zh-hakka, zh-min, zh-min-nan, zh-xiang.

We're free to do whatever we like with the potential
variants in those tags (declare them to be reserved,
or deprecate them).

We cannot preempt bok, min, nan and nyn, and they'll
be used soon (probably):

- bok is a future language subtag
- min is a future language subtag
- nan is a future extlang (and even related, good)
- nyn is a future language subtag

So from the 9 (or 11) critical tags we get a subset
of 4 tags "no-extlang-or-not-what-you-might-expect".

That results in about four subsets for an improved
grandfathered ABNF:

13 i-anything, detailed ABNF enumeration unnecessary
 4 extlang bogeys: no-bok, no-nyn, zh-min, zh-min-nan
 5 variant bogeys: art-lobjan, cel-gaulish, etc.
 2 variant bogeys which might go away before "date C"
 4 less critical: en-GB-oed, and the three sgn-XX-yy

If the ABNF is improved we should consider to support
left-to-right depth-first matching, or in other words
mention <grandfathered> _before_ <langtag>.

Frank

---

From: John Cowan <[email-masked]>
Date: Tue, 19 Sep 2006 01:27:54 -0400

> _before_ it starts to match the "regular" tags.

A non-validating parser doesn't have to distinguish between
semantically decomposable tags and indecomposable ones with
special meanings.  All it has to do is have an ABNF that
specifies what tags do and don't match, to which it also
must add the rule against repeated singletons.

For that purpose, only the irregular-syntax tags need
to appear in the ABNF; the ones with regular syntax but
irregular semantics do not.

>  From that POV x-anything and i-anything are harmless,

i-anything is *not* harmless; it means that i-foobar is
well-formed even though it is neither composed of valid
subtags nor itself a valid grandfathered tag.  i-foobar
should be ill-formed.

Validating parsers can use a different set of productions
if it suits them better, though I don't see why it would:
the simplest approach is to check for well-formedness,
then check the registry to see if the tag is grandfathered,
and if not, decompose them.

-- 
John Cowan     http://ccil.org/~cowan    [email-masked]
Monday we watch-a Firefly's house, but he no come out.  He wasn't home.
Tuesday we go to the ball game, but he fool us.  He no show up.  Wednesday he
go to the ball game, and we fool him.  We no show up.  Thursday was a
double-header.  Nobody show up.  Friday it rained all day.  There was no ball
game, so we stayed home and we listened to it on-a the radio.  --Chicolini

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 18 Sep 2006 22:59:53 -0700

A validating parser is *not* limited to all of the possible strings in the
ABNF. That is already true in 4646 -- duplicate singleton extension tags are
forbidden although that cannot be expressed in the ABNF.

Saying that grandfathered tags are limited to i-anything is certainly an
improvement on the current situation, which is:

 grandfathered = 1*3ALPHA 1*2("-" (2*8alphanum))

Although Frank has a point, I think my preferred approach would be to simply
define but immediately deprecate subtags like lobjan -- that removes them
from the grandfathered list. For the ABNF, we are then only left with the
items that are syntactically ill-formed, which we could express with:

grandfathered = "i-" irregularI
            / "sgn-" irregularSgn
            / irregularOther

irregularI = "ami" / "bnn" / "default" / "enochian" / "hak" / "klingon" /
"lux" / "mingo" / "navajo" / "pwn" / "tao" / "tay" / "tsu"
irregularSgn= "BE-fr" / "BE-nl" / "CH-de"
irregularOther = "en-GB-oed"

We then have the smallest ABNF that (with the exception of duplicate
singletons), describes the well-formed cases without letting anything else
in.

Mark