Subject: Re: [Ltru] Section 2.2.8 needs overhaul

From: "Doug Ewell" <[email-masked]>
Date: Fri, 13 Jun 2008 18:04:41 -0600

John Cowan <cowan at ccil dot org> wrote:

> We now divide grandfathered tags into two classes: those that are 
> accepted by the 'langtag' production and those that are not.  The 
> latter group are necessarily part of the syntax, and are enumerated in 
> the new 'irregular' production.  The former group are enumerated in 
> the prose of the new version of 2.2.8 found in the current editor's 
> draft, and their semantic anomalousness is explained there.

What is the value of dividing them into two classes I and F, both of 
which have productions, if the members of both I and F can only be 
interpreted semantically as whole tags and not as the sum of their 
parts, and the only difference is that the members of class F are 
accepted by the "langtag" production and the members of class I aren't?

Why not combine them into a single "grandfathered" production -- sort of 
like the one we already have, but with the members of class F added --  
and parse that first, as Kent requires for his class F?

--
Doug Ewell  *  Arvada, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Jun 2008 21:29:07 -0400

> and parse that first, as Kent requires for his class F?

ABNF, like most grammar formalisms, doesn't have any notion of "parse that
first", or priority between productions.  You wind up with a grammar which
can parse "zh-hakka" two ways: as a grandfathered tag, and as a language
subtag + variant tag.  The only way out of that is to write the 'variant'
production to exclude 'hakka' and 'lojban' and 'xiang' altogether,
as Kent does, which makes it impossible ever to register any of them
as legitimate variants.  Which is far too arbitrary to be worthwhile.

-- 
I could dance with you till the cows            John Cowan
come home.  On second thought, I'd              http://www.ccil.org/~cowan
rather dance with the cows when you             [email-masked]
came home.  --Rufus T. Firefly
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Fri, 13 Jun 2008 20:16:04 -0600

John Cowan <cowan at ccil dot org> wrote:

> ABNF, like most grammar formalisms, doesn't have any notion of "parse 
> that first", or priority between productions.

That's what I thought, but most of the people involved in this thread 
(as well as most squirrels) know more about ABNF than I do, and I hadn't 
yet heard anyone say "you can't do that" in response to Kent's proposal, 
so I assumed there must be some way to do that.

> You wind up with a grammar which can parse "zh-hakka" two ways: as a 
> grandfathered tag, and as a language subtag + variant tag.  The only 
> way out of that is to write the 'variant' production to exclude 
> 'hakka' and 'lojban' and 'xiang' altogether, as Kent does, which makes 
> it impossible ever to register any of them as legitimate variants. 
> Which is far too arbitrary to be worthwhile.

I would have thought it would be necessary to (a) enumerate all the 
variants, which would mean changing the ABNF every time a new variant is 
registered, or (b) define "variant" by means of a diabolically complex 
regular expression, excluding the forbidden ones while allowing 'hakban' 
and 'lojang' and such.

General question: Is this a problem that can realistically be solved to 
the satisfaction of the IETF LC reviewers, who will surely run the ABNF 
through their parsers and validators looking for errors or 
inconsistencies, as they did last time?

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Jun 2008 23:39:39 -0400

> so I assumed there must be some way to do that.

Not directly in the ABNF in any perspicuous way.  Some other similar
formalisms do have a way to say "A but not B", but ABNF doesn't.

> regular expression, excluding the forbidden ones while allowing 'hakban' 

Kent actually has three proposals: adding comments to the ABNF serving the
role of "A but not B", which would be as he says "informal", and probably
non-normative; adding normative text to 2.1 to exclude them; or the
"diabolically complex regular expression", which as he says would be
"very lengthy and confusing".

The status quo is to allow them to be parsed but give them special
semantics by the prose in 2.2.8.

> inconsistencies, as they did last time?

Parsers and validators will not understand comments or normative text,
and I think everyone would hurl at the sight of the necessary rules to
define 'variant' correctly:

variant = ("a"-"f" / "i"-"k" / "m"-"w" / "y"-"z" / DIGIT) 4*7alphanum
        / "g" ("b"-"t" / "v"-"z" / DIGIT) 3*6alphanum
        / "ga" ("a"-"t" / "v"-"z" / DIGIT) 2*5alphanum
        / "gau" ("a"-"k" / "m"-"z" / DIGIT) 1*4alphanum
        / "gaul" ("a"-"h" / "j"-"z" / DIGIT) 0*3alphanum
        / "gauli" ("a"-"r" / "t"-"z" / DIGIT) 0*2alphanum
        / "gaulis" ("a"-"g" / "i"-"z" / DIGIT) 0*1 alphanum
        / "gaulish" 1alphanum
        / "gu" ("a"-"n" / "p"-"z" / DIGIT) 2*5alphanum
        / "guo" ("a"-"x" / "z" / DIGIT) 1*4alphanum
        / "guoy" ("a"-"t" / "v"-"z" / DIGIT) 0*3alphanum
        / "guoyu" 1*3alphanum
        / "h" ("b"-"z" / DIGIT) 3*6alphanum
        / "ha" ("a"-"j" / "l"-"z" / DIGIT) 2*5alphanum
        / "hak" ("a"-"j" / "l"-"z" / DIGIT) 1*4alphanum
        / "hakk" ("b"-"z" / DIGIT) 0*3alphanum
        / "hakka" 1*3alphanum
        / "l" ("a"-"n" / "p"-"z" / DIGIT) 3*6alphanum
        / "lo" ("a"-"i" / "k"-"z" / DIGIT) 2*5 alphanum
        / "loj" ("a" / "c"-"z" / DIGIT) 1*4alphanum
        / "lojb" ("b"-"z" / DIGIT) 0*3alphanum
        / "lojba" ("a"-"m" / "o"-"z" / DIGIT) 0*2alphanum
        / "lojban" 1*2alphanum
        / "x" ("a"-"h" / "j"-"z" / DIGIT) 3*6alphanum
        / "xi" ("b"-"z" / DIGIT) 2*5alphanum
        / "xia" ("a"-"m" / "o"-"z" / DIGIT) 1*4alphanum
        / "xian" ("a"-"f" / "h"-"z" / DIGIT) 0*3alphanum
        / "xiang" 1*3alphanum
        / DIGIT 3alphanum

This needs to be further obfuscated by expanding the non-ABNF ranges like
"a"-"c" to "%41-43 / %61-63".

-- 
LEAR: Dost thou call me fool, boy?      John Cowan
FOOL: All thy other titles              http://www.ccil.org/~cowan
             thou hast given away:      [email-masked]
      That thou wast born with.
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Kent Karlsson" <[email-masked]>
Date: Sat, 14 Jun 2008 13:00:08 +0200

John Cowan wrote:
> subtag + variant tag.

While that is true, commonly used parser generation tools used for
deterministic parsing (and lexing) of many formal languages, from
simple to complex (command languages, programming languages, and,
why not, language tags, esp. if they are embedded into some other
formal language being parsed), solve this ex-grammatically. And if
you were to write a deterministic parser without using those parser
generation tools, you would solve the prioritisation by choosing
which order to try the productions.

If you insist on using a parser (made by generator or "by hand")
that returns all possible parses of a given string, you would get
two parse trees for these problem cases, one of which can be weeded 
out based on registry data (which at that point in the process is 
quite ok, since *at that point* you will then be looking at registry 
data anyway to determine validity). If you are interested only in 
"pass"/"nopass" for wellformedness, all you need to see is that you
got more than zero parse trees in return. Getting two instead of
just one parse tree is not a problem. (But using a non-deterministic
parser for this seems to be overkill... Non-deterministic here means
"all, maybe more than one", not "choose one arbitrarily".)

But using the -15 grammar, always getting at most one parse tree, but
sometimes reshape it (actually or conceptually) based on registry data,
I find very strange.

>  The only way out of that is to write the 'variant'

Well, that wasn't my suggestion. While I did so for the 'bok', 'nyn',
and 'min' (which are all ineligeble for ever becoming extlangs, IIUC),
I did not suggest that for 'hakka' etc. There, my suggestion was (here
removing the spurious '-'):

  'lojban' must not match if the prefix is 'art' [or 'art-' if you want
	the concrete prefix],
  'gaulish' must not match if the prefix is 'cel',
  'guoyo', 'hakka', and 'xiang' must not match if the prefix is 'zh'

That way 'guoyo' etc. are still available to be registered as variants,
just not for the specific prefixes listed.

While not necessary, it would be fine to use the same approach for
'bok', 'nyn', and 'min' (and that would simplify the actual
implementation of a deterministic-by-disambiguation-rules parser...).

Even so, this exclusion *can* be baked into the grammar making it
deterministic, in a way similar to what you (John) showed. But
no-one would like that solution.

	/kent k

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 14 Jun 2008 10:45:53 -0700

http://inter-locale.com/ID/draft-ietf-ltru-4646bis-16.html#syntax

1. This is really a tempest in a teapot. To address this, we can make the
following change.

The grandfathered tags registered underRFC
3066<http://inter-locale.com/ID/draft-ietf-ltru-4646bis-16.html#RFC3066>
[RFC3066],
a fixed list that can never change, are the only exception to this ability
to infer meaning from subtag structure.

=>

[new paragraph]This ability to infer meaning from subtag structure has a
small, fixed list of exceptions: the grandfathered tags registered under
RFC 3066 [RFC3066].
Some of these grandfathered tags are irregular, and listed in the ABNF
below. There are nine others that
are syntactically
regular, but whose meaning is not derivable from their subtags. For
more information, see section
2.8.8.

2. I do agree with Kent's proposed change to the ABNF for consistency:

                [extlang]              *; sometimes followed by *

=>

                ["-" extlang]              *; sometimes followed by *

and

extlang       = "-" 3ALPHA             *; selected ISO 639 codes*
                *2[ "-" 3ALPHA]        *; permanently reserved*

=>

extlang       = 3ALPHA             *; selected ISO 639 codes*
                *2( "-" 3ALPHA)        *; permanently reserved*

http://inter-locale.com/ID/draft-ietf-ltru-4646bis-16.html#preexisreg

3. I agree with John that the following is not quite grammatical.

The nine grandfathered tags "art-lojban", "cel-gaulish", "no-bok", "no-nyn",
"zh-guoyu", "zh-hakka", "zh-min", "zh-min-nan", and "zh-xiang" match the
'langtag' production in Figure
1<http://inter-locale.com/ID/draft-ietf-ltru-4646bis-16.html#ABNF>
.

I suggest rewriting the sentence for clarity to:

Nine grandfathered tags
match the 'langtag' production in Figure 1: "art-lojban",
"cel-gaulish", "no-bok", "no-nyn", "zh-guoyu", "zh-hakka", "zh-min",
"zh-min-nan", and "zh-xiang".

Mark

On Sat, Jun 14, 2008 at 4:00 AM, Kent Karlsson <[email-masked]>
wrote:

> that

---

From: John Cowan <[email-masked]>
Date: Sat, 14 Jun 2008 14:29:48 -0400

Mark Davis scripsit:

> [new paragraph]This ability to infer meaning from subtag structure has
> a small, fixed list of exceptions: the grandfathered tags registered
> under RFC 3066 [RFC3066].  Some of these grandfathered tags are
> irregular, and listed in the ABNF below. There are nine others that
> are syntactically regular, but whose meaning is not derivable from
> their subtags. For more information, see section 2.8.8.

+1

> 2. I do agree with Kent's proposed change to the ABNF for consistency:

+1

> "zh-min-nan", and "zh-xiang".

+0

-- 
John Cowan  [email-masked]   http://ccil.org/~cowan
It's the old, old story.  Droid meets droid.  Droid becomes chameleon.
Droid loses chameleon, chameleon becomes blob, droid gets blob back
again.  It's a classic tale.  --Kryten, Red Dwarf
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Martin Duerst <[email-masked]>
Date: Mon, 16 Jun 2008 11:35:47 +0900

It is possible to define additional rules such as
'first-match-wins'. As an example, the URI spec
(http://www.ietf.org/rfc/rfc3986.txt) says:

   The syntax rule for host is ambiguous because it does not completely
   distinguish between an IPv4address and a reg-name.  In order to
   disambiguate the syntax, we apply the "first-match-wins" algorithm:
   If host matches the rule for IPv4address, then it should be
   considered an IPv4 address literal and not a reg-name.

Regards,   Martin.

>as legitimate variants.  Which is far too arbitrary to be worthwhile.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 17 Jun 2008 10:03:45 +0200

And similar methods are commonly used when specifying programming
languages. I'd like to see John trying to convince an established
programming language committee to change the grammar they are using
because it is ambiguous...

I don't really see why LTRU has been stricken by the "not invented
here syndrome" on this point. Especially since the replacement
invention is not particularly nice or nifty, and is far from
necessary. Let's employ conventional methods, please.

	/kent k

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Martin Duerst
> Sent: Monday, June 16, 2008 4:36 AM
> To: John Cowan; Doug Ewell
> Cc: LTRU Working Group
> Subject: Re: [Ltru] Section 2.2.8 needs overhaul
>