Subject: [Ltru] Section 2.2.8 needs overhaul

From: John Cowan <[email-masked]>
Date: Wed, 11 Jun 2008 13:33:13 -0400

The text of 2.2.8 is misleading, and has been since RFC 4646: there are
grandfathered tags all of whose subtags are in the registry, such as
"sgn-be-fr".  The essence of a grandfathered tag is that its subtags do
not mean what you think they mean, not that they are or aren't in the
registry.  I propose the following rather more specific replacement text:

   Prior to RFC 4646, whole language tags were registered according
   to the rules in RFC 1766 and/or RFC 3066.  These registered tags
   maintain their validity.  Of those tags, those that were made obsolete
   or redundant by the advent of RFC 4646, by this document, or by
   subsequent registration of subtags are maintained in the registry
   in records as "redundant" records.  Those tags whose interpretation
   cannot be determined by considering the meanings of their subtags
   in the registry (or, in some cases, that contain subtags not in the
   registry at all) are maintained in the registry using records of the
   "grandfathered" type.

   The nine grandfathered tags "art-lojban", "cel-gaulish", "no-bok",
   "no-nyn", "zh-guoyu", "zh-hakka", "zh-min", "zh-min-nan", and
   "zh-xiang" match the 'langtag' production in Figure 1.  However,
   the second subtag in each is neither an extended language subtag
   nor a variant subtag; instead, each tag in its entirety represents a
   language or language collection.  The remaining grandfathered tags do
   not match the 'langtag' production, and are listed in the 'irregular'
   production in Figure 1.  With the exception of "en-GB-oed", which is a
   variant of "en-GB", each of them in its entirety represents a language.

   Many of the grandfathered tags have been superseded by other
   registrations: their record contains a Preferred-Value field that
   ought to be used to form language tags representing that value.

I also propose that bullet point 17 of section 3.4, which deals
with converting grandfathered tags to redundant ones, be eliminated.
After "zh-cmn", "zh-cmn-Hans", "zh-cmn-Hant", "zh-gan", "zh-wuu", and
"zh-yue" are changed to redundant tags by RFC 4645bis, we will have only
the above nine cases, none of which will be made redundant, and the 17
irregular cases, which cannot be.

-- 
You know, you haven't stopped talking           John Cowan
since I came here. You must have been           http://www.ccil.org/~cowan
vaccinated with a phonograph needle.            [email-masked]
        --Rufus T. Firefly
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 12 Jun 2008 11:28:17 -0700

I meant to reply to this earlier, but noted that some work was needed to incorporate it.

This is generally good text. However, it misrepresents the nature of the "redundant" record type. Redundant records must not just have each subtag appear in the registry, but must be truly redundant--that is, the subtags must have the identical semantic meaning. Thus "zh-min" can never be redundant because 'min' means something else.

Therefore, I have incorporated the suggested text with edits as follows:

--
<section anchor="preexisreg" title="Grandfathered Registrations">

<t>Prior to RFC 4646, whole language tags were registered according to the rules in RFC 1766 and/or RFC 3066. All of these registered tags are and remain valid language tags. </t>

<t>Many of these registered tags were made redundant by the advent of RFC 4646, by this document, or by subsequent registration of subtags. A redundant grandfathered tag is one whose individual subtags appear with the same semantic meaning in the regustry. For example, the tag "zh-Hant" (Traditional Chinese) can now be composed from the subtags 'zh' (Chinese) and 'Hant' (Han script traditional variant). These redundant tags are maintained in the registry as "redundant" records.</t>

<t>Those tags that do not match the 'langtag' production in the ABNF in this document or that contain subtags that do not individually appear in the registry are maintained in the registry in records of the "grandfathered" type.</t>

<t>Nine grandfathered tags "art-lojban", "cel-gaulish", "no-bok", "no-nyn", "zh-guoyu", "zh-hakka", "zh-min", "zh-min-nan", and "zh-xiang" match the 'langtag' production in <xref target="ABNF"></xref>.  However, the second subtag in each is neither an extended language subtag nor a variant subtag; instead, each tag, in its entirety, represents a language or collection of languages.</t>

<t>The remaining grandfathered tags do not match the 'langtag' production, and are listed in the 'irregular' production in <xref target="ABNF"></xref>.  With the exception of "en-GB-oed", which is a variant of "en-GB", each of them, in its entirety, represents a language.</t>

<t>Many of the grandfathered tags have been superseded the subsequent addition of new subtags: each superseded record contains a Preferred-Value field that ought to be used to form language tags representing that value. For example, the tag "art-lojban" is superseded by the primary language subtag 'jbo'.</t>

</section>
--

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of John Cowan
> Sent: Wednesday, June 11, 2008 10:33 AM
> To: [email-masked]
> Subject: [Ltru] Section 2.2.8 needs overhaul
>

---

From: John Cowan <[email-masked]>
Date: Thu, 12 Jun 2008 14:59:16 -0400

A few nits:

Phillips, Addison scripsit:

> <t>Many of these registered tags were made redundant by the advent of
> RFC 4646, by this document, or by subsequent registration of subtags. A
> redundant grandfathered tag is one whose individual subtags appear
> with the same semantic meaning in the regustry. For example, the tag
> "zh-Hant" (Traditional Chinese) can now be composed from the subtags
> 'zh' (Chinese) and 'Hant' (Han script traditional variant). These
> redundant tags are maintained in the registry as "redundant"
> records.</t>

s/redundant grandfathered/redundant/; s/regustry/registry/.

> <t>Those tags that do not match the 'langtag' production in the ABNF
> in this document or that contain subtags that do not individually
> appear in the registry are maintained in the registry in records of the
> "grandfathered" type.</t>

I continue my objection to this as missing the point: even if "xiang" were
added as a variant (e.g. for Mandarin as spoken in the town of Xiang),
it would still be true that "zh-xiang" was grandfathered.  I propose:

   Those tags whose subtags do not follow the patterns explained
   in section 2.2 are maintained in the registry in records of the
   "grandfathered" type.

> <t>Nine grandfathered tags "art-lojban", "cel-gaulish", "no-bok",
> "no-nyn", "zh-guoyu", "zh-hakka", "zh-min", "zh-min-nan", and "zh-xiang"
> match the 'langtag' production in <xref target="ABNF"></xref>.  However,
> the second subtag in each is neither an extended language subtag nor
> a variant subtag; instead, each tag, in its entirety, represents a
> language or collection of languages.</t>

Either "The nine grandfathered tags", or "Nine grandfathered tags,
namely" with a comma following the last one.

-- 
"But I am the real Strider, fortunately,"       John Cowan
he said, looking down at them with his face     [email-masked]
softened by a sudden smile.  "I am Aragorn son  http://www.ccil.org/~cowan
of Arathorn, and if by life or death I can
save you, I will."  --LotR Book I Chapter 10
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 12 Jun 2008 12:22:06 -0700

(editor hat ON)

> s/redundant grandfathered/redundant/; s/regustry/registry/.

DONE.

Note that I said:

"A redundant tag is a grandfathered registration whose..."

.. in place of just "redundant tag".

> it would still be true that "zh-xiang" was grandfathered.

We are (trying to) say the same thing, I think. It isn't just the appearance of the subtag but also that the semantic meaning must match.

>  I
> propose:

Section 2.2 doesn't really say, though. So I put all three cases into the sentence thusly:

<t>Those tags that do not match the 'langtag' production in <xref target="ABNF"></xref>, that contain subtags that do not individually appear in the registry, or whose subtags appear but with a different semantic meaning are maintained in the registry as records of the "grandfathered" type.</t>

> namely" with a comma following the last one.
Restored your original wording (The nine...)

Addison

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Jun 2008 00:34:37 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Phillips, Addison

> <t>Those tags that do not match the 'langtag' production in the ABNF in
> this document or that contain subtags that do not individually appear
> in the registry are maintained in the registry in records of the

Catching up, so apologies if you've already addressed this:

You're missing the semantic element; e.g. the subtags in no-bok all appear individually in the registry, and this tag matches the langtag production (up to the point where the prefix value in the registry for the extlang subtag is checked).

Btw, it's not clear to me how a grandfathered tag like "zh-hakka" for which one of the subtags is not in the registry is accommodated at all now by the ABNF. Please explain.

Peter

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Jun 2008 03:39:42 -0400

Peter Constable scripsit:

> Btw, it's not clear to me how a grandfathered tag like "zh-hakka"
> for which one of the subtags is not in the registry is accommodated
> at all now by the ABNF. Please explain.

The BNF thinks it's a language subtag followed by a variant subtag.  The new version of 2.2.8
(grandfathered tags) explains that it isn't.

-- 
Business before pleasure, if not too bloomering long before.
        --Nicholas van Rijn
                John Cowan <[email-masked]>
                    http://www.ccil.org/~cowan
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Jun 2008 00:49:02 -0700

Well, presumably rule 4 of 2.2.5 determines that it is not a variant subtag -- which is why it strikes me as not at all being accommodated by the ABNF. (2.2.8 doesn't provide some additional means to syntactically sanction a tag not sanctioned by 2.1 - 2.2.7.) What am I missing? Don't we still need

   Language-Tag = langtag
                  ...
                  / grandfathered

as we did before?

Peter

> From: John Cowan [mailto:[email-masked]]
> Sent: Friday, June 13, 2008 12:40 AM
> To: Peter Constable
> Cc: [email-masked]
> Subject: Re: [Ltru] Section 2.2.8 needs overhaul
>

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Jun 2008 10:13:23 -0400

Peter Constable scripsit:

> Well, presumably rule 4 of 2.2.5 determines that ['hakka' in "zh-hakka"]
> is not a variant subtag -- which is why it strikes me as not at all
> being accommodated by the ABNF.

No, no, we are fanatical Chomskyites here, and our syntax is autonomous.
We do not allow the semantic rules in section 2.2 to sully in any
way the purity of the phrase-structure rules listed in section 2.1.
Though semantically 'hakka' is not a variant tag, nevertheless "zh-hakka"
is accepted by our productions, and so our colorless green ideas are
licensed to sleep with a fury quite unmitigated by mere selection
restrictions.

(To the non-linguists: this is an in-joke with *no* political
implications.)

> (2.2.8 doesn't provide some additional means to syntactically
> sanction a tag not sanctioned by 2.1 - 2.2.7.)

True.

> What am I missing? Don't we still need
> 
>    Language-Tag = langtag
>                   ...
>                   / grandfathered

> as we did before?

We now divide grandfathered tags into two classes: those that are
accepted by the 'langtag' production and those that are not.  The latter
group are necessarily part of the syntax, and are enumerated in the new
'irregular' production.  The former group are enumerated in the prose
of the new version of 2.2.8 found in the current editor's draft, and
their semantic anomalousness is explained there.

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
You cannot enter here.  Go back to the abyss prepared for you!  Go back!
Fall into the nothingness that awaits you and your Master.  Go! --Gandalf
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Kent Karlsson" <[email-masked]>
Date: Fri, 13 Jun 2008 17:55:53 +0200

Well, Chomsky intended the "context free grammars" (in IETF
instantiated by (IETF) ABNFs) to be useable for parsing *natural*
languages. And while I haven't studied context free grammars
for natural languages, I know that those grammars are *highly*
ambiguous, using other means to weed out hordes of senseless
parsings... (But not, as far as I know, does one twist one parsing
into a completely different one.)

And for denotational semantics (based on an unambiguous, or rule
disambiguated context free grammar), those are *highly* syntax-based,
and a particular *syntactic pattern* (parse tree pattern, if you
like) is given a particular semantics *composed from the semantics*
of the *syntactic* parts.

Either way, the grammar in draft -15 is a no-go.

We are not just plainly and blindly generating a set of strings,
are we?

	/kent k

> Sent: Friday, June 13, 2008 4:13 PM
> To: Peter Constable
> Cc: [email-masked]

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Jun 2008 09:18:06 -0700

> is accepted by our productions

I believe for Chomsky "autonomous syntax" meant a separation between "syntax" (whatever that includes) on the one hand and semantics, pragmatics and general cognition on the other. Radford, in his 1988 book on TG (quoting this rather than more direct source material because that's what's at hand at the moment), describes autonomous syntax thus: "No syntactic rule can make reference to pragmatic, phonological, or semantic information." I'm pretty sure that, for Chomsky, the lexicon was viewed as part of syntax. While section 2.2 does discuss semantics, it also spells out details regarding how the lexicon is defined. The problem I see is that "hakka" is not in the available lexicon.

> their semantic anomalousness is explained there.

But they do not fit anywhere in the lexicon.

Peter

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Jun 2008 13:32:38 -0400

Peter Constable scripsit:

> I'm pretty sure that, for Chomsky, the lexicon was viewed as part
> of syntax.

I think he meant that the properties of lexemes, rather than the lexemes
themselves, were available to the syntax: it is the fact that DOG is
an animate count noun that the syntax can make use of; there can be no
syntactic rule specific to DOG.

In any case, it's clear from the definition of validity in 2.2.9 that we
do not take this view.  A tag like "ab-abc-abcd-abcde" is well-formed
but not valid; specifically, it is like "'Twas brillig and the slithy
toves did gyre and gimble in the wabe" before Humpty Dumpty glosses it.

> But they do not fit anywhere in the lexicon.

Even if 'hakka' were to be added in the lexicon, "zh-hakka" would
still be grandfathered in the sense of 2.2.8.  As I said, the mark
of a grandfathered tag is that its subtags do not mean what you think
they mean; a grandfathered tag has no compositional meaning.

-- 
All Norstrilians knew what laughter was:        John Cowan
it was "pleasurable corrigible malfunction".    [email-masked]
        --Cordwainer Smith, Norstrilia
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Jun 2008 13:47:12 -0400

Kent Karlsson scripsit:

> languages. 

He gave up that idea more than fifty years ago, for the excellent reason
that it cannot be made to work without allowing a denumerably infinite
number of productions.

> of the *syntactic* parts.

Except when it isn't.  In the Basic-Plus language, for example, the
meaning of "ON ERROR GOTO 25" is compositional based on the meanings
of "ON ERROR", "GOTO", and "25" (in this context).  However, "ON ERROR
GOTO 0" is a non-compositional expression meaning "Don't trap errors at all."
Granted that this is a wart, languages do accumulate warts over time.

> We are not just plainly and blindly generating a set of strings,

Yes, we are: those blindly generated strings are the well-formed
language tags.  The rules in 2.2.9 then restrict that set to
a subset called the valid language tags, which is what we care about.
After the application of 4645bis, there will be about 10^11 valid
language tags with compositional meaning, and precisely 32 without.
To call that a wart is to give it far too much prominence.

-- 
An observable characteristic is not necessarily         John Cowan
a functional requirement.  --John Hudson                [email-masked]
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Kent Karlsson" <[email-masked]>
Date: Fri, 13 Jun 2008 20:54:01 +0200

> that it cannot be made to work without allowing a denumerably infinite

That hasn't stopped people from trying to parse (imperfectly) natural
languages using context free grammars.

Leaving natural languages for a moment (which I haven't tried to write
any computational parsers for), denumerably infinite number of
productions is not a hindrance in itself. The Algol68 specification
uses that. The infinite number of productions are generated through
using a context free grammar to generate the productions of the actual
grammar, and embodies "semi-semantic" things such as declaration
checking and type checking, including user defined types.

But all of this is VERY tangential to what we are doing in LTRU.

[Skipping BASIC, a programming language I gave up on 30 years ago ;-)]

> To call that a wart is to give it far too much prominence.

Well, then one would be using two grammars, one official, and one
unofficial to get the parse tree right. And while you're happy with
just a set of strings, I would like to have correct parse trees
for them as well. And I see no reason why one would need to use an
unofficial grammar to get the latter, when it is easy to fix the
official grammar to get the proper parse trees.

	/kent k

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Jun 2008 15:09:04 -0400

Kent Karlsson scripsit:

> The Algol68 specification uses that. The infinite number of productions
> are generated through using a context free grammar to generate
> the productions of the actual grammar, and embodies "semi-semantic"
> things such as declaration checking and type checking, including user
> defined types.

Indeed, the Revised Report on Algol 68 encodes the whole of the
operational semantics using the same mechanism: there is actually an
*interpreter* for A68R in the two-level grammar.  But Turing-complete
grammars are worthless for descriptive purposes, for they can accept any
language however gnarly, and therefore tell us nothing about possible
underlying mechanisms, the goal of Chomsky-style linguistics.

> But all of this is VERY tangential to what we are doing in LTRU.

Indeed, and I'll say no more.

> official grammar to get the proper parse trees.

Just adding a top-level production enumerating the grandfathered tags
will *not* give you proper parse trees.  If you can produce such a
grammar *without* W-grammar-style formalism, I'd be happy to take a look.

-- 
John Cowan   [email-masked]  http://www.ccil.org/~cowan
Most languages are dramatically underdescribed, and at least one is
dramatically overdescribed.  Still other languages are simultaneously
overdescribed and underdescribed.  Welsh pertains to the third category.
        --Alan King
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Jun 2008 12:15:30 -0700

> they mean; a grandfathered tag has no compositional meaning.

Even so, it seems to me that every subtag in the syntactic production must have a lexical source, and it's not clear to me that we have sanctioned a lexical source in these cases.

Peter

---

From: "Kent Karlsson" <[email-masked]>
Date: Fri, 13 Jun 2008 21:33:27 +0200

> grammar *without* W-grammar-style formalism, I'd be happy to 

See the proposal I submitted yesterday. Together with very common
disambiguation rules, often implemented in lexer and parser generating
tools (often used for implementing parsers for programming languages,
and similar formal languages) it is easy to get the proper (abstract)
parse trees. The grammar needs some superficial grammar-syntax changes
to get it into the form accepted by those tools, but will need NO
changes based on registry data if the proposal I submitted is accepted.

I do not propose to use two-level grammars, of course, that would be
a huge leap.

	/kent k

---

From: "Kent Karlsson" <[email-masked]>
Date: Fri, 13 Jun 2008 21:45:16 +0200

Peter Constable wrote:
> Even so, it seems to me that every subtag in the syntactic 
> production must have a lexical source, and it's not clear to 
> me that we have sanctioned a lexical source in these cases.

See the ABNF proposal I submitted yesterday. There "hakka"
and a few other strings that **look like** extlangs or variants
but are not are all disallowed from being parsed as extlangs
or variants, and are handled in separate productions for
"falselyregular" Language-Tags.

	/kent k