Subject: [Ltru] non-terminal names

From: "Kent Karlsson" <[email-masked]>
Date: Wed, 8 Feb 2006 11:43:54 +0100 (CET)

I still do not like that two *closely related* (draft) RFCs use the
*same* names for non-terminals that have *different* productions
(or more precisely: produce different formal languages). Thus I
would still like to see that some non-terminals in section 2 got
their names changed.

I'm not very keen on the exact names or name differences, as long
as the names are sufficiently mnemonic. It still would be nice if
the names for related productions had a systematic difference.
E.g. "langtag" in -registry and "langtagpat" in -matching, "extlang"
in -registry and "extlangpat" in -matching. Non-terminals that
produce the same formal language should of course not have
their names changed (e.g. "alphanum" and "privateuse").

A further elaboration would be to keep the productions from
-registry, and just add some productions, e.g.

regionpat     = region
              / "*"                    ; or wildcard

		/kent k

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 09 Feb 2006 04:12:11 +0100

Kent Karlsson wrote:

> -registry, and just add some productions

Please propose the complete ABNF, ready to be mixed with the
registry ABNF, and passing Bill's validator:  USEFOR consumed
all my energy to tweak ABNF.  With a proposal where the last
issue are nice names for the productions it should be possible
to come to a decision.
                         Bye, Frank

---

From: Martin Duerst <[email-masked]>
Date: Thu, 09 Feb 2006 13:54:41 +0900

This in principle looks like a very good idea.   Regards,   Martin.

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 9 Feb 2006 12:01:32 +0100 (CET)

Frank Ellermann wrote:
> registry ABNF, and passing Bill's validator

Ok. For brevity I will not quote the registry ABNF here, but there
is no change for it at all (maybe not ideal, but it cannot be
changed at this stage). (You did write "mixed with"!) The two
together passes Bill's validator.

I think that the current (v. -09) -matching production

extension = singleton *("-" (2*8alphanum)) [ "-*" ]

is wrong. It allows a singleton *alone* (no dash and second
component at all), which I don't think was intended. See the
last productions below for a correction.

Here are the *additional* productions for (extended) language
tag patterns/ranges (to be mixed with the -registry ABNF):

------------------------------------------------

Language-Tag-pat
              = langtag-pat
              / privateuse       ; private-use tag
              / grandfathered    ; grandfathered registrations

langtag-pat   = (language-pat
                 ["-" script-pat]
                 ["-" region-pat]
                 *("-" variant-pat)
                 *("-" extension-pat)
                 ["-" privateuse])

language-pat  = language
              / (2*3ALPHA extlang-pat)
              / "*"                    ; or wildcard

extlang-pat   = *2("-" 3ALPHA) "-*"    ; reserved for future use
                                       ; wildcard can only appear
                                       ;   at the end

script-pat    = script
              / "*"                    ; or wildcard

region-pat    = region
              / "*"                    ; or wildcard

variant-pat   = variant
              / "*"                    ; or wildcard

extension-pat = singleton 1*("-" (2*8alphanum))
              / singleton *("-" (2*8alphanum)) "-*"
                                       ; extension subtags
                                       ; wildcard can only appear
                                       ;   at the end

-----------------------------------------

		/kent k

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 9 Feb 2006 10:34:04 -0800

I dislike the basic assertion that we should use different non-terminals. I
think it is *more* confusing to have a production for a tag and then, when
choosing my range to select that tag, a differently named sub-entity. Adding
"-pat" doesn't add anything in my opinion.

When selecting an extended language range to do some matching operation, I
think it is useful to know that the "language" production selects the
"language" part of a language tag and the "script" production select the
script part (und so weite). Admittedly, this is a preferential thing, and it
seems I'm in the minority on this.

> last productions below for a correction.

You have a good point. More later.

>                                        ; wildcard can only appear

Nope. This production would require a trailing -* and disallows the
(remotely possible) 3-subtag extlang sequence. The current (more correct
IMO) production is:

extlang       = *2("-" 3ALPHA) ("-" ( 3ALPHA / "*"))
                                       ; reserved for future use
                                       ; wildcard can only appear
                                       ;   at the end

As noted in previous threads, there is a bit of subtlety (which may be lost
on readers) here. If you choose to include an extlang sequence at all, it
will contain at least one subtag (... or you will have omitted the extlang
sequence). The back half of the production covers this. The front half of
the production (*2 etc.) covers any additional subtags as needed.

>               / singleton *("-" (2*8alphanum)) "-*"

You're correct about the meaning of the current ABNF, but I think we need to
be quite as redundant. I would suggest:

   extension = singleton *("-" ext-subtag) ("-" (ext-subtag / "*"))
   ext-subtag = 2*8alphanum

This uses the same subtlety as the extlang, allowing us to explain it in the
text only once and keeping the number of different issues to a minimum. In
particular, I would suggest adding this text immediately following the ABNF:

---
Some subtag types, such as extlang, variant, privateuse, and extension,
allow more than one subtag when forming a language tag. Extended language
ranges permit the wildcard subtag "-*" only at the end of such a sequence in
order to simplify matching implementations. Thus, if "EXA" and "EXB" were
extended language subtags, the range "qx-EXA-EXB-*" would be legal, while
the range "qx-EXA-*-EXB" would not be. This is a subtlety in the ABNF which
might not be immediately obvious.
---

Addison

Addison Phillips
Internationalization Architect - Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 09 Feb 2006 20:09:58 +0100

Kent Karlsson wrote:

> For brevity I will not quote the registry ABNF here

Okay, I have that, added your productions, and it's valid.
Now we have to rename various terms in the matching draft:

2.1 has its own idea of <language-tag>, but that won't fly
with your concept, a <language-tag> is already defined in
3066bis.  We could just s/language/basic/ in 2.1:

   basic-range    = basic-tag / "*"
   basic-tag      = 1*8[alphanum] *["-" 1*8alphanum]

That's for basic 3066-compatible stuff, almost the same as
in 2616.  They had no DIGIT in 2616, but DIGIT is listed in
the 2616-errata at least for <Subtag>:

<http://purl.org/NET/http-errata>

Maybe we could just say "updates 2616" and be done with it ?
Do we really want to mess with the 2616 and 3066 concept of
a <Primary-subtag> ?  IMHO we can keep it as it always was:

   basic-range    = basic-tag / "*"
   basic-tag      = 1*8ALPHA *( "-" 1*8alphanum )

Besides the old 2.1 ABNF in the matching draft -09 is wrong:
                       v        v
   language-tag   = 1*8[alphanum] *["-" 1*8alphanum]

The <Primary-subtag> isn't optional... ;-)

Now the fun stuff, <extended-language-range> in chapter 2.2.

Apparently we could get away with adding "-range" everywhere,
I'd like that better than your "-pat" everywhere.

Something's odd in the -09 ABNF and in your derived ABNF:

Do we want "zero or more" <variant-range>s ?  What's the idea
of more than one wildcard variant like say de-CH-*-*-x-oops ?
No problem with de-CH-1996-*-x-okay, but de-CH-*-1996-x-weird
is weird, or isn't it ?

> component at all), which I don't think was intended.

Yes, your version is better.  Adding the 4234-recommended
"grouping" for alternatives after s/pat/range/ I get:

   extension-range = extension
                   / ( singleton *( "-" 2*8alphanum ) "-*" )

It's not completely obvious to find the <extlang> in your ABNF,
therefore I propose to integrate your <extlang-pat> into the
<language-pat> or <language-ranges> resp.:

   language-range
                 = language / "*"
                 / (2*3ALPHA *2("-" 3ALPHA) "-*")

Putting it all together I get what you see below, ignoring the
question of "two or more wildcard variants" for the moment.

                         Bye, Frank

basic-range   = basic-tag / "*"
basic-tag     = 1*8ALPHA *( "-" 1*8alphanum )

extended-language-range
              = langtag-range

langtag-range = (language-range
                 ["-" script-range]
                 ["-" region-range]
                 *("-" variant-range)
                 *("-" extension-range)
                 ["-" privateuse])

language-range = language / "*"
              / (2*3ALPHA *2("-" 3ALPHA) "-*")

script-range  = script  / "*"
region-range  = region  / "*"
variant-range = variant / "*"

extension-range
              = extension
              / ( singleton *( "-" 2*8alphanum ) "-*" )

; --- copied from 3066bis as is ---

Language-Tag  = langtag
              / privateuse             ; private use tag
              / grandfathered          ; grandfathered registrations

langtag       = (language
                 ["-" script]
                 ["-" region]
                 *("-" variant)
                 *("-" extension)
                 ["-" privateuse])

language      = (2*3ALPHA [ extlang ]) ; shortest ISO 639 code
              / 4ALPHA                 ; reserved for future use
              / 5*8ALPHA               ; registered language subtag

extlang       = *3("-" 3ALPHA)         ; reserved for future use

script        = 4ALPHA                 ; ISO 15924 code

region        = 2ALPHA                 ; ISO 3166 code
              / 3DIGIT                 ; UN M.49 code

variant       = 5*8alphanum            ; registered variants
              / (DIGIT 3alphanum)

extension     = singleton 1*("-" (2*8alphanum))

singleton     = %x41-57 / %x59-5A / %x61-77 / %x79-7A / DIGIT
              ; "a"-"w" / "y"-"z" / "A"-"W" / "Y"-"Z" / "0"-"9"
              ; Single letters: x/X is reserved for private use

privateuse    = ("x"/"X") 1*("-" (1*8alphanum))

grandfathered = 1*3ALPHA 1*2("-" (2*8alphanum))
                ; grandfathered registration
                ; Note: i is the only singleton
                ; that starts a grandfathered tag

alphanum      = (ALPHA / DIGIT)       ; letters and numbers

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 09 Feb 2006 20:55:23 +0100

Addison Phillips wrote:

> I think it is *more* confusing to have a production for a
> tag and then, when choosing my range to select that tag, a
> differently named sub-entity.

The idea to recycle what we have in 3066bis is IMO attractive.
Even if we don't do it the attempt helped to find two bugs ;-)

> it seems I'm in the minority on this.

I'm still rather neutral about it, "import 3066bis ABNF as is"
is nice, but "offer the complete ABNF in RFCTBD" is also fine.

>> extlang-pat   = *2("-" 3ALPHA) "-*"    ; reserved for future
[...]

> Nope. This production would require a trailing -* and
> disallows the (remotely possible) 3-subtag extlang sequence.

Yes, I also got it wrong at the first glance.  But he has the
"<extlang> without wildcard" hidden in the 3066 <language>.

Therefore all he needed was "<extlang-range> with wildcard" as
shown.  I proposed to get rid of the separate production, and
to integrate the wildcard case directly into <language-range>.

> there is a bit of subtlety (which may be lost on readers)
> here. If you choose to include an extlang sequence at all,
> it will contain at least one subtag (... or you will have
> omitted the extlang sequence).

Sounds like a good idea, but I don't see it in your syntax,
you have  extlang = *2("-" 3ALPHA) ("-" ( 3ALPHA / "*"))

That can degenerate into "-*", and then it's not more clear
what this "-*" is, <extlang-range>, <script-range>, etc.

The "-*" is only unambiguous if you have an explicit <script>,
excl. the wilcard case of <script-range>.  Is de-*-1996 the
same as de-1996, and is de-*-*-*-1996 still the same ?

With the ABNF as is (-09 or Kent's) even de-*-*-*-*-1996 is
possible, and I could claim that this is for four wildcard
variants before 1996, preserving the implicit Suppress-Script
Latn for de.  That's messy, isn't it ?

> This uses the same subtlety as the extlang

Here it's clear, the singleton forces us into <extlang-range>.
IMO unnecessary to extract <ext-subtag>, and Kent's idea...

  extension-range
              = extension

...after s/pat/range/ and recycling the simple "<extension>
without wildcard" case of 3066bis should be also good enough.

>| Extended language ranges permit the wildcard subtag "-*"
>| only at the end of such a sequence in order to simplify
>| matching implementations. Thus, if "EXA" and "EXB" were
>| extended language subtags, the range "qx-EXA-EXB-*" would
>| be legal, while the range "qx-EXA-*-EXB" would not be.
>| This is a subtlety in the ABNF which might not be
>| immediately obvious.

Yes, minus the last sentence, because it is obvious... ;-)

Can we use the same rule to kill more than one <variant-range>
with wildcards ?  Neither your nor Kent's ABNF does this at
the moment.
                          Bye, Frank

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 9 Feb 2006 12:59:40 -0800

Hi Frank,

Notes follow.

Addison

Addison Phillips

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Frank Ellermann
> Sent: Thursday, February 09, 2006 11:55 AM
> To: [email-masked]
> Subject: [Ltru] Re: non-terminal names
> 
> Addison Phillips wrote:
> 
> 

I think that might be a good idea, except for the fact that it obscures
extlang and potentially the rules associated with its use.

> what this "-*" is, <extlang-range>, <script-range>, etc.

It isn't clear, but the matching rules later make it clear that it covers
both cases. "qx-*" matches both "qx-EXA" and "qx-Latn" (as well as "qx-DE"
and so on). It is still a strict prefix matching scheme.

The embedded case is the question mark. Does "qx-*-DE" match
"qx-EXA-Latn-DE"? I believe that it does.
> same as de-1996, and is de-*-*-*-1996 still the same ?

The document currently says in the following paragraph:

--
Implementations that normalize extended language ranges SHOULD expand
missing fields to be "*" so that the semantic meaning of the language range
is clear to the user. At the same time, multiple wildcards in a row are
redundant and implementations SHOULD collapse these to a single wildcard
when normalizing the range (for brevity). For example, both the range
"sl-nedis" and the range "sl-*-*-nedis" are equivalent to and should be
normalized as "sl-*-nedis".
--

I believe this makes it quite clear that "de-*-1996" is the same as
"de-1996" and "de-*-*-*-1996".
> Latn for de.  That's messy, isn't it ?

The normalization scheme would smoosh all the redundant stars. The only
question is whether this is intentionally the right thing for extlang. In
Scored Filtering we go the opposite direction, claiming that extlang is
inherently part of the language. 
> without wildcard" case of 3066bis should be also good enough.

No, because you have to be able to specify a item exactly with NO wildcard.
This formula requires the "-*" to appear.
> with wildcards ?  Neither your nor Kent's ABNF does this at

That's a good point.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 09 Feb 2006 20:52:13 +0100

--On 9. februar 2006 10:34 -0800 Addison Phillips <[email-masked]> 
wrote:

> I dislike the basic assertion that we should use different non-terminals.
> I think it is *more* confusing to have a production for a tag and then,
> when choosing my range to select that tag, a differently named
> sub-entity. Adding "-pat" doesn't add anything in my opinion.

I prefer the "-pat" pattern; using one production for "the thing" and 
another production for "either the thing or a wildcard" seems like a clean 
way to name things for me.

                 Harald

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 09 Feb 2006 23:54:37 +0100

> Internationalization is an architecture.

Above all it's messy... :-)  The -09 announcement finally made
it:  <http://permalink.gmane.org/gmane.ietf.announce/11054>

>> I proposed to get rid of the separate production, and to
>> integrate the wildcard case directly into <language-range>.

> I think that might be a good idea, except for the fact that
> it obscures extlang and potentially the rules associated with
> its use.

Yes.  What confused us was Ken't use of *-pat everywhere with
the meaning "with or without wildcard", but for <extlang-pat>
it was "only with with wildcard".

For a clean solution we'd need a different suffix like *-wild:

| language-range = language / "*" / extlang-wild
| extlang-wild   = 2*3ALPHA *2("-" 3ALPHA) "-*"

Then any *-range or *-pat is always with or without wildcard,
and <extlang-wild> is a different beast.

 [ambiguous "-*" cases]
> It isn't clear, but the matching rules later make it clear
> that it covers both cases. "qx-*" matches both "qx-EXA" and
> "qx-Latn" (as well as "qx-DE" and so on). It is still a
> strict prefix matching scheme.

> "qx-EXA-Latn-DE"? I believe that it does.

Okay.  In other words an explicit or implicit "-*" for a script
triggers the same Suppress-Script logic (if applicable).

And qx-*-*-*-DE is a syntax error, as far as any implementation
will care to count stars.

>| Implementations that normalize extended language ranges
>| SHOULD expand missing fields to be "*" so that the semantic
>| meaning of the language range is clear to the user.

>| At the same time, multiple wildcards in a row are redundant
>| and implementations SHOULD collapse these to a single
>| wildcard when normalizing the range (for brevity).

>| For example, both the range "sl-nedis" and the range
>| "sl-*-*-nedis" are equivalent to and should be normalized
>| as "sl-*-nedis".

Is the first SHOULD supposed to add a single trailing star as
in "sl" => "sl-*" ?  The second SHOULD confuses me, apparently
_all_ wildcards are redundant.

Just one star in any free position would be enough if you need
a syntactical distinction between a "plain tag" and a "range".

Thinking loud:  If all I want is to make clear that I want a
range, I could simply take any tag and add a trailing star.

Is that correct, or are there cases where a trailing star won't
do what I want ?  If it's correct, why do we bother with stars
in other positions ?  One trailing star would be always enough.

> "de-*-1996" is the same as "de-1996" and "de-*-*-*-1996".

End of thinking loud:  Is that also the same as de-1996-* ?
Could we even get away with no stars at all as in de-1996 ?

I don't like syntactical contructs without clear purpose, and
the stars are apparently redundant.  Catching qx-*-*-*-DE as
a syntax error doesn't justify the effort to introduce stars.

> No, because you have to be able to specify a item exactly
> with NO wildcard.  This formula requires the "-*" to appear.

The first alternative <extension> is for the case "no star" -
that ABNF was for Kent's idea, where <extension> is what we
have in 3066bis, it's not your matching draft -09 <extension>.

  [subtlety]
>> Can we use the same rule to kill more than one
>> <variant-range> with wildcards ?  Neither your nor Kent's
>> ABNF does this at the moment.

> That's a good point.

Maybe killing all stars except from the "basic Texas case" is
the most simple solution.  Otherwise it could be something like

                 *("-" variant) [ "-*" ]
                 *("-" extension-range)
                 ["-" privateuse])
Bye, Frank

---

From: Martin Duerst <[email-masked]>
Date: Fri, 10 Feb 2006 12:02:26 +0900

At 04:52 06/02/10, Harald Tveit Alvestrand wrote:
 >
 >
 >--On 9. februar 2006 10:34 -0800 Addison Phillips <[email-masked]> wrote:
 >

I have to say that I agree with Harald (and Kent) here.

Others, please state your preferences so that we can move
forward on this point.

Regards,     Martin.

---

From: John Cowan <[email-masked]>
Date: Thu, 9 Feb 2006 22:49:49 -0500

> I have to say that I agree with Harald (and Kent) here.

+1

-- 
I suggest you call for help,                    John Cowan
or learn the difficult art of mud-breathing.    [email-masked]
        --Great-Souled Sam                      http://www.ccil.org/~cowan

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 9 Feb 2006 20:22:39 -0800

Hi -

> From: "Martin Duerst" <[email-masked]>
...
> Sent: Thursday, February 09, 2006 7:02 PM
> Subject: RE: [Ltru] Re: non-terminal names
>
...
> I have to say that I agree with Harald (and Kent) here.

As a technical contributor, I do, too, although I don't think it actually
helps the document very much.

> Others, please state your preferences so that we can move
...

Randy

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 10 Feb 2006 05:16:23 +0100

Martin Duerst wrote:

> please state your preferences so that we can move forward on
> this point.

Thinking about the issue of the stars for some time I'd say we
can radically simplify this.  In chapter 2.1:

| basic-range   = basic-tag / "*"
| basic-tag     = 1*8ALPHA *( "-" 1*8alphanum )

In chapter 2.2:

| language-range = langtag-range
|                / privateuse
|                / grandfathered

| langtag-range  = (( language / "*" )
|                   ["-" script]
|                   ["-" region]
|                   *("-" variant)
|                   *("-" extension)
|                   ["-" privateuse])

We only need the cases "lone star" and "leading star", other
stars are always redundant / implicit:

qx-oops must be script oops, any region, any variant, any
extension, (and later any extlang).

qx-foobar must be any script, any region, almost any variant
as long as it includes 'foobar', any extension, (and later
any extlang).

qx is simply any anthing starting with qx.

-qx isn't possible, that's why we need *-qx for any language
etc. in region qx.

-oops is also impossible, that's why we need *-oops for any
language etc. with script oops.

*-oops-qx is also clear, a "leading star" is all we need, in
addition to the known "lone star" defined in 3066 / 2.1 / 2.2

                            Bye, Frank

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 9 Feb 2006 21:04:38 -0800

I tend to agree with Frank's note below. The main additional concerns I have
are:

1. Originally, when I wrote up the extended grammar, I was concerned with
the difference between "de" and "de-*" in lookup. Extended lookup required
extra stars to make it work. Filtering, though, doesn't need the extra
stars, except for...

2. Good grammars tend to be explicit rather than implicit. "de-*-1996" is
clearer than "de-1996". On some level, I might tend to expect that "de-1996"
matches "de-1996-fubar" but not "de-Oops-1996", while "de-*-1996" might
match both (because it is explicit?).

3. Or course, I said something different in my previous email on this
thread.

4. Filtering still has a problem. I note this text:

--
Missing components in the language range are handled similarly to extended
range lookup: missing internal subtags are expanded to "*". Missing end
subtags are expanded as the empty string. Thus a pattern "en-US" becomes the
quintuple ("en","*","US","","").
--

The trailing star form fixes the trailing items if that is what the user
wants.

Addison

Addison Phillips

> Sent: Thursday, February 09, 2006 8:16 PM
> To: [email-masked]

---

From: r&d afrac <[email-masked]>
Date: Fri, 10 Feb 2006 10:50:01 +0100

At 04:02 10/02/2006, Martin Duerst wrote:
>Others, please state your preferences so that we can move forward on 
>this point.

I do not oppose the proposition. But the implementation is not 
consistent with other terminologies. I am sorry, I have not the time 
to  review the whole thing at this time. But may be my line of 
thinking may help. We are at concept level here.

This effort is attached to the RFC 3066 Bis ABNF. I am interested in 
RFC 3066 Bis to find how to contain the various risks it represents, 
and make it interoperable with other language code systems. Since its 
sponsoring and probable usage will make it to stay, the first target 
was to get a cleaner text. Now my interest in the filtering effort, 
it is to see if could help, how it increases the risks and how it 
impacts the privateuse area.

For this we really need the final document, to understand it, to 
simulate it. But everything related to a better analysis is good, 
except if it unecessary (repeats something we already have: only 
cases study can tell this). My own filtering of the filtering 
proposition will be if and how it matches with our thinking structure 
(ISO 11179 based - itself under review now).

However I can say I have a basic difficulty which is to understad the 
purpose of the intended filtering (solution or protocol - matching or 
profiling) and therefore the underlaying philosophy and needed 
levels/areas of accuracy. Frank just gave a good analysis, which 
could make a good appendix. May be a use case appendix could also 
help better analysing the real needs and risks of confusion?
jfc

---

From: Martin Duerst <[email-masked]>
Date: Fri, 10 Feb 2006 16:10:44 +0900

>helps the document very much.

Hello Randy,

I agree that there are other issues in the document that are more
difficult. Also, I have to admit that I didn't get around to do
much chairing in the last few months. That will change quite a bit
in the next weeks, the term here at the University (and the very
tought final-term/graduation,... work) is finally comming to an
end.

Regards,     Martin.