Subject: Re: [Ltru] Macrolanguage usage

From: "Doug Ewell" <[email-masked]>
Date: Sat, 24 May 2008 12:06:26 -0600

(Note 1: This message requests WG chair action.)
(Note 2: Nothing in this message is meant to express an opinion for or 
against extlang per se.)

Leif Halvard Silli <lhs at malform dot no> wrote:

> Doesn't Cantonese users understand Mandarin?

From what I've heard, in terms of spoken language: absolutely not.

> As you know, many tag Norwegian texts as 'no-no'.

Of course, because that means "Norwegian as used in Norway."  (Which is 
kind of redundant really, since there doesn't seem to be much evidence 
of variation in the Norwegian language associated with regions other 
than Norway; but some creators of language tags and locale identifiers 
feel it is important to apply region subtags consistently.)

> So it is obviously that when 'no-no' falls back to 'no', then of 
> course 'no-nn' or 'no-nno' would fall back just as well. Why should I 
> not believe so?

I ask the co-chairs to settle this matter with a third consensus-call 
question:

Q3: If we did go back to using "extlang," we could combine this subtag
    with the region subtag, and require that at most one of the two be
    used in a single tag.  Possible responses:  (pick ONE)
        A - I would like this.
        B - I could live with this.
        C - I would object to this.

Remember that we did create such a "Leif rule" for the purpose of 
allowing two-letter extlangs, as in "no-nn", then:

1. The region/extlang subtag would have to come AFTER any script 
subtags, thus: "no-Latn-nn" rather than "no-nn-Latn", and "zh-Hant-cmn" 
rather than "zh-cmn-Hant" -- unless we wanted to change that existing 
BCP 47 syntactical rule as well.

2. It would be impossible to tell whether a non-initial two-letter 
subtag such as 'tw' referred to a region, as in "zh-TW" (Taiwan), or an 
extlang, as in "ak-tw" (Twi).  Case is not significant in language 
tags -- unless we wanted to change that existing BCP 47 syntactical rule 
as well.

3. It would be impossible to write a tag for, say, "Cantonese as used in 
Singapore" that also expressed the macrolanguage relationship --  
whatever that may be -- between 'zh' and 'yue'.

> In the draft you sent out you start by saying that "The arguments for 
> extlang are that they give superior results,". However, this is an 
> exaggeration of the standpoint that I for intance have. First, I 
> assume you meant "technical superior". Well, no, I can understand that 
> using short tags is easier to deal with, technically. And therefore 
> superior to extlang. (In my testing with Apache, 'nn' and 'nb' was 
> easier to deal with than 'no-nyn' and 'no-bok'.)
>
> But then a problem is that the users "in the wild" still are tagging 
> Norwegian as if we had an extlang system.

"no-nyn" and "no-bok" are grandfathered tags, registered under RFC 1766 
in 1995, long before anyone ever used the word "macrolanguage" or 
"extlang".  Their similarity to extlangs is to be considered 
coincidental.

> And this is a special kind of language negotiation. For a small 
> macrolanguage like NOrwegian, we suddenly get 3 options. If instead we 
> had extlang for Norwegian, we would in reality only have two options.

This isn't new or sudden.  You've had 3 options since 2000, when ISO 639 
registered 'nb' and 'nn', and actually since 1995, when ietf-languages 
registered the whole tags "no-bok" and "no-nyn" which were not to be 
considered parsable.

> I have allready been told that it is very important to read things out 
> of the tags without needing to look into the registry. And I agree. 
> That is a basic, and very good thing.

See my note 2 above.  Regions and encompassed languages are not at all 
the same, and the "Leif rule" would require tag producers and consumers 
to look in the Registry to see which is intended.  (If Mark can say "a 
la Ewell" then I can say "the Leif rule.")

--
Doug Ewell  *  Arvada, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 24 May 2008 12:35:04 -0600

Gerard Meijssen <gerard dot meijssen at gmail dot com> wrote:

> Hoi,
> The current situation is that the biggest amount of content on the 
> Internet is not tagged. Languages that are "less and least resourced" 
> can not be found because standards and software both do not 
> acknowledge them.

The problem here is the continued unavailability to the public of RFC 
4646bis, due to the surreal delay that has been introduced in this 
second round of LTRU.  We were originally supposed to go to IETF Last 
Call in early 2006, if I remember correctly.

This isn't even about the protracted debate over extlang, which at least 
is fundamental to the update and mentioned prominently in the WG 
charter.  This is about all the other changes -- enhancing validity 
checking, tweaking the ABNF, obsessing over "und" and "mis" and "zxx", 
haggling over "EU" and "UK" -- that are not mentioned in the charter at 
all, or are loosely alluded to but were "not expected to delay the 
progress of the work."  It's also about drafts being published and 
sitting for months without substantive comment from the group.

I share Gerard's concern about the inability to tag minority languages 
(without resorting to private-use tags).  When RFC 4646bis and the new 
Registry are published, it will be possible to tag virtually anything, 
even if the representation or hierarchy or fallback isn't perfect.  But 
I have often wondered if the word isn't "when," but "if."

Please read the current drafts and send your specific comments (or your 
LGTM's) to the list, as well as your responses to Randy's consensus-call 
questions, so we can get these documents out of the WG and start IETF 
Last Call -- which I guarantee will not be a rubber-stamp process 
either.

http://www.ietf.org/internet-drafts/draft-ietf-ltru-4646bis-14.txt
Alternative format available at:
http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-14.html

http://www.ietf.org/internet-drafts/draft-ietf-ltru-4645bis-05.txt
Alternative formats available at:
http://www.ewellic.org/rfc4645bis.html

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 24 May 2008 11:49:16 -0700

Hi -

As co-chair...

> From: "Doug Ewell" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
> Cc: "Randy Presuhn" <[email-masked]>; "Martin Duerst" <[email-masked]>
> Sent: Saturday, May 24, 2008 11:06 AM
> Subject: Re: Macrolanguage usage
...
>         C - I would object to this.
...

So far, I have not seen enough support for such an approach on this
mailing list to merit a consensus call, particularly since technical
opposition has already been voiced.  Until / unless significant support
appears, I'm simply considering the issue "rejected."

Randy

---

From: Peter Constable <[email-masked]>
Date: Sat, 24 May 2008 17:56:11 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Doug Ewell

> allowing two-letter extlangs, as in "no-nn", then:

If the question is whether to allow tags of the form "no-nn", then my answer is

CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCc

I would strongly object.

Peter

---

From: Leif Halvard Silli <[email-masked]>
Date: Sun, 25 May 2008 14:04:56 +0200

Doug Ewell 2008-05-24 20.06:
> kind of redundant really, [...]

It is redundant. But it is still done. And not only on the Web. 
Yesterday I was looking at the hyphenation dictionaries of OpenOffice. 
All of them are identified with a language tag made of two subtags. 
Apparently the double subtags helps users/developers understand better 
what is meant. It gives more context.

So for Nynorsk, they used 'nn_NO'. For Bokmål 'nb_NO'. Which is the same 
level of redundancy as 'nn-NO'. I assume that adding the _NO helps user 
to understand what 'nn' and 'nb' means. Which is kind of backwards. One 
must read it from the right in order to fully understand what it means.

>  since there doesn't seem to be much evidence 

To follow the same pattern throughout, you mean. Could be. But I guess 
this pattern arises in the first place because one knows about the need 
to discern between e.g. en-US and en-GB. To use the region tag *only* 
when needed would be too much hazzle, I guess ...

(The irony, in the OpenOffice case, is that the nn_NO and nb_NO hyph 
dictionaries are *identical*. As is, btw, the en_US, the en_GB and the 
en_CA hyph dictionaries.)

> BCP 47 syntactical rule as well.

But you would still be able to omit the script tag. However, this does 
seem logical, to me.

> tags -- unless we wanted to change that existing BCP 47 syntactical rule 

But since the most important tag is suppose to be the first one, this 
does not seem to be much of an issue. Unless we must concider the 
possibility of a mass exchange of language/population between Ghana/Côte 
d'Ivoire and Taiwan.

> whatever that may be -- between 'zh' and 'yue'.

Yes, one would have to choose between 'yue-sg' and 'zh-yue'. The same 
would go for Mandarin in China. Either 'cmn-cn' or 'zh-cmn'.

> "extlang".  Their similarity to extlangs is to be considered 

Agree - coincidental. But only because 'bok' and 'nyn', by coincident, 
was not registered as the official codes for Nynorsk and Bokmål in 
ISO-639-2.

> This isn't new or sudden.

Pardon me for using colorful language (the word "suddenly").

>   You've had 3 options since 2000, when ISO 639 

For that matter, 'no-nn', 'no-nb' and 'no' would also be 3 options. That 
is not what I meant.

> la Ewell" then I can say "the Leif rule.")

I read this with a sense of humor, so that is ok. :-)
-- 
leif halvard silli

---

From: Leif Halvard Silli <[email-masked]>
Date: Sun, 25 May 2008 15:51:41 +0200

> But you would still be able to omit the script tag. However, this does 

It seems logical to write script before region. Just consider 
Serbo-Croat-Bosnian-Montenegrin. It did not take me long to find 
Serb pages where one discerned between Serbian Latin and Serbian 
Cyrillic through the use of tags for Croatian versus Serbian. This 
  tells me that Script is more important than Region.

Other places, for instance different translated resources at 
www.w3.org, there seems to be tendency to use Serbian as 
"macrolanguage" since for Serbian, it is possible to switch 
between Latin and Cyrillic. (For instance see the translations 
offered at www.w3.org/Style .) I.e. there seems to be little need 
for the other encompassed languages as long as Serbian is covered 
in both scripts. It is not customary to list Region if you list 
Script. The same W3 page also offers translations into Chinese, 
where only the Script is mentioned: 'zh-hant'.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 25 May 2008 15:58:14 -0600

Leif Halvard Silli <lhs at malform dot no> wrote:

>> Of course, because that means "Norwegian as used in Norway."  (Which 
>> is kind of redundant really, [...]
>
> So for Nynorsk, they used 'nn_NO'. For Bokmål 'nb_NO'. Which is the 
> same level of redundancy as 'nn-NO'. I assume that adding the _NO 
> helps user to understand what 'nn' and 'nb' means. Which is kind of 
> backwards. One must read it from the right in order to fully 
> understand what it means.

The language tags you are citing do NOT use 'NO' to mean "Norwegian," as 
you imply, but rather "Norway."  This is correct usage of language and 
region subtags, though as I said earlier, it is somewhat redundant to 
say "(any kind of) Norwegian *as used in Norway*," since as far as I can 
tell there is no relevant difference between Norwegian-in-Norway and 
Norwegian-somewhere-else.  But this is not at all necessarily redundant 
for other languages.

You are continuing to argue that people are using "nn-NO" and "nb-NO" as 
a sort of reverse-extlang syntax (encompassed language followed by 
macrolanguage), and I am continuing to tell you that is NOT what is 
going on here.  It is sheer coincidence that the language code (hence 
language subtag) for Norwegian, and the country code (hence region 
subtag) for Norway, happen to both be 'NO'.  It is not always the case 
for languages and countries that are closely associated.  For example, 
the language subtag for Swedish is 'sv' but the region subtag for Sweden 
is 'SE'.  If you were talking about Swedish instead of Norwegian, I 
doubt you would have headed down this path.

>> but some creators of language tags and locale identifiers feel it is 
>> important to apply region subtags consistently.)
>
> this pattern arises in the first place because one knows about the 
> need to discern between e.g. en-US and en-GB. To use the region tag 
> *only* when needed would be too much hazzle, I guess ...

Why would it be a hassle?

> en_CA hyph dictionaries.)

Exactly.  There is no difference that I'm aware of between hyphenation 
rules in any variety of English.  100 or 200 years ago this may not have 
been the case.  I don't know whether hyphenation rules differ between 
Bokmål and Nynorsk, but I suspect they do not.  I do know they differ 
between de-1901 and de-1996, which is one reason these tags were 
registered back in 2001.

>> subtags, thus: "no-Latn-nn" rather than "no-nn-Latn", and 
>> "zh-Hant-cmn" rather than "zh-cmn-Hant" -- unless we wanted to change 
>> that existing BCP 47 syntactical rule as well.
>

To have the region/extlang come after the script?  Fine, but remember 
this is not compatible with the model established by registered tags 
like "zh-cmn-Hant".

>> subtag such as 'tw' referred to a region, as in "zh-TW" (Taiwan), or 
>> an extlang, as in "ak-tw" (Twi).
>
> does not seem to be much of an issue.

This isn't the point.  You said earlier:

>>> I have allready been told that it is very important to read things 
>>> out of the tags without needing to look into the registry. And I 
>>> agree. That is a basic, and very good thing.

But if you have a combination region/extlang subtag, you MUST look into 
the Registry to find out that 'TW' in "zh-TW" refers to a region, and 
'tw' in "ak-tw" refers to an extlang.

What is more, an example like this proves that a combination 
region/extlang subtag would be structurally impossible.  We already have 
the following two-letter subtags that could be interpreted either way:

bs
    Language: Bosnian
    Region: Bahamas

hr
    Language: Croatian
    Region: Croatia

sr
    Language: Serbian
    Region: Suriname

tw
    Language: Twi
    Region: Taiwan

I'll concede that the semantic difference between 'hr' the language and 
'HR' the region is minimal, but the *syntactic* difference is important. 
A region is not a language.  And furthermore, you have no guarantee that 
ISO 3166 will not assign country codes 'NB' and 'NN', which would apply 
your proposal's syntactic ambiguity to Bokmål and Nynorsk as well. 
Given the number of countries whose name starts with the letter N, it 
would hardly come as a surprise if ISO 3166 did so.

> Unless we must concider the possibility of a mass exchange of 
> language/population between Ghana/Côte d'Ivoire and Taiwan.

I don't understand this.  Of course Akan is not widely spoken in Taiwan, 
and Chinese is not widely spoken in central Africa.  But the fact 
remains that under the your plan, you must examine the Registry to find 
out that 'tw' in 'ak-tw' is an extlang and 'TW' in 'zh-TW' is a region.

Under the existing rules, if you have a tag "xx-TW" you know that 'TW' 
is a region, without having to know what 'xx' stands for, or even what 
'TW' stands for.  This a valuable property, and it is why the syntactic 
rules are the way they are, and I would fight hard to prevent this 
property from being removed.

>> 3. It would be impossible to write a tag for, say, "Cantonese as used 
>> in Singapore" that also expressed the macrolanguage relationship --  

Don't you consider it a problem that it would be syntactically 
impossible to express "Cantonese, encompassed by Chinese" and "Cantonese 
as used in Singapore" in the same tag?  Wouldn't that take away a lot of 
the value of having extlangs in the first place?

>> "no-nyn" and "no-bok" are grandfathered tags, registered under RFC 
>> 1766 in 1995, long before anyone ever used the word "macrolanguage" 
>> or "extlang".  Their similarity to extlangs is to be considered 
>> coincidental.
>

And because official codes were registered in ISO 639-1, which means the 
Registry uses those instead of the ISO 639-2 codes.

>>> macrolanguage like NOrwegian, we suddenly get 3 options. If instead 
>>> we had extlang for Norwegian, we would in reality only have two 
>>> options.
>>

There's nothing wrong with the way you expressed it.  I like colorful 
language.  But there still isn't anything sudden about it.

Randy has already stated that he doesn't see any WG support for your 
proposal, and Kent Karlsson has joined me in being firmly against it. 
There is an important decision yet to be made by the WG -- to have 
extlangs or not -- and I'm afraid your continuing to push the "combined 
region/extlang subtag model" is simply clouding the issue.

---

From: Peter Constable <[email-masked]>
Date: Sun, 25 May 2008 14:59:24 -0700

> Leif Halvard Silli

> So for Nynorsk, they used 'nn_NO'. For Bokmål 'nb_NO'... Which is the
> same

These statements suggest that you don't really understand the basis for those tags at all! The suffix _NO is an ISO 3166-1 ID referring to Norway; take a lot at all the other tags: I'll bet they all have language_country format. I don't debate that there's some redundancy, but the reason for _NO being used is not at all to help the user understand what nn or nb mean, and is not needed to fully understand what either of those means.

(Since 'Norway' is in theory orthogonal to either 'Nynorsk' or 'Bokmål', then _NO would certainly be needed to understand the full meaning of nn_NO or nb_NO -- by definition.)

Peter

---

From: "Kent Karlsson" <[email-masked]>
Date: Mon, 26 May 2008 01:09:43 +0200

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Doug Ewell
...
> > It is redundant. 

For the pure use as a language tag, the NO in nb-NO may be seen as
redundant. But for the derived use as a locale identifier, the NO
part makes a difference, though one not relevant for pure language
tagging. Such things as default currency and maybe even default time
zone (to some extent) may be derived from the territory part. nb_SE
would use a different default currency from nb_NO. And for just 'nb'
there would be no default currency. But as Randy has pointed out,
language tags aren't the same as locale identifiers, though closely
related.

	/kent k

---

From: Leif Halvard Silli <[email-masked]>
Date: Mon, 26 May 2008 02:38:17 +0200

> Leif Halvard Silli <lhs at malform dot no> wrote:

[ macrolanguage things answered at bottom ]

> Why would it be a hassle?

I guess OpenOffice prefers the logic that "all with the same tag 
belong together". (There is both spelling, thesaurus and 
hyphenation for the same language.)

> Exactly.  [...]  I don't know whether hyphenation rules differ between 
> Bokmål and Nynorsk, but I suspect they do not. [...]

Right.

>>> an extlang, as in "ak-tw" (Twi).

   [...]

> 'tw' in "ak-tw" refers to an extlang.

Yes, I was very formalistic when I spoke about "knowing that it is 
in the region/extlang group."

[...]

> 'TW' stands for.  This a valuable property,  [...] 

I can see a value.

> the value of having extlangs in the first place?

You could say so. However, the usefulness of very long tags and 
the fallback behaviour one could get from that, was also 
questioned. You could call the proposal a compromise in that 
sense. A little from both options, but not  - as the co-chair 
suggested (as a last option) - both simultaneously. (No completely 
free choice between extlang and no-extlang - as the co-chair 
hinted, if I interpreted him correctly.) Stricter rules, in order 
to cover all macrolanguages by the same scheme. Such was the thought.

> language.  But there still isn't anything sudden about it.

Whether we got 'nn' and 'nb' then or had "suddenly" gotten 'nob' 
and 'nno' today was still not my point. The point was that a 
macrolanguage tagging method would in fact be simpler for users to 
understand.

> region/extlang subtag model" is simply clouding the issue.

Right. I see that point. I am dropping the proposal.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Mon, 26 May 2008 02:48:38 +0200

Peter Constable 2008-05-25 23.59:

> These statements suggest that you don't really understand the basis for those tags at all! 

Arrested by the Constable. I almost feel Silli. If you prefer to 
assume that I don't understand the what the tags means, so be it.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Sun, 25 May 2008 18:00:44 -0700

> macrolanguage tagging method would in fact be simpler for users to

Well, that's an interesting claim. I've got people (not on this list) claiming the opposite.

Peter

---

From: Leif Halvard Silli <[email-masked]>
Date: Mon, 26 May 2008 04:00:28 +0200

> list) claiming the opposite.

I guess this is what we are debating.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Mon, 26 May 2008 03:00:26 -0400

Doug Ewell scripsit:

> rules in any variety of English.  

Actually en-US and en-GB do have different hyphenation patterns, based
on pronunciation vs. etymology (very roughly): a well-known example is
"prog-ress" vs. "pro-gress".

-- 
John Cowan      [email-masked]
        "Not to know The Smiths is not to know K.X.U."  --K.X.U.
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru