Subject: [Ltru] John Cowan throws in the towel on extlangs

From: John Cowan <[email-masked]>
Date: Thu, 29 Nov 2007 09:39:20 -0500

After yesterday's telcon, it became clear to me that we had no movement
toward reconciling the extlang-yes and the extlang-no factions, and
that if we were ever to make progress, someone would have to concede.
That turns out to be me.  No one else has been actually defending
the use of extlangs, and several people are either mildly or strongly
against them, so all I am doing now is blocking consensus to no purpose.
Therefore, I have reluctantly ceased to block it.

The strongest argument against extlang tags is that our simple
remove-from-right lookup algorithm ends up losing too much information.
If you ask for "zh-yue-Hant", and there are no Cantonese resources, then
by the time you have truncated the tag to just "zh" you have forgotten
that Traditional Han script was required.  In order to prevent it,
we would have to change 4647 to specify a sequence like zh-yue-Hant >
zh-yue > zh-Hant > zh, where the extlang tag is treated magically;
in essence, stripping it restores all other tags.  That would be fine,
except that no one actually does it or is likely to for some time to come.

Everything else is more or less symmetrical.  For example, almost
all content tagged 'zh' is in fact Mandarin, as an uncontroversial
consequence of the fact that almost all tagged content is written,
and almost all written Chinese is in fact Mandarin.  There will be a
problem in getting good matches if people use the new 'cmn' subtag,
as naive matchers will not know its relationship to 'zh', but no worse
than the problems shown above for Cantonese.

In practical terms, this means that (unless someone else wants to take
up the cause on the LTRU mailing list and see Bonnie Prince Charlie
come into his own again) that we will allow all 639-3 code elements
to be language subtags, and deprecate all the grandfathered forms like
"zh-yue" (Cantonese) in favor of just 'yue'.  Similarly, the currently
redundant sign languages like "sgn-US" will be deprecated in favor of
'ase' and friends.  The existing language tags representing macrolanguages
of course remain intact.

IMO we should strongly consider adding a new informative (and mutable)
"Fallback" header in the registry which will inform people about
problematic cases like "cmn" and "arb" (Standard Arabic), instructing
them which language subtag to fall back to in cases of match failure.
These will have to be cherry-picked, like Suppress-Script, but Peter
Constable has estimated that there are no more than 15 such cases
actually in wide use.  For some macrolanguages, there is no dominant
variety, and no special consideration is as yet required; if new dominant
varieties come to exist in future, new Fallback headers can be added.
RFC 4647bis can then be revised to explain how this header MAY be used
to enhance matching.

-- 
Mark Twain on Cecil Rhodes:                    John Cowan
I admire him, I freely admit it,               http://www.ccil.org/~cowan
and when his time comes I shall                [email-masked]
buy a piece of the rope for a keepsake.

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Addison Phillips <[email-masked]>
Date: Thu, 29 Nov 2007 08:30:34 -0800

> RFC 4647bis can then be revised to explain how this header MAY be used

Wouldn't it be easier, more consistent, cleaner, and cherry-free to keep 
what we've already proposed in draft-09, that is, to include the 
Macrolanguage field in the registry as informative information (and then 
note in either 4646bis or 4647bis about possible courses of action in 
cases of match failure)?

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Don Osborn" <[email-masked]>
Date: Thu, 29 Nov 2007 12:09:58 -0500

Some macrolanguage headings might be used also for locales and for describing orthographies (which in Africa often are defined by country and not by categories in ISO 639). How would the draft affect such cases? So in the case of Fula, one would probably be talking about ff-SN, ff-NE, etc., for at least some purposes, without resort to the alpha-3 codes introduced in part 3.

(Sorry I have not been tracking this as closely as I would like to have.)

Don

> -----Original Message-----
> From: Addison Phillips [mailto:[email-masked]]
> Sent: Thursday, November 29, 2007 11:31 AM
> To: John Cowan
> Cc: [email-masked]
> Subject: [Ltru] Fallback vs. Macrolanguage? (was: John Cowan throws
> in...)
> 
> John Cowan wrote:
> >

---

From: Peter Constable <[email-masked]>
Date: Thu, 29 Nov 2007 13:48:51 -0800

> From: John Cowan [mailto:[email-masked]]

> RFC 4647bis can then be revised to explain how this header MAY be used

I've taken a cut at analyzing macrolanguages in terms of two key factors: widespread exposure in ICT (based on support in OSes), and whether there is a dominant language. (Of course, "dominant" varies on a continuum.)

I ended up with 16 considered to have wide exposure. File is attached.

Peter

---

From: "Don Osborn" <[email-masked]>
Date: Thu, 29 Nov 2007 17:01:53 -0500

Apologies in advance if this is a non-sequiteur to this particular thread,
but is there provision for new macrolanguages. A while back there was a
suggestion that we know them all, but I could suggest other possibilities
such as Runyakitara (aready mentioned), Oshiwambo, possible new mix for
Manding languages. This may be ISO 639 territory, but also would relate to
use of tags.

Don

> From: Peter Constable [mailto:[email-masked]]
> Sent: Thursday, November 29, 2007 4:49 PM
> To: John Cowan; [email-masked]
> Subject: RE: [Ltru] John Cowan throws in the towel on extlangs
> 
> > variety, and no special consideration is as yet required; if new
> > dominant varieties come to exist in future, new Fallback headers can
> > be added.
> I've taken a cut at analyzing macrolanguages in terms of two key
> factors: widespread exposure in ICT (based on support in OSes), and
> whether there is a dominant language. (Of course, "dominant" varies on
> a continuum.)
>

---

From: John Cowan <[email-masked]>
Date: Thu, 29 Nov 2007 17:39:47 -0500

> Manding languages. This may be ISO 639 territory, but also would relate to

In principle, 639-3/RA may create them at any time.  So far they have not been
asked to do so.

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
The known is finite, the unknown infinite; intellectually we stand
on an islet in the midst of an illimitable ocean of inexplicability.
Our business in every generation is to reclaim a little more land,
to add something to the extent and the solidity of our possessions.
        --Thomas Henry Huxley

---

From: Addison Phillips <[email-masked]>
Date: Thu, 29 Nov 2007 14:53:39 -0800

> In principle, 639-3/RA may create them at any time.  So far they have not been

Further, if John's towel stays thrown, it wouldn't impact RFC 4646bis. 
It *would* potentially require new information be placed into the 
registry (fallback or macrolanguage fields) and thus might influence 
various tagging choices.

Note that there is no difference between extlang and no-extlang on this: 
most enclosed languages would already be in the registry as primary 
language subtags and thus could not become extlang subtags after the fact.

Addison

---

From: John Cowan <[email-masked]>
Date: Fri, 30 Nov 2007 01:22:29 -0500

> note in either 4646bis or 4647bis about possible courses of action in 

I don't think it's necessarily appropriate to fall back in all cases of
macrolanguages (as Peter's list shows), and it may well be appropriate
to fall back in some cases that aren't macrolanguages at all (Breton ->
French is the example we've talked about a lot).  Using a different name
lets us decouple from 639-3's particular choices.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
Half the lies they tell about me are true.
        -- Tallulah Bankhead, American actress

---

From: Peter Constable <[email-masked]>
Date: Fri, 30 Nov 2007 07:38:00 -0800

> From: Don Osborn [mailto:[email-masked]]

> Apologies in advance if this is a non-sequiteur to this particular

It is not.

> Manding languages. This may be ISO 639 territory, but also would relate
> to use of tags.

New macrolanguage entries can be added to ISO 639.

If RFC4646bis were to use extlang subtags prefixed by macrolanguage subtags, then new macrolanguage IDs could not be prefixed to existing language IDs. The macrolanguage ID could be used on its own, though; it could mean needing data in the LSR to support matching a macrolanguage ID against IDs for the encompassed languages (as would be done if extlangs are not used).

For example, if none of zh, cmn and yue were registered today, they could be added tomorrow and extlang subtags zh-cmn or zh-yue could be used. If, on the other hand, cmn and yue were registered today but zh was not, zh could still be added tomorrow, but cmn and yue would already be used as primary subtags, so zh could not be prefixed. It could still be used as a primary subtag, without extlang subtags; we would just need to allow for possible need to match content tagged "cmn" or "yue" with a request for "zh", or vice versa.

(Btw, the possibility of new macrolanguages encompassing pre-registered languages forms one of the arguments against extlang: you end up needing to support approaches to matching that would be needed without extlangs anyway.)

Now, in the kinds of scenarios you have in mind, it's not clear to me if macrolanguage is the appropriate scope for new entries that might be requested. Keep in mind that the macrolanguage has a measure of ambiguity: it is not one specific variety, but rather one of several distinct varieties. For instance, either Mandarin and Cantonese content can be labeled "Chinese", and content labeled "Chinese" could well be either of those (or any of the other Chinese languages).

If Kwanyama and Ndonga are distinct languages and if "Oshiwambo" can refer to either and is in some contexts used as though it were a single language rather than a collection, then it can be considered a macrolanguage. But if Oshiwambo refers to a distinct variety that is not the same as either Kwanyama or Ndonga but is used or coming into use by speakers of both Kwanyama and Ndonga communities, then it might make more sense to consider Oshiwambo an individual language.

(I don't know which situation applies to Oshiwambo, and I'm not intending to steer discussion in that direction -- which would be off topic.)

There are cases in which language differentiation has occurred so that, over some extended period of time, a single language community has descendents speaking multiple distinct languages; but in which a unifying process has occurred (whether naturally or engineered) with a single language emerging that is used by speakers of the various languages. An example of an engineered case is Filipino; I gather that N'Ko is somewhat an example of natural evolution on this type. These are individual languages, not macrolanguages. E.g. "Filipino" would not be an appropriate label for Cebuano content.

Peter

---

From: [email-masked]
Date: Fri, 30 Nov 2007 09:47:23 -0800

Just a couple points from my perspective on the Chinese languages and 
extlang. 

1) We use "zh" today to indicate either Mandarin or Cantonese audio or 
written language that has not yet been verified to be either Cantonese or 
Mandarin. This is one use of the tag.

2) We tag audio content with the "zh" tag.

3) The audio content we tag with the "zh" tag is sometimes Cantonese.

4)  We need a tag for written Cantonese used in subtitles, and today we 
use "zh-yue."

I apologize for not being able to join any of the conference calls. I am 
OK with whatever the committee decides, but some of the premises seemed 
off based on our usage.

Regards,

Karen Broome
Metadata Systems Designer
Sony Pictures Entertainment
310.244.4384

---

From: John Cowan <[email-masked]>
Date: Fri, 30 Nov 2007 18:07:49 -0500

> Mandarin. This is one use of the tag.

No problem.

> 2) We tag audio content with the "zh" tag.

No problem.

> 3) The audio content we tag with the "zh" tag is sometimes Cantonese.

No problem.  When you know it is Cantonese, you'd be encouraged
to switch to "yue".   When you know it is Mandarin, you may use "cmn",
but that might cause interop problems with applications that don't 
know that tag.

> 4)  We need a tag for written Cantonese used in subtitles, and today we 

That will still be valid, but deprecated; you'd be encouraged
to switch to "yue".

-- 
By Elbereth and Luthien the Fair, you shall     [email-masked]
have neither the Ring nor me!  --Frodo          http://www.ccil.org/~cowan