Subject: [Ltru] ISO 639 codes

From: "Mark Davis" <[email-masked]>
Date: Tue, 6 Mar 2007 00:05:11 -0800

I just dumped some information about the ISO 639 codes, and thought people
might be interested. It does raise one issue, listed below.

    Part Count
Scope Type Samples  Iso639_1 143
Individual Living [aa, ab, af, ak-tw, am, an, as, av, ba, be, bg, bi, bm,
bn, bo, br, ca, ce, ch, co, cs, cv, cy, da,…  Iso639_1 30
Macrolanguage Living [ak, ar, ay, az, cr, fa, ff, gn, ik, iu, kg, kr, ku,
kv, mg, mn, ms, no, oc, oj, om, ps, qu, sc, sq…  Iso639_1 5
Individual Ancient [ae, cu, la, pi, sa]  Iso639_1 5
Individual Constructed [eo, ia, ie, io, vo]  Iso639_1 1
Collection Special [bh]  Iso639_1 1
Individual Extinct [gv]  Iso639_2 520
PrivateUse Special [qaa, qab, qac, qad, qae, qaf, qag, qah, qai, qaj, qak,
qal, qam, qan, qao, qap, qaq, qar, qas, qat…  Iso639_2 168
Individual Living [ace, ach, ada, ady, ain, ak-fat, ale, alt, anp, arn, arp,
arw, ast, awa, ban, bas, bej, bem, bho, …  Iso639_2 66
Collection Special [afa, alg, apa, art, ath, aus, bad, bai, bat, ber, bnt,
btk, cai, cau, cel, cmc, cpe, cpf, cpp, crp…  Iso639_2 25
Macrolanguage Living [bal, bik, bua, chm, del, den, din, doi, gba, gon, grb,
hai, hmn, jrb, kok, kpe, lah, man, mwr, raj…  Iso639_2 15
Individual Historical [ang, dum, enm, frm, fro, gmh, goh, grc, mga, non,
nwc, ota, peo, pro, sga]  Iso639_2 14
Individual Ancient [akk, arc, egy, elx, gez, got, hit, kaw, kho, pal, phn,
sog, sux, uga]  Iso639_2 4
Individual Extinct [chb, chg, cop, sam]  Iso639_2 3
Individual Constructed [afh, jbo, tlh]  Iso639_2 3
Special Special [mul, und, zxx]  Iso639_3 6616
Individual Living [aaa, aab, aac, aad, aaf, aai, aak, aal, aam, aan, aap,
aas, aau, aaw, aax, aay, aaz, aba, abb, abc…  Iso639_3 420
Individual Extinct [aaq, abj, aci, ack, acl, acs, aes, aga, aho, ait, ajw,
akj, akm, akx, aky, ama, ana, anb, ans, aoh…  Iso639_3 94
Individual Ancient [cms, ecr, ecy, ett, gmy, hlu, htx, ims, imy, inm, lng,
nei, nrc, nrp, nxm, oar, obm, och, oht, omn…  Iso639_3 38
Individual Historical [axm, cmg, cnx, hbo, ltc, myz, nci, nwx, oav, obr,
obt, oco, odt, ofs, oge, ohu, ojp, okm, oko, omp…  Iso639_3 9
Individual Constructed [bzt, dws, igs, ldn, nov, occ, qya, rmv, sjn]
Iso639_3 1
Macrolanguage Living [sh]
Notes:

   - Unfortunately, ISO-639-3 is not a full superset of the others, so I
   had to guess that the Collection codes are all and only those ISO-639-2
   codes that are not private-use and are not ISO-639-3. Because of that, I
   also don't have breakdowns for Collections into Ancient, Living, etc.
   - Note an oddity: 'sh' is a deprecated ISO 636-1 code and not found in
   ISO-639-2_values_8bits.txt<http://www.loc.gov/standards/iso639-2/ISO-639-2_values_8bits.txt>,
   but is a valid ISO 639-3 code and found in
   http://www.sil.org/iso639-3/iso-639-3_20070205.tab  (hbs
   sh    M    L    Serbo-Croatian)

Anyway, here is the issue.

http://www.ietf.org/internet-drafts/draft-ietf-ltru-4645bis-01.txt says

   For each language in [ISO639-3] that was not already represented by a
   primary language subtag in the Language Subtag Registry, a new subtag
   was added to the Registry...

   o  If the language was encompassed by a macrolanguage, as determined
      by [iso-fdis-639-3-macrolanguages_20061114], an extended language
      subtag was added, with the primary language subtag of the
      macrolanguage as the value for the Prefix field.

There are few encompassed languages that were already primary language
subtags. They are not affected by this, as Doug quite rightly doesn't touch
them as per the first paragraph above. These are listed below.

   ak-tw Iso639_1 Individual Living Twi  no-nb Iso639_1 Individual
Living Norwegian
Bokmål  no-nn Iso639_1 Individual Living Norwegian Nynorsk  sh-bs Iso639_1
Individual Living Bosnian  sh-hr Iso639_1 Individual Living Croatian  sh-sr
Iso639_1 Individual Living Serbian  ak-fat Iso639_2 Individual Living Fanti
However, that does introduce an inconsistency in the handling of these
languages vs others like
...
     zh-cjy Iso639_3 Individual Living Jinyu Chinese

 zh-cmn Iso639_3 Individual Living Mandarin Chinese

 zh-cpx Iso639_3 Individual Living Pu-Xian Chinese

 zh-czh Iso639_3 Individual Living Huizhou Chinese

 zh-czo Iso639_3 Individual Living Min Zhong Chinese

 ...

I don't know that it is a particularly big deal, but it was a bit of a
surprise to me that the first group existed, and just thought I'd bring it
to the group's attention. (Of course, it would have been nice to have had
this a long time ago, to have taken care of the <no, nb, nn> and the <sh,
sr, hr, bs> messes!)

Mark

BTW, I noticed a whole lot of languages that appeared, from the names, to be
extended languages according to the definitions given, eg for "Zoque":

   zoc Iso639_3 Individual Living Copainalá Zoque  zoh Iso639_3 Individual
Living Chimalapa Zoque  zoq Iso639_3 Individual Living Tabasco Zoque  zor
Iso639_3 Individual Living Rayón Zoque  zos Iso639_3 Individual Living
Francisco
León Zoque
However, once we encode these as primary language tags, they can never be
changed to extended language tags...
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 06 Mar 2007 16:54:54 +0100

Mark Davis wrote:

> subtags. They are not affected by this, as Doug quite rightly doesn't
> touch them as per the first paragraph above. These are listed below.
 
> ak-tw   Iso639_1Individual  Living    Twi
> no-nb   Iso639_1Individual  Living    Norwegian BokmÃ¥l
> no-nn   Iso639_1Individual  Living    Norwegian Nynorsk
> sh-bs   Iso639_1Individual  Living    Bosnian
> sh-hr   Iso639_1Individual  Living    Croatian
> sh-sr   Iso639_1Individual  Living    Serbian
> ak-fat  Iso639_2Individual  Living    Fanti

In other words we already have sh (deprecated), bs, hr, and sr as
languages in the 4646 registry, and 4645bis won't change this.

> zh-cjy  Iso639_3Individual  Living    Jinyu Chinese
> zh-cmn  Iso639_3Individual  Living    Mandarin Chinese
> zh-cpx  Iso639_3Individual  Living    Pu-Xian Chinese
> zh-czh  Iso639_3Individual  Living    Huizhou Chinese
> zh-czo  Iso639_3Individual  Living    Min Zhong Chinese

zh is a language in the 4646 registry, and 4645bis won't change this.

There are no 639-2 languages cjy, cmn, cpx, czh, and czo in the 4645
registry, these are new additions in 4645bis, registered as extlangs.

> I don't know that it is a particularly big deal, but it was a bit of
> a surprise to me that the first group existed, and just thought I'd
> bring it to the group's attention.

Thanks, looking at it I see no problem with the first group, but it
would be weird if cjy (etc.) are "upgraded" to 639-2 languages later:

IIRC we can't move an extlang to a language later.  I'm sure that we
already discussed this, but I forgot the fine print, is there some
kind of guarantee that cjy (etc.) will be never added to ISO 639-2 ?

Frank

---

From: Gerard Meijssen <[email-masked]>
Date: Tue, 06 Mar 2007 17:07:45 +0100

> kind of guarantee that cjy (etc.) will be never added to ISO 639-2 ?
Hoi,
I hope there is no such guarantee because in the case of Chinese, it is 
in essence a written language only.  I would really welcome the  cjy 
(etc.) languages to be promoted to this "higher" level.
Thanks,
    GerardM

---

From: John Cowan <[email-masked]>
Date: Tue, 6 Mar 2007 11:29:08 -0500

> kind of guarantee that cjy (etc.) will be never added to ISO 639-2 ?

There is no such guarantee, but I'd think it would be improbable.
The whole point of a macrolanguage in most (not all) cases is that
there is a single set of writing conventions (639-2 is bibliographic
in origin, remember) that covers several different 639-3 languages.

-- 
One Word to write them all,             John Cowan <[email-masked]>
  One Access to find them,              http://www.ccil.org/~cowan
One Excel to count them all,
  And thus to Windows bind them.                --Mike Champion

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 06 Mar 2007 17:58:08 +0100

> I hope there is no such guarantee because in the case of Chinese, it
> is in essence a written language only.  I would really welcome the cjy

Okay, then we have to consider this possibility for the 4646bis rules.

John confirmed in <[email-masked]>:

| There is no such guarantee, but I'd think it would be improbable.
| The whole point of a macrolanguage in most (not all) cases is that
| there is a single set of writing conventions (639-2 is bibliographic
| in origin, remember) that covers several different 639-3 languages.

Let's assume that cjy will be added to 639-2 later.  4646bis could then
decree "too late, you lose".  It could also play along, adding a new
language cjy, and deprecating the extlang cjy.

John said that's improbable, therefore I'd prefer the second solution,
it's more flexible.  And I like to keep the "all ISO 639-2 languages
are also 3066, 4646, and 4646bis languages" rule (of course excluding
the known cases where 639-1 codes are used instead of 639-2 codes).

Frank

---

From: Addison Phillips <[email-masked]>
Date: Tue, 06 Mar 2007 09:22:03 -0800

We already have rules dealing with this in the 4646 era.

I agree with Gerard that the (spoken) range of Sinitic languages is wide 
and generally not mutually intelligible. But it is not maltreating these 
languages to allow them to share a 'zh' prefix and certainly no one 
should claim that the 'zh' subtag has no utility whatsoever.

It isn't a value judgment on Chinese languages and dialects that they 
need to use extended language subtags. It is recognition of the fact 
that 'zh' is useful in identifying content and that it has been a 
consistent practice for some time.

Permitting (i.e. forcing) users to choose between "cjy-CN" and 
"zh-cjy-CN" ignores the prefix-matching used in RFC 4647. I think it 
would create serious problems.

 > the known cases where 639-1 codes are used instead of 639-2 codes).

If I understand ISO 639's future direction correctly, in the future ISO 
639-2 will be a profile of ISO 639-3. A better description of the 
language subtags would be: all ISO 639-3 codes (which includes all of 
the ISO 639-2 codes) are language subtags, excepting codes which have an 
ISO 639-1 code and historical accidents (in which a macrolanguage is 
later "divided up"). The former use the alpha2 and the later are encoded 
using an extlang.

With luck and care, we won't make many extlangs after the initial set of 
them.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 Mar 2007 11:43:27 -0800

I would not say there's any guarantee that these will never be added to 639-2, nor would I go as far to predict improbability in the general case.

But I don't see why that's an issue. IMO, the difference between inclusion in 639-2 vs. 639-3 only should be irrelevant for us.

Peter

> -----Original Message-----
> From: John Cowan [mailto:[email-masked]]
> Sent: Tuesday, March 06, 2007 8:29 AM
> To: Frank Ellermann
> Cc: [email-masked]
> Subject: Re: [Ltru] Re: ISO 639 codes
>

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 Mar 2007 11:45:19 -0800

> From: Gerard Meijssen [mailto:[email-masked]]

> (etc.) languages to be promoted to this "higher" level.

The quotation marks for '"higher"' are quite appropriate. I don't think we should characterize differences between the language repertoires for 639-1, -2 and -3 in terms of "higher" / "lower".

Peter

---

From: GerardM <[email-masked]>
Date: Tue, 6 Mar 2007 20:50:38 +0100

Hoi,
Effectively an ISO-639-2 recognition for these "spoken" languages, that are
characterised as Chinese, would make it more problematic to maintain that
the notation of zh-xxx is justifiable. Therefore it has the effect as being
of a "higher" value.
Thanks,
    GerardM

On 3/6/07, Peter Constable <[email-masked]> wrote:
> The quotation marks for '"higher"' are quite appropriate. I don't think we
> should characterize differences between the language repertoires for 639-1,
> -2 and -3 in terms of "higher" / "lower".
>
>
> Peter
>

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 06 Mar 2007 21:22:05 +0100

Addison Phillips wrote:
 
> "zh-cjy-CN" ignores the prefix-matching used in RFC 4647.

If the 4645bis extlang cjy is deprecated in favour of a future
language subtag cjy, applications aware of this could replace
zh-cjy by cjy on the fly before trying to match tags.

> I think it would create serious problems.

It's not worse than other cases of deprecated tags or subtags.

OTOH I think it would be a serious problem if some 639-2 codes
aren't permitted without a prefix in 4646bis tags:  Folks will
go for the shortest tag they can get away with and beyond.  My
crystal ball says.  

Frank

---

From: John Cowan <[email-masked]>
Date: Tue, 6 Mar 2007 15:21:35 -0500

Peter Constable scripsit:

> But I don't see why that's an issue. IMO, the difference between
> inclusion in 639-2 vs. 639-3 only should be irrelevant for us.

Mostly it is.  But suppose there is a 639-3 only language code
element 'xxx' which is encompassed by 'yyy'.  At present we will
use the tag 'yyy-xxx'.  Now suppose 'xxx' is added to 639-2.

Do we deprecate "yyy-xxx" in favor of "xxx", or do we ignore the
addition of "xxx"?  This issue *must* be decided for 4646bis.

I favor the latter course.

-- 
John Cowan    <[email-masked]>     http://www.ccil.org/~cowan
But no living man am I!  You look upon a woman.  Eowyn I am, Eomund's daughter.
You stand between me and my lord and kin.  Begone, if you be not deathless.
For living or dark undead, I will smite you if you touch him.

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 6 Mar 2007 12:50:28 -0800

The only reason we have for using the structure "zh-min" instead of "min" is
so that matching with fallback works without recourse to other information.
That is,

   1. if we used "zh-min", and that isn't found, then normal fallback
   will give us "zh".
   2. if we used "min", and that isn't found, then normal fallback would
   not know to give us "zh".
   1. That level of fallback could, however, be done by accessing the
      macrolanguage information in ISO 639-3.

This is not a judgment about "higher" or "lower" at all. One can discuss the
technical merits of approach #1 vs #2, but as technical merits.

Mark

On 3/6/07, GerardM <[email-masked]> wrote:
>
> Hoi,
> Effectively an ISO-639-2 recognition for these "spoken" languages, that
> are characterised as Chinese, would make it more problematic to maintain
> that the notation of zh-xxx is justifiable. Therefore it has the effect as
> being of a "higher" value.
> Thanks,
>     GerardM
>
> > is
> > The quotation marks for '"higher"' are quite appropriate. I don't think
> > we should characterize differences between the language repertoires for
> > 639-1, -2 and -3 in terms of "higher" / "lower".
> >
> >
> > Peter
> >

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 Mar 2007 15:05:28 -0800

As I said in response to Gerard, this should have nothing to do with inclusion in 639-2; it should be based on macrolanguage mappings in 639-3 together with constraints to ensure stability in such a case as this.

In your scenario, a tag 'xxx' in 639-3 can get added to 639-2 without any change in macrolanguage mappings, and without that tag being in a macrolanguage-mapping relation. Also, a macrolanguage entity could get added to 639-3 without being included in 639-2.

Inclusion in 639-2 should be completely irrelevant for such purposes, IMO.

Peter

> Sent: Tuesday, March 06, 2007 12:22 PM
> To: Peter Constable
> Cc: [email-masked]

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 Mar 2007 15:05:28 -0800

Peter

---

From: Addison Phillips <[email-masked]>
Date: Tue, 06 Mar 2007 15:17:51 -0800

> zh-cjy by cjy on the fly before trying to match tags.

That changes a basic assumption of at least some matching 
implementations---that all you need are the strings you're given. 
Matching today doesn't follow deprecation paths. User's tag choices 
(presumably) do.

> It's not worse than other cases of deprecated tags or subtags.

Except that there is a *lot* of content tagged with 'zh'. Most of the 
deprecated codes are pretty small potatoes (important if you happen to 
be involved)... and very few language subtags are (the classical 
"mistaken codes", plus Serbo-Croatian) or likely will be.

> go for the shortest tag they can get away with and beyond.  My

I understand what you're getting at: that some folks may use the ISO 
639-2 codes without looking at the registry to see that they are 
extlangs in need of a prefix. This will, indeed, be a problem. You 
hardly need a crystal ball to predict this. Still, if most users get 
used to seeing the right format, I predict that they will actually rebel 
against shorter forms.

That is, I imagine that Chinese folks will rapidly get the idea that 
they should insert a three-letter code into their zh-XXXX-YY tag to make 
zh-zzz-XXXX-YY, rather than jumping to the conclusion that they should 
be using zzz-XXXX-YY.