Subject: Re: [Ltru] ISO 639 language code addition rules...

From: "Doug Ewell" <[email-masked]>
Date: Sat, 31 May 2008 12:23:18 -0600

"Phillips, Addison" <addison at amazon dot com> wrote:

> Now... if I understand correctly, ISO 639-2 is supposed to be a strict 
> subset of ISO 639-3 going forward.

I thought 639-2 was supposed to be a subset of (the union of 639-3 plus 
639-5) going forward, at least when 639-5 is published (which I don't 
believe has happened yet).

> So I would tend to propose the rules as follows:
>
> - [1] if an ISO 639-1 code is assigned and it does not conflict with 
> existing codes, register it.
>       Note that languages given an ISO 639-1 code cannot be extlangs, 
> even if enclosed by a macrolanguage
> - [2] if an ISO 639-3 code is assigned and it does not conflict with 
> existing codes, register it

Where "existing codes" includes those just registered in accordance with 
[1] above, or about to be.  That is, if 639-3 announces a new code, but 
their code lists show that it will also have a 2-letter 639-1 code, then 
we use the 2-letter code and not the 3-letter.  I don't know if this 
needs to be explicitly stated or not.

I just want to make sure that we don't register the 639-2/3 code *'fob' 
for Foobazian, and then 639-1 assigns *'fb', and the only difference was 
coordination of announcements between the RAs.

> - [3] if an ISO 639-2 code is somehow assigned and it does not 
> conflict with existing codes, register it
>       Note that this should never happen any more

Unless I was right above about additional 639-5 codes making it into 
639-2.

--
Doug Ewell  *  Arvada, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Sat, 31 May 2008 11:42:34 -0700

It's all in the details of the wording, I guess. Here's what I currently have:

<t>Primary and extended language subtags (other than independently registered values created using the registration process) are created according to the assignments of the various parts of ISO 639, as follows:

<list>
   <t>Codes assigned by ISO 639-1 that do not conflict with existing two-letter primary language subtags and which have no corresponding three-letter primary defined in the registry are entered into the IANA registry as new records of type 'language'. Note that languages given an ISO 639-1 code cannot be extended language subtags, even if enclosed by a macrolanguage.</t>

   <t>Codes assigned by ISO 639-3 that do not conflict with existing three-letter primary language subtags and which do not have ISO 639-1 codes assigned are entered into the IANA registry as new records of type 'language' or 'extlang'. Codes that have a defined "macrolanguage" mapping at the time of their registration MUST be entered into the registry as records of type 'extlang' with a 'Prefix' field containing the appropriate prefix tag. They MUST also include a "Macrolanguage" field in their record. All other records are of type 'language'.</t>

   <t>Any other codes assigned by ISO 639-2 that do not conflict with existing three-letter primary or extended language subtags and which do not have ISO 639-1 two-letter codes assigned are entered into the IANA registry as new records of type 'language'. Subtags assigned according to this rule will mostly be language collections.</t>

</list>

</t>

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Doug Ewell
> Sent: Saturday, May 31, 2008 11:23 AM
> To: LTRU Working Group
> Subject: Re: [Ltru] ISO 639 language code addition rules...
>

---

From: Peter Constable <[email-masked]>
Date: Sat, 31 May 2008 12:27:25 -0700

> coordination of announcements between the RAs.

If it's an addition to 639-1, then either it will already have been in 639-2, or it is being added to both simultaneously, and it would get announced together. It if is a new addition to 639-3, then likely it is not something that would make it into 639-1. In general, I expect new additions to 639-1 to be pretty rare.

I've suggested before that we could decide that we will never again register something from 639-1. That would eliminate these concerns entirely.

Peter

---

From: "Phillips, Addison" <[email-masked]>
Date: Sat, 31 May 2008 13:12:46 -0700

>
> If it's an addition to 639-1, then either it will already have been in
> 639-2, or it is being added to both simultaneously, and it would get
> announced together. It if is a new addition to 639-3, then likely it is
> not something that would make it into 639-1. In general, I expect new
> additions to 639-1 to be pretty rare.

Um... what about my contention that parts 2 and 3 are synchronized? I assume that an addition to 639-2 would automatically also be added to 639-3, no?

>
> I've suggested before that we could decide that we will never again
> register something from 639-1. That would eliminate these concerns
> entirely.
>

It seems unlikely that we'll discover that many unknown languages that merit a 639-1 code, so this sounds sensible, as any future additions to 639-1 will probably be covered by the rule prohibiting assignments that match existing three letter codes.

Addison

---

From: Peter Constable <[email-masked]>
Date: Sat, 31 May 2008 13:17:51 -0700

> From: Phillips, Addison [mailto:[email-masked]]

> Um... what about my contention that parts 2 and 3 are synchronized? I
> assume that an addition to 639-2 would automatically also be added to
> 639-3, no?

Indeed, unless it is a collection or is already in 639-3.

Peter

---

From: "Debbie Garside" <[email-masked]>
Date: Sun, 1 Jun 2008 00:00:08 +0100

Doug wrote:

>at least when 639-5 is published (which
> I don't believe has happened yet).

ISO 639-5 was published on the 15/05/2008

Best

Debbie

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Doug Ewell
> Sent: 31 May 2008 19:23
> To: LTRU Working Group
> 639-5) going forward, at least when 639-5 is published (which
> I don't believe has happened yet).
>
> Where "existing codes" includes those just registered in
> accordance with [1] above, or about to be.  That is, if 639-3
> announces a new code, but their code lists show that it will
> also have a 2-letter 639-1 code, then we use the 2-letter
> code and not the 3-letter.  I don't know if this needs to be
> explicitly stated or not.
>
> for Foobazian, and then 639-1 assigns *'fb', and the only
> difference was coordination of announcements between the RAs.
>
> Unless I was right above about additional 639-5 codes making
> it into 639-2.
>
> --
> http://www.alvestrand.no/mailman/listinfo/ietf-languages  ^
>

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 31 May 2008 18:06:56 -0600

> ISO 639-5 was published on the 15/05/2008

I hadn't realized that.  More to the point, I hadn't realized we had 
delayed quite that long.  ISO 639-5 was always one of those things that 
was regarded as being outside the scope of LTRU II since our deadlines 
were long before the scheduled publication of 639-5.

Should we consider incorporating the 639-5 code elements into 
draft-4645bis?  There would be no architectural change to 4646bis, since 
we already have collection codes from 639-2; this would just add more of 
them.  I can do this relatively easily, if someone can provide me with a 
code list.  (I assume access to the full standard is not required.)

I know some WG members will argue that collection codes are Evil, in 
which case I would respond that perhaps we need to think beyond Web 
servers and spell-checkers.  Our intent is for language tags to be 
useful for a wide variety of applications, and as far as I know --  
despite the horror stories -- collection codes haven't caused major 
tagging problems since 2001 (publication date of RFC 3066, which first 
allowed them).

At the very least, if we are going to continue to include the 639-2-only 
collection codes such as 'bad' and 'gem' but exclude the 639-5 ones, we 
should explain this decision in RFC 4646bis so it doesn't look like an 
oversight.

---

From: "Debbie Garside" <[email-masked]>
Date: Sun, 1 Jun 2008 02:08:47 +0100

Hi Doug

BSI have a copy of ISO 639-5 and I am pretty sure that, whilst the standard
itself is not available freely, the code list is.

I can enquire if you like?

Best regards

Debbie

> From: Doug Ewell [mailto:[email-masked]]
> Sent: 01 June 2008 01:07
> To: LTRU Working Group
> Cc: Debbie Garside
> I hadn't realized that.  More to the point, I hadn't realized
> we had delayed quite that long.  ISO 639-5 was always one of
> those things that was regarded as being outside the scope of
> LTRU II since our deadlines were long before the scheduled
> publication of 639-5.
>
> draft-4645bis?  There would be no architectural change to
> 4646bis, since we already have collection codes from 639-2;
> this would just add more of them.  I can do this relatively
> easily, if someone can provide me with a code list.  (I
> assume access to the full standard is not required.)
>
> I know some WG members will argue that collection codes are
> Evil, in which case I would respond that perhaps we need to
> think beyond Web servers and spell-checkers.  Our intent is
> for language tags to be useful for a wide variety of
> applications, and as far as I know -- despite the horror
> stories -- collection codes haven't caused major tagging
> problems since 2001 (publication date of RFC 3066, which
> first allowed them).
>
> At the very least, if we are going to continue to include the
> 639-2-only collection codes such as 'bad' and 'gem' but
> exclude the 639-5 ones, we should explain this decision in
> RFC 4646bis so it doesn't look like an oversight.
>
> --

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 31 May 2008 23:28:10 -0600

Debbie Garside <debbie at ictmarketing dot co dot uk> wrote:

> BSI have a copy of ISO 639-5 and I am pretty sure that, whilst the 
> standard itself is not available freely, the code list is.
>

That would be great.  Can you also check whether it is acceptable to 
"publish" the codes on this mailing list (which would have the effect of 
publishing them on a Web archive)?  It might be helpful for list members 
to see exactly what subtags would be added.

Thanks,

---

From: John Cowan <[email-masked]>
Date: Sun, 1 Jun 2008 01:36:24 -0400

Doug Ewell scripsit:

> 639-5) going forward,

Correct.

> at least when 639-5 is published (which I don't believe has happened
> yet).

It has!  It went to 60.60 (published) state on May 15, and is now
available for the usual exorbitant price (CHF 102, about USD 98 or
EUR 63).

> coordination of announcements between the RAs.

No more 639-1 assignments will happen, except in the unlikely case that a
language is registered simultaneously in -1, -2, and -3 (which would imply
both that it is *not* now known to Ethnologue, and that it is important
enough to justify a 2-letter code, quod absurdum est.

So we can pretty much rule out new -1 code elements, and likewise new
-2 elements except perhaps for a new macrolanguage (but I doubt it).

-- 
As you read this, I don't want you to feel      John Cowan
sorry for me, because, I believe everyone       [email-masked]
will die someday.                               http://www.ccil.org/~cowan
        --From a Nigerian-type scam spam
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Sun, 1 Jun 2008 03:05:52 -0400

Phillips, Addison scripsit:

> <list>
>    <t>Codes assigned by ISO 639-1 that do not conflict with existing
>    two-letter primary language subtags and which have no corresponding
>    three-letter primary defined in the registry are entered into the
>    IANA registry as new records of type 'language'. Note that languages
>    given an ISO 639-1 code cannot be extended language subtags, even
>    if enclosed by a macrolanguage.</t>

As I just posted, I think this should say "no more 639-1 codes as of Date C."

>    <t>Codes assigned by ISO 639-3 that do not conflict with existing
>    three-letter primary language subtags and which do not have ISO
>    639-1 codes assigned are entered into the IANA registry as new
>    records of type 'language' or 'extlang'. Codes that have a defined
>    "macrolanguage" mapping at the time of their registration MUST
>    be entered into the registry as records of type 'extlang' with a
>    'Prefix' field containing the appropriate prefix tag. They MUST
>    also include a "Macrolanguage" field in their record. All other
>    records are of type 'language'.</t>

Presumably "a macrolanguage mapping to "zho" or "sgn".

>    <t>Any other codes assigned by ISO 639-2 that do not conflict with
>    existing three-letter primary or extended language subtags and which
>    do not have ISO 639-1 two-letter codes assigned are entered into the
>    IANA registry as new records of type 'language'. Subtags assigned
>    according to this rule will mostly be language collections.</t>

+1

And to this add:

<t>Any other codes assigned by ISO 639-5 that do not conflict with
existing three-letter primary or extended language subtags and which do
not have ISO 639-1 two-letter codes assigned are entered into the IANA
registry as new records of type 'language'. Subtags assigned according
to this rule will be language collections.

-- 
First known example of political correctness:   John Cowan
After Nurhachi had united all the other         http://www.ccil.org/~cowan
Jurchen tribes under the leadership of the      [email-masked]
Manchus, his successor Abahai (1592-1643)
issued an order that the name Jurchen should       --S. Robert Ramsey,
be banned, and from then on, they were all           The Languages of China
to be called Manchus.
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Sun, 1 Jun 2008 03:45:32 -0400

Doug Ewell scripsit:

> draft-4645bis?  

I think we should do so, despite 639-5 being technically off-charter.

> tagging problems since 2001 (publication date of RFC 3066, which first 

+1 for this reasoning.

By my highly unofficial count, there are 49 new code elements in the
current 639-5 registry, which is embedded in the International Standard.
There are no exact definitions of either the old 639-2 or the new language
collections, but if we identify them with their obvious counterparts
from the Ethnologue, I think these six cases require special consideration:

1) 'euq' codes "Basque (family)".  The Ethnologue lists three languages
in this family, but ISO 639-3 has folded two of them into 'eu', Basque
proper.  The next edition of the Ethnologue will presumably follow.
So 'euq' and 'eu' have the same denotation even though they have
different scopes.  Strong recommendation: exclude 'euq'.

2) 'hyx' represents "Armenian (family)".  Ethnologue shows only one
language in this family.  So 'hyx' and 'hy' have the same denotation even
though they have different scopes.  Strong recommendation: exclude 'hyx'.

3) 'jpx' represents "Japanese (family)".  Ethnologue shows 12 languages in
this family, of which Japanese proper is dominant.  Strong recommendation:
include 'jpx'.

4) 'qwe' represents "Quechuan (family)".  Ethnologue shows 46 languages
in this family, of which all but two (Inga 'inb' and Jungle Inga 'inj'
are included in the macrolanguage 'qu'.  (Presumably these two are
excluded because they do not have "Quechua" or "Quichua" in their names.)
Weak recommendation: exclude 'qwe'.

5) 'sqj' represents "Albanian (family)".  Ethnologue shows four languages
(aln, aae, aat, als) in this family; however, the macrolanguage 'sq'
encompasses these four languages.  So 'sqj' and 'sq' have the same
denotation even though they have different scopes.  Strong recommendation:
exclude 'sqj'.

6) 'zhx' represents "Chinese (family)".  Ethnologue shows 14 languages
in this family; however, the macrolanguage 'zh' encompasses all but one
of these (Dungan 'dng').  (Dungan is excluded because it does not have
"Chinese" in its name.)  Weak recommendation: exclude 'zhx'.

> collection codes such as 'bad' and 'gem' but exclude the 639-5 ones,
> we should explain this decision in RFC 4646bis so it doesn't look like
> an oversight.

+1 to that too.

-- 
John Cowan    http://ccil.org/~cowan  [email-masked]
The Penguin shall hunt and devour all that is crufty, gnarly and
bogacious; all code which wriggles like spaghetti, or is infested with
blighting creatures, or is bound by grave and perilous Licences shall it
capture.  And in capturing shall it replicate, and in replicating shall
it document, and in documentation shall it bring freedom, serenity and
most cool froodiness to the earth and all who code therein.  --Gospel of Tux
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Sun, 1 Jun 2008 01:34:21 -0700

> Should we consider incorporating the 639-5 code elements into

Well, here's an interesting reason for saying that we should: We have discussed before concerns over inherent *in*stability of the "(other)" collections in ISO 639-2: if any member language is coded, then the denotation of the ID for that collection has narrowed, and some unknown number of existing usages become incorrectly tagged. For that reason, I was trying to get agreement within the JAC to change all the exclusive "(other)" collections to regular, inclusive collections. I thought I had obtained a consensus agreement, although the librarians really wanted to continue using the exclusively-defined categories -- I was trying to get agreement to treat that as an implementation decision.

Well, it turns out that 639-5 has baked in an interesting relationship with 639-2 in relation to these "(other)" collections: the categories are part of both codes (indeed, all collections in -2 are in -5), and they denote the same groupings *with the exception* that they are *exclusive* collections in -2 but are *inclusive* in -5. Here's a quotation from clause 3.2 of ISO/FDIS 639-5:

<quote>
There are currently 55 items that are included both in
ISO 639-2 and in ISO 639-5. Of these items, 20
items are identical in the two parts (e.g. “alg –
Algonquian languages”). The remaining 35 items are
intended to cover remainder groups in ISO 639-2 and
entire language groups in ISO 639-5 (e.g. “afa –
Afro-Asiatic (Other)” in ISO 639-2 and “afa – Afro-
Asiatic languages” in ISO 639-5).
</quote>

Thus, if we want to get rid of exclusive collections, it looks like we should probably reference 639-5.

Between 639-5 and 639-3, it looks like 639-2 would be redundant for our purposes.

Peter