Subject: [Ltru] Process for creating 4646bis Registry

From: "Doug Ewell" <[email-masked]>
Date: Sat, 16 Sep 2006 22:29:09 -0700

Here is my first pass at describing the process for generating the RFC 
4646bis Language Subtag Registry.  This is not necessarily the exact 
wording that will appear in RFC 4645bis, but parts of it may end up 
there with little or no change.

This needs to be evaluated by the WG on at least two levels:

* the process itself
* the way the process is described (logic, clarity, ambiguity, 
mechanics, etc.)

1.  Starting point

The existing Registry serves as the starting point for the new Registry. 
RFC 4645bis will include a reference (probably normative) to RFC 4645 
for those who wish to trace the path all the way back to the ISO and UN 
standards and RFC 3066 tag registry.

The new inputs to the Registry are the ISO/FDIS 639-3 online code 
tables.  This includes the code set dated 2006-04-21 and the 
macrolanguage mappings dated 2005-06-14, which are currently 
(2006-09-16) available on the ISO 639-3 home page.  Peter has said there 
are changes in the FDIS document not reflected in these files, and more 
changes will be coming, but I am using these files because they are the 
most up-to-date information available to me.

2.  Language subtags modified

The existing Registry includes 489 primary language subtags, including 
some for "language collections" not included in 639-3.  In cases where 
the reference name(s) in 639-3 for these languages differ(s) from the 
Description fields in the Registry, the new names are added to the 
existing Registry entry, after all existing descriptions.  For example, 
the subtag "gd" currently has the descriptions "Gaelic" and "Scottish 
Gaelic"; the new description "Gaelic (Scots)" from 639-3 will be added. 
This includes inverted forms of existing names, such as "Frisian, 
Northern".

This also includes language names that include a country name, to 
differentiate them from other languages of the same name associated with 
a different country.  For example, the subtag "bas" currently has the 
description "Basa"; to this would be added "Basa (Cameroon)" from 639-3. 
The new subtag "bzw" would have the single description "Basa (Nigeria)".

Reference names in the 639-3 files that contain the strings "(generic)" 
or "(specific)" are changed to "(macrolanguage)" and the empty string, 
respectively, in accordance with Peter's statement on September 12 that 
the FDIS already includes this change.  The single exception is "Zande 
(specific)" (zne), which is left unchanged because there is also a 639-3 
code element "Zande" (znd) and neither of these is a macrolanguage.

No existing Description fields are changed or deleted.

3.  New subtags added

All 639-3 code elements not in the LSR are added as primary language 
subtags if they are not included under a 639-3 macrolanguage, or as 
extended language subtags if they are (with the macrolanguage as 
Prefix).  This is consistent with John Cowan's description of the 
process, which was generally approved by the list.

As a special case, all 639-3 languages with the words "Sign Language" in 
their name are added as extlangs, with "sgn" as the Prefix.  This is the 
same as if 639-3 had listed them under a macrolanguage "sgn".  (The 
entry "sgn", being a collection code, is not present in 639-3 at all.)

All subtags added to the Registry are added in alphabetical order within 
their type, with the 2-letter language subtags appearing before the 
3-letter subtags.  IANA previously added two 3-letter subtags, "anp" and 
"frr", in the section for 2-letter subtags, which is allowed by RFC 4646 
but may surprise some readers.  No attempt is made here to change this.

4.  Grandfathered and redundant tags

As a consequence of adding these new subtags, the following 
grandfathered tags are deprecated, with the indicated tag specified as 
Preferred-Value:

i-ami      -> ami
i-bnn      -> bnn
i-pwn      -> pwn
i-tao      -> tao
i-tay      -> tay
i-tsu      -> tsu
sgn-CH-de  -> sgn-sgg
zh-hakka   -> zh-hak
zh-min     -> (no Preferred-Value, just deprecated)
zh-min-nan -> zh-nan
zh-xiang   -> zh-hns

Each of these grandfathered tags has a Comment field indicating the ISO 
639-3 code element (for example, "replaced by ISO code ami").  This 
duplicates the Preferred-Value information and is not strictly 
necessary, but is added for consistency with other grandfathered tags. 
This comment is also added to the tag "zh-guoyu", which is already 
deprecated in favor of another grandfathered tag, "zh-cmn" (which will 
now be redundant; see below).

The following grandfathered tags are now fully composable and are moved 
to redundant:

zh-cmn
zh-cmn-Hans
zh-cmn-Hant
zh-gan
zh-wuu
zh-yue

The following redundant sign-language tags are deprecated (requiring a 
change in the RFC 4646 rules, which currently disallow this).

sgn-BR -> sgn-bzs
sgn-CO > sgn-csn
sgn-DE > sgn-gsg
sgn-DK > sgn-dsl
sgn-ES > sgn-ssp
sgn-FR > sgn-fsl
sgn-GB > sgn-bfi
sgn-GR > sgn-gss
sgn-IE > sgn-isg
sgn-IT > sgn-ise
sgn-JP > sgn-jsl
sgn-MX > sgn-mfs
sgn-NI > sgn-ncs
sgn-NL > sgn-dse
sgn-NO > sgn-nsl
sgn-PT > sgn-psr
sgn-SE > sgn-swl
sgn-US > sgn-ase
sgn-ZA > sgn-sfs

No change is made to the Description field for these tags, so (for 
example) "sgn-US" will still have the description "American Sign 
Language".  This is to avoid unintended changes of scope.

5.  Availability

A prototype 4646bis Registry built using these rules, with an arbitrary 
File-Date (and Added date for all new subtags) of 2007-01-01, is 
available at:

http://users.adelphia.net/~dewell/lstreg-4646bis.txt    (640,265 bytes)
http://users.adelphia.net/~dewell/lstreg-4646bis.zip    (78,943 bytes, 
PKZip format)

Please use these files as necessary to evaluate the process.  Please do 
not distribute them or build anything for release based on them; 
remember that the rules are not final.  They will not be changed until a 
revised version of this process is posted to LTRU, so please download 
them only once per revision.

--
Doug Ewell
Fullerton, California, USA
http://users.adelphia.net/~dewell/
RFC 4645  *  UTN #14

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 17 Sep 2006 13:22:49 +0200

Doug Ewell wrote:

> RFC 4645bis will include a reference (probably normative) to
> RFC 4645 for those who wish to trace the path all the way
> back to the ISO and UN standards and RFC 3066 tag registry.

Okay, we have already a reference to RFC 4645 in 4646bis-01pre,
but you need it also.

> The new inputs to the Registry are the ISO/FDIS 639-3 online
> code tables.

For the drafts before Last Call.  For the registry it should be
the then existing ISO 639-3:2007.

> the subtag "gd" currently has the descriptions "Gaelic" and
> "Scottish Gaelic"; the new description "Gaelic (Scots)" from
> 639-3 will be added.

As Randy and Debbie wanted it.  You had convinced me that
replacing xxxx by xxxx (yyyy) instead of adding it might be
okay, but for a first draft any consistent rule goes, we know
that it's an open issue.

> No existing Description fields are changed or deleted.

ACK, that's KISS, please note it explicitly.

> extended language subtags if they are (with the macrolanguage
> as Prefix).  This is consistent with John Cowan's description
> of the process, which was generally approved by the list.

It's another open issue, compare
<http://permalink.gmane.org/gmane.ietf.ltru/5420>
For the first draft it's fine, let's look at the output, and
then decide if we really like this "macrolanguage" cruft.  The
job of 4646bis is to define language tags, not to sanction the
POVs of its sources, for that folks should read these sources,
not the registry.

> zh-hakka   -> zh-hak

Resulting in i-hak -> zh-hakka -> zh-hak (JFTR, we have to eat
our dog food)

> zh-min     -> (no Preferred-Value, just deprecated)

That has to be justified explicitly in prose, as an exception.

> http://users.adelphia.net/~dewell/lstreg-4646bis.txt

Thanks, quick check with the W3C validator (8167 records):
One known yi-latn (lower case L) error, anything else okay.

My old registry validator does not yet support extlang XREFs,
otherwise it's happy:
| 8167 (sub)tags in 41651 lines, 804 xrefs okay

Frank

---

From: Martin Duerst <[email-masked]>
Date: Sun, 17 Sep 2006 16:29:34 +0900

Hello Doug,

Many thanks for your work.

As a co-chair, I strongly suggest that you submit your text below
and your data as an Internet-Draft as soon as possible. This is
NOT because I want us to meet the deadline in September, but
because this is the way work is done in the IETF. Also, I'd
like to see how well the Internet-Drafts system takes this
(almost) mega-draft.

As a filename, my suggestion is draft-ietf-ltru-4645bis-00.txt,
but any other suggestion is welcome. The main problem with the
above name is that it may be too close to draft-ietf-ltru-4646bis-00.txt.

Some comments as a technical contributor below.

At 14:29 06/09/17, Doug Ewell wrote:
>Here is my first pass at describing the process for generating the RFC 4646bis Language Subtag Registry.  This is not necessarily the exact wording that will appear in RFC 4645bis, but parts of it may end up there with little or no change.
>
>* the way the process is described (logic, clarity, ambiguity, mechanics, etc.)
>
>1.  Starting point
>
>The existing Registry serves as the starting point for the new Registry. RFC 4645bis will include a reference (probably normative)

No, this should be informative, because there is no need to go back
to RFC 4645 unless for informational purposes.

>to RFC 4645 for those who wish to trace the path all the way back to the ISO and UN standards and RFC 3066 tag registry.
>
>The new inputs to the Registry are the ISO/FDIS 639-3 online code tables.  This includes the code set dated 2006-04-21 and the macrolanguage mappings dated 2005-06-14, which are currently (2006-09-16) available on the ISO 639-3 home page.  Peter has said there are changes in the FDIS document not reflected in these files, and more changes will be coming, but I am using these files because they are the most up-to-date information available to me.

Great.

>The existing Registry includes 489 primary language subtags, including some for "language collections" not included in 639-3.  In cases where the reference name(s) in 639-3 for these languages differ(s) from the Description fields in the Registry, the new names are added to the existing Registry entry, after all existing descriptions.  For example, the subtag "gd" currently has the descriptions "Gaelic" and "Scottish Gaelic"; the new description "Gaelic (Scots)" from 639-3 will be added. This includes inverted forms of existing names, such as "Frisian, Northern".
>
>This also includes language names that include a country name, to differentiate them from other languages of the same name associated with a different country.  For example, the subtag "bas" currently has the description "Basa"; to this would be added "Basa (Cameroon)" from 639-3. The new subtag "bzw" would have the single description "Basa (Nigeria)".
>
>Reference names in the 639-3 files that contain the strings "(generic)" or "(specific)" are changed to "(macrolanguage)" and the empty string, respectively, in accordance with Peter's statement on September 12 that the FDIS already includes this change.  The single exception is "Zande (specific)" (zne), which is left unchanged because there is also a 639-3 code element "Zande" (znd) and neither of these is a macrolanguage.

Can we expect this to be fixed (i.e. the two Zande to be disambiguated)
in a newer version of 639-3? Peter?

>All 639-3 code elements not in the LSR are added as primary language subtags if they are not included under a 639-3 macrolanguage, or as extended language subtags if they are (with the macrolanguage as Prefix).  This is consistent with John Cowan's description of the process, which was generally approved by the list.
>
>As a special case, all 639-3 languages with the words "Sign Language" in their name are added as extlangs, with "sgn" as the Prefix.  This is the same as if 639-3 had listed them under a macrolanguage "sgn".  (The entry "sgn", being a collection code, is not present in 639-3 at all.)
>
>All subtags added to the Registry are added in alphabetical order within their type, with the 2-letter language subtags appearing before the 3-letter subtags.  IANA previously added two 3-letter subtags, "anp" and "frr", in the section for 2-letter subtags, which is allowed by RFC 4646 but may surprise some readers.  No attempt is made here to change this.

I think that if we want things in a certain order, we should put them
into that order, and say so in 4646bis.

>As a consequence of adding these new subtags, the following grandfathered tags are deprecated, with the indicated tag specified as Preferred-Value:
>
>i-ami      -> ami
>i-bnn      -> bnn
>i-pwn      -> pwn
>i-tao      -> tao
>i-tay      -> tay
>i-tsu      -> tsu
>sgn-CH-de  -> sgn-sgg
>zh-hakka   -> zh-hak
>Each of these grandfathered tags has a Comment field indicating the ISO 639-3 code element (for example, "replaced by ISO code ami").  This duplicates the Preferred-Value information and is not strictly necessary, but is added for consistency with other grandfathered tags. This comment is also added to the tag "zh-guoyu", which is already deprecated in favor of another grandfathered tag, "zh-cmn" (which will now be redundant; see below).
>
>The following grandfathered tags are now fully composable and are moved to redundant:
>
>zh-cmn
>zh-cmn-Hans
>zh-cmn-Hant
>zh-gan
>zh-wuu
>zh-yue
>
>The following redundant sign-language tags are deprecated (requiring a change in the RFC 4646 rules, which currently disallow this).
>
>sgn-BR -> sgn-bzs
>sgn-CO > sgn-csn
>sgn-DE > sgn-gsg
>sgn-DK > sgn-dsl
>sgn-ES > sgn-ssp
>sgn-FR > sgn-fsl
>sgn-GB > sgn-bfi
>sgn-GR > sgn-gss
>sgn-IE > sgn-isg
>sgn-IT > sgn-ise
>sgn-JP > sgn-jsl
>sgn-MX > sgn-mfs
>sgn-NI > sgn-ncs
>sgn-NL > sgn-dse
>sgn-NO > sgn-nsl
>sgn-PT > sgn-psr
>sgn-SE > sgn-swl
>sgn-US > sgn-ase
>sgn-ZA > sgn-sfs
>
>No change is made to the Description field for these tags, so (for example) "sgn-US" will still have the description "American Sign Language".  This is to avoid unintended changes of scope.

Great.

>5.  Availability
>
>A prototype 4646bis Registry built using these rules, with an arbitrary File-Date (and Added date for all new subtags) of 2007-01-01, is available at:
>
>http://users.adelphia.net/~dewell/lstreg-4646bis.zip   (78,943 bytes, PKZip format)
>
>Please use these files as necessary to evaluate the process.  Please do not distribute them or build anything for release based on them; remember that the rules are not final.  They will not be changed until a revised version of this process is posted to LTRU, so please download them only once per revision.

Thanks again.   Regards,   Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 17 Sep 2006 22:00:38 -0700

Martin Duerst <duerst at it dot aoyama dot ac dot jp> wrote:

> As a co-chair, I strongly suggest that you submit your text below and 
> your data as an Internet-Draft as soon as possible. This is NOT 
> because I want us to meet the deadline in September, but because this 
> is the way work is done in the IETF. Also, I'd like to see how well 
> the Internet-Drafts system takes this (almost) mega-draft.

I shall get to it, then.