Subject: [Ltru] ISO/FDIS 639-3 tables revised

From: "Doug Ewell" <[email-masked]>
Date: Mon, 20 Nov 2006 00:17:47 -0800

The downloadable ISO/FDIS 639-3 tables have been substantially modified 
as of November 14, and they now contain additional information that will 
allow us to make critical decisions related to alternative and inverted 
language names.

http://www.sil.org/iso639-3/download.asp

The basic "code set" file has the same general format as before, but the 
"Name" or "Inverted_Name" field has been redefined as "Reference_Name". 
This appears to be the non-inverted form of one of the names associated 
with the language, often the most common (e.g. Spanish is listed instead 
of Castillian, Dutch instead of Flemish).  No commas or semicolons are 
present in this file.  An example is:

    crh     crh     crh             I       L       Crimean Tatar

A new 'language names index" file has been added, which lists all of the 
names associated with the language, in both non-inverted and inverted 
forms when present (two identical forms are given when there is no 
inversion).  The file has one record for each distinct name, and does 
not use semicolons to separate multiple names.  Commas are present only 
at the inversion points.  An example is:

    crh     Crimean Tatar   Tatar, Crimean
    crh     Crimean Turkish Turkish, Crimean

The macrolanguage file is essentially the same as before; only the field 
names have been changed.

A "retirements" file has been added to track changes made in the FDIS. 
This include one entry, the change from "fri" to "fry" that reflects the 
ISO 639-2 mapping.

We in LTRU now have all the information related to ISO 639-3 language 
names that we need, and we need to make a decision about which names 
should appear in the RFC 4645bis Registry.  Should we use:

1a.  only the reference names shown in the main file?
1b.  the reference and alternative names, but only in non-inverted form 
(when both exist)?
1c.  the reference and alternative names, but only in inverted form 
(when both exist)?
1d.  the reference and alternative names, in both inverted and 
non-inverted form?

My preference is for option 1d, including all the names.

We also need to decide whether to apply the decision in question 1 above 
to:

2a.  all primary and extended language subtags in the Registry, possibly 
adding more names to existing subtags, or
2b.  only the new subtags added with ISO 639-3.

My preference is for option 2a, retrofitting new names onto existing 
subtags where ISO 639-3 lists them.

Let's discuss our options here, make a decision, and incorporate it into 
the next draft of RFC 4646bis so I can reflect it in a new draft of RFC 
4645bis.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 20 Nov 2006 08:32:44 -0800

I looked over these table this morning, and all I can say is: *fabulous*!
The 639-3 people have done a wonderful job with these tables, and very
thorough work on the names. They have done such a good job that I don't see
any barrier to using the information as is. The only real question is how
best to leverage it. Here is my take:

Fields to add:

   1. Three Letter Code: it would indeed be handy to have the 3 letter
   code around, for those cases where our canonical subtag is 2 letter. While
   that is somewhat peripheral to our main function, it would be useful to have
   this for both the language subtag and the region subtag to help in
   identification, perhaps as "Type-Equivalent"?
   2. Inverted Names: I suggest we add an additional field for these,
   allowing more than one. Maybe "Inverted-Description"? We could change any of
   the current descriptions that are inverted to Inverted-Description, and add
   add the 639-3 ones (1403 of them).
   3. Type: A(ncient), C(onstructed), E(xtinct), H(istorical), L(iving),
   S(pecial). This would be very useful information for identification, and I
   suggest we add it as a Language-Type field

Fields that our current structure can accommodate:

   1. Scope: we don't need to add this, since it can be derived:
   something is a macro language iff there is an extlang that points to it.
   2. Reference Name: If we have the convention that the first
   Description is the reference name, then we could incorporate all of this
   information without changing our structure. Note: 636-3 only has 85 cases of
   multiple names.
   3. Other Names: we can add any alternate names as additional
   descriptions.

Mark

---

From: John Cowan <[email-masked]>
Date: Mon, 20 Nov 2006 12:14:06 -0500

Mark Davis scripsit:

>   code around, for those cases where our canonical subtag is 2 letter.
>   While that is somewhat peripheral to our main function, it would
>   be useful to have this for both the language subtag and the region
>   subtag to help in identification, perhaps as "Type-Equivalent"?

I would be opposed to this: it might easily give the impression that
the use of these forms is licit.  ISO 3166 and ISO 639-3 already
provide such mappings.

>   2. Inverted Names: [... and] 3. Type: A(ncient), C(onstructed),
>   E(xtinct), H(istorical), L(iving), S(pecial). 

I'm fairly neutral about these.

>   something is a macro language iff there is an extlang that points

That's true now, but won't always be true.  If languages 'xxx' and 'yyy'
are encompassed in a newly created macrolanguage 'zzz', then we will not
start using 'zzz-xxx' and 'zzz-yyy', at least according to current plans.

That said, I'm not sure how much having this information
machine-processable adds.  In cases where the names are confusing,
639-3 is adding "(macrolanguage)" anyhow.

>   Description is the reference name,

I am very much opposed to any convention that assigns special semantics
to the first (or any other) instance of a field.  If you want
distinct semantics, use a distinct field name.

-- 
I marvel at the creature: so secret and         John Cowan
so sly as he is, to come sporting in the pool   [email-masked]
before our very window.  Does he think that     http://www.ccil.org/~cowan
Men sleep without watch all night?

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 20 Nov 2006 19:10:02 +0100

> the use of these forms is licit.  ISO 3166 and ISO 639-3 already

+1  My intuitive reaction was "maybe as Comment: Do not use 'abc'".  

> I'm fairly neutral about these.

I like the "inverted names".  If the types are listed let's at least
get rid of "living", declaring this as default.

> I am very much opposed to any convention that assigns special 
> semantics to the first (or any other) instance of a field.  If
> you want distinct semantics, use a distinct field name.

IMO Doug can list the descriptions in "source order", and it will 
automagically do what Mark wants.  Inventing more field names isn't
attractive, maybe the type deserves it.

Frank

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 20 Nov 2006 12:04:22 -0800

Hi -

As a technical contributor...

> From: "Doug Ewell" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
> Sent: Monday, November 20, 2006 12:17 AM
> Subject: [Ltru] ISO/FDIS 639-3 tables revised
...
> My preference is for option 1d, including all the names.

I prefer option 1d, too.

> subtags where ISO 639-3 lists them.
...

Although 2b might be less work, I don't object to 2a.

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 20 Nov 2006 12:16:40 -0800

Hi -

> From: "Mark Davis" <[email-masked]>
> To: "Doug Ewell" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Monday, November 20, 2006 8:32 AM
> Subject: Re: [Ltru] ISO/FDIS 639-3 tables revised
...
>    suggest we add it as a Language-Type field

As a co-chair, one of the things I'm supposed to watch out for is feature-creep.

Consequently, before going any further in discussing these three proposed fields,
I'd like to see a clear statement of why these are necessary to meeting the
requirements established in the ltru WG charter.  If they're not necessary, then
the cost/benefit ratio needs to be *extremely* persuasive, and the benefits still
need to be in terms of supporting language tagging.

>    information without changing our structure. Note: 636-3 only has 85 cases of

As a technical contributor, I think that giving semantics to field order
would be unwise.  If we need special semantics for this (I'm not convinced
that we do) then we should introduce an additional field for the other
names.

>    3. Other Names: we can add any alternate names as additional

This seems adequate.

Randy

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 20 Nov 2006 13:59:18 -0800

On 11/20/06, Randy Presuhn <[email-masked]> wrote:
> While

These are directly related to our charter, which is the addition of the
639-3 codes, since this is relevant data coming directly from 636-3. The
scope of the change, adding 7000 new codes -- means that we need to consider
any information that is useful for people to distinguish them.

Now, as pointed out, we could encode this as specialized-format comments:

Comment: Do not use three-letter code: abc
Comment: 639-3 Type = Ancient

While these are possible, this is kinda hacky, since we are overloading the
comment mechanism. It just makes a bit gummier for anyone needing to machine
parse the data, so I think it would be better to have explicit field types
for these.

> that we do) then we should introduce an additional field for the other

I think that 639-3 has made a significant advance in distinguishing a
reference name, and it would be a great advantage for us to be able to
distinguish it in some way. By far the simplest way to do this is to use the
first Description.

---

From: [email-masked]
Date: Mon, 20 Nov 2006 15:33:15 -0800

I find the inverted names very useful in addition to the reference names 
and alternate names (synonyms). When inverted names are used in 
classification interfaces, it helps promote data quality -- the user sees 
all available variants together before making a choice. I don't think LTRU 
should be in the business of deciding exactly how a UI should be 
implemented as was previously being discussed, but the inverted names are 
common and if already found in 639-3, I would certainly include them. 
However, they are most useful when specifically identified as inverted 
names.

It seems the label "description" is less than descriptive given the 
expanding requirements (it wouldn't meet ISO 11179 element-naming best 
practices).  I'd throw out that terminology altogether in favor of 
something like:

referencelangname
alternatelangname
invertedlangname

So I agree with those who feel that different semantics require different 
fields and don't think that ordering should add to the semantics of a 
field.

You might want more specific terminology for "language type" as well. If 
you look beyond RFC 4646, "language type" is also ambiguous. I might have 
"language usage type" = subtitles, dubbed, EAD, closed captions ... as 
well as "language mode type" = spoken, written, signed, embossed; etc. 
Will future versions of RFC 4646 have similar expanded needs?

I'm failing to come up with a better name, however, as the language types 
listed seem a little bit like apples and oranges. Most of the "language 
type" categories below deal with the current demographics of the language, 
but then "constructed" and "special" don't quite fit that definition. 
Hmm...

Regards,

Karen Broome
Metadata Systems Designer
Sony Pictures Entertainment
310.244.4384

"Mark Davis" <[email-masked]> wrote on 11/20/2006 01:59:18 PM:

> > To: "Doug Ewell" <[email-masked] >
> These are directly related to our charter, which is the addition of 
> the 639-3 codes, since this is relevant data coming directly from 
> 636-3. The scope of the change, adding 7000 new codes -- means that 
> we need to consider any information that is useful for people to 
> distinguish them. 
> 
> While these are possible, this is kinda hacky, since we are 
> overloading the comment mechanism. It just makes a bit gummier for 
> anyone needing to machine parse the data, so I think it would be 
> better to have explicit field types for these. 
> 
> I think that 639-3 has made a significant advance in distinguishing 
> a reference name, and it would be a great advantage for us to be 
> able to distinguish it in some way. By far the simplest way to do 
> this is to use the first Description. 
>

---

From: Martin Duerst <[email-masked]>
Date: Tue, 21 Nov 2006 13:33:07 +0900

[co-chair hat off]

>the use of these forms is licit.  ISO 3166 and ISO 639-3 already

I agree with John here. Mark says that this can be put into a comment,
and that it's helpful for identifying a language. As I understand,
the three-letter codes would be added to two-letter code entries.
I don't think there is too much of a problem identifying two-letter
code entries anyway, so I don't think this is necessary.

>distinct semantics, use a distinct field name.

I agree with John here, too.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 21 Nov 2006 09:43:36 -0000

I like your thoughts, Karen.  I think there should be a discussion wrt
splitting "Description" into a number of more descriptive fields.  I also
think it would be a good idea if, in doing so, this WG considered at least
the basic requirements for conformity to ISO 11179  that the Registry
structure could reflect very easily without necessarily adopting ISO 11179
at this point in time. 
 
Best Regards
 
Debbie

  _____  

From: [email-masked] [mailto:[email-masked]] 
Sent: 20 November 2006 23:33
To: Mark Davis
Cc: LTRU Working Group
Subject: Re: [Ltru] ISO/FDIS 639-3 tables revised

I find the inverted names very useful in addition to the reference names and
alternate names (synonyms). When inverted names are used in classification
interfaces, it helps promote data quality -- the user sees all available
variants together before making a choice. I don't think LTRU should be in
the business of deciding exactly how a UI should be implemented as was
previously being discussed, but the inverted names are common and if already
found in 639-3, I would certainly include them. However, they are most
useful when specifically identified as inverted names. 

It seems the label "description" is less than descriptive given the
expanding requirements (it wouldn't meet ISO 11179 element-naming best
practices).  I'd throw out that terminology altogether in favor of something
like: 

referencelangname 
alternatelangname 
invertedlangname 

So I agree with those who feel that different semantics require different
fields and don't think that ordering should add to the semantics of a field.

You might want more specific terminology for "language type" as well. If you
look beyond RFC 4646, "language type" is also ambiguous. I might have
"language usage type" = subtitles, dubbed, EAD, closed captions ... as well
as "language mode type" = spoken, written, signed, embossed; etc.  Will
future versions of RFC 4646 have similar expanded needs? 

Regards, 

Karen Broome

"Mark Davis" <[email-masked]> wrote on 11/20/2006 01:59:18 PM:

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 21 Nov 2006 06:43:40 -0800

Mark Davis wrote:

> Fields to add:
>
> Three Letter Code: it would indeed be handy to have the 3 letter code 
> around, for those cases where our canonical subtag is 2 letter. While 
> that is somewhat peripheral to our main function, it would be useful 
> to have this for both the language subtag and the region subtag to 
> help in identification, perhaps as "Type-Equivalent"?

I *strongly* disagree, for the reason John stated: it would send users a 
clear message that "fra" is an acceptable alternative for "fr" in RFC 
4646-family language tags.  This is much worse than simply being "not 
relevant" to language tagging.  No, no, a thousand times no.

> Inverted Names: I suggest we add an additional field for these, 
> allowing more than one. Maybe "Inverted-Description"? We could change 
> any of the current descriptions that are inverted to 
> Inverted-Description, and add add the 639-3 ones (1403 of them).

Inverted names in language subtags are easy to identify already; they 
are the only ones that contain a comma.  (ISO 639 and 3166 don't use the 
comma to separate alternative names, as ISO 15924 and some of our 
grandfathered tags do.)  So I don't see a need to label them specially; 
we could just include them, or we could include *only* the inverted 
forms and it would be trivial for users to uninvert them (but I prefer 
the former).

> Type: A(ncient), C(onstructed), E(xtinct), H(istorical), L(iving), 
> S(pecial). This would be very useful information for identification, 
> and I suggest we add it as a Language-Type field

I do support this, and supported it back when John proposed it in early 
October.  Unlike some other ideas for new fields, this one directly 
assists users in determining the appropriateness of a subtag.  I agree 
with Frank that "Living" should be the default, and should not be 
explicitly indicated.

This field should also be able to hold the value "Collection" for those 
collection codes included in 639-2 but not 639-3.  Draft-4646bis-01 
recommends against using these (Section 4.1) but they cannot always be 
identified without this field.

> Scope: we don't need to add this, since it can be derived: something 
> is a macro language iff there is an extlang that points to it.

I agree.  John is right that the concept of "ISO 639-3 macrolanguage" 
and "RFC 4646bis primary language subtag that has extended language 
subtags under it" may diverge in the future, but I'm not sure that's 
relevant for our purposes.

> Reference Name: If we have the convention that the first Description 
> is the reference name, then we could incorporate all of this 
> information without changing our structure. Note: 636-3 only has 85 
> cases of multiple names.

I don't think we need to incorporate it.  As long as one of the 
Descriptions matches an ISO 639-3 reference name, that is enough.  I've 
already fought and lost this particular battle.

Let's put it this way:  Since the order of Description fields doesn't 
matter, I do plan to put them (in RFC 4645bis) in roughly the order in 
which they appear in ISO 639-3: reference name first, then other names, 
then inverted forms.  But we cannot count on that being relevant, or on 
tag consumeres treating the first Description as special in any way.

> Other Names: we can add any alternate names as additional 
> descriptions.

Yes, including any names proposed by individuals that ISO 639 left off 
for whatever reason.

The inclusion of the full list of names, inverted and non-inverted, in a 
downloadable ISO 639-3 text file is a major step forward.  By including 
all of these names in the Registry, we can be assured of fulfilling the 
oft-stated requirement to match the Description field with the 
"official" ISO name.

---

From: Peter Constable <[email-masked]>
Date: Tue, 21 Nov 2006 20:41:04 -0800

From: Doug Ewell [mailto:[email-masked]] 

> This field should also be able to hold the value 
> "Collection" 

In terms of the attributes on entries defined in ISO 639-3, this would
be mixing values for two distinct attributes: type (living, etc.) and
scope (individual language, macrolanguage, collection). The type
attribute applied to collections would more-appropriately have values
such as "genetic" or "geographic" according to how the collection is
defined.

> I agree. 

Well, note that right now you don't have a certain way to tell which
entities have a collection scope.

Peter Constable