Subject: [Ltru] Re: 3066ter encompassed languages

From: "Doug Ewell" <[email-masked]>
Date: Fri, 8 Sep 2006 22:08:20 -0700

John Cowan <cowan at ccil dot org> wrote:

> It looks like these rules will do it:
>
> 1) If a 639-3 code element is already in the registry, either as 
> itself or in the form of its 639-1 equivalent, do nothing.
>
> 2) If a 639-3 code element represents an unencompassed individual 
> language (or a macrolanguage, but fortunately there are no such 
> cases), add it to the registry as a language subtag.
>
> 3) If a 639-3 code element represents an individual language 
> encompassed by a macrolanguage, add it to the registry as an extlang 
> subtag, and specify the required prefix to be the language subtag in 
> place for that macrolanguage.  For this purpose, all sign languages 
> are treated as if encompassed by the macrolanguage "sgn", which is 
> part of 639-2 and the registry but is not in 639-3.

This looks good to me.

>> 1.  what to do about the Description fields when the 639-2 and 639-3 
>> descriptions don't match exactly
>
> Keep both.  The 639-3 descriptions have been reviewed more recently 
> and are more likely to be correct, especially with regard to 
> diacritics; the 639-2 descriptions should be effectively 
> grandfathered.

I also assumed the 639-3 descriptions would be more likely to be 
correct.  Since others have argued that the order of descriptions 
doesn't matter, however, we should add any new descriptions after the 
existing ones.

>> 2.  what to do about sign languages
>
> As above: treat them as encompassed by the macrolanguage "sgn", and 
> therefore add them as extlang subtags.  Deprecate the grandfathered 
> equivalents.

OK.

>> 3.  which grandfathered tags to deprecate in favor of new 639-3-based 
>> subtags (John tackled this in msg05281.html but I'm not sure about 
>> some of his conclusions)
>
> Start picking nits, then.

See my previous mail.

--
Doug Ewell
Fullerton, California, USA
http://users.adelphia.net/~dewell/
RFC 4645  •  UTN #14

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Fri, 8 Sep 2006 22:45:28 -0700

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

>> I hope we don't touch it.  We went to some lengths to build a syntax 
>> that would also work for 3066ter.
>
> I'd make an exception if <extlang> turns out to be unnecessary.

I don't see that happening.  John has proposed rules for assigning 
primary and extended language tags to unencompassed and encompassed 
languages respectively, and nobody else has seriously objected to this.

>> Scenario:  We have "Church Slavic" plus 4 other variants. 639-3 adds 
>> an inverted form, "Slavic, Church".  I assume we add it; first or 
>> last?
>
> The source for Church Slavic was 639-2.  I'd stick to it as is in the 
> registry, don't add it.

What if the descriptions are noticeably different between 639-2 and 
639-3?  Where do we draw the line without being arbitrary?

>> Scenario:  We have "Malay".  639-3 changes this to "Malay (specific)"

(this was an error, should have been "Malay (generic)")

>> because there is also "Malay (specific)" plus a host of others such 
>> as "Malay, Ambonese".
>
> If we can also get away with keeping the 639-2 Malay as is... adding a 
> comment with the more specific name is an option.

My opinion is that that will add a lot of confusion and obfuscate the 
relationship between "generic" Malay and its flavors.

> For the future (not the initial bulk update) the review list could 
> figure out what's best.  E.g. add it if really needed.

I'd recommend adding it, but you're right: this is an ietf-languages 
question.

> For the bulk update:  How many conflicts are there ?

I count approximately 16, not including inverted forms like "Newari, 
Classical" which it has been suggested we ignore.

>> I assume we REPLACE the existing description with the one from 639-3.
>
> I don't like that, so far the description always mirrors the source. 
> "No creative corrections of dubious apostrophes" was an example.

Which source do we mirror, if one says "Waray" and the other says "Waray 
(Philippines)"?  Which is better for avoiding confusion with "Waray 
(Australia)"?

>> Scenario:  We have "Ainu".  639-3 changes this to "Ainu (Japan)" to 
>> differentiate it from the unrelated "Ainu (China)".  Same outcome as 
>> above
>
> Yes, 639-3 shouldn't change 639-2 indirectly via the registry.

In this case 639-3 is not "changing" 639-2, but elaborating on it, 
adding information which was not present before (partly because "aib" 
was not in 639-2).

>> we should probably point out in the registry update draft that these 
>> parenthesized country names are there to disambiguate colliding 
>> language names and not to take the place of region subtags.
>
> ACK.

Bill the Cat?

I'm in favor of adding the parenthesized country names introduced in 
639-3, when they serve to distinguish two languages with otherwise 
identical names.

(subsequent examples of this issue elided; readers know when Frank and I 
stand)

>> We also need to decree that "sgn-US" et al. will be deprecated in 
>> favor of "sgn-ase" et al.
>
> I hope these cases are obvious for you or somebody else here, they're 
> not for me.

Both the grandfathered tag "sgn-US" and the 639-3 code elements "ase" 
refer to American Sign Language.

>> john suggested deprecating "i-enochian" and "i-mingo" but didn't have 
>> anything to replace them with.
>
> Dubious idea under 4646-rules, there's some MUSTard saying that 
> "Preferred-Value" can't be added after deprecation.  I see no 
> compelling reason to change that rule.

John withdrew this.

> For a first impression (and draft) you can take any consistent rules, 
> we then look in the output, especially into all changes wrt existing 
> tags, and if it's dubious tune the rules until the output passes 
> giggle tests (not the "Church, Slavic" example).

That's what we'll end up doing, and it will be fine as long as the rules 
are consistent and not arbitrary or contrived.  The only really 
exceptional situation that I see is "sgn", and I don't see any 
difficulty justifying this when you consider the exceptional nature of 
sign languages.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 09 Sep 2006 12:17:22 +0200

Doug Ewell wrote:
 
> John has proposed rules for assigning primary and extended
> language tags to unencompassed and encompassed languages
> respectively, and nobody else has seriously objected to this.

Yes, those were good rules.  But if 639-anything is guaranteed
to be unique we don't need the extlang construct at all.  

>> The source for Church Slavic was 639-2.  I'd stick to it as
>> is in the registry, don't add it.
 
> What if the descriptions are noticeably different between
> 639-2 and 639-3?  Where do we draw the line without being
> arbitrary?

The line is "whatever 639-2 says" for a tag derived from 639-2.

In that case don't bother to look at 639-3, unless it could be
a good comment to be registered individually under 4646 rules.

 {Malay vs. Malay (generic)]
> My opinion is that that will add a lot of confusion and
> obfuscate the relationship between "generic" Malay and its
> flavors.

I'm not confused, Malay 639-2 is what I want.  And it's fine
if 639-3 Malay (Ambonese) shows up next alphabetically.

> I count approximately 16, not including inverted forms like
> "Newari, Classical" which it has been suggested we ignore.

Good, so a general rule "pick whatever 639-2 say, after that
add anything found in 639-3 not already covered by the first
step" could work.

> Which source do we mirror, if one says "Waray" and the other
> says "Waray (Philippines)"?

639-2.

> Which is better for avoiding confusion with "Waray
> (Australia)"?

639-3 <g>.  Propose a rule how to smuggle Waray (Philippines)
into the 639-2 entry.  "Comment" is my first idea.  Or "let's
see how they sort it out" - Peter said that will be done in
time.

> Bill the Cat?
u+0006 in the US-ASCII repertoire :-)

> I'm in favor of adding the parenthesized country names
> introduced in 639-3, when they serve to distinguish two
> languages with otherwise identical names.

If it has an obvious pattern we can create a rule for it.

Frank

---

From: "Kent Karlsson" <[email-masked]>
Date: Sat, 9 Sep 2006 13:07:51 +0200

> relationship between "generic" Malay and its flavors.

The following is not really an issue for ltru, nor ietf-languages,
but for 639-3.

I'm not too keen on the "(specific)" and "(generic)" parts of those
descriptions.

We already have 'tai': Tai [languages] (other), and 'th'/'tha': Thai. That
is an almost parallel case. Almost because of 1) the "other", and
2) the slightly different spelling th/t. But apart from that, and given
that (almost) parallel, I think that "Malay (generic)" would be better
described as "Malay languages" and "Malay (specific)" would be
better described as just "Malay", or even better use one or more
of the alternate names given in the Ethnologue, like Standard Malay
or Bahasa Malayu.

I find that the 639-3 website does not give a list of languages for
'tai', but it does so for "msa". Why that is so? 'tai' should encompass
all the Tai languages except Thai and Lao, due to the "other" (it is
only "other" w.r.t. 639-2, not w.r.t. 639-3).

		/kent k

---

From: John Cowan <[email-masked]>
Date: Sat, 9 Sep 2006 11:53:29 -0400

Frank Ellermann scripsit:

> But if 639-anything is guaranteed to be unique we don't need the
> extlang construct at all.

The purpose of extlangs is not for uniqueness (639-anything code elements
*are* guaranteed unique) but backward compatibility and ease in matching.

For example, 639-1 (and a fortiori 4646) has "ar", meaning simply
"Arabic", a term encompassing some 30 distinct languages.  Rather than
requiring people in search of any sort of Arabic to search for 30
distinct tags, or requiring matchers to keep a table mapping "ar" to
all those tags, we instead require taggers specifying (say) Algerian
Arabic to use "ar-aao" instead of just "aao".  That way, searches for
"ar" will automatically match.

We have done this for Chinese already, and we attempted to do it for
Norwegian as well, but in the latter case (as well as for Serbo-Croatian),
ISO 639-1/RA took things out of our hands by creating separate code
elements.  So people just have to know that "no" encompasses "nb" and "nn"
and that "sh" encompasses "sr", "hr", and "bs".  We want to avoid
having to deal with any more such tacit knowledge.

-- 
Dream projects long deferred             John Cowan <[email-masked]>
usually bite the wax tadpole.            http://www.ccil.org/~cowan
        --James Lileks

---

From: John Cowan <[email-masked]>
Date: Sat, 9 Sep 2006 12:52:22 -0400

Kent Karlsson scripsit:

> 2) the slightly different spelling th/t. 

Unfortunately, that spelling difference is fundamental.  The difference
between "th" and "t" (aspirated vs. not aspirated) is not very apparent to
Germanic-speaking ears; it is the difference between the /t/ in English
"tie" (which is pronounced like "Thai") and English "sty" (which, if
you excised the /s/ sound, would sound like "Tai").

Furthermore, "Tai" is the name of a genetic subtree of languages, whereas
"Thai" is the name of a specific language.  More on that below.

> only "other" w.r.t. 639-2, not w.r.t. 639-3).

That requires some explanation.  In ISO 639-2, there are two kinds of
code elements; those for individual languages such as "eng" and those for
language collectives such as "tai" or "nai" (North American languages).

The intention was to provide *some* code element for every language,
even if not a unique one.  Thus documents in all Tai languages other
than Thai and Lao could be tagged "tai".  However, this was unstable,
because if an additional Tai language were registered by 639-2/RA,
the denotation of "tai" would technically change to exclude it, possibly
invalidating existing data.  Furthermore, the definition of a Tai language
is dependent on scholarship, and may change over time to include more
or fewer languages.

639-3 is a registry of individual languages only, and does not include
any collective language tags -- ISO 639-5 will contain a selection of
such tags, including all those in 639-2, and ISO 639-6 will contain more.
However, it was found that in many cases individual 639-2 languages mapped
to more than one 639-3 language.  These 639-2 languages were therefore
added to 639-3 as macrolanguages, with explicit and permanent mappings
to the corresponding 639-3 individual languages.

It is these mappings that we will use to establish which 639-3 individual
languages will be registered as language subtags and which as extlang
subtags.

-- 
As we all know, civil libertarians are not      John Cowan
the friskiest group around -- comes from        [email-masked]
forever being on the qui vive for the sound     http://www.ccil.org/~cowan
of jack-booted fascism coming down the pike.           --Molly Ivins

---

From: "Kent Karlsson" <[email-masked]>
Date: Sat, 9 Sep 2006 19:38:33 +0200

> "Thai" is the name of a specific language.  More on that below.

Let me try again, since you managed to completely miss my point:

I think it would be much better (for 639-3) to have descriptions like

	msa: Malay languages
	mly: Standard Malay, Bahasa Malayu

or even

	msa: Malay languages
	mly: Malay

rather than the very artificial

	msa: Malay (generic)
	mly: Malay (specific)

> languages will be registered as language subtags and which as extlang

But the IANA LSTR will not the same as 639-3 (for language codes).
The IANA LSTR will, for stability reasons, continue to contain
"collective" codes. We could either:

1) deprecate them ("collective" codes) all and for each of them refer to
individual language codes (from 639-3 but not in 639-2), or

2) leave them as additional "macrolanguages", giving individual language
codes (from 639-3, not any in 639-2) as "extlangs" for each.

Either way, we would need an authoritative list of the individual
language
codes each of them *currently* cover (not to be updated in future
versions,
for stability, even if would leave the lists as outdated).

I don't think it would be a good idea to leave them as is. I would
prefer
alternative 1 above, but some of them may have been used (despite that
being a bad idea), in which case alternative 2 may be better.

		/kent k

---

From: "Debbie Garside" <[email-masked]>
Date: Sat, 9 Sep 2006 18:56:45 +0100

> rather than the very artificial

In which case please suggest it to the ISO 639 JAC.  To reiterate, the
Registry should reflect the standards exactly. Alternatively, suggest an
additional description but always include the description as per the
underlying standard.

Best regards

Debbie Garside

> -----Original Message-----
> From: Kent Karlsson [mailto:[email-masked]] 
> Sent: 09 September 2006 18:39
> To: 'John Cowan'
> Cc: [email-masked]
> Subject: RE: [Ltru] Re: 3066ter encompassed languages
> 
> John Cowan wrote:
> > Kent Karlsson scripsit:
> > 
> > Unfortunately, that spelling difference is fundamental.  The 
> > difference between "th" and "t" (aspirated vs. not 
> aspirated) is not 
> > very apparent to Germanic-speaking ears; it is the 
> difference between 
> > the /t/ in English "tie" (which is pronounced like "Thai") 
> and English 
> > "sty" (which, if you excised the /s/ sound, would sound like "Tai").
> > 
> > Furthermore, "Tai" is the name of a genetic subtree of languages, 
> > whereas "Thai" is the name of a specific language.  More on that 
> > below.
> 
> > because if an additional Tai language were registered by 
> 639-2/RA, the 
> > denotation of "tai" would technically change to exclude it, 
> possibly 
> > invalidating existing data.  Furthermore, the definition of a Tai 
> > language is dependent on scholarship, and may change over time to 
> > include more or fewer languages.
> > 
> > However, it was found that in many cases individual 639-2 languages 
> > mapped to more than one 639-3 language.  These 639-2 languages were 
> > therefore added to 639-3 as macrolanguages, with explicit and 
> > permanent mappings to the corresponding 639-3 individual languages.
> > 
> The IANA LSTR will, for stability reasons, continue to 
> contain "collective" codes. We could either:
> 
> 1) deprecate them ("collective" codes) all and for each of 
> them refer to individual language codes (from 639-3 but not 
> in 639-2), or
> 
> 2) leave them as additional "macrolanguages", giving 
> individual language codes (from 639-3, not any in 639-2) as 
> "extlangs" for each.
> 
> Either way, we would need an authoritative list of the 
> individual language codes each of them *currently* cover (not 
> to be updated in future versions, for stability, even if 
> would leave the lists as outdated).
> 
> I don't think it would be a good idea to leave them as is. I 
> would prefer alternative 1 above, but some of them may have 
> been used (despite that being a bad idea), in which case 
> alternative 2 may be better.
> 
> 		/kent k
> 
>

---

From: John Cowan <[email-masked]>
Date: Sat, 9 Sep 2006 15:39:41 -0400

> 	mly: Standard Malay, Bahasa Malayu

That would be misleading.  "languages" suggests a collection of
languages, whereas msa is a macrolanguage.  Understanding this
distinction is fundamental.

> rather than the very artificial

It's an artificial situation caused by a mismatch between
639-2 and -3.

> individual language codes (from 639-3 but not in 639-2), or

That would be IMHO overkill.  Sometimes you simply don't know, or need
to know, the specific language of a document.

I would not object to a new "discouraged" status intermediate between
deprecated and encouraged.

> language codes each of them *currently* cover (not to be updated
> in future versions, for stability, even if would leave the lists
> as outdated).

No such lists exist or are likely to exist; there is not even
scholarly agreement on their contents in many cases.

-- 
But that, he realized, was a foolish            John Cowan
thought; as no one knew better than he          [email-masked]
that the Wall had no other side.                http://www.ccil.org/~cowan
        --Arthur C. Clarke, "The Wall of Darkness"

---

From: Peter Constable <[email-masked]>
Date: Sat, 9 Sep 2006 21:57:45 -0700

>rather than the very artificial

No, you have missed the point. In ISO 639, entries with names of the
form "... languages" are *collections*. The entry with the ID msa is
*not* a collection; it is a macrolanguage. 

It is listed in ISO 639-2 as an individual language, "Malay", and as
such has usage as though it denoted a single language. But in ISO 639-3
there are several individual languages that (possibly -- more below) are
encompassed by msa. One of these is the language Malay, also known as
Bahasa Malaysia or Standard Malay, which has its own ID mly. In order to
keep unique reference names for every entry, 639-3 needs some way to
distinguish "Malay" in the case of msa from "Malay" in the case of mly.
When preparing the draft code table for 639-3, most cases of
macrolanguages did not have a problem with the reference name being
identical to the reference name of some individual language, but there
were a few, one of which was Malay.

Now, a few comments are needed here:

- You are looking at a *draft* table for 639-3.

- One of the open issues that the JAC needs to make a decision on is how
the 639-2 entry should be treated in relation to entries in 639-3: is it
equated with mly (in which case mly would be removed from the code
table), or is it treated as a macrolanguage (as reflected in the draft
code table)? 

- If the JFrom [email-masked] Sun Sep 10 00:58:39 2006
Content-Class: urn:content-classes:message
Subject: RE: [Ltru] Re: 3066ter encompassed languages
Date: Sat, 9 Sep 2006 21:57:45 -0700
From: Peter Constable <[email-masked]>
To: <[email-masked]>

>rather than the very artificial

No, you have missed the point. In ISO 639, entries with names of the
form "... languages" are *collections*. The entry with the ID msa is
*not* a collection; it is a macrolanguage. 

It is listed in ISO 639-2 as an individual language, "Malay", and as
such has usage as though it denoted a single language. But in ISO 639-3
there are several individual languages that (possibly -- more below) are
encompassed by msa. One of these is the language Malay, also known as
Bahasa Malaysia or Standard Malay, which has its own ID mly. In order to
keep unique reference names for every entry, 639-3 needs some way to
distinguish "Malay" in the case of msa from "Malay" in the case of mly.
When preparing the draft code table for 639-3, most cases of
macrolanguages did not have a problem with the reference name being
identical to the reference name of some individual language, but there
were a few, one of which was Malay.

Now, a few comments are needed here:

- You are looking at a *draft* table for 639-3.

- One of the open issues that the JAC needs to make a decision on is how
the 639-2 entry should be treated in relation to entries in 639-3: is it
equated with mly (in which case mly would be removed from the code
table), or is it treated as a macrolanguage (as reflected in the draft
code table)? 

- If the JAC decides to treat msa as identical to mly resulting in the
separate entry mly being deleted, then there is no longer an issue with
two entries having otherwise identical reference names.

- If the JAC decides to treat msa as a macrolanguage, then there is
still opportunity for the reference names to be revised, and the problem
might perhaps be avoidable (e.g. by using "Standard Malay" or "Bahasa
Malay" for mly).

> individual language codes (from 639-3 but not in 639-2), or

I've always been in favour of deprecating collective IDs from use in
IETF language tags. 

> codes (from 639-3, not any in 639-2) as "extlangs" for each.

Absolutely not!

> for stability, even if would leave the lists as outdated).

That was not done as part of 639-3 because it was out of scope; it might
be in scope for 639-5, however. In the early work on 639-3, when Gary
Simons and I were preparing proposals for ISO, we did our initial
analysis of the relationship between ISO 639-2 and Ethnologue 14th
edition. That included mappings from collections in 639-2 to individual
languages, and that is available here:

http://www.ethnologue.com/14/iso639/analysis.asp#C

> being a bad idea), in which case alternative 2 may be better.

I think alternative 2 is worse than any existing use of a collective ID.

Peter Constable

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 10 Sep 2006 18:40:14 +0200

John Cowan wrote:

> For example, 639-1 (and a fortiori 4646) has "ar", meaning
> simply "Arabic", a term encompassing some 30 distinct
> languages.  Rather than requiring people in search of any
> sort of Arabic to search for 30 distinct tags, or requiring
> matchers to keep a table mapping "ar" to all those tags, we
> instead require taggers specifying (say) Algerian Arabic to
> use "ar-aao" instead of just "aao".  That way, searches for

So far okay.  And are we sure that this info has to be made
explicit in billions of tags instead of once in the registry ?

In many cases the tags don't offer this redundancy.  If I'm
looking for "de" nobody can (based alone on the registry)
automatically guess that I can also read "nds".

The opposite case is certainly irrelevant, I can't read "ar",
let alone "ar-aao".

 [no]
> in the latter case (as well as for Serbo-Croatian), ISO
> 639-1/RA took things out of our hands by creating separate
> code elements.  So people just have to know that "no"
> encompasses "nb" and "nn" and that "sh" encompasses "sr",
> "hr", and "bs".  We want to avoid having to deal with any
> more such tacit knowledge.

Knowledge is good, inconsistent redundancy is another issue.
So far nothing happened, the syntax exists, all new parsers
can handle it.  Last chance to pull a "Suppress-Macrolanguage"
SHOULD in 4646bis (as in SHOULD aao, or SHOULD NOT ar-aao).

Frank

---

From: "Debbie Garside" <[email-masked]>
Date: Sun, 10 Sep 2006 18:52:07 +0100

> automatically guess that I can also read "nds".

This is perhaps where the hierarchical system of ISO 639-6 comes in. Being
able to lookup the parent tag.  As 639-6 is still in development it would be
easy to incorporate ISO 639-1 codes where relevant; in the same way as ISO
639-6 is being used.

Best regards

Debbie

---

From: John Cowan <[email-masked]>
Date: Sun, 10 Sep 2006 15:00:01 -0400

> automatically guess that I can also read "nds".

True.  For that matter, it takes no account of the fair-to-usable
mutual intelligibility between the mainland Scandinavian
languages, not to mention thousands of other such situations
worldwide.  (English is unusually isolated in this respect,
except for the related creoles.)

However, the fact that you can read "nds" is a fact about you,
not about everyone who can read "de".

-- 
Even the best of friends cannot                 John Cowan
attend each others' funeral.                    [email-masked]
        --Kehlog Albran, The Profit             http://www.ccil.org/~cowan