Subject: [Ltru] Re: Punjabi

From: "Doug Ewell" <[email-masked]>
Date: Thu, 15 Mar 2007 16:09:12 -0700

As someone who's posted off-topic to this list several times, including 
many times when ietf-languages was a more suitable target, I confess not 
knowing how this discussion affects our project to update RFC 4646 and 
4645.  All of the current and future language subtags in the Registry 
are there because they are defined in ISO 639, and they have the 
primary/extended groupings that ISO 639 gives them (except "sgn", which 
is a special case).  We don't change these.

I can think of a few broad courses of action stemming from this 
discussion, but none involves the LTRU WG:

1.  The meanings and/or macrolanguage groupings associated with Panjabi 
and/or Lahnda are not adequately representative of reality, and need to 
be changed or clarified.  This should be brought to the ISO 639-3/RA at 
[email-masked].

2.  The subtags "pa" and/or "lah" should have comments attached to them 
that might help ensure correct usage.  This should be brought to 
[email-masked].

3.  Something about CLDR needs to be changed or re-examined.  This 
should be brought to the CLDR Technical Committee; see 
http://www.unicode.org/cldr/ for information.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 16 Mar 2007 13:05:58 -0700

It is, in fact, quite relevant to what we are doing.

I have had more and more trepidations about the way in which we are thinking
of using extlang. The one and only purpose of having the more complicated
"zh-cmn" in place of just plain "cmn" is to get matching behavior without
hitting the repository (or a local store of it). That is, if I have access
to repository data, then it is clearly possible for me to do a match so that
"cmn-CN" falls back to "cmn" and then to "zh"; I don't need to burden the
language tag itself with data that is redundant with information in the
repository. It is only if I don't have access to the data that it is an
issue.

My concerns are that we are baking into place a hierarchy that may, in
actual practice, be cumbersome and not that useful. We are already not
making use of the macrolanguages that are in ISO 639-1/2, like no and sh,
and there are other cases where fallback behavior != macrolanguage, either
always or often. We just ran into one: vls (Vlaams) wants to naturally
fallback to nl (Dutch/Flemish) in doing lookup. And part of the issue with
"pa" is that it means Panjabi in ISO 639-1, which for most people
encompasses the items grouped under lah.

This is not in any way meant as a criticism of the ISO 639-3 committee; I
have been quite impressed with how well they have handled a very tough
problem. But what I am concerned about is baking in a fallback that may vary
quite significantly according to application, just to avoid hitting the
repository, when it would be simpler to just use all the ISO 639-3 languages
as the first tag, just like we do for ISO 639-1/2.

Mark

-- 
Mark

---

From: John Cowan <[email-masked]>
Date: Fri, 16 Mar 2007 17:05:09 -0400

Mark Davis scripsit:

> actual practice, be cumbersome and not that useful. 

I understand the concern, but I point out that we don't know what the
scope of actual practice will be.  For the first time we are extending
language tagging to languages that either don't exist in written form at
all, or are "written" by writing some closely related language.
This opens up a whole new set of hitherto unknown applications.

-- 
I am expressing my opinion.  When my            John Cowan
honorable and gallant friend is called,         [email-masked]
he will express his opinion.  This is           http://www.ccil.org/~cowan
the process which we call Debate.                   --Winston Churchill

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 16 Mar 2007 15:37:27 -0700

Yes, and it concerns me that we are baking in a particular view of the world
by requiring that some language tags can only be used in conjunction with
others (eg pmu can only be used with lah-pmu). Now maybe I'm missing
something, can you articulate the reasons why we must use lah-pmu (for
example) instead of just pmu?

Mark

---

From: Addison Phillips <[email-masked]>
Date: Fri, 16 Mar 2007 16:03:53 -0700

Mark Davis wrote:
> Yes, and it concerns me that we are baking in a particular view of the 
> world by requiring that some language tags can only be used in 
> conjunction with others (eg pmu can only be used with lah-pmu). Now 
> maybe I'm missing something, can you articulate the reasons why we must 
> use lah-pmu (for example) instead of just pmu?
> 

The reasons are all procedural: any ISO 639-3 code that is contained by 
a macro-language and is not previously encoded by ISO 639-1 or ISO 639-2 
must be an extlang whose Prefix is the macro-language code.

This allows us to piggyback on ISO 639-3's work in this area to create 
tags such as zh-cmn and avoid naked 'cmn' language tags without having, 
ourselves, to squint at the lists and make separate, possibly 
unreasonable, decisions.

One possible way to avoid this would be to limit the "automatic" 
creation of mappings to those languages that have ISO 639-1 codes. This 
greatly limits the impact. The problem here is that it is arbitrary (why 
Aymara and not Baluchi? why Cree and not Delaware?)

To respond to Doug's point, I think that we are not *forced* to delay 
RFC 4646bis or even 4645bis's appearance. What we need are clear rules 
for the incorporation of ISO 639-3 into the registry scheme. This 
*could* take the form of a Big Bang insertion. But it would certainly be 
valid to insert only the language (and not the extlang) codes initially 
or to include only the finalized and stable extlang codes when they are 
mature---on a different day.

I would suggest that a mechanism for doing this would be to take each 
macro-language, as a collection, and vet the contents with the RA and 
on-list with ietf-languages before doing an insert. The Chinese and 
Arabic collections probably get in straight-away. Lahnda's subtags (by 
way of example) could wait---and no one is hurt by that delay. The only 
hurt is getting the mapping wrong. Worst case we have some languages 
that could have been extlangs that become primary language subtags instead.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 16 Mar 2007 16:17:57 -0700

I must not be clearly stating my point. Let me try again.

I'm getting at the fundamental reasoning behind extlang at all. I'm not
arguing that we should use different DATA than ISO 639-3 in order to decide
what are extlangs and what are not. What I'm saying is: why do we need the
extlang construction at all? Why do we need to have zh-cmn instead of just
cmn?

> Worst case we have some languages that could have been extlangs that
become primary language subtags instead.

That is: Why don't we simply have *all* languages be primary language
subtags instead of extlangs? What do extlangs buy us, that we need the
complication that they introduce?

Mark

---

From: Addison Phillips <[email-masked]>
Date: Fri, 16 Mar 2007 16:46:22 -0700

What extlangs buys us is: for languages that are already tagged, the 
current primary language subtags remain consistently the primary 
language subtag of choice. You might add additional subtags, but you 
don't have to retag all of your content.

Thus "zh-Hans-CN" is still valid. You would not (could not) retag with 
"cmn-Hans-CN". You might retag with "zh-cmn-Hans-CN", but this 
interferes less drastically with matching than outright change of the 
primary subtag.

What we want are consistent choices for language tags.

One alternative would be to allow both "zh-cmn" and "cmn". Users would 
have to be careful to use these consistently in their content and range 
selection.

Another alternative would be to forget extlang altogether and permit 
*either* "zh" *or* "cmn" but not both in the same tag (except by 
grandfathering). This frees the extlang up for other, nefarious, purposes.

My surmise is that macro-languages are a one-time event: "discovery" of 
future macro-languages will mostly be prohibited by rule (since most of 
the languages will already have codes in the "primary" position when 
they become part of a macro-language collection). If my surmise is 
correct, we could ban future extlang additions and use the remainder of 
that namespace for (well) nefarious purposes.

The only exception to my surmise would be: a language not previously 
given a code that is part of an existing macro-language. A language that 
isn't currently a macro-language that receives a new sublanguage would 
be a problem (all of the macro-languages are, by definition, one-to-many 
mappings). That is, the sub languages always travel in (at least) pairs.

Does that make sense?

Addison

Mark Davis wrote:
> arguing that we should use different DATA than ISO 639-3 in order to 
> decide what are extlangs and what are not. What I'm saying is: why do we 
> need the extlang construction at all? Why do we need to have zh-cmn 
> instead of just cmn?
> 
> On 3/16/07, *Addison Phillips* <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
> 
> 
>     Mark Davis wrote:

---

From: "Don Osborn" <[email-masked]>
Date: Fri, 16 Mar 2007 20:33:51 -0400

This is getting interesting but tricky. I think John's point...

"we don't know what the scope of actual practice will be.  For the first time we are extending language tagging to languages that either don't exist in written form at all, or are "written" by writing some closely related language.
This opens up a whole new set of hitherto unknown applications."

...was on target. I also concur with Mark's unease with "baking" in a certain approach, given the potential alternative needs. I would say "flexibility" here but that got me into trouble before.

Re Addison's explanations, replies in text...

> -----Original Message-----
> From: Addison Phillips [mailto:[email-masked]]
...
> have to be careful to use these consistently in their content and range

In effect cmn would "map" to zh? I wonder if the existence of the codes doesn't pretty much guarantee that some people will, for whatever reason, use the language code without attention to the macrolanguage code. One would have to consider this possibility and provide for it. Hence this seems to be a good idea.

> grandfathering). This frees the extlang up for other, nefarious,

But presumably there is a relationship between zh and cmn? What about permitting all three options? This goes back to the ff examples. IOW one could tag the macrolanguage, or the language, or both. The latter two having the same function. 

For some languages at least, I wonder if removing the extlang relationship wouldn't lead to some confusion. On the other hand, if overlapping locales and the possibility of alternative tagging being used for essentially the same thing is not a problem, then maybe this could work.

> that namespace for (well) nefarious purposes.

Don't agree here. I think there are de facto macrolanguages out there without the title that may need new codes and the extlang relationships with other 639-3 or even 639-1/2 coded (sub)languages that those would imply (Runyakitara, Oshiwambo, and Beti are examples of possibilities, from my understanding of descriptions). At the very least, the door should not be closed.

It is a truism that language is changing, but for a number of languages there are factors (per John's comment) that are still not clear or are more "changeable" than the ones we have more experience with in ICT.

Don

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 16 Mar 2007 18:05:57 -0700

> You might retag with "zh-cmn-Hans-CN", but this interferes less
drastically with matching than outright change of the primary subtag.

This is the crux of the matter, whether this is in fact true. We've all been
sort-of assuming that it is, but our goal at the time of 4646 was to make
sure that we had some structure to accommodate the possibility in case we
wanted it, long before we'd really take hard looks at the pros and cons. In
order to establish whether this statement is true or not, I think we need to
walk through some of the scenarios, and see how important it is in practice.
I'll take a crack at a few cases, although I think we need a more extensive
look.

Filtering, No Extlang Model

Let's suppose that I have content tagged with the following:

#1 zh-Hant-HK
#2 zh-Hans-HK
#3 yue-SG

If I basic filter for zh, I'll get #1 and #2. With just a bit smarter
filter, using information from ISO 639 (maybe put into our registry), I'll
also match #3. If I search with zh-Hant, I'll get #1 in either case. If I
search for yue, I'll get just #3.

Filtering, Extlang Model

#1 zh-Hant-HK
#2 zh-Hans-HK
#3 zh-yue-SG

If I search for content matching zh, I don't have to use the "bit smarter"
filter. Otherwise the same.

Lookup, No extlang Model

#1 cmn-Hant-HK
#2 cmn-Hant
#3 cmn
#4 yue-Hant-HK
#5 yue-Hant
#6 yue
#7 zh

If I request zh-Hans-CN, I'll fall all the way back to zh. If I didn't have
#7, I'd fail, unless I had some smarts to do an alias to cmn. If I request
yue-Hans, I'd fallback to yue.

Lookup, Extlang Model

#1 zh-cmn-Hant-HK
#2 zh-cmn-Hant
#3 zh-cmn
#4 zh-yue-Hant-HK
#5 zh-yue-Hant
#6 zh-yue
#7 zh

If I request zh-Hans-CN, I'll fall all the way back to zh. If I didn't have
#7, I'd fail, unless I had some smarts to do an alias to cmn. If I request
zh-yue-Hans, I'd fallback to zh-yue.

I'm not seeing a lot of difference here, but I certainly haven't explored
all the possibilities. We need to try some more scenarios to see where we
have some compelling differences. In particular, we need to look at cases
where we are looking up where the target has cmns or yues and the key
doesn't, and vice versa.

Mark

---

From: Peter Constable <[email-masked]>
Date: Fri, 16 Mar 2007 23:34:16 -0700

Perhaps it would help not to consider all macrolanguage (or putative macrolanguages - keep in mind "lah" still needs to be evaluated) as comparable cases. There's a big difference between "lah" and "zh": the former is rare in existing data (certain libraries perhaps being exceptional), while the latter is *very* common. The extlang construct was suggested with cases like the latter in mind, not the former.

Peter

From: Mark Davis [mailto:[email-masked]]
Sent: Friday, March 16, 2007 6:06 PM
To: Addison Phillips
Cc: Doug Ewell; LTRU Working Group
Subject: Re: [Ltru] Re: Punjabi

> You might retag with "zh-cmn-Hans-CN", but this interferes less drastically with matching than outright change of the primary subtag.

This is the crux of the matter, whether this is in fact true. We've all been sort-of assuming that it is, but our goal at the time of 4646 was to make sure that we had some structure to accommodate the possibility in case we wanted it, long before we'd really take hard looks at the pros and cons. In order to establish whether this statement is true or not, I think we need to walk through some of the scenarios, and see how important it is in practice. I'll take a crack at a few cases, although I think we need a more extensive look.

#1 zh-Hant-HK
#2 zh-Hans-HK
#3 yue-SG

If I basic filter for zh, I'll get #1 and #2. With just a bit smarter filter, using information from ISO 639 (maybe put into our registry), I'll also match #3. If I search with zh-Hant, I'll get #1 in either case. If I search for yue, I'll get just #3.

Filtering, Extlang Model

#1 zh-Hant-HK
#2 zh-Hans-HK
#3 zh-yue-SG

If I search for content matching zh, I don't have to use the "bit smarter" filter. Otherwise the same.

Lookup, No extlang Model

#1 cmn-Hant-HK
#2 cmn-Hant
#3 cmn
#4 yue-Hant-HK
#5 yue-Hant
#6 yue
#7 zh

If I request zh-Hans-CN, I'll fall all the way back to zh. If I didn't have #7, I'd fail, unless I had some smarts to do an alias to cmn. If I request yue-Hans, I'd fallback to yue.

Lookup, Extlang Model

#1 zh-cmn-Hant-HK
#2 zh-cmn-Hant
#3 zh-cmn
#4 zh-yue-Hant-HK
#5 zh-yue-Hant
#6 zh-yue
#7 zh

If I request zh-Hans-CN, I'll fall all the way back to zh. If I didn't have #7, I'd fail, unless I had some smarts to do an alias to cmn. If I request zh-yue-Hans, I'd fallback to zh-yue.

I'm not seeing a lot of difference here, but I certainly haven't explored all the possibilities. We need to try some more scenarios to see where we have some compelling differences. In particular, we need to look at cases where we are looking up where the target has cmns or yues and the key doesn't, and vice versa.

Mark
On 3/16/07, Addison Phillips <[email-masked]<mailto:[email-masked]>> wrote:

Does that make sense?

Addison

Mark Davis wrote:
> On 3/16/07, *Addison Phillips* <[email-masked]<mailto:[email-masked]>
> <mailto:[email-masked]<mailto:[email-masked]>>> wrote:
>
>
>
>     Mark Davis wrote:

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 17 Mar 2007 11:54:32 +0100

> grandfathering). This frees the extlang up for other, nefarious, 

Getting rid of it completely might be possible, or using it only for
special purposes like "sgn".  Apparently the "zh" situation isn't much
better than the "sgn" situation, unless the "zh" tag contains a script
and is in essence used instead of "und".

Frank

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 17 Mar 2007 09:50:04 -0700

Addison Phillips <addison at yahoo dash inc dot com> wrote:

> for the incorporation of ISO 639-3 into the registry scheme.

Which will then be a significant part of RFC 4645bis.

> This *could* take the form of a Big Bang insertion. But it would 
> certainly be valid to insert only the language (and not the extlang) 
> codes initially or to include only the finalized and stable extlang 
> codes when they are mature---on a different day.

That's an interesting idea.  It gets us almost 6,700 new primary 
language subtags that weren't available before.  One possible drawback 
is that it leaves the Panjabi/Lahnda question (and others) open, and 
might prolong the period during which things get tagged incorrectly, at 
a time when the increased capacity and visibility of our mechanism may 
spark an increase in the use of language tags.

> way of example) could wait---and no one is hurt by that delay. The 
> only hurt is getting the mapping wrong. Worst case we have some 
> languages that could have been extlangs that become primary language 
> subtags instead.

I wonder how subjective this "vetting" process might be, or might be 
perceived to be.  I can easily imagine other experts or native speakers 
castigating us for getting the decision "wrong," if there are no 
hard-and-fast rules.  At least if we add all encompassed languages that 
aren't currently subtags (a) as extlangs --  the original plan -- or (b) 
as primary language subtags, it can be defended as a rule application 
and not a value judgment on our part.

As you can tell, I'm not at all sure what the best course of action is, 
if we are going to revisit this most basic assumption of LTRU 2.0.

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 17 Mar 2007 10:39:36 -0700

Don Osborn <dzo at bisharat dot net> wrote:

> I also concur with Mark's unease with "baking" in a certain approach, 
> given the potential alternative needs. I would say "flexibility" here 
> but that got me into trouble before.

If the proposal is to allow both "zh-cmn" and "cmn" as synonyms, with 
neither deprecated in favor of the other, then I will indeed object to 
that kind of "flexibility."  The current thinking seems to be that 
matching algorithms will be too dumb to perform anything but basic 
remove-from-right truncation, which would fail dismally under such a 
plan.

> In effect cmn would "map" to zh?

Yes, via the Prefix field.

> I wonder if the existence of the codes doesn't pretty much guarantee 
> that some people will, for whatever reason, use the language code 
> without attention to the macrolanguage code. One would have to 
> consider this possibility and provide for it.

"cmn" would be listed as "Type: extlang".  If people misuse the protocol 
to the point of using extended language subtags alone, without their 
prefix, there is probably nothing we can do about it.  This type of 
person will also invent their own subtags, switch the order of subtags 
from what the ABNF specifies, and so forth.

>> My surmise is that macro-languages are a one-time event: "discovery" 
>> of future macro-languages will mostly be prohibited by rule (since 
>> most of the languages will already have codes in the "primary" 
>> position when they become part of a macro-language collection). If my 
>> surmise is correct, we could ban future extlang additions and use the 
>> remainder of that namespace for (well) nefarious purposes.
>
> Don't agree here. I think there are de facto macrolanguages out there 
> without the title that may need new codes and the extlang 
> relationships with other 639-3 or even 639-1/2 coded (sub)languages 
> that those would imply (Runyakitara, Oshiwambo, and Beti are examples 
> of possibilities, from my understanding of descriptions). At the very 
> least, the door should not be closed.

I think Addison was talking about the combination of (a) ISO 639-3 
adding new macrolanguage relationships with languages that are not 
currently encoded, and (b) our group sticking to its rule about adding 
extlangs.  If we change our rules and allow extlangs to mean something 
other than what they mean in draft-4646bis-02, then anything can happen.

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 17 Mar 2007 10:45:19 -0700

Mark Davis wrote:

> filter, using information from ISO 639 (maybe put into our registry), 
> I'll also match #3. If I search with zh-Hant, I'll get #1 in either 
> case. If I search for yue, I'll get just #3.

Wait a minute.  You've been arguing that we need to put writing-system 
differences in the script subtag and not in the variant, because filters 
will be too stupid to do anything more than prefix matching.  Then why 
would we design for the possibility of a "smarter filter" than could 
associate "yue" with "zh"?

---

From: "Don Osborn" <[email-masked]>
Date: Sun, 18 Mar 2007 00:15:42 -0400

Thanks Doug for the clarifications. 

Your statement "This type of person will also invent their own subtags, switch the order of subtags from what the ABNF specifies, and so forth" had me thinking more about user behavior and actually about an analogy with the "diffusion of innovation" model. (Bear with me...)

Briefly, at the lead end of the bell-curve you have the folks who really get it and go with, no matter how complicated it (in fact, maybe invent it). At the other end of course are those who totally don't get it or don't want to, or worse. In the middle are various shades of understanding and motivation, and I'm suspecting that in there, the well meaning folks who kind of get it may use the codes in a seemingly sensible but not strictly proper way. Like cmn instead of zh-cmn (and so on). There already is the notion out there that ISO-639-3 supersedes 1&2 (in some contexts it may, but the real overall relationship - let alone the intricacies of some specific languages - does not seem to lend itself to a quick and clear explanation). That alone could lead to honest errors in choice.

I guess there is a problem, though, if it turns out that the system is complex enough that human errors are likely and technofixes are hard to devise. It may not be possible to cover for all the possible misuses of tags, but maybe one can cover for the smarter mistakes?

On the other hand, what is the actual number of (macro)languages where extlang issues arise?

This may rate as another non-sequiteur message, but I'm trying to think of end users and scenarios. Have a good rest-of-the-weekend.

Don

PS- I take it then that "new" macrolanguages can be defined in the future, under some circumstances?

> From: Doug Ewell [mailto:[email-masked]]
> Sent: Saturday, March 17, 2007 1:40 PM
> To: LTRU Working Group
> Cc: Don Osborn

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 18 Mar 2007 11:52:58 +0100

Don Osborn wrote:
 
> what is the actual number of (macro)languages where extlang issues arise?

I try to determine this "manually" for Doug's latest 4645bis:
There are 480 "Prefix:" lines in the extlang part.

ar	30		ay	 2		az	 2
bal	 3		bik	 5		bua	 3
chm	 2		cr	 6		del	 2
den	 2		din	 5		doi	 2
fa	 2		ff	 9		gba	 5
gn	 5		gon	 2		grb	 5
hai	 2		hmn	21		ik	 2
iu	 2		jrb	 5		kg	 3
kok	 2		kpe	 2		kr	 3
ku	 3		kv	 2		lah	 8
man	 7		mg	10		mn	 2
ms	13		mwr	 6		oc	 5
oj	 7		om	 4		ps	 3
qu	44		raj	 6		rom	 7
sc	 4		sgn	124		sq	 4
sw	 2		syr	 2		tnh	 4
uz	 2		yi	 2		za	 2
zap	58		zh	13		zza	 2

That's 54 at the moment.

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 18 Mar 2007 11:37:10 -0700

Another way to look at it is on
http://www.sil.org/iso639-3/macrolanguages.asp, which provides the language
names and breakdowns.

Mark

---

From: "Don Osborn" <[email-masked]>
Date: Sun, 18 Mar 2007 15:09:01 -0400

Thanks Frank for the summary list and Mark for the pointer. Actually I was going to reference that page and ask if it is exactly what is in question.

Next (trying my luck here), is there any kind of way(s) that these can be subgrouped? For instance ar ends up referring to standard Arabic, if I understand correctly, and zh has been discussed already a lot; but some other macrolanguages do not necessarily have a single standard form. Some (macro) languages have a lot more in writing than others. What I’m getting at is are there different sets of (possible) complications that can be identified for the macrolanguages, languages and extlang relationships, such that the list of 54 can be disaggregated (perhaps in more than one way)? 

Not as if folks don’t have enough else to think about, but it seems like such an analysis, if it hasn’t been done, might raise other productive questions. 

Don

 

From: Mark Davis [mailto:[email-masked]] 
Sent: Sunday, March 18, 2007 2:37 PM
To: Frank Ellermann
Cc: [email-masked]
Subject: Re: [Ltru] Re: extlang

Another way to look at it is on http://www.sil.org/iso639-3/macrolanguages.asp, which provides the language names and breakdowns.

Mark

ar      30              ay       2              az       2
bal      3              bik      5              bua      3
chm      2              cr       6              del      2
den      2              din      5              doi      2 
fa       2              ff       9              gba      5
gn       5              gon      2              grb      5
hai      2              hmn     21              ik       2
iu       2              jrb      5              kg       3 
kok      2              kpe      2              kr       3
ku       3              kv       2              lah      8
man      7              mg      10              mn       2
ms      13              mwr      6              oc       5 
oj       7              om       4              ps       3
qu      44              raj      6              rom      7
sc       4              sgn     124             sq       4
sw       2              syr      2              tnh      4 
uz       2              yi       2              za       2
zap     58              zh      13              zza      2

That's 54 at the moment.

Frank

-- 
Mark

---

From: Addison Phillips <[email-masked]>
Date: Mon, 19 Mar 2007 08:08:53 -0700

The idea of macro-languages is that they are not, themselves, languages. 
Rather, they are groupings of languages that can be usefully referred to 
collectively. That is, strictly speaking, "Chinese" (meaning "zh") isn't 
a language---and neither is "Arabic" (by which I mean the code "ar").

Unfortunately, that isn't common usage or understanding of the 
situation. To most people, "Arabic" is a language. The idea that it has 
regional or other variations seems natural enough, but not, perhaps, 
that it is a set of somewhat related languages that share a historical 
and/or written tradition.

Allowing both the macro- and plain-language codes on the same level is 
recipe for confusion: does "ar-EG" == "arb-EG" == "arz" == "arz-EG"? 
What is supposed to match? Which tag should be used for a given 
document? How should one distinguish these?

We have a tradition, furthermore, of not having "secret information" in 
language tags requiring extra mapping tables to make sense of or process 
the tags. This tradition is punctuated (and punctured) by an equal 
tradition of assigning "secret" meaning to language tags. Thus, for the 
longest time, "zh-TW" meant "Traditional Chinese".

Extlangs help this a little by avoiding the choice on the initial 
subtag. Thus: "ar-EG" is related to "ar-arz-EG" and "ar-arb-EG" in 
distinct and somewhat logical ways.

Mark's concern is that this tagging system doesn't play nicely with 
basic filtering. He's not cited the fact that we have two filtering 
schemes: extended filtering works more reasonably with extlangs. 
"zh-yue-Hans-CN", "zh-cmn-Hans", and "zh-Hans-CN" all match the range 
"zh-Hans", for example.

The problem with lookup (and I use lookup extensively, so it concerns me 
deeply) might suggest that some extra smarts related to extlangs is 
going to be needed. On the other hand, some of this is going to have to 
be related to Maxim #1: "Tag Content Wisely". Choosing to avoid extlangs 
where they add no distinguishing information (not uncommon in resource 
file lookup for Arabic or Chinese, say) or *consistently* including them 
when they do (Lahnda???) will make things better.

I admit to a good bit of trepidation writing the above, though.

Addison

Don Osborn wrote:
> Thanks Frank for the summary list and Mark for the pointer. Actually I 
> was going to reference that page and ask if it is exactly what is in 
> question.
> 
>  
> 
> Next (trying my luck here), is there any kind of way(s) that these can 
> be subgrouped? For instance ar ends up referring to standard Arabic, if 
> I understand correctly, and zh has been discussed already a lot; but 
> some other macrolanguages do not necessarily have a single standard 
> form. Some (macro) languages have a lot more in writing than others. 
> What I’m getting at is are there different sets of (possible) 
> complications that can be identified for the macrolanguages, languages 
> and extlang relationships, such that the list of 54 can be disaggregated 
> (perhaps in more than one way)?
> 
>  
> 
> Not as if folks don’t have enough else to think about, but it seems like 
> such an analysis, if it hasn’t been done, might raise other productive 
> questions.
> 
>  
> 
> Don
> 
>  
> 
>  
> 
>  
> 
>  
> 
>  
> 
> *From:* Mark Davis [mailto:[email-masked]]
> *Sent:* Sunday, March 18, 2007 2:37 PM
> *To:* Frank Ellermann
> *Cc:* [email-masked]
> *Subject:* Re: [Ltru] Re: extlang
> 
>  
> 
> http://www.sil.org/iso639-3/macrolanguages.asp, which provides the 
> language names and breakdowns.
> 
> Mark
> 
> On 3/18/07, *Frank Ellermann* <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
> Don Osborn wrote:
> 
> [email-masked] <mailto:[email-masked]>
> ------------------------------------------------------------------------
>

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 19 Mar 2007 09:10:23 -0700

I see it a somewhat different way. The fact that there is a macro language
(zh), should not skew the way we encode an individual language (yue). You
say: "Extlangs help this a little by avoiding the choice on the initial
subtag. Thus: "ar-EG" is related to "ar-arz-EG" and "ar-arb-EG" in distinct
and somewhat logical ways."

However, functionally, we have to see advantage in subordinating some
languages as extlangs. There isn't value if they just add complication, as
per your "The problem with lookup (and I use lookup extensively, so it
concerns me deeply) might suggest that some extra smarts related to extlangs
is going to be needed."

In order to have a good case for the extlang model, we would need to see
concrete scenarios where we can demonstrate that "zh-cmn" and "zh-yue" work
better than "cmn" and "yue" resp., and demonstration that those are more
important than the scenarios where they cause problems.

Note that ISO 639-3 does not at all force the use of the extlang model; the
extlang model is just one possible way of expressing the information in ISO
639-3. We already have macrolanguages and "subordinate" languages in BCP 47,
but we put them at the same level. For example, ISO 639-3 already
categorizes no and sh as macrolanguages, and nb, no and sr, hr, and bs as
subordinate to them. We are not going to (and cannot) be forcing users to
encode nb as no-nb, nor sr as sh-sr. We shouldn't be making Cantonese a
subordinate language either.

At one point, I did think that having the extlang structure would be better,
but the more I get into actually implementing them, the more I find that
they are just a complication for no good result. What I think we should
instead be doing is adding a field to the registry that says that X is a
macro language for Y, and adding information to 4647 that indicates how one
can make use of this information in matching. That would also extend to the
current situation with no and sh in a uniform manner. It is also a much less
fragile mechanism: one can add more such X,Y relations over time without
gumming up everything.

Mark

On 3/19/07, Addison Phillips <[email-masked]> wrote:
> > What I'm getting at is are there different sets of (possible)
> > Not as if folks don't have enough else to think about, but it seems like
> > such an analysis, if it hasn't been done, might raise other productive
> > questions.
> >
> >
> >
> > Don
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >

---

From: John Cowan <[email-masked]>
Date: Mon, 19 Mar 2007 21:12:29 -0400

> a language---and neither is "Arabic" (by which I mean the code "ar").

Collections are one thing and macrolanguages are another.  A macrolanguage
is a grouping of languages that can usefully be treated as a single
language in certain contexts.

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
I am he that buries his friends alive and drowns them and draws them
alive again from the water. I came from the end of a bag, but no bag
went over me.  I am the friend of bears and the guest of eagles. I am
Ringwinner and Luckwearer; and I am Barrel-rider.  --Bilbo to Smaug

---

From: Addison Phillips <[email-masked]>
Date: Mon, 19 Mar 2007 18:25:34 -0700

Mark Davis wrote:
> I see it a somewhat different way. The fact that there is a macro 
> language (zh), should not skew the way we encode an individual language 
> (yue). You say: "Extlangs help this a little by avoiding the choice on 
> the initial subtag. Thus: "ar-EG" is related to "ar-arz-EG" and 
> "ar-arb-EG" in distinct and somewhat logical ways."

I'm not sure we see it a different way.

My main problem is not with the body of your argument: most of the
macrolanguages are actually rare in practice and their constituent
languages can be encoded on the top level without rancor. This will
represent a minimum of interruption for users. Some content will be best
tagged with the macro language and some with the constituent language.

My basic problem is with the two widely used and accepted subtags 'ar'
and 'zh'. You're asking for users to change to using a new set of tags
for these languages/"languages". How should users choose between the
current practice ("zh-Hans-CN", "ar-EG") and "modern" practice
("cmn-Hans-CN", "arb-EG")? It isn't clear.

This is the original justification for extlang: to help users of widely
used macro-language tags adopt the right tagging approach in a
compatible manner.

> languages as extlangs. There isn't value if they just add complication, 
> as per your "The problem with lookup (and I use lookup extensively, so 
> it concerns me deeply) might suggest that some extra smarts related to 
> extlangs is going to be needed."

The advantage of subordinating some languages is that there is a
significant body of content already tagged with the macro-language code
and that the macro-language retains a broadly applicable set of uses: 
there is, in many senses, a real thing called "Chinese" or "Arabic".
With extlangs, the "subordinated" language code can be added to tags in 
a manner consistent with the matching schemes and tagging already in place.

"sgn" benefits mightily from this scheme. I believe that at least "zh"
does (given that written forms of Chinese, to a significant degree, are
not very distinctive between the various Sinitic languages).

> concrete scenarios where we can demonstrate that "zh-cmn" and "zh-yue" 
> work better than "cmn" and "yue" resp., and demonstration that those are 
> more important than the scenarios where they cause problems.

I think your citation of Norwegian is not a good example to pick on.
Historically the subtags 'nn' and 'nb' have been problematic: they 
overlap with the more common 'no'. The existence of three subtags for 
the language and its variations is confusing, and, ultimately, not that 
useful... since no one is ever sure if some 'no' labeled resources exist 
(or not). This case can perhaps be extended to the 
Mandarin/Cantonese/etc. case as an example where extlangs would have 
been less painful than full-fledged language codes?

> 
> Note that ISO 639-3 does not at all force the use of the extlang model; 
> the extlang model is just one possible way of expressing the information 
> in ISO 639-3.

That's right. I think I suggested this as one of the solutions.

>  We already have macrolanguages and "subordinate" languages 
> in BCP 47, but we put them at the same level. 

No, we didn't recognize the distinction: each primary language subtag is 
supposed to encode "a language". This has been occasionally confusing 
for users who expect the relationship to have meaning.

>  We shouldn't be 
> making Cantonese a subordinate language either.

The anti-question would be: why *not*? It's quite clear that "zh-HK"
doesn't mean "Cantonese" any more than "zh-TW" meant "Traditional
Chinese". The question of whether to use extlang or not is not a value
judgment about the language, but merely an encoding question.

In fact, for the most part, what you're suggesting is little different
from using extlang. Let's explore...

Let's say I request "zh-yue-Hant-HK". The default lookup scheme produces:

zh-yue-Hant-HK
zh-yue-Hant
zh-yue
zh

One way to have a lookup implementation use macro-language information
would be if it were free to treat extlangs as equivalencies or
ignorable. The fallback could be:

zh-yue-Hant-HK
zh-Hant-HK (implied)
zh-yue-Hant
zh-Hant (implied)
zh-yue
zh

...and let's say we request without extlang (zh-Hant-HK):

zh-Hant-HK
zh-Hant
zh

This misses any content tagged with 'yue'.

We could say that any "zh-yue" or "zh-cmn" content also matches on each
level (an implied match), but this would be more troubling (the user
didn't request that specific content):

zh-Hant-HK
zh-yue-Hant-HK (implied)
zh-Hant
zh-yue-Hant (implied)
zh
zh-cmn (implied, just for the sake of ickiness)

Now consider extended filtering for a second. A range of "zh-Hant"
matches each of:

zh-Hant
zh-yue-Hant
zh-cmn-Hant
zh-Hant-CN
zh-yue-Hant-CN

This is logical. Specifying "zh-yue" would not find a tag "zh-Hant" that
might be Cantonese, though.

Now what you're suggesting is that we put "yue" on the top-level. Some
Cantonese content will thus be tagged as "yue-*" and some as "zh-*". One
of two things has to happen. If I request "yue", I might get some
content labeled "zh". Or if I request some content labeled "yue" I do
NOT get any content labeled "zh" that might actually be Cantonese. This
doesn't solve the lookup problem:

yue-Hant-HK
zh-Hant-HK (implied)
yue-Hant
zh-Hant (implied)
yue
zh (implied)

This pattern is identical to the pattern I get (above, first example),
but with the complication of having the mapping table.

The compelling part of your argument is if we think in terms of a
language priority list instead. Then my fallback is:

yue-Hant-HK
yue-Hant
yue
zh-Hant-HK
zh-Hant
zh

Here we see the advantage for using 'yue' on the top level: we don't get
to the (probably Simplified) 'zh' and (probably Mandarin) 'zh-Hant'
resources too early (or at all, if we omit the pass through 'zh'-bearing 
ranges).

> 
> At one point, I did think that having the extlang structure would be 
> better, but the more I get into actually implementing them, the more I 
> find that they are just a complication for no good result. What I think 
> we should instead be doing is adding a field to the registry that says 
> that X is a macro language for Y, and adding information to 4647 that 
> indicates how one can make use of this information in matching. 

My implementations just don't make me think it is that complicated. Of 
course, most of my implementations have control over both the content 
(tags) and ranges used in the selection---not to mention defaulting 
behavior. I haven't gotten to the point yet of needing to identify the 
spoken Chinese variations in uncontrolled content, so haven't really the 
problems you do yet. Thinking about them suggests, however, that I'll 
want the broader selection profile that extlang gives me than the 
narrower one that (multiple choice) primary language gives me on the top 
level.

That's because the more structure a tag has, the more information it can 
contain and thus the simpler the lookup/filtering is to do. If an 
equivalence table is required, we'll, for the first time, be reliant on 
external data tables to do simple select-type matching (think about 
simplistic implementations such as the :lang pseudo-attribute in CSS for 
a moment!). And I'd like to avoid that if possible.

I recognize what you're saying is valid, for the kinds of data you're
working with. But I'm concerned equally that content tagging will be
mixed up for years to come because users do not understand when and
whether to use 'zh' vs. 'cmn' or 'ar' vs 'arb'.

Addison

---

From: "Don Osborn" <[email-masked]>
Date: Tue, 20 Mar 2007 00:51:37 -0400

Reply below...

> From: John Cowan [mailto:[email-masked]]
...
> language in certain contexts.

Thanks, John. The question I have is what are the contexts? IOW, are the
various macrolanguage-sublanguage relationships, which seem to represent
varying realities in terms of usage, categorizable into certain types? If so
might that help discussions?

"Macrolanguage" is an interesting concept, but whatever its original intent
(and whatever linguists in general think of it), it seems bound to accrue
other meanings and applications. This is not necessarily bad though it might
complicate life for coding.

Part of the issue with Arabic may be that one standard form is used pretty
widely while the spoken forms vary. I've heard from localisers of Arabic
that they only really use one Arabic. In the case of Arabic in Egypt, does
anyone write arz? In literature, theater, journalism, or...? I don't know,
but I suspect that the existence of the code for "Egyptian spoken Arabic"
(which by the way is only one of the colloquial forms in the country it
seems) will prompt some to choose it for what might more appropriately be
ar-EG or arb-EG. The part that I don't know about Arabic far exceeds the
little I think I know, so best to stop here.

When I think of Fula (ff) this is another complex issue. I won't get into it
now, but a couple of things come to mind: publications in the 1970s or so
that treated the (macro)language as a whole, and "pan-Fula" dictionaries
(with notes re different usages among the varieties of Fula). I have been
talking up the idea of a pan-Fula monolingual dictionary online, and hope to
discuss that some more in a few days (at ACAL/ALTA in Gainesville). It
really seems like you can shift perspective and see such tongues from the
point of their functional unity or their functional differences.

<way out in left field>All this has me thinking that a macrolanguage is
maybe something like a Platonic ideal but with a functional aspect. Thus
macrolanguages are actually all over the place, but we don't see them as
such since they so often coincide each with one language. So for instance
Hausa is a macrolanguage with only one language (and some dialect
differences from west to east). On the other hand, something like Fula is a
macrolanguage with several languages that have high degrees of
interintelligibility. IOW, language has an ideological component reflected
in "macrolanguage," which has some practical meaning/use. Take the case of
Runyakitara in western Uganda, which is a conscious effort to provide a
standard for a group of 4 very close languages that also have a common
historical identity - is that creating a macrolanguage or identifying
something that was always there? (conversely, none of the 4 tongues under
that were themselves cases of "macrolanguages" in the sense I'm exploring
here).</way out in left field>

Not sure that any of this helps the coding questions at hand but maybe it
gives another useful perspective on possible use of "macrolanguage."

All the best.

Don

---

From: GerardM <[email-masked]>
Date: Tue, 20 Mar 2007 11:13:44 +0100

Hoi,
Reading Don, it is yet another great argument to be sensible and allow for
the use of tags that are the single lowest denominator for a language. Don
explains quite nicely the ambiguity involved.

When you have a choice between ar-EG and arz and think you can say they are
the same, how would you then code spoken Egyptian as used by the immigrant
community in the USA ?? I would say that ar-arz-US and arz-US are
appropriate. The notion that ar-EG is necessarily the same as arz is imho
wrong. Coding it like ar-EG-US is probably not acceptable.

Thanks,
    Gerard

---

From: "Don Osborn" <[email-masked]>
Date: Tue, 20 Mar 2007 07:54:08 -0400

Hi Gerard, But what I have heard from some people re Arabic localization, “no one” writes “spoken Egyptian Arabic” arz (in Egypt or elsewhere). Maybe that’s an overstatement (I don’t know), but it reflects a certain reality. Which is not to suggest that ar-EG is necessarily the same as arz, but that they describe different realities (perhaps one written, one spoken? – if indeed it is that simple). In such a circumstance, what might seem to be the apparent “LCD” unit of language definition (and coding) may actually not be, or at least may not always be the single best choice.

In other contexts, one might seek an appropriate level of aggregation, rather than the smallest accurate language description. Such as software localization for instance (and hence locale).

So it seems that the context in combination with the specific realities of a particular macrolanguage-sublanguage(s) relationship(s) determines the choice.

All the best.

Don

From: GerardM [mailto:[email-masked]] 
Sent: Tuesday, March 20, 2007 6:14 AM
To: Don Osborn
Cc: John Cowan; Addison Phillips; Frank Ellermann; [email-masked]
Subject: Re: [Ltru] Re: extlang

Hoi,
Reading Don, it is yet another great argument to be sensible and allow for the use of tags that are the single lowest denominator for a language. Don explains quite nicely the ambiguity involved.

When you have a choice between ar-EG and arz and think you can say they are the same, how would you then code spoken Egyptian as used by the immigrant community in the USA ?? I would say that ar-arz-US and arz-US are appropriate. The notion that ar-EG is necessarily the same as arz is imho wrong. Coding it like ar-EG-US is probably not acceptable. 

Thanks,
    Gerard

Reply below...

...

All the best. 

Don

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 20 Mar 2007 13:28:41 +0100

On Tue, Mar 20, 2007 at 11:13:44AM +0100,
 GerardM <[email-masked]> wrote 
 a message of 155 lines which said:

> When you have a choice between ar-EG and arz and think you can say
> they are the same, how would you then code spoken Egyptian as used
> by the immigrant community in the USA ??

Easy: exactly the same as in his mother country. You do not change the
language you speak based on your physical location. It's not because
I'm currently in Prague that I write in en-CZ!

> I would say that ar-arz-US and arz-US are appropriate.

No, unless a special variant appeared among egyptian immigrants in the
USA (which is a possibility but not a necessity).

---

From: Peter Constable <[email-masked]>
Date: Tue, 20 Mar 2007 06:43:37 -0700

From: Don Osborn [mailto:[email-masked]]

> Thanks, John. The question I have is what are the contexts? IOW,
> are the various macrolanguage-sublanguage relationships, which seem
> to represent varying realities in terms of usage, categorizable into
>  certain types?

Answering that involves considering what are the reasons one person might have for grouping and what are the reasons that another might have for splitting.

Splitting

The splitting in most or all cases we're dealing with (where macrolanguages are involved) comes from Ethnologue. The rational is that, on the basis of analysis of the spoken language (the vast majority of languages catalogued being unwritten), including things like lexical similarities, comprehension testing, assessment of usage or assessment of attitudes, the conclusion is drawn that development of literature in one form is unlikely to provide adequate communication for all of the communities involved. (That's the basic principle; it may or may not have been consistently and correctly applied in all instances.)

Note that Ethnologue has not always followed every claim of a split. A familiar example is Serbo-Croatian vs. Bosnian, Croatian and Serbian, for which E only adopted the split only in the latest edition. Or, a less familiar example: I know that, in Mexico, among Mixtec varieties, many communities are wont to make hyper-differentiations, saying that each village is different when researchers find no observable barrier to communication: in these cases, E does not make distinctions the speakers themselves might indicate.

At the same time, the research should be looking for situations in which distinct spoken varieties can be bridged by a common literature, and that has not always happened. Sometimes that may have been because those doing the research didn't have that mindset. But in some cases, probably the research has gotten dated with conditions having changed in ways that make such merging more feasible.

Grouping

The reasons for grouping are various:

- The various communities share in common a single form of communication that de facto is in usage.

- There is a common perception (whether among the communities themselves or among people outside the communities) that there a single identity. (In some cases, the need of any split may have only lately come to light.)

- In spite of differences, policies or practice have been adopted to bring different varieties together with a common literary form.

- Someone gathering information resources in or about the language varieties in question doesn't have the need to differentiate between them. (E.g. a library that has neither the resources or the demand for distinguishing the items -- especially likely if the common perceptions are that there's just one entity.)

Our familiar macrolanguage, Chinese, is an example of the first two: there is a written form that for the most part is shared, and for some time that has contributed to a perception that it's all one language.

The third is something that may relate to cases such as Fula, which you mentioned.

The last is one that certainly pertains to some of the entries in 639-2. There definitely are cases in which librarians have done one of the following:

- A group of undeveloped varieties that share a label (e.g. "Quechua") are treated as a single category for cataloguing purposes.

- An ID is assigned for a major, developed variety (e.g., Malay); but then there are also minor varieties (undeveloped, smaller communities) that share the same label, and the librarians group those with the major language for cataloguing purposes.

- A taxonomy (perhaps a folk taxonomy) exists in which varieties were grouped under a single label for some convenience, and this is used as the basis for cataloguing. (E.g., "Lahnda", "Bihari").

In cases like Quechua, Bihari or Lahnda in which librarians have adopted groupings of varieties at comparable levels of development, the option existed with the introduction of 639-3 to treat the existing 639-2 ID either as a collection or as a macrolanguage. The question was whether there was some reason to warrant the latter. Taking Quechua as an example, I know that there's some ethnic nationalist sentiment that promotes a single identity, and that there has also been some recent language development activity that treated "Quechua" as one. On that basis, I suggested that macrolanguage might be the better property to assign, and the JAC just went along. In the case of Bihari, I wasn't aware of any utility for treating it as a macrolanguage, so suggested it be considered a collection, and the JAC went along. Who can judge whether these were the "right" decisions.

I think cases like Malay are unfortunate: it doesn't seem to me that there's a particular use for the grouping in software applications, but the grouping was done in other applications. Because of the latter, msa can't be treated as an individual-language ID, and I wouldn't like the idea of resources for a major language like Malay getting tagged with a collection ID. So, that leaves the macrolanguage alternative. Again, who can tell me if that was the right decision?

Now, as for IETF lang tags, I think it's reasonable to ask whether it make sense to use the extlang mechanism in all these cases. For Chinese, extlangs were basically introduced years ago. Does it also make sense for Quechua, Lahnda, Malay and all the others to do the same? I don't know.

Peter

---

From: Marion Gunn <[email-masked]>
Date: Tue, 20 Mar 2007 14:42:52 +0000

Exactly! It must be remembered that such linguistic illogicalities in  
the Ethnologue in general can mostly only be accounted for by  
assiduously studying the vagaries of American political/missionary/ 
overseas policies since SIL's foundation.
mg

On 20 Mar 2007, at 13:43, scríobh Peter Constable:

> ...
> Answering that involves considering what are the reasons one person  
> might have for grouping and what are the reasons that another might  
> have for splitting...
> The splitting in most or all cases we're dealing with (where  
> macrolanguages are involved) comes from Ethnologue....
> Note that Ethnologue has not always followed every claim of a  
> split. A familiar example is Serbo-Croatian vs. Bosnian, Croatian  
> and Serbian, for which E only adopted the split only in the latest  
> edition....

- -
Marion Gunn * EGTeo (Estab.1991)
27 Páirc an Fhéithlinn, Baile an
Bhóthair, Co. Átha Cliath, Éire.
* [email-masked] * [email-masked] *

---

From: Peter Constable <[email-masked]>
Date: Tue, 20 Mar 2007 08:49:07 -0700

I don't see how attacking SIL for ideological reasons has any relevance for this list. The "linguistic illogicalit[y]" of "Serbo-Croatian" is by no means unique to SIL or Americans.

Peter

-----Original Message-----
From: Marion Gunn [mailto:[email-masked]]
Sent: Tuesday, March 20, 2007 7:43 AM
To: LTRU Working Group
Subject: Re: [Ltru] Re: extlang

- -

---

From: Addison Phillips <[email-masked]>
Date: Tue, 20 Mar 2007 08:57:00 -0700

Don Osborn wrote:

 > But what I have heard from some people re Arabic
 > localization, “no one” writes “spoken Egyptian Arabic”
 > arz (in Egypt or
 > elsewhere). Maybe that’s an overstatement (I don’t know), but it
 > reflects a certain reality.

All languages seem to have this duality. The written form and the spoken 
form of language are rarely the same (they are different modes, after 
all). Plus, there are, for example, regional variations that don't rise 
to the level of a separately identifiable "dialect" (from the language 
tagging point of view).

While some applications may need ultra-fine-grained identifiers, 
many---even most---and especially on the Internet do not. Speakers (or, 
maybe more especially, *readers*) of Arabic in Egypt are mostly 
interested in the macro-language meaning of Arabic rather than 
specialized dialects (of interest to philologists).

Filtering by origin and other types of "regional" application is best 
done using other kinds of metadata.

Hence my concern that we'll introduce myriad ways of identifying (say) 
Arabic in tags: "ar", "ar-arb", "ar-arz", "arb", "arz", etc. Users 
should not need to know that all these options exist for a single 
document, especially when the distinctions make no difference to the 
document in question.

 >
 > In other contexts, one might seek an appropriate level of aggregation,
 > rather than the smallest accurate language description. Such as software
 > localization for instance (and hence locale).

Yes, hence language tags such as "es-419" [Latin American Spanish, 
essentially]. This is a somewhat artificial distinction useful in 
certain contexts. But most language tags have some basis actual language 
distinctions.

Addison

> 
> So it seems that the context in combination with the specific realities 
> of a particular macrolanguage-sublanguage(s) relationship(s) determines 
> the choice.
> 
This is always true of language tag choice. But the peculiarities of 
macro-language expose this rather more than some other choices one can 
make in forming language tags.

Addison

---

From: "Erkki I. Kolehmainen" <[email-masked]>
Date: Tue, 20 Mar 2007 18:16:53 +0200

+1

Erkki I. Kolehmainen
Tilkankatu 12 A 3, FI-00300 Helsinki, Finland
Puh. (09) 4368 2643, 0400 825 943; Tel. +358 9 4368 2643, +358 400 825 943  

-----Alkuperäinen viesti-----
Lähettäjä: Peter Constable [mailto:[email-masked]] 
Lähetetty: 20. maaliskuuta 2007 17:49
Vastaanottaja: LTRU Working Group
Aihe: RE: [Ltru] Re: extlang

I don't see how attacking SIL for ideological reasons has any relevance for
this list. The "linguistic illogicalit[y]" of "Serbo-Croatian" is by no
means unique to SIL or Americans.

Peter

-----Original Message-----
From: Marion Gunn [mailto:[email-masked]]
Sent: Tuesday, March 20, 2007 7:43 AM
To: LTRU Working Group
Subject: Re: [Ltru] Re: extlang

Exactly! It must be remembered that such linguistic illogicalities in the
Ethnologue in general can mostly only be accounted for by assiduously
studying the vagaries of American political/missionary/ overseas policies
since SIL's foundation. mg

On 20 Mar 2007, at 13:43, scríobh Peter Constable:

> have for splitting... The splitting in most or all cases we're dealing 
> with (where macrolanguages are involved) comes from Ethnologue....

- -

---

From: "Don Osborn" <[email-masked]>
Date: Tue, 20 Mar 2007 12:53:05 -0400

Thanks Addison. Quick comments in text...

...
...
> document, especially when the distinctions make no difference to the

I understand the concern. Problem is that the cat is out of the bag. Public education / outreach does not appear to have been a strong suit for various internationalization / standards efforts, but might benefit from more attention because of such issues. (Just an impression; I know there are reasons for that).

...

Thanks for reminding me of this (I believe John mentioned it a while back). It might be an approach useful for a number of African languages (Swahili and Hausa for starters, although each have country-level locales already) and probably for such as African French or English.

Don

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 20 Mar 2007 13:24:49 -0700

Having arq and the other dozen or so languages on
http://www.sil.org/iso639-3/documentation.asp?id=ara
as primary language tags is not a problem. The real problem is having either
arb or ar-arb, since we already have the tag "ar" which in practice means
the same thing, widely used in a great deal of software. Introducing either arb
or ar-arb essentially introduces a synonym for ar, and synonyms just cause
problems.

Given the set of imperfect options, I think the best one is (a) enhance the
registry, as I suggested, and (b) either don't encode arb, or encode and
deprecate it (and do similar things for the other macrolanguages that are in
widespread use), and (c) not have extlang.

FYI, here is a list of the current ISO 639-3 codes where the macrolanguage
to subordinate language relation exists in ISO 639-3, but we cannot (for
backwards compatibility) use an extlang model.

    ak 4646 Akan fat 4646 Fanti  ak 4646 Akan tw 4646 Twi  no 4646 Norwegian
nb 4646 Norwegian Bokmål  no 4646 Norwegian nn 4646 Norwegian Nynorsk  sh
4646 Serbo-Croatian bs 4646 Bosnian  sh 4646 Serbo-Croatian hr 4646 Croatian
sh 4646 Serbo-Croatian sr 4646 Serbian
And here is the whole list (if you don't get HTML mail, these may be
garbled).

    ak 4646 Akan fat 4646 Fanti  ak 4646 Akan tw 4646 Twi  ar 4646 Arabic
aao new Algerian Saharan Arabic  ar 4646 Arabic abh new Tajiki Arabic  ar
4646 Arabic abv new Baharna Arabic  ar 4646 Arabic acm new Mesopotamian
Arabic  ar 4646 Arabic acq new Ta'izzi-Adeni Arabic  ar 4646 Arabic
acw new Hijazi
Arabic  ar 4646 Arabic acx new Omani Arabic  ar 4646 Arabic acy new Cypriot
Arabic  ar 4646 Arabic adf new Dhofari Arabic  ar 4646 Arabic aeb new Tunisian
Arabic  ar 4646 Arabic aec new Saidi Arabic  ar 4646 Arabic afb new Gulf
Arabic  ar 4646 Arabic ajp new South Levantine Arabic  ar 4646 Arabic apc
new North Levantine Arabic  ar 4646 Arabic apd new Sudanese Arabic  ar 4646
Arabic arb new Standard Arabic  ar 4646 Arabic arq new Algerian Arabic  ar
4646 Arabic ars new Najdi Arabic  ar 4646 Arabic ary new Moroccan Arabic  ar
4646 Arabic arz new Egyptian Arabic  ar 4646 Arabic auz new Uzbeki Arabic
ar 4646 Arabic avl new Eastern Egyptian Bedawi Arabic  ar 4646 Arabic ayh
new Hadrami Arabic  ar 4646 Arabic ayl new Libyan Arabic  ar 4646 Arabic ayn
new Sanaani Arabic  ar 4646 Arabic ayp new North Mesopotamian Arabic  ar
4646 Arabic bbz new Babalia Creole Arabic  ar 4646 Arabic pga new Sudanese
Creole Arabic  ar 4646 Arabic shu new Chadian Arabic  ar 4646 Arabic
ssh new Shihhi
Arabic  ay 4646 Aymara ayc new Southern Aymara  ay 4646 Aymara ayr new Central
Aymara  az 4646 Azerbaijani azb new South Azerbaijani  az 4646 Azerbaijani
azj new North Azerbaijani  bal 4646 Baluchi bcc new Southern Balochi  bal
4646 Baluchi bgn new Western Balochi  bal 4646 Baluchi bgp new Eastern
Balochi  bik 4646 Bikol bcl new Central Bicolano  bik 4646 Bikol bhk new Albay
Bicolano  bik 4646 Bikol bln new Southern Catanduanes Bicolano  bik 4646
Bikol bto new Iriga Bicolano  bik 4646 Bikol cts new Northern Catanduanes
Bicolano  bua 4646 Buriat bxm new Mongolia Buriat  bua 4646 Buriat bxr
new Russia
Buriat  bua 4646 Buriat bxu new China Buriat  chm 4646 Mari (Russia)
mhr new Eastern
Mari  chm 4646 Mari (Russia) mrj new Western Mari  cr 4646 Cree crj
new Southern
East Cree  cr 4646 Cree crk new Plains Cree  cr 4646 Cree crl new Northern
East Cree  cr 4646 Cree crm new Moose Cree  cr 4646 Cree csw new Swampy Cree
cr 4646 Cree cwd new Woods Cree  del 4646 Delaware umu new Munsee  del 4646
Delaware unm new Unami  den 4646 Slave (Athapascan) scs new North Slavey
den 4646 Slave (Athapascan) xsl new South Slavey  din 4646 Dinka dib new South
Central Dinka  din 4646 Dinka dik new Southwestern Dinka  din 4646 Dinka dip
new Northeastern Dinka  din 4646 Dinka diw new Northwestern Dinka  din 4646
Dinka dks new Southeastern Dinka  doi 4646 Dogri (generic) dgo new Dogri
(specific)  doi 4646 Dogri (generic) xnr new Kangri  fa 4646 Persian
pes new Western
Farsi  fa 4646 Persian prs new Dari  ff 4646 Fulah ffm new Maasina Fulfulde
ff 4646 Fulah fub new Adamawa Fulfulde  ff 4646 Fulah fuc new Pulaar  ff
4646 Fulah fue new Borgu Fulfulde  ff 4646 Fulah fuf new Pular  ff 4646
Fulah fuh new Western Niger Fulfulde  ff 4646 Fulah fui new Bagirmi Fulfulde
ff 4646 Fulah fuq new Central-Eastern Niger Fulfulde  ff 4646 Fulah
fuv new Nigerian
Fulfulde  gba 4646 Gbaya (Central African Republic) bdt new Bokoto
gba 4646 Gbaya
(Central African Republic) gbp new Gbaya-Bossangoa  gba 4646 Gbaya (Central
African Republic) gbq new Gbaya-Bozoum  gba 4646 Gbaya (Central African
Republic) gya new Northwest Gbaya  gba 4646 Gbaya (Central African Republic)
mdo new Southwest Gbaya  gn 4646 Guarani gnw new Western Bolivian Guaraní
gn 4646 Guarani gug new Paraguayan Guaraní  gn 4646 Guarani gui new Eastern
Bolivian Guaraní  gn 4646 Guarani gun new Mbyá Guaraní  gn 4646 Guarani nhd
new Chiripá  gon 4646 Gondi ggo new Southern Gondi  gon 4646 Gondi gno
new Northern
Gondi  grb 4646 Grebo gbo new Northern Grebo  grb 4646 Grebo gec new Gboloo
Grebo  grb 4646 Grebo grj new Southern Grebo  grb 4646 Grebo grv new Central
Grebo  grb 4646 Grebo gry new Barclayville Grebo  hai 4646 Haida hax
new Southern
Haida  hai 4646 Haida hdn new Northern Haida  hmn 4646 Hmong blu new Hmong
Njua  hmn 4646 Hmong hea new Northern Qiandong Hmong  hmn 4646 Hmong
hma new Southern
Mashan Hmong  hmn 4646 Hmong hmc new Central Huishui Hmong  hmn 4646 Hmong
hmd new Northeastern Dian Hmong  hmn 4646 Hmong hme new Eastern Huishui
Hmong  hmn 4646 Hmong hmg new Southwestern Guiyang Hmong  hmn 4646 Hmong hmh
new Southwestern Huishui Hmong  hmn 4646 Hmong hmi new Northern Huishui
Hmong  hmn 4646 Hmong hmj new Chonganjiang Hmong  hmn 4646 Hmong hml
new Luopohe
Hmong  hmn 4646 Hmong hmm new Central Mashan Hmong  hmn 4646 Hmong hmp
new Northern
Mashan Hmong  hmn 4646 Hmong hmq new Eastern Qiandong Hmong  hmn 4646 Hmong
hms new Southern Qiandong Hmong  hmn 4646 Hmong hmw new Western Mashan Hmong
hmn 4646 Hmong hmy new Southern Guiyang Hmong  hmn 4646 Hmong huj new Northern
Guiyang Hmong  hmn 4646 Hmong mmr new Western Xiangxi Hmong  hmn 4646 Hmong
muq new Eastern Xiangxi Hmong  hmn 4646 Hmong mww new Hmong Daw  ik 4646
Inupiaq esi new North Alaskan Inupiatun  ik 4646 Inupiaq esk new Northwest
Alaska Inupiatun  iu 4646 Inuktitut ike new Eastern Canadian Inuktitut  iu
4646 Inuktitut ikt new Western Canadian Inuktitut  jrb 4646 Judeo-Arabic ajt
new Judeo-Tunisian Arabic  jrb 4646 Judeo-Arabic aju new Judeo-Moroccan
Arabic  jrb 4646 Judeo-Arabic jye new Judeo-Yemeni Arabic  jrb 4646
Judeo-Arabic yhd new Judeo-Iraqi Arabic  jrb 4646 Judeo-Arabic yud new
Judeo-Tripolitanian
Arabic  kg 4646 Kongo kng new Koongo  kg 4646 Kongo kwy new San Salvador
Kongo  kg 4646 Kongo ldi new Laari  kok 4646 Konkani (generic) gom new Goanese
Konkani  kok 4646 Konkani (generic) knn new Konkani (specific)  kpe 4646
Kpelle gkp new Guinea Kpelle  kpe 4646 Kpelle xpe new Liberia Kpelle  kr
4646 Kanuri kby new Manga Kanuri  kr 4646 Kanuri knc new Central Kanuri  kr
4646 Kanuri krt new Tumari Kanuri  ku 4646 Kurdish ckb new Central Kurdish
ku 4646 Kurdish kmr new Northern Kurdish  ku 4646 Kurdish sdh new Southern
Kurdish  kv 4646 Komi koi new Komi-Permyak  kv 4646 Komi kpv new Komi-Zyrian
lah 4646 Lahnda hnd new Southern Hindko  lah 4646 Lahnda hno new Northern
Hindko  lah 4646 Lahnda jat new Jakati  lah 4646 Lahnda phr new
Pahari-Potwari  lah 4646 Lahnda pmu new Mirpur Panjabi  lah 4646 Lahnda pnb
new Western Panjabi  lah 4646 Lahnda skr new Seraiki  lah 4646 Lahnda xhe
new Khetrani  man 4646 Mandingo emk new Eastern Maninkakan  man 4646
Mandingo mku new Konyanka Maninka  man 4646 Mandingo mlq new Western
Maninkakan  man 4646 Mandingo mnk new Mandinka  man 4646 Mandingo msc
new Sankaran
Maninka  man 4646 Mandingo mwk new Kita Maninkakan  man 4646 Mandingo myq
new Forest Maninka  mg 4646 Malagasy bhr new Bara Malagasy  mg 4646 Malagasy
bjq new Southern Betsimisaraka Malagasy  mg 4646 Malagasy bmm new Northern
Betsimisaraka Malagasy  mg 4646 Malagasy msh new Masikoro Malagasy  mg 4646
Malagasy plt new Plateau Malagasy  mg 4646 Malagasy skg new Sakalava
Malagasy  mg 4646 Malagasy tdx new Tandroy-Mahafaly Malagasy  mg 4646
Malagasy txy new Tanosy Malagasy  mg 4646 Malagasy xmv new Antankarana
Malagasy  mg 4646 Malagasy xmw new Tsimihety Malagasy  mn 4646 Mongolian khk
new Halh Mongolian  mn 4646 Mongolian mvf new Peripheral Mongolian  ms
4646 Malay
(generic) btj new Bacanese Malay  ms 4646 Malay (generic) bve new Berau
Malay  ms 4646 Malay (generic) bvu new Bukit Malay  ms 4646 Malay (generic)
coa new Cocos Islands Malay  ms 4646 Malay (generic) jax new Jambi Malay  ms
4646 Malay (generic) max new North Moluccan Malay  ms 4646 Malay (generic)
meo new Kedah Malay  ms 4646 Malay (generic) mfa new Pattani Malay  ms
4646 Malay
(generic) mly new Malay (specific)  ms 4646 Malay (generic) mqg new Kota
Bangun Kutai Malay  ms 4646 Malay (generic) msi new Sabah Malay  ms 4646 Malay
(generic) vkt new Tenggarong Kutai Malay  ms 4646 Malay (generic) xmm
new Manado
Malay  mwr 4646 Marwari dhd new Dhundari  mwr 4646 Marwari mtr new Mewari
mwr 4646 Marwari mve new Marwari (Pakistan)  mwr 4646 Marwari rwr new Marwari
(India)  mwr 4646 Marwari swv new Shekhawati  mwr 4646 Marwari wry new
Merwari  no 4646 Norwegian nb 4646 Norwegian Bokmål  no 4646 Norwegian nn
4646 Norwegian Nynorsk  oc 4646 Occitan (post 1500) auv new Auvergnat  oc
4646 Occitan (post 1500) gsc new Gascon  oc 4646 Occitan (post 1500) lms new
Limousin  oc 4646 Occitan (post 1500) lnc new Languedocien  oc 4646 Occitan
(post 1500) prv new Provençal  oj 4646 Ojibwa ciw new Chippewa  oj 4646
Ojibwa ojb new Northwestern Ojibwa  oj 4646 Ojibwa ojc new Central Ojibwa
oj 4646 Ojibwa ojg new Eastern Ojibwa  oj 4646 Ojibwa ojs new Severn Ojibwa
oj 4646 Ojibwa ojw new Western Ojibwa  oj 4646 Ojibwa otw new Ottawa  om
4646 Oromo gax new Borana-Arsi-Guji Oromo  om 4646 Oromo gaz new West
Central Oromo  om 4646 Oromo hae new Eastern Oromo  om 4646 Oromo orc new
Orma  ps 4646 Pushto pbt new Southern Pashto  ps 4646 Pushto pbu new Northern
Pashto  ps 4646 Pushto pst new Central Pashto  qu 4646 Quechua cqu new Chilean
Quechua  qu 4646 Quechua qub new Huallaga Huánuco Quechua  qu 4646 Quechua
qud new Calderón Highland Quichua  qu 4646 Quechua quf new Lambayeque
Quechua  qu 4646 Quechua qug new Chimborazo Highland Quichua  qu 4646
Quechua quh new South Bolivian Quechua  qu 4646 Quechua quk new Chachapoyas
Quechua  qu 4646 Quechua qul new North Bolivian Quechua  qu 4646 Quechua qup
new Southern Pastaza Quechua  qu 4646 Quechua qur new Yanahuanca Pasco
Quechua  qu 4646 Quechua qus new Santiago del Estero Quichua  qu 4646
Quechua quw new Tena Lowland Quichua  qu 4646 Quechua qux new Yauyos Quechua
qu 4646 Quechua quy new Ayacucho Quechua  qu 4646 Quechua quz new Cusco
Quechua  qu 4646 Quechua qva new Ambo-Pasco Quechua  qu 4646 Quechua
qvc new Cajamarca
Quechua  qu 4646 Quechua qve new Eastern Apurímac Quechua  qu 4646 Quechua
qvh new Huamalíes-Dos de Mayo Huánuco Quechua  qu 4646 Quechua qvi new Imbabura
Highland Quichua  qu 4646 Quechua qvj new Loja Highland Quichua  qu 4646
Quechua qvl new Cajatambo North Lima Quechua  qu 4646 Quechua qvm new
Margos-Yarowilca-Lauricocha
Quechua  qu 4646 Quechua qvn new North Junín Quechua  qu 4646 Quechua qvo
new Napo Lowland Quechua  qu 4646 Quechua qvp new Pacaraos Quechua  qu 4646
Quechua qvs new San Martín Quechua  qu 4646 Quechua qvw new Huaylla Wanca
Quechua  qu 4646 Quechua qvz new Northern Pastaza Quichua  qu 4646 Quechua
qwa new Corongo Ancash Quechua  qu 4646 Quechua qwc new Classical Quechua
qu 4646 Quechua qwh new Huaylas Ancash Quechua  qu 4646 Quechua qws new Sihuas
Ancash Quechua  qu 4646 Quechua qxa new Chiquián Ancash Quechua  qu 4646
Quechua qxc new Chincha Quechua  qu 4646 Quechua qxh new Panao Huánuco
Quechua  qu 4646 Quechua qxl new Salasaca Highland Quichua  qu 4646 Quechua
qxn new Northern Conchucos Ancash Quechua  qu 4646 Quechua qxo new Southern
Conchucos Ancash Quechua  qu 4646 Quechua qxp new Puno Quechua  qu 4646
Quechua qxr new Cañar Highland Quichua  qu 4646 Quechua qxt new Santa Ana de
Tusi Pasco Quechua  qu 4646 Quechua qxu new Arequipa-La Unión Quechua  qu
4646 Quechua qxw new Jauja Wanca Quechua  raj 4646 Rajasthani bgq new Bagri
raj 4646 Rajasthani gda new Gade Lohar  raj 4646 Rajasthani gju new Gujari
raj 4646 Rajasthani hoj new Harauti  raj 4646 Rajasthani mup new Malvi  raj
4646 Rajasthani wbr new Wagdi  rom 4646 Romany rmc new Carpathian Romani
rom 4646 Romany rmf new Kalo Finnish Romani  rom 4646 Romany rml new Baltic
Romani  rom 4646 Romany rmn new Balkan Romani  rom 4646 Romany rmo new Sinte
Romani  rom 4646 Romany rmw new Welsh Romani  rom 4646 Romany rmy new Vlax
Romani  sc 4646 Sardinian sdc new Sassarese Sardinian  sc 4646 Sardinian sdn
new Gallurese Sardinian  sc 4646 Sardinian src new Logudorese Sardinian  sc
4646 Sardinian sro new Campidanese Sardinian  sh 4646 Serbo-Croatian bs 4646
Bosnian  sh 4646 Serbo-Croatian hr 4646 Croatian  sh 4646 Serbo-Croatian sr
4646 Serbian  sq 4646 Albanian aae new Arbëreshë Albanian  sq 4646 Albanian
aat new Arvanitika Albanian  sq 4646 Albanian aln new Gheg Albanian  sq 4646
Albanian als new Tosk Albanian  sw 4646 Swahili (generic) swc new Congo
Swahili  sw 4646 Swahili (generic) swh new Swahili (specific)  syr 4646
Syriac aii new Assyrian Neo-Aramaic  syr 4646 Syriac cld new Chaldean
Neo-Aramaic  tmh 4646 Tamashek taq new Tamasheq  tmh 4646 Tamashek thv
new Tahaggart
Tamahaq  tmh 4646 Tamashek thz new Tayart Tamajeq  tmh 4646 Tamashek
ttq new Tawallammat
Tamajaq  uz 4646 Uzbek uzn new Northern Uzbek  uz 4646 Uzbek uzs new Southern
Uzbek  yi 4646 Yiddish ydd new Eastern Yiddish  yi 4646 Yiddish yih new Western
Yiddish  za 4646 Zhuang ccx new Northern Zhuang  za 4646 Zhuang ccy
new Southern
Zhuang  zap 4646 Zapotec zaa new Sierra de Juárez Zapotec  zap 4646 Zapotec
zab new San Juan Guelavía Zapotec  zap 4646 Zapotec zac new Ocotlán Zapotec
zap 4646 Zapotec zad new Cajonos Zapotec  zap 4646 Zapotec zae new Yareni
Zapotec  zap 4646 Zapotec zaf new Ayoquesco Zapotec  zap 4646 Zapotec zai
new Isthmus Zapotec  zap 4646 Zapotec zam new Miahuatlán Zapotec  zap 4646
Zapotec zao new Ozolotepec Zapotec  zap 4646 Zapotec zaq new Aloápam Zapotec
zap 4646 Zapotec zar new Rincón Zapotec  zap 4646 Zapotec zas new Santo
Domingo Albarradas Zapotec  zap 4646 Zapotec zat new Tabaa Zapotec  zap 4646
Zapotec zav new Yatzachi Zapotec  zap 4646 Zapotec zaw new Mitla Zapotec
zap 4646 Zapotec zax new Xadani Zapotec  zap 4646 Zapotec zca new Coatecas
Altas Zapotec  zap 4646 Zapotec zoo new Asunción Mixtepec Zapotec  zap 4646
Zapotec zpa new Lachiguiri Zapotec  zap 4646 Zapotec zpb new Yautepec
Zapotec  zap 4646 Zapotec zpc new Choapan Zapotec  zap 4646 Zapotec
zpd new Southeastern
Ixtlán Zapotec  zap 4646 Zapotec zpe new Petapa Zapotec  zap 4646 Zapotec
zpf new San Pedro Quiatoni Zapotec  zap 4646 Zapotec zpg new Guevea De
Humboldt Zapotec  zap 4646 Zapotec zph new Totomachapan Zapotec  zap 4646
Zapotec zpi new Santa María Quiegolani Zapotec  zap 4646 Zapotec zpj
new Quiavicuzas
Zapotec  zap 4646 Zapotec zpk new Tlacolulita Zapotec  zap 4646 Zapotec zpl
new Lachixío Zapotec  zap 4646 Zapotec zpm new Mixtepec Zapotec  zap 4646
Zapotec zpn new Santa Inés Yatzechi Zapotec  zap 4646 Zapotec zpo new Amatlán
Zapotec  zap 4646 Zapotec zpp new El Alto Zapotec  zap 4646 Zapotec
zpq new Zoogocho
Zapotec  zap 4646 Zapotec zpr new Santiago Xanica Zapotec  zap 4646 Zapotec
zps new Coatlán Zapotec  zap 4646 Zapotec zpt new San Vicente Coatlán
Zapotec  zap 4646 Zapotec zpu new Yalálag Zapotec  zap 4646 Zapotec
zpv new Chichicapan
Zapotec  zap 4646 Zapotec zpw new Zaniza Zapotec  zap 4646 Zapotec zpx new San
Baltazar Loxicha Zapotec  zap 4646 Zapotec zpy new Mazaltepec Zapotec  zap
4646 Zapotec zpz new Texmelucan Zapotec  zap 4646 Zapotec zsr new Southern
Rincon Zapotec  zap 4646 Zapotec ztc new Lachirioag Zapotec  zap 4646
Zapotec zte new Elotepec Zapotec  zap 4646 Zapotec ztg new Xanaguía Zapotec
zap 4646 Zapotec ztl new Santiago Lapaguía Zapotec  zap 4646 Zapotec
ztm new San
Agustín Mixtepec Zapotec  zap 4646 Zapotec ztn new Santa Catarina Albarradas
Zapotec  zap 4646 Zapotec ztp new Loxicha Zapotec  zap 4646 Zapotec
ztq new Quioquitani-Quierí
Zapotec  zap 4646 Zapotec zts new Tilquiapan Zapotec  zap 4646 Zapotec ztt
new Tejalapan Zapotec  zap 4646 Zapotec ztu new Güilá Zapotec  zap 4646
Zapotec ztx new Zaachila Zapotec  zap 4646 Zapotec zty new Yatee Zapotec  zh
4646 Chinese cdo new Min Dong Chinese  zh 4646 Chinese cjy new Jinyu Chinese
zh 4646 Chinese cmn new Mandarin Chinese  zh 4646 Chinese cpx new Pu-Xian
Chinese  zh 4646 Chinese czh new Huizhou Chinese  zh 4646 Chinese czo new Min
Zhong Chinese  zh 4646 Chinese gan new Gan Chinese  zh 4646 Chinese
hak new Hakka
Chinese  zh 4646 Chinese hsn new Xiang Chinese  zh 4646 Chinese mnp new Min
Bei Chinese  zh 4646 Chinese nan new Min Nan Chinese  zh 4646 Chinese wuu
new Wu Chinese  zh 4646 Chinese yue new Yue Chinese  zza 4646 Zaza diq new
Dimli  zza 4646 Zaza kiu new Kirmanjki

Mark

---

From: John Cowan <[email-masked]>
Date: Tue, 20 Mar 2007 16:52:54 -0400

Mark Davis scripsit:

> http://www.sil.org/iso639-3/documentation.asp?id=ara as primary language
> tags is not a problem. The real problem is having either arb or ar-arb,
> since we already have the tag "ar" which in practice means the same
> thing, widely used in a great deal of software. Introducing either
> arb or ar-arb essentially introduces a synonym for ar, and synonyms
> just cause problems.

1) Introducing arb should not be lumped with introducing ar-arb, as a
filtering (as distinct from lookup) request for ar will find content
tagged ar-arb, but will not find content tagged arb.  That alone is
enough reason to forbid arb.

2) The argument that ar-arb and ar are in practice the same (an argument
inherently limited to written, rather than recorded spoken, documents)
proves too much.  In practice, fr and fr-FR are the same, but nobody
insists that fr-FR be forbidden, or that fr must not match fr-CA for
filtering purposes.

> Given the set of imperfect options, I think the best one is (a)
> enhance the registry, as I suggested, and (b) either don't encode
> arb, or encode and deprecate it (and do similar things for the other
> macrolanguages that are in widespread use), and (c) not have extlang.

Apart for the substantive objections to this procedure, it also leaves
us open to a (perfectly legitimate) charge of cherry-picking.  It is
not up to us to decide which particular macrolanguage to handle how.

[ak encompasses fat and twi; no encompasses nb and nn; sh encompasses bs,
hr, and sr]

I agree that these are annoyances, but I deny that they are anything
more than annoyances.

-- 
They do not preach                              John Cowan
  that their God will rouse them                [email-masked]
    A little before the nuts work loose.        http://www.ccil.org/~cowan
They do not teach
  that His Pity allows them                         --Rudyard Kipling,
    to drop their job when they damn-well choose.   "The Sons of Martha"

---

From: Addison Phillips <[email-masked]>
Date: Wed, 21 Mar 2007 10:55:53 -0700

> <http://www.sil.org/iso639-3/documentation.asp?id=ara> as primary
 > language tags is not a problem.

Agreed. Provided tag assignment is clear. In actuality, I would guess 
that nearly everyone using Arabic will use 'ar', regardless of the 
availability of specialized primary language subtags.

 > The real problem is having either arb or
 > ar-arb, since we already have the tag "ar" which in practice means the
 > same thing, widely used in a great deal of software. Introducing
 > either
 > arb or ar-arb essentially introduces a synonym for ar,

It would be confusing to exclude 'arb' but allow the enclosing 'ara' 
(better known as 'ar')---and all of the other enclosed languages. And 
other macro-languages don't have such neat/obvious relationships to an 
enclosed language. This would involve cherry-picking. I don't see why we 
would. I would tend to deprecate 'arb' or include a comment pointing out 
that 'ar' is a better choice.

 > ...and synonyms just cause problems.

Yes. Emphatically. And not using extlang creates the synonym problem 
writ large. That is, 'zh' and 'cmn' can both be used to identify the 
same documents ('zh' can also be used on documents that 'cmn' wouldn't 
be used on). Your solution requires implementations and users to have 
this external knowledge about some subtags. Extlang makes the same 
information visible in the tag.

> 
> Given the set of imperfect options, I think the best one is (a) enhance 
> the registry, as I suggested, and (b) either don't encode arb, or encode 
> and deprecate it (and do similar things for the other macrolanguages 
> that are in widespread use), and (c) not have extlang.

Enhancing the registry isn't a big deal. As others have pointed out, 
extlang's "Prefix" is essentially the same information.

For me to support not using extlang would require a very clear 
understanding of how to handle (tag) existing and future documents in 
affected languages. The general population will *not* understand what 
we've done and I'm afraid we'll have a mishmosh of incompatible tags. 
With extlangs, we might have different matching behaviors ("smart" vs. 
"stupid"), but tagging is relatively consistent (all Chinese documents 
begin with 'zh-', some have extlangs and some don't). You haven't 
explained this. I'm not at pains to make implementation easy if it 
results in the expense of making tag choice difficult.

Implementing extlang is not a lot of fun (it would be better if there 
were only one permitted, not three): I've emitted a string of expletives 
when dealing with them.

In my most recent code experiments, I've treated extlang (sequences) as 
part of the primary language and used a boolean to decide whether they 
are ignored (or not) in matching. This actually works reasonably well. 
For lookup I can have this behavior (in the default "don't ignore" mode):

zh-yue-Hant-HK
zh-yue-Hant
zh-yue
(root)

But I can also do the more elaborate:

zh-yue-Hant-HK
zh-Hant-HK
zh-yue-Hant
zh-Hant
zh-yue
zh
(root)

This last pattern is actually more like "zh-yue||zh", but the longer 
range is always first, so the fallback works as shown.

In filtering, using extended filtering in place of basic filtering 
achieves the same result. Users aren't forced to request both Cantonese 
and Chinese to get the same results they used to get from just 
requesting Chinese. That is, the range 'zh-Hant' produces Traditional 
Chinese, including Cantonese (zh-yue-Hant). If you want to be specific, 
include the extlang.

Addison