Subject: [Ltru] Re: extlang

From: "Doug Ewell" <[email-masked]>
Date: Fri, 31 Aug 2007 16:25:01 -0700

Randy Presuhn <randy underscore presuhn at mindspring dot com> wrote:

> Fallback has probably been over-sold.

I think at least two things have been oversold.  Fallback is definitely 
one of them.  As I've said so many times, after all the effort and delay 
we went through to generate RFC 4647, after everything we've created 
that might require RFC 3066-style matching and validation engines to be 
modified or rewritten, we are still hooked on the idea that 
remove-from-right truncation is of paramount importance, and that a 
mechanism like extlang that might exhibit unwanted results in some 
RFR-truncation scenarios is thereby fatally flawed.

The other thing that has been oversold is the identification of a 
macrolanguage with one of its encompassed languages.  From reading this 
thread, you would think people are making a conscious decision to tag 
Mandarin Chinese, and only Mandarin Chinese, as "zh", and tag other 
flavors of Chinese as "zh-yue" or "zh-hakka" or whatever.  Furthermore, 
you would think people are planning to make the same decision with 
respect to Arabic, explicitly equating "ar" with Standard Arabic and not 
with any other flavor, as soon as RFC 4646bis comes out with its 30-odd 
new Arabic subtags.

I don't think this sort of conscious decision is what drives the current 
tagging scenario.  Instead, consider the following:

1.  Most (not all) of the content that is currently tagged with BCP 47 
language tags tends to be written, not spoken, sung, etc.  This will not 
necessarily be the case forever, but it is likely the case today.

2.  Most written Chinese that is tagged tends to be either Mandarin 
Chinese, or written according to Mandarin conventions.

3.  Almost all written Arabic that is tagged tends to be Standard 
Arabic.

4.  The tags registered under RFC 3066, and grandfathered into RFC 4646 
for individual Chinese languages, including Mandarin, are often 
considered (incorrectly) to be exceptions to the "normal" tagging 
syntax, and are used relatively infrequently.  No such tags exist at all 
yet for individual Arabic variations.

Given this, it is natural that in current practice "zh" tends to 
correspond to Mandarin and "ar" tends to correspond to Standard Arabic. 
That doesn't mean we should "bake" this assumption into our solution. 
People are not saying, "I will use plain 'zh' for Mandarin but 'zh-yue' 
for Cantonese."  That's why "zh-cmn" was registered, to make this 
explicit.  "zh" is *not* explicit.

A lot of attention is being paid to the "fallback" scenario and the 
equation of a macrolanguage X with one specific flavor X1, such that the 
question being asked is whether all speakers of X2 and X3 and X4 can 
also comprehend X1, or each other.  This isn't how I understand 
macrolanguages at all.  There are contexts in which X1 and X2 and so 
forth are considered the same language, and others in which they are 
not.  Tagging the unified language as "X" for the contexts where it 
applies, and tagging the individual languages as "X-X1", "X-X2", and so 
forth, where they apply, reflects this relationship and establishes that 
"X" is sometimes a legitimate way to think of individual language X1. 
An informative "Macrolanguage" field doesn't do this, because it exists 
only in the Registry and not in the tag.

In object-oriented (OO) design, there is a concept called "is a" that I 
think describes the macrolanguage model well.  In OO design, you have a 
class called "vehicle" and then you have other classes such as "car," 
"boat," and "airplane" that are subclasses of "vehicle."  One would say 
that a car, boat, or airplane "is a" vehicle.  That doesn't imply that 
any of the subclasses are completely equivalent to the superclass; you 
would not say that a vehicle "is a" car, and you would certainly not 
presume that a car is always a suitable substitute ("fallback") for any 
other type of vehicle, even if it is the most common, and even if people 
sometimes use the word "vehicle" when they specifically mean "car."

*That* is what extlang "bakes into" the tag: the superclass-subclass 
relationship that Mandarin "is a" Chinese, and the notion that sometimes 
it is appropriate to think of it as "Mandarin" and sometimes as 
"Chinese."

Randy wanted specific wording.  There's too much specific to extlangs 
and macrolanguages to call out explicitly in OLD: and NEW: blocks, but I 
suppose my first choice, as I predicted before, would be to go back to 
what we had in draft-4646bis-06.  That also means not changing the rules 
to prohibit adding new extlangs in the future, which Addison and even 
John don't agree with, so I don't really expect this to happen.

My second choice, if we absolutely must jettison extlangs, would be to 
go all the way and not pretend to recognize the ISO 639-3 macrolanguage 
structure at all.  That means not having a Macrolanguage field, since as 
we are often reminded, it is not sufficient for so-called "fallback" to 
determine what language to serve to an arbitrary requester in case the 
requested language is unavailable.  If we don't expect software tools to 
parse and match tags intelligently instead of using Fred Flintstone RFR 
truncation, why would we expect them to consult a Macrolanguage field to 
see if there's a relationship of unspecified nature between X1 and X?

My third choice would be to get rid of extlangs but keep the 
Macrolanguage field.  I still don't agree with keeping both extlangs 
*and* the Macrolanguage field.  John says that one would be normative 
and govern the structure of the tag, and the other would be informative, 
and he and I understand this and so does probably everyone else on this 
list, but I still think it would cause colossal confusion among ordinary 
users who have not been thinking about this stuff for years.

At the very least, the new paragraph in Section 4.1 beginning "In 
particular" needs to be cleaned up to reflect whatever decision is made 
regarding extlangs.  Right now it refers to "languages such as 'yue'" 
and falling back from "yue-Hans-CN" to "zh-Hans-CN"; these "yue" 
references should be "zh-yue" if we decide to keep extlangs.

I'll go along with whatever decision is made, but I don't think one 
implementation from one of the possible language-tagging perspectives, 
with results characterized as "it just doesn't seem to work well," 
should be sufficient to reverse the tide of the past 3 years.

--
Doug Ewell · Fullerton, California, USA · RFC 4645 · UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru