Subject: [Ltru] extlang for users

From: Martin Hosken <[email-masked]>
Date: Tue, 27 May 2008 00:08:59 +0700

Dear All,

So, if we go with extlang, how should I tag some Mandarin text? Let's say I am creating content. I have no history of previous tagging and I have no idea whether the text is also in any other language?

zh-cmn or cmn?

And also please can someone explain why, because I'm going to have to teach this to a very confused set of users (not least of which is me).

Yours,
Martin
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Mon, 26 May 2008 14:12:25 -0400

Martin Hosken scripsit:

> So, if we go with extlang, how should I tag some Mandarin text? Let's
> say I am creating content. I have no history of previous tagging and
> I have no idea whether the text is also in any other language?

I don't understand that: if you know it's Mandarin, you surely know it
isn't anything else.  Text is not both Mandarin and Cantonese at the
same time (except for the highly artificial situation in Hong Kong of
reading Mandarin text word by word using Cantonese equivalents).

But let's say it's Mandarin.

> zh-cmn or cmn?

"cmn" is not an option if we use extlang, so your choice is between
"zh-cmn" and "zh"?  Both have been used in the past, though "zh" much
more so.  "zh-cmn" is theoretically more specific (it excludes all
non-Mandarin varieties); in practice, though, almost all texts tagged
simply "zh" are in Mandarin, partly because almost all Chinese texts
are in Mandarin anyhow.

-- 
We are lost, lost.  No name, no business, no Precious, nothing.  Only empty.
Only hungry: yes, we are hungry.  A few little fishes, nassty bony little
fishes, for a poor creature, and they say death.  So wise they are; so just,
so very just.  --Gollum        [email-masked]  http://ccil.org/~cowan
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 26 May 2008 22:02:37 +0200

On Tue, May 27, 2008 at 12:08:59AM +0700,
 Martin Hosken <[email-masked]> wrote 
 a message of 14 lines which said:

> So, if we go with extlang, how should I tag some Mandarin text?

I assume that the current plan is to go back to
draft-ietf-ltru-4646bis-09, the last one with extlangs.

> zh-cmn or cmn?

Section 2.2.2 of draft-ietf-ltru-4646bis-09 says "zh-cmn" (the
canonical tag for mandarin) and just adds:

   The
   language subtag 'zh' can still be used without an extended language
   subtag to label a resource as some unspecified variety of Chinese
   (which in practice will usually be Mandarin, the dominant variety of
   Chinese, but might also be some other variety).
 
> And also please can someone explain why, because I'm going to have
> to teach this to a very confused set of users (not least of which is
> me).

If someone understands the recent tide that switched the previous
consensus, please reply here :-)

PS: people who understand "fr" are welcome to read
<http://www.bortzmeyer.org/extlang-or-not-extlang.html>
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Martin Duerst <[email-masked]>
Date: Tue, 27 May 2008 16:13:16 +0900

><http://www.bortzmeyer.org/extlang-or-not-extlang.html>

Nice writeup. The only suggestion I have is to make the actual tags
and subtags stick out more, maybe with a light blue background or
something.

Regards,   Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: John Cowan <[email-masked]>
Date: Tue, 27 May 2008 10:04:11 -0400

> <http://www.bortzmeyer.org/extlang-or-not-extlang.html>

I fed it through Systran and fixed a few things.  My apologies for any
errors: I am no translator, and neither is Systran.

NOTE error in the original: for "zaa-zap" read "zap-zaa"; "zap" is the
macrolanguage tag.

Work group LTRU (Language Tag Registry Update) of the IETF is very late
in its project of standardization of the future labels of language,
those short character strings which make it possible to indicate the
language of a document or to specify the language which one wishes to use
on the Internet. One of the reasons of this delay is the endless debate on
"extlangs", these mechanisms making it possible to represent the concept
of "macrolanguage" introduced by the ISO standard 639-3. Approximately,
should Egyptian be labeled ar-arz or arz? And does Cantonese have to be
zh-yue or simply yue?

In the current standard, RFC 4646, language tags are made of several
subtags, each one identifying the language, the writing or the
country. The subtags which identify the language are drawn from ISO 639-1
or ISO 639-2, standards which place all languages on an equal footing.

But the world of human languages is complex. The traditional definition of
a language is the criterion of mutual comprehension. Despite differences
of accent, vocabulary and orthography, the English of Australia can be
understood by the Irish (sometimes with effort). It is thus the same
language, whose tag is "en". In the same way, so close relations who are
French and the Catalan, by their common origin, two speakers of these
two languages cannot understand each other.  They are thus correctly
two distinct languages, whose labels are "fr" and "ca".

Naturally, there exists a grey area: is Danish so different from
Norwegian? Moroccan Arabic from Tunisian? The case is all the more
difficult because certain languages are different in their oral forms
but less with the writing (this is precisely the case of Arabic). In one
way, there exists only one Arab language. In another, there are several
(which do not follow the national borders inevitably exactly). Sometimes,
history or politics contribute to scramble correct linguistic perceptions,
as in the case of Mandarin, often loosely called Chinese because it is
the language of the Chinese state.

To try to model this very complex world which was not conceived
rationally, SIL, the organization which was principally responsible for
the ISO 63 standard9-3 created a new concept, that of macrolanguages. A
macrolanguage is a single language according to certain criteria, and
a group of languages according to others. The two most famous examples
are Chinese (with the subtag "zh") and Arabic ("ar"). The registry of
ISO 639-3 thus indicates for each language if it is encompassed by a
macrolanguage. Cantonese is thus encompassed by Chinese. Immediately let
us note that the encompassed languages (sometimes called pejoratively
"microlanguages") are not dialects, they are distinct languages, typically
lacking mutual comprehensibility.

How to represent macrolanguages in the successor of RFC 4646? The
first idea, whose premises appear in RFC 4646, was to use a new concept,
Extended Language Subtag, the extlang. In this system, as envisaged in the
beginning, the first subtag identified the macrolanguage (or the language
itself for the more frequent cases where there was no macrolanguage)
and one or more extlangs followed it. Thus, Tunisian Arabic was ar-aeb
and Sierra de Juarez Zapotec was zap-zaa (zap being the subtag of the
Zapotec macrolanguage).

To understand the concept of this system, it should be seen that the
software which handles language tags, when it does not find a appropriate
language, search in general by removing the subtags from the right, where
the least significant subtags are (this mechanism is described in RFC
4647). Thus, if one requests from a search engine a az-Arab-IR document
(Azeri written in the Arab script as used in Iran), and that no document
corresponds to this tag, the software can check if it has az-Arab (by
giving up Iranian specificities) or even az (Azeri, whatever its other
characteristics). The extlangs stuck well to this model. A request for
"sq-als" (Tosk) would be thus truncated to "sq" (the macrolanguage for
Albanian), which would not be a bad mapping solution.

But with a more attentive examination of extlangs, several problems were
found. Initially, the method of mapping by truncating from the right
always did not give correct results. Nothing guarantees that the languages
encompassed by the same macrolanguage are mutually comprehensible, quite
to the contrary. Therefore, the "blind" mapping obtained by removing the
subtags from the right will not be satisfactory. Then, extlangs complicate
the model since they represent a new case to be specified, to implement
and explain. Lastly, extlangs can carry an erroneous message, that one
particular language of the group is the principal language. This message
can be desirable (majority of the Arabic-speaking people are very attached
to the concept  of a single Arabic language) or not, but it is always
delicate, since it places the languages in a hierarchy. Many discussions,
sometimes extensive, had animated the work group around these problems.

Thus LTRU had as of December 2007 given up using the extlangs and changed
the Internet-Drafts to implement this decision. In this new version,
the Language Subtag Registry kept the information of which languages
were encompassed by a macrolanguage, but the language tags were only
formed with the language itself. Tosk was thus tagged "als", Sierra de
Juarez Zapotec "zaa", and Mandarin "cmn".

But to the IETF the things do not happen always also simply. Five months
after this change, problems with the new version appeared, often raised
by people who had hardly contributed to the concrete labour, and said
nothing at the time of the abandonment of the extlangs. Whenever the group
around a macrolanguage comprises a dominant language (which is the case
of Chinese with the preponderance of Mandarin, or that of Arabic with
standard Arabic, but not the Zapotec case), many documents were already
tagged with the tag which became that of the macrolanguage. For example,
Mandarin resources should no longer be tagged "zh", as formerly, but
"cmn". What to do with the innumerable Mandarin documents which had been
tagged "zh" while following RFC 4646?

Ideally, the Language Subtag Registry would be enough to find good
solutions. But, in fact, the decisions to be taken by software often
depend on nonlinguistic criteria. For example, if a user configured as
his preferred language Breton, it is not unreasonable serve to him French
text. Not that the two languages are similar (the first is a Celtic
language and the second a Romance language) but because, in practice,
almost everyone who understands Breton also understands French. But one
cannot put this information (of significant size and which often changes)
in the registry! It would thus be necessary to be resigned, which does
not seem easy, not to have in the register of information allowing an
"intelligent" mapping.

After, again, a very hot debate, the LTRU group finally went into reverse
on May 26, 2008, returning to the extlangs. That requires returning to
a previous version of the Internet-Drafts, to re-examine documents and
implementations. And with no guarantee that the process does not start
again in a few months...

My personal view is that the system is unstable: one can put together a
good argument for the two solutions (see Mark Davis's document against
extlangs) but nothing is perfect, because human languages were not
conceived to facilitate the task of the IETF. It is been necessary to
adopt a solution and to stick to it, but the IETF decision mechanisms
do not make that easy.

-- 
Híggledy-pìggledy / XML programmers            John Cowan
Try to escape those / I-eighteen-N woes;        http://www.ccil.org/~cowan
Incontrovertibly / What we need more of is      [email-masked]
Unicode weenies and / François Yergeaus.
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 27 May 2008 20:45:57 +0200

> Norwegian?

While Bokmål and Danish as written languages are quite similar
(for historical reasons), just some minor differences in vocabulary,
spoken Danish and spoken Bokmål are quite different (and indeed
the dialects of Danish are also quite varying, to the extent that
some are nearly incomprehensible even to other Danes). A
macrolanguage "dab" encompassing Danish and Bokmål, anyone...
Or maybe one that covers "Scandinavian"? "Scandinavian" as an
individual language is taught in Iceland, since Icelandic is not
comprehensible by Scandinavians.

I'm not actually proposing to register the abovementioned putative
"macrolanguages" (neither as three- (in 639-3) nor five-letter
(LSR only) code). But a code for "Scandinavian" as an individual
language (used in Iceland & Scandinavia, for Icelanders
communicating with Scandinavians, not among Icelanders) may
be useful. A language ISO 639-3 seems to lack a code for...
 
	/kent k

---

From: John Cowan <[email-masked]>
Date: Tue, 27 May 2008 15:02:49 -0400

> comprehensible by Scandinavians.

Amazing.  Which of the four words for "girl" do they learn?
And do they speak with st/od, tones, or neither?

-- 
John Cowan    http://ccil.org/~cowan  [email-masked]
The Penguin shall hunt and devour all that is crufty, gnarly and
bogacious; all code which wriggles like spaghetti, or is infested with
blighting creatures, or is bound by grave and perilous Licences shall it
capture.  And in capturing shall it replicate, and in replicating shall
it document, and in documentation shall it bring freedom, serenity and
most cool froodiness to the earth and all who code therein.  --Gospel of Tux
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 27 May 2008 21:43:23 +0200

On Tue, May 27, 2008 at 10:04:11AM -0400,
 John Cowan <[email-masked]> wrote 

> NOTE error in the original: for "zaa-zap" read "zap-zaa"; "zap" is the

Fixed, thanks.

---

From: John Cowan <[email-masked]>
Date: Tue, 27 May 2008 16:06:07 -0400

> > NOTE error in the original: for "zaa-zap" read "zap-zaa"; "zap" is the

Great.  Are there any obvious errors in the English version?
I have only a de minimis grasp of French.

-- 
As you read this, I don't want you to feel      John Cowan
sorry for me, because, I believe everyone       [email-masked]
will die someday.                               http://www.ccil.org/~cowan
        --From a Nigerian-type scam spam
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Kent Karlsson" <[email-masked]>
Date: Wed, 28 May 2008 01:42:47 +0200

I haven't studied Scandinavian (the individual Icelandic language)
myself. All I know is that it is some kind of compromise/mix. The
few times I've heard it, it sounds like such a mix with a heavy
Icelandic accent. Sorry, I don't have further details.

	/kent k 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of John Cowan
> Sent: Tuesday, May 27, 2008 9:03 PM
> To: Kent Karlsson
> Cc: 'LTRU Working Group'
> Subject: Re: [Ltru] extlang for users
> 
> Kent Karlsson scripsit:
>