Subject: extlang (was Re: Suggested language for "mis" (Re: [Ltru] RE: ISO 639-2 decision: "mis"))

From: "Mark Davis" <[email-masked]>
Date: Sun, 17 Jun 2007 12:52:16 -0700

I'll move this to a different thread, so we can focus on the "mis" language
under that title.

You had said: "Are we almost ready to go with 4646bis except for "mis"?  Do
any other issues needs to be resolved?  If so, can we spend some time on
those issues as well?"

There are at least three open issues listed in
http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-06.html.

Extlang is one of them. I'll step back a bit, since it appears that we don't
have consensus about making a change in extlang from RFC 4646.

So we need to revive the discussion from back in March.

Mark

On 6/16/07, Doug Ewell <[email-masked]> wrote:
[snip]

You had

> >
> > Example - cleaning up extlang to still be reserved.
>
> Whoa, time out.  When did we agree NOT to use the extlang mechanism for
> new
> 639-3 languages that have a macrolanguage?  I know you wanted that, but
> where was the consensus that makes these wishes into "fixes that need to
> be
> made"?  This is why I wish we were using the issue tracker as we did for
> 3066bis.
>
> --
> Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
> http://users.adelphia.net/~dewell/
> http://www1.ietf.org/html.charters/ltru-charter.html
> http://www.alvestrand.no/mailman/listinfo/ietf-languages
>
>

-- 
Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 17 Jun 2007 22:33:26 +0200

On Sun, Jun 17, 2007 at 12:52:16PM -0700,
 Mark Davis <[email-masked]> wrote 
 a message of 86 lines which said:

> You had said: "Are we almost ready to go with 4646bis except for
> "mis"?

I believe so and I regret that "mis" takes so much energy. IMHO, it is
just a detail (I agree with Doug Ewell's "Type I vs. Type II errors"
message).

> http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-06.html.

Most are either actually closed (I do not think that there are still
serious proposals to use UTF-8 in the registry: even if I regret it,
it seems settled) or only details (additional information related to
Suppress-Script).
 
> Extlang is one of them.

And probably the only serious one. But, on the other hand, I want to
emphasize that ISO 639-3 was issued in february, that some users want
the new language codes now and that any delay in the release of the
RFC undermines the whole language tag effort.

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 17 Jun 2007 22:07:23 -0700

Mark Davis wrote:

> Extlang is one of them. I'll step back a bit, since it appears that we 
> don't have consensus about making a change in extlang from RFC 4646.
>

I agree.  For those who wish to follow the extlang discussion, the question 
is: for languages listed in ISO 639-3 that are not already present in the 
Registry, should we:

1.  add all of them as primary language subtags, or
2.  take the 5% that are encompassed by an ISO 639-3 macrolanguage, add them 
as extended language (extlang) subtags, and add the other 95% as primary 
language subtags?

The discussion in March started with this message from Mark:
http://www1.ietf.org/mail-archive/web/ltru/current/msg07288.html

Chase the "Follow-Ups" links to follow the thread.

My view is that we should keep the primary/extlang relationship that we have 
been planning since 2004 to reflect the ISO 639-3 macrolanguage 
relationship.  It allows identification of the individual languages while 
staying compatible with the way people have tagged them in the past, and in 
some cases with the way people will always perceive them (i.e. "Arabic" 
rather than 30 different flavors).  Expecting matching engines to have "a 
bit more smarts" so they can match "yue" with "zh" doesn't work for me when 
we can't even expect them to be smart enough to match "en-US" with 
"en-Latn-US".

But that's just my view, and the important thing is for the list to discuss 
the matter and reach a consensus so we can move forward.

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 18 Jun 2007 10:06:50 -0700

We added extlang to allow ourselves the freedom to make choices when 639-3
came along. We *very clearly did not define its meaning*, because we didn't
know what 639-3 was finally going to look like, nor did we have agreement on
what we should actually do. We had no commitment to using extlang.

We *already* had macrolanguages with ISO 639-2 in RFC 4646 and we *did not*
use extlang for them: examples are "sr", "hr", "nb", etc. We are not going
to (and cannot) be forcing users to encode nb as no-nb, nor sr as sh-sr.
These include

ak 4646 Akan fat 4646 Fanti  ak 4646 Akan tw 4646 Twi  no 4646 Norwegian nb
4646 Norwegian BokmÃl  no 4646 Norwegian nn 4646 Norwegian Nynorsk  sh 4646
Serbo-Croatian bs 4646 Bosnian  sh 4646 Serbo-Croatian hr 4646 Croatian  sh
4646 Serbo-Croatian sr 4646 Serbian
When we ("Google") tried implementing matching with "zh-yue" and others, we
found it made things *more* difficult, not less. Matching "zh" and "yue" is
not something you want to do automatically. Moreover, because of #2 we had
to have a mechanism for dealing with macrolanguages in RFC 4646 *anyway*.

Thus to make a proposed change from 4646 to use the extlang mechanism for
languages that have macrolanguages, we need a very compelling case that the
additional complication solves more problems than it creates. We haven't
seen that yet, and certainly have no consensus that it is the case.

So my proposal is essentially:

A. Keep the same structure and same philosophy as in RFC 4646. ISO 639-3
codes become primary language subtags, whether they have a macro language or
not. These are the edits I passed out previously, plus the removal of
2.2.2from the draft, and a few other changes (basically reverting to
RFC 4646 for
extlang).

B. (optional) Add a field Macrolanguage: to the language subtag registry.
The suggested text changes are:

* in 3.1.2.  Record Definitions, add in the optional fields

"Macrolanguage

   - For fields of type 'language', if there is a macro language in ISO
   639-3, then this field indicates that."

* After 3.1.9 add a new section.

Macrolanguage

The macro language field is added whenever a language has a corresponding
macro language in ISO 639-3. For example, 'sr' (Serbian) will have the
Macrolanguage value 'sh' (Serbo-Croation). This field is provided for use in
matching algorithms. For more information about the meaning and use of
macrolanguages, see [ISO 639-3].

* Then search for instances of "Suppress-Script" (just as a place to find
where field descriptions are) and make an addition of "Macrolanguage" if
appropriate, eg in the "LANGUAGE SUBTAG REGISTRATION FORM"

Mark

---

From: John Cowan <[email-masked]>
Date: Mon, 18 Jun 2007 21:34:33 -0400

Mark Davis scripsit:

> We added extlang to allow ourselves the freedom to make choices
> when 639-3 came along. We *very clearly did not define its meaning*,
> because we didn't know what 639-3 was finally going to look like,

Not so much.  We had an excellent idea of what 639-3 would both look and
actually be like when 4646 was finalized.  We couldn't include 639-3 or
extlangs because 639-3 itself was not yet final.

> nor did we have agreement on what we should actually do.

We had at least the consensus of silence; at least, I don't remember any
complaints at the time.  Remember that the development of 4646 started at
least a year before LTRU was formally created.

> We *already* had macrolanguages with ISO 639-2 in RFC 4646 and we *did
> not* use extlang for them: examples are "sr", "hr", "nb", etc.

(Rather, these are examples of languages *encompassed* by macrolanguages,
henceforth "encompassed languages".  I realize that's just a slip.)

> We are not going to (and cannot) be forcing users to encode nb as no-nb,
> nor sr as sh-sr.

Nobody has ever proposed that.  Language subtags coming from 639-1
or 639-2 will not change, even if 639-3 says they encode encompassed
languages.

> When we ("Google") tried implementing matching with "zh-yue" and others,
> we found it made things *more* difficult, not less.

Respectfully I suggest that because you ("Google") assign tags to incoming
rather than outgoing content, your use of BCP 47 is essentially private
rather than in interchange.  That makes it potentially important, but
definitely not prototypical.

> Matching "zh" and "yue" is not something you want to do
> automatically. Moreover, because of #2 we had to have a mechanism for
> dealing with macrolanguages in RFC 4646 *anyway*.

Very plausible in your circumstances.  But note that Yue (Cantonese)
content is *already* properly tagged "zh-yue" on precisely the theory
that's being applied to 639-3 encompassed languages.

I realize that this cause is probably not important to you, because
you can (comparatively) easily change all "zh-yue" tags to "yue", but
this is not the case for other users of BCP 47 on and off the Internet,
who will never even hear about the change.

> Thus to make a proposed change from 4646 to use the extlang mechanism
> for languages that have macrolanguages, we need a very compelling
> case that the additional complication solves more problems than it
> creates. We haven't seen that yet, and certainly have no consensus
> that it is the case.

On the contrary, the burden of persuasion is with you.  Tags like
zh-yue are already present in 4646, and it's up to you to provide a
convincing argument to deprecate them.  Furthermore, LTRU and its ad hoc
predecessor has been assuming the extlang structure since at least 2004.
Derailing that is what will take a "very compelling case".

> A. [Don't use extlangs.]

I continue in opposition to this.

> B. (optional) Add a field Macrolanguage: to the language subtag

I am not opposed to this, precisely because encompassed languages and
the corresponding macrolanguage cannot be identified syntactically.

-- 
May the hair on your toes never fall out!       John Cowan
        --Thorin Oakenshield (to Bilbo)         [email-masked]

---

From: Peter Constable <[email-masked]>
Date: Mon, 18 Jun 2007 20:08:26 -0700

From: John Cowan [mailto:[email-masked]]

> But note that Yue (Cantonese)
> content is *already* properly tagged "zh-yue" on precisely
> the theory that's being applied to 639-3 encompassed
> languages.

Just to clarify something, in case anyone isn't familiar or doesn't remember: note that the tag "zh-yue" was not proposed because a scheme based on 639-3 macrolanguages was anticipated. Rather, this and other tags were registered several years before the notion of macrolanguages was first considered for 639-3. In fact, the other way around: it was this and other similar registered language tags that led to the idea of macrolanguages for 639-3.

Peter

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 18 Jun 2007 23:07:15 -0700

John did a fine job articulating most of my position.  Anything not 
explicitly responded to here is +1.

John Cowan <cowan at ccil dot org> wrote:

> Mark Davis scripsit:
>
>> We added extlang to allow ourselves the freedom to make choices when 
>> 639-3 came along. We *very clearly did not define its meaning*, because 
>> we didn't know what 639-3 was finally going to look like,
>

In particular, we knew there would be macrolanguages and encompassed 
languages.  This was spelled out by Peter Constable on 2004-01-30 in a post 
which laid out the extlang concept for the first time:

http://www.alvestrand.no/pipermail/ietf-languages/2004-January/001658.html

> least a year before LTRU was formally created.

I thought it was stronger than that, that just about everyone's expectation 
and understanding was that extlangs were intended for encompassed languages, 
and that encompassed languages would be represented by extlangs.  Tags like 
"zh-cmn" were registered at ietf-languages specifically so they would 
conform to this well-understood model.

>> Matching "zh" and "yue" is not something you want to do automatically.

That's only half true.  All Cantonese is Chinese, and since it's only been 
possible up to this point to tag Cantonese as "zh-something", it's 
reasonable to assume that much of it will continue to be tagged as 
"zh-something" in the future.

> I am not opposed to this, precisely because encompassed languages and the 
> corresponding macrolanguage cannot be identified syntactically.

So, for example:

Type: language
Subtag: qvc
Description: Cajamarca Quechua
Description: Quechua, Cajamarca
Added: 20xx-xx-xx
Macrolanguage: que

Would content in Cajamarca Quechua be required to be tagged as "qvc", or 
would "qu" also be acceptable?  (We all know "qu" would be widely used.) 
Would matching engines be permitted to perform one- or two-way matching 
between "qvc" and "qu"?  What about users who expect we will go through with 
the "qu-qvc" model, and the Web pages that have already started listing such 
tags?  (See www.udhrinunicode.org, for instance.)  Would all this require an 
update to 4647?

---

From: John Cowan <[email-masked]>
Date: Tue, 19 Jun 2007 10:05:47 -0400

Doug Ewell scripsit:

> That's only half true.  All Cantonese is Chinese, and since it's only 
> been possible up to this point to tag Cantonese as "zh-something", it's 

Or, indeed, as "zh" simpliciter, which is probably how most of it is
tagged based on a simplistic subset of RFC 1766.

-- 
Híggledy-pìggledy / XML programmers            John Cowan
Try to escape those / I-eighteen-N woes;        http://www.ccil.org/~cowan
Incontrovertibly / What we need more of is      [email-masked]
Unicode weenies and / François Yergeaus.

---

From: Addison Phillips <[email-masked]>
Date: Tue, 19 Jun 2007 08:26:01 -0700

I agree with Doug and John's comments over the past couple of days.

I also agree with Mark that there are certain problems with extended 
language subtagging. Notably, there are matching implications. I was 
able to address that in my own implementations by:

- using extended rather than basic filtering
- treating the primary plus extlang sequence as atomic in lookup

That is, the fallback from zh-yue-Hant goes:

  zh-yue-Hant
  zh-yue
  (default)

I guess I agree about adding "Macrolanguage" to the registry, except 
that extlang, if preserved, already indicates this with "Prefix".

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: Peter Constable <[email-masked]>
Date: Tue, 19 Jun 2007 10:53:42 -0700

Can someone explain this statement in 3.3.2 (Extended Filtering) of RFC 4647:

"This is the reason why a wildcard in the extended language range is significant in the first position but is ignored in all other positions."

I don't see how "*" is said to be ignored in non-initial position of a range. Step 1 says, "Two subtags match if either they are the same when compared case-insensitively or the language range's subtag is the wildcard '*'." Steps 2 and 3 refer to subtags "match[ing]", where "match" is defined in step 1, and these steps apply only to non-initial positions. Step 3.A also makes explicit reference to "*" in the range subtag -- again, this being non-initial subtags.

I can't figure that sentence out.

Thanks
Peter

---

From: Addison Phillips <[email-masked]>
Date: Tue, 19 Jun 2007 11:03:06 -0700

Ignored is probably the wrong word. The sentence really means: "[t]his 
is the reason why a wildcard in the extended language range is only a 
significant difference in the first position but whose inclusion (or 
omission) has no effect elsewhere in the range."

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 19 Jun 2007 11:30:49 -0700

> extlangs because 639-3 itself was not yet final.

I think perhaps both of our "we"s are overstated. Since we disagree on this
point, neither my "we" nor your "we" is inclusive.

For example, you can't mean that every member of the working group had
looked it over thoroughly, and explored all the implementation
ramifications. I'd like to see a show of hands for those who did -- maybe
everyone except for me had, but I'd be rather surprised at that.

> least a year before LTRU was formally created.

I don't think everyone had reviewed it in detail, but perhaps the show of
hands would prove me the only one.

> henceforth "encompassed languages".  I realize that's just a slip.)

True, thanks.

> or 639-2 will not change, even if 639-3 says they encode encompassed

My point was that anyone who wants to deal with macro languages, has to
already deal with sr, hr, etc. as primary subtags, not as secondary. Thus
whatever mechanisms people have to work with sh, etc. can be extended to
other cases without the extlang mechanism.

> definitely not prototypical.

First of all, that isn't true (but thanks for the "respectfully"!) We
actually use language tags in a huge number of products, many of which have
APIs. We are not fully BCP 47 compliant, but are working towards that.

But the main point is, I want those people who have tried to implement --
professionally, not just in toy programs -- extlang to speak up about their
experiences. So if you want to speak to your experience implementing this
professionally, I'm all ears.

Addison mentions that fallback matching is not a problem. Mechanically it is
a no-brainer. But the *results* of that fallback are what we are having
problems with. And *that's* why I raise this issue, and call it "baking in
an assumption". Let me try to set this out, yet again. I'll call the
"languages encompassed by a macrolanguage" by the term "microlanguage", just
as a term.

I see two options. I may have captured the extlang reasoning incorrectly, so
please bear with me. And if there are other reasons for having extlang,
that'd be good to hear.

Premises

   1. The reason for making microlanguages be extlang instead of primary
   sublanguages is so that truncation-style matching will have better results.
   2. Fallback works when there is mutual comprehensibility (not
   necessarily 100%, but to a high degree); if you fallback to something that
   is not comprehensible, then fallback has failed.

Option A.

   1. Thus for extlang to work for microlanguages, the speakers of any
   microlanguages sharing a macrolanguage need to be able to understand the
   speakers of any other microlanguages sharing that macrolanguage.
   2. Peter and the ISO JAC can verify that A1 is true; that every
   speaker of Hakka can understand Jinyu; every speaker of Shihhi Arabic
   can understand Cypriot Arabic; and so on).
   3. Everything is hunky-dory.

Option B.

   1. The macrolanguage alone is always assumed to be the "standard", and
   that can be identified. That is, "zh" is always assumed to be Mandarin,
   "ar" is always assumed to be "Standard Arabic", etc. (That is, I think, the
   correct approach, but is *not* currently in the spec.)
   2. Thus for extlang to work for microlanguages, the speakers of any
   microlanguages sharing a macrolanguage need to be able to understand the
   speakers of the standard used for that macro language.
   3. Peter and the ISO JAC can verify that B21 is true; that every
   speaker of Hakka can understand Mandarin; every speaker of Shihhi
   Arabic can understand Cypriot Arabic, and so on).
   4. Everything is hunky-dory.

If both A and B are not plausible, and we can't find something other
compelling reason to have extlangs, it is *far* better to add the
Macrolanguage field to the registry, and let people implement their own
matching making use of that AND other factors.

After all, it is trivial to make a 4647bis that adds an optional step for
microlanguages, which is that when you get to a microlanguage, the next step
is to look at its macrolanguage before falling back to the default. That has
the same result (and same problems) as extlang, but is something that is not
baked into the standard -- is something that people can implement if they
want without impacting matching for everyone else.

> who will never even hear about the change.

These are irregular tags anyway, and can stay irregular tags afterwards. We
and everyone else already have to deal with equivalences with grandfathered
and irregular tags anyway; these are not a real problem.

> Derailing that is what will take a "very compelling case".

I disagree strongly. If you can't make a compelling case that extlang will
make BCP 47 better instead of worse, and won't even look at the reasons not
to do it, nor even bother to set out a case for it, then why should we add
it?

> the corresponding macrolanguage cannot be identified syntactically.

Good.

--
>         --Thorin Oakenshield (to Bilbo)         [email-masked]

I hope not....

---

From: John Cowan <[email-masked]>
Date: Tue, 19 Jun 2007 14:51:53 -0400

> that extlang, if preserved, already indicates this with "Prefix".

Except for the cases where the encompassed language is
a 639-2 language.

-- 
He played King Lear as though           John Cowan <[email-masked]>
someone had played the ace.             http://www.ccil.org/~cowan
        --Eugene Field

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 19 Jun 2007 22:58:55 +0200

On Tue, Jun 19, 2007 at 08:26:01AM -0700,
 Addison Phillips <[email-masked]> wrote 

> that extlang, if preserved, already indicates this with "Prefix".

Let me check that I understand. The only way, currently, to see if a
language is a macrolanguage is to check if there exists at least one
extlang which references it in its Prefix field? Correct?

So, in SQL:

SELECT Languages.code AS macrolanguage, count(Extlangs.code) AS Extlangs 
      FROM Languages,Extlangs WHERE Extlangs.prefix = Languages.code 
      GROUP BY Languages.code ORDER BY Languages.code; 

Which yields 54 macrolanguages in 4645bis, zap (Zapotec) being the one
with the most extlangs, 58 (if we do not count the sign languages).

Am I correct? (Sorry for using the IETF for my e-learning but I prefer
to check before commenting.)

---

From: Addison Phillips <[email-masked]>
Date: Tue, 19 Jun 2007 14:14:42 -0700

> to check before commenting.)

Essentially correct.

Note that the proposals I've seen weren't to add pointers in the 
macrolanguage's record to the extlangs, but rather an additional pointer 
in the extlang to the macrolanguage. That strikes me as pointless except 
for the grandfathered ISO 639-2 codes enclosed by macrolanguages that 
can't be extlang.

I take it you're suggesting (or interpreted the proposal) to mean that 
the macrolanguage would contain a pointer or pointers to its 
sublanguages. Hence something like:

%%
Type: language
Subtag: zh
Description: Chinese
...
Encloses: yue,cmn,.....
%%

Is that what you had in mind? Note that this type of field could apply 
even when various grandfathering rules (hello "sh", etc.) don't permit 
the use of extlangs.

Addison

---

From: John Cowan <[email-masked]>
Date: Tue, 19 Jun 2007 17:22:27 -0400

> for the grandfathered ISO 639-2 codes enclosed by macrolanguages that 

Also 639-3 languages that aren't encompassed now but become so in
future.

-- 
John Cowan    http://ccil.org/~cowan    [email-masked]
Economists were put on this planet to make astrologers look good.
        --Leo McGarry

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 19 Jun 2007 14:23:32 -0700

Hi -

> From: "Addison Phillips" <[email-masked]>
> To: "Stephane Bortzmeyer" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>; "Doug Ewell" <[email-masked]>
> Sent: Tuesday, June 19, 2007 2:14 PM
> Subject: Re: extlang (was Re: Suggested language for "mis" (Re: [Ltru] RE:ISO 639-2 decision: "mis"))
... 
...

> > extlang which references it in its Prefix field? Correct?

If whether a language is a macrolanguage is an important property,
then it is worth discussing whether it should be flagged in its
record, rather than requiring it to be computed by examining all
other records.

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 19 Jun 2007 14:28:58 -0700

Hi -

As a technical contributor...

> From: "John Cowan" <[email-masked]>
> To: "Addison Phillips" <[email-masked]>
> Sent: Tuesday, June 19, 2007 2:22 PM
...
...

If there is a real expectation that this will happen, this
will be critical to the extlang discussion, due to the
implications for stability in encoding.

Randy

---

From: Addison Phillips <[email-masked]>
Date: Tue, 19 Jun 2007 14:34:25 -0700

> implications for stability in encoding.

Yes, although I think we dealt with that: the encoding is stable and the 
newly enclosed language remains a primary language. Such an eventuality 
would be supremely icky.

There was also a proposal (I think I made it) to limit extlangs to "at 
4646bis birth only"---that is, only currently enclosed languages that 
are otherwise unencoded would be extlangs. This gets us the (for 
example) the enclosed Chinese dialects (etc.), but not any future 
extlangs. Future registrations would be primary language subtags only.

Addison

---

From: John Cowan <[email-masked]>
Date: Tue, 19 Jun 2007 17:37:50 -0400

Randy Presuhn scripsit:

> If there is a real expectation that this will happen, 

I think it will.

> this will be critical to the extlang discussion, due to the

I don't see why.  Stability says that once a language subtag,
always a language subtag: nothing is demoted to extlang subtag
status.  So languages that are encompassed in 639-2 today
have the same status (language subtag) as languages encompassed
in 639-3 later on.

The more serious case is when an encompassed language is no
longer encompassed or its macrolanguage is changed.
We'd be free to change the proposed Macrolanguage: field,
but the Prefix: field and Type: fields would have to remain.

-- 
Dream projects long deferred             John Cowan <[email-masked]>
usually bite the wax tadpole.            http://www.ccil.org/~cowan
        --James Lileks

---

From: John Cowan <[email-masked]>
Date: Tue, 19 Jun 2007 17:45:13 -0400

> extlang which references it in its Prefix field? Correct?

Alternatively, to look back at 639-3.  There are currently two
such properties in 639-6 that are not represented in the LSR: scope
(individual, macrolanguage, collective) and type (living, [recently]
extinct, ancient, historic [stage of a living language], constructed).

I am neither for nor against including these properties in the LSR.
They aren't necessary for validation, but they are probably useful in
some circumstances.

-- 
John Cowan                              <[email-masked]>
            http://www.ccil.org/~cowan
                .e'osai ko sarji la lojban.
                Please support Lojban!          http://www.lojban.org

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 19 Jun 2007 14:57:32 -0700

Hi -

> To: "Randy Presuhn" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Tuesday, June 19, 2007 2:34 PM
> Subject: Re: [Ltru] Re: extlang
...
...

I think it's important to keep in mind the "division of labor" between
the ltru WG and the function of the [email-masked] list.
Here in the WG, we need to ensure that the registry and registration
process ensure that the right bits get recorded in the right spots,
and that the resulting language tag structure works.  We only have to
agree that there are cases of "enclosing languages" that make it
worthwhile to represent this property in the regsitry, and to account
for in in our matching algorithms.

Deciding which specific languages are to be given such treatment
does not belong here.  We only need to concern oursevles with the
question of whether it is a requirement (that seems to be agreed).
If we cannot agree on a uniform way to handle these cases for the
big batch of tags we'd like add, based entirely on the data from
the standards from which we're importing this information, then
I suggest that those details would most properly be hashed out
on the ietf-languages list on a case-by-case basis.

Randy

---

From: Peter Constable <[email-masked]>
Date: Tue, 19 Jun 2007 15:01:24 -0700

[Mark: the quoting behavior of your mail client is not consistent and extremely hard to follow.]

From: Mark Davis [mailto:[email-masked]]
Sent: Tuesday, June 19, 2007 11:31 AM

> Premises
> 1. The reason for making microlanguages be extlang
> instead of primary sublanguages is so that
> truncation-style matching will have better results.
> 2. Fallback works when there is mutual comprehensibility (not necessarily 100%, but to a high degree); if you fallback to something that is not comprehensible, then fallback has failed.

I'd put this differently. Given macro ID "xxx" and encompassed micro ID "yyy":

1. There is no direct connection between primary subtags "xxx" and "yyy"; a connection would have to be maintained by tables in matching processes. But, if "yyy" always required "xxx" as a prefix, then that relationship is carried in the tag itself, and because truncation gets used in matching (it's part of basic filtering, extended filtering and lookup in 4647), a relationship falls out from the matching process.

2. The relationship is important because there is a significant level of existing usage of "xxx", and we want to allow for future usage of "yyy".

The point about existing usage is significant here, I think. If "xxx" and "yyy" were introduced at the same time and users adopted "yyy" while "xxx" never took off, then there's no particular need to relate them. But if both are going to get used (i.e. "xxx" would get used apart from "yyy", and "yyy" would also get used -- whether that is with or without "xxx"), then we probably want the relationship to be captured.

> Option A.
> 1. Thus for extlang to work for microlanguages, the
> speakers of any microlanguages sharing a macrolanguage
> need to be able to understand the speakers of any
> other microlanguages sharing that macrolanguage.
> 2. Peter and the ISO JAC can verify that A1 is true...

A.2 may be a little more than JAC can guarantee. The idea I worked with is that macrolanguages are coded because in some application the group of microlanguages are being coded as one -- they are not distinguished. Presumably that would happen because they are mutually intelligible, though that may not always be the case.

I don't assume that Mandarin and Cantonese are mutually intelligible at a functional level (though perhaps in their written forms they may be); appropriately or not -- perhaps an accident of history -- a single ID "zh" has been used for both. What we need to do is to be able to relate *as appropriate* "zh" content with "Cantonese", "Mandarin", etc., or Cantonese, etc. content with requests for "zh". Similarly for other cases in which -- for whatever reason -- things are, in practice, sometimes split but sometimes clumped.

> Option B.
> 1. The macrolanguage alone is always assumed to be
> the "standard", and that can be identified. That is,
> "zh" is always assumed to be Mandarin, "ar" is
> always assumed to be "Standard Arabic", etc. (That
> is, I think, the correct approach, but is *not*
> currently in the spec.)

Of course, this assumes that for macro/micro cases there always *is* one variety that can be considered *the* major variety.

When Gary Simons first began to analyze 639-2 with a view to how to map it onto Ethnologue, the original operational principles we established looked for just that -- we referred to it as the "major language variety (MLV) principle". (See http://www.sil.org/silewp/2002/SILEWP2002-004.pdf, page 7.) The idea was that we would equate the existing 639-2 ID with *the* major variety, when one could be identified. Of course, we immediately encountered a reality that there isn't always *one* major variety. See http://www.ethnologue.com/14/iso639/analysis.asp#U for cases we found in our initial analysis that we could not resolve.

I had expected we would try to work with the JAC to come up with some resolution for those cases. That was until the macrolanguage concept occurred to me. And a key thing that prompted it was the pre-existing IANA registrations involving zh: we could not equate "zh" with Mandarin because in existing usage it was clearly being used for Mandarin, Cantonese, Minnan, etc. An available alternative was to consider "zh" to be a collection (and that was, in fact, what we did in our initial analysis), but that just didn't reflect the way it was actually being used: "zh" was in widespread use as though it represented an individual language. Somehow, I had to marry those two: in use as though it represents an individual language, but also in use for multiple languages. Hence the prototype for macrolanguage.

Now, you're introducing the possibility of having the one-language/many-language dichotomy while also maintaining the MLV principle. (Of course, the most widespread use for an *individual* language would have involved for Mandarin.) Of course, that's not formalized in 639-3, but that doesn't block it from being a useful idea. One issue that would need to be worked out is when "xxx" as an individual language is the MLV-preferred microlanguage, when it's the undifferentiated group of microlanguages, or if this difference matters in practice.

But there's also the other issue: not all macrolanguages have one microlanguage that's clearly picked out by the MLV principle.

Peter

---

From: Peter Constable <[email-masked]>
Date: Tue, 19 Jun 2007 15:05:51 -0700

It may help to consider realistic scenarios for future changes in 639, realistic scenarios for usage of language tags, and weigh the value of comparison between two different tags that may be used for comparable content.

Peter

-----Original Message-----
From: Randy Presuhn [mailto:[email-masked]]
Sent: Tuesday, June 19, 2007 2:29 PM
To: LTRU Working Group
Subject: [Ltru] Re: extlang

Hi -

...
...

Randy

---

From: Peter Constable <[email-masked]>
Date: Tue, 19 Jun 2007 15:11:54 -0700

Is there a possibility of an encompassed language getting changed not to be encompassed? If "xxx" is a macrolanguage and "yyy" an encompassed "microlanguage" (to use Mark's term), I could perhaps see "xxx" getting deprecated, but "yyy" getting dissociated from "xxx"? It really shouldn't happen: there should have been *some* good reason for associated "yyy" with "xxx" in the first place, and that shouldn't go away.

Peter

-----Original Message-----
From: John Cowan [mailto:[email-masked]]
Sent: Tuesday, June 19, 2007 2:38 PM
To: Randy Presuhn
Cc: LTRU Working Group
Subject: Re: [Ltru] Re: extlang

I think it will.

---

From: John Cowan <[email-masked]>
Date: Tue, 19 Jun 2007 20:38:20 -0400

Peter Constable scripsit:

> Is there a possibility of an encompassed language getting changed not to
> be encompassed? If "xxx" is a macrolanguage and "yyy" an encompassed
> "microlanguage" (to use Mark's term), I could perhaps see "xxx"
> getting deprecated, but "yyy" getting dissociated from "xxx"? It
> really shouldn't happen: there should have been *some* good reason
> for associated "yyy" with "xxx" in the first place, and that shouldn't
> go away.

Human error would be the obvious candidate.  "Oh, we thought Foovian
was part of the Slobbovian macrolanguage, but that was because Dr. X
put the wrong code on the form back in 2008."

-- 
John Cowan    <[email-masked]>     http://www.ccil.org/~cowan
But no living man am I!  You look upon a woman.  Eowyn I am, Eomund's daughter.
You stand between me and my lord and kin.  Begone, if you be not deathless.
For living or dark undead, I will smite you if you touch him.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 20 Jun 2007 14:31:35 +0900

At 03:30 07/06/20, Mark Davis wrote:

>But the main point is, I want those people who have tried to implement -- professionally, not just in toy programs -- extlang to speak up about their experiences. So if you want to speak to your experience implementing this professionally, I'm all ears. 

Hello Mark,

I seem to remember that you said that you tried to implement them
at Google and found them difficult. But I don't remember you having
explained what exactly the problem is. If you have explained that,
could you point to the relevant message? If not, would you care
to give some more details?

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: John Cowan <[email-masked]>
Date: Wed, 20 Jun 2007 13:15:02 -0400

Mark Davis scripsit:

> already deal with sr, hr, etc. as primary subtags, not as secondary. 

It's not a MUST, it's a MAY.  Matchers are allowed to take into account
any information outside the purely syntactic match algorithms that they
happen to have -- and find useful  For example, a fallback from Scots
to English probably makes sense everywhere -- if you know Scots, you
know English.  On the other hand, a fallback from Swedish to Finnish
would be a Really Bad Idea, except within Finland, where it might make
all the sense in the world.  (Falling back from English to Scots or
Finnish to Swedish essentially never makes sense.)

So while there is no reason to prevent such nonsyntactic matching,
and 4647 explicitly licenses it, there is no reason to require that
every matcher use it either.  The point of the extlang tags, like their
currently-grandfathered predecessors, is to make additional matches easy.

>   sublanguages is so that truncation-style matching will have better 

Agreed.

>   necessarily 100%, but to a high degree); if you fallback to something
>   that is not comprehensible, then fallback has failed.

Two caveats: (a) fallback is one-way, so *mutual* comprehensibility
is not required; (b) Mohawk and English aren't comprehensible at all,
mutually or asymmetrically, but it so happens that the few hundred
Mohawk-speakers know English too.

Also, fallback is a matter of best effort; it does not have to work in
every case to be useful in many cases.  In particular, a failed fallback
leaves you no worse off than before.

>   microlanguages sharing a macrolanguage need to be able to understand
>   the speakers of any other microlanguages sharing that macrolanguage.

Not necessarily all, though the more the better, of course.

>   can understand Cypriot Arabic; and so on).

Of course that's not the case, as you must know.  Even if true in any
particular case, "every speaker" is an inappropriate standard.  If there
are one or two ancient Mohawks who don't have sufficient command of
English, it can't be helped.

>   3. Everything is hunky-dory.

"Hunky-dory" is the wrong standard.

> Option B.

This doesn't differ much from Option A, except that it changes the
semantics of macrolanguages in a way inconsistent with 639-3:

>   1. The macrolanguage alone is always assumed to be the "standard", 

Otherwise my comments above apply.

> After all, it is trivial to make a 4647bis that adds an optional step
> for microlanguages, which is that when you get to a microlanguage,
> the next step is to look at its macrolanguage before falling back to
> the default. That has the same result (and same problems) as extlang,
> but is something that is not baked into the standard -- is something
> that people can implement if they want without impacting matching for
> everyone else.

True enough, but it disregards the behavior of the large number of
naive matchers already out there that lack nonsyntactic information.
It also disregards the well-chosen grandfathered tags (which will become
for the most part redundant in 4646bis along the current lines), which
were picked precisely because they worked tolerably, if not perfectly,
with naive matchers.  This is the primary argument for using extlangs:
they are a conservative extension of what has already been done and what
tags have already been assigned to documents.

Your argument also proves too much: we find it useful to fall back from
az-Cyrl, az-Latn, and az-Arab to plain az (or vice versa for filtering)
without requiring that these be mutually intercomprehensible (your
Option A) or that one of them is the standard (your Option B).  The
results may be less than ideal, but we live with it.

> We and everyone else already have to deal with equivalences with
> grandfathered and irregular tags anyway; these are not a real problem.

Again, every one else does not *have to* deal with them; they MAY
treat them just like all other tags, at the expense of missing some
reasonable matches.  4647 makes such matches entirely optional.

> I disagree strongly. If you can't make a compelling case that extlang
> will make BCP 47 better instead of worse, and won't even look at the
> reasons not to do it, nor even bother to set out a case for it, then
> why should we add it?

It's true, as Dr. Johnson said: "Of an opinion which is no longer
doubted, the evidence ceases to be examined."  I've done my best above.
But I don't understand yet how extlangs make BCP 47 worse, and I think
they actually help given legacy documents and legacy matchers.

-- 
John Cowan      [email-masked]        http://www.ccil.org/~cowan
        Is it not written, "That which is written, is written"?

---

From: "Doug Ewell" <[email-masked]>
Date: Wed, 20 Jun 2007 22:24:51 -0700

As one of the people who got this thread started, I'm not ignoring it or 
anyone's position on it.  Things just suddenly got very busy for me.  I 
promise (or threaten, depending) to jump in as soon as I have more than two 
consecutive minutes to think about it.

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 21 Jun 2007 08:19:34 -0700

I too will be very busy until the end of the month, and won't have a chance
to do much on this until then. MD

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 21 Jun 2007 17:41:04 +0200

On Tue, Jun 19, 2007 at 02:14:42PM -0700,
 Addison Phillips <[email-masked]> wrote 
 a message of 58 lines which said:

> I take it you're suggesting (or interpreted the proposal) to mean
> that the macrolanguage would contain a pointer or pointers to its
> sublanguages.

No, I was not suggesting it myself but it is a good opportunity to
discuss it.

> Encloses: yue,cmn,.....

It seems reasonable to enumerate a macrolanguage "members "in the
macrolanguage itself. But I see some issues with this proposal:

1) What will be the stability rules for Encloses? Can an extlang be
dropped from a Encloses field? (if advances in linguistics show that
it was wrongly added here, for instance.) And can an extlang be added
to an Encloses? If yes (which seems reasonable), it means that the
macrolanguage entry can change quite often (consider Zapotec with its
dozens of enclosed extlangs).

2) It introduces a redundancy in the registry, an information which
was already there and could be computed from other records. Randy
Presuhn noticed it can be seen as a good thing ("it should be flagged
in its record, rather than requiring it to be computed by examining
all other records"). But more redundancy means more possibilities to
introduce contradictions.

3) 4646bis does not indicate an upper bound for the length of a field
("The content of this field is not restricted, except by [...]
reasonable practical size limitations."). Zapotec's "Encloses:" will
be at least 232 bytes long and sign langauges will be worse. While
there is a long discussion of length issues for tags, it seems there
is none for registry-parsing applications. I suspect that some of them
may have hard-wired limits for field lengths.

---

From: Addison Phillips <[email-masked]>
Date: Thu, 21 Jun 2007 08:58:54 -0700

Stephane Bortzmeyer wrote:
> 1) What will be the stability rules for Encloses? 

It's informative and thus could be changed. However, extlangs are 
permanent and stable (including their Prefix relationship), so they 
cannot be removed. Note that Encloses describes the function of the 
subtag, not necessarily the linguistic relationship (although in 
practice we hope that both are true).

> Can an extlang be
> it was wrongly added here, for instance.)

I would assume that an extlang can't be dropped from Encloses, since 
stability rules prevent the Prefix or status of an extlang from changing.

  And can an extlang be added
> dozens of enclosed extlangs).

You mean a new extlang? Of course new extlangs could be added as ISO 
639-3 assigns new codes. But there is no such thing as an existing 
extlang acquiring a new Prefix (and thus a new Encloses relationship). 
It is banned by stability rules.

> introduce contradictions.

The extlang records have a computable relationship. The main problem 
will be grandfathered primary language subtags that are enclosed by 
other subtags linguistically but not in terms of tagging practice. That 
is, not all possible macrolanguage relationships are described by or can 
be computed from entries in the registry.

> 3) 4646bis does not indicate an upper bound for the length of a field

There isn't one.

> may have hard-wired limits for field lengths.

Note that some comments approach or even exceed the sizes you note above.

I would tend to propose that we make Encloses a single field with 
structure simply because I find the multiple repetition of the same 
field unaesthetic. Of course, we went the other direction with Prefix, 
so there is some precedent for repeating the field.

Addison

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 21 Jun 2007 23:10:31 +0200

On Thu, Jun 21, 2007 at 08:58:54AM -0700,
 Addison Phillips <[email-masked]> wrote 
 a message of 73 lines which said:

> You mean a new extlang?

Yes. 

With Encloses: or a similar field, adding a new record (the new
extlang) will change *another* record (its macrolanguage), which is
questionable.

---

From: [email-masked]
Date: Thu, 21 Jun 2007 14:16:26 -0700

On the langtag.net site, it says, "List of grand-fathered *(tags which are 
no longer legal but maintained for compatibility*." 

Are zh-cmn and the other tags on the list now illegal? Or are they 
expected to become redundant as was previously discussed?

Regards,

Karen Broome
Metadata Systems Designer
Sony Pictures Entertainment
310.244.4384

---

From: Addison Phillips <[email-masked]>
Date: Thu, 21 Jun 2007 15:14:14 -0700

> extlang) will change *another* record (its macrolanguage), which is

I don't think it's questionable. Inserting a new record that obsoletes 
an existing item can cause deprecation, change from 
grandfathered->redundant, or other changes.

It *does* mean that inserting an extlang requires two records to be 
submitted to IANA, instead of just one.

Ultimately, though, the question is whether having the information adds 
value in the registry in proportion to the disruption that adding more 
fields, maintenance rules, and so forth adds.

Addison

---

From: Addison Phillips <[email-masked]>
Date: Thu, 21 Jun 2007 15:16:44 -0700

No. The text on langtag.net is wrong.

Grandfathered tags are always legal. These particular ones may become 
redundant (assuming Mark's idea of not implementing extlangs doesn't go 
forwards). But they'll always be legal.

I think what Stephane means is "tags which would otherwise be illegal 
but are maintained for compatibility".

Addison

[email-masked] wrote:
> 
> On the langtag.net site, it says, "List of _grand-fathered_ 
> <http://www.langtag.net/registries/lsr-grandfathered.txt> *(tags which 
> are no longer legal but maintained for compatibility*."
> 
> ------------------------------------------------------------------------
>

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 22 Jun 2007 09:27:07 +0200

On Thu, Jun 21, 2007 at 02:16:26PM -0700,
 [email-masked] <[email-masked]> wrote 
 a message of 59 lines which said:

> On the langtag.net site, it says, "List of grand-fathered *(tags
> which are no longer legal but maintained for compatibility*."

Fixed. Sorry.