Subject: [Ltru] extlang

From: "Mark Davis" <[email-masked]>
Date: Tue, 28 Aug 2007 14:59:57 -0700

On 8/28/07, John Cowan <[email-masked]> wrote:
>
> Mark Davis scripsit:
>
> > Just to open up the discussion, the biggest problem with this version is
> > that it uses a substantial new mechanism (extlang) which was not used in
> RFC
> > 4646, and for which there is no consensus in this group to use in RFC
> > 4646bis.
>
> As far as the record appears, there is no consensus because you are
> single-handedly blocking it.

Not single-handedly: Addison has also commented on it. And on the other
side, I've heard 3 strong proponents. So not exactly overwhelming either
way.

> We haven't removed extlang from the draft yet,
>
> "We haven't yet" implicatures "We will", as you well know.  Its use here
> is quite inappropriate.

I think you are being overly sensitive here. I clearly don't want extlang
in, but the "yet" is only indicating that it is still in the draft. And
leaving it in until we come to a conclusion is reasonable, since it is less
work if we decide we want it.

> but the people that want this new mechanism need to provide
> > justification,
>
> This mechanism has long been planned, but was held up pending the
> publication and stabilization of ISO 639-3.  It is you who wish to change
> the plan, and therefore you who bear the burden of persuasion.

I've said before, and will say again: it was never a foregone conclusion. I
as one of the authors would never have agreed to add a major structure that
was not fully fleshed out, in advance of having a completed standard AND
having had a chance to test that structure in practice. Because we were
locking down the syntax, we wanted to ENABLE the addition of extlang should
the group decide that it was worth adding. Do I really have to repeat this
over and over????

>    1. it is substantially better to have languages like 'cmn' be put in a
> >    secondary position (eg zh-cmn-Hant-CN), rather than being primary
> subtags on
> >    their own (eg cmn-Hant-CN).
>
> It is better because all documents in Sinitic languages have been
> historically tagged "zh", most of them are in Mandarin, and allowing "cmn"
> as a primary language tag will break people who expect "zh" to Just Work.
> Furthermore, people who have been using "zh-yue" for Cantonese should
> not have to switch to "yue" to allow flexibilities such as adding script,
> region, or variant tags.  (Currently "zh-yue" is grandfathered and cannot
> be used generatively.)

You are not revealing some important hidden assumptions in your statements.

   1. The macrolanguage is always a better fallback for every encompassed
   language than other alternatives. Out of the many encompassed languages, you
   implying that a speaker of every encompassed language will be able to
   understand the macrolanguage, or at least better than the alteratives.
   2. People don't lose anything by having the fallback. I dispute this
   as well. As previously noted:
   1. Truncation fallback from zh-cmn-Hant-SG to "zh" loses the Hant and
      the SG; falling back from ar-arb-SA to 'ar' loses the "SA".
      2. It introduces ambiguous language names. Right now, in the
      overwhelming majority of practice, standard Arabic is "ar"; after the
      change, standard Arabic is "ar-arb".
   3. People can't get along without this.
   1. Anyone who has to deal with language issues on all but a trivial
      level must already have a mechanism to deal with sh, sr, hr; with no, nb,
      nn. Those are macrolanguages and encompassed languages. They
exist right now
      WITHOUT an extlang mechanism, and people deal with them. The proposed
      mechanism won't handle these, and anyone who can handle these
doesn't need
      extlang.
      2. With the Macrolanguage field, there is sufficient information
      for *anyone* who wants to to implement extlang-like fallback
(including for
      sh or no), *without* encumbering the IDs with superfluous information.

>    2. it is sufficiently better to warrant making the language tags more
> >    complicated by the addition of this mechanism.
>
> Language tags become more complicated *if* it is desired to make them
> so.  Those who find "zh" sufficient may continue to use it while still
> interoperating with "zh-cmn", "zh-yue", and so on.  Existing deployed
> matchers will continue to work, as will existing deployed software
> that understands specific tags; they will not need to become more
> complicated to understand the out-of-band relationship between "zh",
> "cmn", "yue", etc.

This is untrue. As soon as we implemented extlang in prototype, we ran into
the problems listed above. It *didn't* work out of the box.

--
> A mosquito cried out in his pain,               John Cowan
> "A chemist has poisoned my brain!"              http://www.ccil.org/~cowan
>         The cause of his sorrow                 [email-masked]
>         Was para-dichloro-
> Diphenyltrichloroethane.                                (aka DDT)
>

-- 
Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 28 Aug 2007 15:33:23 -0700

Hi -

As a technical contributor...

> From: "Mark Davis" <[email-masked]>
> To: "John Cowan" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Tuesday, August 28, 2007 2:59 PM
> Subject: [Ltru] extlang
...
>    understand the macrolanguage, or at least better than the alteratives.

I don't see how the use of extlang would require this as an assumption.
Making such an assumption seems a bit like assuming that all languages whose
tags begin in "a" are somehow related.

>       the SG; falling back from ar-arb-SA to 'ar' loses the "SA".

It is the nature of truncation fallback to lose information.  No matter
what order the subtags are trimmed off, someone will be able to argue
that for some particular case, a different order might have been better.
This isn't an argument against extlang; it's an argument against unrealistically
high expectations for truncation fallback.

>       change, standard Arabic is "ar-arb".

This would not be desirable.  However, I wonder whether the semantic associated
by most taggers and users of tags with "ar" is "standard Arabic" or merely "Arabic".

>       mechanism won't handle these, and anyone who can handle these

I think this argument is flawed in that it neglects the cost of supporting
such constellations of languages.  There are already some messes that we're
stuck with, and that have to be handled in an ad hoc manner.  This doesn't
justify requiring ad hoc handling for every other such constellation of
languages.

>       sh or no), *without* encumbering the IDs with superfluous information.

This would be a compelling argument, if fallback were the sole reason for extlang.
However, fallback is not the sole motivation for extlangs; they are also
of use to taggers with incomplete knowledge of the languages used in the
materials they are tagging.  The library staff in my home town would be
doing well if they correctly recognized material as "zh" or "ar".  It would
be quite unrealistic to expect them to be any more precise.

Of course we'd all like everyone who has to tag material to have perfect
knowledge of the languages involved so that tags with sufficient precision
and accuracy would be used.  But we also know that in reality people work
with incomplete knowledge.  Consequently, I think we should allow people to
who by necessity tag with low precision to nonetheless do so accurately.

> the problems listed above. It *didn't* work out of the box.

I'm missing something.  Precisely what scenario was it that was expected to
work that did not?

Randy

---

From: John Cowan <[email-masked]>
Date: Tue, 28 Aug 2007 18:35:36 -0400

Mark Davis scripsit:

> Not single-handedly: Addison has also commented on it. 

AFAIK he has acquiesced in, but not defended, your position.
(I may be wrong.)

> And on the other side, I've heard 3 strong proponents. So not exactly
> overwhelming either way.

FWIW, I have heard privately from several normally active listmembers
supporting my position.  I assume that you know I wouldn't lie about this.

> I've said before, and will say again: it was never a foregone conclusion. 

I have never said so.  I do say that it is the status quo, and it's
a change to the status quo that requires defending.

> Do I really have to repeat this over and over????

Alas, we do seem to be looping.  I wish I saw the way ahead.

>    understand the macrolanguage, or at least better than the alteratives.

RFC 1766 already said that fallback doesn't always work, and sometimes
produces something unintelligible to the requester.

But I believe you are misusing the term "macrolanguage".  A macrolanguage
is not to be equated with a "main" or standardized variety.  Some
macrolanguages like Chinese and Arabic have such varieties, others like
Quechua and Nahuatl do not.

Rather, a macrolanguage is a group of languages that *in some domain*
is recognized as a single language.  By definition, if you speak
Sudan Arabic, you are speaking something that is part of the Arabic
macrolanguage, even if you cannot speak modern standard Arabic at all.
Thus it may be empirically sound to assume that anything (or at least
any text) tagged "ar" is in MSA, it is not certain.

>       the SG; falling back from ar-arb-SA to 'ar' loses the "SA".

So it does.

>       change, standard Arabic is "ar-arb".

You propose in the alternative that "ar" and "arb" both be understood
as modern standard Arabic.  I submit that that is worse.

>       level must already have a mechanism to deal with sh, sr, hr;
>       with no, nb, nn. Those are macrolanguages and encompassed
>       languages. They exist right now WITHOUT an extlang mechanism,
>       and people deal with them. The proposed mechanism won't handle
>       these, and anyone who can handle these doesn't need extlang.

True, but not everyone does.  As I have said before, the majority of
language-sensitive applications, I believe, deal only with recognizing a
short list of languages that they know what to do with, and all others
have the semantics of "unknown language".  It's perfectly compliant
just to look at the first subtag, decide if it is 'en', 'fr', or 'ar',
and throw away all other information; furthermore, I believe this to
be typical.  Most applications just don't involve processing almost
every document known to man.  I want this sort of application to continue
to work without having to be modified to add "arb" as a fourth alternative.

>       (including for sh or no), *without* encumbering the IDs with
>       superfluous information.

I support the Macrolanguage: field for the uses which you mention.
Furthermore, I support extlangs *only* in the context of using newly
introduced 639-3 identifiers that are encompassed by a macrolanguage
registered in 639-2.  Thus, I do *not* support changes to language
tags based on shifting macrolanguage information as SIL learns more,
just the 350+ specific code elements of 639-3, and no more.

> the problems listed above. It *didn't* work out of the box.

I can well understand that in certain applications having an unusual
breadth of scope, both in the matter of documents and in the matter
of languages, that it might well not.

-- 
Dream projects long deferred             John Cowan <[email-masked]>
usually bite the wax tadpole.            http://www.ccil.org/~cowan
        --James Lileks

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 28 Aug 2007 18:12:10 -0700

> Quechua and Nahuatl do not.

I'm not misusing it -- nothing I said had anything to do with the
definition, it has to do with the functional implications. If a language yyy
has the macrolanguage xx, we are talking about two possible representations

a) extlang: xx-yyy
b) lang: yyy

The main reason I've heard from you for doing (a) instead of (b) is that (a)
it has better fallback behavior. For that to be true, xx has to be a good
fallback for users of yyy, in the majority of cases. And the value has to be
worth having the extra structure. So, a natural question to ask is: are you
sure that of ALL the languages on
http://www.sil.org/iso639-3/macrolanguages.asp, the ones that we are talking
about this mechanism for, that the fallback is indeed good.

I really want some reasonable attention paid to the implications of this
whole apparatus before we ship this, because once we do, it is irrevocable!

There are (at least) two cases to consider here.

   1. There is a predominent choice in the industry for the macro
   language. For example, the content for zh is typically always Mandarin; the
   content for ar is typically always standard Arabic. Look at the concrete
   implications. It means that whenever Joe looks for a web page in Hakka
   Chinese, he will typically fall back to Mandarin. Whenever Sarah looks for a
   page in Tunisian Arabic, it will fall back to standard Mandarin.

   If you are saying however, that zh is not necessarily Mandarin, that
   Arabic is not necessarily Standard Arabic, then we fall through to case 2.

   2. There is not a predominant choice in the industry, let's say for
   Hmong. In this case, the situation is different. I could choose any of the
   Hmong for the content for hmn. We then have an even dicer case for the value
   of extlang. I localize my hmn locale with contents appropriate for
   Northeastern Dian Hmong; is that a good default for someone speaking Eastern
   Xiangxi Hmong? for Luopohe Hmong? For all the other Hmongs?

For extlang to be a good apparatus, these always have to be good choices,
since we are baking the structure into the tag.

If Peter Constable came out and said the following, then I would admit to my
sins, give in gracefully, and go along with extlang.

   - "Yes, each of the Hmongs (encompassed by hmn) are mutually
   intelligible, and are better for each one than another fallback like
   Chinese"
      - and the same is true for all the other cases with no
      predominant variant.
   - "Yes, standard Arabic is intelligible for all the encompassed
   languages from Algerian Saharan Arabic to Shihhi Arabic, and are better for
   each one than another fallback like French"
      - and the same is true for all the other cases with a
      predominant variant.

Mark

---

From: Martin Duerst <[email-masked]>
Date: Wed, 29 Aug 2007 12:00:41 +0900

>a change to the status quo that requires defending.

chair hat on:

Given the importance and controversy over the subject,
any assumtion either way of a 'default' would be bad process.
What we need to do is to try to find the technically best solution,
for a wide variety of applications.

Regards,   Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Wed, 29 Aug 2007 12:11:43 +0900

As a technical contributor, I quite agree with Randy.
See below for more details.

>by most taggers and users of tags with "ar" is "standard Arabic" or merely 

Well, probably both ways. Most written Arabic is standard Arabic,
I guess, so even if you assume "ar" just means "whatever Arabic it
means", most stuff tagged "ar" will be Standard Arabic.

The question is whether we can make that assumption stronger,
or whether we can live with such an uncertainity.

I think to some extent, we have always done this.
As an example, "ja" means Japanese, but it also, by suppress-script,
means Japanese written in Kanji-Kana-mixture, and it also, by
"tag wisely", means Japanese as used in Japan.

In general, "tag wisely" seems to include "tag special cases
very precisely, general cases can be tagged more shortly".

So in practice using ar rather than ar-arb for Standard Arabic
seems to be extremely feasible. The question may be how we
can say that in our draft without contradicting ourselves,
and/or how we can help practice moving that way even if we
don't say so in the spec.

>justify requiring ad hoc handling for every other such constellation of

Thinking about what problems Mark actually has, it may be that
what he is saying is: I know I have to deal with some things as
special cases, but I'd like to limit these to single-tag cases
and not to have to combine fallback code and special-case code.

Mark, if this is what you are after, please confirm, and maybe
give some details. Others, if that doesn't look like it would
work, please give some counterexamples.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 29 Aug 2007 12:51:52 +0900

>just the 350+ specific code elements of 639-3, and no more.

Hello John,

I'm sure you have done so somewhere already, but for everybody
to see, could you (pointer is okay) provide a list of the
macrolanguages that would be handled by extlangs?
(and maybe also those that wouldn't).
Also, is your proposal above the same thing as what's in the
current draft, or would the draft need tweaking?
(My reading of Randy's collection of snippets seems to indicate
that our draft would need some work.)
If that's the case, can you provide some proposed text?

Regards,    Martin.

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 28 Aug 2007 23:13:37 -0700

Hi -

Forgive the extended context.  My comments are inline.
As a technical contributor...

> Sent: Tuesday, August 28, 2007 6:12 PM
> Subject: [Ltru] Re: extlang
>
> fallback for users of yyy, in the majority of cases.

As I see it, the qualification "in the majority of cases" is far too strong.
As I see it, if there are *any* significant cases where (a) would work better,
there is reason to consider that approach worth investigating.

> And the value has to be
> worth having the extra structure.

What is the cost of the "extra structure"?

> So, a natural question to ask is: are you

I think this is a mis-statement of the argument.  I think the question is whether
there are any cases for which the fallback would be useful, or for which there
would be other reasons (which I'd like to see spelled out somewhere) to prefer
the (a) format over the (b) format.
 
> whole apparatus before we ship this, because once we do, it is irrevocable!

Full agreement.
 
>    page in Tunisian Arabic, it will fall back to standard Mandarin.

I assume the last Mandarin was Arabic.

>    Arabic is not necessarily Standard Arabic, then we fall through to case 2.

This is the case of what I would call "accurately but imprecisely" tagged material,
 
>    Xiangxi Hmong? for Luopohe Hmong? For all the other Hmongs?

The use of extlang per se has no effect on how this scenario plays out.
What you're assuming here is that the extlang element would always correspond
to some identifiable single language variety, and that there might be content
which would be fully identified by it.  In my opinion, this should happen
solely for those cases where we already have a specific string that is 
already used for a particular language.  The only other case where I could
see it making sense to tag content solely with the extlang is in cases
where a more accurate identification of the language hasn't happened.

>       - and the same is true for all the other cases with a

It's pretty easy to come up with scenarios in which just about any
fallback scheme will produce results other than the desired ones.
That's the problem with heuristics.  They don't predict things like
the fact that 'de' is easier for me than 'en-scottish' even though
my preferred language might be 'en-US'.

Fallback has probably been over-sold.  But I think the real question is
whether using extlang would provide value at all, and what (if any)
cost there would be to supporting it.  THAT's the equation that needs
to be solved.

Randy

---

From: John Cowan <[email-masked]>
Date: Wed, 29 Aug 2007 09:13:55 -0400

> macrolanguages that would be handled by extlangs?

Note that it's the "microlanguages" (not an official term), the languages
encompassed by macrolanguages, that are handled by extlang subtags.

http://www.sil.org/iso639-3/macrolanguages.asp is the core list of
macrolanguage mappings.  For our purposes, the macrolanguages Akan
(Fat, Twi), Norwegian (Bokmal, Nynorsk), and Serbo-Croatian (Serbian,
Croatian, Bosnian) are out of the case, because the "microlanguages"
already have primary language subtags.  (Mark has mentioned the Norwegian
and Serbo-Croatian cases as ones that one needs to handle specially even
under an extlang regime.)

In addition to these, "sgn" is treated as a macrolanguage
encompassing all sign languages.  A list of these can be found at
http://www.ethnologue.com/show_family.asp?subid=90008 (the Deaf ones)
and http://www.ethnologue.com/show_family.asp?subid=90503 (the others).

> (and maybe also those that wouldn't).

That would be all the existing languages of 639-2, plus all
the languages of 639-3 that do *not* appear on the right
side of the above-referenced table.

> current draft, or would the draft need tweaking?

I understand it to be the same.

-- 
There are three kinds of people in the world:   John Cowan
those who can count,                            [email-masked]
and those who can't.

---

From: Peter Constable <[email-masked]>
Date: Thu, 30 Aug 2007 07:29:01 -0700

Here are my responses to this mail from Mark. I critique his argumentation, but be careful not to jump to conclusions about what I’m saying regarding the open issue: what I say here critiques Marks arguments against extlang, but does not attempt to make a sufficient case for extlang.

From: Mark Davis [mailto:[email-masked]]
Sent: Tuesday, August 28, 2007 6:12 PM

> If a language yyy has the macrolanguage xx, we are talking about two possible representations
a) extlang: xx-yyy
b) lang: yyy

>The main reason I've heard from you for doing (a) instead of (b) is that (a) it has better fallback behavior. For that to be true, xx has to be a good fallback for users of yyy, in the majority of cases.
I think “fallback behavior” needs more careful consideration here. It appears that Mark is focusing on a particular scenario: request is for resource in lang A, but that is not available so process needs to fallback to a likely-next-best choice available.
When I first proposed the extlang mechanism, that was not the intent. Rather, the intent was focused on another scenario: author wants to tag content using more specific ID yyy, but many requests, especially from legacy implementations, will use xx, which has been in use for some time. This scenario pertains to Language-range as defined since HTTP/1.1.
                Language-range = xx
                Content to be matched = yyy or xx-yyy
If the content is tagged xx-yyy, there is a match. But if the content is tagged yyy, there is no match using a basic algorithm – one would need a more advanced algorithm that knows about the relationship between xx and yyy.
I think Mark’s is the reciprocal scenario: users prefers the specific variety yyy, but most content is already tagged using xx, which has been in use for some time. This is a particular case in the general set of fallback scenarios, and it seems to be exactly the one Mark is focusing on. But note that language-range does not apply here – or, from a different perspective, doesn’t work here:
language-range = yyy or xx-yyy
content to be matched = xx
Whichever way the language-range is expressed, there is no match with a basic algorithm – one would need a more advanced algorithm that knows about the relationship between xx and yyy.
So, considering just those scenarios, xx-yyy has an advantage over yyy in that it provides an advantage for the one scenario while the two alternatives are equal wrt the other. Of course, those aren’t the only scenarios. We need to consider a broader set of scenarios, and also consider how they rank in priority.

> There are (at least) two cases to consider here.

 1.  There is a predominent choice in the industry for the macro language. For example, the content for zh is typically always Mandarin; the content for ar is typically always standard Arabic.
Typically, yes. We just must not assume that is always the case. The very thing that started me thinking about the macrolanguage concept in the first place, rather than equating existing ISO 639 IDs like zh and ar with the predominant variety, was the fact that there were language tags registered with IANA explicitly associating zh with Chinese languages other than Mandarin. When faced with pre-existing usage “zh-wuu”, we have two choices:
a) wuu is a specific variety of zh
b) wuu relates to zh in some other way, such as zh being a good fallback choice
I suspect that the request for zh-wuu was based on the first perspective, and that was the perspective I assumed.

> Look at the concrete implications. It means that whenever Joe looks for a web page in Hakka Chinese, he will typically fall back to Mandarin. Whenever Sarah looks for a page in Tunisian Arabic, it will fall back to standard Mandarin.
We need to be a bit more careful here: exactly how does Joe go about requesting Hakka? If he asks for “zh” hoping to get content in Hakka, there is a very high probability he will get pages in Mandarin. But if he asks for “zh-hakka”, he will only get pages tagged “zh-hakka” from servers implementing HTTP/1.1 language-range, not “zh” pages: that is how language-range works.

>If you are saying however, that zh is not necessarily Mandarin, that Arabic is not necessarily Standard Arabic, then we fall through to case 2.

 1.  There is not a predominant choice in the industry, let's say for Hmong. In this case, the situation is different. I could choose any of the Hmong for the content for hmn. We then have an even dicer case for the value of extlang. I localize my hmn locale with contents appropriate for Northeastern Dian Hmong; is that a good default for someone speaking Eastern Xiangxi Hmong? for Luopohe Hmong? For all the other Hmongs?
> For extlang to be a good apparatus, these always have to be good choices, since we are baking the structure into the tag.

Again, let’s be more careful in the analysis and argumentation. You’re questioning whether a request for hmn should be able to return content in any Hmong language, and saying that “hmn-hmd” (NE Dian) is bad because someone asking for Hmong might really be a speaker of Luopohe Hmong. It seems to me this is a fallacious argument: it’s premise is that “hmn” can and must be sufficient for any of these various speakers. Well, either it is or it isn’t. If it is, then the argument fails. If it isn’t, then all that proves is that “hmn” really is never sufficient: a more specific language-range really is needed, and anyone requesting their resources using “hmn” is making a vague request that will be subject to somewhat arbitrary results. At that point you decide a more specific language-range is needed, it makes no difference whether the language range used is “hmd” or “hmn-hmd”: both would succeed in obtaining the desired result.

So, we really can’t use the macrolanguage cases like Hmong to decide this open issue. If NE Dian is not a good choice for Luopohe, the *only* thing that points to is that “hmn” is too vague to be useful for requesting resources.

Btw, keep in mind why “hmn” was created in the first place, and why it was created as an individual-language identifier:

-          librarians needed a tag for content

-          they were not Hmong specialists and had no ability to differentiate between varieties

-          as these are not-highly-developed varieties (in the language-development sense – literature, media, standardization), there was no reason for these non-specialists to suppose that these varieties were anything more than dialects of a single language (assuming they had much awareness of any variations in the first place)

Now, as I approached how to deal with “hmn” in ISO 639-2 when it came to creating ISO 639-3, I had two options: argue that “hmn” should really be a collection, or treat “hmn” as a macrolanguage. In the original analysis, (http://www.ethnologue.com/14/iso639/analysis.asp), I had concluded “hmn” is really a collection. But since I also am not a Hmong specialist and didn’t have the capacity (far from it!) of getting an expert analysis of this and every other uncertain case in ISO 639 in any reasonable amount of time, I chose the path of least resistance: treat it as a macrolanguage since ISO 639-2 and its user community considers it an individual language, and that way I don’t have to get the JAC to take action on yet one decision where the impact is unclear and the internal expertise on which to base the decision is minimal.

But note that for our purposes here it really doesn’t matter whether “hmn” ended up as a macrolanguage or as a collection: either way, it is still vague and therefore not a good tag to use for requesting resources if the distinctions matter to you.

> If Peter Constable came out and said the following, then I would admit to my sins, give in gracefully, and go along with extlang.

 *   "Yes, each of the Hmongs (encompassed by hmn) are mutually intelligible, and are better for each one than another fallback like Chinese"

    *   and the same is true for all the other cases with no predominant variant.
I haven’t come out saying that; I assume they are not. But, I’m saying that this case is irrelevant for what we need to decide.

 *   "Yes, standard Arabic is intelligible for all the encompassed languages from Algerian Saharan Arabic to Shihhi Arabic, and are better for each one than another fallback like French"

    *   and the same is true for all the other cases with a predominant variant.
Well, what you’re asking me to say here is that a request for e.g. Algerian Saharan Arabic can appropriately be serviced with Standard Arabic resources rather than, say, French. In other words, a language-range “ar-arq” can appropriately match “ar” content. But that would never happen because that is not how language-range works. So, I’m not sure how this is relevant: whether the language-range is “ar-arq” or “arq” the only results returned will be Algerian Saharan Arabic, unless some more advanced fallback behaviour is invoked.

The question that would be relevant in terms of language-range for the Arabic/Mandarin cases is this: if someone requests “ar”, how bad is it if they get pages in “ar-arq”? If someone requests “zh”, how bad is it if they get pages in “zh-hakka”? These are the cases that relate to the way language-range works.

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 30 Aug 2007 09:39:26 -0700

Thanks Peter; background material like this and specific scenarios really
help to make clear what the requirements are, because they are very
different depending on the kinds of usage patterns people have in mind.
While I may disagree with some of your points, this discussion is going in a
good direction.

It is key to identify exactly how extlang would work in different
situations. We thought long and hard about the introduction of scripts, and
looked at many different aspects -- we need to do the same with the extlang
proposal.

It appears that there are two substantially different kinds of usage that we
need to examine:

Resource Lookup: user requests language X; supplier doesn't have exactly X
but wants to get the best match (in the sense of most likely to be
understood by the user). The extlang proponents appeared to be arguing that
it was better for this case, but it just doesn't seem to work well.

Query: user requests content matching X, in some "loose" fashion, such as a
library lookup. Unless I'm misreading Peter, it appears that this was
actually the primary target of the extlang mechanism. So this bears some
significant attention, to see how the extlang mechanism would work in this
case.

Once we get a sense of the relative benefits/drawbacks of extlang for each
of these -- and weight the relative importance of these with respect to
language tag usage, then we'd be in better shape to know the overall pros
and cons of adding it.

Mark

On 8/30/07, Peter Constable <[email-masked]> wrote:
>
>   Here are my responses to this mail from Mark. I critique his
> argumentation, but be careful not to jump to conclusions about what I'm
> saying regarding the open issue: what I say here critiques Marks arguments
> against extlang, but does not attempt to make a sufficient case for extlang.
>
>
>
> * *
>
> *From:* Mark Davis [mailto:[email-masked]]
> *Sent:* Tuesday, August 28, 2007 6:12 PM
>
> **
>
> >The main reason I've heard from you for doing (a) instead of (b) is that
> (a) it has better fallback behavior. For that to be true, xx has to be a
> good fallback for users of yyy, in the majority of cases.
>
> I think "fallback behavior" needs more careful consideration here. It
> appears that Mark is focusing on a particular scenario: request is for
> resource in lang A, but that is not available so process needs to fallback
> to a likely-next-best choice available.
>
> When I first proposed the extlang mechanism, that was not the intent.
> Rather, the intent was focused on another scenario: author wants to tag
> content using more specific ID yyy, but many requests, especially from
> legacy implementations, will use xx, which has been in use for some time.
> This scenario pertains to Language-range as defined since HTTP/1.1.
>
>                 Language-range = xx
>
> If the content is tagged xx-yyy, there is a match. But if the content is
> tagged yyy, there is no match using a basic algorithm – one would need a
> more advanced algorithm that knows about the relationship between xx and
> yyy.
>
> I think Mark's is the reciprocal scenario: users prefers the specific
> variety yyy, but most content is already tagged using xx, which has been in
> use for some time. This is a particular case in the general set of fallback
> scenarios, and it seems to be exactly the one Mark is focusing on. But note
> that language-range does not apply here – or, from a different perspective,
> doesn't work here:
>
> Whichever way the language-range is expressed, there is no match with a
> basic algorithm – one would need a more advanced algorithm that knows about
> the relationship between xx and yyy.
>
> So, considering just those scenarios, xx-yyy has an advantage over yyy in
> that it provides an advantage for the one scenario while the two
> alternatives are equal wrt the other. Of course, those aren't the only
> scenarios. We need to consider a broader set of scenarios, and also consider
> how they rank in priority.
>
>
>
>
>    content for ar is typically always standard Arabic.
>
> Typically, yes. We just must not assume that is always the case. The very
> thing that started me thinking about the macrolanguage concept in the first
> place, rather than equating existing ISO 639 IDs like zh and ar with the
> predominant variety, was the fact that there were language tags registered
> with IANA explicitly associating zh with Chinese languages other than
> Mandarin. When faced with pre-existing usage "zh-wuu", we have two choices:
>
> > Look at the concrete implications. It means that whenever Joe looks for
> a web page in Hakka Chinese, he will typically fall back to Mandarin.
> Whenever Sarah looks for a page in Tunisian Arabic, it will fall back to
> standard Mandarin.
>
> We need to be a bit more careful here: exactly *how* does Joe go about
> requesting Hakka? If he asks for "zh" hoping to get content in Hakka, there
> is a very high probability he will get pages in Mandarin. But if he asks for
> "zh-hakka", he will only get pages tagged "zh-hakka" from servers
> implementing HTTP/1.1 language-range, not "zh" pages: that is how
> language-range works.
>
>
>    1. There is not a predominant choice in the industry, let's say for
> > For extlang to be a good apparatus, these always have to be good
> choices, since we are baking the structure into the tag.
>
>  Again, let's be more careful in the analysis and argumentation. You're
> questioning whether a request for hmn should be able to return content in
> any Hmong language, and saying that "hmn-hmd" (NE Dian) is bad because
> someone asking for Hmong might really be a speaker of Luopohe Hmong. It
> seems to me this is a fallacious argument: it's premise is that "hmn" can
> and must be sufficient for any of these various speakers. Well, either it is
> or it isn't. If it is, then the argument fails. If it isn't, then all that
> proves is that "hmn" really is never sufficient: a more specific
> language-range really is needed, and anyone requesting their resources using
> "hmn" is making a vague request that will be subject to somewhat arbitrary
> results. At that point you decide a more specific language-range is needed,
> it makes no difference whether the language range used is "hmd" or
> "hmn-hmd": both would succeed in obtaining the desired result.
>
>
>
> So, we really can't use the macrolanguage cases like Hmong to decide this
> open issue. If NE Dian is not a good choice for Luopohe, the **only**
> thing that points to is that "hmn" is too vague to be useful for requesting
> resources.
>
>
>
> Btw, keep in mind why "hmn" was created in the first place, and why it was
> created as an individual-language identifier:
>
>
>
> -          as these are not-highly-developed varieties (in the
> language-development sense – literature, media, standardization), there was
> no reason for these non-specialists to suppose that these varieties were
> anything more than dialects of a single language (assuming they had much
> awareness of any variations in the first place)
>
>
>
> Now, as I approached how to deal with "hmn" in ISO 639-2 when it came to
> creating ISO 639-3, I had two options: argue that "hmn" should really be a
> collection, or treat "hmn" as a macrolanguage. In the original analysis, (
> http://www.ethnologue.com/14/iso639/analysis.asp), I had concluded "hmn"
> is really a collection. But since I also am not a Hmong specialist and
> didn't have the capacity (far from it!) of getting an expert analysis of
> this and every other uncertain case in ISO 639 in any reasonable amount of
> time, I chose the path of least resistance: treat it as a macrolanguage
> since ISO 639-2 and its user community considers it an individual language,
> and that way I don't have to get the JAC to take action on yet one decision
> where the impact is unclear and the internal expertise on which to base the
> decision is minimal.
>
>
>
> But note that for our purposes here it really doesn't matter whether "hmn"
> ended up as a macrolanguage or as a collection: either way, it is still
> vague and therefore not a good tag to use for requesting resources if the
> distinctions matter to you.
>
>
>
>
>
> > If Peter Constable came out and said the following, then I would admit
> to my sins, give in gracefully, and go along with extlang.
>
>     - and the same is true for all the other cases with no predominant
>       variant.
>
> I haven't come out saying that; I assume they are not. But, I'm saying
> that this case is irrelevant for what we need to decide.
>
> >
>
>     - and the same is true for all the other cases with a predominant
>       variant.
>
>  Well, what you're asking me to say here is that a request for e.g.
> Algerian Saharan Arabic can appropriately be serviced with Standard Arabic
> resources rather than, say, French. In other words, a language-range
> "ar-arq" can appropriately match "ar" content. But that would never happen
> because that is not how language-range works. So, I'm not sure how this is
> relevant: whether the language-range is "ar-arq" or "arq" the only results
> returned will be Algerian Saharan Arabic, unless some more advanced fallback
> behaviour is invoked.
>
>
>
> The question that would be relevant in terms of language-range for the
> Arabic/Mandarin cases is this: if someone requests "ar", how bad is it if
> they get pages in "ar-arq"? If someone requests "zh", how bad is it if they
> get pages in "zh-hakka"? These are the cases that relate to the way
> language-range works.
>
>
>
>
>
> Peter
>

---

From: Marion Gunn <[email-masked]>
Date: Thu, 30 Aug 2007 18:05:17 +0000

On 29 Aug 2007, at 13:13, scríobh John Cowan:

> encompassed by macrolanguages, that are handled by extlang subtags...
>

I strongly object to the use of the above unofficial term, which John  
Cowan is now using this list to promote (as evidenced in a recent msg  
from him, as cited below) and I believe that, if the administrators  
of this list _fail_ to deprecate his use of such an objectionable  
term, that this may be an appropriate time for representatives of the  
National Bodies of governments of the countries whose indigenous  
languages he has so denigrated (by his deliberate use of said  
unofficial term) to withdraw any provisional support they may have  
given until now to the use of the term "macrolanguages" (because that  
provisional support for treating that term as "official" would seem  
to be providing the opportunity to use an objectionable word to  
denigrate languages without even naming them).
mg

- -
Marion Gunn * EGTeo (Estab.1991)
27 Páirc an Fhéithlinn, Baile an
Bhóthair, Co. Átha Cliath, Éire.
* [email-masked] * [email-masked] *

---

From: Addison Phillips <[email-masked]>
Date: Thu, 30 Aug 2007 10:13:07 -0700

Peter Constable wrote:
>  These are the cases that relate to the
> way language-range works.
> 

Not exactly. Language-ranges are the expression of the preference. While 
HTTP 1.1 and others didn't put a lot of structure around language 
negotiation, we spent some time doing so in RFC 4647.

HTTP 1.1 and other, similar, items defined what we now call "basic 
filtering". That's what Peter means by the above. There is also 
"lookup", which is widely implemented (not well that I do not say "more 
widely").

In fact, we have different mechanisms for working with language tagged 
materials and these mechanisms each have their own uses and practical 
usage scenarios.

As for extlangs... I'm not sure we can solve this by arguing either 
case. Both the pro- and anti-extlang parties have valid points and in 
some cases the extlang tags *are* more useful and in some cases they 
*are* more harmful.

For a long time I supported extlangs because they were the direction we 
had laid down, in particular for Chinese languages. However, some things 
bother me about this approach:

1. If the languages in question really are distinct languages, why does 
subordinating the language make sense? Yes, you couldn't tag this or 
that language yesterday via the less distinct macrolanguage tag. Is that 
appropriate?

Thus, if I were to support extlang, it would be based solely on John 
Cowan's argument that we need extlang to prevent a "retagging crisis" 
for languages formerly enclosed by a macrolanguage.

2. Randy suggested (a long while ago now) that cherry-picking from the 
macrolanguage list would be a bad idea. Yet tag stability provisions 
prevent us from taking the list wholesale. And I have some concern that 
the macrolanguage "collections" (if you'll pardon this inaccurate term) 
have yet to be thoroughly tested. They may not be stable or suitable in 
the short-to-medium term. This is not a critique of ISO 639-3's work 
here. It is merely a note of caution.

If I were to support doing extlangs, it would consider each 
macrolanguage separately, as a one-time-event, and, again, solely as a 
compatibility item.

Note that we also face our own exception--I assume future sign languages 
would be treated as extlangs for compatibility reasons. Note that sign 
languages are NOT macrolanguages in the first place--they are already 
exceptional. I would favor eliminating this use of extlang too.

3. Mark's arguments about "better matching choices" really don't fit 
with matching as described in RFC 4647. Applications really do need to 
do more than the trivial matching in BCP 47 in many cases, but these 
depend on application specific needs. "Pure" BCP 47 matching has its 
place and cannot do many of the things that Mark suggests (as with 
Breton -> French fallback).

Mark's arguments about losing subsidiary subtags when matching extlangs 
*do* concern me. I had to modify my implementation of filtering to make 
it work in an extlang world. The modifications were not difficult and 
were reasonably successful. I *could* support a very limited application 
of extlang...

... but I feel that we're doing a disservice to the various languages 
involved by making them extlangs. Shouldn't Cantonese, Wu, or Hakka be 
treated fully as languages? Yes, it saves some retagging in the short 
term, but it is cleaner and clearer to tag languages directly. Users can 
still use the macrolanguage tag instead (I doubt we'll see much 'cmn-*' 
when 'zh-*' is available). It eliminates a complexity in tagging and 
language negotiation/selection who's purpose is served by compatibility 
alone. And content will need to be retagged in either case to take 
advantage of the distinction. Content that is not retagged won't be 
distinct.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: Marion Gunn <[email-masked]>
Date: Thu, 30 Aug 2007 18:46:59 +0000

On 30 Aug 2007, at 17:13, scríobh Addison Phillips:

> case. Both the pro- and anti-extlang parties have valid points and  
> in some cases the extlang tags *are* more useful and in some cases  
> they *are* more harmful...

The most harmful aspect of extlangs, Addison, is John Cowan's  
reckless use of an objectionable unofficial term (in his msg below),  
which I am still waiting for a list administrator to challenge/ 
deprecate via the list, because that unofficial term denigrates the  
indigenous languages of certain countries.
mg

- -

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 30 Aug 2007 11:31:11 -0700

Hi -

As co-chair...

> From: "Marion Gunn" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
> Sent: Thursday, August 30, 2007 11:05 AM
> Subject: Re: [Ltru] Re: extlang
>

On what grounds?  If the string '"microlanguages" (not an official term)' is
objectionable to you as a shorthand for "language tag which are constructed
by joining an extlang representing an macrolanguage and an additional subtag
to identify the specific language" (or something like that), please suggest a concise
alternative suitable for use in our discussion.

...

Participation in the IETF is by individuals, not National Body representatives.
Calls such as this are meaningless.  We use "unofficial terms" all the time.
Honestly, I completely fail to see how "microlanguage" (formed by obvious
analogy from "macrolanguage") to describe language tags formed in a
particular way could be though to "denigrate" a language.

Randy

---

From: [email-masked]
Date: Thu, 30 Aug 2007 11:32:14 -0700

For the record, I suspect John used this term -- in quotes indicating that 
this wasn't formal language -- because I had just used this term in an 
offline e-mail to him trying to clarify the straw poll question. I 
certainly meant nothing by it -- I was making sure that we were talking 
about the languages encapsulated by the macrolanguage, not the 
macrolanguage itself as was implied by the text. If this term has offended 
you, I'm sorry for my part in it. I'm quite sure John did not use this in 
the spirit you are suggesting. 

If m*crolanguage is an offensive term, perhaps macrolanguage is as well?

Regards,

Karen Broome

Marion Gunn <[email-masked]> 
08/30/2007 11:46 AM

To
LTRU Working Group <[email-masked]>, Addison Phillips 
<[email-masked]>
cc

Subject
Re: [Ltru] Re: extlang

- -

---

From: John Cowan <[email-masked]>
Date: Thu, 30 Aug 2007 15:02:37 -0400

Marion Gunn scripsit:

> Cowan is now using this list to promote

Promote?  Rubbish.

> if the administrators of this list _fail_ to deprecate his use of such
> an objectionable term

As Randy told you, this is the IETF, where we speak only for ourselves.
In particular, I don't at any time speak for any group.

> that this may be an appropriate time for representatives of the National
> Bodies of governments of the countries whose indigenous languages
> he has so denigrated (by his deliberate use of said unofficial term)
> to withdraw any provisional support they may have given until now to
> the use of the term "macrolanguages" (because that provisional support
> for treating that term as "official" would seem to be providing the
> opportunity to use an objectionable word to denigrate languages without
> even naming them).

However, you ought to know by now that threats against me don't get
you very far.  I'm sure the 639-3/RA and the 639/RA-JAC can take care of
themselves; the notion that one word in one email can put ISO funding
at risk is preposterous.

I'm sorry to take up the list members' time with this, but baseless charges
have to be nailed to the wall.  Dixi.

That said, the term is indeed a poor one, and I do acknowledge the
unconscious influence of Karen's email to me.  "Encompassed languages"
is what I meant and should have said.

-- 
Business before pleasure, if not too bloomering long before.
        --Nicholas van Rijn
                John Cowan <[email-masked]>
                    http://www.ccil.org/~cowan

---

From: Peter Constable <[email-masked]>
Date: Thu, 30 Aug 2007 14:25:43 -0700

I think it's fairly obvious that John was using the term as a convenience for our discussions with purely technical meaning that does not in any way imply a value judgment on any language variety.

For that matter, "macrolanguage" as defined in ISO 639-3 also does not connote any value judgment on language varieties so designated, or even on their sociolinguistic status. It has a technical meaning given in ISO 639-3 that, I think, is clear in this regard.

Peter

> -----Original Message-----
> From: Marion Gunn [mailto:[email-masked]]
> To: LTRU Working Group

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 30 Aug 2007 14:49:56 -0700

I agree. It is obvious that John was not implying anything about the value
of the languages.

Mark

On 8/30/07, Peter Constable <[email-masked]> wrote:
>
> I think it's fairly obvious that John was using the term as a convenience
> for our discussions with purely technical meaning that does not in any way
> imply a value judgment on any language variety.
>
> For that matter, "macrolanguage" as defined in ISO 639-3 also does not
> connote any value judgment on language varieties so designated, or even on
> their sociolinguistic status. It has a technical meaning given in ISO 639-3
> that, I think, is clear in this regard.
>
>
>
> Peter
>
>

---

From: Marion Gunn <[email-masked]>
Date: Fri, 31 Aug 2007 10:25:44 +0000

On 30 Aug 2007, at 18:32, scríobh [email-masked]:

> For the record, I suspect John used this term -- in quotes  
> indicating that this wasn't formal language -- because I had just  
> used this term in an offline e-mail to him trying to clarify the  
> straw poll question.

Thanks, Karen.

> I certainly meant nothing by it -- I was making sure that we were  
> talking about the languages encapsulated by the macrolanguage, not  
> the macrolanguage itself as was implied by the text. If this term  
> has offended you, I'm sorry for my part in it.

For my part, I'm sorry if I seem to sometimes get in with a criticism  
too early (but, in this case, I felt we had already waited way too  
long - days - for the list owners to deprecate that expression).

> I'm quite sure John did not use this in the spirit you are suggesting.

If he abides by what he says below, that will be good.

> If m*crolanguage is an offensive term, perhaps macrolanguage is as  

This episode most certainly casts some doubt on that term, if its use  
can so easily generate a concept we do not want to planted in the  
mind of an innocent professional (such as yourself).
mg

- -