Subject: [Ltru] Re: Extended language tags (long reply)

From: "Doug Ewell" <[email-masked]>
Date: Fri, 12 Oct 2007 00:20:07 -0700

Mark Davis <mark dot davis at icu dash project dot org> wrote:

> It only makes sense to do this as a heuristic when the speaker of the 
> microlanguage is extremely likely to speak the macrolanguage.

And so once again, the concept of "macrolanguage" is represented not as 
ISO 639-3 intended it, as a family of languages that are sometimes 
considered a single language for some purposes, but rather as a 
specific, dominant language within that family.

If this is the definition we apply to the term "macrolanguage," and if 
we understand "zh-yue" as a signal to fall back from Cantonese to 
Mandarin, then the no-extlang proponents are certain to win this battle.

--
Doug Ewell * Fullerton, California, USA * RFC 4645 * UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Fri, 12 Oct 2007 10:15:02 -0400

> specific, dominant language within that family.

Even if you don't take that view (and I don't), it's idle to deny
that almost everything textual tagged "zh" or "ar" is Mandarin Chinese
or Standard Arabic respectively, as an uncontroversial consequence
of the fact that almost all written documents in Chinese or Arabic
*are* Mandarin or Standard respectively.

-- 
I now introduce Professor Smullyan,             John Cowan
who will prove to you that either               [email-masked]
he doesn't exist or you don't exist,            http://www.ccil.org/~cowan
but you won't know which.                               --Melvin Fitting

---

From: Peter Constable <[email-masked]>
Date: Fri, 12 Oct 2007 07:45:40 -0700

From: Doug Ewell [mailto:[email-masked]]

> specific, dominant language within that family.

Doug's comment is consistent with the intent of ISO 639, and I support it.

I also concede, though, that Mark's comment wasn't entirely without basis: the reality is that for a small set of macrolanguages there is a dominant variety that would have been meant in a high proportion of instances of usage of the identifier for that macrolanguage. That's the confusion created by legacy language-identifier implementations and the not-so-simple sociolinguistics they were faced with.

> Mandarin, then the no-extlang proponents are certain to win this battle.

"zh-yue", if we stick with it, should not be a signal to fall back to Mandarin. It should be a signal that, when "zh-yue" is requested, returning a record tagged "zh" may give acceptable results.

Peter

---

From: Addison Phillips <[email-masked]>
Date: Fri, 12 Oct 2007 08:26:29 -0700

> *are* Mandarin or Standard respectively.

Actually, I don't know that I would say that, at least in the case of 
Chinese.

In fact, for most Chinese written documents, I would tend to say that it 
isn't possible to discern which specific Chinese language they are in. 
You could represent that as being "in Mandarin", since that is the 
source of the written form, but in practical terms there is little 
enough difference between Cantonese and Mandarin at the written level to 
really know. In fact, it is a very useful fiction that these languages 
are "the same".

This is one reason that I prefer my workaround for the lookup algorithm 
using extlangs to Mark's recent suggestion on list: it recognizes that, 
for written resources, "generic" Chinese plus subtags is a better 
fallback than the second language in your language priority list.

[[As a reminder Mark's suggestion was to make fallback for the LPL 
"zh-yue-Hant-HK;en" to be:

  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  en
  zh-Hant-HK
  zh-Hant
  zh
  (default)

And mine was:

  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  zh-Hant-HK
  zh-Hant
  zh
  en
  (default)

]]

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 12 Oct 2007 09:59:23 -0700

This is precisely why I think this should not be baked into extlang. For
some applications it might be better off to fallback "yue, en" as

   1. yue-Hant-HK
   2. yue-Hant
   3. yue
   4. zh-Hant-HK
   5. zh-Hant
   6. zh
   7. en

But there is a big maybe here. According to Lee Collins, who is well versed
in this area, there are two cases:
1. Standard written Chinese read as Cantonese.
2. Written Cantonese.

1. Can't be distinguished at the script level from Mandarin,
2. can be as easily discerned as easily as any other language. It's as
different, say, as dutch is from English. Some words look the same, but a
mandarin speaker couldn't read it.

So the situation is not that clear. And certainly in other circumstances
(like audio), it is much better to actually honor the user's request, and
fall back like

   1. yue-Hant-HK
   2. yue-Hant
   3. yue
   4. en
   5. (default, which could expand to)
      1. zh-Hant-HK
      2. zh-Hant
      3. zh

That's why if we leave this up to the application, which knows what domain
it is working in, we're better off than if we hard code a relationship that
sometimes works, but requires code changes anyway, and I think more often
fails.

This situation is very similar to another case that we have already. We
could have used extlang with gsw (Swiss German). We could have said: Hmm,
there was no way to tag Swiss German before, so we might have ended up with
such content tagged as "de" instead. And, hmmm, when the Swiss write German,
they typically use High German, so the written form is essentially the same
(with the exception of sharp-s). So let's make the tag be "de-gsw" instead.

But we didn't bake this relationship into the codes; instead we leave this
to the application, to handle appropriate to the domain.

Mark

-- 
Mark

---

From: Shawn Steele <[email-masked]>
Date: Fri, 12 Oct 2007 11:13:42 -0700

Ø  This is precisely why I think this should not be baked into extlang. For some applications it might be better off to fallback "yue, en" as

 1.  yue-Hant-HK
 2.  yue-Hant
 3.  yue
 4.  zh-Hant-HK
 5.  zh-Hant
 6.  zh
 7.  en
My interpretation of http-accept-lang and rfc4647 is that if they ask for yue;en, then you look for

1.       “yue”

2.      “en”

I don’t know where you’re getting the rest of these from, particularly the HK variations.  If you want to do something app specific, that’s cool, but that’s app specific.

-          Shawn

---

From: Shawn Steele <[email-masked]>
Date: Fri, 12 Oct 2007 11:18:06 -0700

Different idea (so I made it a different mail)

Ø  This is precisely why I think this should not be baked into extlang. For some applications it might be better off to fallback "yue, en" as
Note “some applications”.  If I have “zh-yue” on the wire and I’m an application that prefers to think of that in a “yue” manner, then the conversion from zh-yue to yue is fairly trivial.  (pChar+=3)
If, however we have “yue” and I’m an application that prefers to think of that as “zh-yue”, then I just got a much bigger problem.
So in my view, the problem isn’t if 4646bis says use “zh-yue”, the problem is how 4647bis says to fallback yue, no matter how its labeled.  I also don’t think that 4647bis should say “MUST” (or even “SHOULD”) in this context because it is quite apparent that different applications have different needs for falling back “sublangs”.  I do think it should provide guidelines on that behavior.
Anyway, IMO this isn’t a 4646 tagging problem, it’s a 4647 fallback problem.
- Shawn

---

From: Addison Phillips <[email-masked]>
Date: Fri, 12 Oct 2007 11:20:28 -0700

Shawn Steele wrote:
>  
> 
> Ø  This is precisely why I think this should not be baked into extlang. 
> For some applications it might be better off to fallback "yue, en" as

This was a misquote of the original example?? The LPL would be 
"yue-Hant-HK;en" for this example:

> I don’t know where you’re getting the rest of these from, particularly 
> the HK variations.  If you want to do something app specific, that’s 
> cool, but that’s app specific.

You're referring to the filtering algorithm. I'm referring to the lookup 
algorithm, which is in RFC 4647.

For filtering, if you ask for "yue;en", that matches "yue" or "en", but 
also "yue-HK", "en-Brai-GB", etc.

Addison

---

From: Shawn Steele <[email-masked]>
Date: Fri, 12 Oct 2007 11:24:14 -0700

> also "yue-HK", "en-Brai-GB", etc.

You're right, that's my bias, sorry :)

- Shawn

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 12 Oct 2007 13:17:04 -0700

Sorry for the confusion, I did mean the query's being "yue-Hant-HK, en".

Also, got a note from John Jenkins about Cantonese:

The former is like having Spaniards read and write French but pronounce it
as if it were Spanish.  A Frenchman wouldn't understand it if it they heard
it even though it's the same words and grammar.

The latter is very different and, as Lee says, largely incomprehensible to a
Mandarin speaker.  Not entirely, as there's a lot of overlap between the two
lexically, but enough common words are different as to throw the reader off.
 I've been caught a couple of times not being able to make out ads in the
Hong Kong underground until I realized that they were in Cantonese, not
standard written Chinese.

The example of English and Dutch is good, in that enough words are close
enough you can get a vague sense of what's being said, but you can't really
read it.

E.g., the English sentence, "Thank you; that's a good book." is:

謝謝; 那本書很好。

standard written Chinese, but:

唔該; 嗰本書好好嚹。

in Cantonese. The Mandarin speaker could figure out that it's something
about good books, but that's it. They wouldn't have a clue as to the meaning
of 唔, 嗰, or 嚹, and the way the familiar characters 該 and 好 are being used
would seem odd at best.

And, FWIW, written Cantonese is still something of a avis rara, but there's
been a mini-boom in it since HK went back to the mainland.  When I lived
there twenty-five years ago, it was unheard of except
in pedagogical materials, movie scripts, and graffiti.

Mark

On 10/12/07, Mark Davis <[email-masked]> wrote:
> But there is a big maybe here. According to Lee Collins, who is well
> versed in this area, there are two cases:
> On 10/12/07, Addison Phillips < [email-masked]> wrote:
> >
> > John Cowan wrote:
> > >

---

From: Peter Constable <[email-masked]>
Date: Fri, 12 Oct 2007 15:36:32 -0700

I don’t think there’s debate that written Cantonese and written Mandarin may be quite unintelligible; I didn’t know this much detail, but I certainly don’t find it surprising. The problem is that “zh” can legitimately be applied to either. So, it’s not a matter of saying that it’s OK to fallback from a request to zh-yue to Mandarin resources. It’s a matter of how to deal with the vaguaries of “zh”.

Peter

From: Mark Davis [mailto:[email-masked]]
Sent: Friday, October 12, 2007 1:17 PM
To: Addison Phillips
Cc: LTRU Working Group; Doug Ewell
Subject: Re: [Ltru] Re: Extended language tags (long reply)

The former is like having Spaniards read and write French but pronounce it as if it were Spanish.  A Frenchman wouldn't understand it if it they heard it even though it's the same words and grammar.

The latter is very different and, as Lee says, largely incomprehensible to a Mandarin speaker.  Not entirely, as there's a lot of overlap between the two lexically, but enough common words are different as to throw the reader off.  I've been caught a couple of times not being able to make out ads in the Hong Kong underground until I realized that they were in Cantonese, not standard written Chinese.

The example of English and Dutch is good, in that enough words are close enough you can get a vague sense of what's being said, but you can't really read it.

謝謝; 那本書很好。

唔該 ; 嗰本書好好嚹。

in Cantonese. The Mandarin speaker could figure out that it's something about good books, but that's it. They wouldn't have a clue as to the meaning of 唔, 嗰, or 嚹, and the way the familiar characters 該 and 好 are being used would seem odd at best.

And, FWIW, written Cantonese is still something of a avis rara, but there's been a mini-boom in it since HK went back to the mainland.  When I lived there twenty-five years ago, it was unheard of except in pedagogical materials, movie scripts, and graffiti.
Mark

On 10/12/07, Mark Davis < [email-masked]<mailto:[email-masked]>> wrote:
This is precisely why I think this should not be baked into extlang. For some applications it might be better off to fallback "yue, en" as

 1.  yue-Hant-HK
 2.  yue-Hant
 3.  yue
 4.  zh-Hant-HK
 5.  zh-Hant
 6.  zh
 7.  en

But there is a big maybe here. According to Lee Collins, who is well versed in this area, there are two cases:

1. Can't be distinguished at the script level from Mandarin,
2. can be as easily discerned as easily as any other language. It's as different, say, as dutch is from English. Some words look the same, but a mandarin speaker couldn't read it.

So the situation is not that clear. And certainly in other circumstances (like audio), it is much better to actually honor the user's request, and fall back like

 1.  yue-Hant-HK
 2.  yue-Hant
 3.  yue
 4.  en

    *   zh-Hant-HK
    *   zh-Hant
    *   zh

That's why if we leave this up to the application, which knows what domain it is working in, we're better off than if we hard code a relationship that sometimes works, but requires code changes anyway, and I think more often fails.

This situation is very similar to another case that we have already. We could have used extlang with gsw (Swiss German). We could have said: Hmm, there was no way to tag Swiss German before, so we might have ended up with such content tagged as "de" instead. And, hmmm, when the Swiss write German, they typically use High German, so the written form is essentially the same (with the exception of sharp-s). So let's make the tag be "de-gsw" instead.

But we didn't bake this relationship into the codes; instead we leave this to the application, to handle appropriate to the domain.

Mark

On 10/12/07, Addison Phillips < [email-masked]<mailto:[email-masked]>> wrote:

  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  en
  zh-Hant-HK
  zh-Hant
  zh
  (default)

And mine was:

  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  zh-Hant-HK
  zh-Hant
  zh
  en
  (default)

]]

Addison

_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www1.ietf.org/mailman/listinfo/ltru

--
Mark

---

From: John Cowan <[email-masked]>
Date: Fri, 12 Oct 2007 19:02:46 -0400

Peter Constable scripsit:

> I don't think there's debate that written Cantonese and written
> Mandarin may be quite unintelligible; I didn't know this much detail,
> but I certainly don't find it surprising. The problem is that 'zh'
> can legitimately be applied to either. So, it's not a matter of
> saying that it's OK to fallback from a request to zh-yue to Mandarin
> resources. It's a matter of how to deal with the vaguaries of 'zh'.

Apparently, however, almost all publicly accessible documents that
are tagged "zh" are in Mandarin.  The issue is whether this fact has
special significance.  I think these three cases represent the different
situations that may exist:

1) Cantonese.  Lack of specifically tagged resources will cause Mandarin
ones to be provided (de facto); this is probably okay, because people
who can read Cantonese can (for the most part) read Mandarin.

2) Pular.  Lack of specifically tagged resources might cause resources
in any Fulah language to be produced.  This is probably harmless.

3) Arbereshe Albanian.  Lack of resources specifically tagged for that
language will probably cause the production of resources in Standard
Albanian.  However, the most useful fallback in practice is probably
Italian, which is entirely outside the Albanian macrolanguage.

-- 
John Cowan        http://www.ccil.org/~cowan          [email-masked]
Please leave your values                Check your assumptions.  In fact,
   at the front desk.                      check your assumptions at the door.
     --sign in Paris hotel                   --Cordelia Vorkosigan

---

From: Shawn Steele <[email-masked]>
Date: Fri, 12 Oct 2007 16:36:12 -0700

I think that fallback for certain languages to a macro language is less interesting in some cases than others, but I don't see harm in having the extra tag if the application knows when that is interesting and when it isn't.

I think its clear that some applications have different look up / matching/ filtering /whatever behavior requirements, so I don't think RFC 4647 can say "This is the one way that works."  As an example I think most of the people on this list have mentioned implementations that currently do more that what RFC 4646/4647 specify.

I think that a successor to RFC 4647 should suggest that applications be a bit smarter.  Its reasonably trivial to parse the proposed extlang tag and discover that they have a "zh" macro language, "cmn" language, "hant" script, "HK" regional dialect, etc.

>From there it is simple for the application to note that "zh-HK" or "cmn-Hant" are possible matches.  The application could also go farther and note that with zh-yue-Hant-HK, perhaps zh-HK isn't a good match because the application believes that most "zh" data is Mandarin.  Or it could figure that's OK too because many Cantonese speakers also understand Mandarin.

I think that RFC 4647 should provide matching guidelines rather than "MUST trim from the right".  For extra information, like "Breton users might speak French" or "most zh data is Mandarin", the application can either investigate its own desired behavior, or perhaps this could be included in the registry.

- Shawn

---

From: "Doug Ewell" <[email-masked]>
Date: Fri, 12 Oct 2007 20:54:00 -0700

John Cowan <cowan at ccil dot org> wrote:

>> And so once again, the concept of "macrolanguage" is represented not 
>> as ISO 639-3 intended it, as a family of languages that are sometimes 
> Even if you don't take that view (and I don't), it's idle to deny that 
> almost everything textual tagged "zh" or "ar" is Mandarin Chinese or 
> Standard Arabic respectively, as an uncontroversial consequence of the 
> fact that almost all written documents in Chinese or Arabic *are* 
> Mandarin or Standard respectively.

Yes, but it is a consequence of that fact, not a deliberate tagging 
convention.  People are not making a conscious decision to restrict use 
of "zh" to Mandarin and resort to "zh-specific" for other Chinese 
languages, on those rare occasions that they occur.

This is why I'm not on board with the oft-stated premise that "zh-yue" 
will cause an intentional fallback to Mandarin.  If it causes a fallback 
to "zh", then it may well retrieve Mandarin, since (as you say) almost 
all written documents in Chinese are Mandarin.  But that isn't by 
design.