Subject: [Ltru] zh != Mandarin

From: Shawn Steele <[email-masked]>
Date: Tue, 3 Jun 2008 22:17:46 -0700

I think that lots of people feel that zh != Mandarin and that this should be clear.  I don't think anyone objects to that?

I also believe that there's a general consensus that some applications may want to match/filter zh when looking for Mandarin (or Cantonese), and that this isn't "best practice", but MAY be useful to some applications because of the historical use of zh.

Is there any disagreement with this?  There's been a lot of discussion on this topic, and that seems to be most of Karen's concern with any form of "zh".  I *think* we're all saying basically the same thing.  Does anyone object to being clear that zh != Mandarin?

- Shawn
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 4 Jun 2008 18:22:13 -0700

I agree with what you say. I don't think anyone objects to zh != Mandarin,
given the choices that 639-3 has made. Nor that it is valid to tag Mandarin
content with 'zh'; it is, after all, Chinese. What we worked out before (and
this is orthogonal to extlang vs nextlang) is to say that it is recommended
to use zh-cmn, but that for backwards compatibility people may choose to
continue to use zh for Mandarin.
(More useful advice would be that because of backwards compatibility people
should expect that into the indefinite future, "zh" will predominate, and
should keep that in mind as in using these codes either for tagging content
or matching content. However, we can live with having impractical advice in
the spec as long as it is clear that people aren't forced to follow it for
compliance.)

Just for information, we reran some analysis and in terms of web pages, the
number of pages tagged zh-cmn* is approximately 1/250000 of the number of
pages tagged with zh*. This is way down in the noise level: zh-cmn even
occurs far, far less often than suspect, irregular or bogus tags like:

zh_tw

zh_cn

ch

zh-hk

zh-classical

zh-tw-tw

gbk

zh-en

zhs

zh-cn-cn

zht

zh_hk

zho

zh-ch

zh-cn-utf-8

zh_ch

zht-tw

zhtw

zh-tw; xmlns

简体中文

英文

...

(there are even more frequent completely bogus tags; I filtered these to
ones that had something to do with Chinese).

Mark

On Tue, Jun 3, 2008 at 10:17 PM, Shawn Steele <[email-masked]>
wrote:

> I think that lots of people feel that zh != Mandarin and that this should
> be clear.  I don't think anyone objects to that?
>
> I also believe that there's a general consensus that some applications may
> want to match/filter zh when looking for Mandarin (or Cantonese), and that
> this isn't "best practice", but MAY be useful to some applications because
> of the historical use of zh.
>
> Is there any disagreement with this?  There's been a lot of discussion on
> this topic, and that seems to be most of Karen's concern with any form of
> "zh".  I *think* we're all saying basically the same thing.  Does anyone
> object to being clear that zh != Mandarin?
>
> - Shawn

-- 
Mark

---

From: Peter Constable <[email-masked]>
Date: Wed, 4 Jun 2008 20:37:36 -0700

I suppose there may have been a time when the number of instances of “á” on the Web that were encoded as 0xC3 0xA1 was a tiny fraction of the overall occurrences, and that it might have seemed to some like 0xE1 would predominate for the foreseeable future. But we don’t always know what the future will hold.

Peter

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Wednesday, June 04, 2008 6:22 PM
To: Shawn Steele
Cc: LTRU Working Group
Subject: Re: [Ltru] zh != Mandarin

I agree with what you say. I don't think anyone objects to zh != Mandarin, given the choices that 639-3 has made. Nor that it is valid to tag Mandarin content with 'zh'; it is, after all, Chinese. What we worked out before (and this is orthogonal to extlang vs nextlang) is to say that it is recommended to use zh-cmn, but that for backwards compatibility people may choose to continue to use zh for Mandarin.

(More useful advice would be that because of backwards compatibility people should expect that into the indefinite future, "zh" will predominate, and should keep that in mind as in using these codes either for tagging content or matching content. However, we can live with having impractical advice in the spec as long as it is clear that people aren't forced to follow it for compliance.)

Just for information, we reran some analysis and in terms of web pages, the number of pages tagged zh-cmn* is approximately 1/250000 of the number of pages tagged with zh*. This is way down in the noise level: zh-cmn even occurs far, far less often than suspect, irregular or bogus tags like:

zh_tw

zh_cn

ch

zh-hk

zh-classical

zh-tw-tw

gbk

zh-en

zhs

zh-cn-cn

zht

zh_hk

zho

zh-ch

zh-cn-utf-8

zh_ch

zht-tw

zhtw

zh-tw; xmlns

简体中文

英文

...

(there are even more frequent completely bogus tags; I filtered these to ones that had something to do with Chinese).

Mark

On Tue, Jun 3, 2008 at 10:17 PM, Shawn Steele <[email-masked]<mailto:[email-masked]>> wrote:
I think that lots of people feel that zh != Mandarin and that this should be clear.  I don't think anyone objects to that?

- Shawn
_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ltru

---

From: Shawn Steele <[email-masked]>
Date: Thu, 5 Jun 2008 01:12:14 -0700

And to be clear and address Karen's concern, its also OK for people to continue to use zh for Cantonese?

- Shawn
________________________________
From: [email-masked] [[email-masked]] On Behalf Of Mark Davis [[email-masked]]
Sent: Wednesday, June 04, 2008 6:22 PM
To: Shawn Steele
Cc: LTRU Working Group
Subject: Re: [Ltru] zh != Mandarin

zh_tw

zh_cn

ch

zh-hk

zh-classical

zh-tw-tw

gbk

zh-en

zhs

zh-cn-cn

zht

zh_hk

zho

zh-ch

zh-cn-utf-8

zh_ch

zht-tw

zhtw

zh-tw; xmlns

简体中文

英文

...

Mark

- Shawn

---

From: "Gerard Meijssen" <[email-masked]>
Date: Thu, 5 Jun 2008 10:27:45 +0200

Hoi,
People are entitled to be stupid. If it is a good idea to have zh be
Mandarin, it is equally good to also use zh for Cantonese.
Thanks,
     Gerard

2008/6/5 Shawn Steele <[email-masked]>:

>  ------------------------------
> *From:* [email-masked] [[email-masked]] On
> Behalf Of Mark Davis [[email-masked]]
> *Sent:* Wednesday, June 04, 2008 6:22 PM
> *To:* Shawn Steele
> *Cc:* LTRU Working Group
> *Subject:* Re: [Ltru] zh != Mandarin
>
>  I agree with what you say. I don't think anyone objects to zh !=
> Mandarin, given the choices that 639-3 has made. Nor that it is valid to tag
> Mandarin content with 'zh'; it is, after all, Chinese. What we worked out
> before (and this is orthogonal to extlang vs nextlang) is to say that it is
> recommended to use zh-cmn, but that for backwards compatibility people may
> choose to continue to use zh for Mandarin.
>  (More useful advice would be that because of backwards compatibility
> people should expect that into the indefinite future, "zh" will predominate,
> and should keep that in mind as in using these codes either for tagging
> content or matching content. However, we can live with having impractical
> advice in the spec as long as it is clear that people aren't forced to
> follow it for compliance.)
>
>  Just for information, we reran some analysis and in terms of web pages,
> the number of pages tagged zh-cmn* is approximately 1/250000 of the number
> of pages tagged with zh*. This is way down in the noise level: zh-cmn even
>     (there are even more frequent completely bogus tags; I filtered these
> to ones that had something to do with Chinese).
>
>  Mark
>

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 5 Jun 2008 10:54:36 +0100

I have said this probably too many times now but here goes again...
'zh' is chinese, any sort of chinese, perhaps chinese that at the time of
tagging was not known to be any particular variety.  'zh' ! Mandarin and
'zh' ! Cantonese etc.  'zh' != Mandarin is absolutely not correct!
 
As stated before, whilst we are all now perfectly aware that the 'dominant'
language within data tagged as 'zh' is Mandarin, we should in no way
encourage people to continue using 'zh' for either Mandarin or Cantonese or
anything else other than Chinese unknown/unstated variety.  It is my opinion
that we should clearly state that zh-cmn, or cmn if that is the way the WG
decide to go, MUST be used for all future tagging and that 'zh' will
continue to be treated as Chinese all/any varieties.  If you wish to have an
indicator in the file that clearly shows that a particular extlang is
considered dominant within the Macrolanguage it is quite easy to do.
 
I feel we are going around in circles here.  We have had a straw poll and
yet we seem to still be going down the same route over and over again. 'zh'
does not and should not and must not mean Mandarin!
 
best regards
 
Debbie

  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Shawn Steele
Sent: 05 June 2008 09:12
To: Mark Davis
Cc: LTRU Working Group
Subject: Re: [Ltru] zh != Mandarin

And to be clear and address Karen's concern, its also OK for people to
continue to use zh for Cantonese?
 
- Shawn
  _____  

From: [email-masked] [[email-masked]] On Behalf Of
Mark Davis [[email-masked]]
Sent: Wednesday, June 04, 2008 6:22 PM
To: Shawn Steele
Cc: LTRU Working Group
Subject: Re: [Ltru] zh != Mandarin

zh_tw

zh_cn

ch

zh-hk

zh-classical

zh-tw-tw

gbk

zh-en

zhs

zh-cn-cn

zht

zh_hk

zho

zh-ch

zh-cn-utf-8

zh_ch

zht-tw

zhtw

zh-tw; xmlns

简体中文

英文

...

Mark

I think that lots of people feel that zh != Mandarin and that this should be
clear.  I don't think anyone objects to that?

- Shawn

-- 
Mark

---

From: Peter Constable <[email-masked]>
Date: Thu, 5 Jun 2008 08:01:25 -0700

We know that several of us wholeheartedly hold to the principle that “zh” is any sort of “Chinese”. Shawn asked whether anyone *doesn’t* agree. I’d like to see if there are any who *don’t* agree.

I’m one that does agree with that principle, and because of that, I could not accept having the RFC state the “zh-cmn” (or “cmn”) *MUST* be used for all future tagging of Mandarin. That’s like saying “’zh’ is any kind of Chinese except Mandarin”. We can recommend that “zh-cmn” (or “cmn”) should generally be used for Mandarin, and even that “zh” should not be used for Mandarin in environments that need to distinguish between Mandarin and Cantonese or other Chinese languages. But we need to leave users free to use zh for whatever kind of Chinese they want to use it for, including Mandarin – and potentially only Mandarin, if that suits their needs.

Peter

From: [email-masked] [mailto:[email-masked]] On Behalf Of Debbie Garside
Sent: Thursday, June 05, 2008 2:55 AM
To: Shawn Steele; 'Mark Davis'
Cc: 'LTRU Working Group'
Subject: Re: [Ltru] zh != Mandarin

I have said this probably too many times now but here goes again...
'zh' is chinese, any sort of chinese, perhaps chinese that at the time of tagging was not known to be any particular variety.  'zh' ! Mandarin and 'zh' ! Cantonese etc.  'zh' != Mandarin is absolutely not correct!

As stated before, whilst we are all now perfectly aware that the 'dominant' language within data tagged as 'zh' is Mandarin, we should in no way encourage people to continue using 'zh' for either Mandarin or Cantonese or anything else other than Chinese unknown/unstated variety.  It is my opinion that we should clearly state that zh-cmn, or cmn if that is the way the WG decide to go, MUST be used for all future tagging and that 'zh' will continue to be treated as Chinese all/any varieties.  If you wish to have an indicator in the file that clearly shows that a particular extlang is considered dominant within the Macrolanguage it is quite easy to do.

I feel we are going around in circles here.  We have had a straw poll and yet we seem to still be going down the same route over and over again. 'zh' does not and should not and must not mean Mandarin!

best regards

Debbie

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Shawn Steele
Sent: 05 June 2008 09:12
To: Mark Davis
Cc: LTRU Working Group
Subject: Re: [Ltru] zh != Mandarin
And to be clear and address Karen's concern, its also OK for people to continue to use zh for Cantonese?

- Shawn

zh_tw

zh_cn

ch

zh-hk

zh-classical

zh-tw-tw

gbk

zh-en

zhs

zh-cn-cn

zht

zh_hk

zho

zh-ch

zh-cn-utf-8

zh_ch

zht-tw

zhtw

zh-tw; xmlns

简体中文

英文

...

Mark

- Shawn

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 5 Jun 2008 08:26:41 -0700

I agree wholeheartedly.

2008/6/5 Peter Constable <[email-masked]>:

>  We know that several of us wholeheartedly hold to the principle that "zh"
> is any sort of "Chinese". Shawn asked whether anyone **doesn't** agree.
> I'd like to see if there are any who **don't** agree.
>
>
>
> I'm one that does agree with that principle, and because of that, I could
> not accept having the RFC state the "zh-cmn" (or "cmn") **MUST** be used
> for all future tagging of Mandarin. That's like saying "'zh' is any kind of
> Chinese except Mandarin". We can recommend that "zh-cmn" (or "cmn") should
> generally be used for Mandarin, and even that "zh" should not be used for
> Mandarin in environments that need to distinguish between Mandarin and
> Cantonese or other Chinese languages. But we need to leave users free to use
> zh for whatever kind of Chinese they want to use it for, including Mandarin
> – and potentially only Mandarin, if that suits their needs.
>
>
>
>
>
> Peter
>
>
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf Of
> *Debbie Garside
> *Sent:* Thursday, June 05, 2008 2:55 AM
> *To:* Shawn Steele; 'Mark Davis'
>
> *Cc:* 'LTRU Working Group'
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf Of
> *Shawn Steele
> *Sent:* 05 June 2008 09:12
> *To:* Mark Davis
> *Cc:* LTRU Working Group

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 05 Jun 2008 18:37:39 +0200

Peter Constable 2008-06-05 17.01:

> We know that several of us wholeheartedly hold to the principle
> that “zh” is any sort of “Chinese”. [...]

> I’m one that does agree with that principle, and because of
> that, I could not accept having the RFC state the “zh-cmn” (or
> “cmn”) *MUST* be used for all future tagging of Mandarin.
> That’s like saying “’zh’ is any kind of Chinese except
> Mandarin”.

I agree.

> We can recommend that “zh-cmn” (or “cmn”) should
> generally be used for Mandarin, and even that “zh” should not
> be used for Mandarin in environments that need to distinguish
> between Mandarin and Cantonese or other Chinese languages. But

Agree.

> we need to leave users free to use zh for whatever kind of
> Chinese they want to use it for, including Mandarin – and
> potentially only Mandarin, if that suits their needs.

However, we must require that they use 'zh' in the meaning
'Chinese'. If they need to say 'Mandarin', then they should use
the tag for that.

Apple currently violates this principle for Norwegian, where they
say that 'no' means Bokmål.

Peter, you asked me a question. In my answer [1], I gave an
example of a situation where using 'zh' for tagging Mandarin
content would be good.

[1]http://www.ietf.org/mail-archive/web/ltru/current/msg10807.html
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 5 Jun 2008 10:47:47 -0700

> we should clearly state that zh-cmn, or cmn if that is the way the WG
decide to go,> MUST be used for all future tagging [of Mandarin]...
A bit more on this. (Karen, this one you can mark as a scree ;-)

A standard is not a magic wand. If we cause significant backwards
compatibility problems for implementations, the standard will simply be
ignored or worse yet, we'll get splintering. An example of such a failed
standard is XML 1.1, a story that many of us are familiar with. Even though
the standard was almost completely backwards compatible, because it was not
100% backwards compatible, it failed utterly and completely.

I don't think you realize the scale of the problems that can be caused by
backwards incompatibilities. In a moderately sized organization with large
amounts of code and data, to make a change from zh to zh-cmn (or cmn) could
cost many millions of dollars. The truely complicated part is coordinating
among many internal products plus thousands of vendors and API users so that
there are not breakages. You appear to have no conception of how complicated
this can be. And for a large organization, this is compounded many times.

Now, you may have no sympathy for such organizations, but consider that if a
company is faced with the choice of, say, spending - with *no* particular
benefit to customers - large amounts of money for converting "zh" to
"zh-cmn", versus spending that same amount of money to support more minority
languages, what decision do you think people should make?

If we in RFC 4646bis were to say that one MUST tag Mandarin with zh-cmn (or
cmn), that would be a massive backwards incompatible change from RFC 4646,
inconsistent with the principle of never narrowing, and also inconsistent
with ISO 639-1/2/3. Not to speak of the nonsensical position of denying that
Mandarin is Chinese.

Mark

---

From: John Cowan <[email-masked]>
Date: Thu, 5 Jun 2008 14:44:01 -0400

> 100% backwards compatible, it failed utterly and completely.

This is both OT and controversial, so I'll be brief, say this once,
and then shut up.  IMHO, the hypothetical backward compatibility
issue was the excuse rather than the reason why XML 1.1 was rejected;
the true reasons, insofar as they can be discovered, had more to
do with fear, indifference, and NIH syndrome.

> this can be. And for a large organization, this is compounded many times.

+1 to this and the rest.

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
If a soldier is asked why he kills people who have done him no harm, or a
terrorist why he kills innocent people with his bombs, they can always
reply that war has been declared, and there are no innocent people in an
enemy country in wartime.  The answer is psychotic, but it is the answer
that humanity has given to every act of aggression in history.  --Northrop Frye
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 5 Jun 2008 12:05:26 -0700

I agree that zh-cmn or cmn SHOULD be used. zh will remain valid and includes both Mandarin and Cantonese, so MUST is too strong.

I think this is the last of the disagreement. Mark Davis believes "zh" SHOULD be used. Could we straw poll that?

Karen Broome

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf
>Of Peter Constable
>Sent: Thursday, June 05, 2008 8:01 AM
>To: 'LTRU Working Group'
>We know that several of us wholeheartedly hold to the principle that
>“zh” is any sort of “Chinese”. Shawn asked whether anyone *doesn’t*
>agree. I’d like to see if there are any who *don’t* agree.
>
>
>
>I’m one that does agree with that principle, and because of that, I
>could not accept having the RFC state the “zh-cmn” (or “cmn”) *MUST*
>be used for all future tagging of Mandarin. That’s like saying “’zh’
>is any kind of Chinese except Mandarin”. We can recommend that “zh-
>cmn” (or “cmn”) should generally be used for Mandarin, and even that
>“zh” should not be used for Mandarin in environments that need to
>distinguish between Mandarin and Cantonese or other Chinese
>languages. But we need to leave users free to use zh for whatever
>kind of Chinese they want to use it for, including Mandarin – and
>'zh' is chinese, any sort of chinese, perhaps chinese that at the
>time of tagging was not known to be any particular variety.  'zh' !
>Mandarin and 'zh' ! Cantonese etc.  'zh' != Mandarin is absolutely
>not correct!
>
>
>
>As stated before, whilst we are all now perfectly aware that the
>'dominant' language within data tagged as 'zh' is Mandarin, we
>should in no way encourage people to continue using 'zh' for either
>Mandarin or Cantonese or anything else other than Chinese
>unknown/unstated variety.  It is my opinion that we should clearly
>state that zh-cmn, or cmn if that is the way the WG decide to go,
>MUST be used for all future tagging and that 'zh' will continue to
>be treated as Chinese all/any varieties.  If you wish to have an
>indicator in the file that clearly shows that a particular extlang
>is considered dominant within the Macrolanguage it is quite easy to
>do.
>
>
>
>I feel we are going around in circles here.  We have had a straw
>poll and yet we seem to still be going down the same route over and
>over again. 'zh' does not and should not and must not mean Mandarin!
>
>
>
>best regards
>
>
>
>Debbie
>
>
>
>       And to be clear and address Karen's concern, its also OK for
>people to continue to use zh for Cantonese?
>
>
>
>       - Shawn
>
>       From: [email-masked]
>[[email-masked]] On Behalf Of Mark Davis
>[[email-masked]]
>       I agree with what you say. I don't think anyone objects to zh
>!= Mandarin, given the choices that 639-3 has made. Nor that it is
>valid to tag Mandarin content with 'zh'; it is, after all, Chinese.
>What we worked out before (and this is orthogonal to extlang vs
>nextlang) is to say that it is recommended to use zh-cmn, but that
>for backwards compatibility people may choose to continue to use zh
>for Mandarin.
>
>
>
>       (More useful advice would be that because of backwards
>compatibility people should expect that into the indefinite future,
>"zh" will predominate, and should keep that in mind as in using
>these codes either for tagging content or matching content. However,
>we can live with having impractical advice in the spec as long as it
>is clear that people aren't forced to follow it for compliance.)
>
>
>
>       Just for information, we reran some analysis and in terms of
>web pages, the number of pages tagged zh-cmn* is approximately
>1/250000 of the number of pages tagged with zh*. This is way down in
>the noise level: zh-cmn even occurs far, far less often than
>suspect, irregular or bogus tags like:
>
>
>
>       zh_tw
>
>       zh_cn
>
>       ch
>
>       zh-hk
>
>       zh-classical
>
>       zh-tw-tw
>
>       gbk
>
>       zh-en
>
>       zhs
>
>       zh-cn-cn
>
>       zht
>
>       zh_hk
>
>       zho
>
>       zh-ch
>
>       zh-cn-utf-8
>
>       zh_ch
>
>       zht-tw
>
>       zhtw
>
>       zh-tw; xmlns
>
>       简体中文
>
>       英文
>
>       ...
>
>
>
>       (there are even more frequent completely bogus tags; I
>filtered these to ones that had something to do with Chinese).
>
>
>
>       Mark
>
>
>
>       On Tue, Jun 3, 2008 at 10:17 PM, Shawn Steele
><[email-masked]> wrote:
>
>       I think that lots of people feel that zh != Mandarin and that
>this should be clear.  I don't think anyone objects to that?
>
>       I also believe that there's a general consensus that some
>applications may want to match/filter zh when looking for Mandarin
>(or Cantonese), and that this isn't "best practice", but MAY be
>useful to some applications because of the historical use of zh.
>
>       Is there any disagreement with this?  There's been a lot of
>discussion on this topic, and that seems to be most of Karen's
>concern with any form of "zh".  I *think* we're all saying basically
>the same thing.  Does anyone object to being clear that zh !=
>Mandarin?
>
>       - Shawn

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 5 Jun 2008 12:38:19 -0700

>
> I agree that zh-cmn or cmn SHOULD be used. zh will remain valid and
> includes both Mandarin and Cantonese, so MUST is too strong.
>
> I think this is the last of the disagreement. Mark Davis believes
> "zh" SHOULD be used. Could we straw poll that?
>

SHOULD be used for what? From what I've read Mark Davis believes that he MAY use 'zh' to mean Mandarin.

We already have a "rough consensus": let's work on text instead. Here is what I have revised the draft-15 text to say on the matter:

--
<section title="Using Extended Language Subtags" anchor="choiceUsingExtlang">

<t>The Chinese ('zh') and Arabic ('ar') languages and the various sign languages ('sgn') have a long tradition of using specific primary language subtags, possibly coupled with various region subtags or as part of a registered grandfathered tag, to indicate the language. With the adoption of this document, specific ISO 639-3 assigned subtags became available to identify languages within these diverse language families or groupings. Other than the sign languages, which share a mode of communication rather than any linguistic heritage, this relationship is provided for in ISO 639-3 via a "macrolanguage" mapping. Other languages are encompassed by a macrolanguage and guidance on tagging these languages is provided below. For Arabic and Chinese, however, compatibility with existing tagging practices uses the extended language tag feature to allow tags consistent with user expectations in these language communities.</t>

<t>Chinese ('zh') provides a useful illustration of this.       Many different kinds of content have used tags beginning with the 'zh' subtag, with application specific meaning being associated with region codes, private-use sequences, or grandfathered registered values. This is because historically only the macrolanguage subtag 'zh' was available for forming language tags. However, the languages encompassed by the Chinese subtag are, in the main, not mutually intelligible when spoken. Written forms of these languages also show wide variation in form and usage and many of these languages are written in various contexts.</t>

<t>Rather than require all Chinese content to be retagged, this document provides a special compatibility mechanism: the extended language subtag. Chinese languages encompassed by the 'zh' subtag are in the registry as both primary language subtags and as extended language subtags. For example, the subtag for Cantonese is 'yue'. Content in Cantonese might historically have used a tag such as "zh-HK" (since Cantonese is spoken commonly in Hong Kong), although that tag actually means any type of Chinese as used in Hong Kong. With the availability of ISO 639-3 codes in the registry, content in Cantonese SHOULD use a tag containing the 'yue' subtag. For example, a document written in the Traditional script might use a tag such as "yue-Hant" or "zh-yue-Hant-HK".</t>

<t>Applications MAY use the macrolanguage subtag to form the tag instead of using the more specific encompassed language subtag. For example, an application with large quantities of textual data already using tags with the 'zh' (Chinese) subtag might continue to use this more general subtag even for new data, even though the content could be more precisely be tagged with 'cmn' (Mandarin). Similarly, an application already using tags that start with the 'ar' (Arabic) subtag might continue to use this more general subtag even for new data, which could be more precisely be tagged with 'arb' (Standard Arabic).</t></section>

--

Addison

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 5 Jun 2008 20:41:38 +0100

Hi Mark
 
I would like to understand the problem but first I need to see it... be
patient with me.  See inline comments...

I am not sure how my position wrt 'zh' causes any backward compatibility
problems for implementations.

I am sure that there are many documents that are coded 'zh' that are
Mandarin but I am not saying that you or anyone else should go through all
your documents and re-tag them.  What I am saying is that prior to the
publication of RFC4646bis 'zh' was used to mean Chinese which included
Mandarin and Cantonese. The dominant language encoded with 'zh' is  Mandarin
so future matching scenarios would benefit from some sort of tag (as I
suggested many moons ago) that would identify Mandarin as being the dominant
language within the 'zh' Macrolanguage.  Henceforth all documents should be
encoded with 'zh-cmn'/'cmn'.  I don't see the problem.  Are you saying that
some organizations have tagged *ONLY* Mandarin with 'zh' and they have
created their own tags for Cantonese etc. or not bothered to tag Cantonese
etc. at all?  Historical implementations are not broken by this method as I
am not suggesting deprecating 'zh' and future implementations will benefit
from precision tagging IMHO. 

>> Now, you may have no sympathy for such organizations, but consider that
if a company is faced with the choice of, say, spending - with *no*
particular benefit to customers - large amounts of money for converting "zh"
to "zh-cmn", versus spending that same amount of money to support more
minority languages, what decision do you think people should make?
 
As a business woman with a quarter of a century of experience behind me in
what standards cost in real terms (however big or small the company), I will
have every sympathy if you can give me a true to life *actual*/*factual*
scenario where this is going to cost millions.    As far as I can see
companies/organizations can continue to use 'zh' as they do today but, in
addition, gradually they will see the use of 'zh-cmn' or 'cmn' introduced
with new applications and implementations.  As
ICT's/applications/protocols/implementations are ever changing and software
is often outofdate the minute it hits the streets I cannot see that this is
going to cause a problem for long.  Basically (I feel) you are arguing for
'zh' to mean Mandarin by the back door ('zh'!=Mandarin). I am opposed to
this.  I say again, I am not opposed to saying that the 'dominant' language
within 'zh' is Mandarin. 
 
>> If we in RFC 4646bis were to say that one MUST tag Mandarin with zh-cmn
(or cmn), that would be a massive backwards incompatible change from RFC
4646, inconsistent with the principle of never narrowing, and also
inconsistent with ISO 639-1/2/3. Not to speak of the nonsensical position of
denying that Mandarin is Chinese.
 
Poppycock!  But if you really feel so hot about it you can make it a SHOULD
or PREFERRED.
 
> Not to speak of the nonsensical position of denying that Mandarin is
Chinese.
 
Who said that?
 
I am happy to be corrected.  If I could clearly see that a change from 'zh'
to 'zh-cmn' or 'cmn' would cost industry millions I would be opposed to it
but I am not seeing that here (yet).  I am not hearing this from the
software developers in here.  Instead I am seeing users like Karen saying
(somewhat) the reverse.
 
Debbie

  _____  

From: [email-masked] [mailto:[email-masked]] On
Behalf Of Mark Davis
Sent: 05 June 2008 18:48
To: [email-masked]
Cc: Shawn Steele; LTRU Working Group
Subject: Re: [Ltru] zh != Mandarin

> we should clearly state that zh-cmn, or cmn if that is the way the WG
decide to go, 
> MUST be used for all future tagging [of Mandarin]... 

Mark

 

best regards
 
Debbie

  _____  

Cc: LTRU Working Group

- Shawn
  _____  

zh_tw

zh_cn

ch

zh-hk

zh-classical

zh-tw-tw

gbk

zh-en

zhs

zh-cn-cn

zht

zh_hk

zho

zh-ch

zh-cn-utf-8

zh_ch

zht-tw

zhtw

zh-tw; xmlns

简体中文

英文

...

Mark

- Shawn

-- 
Mark 

-- 
Mark

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 5 Jun 2008 20:54:27 +0100

Addison wrote:

> <t>Applications MAY use the macrolanguage subtag to form the
> tag instead of using the more specific encompassed language
> subtag. For example, an application with large quantities of
> textual data already using tags with the 'zh' (Chinese)
> subtag might continue to use this more general subtag even
> for new data, even though the content could be more precisely
> be tagged with 'cmn' (Mandarin). Similarly, an application
> already using tags that start with the 'ar' (Arabic) subtag
> might continue to use this more general subtag even for new
> data, which could be more precisely be tagged with 'arb'
> (Standard Arabic).</t></section>

I would suggest the following changes:

 <t>Applications MAY use the macrolanguage subtag to form the
 tag instead of using the more specific encompassed language
 subtag. For example, an application with large quantities of
 textual data already using tags with the 'zh' (Chinese)
 subtag might continue to use this more general subtag even
 for new data, even though the content could be more precisely
 be tagged with 'cmn' (Mandarin) or 'yue' (Cantonese). Similarly, an
application
 already using tags that start with the 'ar' (Arabic) subtag
 might continue to use this more general subtag even for new
 data, which could be more precisely tagged with 'arb'
 (Standard Arabic) or any other subtag encompassed by the 'ar'
Macrolanguage.</t></section>

Debbie

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Phillips, Addison
> Sent: 05 June 2008 20:38
> To: Broome, Karen; Peter Constable
> Cc: [email-masked]
> > I think this is the last of the disagreement. Mark Davis
> believes "zh"
> > SHOULD be used. Could we straw poll that?
> >
>
> SHOULD be used for what? From what I've read Mark Davis
> believes that he MAY use 'zh' to mean Mandarin.
>
> We already have a "rough consensus": let's work on text
> instead. Here is what I have revised the draft-15 text to say
> on the matter:
>
> --
> <t>The Chinese ('zh') and Arabic ('ar') languages and the
> various sign languages ('sgn') have a long tradition of using
> specific primary language subtags, possibly coupled with
> various region subtags or as part of a registered
> grandfathered tag, to indicate the language. With the
> adoption of this document, specific ISO 639-3 assigned
> subtags became available to identify languages within these
> diverse language families or groupings. Other than the sign
> languages, which share a mode of communication rather than
> any linguistic heritage, this relationship is provided for in
> ISO 639-3 via a "macrolanguage" mapping. Other languages are
> encompassed by a macrolanguage and guidance on tagging these
> languages is provided below. For Arabic and Chinese, however,
> compatibility with existing tagging practices uses the
> extended language tag feature to allow tags consistent with
> user expectations in these language communities.</t>
>
> <t>Chinese ('zh') provides a useful illustration of this.
>   Many different kinds of content have used tags beginning
> with the 'zh' subtag, with application specific meaning being
> associated with region codes, private-use sequences, or
> grandfathered registered values. This is because historically
> only the macrolanguage subtag 'zh' was available for forming
> language tags. However, the languages encompassed by the
> Chinese subtag are, in the main, not mutually intelligible
> when spoken. Written forms of these languages also show wide
> variation in form and usage and many of these languages are
> written in various contexts.</t>
>
> <t>Rather than require all Chinese content to be retagged,
> this document provides a special compatibility mechanism: the
> extended language subtag. Chinese languages encompassed by
> the 'zh' subtag are in the registry as both primary language
> subtags and as extended language subtags. For example, the
> subtag for Cantonese is 'yue'. Content in Cantonese might
> historically have used a tag such as "zh-HK" (since Cantonese
> is spoken commonly in Hong Kong), although that tag actually
> means any type of Chinese as used in Hong Kong. With the
> availability of ISO 639-3 codes in the registry, content in
> Cantonese SHOULD use a tag containing the 'yue' subtag. For
> example, a document written in the Traditional script might
> use a tag such as "yue-Hant" or "zh-yue-Hant-HK".</t>
>
> <t>Applications MAY use the macrolanguage subtag to form the
> tag instead of using the more specific encompassed language
> subtag. For example, an application with large quantities of
> textual data already using tags with the 'zh' (Chinese)
> subtag might continue to use this more general subtag even
> for new data, even though the content could be more precisely
> be tagged with 'cmn' (Mandarin). Similarly, an application
> already using tags that start with the 'ar' (Arabic) subtag
> might continue to use this more general subtag even for new
> data, which could be more precisely be tagged with 'arb'
> (Standard Arabic).</t></section>
>
> --
>
> Addison

---

From: Peter Constable <[email-masked]>
Date: Thu, 5 Jun 2008 13:34:32 -0700

> From: Broome, Karen [mailto:[email-masked]]

> I think this is the last of the disagreement. Mark Davis believes "zh"

SHOULD be used for what? For any Mandarin? I can't speak for him, but I'd guess his position would be that, for some application scenarios, zh is what should be used for Mandarin, but that we shouldn't make that recommendation across the board -- and I'd agree with that. Every application has to evaluate what makes best sense.

Where there is no reason not to do so and any potential need to differentiate, then I think Mandarin content should be tagged zh-cmn / cmn.

Where there are particular reasons for doing so, I have no problem if implementers of some application decide that, for their purposes, all Mandarin content should be tagged zh. And if someone has no reason why not to tag zh-cmn / cmn but they believe they will never have a need to differentiate and decide they just want to tag their Mandarin content as zh, I don't care at all: it's their prerogative to tag Chinese content as Chinese. And if they decide to describe their content to users as "Mandarin", I might say, "Be careful that doesn't come back to bite you later on," but then leave them to take their own risks.

Peter

---

From: Peter Constable <[email-masked]>
Date: Thu, 5 Jun 2008 13:44:58 -0700

Debbie: you suggested we say that Mandarin content MUST be tagged zh-cmn (or cmn). In terms of back compat, that really is a non-starter. Of course, “cmn” alone would obviously be so, but then you’re not a proponent of that. But even “zh-cmn” will not be an option for some existing implementations that simply have no way to accommodate that tag for any of a variety of reasons (they don’t recognize “cmn”, or they need to use right-truncation in matching zh* and zh-Hant, etc.).

Saying MUST in this case would almost certainly be a last-call blocker for 4646bis: I and I’m sure others here would make every effort to block that.

I don’t think we’re far from a consensus on reasonable recommendations that could be made. Asking for this MUST is definitely moving away from any consensus, IMO.

Peter

From: [email-masked] [mailto:[email-masked]] On Behalf Of Debbie Garside
Sent: Thursday, June 05, 2008 12:42 PM
To: 'Mark Davis'
Cc: 'LTRU Working Group'
Subject: Re: [Ltru] zh != Mandarin

Hi Mark

I would like to understand the problem but first I need to see it... be patient with me.  See inline comments...

>> A standard is not a magic wand. If we cause significant backwards compatibility problems for implementations, the standard will simply be ignored or worse yet, we'll get splintering. An example of such a failed standard is XML 1.1, a story that many of us are familiar with. Even though the standard was almost completely backwards compatible, because it was not 100% backwards compatible, it failed utterly and completely.

I am not sure how my position wrt 'zh' causes any backward compatibility problems for implementations.

>> I don't think you realize the scale of the problems that can be caused by backwards incompatibilities. In a moderately sized organization with large amounts of code and data, to make a change from zh to zh-cmn (or cmn) could cost many millions of dollars. The truely complicated part is coordinating among many internal products plus thousands of vendors and API users so that there are not breakages. You appear to have no conception of how complicated this can be. And for a large organization, this is compounded many times.

I am sure that there are many documents that are coded 'zh' that are Mandarin but I am not saying that you or anyone else should go through all your documents and re-tag them.  What I am saying is that prior to the publication of RFC4646bis 'zh' was used to mean Chinese which included Mandarin and Cantonese. The dominant language encoded with 'zh' is  Mandarin so future matching scenarios would benefit from some sort of tag (as I suggested many moons ago) that would identify Mandarin as being the dominant language within the 'zh' Macrolanguage.  Henceforth all documents should be encoded with 'zh-cmn'/'cmn'.  I don't see the problem.  Are you saying that some organizations have tagged *ONLY* Mandarin with 'zh' and they have created their own tags for Cantonese etc. or not bothered to tag Cantonese etc. at all?  Historical implementations are not broken by this method as I am not suggesting deprecating 'zh' and future implementations will benefit from precision tagging IMHO.

>> Now, you may have no sympathy for such organizations, but consider that if a company is faced with the choice of, say, spending - with *no* particular benefit to customers - large amounts of money for converting "zh" to "zh-cmn", versus spending that same amount of money to support more minority languages, what decision do you think people should make?

As a business woman with a quarter of a century of experience behind me in what standards cost in real terms (however big or small the company), I will have every sympathy if you can give me a true to life *actual*/*factual* scenario where this is going to cost millions.    As far as I can see companies/organizations can continue to use 'zh' as they do today but, in addition, gradually they will see the use of 'zh-cmn' or 'cmn' introduced with new applications and implementations.  As ICT's/applications/protocols/implementations are ever changing and software is often outofdate the minute it hits the streets I cannot see that this is going to cause a problem for long.  Basically (I feel) you are arguing for 'zh' to mean Mandarin by the back door ('zh'!=Mandarin). I am opposed to this.  I say again, I am not opposed to saying that the 'dominant' language within 'zh' is Mandarin.

>> If we in RFC 4646bis were to say that one MUST tag Mandarin with zh-cmn (or cmn), that would be a massive backwards incompatible change from RFC 4646, inconsistent with the principle of never narrowing, and also inconsistent with ISO 639-1/2/3. Not to speak of the nonsensical position of denying that Mandarin is Chinese.

Poppycock!  But if you really feel so hot about it you can make it a SHOULD or PREFERRED.

> Not to speak of the nonsensical position of denying that Mandarin is Chinese.

Who said that?

I am happy to be corrected.  If I could clearly see that a change from 'zh' to 'zh-cmn' or 'cmn' would cost industry millions I would be opposed to it but I am not seeing that here (yet).  I am not hearing this from the software developers in here.  Instead I am seeing users like Karen saying (somewhat) the reverse.

Debbie

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
> we should clearly state that zh-cmn, or cmn if that is the way the WG decide to go,

A standard is not a magic wand. If we cause significant backwards compatibility problems for implementations, the standard will simply be ignored or worse yet, we'll get splintering. An example of such a failed standard is XML 1.1, a story that many of us are familiar with. Even though the standard was almost completely backwards compatible, because it was not 100% backwards compatible, it failed utterly and completely.

I don't think you realize the scale of the problems that can be caused by backwards incompatibilities. In a moderately sized organization with large amounts of code and data, to make a change from zh to zh-cmn (or cmn) could cost many millions of dollars. The truely complicated part is coordinating among many internal products plus thousands of vendors and API users so that there are not breakages. You appear to have no conception of how complicated this can be. And for a large organization, this is compounded many times.

Now, you may have no sympathy for such organizations, but consider that if a company is faced with the choice of, say, spending - with *no* particular benefit to customers - large amounts of money for converting "zh" to "zh-cmn", versus spending that same amount of money to support more minority languages, what decision do you think people should make?

If we in RFC 4646bis were to say that one MUST tag Mandarin with zh-cmn (or cmn), that would be a massive backwards incompatible change from RFC 4646, inconsistent with the principle of never narrowing, and also inconsistent with ISO 639-1/2/3. Not to speak of the nonsensical position of denying that Mandarin is Chinese.

Mark

2008/6/5 Debbie Garside <[email-masked]<mailto:[email-masked]>>:
I have said this probably too many times now but here goes again...
'zh' is chinese, any sort of chinese, perhaps chinese that at the time of tagging was not known to be any particular variety.  'zh' ! Mandarin and 'zh' ! Cantonese etc.  'zh' != Mandarin is absolutely not correct!

best regards

Debbie

________________________________
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Shawn Steele
Sent: 05 June 2008 09:12
To: Mark Davis

Cc: LTRU Working Group

- Shawn
From: [email-masked]<mailto:[email-masked]> [[email-masked]<mailto:[email-masked]>] On Behalf Of Mark Davis [[email-masked]<mailto:[email-masked]>]

zh_tw

zh_cn

ch

zh-hk

zh-classical

zh-tw-tw

gbk

zh-en

zhs

zh-cn-cn

zht

zh_hk

zho

zh-ch

zh-cn-utf-8

zh_ch

zht-tw

zhtw

zh-tw; xmlns

简体中文

英文

...

Mark

- Shawn

--
Mark

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 5 Jun 2008 14:42:45 -0700

Peter wrote:

>> Karen wrote:
>"zh"
>SHOULD be used for what? For any Mandarin?

Yes, any Mandarin. We should not be making application-specific recommendations or suggesting useful hacks. We are making recommendations from the perspective of identifying languages. I believe best practice for identifying these languages -- now that ISO 639-3 is published and will be incorporated -- is to tag Mandarin as "zh-cmn" or "cmn" depending on what we decide on extlang.

>Where there is no reason not to do so and any potential need to
>differentiate, then I think Mandarin content should be tagged zh-cmn
>/ cmn.

If you tag your content as "zh" and I tag mine as "zh-cmn" because I need that specificity, the application that aggregates this content could have problems. Smart applications will need to handle both tags, whether ambiguous or not. But I think we should encourage a transition from the ambiguous tagging to more specific choices. This is important to the data in my industry, which uses a lot more Cantonese tagging than Mark's spiders can see. Increasingly this content is moving to the web and will be tagged. Realistically, many applications will continue to use "zh" variants and this is not wrong. It's just not very specific.

Best practice is to be more specific in Mandarin tagging because of all the audio and video content that is flooding the web today. I don't think we should suggest any use of "zh" alone, though it would be OK to mention that the tag for Chinese languages "zh" remains valid, but should not be used moving forward for Mandarin or Cantonese tagging.

Karen Broome

---

From: Leif Halvard Silli <[email-masked]>
Date: Fri, 06 Jun 2008 00:27:53 +0200

Phillips, Addison 2008-06-05 21.38:

> -- <section title="Using Extended Language Subtags"
> anchor="choiceUsingExtlang">

   [...]

> textual data already using tags with the 'zh' (Chinese) subtag
> data, even though the content could be more precisely be tagged
> with 'cmn' (Mandarin). Similarly, an application already using
> tags that start with the 'ar' (Arabic) subtag might continue to
> use this more general subtag even for new data, which could be
> more precisely be tagged with 'arb' (Standard
> Arabic).</t></section>

I would like these guidelines, for when to avoid and when to 
recommend the macrolanguage tag, to go in:

<t>When using a macrolanguage subtag to form the tag for an 
encompassed language, one should be reluctant to translate its 
meaning – in a user interface or similar – to the spesific name. 
For example, if one wants to label Mandarin as "Chinese", then 
'zh' is fine. But if one wants to label Mandarin as "Mandarin", 
then 'zh' is not appropriate for this. Using 'zh' while displaying 
it as "Mandarin" could lead to compatibility problems and 
confusion amongst consumers and endusers.</t>

<t>Interfaces for choosing between two (or more) encompassed 
languages of the same macrolanguage do wise in considering the 
needs for neutrality and clearness before letting one of the 
encompassed languages being distinguished from the others by the 
use of the macrolanguage tag. For instance, look out for 
macrolanguages whose name equals that of a nationality.</t>

<t>However, the macrolanguage tags might still be suitable even in 
a contexts with several encompassed languages. For example, if a 
system is localized for each and everyone of the encompassed 
langauges of a particular macrolangauge -- for example each of the 
13 encompassed Chinese languages -- the macrolangauge tag might be 
reserved for resources which for some reason are not intended for 
translation, but which still are intended for the users of all the 
encompassed languages.</t>

<t>Or, in a strictly monlinguaal, encompassed language system, for 
example for Mandarin, the 'zh' tag could be reserved -- or open -- 
for content which is not Mandarin, in order to allow coexistens of 
Chinese resourceses which are not Mandarine and wihtout tagging 
them as Mandarin.</t>
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 5 Jun 2008 15:32:44 -0700

That text looks like it represents what I think is the emerging rough
consensus, and I agree that we should work on the text rather than at a
theoretical level.
I have a few minor points below.

> MAY use 'zh' to mean Mandarin.

Yes.

>
>
> We already have a "rough consensus": let's work on text instead. Here is
> what I have revised the draft-15 text to say on the matter:
>
> --
> <t>The Chinese ('zh') and Arabic ('ar') languages and the various sign
> languages ('sgn') have a long tradition of using specific primary language
> subtags, possibly coupled with various region subtags or as part of a
> registered grandfathered tag, to indicate the language. With the adoption of
> this document, specific ISO 639-3 assigned subtags became available to
> identify languages within these diverse language families or groupings.
> Other than the sign languages, which share a mode of communication rather
> than any linguistic heritage, this relationship is provided for in ISO 639-3
> via a "macrolanguage" mapping. Other languages are encompassed by a
> macrolanguage and guidance on tagging these languages is provided below.

=> For most languages encompassed by a macrolanguage, guidance ...

> For Arabic and Chinese, however, compatibility with existing tagging
> practices uses the extended language tag feature to allow tags consistent
> with user expectations in these language communities.</t>

 Did you mean to include sgn here also?

>
>
> <t>Chinese ('zh') provides a useful illustration of this.       Many
> different kinds of content have used tags beginning with the 'zh' subtag,
> with application specific meaning being associated with region codes,
> private-use sequences, or grandfathered registered values. This is because
> historically only the macrolanguage subtag 'zh' was available for forming
> language tags. However, the languages encompassed by the Chinese subtag are,
> in the main, not mutually intelligible when spoken. Written forms of these
> languages also show wide variation in form and usage and many of these
> languages are written in various contexts.</t>
>
> <t>Rather than require all Chinese content to be retagged, this document
> provides a special compatibility mechanism: the extended language subtag.
> Chinese languages encompassed by the 'zh' subtag are in the registry as both
> primary language subtags and as extended language subtags. For example, the
> subtag for Cantonese is 'yue'. Content in Cantonese might historically have
> used a tag such as "zh-HK" (since Cantonese is spoken commonly in Hong
> Kong), although that tag actually means any type of Chinese as used in Hong
> Kong. With the availability of ISO 639-3 codes in the registry, content in
> Cantonese SHOULD use a tag containing the 'yue' subtag. For example, a
> document written in the Traditional script might use a tag such as
> "yue-Hant" or "zh-yue-Hant-HK".</t>

Add "Arabic and Sign Languages are treated similarly."

>
>
> <t>Applications MAY use the macrolanguage subtag to form the tag instead of
> using the more specific encompassed language subtag. For example, an
> application with large quantities of textual data already using tags with
> the 'zh' (Chinese) subtag might continue to use this more general subtag
> even for new data, even though the content could be more precisely be tagged
> with 'cmn' (Mandarin).

I'm ok with Debbie's addition of "or 'yue' (Cantonese)"

> Similarly, an application already using tags that start with the 'ar'
> (Arabic) subtag might continue to use this more general subtag even for new
> data, which could be more precisely be tagged with 'arb' (Standard
> Arabic).</t></section>

I'm ok with Debbie's addition
of "or any other subtag encompassed by the 'ar'
Macrolanguage."

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 5 Jun 2008 15:38:48 -0700

Leif writes:

>meaning - in a user interface or similar - to the spesific name.

I don't think we want to get into specifying user interface names. We are specifying codes for the representation of languages, not names.

>confusion amongst consumers and endusers.</t>

Label specification is out of scope.

>macrolanguages whose name equals that of a nationality.</t>

Not sure this clarifies anything and also seems out of scope.

>encompassed languages.</t>

Agree in principle, but not sure this really helpful.

>Chinese resourceses which are not Mandarine and wihtout tagging

Don't think this is good advice.

Karen

---

From: Shawn Steele <[email-masked]>
Date: Thu, 5 Jun 2008 15:41:54 -0700

Minor note, for the "even though the content could be more precisely be tagged with 'cmn' (Mandarin)." Could it also say Cantonese, "even though the content could be more precisely be tagged with 'cmn' (Mandarin) or 'yue' (Cantonese)"

- Shawn

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Phillips, Addison
Sent: Thursday, June 05, 2008 12:38 PM
To: Broome, Karen; Peter Constable
Cc: [email-masked]
Subject: Re: [Ltru] zh != Mandarin

--

Addison

---

From: Shawn Steele <[email-masked]>
Date: Thu, 5 Jun 2008 15:44:33 -0700

I think I must’ve been confusing, I meant zh != Mandarin as in “zh is NOT equal to Mandarin”, which is what it sounds like you said :)

It sounds like we mostly agree on Addison’s latest text.

- Shawn

From: Debbie Garside [mailto:[email-masked]]
Sent: Thursday, June 05, 2008 2:55 AM
To: Shawn Steele; 'Mark Davis'
Cc: 'LTRU Working Group'
Subject: RE: [Ltru] zh != Mandarin

best regards

Debbie

- Shawn

zh_tw

zh_cn

ch

zh-hk

zh-classical

zh-tw-tw

gbk

zh-en

zhs

zh-cn-cn

zht

zh_hk

zho

zh-ch

zh-cn-utf-8

zh_ch

zht-tw

zhtw

zh-tw; xmlns

简体中文

英文

...

Mark

- Shawn

---

From: Leif Halvard Silli <[email-masked]>
Date: Fri, 06 Jun 2008 00:46:39 +0200

>> subtag might continue to use this more general subtag even

Instead of "might continue etc" I propose "is not forbidden from 
continuing to use".

When you first say that it "MAY", and then you say "might", then 
the "might" sounds much more inviting than the "MAY" does. (It is 
almost like a "feel free to".) Whereas "is not forbidden" sounds 
more like "if there is a good reason for it", while at the same 
time stating that there is nothing wrong with it.

> Macrolanguage.</t></section

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 5 Jun 2008 15:54:01 -0700

Could you summarize what the rough consensus is? I am not so sure we have this. We had one vote, but I don't think it answered all the questions below. As far as I can tell, Mark and I disagree on this point:

Though Mark MAY choose to categorize his new works as "zh", I believe the draft should advise him that he SHOULD NOT do so.

If we say yue is just as good as zh-yue, throw out the shortest tag rule and the idea that there are preferred tags for any given subtag. If there any synonymous preferred subtags, it means ISO 639-6 will be a significantly better solution for me.

Karen

>From: Phillips, Addison [mailto:[email-masked]]

>We already have a "rough consensus": let's work on text instead.
>Here is what I have revised the draft-15 text to say on the matter:
>
>--
><t>The Chinese ('zh') and Arabic ('ar') languages and the various
>sign languages ('sgn') have a long tradition of using specific
>primary language subtags, possibly coupled with various region
>subtags or as part of a registered grandfathered tag, to indicate
>the language. With the adoption of this document, specific ISO 639-3
>assigned subtags became available to identify languages within these
>languages, which share a mode of communication rather than any
>linguistic heritage, this relationship is provided for in ISO 639-3
>macrolanguage and guidance on tagging these languages is provided
>below. For Arabic and Chinese, however, compatibility with existing
>tagging practices uses the extended language tag feature to allow
>tags consistent with user expectations in these language
>communities.</t>
>
>different kinds of content have used tags beginning with the 'zh'
>subtag, with application specific meaning being associated with
>region codes, private-use sequences, or grandfathered registered
>values. This is because historically only the macrolanguage subtag
>'zh' was available for forming language tags. However, the languages
>encompassed by the Chinese subtag are, in the main, not mutually
>intelligible when spoken. Written forms of these languages also show
>wide variation in form and usage and many of these languages are
><t>Rather than require all Chinese content to be retagged, this
>document provides a special compatibility mechanism: the extended
>language subtag. Chinese languages encompassed by the 'zh' subtag
>are in the registry as both primary language subtags and as extended
>language subtags. For example, the subtag for Cantonese is 'yue'.
>Content in Cantonese might historically have used a tag such as "zh-
>HK" (since Cantonese is spoken commonly in Hong Kong), although that
>tag actually means any type of Chinese as used in Hong Kong. With
>the availability of ISO 639-3 codes in the registry, content in
>Cantonese SHOULD use a tag containing the 'yue' subtag. For example,
>a document written in the Traditional script might use a tag such as
><t>Applications MAY use the macrolanguage subtag to form the tag
>instead of using the more specific encompassed language subtag. For
>example, an application with large quantities of textual data
>already using tags with the 'zh' (Chinese) subtag might continue to
>use this more general subtag even for new data, even though the
>content could be more precisely be tagged with 'cmn' (Mandarin).
>Similarly, an application already using tags that start with the
>'ar' (Arabic) subtag might continue to use this more general subtag
>even for new data, which could be more precisely be tagged with
>'arb' (Standard Arabic).</t></section>
>
>--
>
>Addison

---

From: Leif Halvard Silli <[email-masked]>
Date: Fri, 06 Jun 2008 00:57:00 +0200

Shawn Steele 2008-06-06 00.44:

> I think I must’ve been confusing, I meant zh != Mandarin as in
> “zh is NOT equal to Mandarin”, which is what it sounds like you
> said :)

I guess zh is equal to Mandarin. But that it is also equal to
Cantonese. And also equal to both.
-- 
lh
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 5 Jun 2008 16:00:10 -0700

Leif,

zh is not equal to Mandarin. It is equal to "Chinese" which includes Mandarin, Cantonese, and other languages. zh may be used to tag Mandarin, but that doesn't make it synonymous or equal.

Karen

>Of Leif Halvard Silli
>Sent: Thursday, June 05, 2008 3:57 PM
>To: Shawn Steele
>Cc: 'LTRU Working Group'
>> I think I must've been confusing, I meant zh != Mandarin as in
>> "zh is NOT equal to Mandarin", which is what it sounds like you
>> said :)
>

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 5 Jun 2008 16:14:16 -0700

>
> Could you summarize what the rough consensus is? I am not so sure
> we have this. We had one vote, but I don't think it answered all
> the questions below. As far as I can tell, Mark and I disagree on
> this point:

- extlangs: yes, but for 'zh', 'ar', and 'sgn' only
- encompassed languages (i.e. cmn, yue, etc.) are both primary and extlangs
- zh: allowed
- cmn/zh-cmn: recommended

>
> Though Mark MAY choose to categorize his new works as "zh", I
> believe the draft should advise him that he SHOULD NOT do so.

Right this second the specific wording the draft is:

--
The macrolanguage subtag MAY still be used to form the tag instead of the more specific encompassed language subtag. However, the encompassed subtag SHOULD be used to form the language tag.
--

I think this is RFC 2119 equivalent to saying SHOULD NOT. (MAY is more permissive than SHOULD NOT, but SHOULD for the encompassed subtag means "MUST unless you have a Good Reason otherwise", aka "SHOULD NOT").

>
> If we say yue is just as good as zh-yue, throw out the shortest tag
> rule and the idea that there are preferred tags for any given
> subtag.

The "shortest tag" rule is not quite what I think you think it is. The best way to state it would always be: "use as short a tag as possible, but no shorter". In any case, (puts on editor's hat) please suggest textual modifications; please also refer to the complete context rather than isolated items (removes hat again).

> If there any synonymous preferred subtags, it means ISO
> 639-6 will be a significantly better solution for me.

I don't know if ISO 639-6 is a better solution for you (or not). Mark's suggestion to allow both primary and extlang versions of the encompassed languages at the same time is (IMO) ugly, but seems to have support from others. My original suggestion was NOT to include the subtags in both forms. I'm open to either course

Addison

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 5 Jun 2008 16:22:40 -0700

(editor had ON)

Rather than respond to all edit suggestions with lengthy exchanges of a whole section, I have done a new edit, incorporating many of the suggestions. To address Debbie’s comments, I did modify the text extensively to move the normative text out of the Chinese example and to otherwise clarify that we also mean Mandarin and other languages.

Note that I moved the “encompassed languages” section above the “extlangs” section. This is a more natural flow, since the extlangs currently are an exception to the encompassed languages rule.

The documents are on inter-locale. Please send comments to the list.

I note that we do not currently recommend whether xxx or zh-xxx is preferred. We should probably fix that, per Karen’s comments.

Addison

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Thursday, June 05, 2008 3:33 PM
To: Phillips, Addison
Cc: Broome, Karen; Peter Constable; [email-masked]
Subject: Re: [Ltru] zh != Mandarin

That text looks like it represents what I think is the emerging rough consensus, and I agree that we should work on the text rather than at a theoretical level.

Yes.

<t>The Chinese ('zh') and Arabic ('ar') languages and the various sign languages ('sgn') have a long tradition of using specific primary language subtags, possibly coupled with various region subtags or as part of a registered grandfathered tag, to indicate the language. With the adoption of this document, specific ISO 639-3 assigned subtags became available to identify languages within these diverse language families or groupings. Other than the sign languages, which share a mode of communication rather than any linguistic heritage, this relationship is provided for in ISO 639-3 via a "macrolanguage" mapping. Other languages are encompassed by a macrolanguage and guidance on tagging these languages is provided below.

For Arabic and Chinese, however, compatibility with existing tagging practices uses the extended language tag feature to allow tags consistent with user expectations in these language communities.</t>
 Did you mean to include sgn here also?

<t>Applications MAY use the macrolanguage subtag to form the tag instead of using the more specific encompassed language subtag. For example, an application with large quantities of textual data already using tags with the 'zh' (Chinese) subtag might continue to use this more general subtag even for new data, even though the content could be more precisely be tagged with 'cmn' (Mandarin).

I'm ok with Debbie's addition of "or 'yue' (Cantonese)"
Similarly, an application already using tags that start with the 'ar' (Arabic) subtag might continue to use this more general subtag even for new data, which could be more precisely be tagged with 'arb' (Standard Arabic).</t></section>

I'm ok with Debbie's addition of "or any other subtag encompassed by the 'ar'
Macrolanguage."

--

Addison

---

From: Leif Halvard Silli <[email-masked]>
Date: Fri, 06 Jun 2008 02:04:42 +0200

The macrolangauge thing is that two or more languages identify 
themselves the same way (plus a few other things).

So, to split a hair, I consider Chinese a synonym for "All the 
(encompassed) Chinese languages". But I also consider it a synomym 
for Hakka alone. And Mandarin alone. And Cantonese alone. Each 
has a non-exclusive right to target itself with the word "Chinese".

Broome, Karen 2008-06-06 01.00:

> zh is not equal to Mandarin. It is equal to "Chinese" which
> includes Mandarin, Cantonese, and other languages. zh may be
> used to tag Mandarin, but that doesn't make it synonymous or
> equal.

>>> I think I must've been confusing, I meant zh != Mandarin as
>>> in "zh is NOT equal to Mandarin", which is what it sounds
>>> like you said :)
>> 
>>  Cantonese. And also equal to both. -- lh

---

From: Shawn Steele <[email-masked]>
Date: Thu, 5 Jun 2008 17:06:02 -0700

You're describing a superset or set, not a synonym.

---

From: Leif Halvard Silli <[email-masked]>
Date: Fri, 06 Jun 2008 02:29:55 +0200

Perhaps a langauge collection can be described with set theory 
alone. But I don't think a macrolanguage situation can.

Leif

Shawn Steele 2008-06-06 02.06:

> -----Original Message  From:   Leif Halvard Silli

---

From: Leif Halvard Silli <[email-masked]>
Date: Fri, 06 Jun 2008 02:59:29 +0200

Phillips, Addison 2008-06-06 01.22:

> (editor had ON)
> 
> Rather than respond to all edit suggestions with lengthy
> exchanges of a whole section, I have done a new edit,
> incorporating many of the suggestions. To address Debbie’s
> comments, I did modify the text extensively to move the
> normative text out of the Chinese example and to otherwise
> clarify that we also mean Mandarin and other languages.
> 
> Note that I moved the “encompassed languages” section above the
> “extlangs” section. This is a more natural flow, since the
> extlangs currently are an exception to the encompassed
> languages rule.
> 

For the below quote from section 4.1.1, add the word "tag" after 
the word "macrolanguage" (two places) and also move the comma 
after ('wuu') in place.

Currently

  	«For example, Romanian ('ro') and Moldavian ('mo') do not share 
a macrolanguage, [...]  more closely related to each other than 
Cantonese ('yue') and Wu ('wuu') , which do share a macrolanguage.»

Should be

  	«For example, Romanian ('ro') and Moldavian ('mo') do not share 
a macrolanguage **tag**, [...]  more closely related to each other 
than Cantonese ('yue') and Wu ('wuu'), which do share a 
macrolanguage **tag**.»

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 5 Jun 2008 18:35:34 -0700

> the word "macrolanguage" (two places) and also move the comma

The word "tag" is wrong. What they do not share is a macrolanguage. The word "tag" is given a specific meaning, which does not apply in this case.

Addison

---

From: Peter Constable <[email-masked]>
Date: Thu, 5 Jun 2008 18:41:53 -0700

> rule and the idea that there are preferred tags for any given subtag.
> If there any synonymous preferred subtags, it means ISO 639-6 will be a
> significantly better solution for me.

ISO 639-6 is not a better solution for you if it means you abandon working together here to form a consensus. You said in another message,

> If you tag your content as "zh" and I tag mine as "zh-cmn" because I
> need that specificity, the application that aggregates this content
> could have problems. Smart applications will need to handle both tags...

So, evidently interop between your applications and others' is a concern. There will be applications that will want to support A/V scenarios but that have a long history using legacy tags. Because they want to support A/V scenarios, those vendors will likely do work to "handle both tags" within a single system. If you decide to opt for a completely different system, that will only make it harder for them.

Let's all see what are reasonable compromises we can make to reach a consensus.

Peter

---

From: Peter Constable <[email-masked]>
Date: Thu, 5 Jun 2008 18:49:31 -0700

> Leif Halvard Silli

> Currently
>
>         <For example, Romanian ('ro') and Moldavian ('mo') do not share
> Cantonese ('yue') and Wu ('wuu') , which do share a macrolanguage.>
>
> Should be
>
>         <For example, Romanian ('ro') and Moldavian ('mo') do not share
> macrolanguage **tag**.>

If you want to get precise, the categories 'Romanian' and 'Moldavian' do not have in common that there is a single macrolanguage category encompassing both, whereas for the categories 'Cantonese' and 'Wu' there is a single macrolanguage category 'Chinese' that encompasses both.

Peter

---

From: Leif Halvard Silli <[email-masked]>
Date: Fri, 06 Jun 2008 04:09:52 +0200

Phillips, Addison 2008-06-06 03.35:

>> 
>> For the below quote from section 4.1.1, add the word "tag"
>> after the word "macrolanguage" (two places) and also move the
>> comma after ('wuu') in place.
> 
> 
> The word "tag" is wrong. What they do not share is a
> macrolanguage. The word "tag" is given a specific meaning,
> which does not apply in this case.

Do you know if they share a macrolanguage? You only know what is 
registered. Also, human beings may share the same language, 
perhaps, but it sounds strange to me that two languages share 
another language.

New proposal

	"For example, for Romanian ('ro') and Moldavian ('mo') no 
encompassing macrolanguage has been registered, though they are 
far more closely related to each other than Cantonese ('yue') and 
Wu ('wuu') of the Chinese macrolanguage.

-- 
lhs

---

From: Leif Halvard Silli <[email-masked]>
Date: Fri, 06 Jun 2008 04:25:29 +0200

> extlangs currently are an exception to the encompassed

In Section 4.1.1. Because "encompassed" is also used in section 
4.1.2 in slightly other meanings (despite the "spesific meaning" 
that you -- I am sure - try to attribute to it), please change the 
title into

	"Tags for Languages Encompassed by Macrolanguages"

In the same section, please change this text, «The macrolanguages 
Chinese ('zh') and Arabic ('ar') are handled differently. See 
Section 4.1.2.» into this:

	«The encompassed languages of Chinese ('zh') and Arabic ('ar') 
are handled differently. See Section 4.1.2.

Having said this, the above paragraph pretty much hangs in the 
air. Does it say that the previous paragraph does not count for 
Chinees and Arabic? Or that the following pargraphs does not count 
for Chinese/Arabic? Or both?  The next para goes like this:

	«The more specific encompassed language subtag SHOULD be used to 
form the language tag, ...»

Doesn't this rule apply to Chinese and Arabic? Henc eI think the 
entire paragraph regarding "handled differntly - see Section 
4.1.2" should be in the very start of 4.1.1 or in the very end. 
Probably in the very start, and probably it should not say 
"handled differently" but rather something more spesific and 
meaningful (since many things are not different at all).

On a general note,  the extlang section and encompassed lang 
section could have been better sewn together. There are things in 
the extlang section which I think belongs in the encompassed 
langauge section.

I hope to come back to this.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Fri, 06 Jun 2008 05:34:26 +0200

Broome, Karen 2008-06-06 00.38:

>> encompassed language, one should be reluctant to translate
>> its meaning - in a user interface or similar - to the
>> spesific name.
> 
> I don't think we want to get into specifying user interface
> names. We are specifying codes for the representation of
> languages, not names.

Should we be able to trust the tagging - or not?

Users and developers expect that when a certain tag is used, then 
a name which conveys the same meaning is also used. Or else, we 
cannot trust the tagging.

I thought you wanted to avoid zh == Mandarin.

In Mac OS X 10.4 they showed naked tags in the interface. In 10.5 
they translate the tags into names. Including the issue that 'no' 
is being dispalyed as 'Bokmål'. Oh, bad. Suddently the names are 
none of our business anymore, you say?

I don't buy that. One should tag in accordance with one one wants 
to say. One should not twist the meaning of the tags.

>> For example, if one wants to label Mandarin as "Chinese",
>> then 'zh' is fine. But if one wants to label Mandarin as
>> "Mandarin", then 'zh' is not appropriate for this. Using 'zh'
>> while displaying it as "Mandarin" could lead to compatibility
>> problems and confusion amongst consumers and endusers.</t>
> 

The word "label" was chosen to avoid repeating 'tag' over and 
over. I guess this is why the RFC also uses 'label/labeling' 
alltogeter 10 times.

>> languages of the same macrolanguage do wise in considering
>> the needs for neutrality and clearness before letting one of
>> the encompassed languages being distinguished from the others
>> by the use of the macrolanguage tag. For instance, look out
>> for macrolanguages whose name equals that of a
>> nationality.</t>
> 

Why out of scope?

>> <t>However, the macrolanguage tags might still be suitable
>> even in a contexts with several encompassed languages. For
>> example, if a system is localized for each and everyone of
>> the encompassed langauges of a particular macrolangauge --
>> for example each of the 13 encompassed Chinese languages --
>> the macrolangauge tag might be reserved for resources which
>> for some reason are not intended for translation, but which
>> still are intended for the users of all the encompassed
>> languages.</t>
> 

This relates directly to the label thing I mentioed above. If 
Apple had followed these rules, it would have really been helpful 
for Norwegian.

>> <t>Or, in a strictly monlinguaal, encompassed language
>> system, for example for Mandarin, the 'zh' tag could be
>> reserved -- or open -- for content which is not Mandarin, in
>> order to allow coexistens of Chinese resourceses which are
>> not Mandarine and wihtout tagging them as Mandarin.</t>
> 

Ok. Perhaps not.

---

From: Peter Constable <[email-masked]>
Date: Thu, 5 Jun 2008 21:33:53 -0700

> Broome, Karen

> I don't think we want to get into specifying user interface names. We
> are specifying codes for the representation of languages, not names.

I agree with Karen. We should try to impose limitations on implementers in terms of how they relate user-interface to underlying representations. If they want to do something bad, let their users and product-support team correct them.

Peter

---

From: "Debbie Garside" <[email-masked]>
Date: Fri, 6 Jun 2008 08:56:27 +0100

Thanks Peter.  Yes I can see that a MUST on this would be too strong.
 
Best regards
 
Debbie

  _____  

Peter Constable
Sent: 05 June 2008 21:45

Debbie: you suggested we say that Mandarin content MUST be tagged zh-cmn (or
cmn). In terms of back compat, that really is a non-starter. Of course,
“cmn” alone would obviously be so, but then you’re not a proponent of
that. But even “zh-cmn” will not be an option for some existing
implementations that simply have no way to accommodate that tag for any of a
variety of reasons (they don’t recognize “cmn”, or they need to use
right-truncation in matching zh* and zh-Hant, etc.).

Saying MUST in this case would almost certainly be a last-call blocker for
4646bis: I and I’m sure others here would make every effort to block that.

I don’t think we’re far from a consensus on reasonable recommendations
that could be made. Asking for this MUST is definitely moving away from any
consensus, IMO.

Peter

Debbie Garside

Hi Mark

Chinese.

Who said that?

Debbie

  _____  

decide to go, 

Mark

best regards

Debbie

  _____  

Cc: LTRU Working Group

- Shawn

  _____  

zh_tw

zh_cn

ch

zh-hk

zh-classical

zh-tw-tw

gbk

zh-en

zhs

zh-cn-cn

zht

zh_hk

zho

zh-ch

zh-cn-utf-8

zh_ch

zht-tw

zhtw

zh-tw; xmlns

简体中文

英文

...

Mark

- Shawn

-- 
Mark 

-- 
Mark

---

From: "Debbie Garside" <[email-masked]>
Date: Fri, 6 Jun 2008 09:29:20 +0100

I think there is some confusion going on here.  When I first started using
databases (about 25 years ago) I used a system from Informix called SMART
where in SQL '!' means 'Contains' with '=' meaning 'equal'  thus I have
interpreted 'zh!=Mandarin' to mean 'zh' contains or is equal to Mandarin. 
 
Debbie

  _____  

From: Shawn Steele [mailto:[email-masked]] 
Sent: 05 June 2008 23:45
To: [email-masked]; 'Mark Davis'
Cc: 'LTRU Working Group'
Subject: RE: [Ltru] zh != Mandarin
Importance: Low

I think I must’ve been confusing, I meant zh != Mandarin as in “zh is NOT
equal to Mandarin”, which is what it sounds like you said J

- Shawn

best regards

Debbie

  _____  

- Shawn

  _____  

zh_tw

zh_cn

ch

zh-hk

zh-classical

zh-tw-tw

gbk

zh-en

zhs

zh-cn-cn

zht

zh_hk

zho

zh-ch

zh-cn-utf-8

zh_ch

zht-tw

zhtw

zh-tw; xmlns

简体中文

英文

...

Mark

- Shawn

-- 
Mark

---

From: "Debbie Garside" <[email-masked]>
Date: Fri, 6 Jun 2008 11:18:13 +0100

Peter (and Karen) wrote:

> ISO 639-6 is not a better solution for you if it means you
> abandon working together here to form a consensus. You said
> in another message,

I agree with Peter.  It is important to gain a rough consensus here.

> So, evidently interop between your applications and others'
> is a concern. There will be applications that will want to
> support A/V scenarios but that have a long history using
> legacy tags. Because they want to support A/V scenarios,
> those vendors will likely do work to "handle both tags"
> within a single system. If you decide to opt for a completely
> different system, that will only make it harder for them.

ISO 639-6 is not a completely different system, it is an emerging ISO
standard of the ISO 639 family of standards that can (relatively) easily be
incorporated within RFC4646bis or ter if required.  The code (once
published) could be used as private subtags too.  ISO 639-6 subtags could be
placed AFTER the Country Code subtag within the syntax and with a preceding
singleton or it could be situated as a primary language subtag (as the
alpha4 position is already reserved).

Debbie

> Behalf Of Peter Constable
> Sent: 06 June 2008 02:42
> To: [email-masked]
> ISO 639-6 is not a better solution for you if it means you
> abandon working together here to form a consensus. You said
> in another message,
>
> So, evidently interop between your applications and others'
> is a concern. There will be applications that will want to
> support A/V scenarios but that have a long history using
> legacy tags. Because they want to support A/V scenarios,
> those vendors will likely do work to "handle both tags"
> within a single system. If you decide to opt for a completely
> different system, that will only make it harder for them.
>
>

---

From: Martin Duerst <[email-masked]>
Date: Fri, 06 Jun 2008 18:18:39 +0900

[technical hat on]

I generally lean against anything that would specify user interface,
because it's not really considered IETF business. I could immagine
that we have some paragraph or section with advice about user interface.
Maybe we already do, and could add some advice there?

Regards,   Martin.

At 13:33 08/06/06, Peter Constable wrote:
>I agree with Karen. We should try to impose limitations on implementers in 
>terms of how they relate user-interface to underlying representations. If 
>they want to do something bad, let their users and product-support team 
>correct them.
>
>
>
>Peter

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Gerard Meijssen" <[email-masked]>
Date: Fri, 6 Jun 2008 13:57:59 +0200

Hoi,
A user should be able to select the language that he speaks, writes in the
software that he uses. This has nothing to do with the language of the user
interface. The user interface only provides the languages that the software
has been localised in. This is ony a really tiny subset of the languages
that are available.

When people are allowed to express in their writing their own language, it
would be wrong to restrict people in their choice to a subset of the
languages we all agree exist. When people are enabled to express the
languages that they write in, there is no need to allow for ar, sgn or  zh
because a person does not express himself in a multitude of languages and
certainly in the software you cannot have a spellchecker in aggregations of
languages.

When a person has indicated what language he writes in, it is obvious what
language you enter in the meta data. It would be Mandarin when it is
Mandarin, not Chinese.

When we consider it not our business to speak about this, we avoid
discussing what is key in this whole controversy. Key is that zh, ar and sgn
exist to allow for the material that has already been tagged. When we can do
better for new content we should. If this means that companies have to
invest in improving their software in order to allow for more granularity
they should. There is no excuse.

Thanks,
      GerardM

---

From: John Cowan <[email-masked]>
Date: Fri, 6 Jun 2008 09:12:04 -0400

> interpreted 'zh!=Mandarin' to mean 'zh' contains or is equal to Mandarin. 

Ah, no wonder.  No, in this context "!" means "not".  So "zh != Mandarin" means "zh is not equal to
Mandarin", or more precisely "'zh' and 'Mandarin' aren't identical."

-- 
Henry S. Thompson said, / "Syntactic, structural,               John Cowan
Value constraints we / Express on the fly."                 [email-masked]
Simon St. Laurent: "Your / Incomprehensible     http://www.ccil.org/~cowan
Abracadabralike / schemas must die!"
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Fri, 6 Jun 2008 14:57:41 +0100

:-)  I gathered.  But only after the event.

Thanks anyway.

Debbie

> From: John Cowan [mailto:[email-masked]]
> Sent: 06 June 2008 14:12
> To: Debbie Garside
> Cc: 'Shawn Steele'; 'Mark Davis'; 'LTRU Working Group'
> > I think there is some confusion going on here.  When I
> first started
> > using databases (about 25 years ago) I used a system from Informix
> > called SMART where in SQL '!' means 'Contains' with '=' meaning
> > 'equal'  thus I have interpreted 'zh!=Mandarin' to mean
> 'zh' contains or is equal to Mandarin.
>
> Ah, no wonder.  No, in this context "!" means "not".  So "zh
> != Mandarin" means "zh is not equal to Mandarin", or more
> precisely "'zh' and 'Mandarin' aren't identical."
>
> --

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 6 Jun 2008 11:25:45 -0700

Hi -

> From: "Debbie Garside" <[email-masked]>
> To: "'Shawn Steele'" <[email-masked]>; "'Mark Davis'" <[email-masked]>
> Cc: "'LTRU Working Group'" <[email-masked]>
> Sent: Friday, June 06, 2008 1:29 AM

Indeed.  In C, C++, Java, awk, perl, C#, and Python, to name a few,
"!=" is the boolean "is not equal to" operator.  By itself, "!" is
"not" in  many languages.  (I'll ignore the distinction between "~"
and "!" in C-family languages.)  SQL's use of "!" is unusual.

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 6 Jun 2008 11:40:36 -0700

Hi -

As a technical contributor...

> From: "Gerard Meijssen" <[email-masked]>
> To: "Martin Duerst" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Friday, June 06, 2008 4:57 AM
> Subject: Re: [Ltru] User interface (was: Re: zh != Mandarin)
...
> they should. There is no excuse.

Simple counter-example.  If I'm helping out at the library,
I can accurately determine whether printed materials are in Chinese,
Japanese, or Korean.  I cannot, however, reliably distinguish
written Cantonese from Mandarin.  Consequently, 'zh' would be
the correct tag for me to use.  (If they were audio materials,
I'd be able to fairly accurately identify Mandarin and (less accurately)
Cantonese, but not other falvours of 'zh'.)

But back to the point of this thread: I agree that user interface is
out of scope.  We're concerned with tags for the identification of
languages as protocol elements.

Randy

---

From: "Broome, Karen" <[email-masked]>
Date: Fri, 6 Jun 2008 11:53:17 -0700

To put another nail in the user interface coffin, I'd also add that the reason why I piped up is that my experience tells me that the labels one uses in an interface vary significantly depending on

*       the number of language classifications required by the application,
*       whether the labels are being used for classification or retrieval (the same data might be best served with different labels in different contexts),
*       the language the user/classifier speaks, and
*       the sophistication of the classifier.

For example, while inverted forms in English may encourage better tagging in some scenarios, a user interface displaying the user's language choice might do well to display the language name in standard form. This issue has come up in many diverse standards bodies I serve and it's important that one understand the difference between UI labels and back-end metadata naming strategies. Assignment of UI labels are usability decisions, not something that should be standardized.

Cheers,

Karen Broome

>Of Randy Presuhn
>Sent: Friday, June 06, 2008 11:41 AM
>To: [email-masked]

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 6 Jun 2008 14:07:02 -0700

> ISO 639-6 subtags could be
> placed AFTER the Country Code subtag within the syntax

This is not the case, unless you mean in an extension. Variant subtags that start with a letter must be at least 5 characters long. This distinguishes them from script subtags. I know you're thinking "after a country code", but not all language tags have region codes to insulate the variant from the script subtag's position.

> and with a preceding singleton

This would work, either as an extension or in a private-use sequence.

> or it could be situated as a primary language subtag (as
> the alpha4 position is already reserved).

... with modifications in a 4646ter, yes.

Addison

---

From: John Cowan <[email-masked]>
Date: Fri, 6 Jun 2008 17:29:09 -0400

Phillips, Addison scripsit:

> This is not the case, unless you mean in an extension. Variant subtags
> that start with a letter must be at least 5 characters long. This
> distinguishes them from script subtags. I know you're thinking "after a
> country code", but not all language tags have region codes to insulate
> the variant from the script subtag's position.

I have tentatively proposed that in 4646ter, 639-6 code elements that
represent variants (from a 4646 viewpoint) be prefixed with '6' to
make them long enough.  Thus the code element 'zgyu' for Zhongyuanhua
(a dialect of Mandarin) would be "zh-cmn-6zgyu" (or "cmn-6zgyu").

> ... with modifications in a 4646ter, yes.

I'd reserve this use for code elements designating languages or
collections that are not in -3 or -5 respectively.

-- 
Take two turkeys, one goose, four               John Cowan
cabbages, but no duck, and mix them             http://www.ccil.org/~cowan
together. After one taste, you'll duck          [email-masked]
soup the rest of your life.
        --Groucho
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 07 Jun 2008 01:01:10 +0200

Broome, Karen 2008-06-06 20.53:

> To put another nail in the user interface coffin, I'd also add

I demand to open it again.

"User interface" is your tag on this. I spoke about advices for 
how to use the macrolanguage tag.

I don't think that taggers understand MAY and SHOULD - they need 
to understand the issues involved. And what I proposed was not 
absolute in any way.

> that the reason why I piped up is that my experience tells me
> that the labels one uses in an interface vary significantly
> depending on
> 
> * the number of language classifications required by the
> application, * whether the labels are being used for
> classification or retrieval (the same data might be best served
> with different labels in different contexts), * the
> language the user/classifier speaks, and * the
> sophistication of the classifier.

I have no problem understanding this. But I did not speak about 
all these things. I spoke about special issues to be aware of when 
using a macrolanguage tag.

Google has many fine interfaces. Perhaps it is there we shall see 
the 'zh' tag for things labeled as Mandarin first?

> For example, while inverted forms in English may encourage
> better tagging in some scenarios, a user interface displaying
> the user's language choice might do well to display the
> language name in standard form. This issue has come up in many
> diverse standards bodies I serve and it's important that one
> understand the difference between UI labels and back-end

This issue? Which issue? The nittygritty about how to blend the 
macrolanguage tag with the encompassed tags?

> metadata naming strategies. Assignment of UI labels are
> usability decisions, not something that should be standardized.

I can only recommend that you try Mac OS X and see. There 'no' == 
'nb'. Both in the interface (labels). And functionally.

I have probably one of the best, concrete examples of what you 
have been talking about, namely the problem of let the 
macrolanguage tag be seen as equal to one particiular dominant 
encompassed language.
-- 
leif halvard silli

---

From: Peter Constable <[email-masked]>
Date: Fri, 6 Jun 2008 17:19:41 -0700

> incorporated within RFC4646bis or ter if required...

I'm not meaning to suggest that 639-6 isn't that. But 639-6 on its own is a completely different system from the BCP47 series of specs, and alpha-4 language IDs are a significant change for systems that have only ever known alpha-2 or alpha-3 language IDs. That's all I meant.

Peter

---

From: Peter Constable <[email-masked]>
Date: Fri, 6 Jun 2008 17:30:52 -0700

> I demand to open it again.

I consider user interface out of scope. We have had this discussion in the past when people were concerned about description fields in the registry, and we concluded that UI is out of scope. IMO, we should not re-open this topic.

> to understand the issues involved. And what I proposed was not

Anybody is welcome to write a user's guide to language tagging.

> 'nb'. Both in the interface (labels). And functionally.

I sugguest you take the concern to Apple. This WG should not get involved in policing implementations on things that happen to be tangentially related to language tags.

Peter

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 07 Jun 2008 05:32:25 +0200

Quoting ISO 639-3, the section 3.1.9 givs a "what" and a "how" 
about "macrolanguage" and "encompassed language":

  "clusters of closely-related language varieties that [...] can
   be considered distinct individual languages, yet in certain
   usage contexts a single language identity for all is needed"

Hence, I do not understand what the "what" and the "how" in last 
part of section 4.1.1, paragraph 1 really means. Readeres should 
be directed to ask the questions which 639-3 can help answer 
instead of these big "how-s" and "who-s". The French/Spanish 
example also needs to be balanced. I propose a rewording with a 
more direct backlink to 3.1.9:

OLD TEXT:
  "[...] The Macrolanguage mapping doesn't define what the
   relationship between the encompassed language and its
   macrolanguage is, nor does it define how languages encompassed
   by the same macrolanguage are related to each other. Two
   different languages encompassed by the same macrolanguage may
   differ from one another more than say, French and Spanish do."

NEW TEXT:
  "[...] The Macrolanguage mapping doesn't define *the spesific
   usage contexts where the shared language identity of the
   macrolanguage is needed*, nor does it define *any particular
   degree of relationship between the* languages encompassed by
   the same macrolanague. *This single language identity in
   certain usage contexts does for some macrolanguages mean that
   their encompassed languages possess a high degree of mutual
   intelligibility. For other macrolanguages,* two *particular
   encompassed languages* may differ from one another more than
   say, French and Spanish do."

In paragraph 5, the sentence before "Romaninan/Moldavian" also 
makes me wonder. I've rewritten it to say what I think is meant 
(Even if I think that section 4.1.1 is very repetive.)

NEW TEXT:
  "[...] However, the macrolanguage field does not guarantee such
   a degree of relationship nor is such a degree of relationship
   reserved for the members of a macrolanguage. Romanian ('ro')
   and Moldavian ('mo') do not share any macrolanguage, but [...]"
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Sat, 7 Jun 2008 09:30:46 +0100

True :-)

Debbie

> Sent: 07 June 2008 01:20
> Subject: Re: [Ltru] ISO 639 was RE: zh != Mandarin
>
> > standard of the ISO 639 family of standards that can (relatively)
> > easily be incorporated within RFC4646bis or ter if required...
>
> I'm not meaning to suggest that 639-6 isn't that. But 639-6
> on its own is a completely different system from the BCP47
> series of specs, and alpha-4 language IDs are a significant
> change for systems that have only ever known alpha-2 or
> alpha-3 language IDs. That's all I meant.
>
>
>
> Peter

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 07 Jun 2008 17:52:08 +0200

Peter Constable 2008-06-07 02.30:

>> Behalf Of Leif Halvard Silli> 
> I consider user interface out of scope.

I proposed to open the coffin under its real name.

> We have had this

> discussion in the past when people were concerned about
> description fields in the registry, and we concluded that UI is
> out of scope. IMO, we should not re-open this topic.

The key criteria in 639-3 for the 'macrolanguage' designation is 
that for certain "clusters of closely-related language varieties 
[...] in **certain usage contexts** a single language identity for 
all is needed"

And so a focus on "usage contexts" seems appropriate when RFC-ing 
how to use the macrolanguage subtag. Rather than an overfocus on 
whether encompassed languages are good for fallback or not.

I just proposed some changes to section 4.1.1. to alleviate this. 
I hope you find the propoal useful or that you will help improve it.

>> I don't think that taggers understand MAY and SHOULD - they
>> need to understand the issues involved. And what I proposed
>> was not absolute in any way.
> 

So let's put the texts you did not like to the side, and and let 
us consentrate on finding the right way to convey to those user 
guide authors, that the language tags for macrolanguages are 
suitable for "certain usage contexts [where] a single language 
identity for all  [encompassed languages of a particular 
macrolanguage] is needed".

Perhaps we should simply have section in the RFC called

    4.1.x Tagging Macrolanguages

?

>> I can only recommend that you try Mac OS X and see. There
>> 'no' == 'nb'. Both in the interface (labels). And
>> functionally.
> 
> I sugguest you take the concern to Apple. This WG should not
> get involved in policing implementations on things that happen
> to be tangentially related to language tags.

Private solutions to common problems? I am sure anyone gets that 
the above was merly an example of what issues it leads to if there 
is no clear guidane regarding "usage context".

Martin allready said that withouth extlang he rather wanted only 
two language tags for Estonian and Voro. I.e. he did not want 
macrolanguage status for Estonian unless there were extlang. It 
seems that Apple has come to the same conclusion ... and so they 
"undid" the macrolanguage tag.

However, for many language situations (such as for Norwegian) 
there is no choice between going for extlang or encompassed 
languages. And for these the issue of 3 tags for 2 languages will 
be confusing unless there is clear guidance about "usage contexts".

Like I said: I was convinced that one either should have extlang 
for all encompassed languages or no extlang at all, in order to 
ensure that there that the problems of *all* the 
macrolanguage/encompassed languages are taken seriously. I changed 
my view to support those that wanted extlang for only some 
languages. (Those "some languges" have since been reduced to "some 
cherry picked languages".)

I suppose that merely having "no" labeled as macrolanguage will be 
of good help - in communication with Vendor X or Tagger Y. But 
please do no not fullfill my original fear that the problems for 
those that did not get a extlang solution would get ignored.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru