Subject: [Ltru] solving the chinese thing... some text (part 1)

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 8 May 2008 12:58:59 -0700

I'd like to propose a series of edits to try and resolve the Chinese/macrolanguage thread. In particular, I think clear treatment of Chinese tag choice in the section on tag choice makes sense. In a separate email I will address Shawn's proposal regarding lists. In this email I suggest we replace this text:

--
In particular, the Chinese family of languages call for special consideration. Because the written form is very similar for most languages having 'zh' (Chinese) as a macrolanguage (and because historically subtags for the various encompassed languages were not available), languages such as 'yue' (Cantonese) have historically used either 'zh' or a tag (now grandfathered) beginning with 'zh'. This means that macrolanguage information can be usefully applied when searching for content or when providing fallbacks in language negotiation. For example, the information that 'yue' has a macrolangauge of 'zh' could be used in the Lookup algorithm to fallback from a request for "yue-Hans-CN" to "zh-Hans-CN" without losing the script and region information (even though the user did not specify "zh-Hans-CN" in their request).
--

... with this longer text on Chinese:

--
<t>In particular, the family of languages belonging to the macrolanguage Chinese ('zh') call for special consideration. Because the written form is very similar for most languages, textual content SHOULD use the 'zh' subtag, possibly with a script subtag (indicating, for example, the difference between the 'Hant' traditional script and the 'Hans' simplified script), region, or other subtags to deliniate any additional or local variations in language. Use of specific encompassed language subtags is permitted and could make a better choice when there are language-specific variations (as when a written record is created from the spoken material or in colloquial contexts that call for such a differentiation). </t>

<t>For other applications of language tags on Chinese language content, the use of the specific encompassed language subtag is RECOMMENDED. For example, a song, audio recording, or movie soundtrack in Cantonese might use a tag such as "yue-HK", indicating that it uses Cantonese as used in Hong Kong, whereas a similar content item in Mandarin Chinese might use a tag such as "cmn-HK".</t>

<t> Historically subtags for the various encompassed Chinese languages were not available, so  content in these languages have historically used either 'zh' or a tag (now grandfathered) beginning with 'zh'.  These tags were deprecated in the registry upon adoption of this document, but remain valid for use. The modern forms are preferred, however, since implementations might not be able to associate the older tags with the modern equivalents. Implementations are encouraged to canonicalize tags for comparison purposes.</t>

<t> Finally, macrolanguage information can be usefully applied when searching for content or when providing fallbacks in language negotiation.  For example, the information that 'yue' has a macrolangauge of 'zh' could be used in the Lookup algorithm (<xref target="RFC4647"></xref>) to include the range "zh-Hans-CN" as a fallback from a request for "yue-Hans-CN" (preserving the script and region information) even though the user did not specify "zh-Hans-CN" in their request. For the Chinese languages in particular, this practice of conflating the encompassed langauge with its macrolanguage is RECOMMENDED.</t>
--

In addition, I propose that we remove 'zh'/'cmn' from the list of "dominant varieties" in the table in that section. That would seem to address most of the concerns about tagging Chinese as I understand them. Note that a discussion of text about "lists" is intended to follow.

Addison

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Thu, 8 May 2008 16:19:32 -0400

> ... with this longer text on Chinese:

+1

-- 
"But I am the real Strider, fortunately,"       John Cowan
he said, looking down at them with his face     [email-masked]
softened by a sudden smile.  "I am Aragorn son  http://www.ccil.org/~cowan
of Arathorn, and if by life or death I can
save you, I will."  --LotR Book I Chapter 10
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Shawn Steele <[email-masked]>
Date: Thu, 8 May 2008 13:53:32 -0700

I'm concerned about the first paragraph.  From the earlier discussion it sounds like Mandarin and Cantonese written text are NOT interchangeable, so I don't think that the "zh" tag is appropriate for both, particularly if both are available in a resource.

- Shawn

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Phillips, Addison
Sent: Thursday, May 08, 2008 12:59 PM
To: 'LTRU Working Group'
Subject: [Ltru] solving the chinese thing... some text (part 1)

Addison

Addison Phillips

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 8 May 2008 14:16:16 -0700

Mandarin and Cantonese text are not interchangeable. However, most *text* is actually Standard Mandarin. Written Cantonese is covered by the last sentence about colloquial contexts. I note Mark's quoting of John Jenkins quoting:

--
Meanwhile, I quote from Stephen Matthews and Virginia Yip, _Cantonese: A Comprehensive Grammar_ (London: Routledge, 1994), pp. 5-6:  "Traditionally, Cantonese has been regarded as one of the many Chinese dialects. It does not have a standardized written form on a par with standard written Chinese.  No form of written Cantonese is taught in schools or used in academic settings in any Cantonese-speaking community.  When it comes to the written form, it is standard written Chinese that is taught and learnt.  For educated Cantonese speakers, standard written Chinese is the written form they use in most contexts.  However, in colloquial genres such as novels, popular magazines, newspaper gossip columns, informal personal communications, written Cantonese may be used.  When written Cantonese contains too many exclusively Cantonese words and expressions, non-Cantonese speakers may find it totally unintelligible."
--

We don't say they are interchangeable. But the fact is that most *writing* is in a standard form. When you have a distinction, you tag appropriately.

I also caution strongly: this discussion has tended to focus obsessively on Mandarin/Cantonese as an example pair. But zh-coded-Chinese actually encloses 13 languages in ISO 639-3, many of which have dialects of their own with large speaker populations. Suggesting that "standard written Chinese" documents should get 13 extra tags associated with them "just to be sure" is a non-starter.

I don't diminish the need for specificity when appropriate. But nearly all literate Chinese read 'zh' and the distinction for (say) a Cantonese document is from that form. This is in stark contrast from audio content, where the choice is critical.

Addison

Addison Phillips

> From: Shawn Steele [mailto:[email-masked]]
> Sent: Thursday, May 08, 2008 1:54 PM
> To: Phillips, Addison; 'LTRU Working Group'
> Subject: RE: solving the chinese thing... some text (part 1)
>
> I'm concerned about the first paragraph.  From the earlier discussion
> it sounds like Mandarin and Cantonese written text are NOT
> interchangeable, so I don't think that the "zh" tag is appropriate for
> both, particularly if both are available in a resource.
>
>
> - Shawn
>
> I'd like to propose a series of edits to try and resolve the
> Chinese/macrolanguage thread. In particular, I think clear treatment of
> Chinese tag choice in the section on tag choice makes sense. In a
> separate email I will address Shawn's proposal regarding lists. In this
> email I suggest we replace this text:
>
> --
> In particular, the Chinese family of languages call for special
> consideration. Because the written form is very similar for most
> languages having 'zh' (Chinese) as a macrolanguage (and because
> historically subtags for the various encompassed languages were not
> available), languages such as 'yue' (Cantonese) have historically used
> either 'zh' or a tag (now grandfathered) beginning with 'zh'. This
> means that macrolanguage information can be usefully applied when
> searching for content or when providing fallbacks in language
> negotiation. For example, the information that 'yue' has a
> macrolangauge of 'zh' could be used in the Lookup algorithm to fallback
> from a request for "yue-Hans-CN" to "zh-Hans-CN" without losing the
> script and region information (even though the user did not specify
> "zh-Hans-CN" in their request).
> --
>
> <t>In particular, the family of languages belonging to the
> macrolanguage Chinese ('zh') call for special consideration. Because
> the written form is very similar for most languages, textual content
> SHOULD use the 'zh' subtag, possibly with a script subtag (indicating,
> for example, the difference between the 'Hant' traditional script and
> the 'Hans' simplified script), region, or other subtags to deliniate
> any additional or local variations in language. Use of specific
> encompassed language subtags is permitted and could make a better
> choice when there are language-specific variations (as when a written
> record is created from the spoken material or in colloquial contexts
> that call for such a differentiation). </t>
>
> <t>For other applications of language tags on Chinese language content,
> the use of the specific encompassed language subtag is RECOMMENDED. For
> example, a song, audio recording, or movie soundtrack in Cantonese
> might use a tag such as "yue-HK", indicating that it uses Cantonese as
> used in Hong Kong, whereas a similar content item in Mandarin Chinese
> might use a tag such as "cmn-HK".</t>
>
> <t> Historically subtags for the various encompassed Chinese languages
> were not available, so  content in these languages have historically
> used either 'zh' or a tag (now grandfathered) beginning with 'zh'.
> These tags were deprecated in the registry upon adoption of this
> document, but remain valid for use. The modern forms are preferred,
> however, since implementations might not be able to associate the older
> tags with the modern equivalents. Implementations are encouraged to
> canonicalize tags for comparison purposes.</t>
>
> <t> Finally, macrolanguage information can be usefully applied when
> searching for content or when providing fallbacks in language
> negotiation.  For example, the information that 'yue' has a
> macrolangauge of 'zh' could be used in the Lookup algorithm (<xref
> target="RFC4647"></xref>) to include the range "zh-Hans-CN" as a
> fallback from a request for "yue-Hans-CN" (preserving the script and
> region information) even though the user did not specify "zh-Hans-CN"
> in their request. For the Chinese languages in particular, this
> practice of conflating the encompassed langauge with its macrolanguage
> is RECOMMENDED.</t>
> --
>
> In addition, I propose that we remove 'zh'/'cmn' from the list of
> "dominant varieties" in the table in that section. That would seem to
> address most of the concerns about tagging Chinese as I understand them.
> Note that a discussion of text about "lists" is intended to follow.
>
> Addison
>
> Addison Phillips

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 23:27:19 +0200

Phillips, Addison 2008-05-08 21.58:
> <t>In particular, the family of languages belonging

1) 'family' can be interpreted as 'linguistic family'; Perhaps 'group'?
2) perhaps 'covered by' - or 'encompassed by' instead of 'belonging' - 
since that word is again connected to 'family'.

I do of course wonder why it is 'should use "zh"'. But perhaps it is OK. 
I must say, though, that this sets a precendence of other macrolanguages.

>  to the macrolanguage Chinese ('zh') call for special consideration. Because the written form is very similar for most languages, textual content SHOULD use the 'zh' subtag, possibly with a script subtag (indicating, for example, the difference between the 'Hant' traditional script and the 'Hans' simplified script), region, or other subtags to deliniate any additional or local variations in language. Use of specific encompassed language subtags is permitted and could make a better choice when there are language-specific variations (as when a written record is created from the spoken material or in colloquial contexts that call for such a differentiation). </t>
>

This 'for other applications' how does it differ from the application 
thought of in first paragraph? HWat is the application thought of in the 
first para?

Assume this is the context where you will bring in "lists".

> <t> Finally, macrolanguage information can be usefully applied when searching for content or when providing fallbacks in language negotiation.  For example, the information that 'yue' has a macrola…

The phrase "For the Chinese languages *in particular*," means that "this 
practice of conflating the encompassed langauge with its macrolanguage 
is RECOMMENDED" for all  enocompassed languages of any macrolanguage. At 
least, that is how I read 'in particular'.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: [email-masked]
Date: Thu, 8 May 2008 14:32:41 -0700

Addison wrote on 05/08/2008 12:58:59 PM:

> I'd like to propose a series of edits ...

> SHOULD use the 'zh' subtag, possibly with a script subtag 
> (indicating, for example, the difference between the 'Hant' 
> traditional script and the 'Hans' simplified script), region, or 
> other subtags to deliniate any additional or local variations in 
> language. Use of specific encompassed language subtags is permitted 
> and could make a better choice when there are language-specific 
> variations (as when a written record is created from the spoken 
> material or in colloquial contexts that call for such a 
differentiation). </t>
> 
> <t>For other applications of language tags on Chinese language 
> content, the use of the specific encompassed language subtag is 
> RECOMMENDED. For example, a song, audio recording, or movie 
> soundtrack in Cantonese might use a tag such as "yue-HK", indicating
> that it uses Cantonese as used in Hong Kong, whereas a similar 
> content item in Mandarin Chinese might use a tag such as "cmn-HK".</t>
> 
> <t> Historically subtags for the various encompassed Chinese 
> languages were not available, so  content in these languages have 
> historically used either 'zh' or a tag (now grandfathered) beginning
> with 'zh'. 

Not necessarily true. Definitely not true in the data I've seen. 
Indicating that Cantonese docs typically use "zh" is different from the 
statement that the bulk of "zh" documents are Mandarin. My experience is 
that the users who need to specify Cantonese most often make up an illegal 
tag. Not saying that's what we should recommend, but I believe my 
experience does not support the statement as worded. Typo on "deliniate" 
FWIW.

> These tags were deprecated in the registry upon adoption
> of this document, but remain valid for use. 

"zh" is not deprecated -- yet. ;) I don't think that's what what you're 
trying to say, but that's how it reads.

The modern forms are 
> preferred, however, since implementations might not be able to 
> associate the older tags with the modern equivalents. 

Not entirely clear what is meant by "modern forms." 

> Implementations are encouraged to canonicalize tags for comparison 
> purposes.</t>
> 
> region information) even though the user did not specify "zh-Hans-
> CN" in their request. For the Chinese languages in particular, this 
> practice of conflating the encompassed langauge with its 
> macrolanguage is RECOMMENDED.</t>

Add "with written content" to the end of the last sentence? Useless with 
audio.

Regards,

Karen Broome

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 8 May 2008 14:40:58 -0700

Actually, it doesn't, as it is specifically and specially called out:

>
> >  to the macrolanguage Chinese ('zh') call for special consideration.

> > <t>For other applications of language tags on Chinese language
...
> thought of in first paragraph? HWat is the application thought of in

Perhaps instead of "other applications" should say "other types of content" instead.

> Assume this is the context where you will bring in "lists".

No, it isn't. Lists do not belong, I think, in the section on "choice of language tag" (note singular 'tag').

> is RECOMMENDED" for all  enocompassed languages of any macrolanguage.
> At least, that is how I read 'in particular'.

No, the paragraph says, basically, you MAY use macrolanguage information in this manner and you SHOULD use it in particular for Chinese.

Addison

---

From: John Cowan <[email-masked]>
Date: Thu, 8 May 2008 17:49:15 -0400

> 1) 'family' can be interpreted as 'linguistic family'; Perhaps 'group'?

The languages encompassed by zh actually do constitute a linguistic family, Sinitic.
There is one exception, Dungan, which is actually inside the Mandarin dialect
continuum and fully intelligible with neighboring northwestern Mandarin varieties,
but is sociolinguistically separate because its speakers don't live in China and
use a standardized Cyrillic orthography (not transliteration) for all purposes.
Dungan aside, though, it is fair to speak of a family.

> since that word is again connected to 'family'.

"Encompassed by" is probably better.

> least, that is how I read 'in particular'.

That is a plain misreading.  "In particular" does not mean "by way of example";
it emphasizes the uniqueness of the case.

-- 
Time alone is real                      John Cowan <[email-masked]>
  the rest imaginary
like a quaternion       --phma          http://www.ccil.org/~cowan
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 8 May 2008 16:37:11 -0700

> statement that the bulk of "zh" documents are Mandarin.

Ignoring illegal tags for a minute, items that were correctly tagged as being in Cantonese (or some other Chinese encompassed language) used a tag starting with 'zh' because those are the grandfathered ones available.

> My experience is
> experience does not support the statement as worded.

Nothing we can do about illegal tags. Not much we can say about them either, except "don’t do that".

> Typo on
> "deliniate" FWIW.

Thanks.

To address the above comments, I rewrote that paragraph to read:

--
<t>Historically subtags for the various encompassed Chinese languages were not available. To overcome this deficiency, some content in these languages used grandfathered tags registered for specific languages (such as "zh-yue" for Cantonese or "zh-xiang" for Xiang) or used regional subtag combinations such as "zh-HK" (Chinese, Hong Kong) to imply a specific encompassed language.  The grandfathered tags were deprecated in the registry upon adoption of this document, but remain valid for use. The other tag combinations remain valid as well, but their meaning is not sufficiently precise. Instead, tags using the encompassed language subtags  are preferred. Since implementations might not be able to associate the grandfathered tags with these modern equivalents, implementations are encouraged to canonicalize tags for comparison purposes.</t>
--

> Add "with written content" to the end of the last sentence? Useless

Not necessarily. See Peter's scenario for older content classifications... or for mixed-media content... or even because people "make up their own tagging rules" :-). It wouldn't hurt if people completely eschew the use of 'zh-*' for their audio. But my guess is that they won't all "get the memo", so even with audio content it may help. Finding "some Chinese" after you have failed to find any Cantonese might be a better choice for some applications than finding, say, French instead.

Addison

---

From: Shawn Steele <[email-masked]>
Date: Thu, 8 May 2008 20:02:54 -0700

In the case you're describing, it sounds like the text is Mandarin.  It isn't a Chinese macrolanguage, it is "Standard Mandarin".  If it is clearly Mandarin, why recommend tagging it as just "Chinese"?  If I read your statement correctly, it sounds written Chinese is usually Mandarin, not one of 12 other languages.

- Shawn

-----Original Message-----
From: Phillips, Addison [mailto:[email-masked]]
Sent: Thursday, May 08, 2008 2:16 PM
To: Shawn Steele; 'LTRU Working Group'
Subject: RE: solving the chinese thing... some text (part 1)

Addison

Addison Phillips

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 9 May 2008 07:42:59 -0700

I like the direction that this is going. More on Monday.

MD

On Thu, May 8, 2008 at 1:19 PM, John Cowan <[email-masked]> wrote:

> softened by a sudden smile.  "I am Aragorn son  http://www.ccil.org/~cowan<http://www.ccil.org/%7Ecowan>

-- 
Mark

---

From: Peter Constable <[email-masked]>
Date: Sat, 10 May 2008 08:54:23 -0700

> Chinese/macrolanguage thread...

I'm just starting to look at this thread. Some of my comments may already be superseded by discussion I haven't yet looked at; if so, sorry in advance for the distraction.

> I suggest we replace this text:
> SHOULD use the 'zh' subtag

I'm wondering if SHOULD is even too strong. I think I'd feel more comfortable with something along this line:

"Written Mandarin is widely used, and the subtag 'zh' has most often been used for Mandarin textual content. Other Chinese languages are written in some usage contexts, however, and those written forms are distinct from written Mandarin. Because of prevalent use of 'zh' for written Chinese, it will make sense to use 'zh' for written Chinese in many application scenarios, possibly with a script subtag (indicating...), region or other subtags to indicate some variation. Use of specific, encompassed language subtags is permitted, however, and may be a better choice in some application scenarios. For example, Cantonese video subtitles written with Simplified characters could be tagged 'yue-Hans'; or video subtitles in Taiwanese Mandarin written with Traditional characters could be tagged 'cmn-Hant-TW'."

> practice of conflating the encompassed langauge with its macrolanguage

It would probably make sense to mention the use of language priority lists with 'zh' and also a specific encompassed variety to indicate user preferences.

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 10 May 2008 16:16:41 -0600

Works for me.

On Sat, May 10, 2008 at 9:54 AM, Peter Constable <[email-masked]>
wrote:

> I'm just starting to look at this thread. Some of my comments may already
> be superseded by discussion I haven't yet looked at; if so, sorry in advance
> for the distraction.
>
>
> I'm wondering if SHOULD is even too strong. I think I'd feel more
> comfortable with something along this line:
>
> "Written Mandarin is widely used, and the subtag 'zh' has most often been
> used for Mandarin textual content. Other Chinese languages are written in
> some usage contexts, however, and those written forms are distinct from
> written Mandarin. Because of prevalent use of 'zh' for written Chinese, it
> will make sense to use 'zh' for written Chinese in many application
> scenarios, possibly with a script subtag (indicating...), region or other
> subtags to indicate some variation. Use of specific, encompassed language
> subtags is permitted, however, and may be a better choice in some
> application scenarios. For example, Cantonese video subtitles written with
> Simplified characters could be tagged 'yue-Hans'; or video subtitles in
> Taiwanese Mandarin written with Traditional characters could be tagged
> 'cmn-Hant-TW'."
>
>
>
> It would probably make sense to mention the use of language priority lists
> with 'zh' and also a specific encompassed variety to indicate user
> preferences.
>
>
>
> Peter

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 13 May 2008 15:01:43 +0200

On Thu, May 08, 2008 at 12:58:59PM -0700,
 Phillips, Addison <[email-masked]> wrote 
 a message of 33 lines which said:

> For the Chinese languages in particular, this practice of conflating
> the encompassed langauge with its macrolanguage is RECOMMENDED.</t>

I find this text questionable, because it seems to imply there is a
Chinese-specific issue. I'm not sure of it and I'm not convinced that
Arabic is much different in that respect.

I suggest instead:

<t> ... this practice of conflating the encompassed langauge with its
macrolanguage is RECOMMENDED or not, depending on various informations
which are not in the registry (existence of a dominant form of the
language, amount of legacy tagging, etc). These informations must be
collected and analyzed by the programmer before deciding wether or not
the application will conflate the encompassed language with its
macrolanguage.</t>