Subject: [Ltru] tags for Chinese

From: Peter Constable <[email-masked]>
Date: Sun, 1 Jun 2008 18:11:54 -0700

[warning: this is very long]

On May 27, Debbie wrote:

> > > we should consider each of the applications of language tags:
> > > identification, lookup, filtering, and Accept-Language, and
> > be able to
> > > have a reasoned judgment on the technical merits
> >
> > I would only add: we need to do this with *carefully*
> > reasoned judgment, pausing to make sure the arguments
> > presented really do stand up.
>
> I think this is probably a good way forward.  It would be best to start
> a new thread for each application.  I would also add backward
> compatibility and historical usage to the pot.

In the spirit of this, I'm going to attempt some analysis here of Chinese tags. I'm putting this all in one mail rather than separate threads per application since I'm organizing what I write around different conceptual categories and their corresponding tags rather than by applications. There are lots of details to sift, and in trying to get something put down and sent out I know I haven't done as careful analysis as could be done -- certainly I haven't gotten much past cataloguing lots of facts.

I hope this will be useful, and that responses will take the analysis further. If you want to go on a tangent, please start a different thread.

On May 29, Karen wrote:

> here's my real-world Chinese language list:
>
> Chinese (Variant Unknown)
> Chinese (Cantonese, Spoken)
...

I gather Karen's approach here has been to take different "Chinese" categories of interest to her and consider what tag would be recommended for each under different proposals we might adopt. I'd like to approach things from a different direction for a moment: list what "Chinese" tags would be available under each of three proposals and consider what we might say about each tag.

The three proposals are (1) extlang, (2) no-extlang, (3) mixed: extlang, but with IDs for encompassed languages also allowed in first-subtag position.

Before proceeding, I'll briefly review matching processes and scenarios:

- Filtering returns a number of records from a (typically sizeable) corpus. Here, a record tagged with greater specificity than was indicated in the query can always be returned (e.g., if user asks for en, then en-GB is certainly valid); relevance of record that are tagged more generically (e.g., en when user asked specifically for en-GB) is very much in question. A Web search would be a typical scenario in which filtering is appropriate.

- Lookup returns a single record that is a best match for some language request. Here, it is appropriate for a user to indicate very specific preferences and then to match that as best as possible, returning a more-generically-tagged resource if needed. (E.g. if user asks for en-GB is best, but en is certainly appropriate if no en-GB resource is available.) Software UI and Web sites are typical examples in which lookup is appropriate.

Considerations for macrolanguages in filtering: if a user specifies the macrolanguage, the result set may vary considerably in relevance if encompassed languages are not mutually intelligible; and if there is a predominant language, records in that may be relevant for all while records in other languages will be relevant only for certain users -- though there may be many fewer records in other languages. If a record is tagged for the macrolanguage, then it is likely in a predominant language that may be relevant to speakers of any of the encompassed languages (probably true for more developed languages such as Chinese, but not necessarily true for less developed language-nets, such as Ojibwa).

In lookup scenarios, the number of language varieties for a given resource is likely managed as a relatively small set; across a number of resources, the number of language varieties may still be limited, or there may be a very divers set of varieties. For instance, in a closed system such as UI resources for a software title, the number of varieties overall will be limited, and the number of varieties for a single resource may be much smaller. (E.g. there may be resources for en-GB, but there probably won't be resources for a lot of English dialects; and as for distinct languages, 100 would be exceptionally high.) In an open system such as the Web, a given site will likely support a limited set of varieties, though across the Web there are a lot more varieties represented. (E.g. there are going to be sites in many English dialects (though not necessarily tagged distinctly), and there are going to be sites in very many distinct languages.)

Considerations for macrolanguages in lookup: If a user specifies the macrolanguage, only resources tagged exactly the same way will match. If a user specifies an encompassed language, resources tagged for either the encompassed language or the macrolanguage will match. Since resources must be in some particular variety, it may not make sense to tag resources for the macrolanguage unless all the encompassed varieties really are mutual intelligible, or unless the resources are in a predominant language acceptable to speakers of all encompassed languages.

OK, now let's look at "Chinese" tags that would be available under different proposals. I won't give fully exhaustive lists (e.g. won't give tags involving every possible Region subtag), but will cover all the relevant cases with one or more representatives.

Here's the list -- since this will likely get wrapped and be hard to read, I'm also attaching a txt:

extlang          no-extlang    mixed                          meaning                              Notes
----------------------------------------------------------------------------------------------------------
zh               zh            zh                             'Chinese'                            (1)
zh-CN            zh-CN         zh-CN                          'Chinese as used in PRC'             (2)
zh-TW            zh-TW         zh-TW                          'Chinese as used in TW'              (3)
zh-HK            zh-HK         zh-HK                          'Chinese as used in Hong Kong'
zh-Hant          zh-Hant       zh-Hant                        'Chinese, Tdnl writing'              (4)
zh-Hans          zh-Hans       zh-Hans                        'Chinese, Simp writing'              (4)
zh-Hant-CN       zh-Hant-CN    zh-Hant-CN                     'Chinese, Tdnl as used in PRC'       (4)
zh-Hans-CN       zh-Hans-CN    zh-Hans-CN                     'Chinese, Simp as used in PRC'       (4)
zh-Hant-TW       zh-Hant-TW    zh-Hant-TW                     'Chinese, Tdnl as used in Taiwan'    (4)
zh-Hans-TW       zh-Hans-TW    zh-Hans-TW                     'Chinese, Simp as used in Taiwan'    (4)
zh-cmn           zh-cmn        zh-cmn                         'Mandarin'                           (4)(6)
                 cmn           cmn                            'Mandarin'                           (7)
zh-yue           zh-yue        zh-yue                         'Cantonese'                          (5)(6)
                 yue           yue                            'Cantonese'                          (7)
zh-cmn-Hant      zh-cmn-Hant   zh-cmn-Hant                    'Mandarin, Tdnl writing'             (4)(6)
                 cmn-Hant      cmn-Hant                       'Mandarin, Tdnl writing'             (7)
zh-cmn-Hans      zh-cmn-Hans   zh-cmn-Hans                    'Mandarin, Simp writing'             (4)(6)
                 cmn-Hans      cmn-Hans                       'Mandarin, Simp writing'             (7)
zh-yue-Hant      yue-Hant      zh-yue-Hant / yue-Hant         'Cantonese, Tdnl writing'            (7)
zh-yue-Hans      yue-Hans      zh-yue-Hans / yue-Hans         'Cantonese, Simp writing'            (7)
zh-cmn-CN        cmn-CN        zh-cmn-CN / cmn-CN             'Madarin as used in PRC'             (7)
zh-cmn-TW        cmn-TW        zh-cmn-TW / cmn-TW             'Mandarin as used in Taiwan'         (7)
zh-yue-CN        yue-CN        zh-yue-CN / yue-CN             'Cantonese as used in PRC'           (7)
zh-yue-HK        yue-HK        zh-yue-HK / yue-HK             'Cantonese as used in Hong Kong'     (7)
zh-cmn-Hant-CN   cmn-Hant-CN   zh-cmn-Hant-CN / cmn-Hant-CN   'Mandarin, Tdnl as used in PRC'      (7)
zh-cmn-Hans-CN   cmn-Hans-CN   zh-cmn-Hans-CN / cmn-Hans-CN   'Mandarin, Simp as used in PRC'      (7)
zh-cmn-Hant-TW   cmn-Hant-TW   zh-cmn-Hant-TW / cmn-Hant-TW   'Mandarin, Tdnl as used in Taiwan'   (7)
zh-cmn-Hans-TW   cmn-Hans-TW   zh-cmn-Hans-TW / cmn-Hans-TW   'Mandarin, Simp as used in Taiwan'   (7)
zh-yue-Hant-CN   yue-Hant-CN   zh-yue-Hant-CN / yue-Hant-CN   'Cantonese, Tdnl as used in PRC'     (7)
zh-yue-Hans-CN   yue-Hans-CN   zh-yue-Hans-CN / yue-Hans-CN   'Cantonese, Simp as used in PRC'     (7)
zh-yue-Hant-HK   yue-Hant-HK   zh-yue-Hant-HK / yue-Hant-HK   'Cantonese, Tdnl as used in HK'      (7)
zh-yue-Hans-HK   yue-Hans-HK   zh-yue-Hans-HK / yue-Hans-HK   'Cantonese, Simp as used in HK'      (7)

Notes:
1. Most (by far) existing content tagged "zh" is Mandarin. (Not
   clear how much content is tagged simply "zh" rather than
   "zh-CN", etc. or "zh-Hant"/"zh-Hans".)

2. "zh-CN" has often been used to declare content as "Chinese,
   Simp writing" (almost all, of course, being Mandarin), though
   the preferred tag for this, "zh-Hans", has been available
   since mid 2005.

3. "zh-TW" has often been used to declare content as "Chinese,
   Tdnl writing" (almost all, of course, being Mandarin), though
   the preferred tag for this, "zh-Hant", has been available
   since mid 2005.

4. Tag has been available since mid 2005.

5. Tag "zh-yue" has been available since 1999. (Also the case for
   "zh-gan" and "zh-wuu".)

6. Tag is grandfathered in current LSTR.

7. Not valid under RFC 4646.

Of course, some of those possibilities are currently more common than others,* and some are more useful than others. The challenge is to figure out which would be recommended under each proposal, and how to handle those that might not be generally recommended yet are currently in use and so likely to be encountered.

*(I wonder if Mark can get Web stats on all of these.)

So, let's consider each semantic and the tags available:

"Chinese":

The only tag option is "zh". Because "zh" is widely used and couldn't quickly go away (no matter how hard we might want), we will have to deal with it for some time.

It may be a reasonable choice to use "zh" to tag text content (see more below), but it is probably not a good choice for A/V media unless a system is closed and providing Mandarin content only.

Apart from maintaining current practice of using "zh" to tag Mandarin content, it is unclear how it would ever be useful to tag content as "Chinese". There may scenarios in which it would be useful for a user to request "Chinese" in filtering or lookup queries, though association between "zh" and Mandarin in existing usage limits feasibility of using "zh" to indicate a truly general "Chinese" request.

In filtering, users that ask for "zh" probably want Mandarin or would consider Mandarin acceptable, so content tagged "zh-cmn*" or "cmn*" is probably relevant, while content tagged "zh-yue"/"zh-gan"/etc. (or "yue"/"gan"/etc.) has a much lower likelihood of being relevant. In a no-extlang paradigm, "zh" records should probably be returned when "cmn" is requested, but not for requests of "yue"/"gan"/etc.; search engines would need to understand the relationship between "zh" and "cmn". Going into the future, increased importance of Cantonese (especially for A/V media) suggests that "zh" should be discouraged, though transition will be feasible only if requests for "zh" get appropriate results.

Wrt lookup, in a closed system there are no problems using "zh" if only Mandarin resources are provided; it may not be a problem to use for Mandarin if both Mandarin and Cantonese resources are provided -- because the system is closed -- but that's probably not a good choice. For Web sites, many may use "zh", and many browsers may send requests for "zh". In a no-extlang paradigm requests for "cmn" should match "zh", and "cmn" sites should match "zh" requests; the implication is that browsers should send both "cmn" and "zh", and "cmn" shouldn't be used for a site unless the server can match "zh".

"Chinese as used in PRC/Taiwan/etc."

The only tag options are "zh-CN"/"zh-TW"/etc. As noted, these have been used in the past to indicate (Mandarin) content in Simp or Tdnl writing. This is definitely to be discouraged, and hopefully is on the decline. Such tags continue to be used as locale IDs in software systems, and that would take some time to phase out (assuming that's feasible). Because of such usage in the wild we will need to deal with these tags for some time.

Apart from such legacy usage, these categories/tags probably are not that useful, and even if there were usefulness for a user to make a request in terms of the generic "Chinese" language but for a specific region, the association between "zh" and Mandarin limits feasibility of making such a general request.

In filtering, users that ask for "zh-??" probably want Mandarin or would consider Mandarin acceptable. Comments for "Chinese" above are applicable, modulo the added consideration regarding writing system. Users that ask for "zh-CN" or zh-SG probably want Simp or would find that acceptable, so records tagged "zh-Hans*", "zh-cmn-Hans*" (or "cmn-Hans*") would probably have high relevance. On the other hand, users that ask for "zh-TW", "zh-HK" or "zh-MO" probably want Tdnl or would find that acceptable, so records tagged "zh-Hant*", "zh-cmn-Hant*" (or "cmn-Hant*") would probably have high relevance. Similarly, users that ask for "zh-Hant"/"zh-cmn-Hant"/"cmn-Hant" would probably find records tagged "zh-TW" highly relevant; mutatis mutandi for "*-Hans" and "zh-CN".

Wrt lookup: these categories/tags should not be used. In closed systems, they could be used for Simp/Tdnl resources, or these tags used as locale IDs could map to more preferred tags -- "zh-Hant", or "zh-cmn-Hans"/"cmn-Hans", or "zh-Hant-TW", etc.

"Chinese, Tdnl/Simp writing"

The only tag options are "zh-Hant" and "zh-Hans". As noted, these have been available for a few years; they were registered as preferred alternatives to "zh-TW" and "zh-CN". It is unclear how much these have been used for content; they are used in some software platforms (e.g. .Net cultures). Of course, existing content would almost be certainly Mandarin. Also, they are obviously relevant for text but not for A/V.

Wrt filtering, see above for association between "zh" and "cmn", and between "*-Hant"/"*-Hans" and region subtags.

Wrt lookup: these may be useful in closed systems that support only Mandarin, and are definitely preferable to "zh-CN" and "zh-TW". If both Mandarin and Cantonese resources are supported, it is probably best not to use "zh" to represent Mandarin. Some Web sites or browsers may be using these for Mandarin; see comments under "Chinese" for some implications.

"Chinese, Tdnl/Simp as used in PRC/Taiwan/etc."
The tag options are the same for all three proposals: "zh-Hant-CN", "zh-Hans-CN", "zh-Hant-TW", etc. These have been available for a few years, though it is unclear how much they have been used (my guess: very, very little). If they are currently used, "zh" is probably assumed to represent Mandarin. Issues for filtering and lookup described above due to this association between "zh" and Mandarin apply here as well.

Since there is likely little existing usage and these perpetuate issues surrounding "zh" and Mandarin, I don't see any scenario in which it would make sense to use these categories/tags.

"Mandarin", "Cantonese", etc.

For Mandarin, the tag "zh-cmn" would be available under all proposals; the no-extlang and mixed proposals would also make "cmn" available. This category would be appropriate for A/V content. For text, written form should probably also be included (i.e. this is probably not a useful category for text scenarios). The tag "zh-cmn" is in use for A/V content (not sure how widespread already, but it is likely growing); it is probably not currently used for text content. Under the no-extlang and mixed proposals, one of "zh-cmn" and "cmn" would have to be recommended for A/V scenarios, and the other deprecated in general. While "cmn" would be canonical, it is "zh-cmn" that has prior usage.

Similar comments apply to "Cantonese", "zh-yue" and "yue" (and likewise for other languages -- though not all currently registered tags would become redundant in an extlang paradigm; e.g. "zh-hakka").

The particular case of "Mandarin" has the added complication of existing usage of "zh"; the relevant issues are discussed above. In general, the more explicit tags "zh-cmn" or "cmn" are preferable when "Mandarin" is intended, though applications need to take the existing use of "zh" into consideration.

Wrt filtering, the extlang/no-extlang decision affects results. For filtering as spec'd in RFC 4647 in an extlang paradigm, a request for the macrolanguage will return any records tagged for the macrolanguage or any encompassed language. Since a request for "zh" is (at least currently) likely intended to be a request for Mandarin, the result set would include any records in other encompassed languages -- generally, probably low relevance, but also probably low frequency. Reversing things, if "zh-cmn" is requested, it is probably appropriate to return records tagged "zh", though the currently-spec'd algorithm would not do that. (Records tagged "zh" also would not be returned if any other encompassed language is requested -- which is probably appropriate.) In a paradigm of no-extlang 4646bis, "zh" would not match for records tagged for any encompassed language by the currently-spec'd algorithm. The filtering algorithm would have to be modified or tailored to get a match between "zh" and "cmn".

Wrt lookup also, the extlang/no-extlang decision affects results. With extlang, a request for any encompassed language would match resources tagged "zh". This could potentially lead to missing preferred matches. Thus, suppose a user indicated a preference for "zh-yue" then "zh-gan", and after that "zh-cmn" or "zh"; and supposed the available resources included resources tagged "zh-gan" and "zh": the match would be on "zh", not "zh-gan". In a no-extlang 4646bis paradigm, none of the tags for encompassed languages match "zh". This would avoid the problem just described, but it also means that "cmn" would not match "zh" -- unless the algorithm were modified or tailored.

"Mandarin, Tdnl/Simp writing"/"Cantonese, Tdnl/Simp writing"/etc.

As above, the currently-available tags for Mandarin would remain available under all proposals: "zh-cmn-Hant" and "zh-cmn-Hans"; the no-extlang and mixed proposals would add "cmn-Hant" and "cmn-Hans". These categories are appropriate for text but not A/V scenarios. Of course, in text scenarios "zh-Hant"/"zh-Hans" have been used to represent these categories (due to the association between "zh" and Mandarin). Tags using "cmn" should be recommended in general, but applications need to take the legacy tags into consideration. Under the extlang and mixed proposals, a choice to recommend one of "zh-cmn-Han?" or "cmn-Han?" and to deprecate the other would be needed.

Some, but not all, of these comments apply to Cantonese, "zh-yue-Han?" and "yue-Han?". No tags were registered in the past, and so no issues with synonyms will arise: either "zh-yue-Han?" will become available, or "yue-Han?" will become available, but not both. (These comments apply for the other Chinese languages as well.)

In a no-extlang or mixed paradigm, there would be alternate tags for a single category (e.g., "zh-cmn-Hans" and "cmn-Hans"); for each such pair, a choice would need to be made to recommend one and deprecate the other. In the no-extlang paradigm, such a pair would exist only for Mandarin; in that case, consistency would argue for recommending "cmn-Han?" rather than "zh-cmn-Han?". In the mixed paradigm, however, such pairs would exist for every language.

In filtering, users that ask for "zh-Han?" probably want Mandarin or would consider Mandarin acceptable. In this case, the filtering algorithm spec'd in RFC 4647 would not provide a match for content tagged either as "zh-cmn-Han?" or as "cmn-Han?"; that algorithm would need to be tailored or modified. If users request "zh-cmn", the current algorithm would match content tagged "zh-cmn-Han?" but not if tagged "cmn-Han?". (Mutatis mutandi for "zh-yue-Han?", etc.) If no-extlang tags are used in the request, e.g. "cmn", then only no-extlang tags ("cmn", "cmn-CN", "cmn-Hans", etc.) would be matched unless the algorithm were tailored or modified.

In lookup, a request for "zh-cmn-Han?" would match content tagged "zh-cmn" or "zh"; but there would not be a match in the more useful case of "zh-Han?" unless the current algorithm were tailored or modified. Also requests involving other languages, such as "zh-yue-Han?" could produce a match on resources tagged "zh", possibly missing a preferred match -- see the scenario discussed in previous section. (Since these are text scenarios, there is potential consolation in that "zh" resources are likely to be acceptable, provided the text follows the user's Tdnl/Simp preference.) In a no-extlang paradigm, none of the tags for encompassed languages would match "zh", and so this problem is avoided.

"Mandarin as used in PRC/TW/etc." / "Cantonese as used in ..." / etc.

No tags for these categories were ever registered, and so the available tags would depend on the choice between the extlang, no-extlang and mixed proposals: e.g., "zh-cmn-CN" vs. "cmn-CN" vs. "zh-cmn-CN" / "cmn-CN". In the mixed paradigm, there would be two tags available for each semantic, and a choice would need to be made to recommend one and deprecate the other.

These categories would be useful for A/V scenarios, but not text scenarios.

In filtering, a user requesting "zh" may perhaps want Mandarin, though since "zh" is not a great choice for Mandarin for A/V scenarios in general, it probably would not be used that way. In this case, "zh" could perhaps be used for a generic request, and in an extlang paradigm the filtering algorithm spec'd in RFC 4647 would treat that generically, matching any tag "zh*". It's not really clear when users would want to make such a generic request for A/V content, though: users are more likely to specify which particular language they want. A request for "zh-cmn" would match records tagged "zh-cmn-??" by the spec'd filtering algorithm, but not records tagged "cmn-??". Likewise, a request for "cmn" would match recoreds tagged "cmn-??", but not records tagged "zh-cmn-??" (a potential issue only for the mixed paradigm since there are no currently registered tags of the form "zh-cmn-??").

In lookup, a request for "zh-cmn-??" would match resources tagged "zh-cmn" or "zh"; a request for "cmn-??" would match resources tagged "cmn". The problem mentioned in the previous two sections of scenarios in which a preferred match could be missed exists here in an extlang paradigm but not in a no-extlang paradigm.

"Mandarin, Tdnl/Simp as used in PRC/TW/etc." / "Cantonese, Tdnl/Simp as..." / etc.

No tags for these categories were ever registered, and so the available tags would depend on the choice between the extlang, no-extlang and mixed proposals: e.g., "zh-cmn-Hans-CN" vs. "cmn-Hans-CN" vs. "zh-cmn-Hans-CN" / "cmn-Hans-CN". In the mixed paradigm, there would be two tags available for each semantic, and a choice would need to be made to recommend one and deprecate the other.

These categories would be useful for text scenarios, but not A/V scenarios.

In filtering, a user requesting "zh" probably wants Mandarin or would find Mandarin acceptable. In an extlang paradigm, this request would match "zh-cmn-Han?-??". It would also match "zh-yue-Han?-??" (or "zh-gan-Han?-??", etc.), which would likely be of low relevance, but also of low frequency. In a no-extlang paradigm, "zh" would not match the tags for these categories (e.g. "cmn-Hans-CN") unless the spec'd algorithm were tailored or modified.

If a user requests "zh-cmn" (or "zh-yue"), the categories from this set they would get matches on would be relevant in terms of language (though maybe not written form); this would happen using the RFC 4647 algorithm as is in an extlang paradigm, but tailoring or modification would be needed in a no-extlang paradigm.

If a user requests "zh-Han?", they probably want Mandarin or would find Mandarin acceptable. The spec'd algorithm would not produces matches for any of the potential tags for these categories in under any of the proposals unless the algorithm were tailored or modified.

In lookup, a request for "zh-cmn-Han?-??" would match resources tagged "zh-cmn-Han?" or "zh-cmn" or "zh"; a request for "cmn-Han?-??" would match resources tagged "cmn-Han?" or "cmn". The problem mentioned in the previous sections of scenarios in which a preferred match could be missed exists here in an extlang paradigm but not in a no-extlang paradigm.

All this needs further refinement and processing to get to some useful conclusions, but I've gone on long enough for one reading. I'll stop here for now.

Peter
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Mon, 2 Jun 2008 00:33:19 -0400

Peter Constable scripsit:

> - Filtering returns a number of records from a (typically sizeable)
> corpus. Here, a record tagged with greater specificity than was
> indicated in the query can always be returned (e.g., if user asks for
> en, then en-GB is certainly valid); relevance of record that are tagged
> more generically (e.g., en when user asked specifically for en-GB)
> is very much in question. A Web search would be a typical scenario in
> which filtering is appropriate.

Distinguo.

As we have established on this list, filtering is also used when
retrieving from multilingual web sites.  In this context, the corpus is
typically small.  (If filtering produces zero results, some web servers
do lookup instead.)

-- 
John Cowan    http://ccil.org/~cowan  [email-masked]
The Penguin shall hunt and devour all that is crufty, gnarly and
bogacious; all code which wriggles like spaghetti, or is infested with
blighting creatures, or is bound by grave and perilous Licences shall it
capture.  And in capturing shall it replicate, and in replicating shall
it document, and in documentation shall it bring freedom, serenity and
most cool froodiness to the earth and all who code therein.  --Gospel of Tux
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Broome, Karen" <[email-masked]>
Date: Mon, 2 Jun 2008 22:52:04 -0700

Mark,

I think what bothers me more than extlang or no-extlang is the narrowing I perceive in "zh" that seems to come with the no-extlang strategy. The historic registrations of other tags (such as zh-yue and zh-min-nan) clearly shows that "zh" has always included Cantonese and other Asian languages. Yes, I know that a very overwhelming number of these documents are in written Mandarin. But we have an overall rule that indicates the valid tag is the shortest tag that represents the language. It seems like you are indicating that "zh" is the shortest tag for Mandarin, but not Cantonese after RFC 4646bis. How can this be true without there being a narrowing?

While your spiders may be able to give us some kind of an idea of the languages used on the web, xml:lang and ISO tags have a lot of other uses in B2B applications and asset databases. I'm not saying that there is likely more content tagged "zh-yue" when compared to other "Chinese" content, but the percentages for the a/v assets I look at are not statistically insignificant.

We have promised backwards compatibility, and the group has repeatedly said that narrowing an ISO semantic is not allowed. Removing extlang will force me to make some systems tweaks, nothing insurmountable, but I am concerned about the impact on other people who aren't represented in this small group. I don't have a whole lot of insight into some of the other languages that use extlang. I'm wondering if the users of those languages have similar issues where there may be mutually intelligible written forms that are not mutually intelligible when spoken (or vice versa).

Mark wrote:

>Why? While 639-3 now specifies clearly that "de" means (for example)
>just Standard German, while "zh" means Any Chinese, this clarity of
>specification was not present earlier. The code "zh" has been used
>in the past for Mandarin, overwhelmingly so; not just 99% or 99.9%,
>but many 9's.

I agree the "zh" code has always been vague, but treating it as synonymous with Mandarin contradicts previous registrations with IANA. It is the best current tag for Mandarin, but its use for a significant number of Mandarin documents does not prove or disprove whether its historical semantic includes Cantonese and other Chinese languages.

>As you said, the tendency was to use illegal (or
>private use codes) for non-Mandarin content.

A couple of you have been misreading me on this. The overwhelming tendency was to use made-up codes for ANY audio Chinese. I see a lot of "MAN" for Mandarin, which is, of course, a really bad idea. It's typical to see the written variants of Chinese classified as Traditional or Simplified with zh-[etc.], but audio will use bad codes for spoken Mandarin AND Cantonese.

>       Note also that really the whole premise of extlang is that
>'zh' continues to normally map to Mandarin.

But not Cantonese?

>My real point is that if a query for 'ar' really means "give me any
>kind of Arabic", then a query for 'ar' would be almost meaningless,
>since it could return any of a number of mutually incomprehensible
>alternatives.

For anyone other than a giant international search engine, I think fallbacks like these are a business decision. Appropriate fallbacks for a corporate or other content sites likely depend on the content the publisher has to offer and the intended geographical reach of the site. I'm guessing that even rather  forward-thinking organizations will not have content represented in more than 50 languages. Academic archives will certainly have more, but will they use the elaborate fallback logic or a direct lookup?

>Although 639-3 now defines it to be "any Arabic", in
>practice what users will expect to get back is Standard Arabic, and
>they would be unpleasantly surprised to get back other varieties.

Fallback only occurs when the user's language of choice is not available, right?  What would be a better fallback? Is it better not to serve up alternate content at all?

>Note that this is not actually RFC4646 conformant: zh-cmn-TW is not
>valid.

Of course, when this was assigned this was presumed to be future-valid. It's when I started thinking about just using "zh-TW" for this under RFC 4646bis that I began to get concerned.

>Mark: Your list is already ambiguous for columns 1 and 2; you are using
>"yue" for two different things (written and spoken). The only change
>it really makes is that you don't have a term for "any chinese".

The list is not ambiguous as far as I'm concerned. I don't require unique codes for spoken and written forms except when a script tag distinguishes two similar pieces of content. I refuse to use a script tag to identify audio language, but I don't currently have a need to distinguish forms of written Cantonese so "zh-yue" is fine for both and I don't see any potential for misinterpretation. I don't see my competitors making distinctions here either.

>The only purpose I have heard for extlang *is* for fallback; that's
>why the document goes into (painful) depth on that topic.

The arguments I've put forward have had little to do with fallback and more to do with promoting accurate tagging in the future. You just don't seem to think they are valid arguments. So I think your previous statement above is inaccurate.

For
>identification alone, "zh" and "zh-cmn" really mean just the same
>thing. It is only in the context of matching (filtering and lookup)
>that they differ in semantics *because of their behavior*: where
>"cmn" means simply Cantonese, "zh-cmn" effectively means "Cantonese
>but fallback to any Chinese".

Think you mistyped here so not sure if I agree or disagree.

I'd like to see a solution where developers won't need to rely on too much conditional linguistic logic; there are clearly preferred tags; the rules are clear; and ideally, there's no cherry picking. I think cherry picking makes rules harder to explain.

Regards,

Karen Broome

---

From: Peter Constable <[email-masked]>
Date: Mon, 2 Jun 2008 23:26:41 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Broome, Karen

> The historic registrations of other tags (such as zh-yue and
> zh-min-nan) clearly shows that "zh" has always included Cantonese and
> other Asian languages...
> It seems like you are indicating that "zh" is the shortest
> tag for Mandarin, but not Cantonese...

Do all keep in mind comments I made a week or two ago about historical development leading up to macrolanguage:

- the macrolanguage concept means that some treat it as one language while some treat it as many

- the alpha-2 code in ISO 639-1 was initially created by terminologists, for whom developed language varieties would have been of interest, which for "Chinese" would have been Mandarin

- the alpha-3 code in ISO 639-2 was created (in part) by librarians, for whom "Chinese" was explicitly meant to cover both Mandarin and Cantonese

The different points of view expressed by Karen and by Mark are, I think, not surprising or coincidental: the need to be clear that "zh" includes Cantonese as well as Mandarin is appropriate and is consistent with existing use exemplified by library practice; and the utility, in practice, of recognizing a particular association between "zh" and Mandarin in some applications (though not necessarily all) is also appropriate and stems from the same factors that were involved in the practice of Terminologists: for a lot of business application to date, the only Chinese language with particular relevance for text scenarios has been Mandarin.

There are two realities; both need to be recognized. I see that happening by saying something to the effect of, "'zh' _means_ "Chinese"; in some applications, especially text scenarios, it may be appropriate to treat use of 'zh' in existing implementations as being particularly associated with Mandarin".

Peter

Peter

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 3 Jun 2008 11:24:48 -0700

Hi -

As a technical contributor...

> From: "Broome, Karen" <[email-masked]>
> To: "Mark Davis" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Monday, June 02, 2008 10:52 PM
> Subject: Re: [Ltru] Consensus call: extlang
...
> I think what bothers me more than extlang or no-extlang is the narrowing
> I perceive in "zh" that seems to come with the no-extlang strategy.
> The historic registrations of other tags (such as zh-yue and zh-min-nan)
> clearly shows that "zh" has always included Cantonese and other Asian
> languages. Yes, I know that a very overwhelming number of these documents
> are in written Mandarin. But we have an overall rule that indicates the valid tag
> is the shortest tag that represents the language. It seems like you are indicating
> that "zh" is the shortest tag for Mandarin, but not Cantonese after RFC 4646bis.
> How can this be true without there being a narrowing?

I share this concern.  It must be crystal clear that 'zh' does NOT mean Mandarin,
but rather any kind of Chinese.  Of course, there are cases where one might
deliberately be imprecise in tagging, but we should not be so sloppy in our
own definitions to equate the label for the superset with the subset, just as
we must avoid falsely limiting 'de' to "Standard German" or 'ar' to "Standard
Arabic".  (I recognize that this concern is largely independent of the
syntactic decision to use or not use extlang.)

...
> I don't have a whole lot of insight into some of the other languages that use
> extlang. I'm wondering if the users of those languages have similar issues
> where there may be mutually intelligible written forms that are not mutually
> intelligible when spoken (or vice versa).

We need to recognize that there are cases where having
an extlang might make sense, but none is available, so
the extlang approach doesn't help.  Some examples:
  - spoken Serbian and Croatian
  - Romanian and Moldovan.  

Randy

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 3 Jun 2008 11:32:08 -0700

> but rather any kind of Chinese.

+1

> Of course, there are cases where
> one might
> deliberately be imprecise in tagging, but we should not be so

It is always permissible to tag Mandarin as 'zh'. It is also permissible, within one's own application, to require that people who want something other than Mandarin to use a different tag than 'zh'. For example they can be required to use the tag (choose one: yue | zh-yue).

What is not permissible is to dismiss someone else's Cantonese content tagged as 'zh' as wrong.

A corollary example might help:

It is always permissible to tag US English as 'en'. It is also permissible, within one's own application, to require that people who want something other than US English to us a different tag from 'en'. For example, they can be required to use the tag 'en-GB'.

What is not permissible is to dismiss someone else's UK English content tagged as 'en' as wrong.

Addison

---

From: "Broome, Karen" <[email-masked]>
Date: Tue, 3 Jun 2008 11:42:58 -0700

If it wasn't already clear from my previous e-mail, I absolutely agree with this. (And Addison probably said it better than I did. Thanks!)

Karen

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf
>Of Phillips, Addison
>Sent: Tuesday, June 03, 2008 11:32 AM
>To: Randy Presuhn; LTRU Working Group
>It is always permissible to tag Mandarin as 'zh'. It is also
>permissible, within one's own application, to require that people
>who want something other than Mandarin to use a different tag than
>'zh'. For example they can be required to use the tag (choose one:
>yue | zh-yue).
>
>It is always permissible to tag US English as 'en'. It is also
>permissible, within one's own application, to require that people
>who want something other than US English to us a different tag from
>'en'. For example, they can be required to use the tag 'en-GB'.
>

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 3 Jun 2008 11:58:27 -0700

hi -

> From: "Phillips, Addison" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>; "LTRU Working Group" <[email-masked]>
> Subject: RE: [Ltru] Consensus call: extlang
...
> It is always permissible to tag US English as 'en'.
> It is also permissible, within one's own application,
> to require that people who want something other than
> US English to us a different tag from 'en'.
> For example, they can be required to use the tag 'en-GB'.
> 
> What is not permissible is to dismiss someone else's
> UK English content tagged as 'en' as wrong.
...

It is also permissible, within one's own application,
to tag UK English as 'en' and require US English to be
tagged 'en-US'.  But "permissible" doesn't mean "advisable".
I would argue that if an application knows the difference
and cares about the difference, that it would be much better
to use both 'en-GB' and 'en-US', and reserve naked 'en' for
the cases where there's no value in making the distinction
(e.g., it can't tell or doesn't care).

Randy

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 03 Jun 2008 21:26:13 +0200

Phillips, Addison 2008-06-03 20.32:

>> I share this concern.  It must be crystal clear that 'zh'
>> does NOT mean Mandarin, but rather any kind of Chinese.
> 
> +1
> 
>> Of course, there are cases where one might deliberately be
>> imprecise in tagging, but we should not be so
> 

I hear this a lot:

	«'zh' is for all Chinese. But it is permissible to use 'zh' for 
Mandarin.»

If you really want to emphasize that 'zh' is for all, then you 
must start saing that it is permissible to use 'zh' for Cantonese 
- because we allready know that it is allowed for Mandarine.

> permissible, within one's own application, to require that
> people who want something other than Mandarin to use a
> different tag than 'zh'. For example they can be required to
> use the tag (choose one: yue | zh-yue).

Then it is also permissible to require Mandarin users to use 'cmn'.

If we, as Doug said, "believe" in the macrolangauge thing, that it 
describes a real thing, then we should emphasize that it is not at 
all advicable to say that 'zh' means Cantonese or Mandarin in any 
context.

What we *should* say is that the Macrolanguage code should not be 
locked to any particular variety. Application developers should 
strive to keep it as a "free code" - a wild card that can be used 
by any of the encompassed languages.

By wild card I mean this: when I use Mac OS X with Nynorsk as 
preferred langgae, then 'no' should lean towards meaning 'nn'. 
That is: If a text editor is tagged as having 'no' interface, 
then, even if this in reality means that it is Bokmål, still, for 
deciding which spell checking dictionary to use as default (often 
this is synced with the GUI langauge), the application should 
default to 'nn'/Nynorsk. But if I run Mac OS X with Bokmål as 
preferred langauge, then 'no' should default to be interpreted as 
an alias for 'nb' - thus bokmål dictionaries should be picked.

The users will then have to accept that they cannot expect to have 
full control over the interface language of applications tagged as 
'no'. And this is not difficult. Users accept this easily. As long 
as you do not use 'no' but translate it in the user interface as 
'Nynorsk' or 'Bokmål'. Same for zh/Chinese, if it in some language 
is translated as 'Mandarin', then you will get user reactions if 
you use it for any thing else but Mandarin. Hence, such use should 
be avoided.

> permissible, within one's own application, to require that
> people who want something other than US English to us a
> different tag from 'en'. For example, they can be required to
> use the tag 'en-GB'.
> 

Let me reuse your example:

  In Mac OS X I can select UK English as preferred language. 
Unless 'en_UK' localisations are available, then the 'en' tagged 
localisations with 'en-US' content will be served. However, there 
will be one important difference: UK dictioanaries will be 
preferred over US ones. And and certain other things will shift to 
its British values.

Just like the "color" of the "en" tag shifts based on whether the 
user prefers en-US, en-UK, en-Ca etc, the same way the 
Macrolanguage tag should shift according to which variety of the 
Chinese/Norwegian/Arabic language one is preferring.

Btw, since I have been critical about Apple, I think they do the 
right thing when they use 'en' for US English *while still* 
translating its meaning as "English" and not as "US English". It 
is *this* which is permissible. And if this is what you mean, 
Addision, then I agree with you. But if you mean that one can use 
"en" but put a "US English" on it as a label, then I disagree.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 3 Jun 2008 21:37:34 +0200

Phillips, Addison wrote:
> Cantonese content tagged as 'zh' as wrong.
> 
> It is always permissible to tag US English as 'en'. It is 
> also permissible, within one's own application, to require 
> that people who want something other than US English to us a 

You're comparing an apple/pear situation with a peach/nectarine
situation, if you allow me the analogy. There are differences
between ("standard") en-US and ("standard") en-GB, sure. But
it is (most often) VERY small compared to differences between
languages covered by a macrolanguage code. The latter varies
in degree of difference, but for zh, someone (who I think knows
this) compared cmn/yue differences as en par with fr/es
differences, in addition pointing out that the difference was
greater than between Scandinavian languages.

I'm sure you would not want a macrolanguage code for fr/es,
even if there was a way to get that (like way back when...),
nor recommend using a collection code for Gallo-Iberian
for language tagging.

	/kent k

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 3 Jun 2008 12:43:49 -0700

> (e.g., it can't tell or doesn't care).

I don't disagree. However...

The classic software fallback problem (sorry Karen) is that when someone requests "en-AQ" and you don't have any Antarctic English resources, one falls back on some content, which is typically labeled (in software resources) 'en'. This 'en' content can be one variety or another, but not both. In English this isn't always a problem. The foregoing sentence, for example, is good as both "en-US" and "en-GB".

Things are less good when there are disparities and no generic choice exists. Some years ago I work on a project that including localization of a product into Simplified Chinese--but only Simplified, not Traditional. That product also sold in Traditional Chinese speaking markets. Installing 'zh' labeled resources mean that ALL Chinese-locale configured systems might have displayed Simplified Chinese, so I had to implement certain checks to ensure that Traditional Chinese markets didn't see the Simplified (they saw the default resources instead).

Yes, the resources could have be labeled more specifically than 'zh'. But then I would run the risk of having a user with an unfamiliar locale configuration not seeing the Chinese resources they paid for. So the "permissible" and "advisable" bar may be set at different heights under different circumstances.

Addison

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 3 Jun 2008 21:53:22 +0200

Broome, Karen wrote:

> We have promised backwards compatibility,

Backwards compatibility w.r.t. zh-<something> is provided by
that the grandfathered tags are kept in LSR, even if deprecated
with another preferred value. So I consider this a non-argument.

	/kent k

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 3 Jun 2008 12:55:18 -0700

> languages covered by a macrolanguage code.

Yes. Actually, see my just-posted message. I know this is the case, but I was attempting to draw an illustrative analogy without getting carried away.

> The latter varies

There are many flavors of difference. How well-received those differences are depends on the specific case. In the 'en' case, one might be satisfied pretty well with the US English information, regardless of which flavo(u)r one prefers. In the 'zh' case, perhaps this is less the case. It doesn't invalidate the problem.

(laughing) For example, when I typed the above paragraph, my spell checker "fixed" 'flavour' to 'flavor' automagically. I had to fix it to make it stick (before deciding to include the parens). If I had requested UK English, I might have been peeved at that.

> nor recommend using a collection code for Gallo-Iberian

No, but, again, that's not my point. My point is that one can use less-specific tags consciously and deliberately to create a specific effect.

Addison

---

From: "Broome, Karen" <[email-masked]>
Date: Tue, 3 Jun 2008 13:10:39 -0700

What about the backwards compatibility of just "zh" used for Cantonese or Taiwanese? Sure, it will be valid, but if companies develop algorithms assuming that zh=Mandarin because we've advised them that this is a best practice even if that's not exactly what ISO says, that's not so backward compatible. And "zh-yue" becomes a contradiction in terms.

Karen

>Of Kent Karlsson
>Sent: Tuesday, June 03, 2008 12:53 PM
>To: 'LTRU Working Group'

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 03 Jun 2008 22:16:42 +0200

> tagged 'en-US'.  But "permissible" doesn't mean "advisable".

Agree.

> (e.g., it can't tell or doesn't care).

Almost agree. But I do not think that the en/en-US/en-UK example 
covers the Macrolanguage situation 100%.

Where I agree with you is that the 'en' should be used "for the 
cases where there's no value in making the disctinction". 
However, when a system is offered only in one variant of English, 
then were is the value in making any distinction?

I think this is a parallell to macrolanguages. If you only plan on 
supporting the macrolanguage - whichever encompassed language you 
pick to represent it - there is little value in saying "Mandarin" 
or "Bokmål". Just say "Chinese" and "Norwegian".

Then, when the OS vendor switches to support more of the 
encompassed varieties -- and if  translations for the new, 
encompassed languages are not immmediately available for all of 
the languages -- then the  macrolanguage code could be used as a 
tag for applications which only exist in the dominant language, 
thus (hopefully) making them available for all the encompassed 
langauges.

(For example, for Norwegian Mac OS X, Apple could have offered the 
basic OS X GUI in 'nn' tagged Nynorsk and 'nb' tagged Bokmål. Two 
NOrwegian GUIs for the two usergroups. At the same time, they 
could have kept their old base of 'no' tagged Bokmål applications 
in order to make them available for both Nynorsk and Bokmål users.)

So, where I differ from many  of you -- it seems -- is that I see 
a **permanent** value in the Macrolanguage tag.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 03 Jun 2008 22:22:13 +0200

Phillips, Addison 2008-06-03 21.43:

>> It is also permissible, within one's own application, to tag
>> UK English as 'en' and require US English to be tagged
>> 'en-US'.  But "permissible" doesn't mean "advisable". I would
>> argue that if an application knows the difference and cares
>> about the difference, that it would be much better to use
>> both 'en-GB' and 'en-US', and reserve naked 'en' for the
>> cases where there's no value in making the distinction (e.g.,
>> it can't tell or doesn't care).
>> 
> 

   [...]

> Yes, the resources could have be labeled more specifically than
> 'zh'. But then I would run the risk of having a user with an
> unfamiliar locale configuration not seeing the Chinese
> resources they paid for. [...]

A well known issue ...

---

From: Peter Constable <[email-masked]>
Date: Tue, 3 Jun 2008 19:54:34 -0700

> Leif Halvard Silli

> by any of the encompassed languages...

> 'no'... Same for zh/Chinese...

OK, so a user selects "Cantonese" as their preference, and some apps have Cantonese UI but other apps have Mandarin UI? And then when they search on the Web all the results they get are in Mandarin? And they'd find that acceptable?

Is that what you're suggesting?

(I'm not making any judgments; I just want to understand.)

Peter

---

From: Peter Constable <[email-masked]>
Date: Tue, 3 Jun 2008 20:08:30 -0700

> What about the backwards compatibility of just "zh" used for Cantonese
> or Taiwanese? Sure, it will be valid, but if companies develop
> algorithms assuming that zh=Mandarin because we've advised them that
> this is a best practice even if that's not exactly what ISO says,
> that's not so backward compatible. And "zh-yue" becomes a contradiction
> in terms.

Is anyone suggesting that we recommend as best practice that algorithms assume zh = Mandarin?

The closest I have come is to say people should be aware that zh has very often been used in relation to Mandarin resources/content, and that they should evaluate what that may imply for their scenarios.

Peter

---

From: Martin Duerst <[email-masked]>
Date: Wed, 04 Jun 2008 15:27:03 +0900

At 11:54 08/06/04, Peter Constable wrote:

>OK, so a user selects "Cantonese" as their preference, and some apps have 
>Cantonese UI but other apps have Mandarin UI? And then when they search on 
>the Web all the results they get are in Mandarin? And they'd find that 
>acceptable?

"acceptable" is a stretchable term, in most cases. Some Cantonese
users may prefer a Mandarin UI to a UI in French or English or what
not. Given what we have heard on this list about written Cantonese
and Mandarin, it may even be that most Cantonese users would have
such a preference. For search results again, they may prefer
something relevant in Mandarin to nothing/irrelevant in Cantonese,
even though of course their top preference is relevant stuff in
Cantonese.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Leif Halvard Silli <[email-masked]>
Date: Wed, 04 Jun 2008 14:15:25 +0200

Peter Constable 2008-06-04 04.54:

>> [...] On Behalf Of Leif Halvard Silli
>> What we *should* say is that the Macrolanguage code should
>> not be locked to any particular variety. Application
>> developers should strive to keep it as a "free code" - a wild
>> card that can be used by any of the encompassed languages...
> 
>> The users will then have to accept that they cannot expect to
>> have full control over the interface language of applications
>> tagged as 'no'... Same for zh/Chinese...
> 
> OK, so a user selects "Cantonese" as their preference, and some
> apps have Cantonese UI but other apps have Mandarin UI? And

So far, so good.

Yes, this is what I suggest. Those apps which include a 
localisation tagged as "zh" will offer the "zh"-tagged 
localization to the users who are using the Cantonese Window UI 
when and if a Cantonese UI for that particular application is 
lacking. This should not prevent that the same application can 
include a 'zh-cmn' resource to be used in the Mandarin UI, when 
and if needed.

This is similiar to the en/en-UK/en-US analogy. More spesific will 
overrule less spesific.

> then when they search on the Web all the results they get are
> in Mandarin? And they'd find that acceptable?

No, this does not follow. Here the en/en-UK/en-US analogy breaks.

Taking Mac OS X as example, I can allready today place Nynorsk on 
top of its "language list", despite that Mac OS X is lacking any 
Nynorsk interface (except that id does offer daynames in nynorsk 
spelling). Doing so will make Safari - and other application which 
takes its accept-language from the Mac OS X language preference 
list - send the message that it accepts "nn".

Thus, in case of a Cantonese UI preference, an application should 
definitely tell the Web that it accepts Cantonese, whether the UI 
of the app in question is tagged with 'zh' or 'zh-yue'.

> (I'm not making any judgments; I just want to understand.)

A UI tagged as "en-any-variant" should default to the 
accept-language "en". While a "zh-yue" application, even if it may 
accept the macrolanguage as "backup language", should tell the Web 
that it accepts the encompassed language itself. Thus in your 
example, it should tell that it accepts 'zh-yue'/Cantonese. 
(Whether it should *also* tell the web that it also accepts 'zh', 
is technically another question.)
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 4 Jun 2008 17:26:37 +0200

On Tue, Jun 03, 2008 at 11:24:48AM -0700,
 Randy Presuhn <[email-masked]> wrote 
 a message of 46 lines which said:

> just as we must avoid falsely limiting 'de' to "Standard German" or
> 'ar' to "Standard Arabic".

Hold on. For Standard Arabic, the "proper" tag is "arb". But for
Standard German, what could be used instead of "de" if we change "de"
to mean "Any sort of German"?

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 5 Jun 2008 11:02:55 -0700

Hi -

> From: "Stephane Bortzmeyer" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>
> Sent: Wednesday, June 04, 2008 8:26 AM
> Subject: Standard german (Was: Consensus call: extlang
>

An appropriately registered subtag following de, if someone has a need
to make the distinction.  (I don't see it as a change at all.  Perhaps it's
because I used to teach German, but for me "Standard German" is a
narrowly defined and somewhat artificial beast, not to be confused
with the welter of mutually intelligible varieties of German (i.e., 'de')
spoken all over the world.)  To misconstrue 'de' narrowly would, in my
opinion, be like understanding 'fr' as limited to Parisian French.

Randy

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 5 Jun 2008 11:32:15 -0700

> > But for
> > Standard German, what could be used instead of "de" if we change
> "de" to mean "Any sort of German"?
>

I agree. I think all this discussion of "standard" language varieties misses the point. There is a certain lack of specificity to language codes in ISO 639 just as there is a deliberate lack of specificity in most language tag choices.

Often the most useful tag for German is "de". In some cases one might need a "de-CH" or "de-DE". Rarely will a "de-qxx-Latn-CH-1984-x-duerst-martin-2008-06-04-mittwoch" be of any use whatsoever, except as an illustration of tag folly. Ultimately, this thread is really about where "dialect" ends and "language" begins--a distinction that is necessarily indistinct and judgmental. Sometimes the "standard" form of a language forms the peak in the bell-curve of a language's usage and distribution and sometimes it won't. So what?

What's problematic here is the idea that someone might come along and say that any given variant flavor of German is "not German" because it isn't the standard form. At the same time, just about any language can be divided further and further. The question is the utility of a given division.

Addison

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 5 Jun 2008 11:44:16 -0700

Hi -

> Sent: Thursday, June 05, 2008 11:32 AM
> Subject: RE: [Ltru] Standard german (Was: Consensus call: extlang
...
> What's problematic here is the idea that someone might come
> along and say that any given variant flavor of German is
> "not German" because it isn't the standard form. At the same
> time, just about any language can be divided further and further.
> The question is the utility of a given division.

Agreed.

Finishing the thought: Once there's agreement that it's useful
to make a distinction, the next question is whether it's more
useful to regard the newly recognized variant as being a variant
or as being a distinct language, and whether this distinction should
be reflected in the formation of the language tag.

Randy

---

From: Peter Constable <[email-masked]>
Date: Thu, 5 Jun 2008 13:17:29 -0700

> Randy Presuhn

> with the welter of mutually intelligible varieties of German (i.e.,
> 'de') spoken all over the world.)

For which, variant subtags sound just right.

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 5 Jun 2008 15:40:53 -0700

Agreed.

-- 
Mark