Subject: Re: [Ltru] I'm really confused by chinese in 3066bis

From: "Doug Ewell" <[email-masked]>
Date: Sat, 3 May 2008 16:11:10 -0600

<Karen underscore Broome at spe dot sony dot com> wrote:

>> However, because of the de facto predominance of Mandarin, 'zh' has
>> long been used to mean Mandarin only.
>
> Disagree. It was simply the most specific tag one could use for the
> Mandarin at the time. That doesn't mean the tag ever meant "Mandarin"
> to anyone. We might as well say zh-TW means Traditional Mandarin as
> this tag has been used more often than zh-Hant.

I agree with Karen here.  As I said during the extlang debates, I don't
think anyone ever, EVER used "zh" with the intention of excluding
non-Mandarin.  Rather, by necessity they tagged all Chinese content as
"zh", and it so happened that most of that content was Mandarin.  This
is a subtle but important difference.  Saying that people meant
"Mandarin, not Cantonese" when they used "zh" is revisionist.

--
Doug Ewell  *  Arvada, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 3 May 2008 19:08:32 -0700

I'm not the one being revisionist; the people quoting me are. I didn't say
"means", I said "has long been used to mean". There is a difference. I
didn't say "are synonyms", I said "are best treated as synonyms" (and with
qualifications). There's a difference. If I say that for the purpose of
juggling, I treat apples and oranges as the same, it doesn't mean that I am
somehow saying that they mean the same thing. It means for particular
purposes, I think they should be handled the same way. Moreover, I've never
said that 'zh' excludes all but Mandarin.

The question on the table was what advice to give to people. For most
programs, I believe that treating them as synonyms is the right thing to do,
and alternative approaches would be extremely counterproductive. And this
goes for any of the macrolanguage cases where there is a predominant
encompassed language with long usage in the computer industry.

Let's take a scenario where this is not done (a la Ewell). Suppose that a
user picks Arabic as her Accept-Language in her browser. Any existing
browser will represent that with "ar". Then she goes to the BBC site. The
entire site is translated, not into standard Arabic, but into Sudanese
Creole Arabic. The user complains, since she can't understand it, and the
BBC responds that they are just following the standard to the letter and
spirit: "ar" means any kind of Arabic whatsoever, and so in the interests of
fairness, they pick a different encompassed language to serve up each day.
They inform the user that it is her fault for using 'ar' if she really only
wants Standard Arabic.

By your argument, this would be a perfectly reasonable thing to do. But for
users, it would hardly be satisfactory. And this would be a bizarrely stupid
thing for the BBC to do.

You might respond that, well, everyone needs to convert over to cmn for
Mandarin since it is now the Right Thing to Do. Even if that magically
happened, it would take years, and during the transition we would get all
kinds of screwups with different programs transitioning at different paces.
And there isn't much magic around; it is very hard to get people to change
infrastructure that works just fine -- you have to give them a compelling
case for why users are served better by the change. And that would be a very
hard sell, since there isn't any real advantage.

The right approach for the BBC is to treat a request 'ar' as a request for
Standard Arabic*, just as they have always done*. Internally, that means
treating 'ar' and 'arb', and any language tag that starts with them, as a
request for Standard Arabic. That is, treating ar-EG and arb-EG, or ar-SA
and arb-SA, or other combinations, as synonyms for the purpose of lookup.
Now, this "treating as synonyms" could be done in different ways. One way is
to mash on input; the other is to have a fancier fallback, eg arb-EG =>
ar-EG => arb => ar (mutatis mutandis, when starting with ar-EG).

Nor was this solved at all by extlang -- as we discussed at some length, it
discarded all script and region info when falling back, and produces worse
results in many cases, especially where the macrolanguage does not have a
predominant form.

I'm not saying the "treating as synonyms" strategy is always going to be the
right answer. There are undoubtedly scenarios where this strategy is not
necessary, although I can't think of any off the top of my head.

Moreover, I think one of the more productive things we can do is to push for
the incorporation of Language Priority Lists in any query-like protocols.
That way I could say I'd like "ary, fr, ar" if my preferred ordering is
Moroccan Arabic, then French, then as a last resort, Standard Arabic.

Mark

-- 
Mark

---

From: Shawn Steele <[email-masked]>
Date: Sat, 3 May 2008 22:21:08 -0700

> I'm not saying the "treating as synonyms" strategy is always going to be the right answer.
> There are undoubtedly scenarios where this strategy is not necessary, although I can't
> think of any off the top of my head.
I realize you're making a distinction between "are synonymns" and "are best treated (sometimes) as synonyms".  My problem with that is that sometimes will turn into always.

For example you mention that moving from zh to cmn is maybe the right thing to do, but even then it'll take years and people have to use the zh as a synonym in the meantime.  The problem is then nobody has any incentive to make the transition and "sometimes" synonyms will turn into "always" synonyms because there's no clear distinction between the two and when it is appropriate to use each one.

You even state that you can't easily think of any scenarios where you wouldn't treat them as synonyms.  So my suggestiton is that if they aren't distinct, then we shouldn't pretend they are.  (And then if zh == cmn, why bother changing and just keep zh).

We need to clearly explain when cmn would be used (or recommended) instead of zh in a non-synonym case.  If we can't do that, then we shouldn't be making the distinction.

- Shawn

---

From: Leif Halvard Silli <[email-masked]>
Date: Sun, 04 May 2008 08:59:24 +0200

Mark Davis 2008-05-04 04.08:
> treating them as synonyms is the right thing to do,
> and alternative approaches would be extremely counterproductive. 

What alternative, counterproductive, approaches do you see?

> And this goes for any of the macrolanguage cases where there is a
> predominant encompassed language with long usage in the computer
> industry.

Extlang tags would give less reason to advice against being precise. In 
addition, spreading the view that macrolanguage tag ≈ majority language 
tag would be far less less damaging if we had extlang tags.

The problem with your advice is that allthough you are only aiming it at 
the majority language, it fits for the "rest" of the macrolanguage as 
well. Because even for a resource producer of a minority language, the 
macrolanguage tag might be the safest if he wants to reach out. Could 
he, OTOH, use a extlang tag, then it would be less risky to be precise.

[ BBC example: ]
> Sudanese Creole Arabic. [...]  they pick a different encompassed language to serve up each day.

It seems reasonable to discern between tags used to "reach an audience" 
(via the HTTP protocol and content negotation) and tags used to classify 
the content. The HTTP server must focus on what the target group uses. 
The tags used inside the (web) document ought to be precise, though. 
Thus the only thing your BBC example demonstrates is that a site's HTTP 
server cannot use different tags each day even if they "pick a different 
encompassed language to serve up each day".

> thing for the BBC to do. [...]
When first the site was switching between arabic languages, then what 
better to use than 'ar'?
> Standard Arabic*, 
So, a big "sorry no content in standard Arabic, today" would have been 
better?

"For the BBC" is a keyword here.

The real problem was if an radiostation - say "Radio Free Sudan" - used 
the same URL to serve parallell news in Creol Arabic and standard 
Arabic. Then it would have had to look at its target group: If the 
talkers of Creol Arabic were more important to them than the standard 
Arabic users were, then - of course - the station should use 'ar' for 
its Creol Arabic web service.

> results in many cases, especially where the macrolanguage does not have a

Are there any examples or refs to these worse results?

> the incorporation of Language Priority Lists in any query-like protocols [...]

Something like that should be done anyhow.
-- 
leif halvard silli

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 4 May 2008 10:12:02 -0700

There are intelligent points of view on both sides about extlang. We had a
long and exhaustive debate about it some time ago,--long before you raised
this at the last minute--and were finally able to reach a rough consensus. I
don't think anyone really wants to go through that again. Your recent mails
don't really add any new information on the topic, I suggest that you review
the many, many archived mails about it.

Mark

On Sat, May 3, 2008 at 10:21 PM, Shawn Steele <[email-masked]>
wrote:

>  I realize you're making a distinction between "are synonymns" and "are
> best treated (sometimes) as synonyms".  My problem with that is that
> sometimes will turn into always.
>
> For example you mention that moving from zh to cmn is maybe the right
> thing to do, but even then it'll take years and people have to use the zh as
> a synonym in the meantime.  The problem is then nobody has any incentive to
> make the transition and "sometimes" synonyms will turn into "always"
> synonyms because there's no clear distinction between the two and when it is
> appropriate to use each one.
>
> You even state that you can't easily think of any scenarios where you
> wouldn't treat them as synonyms.  So my suggestiton is that if they aren't
> distinct, then we shouldn't pretend they are.  (And then if zh == cmn, why
> bother changing and just keep zh).
>
> We need to clearly explain when cmn would be used (or recommended) instead
> of zh in a non-synonym case.  If we can't do that, then we shouldn't be
> making the distinction.
>
> - Shawn
>
>

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 4 May 2008 11:54:25 -0600

Mark Davis wrote:

> I'm not the one being revisionist; the people quoting me are. I didn't 
> say "means", I said "has long been used to mean". There is a 
> difference. I didn't say "are synonyms", I said "are best treated as 
> synonyms" (and with qualifications). There's a difference. If I say 
> that for the purpose of juggling, I treat apples and oranges as the 
> same, it doesn't mean that I am somehow saying that they mean the same 
> thing. It means for particular purposes, I think they should be 
> handled the same way. Moreover, I've never said that 'zh' excludes all 
> but Mandarin.

What you said [1] was, "'zh' has long been used to mean Mandarin only." 
The way this reads is that people's *intent* in using 'zh' was to mean 
Mandarin only.  I claim that they did not generally have that intent, 
that they intended merely to mean "Chinese," that most of this "Chinese" 
happened to be Mandarin, and that it is revisionist to say that their 
*intent* was to specify Mandarin.  Of course, neither of us can poll the 
entire user community for their intent, so this debate probably can't be 
resolved.

I agree with the apples-and-oranges analogy: sometimes they should be 
handled the same way, sometimes they should not.  That's what 
macrolanguages are all about.

> Let's take a scenario where this is not done (a la Ewell). Suppose 
> that a user picks Arabic as her Accept-Language in her browser. Any 
> existing browser will represent that with "ar". Then she goes to the 
> BBC site. The entire site is translated, not into standard Arabic, but 
> into Sudanese Creole Arabic. The user complains, since she can't 
> understand it, and the BBC responds that they are just following the 
> standard to the letter and spirit: "ar" means any kind of Arabic 
> whatsoever, and so in the interests of fairness, they pick a different 
> encompassed language to serve up each day. They inform the user that 
> it is her fault for using 'ar' if she really only wants Standard 
> Arabic.

If we had gone ahead with extlangs, à la Ewell, the site could have been 
tagged "ar-apd", and the server and browser could have decided between 
themselves whether to match the user's "ar" with the site's "ar-apd", 
without consulting the Registry.  If the user insisted on only Standard 
Arabic, they could have specified "ar-arb", which would never have 
matched "ar-apd".

Without extlangs, either the site is tagged with the specific flavor of 
Arabic ("apd") and the browser must consult the Registry to see whether 
"apd" matches "ar", or the site is tagged "ar" and you get Mark's 
scenario above.  BBC can play games with rotating regional flavors of 
Arabic just as easily with or without extlangs.

> You might respond that, well, everyone needs to convert over to cmn 
> for Mandarin since it is now the Right Thing to Do.

Exactly not.  If we had gone with extlangs, people could have chosen 
"zh-cmn" which would have had some compatibility with "zh".  There would 
not have been this dichotomy between "zh" and "cmn" that we face now. 
(Yes, I know people can choose "zh-cmn" already because it is 
grandfathered in, but solving problems through grandfathered tags is 
probably not a good template.)

> Nor was this solved at all by extlang -- as we discussed at some 
> length, it discarded all script and region info when falling back, and 
> produces worse results in many cases, especially where the 
> macrolanguage does not have a predominant form.

This remove-from-right truncation argument was the one I never bought. 
Implementations could have been changed to perform more intelligent 
matching, without requiring any content to be retagged.  Remember the 
guiding principle behind ANSI C: "Existing code is important, existing 
implementations are not."

In any case -- having said all this -- the no-extlang arguments 
prevailed, and are written into the current drafts (including mine), and 
we should probably not waste time rehashing them.  I concede.

> Moreover, I think one of the more productive things we can do is to 
> push for the incorporation of Language Priority Lists in any 
> query-like protocols. That way I could say I'd like "ary, fr, ar" if 
> my preferred ordering is Moroccan Arabic, then French, then as a last 
> resort, Standard Arabic.

Agreed.  What is missing is a general principle for accomplishing this 
outside of the Web-browser model.  Perhaps this would make a good 
informational RFC, to be discussed and published AFTER the current suite 
of drafts are approved and published as RFCs.

[1] http://www.ietf.org/mail-archive/web/ltru/current/msg09797.html

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 4 May 2008 11:16:04 -0700

Doug,

> BBC can play games with rotating regional flavors of Arabic just as easily
with or without extlangs.

This is drifting off the main topic, which is how 'ar' should be treated.
Whether or not extlangs were defined, the BBC could easily support a
difference between arb and apd. **The question is what they should do about
'ar', which they will continue to get for years to come, even in a 'best
case' scenario.**
**

   1. If someone who supplies 'ar' in their browser really means that
   they would be satisfied with ANY kind of Arabic, then it would be perfectly
   reasonable for the BBC to serve up ANY kind of Arabic in response to 'ar'.
   (your apparent position)
   2. If someone who supplies 'ar' in their browser normally means that
   they would be satisfied with only Standard Arabic, then it would be not be
   reasonable for the BBC to serve up anything but Standard Arabic in response
   to 'ar'. (my position)

Do you really think #1 is sensible?

[extlang discussion omitted]

If you really want to rehash extlang, let's make that a separate thread.

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 4 May 2008 13:09:27 -0600

Mark Davis wrote:

> they would be satisfied with ANY kind of Arabic, then it would be 
> perfectly reasonable for the BBC to serve up ANY kind of Arabic in 
> response to 'ar'. (your apparent position)
>
> they would be satisfied with only Standard Arabic, then it would be 
> not be reasonable for the BBC to serve up anything but Standard Arabic 
> in response to 'ar'. (my position)
>

Sure.  Maybe not so much for Arabic as for Chinese or some other 
macrolanguages, but I don't think we really want to draw a normative 
line here.  The informative list in Figure 8 of draft-4646bis-13 (page 
48 in the text version) should be fine.

> If you really want to rehash extlang, let's make that a separate 

I don't really, and I apologize for rehashing it.  Leif Halvard Silli 
raised some questions about extlangs, and in the course of trying to 
answer them I stepped over the line.  Let's drop this and move on.

---

From: John Cowan <[email-masked]>
Date: Sun, 4 May 2008 15:56:49 -0400

Mark Davis scripsit:

>    2. If someone who supplies 'ar' in their browser normally means
>    that they would be satisfied with only Standard Arabic, then it
>    would be not be reasonable for the BBC to serve up anything but
>    Standard Arabic in response to 'ar'. (my position)

It probably isn't reasonable *for the Beeb* to serve up anything
but Standard Arabic; they probably don't even *have* anything else.
That does not mean that it is not reasonable for *anyone* to serve up
other Arabics in response to 'ar'; a radio-station site in Morocco might
very well serve Moroccan Arabic rather than Standard Arabic.  In Chad,
a site might serve French by default and Chadian Arabic to those who ask
for 'ar'; Standard Arabic is probably not useful in Chad at all.

-- 
That you can cover for the plentiful            John Cowan
and often gaping errors, misconstruals,         http://www.ccil.org/~cowan
and disinformation in your posts                [email-masked]
through sheer volume -- that is another
misconception.  --Mike to Peter
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Shawn Steele <[email-masked]>
Date: Mon, 5 May 2008 09:54:36 -0700

> There are intelligent points of view on both sides about extlang. We had a long
> and exhaustive debate about it some time ago,--long before you raised this at
> the last minute--and were finally able to reach a rough consensus.

As you know, I participated in those debates, and I don't think we reached consensus so much as one "side" caved.

> I suggest that you review the many, many archived mails about it.
Many of which were probably sent by myself :)

> I don't think anyone really wants to go through that again. Your recent mails
> don't really add any new information on the topic,

I think there is new information.  Since we have confusion about the recommendations about zh vs cmn, I think its worth considering if extlang would help resolve the confusion with synonyms.  I realize that everyone (including myself) would like to just be done with this, however I'm still concerned about the aliasing issue.

For example, if I see "zh" and "cmn", I have no way of telling that they're related without looking at the registry, which basically means a hard-coded table.  If "zh" is preferred", then I may want to move from "zh" to "cmn" or whatever.  If I'm using the cmn == zh fallback approach, then I need to know the correlation.

Now what happens when I get a "new" tag that I didn't see in the registry?  How do I know if it belongs to a macrolanguage?  I can't do this at runtime because I can't guarantee internet, ie: registry access, so suddenly I may doubt each tag that I don't recognize.  Is my unknown tag a Chinese variant?  I don't see 7300 entries in the registry, so presumably it'll be updated with some of the newly available languages available in Ethnologue.  Does this require that users of some small language wait for the registry to get updated before they may tag their data?

As Mark notes, many (if not most) macrolanguages aren't appropriate for fallback, so really I have to use the registry to find the correlation, unless someone knowing that the correlation was interesting was able to tag the data directly using extlang.

Live.com might be able to hit the registry for new info whenever it wants (hopefully they're always connected :)), but I have to support Windows in controlled environments where configuration issues, however minor, are very strictly regulated.  That includes secure DoD installations that may have to handle emerging markets and minority languages.

I have to also support mobile devices that may not have connectivity, self-serve kiosks, and other unconnected scenarios.  So in effect at least for some scenarios I have to build in a hard-coded list of possible fallbacks for whatever fallbacks are necessary.  Updating that list from the registry to learn about reletively minor changes for extlang fallback can very costly for some products and can cause an delay if an update has to be prepared..

I will, of course, implement whatever is agreed upon in windows, but I really need good guidance about things like whether or not zh and cmn are going to be an alias, and I really don't want to have an effectively hard-coded list in windows.  I'd rather have the data in the tag and good rules/suggestions derived from RFC4647bis.

The question I really need answered is "In the current draft, is cmn an alias for zh?"  So far I haven't seen any discussion that changes my current interpretation of "yes".  In which case I have to question the value of "cmn".  Yet I know some people are going to argue that cmn is distinct from zh, so therefore we need cmn.  Yet the strong recommendation of using "zh" instead of "cmn", leads me back to the thought that perhaps "zh-cmn" would be a more appropriate recommendation since it recognizes both the distinct aspects of cmn and the legacy macrolanguage behavior of zh.

- Shawn

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 5 May 2008 14:53:57 -0700

> consensus so much as one "side" caved.

That's actually what rough consensus typically means, on controversial
issues. I understand the feeling; I've been on the caving side myself many a
time, but at some point...

> I think there is new information.  Since we have confusion about the
> recommendations about zh vs cmn, I think its worth considering if extlang
> would help resolve the confusion with synonyms.  I realize that everyone
> (including myself) would like to just be done with this, however I'm still
> concerned about the aliasing issue.
>

We did consider that. And we have had text in the draft for some time
describing the situation, and saying "...where the macrolanguage tag has
been historically used to denote a dominant encompassed language, it SHOULD
be used in place of the subtag specific to that encompassed language unless
it is necessary to clearly distinguish the macrolanguage as a whole from
that enclosed dominant language variety". We then list the handful of cases
that we think are the appropriate ones that we know of.

In another posting, ("Concrete text fixes") I said:

The discussion of the treatment of macrolanguages is below that, starting at
"Some of the languages in the registry are labeled "macrolanguages" by ISO
639-3".

I've looked it over, and it does not seem unclear to me. If anyone has
concrete suggestions for additions or changed text, they should present
them.

If you think that text isn't clear, then perhaps you can suggest some
concrete changes?

> For example, if I see "zh" and "cmn", I have no way of telling that
> they're related without looking at the registry, which basically means a
> hard-coded table.  If "zh" is preferred", then I may want to move from "zh"
> to "cmn" or whatever.  If I'm using the cmn == zh fallback approach, then I
> need to know the correlation.
>

You can't tell that from the registry either! Sometimes microlanguages are
related in such a way as to be a good fallback, but usually they are not.
The handful of actual cases where we think it might be a good idea are
listed in the text in that section, *not* in the registry.

(If we wanted to anything with Norwegian, it would be there that we would
add it.)

> Now what happens when I get a "new" tag that I didn't see in the
> registry?  How do I know if it belongs to a macrolanguage?  I can't do this
> at runtime because I can't guarantee internet, ie: registry access, so
> suddenly I may doubt each tag that I don't recognize.  Is my unknown tag a
> Chinese variant?  I don't see 7300 entries in the registry, so presumably
> it'll be updated with some of the newly available languages available in
> Ethnologue.  Does this require that users of some small language wait for
> the registry to get updated before they may tag their data?
>

> As Mark notes, many (if not most) macrolanguages aren't appropriate for
> fallback, so really I have to use the registry to find the correlation,
> unless someone knowing that the correlation was interesting was able to tag
> the data directly using extlang.
>

Knowing that X is a macro/micro language does not necessarily mean -- *and
usually doesn't mean* -- that you want to use it in fallbacks. If there is
no predominant form, then it's a crapshoot as to whether the
macro/microlanguage is a good fallback.

There is no special runtime information in the registry. When a new
macro/microlanguage shows up in the registry *and* you support the one of
the pair, then it may be useful to review whether or not you want to add a
fallback. Automatically updating fallbacks blindly according to the registry
macro/microlanguages is not a good idea. Nor is it complete, since it misses
tl/fil, ro/mo, and many others that are much higher frequency cases than
most of the macro/microlanguages.

>
> Live.com might be able to hit the registry for new info whenever it wants
> (hopefully they're always connected :)), but I have to support Windows in
> controlled environments where configuration issues, however minor, are
> very strictly regulated.  That includes secure DoD installations that may
> have to handle emerging markets and minority languages.
>

Right, and for exactly those reasons you don't want to willy-nilly take
what's in the registry. It sounds like what you want in the registry is:

Subtag: br
Good-Fallback: fr
Good-Fallback: xx

That is, a field that would give you the best fallbacks in order. That is
just not the same as

Subtag: yy
Macrolanguage: xx

Such a "Good-Fallback" field would be require some other changes in the
registry, since you want to apply it to not just subtags. So you'd need
something like the following for Tags that aren't in the registry.

Type: fallback-source
Tag: zh-TW
Good-Fallback: zh-Hant-TW

Type: fallback-source
Tag: zh-Hant
Good-Fallback: zh-TW

It would be possible to discuss such additional structure for a later
revision of BCP 47, but that is really out of scope for this last call. Note
that like Suppress-Script, there are a small number of really clear cases
that would be easy to add -- and a dauntingly large number of cases where we
just don't have good information.

( BTW, we have been collecting data with similar goals in CLDR. See
http://unicode.org/cldr/data/common/supplemental/likelySubtags.xml
http://unicode.org/cldr/data/charts/supplemental/likely_subtags.html )

> I have to also support mobile devices that may not have connectivity,
> self-serve kiosks, and other unconnected scenarios.  So in effect at least
> for some scenarios I have to build in a hard-coded list of possible
> fallbacks for whatever fallbacks are necessary.
>

> Updating that list from the registry to learn about reletively minor
> changes for extlang fallback can very costly for some products and can cause
> an delay if an update has to be prepared..
>

Understood. For the handful of clear cases, we have a recommendation -- for
others we just don't have the data.

> I will, of course, implement whatever is agreed upon in windows, but I
> really need good guidance about things like whether or not zh and cmn are
> going to be an alias, and I really don't want to have an effectively
> hard-coded list in windows.  I'd rather have the data in the tag and good
> rules/suggestions derived from RFC4647bis.
>
> The question I really need answered is "In the current draft, is cmn an
> alias for zh?"  So far I haven't seen any discussion that changes my
> current interpretation of "yes".
>

Right, and that's what the draft says, and has said for some time.

> In which case I have to question the value of "cmn".
>

In an ideal world, cmn would have been introduced at the same time as zh,
which people would have used

> Yet I know some people are going to argue that cmn is distinct from zh, so
> therefore we need cmn. Yet the strong recommendation of using "zh" instead
> of "cmn", leads me back to the thought that perhaps "zh-cmn" would be a more
> appropriate recommendation since it recognizes both the distinct aspects of
> cmn and the legacy macrolanguage behavior of zh.
>

But it doesn't really help. Having zh-cmn instead of cmn means that simple
fallback messes up with regions and scripts, and is only beneficial in the
handful of special cases, not for all macro/microlanguage pairs. Anyway, the
whole extlang issue is complicated, with many pluses and minues, and I don't
want to rehash it here.

So my recommendation would be to simply handle the few special cases, and
then supply APIs and UIs that make it as easy as possible for users to
supply Language Priority Lists, since that is what really gives the user the
ability to say what he/she really wants.

---

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 5 May 2008 14:59:38 -0700

Hi Mark,

This email is extremely difficult to read, as your mail agent didn’t quote Shawn’s points separately from your own. I've been mulling a response on this thread, but want to start near the end :-).

Addison

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Monday, May 05, 2008 2:54 PM
To: Shawn Steele
Cc: LTRU Working Group; Doug Ewell
Subject: Re: [Ltru] I'm really confused by chinese in 3066bis

That's actually what rough consensus typically means, on controversial issues. I understand the feeling; I've been on the caving side myself many a time, but at some point...

We did consider that. And we have had text in the draft for some time describing the situation, and saying "...where the macrolanguage tag has been historically used to denote a dominant encompassed language, it SHOULD be used in place of the subtag specific to that encompassed language unless it is necessary to clearly distinguish the macrolanguage as a whole from that enclosed dominant language variety". We then list the handful of cases that we think are the appropriate ones that we know of.

In another posting, ("Concrete text fixes") I said:
The discussion of the treatment of macrolanguages is below that, starting at "Some of the languages in the registry are labeled "macrolanguages" by ISO 639-3".

I've looked it over, and it does not seem unclear to me. If anyone has concrete suggestions for additions or changed text, they should present them.

If you think that text isn't clear, then perhaps you can suggest some concrete changes?

You can't tell that from the registry either! Sometimes microlanguages are related in such a way as to be a good fallback, but usually they are not. The handful of actual cases where we think it might be a good idea are listed in the text in that section, *not* in the registry.

(If we wanted to anything with Norwegian, it would be there that we would add it.)

Knowing that X is a macro/micro language does not necessarily mean -- and usually doesn't mean -- that you want to use it in fallbacks. If there is no predominant form, then it's a crapshoot as to whether the macro/microlanguage is a good fallback.

There is no special runtime information in the registry. When a new macro/microlanguage shows up in the registry *and* you support the one of the pair, then it may be useful to review whether or not you want to add a fallback. Automatically updating fallbacks blindly according to the registry macro/microlanguages is not a good idea. Nor is it complete, since it misses tl/fil, ro/mo, and many others that are much higher frequency cases than most of the macro/microlanguages.

Right, and for exactly those reasons you don't want to willy-nilly take what's in the registry. It sounds like what you want in the registry is:

Subtag: br
Good-Fallback: fr
Good-Fallback: xx

That is, a field that would give you the best fallbacks in order. That is just not the same as

Subtag: yy
Macrolanguage: xx

Such a "Good-Fallback" field would be require some other changes in the registry, since you want to apply it to not just subtags. So you'd need something like the following for Tags that aren't in the registry.

Type: fallback-source
Tag: zh-TW

Type: fallback-source
Tag: zh-Hant
Good-Fallback: zh-TW

It would be possible to discuss such additional structure for a later revision of BCP 47, but that is really out of scope for this last call. Note that like Suppress-Script, there are a small number of really clear cases that would be easy to add -- and a dauntingly large number of cases where we just don't have good information.

I have to also support mobile devices that may not have connectivity, self-serve kiosks, and other unconnected scenarios.  So in effect at least for some scenarios I have to build in a hard-coded list of possible fallbacks for whatever fallbacks are necessary.

Updating that list from the registry to learn about reletively minor changes for extlang fallback can very costly for some products and can cause an delay if an update has to be prepared..

Understood. For the handful of clear cases, we have a recommendation -- for others we just don't have the data.

The question I really need answered is "In the current draft, is cmn an alias for zh?"  So far I haven't seen any discussion that changes my current interpretation of "yes".

In an ideal world, cmn would have been introduced at the same time as zh, which people would have used

Yet I know some people are going to argue that cmn is distinct from zh, so therefore we need cmn. Yet the strong recommendation of using "zh" instead of "cmn", leads me back to the thought that perhaps "zh-cmn" would be a more appropriate recommendation since it recognizes both the distinct aspects of cmn and the legacy macrolanguage behavior of zh.

But it doesn't really help. Having zh-cmn instead of cmn means that simple fallback messes up with regions and scripts, and is only beneficial in the handful of special cases, not for all macro/microlanguage pairs. Anyway, the whole extlang issue is complicated, with many pluses and minues, and I don't want to rehash it here.

So my recommendation would be to simply handle the few special cases, and then supply APIs and UIs that make it as easy as possible for users to supply Language Priority Lists, since that is what really gives the user the ability to say what he/she really wants.

- Shawn

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 5 May 2008 15:14:10 -0700

Addison,

I tried converting to plaintext. It does preface each of Shawn's
paragraphs with >, although not each line. Hope it helps.

Mark

---------- Forwarded message ----------
From: Mark Davis <[email-masked]>
Date: Mon, May 5, 2008 at 2:53 PM
Subject: Re: [Ltru] I'm really confused by chinese in 3066bis
To: Shawn Steele <[email-masked]>
Cc: Doug Ewell <[email-masked]>, LTRU Working Group <[email-masked]>

That's actually what rough consensus typically means, on controversial
issues. I understand the feeling; I've been on the caving side myself
many a time, but at some point...

We did consider that. And we have had text in the draft for some time
describing the situation, and saying "...where the macrolanguage tag
has been historically used to denote a dominant encompassed language,
it SHOULD be used in place of the subtag specific to that encompassed
language unless it is necessary to clearly distinguish the
macrolanguage as a whole from that enclosed dominant language
variety". We then list the handful of cases that we think are the
appropriate ones that we know of.

The discussion of the treatment of macrolanguages is below that,
starting at "Some of the languages in the registry are labeled
"macrolanguages" by ISO 639-3".

I've looked it over, and it does not seem unclear to me. If anyone has
concrete suggestions for additions or changed text, they should
present them.

> For example, if I see "zh" and "cmn", I have no way of telling that they're related without looking at the registry, which basically means a hard-coded table.  If "zh" is preferred", then I may wan…

You can't tell that from the registry either! Sometimes microlanguages
are related in such a way as to be a good fallback, but usually they
are not. The handful of actual cases where we think it might be a good
idea are listed in the text in that section, *not* in the registry.

(If we wanted to anything with Norwegian, it would be there that we
would add it.)

> As Mark notes, many (if not most) macrolanguages aren't appropriate for fallback, so really I have to use the registry to find the correlation, unless someone knowing that the correlation was inter…

Knowing that X is a macro/micro language does not necessarily mean --
and usually doesn't mean -- that you want to use it in fallbacks. If
there is no predominant form, then it's a crapshoot as to whether the
macro/microlanguage is a good fallback.

There is no special runtime information in the registry. When a new
macro/microlanguage shows up in the registry *and* you support the one
of the pair, then it may be useful to review whether or not you want
to add a fallback. Automatically updating fallbacks blindly according
to the registry macro/microlanguages is not a good idea. Nor is it
complete, since it misses tl/fil, ro/mo, and many others that are much
higher frequency cases than most of the macro/microlanguages.

> Live.com might be able to hit the registry for new info whenever it wants (hopefully they're always connected :)), but I have to support Windows in controlled environments where configuration issue…

Right, and for exactly those reasons you don't want to willy-nilly
take what's in the registry. It sounds like what you want in the
registry is:

Subtag: br
Good-Fallback: fr
Good-Fallback: xx

That is, a field that would give you the best fallbacks in order. That
is just not the same as

Subtag: yy
Macrolanguage: xx

Such a "Good-Fallback" field would be require some other changes in
the registry, since you want to apply it to not just subtags. So you'd
need something like the following for Tags that aren't in the
registry.

Type: fallback-source
Tag: zh-TW

Type: fallback-source
Tag: zh-Hant
Good-Fallback: zh-TW

It would be possible to discuss such additional structure for a later
revision of BCP 47, but that is really out of scope for this last
call. Note that like Suppress-Script, there are a small number of
really clear cases that would be easy to add -- and a dauntingly large
number of cases where we just don't have good information.

> Updating that list from the registry to learn about reletively minor changes for extlang fallback can very costly for some products and can cause an delay if an update has to be prepared..

Understood. For the handful of clear cases, we have a recommendation
-- for others we just don't have the data.

> In which case I have to question the value of "cmn".

In an ideal world, cmn would have been introduced at the same time as
zh, which people would have used

> Yet I know some people are going to argue that cmn is distinct from zh, so therefore we need cmn. Yet the strong recommendation of using "zh" instead of "cmn", leads me back to the thought that per…

But it doesn't really help. Having zh-cmn instead of cmn means that
simple fallback messes up with regions and scripts, and is only
beneficial in the handful of special cases, not for all
macro/microlanguage pairs. Anyway, the whole extlang issue is
complicated, with many pluses and minues, and I don't want to rehash
it here.

So my recommendation would be to simply handle the few special cases,
and then supply APIs and UIs that make it as easy as possible for
users to supply Language Priority Lists, since that is what really
gives the user the ability to say what he/she really wants.

-- 
Mark

---

From: Shawn Steele <[email-masked]>
Date: Mon, 5 May 2008 15:35:06 -0700

I at least mostly remembered what I wrote, so I followed it I think :)

> Right, and that's what the draft says, and has said for some time.

(until the correction is made it also refers to zh-cmn-Hant, which has been a poor example for an even longer time :)

So if cmn == zh, then I don't see any need for cmn, it just makes maintenance more difficult.  I would rather see a statement that Mandarin should use zh and then list cmn as grandfathered.

One additional concern.  You've noted that one cannot generally use macrolanguages since it is only potentially helpful in some cases, such as zh & yue.

2.1 says: ...Thus, a parser need not have a list of
   valid tags or subtags (that is, a copy of some version of the IANA
   Language Subtag Registry) in order to perform common searching and
   matching operations.

The "choice" section also mentions that yue may want to fallback to zh.  When I've decided that my application is interested in these types of fallback, how do I know for which languages this fallback may be appropriate?  (Hardcoding a list is an OK answer if yue->zh is the only language or one of a few with this potential.)

So in summary, I have 3 questions/concerns that I'd like addressed in the draft:

1) If cmn == zh, why do we need cmn?
2) If cmn == zh (or if zh is the only choice) is the lack of an ability to distinguish between Mandarin and the Chinese macrolanguage going to cause difficulty for some applications?
3) How do I know that falling back from a specific language to a macrolanguage may be interesting?

- Shawn

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 5 May 2008 16:20:56 -0700

> So if cmn == zh, then I don't see any need for cmn, it just makes maintenance more difficult.  I would rather see a statement that Mandarin should use zh and then list cmn as grandfathered.

I'm not saying that cmn = zh, just that for compatibility, most
applications want to treat them as the same. I can understand why some
people would want cmn (although it makes our lives harder).

In an ideal world, cmn, yue, and so on would have been around years
ago, and even if there were a zh, it would be unused in practice, like
gem.

> The "choice" section also mentions that yue may want to fallback to zh.  When I've decided that my application is interested in these types of fallback, how do I know for which languages this fallb…

There isn't the data in the registry for that. As I said, one could
have data somewhere for "Good-Fallback", but we don't have it.

Now, in terms of Chinese, I would expect that for most companies for
any xx-CN where xx is a minority language of China, zh-CN would be a
good fallback, because that would be the most likely understood
language. And if xx is predominantly spoken only in China, then zh
would be a good fallback.

But that is orthogonal to whether or not xx has zh as a macrolanguage or not.

> 2) If cmn == zh (or if zh is the only choice) is the lack of an ability to distinguish between Mandarin and the Chinese macrolanguage going to cause difficulty for some applications?

see above

> 3) How do I know that falling back from a specific language to a macrolanguage may be interesting?

BCP 47 doesn't address the issue of best fallback from one language to
another, except in a handful of special cases.

---

From: Shawn Steele <[email-masked]>
Date: Mon, 5 May 2008 20:32:40 -0700

> ago, and even if there were a zh, it would be unused in practice, like

I get that part :)

> people would want cmn (although it makes our lives harder).

In that case I think that the draft should be very clear when "zh" is expected to be used and when "cmn" is expected to be used.  I'm thoroughly confused by this:

"In general, use the most specific subtag to form the language tag. However, where the macrolanguage tag has been historically used to denote a dominant encompassed language, it SHOULD be used in place of the subtag specific to that encompassed language unless it is necessary to clearly distinguish the macrolanguage as a whole from that enclosed dominant language variety."

This sounds to me like "use zh instead of cmn because that's what was used historically."... unless its necessary to clearly distguish the macrolanguage from the dominant language variety.  If normally one tags the dominant language as the macrolanguage, how is it going to be possible to distinguish the macrolanguage from it?  Presumably cmn would clearly distinguish Mandarin, however zh can't be used to specify the Chinese macrolanguage because it is also used to mean Mandarin, in fact that's clearly the preferred meaning.

So if I want to clearly distinguish the macrolanguage, how do I clearly tag it?

- Shawn

---

From: John Cowan <[email-masked]>
Date: Mon, 5 May 2008 23:37:01 -0400

> So if I want to clearly distinguish the macrolanguage, how do I clearly tag it?

You can't.  But a macrolanguage is a group of languages that are treated as one for certain purposes.
Actual, individual documents have to be in a specific language.

-- 
LEAR: Dost thou call me fool, boy?      John Cowan
FOOL: All thy other titles              http://www.ccil.org/~cowan
             thou hast given away:      [email-masked]
      That thou wast born with.
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Shawn Steele <[email-masked]>
Date: Mon, 5 May 2008 20:55:27 -0700

So zh == Mandarin (its not a macrolanguage because docs have to be specific), and cmn == Mandarin...  So I can't clearly distinguish them because they're both the same thing.  So these sound like a mere alias to me.

I'm not trying to be unnecessarily dense, but I still haven't heard any reason for cmn to be permitted at all if we say Mandarin SHOULD be tagged as zh.  The only reason that seems plausible is that cmn has a certain aesthetic symmetry; since yue is permitted then cmn should be allowed.

I would prefer that cmn be disallowed if a clear case cannot be found that makes it useful.  Otherwise we all just say "zh == cmn" in our code and then if a distinction is found it'll be too late.  If we just use "zh" for now, then if a need is discovered we can update a later RFC to allow that case and we won't contradict the bunch of code we're about to write.

- Shawn

________________________________________
From: John Cowan [[email-masked]]
Sent: Monday, May 05, 2008 8:37 PM
To: Shawn Steele
Cc: Mark Davis; LTRU Working Group; Doug Ewell
Subject: Re: [Ltru] I'm really confused by chinese in 3066bis

---

From: John Cowan <[email-masked]>
Date: Tue, 6 May 2008 00:16:07 -0400

Shawn Steele scripsit:

> I'm not trying to be unnecessarily dense, but I still haven't heard
> any reason for cmn to be permitted at all if we say Mandarin SHOULD be
> tagged as zh.  The only reason that seems plausible is that cmn has
> a certain aesthetic symmetry; since yue is permitted then cmn should
> be allowed.
> 
> I would prefer that cmn be disallowed if a clear case cannot be found
> that makes it useful.  Otherwise we all just say "zh == cmn" in our code
> and then if a distinction is found it'll be too late.  

Code isn't everything (indeed, it's fairly easily changed) and the
Goohoosoft (TM) viewpoint isn't the only one.  That is, not everything
is about user locales, Web pages, and search engines, which have to deal
with languages rather crudely and in huge bulk, and for the most part
limit languages to textual objects.

In particular, in Karen Broome's movie database, cmn and yue are totally
different things, and there is no reason for her to blur the distinction
at all by employing zh in the "language of film" field (as opposed to
the "language of subtitles" field).  To be sure, she has to use zh-cmn
and zh-yue at present, because we are still sitting on 4646bis waiting
for the cows to come home, but when we finally get off the pot (to mix
a metaphor) she'll be able to use simply cmn.

-- 
What asininity could I have uttered     John Cowan <[email-masked]>
that they applaud me thus?              http://www.ccil.org/~cowan
        --Phocion, Greek orator
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Shawn Steele <[email-masked]>
Date: Mon, 5 May 2008 22:14:43 -0700

> the "language of subtitles" field).

I don't see where the draft says "zh SHOULD be used instead of cmn ... except for movies".  On the contrary, it seems to state that zh always means Mandarin.

If by "historically used to denote a dominant encompassed language" means only "in text", then it should say "in text".  If "unless it is necessary to clearly distinguish" means "spoken or on film", then it should say so.

I don't want to speak for Karen, but I doubt she wants most Mandarin to be tagged zh, but then use cmn for her own needs.  If she wanted her own private use of language tags I doubt she would've been so vocal on the list :)

So I don't "get it".  The draft says one SHOULD use zh for Mandarin, but when pressed you say cmn and yue are totally different things.  Yet the draft hasn't been able to clarify that zh is different than cmn for her.

You also imply that Goohoosoft (TM) doesn't collide with Karen's space, however it may be that users want to download movies on demand in a language they can understand, in which case the machine must be able to communicate with the movie catalogue.

FYI: My interaction with tags isn't limited to the "Goohoosoft" viewpoint.  I understand that Karen's needs may be different than resource lookup.  I have to support DVD playback devices, subtitled video and other content labeling.  I realize that librarians may have different concerns.  My company does do many different things with language tags, and most of those "what do I do now?" questions end up going to Peter or myself.

The main point I'm trying to make is that the draft is fairly vague on when an exception to the "SHOULD" is permitted.  That allows implementors or taggers or whomever to make different decisions.  That could be a good thing, but I think it'll mean that most users will end up saying "cmn" == "zh" or risk missing scenarios.  Then others like Karen won't be able to make the distinction if she wants to.

At this point I don't care what the solution is, so long as it isn't ambiguous.

- Shawn

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 06 May 2008 09:07:20 +0200

> Actual, individual documents have to be in a specific language.

<philosophy>
Macrolanguages can be compared with font-families.

In CSS we  speak about font-family.  But you can't select a font-family 
- you can only select a concrete font/typeface.

Thus, when you only say "Times New Roman" then you often really mean 
"Times New Roman **Regular**" - and that is also how HTML/CSS interprets 
it. However, that does not mean that "Times New Roman" is (only) 
synonymous to "Times New Roman Regular", as it also covers the Bold 
variant and the Italic and Bold Italic and perhaps others.  But even if 
you by default mean *Regular*", still, if you are in a run of Bold text, 
and say that you want to switch to "Times New Roman", then it means 
"Times New Roman **Bold**".

The same way the meaning of the name of the macrolanguage changes, by 
context. And sometimes it really refers to all at once.

If I can make another comparision: If you can't get the Italic variant 
of Times, then you must often look outside the font-family to find a 
replacement font. Because the font style is actually often more 
important than the font-family. In the same way, if a user can't get a 
certain microlangauge, then he might prefer to jump to another 
(macro)language, rather than picking another microlanguage under the 
same macrolanguage.

For Norwegian, I would turn what you said on its head: Norwegian is one 
language, which is treated as two languages, for certain purposes.  The 
Norwegian case is ultimately about a fight for and a right to the name 
"Norwegian". It is both a linguistic fight, and a name fight.

As for "Chinese", that is also about the right to be called "Chinese", 
you could say.

And I think it would illogical to give special treatment to 'zh' 
compared with other macrolangauge codes. That is, it would be illogical 
to say that zh == cmn. You can't say that it means only 'Mandarin', 
because you can't really remove the difference between "Mandarin" and 
"Chinese" without removing something even from the meaning of "Mandarin".
</philosophy>
-- 
leif halvard silli

---

From: "Andrew Cunningham" <[email-masked]>
Date: Tue, 6 May 2008 18:10:49 +1000

> So zh == Mandarin (its not a macrolanguage because docs have to be specific), and cmn == Mandarin...  So I can't clearly distinguish them because they're both the same thing.  So these sound like a…

this would mean that all relevant locale data should be changed, the
way applications operate should be changed

or does it mean that BCP 47 will not be suitable for defining locales?

so does ISO-639-3 which identifies "zh" as a macrolanguage have to be
changed? or will BCP 47 and ISO-639-3 diverge?

Andrew Cunningham
Vicnet Research and Development Coordinator
State Library of Victoria
Australia

[email-masked]
[email-masked]

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 06 May 2008 10:54:16 +0200

> "In general, use the most specific subtag to form the language tag. However, where the macrolanguage tag has been historically used to denote a dominant encompassed language, it SHOULD be used in p…

How about this change:

    "In general, use the most specific subtag to form the language tag.
    However, where the macrolanguage tag has historically been used as
    if it was a microlanguage tag for one particular microlanguage, the
    macrolanguage tag MAY continue to be used this way, as long as the
    correct microlanguage tag – or another microlanguage tag from the
    same macrolanguage – is not introduced to the same context. The
    macrolanguage tag MUST NOT be used as tag for one particular
    microlanguage if another microlanguage tag from the same
    macrolangauge occurs in the same context."

The above

   1. more clearly covers the possibility that also another
      microlanguage than the dominant one may have been referred to via
      the macrolanguage tag.
   2. speaks clearly against e.g. using "no" and "nn" in the same context.
   3. is more clear about which exception that is hereby permitted: what
      is permitted is to *continue to use* the macrolanguage tag in a
      *narrow* meaning, if - indeed - it has had that narrow meaning in
      a particular context. (For example, that 'zh' always and
      everywhere has meant 'mandarin' just because the text was in
      Mandarin, is, as we know, debated. This must be judged per context.)
   4. gives rules for when to stop using the macrolanguag tag as
      reference to one particular microlanguage.

> So if I want to clearly distinguish the macrolanguage, how do I clearly tag it?

Then, obviously, you must be consequent about using 'cmn' for Mandarin 
so that you can use 'zh' for the macrolanguage. I don't know if the 
draft is clear enought about the need for spesificly tagging things as 
macrolangauge?

(In Mac OS X, Bokmål and Nynorsk is used too much. For instance we have 
a Norwegian list and sorting format, but Mac OS X currently erroneously 
refers to it as "Bokmål sorting" and "Nynorsk sorting". So, there are 
clearly a neeed for the macrolanguage tag to for instance tag things 
that are the same in the entire macrolanguage.)
-- 
leif halvard silli

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 6 May 2008 12:34:32 +0100

I have to say I don't like the way we are contemplating dealing with
Macrolanguages.  Just to outline my perception of the current situation:

A macrolanguage has been designated within ISO 639-3 where there is a one to
many relationship between a subtag from another ISO 639 standard and more
than one subtag within ISO 639-3.  If we look at zh/zho as an example we can
see that this one to many macrolanguage/languages relationship encompasses
the following languages:

  cdo Min Dong Chinese
  cjy Jinyu Chinese
  cmn Mandarin Chinese
  cpx Pu-Xian Chinese
  czh Huizhou Chinese
  czo Min Zhong Chinese
  gan Gan Chinese
  hak Hakka Chinese
  hsn Xiang Chinese
  mnp Min Bei Chinese
  nan Min Nan Chinese
  wuu Wu Chinese
  yue Yue Chinese

In this situation, historically, the macrolanguage subtag has been used to
denote a dominant language in particular - cmn Mandarin Chinese.

However, whilst this is undoubtedly true, undoubtedly other languages within
the macrolanguage have also been encoded using the macrolanguage.

Where we seem to be getting into a bit of a pickle is with the suggestion
that zh should be used instead of cmn.  I think this would be a big mistake
and does not facilitate backward compatibility.

I propose that all future subtags for Mandarin Chinese should use cmn. I
would propose the following text to deal with this (this is a bit of a shot
in the dark to get out of this mess so feel free to shoot me down :-)):

---

Macrolanguages have been designated within ISO 639-3 where there is a one to
many relationship between a subtag from another ISO 639 standard and more
than one subtag within ISO 639-3.  In a number of cases the macrolanguage
has historically been used to denote one dominant language within the
designated macrolanguage.  In other cases the macrolanguage has been widely
used to denote more than one language within a macrolanguage.  In order to
facilitate backward compatibility and legacy systems, for macrolanguages
where this situation occurs, a comment has been inserted within Comment
Field 1 for the dominant related language(s) thus: Macrolanguage =
[macrolanguageSubtag].

Example 1: the dominant language for the macrolanguage subtag 'zh' is 'cmn'
(Mandarin Chinese), therefore Comment Field 1 would read: Macrolanguage = zh
Example 2: there is no particular dominant language within the macrolanguage
'no' which encompasses 'nn' and 'nb' therefore  Comment Field 1 for both
subtags reads: Macrolanguage = no

There are [number] number of subtags within the LSR which incorporate this
facility.

In general, use the most specific subtag to form the language tag. However,
in cases where a macrolanguage has been widely used to denote a dominant
language within the macrolanguage group, the registry includes the
macrolanguage within the first comment field within the registry to
facilitate fallback matching.  Where the macrolanguage tag has been
historically used to denote a dominant encompassed language, a duel language
tag MAY be used in addition to the subtag specific to that encompassed
language unless it is necessary to clearly distinguish the macrolanguage as
a whole from that enclosed
dominant language variety in which case just the macrolangauge subtag can be
used.

Give examples:

---

There are not that many macrolanguages and I think we could do this - it
could also be updated at any time when further info becomes avaiable as to
macrolangauge usage.

Best regards

Debbie

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Leif Halvard Silli
> Sent: 06 May 2008 09:54
> To: Shawn Steele
> > > applications want to treat them as the same. I can understand why
> > > some people would want cmn (although it makes our lives harder).
> >
> > In that case I think that the draft should be very clear
> when "zh" is expected to be used and when "cmn" is expected
> to be used.  I'm thoroughly confused by this:
> >
> > "In general, use the most specific subtag to form the
> language tag. However, where the macrolanguage tag has been
> historically used to denote a dominant encompassed language,
> it SHOULD be used in place of the subtag specific to that
> encompassed language unless it is necessary to clearly
> distinguish the macrolanguage as a whole from that enclosed
> dominant language variety."
> >
>
> How about this change:
>
> > This sounds to me like "use zh instead of cmn because
> that's what was used historically."... unless its necessary
> to clearly distguish the macrolanguage from the dominant
> language variety.  If normally one tags the dominant language
> as the macrolanguage, how is it going to be possible to
> distinguish the macrolanguage from it?  Presumably cmn would
> clearly distinguish Mandarin, however zh can't be used to
> specify the Chinese macrolanguage because it is also used to
> mean Mandarin, in fact that's clearly the preferred meaning.
> >
> Then, obviously, you must be consequent about using 'cmn' for
> Mandarin so that you can use 'zh' for the macrolanguage. I
> don't know if the draft is clear enought about the need for
> spesificly tagging things as macrolangauge?
>
> (In Mac OS X, Bokmål and Nynorsk is used too much. For
> instance we have a Norwegian list and sorting format, but Mac
> OS X currently erroneously refers to it as "Bokmål sorting"
> and "Nynorsk sorting". So, there are clearly a neeed for the
> macrolanguage tag to for instance tag things that are the
> same in the entire macrolanguage.)
> --
> leif halvard silli
>

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 6 May 2008 14:47:29 +0200

On Tue, May 06, 2008 at 10:54:16AM +0200,
 Leif Halvard Silli <[email-masked]> wrote 
 a message of 60 lines which said:

>     However, where the macrolanguage tag has historically been used
>     as if it was a microlanguage tag for one particular
>     microlanguage, the macrolanguage tag MAY continue to be used
>     this way, as long as the correct microlanguage tag ? or another
>     microlanguage tag from the same macrolanguage ? is not
>     introduced to the same context. The macrolanguage tag MUST NOT
>     be used as tag for one particular microlanguage if another
>     microlanguage tag from the same macrolangauge occurs in the same
>     context."

Why not, but remember that the politically correct word for
"microlanguage" is "encompassed language" for reasons already
discussed here.

[I add another reason: the character strings "microlanguage" and
"macrolanguage" are too similar and it is easy to mistake one for the
other.]
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 06 May 2008 16:51:10 +0200

> "macrolanguage" are too similar and it is easy to mistake one for the

New attemt:

    However, in a concretete context where a macrolanguage tag
    historically has been used to denote one, particular encompassed
    language, the macrolanguage tag MAY continue to be used in that
    meaning, provided that its spesific language subtag - or the
    spesific language subtag of another of the encompassed languges - is
    not introduced to the same context. A macrolanguage tag MUST NOT be
    used as tag for one particular encompassed language in a context
    where a another encompassed languages occurs."

---

From: Shawn Steele <[email-masked]>
Date: Tue, 6 May 2008 09:15:12 -0700

I like Debbie's approach much better than Leif's.  Leif delegates my concern to a "context", which I'm not sure makes things any clearer since applications sometimes use multiple "contexts".  Eg: if Karen's movie database is currently offline, then it may fit in a "context" that allows the cmn tag.  If the database then moves online (l ike to be sold on Amazon), then the context may change to a way that specifies the zh tag.)

I get tripped up over the last sentence:  "Where the macrolanguage tag has been
historically used to denote a dominant encompassed language, a duel language
tag MAY be used in addition to the subtag specific to that encompassed
language unless it is necessary to clearly distinguish the macrolanguage as
a whole from that enclosed dominant language variety in which case just the macrolangauge subtag can be
used."

Which sounds to me like the opposite of her introduction.  It seems to me that there are 3 cases:

1) The case where macrolanguages were never used (eg: English).
2) The case where a macrolanguage was used primarily for a dominant language (eg: Mandarin)
3) The case where a non-dominant language has historically been tagged with a macrolanguage because there wasn't another choice (eg: Cantonese).

I think that all three cases should be discoverable from the registry.  My very rough attempt at making this distinction in Debbie's last paragraph:

In general, the most specific subtag to form the language tag MUST be used. However,
in cases where a macrolanguage has been widely used to denote a dominant
language within the macrolanguage group, the registry includes the
macrolanguage within the first comment field within the registry to
facilitate fallback matching.  Where the macrolanguage tag has been
historically used to denote a dominant encompassed language, that macrolanguage
tag SHOULD be used for matching in addition to the tag appropriate for the
specific encompassed language.  When the emcompassed language is not the
dominate language but historically only had the macrolanguage available for
tagging, then the macrolanguage MAY be used for matching in addition tot the
appropriate specific encompassed language.

I'm trying to distinguish between cmn in the first SHOULD case and yue in the second MAY case.

Whatever happens I think examples would be a good idea.

zh -- Data historically tagged with the Chinese macrolanguage.  Presumed to be predominantly Mandarin, but may include other specific languages such as Cantonese.
cmn -- Data newly tagged as Mandarin.  When matching applications SHOULD match zh with cmn so that the historical data is recognized.
yue -- Data newly tagged as Cantonese.  When matching applications MAY match zh if they feel that some Cantonese data in their context may have been tagged as Chinese.

- Shawn
________________________________________
From: Debbie Garside [[email-masked]]
Sent: Tuesday, May 06, 2008 4:34 AM
To: 'Leif Halvard Silli'; Shawn Steele
Cc: 'LTRU Working Group'; 'Doug Ewell'
Subject: RE: [Ltru] I'm really confused by chinese in 3066bis

  cdo Min Dong Chinese
  cjy Jinyu Chinese
  cmn Mandarin Chinese
  cpx Pu-Xian Chinese
  czh Huizhou Chinese
  czo Min Zhong Chinese
  gan Gan Chinese
  hak Hakka Chinese
  hsn Xiang Chinese
  mnp Min Bei Chinese
  nan Min Nan Chinese
  wuu Wu Chinese
  yue Yue Chinese

---

Give examples:

---

Best regards

Debbie

> language tag.

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 May 2008 09:32:56 -0700

From: [email-masked] [mailto:[email-masked]] On Behalf Of Shawn Steele
Sent: Monday, May 05, 2008 9:55 AM

> As you know, I participated in those debates...
> Since we have confusion about the recommendations about zh vs cmn, I think its
> worth considering if extlang would help resolve the confusion with synonyms...

I'm wondering if the more useful discussion (which IIRC hasn't happened at all) is how to create platforms that allow back-compat with all existing usage of "zh" while needing to also start supporting new application scenarios in which Cantonese needs to be distinguished from Mandarin. Allowing zh-cmn may deal with certain fallback or synonymy scenarios, but it doesn't tell me how to create a platform (e.g. Win32, .Net, ICU) that supports Cantonese vs. Mandarin scenarios. Similarly, Mark's suggestion to treat "zh" and "cmn" "for most programs" also doesn't tell me how to do that.

So, I'm suggesting to separate these issues:

1. If I see "zh" and "cmn", do I need to refer to the registry to tell that they're related (perhaps treated synonymously)?

2. In a platform that to date has only known "zh", how do I start to support Mandarin vs. Cantonese scenarios while still maintaining back compat with existing apps?

Peter

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 06 May 2008 18:55:44 +0200

> and does not facilitate backward compatibility.

So, it seems to me that you, in the above, say that the 'zh' must be 
kept "free" so that it - most typically - can stand for Mandaran 
Chinese, but also so that it - in other contexts - can stand for one of 
the other encompassed Chinese languages.

> I propose that all future subtags for Mandarin Chinese should use cmn. I

I agree. In fact, I should perhaps have put more weight on that in the 
rewrite of the text that I made. For instance, new services which are 
not intented to use all encompassed languages, should prepare for the 
future inclusion of other of the encompassed langauges by using a 
spesific subtag.

But in what, future situation would it be advicable to actually use the 
macrolanguage tag, do you think? (Because, what you say could, in my 
mind, mean that 'zh' should almost go out (of future) use.)

Is the macrolanguage tag only a tag for backwards compatibility? Or has 
it any future role?

For instance, is there a difference - or likeness - between using the 
'mul' tag and a macrolanguage tag? Should 'zh' be used as *the* tag for 
resources with several chinese languages, or should 'mul' be used 
instead? Or perhaps a third option?

Currently the draft says about  'mul' that

    "It SHOULD NOT be used when a list of languages (such as
    Content-Language) or individual tags for each content element can be
    used instead."

Ought not the draft have an entry saying something about whether using 
the macrolanguage tag to cover all the encompassed languages is a good idea?

It seems to me that the same rules should count for macrolanguage tags 
as for mul, if or when they are used to play the same - though more 
spesific - role as 'mul'. I.e. if I as author/webserver want to catch 
all persons searching for 'chinese', then I should not simply rely on 
the old 'zh' tag, but should instead - when possible - list all the 
chinese languages. Likewise, for Norwegian, if the goal for me as 
author/server is to catch all Norwegian users to a single-language (one 
encompassing langauge) Norwegian web page, then I should not rely on 
'no', but instead, if possible, use both nn, nb and no. Sensible?

As I see it, a macrolanguage tag can be used on things which are 
"neutrally chinese" or "neutrally Norwegian".  "Neutral" should be 
understood either as field where there is of no interest or perhaps 
irrelevant to classify into encompassed languages. The macrolanguage tag 
might be used in context where there is no theoretical possibility that 
the same kind of resource, in the future, will be split into encompassed 
languages. (One cathegory where it will often be safe to use the 
macrolanguage tage is probably names of persons and juridic persons. 
Though for some public institutions may have spesific encompassing 
language names.)
> in the dark to get out of this mess so feel free to shoot me down :-)):

How about up? ;-)

> where this situation occurs,a comment has been inserted within Comment

This sounds as a good idea.

> subtags reads: Macrolanguage = no

The examples are not so clearly written, I feel. You meant that there 
should be a comment linking from the relevant encompassed language to 
the the macrolanguage tag. As I understand it, you by this comment want 
the list to inform abotu for which sublanguage tags the macrolanguage 
tag should be used as fallback. So, basically, you try to give info 
about which encompassed languages the macrolangauge tag is very direct 
fallback for. (And thus, thereby also inderectly noting for which 
languages it might not be a good fallback.)

I think this should be formulated without using the term "dominant" and 
other postulate like wordings that could lead to debate.

> There are [number] number of subtags within the LSR which incorporate this

There are 50 macrolanguage tags, if my RegEx served me well. And many 
more encompassed languages.

> facilitate fallback matching.

So, if I got this right,  the point about the comment field is not to 
inform the reader that the macrolanguage tag could not be used for 
tagging, but simply to inform applications that they need to facilitate 
fallback matching.

>   Where the macrolanguage tag has been
> tag 

What does 'duel language tag' mean? Two language tags? Such as "cmn, zh" 
pr 'zh-cmn'?

> MAY be used in addition to the subtag specific to that encompassed

Sounds like a good idea to give advice about all of them.
-- 
leif halvard silli

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 6 May 2008 17:56:09 +0100

Ok... now having read the draft in relation to Macrolanguages ... :-)
Ignore my previous message...  I will start again.

I am not in favour of using any macrolanguage subtag for what is
"considered" to be the dominant language as I feel there will be a body of
information tagged that will not be in the dominant language.  I also feel
that we should use the correct ISO 639-3 subtag or there will be a mish mash
of future tagging going on for these languages; with some "in the know"
tagging Mandarin Chinese as 'zh' and others as 'cmn'.  I do, however,
recognise that some applications necessarily need the information with
regard to "dominant" language and that there would appear to be sufficient
knowledge within this community for these languages to be called "dominant".

I would propose the following:

---

   In general, use the most specific subtag to form the language tag.
   Where the macrolanguage tag has been historically used to
   denote a dominant encompassed language, it MAY NOT be used in place of
   the subtag specific to that encompassed language.

   To facilitate fallback matching for the dominant language within a
Macrolanguage the Macrolanguage field for the dominant language should
include, in addition to the Macrolanguage subtag, a rating of number 1
(one).

   For example the Macrolanguage field for Mandarin Chinese 'cmn' will read
as follows: zh;1

   The pairs of macro and encompassed languages affected by this issue
   when this document was published were:

        Arabic 'ar'                    Standard Arabic 'arb'
        Konkani (macrolanguage) 'kok'  Konkani (single language) 'knn'
        Malay (macrolanguage) 'ms'     Malay (single language) 'mly'
        Swahili (macrolanguage) 'sw'   Swahili (single language) 'swh'
        Uzbek 'uz'                     Northern Uzbek 'uzn'
        Chinese 'zh'                   Mandarin Chinese 'cmn'

Each of these cases will include the number 1 (one) within the macrolanguage
field.  This scheme can be further expanded to facilitate additional
fallback matching by adding ratings 2, 3, 4 etc. to other languages
encompassed within the Macrolanguage if deemed appropriate.

---

Accept or reject as you please.  Now I am going to go away and solve my own
problems and leave this to you. :-)

best

Debbie

> Sent: 06 May 2008 15:51
> To: Stephane Bortzmeyer
> Cc: LTRU Working Group

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 06 May 2008 19:28:28 +0200

>    the subtag specific to that encompassed language.

Thinking about nn,nb,no I agree strongly. 'no' should not be used in 
place of the more spesific subtag.

>         Chinese 'zh'                   Mandarin Chinese 'cmn'

Is this intended as the full list? I suppose not?

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 6 May 2008 11:04:30 -0700

User perception of the change (in Windows, in their browser, in tag choices) is that there was "Chinese" (zh-*) and there remains "Chinese"---but we have added "new" languages/locales to their system that weren't there before: here's a choice for "Cantonese" and here's a choice for "Min Nan" and there's a choice for "Mandarin" even. They are free to use "Chinese", but now can accurately choose another language where it suits them. In practice, "Chinese" meant (and means) Mandarin in most written contexts and is further differentiated by script. Users may need to have more than one language to achieve their results for Chinese, since written content generally can be classified as 'zh-*' but audio-visual content generally has to be classified by the actual language/dialect. Most systems and system locales will boot up in "Chinese".

For audio-visual content, the situation is more complicated because "Chinese" really isn't an option. But Chinese speakers of various flavors--I've sought them out and interviewed many--seem to understand the variations in language quite well. They don't understand tagging any better than anyone else in the world, but they understand the relationships between the Chinese languages. So offering this choice works on that level.

It is important to note that the difference between how you handle written Chinese and how you handle spoken Chinese languages is *inherent* in the nature of Chinese. For written content it is entirely reasonable to use 'zh' in many cases, but other forms of content (and certain written cases) require us to identify the specific language/dialect. A glance at the Wikipedia entry for "Chinese language"--even if one dislikes the particular slant given in that article--should make clear why implied meaning for regional subtags is not the answer.

Or, to follow Einstein, language tags should be as simple as possible, but no simpler.

Addison

Addison Phillips

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Peter Constable
> Sent: Tuesday, May 06, 2008 9:33 AM
> To: LTRU Working Group
> I'm wondering if the more useful discussion (which IIRC hasn't happened
> at all) is how to create platforms that allow back-compat with all
> existing usage of "zh" while needing to also start supporting new
> application scenarios in which Cantonese needs to be distinguished from
> Mandarin. Allowing zh-cmn may deal with certain fallback or synonymy
> scenarios, but it doesn't tell me how to create a platform (e.g.
> Win32, .Net, ICU) that supports Cantonese vs. Mandarin scenarios.
> Similarly, Mark's suggestion to treat "zh" and "cmn" "for most
> programs" also doesn't tell me how to do that.
>
> 1. If I see "zh" and "cmn", do I need to refer to the registry to tell
> that they're related (perhaps treated synonymously)?
>
> 2. In a platform that to date has only known "zh", how do I start to
> support Mandarin vs. Cantonese scenarios while still maintaining back
> compat with existing apps?
>
>
>
> Peter

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 May 2008 11:32:49 -0700

> Mark Davis
> Sent: Monday, May 05, 2008 3:14 PM

> That is, a field that would give you the best fallbacks in order. That

Mark, you keep coming back to general fallback scenarios, and I think we've agreed that we don't want to register specific fallback chains. It seems to me that the only reason to jump from the specific problems of "zh" to a discussion of general fallback is that we don't want to consider the "zh" case as somehow special -- because that smacks of cherry picking. But it seems to me that's just trying to ignore the reality that "zh" *is* a special situation because of the dualism in its usage, and is not just one instance of the general fallback scenario.

Peter

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 May 2008 11:44:07 -0700

> changed?

No matter how anybody here answers that one, I think it's safe to say it won't happen: ISO 639-3 treats the scope of zh[o] the way it does, as a macrolanguage, because that is an accurate reflection of pre-existing usage.

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 6 May 2008 11:45:46 -0700

If you look at the context, I was just posing a hypothetical -- I'm not
saying that we should add support like that, just emphasizing that if you
were looking for "best fallback" in the registry, we don't support that, and
it is a mistake to assume that Macrolanguage: is equivalent to that.

Mark

On Tue, May 6, 2008 at 11:32 AM, Peter Constable <[email-masked]>
wrote:

> Mark, you keep coming back to general fallback scenarios, and I think
> we've agreed that we don't want to register specific fallback chains. It
> seems to me that the only reason to jump from the specific problems of "zh"
> to a discussion of general fallback is that we don't want to consider the
> "zh" case as somehow special -- because that smacks of cherry picking. But
> it seems to me that's just trying to ignore the reality that "zh" *is* a
> special situation because of the dualism in its usage, and is not just one
> instance of the general fallback scenario.
>
>
>
> Peter

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 May 2008 11:49:27 -0700

> Leif Halvard Silli
> Sent: Tuesday, May 06, 2008 1:54 AM

>     correct microlanguage tag - or another microlanguage tag from the
>     same macrolanguage - is not introduced to the same context. The

Essentially that says don't use zh in any data category if there's a possibility that someday you may need to use yue (or nan, etc.) in that same data category. That entails, for instance, either that we can use xml:lang="zh" and must never use xml:lang="yue", or that we must henceforth stop ever using xml:lang="zh".

Seems to me that some might find that problematic.

Peter

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 May 2008 11:54:53 -0700

> Debbie Garside

> In this situation, historically, the macrolanguage subtag has been used
> to denote a dominant language in particular - cmn Mandarin Chinese.
>
> However, whilst this is undoubtedly true, undoubtedly other languages
> within the macrolanguage have also been encoded using the macrolanguage.

Just in case anyone is unclear about this, not only is it *undoubtedly true* (which, in my dialect ironically suggests a small potential for doubt), it can be seen to be explicitly, demonstrably and incontrovertibly true either by looking at the IANA LSTR or by comparing the MARC Language Code List with its ISO counterpart, ISO 639-2.

Peter

---

From: Shawn Steele <[email-masked]>
Date: Tue, 6 May 2008 12:08:25 -0700

> Most systems and system locales will boot up in "Chinese".

Perhaps you've oversimplified the problem.  Our customers will want voice interaction to be in the spoken language, Chinese is no longer sufficient for the system.  Even on a web site, audio clips may be available so Chinese doesn't necessarily provide enough granularity.

- Shawn

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 6 May 2008 22:35:32 +0100

Leif wrote:

> So, it seems to me that you, in the above, say that the 'zh'
> must be kept "free" so that it - most typically - can stand
> for Mandaran Chinese, but also so that it - in other contexts
> - can stand for one of the other encompassed Chinese languages.

No.  Not really.  See my other proposed solution.  I think I explain it in a
better (although slighter different) way.
>
> > I propose that all future subtags for Mandarin Chinese
> should use cmn.
> > I
> >
>
> I agree. In fact, I should perhaps have put more weight on
> that in the rewrite of the text that I made. For instance,
> new services which are not intented to use all encompassed
> languages, should prepare for the future inclusion of other
> of the encompassed langauges by using a spesific subtag.
>
> But in what, future situation would it be advicable to
> actually use the macrolanguage tag, do you think? (Because,
> what you say could, in my mind, mean that 'zh' should almost
> go out (of future) use.)

Only use the macrolanguage when you are sure that the item you wish to tag
is in one of the languages encompassed by the macrolanguage but you are
unsure/don't know/cannot presently identify which one.
>
> Is the macrolanguage tag only a tag for backwards
> compatibility? Or has it any future role?

See above.
>
> For instance, is there a difference - or likeness - between
> using the 'mul' tag and a macrolanguage tag? Should 'zh' be
> used as *the* tag for resources with several chinese
> languages, or should 'mul' be used instead? Or perhaps a third option?

I am not overly familiar with 'mul' and would need to read up on it to
answer this question.
> Ought not the draft have an entry saying something about
> whether using the macrolanguage tag to cover all the
> encompassed languages is a good idea?

Indeed.  The draft should give guidance on when the macrolangauge may be
used.  I do not, as I have stated before, beleive we should be advising the
end user to tag the dominant language with the macrolanguage subtag.

> It seems to me that the same rules should count for
> macrolanguage tags as for mul, if or when they are used to
> play the same - though more spesific - role as 'mul'. I.e. if
> I as author/webserver want to catch all persons searching for
> 'chinese', then I should not simply rely on the old 'zh' tag,
> but should instead - when possible - list all the chinese
> languages. Likewise, for Norwegian, if the goal for me as
> author/server is to catch all Norwegian users to a
> single-language (one encompassing langauge) Norwegian web
> page, then I should not rely on 'no', but instead, if
> possible, use both nn, nb and no. Sensible?

Yes.  Sensible.  This is what I would do.  The existing (in draft 13)
macrolaguage field facilitates this.
>
> As I see it, a macrolanguage tag can be used on things which
> are "neutrally chinese" or "neutrally Norwegian".  "Neutral"
> should be understood either as field where there is of no
> interest or perhaps irrelevant to classify into encompassed
> languages. The macrolanguage tag might be used in context
> where there is no theoretical possibility that the same kind
> of resource, in the future, will be split into encompassed
> languages. (One cathegory where it will often be safe to use
> the macrolanguage tage is probably names of persons and
> juridic persons.
> Though for some public institutions may have spesific
> encompassing language names.)

Indeed.  But also, as stated above, where the end user has insufficient
knowledge to tag more accurately/precisely.

> > would propose the following text to deal with this (this is
> a bit of a
> > shot in the dark to get out of this mess so feel free to
> shoot me down :-)):
> >
>
> How about up? ;-)

Hmmm...  Localization is a wonderful thing ;-)

>
> > ---
> >
> > Macrolanguages have been designated within ISO 639-3 where
> there is a
> > one to many relationship between a subtag from another ISO 639
> > standard and more than one subtag within ISO 639-3.  In a number of
> > cases the macrolanguage has historically been used to denote one
> > dominant language within the designated macrolanguage.  In
> other cases
> > the macrolanguage has been widely used to denote more than one
> > language within a macrolanguage.  In order to facilitate backward
> > compatibility and legacy systems, for macrolanguages where this
> > situation occurs,a comment has been inserted within Comment Field 1
> > for the dominant related language(s) thus: Macrolanguage =
> [macrolanguageSubtag].
> >
>
> > (Mandarin Chinese), therefore Comment Field 1 would read:
> > Macrolanguage = zh Example 2: there is no particular
> dominant language
> > within the macrolanguage 'no' which encompasses 'nn' and 'nb'
> > therefore  Comment Field 1 for both subtags reads:
> Macrolanguage = no
> >
>
> The examples are not so clearly written, I feel. You meant
> that there should be a comment linking from the relevant
> encompassed language to the the macrolanguage tag. As I
> understand it, you by this comment want the list to inform
> abotu for which sublanguage tags the macrolanguage tag should
> be used as fallback. So, basically, you try to give info
> about which encompassed languages the macrolangauge tag is
> very direct fallback for. (And thus, thereby also inderectly
> noting for which languages it might not be a good fallback.)

Yes.  Although I don't think I put it very well.

>
> I think this should be formulated without using the term
> "dominant" and other postulate like wordings that could lead
> to debate.
>
> > There are [number] number of subtags within the LSR which
> incorporate
> > this facility.
> >
>
> There are 50 macrolanguage tags, if my RegEx served me well.
> And many more encompassed languages.
>
> > However, in cases where a macrolanguage has been widely
> used to denote
> > a dominant language within the macrolanguage group, the registry
> > includes the macrolanguage within the first comment field
> within the
> > registry to facilitate fallback matching.
>
> So, if I got this right,  the point about the comment field
> is not to inform the reader that the macrolanguage tag could
> not be used for tagging, but simply to inform applications
> that they need to facilitate fallback matching.

Yes.

> denote a
> > dominant encompassed language, a duel language tag
>

I went a bit pear shaped here due to a 12 yr old requiring rescue from a
shopping centre 20 miles away :)  it should have read "dual".  In any case
it was just a thought that I had which went back to something John Cowan had
said previously about having two language tags. But I have changed my mind
on this as it does not fit my (current) view.

> > macrolanguage as a whole from that enclosed dominant
> language variety
> > in which case just the macrolangauge subtag can be used.
> >
> > Give examples:
> >
> > ---
> >
> > There are not that many macrolanguages and I think we could
> do this -
> > it could also be updated at any time when further info becomes
> > avaiable as to macrolangauge usage.
> >
>

Thanks.

Debbie

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 6 May 2008 22:40:25 +0100

Shawn wrote:

> I like Debbie's approach much better than Leif's.

Thank you :-)

 Leif
> delegates my concern to a "context", which I'm not sure makes
> things any clearer since applications sometimes use multiple
> "contexts".  Eg: if Karen's movie database is currently
> offline, then it may fit in a "context" that allows the cmn
> tag.  If the database then moves online (l ike to be sold on
> Amazon), then the context may change to a way that specifies
> the zh tag.)
>
> I get tripped up over the last sentence:  "Where the
> macrolanguage tag has been historically used to denote a
> dominant encompassed language, a duel language tag MAY be
> used in addition to the subtag specific to that encompassed
> variety in which case just the macrolangauge subtag can be used."

Yes.  As stated previously, things went a bit pear shaped here.  Ignore it.

> 3) The case where a non-dominant language has historically
> been tagged with a macrolanguage because there wasn't another
> choice (eg: Cantonese).
>
> I think that all three cases should be discoverable from the
> registry.  My very rough attempt at making this distinction
> in Debbie's last paragraph:
>
> In general, the most specific subtag to form the language tag
> MUST be used. However, in cases where a macrolanguage has
> been widely used to denote a dominant language within the
> macrolanguage group, the registry includes the macrolanguage
> within the first comment field within the registry to
> facilitate fallback matching.  Where the macrolanguage tag
> has been historically used to denote a dominant encompassed
> language, that macrolanguage tag SHOULD be used for matching
> in addition to the tag appropriate for the specific
> encompassed language.  When the emcompassed language is not
> the dominate language but historically only had the
> macrolanguage available for tagging, then the macrolanguage
> MAY be used for matching in addition tot the appropriate
> specific encompassed language.

This sounds pretty good.  But I think I prefer my second solution (adding
;1) to the dominant language with the may on those rated 2 or more (or none
if the additional rating system is unacceptable)
> case and yue in the second MAY case.

I like it.

> Whatever happens I think examples would be a good idea.

Indeed.

>
> zh -- Data historically tagged with the Chinese
> macrolanguage.  Presumed to be predominantly Mandarin, but
> may include other specific languages such as Cantonese.
> cmn -- Data newly tagged as Mandarin.  When matching
> applications SHOULD match zh with cmn so that the historical
> data is recognized.
> yue -- Data newly tagged as Cantonese.  When matching
> applications MAY match zh if they feel that some Cantonese
> data in their context may have been tagged as Chinese.

Seems very reasonable to me.

Debbie
> I have to say I don't like the way we are contemplating
> dealing with Macrolanguages.  Just to outline my perception
> of the current situation:
>
> A macrolanguage has been designated within ISO 639-3 where
> there is a one to many relationship between a subtag from
> another ISO 639 standard and more than one subtag within ISO
> 639-3.  If we look at zh/zho as an example we can see that
> this one to many macrolanguage/languages relationship
> encompasses the following languages:
>
>   cdo Min Dong Chinese
>   cjy Jinyu Chinese
>   cmn Mandarin Chinese
>   cpx Pu-Xian Chinese
>   czh Huizhou Chinese
>   czo Min Zhong Chinese
>   gan Gan Chinese
>   hak Hakka Chinese
>   hsn Xiang Chinese
>   mnp Min Bei Chinese
>   nan Min Nan Chinese
>   wuu Wu Chinese
>   yue Yue Chinese
>
> In this situation, historically, the macrolanguage subtag has
> been used to denote a dominant language in particular - cmn
> Mandarin Chinese.
>
> However, whilst this is undoubtedly true, undoubtedly other
> languages within the macrolanguage have also been encoded
> using the macrolanguage.
>
> Where we seem to be getting into a bit of a pickle is with
> the suggestion that zh should be used instead of cmn.  I
> think this would be a big mistake and does not facilitate
> backward compatibility.
>
> I propose that all future subtags for Mandarin Chinese should
> use cmn. I would propose the following text to deal with this
> (this is a bit of a shot in the dark to get out of this mess
> so feel free to shoot me down :-)):
>
> ---
>
> there is a one to many relationship between a subtag from
> another ISO 639 standard and more than one subtag within ISO
> 639-3.  In a number of cases the macrolanguage has
> historically been used to denote one dominant language within
> the designated macrolanguage.  In other cases the
> macrolanguage has been widely used to denote more than one
> language within a macrolanguage.  In order to facilitate
> backward compatibility and legacy systems, for macrolanguages
> where this situation occurs, a comment has been inserted
> within Comment Field 1 for the dominant related language(s)
> thus: Macrolanguage = [macrolanguageSubtag].
>
> Macrolanguage = zh Example 2: there is no particular dominant
> language within the macrolanguage 'no' which encompasses 'nn'
> and 'nb' therefore  Comment Field 1 for both subtags reads:
> Macrolanguage = no
>
> incorporate this facility.
>
> In general, use the most specific subtag to form the language
> tag. However, in cases where a macrolanguage has been widely
> used to denote a dominant language within the macrolanguage
> group, the registry includes the macrolanguage within the
> first comment field within the registry to facilitate
> fallback matching.  Where the macrolanguage tag has been
> a duel language tag MAY be used in addition to the subtag
> specific to that encompassed language unless it is necessary
> to clearly distinguish the macrolanguage as a whole from that
> enclosed dominant language variety in which case just the
> macrolangauge subtag can be used.
>
> Give examples:
>
> ---
>
> do this - it could also be updated at any time when further
> info becomes avaiable as to macrolangauge usage.
>
> Best regards
>
> Debbie
>
>
>
>
>
> > From: [email-masked] [mailto:[email-masked]]
> On Behalf
> > Of Leif Halvard Silli
> > Sent: 06 May 2008 09:54
> > To: Shawn Steele
> > when "zh" is expected to be used and when "cmn" is expected to be
> > used.  I'm thoroughly confused by this:
> > >
> > historically used to denote a dominant encompassed
> language, it SHOULD
> > be used in place of the subtag specific to that encompassed
> language
> > unless it is necessary to clearly distinguish the
> macrolanguage as a
> > whole from that enclosed dominant language variety."
> > >
> >
> > How about this change:
> >
> >       *narrow* meaning, if - indeed - it has had that
> narrow meaning
> > in
> to clearly
> > distguish the macrolanguage from the dominant language variety.  If
> > normally one tags the dominant language as the
> macrolanguage, how is
> > it going to be possible to distinguish the macrolanguage from it?
> > Presumably cmn would clearly distinguish Mandarin, however
> zh can't be
> > used to specify the Chinese macrolanguage because it is
> also used to
> > (In Mac OS X, Bokmål and Nynorsk is used too much. For instance we
> > have a Norwegian list and sorting format, but Mac OS X currently
> > erroneously refers to it as "Bokmål sorting"

---

From: John Cowan <[email-masked]>
Date: Tue, 6 May 2008 18:53:16 -0400

> unsure/don't know/cannot presently identify which one.

Or "do not care".

-- 
Not to perambulate                 John Cowan <[email-masked]>
    the corridors                  http://www.ccil.org/~cowan
during the hours of repose
    in the boots of ascension.       --Sign in Austrian ski-resort hotel
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Shawn Steele <[email-masked]>
Date: Tue, 6 May 2008 15:55:05 -0700

> Or "do not care".

Is "do not care" a reasonable choice?  Seems that if you knew which specific language it was, then it wouldn't matter much if you used that if you didn't care.

- Shawn

---

From: <[email-masked]>
Date: Tue, 6 May 2008 18:12:14 -0700

Shawn, for the record, you're doing a good job of speaking for me. Please 
continue.... :)

Karen Broome

[email-masked] wrote on 05/05/2008 10:14:43 PM:

> 
> I don't want to speak for Karen, but I doubt she wants most Mandarin
> to be tagged zh, but then use cmn for her own needs.  If she wanted 
> her own private use of language tags I doubt she would've been so 
> vocal on the list :)

---

From: <[email-masked]>
Date: Tue, 6 May 2008 18:49:14 -0700

The reason "zh" is special is because in written form, it is mutually 
intelligible with the written forms used by other dialect speakers even if 
there can be subtle differences (Cantonese subtitles). In spoken form, the 
languages are not mutually intelligible, so while I know the folks on this 
list don't generally like splitting spoken and written language hairs, the 
truth of the matter seems to be this:

Written Form

zh = written Chinese (most likely Mandarin)
zh-yue = written Cantonese
zh-cmn = definitely written Mandarin

These tags can be used with the Hant, Hani, and Hans tags. From my 
experience, most spoken Mandarin and Cantonese language classifiers use a 
made-up tag rather than "zh." I've seen this in moving image archives all 
over the world. 

However, I'm sure I'm not the only one with an archive of "Chinese" 
objects where we don't know whether the language is spoken, written, 
Cantonese, Mandarin, or Taiwanese. The legacy ISO structure has itself 
created a need for "zh" where the semantic equals, "We don't know what 
kind of Chinese this is because the tags weren't specific enough at the 
time we created this database."

Spoken Forms

cmn = spoken Mandarin
yue = spoken Cantonese
cmn-TW = spoken Taiwanese Mandarin

SHOULD NOT use script subtags with these forms.

I'll throw that out there for the sake of argument. 

Regards,

Karen Broome
Metadata Systems Designer
Sony Pictures Entertainment
310.244.4384

[email-masked] wrote on 05/06/2008 11:32:49 AM:

> Mark, you keep coming back to general fallback scenarios, and I 
> think we've agreed that we don't want to register specific fallback 
> chains. It seems to me that the only reason to jump from the 
> specific problems of "zh" to a discussion of general fallback is 
> that we don't want to consider the "zh" case as somehow special -- 
> because that smacks of cherry picking. But it seems to me that's 
> just trying to ignore the reality that "zh" *is* a special situation
> because of the dualism in its usage, and is not just one instance of
> the general fallback scenario.
> 
> 
> 
> Peter

---

From: [email-masked]
Date: Tue, 6 May 2008 18:55:26 -0700

What language does my screen reader speak if the page is coded "zh"?   Is 
"zh" sufficient for this purpose? We can't predict all the ways a language 
tag might be interpreted by an application -- even in text contexts -- so 
I think the muddiness of the "zh" tag is a problem. It does not 
sufficiently identify the language.

K.

[email-masked] wrote on 05/06/2008 12:08:25 PM:

> Perhaps you've oversimplified the problem.  Our customers will want 
> voice interaction to be in the spoken language, Chinese is no longer
> sufficient for the system.  Even on a web site, audio clips may be 
> available so Chinese doesn't necessarily provide enough granularity.
> 
> - Shawn

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 6 May 2008 19:21:40 -0700

I think the answer is Standard Chinese (Mandarin), just like the answer for
German is Standard German (Hochdeutsch), not Schwytzerduetsch or other kinds
of German, even though before 2006 the Schwytzerduetsch would have been
tagged with 'de' or "de-CH".

All of the language tags were muddy to one degree or another.

Mark

---

From: pektiong <[email-masked]>
Date: Wed, 7 May 2008 10:30:38 +0800

Hi there,

   First, I would like to state that even in written form using Hant/s/ 
i, the written Minnan/Hakka/Mandarin CAN BE mutually intelligible.

   Second, in Taiwan, we have the need to distinguish
   Written Mandarin-Hant, Latn
   Written Minnan-Hant, Latn
   Written Hakka-Hant, Latn
   and the (quasi)-official Latin orthography for Mandarin/Minnan/ 
Hakka is very different from each other.
   We of course also need to distinguish spoken Mandarin/Minnan/Hakka

   Finally,  this issue also have social linguistic impact. Treat  
zh=Mandarin and use nan/hak/yue for other "Chinese"
   will either make people feel non-Mandarin Chinese is inferior or  
non-Mandarin Chinese is not Chinese.
   I understand this is not the intent of the language tag but such  
kind of social linguistic impact is certainly there.

best
pektiong

On May 7, 2008, at 9:49 AM, <[email-masked]> <[email-masked]

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 6 May 2008 20:41:25 -0600

John Cowan <cowan at ccil dot org> wrote:

> You can't.  But a macrolanguage is a group of languages that are 
> treated as one for certain purposes.  Actual, individual documents 
> have to be in a specific language.

I thought we had more than a few use cases of content in a 
semi-determined language (e.g. "some Nahuatl language, not sure which 
one") that would be properly tagged with a collection code.  This is 
even less specific than a macrolanguage.

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 6 May 2008 20:50:05 -0600

Leif Halvard Silli <lhs at malform dot no> wrote:

> The Norwegian case is ultimately about a fight for and a right to the 
> name "Norwegian". It is both a linguistic fight, and a name fight.
>

I thought the goal was to *avoid* using language tagging to send 
messages.

---

From: Arne Goetje <[email-masked]>
Date: Wed, 07 May 2008 11:39:08 +0800

[email-masked] wrote:
> there can be subtle differences (Cantonese subtitles).

Also for at least Minnan, Hakka and Zhuang the vocabulary and therefor
the written form differs from Mandarin. They use the same script (Hant
for Minnan and Hakka in Taiwan, Hans for Zhuang in China) like Mandarin,
but different characters.

People familiar with the written forms of those languages can tell the
difference.

(For Karen's case: subtitles in Taiwan are mostly Mandarin (cmn-TW),
while audio tracks can be cmn-TW, nan-TW or hak-TW.)

Also, Minnan and Hakka can be written in Latin script or a mixture of
Latin and Hant script, similar like Japanese is usually written in a
mixture of Hiragana and Kanji.

Therefor I would prefer to avoid the use of 'zh' altogether and only use
 'cmn', 'nan', 'hak', etc.

However, if a software requests the language 'zh' from the operating
system, then it's up to the OS (and probably the user) to decide which
language will be delivered back. The default setting could be that 'cmn'
is delivered back, but the user could override this with 'nan' if he
prefers...

Just my 2 NT$...

Cheers
Arne
- --
Arne Götje (高盛華) <[email-masked]>
PGP/GnuPG key: 1024D/685D1E8C
Fingerprint: 2056 F6B7 DEA8 B478 311F  1C34 6E9F D06E 685D 1E8C
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2008 00:53:28 -0400

> even less specific than a macrolanguage.

Indeed.  Just because a document must be in a specific language does not
mean that the tagger always knows what it is.  One may properly tag a
document und-Cyrl if that's all you know, or just und if *that's* all
you know.  That doesn't change the fact that the document is in fact
in Bulgarian.

-- 
Some people open all the Windows;       John Cowan
wise wives welcome the spring           [email-masked]
by moving the Unix.                     http://www.ccil.org/~cowan
  --ad for Unix Book Units (U.K.)
        (see http://cm.bell-labs.com/cm/cs/who/dmr/unix3image.gif)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 6 May 2008 23:15:02 -0600

John Cowan <cowan at ccil dot org> wrote:

> Indeed.  Just because a document must be in a specific language does 
> not mean that the tagger always knows what it is.  One may properly 
> tag a document und-Cyrl if that's all you know, or just und if 
> *that's* all you know.  That doesn't change the fact that the document 
> is in fact in Bulgarian.

So if all you know about a document is that it's in some flavor of 
Arabic, you tag it "ar".  If you know that it's in Ta'izzi-Adeni Arabic, 
and wish to specify that, then you tag it "acq".  And if you know that 
it's in Standard Arabic, then you tag it either:

- "ar" if you want the emphasis to be on accessibility to everyone who 
requests "ar", or

- "arb" if you want the emphasis to be on the fact that this is Standard 
Arabic and not just any old flavor of Arabic.

Is that even close to right?

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2008 01:23:59 -0400

> Is that even close to right?

I call it 100% right.

I'd also say that "ar" isn't like "zh" (though some people have said it
is), for the simple reason that the parallelism is broken.  Most written
*and* spoken Chinese is Mandarin (probably 70% of Chinese-speakers know
no other Sinitic language), but although almost all written Arabic is
Standard Arabic, the same cannot be said of spoken Arabic, which presents
a much more complex situation.

-- 
John Cowan   http://ccil.org/~cowan    [email-masked]
In might the Feanorians / that swore the unforgotten oath
brought war into Arvernien / with burning and with broken troth.
and Elwing from her fastness dim / then cast her in the waters wide,
but like a mew was swiftly borne, / uplifted o'er the roaring tide.
        --the Earendillinwe
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2008 01:26:43 -0400

> zh-cmn = definitely written Mandarin

[snip]

> cmn-TW = spoken Taiwanese Mandarin

This introduces an artificial distinction between zh-yue and yue, and likewise between
zh-cmn and cmn, that has no real reason to exist.  zh-yue and zh-cmn are RFC 4646
forms; in RFC 4646bis they will be replaced with yue and cmn respectively.
What's more, zh is a suitable tag for spoken Chinese if you do not know
(etc.) what kind of spoken Chinese you have.

-- 
We are lost, lost.  No name, no business, no Precious, nothing.  Only empty.
Only hungry: yes, we are hungry.  A few little fishes, nassty bony little
fishes, for a poor creature, and they say death.  So wise they are; so just,
so very just.  --Gollum        [email-masked]  http://ccil.org/~cowan
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 6 May 2008 23:44:35 -0600

Shawn Steele <Shawn dot Steele at microsoft dot com> wrote:

> In general, the most specific subtag to form the language tag MUST be
> used. However, in cases where a macrolanguage has been widely used to
> denote a dominant language within the macrolanguage group, the
> registry includes the macrolanguage within the first comment field
> within the registry to facilitate fallback matching.

As I've said before, but evidently not loudly enough, I am STRONGLY
OPPOSED to using the Comments field for anything normative, or anything
that software would use to enable or facilitate tag selection or
matching.  I'm not sure I can express how bad an idea I think this is.

Comments fields are supposed to be for free-form, human-readable 
"additional information about the record."  They are completely 
optional, and can be added, changed, and deleted at any time.  Order 
doesn't matter; there is no "Comment Field 1" as far as software is 
concerned.  As both RFC 4646 and the current draft point out, "no 
guarantee of stability is provided."  Comments are just comments.

If we need another field with normative information -- and I'm not
convinced we do, but if -- then for heaven's sake, let's create a new
one.  I don't want to prolong this work any longer than we have to,
believe me, but if the alternative is to overload the Comments field for
algorithmic subtag matching, then we have more work to do.

> I think that all three cases should be discoverable from the registry.

As a software developer, I understand Shawn's desire to find a neat set
of rules that encapsulate this whole, crazy matching circus involving
macrolanguages and encompassed languages, and put those rules in the
Registry where software can deterministically find them.

Unfortunately, we can't even agree among ourselves what the rules should 
be.  Should a request for "cmn" return content tagged "zh" that might be 
Cantonese?  Is it OK for a request for "zh-HK" to be truncated back to 
"zh" (Mark's RFR fallback scenario) and then matched to "cmn"?  Are the 
rules the same for written vs. non-written?  And none of the other 
macrolanguage scenarios is quite like Chinese, or Arabic or Zhuang; 
they're all different.

Additionally, the concept of "a dominant language within the group" may 
be well-defined for Chinese and Arabic, but there are 55 other 
macrolanguages currently defined.  We would essentially be getting 
ourselves into the same controversy that now swirls around 
Suppress-Script, where between this group and ietf-languages, there are 
widely divergent opinions about how liberally Suppress-Script should be 
used, and some who still don't seem to understand it at all.  By 
pronouncing one language "dominant," which even Ethnologue doesn't 
always do, we would open ourselves up to more charges of bias, not only 
outside the group but inside as well.

Finally, the status of any given language as a macrolanguage (and 
consequently the Macrolanguage field within draft-4646bis) is subject to 
change at any time by ISO 639-3/MA.  Already there have been changes to 
the existing macrolanguage membership; we won't see these changes in the 
Registry because they predate our publication date, but others are sure 
to follow.  The draft says the Macrolanguage field "MAY" be changed to 
mirror ISO 639-3 changes, but I can't think of any scenario where this 
would not be the case.  So this information is, well, informative.  Any 
mechanism that treats it as stable or normative is fatally flawed.

IMHO we should limit ourselves to solving problems that we know how to
solve.  Frankly, I think we have done the best that we can do by adding
the Macrolanguage field, for whatever good it provides, and the
explanatory text Addison and Mark (and others) have already written into
draft-4646bis.

---

From: pektiong <[email-masked]>
Date: Wed, 7 May 2008 13:45:20 +0800

According to a survey by PRC government around 2004. Around 53% of  
population in China can speak Mandarin.
But I would also like to comment that the TOTAL SPEAKER of any non- 
Mandarin Chinese is also a LARGE number
compared to other languages in the world.

Ref: http://news.sina.com.cn/c/2004-12-26/17204633373s.shtml

Also, the two dialects within nan (or hak or yue or wuu) COULD be  
mutually UN-intelligible. What if one day NAN itself becomes
a macrolanguage tag?

Cheers
pektiong

---

From: Shawn Steele <[email-masked]>
Date: Tue, 6 May 2008 22:59:19 -0700

I agree with most of your concerns, particularly about "dominant language," but come to the opposite conclusion :)

If we cannot agree amongst ourselves which tags to best choose and how fallback might be used, then application developers will do no better an impliment a variety of mechanisms.  That will only serve to muddle the meaning of any tags, particularly when interoperability between vendors or data sources is required.

Although a "neat set of rules" would be nice, I can't imagine that would be possible, even without the legacy behavior there are too many interesting variations of "language".  I would however like to aim for a meaningful set of rules that are clear enough that people may be able to consistently apply them.

- Shawn

________________________________________
From: Doug Ewell [[email-masked]]
Sent: Tuesday, May 06, 2008 10:44 PM
To: LTRU Working Group
Cc: Shawn Steele
Subject: Re: [Ltru] I'm really confused by chinese in 3066bis

---

From: Shawn Steele <[email-masked]>
Date: Tue, 6 May 2008 23:04:03 -0700

That's kind of my point.  Someone oversimplified my concerns as just matching some web pages, however Microsoft's trying to solve a broader set of scenarios.  It is fairly clear to us that we can't just ask users to pick "zh" for their system locale and expect that to suffice.  In an increasing number of cases we need additional information, such as cmn or yue.

I suspect that given how broadly our features and apis are used that I need to support the same set of scenarios that Karen needs, in addition to those of librarian, in addition to the obvious scenarios of web surfing and OS resource lookup.

- Shawn

________________________________________
From: [email-masked] [[email-masked]]
Sent: Tuesday, May 06, 2008 6:55 PM
To: Shawn Steele
Cc: Phillips, Addison; LTRU Working Group; Peter Constable
Subject: Re: [Ltru] I'm really confused by chinese in 3066bis

K.

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 May 2008 23:10:18 -0700

> Arne Goetje

> the written form differs from Mandarin.

Um... Zhuang is a Tai-Kadai language, not a Sinitic language -- so not at all related to Mandarin.

Peter

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2008 02:14:03 -0400

> population in China can speak Mandarin.

That figure sounds quite low.

> compared to other languages in the world.

Indeed, and the same is true of the other Sinitic languages.  Cantonese
is probably the most widely spoken language in the world that is not
official anywhere.

> mutually UN-intelligible. What if one day NAN itself becomes

It may someday, as might cmn or even en.

-- 
Real FORTRAN programmers can program FORTRAN    John Cowan
in any language.  --Allen Brown                 [email-masked]
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 May 2008 23:27:43 -0700

> Doug Ewell

> ourselves into the same controversy... [b]y
> pronouncing one language "dominant," ...

I will remind people that last November I sent an attachment with my analysis of the macro-language entries in ISO 639-3, attached again here for convenience, in which I take a stab at categorizing according to degree of usage in ICTs and whether there is a single, dominant language. Here is group #2:

2. Macrolanguage ID used in ICT, and there is a single, dominant variety
ara     (E15: arb dominant) = ar
        arb = Standard Arabic
        - aao abh abv acm acq acw acx acy adf aeb aec afb ajp apc
        apd arq ars ary arz auz avl ayh ayl ayn ayp bbz pga shu ssh

kok     (E14: knn dominant)
        knn
        - gom

msa     (E15: mly dominant) = ms
        mly
        - btj bve bvu coa jax max meo mfa mqg msi vkt xmm

swa     (E15: swh dominant) = sw (CLDR)
        swh
        - swc

uzb     (E15: uzn dominant) = uz
        uzn

        - uzs
zho     (E14: collection) = zh
        cmn
        - cdo cjy cpx czh czo gan hak hsn mnp nan wuu yue

> change at any time by ISO 639-3/MA.

Not exactly: ISO 639-3/MA can't narrow the scope of a macro-language category.

Peter

---

From: Leif Halvard Silli <[email-masked]>
Date: Wed, 07 May 2008 17:51:24 +0200

> I thought the goal was to *avoid* using language tagging to send 

That comment was meant to express that we should map the terrain instead 
of trying to make the terrain look like the map.

Don't you agree with that message?
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2008 11:55:12 -0400

> of trying to make the terrain look like the map.

Unfortunately, unlike a mapmaker we have to take into account the
existence of previous maps both formal and informal, because they have
strongly affected people's view of what the terrain actually is.

-- 
Kill Gorgun!  Kill orc-folk!            John Cowan
No other words please Wild Men.         [email-masked]
Drive away bad air and darkness         http://www.ccil.org/~cowan
with bright iron!   --Ghan-buri-Ghan    http://www.ccil.org/~cowan
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Wed, 07 May 2008 18:00:10 +0200

> is delivered back, but the user could override this with 'nan' if he

That was two good cents. Such a behaviour would be good for an OS 
localized for Norwegian as well.
-- 
leif halvard silli

---

From: Leif Halvard Silli <[email-masked]>
Date: Wed, 07 May 2008 18:20:57 +0200

John Cowan 2008-05-07 07.23:
> > and wish to specify that, then you tag it "acq".

I assume that John will agree that it is also 100% right to use 'ar' 
even if you know that it is  Ta'izzi-Adeni Arabic.

> > And if you know that it's in Standard Arabic, then you tag it either:
> > 

If you don't include what I said about  using 'ar' for Ta'izzi-Adeni 
Arabic as well, then this means that when it comes to the "dominant 
encompassing language", then we recommend a less precise tagging.
-- 
leif halvard silli

---

From: Shawn Steele <[email-masked]>
Date: Wed, 7 May 2008 09:28:11 -0700

I missed this before, but it sounds to me like Karen would like a way to recognize the correlation between the written forms of Mandarin and Cantonese yet keep the spoken forms distinct?

So zh-cmn, zh-yue in a written context are interesting to Karen because they are mutually intelligible even though there some differences.  By themselves cmn and yue in a spoken context are also interesting to Karen because they cannot be exchanged at all.

Is that correct?  I'll have to think about that scenario.

- Shawn

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Leif Halvard Silli [[email-masked]]
Sent: Wednesday, May 07, 2008 9:00 AM
To: Arne Goetje
Cc: LTRU Working Group
Subject: Re: [Ltru] Fwd:  I'm really confused by chinese in 3066bis

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 7 May 2008 10:26:29 -0700

According to what I've hear, "mutually intelligible" is too strong. Here is
what I have on the subject from John Jenkins:

   1. I believe you said that a Mandarin speaker can read written
   Cantonese, but will not understand everything (a bit like a Dane reading
   Swedish).

More like French and Spanish, actually.

   1. Some characters would not (normally) be used in Mandarin.

Most famously U+4E5C, the Cantonese for "what".

   1. Some characters would have different meanings than in Mandarin

Best illustrated with U+4FC2, which means "to bind" in Mandarin but is
frequently borrowed to for the Cantonese word for "to be."

   1. Some syntax would be different

Yes, but I can't think of any examples off the top of my head and the book
I've got that lists the differences is successfully playing hide-and-seek at
the moment.  There are actually not an awful lot of these.  The main
differences between the two are phonetic and lexical.  The grammars are very
similar.

   1. Am I understanding correctly?

Yes.

   1. Can you point me to some web pages with written Cantonese that
   would demonstrate that to a Chinese reader?

Nothing better to start with than the Cantonese Wikipedia article on
Cantonese:  <http://zh-yue.wikipedia.org/wiki/粵語<http://zh-yue.wikipedia.org/wiki/%E7%B2%B5%E8%AA%9E>>.
 Similarly, <http://zh-yue.wikipedia.org/wiki/香港<http://zh-yue.wikipedia.org/wiki/%E9%A6%99%E6%B8%AF>>,
<http://zh-yue.wikipedia.org/wiki/Unicode>, and pretty much everything else
in the Cantonese Wikipedia.
I can give you the relative character frequencies in the Cantonese and
traditional (or simplified Chinese) Wikipediae, if you like.  I've still got
that data around somewhere.

The thing you have to emphasize is the difference between
what-Cantonese-speakers-generally-read-and-write, which is just Mandardin
with Cantonese phonetics, and
writing-down-what-Cantonese-speakers-actually-speak, which is what "written
Cantonese" should be used to mean.  Unfortunately, not everybody groks this.
 Fortunately, Wikipedia does.

Meanwhile, I quote from Stephen Matthews and Virginia Yip, _Cantonese: A
Comprehensive Grammar_ (London: Routledge, 1994), pp. 5-6:  "Traditionally,
Cantonese has been regarded as one of the many Chinese dialects. It does not
have a standardized written form on a par with standard written Chinese.  No
form of written Cantonese is taught in schools or used in academic settings
in any Cantonese-speaking community.  When it comes to the written form, it
is standard written Chinese that is taught and learnt.  For educated
Cantonese speakers, standard written Chinese is the written form they use in
most contexts.  However, in colloquial genres such as novels, popular
magazines, newspaper gossip columns, informal personal communications,
written Cantonese may be used.  When written Cantonese contains too many
exclusively Cantonese words and expressions, non-Cantonese speakers may find
it totally unintelligible."

Since they wrote, however, there's been a distinct upsurge in the use of
written Cantonese.  (It's tied in with a kind of Hong Kongese
pseudo-nationalism.)  It's still not exactly *common*, but it's a lot more
common than it used to be.

Mark

On Wed, May 7, 2008 at 9:28 AM, Shawn Steele <[email-masked]>
wrote:

> I missed this before, but it sounds to me like Karen would like a way to
> recognize the correlation between the written forms of Mandarin and
> Cantonese yet keep the spoken forms distinct?
>
> So zh-cmn, zh-yue in a written context are interesting to Karen because
> they are mutually intelligible even though there some differences.  By
> themselves cmn and yue in a spoken context are also interesting to Karen
> because they cannot be exchanged at all.
>

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2008 14:17:12 -0400

Mark Davis scripsit:

> the moment.  

Two that come to mind:

1) Cantonese normally places direct objects bofore indirect objects,
unlike Standard Mandarin.  This property is shared with Shanghainese
and with southern varieties of Mandarin.

2) Mandarin prefers verb-object compounds, Cantonese prefers to make
them object-verb.

-- 
If you have ever wondered if you are in hell,         John Cowan
it has been said, then you are on a well-traveled     http://www.ccil.org/~cowan
road of spiritual inquiry.  If you are absolutely     [email-masked]
sure you are in hell, however, then you must be
on the Cross Bronx Expressway.          --Alan Feuer, NYTimes, 2002-09-20
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2008 16:07:41 -0400

> even if you know that it is  Ta'izzi-Adeni Arabic.

Yes, hence the "and wish to specify that".

-- 
On the Semantic Web, it's too hard to prove     John Cowan    [email-masked]
you're not a dog.  --Bill de hOra               http://www.ccil.org/~cowan
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 00:27:29 +0200

> Yes, hence the "and wish to specify that".

At the same time, I think the draft should specify that the recommended 
use of macrolanguage tags is to allways used them in a list of language 
tags.

Thus, in order to be 200% right, instead of using 'ar' for Ta'izzi-Adeni 
Arabic, on should use 'acq,ar'. Of course, only if the application 
allows it. But the draft says very little about "list of languages". And 
it also does not make the proposal that applications permit specifying 
the language in a list of tags. Only in connection with 'mul' does it 
say that a list of languages is preferred over the use of 'mul'.

It would have been possible to say about 'mul' that you should use 'a 
list of languages' only when you "wish to specify that". But that is not 
what the draft says. It says that you always should use a list of 
languages rather than the 'mul' tag.

And it seems meaningful to me that the draft should likewise always 
RECCOMMEND to that macrolanguage tags are used as part of a list 
consisting of the encompassing language (or languages) and the 
macrolangauge. Because, macrolanguage tags are like 'mul', only  more 
precise.

Thus, if we agree about this, then in the case above, the author is 
RECOMMENDED to make a choice: Either to tag with 'ar', and then the 
draft should recommend to *also* use 'acq', or to only use 'acq'.

Can a list format be specified or RECOMMENDED in this group? In HTML and 
XML, the compatible way to set up a list of language seems to be to use 
the '_' (underscore) between each language tag. Thus, for instance, for 
the list 'Bokmål-Norway, Norwegian-Norway', I could set up 
lang="nb-NO_no-NO'. (Usually lang="nb_no" should be enough.)

If this group specify that listformat, then we don't need to bother HTML 
and XML working group(s) so much.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 01:01:10 +0200

Peter Constable 2008-05-06 20.49:
> > Leif Halvard Silli Sent: Tuesday, May 06, 2008 1:54 AM
>
> > How about this change:
>
> [ ... see reworked version below ...  ]
>   

How about seting up a list of languages? For example xml:lang="cmn,zh". 
Here is new text variant, were I take tat in. See explanation beneath:

    In general, use the most specific subtag to form the language tag.
    When a macrolanguage tag has to be used, for example because
    compatibility issues forbids a complete switch to the more spesific
    language subtag, then it is RECOMMENDED to use it as part of a list
    of language tags. If the application does not permit tag lists, then
    consider nesting.  In both tag lists and in nesting it is
    RECOMMENDED to mention the macrolanguage tag after the spesific
    language subtag(s).  For mark-up languages, the recommended list
    format for compatibility reasons uses the '_' (underscore) character
    as separator. If compatibility forbids lists or nesting, then the
    macrolanguage tag MAY be used alone.

    Examples of language list and nesting:

    HTTP: type text/html, language cmn,zh
    HTML: <html lang=cmn_zh>
    Nesting, HTML: <html lang=yue><body lang=zh>text</div></div>

    It is RECOMMENDED that macrolanguage tags never are used to discern
    one particular encompassed language from a another encompassed
    language of the same macrolanguage. For example, even if it is
    technically possible, it is not meaningful to discern between
    Chinese (zh) and Cantonese (yue), if one really means to discern
    between Mandarin (cmn) and Cantonese (yue). The workaround is to use
    a tag list or nesting for the language one wishes to associate the
    macrolanguage tag with.

    Thus Mandarine, Chinese (cmn,yu) versus Cantonese (yue). Or Bokmål,
    Norwegian (nb,no) versus Nynorsk (nn) rather than Norwegian (no)
    versus Nynorsk (nn).

I think what you are missing is a way to tag the same resource with two 
tags. We discussed it on w3-l18n allready 
<http://www.w3.org/mid/[email-masked]>. And I think the 
compatible way is to use the underscore character as separator between each.

The draft allready says that, quote "a list of languages (such as 
Content-Language" should be used instead of the 'mul' tag. But the draft 
does not seem to have a note were it, in general, recommends that 
applications permits "list of languges" to be used, and neither does it 
recommend a list format. The macrolanguage issue is an issue which 
especially the for such lists urgent.

I think that 4646bis should RECOMMEND the use of several tags when it 
comes to macrolangue tags. Thus, when and if it comes to macrolanguage 
tags, one should *always* RECOMMEND using the macrolanguage as part of a 
list of the encompassed languages one has in mind.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 7 May 2008 16:12:04 -0600

Hi -

As a technical contributor...

> From: "Leif Halvard Silli" <[email-masked]>
> To: "Peter Constable" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Wednesday, May 07, 2008 5:01 PM
...
> list of the encompassed languages one has in mind.
...

I think that's really beyond our scope as a working group.

What might be an appropriate syntax for a list of language tags
will depend very much on the protocol or file format in which the
list is use.  For example, an appropriate syntax in ASN.1 BER would
be completely different from what might make sense in an XML-based
syntax.

Randy

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 7 May 2008 16:29:31 -0700

> use of macrolanguage tags is to allways used them in a list of language

That doesn't make sense. For starters, frequently one CANNOT use a list. Your notes below about HTML and XML are not correct in that regard, at least for the lang/xml:lang attributes, xsd:language data type, and so forth.

> macrolangauge. Because, macrolanguage tags are like 'mul', only  more

There are many (I suspect it's actually most) cases where either the macrolanguage or the enclosed language (but not both) are the right choice. Further, putting multiple tags on content is even more confusing than having multiple tags to choose from. What does it say to tag something "fr,en" (for example)?

I agree that we should provide good guidance on tag choice, especially when it comes to the vexing problem of overlapping subtags (as macro/enclosed languages provide). But the tag choice problem is ever with us--more so, now that we have 7000 more primary language subtags to choose between.

> the '_' (underscore) between each language tag.

Again, I'm not sure what you're referring to here, but I think you'll find that defining lists is out of scope... and doesn't necessarily solve the problem you're trying to solve.

Regards,

Addison

Addison Phillips
Chair -- W3C Internationalization WG

---

From: Shawn Steele <[email-masked]>
Date: Wed, 7 May 2008 16:48:29 -0700

> What does it say to tag something "fr,en" (for example)?

That it's a French/English dictionary, or a DVD with French & English tracks, or a translation where the original is provided as well, or...?

I agree that in most cases a list isn't the right choice, but in some cases I think its very interesting.  I don't think that we should define a MUST list use, but I do think its appropriate to mention that some people may want lists.

- Shawn

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 7 May 2008 17:01:04 -0700

See my other note.

As an indicator of intended audience a list can have meaning. But not:

<p xml:lang="fr,en">Which language should be used to process this sentence?</p>

Lists are sometimes appropriate. But sometimes they are not.

Addison Phillips

> From: Shawn Steele [mailto:[email-masked]]
> Sent: Wednesday, May 07, 2008 4:48 PM
> To: Phillips, Addison; Leif Halvard Silli; John Cowan
> That it's a French/English dictionary, or a DVD with French & English
> tracks, or a translation where the original is provided as well, or...?
>
> I agree that in most cases a list isn't the right choice, but in some
> cases I think its very interesting.  I don't think that we should
> define a MUST list use, but I do think its appropriate to mention that
> some people may want lists.
>
> - Shawn

---

From: Shawn Steele <[email-masked]>
Date: Wed, 7 May 2008 17:18:37 -0700

That's why I said that we shouldn't use MUST :)  But I think it is appropriate to mention lists for those cases where it might be useful.  Obviously your example is not one of those.  Tagging of DVDs might be.

- Shawn

-----Original Message-----
From: Phillips, Addison [mailto:[email-masked]]
Sent: Wednesday, May 07, 2008 5:01 PM
To: Shawn Steele; Leif Halvard Silli; John Cowan
Cc: LTRU Working Group; Doug Ewell
Subject: RE: [Ltru] I'm really confused by chinese in 3066bis

See my other note.

Addison Phillips

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 7 May 2008 17:25:01 -0700

We sometimes don't know exactly what the content is. If you are the author
presumably you know, but if you aren't, you might not. So it does make sense
to say:

<data xml:lang="xtg,xcg">{this is some content that is either Cisalpine
Gaulish or Transalpine Gaulish, or maybe a mixture and I can't figure out
which.}</data>

or even

<data xml:lang="en,fr">chat</data> <!--from language detection, we know that
one of these is true, but we don't know more-->

Using lists both on the query side and on the data side can make sense. On
either side it could be much more precise than the granularity, for example,
of the language collections, or than the use of the almost pointless "mul".

Mark

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 02:30:41 +0200

> That doesn't make sense. For starters, frequently one CANNOT use a list. Your notes below about HTML and XML are not correct in that regard, at least for the lang/xml:lang attributes, xsd:language…

What is the erroneous comment regarding XML and HTML? For starters, it 
validates in the W3 Validator, when it comes to HTML and XML. (While 
whitespace and commas is not permitted in the lang tag.) And otherwise, 
the HTML 4 spesification itself is not accurate, anymore, when it comes 
to language tags.

I partake in the HTML WG, and it is possible to propose things there and 
other places if necessary.

> There are many (I suspect it's actually most) cases where either the macrolanguage or the enclosed language (but not both) are the right choice.

In a context with all the chinese languages, you will tag most of them 
with the spesific tag. Only the "dominant" language will have two tags, 
namely the 'zh' and the 'cmn'.

In the Norwegian situation, I have to (usually) tag the Bokmål document 
as both 'no' an 'nb'.

Then thing is that we often have to list several languages. But nothing 
is specified about this common situation.

>  Further, putting multiple tags on content is even more confusing than having multiple tags to choose from. What does it say to tag something "fr,en" (for example)?
>   

Firstly, I spoke abot the macrolanguage tag. The thing is that when you 
write 'cmn,zn', then there supposedly is *no* conflict between those two 
tags - in that context.

You have co-written a draft which uses the prhase "a list of languages" 
(see about the'mul' tag).

> Again, I'm not sure what you're referring to here, but I think you'll find that defining lists is out of scope... and doesn't necessarily solve the problem you're trying to solve.

I guess we should specify - or suggest - _several_ list formats. For 
HTML/XML I mentioned using the underscore. It validates, and works as a 
CSS selector. For localisation, the undescore is often used as part of 
the languge tag (instead of '-'), and thuse _there_ one might use comma 
separator.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 02:35:57 +0200

> I agree that in most cases a list isn't the right choice, but in some cases I think its very interesting.  I don't think that we should define a MUST list use, but I do think its appropriate to men…

Thanks for the positive assertation!

I agree that it should not bee a MUST use list thing. Perhaps I don't 
use MUST, SHOULD etc correctly, but at least I did use RECOMMENDED with 
a purpose. :-)
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 02:46:00 +0200

> Lists are sometimes appropriate. But sometimes they are not.

It is better than 'mul' - so says at least the draft.

Again, I must make the point that there would often not be any 
processing problems if the list contains one encompassing language tag 
and one macrolanguage tag. On the contrary, it should often be 
prosessing advantages - as they would both effectively say the same thing.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2008 21:08:08 -0400

> use of macrolanguage tags is to allways used them in a list of language 

I know of no mechanism for tagging a document with more than one language tag
simultaneously (as opposed to language A embedded, <span lang="fr">comme
par expres<span>, in language B).

-- 
Even a refrigerator can conform to the XML      John Cowan
Infoset, as long as it has a door sticker       [email-masked]
saying "No information items inside".           http://www.ccil.org/~cowan
        --Eve Maler
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 03:21:58 +0200

> par expres<span>, in language B).

But as matter of fact, using the '_' (underscore) as separator 
character, it is possible to tag a HTML element in such a way that a) it 
validates in the W3 Validator, and b) it becomes usable as CSS 
seleector. The point is also not to do this "as often as possible". But 
only when needed.

<span lang="nb_no"></span>
span:lang(nb_no){color:blue;background:yellow}
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Wed, 7 May 2008 19:26:34 -0600

Leif Halvard Silli <lhs at malform dot no> wrote:

> I partake in the HTML WG, and it is possible to propose things there 
> and other places if necessary.

Then you should understand what Addison and others have been trying to 
say, that the ability to specify multiple language tags -- in, for 
example, HTTP Accept-Language -- falls within the scope of that 
protocol, *not* within the scope of BCP 47.  Our scope is to specify the 
characteristics of a language tag: what it looks like, what it means 
(within limits), how to tell if two tags match.  We do not say whether a 
field belonging to some protocol can contain a single language tag or a 
list of them; that is quite rightly the job of the designers of that 
protocol.

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 03:29:13 +0200

Randy Presuhn 2008-05-08 00.12:
> > But the draft 
> > recommend a list format. [....]
>   

OK. But the very thing that lists should be possible, should be possible 
to say clearer. As it is now, the possibility of lists is only mentioned 
en passant.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 04:26:27 +0200

Doug Ewell 2008-05-08 03.26:
> protocol, *not* within the scope of BCP 47.  Our scope is to specify 
> the characteristics of a language tag: what it looks like, what it 
> means (within limits), how to tell if two tags match.  We do not say 
> whether a field belonging to some protocol can contain a single 
> language tag or a list of them; that is quite rightly the job of the 
> designers of that protocol.

Doug, you allways pick the right parts of my message to comment on. ;-)

I agree with you. (Also a common.) But it can be good to have a picture 
of how it can be used.

And with BCP 47 properly mentioning "the list thing", there is backing 
for taking that issue up, in the relevant working groups and places.
-- 
regards
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2008 22:31:39 -0400

Leif Halvard Silli scripsit:

> validates in the W3 Validator, when it comes to HTML and XML. 

That's a bug in the validator.  In both XML and HTML, only a single BCP
47 language tag is permitted in the xml:lang (and lang) attributes.

-- 
Possession is said to be nine points of the law,                John Cowan
but that's not saying how many points the law might have.       [email-masked]
        --Thomas A. Cowan (law professor and my father)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Wed, 7 May 2008 19:53:22 -0700

> use of macrolanguage tags is to allways used them in a list of language

-1: too restrictive. There may be scenarios in which just using "zh" or "ar" may be perfectly adequate. (E.g. the librarians wouldn't be well served by that recommendation.

> macrolangauge. Because, macrolanguage tags are like 'mul', only  more

I could see something along that line being useful; I'm preparing some thoughts in another msg.

(You're ready to jump to recommendations; I need time to think through scenarios more carefully first.)

> lang="nb-NO_no-NO'.

That would *not* be acceptable in xml:lang; I'm not up on the rules for HTML lang, but my guess is it wouldn't fly there also.

I think it's out of scope for our spec to specify what applications will do. Recommend certain approaches, maybe.

Peter

---

From: Peter Constable <[email-masked]>
Date: Wed, 7 May 2008 20:02:18 -0700

> From: Leif Halvard Silli [mailto:[email-masked]]

>     as separator.

B.A.D. !! What would you expect people to make of "arb_ar"? Looks like "Arabic (Argentina)" to me.

> compatible way is to use the underscore character as separator between

The list idea may have some merits. The underscore, however, is problematic, IMO. Certain making a general recommendation about syntax in 4646bis to use underscore as a delimiter in lists would be a non-starter, IMO.

> list of the encompassed languages one has in mind.

Much too strong, IMO.

Peter

---

From: Shawn Steele <[email-masked]>
Date: Wed, 7 May 2008 20:11:06 -0700

Some systems use _ instead of - (en_US instead of en-US), so using _ to seperate a list is a really bad idea I think.

- Shawn

Sent: Wednesday, May 07, 2008 6:21 PM
To: John Cowan

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 7 May 2008 20:17:46 -0700

HTML <meta> provides it (via Content-Language), but this is more of a hint about the intended audience than part of the document structure. The 'lang' attribute (de jure) does not provide for a list, nor should it, as it would break a wide variety of language selection functionality (such as CSS :lang or XSL).

XML definitely does not supply one. You could construct an element to contain a list of languages (each with their own element).

Addison

Addison Phillips

> John Cowan
> Sent: Wednesday, May 07, 2008 6:08 PM
> To: Leif Halvard Silli

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 05:31:17 +0200

> 47 language tag is permitted in the xml:lang (and lang) attributes.

Of course, currently that is so. And may be it is OK that way.

The more important issue is contexts where an audience has to make a choice.
-- 
leif halvard silli

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 06:05:00 +0200

Phillips, Addison 2008-05-08 05.17:
> HTML <meta> provides it (via Content-Language), but this is more of a hint about the intended audience than part of the document structure. 

I never thought about that I could use a list in the META element before.

> The 'lang' attribute (de jure) does not provide for a list [...].
>   
It was a bad idea, I suppose. It really was audience tagging I was after.

John Cowan wrote:

> I know of no mechanism for tagging a document with more than one
> language tag simultaneously (as opposed to language A embedded, <span
> lang="fr">comme  par expres<span>, in language B).

In Firefox (but not Webkit), if you e.g. have this meta tag

    <meta http-equiv="Content-Language" content="ua, ru, no" >

and no further language tagging of the very HTML structure, then all 
these CSS selectors will work simultaneously: element:lang(ua), 
element:lang(ru), element:lang(no).

I wish that only the first language tag was used, and that it worked 
more or less cross browser.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Wed, 7 May 2008 22:16:44 -0600

> more or less cross browser.

Please take this to the W3C, or the browser makers.  They may be able to 
help.  We cannot; the subject is out of our scope.

I realize that as a relative newcomer to this group, you may not be 100% 
clear about what is within our scope and what is not.  Please accept the 
advice of several of us who have been trying to tell you that this 
HTML/browser business is NOT something over which we have control.  We 
don't define how HTML, HTTP, XML, etc. apply language tags, nor should 
we.

---

From: John Cowan <[email-masked]>
Date: Thu, 8 May 2008 00:53:11 -0400

> seleector. The point is also not to do this "as often as possible". But 

The W3C Validator doesn't seem to care if the language attribute value is
'+++'.  It certainly doesn't enforce the BCP 47 rules.

---

From: Peter Constable <[email-masked]>
Date: Wed, 7 May 2008 22:07:18 -0700

Perhaps it doesn't check beyond whether its CDATA? (I've suspected for a while that might be typical.)

Peter

> Sent: Wednesday, May 07, 2008 9:53 PM
> To: Leif Halvard Silli

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 08:28:42 +0200

Peter Constable 2008-05-08 07.07:
> > John Cowan Wednesday, May 07, 2008 9:53 PM
>   
> > The W3C Validator doesn't seem to care if the language attribute value
> > is '+++'.  It certainly doesn't enforce the BCP 47 rules.
>   

It do not think it treats it as CDATA. In my own testing, it only 
accepted the underscore character. I could at least  not get the '+' to 
validate in any way.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 08:40:07 +0200

Doug Ewell 2008-05-08 06.16:
> help.  We cannot; the subject is out of our scope. [...]
>   

What I said there was related to the "list" issue debated here now. 
Please read it in that context. I was giving one of several examples for 
why it might be a good idea to recommend a certain "list order", in 
order that it becomes simpler to take the message to W3C, browser 
makers, Wikipedia, whatever.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Andrew Cunningham" <[email-masked]>
Date: Fri, 9 May 2008 15:11:28 +1000

> one of these is true, but we don't know more-->

I suspect that it would be better to distinguish between audience (or
language scope) and the current text processing language. They are
different things.

Rather than

<data xml:lang="en,fr">chat</data>

something like

<data audience="en,fr"></data>

or

<data audience="en fr"></data>

since CSS3 selectors, if my understanding is correct expect space
separated lists.

could be wrong here though

some how i thing this is more a question of designing the schema or
dtd correctly rather than changing the meaning and use of xml:lang.

Differentiating between what a reference to a language or group of
languages means at anyone point.

Andrew

[email-masked]
[email-masked]