Subject: [Ltru] going back to the roots to find a solution to "zh"

From: Peter Constable <[email-masked]>
Date: Tue, 6 May 2008 09:13:02 -0700

This is mostly thinking out loud: I don't arrive at a solution, but I evaluate the problem in hope of understanding it better.

The source of ambiguity problems for zh/zho/chi lie in the origins of this category in ISO 639-1 and in the MARC Language Code List. (IANA registrations made in 1999 - zh-yue etc. - could also be cited, but they didn't add any complication that MARC and 639-1 hadn't already brought to the table; they only introduced the extlang construct.) In thinking about how to find a solution to our "zh" problems, I thought I'd consider that in those original contexts.

ISO 639-1 was created by terminologists. They may have had various applications in mind, but it's reasonable to assume terminology applications are representative. For them, "zh" would have meant Mandarin: they didn't differentiate Mandarin from Cantonese or other languages, and just said "Chinese", but in practice they were always dealing with Mandarin terms. Thus, when creating terminologies for food commodities or aircraft components or whatever, the "zh" term for 'sugar' (etc.) would be a Mandarin term.

The MARC Language Code List was the source for ISO 639-2, and was created by librarians. They explicitly used the "Chinese" category ambiguously: that is, they explicitly state that the ID for "Chinese" should be used for either Cantonese or Mandarin.

Bringing these two groups with their respective applications together results in the same kinds of problem we are facing: the librarians give "zh" its generic semantic, the terminologists give it the specific semantic, and all are using the single category in spite of that difference.

I'll pause briefly here to consider how those original two groups relate to our full range of application scenarios. Linguists, translation/localization and speech applications probably side with the terminology model in which the Mandarin/Cantonese distinction is really important. (E.g., localization of software resources is closely related to terminology, and "zh" would be specifically Mandarin; a speech synthesis app needs to differentiate Mandarin from Cantonese, as does a lexicographer.) But general content authors and publishers of first-language content probably side with the librarians who just treat everything as "Chinese" -- most books, Web pages, etc. are tagged "zh" whether they are Mandarin, Cantonese, Min Nan, etc.

Now, suppose the terminologist decided one day that they needed terminologies in both Mandarin and Cantonese: what would they do? Well, for them, "zh" has always been used to mean Mandarin, so the most straight-forward solution would be to decide that "zh" has always meant specifically Mandarin, and to add a new ID for Cantonese. But, since they've decided to use a single coding scheme together with the librarians, they'll check with them before implementing anything.

Now, for the librarian, "zh" has always had generic meaning. If the librarians decide that they need to start differentiating Mandarin and Cantonese, the terminologists' solution doesn't work for them at all. Instead, what would work for them is to create IDs for each distinct language, including Mandarin, and to start adding the additional IDs to their records. So, yesterday record 18736942 was tagged as "zh", but tomorrow it will be tagged "zh, cmn"; and a different record, 39028148 was tagged "zh", but going forward it will be tagged "zh, yue". Their query applications still allow users to search for "Chinese" records, matching on "zh", but users can start to search for "Mandarin" (cmn) or "Cantonese" (yue) records as well. Of course, if a user searches for "Cantonese" but a given record hasn't yet been updated, then that record can't be returned. The mitigation is that, when the user decides to search for "Cantonese", they are given the suggestion that they may also want to consider searching for "Chinese" records. And what if they search for "Mandarin": should they just automatically get every "zh" record? No! The same approach should be used: they should get every cmn record, but given the suggestion to search for Chinese.

Coming back to the terminologists, does the librarians' solution work for them? Well, they could double-tag records going forward; an ambiguous "Chinese" category has never been useful for them, and still isn't, but they can always ignore that. But they have a deeper problem in that all their existing records are really Mandarin, yet are tagged with "zh" which they are being asked to treat as ambiguous. They'd prefer to treat "zh" as synonymous with "cmn", and just use the former ("zh") alongside "yue" etc. Yet, that's a problem when it comes to interoperating with the librarians. They're faced with two options: either find a solution to this dual usage of "zh", or abandon any hope of interoperation with the librarians.

Of course, that's the general problem we're facing: we must find a solution to the dual usage or "zh" or abandon any possibility of allowing "zh" to have its generic meaning, yet existing usage seems to imply that the latter isn't an option, so a solution to the dual usage is essential - but we're at a loss as to how to solve it.

I'll stop here. Maybe I'll come up with some bright idea for a way forward, but maybe not.

Peter

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 6 May 2008 10:41:42 -0700

I have tried valiantly to prune my own version of the below down the past two days, but Peter does an excellent job here.

I think the way forward is to remove Chinese from the list of languages where an enclosing language is dominant and treat it separately. We already have text in place for this purpose, as Chinese is well-known to be a special case. The recommendation to consider macrolanguage in fallback can be expanded with a recommendation to use ‘cmn’ in place of ‘zh’ where appropriate, although noting that ‘zh’ remains a viable and accurate tag for many purposes.

Addison

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Peter Constable
Sent: Tuesday, May 06, 2008 9:13 AM
To: LTRU Working Group
Subject: [Ltru] going back to the roots to find a solution to "zh"

This is mostly thinking out loud: I don’t arrive at a solution, but I evaluate the problem in hope of understanding it better.

The source of ambiguity problems for zh/zho/chi lie in the origins of this category in ISO 639-1 and in the MARC Language Code List. (IANA registrations made in 1999 – zh-yue etc. – could also be cited, but they didn’t add any complication that MARC and 639-1 hadn’t already brought to the table; they only introduced the extlang construct.) In thinking about how to find a solution to our “zh” problems, I thought I’d consider that in those original contexts.

ISO 639-1 was created by terminologists. They may have had various applications in mind, but it’s reasonable to assume terminology applications are representative. For them, “zh” would have meant Mandarin: they didn’t differentiate Mandarin from Cantonese or other languages, and just said “Chinese”, but in practice they were always dealing with Mandarin terms. Thus, when creating terminologies for food commodities or aircraft components or whatever, the “zh” term for ‘sugar’ (etc.) would be a Mandarin term.

The MARC Language Code List was the source for ISO 639-2, and was created by librarians. They explicitly used the “Chinese” category ambiguously: that is, they explicitly state that the ID for “Chinese” should be used for either Cantonese or Mandarin.

Bringing these two groups with their respective applications together results in the same kinds of problem we are facing: the librarians give “zh” its generic semantic, the terminologists give it the specific semantic, and all are using the single category in spite of that difference.

I’ll pause briefly here to consider how those original two groups relate to our full range of application scenarios. Linguists, translation/localization and speech applications probably side with the terminology model in which the Mandarin/Cantonese distinction is really important. (E.g., localization of software resources is closely related to terminology, and “zh” would be specifically Mandarin; a speech synthesis app needs to differentiate Mandarin from Cantonese, as does a lexicographer.) But general content authors and publishers of first-language content probably side with the librarians who just treat everything as “Chinese” -- most books, Web pages, etc. are tagged “zh” whether they are Mandarin, Cantonese, Min Nan, etc.

Now, suppose the terminologist decided one day that they needed terminologies in both Mandarin and Cantonese: what would they do? Well, for them, “zh” has always been used to mean Mandarin, so the most straight-forward solution would be to decide that “zh” has always meant specifically Mandarin, and to add a new ID for Cantonese. But, since they’ve decided to use a single coding scheme together with the librarians, they’ll check with them before implementing anything.

Now, for the librarian, “zh” has always had generic meaning. If the librarians decide that they need to start differentiating Mandarin and Cantonese, the terminologists’ solution doesn’t work for them at all. Instead, what would work for them is to create IDs for each distinct language, including Mandarin, and to start adding the additional IDs to their records. So, yesterday record 18736942 was tagged as “zh”, but tomorrow it will be tagged “zh, cmn”; and a different record, 39028148 was tagged “zh”, but going forward it will be tagged “zh, yue”. Their query applications still allow users to search for “Chinese” records, matching on “zh”, but users can start to search for “Mandarin” (cmn) or “Cantonese” (yue) records as well. Of course, if a user searches for “Cantonese” but a given record hasn’t yet been updated, then that record can’t be returned. The mitigation is that, when the user decides to search for “Cantonese”, they are given the suggestion that they may also want to consider searching for “Chinese” records. And what if they search for “Mandarin”: should they just automatically get every “zh” record? No! The same approach should be used: they should get every cmn record, but given the suggestion to search for Chinese.

Coming back to the terminologists, does the librarians’ solution work for them? Well, they could double-tag records going forward; an ambiguous “Chinese” category has never been useful for them, and still isn’t, but they can always ignore that. But they have a deeper problem in that all their existing records are really Mandarin, yet are tagged with “zh” which they are being asked to treat as ambiguous. They’d prefer to treat “zh” as synonymous with “cmn”, and just use the former (“zh”) alongside “yue” etc. Yet, that’s a problem when it comes to interoperating with the librarians. They’re faced with two options: either find a solution to this dual usage of “zh”, or abandon any hope of interoperation with the librarians.

Of course, that’s the general problem we’re facing: we must find a solution to the dual usage or “zh” or abandon any possibility of allowing “zh” to have its generic meaning, yet existing usage seems to imply that the latter isn’t an option, so a solution to the dual usage is essential – but we’re at a loss as to how to solve it.

I’ll stop here. Maybe I’ll come up with some bright idea for a way forward, but maybe not.

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 6 May 2008 11:01:53 -0700

The situation is messy, and part of the problem is that ISO 639 was
dramatically underspecified. My interpretation, before I got into this
arena, was that when the standard say X, it was short for "Standard X". Thus
it was perfectly reasonable to make the following change:

de - German  (= Standard German, Hochdeutsch)
//later add
gsw - Swiss German
frk - Frankish German
sxu - Saxon German
...

The code de really meant Standard German, so adding a new code gsw is fine.
It is a different language, so we are not narrowing de. We would NOT make de
a macrolanguage and add gsw and then *hde (Standard German). We recognize
that past Swiss German content was tagged with de since that was the 'best'
available code, but we recommend that people switch.

*I was really surprised to see a contrary tack taken with ar. *I would have
expected it to behave just like German.

ar - Arabic (= Standard Arabic)
later add
pga - Sudanese Creole Arabic, etc.

But *don't* add arb = Standard Arabic, because we already have one (ar =
ara).

Water under the bridge, but had this approach been taken with Arabic,
Chinese, and so on, users of the standard would not be faced with a
difficult situation.

Mark

On Tue, May 6, 2008 at 9:13 AM, Peter Constable <[email-masked]>
wrote:

>  This is mostly thinking out loud: I don't arrive at a solution, but I
> evaluate the problem in hope of understanding it better.
>
>
>
> The source of ambiguity problems for zh/zho/chi lie in the origins of this
> category in ISO 639-1 and in the MARC Language Code List. (IANA
> registrations made in 1999 – zh-yue etc. – could also be cited, but they
> didn't add any complication that MARC and 639-1 hadn't already brought to
> the table; they only introduced the extlang construct.) In thinking about
> how to find a solution to our "zh" problems, I thought I'd consider that in
> those original contexts.
>
>
>
> ISO 639-1 was created by terminologists. They may have had various
> applications in mind, but it's reasonable to assume terminology applications
> are representative. For them, "zh" would have meant Mandarin: they didn't
> differentiate Mandarin from Cantonese or other languages, and just said
> "Chinese", but in practice they were always dealing with Mandarin terms.
> Thus, when creating terminologies for food commodities or aircraft
> components or whatever, the "zh" term for 'sugar' (etc.) would be a Mandarin
> term.
>
>
>
> The MARC Language Code List was the source for ISO 639-2, and was created
> by librarians. They explicitly used the "Chinese" category ambiguously: that
> is, they explicitly state that the ID for "Chinese" should be used for
> either Cantonese or Mandarin.
>
>
>
> Bringing these two groups with their respective applications together
> results in the same kinds of problem we are facing: the librarians give "zh"
> its generic semantic, the terminologists give it the specific semantic, and
> all are using the single category in spite of that difference.
>
>
>
> I'll pause briefly here to consider how those original two groups relate
> to our full range of application scenarios. Linguists,
> translation/localization and speech applications probably side with the
> terminology model in which the Mandarin/Cantonese distinction is really
> important. (E.g., localization of software resources is closely related to
> terminology, and "zh" would be specifically Mandarin; a speech synthesis app
> needs to differentiate Mandarin from Cantonese, as does a lexicographer.)
> But general content authors and publishers of first-language content
> probably side with the librarians who just treat everything as "Chinese" --
> most books, Web pages, etc. are tagged "zh" whether they are Mandarin,
> Cantonese, Min Nan, etc.
>
>
>
> Now, suppose the terminologist decided one day that they needed
> terminologies in both Mandarin and Cantonese: what would they do? Well, for
> them, "zh" has always been used to mean Mandarin, so the most
> straight-forward solution would be to decide that "zh" has always meant
> specifically Mandarin, and to add a new ID for Cantonese. But, since they've
> decided to use a single coding scheme together with the librarians, they'll
> check with them before implementing anything.
>
>
>
> Now, for the librarian, "zh" has always had generic meaning. If the
> librarians decide that they need to start differentiating Mandarin and
> Cantonese, the terminologists' solution doesn't work for them at all.
> Instead, what would work for them is to create IDs for each distinct
> language, including Mandarin, and to start adding the additional IDs to
> their records. So, yesterday record 18736942 was tagged as "zh", but
> tomorrow it will be tagged "zh, cmn"; and a different record, 39028148 was
> tagged "zh", but going forward it will be tagged "zh, yue". Their query
> applications still allow users to search for "Chinese" records, matching on
> "zh", but users can start to search for "Mandarin" (cmn) or "Cantonese"
> (yue) records as well. Of course, if a user searches for "Cantonese" but a
> given record hasn't yet been updated, then that record can't be returned.
> The mitigation is that, when the user decides to search for "Cantonese",
> they are given the suggestion that they may also want to consider searching
> for "Chinese" records. And what if they search for "Mandarin": should they
> just automatically get every "zh" record? No! The same approach should be
> used: they should get every cmn record, but given the suggestion to search
> for Chinese.
>
>
>
> Coming back to the terminologists, does the librarians' solution work for
> them? Well, they could double-tag records going forward; an ambiguous
> "Chinese" category has never been useful for them, and still isn't, but they
> can always ignore that. But they have a deeper problem in that all their
> existing records are really Mandarin, yet are tagged with "zh" which they
> are being asked to treat as ambiguous. They'd prefer to treat "zh" as
> synonymous with "cmn", and just use the former ("zh") alongside "yue" etc.
> Yet, that's a problem when it comes to interoperating with the librarians.
> They're faced with two options: either find a solution to this dual usage of
> "zh", or abandon any hope of interoperation with the librarians.
>
>
>
> Of course, that's the general problem we're facing: we must find a
> solution to the dual usage or "zh" or abandon any possibility of allowing
> "zh" to have its generic meaning, yet existing usage seems to imply that the
> latter isn't an option, so a solution to the dual usage is essential – but
> we're at a loss as to how to solve it.
>
>
>

-- 
Mark

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 6 May 2008 12:30:33 -0600

Hi -

As a technical contributor...

> From: "Peter Constable" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
> Sent: Tuesday, May 06, 2008 10:13 AM
...
> Of course, that's the general problem we're facing: we must find a solution
> to the dual usage or "zh" or abandon any possibility of allowing "zh" to have
> its generic meaning, yet existing usage seems to imply that the latter isn't
> an option, so a solution to the dual usage is essential - but we're at a loss
> as to how to solve it.
...

This points to the "soft underbelly" of "tag wisely" - the assumption that
the tagger can reasonably anticipate how the "consumers" of the tag
will want to use that information.

In retrospect, I think it would have been better to have taken the route of
   zh -> some kind of Chinese, likely (but not guaranteed) to be Mandarin
   zh-cmn -> Chinese, specifically Mandarin
   ar -> some kind of Arabic, likely (but not guaranteed) to be Standard Arabic
   ar-arb -> Arabic, specificall Standard Arabic
   de -> some kind of German, overwhelmingly likely (but not guaranteed) to be Hochdeutsch
   de-*hde -> German, specifically Hochdeutsch

Even recognizing that reasoning about "language X is some kind of Y" can be
horribly fuzzy, this would still be better aligned with a "principle of least
astonishment" for folks trying to understand the specification, trying to tag data,
or trying to formulate a query.  Yes, huge amounts of data might end up being tagged
less precisely than we might like, but at least they'd still be tagged accurately.

Randy

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 6 May 2008 20:44:07 +0100

Randy wrote:

In retrospect, I think it would have been better to have taken the route of
   zh -> some kind of Chinese, likely (but not guaranteed) to be Mandarin
   zh-cmn -> Chinese, specifically Mandarin
   ar -> some kind of Arabic, likely (but not guaranteed) to be Standard
Arabic
   ar-arb -> Arabic, specificall Standard Arabic
   de -> some kind of German, overwhelmingly likely (but not guaranteed) to
be Hochdeutsch
   de-*hde -> German, specifically Hochdeutsch

I agree wholeheartedly!

Why can we not take this route.  It is far far better and less troublesome
as well as more in line with reality (IMHO)

Best

Debbie

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Randy Presuhn
> Sent: 06 May 2008 19:31
> To: LTRU Working Group
> Subject: Re: [Ltru] going back to the roots to find a solution to "zh"
>
> Hi -
>
> > solution to the dual usage or "zh" or abandon any possibility of
> > allowing "zh" to have its generic meaning, yet existing
> usage seems to
> > imply that the latter isn't an option, so a solution to the
> dual usage
> > is essential - but we're at a loss as to how to solve it.
> ...
>
> This points to the "soft underbelly" of "tag wisely" - the
> assumption that the tagger can reasonably anticipate how the
> "consumers" of the tag will want to use that information.
>
> Even recognizing that reasoning about "language X is some
> kind of Y" can be horribly fuzzy, this would still be better
> aligned with a "principle of least astonishment" for folks
> trying to understand the specification, trying to tag data,
> or trying to formulate a query.  Yes, huge amounts of data
> might end up being tagged less precisely than we might like,
> but at least they'd still be tagged accurately.
>
> Randy
>

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 06 May 2008 21:50:59 +0200

Debbie Garside 2008-05-06 21.44:
> > less precisely than we might like,
> > but at least they'd still be tagged accurately

Indeed.
-- 
leif halvard silli

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 May 2008 13:47:15 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Randy Presuhn

...

It's my impression that at least part of what kept us from going that route was

(a) the scenarios in which extlang-to-macrolanguage fallback didn't provide useful results,
(b) the scenarios in which extlang might hinder useful right-truncation fallback in existing implementations because region or script are more useful than extlang, and
(c) the assumption that, if we use extlang at all, then we must use it for all macrolanguages rather than being able to specify particular cases (no cherry picking).

Peter

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 6 May 2008 14:22:44 -0600

Hi -

> Sent: Tuesday, May 06, 2008 2:47 PM
...
> (b) the scenarios in which extlang might hinder useful right-truncation fallback
> in existing implementations because region or script are more useful than extlang, and

I see (a) and (b) as being practically the same argument, since they are
both focused on kinds of right-truncation fallback.  Any heuristic that returns
something other than what the user explicitly asked for deserves a "health
warning".  The question is whether we allow a heuristic with known limitations
to override all other concerns.

> (c) the assumption that, if we use extlang at all, then we must use it for all macrolanguages
> rather than being able to specify particular cases (no cherry picking).

I don't see this as an obstacle.

BTW, I'm *not* advocating a change to the current text, but I would
count myself in the camp of "not convinced that this is the right
direction."

==========================================

As co-chair:

I'm very concerned that this issue continues to come up.  That suggests
that either our rationale isn't clear enough in the i-d, or that we've got it
wrong.  It could be that this is just one of those situations where there is
no "right" answer, and we need to just pick one.

When Peter started this thread, he gave us a useful explanation of how
two differing points of view led to our current situation.  Perhaps it might
be useful to work through the pros and cons of the alternatives from
the perspectives of important user communities, both as producers and
as consumers of tagged data, new and legacy.

Randy

---

From: Shawn Steele <[email-masked]>
Date: Tue, 6 May 2008 14:40:41 -0700

> to override all other concerns.

I agree, and also I'm concerned that regardless of the form of the tag, then 4647bis is going to have to address fallback in some way that is smarter than right-truncation.  None of the techniques we've discussed work for all apps with pure right-truncation

> I don't see this as an obstacle.

It seems to me that we're cherry picking zh no matter which direction we go, so I don't think this should be a major concern.

> as consumers of tagged data, new and legacy.

I agree.  I think that we've come to the current point through good intentions, but that the end result doesn't give us the clarity that we originally expected.  I think we need to back up a bit and look at the bigger picture rather than trying to tweak a couple paragraphs.

- Shawn

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 6 May 2008 18:59:56 -0700

Not so sure that we will get accuracy; frankly I think the introduction of
cmn, arb, etc. were well intentioned, but in retrospect a mistake. There was
no reason not to treat zh, ar, etc just like we treated de. There is no
substantive difference between zh:yue and de:gsw.

Given the existence and long usage of zh in IT, the introduction of cmn
(etc) will inevitably just end up with mistagged data, mishandled lookups,
and unsatisfied users, unless one follows the strategy that unless you have
a strong reason not to, treat cmn as a synonym for zh. (In fact, it is
simplest to remap cmn-* to zh-* on input.)

[Our best course of action for compatibility and interoperability would be
to deprecate cmn, arb, and the few others in that section, but I doubt that
we could get consensus on that, so I'd never brought it up.]

Mark

>From earlier, for context:

On Tue, May 6, 2008 at 11:30 AM, Randy Presuhn <[email-masked]>
wrote:

> solution

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2008 01:30:43 -0400

> we could get consensus on that, so I'd never brought it up.]

Indeed not.  If "zh" is to mean "Mandarin and nothing but" (and other languages are to be
considered loose or erroneous, like the use of "de" for Swiss German), then "zh" has
been narrowed, which is unacceptable.

-- 
Possession is said to be nine points of the law,                John Cowan
but that's not saying how many points the law might have.       [email-masked]
        --Thomas A. Cowan (law professor and my father)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Tue, 6 May 2008 23:04:35 -0700

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Tuesday, May 06, 2008 7:00 PM

> Not so sure that we will get accuracy; frankly I think the
> introduction of cmn, arb, etc. were well intentioned, but in
> retrospect a mistake. There was no reason not to treat zh, ar,
> etc just like we treated de. There is no substantive difference
> between zh:yue and de:gsw.

Well, speaking in relation to ISO 639-3, at least, if something were to have been done differently, it would have been to treat de as a macro-language, not to treat zh and ar as limited to the major varieties. (The initial approach Gary Simons and I took *was* to map existing ISO 639-2 entries to major varieties, but that change when I noted existing usage in the IANA registry or in MARC that really made that impossible.

Peter

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 7 May 2008 10:23:55 +0200

On Tue, May 06, 2008 at 02:22:44PM -0600,
 Randy Presuhn <[email-masked]> wrote 
 a message of 51 lines which said:

> It could be that this is just one of those situations where there is

I'm strongly in that camp (or should I say "no-camp"?) The discussion
on extlangs a few months ago was very useful because it allowed to
discover and to list all the good reasons why both solutions had
serious limits. 

That's life (it's not our fault, the world is messy), and, if we want
4646bis to be out one day, we should pick one solution, stick with it
and document its limitations (I suggest a big warning "For being
useful to users, fallback cannot be a dumb right-truncation", examples
as you wish, cmn/zh, sv-FI/fi, br/fr, de-CH/gsw...)

That's why I would not support changing again: extlangs are dead, go
without them. "cmn" is the right tag for Mandarin, but developers and
users should be aware that "zh" was, is and will be used a
lot. That's life.
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 7 May 2008 10:26:04 +0200

On Tue, May 06, 2008 at 08:44:07PM +0100,
 Debbie Garside <[email-masked]> wrote 

...
> Why can we not take this route.  

There are several reasons but one of them is that the original extlang
plan was to use ISO 639-3 macrolanguages (without cherry-picking them)
and "de" is not a macrolanguage... (may be it should but it isn't)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 7 May 2008 10:25:11 +0100

Stephane wrote:

> There are several reasons but one of them is that the
> original extlang plan was to use ISO 639-3 macrolanguages
> (without cherry-picking them) and "de" is not a
> macrolanguage... (may be it should but it isn't)
>

But nothing is set in stone wrt extlang :-)  or is it?

I missed the whole extlang debate so I don't really know all the reasons why
it was dismissed.  However, from what I can see we need to facilitate
fallback matching for *some* subtags that have ISO 639-3 codes but have
also, historically, predominantly used another subtag.  Maybe we should
concentrate on this aspect rather than extlang and macrolanguages
specifically.  You might find that by doing so we solve all the problems.
But it needs to be done with an open mind and I don't see the need to be
restrictive to facilitating fallback *just* for macrolanguages.  We could
include a "Fallback" field and devise a registration procedure to
incorporate information with regard to "known fallback relationships" that
would encompass a good deal of this.  I see it working along similar lines
as suppress script (as and when info becomes available). As I say, I  missed
pretty much the whole debate on this so maybe this has already been covered.

Best

Debbie

> From: Stephane Bortzmeyer [mailto:[email-masked]]
> Sent: 07 May 2008 09:26
> To: Debbie Garside
> Cc: 'LTRU Working Group'
> Subject: Re: going back to the roots to find a solution to "zh"
>
> On Tue, May 06, 2008 at 08:44:07PM +0100,  Debbie Garside
> <[email-masked]> wrote  a message of 89 lines which said:
>
> There are several reasons but one of them is that the
> original extlang plan was to use ISO 639-3 macrolanguages
> (without cherry-picking them) and "de" is not a
> macrolanguage... (may be it should but it isn't)
>
>
>
>

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2008 09:19:11 -0400

> But nothing is set in stone wrt extlang :-)  or is it?

The decisions about which languages are macrolanguages and which languages
are encompassed by them are made by 639-3/RA, and are in stone as far
as we are concerned.  De is not a macrolanguage and that's that.

The only thing we can change is our syntax, and that is set in concrete,
which is drying fast.

> pretty much the whole debate on this so maybe this has already been covered.

It's too big a job, and "known fallback relationships" depend on too many things,
not just language relatedness.

-- 
The Imperials are decadent, 300 pound   John Cowan <[email-masked]>
free-range chickens (except they have   http://www.ccil.org/~cowan
teeth, arms instead of wings, and
dinosaurlike tails).                        --Elyse Grasso
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: 'Stephane Bortzmeyer' <[email-masked]>
Date: Wed, 7 May 2008 15:47:55 +0200

On Wed, May 07, 2008 at 10:25:11AM +0100,
 Debbie Garside <[email-masked]> wrote 
 a message of 55 lines which said:

> I missed the whole extlang debate so I don't really know all the
> reasons why it was dismissed.

[My personal summary]

* stability issues (what if a "standalone" language becomes
encompassed or the opposite?).

* "dummy" right truncation, as allowed by extlangs, is not a good idea
for macrolanguages, anyway (two encompassed languages of the same
macrolanguage may not be mutually intelligible).

* politically, it may send the message that the encompassed language
is inferior in some way.

As Randy said here, you can also make a similar list of the good
reasons to keep extlangs. The case is really a 50-50 but we have to
decide on one solution.

> But it needs to be done with an open mind and I don't see the need
> to be restrictive to facilitating fallback *just* for
> macrolanguages.

The idea was not to facilitate fallback (a complicated thing, which
depend a lot of non-linguistical data), just to use an information
that ISO 639-3 gives us (macrolanguages).

In the current 4646bis model, "smart" fallback is not handled (because
it requires a lot of outside information).

> We could include a "Fallback" field and devise a registration
> procedure to incorporate information with regard to "known fallback
> relationships" that would encompass a good deal of this.

This is a huge work, compared to with Suppress-Script discussions is
peanuts :-)

To mention again typical examples (my examples, criticisms welcome):

"fr" (french) is a good fallback for "br" (breton), not because the
two languages are related (they are quite distinct) but because
Brittany is in France and, since the 19th century, there are no longer
breton speakers that do not speak french.

"fi" (finnish) is probably a good fallback for "xx-FI" because finnish
is the most common language in Finland so someone who requests "sv-FI"
can probably read finnish. But "fi" is not a good fallback for "sv"
alone.

In writing, "arb" is a good fallback for every encompassed language of
the macrolanguage "ar" (arabic). But it is not true in spoken form.

These examples give an idea of the amount of information required.

---

From: Peter Constable <[email-masked]>
Date: Wed, 7 May 2008 07:08:24 -0700

> John Cowan

> > We could
> > would encompass a good deal of this...

> not just language relatedness.

+1

Peter

---

From: Peter Constable <[email-masked]>
Date: Wed, 7 May 2008 08:13:56 -0700

> 'Stephane Bortzmeyer'

> encompassed or the opposite?).

Wasn't an issue if we used extlang only for pre-existing cases, not new cases.

> macrolanguage may not be mutually intelligible).

An issue only for the cases in which the macro-language ID has been used for some dominant, encompassed language: content would be tagged with (and hence matchable against) the macro-language ID only if there were some conventional form of linguistic expression it was associated with. That would happen only if it were associated with a dominant encompassed language, or if there were some variety of common communication (à la Chinese or African cases Don has alluded to). But in both those scenarios, the fallback probably *would* provide intelligible results for both.

> * politically, it may send the message that the encompassed language

It seems to me that would only arise if the macro-language ID is particularly associated with one dominant variety -- and that's true whether extlang is used or not. Thus, if we ended up using zh for Mandarin and either yue or zh-yue for Cantonese, that attitude might be conveyed.

I think there were a few other reasons cited:

* matching processes need to be made more complex / right-truncation doesn't work as expected in some scenarios -- an issue for tags with extlang and script or region subtags

The counter-argument made was that matching processes need to be made more complex without extlang in order to capture useful fallback associations between encompassed and macro-language IDs.

The rebuttal to that was that the whole point of extlang was to facilitate right-truncation matching involving existing usage of macro-language IDs, yet extlang breaks some desired right-truncation behaviours.

I'd note that limiting extlang to certain, pre-existing cases such as zh would cover the bulk of cases of concern, though as we know zh is a case for which script or region subtags may have particular significance.

* Not all macro-languages in ISO 639-3 are useful for matching and fallback.

Not an issue if not all macro-language IDs are used with extlangs.

* Fallback requirements are more involved than what extlang can support.

True; but extlang wasn't proposed as a general fallback solution, and the issues exist with or without extlang.

> reasons to keep extlangs. The case is really a 50-50 but we have to

That's mostly what it amounted to: a compelling case couldn't be made either way, and those arguing for extlang gave up sooner than those arguing against extlang.

Peter

---

From: Shawn Steele <[email-masked]>
Date: Wed, 7 May 2008 08:54:53 -0700

> * politically, it may send the message that the encompassed language

Politically saying that Mandarin should be tagged Chinese (and therefore implying that Chinese is Mandarin), and then suggesting that Cantonese may want to fall back to Chinese is much worse than extlang ever was.

- Shawn

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 01:07:23 +0200

Peter Constable 2008-05-07 08.04:
> Well, speaking in relation to ISO 639-3, at least, if something were to have been done differently, it would have been to treat de as a macro-language, not to treat zh and ar as limited to the major varieties.  [ ... snip ... ]

Agree.

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 01:31:17 +0200

'Stephane Bortzmeyer' 2008-05-07 15.47:
> May 07, 2008 at 10:25:11AM +0100 Debbie Garside [...] said:
>

I start to realise that the extlang issue is dead, and I might also 
agree that it was not a perfect approach. I think we should instead see 
the macrolanguage tags in relationship to the 'mul' tag (since the draft 
says that a list format should be used instead of 'mul') and specify a 
list format, because that is obviously what is needed. For mark-up 
languages, one could use the '_' (underscore) character as separator and 
write lang="cmn_zh" in order to tag it with two tags simultaneously, 
when that is needed.

That said, and as a speaker of a "microlanguage", I have to say that of 
all the arguments against extlang, the above "political" argument is the 
most ill taken an most illogical. The opposite is true: It lifts the 
extlang language to a higher status, so to speak. For instance, if 
de-gsw was allowed, then it would mean that Swiss German was formally 
counted in as a German language. It would also mean that de-ger was 
right. And this would place all the encompassed languages of a 
macrolanguage on a equal footing. Perhaps the macrolanguage would feel 
that as a degradation, though - *that* would be a much more real issue.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 01:42:21 +0200

> Wasn't an issue if we used extlang only for pre-existing cases, not new cases.

It would also not be an issue if we set up a list format. Or, well, if 
'cmn' was removed from 'zh', then that would be a problem if you had 
tagged it as 'cmn,zh'.

But assuming that the real problems will consist of "standalone" 
languages being "promoted" to macrolanguage, not that it is is "demoted" 
from that status, then, if for example "de" became a macrolanguage tag, 
then documents tagged as "de" would not be incorrectly tagged, but 
simply not as presice as possible. To be presice and backward 
compatible, one would have to start to tag all Standard German documents 
with the list "ger,de".
-- 
leif halvard silli

---

From: Shawn Steele <[email-masked]>
Date: Wed, 7 May 2008 16:44:48 -0700

Peter and I and a few others here at Microsoft chatted a bit, and our current thinking is that zh has to have a distinct meaning from cmn.  Existing "zh" data isn't necessarily Mandarin, and requiring Mandarin to be tagged more generically would mean that one would never know if "zh" was generic Chinese or Mandarin.

We are also happy with the "cmn" and "yue" codes as currently drafted (no extlang).

My concerns regarding fallback are probably solvable by "lists".  If the user specifies "cmn" and also "zh" as their language, then I can find cmn or zh resources.  Similarly if a document is tagged as cmn and also zh, then it is more likely to be found by legacy software.

Which leads me to thinking of lists in general.  What I like about extlang was the association of "zh" with "cmn" for possible fallback purposes.  It seems that Karen has similar concerns about the text content of her media with the zh-cmn preference that lets her know it is Chinese as well as Mandarin.  If the content could be tagged with both zh & cmn (dueling tags ;-), then perhaps her needs are met.

I think that there are many cases were lists are interesting.  In the Arabic cases, someone might want the acq,ar behavior for some resources because they were particularly interested in those users, but may also have more general ar resources for the rest of the Arabic speakers.  (I'm shooting for the general idea, I don't know that much about the Arabic differences.)

Similarly I could easily see a French/English dictionary as needing to be tagged "fr,en".  I've seen books with the translations in columns next to the original text.  I could easily imagine looking in a library to find a DVD that has en & fr & es voice and additionally en, fr, es, de & jp subtitles.  The DVD should maybe be tagged with all the tags, but the tracks themselves would only need their specific tag.

So I think that its worth a section in the draft to discuss language lists and their potential value for implementers, particularly those building other standards off of RFC 4646bis.  I don't think there should be MUSTard in that section, but I think it would be helpful to those trying to figure out how to apply RFC 4646bis to their specific needs.

Summary:
* We're OK with cmn & yue
* zh and cmn have to have different meanings
* Its worth thinking about cases when data or queries need multiple language tags.

- Shawn

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 7 May 2008 16:58:49 -0700

What you are describing has been described by the W3C Internationalization working group. We have tutorial material on the difference between the "content processing language" and the "language of the intended audience".

A list of language tags can be useful for the later. For the former, most data formats (for example, XML) force you to specify a single language per "segment"---which is really what you want for processing. For audience, you might use a list. Although, frankly, most content serves a specific language audience. Not "all", must "most".

Addison

Addison Phillips

> Shawn Steele
> Sent: Wednesday, May 07, 2008 4:45 PM
> To: Peter Constable
> Cc: LTRU Working Group
> Peter and I and a few others here at Microsoft chatted a bit, and our
> current thinking is that zh has to have a distinct meaning from cmn.
> Existing "zh" data isn't necessarily Mandarin, and requiring Mandarin
> to be tagged more generically would mean that one would never know if
> "zh" was generic Chinese or Mandarin.
>
> My concerns regarding fallback are probably solvable by "lists".  If
> the user specifies "cmn" and also "zh" as their language, then I can
> find cmn or zh resources.  Similarly if a document is tagged as cmn and
> also zh, then it is more likely to be found by legacy software.
>
> Which leads me to thinking of lists in general.  What I like about
> extlang was the association of "zh" with "cmn" for possible fallback
> purposes.  It seems that Karen has similar concerns about the text
> content of her media with the zh-cmn preference that lets her know it
> is Chinese as well as Mandarin.  If the content could be tagged with
> both zh & cmn (dueling tags ;-), then perhaps her needs are met.
>
> I think that there are many cases were lists are interesting.  In the
> Arabic cases, someone might want the acq,ar behavior for some resources
> because they were particularly interested in those users, but may also
> have more general ar resources for the rest of the Arabic speakers.
> (I'm shooting for the general idea, I don't know that much about the
> Arabic differences.)
>
> Similarly I could easily see a French/English dictionary as needing to
> be tagged "fr,en".  I've seen books with the translations in columns
> next to the original text.  I could easily imagine looking in a library
> to find a DVD that has en & fr & es voice and additionally en, fr, es,
> de & jp subtitles.  The DVD should maybe be tagged with all the tags,
> but the tracks themselves would only need their specific tag.
>
> So I think that its worth a section in the draft to discuss language
> lists and their potential value for implementers, particularly those
> building other standards off of RFC 4646bis.  I don't think there
> should be MUSTard in that section, but I think it would be helpful to
> those trying to figure out how to apply RFC 4646bis to their specific
> needs.
>
> Summary:

---

From: [email-masked]
Date: Wed, 7 May 2008 16:55:52 -0700

I am OK with this. I think it's OK to use "cmn" and "yue" on their own as 
long as the recommendation for Mandarin (written and spoken) is clearly 
"cmn"; and "zh" and "cmn" are not viewed in theory or in practice as 
synonyms. I agree very much with the reasoning expressed below. I might 
have a preference for extlang, but the solution below results in neither a 
semantic narrowing to "zh" nor conditional tag recommendations that will 
be impossible to explain.

Best regards,

Karen Broome

[email-masked] wrote on 05/07/2008 04:44:48 PM:

> Peter and I and a few others here at Microsoft chatted a bit, and 
> our current thinking is that zh has to have a distinct meaning from 
> cmn.  Existing "zh" data isn't necessarily Mandarin, and requiring 
> Mandarin to be tagged more generically would mean that one would 
> never know if "zh" was generic Chinese or Mandarin.
> 
> find cmn or zh resources.  Similarly if a document is tagged as cmn 
> and also zh, then it is more likely to be found by legacy software.
> 
> content of her media with the zh-cmn preference that lets her know 
> it is Chinese as well as Mandarin.  If the content could be tagged 
> with both zh & cmn (dueling tags ;-), then perhaps her needs are met.
> 
> I think that there are many cases were lists are interesting.  In 
> the Arabic cases, someone might want the acq,ar behavior for some 
> resources because they were particularly interested in those users, 
> but may also have more general ar resources for the rest of the 
> Arabic speakers.  (I'm shooting for the general idea, I don't know 
> that much about the Arabic differences.)
> 
> Similarly I could easily see a French/English dictionary as needing 
> to be tagged "fr,en".  I've seen books with the translations in 
> columns next to the original text.  I could easily imagine looking 
> in a library to find a DVD that has en & fr & es voice and 
> additionally en, fr, es, de & jp subtitles.  The DVD should maybe be
> tagged with all the tags, but the tracks themselves would only need 
> their specific tag.
> 
> should be MUSTard in that section, but I think it would be helpful 
> to those trying to figure out how to apply RFC 4646bis to their 
> specific needs.
> 
> Summary:

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 7 May 2008 17:02:12 -0700

I'm okay with this. What we need to do is translate that in specific text in draft-14.

I *think* what we're doing here is removing the "SHOULD" sentence about macrolanguage-used-for-dominant-enclosed-variety.

Addison

Addison Phillips

> [email-masked]
> Sent: Wednesday, May 07, 2008 4:56 PM
> To: Shawn Steele
> Cc: LTRU Working Group

---

From: Shawn Steele <[email-masked]>
Date: Wed, 7 May 2008 17:17:18 -0700

My hope was that by mentioning a list and its possible uses, then those standards which derive from RFC 4646bis may consider whether or not list-like behavior is interesting for them.  Lists certainly shouldn't be required.  There are probably many other groups that could use lists that aren't part of the W3C effort.

- Shawn

-----Original Message-----
From: Phillips, Addison [mailto:[email-masked]]
Sent: Wednesday, May 07, 2008 4:59 PM
To: Shawn Steele; Peter Constable
Cc: LTRU Working Group
Subject: RE: [Ltru] going back to the roots to find a solution to "zh"

Addison

Addison Phillips

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 7 May 2008 17:19:50 -0700

We're playing telegram here (laughing).

In the response I just sent, I point out that lists are extensively dealt with in RFC 4647, which is, after all, part of BCP 47.

We can mention lists more in 4646 too. Putting on my editor hat, though, do you have specific text to suggest? It is usually easier (and more productive) to work directly on text.

Addison

Addison Phillips

> From: Shawn Steele [mailto:[email-masked]]
> Sent: Wednesday, May 07, 2008 5:17 PM
> To: Phillips, Addison; Peter Constable
> Cc: LTRU Working Group
> My hope was that by mentioning a list and its possible uses, then those
> standards which derive from RFC 4646bis may consider whether or not
> list-like behavior is interesting for them.  Lists certainly shouldn't
> be required.  There are probably many other groups that could use lists
> that aren't part of the W3C effort.
>
> - Shawn
>
> What you are describing has been described by the W3C
> Internationalization working group. We have tutorial material on the
> difference between the "content processing language" and the "language
> of the intended audience".
>
> A list of language tags can be useful for the later. For the former,
> most data formats (for example, XML) force you to specify a single
> language per "segment"---which is really what you want for processing.
> For audience, you might use a list. Although, frankly, most content
> serves a specific language audience. Not "all", must "most".
>
> Addison
>
> Addison Phillips

---

From: Shawn Steele <[email-masked]>
Date: Wed, 7 May 2008 17:21:27 -0700

I think we mostly agree :) I'd like a mention in RFC4646, but obviously 4647 would be more appropriate.

I'll try to think of some text.

- Shawn

-----Original Message-----
From: Phillips, Addison [mailto:[email-masked]]
Sent: Wednesday, May 07, 2008 5:20 PM
To: Shawn Steele; Peter Constable
Cc: LTRU Working Group
Subject: RE: [Ltru] going back to the roots to find a solution to "zh"

Addison

Addison Phillips

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 7 May 2008 17:23:30 -0700

Okay, but maybe see if 4647 doesn't already meet the need. I would hate to open up that document to editing unless strictly necessary.

What's more immediate would be thinking about text in the section on "choice" draft-14 or possibly incorporating some of the "language of content processing" vs. "language of audience" text into the document (via reference, perhaps??)

Addison

Addison Phillips

> Sent: Wednesday, May 07, 2008 5:21 PM
> In the response I just sent, I point out that lists are extensively
> dealt with in RFC 4647, which is, after all, part of BCP 47.
>
> We can mention lists more in 4646 too. Putting on my editor hat, though,
> do you have specific text to suggest? It is usually easier (and more
> productive) to work directly on text.
>
> Addison
>
> Addison Phillips

---

From: Peter Constable <[email-masked]>
Date: Wed, 7 May 2008 17:49:48 -0700

> serves a specific language audience. Not "all", must "most".

Could you explain these two notions a little more? I'm not sure I'm fully grasping the ideas.

Thanks
Peter

---

From: Shawn Steele <[email-masked]>
Date: Wed, 7 May 2008 18:00:18 -0700

I'm thinking something like this:

4.6? Lists of Language Tags

In some applications a list of language tags is necessary, such as a user's language priority list, when multiple language apply equally to the same content, or when matching of older tags is expected.  Some example scenarios are:

* A user could specify "en, de, jp" for a language priority list as described in RFC4647bis
* A French/English, English/French dictionary could be tagged as "en, fr" to specify that it applies equally to French and English
* A DVD could be tagged as having audio content in "en, fr" and subtitles in "en, fr, es, jp".
* An English translation of a classical work could have the code "la, en".
* Mandarin content expected to interchange with existing systems could be tagged "cmn, zh".

Standards and protocols specifying this RFC MAY choose to allow lists of language tags.  Such standards and protocols MUST NOT be required to allow lists of language tags if not appropriate.

If a list of language tags is used it MUST contain well formed language tags and strongly RECOMMENDED that those tags are chosen using the Choice of Language Tag guidelines.

When a list of language tags is used, the protocols MUST specify whether or not an order is or is not implied or required for the list.

When a collection of content, such as a DVD with multiple tracks, can be tagged with multiple languages, the collection SHOULD be tagged with the appropriate language tags, and the individual pieces of content SHOULD be tagged with a single language tag from the list of the collection.

Recommendations for searching and matching of lists of language tags should be addressed in the successor to RFC 4647

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 7 May 2008 17:17:44 -0700

+1

+1

> also zh, then it is more likely to be found by legacy software.

Agreed: this is why RFC 4647 is written in terms of Language Priority Lists.

> both zh & cmn (dueling tags ;-), then perhaps her needs are met.

This would be up to the specific format. If required to choose a SINGLE tag, she will still need to choose between 'zh' (possibly Mandarin, definitely Chinese, perhaps written since written forms tend to be indeterminate) and 'cmn' (definitely Mandarin). What RFC 4646bis needs to provide is good guidance for this case.

Implementations should think in terms of RFC 4647 rather than 4646 for most operations (since frequently tag-related operations have to do with matching).

> (I'm shooting for the general idea, I don't know that much about the

It depends on the content and on the algorithm.

If you have "a??,ar" tags on all your Arabic content (where 'a??' is some local variety), a matching algorithm can find all content in any kind of Arabic. But this also means that it can fall back into the "wrong" variety if not handled carefully. For example, a request for "arb,ar" could return some content tagged "acq,ar" (because it is tagged with 'ar', after all). Probably this isn't desirable.

> but the tracks themselves would only need their specific tag.

This is the difference between audience and processing language.

Karen's problem, though (mine too: the Amazon Kindle is both a book and an audio device), is that the subtitles (text) don't necessarily match the soundtrack (audio). The subtitles on a Chinese DVD are almost certainly one of "zh-Hant" or "zh-Hans". The soundtracks are almost certainly 'cmn', 'yue' or something like that. Unless the subtitles are written in a colloquial form, you probably do NOT want to specify them as Mandarin. Books, vs. recordings of books, have the same problem. If you are a reader, Simplified vs. Traditional Chinese (zh-Han?) is most important to you. If you are listening to a book, the spoken language is primary.

> those trying to figure out how to apply RFC 4646bis to their specific

We have one: it is called "RFC 4647"

> * zh and cmn have to have different meanings

They have never not had different meanings, the SHOULD in the section on choice notwithstanding.

Addison

Addison Phillips

---

From: Shawn Steele <[email-masked]>
Date: Wed, 7 May 2008 18:08:59 -0700

> Okay, but maybe see if 4647 doesn't already meet the need.
> I would hate to open up that document to editing unless strictly necessary.

I presumed that 4647 would have to be updated if for no other reason than to address the zh/cmn fallback matching questions.

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 03:11:26 +0200

> Politically saying that Mandarin should be tagged Chinese (and therefore implying that Chinese is Mandarin), and then suggesting that Cantonese may want to fall back to Chinese is much worse than e…

I agree very much.

I therefore want to whisper about "a list of languages" format, again.

I think the thing is that macrolanguage tags for the most part should 
only be used for backward compatibility. Thus, if you want to cover all 
the languages that a macrolanguage tag (currently) covers, then you are 
RECOMMENDED to list all the languages which that tag covers. You are not 
RECOMMENDED to simply use the macrolanguage tag and hope that it works.

That should - I hope - cover what you said here:

Shawn Steele 2008-05-06 05.32:
> This sounds to me like "use zh instead of cmn because that's what was used historically."... unless its necessary to clearly distguish the macrolanguage from the dominant language variety.  If normally one tags the dominant language as the macrolanguage, how is it going to be possible to distinguish the macrolanguage from it?  Presumably cmn would clearly distinguish Mandarin, however zh can't be used to specify the Chinese macrolanguage because it is also used to mean Mandarin, in fact that's clearly the preferred meaning.
>
> So if I want to clearly distinguish the macrolanguage, how do I clearly tag it?
>   

Worldwide, or in a context?

If I understand what you meant, then I'd say you should list all the 
languages which the macrolanguage covers. So, if you have an Arabic 
resource which should be accessible to "components" which are spesificly 
Egyptian Arabic, Gulf Arabic, Saidi Arabic etc, then you should really 
tag that resource with all arabic subtags which that resource is meant 
for, in addition to the macrolanguage 'ar' tag.

This would be in line with what has been said here about never to take 
for granted any fallback from spesific language to macrolanguage.
-- 
leif halvard silli

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 04:17:06 +0200

Two notes regarding this list thing.

1. Regarding processing versus audicence tagging:

    The thing is that in HTML/CSS, the audience languge (HTTP language) 
is (halfly) permitted and (hafly) implemented as fallback language if no 
processing language tags are specified inside the document.

It was recently asked in the HTML working group if taking _full_ note of 
the HTTP tagging could be harmfull. See this message and the short thrad 
that followed: 
<http://www.w3.org/mid/[email-masked]>

And in the context of tag lists, I now understand that it could be 
dangerous for the user agent to give full heed to the HTTP/Meta language 
tag.

But at least this also shows that there is not always a clearcut 
difference between audience and processing tagging. Also, if you need to 
tag the root element with a "list" (and isn't the root element sometimes 
used for audience tracking?), then that does not mean that one cannot 
tag the BODY element etc with a processing language tag.

Howeve, regarding the possible harmfullness, if we cannot specify list 
*formats*, then at least we could say something about listing the most 
spesific language first, so that the first item of the list more 
reliably can be usd as a fallback language in User Agents and by CSS if 
the very document itself is lacking proper tagging.

2. Regarding HTTP:

    There is as we konw no problem adding several extension to a HTML 
file, so that HTTP tags that file get tagged with e.g. both English, 
French, German. Typically, in Apache, such a file would look like 
'index.en.fr.de.html'

But what would have been great was if I could say this:

    'en', 'fr', 'de' == .extension

For instance, in Apache, now they have for instance this:

    AddLanguage zh-CN .zh-cn

Now, when RFX 4646bis goes into business, there are therefore millons of 
pages with the .zh-cn extension, I suppose. Many of those pages could be 
properly and backward compatibly tagged with the list tag 'cmn,zh', in 
one chunch. But changing all those tags is massive work. Therefore, what 
we need is to say that the exention  'zh-cn' means 'cmn,zh'. Like this:

    AddLanguage cmn,zh,zh-CN .zh-cn

Then it became simple to fix that problem. The same should be done for 
Norwegian. The '.no' extension could be remappe like this

    AddLanguage nb,no .no

The problem - at least in Apache - is that the abvoe seems not work. (At 
least I could not get it to work.) But at least, if we are clear about 
the need for 'list' etc, then we can ask Microsoft and Apache and othe 
web server vendors to make it simple to tag HTML documents with "lists".

So, the RFC could recognize the need for concanating lists intoi a 
single extension.

with regards
leif halvard silli

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 05:01:06 +0200

...
> When a list of language tags is used, the protocols MUST specify whether or not an order is or is not implied or required for the list.

I think that a crucial thing to say in this RFC is whether the *first* 
tag in such a list MAY/SHOULD have a special status, or not. It would 
kind of reflect how the tag system is working if it was said that the 
first tag will often be expected to have the same degree of specificity 
- or a higher degree of specificity -  than then one(s) that follows. 
For instance, to list the 'mul' tag first would not be so helpful?

Protocols could be allowed to opt out of giving the first tag special 
heed, though.

As I said in a subsequent e-mail: In HTML, the HTTP language is used as 
fallback if HTML element(s) are lacking any  language tagging. So that 
is at least 3 protocols that must work together - HTML, CSS, HTTP. So, 
if the RFC lead the way a bit, then it would be good.

(Off topic land: In a META element in HTML, I am not aware of the 
possibility of adding lists, allthough it is supposed to reflect the 
HTTP header. (HTTP could/can read the serving langauge from the META 
element.) But perhaps it is possible to use several META elements in the 
same document - that would constitute a kind of list.)
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 7 May 2008 20:03:45 -0600

Hi -

> From: "Leif Halvard Silli" <[email-masked]>
> To: "Phillips, Addison" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Wednesday, May 07, 2008 8:17 PM
...
> the very document itself is lacking proper tagging.

In RFC 4647 I think we've said about as much as we can usefully say on this topic.
I don't think it would be helpful to repeat it.  Also, language tags and fallback have
applicability in situations other than the web content, so we should be wary of letting
that particular use case drive our thinking to the exclusion of other cases.

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 7 May 2008 20:11:20 -0600

Hi -

> To: "Shawn Steele" <[email-masked]>
> Sent: Wednesday, May 07, 2008 9:01 PM
...
> tag in such a list MAY/SHOULD have a special status, or not. 
...

I strongly disagree with the "this RFC" part of the comment.
Any text to that effect would belong in an update to RFC 4647.
We're currently working on an update to the documents
describing the syntax of tags and the procedures for managing
the registry.  The working group might ask permission to add a
new work item to its charter, but I suspect at least one of the co-chairs
would take a very dim view of additional work items.  :-)

Randy

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 7 May 2008 20:13:59 -0700

Sorry, I should have provided the link.

This is most thoroughly discussed in the tutorial "Declaring Language in XHTML and HTML":

  http://www.w3.org/International/tutorials/language-decl/

In the Internationalization Best Practices we find items that speak directly to this thread:

  http://www.w3.org/TR/i18n-html-tech-lang/#ri20030510.102829377
  http://www.w3.org/TR/i18n-html-tech-lang/#ri20040728.121358444
  http://www.w3.org/TR/i18n-html-tech-lang/#ri20040728.121940236

The last item is Best Practice 10, whose description reads:

--
Where a document contains content aimed at speakers of more than one language, use Content-Language with a comma-separated list of language tags.
--

Addison

Addison Phillips
Chair -- W3C Internationalization Core WG

> From: Peter Constable [mailto:[email-masked]]
> Sent: Wednesday, May 07, 2008 5:50 PM
> To: Phillips, Addison
> Cc: LTRU Working Group

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 7 May 2008 20:20:58 -0700

Not necessarily. It only has to change if the algorithms change. If 4646bis provides a MAY or SHOULD allowing for fallback schemes to consider the macrolanguage, implementations of Filtering or Lookup are already given the necessary latitude.

Extlang, by contrast, would probably have required editing the Matching document. RFR is not broken if you don't have extlangs. If users use lists (already in 4647) or implementations supply lists on their behalf, then the algorithms are already complete.

Admittedly, some errata might be provided, perhaps in the form of an additional RFC, but that's not strictly necessary at first blush. But again, look through it first and see if you see where it's required.

Addison

Addison Phillips

> Sent: Wednesday, May 07, 2008 6:09 PM
> I presumed that 4647 would have to be updated if for no other reason
> than to address the zh/cmn fallback matching questions.
>

---

From: Shawn Steele <[email-masked]>
Date: Wed, 7 May 2008 20:23:19 -0700

It seems to me that Filtering and Lookup works only if a language priority list contains the tags being looked for.  So if my content is "cmn" and my list only has "zh", then there's no hit.

So I guess I'm looking for wording saying that in some cases if you can't match using cmn-xx-xx-xx-xx that you could try again using zh-xx-xx-xx-xx.  Even then I'm not sure that's the best behavior.  cmn-HK may prefer to look in the order cmn-HK, zh-HK, cmn, zh.  That's really unclear....

- Shawn

________________________________________
From: Phillips, Addison [[email-masked]]
Sent: Wednesday, May 07, 2008 8:20 PM
To: Shawn Steele; Peter Constable
Cc: LTRU Working Group
Subject: RE: [Ltru] going back to the roots to find a solution to "zh"

Addison

Addison Phillips

---

From: Peter Constable <[email-masked]>
Date: Wed, 7 May 2008 20:41:16 -0700

This is going to be a bit more thinking out loud on my part.

> From: Shawn Steele

> Which leads me to thinking of lists in general...

> be tagged "fr,en"...

I'm wondering if we don't need to separate different purposes for having lists. I see three distinct cases:

1. declaring a fallback / user-preference list: this is a (possibly weighted) list of potentially-unrelated varieties, and the list is an expression of preference, not a declaration about actual content/resources

2. declaring that content/resources are mixed -- that is, they pertain to multiple, distinct varieties

3. declaring a single variety for content/resources at multiple levels of specificity

With others seeding the "list" idea, I'm starting to think our problems with zh and similar macro-languages could be solved by a combination of #1 and #3: if a user specifies that they will accept content in zh or cmn, then they will get results that were tagged either way; and if content is tagged to declare it as zh but also as cmn, then users asking for either would get it.

Returning to the start of this thread, this is basically what I suggested as the solution those in the "librarian" camp would take when they want to differentiate Mandarin and Cantonese: they'd start declaring their information objects as "zh,cmn" or "zh,yue", and suggest that users may want query for "zh" in addition to either "cmn" or "yue" if they want to catch objects that haven't yet had their metadata updated. Of course, the relevance-weighting for "zh" should probably be a lot higher if searching for "cmn" than if searching for "yue".

Addison mentioned (in a later msg) a potential issue:

> For example, a request for "arb,ar" could return some content
> tagged "acq,ar" (because it is tagged with 'ar', after all).
> Probably this isn't desirable.

This requires some added complexity in matching, but there is a potential solution: if you note that there's a generic/specific tag pair in the request and in the content, with the generics matching, then you must compare the specific values, not the matching generic value, to determine a resource match.

Now, what's this do for the "terminologist" camp?

Well, they can continue to use "zh" for Mandarin resources as long as Mandarin is all they care about. If they need to interop with "librarian" types who have started to differentiate, then fallback to "zh" could be set up on the serving side of the interaction. When it make sense -- perhaps only if and when they start to deal with Cantonese as well as Mandarin resources -- they would probably want to transition to using "cmn" for Mandarin, relying again on fallback, this time from "zh" to "cmn". (Of course, how server-side fallback should interact with user-specified fallback preferences would need to be considered.) I don't know how well this brief sketch covers all the relevant scenarios, but it's a starting point / strawman to work from.

So, part of this involves being able to declare multiple values with differing degrees of specificity on a given information object (whether a span of content to entire documents). Again, note that I am not mixing this with declaring "mul" kinds of combinations, such as "en,fr". (Leif compared pairs like "no,nb" to "mul", but I think they are not really comparable.) Of course, the big issue with this is that implementing it belongs to the application. That absolutely is appropriate, and anything RFC4646bis might say on this would need to be sufficiently circumspect: "SHOULD" would be too strong; "zh" may continue to be perfectly reasonable for some applications (indeed, MARC would insist on it). But the question we can consider is if this is a reasonable suggestion to present to applications in general, or if we can see up front that it would be so problematic in some situations as to have left some users without acceptable options or guidance for "zh" and similar cases.

Peter

---

From: Peter Constable <[email-masked]>
Date: Wed, 7 May 2008 20:48:09 -0700

> dealt with in RFC 4647, which is, after all, part of BCP 47.

But only one kind of list: in the three cases I mentioned in another msg I just sent, repeated below, Language Priority Lists represent case #1. "mul" is case #2. But there is a third potential kind of list, #3:

Peter

---

From: John Cowan <[email-masked]>
Date: Thu, 8 May 2008 00:51:10 -0400

Phillips, Addison scripsit:

> Where a document contains content aimed at speakers of more than
> one language, use Content-Language with a comma-separated list of
> language tags.

Huh.  I didn't know that was legal.  Shows what I know about HTTP (viz. a lot about etags and timestamps,
not much about anything else).

-- 
Man has no body distinct from his soul,                 John Cowan
for that called body is a portion of the soul           [email-masked]
discerned by the five senses,                           http://www.ccil.org
the chief inlets of the soul in this age.  --William Blake
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Thu, 8 May 2008 00:57:52 -0400

Peter Constable scripsit:

> Returning to the start of this thread, this is basically what
> I suggested as the solution those in the "librarian" camp would
> take when they want to differentiate Mandarin and Cantonese: they'd
> start declaring their information objects as "zh,cmn" or "zh,yue",
> and suggest that users may want query for "zh" in addition to either
> "cmn" or "yue" if they want to catch objects that haven't yet had their
> metadata updated. Of course, the relevance-weighting for "zh" should
> probably be a lot higher if searching for "cmn" than if searching for
> "yue".

That could be done not by changing the tagging, but by changing the
Web server's notion of what "zh" means (configurably, of course, not
unconditionally) so that requests for "zh" resources will find both "zh"
ones and "cmn" ones, and perhaps vice versa, depending on the needs of
the site.

-- 
One Word to write them all,             John Cowan <[email-masked]>
  One Access to find them,              http://www.ccil.org/~cowan
One Excel to count them all,
  And thus to Windows bind them.                --Mike Champion
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 7 May 2008 22:13:01 -0700

> ones and "cmn" ones, and perhaps vice versa, depending on the needs of

Yes. And I would stipulate the "depending on the needs of the site" (or application). Note that Lookup provides for this, at least somewhat, by allowing for customized defaulting rules. And we *do* have a Macrolanguage field in the registry, presumably for the use of implementations that have need of it.

Addison

Addison Phillips

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 07:31:22 +0200

Peter Constable 2008-05-08 05.41:
> > From: Shawn Steele, Wednesday, May 07, 2008 4:45 PM
> > My concerns regarding fallback are probably solvable by "lists". [...]
>   
> 1. declaring a fallback / user-preference list: [...] a (possibly weighted) list 
>   
> 2. declaring that content/resources are mixed  [...]
>   

Could usecase 2. include the use of 'mul' as one of the tags? E.g. 
'en,fr,mul'? For instance, some might be especially interested in 
multlingual material? This would also make the use of 'mul' quite 
similar in logic to the use of macrolanguage tags. See below.

> With others seeding the "list" idea, I'm starting to think our problems with zh and similar macro-languages could be solved by a combination of #1 and #3: if a user specifies that they will accept content in zh or cmn, then they will get results that were tagged either way; and if content is tagged to declare it as zh but also as cmn, then users asking for either would get it.  [...]
>   

The above is the Norwegian situation ... on the web.

> This requires some added complexity in matching, but there is a potential solution: if you note that there's a generic/specific tag pair

What about "cmn,yue,zh"? I suppose then both "cmn" and "yue" would then 
have the same relative value?

The pair thing Should work fine for e.g. Norwegian,Bokmål (nb,no), when 
I only want to reach Norwegian speaking people.

>  in the request and in the content, with the generics matching, then you must compare the specific values, not the matching generic value, to determine a resource match.
>   

It could sound as if you think that the order should not matter? 'no,nn' 
and 'nn,no'?

> Now, what's this do for the "terminologist" camp?
[...]
> So, part of this involves being able to declare multiple values with differing degrees of specificity on a given information object (whether a span of content to entire documents). Again, note that I am not mixing this with declaring "mul" kinds of combinations, such as "en,fr". (Leif compared pairs like "no,nb" to "mul", but I think they are not really comparable.)

What I actually compared was not "no,nb" to "mul"  but "no" to "mul" and 
"zh" to "mul".  'mul' = all languges are possible. 'zh' = all chinese 
languages possible. I was intrigued by the recomendation to use a list 
instead of 'mul', while at the same time there were no recommendation to 
use a list instead of macrolanguage tags - when the purpose of using the 
macrolanguage tag was to encompass all the encompassed languages. (There 
is a passage, in section 4.1, which advices against "subtags for 
language collections", but that passages is not linked to the spesific 
issue of macrolanguages.)

Also, if we only list "fr,en" then one might not know *why* it is tagged 
that way. But if one "pair" them with the "mul" tag - 'fr,en,mul' - then 
one should know better why.  For a resource in Chinese, were the content 
is mixed, then one could use 'spesific|generic|mul' - 'cmn,yue,zh,mul' - 
to signify that it is multilingual Chinese of those languages. To 
exclude the "mul" tag - "cmn,yue,zh" - means that the tags say nothing 
about whether the content is multilingual.
-- 
leif halvard silli

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 08:13:42 +0200

Phillips, Addison 2008-05-08 07.13:
> >> they'd start declaring their information objects as "zh,cmn" or
> >> "zh,yue", and suggest that users may want query for "zh" in
> >> addition to either "cmn" or "yue" if they want to catch objects
> >> that haven't yet had their metadata updated. [...]
> >
> > Web server's notion of what "zh" means  [...]
>
> Yes. And I would stipulate the "depending on the needs of the site" (or application).  [...]
>   

Web servers seems to link the tag(s) to a spesific file exensions. Thus 
web servers has double tagging: a file extension tag, and a the real 
language tag. Often those file extensions look the same as the language tag.

It is those file extensions that it should be possible to twist, to say 
that '.zh' now means not 'zh' anymore but 'cmn,zh'. That, rather than to 
say that the http tag now stands for two language tags.

Of course, the principle of "double tagging" could be used outside the 
web server scope.
-- 
leif halvard silli

---

From: Peter Constable <[email-masked]>
Date: Wed, 7 May 2008 23:26:30 -0700

Btw, in relation to "lists" for declaring dual language values on a given bit of content, and specifically the #3 case in which the "dual" values are the generic/specific pair for the "same" thing (to the extent that e.g. "zh" is the "same" as "yue" or as "nan" etc.), it occurs to me that there is one way we could be completely certain about being able to express the generic/specific "list" in a way that all BCP 47 apps could support: extlang!

;-)
Peter

> Peter Constable
> Sent: Wednesday, May 07, 2008 8:48 PM
> To: LTRU Working Group
> But only one kind of list: in the three cases I mentioned in another
> msg I just sent, repeated below, Language Priority Lists represent case
> #1. "mul" is case #2. But there is a third potential kind of list, #3:
>
> 1. declaring a fallback / user-preference list: this is a (possibly
> weighted) list of potentially-unrelated varieties, and the list is an
> expression of preference, not a declaration about actual
> content/resources
>

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 8 May 2008 09:41:23 +0100

John wrote:

> It's too big a job, and "known fallback relationships" depend
> on too many things, not just language relatedness.
>

Not for just the handful of subtags that are causing problems where there
has been historical usage of other subtags.  I am not proposing that we
create a fallback mechanism for fr to br etc.  That sort of thing is way
beyond the scope/knowledge realm of this WG. However, where there is
ambiguity in tagging I feel it is within scope to try to establish a
mechanism for dealing with such cases.  In essence we are talking about
zh/cmn but I wanted to, perhaps, establish a mechanism where such cases as
de could also benefit.

Best

debbie

> From: John Cowan [mailto:[email-masked]]
> Sent: 07 May 2008 14:19
> To: Debbie Garside
> Cc: 'Stephane Bortzmeyer'; 'LTRU Working Group'
> The decisions about which languages are macrolanguages and
> which languages are encompassed by them are made by 639-3/RA,
> and are in stone as far as we are concerned.  De is not a
> macrolanguage and that's that.
>
> The only thing we can change is our syntax, and that is set
> in concrete, which is drying fast.
>
> > reasons why it was dismissed.  However, from what I can see
> we need to
> > facilitate fallback matching for *some* subtags that have ISO 639-3
> > codes but have also, historically, predominantly used
> another subtag.
> > Maybe we should concentrate on this aspect rather than extlang and
> > macrolanguages specifically.  You might find that by doing
> so we solve all the problems.
> > But it needs to be done with an open mind and I don't see
> the need to
> > be restrictive to facilitating fallback *just* for
> macrolanguages.  We
> > could include a "Fallback" field and devise a registration
> procedure
> > to incorporate information with regard to "known fallback
> > relationships" that would encompass a good deal of this.  I see it
> > working along similar lines as suppress script (as and when info
> > becomes available). As I say, I  missed pretty much the
> whole debate on this so maybe this has already been covered.
>
> It's too big a job, and "known fallback relationships" depend
> on too many things, not just language relatedness.
>
> --

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 8 May 2008 09:31:19 +0100

Thanks

Debbie

> From: 'Stephane Bortzmeyer' [mailto:[email-masked]]
> Sent: 07 May 2008 14:48
> To: Debbie Garside
> Cc: 'LTRU Working Group'
> On Wed, May 07, 2008 at 10:25:11AM +0100,  Debbie Garside
> <[email-masked]> wrote  a message of 55 lines which said:
>
> * "dummy" right truncation, as allowed by extlangs, is not a
> good idea for macrolanguages, anyway (two encompassed
> languages of the same macrolanguage may not be mutually intelligible).
>
> * politically, it may send the message that the encompassed
> language is inferior in some way.
>
> As Randy said here, you can also make a similar list of the
> good reasons to keep extlangs. The case is really a 50-50 but
> we have to decide on one solution.
>
> > be restrictive to facilitating fallback *just* for macrolanguages.
>
> The idea was not to facilitate fallback (a complicated thing,
> which depend a lot of non-linguistical data), just to use an
> information that ISO 639-3 gives us (macrolanguages).
>
> In the current 4646bis model, "smart" fallback is not handled
> (because it requires a lot of outside information).
>
> This is a huge work, compared to with Suppress-Script
> discussions is peanuts :-)
>
> "fr" (french) is a good fallback for "br" (breton), not
> because the two languages are related (they are quite
> distinct) but because Brittany is in France and, since the
> 19th century, there are no longer breton speakers that do not
> speak french.
>
> "fi" (finnish) is probably a good fallback for "xx-FI"
> because finnish is the most common language in Finland so
> someone who requests "sv-FI"
> In writing, "arb" is a good fallback for every encompassed
> language of the macrolanguage "ar" (arabic). But it is not
> true in spoken form.
>

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 8 May 2008 08:54:30 +0100

And so, the discussion comes full circle. :-)

This is the very reason why extlang was introduced in the first place (I
think).

Debbie

> Behalf Of Peter Constable
> Sent: 08 May 2008 07:27
> To: LTRU Working Group
> Btw, in relation to "lists" for declaring dual language
> values on a given bit of content, and specifically the #3
> case in which the "dual" values are the generic/specific pair
> for the "same" thing (to the extent that e.g. "zh" is the
> "same" as "yue" or as "nan" etc.), it occurs to me that there
> is one way we could be completely certain about being able to
> express the generic/specific "list" in a way that all BCP 47
> apps could support: extlang!
>
> ;-)
> Peter
>
>
>
> > From: [email-masked] [mailto:[email-masked]]
> On Behalf
> > Of Peter Constable
> > msg I just sent, repeated below, Language Priority Lists represent
> > case #1. "mul" is case #2. But there is a third potential
> kind of list, #3:
> >
> > 2. declaring that content/resources are mixed -- that is,
> they pertain
> > to multiple, distinct varieties
> >
> > 3. declaring a single variety for content/resources at
> multiple levels
> > of specificity
> >
> >
> >
> > Peter

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 08 May 2008 10:30:05 +0200

Peter Constable 2008-05-08 08.26:
> > Peter Constable, Wednesday, May 07, 2008 8:48 PM
> > > From: Phillips, Addison Wednesday, May 07, 2008 5:20 PM
> > #1. "mul" is case #2. But there is a third potential kind of list, [...]

Regarding #3, yes, for those languages for which it has been judged that 
extlang would be allowed. Not all macrolanguages was planned included in 
the extlang feature.

To cover all macrolangues - or "same" situations, just as one could have 
said that "mul-fr-en" could be used to tell what kind of langues a "mul" 
covered, it should also be possible to define a macrolanguage tag - 
"mac" - which could be used for such cases. For example "mac-zh-cmn", 
and "mac-no-nn". Or, perhaps more backward compatible: zh-mac-cmn, 
no-mac-nb. Or perhaps nother keyword subtag than "mac" - how about "=" 
or "plus" or ...

---

From: Shawn Steele <[email-masked]>
Date: Thu, 8 May 2008 08:55:32 -0700

> ones and "cmn" ones, and perhaps vice versa, depending on the needs of

And isn't a list a type of configurabilty?  It'd probably have to be configurable at the page level.  Also the server probably needs to understand the intent of the user and the page designer.  Presumably http-accept-lang can include multiple variants if the user thinks its interesting.

A random though I had from Peter's suggesting that "isn't zh-cmn a type of list?" is that we *could* allow zh for Chinese, cmn for Mandarin, and zh-cmn for the "list" syntax, in a somewhat portable format.  That doesn't solve the german/english dictionary scenario, but it would address all the variations of Chinese "lists".

- Shawn

---

From: Peter Constable <[email-masked]>
Date: Thu, 8 May 2008 09:12:13 -0700

> From: Debbie Garside [mailto:[email-masked]]

> This is the very reason why extlang was introduced in the first place

Well, not quite, really.

It didn't come about by this line of reasoning. The reasoning that initially led to extlang was tied up with the reasoning that led to macro-languages in ISO 639-3. My point of departure was having assumed I'd approach ISO 639-2 entries that potentially would map to multiple languages using a major-language-variety operational principle I had defined.

a. IANA already has registered entries zh-yue, zh-guoyu, etc. that would contradict the application of the MLV principle.

b. MARC explicitly treats zh/zho/chi as encompassing Mandarin and Cantonese; that also contradicts the application of the MLV principle. (Ditto for most of the other cases in ISO 639-2.)

c. Maybe I need to abandon MLV and go with some notion of an intermediate-level grouping between individual language and collection. I'll call that "macro-language" until I find a better name. So long, MLV; hello "macro-language".

d. Hmmm... Several of the IANA-registered tags for individual Chinese languages use alpha-3 subtags; if all of them did that, then we could have a general principle applicable to macro-language cases, and these would go away as special cases.

e. And if we did that, then the right-truncation matching would automatically match uses of the new, more specific IDs with the pre-existing macro-language IDs. (Of course, that was recognized for zh-yue etc. when they were first registered.)

f. Looks like two potential benefits to the xx(x)-yyy syntax for macro-language cases: the current special cases are no longer special, and right-truncation matching works for matching the newer individual-language IDs with existing macro-language IDs. Let's float "extlang" for 3066bis.

The line of reasoning I gave here was slightly different:

A. It may be useful in some scenarios to be able to make a paired declaration about the language attributes of an information object, giving both specific and generic values.

B. Some applications might allow specifying two language tags for an IO, but for some applications that might require a breaking change.

C. If a single tag contained both the specific and generic IDs in a way that fit 4646 syntax, then there'd be no need for any breaking changes: all BCP47 apps would already permit it.

d. Extlang does that.

Peter

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 8 May 2008 17:50:03 +0100

Thanks for the explanation.

I find your reasoning quite sound.

Best

Debbie

> Sent: 08 May 2008 17:12
> To: 'LTRU Working Group'
> > This is the very reason why extlang was introduced in the
> first place
> > (I think).
>
> Well, not quite, really.
>
> It didn't come about by this line of reasoning. The reasoning
> that initially led to extlang was tied up with the reasoning
> that led to macro-languages in ISO 639-3. My point of
> departure was having assumed I'd approach ISO 639-2 entries
> that potentially would map to multiple languages using a
> major-language-variety operational principle I had defined.
>
> a. IANA already has registered entries zh-yue, zh-guoyu, etc.
> that would contradict the application of the MLV principle.
>
> b. MARC explicitly treats zh/zho/chi as encompassing Mandarin
> and Cantonese; that also contradicts the application of the
> MLV principle. (Ditto for most of the other cases in ISO 639-2.)
>
> c. Maybe I need to abandon MLV and go with some notion of an
> intermediate-level grouping between individual language and
> collection. I'll call that "macro-language" until I find a
> better name. So long, MLV; hello "macro-language".
>
> d. Hmmm... Several of the IANA-registered tags for individual
> Chinese languages use alpha-3 subtags; if all of them did
> that, then we could have a general principle applicable to
> macro-language cases, and these would go away as special cases.
>
> e. And if we did that, then the right-truncation matching
> would automatically match uses of the new, more specific IDs
> with the pre-existing macro-language IDs. (Of course, that
> was recognized for zh-yue etc. when they were first registered.)
>
> f. Looks like two potential benefits to the xx(x)-yyy syntax
> for macro-language cases: the current special cases are no
> longer special, and right-truncation matching works for
> matching the newer individual-language IDs with existing
> macro-language IDs. Let's float "extlang" for 3066bis.
>
>
> A. It may be useful in some scenarios to be able to make a
> paired declaration about the language attributes of an
> information object, giving both specific and generic values.
>
> B. Some applications might allow specifying two language tags
> for an IO, but for some applications that might require a
> breaking change.
>
> C. If a single tag contained both the specific and generic
> IDs in a way that fit 4646 syntax, then there'd be no need
> for any breaking changes: all BCP47 apps would already permit it.
>
> d. Extlang does that.
>
>
>
> Peter

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 8 May 2008 14:26:27 -0700

I took Shawn's suggestion below and Peter's email about use cases for lists and molded them into a single, new, section for draft-14. At first I wanted to include them into the section on choice or on meaning, but it just doesn't fit into either. Here's my proposed text:

--
<section anchor="lists" title="Lists of Languages">

<t>In some applications, a single content item might best be associated with more than one language tag. Examples of such a usage include:<list style="symbols">

   1.<t>A language priority list <xref target="RFC4647"></xref> describing a user's language preferences. This is a (possibly weighted) list of potentially-unrelated varieties, expressing a preference, rather than as a declaration about actual content.</t>

   2.<t>Content items that contain multiple, distinct varieties. Often this is used to indicate an appropriate audience for a given content item when multiple choices might be appropriate. Examples of this could include: <list style="symbols">

      a.<t>Metadata about the appropriate audience for a movie title. For example, a DVD might label its individual audio tracks 'de' (German), 'fr' (French), and 'es' (Spanish), but the overall title would list "de, fr, es" as its overall audience.</t>

      b.<t>A French/English, English/French dictionary tagged as both "en" and "fr" to specify that it applies equally to French and English</t>

      c.<t>A side-by-side or interlinear translation of a document, as is commonly done with classical works in Latin or Greek</t>

   </list> </t>

   3.<t>Content items that contain a single language but which require multiple levels of specificity. For example, a library might wish to classify a particular work as both Chinese ('zh') and as Min Nan ('nan') for audiences capable of appreciating the distinction or needing to select content more narrowly.</t></list>

</t>

</section>
--

I can't seem to think of any normative language to wrap around this that isn't already handled by the choice or meaning sections or by 4647.

Comments?

Addison

Addison Phillips

> Sent: Wednesday, May 07, 2008 6:00 PM
> To: Shawn Steele; Phillips, Addison; Peter Constable
> Cc: LTRU Working Group
> In some applications a list of language tags is necessary, such as a
> user's language priority list, when multiple language apply equally to
> the same content, or when matching of older tags is expected.  Some
> example scenarios are:
>
> * A French/English, English/French dictionary could be tagged as "en,
> fr" to specify that it applies equally to French and English
> Standards and protocols specifying this RFC MAY choose to allow lists
> of language tags.  Such standards and protocols MUST NOT be required to
> allow lists of language tags if not appropriate.
>
> If a list of language tags is used it MUST contain well formed language
> tags and strongly RECOMMENDED that those tags are chosen using the
> Choice of Language Tag guidelines.
>
> When a list of language tags is used, the protocols MUST specify
> whether or not an order is or is not implied or required for the list.
>
> When a collection of content, such as a DVD with multiple tracks, can
> be tagged with multiple languages, the collection SHOULD be tagged with
> the appropriate language tags, and the individual pieces of content
> SHOULD be tagged with a single language tag from the list of the
> collection.
>
> Recommendations for searching and matching of lists of language tags
> should be addressed in the successor to RFC 4647

---

From: [email-masked]
Date: Thu, 8 May 2008 14:50:27 -0700

Addison wrote on 05/08/2008 02:26:27 PM:

> 
>       a.<t>Metadata about the appropriate audience for a movie 
> title. For example, a DVD might label its individual audio tracks 
> 'de' (German), 'fr' (French), and 'es' (Spanish), but the overall 
> title would list "de, fr, es" as its overall audience.</t>

Don't add this text to help me. There are some reasons why this isn't 
quite right. DVDs are sold by region, not language -- to oversimplify. 
fr-FR audience is not going to want the DVD we've targeted for fr-CA. Not 
sure this is a best practice or if it's within the scope of this working 
group to listen to me rattle on about why. (Though I know you've already 
done so on other occasions. Cheers.)

>    3.<t>Content items that contain a single language but which 
> require multiple levels of specificity. For example, a library might
> wish to classify a particular work as both Chinese ('zh') and as Min
> Nan ('nan') for audiences capable of appreciating the distinction or
> needing to select content more narrowly.</t></list>

Is this better practice than just deciding that maybe extlang wasn't such 
a bad idea after all? zh-nan does the trick nicely and the order of the 
tags clearly indicates the language hierarchy.

Regards,

Karen Broome

---

From: Shawn Steele <[email-masked]>
Date: Thu, 8 May 2008 19:58:36 -0700

> done so on other occasions. Cheers.)

I wasn't specifically trying to "solve DVD" with this example, it just seemed like an example of mixed content that everyone could identify with :)  I get that it isn't strictly right.  I could imagine a tagging mechanism that included a scheme like mine and also the region code.  Note that these could be fr-FR and fr-CA if that was helpful.

Anyway, I get that the example might not work for you, however is it a problem for you DVD guys if it was in the draft?  We could find a different example, but I think this is clear to people trying to understand the scenario, even if it doesn't really completely apply to DVD.

- Shawn

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 9 May 2008 09:03:04 -0700

Actually, one more:

e.<t>A single tag needs to be replaced by multiple tags in order to remove
deprecated tags, or get more precise results than macrolanguages. Example:
"sr-CS" => "sr-RS, sr-ME"; "sh" => "sr, bs, hr"</t>

On Fri, May 9, 2008 at 7:41 AM, Mark Davis <[email-masked]>
wrote:

> This looks like a good direction (I have to be brief; won't have time to
> look at these in detail until Monday).
>
> The one thing I would add, which in certain ways is the most important, is
>
>      d.<t>A document or segment of text that could equally well be tagged
> by each of two or more tags. This can happen where the exact language is not
> known, or multiple tags could validly apply.</t>
>
>
>
> On Thu, May 8, 2008 at 2:26 PM, Phillips, Addison <[email-masked]>
> wrote:
>
>> I took Shawn's suggestion below and Peter's email about use cases for
>> lists and molded them into a single, new, section for draft-14. At first I
>> wanted to include them into the section on choice or on meaning, but it just
>> doesn't fit into either. Here's my proposed text:
>>
>> --
>> <t>In some applications, a single content item might best be associated
>> with more than one language tag. Examples of such a usage include:<list
>> style="symbols">
>>
>>   1.<t>A language priority list <xref target="RFC4647"></xref> describing
>> a user's language preferences. This is a (possibly weighted) list of
>> potentially-unrelated varieties, expressing a preference, rather than as a
>> declaration about actual content.</t>
>>
>>   2.<t>Content items that contain multiple, distinct varieties. Often this
>> is used to indicate an appropriate audience for a given content item when
>> multiple choices might be appropriate. Examples of this could include: <list
>> style="symbols">
>>
>>      a.<t>Metadata about the appropriate audience for a movie title. For
>> example, a DVD might label its individual audio tracks 'de' (German), 'fr'
>> (French), and 'es' (Spanish), but the overall title would list "de, fr, es"
>> as its overall audience.</t>
>>
>>      b.<t>A French/English, English/French dictionary tagged as both "en"
>> and "fr" to specify that it applies equally to French and English</t>
>>
>>      c.<t>A side-by-side or interlinear translation of a document, as is
>> commonly done with classical works in Latin or Greek</t>
>>
>>   </list> </t>
>>
>>   3.<t>Content items that contain a single language but which require
>> multiple levels of specificity. For example, a library might wish to
>> classify a particular work as both Chinese ('zh') and as Min Nan ('nan') for
>> audiences capable of appreciating the distinction or needing to select
>> content more narrowly.</t></list>
>>
>> </t>
>>
>> </section>
>> --
>>
>> I can't seem to think of any normative language to wrap around this that
>> isn't already handled by the choice or meaning sections or by 4647.
>>
>> Comments?
>>
>> Addison
>>
>> Addison Phillips

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 9 May 2008 07:41:48 -0700

d.<t>A document or segment of text that could equally well be tagged by
each of two or more tags. This can happen where the exact language is not
known, or multiple tags could validly apply.</t>

On Thu, May 8, 2008 at 2:26 PM, Phillips, Addison <[email-masked]>
wrote:

> I took Shawn's suggestion below and Peter's email about use cases for lists
> and molded them into a single, new, section for draft-14. At first I wanted
> to include them into the section on choice or on meaning, but it just
>   1.<t>A language priority list <xref target="RFC4647"></xref> describing a
> user's language preferences. This is a (possibly weighted) list of

---

From: Peter Constable <[email-masked]>
Date: Sat, 10 May 2008 07:48:21 -0700

I don’t see what d adds: a-c give examples in which multiple tags apply, and if the language is unknown I don’t know how why a list declaring multiple languages would be used.

I don’t understand what you have in mind for e. Do you mean that a list may be used to make a declaration using a set of current tags for some set of varieties encompassed by a now-deprecated tag that had previously been used for that record? This doesn’t strike me as an instance of “Content items that contain multiple, distinct varieties”; rather the examples you give are cases of a single variety that over time, for geo-political reasons, have begun to be given distinct identities.

Btw, it’s not entirely clear to me why sh should be deprecated in our usage if there is content out there than can appropriately be described as “Serbo-Croatian”.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Friday, May 09, 2008 9:03 AM
To: Phillips, Addison
Cc: Shawn Steele; Peter Constable; LTRU Working Group
Subject: Re: [Ltru] solving the chinese thing... lists

Actually, one more:

e.<t>A single tag needs to be replaced by multiple tags in order to remove deprecated tags, or get more precise results than macrolanguages. Example: "sr-CS" => "sr-RS, sr-ME"; "sh" => "sr, bs, hr"</t>
On Fri, May 9, 2008 at 7:41 AM, Mark Davis <[email-masked]<mailto:[email-masked]>> wrote:
This looks like a good direction (I have to be brief; won't have time to look at these in detail until Monday).

     d.<t>A document or segment of text that could equally well be tagged by each of two or more tags. This can happen where the exact language is not known, or multiple tags could validly apply.</t>

On Thu, May 8, 2008 at 2:26 PM, Phillips, Addison <[email-masked]<mailto:[email-masked]>> wrote:
I took Shawn's suggestion below and Peter's email about use cases for lists and molded them into a single, new, section for draft-14. At first I wanted to include them into the section on choice or on meaning, but it just doesn't fit into either. Here's my proposed text:

  </list> </t>

</t>

</section>
--

Comments?

Addison

Addison Phillips

> From: Shawn Steele [mailto:[email-masked]<mailto:[email-masked]>]

_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ltru

--
Mark

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 10 May 2008 19:28:57 +0200

Wikipedia http://en.wikipedia.org/wiki/Serbo-Croatian_language

>     Linguists are divided on questions regarding whether the name is
>     deprecated. It is still used, for lack of a more succinct
>     alternative, to denote the "daughter" languages as a collectivity.
>     An alternative name has emerged in official use abroad —
>     Bosnian/Croatian/Serbian (BHS). It is also known in the regional
>     linguistic community as the Central South Slavic diasystem.

The study of Serbian-Croatic at University of Oslo were soon to switch 
name to Bosnia-Kroatian-Serbian.

So, the issue - the macrolanguage - seems indeed to exist, merely under 
- in some "abroad" circles - another name.

Leif Halvard Silli

Leif Halvard Silli 2008-05-10 18.40:
> The Serbocroatian Wikipedia seems to be live and vibrant.
>
> http://sh.wikipedia.org/
>
> Leif Halvard Silli
>
> Peter Constable 2008-05-10 16.48:
>> ...
>> Btw, it’s not entirely clear to me why sh should be deprecated in our 
>> usage if there is content out there than can appropriately be 
>> described as “Serbo-Croatian”.
>>
>> From: [email-masked] 
>> [mailto:[email-masked]] On Behalf Of Mark Davis
>> e.<t>A single tag needs to be replaced by multiple tags in order to 
>> remove deprecated tags, or get more precise results than 
>> macrolanguages. Example: "sr-CS" => "sr-RS, sr-ME"; "sh" => "sr, bs, 
>> hr"</t>
>> On Fri, May 9, 2008 at 7:41 AM, Mark Davis 
>> <[email-masked]<mailto:[email-masked]>> wrote:
>> This looks like a good direction (I have to be brief; won't have time 
>> to look at these in detail until Monday).
>
>

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 10 May 2008 18:40:28 +0200

http://sh.wikipedia.org/

Leif Halvard Silli

---

From: John Cowan <[email-masked]>
Date: Sat, 10 May 2008 14:15:25 -0400

> - in some "abroad" circles - another name.

There is a dialect continuum spoken in four different countries (with
outliers elsewhere).

There are also two separate standard languages, Serbian and Croatian,
which merged for a while (in the sense that users of each agreed to
accept the orthography and lexis of the other as also instances of their
standard form) and have since separated again.

There is an incipient third standard language, and a label for a fourth
but no actual separate standard yet, perhaps not ever.

Or as a Croat linguist put it to me back in the 1980s.  "I am a native
speaker of Croatian, and therefore have a native command of Serbian
as well."

-- 
An observable characteristic is not necessarily         John Cowan
a functional requirement.  --John Hudson                [email-masked]
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 10 May 2008 21:00:32 +0200

> but no actual separate standard yet, perhaps not ever.

Not sure what you mean by "third" and "fourth".

> speaker of Croatian, and therefore have a native command of Serbian

You forgot one, important, unifying thing, namely the special alphabet - 
or alphabets - which were crafted to unify the two - or more - "breeds" 
of these encompassed languages. The Serbian Cyrillic alphabet and its 
corresponding Croatian Latin alphabet, are the only two Latin and 
Cyrillic alphabets in use which are 100% interchangable - and created to 
be like that - except when it comes to sorting. (Though I have observed 
an - perhaps post-Yugoslavian - tendency to perform Cyrillic enumeration 
and sorting based on the Latin order values.)

Only the Serbocroatian languages share these alphabets. (The Macedonian 
Cyrillic alphabet differ a little from the Serbian one. And the 
Slovenian differs a little from the Croatian one.)

I have not heard that any of the Serbocroatian languages since 
Yugoslavia broke up have switched back to any "pre-Yugoslavian" alphabet 
or something.

When it comes to Chinese, then the more or less common script is one of 
the most important reasons, as I  understand it, for talking about 
macrolanguage. So where is the reason to view it otherwise with regard 
to Serbocroatian.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 10 May 2008 16:07:30 -0600

On Sat, May 10, 2008 at 8:48 AM, Peter Constable <[email-masked]>
wrote:

>  I don't see what d adds: a-c give examples in which multiple tags apply,
> and if the language is unknown I don't know how why a list declaring
> multiple languages would be used.
>
>
>
> I don't understand what you have in mind for e. Do you mean that a list may
> be used to make a declaration using a set of current tags for some set of
> varieties encompassed by a now-deprecated tag that had previously been used
> for that record?
>

Yes. The issue is that sr-CS is deprecated. As such, one should replace it.
However, because CS was replaced by RS/ME, one cannot replace it by a single
entity without being more specific than the original data was. And if you
don't know how the original data was derived, then you don't know which of
the choices it should be. (This is just one example of the general case.).

> This doesn't strike me as an instance of "Content items that contain
> multiple, distinct varieties"; rather the examples you give are cases of a
> single variety that over time, for geo-political reasons, have begun to be
> given distinct identities.
>

It is a case where one tag would be nowadays ambiguously one of two (or
more). [Even worse for cases like SU.]. If you can use a list, you can tag
it with nondeprecated codes, and neither lose nor add information.

>
>
> Btw, it's not entirely clear to me why sh should be deprecated in our usage
> if there is content out there than can appropriately be described as
> "Serbo-Croatian".
>

I share that view, but it is in fact deprecated.

>
>
> *From:* [email-masked] [mailto:[email-masked]] *On
> Behalf Of *Mark Davis
> *Sent:* Friday, May 09, 2008 9:03 AM
> *To:* Phillips, Addison
> *Cc:* Shawn Steele; Peter Constable; LTRU Working Group
> *Subject:* Re: [Ltru] solving the chinese thing... lists
>
>
>
> Actually, one more:
>

---

From: John Cowan <[email-masked]>
Date: Sun, 11 May 2008 02:21:37 -0400

> Not sure what you mean by "third" and "fourth".

Standard Bosnian is not yet well-defined.  Standard Montenegrin doesn't
yet exist; although the new Constitution says the national language is
the Montenegrin language, the national language academy says the national
language is Serbian.

> be like that - except when it comes to sorting. 

True that that's very convenient.  AFAIK, though, Standard Croatian is only
written in Latin script, so the advantage is solely for Standard Serbian.
I'd expect to see sr-Latn disappear over time.

-- 
Barry gules and argent of seven and six,        John Cowan
on a canton azure fifty molets of the second.   [email-masked]
        --blazoning the U.S. flag               http://www.ccil.org/~cowan
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru