Subject: Re: [Ltru] Macrolanguage usage

From: "Doug Ewell" <[email-masked]>
Date: Thu, 15 May 2008 19:39:58 -0600

Mark Davis <mark dot davis at icu dash project dot org> wrote:

> - where content written in an encompassed language is also 
> understandable in the predominant language (that being a distinct 
> language encompassed by the same macrolanguage), the content could 
> also be tagged with the macrolanguage identifier. Thus if a Cantonese 
> passage is understandable if read as Mandarin, it could also be tagged 
> with "zh", or where a Tajiki Arabic passage is also understandable in 
> Standard Arabic it could be tagged with "ar".

Cantonese "is a" Chinese, it falls under the Chinese macrolanguage 
umbrella, and therefore it could legitimately be tagged with "zh" 
whether or not it is understandable if read as Mandarin.

Unfortunately, this wording takes us back to the mindset that Chinese = 
Mandarin = "zh", and other Chinese languages are not "zh".  I thought we 
had just agreed not to do that.

--
Doug Ewell  *  Arvada, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Fri, 16 May 2008 08:24:16 +0100

+1

Debbie

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Doug Ewell
> Sent: 16 May 2008 02:40
> To: LTRU Working Group
> Subject: Re: [Ltru] Macrolanguage usage
>
> Cantonese "is a" Chinese, it falls under the Chinese
> macrolanguage umbrella, and therefore it could legitimately
> be tagged with "zh"
> Unfortunately, this wording takes us back to the mindset that
> Chinese = Mandarin = "zh", and other Chinese languages are
> not "zh".  I thought we had just agreed not to do that.
>
> --
> http://www.alvestrand.no/mailman/listinfo/ietf-languages  ^
>

---

From: Peter Constable <[email-masked]>
Date: Fri, 16 May 2008 07:10:11 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Doug Ewell

> had just agreed not to do that.

Well, you demonstrate that it evidently reads that way, though it's not quite saying that: this is a exception to the general recommendation. In general, we would not recommend that people tag Cantonese content as "zh", although they certainly can. The situation in which it's *most* reasonable to tag Cantonese content "zh" is when it is understandable to those that read Mandarin -- meaning it's understandable by the vast majority.

Do you only have a critique? Did you not find the overall text useful enough to even consider suggestions for how to improve? Overall, was it of no value?

Peter

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 16 May 2008 08:18:47 -0700

>
> Well, you demonstrate that it evidently reads that way, though it's not
> quite saying that: this is a exception to the general recommendation.
> In general, we would not recommend that people tag Cantonese content as
> "zh", although they certainly can. The situation in which it's *most*
> reasonable to tag Cantonese content "zh" is when it is understandable
> to those that read Mandarin -- meaning it's understandable by the vast
> majority.

So we have a small impasse, indicative of the larger problem here: do we or do we not recommend against the use of 'zh' to tag Mandarin Chinese? It isn't the tagging of Cantonese that is the controversy. It is whether 'zh' is equated with Mandarin.

>
> Do you only have a critique? Did you not find the overall text useful
> enough to even consider suggestions for how to improve? Overall, was it
> of no value?

It's not of "no value", but it is different from "option #2" and it isn't a commentary on the currently developing draft. I have pondered how to incorporate these divergent views carefully and suggest that the section on Chinese be expanded and edited as follows:

--
<t>The family of languages encompassed by the macrolanguage Chinese ('zh') provides a useful illustration of this. Historically many different kinds of content have been tagged with variations of the 'zh' subtag, with application specific meaning being associated with region codes in particular. This is because historically only the macrolanguage subtag was available for forming language tags. However, these languages are, in the main, not mutually intelligible when spoken. Written forms of these languages also show wide variation in form and usage and many of these languages are written in various contexts. However, note that the majority of literate Chinese can read "Standard Mandarin" documents and many, if not most, written documents use this form. This should not be taken as meaning that the other languages are never (or rarely) written, only that the standard Mandarin language form is most common and serves as a kind of written 'lingua franca'.</t>

<t>With the adoption of this document, subtags for the encompassed languages became available for use in language tags. These subtags SHOULD be used instead of the macrolanguage subtag 'zh' to identify Chinese language content. While documents written in Standard Mandarin could use the 'cmn' (Mandarin) subtag, their wide accessibility can be indicated by using the 'zh' subtag in this case.</t>

<t>For example, before script codes were available, Chinese written in the Traditional script was sometimes associated with the "zh-TW" (Chinese, Taiwan) tag. Another example would be the association of the various spoken or written forms of the Cantonese language with the "zh-HK" (Chinese, Hong Kong) tag. However, each of these tags could also be (and actually were) associated with other language forms as well. For example, "zh-TW" might also indicate the Min Nan language or "zh-HK" could indicate a Chinese content item adapted for Hong Kong (but not necessarily a Cantonese item: as noted, many written documents are in Mandarin Chinese).</t>

<t>Using the encompassed language subtags (in concert with other appropriate subtags) makes clear the actual language. Script, region, or other subtags can still delineate any additional or local variations in language (such as using the 'Hans' subtag to identify the simplified Chinese script or using the 'TW' region subtag to identify a Taiwanese regional form of a language). Thus, a Min Nan document written in the Traditional script and using a Taiwanese region form would use the tag "nan-Hant-TW", while a Cantonese sound recording could use the tag "yue" (Cantonese) or possibly "yue-HK" (Cantonese, Hong Kong) if the recording were appropriate to that region.</t>
--

Addison

---

From: Shawn Steele <[email-masked]>
Date: Fri, 16 May 2008 09:30:28 -0700

> So we have a small impasse, indicative of the larger problem here: do we or do we not
> recommend against the use of 'zh' to tag Mandarin Chinese? It isn't the tagging of
> Cantonese that is the controversy. It is whether 'zh' is equated with Mandarin.

I would recommend against tagging Mandarin chinese with zh.  Most people seem to be fairly clear that "zh" means "general unspecified chinese" and "cmn" means Mandarin.

--
<t>With the adoption of this document, subtags for the encompassed languages became available for use in language tags. These subtags SHOULD be used instead of the macrolanguage subtag 'zh' to identify Chinese language content. While documents written in Standard Mandarin could use the 'cmn' (Mandarin) subtag, their wide accessibility can be indicated by using the 'zh' subtag in this case.</t>
--
<shawn> this makes no sense :)  The middle sentence ways "use cmn instead of zh". and the final sentence says "use zh instead of cmn".  If the last sentence is necessary, then it should mention back compat or other mechanisms (like as part of a list that also includes cmn).  The way its written is like saying "tag english as 001 (world) because lots of people can read it".  I don't think that zh should ever be appropriate for cmn (excluding maybe back compat concerns), unless people really don't know which chinese it is.  It has been made very clear that although most Cantonese speakers may be able to read Mandarin, it is not sort of general chinese readable by all when read, it is Mandarin.

- Shawn

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 16 May 2008 09:47:27 -0700

Shawn wrote:
> I would recommend against tagging Mandarin chinese with zh.  Most
> people seem to be fairly clear that "zh" means "general unspecified
> chinese" and "cmn" means Mandarin.

Yes. I agree. I'm trying to incorporate everyone's comments into a coherent whole. Since the situation is complex, the "solution" or at least the explantation will necessarily be complex.

>
> --
> <t>With the adoption of this document, subtags for the encompassed
> languages became available for use in language tags. These subtags
> SHOULD be used instead of the macrolanguage subtag 'zh' to identify
> Chinese language content. While documents written in Standard Mandarin
> could use the 'cmn' (Mandarin) subtag, their wide accessibility can be
> indicated by using the 'zh' subtag in this case.</t>
> --
> <shawn> this makes no sense :)  The middle sentence ways "use cmn
> instead of zh". and the final sentence says "use zh instead of cmn".

No, not exactly. What is says is: you *can* label these as 'cmn', but you *can* (the 2119 keyword is MAY, maybe I should use that) indicate your *intention* of making the text (it is text) generally accessible.

Maybe rephrase as:

---
While documents written in Standard Mandarin SHOULD use the 'cmn' (Mandarin) language subtag to form the larger language tag, for some applications where it is necessary to indicate that the text is intended for wide accessibility MAY be indicated by using the 'zh' subtag.
--

> If the last sentence is necessary, then it should mention back compat
> or other mechanisms (like as part of a list that also includes cmn).
> The way its written is like saying "tag english as 001 (world) because
> lots of people can read it".

Actually, we have a tag for that: it is "i-default" and that is, in fact, the purpose of "i-default".

> I don't think that zh should ever be
> appropriate for cmn (excluding maybe back compat concerns), unless
> people really don't know which chinese it is.  It has been made very
> clear that although most Cantonese speakers may be able to read
> Mandarin, it is not sort of general chinese readable by all when read,
> it is Mandarin.
>
It *is* Mandarin, but using the 'zh' tag might indicate that it is "general purpose" text, and not specific to a particular audience. Software resources, for example, typically use this form of Mandarin. My copy of Windows, for example, offers a number of flavors of Chinese locale--and they all appear to be Mandarin.

Addison

---

From: Peter Constable <[email-masked]>
Date: Fri, 16 May 2008 09:54:20 -0700

> From: Phillips, Addison [mailto:[email-masked]]

> So we have a small impasse, indicative of the larger problem here: do
> we or do we not recommend against the use of 'zh' to tag Mandarin
> Chinese? It isn't the tagging of Cantonese that is the controversy. It
> is whether 'zh' is equated with Mandarin.

Again, I don't see why what I said is interpreted as equating 'zh' with Mandarin. I'm completely opposed to that. Re carefully the text that Mark sent: nowhere does it say, "If your content is Mandarin, you SHOULD use 'zh'," or, "It can be assumed that content tagged 'zh' is in Mandarin."

Peter

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 16 May 2008 10:13:27 -0700

Peter wrote:
>
> Again, I don't see why what I said is interpreted as equating 'zh' with
> Mandarin. I'm completely opposed to that. Re carefully the text that
> Mark sent: nowhere does it say, "If your content is Mandarin, you
> SHOULD use 'zh'," or, "It can be assumed that content tagged 'zh' is in
> Mandarin."
>

The text said (in part):

--
•       where content written in an encompassed language is also understandable in the predominant language (that being a distinct language encompassed by the same macrolanguage), the content could also be tagged with the macrolanguage identifier.
--

It then goes on to say:

--
Thus if a Cantonese passage is understandable if read as Mandarin, it could also be tagged with "zh",...
--

You are correct that this doesn't explicitly *say* 'zh' == 'cmn'. But it at least implies that Mandarin is the predominant language.

The real issue with the text you and Mark proposed is that it does not provide clear guidance, which, as I understand it, is one of the key objections that Karen has (I happen to know that she's away from email for a couple of days, so may not be able to respond in a timely fashion; I ask pardon if I misstate or misinterpret her position). The three recommendations in your bullet list, when applied to cmn/zh, as I interpret them are:

1. You can choose to tag or lookup Mandarin as either 'cmn' or 'zh', since Mandarin is the predominant language.
2. You can choose to consider 'zh' and 'cmn' to be synonyms or not as determined by your application's needs.
3. If the content is consistent with 'cmn', you can tag it as 'zh', since 'cmn' is the predominant form of 'zh'.

I don't have an argument with any of these, excepting #3. The guidance on this last is not clear. Why the predominant language only? What about other forms? What about the example text I just suggested (note that it needed some editing):

--
While documents written in Standard Mandarin SHOULD use the 'cmn' (Mandarin) language subtag to form the larger language tag, where it is necessary to indicate that the text is intended for wide accessibility applications MAY use the 'zh' subtag.
--

Was this your intention with the above?

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 16 May 2008 11:08:27 -0700

The text does NOT say that. And I think there is a confusion regarding what
is *permissible* and what is "a good idea". Let me illustrate the difference
between these.

   - It is *permissible* (morally and legally) for me to walk through back
   alleys in the San Francisco Tenderloin district at 3 in the morning,
   blindfolded, with hundred dollar bills pinned all over my body.
   - It is, however, not a *good idea*.

So look again at language tags.

*Non-Predominant Encompassed Languages
*

   - It is *permissible* to tag Cantonese text with "und" or "zh" or "yue".
      - It is a *good idea* to tag/lookup with "yue".
   - It is *permissible* to tag Tajiki Arabic text with "und" or "ar" or
   "abh".
      - It is a *good idea* to tag/lookup with "abh".

This point should not be at issue, since we say in the spec *and have said
for some time* that people should tag as specifically as needed. Just
tagging with "zh" or "ar" would not be sufficient to distinguish it from the
vast amount of material in the predominant encompassed language.

*Predominant Encompassed Languages*

The only thing that the text is saying is that when there is a predominant
language, for backwards compatibility -- AND with some careful caveats --
the above is broadened. This is *not* with respect to permissibility, but
with respect to recommended practice.

   - It is *permissible* to tag Mandarin text with "und" or "zh" or "cmn".
      - It is a *good idea* to tag/lookup with "cmn"*, except that for
      backwards compatibility, "zh" may be better for many implementations.*
   - It is *permissible* to tag Standard Arabic text with "und" or "ar" or
   "arb".
      - It is a *good idea* to tag/lookup with "arb"*, except that for
      backwards compatibility, "ar" may be better for many implementations.*

That lets people like Google, which has a very serious and important issue
with backwards compatibility, support the vast amount of Mandarin/Arabic/...
that is tagged with "zh", and continue to tag Mandarin content with "zh". It
also lets people like Shawn's group at Microsoft, having no backwards
compatibility issues with "cmn" and "zh", to shift over immediately.

It is also orthogonal to the recommendation for non-predominant encompassed
languages, where we would tag with the more specific tags: "yue",
"yue-Hant-HK", "yue-Hant-US", "abh-AF", and so on, once they are available.

Mark

-- 
Mark

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 16 May 2008 12:03:47 -0700

Hi -

As a technical contributor...

> From: "Mark Davis" <[email-masked]>
> To: "Doug Ewell" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Friday, May 16, 2008 11:08 AM
...
> "yue-Hant-HK", "yue-Hant-US", "abh-AF", and so on, once they are available.
...

I'd re-frame it in slightly different terms, as an example of the "tag wisely"
principle, rather than as anything that might be understood as a recommendation.
"For some applications, it is possible that there will be cases where it could
be advantageous to deliberately use a tag which is imprecise.  For example,
due to the peculiar situation of written Chinese, an application with large
quantities of legacy textual data already tagged with 'zh' might countinue
to use this over-broad tag even for new data which would more precisely be
tagged with 'cmn'."

Randy

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 16 May 2008 12:36:03 -0700

Hmm... that's a good bit of text.

I incorporated it and did some more editing to cut down the extraneous text, incorporate some of Mark/Peter's text, etc. and posted the result as editor copy "b" here:

  http://www.inter-locale.com/ID/editor/draft-ietf-ltru-4646bis-14b.html#macrolanguages
  http://www.inter-locale.com/ID/editor/draft-ietf-ltru-4646bis-14b.txt

Comments?

Addison

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

> Randy Presuhn
> Sent: Friday, May 16, 2008 12:04 PM
> To: LTRU Working Group

---

From: Shawn Steele <[email-masked]>
Date: Fri, 16 May 2008 12:39:31 -0700

---

What I'm looking for is a subtle distinction.  What this sounds like is "use the cmn subtag for Mandarin, but if you want 'wide accessibility' they you're going to want zh instead".  I don't want developers to think "gee, do I want to support only Mandarin, or all of China?  Well, China's got more users, so I'll tag it zh".

Apps should be able to make the correlation between zh and cmn and yue if appropriate for them, so the zh tag only sounds that helpful in the "legacy" context, or when you don't know if its truely Mandarin.  I do not want Mandarin to be commonly tagged as zh when it can be avoided.

So I'm thinking something more like this (draft, I don't really like how well I said it)

---
While documents written in Standard Mandarin SHOULD use the 'cmn' (Mandarin) language subtag to form the larger language tag, the 'zh' (General Chinese) subtag MAY be used if the actual variation is not known or for compatibility with older applications.
---

- Shawn

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 16 May 2008 12:43:30 -0700

Here's the 14 -> 14b diff:

  http://tinyurl.com/5fuw3j

Here's the 14a -> 14b diff:

  http://tinyurl.com/5tshfa

Addison

Addison Phillips

> Phillips, Addison
> Sent: Friday, May 16, 2008 12:36 PM
> To: Randy Presuhn; LTRU Working Group
> I incorporated it and did some more editing to cut down the extraneous
> text, incorporate some of Mark/Peter's text, etc. and posted the result
> as editor copy "b" here:
>
> > > the above is broadened. This is *not* with respect to
> permissibility,
> > but
> > > That lets people like Google, which has a very serious and
> important
> > issue
> > I'd re-frame it in slightly different terms, as an example of the
> "tag
> > wisely"
> > to use this over-broad tag even for new data which would more
> precisely
> > be
> > tagged with 'cmn'."
> >
> > Randy
> >
> >

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 16 May 2008 12:46:35 -0700

Argh. Make the second diff (14a -> 14b):

  http://tinyurl.com/5hazk2

Addison Phillips

> From: Phillips, Addison
> Sent: Friday, May 16, 2008 12:44 PM
> To: Phillips, Addison; Randy Presuhn; LTRU Working Group
> Subject: RE: [Ltru] Macrolanguage usage
>
> Behalf
> > Of
> > > >    - It is *permissible* to tag Mandarin text with "und" or "zh"
> or
> > > "cmn".
> > > >       - It is a *good idea* to tag/lookup with "cmn"*, except
> that
> > > for
> > > >    - It is *permissible* to tag Standard Arabic text with "und"
> or
> > > "ar" or
> > > >    "arb".
> > > >       - It is a *good idea* to tag/lookup with "arb"*, except
> that
> > > for
> > > > that is tagged with "zh", and continue to tag Mandarin content
> with
> > > "zh". It
> > > "For some applications, it is possible that there will be cases
> where
> > > it could
> > > due to the peculiar situation of written Chinese, an application
> with
> > > large

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 16 May 2008 13:09:14 -0700

I think that would be fine.

---

From: Shawn Steele <[email-masked]>
Date: Fri, 16 May 2008 13:35:23 -0700

> compatibility issues with "cmn" and "zh", to shift over immediately.
This seems to be oversimplifying the issue.  I presume that Google would also be interested in recognizing cmn tagged data and cmn requests when they see them?  If its "just" some internal database of languages and you have to map between the actual request/content names anyway, then I don't know why this working group would care if Google called their internal data "zh" or "abc" or whatever.  It the intent is interchange, then continuing to use "zh" when what is really meant is a large subset of zh (Mandarin) seems to perpetuate the existing ambiguity.

Microsoft is also concerned with the backwards compatibility issues, however we recognize that existing "zh" tagged data is not necessarily Mandarin (even though its likely).  We don't have language detection tools to try to guess what the application's resources or a web page actually has for a language.  For back-compat, you'll recall that I thought zh-cmn helped, which would seem to solve both problems.  (Google'd have the zh it wants and I'd have the specificity that I'm looking for.)

Presumably (unless its only for internal use), Microsoft and Google can't have different definitions of "zh" for interchange.  If the definitions differ, then indexing of IIS served pages or requests from IE browsers would not necessarily provide the expected results.

What happens for other language tags that change?  Like serbian (serbian remains the same, but data tagged at the region level will change.)  Needing to support zh + cmn isn't that different than other common scenarios.

- Shawn

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 16 May 2008 14:15:26 -0700

On Fri, May 16, 2008 at 1:35 PM, Shawn Steele <[email-masked]>
wrote:

>  This seems to be oversimplifying the issue.  I presume that Google would
> also be interested in recognizing cmn tagged data and cmn requests when they
> see them?
>

Yes, clearly. Although even that takes time and effort to do; there is no
magic wand. (As I've said, the addition of non-predominant encompassed
language subtags like "yue" is important and useful for us. The addition of
the predominant encompassed language subtags just means a lot of work for no
additional benefit to us or our users. Following the de/gsw paradigm would
have been *much* simpler. But that is water under the bridge; we just have
to deal with the situation as it is, and try to give people guidance as to
the best way to handle them.)

> If its "just" some internal database of languages and you have to map
> between the actual request/content names anyway, then I don't know why this
> working group would care if Google called their internal data "zh" or "abc"
> or whatever.
>

The working group might not, but we have a goal of using BCP 47 not only
externally, but in communication *within* Google among the many different
products and programs. I assume that's not a bad thing ;-)

It the intent is interchange, then continuing to use "zh" when what is
> really meant is a large subset of zh (Mandarin) seems to perpetuate the
> existing ambiguity.
>

The issue is on output. If a program switches to "cmn" from "zh" on output,
then an external party who doesn't recognize "cmn" breaks. So that program
needs to output "zh" until it is certain that the recipients would all
recognize "cmn".

We could remain silent on this issue in the spec, but that would just be
withholding useful advice for people in terms of "tagging wisely", advice
that would allow people to interoperate more effectively.

> Microsoft is also concerned with the backwards compatibility issues,
> however we recognize that existing "zh" tagged data is not necessarily
> Mandarin (even though its likely).  We don't have language detection tools
> to try to guess what the application's resources or a web page actually has
> for a language.  For back-compat, you'll recall that I thought zh-cmn
> helped, which would seem to solve both problems.  (Google'd have the zh it
> wants and I'd have the specificity that I'm looking for.)
>

No, it doesn't solve the problem at all. There are many, many circumstances
where you want precise communication, not lookup&fallback. If recipient
expecting "zh" (and who doesn't know about the new codes) will break when
they get "zh-cmn" just like they will break when they get "cmn": extlang or
not makes no difference. (Extlang only really has an effect with lookup, you
think positive, I think negative, but let's not discuss that here -- we've
all agreed to look at that issue *after* this round.)

>
> Presumably (unless its only for internal use), Microsoft and Google can't
> have different definitions of "zh" for interchange.  If the definitions
> differ, then indexing of IIS served pages or requests from IE browsers would
> not necessarily provide the expected results.
>

Once both parties can handle both, it is not a problem. And where there is a
handshaking protocol it is not a problem. But there will be a *long*
transition period where only "zh" can be depended on to work.

> What happens for other language tags that change?  Like serbian (serbian
> remains the same, but data tagged at the region level will change.)  Needing
> to support zh + cmn isn't that different than other common scenarios.
>

Right, I've been saying all along that this is the same issue with any
predominant encompassed language.

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 20 May 2008 13:51:07 +0200

> While documents written in Standard Mandarin SHOULD use the 'cmn' (Mandarin) language subtag to form the larger language tag, the 'zh' (General Chinese) subtag MAY be used if the actual variation i…

I disagree with the above text.

I would however agree with this text:

<t>While **for example** documents written in Standard Mandarin SHOULD use the 'cmn' (Mandarin) language subtag to form the larger language tag,  the 'zh' (General Chinese) subtag MAY be used for compatibility with older applications. **The same rule applies to all the other Chinese languages. Thus also for example Cantonese may be tagged as 'zh' for the exact same reasons as Mandarin. Another legal usecase for 'zh' is if the actual variation is not known.</t>

-- 
leif halvard silli

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 20 May 2008 14:20:42 +0200

Mark Davis 2008-05-16 23.15:
> additional benefit to us or our users.

Sad to hear a Google representative speak like that. Because it does add 
benefit to your users. Macrolanguage information represents reality - it 
is not just teory.

And, btw, for Google, each language tag is also an potential advertising 
channel, for instance. In order to be able to use Nynorsk in 
adverticing, I must be reasonably certain that those who are not turned 
off by that receives it. However, as long as Google associates 'no' with 
'nb' then you are sucking up all the unknowing nynorskusers who just 
take what the browser/OS offers them.

And for searching, being able to discern between Nynorsk, Bokmål or both 
taken together, would be useful.

>  Following the de/gsw paradigm would

Often it is much simpler to ignore than to take notice of, it seems.

> products and programs. I assume that's not a bad thing ;-)

It is a bad thing if you force your possibly wrong understanding upon 
the rest of the world.

> needs to output "zh" until it is certain that the recipients would all

Firstly, you are in fact making an argument her for the use of extlang.

And where is this an issue for Google? You control your own applications 
- so it can't be that. The only place I see is in lang negoation for web 
browsers. However, there you are masters in using cookies and IP 
information anyhow.

> that would allow people to interoperate more effectively.

As long as you are not silent on the issue that 'zh' equally well can be 
used for Cantonese, then pleas also tell that it can be used for 
Mandarin. Both things are true and must be stated - not hidden.

> all agreed to look at that issue *after* this round.)

But if a recpipient expecting 'no' to mean all forms of Norwegian breaks 
when he sees that you uses it only for Bokmål, then that doesn't matter 
for you.

Yes, negatively.

> transition period where only "zh" can be depended on to work.

The most important thing is probably the goal.

> predominant encompassed language.

As it is the same issue as with any predominant region variant of 
English, for instance.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Don Osborn" <[email-masked]>
Date: Tue, 20 May 2008 10:24:27 -0400

Is anyone on this list following this discussion and the evolution of the standard from the point of view of African languages? I haven't been able to stay with it for several reasons, so apologies in advance if the following is redundant to what has already been taken into account.

Much of my concern re the form and use of language tags have been voiced previously on this list and IETF-languages. I do think that as we get more people working on African language web content, software localization, etc., there will be more questions about available ISO 639 code elements and how those are (and can be) used. 

Just yesterday I learned that a small group has translated Opera in Songhai and has plans to localize some other open source software. The tag used in the .lng file is "soŋ" which fills their need and appears to approximate most closely to the ISO 639-2 code "son" - the latter representing a cluster, not even a macrolanguage. There is no locale for any variety of Songhai (which would be a topic for another list of course). I'm in communication with them for more information on the intent of their current projects but my impression is that they are expressedly not intending to limit themselves to any single currently encoded entity (khq, ses, dje, hmb, ddn, or whatever). 

It may be that for some purposes, localizers and content developers may prefer a broader tag than what is available in ISO 639-3, while in other cases find that the latter fits the need. In some cases perhaps the optimal tags do not exist yet. What this says to me is that there is a need both for flexibility and for various ways to define and refine relationships among tags

The reality of language is of course quite fluid, and we see that especially in cases where there is no single standard language form among several very closely related tongues. Even where linguists or diverse country authorities have established rules of orthography and so on, you may have situations where the optimal localization may be a series of compromises (or a reflection of either dialect leveling or an awareness of speakers of how things are expressed in different varieties of their language). That in turn may require feedback from users to know how well it works for them, but the end result in any case may be new information about tags and relationships among them.

The needs of information technology of course tend to be for clear, mutually exclusive categories (a bit cannot be somewhere between 0 and 1, nor sometimes both 0 and 1, and so on, on up). The only way to accommodate the fluidity of language in this environment seems to be with options of resort to wider or narrower definitions.

Hopefully what I'm suggesting has already been taken into account, and if so, apologies for the redundancy. But whatever the case, I do think that as more languages enter "cyberspace" mure fully in their various forms, there will be more questions by more people about how existing tags, and the systems for using them to identify content, write locales etc., respond to their needs.

Don

Don Osborn
Bisharat.net

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 20 May 2008 10:44:42 -0700

Hi -

As working group co-chair...

> From: "Leif Halvard Silli" <[email-masked]>
> To: "Mark Davis" <[email-masked]>
> Sent: Tuesday, May 20, 2008 5:20 AM
...
> Sad to hear a Google representative speak like that.
...

Participation in the IETF is by INDIVIDUALS, not organizations
or represenatatives of organizations.  There should be no assumption
that anyone's positions will be aligned with the interests of their
employer(s).  It's ok for someone to describe how a particular
company has approached an issue, but that information has no bearing
on a final determination of consensus.

Randy
ltru co-chair

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 20 May 2008 13:44:18 -0700

On Tue, May 20, 2008 at 5:20 AM, Leif Halvard Silli <[email-masked]> wrote:

> benefit to your users. Macrolanguage information represents reality - it is
> not just teory.

Actually no. According to everything I've heard, the macrolanguage was
devised as a construction which is an attempt to rationalize inconsistent
approach to languages used by previous versions of ISO 639. It does not
represent any particularly reality beyond that. As said before, whether two
languages are closely related or not is pretty much orthogonal to whether or
not they share a macrolanguage: ro and mo are examples.

You appear to be generalizing from the case of no, nn, nb to all
macrolanguages.

> channel, for instance. In order to be able to use Nynorsk in adverticing, I
> must be reasonably certain that those who are not turned off by that
> receives it. However, as long as Google associates 'no' with 'nb' then you
> are sucking up all the unknowing nynorskusers who just take what the
> browser/OS offers them.
>

I'm not going to speak to the issue of particular language support by Google
or any others except to say that it is always a tradeoff. Each new language
takes effort. Of the hundreds of languages that could be supported, one has
to consider the priority among those: is it better to do Bengali next, or
Nynorsk, and so on.

> Often it is much simpler to ignore than to take notice of, it seems.

We cannot ignore the situation; the question is what are the best strategies
for handling them going forward.

> It is a bad thing if you force your possibly wrong understanding upon the
> rest of the world.

Nobody's forcing anything.

> Firstly, you are in fact making an argument her for the use of extlang.

No. And notice again that extlang is not relevant to your issue: the extlang
issue does not pertain to no, nb, and nn; those are already coded. *None of
the extlang proponents except for you are proposing to have "no-nb" or
"no-nn".*

>
>
> And where is this an issue for Google? You control your own applications -
> so it can't be that. The only place I see is in lang negoation for web
> browsers. However, there you are masters in using cookies and IP information
> anyhow.

Modern companies do not live in a vacuum; there are many partners and other
organizations that they need to have effective communication and
interoperability with. Thus the reason for standards.

> used for Cantonese, then pleas also tell that it can be used for Mandarin.
> Both things are true and must be stated - not hidden.

My opinion, and I've hardly been quiet on the issue, is that for a large
class of implementations it is best for backwards compatibility to continue
to have resource lookup map 'ar' to Standard Arabic, 'zh' map to Mandarin,
and so on, and use the new codes for other languages, eg 'yue' for
Cantonese. And the text of draft #14 allows that possibility, which is after
all perfectly legal. If you choose to map 'zh' to Gan in your resource
lookup, be my guest.

> when he sees that you uses it only for Bokmål, then that doesn't matter for
> you.
>
> Yes, negatively.

Again, 'no' is not relevant to the extlang discussion.

> The most important thing is probably the goal.

This is completely unrealistic. If you don't have a clear migration path for
people to take, you will never reach that goal.

> As it is the same issue as with any predominant region variant of English,
> for instance.

I have no idea what you mean by this.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 21 May 2008 09:43:41 +0200

On Tue, May 20, 2008 at 10:24:27AM -0400,
 Don Osborn <[email-masked]> wrote 
 a message of 11 lines which said:

> Is anyone on this list following this discussion and the evolution
> of the standard from the point of view of African languages? 

No, I have enough trouble with the languages I know (or can find
information about).

> Just yesterday I learned that a small group has translated Opera in
> Songhai and has plans to localize some other open source
> software. The tag used in the .lng file is "so??" which fills their
> need and appears to approximate most closely to the ISO 639-2 code
> "son" - the latter representing a cluster, not even a
> macrolanguage. 

It is not clear why they did not use the standard code "son". They are
looking for trouble. It is a valid ISO 639-2 code and "son" is a
valid language tag.

> It may be that for some purposes, localizers and content developers
> may prefer a broader tag than what is available in ISO 639-3,

Hold on, it seems an issue with ISO 639, not with us, no? I do not
mean that ISO 639 is perfect, just that we decided to build on it and,
frankly, we (LTRU / IETF) cannot certainly produce something better.

The future ISO 639-5 may help them here, if they want code for broad
language families.

---

From: John Cowan <[email-masked]>
Date: Wed, 21 May 2008 09:04:01 -0400

Don Osborn scripsit:

> may prefer a broader tag than what is available in ISO 639-3, while in
> other cases find that the latter fits the need. In some cases perhaps
> the optimal tags do not exist yet. What this says to me is that there
> is a need both for flexibility and for various ways to define and
> refine relationships among tags.

There is no dispute about the usability of broader tags from 639-2
or 639-3 (or the forthcoming 639-5, presumably).  The dispute is
solely about the best way to represent those languages which are
treated as single in 639-2 and multiple in 639-3: whether a compound
or a simple subtag should be used.

-- 
John Cowan                                [email-masked]
I amar prestar aen, han mathon ne nen,    http://www.ccil.org/~cowan
han mathon ne chae, a han noston ne 'wilith.  --Galadriel, LOTR:FOTR
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 22 May 2008 04:12:35 +0200

> You appear to be generalizing from the case of no, nn, nb to all

When this group first has one member which speaks and writes as minority 
encompassed language of a Macrolanguage, then it is of course best to 
write him off.

So, why do you think ISO 639-1 is inconistent? By accident?

The inconsistencies of ISO 639-1 are very much based on reality.

> all perfectly legal. If you choose to map 'zh' to Gan in your resource

This is hiding facts. You can have that opinion that 'zh' is best for 
Mandarin.

But you should not hide that 'zh' is perfect for Cantonese as well. It 
is the best choice for Cantonese, if you want backward compatibility.

In fact, the best thing is the opposite of what you say.

The mayority has lots of power. The minority gets ignored. THerefore, be 
careful so you not destroy what the minority rely on, because the 
minority is the last ones to be handled. They will receive web browsers 
with support for the new Cantonese tag much later than the Mandarine 
users will get the same tag. (Not to speak about web pages using those 
tags.)

I must again generalize based upon my experience as minority extlang 
user:  In Mac OS X "Leopard" Apple introduced support for "Nynorsk". But 
in a way that dissassociated it from "Norwegian". So, because Norwegians 
often are of the kind that "either Nynork or Bokmål", if I disseleect 
support for Bokmål, then my applicatinos will launch in English instead 
of "Norwegian" - or, if you wish, Bokmål.

And in my view, this "Leopard" solution for Nynorsk is exactly what you 
propose for Cantonese.  What you propose would, in a very likely worst 
case, cause that selecting Cantonese leads the user to get English 
instead of Mandarine when Cantonese is lacking.

> I have no idea what you mean by this.

I am not surprised that you say so.

Basically, for English the "remove from right" language fallback 
negotiation method works all the time.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 22 May 2008 05:42:50 +0200

Leif Halvard Silli 2008-05-22 04.12:
> >>> What happens for other language tags that change? Like serbian
> >>> (serbian remains the same, but data tagged at the region level will
> >>> change.)  Needing to support zh + cmn isn't that different than
> >>> other common scenarios.
> >>>
> >> Right, I've been saying all along that this is the same issue with
> >> any predominant encompassed language.  As it is the same issue as
> >> with any predominant region variant of English, for instance.
> >
> > 

What I mean is: You constantly diminish the need for automatic fallback, 
and emphasize the problems: it could happen that this solution creates 
some problems.

Could it not happen that your proposal creates problems? And what are they?

You also say emphasize that sometimes "same macrolanguage" do not mean 
that the different encompassed language speakers can understand each 
other. Ok, it could happen.

You also brings up very artificial things like the socalled 
discrimination someone could fell if they cannot have their own single 
tag. (I think both 'zh-yue' and 'yue' should be possible, though, so I 
go free from that "problem".)

I say: treat Macrolanguages (including their encompassed languages) 
equally to your own, English language. You rely on automatic fallback 
for your language. Let macrolanguage users have the same opportunity.

You and others also emphasize that "The users know best. Let them choose."

I say, that, in my own experience, belonging to a minority of a 
macrolanguage, one of the things that is harming us most, is that we 
must constantly be so damned conscious about everything. People hate 
being conscious all the time.

Many public web pages in Norway are in both Nynorsk and Bokmål. But if 
Bokmål is served by default, and I understand what I read, why shall I 
click on the Nynorsk link? It *must* work automatically. Or else it 
doesn't work.

This will be the same for the Chinese (macro)language in those places 
where Mandarin is dominating. The Mandarin dominanse is so strong that 
if saying "I prefer Cantonese" only leads to trouble, then people will 
not say it. It is much easier to say "I prefer Chinese, preferrably in 
the form of Cantonese". It *must* be easy to state which encompassed 
langague one prefers. Or else, the reality often is that there isn't 
much of a choice.

In fact, this is a choice about values.

Finally, the standpoint that we should not have extlang tags is OK in 
itself. One can be a strong supporter of multilingualism and so on, yet 
still have that view.

But you not only support this viewpoint, but are in addition getting all 
those that *could* have supported you - provided it was properly 
implemented and taking full notice of all the problems -  against you 
because you are demonstrating that you are, IMHO, neglecting the 
problems which the extlang tags were supposed to bridge.

So, what sanity is there in writing into the spec that the dominant 
language can tagged with the macrolanguag tag? Do we need to bang in 
open doors?

It is, however, crucial to emphasize that the minority language can be 
tagged that way. It is they who are loosing the battle. If the spec 
should correct and state something very firmly, then is that that it is 
not only legal, but often very useful, to tag minority encompassed 
languages that way.

This is even more important to state since it seems that not even all in 
this group get this point very well.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 21 May 2008 23:57:49 -0700

Comments below.

[snip]

> encompassed language of a Macrolanguage, then it is of course best to write
> him off.

I'm writing off nobody based on what language they speak and write.

What I'm saying is that the relationship between no and nn/nb, in terms of
mutual comprehensibility, are not typical of other macrolanguage
relationships, based on conversations with Peter Constable, who played a key
role in the development of the macrolanguage concept. If you have evidence
to the contrary for the other macrolanguages (see all of
http://www.sil.org/iso639-3/macrolanguages.asp), it would be good to bring
that forward.

[snip]

> But you should not hide that 'zh' is perfect for Cantonese as well. It is
> the best choice for Cantonese, if you want backward compatibility.

Not at all. When filtering for just Cantonese, you absolutely don't want to
get Mandarin content as well, since you would be swamped with irrelevant
results. If you want Cantonese + Mandarin, you should ask for it. When doing
fallback for Bokoto, you don't want to get Gbaya-Mbodomo. If you want Bokoto
OR Gbaya-Mbodomo you should ask for it.

You seem to think of "no-nno" or "no-nob" as a convenient way to get
automatic fallback. Someone might equivalently think that "ro-mol" is a
convenient way to get fallback from Moldavian to Romanian, or that "tl-fil"
is a convenient way to get fallback from Filipino to Tagalog (or "fil-tgl"
for the reverse), or "de-gsw" to get fallback from Swiss German to German,
or any number of other cases. After all, those relationships are far closer
than the vast majority of relationships between macrolanguages and
encompassed languages.

But in practice, having such compound tags complicates matters more than it
helps, as detailed in the document I sent out.

[snip]

> propose for Cantonese.  What you propose would, in a very likely worst case,
> cause that selecting Cantonese leads the user to get English instead of
> Mandarine when Cantonese is lacking.

What you are really asking for is for a reasonable fallback strategy that is
likely to get you to a language that you want. After all, exactly the same
situation applies for the Moldavian speaker who would rather see Romanian
than English, and they have no macrolanguage relationship. For that matter,
someone might think that "da-nor-nno-nob" is the answer for them, since they
can understand Danish better than English.

The solution is not to complicate language tags. This is looking in the
wrong place for the solution. What you really want are provided in
LanguagePriorityLists, where you can precisely specify exactly which
languages you can understand, and in what order of preference.

[snip]

> negotiation method works all the time.

Ad hominem attacks are hardly a way to convince anyone. I've had a
reasonably long career devoted to software internationalization, with
*the*goal being to let people be able to use their languages on
computers no
matter what language. To accuse me or any others on this list of only caring
about English makes me, let's just say, irritated.

Our position on extlang is the result of fairly extensive software testing
of a number of different models of language fallback, filtering, and
matching. In our experience in implementations it causes many more problems
than it solves, and is especially negative for the minority languages, even
more so than the predominant ones. The macrolanguage relationships are
published in the registry, so it is trivial for any implementation to
achieve exactly the same effects as extlang if desired for their
application. (Although as I've said, in general the results would not be
particularly good, because the question of whether A is a good fallback for
B is simply orthogonal to the question of whether A is a macrolanguage for
B.)

If you have implementation experience with a variety of different models --
covering all the combinations with scripts and countries -- dealing with
closely related languages like ro/mo -- and handling both filtering and
lookup -- if you have some basis for your statements, you should bring that
forward, or encourage others to do so.

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 24 May 2008 14:18:12 +0200

> I'm writing off nobody based on what language they speak and write.

I did not mean it like that. I only meant that my experience as 
"encompassed language user" is relevant. Somewhat opposed to what you 

> http://www.sil.org/iso639-3/macrolanguages.asp), it would be good to bring
Doesn't Cantonese users understand Mandarin? And most Arabian speakers 
can deal with Standard Arabic? How different the encompassed languages 
are isn't necessarily relevant. The relevant thing is that they are 
conceived as one language in some important areas. Education of some 
sort is necessary to maintain intelligibility and command of more than 
one encompassed language. (This is even the case for Norwegian.)

> OR Gbaya-Mbodomo you should ask for it.

I understand this technological wish. But what do the Cantonese users 
wish to search for?  (The most important thing, anyhow, is to provide a 
Cantonese interface.)

What I said before is important: Cantonese if available, Mandarin else. 
I am much more concerned with that the encompassed minority langauge 
user would not get even the majority encompassed language resources.

> than the vast majority of relationships between macrolanguages and

As you know, many tag Norwegian texts as 'no-no'. So it is obviously 
that when 'no-no' falls back to 'no', then of course 'no-nn' or 'no-nno' 
would fall back just as well. Why should I not believe so?

> helps, as detailed in the document I sent out.

In the draft you sent out you start by saying that "The arguments for 
extlang are that they give superior results,". However, this is an 
exaggeration of the standpoint that I for intance have. First, I assume 
you meant "technical superior". Well, no, I can understand that using 
short tags is easier to deal with, technically. And therefore superior 
to extlang. (In my testing with Apache, 'nn' and 'nb' was easier to deal 
with than 'no-nyn' and 'no-bok'.)

But then a problem is that the users "in the wild" still are tagging 
Norwegian as if we had an extlang system.

> likely to get you to a language that you want.

I am *not* asking for something individualistic, though.

Fine, the users should be able to customize from A to Z. And fallback 
strategies should be developed. But *that* is a different, allthough 
very related, subject!

I care just as much about "fall forward" as "fallback".  For instance, a 
developer must also be able to trust that things works. He must be able 
to use the Macrolangauge tag in the trust that it will "fall forward" to 
the encompassed languages.

How did I find out about the Nynorsk issues of OS X?

Well, I translated an application to Nynorsk. In Tiger, which only had 
'no' (Imagine that it only had 'zh') with Bokmål content, but 
"Norwegian" as label, it was fine. I used the 'no' tag, which said 
'Norwegian'. (Imagine that I tagged something Cantonese as 'zh'). Then 
suddenly in Leopard, using the same tag, it turned up in the interface 
as 'Bokmål'. (Imagine that 'Cantonese' suddenly turned up as 
'Mandarin'.) And since you are so wary about users that complain: This 
lead to user reactions. While there were no reactions under Tiger.

Macrolanguages represent a market. And if you say that 'zh' means 
Mandarin, or 'no' means 'Bokmål', then you might confuse both 
Mandarin/Cantonese/Bokmål/Nynorsk users.

The compatibility things that I spoke about has to do with a) being able 
to view a resource both as Chinese/Macrolangauge and as 
Cantonese/Encompassed lnagues. b) Having both the Macrolanguage tag and 
one's own Encompassed language tag at disposal.

>  After all, exactly the same

Moldovian and Romanian is perhaps best regardes as the same language.

But anyhow, I do not mean or say that the Macrolanguage conscept solves 
all problems, such as that of Moldovan and Romanian.

> languages you can understand, and in what order of preference.

According to you, then, Norwegian has the perefect situation. We have 
'no', 'nn' and 'nb'. The only thing - if I interpret you corectly - that 
could have made it more perfect, was if 'no' meant Bokmål and 'nn' meant 
Nynorsk.

However, this hasn't developed perfect. The fact is that there are lots 
of hierarchical tags around - de-DE, de-AT, en-GB etc. And people look 
and deduct and come up with 'no-nn' and 'no-nb' all by themselves. (Now, 
in this round, I will not repeat that this should be permitted - I just 
use it as an example.)

The problem is, that there is a "natural" tendency to think that 
Macrolanguage = majority language.  Not in a semantic sense, but in a 
practical sense. You convey this yourself.

And - if we can keep out those very few (as I see it) encompassed 
languages that do not need to compete (Read: fallback/negotiate) with 
another encompassed language, then the fact is that we need to perform 
fallback/language negotiation between encompassed languages of the same 
macrolanguage.

And this is a special kind of language negotiation. For a small 
macrolanguage like NOrwegian, we suddenly get 3 options. If instead we 
had extlang for Norwegian, we would in reality only have two options.

No, it is very important to realise the need for Macrolangauge support. 
The Macrolanguage tag must be free to be used by anyone who want to 
reach more than those that are tagged as belloing to only one spesific 
encompassaded language.

> *the* goal being to let people be able to use their languages on
> computers no matter what language. To accuse me or any others on this list of only caring about English makes me, let's just say, irritated.
>   
I did not put it exactly that way. And let's say I was a little 
irritated for what I saw as lack of trying to understand what I meant.

What you say is that fallback with tags that are longer than two 
subtags, is complicated.

But the fact is that such long tags will almost solely be needed for 
Chinese. For the other macrolanguages, the situation is not that 
complicated. And thus will yield more positive results - since the tags 
will have the same lenght as for example typical tags for British 
Englsih, which as we know, works well.

> more so than the predominant ones. 

How have you counted the negative things?

I believe full well that you have tested. But it is one thing to look at 
it from the outside. What you think is needed, might not be what is 
really needed.

Big corps with 100 of languages, is though a very particular breed. I 
have in mind more typical "within the same country" negotiation between 
to encompassed languages.

> The macrolanguage relationships are

I have allready been told that it is very important to read things out 
of the tags without needing to look into the registry. And I agree. That 
is a basic, and very good thing.

> forward, or encourage others to do so.

I do at least have some experience with the model you recommend, I feel.
-- 
leif halvard sillli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Sat, 24 May 2008 17:01:47 -0400

Leif Halvard Silli scripsit:

> can deal with Standard Arabic?

In both cases the answer is "no, not unless they have been educated
through the medium of the standard languages."  Currently there is
nowhere where you can be educated through the medium of any non-standard
Chinese or Arabic variety (with the marginal exception of Hong Kong),
so this amounts to "not unless they have been educated".  (This obviously
excludes people in other countries who speak a non-standard variety but
have been educated through the medium of some other language altogether.)

-- 
Babies are born as a result of the              John Cowan
mating between men and women, and most          http://www.ccil.org/~cowan
men and women enjoy mating.                     [email-masked]
    --Isaac Asimov in Earth: Our Crowded Spaceship
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Sat, 24 May 2008 17:51:50 -0700

> Leif Halvard Silli

> How different the encompassed languages
> one encompassed language.

Please note very carefully: the definition of macrolanguage entails that the range of varieties is treated as a single language in some application context. That does *not* entail that the encompassed varieties are mutually intelligible, or that there is any one encompassed variety that is intelligible to all the others. Certainly no claim is made in ISO 639 that either is true. Whether or not either is true for any macrolanguage is an empirical question.

Peter

---

From: Leif Halvard Silli <[email-masked]>
Date: Sun, 25 May 2008 13:38:12 +0200

Peter Constable 2008-05-25 02.51:
> > From: [email-masked] On Behalf Of Leif Halvard Silli
>
> Please note very carefully: the definition of macrolanguage entails that the range of varieties is treated as a single language in some application context. 

Quoting 639-3: (http://www.sil.org/iso639-3/scope.asp#M)

|...] considered distinct individual languages, yet in certain usage 
contexts a single language identity for all is needed [...]

The wording "is needed" is one that I think Mark should take note of.

> That does *not* entail that the encompassed varieties are mutually
> intelligible, or that there is any one encompassed variety that is
> intelligible to all the others. Certainly no claim is made in ISO 639
> that either is true. Whether or not either is true for any
> macrolanguage is an empirical question.

The strict definition might not entail this. But the empirical 
investigation that lead a group of languages to be placed in a 
Macrolanguage cathegory does most often entail it, it seems.

The fragment I quoted above mentions 3 typical situations for when the 
Macrolanguage definition is suitable - listed here in my order of 
preference:

   1. Transitional socio-linguistic situation; (The
      Serbian/Croatian/Bosnian/Montenegrin situation)
   2. Dominant encompassed language situation; (Arabic)
   3. A "common written form used for multiple closely-related
      languages" situation. (Chinese)

(The 2nd and 3rd sitution seems to float together, in my view. Well, 
even for the 1st situation - e.g. think serbo-croat-bosnian- - the 3rd 
("common written form") is very important there too. And, btw, when 
forced, I would place Norwegian in the first group.)

In both situation 1 and 2, I would use your words and say that we have a 
situation where "the encompassed varieties are mutually intelligible, or 
that there is any one encompassed variety that is intelligible to all 
the others".

In the third situation, Chinese, it seems partly to be the case - at 
least in China - that Mandarin is the lingua franca - thus we partly  
have a dominant language situation. The common written form is also very 
important, however, so that it perhaps serves as a "lingua franca" in 
itself.
-- 
leif halvard silli

---

From: "Kent Karlsson" <[email-masked]>
Date: Sun, 25 May 2008 20:02:16 +0200

Peter Constable wrote:
> Please note very carefully: the definition of macrolanguage 
> entails that the range of varieties is treated as a single 
> language in some application context. That does *not* entail 
> that the encompassed varieties are mutually intelligible, or 

I have stayed out of this heated debate, but this clarification
makes up my mind. I had, falsely, the impression that all
languages encompassed by a macrolanguage were largely mutually
intelligible in all cases (except for spoken 'zh'). But not so.

So on the consensus call questions, my answer is:

Q1: A
Q2: C
Doug's Q3: C, very much C

Reasons (not limited to one sentence...):

 a) See above quote, which I see as saying that extlangs will
    not provide the benefits claimed by the extlang proponents.
 b) The exceptions to extlang application, for various reasons,
    like for nn/nb and other instances, are already numerous.
 c) No extlangs is a simpler model, both syntactically and
    use-wise, and often have advantages over extlangs.

So a no-extlang model is simpler to explain and apply, and
often have use-advantages over an extlang model. While the
extlang model is harder to defend, especially given the
exceptions, and though it may sometimes have advantages over
a no-extlang model, I don't think those occasional advantages
outweigh the disadvantages in general (including the pedagogical
ones).

Even if we were to change back (in itself unappealing, and
I don't see a strong reason to do so) to an extlang model,
I would very much expect to see a non-extlang model being
applied in practice, even though some language tags then
would not be valid (though well-formed). I.e. even if
we were to go back to an extlang model, I would expect
quite a lot of content be tagged with an extlang at
the lang position (i.e. first). Better stick to a simpler
model, and also formally let would-be-extlangs be langs.

For the almost-running-example of Cantonese and language
preference list (like for a web browser), and drawing from
John Cowan's mail of today, a Cantonese user may use

1. yue	for spoken and written Cantonese (any script)
2. cmn-Hans	for written Mandarin (in simplified Han)
3. zh-Hans	for written "Chinese" (in simplified Han) for
		backwards compatibility, ideally to be phased
		out (except for archived old data)
4. NOT listing cmn, to avoid spoken Mandarin, which this
		hypothetical user does not understand.
		(Hopefully written Mandarin will be tagged
		including a script subtag, due to the Hant/
		Hans split (not to speak of Hani...).)

For the initial example for the current "thread bundle",
no/nn/nb, there will be no change in either case. And
yes, I think the current situation for Norwegian is
much an accident. Compare the similar case of Frisian,
where the old code was reused for West Frisian, when
new codes were introduced for North Frisian and East
Frisian. The old code was NOT kept as a (would-be)
macrolanguage. My guess is that no repeat of the
"Norwegian mess" was desired, and a narrowing of the
code "fy" was better (accepted without much discussion,
IIRC, for the Language Subtag Registry).

		/kent k

(Going back to being (mostly) silent on this issue.)

---

From: Peter Constable <[email-masked]>
Date: Sun, 25 May 2008 14:50:51 -0700

> From: Leif Halvard Silli [mailto:[email-masked]]

> > Please note very carefully: the definition of macrolanguage entails
> that the range of varieties is treated as a single language in some
> application context.
>
...

> > intelligible to all the others...
>

I think I know what the intent of the text in ISO 639-3 is better than anyone here, whether it reads the same way to all people or not. As for the "empirical investigation that lead a group of languages to be placed in a macrolanguage category," I wonder what you might actually know about that. I happen to know exactly how the current macrolanguage categories got defined, and the only empirical part of it I'm personally aware of was mostly comparison of ISO 639-2 and MARC with content being prepared for Ethnologue 15.

> Macrolanguage definition is suitable

"Typical situations in which this need can occur include the following..." These are three prototypical scenarios, but not the only scenarios. None of these three scenarios hold in several macrolanguage cases, including Quechuan, Zapotec, Cree, Ojibwa...

Peter

---

From: Leif Halvard Silli <[email-masked]>
Date: Mon, 26 May 2008 03:54:50 +0200

> I think I know what the intent of the text in ISO 639-3 is
> better than anyone here, whether it reads the same way to all
> people or not. 

   [ ... ]

> comparison of ISO 639-2 and MARC with content being prepared for
> Ethnologue 15.

1. I suppose Ethnologue has som empirical basis?
2. Was it ISO 639-2 that came up with 'Macrolanguage'?

> "Typical situations in which this need can occur include the
> following..." These are three prototypical scenarios, but not the
> only scenarios. None of these three scenarios hold in several
> macrolanguage cases, including Quechuan, Zapotec, Cree, Ojibwa...

So, the typical examples that ISO 6369-3 mentions, are not 
typical, but exceptions. The largest, most well known, 
macrolanguages are to be ignored. And if there happens to be a 
large degree of inteligibility between them, then then it is an 
exceptional exception.

Ok. I better write it up in my fact book.

I just let it be known that I, unlike some others, have not taken 
part in this debate in the belief that all encompassed languages 
of a Macrolanguage are intelligible.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Mon, 26 May 2008 04:49:38 +0200

> intelligible in all cases (except for spoken 'zh'). But not so.

You cut Peter's statement in the middle.  However, if this is what
you thought - that the encompassed varietes of each macrolanguage
always were mutually intelligible, then OK, that part of the quote
answers your question.

>     not provide the benefits claimed by the extlang proponents.

The benefits are for the minority language users, in contexts of
negotiation between majority and minority encompassed language.
Would be interesting to hear more why you don't see any benefits.

>     like for nn/nb and other instances, are already numerous.

nn/nb is a situation where the the mutual command of all the (two)
encompassed languages by all native speakers is fostered. Wheras
in most other situations there is perhaps only one (and perhaps
sometimes even none) encompassed language that dominates
throughout each "subtag Region".

This confirms what John has said - as I interpret him (and I has
said it too) that it is education that "keeps up" a macrolanguage
situation. Or - perhaps I should say that it is education that
makes a macrolangauge situation useful knowledge for deciding
fallback.

> So a no-extlang model is simpler to explain and apply, and

For whom?

> outweigh the disadvantages in general (including the pedagogical

In the areas where needed, the advantages are constant - not
occasional.

> model, and also formally let would-be-extlangs be langs.

Not my experience. Not sure on what the simplicity claim is based.

> 		Hans split (not to speak of Hani...).)

A good proposal for the no-Extlang situation.

> IIRC, for the Language Subtag Registry).

I shall not speculate about Frisian. But  my guess about no, nn,
nb situation is rooted in what I said about education and
fostering. A language is a dialect with an army behind. It has
always been institutions behind both Nynorsk and Bokmål. And it
has allways been 100% clear that Norwegian/'no' covered both.
-- 
leif halvard silli

---

From: "Kent Karlsson" <[email-masked]>
Date: Mon, 26 May 2008 10:30:44 +0200

> Would be interesting to hear more why you don't see any benefits.

As others have said already:

1) Fallback to language tag prefix does not apply to content
   negotiation (except as an extraordinary measure for misconfigured
   web browsers).

2) Even when fallback to language tag prefix does apply, it is more
   often than not inappropriate to fall back to a macrolanguage. In
   many cases it would be *akin to* letting Norwegian fall back to
   German (or any other Germanic language), just because both languages
   are Germanic (and German is "larger" than Norwegian; English is even
   "larger", but using that in this example would cloud my point since
   most (not very old) people that know Norwegian also know English).
   That would be bad, not only for language understandability reasons,
   but in some cases it can also be politically sensitive. Both of
   which we should stay out of.

3) Language preference lists, for content negotiation and similar, is
   a more general approach, and is easier to apply in a non-extlang
   model than in an extlang model (as examples from others have shown),
   avoiding numerous "q=0" entries to rule out some languages.

> encompassed languages by all native speakers is fostered. Wheras

Even so, and even though 'no' ('nor') is a macrolanguage code for nn/nb
(or nno/nob), extlang does not apply to nn/nb for backwards compatibility
reasons.

> > So a no-extlang model is simpler to explain and apply, and

Everyone. Users, software developers, you and me, this WG, IETF, IANA, ...
A language code can then always be in the first position, not sometimes
mostly arbitrarily forced to be in the second position (as an extlang).
It will not introduce new false fallback to prefix, not introduce new
politically sensitive fallbacks/prefixes, leave users with more easily
managed language preference lists, etc.

	/kent k

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 26 May 2008 12:13:56 +0200

On Mon, May 26, 2008 at 10:30:44AM +0200,
 Kent Karlsson <[email-masked]> wrote 
 a message of 63 lines which said:

> A language code can then always be in the first position, not
> sometimes mostly arbitrarily forced to be in the second position (as
> an extlang).

If we go back to what was planned in RFC 4646, an extlang can be in
the third or fourth position as well :-}
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Mon, 26 May 2008 16:26:44 +0200

>    negotiation (except as an extraordinary measure for misconfigured

This misunderstanding has been stated by both sides. I have 
accepted the facts here.

>    which we should stay out of.

1. If the user has stated a preference of "zh-yum, fr", then "fr" 
  will be preferred before the server starts looking for 
"zh-whatever". Hardly a problem.

2. If, however, neither "fr" nor "zh-yum" nor "zh-whatever" is 
found, causing the web server to try the extraordinary fallback 
that you mention in your first point, then the user will get 'zh', 
if available. Hardly politically problematic.

3. I will tell you (repeat, or perhaps I said it on another list) 
one problem that I see: You might have problems getting Apache's 
ForceLanguagePriority settings to work as you wish. Imagine that 
you offer 'sv' and 'zh-cmn' on your web server, and if you have 
set up 'sv' to be the ForcedLanguagePriority when the user neither 
prefers 'zh-cmn' nor 'sv'. Then a user who prefers "zh-yum, fr" 
will still get "zh-cmn"  (and even 'zh', if you offer it, via the 
"extraordinary fallback") instead of 'sv'. This could be a problem.

However, even if Apach considers "en-GB" a misconfigured browser 
and performs the "extraordinary fallback" to "en", it should  not 
treat "zh-yum" the same way and fallback to "zh". That is one 
issue solved. I don't know if the "normal fallback" can be solved.

As I see it, extlang is supposed to handle the issues dealing with 
negotiation between encompassed languages. It is supposed to 
handle teh problems of minority language users inside a sea of 
dominant encompassed language resources. And it handles such cases 
very well, I think.

But outside that "sea", in the diaspora and such things, the 
extlang solution has the above mentioned problem. However, even 
then, the problem could be rather small, because, it is likely 
that a user visiting a page which is offered in "sv" and "zh-cmn" 
will have a browser that is set up to prefer "sv" or "zh-cmn".

Some would say that if you are in macrolanguage situation where 
ther is close to zero intelligibility between the encompassed 
languges, then this problem would cause problems. Well, no. If we 
have a situation where each encompassed language web site only 
offers one language, then being served the available encompassed 
language equals a "Forced Fallback" situation. If, however, we are 
in a situation - as we often are - were Spanish or another 
language outside the Macrolanguage is used as linguage franca, 
then the browsers will include Spanish in its preferences, and 
then Spanish will always be preferred over the encompassed 
language (unless the user has used the Macrolanguage tag - but 
that would be a problem both with and without extlang).

>    avoiding numerous "q=0" entries to rule out some languages.

The political problems are exaggerated. See above. And there are 
also political advantages.

> (or nno/nob), extlang does not apply to nn/nb for backwards compatibility

The Norwegian case is still valid as an example of what extlang 
can/cannot do for a Macrolanguage situation.

>> > So a no-extlang model is simpler to explain and apply, an	d
>> 
>> For whom?
> 

I answered the claim about "false fallback" and the political 
"dimenson" above.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Mon, 26 May 2008 09:30:21 -0700

> Kent Karlsson

>    often than not inappropriate to fall back to a macrolanguage...

I'd suggest that, in just those cases, it's probably not helpful, in general, to tag content/resources with the macrolanguage ID. Of course, that is just what librarians have sometimes done, though their situation may be exceptional: I suspect they do this when dealing with smaller volumes of content and/or in languages of lower demand such that it is not cost effective to tag the content more specifically.

> managed language preference lists, etc.

This covers concerns expressed to me about complexity of extlang.

Peter

---

From: Peter Constable <[email-masked]>
Date: Mon, 26 May 2008 09:54:34 -0700

> negotiation between encompassed languages.

The reasons extlang was originally proposed were:

- it would allow a set of already-registered tags such as zh-yue to change from being grandfathered to being redundant (they would fall out generatively)

- it would all truncate-from-right fallback to provide a match in cases in which the macrolanguage ID was already in widespread use

The former was relevant only to Chinese.

The latter makes sense only if it can be assumed that content tagged aa-bbb would be a reasonable match for requests for aa. That may be valid in cases such as content in zh-cmn or ar-arb (i.e. a predominant language) or in cases in which the encompassed languages really are mutually intelligible (sh-??? would be fitting examples except that sh is among cases in which extlang would not have been used). It is probably not valid, however, in all the other cases, e.g. when the content is zh-yue or ar-auz. Thus, I think it may be fair to say that when extlang was first proposed the putative benefits wrt right-truncation fallback had not been thought through carefully.

> can/cannot do for a Macrolanguage situation.

Yet, indeed, Norwegian is one case for which it had been assumed extlang would *not* be used.

Peter

---

From: John Cowan <[email-masked]>
Date: Mon, 26 May 2008 13:47:18 -0400

> "zh-whatever". Hardly a problem.

Not quite.

If you specify "zh, fr" you will get the available "zh-*" (or a 300
Multiple Responses result if there is more than one).  Only if there
are none will the server look for "fr-*".

If you specify "zh-yue, fr", you will get the available "zh-yue-*"
resource, or if none, then the available "fr-*"; by extension to HTTP,
some servers will then look for "zh".  In no case will "zh-*" other than
"zh-yue" be returned.

-- 
John Cowan  [email-masked]   http://ccil.org/~cowan
"The exception proves the rule."  Dimbulbs think: "Your counterexample proves
my theory."  Latin students think "'Probat' means 'tests': the exception puts
the rule to the proof."  But legal historians know it means "Evidence for an
exception is evidence of the existence of a rule in cases not excepted from."
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 27 May 2008 06:31:39 +0200

> some servers will then look for "zh".  In no case will "zh-*" other than

After a new test with new files, I stand corrected.

(I have a file which, for some reason, makes Apache behave like I 
said - well, almost. Tested on two computers, Apache 1.3 and 2.)
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru