Subject: [Ltru] Consensus call: extlang

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 22 May 2008 01:26:34 -0700

Hi -

As co-chair...

Later in this message I will be asking some questions.  Please
respond to this mailing list so that the co-chairs can decide
what to instruct the editors to do regarding the general disposition
(*not* detailed wordsmithing!) of the extlang question.

Please respond no later than Saturday, May 24th, wherever you are.
Please DO NOT respond to others' responses.
If you feel compelled to include a rationale with your response,
PLEASE keep it down to a single brief sentence, and don't expect
anyone to respond to it.  If you want a conversation, start a
new thread!

We've discussed this quite enough.  About 600 messages have been
posted this month on this list, mostly on this topic.  After wading
through this mass, I'm quite convinced that while further wordsmithing
(of whatever solution we agree to) will be needed, the arguments for
and against each of the approaches were made quite some time ago.
There are conflicting considerations and analyses, and it's clear
to me there's no perfect answer that will make everyone happy.

Q1: The current draft (http://www.ietf.org/internet-drafts/draft-ietf-ltru-4646bis-14.txt)
    does away with the "extlang" production. Possible responses: (pick ONE)
        A - I like this.
        B - I can live with this.
        C - I object to this.   

Q2: We could go back to using "extlang" in the grammar, requiring its
    use for representing "encompassed" languages that are being added.
    Possible responses:  (pick ONE)
        A - I would like this.
        B - I could live with this.
        C - I would object to this.

Randy

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 22 May 2008 10:04:10 +0100

Q1 = C
Q2 = A

Debbie

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Randy Presuhn
> Sent: 22 May 2008 09:27
> To: LTRU Working Group
> Subject: [Ltru] Consensus call: extlang
>
> Hi -
>
> As co-chair...
>
> Later in this message I will be asking some questions.
> Please respond to this mailing list so that the co-chairs can
> decide what to instruct the editors to do regarding the
> general disposition
> If you feel compelled to include a rationale with your
> response, PLEASE keep it down to a single brief sentence, and
> don't expect anyone to respond to it.  If you want a
> conversation, start a new thread!
>
> We've discussed this quite enough.  About 600 messages have
> been posted this month on this list, mostly on this topic.
> After wading through this mass, I'm quite convinced that
> while further wordsmithing (of whatever solution we agree to)
> will be needed, the arguments for and against each of the
> approaches were made quite some time ago.
> There are conflicting considerations and analyses, and it's
> clear to me there's no perfect answer that will make everyone happy.
>
> Q1: The current draft
> (http://www.ietf.org/internet-drafts/draft-ietf-ltru-4646bis-14.txt)

---

From: Martin Hosken <[email-masked]>
Date: Thu, 22 May 2008 17:13:49 +0700

Dear Randy,

> (*not* detailed wordsmithing!) of the extlang question.

1A
2B

Yours,
Martin Hosken
Script Technologist

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 22 May 2008 14:06:32 +0200

On Thu, May 22, 2008 at 01:26:34AM -0700,
 Randy Presuhn <[email-masked]> wrote 
 a message of 43 lines which said:

> posted this month on this list, mostly on this topic.

I fully agree.

>         B - I can live with this.

Rationale: I have a small preference *against* extlangs (see
<http://www.ietf.org/mail-archive/web/ltru/current/msg10362.html> for
a good rationale) but I feel that one can write a very convincing
argument *for* or *against* extlangs. This is a case where we are
right on the border.

>         C - I would object to this.

Rationale: To me, it has already been discussed and settled. I find
unwise to reopen it.
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Felix Sasaki <[email-masked]>
Date: Thu, 22 May 2008 21:15:52 +0900

Q1: B
Q2: A

Felix

---

From: John Cowan <[email-masked]>
Date: Thu, 22 May 2008 10:00:27 -0400

-- 
You're a brave man! Go and break through the            John Cowan
lines, and remember while you're out there              [email-masked]
risking life and limb through shot and shell,           http://ccil.org/~cowan
we'll be in here thinking what a sucker you are!
        --Rufus T. Firefly
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Ira McDonald" <[email-masked]>
Date: Thu, 22 May 2008 13:11:37 -0400

Q1-B
Q2-A

- Ira

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 22 May 2008 10:13:02 -0700

1: C
2: A

Feel like the question is a bit of a false choice and that fallback rather than the identification of languages has been driving the solution.

>From: [email-masked] [mailto:[email-masked]] On Behalf
>Of Randy Presuhn
>Sent: Thursday, May 22, 2008 1:27 AM
>To: LTRU Working Group

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 22 May 2008 10:29:04 -0700

Q1: (no extlang) A - I like this.
Q2: (extlang) C - I would object to this.

Mark

---

From: Peter Constable <[email-masked]>
Date: Thu, 22 May 2008 15:30:13 -0700

Q1: B

Q2: B
I lean slightly on the C side of this question, but am very concerned that the AV media sector buys into the same tagging scheme as everyone else; if it would help to that end, I’d want it to be clear that grandfathered tags “zh-yue” etc. always remain fully legitimate.

Peter

---

From: Shawn Steele <[email-masked]>
Date: Thu, 22 May 2008 15:42:16 -0700

Q1: B

Q2: B (very mild bias toward A, if that would help other groups.  This is a change from my previously strong opinion)

I agree with Peter:  in particular I don’t want other groups and standards to feel disenfranchised with either solution, particularly if that would cause them to adopt a different system.  If it helps those groups to accept a non-extlang path, I would also want to ensure that Chinese zh-yue, etc. remain a legitimate alias for yue.

- Shawn

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Peter Constable
Sent: Thursday, May 22, 2008 3:30 PM
To: LTRU Working Group
Subject: Re: [Ltru] Consensus call: extlang

Q1: B

Q2: B

Peter

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 22 May 2008 16:01:57 -0700

Either way, zh-yue will remain valid, but not necessarily preferred. Having clear preferences is very important to me. I really don't want to recommend that people use "zh" for Mandarin or have to explain to someone why I think a deprecated tag is preferred in this industry but not others.

>Of Shawn Steele
>Sent: Thursday, May 22, 2008 3:42 PM
>To: Peter Constable; LTRU Working Group
>Q2: B (very mild bias toward A, if that would help other groups.
>This is a change from my previously strong opinion)
>
>I agree with Peter:  in particular I don’t want other groups and
>standards to feel disenfranchised with either solution, particularly
>if that would cause them to adopt a different system.  If it helps
>those groups to accept a non-extlang path, I would also want to
>ensure that Chinese zh-yue, etc. remain a legitimate alias for yue.
>
>- Shawn
>
>Of Peter Constable
>I lean slightly on the C side of this question, but am very
>concerned that the AV media sector buys into the same tagging scheme
>as everyone else; if it would help to that end, I’d want it to be
>clear that grandfathered tags “zh-yue” etc. always remain fully
>legitimate.
>
>
>Peter

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 22 May 2008 16:08:06 -0700

I haven't looked at the exact language around this that we have already, but
I think we should clearly state (if we don't already) that for any cases
where we have such equivalences, implementations SHOULD treat them as the
same on input. (They may do this by internally converting one way or the
other, but that's not our business.)

Mark

On Thu, May 22, 2008 at 4:01 PM, Broome, Karen <[email-masked]>
wrote:

> Either way, zh-yue will remain valid, but not necessarily preferred. Having
> clear preferences is very important to me. I really don't want to recommend
> that people use "zh" for Mandarin or have to explain to someone why I think
> a deprecated tag is preferred in this industry but not others.
>
>
> >I agree with Peter:  in particular I don't want other groups and
> >as everyone else; if it would help to that end, I'd want it to be
> >clear that grandfathered tags "zh-yue" etc. always remain fully
> >legitimate.
> >
> >
> >Peter

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 22 May 2008 16:31:54 -0700

I care more about identification than applications. - K.

Mark wrote:

I haven't looked at the exact language around this that we have already, but I think we should clearly state (if we don't already) that for any cases where we have such equivalences, implementations SHOULD treat them as the same on input. (They may do this by internally converting one way or the other, but that's not our business.)

Mark

---

From: Peter Constable <[email-masked]>
Date: Thu, 22 May 2008 16:58:16 -0700

[changing the subject per Randy's request]

> From: Broome, Karen [mailto:[email-masked]]

> Either way, zh-yue will remain valid, but not necessarily preferred.
> Having clear preferences is very important to me. I really don't want
> to recommend that people use "zh" for Mandarin or have to explain to
> someone why I think a deprecated tag is preferred in this industry but
> not others.

And in another msg,

> I care more about identification than applications.

So, it sounds to me like what you really care about are the following:

- Don't have a recommendation "tag Madarin content as zh"

- Have a single clear recommendation for how to tag each language (one or the other of "cmn" or "zh-cmn" for Mandarin would do).

Is that right?

Peter

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 22 May 2008 17:01:29 -0700

Yes, Peter.

>Sent: Thursday, May 22, 2008 4:58 PM
>To: LTRU Working Group
>Subject: [Ltru] clear guidance on tagging in cases involving a
>related macrolanguage
>
>- Have a single clear recommendation for how to tag each language
>(one or the other of "cmn" or "zh-cmn" for Mandarin would do).
>
>
>Is that right?
>
>
>Peter

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 22 May 2008 18:23:12 -0700

Hi -

As co chair:

PLEASE don't use this thread for discussion!
I'm tallying the rough consensus.  If you want to
spin off a discussion, PLEASE start a new thread!

Randy

---

From: Peter Constable <[email-masked]>
Date: Thu, 22 May 2008 18:40:33 -0700

Could you live without extlang provided those conditions were met, or do you think those conditions can't be met without extlang?

Peter

> Sent: Thursday, May 22, 2008 5:01 PM
> To: Peter Constable
> Cc: LTRU Working Group
> Subject: RE: clear guidance on tagging in cases involving a related
> macrolanguage
>
> Yes, Peter.
>

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 22 May 2008 18:50:07 -0700

I think extlang will likely be a cleaner solution to me than what you might offer, but I'm open to that as long as the preferred tagging and associated semantics are clear.

I think de-CH is a problem. I see more problems like this if we move forward extlangless with multiple "preferred" tag choices for the same thing and subjective distinctions. In the face of that, I start to like ISO 639-6.

I *like* hierarchies.

Karen

________________________________________
From: Peter Constable [[email-masked]]
Sent: Thursday, May 22, 2008 6:40 PM
To: Broome, Karen
Cc: LTRU Working Group
Subject: RE: clear guidance on tagging in cases involving a related     macrolanguage

Peter

---

From: Felix Sasaki <[email-masked]>
Date: Fri, 23 May 2008 11:17:52 +0900

> I think extlang will likely be a cleaner solution to me than what you might offer, but I'm open to that as long as the preferred tagging and associated semantics are clear.

this sounds like your vote

1: C
2: A

could also be

1: B (if there is a clear guidance on tagging in cases involving a related
macrolanguage available)
2: A

but given the long discussion about sec. 
http://tools.ietf.org/html/draft-ietf-ltru-4646bis-14#section-4.1.1
it seems difficult to formulate that guidance.

Felix

>>>> Either way, zh-yue will remain valid, but not necessarily
>>>>         
>>> preferred.
>>>       
>>>> Having clear preferences is very important to me. I really don't
>>>>         
>>> want
>>>       
>>>> to recommend that people use "zh" for Mandarin or have to explain
>>>>         
>>> to
>>>       
>>>> someone why I think a deprecated tag is preferred in this industry
>>>>         
>>> but
>>>       
>>>> not others.
>>>>         
>>> And in another msg,
>>>
>>>

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 22 May 2008 19:47:25 -0700

Yes, that's why I thought the question was a false choice. I could imagine scenarios where I might find it acceptable to use only "cmn". But I'm not convinced this is the best solution given the history of the tag and my needs. I'm not so sure my needs are unique. Don Osborn's situation sounded similar to mine. As did Leif's, though I think I disagree with him about various solutions.

When there starts to be too much preferred tag logic based on subjective judgments (such as the relative dominance of encompassed languages) and there are different tags for the same thing -- that *could* in some rare cases mean something completely different (zh and cmn); and you're about to make nonsense of a rather popular set of grandfathered tags as grandfathered tags go (zh-yue, zh-min-nan, etc.)... Well? I'm having a hard time just writing that sentence, much less explaining that to the people that scowl at my explanation for the de-CH/gsw situation.

I'm thinking a lot of fallback situations in my world are really carefully considered business decisions so my primary concern is precise identification of the language. I figure if I can do that and have confidence in what those tags mean, I can throw any logic I want on top of it.

Regards,

Karen Broome

________________________________________
From: Felix Sasaki [[email-masked]]
Sent: Thursday, May 22, 2008 7:17 PM
To: Broome, Karen
Cc: Peter Constable; [email-masked]
Subject: Re: [Ltru] clear guidance on tagging in cases involving a related macrolanguage

1: C
2: A

could also be

Felix

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 24 May 2008 09:37:21 +0200

C - I object to this.

A - I would like this.

While I agree with Karen that it is a bit of a false choice, I also 
agree with her that it is the identication that it is important, and I 
think that hierarchies is an very important method for identifying 
languges (as is evident for the two encompassed langauges of Norwegian: 
In the RFC, both names begins with the word "Norwegian" => "Norweigan 
Bokmål" and "Norwegian Nynorsk" - despite that the common name amongst 
Norwegians are only "Bokmål" and "Nynorsk".)

(One sentence, right?) ;-)
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 24 May 2008 12:21:23 -0700

Hi -

As a technical contributor...

Q1 - B
Q2 - A

Randy

---

From: Peter Constable <[email-masked]>
Date: Sat, 24 May 2008 14:22:20 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of

> Leif Halvard Silli

> Norwegians are only "Bokmål" and "Nynorsk".)

While it is clear that "Norwegian" is used in the registered language name for nb and nn, that fact does not make it evident that hierarchies are a very important method for constructing tags.

For some tag α, if it is important for some purposes to know that the tag denotes some language variety Lα which is a variety encompassed by a broader-scoped variety Mα, the hierarchical relationship between Lα and Mα *can* be captured structurally within the tag α itself, but that information can also be retained elsewhere with other information maintained regarding α.

Peter

---

From: "Randy Presuhn" <[email-masked]>
Date: Sun, 25 May 2008 19:58:59 -0700

Hi -

As WG co-chair...

Ok, I've tallied the responses.  One was sent directly to me, rather
than to the [email-masked] mailing list.  One was sent after the deadline.
Here's the tally including those two:

Q1: A-4 B-9 C-4
Q2: A-8 B-5 C-4

Excluding those two responses, it looks like:
Q1: A-3 B-9 C-3
Q2: A-7 B-5 C-3

>From this I'm concluding that we have a *rough* consensus to
"reinstate" extlang.

This was not an easy decision, and it was not undertaken lightly,
particularly since we'd been working on the assumption that we'd
polished of the extlang issue some months ago.

However, with equal numbers objecting to each of the proposals,
and a 2-to-1 preference for Q2 among those favoring a proposal,
I think the sense of the WG is clear enough for us to move on.

Randy
ltru co-chair

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 25 May 2008 21:32:02 -0700

This is crazy; we do not have consensus to introduce a major architectural
change from RFC4646.
Mark

---

From: "Randy Presuhn" <[email-masked]>
Date: Sun, 25 May 2008 22:46:41 -0700

Hi -

> From: "Mark Davis" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Sunday, May 25, 2008 9:32 PM
...

As I see it, the alternative, since we've been unable to
reach closure, is to shut down the WG.  Folks interested in the
subject could then request a new working group when they come up
with a solution.  We're far behind schedule, and the debate, while
excruciatingly voluminous, has produced little progress.  I'm
ready to say "to heck with it" and recommend closure of the WG,
as unsatisfactory as I know that would be.

But let's wait to see whether Martin reads the responses differently
from how I did, or if he adds the numbers differently.  If he comes
to the same conclusion as I did, then you may, if you wish, appeal
to our Area Director under RFC 2026 section 6.5.1, challenging whether
we've provided adequate opportunity for your views to be considered,
or claiming that this is a technical error which would put the BCP 47
update "in significant jeopardy."

Randy
ltru co-chair

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 26 May 2008 21:12:51 +0200

On Sun, May 25, 2008 at 07:58:59PM -0700,
 Randy Presuhn <[email-masked]> wrote 
 a message of 33 lines which said:

> As WG co-chair...
> 
> Ok, I've tallied the responses. 

Do note that your message in the archives is truncated
<http://www.ietf.org/mail-archive/web/ltru/current/msg10482.html>,
probably because of the word "F r o m" at the beginning of a line.

The problem has been reported to the secretariat [rt.amsl.com
#7581]. In the mean time, it is better to avoid the word which starts
with F :-)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Martin Duerst <[email-masked]>
Date: Mon, 26 May 2008 20:04:49 +0900

At 14:46 08/05/26, Randy Presuhn wrote:
>from how I did, or if he adds the numbers differently.

I will look into this tomorrow. Sorry for the delay.  Regrads,  Martin.

>If he comes

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 26 May 2008 20:03:11 -0700

I will. The problem with this way of settling the question is that it is
clear that many people who responded still don't understand some of the
basic facts, such as what Peter relates:

"Please note very carefully: the definition of macrolanguage entails that
the range of varieties is treated as a single language in some application
context. That does *not* entail that the encompassed varieties are mutually
intelligible, or that there is any one encompassed variety that is
intelligible to all the others. Certainly no claim is made in ISO 639 that
either is true. Whether or not either is true for any macrolanguage is an
empirical question."

At least one of those voting appears to be against Q1 while not realizing
that Q2 also means no "no-nb" or "no-nn". (At least by the text of pre
December.)

If these technical decisions are to be decided on technical merit, and not
just a popularity contest among those that happen to be on this list, then
we should consider each of the applications of language tags:
identification, lookup, filtering, and Accept-Language, and be able to have
a reasoned judgment on the technical merits.

We will also need to see exact proposed text for the alternative (extlang).
The text of 5 months ago has not undergone the same kind of thorough
examination that draft 14 has.

As to my remark earlier, RFC4646 says nothing about macrolanguages, nothing
about relegating ISO 639 languages -- for whatever reason -- to the extlang
position. And extlang breaks mutual intelligibility of fallback in a
fundamentally different way than 4646. So I do view this as an major
architectural change.

I have been a strong proponent of RFC 4646. But I can't see any way to sell
software developers on the ways in which extlang would require a radical
change, eg that the Accept-Language value meaning 'Mandarin then French'
would be

   - under RFC 4646: "zh, fr"
   - under this proposal: "zh-cmn, zh, fr, zh-cjy;q=0, zh-cpx;q=0,
   zh-czh;q=0, zh-czo;q=0, zh-gan;q=0, zh-hak;q=0, zh-hsn;q=0, zh-mnp;q=0,
   zh-nan;q=0, zh-wuu;q=0, zh-yue;q=0".

Mark

---

From: John Cowan <[email-masked]>
Date: Mon, 26 May 2008 23:21:20 -0400

>    zh-nan;q=0, zh-wuu;q=0, zh-yue;q=0".

Actually, to get *exactly* the same effect under RFC 4646 you'd need to
say "zh-cmn, zh, fr, zh-gan;q=0, zh-hakka;q=0, zh-min-nan;q=0, zh-wuu;q=0;
zh-xiang;q=0, zh-yue;q=0", and that's not counting the deprecated forms.

-- 
In politics, obedience and support      John Cowan <[email-masked]>
are the same thing.  --Hannah Arendt    http://www.ccil.org/~cowan
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 26 May 2008 21:56:13 -0700

Hi -

From: "Stephane Bortzmeyer" <[email-masked]>
To: "Randy Presuhn" <[email-masked]>
Cc: "LTRU Working Group" <[email-masked]>
Sent: Monday, May 26, 2008 12:12 PM
Subject: Re: Consensus call: extlang
...

Argh!  Here it is again, for the sake of the archives:

> From: "Randy Presuhn" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
> Sent: Sunday, May 25, 2008 7:58 PM

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 26 May 2008 22:08:02 -0700

Hi -

> Sent: Monday, May 26, 2008 8:03 PM
...
...

That is precisely why the consensus call was put in very narrow terms of
bringing back the extlang production.  If one brings it back, there remain
plenty of other questions, and it would hardly be fair to bundle those
questions with this one.

Randy

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 27 May 2008 07:45:57 +0200

Mark Davis 2008-05-27 05.03:

> At least one of those voting appears to be against Q1 while not 
> realizing that Q2 also means no "no-nb" or "no-nn". (At least by the 
> text of pre December.)

One of those can tell you that your interpretation of his vote  is 
wrong. (But why not, when everything else I say gets turned and 
twisted.)

The one steadfast opinion I have had, is that since Norwegian and 
Serbo-Croatian and perhaps others two letter coded languages that 
are macrolanguages or encompassed langauges cannot be part of the 
Extlang approach, then no languages should follow the Extlang 
approach. This was also the reason behind why I made a proposal 
which would have made it possible to include even Norwegian.

Because, I thought we would all be given better treatment, better 
understanding of the Macrolanguage issues, if we had the same 
approach for all Macrolanguages. And I also therefore, while 
extlang was ruled out, tried to work with Addision to get 
improvements for the RFC to make is as good as possible, without 
Extlang.

However, finally I came to the conclusion, that an extlang 
approach is what I would have wanted for Norwegian. And therefore 
I decided to vote for Extlang so that at least other languages in 
similar situations could have the benefit of it.

So, I did not believe that I voted for anything else than the 
co-chair asked for. (He refuted my idea before we voted.) And it 
was Doug who suggested a vote on my proposal about allowing 
2-letter codes as extlang codes - so I must ask him to take 
responsibility for that that. When Doug took up the issue, and 
commented it, then I wanted to defend it, though. And it is true 
that I did not publically "put down the arms" before Doug asked 
me, after the vote.

I hope this clarifies.

---

From: Shawn Steele <[email-masked]>
Date: Mon, 26 May 2008 23:11:13 -0700

> As to my remark earlier, RFC4646 says nothing about macrolanguages..
2.2.2 Clearly states that "extlang" is reserved for future standardization.  Enough information is given that eager early adopters could've coded for that as well.

> ... that the Accept-Language value meaning 'Mandarin then French' would be

 *   under RFC 4646: "zh, fr"
 *   under this proposal: "zh-cmn, zh, fr, zh-cjy;q=0, zh-cpx;q=0, zh-czh;q=0, zh-czo;q=0, zh-gan;q=0, zh-hak;q=0, zh-hsn;q=0, zh-mnp;q=0, zh-nan;q=0, zh-wuu;q=0, zh-yue;q=0".

Under RFC 4646 "zh, fr" doesn't mean "Mandarin, then French" it means "Chinese, then french".

Under the latest draft "cmn, zh, fr" also wouldn't mean "Mandarin, then French", it'd be "Mandarin, then Chinese, then French".  Without the "zh" you'd fail to match legacy data.

I'm truly on the fence, leaning a bit toward no-extlang, and if extlang was permitted, then only allow it in explicit cases like Chinese, however your statements seem to ignore the problems with a no-extlang solution.  It is also unclear to me if you want "zh" to be an alias for Mandarin.  From your comments above it would seem that you do.  I don't think that's acceptable, I think that "zh" must allow for the possibility that the language may not be Mandarin.

- Shawn

---

From: Shawn Steele <[email-masked]>
Date: Mon, 26 May 2008 23:17:50 -0700

I'm wondering if there is a compromise?

Specifically I'm wondering if people would consider a no-extlang solution, however allow zh-cmn and zh-yue as a true alias (eg: not grandfathered) of cmn and yue?  My suggestion is that this would be limited to only the specific languages known to be a problem.  This would have its own set of concerns, however perhaps we could continue from the current draft...

Mostly I'm just thinking out loud.  I would like to make sure that most (I'd say all, but there seem to be a few diametrically opposed views) everyone's concerns are addressed in a way they feel comfortable with.

- Shawn

---

From: Peter Constable <[email-masked]>
Date: Tue, 27 May 2008 01:08:55 -0700

> approach...

> similar situations could have the benefit of it.

And yet, not all other macrolanguage cases are similar situations to Norwegian or to Serbo-Croatian.

So is it perhaps still the case that you might have voted against Q1 and for Q2 without realizing that Q2 did not have the same benefits for as many cases as you thought?

Peter

---

From: Peter Constable <[email-masked]>
Date: Tue, 27 May 2008 01:54:13 -0700

From: [email-masked] [mailto:[email-masked]] On Behalf Of Shawn Steele

>> * under this proposal: "zh-cmn, zh, fr, zh-cjy;q=0, zh-cpx;q=0, zh-czh;q=0,
>> zh-czo;q=0, zh-gan;q=0, zh-hak;q=0, zh-hsn;q=0, zh-mnp;q=0, zh-nan;q=0,
>> zh-wuu;q=0, zh-yue;q=0".
>

Mark: Shawn has a valid point here: you present the two from unequal sets of premises without clarifying that difference, thus (unintentionally, I'm sure) giving a subtly deceiving argument.

All of the zh-??? tags have be valid for 4646 as they would be for a 4646bis with extlang. Thus, all other things being equal, the Accept-Language value "zh, fr" should have just the same meaning and effect in the latter as for the former, as should the value "zh-cmn, zh, ...".

I think that perhaps the contrast you present here is best described not as that between 4646 and a 4646bis *specifically with extlang*, but rather between a situation in which "zh" is predominantly used and a situation in which it is recommended that content be tagged for the specific, encompassed Chinese language rather than being tagged generically as "zh" -- a situation that could obtain under 4646 as well as under 4646bis. And if that situation obtained in a 4646-without-extlang world, then the corresponding Accept-Language value would be "cmn, zh, fr, cjy;q=0, cpx;q=0, ..."

While I was the proto-proponent for extlang, I've come to lean toward no-extlang since, to the extent that I've been able to think through scenarios,

- it seems to me that the benefits are far more limited than what I initially was supposing, and

- because one of the benefits is to slip fallback relationships into a small set of tags whereas in general good fallback needs a lot more -- true for cases related to macrolanguages let alone for the very many more cases not related to any macrolanguages.

I'm concerned, though, about such not-quite-careful-enough cases being made for (or against): someone is likely to sense something not quite right and come back, perhaps with an equally slightly-flawed case, in which event we end up not making any progress. So, to your comment,

> identification, lookup, filtering, and Accept-Language, and be
> able to have a reasoned judgment on the technical merits

I would only add: we need to do this with *carefully* reasoned judgment, pausing to make sure the arguments presented really do stand up.

Peter

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 27 May 2008 10:27:07 +0100

Peter and Mark wrote:

> > identification, lookup, filtering, and Accept-Language, and
> be able to
> > have a reasoned judgment on the technical merits
>
> I would only add: we need to do this with *carefully*
> reasoned judgment, pausing to make sure the arguments
> presented really do stand up.

I think this is probably a good way forward.  It would be best to start a
new thread for each application.  I would also add backward compatibility
and historical usage to the pot.

Best regards

Debbie

> Behalf Of Peter Constable
> Sent: 27 May 2008 09:54
> To: LTRU Working Group
> Behalf Of Shawn Steele
> >> * under this proposal: "zh-cmn, zh, fr, zh-cjy;q=0, zh-cpx;q=0,
> >>zh-czh;q=0,  zh-czo;q=0, zh-gan;q=0, zh-hak;q=0, zh-hsn;q=0,
> >>zh-mnp;q=0, zh-nan;q=0,  zh-wuu;q=0, zh-yue;q=0".
> >
> > Under RFC 4646 "zh, fr" doesn't mean "Mandarin, then
> French" it means
> > "Chinese, then french".
>
> Mark: Shawn has a valid point here: you present the two from
> unequal sets of premises without clarifying that difference,
> thus (unintentionally, I'm sure) giving a subtly deceiving argument.
>
> All of the zh-??? tags have be valid for 4646 as they would
> be for a 4646bis with extlang. Thus, all other things being
> equal, the Accept-Language value "zh, fr" should have just
> the same meaning and effect in the latter as for the former,
> as should the value "zh-cmn, zh, ...".
>
> I think that perhaps the contrast you present here is best
> described not as that between 4646 and a 4646bis
> *specifically with extlang*, but rather between a situation
> in which "zh" is predominantly used and a situation in which
> it is recommended that content be tagged for the specific,
> encompassed Chinese language rather than being tagged
> generically as "zh" -- a situation that could obtain under
> 4646 as well as under 4646bis. And if that situation obtained
> in a 4646-without-extlang world, then the corresponding
> Accept-Language value would be "cmn, zh, fr, cjy;q=0, cpx;q=0, ..."
>
>
> While I was the proto-proponent for extlang, I've come to
> lean toward no-extlang since, to the extent that I've been
> able to think through scenarios,
>
> - because one of the benefits is to slip fallback
> relationships into a small set of tags whereas in general
> good fallback needs a lot more -- true for cases related to
> macrolanguages let alone for the very many more cases not
> related to any macrolanguages.
>
> I'm concerned, though, about such not-quite-careful-enough
> cases being made for (or against): someone is likely to sense
> something not quite right and come back, perhaps with an
> equally slightly-flawed case, in which event we end up not
> making any progress. So, to your comment,
>
> > identification, lookup, filtering, and Accept-Language, and
> be able to
> > have a reasoned judgment on the technical merits
>
> I would only add: we need to do this with *carefully*
> reasoned judgment, pausing to make sure the arguments
> presented really do stand up.
>
>
>
> Peter
>
>

---

From: Martin Duerst <[email-masked]>
Date: Tue, 27 May 2008 17:27:17 +0900

[co-chair hat on]

Shawn, others, please rest assured that if we can find something that makes
more people happier than either of the two solutions that were recently
up for the consensus call, we will all I guess be very glad to go for that.

In the meantime, everybody, please change your subject a bit more often.

Regards,    Martin.

At 15:17 08/05/27, Shawn Steele wrote:
>Specifically I'm wondering if people would consider a no-extlang solution, 
>however allow zh-cmn and zh-yue as a true alias (eg: not grandfathered) of 
>cmn and yue?  My suggestion is that this would be limited to only the 
>specific languages known to be a problem.  This would have its own set of 
>concerns, however perhaps we could continue from the current draft...
>
>Mostly I'm just thinking out loud.  I would like to make sure that most 
>(I'd say all, but there seem to be a few diametrically opposed views) 
>everyone's concerns are addressed in a way they feel comfortable with.
>
>- Shawn

---

From: Martin Duerst <[email-masked]>
Date: Tue, 27 May 2008 18:30:48 +0900

[co-chair hat on]

At 12:03 08/05/27, Mark Davis wrote:
>I will. The problem with this way of settling the question is that it is clear that many people who responded still don't understand some of the basic facts,

Some people have been involved in this work for very long,
some others have joined very recently. Each person clearly
has a different background. For example, some focus more
on tagging, others focus more on processing, and so on.

>If these technical decisions are to be decided on technical merit, and not just a popularity contest among those that happen to be on this list,

The way the IETF works is to make sure all the technical
arguments are duely discussed. This discussion has been
going on for a very long time, so it's difficult to claim
that the arguments aren't on the table. (In particular, your
arguments are very nicely summarized on a page of yours
(Google document), which I have (re)read just recently.)
As we have seen, people can draw different conclusions
from all these arguments.

The IETF process is geared towards making the best technical
solution win, but when there are honest differences about
what's the best technical solution, it's very difficult to
satisfy everybody. That's why the IETF has the concept of
"rough" consensus.

>then we should consider each of the applications of language tags: identification, lookup, filtering, and Accept-Language, and be able to have a reasoned judgment on the technical merits.

I assume everybody has done this. They may have weighted
different applications differently, or may have some
additional reasons for their judgement.

>We will also need to see exact proposed text for the alternative (extlang).

Of course, in due course.

>The text of 5 months ago has not undergone the same kind of thorough examination that draft 14 has.

My rough guess is that at least some of the text that was worked out
for draft 14 and before can be reused, with some changes, in particular
e.g. the text talking about 'zh' and whether and how much it means
Mandarin.

>As to my remark earlier, RFC4646 says nothing about macrolanguages,

The word 'macrolanguage' indeed doesn't appear in RFC 4646.

I might not be overly surprised if somebody who had only read
RFC 4646, and hadn't been involved in its production, would
come up with some claim like "a major architectural change from RFC4646".
However, you have been closely involved in the production of RFC 4646
(being one of it's editors) and in the email discussions surrounding
it and RFC 4646bis.

If you, or anybody else closely involved with this work, wants to
claim that they had absolutely no idea that the extlang production
in RFC 4646 was intended, at least among else if not primarily or
only, for encompassed languages, or more concretely, for cases such
as zh-yue and friends, then I'd be extremely surprised.

Also, please note that our charter says:

The working group will prepare an update to the Language Subtag
Registry procedures to allow the use of 3-letter code elements
from ISO 639-3 as primary language subtags or extended language
subtags as appropriate.

So it at least leaves open the possibility to use 639-3 codes
for extended language subtags. It is your due right to claim
that no 639-3 letter code is appropriate for an extended language
subtag, but there are obviously others who think differenty.

Please note that what I try to say above isn't in any way a technical
argument for or against extlangs; I'm just not sure where your
claim is comming from that the recent decision to go back to extlang
is "a major architectural change from RFC 4646".

>nothing about relegating ISO 639 languages -- for whatever reason -- to the extlang position.

'relegating' is an expression of a value judgement,
not a technical argument.

>On Sun, May 25, 2008 at 10:46 PM, Randy Presuhn <<mailto:[email-masked]>[email-masked]> wrote:  
>Hi -
>> From: "Mark Davis" <<mailto:[email-masked]>[email-masked]> 
>> To: "Randy Presuhn" <<mailto:[email-masked]>[email-masked]> 
>> Cc: "LTRU Working Group" <<mailto:[email-masked]>[email-masked]> 

I have been working on this, but need a bit more time, sorry.

Regards,   Martin.

><mailto:[email-masked]>[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Tue, 27 May 2008 17:36:38 +0900

[co-chair hat on]

I don't think there is any specific IETF tradition on this point,
but I'd advise people to refrain from second-guessing other's
statements of preference.

Regards,    Martin.

...

>So is it perhaps still the case that you might have voted against Q1 and 
>for Q2 without realizing that Q2 did not have the same benefits for as many 
>cases as you thought?
>
>
>
>Peter

---

From: Martin Duerst <[email-masked]>
Date: Tue, 27 May 2008 19:06:42 +0900

[technical hat on]

>zh-xiang;q=0, zh-yue;q=0", and that's not counting the deprecated forms.

Yes, and it's not covering languages that haven't been registered
(e.g. zh-cjy, Jinyu Chinese, which may simply have been tagged 'zh'
in lack of anything better), so it seems impossible to get *exactly*
the same, anyway.

Regards,    Martin.

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 27 May 2008 20:24:46 +0200

Peter Constable 2008-05-27 10.08:
 >>Behalf Of  Leif Halvard Silli:

>> approach is what I would have wanted for Norwegian. And
>> therefore I decided to vote for Extlang so that at least
>> other languages in similar situations could have the benefit
>> of it.

> And yet, not all other macrolanguage cases are similar 
> situations to Norwegian or to Serbo-Croatian.
> 
> So is it perhaps still the case that you might have voted 
> against Q1 and for Q2 without realizing that Q2 did not have 
> the same benefits for as many cases as you thought?

I am thankful for the Co-Chair who adviced against second 
guessing. I feel I've second guessed a lot. Never the less, I'll 
at least provide some thoughts as response, but under a new 
heading, as we were adviced.

Let's look at Sami. Those languages are not grouped under a 
Macrolanguage. But could the Sami language(s) be worth an upgrade 
to "Macrolanguage" situation?

Firstly, the encompassed languages of a Macrolangauge always share 
some lingustic "thing" that makes it possible to group them 
together - I suppose encompassed languages always share that much.

The Macrolanguage "thing", then, is that there is one or several 
Areas of Life where these languages are considered as one.

Hence, I wonder: We in Norway often speak about the Sami languages 
(plural) as just "the Sami language". So, for us, these languages 
are often considered as one thing. I therefore suppose that this 
qualifies for Macrolanguage status.

However, the Sami languages are not mutually intelligible, at 
least not generally. In Norway, though, Northern Sami is pretty 
much the dominating encompassed language. And since Norway is also 
  the place in Scandinavia - and likely also in Russia - were the 
sami population are "best off", in many ways, it perhaps has some 
dominating role also outside Norway. (Sami reindeer groups often 
moved/move accross the borders, anyhow.)

If, for simplicity, we say that samis all have a sami languag as 
primary languages, then as secondary language they might have 
Swedish, Norwegian, Finnish, Russian. The Samis in the Russian 
Federation will also eventually write their own languages in 
Cyrillic script.

Now, the million dollar question: Macrolanguage? Would it have 
benefitted them? And secondly, would it have benefitted these 
groups to have Extlang? And what about technially solutions - 
would the RFC and syntax rules stand in the way?

Answers:

1. Technical first: Currently, only Northern Sami has a twoletter 
subtag, 'se'. I think that for the Sami language to be "upgraded" 
to Macrolanguage, then the "se" tag would have had to be change 
meaning and become a Macrolanguage tag for all Sami languages. (I 
don't know if, historically, it has had such a broad meaning 
earlier?)

Would it be technical possible to upgrade 'se' to Macrloanguag 
tag?  If not, are there other ways to get an upgrade? (The whole 
Macrolanguage and Extlang thing becomes very accidental if the 
fact that 'se' has been registered long ago can be used against 
it, in this case.)

2. Macrolanguage at all Would it have benefitted the Sami to have 
a Macrolanguag tag? Well, should it not be much simpler for users 
and implementers if we could "catch" any Sami with the "se" tag? 
The Sami have, and try to foster a cross border Sami identity. 
They have a Sami flag used in all the different regions where they 
are present.  We can simply look at Norway: The smalles Sami group 
is very small. I think a Macrolangauge tag that catched also that 
group, would be good - for users and implementors.

3. Extlang: Thanks to John, I now understand language negotiation 
a little bit better ... Extlang would mean that 'se' would be used 
much more, all over. It would mean fewer and more general tags, 
which coudl be made more spesific by adding the extlang tag.

Would this be good? Yes, I think it would be good. I think it 
would be simpler and lead to more general but on the whole more 
correct tagging. (I think Addison made that point once.) I also 
think Sami people would like it. Then they could, via the use of 
the 'se' tag, browse Cyrillic Sami pagese in Russia via the 'se' 
tag for instanse.

If you have an thoughts regarding when it would be bad, then 
please provide them.

So, to answer your question, Peter: Yes, I see a value of 
Macrolanguage and Extlang also when languages are not mutually 
intelligble. And I knew this when I voted. Hence, the question of 
whether I would have voted the same way, does not apply.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Tue, 27 May 2008 20:30:52 +0200

Leif Halvard Silli 2008-05-27 20.24:

> I feel I've second guessed a lot

Should be "I feel I've **been** second guessed a lot". ;-)
-- 
leif halvard silli

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 27 May 2008 11:50:49 -0700

Those are good additions.

---

From: "Don Osborn" <[email-masked]>
Date: Tue, 27 May 2008 14:55:54 -0400

Leif, The situation of Sami as you describe it is similar to that observed
for a lot of languages, for instance many in Africa. Ultimately we're
talking about amendments to the existing ISO 639 standards, and for that
there needs to be more input from experts on the languages. That input can
come ad hoc as people observe it, or as workshops 

(I'll cc to A12n-collab for the info of people there concerning the
impending decisions regarding language tagging, which has to do with how the
terms in ISO 639 are used, not the content of the ISO 639 standard.)

Don Osborn
Bisharat.net

> Sent: Tuesday, May 27, 2008 2:25 PM
> To: Peter Constable
> Cc: LTRU Working Group
> Subject: [Ltru] Macrolanguage, Extlang. The Sami language situatation
> as example
> 
...

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 27 May 2008 11:38:30 -0700

That's a good point, John. My corrected text should would read:

I want to get Mandarin and French. For backwards compatibility, I have to
use "zh" for Mandarin, now and the the foreseeable future (I'd pointed this
out before).

What I'd like to say is

As John points out,  RFC4646 is already complicated by codes with zh-
prefixes, and what I actually have to say is:

   - under RFC 4646: "zh-cmn, zh, fr, zh-gan;q=0, zh-hakka;q=0,
   zh-min-nan;q=0, zh-wuu;q=0; zh-xiang;q=0, zh-yue;q=0"

And under the proposed extlang additions, I have to say:

   - under this proposal: "zh-cmn, zh, fr, zh-cjy;q=0, zh-cpx;q=0,
   zh-czh;q=0, zh-czo;q=0, zh-gan;q=0, zh-hak;q=0, zh-hsn;q=0, zh-mnp;q=0,
   zh-nan;q=0, zh-wuu;q=0, zh-yue;q=0; zh-hakka;q=0, zh-min-nan;q=0,
   zh-yue;q=0"

Thus RFC4646 is already complicated by these codes, and the proposed change
will just make it worse.

To choose a clearer example, if I want Standard Arabic and French:

What I'd like to say, and what I use now under RFC4646 is:

   - "ar, fr"

And under the proposed extlang additions, I have to say:

   - "ar, fr, ar-aao;q=0, ar-abh;q=0, ar-abv;q=0, ar-acm;q=0, ar-acq;q=0,
   ar-acw;q=0, ar-acx;q=0, ar-acy;q=0, ar-adf;q=0, ar-aeb;q=0, ar-aec;q=0,
   ar-afb;q=0, ar-ajp;q=0, ar-apc;q=0, ar-apd;q=0, ar-arq;q=0, ar-ars;q=0,
   ar-ary;q=0, ar-arz;q=0, ar-auz;q=0, ar-avl;q=0, ar-ayh;q=0, ar-ayl;q=0,
   ar-ayn;q=0, ar-ayp;q=0, ar-bbz;q=0, ar-pga;q=0, ar-shu;q=0, ar-ssh;q=0"

Shawn,
Yes, you're right: "zh, fr" means "Chinese and French". The ISO standard
that RFC4646 is based on is woefully underspecified, so you don't know
whether that "Chinese" means Standard Chinese or also includes other
languages that people call Chinese. (Magically, "German" means only
"Standard German", and "French" means "Standard French", and so on, but
"Arabic" doesn't mean "Standard Arabic",...) The text in draft 12 makes it
clear what the extent of zh is. Yet for now and into the future, if I want
Mandarin, I have to include "zh". ("zh-cmn" has been around for almost 3
years (2005-07-15), but we are seeing **no** uptake in any web documents or
Accept-Language values.)

As far as a compromise goes, I could even live with allowing either extlang
or not, since they are not semantically or syntactically the same. Then the
user has the choice of indicating s/he wants

   - Cantonese alone ("yue"), or Cantonese-with-fallback ("zh-yue"), or
   - Babalia Creole Arabic ("bbz") alone, or Babalia Creole
   Arabic-with-fallback ("ar-bbz").

Peter,
I agree with what you were saying, I didn't intend for it to be misleading.
I was really thinking about the syntactic matching, since extlang is a
syntactic device, and Accept-Language works in terms of that syntax. If we
are dealing with semantic matching, then the situation is a bit different.

Martin,
>> "consider each of the applications of language tags: identification,
lookup, filtering, and Accept-Language, and be able to have a reasoned
judgment on the technical merits."
> I assume everybody has done this.

Based on the comments, I really doubt that. Do you really think that
everyone has considered the technical ramifications of using this with
Accept-Language, even now?

I dem fall, chani di üeberasche ;-)

I can at least speak for myself -- and I was "closely involved" -- that the
extlang mechanism was to allow us the syntactic wiggle room we needed should
we decide to use it for 639-3 in any way. We knew about macrolanguages, but
we had *not* weighted all of the implications at the time, and had not
committed to the use for macrolanguages, because we didn't know what shape
639-3 might take. We even allowed for multiple extlangs in a tag.

Mark

On Mon, May 26, 2008 at 8:21 PM, John Cowan <[email-masked]> wrote:

> are the same thing.  --Hannah Arendt    http://www.ccil.org/~cowan<http://www.ccil.org/%7Ecowan>
>

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 27 May 2008 12:02:49 -0700

Hi -

> To: "John Cowan" <[email-masked]>
> Cc: "Randy Presuhn" <[email-masked]>; "LTRU Working Group" <[email-masked]>
> Sent: Tuesday, May 27, 2008 11:38 AM
...
> (Magically, "German" means only
> "Arabic" doesn't mean "Standard Arabic",...)

I'm very surprised by this claim.  I thought 'de' encompassed any variety
of German that wasn't covered by some other tag (like gsw), just as 'fr'
encompasses many flavors of French.  It would be only slightly more
extreme to claim that 'en' means American English, simply because that
variety makes up a significant percentage of the content tagged 'en'.

Regardless of whether we use extlangs, it's becoming clearer
and clearer to me that we should be careful to advise
*against* practices like using naked 'zh' for content known to be Mandarin,
or naked 'ar' for content known to be standard Arabic, even when there's
substantial legacy data.  To do otherwise would lead to a practical narrowing
of the subtags' meanings, and to dangerous analogies like "de is limited to
standard German." That would be a very bad thing.

Randy

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 27 May 2008 20:10:10 +0100

We are not here to classify languages.  We do not have that level of
knowledge in this working group.  It is totally out of scope.

We are here to facilitate the availability of the ISO 639-3 code within the
LSR via the design of an RFC - RFC4646bis.

Best regards

Debbie

> Behalf Of Leif Halvard Silli
> Sent: 27 May 2008 19:25
> To: Peter Constable
> Cc: LTRU Working Group
> Subject: [Ltru] Macrolanguage,Extlang. The Sami language
> situatation as example
>
> approach
> >> is what I would have wanted for Norwegian. And therefore I
> decided to
> >> vote for Extlang so that at least other languages in similar
> >> situations could have the benefit of it.
>
>
>

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 27 May 2008 20:20:14 +0100

Randy wrote:

> *against* practices like using naked 'zh' for content known
> to be Mandarin, or naked 'ar' for content known to be
> standard Arabic, even when there's substantial legacy data.
> To do otherwise would lead to a practical narrowing of the
> subtags' meanings, and to dangerous analogies like "de is
> limited to standard German." That would be a very bad thing.

Quite so!

Debbie

> Sent: 27 May 2008 20:03
> Subject: [Ltru] Does 'de' really mean "only standard German"?
>
> Hi -
>
> > "Standard German", and "French" means "Standard French", and so on,
> > but "Arabic" doesn't mean "Standard Arabic",...)
>
> I'm very surprised by this claim.  I thought 'de' encompassed
> any variety of German that wasn't covered by some other tag
> (like gsw), just as 'fr'
> encompasses many flavors of French.  It would be only
> slightly more extreme to claim that 'en' means American
> English, simply because that variety makes up a significant
> percentage of the content tagged 'en'.
>
> *against* practices like using naked 'zh' for content known
> to be Mandarin, or naked 'ar' for content known to be
> standard Arabic, even when there's substantial legacy data.
> To do otherwise would lead to a practical narrowing of the
> subtags' meanings, and to dangerous analogies like "de is
> limited to standard German." That would be a very bad thing.
>
> Randy
>

---

From: John Cowan <[email-masked]>
Date: Tue, 27 May 2008 15:26:40 -0400

Randy Presuhn scripsit:

> I'm very surprised by this claim.  I thought 'de' encompassed any
> variety of German that wasn't covered by some other tag (like gsw),

It does encompass national varieties as well as different orthographic
reforms of Standard German.  But it does not cover the various other
West Germanic languages (Mundarten), High or Low, spoken in Germany.

How do we know this?  The ISO 639-2/639-3 equivalent of 'de' is 'deu',
and when we look at http://www.sil.org/iso639-3/documentation.asp?id=deu
(which defines 'deu'), we are pointed to encyclopedic information at
http://www.ethnologue.com/show_language.asp?code=deu , which makes it
clear that the denotation of 'de' is Standard German, the language
spoken by almost a million people in Kazakhstan.

> just as 'fr' encompasses many flavors of French.

For whatever reasons, Ethnologue (and therefore ISO 639-3) does not
distinguish the various langues d'oil except for Picard and Walloon,
so all the rest are indeed lumped under 'fr'.

-- 
A rose by any other name                            John Cowan
may smell as sweet,                                 http://www.ccil.org/~cowan
but if you called it an onion                       [email-masked]
you'd get cooks very confused.          --RMS
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Shawn Steele <[email-masked]>
Date: Tue, 27 May 2008 12:52:05 -0700

Mark’s examples for accept-lang, and the other discussions lead me to believe that there’s no simple solution that will solve lookup/matching/whatnot.  I don’t think that the “perfect” behavior for all applications can be had just by choosing the legal tags wisely, and I think that we’ll need to address the impact on 4647 separately.

With zh-HK you may want to match zh-Hant or zh.  With cmn-HK you may want to match cmn-Hant or zh-HK or whatever.  With zh-cmn-HK you may want to match zh-HK or zh-Hant or whatever.  So I don’t think the “simple” remove from right syntax works any more, even with 4646, and even if you normalize the names to whatever’s appropriate for your app.

If we concede that lookup isn’t going to be simple, then for 4646bis we “merely” have to pick tags that identify the language.

Mark provided some interpretations of the meanings of the tags:

As far as a compromise goes, I could even live with allowing either extlang or not, since they are not semantically or syntactically the same. Then the user has the choice of indicating s/he wants

 *   Cantonese alone ("yue"), or Cantonese-with-fallback ("zh-yue"), or
 *   Babalia Creole Arabic ("bbz") alone, or Babalia Creole Arabic-with-fallback ("ar-bbz").
To that example I would add for clarity:

 *   Unspecified Chinese (“zh”).
So which label(s) best describe the value(s) we need to tag?

- Shawn

---

From: "Don Osborn" <[email-masked]>
Date: Tue, 27 May 2008 16:04:03 -0400

Hi Debbie, You are correct on this point. I do however think however that
work such as that on RFC4646bis and other efforts such as writing of locale
data (a subproject of the new PanAfrican Localisation Network project will
coordinate efforts for up to 100 locales for African languages soon) can
provide very useful information back to the standard. Basically in using or
applying the codes in ISO 639 to various practical applications, we get a
better idea of gaps, unmet needs, and even problems with the existing set of
codes. 

Of course this is not a discussion to pursue on the LTRU list, but I hope
there is no problem to have such issues mentioned.

Don 

PS- For the cc to A12n-collab, more info on RFC 4646 is available at
http://tools.ietf.org/html/rfc4646 , and on RFC's generally at
http://en.wikipedia.org/wiki/Request_for_Comments 

> Debbie Garside
> Sent: Tuesday, May 27, 2008 3:10 PM
> To: 'Leif Halvard Silli'
> Cc: 'LTRU Working Group'
> Subject: Re: [Ltru] Macrolanguage, Extlang. The Sami language
> situatation as example
> 
...

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 27 May 2008 13:31:24 -0700

Hi -

> From: "John Cowan" <[email-masked]>
> Sent: Tuesday, May 27, 2008 12:26 PM
> Subject: Re: [Ltru] Does 'de' really mean "only standard German"?
...
...

Perhaps this would be a question for [email-masked], but I'll ask
it here:  how then would one tag the "Stadtsprache" (city dialect) of Mannheim?
While clearly not "standard German", it seems quite a stretch to lump it
with the Pfaelzisch one might run into among the elderly in the countryside.

Randy

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 27 May 2008 21:36:56 +0100

Hi Don

IMHO, possible use cases of the RFC, needs, unmet needs and gaps are
relevant to this WG and the development of the standard.  Where we need to
draw the line is when the discussion comes into the realm of re-designing
ISO 639 or making language classification decisions.  This is absolutely
beyond the scope of this WG and should be directed to ISO TC37/SC2/WG1 or
the ISO 639 JAC.  As this RFC is way over the timescale originally envisaged
for completion it is in everyone's interest to try to resolve the issues
that are within scope rather than focussing on issues that cannot be
resolved other than by external parties.  But this is just my humble opinion
in the light of trying to get the important issues, such as extlang/no
extlang resolved, the ultimate call is with the co-chairs.

Best wishes

Debbie

> From: Don Osborn [mailto:[email-masked]]
> Sent: 27 May 2008 21:04
> To: [email-masked]; 'Leif Halvard Silli'
> Cc: 'LTRU Working Group'; 'A12n tech support'
> Subject: RE: [Ltru] Macrolanguage, Extlang. The Sami language
> situatation as example
>
> Hi Debbie, You are correct on this point. I do however think
> however that work such as that on RFC4646bis and other
> efforts such as writing of locale data (a subproject of the
> new PanAfrican Localisation Network project will coordinate
> efforts for up to 100 locales for African languages soon) can
> provide very useful information back to the standard.
> Basically in using or applying the codes in ISO 639 to
> various practical applications, we get a better idea of gaps,
> unmet needs, and even problems with the existing set of codes.
>
> Of course this is not a discussion to pursue on the LTRU
> list, but I hope there is no problem to have such issues mentioned.
>
> Don
>
> http://tools.ietf.org/html/rfc4646 , and on RFC's generally
> at http://en.wikipedia.org/wiki/Request_for_Comments
>
> > Of Debbie Garside
> > We are here to facilitate the availability of the ISO 639-3 code
> > within the LSR via the design of an RFC - RFC4646bis.
> >
> > Best regards
> >
> > Debbie
> >
> > > Of Leif Halvard Silli
> situatation

---

From: "Gerard Meijssen" <[email-masked]>
Date: Tue, 27 May 2008 22:55:17 +0200

Hoi,
There are clear differences between Ethnologue and the ISO-639-3. There are
languages in Ethnologue that do not exist in the ISO-639--3 and the other
way round. It is not safe to consider Ethnologue to have the authority that
the standard has.
Thanks,
      Gerard

On Tue, May 27, 2008 at 9:26 PM, John Cowan <[email-masked]> wrote:

> may smell as sweet,
> http://www.ccil.org/~cowan <http://www.ccil.org/%7Ecowan>

---

From: John Cowan <[email-masked]>
Date: Tue, 27 May 2008 17:16:24 -0400

Gerard Meijssen scripsit:

> way round. 

Only in very minor ways.

1) Ethnologue doesn't handle ancient or artificial languages -- the
sub-registrar for them is Linguist List.

2) Macrolanguages aren't in Ethnologue either.

3) There may be some version skew where 639-3 has been updated but
Ethnologue has not.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
In computer science, we stand on each other's feet.
        --Brian K. Reid
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Tue, 27 May 2008 17:23:43 -0400

> with the Pfaelzisch one might run into among the elderly in the countryside.

Well, you have your choice between de-mannheim or pfl-mannheim.  Perhaps
both are worth registering.  I certainly don't claim to know.

-- 
Andrew Watt on Microsoft:                       John Cowan
Never in the field of human computing           [email-masked]
has so much been paid by so many                http://www.ccil.org/~cowan
to so few! (pace Winston Churchill)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 27 May 2008 14:55:12 -0700

ISO 639-1 and ICU 639-2 *didn't* do this in the RFC 4646 time frame, and so
RFC 4646 doesn't either.

ISO 639-3 chose to interpret 'de'/'deu' as "Standard German", and 'ar'/'arb'
as "Any Arabic", but someone reading 639-1/2 and applying RFC 4646 *in good
faith and with the information available in the standard* would have no idea
that there was any fundamental difference in the way that they were going to
be defined in 639-3.

ISO 639-3 does have much more concrete definitions (all to the good), but
someone implementing ISO 639-1/2 or RFC 4646 or any of its predecessors
would have no idea that German would get defined with one model, and Arabic
get defined with a completely different model. It was reasonable for someone
implementing RFCs 1766, 3066, and 4646 to interpret "de" as Standard German
(Hochdeutsch) and "zh" as Standard Chinese (Mandarin). It would *also* have
been reasonable for someone to interpret 'de' as Any German (and up until
2006-03-08, encompassing Swiss German), and 'zh' as Any Chinese (thus
encompassing Cantonese).

The standard was underspecified.

I am not arguing against the way ISO 639-3 has done things; that is water
under the bridge -- but we need to recognize that because of the imprecision
of the previous ISO 639 standards, we can't make reasonable interpretations
of RFCs 1766, 3066, and 4646, widely followed in industry, be
non-conformant.

Mark

---

From: "Broome, Karen" <[email-masked]>
Date: Tue, 27 May 2008 15:09:48 -0700

Mark writes:

I have been a strong proponent of RFC 4646. But I can't see any way to sell software developers on the ways in which extlang would require a radical change, eg that the Accept-Language value meaning 'Mandarin then French' would be

• under RFC 4646: "zh, fr"
• under this proposal: "zh-cmn, zh, fr, zh-cjy;q=0, zh-cpx;q=0, zh-czh;q=0, zh-czo;q=0, zh-gan;q=0, zh-hak;q=0, zh-hsn;q=0, zh-mnp;q=0, zh-nan;q=0, zh-wuu;q=0, zh-yue;q=0".

Karen responds:

Of course, if you had actually asked for Mandarin using a precise code for Mandarin (either "cmn" or "zh-cmn"), that list doesn't look so ugly, does it?

But you didn't ask for Mandarin, you asked for Chinese. Your request as written *should* match all those Chinese variants because "zh" includes all those languages. Given that, your first request may pull back Cantonese, Nan, or Hakka and will never be precise. Given that there is a certain hierarchical relationship between the languages in ISO 639-1/2 and 639-3, this is what I would want software to represent.

This is where I think we are prioritizing generalized fallback over precision in identification -- even if what we are precisely identifying is intentionally vague. I lose the ability to identify "some kind of Chinese, I don't know what kind" if zh is thought to be synonymous with Mandarin. (A real need given the history of these tags.) I also will not know what the user intended if they use the tag "zh". When it becomes possible to tag more specifically, we won't be able to know whether the decision to use the more general tag ("zh") was a conscious choice or not.

zh is a bad tag; its semantics have always been muddy. We are going to rather great lengths to avoid deprecating it.

Regards,

Karen Broome

---

From: "Broome, Karen" <[email-masked]>
Date: Tue, 27 May 2008 15:30:15 -0700

For the record, John, with all the changes that have gone into ISO 639-3 since it was published, I think the skew (#3) may be more than you think. There are also issues where Ethnologue is not clear, such as the Erzgebirgische ambiguity we stumbled on by accident.

Regards,

Karen

>Of John Cowan
>Sent: Tuesday, May 27, 2008 2:16 PM
>To: Gerard Meijssen
>Cc: LTRU Working Group

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 27 May 2008 23:44:24 +0100

Re: Version Skew. As I mentioned more than a month ago on this list, the
Ethnologue online database does not keep pace with ISO 639-3 but (according
to SIL) will be updated prior to publication of the 16th Edition.

Please remember that Ethnologue is not normative in any way.  If you want
information about ISO 639-3 go to the ISO 639-3 site. From there you can
access the linked informative data from Ethnologue whilst at the same time
being assured that the code you are viewing is up to date.

Best

Debbie

> Behalf Of Broome, Karen
> Sent: 27 May 2008 23:30
> To: John Cowan; Gerard Meijssen
> Cc: LTRU Working Group
> For the record, John, with all the changes that have gone
> into ISO 639-3 since it was published, I think the skew (#3)
> may be more than you think. There are also issues where
> Ethnologue is not clear, such as the Erzgebirgische ambiguity
> we stumbled on by accident.
>
> Regards,
>
> Karen
>
> >John Cowan
> >John Cowan  [email-masked]  http://ccil.org/~cowan In
> computer science,
> >we stand on each other's feet.
> >        --Brian K. Reid

---

From: "Kent Karlsson" <[email-masked]>
Date: Wed, 28 May 2008 01:42:48 +0200

Sami already has a code that covers any Sami language, 'smi'.

It is a collection code, and already in LSR. Ignore the
"(Other)" part of the Description, it's going away.

Collection codes, are (IIUC) in general not candidates for 
being reclassified as macrolanguages, given how the concept
of macrolanguage got started.

Indeed, as I mentioned, I think it would be better to
just regard "macrolanguage" codes as any other collection
code. Available, but not recommended for use in language
tagging of individual documents.

	/kent k 

> Sent: Tuesday, May 27, 2008 8:25 PM
> To: Peter Constable
> Cc: LTRU Working Group

---

From: Leif Halvard Silli <[email-masked]>
Date: Wed, 28 May 2008 02:36:40 +0200

Broome, Karen 2008-05-28 00.09:

> Mark writes: [...]

> • under RFC 4646: "zh, fr" • under this proposal: "zh-cmn, zh,
> fr, zh-cjy;q=0, zh-cpx;q=0, zh-czh;q=0, zh-czo;q=0, zh-gan;q=0,
> zh-hak;q=0, zh-hsn;q=0, zh-mnp;q=0, zh-nan;q=0, zh-wuu;q=0,
> zh-yue;q=0".
> 
> Karen responds: [...]

> But you didn't ask for Mandarin, you asked for Chinese. [...]
> This is where I think we are prioritizing generalized fallback
> over precision in identification -- even if what we are
> precisely identifying is intentionally vague. I lose the
> ability to identify "some kind of Chinese, I don't know what
> kind" if zh is thought to be synonymous with Mandarin. (A real
> need given the history of these tags.) I also will not know
> what the user intended if they use the tag "zh". 

So, keeping 'zh' general, we will know what it meant until now.

> When it becomes possible to tag more specifically, we won't be
> able to know whether the decision to use the more general tag
> ("zh") was a conscious choice or not.

True. We won't know.

> zh is a bad tag; its semantics have always been muddy. We are
> going to rather great lengths to avoid deprecating it.

Then I want to ask: Do you think that we should/could deprecate 
'zh' but still have extlang? And, do you think that all 
macrolanguage tags for which there exist extlang tags -  in 
principle should be deprecated? (Or would you even prefer to 
deprecate the other macrolangauge tags?)

[ Browsers would then need to use the 'zh' in order to match old 
resources, only. (We could perhaps expect e.g server software to 
perform the extra hoop that Apache allready does for browsers 
which asks for 'en-GB' when they should be asking for 'en'. In 
fact Apache would do the same trick if the browser asks for 
'zh-cmn'.) ]

Thus is it so that the fallback that you want is only fallback 
from current tagging to deprecated tagging?
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Wed, 28 May 2008 03:23:53 +0200

Debbie Garside 2008-05-27 22.36:

> ISO 639 or making language classification decisions. [...]

This was mostly a reply to Peter. On the whole I agree with you.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 27 May 2008 21:13:27 -0700

Hi -

> Sent: Tuesday, May 27, 2008 2:23 PM

Limiting "de"  to mean "standard German" seems a non-useful narrowing.
 Consider a clearer example: the many flavors of "Gastarbeiterdeutsch"
(Guest-worker German).  These are clearly not "standard German", but I
can't believe that their initial subtag should be anything other than "de".

Randy

---

From: "Broome, Karen" <[email-masked]>
Date: Tue, 27 May 2008 22:57:08 -0700

Not "gem"?

Karen

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Randy Presuhn [[email-masked]]
Sent: Tuesday, May 27, 2008 9:13 PM
To: LTRU Working Group
Subject: Re: [Ltru] Does 'de' really mean "only standard German"?

Hi -

Randy

---

From: Peter Constable <[email-masked]>
Date: Tue, 27 May 2008 23:31:47 -0700

> being assured that the code you are viewing is up to date.

Debbie's descriptions look right to me -- thanks for covering these points.

I do hope that the situation described is good enough for people. I don't know how realistically things could be improved a whole lot. There are still plenty of vague spots and issues, but overall the semantics behind the coding are far more concrete and well defined than they were before.

Peter

Peter

---

From: John Cowan <[email-masked]>
Date: Wed, 28 May 2008 10:52:54 -0400

> can't believe that their initial subtag should be anything other than "de".

Again, I can't say, not knowing enough about the domain.  Are they
reasonably mutually intelligible with Standard German, or are we dealing
with a pidgin or creole here, or perhaps several?

Most of the English varieties I hear about me daily in NYC are clearly
"en", even if with a thousand different non-native phonologies.  Some of
them are English-based creoles, or varieties intermediate between creoles
and "en" itself.

But then again, English doesn't work like German or Chinese, where
there is at every location a contrast between "local language"
and "locally influenced variety of the standard language." 
(The creole-English-speaking areas are an exception, as is Scotland;
non-regionally, AAVE works like this too.)

There is no One True Way anywhere.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
Any sufficiently-complicated C or Fortran program contains an ad-hoc,
informally-specified bug-ridden slow implementation of half of Common Lisp.
        --Greenspun's Tenth Rule of Programming (rules 1-9 are unknown)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Wed, 28 May 2008 07:53:18 -0700

> Randy Presuhn

> with the Pfaelzisch one might run into among the elderly in the

I don't have an answer but rather another question -- a thought experiment:

The colour space is homogeneously continuous and for that reason we can describe individual colour varieties with fine granularity as an array of integers over some range. (I suppose in principle we could use real numbers instead of integers.) But suppose it were not always continuous and certainly not homogeneous: how would we create a system to describe colour varieties? How would we define basic, commonly-used concepts such as RED? In terms of a range or in terms of a particular, focal target colour?

It happens that there was some linguistic research done related to this in the 1960s by Brent Berlin and Paul Kay; this is described in their book _Basic Color Terms_. First, they found certain regularities regarding the basic colour terms found in different languages, but then they also looked at how those basic terms could be defined. It turns out that if you define them in terms of ranges over the spectrum, there are no regularities, either across languages or across speakers within any given language. On the other hand, if you ask individuals to select a best example for a given basic colour term out of a large set of candidate colours (they used a standardized chart of 320 small colour chips), then there is a high degree of regularity across individuals both within a single language and also across languages.

Now, there has been subsequent research suggesting that there may be human-physiological factors contributing to that regularity. I don't know of any such factors contributing to how we define concepts _language X_ or _language Y_, and I doubt any such physiological factors exist. Even so, I find the situation with colour terms to be a useful analogue for considering how to define language concepts: it suggests to me that we would probably have an easier time attaching labels like "de" to certain focal points in the plane rather than trying to define them in terms of some specific set of limits.

Now, _defining_ those basic labels in terms of particular focal points doesn't mean that we never need to have labels (tags) for varieties other than the focal points -- of course, we do. It just means that we face a certain challenge in devising tags for those other varieties. In particular, it's likely that we might encounter some issues in arriving at a single consensus on a tag for particular non-focal varieties: because there are no regularities in how people define the boundaries, one will consider a particular variety to be a variant of X while another will prefer to consider it a variant of Y.

So, let me come back to Randy's question. What's the best way to define something like "de"? If you're a contributor to Ethnologue, then a very reasonable answer is probably Standard High German -- the focal value (best example). How should something like Stadtsprache be tagged? Well, quite possibly with a tag of the form de-X (for some X), though it should be recognized in advance that there may be some debate as to whether it shouldn't rather use a tag of the form Y-Z for some other ISO 639 code element Y.

Peter

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 28 May 2008 17:43:56 +0200

On Mon, May 26, 2008 at 09:12:51PM +0200,
 Stephane Bortzmeyer <[email-masked]> wrote 
 a message of 19 lines which said:

> #7581]. 

May I should not have reported it, it is now completely scrambled in
the archive :-(

[I have no news from the secretariat.]
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Wed, 28 May 2008 09:13:58 -0700

> Broome, Karen

> Of course, if you had actually asked for Mandarin using a precise code
> for Mandarin (either "cmn" or "zh-cmn"), that list doesn't look so ugly,
> does it?
>
> But you didn't ask for Mandarin, you asked for Chinese...

I completely agree. I also note some validity in Mark's concern: there are a lot of users that have been and are still using zh rather than cmn/zh-cmn, and probably a large proportion of them are expecting Mandarin content. But as your following comments note, there's no silver bullet for dealing with that.

> This is where I think we are prioritizing generalized fallback over
> precision in identification -- even if what we are precisely
> identifying is intentionally vague. I lose the ability to identify
> "some kind of Chinese, I don't know what kind" if zh is thought to be
> synonymous with Mandarin. (A real need given the history of these tags.)

I think there's general consensus here that zh is *not* thought to be synonymous with Mandarin; there's just a high-ish degree of statistical correlation in existing usage.

> I also will not know what the user intended if they use the tag "zh".

At best, we can only make a statistical inference: based on usage to date, Mandarin is most likely meant.

> When it becomes possible to tag more specifically, we won't be able to
> know whether the decision to use the more general tag ("zh") was a
> conscious choice or not.

In some closed systems, it may be possible to know, but in an open system involving interchange with other parties in other spheres of control, no.

> zh is a bad tag; its semantics have always been muddy. We are going to
> rather great lengths to avoid deprecating it.

Deprecation for something that has been so widely used is a big step, and it may take some time for people to buy into that. But you are right in saying it is problematic: if there are other Chinese choices besides Mandarin, zh just leaves people forced to make assumptions or guess that may not be valid.

I'd point out that all these things are true whether we use extlang or not, and that extlang is helpful in relation to that problem only to the extent that it can help processes take actions that will match user expectations better.

I think initially I thought it might on the basis that anyone asking for zh-xxx might, by right-truncation fallback, still get matches with existing zh content. I was assuming that would generally be a useful response from the user perspective, and while it probably is for Chinese it isn't in the general case, across all macrolanguages -- if for no other reason, because there isn't a bunch of existing content tagged aym or bal or bik etc. that people want to continue getting matches for.

Also, I was thinking of filtering (in RFC4747 terms), and for filtering I got it backwards: right-truncation fallback allows a request for zh to match content tagged zh-xxx, and in that case it *isn't* helpful in the general case: if someone is making a request using a macrolanguage ID, they're still looking for some particular variety, and in most cases returning *any* encompassed variety probably isn't helpful.

Rather than filtering, it's *lookup* for which a request for zh-cmn would match existing zh content. But I think it's fair to say that lookup is most typically used for UI resource matching or similar scenarios in which the options for available content are likely to be fairly limited and controlled, with users being presented with the set of specific options available. In that kind of controlled environment, fallback from (zh-)cmn to zh is less likely to be needed.

Peter

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 28 May 2008 09:41:46 -0700

Peter wrote:

> Rather than filtering, it's *lookup* for which a request for zh-cmn
> would match existing zh content. But I think it's fair to say that
> lookup is most typically used for UI resource matching or similar
> scenarios in which the options for available content are likely to be
> fairly limited and controlled, with users being presented with the set
> of specific options available. In that kind of controlled environment,
> fallback from (zh-)cmn to zh is less likely to be needed.
>

In fact, in my experiments at least, lookup requires some modification in order to produce the best results when working with extlang. For my applications, at least, it seems that treating the primary-extlang pair as "atomic" for at least the first pass makes the most sense. That is, the fallback pattern works better when it goes like this:

zh-cmn-Hans-HK
zh-cmn-Hans
zh-cmn
zh-Hans-HK
zh-Hans
zh
(default)

... then when it goes according to the current rules:

zh-cmn-Hans-HK
zh-cmn-Hans
zh-cmn
zh
(default)

This is especially true when one has mixed content regimes (mixed scripts, regions, dialects, languages, etc.) in which 'zh' can represent only one variety (and frequently the wrong value).

Extlangistas will note that you have to make the same modification if you omit extlang to obtain the *exact* same results:

cmn-Hans-HK
cmn-Hans
cmn
zh-Hans-HK (presumably from Macrolanguage field??)
zh-Hans
zh
(default)

Addison

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 28 May 2008 13:11:08 -0700

Randy wrote:

> Later in this message I will be asking some questions.... Please respond no later than Saturday, May 24th, wherever you are.

I realize I am late to the party. However, having been on vacation, I would like to add my response to this call (not having found a consensus determination in the thread yet and understanding that my response may be ignored by the chairs).

> Q1: The current draft does away with the "extlang" production.

Addison

Addison Phillips

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 28 May 2008 13:58:09 -0700

> can't believe that their initial subtag should be anything other than

This argument can easily be extended to any language one cares to name, since languages provide a wide range of expressiveness and people have varying degrees of idiosyncrasy or command of the language in question. In one of my presentations about language tags I quote this bit of introduction by Mark Twain which seems instructive here:

--
IN this book a number of dialects are used, to wit: the Missouri negro dialect; the extremest form of the backwoods Southwestern dialect; the ordinary "Pike County" dialect; and four modified varieties of this last. The shadings have not been done in a haphazard fashion, or by guesswork; but painstakingly, and with the trustworthy guidance and support of personal familiarity with these several forms of speech. I make this explanation for the reason that without it many readers would suppose that all these characters were trying to talk alike and not succeeding.
--

That language tags do not capture the full richness of human expression should not be surprising. Attempts to capture every nuance of a given language artifact in a single tag are counter-productive.

Addison

Addison Phillips

---

From: Peter Constable <[email-masked]>
Date: Wed, 28 May 2008 16:13:12 -0700

> From: Texin, Tex [mailto:[email-masked]]

> Actually, I have the research book of Berlin and Kay. I thought I might
> have been the only one in the world to have read it.

I have to admit, while I've known about their work for a long time, I've never seen or read the book; I was relying on other references (for this discussion, a book by George Lakoff discussing cognitive models by which we define concepts).

> But you make quite a leap in the last para when you equate the focal
> point to the tag.
> Making the ideal equivalent to the tag, then makes everything non-ideal
> excluded, when in fact until now it is a superset. (Which is what this
> discussion revolves around.)

What I meant was that the best way to document the denotation of a code element like "de" is in terms of the best example rather than in terms of specifying a range, though in terms of application of that code element we would allow it to be used over some range of varieties (even though the limits to the range haven't been defined).

If you think about it, that's just the same things we do in Unicode for character coding.

> Rather, I would think you would make the ideal a subtag. In fact
> standardizing the subtag for the most representative dialect would then
> help define the more general case while not limiting it. So if N1 is
> used to represent number 1 or best case, then you can identify best
> cases for each language and still have primary tags be more inclusive;
> de-N1 Also people can then specify if they want the most general or the
> most representative languages for langs like zh.

Every tag must allow for some range of variation. For instance, even if we're considering some ideal, conventional variety, two native speakers of that language do not express things in exactly the same way. There will always be some degree of variation for any language category we define. So, it's not a matter of picking one specific value versus allow some range of variation; rather, it's a matter of deciding what order of range of variation is scoped by a given category.

In evaluating ISO 639-2 to decide how the denotation of existing code elements should be characterized with a greater degree of precision, there were a few hundred choices that had to be made regarding scope. When faced with something like "ar" or "de", the approach that seemed to make most sense, considering common usage wrt tagged content and resources, was to scope at the individual-language level, which in those cases meant the standard varieties. We went as far as to come up with some operational definitions to apply that approach consistently across the board, but then we encountered some division in approach to usage in the case of some languages. Chinese was a particularly obvious case because of the pre-existing tags registered under RFC 1766. So, we ended up adopting a different principle for those cases, leading to the macrolanguage concept. Should we have taken the different approach for even more cases, such as "de"? Well, I had no information pointing to "de" being used for anything other than the standard variety: A/V content was not yet a major consideration in application of language tags, and wrt the most likely non-standard variety that could potentially have been lumped into "de", Swiss German (i.e. gsw), I kept hearing that written usage was not significant. If "de" had been analyzed the same way as "zh", then we'd be facing all the same problems for "de" as we are facing now for "zh".

So, while it isn't all perfect, I think what was done was reasonable and workable. And I think there can appropriate cases in which tags of the form de-variantx might be used to describe some specific variaties that could reasonably be considered as within the scope for the standard variety. (But don't ask me which are or are not appropriate cases for "de".)

Peter

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 28 May 2008 17:22:15 -0700

Hi -

Martin and I double-checked the tally, and it turns out that
*two* responses were sent directly to me, rather  than to the
[email-masked] mailing list.  One was sent after the deadline.
Here's the tally including those three:

Q1: A-4 B-9 C-4
Q2: A-8 B-5 C-4

Excluding those three responses, it looks like:
Q1: A-2 B-9 C-3
Q2: A-7  B-5 C-2

Either way, the support for an extlang-less approach appears to
be in he distinct minority, and there appears to be a marked
preference (which we'll call a *rough* consensus) for
"reinstating" extlang.

Or have a bunch of people changed their minds?

Randy

---

From: Peter Constable <[email-masked]>
Date: Wed, 28 May 2008 18:11:29 -0700

> Or have a bunch of people changed their minds?

Well, if it's worth anything, I think I mentioned that my B/B vote for Q1/Q2 was leaning on the A/C side provided we could find a way to satisfy major concerns and keep all industry sectors represented following one spec. I haven't changed in that regard.

Peter

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 28 May 2008 22:33:55 -0700

Hi -

> From: "Peter Constable" <[email-masked]>
> Sent: Wednesday, May 28, 2008 4:13 PM
...
> So, while it isn't all perfect, I think what was done was reasonable
> and workable. And I think there can appropriate cases in which tags
> of the form de-variantx might be used to describe some specific variaties
> that could reasonably be considered as within the scope for the standard
> variety. (But don't ask me which are or are not appropriate cases for "de".)

I could agree with saying that "de" is "exemplified by standard German".
This reflects reality and does not cause problems.

I cannot agree with saying that "de" somehow *is* "standard German".
This leaves all non-standard varieties which have not been given language
codes of their own in some sort of limbo, potentially requiring new top-level
codes for things which really should not need them.  (I have great difficulty
understanding how a non-standard variety would be "within the scope for
the standard variety".)

Randy

---

From: John Cowan <[email-masked]>
Date: Thu, 29 May 2008 02:17:39 -0400

> I cannot agree with saying that "de" somehow *is* "standard German".

Well, no actually extant variety *is* Standard German, strictly speaking:
Standard German (as defined, say, in Duden) doesn't *exactly* match
anybody's idiolect.  However, what's described there is close enough
to many other idiolects that we can collectively call them all "de"
without crossing any mutual-intelligibility barriers.  (To me, "de"
is the set of idiolects mutually intelligible with my mother's; her
German was about as uninfluenced by locality as you could imagine,
bar a very occasional /x/ for "-g" other than "-ig", which she would
conscientiously warn her first-year German students not to emulate.)

> This leaves all non-standard varieties which have not been given
> language codes of their own in some sort of limbo, potentially requiring
> new top-level codes for things which really should not need them.
> (I have great difficulty understanding how a non-standard variety
> would be "within the scope for the standard variety".)

By being mutually intelligible with it, and therefore in some sense
part of it.  That way, new top-level codes can be, and are, properly
reserved for things that are *not* mutually intelligible with "de".
(I am using "mutually intelligible" as shorthand for the actual criteria
used by 639-3/RA; see http://www.sil.org/iso639-3/scope.asp#I .)

-- 
John Cowan  [email-masked]   http://ccil.org/~cowan
Promises become binding when there is a meeting of the minds and consideration
is exchanged. So it was at King's Bench in common law England; so it was
under the common law in the American colonies; so it was through more than
two centuries of jurisprudence in this country; and so it is today.
       --Specht v. Netscape
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 29 May 2008 07:51:20 -0700

> One was sent after the deadline.

If this was my vote (sent yesterday prior to your message), your "revised totals" appear wrong, as I voted "A,B". If this wasn't my vote, then including my vote in the late count doesn't make the numbers wholly different.

> preference (which we'll call a *rough* consensus) for

This is not a situation in which either camp has a totally decisive argument up its sleeve. While I think this is the wrong way to go (obviously, as I supported removing extlang), some kind of decision is desirable.

As co-editor, my immediate concern is how to implement this. Assuming that we don't do a new round of voting and persuasion, I can hardly restore the text we had prior to removing extlang because some serious editing has occurred since then. We will need to decide what criteria are used to register and maintain extlangs as a result (a problem we were struggling with prior to the decision to remove extlang). What will be necessary, I think, is a stem-to-stern restore-and-edit pass with the particular notes of interest sent to the list. The resulting draft will then need to be reviewed by the list.

If there is some doubt about whether the results of this consensus call will hold, I would point out that this might be a useful exercise in any event, as we can compare this putative "draft-15" (incorporating feedback from the list, of course) to the current draft-14 more directly: we can compare each other's cherry picking machines for elegance :-).

Is this the decision, Randy and Martin? Or should we waffle along a bit longer?

Addison

Addison Phillips

> Sent: Wednesday, May 28, 2008 5:22 PM
> To: LTRU Working Group

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 29 May 2008 11:23:54 -0700

Hi -

> Sent: Wednesday, May 28, 2008 11:17 PM

> Well, no actually extant variety *is* Standard German, strictly speaking:

Violent agreement.

> without crossing any mutual-intelligibility barriers.

This seems a slight over-statement.  Consider, for example, regionalisms
in vocabulary.  I don't think it's terribly useful to regard them as loan words
from some other language, even if they happen to be the same in a local
dialect which enjoys a language code of its own.

...
> part of it. 

Sorry, I just can't make sense of that "in some sense part of it".  "Part of"
to me implies a subset / superset relationship, and what I'm sure we're
talking about is set intersection.  (The extent of that set intersection is a
factor in making mutual intelligibility is possible.)

I'm in complete agreement with the idea of treating the mythical Hochdeutsch
in a manner analogous to a "focal color".  What I have a problem with are
descriptions that lead to readings like "stuff tagged 'de' is Standard German"
rather than "stuff tagged 'de' should be some kind of German which would be
mutually intelligible with Standard German."

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 29 May 2008 11:47:59 -0700

Hi -

> From: "Phillips, Addison" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>; "LTRU Working Group" <[email-masked]>
> Sent: Thursday, May 29, 2008 7:51 AM
> Subject: RE: [Ltru] Consensus call: extlang
>

> If this was my vote (sent yesterday prior to your message),
> your "revised totals" appear wrong, as I voted "A,B".

The summary (with the late response) was sent out before
your much later response.  Including the three late responses
I've seen, it now comes to 
Q1 a-6 b-9 c-4 
Q2 a-8 b-6 c-5

Not the clear-cut result any of us would have liked.
(And for me, having a clear decision is much more important
than the decision itself!)

...
> This is not a situation in which either camp has a totally
> decisive argument up its sleeve.

This has been apparent for many, many months.

> While I think this is the wrong way to go (obviously, as I
> supported removing extlang), some kind of decision is desirable.

Agreed.  However, the recent discussion indicates that the group
doesn't feel much urgency in moving on.  If the decision were
solely up to me, I'd recommend shutting down the working group,
letting the people that care about it fight it out, and then let
them request a BOF at some future IETF meeting when they think
they're ready to resume work as a working group.  (This is
what happened when the SNMPv2 working group missed its charter
deadline by a far, far smaller margin.)

> As co-editor, my immediate concern is how to implement this.
> Assuming that we don't do a new round of voting and persuasion,
> I can hardly restore the text we had prior to removing extlang
> because some serious editing has occurred since then.

Agreed.

> We will need to decide what criteria are used to register and
> maintain extlangs as a result (a problem we were struggling with
> prior to the decision to remove extlang). What will be necessary,
> I think, is a stem-to-stern restore-and-edit pass with the particular
> notes of interest sent to the list. The resulting draft will then
> need to be reviewed by the list.

Sounds reasonable.
 
> If there is some doubt about whether the results of this consensus call
> will hold, I would point out that this might be a useful exercise in
> any event, as we can compare this putative "draft-15" (incorporating
> feedback from the list, of course) to the current draft-14 more directly:
> we can compare each other's cherry picking machines for elegance :-).

:-)  Since we're in "the lesser of the evils" mode, I agree it would be helpful
to work through the devilish details.
 
> Is this the decision, Randy and Martin? Or should we waffle along a bit longer?

Creating such a draft sounds good to me.  Martin, any insights?

Randy

---

From: John Cowan <[email-masked]>
Date: Thu, 29 May 2008 17:00:29 -0400

> mutually intelligible with Standard German."

Okay, then we disagree on what "is" is.  When I say "'de' is Standard
German", mentally translate that to "'de' denotes the cluster of varieties
mutually intelligible (etc. etc.) with Standard German."  Similarly, "'en'
is English" means that 'en' denotes the varities mutually intelligible
with, say, Peter Constable's (Canadian) English.  Hey, it's better than
Mandarin, which routinely tolerates sentences like "I am [the one who
ordered] ice cream" and "She is also [a case of someone married to]
an American husband."  (Li & Thompson)

("_A Course In Phonetics_ explains everything about phonetics except
how to pronounce 'Ladefoged'.")

-- 
Time alone is real                      John Cowan <[email-masked]>
  the rest imaginary
like a quaternion       --phma          http://www.ccil.org/~cowan
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 29 May 2008 23:08:52 +0100

Randy wrote:

> Agreed.  However, the recent discussion indicates that the
> group doesn't feel much urgency in moving on.  If the
> decision were solely up to me, I'd recommend shutting down
> the working group, letting the people that care about it
> fight it out, and then let them request a BOF at some future
> IETF meeting when they think they're ready to resume work as
> a working group.  (This is what happened when the SNMPv2
> working group missed its charter deadline by a far, far
> smaller margin.)

I think this would be a huge mistake.  It is very very important to make ISO
639-3 available within the LSR as soon as possible.  However, as I have
stated before, after this length of time it is important that we 'please' as
many end users as possible whilst bearing in mind that we are discussing
some of the worlds most spoken languages and their historical codes.

On the 27/5/2008 the following message was posted:

Debbie wrote in response:
>I think this is probably a good way forward.  It would be best to
>start a new thread for each application.  I would also add
>backward compatibility and historical usage to the pot.

I would ask the co-chairs to guide the WG in this direction and to advise
the WG that any other threads will be considered as temporarily off topic
until the extlang/no extlang issue is resolved.

Somebody may have already posted a similar request but I have not had time
to read the avalanche of mail this evening. We really need the WG to focus
now.

Best regards

Debbie

> Sent: 29 May 2008 19:48
> > If this was my vote (sent yesterday prior to your message), your
> > "revised totals" appear wrong, as I voted "A,B".
>
> The summary (with the late response) was sent out before your
> much later response.  Including the three late responses I've
> seen, it now comes to
> Q1 a-6 b-9 c-4
> Q2 a-8 b-6 c-5
>
> > This is not a situation in which either camp has a totally decisive
> > argument up its sleeve.
>
> supported
> > removing extlang), some kind of decision is desirable.
>
> Agreed.  However, the recent discussion indicates that the
> group doesn't feel much urgency in moving on.  If the
> decision were solely up to me, I'd recommend shutting down
> the working group, letting the people that care about it
> fight it out, and then let them request a BOF at some future
> IETF meeting when they think they're ready to resume work as
> a working group.  (This is what happened when the SNMPv2
> working group missed its charter deadline by a far, far
> smaller margin.)
>
> > Assuming that we don't do a new round of voting and
> persuasion, I can
> > hardly restore the text we had prior to removing extlang
> because some
> > We will need to decide what criteria are used to register
> and maintain
> > extlangs as a result (a problem we were struggling with
> prior to the
> > decision to remove extlang). What will be necessary, I think, is a
> > stem-to-stern restore-and-edit pass with the particular notes of
> > interest sent to the list. The resulting draft will then need to be
> > reviewed by the list.
>
> Sounds reasonable.
>
> > If there is some doubt about whether the results of this consensus
> > call will hold, I would point out that this might be a
> useful exercise
> > in any event, as we can compare this putative "draft-15"
> > (incorporating feedback from the list, of course) to the
> current draft-14 more directly:
> :-)  Since we're in "the lesser of the evils" mode, I agree
> it would be helpful to work through the devilish details.
>

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 29 May 2008 15:59:41 -0700

What would be useful is to hear from the extlangistas what their concerns
are specifically; many have not given reasons for favoring encompassed
languages into extlang instead of into the primary language subtag. It would
be useful for them to give the scenarios where they think extlang is an
improvement. It would be useful to find out why they think the scenarios
such as in http://docs.google.com/Doc?docid=dfqr8rd5_676kxxxjhd&hl=en are
not a problem.
Clearly people think that using the extlang model solves more problems than
it causes, so it would be useful to example specific cases and see if that
is, in fact, true.

Mark

---

From: "Broome, Karen" <[email-masked]>
Date: Thu, 29 May 2008 18:04:45 -0700

Mark,

One thing I think you aren't acknowledging is that "treat as synonyms" means something very different to the vast numbers of content creators who use this standard than it does the handful of search engines that use the fuzzy logic associated with companion standards. As you note in your document, "It is clear that companies like Google or Yahoo can work around the problems with extlang." How many other users need and can afford to implement the extended fallback and filtering logic? Enough that this logic should be the primary driver behind the chosen solution?

Before I spend too much time picking apart your lengthy screed involving a scenario where the BBC presents its web site in Sudanese Creole Arabic with rotating languages code logic for each day of the week ... (ahem) ... here's my real-world Chinese language list:

Chinese (Variant Unknown)
Chinese (Cantonese, Spoken)
Chinese (Cantonese, Written)
Chinese (Mandarin, Spoken)
Chinese (Mandarin, Spoken Taiwanese)
Chinese (Mandarin, Simplified)
Chinese (Mandarin, Traditional)
Chinese (Taiwanese, Spoken)
Chinese (Taiwanese, Written)

(Apologies, this is hard to represent in ASCII. I have a mini-spreadsheet if someone wants it.)

1             2       3   4
zh            zh        zh        zh
zh-yue  yue           yue         yue
zh-yue  yue           yue         yue
zh-cmn  cmn           zh          cmn
zh-cmn-TW       cmn-TW  zh-TW     cmn-TW
zh-cmn-Hans     cmn-Hans        zh-Hans zh-Hans
zh-cmn-Hant     cmn-Hant        zh-Hant zh-Hant
zh-min-nan      nan           nan         nan
zh-min-nan      nan           nan         nan

* Option #1 (RFC 4646) contains the codes as I have them today.
* Option #2 (RFC 4646bis) contains the codes if I choose to go against the grain and use "cmn".
* Option #3 (RFC 4646bis) treats "zh" and "cmn" as synonyms; avoids using "cmn" for compatibility.
* Option #4 (RFC 4646bis) contains the codes "cmn" for spoken context (where distinction is essential) and "zh" for written context.

Comments:

* Option #1 is unambiguous and shows that there is a relationship between these languages. It also preserves the legacy "zh" tag so developers that aren't hip to later versions of BCP 47 or 639-3 will have some idea what these tags mean. The tags are maybe longer than they need to be, but if I need a fixed-length tag, I can wait for 639-6. The languages may not be mutually intelligible in some contexts, but they are related.

* Option #2 is unambiguous, but Microsoft, Google, and Amazon won't be using the same tags for Chinese that I do. Even if I don't follow their lead, others likely will. This worries me. Also, the rules for #2 must include fuzzy guidelines such as, "use the 'zh' tag except when you think it's a bad idea" and "use the shortest tag except when you don't want to." This presents complications in trying to explain some sort of consistent method to the LTRU madness to others. Given this, I start to wish ISO 639-6 a safe and speedy passage.

* Option #3 is what I believe you might suggest, but for me, that's the worst list of all. There are five ambiguous "zh" categories on that list. It follows the "always use the shortest tag" rule and respects history, but it's useless to me from an identification perspective.

* Option #4 has three ambiguous tags and means I have to explain to people who aren't in this industry about why I use different tags for the same language. This strategy is less ambiguous that #3, but I'm not sure I can explain it to other content creators for the same reasons as #2 and presents the spoken/written complication others may not want. In the long run, this seems messy and unclear enough that it will result in bad tagging.

* Options #2,3,4: In general, it worries me that RFC 4646bis offers so many "preferred" options for the same thing. I really can't see how this simplifies things for anyone.

I don't have a need for fuzzy fallback scenarios. I need precise tags and mostly simple lookup. I think if you take the fallback scenarios and absurdities out of the document you reference, I don't think there's much left.

Regards,

Karen Broome

>Of Mark Davis
>Sent: Thursday, May 29, 2008 4:00 PM
>To: [email-masked]
>Cc: LTRU Working Group
>What would be useful is to hear from the extlangistas what their
>concerns are specifically; many have not given reasons for favoring
>encompassed languages into extlang instead of into the primary
>language subtag. It would be useful for them to give the scenarios
>where they think extlang is an improvement. It would be useful to
>find out why they think the scenarios such as in
>http://docs.google.com/Doc?docid=dfqr8rd5_676kxxxjhd&hl=en are not a
>problem.
>
>Clearly people think that using the extlang model solves more
>problems than it causes, so it would be useful to example specific
>cases and see if that is, in fact, true.
>
>
>Mark

---

From: John Cowan <[email-masked]>
Date: Fri, 30 May 2008 00:57:09 -0400

> zh-min-nan      nan           nan         nan

Note that if we do go with extlangs, or extlangs-for-zh, we would still
deprecate zh-min-nan in favor of zh-nan, though of course zh-min-nan
will remain valid.

-- 
Principles.  You can't say A is         John Cowan <[email-masked]>
made of B or vice versa.  All mass      http://www.ccil.org/~cowan
is interaction.  --Richard Feynman
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Thu, 29 May 2008 22:50:13 -0700

> Before I spend too much time picking apart your lengthy screed
> involving a scenario where the BBC presents its web site in Sudanese
> Creole Arabic with rotating languages code logic for each day of the
> week ... (ahem) ... here's my real-world Chinese language list:
>

> Comments:
>
> * Option #1 is unambiguous and shows that there is a relationship
> between these languages.

The only difference with #2 is that the relationship is not reflected directly in the tags.

Since you mention having the relationship shown, I gather you think there's a benefit, but you don't explain what that benefit is.

> It also preserves the legacy "zh" tag so
> developers that aren't hip to later versions of BCP 47 or 639-3 will
> have some idea what these tags mean.

I'm not sure how significant a benefit that is. The developers may encounter many subtags that they're not familiar with, and they'll need to do some learning to make sense of them. And if feedback we got the first time we tried a last call for 3066bis were to be believed (before this WG was formed), there are developers out there who will be confused by any new patterns for forming tags. I agree that the presence of "zh" will give them a hint, but I expect in any of these situations people that need to will find out what these things they don't recognize are.

> The tags are maybe longer than
> they need to be, but if I need a fixed-length tag, I can wait for 639-6.

The length is not a problem, nor is the variation in length a problem.

I've got some people in my company expressing concern over people (meaning people in my company, not users) being confused by the fact that some languages will be represented as "aa", others "aaa", and (worst of all) still others "aa-bbb" or "aaa-bbb". While I expect people in a technology company who need to work directly with these things ought to be able to learn about and understand extlangs, I think if any of these things we've been talking about is likely to be a source of confusion, it's probably this (extlang). I'm still not sure that's a serious concern, but my point is that I think something like unfamiliarity with "cmn" is less of a concern.

> The languages may not be mutually intelligible in some contexts, but
> they are related.

English and Frisian are related. French and Spanish are related. Vietnamese and Khmer are related. I'm not sure how those facts are especially significant for tagging.

> * Option #2 is unambiguous, but Microsoft, Google, and Amazon won't be
> using the same tags for Chinese that I do. Even if I don't follow their
> lead, others likely will. This worries me.

The potential for different parties to use different tags exists in any of the schemes you list.

While we may not always use the same tags that you do, it isn't necessarily the case that we would never use the same tags that you do. I can't speak for other vendors, but I wouldn't at all rule out MS using the same tags you use in scenarios where your content needs to be supported.

> Also, the rules for #2 must
> include fuzzy guidelines such as, "use the 'zh' tag except when you
> think it's a bad idea" and "use the shortest tag except when you don't
> want to."

That's the same for #1, though.

> This presents complications in trying to explain some sort of
> consistent method to the LTRU madness to others. Given this, I start to
> wish ISO 639-6 a safe and speedy passage.

Have you at all considered what complications there would be in going to that? If all you're thinking about is "identification" (how to declare an identity), then I can imaging you might not have. Being on the side of thinking about how to consume tags and do something with them, I don't see that as a rosy and simple alternative.

> * Option #3 is what I believe you might suggest, but for me, that's the
> worst list of all. There are five ambiguous "zh" categories on that
> list. It follows the "always use the shortest tag" rule and respects
> history, but it's useless to me from an identification perspective.

I won't argue against your concerns here: they are valid. You do need to acknowledge, though, that all of the tags you find problematic in this case (zh, zh-TW, zh-Han?) would all be valid in *any* case, no matter what we decide, and these *will* be encountered.

> * Option #4 has three ambiguous tags and means I have to explain to
> people who aren't in this industry about why I use different tags for
> the same language. This strategy is less ambiguous that #3, but I'm not
> sure I can explain it to other content creators for the same reasons as
> #2 and presents the spoken/written complication others may not want. In
> the long run, this seems messy and unclear enough that it will result
> in bad tagging.

Again, with or without extlang, some kind of guidance needs to be given, and some reasonable means must be available for those who must somehow support legacy processes and data.

> * Options #2,3,4: In general, it worries me that RFC 4646bis offers so
> many "preferred" options for the same thing. I really can't see how
> this simplifies things for anyone.

There is no avoiding certain options for Chinese, whether with or without extlang, whether with RFC 4646bis or some entirely new scheme based on ISO 639-6 or whatever (since IETF language tags as we know them aren't about to disappear).

Peter

---

From: "Andrew Cunningham" <[email-masked]>
Date: Fri, 30 May 2008 16:28:53 +1000

> Chinese (Taiwanese, Written)

wouldn't Chinese (Cantonese, Written) be divided into

Chinese (Cantonese, Simplified)  and  Chinese (Cantonese, Traditional)

I'm assuming a citizen of Guangzhou province, if they decide to write
in Cantonese, are unlikely to be doing so in Traditional characters.
Although my assumptions may be wrong.

-- 
Andrew Cunningham
Vicnet Research and Development Coordinator
State Library of Victoria
Australia

[email-masked]
[email-masked]

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 30 May 2008 09:11:06 -0700

Peter's addressed some of the questions. Back to your original question.
For backward compatibility, we'll continue to represent Mandarin as "zh",
Standard Arabic as "ar", and so on. Note that this is independent of whether
extlang is used or not. That is, if extlang exists, we'll treat incoming
"zh-cmn" as if it were "zh"; if it doesn't, we'll treat "cmn" as if it
were "zh". And under either scenario it is conformant to tag Mandarin as
'zh'.

Why?
While 639-3 now specifies clearly that "de" means (for example) just
Standard German, while "zh" means Any
Chinese, this clarity of specification was not present earlier. The code
"zh" has been used in the past for Mandarin, overwhelmingly so; not just 99%
or 99.9%, but many 9's. As you said, the tendency was to use illegal (or
private use codes) for non-Mandarin content. All of our internal software
and any external software that we talk to will expect Mandarin to be tagged
as 'zh' for the forseeable future.  Of course, we recognize that others may
end up using 'zh-cmn' / 'cmn', so we're prepared to deal with that.

Note also that really the whole premise of extlang is that 'zh' continues to
normally map to Mandarin. After all, if 'zh' really meant that you were as
likely to get Gan or Hakka as Mandarin, then having "zh-yue" in order to get
some kind of automatic fallback wouldn't make any sense.

Other comments below.

On Thu, May 29, 2008 at 6:04 PM, Broome, Karen <[email-masked]>
wrote:

> Mark,
>
> One thing I think you aren't acknowledging is that "treat as synonyms"
> means something very different to the vast numbers of content creators who
> use this standard than it does the handful of search engines that use the
> fuzzy logic associated with companion standards. As you note in your
> document, "It is clear that companies like Google or Yahoo can work around
> the problems with extlang." How many other users need and can afford to
> implement the extended fallback and filtering logic? Enough that this logic
> should be the primary driver behind the chosen solution?
>
> Before I spend too much time picking apart your lengthy screed involving a
> scenario where the BBC presents its web site in Sudanese Creole Arabic with
> rotating languages code logic for each day of the week ... (ahem) ... here's
> my real-world Chinese language list:
>

Sorry you consider it a scree. I realize that the emails have sometimes
gotten heated -- email really is a poor substitute for audio discussions in
controversial issues; I've seen many, many issues in Unicode and other
standards flare for months in email, and be resolved in a few hours of
discussion.

My real point is that if a query for 'ar' really means "give me any kind of
Arabic", then a query for 'ar' would be almost meaningless, since it could
return any of a number of mutually incomprehensible alternatives. Although
639-3 now defines it to be "any Arabic", in practice what users will expect
to get back is Standard Arabic, and they would be unpleasantly surprised to
get back other varieties. And our purpose should be to avoid our users'
getting unpleasant surprises.

> a. zh            zh            zh          zh
> b. zh-yue        yue           yue         yue
> c. zh-yue        yue           yue         yue
> d. zh-cmn        cmn           zh          cmn
> e. zh-cmn-TW     cmn-TW        zh-TW       cmn-TW
> f. zh-cmn-Hans   cmn-Hans      zh-Hans     zh-Hans
> g. zh-cmn-Hant   cmn-Hant      zh-Hant     zh-Hant
> h. zh-min-nan    nan           nan         nan
> i. zh-min-nan    nan           nan         nan

above modified slightly to add row references.

> * Option #1 (RFC 4646) contains the codes as I have them today.

Note that this is not actually RFC4646 conformant: zh-cmn-TW is not valid.

> * Option #4 (RFC 4646bis) contains the codes "cmn" for spoken context
> (where distinction is essential) and "zh" for written context.
>
> Comments:
>
> * Option #1 is unambiguous and shows that there is a relationship between
> these languages. It also preserves the legacy "zh" tag so developers that
> aren't hip to later versions of BCP 47 or 639-3 will have some idea what
> these tags mean. The tags are maybe longer than they need to be, but if I
> need a fixed-length tag, I can wait for 639-6. The languages may not be
> mutually intelligible in some contexts, but they are related.
>
> lead, others likely will. This worries me. Also, the rules for #2 must
> include fuzzy guidelines such as, "use the 'zh' tag except when you think
> it's a bad idea" and "use the shortest tag except when you don't want to."
> This presents complications in trying to explain some sort of consistent
> method to the LTRU madness to others. Given this, I start to wish ISO 639-6
> a safe and speedy passage.
>
> worst list of all. There are five ambiguous "zh" categories on that list. It
> follows the "always use the shortest tag" rule and respects history, but
> it's useless to me from an identification perspective.

Your list is already ambiguous for columns 1 and 2; you are using "yue" for
two different things (written and spoken). The only change it really makes
is that you don't have a term for "any chinese".

RFC 4646 lacks terms for many, many combinations of things: a term for "any
german" (including de, gsw, ...), "any french", "any scandinavian", or any
one of the countless other possible sets of languages that people consider
to be important for some particular purpose. That's why lists of languages
are really the appropriate vehicle.

>
> * Option #4 has three ambiguous tags and means I have to explain to people
> who aren't in this industry about why I use different tags for the same
> language. This strategy is less ambiguous that #3, but I'm not sure I can
> explain it to other content creators for the same reasons as #2 and presents
> the spoken/written complication others may not want. In the long run, this
> seems messy and unclear enough that it will result in bad tagging.
>
> * Options #2,3,4: In general, it worries me that RFC 4646bis offers so many
> "preferred" options for the same thing. I really can't see how this
> simplifies things for anyone.
>
> I don't have a need for fuzzy fallback scenarios. I need precise tags and
> mostly simple lookup. I think if you take the fallback scenarios and
> absurdities out of the document you reference, I don't think there's much
> left.

The only purpose I have heard for extlang *is* for fallback; that's why the
document goes into (painful) depth on that topic. For identification alone,
"zh" and "zh-cmn" really mean just the same thing. It is only in the context
of matching (filtering and lookup) that they differ in semantics *because of
their behavior*: where "cmn" means simply Cantonese, "zh-cmn" effectively
means "Cantonese but fallback to any Chinese".

---

From: Felix Sasaki <[email-masked]>
Date: Sat, 31 May 2008 11:47:46 +0900

Mark Davis さんは書きました:
> For backward compatibility, we'll continue to represent Mandarin as 
> "zh", Standard Arabic as "ar", and so on. Note that this is 
> independent of whether extlang is used or not. That is, if extlang 
> exists, we'll treat incoming "zh-cmn" as if it were "zh"; if it 
> doesn't, we'll treat "cmn" as if it were "zh". And under either 
> scenario it is conformant to tag Mandarin as 'zh'.
>
> Why? 
> While 639-3 now specifies clearly that "de" means (for example) just Standard German, while "zh" means Any 
> Chinese, this clarity of specification was not present earlier. The 
> code "zh" has been used in the past for Mandarin, overwhelmingly so; 
> not just 99% or 99.9%, but many 9's. As you said, the tendency was to 
> use illegal (or private use codes) for non-Mandarin content. All of 
> our internal software and any external software that we talk to will 
> expect Mandarin to be tagged as 'zh' for the forseeable future.  Of 
> course, we recognize that others may end up using 'zh-cmn' / 'cmn', so 
> we're prepared to deal with that.
>
>     Note also that really the whole premise of extlang is that 'zh'
>     continues to normally map to Mandarin. After all, if 'zh' really
>     meant that you were as likely to get Gan or Hakka as Mandarin,
>     then having "zh-yue" in order to get some kind of automatic
>     fallback wouldn't make any sense.
>
>
> Other comments below.
>
> On Thu, May 29, 2008 at 6:04 PM, Broome, Karen 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Mark,
>
>     One thing I think you aren't acknowledging is that "treat as
>     synonyms" means something very different to the vast numbers of
>     content creators who use this standard than it does the handful of
>     search engines that use the fuzzy logic associated with companion
>     standards. As you note in your document, "It is clear that
>     companies like Google or Yahoo can work around the problems with
>     extlang." How many other users need and can afford to implement
>     the extended fallback and filtering logic? Enough that this logic
>     involving a scenario where the BBC presents its web site in
>     Sudanese Creole Arabic with rotating languages code logic for each
>     day of the week ... (ahem) ... here's my real-world Chinese
>     language list:
>
> Sorry you consider it a scree. I realize that the emails have 
> sometimes gotten heated -- email really is a poor substitute for audio 
> discussions in controversial issues; I've seen many, many issues in 
> Unicode and other standards flare for months in email, and be resolved 
> in a few hours of discussion.

Which let's me think that a few hours meeting, maybe even face-to-face, 
between the two of you and potentially others, could be very helpful.

Felix

>
> My real point is that if a query for 'ar' really means "give me any 
> kind of Arabic", then a query for 'ar' would be almost meaningless, 
> since it could return any of a number of mutually incomprehensible 
> alternatives. Although 639-3 now defines it to be "any Arabic", in 
> practice what users will expect to get back is Standard Arabic, and 
> they would be unpleasantly surprised to get back other varieties. And 
> our purpose should be to avoid our users' getting unpleasant surprises.
>  
>
>
>
>     * Option #4 (RFC 4646bis) contains the codes "cmn" for spoken
>     context (where distinction is essential) and "zh" for written context.
>
>     Comments:
>
>     between these languages. It also preserves the legacy "zh" tag so
>     developers that aren't hip to later versions of BCP 47 or 639-3
>     will have some idea what these tags mean. The tags are maybe
>     longer than they need to be, but if I need a fixed-length tag, I
>     can wait for 639-6. The languages may not be mutually intelligible
>     in some contexts, but they are related.
>
>     * Option #2 is unambiguous, but Microsoft, Google, and Amazon
>     won't be using the same tags for Chinese that I do. Even if I
>     don't follow their lead, others likely will. This worries me.
>     Also, the rules for #2 must include fuzzy guidelines such as, "use
>     the 'zh' tag except when you think it's a bad idea" and "use the
>     shortest tag except when you don't want to." This presents
>     complications in trying to explain some sort of consistent method
>     to the LTRU madness to others. Given this, I start to wish ISO
>     639-6 a safe and speedy passage.
>
>     * Option #3 is what I believe you might suggest, but for me,
>     that's the worst list of all. There are five ambiguous "zh"
>     categories on that list. It follows the "always use the shortest
>     tag" rule and respects history, but it's useless to me from an
>     identification perspective.
>
>
> Your list is already ambiguous for columns 1 and 2; you are using 
> "yue" for two different things (written and spoken). The only change 
> it really makes is that you don't have a term for "any chinese".
>
> RFC 4646 lacks terms for many, many combinations of things: a term for 
> "any german" (including de, gsw, ...), "any french", "any 
> scandinavian", or any one of the countless other possible sets of 
> languages that people consider to be important for some particular 
> purpose. That's why lists of languages are really the appropriate vehicle.
>
>
>
>     * Option #4 has three ambiguous tags and means I have to explain
>     to people who aren't in this industry about why I use different
>     tags for the same language. This strategy is less ambiguous that
>     #3, but I'm not sure I can explain it to other content creators
>     for the same reasons as #2 and presents the spoken/written
>     complication others may not want. In the long run, this seems
>     messy and unclear enough that it will result in bad tagging.
>
>     * Options #2,3,4: In general, it worries me that RFC 4646bis
>     offers so many "preferred" options for the same thing. I really
>     can't see how this simplifies things for anyone.
>
>     I don't have a need for fuzzy fallback scenarios. I need precise
>     tags and mostly simple lookup. I think if you take the fallback
>     scenarios and absurdities out of the document you reference, I
>     don't think there's much left.
>
>
> The only purpose I have heard for extlang *is* for fallback; that's 
> why the document goes into (painful) depth on that topic. For 
> identification alone, "zh" and "zh-cmn" really mean just the same 
> thing. It is only in the context of matching (filtering and lookup) 
> that they differ in semantics *because of their behavior*: where "cmn" 
> means simply Cantonese, "zh-cmn" effectively means "Cantonese but 
> fallback to any Chinese". 
>
>
>
>     Regards,
>
>     Karen Broome
>
>
>
>
>     >From: [email-masked] <mailto:[email-masked]>
>     [mailto:[email-masked] <mailto:[email-masked]>] On
>     Behalf
>     >Of Mark Davis
>     >To: [email-masked] <mailto:[email-masked]>
>     >Cc: LTRU Working Group
>     >http://docs.google.com/Doc?docid=dfqr8rd5_676kxxxjhd&hl=en
>     <http://docs.google.com/Doc?docid=dfqr8rd5_676kxxxjhd&hl=en> are not a
>     >problem.
>     >
> ------------------------------------------------------------------------
>