Subject: Re: [Ltru] Ticket #45: AD Issue #12: reason for SHOULD in 4.5 extlang mapping

From: "Doug Ewell" <[email-masked]>
Date: Sat, 2 May 2009 19:22:29 -0600

Kent Karlsson <kent dot karlsson14 at comhem dot se> wrote:

> Well, that was the point. I think the extlang canonical form for the 
> sign languages should be "sgn-zzz" for zzz as one of the sign language 
> code. We made sign languages extlangs despite that sgn is a collection 
> code and not a macrolanguage code, just because the exlang proponents 
> wanted "sgn" at the beginning of sign language language tags, did we 
> not?

Kent is correct.  The Scope field attached to a primary language subtag 
reflects the ISO 639-3 assignment for that code element.  It is not 
intended to indicate whether there is a primary/extlang association. 
That is the role of the Prefix field attached to the extlang.

The canonicalization rules for "swl" versus "sgn-swl" are supposed to be 
exactly the same as those for "yue" versus "zh-yue".  Section 4.1.2 
affirms this.  Any change in wording that treats these two examples 
differently, or assigns them different canonicalization rules on the 
basis that "zh" is defined as a macrolanguage in ISO 639-3 and "sgn" is 
not, is a TECHNICAL change, and should be ruled out of scope for this 
post-WGLC, post-IETF LC period.

--
Doug Ewell  *  Thornton, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

---

From: Mark Davis <[email-masked]>
Date: Sun, 3 May 2009 02:42:44 -0700

On Sat, May 2, 2009 at 18:22, Doug Ewell <[email-masked]> wrote:

>  Well, that was the point. I think the extlang canonical form for the sign
>> languages should be "sgn-zzz" for zzz as one of the sign language code. We
>> made sign languages extlangs despite that sgn is a collection code and not a
>> macrolanguage code, just because the exlang proponents wanted "sgn" at the
>> beginning of sign language language tags, did we not?
>>
>
> reflects the ISO 639-3 assignment for that code element.

If you look at my example, I was *not* talking about the Scope field. I
carefully marked what I was talking about, which was the Macrolanguage field
in the extlang record, NOT the Scope field in the primary language subtag
record.

*Macrolanguage: zh* *// only material difference*

 It is not intended to indicate whether there is a primary/extlang
> association. That is the role of the Prefix field attached to the extlang.
>
> exactly the same as those for "yue" versus "zh-yue".  Section 4.1.2 affirms
> this.

This was not trying to make any material change; I'd actually thought that
that we had not wanted to add 'sgn'. As I recall the discussion, while there
is some case to be made that falling back from yue to zh (with likely
content = cmn) may make sense, the fallback to an arbitrary other sign
language is probably completely bogus. 4.1.2 actually doesn't affirm that
one actively wants to add 'sgn' to, say, bqy.

And note that Addison used the same terminology "*macrolanguage-extlang"* --
I had copied his language -- when the correct term would be "prefix-extlang"
according to your interpretation, since the prefix may not be a
macrolanguage. But if you both are convinced that we'd had a consensus to
add 'sgn', I have no interest in even looking back at old emails to verify.
I don't care either way about 'sgn'; I was just trying to supply explicit
language for how to deal with adding the macrolanguage. (You are rather
eager to assume a Conspiracy instead of a mistake.)

Because this is the first time the *addition* of primary language subtags
has really been clearly stated, it only now that we are seeing that people
could misinterpret the text, so I'm glad we are having this discussion.

My proposed text on
http://sites.google.com/site/macchiato/unicode/ltru-canonicalization has
been modified to be:

*4. In the 'extlang' canonical form (but not the 'default' form), primary
language subtags that are also extlang subtags are prepended with the
extlang's Prefix.**

   - For example, "hak-CN" (Hakka, China) has primary language subtag 'hak',
   which in turn has an 'extlang' record with a Prefix 'zh' (Chinese).
The 'extlang'
   canonical form would be "zh-hak-CN" (Chinese, Hakka, China)

*If you have any other suggestions for making this clearer, let me know.

>  Any change in wording that treats these two examples differently, or
> assigns them different canonicalization rules on the basis that "zh" is
> defined as a macrolanguage in ISO 639-3 and "sgn" is not, is a TECHNICAL
> change, and should be ruled out of scope for this post-WGLC, post-IETF LC
> period.

> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ltru
>

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 3 May 2009 09:53:07 -0600

Mark Davis wrote:

> If you look at my example, I was *not* talking about the Scope field. 
> I carefully marked what I was talking about, which was the 
> Macrolanguage field in the extlang record, NOT the Scope field in the 
> primary language subtag record.
>
> Macrolanguage: zh // only material difference

You are right and I apologize for the confusion.  You did suggest using 
the Macrolanguage field on the encompassed language, not the Scope field 
of the encompassing language, to determine whether extlang 
canonicalization would be used.

However, these two fields are supposed to exist in perfect balance with 
one another: for every language subtag A with "Scope: macrolanguage", 
there exist one or more language subtags B1 through Bn with 
"Macrolanguage: A".  Both the Macrolanguage (3.1.10) and Scope (3.1.11) 
fields reflect ISO 639-3 usage.  And in fact, the Registry includes 
dozens of ISO 639-3 macrolanguage/encompassed language relationships 
that have no bearing on BCP 47 extlang usage:

Type: language
Subtag: bal
Description: Baluchi
Added: 2005-10-16
Scope: macrolanguage
...
Type: language
Subtag: bcc
Description: Southern Balochi
Added: 2029-09-09
Macrolanguage: bal

As I wrote earlier, the Prefix field attached to an extlang record is 
the field that is supposed to indicate whether a primary/extlang 
relationship exists.  Actually, I suppose the Preferred-Value carries 
this meaning at least as well, if not more so, since it contains the 
primary language subtag that is "supposed" to be used instead if the 
goal is to canonicalize away extlangs.

Note that the sign language extlangs do have both a Prefix ("sgn") and a 
Preferred-Value, while the Southern Balochi example above has neither. 
That is because sign languages were intended to participate in THE 
extlang behavior while the Baluchi languages were not.  There were not 
two different sets of rules for extlang behavior, one for "sgn" and 
another for the "Big Six" (Arabic, Konkani, etc.) described in 
draft-4645bis, Section 2.2.  The only difference was that we were 
applying the one set of rules to sign languages even though ISO 639-3 
recognizes no macro/encompassed relationship (and in fact doesn't even 
include the code element "sgn").

> This was not trying to make any material change; I'd actually thought 
> that that we had not wanted to add 'sgn'. As I recall the discussion, 
> while there is some case to be made that falling back from yue to zh 
> (with likely content = cmn) may make sense, the fallback to an 
> arbitrary other sign language is probably completely bogus. 4.1.2 
> actually doesn't affirm that one actively wants to add 'sgn' to, say, 
> bqy.

I think it is fair to say that attempting to fall back from one sign 
language to another is risky behavior.  The last paragraph of 4.1.2 
explains, I think adequately, why sign languages in "sgn" were included 
to be representable with either primary language subtags or extlangs. 
The combination of primary and extended language subtags denotes a 
relationship between the subtags; it is not only for Web browser 
fallback.

> And note that Addison used the same terminology 
> "macrolanguage-extlang" -- I had copied his language -- when the 
> correct term would be "prefix-extlang" according to your 
> interpretation, since the prefix may not be a macrolanguage.

That is correct.

> But if you both are convinced that we'd had a consensus to add 'sgn', 
> I have no interest in even looking back at old emails to verify. I 
> don't care either way about 'sgn'; I was just trying to supply 
> explicit language for how to deal with adding the macrolanguage.

Wording that clarifies might be helpful, but I agree with Randy that we 
are engaging in an awful lot of textual revision when the AD's original 
question was simply to clarify a SHOULD.  The more text we revise, the 
greater the chance of slipping away from the intent of the WG, whether 
deliberately or accidentally.

> (You are rather eager to assume a Conspiracy instead of a mistake.)

I am in this case.  We had a delicately brokered truce, and I don't want 
to see it touched on either side of the table.  (Notice that I haven't 
proposed any text to remove the SHOULD from the no-extlang form, even 
though I wanted that.)  I understand that your studies showed that 
search engines, such as Google, did not work with extlangs as well as 
anticipated.  There were other use cases for language tags besides Web 
searches, in which extlangs provided more benefit.

> Because this is the first time the *addition* of primary language 
> subtags has really been clearly stated, it only now that we are seeing 
> that people could misinterpret the text, so I'm glad we are having 
> this discussion.

That is a good point.

> My proposed text on 
> http://sites.google.com/site/macchiato/unicode/ltru-canonicalization 
> has been modified to be:
>
> 4. In the 'extlang' canonical form (but not the 'default' form), 
> primary language subtags that are also extlang subtags are prepended 
> with the extlang's Prefix.
> For example, "hak-CN" (Hakka, China) has primary language subtag 
> 'hak', which in turn has an 'extlang' record with a Prefix 'zh' 
> (Chinese). The 'extlang' canonical form would be "zh-hak-CN" (Chinese, 
> Hakka, China)
> If you have any other suggestions for making this clearer, let me 
> know.

I think that should be fine.  But we have revised so much text here that 
a full regression text of the section is probably necessary, which I 
find unfortunate at this stage.

---

From: Mark Davis <[email-masked]>
Date: Sun, 3 May 2009 17:15:47 -0700

...
>> 4. In the 'extlang' canonical form (but not the 'default' form), primary
>> extlang's Prefix.
>> For example, "hak-CN" (Hakka, China) has primary language subtag 'hak',
>> which in turn has an 'extlang' record with a Prefix 'zh' (Chinese). The
>> 'extlang' canonical form would be "zh-hak-CN" (Chinese, Hakka, China)
>> If you have any other suggestions for making this clearer, let me know.
>>
>
> I think that should be fine.  But we have revised so much text here that a
> full regression text of the section is probably necessary, which I find
> unfortunate at this stage.

I'm not sure what you mean by a "full regression text". Do you mean full
replacement of the text, or something different?

I'm not particularly pleased by this level of change either, but it has
become clear that the previous text was not actually coherent (usually a bad
thing in a spec ;-). We have striven to have no material impact, so if the
WG can look over the proposed text and assent to it, I'm hoping that we can
move on.

...

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 3 May 2009 21:14:45 -0600

Mark Davis wrote:

> I'm not sure what you mean by a "full regression text". Do you mean 
> full replacement of the text, or something different?

Sorry, that was a typo for "full regression test."  What I meant was 
that we would have to go back through the entire section and check it 
all over again for accuracy, not just the parts we thought were changed, 
because the changes are so extensive.

> I'm not particularly pleased by this level of change either, but it 
> has become clear that the previous text was not actually coherent 
> (usually a bad thing in a spec ;-). We have striven to have no 
> material impact, so if the WG can look over the proposed text and 
> assent to it, I'm hoping that we can move on.

As long as there is no change in the meaning intended by the WG, and as 
long as the cycle of reading the text and finding new ways to improve it 
eventually comes to an end, I agree that this is a good thing.

---

From: Mark Davis <[email-masked]>
Date: Mon, 4 May 2009 07:43:21 -0700

Mark

On Sun, May 3, 2009 at 20:14, Doug Ewell <[email-masked]> wrote:

> Sorry, that was a typo for "full regression test."  What I meant was that
> we would have to go back through the entire section and check it all over
> again for accuracy, not just the parts we thought were changed, because the
> changes are so extensive.
>

Good. By the way, those of in on this Nefarious Scheme do try to follow the
rules of http://www.eviloverlord.com/lists/overlord.html ;-)