Subject: Re: [Ltru] Ticket #45: AD Issue #12: reason for SHOULD in 4.5 extlang mapping

From: "Doug Ewell" <[email-masked]>
Date: Wed, 29 Apr 2009 21:32:17 -0600

Mark Davis <mark at macchiato dot com> wrote:

> The changes are marked in yellow.

Not useful when reading the plain-text digest.

> A language tag is in canonical form when:
> ...
> 2.  It has been canonicalized according to the following process:
> ...
> 2.  Subtags of type 'extlang' MUST be mapped to their Preferred-Value.

As Addison pointed out, the existing wording with SHOULD was a 
compromise.  The battle between extlang and no-extlang camps was lengthy 
and painful, and this was the wording the WG finally agreed upon.  I 
object to using the IETF Last Call process to undo this compromise and 
swing the wording back in favor of the no-extlang side.

--
Doug Ewell  *  Thornton, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 30 Apr 2009 08:20:32 -0700

Okay, I've finished generating the new draft and diffing it. Our only remaining open item is this issue.

Last night I proposed defining two canonical forms to resolve the muddiness of "SHOULD". I now have a separate editor's copy of the draft with a proposed edit to accomplish this. Co-chair guidance on this issue is very much desired.

Below is my proposed edited text for section 4.5. Suggestions and fixes are welcome. In particular, I haven't said anything about when to use which form.

--
4.5.  Canonicalization of Language Tags

   Since a particular language tag is sometimes used by many processes,
   language tags SHOULD always be created or generated in a canonical
   form.

   There are two canonical forms for language tags.  The 'default'
   canonical form maps each 'extlang' subtag to its Preferred-Value.
   The 'extended' canonical form includes the macrolanguage primary
   language subtag before eligible (extended) language subtags.

   A language tag is in a canonical form when:

   1.  The tag is well-formed according the rules in Section 2.1 and
       Section 2.2.

   2.  Redundant or grandfathered tags that have a Preferred-Value
       mapping in the IANA registry (see Section 3.1) MUST be replaced
       with their mapped value.  These items either are deprecated
       mappings created before the adoption of this document (such as
       the mapping of "no-nyn" to "nn" or "i-klingon" to "tlh") or are
       the result of later registrations or additions to this document
       (for example, "zh-hakka" was deprecated in favor of the ISO 639-3
       code 'hak' when this document was adopted).  These mappings
       SHOULD be done before additional processing, since there can be
       additional changes to subtag values.  These field-body of the
       Preferred-Value for grandfathered and redundant tags is an
       "extended language range" ([RFC4647]) and might consist of more
       than one subtag.

   3.  In the 'default' canonical form, subtags of type 'extlang' MUST
       be mapped to their Preferred-Value.  The field-body of the
       Preferred-Value for extlangs is an "extended language range",
       typically a primary language subtag (in all such cases, the
       primary language subtag is removed).  For example, the subtag
       sequence "zh-hak" (Chinese, Hakka) would be replaced with the tag
       "hak" (Hakka).

   4.  In the 'extended' canonical form, primary language subtags with a
       'Macrolanguage' field that are also registered as 'extlang'
       subtags MUST be replaced by their macrolanguage-extlang
       combination.  For example, the language tag "hak" (Hakka) has a
       Macrolanguage of 'zh' (Chinese) and an existing 'extlang'
       registration.  The tag would be replaced with that tag "zh-hak"
       (Chinese, Hakka).

   5.  Other subtags that have a Preferred-Value field in the IANA
       registry (see Section 3.1) MUST be replaced with their mapped
       value.  Most of these are either Region subtags where the country
       name or designation has changed or clerical corrections to ISO
       639-1.

   6.  If more than one extension subtag sequence exists, the extension
       sequences are ordered into case-insensitive ASCII order by
       singleton subtag (that is, the subtag sequence '-a-babble' comes
       before '-b-warble').

   Example: The language tag "en-a-aaa-b-ccc-bbb-x-xyz" is in canonical
   form, while "en-b-ccc-bbb-a-aaa-X-xyz" is well-formed and potentially
   valid (extensions 'a' and 'b' are not defined as of the publication
   of this document) but not in canonical form (the extensions are not
   in alphabetical order).

   Example: Although the tag "en-BU" (English as used in Burma)
   maintains its validity, the language tag "en-BU" is not canonical
   because the 'BU' subtag has a canonical mapping to 'MM' (Myanmar).

   Canonicalization of language tags does not imply anything about the
   use of upper or lowercase letters when processing or comparing
   subtags (and as described in Section 2.1).  All comparisons MUST be
   performed in a case-insensitive manner.

   When performing canonicalization of language tags, processors MAY
   regularize the case of the subtags (that is, this process is
   OPTIONAL), following the case used in the registry (see
   Section 2.1.1).

   If more than one variant appears within a tag, processors MAY reorder
   the variants to obtain better matching behavior or more consistent
   presentation.  Reordering of the variants SHOULD follow the
   recommendations for variant ordering in Section 4.1.

   If the field 'Deprecated' appears in a registry record without an
   accompanying 'Preferred-Value' field, then that tag or subtag is
   deprecated without a replacement.  These values are canonical when
   they appear in a language tag.  However, tags that include these
   values SHOULD NOT be selected by users or generated by
   implementations.

   An extension MUST define any relationships that exist between the
   various subtags in the extension and thus MAY define an alternate
   canonicalization scheme for the extension's subtags.  Extensions MAY
   define how the order of the extension's subtags are interpreted.  For
   example, an extension could define that its subtags are in canonical
   order when the subtags are placed into ASCII order: that is, "en-a-
   aaa-bbb-ccc" instead of "en-a-ccc-bbb-aaa".  Another extension might
   define that the order of the subtags influences their semantic
   meaning (so that "en-b-ccc-bbb-aaa" has a different value from "en-b-
   aaa-bbb-ccc").  However, extension specifications SHOULD be designed
   so that they are tolerant of the typical processes described in
   Section 3.7.
--

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Doug Ewell
> Sent: Wednesday, April 29, 2009 8:32 PM
> To: LTRU Working Group
> Subject: Re: [Ltru] Ticket #45: AD Issue #12: reason for SHOULD in
> 4.5 extlang mapping
> 
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Thu, 30 Apr 2009 11:32:37 -0400

>    4.  In the 'extended' canonical form, primary language subtags with a

I propose the names "short" and "long" respectively.

-- 
John Cowan              [email-masked]          http://www.ccil.org/~cowan
Any day you get all five woodpeckers is a good day.  --Elliotte Rusty Harold

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 30 Apr 2009 09:13:11 -0700

(as individual contributor)
> I propose the names "short" and "long" respectively.

'extended' seems like a more natural name for the latter form. I acknowledge that the name 'default' is prejudicial, in its way, but it was the previous intent of 'SHOULD' and it is the canonical form that the registry is optimized for. If we don't want to say that it's the default, we could pick a name that is partial antonym for 'extended', such as 'compact', 'regular', or (hmm...) 'short'. How about 'short' and 'extended'?

If we don't say 'default' (or even if we do), oughtn't we to have some text explaining when to use which?

Addison

> Any day you get all five woodpeckers is a good day.  --Elliotte

All five woodpeckers?!?

---

From: Mark Davis <[email-masked]>
Date: Thu, 30 Apr 2009 11:51:35 -0700

Default is fine; that preserves the sense of the SHOULD. I had a couple of
other comments on the proposed text, will send separately.

Mark

On Thu, Apr 30, 2009 at 09:13, Phillips, Addison <[email-masked]> wrote:

> 'extended' seems like a more natural name for the latter form. I
> acknowledge that the name 'default' is prejudicial, in its way, but it was
> the previous intent of 'SHOULD' and it is the canonical form that the
> registry is optimized for. If we don't want to say that it's the default, we
> could pick a name that is partial antonym for 'extended', such as 'compact',
> 'regular', or (hmm...) 'short'. How about 'short' and 'extended'?
>

---

From: Mark Davis <[email-masked]>
Date: Thu, 30 Apr 2009 12:00:36 -0700

The changes are not an attempt to sway one way or another; it is to have
comprehensible text.

When you define what a term is, you can't say SHOULD. It is when you talk
about the *usage* of that defined term that you can say SHOULD or MUST.

The very nature of a canonicalization is that once you are done, the results
erase differences where two forms are equivalent. If there is only a SHOULD,
then that doesn't work. We all agree that "cmn" and "zh-cmn" represent the
same underlying entity, so any canonicalization has to go one way or the
other. We can't have implemenation A canonicalize <ar-arb, zh-cmn, yue> to
<arb, zh-cmn, yue> and implemenation B canonicalize to <ar-arb, zh-cmn,
yue>, and implementation C canonicalize to <arb, zh, zh-yue>. Yet having the
SHOULD allows all of these. The SHOULD is in the wrong place.

We need to have two form of canonicalization, and have the SHOULD be on *
which* form you should use. I went part way in that direction, and Addison's
extentions are go further.

Mark

---------- Forwarded message ----------
From: Doug Ewell <[email-masked]>
Date: Wed, Apr 29, 2009 at 20:32
Subject: Re: [Ltru] Ticket #45: AD Issue #12: reason for SHOULD in 4.5
extlang mapping
To: LTRU Working Group <[email-masked]>

> 2.  Subtags of type 'extlang' MUST be mapped to their Preferred-Value.

As Addison pointed out, the existing wording with SHOULD was a compromise.
 The battle between extlang and no-extlang camps was lengthy and painful,
and this was the wording the WG finally agreed upon.  I object to using the
IETF Last Call process to undo this compromise and swing the wording back in
favor of the no-extlang side.

---

From: Mark Davis <[email-masked]>
Date: Thu, 30 Apr 2009 12:17:12 -0700

Mark

On Thu, Apr 30, 2009 at 08:20, Phillips, Addison <[email-masked]> wrote:

> Last night I proposed defining two canonical forms to resolve the muddiness
> of "SHOULD". I now have a separate editor's copy of the draft with a
> proposed edit to accomplish this. Co-chair guidance on this issue is very
> much desired.
>
> Below is my proposed edited text for section 4.5. Suggestions and fixes are
> welcome. In particular, I haven't said anything about when to use which
> form.
>
> --

I agree with these names. If we change the name of 'default' to anything
else, we'd have to make an additional change to preserve the compromise
SHOULD from the previous text, something like:

The default canonical form SHOULD normally be used for
canonicalization. The extended canonical form may be useful
in environments where the presence of the macrolanguage is beneficial
in matching or selection.

>   1.  The tag is well-formed according the rules in Section 2.1 and

This omits the significant consistency problem muddling defining a canonical
form, and defining the process for canonicalizing. Please change the
numbering/indent for 2-4 and add:

>        sequences are ordered into case-insensitive ASCII order by

Please change "are ordered" to "MUST be ordered". This is not optional in
exactly the same sense as all the other clauses are not optional. (That is,
they must all be "are" or all be "MUST".)

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 30 Apr 2009 13:53:30 -0700

Although I agree with your sentiments, I think we can avoid some additional working group bloodletting by being careful with our wording choices. I particularly note that RFC 2119 keywords, while have a normative weight to them, are not the only way to be “normative” in a spec. Thus, I propose to take your suggestion:

--
The default canonical form SHOULD normally be used for canonicalization. The extended canonical form may be useful
in environments where the presence of the macrolanguage is beneficial in matching or selection.
--

… but modified to read…

--
<t>Normally, the 'default' canonicalization is preferred. However, the 'extended' canonical form is useful
in environments where the presence of the macrolanguage is beneficial in matching or selection (see <xref target="choiceUsingExtlang"></xref>).</t>
--

In incorporating your proposed changes, I noticed that there was no need for double- or triple-list embedding. I also note that, since we are describing a canonicalization process, using 2119 keywords isn’t necessary. We can just say “do X”.

The resulting text reads as:

--
A language tag is in a canonical form when the tag is well-formed according the rules in <xref target="syntax"/> and
<xref target="sources"/> and it has been canonicalized as follows:

 <list style="numbers">

<t>Redundant or grandfathered tags that have a Preferred-Value mapping in the IANA registry (see <xref target="ianaformat"/>) are replaced with their mapped value. These items are either deprecated mappings created before the adoption of this document (such as the mapping of "no-nyn" to "nn" or "i-klingon" to "tlh") or are the result of later registrations or additions to this document (for example, "zh-hakka" was deprecated in favor of the ISO 639-3 code 'hak' when this document was adopted). These mappings MUST be done before additional processing, since there can be additional changes to subtag values. These field-body of the Preferred-Value for grandfathered and redundant tags is an "extended language range" (<xref target="RFC4647"></xref>) and might consist of more than one subtag.</t>

<t>One of the following canonical forms has been applied:

<list style="letters">

<t>In the 'default' canonical form, subtags of type 'extlang' are mapped to their Preferred-Value. The field-body of the Preferred-Value for extlangs is an "extended language range", typically a primary language subtag (in all such cases, the primary language subtag is removed). For example, the subtag sequence "zh-hak" (Chinese, Hakka) would be replaced with the tag "hak" (Hakka).</t>

<t>In the 'extended' canonical form, primary language subtags with a 'Macrolanguage' field that are also registered as 'extlang' subtags are replaced by their macrolanguage-extlang combination. For example, the language tag "hak" (Hakka) has a Macrolanguage of 'zh' (Chinese) and an existing 'extlang' registration. The tag would be replaced with that tag "zh-hak" (Chinese, Hakka).</t>

</list></t>

<t>Other subtags that have a Preferred-Value field in the IANA registry (see <xref target="ianaformat"/>) have been replaced with their mapped value. Most of these are either Region subtags where the country name or designation has changed or clerical corrections to ISO 639-1.</t>

<t>If more than one extension subtag sequence exists, the extension sequences are ordered into case-insensitive ASCII order by singleton subtag (that is, the subtag sequence '-a-babble' comes before '-b-warble').</t>

</list>
--

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Thursday, April 30, 2009 12:17 PM
To: Phillips, Addison
Cc: Doug Ewell; LTRU Working Group
Subject: Re: [Ltru] Ticket #45: AD Issue #12: reason for SHOULD in 4.5 extlang mapping

Mark

On Thu, Apr 30, 2009 at 08:20, Phillips, Addison <[email-masked]<mailto:[email-masked]>> wrote:
Okay, I've finished generating the new draft and diffing it. Our only remaining open item is this issue.

I agree with these names. If we change the name of 'default' to anything else, we'd have to make an additional change to preserve the compromise SHOULD from the previous text, something like:

The default canonical form SHOULD normally be used for canonicalization. The extended canonical form may be useful

in environments where the presence of the macrolanguage is beneficial in matching or selection.

This omits the significant consistency problem muddling defining a canonical form, and defining the process for canonicalizing. Please change the numbering/indent for 2-4 and add:

Please change "are ordered" to "MUST be ordered". This is not optional in exactly the same sense as all the other clauses are not optional. (That is, they must all be "are" or all be "MUST".)

Addison Phillips

> From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On
> Behalf Of Doug Ewell
> [email-masked]<mailto:[email-masked]>
_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 30 Apr 2009 13:55:27 -0700

One additional note: pursuant to your comment that people either MUST or MUST NOT do things in canonicalization, I think we need to modify this text:

--
These mappings
      SHOULD be done before additional processing, since there can be
      additional changes to subtag values. 
--

.. and require it as the first step. It should read:

--
These mappings MUST be done before additional processing, since there can be additional changes to subtag values.
--

Addison Phillips

Mark

Addison Phillips

---

From: Mark Davis <[email-masked]>
Date: Thu, 30 Apr 2009 13:56:34 -0700

Agreed, good catch.

Mark

On Thu, Apr 30, 2009 at 13:55, Phillips, Addison <[email-masked]> wrote:

> One additional note: pursuant to your comment that people either MUST or
> MUST NOT do things in canonicalization, I think we need to modify this text:
>
> --
> These mappings
> The default canonical form SHOULD normally be used for canonicalization.
> The extended canonical form may be useful
> This omits the significant consistency problem muddling defining a
> canonical form, and defining the process for canonicalizing. Please change
> the numbering/indent for 2-4 and add:

---

From: Mark Davis <[email-masked]>
Date: Thu, 30 Apr 2009 14:16:01 -0700

works for me; comments below.

Mark

On Thu, Apr 30, 2009 at 13:53, Phillips, Addison <[email-masked]> wrote:

>  Although I agree with your sentiments, I think we can avoid some
> additional working group bloodletting by being careful with our wording
> choices. I particularly note that RFC 2119 keywords, while have a normative
> weight to them, are not the only way to be “normative” in a spec. Thus, I
> propose to take your suggestion:
>
>
>
> --
>
> in environments where the presence of the macrolanguage is beneficial in
> matching or selection (see <xref target="choiceUsingExtlang"></xref>).</t>
>

I'm fine with that. My interest is not in further blood-letting, but rather
just to preserve the consensus exactly as we have had it, but remove
inconsistencies.

> --
>
>
>
> In incorporating your proposed changes, I noticed that there was no need
> for double- or triple-list embedding. I also note that, since we are
> describing a canonicalization process, using 2119 keywords isn’t necessary.
> We can just say “do X”.
>
>
>

That works also. I prefer definitions in this style.

>
>
>
> <t>Redundant or grandfathered tags that have a Preferred-Value mapping in
> the IANA registry (see <xref target="ianaformat"/>) are replaced with their
> mapped value. These items are either deprecated mappings created before the
> adoption of this document (such as the mapping of "no-nyn" to "nn" or
> "i-klingon" to "tlh") or are the result of later registrations or additions
> to this document (for example, "zh-hakka" was deprecated in favor of the ISO
> 639-3 code 'hak' when this document was adopted). These mappings MUST be
> done before additional processing, since there can be additional changes to
> subtag values. These field-body of the Preferred-Value for grandfathered and
> redundant tags is an "extended language range" (<xref
> target="RFC4647"></xref>) and might consist of more than one subtag.</t>
>
>
>
> <t>In the 'default' canonical form, subtags of type 'extlang' are mapped to
> their Preferred-Value. The field-body of the Preferred-Value for extlangs is
> an "extended language range", typically a primary language subtag (in all
> such cases, the primary language subtag is removed). For example, the subtag
> sequence "zh-hak" (Chinese, Hakka) would be replaced with the tag "hak"
> (Hakka).</t>
>
>
>
> <t>In the 'extended' canonical form, primary language subtags with a
> 'Macrolanguage' field that are also registered as 'extlang' subtags are
> replaced by their macrolanguage-extlang combination. For example, the
> language tag "hak" (Hakka) has a Macrolanguage of 'zh' (Chinese) and an
> existing 'extlang' registration. The tag would be replaced with that tag
> "zh-hak" (Chinese, Hakka).</t>
>
>
>
> </list></t>
>
>
>
> <t>Other subtags that have a Preferred-Value field in the IANA registry
> (see <xref target="ianaformat"/>) have been replaced with their mapped
> value. Most of these are either Region subtags where the country name or
> designation has changed or clerical corrections to ISO 639-1.</t>
>
>
>
> <t>If more than one extension subtag sequence exists, the extension
> sequences are ordered into case-insensitive ASCII order by singleton subtag
> (that is, the subtag sequence '-a-babble' comes before '-b-warble').</t>
>
>
>
> </list>
>
> --
>
>
>
> Addison Phillips
>
> *From:* [email-masked] [mailto:[email-masked]] *On
> Behalf Of *Mark Davis
> *Sent:* Thursday, April 30, 2009 12:17 PM
> *To:* Phillips, Addison
> *Cc:* Doug Ewell; LTRU Working Group
>
> *Subject:* Re: [Ltru] Ticket #45: AD Issue #12: reason for SHOULD in 4.5
> extlang mapping
>
>
>
>
> Mark
>

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 30 Apr 2009 14:31:12 -0700

Hi -

As co-chair...

I'm concerned about the amount of textual change proposed in
resultion of this item.  Without prejudicing the merits of these
changes, we should recall that the AD's comment merely asked for
an explanation of the "SHOULD".  I'm reluctant to declare consensus
on a change this extensive without hearing from more working
group participants, particularly since this was such a hotly
contested portion of the specification.

Randy