Subject: [Ltru] Ticket #45: updated editor's copy available

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 5 May 2009 11:07:16 -0700

All,

I am back from a recent soggy camping trip and catching up with this thread. I have just now posted an editor's copy of the text that has been discussed on this thread to inter-locale. Note that I have made some minor edits to the proposed text (to ensure it matches the style of the document; uses RFC 2119 keywords properly; and is grammatically correct). 

Here are the links:

  Diff: http://tinyurl.com/cyrmju
  HTML: http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed-md.html
  TXT: http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-222-ed-mt.txt 

Here is the proposed text for Section 4.5:

--
4.5.  Canonicalization of Language Tags

   Since a particular language tag is sometimes used by many processes,
   language tags SHOULD always be created or generated in a canonical
   form.

   There are two canonical forms for language tags: the 'default'
   canonical form contains no extended language subtags, while the
   'extlang' canonical form contains extended language subtags where
   required.  Normally, the 'default' canonicalization is preferred.
   However, the 'extlang' canonical form can be useful in environments
   where the presence of the enclosing primary language subtag is
   considered beneficial to matching or selection (see Section 4.1.2)

   A language tag is in a canonical form, either default or extended,
   when the tag is well-formed according the rules in Section 2.1 and
   Section 2.2 and it has been canonicalized by applying each of the
   following steps in order, using data from the IANA registry (see
   Section 3.1):

   1.  Extension sequences are ordered into case-insensitive ASCII order
       by singleton subtag.

       *  That is, the subtag sequence '-a-babble' comes before
          '-b-warble'.

   2.  Redundant or grandfathered tags are replaced by their Preferred-
       Value, if there is one.

       *  These items are either deprecated mappings created before the
          adoption of this document (such as the mapping of "no-nyn" to
          "nn" or "i-klingon" to "tlh") or are the result of later
          registrations or additions to this document (for example, "zh-
          hakka" was deprecated in favor of the ISO 639-3 code 'hak'
          when this document was adopted).

       *  Note: The field-body of the Preferred-Value for grandfathered
          and redundant tags is an "extended language range" ([RFC4647])
          and might consist of more than one subtag.

   3.  Subtags are replaced by their Preferred-Value, if there is one.
       For extended language subtags, the original primary language
       subtag is also replaced if there is a primary language subtag in
       the Preferred-Value.

       *  The field-body of the Preferred-Value for extlangs is an
          "extended language range" and almost always consists of a
          single, primary language subtag.  For example, the subtag
          sequence "zh-hak" (Chinese, Hakka) would be replaced with the
          tag "hak" (Hakka).

       *  The field-body of the Preferred-Value for all other types of
          subtags consists of a subtag of the same type.  Most of these
          non-extlang subtags are either Region subtags where the
          country name or designation has changed or are clerical
          corrections to ISO 639-1.

   4.  In the 'extlang' canonical form (but not the 'default' canonical
       form), primary language subtags that are also extlang subtags are
       prepended with the extlang's Prefix.

       *  For example, "hak-CN" (Hakka, China) has a primary language
          subtag of 'hak', which also appears in the registry as an
          'extlang' record with a Prefix 'zh' (Chinese).  The 'extlang'
          canonical form would be "zh-hak-CN" (Chinese, Hakka, China).

       *  Note that this step can restore a subtag that was removed by
          the previous step.

   Example: The language tag "en-a-aaa-b-ccc-bbb-x-xyz" is in a
   canonical form, while "en-b-ccc-bbb-a-aaa-X-xyz" is well-formed and
   potentially valid (extensions 'a' and 'b' are not defined as of the
   publication of this document) but not in a canonical form (the
   extensions are not in alphabetical order).

   Example: Although the tag "en-BU" (English as used in Burma)
   maintains its validity, the language tag "en-BU" is not in a
   canonical form because the 'BU' subtag has a canonical mapping to
   'MM' (Myanmar).

   Canonicalization of language tags does not imply anything about the
   use of upper or lowercase letters when processing or comparing
   subtags (and as described in Section 2.1).  All comparisons MUST be
   performed in a case-insensitive manner.

   When performing canonicalization of language tags, processors MAY
   regularize the case of the subtags (that is, this process is
   OPTIONAL), following the case used in the registry (see
   Section 2.1.1).

   If more than one variant appears within a tag, processors MAY reorder
   the variants to obtain better matching behavior or more consistent
   presentation.  Reordering of the variants SHOULD follow the
   recommendations for variant ordering in Section 4.1.

   If the field 'Deprecated' appears in a registry record without an
   accompanying 'Preferred-Value' field, then that tag or subtag is
   deprecated without a replacement.  These values are canonical when
   they appear in a language tag.  However, tags that include these
   values SHOULD NOT be selected by users or generated by
   implementations.

   An extension MUST define any relationships that exist between the
   various subtags in the extension and thus MAY define an alternate
   canonicalization scheme for the extension's subtags.  Extensions MAY
   define how the order of the extension's subtags are interpreted.  For
   example, an extension could define that its subtags are in canonical
   order when the subtags are placed into ASCII order: that is, "en-a-
   aaa-bbb-ccc" instead of "en-a-ccc-bbb-aaa".  Another extension might
   define that the order of the subtags influences their semantic
   meaning (so that "en-b-ccc-bbb-aaa" has a different value from "en-b-
   aaa-bbb-ccc").  However, extension specifications SHOULD be designed
   so that they are tolerant of the typical processes described in
   Section 3.7.
--

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 5 May 2009 11:13:17 -0700

Argh... one of those links is a typo.

TXT: http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed-md.txt 

Addison Phillips

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Phillips, Addison
> Sent: Tuesday, May 05, 2009 11:07 AM
> To: LTRU Working Group
> Subject: [Ltru] Ticket #45: updated editor's copy available
> 
> All,
> 
> I am back from a recent soggy camping trip and catching up with
> this thread. I have just now posted an editor's copy of the text
> that has been discussed on this thread to inter-locale. Note that I
> have made some minor edits to the proposed text (to ensure it
> matches the style of the document; uses RFC 2119 keywords properly;
> and is grammatically correct).
> 
> Here are the links:
> 
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ltru

---

From: Mark Davis <[email-masked]>
Date: Tue, 5 May 2009 12:24:30 -0700

I don't think we want to have single quotes in the

'default' canonical form
'extlang' canonical form

These are names of forms, not subtags. Should be:

default canonical form
extlang canonical form

Mark

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 5 May 2009 12:39:23 -0700

Okay. Note that I (continue to) use the single quotes the first time to introduce the names. I have eliminated the other uses of the quotes.

I also note that my copy of the text read:

… but should read:

--
A language tag is in a canonical form, either default or extlang,
--

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Tuesday, May 05, 2009 12:25 PM
To: Phillips, Addison
Cc: LTRU Working Group
Subject: Re: [Ltru] Ticket #45: updated editor's copy available

'default' canonical form
'extlang' canonical form

default canonical form
extlang canonical form

Mark

On Tue, May 5, 2009 at 11:07, Phillips, Addison <[email-masked]<mailto:[email-masked]>> wrote:

---

From: Mark Davis <[email-masked]>
Date: Tue, 5 May 2009 12:53:33 -0700

Ah, I get it. They are substituting for italic on first usage since we are
limited in RFCs to plaintext*. In that case, the whole name (the part that
you would italicize) would be in quotes, eg.

... the 'default canonical form' is ...

Mark

* And since it's ASCII, we can't even use
http://unicode.org/cldr/utility/list-unicodeset.jsp?a=[\u210E\U0001D434-\U0001D467]
;-)

On Tue, May 5, 2009 at 12:39, Phillips, Addison <[email-masked]> wrote:

>  Okay. Note that I (continue to) use the single quotes the first time to
> introduce the names. I have eliminated the other uses of the quotes.
>
>
>
> *From:* [email-masked] [mailto:[email-masked]] *On
> Behalf Of *Mark Davis
> *Sent:* Tuesday, May 05, 2009 12:25 PM
> *To:* Phillips, Addison
> *Cc:* LTRU Working Group
> *Subject:* Re: [Ltru] Ticket #45: updated editor's copy available
>
>
>

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 5 May 2009 14:09:24 -0700

I changed the embrace of the ASCII quotes. And lit another candle at the shrine of UTF-8.

Addison

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Tuesday, May 05, 2009 12:54 PM
To: Phillips, Addison
Cc: LTRU Working Group
Subject: Re: [Ltru] Ticket #45: updated editor's copy available

Ah, I get it. They are substituting for italic on first usage since we are limited in RFCs to plaintext*. In that case, the whole name (the part that you would italicize) would be in quotes, eg.

Mark

* And since it's ASCII, we can't even use http://unicode.org/cldr/utility/list-unicodeset.jsp?a=[<http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5b>\u210E\U0001D434-\U0001D467] ;-)

On Tue, May 5, 2009 at 12:39, Phillips, Addison <[email-masked]<mailto:[email-masked]>> wrote:

--

--

… but should read:

--

--

Addison Phillips

It is an architecture.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Mark Davis
Sent: Tuesday, May 05, 2009 12:25 PM
To: Phillips, Addison
Cc: LTRU Working Group
Subject: Re: [Ltru] Ticket #45: updated editor's copy available

'default' canonical form
'extlang' canonical form

default canonical form
extlang canonical form

Mark

_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ltru

---

From: Mark Davis <[email-masked]>
Date: Tue, 5 May 2009 14:29:38 -0700

LOL

Mark

On Tue, May 5, 2009 at 14:09, Phillips, Addison <[email-masked]> wrote:

> *Sent:* Tuesday, May 05, 2009 12:54 PM
>
> *To:* Phillips, Addison
> *Cc:* LTRU Working Group
> http://unicode.org/cldr/utility/list-unicodeset.jsp?a=[<http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5b>\u210E\U0001D434-\U0001D467]
> ;-)
>

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 5 May 2009 23:57:31 -0700

Hi -

As a technical contributor...

> From: "Phillips, Addison" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
...
>    considered beneficial to matching or selection (see Section 4.1.2)
...

The word "required" looks a lot like the RFC 2219 keyword "REQUIRED", which
is essentially a "MUST".  Reading the proposed text in that light, the
"default" form makes no sense.

I really really really really really dislike, loathe, and abhor the very idea
of having two "canonical" forms - it seems oxymoronic.  Yes, there are
two use cases, but that does not mean that both have to be called
"canonical", or that the one true canonical form (whichever that is)
needs to serve both purposes.

Randy

---

From: Mark Davis <[email-masked]>
Date: Wed, 6 May 2009 07:42:30 -0700

The other formulation we could use goes back a bit more towards the v21 text
with a single canonical form, yet still have a clearly defined extlang form
for people who need it.  If you can use the web, it is a new third column on
http://www.macchiato.com/unicode/ltru-canonicalization. The differences from
the last formulation (B) I sent out are:

*[Remove from B: "There are two canonical forms... ...considered beneficial
in matching or selection."]**[Replace **B: "4. **In the extlang canonical
form (but not...
...Step 3." by the following]***The canonical form has no extlang subtag.
There is an alternate 'extlang form' that modifies the canonical form so
that primary language subtags that are also extlang subtags are prepended
with the extlang's Prefix. This form may be useful in environments where the
presence of the macrolanguage subtag is considered beneficial in matching or
selection (see <xref target="choiceUsingExtlang"></xref>).

   - For example, "hak-CN" (Hakka, China) has primary language subtag 'hak',
   which in turn has an 'extlang' record with a Prefix 'zh' (Chinese). The
   extlang form would be "zh-hak-CN" (Chinese, Hakka, China).

I could live with either formulation.

Mark

On Tue, May 5, 2009 at 23:57, Randy Presuhn <[email-masked]>wrote:

> ______________________________ _________________
> Ltru mailing list
> [email-masked]

---

From: Mark Davis <[email-masked]>
Date: Wed, 6 May 2009 11:42:45 -0700

Doug had a small fix, for the use of macrolanguage subtag. So

This form may be useful in environments where the presence of the
macrolanguage subtag is considered beneficial in matching or selection (see
<xref target="choiceUsingExtlang"></xref>).
=>
This form may be useful in environments where the presence of the Prefix
subtag is considered beneficial in matching or selection (see <xref
target="choiceUsingExtlang"></xref>).

Also fixed on http://www.macchiato.com/unicode/ltru-canonicalization

Mark

On Wed, May 6, 2009 at 07:42, Mark Davis <[email-masked]> wrote:

> The other formulation we could use goes back a bit more towards the v21
> text with a single canonical form, yet still have a clearly defined extlang
> form for people who need it.  If you can use the web, it is a new third
> column on http://www.macchiato.com/unicode/ltru-canonicalization. The
> differences from the last formulation (B) I sent out are:
>
> *[Remove from B: "There are two canonical forms... ...considered
> beneficial in matching or selection."]* *[Replace **B: "4. **In the
> extlang canonical form (but not...
> ...Step 3." by the following]*** The canonical form has no extlang subtag.
> with the extlang's Prefix. This form may be useful in environments where
> the presence of the macrolanguage subtag is considered beneficial in
> matching or selection (see <xref target="choiceUsingExtlang"></xref>).
>
>    - For example, "hak-CN" (Hakka, China) has primary language subtag
>    'hak', which in turn has an 'extlang' record with a Prefix 'zh' (Chinese).
>    The extlang form would be "zh-hak-CN" (Chinese, Hakka, China).
>
>

---

From: Leif Halvard Silli <[email-masked]>
Date: Wed, 06 May 2009 23:33:15 +0200

Mark Davis 2009-05-06 16.42:
> On Tue, May 5, 2009 at 23:57, Randy Presuhn wrote:

>> I really really really really really dislike, loathe, and abhor the 
>> very idea of having two "canonical" forms - it seems oxymoronic. 
>> Yes, there are two use cases, but that does not mean that both have
>> to be called "canonical", or that the one true canonical form
>> (whichever that is) needs to serve both purposes.

> for people who need it. [...]

It seems better to just edit the text a little by dealing with 
'default': 'default' indicates that the "non-extlang" variant 
could need a name of its own that (just as "extlang") gives a hint 
about its nature - for example "plain" or "common". Whether this 
canonicalization also is the default one, can then be expressed 
independently of the name.

Additionally, the text could - in line with the title of the 
section - focus on process (canonicalization) rather than result 
(canonical form) - default type of canonicalization, rather than 
a (default) canonical form.

Thus Addison's two first paragraphs could then e.g. become:

<t>Since a particular language tag is sometimes used by many 
processes, language tags SHOULD always be created or generated in 
a canonicalized form.</t>

<t>Canonicalization is either of type 'plain' or of type 
'extlang'. 'Plain canonicalization' leads to a language tag with 
no extended language subtags, while 'extlang canonicalization' 
leads to a language tag with extended language subtags. 'Plain 
canonicalization' is the default option. But 'extlang 
canonicalization' can be useful in environments where the presence 
of the enclosing primary language subtag is considered beneficial 
to matching or selection (see Section 4.1.2)</t>
-- 
leif halvard silli

---

From: Mark Davis <[email-masked]>
Date: Mon, 11 May 2009 16:31:01 -0700

It appears to me that we have rough consensus on the text. I'd like to see
if we can move forward on this. That text (also in Column C of
http://www.macchiato.com/unicode/ltru-canonicalization) is below.

===

Since a particular language tag is sometimes used by many processes,
language tags SHOULD always be created and processed in a canonical form.

A language tag is in a canonical form, either default or extended, when the
tag is well-formed according the rules in <xref target="syntax"/> and <xref
target="sources"/> and it has been canonicalized by applying each of the
following steps in order, using data from the IANA registry (see <xref
target="ianaformat"/>):

1. Extension sequences are ordered into case-insensitive ASCII order by
singleton subtag.

   - That is, the subtag sequence '-a-babble' comes before '-b-warble'.

2. Redundant or grandfathered tags are replaced by their Preferred-Value, if
there is one.

   - These items are either deprecated mappings created before the adoption
   of this document (such as the mapping of "no-nyn" to "nn" or "i-klingon" to
   "tlh") or are the result of later registrations or additions to this
   document (for example, "zh-hakka" was deprecated in favor of the ISO 639-3
   code 'hak' when this document was adopted).
   - These field-body of the Preferred-Value for grandfathered and redundant
   tags is an "extended language range" (<xref target="RFC4647"></xref>) and
   might consist of more than one subtag.

3. Subtags are replaced by their Preferred-Value, if there is one. For
extlangs, the original primary language subtag is also replaced if there is
a primary language subtag in the Preferred-Value.

   - The field-body of the Preferred-Value for extlangs is an "extended
   language range" and typically maps to a primary language subtag. For
   example, the subtag sequence "zh-hak" (Chinese, Hakka) would be replaced
   with the tag "hak" (Hakka).
   - Most of the non-extlang subtags are either Region subtags where the
   country name or designation has changed or clerical corrections to ISO
   639-1.

The canonical form has no extlang subtag. There is an alternate 'extlang
form' that modifies the canonical form so that primary language subtags that
are also extlang subtags are prepended with the extlang's Prefix. This form
may be useful in environments where the presence of the Prefix subtag is
considered beneficial in matching or selection (see <xref
target="choiceUsingExtlang"></xref>).

---

From: Kent Karlsson <[email-masked]>
Date: Tue, 12 May 2009 09:14:10 +0200

A language tag is in a canonical form, either default or extended,
-->
A language tag is in 'canonical form', or in the alternative 'extlang form'
(see below), 

(since there is now only one "canonical form"; the extlang form is no longer
termed "canonical")

        /kent k

---

From: Mark Davis <[email-masked]>
Date: Tue, 12 May 2009 07:00:16 -0700

Whoops, missed that. Thanks,

Mark

On Tue, May 12, 2009 at 00:14, Kent Karlsson <[email-masked]>wrote:

> (since there is now only one "canonical form"; the extlang form is no
> longer termed "canonical")
>
>         /kent k
>

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 14 May 2009 20:15:17 -0700

Hi -

It's time to wrap this one up.  Editors, please post to
the mailing list your current understanding of the new
text to resolve this item.  (I see there have been some
minor adjustments discussed along the way.)  Everyone:
is this text something you can live with?

Perfection is antithetical to completion.

Randy

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 18 May 2009 20:07:03 +0900

[shepherd hat on]

It appears that this is the last full version of the proposed text.
Mark nodded at
http://www.ietf.org/mail-archive/web/ltru/current/msg12589.html
to a small fix from Kent at
http://www.ietf.org/mail-archive/web/ltru/current/msg12588.html

Is that a correct summary of the current state? There is some 
back-and-forth between Mark and CE Whitehead after that, but it's not 
clear what the result of that is.

When searching through 
http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21, I found the word 
"canonical" all over the place. I would suggest that some people do a 
careful read of these places, checking whether they fit together with 
the new text. In particular, if there is text elsewhere in the document 
that assumes that canonicalization may be without or with extlangs, then 
we would have to revise the text proposal below.

Regards,    Martin.

On 2009/05/12 8:31, Mark Davis wrote:
> tag is well-formed according the rules in<xref target="syntax"/>  and<xref
> following steps in order, using data from the IANA registry (see<xref
> target="ianaformat"/>):
>
> considered beneficial in matching or selection (see<xref

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Kent Karlsson <[email-masked]>
Date: Mon, 18 May 2009 14:03:54 +0200

the
> new text. In particular, if there is text elsewhere in the document 
that
> assumes that canonicalization may be without or with extlangs, then 
we would
> have to revise the text proposal below.

Ok, below is my walkthrough for instances of "canonical" in 4646bis-21.
I did not find any issue of the nature you suggest, but I found one that
should be an easy fix ("canonical" -> "preferred"), and one that may be
more major, but has an unclear fix, and I don't want to raise an issue.
There is also a possible omission in the definition of canonical form
w.r.t. extention subtags.

        /kent k

==========================================

   In order to avoid instability in the canonical form of tags, if a
   two-character code is added to ISO 639-1 for a language for which a
   three-character code was already included in either ISO 639-2 or ISO
   639-3, the two-character code MUST NOT be registered.  See
   Section 3.4.

----------------------------

This seems to be excessive, since we don't require absolute stability
of canonical forms. It also seems unnecessary, since apparently (see next
entry just below) ISO has 'promised' not to add any more two-letter
language codes.

Anyway, this text is not problematic w.r.t. the current edits
re. canonical form definition.

==================================================

   To avoid these problems with versioning and subtag choice (as
   experienced during the transition between RFC 1766 and RFC 3066), as
   well as to ensure the canonical nature of subtags defined by this
   document, the ISO 639 Registration Authority Joint Advisory Committee
   (ISO 639/RA-JAC) has included the following statement in
   [iso639.prin]:

    ...

----------------------------

See previous entry (just above).

=====================================================

   9.  In the event that more than one extension appears in a single
       tag, the tag SHOULD be canonicalized as described in Section 4.5,
       by ordering the various extension sequences into case-insensitive
       ASCII order.

----------------------------

This is about Extension Subtags (not to be confused with Extended
Language Subtags; I still find this naming needlessly confusing),
and there is no change to canonicalisation w.rt. that in section 4.5.

===============================================

      *  Preferred-Value's field body contains a canonical mapping from
         this record's value to a modern equivalent that is preferred in
         its place.  Depending on the value of the 'Type' field, this
         value can take different forms:

----------------------------

I see no problem with this text.

==============================================

   The field 'Preferred-Value' contains a mapping between the record in
   which it appears and another tag or subtag (depending on the record's
   'Type').  The value in this field is used for canonicalization (see
   Section 4.5).  In cases where the subtag or tag also has a
   'Deprecated' field, then the 'Preferred-Value' is RECOMMENDED as the
   best choice to represent the value of this record when selecting a
   language tag.

----------------------------

I see no problem with this text.

================================================

   4.  Extended language subtags always have a mapping to their
       identical primary language subtag.  For example, the extended
       language subtag 'yue' (Cantonese) can be used to form the tag
       "zh-yue".  It has a Preferred-Value mapping to the primary
       language subtag 'yue', meaning that a tag such as
       "zh-yue-Hant-HK" can be canonicalized to "yue-Hant-HK".

----------------------------

This refers to what is the latest (and single) version of "canonical",
so no problem with this text.

==========================================

   Occasionally the deprecated code is preferred in certain contexts.
   For example, both "iw" and "he" can be used in the Java programming
   language, but "he" is converted on input to "iw", which is thus the
   canonical form in Java.

----------------------------

This is a Java specific problem, not one with LTRU. Still, it would be
better to say "preferred" instead of "canonical" in this paragraph.

========================================

   o  The specification MUST specify a canonical representation.

----------------------------

This is a requirement on registrations of "Extension subtags".
One that is somehow NOT taken advantage of in the current definition
of canonicalisation. Forgotten point in the def. of canonicalisation?

=========================================

   Extension authors are strongly cautioned that many (including most
   well-formed) processors will be unaware of any special relationships
   or meaning inherent in the order of extension subtags.  Extension
   authors SHOULD avoid subtag relationships or canonicalization
   mechanisms that interfere with matching or with length restrictions
   that sometimes exist in common protocols where the extension is used.
   In particular, applications MAY truncate the subtags in doing
   matching or in fitting into limited lengths, so it is RECOMMENDED
   that the most significant information be in the most significant
   (left-most) subtags and that the specification gracefully handle
   truncated subtags.

----------------------------

I did not know that we classified *processors* (of any kind) as being
well-formed or not...

This text seems to refer to some other canonicalisation mechanisms
than that in section 4.5. And the current def. of canonicalisation
requires sorting the extention subtags. Maybe this refers to
canonicalisation *within* an extension subtag.

I think this text is problematical (in any case, quite regardless of
the changes to section 4.5), but I don't want to raise an issue on
this now.

==========================================

   In some cases, the encompassed languages had tags registered for them
   during the RFC 3066 era.  Those grandfathered tags not already
   deprecated or rendered redundant were deprecated in the registry upon
   adoption of this document.  As grandfathered values, they remain
   valid for use and some content or applications might use them.  As
   with other grandfathered tags, since implementations might not be
   able to associate the grandfathered tags with the encompassed
   language subtag equivalents that are recommended by this document,
   implementations are encouraged to canonicalize tags for comparison
   purposes.  Some examples of this include the tags "zh-hakka" (Hakka)
   and "zh-guoyu" (Mandarin or Standard Chinese).

----------------------------

I see no problem with this text.

=========================================

----------------------------

Heading for the section on canonicalisation, a section which naturally has
many instances of the word "canonical" (and inflections of that), which I
don't list here.

===========================================
===========================================

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 18 May 2009 10:22:27 -0700

Hi -

> From: "Martin J. Dürst" <[email-masked]>
> To: "Mark Davis" <[email-masked]>; "Alexey Melnikov" <[email-masked]>
> Cc: "Randy Presuhn" <[email-masked]>; "LTRU Working Group" <[email-masked]>
> Sent: Monday, May 18, 2009 4:07 AM
...
> clear what the result of that is.

I *think* so.  I'd *REALLY* like to see a post of whatever the current
proposed text is.

> we would have to revise the text proposal below.

A big THANK YOU to Kent for his post on that topic!

Randy

---

From: Kent Karlsson <[email-masked]>
Date: Mon, 18 May 2009 22:16:11 +0200

version 21 versus version 22-ed:
21:   Example: Although the tag "en-BU" (English as used in Burma)
21:   maintains its validity, the language tag "en-BU" is not canonical
21:   because the 'BU' subtag has a canonical mapping to 'MM' (Myanmar).

22:   Example: Although the tag "en-BU" (English as used in Burma)
22:   maintains its validity, the language tag "en-BU" is not in a
22:   canonical form because the 'BU' subtag has a canonical mapping to
22:   'MM' (Myanmar).

Since there is now only one form that is named "canonical":
"a canonical form" -> "canonical form" (so the -21 text was better).

I don't think we have any "canonical mappings", they are called
"Preferred-Value mappings".

    /kent k

---

From: Mark Davis <[email-masked]>
Date: Mon, 18 May 2009 14:18:21 -0700

Addison's text in
http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html was drawn
from mine, with the one change covered by Kent. And that text looks pretty
good.

There is one difference: I didn't have #4 as a numbered item. Making that
paragraph a step 4, muddies the distinction in the definitions of the two
entities. I think that #4 should not be numbered. Removing the number makes
it much clearer
(a) that the canonical form is defined by well-formedness + Steps 1, 2, and
3 alone, and
(b) exactly how the alternative form is generated from the canonical form
(what was step 4).

That would be done by the following two textual changes in Addison's text of
http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html

=====

A. A language tag is in 'canonical form' (or in the alternative 'extlang
form', see below), when...
=>
A language tag is in 'canonical form' when...

=====

B. "De"number #4. That it, it becomes a regular paragraph after #3. The
change would be:

4. The canonical form contains no 'extlang' subtags. There is an alternate
'extlang form' that modifies the canonical form...

   - ...
   - Note that this step can restore a subtag that was removed by the
   previous step.

=> (no #4, outdented)

The canonical form contains no 'extlang' subtags. There is an alternate
'extlang form' that modifies the canonical form ...

   - ...
   - Note that this step can restore a subtag that was removed by the
   previous step.

=====

Mark

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 18 May 2009 14:39:41 -0700

Hi -

Both as co-chair and as a technical contributor...

> From: "Mark Davis" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Monday, May 18, 2009 2:18 PM
...
...

I'm still waiting for someone to post the *exact* text being proposed
to this mailing list, rather than yet another bunch of little deltas
and wouldn't-it-be-nice-ifs.  Until that is done, I can't agree to
support the changes because I really still don't know what they are.

Randy

---

From: Mark Davis <[email-masked]>
Date: Mon, 18 May 2009 14:42:41 -0700

I did that. I referred very explicitly to Addison's text of
http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html , which is
the latest working copy of the text.

Mark

---

From: Mark Davis <[email-masked]>
Date: Mon, 18 May 2009 14:44:50 -0700

Here is the complete amended text, copied from
http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html with the
two suggested changes.

Since a particular language tag is sometimes used by many processes,
language tags SHOULD always be created or generated in a canonical form.

A language tag is in 'canonical form', when the tag is well-formed according
the rules in Section 2.1
(Syntax)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#syntax>and
Section 2.2 (Language Subtag Sources and
Interpretation)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#sources>and
it has been canonicalized by applying each of the following steps in
order, using data from the IANA registry (see Section 3.1 (Format of the
IANA Language Subtag
Registry)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#ianaformat>
):

   1. Extension sequences are ordered into case-insensitive ASCII order by
   singleton subtag.
      - That is, the subtag sequence '-a-babble' comes before '-b-warble'.
   2. Redundant or grandfathered tags are replaced by their Preferred-Value,
   if there is one.
      - These items are either deprecated mappings created before the
      adoption of this document (such as the mapping of "no-nyn" to "nn" or
      "i-klingon" to "tlh") or are the result of later registrations
or additions
      to this document (for example, "zh-hakka" was deprecated in
favor of the ISO
      639-3 code 'hak' when this document was adopted).
      - These field-body of the Preferred-Value for grandfathered and
      redundant tags is an "extended language range" ([RFC4647] (Phillips,
      A. and M. Davis, “Matching of Language Tags,” September
2006.)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#RFC4647>)
      and might consist of more than one subtag.
   3. Subtags are replaced by their Preferred-Value, if there is one. For
   extlangs, the original primary language subtag is also replaced if there is
   a primary language subtag in the Preferred-Value.
      - The field-body of the Preferred-Value for extlangs is an "extended
      language range" and typically maps to a primary language subtag. For
      example, the subtag sequence "zh-hak" (Chinese, Hakka) would be replaced
      with the tag "hak" (Hakka).
      - Most of the non-extlang subtags are either Region subtags where the
      country name or designation has changed or clerical corrections to ISO
      639-1.

The canonical form contains no 'extlang' subtags. There is an alternate
'extlang form' that modifies the canonical form so that primary language
subtags that are also extlang subtags are prepended with the extlang's
Prefix. This form may be useful in environments where the presence of the
Prefix subtag is considered beneficial in matching or selection (see
Section 4.1.2 (Using Extended Language
Subtags)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#choiceUsingExtlang>).

Mark

On Mon, May 18, 2009 at 14:42, Mark Davis <[email-masked]> wrote:

> http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html , which
> is the latest working copy of the text.
>
> Mark
>
>
>
> On Mon, May 18, 2009 at 14:39, Randy Presuhn <[email-masked]
> > wrote:
>
>> Hi -
>>

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 18 May 2009 14:53:18 -0700

Hi -

Thank you Mark, for getting a complete proposal onto the mailing list.
Everyone: The time for discussion is drawing to a close.
Can you live with this text?

Randy

----- Original Message ----- 
From: "Mark Davis" <[email-masked]>
To: "Randy Presuhn" <[email-masked]>
Cc: "LTRU Working Group" <[email-masked]>
Sent: Monday, May 18, 2009 2:44 PM
Subject: Re: [Ltru] Ticket #45: updated editor's copy available

Mark

---

From: Kent Karlsson <[email-masked]>
Date: Mon, 18 May 2009 23:58:19 +0200

> Hi -

Thank you Mark, for getting a complete proposal onto the mailing
> list.
Can you live
> with this text?

Randy

----- Original Message ----- 
From: "Mark Davis"

> language tags SHOULD always be created or generated in a canonical form.

Again, there is now only one canonical form, so
"a canonical form" -> "canonical form"

(Though I may guess that there are still those that would prefer
generating/creating tags in extlang form. Not me though.)

    /kent k

---

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 18 May 2009 15:27:34 -0700

I don’t see the changes. Can you break them out?

Addison

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Monday, May 18, 2009 2:45 PM
To: Randy Presuhn
Cc: LTRU Working Group
Subject: Re: [Ltru] Ticket #45: updated editor's copy available

Here is the complete amended text, copied from http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html with the two suggested changes.
4.5.  Canonicalization of Language Tags

Since a particular language tag is sometimes used by many processes, language tags SHOULD always be created or generated in a canonical form.

A language tag is in 'canonical form', when the tag is well-formed according the rules in Section 2.1 (Syntax)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#syntax> and Section 2.2 (Language Subtag Sources and Interpretation)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#sources> and it has been canonicalized by applying each of the following steps in order, using data from the IANA registry (see Section 3.1 (Format of the IANA Language Subtag Registry)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#ianaformat>):

 1.  Extension sequences are ordered into case-insensitive ASCII order by singleton subtag.

    *   That is, the subtag sequence '-a-babble' comes before '-b-warble'.

 1.  Redundant or grandfathered tags are replaced by their Preferred-Value, if there is one.

    *   These items are either deprecated mappings created before the adoption of this document (such as the mapping of "no-nyn" to "nn" or "i-klingon" to "tlh") or are the result of later registrations or additions to this document (for example, "zh-hakka" was deprecated in favor of the ISO 639-3 code 'hak' when this document was adopted).
    *   These field-body of the Preferred-Value for grandfathered and redundant tags is an "extended language range" ([RFC4647] (Phillips, A. and M. Davis, “Matching of Language Tags,” September 2006.)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#RFC4647>) and might consist of more than one subtag.

 1.  Subtags are replaced by their Preferred-Value, if there is one. For extlangs, the original primary language subtag is also replaced if there is a primary language subtag in the Preferred-Value.

    *   The field-body of the Preferred-Value for extlangs is an "extended language range" and typically maps to a primary language subtag. For example, the subtag sequence "zh-hak" (Chinese, Hakka) would be replaced with the tag "hak" (Hakka).
    *   Most of the non-extlang subtags are either Region subtags where the country name or designation has changed or clerical corrections to ISO 639-1.
The canonical form contains no 'extlang' subtags. There is an alternate 'extlang form' that modifies the canonical form so that primary language subtags that are also extlang subtags are prepended with the extlang's Prefix. This form may be useful in environments where the presence of the Prefix subtag is considered beneficial in matching or selection (see Section 4.1.2 (Using Extended Language Subtags)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#choiceUsingExtlang>).

 *   For example, "hak-CN" (Hakka, China) has primary language subtag 'hak', which in turn has an 'extlang' record with a Prefix 'zh' (Chinese). The extlang form would be "zh-hak-CN" (Chinese, Hakka, China).
 *   Note that this step can restore a subtag that was removed by the previous step.

Mark

On Mon, May 18, 2009 at 14:42, Mark Davis <[email-masked]<mailto:[email-masked]>> wrote:
I did that. I referred very explicitly to Addison's text of http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html , which is the latest working copy of the text.

Mark

On Mon, May 18, 2009 at 14:39, Randy Presuhn <[email-masked]<mailto:[email-masked]>> wrote:
Hi -

> From: "Mark Davis" <[email-masked]<mailto:[email-masked]>>
> To: "Randy Presuhn" <[email-masked]<mailto:[email-masked]>>
> Cc: "LTRU Working Group" <[email-masked]<mailto:[email-masked]>>
...
...

Randy

---

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 18 May 2009 15:37:31 -0700

A. A language tag is in 'canonical form' (or in the alternative 'extlang form', see below), when...
=>
A language tag is in 'canonical form' when...

Addison> DONE.

=====

B. "De"number #4. That it, it becomes a regular paragraph after #3. The change would be:
4. The canonical form contains no 'extlang' subtags. There is an alternate 'extlang form' that modifies the canonical form...
• ...
• Note that this step can restore a subtag that was removed by the previous step. 
=> (no #4, outdented)

Addison> DONE.

Addison

Addison Phillips

---

From: Mark Davis <[email-masked]>
Date: Mon, 18 May 2009 15:39:29 -0700

Looks like you got them from the followup message. Thanks,

Mark

On Mon, May 18, 2009 at 15:27, Phillips, Addison <[email-masked]> wrote:

> *From:* [email-masked] [mailto:[email-masked]] *On Behalf Of
> *Mark Davis
> *Sent:* Monday, May 18, 2009 2:45 PM
> *To:* Randy Presuhn
> *Cc:* LTRU Working Group
>
> A language tag is in 'canonical form', when the tag is well-formed
> according the rules in Section 2.1 (Syntax)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#syntax>and Section 2.2
> (Language Subtag Sources and Interpretation)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#sources>and it has been canonicalized by applying each of the following steps in
> IANA Language Subtag Registry)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#ianaformat>
> ):
>
>       "i-klingon" to "tlh") or are the result of later registrations or additions
>       to this document (for example, "zh-hakka" was deprecated in favor of the ISO
>       A. and M. Davis, “Matching of Language Tags,” September 2006.)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#RFC4647>)
>    1. Subtags are replaced by their Preferred-Value, if there is one. For
>       - Most of the non-extlang subtags are either Region subtags where
>       the country name or designation has changed or clerical corrections to ISO
>       639-1.
>
> Prefix subtag is considered beneficial in matching or selection (see Section 4.1.2
> (Using Extended Language Subtags)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#choiceUsingExtlang>).
>
>
>  On Mon, May 18, 2009 at 14:39, Randy Presuhn <
> [email-masked]> wrote:
>
> Hi -
>

---

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 18 May 2009 15:40:11 -0700

Yep. Thanks.

Addison

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Monday, May 18, 2009 3:39 PM
To: Phillips, Addison
Cc: Randy Presuhn; LTRU Working Group
Subject: Re: [Ltru] Ticket #45: updated editor's copy available

Mark

On Mon, May 18, 2009 at 15:27, Phillips, Addison <[email-masked]<mailto:[email-masked]>> wrote:

Addison

Addison Phillips

It is an architecture.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Mark Davis
Sent: Monday, May 18, 2009 2:45 PM
To: Randy Presuhn
Cc: LTRU Working Group

Mark

Mark

Hi -

...

...

Randy

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 19 May 2009 10:18:20 +0900

[as a technical contributor]

Here are my comments. In general, I agree with the wording, except where 
indicated below, and modulo the way cross-references are displayed).

> Registry)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#ianaformat>

Editorial: I'd prefer to avoid double parentheses. Just change to "(see 
Section 3.1, Format of the IANA Language Subtag Registry):"

>        - That is, the subtag sequence '-a-babble' comes before '-b-warble'.

Please replace "That is" by "for example". "That is" is usually used, 
and is used in the rest of our draft, to give equivalent explanations, 
not examples.

>        639-3 code 'hak' when this document was adopted).

I'd prefer to replace "These items" with "Redundant or grandfathered 
tags". But even then, these are not deprecated mappings. So something 
here needs fixing.

>        and might consist of more than one subtag.

"These field-body" doesn't make sense at all to me. Please fix.

>        with the tag "hak" (Hakka).

I suggest replacing "would" by "is".

> Subtags)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#choiceUsingExtlang>).

It seems the text here defines 'extlang form' just in passing, it's not 
even clear whether this is a definition or not. I'd strongly prefer a 
slightly different wording, e.g.:

The canonical form contains no 'extlang' subtags. There is an alternate
'extlang form' that maintains or reinstates extlang subtags. This form 
MAY be used in environments where the presence of the Prefix subtag is 
considered beneficial in matching or selection (see Section 4.1.2 (Using 
Extended Language Subtags).

A language tag is in 'extlang form' if it has been processed by the 
following procedure, or any alternative procedure that leads to the same 
result.

      1. The language tag is brought into canonical form.

      2. If the language tag starts with a primary language subtag that
      is also an extlang subtag, then the language tag is prepended with
      the extlang's Prefix.

      - For example, "hak-CN" (Hakka, China) has primary language subtag 
'hak',
      which in turn has an 'extlang' record with a Prefix 'zh' 
(Chinese). The
      extlang form is "zh-hak-CN" (Chinese, Hakka, China).
      - Note that this step can restore a subtag that was removed by the
      previous step.

Regards,   Martin.

>
>
> Mark
>
>
> On Mon, May 18, 2009 at 14:42, Mark Davis<[email-masked]>  wrote:
>
>> On Mon, May 18, 2009 at 14:39, Randy Presuhn<[email-masked]
>>> wrote:
>>> Hi -
>>>
>>>> From: "Mark Davis"<[email-masked]>
>>>> To: "Randy Presuhn"<[email-masked]>
>>>> Cc: "LTRU Working Group"<[email-masked]>
> ------------------------------------------------------------------------
>

---

From: Mark Davis <[email-masked]>
Date: Mon, 18 May 2009 18:38:50 -0700

Mark

On Mon, May 18, 2009 at 18:18, "Martin J. Dürst" <[email-masked]>wrote:

>> according
>> (Syntax)<
>> http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#syntax
>> >and
>> Interpretation)<
>> http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#sources
>> >and
>> Registry)<
>> http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#ianaformat
>> >
>> ):
>>
>

fine by me

> Please replace "That is" by "for example". "That is" is usually used, and
> is used in the rest of our draft, to give equivalent explanations, not
> examples.

fine by me

> I'd prefer to replace "These items" with "Redundant or grandfathered tags".
> But even then, these are not deprecated mappings. So something here needs
> fixing.

I suggest:

These items are either deprecated mappings created...
=>
The mappings to Preferred-Value are mappings of deprecated or redundant
tags, some of which predate this document such as the mapping of "no-nyn" to
"nn" or "i-klingon" to "tlh", and some of which are the result of additions
in this document, such as the mapping of "zh-hakka" to the ISO 639-3 code
'hak'.

>> 2006.)<
>> http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#RFC4647
>> >)

should be "The" instead of "These"

> I suggest replacing "would" by "is".

fine by me.

>> Subtags)<
>> http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#choiceUsingExtlang
>> >).
>>
>
> 'extlang form' that maintains or reinstates extlang subtags. This form MAY
> be used in environments where the presence of the Prefix subtag is

I'm happy with this direction, so that we are more formal about the
definition, but have some wording fixes.

> following procedure, or any alternative procedure that leads to the same

=>
A language tag is in 'extlang form', when the tag is well-formed according
the rules in Section 2.1
(Syntax)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#syntax>and
Section 2.2 (Language Subtag Sources and
Interpretation)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#sources>and
it has been processed by applying each of the following two steps in
order, using data from the IANA registry (see Section 3.1 (Format of the
IANA Language Subtag

>     1. The language tag is brought into canonical form.

=> The language tag is first transformed into canonical form, as described
above.

>     - Note that this step can restore a subtag that was removed by the
=> Note that Step 2 (prepending a prefix) can restore a subtag that was
removed by Step 1 (canonicalizing).

>>>>> That would be done by the following two textual changes in Addison's
>>>>>
>>>> text of
>>>> ...
>>>>

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 18 May 2009 19:36:20 -0700

Hi -

> To: "Martin J. Dürst" <[email-masked]>
> Sent: Monday, May 18, 2009 6:38 PM
...

Though the number of comments on #45 is disconcerting, there seems to
be the potential for rapid progress and closure here.  To that end,
I suggest that revised proposed text, incorporating the suggestions
on which there seems to be agreement, be posted to the WG mailing
list for the archives.

Randy

---

From: Mark Davis <[email-masked]>
Date: Mon, 18 May 2009 20:38:27 -0700

Ok. Here is complete revised text, taking Martin's comments into account
(modified slightly according to my comments).

====

A language tag is in 'canonical form', when the tag is well-formed according
the rules in Section 2.1
(Syntax)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#syntax>and
Section 2.2 (Language Subtag Sources and
Interpretation)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#sources>and
it has been canonicalized by applying each of the following steps in
order, using data from the IANA registry (see Section 3.1, Format of the
IANA Language Subtag
Registry<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#ianaformat>
):

   1. Extension sequences are ordered into case-insensitive ASCII order by
   singleton subtag.
      - For example, the subtag sequence '-a-babble' comes before
      '-b-warble'.
   2. Redundant or grandfathered tags are replaced by their Preferred-Value,
   if there is one.
   - The mappings to Preferred-Value are mappings of deprecated or redundant
      tags, some of which predate this document such as the mapping of
"no-nyn" to
      "nn" or "i-klingon" to "tlh", and some of which are the result
of additions
      in this document, such as the mapping of "zh-hakka" to the ISO 639-3 code
      'hak'.
      - The field-body of the Preferred-Value for grandfathered and
      redundant tags is an "extended language range" ([RFC4647] (Phillips,
      A. and M. Davis, “Matching of Language Tags,” September
2006.)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#RFC4647>)
      and might consist of more than one subtag.
   3. Subtags are replaced by their Preferred-Value, if there is one. For
   extlangs, the original primary language subtag is also replaced if there is
   a primary language subtag in the Preferred-Value.
      - The field-body of the Preferred-Value for extlangs is an "extended
      language range" and typically maps to a primary language subtag. For
      example, the subtag sequence "zh-hak" (Chinese, Hakka) is
replaced with the
      tag "hak" (Hakka).
      - Most of the non-extlang subtags are either Region subtags where the
      country name or designation has changed or clerical corrections to ISO
      639-1.

A language tag is in 'extlang form', when the tag is well-formed according
the rules in Section 2.1
(Syntax)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#syntax>and
Section 2.2 (Language Subtag Sources and
Interpretation)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#sources>and
it has been processed by applying each of the following two steps in
order, using data from the IANA registry (see Section 3.1, Format of the
IANA Language Subtag
Registry<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#ianaformat>):

   1. The language tag is first transformed into canonical form, as
   described above.
   2. If the language tag starts with a primary language subtag that is also
   an extlang subtag, then the language tag is prepended with the extlang's
   Prefix.
      - For example, "hak-CN" (Hakka, China) has primary language subtag
      'hak', which in turn has an 'extlang' record with a Prefix 'zh'
(Chinese).
      The extlang form would be "zh-hak-CN" (Chinese, Hakka, China).
      - Note that Step 2 (prepending a prefix) can restore a subtag that was
      removed by Step 1 (canonicalizing).

Mark

---

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 18 May 2009 21:31:40 -0700

Martin wrote:
> Editorial: I'd prefer to avoid double parentheses. Just change to
> "(see Section 3.1, Format of the IANA Language Subtag Registry):"

Note that this is an artifact of copying the HTML, not what actually appears in the document. In the text version, only the section number shows. I.e.:

--
   Section 2.1 and Section 2.2 and it has been canonicalized by applying
   each of the following steps in order, using data from the IANA
   registry (see Section 3.1):
--
> Please replace "That is" by "for example". "That is" is usually
> used, and is used in the rest of our draft, to give equivalent
> explanations, not examples.

Okay.

> I'd prefer to replace "These items" with "Redundant or
> grandfathered tags". But even then, these are not deprecated mappings. So
> something here needs fixing.

I would suggest:

--
Many of these are deprecated mappings...
--

> "These field-body" doesn't make sense at all to me. Please fix.

This is the typo I referred to in a previous email. Fixed already.

> I suggest replacing "would" by "is".

Okay.

> It seems the text here defines 'extlang form' just in passing, it's
> not even clear whether this is a definition or not. I'd strongly prefer
> a slightly different wording, e.g.:

I agree with your position, but wasn't the idea that the 'extlang form' wasn't "the" canonical form?

Addison

Addison Phillips

---

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 18 May 2009 21:50:29 -0700

Ur, uh… let me post it from the actual editor’s copy please. I did some additional editing as part of incorporating all these changes. Also, it is useful to see the entire section.

Here is what I have, after my edits, the email thread with Martin, Doug’s note, and the below:

   Since a particular language tag can be used by many processes,
   language tags SHOULD always be created or generated in canonical
   form.

   A language tag is in 'canonical form' (or in the alternative 'extlang
   form', see below), when the tag is well-formed according the rules in
   Section 2.1 and Section 2.2 and it has been canonicalized by applying
   each of the following steps in order, using data from the IANA
   registry (see Section 3.1):

       *  For example, the subtag sequence '-a-babble' comes before
          '-b-warble'.

       *  The field-body of the Preferred-Value for grandfathered and
          redundant tags is an "extended language range" ([RFC4647]) and
          might consist of more than one subtag.

       *  Many of these Preferred-Value mappings are either deprecated tags
          created before the adoption of this document (such as the
          mapping of "no-nyn" to "nn" or "i-klingon" to "tlh") or are
          the result of later registrations or additions to this
          document (for example, "zh-hakka" was deprecated in favor of
          the ISO 639-3 code 'hak' when this document was adopted).

   3.  Subtags are replaced by their Preferred-Value, if there is one.
       For extlangs, the original primary language subtag is also
       replaced if there is a primary language subtag in the Preferred-
       Value.

       *  The field-body of the Preferred-Value for extlangs is an
          "extended language range" and typically maps to a primary
          language subtag.  For example, the subtag sequence "zh-hak"
          (Chinese, Hakka) is replaced with the subtag 'hak' (Hakka).

       *  Most of the non-extlang subtags are either Region subtags
          where the country name or designation has changed or clerical
          corrections to ISO 639-1.

   The canonical form contains no 'extlang' subtags.  There is an
   alternate 'extlang form' that maintains or reinstates extlang
   subtags.  This form can be useful in environments where the presence
   of the Prefix subtag is considered beneficial in matching or
   selection (see Section 4.1.2).

   A language tag is in 'extlang form', when the tag is well-formed
   according the rules in Section 2.1 and Section 2.2 and it has been
   processed by applying each of the following two steps in order, using
   data from the IANA registry:

   2.  If the language tag starts with a primary language subtag that is
       also an extlang subtag, then the language tag is prepended with
       the extlang's Prefix.

       *   For example, "hak-CN" (Hakka, China) has primary language
           subtag 'hak', which in turn has an 'extlang' record with a
           Prefix 'zh' (Chinese).  The extlang form would be "zh-hak-CN"
           (Chinese, Hakka, China).

       *   Note that Step 2 (prepending a prefix) can restore a subtag
           that was removed by Step 1 (canonicalizing).

--

I’d also be in favor of losing the two “Example” paragraphs as redundant (or incorporating them into the canonicalization rules). They detract from the overall flow, are confusing, and are already covered by other examples in this final text.

Addison

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Monday, May 18, 2009 8:38 PM
To: Randy Presuhn
Cc: LTRU Working Group
Subject: Re: [Ltru] Ticket #45: updated editor's copy available

Ok. Here is complete revised text, taking Martin's comments into account (modified slightly according to my comments).

====

A language tag is in 'canonical form', when the tag is well-formed according the rules in Section 2.1 (Syntax)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#syntax> and Section 2.2 (Language Subtag Sources and Interpretation)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#sources> and it has been canonicalized by applying each of the following steps in order, using data from the IANA registry (see Section 3.1, Format of the IANA Language Subtag Registry<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#ianaformat>):

    *   For example, the subtag sequence '-a-babble' comes before '-b-warble'.

    *   The mappings to Preferred-Value are mappings of deprecated or redundant tags, some of which predate this document such as the mapping of "no-nyn" to "nn" or "i-klingon" to "tlh", and some of which are the result of additions in this document, such as the mapping of "zh-hakka" to the ISO 639-3 code 'hak'.

    *   The field-body of the Preferred-Value for grandfathered and redundant tags is an "extended language range" ([RFC4647] (Phillips, A. and M. Davis, “Matching of Language Tags,” September 2006.)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#RFC4647>) and might consist of more than one subtag.

    *   The field-body of the Preferred-Value for extlangs is an "extended language range" and typically maps to a primary language subtag. For example, the subtag sequence "zh-hak" (Chinese, Hakka) is replaced with the tag "hak" (Hakka).
    *   Most of the non-extlang subtags are either Region subtags where the country name or designation has changed or clerical corrections to ISO 639-1.
The canonical form contains no 'extlang' subtags. There is an alternate 'extlang form' that maintains or reinstates extlang subtags. This form MAY be used in environments where the presence of the Prefix subtag is considered beneficial in matching or selection (see Section 4.1.2 (Using Extended Language Subtags).

A language tag is in 'extlang form', when the tag is well-formed according the rules in Section 2.1 (Syntax)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#syntax> and Section 2.2 (Language Subtag Sources and Interpretation)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#sources> and it has been processed by applying each of the following two steps in order, using data from the IANA registry (see Section 3.1, Format of the IANA Language Subtag Registry<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#ianaformat>):

 1.  The language tag is first transformed into canonical form, as described above.
 2.  If the language tag starts with a primary language subtag that is also an extlang subtag, then the language tag is prepended with the extlang's Prefix.

    *   For example, "hak-CN" (Hakka, China) has primary language subtag 'hak', which in turn has an 'extlang' record with a Prefix 'zh' (Chinese). The extlang form would be "zh-hak-CN" (Chinese, Hakka, China).
    *   Note that Step 2 (prepending a prefix) can restore a subtag that was removed by Step 1 (canonicalizing).
Mark

On Mon, May 18, 2009 at 19:36, Randy Presuhn <[email-masked]<mailto:[email-masked]>> wrote:
Hi -

> To: "Martin J. Dürst" <[email-masked]<mailto:[email-masked]>>
> Cc: "Randy Presuhn" <[email-masked]<mailto:[email-masked]>>; "LTRU Working Group" <[email-masked]<mailto:[email-masked]>>
...

Randy

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 19 May 2009 15:37:35 +0900

Hello everybody,

> I agree with your position, but wasn't the idea that the 'extlang form' wasn't "the" canonical form?

I'm fine if it's not the canonical form. But I think we either define it 
clearly, or not (and I'd definitely prefer the former). In the text I 
commented on, it was just mentioned, leaving the reader wondering where 
this was actually defined anywhere.

Regards,  Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 19 May 2009 16:00:56 +0900

[with shepherd hat on]

Addison and Mark, please publish -22 at your earliest convenience.
I share Randy's assessment that we might be very close to closure on
#45, but then again we might not be. Don't wait with publishing just
because we might be done tomorrow (integers are cheap).

[as a technical contributor (or should I say textual contributor?)]

Some nits below.

> Here is what I have, after my edits, the email thread with Martin, Doug’s note, and the below:

according the -> according to the

, when -> when

(all due to Doug)

>     according the rules in Section 2.1 and Section 2.2 and it has been

according the -> according to the

>         *   For example, "hak-CN" (Hakka, China) has primary language

has primary language -> has a primary language

>             Prefix 'zh' (Chinese).  The extlang form would be "zh-hak-CN"

would be -> is

>     canonical form, while "en-b-ccc-bbb-a-aaa-X-xyz" is well-formed and

in a canonical form -> in canonical form

>     canonical form because the 'BU' subtag has a canonical mapping to

in a canonical form -> in canonical form

> I’d also be in favor of losing the two “Example” paragraphs as redundant
 > (or incorporating them into the canonicalization rules).
 > They detract from the overall flow, are confusing, and are
 > already covered by other examples in this final text.

Would be fine by me either way. [as a technical contributor]
With the fixes as given above. [as a textual contributor]

Better not start another discussion, the examples above don't hurt, or 
do they. Let's converge quickly [shepherd hat on]

Regards,   Martin.

---

From: Mark Davis <[email-masked]>
Date: Tue, 19 May 2009 00:07:39 -0700

Other than the items below, works for me.

Mark

On Mon, May 18, 2009 at 21:50, Phillips, Addison <[email-masked]> wrote:

>  Ur, uh… let me post it from the actual editor’s copy please. I did some
> additional editing as part of incorporating all these changes. Also, it is
> useful to see the entire section.
>
>
>
Remove the parenthetical, since Steps 1-3 *don't* define the extlang form;
the extlang form is defined separately. I've been trying for a while to get
you to fix this ;-)

according => according to

>        *  Many of these Preferred-Value mappings are either deprecatedtags
>
A mapping isn't a tag. You could add "for" after "either" to fix that.

Grammatically you need to structure this sentence as one of the following. I
don't care which.

... mappings either are ... or are ...

or

... mappings are either ... or ...

> (or incorporating them into the canonicalization rules). They detract from
> the overall flow, are confusing, and are already covered by other examples
> in this final text.
>

The examples do cover cases not otherwise covered. To help with the flow,
how about as a list?

Examples:
    1. ...
    2. ...

> *Sent:* Monday, May 18, 2009 8:38 PM
> *To:* Randy Presuhn
> *Cc:* LTRU Working Group
> IANA Language Subtag Registry<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#ianaformat>
> ):
>
>     - The mappings to Preferred-Value are mappings of deprecated or
>       redundant tags, some of which predate this document such as the mapping of
>       "no-nyn" to "nn" or "i-klingon" to "tlh", and some of which are the result
>       of additions in this document, such as the mapping of "zh-hakka" to the ISO
>       639-3 code 'hak'.
>
>
>       example, the subtag sequence "zh-hak" (Chinese, Hakka) is replaced with the
>       tag "hak" (Hakka).
> the rules in Section 2.1 (Syntax)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#syntax>and Section 2.2
> (Language Subtag Sources and Interpretation)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#sources>and it has been processed by applying each of the following two steps in
> IANA Language Subtag Registry<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22-ed.html#ianaformat>):
>
>
>    also an extlang subtag, then the language tag is prepended with the
>    extlang's Prefix.
>
>
>       - Note that Step 2 (prepending a prefix) can restore a subtag that
>       was removed by Step 1 (canonicalizing).
>
> Mark
>
>  On Mon, May 18, 2009 at 19:36, Randy Presuhn <

---

From: Mark Davis <[email-masked]>
Date: Tue, 19 May 2009 00:08:55 -0700

Mark

> according the -> according to the

Agreed.

> according the -> according to the

Agreed.

> has primary language -> has a primary language

Agreed.

>>            Prefix 'zh' (Chinese).  The extlang form would be "zh-hak-CN"

Agreed

> in a canonical form -> in canonical form

Agreed.

> in a canonical form -> in canonical form

Agreed

> Better not start another discussion, the examples above don't hurt, or do
> they. Let's converge quickly [shepherd hat on]
>
> Regards,   Martin.
>
>

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 19 May 2009 06:55:13 -0700

> according the -> according to the

DONE

> , when -> when

DONE
> according the -> according to the

DONE
> >
> >         *   For example, "hak-CN" (Hakka, China) has primary
> language subtag
> 

DONE, except I made it say "has the primary language subtag"

> >             Prefix 'zh' (Chinese).  The extlang form would be

DONE

> in a canonical form -> in canonical form

DONE

> in a canonical form -> in canonical form

DONE

> do they. Let's converge quickly [shepherd hat on]

The problem with the examples is that they are already covered by the new text and interrupt the flow of the text. They make the document longer without really adding anything. However, I didn't and won't remove them now in favor of finishing.

Addison

Addison Phillips

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 19 May 2009 07:27:56 -0700

A language tag is in 'canonical form' (or in the alternative 'extlang
   form', see below), when the tag is well-formed according the rules in
Remove the parenthetical, since Steps 1-3 *don't* define the extlang form; the extlang form is defined separately. I've been trying for a while to get you to fix this ;-)

AP> DONE

AP> DONE in response to Martin's note.

Grammatically you need to structure this sentence as one of the following. I don't care which.
... mappings either are ... or are ...

or

AP> Good points both. I did a more extensive edit to fix this. In the new document, it isn't "many" but rather "all" of the redundant/grandfathered mappings that fit the two categories provided: there are no other kinds of P-V values! To convey all this I put:

--
Preferred-Value fields in the registry provide mappings from deprecated tags to modern equivalents. Many of these were created before the adoption of this document (such as the mapping of "no-nyn" to "nn" or "i-klingon" to "tlh"). Others are the result of later registrations or additions to the registry as permitted or required by this document (for example, "zh-hakka" was deprecated in favor of the ISO 639-3 code 'hak' when this document was adopted).
--

--

The examples do cover cases not otherwise covered. 

AP> The first one illustrates extension ordering. The second one illustrates preferred-value mapping. These are covered by the text now in a way that wasn't as apparent before. But I digress, since I already said I wasn't going to muck with them anymore.

Addison

---

From: Mark Davis <[email-masked]>
Date: Tue, 19 May 2009 10:43:03 -0700

Very good.

Now that I see the result, I think you are right about moving the examples.
I'd suggest the following locations (yellow for those who can see it), and a
slight shortening.

Since a particular language tag can be used by many processes, language tags
SHOULD always be created or generated in canonical form.

A language tag is in 'canonical form' when the tag is well-formed according
to the rules in Section 2.1
(Syntax)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22.html#syntax>and
Section 2.2 (Language Subtag Sources and
Interpretation)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22.html#sources>and
it has been canonicalized by applying each of the following steps in
order, using data from the IANA registry (see Section 3.1 (Format of the
IANA Language Subtag
Registry)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22.html#ianaformat>
):

   1. Extension sequences are ordered into case-insensitive ASCII order by
   singleton subtag.
      - For example, the subtag sequence '-a-babble' comes before
      '-b-warble'.
      - For example, the language tag "en-b-ccc-bbb-a-aaa-X-xyz" is
      well-formed and potentially valid (extensions 'a' and 'b' are
not defined as
      of the publication of this document) but not in a canonical form
because the
      extensions are not in alphabetical order; the canonical form would be
      "en-a-aaa-b-ccc-bbb-X-xyz".
       2. Redundant or grandfathered tags are replaced by their
   Preferred-Value, if there is one.
      - The field-body of the Preferred-Value for grandfathered and
      redundant tags is an "extended language range" ([RFC4647] (Phillips,
      A. and M. Davis, “Matching of Language Tags,” September
2006.)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22.html#RFC4647>)
      and might consist of more than one subtag.
      - Preferred-Value fields in the registry provide mappings from
      deprecated tags to modern equivalents. Many of these were
created before the
      adoption of this document (such as the mapping of "no-nyn" to "nn" or
      "i-klingon" to "tlh"). Others are the result of later registrations or
      additions to the registry as permitted or required by this document (for
      example, "zh-hakka" was deprecated in favor of the ISO 639-3
code 'hak' when
      this document was adopted).
   3. Subtags are replaced by their Preferred-Value, if there is one. For
   extlangs, the original primary language subtag is also replaced if there is
   a primary language subtag in the Preferred-Value.
      - For example, although the tag "en-BU" (English as used in Burma) is
      valid, it is not in canonical form because the 'BU' subtag has a
canonical
      mapping to 'MM' (Myanmar).
      - The field-body of the Preferred-Value for extlangs is an "extended
      language range" and typically maps to a primary language subtag. For
      example, the subtag sequence "zh-hak" (Chinese, Hakka) is
replaced with the
      subtag 'hak' (Hakka).
      - Most of the non-extlang subtags are either Region subtags where the
      country name or designation has changed or clerical corrections to ISO
      639-1.

Mark

On Tue, May 19, 2009 at 07:27, Phillips, Addison <[email-masked]> wrote:

> AP> Good points both. I did a more extensive edit to fix this. In the new
> document, it isn't "many" but rather "all" of the redundant/grandfathered
> mappings that fit the two categories provided: there are no other kinds of
> P-V values! To convey all this I put:
>
> --
> Preferred-Value fields in the registry provide mappings from deprecated
> tags to modern equivalents. Many of these were created before the adoption
> "tlh"). Others are the result of later registrations or additions to the
> registry as permitted or required by this document (for example, "zh-hakka"
> was deprecated in favor of the ISO 639-3 code 'hak' when this document was
> adopted).
> --
>
>
> --
>
> AP> The first one illustrates extension ordering. The second one
> illustrates preferred-value mapping. These are covered by the text now in a
> way that wasn't as apparent before. But I digress, since I already said I
> wasn't going to muck with them anymore.
>
> Addison
>

---

From: Mark Davis <[email-masked]>
Date: Tue, 19 May 2009 10:45:40 -0700

I should quickly add that I'm also ok with leaving the examples where they
are in http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22.html, just
in the interests of timeliness.

Mark

On Tue, May 19, 2009 at 10:43, Mark Davis <[email-masked]> wrote:

> Since a particular language tag can be used by many processes, language
> tags SHOULD always be created or generated in canonical form.
>
> to the rules in Section 2.1 (Syntax)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22.html#syntax>and
> Section 2.2 (Language Subtag Sources and Interpretation)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22.html#sources>and it has been canonicalized by applying each of the following steps in
> IANA Language Subtag Registry)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22.html#ianaformat>
> ):
>
>       well-formed and potentially valid (extensions 'a' and 'b' are not defined as
>       of the publication of this document) but not in a canonical form because the
>    - The field-body of the Preferred-Value for grandfathered and redundant
>       tags is an "extended language range" ([RFC4647] (Phillips, A. and M.
>       Davis, “Matching of Language Tags,” September 2006.)<http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-22.html#RFC4647>)
>       deprecated tags to modern equivalents. Many of these were created before the
>       example, "zh-hakka" was deprecated in favor of the ISO 639-3 code 'hak' when
>    For extlangs, the original primary language subtag is also replaced if there
>    is a primary language subtag in the Preferred-Value.
>       valid, it is not in canonical form because the 'BU' subtag has a canonical
> On Tue, May 19, 2009 at 07:27, Phillips, Addison <[email-masked]>wrote:
>