Subject: [Ltru] Ticket #45: AD Issue #12: reason for SHOULD in 4.5 extlang mapping

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 29 Apr 2009 15:43:22 -0700

The AD asked:

--
AD review comment #12 from
http://www.ietf.org/mail-archive/web/ltru/current/msg12399.html

12).

    4.5. Canonicalization of Language Tags

    [...]

    3. Subtags of type 'extlang' SHOULD be mapped to their Preferred-
    Value.

Why use of SHOULD here? I.e. what is a good reason for violating this rule?
A pointer here to some case(s) would be appreciated here.

    The field-body of the Preferred-Value for extlangs is an
    "extended language range" and typically maps to a primary
    language subtag. For example, the subtag sequence "zh-hak"
    (Chinese, Hakka) would be replaced with the tag "hak" (Hakka
--

The keyword is (correctly) SHOULD because removing the primary language subtag removes information that some applications may wish to preserve.

Proposed resolution:

Add this text to the above paragraph, indicating why you might not perform the mapping:

--
Because this involves removing the macrolanguage information from the tag, an implementation might choose not to perform this particular canonicalization, if such a canonicalization would reduce the effectiveness of that resulting tags for matching or selection later.
--

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

---

From: Mark Davis <[email-masked]>
Date: Wed, 29 Apr 2009 16:30:08 -0700

This doesn't really solve the problem. A language tag is either in canonical
format or it is not, and we need to be crystal-clear about what the format
is. The problem arises from the numbered clauses being a mixture of
descriptive (*a language-tag is in canonical form WHEN X, Y, Z*) text, and
procedural text (*in order to canonicalize a language-tag, you MUST do A, B,
C*).

For example:

   - A language tag *IS* in canonical form when it contains no tags/subtags
   with Preferred-Values.
   - In order to *put* a language-tag into a canonical form, the
   tags/subtags *MUST be* replaced by their Preferred-Values.

The text should be restructured to something like the following, and have
consistent use of MUST for each clause. I tried to make the minimal changes
necessary to resolve the issue. At the end, we can then make clear that
there are perfectly good circumstances where people can -- with perfect
reason -- avoid doing the extlang mapping, using a recast version of your
paragraph.

The changes are marked in yellow.

   A language tag is in canonical form when:

   1.  The tag is well-formed according the rules in Section 2.1
<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-2.1>
and
       Section 2.2
<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-2.2>.

   2.  It has been canonicalized according to the following process:

     1.  Redundant or grandfathered tags that have a Preferred-Value
         mapping in the IANA registry (see Section 3.1
<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-3.1>)
MUST be replaced
         with their mapped value.  These items either are deprecated
         mappings created before the adoption of this document (such as
         the mapping of "no-nyn" to "nn" or "i-klingon" to "tlh") or are
         the result of later registrations or additions to this document
         (for example, "zh-hakka" was deprecated in favor of the ISO 639-3
         code 'hak' when this document was adopted).  These mappings
         MUST be applied before subsequent canonicalization steps,
         since there can be additional changes to the mapped subtag values.
         These field-body of the
         Preferred-Value for grandfathered and redundant tags is an
         "extended language range" ([RFC4647
<http://tools.ietf.org/html/rfc4647>]) and might consist of more
         than one subtag.

     2.  Subtags of type 'extlang' MUST be mapped to their Preferred-
         Value.  The field-body of the Preferred-Value for extlangs is an
         "extended language range" and typically maps to a primary
         language subtag.  For example, the subtag sequence "zh-hak"
         (Chinese, Hakka) would be replaced with the tag "hak" (Hakka).

     3.  Other subtags that have a Preferred-Value field in the IANA
         registry (see Section 3.1
<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-3.1>)
MUST be replaced with their mapped
         value.  Most of these are either Region subtags where the country
         name or designation has changed or clerical corrections to ISO
         639-1.

     4.  If more than one extension subtag sequence exists, the extension
         sequences MUST be ordered into case-insensitive ASCII order by
         singleton subtag (that is, the subtag sequence '-a-babble' comes
         before '-b-warble').

   Step 2.2 maps extlang fields by removing the macrolanguage
information from the
   tag. That information can be restored with access to the language
subtag registry.
   A partial canonicalization (omitting step 2.2) may also be performed. Such a
   partial canonicalization may be useful in environments where the
macrolanguage
   is used in matching or selection and there is no access to the registry.

For comparison, the original text is:

The original text is:

   A language tag is in canonical form when:

   1.  The tag is well-formed according the rules in Section 2.1
<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-2.1>
and
       Section 2.2
<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-2.2>.

   2.  Redundant or grandfathered tags that have a Preferred-Value
       mapping in the IANA registry (see Section 3.1
<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-3.1>)
MUST be replaced
       with their mapped value.  These items either are deprecated
       mappings created before the adoption of this document (such as
       the mapping of "no-nyn" to "nn" or "i-klingon" to "tlh") or are
       the result of later registrations or additions to this document
       (for example, "zh-hakka" was deprecated in favor of the ISO 639-3
       code 'hak' when this document was adopted).  These mappings
       SHOULD be done before additional processing, since there can be
       additional changes to subtag values.  These field-body of the
       Preferred-Value for grandfathered and redundant tags is an
       "extended language range" ([RFC4647
<http://tools.ietf.org/html/rfc4647>]) and might consist of more
       than one subtag.

   3.  Subtags of type 'extlang' SHOULD be mapped to their Preferred-
       Value.  The field-body of the Preferred-Value for extlangs is an
       "extended language range" and typically maps to a primary
       language subtag.  For example, the subtag sequence "zh-hak"
       (Chinese, Hakka) would be replaced with the tag "hak" (Hakka).

   4.  Other subtags that have a Preferred-Value field in the IANA
       registry (see Section 3.1
<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-3.1>)
MUST be replaced with their mapped
       value.  Most of these are either Region subtags where the country
       name or designation has changed or clerical corrections to ISO
       639-1.

   5.  If more than one extension subtag sequence exists, the extension
       sequences are ordered into case-insensitive ASCII order by
       singleton subtag (that is, the subtag sequence '-a-babble' comes
       before '-b-warble').

Mark

On Wed, Apr 29, 2009 at 15:43, Phillips, Addison <[email-masked]> wrote:

> The keyword is (correctly) SHOULD because removing the primary language
> subtag removes information that some applications may wish to preserve.
>
> Proposed resolution:
>
> Because this involves removing the macrolanguage information from the tag,
> an implementation might choose not to perform this particular
> canonicalization, if such a canonicalization would reduce the effectiveness
> of that resulting tags for matching or selection later.
> --
>
> Addison Phillips
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ltru
>

---

From: John Cowan <[email-masked]>
Date: Wed, 29 Apr 2009 19:56:35 -0400

> reason -- avoid doing the extlang mapping, using a recast version of your

+1 to all points.

-- 
I suggest you solicit aid of my followers       John Cowan
or learn the difficult art of mud-breathing.    [email-masked]
        --Great-Souled Sam                      http://www.ccil.org/~cowan

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 29 Apr 2009 19:49:02 -0700

Ulp.

Yes, of course, you’re right. However the canonicalization of extlangs, IIRC, was  a stumbling block and this was the compromise. It’s pure fudge. I support your changes.

One bit of text you suggest is:

--
That information can be restored with access to the language subtag registry.
--

Actually, that’s not quite right. There are two canonical forms: “no extlang” and “always extlang”. Once the primary language subtag has been removed, you can’t tell if it was there (or not), so “restoration” might be changing the original tag. So one canonicalization form is as you write below. The other form always pushes the macrolanguage onto the tag.

“NO” form:

Zh-hak -> hak
Zh-cmn-Hant-CN -> cmn-Hant-CN

”EXTLANG” form:

Zh-hak => zh-hak
Zh-cmn-hant-cn => zh-cmn-Hant-CN
Hak => zh-hak
Cmn-hant-cn => zh-cmn-Hant-CN

The no-extlang form requires no registry access and is the default. I would suggest that we define both forms, with the “no” form as default, and have a note that says that canonicalization is not required using your text.

Addison

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Wednesday, April 29, 2009 4:30 PM
To: Phillips, Addison
Cc: LTRU Working Group
Subject: Re: [Ltru] Ticket #45: AD Issue #12: reason for SHOULD in 4.5 extlang mapping

This doesn't really solve the problem. A language tag is either in canonical format or it is not, and we need to be crystal-clear about what the format is. The problem arises from the numbered clauses being a mixture of descriptive (a language-tag is in canonical form WHEN X, Y, Z) text, and procedural text (in order to canonicalize a language-tag, you MUST do A, B, C).

For example:

 *   A language tag IS in canonical form when it contains no tags/subtags with Preferred-Values.
 *   In order to put a language-tag into a canonical form, the tags/subtags MUST be replaced by their Preferred-Values.
The text should be restructured to something like the following, and have consistent use of MUST for each clause. I tried to make the minimal changes necessary to resolve the issue. At the end, we can then make clear that there are perfectly good circumstances where people can -- with perfect reason -- avoid doing the extlang mapping, using a recast version of your paragraph.

   1.  The tag is well-formed according the rules in Section 2.1<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-2.1> and

       Section 2.2<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-2.2>.

         mapping in the IANA registry (see Section 3.1<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-3.1>) MUST be replaced

         These field-body of the

         "extended language range" ([RFC4647<http://tools.ietf.org/html/rfc4647>]) and might consist of more

         than one subtag.

         registry (see Section 3.1<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-3.1>) MUST be replaced with their mapped

         639-1.

         before '-b-warble').

   Step 2.2 maps extlang fields by removing the macrolanguage information from the

   tag. That information can be restored with access to the language subtag registry.

   partial canonicalization may be useful in environments where the macrolanguage

The original text is:

   1.  The tag is well-formed according the rules in Section 2.1<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-2.1> and

       Section 2.2<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-2.2>.

       mapping in the IANA registry (see Section 3.1<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-3.1>) MUST be replaced

       "extended language range" ([RFC4647<http://tools.ietf.org/html/rfc4647>]) and might consist of more

       than one subtag.

       registry (see Section 3.1<http://tools.ietf.org/html/draft-ietf-ltru-4646bis-21#section-3.1>) MUST be replaced with their mapped

       639-1.

       before '-b-warble').

Mark

On Wed, Apr 29, 2009 at 15:43, Phillips, Addison <[email-masked]<mailto:[email-masked]>> wrote:
The AD asked:

12).

   [...]

Proposed resolution:

Addison Phillips

_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ltru