Subject: Re: [Ltru] Ticket #45: AD Issue #12: reason for SHOULD in 4.5 extlang mapping

From: Kent Karlsson <[email-masked]>
Date: Fri, 01 May 2009 16:02:23 +0200

See below. This message includes proposed text towards the end of the
message.

Den 2009-04-30 22.53, skrev "Phillips, Addison" <[email-masked]>:

> Although I agree with your sentiments, I think we can avoid some additional
> working group bloodletting by being careful with our wording choices. I
> particularly note that RFC 2119 keywords, while have a normative weight to
> them, are not the only way to be ³normative² in a spec. Thus, I propose to
> take your suggestion:
>  
> --
> The default canonical form SHOULD normally be used for canonicalization. The
> extended canonical form may be useful
> in environments where the presence of the macrolanguage is beneficial in
> matching or selection.
> --
>  
> Š but modified to readŠ
>  
> --
> <t>Normally, the 'default' canonicalization is preferred. However, the
> 'extended' canonical form is useful
> in environments where the presence of the macrolanguage is beneficial in
> matching or selection (see <xref target="choiceUsingExtlang"></xref>).</t>
> --
>  
> In incorporating your proposed changes, I noticed that there was no need for
> double- or triple-list embedding. I also note that, since we are describing a
> canonicalization process, using 2119 keywords isn¹t necessary. We can just say
> ³do X².
>  
> The resulting text reads as:
>  
> --
> A language tag is in a canonical form when the tag is well-formed according
> the rules in <xref target="syntax"/> and
> <xref target="sources"/> and it has been canonicalized as follows:

I think the fact that there are two canonical forms should be made clear
from the outset, not just further down in the definition.

>  <list style="numbers">
>                                                                               
>                  
> <t>Redundant or grandfathered tags that have a Preferred-Value mapping in the
> IANA registry (see <xref target="ianaformat"/>) are replaced with their mapped
> value.

That takes care of "redundant" and "grandfathered" (whole) tags, but not
subtag entries like "iw"-> "he", "in" -> "id", ... They are left to much
later. I don't see why. Logically they belong here.

Here the term "mapped value" is used, but I think "preferred value" is
clearer, and does not introduce another term.

> These items are either deprecated mappings created before the adoption
> of this document (such as the mapping of "no-nyn" to "nn" or "i-klingon" to
> "tlh") or are the result of later registrations or additions to this document
> (for example, "zh-hakka" was deprecated in favor of the ISO 639-3 code 'hak'
> when this document was adopted).

This is auxiliary text that is not needed for the definition of canonical
form(s).

> These mappings MUST be done before additional
> processing, since there can be additional changes to subtag values.

"additional processing"? Do you mean that they are to be performed as the
first step in canonicalisation, since the following steps may do additional
changes? I guess so, but the text does not say that. It seems to refer to
some other processing (that is "additional"). And if it refers to
canonicalisation, which I think it does, grandfathered and redundant tags
can't have more subtags applied to them, so for them the only following
changes to subtags must be within the preferred value tag, and then (as long
as Hant and Hans are not deprecated) it is only replacement of a primary
language tag with a macrolangauge-extlang combination (for the extended
canonical form) that may happen.

> These 
> field-body of the Preferred-Value for grandfathered and redundant tags is an
> "extended language range" (<xref target="RFC4647"></xref>) and might consist
> of more than one subtag.</t>

> <t>One of the following canonical forms has been applied:

That sentence is odd, a definition of canonical forms within the definition
of canonical form...

> <list style="letters">
>   
> <t>In the 'default' canonical form, subtags of type 'extlang' are mapped to
> their Preferred-Value. The field-body of the Preferred-Value for extlangs is
> an "extended language range",

Mapping just the subtag will not do anything here (if we just consider the
strings, ignoring the primary langauge vs. extlang classification). One
needs to map the language tag prefix, up to and including the extlang subtag
in question, to the preferred value. Due to the restriction to at most one
level of extlang, it suffices to say that needs to map the
macrolangauge-extlang combination (as a prefix) to the preferred value.

> typically a primary language subtag (in all such
> cases, the primary language subtag is removed).

I don't think one need to say "typically", particularly not next to saying
"in all such cases", is a good formulation. And the primary language subtag
is of course not removed, all language tags have a primary language subtag,
but the macrolanguage-extlang combination, as a prefix, is replaced by
another primary language subtag (which **as a string** in all cases are
equal to the extlang subtag).

> For example, the subtag
> sequence "zh-hak" (Chinese, Hakka) would be replaced with the tag "hak"
> (Hakka).</t>
>  
> <t>In the 'extended' canonical form, primary language subtags with a
> 'Macrolanguage' field that are also registered as 'extlang' subtags are

The sign language primary language subtags have no macrolanguage. Do you
mean to exclude those from getting "sgn-" prepended? I guess not.

> replaced by their macrolanguage-extlang combination. For example, the language
> tag "hak" (Hakka) has a Macrolanguage of 'zh' (Chinese) and an existing
> 'extlang' registration. The tag would be replaced with that tag "zh-hak"
> (Chinese, Hakka).</t>
>  
> </list></t>
>  
> <t>Other subtags that have a Preferred-Value field in the IANA registry (see
> <xref target="ianaformat"/>) have been replaced with their mapped value.

> Most 
> of these are either Region subtags where the country name or designation has
> changed or clerical corrections to ISO 639-1.</t>

Auxiliary text that is not needed in the definition.

> <t>If more than one extension subtag sequence exists, the extension sequences
> are ordered into case-insensitive ASCII order by singleton subtag (that is,
> the subtag sequence '-a-babble' comes before '-b-warble').</t>

The extension subtags and the name "extended canonical form" could make a
reader believe that they are somehow related. I'd prefer to use the term
"extlang canonical form", since that form keeps/introduces extlangs (as
opposed to the default canonical form) and is unrelated to extensions.

===========================================================
===========================================================
PROPOSED TEXT:

A language tag is in canonical form, either default canonical form or
extlang canonical form, when the tag is well-formed according to the
rules in <xref target="syntax"/> and <xref target="sources"/> and
canonicalising it does not change it.

================================
Comment: maybe it would be clearer to have "well-formed" refer to just
the syntax (which is not dependent on current registry data), and have
"well-defined" for the additional requirement that the subtags are
(currently) registered.
================================

Canonicalisation of a well-formed [or well-defined, see comment above]
language tag is defined by doing the following steps, in order, using data
from the current IANA language subtag registry (<xref
target="ianaformat"/>).

1. Extension sequences in the tag are ordered into case-insensitive ASCII
   order by the singleton subtags. (At the time of publication of this
   document, there were no extension subtags registered.)

2. A redundant or grandfathered tag that has a Preferred-Value field in
   the IANA registry is replaced with its preferred value.

3. A non-extlang subtag that has a Preferred-Value field in the IANA
   registry is replaced with its preferred value.

4. A tag prefix of the form language-extlang is replaced by the
   preferred value registered for the extlang.

5. For the extlang canonical form (but not for the default canonical form),
   a primary language subtag that is also registered as an 'extlang' subtag
   is replaced by the corresponding language-extlang combination, where the
   primary language subtag here is the Prefix registered for the extlang.

=======================================================
Comment: This would be a clear and simple step-by-step definition of the
canonical forms. After this definition you can put text about usage,
examples, and auxiliary explanatory text to your hearts content (those in
the original text are fine in general, just move such text to after the
definition).
====================================================

===========================================================
===========================================================

        /kent k

---

From: CE Whitehead <[email-masked]>
Date: Fri, 1 May 2009 16:31:32 -0400

Hi!
From: "Phillips, Addison" <addison at amazon.com> 

Date: Thu, 30 Apr 2009 09:13:11 -0700 
>(as individual contributor)
 
>> >    3.  In the 'default' canonical form, subtags of type 'extlang'
>> MUST
>> >    4.  In the 'extended' canonical form, primary language subtags
>> with a
 
>> I propose the names "short" and "long" respectively.

> 'extended' seems like a more natural name for the latter form.* Yes >I acknowledge that the name 'default' is prejudicial, in its way, but it was the previous intent of 'SHOULD' and it is the canonical form that the registry is optimized for. > If we don't want to say that it's the default, we could pick a name that is partial antonym for 'extended', such as 'compact', 'regular', or (hmm...) 'short'. 'abbreviated'?  'brief'?  But I have no problem with 'default'. Also, I like Mark's text; it's clear: 4.5.  Canonicalization of Language Tags

  Since a particular language tag is sometimes used by many processes,
  language tags SHOULD always be created or generated in a canonical
  form.

  There are two canonical forms for language tags.  The 'default'
  canonical form maps each 'extlang' subtag to its Preferred-Value.
  The 'extended' canonical form includes the macrolanguage primary
  language subtag before eligible (extended) language subtags." And I think I understand Addison's amendment (I'm just not up on the changes to the values that occur during processing--the length is changed?) & agree there needs to be an amendment: ""These mappings MUST be done before additional processing, since there can be additional changes to subtag values."
  Best,--C. E. [email-masked]> How about 'short' and 'extended'?
>If we don't say 'default' (or even if we do), oughtn't we to have some text explaining when to use which?

>Addison