Subject: Re: [Ltru] Ticket #45: AD Issue #12: reason for SHOULD in 4.5 extlang mapping

From: CE Whitehead <[email-masked]>
Date: Sat, 2 May 2009 10:28:40 -0400

Hi!

I.  
I still think Mark’s text reads more clearly than other possible introductory text and that it is the text to start with in the introduction to canonicalization . . . I can't do anything with:
"A language tag is in canonical form, either default canonical form or
extlang canonical form, when the tag is well-formed according to the
rules in <xref target="syntax"/> and <xref target="sources"/> and
canonicalising it does not change it" 

Mark's text clearly introduces the two types of canonicalization, which you all seem to think you need to do.

)

Mark’s suggested introductory text:

"4.5.  Canonicalization of Language Tags

  Since a particular language tag is sometimes used by many processes,
  language tags SHOULD always be created or generated in a canonical
  form.

  There are two canonical forms for language tags.  The 'default'
  canonical form maps each 'extlang' subtag to its Preferred-Value.
  The 'extended' canonical form includes the macrolanguage primary
  language subtag before eligible (extended) language subtags."

However, Kent points out that,

"Mapping just the subtag will not do anything here (if we just consider the
strings, ignoring the primary language vs. extlang classification). One
needs to map the language tag prefix, up to and including the extlang subtag
in question, to the preferred value." 

So then, do we replace: 

"each 'extlang' subtag" 

with 

"each subtag in the primary language-extension language combination" ???

Also, I want to clarify the implication of Kent’s suggested change to the steps in canonicalization going to affect the use of the term "macrolanguage" in the above?(Kent's worried about sign language subtags that are extension language subtags but don't have a macrolanguage):

"5. For the extlang canonical form (but not for the default canonical form),   a primary language subtag that is also registered as an 'extlang' subtag   is replaced by the corresponding language-extlang combination, where the

   primary language subtag here is the Prefix registered for the extlang."

Does the above mean that we need to say here:

"The ‘extlang’ canonical form includes the registered prefix for the extension language before eligible (extlang) language subtags"

???

(I noted that Kent did not like "extended canonical form" and acted accordingly.)

These changes would result in:

  "Since a particular language tag is sometimes used by many processes,
  language tags SHOULD always be created or generated in a canonical
  form.

  There are two canonical forms for language tags.  The 'default'
  canonical form maps each each subtag in the primary language-extension language combination to its Preferred-Value.
  The 'extended' canonical form includes the registered prefix for the extension language before eligible (extlang) language subtags."

II.

I also find the following text useful:

<t>Normally, the 'default' canonicalization is preferred. However, the 'extlang' canonical form  is useful

in environments where the presence of the macrolanguage is beneficial in matching or selection (see <xref target="choiceUsingExtlang"></xref>).</t>

III.

My goof on Addison’s text (below)—as Kent pointed out, Addison means that: “mapping a subtag to its preferred value should occur before any additional steps in canonicalization”

>"These mappings MUST be done before additional processing, since there can be additional changes to subtag values." 

>form(s)."

But then would not Kent's suggested step 2  (below) become step 1? but otherwise everything follows mapping:

>Canonicalisation of a well-formed [or well-defined, see comment above]

>language tag is defined by doing the following steps, in order, using data

>from the current IANA language subtag registry (<xref

>target="ianaformat"/>).

>1. Extension sequences in the tag are ordered into case-insensitive ASCII

>   order by the singleton subtags. (At the time of publication of this

>   document, there were no extension subtags registered.)

>2. A redundant or grandfathered tag that has a Preferred-Value field in

>   the IANA registry is replaced with its preferred value.

> 3. A non-extlang subtag that has a Preferred-Value field in the IANA

>   registry is replaced with its preferred value.

> 4. A tag prefix of the form language-extlang is replaced by the

>   preferred value registered for the extlang.

> 5. For the extlang canonical form (but not for the default canonical form),

>   a primary language subtag that is also registered as an 'extlang' subtag

>   is replaced by the corresponding language-extlang combination, where the

>   primary language subtag here is the Prefix registered for the extlang.

(NOTE: Hope my suggestions make some sense.  Sorry if I'm not keeping up with the discussion on this (the postings online aren't quite up-to-date???)

--C. E. Whitehead

[email-masked]

---

From: Kent Karlsson <[email-masked]>
Date: Sat, 02 May 2009 17:24:21 +0200

Den 2009-05-02 16.28, skrev "CE Whitehead" <[email-masked]>:

> Hi!
>  
> I.  
> I still think Mark¹s text reads more clearly than other possible introductory
> text and that it is the text to start with in the introduction to
> canonicalization . . . I can't do anything with:

Part of my suggestion was to *first* have the definition, then say something
about recommended use.
>  
> Mark¹s suggested introductory text:
>  
> Also, I want to clarify the implication of Kent¹s suggested change to the
> steps in canonicalization going to affect the use of the term "macrolanguage"
> in the above?(Kent's worried about sign language subtags that are extension
> language subtags but don't have a macrolanguage):
> "5. For the extlang canonical form (but not for the default canonical form),
> a primary language subtag that is also registered as an 'extlang' subtag   is
> replaced by the corresponding language-extlang combination, where the
> "The Œextlang¹ canonical form includes the registered prefix for the extension
> language before eligible (extlang) language subtags"
> ???
> 
I think having step 5 saying that is sufficient, no need to duplicate that
statement.

>   canonical form maps each each subtag in the primary language-extension
> language combination to its Preferred-Value.
>   The 'extended' canonical form includes the registered prefix for the
> extension language before eligible (extlang) language subtags."
>  
> II.
>  
> in environments where the presence of the macrolanguage is beneficial in
> matching or selection (see <xref target="choiceUsingExtlang"></xref>).</t>
 
Saying "macrolanguage" here is not exactly correct.  Otherwise text like
that can go after the definition of the normal forms.

> III.
>  
> My goof on Addison¹s text (below)‹as Kent pointed out, Addison means that:
> ³mapping a subtag to its preferred value should occur before any additional
> steps in canonicalization²
>  
 
The ordering does not matter for all steps. But I suggest having one order
that is correct (of course), makes some kind of logical grouping, and have
the optional part last. And then say "in order" without making a great fuss
using "MUST". Also: have the steps as clean as possible, moving all
examples, explanations, text on usage, to after the definition of the
canonical forms rather than bog down the steps in the (algorithmic)
definition with such text.

    /kent k

> (NOTE: Hope my suggestions make some sense.  Sorry if I'm not keeping up with
> the discussion on this (the postings online aren't quite up-to-date???)
> 
> --C. E. Whitehead
> [email-masked]
> 
> 
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ltru

---

From: CE Whitehead <[email-masked]>
Date: Sat, 2 May 2009 12:02:56 -0400

Hi!

Date: Sat, 2 May 2009 17:24:21 +0200
From: [email-masked]

> Den 2009-05-02 16.28, skrev "CE Whitehead" <[email-masked]>:

. . .
   
>>I still think Mark’s text reads more clearly than other possible introductory text and that it is the text to start with >>in the introduction to canonicalization . . . I can't do anything with:

> Part of my suggestion was to *first* have the definition, then say something about recommended use.
Yes, I know.
Mark's text gives a definition I think when he explains the two forms.  We can reorder it a bit but I like it.

>>. "4.5.  Canonicalization of Language Tags

So then, do we replace: 
"each 'extlang' subtag" 
with 

> I think having step 5 saying that is sufficient, no need to duplicate that statement.
Hmm  I was just trying to be consistent as far as vocabulary goes.

 
 
. . .

> Saying "macrolanguage" here is not exactly correct.  
Sorry, my goof!
> Otherwise text like that can go after the definition of the normal forms.
That makes me happy.
. . . 
 
>>. . . as Kent pointed out, Addison means that: “mapping a subtag to its preferred value should occur before any additional >>steps in canonicalization”

> The ordering does not matter for all steps. But I suggest having one order that is correct (of course), makes some kind of 
> logical grouping, and have the optional part last. And then say "in order" without making a great fuss using "MUST". Also: have > the steps as clean as possible, moving all examples, explanations, text on usage, to after the definition of the canonical forms 
> rather than bog down the steps in the (algorithmic) definition with such text.

    /kent k
O.k. maybe we could specify what the mapping to the preferred value should precede.  Grandfathered tags have to be mapped before other tags.  And, in the case that the extlang canonical form is desired, the mapping has to be done before the primary language subtag is replaced by the corresponding language-extension-language combination. 
 
Best,
 
C. E. Whitehead
[email-masked]

---

From: Mark Davis <[email-masked]>
Date: Sat, 2 May 2009 11:10:14 -0700

*I started with Addison's formulation, and then tried to combine all the
suggestions that did not materially change the content; I also separated out
the examples and explanation into sub-bullets for to keep them from muddling
the main rules, but did not move them to a different point in the text just
to reduce the extent of the changes. I retained other cases of original text
or structure even where I felt Kent's to be somewhat better, also to reduce
changes.

I correlated this with the original (v21) text to make it clear what the
changes were. Please see:
*
http://sites.google.com/site/macchiato/unicode/ltru-canonicalization

Please let me know of any further suggestions.

Mark

---

From: Kent Karlsson <[email-masked]>
Date: Sat, 02 May 2009 22:31:40 +0200

So which is the intended extended canonical form for "swl", for instance?
According to my suggestion it would be "sgn-swl". But it is unclear
according to Mark's suggestion, as that case is not covered.

I still don't like the name "extended canonical form". How is it extended?
(It's not related to language tag extensions, though appear to be by name.)
"Extlang canonical form" is better, as this form uses the extlang feature.
 
    /kent k

Den 2009-05-02 20.10, skrev "Mark Davis" <[email-masked]>:

> I started with Addison's formulation, and then tried to combine all the
> the main rules, but did not move them to a different point in the text just to
> reduce the extent of the changes. I retained other cases of original text or
> structure even where I felt Kent's to be somewhat better, also to reduce
> changes. 
>

---

From: Mark Davis <[email-masked]>
Date: Sat, 2 May 2009 15:48:06 -0700

> according to Mark's suggestion, as that case is not covered.

Let's look at two examples, so that we can clarify the text. I marked the
only material difference between them below.

Type: language
Subtag: zh
Description: Chinese
Added: 2005-10-16
Scope: macrolanguage

Type: language
Subtag: sgn
Description: Sign languages
Added: 2005-10-16
Scope: collection

Type: extlang
Subtag: yue
Description: Yue Chinese
Added: 2029-09-09
Preferred-Value: yue
Prefix: zh
*Macrolanguage: zh* *// only material difference*

Type: extlang
Subtag: swl
Description: Swedish Sign Language
Added: 2029-09-09
Preferred-Value: swl
Prefix: sgn

We find that the primary language "yue" is also an extlang. Because it has a
Macrolanguage field 'zh', we change 'yue' to 'zh-yue'. In the second case,
the primary language "swl" is also an extlang, but it doesn't have a
Macrolanguage field, so it isn't replaced.

My language was:*

* In the 'extended' canonical form (but not the 'default' form), primary
language subtags that are also registered as 'extlang' subtags are replaced
by their macrolanguage-extlang combinations.

This is the same as Addison's, although I added a qualifying note from your
text: "In the 'extended' canonical form, primary language subtags with a
'Macrolanguage' field that are also registered as 'extlang' subtags are
replaced by their macrolanguage-extlang combination."
* *
While I think the language is ok, especially in light of the immediately
following example, I reworded to:

In the 'extended' canonical form (but not the 'default' form), primary
language subtags are replaced by macrolanguage-extlang combinations if they
are also 'extlang' subtags with a Macrolanguage field.

*I also fixed the example following this text.*

You had suggested:
For the extlang canonical form (but not for the default canonical form), a
primary language subtag that is also registered as an 'extlang' subtag is
replaced by the corresponding language-extlang combination, where the
primary language subtag here is the Prefix registered for the extlang.

Your language doesn't correctly distinguish these two cases, which are only
different because of the Macrolanguage field in the extlang record, NOT
whether or not there is a Prefix (which both of them have). However, due to
your raising this issue, I think the text is now better.

> "Extlang canonical form" is better, as this form uses the extlang feature.

I'm fine with "extlang canonical form", I agree that it is clearer. I'll go
ahead and make that change in my proposed copy, but I can revert if anyone
has a better suggestion.

> ------------------------------

---

From: Kent Karlsson <[email-masked]>
Date: Sun, 03 May 2009 01:21:21 +0200

Mark Davis wrote:
============
> My language was:
> language subtags that are also registered as 'extlang' subtags are replaced by
> their macrolanguage-extlang combinations.
> 
> I also fixed the example following this text.
> 
> You had suggested:
> replaced by the corresponding language-extlang combination, where the primary
> language subtag here is the Prefix registered for the extlang.
> 
===========

Well, that was the point. I think the extlang canonical form for the sign
languages should be "sgn-zzz" for zzz as one of the sign language code. We
made sign languages extlangs despite that sgn is a collection code and not a
macrolanguage code, just because the exlang proponents wanted "sgn" at the
beginning of sign language language tags, did we not?

    /kent k

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 2 May 2009 21:08:02 -0700

Hi -

As Co-chair...

> From: "Mark Davis" <[email-masked]>
> To: <[email-masked]>
> Sent: Saturday, May 02, 2009 11:10 AM
> Subject: Re: [Ltru] Ticket #45: AD Issue #12: reason for SHOULD in 4.5extlang mapping
...
> http://sites.google.com/site/macchiato/unicode/ltru-canonicalization
...

Since this discussion is not particularly easy to follow, and in
the interest of keeping a historical record of the actual text
being considered by the working group, please post the current
proposed text to the mailing list, too.

Randy