Subject: [Ltru] Canonicalizing variants

From: "Doug Ewell" <[email-masked]>
Date: Sun, 4 Mar 2007 19:17:20 -0800

Mark's proposed changes to draft-4646bis [1] include the following in 
Section 4.4 (my emphasis):

"... language tags SHOULD always be *created or generated* in a 
canonical form.  A language tag is in canonical form when:
...
6. Any sequence of variant tags MUST be reordered such that for each 
variant subtags A and B, A comes before B if
    1.  A has no prefix and B has a prefix
    2.  or A and B are not ordered by #1, and A is part of a prefix 
(given the previous non-variant subtags) for B
    3.  or A and B are not ordered by #1 or #2, and A is alphabetically 
before B"

Elsewhere I thought I remembered Mark saying that this step was intended 
for tag *consumers* to aid matching.

I expressed concerns that tag producers would make the necessary 
distinction between prefix and no-prefix variants, and suggested that 
the "natural" order of thinking about certain tags might be at odds with 
the proposed canonical form.

Is this step proposed as a constraint on tag producers or tag consumers?

[1] http://docs.google.com/Doc?id=dfqr8rd5_4gh864p

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Martin Hosken <[email-masked]>
Date: Mon, 05 Mar 2007 13:00:14 +0700

Dear Doug,
> Elsewhere I thought I remembered Mark saying that this step was
> intended for tag *consumers* to aid matching.
>
> the "natural" order of thinking about certain tags might be at odds
> with the proposed canonical form.

And then I produced an alternative canonicalisation that seems to
address that. Interestingly there seems to be little response to it.
Perhaps it needs simplifying a little:

  1. A is a language variant and B is one of a script, or regional variant or has no prefix
  2. A is a script variant and B is one of a regional variant or has no prefix
  3. A is a regional variant and B has no prefix
  4. A is part of a prefix for B
  5. A is alphabetically before B

Where a variant is considered:

  1. A language variant if its prefix(es) or ancestoral prefix(es) consist only of language subtags
  2. A script variant if its prefix(es) or ancestoral prefix(es) contains a script subtag but no regional subtag.
  3. A regional variant if its prefix(es) or ancestoral prefix(es) contains a regional subtag

Where the ancestoral prefixes are the prefixes of the ancestoral prefixes or the variant in question.

Yours,
Martin

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 4 Mar 2007 23:21:06 -0800

Martin Hosken <martin underscore hosken at sil dot org> wrote:

>  2. A script variant if its prefix(es) or ancestoral prefix(es) 
> contains a script subtag but no regional subtag.

If anything, I think this more complex hierarchy would be even more 
difficult for tag producers to grok, and even less likely to be 
observed.

My suggestion is to throw the burden of tag matching back onto the tag 
consumer.

---

From: Martin Duerst <[email-masked]>
Date: Mon, 05 Mar 2007 21:05:49 +0900

At 12:17 07/03/05, Doug Ewell wrote:

>I expressed concerns that tag producers would make the necessary distinction between prefix and no-prefix variants, and suggested that the "natural" order of thinking about certain tags might be at odds with the proposed canonical form.
>

[general technical comment]

If it's on the tag consumer, then we have to either go back and revise RFC 4647,
or we admit that this is only wishful thinking anyway. If it's on the producer,
we risk that some will get it wrong, but whoever gets it wrong will have to
pay themselves (their material not being found or their request not returning
many documents).

As for people not getting the right sequence, it shouldn't be magic for somebody
to produce a Web page somewhere that gives the right advice (i.e. canonicalizes
some input).

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 5 Mar 2007 09:21:07 -0800

1. I don't know why you thought that I wrote "tags SHOULD always be *created
or generated* ", unless you are more color-blind than I am ;-). All of my
changes were marked in yellow. All of the other text was from the previous
draft, and is not part of my proposed changes.

However, I don't think there is anything wrong with that phrasing. I
should produce
"en-US" and not "En-Us" even though the spec says that they are equivalent,
because the more I hew to the canonical form, the less likely that some
errant process will screw up matching. Giving that strong guidance to users
of the specification is appropriate. Even if the ordering of variants seems
somewhat less natural, having a unique ordering will at least result in
determinant behavior instead of random breakage.

It is a SHOULD, not a hard and fast constraint. So if My Environment
Requires Titlecase, I Could Use "En-Us" -- but I should recognize that it
would be better to use the canonical form.

2. prefix vs no-prefix variants
> distinction between prefix and no-prefix variants,

Because we already have the distinction between these in the last version of
the spec, and the requirements on them are different, it would be
irresponsible of us not to indicate to people that they have to take certain
steps to avoid invalid tags. When we have a variant A with a prefix
containing B, it is simply invalid to have ...-B-A instead of the reverse.
So we just need to byte the bullet and make the implications as clear as we
can.

3. As far as the ordering of variants goes, we really can't have it both
ways. Aside from constraints imposed by prefixing, either

A. Order is NOT significant for variants
   or
B. Order IS significant for variants.

In case A, we can clearly canonicalize to anything we want, so that we get a
unique form. And the simplest way to do that is alphabetical.

In case B, we MUST provide enough structure so that people producing and
matching tags can actually interpret the specification so as to understand
what that order is.

Now, I favor A. I think we do need to correctly clarify the ordering imposed
by prefixes, and beyond that just use alphabetical. I don't think it is
worth the time, and effort, and delay, to try to impose more structure on
variants. More below.

4. Sorry for not responding to Martin H's message last week, but I was
inconvenienced. I think it is an interested approach, but ultimately is not
worth the effort. Our original goal was to have all subtags be in order of
importance. This reflects the fact that most matching implementations will
simply truncate from the end. That is why the script tag is second, not
third. Unfortunately, this model has gotten pretty well screwed up. Given a
tag:

en-CA-fonipa

and resources for

en-US-fonipa
en-CA
en

We will end up matching to the en-CA, which is a really bad match, since the
differences between IPA and the normal orthography completely swamp the
differences based on country code for the vast majority of users. So even if
an orthographic variant is far more important than the country code
variations, it will be discarded first. The right thing would have been a
script/orthographic code like Fipa corresponding to fonipa. Then we would
have had

en-Fipa-CA

and resources for

en-Fipa-US
en-CA
en

We would get a far better match. By a slavish adherence to a notion of
script that is neither required by 15924 nor consistently adhered to (Latf),
rather than looking at the functional and operational aspects of codes,
we've gotten into a muddle with these.

So what am I saying? It is that the variant ordering is already so screwed
up that it is pointless to waste much time on it. The best we can to is
provide a canonical ordering for variants that is at least (α) consistent
with prefixes, and (β) unique. That's the basis for my proposal.

Mark

-- 
Mark

---

From: Addison Phillips <[email-masked]>
Date: Mon, 05 Mar 2007 09:54:08 -0800

I have been opposing the registration of general purpose variants for 
awhile now, not so much because the registrations themselves were 
misguided, but because I recognized that they create problems like this 
that we are ill-equipped to handle. While I supported watering down MUST 
on Prefixes during the "3066bis" era, I thought it would be most prudent 
not to rush out and make a bunch of these to wrestle with.

 > Because we already have the distinction between these in the last
 > version of the spec, and the requirements on them are different, it
 > would be irresponsible of us not to indicate to people that they have
 > to take certain steps to avoid invalid tags.

This isn't quite correct. There is a MAY in Section 2.2.5 that suggests 
variants can be registered without a Prefix. And Section 3.5 goes 
further by saying:

--
Variant registration requests SHOULD include at least one "Prefix" field 
in the registration form.
--

General variants are not defined and their handling was controversial. 
Attempts to registry truly general subtags ("east", "west", etc.) as 
trial balloons were shot down initially.

This canonicalization problem does not exist for traditional, 
Prefix-bearing variants. It only exists when one must mix variants not 
tied to a particular language with ones that are. Thus:

1. We could act to ban *future* registrations of "general purpose" 
variants (by requiring a Prefix). This would be a good move on our 
parts, I think, since they introduce all manner of problems for language 
tag users and BCP 47 implementers.

2. Our problems with ISO 15924 should be handled via another mechanism. 
The likely candidates here would be:

   -- Ask ISO 15924 about adding a "Part II: Codes for script 
variations". Their namespace is enormous and unlikely to ever fill up, 
and they really should manage all sorts of these variations that really 
*should* be scripts.

   -- Add some ABNF mutation of our own to the "script" production for 
"registered scripts" (probably ALPHA DIGIT 2ALPHA or maybe 3ALPHA DIGIT) 
and repent the registration woes at our leisure (I predict many such 
woes: the history of ietf-languages acting quickly and some ISO body 
catching up later is long and ugly).

   -- Tell people to write an extension for transliterations, 
transcriptions, and other script-like mutations.

> B. Order IS significant for variants.

Variant order may not be significant to the meaning of some tags 
(considered as an atomic unit), as understood by humans. But matching 
schemes of all stripes make order significant, even where it is 
inappropriate.

> script that is neither required by 15924 nor consistently adhered to 
> (Latf), rather than looking at the functional and operational aspects of 
> codes, we've gotten into a muddle with these.
> 
> So what am I saying? It is that the variant ordering is already so 
> screwed up that it is pointless to waste much time on it. 

Yay! +1 to that sentiment... when applied to general-purpose variants. 
There *is no problem with traditional variants*.

> The best we 
> can to is provide a canonical ordering for variants that is at least (α) 
> consistent with prefixes, and (β) unique. That's the basis for my proposal.
> 

Yes, but I oppose including it. Let people choose tags wisely. Warn them 
to order the subtags in the best order. And don't register any more 
universal variants.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 5 Mar 2007 10:12:49 -0800

Briefly,

1. I think adding ALPHA 2alphanum DIGIT (or some variant) to fix the
situation would be a great step forward.

2. I think prefixlessness is orthogonal to the issue. As long as one tag
could have both prefixes (which we do not rule out), the ordering problem
arises. And, frankly, I don't see any problem with general prefixes (but
let's not get into that right now).

> Let people choose tags wisely. Warn them
> to order the subtags in the best order.

3. I would agree with the above IF the "best order" were so self-evident
that everyone composing and interpreting those tags knows what it is. But I
see little hope of that happening. And my software is pretty dumb. It needs
to be told explicitly which orderings are the best -- it's not so good at
guessing....

Mark

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 05 Mar 2007 19:24:02 +0100

> Prefix-bearing variants. It only exists when one must mix variants
> not tied to a particular language with ones that are. Thus:
 
> parts, I think, since they introduce all manner of problems for
> language tag users and BCP 47 implementers.

ACK so far.  Keeping the "fonipa" cruft until it's deprecated by a
proper extension (or one of your other proposals).  If there's a
MUST instead of a SHOULD in 4646bis folks wanting something new
in the direction of "fonipa" are anyway forced to figure out a
general solution also deprecating "fonipa" (and the other "fon").

>> So what am I saying? It is that the variant ordering is already
>> so screwed up that it is pointless to waste much time on it.
 
> Yay! +1 to that sentiment... when applied to general-purpose
> variants.  There *is no problem with traditional variants*.

+2  If a new idea isn't ready for an extension folks can still
play with it using a bunch of x-whatever.

Frank

---

From: Addison Phillips <[email-masked]>
Date: Mon, 05 Mar 2007 10:42:51 -0800

> general solution also deprecating "fonipa" (and the other "fon").

Yes, I forgot to call this out explicitly (although note the 
highlighting of "future"). This would work simply enough: put a MUST in 
Section 3.5 (requiring a Prefix in future registrations) and a note in 
Section 2.2.5 near the MAY (that a couple of items have no Prefix, but 
any new registrations will have a Prefix).

Addison

---

From: John Cowan <[email-masked]>
Date: Mon, 5 Mar 2007 13:52:43 -0500

> situation would be a great step forward.

Too late now; that entails changing the syntax.

> >variations". Their namespace is enormous and unlikely to ever fill up,

Actually it is not, because every 4-alpha code has a 3-digit equivalent.
That's why there are so few private-use tags.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
Nobody expects the RESTifarian Inquisition!  Our chief weapon is
surprise ... surprise and tedium  ... tedium and surprise ....
Our two weapons are tedium and surprise ... and ruthless disregard
for unpleasant facts....  Our three weapons are tedium, surprise, and
ruthless disregard ... and an almost fanatical devotion to Roy Fielding....

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 05 Mar 2007 20:26:42 +0100

John Cowan wrote:

>>> Ask ISO 15924 about adding a "Part II: Codes for script variations".
>>> Their namespace is enormous and unlikely to ever fill up,
 
> Actually it is not, because every 4-alpha code has a 3-digit 
> equivalent.  That's why there are so few private-use tags.

They could partition it using a subset of alpha4 codes without
associated numbers.  And I'm tempted to ask who uses this script
enumeration anyway.

Frank

---

From: Addison Phillips <[email-masked]>
Date: Mon, 05 Mar 2007 17:28:09 -0800

> That's why there are so few private-use tags.

The alpha4 namespace can't ever fill up then. Thus the "excess" codes 
can be used for other things (some on the 15924RA might define "things" 
as "garbage") that don't have three-digit equivalents.

Addison

---

From: John Cowan <[email-masked]>
Date: Mon, 5 Mar 2007 21:30:57 -0500

> as "garbage") that don't have three-digit equivalents.

Where's it to stop?  Who gets to allocate (say) the Z??? space?
If it's the RA, they won't want to contaminate the script standard
with nonscripts.  If not, it's just another private-use
area from the standard's viewpoint, and we can use it, but so
can anyone else.

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
I come from under the hill, and under the hills and over the hills my paths
led. And through the air. I am he that walks unseen.  I am the clue-finder,
the web-cutter, the stinging fly. I was chosen for the lucky number.  --Bilbo

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 5 Mar 2007 22:21:20 -0800

Mark Davis wrote:

> 1. I don't know why you thought that I wrote "tags SHOULD always be 
> *created or generated* ", unless you are more color-blind than I am 
> ;-). All of my changes were marked in yellow. All of the other text 
> was from the previous draft, and is not part of my proposed changes.

I quoted the existing text (with *my emphasis*):

"... language tags SHOULD always be *created or generated* in a 
canonical form.  A language tag is in canonical form when:"

to lend context to Mark's proposed change:

"6. Any sequence of variant tags MUST be reordered such that for each 

Sorry if anyone was confused by this.

I'm basically just digging myself deeper and deeper here, so I will back 
off and let whatever happens happen, on whatever time schedule it 
happens.

---

From: John Cowan <[email-masked]>
Date: Tue, 6 Mar 2007 17:52:27 -0500

Mark Davis scripsit:

> I don't think it is too late. We are not talking about changing
> backwards compatibility; all old language tags would continue to
> work. What would happen is that old parsers wouldn't recognize the new
> codes, but the changes that we are considering for the extlang would
> mean that any old parsers wouldn't be able to recognize new codes like
> zh-min anyway.

No, it means that old parsers that used the BNF would recognize the tags
using the new subtags as not well-formed.  One of the stated purposes
of 4646 was to get the syntax stabilized once and for all.  zh-min is
well-formed by 4646 rules, en-0ipa is not.

-- 
John Cowan  [email-masked]   http://ccil.org/~cowan
"The exception proves the rule."  Dimbulbs think: "Your counterexample proves
my theory."  Latin students think "'Probat' means 'tests': the exception puts
the rule to the proof."  But legal historians know it means "Evidence for an
exception is evidence of the existence of a rule in cases not excepted from."

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 6 Mar 2007 13:11:31 -0800

I don't think it is too late. We are not talking about changing backwards
compatibility; all old language tags would continue to work. What would
happen is that old parsers wouldn't recognize the new codes, but the changes
that we are considering for the extlang would mean that any old parsers
wouldn't be able to recognize new codes like zh-min anyway.

I agree with you that ideally we wouldn't need to do this, but the current
use of variants for orthographic differences is just broken, so I think
Addison's suggestion is pretty reasonable.

Mark

---

From: Martin Hosken <[email-masked]>
Date: Thu, 08 Mar 2007 09:27:28 +0700

Dear Mark,

As a technical implementor of the standard ;)

> inconvenienced. I think it is an interested approach, but ultimately
> is not worth the effort. 

It's aim was to resolve the: I don't like your alphabetical order since
it can result in an order that doesn't reflect the meaning of the tag.
What I think I've managed to do with the added complexity and the
requirement to DO AWAY WITH FREE VARIANTS AND TURN THEM INTO SCRIPT
VARIANTS (which is what they are) is to give a canonical order that also
reflects meaning rather than an arbitrary order that is only good for
matching.

> Our original goal was to have all subtags be in order of importance.
> This reflects the fact that most matching implementations will simply
> truncate from the end.

If that is really true then we are dead in the water. I would suspect
that most implementations will compare the whole tag with some
truncating it for buffer length reasons.

> That is why the script tag is second, not third. Unfortunately, this
> model has gotten pretty well screwed up. Given a tag:
>
> en-CA-fonipa
>
> and resources for
>
> en-US-fonipa
> en-CA
> en
>
> We will end up matching to the en-CA, which is a really bad match,
> since the differences between IPA and the normal orthography
> completely swamp the differences based on country code for the vast
> majority of users. So even if an orthographic variant is far more
> important than the country code variations, it will be discarded
> first. The right thing would have been a script/orthographic code like
> Fipa corresponding to fonipa. Then we would have had
>
> en-Fipa-CA
>
> and resources for
>
> en-Fipa-US
> en-CA
> en
>
> (Latf), rather than looking at the functional and operational aspects
> of codes, we've gotten into a muddle with these.

Well a long time ago I did suggest having 4 letter script variants that
could go before the region tag, but that would change the hallowed
regexp slightly and so I was shouted out of court. But it would have
resolved the script interpretation mismatch.

> screwed up that it is pointless to waste much time on it. The best we
> can to is provide a canonical ordering for variants that is at least
> (α) consistent with prefixes, and (β) unique. That's the basis for my
> proposal.

And I'm happy with it. I'd be happy with anything that means I can
create tags in a way that makes them easy to match by non RFC aware
interpreters. After all, who is going to add a RFC 4647 capable tag
matcher to xpath for example?

Yours,
Martin

---

From: John Cowan <[email-masked]>
Date: Wed, 7 Mar 2007 21:41:17 -0500

> truncating it for buffer length reasons.

Not that kind of truncation.  The kind meant here is that most matchers
will say that foo-bar-baz matches foo-bar but not foo-baz, because
foo-bar is a truncation of foo-bar-baz, but foo-baz is not.
(Don't draw any conclusions from the fact that foo, bar, and baz
are all three letters long here.)

-- 
Híggledy-pìggledy / XML programmers            John Cowan
Try to escape those / I-eighteen-N woes;        http://www.ccil.org/~cowan
Incontrovertibly / What we need more of is      [email-masked]
Unicode weenies and / François Yergeaus.