Subject: [Ltru] Ordering of variants

From: "Mark Davis" <[email-masked]>
Date: Wed, 7 Mar 2007 18:07:34 -0800

Addison wrote:
We disagree here: variant order is always important to matching, even
though it isn't usually that important to tag meaning. Therefore users
need to be aware of the subtag order. I'm on record as favoring
exclusion of future "general purpose" variants to avoid the
canonicalization problem, ...
The order is important. The order that the user puts them in is the
order to use. General purpose variants are the source of the problem and
their elimination would put paid to the issue (language-specific
variants are either mutually exclusive or have Prefix relationships).

I think the exclusion of general purpose variants doesn't solve the problem.
Suppose we have

Prefix: en
Variant: highland
...
Prefix: en
Variant: suburban

A fundamental question that we have to be able to answer when we are
defining identifiers is the question of, well, identity. That is, are two
tags equivalent or not, and if they are not equivalent, what is the
difference in denotion?

We decided against making case or ordering of extension sequences
significant. So we don't have to explain what any differences mean in terms
of denotation. Had we said that case was significant, we would have needed
to explain what exactly the difference was. For example, \s vs \S in Perl is
different, and the difference can be explained. Or in Java, upper and
lowercase are just handled as different letters, so int Int = 3; is valid.
But since case is irrelevant for us, we can canonicalize it away and ignore
in matching.

We have a required order for subtags of language, script, region, variant,
so there is no variation there that we have to provide a difference in
denotation for.

But we just don't have a good story for variants.

If I encounter "en-highland-suburban" and a separate tag
"en-suburban-highland", I have to be able to determine whether they are the
same or different, and if they are different, what is that difference. The
same for any en-avariant-bvariant. My view is, like extensions, like casing,
it is impossible for us to give a consistent, principled account of what the
difference is supposed to be if I rearrange the variant subtags. Given that,
I should be able to canonicalize them both to one unique form; I don't care
what that form is as long as it is unique.

Now, of course, if we have a prefix ordering, then that does impose a
meaning to the ordering. So if we had:

Prefix: en
Variant: highland

Prefix: en-highland
Variant: alpine

Then there is a determinate ordering that we have to have: we have to use
en-highland-alpine.

If we say that there is a difference in meaning between en-avariant-bvariant
and en-bvariant-avariant, but we don't define what that is, then what will
happen is that users of the standard will willy-nilly choose one random
order or another. And our matching of what they choose will be equally
random. That level of indeterminacy does nobody any good.

Those who really want en-avariant-bvariant and en-bvariant-avariant to be
different, need to step up to the plate and spell out what that difference
is supposed to be, otherwise users and developers will just screw up.

Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Addison Phillips <[email-masked]>
Date: Wed, 07 Mar 2007 18:35:19 -0800

> I think the exclusion of general purpose variants doesn't solve the
 > problem. Suppose we have
 >
 > Prefix: en
 > Variant: highland
 > ...
 > Prefix: en
 > Variant: suburban

The issue is avoided because we have this text:

--
Most variants that share a prefix are mutually exclusive. For example, 
the German orthographic variations '1996' and '1901' SHOULD NOT be used 
in the same tag, as they represent the dates of different spelling 
reforms. A variant that can meaningfully be used in combination with 
another variant SHOULD include a 'Prefix' field in its registry record 
that lists that other variant.
--

That is, 'highland' and 'suburban' are probably mutually exclusive. If 
one can reasonably be used in combination with another, it'll have that 
subtag in one of its prefixes. We don't outright forbid the combination, 
but we frown on it here (and elsewhere). Excepting general purpose 
variants, there aren't any current non-mutually-exclusive variants.

 > "en-suburban-highland", I have to be able to determine whether they are
 > the same or different, and if they are different, what is that
 > difference.

Besides the likelihood of it being a poor choice, they are (from at 
least the basic filtering and lookup perspective) different tags. The 
difference may not have any real world significance, but in one case 
'highland' was wider than 'suburban' and in the other, the opposite.

> But we just don't have a good story for variants.

The story, basically, is that: in *theory* you can have an arbitrarily 
long list of variants, each one ever narrower than the previous one. In 
practice you'll likely see only one or maybe two in exceptional 
circumstances. In fact, we show this in Figure 7 in Section 4.3.1 
(Length Considerations):

--
variant1      =  9 (MUST have language as a prefix)
variant2      =  9 (MUST have language-variant1 as a prefix)
--

That is, the minimum buffer recommendation already makes use of the 
unlikeliness of useful long variant sequences. If we want to change 
something to preserve determinacy, we can strengthen the SHOULD in the 
section on variants to match the MUST in 4.3.1.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 08 Mar 2007 16:24:30 +0900

At 11:35 07/03/08, Addison Phillips wrote:

>That is, 'highland' and 'suburban' are probably mutually exclusive. If one can reasonably be used in combination with another, it'll have that subtag in one of its prefixes. We don't outright forbid the combination, but we frown on it here (and elsewhere). Excepting general purpose variants, there aren't any current non-mutually-exclusive variants.

So what if we have, for example for German, two variants that
correspond to something like 'highland' and 'suburban', and these
can be freely combined with orthographic variants?

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Addison Phillips <[email-masked]>
Date: Thu, 08 Mar 2007 07:28:06 -0800

> can be freely combined with orthographic variants?

Then list the orthographic variants among the prefixes. Yes, this is 
potentially ugly, but there aren't *that* many language variants 
registered, nor, for any given language, are there likely to be.

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 8 Mar 2007 08:17:31 -0800

The "we'll never really be faced with this as a practical problem" is a
reasonable answer. If we can get a solution for the orthographies, I'm can
live with that.

Mark

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 08 Mar 2007 18:14:17 +0100

Mark Davis wrote:

> "en-suburban-highland", I have to be able to determine whether they
> are the same or different, and if they are different, what is that
> difference.

The difference should be explained in the registration template of
the variant registered later.  Let's say there is _no_ difference,
and "suburban" was registerd after "highland".  The "suburban" could
offer prefixes "en" and "en-highland" to indicate a preferred order
"en-highland-suburban".

If "suburban" and "highland" are mutually exclusive (like "1901" and
"1996") only the registration template says so, the registry can't...
we should really consider links to the original registration template.

If the preferred order is "en-suburban-highland", but "suburban" is
registered after "highland", then a prefix "en-suburban" added to the
old "highland" could indicate this.

Finally if both orders are fine meaning different things the template
should say so.  And we maybe should have a link to the registration 
template in the registry (?).  Or we should have a directory of these
registration templates.  This affects all variants, all grandfathered 
tags, and a few modifications like Debbie's comment for UK.

Frank

---

From: John Cowan <[email-masked]>
Date: Thu, 8 Mar 2007 12:58:39 -0500

> old "highland" could indicate this.

+1 to all this.

> tags, and a few modifications like Debbie's comment for UK.

IANA maintains a directory of 1766/3066 registrations, and maybe
they should be maintaining one for 4646 registrations too.  As
things now stand, the references, purpose, and such have to be
searched for in the list archives.

-- 
There is no real going back.  Though I          John Cowan
may come to the Shire, it will not seem         [email-masked]
the same; for I shall not be the same.          http://www.ccil.org/~cowan
I am wounded with knife, sting, and tooth,
and a long burden.  Where shall I find rest?           --Frodo