Subject: [Ltru] RE: duel(ing) tags

From: Shawn Steele <[email-masked]>
Date: Wed, 10 Oct 2007 10:43:43 -0700

I was sort of wondering the same thing.

I was thinking that to support mistakes, legacy tags & other randomness it might be best if any matching stuff I wrote recognized that cmn & zh-cmn were interchangeable.  It is unclear to me if data is currently tagged as cmn though, although I know some is tagged as zh-cmn.

- Shawn

------------------------------

Date: Wed, 10 Oct 2007 10:46:18 +0100
From: "Debbie Garside" <[email-masked]>
Subject: RE: [Ltru] Re: Extended language tags
To: <[email-masked]>, "'Doug Ewell'"
        <[email-masked]>,        "'LTRU Working Group'" <[email-masked]>
Message-ID: <04f501c80b22$68debac0$0d00a8c0@CPQ86763045110>

Hi

I may have missed the answer to this question, apologies if I have, but why
not include cwm within the registry and then deprecate in favour of zh-cwm?
This way both are in the registry and the fallback, as described by Mark, is
facilitated. The same could be done for zh-yue and yue and no-nb etc.  Thus
if people tag "incorrectly", with, for example, cwm or yue, applications
would come up with the correct match anyway.

I cannot pretend that I understand the entirety of the current conversations
wrt extended language tags but I do think their inclusion would be useful
for matching purposes.

Best regards

Debbie Garside

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 10 Oct 2007 11:23:48 -0700

Even if we have extlang, I have no doubt that in practice we'll need to
treat "yue" and "zh-yue" as equivalent, and so forth. In addition, in
practice, you'll need to treat "cmn"/"zh-cmn" and "zh" as equivalent.

Mark

On 10/10/07, Shawn Steele <[email-masked]> wrote:
> I was thinking that to support mistakes, legacy tags & other randomness it
> might be best if any matching stuff I wrote recognized that cmn & zh-cmn
> were interchangeable.  It is unclear to me if data is currently tagged as
> cmn though, although I know some is tagged as zh-cmn.
>
> - Shawn
>

-- 
Mark

---

From: Addison Phillips <[email-masked]>
Date: Wed, 10 Oct 2007 13:00:38 -0700

'cmn' is not a valid subtag at present. It is part of the 7000+ 
additional subtags in ISO 639-3 that are being added. Presumably nothing 
is tagged with 'cmn' today.

I definitely am against providing both tagging options.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 10 Oct 2007 13:06:58 -0700

I am definitely against providing both options in the specification as well
-- we should settle on either "ar-arb" or "arb". However, we also must
recognize that, even if we require "ar-arb", in practice we would see a lot
of simple "arb" as input, and implementations would need to deal with it,
typically by equating it to "ar-arb".

Mark

On 10/10/07, Addison Phillips <[email-masked]> wrote:
> > I was thinking that to support mistakes, legacy tags & other randomness
> it might be best if any matching stuff I wrote recognized that cmn & zh-cmn

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 10 Oct 2007 21:10:28 +0100

> I definitely am against providing both tagging options.

But where is the harm in including cmn in the registry and then deprecating
in favour of zh-cmn?  As stated previously, this would catch those who
unwittingly tag with any 639-3 code and are unaware of macrolanguages and
4646bis.

Best regards

Debbie

> -----Original Message-----
> From: Addison Phillips [mailto:[email-masked]]
> Sent: 10 October 2007 21:01
> To: Shawn Steele
> Cc: [email-masked]
> Subject: Re: [Ltru] RE: duel(ing) tags
>
> 'cmn' is not a valid subtag at present. It is part of the
> 7000+ additional subtags in ISO 639-3 that are being added.
> Presumably nothing is tagged with 'cmn' today.
>
> > I was thinking that to support mistakes, legacy tags &
> other randomness it might be best if any matching stuff I
> wrote recognized that cmn & zh-cmn were interchangeable.  It
> is unclear to me if data is currently tagged as cmn though,
> although I know some is tagged as zh-cmn.
> >
> > - Shawn
> >
> > I may have missed the answer to this question, apologies if I have,
> > but why not include cwm within the registry and then
> deprecate in favour of zh-cwm?
> > This way both are in the registry and the fallback, as described by
> > Mark, is facilitated. The same could be done for zh-yue and yue and
> > no-nb etc.  Thus if people tag "incorrectly", with, for
> example, cwm
> > or yue, applications would come up with the correct match anyway.
> >
> > I cannot pretend that I understand the entirety of the current
> > conversations wrt extended language tags but I do think their
> > inclusion would be useful for matching purposes.
> >
> > Best regards
> >
> > Debbie Garside
> >
> >
> >

---

From: Shawn Steele <[email-masked]>
Date: Wed, 10 Oct 2007 13:29:52 -0700

> unwittingly tag with any 639-3 code and are unaware of macrolanguages and

I wasn't suggesting registering both.  Its just my experience that people don't really always pay that much attention, so they might use both.  I was merely suggesting that robust applications might want to recognize both forms, even if one was "illegal".

- Shawn

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 10 Oct 2007 13:35:21 -0700

Right.

On 10/10/07, Shawn Steele <[email-masked]> wrote:
> I wasn't suggesting registering both.  Its just my experience that people
> don't really always pay that much attention, so they might use both.  I was
> merely suggesting that robust applications might want to recognize both
> forms, even if one was "illegal".
>
> - Shawn
>
>

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 10 Oct 2007 21:38:15 +0100

Shawn wrote:

Its just my experience
> that people don't really always pay that much attention, so
> they might use both.  I was merely suggesting that robust
> applications might want to recognize both forms, even if one
> was "illegal".

But if we already know that is what people will do (which is the case) and
if we are to use extended language subtags with the inclusion of 639-3
within the registry why not include a way to catch the "illegal" tags that
is easy for applications to implement?   I see more use than harm in using
extended language subtags and deprecating the individual 639-3 codes to
facilitate matching.  This can be documented within 4646bis.

Best regards

Debbie

> From: Shawn Steele [mailto:[email-masked]]
> Sent: 10 October 2007 21:30
> To: [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: RE: [Ltru] RE: duel(ing) tags
>
> > But where is the harm in including cmn in the registry and then
> > deprecating in favour of zh-cmn?  As stated previously, this would
> > catch those who unwittingly tag with any 639-3 code and are
> unaware of
> > macrolanguages and 4646bis.
>
> I wasn't suggesting registering both.  Its just my experience
> that people don't really always pay that much attention, so
> they might use both.  I was merely suggesting that robust
> applications might want to recognize both forms, even if one
> was "illegal".
>
> - Shawn
>
>
>

---

From: Addison Phillips <[email-masked]>
Date: Wed, 10 Oct 2007 15:45:18 -0700

Read the spec. Items are not permitted to exist in both language and 
extlang. Furthermore, 'deprecated' does not mean illegal. In fact, you'd 
be required to support it as a valid tagging option.

Whether my software chooses to deal with bad tags by equating them to 
something else is up to me (and how it does so), is my problem. Making 
such mappings obligatory (which, as I'll point out, requires registry 
access, something we have historically tried hard to avoid as an 
implementation requirement), I oppose.

Addison

---

From: Addison Phillips <[email-masked]>
Date: Wed, 10 Oct 2007 16:06:17 -0700

> why not include a way to catch the "illegal" tags that
 > is easy for applications to implement?

Ur... that's what the registry does. I can catch the fact that "cmn-HK" 
is not valid and it's really easy to implement. What you mean is that I 
should be able to infer that the user really means "zh-cmn-HK", which I 
might be able to do in my software (but which I am free *not* to do at 
present).

After all, plenty of content is labeled with the language tag "english". 
Implementations are free to infer that this really means "en". However, 
they are not obliged to do so.

 > I see more use than harm in using

I sense perhaps some confusion here?

We are not talking about deprecated any ISO 639-3 codes. We are talking 
about how ISO 639-3 codes are incorporated into the registry. 'cmn' is a 
perfectly valid ISO 639-3 code. We have several choices for how to 
include it in the registry:

- as a primary language subtag (making tags like "cmn-Hant-HK" valid)
- as an extlang of 'zh' (making tags like "zh-cmn-Hant-HK" valid)
   // note: notice that I chose an un-grandfathered tag there
- as a primary language subtag that is deprecated (making tags like 
"cmn-Hant-HK" valid, but deprecated)
- not at all (none of the tags in this example would be valid then)

Let me note that "special case" handling and mapping in matching 
algorithms is incredibly painful to build and maintain. If you choose to 
canonicalize tags and ranges before processing, this isn't so bad. But 
if you are a "good citizen" and preserve the original form of the 
tags/ranges for later consumption, one must implement quite complex 
logic for doing what's supposed to be quite simple matching.

Furthermore, this requires, absolutely, access to information above and 
beyond that embodied in the tags/ranges themselves. That is a wholesale 
departure from current practice in matching. While my own 
implementations might very well use such information (and more) to "do 
the right thing", that is *my* application and you might not care for my 
particular implementation choices (they may not be right for your 
application). To the extent that we can describe behaviors that do not 
require registry access, I am content with either primary language 
subtags or extlangs (but NOT both for the same language).

Remember: a deprecated subtag is still a valid subtag.

Addison

PS> Note that the word "valid" in the foregoing refers explicitly to the 
terminology defined in Section 2.2.9 "Classes of Conformance" in the 
current draft of RFC 4646bis (and by intention to the same section in 
RFC 4646).

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 11 Oct 2007 00:44:03 +0100

Hi Addison

Comments in line...

>  > why not include a way to catch the "illegal" tags that  >
> is not valid and it's really easy to implement. What you mean
> is that I should be able to infer that the user really means
> "zh-cmn-HK", which I might be able to do in my software (but
> which I am free *not* to do at present).

That is exactly what I mean.

> Implementations are free to infer that this really means
> "en". However, they are not obliged to do so.
>
>  > extended language subtags and deprecating the individual
> > 639-3 codes to  > facilitate matching.  This can be
> documented within 4646bis.
>

I don't think I am confused at all... Having read your email :-)

> We are not talking about deprecated any ISO 639-3 codes. We
> are talking about how ISO 639-3 codes are incorporated into
> the registry.

That is exactly what I am talking about.

>'cmn' is a perfectly valid ISO 639-3 code.

Agreed

> We
> have several choices for how to include it in the registry:
>
> - as a primary language subtag that is deprecated (making
> tags like "cmn-Hant-HK" valid, but deprecated)

Indeed.  I understood this perfectly well.  I was attempting to show how cmn
could be incorporated within the registry as an extlang of 'zh' whilst also
covering the back door by registering 'cmn' and then immediately deprecating
'cmn' in favour of 'zh-cmn'.  You already have several deprecated subtags
what is the problem  with a few more to deal with the macrolanguage
scenario?

> Let me note that "special case" handling and mapping in
> matching algorithms is incredibly painful to build and
> maintain. If you choose to canonicalize tags and ranges
> before processing, this isn't so bad. But if you are a "good
> citizen" and preserve the original form of the tags/ranges
> for later consumption, one must implement quite complex logic
> for doing what's supposed to be quite simple matching.

What I have proposed is fairly simple logic for matching purposes.

> Furthermore, this requires, absolutely, access to information
> above and beyond that embodied in the tags/ranges themselves.
> That is a wholesale departure from current practice in
> matching. While my own implementations might very well use
> such information (and more) to "do the right thing", that is
> *my* application and you might not care for my particular
> implementation choices (they may not be right for your
> application). To the extent that we can describe behaviors
> that do not require registry access, I am content with either
> primary language subtags or extlangs (but NOT both for the
> same language).

I am not proposing both for the same language as such.  I am proposing to
deprecate one in favour of the other.

> Remember: a deprecated subtag is still a valid subtag.

Yes, I know.  Perhaps you would like to call it something else?  Retired as
part of macrolanguage?  Replaced by?

The whole point is to have both within the registry; one current, one not
current but both valid for matching.  By deprecating cmn to zh-cmn matching
is facilitated.  Text within RFC4646bis can be quite simply written to
explain this.

Now I am going to shut up about it as I am obviously banging my head against
a brick wall.  But I really cannot see what the problem is; other than
people will tag with both zh-cmn and cmn whether we like it or not and I
think it is far better to deal with that scenario within the registry rather
than leave it to application designers who may not be as conversant with
RFC4646bis as you (and maybe) me :-)

Best regards

Debbie

> terminology defined in Section 2.2.9 "Classes of Conformance"
> in the current draft of RFC 4646bis (and by intention to the
> same section in RFC 4646).
>
> Debbie Garside wrote:
> > Shawn wrote:
> >
> > Its just my experience
> >> that people don't really always pay that much attention, so they
> >> might use both.  I was merely suggesting that robust applications
> >> might want to recognize both forms, even if one was "illegal".
> >
> > But if we already know that is what people will do (which
> is the case)
> > and if we are to use extended language subtags with the
> inclusion of
> > 639-3 within the registry why not include a way to catch
> the "illegal" tags that
> > extended language subtags and deprecating the individual
> 639-3 codes
> > to facilitate matching.  This can be documented within 4646bis.
> >
> > Best regards
> >
> > Debbie
> >
> >
> >> I wasn't suggesting registering both.  Its just my experience that
> >> people don't really always pay that much attention, so
> they might use
> >> both.  I was merely suggesting that robust applications
> might want to
> >> recognize both forms, even if one was "illegal".
> >>
> >> - Shawn
> >>
> >>
> >>
> >
> >
> >
> >
>
> --
> Addison Phillips

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 11 Oct 2007 08:07:05 +0200

Debbie Garside wrote:

> if we already know that is what people will do (which is the case) and
> within the registry why not include a way to catch the "illegal" tags
> that is easy for applications to implement?

Your proposal is a cute hack.
 
If we're *_not_* using the extlang-construct we'd deprecate zh-cmn with
preferred value cmn.  In both cases we'd end up with about 200 deprecated
aliases, don't we ?

 Frank

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 11 Oct 2007 07:54:33 +0100

Frank wrote:

> If we're *_not_* using the extlang-construct we'd deprecate
> zh-cmn with preferred value cmn.  In both cases we'd end up
> with about 200 deprecated aliases, don't we ?

Exactly :-)

Debbie

> From: Frank Ellermann [mailto:[email-masked]]
> Sent: 11 October 2007 07:07
> To: [email-masked]
> Subject: [Ltru] Re: RE: duel(ing) tags
>
> Debbie Garside wrote:
>
> If we're *_not_* using the extlang-construct we'd deprecate
> zh-cmn with preferred value cmn.  In both cases we'd end up
> with about 200 deprecated aliases, don't we ?
>
>  Frank
>
>
>

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 11 Oct 2007 09:48:41 +0100

> with about 200 deprecated aliases, don't we ?

In fact, it is more preferable to deprecate zh-cmn in favour of cmn.  This
way application designers need not include matching for extlang if they
don't want to;  they can purely match to primary language subtags.

Best regards

Debbie

---

From: Addison Phillips <[email-masked]>
Date: Thu, 11 Oct 2007 10:42:07 -0700

>> with about 200 deprecated aliases, don't we ?

In the former case (deprecating "zh-cmn" and associates), we would be 
deprecating *existing* grandfathered tags.

In the latter case (deprecating "cmn" and friends), we would be *adding* 
items into the registry in a deprecated state. This is, in part, what 
bothers me.

There is no need to make these codes valid as primary language tags if 
they serve no purpose there. Making it legal for people to create 
undesirable codes "just because they will anyway" will contribute to 
poor tagging practice and a proliferation of tags synonymous in meaning 
but not in form. We already know this is bad: why encourage it?

Addison

---

From: Addison Phillips <[email-masked]>
Date: Thu, 11 Oct 2007 10:42:07 -0700

Addison

---

From: "Debbie Garside" <[email-masked]>
Date: Fri, 12 Oct 2007 11:45:15 +0100

Hi Addison

>Making it legal
> for people to create undesirable codes "just because they
> will anyway" will contribute to poor tagging practice and a
> proliferation of tags synonymous in meaning but not in form.
> We already know this is bad: why encourage it?

I don't think we would be encouraging it.  Users of the RFC or BCP who have
good knowledge of the system will still tag correctly but users who have
little or no knowledge will, for sure, tag incorrectly anyway.  I think
prevention is better than cure.

If we do not do this it will result in application designers having to
incorporate it within their systems in an ad hoc way (cure).  Will this
facilitate interchange with other systems?  Not unless it is standardized
(prevention) IMHO.

Maybe I am wrong, but I can see good use for extlang (probably because ISO
639-6 is hierarchical and works in the same way :-)).

Best regards

Debbie

> Sent: 11 October 2007 18:42
> To: [email-masked]
> Cc: [email-masked]; [email-masked]
> Subject: Re: [Ltru] Re: RE: duel(ing) tags
>
> Debbie Garside wrote:
> >
> >> If we're *_not_* using the extlang-construct we'd deprecate zh-cmn
> >> with preferred value cmn.  In both cases we'd end up with
> about 200
> >> deprecated aliases, don't we ?
> >
> > Exactly :-)
> >
>
> In the former case (deprecating "zh-cmn" and associates), we
> would be deprecating *existing* grandfathered tags.
>
> In the latter case (deprecating "cmn" and friends), we would
> be *adding* items into the registry in a deprecated state.
> This is, in part, what bothers me.
>
> There is no need to make these codes valid as primary
> language tags if they serve no purpose there. Making it legal
> for people to create undesirable codes "just because they
> will anyway" will contribute to poor tagging practice and a
> proliferation of tags synonymous in meaning but not in form.
> We already know this is bad: why encourage it?
>
> Addison
>
> --
> Addison Phillips