Subject: [Ltru] Re: Preparations for smart matching

From: "Doug Ewell" <[email-masked]>
Date: Sun, 20 Mar 2005 22:20:18 -0800

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

> Out of curiosity, what's the idea of ISO 639-3 for the
> reserved "extlang" ?  Is it something in the general direction
> of my "region codes in language tags considered harmful" idea ?

It is a way of identifying more languages.  It does not replace region
subtags in cases where the essential language is the same.  There are
codes for things like "Liberian English," which differs dramatically
from Standard English, but there are not separate codes for en-US and
en-GB and en-AU and en-ZA.

>> I think you missed my point.  Trying to determine the
>> "default" script and perform matching based on this
>> assessment is probably *not* a good idea for those languages
>> that are customarily written in more than one script
>
> No, that point is clear.  We don't want a mad tag reviewer who
> sees no problem in deciding what the default script is where
> it's _not_ obvious.  In that case we prefer no default at all.
> Add TINW where necessary.

So the problem is reduced to defining a "default" script where that is
considered "obvious" -- best to be conservative here -- and establishing
a mechanism to indicate the default script in the registry, a matching
algorithm that will use this information to dike out "redundant" script
subtags, and a recommendation in the draft that users not use any script
subtag explicitly that matches the default.

Does anyone else besides Frank agree with this, or think it bears
investigating?  I'd like to hear someone else besides just Frank and me.

>> and probably unnecessary for those that are not.
>
> That's what I think is incorrect, see the parallel thread with
> Debbie with issues like legacy software, audio data, limited
> space in RfC 2231 header fields, etc.

Debbie is saying we should encode as much information as possible.  You
are saying that we should not encode the script if it is assumed to be
the "default."  These are fundamentally opposite viewpoints.

Let's not talk about audio data and script subtags in the same breath.

> I'd also like to get as much info as possible, but above all I
> like correct data, no guesses, and backwards compatible data
> where possible.

Collapsing "en-Latn" to "en" is very much a matter of employing
guesswork.  You are guessing that users who request "en" don't want the
very small amount of content that exists in any script other than Latin,
and will know how to request "en-Dsrt" or "en-Shaw" or "en-Brai" if they
want that.

> If 3066bis makes it this year I'm confident
> that compatibility issues with RfC 3066 begin to be irrelevant
> after 2038 with its signed time_t 32 bits issue.

I guess this means "RFC 3066bis has serious compatibility issues with
RFC 3066."  Can you enumerate these, please, in a bullet-point list?

>> Section 2.3, items 1 and 2 tell the user to avoid subtags
>> that don't add information.
>
> It's not always the user who tags data, often it's software.
> As user you would probably never say "utf-8 7bit".  We need
> defaults for software implementors, not for ordinary users.

"User" in software engineering does not have to mean a human.  It could
be a software process.

> For an I-D and IANA I'd prefer a format as in the IANA charset
> registry.  Maybe with "%%" lines instead of empty lines as the
> terminator for individual entries.  I forgot the name for this
> format and who proposed <CRLF>%%<CRLF> instead of <CRLF>CRLF>.

John Cowan suggested the "record-jar" format.  I could certainly put
together a record-jar version of the registry if desired.  Is anyone
else interested in seeing this?

> Yes, probably they are both important.  If a client supports
> both en-misc and en-boont but not en-misc-boont let's hope
> that it "knows" the best alternative or offers a choice.

As I said: it is very difficult to tell, simply from the presence of a
variant subtag, whether it represents the most important or least
important information for the user.

If I request "de" content, say over the Web, I definitely want to
receive content in both "de-1901" and "de-1996", not just one or the
other.  However, if I request "zh" I may not be able to do anything with
"zh-hakka".

> E.g a user could configure somewhere that he only knows Latn.
> Or that he's really interested in misc, not Latn.  Or that he
> loves en-boont independent of the script.

This is all part of the "good" matching strategy I described.  You could
specify "en-Latn" or "en-Misc" (not an actual script, of course) or
"en-boont", and get content in whatever combination you like.  All that
is needed is for the matching module to compare the langtag of the
content with that of your request, decompose the two into subtags, and
compare the subtags.  If you had something like this, you wouldn't need
"en-Latn" to be collapsed to "en" or anything else.

>> Who are we to say she can't read Ogham?
>
> Of course we won't say this.  We only want existing RfC 3066
> software to do the right thing for "Latn".  This software has
> no idea what "Latn" is.

If you declare 'Latn' to be the default script for 'en', and use that
information to collapse the tag "en-Latn" to "en", then you remove the
user's ability to get "en-Ogam" content unless she specifically requests
it.  This essentially says, "Since you requested English, I'll assume
you can only read it in Latin script, and I won't send you English
content in any other script."

-Doug Ewell
 Fullerton, California
 http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 20 Mar 2005 22:48:21 -0800

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

>>> Step 2:  Deprecated ISO 639 codes
>
> How about a special rule "skip jw" ?  The normative references
> are ISO 639-1:2002 and ISO 639-2:1998, that could justify to
> omit an old typo fixed 1988 (?) or 2001 (?).

If you don't mind, I'd like to phrase this as a question for the group,
instead of simply continung to say "such-and-so could be done," which
doesn't seem to generate any WG feedback.

QUESTION:  Should the rules for populating the registry be changed to
exclude ISO 639 codes deprecated before a certain date?  If so, what
should that date be?

PRO:  This might reduce user confusion, and would leave the door open
for ISO 639 to reassign these codes (which they have never done).

CON:  Many reference sources actually encouraged users to continue using
the old codes 'in', 'iw', and 'he' for years after they were deprecated,
on the basis that old software would be more likely to recognize the old
codes.

A side note: Please don't take the fact that I am presenting this issue
as a question for the group as evidence that I support the idea.

> Your second source (codechanges.html) says that it's currently
> updated.  Apparently they forgot the "-" for a "-jw", but you
> said that your script ignores the "-".

Yes, they did forget the hyphen to indicate that "jw" is now deprecated.

BTW, for those who found my e-mail message on this subject difficult to
read, I've turned it into a Web page (including errata).  Eventually
this will include steps to populate the entire registry.  I'm waiting
for feedback on the language part before finishing the other parts.  The
address is:

http://users.adelphia.net/~dewell/init-registry.html

-Doug Ewell
 Fullerton, California

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 21 Mar 2005 10:09:08 +0100

Doug Ewell wrote:

> Debbie is saying we should encode as much information as
> possible.  You are saying that we should not encode the
> script if it is assumed to be the "default."  These are
> fundamentally opposite viewpoints.

Yes, because I dare not declare all existing RfC 3066 software
to be obsolete soon.  For a completely new scheme that's by
definition incompatible I'd opt for Debbie's idea.

> You are guessing that users who request "en" don't want the

I'm guessing that users who request "en" want the RfC 3066 idea
of "en", any en-?? plus a few registerd en- like en-boont.

> and will know how to request "en-Dsrt" or "en-Shaw" or
> "en-Brai" if they want that.

Yes, if they want that they can say so.  I think you've just
found another design problem.  Even a default "Latn" does not
allow to say "en-Latn" and to get any old "en" minus the new
"en-Brai" etc.

> I guess this means "RFC 3066bis has serious compatibility
> issues with RFC 3066."  Can you enumerate these, please, in
> a bullet-point list?

1 - old software won't recognize en-Brai-LR, but maybe knows
    en-LR (you said that it's very different from en)

2 - old content is tagged as en-US etc., We can guess that it's
    en-Latn-US etc. (if we define default scripts), but that
    guess would be incorrect if it's actually en-Brai-US etc.

For "old" read "almost all" until a new 3066bis is published.

> If I request "de" content, say over the Web, I definitely
> want to receive content in both "de-1901" and "de-1996", not
> just one or the other.

But not necessarily de-Brai, so there's definitely a problem.
So far I thought that a "default script" would help, but now
you confused me, is a "default script" actually worse ?

Assuming that it is worse, then we'd need Debbie's idea, we
SHOULD say de-Latn in new content, and pray that all old pages
are updated.  Hell, that's impossible, somewhere I lost it.

> This essentially says, "Since you requested English, I'll
> assume you can only read it in Latin script, and I won't
> send you English content in any other script."

Bad.  Could I try again with en-Zyyy ?  Bye, Frank

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 21 Mar 2005 10:55:55 -0000

Frank Ellerman: wrote

"Assuming that it is worse, then we'd need Debbie's idea, we SHOULD say
de-Latn in new content, and pray that all old pages are updated.  Hell,
that's impossible, somewhere I lost it." 

What I said about detail was really about variant subtags but it is OK to
apply it everywhere...consistency is also the name of the game... as it also
assists the end user which is my primary focus.

I hate to throw a spanner in the works here but still on the subject of
detail... Exactly how much detail can be gleaned from a script code...
Should we perhaps be looking in even finer detail and at extensibility
issues e.g. 
en-Character_Set-US
At least you would end up with something useful, forward looking and
extendable... :-)

Debbie

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Frank Ellermann
Sent: 21 March 2005 09:09
To: [email-masked]
Subject: [Ltru] Re: Preparations for smart matching

Doug Ewell wrote:

> Debbie is saying we should encode as much information as possible.  
> You are saying that we should not encode the script if it is assumed 
> to be the "default."  These are fundamentally opposite viewpoints.

Yes, because I dare not declare all existing RfC 3066 software to be
obsolete soon.  For a completely new scheme that's by definition
incompatible I'd opt for Debbie's idea.

> You are guessing that users who request "en" don't want the very small 
> amount of content that exists in any script other than Latin,

I'm guessing that users who request "en" want the RfC 3066 idea of "en", any
en-?? plus a few registerd en- like en-boont.

> and will know how to request "en-Dsrt" or "en-Shaw" or "en-Brai" if 
> they want that.

Yes, if they want that they can say so.  I think you've just found another
design problem.  Even a default "Latn" does not allow to say "en-Latn" and
to get any old "en" minus the new "en-Brai" etc.

> other.

But not necessarily de-Brai, so there's definitely a problem.
So far I thought that a "default script" would help, but now you confused
me, is a "default script" actually worse ?

Assuming that it is worse, then we'd need Debbie's idea, we SHOULD say
de-Latn in new content, and pray that all old pages are updated.  Hell,
that's impossible, somewhere I lost it.

> This essentially says, "Since you requested English, I'll assume you 
> can only read it in Latin script, and I won't send you English content 
> in any other script."

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 21 Mar 2005 09:10:44 -0800

> on the basis that old software would be more likely to recognize the old

The safest and most robust policy is allow all codes that have ever been
permitted by the source standards. My reasons are that
(a) because of the instability of such sources, it was indeed a reasonable
tactic to take use the first defined code for a given semantic, and that
tactic was in widespread use, and
(b) because 3066 and 1766 did not take into account the instability of the
source standards, it was never clear which codes were in fact valid at any
point in time for use in 3066 or 1766, so there was no guidance as to which
code to use, or with what semantics.

However, because of the reuse of codes for unrelated entities by the source
standards, we do have to impose some order in those cases where there is a
conflict. For that, we have had in the drafts the use of a cut-off date for
deciding which semantic to use in case of a conflict, such as with 'CS'.  In
that particular case there is sufficient consensus to make that after the
redefinition of CS to mean "Serbia and Montenegro": a date of January 1 2005
would capture that consensus. It would be safest not to put it at date of
publication, simply because that take the risk (probably small, but
unfortunately real) that the source standards would redefine a code after
the draft has been discussed and approved, but before publication.

‎Mark

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 22 Mar 2005 10:36:28 +0100

Mark Davis wrote:
 
> The safest and most robust policy is allow all codes that
> have ever been permitted by the source standards.

IBTD, a "get rid of old nonsense" strategy for a definition
of "old" = before RfC 1766 was published allows to capture
the _actual_ state of the relevant standards.

Nobody needs an old 639-typo like "jw" in the Internet, it's
a zombie like UNICODE-1-1-UTF-7 (yes, I got some mails with
this beast in 2003 and 2004)

> (a) because of the instability of such sources

True.  The tactic to get stability by a "frozen copy" is not
necessarily convincing, at least not for the region codes.

> we do have to impose some order in those cases where there
> is a conflict.

Or to accept the instability and offer a stable alternative:
3166-1 as it is before 3166-1 as it was (instable), stable
version UN M49 as it is or was.  3166-3 as guide to populate
some canonical values (in addition to UN M49).

> It would be safest not to put it at date of publication,
> simply because that take the risk (probably small, but
> unfortunately real) that the source standards would redefine
> a code after the draft has been discussed and approved, but
> before publication.

Then let's always take the state of the art at the date of
the creation of the future draft registry I-Ds based on the
future registry draft I-Ds.
                           Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 22 Mar 2005 11:05:47 +0100

Debbie Garside wrote:

> consistency is also the name of the game... as it also

I'm that end user, I use things like lang="en-GB-OED" or
xml:lang="de-CH-1996".  Not often, generally I prefer a
simple lang="en" or xml:lang="de".

No problem with RfC 3066 for the wrong reasons, but still
no problem is no problem, good enough for an end user.

Later with 3066bis I _could_ add -Latn everywhere.  Or I
could try a stunt like adding -Latn only in the xml:lang
but not in the lang, if XHTML allows it.  Probably it does
not allow it, but it's a very bad sign that I'd check this
before dropping the horrible idea.

> Should we perhaps be looking in even finer detail and at
> extensibility issues e.g.
> en-Character_Set-US

RfC 3066 and 3066bis are for the Internet (incl. WWW), and
there you generally have a charset defined elsewhere, not in
a language tag.

The worst cases I have on my pages are links to plain text
documents (actually scripts), and there I use something like:

href="kex/german.kex"
type="text/plain"
charset="PC-Multilingual-850+euro"

No hreflang= in this case because source and target are both
"en", and the hotspot text for a potential lang= is also "en".

I'm too lazy to find it now, but I'm sure that I have some
links with type=, charset=, lang=, and hreflag=.  That I don't
use xml:lang= in a body of XHTML documents is an idiosyncrasy.

There's a certain point where I think that Martin and his I18N
cohorts should leave me alone, a redundant xml:lang= is ugly,
I won't have it more than once per document.

> At least you would end up with something useful, forward
> looking and extendable... :-)

Where do you miss a charset in language tags ?  Bye, Frank

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 24 Mar 2005 22:06:13 -0800

Mark Davis <mark dot davis at jtcsv dot com> wrote:

> The safest and most robust policy is allow all codes that have ever
> been permitted by the source standards. My reasons are that
> (a) because of the instability of such sources, it was indeed a
> reasonable tactic to take use the first defined code for a given
> semantic, and that tactic was in widespread use, and
> (b) because 3066 and 1766 did not take into account the instability of
> the source standards, it was never clear which codes were in fact
> valid at any point in time for use in 3066 or 1766, so there was no
> guidance as to which code to use, or with what semantics.

OK, so far we have Mark taking this stand, and Frank taking the stand
that a cutoff date should be set, such as the publication of RFC 1766,
such that language and country codes deprecated before that date would
not be allowed in language tags.  And I agree with Mark.

Anyone else?  This is only 3 voices out of at least 15 active
participants in the group, which is not what anyone would call
consensus.

> However, because of the reuse of codes for unrelated entities by the
> source standards, we do have to impose some order in those cases where
> there is a conflict. For that, we have had in the drafts the use of a
> cut-off date for deciding which semantic to use in case of a conflict,
> such as with 'CS'.  In that particular case there is sufficient
> consensus to make that after the redefinition of CS to mean "Serbia
> and Montenegro": a date of January 1 2005 would capture that
> consensus. It would be safest not to put it at date of publication,
> simply because that take the risk (probably small, but unfortunately
> real) that the source standards would redefine a code after the draft
> has been discussed and approved, but before publication.

I wonder if we could solicit some sort of temporary moratorium from the
MAs that they would refrain from doing this at least until we can get
the RFCs out, presumably by the end of 2005.

-Doug Ewell
 Fullerton, California