Subject: [Ltru] Re: Preparations for smart matching

From: "Doug Ewell" <[email-masked]>
Date: Sun, 20 Mar 2005 20:33:34 -0800

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

>> For instance, using the examples from Section 2.3,
>> "de-CH-1996" and "en-Latn-US" would be considered
>> preferable to "de" or ("en" or "en-US").
>
> Depends.  When I use de-CH-1996 what I mean is that after one
> decade too much with US-keyboard layouts I forgot all rules
> about &szlig; vs. ss in de, and what I learned in school was
> de-1901, so de-CH-1996 without &szlig; is the best I can use
> without too many stupid errors.

Debbie's premise is that IF you can identify your data correctly using
detailed tags, then use the detailed tag rather than a less specific
one.  Obviously not all data fits neatly into a category like
"de-CH-1996"; you might use some elements that are characteristic of
Swiss German and some that are not, and you might follow some rules of
the spelling reform but not others.  In that case, the rule should
always be "Don't lie with language tags."

> For another example see what your UA did with your article.
> The default for MIME 1.0 text/plain is us-ascii 7bit.  You
> can explicitly say so or omit it, and arguably saying so is
> better.
>
> But your UA claims that it's using utf-8 7bit.  I know that
> this is the same as us-ascii, but my UA doesn't know utf-8
> at all, it's just a piece of software and can't think.
>
> Fortunately they made it very robust (= stupid), with the
> correct result in this case of utf-8 7bit.

I have no idea what this has to do with language tags.  I always change
the encoding of my messages to UTF-8 (except when I forget) and Outlook
Express is probably tagging it as UTF-8.  I don't know what is "7-bit"
about it (UTF-8 is an 8-bit encoding) but I guess if I had any non-ASCII
characters in my message they might be wrapped in a 7-bit encoding of
some sort.  If you care, see what it does with this character: [×]

(I'm sorry to hear that your e-mail UA doesn't know what UTF-8 is, by
the way; the Internet Mail Consortium considers it "deficient and
lacking in standard internationalization capabilities" for that reason.)

> For existing RfC 3066 software en-Latn-US is gibberish.

Well, of course it is.  I'm talking about parsers that are expected to
be able to handle RFC 3066bis tags.

To update what I said on ietf-languages:  If our goal is to ensure that
all future tags can be interpreted by existing RFC 3066 parsers, with no
modification, then the best way to do that is to close this WG right now
and stop registering new RFC 3066 tags.

The RFC 3066bis generative syntax is a superset of the RFC 3066
generative syntax.  It allows things to be generated that cannot be
generated under RFC 3066.  However, it is completely downward
compatible, in that everything valid under RFC 3066 is also valid under
RFC 3066bis (which would not be true if, for example, we got rid of
country codes).

> That's why I want default scripts and a SHOULD NOT for cases
> like en-Latn-*.  The recommendation to tag wisely is a waste
> of time for software, it can't think.  It doesn't know what
> audio data is, and if it tries it could fail miserably for
> captions in video data.

Audio and video data would not be tagged for script in any event.

-Doug Ewell
 Fullerton, California
 http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Sun, 20 Mar 2005 23:49:11 -0500

Doug Ewell scripsit:

> (1) The RFC 3066bis generative syntax is a superset of the RFC 3066
> generated under RFC 3066.

> (2) [E]verything valid under RFC 3066 is also valid under RFC 3066bis
> (which would not be true if, for example, we got rid of country codes).

(3) The RFC 3066bis generative syntax is a subset of the RFC 3066
recognizer syntax: every tag that can be generated under RFC 3066bis
could be registered under RFC 3066.

(I think this point is worth emphasizing.)

-- 
Long-short-short, long-short-short / Dactyls in dimeter,
Verse form with choriambs / (Masculine rhyme):  [email-masked]
One sentence (two stanzas) / Hexasyllabically   http://www.reutershealth.com
Challenges poets who / Don't have the time.     --robison who's at texas dot net

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 21 Mar 2005 08:07:03 +0100

Doug Ewell wrote:

> Debbie's premise is that IF you can identify your data
> correctly using detailed tags, then use the detailed tag
> rather than a less specific one.

Sure, she would say en-Latn-US instead of en-US and maybe be
disappointed when today's software matches it with en instead
of en-US.  Browsers, googlebot, spell checkers, etc. for many
years.

3066bis could avoid this potential problem by saying that
en-Latn-* is NOT RECOMMENDED.  Or ru-Cyrl-*, it's not limited
to Latn.

 [utf-8 7bit vs. us-ascii 7bit]
> I have no idea what this has to do with language tags.

It was only an example for backwards compatibility, my rather
old UA often "helps" me to identify such issues.  Not for this
example, I simply looked into the header.  A clever UA could
try to guess the script based on the charset, for us-ascii I'd
expect it to guess "Latn".  And for utf-8 I'd expect that it
doesn't try to guess.

> I don't know what is "7-bit" about it

Another logic decided that you used only us-ascii, and then a
transfer-encoding is unnecessary.  It was only an example for
"many details are not always consistent", keeping it simple is
often better.  Backwards compatible defaults like us-ascii in
7bit mail or Latn for en are good.

> sorry to hear that your e-mail UA doesn't know what UTF-8 is

It's no big problem, otherwise I'd change the UA.  Generally
Latin-1 works for me, and I also know when that's a lie and I
have to replace it by windows-1252 manually.

> the Internet Mail Consortium considers it "deficient and
> lacking in standard internationalization capabilities"

Only for UAs created or revised _after_ 1998, and that was two
or three years after somebody said "make" for my UA.  And it's
already in the binary, unfortunately disabled for my platform.

Thanks for <http://www.imc.org/mail-i18n.html>, I've added it
to my collection of charset links.  I just checked the final
<http://www.w3.org/TR/charmod/#C049> - it still allows me to
use windows-1252 where necessary for backwards compatibility.
Also added to my link collection.

Your test character was ž, okay, what I've now done is evil.
Don't get me started with charsets, it's worse than ccTLDs ;-)

> Well, of course it is.

Then it should be avoided where possible.  We only have to make
it possible by introducing "default" scripts for the 80% where
it's easy, as you said.

> everything valid under RFC 3066 is also valid under
> RFC 3066bis (which would not be true if, for example, we got
> rid of country codes).

My proposal to use the last official meaning for all ISO 3166-1
country codes as an alias to stable UN numbers is compatible
with RfC 3066.  That past or present countries as defined by
the UN are no good choice for regional differences in languages
is IMHO clear.

Maybe 693-3 will allow me to say "Texas", "Wales", or "Bavaria"
somehow where that's relevant.

> Audio and video data would not be tagged for script in any

Not by humans.  But with software you never know, if there's a
way to get it wrong somebody will implement it, it could be me.

                    Bye, Frank

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 24 Mar 2005 21:58:53 -0800

Debbie Garside <debbie at ictmarketing dot co dot uk> wrote:

> RFC 3066bis seems intent on making
> tags as small as possible (am I wrong? Just my perception)

I do think this is overstating things a bit.  There's a point of view
that tags should be detailed, for accuracy, and another point of view
that tags should be general, for compatibility and to prevent searches
from missing them.  I don't think either side is 100% right or wrong.

-Doug Ewell
 Fullerton, California