Subject: [Ltru] Re: Preparations for smart matching

From: "Doug Ewell" <[email-masked]>
Date: Mon, 21 Mar 2005 08:30:41 -0800

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

>> Debbie is saying we should encode as much information as
>> possible.  You are saying that we should not encode the
>> script if it is assumed to be the "default."  These are
>> fundamentally opposite viewpoints.
>
> Yes, because I dare not declare all existing RfC 3066 software
> to be obsolete soon.  For a completely new scheme that's by
> definition incompatible I'd opt for Debbie's idea.

The vast majority of tags created under RFC 3066bis will NOT contain
script subtags.  They are there for when someone deems them necessary.

>> You are guessing that users who request "en" don't want the
>> very small amount of content that exists in any script other
>> than Latin,
>
> I'm guessing that users who request "en" want the RfC 3066 idea
> of "en", any en-?? plus a few registerd en- like en-boont.

That is not what your suggestion does, though.  It collapses "en-Latn"
to "en" in matching, on the basis of Latin being considered the "default
script" for English.  This means either (a) a search for "en" will
return only "en-Latn", which is not compatible with the RFC 3066
interpretation, or (b) a search for "en-Latn" will return all "en",
which is not what the user asked for.

(Bear in mind, throughout this post, that "user" could mean "software"
or "user agent."  It's software-engineering shorthand.)

>> and will know how to request "en-Dsrt" or "en-Shaw" or
>> "en-Brai" if they want that.
>
> Yes, if they want that they can say so.  I think you've just
> found another design problem.  Even a default "Latn" does not
> allow to say "en-Latn" and to get any old "en" minus the new
> "en-Brai" etc.

Well, I think that's a design problem with the whole "default script"
idea.  And I think asking for "all scripts except Braille" is probably
out of scope for this project.

>> I guess this means "RFC 3066bis has serious compatibility
>> issues with RFC 3066."  Can you enumerate these, please, in
>> a bullet-point list?
>
> 1 - old software won't recognize en-Brai-LR, but maybe knows
>     en-LR (you said that it's very different from en)

If the user requests "en-Brai-LR", she is making a very particular
request that could not have been fulfilled under RFC 3066 in any
circumstance (unless "en-Brai-LR" were explicitly registered.  Since
this wouldn't have worked under RFC 3066, it seems odd to consider it an
incompatibility, unless you are saying that the whole script-subtag idea
should be deleted (which would take away one of the main purposes of RFC
3066bis).

> 2 - old content is tagged as en-US etc., We can guess that it's
>     en-Latn-US etc. (if we define default scripts), but that
>     guess would be incorrect if it's actually en-Brai-US etc.

Content tagged as "en-US" cannot be assumed to be in any particular
script, although Latin would be a good guess.

Content tagged as "en-Latn-US" or "en-Brai-US" can be assumed to be in
Latin or Braille respectively.

>> If I request "de" content, say over the Web, I definitely
>> want to receive content in both "de-1901" and "de-1996", not
>> just one or the other.
>
> But not necessarily de-Brai, so there's definitely a problem.
> So far I thought that a "default script" would help, but now
> you confused me, is a "default script" actually worse ?

I think it is.  It assumes certain information that I don't think is
safe to assume, and it makes certain types of requests (like "English in
any script imaginable") impossible (or at least counterintuitive and
non-3066-compatible; see Zyyy example below).

A better solution would be to perform matching based on subtags.  That
way, content can be marked as "en" or "en-US" or "en-Latn-US", and the
user can request any of those, and matching will take place as
appropriate.

> Assuming that it is worse, then we'd need Debbie's idea, we
> SHOULD say de-Latn in new content, and pray that all old pages
> are updated.  Hell, that's impossible, somewhere I lost it.

Nobody proposes that old data be "updated."

>> This essentially says, "Since you requested English, I'll
>> assume you can only read it in Latin script, and I won't
>> send you English content in any other script."
>
> Bad.  Could I try again with en-Zyyy ?  Bye, Frank

OK, you have a point.  There is a subtag 'Zyyy' which means "Code for
undetermined script."  That would allow users to request "English in any
script," but in a way that is incompatible with RFC 3066 -- which I know
is important to you.

Under RFC 3066:
    "en" means English in any script
    "en-Latn" is not registered, so not allowed
    "en-Zyyy" is not registered, so not allowed

Under RFC 3066bis, current draft:
    "en" means English in any script
    "en-Latn" means English in Latin script
    "en-Zyyy" means English in any script

Under RFC 3066bis with your "default script" idea:
    "en" means English in Latin script (default)
    "en-Latn" means English in Latin script (explicit)
    "en-Zyyy" means English in any script

-Doug Ewell
 Fullerton, California
 http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 23 Mar 2005 11:20:23 +0100

Doug Ewell wrote:

> The vast majority of tags created under RFC 3066bis will NOT
> contain script subtags.  They are there for when someone
> deems them necessary.

And how am I supposed to decide this without a default for my
pages ?  I have some "en" pages and one "en-GB-OED", but I've
no en-Ogam.

Apparently you're saying that there are defaults, but you do
not want to document them.  If somebody tries to implement it
he probably hardwires some cases like en-Latn, but ignores
other cases like fy-Latn, because he has no idea what fy is.

> I think asking for "all scripts except Braille" is probably

Yes, but "only Braille" and "only Latin" are relevant.  The
former works, the latter doesn't (for languages where "Latn"
is the undocumented default).

> If the user requests "en-Brai-LR", she is making a very
> particular request that could not have been fulfilled under
> RFC 3066 in any circumstance

RfC 3066 has en-LR, and that includes any future en-Brai-LR.

> it seems odd to consider it an incompatibility

You could also consider it as a case of tough luck if the user
doesn't find existing en-Brai-LR content looking for en-Brai-LR
only because it was tagged as en-LR under 1766 / 3066.

In that case I'd say that any "old" en-LR is probably "Latn",
so not finding it as en-Brai-LR is a feature and no bug.  But
it's less funny if en-Latn-LR doesn't find en-LR.

> Content tagged as "en-US" cannot be assumed to be in any
> particular script, although Latin would be a good guess.

And who tells me what a good guess is for languages I've never
before heard of ?  The publisher of the documents should know
what a good guess is, but he's not the implementor of the used
software.  There are numerous languages in your list where I'm
lost, I couldn't implement a "guess" for these languages.

What I could do, if you offer good guesses (= defaults), is to
implement a ??-Zyyy plan B.

> Nobody proposes that old data be "updated."

I'd consider it, after all I really offer no Brai or Ogam, so
users looking for these scripts shouldn't be forced to check
my pages.
                        Bye, Frank