Subject: [Ltru] Re: Great Script Debate "the Next Generation"... (long)

From: "Doug Ewell" <[email-masked]>
Date: Thu, 5 Oct 2006 23:06:04 -0700

Mark Davis <mark dot davis at icu dash project dot org> wrote:

> I am in favor of this approach; it would be much more useful, and more 
> maintainable. I would add that it would be useful information to say 
> that the order of the scripts is a rough indication of the volume of 
> text in the language in that script.

Nope.  Several people have already stated that order is not relevant in 
the Registry or in record-jar generally.  I asked about this with regard 
to the relative order of Description fields, and this was the answer I 
got.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 6 Oct 2006 07:55:22 -0700

"Nope" is a bit too simple.

a) we don't say in the spec that the fields in a record must be considered
to be unordered (unless I've missed something).

b) even if that were the case, we could have

   Script: Hans, Hant

===

Zero and One don't mean the same thing in the Script (S) proposal.

S:
Zero means: there isn't any information in the registry about the use of
script.
One means: as far as we know, the language tag is only customarily used with
this script.
>=2 means: as far as we know, the language tag is customarily used with all
of these scripts, and no others.

So this provides more infomation that the structure with the SS, which has:

SS:
Zero means: either there isn't any information in the registry about the use
of the script, or as far as we know, the language tag is customarily used
with multiple scripts (but we're not going to tell you which ones).
One means: as far as we know, the language tag is only customarily used with
this script.

In terms of usage, the recomendations would be:

SS:
Zero: we can't tell you much, but if you know that the script is only
customarily used with a single script, do like the Zero case, and if you
know it is customarily used with more scripts, do like the >=2 case.

One: unless you have a good reason, or your script actually differs from
this one, don't use a script.

>=2: unless you have a good reason, don't use a script that isn't in this
list. And don't use a script at all:
a) in tagging content or in lookup unless it applies (eg written material
only in that script), or
b) in filtering, unless you want to restrict yourself to only one script.

===

Stability. Our guarantee is that the language tags themselves are stable,
not the precise format of the registry. And by stable, that doesn't mean
unchanging, either. We can add valid language tags, and we can broaden the
semantics of existing language tags.

Mark

---

From: John Cowan <[email-masked]>
Date: Fri, 6 Oct 2006 11:26:55 -0400

> b) even if that were the case, we could have

I think we'd have to go with this style (though perhaps without
the commas) if we wanted to impose ordering.

> >=2 means: as far as we know, the language tag is customarily used with 
> >all of these scripts, and no others.

Unfortunately, this interpretation breaks down because of Braille.
There are hundreds of languages that can be written in Braille script,
not one of which is written solely in that script.  So to specify Braille
alone would be misleading, to specify Braille and other scripts would
be as hard as Suppress-Script, and to omit Braille would be to say that
it should not be used with (otherwise) multiscript languages.

> unchanging, either. We can add valid language tags, and we can broaden 
> the semantics of existing language tags.

+1, except to note that existing language tags can be narrowed as
well, provided they are not narrowed too much.  If it is learned
that Upper and Lower Slobbovian are really separate languages, not
just divergent dialects of Slobbovian, we might expect ISO 639-3/RA
to create a macrolanguage; but if it turns out that a few isolated
villages in the little-known North Central district of Gibberistan
speak something distinct enough from Gibberish to warrant being called
a separate language, there will probably simply be a new code for it,
and the code for Gibberish will be left alone.

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
The known is finite, the unknown infinite; intellectually we stand
on an islet in the midst of an illimitable ocean of inexplicability.
Our business in every generation is to reclaim a little more land,
to add something to the extent and the solidity of our possessions.
        --Thomas Henry Huxley

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 6 Oct 2006 08:46:46 -0700

Braille, Zxxx (unwritten), etc. are exceptional cases, and need to be called
out as not being governed by SS.

Mark

---

From: Addison Phillips <[email-masked]>
Date: Fri, 06 Oct 2006 09:02:35 -0700

> it should not be used with (otherwise) multiscript languages.

Braille is an example of a script you would always have to identify. 
That is, it represents *exactly* the sort of "distinguishing value" that 
requires a script subtag.

It is useful to note that *no* languages are "natively written" in 
Braille. There are hundreds of languages that are transcribed into 
Braille, and, while there is no doubt that some--even many--texts are 
authored natively in Braille, there is also no doubt in my mind that no 
language claims it as a primary script (i.e. documents are not 
"customarily" written in it, they are always *exceptionally* written in it).

It's non-listing in the registry means exactly the foregoing: if you 
have a document in Braille, then you, by all rights, really SHOULD tag 
the content as being in Braille, as it will most probably be 
inaccessible to native speakers of the language. Just as if, when I find 
an English document written in the Arabic script, I would naturally feel 
compelled to identify that fact: "en-Arab".

The "two or more" rule for Script would mean, basically: yes, this 
language (say Serbian) is customarily written in the script you are 
using (say Cyrillic). You still need to identify the script because 
there are other common scripts for this particular language.

addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: Peter Constable <[email-masked]>
Date: Fri, 6 Oct 2006 09:53:17 -0700

One problem with scripts is that scenarios fall along one or more continua rather than falling into a few distinct buckets: 

English is 99.99% of the time written in Latin script with certain limited orthographic variations. Arabic is similar. At the other extreme, we know that Hans and Hant are both viable and widely-used written forms for zh. 

Now consider Somali: any new text is written in Latin, but just a few decades ago this was transitional. Or consider Balkan languages: a language might be written in more than one script, with one clearly the majority case though the other can't be disregarded as rare. These and other cases fall somewhere between the two extremes.

For the cases at the two extremes, whether we declare a known script (en uses Latn; zh uses Hans, Hant, Hani) or declare scripts to be suppressed (don't use Latn with en) might not be a huge difference. It's the in-between cases that present the biggest problems. 

Although, even in the cases at the two extremes, we need to consider how to deal with the less common usages: e.g. English in shorthand, Chinese in Bopomofo. Clearly tags for these should have a script tag; it's just not clear what we ought to declare about the languages in the registry. If we're declaring scripts, then for en do we just list Latn? If so, then we're not really declaring all scripts used for that language; we're only declaring commonly-used scripts.

Maybe that's another alternative to consider: we have two data fields that can be used, one to declare scripts in common usage for modern orthographies, and another to declare other scripts that are known to be used (but only in limited, uncommon scenarios). If the former field has only one value, then we can assume it is safe to suppress that.

Just thinking out loud...

Peter Constable

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 7 Oct 2006 00:17:19 -0700

Mark Davis wrote:

> a) we don't say in the spec that the fields in a record must be 
> considered to be unordered (unless I've missed something).

Well, I can't find it either, in RFC 4646 or Addison's record-jar draft 
or in the Art of Unix Programming.  But I know people have been telling 
me for at least a year that the order of Description fields is 
meaningless, and we can't make it meaningful, because:

a)  record-jar says order is meaningless
b)  RFC 4646 saysorder is meaningless
c)  (a) and (b) above

> b) even if that were the case, we could have

You could certainly do this.  I don't know, however, if trying to order 
the scripts in this way is pertinent to the main goal of improving 
script subtag practice is in line.  If the Registry included either of 
these two Script fields for the Mongolian language:

Script: Mong, Cyrl
Script: Cyrl, Mong

it would have exactly the same meaning that you stated: don't use a 
script that isn't in this list.  Trying to order the entries by usage 
statistics would cause controversy, and would be perceived by some as 
expressing a preference for one script over another, which is not our 
job.

> S:
> Zero means: there isn't any information in the registry about the use 
> of script.

Just like the absence of Suppress-Script.  Fine so far.

> One means: as far as we know, the language tag is only customarily 
> used with this script.

Just like the presence of Suppress-Script.  Fine so far.

> >=2 means: as far as we know, the language tag is customarily used 
> >with all of these scripts, and no others.

This is where it gets better: sort of like an Accept-Script for just 
those two (or whatever).

OK, I'm convinced that this solution might be better than S-S.  The 
question is whether it is "better enough" to justify changing the 
mechanism.

Also, we had better try hard (and fail) to poke holes in this mechanism, 
from all perspectives: ease of understanding, usability by tag producers 
and consumers, ease of administration by ietf-languages.  If there are 
flaws in this mechanism, we'd better find out about them and address 
them before making a change to the system, because we (read: "I") don't 
want to end up scrapping yet another of our own inventions in favor of 
something "better."

> Stability. Our guarantee is that the language tags themselves are 
> stable, not the precise format of the registry. And by stable, that 
> doesn't mean unchanging, either. We can add valid language tags, and 
> we can broaden the semantics of existing language tags.

That kind of instability doesn't bother me half as much as inventing a 
feature one year and getting rid of it the next, or creating "deprecated 
at birth" subtags, or creating mechanisms that we don't really want to 
see used, like extensions.