Subject: [Ltru] Great Script Debate "the Next Generation"... (long)

From: Addison Phillips <[email-masked]>
Date: Thu, 05 Oct 2006 11:58:02 -0700

All,

Taking up the gauntlet flung down by John Cowan :-), herewith my 
proposal for fixing Suppress-Scripts

----
First, some recap of the problem. Suppress-Scripts are meant to identify
languages that are written predominantly in a single script. This is to
warn users and implementers not to form language tags using a script
subtag that is usually redundant, for compatibility with tagging
practices prior to RFC 4646.

S-S poses a number of interesting problems.

I've cited previously the registration problems. Mainly the problem here
is that most languages fit the pattern of "wanting" some form of S-S
field. Even languages that might not have a clear relationship to a
specific script subtag (cf. Doug's research on Korean) probably should
not use a script subtag.

In particular, creating accurate values for even the ISO 639-1 and ISO
639-2 set of languages would require significant knowledge about the
current and recent historical writing traditions of each given language,
plus, possibly, some knowledge of public policy and/or potential
suppression or abuse of minority tradition in regard to that language.

S-S indicates that a given language is written predominantly in a
specific script, so the burden of proof for a less common language might
be very difficult to achieve, since the presence of many texts in a
specific script does not "prove the negative", that is, that a
significant body of texts or a specific writing tradition does not exist
that uses a separate script.

The main alternative we've dealt with in the past would be an
"Accept-Script" or "Recommend-Script" approach. That design, which was
not adopted in RFC 4646, involves documenting the known cases in which a
script subtag *should* be used, and, in effect, recommending that
languages that do not have an "A-S" field not use a script subtag except
when indicating a specific difference important within a given group of
information items.

A-S avoids the "proving the negative" problem of S-S. Since it applies
to a much smaller set of languages, it probably requires less
registration overhead. The burden of proof may be just as difficult to
achieve and is encumbered with essentially the same problems that attend
S-S, since assertions about multiple script usage are just as
potentially disruptive as assertions about the "single scriptness" of a
language.

Removing script information from the registry altogether is appealing as
an alternative. As Mark points out, script subtags are entirely
voluntary and entirely valid. The informational nature of the S-S field
is merely to help guide implementers and users to try and do the right
things. If maintenance is a nightmare, why persist in maintaining
somewhat fictional information?

I do think that guidance for users/implementers is a valid goal here. My
experience as an "eminence grise" for language tags over the past couple
of years is that the level of ill-informedness and mythology surrounding
language tags is pretty deep. Anything we can do to help speed proper
implementation of language tags will help.

The problem here is that I think we're miscasting the role of the script
advisory field or fields in the registry. If we only document the "do
not use" case, users and implementers will remain ignorant of what to do
for languages without the S-S field. Having explained the Chinese issue
several times, it's clear to me that many implementers will not stumble
over the right subtags by accident... and certainly not for languages 
such as Serbian, Uzbek, or Azerbaijani.

If we only document the "do use" cases, though, users and implementers
may not notice the warnings against use of scripts elsewhere. Leading to
the problem we initially sought to prevent.

Thus, my proposal for solving the problem:

1. Include the strongest possible warnings about not using script
subtags in 4646bis. This is probably embodied in Karen Broome's
suggested texts.

2. Replace "Suppress-Script" with "Script". If no script field is
supplied, language tags/ranges should still not use a script subtag
unless one is warranted by the information item or request. If the
script field is present and contains a single item, the language is
known to use that script predominantly. If two or more items are
present, the language is commonly written in more than one script. Users 
are advised *not* to use a script subtag unless the language has more 
than one item in the Script field.

Potential issues:

1. zero or one script subtag have the same behavior. Acquiring a second 
script, however, requires extra scrutiny by ietf-languages because it 
changes the potential default behavior for tag formation for that language.

Reactions?

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 5 Oct 2006 13:19:09 -0700

I am in favor of this approach; it would be much more useful, and more
maintainable. I would add that it would be useful information to say that
the order of the scripts is a rough indication of the volume of text in the
language in that script.

Mark

---

From: [email-masked]
Date: Thu, 5 Oct 2006 12:52:16 -0700

Thoughts:

I wonder if in assigning only commonly used scripts to a "script" 
designator -- as opposed to a "suppress-script" designator -- it might be 
assumed that other more obscure scripts not entered into the registry are 
invalid.  (Granted, this problem may exist with today's suppress-script 
field as well, but adding more than one option might present new 
problems.)

If a user had a Somali document written in Osmanya and didn't find Osma as 
a choice for "script" because it's not as commonly used (today), would the 
classifier assume that so-Osma and so-Arab were invalid or deprecated 
under RFC 4646bis? To avoid this confusion, I fear the theoretical Army of 
Charitable Linguists might be required to do the script research to 
populate the registry with additional script variations.

On the other hand, it would be very useful to be able to see all common 
script options for a particular language in the registry when setting up 
systems.

Regards,

Karen Broome
Metadata Systems Designer
Sony Pictures Entertainment
310.244.4384

Addison Phillips <[email-masked]> 
10/05/2006 11:58 AM

To
"'LTRU Working Group'" <[email-masked]>
cc

Subject
[Ltru] Great Script Debate "the Next Generation"... (long)

All,

----

Potential issues:

1. zero or one script subtag have the same behavior. Acquiring a second 
script, however, requires extra scrutiny by ietf-languages because it 
changes the potential default behavior for tag formation for that 
language.

Reactions?

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 05 Oct 2006 23:34:38 +0200

Addison Phillips wrote:

> A-S avoids the "proving the negative" problem of S-S.
[...]
> The burden of proof may be just as difficult to achieve and is
> encumbered with essentially the same problems that attend S-S,
> since assertions about multiple script usage are just as
> potentially disruptive as assertions about the "single
> scriptness" of a language.

I pick the latter, A-S does not avoid the "proving of a negative"
problem.

> Removing script information from the registry altogether is
> appealing as an alternative. As Mark points out, script subtags
> are entirely voluntary and entirely valid. The informational
> nature of the S-S field is merely to help guide implementers
> and users to try and do the right things.

It was and is _not_ "merely informational", it was and is a
sine qua non for the concept of script subtags:

| the script subtag SHOULD be omitted when it adds no
| distinguishing value to the tag or when the primary language
| subtag's record includes a Suppress-Script field listing the
| applicable script subtag.

A SHOULD is normative, not informative.  It's also not a MAY.

> the level of ill-informedness and mythology surrounding
> language tags is pretty deep. Anything we can do to help speed
> proper implementation of language tags will help.

Yes, that's what among others the Suppress-Scripts do.

> If the script field is present and contains a single item, the
> language is known to use that script predominantly.

In other words the same as a Suppress-Script today.

> If two or more items are present, the language is commonly
> written in more than one script.

That might be very near to what CLDR tries.  Excluding cases
like Latn+Arab for tr, or (only) Cyrl for tt, where CLDR might
have different goals.

> Reactions?

That won't change much for the known frr, frs, fy, ngo, jp,
etc. cases.  I don't see that we're entitled to rewrite 4646
from scratch by introducing new fields or modifying existing
fields.  We're also not entitled to return to 3066, otherwise
I'd think about that as another option.

Frank

---

From: John Cowan <[email-masked]>
Date: Sat, 14 Oct 2006 13:49:06 -0400

> Reactions?

<span lang="sco">It's jist awfu complicated.</span>

On reflection I think we should remove all traces of Suppress-Script
from 4646bis and 4645bis as well, and go with text in 2.2.3 like this:

5.  There MUST be at most one script subtag in a language tag.

6.  A script subtag SHOULD be used if the document is written
    in a script other than that normally used for the language
    (i.e. zh-Latn, en-Brai).

7.  A script subtag SHOULD be used if the language of the document
    is customarily written in more than one script (e.g. sr-Cyrl,
    sr-Latn).

8.  A script subtag SHOULD NOT be used if the document being tagged is
    written in the script normally used for the language (e.g. en-Latn
    SHOULD NOT be used).

And leave it at that.

After all, script is one of the most observable properties of a document
(people may debate which language a document is written in, but scarcely
which script it is written in, and even computers can tell which if the
document is Unicoded), so we don't really have to worry about bad
script tags, and this advice will warn people off using them unnecessarily.

-- 
One Word to write them all,             John Cowan <[email-masked]>
  One Access to find them,              http://www.ccil.org/~cowan
One Excel to count them all,
  And thus to Windows bind them.                --Mike Champion

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 14 Oct 2006 11:04:51 -0700

I'm very much in agreement with this approach.

The text needs a few fixes. A language tag can be used to tag content (and
not just documents as you have below), or can be used to query content (in
which case slightly different conditions apply). And the content may not be
written -- and some script tags distinguish unwritten content. And "normally
used" is too weak -- someone might interpret that as >= 50% of the time,
which would mean not using "Hans", for example, since Hans occurs more
frequently than Hant. And using sometimes "customarily" and sometimes
"normally" might lead a reader to think that some distinction is meant
between them.

So here is a suggested rewrite:

6.  A script subtag SHOULD be used when tagging content if the content is
   in a script other than that customarily used for the language
   (i.e. zh-Latn, en-Brai).

7.  A script subtag SHOULD be used when tagging content if the language of
the content
   is customarily written in more than one script (e.g. sr-Cyrl,
   sr-Latn).

8.  A script subtag SHOULD NOT be used if the content being tagged is
written
   in the script customarily used for the vast majority of content in that
language
(e.g. en-Latn SHOULD NOT be used).

Mark

---

From: John Cowan <[email-masked]>
Date: Sat, 14 Oct 2006 14:59:49 -0400

-- 
Híggledy-pìggledy / XML programmers            John Cowan
Try to escape those / I-eighteen-N woes;        http://www.ccil.org/~cowan
Incontrovertibly / What we need more of is      [email-masked]
Unicode weenies and / François Yergeaus.