Subject: [Ltru] Re: Great Script Debate "the Next Generation"...

From: "Doug Ewell" <[email-masked]>
Date: Sun, 15 Oct 2006 00:46:15 -0700

Warning: micro-editing suggestions follow.

Mark Davis <mark dot davis at icu dash project dot org> wrote:

> 5.  There MUST be at most one script subtag in a language tag.

I think this could be too easily read as "There MUST be one script 
subtag..." and would prefer turning the sentence around to something 
like "There MUST NOT be more than one script subtag..."

> 6.  A script subtag SHOULD be used when tagging content if the content 
> is in a script other than that customarily used for the language (i.e. 
> zh-Latn, en-Brai).

Would this sentence mean the same, but be more concise, if the words "if 
the content is" were replaced by "written"?

Also, the examples should be marked "e.g." instead of "i.e."

> 7.  A script subtag SHOULD be used when tagging content if the 
> language of the content is customarily written in more than one script 
> (e.g. sr-Cyrl, sr-Latn).

Again for conciseness, if we don't need "of the content" in the previous 
item, we may not need it here.

> 8.  A script subtag SHOULD NOT be used if the content being tagged is 
> written in the script customarily used for the vast majority of 
> content in that language (e.g. en-Latn SHOULD NOT be used).

How about "... if the content being tagged is not written (e.g. spoken 
or sung) or if it is written in the script customarily used..."

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Sun, 15 Oct 2006 13:25:39 +0100

Doug wrote:

> I think this could be too easily read as "There MUST be one 
> script subtag..." and would prefer turning the sentence 
> around to something like "There MUST NOT be more than one 
> script subtag..."

I agree with Doug re Mark's wording but would suggest the following:

----

6. No more than one script subtag may be used in a language tag.

----

Or 

----

6. Script sub tags are limited to one per language tag.

----

> How about "... if the content being tagged is not written 
> (e.g. spoken or sung) or if it is written in the script 
> customarily used..."

Suggestion:

----

8. A script subtag SHOULD NOT be used where the majority of content is
customarily written in one script (e.g. en-Latn SHOULD NOT be used) or where
the item being tagged is not written (e.g. spoken content).

----

Debbie

> -----Original Message-----
> From: Doug Ewell [mailto:[email-masked]] 
> Sent: 15 October 2006 08:46
> To: LTRU Working Group
> Subject: [Ltru] Re: Great Script Debate "the Next Generation"...
> 
> I think this could be too easily read as "There MUST be one 
> script subtag..." and would prefer turning the sentence 
> around to something like "There MUST NOT be more than one 
> script subtag..."
> 
> Would this sentence mean the same, but be more concise, if 
> the words "if the content is" were replaced by "written"?
> 
> Again for conciseness, if we don't need "of the content" in 
> the previous item, we may not need it here.
> 
> How about "... if the content being tagged is not written 
> (e.g. spoken or sung) or if it is written in the script 
> customarily used..."
> 
> --
> Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  
> UTN #14 http://users.adelphia.net/~dewell/

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 15 Oct 2006 10:48:09 -0700

> like "There MUST NOT be more than one script subtag..."

agreed

> the content is" were replaced by "written"?

I was trying to avoid the use of written -- since the script Zxxx could be
being used for unwritten material (see below).

agreed

> item, we may not need it here.

Actually, for precision, I think we should put it in both places.

> or sung) or if it is written in the script customarily used..."

Because of edge cases, I think we need to settle on the usage first. Let's
suppose I have the following cases of content, where we follow SHOULDs, and
show what it could be tagged with (with varying degrees of specificity):

1. American English, written in Latin.
en
en-US

2. American English, written in Cyrillic (don't laugh; a collegue of mine
has a printed ad in this.)
en
en-US
en-Cyrl
en-Cyrl-US

3. American English, only spoken -- no written content
en
en-US
en-Zxxx
en-Zxxx-US

4. American English in a video, some parts spoken, and with Cyrllic
subtitles. In protocols limited to a single language, this is now limited
to:

en
en-US

If a protocol allows tagging with multiple languages, then it could have any
of the combinations of #2 & #3.

--

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 15 Oct 2006 16:20:13 -0700

Mark Davis wrote:

>>> 6.  A script subtag SHOULD be used when tagging content if the 
>>> content is in a script other than that customarily used for the 
>>> language (i.e. zh-Latn, en-Brai).
>>
>> Would this sentence mean the same, but be more concise, if the words 
>> "if the content is" were replaced by "written"?
>
> I was trying to avoid the use of written -- since the script Zxxx 
> could be being used for unwritten material (see below).

That assumes that your proposed interpretation of "Zxxx" to mean 
"non-written content" is agreed to by the list, and will be part of 
4646bis.  You made this argument in [1] but I don't remember any other 
discussion of it.  As a reminder, the Description field for "Zxxx" is 
"Code for unwritten languages" and I claimed that this is not quite the 
same thing.

[1] http://www1.ietf.org/mail-archive/web/ltru/current/msg05971.html

>>> language of the content is customarily written in more than one 
>>> script (e.g. sr-Cyrl, sr-Latn).
>>
>> Again for conciseness, if we don't need "of the content" in the 
>> previous item, we may not need it here.
>

No objection.

> 2. American English, written in Cyrillic (don't laugh; a collegue of 
> mine has a printed ad in this.)

I've seen it too, on TV.

> 3. American English, only spoken -- no written content

See above; if there is agreement to do this, then 4646bis should 
probably include an example of this usage, and/or the entry for "Zxxx" 
in the Registry should have a comment added.  To me this is a semantic 
stretch almost equivalent to "sign languages used in the U.S. = American 
Sign Language."

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 19 Oct 2006 17:07:29 +0200

> content in that language (e.g. en-Latn SHOULD NOT be used). 

There may be exceptions. While the following is about locale naming rather
than language tagging (ready-made) text content, those are closely related,
and a language tag can well be used to select the (almost) same-named locale.
 
While 'th' currently has a Suppress-Script of 'Thai', and would still be covered
by the above "SHOULD" if we remove "Suppress-Script", 'th' and 'th-Thai'
(or as a locale name 'th_Thai'), in that generated digits (generated from some
computation, or some database access, or similar) may be formatted differently
for presentation: 'th' would imply the use of the digits 0-9 (which I understand
is common in modern Thai), while 'th-Thai' (or rather a 'th_Thai' locale) would
imply that they are presented using the Thai digits (U+0E50-U+0E59).
 
Similarly, Arabic mixes the use of 0-9 and Arabic-Indic digits, and one may
want to have separate locales for presenting computed number/date values
using 0-9 (ar) or using Arabic-Indic digits (ar-Arab).
 
It would be possible to use variants for this, but that seems needless. Or would
you consider such use as described above of script subtags as inappropriate?
 
        /kent k

---

From: John Cowan <[email-masked]>
Date: Thu, 19 Oct 2006 12:30:01 -0400

Kent Karlsson scripsit:

> While 'th' currently has a Suppress-Script of 'Thai', and would still
> be covered by the above "SHOULD" if we remove "Suppress-Script", 'th'
> and 'th-Thai' (or as a locale name 'th_Thai'), in that generated digits
> (generated from some computation, or some database access, or similar)
> may be formatted differently for presentation: 'th' would imply the
> use of the digits 0-9 (which I understand is common in modern Thai),
> while 'th-Thai' (or rather a 'th_Thai' locale) would imply that they
> are presented using the Thai digits (U+0E50-U+0E59).

I think this is a very hacky trick, and extensions of it would be disastrous.
All the Indic scripts have their own digit runs: some are used routinely,
others only in historical documents.

> It would be possible to use variants for this, but that seems
> needless. Or would you consider such use as described above of script
> subtags as inappropriate?

Variants, or failing to tag at all when both styles are understood by the
user community, seem ideal for this sort of very language-specific thing.

-- 
He played King Lear as though           John Cowan <[email-masked]>
someone had played the ace.             http://www.ccil.org/~cowan
        --Eugene Field

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 19 Oct 2006 19:54:50 +0200

> It would be really impractical to depend on any consistent distinction
> ever being drawn between th-Thai and th, either as languages or as
> locales (in the CLDR sense, which is essentially the same as
> language). The only difference would be that in matching, th could
> match th-Latn (a romanization), for example. But that is going to be
> extremely rare cases.

Well, th_Latn (and I'm still considering locale names) would have all
data associated with it in the Latin script.

Whereas th would have all data but the LDML/<nativeZeroDigit> in Thai,
and th_Thai would differ from th only in <nativeZeroDigit>.

And I'm looking at an actual, and rather widespread, system that already
has a th_TH locale as well as a th_TH_TH locale, with just this difference,
and the only difference, but (currently) with a worse naming.

I'm not sure why John finds using th_Thai, and similar for some other
languages, disastrous, nor why variants would be ideal. Indeed, using
variants for this seems *very* unpractical, especially if one prefers a
restriction to (for language tags) registered variants. I'm not going to
push for registering such (a) variant(s), and I don't think one should be
registered, nor that ietf-languages group would accept such a variant.
An (explicit, -x-)  private use variant for this would be out of the
question, I'd say.

		/kent k

---

From: Addison Phillips <[email-masked]>
Date: Thu, 19 Oct 2006 11:07:50 -0700

> you consider such use as described above of script subtags as inappropriate?

I don't understand why the "thai" script subtag would imply this at all, 
even in a locale identifier. Minor variations in user preference such as 
this really shouldn't be inferred from a script subtag. Similarly the 
numeric shaping in Arabic locales (or in the Indic languages, for that 
matter).

For example, I wouldn't expect a script subtag to control whether the 
date format in a Chinese locale is all-numeric or presents ideographs. 
These are properly API options or minor variations within a locale 
hierarchy.

I *do* agree that there are cases in which, even when normally it would 
be "suppressed", the script subtag should be visible and this could even 
apply to locale identifiers (although I can't think of a good example 
off the top of my head). But I don't think this is a valid example of 
such a case.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 19 Oct 2006 21:16:04 +0200

> even in a locale identifier.

You could see it as knd of emphasis: "really all Thai, digits too".

> Minor variations in user preference such as 
> numeric shaping in Arabic locales (or in the Indic languages, 
> for that matter).
> 

As for using ideographs, that use seems mostly comparable to using
number spellout, like 120 -> "one-two-zero" or "one hundred and twenty".

But looking at the same system as I mentioned before, there is a "ko_KR"
locale and a "ko_KR_HANI" locale, which differs in just that: The number
*spellout* (for datetimes, currently) in ko_KR is in Sino-Korean Hangul,
whereas in ko_KR_HANI (yes, the current name has the script code after
the territory code, but that is a separate issue; they were named before
RFC 4646) the *spellout* is in Hanja ideographs. Both have non-spellout
digits as 0-9 (both for ordinary numbers and datetimes). But ko_KR_HANI
has all data in Hanja, not just for number spellout. ko_KR has, of course,
all data in Hangul.

That th_TH_TH is named that way instead of th_TH_THAI is more
an accident than design, I think. I did not name these locales.

> off the top of my head). But I don't think this is a valid example of 

There are already such cases, see above.

		/kent k

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 19 Oct 2006 09:06:08 -0700

Mark

---

From: Martin Duerst <[email-masked]>
Date: Sat, 21 Oct 2006 10:27:55 +0900

I agree with Mark and others that it's a bad idea to let the presence
or absence of a script tag indicate the useage of a certain kind of
digits. This is not what the script tag was/is intended for, and it's
very brittle.

Regards,    Martin.

>> you consider such use as described above of script subtags as inappropriate?

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Kent Karlsson" <[email-masked]>
Date: Sat, 21 Oct 2006 14:25:41 +0200

So how would you suggest dealing with the (existing!) locale
currently named th_TH_TH if one were to rename it more
in the spirit of RFC4646 language tags? Granted, locale
names are not necessarily bound by how IETF language tags
are formed. Almost all locales in CLDR are named according to
IETF language tags, with the exceptions hy_AM_REVISED,
aa_ER_SAAHO, and el_POLYTONI, for which I do think variant 
language subtags would be warranted, en_US_POSIX, and root
which are indeed special cases.  th_TH_TH isn't a CLDR
locale, but I don't want to make another exception for it in
naming, CLDR or not. There is still a need to support
th_TH_TH (*and* th_TH, but th_TH_TH will likely be more
used than th_TH for the applications I have in mind, which
use generated digits quite a lot), but I'd rather see its name
mapped to something more up-to-date. And I would guess
there is a similar need for other languages, Indic ones in
particular.

As I mentioned, registering a *language* variant subtag for
this (digit preference for **generated** decimal digits, not
(generated) spellout digits) would be inappropriate, I think
(I would not support such a registration, at least). So would
using a private use subtag.

	/kent k

> From: Martin Duerst [mailto:[email-masked]] 
> Sent: Saturday, October 21, 2006 3:28 AM
> To: Mark Davis; Kent Karlsson
> Cc: LTRU Working Group
> Subject: Re: [Ltru] Re: Great Script Debate "the Next Generation"...
> 
> 
> >> and a language tag can well be used to select the (almost) 
> same-named
> >> locale.
> >>
> >> While 'th' currently has a Suppress-Script of 'Thai', and 
> would still be
> >> covered
> >> (or as a locale name 'th_Thai'), in that generated digits 
> (generated from
> >> some
> >> for presentation: 'th' would imply the use of the digits 
> 0-9 (which I
> >> understand
> >> is common in modern Thai), while 'th-Thai' (or rather a 
> 'th_Thai' locale)
> >> would
> >> It would be possible to use variants for this, but that 
> seems needless. Or
> >> would

---

From: John Cowan <[email-masked]>
Date: Sat, 21 Oct 2006 11:41:24 -0400

Kent Karlsson scripsit:

> (I would not support such a registration, at least). 

I conclude, then, that preferences are not languages, and that
this question is simply out of scope for RFC 4646.

-- 
What has four pairs of pants, lives             John Cowan
in Philadelphia, and it never rains             http://www.ccil.org/~cowan
but it pours?                                   [email-masked]
        --Rufus T. Firefly

---

From: Addison Phillips <[email-masked]>
Date: Sat, 21 Oct 2006 09:21:44 -0700

> in the spirit of RFC4646 language tags?

The "obvious" first response: this is what private use is for!

   th-TH-x-{your subtags here}

I note that no short identifier is going to be able to manage all of the 
variations possible *within* a locale. Locales don't provide for every 
permutation of a user's preferences and shouldn't try to. To indicate 
digit shaping choices at the locale (or language tag) level strikes me 
as somewhat overly specific.

 > (I would not support such a registration, at least).

A case could be made for a language tag "extension" specific to locale 
identification. This would allow whatever additional stuff necessary to 
our information systems to be identified without interfering with the 
language tag proper. Specific variations, such as collation, calendar, 
and, well, digit shaping, aren't really "locale differences" anyway, 
they are tailorings within a particular locale. These could be 
identified in an extension sequence quite comfortably without impinging 
on the language tag or what I call the "gross locale identifier" (the 
"th-TH" part).

The use of variants in POSIX locales shows the limitations of the 
tri-partite locale identifier. Variants in particular have a checkered 
history of being annoying and accumulating junk. Using tailorings is a 
much more scalable, manageable approach to this problem. *And* is keeps 
the purity of the language tag/locale identifer!

In fact, I'd go so far as to say I'm surprised CLDR hasn't produced an 
extension for this purpose or that one hasn't been proposed for W3C's 
LTLI. I think CLDR in particular has stretched about as far as it can 
against the boundaries of what can be done with variants.

Before I pooh-pooh variant registration, I'll point out that it is, in 
many ways, difficult to divide a locale from a language. The computer 
generated part of a text is just as much a part of the text as the 
static strings are. We all look oddly on a sentence that mixes one with 
another:

What's this? en-CH?: "There were 1'347 errors in the last 13,75 seconds."

So it is possible that some variations of this nature might merit 
registration as language tag variants, although, as you say, I'm also 
pretty sure I can't think of a case where it would be appropriate today 
and I'm pretty sure I wouldn't support most proposals to try one.

Addison

---

From: "Kent Karlsson" <[email-masked]>
Date: Sat, 21 Oct 2006 21:26:28 +0200

>    th-TH-x-{your subtags here}

Definitely not. The use I have in mind is as *public* a use as
one can get.

> as somewhat overly specific.

Maybe, maybe not. But firstly, those specifying which digit rendering
to use will in most cases not be programmers, so a programming
API for selecting such options are out of the question. Further, selecting
options for this comes very close to selecting another locale. So I'm
not surprised that in this case it is currently done by locale selection.

> on the language tag or what I call the "gross locale identifier" (the 

In the case of th, the -TH part is really overspecifying. An advantange
of th/th-Thai over th/th-<somevariant>, is that adding (the superfluous)
TH in the first case makes no real difference (prefix matching), whereas
in the second case it comes between the th and the <somevariant>
interfering with prefix matching.

> against the boundaries of what can be done with variants.

If the CLDR project or W3C applies for an extension tag for this and
similar matters, I would not oppose that. But I still don't find it to be
the niftiest of solutions, since an extraneous -TH still interferes
with prefix matching. Extended matching isn't necessarily available.

	/kent k

> static strings are. We all look oddly on a sentence that 
> mixes one with another:
>

---

From: Peter Constable <[email-masked]>
Date: Sun, 22 Oct 2006 22:31:44 -0700

IMO, this is clearly a case where the distinction between a locale ID
and a language tag that I've talked about for several years is critical:
the use of digit substitution in certain processes on a particular
computer system has nothing whatsoever to do with describing the
linguistic attributes of content. IMO, digit substitution as part of a
CLDR locale is out of scope for RFC 4646 language tags.

Peter

-----Original Message-----
From: Kent Karlsson [mailto:[email-masked]] 
Sent: Saturday, October 21, 2006 5:26 AM
To: 'Martin Duerst'; 'Mark Davis'
Cc: 'LTRU Working Group'
Subject: RE: [Ltru] Re: Great Script Debate "the Next Generation"...

	/kent k

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 24 Oct 2006 07:14:58 -0700

Once again, I have to correct some misperceptions. There is no "digit
substitution as part of a
CLDR locale".

Before people talk about CLDR it might be a good idea to look at it first.
And Randy's right -- this is getting way off topic.

Mark

On 10/22/06, Peter Constable <[email-masked]> wrote:
> From: Kent Karlsson [mailto: [email-masked]]
> > >> > > content in that language ( e.g. en-Latn SHOULD NOT be used).
> > >>
> > #-#-#  http://www.sw.it.aoyama.ac.jp
> > mailto: [email-masked]
> >
> >
> >