Subject: [Ltru] Re: Suppress-Script batch 1

From: "Doug Ewell" <[email-masked]>
Date: Wed, 27 Sep 2006 11:06:50 -0700

(Cross-posted to ietf-languages, since it really belongs there.)

Addison Phillips <addison at yahoo dash inc dot com> wrote:

> Here's my thought:
>
>   -1 to all of them
>
> I still believe that Suppress-Script was the wrong solution to the 
> script subtag problem. Here we are seeing *exactly* the kinds of 
> problems with it that I predicted, which is that it is very difficult 
> indeed to make a complete, comprehensive, and verifiably correct list.

We can't put that particular toothpaste back into that particular tube. 
By creating an initial set of 126 Suppress-Script entries, I'm afraid 
the stage is set to investigate the remaining RFC 3066-era languages, 
and either add Suppress-Scripts or not.

> Furthermore, the possibility of abuse is clearly present. I see 'tt' 
> on the list below, despite someone's vehement belief that Tatar 
> speakers wish to write using the Latin script and are being 
> suppressed. Suppress-Script would aid that, if that case turns out to 
> be true.

This is a good observation, and it's why John urged everyone to review 
his list.  For the kind of bulk update that John and others have in 
mind, we (ietf-languages) would probably have to raise the bar for 
acceptance even higher, something like "Language A written in Script B 
would be just plain bizarre."

Last March I began a small research project to determine whether Korean 
should have a Suppress-Script of Hangul.  I read books, scanned 
Korean-language newspapers published in Seoul and in Los Angeles, 
checked numerous Web sites, talked to native Korean speakers in person 
and by e-mail.  The result was that Hangul is, in fact, used an 
overwhelming proportion of the time to write modern Korean -- a 
colleague at work actually used the word "overwhelming" -- but there is 
still a certain, tiny, regular pattern of usage of Han (hanja) in 
scholarly works and newspapers.  "Regular" was the key here.  So I 
concluded that a Suppress-Script for Korean actually would NOT be a good 
idea, which is not what I expected to find.

One important fact to keep in mind is that Suppress-Script values may be 
not only added, but also removed (Section 3.4, item 7).  If we make a 
mistake, we can go back and fix it.

> Since we didn't do that [Accept-Script], I think we should only create 
> Suppress-Script fields when there is clear and compelling reason to do 
> so... and I think that a clear and compelling reason for me would be 
> someone requesting it on purpose.

Like all other requests, though, it needs to be confirmed that such a 
request relates to actual usage, and is not a "test" of the system or an 
attempt to "prove" something about the system or its participants or 
related SDOs.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Addison Phillips <[email-masked]>
Date: Wed, 27 Sep 2006 12:11:01 -0700

> (Cross-posted to ietf-languages, since it really belongs there.)

It does NOT belong there. I object in the strongest possible terms to my 
being cross-posted, especially in an edited form.

> and either add Suppress-Scripts or not.

Yeah, I lost that battle and don't really want to revisit it... except 
that I am horrified to note that my predictions about the unwieldy 
nature of Suppress-Script all came true. I do not believe that 
ietf-languages is the right forum for deciding how languages are written 
customarily and would prefer it if only the absolutely necessary 
Suppress-Script fields were created.

I also believe that, since we appear to be willing to consider trashing 
any amount of RFC 4646 in a quest to have a better file format (XML) or 
a different reviewing mechanism (LSRB), why not fix everything to our 
current taste? I don't want to be alone in standing on the barricades 
fighting extraneous change. :-)

> would be just plain bizarre."

It is exactly why Suppress-Script is an awful solution to the problem of 
preventing unwanted script subtags. Too much information is required 
about too many languages. Most languages are customarily written in only 
one script, but determining which ones that is true of and what the 
script subtag should be is moderately difficult, especially for 
languages of limited diffusion.

By contrast, determining which small number of languages are customarily 
written in more than one script (Note Well: the word 'customarily'!) is 
a far easier task.

> idea, which is not what I expected to find.

It took a lot of work to arrive at that conclusion, no? Why are these 
other languages different?

> attempt to "prove" something about the system or its participants or 

This may be difficult to evaluate. For all I know, 'tt' is, in fact, 
customarily written in Cyrillic and the person proposing Latin script is 
an outlying opinion. I have no way of evaluating it without finding a 
large number of documents written in Tatar and examining their script.

I've said enough. I think this an awful idea, but no one else seems to 
see the difficulties involved. So I'm going to quit raising my sour 
grapes up for general viewing. On the other hand, I note that attempts 
to create this informative field have already demonstrated some 
workability problems.

So, let's leave it at:

   -1 for proposing these Suppress-Script fields on ietf-languages, 
because I don't believe there is enough compelling information to 
support them.

When they are proposed, you can count on my emitting a quiet -1 for each 
of them. Consensus, after all, is not unanimity.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 27 Sep 2006 13:03:29 -0700

I also agree that suppress script was a bad idea. Better would have been an
informative field indicating that "this language is customarily written in
multiple scripts"; that would provide the indication that a script might be
necessary.

(Notice that in lookup matching -- our principal concern -- we end up always
logically having a "default script" even for languages customarily written
in multiple scripts -- that is the script we will use for content that
matches the case where the language tag doesn't contain a script. For that
matter, we also always have a "default region" in that sense; the content is
appropriate for the region that is most populous for that language.)

Mark

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 27 Sep 2006 21:04:40 +0100

I have to say, I am mostly in agreement with Addison.

I haven't time to go into details on this now, but I have done extensive
research on a number of languages and their writing systems.  It is not as
easy as it sounds and really needs thorough research or the Registry is
going to be in trouble.

A few weeks ago I corrected a Unicode document that was displaying two
separate languages of the same name, spoken in different regions with
scripts designated.  The script information was correct for one of the
languages only. This type of error highlights the muddle that can be created
by "official" documentation compiled without adequate research. On this
occasion a simple human error could be easily corrected.  This is not the
case within the Registry.

I do realise that nothing is ever 100% accurate, and certainly linguistics
is a field where there are a number of differing opinions within various
aspects; no one of them being necessarily correct. 

But I also think that if we are to consider seriously supporting
suppress-script within the Registry a thorough methodology should be devised
in order to deal with this. As within many aspects of business today, we can
design out the error rating insofar as is reasonably possible.  

However, I do not think that research of this nature is within the remit of
LTRU or IETF-languages. I know suppress-script is there now and I have no
suggestions to make to repair this decision other than to only add
suppress-script on request and when presented with "thorough" supporting
research; "thorough" should be defined in RFC4646bis.

Best regards

Debbie Garside

> -----Original Message-----
> From: Addison Phillips [mailto:[email-masked]] 
> Sent: 27 September 2006 20:11
> To: Doug Ewell
> Cc: LTRU Working Group
> Subject: [Ltru] Re: Suppress-Script batch 1
> 
> 
> 
> Doug Ewell wrote:
> It does NOT belong there. I object in the strongest possible 
> terms to my being cross-posted, especially in an edited form.
> 
> 
> >

---

From: "Doug Ewell" <[email-masked]>
Date: Wed, 27 Sep 2006 13:16:16 -0700

Addison Phillips <addison at yahoo dash inc dot com> wrote:

> It does NOT belong there. I object in the strongest possible terms to 
> my being cross-posted, especially in an edited form.

OK, I apologize.  I thought ietf-languages was where decisions were made 
about specific items in the Registry, and LTRU was where the governing 
documents and policy decisions about them were discussed.  I guess this 
fell on the policy side.

> ietf-languages is the right forum for deciding how languages are 
> written customarily and would prefer it if only the absolutely 
> necessary Suppress-Script fields were created.

I didn't want Suppress-Script, Require-Script, Suggest-Script, or 
Gently-Nudge-In-The-Drection-Of-Script.  I thought the guiding principle 
was supposed to be "Tag content wisely."

> I also believe that, since we appear to be willing to consider 
> trashing any amount of RFC 4646 in a quest to have a better file 
> format (XML) or a different reviewing mechanism (LSRB), why not fix 
> everything to our current taste? I don't want to be alone in standing 
> on the barricades fighting extraneous change. :-)

The only reason I supported XML was that the General Manager of IANA 
reported they were moving to it.

The LSRB basically would codify what we do now -- Michael reviews 
subtags, I prepare paperwork for him, Harald moderates the list.  It 
would allow life to go on in the event Michael is on one of his extended 
trips.

You're not alone in your frustration -- remember how I feel about 
extension subtags and the group's reluctance to consider using them for 
transcriptions or any other purpose.

> It is exactly why Suppress-Script is an awful solution to the problem 
> of preventing unwanted script subtags. Too much information is 
> required about too many languages. Most languages are customarily 
> written in only one script, but determining which ones that is true of 
> and what the script subtag should be is moderately difficult, 
> especially for languages of limited diffusion.

No argument.  As I said from the outset, S-S -- more than any other 
feature of RFC 4646 -- makes it look like we are trying to be the 
ultimate arbiters of language usage, instead of a body that gathers the 
research of others and makes it usable in a regularized fashion on the 
Internet, which is the body I joined.

> By contrast, determining which small number of languages are 
> customarily written in more than one script (Note Well: the word 
> 'customarily'!) is a far easier task.

Even that carries the potential for controversy.  Is Santali 
"customarily" written in Ol Chiki?  That probably depends on who you 
ask.

> It took a lot of work to arrive at that [Korean] conclusion, no? Why 
> are these other languages different?

They aren't, necessarily.  Some might be less work, some more.  This is 
one reason I don't just starting ticking boxes when a list like John's 
comes up, like I'm voting for an All-Star team.  I don't personally have 
the expertise to make most of those calls, and they should only be made 
by those who do.

> to create this informative field have already demonstrated some 

S-S provides a long-term solution for a world that continues to use 
remove-from-right prefix matching and unnecessary region subtags, 
instead of using the recommendations in RFC 4647 which would solve all 
of these problems.  We assume that tag producers will start adding 
script subtags indiscriminately, but that tag consumers will *not* 
update their matching engines to understand them.

> When they are proposed, you can count on my emitting a quiet -1 for 
> each of them. Consensus, after all, is not unanimity.

Would you support removing some of the ones that are there already?

Example: Timne.  In order for Suppress-Script to work in the case of 
Timne, there has to be content currently tagged "tem-SL" (with a region 
subtag, not just "tem"), and there must be a concern that the content 
will not be found if it is tagged "tem-Latn-SL" instead (thus using TWO 
unnecessary subtags instead of one).  We added Timne to the initial S-S 
list, not because this scenario played out, but because it could be 
easily determined that Timne is written overwhelmingly in Latin script.

I don't like S-S now any more than when I started writing this.  My 
concern is that we already did 50% of this bulk adding job, and we are 
better off doing 100% or 0% rather than 50%.

---

From: Addison Phillips <[email-masked]>
Date: Wed, 27 Sep 2006 13:32:08 -0700

Hi,

Some notes follow.

> documents and policy decisions about them were discussed.  I guess this 

No, my bad for being grouchy.

> "customarily" written in Ol Chiki?  That probably depends on who you ask.

Yes, but the demonstration for multiple scripts is relatively easier to 
prove: find some moderately large number of documents in each of the 
scripts. Presence of more than one script, plus, is possible, 
descriptions of the language as being written in more than one script is 
a *positive* demonstration of multiple scripts.

Whereas, you can produce all the documents in Tatar in Cyrillic that you 
want: it doesn't prove that there are (or are not) a large number of 
Tatar documents lurking somewhere in the Latin script. Proving the 
negative case is hard without a Tatar library card.

> the expertise to make most of those calls, and they should only be made 

Avoiding the necessity would seem like a better choice. The 
compatibility issues are avoided if we can make strong enough positive 
statements about not using script subtags except where they are 
necessary and providing for ways that registrants can create information 
showing the minority of cases where they are.

> update their matching engines to understand them.

Assumptions that may or may not be true. But the problem here is that 
S-S seeks to provide, at great cost and difficulty, mechanisms to 
address the problem which could be better solved through simpler, more 
manageable means.

> Would you support removing some of the ones that are there already?

Yes. I support (one might infer) removing them *all* and adding another 
kind of field to solve the problem. The reason S-S was chosen was the 
impression that no one would ever read RFC 4646 (maybe not such a bad 
assumption) and just look at the registry. S-S provides a very positive 
hint to authors of "major" languages (i.e. those with 639-1 codes) not 
to start using the script subtag. I'd almost prefer limiting it to the 
alpha2 codes.

> better off doing 100% or 0% rather than 50%.

0% does less potential harm than 100% does ;-)

Addison

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 28 Sep 2006 02:57:42 +0200

Doug Ewell wrote:
 
> it needs to be confirmed that such a request relates to
> actual usage, and is not a "test" of the system or an
> attempt to "prove" something about the system or its
> participants or related SDOs.

1: The intended cleanup of the missing Suppress-Scripts did
   not work so far.
2: What's okay for ngo should be also okay for fy and frr.
3: I consider statements that my registration request for 
   frr was a bad faith effort as insulting.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 28 Sep 2006 13:53:44 +0900

At 03:06 06/09/28, Doug Ewell wrote:

>Last March I began a small research project to determine whether Korean should have a Suppress-Script of Hangul.  I read books, scanned Korean-language newspapers published in Seoul and in Los Angeles, checked numerous Web sites, talked to native Korean speakers in person and by e-mail.  The result was that Hangul is, in fact, used an overwhelming proportion of the time to write modern Korean -- a colleague at work actually used the word "overwhelming" -- but there is still a certain, tiny, regular pattern of usage of Han (hanja) in scholarly works and newspapers.  "Regular" was the key here.  So I concluded that a Suppress-Script for Korean actually would NOT be a good idea, which is not what I expected to find.

This is a particularly tricky one. In modern Korean, what you may want to distinguish
in practice is texts written exclusively in Hangul and texts written mostly in
Hangul, but with a few Hanja interspersed. Historically, you also want to distinguish
texts written exclusively in Hanja.

The "a few Hanja interspersed" can, as far as I understand, go from a very low percentage
(such as <1%, an example I have seen is books for the board game of Go, where
Hanja are used for 'black' and 'white', but nothing else) to a much higher
percentage (a potential example might be a political treatise where all the
names and all the technical terms are in Hanja). In terms of readability, these
two examples, althoug both mixtures, have to be evaluated quite differently:
The former is about as easy to read as pure Hangul, while the later is quite
a bit more difficult.

In that sense, one can ask oneself whether an example like the former cannoFrom [email-masked] Thu Sep 28 00:54:29 2006
Date: Thu, 28 Sep 2006 13:53:44 +0900
To: "Doug Ewell" <[email-masked]>, "LTRU Working Group" <[email-masked]>, 
	<[email-masked]>
From: Martin Duerst <[email-masked]>
Cc: 
Subject: [Ltru] Korean (and Japanese) (was: Re: Suppress-Script batch 1)

At 03:06 06/09/28, Doug Ewell wrote:

>Last March I began a small research project to determine whether Korean should have a Suppress-Script of Hangul.  I read books, scanned Korean-language newspapers published in Seoul and in Los Angeles, checked numerous Web sites, talked to native Korean speakers in person and by e-mail.  The result was that Hangul is, in fact, used an overwhelming proportion of the time to write modern Korean -- a colleague at work actually used the word "overwhelming" -- but there is still a certain, tiny, regular pattern of usage of Han (hanja) in scholarly works and newspapers.  "Regular" was the key here.  So I concluded that a Suppress-Script for Korean actually would NOT be a good idea, which is not what I expected to find.

This is a particularly tricky one. In modern Korean, what you may want to distinguish
in practice is texts written exclusively in Hangul and texts written mostly in
Hangul, but with a few Hanja interspersed. Historically, you also want to distinguish
texts written exclusively in Hanja.

The "a few Hanja interspersed" can, as far as I understand, go from a very low percentage
(such as <1%, an example I have seen is books for the board game of Go, where
Hanja are used for 'black' and 'white', but nothing else) to a much higher
percentage (a potential example might be a political treatise where all the
names and all the technical terms are in Hanja). In terms of readability, these
two examples, althoug both mixtures, have to be evaluated quite differently:
The former is about as easy to read as pure Hangul, while the later is quite
a bit more difficult.

In that sense, one can ask oneself whether an example like the former cannot
just be tagged as Hangul, in the same way as e.g. a Latin text with an
occasional Greek or other character would still be tagged as Latin.
For the later example, the question is whether something similar to
    Subtag: Jpan
    Description: Japanese (alias for Han + Hiragana + Katakana)
should be defined for the Korean case. The problem is that while
conventions for what to write with Kanji, Hiragana, and Katakana in
modern Japanese are quite well established, the practice is quite
a bit more varying in Korean, the way I understand it.

Speaking about Japanese, I'm quite surprised that we don't have
    Supress-script: Jpan
at
    Type: language
    Subtag: ja
    Description: Japanese
    Added: 2005-10-16
When something is tagged ja, the assumption is that it's written in
Kanji-Kana mixture, because that's how Japanese is written. Not having
a Suppress-script would mean that strictly speaking, ja-Jpan should be
used, which nobody does and would be silly anyway.

>One important fact to keep in mind is that Suppress-Script values may be not only added, but also removed (Section 3.4, item 7).  If we make a mistake, we can go back and fix it.

Oh, at least one thing that we can fix. We should definitely
leave it that way.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Masayasu Ishikawa <[email-masked]>
Date: Thu, 28 Sep 2006 18:24:49 +0900

Martin Duerst wrote:

> Kanji-Kana mixture, because that's how Japanese is written.

In many cases yes, but you can find quite a few exceptions.
For example, "Ishikawa" in romaji is Japanese, even if it is
written in Latin script, and I used the language tag "ja" in
quite a few places for that.  With RFC 4646 I could use "ja-Latn"
in that case, but anyway, there's more than one way to write
Japanese and I wouldn't simply assume that "ja" means "written
in Kanji-Kana mixture".

-- 
Masayasu Ishikawa

---

From: Martin Duerst <[email-masked]>
Date: Thu, 28 Sep 2006 19:42:16 +0900

Hello Masayasu,

>Japanese and I wouldn't simply assume that "ja" means "written

I guess this makes sense. But my guess is that this is on a
micro level. On that level, the main usages of the information
are e.g. for text-to-speach. What you want is that your name
is pronounced in the Japanese way rather than the way an
English (or some other language) speaker would pronounce it.
The characters are there in the data, so tagging the script
would actually be overkill, either redundant or contradictory.

I think this is an interesting aspect of language tagging that
maybe we should mention in the RFC 4646bis: The script can
be left out if it's obvious from the data. This would just
be a special case of John's "tag wisely" general principle.
This in my view doesn't justify to not use Jpan as a supress-
script, but it gives you the licence to just use ja even if
ja-Latn would be more precise.

I think the main use case for suppress-script is for things
such as whole-document search: You don't want to have to look
into each document to check what script(s) it uses. In that
case, to me the only reasonable way for things to work is that
ja means "Japanese, as usually written (Kanji-Kana mixture),
and ja-Latn is used for romanized documents (except for learning
material, and some stuff in the early years of computer use,
I haven't seen a single ja-Latn document, and I look at
Japanese all day).

Would you agree with this analysis?

Regards,      Martin.

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 28 Sep 2006 06:00:37 -0700

> Speaking about Japanese, I'm quite surprised that we don't have

That's easy: Jpan was added to ISO 15924 this past July, after we went 
through our first round of adding Suppress-Scripts in bulk, and nobody 
thought to add it to ja.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 29 Sep 2006 02:19:47 +0200

Martin Duerst wrote:

> an interesting aspect of language tagging that maybe we
> should mention in the RFC 4646bis: The script can be left
> out if it's obvious from the data.

I don't get this, "language tagging is unnecessary if the
language is obvious from the data" would be a very similar
statement.  What's "obvious" about a script in the age of
Unicode ?  Of course the script should be left out if it's
irrelevant (spoken content), but that's another case.

> This in my view doesn't justify to not use Jpan as a
> supress-script, but it gives you the licence to just use
> ja even if ja-Latn would be more precise.

Legacy applications and content don't need a licence, they
couldn't use ja-Latn before it was specified.  And they'll
continue to use ja until they are upgraded to 4646 or its
successor(s), that could take decades (plural).

All new applications better use a tag ja-Latn where that's
relevant.  And the same new applications need a licence to
suppress Jpan in ja-Jpan for backwards compatibility, they
need your proposed Suppress-Script.

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 28 Sep 2006 18:24:53 -0700

You're reading into the spec rules that don't exist; Suppress-Script is a
SHOULD not a MUST. It is perfectly valid -- and reasonable -- for me to tag
as follows:

A. *<blah xml:lang='ja'>kanji*
(漢字<http://en.wiktionary.org/wiki/%E6%BC%A2%E5%AD%97>
)</blah>

I don't *have* to indicate the script. Now, I could also correctly tag as:

*B. <blah xml:lang='ja-Latn'>kanji*</blah><blah *xml:lang='ja-Japn'*>
(漢字<http://en.wiktionary.org/wiki/%E6%BC%A2%E5%AD%97>
)</blah>
*C. <blah xml:lang='ja'>kanji*</blah><blah *xml:lang='ja-Japn'*>
(漢字<http://en.wiktionary.org/wiki/%E6%BC%A2%E5%AD%97>
)</blah>
*D. <blah xml:lang='ja-Latn'>kanji*</blah><blah *xml:lang='ja'*>
(漢字<http://en.wiktionary.org/wiki/%E6%BC%A2%E5%AD%97>
)</blah>

And of course, I could incorrectly tag as the following, or other invalid
variants:
*E. <blah xml:lang='ja-Japn'>kanji*
(漢字<http://en.wiktionary.org/wiki/%E6%BC%A2%E5%AD%97>
)</blah>
F. *<blah xml:lang='ja-Latn'>kanji*
(漢字<http://en.wiktionary.org/wiki/%E6%BC%A2%E5%AD%97>
)</blah>

What Martin was pointing out is that in situations like this, I can
externally verify programmatically that E and F are incorrect; libraries
that use the Unicode Character Database (like ICU and others) can determine
that there are characters in the span that are not Japn and there are
characters in F that are not Latn. In such circumstances, I don't need to
supply the script. [Script variants are a different matter; Hant and Hans
overlap: some characters are clearly only Hant and some clearly only Hans,
but many are shared. (There are other issues with these that I won't go
into).]

Moreover, when I said *incorrect*, it is not, of course, a conformance
issue. There is nothing in RFC 4646 that requires that any particular text
be correctly tagged. The only thing the RFC can do determine the meaning of
'ja-Japn', not its correct application.

Mark

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 29 Sep 2006 04:49:00 +0200

Mark Davis wrote:

> You're reading into the spec rules that don't exist;
> Suppress-Script is a SHOULD not a MUST.
[...]
>> All new applications better use a tag ja-Latn where
>> that's relevant.
------------------------^^^^^^^^^^
This "better use" reflects the meaning of a SHOULD, or
are you talking about something else ?

See also <http://purl.net/xyzzy/kex/mustard.kex> :-)

---

From: John Cowan <[email-masked]>
Date: Thu, 28 Sep 2006 23:44:32 -0400

Mark Davis scripsit:

> that use the Unicode Character Database (like ICU and others) can
> determine that there are characters in the span that are not Japn and
> there are characters in F that are not Latn.

Well, only probabilistically.  An English text is still in Latn even
if (like the one I'm reading now) it contains the occasional Greek word
in the Greek script.  For that matter, it's still in English even
though there are occasional quotations in German.

-- 
What is the sound of Perl?  Is it not the       John Cowan
sound of a [Ww]all that people have stopped     [email-masked]
banging their head against?  --Larry            http://www.ccil.org/~cowan

---

From: Mark Crispin <[email-masked]>
Date: Thu, 28 Sep 2006 08:44:25 -0700 (PDT)

On Thu, 28 Sep 2006, Martin Duerst wrote:
> On that level, the main usages of the information

Interesting point.

So I assume that means that "karaoke" tagged as ja-Latn would be 
pronounced (somewhat like "kah-rah-oh-kay" to an English speaker), whereas 
"karaoke" tagged as "en" would be pronounced as "kerry-oh-key"?  :-)

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 29 Sep 2006 07:40:42 -0700

I wasn't talking about that. I was talking about your sentence:

"And the same new applications need a licence to
suppress Jpan in ja-Jpan for backwards compatibility, they
need your proposed Suppress-Script."

This is untrue. I need no licence to tag anything as simply 'ja' (instead of
ja-<somescript>), if that's all I want to tag it with.

Mark

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 29 Sep 2006 08:09:35 -0700

Hard to make such a catagorical statement; it really depends. If I said that
all of the text (meaning every word) in the following was English written in
Latin, I would be clearly wrong. If I said that the sentence was English
written in Latin, with some Japanese words, I'd be right.

"In Japan people write in * kanji*
(漢字<http://en.wiktionary.org/wiki/%E6%BC%A2%E5%AD%97>
)"

Mark

On 9/28/06, John Cowan <[email-masked]> wrote:
> banging their head against?  --Larry            http://www.ccil.org/~cowan<http://www.ccil.org/%7Ecowan>
>

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 29 Sep 2006 08:17:30 -0700

A more realistic (but still, of course, somewhat contrived) example would
be:

"He didn't do it for her beer, he did it for her sake."

It could be interpreted in two ways, depending on whether "sake" was read as
/sek/ or as /sake/. But, of course, the same thing could happen with two
native English words that happen to have different pronunciations, where
language tags wouldn't help to disambiguate.

And when push comes to shove, only in extremely rare instances will people
tag language on a very fine grained level. Frankly, at Google we end up
having to basically disregard language tagging of web pages because it is
so, so often wrong.

Mark

On 9/28/06, Mark Crispin <[email-masked]> wrote:
> Ietf-languages mailing list
> [email-masked]

---

From: John Cowan <[email-masked]>
Date: Fri, 29 Sep 2006 11:44:56 -0400

> ja-<somescript>), if that's all I want to tag it with.

Quite so.  The point of Suppress-Script: is to discourage pointless "ja-Jpan"
tags, because they damage the language-tag ecology.

-- 
John Cowan  [email-masked]   http://ccil.org/~cowan
Assent may be registered by a signature, a handshake, or a click of a computer
mouse transmitted across the invisible ether of the Internet. Formality
is not a requisite; any sign, symbol or action, or even willful inaction,
as long as it is unequivocally referable to the promise, may create a contract.
       --Specht v. Netscape

---

From: Mark Crispin <[email-masked]>
Date: Fri, 29 Sep 2006 08:51:24 -0700 (PDT)

> "He didn't do it for her beer, he did it for her sake."

Indeed.  There is a joke about a sansei named Peter Yamamoto who decided 
to brew sake here in the US, and how people would do anything for Pete's 
sake...

> having to basically disregard language tagging of web pages because it is

I agree.  It isn't just language tagging.  Charset tagging is often also 
wrong.  Some people are under the mistaken impression that the way to tag 
"8-bit content" is to set charset to ISO-8859-1, even though the actual 
content is Korean, etc.

For what it's worth, I just got private email from the mighty Morfin power 
ranger.  I sent back a sharp response telling him not to send me any more 
email.

-- Mark --

http://panda.com/mrc

---

From: John Cowan <[email-masked]>
Date: Fri, 29 Sep 2006 17:39:25 -0400

> written in more than one script (Note Well: the word 'customarily'!) is 

The trouble is, what's the alternative?  If we went to an Accept-Script:
header specifying a script that works with multiscript languages like
az, then we have two possible tacks:

a) If there are no Accept-Script headers, no script subtags can be used.

b) If there are no Accept-Script headers, any script subtag can be used.

The trouble with (a) is that it defeats the generativity that was one
of the main purposes of RFC 4646.  Instead of just being able to tag
your Cyrillic orthography for Irish as "ga-Cyrl", you have to come to
us hat in hand and ask for it to be registered.  Bad.

The trouble with (b) is that it's not monotonic.  If we learn that
'aa' is often written in 'Aaaa', and we add it to the registry, then
the formerly valid tag "aa-Zzzz" becomes invalid.  Very, very bad,
particularly if "aa-Zzzz" is already in use in the wild.

What's more, almost any language that can be written at all, can be
written in alternative scripts on some occasions.  'Latn' and 'Brai'
are notable candidates.  So we end up having to provide Accept-Script:
headers for most languages anyhow.

> It took a lot of work to arrive at [the Korean] conclusion, no? Why

Because Korean is an edge case: it has moved from complete use of Han
script to a position where Hangul is almost always used but there are
a few domains where Han is still required.  In South Korea, at least,
one must still learn a limited number of Han characters to be
considered literate.  There is nothing comparable for English or
French or German or Igbo or Hebrew.

I'm happy to keep Suppress-Script confined (as a matter of administrative
policy, not RFC 4646bis rule) to the languages of 639-1 and 639-2.

-- 
In politics, obedience and support      John Cowan <[email-masked]>
are the same thing.  --Hannah Arendt    http://www.ccil.org/~cowan

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 29 Sep 2006 15:03:28 -0700

If I had had my druthers, we would have had a tag that said (perhaps in
nicer language):

Commonly-Multiscript: Yes

We would have marked just those languages that are commonly written with
multiple scripts. That is enough of a signal to people that they don't need
to add script to other language subtags. That is enough information to solve
the compatibility problem ("Which language tags might I normally need to put
a script tag on."). If we wanted to supply just a bit more useful
information, it would be possible to change that slightly to actually name
the scripts, with most commonly used script first.

Common-Scripts: Hans, Hant

> If we learn that
> the formerly valid tag "aa-Zzzz" becomes invalid.

Like the current Suppress-Script, *none* of this is a matter of validity.
Any combination of valid language tag and valid script tag is valid; it is
just that certain combinations are discouraged in general use.

Mark

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 30 Sep 2006 14:23:56 +0200

John Cowan wrote:
 
> I'm happy to keep Suppress-Script confined (as a matter of
> administrative policy, not RFC 4646bis rule) to the languages
> of 639-1 and 639-2.

The registry won't tell users what's what, so I'm not happy
with your proposal.  If this boils down to "let's try hard
to get it right at least for 639-1 and 639-2" it's okay.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 30 Sep 2006 14:32:57 +0200

> This is untrue. I need no licence to tag anything as simply
> 'ja' (instead of ja-<somescript>), if that's all I want to
> tag it with.

You're a human user, not a "new application".  It's the job of
the application to accept ja-Latn if you say so, and it could
ask "are you sure" if you say "ja-Jpan".  Without a "licence"
in the form of a Suppress-Script the application would accept
ja-Latn exactly like ja-Jpan.  Getting no match for any legacy
ja-JP bogeys.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 30 Sep 2006 14:48:22 +0200

[email-masked] wrote:
 
> I just wanted to note that these two private use examples
> likely need to be changed reflecting the recent change to
> ISO 3166-1:
 
> sr-Latn-QM (Serbian, Latin-script, private region)

Could be Kosovo or similar, I think that's still okay.

> sr-Qaaa-CS

Yes, that has to be fixed.  In theory I could reactivate
my meta wikimedia account (if I recall the password) and
ask them if we can use their server to "note" pending
464xbis issues, after all meta is their I18N central as
far as it doesn't concern code.

In practice I hope somebody here has a more straight
forward idea, what about say the CVS servers for ICU or
CLDR ?  For subversion there is even an optional module
"Trac" with a tracker and built-in Wiki.

The IESG uses "Trac", maybe they'd be willing to let us
test it.

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 30 Sep 2006 11:13:03 -0700

I don't know what kind of scenario you have in mind, but for most I'm
familiar with, having suppress-script in the registry is of very little aid
in the UI. Let's suppose I'm picking the language in my browser. Typically
I'll get a pull-down list with the most common languages. So that's already
pre-canned, and -- notably -- takes some intelligence to produce. Nobody can
take arbitrary combinations of language, script, and region subtags, since
that would produce a huge, unusable list. Instead, someone needs to take the
time to figure out which ones are worth separating out for presentation to
the user. That involves *far* more than suppress-script: you need to know,
for example, which region-based variations are important to separate out.

Then, if you are lucky the UI will give you a way to select additional
choices (Firefox, for example, does even let you do this). A good UI will
not just have the user type in some random string and validate -- it will
provide pull-down menus that help to guide the user to picking reasonable
choices.  (Even the choice of language subtag alone gets will get
complicated with 639-3. Probably needs some fairly careful thought to allow
people to get to the one they want without too much hassle.) With the UI
selection for script, one could simply have the default choice be "any"
(meaning no language tag), with a message that the script should not
normally be selected unless your language is customarily written with
multiple scripts. And *real* guidance to the user would not use suppress
script -- instead it would give a choice, based on the language subtag, of
what the customary values are. So it really wants to show, for example,
something like:

Language: zh
Script:
   any
   simplified-only
   traditional-only
   other...

That is, only bring up the full list of scripts in unusual cases.

But all of this may vary by circumstance. If I had a library search for
specialized terms, I might allow for Japanese written in Latin -- if I have
content marked that way. If I am picking languages for use in resource
lookup, I wouldn't even have zh-Hans listed in the above choices, since my
zh content would default to that -- and the choices for region would also
just list the ones that made a difference, acording to the chosen language
subtag and script. And so on....

I think it would be helpful to describe in some detail some of the UI
scenarios that you envision where suppress-script, and that alone, is really
both necessary and sufficient to guide people in selecting language tags.

Mark

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 30 Sep 2006 23:00:51 +0200

Mark Davis wrote:

> I don't know what kind of scenario you have in mind

Lazy / clueless implementor + clueless user / tagger.

As an example I had no clue that Jpan might be a good
Suppress-Script for jp.

> Typically I'll get a pull-down list with the most
> common languages. So that's already pre-canned

My browser didn't know de-CH-1996.  Admittedly I
somehow managed to disable its Accept-Language and
don't know how to enable it again, but in theory
I'd have to configure it manually.

BTW, the stunt to screw-up the Accept-Language was
a side effect of Google's geo-location - I hated its
auto-redirection to the censored-for-Germany edition,
I wanted the "international" (= US) edition.

Maybe BE users are now in a similar position, trying
to get an edition that's not censored-for-Belgium.
Sometimes Geo-IP is cute, but it can be also a pain.

> Even the choice of language subtag alone gets will
> get complicated with 639-3.

Yes, an ordinary select box with thousands of choices
won't work, certainly not with my old browser.

> With the UI selection for script, one could simply
> have the default choice be "any" (meaning no language
> tag), with a message that the script should not
> normally be selected unless your language is customarily
> written with multiple scripts.

The registry offers Suppress-Script as indication when
using a script could be necessary:  For languages without
Suppress-Script.

> *real* guidance to the user would not use suppress
> script -- instead it would give a choice, based on the
> language subtag, of what the c ustomary values are.

That info isn't in the registry.  And if the application
checks what CLDR says the implementor could offer Arab
or Latn for tr.  Resulting in tr-Latn.  But the registry
clearly says "don't do that, it's probably not what you
want for tr".  The Suppress-Script is a good feature of
the registry.

BTW, that's odd, we copied CLDR...  Maybe it used to be
only Latn, and Arab was added later.  And of course the
Suppress-Scripts and CLDR have different purposes.

Frank

---

From: Martin Duerst <[email-masked]>
Date: Sat, 30 Sep 2006 18:09:44 +0900

>policy, not RFC 4646bis rule) to the languages of 639-1 and 639-2.

As a technical contributor, I'd tend to disagree. There is no need
to force people to be unnecessarily detailled when tagging. If it's
widely accepted general practice that a language is written in a single
script only, then just the language tag should do the job. Scripts,
like regions, should only be added when they are really necessary.

Regards,    Martin.