Subject: [Ltru] RE: Updated article: Two-letter or three-letter language codes

From: Misha Wolf <[email-masked]>
Date: Fri, 22 Sep 2006 16:07:23 +0100

That would be seriously broken.  It would encourage 
people to violate BCP 47.

Misha

-----Original Message-----
From: Stephen Deach [mailto:[email-masked]] 
Sent: 22 September 2006 16:05
To: Misha Wolf; Richard Ishida
Cc: [email-masked]
Subject: RE: Updated article: Two-letter or three-letter language codes

I would strongly recomment taht all processing applications support both
2 
& 3 letter ISO codes. It was the only way to get some countries and some

applications (especially in business databases) simply always use the 3 
letter coded.

This email was sent to you by Reuters, the global news and information company. 
To find out more about Reuters visit www.about.reuters.com

Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Ltd.

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Martin Duerst <[email-masked]>
Date: Sat, 23 Sep 2006 11:29:48 +0900

Exactly. Codes should be converted at the boundaries to systems that
can't handle anything else that three-letter codes. It has to be done
one way, so it can as well be done both ways.

Regards,   Martin.

At 00:07 06/09/23, Misha Wolf wrote:
>Any views expressed in this message are those of the individual sender, 
>except where the sender specifically states them to be the views of Reuters Ltd.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 23 Sep 2006 09:31:58 -0700

I can appreciate the goal. In the case of language tags, we've done
some analysis here at Google, and at least in a (large) sample of web
pages and xml documents, the three-letter codes don't account for many
of the problem cases.

Total-Valid	99.62%
Total-WellFormed	99.71%

Here are some examples of what we do find.

Ill-formed:
(the second one has a space at the end. this also excludes x-....
where the ... is a subtag longer than 8 -- that has a pretty high
frequency)
Rank	Frequency	tag
102	0.015999%	en-us.
122	0.010068%	en-us
219	0.001668%	es-es-ts
302	0.000638%	q=0.5
304	0.000634%	undefined
339	0.000429%	espa�ol
391	0.000325%	Indonesian
458	0.000185%	utf-8
464	0.000178%	pt-br
467	0.000173%	t�rk�e
481	0.000158%	portugues
503	0.000138%	de
518	0.000126%	es-ES-TS
529	0.000120%	Vietnamese
547	0.000107%	sr-sp-latn
549	0.000107%	e
555	0.000102%	Language T20029 2005-05-18
...

Well-Formed but Invalid
88	0.024632%	en-securid
133	0.007796%	English
136	0.007353%	xl
160	0.004739%	Chinese
176	0.003235%	zs
182	0.003062%	us
183	0.003054%	chinese
184	0.002891%	eses
188	0.002497%	in
189	0.002461%	pdf
210	0.001827%	en-sp
213	0.001771%	es-sp
248	0.001150%	zh-chs
254	0.001088%	French
262	0.001019%	po
276	0.000873%	sr-SP
279	0.000865%	no-bok
284	0.000803%	Arabic
293	0.000733%	sr_SP
299	0.000667%	en-en
303	0.000637%	ua
318	0.000570%	jp
...

On 9/23/06, Stephen Deach <[email-masked]> wrote:
>
> I just wanted to make sure this "shortest code" issue was considered
> carefully.
>    A lot of people I've talked to about internationalization issues over
> the years simply had "assumed" that the 3-letter ISO codes superceded the
> 2-letter ones, or chose to use all 3-letter codes rather than a mix of 2 &
> 3 because it was easier to make it a fixed-length field.
>
> I understand your goal is to eventually make this simpler, by eliminating
> multiple formats for each subtoken and moving to a single registry/list. As
> a general process I always try to accept ill-formed input, but emit
> corrected output (since you pretty much have to grandfather all past formats).
>
>
> At 2006.09.23-11:29(+0900), Martin Duerst wrote:
> ---Steve Deach
>     [email-masked]

---

From: Stephen Deach <[email-masked]>
Date: Sat, 23 Sep 2006 07:58:45 -0700

---Steve Deach
    [email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Sun, 24 Sep 2006 11:56:41 +0900

Hello Mark,

Many thanks for this interesting data. In my mailer, the line endings
didn't work very well, so I refomatted it (at the same time, my
mailer messed up the non-ASCII stuff, sorry).
The only three-letter code I can see is pdf, for which we can blame
Steve's company :-).

On the other hand, in the list, there are a few items (such as en-us,
pt-br) that look perfectly fine. What was wrong with them?

Regards,   Martin.

>102    0.015999%       en-us.
>122    0.010068%       en-us
>219    0.001668%       es-es-ts
>302    0.000638%       q=0.5
>304    0.000634%       undefined
>339    0.000429%       espa$B~A%9(Bol
>391    0.000325%       Indonesian
>458    0.000185%       utf-8
>464    0.000178%       pt-br
>467    0.000173%       t$B~A%9(Brk$B~A%9(Be
>481    0.000158%       portugues
>503    0.000138%       de
>518    From [email-masked] Sun Sep 24 02:04:09 2006
Date: Sun, 24 Sep 2006 11:56:41 +0900
To: "Mark Davis" <[email-masked]>,
	"Stephen Deach" <[email-masked]>
From: Martin Duerst <[email-masked]>
Cc: Misha Wolf <[email-masked]>, [email-masked],
	[email-masked]
Subject: [Ltru] Re: Updated article: Two-letter or three-letter language
	codes

Hello Mark,

Many thanks for this interesting data. In my mailer, the line endings
didn't work very well, so I refomatted it (at the same time, my
mailer messed up the non-ASCII stuff, sorry).
The only three-letter code I can see is pdf, for which we can blame
Steve's company :-).

On the other hand, in the list, there are a few items (such as en-us,
pt-br) that look perfectly fine. What was wrong with them?

Regards,   Martin.

>102    0.015999%       en-us.
>122    0.010068%       en-us
>219    0.001668%       es-es-ts
>302    0.000638%       q=0.5
>304    0.000634%       undefined
>339    0.000429%       espa$B~A%9(Bol
>391    0.000325%       Indonesian
>458    0.000185%       utf-8
>464    0.000178%       pt-br
>467    0.000173%       t$B~A%9(Brk$B~A%9(Be
>481    0.000158%       portugues
>503    0.000138%       de
>518    0.000126%       es-ES-TS
>529    0.000120%       Vietnamese
>547    0.000107%       sr-sp-latn
>549    0.000107%       e

---

From: Martin Duerst <[email-masked]>
Date: Sun, 24 Sep 2006 12:23:15 +0900

At 23:58 06/09/23, Stephen Deach wrote:
>I just wanted to make sure this "shortest code" issue was considered carefully.
>   A lot of people I've talked to about internationalization issues over the years simply had "assumed" that the 3-letter ISO codes superceded the 2-letter ones, or chose to use all 3-letter codes rather than a mix of 2 & 3 because it0.000126%       es-ES-TS
>529    0.000120%       Vietnamese
>547    0.000107%       sr-sp-latn
>549    0.000107%       e

---

From: Martin Duerst <[email-masked]>
Date: Sun, 24 Sep 2006 12:23:15 +0900

At 23:58 06/09/23, Stephen Deach wrote:
>   A lot of people I've talked to about internationalization issues over the years simply had "assumed" that the 3-letter ISO codes superceded the 2-letter ones, or chose to use all 3-letter codes rather than a mix of 2 & 3 because it was easier to make it a fixed-length field.

IETF language tags always have been variable-length.

>I understand your goal is to eventually make this simpler, by eliminating multiple formats for each subtoken and moving to a single registry/list. 

This is not a matter of future development. RFC 3066,
which sorted out this issue, has been published in January 2001.
Mark's data also seems to show that this is quite well respected.

>As a general process I always try to accept ill-formed input, but emit corrected output (since you pretty much have to grandfather all past formats).

In the case of language tags, this is a very good policy for interfaces
between other systems and the Internet protocols and formats that use
RFC 4646 language tags.

Regards,   Martin.

 was easier to make it a fixed-length field.

IETF language tags always have been variable-length.

>I understand your goal is to eventually make this simpler, by eliminating multiple formats for each subtoken and moving to a single registry/list. 

This is not a matter of future development. RFC 3066,
which sorted out this issue, has been published in January 2001.
Mark's data also seems to show that this is quite well respected.

>As a general process I always try to accept ill-formed input, but emit corrected output (since you pretty much have to grandfather all past formats).

In the case of language tags, this is a very good policy for interfaces
between other systems and the Internet protocols and formats that use
RFC 4646 language tags.

Regards,   Martin.

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 24 Sep 2006 13:14:55 -0700

1. Details.

> Capitalization.

No. (trying for equal brevity)

More completely:

102     0.015999%       en-us.
122     0.010068%       en-us

As I said in my original message, the second one has a space at the end. The
first one has a period at the end. So both are ill-formed. The same is true
of pt-br (extra space at end).

2 Correction

A correction: this is actually Accept-Language values, not documents -- we
get different results looking at documents. A very interesting point,
however, is that the errors here could be corrected if the browsers checked
for well-formedness, or at least partial well-formedness, when allowing the
user to pick his/er browser's language. That would eliminate a lot of cruft.

3. Guidance for User Agents. This raises a point we should probably have
language in 4646bis for. Here's rough text for it; I anticipate that this
will generate some discussion ;-)

When a user agent, such as a browser, allows users to enter a language tag
by typing, the results SHOULD be checked for well-formedness. If the user
agent is not regularly updated to the latest registry, it SHOULD NOT require
validity, because that could exclude current, valid language tags. It is
recommended, however, that the user be notified that the language tag may
not be valid.

4. Basic Well-Formedness. We may also want to have the notion of basic
well-formedness, which that part of validity which can be checked with a
regular expression. The difference is that basic well-formedness doesn't
check for multiple singleton extensions. The value of doing this is that (a)
it covers 99.999...% of the value of a well-formedness check, and (b) it is
a much easier sell to implementers that all they need is a simple regex
check.

Mark

---

From: Chris Lilley <[email-masked]>
Date: Sun, 24 Sep 2006 16:56:56 +0200

On Sunday, September 24, 2006, 4:44:03 PM, Stephen wrote:

>>pt-br) that look perfectly fine. What was wrong with them?

SD> Capitalization.

Capitalization is, IIRC, irrelevant.

However "en-us." and "en-us " (note trailing space and trailing period) are not correct. en-US or En-Us or any other variant therof are fine.

-- 
 Chris Lilley                    mailto:[email-masked]
 Interaction Domain Leader
 Co-Chair, W3C SVG Working Group
 W3C Graphics Activity Lead
 Co-Chair, W3C Hypertext CG

---

From: Chris Lilley <[email-masked]>
Date: Sun, 24 Sep 2006 17:01:54 +0200

On Sunday, September 24, 2006, 4:54:45 PM, Stephen wrote:

SD> The fact that "utf-8" and "pdf" (which I would call "encodings") show up in
SD> this list points out something interesting about the term "language" that
SD> should probably be emphasized in the paper/article.

I agree.

The fact that "Chinese" and "Indonesian" show up is less surprising but should also be explained as being incorrect, and the correct way for those should be given in examples.

---

From: Martin Duerst <[email-masked]>
Date: Mon, 25 Sep 2006 16:28:34 +0900

[I have removed [email-masked] because this gets more
LTRU-specific]

At 05:14 06/09/25, Mark Davis wrote:

>3. Guidance for User Agents. This raises a point we should probably have language in 4646bis for. Here's rough text for it; I anticipate that this will generate some discussion ;-)
>
>When a user agent, such as a browser, allows users to enter a language tag by typing, the results SHOULD be checked for well-formedness. If the user agent is not regularly updated to the latest registry, it SHOULD NOT require validity, because that could exclude current, valid language tags. It is recommended, however, that the user be notified that the language tag may not be valid. 

I think we already have started discussing this, and I think
some more guidance in this direction may be appropriate, but
I hope we can word it more generally, not so browser-specific.

Thinking out lound, in practical terms, what I would do would be:
- Check for well-formedness. If not well-formed, alert the user
  about as follows: "$@*&%" cannot be a language tag. Please change.
- Check for validity against a built-in list. If not valid
  against that list, ask the user, about as follows:
  I have a large list of tags and subtags (about 7000), but
  your subtag "foo" isn't in that list. For a complete, up-to-date
  list, please check IANA (address here).
  After this, the two alternatives [go ahead anyway] [change]
The reason for why the second one makes sense is that for 99.99%
and more of actual uses, even if some software doesn't have the
latest version, tags can be validated. So if the software can't
validate a tag, the chance there is a mistake is much higher
than the chance that there was an update recently.

This is also true for the current (RFC 4646, iso-639-1/2 only)
state, because in terms of occurrence, these subtags are way
more frequent than iso-639-3. In some sense, it's actually
more true for iso-639-1/2 only: iso-639-3 populates the 3-letter
code space rather densly (40%). At that density, the chance
of a typo hitting another valid code and being accepted is
getting higher.

>4. Basic Well-Formedness. We may also want to have the notion of basic well-formedness, which that part of validity which can be checked with a regular expression. The difference is that basic well-formedness doesn't check for multiple singleton extensions. The value of doing this is that (a) it covers 99.999...% of the value of a well-formedness check, and (b) it is a much easier sell to implementers that all they need is a simple regex check.

Shouldn't you say "that part of well-formedness" rather than
"that part of validity" above?
Also, I think that having three levels is a bit much. The difference
is not so big, the implementation effort also not too different
(well, it depends a lot on the language used), and implementors
are known to cut corners when they can anyway, with or without
permission in the spec. One choice would be to loosen well-formedness
a bit, but I'm not convinced yet that that's the way to go.

Regards,    Martin.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 26 Sep 2006 09:04:07 +0200

On Mon, Sep 25, 2006 at 04:28:34PM +0900,
 Martin Duerst <[email-masked]> wrote 
 a message of 60 lines which said:

> - Check for validity against a built-in list. [...] iso-639-3
> populates the 3-letter code space rather densly (40%). At that
> density, the chance of a typo hitting another valid code and being
> accepted is getting higher.

A good thing about validity testing is that you can display the full
description, not only the 3-letter code. How about using the built-in
list to display in an human-friendly way the semantics of the tag,
before asking confirmation?

% ./display-tag man-Nkoo-GN
man-Nkoo-GN: (man: language "Mandingo", Nkoo: script  "N&#x2019;Ko", GN: region  "Guinea")