Subject: [Ltru] updated demo

From: Mark Davis ⌛ <[email-masked]>
Date: Sat, 27 Jun 2009 11:38:07 -0700

I updated the demo at http://unicode.org/cldr/utility/languageid.jsp to
parse extlangs. The samples include official languages and the scripts they
use (based on CLDR data), and the names have localizations where available.

Comments welcome.

Mark

---

From: Felix Sasaki <[email-masked]>
Date: Sun, 28 Jun 2009 14:10:11 +0200

Resent, now this should reach the list.

Hello Mark,

this looks similar to
http://www.w3.org/2008/05/lta/
my language tag parser currently based on draft 21 of rfc4646bis. lta also
contains some error checking mechanisms, see examples like

http://www.w3.org/2008/05/lta/language-tags/q?input=de-x
http://www.w3.org/2008/05/lta/language-tags/q?input=xa
http://www.w3.org/2008/05/lta/language-tags/q?input=en-latn
http://www.w3.org/2008/05/lta/language-tags/q?input=ja-1901
http://www.w3.org/2008/05/lta/language-tags/q?input=fr-cmn
http://www.w3.org/2008/05/lta/language-tags/q?input=zh-cmn-cmn
http://www.w3.org/2008/05/lta/language-tags/q?input=zh-cmn-a-bbb-a-ccc
http://www.w3.org/2008/05/lta/language-tags/q?input=de-de-1901-1901

Output is available in HTML with German UI and English, and in an XML
format, see e.g.
http://www.w3.org/2008/05/lta/language-tags/q?input=de-de-1901-1901&output=xml

My comment on your tool is that to co-ordinate such efforts it would be
great to have a common machine-readable output format for language tag
parsing, also e.g. to deal with error descriptions like

 <lta:variant>
      <lta:subtag>1901</lta:subtag>
      <lta:registryInfo>

         <lta:var ty="variant" su="1901" ad="2005-10-16">

            <lta:ds>Traditional German orthography

</lta:ds>
            <lta:pref>de</lta:pref>

         </lta:var>
      </lta:registryInfo>
      <lta:matchedPrefix>de</lta:matchedPrefix>

      <lta:error type="e007">
         <lta:errorText>Variant repetition</lta:errorText>

         <lta:errorAddInfo>

            <lta:subtag>1901</lta:subtag>
         </lta:errorAddInfo>

      </lta:error>
   </lta:variant>

Felix

2009/6/27 Mark Davis ⌛ <[email-masked]>
I updated the demo at http://unicode.org/cldr/utility/languageid.jsp to
parse extlangs. The samples include official languages and the scripts they
use (based on CLDR data), and the names have localizations where available.

Comments welcome.

Mark

______________________________
>
> _________________
> Ltru mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ltru

---

From: Mark Davis ⌛ <[email-masked]>
Date: Sun, 28 Jun 2009 13:07:24 -0700

It does look similar. After looking them over, I think sometimes one is
better and sometimes the other is. The differences I can see (other than UI)
are:

*Feedback on ill-formed, invalid, or non-preferred values.*

de-x

   - http://unicode.org/cldr/utility/languageid.jsp?a=de-x shows the tag
   where the problem lies, but not potential fixes. (I use a regex based on the
   ABNF for well-formedness, and if it fails I just show the tag where the
   problem is.)
   - http://www.w3.org/2008/05/lta/language-tags/q?input=de-x shows what the
   potential subtags at that point might be.

iw-su

   - http://unicode.org/cldr/utility/languageid.jsp?a=iw-su&l=en shows the
   replacement values for iw and su.
   -
   http://www.w3.org/2008/05/lta/language-tags/q?input=iw-su&output=html&hl=enjust
says they are valid. It does show all the registry information, like
   when the code was added.

eng-840

   - http://unicode.org/cldr/utility/languageid.jsp?a=eng-840&l=en shows the
   replacements for the wrong choice of source code (3 letter language when 2
   letter exists (common in the field), 3 digit region when 2 letter exists)
   -
   http://www.w3.org/2008/05/lta/language-tags/q?input=eng-840&output=html&hl=enjust
says they are invalid.

*Localization:*

sl-Cyrl-YU - Arabic, German

   - http://unicode.org/cldr/utility/languageid.jsp?a=sl-Cyrl-YU&l=ar and
   http://unicode.org/cldr/utility/languageid.jsp?a=sl-Cyrl-YU&l=de show
   localized subtag names.
   -
   http://www.w3.org/2008/05/lta/language-tags/q?input=sl-Cyrl-YU&output=html&hl=aromits
text;
   http://www.w3.org/2008/05/lta/language-tags/q?input=sl-Cyrl-YU&output=html&hl=dehas
a localized UI, but not localized subtag names.

*Prefix Warnings*

en-cmn-rozaj

   - http://unicode.org/cldr/utility/languageid.jsp?a=en-cmn&l=ar doesn't
   give a warning (it just applies strict validity).
   -
   http://www.w3.org/2008/05/lta/language-tags/q?input=en-cmn-rozaj&output=html&hl=endoes
supply warnings for missing variant prefixes.

*Canonical Form*

sl-cyrl-Yu-rozaj-Solba-1994-b-1234-a-Foobar-x-b-1234-a-Foobar

   -
   http://unicode.org/cldr/utility/languageid.jsp?a=sl-cyrl-Yu-rozaj-Solba-1994-b-1234-a-Foobar-x-b-1234-a-Foobar&l=enputs
the results in canonical casing and order (and shows canonical
   replacements). It does not validate extensions, like "b-1234". (It follows
   LDML canonical order for variants - alphabetical.)
   -
   http://www.w3.org/2008/05/lta/language-tags/q?input=sl-Cyrl-YU-rozaj-solba-1994-b-1234-a-Foobar-x-b-1234-a-Foobar&output=html&hl=endoesn't.
It also gives a validation error on extensions.

Validating extensions is debatable - the validity of these is established
outside of the spec and iana subtag registry. Probably best would be neither
of the above: a warning, not an error.

Note that http://unicode.org/cldr/utility/languageid.jsp<http://unicode.org/cldr/utility/languageid.jsp?a=sl-cyrl-Yu-rozaj-Solba-1994-b-1234-a-Foobar-x-b-1234-a-Foobar&l=en>says
"suggested canonical form", since in the case of multiple replacements
it doesn't try to pick the best one. Eg the best guess for ru-SU is ru-RU,
but the best guess for az-SU would be az-AZ. It also doesn't try to find
missing prefix values for variants; that's probably of such low frequency
that it doesn't pay.

*Completeness*

   - http://unicode.org/cldr/utility/languageid.jsp?a=i-default&l=en doesn't
   allow grandfathered codes. (Following LDML.)
   - http://www.w3.org/2008/05/lta/language-tags/q?input=i-default does.

FYI, the regex it uses is:

      (?: ( [a-z A-Z]{2,8} | [a-z A-Z]{2,3} [-_] [a-z A-Z]{3} )
      (?: [-_] ( [a-z A-Z]{4} ) )?
      (?: [-_] ( [a-z A-Z]{2} | [0-9]{3} ) )?
      (?: [-_] ( (?: [0-9 a-z A-Z]{5,8} | [0-9] [0-9 a-z A-Z]{3} ) (?: [-_]
(?: [0-9 a-z A-Z]{5,8} | [0-9] [0-9 a-z A-Z]{3} ) )* ) )?
      (?: [-_] ( [a-w y-z A-W Y-Z] (?: [-_] [0-9 a-z A-Z]{2,8} )+ (?: [-_]
[a-w y-z A-W Y-Z] (?: [-_] [0-9 a-z A-Z]{2,8} )+ )* ) )?
      (?: [-_] ( [xX] (?: [-_] [0-9 a-z A-Z]{1,8} )+ ) )? )
    | ( [xX] (?: [-_] [0-9 a-z A-Z]{1,8} )+ )

Mark

On Sun, Jun 28, 2009 at 02:07, Felix Sasaki <[email-masked]>wrote:

>> _______________________________________________
>> Ltru mailing list
>> [email-masked]

---

From: Felix Sasaki <[email-masked]>
Date: Sun, 28 Jun 2009 22:47:18 +0200

2009/6/28 Mark Davis ⌛ <[email-masked]>

>    ABNF for well-formedness, and if it fails I just show the tag where the
FYI, my proposed fixes are based on an LL(1) parsing implementation of the
rfc4646bis ABNF, see
http://www.w3.org/2008/05/lta/04/abnf-check.xsl
of course one might argue whether LL(1) parsing is the appropriate means for
generating the proposals, but the proposals looked intuitive to me

>
>    -
>    - http://www.w3.org/2008/05/lta/language-tags/q?input=de-x shows what
>    the potential subtags at that point might be.
>
> iw-su
>
>    http://www.w3.org/2008/05/lta/language-tags/q?input=iw-su&output=html&hl=enjust says they are valid. It does show all the registry information, like
>    when the code was added.
>
>
correct. That and the replacement values you mention below are still on my
TODO list.

>
>    -
>
> eng-840
>
>    - http://unicode.org/cldr/utility/languageid.jsp?a=eng-840&l=en shows
>    the replacements for the wrong choice of source code (3 letter language when
>    2 letter exists (common in the field), 3 digit region when 2 letter exists)
>    -
>    http://www.w3.org/2008/05/lta/language-tags/q?input=eng-840&output=html&hl=enjust says they are invalid.
>
> *Localization:*
>
>    http://www.w3.org/2008/05/lta/language-tags/q?input=sl-Cyrl-YU&output=html&hl=aromits text;
>    http://www.w3.org/2008/05/lta/language-tags/q?input=sl-Cyrl-YU&output=html&hl=dehas a localized UI, but not localized subtag names.
>
>
Yes. I guess you are using CLDR data for the localized subtag names? For
including such data easily, a common result format for language tag analysis
would be good.

>    http://www.w3.org/2008/05/lta/language-tags/q?input=en-cmn-rozaj&output=html&hl=endoes supply warnings for missing variant prefixes.
>
>
Yes, for extlang prefixes and for variant prefixes.

>    http://unicode.org/cldr/utility/languageid.jsp?a=sl-cyrl-Yu-rozaj-Solba-1994-b-1234-a-Foobar-x-b-1234-a-Foobar&l=enputs the results in canonical casing and order (and shows canonical

>
>    -
>    -
>    http://www.w3.org/2008/05/lta/language-tags/q?input=sl-Cyrl-YU-rozaj-solba-1994-b-1234-a-Foobar-x-b-1234-a-Foobar&output=html&hl=endoesn't. It also gives a validation error on extensions.
>

Reading
"Note that there might not be a registry of these subtags and validating
processors are not required to validate extensions."
from sec. 2.2.6 of
http://www.ietf.org/internet-drafts/draft-ietf-ltru-4646bis-23.txt I think
you are correct, I wil change the error to a warning in my next version.

>
>
> Note that http://unicode.org/cldr/utility/languageid.jsp<http://unicode.org/cldr/utility/languageid.jsp?a=sl-cyrl-Yu-rozaj-Solba-1994-b-1234-a-Foobar-x-b-1234-a-Foobar&l=en>says "suggested canonical form", since in the case of multiple replacements
>    - http://unicode.org/cldr/utility/languageid.jsp?a=i-default&l=endoesn't allow grandfathered codes. (Following LDML.)
yes, implementing the ABNF of rfc4646bis, see
http://www.w3.org/2008/05/lta/04/abnf.xsl

> FYI, the regex it uses is:

Thank you for this and for your feedback! FYI, the goal of lta is a)
educational about language tags, probably similar to your tool, and b) to be
used in RESTful web services which need language tag information. For b)
there is the XML output, see
http://www.w3.org/2008/05/lta/language-tags/q?input=sl-Cyrl-YU-rozaj-solba-1994-b-1234-a-Foobar-x-b-1234-a-Foobar&output=xml
and others like json or RDF might follow, depending on (users) need.

Felix

---

From: Felix Sasaki <[email-masked]>
Date: Sun, 28 Jun 2009 11:07:40 +0200

Hello Mark,

this looks similar to

 <lta:variant>

</lta:ds>
            <lta:pref>de</lta:pref>
         </lta:var>
      </lta:registryInfo>

Felix

---

From: Mark Davis ⌛ <[email-masked]>
Date: Sun, 28 Jun 2009 14:51:32 -0700

Some comments on one point you raise:

On Sun, Jun 28, 2009 at 13:47, Felix Sasaki <[email-masked]>wrote:
...

> including such data easily, a common result format for language tag analysis

For localization, CLDR uses the structure as you see in
http://unicode.org/cldr/data/common/main/de.xml

Language subtags illustrate this:

<ldml>
	<localeDisplayNames>
		<languages>
			<language type="aa">Afar</language>
...

However, the data can also be used to translate compounds, such as

			<language type="de_AT">Österreichisches Deutsch</language>
			<language type="de_CH">Schweizer Hochdeutsch</language>

That means that when getting the localized name for a tag, you have to first
try with lang+script+region, then lang+script, then lang+region to see if
there are any matches, then remove the fields you got in order to look up
the rest.

Scripts are similar, but don't have the compounds:

		<scripts>
			<script type="Arab">Arabisch</script>
...

Regions use the name 'territory', having predated BCP 47:

		<territories>
			<territory type="001">Welt</territory>

Variants are simple.

		<variants>
...
			<variant type="1994">Standardisierte Resianische Rechtschreibung</variant>
...

Ideally, however, they should allow for compounds, since for goofy compound
variant tags like sl-SI-rozaj-njiva-1994, you don't want a term like:

Slovenian (Slovenia, standardized resian orthography, resian, gniva/njiva
dialect)

but rather something a bit more readable and less repetitious like:

Slovenian (Slovenia, gniva/njiva dialect with standardized resian
orthography)

And we don't have support for extensions yet.

To put components together, there are localizable patterns:

		<localeDisplayPattern>
			<localePattern>{0} ({1})</localePattern>
			<localeSeparator>, </localeSeparator>

While it might be more natural to have structure for something like the
following, that is a real challenge for generative localization of language
tags, because of grammatical changes required by composition in more complex
languages.

Chinese written in traditional script as used in Hong Kong

There is, however, the ability to have abbreviations, like:

			<territory type="HK">Sonderverwaltungszone Hongkong</territory>
			<territory type="HK" alt="short">Hongkong</territory>

What we don't have yet is the ability to allow different forms of names for
different target environments. In flowing text, you might want to say
"traditional Chinese" for "zh-Hant", but in an alphabetized menu, something
like "Chinese, Traditional". That is, order, casing, and wording might
change between those two environments.

---

From: Felix Sasaki <[email-masked]>
Date: Mon, 29 Jun 2009 01:39:07 +0200

Hello Mark,
thank you for your helpful explanations. I have used this part of CLDR data
before in a different context (localization of XML schema). My point "For
including such data easily, a common result format for language tag analysis
would be good. " was that I would not need to implement the CLDR analysis
"again" if we would have one result format for language tag analysis - I
would just fetch parts of your results (e.g. the localization bit), and you
might fetch parts of mine (e.g. the subtag proposals).

Felix

> That means that when getting the localized name for a tag, you have to
> first try with lang+script+region, then lang+script, then lang+region to see
> if there are any matches, then remove the fields you got in order to look up
> the rest.
>