Subject: Re: [Ltru] Re: Test suite for language tags?

From: "Mark Davis" <[email-masked]>
Date: Sat, 16 Sep 2006 16:28:01 -0700

BTW, I had updated my regex to the final spec for 4646. Here is a single
Perl or Java regex that does most of the parse:

Regex: ((?: [a-z A-Z]{2,3} (?: [-] [a-z A-Z]{3} ){0,3} | [a-z A-Z]{4,8}
))(?: [-] ((?: [a-z A-Z]{4} )) )?(?: [-] ((?: [a-z A-Z]{2} | [0-9]{3} ))
)?(?: [-] ((?: (?: [0-9] [a-z A-Z 0-9]{3} | [a-z A-Z 0-9]{5,8} ) (?: [-] (?:
[0-9] [a-z A-Z 0-9]{3} | [a-z A-Z 0-9]{5,8} ) )* )) )?(?: [-] ((?: (?: [a-w
y-z A-W Y-Z] (?: [-] [a-z A-Z 0-9]{2,8} )+ ) (?: [-] (?: [a-w y-z A-W Y-Z]
(?: [-] [a-z A-Z 0-9]{2,8} )+ ) )* )) )?(?: [-] ((?: [xX] (?: [-] [a-z A-Z
0-9]{1,8} )+ )) )?| ( (?i) art [-] lojban| cel [-] gaulish| en [-] (?: boont
| GB [-] oed | scouse )| i [-] (?: ami | bnn | default | enochian | hak |
klingon | lux | mingo | navajo | pwn | tao | tay | tsu )| no [-] (?: bok |
nyn)| sgn [-] (?: BE [-] fr | BE [-] nl | CH [-] de)| zh [-] (?: cmn | zh
[-] cmn [-] Hans | cmn [-] Hant | gan | guoyu | hakka | min | min [-] nan |
wuu | xiang | yue))| ((?: [xX] (?: [-] [a-z A-Z 0-9]{1,8} )+ ))

It checks for the grandfathered tags, since otherwise too much cruft sneaks
in. You can't check in regex that there are only single instances of each
singleton extension. (In retrospect we could have allowed multiple
singletons: we could have accepted en-a-bcdef-ghijk-b-123-a-lmnop as
equivalent to the canonical form en-a-bcdef-ghijk-lmnop-b-123, but that's
water under the bridge at this point.) Of course, I didn't put this together
by hand. The table used to build it is much more readable, at

http://unicode.org/cldr/data/tools/java/org/unicode/cldr/util/data/langtagRegex.txt

and a test file that includes strings mentioned on this list is at:

http://unicode.org/cldr/data/tools/java/org/unicode/cldr/util/data/langtagTest.txt
Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Martin Duerst <[email-masked]>
Date: Sun, 17 Sep 2006 11:56:56 +0900

There's a mistake in the regex, from the underlying
langtagRegex.txt, that allows zh-zh-cmn-Hans.

Regards,    Martin.

At 08:28 06/09/17, Mark Davis wrote:
>BTW, I had updated my regex to the final spec for 4646. Here is a single Perl or Java regex that does most of the parse:
>
>Regex: ((?: [a-z A-Z]{2,3} (?: [-] [a-z A-Z]{3} ){0,3} | [a-z A-Z]{4,8} ))(?: [-] ((?: [a-z A-Z]{4} )) )?(?: [-] ((?: [a-z A-Z]{2} | [0-9]{3} )) )?(?: [-] ((?: (?: [0-9] [a-z A-Z 0-9]{3} | [a-z A-Z 0-9]{5,8} ) (?: [-] (?: [0-9] [a-z A-Z 0-9]{3} | [a-z A-Z 0-9]{5,8} ) )* )) )?(?: [-] ((?: (?: [a-w y-z A-W Y-Z] (?: [-] [a-z A-Z 0-9]{2,8} )+ ) (?: [-] (?: [a-w y-z A-W Y-Z] (?: [-] [a-z A-Z 0-9]{2,8} )+ ) )* )) )?(?: [-] ((?: [xX] (?: [-] [a-z A-Z 0-9]{1,8} )+ )) )?| ( (?i) art [-] lojban| cel [-] gaulish| en [-] (?: boont | GB [-] oed | scouse )| i [-] (?: ami | bnn | default | enochian | hak | klingon | lux | mingo | navajo | pwn | tao | tay | tsu )| no [-] (?: bok | nyn)| sgn [-] (?: BE [-] fr | BE [-] nl | CH [-] de)| zh [-] (?: cmn | zh [-] cmn [-] Hans | cmn [-] Hant | gan | guoyu | hakka | min | min [-] nan | wuu | xiang | yue))| ((?: [xX] (?: [-] [a-z A-Z 0-9]{1,8} )+ )) 
>
>It checks for the grandfathered tags, since otherwise too much cruft sneaks in. You can't check in regex that there are only single instances of each singleton extension. (In retrospect we could have allowed multiple singletons: we could have accepted en-a-bcdef-ghijk-b-123 -a-lmnop as equivalent to the canonical form en-a-bcdef-ghijk-lmnop-b-123, but that's water under the bridge at this point.) Of course, I didn't put this together by hand. The table used to build it is much more readable, at 
>
><http://unicode.org/cldr/data/tools/java/org/unicode/cldr/util/data/langtagRegex.txt>http://unicode.org/cldr/data/tools/java/org/unicode/cldr/util/data/langtagRegex.txt 
>
><http://unicode.org/cldr/data/tools/java/org/unicode/cldr/util/data/langtagTest.txt>http://unicode.org/cldr/data/tools/java/org/unicode/cldr/util/data/langtagTest.txt 
>Mark

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 17 Sep 2006 18:50:50 +0200

On Sat, Sep 16, 2006 at 04:28:01PM -0700,
 Mark Davis <[email-masked]> wrote 
 a message of 98 lines which said:

> BTW, I had updated my regex to the final spec for 4646. Here is a
> single Perl or Java regex that does most of the parse:

Isn't it too lax? When testing it in a Perl script, I find it accepts
all my well-formed tags (OK) but also accepts wrongly:

fr-Latn-F is well-formed
en-a-bbb-a-ccc is well-formed
tlh-a-b-foo is well-formed
abcdefghi-012345678 is well-formed
ab-abc-abc-abc-abc is well-formed
ab-abcd-abc is well-formed
ab-ab-abc is well-formed
ab-123-abc is well-formed
ab-abcde-abc is well-formed
ab-1abc-abc is well-formed
ab-ab-abcd is well-formed
ab-123-abcd is well-formed
ab-abcde-abcd is well-formed
ab-1abc-abcd is well-formed
ab-a-b is well-formed
ab-a-x is well-formed
ab--ab is well-formed
ab-abc- is well-formed
ab-c-abc-r-toto-c-abc is well-formed
abcd-efg is well-formed
aabbccddE is well-formed

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 17 Sep 2006 14:49:01 -0700

Fixed, thanks!

Mark

On 9/16/06, Martin Duerst <[email-masked]> wrote:
> >It checks for the grandfathered tags, since otherwise too much cruft
> sneaks in. You can't check in regex that there are only single instances of
> each singleton extension. (In retrospect we could have allowed multiple
> singletons: we could have accepted en-a-bcdef-ghijk-b-123 -a-lmnop as
> ><
> ><

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 17 Sep 2006 14:54:36 -0700

That's odd. Testing in Java on the file
http://unicode.org/cldr/data/tools/java/org/unicode/cldr/util/data/langtagTest.txt(which
includes those cases) the only failures I get are the ones that
repeat extensions (like en-a-bbb-a-ccc), which can't be tested for with
regex. I don't think I'm using any regex syntax that differs between Java
and Perl (there are very few instances).

Note: I added a number of mechanically generated items to the test file.

Mark

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 17 Sep 2006 14:57:28 -0700

BTW, here is the regex with the fix for the problem that Martin noticed, and
also restricted to the irregular grandfathered codes.

(?: ((?: [a-z A-Z]{2,3} (?: [-] [a-z A-Z]{3} ){0,3} | [a-z A-Z]{4,8} ))(?:
[-] ((?: [a-z A-Z]{4} )) )? (?: [-] ((?: [a-z A-Z]{2} | [0-9]{3} )) )? (?:
[-] ((?: (?: [0-9] [a-z A-Z 0-9]{3} | [a-z A-Z 0-9]{5,8} ) (?: [-] (?: [0-9]
[a-z A-Z 0-9]{3} | [a-z A-Z 0-9]{5,8} ) )* )) )? (?: [-] ((?: (?: [a-w y-z
A-W Y-Z] (?: [-] [a-z A-Z 0-9]{2,8} )+ ) (?: [-] (?: [a-w y-z A-W Y-Z] (?:
[-] [a-z A-Z 0-9]{2,8} )+ ) )* )) )? (?: [-] ((?: [xX] (?: [-] [a-z A-Z
0-9]{1,8} )+ )) )?  ) |    ((?: (?i)en [-] GB [-] oed|    i [-] (?: ami |
bnn | default | enochian | hak | klingon | lux | mingo | navajo | pwn | tao
| tay | tsu )|    sgn [-] (?: BE [-] fr | BE [-] nl | CH [-] de))) |    ((?:
[xX] (?: [-] [a-z A-Z 0-9]{1,8} )+ ))

Mark

On 9/17/06, Mark Davis <[email-masked]> wrote:
> http://unicode.org/cldr/data/tools/java/org/unicode/cldr/util/data/langtagTest.txt(which includes those cases) the only failures I get are the ones that
> On 9/17/06, Stephane Bortzmeyer < [email-masked]> wrote:
> >
> > Mark Davis < [email-masked]> wrote

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 19 Sep 2006 14:12:08 +0200

> http://unicode.org/cldr/data/tools/java/org/unicode/cldr/util/data/langtagTest.txt

May I ask a question about some of the tags in this file? 

xE-Lxs-qu is indicated as well-formed but I do not think so. Because
of "Lxs", it can only be and extension and therefore, "qu" is
forbidden (xE-Lxs is well-formed, as well as xE-Lxs-quT). 

(There are in your file several other tags with 2letters then
3letters, which have the same problem.)

---

From: John Cowan <[email-masked]>
Date: Tue, 19 Sep 2006 08:31:24 -0400

> forbidden (xE-Lxs is well-formed, as well as xE-Lxs-quT). 

No, it's well-formed all right:  language-extlang-region.  Nothing
wrong with that, as in "zh-hak-TW", the variety of Hakka Chinese
spoken on Taiwan.  Of course it won't be valid until 4646bis.

-- 
What asininity could I have uttered     John Cowan <[email-masked]>
that they applaud me thus?              http://www.ccil.org/~cowan
        --Phocion, Greek orator

---

From: Addison Phillips <[email-masked]>
Date: Tue, 19 Sep 2006 07:45:05 -0700

> forbidden (xE-Lxs is well-formed, as well as xE-Lxs-quT). 

"Lxs" would be an extlang and "qu" would be a region.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 19 Sep 2006 17:08:34 +0200

On Tue, Sep 19, 2006 at 07:45:05AM -0700,
 Addison Phillips <[email-masked]> wrote 

> "Lxs" would be an extlang and "qu" would be a region.

Yes, of course, my bug. Fixed, thanks.