Subject: [Ltru] Re: 3066ter encompassed languages

From: "Doug Ewell" <[email-masked]>
Date: Tue, 12 Sep 2006 22:06:15 -0700

Peter Constable <petercon at microsoft dot com> wrote:

> Btw, if you've had a chance to read the FDIS, you'll see that 
> "(generic)" and "(specific)" are no longer the distinguishing 
> qualifiers that are going to be used. Rather, the individual language 
> has no added qualifier, and "(macrolanguage)" gets added to the 
> macrolanguage. So, it would be "Malay" and "Malay (macrolanguage)".

I had not seen this.  You may need to remind me of the URL where the 
document can be found.

I dearly wish the downloadable tables 
(http://www.sil.org/iso639-3/download.asp) would be updated to reflect 
the changes you have described.

>> OK, everyone: If we do what Frank wants, we will have:
>>
>>    Type: language
>>    Subtag: ms
>>    Description: Malay
>>    Description: Malay (generic)
>
> I don't see a need for that duplication; and given that what we'd end 
> up with per the change I mentioned above is
>
>    Type: language
>    Subtag: ms
>    Description: Malay
>    Description: Malay (macrolanguage)
>    Added: 2005-10-16
>    Suppress-Script: Latn
>    ...
>    Type: extlang
>    Subtag: mly
>    Description: Malay
>    Added: xxxx-xx-xx
>    Prefix: ms
>
> I don't particularly care for having the same description show up in 
> two different entries for things that have different meanings.

+1.  Actually, this makes the confusion even worse if we have to keep 
the 639-2 name, because now there will be TWO subtags called simply 
"Malay" without qualification.

--
Doug Ewell
Fullerton, California, USA
http://users.adelphia.net/~dewell/
RFC 4645  *  UTN #14

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 13 Sep 2006 17:19:06 +0200

Doug Ewell wrote:
 
> You may need to remind me of the URL where the document can
> be found.

If it's "internal" the I* could appoint you as LTRU-liaison
to "whatever it takes to get read access".  Pointless if that
puts you under non-disclosure rules.

> this makes the confusion even worse if we have to keep
> the 639-2 name, because now there will be TWO subtags called
> simply "Malay" without qualification.

If one is type extlang and the other type language it's not
too bad - we have some identical descriptions with different
types.

For the "find a simple rule" project:  If an old description
matches the begin of a new description, only the (longer) new
description is registered.

Needs some tuning, it's not necessarily a question of "old"
vs. "new".  Let's keep the inversions, please.

Frank

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 14 Sep 2006 06:50:25 -0700

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

>> UTF-8 is far superior to markup, in my (biased, Unicadette) opinion, 
>> since it makes the file plain text in a well-known, well-declared, 
>> highly-desirable encoding.
>
> Yes.  Not the maximal Latin-1 compatibility IFF you're willing to 
> spend up to 7 octets.  But apparently the optimal encoding for maximal 
> ASCII compatibility.

Actually up to 8 octets for the NCR format:  [ & # x 2 0 1 9 ; ]

I don't think it's been established that Latin-1 compatibility is a goal 
for an IANA registry.

>> perhaps we should leave well enough alone.
>
> Yes, for other reasons like not annoying folks who implemented 4646 
> validators (probably there are many more than only Doug, Stephane, and 
> you).

I hope my earlier message made it clear that I do not mind redesigning 
my software for a different format, if the different format is judged 
better in some way.  Old-timers may remember that the Registry was once 
planned to be bar-delimited, one line per entry, and I had to change my 
software when we moved to record-jar.

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 14 Sep 2006 15:09:37 +0100

Hi 

I apologise if you have already discussed this; I'm playing catchup after a
Name Server error destroyed my email.

>From the a non-technical aspect, I think the record jar format of the
registry sucks.  Can anyone see a situation where users will want to import
the data into proprietary software - such as an Access database?  For my own
purposes, I would like to and the present format is not at all human/user
friendly.  A nice tab or comma delimited format would suit me.  Or maybe I
have missed a way to import the data (I tried many moons ago).

Best regards

Debbie Garside

> -----Original Message-----
> From: Doug Ewell [mailto:[email-masked]] 
> Sent: 14 September 2006 14:50
> To: LTRU Working Group
> Subject: [Ltru] Re: UTF-8
> 
> I don't think it's been established that Latin-1 
> compatibility is a goal for an IANA registry.
> 
> I hope my earlier message made it clear that I do not mind 
> redesigning my software for a different format, if the 
> different format is judged better in some way.  Old-timers 
> may remember that the Registry was once planned to be 
> bar-delimited, one line per entry, and I had to change my

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 14 Sep 2006 16:28:46 +0200

On Thu, Sep 14, 2006 at 03:09:37PM +0100,
 Debbie Garside <[email-masked]> wrote 
 a message of 80 lines which said:

> From the a non-technical aspect,

>From a technical aspect, remember that there are many possible formats
for structured data and there is little chance of a consensus, even a
rough one, on the "best" format. That's why, although I personnaly
prefer XML, I have sympathy for the people who say "Do not touch it,
do not reopen the discussion".

> Can anyone see a situation where users will want to import the data
> into proprietary software - such as an Access database?

Sure, see later for a possible solution.

> A nice tab or comma delimited format would suit me.

CSV and similar formats have two big problems, for the *authoritative*
copy of the registry:

1) They cannot handle structure of more than one level (not a problem
for the language tag registry)

2) And, more important, they are not self-documented. When you find a
JSON, XML or record-jar file, even if you have never seen these
formats, you can reverse-engineer them and the field names are
present. With CSV, you are lost.

> Or maybe I have missed a way to import the data (I tried many moons
> ago).

Since there will be no consensus and because there are many use-cases
such as "Import to Access" or "Process with awk", I suggest the
following:

1) Let the reference format be record-jar, in the name of stability,

2) Create somewhere a Web site with translation tools *and the
results* of these translations. That way, we will still have one
authoritative form but people looking for such or such formats will be
happy.

I do not know if IANA (which seems quite busy) can handle this but we,
as a community, certainly can. I presume we will not lack volunteers
to produce recordjar2anything converters in Awk, Visual Basic, Haskell
or Perl ?

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 14 Sep 2006 15:34:42 +0100

Good answer Stephane!

Let's hope someone creates those conversion tools.

Best regards

Debbie 

> From: Stephane Bortzmeyer [mailto:[email-masked]] 
> Sent: 14 September 2006 15:29
> To: Debbie Garside
> Cc: 'LTRU Working Group'
> Subject: Re: Registry Format (was UTF-8)
> 
> On Thu, Sep 14, 2006 at 03:09:37PM +0100,  Debbie Garside 
> <[email-masked]> wrote  a message of 80 lines which said:
> 
> From a technical aspect, remember that there are many 
> possible formats for structured data and there is little 
> chance of a consensus, even a rough one, on the "best" 
> format. That's why, although I personnaly prefer XML, I have 
> sympathy for the people who say "Do not touch it, do not 
> reopen the discussion".
> 
> CSV and similar formats have two big problems, for the 
> *authoritative* copy of the registry:
> 
> 1) They cannot handle structure of more than one level (not a 
> problem for the language tag registry)
> 
> 2) And, more important, they are not self-documented. When 
> you find a JSON, XML or record-jar file, even if you have 
> never seen these formats, you can reverse-engineer them and 
> the field names are present. With CSV, you are lost.
> 
> Since there will be no consensus and because there are many 
> use-cases such as "Import to Access" or "Process with awk", I 
> suggest the
> following:
> 
> results* of these translations. That way, we will still have 
> one authoritative form but people looking for such or such 
> formats will be happy.
> 
> I do not know if IANA (which seems quite busy) can handle 
> this but we, as a community, certainly can. I presume we will 
> not lack volunteers to produce recordjar2anything converters 
> in Awk, Visual Basic, Haskell or Perl ?
> 
> 
> 
>

---

From: John Cowan <[email-masked]>
Date: Thu, 14 Sep 2006 10:40:13 -0400

> for the language tag registry)

Actually, we do have such structure, though not explicitly.  Certain
line types such as "Prefix" can be repeated, implying a two level
structure Record >> Prefixes >> Prefix.

> present. With CSV, you are lost.

3) Newly proposed records are passed around in email through most of their
lifetimes: from proposer to list to LSR to IANA.  Passing single very long
lines through email is not a very reliable process.  Furthermore, people
often want to put commas into description/comment fields, requiring
extra messy conventions using quotes or backslashes.  Tabs are worse
yet, since they are very easily transformed into spaces by mistake.
Record-jar involves a small block of lines none of which has to be very
long, thanks to the wrapping convention.

(I might have proposed JSON if I'd known about it at the time.)

-- 
Do what you will,                       John Cowan
   this Life's a Fiction                [email-masked]
And is made up of                       http://www.ccil.org/~cowan
   Contradiction.  --William Blake

---

From: John Cowan <[email-masked]>
Date: Thu, 14 Sep 2006 10:42:46 -0400

> Actually up to 8 octets for the NCR format:  [ & # x 2 0 1 9 ; ]

Actually up to 11: [ & # x 1 0 F F F F ; ], although in practice nothing
more than 10 will ever be required.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
In computer science, we stand on each other's feet.
        --Brian K. Reid

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 14 Sep 2006 16:43:02 +0200

On Thu, Sep 14, 2006 at 03:34:42PM +0100,
 Debbie Garside <[email-masked]> wrote 

> Let's hope someone creates those conversion tools.

My vision was more "community-oriented" (aka "Do It Yourself and Then
Share"): a volunteer sets up and runs the Web site (AFNIC, my
employer, certainly can host it) and then various volunteers submit
tools and translations of the registry (not all tools will run on the
platform used for the Web site, so not all translations will be
produced by the Web site administrator).

Of course, this will be a best-effort-no-warranty project, the
authoritative registry being at IANA only.

"Better to light a candle, than to complain about darkness"

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 14 Sep 2006 15:53:19 +0100

Stephane wrote:

> employer, certainly can host it) and then various volunteers 
> submit tools and translations of the registry (not all tools 
> will run on the platform used for the Web site, so not all 
> translations will be produced by the Web site administrator).

I think that is an excellent idea with one exception, the site should host
the translation/conversion tools and a link to the IANA Registry; that way a
translation would always be up-to-date and there would be no registry
conflicts. The tool could then be used when new Registry items are inserted
or the user can make a decision to update "by hand" (although this option is
inadvisable IMHO as it introduces human error).

> Of course, this will be a best-effort-no-warranty project, 
> the authoritative registry being at IANA only.

This would not be an issue if the "alternative" registry is not hosted.

> "Better to light a candle, than to complain about darkness"

We have the candle, just need a lighter! ;-)

Debbie

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 14 Sep 2006 09:46:55 -0600

I agree; that's what we do for CLDR, for example. For example, for
French there is an XML file at:

http://unicode.org/cldr/data/common/main/fr.xml

Then there are translations (actually multiple ones, with different
slices of the data) at http://unicode.org/cldr/comparison_charts.html,
such as:

http://www.unicode.org/cldr/data/charts/summary/fr.html?hide

Mark

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 14 Sep 2006 18:13:24 +0200

John Cowan wrote:

> up to 11: [ & # x 1 0 F F F F ; ], although in practice
> nothing more than 10 will ever be required.

My "7" was for a Latin-1 friendly UTF-4, not escape sequences:

Hex. 86 91 90 9F 9F 9F 9F (lead byte + 6 hex. digits), in that
form a legacy text viewer or editor could display 191 visible
Latin-1 characters as is, instead of "only" 95 ASCII for UTF-8.

Of course we can't use this for the registry, and we also won't
try BOCU-1.  But maybe IANA should offer a gzip-ped version.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 15 Sep 2006 10:26:09 +0200

Debbie Garside wrote:

> the present format is not at all human/user friendly.

Matter of taste, I like it.

>  A nice tab or comma delimited format would suit me.

I've no clue how you would handle multiple descriptions,
comments, or prefixes in such formats, you'd need two
separators, a field separator (tab or comma), and some
secondary separator.  And for that you would have to
escape these two separators somehow if they appear as
ordinary characters (e.g. a comma in a description).

Just for fun I've created a gawk-script to convert the
registry to XML - it still needs a DTD.  Does that help
you ?  See below.

Frank
#! /usr/common/bin/gawk -f
#
# usage: ltru2xml.awk file > file.xml
#
#### remove leading and trailing spaces ###########################
function STRIP( STR )
{              sub( /^[\t ]+/, "", STR )
               sub( /[\t ]+$/, "", STR )
               return STR
}
#### error ########################################################
function FATAL( STR )
{              print "error " NR ": " STR
               return 1
}
#### save unfolded field body #####################################
function FIELD()
{              if ( NAME == "description" )   D[ ++DD ] = BODY
               else if ( NAME == "comments" ) C[ ++CC ] = BODY
               else if ( NAME == "prefix" )   P[ ++PP ] = BODY
               else if ( F[ NAME ] == "" )    F[ NAME ] = BODY
               else exit FATAL( NAME ": " BODY )
               return
}
#### output collected record fields ###############################
function READY()
{              T = F[ "type" ]
               if ( T == "" ) exit FATAL( "missing type" )
               L = "<" T

               A = F[ "added" ]
               if ( A == "" ) exit FATAL( "missing date" )
               L = L " date='" A "' "

               S = F[ "subtag" ]
               if ( S == "" ) S = F[ "tag" ]
               if ( S == "" ) exit FATAL( "missing tag" )
               if ( F[ "tag" ] != "" && F[ "subtag" ] != "" )
                    exit FATAL( "conflicting subtag " S )
               print L "tag='" S "'>"

               S = F[ "preferred-value" ]
               A = F[ "deprecated" ]
               if ( A != "" )
               {    L = "    <deprecated date='" A "' "
                    if ( S == "" ) print L "/>"
                    else print L "tag='" S "' />"
               }
               else if ( S != "" )
                    exit FATAL( "missing deprecated" )

               while ( PP )
                    print "    <prefix tag='" P[ PP-- ] "' />"

               while ( CC )
               {    L = "    <comment> " C[ CC-- ]
                    print L " </comment>"
               }

               if ( DD == 0 ) exit FATAL( "missing description" )
               while ( DD )
               {    L = "    <description> " D[ DD-- ]
                    print L " </description>"
               }

               S = F[ "suppress-script" ]
               if ( S != "" && T == "language" )
                    print "    <suppress tag='" S "' />"
               else if ( S != "" )
                    exit FATAL( "unexpected Suppress-Script" S )

               print "</" T ">"
               return
}
#### init #########################################################
BEGIN          {    L = "<?xml version=\"1.0\" "
                    L = L "encoding=\"US-ASCII\" "
                    print L "?>"

                    DOCT = "ltru"
               }

#### record #######################################################
/^%%$/         {    FIELD()

                    if ( NN++ )    READY()
                    else
                    {    L = "<" DOCT " date='"
                         A = F[ "file-date" ]
                         if ( A == "" )
                              exit FATAL( "missing File-Date" )
                         print L A "' />"
                    }

                    for ( NAME in F ) delete F[ NAME ]
                    NAME = "" ;    CC = 0    ;    PP = 0
                    BODY = "" ;    DD = 0    ;    next
               }
#### field ########################################################
/^[A-Za-z0-9]/ {    FIELD()
                    if ( ! match( $0, ":" )) exit FATAL( $0 )
                    NAME = tolower( substr( $0, 1, RSTART - 1 ))
                    BODY = STRIP( substr( $0, RSTART + 1 ))
                    next
               }
#### LWSP #########################################################
/^ /           {    BODY = BODY " " STRIP( $0 )
                    next
               }
#### toast ########################################################
               {    exit FATAL( $0 )
               }
###################################################################
END            {    if ( NN++ )
                    {    FIELD()   ;   READY()
                    }
                    if ( --NN ) print "</" DOCT ">"
                    print NN " records processed" > "/dev/tty"
               }

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 17 Sep 2006 18:17:53 +0200

On Fri, Sep 15, 2006 at 10:26:09AM +0200,
 Frank Ellermann <[email-masked]> wrote 
 a message of 160 lines which said:

> Just for fun I've created a gawk-script to convert the registry to
> XML

The output is not well-formed XML. Here is a patch:

--- ltru2xml.awk.orig   2006-09-17 18:08:18.656912781 +0200
+++ ltru2xml.awk        2006-09-17 18:08:10.536819185 +0200
@@ -89,7 +89,7 @@
                          A = F[ "file-date" ]
                          if ( A == "" )
                               exit FATAL( "missing File-Date" )
-                         print L A "' />"
+                         print L A "' >"
                     }
 
                     for ( NAME in F ) delete F[ NAME ]
@@ -116,4 +116,4 @@
                     }
                     if ( --NN ) print "</" DOCT ">"
                     print NN " records processed" > "/dev/tty"
-               }
\ No newline at end of file
+               }

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 17 Sep 2006 18:17:53 +0200

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 18 Sep 2006 06:09:36 +0200

> The output is not well-formed XML. Here is a patch:
[...]
> +                         print L A "' >"

Thanks, took me some time to find this after the W3C validator
told me that my DTD doesn't allow the very first <language> :-)

> \ No newline at end of file

Your gawk is rather pedantic with lineends.  That "}" is the
last line, why shouldn't it have a lineend ?

Frank