Subject: [Ltru] Re: Registry in record-jar format

From: "Doug Ewell" <[email-masked]>
Date: Sat, 26 Mar 2005 08:24:24 -0800

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

>> You can now buy a 512 MB JumpDrive for 40 USD, or a 200 GB
>> hard drive for 100 USD
>
> I'm not planning to buy a new hard drive only to read a IANA
> registry with language tags ;-)

I said the registry would grow by 600,000 bytes.  I don't think that's
quite big enough to require you to buy a new hard drive.

My point was that 600,000 bytes isn't all that much in comparison to the
things we store and pass around nowadays.  That's less than half a
floppy disk, and everyone knows we don't use floppies any more because
they're considered too small to hold anything useful.

> I trust that IANA will split
> it when necessary on its Web pages, and as you said it's no
> problem for a 3066bis without the future ISO 639-3 languages.

Whether the registry gets split into pieces is, I suppose, IANA's
business.  I would have thought the extra hassle of downloading and
maintaining multiple files would militate against this.  And it's still
a non-issue until ISO 639-3 codes become available as extended-language
subtags, which won't happen until (a) ISO 639-3 is published and (b) the
registry RFC we are working on is updated to accommodate them.

-Doug Ewell
 Fullerton, California
 http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 26 Mar 2005 09:14:12 -0800

JFC (Jefsey) Morfin <jefsey at jefsey dot com> wrote:

> I extensively use my footer to evangelize about this WG's work (it
> informs people and leave them free to make their mind). I would
> appreciate (as well as anyone else maintaining a file on line of
> relevance to this work) to give the exact wording they wish I add for
> their file.

"Extensive" is the word, all right.  I have other words for it as well.

Your signature block:

* is 33 lines and almost 2000 characters long, far beyond the accepted
limits for Internet mail;

* implies that we are trying "to standardize languages and to unify the
world under a dominant culture," which is not only demonstrably
inaccurate but highly offensive as well;

* includes a Jon Postel quote, the relevance of which is puzzling at
best, since region subtags *are* based on ISO 3166 codes, at least as
much so as ccTLDs;

* includes a Brian Carpenter quote, also with dubious relevance, since
ccTLDs and regions in language tags are not "the same problem," and do
not therefore require the same solution, especially in the context of
"protocol functionality";

* makes a comparison between ISO 3166 and ISO "693" [sic] which is
puzzling and obscure even without the lingering typo.

Instead of adding even more wording, I would suggest a simple signature
block of no more than four lines (some say six is acceptable), with your
name, e-mail address, affiliation, maybe a single short quote, and
probably a link to the LTRU archive so interested people can visit,
join, and make their own judgments.  They don't need "evangelizing."

Now, back on topic.

>> (BTW, nobody has commented yet on my recipe for populating the
>> language subtags, so I assume there's nothing dreadfully wrong with
>> it.  I'll do the other sections in similar fashion, perhaps this
>> weekend.)
>
> Could you please make it a clean part that we could include in our
> Draft?

I'm working on it.  Easter weekend activities may prevent me from
posting additional installments in the next few days, but it is on its
way.

I think the current theory is indeed to put the description in the
draft, rather than relying on the pre-built registry.

> The problem as usual is the usage made of it, and the load imposed on
> IANA and private mirrors...
>
> Another point we try to consider is the developing countries and
> islands problems. ADSL US or European or Korean traffic costs nothing.
> But 600,0000 at real 500 characters per second or less is 1200
> seconds. When you know that in some countries one hour of internet
> equals one day work salary, you see the digital divide this represents
> for some of the really most involved searchers.

Happily, nobody will be required to download a fresh copy of the
registry every time they wish to read or write a language tag.  Section
4, "Security Considerations," says:

"Although the specification of valid subtags for an extension MUST be
available over the Internet, implementations SHOULD NOT mechanically
depend on it being always accessible, to prevent denial-of-service
attacks."

This applies to cost-of-access concerns as well as security concerns.

> Permitting people to exchange in different languages over real time
> information is of the essence for international assistance for
> example. So languages, terminology, etc. are an important elements.
> But at the same time we need to qualify that information, to keep it
> as dense as possible, to "culturalize it" to that end (because a DoS
> costs too much - and this is why IP is not a proper vehicle, and
> IPSec is costly). This is where you find that ISO 3166 is of
> importance but it does not address the diaspora, the immigration, de
> cultural, the corporate, the reader/viewers classes, etc. This is
> where you understand that Governments on_the_network are also
> perceived through their e-regalian embodiment of services - and
> possible interference with CRCs.

I don't see what 90% of this has to do with language tags -- surprise -- 
but I would suggest that ISO 3166-based region subtags used to indicate
"language variations used in a specific region, geographic, or political
area" apply equally well to a diaspora or to other types of immigrants.
That is, if you were a Frenchman living in Canada, your language variety
might well be "fr-FR" rather than "fr-CA", and in some cases it might be
important to tag that distinction.

Newer versions of Windows include a "region" setting in Control Panel,
distinct from the traditional "language variant/locale" setting that has
always been there.  The idea is that (assuming the above scenario) you
could set your language and other locale preferences to "French as used
in France," but still specify that you are physically in Canada for
applications where that matters.

> FYI our DIA effort will be dedicated to IEMS a small team of a WU shop
> who had started an inter-mail systems bridge. Mexico City known a big
> earth break and for hours all the international assistance
> coordination went though one of their customer's mail system to their
> ITT mailbox. No one really praised them, but many lifes and
> sufferances were spared that day because of their small async 300 bps
> service.

Their efforts are much appreciated.  Please reassure them that they will
not be required to perform a 20-minute download to tag their text.

-Doug Ewell
 Fullerton, California

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Sat, 26 Mar 2005 22:06:26 +0100

Dear Doug,
I will certainly be polite and address all your points, but I hope that at 
the end of the day you will read and accept what I say in responding your 
first part. We are both accustomed to the political aspects and tricks of 
normalization.

>"Extensive" is the word, all right.  I have other words for it as well.

"Of or relating to the cultivation of vast areas of land with a minimum of 
labor or expense". Yes.

>limits for Internet mail;

Reference?
1869 bytes isn't all that much in comparison to the things we store and 
pass around nowadays.

>inaccurate but highly offensive as well;

It implies nothing. It is a very lenient form of the comments I hear all 
the day long about this work. I am not sure why you oppose it?

>best, since region subtags *are* based on ISO 3166 codes, at least as

So, I do not see why it is puzzling?

>"protocol functionality";

IMHO this is inaccurate and highly offensive as well, if you accept to 
think about what you say implies.

>puzzling and obscure even without the lingering typo.

Thank you for correcting the typo. This remark is puzzling and obscure to me.

>join, and make their own judgments.  They don't need "evangelizing."

Thank you, dad, for your advise. We really live in two worlds appart :-)

>I'm working on it.  Easter weekend activities may prevent me from osting 
>additional installments in the next few days, but it is on its way.

Yes. Thank you. Same for me. I may have an extra week delay, because a lot 
of printing I ordered for our work is delayed: the printing shop closed for 
the Easter WE ....

>I think the current theory is indeed to put the description in the draft, 
>rather than relying on the pre-built registry.

IMHO it would be better in a second RFC. To preserve the weight of the main 
RFC (when quoted, printed). To permit to update that RFC separately.

>This applies to cost-of-access concerns as well as security concerns.

I am afraid you missed the concern. SHOULD NOT is not a MUST NOT. You 
should understand that the format will make a lot of difference. If the 
presentation is in XML the unnecessary usage will probably be 10 times 
larger. There is a good example: the http://ref-editor.org In 1200 days 
there have been 1.800.000, ie 1500 accesses a day. What is far lower. But 
you understand that 600.000 bytes are not the 3 to 10 000 bytes of an RFC.

This Draft must define all that in detail in IANA considerations. The 
solution requested must be workable. I know that our Chairs do not want 
Doug Barton to share in this WG, but we would certainly need to have 
precise inputs.

I want also to underline that the figure I quote are not for "every time", 
but very conservative (1 a year) as you call for.

>applications where that matters.

Thank you for the 10% (in standardization, there are not such a way as %, 
if there is a need it must be supported 100% or not). What you quote about 
Microsoft is interesting. But it looks like a way to tune an obligation 
imposed elsewhere? (if you speak French and are in Canada you are supposed 
to speak fr-CA?). But this is a good point. I recall you that my lang5tag 
proposition is not only in line with every dictionary, but also with Word 
and with the "preferences" quoted in the Charter.

jfc

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 26 Mar 2005 16:17:24 -0800

JFC (Jefsey) Morfin <jefsey at jefsey dot com> wrote:

> IMHO it would be better in a second RFC. To preserve the weight of the
> main RFC (when quoted, printed). To permit to update that RFC
> separately.

The registry will need to be updated whenever a new code element is
added to ISO 639, 3166, or 15924, or whenever the Language Subtag
Reviewer approves the registration of a variant subtag.  By my estimate,
this is anywhere from 3 to 12 times a year.  I don't think an RFC is the
right vehicle for something as dynamic as that.

> should understand that the format will make a lot of difference. If
> the presentation is in XML the unnecessary usage will probably be 10
> times larger. There is a good example: the http://ref-editor.org In
> 1200 days there have been 1.800.000, ie 1500 accesses a day. What is
> far lower. But you understand that 600.000 bytes are not the 3 to
> 10 000 bytes of an RFC.

I am not necessarily arguing in favor of the XML format, which is
clearly more bulky than the others being considered.  Nobody, least of
all the W3C, claims that XML is a terse and compact format.

However, we are talking about ISO 639-3 code elements used as
extended-language subtags, and it does need to be noted that there are
more than 7,000 of these and there is NO plain-text format that can
express them all without chewing up quite a few bytes.  I created a text
file from the tables in a draft version of 639-3, in about as compact a
format as you can imagine:

# Reference name|ID|Scope|Part 2|Status|Macro|
!O!ung|oun|I||||
!Xóõ|nmn|I||||
//Ani|hnh|I||||
//Gana|gnk|I||||
//Xegwi|xeg|I||||
/Gwi|gwj|I||||
/Xam|xam|I||||
etc.

and even this file, with 7,617 records (479 of which are already in
639-1 or 639-2 and thus won't need to be added to the registry), is more
than 161,000 bytes.  There's no getting around it; no matter how you
slice it up, no matter what format is chosen, a registry with seven
thousand languages is a large registry.

> I want also to underline that the figure I quote are not for "every
> time", but very conservative (1 a year) as you call for.

You are correct, and I did miss that.

I suppose a valid question might be, how often does an application need
to update its copy of the registry if it attempts to be "conformant"
with RFC 3066bis (however that is defined)?  Currently we have
applications that claim to support RFC 3066, but don't necessarily
support all registered tags, such as "bs-Latn" which was registered just
last month.  (For that matter, even the IANA pages don't list any of the
tags they have registered since mid-2004.)

As you mentioned, anti-virus software requires virus definition files to
be updated frequently for maximum effectiveness.  This is not the same
as requiring frequent updates in order for the software to work at all.
I wonder what the situation is with other applications that rely on data
from IANA registries.  Are they considered non-conformant if they don't
have the latest version, or just behind the times?

Maybe there should be a way to provide incremental updates to the
registry, so that the addition of one or two new subtags does not
require the whole registry to be replaced.  This is probably
application-specific, and out of our scope.

> What you quote about Microsoft is interesting. But it looks like a way
> to tune an obligation imposed elsewhere? (if you speak French and are
> in Canada you are supposed to speak fr-CA?).

No, that's the exact opposite of my point.  If you are from France and
travel to Canada, you probably still speak fr-FR, not fr-CA.  And you
may want locale settings, sduch as spelling dictionaries and sort order,
appropriate for France as well.  Microsoft's approach allows you to
specify that your physical location is Canada, but you want the settings
for France for language and other things.  (This is not to say only
Microsoft has addressed this issue; I'm sure other vendors have as
well.)

You had stated that "ISO 3166 is of importance but it does not address
the diaspora, the immigration, de cultural, the corporate, the
reader/viewers classes, etc."  I wanted to show that ISO 3166 codes in
language tags are orthogonal to the existence of a diaspora or other
immigrants.  People speak whatever language or variant they speak,
regardless of their physical location.

> But this is a good point. I recall you that my lang5tag proposition is
> not only in line with every dictionary, but also with Word and with
> the "preferences" quoted in the Charter.

We'll see what your draft says.  Being "in line with" Microsoft Word is
not a goal of the current draft, though.

-Doug Ewell
 Fullerton, California

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Sun, 27 Mar 2005 04:47:54 +0200

>right vehicle for something as dynamic as that.

What I mean is that:
- if you put the data in the BCP 047, it would become obsolete when there 
would be an ISO update.
- if you put them in a separate RFC, the IANA will load them. At the first 
change, you update the IANA according to the BCP047 procedure (new 
document) and obsolete the data RFC.

>!XÃ³Ãµ|nmn|I||||

We fully agree. I rose the question of the load when we started discussing 
XML. IMHO, but this may be subjective, this is why I asked, the least 
directly usable is the format, the least there will be chances that someone 
builds an application calling the IANA.

I will be frank IMHO the IANA registryis has only a political and and a 
central exposure interes: it would be better just to document where to find 
the data supported by their referent. We have the experience of the DNS 
database: the IANA delays/moods in getting the DNS root updated are the 
main problem of TLD Managers. Also, be sure that if you start building a 
formal system, imposing constraints, usage will just partly move away from 
it. Like for the DNS, with alt-roots and PADs.

NB. I keep refering to the DNS because (I quoted the STD 013) it could be 
the proper vehicle to support the data and permit immediate access by 
applications.

I do not know if I will be able to present/discuss our draft as we planned 
due to the Easter delay, so let me introduce the idea and it can be 
discussed before the draft (at this stage it is just a concept, but I have 
the tools and the possibly useful plug-in).
1. there are four descriptive subtags (language, script, culture, style)
2. there is an authoritative subtag (referent).

Let assume that the referent is designated by his domain 
name.  "style.culture.language.script.referent" can be a domain name where 
an information is stored (exists or not, to start with) in an RR. Exemple: 
"casual.fra.fr.latn.jefsey.com" will be where to find my description of the 
way I understand casual French of France. I can certainly have a CNAME with 
"casual.fr.fr.latn.jefsey.com".

The reason why I put the script first is that "latn.jefsey.com" can be 
transformed by a plug-in (currently used by scores of millions of users) 
into ".langtag" in Latin character. The call is then casual.fr.fr.langtag, 
with a RX being responded to the application documenting the registration 
(it can simply be the Handle of the documentation - permanent document 
addressing system used by the LOC and many others).

It just calls for a db.file description to immediately support 7500 
languages and 20000 dialects and all what you want. 15.000.000 tags are no 
big deal vs. hundreds of million of domain names supported by nearly one 
million of nameservers. And it costs nothing.

>No, that's the exact opposite of my point.

We are in agreement. My Franglish probably the problem. I meant you are 
supposed to use a Canadian locale, but Microft allows you to evade  from it.

How do you support Tamils?

>not a goal of the current draft, though.

You are right. I only assume that Word is probably in line with the users 
otherwise they would use something else. So being level with Words and 
Dictionaries (which usually document a word in giving the style, the 
reference and possibly the date) gives chances to be level wit the  real needs.

jfc

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 26 Mar 2005 21:14:45 -0800

JFC (Jefsey) Morfin <jefsey at jefsey dot com> wrote:

> What I mean is that:
> - if you put the data in the BCP 047, it would become obsolete when
> there would be an ISO update.

Of course.  I don't know anyone who wants to do that.

> - if you put them in a separate RFC, the IANA will load them. At the
> first change, you update the IANA according to the BCP047 procedure
> (new document) and obsolete the data RFC.

I don't think there should be a data RFC in the first place.

> I will be frank IMHO the IANA registryis has only a political and and
> a central exposure interes: it would be better just to document where
> to find the data supported by their referent.

I strongly disagree.  Nobody should have to go through all the tedious
and error-prone steps to derive the registry from all the various source
standards.  Take another look at
http://users.adelphia.net/~dewell/init-registry.html, and bear in mind
that's just the language subtags.  The region subtags will be worse.

I don't believe that any of us who support the creation of a registry do
so on the basis of any political agenda or desire for "exposure," if
that is what you mean.  I certainly do not have those goals.

> We have the experience of the DNS database: the IANA delays/moods in
> getting the DNS root updated are the main problem of TLD Managers.

I have to admit, I am worried about whether the subtag registry will get
updated in a timely manner, if the existing whole-tag registry hasn't
been updated since mid-2004 even though nine tags have been registered
since then.

> Also, be sure that if you start building a formal system, imposing
> constraints, usage will just partly move away from it. Like for the
> DNS, with alt-roots and PADs.

The "system" we are "building" is just a significant update to the RFC
1766/3066 system that already exists.  And yes, some people have abused
it.

Not to pick on Microsoft, because everyone else does, but the .NET
Framework has a class called CultureInfo, which has a property called
Name, which is a string that supposedly "follows the RFC 1766 standard."
Only it doesn't; it extends it in certain ways, by assigning names like
"zh-CHS" and "zh-CHT" for Simplified and Traditional Chinese, and using
the 3-letter "div" instead of "dv" for Divehi, and by using the
non-existent country code 'SP' for Serbia and Montenegro, and by using
language-region-script codes like "az-AZ-Cyrl" which aren't registered
under either RFC 1766 or 3066.

(I wish, oh how I wish, that 'SP' really were the country code that ISO
3166/MA had chosen to replace 'YU'.  But it isn't.)

So sure, somebody is bound to abuse RFC 3066bis.  I don't think that's a
convincing argument not to build it.

> I do not know if I will be able to present/discuss our draft as we
> planned due to the Easter delay, so let me introduce the idea and it
> can be discussed before the draft (at this stage it is just a concept,
> but I have the tools and the possibly useful plug-in).
> Let assume that the referent is designated by his domain name.
> "style.culture.language.script.referent" can be a domain name where an
> information is stored (exists or not, to start with) in an RR.
> Exemple: "casual.fra.fr.latn.jefsey.com" will be where to find my
> description of the way I understand casual French of France. I can
> certainly have a CNAME with "casual.fr.fr.latn.jefsey.com".

Others can discuss this now if they choose to.  I will wait for the
draft.

>>> What you quote about Microsoft is interesting. But it looks like a
>>> way to tune an obligation imposed elsewhere? (if you speak French
>>> and are in Canada you are supposed to speak fr-CA?).
>>
> We are in agreement. My Franglish probably the problem. I meant you
> are supposed to use a Canadian locale, but Microft allows you to evade
> from it.

I still don't understand.  Microsoft lets you use a France locale, and
simultaneously say, "but I am physically in Canada."  Maybe we should
just drop this.

> How do you support Tamils?

Very well, thank you.

No, seriously, there must be an inner meaning to this that I don't
understand.  You can write "ta" for Tamil, or "ta-IN" for Tamil as used
specifically in India, or "ta-LK" for Tamil in Sri Lanka, etc. etc.  You
can do that today, with RFC 3066.  Under RFC 3066bis you could also use
the script subtag 'Taml', or a different one if another script is being
used to write Tamil (I don't know if that happens).

>> We'll see what your draft says.  Being "in line with" Microsoft Word
>> is not a goal of the current draft, though.
>
> You are right. I only assume that Word is probably in line with the
> users otherwise they would use something else. So being level with
> Words and Dictionaries (which usually document a word in giving the
> style, the reference and possibly the date) gives chances to be level
> wit the  real needs.

Sorry, you lost me.  I use Word, at home because I think it's good
software, and at work because I have to.  As for dictionaries, I have
one of those huge Random House unabridged dictionaries and enjoy being
able to look up etymologies, origins, and dates of first usage.  But I
don't recall it classifying every word as to "style," except for obvious
non-standard words like "ain't," and I certainly wouldn't consider that
to be a "real need" in the sense that I would need that information from
the dictionary in order to form a sentence.

-Doug Ewell
 Fullerton, California

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 27 Mar 2005 19:20:26 +0200

> I said the registry would grow by 600,000 bytes.  I don't
> think that's quite big enough to require you to buy a new
> hard drive.

Sure, I was joking.  It would be serious if you were talking
about 50 MB.  Or maybe about cell phones / PDAs / etc.

> Whether the registry gets split into pieces is, I suppose,
> IANA's business.

Scott said that we should assume the worst, or at leaast that
was my interpretation.  One way to organize this could be a
"one tage = one file" subdirectory (+ ZIP or similar), for an
example see the xml2rfc subdirectories:

<http://xml.resource.org/public/rfc/bibxml/>
<http://xml.resource.org/public/rfc/txt/>

There are similar subdirectories for fresh I-Ds, other formats,
other popular references, etc.  One example you could need in
a future "draft registry" with xml2rfc is:

<http://xml.resource.org/public/rfc/bibxml3/reference.I-D.ietf-ltru-registry.xml>

> it's still a non-issue until ISO 639-3 codes become available
> as extended-language subtags, which won't happen until (a)
> ISO 639-3 is published and (b) the registry RFC we are 
> working on is updated to accommodate them.

Sure, but IANA and many users won't be happy if we change the
format of their langtag registry more than once in this decade.
Yes, I know when RfC 3066 was published.  Bye, Frank

---

From: "Jon Hanna" <[email-masked]>
Date: Sun, 27 Mar 2005 19:36:44 +0100

What is the "dumbest" device we're likely to want to have a cache of the
registry?

This clearly isn't an elevator-control matter, but how much smarter than
that would something have to be before having a access to the registry is
useful, and hence before the size of the cache becomes a significant matter?

Regards,
Jon Hanna
Work: <http://www.selkieweb.com/>
Play: <http://www.hackcraft.net/>
Chat: <irc://irc.freenode.net/selkie>

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Sun, 27 Mar 2005 23:56:49 +0200

>useful, and hence before the size of the cache becomes a significant matter?

Could the authors (or anyone have a good command of their intent) document 
the exact _use_ of the IANA files as they see it.

- an application reads a langtag.
- what does it do?
- when is it going to need a IANA originated data?
- will it be usually in the application code? will be it be updated in a 
parameter file? will it call the IANA? Should it call a cache/mirror? Then 
who should manage them?
- which applications will do that?
- how many times a day/month/year will this happen for a single user? or 
ISP? or designer?

For example, I understand that the non documented CLDR application mostly 
needs the langtag as a reference and will maintain its own files to be 
themselves used by OS programmers. So the load should be very low, unless 
someone develops an updating tool direct from the IANA and people use it at 
each reboot. I am not sure about HTML and XML applications? I am not sure 
about the need of updates? Will it be on a regular basis, or when an error 
is met an update will be requested? Should there be a solution to prevent 
an application to loop (like there are tools for Whois from hat I know)? etc.

We do have to document in detail the IANA section, and I am really at loss 
understanding what we are talking about. I maintain a daily review of the 
IANA files for ccTLDs (http://afrac.org/intldate.txt). It actually 
decreases the load on IANA since people know if there is an update of not 
for them to look. But if many people made the same thing as me for their 
own us, or if my cron had a problem, it could create an overload.

Without that information the whole Draft does not make any sense, because 
it is to document a IANA service and there are different strategies to 
address the need, depending on the load it represents.

jfc