Subject: [Ltru] Re: Preparations for smart matching

From: "Doug Ewell" <[email-masked]>
Date: Sun, 20 Mar 2005 10:56:59 -0800

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

>> Have you read the new draft on "Matching Language
>> Identifiers"?
>> ...
>> It explains that the "default" matching algorithm is not the
>> only possibility.
>
> IIRC that's also the same as in draft 9 and earlier.

Section 2.2.1 explains that the matching algorithm which involves
truncating from the right is simply "the most common implementation,"
and Section 2.3 describes the possibility of choosing other algorithms.

The point is that we don't have to make the grand decision for everyone
of whether script, region, or variant is more important, and therefore
should appear ahead of the others, just to get matching to work.

>> "pu-rud-Blaz-FU-scrog-f-vlad-x-druz",

(we seem to agree on this point)

> So you always have a 6th colum, and for all variants it's the
> pointer to a language - as I wanted it, I only didn't see that
> you already have it.
>
> Could we rename "recommended_prefix" to "associated_tag", or
> something in this direction ?  Then the 6th column of languages
> would be free for a pointer to a default script.

I think you missed my point.  Trying to determine the "default" script
and perform matching based on this assessment is probably *not* a good
idea for those languages that are customarily written in more than one
script, and probably unnecessary for those that are not.

If I make a request for Mongolian-language content, script unspecified,
then I need to be prepared for the possibility that I will receive
content in either the Mongolian or Cyrillic script (and maybe even
Latin, since everything seems to get transliterated into Latin sooner or
later).  If I want to restrict the content to Cyrillic script, I need to
specify "mn-Cyrl" instead of just "mn".

Having my "mn" data limited to Mongolian script only, excluding the
Cyrillic data, because the LTRU WG decided for me that Mongolian was the
"default" script for the Mongolian language, is not a satisfactory
option.  Maybe I'm able to read both, and I really do want content in
either one.  If we decide for the user than "mn" is the same as
"mn-Mong", then we make it impossible to form a request for "Mongolian,
script unspecified."  (And no, I do not suggest an additional
syntactical construct like "mn-*" or "mn-{Mong, Cyrl}" to patch this
defect.)

In the opposite case, limiting "en" to mean "en-Latn" is probably an
unnecessary layer.  Yes, I know about Deseret and Shavian and the like,
but the fact is that the overwhelming majority of English-language data
is in the Latin script, and people who want English-language data in
other scripts will probably ask for it specifically.  Requests for "en"
alone typically will not fill the user's box with useless (to him)
Deseret and Shavian content, because there is so little of it.  Adding
this "default script" assumption to the matching logic for English
accomplishes little.

> At the moment the draft says:
>
> | The "recommended_prefix" field is empty, except where the
> | type is "variant"
>
> That could be changed to e.g.
>
>   The "recommended_prefix" is generally empty.  In the case of
>   a "variant" it's often the recommended "language" prefix, but
>   could be also a language-script, language-script-region, or
>   language-region prefix.
>
>   For the type "language" this field can be used to define a
>   default "script" for the correspoding "laguage", and then the
>   resulting tag or prefix language-sript is NOT RECOMMENDED.

I really don't think this buys us much.  Section 2.3, items 1 and 2 tell
the user to avoid subtags that don't add information.  I think that is
sufficient.  I'm not comfortable with explicitly recommending against
"en-Latn" the way we do with "iw" or "und".

> > (N.B.  Note the difference between "registry draft" and
> > "draft registry."  We need better terminology for this, don't
> > we?)
>
> As long as <http://users.adelphia.net/~dewell/lstreg.html> is
> the state of the art I don't care about the terminology.  If
> you'll submit it as an I-D it would be the "draft registry".

That's what I'm worried about.  "Draft registry" implies that it's an
I-D, which it's not, at least not at present.  But it seems to be more
than just a prototype.

>> Thus if you want to match "en-Latn-US-boont" with "en-boont",
>> it is enough to disassemble both tags and observe that all
>> the subtags in the second tag are also present in the first.
>
> That's IMFO not good enough.  If a client offers en-boont and
> en-misc (script "misc"), and the server has en-misc-US-boont,
> then the client should try en-misc, but not en-boont.
>
> Because "misc" is probably very important, if it's not "Latn"
> as in the default for en.

But "boont" may also be important, because most English is not
Boontling.

Variants are a catch-all category used for all kinds of things, anywhere
from relatively minor orthographic variations ('1901' and '1996') to
regional dialects ('boont' and 'scouse') to completely different
languages ('guoyu' and 'xiang').  The presence of a variant subtag, in
and of itself, really doesn't tell much about whether the tagger
considered the variation to be the most important detail or the least.

Boontling is cute and fun, but I'd really rather not use it as the
metric by which important decisions about tagging and matching are made.

>> If "en-Latn-US-boont" is a match for "en-boont", then so is
>> "en-Ogam-MV-boont".
>
> Unless you have en-Ogam, then you need to know the default
> script for en (which isn't Ogam).

Suppose there were such a thing as Boontling written in Ogham.  (I can't
believe I just wrote that sentence.)  Now if a user requests "en-boont",
she may want this content anyway, even though it's in Ogham and not
Latin script.  Who are we to say she can't read Ogham?  And if another
user requests "en-Ogam", he may want this Boontling content as well.
The proportion of English (any flavor) written in Ogham is likely to be
so small that this may be just what he was looking for.

My point is that this is not for us to dictate.

>> the project of determining default scripts for *all*
>> languages (not just the 80% or 90% for which this task is
>> easy) is not something that anyone on this list, or perhaps
>> anywhere else, is qualified to tackle.
>
> *all* is not what I proposed.  I'm only interested in the 80%
> where it's easy for humans, but not for parsers.

I don't think most people will be satisfied with the result if we solve
this problem for the obvious cases, but not the marginal ones.  The
nature of this group is to pay attention to the edge cases.  Look how
much attention we are paying to Boontling!

-Doug Ewell
 Fullerton, California
 http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 21 Mar 2005 02:48:07 +0100

> (we seem to agree on this point)

Yes.  Out of curiosity, what's the idea of ISO 639-3 for the
reserved "extlang" ?  Is it something in the general direction
of my "region codes in language tags considered harmful" idea ?

> I think you missed my point.  Trying to determine the
> "default" script and perform matching based on this
> assessment is probably *not* a good idea for those languages
> that are customarily written in more than one script

No, that point is clear.  We don't want a mad tag reviewer who
sees no problem in deciding what the default script is where
it's _not_ obvious.  In that case we prefer no default at all.
Add TINW where necessary.

> and probably unnecessary for those that are not.

That's what I think is incorrect, see the parallel thread with
Debbie with issues like legacy software, audio data, limited
space in RfC 2231 header fields, etc.

I'd also like to get as much info as possible, but above all I
like correct data, no guesses, and backwards compatible data
where possible.  If 3066bis makes it this year I'm confident
that compatibility issues with RfC 3066 begin to be irrelevant
after 2038 with its signed time_t 32 bits issue.

So for the next 33 years I want compatibility where possible.
Your "mn" example is clear, it's not always possible to get
full backwards compatibility.  I don't want a default script
for "mn" or similar languages, where it's not obvious.

> Section 2.3, items 1 and 2 tell the user to avoid subtags
> that don't add information.

It's not always the user who tags data, often it's software.
As user you would probably never say "utf-8 7bit".  We need
defaults for software implementors, not for ordinary users.

 ["draft registry"]
> it seems to be more than just a prototype.

It's just a nice way to see what the actual draft really means.

For an I-D and IANA I'd prefer a format as in the IANA charset
registry.  Maybe with "%%" lines instead of empty lines as the
terminator for individual entries.  I forgot the name for this
format and who proposed <CRLF>%%<CRLF> instead of <CRLF>CRLF>.

  [en-misc-US-boont]
>> "misc" is probably very important, if it's not "Latn"
 
> But "boont" may also be important, because most English is
> not Boontling.

Yes, probably they are both important.  If a client supports
both en-misc and en-boont but not en-misc-boont let's hope
that it "knows" the best alternative or offers a choice.

I still think that the prcocessing of en-Latn-US-boont or the
"canonical" en-boont could be different from the processing
of en-misc-US-boont / en-misc-boont.

E.g a user could configure somewhere that he only knows Latn.
Or that he's really interested in misc, not Latn.  Or that he
loves en-boont independent of the script.
 
> Boontling is cute and fun, but I'd really rather not use it
> as the metric by which important decisions about tagging and
> matching are made.

Yes, I have no idea what "boont" is, that's a problem.  Maybe
the draft should use another standard example for matching
strategies and dialects (my editor counts 17 occurences in 13
lines for "boont" in draft -00)

> Who are we to say she can't read Ogham?

Of course we won't say this.  We only want existing RfC 3066
software to do the right thing for "Latn".  This software has
no idea what "Latn" is.  <joke> Maybe it asks whois.iana.org
why they now use a funny FOURCC for the country codes </joke>

                         Bye, Frank

---

From: John Cowan <[email-masked]>
Date: Sun, 20 Mar 2005 21:45:03 -0500

This posting is mostly for background information.  The only immediately
WG-relevant point is that I agree with Frank Ellerman that using "boont"
so heavily as an example in the draft is a bad idea.

> of my "region codes in language tags considered harmful" idea ?

<unofficial> ISO 639-3 will be (if all goes as expected) an extension
of ISO 639-2 to provide a 3-letter code, compatible with ISO 639-2,
for every known language in the world (about 7000).  Most of the
work has already been done, and what remains is basically cranking the
standardization machinery.

Some ISO 639-1/2 codes are, from the viewpoint of ISO 639-3, actually
cover terms for multiple languages.  For example, "zh" actually covers
13 distinct ISO 639-3 languages: czo, mnp, yue, hsn, wuu, cpx, gan, nan,
cmn, cjy, czh, hak, cdo.  Instead of allowing these as initial tags in RFC
3066ter, the intent is to allow zh-czo, zh-mnp, zh-yue, etc.  (This will
mean that several grandfathered zh-??? tags will become redundant instead.)

The reason for putting this facility off to RFC 3066ter is that ISO 639-3
is only an editor's draft at this point.  The reason for including the
syntax in 3066bis is to make 3066bis parsers 3066ter-ready.  </unofficial>

> format and who proposed <CRLF>%%<CRLF> instead of <CRLF>CRLF>.

"Record-jar" format, a merger of RFC-822 headers and the cookie-jar
format devised by Ken Arnold.

> Yes, I have no idea what "boont" is, that's a problem.  

Most other people don't either.  It is a jargon or Rotwelsch used by most
of the people living in the small and (then) isolated town of Booneville,
California between 1880 and 1920, and sporadically by hobbyists since
then.  The syntax is mostly that of informal American English,
but the jargon vocabulary is so all-encompassing as to make Boontling
(itself a jargon term) mutually unintelligible with English.

Here are a nursery rhyme and part of a mushroom-collecting guide:

#	The eeld'm piked for the chigrel nook
#		For gorms for her bahl beljeemer;
#	The gorms had shied, the nook was strung,
#		And the bahl beljeemer had nemer.

Literal translation:  The old woman went to the cupboard/For food for
her excellent hunting dog/The food was gone, the cupboard was empty/
And the excellent hunting dog had nothing.

Standard English version: Old Mother Hubbard/Went to the cupboard/
To fetch her poor dog a bone/But when she came there/The cupboard was
bare/And so the poor dog had none.

#	You must do much graymatterin fore pikin for seekin Ite steaks
#	to gorm, cause the sockers might not be bahlers, but nonchers
#	with dusties dust, so deek your bok well.

Literal translation:  You must do much thinking before going out to
look for mushrooms (literally "Italian steaks") to eat, because the
mushrooms might not be good ones, but bad ones with poison (literally
"graveyard dust"), so read your manual well.

Don't be troubled that you can't understand it; anglophones can't either.

-- 
Clear?  Huh!  Why a four-year-old child         John Cowan
could understand this report.  Run out          [email-masked]
and find me a four-year-old child.  I           http://www.ccil.org/~cowan
can't make head or tail out of it.              http://www.reutershealth.com
        --Rufus T. Firefly on government reports

---

From: "Randy Presuhn" <[email-masked]>
Date: Sun, 20 Mar 2005 20:03:04 -0800

Hi -

> From: "John Cowan" <[email-masked]>
> To: "Frank Ellermann" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Sunday, March 20, 2005 6:45 PM
> Subject: [Ltru] [semi-OT] ISO 639-3, record-jars, and Boontling
...
...

(co-chair hat OFF)
Do you have a rough idea how many ISO 639-1/2 codes will be "cover terms"
from the perspective of ISO 639-3?

Is the expectation that these other cases would all be handled in the same
way as (zh-cao instead of cao)?

If we already have a good idea how the ISO specifications will look, wouldn't
it make sense to ensure that 3066bis works correctly for those cases, so
we don't have to bother doing a 3066ter?

Randy

---

From: John Cowan <[email-masked]>
Date: Sun, 20 Mar 2005 23:41:13 -0500

> from the perspective of ISO 639-3?

The current draft contains 56 of them; the technical ISO 639-3 term is
"macro-languages".  A total of 365 ISO 639-3 languages would be covered
by a macro-language code.  They do not overlap.

> Is the expectation that these other cases would all be handled in the same way

Yes.  Similarly, fy-fri (West Frisian), fy-frs (East Frisian), and fy-frr
(North Frisian), rather than fri, frs, frr.  Not all macro-languages have
ISO 639-1 codes: the three Baluchi languages would be bal-bcc (Southern),
bal-bgn (Western), bal-bgp (Eastern).

There would be a few exceptions due to ISO 639 grandfathering: the ISO
639-1 codes nb (Bokmal Norwegian) and nn (Nynorsk Norwegian) would be
preferred in RFC 3066ter to no-nob and no-nno.

> as (zh-cao instead of cao)?

(Presumably zh-czo (Min Zhong) or zh-cdo (Min Dong).  cao is a different
language altogether, not covered by any macro-language code.)

> we don't have to bother doing a 3066ter?

I think everyone is okay with that except for the timing issue.

Currently, ISO 639-3 is an editor's draft, the ISO equivalent of an I-D.
There is no guarantee that it would make it through the ISO process
before RFC 3066bis makes it through the IETF BCP process, nor that the
current concepts will survive intact.  Fortunately, Peter Constable is
the ISO 639-3 editor, and has been able to bring enormous energy to the
task so far.

>From what I understand of the IETF process, this WG could go dormant
until ISO 639-3 was released, and then wake up for a relatively brief
period to incorporate the necessary few paragraphs in RFC 3066bis to
create RFC 3066ter.

-- 
John Cowan  www.ccil.org/~cowan  [email-masked]  www.reutershealth.com
[P]olice in many lands are now complaining that local arrestees are insisting
on having their Miranda rights read to them, just like perps in American TV
cop shows.  When it's explained to them that they are in a different country,
where those rights do not exist, they become outraged.  --Neal Stephenson

---

From: "Randy Presuhn" <[email-masked]>
Date: Sun, 20 Mar 2005 21:41:06 -0800

Hi -

(co-chair hat off)

> To: "Randy Presuhn" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Sunday, March 20, 2005 8:41 PM
> Subject: Re: [Ltru] [semi-OT] ISO 639-3, record-jars, and Boontling
...
...

I think it would be really nice if we could put in text now that would not
need revision later.  It sounds like all that is needed is text that spells
out precisely when using the macro-language code is required.

However, I wonder if the approach of mandating the use of the macro-language
code in some, but not all, cases will make matching algorithms overly complicated.

Randy

---

From: John Cowan <[email-masked]>
Date: Mon, 21 Mar 2005 02:15:34 -0500

> out precisely when using the macro-language code is required.

The difficulty there is that we put the extended-language concept in to 
support a specific mechanism of ISO 639-3 as it is currently conceived of.
If ISO/IEC JTC 1 / TC 37 / SC 2 / WG 1 changed its mind about this
mechanism, we'd be left hanging out to dry.

That said, it seems I was incorrect to say that ISO 639-3 is an editor's
draft: it is now a Draft International Standard out for ballot.

> code in some, but not all, cases will make matching algorithms overly complicated.

I don't think so.  It's much like mandating the use of ISO 639-1 codes where
they exist in place of ISO 639-2 codes.  It means that you need to know
more to assign a code, but not more to match on one.

-- 
John Cowan  [email-masked]  www.reutershealth.com  www.ccil.org/~cowan
This great college [Trinity], of this ancient university [Cambridge],
has seen some strange sights. It has seen Wordsworth drunk and Porson
sober. And here am I, a better poet than Porson, and a better scholar
than Wordsworth, somewhere betwixt and between.  --A.E. Housman

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 21 Mar 2005 09:04:05 +0100

John Cowan wrote:
 
> This posting is mostly for background information.

Very interesting, thanks.  It promises that I'll be able to say
what I mean instead of ad hoc (ab)uses of country codes or UN
numbers to identify regional differences.

> the intent is to allow zh-czo, zh-mnp, zh-yue, etc.

It's probably difficult to find the appropriate sets, but not
impossible.

> "Record-jar" format, a merger of RFC-822 headers and the
> cookie-jar format devised by Ken Arnold.

Yes, that was the name.  It's not strictly necessary for Doug's
registry, if he has something like Section: Language, then the
start of the next registry entry is always clear, even without
empty line or "%%".  But if the "record-jar" format is common,
why not.

> It is a jargon or Rotwelsch

Just for fun, here's a difficult case for a future ISO 639-3:
<http://www.ethnologue.com/show_language.asp?code=YEC>
"German-Yiddish-Romani-Rotwelsch"

>#               And the bahl beljeemer had nemer.

I see, whatever we do, we must try hard that it never matches
"en" as long as there is any better choice.

                         Bye, Frank

---

From: John Cowan <[email-masked]>
Date: Mon, 21 Mar 2005 10:08:36 -0500

> "German-Yiddish-Romani-Rotwelsch"

ISO 639-3 is basically Ethnologue with some of the codes changed for
various reasons.

> "en" as long as there is any better choice.

Indeed.  That's why it really should be registered as a language IMHO.

-- 
John Cowan  [email-masked]  www.reutershealth.com  ccil.org/~cowan
Dievas dave dantis; Dievas duos duonos          --Lithuanian proverb
Deus dedit dentes; deus dabit panem             --Latin version thereof
Deity donated dentition;
  deity'll donate doughnuts                     --English version by Muke Tever
God gave gums; God'll give granary              --Version by Mat McVeagh

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 22 Mar 2005 10:12:31 +0100

John Cowan wrote:
 
> ISO 639-3 is basically Ethnologue with some of the codes
> changed for various reasons.

And where does it place YEC, de-yec, yi-yec, gem-yec, ... ?

RfC 3066 software seeing YEC as "de" content would be a PITA,
lust like en-boont as a part of "en".

                        Bye, Frank

---

From: John Cowan <[email-masked]>
Date: Tue, 22 Mar 2005 08:49:48 -0500

Frank Ellermann scripsit:

> And where does [RFC 3066ter] place YEC,

"yec" is indeed the correct code for Yeniche in RFC 3066ter.

> de-yec,

Nobody uses "German" as a cover term to include both German and Yeniche.
Consequently, the tag "de" refers to modern standard German and its regional
variants (but not the German dialects proper, which have their own codes).

> yi-yec,

Nobody uses "Yiddish" as a cover term for "Yiddish and Yeniche" either.
However, "Yiddish" *is* used as a cover term for both Eastern Yiddish and
the nearly extinct Western Yiddish (formerly coterminous with German).
Consequently, RFC 3066ter would tag these yi-ydd and yi-yih respectively.
Of course, yi will still be usable when the distinction is unimportant.

> gem-yec,

gem is a 639-2 code for a language collection, the Germanic languages.
In current practice, it is only used when more specific 639-2 codes
fail to apply.  The expectation is that language-collection codes will
be rarely if ever used in RFC 3066ter practice.

> just like en-boont as a part of "en".

Indeed.

-- 
Go, and never darken my towels again!           John Cowan
        --Rufus T. Firefly                      www.ccil.org/~cowan

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 22 Mar 2005 16:18:51 +0100

> "yec" is indeed the correct code for Yeniche in RFC 3066ter.

Hm, you said that 639-3 codes are organized into about 50 (?)
639-1/2 "sets", and that 3066ter would use the 639-3 code as
an "extlang" together with the "set" as "language.  That's
why I'd expect something like ??-yec or ???-yec in 3066ter.

But now you're saying that 639-3 yec will be a language in
3066ter, where does that leave the reserved 3066bis "extlang" ?

Do you have an example where "extlang" is needed ?  Bye, Frank

---

From: John Cowan <[email-masked]>
Date: Tue, 22 Mar 2005 11:54:04 -0500

> why I'd expect something like ??-yec or ???-yec in 3066ter.

Ah, I see.  No, I said that about 350 of the nearly 7000 languages in
639-3 are included in about 50 macro-languages listed in ISO 639-1/2 as
well as 639-3 itself.  These will be coded in 3066ter using xx-yyy codes.
The remaining languages of 639-3 will be coded just like the languages
of 639-1/2, as a language appearing in both 639-2 and 639-3 will have
the same 3-alpha code in both parts.

> 3066ter, where does that leave the reserved 3066bis "extlang" ?

The extlang syntax is solely for use with the 56 macro-languages.

> Do you have an example where "extlang" is needed ?  

For example, "ar" is an ISO 639-1/2 code meaning "Arabic".  639-3
interprets this code as a macro-language.  Normally, it refers to modern
standard Arabic, but it is also used as a cover term for 30 Arabic
languages ("dialects", "colloquials") listed in ISO 639-3.

In an RFC 3066ter regime, one could tag something as "ar" to say that
it was Arabic generally, or "ar-arb" to refer specifically to modern
standard Arabic, or "ar-arz" to refer to Egyptian Spoken Arabic, or
"ar-shu" to refer to Chadian Spoken Arabic (which is barely, if at all,
mutually intelligible with ar-arb).

(Note: the term "Spoken" in the names of these languages does not mean
that they are not or cannot be written, but simply that they are not
generally written in the countries named; ar-arb or another language
altogether is normally used for writing.)

Maltese, although linguistically it is just another Arabic colloquial,
does not function as one sociolinguistically -- nobody refers to it as
"Arabic" -- and in RFC 3066ter it would be coded as "mlt", its ISO
639-3 code.  No extlang code is required or permitted.

The various Chinese, Nahuatl, Zapotec, Quechua, Romany, Sardinian, and
Albanian languages are treated in exactly the same way.  Cantonese is
"zh-yue" rather than just "yue" because it is thought of as part of the
macro-language called ""Chinese; Dungan (which linguistically is just
a Mandarin dialect) is not, and so its code is "dng".

Likewise, the four main Sardinian languages are "sc-sdn" (Gallurese),
"sc-sdc" (Sassarese), "sc-src" (Logudorese), and "sc-sro" (Campidanese),
but Algherese is a dialect of Catalan, and coded as "ca", or "ca-alghero"
if someone finds the distinction important and registers "alghero".

-- 
John Cowan  [email-masked]  www.ccil.org/~cowan  www.reutershealth.com
And now here I was, in a country where a right to say how the country should
be governed was restricted to six persons in each thousand of its population.
For the nine hundred and ninety-four to express dissatisfaction with the
regnant system and propose to change it, would have made the whole six
shudder as one man, it would have been so disloyal, so dishonorable, such
putrid black treason.  --Mark Twain's Connecticut Yankee