Subject: [Ltru] To invert or not to invert?

From: "Doug Ewell" <[email-masked]>
Date: Fri, 8 Dec 2006 06:23:13 -0800

The new RFC 4646bis draft does finally touch on the issue of mapping the 
language names from ISO 639-3 to Description fields in the Registry. 
But there is still some wiggle room.  From Section 3.1.4:

   Each 'Description' field MUST be unique, both within the record in
   which it appears and for the collection of records of the same type.
   Moreover, formatting variations of the same description MUST NOT
   occur in that specific record or in any other record of the same
   type.  For example, while the ISO 639-1 code 'fy' contains both the
   descriptions "Western Frisian" and "Frisian, Western", only one of
   these descriptions appears in the registry.

   ...

   When creating a new registry entry, duplicate, redundant,
   conflicting, or otherwise problematic descriptions MUST either be
   corrected or omitted.  Parenthetical comments, inverted names, and
   other irregularities SHOULD be regularized according to the
   guidelines used to update the registry in [registry-update].

The first passage indicates that EITHER the inverted form of a name OR 
the uninverted form must appear as a Description.  The second passage 
speaks of "regularizing" inverted names, which suggests that it is the 
uninverted name which is used, but does not say so explicitly, and it 
essentially pushes the problem to RFC 4645bis, which is where it belongs 
as far as the bulk update is concerned.

Either the inverted or uninverted name would equally serve the purpose 
of matching the Description fields for a language subtag to the ISO 
standard,  I haven't seen a requirement from anyone that *all* of the 
various ISO names, inverted and uninverted, must be matched.

What is the preference of the WG on this issue?  Should the Registry use 
"Western Frisian" or "Frisian, Western", given that it must use one and 
only one?  I can do it either way with roughly equal ease.  Please send 
your preference to the list.  I'd like to do a 4645bis draft this 
weekend.  If I don't hear a strong preference for "inverted," then I'll 
go with "uninverted" since that seems to be the direction in which 
4646bis leans.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Addison Phillips <[email-masked]>
Date: Fri, 08 Dec 2006 09:45:31 -0800

Hi Doug,

Put me down in favor of uninversion of names.

> as far as the bulk update is concerned.

I'm not sure that it suggests un-inversion specifically. It just as well 
suggests (per a long thread on this list) that inversion, when it 
appears, should be consistent in form/format. Either approach would be 
correct, but I'm not wild about creating monsters like "Slavonic, 
Church, Old" or even preserving the current "Slavonic, Old Church".

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: John Cowan <[email-masked]>
Date: Sun, 10 Dec 2006 01:31:03 -0500

Doug Ewell scripsit:

> only one?  

I still plump for "Frisian, Western", from which "Western Frisian" can
be reliably reconstructed, but not vice versa.  To exclude inverted
names is to throw away potentially valuable information.

-- 
How they ever reached any  conclusion at all    <[email-masked]>
is starkly unknowable to the human mind.        http://www.ccil.org/~cowan
        --"Backstage Lensman", Randall Garrett

---

From: "Kent Karlsson" <[email-masked]>
Date: Sun, 10 Dec 2006 14:12:49 +0100

> names is to throw away potentially valuable information.

Maybe, but there's a LOT of "potentially useful information" about
languages that are not at all covered by the registry.

Also, when translating the language names (as done in CLDR),
translating "Western Frisian" to "västfrisiska" is straightforward,
while translating "Frisian, Western" to "frisiska, väst-" is clumsy
and reconstructing the uninverted name means also to know when
to and when not insert a space (admittedly easy, just check if there
is a hyphen at the end or not), and when to and when not to delete
the hyphen (this may be language and case by case dependent).

	/kent k

---

From: John Cowan <[email-masked]>
Date: Sun, 10 Dec 2006 14:15:21 -0500

> languages that are not at all covered by the registry.

I was speaking of information contained in 639-3, not information in
Ethnologue or floating freely somewhere.

> the hyphen (this may be language and case by case dependent).

Uninvert first, translate, re-invert if it makes sense in the new
language.  I am concerned only with the LSR, which is fundamentally
in English.

-- 
Winter:  MIT,                                   John Cowan
Keio, INRIA,                                    [email-masked]
Issue lots of Drafts.                           http://www.ccil.org/~cowan
So much more to understand!
Might simplicity return?                        (A "tanka", or extended haiku)

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 10 Dec 2006 15:44:03 -0800

>I am concerned only with the LSR, which is fundamentally
in English.

You may only be concerned with English, but a good many of us are concerned
with other languages, and translations into those languages. Even in
English, for a full index, one would want all reasonable variants; that is,
I would want to find "Old Church Slavonic" listed under

Old Church Slavonic
  AND
Church Slavonic, Old
  AND
Slavonic, Old Church

As Kent says, the inversion may be very different in other languages, so one
MUST be able to extract the uninverted name mechanically.

Now, the latest version of 639-3 data has made very significant
improvements, which make me much more comfortable with inverted text. But we
still need to decide among the viable options. I can see the following:

1. List both inverted and uninverted forms under the same tag; also ensure
that the uninverted is first, eg

Description: Old Church Slavonic
Description: Church Slavonic, Old
Description: Slavonic, Old Church

2. List the inverted forms under a separate field, eg

Description: Old Church Slavonic
Inverted-Description: Church Slavonic, Old
Inverted-Description: Slavonic, Old Church

3. Only list the (really "an") inverted form

Description: Slavonic, Old Church

In all cases, we should verify that comma outside of parens *only* occurs in
the case of inversion.

Mark

---

From: John Cowan <[email-masked]>
Date: Sun, 10 Dec 2006 20:16:05 -0500

Mark Davis scripsit:

> You may only be concerned with English, 

Not what I said.  I am concerned with the LSR.

> I would want to find "Old Church Slavonic" listed under

I think the third is very misleading, and would not wish to see it.
The language is the old variety of Church Slavonic, not the old-church
variety of Slavonic.  This is precisely why inversion cannot be done
mechanically.

> As Kent says, the inversion may be very different in other languages,
> so one MUST be able to extract the uninverted name mechanically.

I agree absolutely, which is why I want to see the inverted names used --
there is no method of inverting the uninverted form.

> 1. List both inverted and uninverted forms under the same tag; also
> ensure that the uninverted is first,

I continue to believe that order of field instances should never be
treated as significant.

> 2. List the inverted forms under a separate field, 

I could live with this.

> 3. Only list the (really "an") inverted form

This is what I favor.  I doubt that there are many languages, if any,
that could be inverted in more than one way with appropriate semantics.

> In all cases, we should verify that comma outside of parens *only*
> occurs in the case of inversion.

That has already been done by the Official Doug.

-- 
We call nothing profound                        [email-masked]
that is not wittily expressed.                  John Cowan
        --Northrop Frye (improved)

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 10 Dec 2006 17:42:46 -0800

one comment

> I continue to believe that order of field instances should never be

And I continue to believe that they must be. When translating, we simply
*can't* translate all of the lines, so I don't see any choice but to use the
first one as the main description, and trust that that is a reasonable
choice. You can make lives more difficult for users if they are in random
order, that's all.

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 10 Dec 2006 20:37:26 -0800

"Slavonic, Old Church" is only a bad choice if people would never look it up
by "Slavonic". I don't want to belabor the point, since Slavonic may or may
not be a good example, but there are plenty of others.

Mark

On 12/10/06, Mark Davis <[email-masked]> wrote:
>
> one comment
>
> On 12/10/06, John Cowan < [email-masked]> wrote:
> >
> > Mark Davis scripsit:
> >

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 10 Dec 2006 23:00:31 -0800

Mark Davis wrote:

> Even in English, for a full index, one would want all reasonable 
> variants; that is, I would want to find "Old Church Slavonic" listed 
> under
>
> Old Church Slavonic
>   AND
> Church Slavonic, Old
>   AND
> Slavonic, Old Church

In that case, you don't need inverted forms in the Registry; you can 
create all of them yourself algorithmically.  It's only if you want 
certain inversions, such as "Slavonic, Old Church" (which is in ISO/FDIS 
639-3), and don't want others, such as "Church Slavonic, Old" (which 
isn't), that the ones you want have to be spelled out.

> so one MUST be able to extract the uninverted name mechanically.

But as Kent implied, the notion of "inversions" gets tricker in other 
languages when you start talking about prefixes instead of separate 
words.

> improvements, which make me much more comfortable with inverted text.

Indeed, the latest version regularizes the whole concept of inversions 
for the first time.

> But we still need to decide among the viable options. I can see the 
> following:
>
> ensure that the uninverted is first, eg
>

Again, if you want every possible inversion ("and Barbuda Creole 
English, Antigua") you don't need to indicate any of them.  Just add a 
rule that Descriptions can be inverted between any two words, noting 
that parenthetical comments need to be handled intelligently.

Regarding "first," see below.

> Inverted-Description: Slavonic, Old Church

I don't see the advantage of this approach for listing all possible 
inversions.  Obviously "Church Slavonic, Old" is an inverted form of 
"Old Church Slavonic."  There's no need for the Registry to point that 
out redundantly.

> Description: Slavonic, Old Church

I count that as a vote on the original question, in favor of using the 
inverted form instead of the uninverted form.  If you want both -- in 
particular, (1) or (2) above -- then that will require a change to 
4646bis, which currently specifies that only one or the other may 
appear.

> occurs in the case of inversion.

Done.  The 639-3 folks have really done a good job with this.

> variety of Slavonic.  This is precisely why inversion cannot be done 

Strange, then, that 639-3 lists "S, O C" but not "C S, O".

> I continue to believe that order of field instances should never be 

That was the way this particular riot act was read to me, anyway.  I do 
plan to list them with the 639-3 reference name first, which is harmless 
(order is not significant, right?), but applications cannot count on 
this convention and cannot assume it will be observed in the future.

>> 2. List the inverted forms under a separate field,

No, it really is as simple as this:

1. If you have the inverted names, you don't need the uninverted names; 
you can uninvert them yourself.

2. If you want all possible inversions, it doesn't matter which one you 
start with; you can create all the others yourself (including 
uninverted).

That is why 4646bis currently disallows:

Description: Foo Bar
Description: Bar, Foo

Mark replied:

> And I continue to believe that they must be. When translating, we 
> simply *can't* translate all of the lines, so I don't see any choice 
> but to use the first one as the main description, and trust that that 
> is a reasonable choice. You can make lives more difficult for users if 
> they are in random order, that's all.

Then what you really want is a Reference-Description:

Type: language
Subtag: cu

Reference-Description: Church Slavic
Description: Church Slavic
Description: Slavic, Church
Description: Old Church Slavonic
Description: Slavonic, Old Church
Description: Church Slavonic
Description: Slavonic, Church
Description: Old Bulgarian
Description: Bulgarian, Old
Added: 2005-10-16

If the Registry were converted to XML and accessed using normal XML 
methods, the multiple Description fields really would be in 
non-determininstic order, unless one was given a different name (or 
unless "rank" values were assigned).

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 11 Dec 2006 10:07:58 -0000

> Then what you really want is a Reference-Description:

Yes! You (LTRU) really must read the standard and use a Reference Name
(Description).

As to inverted or non-inverted. Have both.  But mirror the ISO 639-3 file
other than when there is a perceived need for adding additional inverted
forms.

BTW, IMHO there is a marked difference between "Church Slavonic" and "Old
Church Slavonic"; two separate entities.

Debbie

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 11 Dec 2006 10:46:38 -0000

>That has already been done by the Official Doug.
 
This is a rule that is now set within ISO 639-3.
 
Debbie

  _____  

From: Mark Davis [mailto:[email-masked]] 
Sent: 11 December 2006 01:43
To: John Cowan
Cc: Doug Ewell; LTRU Working Group; Kent Karlsson
Subject: Re: [Ltru] To invert or not to invert?

one comment

I could live with this.

---

From: Peter Constable <[email-masked]>
Date: Mon, 11 Dec 2006 17:26:18 -0800

> From: Doug Ewell [mailto:[email-masked]]

> Strange, then, that 639-3 lists "S, O C" but not "C S, O".

I don't see why that's particularly strange. True, "C S, O" reflects a better parse of "Old Church Slavonic". But I don't think the purpose of an inverted name is, at all, to convey something like whether it is the "Old Church" variety of "Slavonic" or the "Old" variety of "Church Slavonic". Rather, I think the purpose is to facility searching under a root name. Now, one might contend that both "Church Slavonic" and "Slavonic" could be considered root names, but in the former "Slavonic" actually is the unmodified root -- making "Slavonic" the root in both cases.

The inverted name only exists to facilitate one way of searching, and especially in hard-coded lists where no free-text search is available. It's not the only way to search; there are various other ways a user might want to search for a variety. In that light, listing "S, O C" is useful for a certain purpose, and the omission of "C S, O" isn't particularly hindering that purpose.

Peter Constable

---

From: Martin Duerst <[email-masked]>
Date: Tue, 12 Dec 2006 10:22:56 +0900

At 16:00 06/12/11, Doug Ewell wrote:

>If the Registry were converted to XML and accessed using normal XML methods, the multiple Description fields really would be in non-determininstic order, unless one was given a different name (or unless "rank" values were assigned).

No. The order of elements in XML is significant. The only
place where order isn't significant is attributes, but
you can't put multiple descriptions into attributes
without giving them different names, so you again break
the symmetry. Applications can declare that for some
cases, element order isn't significant, but generic XML
tools keep element order. Otherwise, we'd all read Web
pages in random sequence :-).

Of course, if the tool that converts from record-jar to
XML uses some random numbers, then the XML file will
contain the descriptions in random order, but once they
are in XML, they are fixed.

Regards,     Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Tue, 12 Dec 2006 10:30:19 +0900

>variety of Slavonic.  This is precisely why inversion cannot be done

What about the following compromize:

1) For names that consist of two pieces, list the uninverted form,
   assuming that it can always be inverted.
2) For names that consist of three or more pieces, list the uninverted
   form as well as all the possible inverted forms.

There are variants on this, e.g. in 2), don't list the uninverted
form to reduce duplication. The proposal is based on the assumption
that two-piece names can always be inverted, and that names with
three pieces or more are relatively rare. If that's not true, please
correct me.

This would allow to hopefully satisfy both those who want to see
the uninverted names, because they are more straightforward, and
those who want to be able to invert them without creating wrong
inversions.

Regards,     Martin.

d: from scmailgw1.scop.aoyama.ac.jp ([133.2.251.194])
	by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Gtwqw-0006AD-6N
	for [email-masked]; Mon, 11 Dec 2006 20:54:43 -0500
Date: Tue, 12 Dec 2006 10:30:19 +0900
To: John Cowan <[email-masked]>, Mark Davis <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: [Ltru] To invert or not to invert?
Cc: Doug Ewell <[email-masked]>, LTRU Working Group <[email-masked]>,
	Kent Karlsson <[email-masked]>

>variety of Slavonic.  This is precisely why inversion cannot be done

What about the following compromize:

1) For names that consist of two pieces, list the uninverted form,
   assuming that it can always be inverted.
2) For names that consist of three or more pieces, list the uninverted
   form as well as all the possible inverted forms.

There are variants on this, e.g. in 2), don't list the uninverted
form to reduce duplication. The proposal is based on the assumption
that two-piece names can always be inverted, and that names with
three pieces or more are relatively rare. If that's not true, please
correct me.

This would allow to hopefully satisfy both those who want to see
the uninverted names, because they are more straightforward, and
those who want to be able to invert them without creating wrong
inversions.

Regards,     Martin.

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 11 Dec 2006 21:10:14 -0800

> pages in random sequence :-).

Oops.  You're right.  I hope nobody's keeping track of these...

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 11 Dec 2006 23:07:52 -0800

>   assuming that it can always be inverted.

Fine.

>   form as well as all the possible inverted forms.

Not fine.  Apparently my earlier point about "and Barbuda Creole 
English, Antigua" was a bit too subtle.  Let's try another example with 
ALL POSSIBLE inverted forms:

Description: Santa Ana de Tusi Pasco Quechua
Description: Ana de Tusi Pasco Quechua, Santa
Description: de Tusi Pasco Quechua, Santa Ana
Description: Tusi Pasco Quechua, Santa Ana de
Description: Pasco Quechua, Santa Ana de Tusi
Description: Quechua, Santa Ana de Tusi Pasco

Silly, isn't it?  And not all of them even make sense.

And as I said before, if you really want ALL POSSIBLE inverted forms --  
for whatever reason -- there is no need to list ANY of them, for the 
same reason that there is no need to list all possible inversions of a 
two-word name: because it is obvious.  Listing all of these redundant 
forms would do nothing but chew up space in the Registry, which will be 
big enough already without them.

There are 2 six-word names, 12 five-word names, 98 four-word names, and 
504 three-word names.  That's a lot of pointless repetition.  And that 
doesn't even count hyphens.

I understand adding only the uninverted form, or only the inverted 
form(s) given in the source standard.  I could even *sort of* go along 
with adding all the versions from the source standard, inverted and 
uninverted, although that seems quite unnecessary.  But creating our own 
inverted forms, just to make sure every mathematical indexing 
possibility is covered, makes no sense at all.  Strongly opposed.

> There are variants on this, e.g. in 2), don't list the uninverted form 
> to reduce duplication. The proposal is based on the assumption that 
> two-piece names can always be inverted, and that names with three 
> pieces or more are relatively rare. If that's not true, please correct 
> me.

See statistics above.

> This would allow to hopefully satisfy both those who want to see the 
> uninverted names, because they are more straightforward, and those who 
> want to be able to invert them without creating wrong inversions.

My preferences, in order:

   1.  uninverted forms only
   2.  inverted forms from 639-3 only
   3.  both 1. and 2. above
 999.  anything else

---

From: John Cowan <[email-masked]>
Date: Tue, 12 Dec 2006 10:18:17 -0500

> ALL POSSIBLE inverted forms:

I think you are misreading Martin: by "possible forms" he probably meant
"suitable forms".  Obviously there would be no point in enumerating
all mechanically possible forms, so the intention would be to enumerate
the ones which make sense, and leave the rest unenumerated.

However, I'm not even happy with the assumption that you can invert
any two-word name:  "Bella Coola" (blc) is written as two words, but
it does not mean the Bella variety of Coola!

-- 
I marvel at the creature: so secret and         John Cowan
so sly as he is, to come sporting in the pool   [email-masked]
before our very window.  Does he think that     http://www.ccil.org/~cowan
Men sleep without watch all night?

---

From: John Cowan <[email-masked]>
Date: Tue, 12 Dec 2006 10:33:57 -0500

Peter Constable scripsit:

> I don't see why that's particularly strange. True, "C S, O" reflects a
> better parse of "Old Church Slavonic". But I don't think the purpose
> of an inverted name is, at all, to convey something like whether
> it is the "Old Church" variety of "Slavonic" or the "Old" variety of
> "Church Slavonic". 

Ooookay.  Based on this example and its justification, I throw in the
towel.  I had believed that there was semantic information in the
inverted names that made them worth preserving, but now I see that
there isn't, or at least not reliably.  (What next, "Language, American
Sign"?)

So I now support the use of uninverted names only.

-- 
John Cowan    http://ccil.org/~cowan  [email-masked]
'Tis the Linux rebellion / Let coders take their place,
The Linux-nationale / Shall Microsoft outpace,
We can write better programs / Our CPUs won't stall,
So raise the penguin banner of / The Linux-nationale.  --Greg Baker

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 12 Dec 2006 20:16:27 -0800

John Cowan <cowan at ccil dot org> wrote:

> I think you are misreading Martin: by "possible forms" he probably 
> meant "suitable forms".  Obviously there would be no point in 
> enumerating all mechanically possible forms, so the intention would be 
> to enumerate the ones which make sense, and leave the rest 
> unenumerated.

Well, I only saw one reasonable interpretation of "possible" at the 
time.  But I do feel better now.

I still think if we add inverted names to the Registry (without or 
without their uninverted brethren) we should stick to exactly the ones 
listed in ISO 639-3, and not go off inventing our own.

> it does not mean the Bella variety of Coola!

Nor Hiri Motu, nor Tok Pisin.

In general I don't favor making one rule for two-word names and another 
for names of three or more words.  That seems arbitrary and overly 
clever.  Some two-word names have inversions in 639-3 and some do not; 
the same is true for at least one of the six-letter names ("Ho Chi Minh 
City Sign Language").  Actually, of all the "X Sign Language" names --  
by definition at least three words long -- only one has an inversion in 
639-3: "Kentish Sign Language, Old".

---

From: Martin Duerst <[email-masked]>
Date: Tue, 12 Dec 2006 17:36:04 +0900

>>   form as well as all the possible inverted forms.

Sorry, by "possible", I meant "sensible", which I think we can
take as "listed in the source standard". Sorry for the confusion.

Regards,    Martin.

>Not fine.  Apparently my earlier point about "and Barbuda Creole English, Antigua" was a bit too subtle.  Let's try another example with ALL POSSIBLE inverted forms:
>
>for whatever reason -- there is no need to list ANY of them, for the same reason that there is no need to list all possible inversions of a two-word name: because it is obvious.  Listing all of these redundant forms would do nothing but chew up space in the Registry, which will be big enough already without them.
>
>There are 2 six-word names, 12 five-word names, 98 four-word names, and 504 three-word names.  That's a lot of pointless repetition.  And that doesn't even count hyphens.
>
>I understand adding only the uninverted form, or only the inverted form(s) given in the source standard.  I could even *sort of* go along with adding all the versions from the source standard, inverted and uninverted, although that seems quite unnecessary.  But creating our own inverted forms, just to make sure every mathematical indexing possibility is covered, makes no sense at all.  Strongly opposed.
>
>> There are variants on this, e.g. in 2), don't list the uninverted form to reduce duplication. The proposal is based on the assumption that two-piece names can always be inverted, and that names with three pieces or more are relatively rare. If that's not true, please correct me.
>
>See statistics above.
>
>> This would allow to hopefully satisfy both those who want to see the uninverted names, because they are more straightforward, and those who want to be able to invert them without creating wrong inversions.
>