Subject: [Ltru] Inversions and other problems

From: "Mark Davis" <[email-masked]>
Date: Mon, 16 Oct 2006 18:33:40 -0700

I processed through the 01 file and generated the following data, listing
every case where a Tag or Subtag contains a Description with a comma or "(".
It is comma-delimited, so a useful way to view it is in Excel or OpenOffice,
where you can filter and sort.

http://macchiato.com/ltru/ltru-commas.txt
(excel version http://macchiato.com/ltru/ltru-commas.xls)

My conclusions are that it would be possible to support option A (allow
inverted names), but it would definitely require some considerable cleanup.
The key is that the inversion has to be predictable, otherwise people can
get garbage when they try to reconstruct the "real" name.

The algorithm to "de-invert" used the following. It assumes the Description
is of the form:

precomma ("," postComma) ("(" parenComment ")")?

When there is a comma outside of the parenComment, the inversion is formed
as:

postComma " " preComma                 // plus "(" parenComment ")" if found

The deinversion is listed after "=>" in the file. It doesn't work in many
cases. I marked certain cases with "@" in column E that could be
mechanically distinguished.

1. There are examples containing a double comma; those don't invert properly
Resian, Resianic, Rezijan

2. There is one case where a description contains nothing but comment. This
should just be deparened, IMO.
(alias for Hiragana + Katakana)

3. There are multiple duplicates.
"Ainu (Japan)" and "Ainu"

[The vast majority of the items with comments don't duplicate, so it is
clearly more consistent not to.]

4. Sometimes the inverted form is *also* listed as a name; much more often
it is not. This is also an inconsistency that should be fixed.
Altai, Southern

5. There are multiple other failures of inversion, like the following. These
cannot be detected automatically; they must be reviewed by knowledgeable
people (column F)

Natisone dialect, Nadiza dialect => Nadiza dialect Natisone dialect

6. There are missing inversions, which for consistency should be present.
*Bulgarian, Old

(These were done with a simple check, looking for Northern, North, ....) I'm
sure that there are many more, but those would require experts to review.

Now, my preference is still to do option B -- I think A is inherently
troublesome for translation -- but I wanted to give people the data I found.

What I think would also be useful, as a by-product of this, would be to move
all of the parenComments into real Comment fields.

Mark

On 10/16/06, Kent Karlsson <[email-masked]> wrote:
>
>
> +1 as modified below (but needs wordsmithing).
>
> > From: Addison Phillips [mailto:[email-masked]]
>
> > I'd be happier with a rule for Doug and Michael to follow that is clear
> > and unambiguous and has the required latitude---and then letting the
> > problem take care of itself, as, so far, it has. Perhaps:
> >
> > --
> > If more than one name is provided by the underlying standard
> > for a new
> > or updated record or the same name appears in multiple formats, the
> > Language Subtag Reviewer MAY select which name or names appear in a
> > Description field in the associated registry record (subject to
> > community consensus) and MAY edit the names to correct spelling,
> > orthographic, or formatting errors.
>
> even if there is just one name given in the source registry
>
> > The Reviewer MAY split a list of
>
> in this case: "MAY" -> "SHOULD", or (preferably) even "SHALL"
>
> > names into separate Description fields. Note: Additional Description
> > fields MAY be added in various formats via the registration process.
>
>                 /kent k
>
>
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ltru
>
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Addison Phillips <[email-masked]>
Date: Mon, 16 Oct 2006 20:39:23 -0700

Mark,

I think my solution would be the most elegant way of dealing with this. 
Doug Ewell is a better approximation of a Turing machine than any 
inversion algorithm. Perhaps the only addition to my message below would 
be to permit the same form of editing of the description fields in 4645bis.

IOW, the rules would be:

1. Follow the source standard.
2. Split multiple names into multiple description fields.
3. Omit or edit problematic items, subject to consensus.

Addison

Mark Davis wrote:
> I processed through the 01 file and generated the following data, 
> listing every case where a Tag or Subtag contains a Description with a 
> comma or "(". It is comma-delimited, so a useful way to view it is in 
> Excel or OpenOffice, where you can filter and sort.
> 
> (excel version http://macchiato.com/ltru/ltru-commas.xls )
> 
> inverted names), but it would definitely require some considerable 
> cleanup. The key is that the inversion has to be predictable, otherwise 
> people can get garbage when they try to reconstruct the "real" name.
> 
> The algorithm to "de-invert" used the following. It assumes the 
> Description is of the form:
> 
> When there is a comma outside of the parenComment, the inversion is 
> formed as:
> 
> The deinversion is listed after "=>" in the file. It doesn't work in 
> many cases. I marked certain cases with "@" in column E that could be 
> 2. There is one case where a description contains nothing but comment. 
> This should just be deparened, IMO.
> 4. Sometimes the inverted form is *also* listed as a name; much more 
> often it is not. This is also an inconsistency that should be fixed.
> Altai, Southern
> 
> 5. There are multiple other failures of inversion, like the following. 
> These cannot be detected automatically; they must be reviewed by 
> knowledgeable people (column F)
> 
> (These were done with a simple check, looking for Northern, North, ....) 
> I'm sure that there are many more, but those would require experts to 
> review.
> 
> What I think would also be useful, as a by-product of this, would be to 
> move all of the parenComments into real Comment fields.
> 
> Mark
> 
> On 10/16/06, *Kent Karlsson * <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
> 
>      > From: Addison Phillips [mailto:[email-masked]
>     <mailto:[email-masked]>]
> 
>     [email-masked] <mailto:[email-masked]>
> ------------------------------------------------------------------------
> 

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 17 Oct 2006 06:34:10 +0200

> 3. Omit or edit problematic items, subject to consensus.

Where (2) is "normally indicated by a semicolon" (?)

I like the "inverted form", it allows to spot all related
subtags in a list sorted by descriptions, e.g. all subtags
with one or more descriptions starting with "Arabic".

Or very visible in Mark's list, find out if "Quechua" is
the same as "Quichua" - but I didn't try that ;-)

Creating a derived list without inversions is apparently
simple, but the opposite direction isn't necessarily.

The @DOUBLE_COMMA stuff is apparently limited to a few 
enumerations, for "redundant" subtags we could improve it:

Instead of (example, one of six):
- German, Swiss variant, traditional orthography
+ German traditional orthography, Swiss variant

The nedis / rozaj cases could get a semicolon...

- Natisone dialect, Nadiza dialect
- Resian, Resianic, Rezija
+ Natisone dialect; Nadiza dialect
+ Resian; Resianic; Rezija

...resulting in multiple descriptions.  But IMO that's
unnecessary, also unnecessary for zh-min and zh-min-nan,
where the @DOUBLE_COMMA is an ordinary enumeration in
grandfathered tags.  

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 17 Oct 2006 07:49:33 -0700

As I said before, I can live with inversion IF it is fixed. That means an
extension of Addison's proposed rules.

2. Split multiple names into multiple description fields.
I believe Addison meant: X, Y, Y =>
Description: X
Description: Y
Description: Z

plus

4. Ensure that there are no duplicates (where inversion and/or removal of
parenComments results in a duplicate within the same subtag, or where names
are the same across different subtags of the same type)
Eg no cases like
Subtag: ain
Description: Ainu
Description: Ainu (Japan)
or
Description: Northern Sotho
Description: Sotho, Northern
or (there are currently none of these)
Subtag: x
Description: y
Subtag: z
Description: y

5. The only information in parentheses is a comment, eg a clarification of
the description.

6. The final result being that there are NO description items with commas
outside of parenComments, except for the case that there is a single comma
indicating inversion.

If these are the rules, then I'm satisfied.

==

Ideally, we'd have two additional rules:

7. The most commonly known name is in the first Description.
(This may already be the case -- I don't know.)

8. A description is unique even if the parenComment is removed.

As it stands, there are 138 cases where a description is only distinguished
by a parenComment. Ideally, those would be fixed so that the rest of the
description was unique. These are in that same directory:
http://macchiato.com/ltru/, in the file ltru-duplicates.txt

Example:

126    Aja (Benin)    language    ajg
127    Aja (Sudan)    language    aja

For example, if these were the following, then the comments would not be
necessary.
ajg: Aja, Benin
aja: Aja, Sudan
That would also be more consistent with other uses like Sotho, Northern vs
Sotho, Southern -- the adjective is not just in a parenComment.

For those interested,

   - ltru-reverse.txt is a "reverse" list, ordered by description.
   - ltru-reverse-no-comments.txt is the same, but with the parenComments
   removed.

Mark

---

From: Addison Phillips <[email-masked]>
Date: Tue, 17 Oct 2006 08:32:06 -0700

Mark Davis wrote:
> As I said before, I can live with inversion IF it is fixed. That means 
> an extension of Addison's proposed rules.
> 

Yes, exactly so.

> 3. Omit or edit problematic items, subject to consensus.

Well... my goal was to avoid having to spell all of this out. We've got 
enough rules, so I think some hand-waving is in order. While I note the 
problem projects like CLDR might have producing localizations, the use 
of an actual Turing machine (aka Doug Ewell) to create the entries 
should eliminate a lot of woes. Furthermore, I tend to think that the 
*detailed* rules you propose should go into 4645bis (i.e. "how did the 
entries get this way"), while the simpler version I proposed can go into 
4646bis (perhaps with a pointer to "pre-existing customary practice" in 
4645bis).

> 
> 4. Ensure that there are no duplicates (where inversion and/or removal 
> of parenComments results in a duplicate within the same subtag, or where 
> names are the same across different subtags of the same type)

I wasn't specific enough in my little bit of text above. "Duplicates" 
(including inverted duplicates) are members of the class "problematic".

> 
> 5. The only information in parentheses is a comment, eg a clarification 
> of the description.

This could be guidance added to rule 3.

> 
> 6. The final result being that there are NO description items with 
> commas outside of parenComments, except for the case that there is a 
> single comma indicating inversion.
> 

Except that you go on to demonstrate below that sometimes inversion is 
illusory.

> (This may already be the case -- I don't know.)

I don't think this is appropriate. The level of judgment required is too 
great for the ietf-languages list to manage.

> 8. A description is unique even if the parenComment is removed.

We need a "MUST" here. I'd propose:

--
For a given record type, each Description field MUST be unique.
--

Addison

---

From: Addison Phillips <[email-masked]>
Date: Tue, 17 Oct 2006 08:56:48 -0700

Maybe more to the point, I propose we insert this text into the 
'Description' section of the document (the current Section 3.1.3.2):

<t>For a records of a given 'Type', the value of the 'Description' field 
MUST be unique. For records taken from a source standard (such as ISO 
639 or ISO 3166), the 'Description' value(s) SHOULD be taken from the 
source standard. Multiple descriptions in the source standard MUST be 
split into separate 'Description' fields. The source standard's 
descriptions MAY also be edited prior to insertion. Duplicate, 
redundant, conflicting, or otherwise problematic descriptions SHOULD be 
omitted. Parenthetical comments, inverted names, and other formatting 
quirks SHOULD be regularized according to the guidelines used to update 
the registry in <xref target="registry-update"></xref>.</t>

Addison

Addison Phillips wrote:
>> of parenComments results in a duplicate within the same subtag, or 
>> where names are the same across different subtags of the same type)
> 
>> 5. The only information in parentheses is a comment, eg a 
>> clarification of the description.
>

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 17 Oct 2006 12:37:13 -0700

> 4646bis (perhaps with a pointer to "pre-existing customary practice" in

That's ok, as long as they are sufficiently spelled out such that they
prevent mistakes.

> (including inverted duplicates) are members of the class "problematic".

ok

> This could be guidance added to rule 3.

ok

> Except that you go on to demonstrate below that sometimes inversion is

No. What I mean is that any cases of false inversion need to be fixed. There
are not that many of them.

> great for the ietf-languages list to manage.

I'm not thinking of a requirement, more of a weak should.

> We need a "MUST" here. I'd propose:

I'd like slightly string.

For a given record type, each Description field (after removing any
parantheticals and inverting any comma-separated strings) MUST be unique.

--

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 17 Oct 2006 12:43:37 -0700

I'm ok with the direction. I think it needs a bit more qualification, so I
suggest the text below.

<t>For a records of a given 'Type', the value of the 'Description' field
MUST be unique, after removing parentheticals and inverting any string
containing commas. For records taken from a source standard (such as ISO
639 or ISO 3166), the 'Description' value(s) SHOULD be taken from the
source standard. Multiple descriptions in the source standard MUST be
split into separate 'Description' fields. Outside of parentheticals, commas
MUST only be used for inverted strings. The source standard's
descriptions MAY also be edited prior to insertion. Duplicate,
redundant, conflicting, or otherwise problematic descriptions MUST be
corrected or omitted. Parenthetical comments, inverted names, and other
formatting variants SHOULD be regularized according to the guidelines used
to update the registry in <xref target="registry-update"></xref>.</t>

Mark

---

From: Addison Phillips <[email-masked]>
Date: Tue, 17 Oct 2006 12:46:05 -0700

> I'd like slightly string.

Mnrgle?

> parantheticals and inverting any comma-separated strings) MUST be unique.

No, actually, if two of the same description field appear anywhere in 
the file, something is wrong. We wouldn't want:

Subtag: xx
Description: Western Xish
%%
Subtag: xy
Description: Western Xish

If what you're say is that the following should also be prohibited, I agree:

Subtag: xx
Description: Western Xish
%%
Subtag: xy
Description: Xish, Western

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 17 Oct 2006 13:03:19 -0700

I generated a file at http://macchiato.com/ltru/ltru-inversions.txt. It
contains a mapping from all of the descriptions (sans parentheticals) to
their uninverted forms. It also *adds* cases that are "possibly missing",
marked with @MISSING. The file, as usuall, is tab delimited, so best pulled
into a program that lines them up into columns, like Excel or equivalent.

The "possibly missing" inversions are produced by collecting all of the
prefixes and suffixes of existing inversions, and seeing if any of the
uninverted string matches such an existing prefix or suffix. For example, if
there is some preexisting inversion such as "Armenian, X" in the 01 data
file, and the program sees "Y Armenian", then it includes "Armenian, Y
@MISSING" for review in this file.

There are 251 of these items. So that we have consistent inversion, these
should be reviewed before we finalize the descriptions. For example, in the
list we see:

103	Armenian, Classical	Classical Armenian
104	Armenian, Eastern	Eastern Armenian	@MISSING
105	Armenian, Middle	Middle Armenian
106	Armenian, Western	Western Armenian	@MISSING

#104 and #106 are missing, meaning that we don't invert those (although,
IMO, clearly we should if we are doing inversion).

The whole list could probably stand review -- it exposes some clear spurious
differences, eg "Alaska" vs "Alaskan" in the following:

31	Alaska Inupiatun, Northwest	Northwest Alaska Inupiatun	@MISSING
32	Alaskan Inupiatun, North	North Alaskan Inupiatun	@MISSING

Mark

---

From: Addison Phillips <[email-masked]>
Date: Tue, 17 Oct 2006 12:54:33 -0700

Mark Davis wrote:
> I'm ok with the direction. I think it needs a bit more qualification, so 
> I suggest the text below.
> 

Additional edits were needed, notably for the "For a records" typo. I 
ended up with:

--
<t>For each collection of records of a particular 'Type', each of the 
values that appears in a 'Description' field MUST be unique. For records 
taken from a source standard (such as ISO 639 or ISO 3166), the 
'Description' value(s) SHOULD be taken from the source standard. 
Multiple descriptions in the source standard MUST be split into separate 
'Description' fields. The source standard's descriptions MAY also be 
edited prior to insertion. Duplicate, redundant, conflicting, or 
otherwise problematic descriptions MUST either be corrected or omitted. 
Parenthetical comments, inverted names, and other formatting quirks 
SHOULD be regularized according to the guidelines used to update the 
registry in <xref target="registry-update"></xref>.</t>
--

Addison

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 18 Oct 2006 02:09:18 +0200

> Mnrgle?

"slightly more stringent MUSTard" ;-)

> the file, something is wrong.

That could happen with deprecated tags or subtags.  We have three records
with description "Hebrew" in 4645bis-00, and one of those is the script.

> Description: Xish, Western

That's apparently what it says for readers of this thread.  Is "inverting
any comma-separated string" also obvious for readers of 4646bis, or does
it need an example ?

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 17 Oct 2006 17:20:21 -0700

Good point: we need to exclude the deprecated cases (I did that in my
program, but neglected to note it).

Mark

> any comma-separated string" also obvious for readers of 4646bis, or does

I always think examples are worthwhile: we could include Xish cases for
clarity.

Frank

---

From: John Cowan <[email-masked]>
Date: Tue, 17 Oct 2006 22:08:11 -0400

> program, but neglected to note it).

It's not uncommon for a script and a language to have the same name,
and I don't think we should insist on an artificial distinction here.

-- 
John Cowan              http://www.ccil.org/~cowan      [email-masked]
"After all, would you consider a man without honor wealthy, even if his
Dinar laid end to end would reach from here to the Temple of Toplat?"
"No, I wouldn't", the beggar replied.  "Why is that?" the Master asked.
"A Dinar doesn't go very far these days, Master.        --Kehlog Albran
Besides, the Temple of Toplat is across the street."      The Profit

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 17 Oct 2006 13:11:38 -0700

> > I'd like slightly string.

ypu, srory!

> If what you're say is that the following should also be prohibited, I

Yes, that's what it means. "For a given record type, each Description field
(after removing any parantheticals and inverting any comma-separated
strings) MUST be unique." means that if we have

then each of the following is disallowed, anywhere else (under the same
type, of course!):

Description: Western Xish
Description: Western Xish (Amusing)
Description: Xish, Western (Amusing)
Description: Xish, Western

Subtag: xx

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 17 Oct 2006 19:54:10 -0700

You misunderstand. There are 3 items called "Hebrew" The script is
distinct from the others, and is not at issue. Everything we have been
saying is that the items must be distinct *within a type*. See the
line:

 ><t>For a records of a given 'Type', the value of the 'Description'

The problem is that there are two cases with type=language &
description=Hebrew: he and iw. For that reasons, we need to restrict
the clause to only worry about non-deprecated languages.

Fleshing it out with examples, reordering slightly, and adding the
qualification, here is my suggested language:

<t>For each collection of records of a particular 'Type' (excluding
records with a Deprecated field), each of the values that appears in a
'Description' field MUST be unique. The description values MUST also
be unique even after removing parentheticals and inverting any string
containing commas. Outside of parentheticals, commas MUST only be used
for inverted strings.</t>

<t>An inverted name such as "Xishu, Gibrus" is used to facilitate
sorting in English according to most salient feature (in this case,
"Xishu"). Such inversions would not necessarily be reflected in
translations to other languages. Commas (outside of parentheticals)
always indicates inversion, not multiple names or other constructs.
Thus "Xishu, Gibrus" is always equivalent to "Gibrus Xishu", just
differing in presentation.</t>

<t>For example, suppose that the following description is in a
language subtag record:</t>

<pre>Description: Gibrus Xishu</pre>

<t>Then that line cannot occur again in that record, nor in any other
language subtag record. Moreover, none of the following can occur in
any language subtag record:</t>

<pre>Description: Gibrus Xishu (any comment)</pre>
<pre>Description: Xishu, Gibrus </pre>
<pre>Description: Xishu, Gibrus (any comment)</pre>

</t>For records taken from a source standard (such as ISO 639 or ISO
3166), the 'Description' value(s) SHOULD be based on the source
standard. The source standard's descriptions MAY also be edited prior
to insertion. Multiple descriptions in the source standard MUST be
split into separate 'Description' fields. Duplicate, redundant,
conflicting, or otherwise problematic descriptions MUST either be
corrected or omitted. Parenthetical comments, inverted names, and
other formatting variants SHOULD be regularized according to the
guidelines used to update the registry in <xref
target="registry-update"></xref>.</t>

Mark

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 18 Oct 2006 05:36:24 +0200

Mark Davis wrote:

> here is my suggested language:
 
>| For each collection of records of a particular 'Type' (excluding
>| records with a Deprecated field), each of the values that appears
>| in a 'Description' field MUST be unique.

ACK so far, that just makes sense.  It doesn't catch the fine print
of a shared namespace for language and extlang (JFTR, I'm not saying
that it should) 

>| The description values MUST also be unique even after removing 
>| parentheticals
[...]
I'm not sure about this parenthical bit, see below,
[...]
>| and inverting any string containing commas.
[...]
That's again clear, and I like it.
[...]
>| Outside of parentheticals, commas MUST only be used for inverted 
>| strings.

If you insist on this new comma-rule we're forced to patch several
redundant and grandfathered descriptions, as noted before in another
article - for the six 1901/1996 cases it would be simple.

>| For example, suppose that the following description is in a
>| language subtag record:
 
> <pre>Description: Gibrus Xishu</pre>

My preferred style would be what we get from the "inverted name"
column in the table derived from the 639-3 draft:  Xishu, Gibrus

How does your parenthical rule handle Ainu (China) vs. Ainu (Japan),
or similar Aja (Sudan) vs. Aja (Benin) ?  

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 18 Oct 2006 07:28:51 -0700

> of a shared namespace for language and extlang (JFTR, I'm not saying

We could handle that by adding: "For the purposes of determining uniqueness,
extlang records are treated as if they were language tags."

> article - for the six 1901/1996 cases it would be simple.

Yes, we would have to fix a few items. But there are few, and the result
would then be consistent. Better to do that than to have some descriptions
be inverted and some not, with no way to tell which is which. (And adding
additional structure or description to allow people to determine which are
inverted and which not is *much* more painful than just fixing the instances
that are not). As I've said, my preference would be not to have inverted
names, but since the majority seem to want them, we have to make their usage
completely predictable.

> column in the table derived from the 639-3 draft:  Xishu, Gibrus

The example was not mean to prefer one form other another, just to say that
if any form exists, the others can't. Maybe that is better captured by
changing:

OLD:

NEW:

<t>For example, if any one of the following occurs in a language subtag
record, then none of the others can occur in any language subtag record.</t>

END

> or similar Aja (Sudan) vs. Aja (Benin) ?

Those would need to be fixed. There are 138 such cases, as I indicated in
previous messages, so that would involve changing 69..138 values, depending
on the resolution. For example, since "Ainu" alone is already a description
on 'ain', I think the correct change there would be to drop the description
"Ainu (Japan)" on 'ain' and change the "Ainu (China)" to "Ainu, China" on
'aib'. Now, I want to make clear, I distinguished this as a "nice to have"
in my original list (#8); it makes it easier for translation, and better we
fix 1% or so of the cases than to leave them in. But it would also be
possible to adjust the text to allow descriptions only differing by
parentheticals.

Frank

---

From: Martin Duerst <[email-masked]>
Date: Wed, 18 Oct 2006 17:10:57 +0900

At 00:56 06/10/18, Addison Phillips wrote:
>Maybe more to the point, I propose we insert this text into the 'Description' section of the document (the current Section 3.1.3.2):
>
><t>For a records of a given 'Type', the value of the 'Description' field MUST be unique.

This doesn't fly, as there are multiple 'Description' fields.

Is it "The 'Description' fields of a record each have to be unique among themselves."
or "All the 'Description' fields across a given 'Type' have to be unique."
or "Each record of a given type has to contain at least a 'Description' that is unique
among all the 'Description' fields of a type." or something else?

(MUSTardization at your disgression)

Regards,     Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 18 Oct 2006 18:48:58 -0700

Is this any clearer? (would need some reworking, but I want to get the
concept across.)

Given a description, a variant description is any that could be created by
adding or removing a parenthetical, or inverting or uninverting the string,
or both.

Within a given type, a given description MUST be unique. That is, it MUST
NOT occur more than once in that record, and MUST NOT occur at all in any
other record of that type. Moreover, no variant description MAY occur in
that record or any other record of that type.

On 10/18/06, Martin Duerst <[email-masked]> wrote:
> MUST be unique.

---

From: Marion Gunn <[email-masked]>
Date: Thu, 19 Oct 2006 09:37:00 +0000

"A given description within a given type must be unique. That means  
it may not occur more than once in any specific record and may not  
occur in any other record of the same type. Moreover, no variant  
description may occur in that specific record or in any other record  
of the same type."
mg

On 19 Oct 2006, at 01:48, scríobh Mark Davis:

> Is this any clearer? (would need some reworking, but I want to get  
> the concept across.)
>
> Given a description, a variant description is any that could be  
> created by adding or removing a parenthetical, or inverting or  
> uninverting the string, or both.
>
> Within a given type, a given description MUST be unique. That is,  
> it MUST NOT occur more than once in that record, and MUST NOT occur  
> at all in any other record of that type. Moreover, no variant  
> description MAY occur in that record or any other record of that type.
>
>
> field MUST be unique.

- -
Marion Gunn * EGTeo (Estab.1991)
27 Páirc an Fhéithlinn, Baile an
Bhóthair, Co. Átha Cliath, Éire.
* [email-masked] * [email-masked] *

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 19 Oct 2006 15:38:13 +0200

Mark Davis wrote:

Those would need to be fixed. There are 138 such cases, as I indicated in previous messages, so that would involve changing 69..138 values, depending on the resolution. For example, since "Ainu" alone is already a description on 'ain', I think the correct change there would be to drop the description "Ainu (Japan)" on 'ain' and change the "Ainu (China)" to "Ainu, China" on 'aib'. 

I'm not fond of the idea "to drop the description "Ainu (Japan)" on 'ain'" (leaving just "Ainu"), since that does not hint that there *may* be more than one "Ainu". Nor do I like the idea of "change the "Ainu (China)" to "Ainu, China" on 'aib'", since 1) that would be
"equivalent" to "China Ainu" which looks strange ("Chinese Ainu" would seem better), and 2) if the country name is replaced by a language group name (which I would prefer, compare "Slave (Athapascan)"), both the inverted and non-inverted name would look really strange.
 
One more reason that I don't like using country names for distinguishing same-named but different languages is that name tags,
as per 1766 and successors, already allow region codes. One may well be tempted to think that:
 
* "ain-CN" is the same as "aib", which it is not and "Ainu (China)" is therefore misleading (and many UIs might not make a display name distinction), and that
 
* "swa-CG" (or "swa-CD") is the same as "swa-swc", which I cannot tell if it is or isn't (swa is a macrolanguage covering swc); I guess they may be the "same", and if so, using the country name in the language description would be more appropriate (but still subject to name instability, (one of the) Congo(s) was called Zaïre not so very long ago). There is no requirement that systems treat them the same though, but it may warrant a comment that swa-swc is preferred (or not) over swa-CG or swa-CD.
 
There are several instances of a) (clearly?) unrelated, and b) (apparently?) closely related languages that have the same name but (currently) distinguished by country name in the draft (yes, I know it is a draft!) 639-3 registry tables. The former should certainly be distinguished by something else than country name (in 639-3...).
 
                /kent k

---

From: John Cowan <[email-masked]>
Date: Thu, 19 Oct 2006 09:39:28 -0400

> that record or any other record of that type.

Well, that's fine, but it completely cabbages the requirement to keep the
Description fields in sync with the underlying standards.  Can't have it
both ways as long as 639-2 and 639-3 haven't settled their differences,
and we can only hope that they *will* settle them.

This is as good a place as any to report that I heard back from Joan
Spanne at the 639-3/RA-to-be saying that there while my complaints about
Yinglish and the bogus conlangs would be considered after 639-3 became
a standard, no changes would be looked at between the FDIS (which has
already happened) and the IS.  Oh well.  So we will probably have a
bunch of deprecations later.

-- 
He made the Legislature meet at one-horse       John Cowan
tank-towns out in the alfalfa belt, so that     [email-masked]
hardly nobody could get there and most of       http://www.ccil.org/~cowan
the leaders would stay home and let him go      --H.L. Mencken's
to work and do things as he pleased.              Declaration of Independence

---

From: John Cowan <[email-masked]>
Date: Thu, 19 Oct 2006 09:43:48 -0400

Kent Karlsson scripsit:

> I'm not fond of the idea "to drop the description "Ainu (Japan)" on
> 'ain'" (leaving just "Ainu"), since that does not hint that there *may*
> be more than one "Ainu". Nor do I like the idea of "change the "Ainu
> (China)" to "Ainu, China" on 'aib'", since 1) that would be "equivalent"
> to "China Ainu" which looks strange ("Chinese Ainu" would seem better),
> and 2) if the country name is replaced by a language group name (which
> I would prefer, compare "Slave (Athapascan)"), both the inverted and
> non-inverted name would look really strange.

In addition, both "China Ainu" and even more so "Chinese Ainu" would suggest
some relation to the Ainu language of Japan, but there is none.

> There are several instances of a) (clearly?) unrelated, and b)
> (apparently?) closely related languages that have the same name but
> (currently) distinguished by country name in the draft (yes, I know
> it is a draft!) 639-3 registry tables. The former should certainly be
> distinguished by something else than country name (in 639-3...).

As a practical matter, country name is probably the most well-known
attribute in these cases, and is fairly uncontroversial (though as you
say subject to change).  Most people know nothing about language affiliations.

-- 
We do, doodley do, doodley do, doodley do,        John Cowan <[email-masked]>
What we must, muddily must, muddily must, muddily must; 
Muddily do, muddily do, muddily do, muddily do,    http://www.ccil.org/~cowan
Until we bust, bodily bust, bodily bust, bodily bust.  --Bokonon

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 19 Oct 2006 06:46:53 -0700

I agree with you that country names -- whether in parentheticals or
not -- are not a good method of distinguishing language subtags. So if
we have other information to help to name them, that would be better.

I differ on the simultaneous use of

description(xxx) = A B
description(yyy) = B.

We do that a lot, and I don't think it leads to that much confusion.
It is reasonable when what is denoted by B (that is, yyy) is the
predominant usage. When someone says Ainu, it really means Ainu, as in
Japan, unless it is otherwise qualified. It is the unusual case that
needs a qualification.

Mark

On 10/19/06, Kent Karlsson <[email-masked]> wrote:
> 'aib'.
> I'm not fond of the idea "to drop the description "Ainu (Japan)" on 'ain'"
> (leaving just "Ainu"), since that does not hint that there *may* be more
> than one "Ainu". Nor do I like the idea of "change the "Ainu (China)" to
> "Ainu, China" on 'aib'", since 1) that would be
> "equivalent" to "China Ainu" which looks strange ("Chinese Ainu" would seem
> better), and 2) if the country name is replaced by a language group name
> (which I would prefer, compare "Slave (Athapascan)"), both the inverted and
> One more reason that I don't like using country names for distinguishing
> same-named but different languages is that name tags,
> * "ain-CN" is the same as "aib", which it is not and "Ainu (China)" is
> therefore misleading (and many UIs might not make a display name
> distinction), and that
>
> * "swa-CG" (or "swa-CD") is the same as "swa-swc", which I cannot tell if it
> is or isn't (swa is a macrolanguage covering swc); I guess they may be the
> "same", and if so, using the country name in the language description would
> be more appropriate (but still subject to name instability, (one of the)
> Congo(s) was called Zaïre not so very long ago). There is no requirement
> that systems treat them the same though, but it may warrant a comment that
> swa-swc is preferred (or not) over swa-CG or swa-CD.
>
> There are several instances of a) (clearly?) unrelated, and b) (apparently?)
> closely related languages that have the same name but (currently)
> distinguished by country name in the draft (yes, I know it is a draft!)
> 639-3 registry tables. The former should certainly be distinguished by
> something else than country name (in 639-3...).
>
>                 /kent k
>

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 19 Oct 2006 06:51:24 -0700

> and we can only hope that they *will* settle them.

That is clear in the wording that Addison and I have been working on,
and also clear as soon as we have separate names split into different
description fields. I think we need to have a strong SHOULD to
parallel the underlying standard as much as possible, but we owe it to
our users to present unambiguous descriptions, so if the source
standards do not do that, we should. After all, if someone wants to
look at the original source, they can clearly do that.

> bunch of deprecations later.

dang!

---

From: John Cowan <[email-masked]>
Date: Thu, 19 Oct 2006 10:10:32 -0400

> look at the original source, they can clearly do that.

Fair enough.

I am absolutely opposed, however, to creating inversions where the source
has none.  Mark's list <http://macchiato.com/ltru/ltru-inversions.txt>
shows its absurdity in its first few entries.  "South Africa" is the name
of a country, and inverting it to "Africa, South" makes no sense at all;
it winds up mixed in with the U.N. regions "Southern Africa", "Northern
Africa", etc. etc.  We also wind up with "African Sign Language, South"
which of course is the sign language of South Africa, but when inverted
in this fashion looks like the Southern variety of a fictitious "African
Sign Language".

Where we have an inverted form we should use it, because it contains
more information than the uninverted form.  Where we have only an uninverted
form, we must live with it.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
        Sound change operates regularly to produce irregularities;
        analogy operates irregularly to produce regularities.
                --E.H. Sturtevant, ca. 1945, probably at Yale

---

From: John Cowan <[email-masked]>
Date: Thu, 19 Oct 2006 10:11:25 -0400

> Japan, unless it is otherwise qualified. It is the unusual case that

Probably true for people referring to Ainu in anglophone countries --
but in China things may be otherwise.

-- 
"The serene chaos that is Courage, and the phenomenon   [email-masked]
of Unopened Consciousness have been known to the        John Cowan
Great World eons longer than Extaboulism."
"Why is that?" the woman inquired.
"Because I just made that word up", the Master said wisely.
        --Kehlog Albran, The Profit             http://www.ccil.org/~cowan

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 19 Oct 2006 10:12:34 -0700

> shows its absurdity in its first few entries.  "South Africa" is the name

"absurdity"? It is pretty darned annoying when no matter how I qualify
messages, some people completely skip over that. Please look at what I
wrote. I DID NOT SAY THAT ALL OF THE ITEMS SHOULD BE DONE. I SAID THAT
THEY ARE "POSSIBLY MISSING" AND PUT THEM UP FOR REVIEW. It is not
absurd to put up information for review.

To make it dirt-simple, I put up a file called
http://macchiato.com/ltru/ltru-inversions-possibly-missing.xls
(notice the "possibly" in the name) that has all of the possible items
sorted by type. It is an Excel file, but if you are a parathyrophobe
you can use OpenOffice.

Now, I am, as I have said many times, not in favor of inversion. But
if we are going to invert, we should make at least *some* effort to
make it work consistently. Given *all* the other inversions, it is
not, at least prima facie, absurd to have:

Bolivian Guaraní, Eastern
Bolivian Guaraní, Western
or
Canadian Inuktitut, Eastern
Canadian Inuktitut, Western
or
Guiyang Hmong, Northern
Guiyang Hmong, Southern
Guiyang Hmong, Southwestern

or many others listed there.

Of course, some of the inversions shouldn't be done, such as those
splitting South Africa where it means the country. That's why these
where characterized as "possible". What would be useful is if you were
to go through the document and pick out the ones that you think
shouldn't be done.

---

From: Addison Phillips <[email-masked]>
Date: Thu, 19 Oct 2006 12:01:24 -0700

We've been batting this around awhile. I did a major edit on that 
section (incorporating some of Marion's text) to produce this:

--
<section anchor="descriptionfield" title="Description Field">

<t>The field 'Description' contains a description of the tag or subtag 
in the record. The 'Description' field MAY appear more than once per 
record, that is, there can be multiple descriptions for a given record. 
At least one of the 'Description' fields MUST be written or transcribed 
into the Latin script; additional 'Description' fields MAY also include 
a description in a non-Latin script. Each 'Description' field MUST be 
unique, both within the record in which it appears and for the 
collection of records of the same type. Moreover, formatting variations 
of the same description MUST NOT occur in that specific record or in any 
other record of the same type.  For example, while the ISO 639-1 code 
'fy' contains both the descriptions "Western Frisian" and "Frisian, 
Western", only one of these descriptions appears in the registry.</t>

<t>The 'Description' field is used for identification purposes and 
SHOULD NOT be taken to represent the actual native name of the language 
or variation or to be in any particular language.</t><t>For records 
taken from a source standard (such as ISO 639 or ISO 3166), the 
'Description' value(s) SHOULD be taken from the source standard. 
Multiple descriptions in the source standard MUST be split into separate 
'Description' fields. The source standard's descriptions MAY be edited, 
either prior to insertion or via the registration process. </t>

<t>When creating a new registry entry, duplicate, redundant, 
conflicting, or otherwise problematic descriptions MUST either be 
corrected or omitted. Parenthetical comments, inverted names, and other 
irregularities SHOULD be regularized according to the guidelines used to 
update the registry in <xref target="registry-update" />.</t>

<t>Note: Descriptions in registry entries that correspond to ISO 639, 
ISO 15924,  ISO 3166, or UN M.49 codes are intended only to indicate the 
meaning of that identifier as defined in the source standard at the time 
it was added to the registry. The description does not replace the 
content of the source standard itself. The descriptions are not intended 
to be the English localized names for the subtags. Localization or 
translation of language tag and subtag descriptions is out of scope of 
this document.</t></section>
--

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 19 Oct 2006 21:08:56 +0100

Playing catchup.  Apologies if I am just reiterating my position on this.
 
It is my humble opinion that if the ISO standard says Ainu (Japan) then that
should be copied EXACTLY within the registry.  Please do not make changes to
the data from the underlying standards where this data has been added in
order to differentiate between language (or other) entities. I have no
objection to spelling errors etc being corrected and the JAC being informed
of such corrections. 
 
Best regards
 
Debbie

Mark Davis wrote:

 
One more reason that I don't like using country names for distinguishing
same-named but different languages is that name tags,
as per 1766 and successors, already allow region codes. One may well be
tempted to think that:

 
 
                /kent k

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 20 Oct 2006 02:21:55 +0200

Mark Davis wrote:

> if you are a parathyrophobe you can use OpenOffice.

Your TAB-separated files were perfectly readable from my POV,
the XLS-file isn't.  What is a "parathyrophobe" ?  Am I safe
from this condition without "OpenOffice" ?

>> This is as good a place as any to report that I heard back
>> from Joan Spanne at the 639-3/RA-to-be saying that there
>> while my complaints about Yinglish and the bogus conlangs
>> would be considered after 639-3 became a standard, no changes
>> would be looked at between the FDIS (which has already
>> happened) and the IS.  Oh well.  So we will probably have a

> dang!

An "orq" registry wouldn't pass a giggle test, we can't submit it.

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 19 Oct 2006 17:40:07 -0700

parathyrophobe - someone who doesn't like Windows (my coinage)

OpenOffice is an open-source office product that let's people on other
platforms read Windows files, like xls files.

Mark

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 20 Oct 2006 05:57:54 +0200

> parathyrophobe - someone who doesn't like Windows (my coinage)

"Can't read XLS" and "don't like a 'parathyro'" isn't the same,
as far as I know it W2K is a nice OS, as shipped without Excel ;-)

> OpenOffice is an open-source office product that let's people
> on other platforms read Windows files, like xls files.

On _some_ other platforms _if_ installed.  I should finally delete
a vintage '95 "IBM works for OS/2" directory, it's in an unusable 
state.  MIME text/csv got its own RFC 4180 a year ago, with a link
to the same TaoUP as in RFC 4646 (incl. among others "record-jar"): 

<http://www.catb.org/~esr/writings/taoup/html/ch05s02.html>

And RFC 4180 mentions that there's a text/tab-separated-values
MIME type, nice, I didn't know that:

<ftp://ftp.iana.org/assignments/media-types/text/tab-separated-values>

Frank

---

From: Addison Phillips <[email-masked]>
Date: Thu, 19 Oct 2006 21:15:24 -0700

>> parathyrophobe - someone who doesn't like Windows (my coinage)

(laughing) Παράθυρα. Ah, it took me a couple of minutes to figure that 
out. Knowledge of Greek roots doesn't help at all here.

Addison

---

From: John Cowan <[email-masked]>
Date: Sun, 22 Oct 2006 00:05:33 -0400

Mark Davis scripsit:

> make it work consistently.

We aren't the right target for that:  the ISO RA/MAs are.

> Given *all* the other inversions, it is

[snip]

> Guiyang Hmong, Southwestern

Or is it "Hmong, Northern Guiyang" etc.?  We don't know and we
shouldn't strive to find out.

> >more information than the uninverted form.  Where we have only an 
> >uninverted form, we must live with it.

I stand by what I said.

-- 
What is the sound of Perl?  Is it not the       John Cowan
sound of a [Ww]all that people have stopped     [email-masked]
banging their head against?  --Larry            http://www.ccil.org/~cowan

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 22 Oct 2006 06:52:04 +0200

> We aren't the right target for that:  the ISO RA/MAs are.

We could collect such questions and submit the collection,
and then decide what to do in 4646bis based on the answer.

The beautification of descriptions is no showstopper, the
RFC 4646 procedure to update descriptions can handle bulk
updates - at least in theory.

In practice there's a lot of stuff piling up:  the script
issues, the review issues, the language type/scope issue,
some type "C" oddities, now the descriptions, etc.

Frank