Subject: [Ltru] Apostrophes

From: "Doug Ewell" <[email-masked]>
Date: Sun, 5 Aug 2007 23:52:04 -0700

This is a proposal for RFC 4645bis and ultimately for the registration 
process performed on the ietf-languages list, although I have not copied 
this message to that list.  I expect it to be controversial, and I am 
willing to be talked out of it, but I believe it has merit.

In arguing for the conversion of the Registry from hex NCRs to UTF-8, 
the issue of readability of both formats came up repeatedly.  There were 
several calls, mostly from one participant, to supply an ASCII-only 
"compatibility" Registry, similar to the Latin-1-only code lists 
supplied by the ISO 639-2 and -3 maintenance agencies.  There have also 
been proposals to add additional Description or Comments fields which 
simply duplicate the existing fields in plain ASCII.

I don't support either of these approaches because non-ASCII characters 
are often an integral part of the description or comment, and will 
become even more so in the RFC 4646bis era, as we add pairs of languages 
whose names differ only by an accented vs. unaccented letter (such as 
Arua and Aruá; the latter is spelled A, r, u, a-with-acute).  We need to 
maintain these distinctions to preserve the identity of the language or 
other entity represented by each subtag, which is what descriptions are 
for.

However, there is one place where we could simplify, or regularize, the 
characters used in certain Description fields without sacrificing 
legibility, uniqueness, or identity.  These are the various apostrophes 
and apostrophe-like modifier letters.

Right now, the following apostrophes and modifier letters appear in 
Description fields in the Registry:

U+0027 APOSTROPHE
    Description: Mi'kmaq
    Description: Ethiopic (Ge'ez)
    Description: Ol Chiki (Ol Cemet', Ol, Santali)
    Description: C&#xF4;te d'Ivoire
    Description: Korea, Democratic People's Republic of
    Description: Lao People's Democratic Republic

U+00B4 ACUTE ACCENT
    Description: Gwich&#xB4;in

U+02BB MODIFIER LETTER TURNED COMMA
    Description: Ethiopic (Ge&#x2BB;ez)

U+2019 RIGHT SINGLE QUOTATION MARK
    Description: N&#x2019;Ko  (both language and script)

Most people seem to agree that the acute accent used in Gwich'in was a 
mistake, but we used it in the Registry anyway to be perfectly faithful 
to the source standard.  RFC 4646, however, does not require this type 
of perfect fidelity, while draft-4646bis-07 adds only the following of 
relevance:

    "For fields of type 'language' or 'extlang', the first 'Description'
    field appearing in the Registry corresponds to the Reference Name
    assigned by ISO 639-3."

"Corresponds to" is not the same as "is character-for-character 
equivalent to."

The ISO 639-3 data includes 117 names containing U+0027, and none 
containing any other form of apostrophe.  This introduces a kind of 
inconsistency in which we have chosen certain types of apostrophes or 
apostrophe-like modifier letters in isolated cases, in a noble attempt 
to use "the right character," but also want to maintain fidelity with 
source standards that may not have this as a goal.  We frankly don't 
know which of the names in ISO 639-3 really "should" use some other type 
of apostrophe; indeed, ISO 639-3 spells "N'Ko" with U+0027.

In reality, even where U+0027 is not "the right character" for a given 
name, most users will expect to see it and to be able to search on it 
any time they see an apostrophe-like character.  Unlike the various 
accented letters, the apostrophes all look pretty much the same in small 
type sizes.  No Description fields in the Registry are now, or will be 
in the 4646bis era, distinguished only by the type of apostrophe used; 
this will not be true for other types of non-ASCII letters, as mentioned 
above.

Likewise, it seems quite unlikely that any user will perform lexical 
analysis on Description or Comments fields, such that strings containing 
the "wrong" apostrophe will be processed incorrectly in some way.

I propose the following:

1.  In RFC 4645bis, all apostrophes and apostrophe-like modifier letters 
that appear either in the current Registry or in one of the source 
standards be transformed to the plain ASCII apostrophe, U+0027.

2.  Existing pairs of Description fields that differ only in the type of 
apostrophe (e.g. "Ethiopic (Ge'ez)") be reduced to a single field.

3.  Future registrations use only U+0027 in place of other 
apostrophe-like characters.

I believe items 1 and 2 affect only RFC 4645bis, but I invite the 
editors and others to check whether 4646bis is affected as well.

Again, cogent arguments against this proposal are likely to be more 
productive than flames.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Michael Everson <[email-masked]>
Date: Mon, 6 Aug 2007 09:58:03 +0100

Why are you asking me, Doug? You know what my view is.
-- 
Michael Everson * http://www.evertype.com

---

From: Addison Phillips <[email-masked]>
Date: Mon, 06 Aug 2007 09:42:43 -0700

> "Corresponds to" is not the same as "is character-for-character 

You're right, although I tend to think that "corresponds to" usually 
means "equals" rather than "is similar to".

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: Addison Phillips <[email-masked]>
Date: Mon, 06 Aug 2007 09:53:39 -0700

> apostrophe (e.g. "Ethiopic (Ge'ez)") be reduced to a single field.

Yes, please.

> editors and others to check whether 4646bis is affected as well.

RFC 4646bis would be affected if and only if we feel the need to 
prescribe this. I note that the sentence you quoted elsewhere in this 
message occurs in this paragraph:

--
For records taken from a source standard (such as ISO 639 or ISO 3166), 
the 'Description' value(s) SHOULD also be taken from the source 
standard. Multiple descriptions in the source standard MUST be split 
into separate 'Description' fields. The source standard's descriptions 
MAY be edited, either prior to insertion or via the registration 
process. For fields of type 'language' or 'extlang', the first 
'Description' field appearing in the Registry corresponds to the 
Reference Name assigned by ISO 639-3. This helps facilitate 
cross-referencing between ISO 639 and the registry.
--

This suggests to me that we can safely ignore this issue in 4646bis, 
trusting to the Language Subtag Reviewer and/or the reviewer's trusty 
deputy to make these sorts of edits as a matter of policy.

At that same time, I tend to think the the first 'Description' field in 
a language subtag should be character-for-character equivalent to ISO 
639-3, regardless of whether we like it or not. The stated purpose is 
cross-referencing and my browser's search function does 
character-by-character identity. You can always have a second, edited, 
version.

Addison

---

From: John Cowan <[email-masked]>
Date: Mon, 6 Aug 2007 13:45:57 -0400

> I propose the following:

+1 to all

-- 
You're a brave man! Go and break through the            John Cowan
lines, and remember while you're out there              [email-masked]
risking life and limb through shot and shell,           http://ccil.org/~cowan
we'll be in here thinking what a sucker you are!
        --Rufus T. Firefly

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 6 Aug 2007 12:32:26 -0700

Hi -

> From: "Doug Ewell" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
> Cc: "Michael Everson" <[email-masked]>
> Sent: Sunday, August 05, 2007 11:52 PM
> Subject: [Ltru] Apostrophes
...
> apostrophe-like characters.
...

As a co-chair, I'm concerned that (1) and (2) so would be exceding this
WG's proper role.  Such a decision belongs to the [email-masked]
list and the subtag reviewer, rather than this WG, in my opinion.

As a technical contributor, though I understand the motivation for (3),
I think that hard-coding this kind of a policy detail is something we might
eventually regret.  The whole point of having a reviewer and discussion list
is that we know that specifying a purely mechanical process for handling all
aspects of a registration request is impractical.  We should spend time
spelling out policy where we think the odds are good that submitters, the
reviewer, or the discussion list would otherwise be likely to err.  I'm also
concerned that (3) is just the tip of the look-alike iceberg, and that before
too long we might find ourselves specifying a stringprep profile for registry
requests.  This really feels like over-engineering to me.

Randy

---

From: Addison Phillips <[email-masked]>
Date: Mon, 06 Aug 2007 13:10:22 -0700

> list and the subtag reviewer, rather than this WG, in my opinion.

Wouldn't #1 and #2 be within scope, since it affects a work item for 
this WG? It might be a simple policy decision or a bit of editorial 
discretion (we've been satisfied with those previously). The 
registration process can always be used to alter or amend any resulting 
crud.

> requests.  This really feels like over-engineering to me.

+1

Addison

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 6 Aug 2007 13:33:00 -0700

Hi -

> From: "Addison Phillips" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Monday, August 06, 2007 1:10 PM
> Subject: Re: [Ltru] Apostrophes
>
> Randy Presuhn wrote:
...

Here's how I look at it:  the WG's job is to define policy/process, 
structure and semantics.

The reviewer & list are responsible for content.

There's a bit of an intersection for bulk generation of entries derived
from other standards.  The question here is when do "automatic" transformations
of the data provided by other standards cross the line and become decisions
about content, that should properly be discussed by the language experts?

I can see how this particular case arguably falls into a grey area,
but since Doug has already done the work to identify the specific cases, my own
preference would be for those cases to be handled on the [email-masked]
list as change requests, rather than for us here to try to anticipate the
outcome of the discussions in that forum.

Randy

---

From: Addison Phillips <[email-masked]>
Date: Mon, 06 Aug 2007 13:36:57 -0700

So... I think we're saying the same thing, which is that Doug should: 
take the existing registry as an *exact* source; modify those things 
that *require* modification; add the new records (copying the source 
standards exactly); and leave any additional changes (such as making the 
apostrophes "nicer") up to ietf-langauges?

Addison

---

From: [email-masked]
Date: Mon, 6 Aug 2007 13:34:58 -0700

> character-by-character identity. You can always have a second, edited, 

I agree with Doug's apostrophe proposal to the extent that it supports 
Addison's words above. If there is an incorrectly used apostrophe and 
someone would like to make issue of it, I would suggest making that change 
through a change request to the source ISO standard. Any changes that have 
been made within the registry to date, should likely change back to 
whatever is specified in the source standard.

Two cents,

Karen Broome

---

From: Addison Phillips <[email-masked]>
Date: Mon, 06 Aug 2007 13:40:54 -0700

PS> Please note that I personally approve of fixing the various 
apostrophes. It's just that it does seem unnecessary to modify 4646bis 
to do it.

Addison

Addison Phillips wrote:
>>>>> 1.  In RFC 4645bis, all apostrophes and apostrophe-like modifier 
>>>>> letters that appear either in the current Registry or in one of the 
>>>>> source standards be transformed to the plain ASCII apostrophe, U+0027.
>>>>>
>>>>> 2.  Existing pairs of Description fields that differ only in the 
>>>>> type of apostrophe (e.g. "Ethiopic (Ge'ez)") be reduced to a single 
>>>>> field.
>>>>>
>>> registration process can always be used to alter or amend any 
>>> resulting crud.
>> ...
>>

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 6 Aug 2007 14:04:22 -0700

Hi -

> Sent: Monday, August 06, 2007 1:36 PM
...

That would be my preference, both as a co-chair and as a technical
contributor.

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 6 Aug 2007 14:06:13 -0700

Hi -

> To: "Addison Phillips" <[email-masked]>
> Cc: "Randy Presuhn" <[email-masked]>; "LTRU Working Group" <[email-masked]>
> Sent: Monday, August 06, 2007 1:40 PM

...

Ditto.

Randy

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 6 Aug 2007 20:42:45 -0700

Addison Phillips <addison at yahoo dash inc dot com> wrote:

> At that same time, I tend to think the the first 'Description' field 
> in a language subtag should be character-for-character equivalent to 
> ISO 639-3, regardless of whether we like it or not. The stated purpose 
> is cross-referencing and my browser's search function does 
> character-by-character identity.

If the intent is for the first Description field to be identical to the 
639-3 reference name -- and this is just what I have been doing in RFC 
4645bis, *except* that I proposed to ASCII-slam the apostrophes -- then 
I think the text should emphasize this a bit more.  There is lots of 
text about how Description fields are not normative.

> You can always have a second, edited, version.

No, please, no.  Let's not proliferate this kind of thing:

Frankly, I would much rather see just the "official" line with U+02BB 
than both lines.  It's back to what I said about "transcribing" accented 
Latin letters into ASCII: it proves nothing except that we created a 
character-encoding problem and had to hack our way out of it.  With 
UTF-8 this will be worse than it is today, because the two lines will 
look virtually identical:

    Description: Ethiopic (Geʻez)
    Description: Ethiopic (Ge'ez)

Randy Presuhn <randy underscore presuhn at mindspring dot com> wrote:

> As a co-chair, I'm concerned that (1) and (2) so would be exceding 
> this WG's proper role.  Such a decision belongs to the 
> [email-masked] list and the subtag reviewer, rather than this 
> WG, in my opinion.

Addison answered this adequately: (1) and (2) are questions about what 
we should do in RFC 4645bis.

> As a technical contributor, though I understand the motivation for 
> (3), I think that hard-coding this kind of a policy detail is 
> something we might eventually regret.  The whole point of having a 
> reviewer and discussion list is that we know that specifying a purely 
> mechanical process for handling all aspects of a registration request 
> is impractical.  We should spend time spelling out policy where we 
> think the odds are good that submitters, the reviewer, or the 
> discussion list would otherwise be likely to err.

One of the main reasons I bring this up here is that ietf-languages has 
been unable to establish a consistent pattern for apostrophes and 
meta-apostrophes, despite much discussion.

> I'm also concerned that (3) is just the tip of the look-alike iceberg, 
> and that before too long we might find ourselves specifying a 
> stringprep profile for registry requests.  This really feels like 
> over-engineering to me.

I have no desire to extend this beyond apostrophe-like characters.  Of 
course, others might pick up this particular ball and run with it.

Later:

> but since Doug has already done the work to identify the specific 
> cases, my own preference would be for those cases to be handled on the 
> ietf-languages at iana.org list as change requests, rather than for us 
> here to try to anticipate the outcome of the discussions in that 
> forum.

I can promise that if this is brought up on the ietf-languages list, the 
net result will be to convert the U+0027 apostrophes to more "correct" 
characters, to de-regularize them, which is exactly the opposite of what 
I had in mind.  I've already gotten one private reply to this effect.

Addison replied:

> standards exactly); and leave any additional changes (such as making 
> the apostrophes "nicer") up to ietf-langauges?

and Karen and Randy agreed.

Oh well, I tried.  Thanks to John for his summary "+1", but it looks 
like this isn't going to happen.