Subject: [Ltru] Language and script encoding standards

From: John Cowan <[email-masked]>
Date: Wed, 12 Jul 2006 15:59:02 -0400

I have just sent the following email to Neel Smith with respect to his
page "Developing standards for encoding languages and writing systems
in the editing of Greek and Latin texts".

Dear Dr. Smith:

I write to you as a member of LTRU, the IETF working group responsible
for RFC 3066bis, and as a long-term member of ietf-languages, the mailing
list that actually registers language tags.  I do not, however, speak
officially for either of these.

I have read with interest your page on language and script encoding
at http://chs75.harvard.edu/projects/diginc/techpub/language-script after
Chris Lilley of W3C drew attention to it on the www-international
mailing list.  I'll send all three lists copies of this email and (with
your permission) any reply you send me.

I wish to bring to your attention a variety of minor difficulties with
the statements and suggestions on that page, in the hope that we can
work together to bring about a satisfactory resolution which provides
for all the concerns of the classics community.

1.  The xml:lang attribute technically does not yet support RFC 3066bis,
and RFC 3066bis does not incorporate all the codes of ISO 639-3 (that
must wait until ISO finalizes ISO 639-3 and IETF issues RFC 3066ter).
However, these are mere matters of timing, and in substance there is no
reason why such codes cannot be used immediately.

2.  The distinction between standard Greek and the epichoric alphabets is
not one of "script" as that term is defined in ISO 15924.  The Estrangelo,
Western, and Eastern varieties of Syriac are distinguished there because
they use fundamentally different letter shapes, on a par with the
difference between Carolingian/Antiqua, insular, and Fraktur varieties
of Latin script.  Epichoric alphabets, on the other hand, differ in
orthography rather than in script: they use different conventions for
assigning sounds to Greek letters, and in some cases use additional
letters, just as is the case for English, German, and Icelandic, all
of which share the Latin script.  The same remarks apply to 23-letter
and 26-letter varieties of the Latin language: these are different
orthographies rather than different scripts.

3.  Likewise, the dialects of Ancient Greek were not (as far as I know)
mutually unintelligible, and therefore should not be given separate
language codes in ISO 639-3 according to the principles of that standard.
(The line drawn between Ancient and Modern Greek there is obviously
arbitrary, and is inherited from earlier parts of ISO 639.)

4. Treating beta code and UTF-8 on a par with each other is a confusion
of levels.  UTF-8 (and other kinds of Unicode), like ASCII or the various
ISO 8859 standards, are encodings representing a mapping from characters
to bits.  Beta code, on the other hand, is a transliteration standard
for Ancient Greek, representing a mapping from the Greek character
repertoire to the ASCII repertoire.  There is nothing preventing a
document in beta code from being represented in an encoding other than
ASCII, as long as that encoding supports the ASCII repertoire (as in
practice all encodings do).  There is no need to represent the encoding
of an XML document using xml:lang (you cannot even parse the document
until you have determined its encoding), but there is need to represent
any transliteration standard that is in use.

5.  Fortunately, the variant subtag mechanism of RFC 3066bis provides a
solution for all three of these problems.  By registering variant subtags
(a fairly quick and easy process), it is possible to create tags that
specify epichoric orthographies, dialects, and transliteration methods
to any desired degree of detail.  In order to do this, it would be
necessary to fix an order in which these variants should appear (with
the understanding that any or all may be omitted) and then propose the
variants themselves, each with an associated 5-letter to 8-letter subtag.
We already have variant subtags for the old and new German orthographies,
for Slovenian dialects (as you note), and have discussed transliteration
subtags, though without coming to definite conclusions.

6.  The rules of RFC 3066bis require that "la" rather than "lat" be used
to represent the Latin language (always use 2-letter ISO 639-1 rather
than 3-letter ISO 639 tags when available), and strongly recommend that
"Grek" be omitted from tags beginning with "grc", since Greek is the
normal and usual script for Ancient Greek (grc-Linb would be suitable
for Linear B texts, though).

I would urge you to join ietf-languages using the web page at
http://www.alvestrand.no/mailman/listinfo/ietf-languages and discuss
the matter further in a public forum.

-- 
John Cowan      [email-masked]        http://www.ccil.org/~cowan
        Is it not written, "That which is written, is written"?

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Martin Duerst <[email-masked]>
Date: Fri, 14 Jul 2006 17:46:28 +0900

Hello John,

Many thanks for doing this so quickly and precisely.

With regards to transliterations, I think two uses have to be
distinguished:
a) Transliterations for human users. These will continue to be
   used.
b) Transliterations for computers, i.e. to get around limitations
   in encodings or software. Beta coding is clearly such an example.
   These will die out. Rather than copying beta code into XML,
   converting it back to the real stuff is highly advisable
   (and a tool is apparently provided on that Web site).

With respect to not yet encoded scripts, such as those mentioned in
the article, there are various cases:
- Things not really scripts, rejected (e.g. Klingon)
- Variants that are not considered independent scripts
- Not yet encoded scripts. In this case, it's much more
  useful to work on documentation and stuff for getting
  the script encoded in Unicode rather than to invent yet
  another transliteration.

Regards,    Martin.

>the matter further in a public forum.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Addison Phillips" <[email-masked]>
Date: Fri, 14 Jul 2006 08:16:23 -0700

>   the script encoded in Unicode rather than to invent yet

Which makes me curious: is there any reason that a not-yet-encoded script
cannot have an ISO 15924 script code? There doesn't seem to be a reason not
to assign a script code if the script is well identified.

Addison

Addison Phillips
Internationalization Architect - Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: John Cowan <[email-masked]>
Date: Fri, 14 Jul 2006 11:20:39 -0400

> to assign a script code if the script is well identified.

There is not only no reason, but such codes actually already exist.
Consider Blissymbolics, Cirth, Cypriot, Egyptian this'n'that, Indus ...
the list goes on.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
   There was an old man                Said with a laugh, "I
     From Peru, whose lim'ricks all      Cut them in half, the pay is
       Look'd like haiku.  He              Much better for two."
                                             --Emmet O'Brien

---

From: Michael Everson <[email-masked]>
Date: Fri, 14 Jul 2006 16:47:19 +0100

>to assign a script code if the script is well identified.

No, and there are some non-Unicode scripts which have script codes. 
We tend toward conservatism here, but are waiting for the proposal 
for Ewellic to come through. ;-)
-- 
Michael Everson * http://www.evertype.com

---

From: "Addison Phillips" <[email-masked]>
Date: Fri, 14 Jul 2006 08:25:39 -0700

> Consider Blissymbolics, Cirth, Cypriot, Egyptian this'n'that, 
So you can see where I'm going with this: script variations should not use
variants or private use codes. They should be referred to ISO 15924.

Addison

---

From: John Cowan <[email-masked]>
Date: Fri, 14 Jul 2006 12:12:25 -0400

> variants or private use codes. They should be referred to ISO 15924.

Doubtless.  But I doubt very much that any of these supposed script variations
are anything of the sort.

-- 
Winter:  MIT,                                   John Cowan
Keio, INRIA,                                    [email-masked]
Issue lots of Drafts.                           http://www.ccil.org/~cowan
So much more to understand!
Might simplicity return?                        (A "tanka", or extended haiku)

---

From: John Cowan <[email-masked]>
Date: Fri, 14 Jul 2006 13:38:07 -0400

> a) Transliterations for human users. These will continue to be

Agreed.

>    These will die out. 

Beta code still has substantial advantages in its niche market:

1) It's easier to keyboard than full Unicode;

2) It's easier to extend by agreement, and continues to handle
matters that are either not yet in Unicode or will never be in
Unicode, being appropriate for high-level protocols.

Don't count it out yet.

-- 
A: "Spiro conjectures Ex-Lax."                  John Cowan
Q: "What does Pat Nixon frost her cakes with?"  [email-masked]
  --"Jeopardy" for generative semanticists      http://www.ccil.org/~cowan

---

From: Tom Elliott <[email-masked]>
Date: Fri, 14 Jul 2006 10:13:37 -0500

Dear list members,

Apologies in advance for this intervention from a new subscriber, but I 
am directly involved in some of these matters (details below).

I greatly appreciate Martin's distinction between types of 
transliteration -- it is very important that we have a way to indicate 
human-use transliterations for a number of reasons (I'll spare you 
specific use cases unless requested).

I would like to offer a caveat with regard to his observation concerning 
the second class of transliteration:

>    in encodings or software. Beta coding is clearly such an example.

Certainly this will eventually be the case, and some of us hope that day 
comes sooner than later; however, the fact is that there are at least 2 
factors that will delay the demise of beta code:

1. There are some well-established projects with large beta-coded text 
collections and V&V'd tooling for working with those texts (think 
search; indexing; linguistic parsing; transformation to other 
transliteration schemes, legacy font encodings and Unicode Greek of one 
or another composition flavor). There is no guarantee that these 
projects have the overhead or funding to do retrospective conversion of 
these text bases -- let alone the re-engineering of all their tooling -- 
in any kind of rapid way. There's just not that much money sloshing 
around in humanities tech.

Now, I'm willing to admit that what goes on behind the scenes in such 
systems needn't be foisted upon the wider, web-using public; however, I 
can envision scenarios in which such projects might care to share data 
with each other (e.g., via a web service) in which beta remains the 
carrier for the Greek. Would they not want a way to communicate 
explicitly about this fact?

2. There is a subset of the user base that would prefer to work with 
beta-coded Greek texts natively (I can't quantify the number, but many 
are actively productive scholars and some are fully tech-savvy). These 
are mostly people for whom one or more of the following is true:

(a) they learned to type and read beta a long time ago and are 
completely comfortable and speedily effective with it (more so than, 
say, even with a good keyboard interpreter);

(b) they work on older platforms where either it's not possible (or a 
giant pain) to do either/both Unicode input/or and display. They are not 
likely to pursue upgrades any time soon, as their funds are limited 
and/or they do not see the point: they're focused on achieving certain 
research and publishing goals and the methods they currently use are 
efficient and effective in achieving those goals. For most, if a clear 
case is made for *more* efficiency and effectiveness in achieving those 
*specific* personal goals, then the willingness to upgrade and learn new 
tricks follows as a matter of course. But this sort of narrowly-focused 
calculus can often be difficult to achieve.

>   the script encoded in Unicode rather than to invent yet

There are scholarly communities that will continue to use established 
transliteration schemes -- for various considered reasons -- regardless 
of the presence of the script-in-question's presence in Unicode. 
Etruscan is an example.

I make these comments, not in any attempt to disagree, but rather to 
communicate some of the background and facts on the ground in the hopes 
that complete and effective solutions can be found. I should also say 
that I cannot take credit for authorship of Neel's document, but I have 
been involved in some of the CHS Technical Working Group discussions 
that lie behind and around it. The issues are directly relevant to my 
work in a number of contexts, including:

http://www.unc.edu/awmc/pleiades.html
http://epidoc.sf.net

Best,
Tom

-- 
Tom Elliott, Ph.D.
Director, Pleiades Project
Ancient World Mapping Center
University of North Carolina at Chapel Hill
http://www.unc.edu/awmc/pleiades.html