Subject: [Ltru] 639 coding wrt historic varieties (was RE: Request for variant subtag fr 16th-c 17th-c Resubmitted!)

From: Peter Constable <[email-masked]>
Date: Thu, 11 Jan 2007 04:55:51 -0800

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Monday, December 18, 2006 6:35 PM

Mark: Sorry for the delay in responding to this.

> Peter, I'm a little bit fuzzy on where the lines were 
> drawn with historic versions of the same language in ISO. 
> This is relevant to the LTRU group for ISO 639-3, so am 
> cc'ing that group.

Entities coded in 639-1 (at least, any coded in 639-1 prior to 639-2) were probably done for terminology work, which would almost certainly mean they were done for modern forms of languages.

The historic varieties coded in 639-2 probably originated in MARC and hence were done on the basis of the practice of librarians (and those primarily in Western nations). I don't know on what basis they chose the boundaries that they did. 

> I take it from your discussion that "fr" means *only* 
> modern French, and that if I want to have a tag for any 
> French, modern or not, I would have to use (fr OR frm 
> OR fro). Similarly, if I wanted any English, I would 
> have to use (en OR enm OR ang). 

That's my understanding. 

Since 639-1/-2 only provide English and French names as a guide to the semantics of each coded entity, it isn't always clear from a given entry what it means. To gain a more complete idea of the semantics of entries, there are a couple of things we can look at: what distinctions are made within the coded set, and how a given ID been used. (For the latter, MARC is particularly relevant since it was the source from which most of the entities coded in 639-2.)

As I mentioned above, "fr" was originally used by terminologists, in which context it very likely has meant the modern variety only. The corresponding alpha-3 ID "fre" was originally used by librarians. While in that context it's more likely that the ID potentially could have been used in a way that encompassed historic varieties, the contrastive coded entities "frm" and "fro" that they also used clearly suggests that that was probably not how they used it. The MARC Code List for Languages (http://www.loc.gov/marc/languages/) is consistent with that: in describing how "fre" is used, they document it as encompassing varieties from different regions, but not historic varieties.

Now, I admit the question of whether a coded entity like "fr" could encompass historic varieties had not occurred to me until it came up recently in ietf-languages / LTRU (whichever it was). And my guess is that it has never been considered by the JAC as well: the idea that an ID might be used for multiple varieties certainly existed in MARC, but it appears that that was never done in MARC with historic varieties; and apart from that, the JAC had no occasion to discuss an ID encompassing multiple varieties (except for the obvious case of collections) until ISO/CD 639-3 introduced macrolanguages. But given the distinctions made in the code set and the usage of 639-1/-2 IDs by terminologists and librarians, I would conclude that ISO 639 currently does not have identifiers that are intended to encompass varieties from multiple time depths.

> My question is how this is managed over time in ISO, 
> since there are significant implications for language tags.

It's a good question, and one I'm sure hasn't been considered by the JAC.

> Let's take Czech, for example, where we only currently have 
> 'cs'. I see the following possibilities. 
>
> 1. This means only Modern Czech.
> That implies that there is no code for Old Czech, so if I 
> want to tag something with that, I need to petition ISO 
> for an language tag for Old Czech (let's say 'ceu'). Once 
> that is added, I can refer to Old Czech. 
>
> 2. This currently means any Czech, but ISO may introduce 
> a code for Old Czech (let's say 'ceu'). 

As I mentioned above, I think for use of 639-1 in terminology probably only modern varieties would have been relevant. As for 639-2, the fact that the practice has been to differentiate between historic varieties could be seen to suggest that IDs refer specifically to modern varieties unless otherwise noted. On the other hand, it's easy to imagine that there are users out there (including librarians) that might have done otherwise.

> I see three possible approaches:
>
> 2a. The denotation of 'cs' is changed to mean only modern 
> Czech. This would be a breaking change for the language 
> subtag registry, since the meaning of a subtag would be 
> narrowed, invalidating any tags that had a broad 
> application. This would be rather disturbing, since we are 
> guaranteeing stability. 

If "cs" were deemed to encompass historic Czech varieties as well as modern, then this would be a breaking change for users in general, and so would not be a good idea.

> 2b. The denotation of 'cs' remains "any" Czech, and to get 
> only modern Czech I would need to use (cs AND NOT ceu). Note 
> while OR can be handled with a list, as per RFC 4647, AND 
> NOT cannot. This, however, would not break stability. 

Of course, if "cs" encompasses all historic varieties, then just as an ID for Old Czech can be added, and ID for specifically Modern Czech can also be added.

> 2c. The denotation of 'cs' remains "any" Czech, ceu becomes 
> an extlang. This I see as the least unpleasant outcome, but 
> I can't tell if whether this would be the ISO policy.

I think if "cs" were deemed to encompass historic varieties, then effectively that makes it like a macrolanguage entity, even if each of the individual languages (modern, middle, etc.) are not currently coded. So then, if one of the historic varieties does get coded in 639, then it would make complete sense to treat that as extlang in 4646bis.

Of course, the crucial question is whether "cs" (or "hi" or "ka" or ...) is intended to mean only the modern language or is intended to encompasses modern and historic varieties. This is an open question that could be put to the JAC. 

I'm somewhat inclined to say that all the coded entities in 639 should be understood to be the modern variety unless explicitly indicated otherwise. Given the vast majority of existing usage of IDs and the existing coding practice of distinguishing historic varieties from modern ones, that would maintain the greatest level of consistency. I think it would be confusing for users if the historic varieties were treated in different ways for different cases. Of course, there will probably be situations in which some user uses an ID for something that arguably is a distinct historic variety and should have a different ID (but no separate ID exists). But I think that when there is a significant need for an ID for a historic variety, users can request them, and the JAC is likely to grant them. I'm open to other ideas on this, though.

Peter Constable
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 11 Jan 2007 06:54:25 -0800

Thanks for the detailed responses, which all sound sensible. Can you put the
"crucial question" to the JAC, so we can get some clarity on how to proceed?

I'm inclined to agree with your inclination "to say that all the coded
entities in 639 should be understood to be the modern variety unless
explicitly indicated otherwise". (The boundaries for the modern languages
will come up for the ietf languages group, for example, if someone tries to
register a variant for 12th century Czech. We will need to know that that
requires first a language subtag for "Old Czech (ca. 1000-1400)" to be
encoded as a prefix for that, and route the applicant to the JAC.)

Mark

On 1/11/07, Peter Constable <[email-masked]> wrote:
> Entities coded in 639-1 (at least, any coded in 639-1 prior to 639-2) were
> probably done for terminology work, which would almost certainly mean they
> were done for modern forms of languages.
>
> The historic varieties coded in 639-2 probably originated in MARC and
> hence were done on the basis of the practice of librarians (and those
> primarily in Western nations). I don't know on what basis they chose the
> boundaries that they did.
>
>
> Since 639-1/-2 only provide English and French names as a guide to the
> semantics of each coded entity, it isn't always clear from a given entry
> what it means. To gain a more complete idea of the semantics of entries,
> there are a couple of things we can look at: what distinctions are made
> within the coded set, and how a given ID been used. (For the latter, MARC is
> particularly relevant since it was the source from which most of the
> entities coded in 639-2.)
>
> As I mentioned above, "fr" was originally used by terminologists, in which
> context it very likely has meant the modern variety only. The corresponding
> alpha-3 ID "fre" was originally used by librarians. While in that context
> it's more likely that the ID potentially could have been used in a way that
> encompassed historic varieties, the contrastive coded entities "frm" and
> "fro" that they also used clearly suggests that that was probably not how
> they used it. The MARC Code List for Languages (
> http://www.loc.gov/marc/languages/) is consistent with that: in describing
> how "fre" is used, they document it as encompassing varieties from different
> regions, but not historic varieties.
>
> Now, I admit the question of whether a coded entity like "fr" could
> encompass historic varieties had not occurred to me until it came up
> recently in ietf-languages / LTRU (whichever it was). And my guess is that
> it has never been considered by the JAC as well: the idea that an ID might
> be used for multiple varieties certainly existed in MARC, but it appears
> that that was never done in MARC with historic varieties; and apart from
> that, the JAC had no occasion to discuss an ID encompassing multiple
> varieties (except for the obvious case of collections) until ISO/CD 639-3
> introduced macrolanguages. But given the distinctions made in the code set
> and the usage of 639-1/-2 IDs by terminologists and librarians, I would
> conclude that ISO 639 currently does not have identifiers that are intended
> to encompass varieties from multiple time depths.
>
>
> As I mentioned above, I think for use of 639-1 in terminology probably
> only modern varieties would have been relevant. As for 639-2, the fact that
> the practice has been to differentiate between historic varieties could be
> seen to suggest that IDs refer specifically to modern varieties unless
> otherwise noted. On the other hand, it's easy to imagine that there are
> users out there (including librarians) that might have done otherwise.
>
>
> If "cs" were deemed to encompass historic Czech varieties as well as
> modern, then this would be a breaking change for users in general, and so
> would not be a good idea.
>
>
> Of course, if "cs" encompasses all historic varieties, then just as an ID
> for Old Czech can be added, and ID for specifically Modern Czech can also be
> added.
>
> I think if "cs" were deemed to encompass historic varieties, then
> effectively that makes it like a macrolanguage entity, even if each of the
> individual languages (modern, middle, etc.) are not currently coded. So
> then, if one of the historic varieties does get coded in 639, then it would
> make complete sense to treat that as extlang in 4646bis.
>
> Of course, the crucial question is whether "cs" (or "hi" or "ka" or ...)
> is intended to mean only the modern language or is intended to encompasses
> modern and historic varieties. This is an open question that could be put to
> the JAC.
>
> I'm somewhat inclined to say that all the coded entities in 639 should be
> understood to be the modern variety unless explicitly indicated otherwise.
> Given the vast majority of existing usage of IDs and the existing coding
> practice of distinguishing historic varieties from modern ones, that would
> maintain the greatest level of consistency. I think it would be confusing
> for users if the historic varieties were treated in different ways for
> different cases. Of course, there will probably be situations in which some
> user uses an ID for something that arguably is a distinct historic variety
> and should have a different ID (but no separate ID exists). But I think that
> when there is a significant need for an ID for a historic variety, users can
> request them, and the JAC is likely to grant them. I'm open to other ideas
> on this, though.
>
>
>
> Peter Constable
>

-- 
Mark

---

From: "David Starner" <[email-masked]>
Date: Thu, 11 Jan 2007 15:01:43 -0600

> encoded as a prefix for that, and route the applicant to the JAC.)

There's several problems I see. The chronological lines between
languages are as arbitrary as any, and there is no way to predict when
the committee would draw the line between modern and pre-modern Czech.
So a user _cannot_ know whether cs applies to their 15th century Czech
books or not without bringing it up to the committee.

Secondly, this breaks at least some current usage. For example, I
tagged "Primitive & mediaeval Japanese texts" (10th-12th century
Japanese) and "Ars grammaticae Iaponicae linguae" (17th century
Japanese) as including ja when entering them into the Distributed
Proofreaders database. When it comes to Project Gutenberg, it will be
entered as ja. The LoC seems to have entered the first as eng, despite
having one volume of ja-Latn, but I'm sure there's medieval Japanese
in a library database marked as ja somewhere.

Third, right now, the tagging for ancient languages is woefully
insufficient. Unless someone's willing to put a lot of work into
cataloging recorded ancient languages, a library like PG will end up
using the modern language tags for ancient languages instead of
fighting for dozens of new tags. To follow this path involves dozens
of new tags, which probably won't successfully be created piece-meal.

PS. I'll point out this has slightly absurd results when applied to
la. I think we can assume that la was designed to apply to Classical
Latin, and not (not just?) the modern variety.

PS. #2 I can't think of any way this affects tlh. Isn't everyone happy
about that?

---

From: "CE Whitehead" <[email-masked]>
Date: Thu, 11 Jan 2007 14:38:52 -0500

>"crucial question" to the JAC, so we can get some clarity on how to 

I guess we could question the JAC
>explicitly indicated otherwise".

>>regions, but not historic varieties.

Two things:

1.
In my French studies we used Old French to refer to Old French.  It is 
semi-comprehensible but not completely so to speakers of modern French.

We rarely talked about Middle French however although the term is there.

However all these could be encompassed (albeit somewhat loosely, laxly, or 
something) under the larger term French but this is an extension of course 
of the meaning of French which actually means a very specific modern 
language.

2.
The regional varieties sometimes can be as different as the historical ones. 
  I guess I need to find out which regional varieties were encompassed by 
fre, by checking the document at http://www.loc.gov/marc/languages/ (can I 
find out there?) and getting back to you.

I suppose one option would be to have a tag
fr-ext
and then that would be more like a macro tag for an extended meaning of 
French
and then you could have
fr-ext-16s
fr-ext-17s
fr-ext-frm

but that is bizarre and the variant form ext extending the French would be 
nesting (I guess, or something?) which I understand is not to be done.

So we might need some new tag but not fr with a variant ext then;
and not fre which aready means something else; fra I do not like either; and 
fr by itself of course has only its specific meaning perhaps

frx?  I'm not sure what I think of that.

--C. E. Whitehead
[email-masked]

_________________________________________________________________
The MSN Entertainment Guide to Golden Globes is here.  Get all the scoop. 
http://tv.msn.com/tv/globes2007/?icid=nctagline2

---

From: John Cowan <[email-masked]>
Date: Fri, 12 Jan 2007 13:00:08 -0500

CE Whitehead scripsit:

> By the modern variety are you referring to the Latin that was used in the 
> Catholic service until recently and to the Latin used in various phrases 
> such as "e pluribus unum?"

There's a lot more than that, from Newton's _Principia Mathematica_
to the weekly broadcast of world news in Latin from Finland.
See http://en.wikipedia.org/wiki/Neo-Latin for details, but generally
everything written after 1600 is considered Modern Latin.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
The whole of Gaul is quartered into three halves.
        -- Julius Caesar

---

From: "CE Whitehead" <[email-masked]>
Date: Fri, 12 Jan 2007 12:41:05 -0500

>books or not without bringing it up to the committee.

+1 to this; this is the case with the French stuff too.
In the late 16th century some of the texts are almost modern French (15th 
century French is clearly moyen francais); but even into the 17th century, 
when at the peripheries of French culture, some of the texts still are more 
like the earlier 16th century moyen francais texts.

So while I support maybe having more macrolanguages incuding a macrolanguage 
for the various languages related to French, I do not think that the 
16th-17th century issue can be resolved without a macrolanguage, by allowing 
the content writer to decide what tag is most appropriate for a given text, 
fr (French) or frm (moyne francais).

By the modern variety are you referring to the Latin that was used in the 
Catholic service until recently and to the Latin used in various phrases 
such as "e pluribus unum?"
There's also the variety used by the Goliard poets, Medieval Latin.

>of new tags, which probably won't successfully be created piece-meal.

I suspect that tagging is still worse for ancient varieties of non-Western 
non-Classical languages.
I think we'll probably have to go about it all piecemeal though (on an as 
sought-for basis) and that way each case may get the attention it merits, as 
I think all these varieties merit special attention to prevent the tags from 
becoming just rigidly applied and ultra-systematic (for the sake of 
expediency) and perhaps not as meaningful and as useful as they should be.

--C. E. Whitehead
[email-masked]

_________________________________________________________________
Your Hotmail address already works to sign into Windows Live Messenger! Get 
it now 
http://clk.atdmt.com/MSN/go/msnnkwme0020000001msn/direct/01/?href=http://get.live.com/messenger/overview