Subject: [Ltru] Re: Registration request: "mis" comment clarifying meaning

From: "Doug Ewell" <[email-masked]>
Date: Tue, 15 May 2007 08:26:13 -0700

Kent Karlsson <kent dot karlsson14 at comhem dot se> wrote:

> Not at all. There is no hierarchy that is **given explicitly**, but there 
> is certainly an implicit one. Vide:
>
> (set of 'gem' [Germanic] languages)  subset
> (set of 'ine' [Indo_European] languages) subset
> (set of 'mis' [natural] languages) subset
> (set of stuff covered by 'und', i.e. anything)

> With the exception of 'mis' (which must cover all natural languages) the 
> rest of the hierarchy must be figured out via some other sources of 
> information. But the intention is certainly a hierarchy, otherwise none of 
> the collection codes would make any sense.

I don't remember if I said before "this is my last post on this subject," 
but this really will be my last post on this subject.

I do not agree with Kent's model in the slightest.  "Miscellaneous" is, BY 
ITS NATURE, a different concept from all of the other "collection" concepts. 
Intuitively it means "everything that doesn't fit into some other category."

I would very much like it if ISO 639 JAC or MARC or somebody would issue 
some sort of official statement about the intended purpose of "mis" so we 
could put this discussion to rest.  I think such a statement will support my 
view of "mis", but whether it does or not, we should accept it and move on. 
We are spending great gobs of time debating these particular angels and this 
particular pin, and meanwhile there are other issues to resolve before RFC 
4646bis will be ready for WGLC.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 15 May 2007 22:54:17 +0100

Doug wrote:

> I would very much like it if ISO 639 JAC or MARC or somebody 
> would issue some sort of official statement about the 
> intended purpose of "mis" so we could put this discussion to 
> rest.  I think such a statement will support my view of 
> "mis", but whether it does or not, we should accept it and move on. 

Patience is a virtue... I believe it is on its way via the ISO 639 JAC :-)

Best

Debbie Garside

www.geolang.com 

> -----Original Message-----
> From: Doug Ewell [mailto:[email-masked]] 
> Sent: 15 May 2007 16:26
> To: LTRU Working Group
> Subject: [Ltru] Re: Registration request: "mis" comment 
> clarifying meaning
> 
> > Not at all. There is no hierarchy that is **given explicitly**, but 
> > there is certainly an implicit one. Vide:
> >
> > (set of 'gem' [Germanic] languages)  subset (set of 'ine' 
> > [Indo_European] languages) subset (set of 'mis' [natural] 
> languages) 
> > subset (set of stuff covered by 'und', i.e. anything)
> 
> > With the exception of 'mis' (which must cover all natural 
> languages) 
> > the rest of the hierarchy must be figured out via some 
> other sources 
> > of information. But the intention is certainly a hierarchy, 
> otherwise 
> > none of the collection codes would make any sense.
> 
> I do not agree with Kent's model in the slightest.  
> "Miscellaneous" is, BY ITS NATURE, a different concept from 
> all of the other "collection" concepts. 
> I would very much like it if ISO 639 JAC or MARC or somebody 
> would issue some sort of official statement about the 
> intended purpose of "mis" so we could put this discussion to 
> rest.  I think such a statement will support my view of 
> "mis", but whether it does or not, we should accept it and move on. 
> We are spending great gobs of time debating these particular 
> angels and this particular pin, and meanwhile there are other 
> issues to resolve before RFC 4646bis will be ready for WGLC.
> 
> --
> Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  
> UTN #14 http://users.adelphia.net/~dewell/

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 15 May 2007 23:57:27 +0200

> Intuitively it means "everything that doesn't fit into some 

And 'gem' BY ITS NATURE "intuitively" means "every Germanic language
[extent] that doesn't fit into some other category [use rec.]".

There is no difference in principle.

I feel like as if I was trying to explain "0" when some are only used to
a number system where "0" is unknown. But it is quite obvious once you've
had the aha, but apparently very hard before one has had the aha (compare
Roman numerals and other ancient numbering systems).

	/kent k

---

From: Martin Duerst <[email-masked]>
Date: Wed, 16 May 2007 10:57:09 +0900

[chair hat on]

Given the statement below from Debbie, the fact that this
discussion has used an enormous amount of emails, the fact
that currently arguments are mostly repeated, and my
understanding that much of the discussion is about
underlying models rather than about practical consequences,
I herewith declare to put this discussion on hold.

If you have any additional contributions, please write
them up, but don't send them as long as the discussion
is on hold.

If you have any information that may suggest that the
discussion should be resumed, please send such information
to the co-chairs.

Many thanks for your cooperation in assuring smooth
and efficient WG progress.

Regards,    Martin.

>Patience is a virtue... I believe it is on its way via the ISO 639 JAC :-)

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 22 May 2007 20:29:10 -0700

I see two tenable choices for "mis".

   1. Linguistic root: "mis" would serve a somewhat useful function if
   interpreted as the "root" of all languages, as Kent has proposed. That
   function is not the same as "und", since "und" indicates no knowledge
   about the status of the content, while "mis" says that there is
some linguistic
   content, and is thus the counterpart to zxx (there is no linguistic
   content).
   2. Deprecated. If mis is not interpreted as Root, but as "Not any of
   the above", it completely breaks stability, and we need to completely
   deprecate it, as discussed before.

Although I think the first choice would be a better one in terms of overall
usefulness for clients of 4646bis, I wouldn't be strongly against #1 either.

Related to that, Peter raise the issue of getting clarification from ISO as
to the status of the collection codes. While I think this would be very
useful, there is one important caveat: If a version of ISO 639 ever narrows
the documented, written meaning of a code according to a previously released
version ISO 639, for stability we cannot follow that narrowing, and must
keep the original broader meaning. If they broaden the meaning of a code, we
can and should follow their lead.

Mark

On 5/15/07, Doug Ewell <[email-masked]> wrote:
> http://users.adelphia.net/~dewell/ <http://users.adelphia.net/%7Edewell/>

-- 
Mark

---

From: John Cowan <[email-masked]>
Date: Wed, 23 May 2007 01:01:59 -0400

Mark Davis scripsit:

> If a version of ISO 639 ever narrows the documented, written meaning
> of a code according to a previously released version ISO 639, for
> stability we cannot follow that narrowing, and must keep the original
> broader meaning.

That's reasonable in the case of a conscious change, but doesn't apply
to the clarification of a code element whose meaning is ambiguous or
unintentionally vague.  For example, 639-2 defines the code element
'ain' as simply "Ainu".  639-3 recognizes that there are two completely
unrelated languages with that name, the fairly well-known language
isolate in Japan and an obscure Turkic language closely related to
Uyghur and spoken in China.  The former is assigned the code element
'ain' by 639-3, whereas the latter has the code element 'aib'.

A strict application of the "no narrowing" principle would require that
"ain" be applicable to both Ainu-Japan and Ainu-China, and that a new
code element has to be introduced to represent Ainu-Japan exclusively.
This would *not* help stability: on the contrary, it would damage
stability, since both 'ain' and the new code would wind up being applied
to Ainu-Japan resources, making matching less likely to succeed.

Rather we must recognize that in fact 'ain' has been applied exclusively
to Ainu-Japan resources and should continue to be so applied, despite
the hitherto unrecognized existence of Ainu-China.

A second example would be the code element 'aus', for "Australian
languages".  639-2 does not make clear whether this includes only
languages belonging to the Australian language family, or also includes
languages such as English, Greek, Turkish, and Basque which are also
spoken in the country of Australia, since both the English adjective
"Australian" and the French adjective "australiennes" are ambiguous in
this respect.  A strict interpretation of "no narrowing" would require
us to ignore any clarification of this term to exclude languages outside
the Australian family.

As a third example, the code element 'nld' (or 'nl' in 639-1) is defined
as "Dutch; Flemish".  However, "Flemish" is also used as the English
name of the nonstandard Low Franconian language Vlaams that is spoken in
Flanders.  Strictly, 'nl' could be used to represent this language as well
as standard Dutch, but again this reduces rather than increases stability.

-- 
In politics, obedience and support      John Cowan <[email-masked]>
are the same thing.  --Hannah Arendt    http://www.ccil.org/~cowan

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 23 May 2007 07:28:07 -0700

I was reminded by Martin that he had wanted no more messages on this topic
for now, so I'll reply later when he opens up the discussion again.

Mark

---

From: John Cowan <[email-masked]>
Date: Wed, 23 May 2007 18:59:02 -0400

Mark Davis scripsit:

> I was reminded by Martin that he had wanted no more messages on this
> topic for now, so I'll reply later when he opens up the discussion
> again.

AFAIK that applied only to specific discussion of "mis" (which I deleted
from your message) rather than to the general point I'm making here,
which affects a policy question.  So I look forward to your reply.

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 23 May 2007 17:27:30 -0700

Sure. The stability of codes is crucial, and at the core of the reason for
the changes in 4646. But the standard for application of any body of text,
including the RFC, has to be what the interpretation by a "reasonable"
person would be. If a reasonable, informed person can sensibly tag content X
with tag T, based on a sensible reading of a copy of RFC 4646, then we
cannot later make that application of that T to X be invalid.

Thus if Joe uses the tag "lat" to describe content in "pig latin", we can
clearly view that as not something a reasonable, informed person would do.
Thus we could in good conscience add a note clarifying that Latin doesn't
include "pig latin" (not that I think this is necessary ;-). And as in the
case you cite, the normal expectation is that "Ainu" refers to the language
spoken in Japan rather than some relatively unknown language. So adding
clarifying language is fine for such cases.

If, on the other hand, ISO said that "alg" no longer means "Algonquian
languages", it only means *some* Algonquian languages, that would clearly be
a narrowing, which we cannot support.

For "aus", it would be ok to add a clarifying note. It's generally
understood that the geographic terms in language names such as that are to
be interpreted as pertaining to languages originating in that area, and thus
not include Italian or Russian even though many Australians are fluent in
those languages.

Flemish has come up in Google, so we had to figure this out. We canvased
people, including Flemish speakers, and all equate "Dutch as used in
Belgium" (nl-BE) with "Flemish". Using the term "Flemish" for something else
is clearly very unusual, so I have no qualms about clarifying the situation
there.

Mark

---

From: [email-masked]
Date: Wed, 23 May 2007 18:13:21 -0700

A question with respect to Flemish: ISO 639-3's Dutch tag maps back to 
"nl" and "nld" but today Vlaams does not, though "Flemish" is listed as a 
synonym for Vlaams on Ethnologue. The term "Flemish" does not appear in 
ISO 639-3 though clearly that term is most commonly used in 
English-language applications.

So while I've been using "nl-BE" for Flemish, I'm now wondering why Vlaams 
is not mapped to "nl" in 639-3. Is this an oversight or intentional? 
Should recommend "vls", "nl-vls", or "nl-BE" for use in our applications 
once RFC 4646bis is released?  I scanned the list of requested changes to 
ISO 639-3 and I didn't see this.

Regards,

Karen Broome
Metadata Systems Designer
Sony Pictures Entertainment
310.244.4384

"Mark Davis" <[email-masked]> 
05/23/2007 05:27 PM

To
"John Cowan" <[email-masked]>
cc
[email-masked]
Subject
[Ltru] Re: "No narrowing" does not apply to ambiguous code elements

Sure. The stability of codes is crucial, and at the core of the reason for 
the changes in 4646. But the standard for application of any body of text, 
including the RFC, has to be what the interpretation by a "reasonable" 
person would be. If a reasonable, informed person can sensibly tag content 
X with tag T, based on a sensible reading of a copy of RFC 4646, then we 
cannot later make that application of that T to X be invalid. 

Thus if Joe uses the tag "lat" to describe content in "pig latin", we can 
clearly view that as not something a reasonable, informed person would do. 
Thus we could in good conscience add a note clarifying that Latin doesn't 
include "pig latin" (not that I think this is necessary ;-). And as in the 
case you cite, the normal expectation is that "Ainu" refers to the 
language spoken in Japan rather than some relatively unknown language. So 
adding clarifying language is fine for such cases. 

If, on the other hand, ISO said that "alg" no longer means "Algonquian 
languages", it only means *some* Algonquian languages, that would clearly 
be a narrowing, which we cannot support.

For "aus", it would be ok to add a clarifying note. It's generally 
understood that the geographic terms in language names such as that are to 
be interpreted as pertaining to languages originating in that area, and 
thus not include Italian or Russian even though many Australians are 
fluent in those languages. 

Flemish has come up in Google, so we had to figure this out. We canvased 
people, including Flemish speakers, and all equate "Dutch as used in 
Belgium" (nl-BE) with "Flemish". Using the term "Flemish" for something 
else is clearly very unusual, so I have no qualms about clarifying the 
situation there. 

Mark

--
In politics, obedience and support      John Cowan < [email-masked]>
are the same thing.  --Hannah Arendt    http://www.ccil.org/~cowan

-- 
Mark _______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Thu, 24 May 2007 00:06:11 -0400

[email-masked] scripsit:

> "nl" and "nld" but today Vlaams does not, though "Flemish" is listed
> as a synonym for Vlaams on Ethnologue. The term "Flemish" does not
> appear in ISO 639-3 though clearly that term is most commonly used in

There are two different languages involved here:  'nl(d)', which is
standard Dutch and closely related dialects; the other is 'vls', which
as I said earlier is a nonstandard language closely related to, but not
the same as, Dutch.  The terms "Flemish" and "Vlaams" have been used for
both 'vls' and for the variety of 'nl(d)' spoken and written in Flanders,
and the actual speech of Flemings often involves a mixture of the two
languages, making things extremely confusing.

Nevertheless, ISO 639-3 does not consider 'nl(d)' a macrolanguage
encompassing 'vls', and so the appropriate tag in the 4646bis regime
will be "vls".

> So while I've been using "nl-BE" for Flemish, I'm now wondering
> why Vlaams is not mapped to "nl" in 639-3. Is this an oversight or
> intentional?  Should recommend "vls", "nl-vls", or "nl-BE" for use in
> our applications once RFC 4646bis is released?  I scanned the list of
> requested changes to ISO 639-3 and I didn't see this.

"nl-vls" is definitely out, so any of "nl" or "nl-BE" or "vls" might
be appropriate, depending on how close the language being used is to
standard Dutch.  Note that "vls" has three orthographies in use, which
can be reasonably labeled "vls-BE", "vls-FR", and "vls-NL".

The situation is essentially parallel to "en" vs. "en-x-scottish"
(Scottish English) vs. "sco" (Scots).  There are many other examples
the world over.

---

From: John Cowan <[email-masked]>
Date: Thu, 24 May 2007 15:57:59 -0400

Mark Davis scripsit:

> Sure. The stability of codes is crucial, and at the core of the reason
> for the changes in 4646. But the standard for application of any body
> of text, including the RFC, has to be what the interpretation by a
> "reasonable" person would be. If a reasonable, informed person can
> sensibly tag content X with tag T, based on a sensible reading of a
> copy of RFC 4646, then we cannot later make that application of that
> T to X be invalid.

Okay, so we agree that the text alone is not enough -- one must know the
context to understand a phrase such as "Australian languages".  So much
for _sola scriptura_.  And I agree with all of your particular points.

What remains, then, is whether it's enough just to look at information
from linguistics alone, or whether we also have to look at the practices
of large segments of the tagging community, particularly when (as in
the case of MARC and Ethnologue) the ISO tags are originally derived
from their systems.  I say yes, that the MARC interpretation controls
unless specifically overridden by ISO.

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 24 May 2007 13:11:20 -0700

No. We cannot reasonably expect people to traverse through the history of
how codes came to be; that would be a bizarre expectation, since they would
not only have to research exactly how they came to get into the standard,
but they would also have to know all of the internal history involved, often
only in people's heads, in order to know whether or not the meaning changed
in the adoption.

To take an analogy, if I want to know what a particular character in Unicode
means, I need to look at the text of the standard and associated data, as
viewed by a reasonably knowledgeable person. We CANNOT expect such people to
go look at all the internal documents that led up to the encoding of the
character, (a) because it's just plain wrong to have that expectation, and
(b) because the interpretation and usage often changes by the time it is
encoded.

Mark

---

From: "McDonald, Ira" <[email-masked]>
Date: Thu, 24 May 2007 18:44:52 -0700

Hi,

While I sympathize with Mark's reply below, I think
John's making a very important point - the expectation
about the behavior of a "reasonable" person in tagging
language content based on RFC 4646(bis) is in fact
unreasonable.

To say that we can't allow in a future revision of
the spec (or the registry) precluding a previously
"reasonable" language tagging behavior is much too
fuzzy.

If only a reasonably knowledgeable human being can
"understand" that, in the future, the tag "vls" would
be preferred over "nl-BE" for (one of the) languages
spoken in the northern part of modern Belgium, how
is a mere piece of software supposed to figure this
out and concatenate two apparently disjoint match
sets?

The Wikipedia article on "Flanders" is fascinating
<http://en.wikipedia.org/wiki/Flanders>, but it
leaves the use of the term "Flanders" and "Flemish"
as largely eye-of-the-beholder.  

Cheers,
- Ira

Ira McDonald (Musician / Software Architect)
Chair - Linux Foundation Open Printing WG
Blue Roof Music / High North Inc
PO Box 221  Grand Marais, MI  49839
phone: +1-906-494-2434
email: [email-masked] 
-----Original Message-----
From: Mark Davis [mailto:[email-masked]]
Sent: Thursday, May 24, 2007 3:11 PM
To: John Cowan
Cc: [email-masked]
Subject: [Ltru] Re: "No narrowing" does not apply to ambiguous code elements

No. We cannot reasonably expect people to traverse through the history of how codes came to be; that would be a bizarre expectation, since they would not only have to research exactly how they came to get into the standard, but they would also have to know all of the internal history involved, often only in people's heads, in order to know whether or not the meaning changed in the adoption. 

<...snip...>

Mark

No virus found in this outgoing message.
Checked by AVG Free Edition. 
Version: 7.5.467 / Virus Database: 269.8.0/817 - Release Date: 5/24/2007 4:01 PM

---

From: John Cowan <[email-masked]>
Date: Thu, 24 May 2007 22:13:40 -0400

Mark Davis scripsit:

> To take an analogy, if I want to know what a particular character in
> Unicode means, I need to look at the text of the standard and associated
> data, as viewed by a reasonably knowledgeable person.

ISO 639 is severely under-documented compared to Unicode or even
ISO 10646.