Subject: [Ltru] Re: "mis" update review request

From: "Mark Davis" <[email-masked]>
Date: Fri, 13 Apr 2007 11:49:36 -0700

You are right about number one; I concede that point fully -- I'd overlooked
that phrasing, sorry. That means that in ISO 639-2 the following are
collections:

mul             Multiple languages
art             Artificial (Other)

plus more normal cases:

afa             Afro-Asiatic (Other)
alg             Algonquian languages
...

and the following are not collections.

und             Undetermined
zxx             No linguistic content

I disagree with your point number two. The only reason that I can conclude
that "Miscellaneous Expenses" in a spreadsheet excludes the other listed
categories is that I know that everything listed is intended to be a
partition. The language codes clearly do not form a partition, since some
collections encompass other codes. The collection codes that are tagged with
(Other) are clearly meant to be the remainder of partitions, but for the
collections that are not tagged with (Other) there is no evidence that they
were intended to exclude other cases -- if anything, the contrary -- if they
had meant to be the remainder of partitions, they would have said (Other).

Moreover, while some may be perfectly willing to have stability go by the
wayside, it is extremely important to us, and that is one of the guiding
principles of and reasons for BCP 47. That means that if I validly and
correctly tag content with "mis", that application cannot be made incorrect
by any future change to BCP 47. That is why we can broaden the application
of codes, but cannot narrow them. There is no evidence in BCP 47 that I
cannot correctly tag the content "kind" with "mis". Now, of course, we all
know that we should tag with as much information as we can, so I *should*
tag that with "en" if I know I mean the English word, and "de" if I mean the
German, or possibly others. If I don't know which one it is, if my protocol
allows multiple tags I can use "en, de", and if not I am forced into a
choice between "mul" or choosing the 'most likely' language of the set.

Mark

On 4/13/07, John Cowan <[email-masked]> wrote:
>
> Mark Davis scripsit:
>
> > Saying that mis is a collection is not breaking, but also not
> > substantiated by ISO 639-2.
>
> It's just unfathomable to me how you can get that reading of the standard.
>
> > http://www.loc.gov/standards/iso639-2/normtext.html
>
> which says (section 4.1.1, second sentence):
>
>         The words *languages* or *(other)* as part of a language name
>         in the following tables may be taken to indicate that a language
>         code is a collective language code.
>
> > http://www.loc.gov/standards/iso639-2/php/code_list.php
>
> which says (s.v. "mis")
>
>         Miscellaneous languages
>
> Note the word *languages*.  Game, set, and match (or Q, E, and D).
>
>
> > Saying that "which don't belong to any other collection" is a breaking
> > change, *and* is not substantiated by ISO 639-2 at the time the code was
> > added to the registry (or even now).
>
> Come, come.  Do you expect us, the members of this list, to suppose
> that when a spreadsheet contains the line "Miscellaneous expenses"
> you will find there charges for capital construction or salaries?
> You will not.  You will find expenses *that do not fit into any
> other category* on the spreadsheet.
>
> --
> One art / There is                      John Cowan <[email-masked]>
> No less / No more                       http://www.ccil.org/~cowan
> All things / To do
> With sparks / Galore                     -- Douglas Hofstadter
>

-- 
Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 13 Apr 2007 12:21:47 -0700

Hi -

The thing that bothers me about the comment "A collection of languages which
don't belong to any other collection" is that it isn't compatible with the
possibility that one or more of those languages might eventually be included
in another (possibly new) collection.  If that langauage were left in the mis
collection as well, the comment would be incorrect.  If the language were
removed from the mis collection, stability goes out the window.  Either way,
changing the comment wouldn't help the situation.

However, unlike some other collections, I find it very difficult to imagine
a case where "mis" would be useful in tagging data.  Without a clear use case
for "mis" in constructing a language tag, perhaps we could conclude that this
whole debate is really academic, and that no action is needed.

Randy

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 13 Apr 2007 13:04:48 -0700

I agree with you about the stability issue.

I think part of the problem in communicating about this is that people may
have somewhat different usage scenarios in mind. If you think of tagging as
something that a person does with content that they originate or have
control over, then it is (probably) fairly straightforward for that person
to tag as specifically as possible.

Another scenario is where you have incoming content, and you need to tag it
for use by other components. This might be done, for example, in a search
engine, where you fetch and process a page, and use that information later
in doing searches. The tag serves to communicate language between the
different components.

In that case, you have far from perfect information about the content: what
you have being typically the result of some level statistical analysis, plus
other factors about the document. You need to tag with as much information
as you have, *but no more*. It is in that case where you need to have the
tags that indicate some level of imperfect knowledge about the source, such
as "I have no idea what this is", or "It looks like linguistic content, but
I don't know which language", or "it doesn't look like linguistic content".
(You may also have more detailed knowledge, like that some document appears
to have 70% English content (probability 95%) and 20% French content
(probability 65%)).

Both scenarios are equally valid use cases for BCP 47 (in fact, as a
percentage of data flow on the web, I'd wager strongly that the second
scenario completely swamps the first).

Mark

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 16:15:41 -0400

Mark Davis scripsit:

> That means that in ISO 639-2 the following are collections:

> mul             Multiple languages

Yes, unfortunately.  639-3 takes a more sensible stand, and calls
'und', 'zxx', and 'mul' special (as distinct from individual language,
macrolanguage, collection, or private-use).

> The language codes clearly do not form a partition, since some
> collections encompass other codes.

True.  The collection codes, however, form a rootless hierarchy,
a rather flat one, of which "mis" is one top-level component.

> The collection codes that are tagged with (Other) are clearly meant
> to be the remainder of partitions, but for the collections that are
> not tagged with (Other) there is no evidence that they were intended
> to exclude other cases -- if anything, the contrary -- if they had
> meant to be the remainder of partitions, they would have said (Other).

I think that's reading too much into them; the entries are the work
of many hands.  FWIU, all the "(Other)" epithets are being removed,
anyhow.

> Moreover, while some may be perfectly willing to have stability go by
> the wayside, it is extremely important to us, and that is one of the
> guiding principles of and reasons for BCP 47. That means that if I
> validly and correctly tag content with "mis", that application cannot
> be made incorrect by any future change to BCP 47. That is why we can
> broaden the application of codes, but cannot narrow them.

+1

> There is no evidence in BCP 47 that I cannot correctly tag the content
> "kind" with "mis".

There is no *evidence* that you cannot tag "George slipped on a banana
peel and fell down" as "fr".  It's a quality of implementation issue.

-- 
Eric Raymond is the Margaret Mead               John Cowan
of the Open Source movement.                    [email-masked]
        --Bruce Perens,                         http://www.ccil.org/~cowan
          some years ago

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 13 Apr 2007 13:44:39 -0700

> of many hands.  FWIU, all the "(Other)" epithets are being removed,

What is in the specification is what we can use to establish the semantics
of the code at introduction into BCP 47, which then determines whether any
successive changes to the Description are allowed (broadenings) or not.

> peel and fell down" as "fr".  It's a quality of implementation issue.

I disagree. Any reasonable person with a knowledge of French and looking
over the text of the standard would say that tagging that sentence with "fr"
is a mistake. The same is not true of 'mis'.

--

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 13 Apr 2007 13:53:24 -0700

Also,

> > a rather flat one, of which "mis" is one top-level component.
You might like this to be true, but I don't see any substantiation of it in
the standard. If you could point me to that, I'd appreciate it.

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 17:06:50 -0400

Mark Davis scripsit:

> You might like this to be true, but I don't see any substantiation of
> it in the standard. If you could point me to that, I'd appreciate it.

It seems rather self-evident to me that 'ber' is a subset of 'afa',
and so on; but no, the standard doesn't say so.  It does, however, say:

	A collective language code is not intended to be used when an
	individual language code or another more specific collective
	language code is available.

http://www.loc.gov/standards/iso639-2/normtext.html section 4.1.1

I take that to mean that "afa" is unsuitable for a Berber language, and
"ger" is unsuitable for English.  A fortiori, "mis" is unsuitable for
a language for which a better code is available.

Questionless, this contradicts the desire for stability, but I don't
see what's to be done about it.  I tried at one point to get all
language collection codes deprecated, but it was pointed out that
there are good reasons for having them, as when insufficient evidence
is available.

-- 
John Cowan   [email-masked]   http://ccil.org/~cowan
I must confess that I have very little notion of what [s. 4 of the British
Trade Marks Act, 1938] is intended to convey, and particularly the sentence
of 253 words, as I make them, which constitutes sub-section 1.  I doubt if
the entire statute book could be successfully searched for a sentence of
equal length which is of more fuliginous obscurity. --MacKinnon LJ, 1940

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Apr 2007 23:06:37 +0200

Randy Presuhn wrote:

> I find it very difficult to imagine a case where "mis" would
> be useful in tagging data.

Mark finds new wild and wonderful ways to (ab)use "mis" while
we're talking.  Based on exactly the same mis-conception I had
until yesterday when John shed some light on it.

There's a "high astonishing factor" in what "mis" really means,
the very minimum is to document it, preventing further havoc.

The only other alternative I see is to deprecate the "mis" tag
either by convincing ISO 639, or with brute force in 4646bis.

> perhaps we could conclude that this whole debate is really 
> academic, and that no action is needed.

Strong NAK.  Mark has also plausible ideas how to (ab)use "art",
maybe that also deserves a comment in the registry.  IMO less
critical because 4646 is very clear that it's really not about
programming languages.

Last but not least Mark found a plausible way to use "und" as
a kind of anchor for "whatever it is, I know the script", as
in und-Latn.  The SHOULD NOT in RFC 4646 is apparently wrong,
und-Latn is an excellent use case.

Frank

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 18:25:59 -0400

Randy Presuhn scripsit:

> The thing that bothers me about the comment "A collection of languages
> which don't belong to any other collection" is that it isn't compatible
> with the possibility that one or more of those languages might
> eventually be included in another (possibly new) collection.  If that
> langauage were left in the mis collection as well, the comment would
> be incorrect.  If the language were removed from the mis collection,
> stability goes out the window.  Either way, changing the comment
> wouldn't help the situation.

That's true, but it's also true of all other collection subtags.  "Mis"
is a particularly bad case, but it's not hard to devise other problematic
situations; I've changed the Subject: line accordingly.

For example, today Slobbovian (hypothetical) might be considered a Slavic
language with a heavy Germanic substrate, but tomorrow Thingummy might
be able to show conclusively that it was really a Germanic language
with a heavy Slavic superstrate.  That would invalidate the old "sla"
tag for the Slobbovian Declaration of Independence in favor of a "ger"
one, but since collections are not and cannot in practice be defined by
enumeration, such things can't be avoided.

I think we must accept that there is some risk of instability when
language collection subtags are used; that's a tradeoff against their
convenience.

Bullet point 3 of section 4.1 of rfc-4646-04 currently says:

	Use specific language subtags or subtag sequences in preference
	to subtags for language collections. A "language collection"
	is a subtag derived from one of the ISO 639-2 codes that
	represents multiple related languages. For example, the code 'cmc'
	represents "Chamic languages". The registry contains values for
	each of the approximately ten individual languages represented
	by this collective code. For example 'jra' (Jarai) and 'cja'
	(Western Cham).

I suggest adding the following additional text:

	Using a collective language code may often be convenient or
	necessary when detailed information is not available.  However,
	collections are defined not by enumerating specific languages,
	but by genetic or other criteria, and so a specific language may
	be moved out of a given collection if further information about
	the language becomes available.  Thus collective language codes
	are inherently more unstable.

For guidance in interpreting these suggestions, I think we now
need to include the 639-3 scope information in language subtags:
"individual language", "macrolanguage", "collective" or "private use".

-- 
John Cowan                              <[email-masked]>
            http://www.ccil.org/~cowan
                .e'osai ko sarji la lojban.
                Please support Lojban!          http://www.lojban.org

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Apr 2007 16:03:20 -0700

It's my impression that, in MARC, the categories were considered to be an exception. At this point in ISO 639, I think the collections cannot be treated as partitions: the additions in 639-3 basically undermine that assumption. However, it might still be possible to treat the *collections* as a partition.

None of that changes the intention of mis in MARC, which I'd have to say is the same intention in ISO 639: mis is the "else" case - i.e. none of the other coded categories fits. The problem with mis is one of maintainability: any time a new entry is added in ISO 639, some documents tagged with mis may suddenly become inappropriately tagged.

Peter

From: Mark Davis [mailto:[email-masked]]
Sent: Friday, April 13, 2007 11:50 AM
To: John Cowan
Cc: Frank Ellermann; LTRU Working Group; [email-masked]
Subject: [Ltru] Re: "mis" update review request

You are right about number one; I concede that point fully -- I'd overlooked that phrasing, sorry. That means that in ISO 639-2 the following are collections:

mul             Multiple languages
art             Artificial (Other)

plus more normal cases:

afa             Afro-Asiatic (Other)
alg             Algonquian languages
...

und             Undetermined

I disagree with your point number two. The only reason that I can conclude that "Miscellaneous Expenses" in a spreadsheet excludes the other listed categories is that I know that everything listed is intended to be a partition. The language codes clearly do not form a partition, since some collections encompass other codes. The collection codes that are tagged with (Other) are clearly meant to be the remainder of partitions, but for the collections that are not tagged with (Other) there is no evidence that they were intended to exclude other cases -- if anything, the contrary -- if they had meant to be the remainder of partitions, they would have said (Other).

Moreover, while some may be perfectly willing to have stability go by the wayside, it is extremely important to us, and that is one of the guiding principles of and reasons for BCP 47. That means that if I validly and correctly tag content with "mis", that application cannot be made incorrect by any future change to BCP 47. That is why we can broaden the application of codes, but cannot narrow them. There is no evidence in BCP 47 that I cannot correctly tag the content "kind" with "mis". Now, of course, we all know that we should tag with as much information as we can, so I *should* tag that with "en" if I know I mean the English word, and "de" if I mean the German, or possibly others. If I don't know which one it is, if my protocol allows multiple tags I can use "en, de", and if not I am forced into a choice between "mul" or choosing the 'most likely' language of the set.

Mark
On 4/13/07, John Cowan <[email-masked]<mailto:[email-masked]>> wrote:

which says (s.v. "mis")

        Miscellaneous languages

--
One art / There is                      John Cowan < [email-masked]<mailto:[email-masked]>>
No less / No more                       http://www.ccil.org/~cowan
All things / To do
With sparks / Galore                     -- Douglas Hofstadter

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Apr 2007 16:06:04 -0700

For the imperfect-knowledge scenario you describe where the process ends with an "I don't know" conclusion, und is the appropriate tag, not mis.

Peter

From: Mark Davis [mailto:[email-masked]]
Sent: Friday, April 13, 2007 1:05 PM
To: Randy Presuhn
Cc: [email-masked]; [email-masked]
Subject: Re: [Ltru] Re: "mis" update review request

I think part of the problem in communicating about this is that people may have somewhat different usage scenarios in mind. If you think of tagging as something that a person does with content that they originate or have control over, then it is (probably) fairly straightforward for that person to tag as specifically as possible.

Another scenario is where you have incoming content, and you need to tag it for use by other components. This might be done, for example, in a search engine, where you fetch and process a page, and use that information later in doing searches. The tag serves to communicate language between the different components.

In that case, you have far from perfect information about the content: what you have being typically the result of some level statistical analysis, plus other factors about the document. You need to tag with as much information as you have, *but no more*. It is in that case where you need to have the tags that indicate some level of imperfect knowledge about the source, such as "I have no idea what this is", or "It looks like linguistic content, but I don't know which language", or "it doesn't look like linguistic content". (You may also have more detailed knowledge, like that some document appears to have 70% English content (probability 95%) and 20% French content (probability 65%)).

Both scenarios are equally valid use cases for BCP 47 (in fact, as a percentage of data flow on the web, I'd wager strongly that the second scenario completely swamps the first).

Mark
On 4/13/07, Randy Presuhn <[email-masked]<mailto:[email-masked]>> wrote:
Hi -

Randy

_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Apr 2007 16:17:42 -0700

I think ISO 639-2 is clear that the most specific category should be used. That principle is implicit in the "(Other)" collections. I also think that principle in combination with collections creates maintenance problems. (Which is why I suggested that all the "(Other)" entries should just be "languages" entries.)

(Btw, I think I suggested some time ago it might not be a bad thing to deprecate use of collection IDs in IETF language tags.)

Peter

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Apr 2007 16:21:13 -0700

From: Frank Ellermann [mailto:[email-masked]]

> either by convincing ISO 639, or with brute force in 4646bis.

I don't think it would be appropriate for ISO 639 to deprecate mis. It is assumed that a large set of coded categories are provided and that particular applications may wish to use a "profile" -- i.e. some subset of the inventory. In the context of a particular application context using some particular profile, mis may be appropriate as the "else" category. But it would be up to that application to determine appropriate usage, including how to deal with maintenance issues.

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 13 Apr 2007 17:20:13 -0700

That, I think, we are all in agreement on. And that follows what we do in
BCP 47, which is that we say that people *should* tag as specifically as
possible. So if I know that content is "en-US", I *should* say "en-US" and
not just "en". But I *can* also use "en". It might not be the best choice,
but it is a legitimate usage (although not optimal) usage. However, it is a
perfectly reasonable choice if I don't know whether it is "en-US" or
"en-CA", or it could be both.

So what about "mis"? Once again, I *should* tag more specifically, if I have
the information. No argument at all there. The question is whether it is
non-conformant to BCP 47 to tag "kind" as "mis". For that, we need to
establish whether there is sufficient grounds in the text and data of ISO
639-2 as of the time that "mis" was taken into BCP 47 to conclusively
determine that "mis" is disjoint from other language codes. I don't see a
conclusive case from what you and John have said so far, unless I'm missing
something.

I would not at all be adverse to saying that you shouldn't use "und" or
"mis" or "mul" or any collections if you have any more specific information
about the content. And I think it is clear that we need much more guidance
in BCP 47 as to intended usage.

Mark

On 4/13/07, Peter Constable <[email-masked]> wrote:
>
> I think ISO 639-2 is clear that the most specific category should be used.
> That principle is implicit in the "(Other)" collections. I also think that
> principle in combination with collections creates maintenance problems.
> (Which is why I suggested that all the "(Other)" entries should just be
> "languages" entries.)
>
> (Btw, I think I suggested some time ago it might not be a bad thing to
> deprecate use of collection IDs in IETF language tags.)
>
>
> Peter
>
>

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 13 Apr 2007 17:33:01 -0700

What is truly unfortunate is that ISO defined these (Other) items by
exclusion. If they had defined the tag "ger" to simply meant Germanic, it
would remain valid indefinitely. They would then behave like macrolanguages.

In LTRU what we need to decide is whether we bail on stability, or we impose
stability. We can do the latter by deciding, like we do with other unstable
ISO codes, that the meaning of the tag "ger" is set at the time we add it to
the registry, and can only be narrowed, not broadened. Then if ISO decides
to add a new code for a Germanic language "gxx", it then simply overlaps
with "ger". It is then a strong SHOULD to use "gxx" in preference to "ger"
where it applies, but it doesn't invalidate previous usage.

Mark

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 21:11:25 -0400

Mark Davis scripsit:

> What is truly unfortunate is that ISO defined these (Other) items
> by exclusion. If they had defined the tag "ger" to simply meant
> Germanic, it would remain valid indefinitely. They would then behave
> like macrolanguages.

Well, unless a language were to be reclassified.  Very unlikely for
Germanic, but not so unlikely for some of the others.

In any case, the "(Other)" is, IIRC, in the process of being removed.

> In LTRU what we need to decide is whether we bail on stability, or
> we impose stability. We can do the latter by deciding, like we do
> with other unstable ISO codes, that the meaning of the tag "ger" is
> set at the time we add it to the registry, and can only be narrowed,
> not broadened.

The trouble is that we don't really know the extensional definition
of these collective language tags:  they are defined purely intensionally.
So they change when the literature says they change.

> Then if ISO decides to add a new code for a Germanic language "gxx",
> it then simply overlaps with "ger". It is then a strong SHOULD to use
> "gxx" in preference to "ger" where it applies, but it doesn't invalidate
> previous usage.

Again, what happens if a language is found *not* to belong to some
collection in which it had been included until now?  How do we even know
about such an event?

-- 
John Cowan                                [email-masked]
I amar prestar aen, han mathon ne nen,    http://www.ccil.org/~cowan
han mathon ne chae, a han noston ne 'wilith.  --Galadriel, LOTR:FOTR

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Apr 2007 18:22:03 -0700

(Is "kind" meant to be a word in English, a word in some other language, or a reference to something else? I assume the first.)

I don't think we can say it is *non-conformant* to tag "hello" as mis any more than we can say it is non-conformant to tag "hello" as fr. It's just bad tagging. That's comparable to a spelling checker correcting "helo" as "holp" - it may not be useful, that that doesn't make it non-conformant to Unicode. Non-conformance to BCP 47 would be to see fr and apply an English spelling checker.

But that's a different question from how (or if) people should use mis. IMO we should say that implementers of BCP 47 SHOULD NOT use mis except if they have an immediate need to apply a language subtag and have determined that there is no available language subtag encompassing the language of the given content. E.g. I know it's Martian and there's no ISO 639 ID for Martian. (Hypothetical example assumes Martians exist and that this is not encompassed within art.) Of course, in many cases people/processes may not know enough about every language to be able to rule out all the available possibilities, but in that case the appropriate thing to use would be und.

Peter

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Friday, April 13, 2007 5:20 PM
To: Peter Constable
Cc: LTRU Working Group; [email-masked]
Subject: Re: [Ltru] Re: "mis" update review request

That, I think, we are all in agreement on. And that follows what we do in BCP 47, which is that we say that people *should* tag as specifically as possible. So if I know that content is "en-US", I *should* say "en-US" and not just "en". But I *can* also use "en". It might not be the best choice, but it is a legitimate usage (although not optimal) usage. However, it is a perfectly reasonable choice if I don't know whether it is "en-US" or "en-CA", or it could be both.

So what about "mis"? Once again, I *should* tag more specifically, if I have the information. No argument at all there. The question is whether it is non-conformant to BCP 47 to tag "kind" as "mis". For that, we need to establish whether there is sufficient grounds in the text and data of ISO 639-2 as of the time that "mis" was taken into BCP 47 to conclusively determine that "mis" is disjoint from other language codes. I don't see a conclusive case from what you and John have said so far, unless I'm missing something.

I would not at all be adverse to saying that you shouldn't use "und" or "mis" or "mul" or any collections if you have any more specific information about the content. And I think it is clear that we need much more guidance in BCP 47 as to intended usage.

Mark
On 4/13/07, Peter Constable <[email-masked]<mailto:[email-masked]>> wrote:
I think ISO 639-2 is clear that the most specific category should be used. That principle is implicit in the "(Other)" collections. I also think that principle in combination with collections creates maintenance problems. (Which is why I suggested that all the "(Other)" entries should just be "languages" entries.)

Peter

-----Original Message-----
From: John Cowan [mailto: [email-masked]<mailto:[email-masked]>]
Sent: Friday, April 13, 2007 2:07 PM
To: Mark Davis
Cc: LTRU Working Group; [email-masked]<mailto:[email-masked]>
Subject: [Ltru] Re: "mis" update review request

--
John Cowan   [email-masked]<mailto:[email-masked]>   http://ccil.org/~cowan

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Apr 2007 18:25:30 -0700

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis

> What is truly unfortunate is that ISO defined these (Other) items by exclusion.

I am pushing for these to be generalized with the exclusions removed - all cases of "Foo (Other)" changed to "Foo languages".

> We can do the latter by deciding, like we do with other unstable ISO codes,
> that the meaning of the tag "ger" is set at the time we add it to the registry,
> and can only be narrowed, not broadened.

I think you mean, "...can only be broadened, not narrowed.

Peter

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Apr 2007 18:29:51 -0700

> The trouble is that we don't really know the extensional
> definition of these collective language tags:  they are
> defined purely intensionally.

At one point early in our involvement with ISO 639, Gary Simons and I suggested that the collections be given extensional definitions, as has been done with macrolanguages (in the macrolanguage mapping table). But as our interest was in coverage for individual languages and no in the collections, we didn't take any interest in pursuing either evaluating the merits or getting that implemented.

Peter

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 21:38:23 -0400

Peter Constable scripsit:

> (Is "kind" meant to be a word in English, a word in some other language,
> or a reference to something else? I assume the first.)

I think Mark's point was that "kind" is ambiguously English or Dutch
(or German, if you discount the lower-case "k").

The normative text of ISO 639-2 says (section 4.1.2):

	The language code mul (for multiple languages) should be applied
	when several languages are used and it is not practical to
	specify all the appropriate language codes.

I think that is meant to cover mixed-language documents, not documents
whose language is ambiguous, but I don't want to start yet another hooraw.

> But that's a different question from how (or if) people should use
> mis. IMO we should say that implementers of BCP 47 SHOULD NOT use mis
> except if they have an immediate need to apply a language subtag and
> have determined that there is no available language subtag encompassing
> the language of the given content. E.g. I know it's Martian and there's
> no ISO 639 ID for Martian. (Hypothetical example assumes Martians
> exist and that this is not encompassed within art.)

Or, until 639-3 codes are available in BCP 47, any language that doesn't
have its own 639-2 code or plausible covering collection.

-- 
John Cowan   http://ccil.org/~cowan    [email-masked]
In might the Feanorians / that swore the unforgotten oath
brought war into Arvernien / with burning and with broken troth.
and Elwing from her fastness dim / then cast her in the waters wide,
but like a mew was swiftly borne, / uplifted o'er the roaring tide.
        --the Earendillinwe

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 22:06:56 -0400

Peter Constable scripsit:

> At one point early in our involvement with ISO 639, Gary Simons and
> I suggested that the collections be given extensional definitions,
> as has been done with macrolanguages (in the macrolanguage mapping
> table). But as our interest was in coverage for individual languages
> and no in the collections, we didn't take any interest in pursuing
> either evaluating the merits or getting that implemented.

Your obsolete-but-still-interesting cut at that is still online at
www.ethnologue.com/14/iso639 .

Reminder: in addition to replacing "(Other)" with "languages", 'bih' and
'him' should be updated to add "languages".

-- 
Cash registers don't really add and subtract;           John Cowan
        they only grind their gears.                    [email-masked]
But then they don't really grind their gears, either;   
        they only obey the laws of physics.  --Unknown

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 22:12:30 -0400

Mark Davis scripsit:

> So what about "mis"? Once again, I *should* tag more specifically,
> if I have the information. No argument at all there. The question is
> whether it is non-conformant to BCP 47 to tag "kind" as "mis". For that,
> we need to establish whether there is sufficient grounds in the text
> and data of ISO 639-2 as of the time that "mis" was taken into BCP 47 to
> conclusively determine that "mis" is disjoint from other language codes.

In order to determine the semantics of 639-2, we must look to the MARC
list and the practice of the Library of Congress (just as to determine
the semantics of 639-3, we must look to the Ethnologue).  Peter gave
us the list of languages and language groups which MARC tags using
'mis':  see http://www.loc.gov/marc/languages/lang_l2n.html .
There may not be 100% agreement between this list and 639-2, but it's
the best we have right now, especially when supplemented with
http://www.ethnologue.com/14/show_iso639.asp?code=mis .

-- 
In my last lifetime,                            John Cowan
I believed in reincarnation;                    http://www.ccil.org/~cowan
in this lifetime,                               [email-masked]
I don't.  --Thiagi

---

From: Peter Constable <[email-masked]>
Date: Fri, 13 Apr 2007 19:40:31 -0700

> I think Mark's point was that "kind" is ambiguously English or Dutch...

In a case like "kind" where there's ambiguity and you don't know the specific language, the appropriate tag would be und.

> whose language is ambiguous

Correct.

> have its own 639-2 code or plausible covering collection.

Sure; the determination can be made based on what's available at the time the determination is being made. That presents a stability issue, but I don't see how that can be avoided if you want to use mis. The other option is to simply use und on the assumption that the conclusion being made is "I don't know how I can possibly tag it using the available tags". That's stable, though it is making a different statement.

Peter

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 13 Apr 2007 21:21:23 -0700

Hi -

> From: "Mark Davis" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>
> Cc: <[email-masked]>; <[email-masked]>
> Sent: Friday, April 13, 2007 1:04 PM
... 
...

I agree that groupings are useful, and have argued for them in the past.
I'm not going to argue that we should exclude "mis", but I doubt that
it would be terribly useful.

I find it very hard to believe that a reasonable analysis (whether done
by human or machine) would classify a text a being "mis" without being
able to recognize which of the languages in that grouping the text belonged
to.  I can believe someone could look at text and say "it's a slavic language,
but I'm not sure which one."  Do we really think someone or something would
look at some text and say "it's Ainu, Andamanese, or Etruscan, but I can't
tell which, so I'll tag it 'mis'"?

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 13 Apr 2007 21:45:33 -0700

Hi -

As a technical contributor...

> To: "Peter Constable" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>; <[email-masked]>

> "en-CA", or it could be both.

What provision in BCP 47 recommends that text should be tagged
"as specifically as possible"?  That seems quite a stretch from
the spirit of "tag wisely", and to be contrary to the section 4.1's
statement that "A subtag SHOULD only be used when it adds useful
distinguishing information to the tag."

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 13 Apr 2007 21:21:23 -0700

Hi -

... 
...

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 13 Apr 2007 21:45:33 -0700

Hi -

Randy

---

From: John Cowan <[email-masked]>
Date: Sat, 14 Apr 2007 01:20:19 -0400

> distinguishing information to the tag."

We are talking about language subtags specifically, not other kinds;
see section 4.1, bullet point 3, of the current 4646bis draft,
which prefers language subtags to language-collection subtags,
and more specific collections to less specific ones.

-- 
John Cowan
        [email-masked]
                I am a member of a civilization. --David Brin

---

From: Peter Constable <[email-masked]>
Date: Sat, 14 Apr 2007 00:13:55 -0700

I have not forgotten bih and him; they are among the last handful of issues I intend to present to the JAC for some resolution to complete the integration of the Ethnologue 15 inventory with the existing inventory of ISO 639-2. I am simply waiting for them to wrap up decision regarding Aramaic varieties, and then Tibetan before I present the short list of remaining issues, all have to do with South Asian varieties.

Peter

---

From: Peter Constable <[email-masked]>
Date: Sat, 14 Apr 2007 00:20:44 -0700

From: Randy Presuhn [mailto:[email-masked]]

> I find it very hard to believe that a reasonable analysis
> (whether done by human or machine) would classify a text a
> being "mis" without being able to recognize which of the
> languages in that grouping the text belonged to.  I can
> believe someone could look at text and say "it's a slavic
> language, but I'm not sure which one."  Do we really think
> someone or something would look at some text and say "it's
> Ainu, Andamanese, or Etruscan, but I can't tell which, so
> I'll tag it 'mis'"?

If someone were so tempted, I would argue that would be inappropriate use of mis. Since they do not know what it is, their declaration is that the language identity is not determined, and the appropriate tag for that is und. Appropriate use of mis does not require that one know the language of the content; it does, however, require that one know it is *not* a language covered by any of the available tags.

Peter

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 14 Apr 2007 13:28:05 +0200

> distinguishing information to the tag."

Yes, I was also puzzled by Mark's statement, but missed what you
have quoted when I tried to check it.  I found the SHOULD NOT for
script subtags, confirming the general line of "tag wisely".

IMO that's both:   not too unspecific, but also not too specific.
And above all don't lie.

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 16 Apr 2007 15:49:16 -0700

1. I think we have to be very careful here. The meaning of a standard like
ISO 639-2 is established not by what we wish it would have said, nor by what
we would find out if we were able to read Peter's mind. It is established by
the wording in the standard, and how reasonable people could interpret it.
The fact that "mis" was incorporated in order to account for MARC codes is
interesting, but is not in the text of the standard. We can't expect users
of BCP 47 to all be able to read Peter's mind before tagging.

2. When we are looking at stability, that is very important: our goal is
that once content is correctly tagged, people can depend on the fact that we
will not change the meaning of a tag out from under them. So clarifications
that we add in future versions of 4646 or the registry are fine, as long as
they do not narrow the range of reasonable interpretations. We can broaden
them. So in the case of "mis", a proposed narrowing to include just the MARC
codes is clearly disallowed, since it was nowhere stated in ISO 639-2 at the
time that "mis" was added to the language registry (the BCP 47 semantics are
established at the time we add the code). That is one of the key principles
of BCP 47, is to isolate us where necessary from instabilities in the source
standards.

(The one exception we might be able to make is where something is so badly
defined that most reasonable people couldn't come up with any consistent
definition for it.)

3. Now, I think there are steps that can be taken to make the above moot. I
think Peter's suggestion for ISO 639-X of broadening all of the Collections
to remove the (Other) is exactly the right strategy, and if this can be done
before 4646bis is issued, all the better. So having

   - aus    Australian languages means any of the languages on
   http://www.ethnologue.com/show_family.asp?subid=90498
   - bat    Baltic (Other) => Baltic languages, means any of the
   languages on http://www.ethnologue.com/show_family.asp?subid=90207
   - mis    Miscellaneous languages, essentially the root for
   http://www.ethnologue.com/family_index.asp

and so on. This is useful on a number of levels; it resolves a number of
problems in the interpretation of language codes, and makes the source
standards themselves more stable. (In the ideal case, we would have codes
for each of the possible "decision points" in the language tree. That is, if
we look at any language code such as
http://www.ethnologue.com/show_lang_family.asp?code=eng we'd have codes for
each of the parent groupings, not just some of them, like "Australian
languages".)

3. Randy raised the issue as to whether "mis" in the broad sense is useful
(as something that has linguistic content, but I don't know what it is). It
very much follows the model in #3. There are times when detection can only
determine that it looks like there is some linguistic content -- it is not
just binary data -- but current detection can't really determine what it
might be. That is, a code that means "according to our best available
detection methods this doesn't look like it is zxx".

4. I'm leery of using zxx for programming languages, instead of just binary.
There is clearly some linguistic content in "if (content == null) { /*
remove the item in the lookup table */ ...}". Maybe we need another code for
this, something different than either 'art' or 'zxx'.

Mark

On 4/14/07, Peter Constable <[email-masked]> wrote:
> If someone were so tempted, I would argue that would be inappropriate use
> of mis. Since they do not know what it is, their declaration is that the
> language identity is not determined, and the appropriate tag for that is
> und. Appropriate use of mis does not require that one know the language of
> the content; it does, however, require that one know it is *not* a language
> covered by any of the available tags.
>
>
>
> Peter
>

---

From: Peter Constable <[email-masked]>
Date: Mon, 16 Apr 2007 17:18:30 -0700

Re 1: Yes, be careful: (a) the majority of existing legacy usage of mis is bound to be in MARC, and (b) any existing usage would assume the context of ISO 639-2 (i.e. mis in existing usage is the exception list for ISO 639-2).

Re 2: The mis collection is inherently unstable - unavoidably so. Prior to 2005-08-16, an implementation of ISO 639-2 would have tagged Ainu content as mis; after that date, an implementation of ISO 639-2 would have tagged Ainu content as ain; existing content tagged before that date would not get retrieved by request for ain, and it would be conformant to suppose that requests for mis would not return Ainu content. The mis collection is ugly, pure and simple. So, I don't see what the point is of getting worried over whether we're making mis unstable: it's been that way for some time.

(Note: mis is badly defined from a stability perspective, though I don't think there's much question of how it's defined.)

Re 3(b): "There are times when detection can only determine that it looks like there is some linguistic content -- it is not just binary data -- but current detection can't really determine what it might be. That is, a code that means "according to our best available detection methods this doesn't look like it is zxx"." If you want to use mis for that, I would argue that that is significantly changing the semantics of mis. (Even though mis is unstable, it is unstable on a qualitative level; this is a categorical change.) I definitely oppose that. If you want an ID for "undetermined human language", then that should be proposed. We should not usurp an existing ID for that purpose.

Re 4: I don't see how your example differs from this: "Nous avons une phrase en français (but this is in English)". The fact that the parenthetical text is in English doesn't change the fact that the other text is in French. Similarly, in your example, the fact that there is a comment in English does not change the fact that the rest of the text is not in a human language. Do we create tags for "French with embedded bits of English"?

Peter

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Monday, April 16, 2007 3:49 PM
To: Peter Constable
Cc: [email-masked]; [email-masked]
Subject: Re: [Ltru] Re: "mis" update review request

1. I think we have to be very careful here. The meaning of a standard like ISO 639-2 is established not by what we wish it would have said, nor by what we would find out if we were able to read Peter's mind. It is established by the wording in the standard, and how reasonable people could interpret it. The fact that "mis" was incorporated in order to account for MARC codes is interesting, but is not in the text of the standard. We can't expect users of BCP 47 to all be able to read Peter's mind before tagging.

2. When we are looking at stability, that is very important: our goal is that once content is correctly tagged, people can depend on the fact that we will not change the meaning of a tag out from under them. So clarifications that we add in future versions of 4646 or the registry are fine, as long as they do not narrow the range of reasonable interpretations. We can broaden them. So in the case of "mis", a proposed narrowing to include just the MARC codes is clearly disallowed, since it was nowhere stated in ISO 639-2 at the time that "mis" was added to the language registry (the BCP 47 semantics are established at the time we add the code). That is one of the key principles of BCP 47, is to isolate us where necessary from instabilities in the source standards.

(The one exception we might be able to make is where something is so badly defined that most reasonable people couldn't come up with any consistent definition for it.)

3. Now, I think there are steps that can be taken to make the above moot. I think Peter's suggestion for ISO 639-X of broadening all of the Collections to remove the (Other) is exactly the right strategy, and if this can be done before 4646bis is issued, all the better. So having

 *   aus    Australian languages means any of the languages on http://www.ethnologue.com/show_family.asp?subid=90498
 *   bat    Baltic (Other) => Baltic languages, means any of the languages on http://www.ethnologue.com/show_family.asp?subid=90207
 *   mis    Miscellaneous languages, essentially the root for http://www.ethnologue.com/family_index.asp
and so on. This is useful on a number of levels; it resolves a number of problems in the interpretation of language codes, and makes the source standards themselves more stable. (In the ideal case, we would have codes for each of the possible "decision points" in the language tree. That is, if we look at any language code such as http://www.ethnologue.com/show_lang_family.asp?code=eng we'd have codes for each of the parent groupings, not just some of them, like "Australian languages".)

3. Randy raised the issue as to whether "mis" in the broad sense is useful (as something that has linguistic content, but I don't know what it is). It very much follows the model in #3. There are times when detection can only determine that it looks like there is some linguistic content -- it is not just binary data -- but current detection can't really determine what it might be. That is, a code that means "according to our best available detection methods this doesn't look like it is zxx".

4. I'm leery of using zxx for programming languages, instead of just binary. There is clearly some linguistic content in "if (content == null) { /* remove the item in the lookup table */ ...}". Maybe we need another code for this, something different than either 'art' or 'zxx'.

Mark
On 4/14/07, Peter Constable <[email-masked]<mailto:[email-masked]>> wrote:
From: Randy Presuhn [mailto:[email-masked]<mailto:[email-masked]>]

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 16 Apr 2007 18:51:50 -0700

On 4/16/07, Peter Constable <[email-masked]> wrote:
>
>  Re 1: Yes, be careful: (a) the majority of existing legacy usage of mis
> is bound to be in MARC, and (b) any existing usage would assume the context
> of ISO 639-2 (i.e. mis in existing usage is the exception list for ISO
> 639-2).
>
>
>
> Re 2: The mis collection is inherently unstable – unavoidably so. Prior to
> 2005-08-16, an implementation of ISO 639-2 would have tagged Ainu content as
> mis; after that date, an implementation of ISO 639-2 would have tagged Ainu
> content as ain; existing content tagged before that date would not get
> retrieved by request for ain, and it would be conformant to suppose that
> requests for mis would not return Ainu content. The mis collection is ugly,
> pure and simple. So, I don't see what the point is of getting worried over
> whether we're making mis unstable: it's been that way for some time.
>

What I'm saying is that

   1. Right now in ISO 639-2, we have a number of collections defined by
   exclusion, where XXX (Other) means any XXX that is not already defined. Thus
   "bat" means "Any Baltic language that doesn't already have a code".
   2. Those collections are inherently unstable in ISO 639-2, since they
   contract each time a new XXX language is added.
   3. The way to make an collection code XXX not unstable is to make it
   not be not defined as an exclusion: removing the (Other). [your proposal]
   4. Then XXX is stable into the future, since adding a new language of
   the type XXX doesn't affect it.
   5. Thus if we change "bat" from Baltic (Other)" into "Baltic", meaning
   any of the Baltic languages, it becomes stable.
   6. Such a change, being a broadening, can be carried into BCP 47.
   7. We can apply the same methodology to "mis". That would change it
   from the fairly useless -- and unstable -- "Any Language not otherwise
   encoded", into "Any language".
   8. It then becomes stable, and useful.

(Note: mis is badly defined from a stability perspective, though I don't
> think there's much question of how it's defined.)
>

I agree that that is not the current definition of "mis" (see below).

Re 3(b): "There are times when detection can only determine that it looks
> like there is some linguistic content -- it is not just binary data -- but
> current detection can't really determine what it might be. That is, a code
> that means "according to our best available detection methods this doesn't
> look like it is zxx"." If you want to use mis for that, I would argue that
> that is significantly changing the semantics of mis. (Even though mis is
> unstable, it is unstable on a qualitative level; this is a categorical
> change.) I definitely oppose that. If you want an ID for "undetermined human
> language", then that should be proposed. We should not usurp an existing ID
> for that purpose.
>

It is a significant broadening of the semantics. And I'm not fixed on that.
It just seems that doing that broadening is congruent with the removal of
"(Other)" that you've proposed in other cases, and transforms a useless and
dangerous (for stability) code into a useful code. And since it is a
broadening, it is consistent with BCP 47.

However, if that is too big a step to stomach, the alternative is to
strongly recommend that people never use "mis", and propose a new code for
ISO 639-2 that has the meaning of "Any language".

Re 4: I don't see how your example differs from this: "Nous avons une phrase
> en français (but this is in English)". The fact that the parenthetical text
> is in English doesn't change the fact that the other text is in French.
> Similarly, in your example, the fact that there is a comment in English does
> not change the fact that the rest of the text is not in a human language. Do
> we create tags for "French with embedded bits of English"?
>

You have a good point. Again, I'm not hard and fast about this issue, but I
think there is definitely a significant distinction in usage between "this
is a chunk of stuff that looks like random binary data, like a JPEG", and
"this is stuff that looks like it might be written a programming language.",
a distinction that I think would be useful to provide for in BCP 47. On the
detection front, it is much easier to determine "this is random binary",
while not necessarily very easy to determine "this is a programming language
fragment".

Peter
>
>
>
> *From:* [email-masked] [mailto:[email-masked]] *On
> Behalf Of *Mark Davis
> *Sent:* Monday, April 16, 2007 3:49 PM
> *To:* Peter Constable
> *Cc:* [email-masked]; [email-masked]
> *Subject:* Re: [Ltru] Re: "mis" update review request
>
>
>
> ISO 639-2 is established not by *what we wish it would have said, *nor by
> *what we would find out if we were able to read Peter's mind.* It is
> established by the wording in the standard, and how reasonable people could
> interpret it. The fact that "mis" was incorporated in order to account for
> MARC codes is interesting, but is not in the text of the standard. We can't
> expect users of BCP 47 to all be able to read Peter's mind before tagging.
>
> 3. Now, I think there are steps that can be taken to make the above moot.
> I think Peter's suggestion for ISO 639-X of broadening all of the
> Collections to remove the (Other) is exactly the right strategy, and if this
> can be done before 4646bis is issued, all the better. So having
>
> http://www.ethnologue.com/show_lang_family.asp?code=eng we'd have codes
> for each of the parent groupings, not just some of them, like "Australian
> languages".)
>
> 4. I'm leery of using zxx for programming languages, instead of just
> binary. There is clearly some linguistic content in "if (content == null) {
> /* remove the item in the lookup table */ ...}". Maybe we need another code
> for this, something different than either 'art' or 'zxx'.
>
> Mark
>
> On 4/14/07, *Peter Constable* <[email-masked]> wrote:
>

---

From: Peter Constable <[email-masked]>
Date: Mon, 16 Apr 2007 19:47:29 -0700

I get - and agree with - what you're saying about changing things like bat from "Baltic (Other)" to "Baltic Languages". But to change mis from its current intensional meaning to "any language" is not just broadening - it's to change it into something completely different altogether. It could appropriately be named "Other Languages" - and getting rid of "other" changes it utterly. It is one case where getting rid of the "other" aspect is not appropriate because that is *the* defining criterion of the concept it represents.

I would have little difficulty with the idea that we tell people not to use mis. (Naturally -- again, at some point I suggested we not even allow collections of any kind in IETF tags.) We could deprecate it in the registry, or put a clear SHOULD NOT (if we think just maybe there are some application scenarios in which informed and consenting adults might have a reasonable use for it).

I'd have no problem with a request to the ISO 639 JAC for a special-case ID for "any language" or "some unspecified language". (We'd need to clarify for them the difference between this and und - which may or may not actually be a human language.)

For programming languages, I think a zxx variant would be a reasonable solution.

Peter

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Monday, April 16, 2007 6:52 PM
To: Peter Constable
Cc: [email-masked]; [email-masked]
Subject: Re: [Ltru] Re: "mis" update review request

On 4/16/07, Peter Constable <[email-masked]<mailto:[email-masked]>> wrote:

What I'm saying is that

 1.  Right now in ISO 639-2, we have a number of collections defined by exclusion, where XXX (Other) means any XXX that is not already defined. Thus "bat" means "Any Baltic language that doesn't already have a code".
 2.  Those collections are inherently unstable in ISO 639-2, since they contract each time a new XXX language is added.
 3.  The way to make an collection code XXX not unstable is to make it not be not defined as an exclusion: removing the (Other). [your proposal]
 4.  Then XXX is stable into the future, since adding a new language of the type XXX doesn't affect it.
 5.  Thus if we change "bat" from Baltic (Other)" into "Baltic", meaning any of the Baltic languages, it becomes stable.
 6.  Such a change, being a broadening, can be carried into BCP 47.
 7.  We can apply the same methodology to "mis". That would change it from the fairly useless -- and unstable -- "Any Language not otherwise encoded", into "Any language".
 8.  It then becomes stable, and useful.

It is a significant broadening of the semantics. And I'm not fixed on that. It just seems that doing that broadening is congruent with the removal of "(Other)" that you've proposed in other cases, and transforms a useless and dangerous (for stability) code into a useful code. And since it is a broadening, it is consistent with BCP 47.

However, if that is too big a step to stomach, the alternative is to strongly recommend that people never use "mis", and propose a new code for ISO 639-2 that has the meaning of "Any language".

You have a good point. Again, I'm not hard and fast about this issue, but I think there is definitely a significant distinction in usage between "this is a chunk of stuff that looks like random binary data, like a JPEG", and "this is stuff that looks like it might be written a programming language.", a distinction that I think would be useful to provide for in BCP 47. On the detection front, it is much easier to determine "this is random binary", while not necessarily very easy to determine "this is a programming language fragment".

Peter

From: [email-masked]<mailto:[email-masked]> [mailto: [email-masked]<mailto:[email-masked]>] On Behalf Of Mark Davis
Sent: Monday, April 16, 2007 3:49 PM
To: Peter Constable
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Ltru] Re: "mis" update review request

Mark

Peter

--
Mark

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 16 Apr 2007 20:42:14 -0700

Hi -

> From: "Peter Constable" <[email-masked]>
> To: <[email-masked]>; <[email-masked]>
> Sent: Monday, April 16, 2007 7:47 PM
> Subject: RE: [Ltru] Re: "mis" update review request
...
...

Recall that long ago, the consensus of the ltru WG (issue #955) was to
keep programming languages out of scope.  Are you proposing that
this should be revisited?

Randy
ltru co-chair

---

From: John Cowan <[email-masked]>
Date: Tue, 17 Apr 2007 01:16:38 -0400

>   any of the Baltic languages, it becomes stable.

As I've pointed out repeatedly, this simply isn't true.  If scholars decide that
language X is not a Baltic language after all, then the denotation of "bat"
changes whether we like it or not.

Granted, this isn't likely to happen with Baltic, but a lot of the other
language families aren't so well understood.  Short of defining each
collection by extension, which no one has undertaken to do (the
Ethnologue 14th edition is halfway there), no blanket stability
guarantees can be offered.

> ISO 639-2 that has the meaning of "Any language".

That suits me much better.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
The whole of Gaul is quartered into three halves.
        -- Julius Caesar

---

From: Peter Constable <[email-masked]>
Date: Mon, 16 Apr 2007 22:20:33 -0700

Sorry; if that's a decision that was already made, then I don't mean to re-open it. I'll stick with "It's out of scope for ISO 639, and zxx would apply."

Peter

-----Original Message-----
From: Randy Presuhn [mailto:[email-masked]]
Sent: Monday, April 16, 2007 8:42 PM
To: [email-masked]
Subject: [Ltru] zxx variants for programming languages?

Hi -

...
...

Randy
ltru co-chair

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 16 Apr 2007 22:23:05 -0700

I think there are two somewhat different questions.

1. Should we have tags for individual programming languages (I think not).
2. How should we best tag content that happens to be in a programming
language (und?, art?, zxx?, other?). It is this latter question we've been
discussing.

Mark

On 4/16/07, Randy Presuhn <[email-masked]> wrote:
> > To: <[email-masked]>; < [email-masked]>

---

From: Peter Constable <[email-masked]>
Date: Mon, 16 Apr 2007 22:59:18 -0700

If the answer to the first question is No, then there's no real issue wrt the second question: it's none of the things we tag, hence zxx.

Peter

From: Mark Davis [mailto:[email-masked]]
Sent: Monday, April 16, 2007 10:23 PM
To: Randy Presuhn
Cc: [email-masked]
Subject: Re: [Ltru] zxx variants for programming languages?

1. Should we have tags for individual programming languages (I think not).
2. How should we best tag content that happens to be in a programming language (und?, art?, zxx?, other?). It is this latter question we've been discussing.

Mark
On 4/16/07, Randy Presuhn < [email-masked]<mailto:[email-masked]>> wrote:
Hi -

> From: "Peter Constable" <[email-masked]<mailto:[email-masked]>>
> To: < [email-masked]<mailto:[email-masked]>>; < [email-masked]<mailto:[email-masked]>>
...
...

Randy
ltru co-chair

---

From: Marion Gunn <[email-masked]>
Date: Tue, 17 Apr 2007 18:09:26 +0000

Me, too. We do. And after that, to  make an ISO standard for  
cataloguing computer languages - which couldn't take long to do at  
all, compared to the long-drawn-out efforts to catalogue human  
languages.

mg

On 16 Apr 2007, at 22:49, scríobh Mark Davis:

> ...
>
> 4. I'm leery of using zxx for programming languages, instead of  
> just binary. There is clearly some linguistic content in "if  
> (content == null) { /* remove the item in the lookup table  
> */ ...}". Maybe we need another code for this, something different  
> than either 'art' or 'zxx'.

- -
Marion Gunn * EGTeo (Estab.1991)
27 Páirc an Fhéithlinn, Baile an
Bhóthair, Co. Átha Cliath, Éire.
* [email-masked] * [email-masked] *

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 17 Apr 2007 21:09:59 +0200

on 1:
 
I don't see why 'mis' would have to be an exception when doing a semantic change of removing (implicit or explicit) "other" for
various language codes. Doing so is equally much a semantic change for 'tai' (or any other "other" collection), and of exactly the
same kind, so if it is not ok for 'mis' it would not be ok for 'tai' either. (If you prefer another acronym, say 'any' instead of
'mis', that is another ball-game.)
 
Furthermore, since 'mul' is the only code intended for multiple languages (when it is not practical to list which languages, per
fragment of the document preferably), all of the "languages" codes should instead refer to "language" in singular. This would not be
a semantic change, just referring to each of the items that may be tagged, not a set of items [book shelf...] so tagged.
 
on 4:
 
Programming languages of various sorts are out of scope (like 'zxx', but unlike 'art'), but I may agree that they are out of scope
in a different way than 'zxx'. Perhaps "formal language" ('for'), with no further subdivision (they are still out of scope).
 
        /kent k

  _____  

From: Peter Constable [mailto:[email-masked]] 
Sent: Tuesday, April 17, 2007 2:19 AM
To: [email-masked]; [email-masked]
Subject: RE: [Ltru] Re: "mis" update review request

Re 1: Yes, be careful: (a) the majority of existing legacy usage of mis is bound to be in MARC, and (b) any existing usage would
assume the context of ISO 639-2 (i.e. mis in existing usage is the exception list for ISO 639-2).

Re 2: The mis collection is inherently unstable – unavoidably so. Prior to 2005-08-16, an implementation of ISO 639-2 would have
tagged Ainu content as mis; after that date, an implementation of ISO 639-2 would have tagged Ainu content as ain; existing content
tagged before that date would not get retrieved by request for ain, and it would be conformant to suppose that requests for mis
would not return Ainu content. The mis collection is ugly, pure and simple. So, I don’t see what the point is of getting worried
over whether we’re making mis unstable: it’s been that way for some time.

(Note: mis is badly defined from a stability perspective, though I don’t think there’s much question of how it’s defined.)

Re 3(b): “There are times when detection can only determine that it looks like there is some linguistic content -- it is not just
binary data -- but current detection can't really determine what it might be. That is, a code that means "according to our best
available detection methods this doesn't look like it is zxx".” If you want to use mis for that, I would argue that that is
significantly changing the semantics of mis. (Even though mis is unstable, it is unstable on a qualitative level; this is a
categorical change.) I definitely oppose that. If you want an ID for “undetermined human language”, then that should be proposed. We
should not usurp an existing ID for that purpose.

Re 4: I don’t see how your example differs from this: “Nous avons une phrase en français (but this is in English)”. The fact that
the parenthetical text is in English doesn’t change the fact that the other text is in French. Similarly, in your example, the fact
that there is a comment in English does not change the fact that the rest of the text is not in a human language. Do we create tags
for “French with embedded bits of English”?

Peter

1. I think we have to be very careful here. The meaning of a standard like ISO 639-2 is established not by what we wish it would
have said, nor by what we would find out if we were able to read Peter's mind. It is established by the wording in the standard, and
how reasonable people could interpret it. The fact that "mis" was incorporated in order to account for MARC codes is interesting,
but is not in the text of the standard. We can't expect users of BCP 47 to all be able to read Peter's mind before tagging. 

2. When we are looking at stability, that is very important: our goal is that once content is correctly tagged, people can depend on
the fact that we will not change the meaning of a tag out from under them. So clarifications that we add in future versions of 4646
or the registry are fine, as long as they do not narrow the range of reasonable interpretations. We can broaden them. So in the case
of "mis", a proposed narrowing to include just the MARC codes is clearly disallowed, since it was nowhere stated in ISO 639-2 at the
time that "mis" was added to the language registry (the BCP 47 semantics are established at the time we add the code). That is one
of the key principles of BCP 47, is to isolate us where necessary from instabilities in the source standards. 

(The one exception we might be able to make is where something is so badly defined that most reasonable people couldn't come up with
any consistent definition for it.)

3. Now, I think there are steps that can be taken to make the above moot. I think Peter's suggestion for ISO 639-X of broadening all
of the Collections to remove the (Other) is exactly the right strategy, and if this can be done before 4646bis is issued, all the
better. So having 

*	bat    Baltic (Other) => Baltic languages, means any of the languages on
http://www.ethnologue.com/show_family.asp?subid=90207 

and so on. This is useful on a number of levels; it resolves a number of problems in the interpretation of language codes, and makes
the source standards themselves more stable. (In the ideal case, we would have codes for each of the possible "decision points" in
the language tree. That is, if we look at any language code such as http://www.ethnologue.com/show_lang_family.asp?code=eng we'd
have codes for each of the parent groupings, not just some of them, like "Australian languages".) 

3. Randy raised the issue as to whether "mis" in the broad sense is useful (as something that has linguistic content, but I don't
know what it is). It very much follows the model in #3. There are times when detection can only determine that it looks like there
is some linguistic content -- it is not just binary data -- but current detection can't really determine what it might be. That is,
a code that means "according to our best available detection methods this doesn't look like it is zxx". 

4. I'm leery of using zxx for programming languages, instead of just binary. There is clearly some linguistic content in "if
(content == null) { /* remove the item in the lookup table */ ...}". Maybe we need another code for this, something different than
either 'art' or 'zxx'. 

Mark

> Ainu, Andamanese, or Etruscan, but I can't tell which, so

If someone were so tempted, I would argue that would be inappropriate use of mis. Since they do not know what it is, their
declaration is that the language identity is not determined, and the appropriate tag for that is und. Appropriate use of mis does
not require that one know the language of the content; it does, however, require that one know it is *not* a language covered by any
of the available tags. 

Peter

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru  <https://www1.ietf.org/mailman/listinfo/ltru> 

-- 
Mark

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 17 Apr 2007 21:34:16 +0200

Kent Karlsson wrote:

> @font-face { font-family: Wingdings; } @font-face { font-family:
> SimSun; } @font-face { font-family: Cambria Math; } @font-face {
> font-family: Calibri; } @font-face { font-family: Tahoma; } @font-face
> { font-family: @SimSun; } @page Section1 {size: 8.5in 11.0in; margin:

And so on, followed by top-posting.  Is this a kind of joke wrt "zxx",
or did Mark's HTML trigger the "use same format" default of some UAs ?

---

From: Peter Constable <[email-masked]>
Date: Tue, 17 Apr 2007 17:33:27 -0700

On 1: I disagree: taking "other" out of mis is a categorical change - it creates a completely different concept, because the heart of the concept of mis is "other".

On 1b ("language" vs. "languages"): I disagree: while the content tagged is in a single language, the concept that the ID represents is a collection of languages. The ID represents that concept, not the content; we associate the ID with the content to indicate an association of the concept with the content.

On 4: Again, I disagree. This is like saying, "It's out of scope, mostly but not completely." Either it's in scope or it's out of scope.

Peter

From: Kent Karlsson [mailto:[email-masked]]
Sent: Tuesday, April 17, 2007 12:10 PM
To: Peter Constable; [email-masked]; [email-masked]
Subject: RE: [Ltru] Re: "mis" update review request

on 1:

I don't see why 'mis' would have to be an exception when doing a semantic change of removing (implicit or explicit) "other" for various language codes. Doing so is equally much a semantic change for 'tai' (or any other "other" collection), and of exactly the same kind, so if it is not ok for 'mis' it would not be ok for 'tai' either. (If you prefer another acronym, say 'any' instead of 'mis', that is another ball-game.)

Furthermore, since 'mul' is the only code intended for multiple languages (when it is not practical to list which languages, per fragment of the document preferably), all of the "languages" codes should instead refer to "language" in singular. This would not be a semantic change, just referring to each of the items that may be tagged, not a set of items [book shelf...] so tagged.

on 4:

Programming languages of various sorts are out of scope (like 'zxx', but unlike 'art'), but I may agree that they are out of scope in a different way than 'zxx'. Perhaps "formal language" ('for'), with no further subdivision (they are still out of scope).

        /kent k

________________________________

Peter

Mark

Peter

---

From: Peter Constable <[email-masked]>
Date: Tue, 17 Apr 2007 19:24:06 -0700

I continue to see comments in this thread in which people are trying to stretch the semantic of mis or zxx or und to cover some pet scenario. As long as we keep stretching these to mean concepts that are slightly (or entirely) different from what they were intended to be - semantics that IMO are fairly straight forward once stated - then we're going to continue debating what that should mean, and that portends that users will *never understand our intent and use them consistently.

We should simply take them for what they are. If we need concepts that are different, then we can consider identifiers to represent those concepts.

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 17 Apr 2007 19:37:41 -0700

As long as "what they are" means any reasonable interpretation given by a
reading of the standards involved, I'm in complete agreement. If "what they
are" means anything outside of the text of the standard, I'd disagree.

That being said, I recognize that broadening 'mis' would be a change from
the current standard, and while allowable, as I said I'm willing to give up
on that. (However, your using "pet scenario" to characterize something that
I and many others consider "important scenario", like language detection, is
unnecessary.)

Mark

On 4/17/07, Peter Constable <[email-masked]> wrote:
>
>  I continue to see comments in this thread in which people are trying to
> stretch the semantic of mis or zxx or und to cover some pet scenario. As
> long as we keep stretching these to mean concepts that are slightly (or
> entirely) different from what they were intended to be – semantics that IMO
> are fairly straight forward once stated – then we're going to continue
> debating what that should mean, and that portends that users will *never
> understand our intent and use them consistently.
>
>
>
> We should simply take them for what they are. If we need concepts that are
> different, then we can consider identifiers to represent those concepts.
>
>
>
>
>
> Peter
>

---

From: John Cowan <[email-masked]>
Date: Wed, 18 Apr 2007 01:16:14 -0400

Mark Davis scripsit:

> As long as "what they are" means any reasonable interpretation given
> by a reading of the standards involved, I'm in complete agreement. If
> "what they are" means anything outside of the text of the standard,
> I'd disagree.

Not *any* reasonable interpretation, and the nature of ISO 639 (and
3166 and 15924) is that the semantics of the code elements mostly
lies outside the standard.

-- 
Knowledge studies others / Wisdom is self-known;      John Cowan
Muscle masters brothers / Self-mastery is bone;       [email-masked]
Content need never borrow / Ambition wanders blind;   http://ccil.org/~cowan
Vitality cleaves to the marrow / Leaving death behind.    --Tao 33 (Bynner)

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 17 Apr 2007 23:50:12 -0700

I said "standards involved". With an "s". That includes the source
standards. And I mean "any reasonable interpretation", not "any
interpretation, reasonable or unreasonable", not an "average
interpretation", not a "misinformed interpretation". The customary norms for
interpretation of contracts, standards, and other formal documents is based
on what a "reasonable" person could interpret the text as meaning.

I really don't understand your position on this. When I say that I follow a
standard, like the XML standard, I have to follow what the standard actually
says, not what someone wishes it had said. If it says "[#x7F-#x84]" that's
what it means. A stance that I can just interpret the standard according to
what I think the authors had in mind, rather than what the text actually
says, makes any notion of compliance -- even in good faith -- completely
meaningless. If that were the case we might as well write any damn thing we
want in 4646bis, since users of the standard could interpret it pretty much
any way they would want to anyway.

If the standard (in this case, ISO 639-2) says "Baltic languages", that's
what it means. Not "Other Baltic languages", not "Baltic languages except
for ones starting with the letter L", but "Baltic languages". And if I see
"Baltic languages (Other)" that is clearly meant to be an exclusion of some
kind.

That doesn't mean that a standard is holy writ. There are always mistakes,
and always corrections and clarifications that may be added to a future
version. But whenever the new text narrows the interpretation from what it
could reasonably be interpreted as in the previous version, then that is a
compatibility breakage. It needs to be recognized as such. And any decision
to take such a step that breaks backwards compatibility has to be a
deliberate, considered decision -- especially when one of of the primary
goals of a standard like BCP 47 is stability.

Mark

---

From: "Kent Karlsson" <[email-masked]>
Date: Wed, 18 Apr 2007 10:16:39 +0200

1a: It is just as much a categorical change for 'tai', 'gem', etc. But it is a change that I support, since without such a change,
almost all of the collection codes would have an empty set of applicable languages in the context of 639-3. However, if 'mis' is
deprecated and replaced by a code for 'any language' (rather than handling 'mis' like all of the other "other" codes) in the process
of doing this, I'm not going to complain. To be nit-picking, all of the collection codes should then also be replaced (but I'm not
going to complain if they are not so replaced when removing the "other" part of the semantics). Of course, it would be helpful if
the standard defining the collection codes (639-4?) also gave an explicit hierarchy of the codes (like "'tai' covers ..., ..., ...,
and any Tai language not given a code"). Maybe that is the case already (I haven't seen a draft).
 
1b: A document (or text fragment) tagged as e.g. 'tai' is supposed to be in ONE 'tai' language, not in several 'tai' languages. So
in each individual application, a collection
code refers to one language, not several languages, even though the set of languages covered by a collection code usually has more
than one element (or no elements... see 1a).
 
4: So why is there a code for 'zxx' if it is out of scope? Furthermore, 'zxx' is supposed to mean "no linguistic content", not "out
of scope".
 
        /kent k

  _____  

From: Peter Constable [mailto:[email-masked]] 
Sent: Wednesday, April 18, 2007 2:33 AM
To: [email-masked]; [email-masked]
Subject: RE: [Ltru] Re: "mis" update review request

On 1: I disagree: taking “other” out of mis is a categorical change – it creates a completely different concept, because the heart
of the concept of mis is “other”.

On 1b (“language” vs. “languages”): I disagree: while the content tagged is in a single language, the concept that the ID represents
is a collection of languages. The ID represents that concept, not the content; we associate the ID with the content to indicate an
association of the concept with the content.

On 4: Again, I disagree. This is like saying, “It’s out of scope, mostly but not completely.” Either it’s in scope or it’s out of
scope.

Peter

on 1:

on 4:

        /kent k

  _____  

 

Peter

Mark

Peter

-- 
Mark

---

From: Peter Constable <[email-masked]>
Date: Wed, 18 Apr 2007 01:54:32 -0700

You don't seem to be grasping: mis is fundamentally different from every other collection. They are all defined in terms of membership in some set; mis is defined in terms of exclusion from everything else. The change to 'tai' is not a categorical change: the essence of the category remains membership in a particular language family. But the suggested change to mis is absolutely categorical: from exclusion from all other sets to inclusion in every set.

Zxx is a special-case identifier to indicate, "whatever content is here, it doesn't have an in-scope attribute". Zxx is no more in-scope than und - which is why these are deemed special cases. I don't see what the problem is. In terms of ISO 639, the coded concepts of the standard are the things that linguists refer to as "natural languages" or "human languages" - the things that humans speak. If content has meaning expressed in some form of communication *other than* one of these things, then that content would be considered non-linguistic, and the form of communication would be considered out of scope. The two characterizations, 'non-linguistic' and 'out of scope' correlate.

Peter

From: Kent Karlsson [mailto:[email-masked]]
Sent: Wednesday, April 18, 2007 1:17 AM
To: Peter Constable; [email-masked]; [email-masked]
Subject: RE: [Ltru] Re: "mis" update review request

1a: It is just as much a categorical change for 'tai', 'gem', etc. But it is a change that I support, since without such a change, almost all of the collection codes would have an empty set of applicable languages in the context of 639-3. However, if 'mis' is deprecated and replaced by a code for 'any language' (rather than handling 'mis' like all of the other "other" codes) in the process of doing this, I'm not going to complain. To be nit-picking, all of the collection codes should then also be replaced (but I'm not going to complain if they are not so replaced when removing the "other" part of the semantics). Of course, it would be helpful if the standard defining the collection codes (639-4?) also gave an explicit hierarchy of the codes (like "'tai' covers ..., ..., ..., and any Tai language not given a code"). Maybe that is the case already (I haven't seen a draft).

1b: A document (or text fragment) tagged as e.g. 'tai' is supposed to be in ONE 'tai' language, not in several 'tai' languages. So in each individual application, a collection
code refers to one language, not several languages, even though the set of languages covered by a collection code usually has more than one element (or no elements... see 1a).

4: So why is there a code for 'zxx' if it is out of scope? Furthermore, 'zxx' is supposed to mean "no linguistic content", not "out of scope".

        /kent k

Peter

on 1:

on 4:

        /kent k

Peter

Mark

Peter

---

From: John Cowan <[email-masked]>
Date: Wed, 18 Apr 2007 14:07:11 -0400

Mark Davis scripsit:

> I really don't understand your position on this.

I think we have been talking past one another, and I think I see why.

> When I say that I follow a standard, like the XML standard, I have
> to follow what the standard actually says, not what someone wishes it
> had said. If it says "[#x7F-#x84]" that's what it means.

The XML standard is very different in kind from the ISO standards we are
dealing with here.  If you read the XML standard and the other standards
it normatively references, like Unicode, you know everything you need
to know about determining what is and what is not an XML document.

However, ISO 639 tells you only that "en" is a code for the language named
"English".  It does not tell you which documents are and are not English.
It also gives you the code "bal" for Baltic languages, but it does not
say which those languages might be, still less which documents are and
are not in a Baltic language.  Indeed, the formal name of 639 tells
us that it is, strictly speaking, not a code for languages as such,
but for their names (ditto 3166 and 15924).

(This remark will draw a troll from HWMNBN, but I won't see it because
it'll be in my spam bucket.)

In order to avoid the Scylla of "'en' means 'English' and nothing more"
and the Charybdis of "'en' means whatever I want it to mean", we can,
indeed must, look behind the mere text of the standard to the encyclopedic
non-ISO standards from which the ISO versions are abstracted:  MARC,
Ethnologue, UNSD, Unicode.  That's what's happening here:  Peter is
telling you the way these tags have been used in the central community
of practice.

> A stance that I can just interpret the standard according to what I
> think the authors had in mind, rather than what the text actually says,

As I note above, this is a false dichotomy.

-- 
Only do what only you can do.               John Cowan <[email-masked]>
  --Edsger W. Dijkstra's advice
    to a student in search of a thesis

---

From: [email-masked]
Date: Wed, 18 Apr 2007 11:49:18 -0700

Peter wrote:

> Zxx is a special-case identifier to indicate, “whatever content is 
> here, it doesn’t have an in-scope attribute”. Zxx is no more in-
> scope than und – which is why these are deemed special cases. I don’
> t see what the problem is. In terms of ISO 639, the coded concepts 
> of the standard are the things that linguists refer to as “natural 
> languages” or “human languages” – the things that humans speak. 

If ISO 639 was primarily aimed at spoken languages, it wouldn't have taken 
this long to get codes for Mandarin, Cantonese, or Swiss German. 

I'm much more comfortable with "whatever content is in here, it doesn't 
have an in-scope attribute" than adding the phrase "could be a programming 
language." I'd agree that code is not generally a spoken human language 
and is out of scope if ISO 639 intends to identify spoken human languages. 

However, Amazon.com currently has 44,000 titles on Java alone. These 
technical books are heavily weighted with code snippets that are *written* 
by one human for communication with another. This code is never read by 
machines, it's read by humans. There is a need to mark these code sections 
separate from the English language text as is seen on the site link I sent 
before, which is heavily marked up with <code> and <pre> tags. The same is 
true for technical books sitting on your shelves and taking over large 
sections of your local bookstore. That doesn't necessarily mean that 
xml:lang tags are the best way to indicate this code. There are already 
recommended ways to do this in HTML and XHTML that do not include the 
xml:lang tag. We should not imply that the xml:lang tag and "zxx" replace 
these uses.

For the sake of argument, don't most of us read at least some type of code 
as a written language? It may not have the same properties as other human 
language, and I'll agree this is outside the scope of the 639 standards 
according to the document text. But to call code "non-linguistic" still 
just strikes me as wrong.

Regards,

Karen Broome

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 19 Apr 2007 01:38:10 +0200

(As a curtsey to Frank I converted this email to plain text. However, I may well respond
to HTML format emails in HTML format in future as well.)

Peter Constable wrote:

> You don’t seem to be grasping: mis is fundamentally different from every other collection.
> They are all defined in terms of membership in some set; mis is defined in terms of
> exclusion from everything else.

Indeed I don't grasp that, especially since your interpretation is not supported by the
normative text of 639-2 as given at http://www.loc.gov/standards/iso639-2/normtext.html.

Only und and mul are listed as "special" (subclause 4.1.2, which I guess has not been
updated when zxx was added, which is admittedly "special"). The collective codes are
covered by the text in subclause 4.1.1. And they are all given as semi-inclusive/semi-exclusive.

|| The words languages or (other) as part of a language name in the following tables
|| may be taken to indicate that a language code is a collective language code.
|| A collective language code is not intended to be used when an individual language code
|| or another more specific collective language code is available.

So it does not matter whether the word "languages" or the word "other" is used, the semantics
is the same at the meta-level. "mis", "tai", "gem", and other collective codes are on an
equal standing. But the text does not say "shall not be used when...", not even "should not...",
but just a feeble "not intended to...". Thus it is fully conformant with 639-2 to use all
and any of the collective codes ("mis" included) as inclusive, and only have a recommendation
to use a more specific code "is available". Note that also the "is available" part is weak.
Is available where? In the 639-2 code list? In the 639-3 code list? In some arbitrary
selection of the codes? Ok, the latter does not really matter, since the recommendation
to use a more specific code is not even a weak "should", it's not there at all, its
just a *usage recommendation*, **not** a coverage limitation. Thus "mis" already covers
all languages, there is no exclusion in coverage, just an "avoid use when possible"
recommendation. The word "other" is thus just misleading when it comes to the coverage
definition as is the pluralness of "languages"; both should be replaced by "language".
That would just be a clarification, not a semantic change (as it turns out, reading the
normative text of the standard).

Ok, so the text in 639-2 isn't the best standardese ever written, but I think it is
clear enough on these points: "mis" is already "any language" (with a **weak**
recommendation to use a more specific code "when available", similar to the "tag wisely"
recommendation in 4646). "und" is even more general, covering all of "mis", as well as
"mul" and the new "zxx". The "may be taken" part (see quote above) is unnecessarily weak,
though.

B.t.w. an image may well contain linguistic data: not just photographs of text,
but generated from scratch as well (instead of generating, say, a PDF document,
one can generate a series of images of the pages). Images of text, as well as
sound tracks, may perfectly well be language tagged with ordinary (non-zxx, and
preferably as specific as possible among the 639-x language codes) language tags.

		/kent k

---

From: Peter Constable <[email-masked]>
Date: Wed, 18 Apr 2007 16:38:22 -0700

From: [email-masked] [mailto:[email-masked]]

> I'm much more comfortable with "whatever content is in here,
> it doesn't have an in-scope attribute" than adding the phrase
> "could be a programming language." I'd agree that code is not
> generally a spoken human language and is out of scope if ISO
>  639 intends to identify spoken human languages.

ISO 639 is explicit: programming languages are out of scope.

> That doesn't necessarily mean that xml:lang tags are the best
> way to indicate this code. There are already recommended ways
> to do this in HTML and XHTML that do not include the xml:lang
> tag. We should not imply that the xml:lang tag and "zxx"
> replace these uses.

I completely agree. All I have said is that, if someone must (for whatever reason) tag programming code with an ISO 639 ID, then zxx is the appropriate choice.

> For the sake of argument, don't most of us read at least some
> type of code as a written language? It may not have the same
> properties as other human language, and I'll agree this is
> outside the scope of the 639 standards according to the document
> text. But to call code "non-linguistic" still just strikes me
> as wrong.

ISO 639 treats "linguistic" as those things that are in scope for coding in that standard. That may not align exactly with everyone's idea of "linguistic", but that is not a problem for ISO 639. It is clear about what it does and does not code.

Peter

---

From: Peter Constable <[email-masked]>
Date: Wed, 18 Apr 2007 16:42:12 -0700

> However, your using "pet scenario" to characterize something that I and
> many others consider "important scenario", like language detection, is
> unnecessary.)

I was not referring to tasks like language detection. I was referring to cases like "content is programming code which I perceive to be in some way linguistic". Such individual perceptions are entirely valid as just that - individual perceptions - but have no direct connection to the concepts coded in ISO 639 and should not be confused with them.

Peter

---

From: John Cowan <[email-masked]>
Date: Wed, 18 Apr 2007 21:50:28 -0400

Kent Karlsson scripsit:

> (As a curtsey to Frank I converted this email to plain text. However,
> I may well respond to HTML format emails in HTML format in future
> as well.)

As a courtesy to me and everyone else, please don't do so, but convert
all your list emails to plain text.

> Indeed I don't grasp that, especially since your interpretation
> is not supported by the normative text of 639-2 as given at
> http://www.loc.gov/standards/iso639-2/normtext.html.

Normatively, then, you can use any other code element for anything at all?
Or is it that you can use "se" for any language(s) you choose to
call "Swedish", however much the rest of us may prefer the term "Fulani"?

> So it does not matter whether the word "languages" or the word "other"

[long rant snipped]

> That would just be a clarification, not a semantic change (as it turns
> out, reading the normative text of the standard).

This is a marvelous illustration of why, in the case of 639, the standard
alone does not suffice.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
If I have seen farther than others, it is because I am surrounded by dwarves.
        --Murray Gell-Mann

---

From: Peter Constable <[email-masked]>
Date: Wed, 18 Apr 2007 21:13:04 -0700

> Indeed I don't grasp that, especially since your
> interpretation is not supported by the normative text
> of 639-2 as given at http://www.loc.gov/standards/iso639-2/normtext.html.

All your words and careful attempt to make logical deductions from the normative text of ISO 639-2 does not change the origin, legacy, intended meaning and most-representative existing usage of mis.

> Only und and mul are listed as "special"

Because the people that prepared the text for ISO 639-2 did not think through and analyze the code set in any formally-rigourous way -- they weren't paying anywhere as close attention as you just did in your message to the details of what they were doing.

> So it does not matter whether the word "languages" or
> the word "other" is used, the semantics is the same at
> the meta-level. "mis", "tai", "gem", and other collective
> codes are on an equal standing.

In terms of what the authors intended, I believe that's not the case: IIUC, the "other" collections were assumed to be cases in which there was at least one individual-language member of the general category that was coded, while for the "languages" collection it was assumed there was not. Careful examination reveals that those principles do not hold up, but that was still the intent and assumption of the authors.

(Note that, in ISO 639-3, where I was trying to ensure a good measure of formal rigour, we did not differentiate two different sub-types of collection scope.)

> Thus it is fully conformant with 639-2 to use all and any of
> the collective codes ("mis" included) as inclusive... Thus "mis"
> already covers all languages, there is no exclusion in coverage,
> just an "avoid use when possible"

Again, all your words and logic don't change what original sources, existing usage, etc. tell us about the semantic of mis, which clearly is not as you're describing. The source standard from which mis was taken into ISO 639-2 is quite clear: mis is not used for just any language, but for a relative handful -- I enumerated them in mail last week.

Peter

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 19 Apr 2007 10:29:19 +0200

> all your list emails to plain text.

But it's a pain and entails lots of manual editing of quoted texts. (No, I
do NOT want to change any default setting or anything.) Why am I the only
one to be complained upon for sending HTML mails? Mark and Peter send such
mails regularly.

> call "Swedish", however much the rest of us may prefer the 

I haven't got the foggiest idea what you are trying to say here.
(Esp. since se is "northern Sami", but even disregarding that
I cannot make any sense of what you say here.)

> > So it does not matter whether the word "languages" or the 

The text of 639-2 DOES lend itself to being interpreted, and I
gave an interpretation (which I do not think is in any way at all
"gerrymandered", but instead rather straight-forward). That you and
Peter don't agree with it is another matter, but I don't see how your
or Peter's interpretations (as given in previous emails) can at all
be supported by the text of 639-2 (without "gerrymandering").

> This is a marvelous illustration of why, in the case of 639, 
> the standard alone does not suffice.

Indeed it does not suffice, but still there is no way it can be
interpreted the way you and Peter want to. Your interpretations
are in quite distinct non-conformity with the 639-2 standard itself.

	/kent k

---

From: Addison Phillips <[email-masked]>
Date: Thu, 19 Apr 2007 15:35:59 +0100

> ISO 639 is explicit: programming languages are out of scope.

So is BCP 47. Specifically, Section 2 clearly states:

--
Language tags are used to help identify languages, whether spoken, 
written, signed, or otherwise signaled, for the purpose of 
communication. This includes constructed and artificial languages, but 
excludes languages not intended primarily for human communication, such 
as programming languages.
--

> 
> I completely agree. All I have said is that, if someone must (for 
> whatever reason) tag programming code with an ISO 639 ID, then zxx is 
> the appropriate choice.

+1, although you SHOULD NOT use a language tag for programming code. The 
use of 'zxx' for programming code or other non-linguistic data is only 
when one is *forced* to, unavoidably, provide a language tag.

Personally, I was always much more fond of the empty string for this in XML.

Addison
-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 20 Apr 2007 12:57:58 +0200

Peter Constable wrote:

>> However, your using "pet scenario" to characterize something that I
>> and many others consider "important scenario", like language detection,
>> is unnecessary.

> I was not referring to tasks like language detection. I was referring
> to cases like "content is programming code which I perceive to be in
> some way linguistic". Such individual perceptions are entirely valid
> as just that "individual perceptions" but have no direct connection
> to the concepts coded in ISO 639 and should not be confused with them.

[Quote marks edited by me, keeping LTRU on the CC list, but actually I
 think this can be solved on the subtag review list under BCP 47 rules.]

I said "zxx for code is barbaric" based on its description "no linguistic
content".  While I agree that this is an "individual perception" and not
covered by BCP 47 I also propose that we need a short comment in the
"zxx" record stating that this tag can be used for code.  The description
"no linguistic content" is misleading in the case of code.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 20 Apr 2007 13:15:35 +0200

Addison Phillips wrote:
 
> you SHOULD NOT use a language tag for programming code. The use of 
> 'zxx' for programming code or other non-linguistic data is only
 
> Personally, I was always much more fond of the empty string for 

+1 for the latter.  I'd also consider "i-default" if forced to use
a tag for all programming languages I know.  Do we have one tag too
much for this situation ?  Admittedly "zxx" is more flexible, it 
can have scripts, extensions (in theory), and annotations in the
style x-whatever.  OTOH we might not like this "zxx" flexibility. 

Frank

---

From: John Cowan <[email-masked]>
Date: Fri, 20 Apr 2007 07:39:53 -0400

Frank Ellermann scripsit:

> I'd also consider "i-default" if forced to use a tag for all programming
> languages I know.

i-default is defined very carefully by RFC 2277, section 4.5; it's not
a programming language.  Here's the relevant text:

4.5.  Default Language

   When human-readable text must be presented in a context where the
   sender has no knowledge of the recipient's language preferences (such
   as login failures or E-mailed warnings, or prior to language
   negotiation), text SHOULD be presented in Default Language.

   Default Language is assigned the tag "i-default" according to the
   procedures of RFC 1766. It is not a specific language, but rather
   identifies the condition where the language preferences of the user
   cannot be established.

   Messages in Default Language MUST be understandable by an English-
   speaking person, since English is the language which, worldwide, the
   greatest number of people will be able to get adequate help in
   interpreting when working with computers.

   Note that negotiating English is NOT the same as Default Language;
   Default Language is an emergency measure in otherwise unmanageable
   situations.

   In many cases, using only English text is reasonable; in some cases,
   the English text may be augumented by text in other languages.

-- 
By Elbereth and Luthien the Fair, you shall     [email-masked]
have neither the Ring nor me!  --Frodo          http://www.ccil.org/~cowan

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 20 Apr 2007 08:08:02 -0700

I agree, although I consider it bizarre rather than barbaric ;-)

As in example #9 of http://docs.google.com/Doc?id=dfqr8rd5_11g425c9, to
think that the following contains "no linguistic content" is bizarre. It
obviously contains linguistic content.

if (linguisticContent == null) { throw new Exception(""); }

Mark

On 4/20/07, Frank Ellermann <[email-masked]> wrote:
> Ietf-languages mailing list
> [email-masked]
> http://www.alvestrand.no/mailman/listinfo/ietf-languages
>

---

From: Peter Constable <[email-masked]>
Date: Fri, 20 Apr 2007 08:24:31 -0700

> As in example #9 of http://docs.google.com/Doc?id=dfqr8rd5_11g425c9 ,
> to think that the following contains "no linguistic content" is bizarre.
> It obviously contains linguistic content.

You could say the same of this:
MZ
________________________________

________________________________
   ÿÿ  ¸       @                                   à
º
 ´            Í!¸LÍ!This program cannot be run in DOS mode.
$       Tbï›
________________________________
È
________________________________
È
________________________________
È7ÅïÈ
________________________________
È7ÅüÈ
________________________________
È7ÅúÈ
________________________________
È
________________________________
€ÈÉ
________________________________
È7ÅìÈ3
________________________________
È7ÅýÈ
________________________________
È7ÅùÈ
________________________________
ÈRich
________________________________
È

We could probably come up with all kinds of boundary cases for which there is no “right” answer. I don’t know what use it would be.
Peter
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 20 Apr 2007 08:59:15 -0700

I don't think the programming language fragment is really a boundary
condition. Most code source nowadays are not just random hex, there
typically, not exceptionally, some real linguistic content. I would agree
with you that a hex dump of a compiled program, such as perhaps you used for
your example, is sensible to tag as zxx, but based on the wording of the
standards, I don't think we can expect zxx to apply to typical code source.
Yet, while there may be is some embedded English, we don't want to call it
"en" either.

It looks to me like the best choice currently would be "und"; as I said, I
think it might be useful to have a special tag for this just because it is a
reasonably common case that is otherwise difficult to categorize. An
alternative would be to explicitly broaden the description of "zxx" to be
"no linguistic content, or programming source code". That would be a
compatible change to 4646bis, since it is a broadening.

Mark

On 4/20/07, Peter Constable <[email-masked]> wrote:
>
>   *From:* Mark Davis [mailto:[email-masked]]
>
> **
>
> *> *As in example #9 of http://docs.google.com/Doc?id=dfqr8rd5_11g425c9 ,
>
> ------------------------------
>
> ------------------------------
>    ÿÿ  ¸       @                                   à
> º
> ------------------------------
> È
> ------------------------------
> È
> ------------------------------
> È7ÅïÈ
> ------------------------------
> È7ÅüÈ
> ------------------------------
> È7ÅúÈ
> ------------------------------
> È
> ------------------------------
> €ÈÉ
> ------------------------------
> È7ÅìÈ3
> ------------------------------
> È7ÅýÈ
> ------------------------------
> È7ÅùÈ
> ------------------------------
> ÈRich
> ------------------------------
> È
>
>
>
> We could probably come up with all kinds of boundary cases for which there
> is no "right" answer. I don't know what use it would be.
>
> Peter
>

---

From: Addison Phillips <[email-masked]>
Date: Fri, 20 Apr 2007 17:15:05 +0100

> Most code source nowadays are not just random hex, there
 > typically, not exceptionally, some real linguistic content. I would
 > agree with you that a hex dump of a compiled program, such as perhaps
 > you used for your example, is sensible to tag as zxx, but based on the
 > wording of the standards, I don't think we can expect zxx to apply to
 > typical code source. Yet, while there may be is some embedded English,
 > we don't want to call it "en" either.

I don't understand. Why is this really a problem? BCP 47 has always 
said, quite clearly, that programming languages are not what we mean by 
"languages". The following fragment is utterly perverse (but not illegal):

---

Content-Type: application/javascript
Content-Language: zxx, en

function init() {

    var stdmenu = new widget.Menu('stditems', { position:'static' });
    var menuitems = stdmenu.getItemGroups();

    for (var x=0; x<menuitems.length; x++) {
       stdmenu.getItem(x).mouseOverEvent.subscribe(doMouseEvent);
       stdmenu.getItem(x).keyDownEvent.subscribe(doMouseEvent);
    }

    stdmenu.render();
    stdmenu.show();
}
---

Note that there are no "English" words above (except, possibly, the 
keyword "static"). Calling programming language keywords or variable 
names "English" is mostly daft or especially pedantic.

Really, of course, it probably makes more sense to say what BCP 47 has 
always said: (BCP 47) "... excludes languages not intended primarily for 
human communication, such as programming languages..." and leave it at 
that. Allowing the use of 'zxx' for programming languages when utterly 
required I have no problem with, but why break out programming languages 
as anything other than an example?

Addison

---

From: "Kent Karlsson" <[email-masked]>
Date: Fri, 20 Apr 2007 18:42:27 +0200

Peter Constable wrote:

> From: Kent Karlsson
> All your words and careful attempt to make logical deductions from the
> normative text of ISO 639-2 does not change the origin, legacy, intended
> meaning and most-representative existing usage of mis.

My conlusions are in no way in contrast with that. It is just that "coverage"
and "recommended use" aren't the same thing, but has been mixed somewhat in
639-2, especially when it comes to the "other" designations. The "other"
refers to the recommended use, not the coverage.

> Because the people that prepared the text for ISO 639-2 did not think
> through and analyze the code set in any formally-rigourous way -- > they
> weren't paying anywhere as close attention as you just did in your
> message to the details of what they were doing.

Though I agree that the formulations could be improved, I think you are
giving too little credit to the authors of the 639-2 text. (Any one of
the authors listening in on this list?) The formulations are quite logical
as they are, and I do think that they had a difference between coverage
and recommended use in mind, though I agree that wasn't as *clearly* and
directly formulated as one might wish. But still I think that is the only
logical way of reading the 639-2 text.

> In terms of what the authors intended, I believe that's not the case:
> IIUC, the "other" collections were assumed to be cases in which there
> was at least one individual-language member of the general category that
> was coded, while for the "languages" collection it was assumed there
> was not. Careful examination reveals that those principles do not hold up,
> but that was still the intent and assumption of the authors.

Again, _coverage_ and _recommended use_ **do not coinside** for the collection
codes, and they cannot coinside for a reasonable reading of 639-2.

> (Note that, in ISO 639-3, where I was trying to ensure a good measure
> of formal rigour, we did not differentiate two different sub-types
> of collection scope.)

"two different sub-types of collection scope"?? I'm not sure what you are
referring to here. 639-2 and 639-3 seem to be in agreement w.r.t. collection
codes ('mis' also) as 639-3 just refers back to 639-2 on collection codes,
as well as "special codes" (just that 639-3 mentions the **new** special
code "zxx" which the normative text of 639-2 hasn't cought up with yet).

[I've been looking at http://www.sil.org/iso639-3/scope.asp, which I think
is an excerpt of the 639-3 text.]

> Again, all your words and logic don't change what original sources,
> existing usage, etc. tell us about the semantic of mis, which clearly
> is not as you're describing. The source standard from which mis was
> taken into ISO 639-2 is quite clear: mis is not used for just any
> language, but for a relative handful -- I enumerated them in mail last week.

(Not quite sure which enumaration you refer to here.)

Again, _coverage_ and _recommended use_ **do not coinside** for the collection
codes, and they cannot coinside for a reasonable reading of 639-2.

		/kent k

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 20 Apr 2007 18:41:29 +0200

Mark Davis wrote:

> An alternative would be to explicitly broaden the description of
> "zxx" to be "no linguistic content, or programming source code".
> That would be a compatible change to 4646bis, since it is a
> broadening.

I'd like the proposed "zxx" comment for rather different reasons:

IMO it is _not_ changing the existing "zxx" semantics in any way.

By definition BCP 47 claims to be only about human languages, and
draws the line roughly with "art".  Anything outside of this field
is considered as "zxx", and programming languages are explicitly
mentioned in BCP 47.

So if they get any language tag at all "zxx" was and is possible,
only the description is "mis"-leading when read outside of its
BCP 47 context, i.e. in the "zxx" record in the raw registry data.

Adding a comment to reenforce the intended context doesn't change
or broaden the meaning, it clarifies what "zxx" is.  Same idea as
for the proposed "mis" comment:

Comments added to subtags adopted from ISO standards or from UN
region numbers should IMO never change, restrict, or broaden the
real meaning in those sources, but they can reflect or clarify
this meaning when the description "mis"-erably fails to do this.

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 20 Apr 2007 12:17:22 -0700

It is clear that they aren't natural languages (although one could make a
case for them being human -- after all, martians didn't write the code you
included ;-). The question is what to tag them with in circumstances where
you need to have them tagged. My conclusion is that according to 4646, zxx
is not appropriate, which leaves someone with "und". (
http://docs.google.com/Doc?id=dfqr8rd5_11g425c9)

Mark

---

From: Peter Constable <[email-masked]>
Date: Fri, 20 Apr 2007 13:44:06 -0700

I say your programming code example is a boundary case in the sense that I don’t expect to see .cxx, .h, etc. files tagged with language tags any time soon, and I don’t expect to see a book on programming concepts tagged as anything other than en, no matter how many pages of source code samples it has.

(Granted, in an XML representation of that book there may be a question as to how individual elements should be tagged, but it’s not clear to me in that scenario what difference it really makes whether you have <code sample xml:lang=”en”> or <code sample xml:lang=”zxx”> or <code sample xml:lang=”und”> or <code sample xml:lang=””> or whatever.)

Peter

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Friday, April 20, 2007 8:59 AM
To: Peter Constable
Cc: [email-masked]; [email-masked]
Subject: Re: [Ltru] Re: "mis" update review request

I don't think the programming language fragment is really a boundary condition. Most code source nowadays are not just random hex, there typically, not exceptionally, some real linguistic content. I would agree with you that a hex dump of a compiled program, such as perhaps you used for your example, is sensible to tag as zxx, but based on the wording of the standards, I don't think we can expect zxx to apply to typical code source. Yet, while there may be is some embedded English, we don't want to call it "en" either.

It looks to me like the best choice currently would be "und"; as I said, I think it might be useful to have a special tag for this just because it is a reasonably common case that is otherwise difficult to categorize. An alternative would be to explicitly broaden the description of "zxx" to be "no linguistic content, or programming source code". That would be a compatible change to 4646bis, since it is a broadening.

Mark
On 4/20/07, Peter Constable <[email-masked]<mailto:[email-masked]>> wrote:

From: Mark Davis [mailto:[email-masked]<mailto:[email-masked]>]

MZ

$       Tbï›

We could probably come up with all kinds of boundary cases for which there is no "right" answer. I don't know what use it would be.

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 20 Apr 2007 13:52:59 -0700

> I don't expect to see .cxx, .h, etc. files tagged with language tags any
time soon

Well, every file available on the web, like
http://www.cs.duke.edu/csed/tapestry/win/date.h (chosen at random) gets some
language tag when processed at Google (I can't say what MSN, Yahoo, and
other search engines do). So right under your nose millions of pages of
source code are getting tagged, all the time. We are faced with the
practical problem of what the best thing to do is according to the standard.

Mark

On 4/20/07, Peter Constable <[email-masked]> wrote:
>
>  I say your programming code example is a boundary case in the sense that
> I don't expect to see .cxx, .h, etc. files tagged with language tags any
> time soon, and I don't expect to see a book on programming concepts tagged
> as anything other than en, no matter how many pages of source code samples
> it has.
>
>
>
> (Granted, in an XML representation of that book there may be a question as
> to how individual elements should be tagged, but it's not clear to me in
> that scenario what difference it really makes whether you have <code sample
> xml:lang="en"> or <code sample xml:lang="zxx"> or <code sample
> xml:lang="und"> or <code sample xml:lang=""> or whatever.)
>
>
>
>
>
> Peter
>
>
>
> *Sent:* Friday, April 20, 2007 8:59 AM
> *To:* Peter Constable
> *Cc:* [email-masked]; [email-masked]
> with you that a hex dump of a *compiled* program, such as perhaps you used
> for your example, is sensible to tag as zxx, but based on the wording of the
> It looks to me like the best choice *currently *would be "und"; as I said,
> I think it might be useful to have a special tag for this just because it is
> a reasonably common case that is otherwise difficult to categorize. An
> alternative would be to *explicitly *broaden the description of "zxx" to
> be "no linguistic content, or programming source code". That would be a
> On 4/20/07, *Peter Constable* <[email-masked]> wrote:
>

---

From: Peter Constable <[email-masked]>
Date: Fri, 20 Apr 2007 16:34:17 -0700

>> I don't expect to see .cxx, .h, etc. files tagged with language tags any time soon
>
> http://www.cs.duke.edu/csed/tapestry/win/date.h (chosen at random) gets
> some language tag when processed at Google (I can't say what MSN,
> Yahoo, and other search engines do). So right under your nose millions of
> pages of source code are getting tagged, all the time. We are faced with the

Well, I suppose you have to decide: when an English user does a search on "date", what's the likelihood that the content in a file date.h is going to be of interest to them.

All I've said all along is that the only appropriate language tag in ISO 639 for programming languages (when you have determined the content is in a programming language) is "zxx". That refers to content such as "#ifndef _DATE_H", not content such as "// a class for manipulating dates". Deciding how Google should tag a file like date.h is an entirely different question. Absent any context, there is no right or wrong answer about how to tag *as a whole* a file like date.h: clearly there is about as much English content in that file as there is C (or whatever prog lang it is). In a given application context, such as how Google should tag that file as a whole, there may be good reasons to tag it one way or another. But that's a question to be worked out in the context of that application; IMO it is not a question that should be answered in BCP 47.

If you want to add text suggesting that different things may be done in different application scenarios, including tagging programming-code content with English comments or string literals as "en" when that provides the attribute most useful for implementation and user experience, I have no problem with that. But a *general* statement along lines such as that using an IETF language tag/ISO 639 ID "en" for such programming code is appropriate or that the appropriate tag for your example 9 would be "und" is, IMO, not something that should appear in BCP 47.

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 20 Apr 2007 17:28:13 -0700

And as a *general* statement about applying zxx to programming languages, I
think that unless and until we add ", or programming languages" to the
Description, it is unsuitable for application to programming languages. I
don't think we are that far apart here -- we both agree that zxx ought to be
the right thing. Where we disagree is that you think it already has that
meaning, and I can't see it at all in the standard.

But since it is a broadening, we can make a change and then both of us would
be happy (or at least happier). So what do you say we agree on that, and be
done with this part of the issue?

Mark

On 4/20/07, Peter Constable <[email-masked]> wrote:
>
>  *From:* [email-masked] [mailto:[email-masked]]
> *On Behalf Of *Mark Davis
>
> **
>
> Well, I suppose you have to decide: when an English user does a search on
> "date", what's the likelihood that the content in a file date.h is going
> to be of interest to them.
>
>
>
> All I've said all along is that the only appropriate language tag in ISO 639 for programming languages (when you have determined the content is in a programming language) is "zxx". That refers to content such as "#ifndef _DATE_H", not content such as "// a class for manipulating dates". Deciding how Google should tag a file like date.h is an entirely different question. Absent any context, there is no right or wrong answer about how to tag **as a whole** a file like date.h: clearly there is about as much English content in that file as there is C (or whatever prog lang it is). In a given application context, such as how Google should tag that file as a whole, there may be good reasons to tag it one way or another. But that's a question to be worked out in the context of that application; IMO it is not a question that should be answered in BCP 47.
>
>
>
> If you want to add text suggesting that different things may be done in
> different application scenarios, including tagging programming-code content
> with English comments or string literals as "en" when that provides the
> attribute most useful for implementation and user experience, I have no
> problem with that. But a **general** statement along lines such as that
> using an IETF language tag/ISO 639 ID "en" for such programming code is
> appropriate or that the appropriate tag for your example 9 would be "und"
> is, IMO, not something that should appear in BCP 47.
>
>
>
>
>
> Peter
>
>
>
>

---

From: John Cowan <[email-masked]>
Date: Sat, 21 Apr 2007 00:45:32 -0400

Mark Davis scripsit:

> typically, not exceptionally, some real linguistic content. 

That's a matter of definition: taggers must draw the line somewhere, and
ISO 639 by its nature gives no help.  Is this email in English?  Yes.
Is the OED written in English?  Unquestionably.  Is a mere wordlist
like /usr/share/dict/words in English?  Probably.  What about a shorter
list of English words which on inspection turns out to be the reserved
words of Cobol-68?  Perhaps.  What about the text of _Finnegans Wake_?
Scholars disagree, though several translations have been made.  What about
the source of TeX, which is a complex admixture of English prose (well
beyond line-by-line comments) and Pascal code fragments?  I don't know.

A document in English may be represented in a computer as text or as a
graphic; in either case "en" is applicable.  How large a contribution
to the graphic must letterforms make before "en" is reasonable?  Is a
photograph with some English text shown in the image (on a sign, for
example) a reasonable instance of "en"?  What about a subtitle embedded
in the image?  What about a copyright notice using the word "Copyright",
which is an English word but can be used, by international agreement,
in any copyright notice in place of the copyright symbol?  What if
"Copr." is used, which is also equivalent to the copyright symbol?

There is "en" and there is "cpe", which represents the collection of
pidgin and creole languages with an English lexifier.  639-3 provides 30
such languages, or 32 if you count Bislama and Tok Pisin, which have their
own 639-2 code elements.  However, in classifying spoken-word documents
(whether recordings or transcriptions) made where a creole language
is spoken, one generally finds that they contain an extraordinarily
complex mixture of the local variety of English and of the creole, with
the proportions and details of the mixture varying with the speaker, the
audience, the subject matter, and a host of other factors.  The behavior
of "en" and "sco" both in speech and in writing is essentially similar.
It is and always will be a matter of judgement whether to use "en",
"cpe" (or "sco"), or "mul" in such situations.

In short, "English" is not a bright-line notion.  Formally, all that
ISO 639 tells us is that if you have something that you believe to be
English, you may tag it "en", and a counterparty may reconstruct the fact
that you believe it to be English.  There is a large class of documents
where all will agree that the language is English and that the "en"
tag is appropriate, and another large class where all will agree that
it is inappropriate.  In between there are judgement calls.

> but based on the wording of the standards, I don't think we can expect
> zxx to apply to typical code source.  Yet, while there may be is some
> embedded English, we don't want to call it "en" either.

Maybe we do, maybe we don't; it depends on our inividual purposes.

> It looks to me like the best choice currently would be "und";

"und" is always a correct choice, as it is a formal expression of
ignorance or indifference or both.

> I think it might be useful to have a special tag for this just
> because it is a reasonably common case that is otherwise difficult
> to categorize.

I don't think you will make any progress with 639/RA on that one, nor
is it appropriate for BCP 47 to add tags for entirely new purposes, nor
do you seem to have much traction from this mailing list for doing so.
A private-use tag, perhaps based on zxx, perhaps not, is surely suitable
for Google's private purposes.

> compatible change to 4646bis, since it is a broadening.

It may or may not be a broadening, depending on whether you think
programming source code has linguistic content or not.

-- 
If you have ever wondered if you are in hell,         John Cowan
it has been said, then you are on a well-traveled     http://www.ccil.org/~cowan
road of spiritual inquiry.  If you are absolutely     [email-masked]
sure you are in hell, however, then you must be
on the Cross Bronx Expressway.          --Alan Feuer, NYTimes, 2002-09-20

---

From: John Cowan <[email-masked]>
Date: Sat, 21 Apr 2007 00:51:09 -0400

Mark Davis scripsit:

> http://www.cs.duke.edu/csed/tapestry/win/date.h (chosen at random)
> gets some language tag when processed at Google (I can't say what MSN,
> Yahoo, and other search engines do). So right under your nose millions
> of pages of source code are getting tagged, all the time. We are faced
> with the practical problem of what the best thing to do is according
> to the standard.

Is "we" Google here, or the WG?  I don't think the WG is faced with any
such problem, though I agree that Google is; however, Google has a variety
of moves (including ignoring the issue, using "zxx", using a private tag,
or using a 'zxx' language subtag with a private subtag.

Without wishing to induce a breach of the rules of Fight Club, I'd be
interested to know what Google does currently, and whether it recognizes
such a file as C source or not.

-- 
And through this revolting graveyard of the universe the muffled, maddening
beating of drums, and thin, monotonous whine of blasphemous flutes from
inconceivable, unlighted chambers beyond Time; the detestable pounding
and piping whereunto dance slowly, awkwardly, and absurdly the gigantic
tenebrous ultimate gods -- the blind, voiceless, mindless gargoyles whose soul
is Nyarlathotep. (Lovecraft) John [email-masked]|ccil.org/~cowan

---

From: John Cowan <[email-masked]>
Date: Sat, 21 Apr 2007 01:00:47 -0400

Mark Davis scripsit:

> And as a *general* statement about applying zxx to programming
> languages, I think that unless and until we add ", or programming
> languages" to the Description, it is unsuitable for application to
> programming languages.

That would only be true if there were a clear and convincing argument,
acceptable to all, that code is written in a natural language.  Otherwise,
it is a mere judgement about a class of borderline cases.

> don't think we are that far apart here -- we both agree that zxx ought
> to be the right thing. Where we disagree is that you think it already
> has that meaning, and I can't see it at all in the standard.

All the standard says is "Non-linguistic content".  So to assert that
code is not "zxx", you must say that it is linguistic content, though
you are unwilling to apply "en" (or, presumably, any other specific or
collective language tag).  What is this linguistic content that is not
in any language?

You have for whatever reason decided that reading and writing code
(presumably exclusive of comments, which are unquestionably in English
or French or Japanese or what have you) is linguistic behavior, but why?
Learning to program is very unlike either acquiring French or Esperanto
as a first language, or learning them as a second language, though
admittedly more like the latter than the former.  Are you also willing
to say that mathematical formulae embedded in a math paper in English,
French, German, Russian, or what have you are linguistic content, though
certainly not in any of those languages?

> But since it is a broadening, we can make a change and then both of
> us would be happy (or at least happier). So what do you say we agree
> on that, and be done with this part of the issue?

Because it is not clear that it is a broadening, and it is very colorably
an out-of-scope extension.  Why not mention musical notation (of
instrumental compositions, where there are no lyrics to muddy the
waters) as well?

-- 
I am expressing my opinion.  When my            John Cowan
honorable and gallant friend is called,         [email-masked]
he will express his opinion.  This is           http://www.ccil.org/~cowan
the process which we call Debate.                   --Winston Churchill