Subject: [Ltru] Re: Language Tag Special Cases

From: "Doug Ewell" <[email-masked]>
Date: Sat, 21 Apr 2007 14:23:43 -0700

Kent Karlsson <kent dot karlsson14 at comhem dot se> wrote:

> 'mis' is NOT a "special" code in the 639 sense. It is a collection 
> code (among many other collection codes). The only thing "special" 
> about it is that it is the "top level" collection (in the inclusion 
> hierarchy), but that does not make it "special" in the ISO 639 sense.

It is a special code from the standpoint of RFC 4646, and it is *not* 
just like all the other collections.  As John tried to point out some 
time ago, the concept of "miscellaneous" generally implies "does not fit 
into any other category," and does not imply "fits into all categories."

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Kent Karlsson" <[email-masked]>
Date: Sat, 21 Apr 2007 23:42:09 +0200

Doug Ewell wrote:
> Kent Karlsson wrote:
> just like all the other collections.

That is not true. 'mis' is not taken out as a special case in 4646,
nor should it be. It thus works just like in 639-*, and there it
is one of the collection codes, and work just like all the other
collection codes.

	/kent k

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 21 Apr 2007 15:30:21 -0700

Kent Karlsson <kent dot karlsson14 at comhem dot se> wrote:

> nor should it be. It thus works just like in 639-*, and there it is 
> one of the collection codes, and work just like all the other 
> collection codes.

Peter has repeatedly tried to show that the intent of "mis", going back 
to its MARC origin, has always been to mean "languages not covered by 
any other individual or collection code."

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 21 Apr 2007 16:21:26 -0700

Hi -

> From: "Doug Ewell" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
> Cc: "Kent Karlsson" <[email-masked]>
> Sent: Saturday, April 21, 2007 3:30 PM
> Subject: Re: [Ltru] Re: Language Tag Special Cases
...
> any other individual or collection code."
...

The cases where "mis" would be of use for our purposes seem to be
vanishingly small.

If the recognizer can identify the language(s) of the material,
there are three possibilities that seem practical to me:
    (1) an existing language code
    (2) a meaningful collection code
    (3) a private-use code until a tag for the language in question has
        been registered

The members of the set of languages in "mis" are so scattered that I'd
really be surprised if anyone built a recognizer that would be able to
figure out "it's one of the mis languages, but I can't say which one".

If the recognizer doesn't recognize the language of the material, "mis"
isn't appropriate.

Randy

---

From: "Kent Karlsson" <[email-masked]>
Date: Sun, 22 Apr 2007 01:37:50 +0200

I think you should catch up on your LTRU email backlog...
Peter and I are actually in agreement, it seems. (Not that
I have changed my mind in any way, after reading the
ISO 639-2 text.)

	/kent k

> -----Original Message-----
> From: Doug Ewell [mailto:[email-masked]] 
> Sent: Sunday, April 22, 2007 12:30 AM
> To: LTRU Working Group
> Cc: Kent Karlsson
...

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 21 Apr 2007 16:48:04 -0700

> I have changed my mind in any way, after reading the

On Sat, 21 Apr 2007 08:40:18 -0700, Peter Constable <petercon at 
microsoft dot com> wrote:

> Using an example, an application of ISO 639 could choose to use as its 
> inventory {eng, fra, deu, und, zxx, mis}, in which case mis would 
> encompass Swedish and Italian and several thousand other languages *in 
> that application context*.
>
> I think you'd agree with that.
>
> In the context of BCP47, mis would encompass only a very small set of 
> languages (once 639-3 is included, a very small set indeed).

---

From: "Kent Karlsson" <[email-masked]>
Date: Sun, 22 Apr 2007 01:55:21 +0200

Doug,

Since you still don't seem to have caught up with your backlog,
I'd better repeat my reply to Peter on the email you just copied
(I seem to have to repeat myself quite a lot):

> From: Kent Karlsson [mailto:[email-masked]] 
> Sent: Saturday, April 21, 2007 10:28 PM
> To: [email-masked]
> Subject: RE: [Ltru] Re: Language Tag Special Cases
> 
> Peter Constable wrote:
> > From: Kent Karlsson
> > 
> > > "Miscellaneous" does mean "anything". But it is an "anything"
> > > for things not otherwise classified. *coverage* (everything)
> > > vs. *recommended use* (not otherwise classified).
> > 
> > I don't think I'd express things the way you do, but I start
> > to think that, in practical terms, our views are similar.
> > 
> > Using an example, an application of ISO 639 could choose to use
> > as its inventory {eng, fra, deu, und, zxx, mis}, in which case
> > mis would encompass Swedish and Italian and several thousand other
> > languages *in that application context*.
> 
> Yes.
> 
> Yes, modulo the possibility that 'mis' and other collection codes
> maybe (as John Cowan suggested) could be used for a partial 
> information
> scenario. E.g.
> 
> 'smi' - this has been determined (or deemed) to have a Saami language
> content, but have not determined (yet) which Saami language;
> 
> 'mis' - this has been determined (or deemed) to have content 
> in a (one)
> natural language, but have not determined (yet) which one;
> 
> 'und' - no determination made (yet), may be in one or more natural
> languages, or may (predominantly) be something else.
> 
> But that is not clear from the text of 639-2, nor IIRC from the
> much longer and intricate BCP47 text, that collection codes can
> be used like that, just that 'und' can be used like that.
> 
> 		/kent k
> 
>

---

From: Peter Constable <[email-masked]>
Date: Sat, 21 Apr 2007 16:59:43 -0700

> I'd better repeat my reply to Peter on the email you just copied

Actually, you also are not caught up. I agree with you on some points, but I do not agree with you on all, including how mis should be used in the BCP47 context.

Peter

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 21 Apr 2007 17:08:24 -0700

> (I seem to have to repeat myself quite a lot):

I subscribe to the digest, and only received your most recent exchange 
with Peter a minute ago.  You wrote:

> 'mis' - this has been determined (or deemed) to have content in a 
> (one) natural language, but have not determined (yet) which one;

and Peter replied:

> This is where we depart. Here, I'd use 'und'. Again, in the context of 
> BCP47, I consider mis to encompass a fairly small set of languages. I 
> don't have a big problem with 'smi' when the confidence level is at 
> the level of "it's a Saami language", but I still maintain that 'mis' 
> is for the exceptions not covered by available IDs, and 'und' is for 
> the general "don't know" case.

I wonder which one of us needs to catch up.

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 21 Apr 2007 17:19:27 -0700

Hi -

As a technical contributor...

> From: "Kent Karlsson" <[email-masked]>
> To: "'Doug Ewell'" <[email-masked]>
> Cc: "'LTRU Working Group'" <[email-masked]>
> Sent: Saturday, April 21, 2007 4:55 PM

I must admit to sharing in Doug's bewilderment.  See below.
 
> > natural language, but have not determined (yet) which one;

I disagree.  mis is appropriate only if the language is known to
be a language not covered by any other grouping, and the tagger
has decided for some perverse reason that using that language's
code would be inappropriate.  The case you describe would be more
appropriately handled by 'und'.

> > be used like that, just that 'und' can be used like that.

The RFC 4646 text seems quite clear about the use of 'und':

   4.  The 'und' (Undetermined) primary language subtag SHOULD NOT be
       used to label content, even if the language is unknown.  Omitting
       the language tag altogether is preferred to using a tag with a
       primary language subtag of 'und'.  The 'und' subtag MAY be useful
       for protocols that require a language tag to be provided.  The
       'und' subtag MAY also be useful when matching language tags in
       certain situations.

Randy

---

From: "McDonald, Ira" <[email-masked]>
Date: Sat, 21 Apr 2007 18:19:04 -0700

Hi,

The meaning of 'mis' (in ISO 639-*) is obviously going to
change over time.  There is no possible *safe* use of 'mis'
in the BCP47 context.

RFC4646bis should explicitly prohibit the use of 'mis'.

Digital ink is cheap, but really folks this is far too much
of a waste of everyone's time.

Cheers,
- Ira

Ira McDonald (Musician / Software Architect)
Chair - Linux Foundation Open Printing WG
Blue Roof Music / High North Inc
PO Box 221  Grand Marais, MI  49839
phone: +1-906-494-2434
email: [email-masked]

No virus found in this outgoing message.
Checked by AVG Free Edition. 
Version: 7.5.463 / Virus Database: 269.5.7/771 - Release Date: 4/21/2007 11:56 AM

---

From: "Kent Karlsson" <[email-masked]>
Date: Sun, 22 Apr 2007 12:56:17 +0200

(Catching up ;-)

Peter Constable wrote:

> > 'mis' - this has been determined (or deemed) to have content in a (one)

> This is where we depart.

Actually, using collection tags (not just the special code 'und')
this way was John Cowan's suggestion.

(And recall that according to 639, 'mis' is just a collection tag,
there is nothing special or exceptional about it standardswise.)

> Here, I'd use 'und'.

One can use 'und', but 'und' is even more undetermined than 'mis'
(if you allow collection codes for the partial information scenario).

> Again, in the context of BCP47,
> I consider mis to encompass a fairly small set of languages.

The recommended use would be very small, yes. EXACTLY THE SAME goes
for ALL of the collection codes in a 4646bis context which has
a separate code for each (known, semi-modern, natural) language.

> I don't have a
> big problem with 'smi' when the confidence level is at the level
> of "it's a Saami language",

If you take the recommended use restriction strictly, 'smi' "can"/
"should" only be used for Saami languages that have no code of their
own (in a 4646bis context, or even a 4646 context, as both provide
several more specific Saami codes).

As another example: In a 4646 (and current registry content), 'cpp'
is the only recommended available code for Cap Verdian (Kabuverdianu).
But in a 4646bis context (given the draft registry) 'kea' would be
the only recommended available code for Cap Verdian. 'cpp', and
indeed 'mis', as well as the special code 'und', are still valid for
Kabuverdianu, as the coverage (of course) does not change, but the
recommended use changes as the more specific code 'kea' becomes
available.

> but I still maintain that 'mis' is for the exceptions not
> covered by available IDs, and 'und' is for the general "don't know" case.

There is no exception in principle between 'mis' and the other collection
codes, neither in 639, nor in 4646, nor should there be in 4646bis.

'mis' covers all natural languages, but the recommended use would be
to only use it for languages for which "an individual language code or
another more specific collective language code is available" does not
hold. Note again that this phrase, quoted from the 639-2 standard,
does not even make sense for 'mis' if 'mis' does not COVER ALL languages.
There is a difference between COVERAGE on one hand, and RECOMMENDED USE
on the other. The 639-2 standard itself does not seem to favour a
partial information interpretation for any of the collection codes,
only for 'und'.

But if you allow a partial information scenario, as John Cowan suggested
(not at all clear that the normative text of 639-2 covers such use),
there is no reason to take out 'mis' as a special case. Taking two
different approaches at the same time (one for some collection codes,
another for another set of collection codes) would be inconsistent.

=============================================================
 
> change over time.  There is no possible *safe* use of 'mis'

The *coverage* of 'mis' does not really change over time, the coverage
of 'mis' is the scope of the 639 standards. Likewise, the *coverage*
of 'smi' does not change much over time (language development and
diversification is slow), but the *recommended use* for any of the
collection codes may change in an instant (if one takes aboard another
Saami language code not previously used in that 639 use context).

The *recommended use*, however, not only may change over time, but
also over each application context of 639 (see above). 

> RFC4646bis should explicitly prohibit the use of 'mis'.

Either we prohibit all of the collection codes, or none of them
(in 4646bis).

Or allow for a partial information interpretation for all of the
collection codes, in a consistent manner, or for none of the
collection codes.

> of a waste of everyone's time.

Well, I think we need to flesh this out, so that the interpretation
and possible (or not) use of collection codes is made clear.
(Though I do think 639(-2) is rather clear on 'mis'; it's not all
that hard, strange, or exceptional at all. But it is very succinctly
formulated in 639-2.)

		/kent k