Subject: [Ltru] RE: RE: ISO 639-2 decision: "mis"

From: Peter Constable <[email-masked]>
Date: Thu, 14 Jun 2007 18:48:47 -0700

[re-sending to appropriate list]

+1 to what Doug said. The name has changed; the intended semantic has not.

I think we'd all agree it's not a good idea for anybody to use 'mis' in the context of 4646bis. But I don't see why we need to do more than recommend against its use except as a last resort: if an informed user feels that it's really what they need to do because they believe they know the language of their content and believe there is no subtag for it in the LSTR and feel they need to declare that condition, that should be their prerogative.

I would make some minor changes to the current text: from

"The 'mis' (Miscellaneous) primary language subtag is derived from a collective code and is used to identify linguistic content whose language is known but cannot otherwise be identified. It is commonly used when the range of language tags is constrained or for languages not otherwise categorized... "

To

"The 'mis' (Uncoded languages) primary language subtag is used to declare that linguistic content is in a known language but that this language cannot be indicated. It is typically used when the range of language tags supported in a given application context is constrained or for languages not otherwise categorized... "

(I particular think we should avoid "commonly used": we don't want to suggest to the reader that it is commonly used.

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Doug Ewell
Sent: Thursday, June 14, 2007 2:13 PM
To: [email-masked]
Subject: Re: ISO 639-2 decision: "mis"

Kent Karlsson <kent dot karlsson14 at comhem dot se> wrote:

> I agree with Mark here.
>
> With this change, the use recommendation effectively hasn't changed, but
> the coverage has, and it has changed in a way to make it inherently
> unstable.

I agree with Peter here.  The intent of this code element has not changed,
but the new name reflects the intent much better than the old name did.  The
code element was always inherently unstable.

> I see two ways of dealing with this:
>
> 1) Ignore the change, and let the coverage still be "all languages" (one a
> a time).

Strongly, strongly opposed.

> 2) Deprecate 'mis', and use 'und' ("all languages (or not a language)") in
> its place (despite the different intention).

3) Deprecate "mis", and use "und" to mean "the language of this content is
undetermined" as it has always meant.

4) Don't deprecate "mis", and use "und" to mean "the language of this
content is undetermined" as it has always meant.

Please stop trying to postulate a relationship between "mis" and "und" that
isn't there.  They have always meant two totally different things.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ietf-languages mailing list
[email-masked]
http://www.alvestrand.no/mailman/listinfo/ietf-languages
_______________________________________________
Ietf-languages mailing list
[email-masked]
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Fri, 15 Jun 2007 16:44:38 +0100

I agree with Peter and Doug.  However, I would change Peter's proposed text
to read as follows:

 "The 'mis' (Uncoded languages) primary language subtag MAY be used when
there is an identified need to declare that linguistic content is in a known
language that is not currently supported within the Registry by any other
language subtag or when the range of language tags supported in a given
application are constrained. "

Debbie Garside

> "The 'mis' (Uncoded languages) primary language subtag is 
> used to declare that linguistic content is in a known 
> language but that this language cannot be indicated. It is 
> typically used when the range of language tags supported in a 
> given application context is constrained or for languages not 
> otherwise categorized... "

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of 
> Peter Constable
> Sent: 15 June 2007 02:11
> Subject: RE: ISO 639-2 decision: "mis"
> 
> I think we'd all agree it's not a good idea for anybody to 
> use 'mis' in the context of 4646bis. But I don't see why we 
> need to do more than recommend against its use except as a 
> last resort: if an informed user feels that it's really what 
> they need to do because they believe they know the language 
> of their content and believe there is no subtag for it in the 
> LSTR and feel they need to declare that condition, that 
> should be their prerogative.
> 
> "The 'mis' (Miscellaneous) primary language subtag is derived 
> from a collective code and is used to identify linguistic 
> content whose language is known but cannot otherwise be 
> identified. It is commonly used when the range of language 
> tags is constrained or for languages not otherwise categorized... "
> 
> To
> 
> "The 'mis' (Uncoded languages) primary language subtag is 
> used to declare that linguistic content is in a known 
> language but that this language cannot be indicated. It is 
> typically used when the range of language tags supported in a 
> given application context is constrained or for languages not 
> otherwise categorized... "
> 
> (I particular think we should avoid "commonly used": we don't 
> want to suggest to the reader that it is commonly used.
> 
> Peter
> 
> 
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Doug Ewell
> > With this change, the use recommendation effectively hasn't 
> changed, 
> > but the coverage has, and it has changed in a way to make it 
> > inherently unstable.
> 
> I agree with Peter here.  The intent of this code element has 
> not changed, but the new name reflects the intent much better 
> than the old name did.  The code element was always 
> inherently unstable.
> 
> > 1) Ignore the change, and let the coverage still be "all languages" 
> > (one a a time).
> 
> > 2) Deprecate 'mis', and use 'und' ("all languages (or not a 
> > language)") in its place (despite the different intention).
> 
> 3) Deprecate "mis", and use "und" to mean "the language of 
> this content is undetermined" as it has always meant.
> 
> 4) Don't deprecate "mis", and use "und" to mean "the language 
> of this content is undetermined" as it has always meant.
> 
> Please stop trying to postulate a relationship between "mis" 
> and "und" that isn't there.  They have always meant two 
> totally different things.
> 
> --
> Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  
> UTN #14 http://users.adelphia.net/~dewell/

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 15 Jun 2007 09:45:54 -0700

Hi -

As a technical contributor...

> From: "Debbie Garside" <[email-masked]>
> To: <[email-masked]>; <[email-masked]>
> Cc: "'LTRU Working Group'" <[email-masked]>
> Sent: Friday, June 15, 2007 8:44 AM
> Subject: [Ltru] RE: ISO 639-2 decision: "mis"
>
...

I disagree with the phrase "or when the range of language tags supported in
a given application are constrained."  If the application knows what language
it is, it should use the correct tag, if one exists.  If the application doesn't
know what language is in use, "und" would be correct.

I also suggest that we consider what RFC 2119 says regarding "MAY" versus
"SHOULD NOT" qualifiers for features:

4. SHOULD NOT   This phrase, or the phrase "NOT RECOMMENDED" mean that
   there may exist valid reasons in particular circumstances when the
   particular behavior is acceptable or even useful, but the full
   implications should be understood and the case carefully weighed
   before implementing any behavior described with this label.

5. MAY   This word, or the adjective "OPTIONAL", mean that an item is
   truly optional.  One vendor may choose to include the item because a
   particular marketplace requires it or because the vendor feels that
   it enhances the product while another vendor may omit the same item.
   An implementation which does not include a particular option MUST be
   prepared to interoperate with another implementation which does
   include the option, though perhaps with reduced functionality. In the
   same vein an implementation which does include a particular option
   MUST be prepared to interoperate with another implementation which
   does not include the option (except, of course, for the feature the
   option provides.)

Randy

---

From: GerardM <[email-masked]>
Date: Fri, 15 Jun 2007 18:46:11 +0200

Hoi,
Standards have a tendency to lag behind what is accepted to be true. When a
standard does not acknowledge a specific language of linguistic entity yet
because it is not in that version of the standard, a tag could be not
available because a specific version of standard has not been ratified yet.
Thanks,
     Gerard

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 15 Jun 2007 10:08:00 -0700

Hi -

As technical contributor...

> From: "GerardM" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Friday, June 15, 2007 9:46 AM
> Subject: Re: [Ltru] RE: ISO 639-2 decision: "mis"
>
> Hoi,

This is the *sole* case where "mis" would be appropriate,

However, I would also argue that if I were a field linguist collecting data on
previously unknown languages (the most likely scenario in which one could legitimately
encounter a need for "mis") that I'd be better off using private-use subtags as
an *interim* measure for my data.  With private-use subtags I'd know which was which.
With "mis" I'd have thrown everything into one basket.

Randy

---

From: Peter Constable <[email-masked]>
Date: Fri, 15 Jun 2007 11:02:14 -0700

Randy:

MARC is an application context -- an application of language tags, not an application in the sense of a piece software. In that application context, only IDs from 639-2 can be used: that is a constraint on the range of language tags supported in that application context. Whether a user or a software process is assigning tags, the options are not necessarily binary, as you suggest: in that application context it is entirely possible for the language of given content to be known but not to be within the restricted set permitted by MARC.

Peter

-----Original Message-----
From: Randy Presuhn [mailto:[email-masked]]
Sent: Friday, June 15, 2007 9:46 AM
To: LTRU Working Group
Subject: Re: [Ltru] RE: ISO 639-2 decision: "mis"

Hi -

...

Randy

---

From: Peter Constable <[email-masked]>
Date: Fri, 15 Jun 2007 11:06:40 -0700

What you say wrt private-use subtags being better than 'mis' is true if the application context is a linguist collecting field data. It may not be true in all application contexts, however. In particular, it is NOT the practice in MARC, nor would it be in the interest of that standard to start defining private-use IDs for use in MARC records. (It may make sense for a local implementation of MARC, however.)

Peter

-----Original Message-----
From: Randy Presuhn [mailto:[email-masked]]
Sent: Friday, June 15, 2007 10:08 AM
To: LTRU Working Group
Subject: Re: [Ltru] RE: ISO 639-2 decision: "mis"

Hi -

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 15 Jun 2007 11:15:46 -0700

Hi -

> From: "Peter Constable" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
> Sent: Friday, June 15, 2007 11:02 AM
> Subject: RE: [Ltru] RE: ISO 639-2 decision: "mis"
...
> MARC is an application context -- an application of language tags, not an
> application in the sense of a piece software. In that application context, only
> IDs from 639-2 can be used: that is a constraint on the range of language tags
> supported in that application context.

As such, I'd argue that these constraints mean it isn't an application of BCP 47.

> Whether a user or a software process is assigning tags, the options are not
> necessarily binary, as you suggest: in that application context it is entirely possible
> for the language of given content to be known but not to be within the restricted set
> permitted by MARC.

These things might be called tags, and they might even be a subset of the
set of strings that are well-formed, valid BCP 47 tags, but this is *NOT* an
application of BCP 47, and, consequently, is not our problem.

Randy

---

From: Addison Phillips <[email-masked]>
Date: Fri, 15 Jun 2007 11:19:07 -0700

> know what language is in use, "und" would be correct.

That's nice in theory. But some applications use a subset of tags (I 
used the MARC21 example in the text) and then transmit them through 
another system (where the larger range of RFC 4646 is available). Those 
systems only know that the content is 'mis' (because it is tagged that 
way) and not what the miscellaneous language happens to be.

We don't define what subtags applications are capable of supporting or 
required to support elsewhere. We should focus on providing appropriate 
guidance (which is not to use the subtag). I note that Peter's use of 
MAY doesn't reflect the current draft.

I propose that we modify the draft to say the following. Please note: 
the paragraph format is consistent with the other items in that section. 
I think that folks should glance at that text when proposing edits.

--
The 'mis' (Uncoded) primary language subtag is used to identify 
linguistic content whose language is known but cannot otherwise be 
identified. It is intended for use when the range of language tags is 
constrained or for languages not otherwise categorized. It SHOULD NOT be 
used except when other means of identifying the language are not 
available. For example, a library application might be limited to the 
set of subtags defined for use by the [MARC21] standard. The 'mis' 
subtag might be used by this application for languages not included in 
that set.
--

Comments?

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 15 Jun 2007 11:50:42 -0700

Hi -

> From: "Addison Phillips" <[email-masked]>
> Sent: Friday, June 15, 2007 11:19 AM

This is an interworking problem, where data from a non-BCP 47 environment
needs to travel through a BCP 47 one.  In this particular case, things
work out.  But as for the proposed text...

...
> identified.

The phrase "known but cannot otherwise be identified" is a bit obscure.
I suggest replacing it with "known but for which no subtag has been defined."

> It is intended for use when the range of language tags is 
> constrained or for languages not otherwise categorized.

This only makes sense if we somehow want to shoehorn MARC into being
BCP 47 compliant.  I propose deleting this sentence.

> It SHOULD NOT be 
> available.

Minor edit: I think the would be much easier to read if worded:
 It SHOULD NOT be used when other means of identifying the language are available.

> For example, a library application might be limited to the 

I think this is not a good example of a BCP 47 use of "mis".

Although I think it's quite reasonable and fortunate that MARC data
could traverse a BCP 47 environment unharmed (and this falls under the
"SHOULD NOT" rules of RFC 2119), I do *not* think we should stretch things
to where we'd claim that MARC is "compliant".  It's just a happy circumstance
that data encoded under the MARC regime is still meaningful when interpreted
under BCP 47 rules, unless tagged with "mis".

Randy

---

From: Peter Constable <[email-masked]>
Date: Fri, 15 Jun 2007 12:04:33 -0700

You appear to be assuming that for something to qualify as "an application of BCP 47" it must support all tags defined by BCP 47.

I would certainly agree that a BCP 47-conformant parser must fully support the syntax that's defined, and for a BCP-47 validator to validate all and only tags that are defined in relation to a given snapshot of the registry.

But I see no reason why a higher-level protocol cannot restrict the set of tags it accepts and still be considered "an application of BCP 47". I wouldn't be surprised if was actually common for people to create higher-level protocols that normatively reference BCP 47 but that define some subset of tags that are permitted. What is the problem in saying those are "applications of BCP 47"? It would certainly be getting applied in those cases.

Peter

-----Original Message-----
From: Randy Presuhn [mailto:[email-masked]]
Sent: Friday, June 15, 2007 11:16 AM
To: LTRU Working Group
Subject: Re: [Ltru] RE: ISO 639-2 decision: "mis"

Hi -

...

Randy

---

From: [email-masked]
Date: Fri, 15 Jun 2007 12:07:39 -0700

But MARC uses the three-character 639-2 tags for the most common 
languages, right? If so, that is not a restriction, it's a violation.

Karen Broome

Peter Constable <[email-masked]> 
06/15/2007 12:04 PM

To
Randy Presuhn <[email-masked]>, LTRU Working Group 
<[email-masked]>
cc

Subject
RE: [Ltru] RE: ISO 639-2 decision: "mis"

You appear to be assuming that for something to qualify as "an application 
of BCP 47" it must support all tags defined by BCP 47.

I would certainly agree that a BCP 47-conformant parser must fully support 
the syntax that's defined, and for a BCP-47 validator to validate all and 
only tags that are defined in relation to a given snapshot of the 
registry.

But I see no reason why a higher-level protocol cannot restrict the set of 
tags it accepts and still be considered "an application of BCP 47". I 
wouldn't be surprised if was actually common for people to create 
higher-level protocols that normatively reference BCP 47 but that define 
some subset of tags that are permitted. What is the problem in saying 
those are "applications of BCP 47"? It would certainly be getting applied 
in those cases.

Peter

-----Original Message-----
From: Randy Presuhn [mailto:[email-masked]]
Sent: Friday, June 15, 2007 11:16 AM
To: LTRU Working Group
Subject: Re: [Ltru] RE: ISO 639-2 decision: "mis"

Hi -

...
> supported in that application context.

As such, I'd argue that these constraints mean it isn't an application of 
BCP 47.

These things might be called tags, and they might even be a subset of the
set of strings that are well-formed, valid BCP 47 tags, but this is *NOT* 
an
application of BCP 47, and, consequently, is not our problem.

Randy

---

From: Peter Constable <[email-masked]>
Date: Fri, 15 Jun 2007 13:22:04 -0700

Let's not get too hung up on MARC as an example. It serves as an example of something that limits the set of tags, not specifically as an application of BCP 47, which it does not claim to be (and, as Karen points out, is not).

Suppose a corpus that has content in a number of languages with primary focus on particular languages, but in which the criteria for inclusion of records is based not solely on language, with the result that some records are not in one of the languages of primary focus. Suppose that content is recorded in an XML language, with language declared using xml:lang. Because XML is used, that would be an application of BCP 47 according to my understanding of conventional uses of "application of". The policy for that corpus, however, might well be to use 'mis' for any records not in some specified list of languages. IMO, that particular fact should not nullify describing that system as "an application of BCP 47".

Peter

From: [email-masked] [mailto:[email-masked]]
Sent: Friday, June 15, 2007 12:08 PM
To: Peter Constable
Cc: LTRU Working Group; Randy Presuhn
Subject: RE: [Ltru] RE: ISO 639-2 decision: "mis"

But MARC uses the three-character 639-2 tags for the most common languages, right? If so, that is not a restriction, it's a violation.

Karen Broome

06/15/2007 12:04 PM

To

Randy Presuhn <[email-masked]>, LTRU Working Group <[email-masked]>

cc

Subject

Peter

-----Original Message-----
From: Randy Presuhn [mailto:[email-masked]]
Sent: Friday, June 15, 2007 11:16 AM
To: LTRU Working Group
Subject: Re: [Ltru] RE: ISO 639-2 decision: "mis"

Hi -

...

Randy

---

From: [email-masked]
Date: Fri, 15 Jun 2007 14:10:20 -0700

I've restricted our internal implementations not to use script tags with 
audio languages, though that is not currently part of the published work 
and when it is, it will be a Should not a Shall. 

Also, in a standard outside of Sony Pictures, the length of the tag is 
restricted by the implementation. None of these provisions make for 
invalid tags according to BCP 47, so I consider these to be 
application-specific restrictions within a BCP 47-compliant framework. So 
yes, there are good reasons to place certain restrictions on BCP 47 in 
context -- I didn't think there was any controversy around this.

Regards,

Karen Broome
Metadata Systems Designer
Sony Pictures Entertainment
310.244.4384

06/15/2007 01:22 PM

To
LTRU Working Group <[email-masked]>
cc

Subject

Let’s not get too hung up on MARC as an example. It serves as an example 
of something that limits the set of tags, not specifically as an 
application of BCP 47, which it does not claim to be (and, as Karen points 
out, is not).
 
Suppose a corpus that has content in a number of languages with primary 
focus on particular languages, but in which the criteria for inclusion of 
records is based not solely on language, with the result that some records 
are not in one of the languages of primary focus. Suppose that content is 
recorded in an XML language, with language declared using xml:lang. 
Because XML is used, that would be an application of BCP 47 according to 
my understanding of conventional uses of “application of”. The policy for 
that corpus, however, might well be to use ‘mis’ for any records not in 
some specified list of languages. IMO, that particular fact should not 
nullify describing that system as “an application of BCP 47”.
 
Peter

Karen Broome 

To

Subject

Peter

-----Original Message-----
From: Randy Presuhn [mailto:[email-masked]]
Sent: Friday, June 15, 2007 11:16 AM
To: LTRU Working Group
Subject: Re: [Ltru] RE: ISO 639-2 decision: "mis"

Hi -

...

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 15 Jun 2007 14:17:10 -0700

Hi -

As a technical contributor....

> Sent: Friday, June 15, 2007 1:22 PM
...
> focus on particular languages, but in which the criteria for inclusion of records
> is based not solely on language, with the result that some records are not in
> one of the languages of primary focus. Suppose that content is recorded in an XML
> language, with language declared using xml:lang. Because XML is used, that
> would be an application of BCP 47 according to my understanding of conventional
> uses of "application of".

So far so good.

> The policy for that corpus, however, might well be to use 'mis' for any records
> not in some specified list of languages.  IMO, that particular fact should not
> nullify describing that system as "an application of BCP 47".

This is where we differ.  While the syntax is still ok, the semantic is changed
past any hope of interoperability.  (Rather like that classic paper on Recoverability
Under Deletion.  :-)  It would rather be like using "sw" to mean "Swiss German"
rather than Swahili.  I'd consider that an abuse, rather than an application, of the BCP.

Randy

---

From: Peter Constable <[email-masked]>
Date: Fri, 15 Jun 2007 14:39:08 -0700

> past any hope of interoperability.

I don't get that at all. Every user of that system, which may well be public, would understand by the policy for that system, what to expect. The semantics of 'mis' would be clearly documented both in ISO 639 and in the LSTR, and the usage would be consistent with the documented semantics. If any data is exported outside that system, then there would be some semantic loss because the context is lost, but that is part of the documented contract around 'mis' -- and why users need to understand its shortcomings, so also why users would probably be warned against exporting data.

The only issue here is the inherent instability and shortcomings of 'mis'. I don't see any abuse of BCP 47. There is nothing whatsoever like using "sw" to mean 'Swiss German'.

Peter

---

From: "Kent Karlsson" <[email-masked]>
Date: Fri, 15 Jun 2007 23:49:28 +0200

...
> This is the *sole* case where "mis" would be appropriate,

With the given recent change for ISO 639 w.r.t. 'mis', 'mis'
is not applicable, when using BCP 47 tags, in such cases
either, due to inherent instability of 'mis' that result from
the recent change. 'und' is still applicable (but has admittedly
a different intension).

> I'd know which was which.

Agreed.

	/kent k

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 15 Jun 2007 14:56:20 -0700

Hi -

> To: "Randy Presuhn" <[email-masked]>; "LTRU Working Group" <[email-masked]>
> Sent: Friday, June 15, 2007 2:39 PM
...
> I don't get that at all. Every user of that system, which may well be public,
> would understand by the policy for that system, what to expect. The semantics
> of 'mis' would be clearly documented both in ISO 639 and in the LSTR, and
> the usage would be consistent with the documented semantics. If any data
> is exported outside that system, then there would be some semantic loss
> because the context is lost,

You've just explained why such a usage is not interoperable.

> but that is part of the documented contract around 'mis'

Which "documented contract"?

> -- and why users need to understand its shortcomings, so also why users
> would probably be warned against exporting data.

This is making the case for using "MUST NOT" rather than "SHOULD NOT".

> The only issue here is the inherent instability and shortcomings of 'mis'.
> I don't see any abuse of BCP 47. There is nothing whatsoever like using
> "sw" to mean 'Swiss German'.

Yes, it is in that permitting the tag to be used for languages for which tags
already exist is in direct contradiction to its description.

Randy

---

From: "Debbie Garside" <[email-masked]>
Date: Fri, 15 Jun 2007 23:20:22 +0100

Just a slight comment:

Addison wrote:

>It SHOULD NOT be used except when 
> other means of identifying the language are not available. 

Double nots are not good :-)

I propose:

"It should only be used when other means of identifying the language are
unavailable."

Debbie

> From: Addison Phillips [mailto:[email-masked]] 
> Sent: 15 June 2007 19:19
> To: Randy Presuhn
> Cc: LTRU Working Group
> > I disagree with the phrase "or when the range of language tags 
> > supported in a given application are constrained."  If the 
> application 
> > knows what language it is, it should use the correct tag, if one 
> > exists.  If the application doesn't know what language is 
> in use, "und" would be correct.
> 
> That's nice in theory. But some applications use a subset of 
> tags (I used the MARC21 example in the text) and then 
> transmit them through another system (where the larger range 
> of RFC 4646 is available). Those systems only know that the 
> content is 'mis' (because it is tagged that
> We don't define what subtags applications are capable of 
> supporting or required to support elsewhere. We should focus 
> on providing appropriate guidance (which is not to use the 
> subtag). I note that Peter's use of MAY doesn't reflect the 
> current draft.
> 
> The 'mis' (Uncoded) primary language subtag is used to 
> identify linguistic content whose language is known but 
> cannot otherwise be identified. It is intended for use when 
> the range of language tags is constrained or for languages 
> not otherwise categorized. It SHOULD NOT be used except when 
> other means of identifying the language are not available. 
> subtag might be used by this application for languages not 
> included in that set.
> --
> 
> Comments?
> 
> Addison
> 
> --
> Addison Phillips

---

From: Peter Constable <[email-masked]>
Date: Fri, 15 Jun 2007 15:33:43 -0700

> Which "documented contract"?

I expect the ISO 639 RAs will be providing a usage note wrt 'mis' explaining that its extensional semantic must be defined in relation to an application context. This was one of the action items listed if the proposal to change the name were approved. I will follow up with the RAs on getting that done.

> This is making the case for using "MUST NOT" rather than "SHOULD NOT".

IMO, this is in the same line as people using a tag like "en-Latn": we say "the script subtag SHOULD be omitted" in such cases, but not that they MUST NOT include it in such cases. We expect users to make informed choices; we're not going to police them on certain things they *could* do but that wouldn't be wise to do in general.

> already exist is in direct contradiction to its description.

The issue here is evaluating "languages for which tags already exist". You're assuming that every application of BCP 47 must determine that based on the complete inventory of the LSTR. I'm assuming that a higher-level protocol can restrict the contents of the LSTR it permits and that it can also determine the extension of 'mis' on the basis of its restricted set. I see these as two options we can permit; I don't see one of these excluded a priori. It's a choice we should discuss and probably clarify in documenting 'mis' (unless we decide to deprecate it).

Peter

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 15 Jun 2007 16:23:37 -0700

Hi -

> Sent: Friday, June 15, 2007 3:33 PM
...
> The issue here is evaluating "languages for which tags already exist". 
> You're assuming that every application of BCP 47 must determine that
> based on the complete inventory of the LSTR.

Yes.  If an application lacks the knowledge to make that determination, 
"und" would be the correct choice.

> I'm assuming that a higher-level protocol can restrict the contents of the
> LSTR it permits

As such, it would no longer support BCP 47.
At a minimum, I would argue that a protocol that claims to support BCP 47
must be able to carry all valid tags, and should probably be able to carry
all well-formed tags.  Anything less is merely "based on" BCP 47.

> and that it can also determine the extension of 'mis' on the basis of its
> restricted set.

This is simply not interoperable, as you've already explained.

> I see these as two options we can permit; I don't see one of these
> excluded a priori. It's a choice we should discuss and probably clarify
> in documenting 'mis' (unless we decide to deprecate it).

If someone decides a subset of BCP 47 is appropriate to their needs,
that is fine, just as long as they don't claim to sully support BCP 47.
Taken to absurdity, if my protocol offers a choice limited to "en" and "mis", 
should I be able to claim to support BCP 47?  In any case, I'd really hate
 to have this WG go down the path of defining standardized
"profiles" of BCP 47 unless there were a *huge* benefit to doing so.

Just like the "adapted subset" of ASN.1 used in SMI MIB modules, at
some point one has to admit that the subsetting and adaptation can result
in a quite different beast from what one might have started with.

Randy

---

From: Peter Constable <[email-masked]>
Date: Fri, 15 Jun 2007 17:21:35 -0700

> This is simply not interoperable, as you've already explained.

Even treated as you proposed, 'mis' is inherently unstable wrt extension (and always has been). That is part of the contract taken on by anybody who chooses to use it with any profile of language tags that is open to change. You say that's not interoperable; I say it's interoperable given appropriately low expectations of the usefulness of 'mis' in interchange.

> In any case, I'd really hate

Absolute agreement. I have never suggested we should define standardized "profiles". If a consuming protocol wishes to do so, that's entirely up to them -- we certainly should not do it for them.

Peter

---

From: [email-masked]
Date: Fri, 15 Jun 2007 17:20:13 -0700

> that is fine, just as long as they don't claim to sully support BCP 47.

There's a difference between "fully supporting" BCP 47 and being compliant 
with BCP 47. I don't think there's any organization that has a real-life 
use for fully supporting BCP 47. These codes are frequently used in 
databases; if a developer limits the size of that field such that the most 
granular private use tag imaginable does not fit the space allocated, you 
consider that to an implementation "based on" BCP 47 even if all tags in 
the system are valid?

I don't think what you suggest is a useful or realistic way of looking at 
this. If I only need en, fr, de, and it, I think that's fully compliant 
with BCP 47. Must I support gsw-Hant-BV-x-chicago-mwbstr14-spoken to claim 
to be aligned with BCP 47? 

Regards,

Karen Broome

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 15 Jun 2007 20:17:16 -0700

Hi -

> From: <[email-masked]>
> Sent: Friday, June 15, 2007 5:20 PM
...
> consider that to an implementation "based on" BCP 47 even if all tags in 

RFC 4646 says support for a maximum of at least 33 characters is a MUST,
42 characters is a SHOULD.  Anything less is not BCP 47.

> to be aligned with BCP 47? 

If someone designs a protocol, and it can only handle the language tags
en, fr, de, and it, that protocol, in my judgement, would not be compliant
with BCP 47.  This doesn't say anything about the tags that a particular
application using that protocol might *generate*.  I'm just saying that it
must be able to cope with receiving valid tags of up to 33 characters in
length.

I'm urging folks to think about where this specification sits in the "food chain".
It specifies an element that is used by other protocols.  If other protocols
can claim to use this specification, without actually being able to carry the
full range of values, there will be little value to having this specification or to
claiming to support it.  It'd be like claiming to support ASCII, but not permitting
use of the letter "Z".  I fully recognize that for good historical reasons there
will be protocols that are restricted to a subset.  However, I'd strongly urge
against claiming that such protocols "supported" or were "aligned with",
much less "conformant with" BCP 47.

I think part of the disconnect is that some have been thinking about databases,
rather than protocols.

Randy

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 16 Jun 2007 08:37:24 -0700

I agree strongly with Peter. Let's suppose I have a SOAP service using the
BCP 47 protocol for communicating UI language preferences. My application
doesn't support (surprise) all possible BCP 47 language subtags, and
communicates an error back to the requester indicating when I don't accept a
particular language. Such action doesn't make me uncompliant.

The key for compliance is that if I say I'm interpreting a field according
to BCP 47,

   - I do not accept anything that isn't BCP 47 compliant,
   - Anything that I accept has to be interpreted according to BCP 47
   semantics.

There are other kinds of reasonable claims that I can make. I can say, for
example, that I interpret a field according to BCP 47, but with the addition
of 4 exceptions for backwards compatibility with previous versions of my
software.

Mark

On 6/15/07, Peter Constable <[email-masked]> wrote:
> only tags that are defined in relation to a given snapshot of the registry.
>
> some subset of tags that are permitted. What is the problem in saying those
> are "applications of BCP 47"? It would certainly be getting applied in those
> cases.
>
>
> Peter
>
>

-- 
Mark

---

From: "Jukka K. Korpela" <[email-masked]>
Date: Sat, 16 Jun 2007 19:39:34 +0300 (EEST)

> "It should only be used when other means of identifying the language are

The meaning of "mis" does not quite fall into the scope of the meaning of 
"to identify", the way I see it. The language is not identified, but some 
indirect information about language is given.

Perhaps "indicating" could be used instead of "identifying", but the 
following formulation might be better:

"It should only be used when more specific means of giving 
information about the language are unavailable."

-- 
Jukka "Yucca" Korpela, http://www.cs.tut.fi/~jkorpela/

---

From: Martin Duerst <[email-masked]>
Date: Sun, 17 Jun 2007 16:40:18 +0900

At 09:21 07/06/16, Peter Constable wrote:
>Even treated as you proposed, 'mis' is inherently unstable wrt extension 
>(and always has been). That is part of the contract taken on by anybody who 
>chooses to use it with any profile of language tags that is open to change. 
>You say that's not interoperable; I say it's interoperable given 
>appropriately low expectations of the usefulness of 'mis' in interchange.

I think I have to side with Peter here. We inherently know that
no tag at all and things such as 'und' and 'mis' have a rather low
value. On top of that, we have the problem of wrong tagging in the
first place. I'd vastly prefer a repository that tagged documents
from 200 languages *correctly* and used 'mis' or so in the rare case
it got something in a language that the system wasn't envisioned to
handle, rather than a system that tried to tag everything with a
specific code but got half of that wrong.

The interoperability is much more on the level of correctly
using the available tags, rather than on the details of the use
of 'mis'.

Also, while it's easy to write 'don't use "mis" unless there is
no subtag for the language in question', it's very difficult to
test.

>Absolute agreement. I have never suggested we should define standardized 
>"profiles". If a consuming protocol wishes to do so, that's entirely up to 
>them -- we certainly should not do it for them.

I agree.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Sun, 17 Jun 2007 16:40:18 +0900

I agree.

Regards,    Martin.

---

From: "Kent Karlsson" <[email-masked]>
Date: Sun, 17 Jun 2007 20:40:20 +0200

Martin Dürst wrote:
> I'm a bit confused here. We want to try to give absoulte stability
> guarantees. But then we recommend 'und' (wrong to start with, because
> we know what language it is, so calling it undetermined is lying)
> or a private tag (inherently unstable and non-interoperable).
> Why don't we just accept that 'mis' is unstable (no disagreement
> here, of course), but instead of saying "invalid at any point without
> any warning -- and hence incompatible with the stability goals of BCP 47",
> warn the receiver that 'mis' may refer to a language that in the meantime
> has been coded, and change our language, e.g. 'invalid' -> 'suboptimal'.

That was the *very thing* that was changed by the recent change to 'mis'.
(And there was a real change!) What used to be 'suboptimal' is now
'invalid' w.r.t. the use of 'mis'.

'mis' used to be interpretable in a stable way, but that is no longer
possible given the change. (Which is why I've been arguing against this
change all along.)

	/kent k