Subject: [Ltru] Macrolanguage usage

From: "Mark Davis" <[email-masked]>
Date: Thu, 15 May 2008 08:29:03 -0700

Peter Constable and I were at the Unicode Technical Committee meeting, and
had a chance to talk about macrolanguages. One of the key points is to
provide implementers with enough guidance as to what they should do, while
not precluding reasonable alternatives. Here's what we were thinking about
(not in formal language, but the points we wanted to make).

===

Formally, a macrolanguage identifier could be used to tag or look up content
in any encompassed language; alternately, specific, individual-language
identifiers could be used to tag or look up content in those languages. In
consideration of these alternatives, the following provides guidance for how
to maintain backwards compatibility while giving the ability to clearly tag
and lookup content.

1. Implementations generally should tag and/or lookup content with the
specific language where possible (this is the general recommendation with
language tags). In the case of macrolanguages, this means that Cantonese
should be tagged and looked up with "yue", Hakka with "hak", Tajiki Arabic
with "abh", Plains Cree with "crk", and so on.

2. An exception to this general recommendation may apply in the case of
macrolanguages with predominant forms, listed in Table 8. For backwards
compatibility in those cases:

   - an implementation could tag and/or lookup content in the predominant
   language either with the macrolanguage or the encompassed language. (eg
   either "ar" or "arb" for Standard Arabic).
   - an implementation could make a distinction between these in lookup, or
   could return the same content. That is, lookup for "zh-SG" and "cmn-SG" may
   return the same content, or may return different content if the
   implementation needs to make a distinction for some purpose.
   - where content written in an encompassed language is also understandable
   in the predominant language (that being a distinct language encompassed by
   the same macrolanguage), the content could also be tagged with the
   macrolanguage identifier. Thus if a Cantonese passage is understandable if
   read as Mandarin, it could also be tagged with "zh", or where a Tajiki
   Arabic passage is also understandable in Standard Arabic it could be tagged
   with "ar".

3. Another exception to this general recommendation applies in the case of
applications that have limitations that exclude the identifiers for
encompassed, individual languages of a macrolanguage. For example, some
content cataloguing systems limit language identifiers to those in ISO
639-2; as a result, they may support a macrolanguage identifier but not the
identifiers for the encompassed languages of that macrolanguage.

Mark and Peter

--
Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Shawn Steele <[email-masked]>
Date: Thu, 15 May 2008 13:09:25 -0700

Sounds reasonable to me, but it seems like RFC4647 would need updated to handle the 2nd bullet.

-          Shawn

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Thursday, May 15, 2008 8:29 AM
To: LTRU Working Group
Subject: [Ltru] Macrolanguage usage

Peter Constable and I were at the Unicode Technical Committee meeting, and had a chance to talk about macrolanguages. One of the key points is to provide implementers with enough guidance as to what they should do, while not precluding reasonable alternatives. Here's what we were thinking about (not in formal language, but the points we wanted to make).

===

Formally, a macrolanguage identifier could be used to tag or look up content in any encompassed language; alternately, specific, individual-language identifiers could be used to tag or look up content in those languages. In consideration of these alternatives, the following provides guidance for how to maintain backwards compatibility while giving the ability to clearly tag and lookup content.

1. Implementations generally should tag and/or lookup content with the specific language where possible (this is the general recommendation with language tags). In the case of macrolanguages, this means that Cantonese should be tagged and looked up with "yue", Hakka with "hak", Tajiki Arabic with "abh", Plains Cree with "crk", and so on.

2. An exception to this general recommendation may apply in the case of macrolanguages with predominant forms, listed in Table 8. For backwards compatibility in those cases:

 *   an implementation could tag and/or lookup content in the predominant language either with the macrolanguage or the encompassed language. (eg either "ar" or "arb" for Standard Arabic).
 *   an implementation could make a distinction between these in lookup, or could return the same content. That is, lookup for "zh-SG" and "cmn-SG" may return the same content, or may return different content if the implementation needs to make a distinction for some purpose.

 *   where content written in an encompassed language is also understandable in the predominant language (that being a distinct language encompassed by the same macrolanguage), the content could also be tagged with the macrolanguage identifier. Thus if a Cantonese passage is understandable if read as Mandarin, it could also be tagged with "zh", or where a Tajiki Arabic passage is also understandable in Standard Arabic it could be tagged with "ar".
3. Another exception to this general recommendation applies in the case of applications that have limitations that exclude the identifiers for encompassed, individual languages of a macrolanguage. For example, some content cataloguing systems limit language identifiers to those in ISO 639-2; as a result, they may support a macrolanguage identifier but not the identifiers for the encompassed languages of that macrolanguage.

Mark and Peter

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 15 May 2008 14:00:41 -0700

Perhaps it does, but I don't see why it would need to. Could you explain a
bit more?

Mark

On Thu, May 15, 2008 at 1:09 PM, Shawn Steele <[email-masked]>
wrote:

> *From:* [email-masked] [mailto:[email-masked]] *On Behalf Of
> *Mark Davis
> *Sent:* Thursday, May 15, 2008 8:29 AM
> *To:* LTRU Working Group
> *Subject:* [Ltru] Macrolanguage usage
>
>
>
> Formally, a macrolanguage identifier could be used to tag or look up
> content in any encompassed language; alternately, specific,
> individual-language identifiers could be used to tag or look up content in
> those languages. In consideration of these alternatives, the following
> provides guidance for how to maintain backwards compatibility while giving
> the ability to clearly tag and lookup content.
>
>    - an implementation could make a distinction between these in lookup,
>    or could return the same content. That is, lookup for "zh-SG" and "cmn-SG"
>    may return the same content, or may return different content if the
>    - where content written in an encompassed language is also
>    understandable in the predominant language (that being a distinct language
>    encompassed by the same macrolanguage), the content could also be tagged
>    with the macrolanguage identifier. Thus if a Cantonese passage is
>    understandable if read as Mandarin, it could also be tagged with "zh", or
>    where a Tajiki Arabic passage is also understandable in Standard Arabic it
>    could be tagged with "ar".
>

---

From: Shawn Steele <[email-masked]>
Date: Thu, 15 May 2008 14:26:23 -0700

Its not clear to me where 4647 says that zh-SG and cmn-SG may be treated as equal (or may return different content).  On the contrary, it seems that applications needn’t have any additional information.

“Applications, protocols, and specifications are not required to validate or understand any of the semantics of the language tags or ranges or of the subtags in them”

The canonicalizing text also discusses various forms, including zh-CN vs zh-Hans, but doesn’t mention this cmn/zh variant.  It also mentions looking in the registry to recognize & canonicalize grandfathered tags, but neither zh nor cmn are grandfathered.

It could be done with lists, but the idea is that a list isn’t necessary for correct filtering/matching/lookup.  It also mentioned the remove-from-right behavior, but not its implications WRT zh/cmn.

I may have overlooked where this is covered in 4647, but it seems to me like zh/cmn type matching isn’t already covered by 4647.  Obviously it should be an optional thing, not a must thing, but I don’t seem language permitting this sort of matching.

- Shawn

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Thursday, May 15, 2008 2:01 PM
To: Shawn Steele
Cc: LTRU Working Group
Subject: Re: [Ltru] Macrolanguage usage

Perhaps it does, but I don't see why it would need to. Could you explain a bit more?

Mark
On Thu, May 15, 2008 at 1:09 PM, Shawn Steele <[email-masked]<mailto:[email-masked]>> wrote:

-          Shawn

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Mark Davis
Sent: Thursday, May 15, 2008 8:29 AM
To: LTRU Working Group
Subject: [Ltru] Macrolanguage usage

===

Mark and Peter

--
Mark

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 15 May 2008 14:31:28 -0700

Ah, now I understand. We'd decided some time ago to try to put the
discussion of the effects of macro/micro languages into 4646 rather than
4647, just to keep from having to open it up.

Mark

On Thu, May 15, 2008 at 2:26 PM, Shawn Steele <[email-masked]>
wrote:

>  Its not clear to me where 4647 says that zh-SG and cmn-SG may be treated
> as equal (or may return different content).  On the contrary, it seems that
> applications needn't have any additional information.
>
>
>
> "Applications, protocols, and specifications are not required to validate
> or understand any of the semantics of the language tags or ranges or of the
> subtags in them"
>
>
>
> The canonicalizing text also discusses various forms, including zh-CN vs
> zh-Hans, but doesn't mention this cmn/zh variant.  It also mentions looking
> in the registry to recognize & canonicalize grandfathered tags, but neither
> zh nor cmn are grandfathered.
>
>
>
> It could be done with lists, but the idea is that a list isn't necessary
> for correct filtering/matching/lookup.  It also mentioned the
> remove-from-right behavior, but not its implications WRT zh/cmn.
>
>
>
> I may have overlooked where this is covered in 4647, but it seems to me
> like zh/cmn type matching isn't already covered by 4647.  Obviously it
> should be an optional thing, not a must thing, but I don't seem language
> permitting this sort of matching.
>
>
>
> - Shawn
>
>
>
>
>
>
>
> *From:* [email-masked] [mailto:[email-masked]] *On
> Behalf Of *Mark Davis
> *Sent:* Thursday, May 15, 2008 2:01 PM
> *To:* Shawn Steele
> *Cc:* LTRU Working Group
> *Subject:* Re: [Ltru] Macrolanguage usage
>
>
>

---

From: Shawn Steele <[email-masked]>
Date: Thu, 15 May 2008 14:50:18 -0700

I can understand that desire, however I’m not sure the full implications of lookup can be discussed in 4646bis.  You’d have to refer to 4647 to understand the comments and then 4646bis would have to make exceptions/additions to the 4647 behavior.

Perhaps there could be a limited effort to add one section to address this in 4647 rather than rewriting the whole thing?

I’m also a bit concerned that a harried developer that thinks “4646bis is names, I know how those work, 4647 is lookup, that’s what I’m implementing, I’ll read that” could easily miss the possible considerations of zh/cmn and whether or not his app needs to be concerned with that.

- Shawn

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Thursday, May 15, 2008 2:31 PM
To: Shawn Steele
Cc: LTRU Working Group
Subject: Re: [Ltru] Macrolanguage usage

Ah, now I understand. We'd decided some time ago to try to put the discussion of the effects of macro/micro languages into 4646 rather than 4647, just to keep from having to open it up.

Mark
On Thu, May 15, 2008 at 2:26 PM, Shawn Steele <[email-masked]<mailto:[email-masked]>> wrote:

Its not clear to me where 4647 says that zh-SG and cmn-SG may be treated as equal (or may return different content).  On the contrary, it seems that applications needn't have any additional information.

"Applications, protocols, and specifications are not required to validate or understand any of the semantics of the language tags or ranges or of the subtags in them"

The canonicalizing text also discusses various forms, including zh-CN vs zh-Hans, but doesn't mention this cmn/zh variant.  It also mentions looking in the registry to recognize & canonicalize grandfathered tags, but neither zh nor cmn are grandfathered.

It could be done with lists, but the idea is that a list isn't necessary for correct filtering/matching/lookup.  It also mentioned the remove-from-right behavior, but not its implications WRT zh/cmn.

I may have overlooked where this is covered in 4647, but it seems to me like zh/cmn type matching isn't already covered by 4647.  Obviously it should be an optional thing, not a must thing, but I don't seem language permitting this sort of matching.

- Shawn

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Mark Davis
Sent: Thursday, May 15, 2008 2:01 PM
To: Shawn Steele
Cc: LTRU Working Group
Subject: Re: [Ltru] Macrolanguage usage

Mark

-          Shawn

===

Mark and Peter

--
Mark

--
Mark

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 15 May 2008 16:39:17 -0700

Hi -

As a technical contributor...

> From: "Shawn Steele" <[email-masked]>
> To: "Mark Davis" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Thursday, May 15, 2008 2:50 PM
> I can understand that desire, however I’m not sure the full
> implications of lookup can be discussed in 4646bis.  You’d
> have to refer to 4647 to understand the comments and then
> 4646bis would have to make exceptions/additions to the 4647 behavior.
>
> Perhaps there could be a limited effort to add one section to address
> this in 4647 rather than rewriting the whole thing?
>
> I’m also a bit concerned that a harried developer that thinks
> “4646bis is names, I know how those work, 4647 is lookup, that’s
> what I’m implementing, I’ll read that” could easily miss the
> possible considerations of zh/cmn and whether or not his app
> needs to be concerned with that.
...

I agree.  Though these RFCs need to be taken together as a BCP,
there's good reason to maintain modularity.  Consequently, I hate
to see matching algorithm details "leaking" into 4646bis.
Architecturally, it smells like something is broken.

Randy

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 15 May 2008 17:11:55 -0700

> Architecturally, it smells like something is broken.

I agree that we might need to do some revision in 4647. However, I think that such work would be limited in scope (this is one area where extlang has *more* impact). Certainly my implementation experience suggests it. There is one very permissive "MAY" in the current 4646bis [1] that users might miss and perhaps some additional text is needed about selecting ranges. This might be suitable as an erratum, although IETF policy may not permit that. We could also break out some of the very detailed information we've had on this list into an Informational RFC that both 4646bis and 4647 (with erratum) could reference. That strikes me as potentially an appropriate mechanism for helping implementers without polluting the RFCs with extensive information about specific languages.

Addison

[1] The quote I have in mind is:

--
Applications MAY use macrolanguage information to improve matching or language negotiation. For example, the information that 'sr' (Serbian) and 'hr' (Croatian) share a macrolanguage expresses a closer relation between those languages than between, say, 'sr' (Serbian) and 'ma' (Macedonian). While it is valid to use either the subtag of the encompassed language or of the macrolanguage to form a language tag, many matching applications will not be aware of the relationship between the languages.
--

http://www.inter-locale.com/ID/editor/draft-ietf-ltru-4646bis-14a.html#macrolanguages
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru