Subject: Re: [Ietf-languages] language variant subtag "pinyin" modification request

From: "Doug Ewell" <[email-masked]>
Date: Tue, 20 Nov 2018 14:06:52 -0700

Phake Nick wrote:

> Type: variant
> Subtag: pinyin
> Description: Pinyin romanization
> Prefix: zh-Latn
> Prefix: zh
> Prefix: bo-Latn

I'm not entirely sure of the benefits of having both "zh" and "zh-Latn"
as Prefix fields for the same variant, since the more general Prefix
generally encompasses the more specific. But since it's an example in
Section 2.2.5, and we have some cases of this in the Registry ('1994',
'ekavsk', 'ijekavsk'), I won't argue, except perhaps to wonder why this
request isn't made for ''bo' as well.

> Suppress-Script:
> Macrolanguage:
> Comments:

As an administrative note, it's not necessary to include these blank
fields in a proposal. The form in Section 3.5 merely shows all the
fields that /could/ be included if necessary. I realize the document
doesn't make this very clear.

> The request is a request to modify an existing entry, with the aim of
> adding an additional prefix for the variant subtag "pinyin". The
> proposed "zh-pinyin" tag would have same effect as "zh-Latn-pinyin"
> but it is shorter and is currently in use in multiple places to
> reflect Chinese content being written in Pinyin romanization,

There are pros and cons to this. Suggesting "zh-pinyin" instead of, or
in addition to, "zh-Latn-pinyin" does make the tag shorter (remember
that Prefix is only a suggestion, not a matter of "permit" vs.
"prohibit," although not everyone understands this). Since Pinyin is a
romanization scheme, it may seem a bit redundant to suggest 'Latn' as
well. Many tags do assume a certain amount of knowledge on the user's
part (e.g. what exactly distinguishes "en-GB" from en-US"?); assuming
the user knows that Pinyin is Latin script doesn't seem far-fetched.

On the other hand, "zh-Latn-pinyin" would correctly match a request for
"zh-Latn" using the truncate-from-right approach, while "zh-pinyin"
would not. It would be identifiable to any human reader as "Chinese in
Latin script," which is what it is. (Question: why is this request not
also being made for 'wadegile' and 'hepburn' and 'jyutping'?) And Prefix
is only a suggestion, so it isn't really necessary. Plus, there is a
school of thought on this list that redundancy of the type exemplified
by "zh-Latn-pinyin" is desirable.

I will defer to other as to the merits of this request.

> including OpenStreetMap. See
> https://wiki.openstreetmap.org/wiki/Multilingual_names#China

That page makes several false or questionable statements about BCP 47
language tagging, and I might not want to present it as representative
of the users who would benefit from your request. Among other things,
the page states under "Issues" that only valid BCP 47 tags can be used
in OSM, but under "China" it recommends using the invalid "zh_pinyin"
(with underscore) instead of a valid tag, even the currently
non-suggested but valid "zh-pinyin". It also suggests using invalid tags
in the Japan section, where it makes the bizarre statement that "BCP 47
makes no difference between '-' and '_'". Be careful when citing this
project.

--
Doug Ewell | Thornton, CO, US | ewellic.org

---

From: Mark Davis ☕️ <[email-masked]>
Date: Wed, 21 Nov 2018 19:03:46 +0100

Nice discussion, Doug.

I'd add that very few systems support variants, especially for language
matching. So one would expect currently a request for "zh-pinyin" to return
the same as "zh".  With "zh-Latn-pinyin", on the other hand, is likely to
return the same as "zh-Latn" if available, namely pinyin content.

So in practice,  "zh-pinyin" is really best avoided until there is good
support for variants.

Mark

On Tue, Nov 20, 2018 at 10:07 PM Doug Ewell <[email-masked]> wrote:

> _______________________________________________
> Ietf-languages mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-languages
>

---

From: Mark Davis ☕️ <[email-masked]>
Date: Wed, 21 Nov 2018 19:03:46 +0100

Nice discussion, Doug.

Mark

---

From: Hugh Paterson <[email-masked]>
Date: Sat, 24 Nov 2018 12:06:19 -0800

@Doug Ewell <[email-masked]>

I'm interested in some clarification on the statement below:

> It also suggests using invalid tags in the Japan section, where it makes
the bizarre statement that "BCP 47 makes no difference between '-' and
'_'". Be careful when citing this project.

I'm wondering if the source of confusion at the OSM project might be CLDR.
I notice that the documentation at CLDR for suggested tags includes
underscores. See the documentation at [1].

For my own understanding, am I correct to understand that CLDR tags are not
BCP47 tags? or put another way what is the relationship between CLDR tags
(approved ones) and BCP47 tags (approved ones).

[1]:
http://www.unicode.org/cldr/charts/latest/supplemental/likely_subtags.html

all the best,

-- 
*Hugh Paterson III *Innovation Analyst
*Innovation Development & Experimentation*, *SIL International*

*Web*: Contact & CV <http://hughandbecky.us/Hugh-CV/>

---

From: Hugh Paterson <[email-masked]>
Date: Sat, 24 Nov 2018 12:06:19 -0800

[1]:

all the best,

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 24 Nov 2018 16:04:28 -0700

Hugh Paterson wrote:

>> It also suggests using invalid tags in the Japan section, where it
>> makes the bizarre statement that "BCP 47 makes no difference between
>> '-' and '_'". Be careful when citing this project.
>
> I'm wondering if the source of confusion at the OSM project might be
> CLDR. I notice that the documentation at CLDR for suggested tags
> includes underscores. See the documentation at [1].
>
> For my own understanding, am I correct to understand that CLDR tags
> are not BCP47 tags? or put another way what is the relationship
> between CLDR tags (approved ones) and BCP47 tags (approved ones).
>
> [1]: 

CLDR defines its own extensions of the BCP 47 syntax. Here is the 
relevant passage from UTS #35, which defines the syntax for identifiers 
(and much more) in CLDR:

> 3.3 BCP 47 Conformance
>
> Unicode language and locale identifiers inherit the design and the
> repertoire of subtags from [BCP47] Language Tags. There are some
> extensions and restrictions made for the use of the Unicode locale
> identifier in CLDR:
>
> [...]
>
> It allows certain syntax for backwards compatibility (not BCP 47-
> compatible):
>
> The "_" character for field separator characters, as well as the "-"
> used in [BCP47] (however, the canonical form is with "-")

So underscores are not allowed in BCP 47 tags, but they are allowed in 
CLDR identifiers.

As a side note, CLDR "likely subtags" are not necessary the "suggested 
tags" for language tagging. As Section 4.3 of UTS #35 notes, the purpose 
of these is to indicate the most likely language, script, and/or region 
subtag, given one or two of the others. They do not supersede the 
general BCP 47 advice to omit needless subtags.

For example, although likelySubtags.xml defines this:

<likelySubtag from="ja" to="ja_Jpan_JP"/>

there is no reason in normal language tagging to use either 'Jpan' or 
'JP' in this tag. It is just telling you what the most likely script and 
region subtags for Japanese would be.

CLDR is an important user of BCP 47, but its syntax and rules are its 
own.

---

From: Hugh Paterson <[email-masked]>
Date: Sat, 24 Nov 2018 17:45:23 -0800

Thanks for the clarification.

---

From: Mark Davis ☕️ <[email-masked]>
Date: Sun, 25 Nov 2018 14:17:52 +0100

The Unicode locale identifiers support a legacy syntax as well as proper
BCP 47 syntax. There is a defined subset of the Unicode locale identifiers
called *Unicode BCP 47 locale identifiers*, which are all well-formed BCP
47 language tags. That is the target syntax that any new implementation
should use.

The data files on the CLDR site use the legacy syntax for backwards
compatibility. However, the transform to BCP 47 syntax is simple, and
defined in
http://unicode.org/reports/tr35/#Unicode_Locale_Identifier_CLDR_to_BCP_47,
so it is quite straightforward for implementations to read the CLDR data
and just use the BCP47 syntax.

We should probably make that clear in the chart documentation.

Mark

On Sat, Nov 24, 2018 at 9:07 PM Hugh Paterson <[email-masked]> wrote:

> For my own understanding, am I correct to understand that CLDR tags are
> not BCP47 tags? or put another way what is the relationship between CLDR
> tags (approved ones) and BCP47 tags (approved ones).
>
> [1]:

> [snip]

---

From: Mark Davis ☕️ <[email-masked]>
Date: Sun, 25 Nov 2018 14:17:52 +0100

Mark

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 25 Nov 2018 14:02:43 -0700

<flame>

I wish individuals and organizations that want to use BCP 47 language 
tags would read the BCP and bring their questions or confusion to THIS 
list.

This isn't 1996, when authoritative lists of ISO 639 and 3166 code 
elements weren't available on the web, and users of language tags had to 
rely on outdated, error-filled third-party lists.

</flame>

---

From: Phake Nick <[email-masked]>
Date: Mon, 26 Nov 2018 13:35:34 +0800

.... The interesting thing is that, in the OSM wiki url I linked, in the
#Issue section on the top, it have correctly stated that subtags are
separated by hyphens, yet in the #Japan section it have a note saying 'BCP
47 makes no difference between "-" and "_" '.

2018年11月26日 05:03 於 "Doug Ewell" <[email-masked]> 寫道：

<flame>

</flame>

---

From: Mark Davis ☕️ <[email-masked]>
Date: Mon, 26 Nov 2018 08:31:09 +0100

See also: Locale Identifiers <https://goo.gl/kizkrm>

Mark

On Sun, Nov 25, 2018 at 2:17 PM Mark Davis ☕️ <[email-masked]> wrote:

>> makes the bizarre statement that "BCP 47 makes no difference between '-'
>> and '_'". Be careful when citing this project.
>>
>> CLDR. I notice that the documentation at CLDR for suggested tags includes

---

From: Mark Davis ☕️ <[email-masked]>
Date: Mon, 26 Nov 2018 08:31:09 +0100

Mark