Subject: [Ltru] Extended language tags

From: Shawn Steele <[email-masked]>
Date: Thu, 4 Oct 2007 14:33:05 -0700

There's been lots of discussion about the extended language tags, but it doesn't seem like there've been many conclusions.  I've gleaned the following points for the discussion:

        * We (the bigger software ecosystem we, tools, etc) have encouraged use of zh-HK, etc for labeling both Mandarin and Cantonese.  Whatever happens in the future a large number of legacy documents, both Manderin and Cantonese, will be labeled with the existing language tag.

        * Whether or not yue or zh-yue was used, it is a change from the current label (in most cases).  It seems that in nearly all cases this will require a code change.  In particular applications that want to include zh-HK in lists containing zh-cmn-HK or cmn-HK will need extra awareness.

        * I don't think the Breton example applies.  http-accept-lang allows for "br-FR;fr-FR" type fallback, so that is a solution.  If an application independently wanted to make this assumption that's fine by me, but this seems orthogonal to the problem we're trying to solve here.

        * Lots of zh data right now is zh-cmn.  Most of "us" seem to agree that we can't narrow the meaning of zh because it does allow Cantonese.  However if some application wanted to make an assumption that zh == zh-cmn, then that seems up to the application for fallback.

        * Even for the current tags, many of the people in the teleconference seem to extend RFC 4647 in ways that are best for them.  Strict use of 4647 behavior seems rare.  It seems reasonable to me to expect that in the future people may continue to do so and that RFC 4647 and the registry can only provide guidelines.  I don't think that it can solve all problems for all applications, and I'm fine with that.

        * It was proposed that the registry could provide a link from cmn to zh for applications that want this.  My experience has been that the registry is good for providing recommendations, but in practice software is rarely kept current with the actual data currently in the registry.

        * FWIW: some documents are currently tagged zh-yue.

        * RFC 4646 strongly implies that that direction is zh-cmn rather than cmn.  By itself this isn't a big concern to me, but taken with the grandfathered and existing zh-yue data I would think there'd need to be a strong case to change directions.

So from these points, my conclusion is that the zh-cmn form is preferable.

My reasoning is that either cmn by itself or zh-cmn will require code change in nearly all cases, either to include or exclude existing data or fallback rules.   Either was will require a code change, and either way may require knowledge that zh-HK might be interesting if the request is for {Mandarin tag}-HK.  Searching may need to include zh files for {Manderin tag} queries and exclude them for {Cantonese tag} queries, but the actual tag doesn't really change this logic.  Neither variation is likely to work with existing code when the request is for the new name and the data is tagged with the old tag.

The deciding factor for me is that to know that cmn is related to zh I'd have to look in the registry, but zh-cmn contains that information.  Otherwise I don't really see advantages with either method.

- Shawn

SDE
Windows International
Visual Studio .Net
Microsoft

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Andrew Cunningham" <[email-masked]>
Date: Fri, 5 Oct 2007 11:01:35 +1000

Throwing in my two cents worth:

One aspect of language tagging is the preference for the tags to be as
specific as possible. In some cases it is desirable to be less
specific.

An example i mentioned on the teleconference was the case of Dinka. In
ISO-639-2 this is represented by the language code "din".

While ISO-639-3 have the following language codes:

dib  	South Central Dinka
dik 	Southwestern Dinka
dip 	Northeastern Dinka
diw 	Northwestern Dinka
dks 	Southeastern Dinka

If I was applying a langauge tag to a Rek grammar, then i'd use "dik",
for a collection of Ciec folktales I'd use "dib". For a collection of
Bor proverbs I'd use "dks".

To describe the literacy materials and class room materials being
developed in Australia by the Dinka community, I'd use "din". Within
the diaspora and in Australia specifically the literacy and language
teachers, translators and interpreters are discussing a standardized
approach to written Dinka.

The original SPLA/M education policies highlighted Rek as a standard
for written Dinka. What seems to be occurring is an amalgam based on
Rek, but including aspects and vocab from other Dinka dialects. There
will be a locally hosted conference next year to thrash out some of
the issues.

In this context "din": would be the most appropriate way of tagging
the new educational material, while keeping the existing five
iso-639-3 language tags to more accurately describe information and
data written in one of the 20 odd specific dialects.

from the perspective of the user community, an extlang approach would
make more sense, i.e. Rek labeled as "din-dik" makes more sense than
"dik". To the community Rek is a Dinka langauge, Dinak Rek, not a
separate language called Rek.

In this sense extlang reflects the communities understanding of their language.

This is just an observation, i'm neither for or against extlang.

Although from the perspective of web development and how CSS and web
browsers handle psuedo langauge selectors and attribute selectors, I'd
suggest that extlang approach simplifies things for those rare
individuals amongst us that use these selectors.

Andrew
-- 
Andrew Cunningham
State Library of Victoria, Australia

[email-masked]
[email-masked]

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 4 Oct 2007 18:25:27 -0700

>  In this sense extlang reflects the communities understanding of their
language.
=> In this sense macro languages reflects the communities understanding of
their language.

While the issue of synonyms is related to the issue of macro language
registry info vs extlang, they aren't the same.

Mark

-- 
Mark

---

From: "Don Osborn" <[email-masked]>
Date: Thu, 4 Oct 2007 21:39:44 -0400

Just to confirm that what Andrew is saying applies to a number of languages in Africa. I've brought up similar concerns in the past wrt (macro) languages like Fula and Manding (though the latter might be called a cluster), though I believe I addressed them to the ietf-languages list. These are just a few examples.

I haven't been following the recent discussions here as much as I should have, but do believe that if there is a bias in the system to specificity, when specificity is not always the ideal approach, then it will inevitably encounter calls for flexibility later. At this point I can't say how extlang responds to this so can only offer these general comments on the context of some languages.

This is not to say that specificity in language tags is wrong, but that they describe one aspect of a more complex reality. Large degrees of interintelligibility, the sense of speakers regarding the essential unity of a language (based on their experience and the common wisdom in the culture), and the hopes and potentials for standardized versions, all probably mean that less specific tags will become more important as these languages emerge more fully into digital culture and the digital economy.

The language situations are dynamic, and the medium in which we are trying to describe and categorize languages is adding a new dimension to the development of lesser-resourced languages and their variants.

Hope this makes some sense.

Don Osborn
Bisharat.net
PanAfriL10n.org

> -----Original Message-----
> From: Andrew Cunningham [mailto:[email-masked]]
> Sent: Thursday, October 04, 2007 9:02 PM
> To: Shawn Steele
> Cc: [email-masked]
> Subject: Re: [Ltru] Extended language tags
>

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 4 Oct 2007 18:44:52 -0700

Hi -

As a technical contributor...

> From: "Andrew Cunningham" <[email-masked]>
> To: "Shawn Steele" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Thursday, October 04, 2007 6:01 PM
... 
> data written in one of the 20 odd specific dialects.
...

While agreeing with the thrust of this argument, something like "din-common"
would make more sense to me in this example as a preferred way of tagging
this educational material.  I do not think it is desirable for a subtag
to formally convey both the sense "some unspecified variety or varieties of
this language family" and "the most common or standardardized variety".

This would allow such material to be identified as a distinct variety,
while recognizing the commonality that makes such standardization possible.

Randy

---

From: "Andrew Cunningham" <[email-masked]>
Date: Fri, 5 Oct 2007 12:23:31 +1000

Thanks Mark for the correction, my use of terminology can be sloppy at times ;)

-- 
Andrew Cunningham
Language IT support
Dinka Language Institute
Australia
http://home.vicnet.net.au/~andrewc/
http://home.vicnet.net.au/~agamlong/dlia/
http://www.openroad.net.au/languages/african/dinka/

---

From: "Andrew Cunningham" <[email-masked]>
Date: Fri, 5 Oct 2007 12:36:45 +1000

Randy,

On 05/10/2007, Randy Presuhn <[email-masked]> wrote:

> I do not think it is desirable for a subtag

True, and if a standardised Dinka is formalised then an approprian
unspecified variety of Dinka not covered by the current ISO-639-3
codes". I'm not suggesting "din" be used for a standardised or common
Dinka. Just at the moment things are heading in the development of a
standardised Dinka (if that ever actually happens). At which time a
more appropriate subtag may be put forward.

if that makes sense.

-- 
Andrew Cunningham

---

From: "Andrew Cunningham" <[email-masked]>
Date: Fri, 5 Oct 2007 12:40:45 +1000

That should have read (and for some reason didn't):

True, and if a standardised Dinka is formalised then an
appropriate subtag should/may be registered.

But at the moment, the way I'm using "din" is as an
unspecified variety of Dinka not covered by the current
ISO-639-3 codes".

I'm not suggesting "din" be used for a standardised or
common Dinka.

Just saying that at the moment things are heading in
the direction of the development of a standardised
Dinka (if that ever actually happens). At which time
a more appropriate subtag may be put forward.

If that makes sense?

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 4 Oct 2007 19:51:50 -0700

Hi -

> To: <[email-masked]>
> Sent: Thursday, October 04, 2007 7:40 PM
...

We're in violent agreement.  :-)

Randy

---

From: John Cowan <[email-masked]>
Date: Thu, 4 Oct 2007 23:21:51 -0400

Shawn Steele scripsit:

> * I don't think the Breton example applies.  http-accept-lang allows for
> "br-FR;fr-FR" type fallback, so that is a solution.  If an application
> independently wanted to make this assumption that's fine by me, but
> this seems orthogonal to the problem we're trying to solve here.

That works for filtering (which is what Accept-Lang does), but not so
well in practice for lookup.  Normally people don't specify multiple match patterns
in lookup situations, they just say "I speak Breton, what's the best
match you have?"

-- 
You're a brave man! Go and break through the            John Cowan
lines, and remember while you're out there              [email-masked]
risking life and limb through shot and shell,           http://ccil.org/~cowan
we'll be in here thinking what a sucker you are!
        --Rufus T. Firefly

---

From: Addison Phillips <[email-masked]>
Date: Thu, 04 Oct 2007 20:27:31 -0700

> in lookup situations, they just say "I speak Breton, what's the best

Although actual implementations of lookup, as specified, do allow for 
it. My implementations handle this situation well and it turns out to be 
pretty useful to use a language priority list for lookup.

Traditional locale systems, however, work as John notes. I am hopeful 
that they will adapt.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Don Osborn" <[email-masked]>
Date: Thu, 4 Oct 2007 23:42:30 -0400

...
...

Which seems to be saying that in this case there is need for an appropriate level of imprecision in the tagging, and that this is met by the macrolanguage code alone (which is in 639-3 also, as I read it). I'm thinking that this is not so much the exception, but a situation that can arise with many other (macro)languages where there is no formal standard.

I can see the point of having a tag to describe a standardized form, if such a formal standardization is arrived at (though I'm not sure where discussions are re ar and Standard Arabic). But what if usages just keep crossing or blurring the lines among "languages" within the "macrolanguage"?  This is not just limited to Dinka - there were a few publications in Fulfulde a while back (one title was Janng'en Fulfulde as I recall) that deliberately tried to address a range of speakers (to the point of having footnotes to clarify usage in specific dialects where these were markedly different). Nowadays, anyone could try that on a website, blog or Wikipedia. In such a case wouldn't the ISO 639-1/2 code (ff/ful) be appropriate?

Maybe there's no argument here, but I wanted to underscore this perspective before letting it be.

Don

---

From: John Cowan <[email-masked]>
Date: Fri, 5 Oct 2007 00:00:36 -0400

Don Osborn scripsit:

> I can see the point of having a tag to describe a standardized form,
> if such a formal standardization is arrived at (though I'm not sure
> where discussions are re ar and Standard Arabic). 

Arabic's like Chinese: there's a prestige variety, which has its own
639-3 code, and then there are the other varieties, which have 639-3
codes too.  Each set is encompassed by a macrolanguage that is in both
639-2 and 639-3.

Most textual documents are in the prestige variety (as an uncontroversial
consequence of the prestige variety being the only one normally used
for writing at all), and are tagged with the macrolanguage code (as an
uncontroversial consequence of it being either the only code available
or the only code that most people know about).  Audio/video content may
well be another story, but most of it is neither manually tagged nor
(as yet) automatically taggable.

Macrolanguages like Quechua or Zapotec are another story: there is no
prestige variety particularly.  The African cases sound like they are
moving from the latter group toward the former, with the appearance of
prestige varieties at least in the cities or the diaspora.

-- 
Possession is said to be nine points of the law,                John Cowan
but that's not saying how many points the law might have.       [email-masked]
        --Thomas A. Cowan (law professor and my father)

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 4 Oct 2007 21:19:50 -0700

Hi -

> From: "Don Osborn" <[email-masked]>
> To: "'Andrew Cunningham'" <[email-masked]>; <[email-masked]>
> Sent: Thursday, October 04, 2007 8:42 PM
> Subject: RE: [Ltru] Extended language tags
...
> Which seems to be saying that in this case there is need for an appropriate level of
> imprecision in the tagging, and that this is met by the macrolanguage code alone
> (which is in 639-3 also, as I read it). I'm thinking that this is not so much the
> exception, but a situation that can arise with many other (macro)languages where
> there is no formal standard.
...

I think we're mostly agreed on the need to be able to do this.  The
question that we need to resolve is which of three choices to make:

(1) include the macrolanguage information in the registry and 
reflect it in the encoding of the language tag for the newly-added
languages which are encompassed by a macrolanguage

(2) do not include the macrolanguage information in the encoding
of the language tag, but do include it in the registry so intelligent
matching algorithms can act as though the information had been supplied

(3) do not include the macrolanguage information in the tag encoding,
and do not include it in the registry.

There are other possibilities, but these seem to be the most defensible.
Which of these makes the most sense seems to depend on the use case
one has in mind.  Overstating merely to drive home the point:  The
discussion of Dinka seems to favor (1).  Discussion of Norwegian
leads one to favor (2).  Discussion of Breton leads one to (3).

As co-chair...

At this moment, the discussion seems to favor (1), but it's clear that
we need to work through what we'd want do to the matching specification
before we could declare consensus on this.  If we find that an update
to the matching specification is necessary, we'll need to ask our AD
to get the IESG to add it to our charter.

Randy

---

From: Shawn Steele <[email-masked]>
Date: Thu, 4 Oct 2007 22:14:32 -0700

> but do believe that if there is a bias in the system to specificity, when specificity is not always the ideal
> approach, then it will inevitably encounter calls for flexibility later.

My opinion is that I don't mind the extra detail.  An application could choose to ignore that detail if it wasn't warranted in their case, or use it if necessary, but if they don't have the detail available, then they don't have the option.  (And IMO having the detail as a field in the registry doesn't count as being readily available)

In truth I don't mind much if we use extlang or not.  To me it makes a little more sense to have it and I don't see any disadvantages to its presence.

What I would like to see is this question being resolved :)  At some point I have to turn any decision/RFC into code for our various products, and I'd like as much time as possible.

- Shawn

---

From: Shawn Steele <[email-masked]>
Date: Thu, 4 Oct 2007 22:14:32 -0700

- Shawn

---

From: Shawn Steele <[email-masked]>
Date: Thu, 4 Oct 2007 22:09:20 -0700

>in lookup situations, they just say "I speak Breton, what's the best

IMO if the user only provides Breton, then any further lookup is the application's best guess.  To wildly mangle the analogy, if someone asked for en-US, an application could say "gee, a bunch of en-US speakers also know spanish, so I'll return es-US if I can't find en-US".  Such decisions might be appropriate for some applications and not for others.

I just don't think fallback from Breton to French is appropriate at the registry/RFC level.  I think its more appropriately decided by the applications that might want that fallback.

- Shawn

rom ltru by megatron.ietf.org with local (Exim 4.43)
	id 1IdfUk-0005og-0O
	for [email-masked]; Fri, 05 Oct 2007 01:13:02 -0400
From: Shawn Steele <[email-masked]>
To: John Cowan <[email-masked]>
Date: Thu, 4 Oct 2007 22:09:20 -0700
Subject: RE: [Ltru] Extended language tags
Accept-Language: en-US
acceptlanguage: en-US
Cc: "[email-masked]" <[email-masked]>

>in lookup situations, they just say "I speak Breton, what's the best

IMO if the user only provides Breton, then any further lookup is the application's best guess.  To wildly mangle the analogy, if someone asked for en-US, an application could say "gee, a bunch of en-US speakers also know spanish, so I'll return es-US if I can't find en-US".  Such decisions might be appropriate for some applications and not for others.

I just don't think fallback from Breton to French is appropriate at the registry/RFC level.  I think its more appropriately decided by the applications that might want that fallback.

- Shawn

---

From: John Cowan <[email-masked]>
Date: Fri, 5 Oct 2007 09:09:06 -0400

Randy Presuhn scripsit:

> (1) [registry and extlang tag]
> (2) [registry only]
> (3) [neither]

AFAIK no one is advocating for case 3; the split is between cases 1 and 2.

> to get the IESG to add it to our charter.

Based on Mark's and Addison's evidence (as discussed in the telcons),
I propose that a new subsection be added to Section 3 of 4647,
explaining how to do matching in the presence of extlang tags (case 1)
or Macrolanguage headers (case 2).

The trouble with directly using the 4647 algorithms is that they will
fall back poorly in case 1, because all other types of subtags will
have been disregarded before the conversion from zh-yue to zh is made.
In case 2, of course, the algorithms will not fall back at all, unless
by a private extension.

Therefore, matchers SHOULD, given a range with a 'yue' language subtag
(case 1), or 'zh-yue' language and extlang subtags (case 2), add to
the range list the corresponding range with the macrolanguage 'zh'.
Likewise, a matcher given a range with a language subtag of 'nn' SHOULD
add to the range list the corresponding range with the macrolanguage
'no', provided the LSR is available.

All this is orthogonal to the question of representation.  I favor 1
over 2 simply because nothing in 4647 MUSTard requires the availability
of the registry, and I'd like to see it stay that way.  (Currently,
matchers are "encouraged", a non-MUSTard term, to use the registry for
canonicalization, and people who create language ranges (as opposed to
matchers) SHOULD NOT include suppressed scripts.)

-- 
The experiences of the past show                John Cowan
that there has always been a discrepancy        [email-masked]
between plans and performance.                  http://www.ccil.org/~cowan
        --Emperor Hirohito, August 1945

---

From: Peter Constable <[email-masked]>
Date: Sat, 6 Oct 2007 23:49:27 -0700

The Dinka language development activities sound like there are similarities to what has been going on in the Philippines, where it was decided to engineer a pan-Philippine language "Filipino" as an amalgam with greatest dependence initially on Tagalog.

In that case, a new language name was created -- and a new ISO 639 ID. One could apply a similar approach in the Dinka case, except that the sociolinguistics appear to be different: from your description, it sounds like the speaker community has a conception of an encompassing linguistic unity, and 639-2 has had an ID that is comparable in scope.

Peter

-----Original Message-----
From: Andrew Cunningham [mailto:[email-masked]]
Sent: Thursday, October 04, 2007 6:02 PM
To: Shawn Steele
Cc: [email-masked]
Subject: Re: [Ltru] Extended language tags

dib     South Central Dinka
dik     Southwestern Dinka
dip     Northeastern Dinka
diw     Northwestern Dinka
dks     Southeastern Dinka

Andrew

[email-masked]
[email-masked]

---

From: Peter Constable <[email-masked]>
Date: Sat, 6 Oct 2007 23:57:48 -0700

Don, do you have a sense of how well the macrolanguage entities that have been in 639 cover the set of African developed macro-varieties that we're likely to see developing in the next (say) 20 years?

Put another way, are we likely to need a number of new IDs in 639 (as in the N'Ko case)? If so, should they have macrolanguage mappings in 639 to sets of existing entries? If so, what can we expect to happen in the way of tag use in the short term, while these varieties have not yet emerged or are only just starting to emerge, and what would pros/cons be of the extlang approach to tagging?

Peter

-----Original Message-----
From: Don Osborn [mailto:[email-masked]]
Sent: Thursday, October 04, 2007 6:40 PM
To: 'Andrew Cunningham'; Shawn Steele
Cc: [email-masked]
Subject: RE: [Ltru] Extended language tags

Don Osborn
Bisharat.net
PanAfriL10n.org

---

From: Peter Constable <[email-masked]>
Date: Sun, 7 Oct 2007 00:03:48 -0700

But I think Shawn's point is that the Breton/French example doesn't really tell us much of anything of interest wrt extlangs -- neither language has any macrolanguage / extlang issues involved. For Lookup, the best fallback for Dyirbal is probably English; the best fallback for Ainu is Japanese. So?

Peter

---

From: Addison Phillips <[email-masked]>
Date: Sun, 07 Oct 2007 10:11:19 -0700

I think this is a mis-reading of Mark's example. His presupposition goes 
something like: "suppose Breton had a macrolanguage of Celtic..."

It would be a more effective example, I think, if we used an actual 
enclosed language. I've been writing a long email in response to Shawn's 
(and various responses thereto) that I haven't quite been able to make 
myself send. Part of it includes this rephrasing of the example:

--
Let me rephrase Mark's Breton case: when you want Chippewa, any old 
Ojibwa (its macrolanguage) -- which might be a language such as Ottawa 
-- will not do. It could be unintelligible (I don't actually know). If 
it is unintelligible, you're probably better off *not* mapping Chippewa 
to Ojibwa. You're better off serving some useful default (for Chippewa, 
this is probably English, but the default is up to the application, not 
something RFC 4647 or the registry do). Or the app could fail to find 
content.
--

When considered like this, the Breton vs. French example is more useful. 
Otherwise it is pointless.

Addison

---

From: Addison Phillips <[email-masked]>
Date: Sun, 07 Oct 2007 10:16:31 -0700

(warning: this is a long screed, which, now that I've alluded to it on 
list--and even quoted from the abominable thing-- I may as well share, 
even though I'll probably think better of it in about an hour when the 
responses start to arrive)

>         * We (the bigger software ecosystem we, tools, etc) have encouraged use of zh-HK, etc for labeling both Mandarin and Cantonese.  Whatever happens in the future a large number of legacy docu…

Not exactly. "We" have (at least historically) abused the tags "zh-HK"
and "zh-TW" to represent Traditional Chinese and ignored the
Cantonese/Mandarin split (which is possible if one only means written
content, which historically has composed most of the material in our
information systems and on the Web).

More recently, language tags have allowed for direct tagging of these
differences as "zh-Hant" and "zh-Hans" (Traditional vs. Simplified)
allowing TW and HK to represent regional differences (there are regional
preferential differences, even in written texts, even when ignoring the
Cantonese/Mandarin split--we encounter these at Yahoo!, for example). It
is unclear how quickly the transition to these subtags is proceeding.
Some user-agents don't support these tags yet.

With the advent of audio and visual content in particular, tagging all
Chinese as 'zh' becomes less useful (at least for those cases), since we
suddenly find applications for which we need to describe specific spoken
variations not captured by 'zh' and not directly discernible from the
region subtag.

I note that Chinese *may* be a special case, in comparison to others, in
which we have very large document bases in a variety of Sinitic
languages tagged with a wide variety of regional subtags representing
something other than regional variations. Other macro languages don't
exhibit this sort or range of imputed meaning.

For example, the Arabic "sub-languages" are mostly regionally based, so
one might very well consider 'arz' (Egyptian Arabic) to be a synonym for
"ar-EG" (Arabic as used in Egypt). Of course, some of the other Arabic
languages don't map so closely to modern nation-states... for the
record, I oppose assigning "secret handshake" meaning to region subtags.
Explicit subtags seem to work better. [Yes, there really are regional
differences, for which region subtags are good, even when the region
where the language is spoken leaks over borders a bit.]

>         * Whether or not yue or zh-yue was used, it is a change from the current label (in most cases).  It seems that in nearly all cases this will require a code change.  In particular applicatio…

It requires tag changes to indicate things with more precision, yes.
Nothing says that we might not have continued use of the existing tags;
indeed, I would be surprised by a wholesale conversion over to the new
scheme (whatever it is). Most people won't "get the message" for awhile.
And sometimes the additional specificity doesn't matter ('zh' is
perfectly good for many written documents). However, for many
applications where content labeling can be controlled, converting over
makes life easier. For Chinese in particular, I expect a pretty messy
tagging situation to persist for quite some time (since we have
overlapping levels of imputed meaning in region subtags and elsewhere).

For other languages, a messy situation may not be necessary. Arabic
probably doesn't *require* extlangs, although that may be my parochial
perspective. While Andrew and Don point out the usefulness of deliberate
vagaries in tagging, mightn't we be just as well be served by tagging
"generic" Dinka documents (for example) as 'din' and specific Dinka
documents as, for example, 'dib' or 'dik'? Again, the question here
seems to depend on whether tags or ranges are what matter most.

>         * I don't think the Breton example applies.  http-accept-lang allows for "br-FR;fr-FR" type fallback, so that is a solution.  If an application independently wanted to make this assumption…

I agree that this fallback case is valid as a use-case for language
priority lists, but not for language tags. But I think this misses the
point. It is quoted without Mark's preamble, in which he posits Breton
as a sublanguage of Welsh. Mark is using it more as a parable to
illustrate the extlang case, not as an example-in-fact.

Let me rephrase Mark's Breton case: when you want Chippewa, any old
Ojibwa (its macrolanguage), which might be a language such as Ottawa,
will not do. It is unintelligible (maybe: I don't actually know in this
case). If it is unintelligible, you're probably better off *not* mapping
Chippewa to Ojibwa. You're better off serving some useful default (for
Chippewa, this is probably English, but is up to the application, not
something RFC 4647 or the LSR do) or even failing.

The counter-example would be the ietf-languages adventure in Norwegian
tagging. Were we better off when Norwegian was represented by the older
registered values (which are quite extlang like, eh?):

   no-nyn
   no-bok

Than by the codes:

   nn
   nb
   no

Things certainly were easier when 'no' meant generic Norwegian (which,
it turns out, tends to be 'nb') and Nynorsk could be represented
extlang/variant-like. In code that deals with Norwegian language tags
today (as well as other odd cases, such as the mistaken 'he'/'iw' pair),
you often have this irksome special mapping table of the (what would be)
sublanguage codes to its macrolanguage, which is what we "mean". But you
can't eat/change the original subtag---other processors might interpret
or map the codes differently or be able to serve specific resources
tailored to the original request. This is a flaw in the no-extlang case
and why Mark and others have insisted that Macrolanguage becomes an
important piece of information in the registry.

>         * Lots of zh data right now is zh-cmn.  Most of "us" seem to agree that we can't narrow the meaning of zh because it does allow Cantonese.  However if some application wanted to make an ass…

Agreed. More to the point, the "zh" tag encompasses a resource
containing *some* (which is to say, exactly ONE) variety of Chinese. It
is probably also Simplified Chinese, for example--it certainly can't be
both Simplified and Traditional.

But you can't make that assumption safely! There are also differences in
(for example) the accepted level of English word borrowing between (say)
Taiwan and HK, etc. "zh" embodies the least common denominator, which
may not be very common in certain circumstances (if you're talking about
our HK web site, 'zh' probably means content that could be tagged
"zh-yue-Hant-HK", whereas I'm pretty sure other sites or applications
would prefer it to be pretty similar to the stuff that could be tagged
"zh-cmn-Hans-CN"). Heck, when I started my internationalization career,
'zh' meant Traditional, because many of us did business in Taiwan and
hardly anyone in the PRC. Now 'zh' mostly "means" Simplified.

>         * Even for the current tags, many of the people in the teleconference seem to extend RFC 4647 in ways that are best for them.  Strict use of 4647 behavior seems rare.  It seems reasonable t…

I disagree. RFC 4647 is fine for many purposes. I'm proposing,
if we do extlangs, a small modification due to finding a slightly better
version of just one algorithm (lookup).

I think our ideal goal would be for RFC 4647 matching schemes to work
fully for everyone for the scope they are designed for.

We might make one or other tweak to the document and possibly the
algorithms based on further testing. For now, the existing schemes
mostly work. They exhibit varying impact on tag and range choice in
either extlang or no-extlang clothes.

I disagree that you have to have far more complex matching systems in
all cases. Yes, there are use cases that call for more complex systems.
But most low-level users have no need of these elaborations. For basic
protocols, I think we would prefer the simplest possible algorithms that
produce mostly the right results. Locale systems, for example, are based
on lookup (well, more like lookup is based on them), and they produce
meaningful results for resource lookup pretty reliably. I have a concern
that lookup work well, because I have a vested interest in the continued
good behavior of systems such as CLDR, which use and depend on these
simplest possible matching systems, such as lookup.

So we can change the algorithm or "change" the (as yet undefined) tags,
which is what is suggested as the other route.

> The deciding factor for me is that to know that cmn is related to zh I'd have to look in the registry, but zh-cmn contains that 
information.  Otherwise I don't really see advantages with either method.

The question is: do we want to do this for all time? Or can we help
people migrate and just get started on the migration? Yes, retagging
with "yue-HK" is a PITA, but so is retagging with "zh-yue-HK". If we
have to convince folks to retag their data anyway, what is the best
solution/most sensible?

The benefit to extlang partially comes from the expectation that the 
tags will be changed but users won't update their ranges. For that case, 
you can switch to extended filtering and a slightly modified lookup and 
use extlangs and go about your business. Of course, you have to explain
these fairly complex tags to the Chinese...... (and Arabic, Dinka, 
Quechua, Zapotec, and etc. speakers)

On the other hand, if you go with primary language subtags for everyone,
you are expecting that users will change or augment their language
priority lists (i.e. "yue-HK; zh-HK") to deal with un-retagged data or
provide alternate lookup tables in your implementations. Matching
algorithms do not change, but tags eventually have to catch up for the
majority of documents or there is (probably quite icky) tagging chaos.

So, for me, the main issue is whether we are going to explicitly break 
the connection between the language and its macrolanguage (at the tag
level). In some cases (Norwegian) we already know this can be 
problematic; in others, it may actually be desirable.

Addison

---

From: "Andrew Cunningham" <[email-masked]>
Date: Mon, 8 Oct 2007 10:15:57 +1000

> problematic; in others, it may actually be desirable.

It probably comes down to philosophical differences.

So, cutting to the chase, which way forward?

Andrew

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 7 Oct 2007 17:40:33 -0700

I think what might help is for us to draw up in detail what the matching
algorithms (changes to 4747) would be like in either of the two proposals,
and look at what happens with different cases 'ar', 'zh', 'no', and others.

Mark

---

From: Addison Phillips <[email-masked]>
Date: Sun, 07 Oct 2007 20:29:36 -0700

I would say:

1. With Extlangs.

- change to filtering: none, but you probably want to use extended 
filtering instead of basic filtering (i.e. "zh-Hant-HK" matches 
"zh-yue-Hant-HK" and "zh-cmn-Hant-HK")

- change to lookup: treat extlang as atomic with the primary language 
subtag; potentially loop-back through the subtags. That is, given the 
range "zh-yue-Hant-HK", the fallback pattern is this:

  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  zh-Hant-HK
  zh-Hant
  zh
  (default)

Or this:

  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  (default)

2. Without extlangs.

- change to filtering: none

- change to lookup: none

BUT... you want to include the macro language in your ranges in some 
cases. Alternatively, we would have to define new filtering and lookup 
options that include mapping to macrolanguages. For example, with the 
range "yue-Hant-HK", you would want the fallback to be:

  yue-Hant-HK
  yue-Hant
  yue
  zh-Hant-HK
  zh-Hant
  zh
  (default)

This can be achieved either by having the language priority list 
"yue-Hant-HK;zh-Hant-HK" or by inferring it using registry data about 
macro languages.

A similar case can be made for filtering.

Addison

Mark Davis wrote:
> algorithms (changes to 4747) would be like in either of the two 
> proposals, and look at what happens with different cases 'ar', 'zh', 
> 'no', and others.
> 
> Mark
> 
> On 10/7/07, *Andrew Cunningham* <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     On 08/10/2007, Addison Phillips <[email-masked]
>     <mailto:[email-masked]>> wrote:
> 
>      >
>     [email-masked] <mailto:[email-masked]>
>     <https://www1.ietf.org/mailman/listinfo/ltru>
> 
> 
> 
> 
> -- 
> Mark

---

From: Peter Constable <[email-masked]>
Date: Mon, 8 Oct 2007 00:43:27 -0700

From: Addison Phillips [mailto:[email-masked]]

> -- will not do...

> Otherwise it is pointless.

Well, what this points out is that the macrolanguage cases are not all comparable.

- Some are like Chinese in having one encompassed language (I'll call this the Major Language Variety - MLV) that is quite developed and probably intelligible to a considerable extent to speakers of all the encompassed languages.

- Some are like Ojibwa in not having one such MLV among the encompassed languages.

For purposes of lookup, the MLV of the former cases may well be an acceptable fallback for the other encompassed languages. But in the latter cases, there is no MLV, and none of the encompassed languages is likely to be an acceptable fallback in general.

For purposes of filtering, if the records are all tagged specifically (Mandarin, Yue, etc. instead of just "zh"; "Ottawa, Chippewa, etc. instead of just "oji"), then a user can always get what they want -- narrow or specifc. But, of course, records aren't always going to be tagged specifically, so let's consider those scenarios. In the Ojibwa-type cases, results using "oji" as language-range will likely be comparably good (or bad) for Chippewa as for Ottawa (or etc.). In the Chinese-type cases, results using "zh" as language-range are likely to be weighted in favour of Mandarin versus Yue, etc.

I haven't thought yet how this all relates to the issue of xxx-yyy vs. yyy.

Peter

---

From: Peter Constable <[email-masked]>
Date: Mon, 8 Oct 2007 01:10:46 -0700

>>         * I don't think the Breton example applies...

> will not do.

In that case, your language-range would be "(oji-)ciw"; resources might be tagged "oji" or "(oji-)ciw". The request should match "(oji-)ciw" if availale; otherwise, "oji" may be matched -- which may or may not be acceptable.

But note: you get the potential of an unacceptable result whether "oji-ciw" is used or "ciw" is used and the association between this an "oji" is recorded in the registry. Either way, a connection is made between macrolanguage and encompassed language -- whether in the tag or in the registry.

The claim is valid that (i) some macrolanguages have MLVs while others do not, and (ii) that a lookup match in the latter cases a lookup match using the macrolanguage could result in an unacceptable / unintelligible.

*But*, these facts do not differentiate between the extlang vs. macrolanguage field proposals.

Peter

---

From: John Cowan <[email-masked]>
Date: Mon, 8 Oct 2007 10:16:11 -0400

>  (default)

This is equivalent to what I proposed: the matching engine should note
that "zh-yue-Hant-HK" has an extlang, and add "zh-Hant-HK" to the list
of language ranges immediately after it.

>  (default)

This likewise is equivalent: if "yue-Hant-HK" is in the list, add
"zh-Hant-HK" after it.

-- 
Even the best of friends cannot                 John Cowan
attend each others' funeral.                    [email-masked]
        --Kehlog Albran, The Profit             http://www.ccil.org/~cowan

---

From: Marion Gunn <[email-masked]>
Date: Mon, 8 Oct 2007 16:49:10 +0000

Short answer to a short question - go for language groups (tag  
languages in groups which make sense and require little or no  
explaining), rather than inventing confusing concepts and naming them  
"extlang", "macrolanguage", which, at least to date, seem to beg more  
questions than they can answer.
mg

> So, cutting to the chase, which way forward?

- -
Marion Gunn * EGTeo (Estab.1991)
27 Páirc an Fhéithlinn, Baile an
Bhóthair, Co. Átha Cliath, Éire.
* [email-masked] * [email-masked] *

---

From: Addison Phillips <[email-masked]>
Date: Mon, 08 Oct 2007 08:51:06 -0700

> This likewise is equivalent: if "yue-Hant-HK" is in the list, add

They are equivalent *functionally*, but not equivalent in some other 
important ways. Notably: the first version uses only information 
embedded in the tags themselves. The second requires the registry. The 
first version requires code changes in the lookup algorithm. The second 
requires changes to the user's language priority list.

Addison

---

From: John Cowan <[email-masked]>
Date: Mon, 8 Oct 2007 11:58:23 -0400

> requires changes to the user's language priority list.

Sorry, I cut too much context.  I was saying that extending the user's
priority list is equivalent to making changes to the actual algorithms,
and is less intrusive, because if an older implementation does not
provide this extension, the user can do so instead.  This is so
under either the extlang or the no-extlang scheme.

-- 
John Cowan    http://ccil.org/~cowan    [email-masked]
Mr. Henry James writes fiction as if it were a painful duty.  --Oscar Wilde

---

From: Shawn Steele <[email-masked]>
Date: Tue, 9 Oct 2007 12:32:53 -0700

Your 1) & 2) don't have the same functionality.  If you don't change the lookup for 2), then you don't get the behavior that your changed lookup in 1) provided.

Changing a language priority list in 2) does count as a change.  This doesn't help a "new" web app talking to an "old" client that hasn't been updated, for example.  In order to solve that you have to infer the matching suggested, which I think is simpler with 1).

Note that your solution in 2) could be used if you felt it was simpler:  zh-yue-Hant-HK;zh-Hant-HK could be used to solve 1) as well, but with 1) you have more options.

- Shawn

-----Original Message-----
From: Addison Phillips [mailto:[email-masked]]
Sent: Sunday, October 07, 2007 8:30 PM
To: Mark Davis
Cc: [email-masked]
Subject: Re: [Ltru] Extended language tags (long reply)

I would say:

1. With Extlangs.

  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  zh-Hant-HK
  zh-Hant
  zh
  (default)

Or this:

  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  (default)

2. Without extlangs.

- change to lookup: none

  yue-Hant-HK
  yue-Hant
  yue
  zh-Hant-HK
  zh-Hant
  zh
  (default)

Addison

---

From: Addison Phillips <[email-masked]>
Date: Tue, 09 Oct 2007 12:49:01 -0700

Shawn Steele wrote:
> Your 1) & 2) don't have the same functionality.  If you don't change the lookup for 2), 
 >  then you don't get the behavior that your changed lookup
 > in 1) provided.
> 
> Changing a language priority list in 2) does count as a change. 

Yes. This is the Norwegian case I cited elsewhere: you have to have 
"special" or "secret" information in order to perform the complete 
lookup. Which is a pain.

However... we aren't *required* to make a change to the algorithm 
itself. We could just pretend that everyone is done with their retagging 
:-).

> This doesn't help a "new" web app talking to an "old" client that 
 > hasn't been updated, for example.  In order to solve that
 > you have to infer the matching suggested, which I think is
 > simpler with 1).

Yes.

> 
> Note that your solution in 2) could be used if you felt it was simpler:
 >   zh-yue-Hant-HK;zh-Hant-HK
 > could be used to solve 1) as well, but with 1) you have more options.

No: this is why the modification is necessary. The first fallback, 
without the change goes:

  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  zh        <-- wrong result, possibly similar to zh-Hans-CN, returned)
  zh-Hant-HK
  zh-Hant
  zh
  (default)

I'm not saying that (1) is a bad option. It works for me. I'm just 
trying to make the cases clear.

Addison

---

From: Shawn Steele <[email-masked]>
Date: Tue, 9 Oct 2007 13:14:27 -0700

> > problematic; in others, it may actually be desirable.

Mostly I agree with most of the points in Addison's "long reply", I just disagree with the conclusion :)

> It probably comes down to philosophical differences.

:)

> So, cutting to the chase, which way forward?

:)

I do have one other concern that neither suggest addresses directly: My understanding (possibly incorrect) is that Brazilian Portuguese and Portuguese Portuguese are different enough to require different localizations.  Labeling them pt-BR and pt-PT are fine, except that if I end up with pt-MZ how do I find the best match?

- Shawn

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 9 Oct 2007 13:27:48 -0700

Hi -

> From: "Shawn Steele" <[email-masked]>
> To: "Andrew Cunningham" <[email-masked]>; "Addison Phillips" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Tuesday, October 09, 2007 1:14 PM
> Subject: RE: [Ltru] Extended language tags (long reply) - portuguese
...
> I do have one other concern that neither suggest addresses directly:
> My understanding (possibly incorrect) is that Brazilian Portuguese
> and Portuguese Portuguese are different enough to require different
> localizations.  Labeling them pt-BR and pt-PT are fine, except that
> if I end up with pt-MZ how do I find the best match?
...

Not our problem.  :-)

But seriously, this is exactly the same kind of problem as worries the
folks who don't like macrolangauge/extlang constructs; there are cases
where the results of remove-from-right heuristics won't always be helpful
in all situations.

Randy

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 10 Oct 2007 08:55:18 -0700

Some detailed comments on these scenarios.

> "zh-yue-Hant-HK" and "zh-cmn-Hant-HK")

Actually, I think the change is the reverse; you have to disable extended
filtering. In the filtering / search business, it is just as bad (if not
worse) to give too many responses as it is to give too few. Otherwise your
user is swamped in irrelevant documents.

Except in very few cases, notably Arabic and Chinese, according to what I
have heard Peter say, we have no reason to believe that a speaker of one
microlanguage speaks any other particular macrolanguage. If my user searches
for Fulah, and does s/he want to get Maasina Fulfulde, Adamawa Fulfulde,
Pulaar, Central-Eastern Niger Fulfulde, and so on, which I have no reason to
believe that the original person speaks?

And even for Arabic and Chinese, unmodified extended filtering with extlang
might make sense only because ar-arb is equivalent to arb and zh-cmn is
equivalent to cmn, for all practical purposes, and where all speakers of the
microlanguages understand the macrolanguages (which is unclear -- I've asked
on the list whether the latter is true -- and it is a crucial fact for
extlang -- but get no answer). Better is that if someone really wants both
standard Arabic and Tajiki Arabic, than the input be "ar, abh", just like
now with Norwegian or Serbo-croation, where someone can just list "nn, nb,
no" and "sr, hr, bs, sh",

So with extlang we have a backwards compatibility issue with extended
filtering; we need to disable macrolanguage lookup in all but a few cases,
and even there it is better for the user to just supply the list of items
they want.

http://www.sil.org/iso639-3/macrolanguages.asp

> range "zh-yue-Hant-HK", the fallback pattern is this:

And again, we disable this for all but a few specific cases. It only makes
sense to do this as a heuristic when the speaker of the microlanguage is
extremely likely to speak the macrolanguage.

If the query is Maasina Fulfulde as used in Ghana (ful-)ffm-GH, we don't
want to fall back to some arbitrary Fulah; instead we want to probably fall
back to Hausa.

And this would need to be handled even more carefully where the input is
multiple tags, as in Accept-Language. If the user supplies "zh-yue; en",
meaning that they prefer Cantonese, then English -- and may not know
Mandarin at all! -- then the stated algorithm, the one that everyone uses
now, would give completely incorrect results.

(1) Incorrect
  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  zh
  en
  (default)

Even the modified version doesn't work

(2) Still incorrect
  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  zh-Hant-HK
  zh-Hant
  zh
  en
  (default)

What we need to do is more complicated:

(3) what the user asked for, Cantonese, then English, then maybe something
else.
  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  en
  zh-Hant-HK
  zh-Hant
  zh
  (default)

Number 3 is especially needed whenever the macrolanguage is not
overwhelmingly likely to be understood by the macrolanguage users.

So with extlang we also have a backwards compatibility issue with lookup;
code that used to work fine will fail.

>   (default)

This would be better, but still requires a change to algorithms, and then
simply amounts to adding extlang, but coding around it so that the results
were as if there were no extlang -- better to not use the extlang mechanism.

2. Without extlangs.
> - change to filtering: none

Agreed so far, but...

> range "yue-Hant-HK", you would want the fallback to be:

In the case of no extlangs, we really do not want to make any changes to the
algorithms. Instead, we want to point out in the text that macro languages
*MAY* be a useful resource for having some extended fallback. That is, it
*MAY* be reasonable to fall back from the Chinese and Arabic microlanguages
to their macrolanguages, but also may not. And the interaction between this
additional fallback and multiple tags needs to be taken into consideration.

Suppose that the input is "yue-Hant-HK; en"; the current algorithm works.

(1)
  yue-Hant-HK
  yue-Hant
  yue
  en
  (default)

If a particular implementation wants to change the default processing to
have an extra step that adds fallback to zh-Hant-HK (and so on) for certain
specific languages, such as particular Chinese and Arabic microlanguages,
that's a small amount of work. Moreover, it is just what a more enhanced
algorithm *currently* would do with any of "nn, nb, no" and "sr, hr, bs,
sh", -- in this case they are known to be good, practical fallbacks.

Alternatively, if the user just feeds in "yue-Hant-HK, en, zh-Hant-HK" or
the reverse "yue-Hant-HK, zh-Hant-HK, en", that will work without any code
changes, or any enhancements; the user gets what is asked for.

================

What we don't want to do is make recommendations that if implemented, are
harder for people to control and get the right answer. And baking extlang
into the tags is even worse -- since it introduces backwards
incompatibilities that require old code to be modified to work around.

Someone mentioned on this list that they thought that these were only
philosophical differences. I don't see it that way at all. The choices we
make here will affect the ability of people to work with their own languages
for a long time -- it is important to get this right, and pay attention both
to backwards compatibility and to future capabilities -- the choice of
architecture will make real, practical differences for people, especially
for minority language users. My fear is exactly minority language users will
be disadvantage with extlang. It isn't the French or German speakers that
will end up with problems; it is the Gondi and Grebo speakers.

Mark

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 10 Oct 2007 17:24:09 +0100

Hi Mark
 
I haven't time at the moment to put together the fallback for this (as it
would take me ages) but I think fas-prs and fas-pes are good cases for
macrolanguages to be included as extended language tags.  These languages
are mutually intelligible and to be able to choose prs and have a system
create a fallback based on fas would be a huge bonus.  Implementations
should have the extended language tags option and end users should be
presented with a question asking if they want macrolanguage related
languages matched IMHO. It is not just about whether they understand the
macrolanguage it is whether a user wants to return the related languages
within the macrolanguage; I do understand that the end user would not want
this in many cases but where is the harm in the provision?  If I was
querying using prs I would certainly want to have pes and fas tagged items
returned as a secondary match. 
 
Best regards
 
Debbie  

  _____  

From: Mark Davis [mailto:[email-masked]] 
Sent: 10 October 2007 16:55
To: Addison Phillips
Cc: [email-masked]
Subject: Re: [Ltru] Extended language tags (long reply)

I would say:

1. With Extlangs.

  zh-yue-Hant-HK 
  zh-yue-Hant
  zh-yue
  zh-Hant-HK
  zh-Hant
  zh
  (default)

(1) Incorrect
  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  zh
  en
  (default)

(2) Still incorrect
  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue 
  zh-Hant-HK
  zh-Hant
  zh
  en
  (default)

Or this:

  zh-yue-Hant-HK
  zh-yue-Hant
  zh-yue
  (default)

2. Without extlangs.

- change to lookup: none 

Agreed so far, but... 

  yue-Hant-HK
  yue-Hant
  yue
  zh-Hant-HK
  zh-Hant
  zh
  (default)

(1) 
  yue-Hant-HK
  yue-Hant
  yue
  en
  (default)

================

Mark

---

From: Shawn Steele <[email-masked]>
Date: Wed, 10 Oct 2007 11:02:01 -0700

================

What we don't want to do is make recommendations that if implemented, are harder for people to control and get the right answer. And baking extlang into the tags is even worse -- since it introduces backwards incompatibilities that require old code to be modified to work around.

================

cmn is completely incompatible with existing practice anyway, so you can’t claim that it solves the problem.  Existing clients ask for zh-HK (or whatever) and code is tagged as zh-HK (or whatever).  Those won’t match cmn using RFC 4647.

So for backwards compatibility zh-cmn is no worse than cmn.  And if you don’t like the inference of the zh, then you can ignore that part, but at least the data’s there if people do want it.

For some macro languages the strict fallback is probably inappropriate, however I don’t expect to find matches in that case (because I don’t expect correctly tagged data to be “zh”).  If this is a concern, those are easily filtered out.

From the discussion I don’t think the bigger problem is whether or not we go with zh-cmn or just cmn.  The bigger issue seems to be how to modify RFC 4647 to provide meaningful fallback with whichever mode is used.  Since some applications may (or may not) want to consider zh-HK or other legacy behaviors, such recommendations aren’t trivial.  Given the differing requirements amongst us, I suspect a certain flexibility of the applications will be necessary, perhaps several suggestions rather than the strict behavior of RFC 4647 (which everyone seems to modify for their purposes anyway)

- Shawn

---

From: [email-masked]
Date: Wed, 10 Oct 2007 11:25:53 -0700

I think if meaningful fallback is the goal, you need to consider both 
audio and textual forms of languages and their respective fallback 
mechanisms. While a fallback to "zh" (from "zh-cmn" or "zh-HK", etc.) 
would likely be useful in written contexts, falling back to spoken "zh" 
from spoken "cmn" may provide something unintelligible. 

In the context of my industry, it would likely be useful for a Cantonese 
speaker to be able to find subtitled films in "zh" if "zh-yue" is not 
available. Obviously there are questions of simplified vs. traditional 
orthography, but the "zh" tag alone encompasses both orthographies. But if 
the Cantonese speaker is looking at dubbed films, a fallback to spoken 
"zh-cmn" is not as likely to be useful.

From what I understand the same is true in written Dutch vs. spoken 
Flemish -- a Flemish speaker may be able to easily read standard Dutch, 
but the spoken forms can create problems of understanding to the point 
where Dutch is subtitled in Belgium. (I still don't know whether "vls" 
(639-3) falls back to "nld/dut" in 639-2, though Flemish is cited as a 
synonym for Vlaams in 639-3.)

Please consider the nature of both audio and written language with respect 
to ext langs. I think you'll find what makes sense for fallback depends on 
these distinctions in many other cases than those I've mentioned. 

Regards,

Karen Broome
Metadata Systems Designer
Sony Pictures Entertainment
310.244.4384

Shawn Steele <[email-masked]> 
10/10/2007 11:02 AM

To
Mark Davis <[email-masked]>, Addison Phillips 
<[email-masked]>
cc
"[email-masked]" <[email-masked]>
Subject
RE: [Ltru] Extended language tags (long reply)

================

================

cmn is completely incompatible with existing practice anyway, so you can’t 
claim that it solves the problem.  Existing clients ask for zh-HK (or 
whatever) and code is tagged as zh-HK (or whatever).  Those won’t match 
cmn using RFC 4647.
 
So for backwards compatibility zh-cmn is no worse than cmn.  And if you 
don’t like the inference of the zh, then you can ignore that part, but at 
least the data’s there if people do want it.
 
For some macro languages the strict fallback is probably inappropriate, 
however I don’t expect to find matches in that case (because I don’t 
expect correctly tagged data to be “zh”).  If this is a concern, those are 
easily filtered out.
 
From the discussion I don’t think the bigger problem is whether or not we 
go with zh-cmn or just cmn.  The bigger issue seems to be how to modify 
RFC 4647 to provide meaningful fallback with whichever mode is used.  
Since some applications may (or may not) want to consider zh-HK or other 
legacy behaviors, such recommendations aren’t trivial.  Given the 
differing requirements amongst us, I suspect a certain flexibility of the 
applications will be necessary, perhaps several suggestions rather than 
the strict behavior of RFC 4647 (which everyone seems to modify for their 
purposes anyway)
 
- Shawn_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Wed, 10 Oct 2007 11:36:01 -0700

> If my user searches for Fulah, and does s/he want to get
> Maasina Fulfulde, Adamawa Fulfulde, Pulaar, Central-Eastern
> Niger Fulfulde, and so on, which I have no reason to

The user probably doesn't want all that. But how do you know *what* they want? If they use a generic qualifier in their query, I don't know how you do better than return generic results, in the absence of additional information.

Peter

---

From: [email-masked]
Date: Wed, 10 Oct 2007 11:37:24 -0700

I wrote:

(I still don't know whether "vls" 
(639-3) falls back to "nld/dut" in 639-2, though Flemish is cited as a 
synonym for Vlaams in 639-3.)

Sorry, I meant to say that "Flemish" is cited as a synonym on 
Ethnologue.com.

Karen Broome

[email-masked] 
10/10/2007 11:25 AM

To

Regards,

Karen Broome

To

================

================

cmn is completely incompatible with existing practice anyway, so you can?t 

claim that it solves the problem.  Existing clients ask for zh-HK (or 
whatever) and code is tagged as zh-HK (or whatever).  Those won?t match 
cmn using RFC 4647.
 
So for backwards compatibility zh-cmn is no worse than cmn.  And if you 
don?t like the inference of the zh, then you can ignore that part, but at 
least the data?s there if people do want it.
 
For some macro languages the strict fallback is probably inappropriate, 
however I don?t expect to find matches in that case (because I don?t 
expect correctly tagged data to be ?zh?).  If this is a concern, those are 

easily filtered out.
 
>From the discussion I don?t think the bigger problem is whether or not we 
legacy behaviors, such recommendations aren?t trivial.  Given the

---

From: Peter Constable <[email-masked]>
Date: Wed, 10 Oct 2007 11:46:33 -0700

Where do you see "Flemish" cited as synonymous with "Vlaams"?

The closest I see is a statement that Dutch is "[c]alled 'Vlaams' in Belgium, even though it is different from the (West) Vlaams spoken there."

Peter

> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, October 10, 2007 11:37 AM
> To: [email-masked]
> Subject: RE: [Ltru] Extended language tags (long reply)
>
> I wrote:
>

---

From: [email-masked]
Date: Wed, 10 Oct 2007 11:57:15 -0700

http://www.ethnologue.com/show_language.asp?code=vls

"Flemish" is listed as an alternate name in Netherlands and France.

Karen Broome

Peter Constable <[email-masked]> wrote on 10/10/2007 11:46:33 AM:

> The closest I see is a statement that Dutch is "[c]alled 'Vlaams' in
> Belgium, even though it is different from the (West) Vlaams spoken 
there."
> > orthography, but the "zh" tag alone encompasses both orthographies. 
But
> > if
> > to ext langs. I think you'll find what makes sense for fallback 
depends
> > on
> > >From the discussion I don?t think the bigger problem is whether or 
not
> > we

---

From: John Cowan <[email-masked]>
Date: Wed, 10 Oct 2007 15:31:16 -0400

> synonym for Vlaams in 639-3.)

It does not.

> Sorry, I meant to say that "Flemish" is cited as a synonym on 

That is to say, the name "Flemish" has been used for both vls and
the Belgian version of nl.

-- 
Mark Twain on Cecil Rhodes:                    John Cowan
I admire him, I freely admit it,               http://www.ccil.org/~cowan
and when his time comes I shall                [email-masked]
buy a piece of the rope for a keepsake.

---

From: [email-masked]
Date: Wed, 10 Oct 2007 13:58:53 -0700

Without additional disambiguation, this is very confusing to just about 
anyone with a language list that includes Flemish. It is especially 
confusing to me as the editor of ISO 639-3 previously indicated to me and 
the MPAA that this was indeed the tag for the Flemish dialect of Dutch. 
Granted, 639-3 was in DIS status at that point. After ISO 639-3 was 
published, I recommended the "vls" tag for a non-RFC-4646 usage than 
spanned efforts of several studios. 

Could we make a request for just a bit more disambiguation in the 
Ethnologue entry?

This is text I received in a private e-mail about the tag when I presented 
a list of film languages a couple years back and asked for equivalents in 
ISO 639-3. I specifically noted the need to distinguish the Flemish 
dialect of Dutch from the Netherlands variant:

"ISO 639-3 does have an entry for "Flemish", though it happens to use the 
autonym "Vlaams" for the reference name. That is, on the one hand, a 
documentation problem and, on the other, a UI problem: a developer needs 
to be able to discover that the entry "vls, 'Vlaams'" is for Flemish, and 
then they need to determine the UI string they should present to users 
that will allow *them* to find the correct item in a list. 

Regards,

Karen Broome

John Cowan <[email-masked]> 
10/10/2007 12:31 PM

To
[email-masked]
cc
"[email-masked]" <[email-masked]>
Subject
Re: [Ltru] Extended language tags (long reply)

It does not.

---

From: Peter Constable <[email-masked]>
Date: Wed, 10 Oct 2007 17:45:12 -0700

[cc to JAC]

The documentation problem is something that ISO 639 RAs potentially can and probably should address so that it's clear to implementers that "vls" is indeed the identifier for the language variety typically meant by the appellation "Flemish".

Note, though, that there's no way to escape the issue Ethnologue is calling out: that "Flemish" and "Vlaams" are sometimes used to refer to the variety usually called (in English) Dutch. If someone wants to find the ID for "Flemish", they must first make sure they know what language variety it is they are really dealing with. That is not a problem that ISO 639 can deal with.

Also, ISO 639 cannot really deal with the UI issue. Languages can have many names. Some names may go in or out of vogue; some names may be strongly preferred in some places and simultaneously strongly disparaged in other places. It is not the role of the ISO 639 JAC or the RAs to give out recommendations as to what names should be exposed to users in products.

Finally, wrt our email exchange in 2005, I think I may have expressed things in a way that was misunderstood. I believe this is the bit in question:

<quote>
E.g. Karen mentioned "nld-BE" (3066 would use require nd-BE) for Flemish, though 639-3 will provide "vls". I expect a successor to RFC 3066 will consider both valid, though we can't force people to use "vls" if they happen to come up with "nd-BE".
</quote>

I'm pretty sure I did not mean to imply by these comments that I considered "vls" appropriate for Dutch spoken in the Flemish region. I believe I thought we were discussing Flemish aka Vlaams.

Peter

> Sent: Wednesday, October 10, 2007 1:59 PM
> To: John Cowan
> Cc: [email-masked]

---

From: [email-masked]
Date: Wed, 10 Oct 2007 18:39:41 -0700

The comment I quoted was in reply to this comment from me. I think it was 
pretty clear that I was talking about dialects of Dutch. The context for 
this comment was not RFC 4646, it was ISO 639-6 and some of the 
interesting things I saw in its hierarchical structure and spoken/written 
distinctions:

Dialects represented here are critical to us even if we only use a small 
subset of these codes. The difference between Dutch and Flemish is 
important in our world. I understand that Dutch productions are often 
subtitled in Flanders because the accent is so different. In written form, 
there is very little to distinguish the two dialects. Even 639-3 doesn't 
recognize Flemish. 

The people tasked with classifying content and developing linguistic 
classifications are usually not people who can tell the difference between 
Flemish and Vlaams given a piece of data that merely says "Flemish." For 
this reason, I request clarification on both the Dutch and Vlaams pages on 
the Ethnologue site. This would help.

This is now probably off-topic for the IETF list, but serves as a reminder 
of the confusion that can result from the combination of such different 
language standards in BCP 47 and the need for adequate documentation and 
disambiguation at the Ethnologue site. I think ISO 639-3 may prove to be 
too complex for the average classifier of linguistic content. 

Regards,

Karen Broome

Peter Constable <[email-masked]> 
10/10/2007 05:45 PM

To
"[email-masked]" <[email-masked]>
cc
'ISO 639 Joint Advisory Committee' <[email-masked]>
Subject
RE: [Ltru] Extended language tags (long reply)

[cc to JAC]

The documentation problem is something that ISO 639 RAs potentially can 
and probably should address so that it's clear to implementers that "vls" 
is indeed the identifier for the language variety typically meant by the 
appellation "Flemish".

Note, though, that there's no way to escape the issue Ethnologue is 
calling out: that "Flemish" and "Vlaams" are sometimes used to refer to 
the variety usually called (in English) Dutch. If someone wants to find 
the ID for "Flemish", they must first make sure they know what language 
variety it is they are really dealing with. That is not a problem that ISO 
639 can deal with.

Also, ISO 639 cannot really deal with the UI issue. Languages can have 
many names. Some names may go in or out of vogue; some names may be 
strongly preferred in some places and simultaneously strongly disparaged 
in other places. It is not the role of the ISO 639 JAC or the RAs to give 
out recommendations as to what names should be exposed to users in 
products.

Finally, wrt our email exchange in 2005, I think I may have expressed 
things in a way that was misunderstood. I believe this is the bit in 
question:

<quote>
E.g. Karen mentioned "nld-BE" (3066 would use require nd-BE) for Flemish, 
though 639-3 will provide "vls". I expect a successor to RFC 3066 will 
consider both valid, though we can't force people to use "vls" if they 
happen to come up with "nd-BE".
</quote>

I'm pretty sure I did not mean to imply by these comments that I 
considered "vls" appropriate for Dutch spoken in the Flemish region. I 
believe I thought we were discussing Flemish aka Vlaams.

Peter