Subject: [Ltru] extlang (was: Punjabi)

From: "Mark Davis" <[email-masked]>
Date: Sat, 17 Mar 2007 11:49:13 -0700

(I agree with Frank, we should change the subject line.)

Let me try another way to state it. We've been looking into how to implement
extlang at Google, and it has a number of complications when you start to
look at actual implementations. We want the default fallback to work well,
but using the extlang model we get the equivalent of the "suppress script"
problem.

Right now, 99% of the usage of zh means Mandarin, 99% of the usage of ar
means standard Arabic (fuṣḥā) If we introduce zh-cmn and ar-arb it
complicates lookup considerably. Compatibility forces us to treat "ar" and
"ar-arb" as equivalents, which means we already have non-trivial lookup,
since we have to have all child languages match, eg ar-arb-Arab-EG and
ar-Arab-EG, etc. Encoding arq as ar-arq to do a bit of magic in fallback
isn't worth the trouble introduced by that. It may well be that arb is the
best fallback for arq, but it may also be that nb is the best fallback for
da, which we don't try to incorporate in the language tags themselves. That
is, trying to guess what the best fallback is AND bake it into the encoding
of the tags itself for the case of macro languages is simply a complication
for little benefit.

At this point, we'd be better off to add information to the registry about
related languages than we would to use the extlang structure. That is, add
"cmn-CN", because we need to have it, and have an extra field to say that
this is related in a certain way to "zh", information that can be used in
matching.

Mark

On 3/17/07, Doug Ewell <[email-masked]> wrote:
>
> Mark Davis wrote:
>
> > Let's suppose that I have content tagged with the following:
> >
> > #1 zh-Hant-HK
> > #2 zh-Hans-HK
> > #3 yue-SG
> >
> > If I basic filter for zh, I'll get #1 and #2. With just a bit smarter
> > filter, using information from ISO 639 (maybe put into our registry),
> > I'll also match #3. If I search with zh-Hant, I'll get #1 in either
> > case. If I search for yue, I'll get just #3.
>
> Wait a minute.  You've been arguing that we need to put writing-system
> differences in the script subtag and not in the variant, because filters
> will be too stupid to do anything more than prefix matching.  Then why
> would we design for the possibility of a "smarter filter" than could
> associate "yue" with "zh"?
>
> --
> Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
> http://users.adelphia.net/~dewell/
> http://www1.ietf.org/html.charters/ltru-charter.html
> http://www.alvestrand.no/mailman/listinfo/ietf-languages
>
>

-- 
Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 17 Mar 2007 14:32:00 -0700

Mark Davis wrote:

> At this point, we'd be better off to add information to the registry 
> about related languages than we would to use the extlang structure. 
> That is, add "cmn-CN", because we need to have it, and have an extra 
> field to say that this is related in a certain way to "zh", 
> information that can be used in matching.

I see your point (which is not the same as agreeing with it), but you 
still haven't convinced me that these matching filters which are too 
simplistic to match "zh-Hans-SG" with "zh-SG" are going to be clever 
enough to find the relationship between "zh" and "cmn" in the Registry.

---

From: Gerard Meijssen <[email-masked]>
Date: Sun, 18 Mar 2007 13:16:04 +0100

Hoi,
Modern computers are quite capable of this "non-trivial" lookup that you 
refer to. The reason why we should allow for both zh-cmn and cmn is 
because it is much easier to explain to *people*. I utterly disagree 
that having both options is of little benefit. It helps people; it helps 
people understand the code. We give meaning to these codes and it is 
extremely hard to explain how these codes work. For people it makes a 
hell of a difference. You can explain that zh is old and Mandarin can be 
used as zh-cmn and cmn as a consequence. People will opt for cmn if they 
have the option and to it right, they will still opt for cmn and you 
would call them wrong.

We get a much better adoption by being understood. It is more important 
than having computers to know that two codes are equivalent. Computers 
are good at that !!

Thanks,
    Gerard

Mark Davis schreef:
> Let me try another way to state it. We've been looking into how to 
> implement extlang at Google, and it has a number of complications when 
> you start to look at actual implementations. We want the default 
> fallback to work well, but using the extlang model we get the 
> equivalent of the "suppress script" problem.
>
> Right now, 99% of the usage of zh means Mandarin, 99% of the usage of 
> ar means standard Arabic (fuṣḥā) If we introduce zh-cmn and ar-arb it 
> complicates lookup considerably. Compatibility forces us to treat "ar" 
> and "ar-arb" as equivalents, which means we already have non-trivial 
> lookup, since we have to have all child languages match, eg 
> ar-arb-Arab-EG and ar-Arab-EG, etc. Encoding arq as ar-arq to do a bit 
> of magic in fallback isn't worth the trouble introduced by that. It 
> may well be that arb is the best fallback for arq, but it may also be 
> that nb is the best fallback for da, which we don't try to incorporate 
> in the language tags themselves. That is, trying to guess what the 
> best fallback is AND bake it into the encoding of the tags itself for 
> the case of macro languages is simply a complication for little benefit.
>
> On 3/17/07, *Doug Ewell* <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Mark Davis wrote:
>
>     <http://users.adelphia.net/%7Edewell/>

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 18 Mar 2007 09:47:10 -0700

Gerard Meijssen <gerard dot meijssen at gmail dot com> wrote:

> You can explain that zh is old and Mandarin can be used as zh-cmn and 
> cmn as a consequence. People will opt for cmn if they have the option 
> and to it right, they will still opt for cmn and you would call them 
> wrong.

This continues to be a misrepresentation of the nature and intent of 
macrolanguages.  It is not a question of one being "old" and the other 
being "new."  If it were, the "old" code element would not exist in ISO 
639-3.

The ISO 639-3/RA has determined that for some languages, including 
Chinese and I guess 53 others, it is suitable or common in some 
situations to think of them as a single language and in other situations 
as multiple languages.  That is the decision they have made, based on 
their knowledge and research.  I do not consider it my job, at least, to 
second-guess their knowledge or research about Chinese and declare that 
it is never appropriate to think of Chinese as a single language, nor to 
"bake" (as Mark would say) my opinion into the tagging standard.

---

From: Gerard Meijssen <[email-masked]>
Date: Sun, 18 Mar 2007 18:01:12 +0100

Hoi,
The point was if it makes sense to allow for both zh-cmn and cmn. The 
fact that some languages are considered together a single language is 
very much beside the point. I will not argue that. One point is that 
computers can easily equate these two codes as being the same.

The other point to my argument is not addressed; this is that it is 
easier to explain to people; a single code per language and for 
compatibilities sake, zh is understood to be all these languages and is 
therefore not as discrete it can be appended with the later code. Of 
importance is that in this way more content will be tagged in a way that 
is correct. When you allow for one language one tag, you make it easier 
for people to do a right thing.

Thanks,
    GerardM

Doug Ewell schreef:
> being "new."  If it were, the "old" code element would not exist in 
> ISO 639-3.
>
> situations to think of them as a single language and in other 
> situations as multiple languages.  That is the decision they have 
> made, based on their knowledge and research.  I do not consider it my 
> job, at least, to second-guess their knowledge or research about 
> Chinese and declare that it is never appropriate to think of Chinese 
> as a single language, nor to "bake" (as Mark would say) my opinion 
> into the tagging standard.
>

---

From: Martin Duerst <[email-masked]>
Date: Thu, 22 Mar 2007 17:54:11 +0900

At 21:16 07/03/18, Gerard Meijssen wrote:
>Hoi,
>Modern computers are quite capable of this "non-trivial" lookup that you refer to.

On the level of implementation in a single single program, this is
definitely true. 

But what we have to consider is world-wide development (to some extent,
every Web browser and server, and lots of other software). At this level,
things look absolutely, totally, different. Things that look extremely
simple.

>The reason why we should allow for both zh-cmn and cmn is because it is much easier to explain to *people*. I utterly disagree that having both options is of little benefit. It helps people; it helps people understand the code. We give meaning to these codes and it is extremely hard to explain how these codes work. For people it makes a hell of a difference. You can explain that zh is old and Mandarin can be used as zh-cmn and cmn as a consequence. People will opt for cmn if they have the option and to it right, they will still opt for cmn and you would call them wrong.

I think that any way you slice it, some people will follow, and
others won't. And some people will understand things one way
("this is old and this is new") where others will understand
things another way ("of course Mandarin is Chinese, so zh-cmn
makes perfect sense"). To predict which one will work better
is extremely difficult.

The important thing for us, when creating a standard,
is to try hard to create only as many promises as we think can
reasonably be kept. To use the above example, even if a lot of people
may just choose 'cmn', it is way better for us to say that they
should use 'zh-cmn' to make sure things match with 'zh' than to
define that 'zh-cmn' and 'cmn', or 'zh-cmn' and 'cmn', match
when we don't have the confidence that these matches will actually
work in wide practice.

>We get a much better adoption by being understood. It is more important than having computers to know that two codes are equivalent. Computers are good at that !!

On a small level (one human, one computer), yes. In the big picture,
things may look quite a bit different.

Regards,     Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp      mailto:[email-masked]