Subject: [Ltru] my technical position on extlang

From: "Mark Davis" <[email-masked]>
Date: Sun, 18 May 2008 11:49:42 -0700

Here are some thoughts on extlang. The more readable version is at:

http://docs.google.com/Doc?docid=dfqr8rd5_676kxxxjhd&hl=en

Copied here for the archive:

Extlang

The arguments for extlang are that they give superior results, and are thus
worth the complication of having some languages be unavailable as langtag,
and only in a secondary position. I believe that when extlang is examined
carefully, by people who have implemented language tag lookup, that on
balance most people will be worse off than if we retain the structure of
RFC4646, and do not complicate the structure to the overall detriment of
implementations, do not make encompassed languages be in the inferior,
extlang position.

* Links*

   - This document: http://docs.google.com/Doc?id=dfqr8rd5_676kxxxjhd
   - BCP 47: http://www.rfc-editor.org/rfc/bcp/bcp47.txt
   - Current draft:
   http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-14.html
   - Macrolanguage mappings <http://www.sil.org/iso639-3/macrolanguages.asp>

 Process

I looked back over the emails, and people may not remember everything that
was discussed in the emails around the time that we came to (rough)
consensus on extlang. While Shawn claimed that the topic should be reopened
-- well after the last call went out -- as far as I can tell there is in
fact *no* new information being presented that was not available on the list
6 months ago.

   - everything mentioned recently about the use of 'zh' and 'cmn' was
   discussed long ago in the email of September through December last year.
   - the teleconferences were discussed at length during September (see "[Ltru]
   70th IETF - ltru session?", and we ended up with a few sessions. We made
   efforts to have them at times that would work for all those interested in
   discussing the issue; we also had jabber available for those who couldn't
   phone in. Nobody objected to the time except for Peter, and we delayed a
   week so that we could then include him. A number of people from both 'camps'
   evidenced interest in having them. We took notes and distributed them. (Doug
   bailed out at one point: "Go ahead and include me out of future
   teleconferences, and feel free to move them back to a more comfortable time
   for all. ")
   - as a result of those telecons, we were able to come to rough consensus
   - by December, we were moving on to whether to remove extlang from the
   ABNF or not, and settled on doing that.
   - the results of the telecon were announced on the list. Nothing
   prevented people who might have felt left out from contesting the results at
   that point or any time since, including the last call.
   - there have been no new technical reasons provided since December that
   would give us any reason for overturning the result of the telecon, nor for
   believing that we would get consensus for having encompassed languages be
   extlangs in RFC4646bis
   - moreover, we now have workable text in draft 14 for handling the
   macrolanguage/encompassed language issues without recourse to a new,
   untested mechanism.

Since this issue seems to be reopening (for no good reason that I can see),
I put together some responses from past emails on the topic. They are not
wonderfully organized; I just tried to cast them as Q/A pairs to make them a
bit more readable. There is definitely some repetition that ought to be
edited away. The tone may seem too harsh at times -- sometimes that was in
the heat of the moment, and I haven't had time to moderate the text, so I
apologize in advance for any offense I may give.
 Q. Where would extlang make a difference?
 A. Where As RFC 4647 describes, there are two main processes for matching
using language tags, filtering and lookup. There is reason to look at how
the two different macrolanguage models in affect 4647. If primary reason
cited for the extlang model is compatibility, we have to see what effects
each model has on commonly used matching functions, which is where the
rubber hits the road.

For filtering, extlang offers no particular advantage. Let's look at queries
of "ar-ary" Moroccan Arabic vs "ary". In either case I need a way to match
all and only Moroccan Arabic; I must not fallback to "ar". If I fallback to
include all Arabic, the actual content that is in Moroccan Arabic would be
completely and utterly swamped by Standard Arabic. So for filtering, the
extlang model just gives us a more complicated syntax, with no benefit.

The only possible advantage of extlang would be in Lookup.
 Q. Isn't extlang better for lookup? A. Take an example

   - My site has support for zh, zh-Hans, and zh-Hant. zh has Mandarin
   content, since that is what 99.999% of web sites mean by "zh" currently. As
   is customary in fallback, my site's 'zh' content uses the predominent form
   (zh-Hans) [this is just for example; a TW site can use the opposite
   convention.]
   - A user comes in with different requests, listed below.

* Scenario 1. *The user's browser has the proposed "zh-yue-Hant-US". My
lookup falls back to zh, so I serve it up to the user. So even if the target
of the match (zh) is not Cantonese, you want a fallback to zh. I'm guessing
that you see this as better than if we defined the tag as "yue-Hant-US",
since it gets to some fallback that the user is likely to understand. But I
don't see this as much different than if we had fr-br-BE (meaning Breton,
but fall back to French), or ro-mo (meaning Moldavian, but falling back to
Romanian).* And note that in the fallback, the script and region are
completely lost.*

* Scenario 2. *The user's browser has zh-cmn-Hant-US. In matching, we fall
back to zh. Note than in the fallback, the script and region are completely
lost. *We have essentially just introduced a synonym for zh which causes
fallback to lose information, for no good reason. *
 Q. What's wrong with simply including extlang?
A. If we bake it in, then every simple algorithm will in practice
automatically fall back from Cantonese to Mandarin, fall back from Dari to
Persian, fall back from Khetrani to Lahnda -- and in doing so, strip the
script and country information. That, unless they fix the algorithm to
pretend that the secondary language is in fact a primary language. So we are
forcing people into a model that is often, or mostly, wrong.

*If we supply the information in the registry, then implementations can
choose whatever they think is appropriate, given the particular facts about
languages and the particular needs of their applications without having to
work around the **extlang** mechanism.*

I'm reminded again of a similar case with C++. The assignment operator gets
a default implementation. That must have seemed like a nice convenience for
the user, but *except for toy programs, it is always, always wrong. * So
supplying that default just means that people usually have to take extra
steps to disable it, and prevent it from causing bugs in their programs. I'm
worried about this being similar.

It is clear that companies like Google or Yahoo can work around the problems
with extlang-- what I'm worried about are the people who don't have a lot of
experience with these matters, and are just led down a garden path. We need
to look long and hard at the experience of people who have had detailed
implementation experience with filtering and matching these tags in
production environments.
 Q. Where are some cases where extlang works particularly badly?
Extlang plays especially badly in many cases. Suppose that we have
macrolanguage m1, and microlanguages x1 and x2. By the design of ISO 639, we
*can't* assume that a speaker of x1 can also speaker x2 or vice versa. If a
user has as accept language the list <x1-Ssss-Rr, en, fr>, it works fine
without extlang: she gets the fallback

(A) Script fallback

   1. x1-Ssss-Rr
   2. x1-Ssss
   3. x1
   4. en
   5. fr

If she also speaks/reads x2, then she can specify <x1-Ssss-Rr, x2, en, fr>
or <x1-Ssss-Rr, en, fr, x2>; that is, putting x2 in the list in the position
she wants it. If x2 is the predominant microlanguage, meaning that m1 is
essentially always assumed to be x2, then the priority list can be
<x1-Ssss-Rr, en, fr, m1>, also wherever it belongs. Thus, the user gets
something she can understand, based on the list she supplied.

If we are using extlang, and x2 is the content for m1, then we get the
fallback

(B) Script + extlang fallback

   1. m1-x1-Ssss-Rr
   2. m1-x1-Ssss
   3. m1-x1
   4. m1
   5. en
   6. fr

That has two problems: first, the script and region are lost. That can be
fixed by hacking the fallback (although there is a *lot *of installed base
that won't do this), to

(C) Hacked Script + extlang fallback

   1. m1-x1-Ssss-Rr
   2. m1-x1-Ssss
   3. m1-x1
   4. m1-Ssss-Rr
   5. m1-Ssss
   6. m1
   7. en
   8. fr

But even more importantly, we are disabling the user's explicit choice. If
the user doesn't speak x2 (or whatever the content of m1 is), he's screwed.
There is no way that he can indicate that he wants x1 * but no other version
of m1 because he can't understand them.
*
We'd have to change the fallback to be quite substantially different to get
around this, with

(D) More Hacked Script + extlang fallback

   1. m1-x1-Ssss-Rr
   2. m1-x1-Ssss
   3. m1-x1
   4. en
   5. fr
   6. m1-Ssss-Rr
   7. m1-Ssss
   8. m1

This is, however, still only appropriate if it is likely that a user of x1
speaks *whatever happens to be the content for m1*. That is an extremely
shaky assumption.

If Peter Constable said, for each and every macrolanguage on
http://www.sil.org/iso639-3/macrolanguages.asp, there is at least one
microlanguage that all speakers (or even most speakers) of each of the other
microlanguages would understand, I'd say: fine, let's do extlang and
incorporate that information into the registry, with the "default
microlanguage" for each macrolanguage. Then, for example, implementers would
know that, say " fuf
<http://www.sil.org/iso639-3/documentation.asp?id=fuf>Pular" is
understood by all the speakers of the microlanguages under
"ful <http://www.sil.org/iso639-3/documentation.asp?id=ful> Fulah", so we
can tell people to always have the content of "ful" be "fuf", and bake the
macrolanguages in as extlang.

The point of the suggested text is that if your application wants to use
macrolanguages to support extlang-equivalent fallback, there is nothing
stopping you from doing so. If there are particular environments where an
extlang-like fallback is right for a particular language community, it is
simple to do. But we don't need to bake shaky assumptions into the structure
of language tags.
 Q. Isn't extlang just like script fallback?
A. The problem with extlang is that the fallback from encompassed language
to macrolanguage is fundamentally different in kind than a fallback from
region to script to base language. In the case of script, like uz-Arab and
uz-Latn, or en-US vs en-GB, we really have variations on the same language,
and fallback makes sense. We ordered the subtags so that it works optimally
overall.

The encompassed languages, on the other hand, are not just dialects, not
just variants. *They are languages in their own right. *Trying to insert
them into the fallback process just screws things up, because they need a
"sideways" matching not just simple truncation fallback. If you want to do
any fallback with extlang, it would be to fall back from zh-yue-<other
stuff> to zh-<other stuff>. That means that in order to do reasonable
fallback, you can't just use truncation fallback anyway. So I see the
situation this way:

   1. The only reason for adding the complication of the extlang mechanism
   is to make truncation fallback work better.
   2. Truncation fallback with extlang doesn't work better.
   3. So there is no need to make encompassed languages be "secondary"
   languages by making them be "secondary" subtags.

The goals of extlang are good, to make matching work better, but in practice
it just makes things worse. [Speaking to those familiar with C++, it feels a
bit like the default assignment operator in C++. Nice in theory, but in
practice it gums things up more than it fixes, since once you are beyond
very simple (toy) classes, the default is almost always wrong -- but because
it is supplied behind your back you don't realize it.]

So instead of adding the extlang mechanism to RFC 4646, what we really need
to do is to point people to how to handle yue and other encompassed
languages along with mo/ro, tl/fil, and other edge cases in a reasonable
way, by augmenting matching.
 Q. Where might the macrolanguage be useful? A. An implementation may choose
to use that information in falling back from some encompassed languages to
macro languages. For example, given the language priority list with
Cantonese in Traditional Script as used in Hong Kong, followed by French
("yue-Hant-HK, fr"), the lookup could be the following:

1. yue-Hant-HK
2. yue-Hant
3. yue-HK
4. fr
5. implementation defined default:
  5a. zh-Hant-Hk
  5b. zh-Hant
  5c. zh
  5d. en

Whether such fallback should be used -- and if so, the precise way in which
such a fallback is done -- is application-dependent.  Where it is very
likely that the audience requesting Cantonese (as above) will accept and
understand Mandarin (the predominant content for 'zh'), then this fallback
might be useful. Where there is risk that that the audience requesting
Cantonese will not be conversant with Mandarin, and would prefer an
alternative in the language priority list, it should be avoided. (This might
be the case, for example, with audio using yue-Zxxx-US.
 Q. Why not have extlang for using macrolanguages if the suggested text adds
macrolanguages back into the fallback chain?
A. The suggested text doesn't add it back in. It only says that *IF *an
application wants to do extlang-equivalent fallback, the text in BCP 47
already allows for that.

We really have *no* idea whether using macro languages in the fallback chain
is a good idea or not. Some people think it will be an advantage for a few
specific examples that are cited. (Cantonese comes up, but when I tested
some of the assumptions with Cantonese speakers, they didn't quite hold up.)
But nobody has substantiated that it will give better results for all or
most macrolanguages. Or any indication of an even rough list of those for
which it will be better. Nor has anyone effectively argued that the
situation between yue and zh is substantially different than the situation
between gsw and de, where we get along just fine without extlang.

So the suggested text just provides it as an option, and leaves it up to the
application.
 Q. How should we look at macrolanguages?
I think one of the things that we realized when looking at how this would
work in practice is that we are better off if we treat macrolanguage as a
piece of *perhaps* information for matching, but one that can be enhanced
(that is, changed) over time as more information becomes available.
Hard-coding it into extlang doesn't serve that purpose, and causes other
problems, notably that the other fields are lost in fallback: if we had
zh-yue-Hant, then by the time we get to zh in fallback, we've lost the Hant.

So that is the origin of the text we are proposing.

There are a number of edge cases such as deprecated codes, closely related
languages, or practical fallbacks (eg if someone speaks X they are likely to
speak Y, even if X and Y are not linguistically related) that are simply
unsuited to hard-coding in the tag. If I hit a code like "iw-PL", I want to
match that with "he-PL", not depend on some kind of fallback between "iw"
and "he"; otherwise it loses information (the PL). We do provide that kind
of information in the deprecated field, and with the macrolanguage field we
would provide more (for example, it would make clear the relation between
no, nb, and nn and how that could be used in matching). Other useful
information is the scripts used with a language in practice; suppress script
supplies just a little information, but doesn't tell me that Uzbek is
customarily written with Arabic, Latin, or Cyrillic, but not with (say)
Tagalog.

The more information there is available, whether it be in the language
subtag registry or somewhere else, the better a job people can do in dealing
with some of the edge cases that turn up in matching.

Q. Doesn't the macrolanguage relationship uniquely define the best fallback?

A. Certain languages are closely related, and the lookup process may take
that into account. Macrolanguage is just one factor that may (or might not)
be useful. For example, since the the tag "gsw-CH" (for Swiss German as used
in Switzerland) was first available on 2006-12-08, Swiss German
("Schwyzerduetsch") text may have been tagged with "de-CH" instead. ISO 639
was not (and is still not) clear on whether "de" meant only High German or
also included variants such as Low German or not. Thus Swiss German material
may have been, and may still be tagged with "de". Essentially all Swiss
German speakers are comfortable in High German, so where Swiss German is not
available, High German is a very good fallback. Thus when given the language
priority list: "gsw-CH, fr-CH", an implementation using lookup may augment
the default values to also include the lookup of related values, such as the
following search order:

1. gsw-CH
2. gsw
3. fr-CH // next language
4. fr
5. implementation defined default:
    5a. de-CH // special fallback from gsw-CH
    5b. de
    5c. en // root

In this way, other likely possibilities are tried before the final fallback
to the root value. Note that typically the fallback to related languages
should include the script and region codes if available.

In this way, the lookup process may take into account what languages people
are likely to understand, given a language priority list. Similarly, the
close relations between Romanian and Moldavian, Tagalog and Filipino,
Serbo-Croatian and Croatian, and so on may all be useful in doing related
language lookup.

This is not restricted to related languages. For example, a Breton speaker
is very likely to also understand French, given the language priority list.
Thus the implementation may choose to use the following lookup for the
language priority list "br-FR, de":

1. br-FR
2. br
3. de
4. implementation defined default:
    4a. fr-FR // special fallback from br (Breton)
    4b. fr
    4c. en
Q. I see "zh" and "cmn", I have no way of telling that they're related
without looking at the registry, which basically means a hard-coded table.
If "zh" is preferred", then I may want to move from "zh" to "cmn" or
whatever. A. *You can't tell that from the **registry** either! *Sometimes
microlanguages are related in such a way as to be a good fallback, but *usually
they are not.* The handful of actual cases where we think it might be a good
idea are listed in the text around Table 8, *not* in the registry. Knowing
that X is a macro/micro language does not necessarily mean -- *and usually
doesn't mean* -- that you want to use it in fallbacks. If there is no
predominant form, then it's a crapshoot as to whether the
macro/microlanguage is a good fallback.

There is no special runtime information in the registry. When a new
macro/microlanguage shows up in the registryregistry macro/microlanguages is
not a good idea. Nor is it complete, since it misses tl/fil, ro/mo, and many
others that are much higher frequency cases than most of the
macro/microlanguages.

If an implementation provides a UI for selecting language priority lists, it
may be better to give the user the option of having explicit fallbacks (such
as from Cantonese to Mandarin or Tagalog to Filipino), rather than trying to
guess the user's intent (and run the distinct risk of getting it wrong). For
that purpose, when a user adds a language to the priority list, the UI may
suggest macrolanguages, or other related languages, as additional fallbacks.
*and* you support the one of the pair, then it may be useful to review
whether or not you want to add a fallback. Automatically updating fallbacks
blindly according to the Q. In lookup, if there is a predominant form how is
it best (in practice) to deal with the macrolanguage? For most programs, I
believe that treating them as synonyms is the right thing to do, and
alternative approaches would be extremely counterproductive. And this goes
for any of the macrolanguage cases where there is a predominant encompassed
language with long usage in the computer industry.

Let's take a scenario where this is not done (a la Ewell). Suppose that a
user picks Arabic as her Accept-Language in her browser. Any existing
browser will represent that with "ar". Then she goes to the BBC site. The
entire site is translated, not into standard Arabic, but into Sudanese
Creole Arabic. The user complains, since she can't understand it, and the
BBC responds that they are just following the standard to the letter and
spirit: "ar" means any kind of Arabic whatsoever, and so in the interests of
fairness, they pick a different encompassed language to serve up each day.
They inform the user that it is her fault for using 'ar' if she really only
wants Standard Arabic. So they have the following schedule:

  Monday
 aao <http://www.sil.org/iso639-3/documentation.asp?id=aao>  Algerian
Saharan Arabic   Tuesday
 abh <http://www.sil.org/iso639-3/documentation.asp?id=abh>  Tajiki
Arabic  Wednesday
abv <http://www.sil.org/iso639-3/documentation.asp?id=abv>  Baharna
Arabic   ...
acm <http://www.sil.org/iso639-3/documentation.asp?id=acm>  Mesopotamian
Arabic      acq <http://www.sil.org/iso639-3/documentation.asp?id=acq>
Ta'izzi-Adeni Arabic
acw <http://www.sil.org/iso639-3/documentation.asp?id=acw>  Hijazi Arabic
acx <http://www.sil.org/iso639-3/documentation.asp?id=acx>  Omani Arabic
 acy <http://www.sil.org/iso639-3/documentation.asp?id=acy>  Cypriot Arabic
adf <http://www.sil.org/iso639-3/documentation.asp?id=adf>  Dhofari Arabic
aeb <http://www.sil.org/iso639-3/documentation.asp?id=aeb>  Tunisian Arabic
aec <http://www.sil.org/iso639-3/documentation.asp?id=aec>  Saidi Arabic
 afb <http://www.sil.org/iso639-3/documentation.asp?id=afb>  Gulf Arabic
 ajp <http://www.sil.org/iso639-3/documentation.asp?id=ajp>  South Levantine
Arabic      apc <http://www.sil.org/iso639-3/documentation.asp?id=apc>
North Levantine Arabic
apd <http://www.sil.org/iso639-3/documentation.asp?id=apd>  Sudanese Arabic
arb <http://www.sil.org/iso639-3/documentation.asp?id=arb>  Standard Arabic
arq <http://www.sil.org/iso639-3/documentation.asp?id=arq>  Algerian Arabic
ars <http://www.sil.org/iso639-3/documentation.asp?id=ars>  Najdi Arabic
 ary <http://www.sil.org/iso639-3/documentation.asp?id=ary>  Moroccan Arabic
     arz <http://www.sil.org/iso639-3/documentation.asp?id=arz>  Egyptian
Arabic      auz <http://www.sil.org/iso639-3/documentation.asp?id=auz>
Uzbeki Arabic
avl <http://www.sil.org/iso639-3/documentation.asp?id=avl>  Eastern Egyptian
Bedawi Arabic
ayh<http://www.sil.org/iso639-3/documentation.asp?id=ayh> Hadrami
Arabic
ayl <http://www.sil.org/iso639-3/documentation.asp?id=ayl>  Libyan Arabic
ayn <http://www.sil.org/iso639-3/documentation.asp?id=ayn>  Sanaani Arabic
ayp <http://www.sil.org/iso639-3/documentation.asp?id=ayp>  North
Mesopotamian Arabic
bbz<http://www.sil.org/iso639-3/documentation.asp?id=bbz> Babalia
Creole Arabic
pga <http://www.sil.org/iso639-3/documentation.asp?id=pga>  Sudanese Creole
Arabic      shu <http://www.sil.org/iso639-3/documentation.asp?id=shu>
Chadian Arabic
ssh <http://www.sil.org/iso639-3/documentation.asp?id=ssh>  Shihhi Arabic

If 'ar' means any Arabic, without any preference, this would be a perfectly
reasonable thing to do. But for users, it would hardly be satisfactory. And
this would be a bizarrely stupid thing for the BBC to do.

Someone might respond that, well, everyone needs to convert over to cmn for
Mandarin since it is now the Right Thing to Do. Even if that magically
happened, it would take years, and during the transition we would get all
kinds of screwups with different programs transitioning at different paces.
And there isn't much magic around; it is very hard to get people to change
infrastructure that works just fine -- you have to give them a compelling
case for why users are served better by the change. And that would be a very
hard sell, since there isn't any real advantage.

The right approach for the BBC is to treat a request 'ar' as a request for
Standard Arabic*, just as they have always done*. Internally, that means
treating 'ar' and 'arb', and any language tag that starts with them, as a
request for Standard Arabic. That is, treating ar-EG and arb-EG, or ar-SA
and arb-SA, or other combinations, as synonyms for the purpose of lookup.
Now, this "treating as synonyms" could be done in different ways. One way is
to mash on input; the other is to have a fancier fallback, eg arb-EG =>
ar-EG => arb => ar (mutatis mutandis, when starting with ar-EG).

Nor was this solved at all by extlang -- as we discussed at some length, it
discarded all script and region info when falling back, and produces worse
results in many cases, especially where the macrolanguage does not have a
predominant form.

The "treating as synonyms" strategy is always going to be the right answer.
There are undoubtedly scenarios where this strategy is not necessary,
although I can't think of any off the top of my head.

Moreover, I think one of the more productive things we can do is to push for
the incorporation of Language Priority Lists in any query-like protocols.
That way I could say I'd like "ary, fr, ar" if my preferred ordering is
Moroccan Arabic, then French, then as a last resort, Standard Arabic
Q. How do I use fancier fallback for the predominant form?
Here is a more detailed case

   1. cmn-Hant-HK
   2. zh-Hant-Hk
   3. cmn-Hant
   4. zh-Hant
   5. yue-HK
   6. zh-HK
   7. yue
   8. zh

 Q. Haven't people always interpreted zh as meaning anything from Mandarin
to Hakka to Min?
A. That's very unclear. People usually choose 'zh' not literally, but
through a UI that shows a human readable form. So the question is, how many
people have looked at interfaces that say simply " 中文" and think

   1. "that could mean Mandarin but could also mean Hakka" vs
   2. "that means just Mandarin, they don't offer Hakka so I'll pick
   something else", vs
   3. "that means just Mandarin, they don't offer Hakka, but Mandarin is the
   closest to Hakka that is offered so I'll pick that."

In lookup on computer systems it is clear that nobody's expectation is that
by picking 'zh', they will get Hakka. And anything but Mandarin is a
vanishingly small percentage of tagged text; the same is true of 'ar';
anything but Standard Arabic is a vanishingly small percentage.

As Karen said on a related topic: *"My experience is that the users who need
to specify Cantonese most often make up an **illegal** tag. Not saying
that's what we should recommend, but I believe my experience does not
support the statement as worded. "*

Q. Isn't the written form of Cantonese the same as Mandarin?A. No, no more
than the "written form of Swiss German is the same as High German". What is
true is that when the Swiss write, they write in High German; that's
different. They are using different words than what is spoken, and very
different syntax: "I bi doo gsy." => "Ich war hier."

Here is what I have on the subject from John Jenkins:

I believe you said that a Mandarin speaker can read written Cantonese, but
will not understand everything (a bit like a Dane reading Swedish).

More like French and Spanish, actually.

Some characters would not (normally) be used in Mandarin.

Most famously U+4E5C, the Cantonese for "what".

Some characters would have different meanings than in Mandarin

Best illustrated with U+4FC2, which means "to bind" in Mandarin but is
frequently borrowed to for the Cantonese word for "to be."

Some syntax would be different

Yes, but I can't think of any examples off the top of my head and the book
I've got that lists the differences is successfully playing hide-and-seek at
the moment.  There are actually not an awful lot of these.  The main
differences between the two are phonetic and lexical.  The grammars are very
similar.

Can you point me to some web pages with written Cantonese that would
demonstrate that to a Chinese reader?

Nothing better to start with than the Cantonese Wikipedia article on
Cantonese:  <http://zh-yue.wikipedia.org/wiki/粵語<http://zh-yue.wikipedia.org/wiki/%E7%B2%B5%E8%AA%9E>>.
Similarly, <http://zh-yue.wikipedia.org/wiki/香港<http://zh-yue.wikipedia.org/wiki/%E9%A6%99%E6%B8%AF>>,
<http://zh-yue.wikipedia.org/wiki/Unicode>, and pretty much everything else
in the Cantonese Wikipedia.

I can give you the relative character frequencies in the Cantonese and
traditional (or simplified Chinese) Wikipediae, if you like.  I've still got
that data around somewhere.

*The thing you have to emphasize is the difference between what-**Cantonese*
*-speakers-generally-read-and-write, which is just Mandardin with **
Cantonese** phonetics, and
writing-down-what-**Cantonese**-speakers-actually-speak,
which is what "written **Cantonese**" should be used to mean.
 Unfortunately, not everybody groks this.  Fortunately, Wikipedia does.*

Meanwhile, I quote from Stephen Matthews and Virginia Yip, _Cantonese: A
Comprehensive Grammar_ (London: Routledge, 1994), pp. 5-6:

"Traditionally, Cantonese has been regarded as one of the many Chinese
dialects. It does not have a standardized written form on a par with
standard written Chinese.  No form of written Cantonese is taught in schools
or used in academic settings in any Cantonese-speaking community.  When it
comes to the written form, it is standard written Chinese that is taught and
learnt.  For educated Cantonese speakers, standard written Chinese is the
written form they use in most contexts.  However, in colloquial genres such
as novels, popular magazines, newspaper gossip columns, informal personal
communications, written Cantonese may be used.  When written Cantonese
Cantonese words and expressions, non-Cantonese speakers may find it totally
unintelligible."   contains too many exclusively

Since they wrote, however, there's been a distinct upsurge in the use of
written Cantonese.  (It's tied in with a kind of Hong Kongese
pseudo-nationalism.)  It's still not exactly *common*, but it's a lot more
common than it used to be.

-- 
Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Fri, 23 May 2008 00:43:05 -0400

> extlang model just gives us a more complicated syntax, with no benefit.

I agree with most of your document, which is almost entirely about
lookup, but I can't agree with your characterization of filtering.
It would apply to filtering applied to a search engine, but a search
engine is a very atypical case of a Web server -- its content is vast
and not controlled by the server's owner except in the broadest sense.

Filtering is the algorithm applied to language-tag negotiation by the HTTP
RFC (2616); indeed, 4647's simple filtering algorithm is taken directly
from 2616.  So as an IETF group we must take filtering seriously, and
it is my contention that using extlang syntax is a big win for filtering.

Remember that the operational difference between filtering and lookup
is that in filtering, a range with *fewer* subtags can match a resource
tagged with *more* subtags; whereas in lookup, a range with *more*
subtags can match a resource tagged with *fewer* subtags if no longer
match is available.

Suppose we are downloading Arabic-language audio, and we wish to receive
Egyptian Arabic by choice; failing that, Standard Arabic; and failing
that, any available Arabic variety except Shuwa Arabic, which we do not
understand at all.  We can inform the server of our wishes using the
following header:

	Accept-Language: ar-arz, ar-arb, ar, ar-shw;q=0

A 2616-conformant server will apply the filtering algorithm and return
anything beginning with "ar-arz"; if none, then anything beginning with
"ar-arb"; if none, then anything beginning with "ar" except something
beginning with "ar-shw".  (Some web servers ignore "q=0", making the
exclusion of Shuwa not work, but that's just a bug; RFC 2616 section
3.9 is clear that ";q=0" means content so tagged is not acceptable to
the client.)

What is more (and this is key) if the server has more than one match
for ar-arb, say ar-arb-koranic and ar-arb-modern, it can return either,
or return a disambiguation page allowing the user to choose.  This is
entirely different from lookup, where at most one resource must be
returned, and we remove subtags until we get an exact match.

Now what if extlang syntax is not used?  Then we must say

	Accept-Language: arz, arb, ar, shw;q=0

And in order for this to mean what it is intended to mean, the server
must extend RFC 2616 and treat filtering for "ar" as magically matching
resources that begin with "arq", "aao", "bbz", ..., "aeb", "auz" (see
http://www.sil.org/iso639-3/documentation.asp?id=ara ).  Either that,
or the user must specify all these tags in Accept-Language.  What's more,
if the list grows, the server or the client must be updated.  Whereas if
all Arabics begin with "ar", there is no such issue.

Obviously extlang syntax won't solve *all* problems with filtering,
but it will help a great deal with some important cases.  That's why I
think the issue needs to be reconsidered.

-- 
If you understand,                      John Cowan
   things are just as they are;         http://www.ccil.org/~cowan
if you do not understand,               [email-masked]
   things are just as they are.
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 23 May 2008 08:51:15 -0700

The contrast is between:

   - ar-arz, ar-arb, ar, ar-shu;q=0 // with extlang=macrolanguage
   - arz, arb, ar, shu;q=0 // without extlang=macrolanguage, eg plain RFC
   4646

There are two cases where people use lists, filtering and content
negotiation.

   1. For content negotiation most people use some form of fallback
   (implicit or explicit). For example, if an Accept-Language list is handed to
   a Java implementation, many implementations will actually do fallback on the
   items.
   2. We'd have to look at how Accept-Language is implemented in practice.
   The spec is pretty lame, and a lot of implementations just disregard the q
   values. Take Tomcat, for example. With the Accept-Language string "ar-arz,
   ar-arb, ar, ar-shu;q=0, de;q=0.5", it gets the locales locales: [ar_ARZ,
   ar_ARB, ar, ar_SHU, de]. Note that they are not even in order or precedence,
   and that mentioning ar-shu actually includes it explicitly instead of
   removing it!!!
   3. Even if they pay attention to q values, normally implementations just
   reorder the list, then walk through the list and pick the first one that
   works. By your definition, the content for "ar" could be any of the
   microlanguages, including "arb" but also "shu" or "pga", so this would not
   control that unless there were some metalanguage in the system that said
   that the content for "ar" happened to be "shu". I don't know of anyone that
   does that, or that it would ever be worth the effort to support.
      - Interpreting 'ar' as Standard Arabic is permissible now and after,
      whatever happens with extlang. With the above list, the practical impact
      will be that it will stop at 'ar' anyway and return Standard Arabic. The
      odds of an implementation supporting "pga" but not ar/arb are
pretty darn'd
      low. In practice, of course, very implementation that I know of
will treat
      'ar' as simply Standard Arabic.

Look then at filtering.

Even if the server doesn't support extlang matching, you can get exactly
what you said you want -- reliably -- with an explicit list:

arz, arb, ar, arq, aao, bbz, abv, acy, adf, avl, afb, ayh, acw, ayl, acm,
ary, ars, apc, ayp, acx, aec, ayn, ssh, ajp, apd, pga, acq, abh, aeb, auz
// excluding shu

And this will work even if filtering isn't supported according to the spec;
and note that we have no guarantee that the server will filter according to
the spec -- based on the experience with Accept-Language, it's actually
unlikely.

And really, what are the odds that someone wants that exact list above? And
MUST have it supported with a short enumeration? And does not care if
additional encompassed languages are added?

According to information from Peter, the microlanguages are all independent
languages, meaning that they are not mutually comprehensible. Someone might
know more than one, but there is no particular expectation in the absence of
language-specific information. So while it is possible that someone happens
to know 15-odd different microlanguages under "ar" (but not "shu"), it is
pretty darned unlikely. What is more likely is that someone knows a few of
them, and simply lists the ones they know. The usefulness of having such a
shorthand is really not there.

Peter did his best with the macrolanguage construction faced with what was
there. But the information content useful to implementations are extremely
low, and cannot be relied on.

   - Languages that are *very* closely related do not have a macrolanguage
   ("ro" and "mo").
   - Languages that are mutually incomprehensible do share a macrolanguage
   (most examples).

I don't see extlang as any really practical benefit in filtering. It isn't
useful in a great many other cases of related languages; if I want to use
with nn, nb, no it doesn't help, nor with ro/mo, various varieties of
German, and so on. Moreover, if a customer searches for 'ar' and get a bunch
of documents in shu, they are as or more likely to be unhappy as if they
search for 'de' documents and get back a bunch of 'gsw'.

Extlang is really just not a good mechanism for implementation in terms of
getting good matches, and baking that into the syntax just makes it worse.
It is a negative in lookup, and no positive for filtering either.

Putting some ISO language tags into the extlang position just because they
have a macrolanguage is an unnecessary complication for implementations and
represents a substantive, controversial change to RFC 4646.

Mark

On Thu, May 22, 2008 at 9:43 PM, John Cowan <[email-masked]> wrote:

>   things are just as they are;         http://www.ccil.org/~cowan<http://www.ccil.org/%7Ecowan>

---

From: John Cowan <[email-masked]>
Date: Fri, 23 May 2008 12:09:05 -0400

Mark Davis scripsit:

[points 1, 2, 3a snipped]

These all amount to "Some implementations are buggy and don't follow
RFC 2616."  Such bugs that could be fixed at any time, in which
case the servers would come into compliance.  Historical practice in
tagging documents is important, buggy protocol implementations are not.
(A variant of the ISO C motto about "code matters, implementations
don't".)

> [3b]  - Interpreting 'ar' as Standard Arabic is permissible now
> and after,

Certainly.

> With the above list, the practical impact will be that it will stop at
> 'ar' anyway and return Standard Arabic.

As I say, that's a clear violation of the RFC.

> The odds of an implementation supporting "pga" but not ar/arb are
> pretty darn'd low.

Unless it just happens to be a site with lots of audio from around
the arabophone world.  *Across all sites* all non-standard Arabics are
rare, but that doesn't apply to *specific* sites that really need to do
language negotiation.

> ary, ars, apc, ayp, acx, aec, ayn, ssh, ajp, apd, pga, acq, abh, aeb, auz

Which is just what I said.  However, that isn't very stable with time
as new Arabic varieties get encoded, not to mention extremely obnoxious
to the user without special support in the client.

> the spec -- based on the experience with Accept-Language, it's actually

Most authors don't tag according to the spec either, so why have language
tags at all?  Search engines can and should ignore them; that doesn't
mean they are worthless for other purposes.  We design around the central
point that people tag wisely and programs behave properly, which is not
necessarily the best *implementation* strategy for special cases like
search engines.

> And really, what are the odds that someone wants that exact list
> above? And MUST have it supported with a short enumeration? And does
> not care if additional encompassed languages are added?  According
> to information from Peter, the microlanguages are all independent
> languages, meaning that they are not mutually comprehensible.

The Arabic languages don't have full mutual intelligibility, but that's not
the same as saying intelligibility is zero.  I specifically picked Shuwa
(Chadian/Nigerian) to exclude because it's very remote from most others.

> - Languages that are *very* closely related do not have a macrolanguage

I know.  That doesn't mean not to take advantage of the ISO 639-3
information we do have.

> search for 'de' documents and get back a bunch of 'gsw'.

As I keep saying, *search* isn't the issue here.  It's *negotiation
by filtering*.

> represents a substantive, controversial change to RFC 4646.

*Now* it does, yes.  All that shows is that I should have thought of
this argument before giving in earlier.

-- 
Values of beeta will give rise to dom!          John Cowan
(5th/6th edition 'mv' said this if you tried    http://www.ccil.org/~cowan
to rename '.' or '..' entries; see              [email-masked]
http://cm.bell-labs.com/cm/cs/who/dmr/odd.html)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 23 May 2008 14:05:41 -0700

A general comment, repeated below.

People keep confusing two orthogonal concepts, and I think this is at the
root of the extlang issue. These are:

   1. "get me languages that are mutually intelligible with X" (maybe to
   degree Y), and
   2. "get me the languages that have the same macrolanguage as X"

Number 1 is very interesting, and would be very useful; but it is not at all
the same as #2. If macrolanguage were defined as #1, I would probably be all
in favor of baking it into extlangs. But is not at all the same, as many,
many examples illustrate. Moreover, "mutual intelligibility" differs whether
the content is written or spoken - forcing it to be baked into the syntax
does not allow for that difference.

Also, as I read your response, I think at least part of our apparent
differences is the use of different terminology.

I was using "content negotiation"  in the lookup sense, which is what is
typically done with Accept-Language (not always, but typically). That is,
the client is supplying a list of languages, perhaps with q values, and the
expectation is that s/he will get one thing back. For example, a web page in
one of the requested languages, but could be any sort of resource. And in
such interactions, you do want lookup on the individual items; "ar" is not
treated like "ar(-.*)"

Filtering is much different. An example would be: give me a list of all the
documents in my index that are "fr, ar, sku" that have X in field 5. In that
case, I do want to treate "ar" as if it were "ar(-.*)". Accept-Language
could also be used for this, but with a different meaning.

Bearing that in mind, let me see if I can make some useful responses.

> (A variant of the ISO C motto about "code matters, implementations

I was really referring to the ' lookup content negotiation' side of things,
which may or may not be relevant to what you are discussing. And I think
reality is not completely irrelevant to our discussion. ;-)

> As I say, that's a clear violation of the RFC.

No, not if we are talking about content negotiation (lookup) rather than
filtering. (this may be due to our differences in terminology)

> rare, but that doesn't apply to *specific* sites that really need to do

Again, if we are talking about content negotiation, where we return a 'best
fit', it is going to be a rare case where 'pga' is supported but 'ar' is
not. So if one stops at 'ar', "pga" would never be seen, whether it were
expressed as "ar-pga" or not.

If we are talking about real filtering (all documents that match the list),
see below.

> to the user without special support in the client.

But again, this is an extremely unusual query, since these languages are not
mutually comprehensible. And the very fact that you excluded "shu" in your
example means that you don't want all the things that could be encompassed
by "ar". If a new one shows up in the future, maybe you can understand it,
maybe you can't and would want it excluded like "shu". And what happens with
other macrolanguages; will a speaker of gan be able to understand hakka?
Better to just explicitly list the ones that are wanted.

> necessarily the best *implementation* strategy for special cases like

That isn't my point. My point is that the explicit list is

   - currently works.
   - handles everything that extlang could, with finer control
   - more powerful, since you can express things that extlang simply can't
   (eg "ro, mo, no, nn, nb"), and provide exactly the list you want.

> (Chadian/Nigerian) to exclude because it's very remote from most others.

People keep confusing two orthogonal concepts, and I think this is at the
root of the extlang issue.

   1. "get me languages that are mutually intelligible with X" (maybe to
   degree Y), and
   2. "get me the languages that have the same macrolanguage as X"

Number 1 is very interesting, and would be very useful; but it is not at all
the same as #2.

If macrolanguage were defined as #1, I would probably be all in favor of
baking it into extlangs. But is not at all the same.

> I know.  That doesn't mean not to take advantage of the ISO 639-3

See comment on mutual intelligibility.

> As I keep saying, *search* isn't the issue here.  It's *negotiation

See my discussion above.

> this argument before giving in earlier.

I didn't quite understand this.

> (5th/6th edition 'mv' said this if you tried    http://www.ccil.org/~cowan<http://www.ccil.org/%7Ecowan>

---

From: John Cowan <[email-masked]>
Date: Fri, 23 May 2008 18:54:00 -0400

Mark Davis scripsit:

>    1. "get me languages that are mutually intelligible with X" (maybe
>    to degree Y), and
> Number 1 is very interesting, and would be very useful; but it is
> not at all the same as #2. If macrolanguage were defined as #1, I
> would probably be all in favor of baking it into extlangs. But is
> not at all the same, as many, many examples illustrate. Moreover,
> "mutual intelligibility" differs whether the content is written or
> spoken - forcing it to be baked into the syntax does not allow for
> that difference.

This is as much as to say, Because we can't have everything, let us
have nothing.  To which I reply, The best is the enemy of the good.

> differences is the use of different terminology.

Indeed.

> typically done with Accept-Language (not always, but typically).

That just turns out not to be the case.

> That is, the client is supplying a list of languages, perhaps with q
> values, and the expectation is that s/he will get one thing back. For
> example, a web page in one of the requested languages, but could be
> any sort of resource. And in such interactions, you do want lookup on
> the individual items; "ar" is not treated like "ar(-.*)"

If you look at RFC 2616, you'll see that what it describes in section
14.4 is filtering:

   A language-range matches a language-tag if it exactly equals the tag,
   or if it exactly equals a prefix of the tag such that the first tag
   character following the prefix is "-".

That's the standard.

Now, if we look at the documented behavior of the dominant web server,
Apache, at http://httpd.apache.org/docs/2.2/content-negotiation.html ,
you'll find that it implements that standard.

It's true that if filtering fails to find anything, Apache provides
additional lookup-style fallback, thus:

	When a client requests a page on your server, but the server
	cannot find a single page that matches the Accept-language sent
	by the browser, the server will return either a "No Acceptable
	Variant" or "Multiple Choices" response to the client.

	[snipped explanation of configuring ultimate fallback]

	The server will also attempt to match language-subsets when
	no other match can be found. For example, if a client requests
	documents with the language en-GB for British English, the server
	is not normally allowed by the HTTP/1.1 standard to match that
	against a document that is marked as simply en. (Note that it is
	almost surely a configuration error to include en-GB and not en
	in the Accept-Language header, since it is very unlikely that
	a reader understands British English, but doesn't understand
	English in general. Unfortunately, many current clients have
	default configurations that resemble this.)

	However, if no other language match is possible and the server
	is about to return a "No Acceptable Variants" error or fallback
	to the LanguagePriority, the server will ignore the subset
	specification and match en-GB  against en documents. Implicitly,
	Apache will add the parent language to the client's acceptable
	language list with a very low quality value. But note that if
	the client requests "en-GB; q=0.9, fr; q=0.8", and the server
	has documents designated "en" and "fr", then the "fr" document
	will be returned. This is necessary to maintain compliance with
	the HTTP/1.1 specification and to work effectively with properly
	configured clients.

This is very clear that the use of lookup is an extension to provide
ultimate fallback for misconfigured browsers, *not* the standard behavior
in handling HTTP Accept-Language: headers.

> not. 

Consider a collection of oral histories by people in arabophone countries.
Standard Arabic would be rare among them, as most people can't speak it.

> Better to just explicitly list the ones that are wanted.

I may even agree with you here, but I don't think it's our business to
tell people how to configure their browsers.

>    (eg "ro, mo, no, nn, nb"), and provide exactly the list you want.

It's also painful to set up, less stable than the variety using
macrolanguages, and so on.

> > > Putting some ISO language tags into the extlang position just
> > > because they have a macrolanguage is an unnecessary complication
> > > for implementations and represents a substantive, controversial
> > > change to RFC 4646.
> >

In other words, extlangs were the status quo before the teleconferences,
and if I had held out, they might well still be.  Because I gave in then
(prematurely, as I think now), no-extlangs is the status quo.

-- 
                Si hoc legere scis, nimium eruditionis habes.
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 23 May 2008 16:39:43 -0700

> have nothing.  To which I reply, The best is the enemy of the good.

This is a key issue.

If you are claiming that "sharing macrolanguages" are a good or at least
reasonable proxy for "mutually comprehensible", then we'd really need some
figures. Especially since where the languages are not mutually
comprehensible it will cause problems.

   1. What percentage of languages sharing macrolanguages are mutually
   comprehensible?
   2. How many other languages that don't share macrolanguages are mutually
   comprehensible?

Both of these can have different figures for written vs spoken.

*If you don't have the data for this, you have NO justification to say that
we'd be better off with extlang. Moreover, from what I understand from
Peter, #1 is low, which is very bad.*

---

From: John Cowan <[email-masked]>
Date: Fri, 23 May 2008 19:44:27 -0400

> comprehensible it will cause problems.

You ask for more than can be had, since "mutually comprehensible" is itself
very much a matter of degree.  There are dialects of English (not creoles)
that I find very difficult to understand.

-- 
"Well, I'm back."  --Sam        John Cowan <[email-masked]>
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 23 May 2008 16:55:24 -0700

Pick whatever degree you want. At that degree:

If you don't have any data to back up your assertion that "The best is the
enemy of the good."....

Mark

---

From: John Cowan <[email-masked]>
Date: Fri, 23 May 2008 20:11:51 -0400

> If you don't have any data to back up your assertion that "The best is the

So your argument is that languages are *less* likely to be mutually
intelligible (to some degree) if they are co-members of a macrolanguage
than if they are not?

All I need is to show that the tendency is the other way: that (a)
getting Cantonese when you want Mandarin is no worse than getting
Portuguese when you want French (i.e. both are equally bad); and (b)
that getting Egyptian Arabic when you want Sa'idi Arabic is better
than getting English when you want Greek.

-- 
One Word to write them all,             John Cowan <[email-masked]>
  One Access to find them,              http://www.ccil.org/~cowan
One Excel to count them all,
  And thus to Windows bind them.                --Mike Champion
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 23 May 2008 17:30:45 -0700

Hi -

As co-chair...

Gentlemen, you have been repeating yourselves for quite
some time now.  Unless you have something truly novel
to say, I recommend restraint in posting.

Randy

---

From: Peter Constable <[email-masked]>
Date: Fri, 23 May 2008 18:03:10 -0700

> Moreover, from what I understand from Peter, #1 is low…

I’d be happier if people did not attribute such general claims to me.

When initially published, ISO 639-3 encoded categories that were already encoded in ISO 639-2, that were catalogued in Ethnologue 15, or that were catalogued by LinguistList as supplemental to Ethnologue 15. Those sources deem each individual language to be distinct languages, by some definition – typically based on evidence of mutual non-intelligibility, though sometimes also considering other factors. All that I added was to attempt to make sense of the cases in which the denotation of categories in ISO 639-2 were not clear, or in which it appeared that they did not align in terms of granularity with categories in the Ethnologue or Linguist List catalogues, the latter leading to instances of macrolanguage.

So, claims about inter-intelligibility between varieties go back to those sources, not to me; and while the prototypical pairing of individual languages would be assumed by those sources to be mutually non-intelligible, that is not necessarily the assumption made in every case.

Peter

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Friday, May 23, 2008 4:40 PM
To: John Cowan
Cc: LTRU Working Group
Subject: Re: [Ltru] my technical position on extlang

On Fri, May 23, 2008 at 3:54 PM, John Cowan <[email-masked]<mailto:[email-masked]>> wrote:

This is a key issue.

If you are claiming that "sharing macrolanguages" are a good or at least reasonable proxy for "mutually comprehensible", then we'd really need some figures. Especially since where the languages are not mutually comprehensible it will cause problems.

 1.  What percentage of languages sharing macrolanguages are mutually comprehensible?
 2.  How many other languages that don't share macrolanguages are mutually comprehensible?
Both of these can have different figures for written vs spoken.

If you don't have the data for this, you have NO justification to say that we'd be better off with extlang. Moreover, from what I understand from Peter, #1 is low, which is very bad.

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 24 May 2008 10:16:42 +0200

> than getting English when you want Greek.

My support to John here.

At the same time, I think that the really important and compelling 
argument is *identification*.

I feel that Mark is willing to accept semantically less clear 
identification in order to, technically, be better able to prevent 
(sometimes) unwanted fallback. Thus if I, in my argumentation have been 
eager to secure fallback, Mark has been very eager to prevent it. These 
are two sides of the same coin.

However, some unwanted fallback is always unavoidable, even across 
un-encompassed languages.  And I agree with John that fallback within 
the same macrolangauge very often is wanted - or at least totally 
acceptable and understandable.

The one argument against extlang I would bring - though I don't know if 
it is an argument against anything - it is merely a fact: In the talk 
about dropping extlang, it was set up as the ultimate goal taht 'zh' 
would fall out of use, sometime in the very distant future.

Contrary to this, the extlang is almost like a blessing of 'zh' and 
'ar'. And if extlang had been introduced to Norwegian as well, then it 
would have made 'no' much more used. I am not sure that this is bad, 
though.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Gerard Meijssen" <[email-masked]>
Date: Sat, 24 May 2008 10:53:43 +0200

Hoi,
The current situation is that the biggest amount of content on the Internet
is not tagged. Languages that are "less and least resourced" can not be
found because standards and software both do not acknowledge them. There is
a glimmer of a hierarchy in extlangs but in reality they are there to
explain historic facts and make all these "hierarchies" seem to be the same
type, something they are not.

In my understanding it is only in the ISO-639-6 that a true hierarchy of
linguistic entities becomes available. When you consider the complexity and
size of this work, it will take years to get to grips with it. The current
tools ie mail and mailing lists are completely inadequate to the task for
the LTRU to deal with this.

I feel that this is a true committee and the result is likely to show it. If
that is the best that is on offer then so be it. In a way it is business as
usual and the biggest advantage is that it still prevents a move forward
warts and all.
Thanks,
      Gerard

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 24 May 2008 09:43:25 -0700

Just to make sure that we're talking about content negotiation, eg using
Accept-Language to get the language for a website, where "zh" means "zh" or
anything starting with it. The key difference between the models is
identification. With the extlang model, I can't *restrict* the match to what
I want without really jumping through hoops.
Here's an example. Let's suppose that content tagged with "fr", "gsw", and
"yue" (or "zh-yue") are available, and nothing else. Let's suppose also
that I understand German, French, and Mandarin, but not any of the
following.

   - gct Colonia Tovar German
   - geh Hutterite German
   - gmh Middle High German
   - gml Middle Low German
   - goh Old High German
   - gsw Swiss German
   - nds Low German
   -
   - cjy Jinyu Chinese
   - cpx Pu-Xian Chinese
   - czh Huizhou Chinese
   - czo Min Zhong Chinese
   - gan Gan Chinese
   - hak Hakka Chinese
   - hsn Xiang Chinese
   - mnp Min Bei Chinese
   - nan Min Nan Chinese
   - wuu Wu Chinese
   - yue Yue Chinese

Both models

   - If I specify "de fr", I'll get "fr" under either model. Fine, I can
   understand "fr".
   - If I wanted to also request Swiss German as well as German, I would use
   "gsw de fr", and get "gsw".
   - I only get "gsw" if I request it. That's good, because I don't
   understand "gsw".

No Extlang Model

   - If I specify "zh cmn fr", I'll get "fr" under either model. Fine, I can
   understand "fr".
   - If I wanted to also request Cantonese as well as Mardarin, I would use
   "yue cmn zh fr" and get "yue"
   - I only get "yue" if I request it. That's good, because I don't
   understand "yue".
   - I need to include "zh", and will for the indefinite future because the
   vast majority of Mandarin content is tagged that way.
   - I might get content tagged "zh" but which is actually "yue" but the
   chances of that are extremely remote.

Extlang Model

   - If I specify "zh zh-cmn fr", I would get "zh-yue". Broken, I don't
   understand Cantonese!
   - In order to specify that I want "zh" or "cmn" but no other languages, I
   have to use "zh-cjy;q=0, zh-cpx;q=0, zh-czh;q=0, zh-czo;q=0, zh-gan;q=0,
   zh-hak;q=0, zh-hsn;q=0, zh-mnp;q=0, zh-nan;q=0, zh-wuu;q=0, zh-yue;q=0 zh
   zh-cmn fr".
   - Moreover, as you said, if another encompassed language shows up for zh,
   I could get that inadvertently unless I add this to my list.
   - Again, I need to include "zh", and will for the indefinite future
   because the vast majority of Mandarin content is tagged that way.

Mark

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 24 May 2008 09:43:32 -0700

While John and I disagree on the outcome of the conversation, I think he has
raised a legitimate issue. We have focused in on lookup for most of our
discussions, and he raised the different issues of filtering and content
negotiation.
Mark

---

From: John Cowan <[email-masked]>
Date: Sat, 24 May 2008 14:46:40 -0400

Gerard Meijssen scripsit:

> linguistic entities becomes available. 

It's a true hierarchy, all right, but it's a genetic hierarchy, which is
only loosely connected with mutual intelligibility.  Sometimes closely
related languages are mutually intelligible with only small adjustments,
sometimes the barriers between them are absolute.  Consider my comments
about English/Scots and the Frisian languages, which form a sister group in
genetic terms:

	Evidence from earlier stages of the languages would lead us
	to expect all three Frisians to be quite similar and the most
	similar language outside the group to be English -- but West
	Frisian has been so influenced by Dutch, North Frisian by High
	German and Low Saxon and Danish, Saterfrisian by High German and
	Low Saxon, and English by French and Latin that the connections
	are pretty much obscured.

-- 
In my last lifetime,                            John Cowan
I believed in reincarnation;                    http://www.ccil.org/~cowan
in this lifetime,                               [email-masked]
I don't.  --Thiagi
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Sat, 24 May 2008 22:16:40 -0400

Mark Davis scripsit:

> anything starting with it. 

Check.

> that I understand German, French, and Mandarin, but not [anything else].

> Both models
> 
>    - If I specify "de fr", I'll get "fr" under either model. Fine,
>    I can understand "fr".
>    - If I wanted to also request Swiss German as well as German,
>    I would use "gsw de fr", and get "gsw".

Check.

> No Extlang Model
> 
>    - If I specify "zh cmn fr", I'll get "fr" under either model. Fine,
>    I can understand "fr".
>    - If I wanted to also request Cantonese as well as Mardarin,
>    I would use "yue cmn zh fr" and get "yue"

That would only work if the Cantonese content was labeled "yue", which is
not possible until 4646bis goes into effect.  So for practical purposes
you'd need to say "yue zh-yue cmn zh fr", since "zh-yue" is currently
the *only* way to tag Cantonese.

>    - I only get "yue" if I request it. That's good, because I don't

That's literally true, but you *will* get Cantonese content tagged "zh-yue".

>    - I need to include "zh", and will for the indefinite future because
>    the vast majority of Mandarin content is tagged that way.

Check.

>    chances of that are extremely remote.

Not so.  If the Cantonese content is tagged "zh-yue" (and again, that is the
*only* way to tag Cantonese today), you will get it.

>    - If I specify "zh zh-cmn fr", I would get "zh-yue". Broken, I don't

Check.

>    zh-hak;q=0, zh-hsn;q=0, zh-mnp;q=0, zh-nan;q=0, zh-wuu;q=0, zh-yue;q=0 zh

Check.  And yet:

If most users can handle most of the encompassed languages, then the
no-extlang model is painful (you have to specify too many positive-q
tags).  If most users can only handle a few encompassed languages, then
the extlang model is painful (you have to specify too many zero-q tags).
The situation is symmetrical.

>    - Moreover, as you said, if another encompassed language shows up
>    for zh, I could get that inadvertently unless I add this to my list.

Again, it's symmetrical: if another encompassed language shows up, I
could miss that inadvertently (in the no-extlang model) unless I add
it to my list.

>    because the vast majority of Mandarin content is tagged that way.

Check.

-- 
You know, you haven't stopped talking           John Cowan
since I came here. You must have been           http://www.ccil.org/~cowan
vaccinated with a phonograph needle.            [email-masked]
        --Rufus T. Firefly
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Sat, 24 May 2008 23:20:21 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> John Cowan

> The situation is symmetrical.

My hunch is that, for most macrolanguage cases, most users would not be able to handle most of the encompassed languages. If that's the case, then you're making an argument *against* extlang.

Peter

---

From: John Cowan <[email-masked]>
Date: Sun, 25 May 2008 02:31:35 -0400

Peter Constable scripsit:

> My hunch is that, for most macrolanguage cases, most users would not be
> able to handle most of the encompassed languages.

You're probably right, with Norwegian as a conspicuous exception;
hence the ongoing brouhaha.

> If that's the case, then you're making an argument *against* extlang.

Hmm.  You may be right at that.

-- 
Some people open all the Windows;       John Cowan
wise wives welcome the spring           [email-masked]
by moving the Unix.                     http://www.ccil.org/~cowan
  --ad for Unix Book Units (U.K.)
        (see http://cm.bell-labs.com/cm/cs/who/dmr/unix3image.gif)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Sun, 25 May 2008 11:09:08 +0200

Mark Davis 2008-05-24 18.43:
> Let's suppose that content tagged with "fr", "gsw", and

>    chances of that are extremely remote.

A fundamental question:  No matter how "extremly remote" the "yue" 
content which is tagged as "zh" is, why do you place "zh" first? Where 
is the benefit in having 'zh cmn' instead of 'cmn zh'?  If you are 
against Extlang, why are you rejecting the benefit of not having Extlang 
by placing 'zh' first?

>    - If I specify "zh zh-cmn fr", I would get "zh-yue". Broken, I don't

Ok, so you want to discus your 0.00001% of Chinese resources which are 
in Cantonese ...

    * Firstly, you should not need to say 'zh' when you have said
      'zh-cmn', since Web servers wiill see that zh-cmn as a partly
      match for 'zh'. So I disagree that you really need to include 'zh'
      here.
    * Secondly, as I understand it, a Mandarin user very likely
      understands some Cantonese. It all depends on the kind of
      Cantonese - if we are to believe what was mentioned on this list
      earlier.  Do the unlikelyhood that the Mandarin user could meet a
      Cantonese page he/she was completely unable to comprehend really
      defend "going through hoops"?
          o Thus I feel that this need for a Mandarin speaker to avoid
            Cantonese at all cost is not a a real issue for a native
            Chinese speaker. I.e. you describe an edge case.

>    zh-hak;q=0, zh-hsn;q=0, zh-mnp;q=0, zh-nan;q=0, zh-wuu;q=0, zh-yue;q=0 zh

A simpler solution, for laymen, would be this order of preference: 
"zh-cmn fr zh-yue zh-cjy zh-cpx zh-czh zh-czo zh-gan zh-hak zh-hsn 
zh-mnp zh-nan zh-wuu"

In other words, to completely control the languages of a macrolanguage, 
you must list them all.

This is the benefit, and downside, of macrolanguages.

>    because the vast majority of Mandarin content is tagged that way.

All users of the Chinese Macrolanguae will, eventually, need to include 
'zh'.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Sun, 25 May 2008 12:11:19 +0200

Peter Constable 2008-05-25 08.20:
> >> - In order to specify that I want "zh" or "cmn" but no other 
> >> languages, I have to use "zh-cjy;q=0, zh-cpx;q=0, zh-czh;q=0,
> >> zh-czo;q=0, zh- gan;q=0, zh-hak;q=0, zh-hsn;q=0, zh-mnp;q=0,
> >> zh-nan;q=0, zh-wuu;q=0, zh- yue;q=0 zh zh-cmn fr".
> >
> > Check.  And yet:
> >

And if your hunch is wrong, you are making an argument for Extlang. The 
question is: how do you count "most users"? 

My answer is that firstly, one should focus on the native Macrolanguage 
speakers. Secondly, one should focus on web pages made in the countries 
were these langauges are a dominant language. Their needs should count 
the most.

Fallback is an issue for the web, mostly. Or, perhaps we sould say, it 
is an issue for interactive media. Then the question is,

    * how many web pages are there that exist in "the dominant
      encompassed language" but not in "an undominant encompassed
      language"? (And I count in that which encompassed language is
      dominant, might vary in different countries.)

If it is a general resource aimed at users from China, then you will use 
the dominant language of China.  And may be fill in with pages in one or 
several of the other languages of China.

If it is a resource aimed spesificly at the users of "an undominant 
encompassed language", then,

    * firstly, it is very likely that the Macrolangauge tag would be
      used for that language anyhow. And thus, there would not be any
      possibility of escaping it, anyway.
    * secondly, should we really "go through hoops", in this spec, in
      order to secure that users of "the dominant encompassed language"
      can get the page in French or English instead? How often will
      there be a *real* need for that?

What we should NOT focus strongly on is the needs of Web sites that aims 
at demonstrating that the World is a multilingual place. Pages where all 
the x-thousand languages of the world are given equal space. Such pages 
are not real world examples in my view.

(I'm now very exited if the above can be commented without a reference 
to how close Nynorsk and Bokmål are.)
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Gerard Meijssen" <[email-masked]>
Date: Sun, 25 May 2008 12:12:48 +0200

Hoi,
I am lost when I try to understand you. For me it is simple; with a zh
article as it is tagged today, we know nothing for sure. It can be anything.
This is the sad truth of the matter. Keeping  the zh tag only prolongs this
for the content that will be tagged in the future. What we need is to
understand what the historic tags means and interpret it for what it stands
for. This interpretation is done best by computers.

As long as you expect people to be able to tag manually, you do in my
opinion best to cut this gordian knot. This leaves you with ends of rope
that you can call "cmn" "yue" et all. When you are sure that people use
software that hides the issues involved, that present the choises in natural
language, then and only then does it not matter what the codes say, they
could even be a binary code.
Thanks,
     Gerard

On Sun, May 25, 2008 at 11:09 AM, Leif Halvard Silli <[email-masked]> wrote:

> the
> languages, I

---

From: Leif Halvard Silli <[email-masked]>
Date: Sun, 25 May 2008 12:56:45 +0200

Gerard Meijssen 2008-05-25 12.12:

> I am lost when I try to understand you.

Me? This group? Which passage of my letter?

> For me it is simple; with a zh article as it is tagged today, we know
> nothing for sure. It can be anything.

Agree. It can be any Chinese. In theory and in real life. Unfortunatly, 
there doesn't seem to be agreement on this.

> This is the sad truth of the matter. Keeping  the zh tag only
> prolongs this for the content that will be tagged in the future.

I have seen no serious proposal to deprecate it.

> What we need is to understand what the historic tags means and
> interpret it for what it stands for.

I take this to mean that "we must understand that it means 'any Chinese'".

> This interpretation is done best by computers.

If so, then we must be really firm on that it means 'any Chinese'.

>  opinion best to cut this gordian knot. This leaves you with ends of
> rope that you can call "cmn" "yue" et all.

There is much in this, I feel. My involvement in this debate began with 
a message to the W3-I18N mailinglist, where I claimed that there are too 
many tags for Norwegian and that people don't get it.

Unfortuntaly, it is hard to get agrement on how the automatic should work.

> When you are sure that people use software that hides the issues
> involved, that present the choises in natural language, then and only
> then does it not matter what the codes say, they could even be a
> binary code.

So, do you mean that 'zh' should always be included, if you select any 
of the Chinese encompassed languages?
-- 
leif halvard silli

---

From: John Cowan <[email-masked]>
Date: Sun, 25 May 2008 12:30:03 -0400

Leif Halvard Silli scripsit:

>      match for 'zh'. 

No, that's wrong.  That's the mistake I've been pointing out this
last week.

The way Web servers work is that if you ask for "zh", you will get
anything that begins with "zh" as well.  This is the opposite of the
way locale fallback works, where asking for "zh-cmn" resources and not
finding them will cause fallback to "zh".

>      understands some Cantonese. 

That's a gross exaggeration.

1) Spoken Cantonese and spoken Mandarin have zero mutual intelligibility,
the same as Norwegian and English.

2) Most Mandarin-speakers never learn any Cantonese at all.  [Most
English-speakers never learn any Norwegian.]

3) Cantonese-speakers in the People's Republic and in Singapore
typically learn Mandarin in school, including how to read and write it.
They then use Mandarin for all written purposes and some spoken purposes,
while of course continuing to use Cantonese for other spoken purposes.
[Norwegians learn and use English this way too, while of course also
learning to read and write Norwegian.]

4) In Hong Kong, people learn Mandarin in written form only, and for oral
purposes pronounce each written word using its Cantonese equivalent.
(I'm simplifying by using the word "word" here.)  This produces unnatural
but intelligible Cantonese.  After that they read and write Mandarin
and speak Cantonese.  [You could imagine a Norwegian learning written
English like this and reading it aloud word by word in Norwegian.]

5) There is *no* standard way to write Cantonese.  When it is necessary
to write it exactly, as in recording the lyrics of Cantonese pop songs,
a variety of hacks are used, similar in principle to the hacks used
to record dialects exactly in English (and, I suppose, Norwegian too).
The more hacks there are (unknown spellings, use of one word to represent
another with no Mandarin equivalent, unusual word orderings), the less
likely that someone who can read and write Mandarin will be able to
understand.  [Norwegian, fortunately, has a written standard, in fact
two of them.]

This set of facts is not reducible to "Mandarin- and Cantonese-speakers
understand each other" or "They don't understand each other."
Understanding is partial, not necessarily mutual, a matter of degree.

-- 
John Cowan      http://www.ccil.org/~cowan      [email-masked]
Be yourself.  Especially do not feign a working knowledge of RDF where
no such knowledge exists.  Neither be cynical about RELAX NG; for in
the face of all aridity and disenchantment in the world of markup,
James Clark is as perennial as the grass.  --DeXiderata, Sean McGrath
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 25 May 2008 19:21:05 +0200

On Fri, May 23, 2008 at 02:05:41PM -0700,
 Mark Davis <[email-masked]> wrote 

>    2. "get me the languages that have the same macrolanguage as X"

Most people, in practice, say neither #1 nor #2. They say "get the
languages I understand and like". It is not restricted to languages
that are mutually intelligible, users expect the software to know
everything *they* know (for instance that most literate people in
Tunisia can read French, even if this language has no official status
in Tunisia).

Since nobody considers adding this complicated and often-changing
information to the LSR, we must accept that we will not be able to
deliver to the end-users what they really want.

Therefore, we should stay focused on delivering something which may
not be the perfect solution to every language problem, but which is at
least useful and consistent.

I would prefer something that allows unambiguous tagging. I would
leave the "right fallback" issue to complicated programs, filled with
information produced by someone else and developed outside of LTRU. We
cannot specify rules for the right fallback, we should regard this
issue as taboo and no longer discuss it.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 25 May 2008 19:30:50 +0200

On Fri, May 23, 2008 at 07:44:27PM -0400,
 John Cowan <[email-masked]> wrote 
 a message of 17 lines which said:

> You ask for more than can be had, since "mutually comprehensible" is
> itself very much a matter of degree.  There are dialects of English
> (not creoles) that I find very difficult to understand.

A question for the linguists: I suspect that "mutual intelligibility"
is not transitive. That is, A can be "mutually comprehensible" with B
and B with C but not A with C. Are there documented cases of
non-transitivity?
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 25 May 2008 11:02:32 -0700

I agree.

---

From: John Cowan <[email-masked]>
Date: Sun, 25 May 2008 15:34:41 -0400

> and B with C but not A with C. Are there documented cases of

Indeed.  English being a pluricentric language, it's easy to set up
chains where each adjacent pair of varieties is mutually intelligible,
but the endpoints are not.  For that matter, there is no sharp linguistic
boundary between Italy and France at the level of local dialect (or
insofar as there is one, it is well inside the political boundaries of
Italy), though obviously Standard Italian and Standard French are not
mutually intelligible.

-- 
John Cowan                                [email-masked]
I amar prestar aen, han mathon ne nen,    http://www.ccil.org/~cowan
han mathon ne chae, a han noston ne 'wilith.  --Galadriel, LOTR:FOTR
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Sun, 25 May 2008 21:37:44 +0200

> No, that's wrong.  That's the mistake I've been pointing out this

I agree, this was not correct. Only if the users ask *only* for 
'zh-cmn' and no other languages, would what I said above hold true 
for Apache. The user would then experience this as the following:

   Ask zh-cmn; Get zh-cmn, zh-abc, zh-def, zh-ghi *or* zh

I.e. you will get 'zh-whatever' before you get 'zh'.

However, in Mark's example, the User Agent also asked for other 
languages. And when asking for other languages as well, then you 
will get 'other language' (fr, en, etc) before any fallback to 
zh-whatever happens.

Hence, the User Agent must, as Mark said, ask for both zh-cmn and 
zh in order to receive such content before for example 'en' content.

However, this also confirms that one should place 'zh-cmn' before 
  'zh'. At least if the user wants to be served 'zh-cmn' before 
'zh-whatever'.

> finding them will cause fallback to "zh".

So, if I got it right, you say this:

  Web: Ask zh; Get zh, zh-cmn, zh-yum, etc - in alphabetical order.
  Loc: Ask 'zh'; Get 'zh' only.
  Loc: Ask 'zh-cmn'; Get zh-cmn or zh.
.

> That's a gross exaggeration.

   [ snip ]

> Understanding is partial, not necessarily mutual, a matter of degree.

This was a useful read. Thank you. I agree with your conclusion.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Sun, 25 May 2008 14:30:47 -0700

> From: Leif Halvard Silli [mailto:[email-masked]]

> > My hunch is that, for most macrolanguage cases, most users would not
> be able to handle most of the encompassed languages. If that's the case,
> then you're making an argument *against* extlang.
> >
>
> And if your hunch is wrong, you are making an argument for Extlang.

Not necessarily. If a implies b, it does not logically follow that not b implies not a.

Besides, I doubt my hunch is wrong: AFAIK, it's not the case that most Chinese are able to handle most Chinese languages; it's not the case that most Quechua are able to handle most Quechuan languages; it's not the case that most Zapotec are able to handle most Zapotec languages.

> The

Well, you tell me: it was needed for the pro-extlang argument being made.

> speakers.

That's fallacious in the general case: nobody speaks a macrolanguage, unless you consider a macrolanguage to be equal to one of the encompassed languages or believe that all of the presumed individual languages are really not distinct at all. That may hold in a case like Serbo-Croatian, in which socio- and politico-linguistic factors have been the basis for positing multiple languages rather than purely linguistic factors. But that's the exceptional case.

> Secondly, one should focus on web pages made in the countries

As has been observed before, 4646bis is not creating language tags solely for the Web.

Peter

---

From: Peter Constable <[email-masked]>
Date: Sun, 25 May 2008 14:35:19 -0700

> Leif Halvard Silli

>       match for 'zh'. So I disagree that you really need to include
>       'zh' here.

Please read RFC 2616, section 14.4: if an HTTP header requests zh-cmn and the content is tagged zh, there will not be a match. If the content is tagged zh, then zh must be requested.

Peter

---

From: Peter Constable <[email-masked]>
Date: Sun, 25 May 2008 15:01:37 -0700

> Stephane Bortzmeyer

> and B with C but not A with C. Are there documented cases of

Certainly. There are also cases of non symmetry: speakers of A may understand B at a functional level, but not vice versa.

Peter

---

From: Peter Constable <[email-masked]>
Date: Sun, 25 May 2008 15:09:10 -0700

> information produced by someone else and developed outside of LTRU.

I can agree with that.

Peter

---

From: "Kent Karlsson" <[email-masked]>
Date: Mon, 26 May 2008 01:05:22 +0200

Maybe, just maybe, in hindsight, it would have been better to either
 
a) reclassify some language codes to be collection codes ('ar', 'zh', ...), or
 
b) retarget some language codes to the "dominant language" (in some sense)
    like 'ar' to stand for just standard Arabic (a similar change has been
    accepted for LTRU re. 'fy'), even though that could be seen as a narrowing,
    but better seen as a clarification (in view of stability requirements);
 
or a mix of these these, rather than to invent the new concept of macrolanguage.
In the current situation, it seems best to in practice regard macrolanguage
codes as collection codes.
 
        /kent k

  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of Peter Constable
Sent: Saturday, May 24, 2008 3:03 AM
To: LTRU Working Group
Subject: Re: [Ltru] my technical position on extlang

 

Peter

 

This is a key issue.

-- 
Mark

---

From: Shawn Steele <[email-masked]>
Date: Sun, 25 May 2008 16:15:12 -0700

RFC 2616 may say that, but in practice that's not really true.  Many servers will return en if en-US is requested and en content exists but en-US doesn't.

IE 6 was also pretty mixed up with the default languages, knowing specific languages for en-US, etc., but French just got fr, no fr-FR option.  Region-neutral resources aren't requested by default from IE 6 (unless you happened to be French or some other country where your country code matches your language code by accident).

- Shawn

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Peter Constable
Sent: Sunday, May 25, 2008 2:35 PM
To: LTRU Working Group
Subject: Re: [Ltru] my technical position on extlang

Peter

---

From: Leif Halvard Silli <[email-masked]>
Date: Mon, 26 May 2008 02:59:19 +0200

Shawn Steele 2008-05-26 01.15:

> RFC 2616 may say that, but in practice that's not really true.
> Many servers will return en if en-US is requested and en content
> exists but en-US doesn't.

This is, I think, what John explained about Apache to Mark.

LH

> IE 6 was also pretty mixed up with the default languages,
> knowing specific languages for en-US, etc., but French just got
> fr, no fr-FR option. Region-neutral resources aren't requested by
> default from IE 6 (unless you happened to be French or some other
> country where your country code matches your language code by
> accident).

> From: Peter Constable

> Please read RFC 2616, section 14.4: if an HTTP header requests

zh-cmn and the content is tagged zh, there will not be a match. If
the content is tagged zh, then zh must be requested.
-- 

leif halvard silli

---

From: Leif Halvard Silli <[email-masked]>
Date: Mon, 26 May 2008 04:29:24 +0200

>>> My hunch is that, for most macrolanguage cases, most users
>>> would not be able to handle most of the encompassed
>>> languages. If that's the case, then you're making an
>>> argument *against* extlang.

> Not necessarily. If a implies b, it does not logically follow

> that not b implies not a.

I don't protest.

> Besides, I doubt my hunch is wrong: AFAIK, it's not the case
> that most Chinese are able to handle most Chinese languages; it's
> not the case that most Quechua are able to handle most Quechuan
> languages; it's not the case that most Zapotec are able to handle
> most Zapotec languages.

For me, it has not been the point that most Chinese are able to 
handle most Chinese languages. I have never said that.

>> The question is: how do you count "most users"?
> 

> That's fallacious in the general case: nobody speaks a
> macrolanguage, unless you consider a macrolanguage to be equal to
> one of the encompassed languages or believe that all of the

> presumed individual languages are really not distinct at all. 

It is perhaps an interesting for you to make that logical point. 
But anyone who speaks one of the encompassed Chinese languages 
speaks Chinese. For my own part, I write and speak Norwegian. For 
other macrolanguages there is perhaps not a common name.

 > That

> may hold in a case like Serbo-Croatian, in which socio- and
> politico-linguistic factors have been the basis for positing
> multiple languages rather than purely linguistic factors. But
> that's the exceptional case.

I'll just take it as "revelation from above" that it is an 
exception if the speakers that belongs to a macrolanguage 
understands any other encompassed language than his/her own.

> As has been observed before, 4646bis is not creating language

And that is why I made a point about clarifying that I spoke only 
about the web in this case.
-- 
leif halvard silli

---

From: Martin Duerst <[email-masked]>
Date: Mon, 26 May 2008 18:32:18 +0900

This is off-topic, but may be helpful anyway.

>Not necessarily. If a implies b, it does not logically follow that not b 

Well, in classical logic, "not b implies not a" indeed logically
follows from "a implies b". That only breaks down if you remove
the law of the excluded middle, if you no longer have "a or not a = TRUE".

Regards,   Martin.

>Besides, I doubt my hunch is wrong: AFAIK, it's not the case that most 
>Chinese are able to handle most Chinese languages; it's not the case that 
>most Quechua are able to handle most Quechuan languages; it's not the case 
>that most Zapotec are able to handle most Zapotec languages.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Peter Constable <[email-masked]>
Date: Mon, 26 May 2008 09:57:36 -0700

> From: Martin Duerst [mailto:[email-masked]]

> > If a implies b, it does not logically follow that not
> b
> >implies not a.
>

Doh!! My VERY bad. Of course, what I was thinking of is that "not a implies not b" does not logically follow from "a implies b".

Peter