Subject: [Ltru] Matching Text

From: "Mark Davis" <[email-masked]>
Date: Wed, 10 Oct 2007 18:03:36 -0700

We've been talking about two different models for dealing with
macrolanguages: extlang and no-extlang. In our telephone conversations, a
third has come up: partial-extlang.

The partial-extlang model is where we only make X a secondary language for Y
when it can reliably be assumed that someone who understands X will also
understand Y. That would be the case if all of the microlanguages of Y are
mutually intelligible, or if Y is generally identified with a "prestige"
variant. With this model we wouldn't make all the microlanguages be
secondary, just certain ones like those having ar and zh as macrolanguages.
An example might be zh-yue.

To assess these options, I suggest that the right way to do it is to look at
the textual changes that we would need to make to RFC 4747 in order to have
each of these work -- AND describes the implications for implementers. That
way we can assess actual text that reflects the impact of the models on the
operations that use their structure.

Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 11 Oct 2007 08:35:39 +0200

Mark Davis wrote:

> The partial-extlang model is where we only make X a secondary language for
> Y when it can reliably be assumed that someone who understands X will also
> understand Y.

Hard to judge.   One of the sources I used for the Suppress-Script modification
requests claims that folks talking "mt" understand "ar-015" but can't read it. 

> That would be the case if all of the microlanguages of Y are mutually intelligible,
> or if Y is generally identified with a "prestige" variant.

What's a "prestige variant", is that a weasel word for "zh" ?  I also don't like
the term "microlanguage" as long as it's undefined.  Are say "mt", "lb", "nds", 
or "gsw" microlanguages ?

 Frank

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 11 Oct 2007 12:20:32 +0100

If I was designing a UI for a matching application I would expect the user
to enter the 639-3 code or the language name.  I would then wish to inform
the user that this code forms part of a macrolanguage that encompasses a
number of other languages and offer the user an option to include, via some
check box facility,  any or all or none of the related codes as secondary
matches.  This puts the decision on the end user and negates the need for us
to make any decisions as to intelligibility. Just my limited application :-)
 
Best regards
 
Debbie

  _____  

From: Mark Davis [mailto:[email-masked]] 
Sent: 11 October 2007 02:04
To: LTRU Working Group
Subject: [Ltru] Matching Text

Mark

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 11 Oct 2007 13:35:43 +0200

Debbie Garside wrote:

> If I was designing a UI for a matching application I would expect the
> user to enter the 639-3 code or the language name.  I would then wish
> to inform the user that this code forms part of a macrolanguage that
> encompasses a number of other languages and offer the user an option
> to include, via some check box facility,  any or all or none of the
> related codes as secondary matches.  This puts the decision on the end
> user and negates the need for us to make any decisions as to
> intelligibility. Just my limited application :-)

Sounds fine for a user configuring Accept-Languages for his browser,
at least for languages with a Suppress-Script.  But that's only one
side of it.  

My limited application is tagging Web pages, and I wasn't too thrilled
when Google searches for English pages missed my en-GB-oed pages.

 Frank

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 11 Oct 2007 14:35:16 +0200

On Wed, Oct 10, 2007 at 06:03:36PM -0700,
 Mark Davis <[email-masked]> wrote 
 a message of 54 lines which said:

> The partial-extlang model is where we only make X a secondary
> language for Y when it can reliably be assumed that someone who
> understands X will also understand Y.

Won't it be a problem similar to the Suppress-Script issue?
ietf-languages will have to evaluate languages and assess wether they
are or not compatible, since the information is not in ISO-639. 

Unlike the other two proposals, this one is apparently not
implementable by using only the ISO 639 standard.
 
> To assess these options, I suggest that the right way to do it is to
> look at the textual changes that we would need to make to RFC 4747
> in order to have each of these work -- AND describes the
> implications for implementers.

IMHO, the first thing to do would be to try to classify every
macrolanguage in 639-3 according to this plan. We will probably
discover it is complicated and flame-generating.

---

From: John Cowan <[email-masked]>
Date: Thu, 11 Oct 2007 09:06:56 -0400

Frank Ellermann scripsit:

> One of the sources I used for the Suppress-Script modification requests
> claims that folks talking "mt" understand "ar-015" but can't read it.

Reasonable.  Of course, they are going to lack vocabulary items that
have been replaced in Maltese by English and Italian borrowings.

> What's a "prestige variant", is that a weasel word for "zh" ?

Macrolanguages come in two basic types.  In the first type, there is a
single encompassed language (Mandarin for Chinese, Standard Arabic for
Arabic, Eastern Yiddish for Yiddish, Swahili (individual language) for
Swahili (macrolanguage), e.g.) which has much higher numbers and status
than the others.  In the other type, there is no such dominance among
the group of encompassed languages (Sardinian, Quechua, Romany, e.g.).
Sometimes the answer is unclear: Tosk Albanian is the basis of Standard
Albanian, but is spoken by only about 48% of all Albanian-speakers.

> I also don't like the term "microlanguage" as long as it's undefined.

It's the slang of this list for "language encompassed, according to
ISO 639-3, by a macrolanguage".  It will never be an official term.

> Are say "mt", "lb", "nds", or "gsw" microlanguages ?

Since none of them appear at
http://www.sil.org/iso639-3/macrolanguages.asp , the answer is "no"
by definition.  All of them are part of the German dialect continuum,
but then so is "nl".  Sociolinguistically they all function as separate
languages.

-- 
Unless it was by accident that I had            John Cowan
offended someone, I never apologized.           [email-masked]
        --Quentin Crisp                         http://www.ccil.org/~cowan

---

From: John Cowan <[email-masked]>
Date: Thu, 11 Oct 2007 10:33:11 -0400

> discover it is complicated and flame-generating.

Indeed.  I've already mentioned the case of Albanian, and Azerbaijani
is even more contentious:  North Azerbaijani is the basis for the official
language of Azerbaijan, but only about 22% of Azerbaijani speakers use it.
What's more, as Ethnologue delicately says:

        North and South Azerbaijani are spoken by one ethnic group. Each
        language group is reluctant to accept the written form of
        the other.

That is, North Azerbaijani is written in the Latin and Cyrillic scripts,
South Azerbaijani in the Arabic script.  Ethnologue says there are
"significant differences" between the two, but provides no estimates
of shared lexicon; other sources claim that mutual intelligibility
is easily attained, and even with eastern Turkish and with Turkmen.

-- 
There is no real going back.  Though I          John Cowan
may come to the Shire, it will not seem         [email-masked]
the same; for I shall not be the same.          http://www.ccil.org/~cowan
I am wounded with knife, sting, and tooth,
and a long burden.  Where shall I find rest?           --Frodo

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 11 Oct 2007 08:20:38 -0700

At Google we end up having to only regard the tags on Web pages as (weak)
hints, because so many pages are so badly tagged. Instead, we have to do
automated detection. I believe the other search engines do this as well.
Detection cannot effectively distinguish between relatively small variants
of a language, as typically expressed by variant subtags, or even
significant ones like most country variants. So for those cases, those
features are not distinguished when pages are indexed.

The situation is far worst for charsets, since a bad tag there has a much more
serious impact than just getting the language wrong.

The situation is very different for localization, rather than search. There
we can support country variants, such as en vs en-GB, more effectively.

Mark

BTW. For those of you coming to the Unicode conference next week, I'll be
giving a paper on how Google handles Unicode, charsets, and languages; I
think there will be discussions of those issues by other companies as well.

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 11 Oct 2007 08:23:00 -0700

I should have mentioned the other part of it. The extlang mechanism is only
for backwards compatibility in the first place, so for the partial-extlang
model we'd only need to look at cases where there has been a lot of tagging
data with the macrolanguage tag because the microlanguage tags were not
available.

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 11 Oct 2007 10:39:51 -0700

Hi -

As a technical contributor...

> From: "John Cowan" <[email-masked]>
> To: "Frank Ellermann" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Thursday, October 11, 2007 6:06 AM
> Subject: Re: [Ltru] Re: Matching Text
>
...
> > What's a "prestige variant", is that a weasel word for "zh" ?

It's a term from the field of sociolinguistics, used to describe a variant
that is favored in such ways as being taught in schools, used by TV
and radio announcers, etc. 

> Albanian, but is spoken by only about 48% of all Albanian-speakers.

I think this perspective can help clearly illustrate the difference in how different people
in this WG understand the problem.  Here's how I would have put it:

Macrolanguages come in two basic types.  In the first type, the string which forms
the macrolanguage subtag has historically been used as a tag for a particular
language.  As a consequence of the dominance of a particular variety of that language,
common usage has resulted in a situation where, in the field, the macrolanguage
subtag is effectively equivalent to the tag for the dominant variety.  In the second
type, there is no such history of usage, or no significant body of data tagged
in that way exists.  In this case, the macrolanguage subtag by itself merely has
the semantic "undifferentiated Foo", meaning the content could be from any
member of that macrolanguage family.

More succinctly: some view "macrolanguage" as being effectively equivalent
to some dominant or (likely) mutually intelligible member of the group, while
others view "macrolanguage" as being simply an abstraction, not corresponding
to any specific member of the group.

Randy

---

From: Addison Phillips <[email-masked]>
Date: Thu, 11 Oct 2007 10:41:29 -0700

> to any specific member of the group.

I would put a slight twist on that. I would say that "some view some of 
the macrolanguages as being effectively equivalent to some dominant 
(etc.) member of the group while viewing other macrolanguages of being 
simply an abstraction".

That is, I think both cases exist simultaneously.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

.org>
Date: Thu, 11 Oct 2007 10:41:29 -0700
From: Addison Phillips <[email-masked]>
To: Randy Presuhn <[email-masked]>
Subject: Re: [Ltru] Re: Matching Text
Cc: LTRU Working Group <[email-masked]>

> to any specific member of the group.

I would put a slight twist on that. I would say that "some view some of 
the macrolanguages as being effectively equivalent to some dominant 
(etc.) member of the group while viewing other macrolanguages of being 
simply an abstraction".

That is, I think both cases exist simultaneously.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: Shawn Steele <[email-masked]>
Date: Thu, 11 Oct 2007 12:05:55 -0700

> when Google searches for English pages missed my en-GB-oed pages.

Google doesn't like my tlh-US user locale either :-)

- Shawn

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 15 Oct 2007 08:32:16 +0200

Randy Presuhn wrote:

> More succinctly: some view "macrolanguage" as being effectively
> equivalent to some dominant or (likely) mutually intelligible
> member of the group, while others view "macrolanguage" as being
> simply an abstraction, not corresponding to any specific member
> of the group.

Thanks to all for your explanations of "prestige variant" and the
definition of "microlanguage".  

 Frank

P.S. to Mark, please post an URL to your paper if it's published.

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 15 Oct 2007 19:12:55 -0700

It's on my homepage, http://macchiato.com/. However, it's a presentation,
not a paper, so it is only a sketch of what I will actually discuss.

Mark

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 25 Oct 2007 21:07:46 -0700

Back in town, so will be attending the 6:00PT meeting tomorrow (Addison, I
assume it is the same call-in numbers).

I had suggested that we include proposed text changes for each of the
possible options that we are considering, where it affects implementations,
in RFC 4747. I haven't seen anything on the list about that, so I'll start
off with the no-extlang option. I suggest that others try their hands at the
extlang or partial extlang options so that we can compare the issues with
real text in front of us.

================

Add to end of 3.4.1 (http://www.rfc-editor.org/rfc/bcp/bcp47.txt, most of
the way to the end)

...
Often the implementation-defined default result is English, since that is
quite often the "root" used for common resource lookup. However, that is
often not the best choice where other information is available.

Certain languages are closely related, and the lookup process may take that
into account. For example, since the the tag "gsw-CH" (for Swiss German as
used in Switzerland) was first available on 2006-12-08, Swiss German
("Schwyzerduetsch") text may have been tagged with "de-CH" instead. ISO 639
was not (and is still not) clear on whether "de" meant only High German or
also included variants such as Low German or not. Thus Swiss German material
may be tagged with "de". Moreover, essentially all Swiss German speakers are
comfortable in High German, so where Swiss German is not available, High
German is a very good fallback. Thus when given the language priority list:
"gsw-CH, fr-CH", an implementation using lookup may augment the default
values to also include the lookup of related values, such as the following
search order:

1. gsw-CH
2. gsw
3. fr-CH // next language
4. fr
5. implementation defined default:
    5a. de-CH // special fallback from gsw-CH
    5b. de
    5c. en // root

In this way, other likely possibilities are tried before the final fallback
to the root value. Note that typically the fallback to related languages
should include the script and region codes if available.

Thus the lookup process may take into account what languages people are
likely to understand, given a language priority list. For example, a Breton
speaker is very likely to also understand French, given the language
priority list. Thus the implementation may choose to use the following
lookup for the language priority list "br-FR, de":

1. br-FR
2. br
3. de
4. implementation defined default:
    4a. fr-FR // special fallback from br (Breton)
    4b. fr
    4c. en

Similarly, Romanian and Moldavian, Tagalog and Filipino, Serbo-Croatian and
Croatian, may all be useful in doing related language lookup.

The Macrolanguage information in the registry MAY be useful to
implementations in this regard. That is, an implementation MAY choose to use
that information in falling back from encompassed languages to macro
languages. For example, given the language priority list with Cantonese in
Traditional Script as used in Hong Kong, followed by French ("yue-Hant-HK,
fr"), the lookup could be the following:

1. yue-Hant-HK
2. yue-Hant
3. yue-HK
4. fr
5. implementation defined default:
  5a. zh-Hant-Hk
  5b. zh-Hant
  5c. zh
  5d. en

Whether such fallback should be used -- and if so, the precise way in which
such a fallback is done -- is application-dependent.  Where it is very
likely that the audience requesting Cantonese (as above) will accept and
understand Mandarin (the predominant content for 'zh'), then this fallback
might be useful. Where there is risk that that the audience requesting
Cantonese will not be conversant with Mandarin, and would prefer an
alternative in the language priority list, it should be avoided. (This might
be the case, for example, with audio using yue-Zxxx-US.)

For many macrolanguages, there is a single predominant form which is
strongly identified with the macrolanguage. For example, "zh" is
overwhelmingly used for Mandarin content (formally 'cmn'), and "ar" is
overwhelmingly used for Standard Arabic content (formally 'arb'). While
theoretically there is a distinction between these pairs, in practice they
are treated as interchangeable, so no such distinction can be depended upon.
Therefore in such cases, lookup SHOULD typically fall back from either one
to the other. For other macrolanguages, this is not the case, and fallback
to content that might be any any of the encompassed languages may be worse
than falling back to other default values, such as the predominant language
for a region.

Care must be taken with any of these choices. If an implementation provides
a UI for selecting language priority lists, it may be better to give the
user the option of having explicit fallbacks (such as from Cantonese to
Mandarin or Tagalog to Filipino), rather than trying to guess the user's
intent (and run the distinct risk of getting it wrong). For that purpose,
when a user adds a language to the priority list, the UI may suggest
macrolanguages, or other related languages, as additional fallbacks.
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 26 Oct 2007 10:10:22 +0200

Mark Davis wrote:

> I had suggested that we include proposed text changes for each of
> the possible options that we are considering, where it affects
> implementations, in RFC 4747. I haven't seen anything on the list
> about that, so I'll start off with the no-extlang option.

Thanks.

> Add to end of 3.4.1 (http://www.rfc-editor.org/rfc/bcp/bcp47.txt
[...]

You're talking about a section in RFC 4647, it follows that you
intend to create a 4647bis.  (No problem, I just didn't know it.)

 Frank