Subject: [Ltru] Issue on usability of extlang (was Re: Fragmentary notes from telecon)

From: "Mark Davis" <[email-masked]>
Date: Mon, 26 Nov 2007 08:47:35 -0800

No problem with copying it.

By the way, Martin raise an issue during the discussion which I didn't have
time to address.

Q. Why not have extlang for using macrolanguages if the suggested text in
http://docs.google.com/Doc?id=dfqr8rd5_42gzc36s adds macrolanguages back
into the fallback chain? (These are not Martin's words, but it was roughly
that.)

A. The suggested text doesn't add it back in. It only says that *IF *an
application wants to do extlang-equivalent fallback, the text in BCP 47
already allows for that.

We really have no idea whether using macro languages in the fallback chain
is a good idea or not. Some people think it will be an advantage for a few
specific examples that are cited. (Cantonese comes up, but when I tested
some of the assumptions with Cantonese speakers, they didn't quite hold up.)
But nobody has substantiated that it will give better results for all or
most macrolanguages. Or any indication of an even rough list of those for
which it will be better. Nor has anyone effectively argued that the
situation between yue and zh is substantially different than the situation
between gsw and de, where we get along just fine without extlang.

So the suggested text just provides it as an option, and leaves it up to the
application.

In particular, extlang plays especially badly in many cases. Suppose that we
have macrolanguage m1, and microlanguages x1 and x2. By the design of 639,
we can't assume that a speaker of x1 can also speaker x2 or vice versa. If a
user has as accept language the list <x1-Ssss-Rr, en, fr>, it works fine
without extlang: she gets the fallback (A)

   1. x1-Ssss-Rr
   2. x1-Ssss
   3. x1
   4. en
   5. fr

If she also speaks/reads x2, then she can specify <x1-Ssss-Rr, x2, en, fr>
or <x1-Ssss-Rr, en, fr, x2>; that is, putting x2 in the list in the position
she wants it. If x2 is the predominant microlanguage, meaning that m1 is
essentially always assumed to be x2, then the priority list can be
<x1-Ssss-Rr, en, fr, m1>, also wherever it belongs. Thus, the user gets
something she can understand, based on the list she supplied.

If we are using extlang, and x2 is the content for m1, then we get the
fallback (B)

   1. m1-x1-Ssss-Rr
   2. m1-x1-Ssss
   3. m1-x1
   4. m1
   5. en
   6. fr

That has two problems: first, the script and region are lost. That can be
fixed by hacking the fallback (although there is a *lot *of installed base
that won't do this), to (C)

   1. m1-x1-Ssss-Rr
   2. m1-x1-Ssss
   3. m1-x1
   4. m1-Ssss-Rr
   5. m1-Ssss
   6. m1
   7. en
   8. fr

But even more importantly, we are disabling the user's explicit choice. If
the user doesn't speak x2 (or whatever the content of m1 is), he's screwed.
There is no way that he can indicate that he wants x1 *but no other version
of m1 because he can't understand them.
*We'd have to change the fallback to be quite substantially different to get
around this, with (D)*
*

   1. m1-x1-Ssss-Rr
   2. m1-x1-Ssss
   3. m1-x1
   4. en
   5. fr
   6. m1-Ssss-Rr
   7. m1-Ssss
   8. m1

This is, however, still only appropriate if it is likely that a user of x1
speaks *whatever happens to be the content for m1*. That is an extremely
shaky assumption.

If Peter Constable said, for each and every macrolanguage on
http://www.sil.org/iso639-3/macrolanguages.asp, there is at least one
microlanguage that all speakers (or even most speakers) of each of the other
microlanguages would understand, I'd say: fine, let's do extlang and
incorporate that information into the registry, with the "default
microlanguage" for each macrolanguage. Then, for example, implementers would
know that, say "fuf
<http://www.sil.org/iso639-3/documentation.asp?id=fuf>Pular" is
understood by all the speakers of the microlanguages under
"ful <http://www.sil.org/iso639-3/documentation.asp?id=ful> Fulah", so we
can tell people to always have the content of "ful" be "fuf", and bake the
macrolanguages in as extlang.

The point of the suggested text is that if your application wants to use
macrolanguages to support extlang-equivalent fallback, there is nothing
stopping you from doing so. If there are particular environments where an
extlang-like fallback is right for a particular language community, it is
simple to do. But we don't need to bake shaky assumptions into the structure
of language tags.

------------------------------

And here is the suggested text from
http://docs.google.com/Doc?id=dfqr8rd5_42gzc36s, just so there is a record
for the IETF.
 LTRU text Latest draft of 4646:
http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-09.html
Latest draft of 4647:
*http://www.rfc-editor.org/rfc/bcp/bcp47.txt*<http://www.rfc-editor.org/rfc/bcp/bcp47.txt>
------------------------------

[Here are additions proposed for rfc4647bis, to account for macrolanguages
without extlang. ]

*Add to end of 3.4.1 (**most of the way to the end)*

...
Often the implementation-defined default result is English, since that is
quite often the "root" used for common resource lookup. However, that is
often not the best choice where other information is available.

Certain languages are closely related, and the lookup process may take that
into account. For example, since the the tag "gsw-CH" (for Swiss German as
used in Switzerland) was first available on 2006-12-08, Swiss German
("Schwyzerduetsch") text may have been tagged with "de-CH" instead. ISO 639
was not (and is still not) clear on whether "de" meant only High German or
also included variants such as Low German or not. Thus Swiss German material
may have been, and may still be tagged with "de". Essentially all Swiss
German speakers are comfortable in High German, so where Swiss German is not
available, High German is a very good fallback. Thus when given the language
priority list: "gsw-CH, fr-CH", an implementation using lookup may augment
the default values to also include the lookup of related values, such as the
following search order:

1. gsw-CH
2. gsw
3. fr-CH // next language
4. fr
5. implementation defined default:
    5a. de-CH // special fallback from gsw-CH
    5b. de
    5c. en // root

In this way, other likely possibilities are tried before the final fallback
to the root value. Note that typically the fallback to related languages
should include the script and region codes if available.

In this way, the lookup process may take into account what languages people
are likely to understand, given a language priority list. Similarly, the
close relations between Romanian and Moldavian, Tagalog and Filipino,
Serbo-Croatian and Croatian, and so on may all be useful in doing related
language lookup.

This is not restricted to related languages. For example, a Breton speaker
is very likely to also understand French, given the language priority list.
Thus the implementation may choose to use the following lookup for the
language priority list "br-FR, de":

1. br-FR
2. br
3. de
4. implementation defined default:
    4a. fr-FR // special fallback from br (Breton)
    4b. fr
    4c. en

The Macrolanguage information in the registry MAY be useful to
implementations in this regard. That is, an implementation MAY choose to use
that information in falling back from encompassed languages to macro
languages. For example, given the language priority list with Cantonese in
Traditional Script as used in Hong Kong, followed by French ("yue-Hant-HK,
fr"), the lookup could be the following:

1. yue-Hant-HK
2. yue-Hant
3. yue-HK
4. fr
5. implementation defined default:
  5a. zh-Hant-Hk
  5b. zh-Hant
  5c. zh
  5d. en

Whether such fallback should be used -- and if so, the precise way in which
such a fallback is done -- is application-dependent.  Where it is very
likely that the audience requesting Cantonese (as above) will accept and
understand Mandarin (the predominant content for 'zh'), then this fallback
might be useful. Where there is risk that that the audience requesting
Cantonese will not be conversant with Mandarin, and would prefer an
alternative in the language priority list, it should be avoided. (This might
be the case, for example, with audio using yue-Zxxx-US.)

For many macrolanguages, there is a single predominant form which is
strongly identified with the macrolanguage. For example, "zh" is
overwhelmingly used for Mandarin content (formally 'cmn'), and "ar" is
overwhelmingly used for Standard Arabic content (formally 'arb'). While
there is logically a distinction between these pairs, in practice they are
treated as interchangeable, so no such distinction can be depended upon.
Therefore in such cases, lookup SHOULD typically fall back from either one
to the other. For other macrolanguages, this is not the case, and fallback
to content that might be any any of the encompassed languages may be worse
than falling back to other default values, such as the predominant language
for a region.

Care must be taken with any of these choices. If an implementation provides
a UI for selecting language priority lists, it may be better to give the
user the option of having explicit fallbacks (such as from Cantonese to
Mandarin or Tagalog to Filipino), rather than trying to guess the user's
intent (and run the distinct risk of getting it wrong). For that purpose,
when a user adds a language to the priority list, the UI may suggest
macrolanguages, or other related languages, as additional fallbacks.

On Nov 22, 2007 6:01 AM, Stephane Bortzmeyer <[email-masked]> wrote:

> On Wed, Nov 21, 2007 at 03:10:28PM -0800,
>  Mark Davis <[email-masked]> wrote
>  a message of 38 lines which said:
>
> > http://docs.google.com/Doc?id=dfqr8rd5_43dnw36c
>
> IMHO, the IETF rules for mailing lists do not mix well with "outside"
> documents, which may not be available in the future, when someone will
> read the archive. So, I take the liberty to copy it here:
>
>
>
> LTRU Telecon Notes
> Felix, Randy, Martin, John, Addison, Mark
>
> Fragmentary notes (see end)
>
> Mark:
> We could look over the suggested changes in
> http://docs.google.com/Doc?id=dfqr8rd5_42gzc36s to see how this would
> be/change depending on the choices we make.
>
> Addison:
> Mostly in agreement about difficulties with macrolanguages.
> Disagreement:
>  extlang is a useful mechanism, gives better results
>  extlang is not a useful mechanism, gives worse results
>
> (Someone): Conflation of macrolanguage and majority language is a mistake.
>
> John: with extlang, should deviate from truncate from right in lookup
>
> Mark: sole reason for extlang is for not changing the algorithm
>
> Others: reason is to avoid need to look at registry
>
> Mark: doesn't matter -- no big deal to look at registry. Not proposing
> that it be required, just optional. And practically, software can update
> every so often to get the cases it cares about. Not a lot of data. Even for
> limited-memory environments, like cellphones, can
>
> Addision: people would be recommended to use the languages they can
> handle: eg zh, yue
>
> John: have registered tags encoded with extlang style
>
> Mark: those are not productive in tags
>
> Addision: those were in good faith, but can deprecate and remap; have done
> in other cases.
>
> Addision: lookup with extlang would look like:
>
> zh-yue-Hant-TW
> zh-yue-Hant
> zh-yue
> zh-hant-tw
> zh-hant
> zh
> (default)
>
> ...
> (many interesting points made, but was too involved in the conversation to
> keep up with the notes)
> ...
>
> Martin: Why not have extlang (question about google)
>
> Mark: Concerned with more than just Google: impacts Unicode (CLDR), ICU,
> Java, ... Concern is that better results overall are returned if we don't
> have extlang; we're inventing a mechanism and complications that we don't
> need. Example, zh/yue -- we got along without extlang for the analogous case
> of de/gsw. ... (Q about filtering). For filtering, in other cases, we
> specify the languages we want; if I want de and gsw, I list them. Similar
> for yue and zh.
> ...
>
> At the end.
>
>    * Will have weekly meeting.
>    * We talked about assigning a scribe, or having people note what they
> just said -- either way, set up something before we start next time.
>    * Addison will set up a collaboration tool for next time, and notify us
> ahead of time.
>
>
>
>
>
>
>
>
>
> Google Docs -- Web word processing, presentations and spreadsheets.
>

-- 
Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru