Subject: Re: [Ltru] matching: Language Range Lists

From: "Jukka K. Korpela" <[email-masked]>
Date: Sat, 1 Oct 2005 10:10:46 +0300 (EEST)

On Fri, 30 Sep 2005, Martin Duerst wrote:

> I wanted to bring [language negotation in HTTP] up because:
> - We could decide that this was the right thing (or good enough at least),
>  and simply point to it (other specs can easily reference that part of
>  the HTTP spec if they want).
> - We could decide that this was the right thing (or good enough at least),
>  and copy the text for one-stop-shopping by other standards.
> - We could look at it, try to understand why it is the way it is,
>  and come away with the conclusion that we need something different.

That's quite a challenge. Starting from the last option, it would probably 
take a lot of time it would require experimentation - not just technical
but with "normal users" testing the feasibility of the ideas and 
principles, for different language skill and preferences profiles.

The other two options sound tempting, and they would be technically easy.
Yet, the complexity of the third option implies that we could not know
whether we have taken the right path. I'm afraid the only realistic 
alternative would be the fourth one:
- mention the language negotiation mechanism as one possible
   use of language preferences information, in a context where
   there is definitely need for more detailed expression of
   preferences than simply naming (by code) the most preferred language

>> It deals with a specific mechanism, content negotiation, which lets an HTTP 
> server and a client negotiate on the choice of a language version.
>
> Well, yes, it's a specific thing, but it's not too difficult to see
> how this could be used in other cases.

To me, it is. Or let's say that I don't think it's a suitable approach in 
many other cases, and I have little idea of what might be suitable,
except in a few cases.

>> (This has, sadly enough, been used rather on multilingual sites.)
>
> Not sure I get this. It was designed for multilingual sites.
> Did you want to say "it is used rather rarely on multilingual sites"?

Yes. Well guessed. (It seems I all too often mistype so that I omit
the most important word.)

Language negotiation is used on some sites, often on sites that somehow 
relate to technologies of internationalization and localization, and 
that's fine and natural. But on most sites that are genuinely 
multilingual, language negotiation is not used. Even the site of the 
European Union does not use it, and this can hardly depend on lack of
resources or on lack of political will to support multilingualism.

> Or is there something else that you don't like in the way it is used?

Quite a many features actually. For one thing, the language preferences 
settings in users browsers are so often wrong in the sense of not 
actually reflecting the user's understanding and preferences.
I would say that they are wrong more often than correct, e.g. because
the user has an English-language version of a browser, with some default 
settings that make "en" the only language, or worse still "en-US",
quite independently of the user's native language and other languages.
How _could_ the preferences be correct e.g. in a computer used in a 
public library?

This is probably one explanation to why the mechanism is rarely used.
Why take the trouble when you know that the results will so often be 
wrong?

In the long run, such problems could be solved, if web browsers
explicitly requested for the user's language preferences upon installation
and the preferences themselves could be easily stored so that they can
be transferred from one computer or browser to another, perhaps as part
of a collection of preferences. (Most of us use different computers, if 
not different browsers, and it's really frustrating to set all the 
personal preferences, including language preferences, every time you start 
using a new computer. In reality, people mostly don't even try.)

Apart from such issues, which are of general nature, the language 
negotiation mechanism itself as defined in HTTP is problematic
for a wide range of uses. It is based on the assumption that a user's
preferences are expressed as a set of language/q-value pairs. (It's really 
a set, not a list; the order is insignificant, the q-values define the
weights.) In practice, most browsers assign the q-values themselves, on 
the basis of the order of languages. This indicates that browser vendors 
regard the q-value system as too difficult to users. And it isn't really
that intuitive. Yet, the actual mechanism works on them.

The q-values are defined as weights, but what do the numbers relate to.
If I have q=0.95 for "de", would that mean that I understand 95 % of
German text on the average? If not, what does it mean?

-- 
Jukka "Yucca" Korpela, http://www.cs.tut.fi/~jkorpela/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: r&d afrac <[email-masked]>
Date: Sat, 01 Oct 2005 14:57:22 +0200

_Full_ agreement with Jukka.

At 09:10 01/10/2005, Jukka K. Korpela wrote:
>That's quite a challenge. Starting from the last option, it would 
>probably take a lot of time it would require experimentation - not 
>just technical

Martin's last option is however a necessary prerequisite to fully 
understand what we talk about. I was mistaken in believing that this 
was part of this WG Charter, and we could this way maintain 
compatibility through the digital convergence and avoid errors. 
Experience, and comments on the EITF main list about WSIS, show the 
subject is too "challenging" for the IETF Charter. Too bad because I 
know no other area to carry it: it has to be created. This is not 
small task. In the meanwhile I support Jukka's fourth option.

Jukka's solution is perfect as it permits to work in parallel on 
general negotiation mechanisms when needed - I would think for fine 
tuning like between Web Services or dialects, etc. (what does not 
prevent the working use of major languages to better see the issues). 
I suggested Addison a dedicated RFC so the IETF would have a 
consistent interapplication/protocol negotiation mechanism (giving 
the same result whatever the application/OPES).

It also permits the qualification of the space of exchanges the users 
has selected. Which is a part of his right "not to receive what he 
does not want" - this is a major user comportment oriented 
right/demand (ex. spam) which definitely oppose the langtag concept.

>To me, it is. Or let's say that I don't think it's a suitable 
>approach in many other cases, and I have little idea of what might be suitable,
>except in a few cases.

Yes. This is why a lot of work has to be carried. You cannot trust 
negotiation mechanisms at this stage to accept the proper content you 
want.  At this stage one can consider it an insecure/uncertain help.

>Language negotiation is used on some sites, often on sites that 
>somehow relate to technologies of internationalization and 
>localization, and that's fine and natural. But on most sites that 
>are genuinely multilingual, language negotiation is not used. Even 
>the site of the European Union does not use it, and this can hardly 
>depend on lack of resources or on lack of political will to support 
>multilingualism.

This is the overlooked fundamental problem. Harald's RFC line is 
limited to internationalisation/localisation. Multilingual and 
Vernacular are foreign concepts to IETF. Paul Hoffman's RFC 3536 on 
vocabulary shows it. Remarks of John Klensin on this list too (from 
past debates I feel to recall that John has a vision of 
vernacularisation he opposes? But it is true that the "mono-Internet" 
technology imposes all the related features to be supported as 
options what increases complexity and risks of conflicts).

One reproaches me to extend to principles when discussing details. 
The problem - and this is the very reason of the European Union 
current attitude so debated on the main list - is that the current 
Internet architecture does not deliver enough for all these details 
to be supported. I accept that people usually do not phrase it as I 
do, not being network architects. But this is what they identify 
everywhere, in their own words.

As I will be contested on this, let me document it. I created with 
Louis Pouzin and others people Eurolinc (European Languages Internet 
Conference). This is a ccTLD and WSIS accredited lingual 
organisation. Nearly one page of the Geneva action plan has been 
proposed by Louis as Eurolinc. I sponsored our first Chair, 
Jean-Louis Grange who contributed with him to the TCP/IP design (ask 
Vin Cerf). This is to say they are Internet orthodox network 
founders. JLG had some years ago to direct the site of the French 
European presidency. He did not lack experience, money, time and 
expertise around him. His presentation fully confirms what Jukka 
says. On July 1st, I hold a workshop at EGENI 2005 on 
multilingualism. The person in charge of the European policy and 
culture was one the guest speakers I invited. He said nothing 
different. In the few ISO TC37 French,  European and international 
meetings I attended, the position was the same. etc. etc.

>Quite a many features actually. For one thing, the language 
>preferences settings in users browsers are so often wrong in the 
>sense of not actually reflecting the user's understanding and 
>preferences. I would say that they are wrong more often than 
>correct, e.g. because the user has an English-language version of a 
>browser, with some default settings that make "en" the only 
>language, or worse still "en-US", quite independently of the user's 
>native language and other languages.

The word "wrong" may be too strong in most of the cases (except when 
the mental structure inherited from the language is hardly compatible 
with the one inherited from en-US)? Anyway this is why dual language 
standardisation using two sister languages of strong different 
perception (French and English) is important for pertinence and QA.

To make this easier to understand, please consider the word 
"understand, en" and "comprendre, fr". This is the basic word for a 
user facing a machine and trying to use its screen. In the two 
languages they correspond to the same function for this user. But 
conceptually they nearly oppose. The English "understanding" will be 
bottom-up (what is under what stands) and the French "comprendre" 
will be of "everything together". If you consider the Italian 
"capire" what comes from the Chief (top down), the Spanish "entiende" 
for what are the links between, etc. In German, Japanese, etc. This 
underlay concepts which shape cultures and ways of interfacing with 
the reality. You better understand why the way a browser, Word, etc. 
SHOULD be designed to fully support the user/author brain might have 
to be very different.

>How _could_ the preferences be correct e.g. in a computer used in a 

How _could_ the preferences be correct e.g. in a computer built to 
fit the world?

>Why take the trouble when you know that the results will so often be wrong?

Just a minute: this is questioning the legitimacy of this whole WG!

However there are cases where another cultural/lingual vision may 
take over. This is in particular true in global needs such as legal 
aspects, technology issues, for the part the participants want 
global. For example international e-commerce, international 
commercial right, IETF, etc. In such cases, with people educated to 
that kind of cultural/language overlay, it can perfectly work.

This is why I do not object the deliverable of this WG (only its 
pretension to exclusive) and I tried to get it as precise as 
possible. The Internet legacy can permit to use it as a default, 
during some while, among people who have been initially used to 
en-US. As long as the proposed solution also supports the 
multilingualisation and the vernacularisation they need. I hope the 
debate on the IETF main list on the WSIS and the attitude of all the 
countries but one (including Europe) will help en-US people (American 
mother tongue people and people from countries where there is an 
important domain loss to English [one of the most usually quoted ones 
being Norway]) to understand that.

>explicitly requested for the user's language preferences upon 
>installation and the preferences themselves could be easily stored 
>so that they can be transferred from one computer or browser to 
>another, perhaps as part
>of a collection of preferences. (Most of us use different computers, 
>if not different browsers, and it's really frustrating to set all 
>the personal preferences, including language preferences, every time 
>you start using a new computer. In reality, people mostly don't even try.)

This is a good description of a part of the work we engaged and we 
name the DRS (distributed registry system) and Unicode people work on 
a small but important part with CLDR (locales) - what could make us 
unnecessary competitors. This work can be achieved in different ways 
and the task is enormous. In staying at what I call the 
"mono-Internet" level one has not the vision and the tools to address 
it another way than by constraint. The RFC 3066 has leaks listed in 
our Charter. The response the WG is let lock the bolts, when I say 
let open the doors and let the inside and outside be the same.

The way to do that is in having everything ISO 11179 compliant (the 
locale is a registry). The variation of country, community, 
corporation, family, personal "locales", not only for the computer, 
but for the network, the applications, the Telephone, the TV, the 
appliances, the cars, etc. etc. IS a complex thing. It is also the 
true spine of the pervasive network continuity. The first problem is 
to describe it and to standardise that description.

When collectively debating this we may want to recall that most of 
the Members of this WG see these need as an added need: their 
common/basic personal need, experience and usage is satisfied by the 
built-in en-US oriented defaults.

>for a wide range of uses. It is based on the assumption that a 
>user's preferences are expressed as a set of language/q-value pairs. 
>(It's really a set, not a list; the order is insignificant, the 
>q-values define the weights.) In practice, most browsers assign the 
>q-values themselves, on the basis of the order of languages. This 
>indicates that browser vendors regard the q-value system as too 
>difficult to users. And it isn't really
>The q-values are defined as weights, but what do the numbers relate 
>to. If I have q=0.95 for "de", would that mean that I understand 95 % of

An interesting test protocol we discussed at AILF (French Speaking 
Engineers Association) meeting attended by academics from Sweden, 
Tunisia,  Algeria, Japan, China, Senegal, etc.  was that every member 
of a multilingual mailing list would give his DIQ (Differential IQ), 
i.e. the loss of IQ when tested in the different languages of the 
people of the list (you can certainly guess questions in languages 
you do not know, the same as you may have to guess what others may 
want to say in the list language).

They carried a test in Sweden, to analyse the mental speed (time to 
address a set of questions) in Swedish and in English. The panel was 
more than 1000 students if I am correct. And the results consistent. 
I have the name of a searcher to contact, somewhere in my archives if 
someone is interested. The speed loss was significant. This is the 
test which triggered the http://nicso.org/equilang.htm proposition.

jfc

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 1 Oct 2005 13:43:49 -0700

Hi -

(as a technical contributor)

> From: "Jukka K. Korpela" <[email-masked]>
> To: <[email-masked]>
> Sent: Saturday, October 01, 2005 12:10 AM
> Subject: Re: [Ltru] matching: Language Range Lists
...
>    preferences than simply naming (by code) the most preferred language
...

Total agreement.    Our charter is quite narrow and specific regarding this
deliverable.  It says:

| The second document will describe matching algorithms for use with
| language tags. Language tags are used in a broad variety of contexts
| and it is not expected that any single matching algorithm will fit all
| needs. Developing a small set of common matching algorithms does seem
| likely to contribute to interoperability, however, as it seems likely
| that using protocols could reference these well-known algorithms in
| their specifications.

I believe the fact that it talks about "matching algorithms" rather than
"negotiation mechanisms" is important, and has direct consequences
for the document.  Specifically, what we say about negotiation mechanisms
will need to be non-normative, included merely as use cases justifying
our decision to document specific algorithms.

Now, if the working group wants to define algorithms that are not used
by any existing negotiation or retrieval mechanism, that's ok, but keep
in mind that if this document is to ever advance reach the Draft Standard
level, any mechanisms that are not used by multiple interoperable
implementations will need to be removed, under the provisions of
RFC 2026 section 4.1.2 which says:

|   The requirement for at least two independent and interoperable
|   implementations applies to all of the options and features of the
|   specification.  In cases in which one or more options or features
|   have not been demonstrated in at least two interoperable
|   implementations, the specification may advance to the Draft Standard
|   level only if those options or features are removed.

So, before we get too creative, I think its good to consider which
protocols require and can actually use the mechanisms proposed.

Randy