Subject: [Ltru] Extlang issues

From: Leif Halvard Silli <[email-masked]>
Date: Fri, 02 May 2008 15:38:08 +0200

Ok, I come stumbling in here, in the last second... but I wondered if 
all issues regarding extended languages really have been considered - 
now that publcation is near.

I want to compare the Norwegian situation, which I know well, with the 
Chinese, that I know less well. Because, it seems to me that the 
proposed changes will create a situation for Chinese that resembles the 
situation we allready have for Norwegian today - which is less than 
optimal. (For Chinese, it looks as if it can become even far less than 
optimal than currently for Norwegian.)

I have shown [1] that authors, users, web servers do not understand the 
langauge subtags for Norwegian. Norwegian ('no') is the macrolangauge 
(as per ISO 639-3) for Nynorsk ('nn') and Bokmål ('nb'). But

   1. there is a tendency to use 'no' as a tag for the hegemony language
      (Bokmål). Last version of Apache, Apache 2.2.8 is set up like this
      exacalty this way - with .no for 'no' and .nn for 'nn'. This
      steals playground from the minority language. (But also the
      opposite can happen: a friend just discovered that on Mac OS X,
      Safari only asks for 'nb-NO' ...);
   2. users and browsers tend to have 'no' as a defacto first preference
      (if not the only preference), which can cause Nynorsk users to
      loose content tagged as 'nn';
   3. and authors tend to think that the tags should work in a
      hierarchical way, meaning that they find 'no-nn' and 'no-nb' more
      natural than 'nn-NO' and 'nb-NO'. The official goverment web site
      of Norway currently has this error, except that instead of nn they
      write ny: 'no-NY', and they - as well - use 'no' for Bokmål. See
      http://www.regjeringa.no ;
   4. provided 'no-nb' and 'no-nn' could be used, they would be more
      backwards compatible than 'nn' and 'nb', since (in Apache), if the
      browser asks for 'no-nn', then a file tagged 'no' will be served
      if 'no-nn' can't be found. (For this reason, the grandfathered
      'no-nyn' and 'no-bok' are more compatible with a browser asking
      for 'no' than are the preferred 'nb' and 'nn'.)

Now, for Chinese, you create a situation were for instance 'yue' 
should/can be used for written Cantonese. So, let's imagine I get a web 
browser which is set up to ask for 'yue', only. Then, suddenly, I will 
loose any content tagged with 'zh'. This is unfortunate, as I think all 
those users want written Mandarin if they can get written Cantonese.

However, had the code for written Cantonese been zh-yue, then, setting 
the browser to ask for zh-yue would, in case a Cantonese resource was 
lacking, give you Mandarine (zh, or zh-something) instead, if available, 
since Apache sees zh as a partial match for zh-yue.

The only solution I see to this problem, both for Norwegian and for 
Chinese, is to make sure that files are served with several alternative 
language tags.  Thus, for Norwegian, I will serve a Bokmål file both 
with a 'no' tag and with a 'nb' tag, so that both requests for 'no' as 
well as for 'nb' will yield the same Bokmål file.

Likewise, for a file tagged as 'yue', I could make sure that it was also 
tagged as 'zh'. However, then I would, at the same time, also have to be 
certain that the written Mandarin file could use 'zh-Hans' or 'zh-Hant', 
or else they would get Cantonese instead of Mandarin. Are Chinese 
users/browser applications for Chinese good at specifying whether they 
want simplified or traditional Mandarin, so that 'zh' would work fine as 
'back-up tag' for Cantonese? (This is also something that resembles the 
Norwegian situation: Setting up 'nb' for Bokmål, while using the general 
'no' for Nynorsk is also something I have considered. If I could be 
certain that all Bokmål users had 'nb' first on their list, then this 
would work fine. Unfortunatly, I can't be sure about that.)

In the end, tagging with several tags might be more complicated for 
Chinese than for Norwegian. And, regardless, was it the intention, that 
files should be tagged with several tags?

Here is another scenario: I serve 'file.nn.html' and 'file.nb.html' in 
parallell. But inside those files I have embedded an audio file, with 
the tag 'no' (e.g. audio.no.mp3, referenced in the HTML source as just 
'audio.mp3'), because it is aimed for use in both languages. (Typically, 
for example an audio file containing a interview would be considered 
suitable both in a Nynorsk file as well as in a Bokmål file, unless the 
"interview" is actually a theatrical play in one of our two literature 
languages - Bokmål/Nynorsk.) Now, if browsers were asking for 'nb' or 
'nn' only, respectively, then they might not get the 'no' tagged audio file.

However, if tags for Nynorsk and Bokmål were 'no-nno' and 'no-nob' 
respectively (or 'no-nn' and 'no-nb'), so that the Bokmål file was 
tagged 'no-nob' and the Nynorsk file 'no-nno' and the audio file was 
tagged 'no', then both browsers asking for Nynorsk as well as browsers 
asking Bokmål would get that audio file, since 'no' was a partial match 
for. Of course, in a Norwegian only context, this would not be a problem 
- I did no need to tag the file with any language tag at all, then. But 
else, on a multilingual web site I would, again, have to tag the file as 
both 'nn', 'nb' and perhaps even 'no', in order to be certain that all  
Norwegian users would get the audio file.

[1] http://www.w3.org/mid/[email-masked]

Regards,
Leif Halvard Silli,
Oslo
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Fri, 02 May 2008 15:47:24 +0200

Correction:

Leif Halvard Silli 2008-05-02 15.38:
> [...] This is unfortunate, as I think all those users want written Mandarin if they can get written Cantonese. [...]
>   

The above should be  '... if they can _NOT_ get written Cantonese'.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 2 May 2008 12:07:19 -0600

Hi -

As a technical contributor...

> From: "Leif Halvard Silli" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
> Sent: Friday, May 02, 2008 7:38 AM
> Subject: [Ltru] Extlang issues
...
> well as for 'nb' will yield the same Bokmål file.
...

I agree the situation is a mess, and I really dislike "zh" being used to
*mean* Mandarin, rather than meaning "something Chinese, which
is consequently overwhelmingly likely to be Mandarin".

However, to address the cases you cite, wouldn't it be simpler and
more robust (and also more likely to give these users exactly what
they want) for them to simply put "no" / "zh" as their second choice
in their lists of language preferences?

Truncation fallback is at best a helpful heuristic, and can fail even for
seemingly easy cases.  My sense of the discussion on this list has
been that though there may be macrolanguage cases where truncation
fallback happens work nicely, that this is not generally the case,
and that the failure cases are too numerous to justify organizing the
specification around the few cases where it can be made to "do the
right thing".

I think the reason it's so ugly is that what the appropriate fallback
language(s) will be for a particular language is not a matter of genetic
relationships between them, but rather has to do with the history and
politics of the speaker communities of the languages, and the extent
to which one can legitimately assume that members of a particular
community will have some competance in the fallback language.

Randy

---

From: John Cowan <[email-masked]>
Date: Fri, 2 May 2008 15:18:02 -0400

> in their lists of language preferences?

Apparently, some browsers insist on copying your language preference
from a system-wide locale setting, and then make it annoying (or perhaps
impossible) to specify a more intelligent Accept-Language: header.

Furthermore, servers tend to have a simplistic model of tags in which
rules like "if nn is lacking, fall back to no" cannot be specified,
and a simplistic model of content negotiation where each resource is
associated with at most one language tag.

These are IMHO the real issues here, and unfortunately they are out of
scope for us.

-- 
I marvel at the creature: so secret and         John Cowan
so sly as he is, to come sporting in the pool   [email-masked]
before our very window.  Does he think that     http://www.ccil.org/~cowan
Men sleep without watch all night?
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 03 May 2008 08:38:11 +0200

> in their lists of language preferences?

To get 'no' inserted as second choice for Nynorsk as well as for Bokmål 
users, would be right, in the current circumstances. It should be added 
as default from the application developer/localizer.

I think truncation fallback could work better than current situation for 
most authors,  but perhaps be less simple to customize for (semi) server 
experts.

> specification around the few cases where it can be made to "do the

Is there an example of where it does not work well? And why are you only 
having macrolanguages in mind? Truncation fallback is what takes place 
when en-GB falls back to en. And that works well?

> community will have some competance in the fallback language.

Picking the default fallback languages should be part of the 
localisation process, therefore.

And perhaps those fallback languages should be possible to link to 
domains? E.g. a Finland-Swedish user would accept Finnish as fallback 
under (most of) the .fi domain, but not when visiting e.g. Google. (And 
some CMS-es are almost too language sensitive. I once got out-of-context 
Nynorsk menus when visiting the web site of a US university.)

> associated with at most one language tag.

Right. Though, you can associate (in Apache) a file with several 
languages if you add several corresponding language extensions to it.

> These are IMHO the real issues here, and unfortunately they are out of

Solving those technical things could ge us a long way.

Extlang tags would, I think, lead to more *legitimate* use of the 
macrolanguage tags ('no', 'zh') than would a regime where there are 
"independent" language tags. At the same time, to actually use the 
spesific tag would, with extlang tags, not be so dangerous as it can be 
without them.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 4 May 2008 23:17:27 +0200

On Fri, May 02, 2008 at 03:38:08PM +0200,
 Leif Halvard Silli <[email-masked]> wrote 

>       http://www.regjeringa.no ;

[Note: the mistake in this site seems to be only in the Dublin Core
metadata, the xml:lang and the HTML lang attribute seems correct.]

This is an issue what we'll have, whatever technical choice we
make. Users will make mistakes, not because LTRU work is
undecipherable but because language issues are complicated. Most of
this issue is out of scope for the RFC. Of course, we should be sure
the RFC is clear and well-explained but that won't make all Web sites
valid.

More specifically, in the case of extlangs, the discussion a few
months ago clearly showed that we will have misunderstandings in both
cases (with or without extlangs).

IMHO, this is a meta-issue which requires a different kind of effort:
education, capacity building, etc. If someone wants to write texts for
http://www.langtag.net/ "Users" section, for instance on the proper
tagging of norwegian and chinese, that would be great. Also, did you
report the problem to the webmaster of http://www.regjeringa.no?

---

From: Leif Halvard Silli <[email-masked]>
Date: Mon, 05 May 2008 01:08:24 +0200

> metadata, the xml:lang and the HTML lang attribute seems correct.]

You will also find this META tag:
<meta http-equiv="Content-Language" content="NO-NY" />

> report the problem to the webmaster of http://www.regjeringa.no?

They are aware of it. Writing for Langtag.net is good tip. Thanks.
-- 
leif halvard silli