Subject: [Ltru] Summary of 2007-10-03 teleconference

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 3 Oct 2007 11:09:50 -0700

Hi -

There was a teleconference to discuss remaining open issues in
the ltru drafts this morning.  I volunteered to serve as scribe.
I hope I got it right.  If I didn't, I'm sure someone will
correct me.  In any case, let's see if we can finish this discussion
by working through the tradeoffs and agreeing on something, since
it's pretty clear at this point that neither approach is perfect.

The primary topic of discussion was the question of what to do
about macrolanguages and extlangs.  The two major alternatives
both had implications for matching.

There was general agreement that remove-from-right matching is not
a panacea.  There was less agreement about whether a macrolanguage-
extlang construct is ever helpful, or actually counterproductive,
in its interactions with matching.

One consequence of the "flat" approach (not including encapsulating
subtags in a language tag encoding) is that more intelligent matching
would require additional information from the registry, which would
increase the risk that someone might build applications that would
read the registry, presenting a possible DoS with respect to IANA.

There was some disagreement about what is meant by a macrolanguage
subtag by itself.  Two examples: does zh mean the same thing as zh-cmn,
and does ar mean the same thing as ar-arb?  A *rough* consensus
of those participating in the teleconference (this must not be
understood as a determination of WG consensus) was that they mean
different things.  In this view, "zh" means "undifferentiated Chinese"
(much of which happens to be "zh-cmn"), and "ar" means "undifferentiated
Arabic".  A minority perspective was that because so much content
tagged with "zh" is "zh-cmn", they should be understood as effectively
synonymous.

In fleshing out the details of the alternative approaches, we need to
consider the potential for synonyms, and what guidance we provide to
taggers in the cases where synonyms could arise, as well as the extent
that we might want to distinguish between "true" and "effective" ones.

There was a side discussion about the possibility of additional
macrolanguages being registered, and about the possibility of the
registration of additional exncapsulated languages.  None of the
likely scenarios appeared to create any serious problems for either
proposal.

Yet another consideration discussed is the extent to which either approach
would require the retagging of existing data.

A final consideration will be how to explain it all to the BCP's users.

Randy

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 4 Oct 2007 10:20:05 -0700

> subtags in a language tag encoding) is that more intelligent matching

is that some people felt
[this was not universal]

> read the registry, presenting a possible DoS with respect to IANA.

(Suppress Script already requires access to the registry. One could
theoretically argue that this is only required on the tagging side, not the
matching side, but in practice any reasonable application is going to
probably need to have a snapshot of information from the registry.)

> tagged with "zh" is "zh-cmn", they should be understood as effectively

One of the key issues we need to face are the problems introduced by either
of the models:

no-NO
nn-NO (or no-nn-NO*)
nb-NO (or no-nb-NO*)

vs.

zh-Hant-HK
zh-yue-Hant-HK or yue-Hant-HK
zh-cmn-Hant-HK or cmn-Hant-HK

While theoretically no-NO and nb-NO are different, in practice every
implementation I know of ends up having to treat them as synonyms -- that
wouldn't have changed if they had been encoded as no-nb-NO and no-NO. The
same is true of zh-Hant-HK and zh-cmn-Hant-HK, or ar-SA and ar-arb-SA.

-- 
Mark

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 4 Oct 2007 11:04:37 -0700

Hi -

As a technical contributor...

> From: "Mark Davis" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Thursday, October 04, 2007 10:20 AM
> Subject: Re: [Ltru] Summary of 2007-10-03 teleconference
...
> same is true of zh-Hant-HK and zh-cmn-Hant-HK, or ar-SA and ar-arb-SA.
...

We need to be careful when talking about "synonyms", when what we're really
talking about are the statistical properties of imprecisely tagged data.
Such tags are "synonyms" only for the purposes of some applications' heuristics.
In hindsight, "no-nn"* / "no-nb"* would have been less horrible that the
current mess, even if data (imprecisely due to broadening) tagged "no"
would have been mostly of one type.  (I read "zh" as "some kind of Chinese",
and based on experience know that if a text is "some kind of Chinese" that
it's most likely, but not guaranteed to be, Mandarin.

An inevitable consequence of our work to make more precise tagging possible
it that there will be legacy data tagged less precisely than we might like.
There are some cases where the ability to tag imprecisely, and to match imprecisely,
are useful. The question is the extent to which we want to support it
directly in tag structure, rather than requiring more thorough registry
knowledge to make use of it.

Of course, it's legitimate to ask whether the cases where such imprecision
is actually useful are worth supporting.

Randy

---

From: John Cowan <[email-masked]>
Date: Thu, 4 Oct 2007 15:02:13 -0400

Randy Presuhn scripsit:

> are useful. 

Indeed there are.  For example, if I'm tagging documents of modern
origin, I can recognize Chinese documents (in Han script) with pretty
close to 100% reliability.  That is, I can tell if a printed document
is entirely in hanzi (with maybe some Latin) as opposed to being in
kanji+kana or hanja+hangeul.  But Mandarin or Cantonese?  No clue.
Simplified or traditional? Ya got me.  Country of origin?  A mystery.

At present I can unequivocally and correctly tag these documents "zh".

-- 
John Cowan <[email-masked]>             http://www.ccil.org/~cowan
It's like if you meet an really old, really rich guy covered in liver
spots and breathing with an oxygen tank, and you say, "I want to be
rich, too, so I'm going to start walking with a cane and I'm going to
act crotchety and I'm going to get liver disease. --Wil Shipley

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 4 Oct 2007 12:55:44 -0700

> talking about are the statistical properties of imprecisely tagged data.

True. But we don't want to build an ivory-tower system that works in theory
but fails in practice: we also have to build on the experience that we have
in dealing with tags. We know that tags are underspecified and misused, and
not just in edge cases -- in large percentages. So we have to be cognizant
of that, and design a system that is as clear and robust as possible. The
more complicated we make the system, the more likely it is that people will
make errors. In the above cases, essentially every implementation will end
up having to treat these as synonyms if they want to deal realistically with
tags. We can ignore that fact and end up with a fragile system, or we can
recognize that fact and end up with a more robust system.

And the whole goal mentioned for extlang like suppress-script is backwards
compatibility. In that case we absolutely do have to be cognizant of past
usage and implications of our model for that. Recall the whole discussion
around "gsw" as meaning Alemanish vs Swiss German.

Such tags are "synonyms" only for the purposes of some applications'
> heuristics.
> would have been mostly of one type.

Actually, the best case would have been neither. It would have been add
*only* "nn" -- not "nb", and have documentation to the effect that:
Although theoretically "no" means any Norwegian, in practice it is
interpreted as Bokmal. People tagging nynorsk are encouraged to use "nn"
instead.

We would have then ended up without having to jump through hoops. Moreover,
my contention is that when you actually look at the code, having no-nb
instead of nb doesn't make life any easier at all; overall it simply adds
complication which is better for some algorithms and worse for more.

And as I pointed out in the call, ISO is giving us no guidance here. Take
the case of Old Czech; if a code is introduced for that does that narrow the
scope of "cz" to be just modern Czech, or does it leave it as broad? If they
introduce the old code, then we are probably best documenting "cz" as best
used for just modern Czech, just like "en" is best used for just modern
English, and "ang" for Old English.

 (I read "zh" as "some kind of Chinese",
> it's most likely, but not guaranteed to be, Mandarin.

In an ideal world, we might never have this situation. But the situation is
far from clear. How many people have looked at interfaces that say simply "中文"
and think "that could mean Mandarin but could also mean Hakka" and how many
think "that means just Mandarin, they don't offer Hakka so I'll pick
something else", and how many think "that means just Mandarin, they don't
offer Hakka, but Mandarin is the closest to Hakka that is offered so I'll
pick that."

> knowledge to make use of it.

Yes, I think that is the crux of the matter.  My position is that we will
have more satisfactory results overall if we (a) treat all languages as
equal, and not bake in some of them into a secondary position, and (b)
provide the macro language information so that people can use if if they
want.

If we bake it in, then every simple algorithm will in practice automatically
fall back from Cantonese to Mandarin, fall back from Dari to Persian, fall
back from Khetrani to Lahnda -- and in doing so, strip the script and
country information. That, unless they fix the algorithm to pretend that the
secondary language is in fact a primary language. So we are forcing people
into a model that is often, or mostly, wrong. If we supply the information
in the registry, then implementations can choose whatever they think is
appropriate, given the particular facts about languages and the particular
needs of their applications without having to work around the extlang
mechanism.

I'm reminded again of a similar case with C++. The assignment operator gets
a default implementation. That must have seemed like a nice convenience for
the user, but except for toy programs, it is always, always wrong. So
supplying that default just means that people usually have to take extra
steps to disable it, and prevent it from causing bugs in their programs. I'm
worried about this being similar.

It is clear that companies like Google or Yahoo can work around the problems
with extlang -- what I'm worried about are the people who don't have a lot
of experience with these matters, and are just led down a garden path. I'm
really not trying to be obstreperous on this issue, but there does seem to
be a positive correlation between the people who have had detailed
implementation experience with filtering and matching these tags in
production environments and those who are have doubts about extlang.

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 4 Oct 2007 12:56:15 -0700

I can't tell whether "chat" is in French or English either. I'm not sure
that's a good argument for having a code that means either.

More seriously, one could argue for having a root for each set of languages
could sometimes not be detectably different. So we'd have

rom*  for Romanian and Moldavian

That is an interesting thought, though perhaps although difficult to draw
the lines in: what or who determines what is detectably different? For
example, you said that you personally couldn't tell whether some text was
simplified or traditional -- but for any substantial body of text, they are
pretty easy to tell apart statistically, and much easier for human readers.
And from what John Jenkins has said, Cantonese text is also often
distinguishable from Mandarin by character choice and phrasing.

Anyway, when in doubt, one can use lists of the likely languages.

Mark