Subject: [Ltru] Re: Preparations for smart matching

From: "Doug Ewell" <[email-masked]>
Date: Sat, 19 Mar 2005 11:37:19 -0800

Randy Presuhn <randy underscore presuhn at mindspring dot com>  wrote:

>> Have you read the new draft on "Matching Language Identifiers"?  It
>> explains that the "default" matching algorithm is not the only
>> possibility.
> ...
>
> Has the i-d been posted?

No, sorry.  My mistake.  There is a document at
http://www.inter-locale.com/ID/draft-ietf-ltru-matching-00.html, but as
such it is completely unofficial.

-Doug Ewell
 Fullerton, California
 http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 19 Mar 2005 14:26:16 -0800

John Cowan <jcowan at reutershealth dot com> wrote:

> In fact it does work, provided that the algorithm keeps around a short
> list of exceptions (much less than the whole registry).  Other than
> the exceptions, one can identify a tag's class knowing only if it is
> initial or non-initial, thus:
> ...
> anything else non-initial: variety

Additionally, the singleton 'x' introduces a sequence of one or more
private-use subtags until the end of the tag, while singletons other
than 'x' introduce a sequence of one or more extension subtags until the
next singleton.

-Doug Ewell
 Fullerton, California

---

From: "Addison Phillips" <[email-masked]>
Date: Sat, 19 Mar 2005 17:29:14 -0800

What's currently "official" (but not the work of this WG) is the previous individual submission version of that document:

http://www.ietf.org/internet-drafts/draft-phillips-langmatching-00.txt

The "editor's copy" at the link Doug listed below is much more up-to-date, of course, as it is being prepared for submission as the starting point on matching. Both documents make the point that the default matching scheme is not the only one possible.

Addison

Addison P. Phillips
Globalization Architect, Quest Software
Chair, W3C Internationalization Core Working Group

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Doug Ewell
> Sent: samedi 19 mars 2005 11:37
> To: LTRU Working Group
> Subject: [Ltru] Re: Preparations for smart matching
>

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 21 Mar 2005 22:38:34 -0800

L.Gillam <L dot Gillam at surrey dot ac dot uk> wrote:

> (OT - I've not yet understood if in having a .uk domain we can't
> include content in non-UK languages on a web-site. Perhaps I've
> missed the point here too).

As I keep saying:  Country codes used as ccTLDs have NOTHING to do with
country codes used in language tags.

Of course, you can post content on your site in any language you like.

-Doug Ewell
 Fullerton, California

---

From: "Peter Constable" <[email-masked]>
Date: Mon, 21 Mar 2005 23:58:14 -0800

> Behalf Of Frank Ellermann
> Sent: Saturday, March 19, 2005 5:57 AM

> Hi, so far en-Latn-US-boont not matching en-boont is one of my
> standard examples why I have a problem with 3066bis as it is.
> 
> This could be solved later by a smarter matching strategy.  Or
> by subtags used in the order of their importance, something
> like en-boont-Latn-US would match en-boont with the normal RTL
> strategy.

A couple of comments. 

First, you haven't defined what the "importance" metric is, and on the
surface it appears to be merely that e.g. "boont" is more important wrt
"en" than are "Latn" or "US" because there is a registered legacy tag
"en-boont". That alone is not a particularly insightful metric -- it is
based only on formal properties of the tags and nothing else. I think it
would be more useful to define a metric in terms of something like
desired functionality in particular scenarios.

Secondly, I think this example is not a particularly good one for what
you're evaluating since the qualifiers "Latn" and "US" would, in the
99.9% case be redundant / assumed and so would not be included in the
tag. It's far more useful to consider scenarios in which qualifying
subtags are reasonably likely to be used, as in cases such as e.g.
zh-Hans-HK. (If you want to use "en-Latn-US-boont" merely as an example
for sake of discussion, that's fine, provided we keep in mind those
considerations.)

Thirdly, no matter what the order of subtags, there will always be valid
scenarios in which left-prefix-based matching does not provide the
functionality you'd like. It's just an inherent limitation of that
approach to finding matches. (I pointed this out in 2001 when we were
discussing the request for de-1901 etc. tags.) The appropriate steps,
then, are (i) to identify what ordering of subtags provides the best
results generally for important or widely-used algorithms that have this
limitation, and (ii) to consider what alternative approaches to matching
might be possible. (Of course, hence this thread and work item.)

> So the last chance is smart matching.  To be smart the matching
> algorithm needs some input...

Of course, one of the goals for the syntax proposed in the draft is to
narrow the set of possible tags from what was allowed under RFC 3066 in
a way that allows the type of each subtag always to be determined
without knowing what specific entity the subtag actually denotes or
refers to. Even if I don't know what specific script "Jpqx" referred to,
I know for certain that in "de-Jpqx" this subtag must refer to a script.

> So de-Latn-US-boot won't match de-boont, because boont is only
> registered for use with en.  Does that make any sense ?  If yes
> we need optional default scripts for languages, and maybe also
> default languages for all variants in the registry.  Bye, Frank

I think it makes sense. On the other hand, the practicality of
documenting default scripts for languages and other things of that ilk
is not obvious. Not long ago, there was a discussion regarding compiling
a list of valid language/country combinations, and one thing that
reinforced for me that such tasks are far from trivial undertakings.

Peter Constable

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 22 Mar 2005 15:11:18 -0800

Yes, exactly. In fact I've been mentally ruminating (but haven't had time to set down in email) about how better to reflect these issues in the matching draft.

In my thinking about the below I mentally envisioned a separate data table, somewhat like the ancillary tables in the UCD. Wanting the core data table to do everything strikes me as a bit messy for no apparent gain in utility.

The real issue here is what form language tags should take. Matching issues may guide us in selecting a particular subtag pattern, but I haven't seen any particular ideas about how to proceed that don't entail matching issues of any sort.

The matching scheme described in RFC 3066 is not suitable for (and, in practice, isn't used in) all cases even today.

In fact, if you think about it, most of what is called "language tag matching" is *really* a form of content filtering. The RFC 3066 matching algorithm (what Mark describes in #1 below) matches a prefix against tags. Specifying "xx-YY" filters out all content that is not *at least* tagged "xx-YY" (and selects content that may be inappropriate, like our friend "en-boont" is picked up by "en")

The de facto content lookup mechanism (what Mark describes in #2 below and which is also used by Web applications of all stripes for controlling locale preferences when generating content or formatting embedded in text) falls back from the specified tag toward default content. This is not filtering, please note.

In these cases, I think Mark is correct about how to proceed. In addition, I would stipulate that once one introduces additional fields into the generative mechanism (even via RFC 3066 registration, which, while it doesn't recognize subtags as such, does recognize their use in matching!!), there will always be cases where traditional "matching" produces results that might be questioned by a human end user (who can see the tag and reason its meaning). 

In other words, registered tags RFC 3066 already have matching problems. Consider the "de-19xx"/"de-CC-19xx" collection of tags, which already demonstrates the "subtag in the middle" selection problem.

Addison

Addison P. Phillips

> From: Mark Davis [mailto:[email-masked]]
> Sent: mardi 22 mars 2005 13:26
> To: Addison Phillips; Peter Constable; Frank Ellermann; [email-masked]
> Subject: Re: [Ltru] Re: Preparations for smart matching
> 
> Some languages are customarily written in multiple scripts, while others
> are
> customarily written in a single script. In the case of English, the
> overwhelming majority of English text is written with Latin, just as the
> overwhelming majority of Arabic is written in the Arabic script. But there
> are, of course, documents that do have ar-Latn, such as certain books
> teaching Arabic. So to *always* map ar to ar-Arab is a mistake. And it
> gets
> fuzzier for cases like Serbian, where while the predominant case of sr is
> sr-Cyrl, but there are significant amounts of sr-Latn.
> 
> It really depends on what the matching process is *for*. There are
> different
> usages of language tags that may have different matching strategies. For
> example:
> 
> 1. Matching. Use of 'sr' says that you want to match Serbian of
> whatever form: doesn't depend on the country, doesn't depend on the script.
> Use of, say, 'sr-Latn-IT' means that you want Serbian, but limited to
> Latin script, and as used in Italy. If I want to have a document query,
> then
> I could use one of the above to restrict the documents that match my query.
> I could get back 1000 documents or none, according to what database of
> documents I am searching and how restrictive my query is. I can also
> probably structure queries with more capabilities, such as (sr AND (Latn
> OR
> Cyrl)) OR (ru-Cyrl))
> 
> 2. Lookup. When you lookup language data, for example, for display on a
> web
> page, it is a different process. You typically don't have the choice of
> displaying nothing if there is not an exact match. Instead, you fall back.
> If you don't have data exactly matching 'sr-Latn-IT', you fallback to data
> for 'sr-Latn'. If you don't have that either, you fallback to data for
> 'sr'.
> Now, whatever data contents someone has associated with 'sr', it has to be
> a
> single consistent type of data, so it will be in one of 'Latn' or 'Cyrl'.
> 
> Similarly, American English may be used in data associated with en for
> lookup purposes. Any distinctions according to country would be stored
> separately in an 'en-AU', 'en-CA', 'en-IE', etc., and if available, would
> be
> found in a lookup. But if someone came in with 'en-JP', and there was no
> separate data source for that, it would fall back to the data associated
> with 'en', which would be American English.
> 
> Thus, I think it may be reasonable, in support of lookup fallback
> strategies, to have a table of data designed for "default" lookup fallback,
> structured along something like the following lines:
> 
> fallback | en | en-Latn
> fallback | sr | sr-Cyrl
> fallback | pt | pt-BR pt-Latn-BR
> 
> However, to me this is information that is much more closely associated
> with
> matching than with the structure of language tags, and thus should be
> discussed and elaborated in the context of the matching document, when we
> start work on that.
> 
> ‎Mark
> 
> ----- Original Message -----
> From: "Addison Phillips" <[email-masked]>
> To: "Peter Constable" <[email-masked]>; "Frank Ellermann"
> <[email-masked]>; <[email-masked]>
> Sent: Tuesday, March 22, 2005 09:47
> Subject: RE: [Ltru] Re: Preparations for smart matching
> 
> 
> > > At the moment the draft says:
> > >
> > > | The "recommended_prefix" field is empty, except where the
> > > | type is "variant"
> > >
> > > That could be changed to e.g.
> > >
> > >   The "recommended_prefix" is generally empty.  In the case of
> > >   a "variant" it's often the recommended "language" prefix, but
> > >   could be also a language-script, language-script-region, or
> > >   language-region prefix.
> >
> > That's an interesting suggestion worth some thought.
> 
> Um...that's what it does say, except that the draft doesn't say what Frank
> says it says. It actually says:
> 
> <quote>
> The field 'recommended prefix' is for use with registered variants and
> contains a semicolon separated list of language-ranges considered most
> appropriate for use with this subtag. Additional values can be added to
> this
> field for variants only via additional registration. Other modification of
> this field (such as removing or changing values) is not permitted.
> </quote>
> 
> A language range is defined to allow inclusion of all of the examples
> Frank
> gives. We can (and should) remove the restriction to registered variants
> in
> the text.
> 
> Another way to accomplish what I think Frank is suggesting is to provide a
> canonical mapping section in the registry. We could create specific
> "range"
> entries with canonical mappings in this manner:
> 
> 
> canonical | en-Latn | English written in Latin Script | en | | # use 'en'
> prefix, not en-Latn
> 
> If users feel strongly about certain combinations they could register them.
> Rules would have to be created to prevent abuse (for example, you couldn't
> create mappings that create ambiguity, such as mapping both zh-Hant and
> zh-Hans to zh). Note that these canonical mappings would be informational
> in
> nature. Tags such as en-Latn-US would still be well-formed and valid. This
> would merely allow tag processors in possession of the registry to ignore
> the subtag Latn when matching the tag (by canonicalizing the tag to en-US
> before matching).
> 
> Best Regards,
> 
> Addison
> 
> Addison P. Phillips
> > >   For the type "language" this field can be used to define a
> > >   default "script" for the correspoding "laguage", and then the
> > >   resulting tag or prefix language-sript is NOT RECOMMENDED.
> >
> > Ah! You'd use this to indicate a default script for cases in which there
> > is one that is predominant. A better suggestion than what I thought you
> > had in mind. Note, though, that there will be gray areas: 99.99% of all
> > en content is in the common Latin orthography rather than other writing
> > systems, such as IPA or Gregg Shorthand; it's clear we'd consider Latn
> > the default in such a case. What if the proportion were 95%? 90%? 85%?
> > ...  What's the threshold at which we begin to assume a default?
> >
> >
> > > > Thus if you want to match "en-Latn-US-boont" with "en-boont",
> > > > it is enough to disassemble both tags and observe that all
> > > > the subtags in the second tag are also present in the first.
> > >
> > > That's IMFO not good enough.  If a client offers en-boont and
> > > en-misc (script "misc"), and the server has en-misc-US-boont,
> > > then the client should try en-misc, but not en-boont.
> > >
> > > Because "misc" is probably very important, if it's not "Latn"
> > > as in the default for en.
> > >
> > > > If "en-Latn-US-boont" is a match for "en-boont", then so is
> > > > "en-Ogam-MV-boont".
> > >
> > > Unless you have en-Ogam, then you need to know the default
> > > script for en (which isn't Ogam).
> >
> > Hang on. It's unclear what you're comparing. You need to start
> > qualifying the argument structure here: you're comparing two tags: one
> > is the what is being requested, the other is the attribute of the
> > resource in the repository. You've got to specify which is which.
> >
> > If the request is for "en-boont", then "en-Ogam" in the repository
> > should not be a match (except in a fallback situation, when you must
> > return something, haven't found a match and are looking for the best
> > available alternative -- e.g. software UI resource loading). That's not
> > because the script isn't Latn; it's because the request was specifically
> > for Boontling, and "en-Ogam" is not constrained to that.
> >
> > If the request is for "en-boont", then ideally you'd want
> > "en-Latn-US-boont" (if that were actually used) in the repository to be
> > a match.
> >
> >
> >
> > Peter Constable
> >

---

From: "Peter Constable" <[email-masked]>
Date: Tue, 22 Mar 2005 21:50:22 -0800

> Behalf Of Debbie Garside

> My point is... if the information on variant subtag is available, use
it...
> even though you may not need it for your own purpose.  This is what I
see
> as
> facilitating, in particular, the semantic web, knowledge management,
> archiving and search/query routines.  RFC 3066bis seems intent on
making
> tags as small as possible (am I wrong? Just my perception), whereas I
for
> my
> purposes, always want as much information as possible, split into the
> tiniest of pieces, I can then include/exclude/use/concatenate in any
way
> required.

There's validity to this point: you can't reliably infer information
that isn't there, so if an item is tagged in a way that doesn't specify
all the relevant attributes that are known, then anybody making use of
that metadata later will not be able to benefit from the knowledge that
existed but was not recorded and therefore lost.

Putting it another way, you can't query using finer distinctions than
exist in the data.

This has to be applied within a given context, however. For instance, if
a library in Thailand is serving primarily a Thai-speaking population
and somehow happens to have a dozen items in various South American
Indian languages, they could tag the items down to dialects of
particular villages, but it would likely be costly for them to do that
and there would be a limited expectation of realized benefit for that
expense. For them, tagging the language of those items in a non-granular
way such as "sai" is adequate for their purposes.

If attributes for an item are already known / inexpensive to determine,
then in general it shouldn't hurt to specify more than may be needed for
immediate purposes -- provided the system within which the info is being
used can accommodate the extra details without adding to the cost of
developing or maintaining the system and without hindering the required
functionality. 

I.e., if it doesn't cost to use a more specific tag, you might as well
use it, as you may not be able to anticipate when that knowledge about a
given item will provide to be useful (and if that knowledge turns out to
be needed later, it will likely be costly to rediscover that knowledge).

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Tue, 22 Mar 2005 22:29:52 -0800

> Yes.  Out of curiosity, what's the idea of ISO 639-3 for the
> reserved "extlang" ?

This was my idea, which arose as a result of a convergence between a
couple of issues I was facing. 

In preparing to work on ISO 639-3, my colleague, Gary Simons, and I had
to figure out how an ID like "zh" should relate to a collection of IDs
for specific languages such as Mandarin, Cantonese, etc. Initially, we
adopted a set of principles for mapping every item in ISO 639-1/-2 to
entries in the Ethnologue. Using those principles, we probably would
have said that "zh" denotes specifically Mandarin, it being the most
developed of the Chinese languages. But then on reflecting on the tags
already registered under RFC 3066, I realized this would be a problem as
there already were tags "zh-yue" etc. that applied "zh" in a tag for
things other than Mandarin. At that point, my choice was to force a
deprecation of  the "zh-yue" etc. tags, or to accept that "zh" means
something broader than one individual language.

There were other cases in which a similar struggle between IDs for
specific languages versus IDs being used more broadly seemed to exist.
This led me, while preparing ISO/CD 639-3 to add an additional level of
scope for language IDs, which I called "macrolanguage". (Not my proudest
act of naming, but I still can't think of anything better.)

The point of macrolanguage IDs is that they de facto are used in ways
that encompass multiple varieties that, for other purposes, would be
considered distinct individual languages. To put another angle on it,
macrolanguage categories are part of the folk taxonomy that people
actually use every day that don't correspond to the categories that a
linguist would otherwise identify.

Now, coming back to the "zh-yue" tags, I noticed that several of these
had a second subtag of three letters, and it occurred to me that, if
there were way that these could be made redundant under a revised
language tagging spec rather than grandfathered, that that might be
beneficial.

I also had to think about what to do with ISO 639-3 IDs for things like
Cantonese, or like Northern Frisian, at a point at which ISO 639-3 were
integrated into a revised language tag RFC: if someone wants to use an
ISO 639-3 alpha-3 tag to identify the content specifically as Northern
Frisian, but we've already got content being tagged using "fy" for
'Frisian' (without further qualification), then how do you do that in a
way that lends itself to interoperability? 

The obvious answer is that you do what's already done for Cantonese
(using a registered tag): you combine the macrolanguage ID "fy" with the
individual-language ID "frr" to create a composite tag "fy-frr",
analogous to "zh-yue". That way you can tag an items specifically as
Frisian and people can request at that level of granularity, but you
will still be able to provide a match for the more likely requests for
generic 'Frisian' using "fy".

I'll leave it at that for the moment.

Peter Constable

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 22 Mar 2005 22:30:12 -0800

Mark Davis <mark dot davis at jtcsv dot com> wrote:

> whatever form: doesn't depend on the country, doesn't depend on the
> script. Use of, say, 'sr-Latn-IT' means that you want Serbian, but
> limited to Latin script, and as used in Italy.

Fine so far.

> 2. Lookup. When you lookup language data, for example, for display on
> a web page, it is a different process. You typically don't have the
> choice of displaying nothing if there is not an exact match. Instead,
> you fall back. If you don't have data exactly matching 'sr-Latn-IT',
> you fallback to data for 'sr-Latn'. If you don't have that either, you
> fallback to data for 'sr'. Now, whatever data contents someone has
> associated with 'sr', it has to be a single consistent type of data,
> so it will be in one of 'Latn' or 'Cyrl'.

But this is just prefix matching.  Falling back from "sr-Latn-IT" to
"sr-Latn" to "sr" doesn't require us or anyone else to define a default
script for Serbian.  And when we get to "sr" and finally find a match,
we have whatever resources we have.  They might be either Latin or
Cyrillic.  The choice won't depend on which script someone decided was
the default.

> separately in an 'en-AU', 'en-CA', 'en-IE', etc., and if available,
> would be found in a lookup. But if someone came in with 'en-JP', and
> there was no separate data source for that, it would fall back to the
> data associated with 'en', which would be American English.

Not if the content provider is in Australia or Canada or Ireland.  Their
data will be in their own flavor of English, which is what they would
logically call "en".

> strategies, to have a table of data designed for "default" lookup
> fallback, structured along something like the following lines:
>
> fallback | en | en-Latn
> fallback | sr | sr-Cyrl

I really don't like the idea of this.  People will see it as a
recommendation or a Diktat that en "means" en-Latn, or (worse) that sr
"means" sr-Cyrl.

We've never gotten involved with second-guessing what people mean when
they write a certain language tag, or choosing one script or region as
"preferred" over another.  I see no reason to start doing so now.

-Doug Ewell
 Fullerton, California

---

From: "Peter Constable" <[email-masked]>
Date: Tue, 22 Mar 2005 22:39:42 -0800

> The RFC 3066bis generative syntax is a superset of the RFC 3066
> generative syntax.  It allows things to be generated that cannot be
> generated under RFC 3066.

This needs a little clarification: 

RFC 3066 has a weak syntax that permits lots of strings; it sanctions a
small set of those, requiring others to be registered explicitly.

RFC3066bis has a stronger syntax that permits a much more restrictive
set of strings than does RFC 3066. On the other hand, it sanctions a
much larger portion of the strings its syntax permits, making the need
for registration (as well as the possibilities for registration) more
limited.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Tue, 22 Mar 2005 23:05:04 -0800

> If I request "de" content, say over the Web, I definitely want to
> receive content in both "de-1901" and "de-1996", not just one or the
> other.  However, if I request "zh" I may not be able to do anything
with
> "zh-hakka".

Of course, if you request "de" content, you might also get back content
that's German but written in various ways not useful to the user, such
as phonetic transcription, some kind of shorthand, or in Devanagari
script, say. That's not different from the "zh-hakka" example.

> If you declare 'Latn' to be the default script for 'en', and use that
> information to collapse the tag "en-Latn" to "en", then you remove the
> user's ability to get "en-Ogam" content unless she specifically
requests
> it.  This essentially says, "Since you requested English, I'll assume
> you can only read it in Latin script, and I won't send you English
> content in any other script."

If a user requests specifically "en-Ogam", that request should never be
reduced to "en" -- and I don't think Frank was suggesting that. The real
issue is whether a request for "en" should return content tagged
"en-Ogam". Existing left-prefix implementations would match, though a
"smarter" matching algorithm might of course work differently.

It is an issue to be considered. So is the reality that there's a ton of
data tagged "en" where that's assumed to mean "en-Latn".

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Tue, 22 Mar 2005 23:21:19 -0800

> But not necessarily de-Brai, so there's definitely a problem.
> So far I thought that a "default script" would help, but now
> you confused me, is a "default script" actually worse ?

Well, there is an issue (one that has been pointed out in the past --
this isn't a new discovery, though it hasn't been heavily discussed
before now).

But it's hard to say the idea of a default script makes things worse
when that is effectively what gets used now -- "en" being used in
99.9999% of cases for English written using the common English alphabet
is effectively making "en" to be an alias for "en-Latn".

> Assuming that it is worse, then we'd need Debbie's idea, we
> SHOULD say de-Latn in new content, and pray that all old pages
> are updated.  Hell, that's impossible, somewhere I lost it.

As long as that's impossible, then we can't say that the idea of
recognizing certain defaults is worse.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Tue, 22 Mar 2005 23:30:05 -0800

> Behalf Of John Cowan

> I think it worth mentioning that there is no ISO 639-6 on the current
> programme of work of ISO TC 37 / SC 2.  (My reference in an earlier
> posting to ISO/IEC JTC 1 was a mistake, and should be disregarded.)
> So at the moment Debbie's draft has no official status with ISO.

Not so. There is an approved new work item for ISO 639-6, so Debbie is
indeed doing ISO TC 37/SC 2-sanctioned work. In terms of ISO stages, I'd
guess it's probably at stage 10.99 -- the project has been approved, but
it is not yet registered in the TC 37/SC 2 program of work (hasn't yet
shown up on the ISO site listed as a new project).

> ISO 639-4 (general principles) and ISO 639-5 (codes for groups of
> languages) are in progress, but are irrelevant to RFC 3066bis.

I can't say for certain that ISO 639-4 would not have any relevance (it
may prove to have some relevance, or it may not); I'd agree though wrt
ISO 639-5, as I don't think we should allow additional collective IDs in
subtags (if anything, I'd restrict what we already allow).

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Tue, 22 Mar 2005 23:39:30 -0800

> Behalf Of L.Gillam

> <a_light_note>I think this is a time where one should at least ask
> questions of the messenger, and perhaps consider less extreme forms of
> punitive actions</a_light_note> - please see:
> 
>
http://isotc.iso.org/livelink/livelink/fetch/2000/2122/687806/ISO_TC_037
__
> Terminology_and_other_language_resources_.pdf?nodeid=1160801&vernum=0

Please note that this document is a little bit out of date. For
instance, "ISO NWIP 639-3", described as being at stage 20.00, should
actually be "ISO DIS 639-3", -- it is at stage 40.20.

Peter Constable

---

From: "L.Gillam" <[email-masked]>
Date: Wed, 23 Mar 2005 09:23:36 -0000

> From: John Cowan [mailto:[email-masked]]

[snip]

> http://isotc.iso.org/livelink/livelink/fetch/2000/2122/687806/
> ISO_TC_037__Terminology_and_other_language_resources_.pdf?node
> id=1160801&vernum=0
> 
> Thanks for this pointer.  This says that 639-3,4,5,6 are all at stage
> 20.00, meaning that they are part of the programme of work.  This is
> clearly inaccurate for 639-3, which is elsewhere stated to be at 40.20
> (DIS out for ballot).

Ah, beaten to it by Peter Constable. Since the document is dated 2004-12-15
and 639 DIS ballot began on 2005-02-04 and terminates on 2005-07-04, I don't
think the accuracy of the document is in question - just that progress has
since been made.

[snip]

> > General principles and groups of languages may be irrelevant to
> > 3066bis, but eventually - in some none-too-distant future one
> > suspects - smart matching will also have to take into account the
> > specialisation/generalisation relationships as composed by these 2 -
> > which is somewhat orthogonal to current 3066 "tag-dropping" 
> practice.
> 
> I hope not.  The trouble with language collections is that if they are
> defined extensionally (en+nl+de+yi+...) they may or may not reflect
> anything real, and if they are defined intensionally 
> ("Germanic language")
> they are theory-dependent and are unstable, changing as 
> linguistic theories
> change.  (Granted, the Germanic family is quite stable, but this isn't
> true in general.)

Would you include the 639-3 "macrolanguages" that Peter described as
being so extensionally defined? The 639-5 language groups?

I wonder how you'd create a theory-independent system?

> 
> > At this point I should promote a related research activity
> > 
> > http://portal.surrey.ac.uk/portal/page?_pageid=798,337934&_dad=portal&_schema=PORTAL
> > for which the  actual website is in development at INRIA.

> Hmm.  It would be nice to know what this is actually about.  :-)

Ok, I'll point you at the project website when it's ready - and user-focussed workshops
are planned in the near future also.

---

From: "L.Gillam" <[email-masked]>
Date: Wed, 23 Mar 2005 09:27:31 -0000

These people should be given a little test:

Q: What is the language of the UK?

If they say it's English, they just offended quite a lot of users.

> From: [email-masked] 
> [mailto:[email-masked]]On
> Behalf Of JFC (Jefsey) Morfin
>
> Just beware that this may change with the IDNs. Registrations 
> are subject 
> to the TLD Manager terms and conditions. The same ICANN 
> imposes conditions 
> on specialized TLDs. There is interest by some ccTLD to 
> require that the 
> language of the IDN be used on the sites, by respect to the users.
> jfc  
>

---

From: "L.Gillam" <[email-masked]>
Date: Wed, 23 Mar 2005 09:35:03 -0000

> subtags (if anything, I'd restrict what we already allow).

I guess the response to this one is along the lines of 
- They are irrelevant to 3066bis. They may be relevant to 3066ter. Or
some future iteration.

---

From: John Cowan <[email-masked]>
Date: Wed, 23 Mar 2005 09:14:07 -0500

L.Gillam scripsit:

> being so extensionally defined?

They are extensionally defined, and what they reflect is that (as Peter
doesn't quite say) ISO 639-2 is a folk taxonomy, and as such doesn't
match up very well with 639-3's rather more scientific coverage of the
same ground.  In principle, there's an ISO 639-2 code for every text,
but only on the principle of "split local things, lump remote things"
that folk taxonomies generally employ.

> The 639-5 language groups?

I haven't seen even an editor's draft of -5 [*], so I can't comment
specifically.  But I will note that in Linnaean taxonomy, heroic
attempts are made to preserve stability for both entities (where the
facts permit it) and labels (to the point where ordinary meaning is
flatly disregarded) at the species level, but no such attempts are made
at higher taxonomic levels, which change all the time and are the subject
of many disagreements.

([*] The notion of ISO that people are going to *pay* for drafts, and
in the two figures of CHF at that, is simply laughable.)

> I wonder how you'd create a theory-independent system?

I wouldn't even try.  I'd accept the collective-language codes of -2
as a regrettable consequence of the way it was designed, or rather not
designed, and leave it at that.

-- 
John Cowan        http://www.ccil.org/~cowan          [email-masked]
Please leave your values                Check your assumptions.  In fact,
   at the front desk.                      check your assumptions at the door.
     --sign in Paris hotel                   --Cordelia Vorkosigan

---

From: "Addison Phillips" <[email-masked]>
Date: Wed, 23 Mar 2005 09:56:07 -0800

> Oops, that was a quote of lines 1131 and 1132 in the file
> draft-ietf-ltru-registry-00.txt.  What you quote is in lines
> 1192..1197.

Yes, you're right. I missed that sentence when I searched for it somehow. And (I think) you're correct that your original quote should be modified to match the later content I quoted in meaning.

> 
> Yes, that could allow other uses for other types.  I've no
> idea what that could be except from "default script" for a
> language.  But maybe there are scripts which are only used
> for a small set of languages, this set could be enumerated
> in the "recommended_prefix" column of the script.

The difference is that any subtag could then have a prefix. Scripts would be one possibility. Regions are another (if we were to encode 200, for example, it might bear a couple of prefixes).

Note that recommended prefix is a different thing than canonical mapping or implied suffix, either of which is what I take default script to be. 

Addison

Addison P. Phillips

> Sent: mercredi 23 mars 2005 01:37
> To: [email-masked]
> Addison Phillips wrote:
> > the draft doesn't say what Frank says it says
> 
> Oops, that was a quote of lines 1131 and 1132 in the file
> draft-ietf-ltru-registry-00.txt.  What you quote is in lines
> 1192..1197.
> 
> > We can (and should) remove the restriction to registered
> > variants in the text.
> 
> Yes, that could allow other uses for other types.  I've no
> idea what that could be except from "default script" for a
> language.  But maybe there are scripts which are only used
> for a small set of languages, this set could be enumerated
> in the "recommended_prefix" column of the script.
> 
>                             Bye, Frank
> 
> 
>

---

From: "Addison Phillips" <[email-masked]>
Date: Wed, 23 Mar 2005 09:58:13 -0800

> "smarter" matching algorithm might of course work differently.

This problem exists (albeit in a very small way) with en-boont, eh? Left-prefix matching relies on the premise (stated to often be false in RFC 3066 itself!) that content which shares a prefix is mutually understandable.
> data tagged "en" where that's assumed to mean "en-Latn".

Not really. The problems are:

1. Users who request en-something (en-US for example) won't get content tagged "en-Latn-something" using left-prefix matching.

2. Users requesting en-Latn won't get content tagged merely "en" using left-prefix matching. (which is what you're saying above) 

The ton of data tagged "en" is *probably* in Latin script, but the author's intentions in that regard are unknown. This has bearing on those languages where script is a possible confusion.

Addison

Addison P. Phillips

> Behalf Of Peter Constable
> Sent: mardi 22 mars 2005 23:05
> To: LTRU Working Group

---

From: "Addison Phillips" <[email-masked]>
Date: Wed, 23 Mar 2005 10:07:52 -0800

> data will be in their own flavor of English, which is what they would

With the proviso that the country of textual origin and the language tag do not always match. Regions are helpful in identifying language only insofar as they describe a particular language variation. Some variations just don't occur, even if the language is spoken there.

For example, there are plenty of Japanese who speak English, but "en-JP" isn't the language that they are speaking: it is probably "en" or one of the "en" variations such as "en-US" or "en-GB".

> "preferred" over another.  I see no reason to start doing so now.

I tend to agree. But this issue keeps being brought up, so it makes sense to talk about alternative ways of addressing it. The idea of a default script seems rational or plausible at the outset, but it only becomes a problem when used in conjunction with other subtags in the default matching scheme.

That is, there is no problem with "zh-Hant" (to pick an example) or with "zh-TW". Both of these tags identify specific aspects of Chinese language. The problems only occur when you wish to combine the script and regional variations into "zh-Hant-TW" because prefix-based matching (seems to) create a tension between choosing the script and choosing the region (coupled with the fact that one tag predates the other and has been used rather liberally to mean the other in some cases).

I believe that 3066bis solves most of the *identification* problems one can envision (on varying scales of granularity). The problem it does not completely solve is that of matching in the face of existing implementations (of left-prefix matching). Only compromises are possible once you consider additional subtags (no matter what order you put the subtags into, although, pace Frank Ellermann, we can discuss how best to order them when forming tags).

I think that, in this light, what I was saying is that some informative information to assist current and future implementations with this is worthy of consideration, probably in the matching draft.

And, of course, Rule 0 of language tagging should be restated here: "Choose language tags wisely."

The definition of wisdom is open and fungible over time. One possibility that I would like to (re-)raise is that the draft might not need to address directly the choice of whether or not to use a particular script subtag. It may turn out that, over time, Debbie Garside is correct and that content authors will adopt the currently superfluous 'Latn' subtag for use with "en" tags (for example), as more capable language tag processing becomes the norm. Think of it as the "natural selection" of language tags in response to the evolution of algorithms.

Best Regards,

Addison

Addison P. Phillips

> Sent: mardi 22 mars 2005 22:30
> To: LTRU Working Group

---

From: "Peter Constable" <[email-masked]>
Date: Wed, 23 Mar 2005 19:40:57 -0800

> Not legacy code so much as legacy tagging.  There may be documents out
> there
> tagged "nai", worthless as that is for almost all purposes except
> completeness,
> so we can't just deprecate it wholesale -- the argument has been made
that
> expecting people on the other side of the planet to discriminate
between
> various North American Indian languages in their holdings is just
> unreasonable.

One additional point regarding this: there's a difference between an
author tagging a resource and a cataloguer tagging a resource: the
author will generally know the language the content was authored in;
whether a cataloguer can recognize the language of an item to be
catalogued is dependent on the cataloguer and the content and cannot be
guaranteed.

Peter Constable