Subject: RE: [Ltru] Re: Preparations for smart matching

From: "Addison Phillips" <[email-masked]>
Date: Tue, 22 Mar 2005 09:47:35 -0800

> > At the moment the draft says:
> >
> > | The "recommended_prefix" field is empty, except where the
> > | type is "variant"
> >
> > That could be changed to e.g.
> >
> >   The "recommended_prefix" is generally empty.  In the case of
> >   a "variant" it's often the recommended "language" prefix, but
> >   could be also a language-script, language-script-region, or
> >   language-region prefix.
> 
> That's an interesting suggestion worth some thought.

Um...that's what it does say, except that the draft doesn't say what Frank says it says. It actually says:

<quote>
The field 'recommended prefix' is for use with registered variants and contains a semicolon separated list of language-ranges considered most appropriate for use with this subtag. Additional values can be added to this field for variants only via additional registration. Other modification of this field (such as removing or changing values) is not permitted.
</quote>

A language range is defined to allow inclusion of all of the examples Frank gives. We can (and should) remove the restriction to registered variants in the text.

Another way to accomplish what I think Frank is suggesting is to provide a canonical mapping section in the registry. We could create specific "range" entries with canonical mappings in this manner:

canonical | en-Latn | English written in Latin Script | en | | # use 'en' prefix, not en-Latn

If users feel strongly about certain combinations they could register them. Rules would have to be created to prevent abuse (for example, you couldn't create mappings that create ambiguity, such as mapping both zh-Hant and zh-Hans to zh). Note that these canonical mappings would be informational in nature. Tags such as en-Latn-US would still be well-formed and valid. This would merely allow tag processors in possession of the registry to ignore the subtag Latn when matching the tag (by canonicalizing the tag to en-US before matching).

Best Regards,

Addison

Addison P. Phillips
Globalization Architect, Quest Software
Chair, W3C Internationalization Core Working Group

Internationalization is not a feature.
It is an architecture. 

> 
> 
> >   For the type "language" this field can be used to define a
> >   default "script" for the correspoding "laguage", and then the
> >   resulting tag or prefix language-sript is NOT RECOMMENDED.
> 
> Ah! You'd use this to indicate a default script for cases in which there
> is one that is predominant. A better suggestion than what I thought you
> had in mind. Note, though, that there will be gray areas: 99.99% of all
> en content is in the common Latin orthography rather than other writing
> systems, such as IPA or Gregg Shorthand; it's clear we'd consider Latn
> the default in such a case. What if the proportion were 95%? 90%? 85%?
> ...  What's the threshold at which we begin to assume a default?
> 
> 
> > > Thus if you want to match "en-Latn-US-boont" with "en-boont",
> > > it is enough to disassemble both tags and observe that all
> > > the subtags in the second tag are also present in the first.
> >
> > That's IMFO not good enough.  If a client offers en-boont and
> > en-misc (script "misc"), and the server has en-misc-US-boont,
> > then the client should try en-misc, but not en-boont.
> >
> > Because "misc" is probably very important, if it's not "Latn"
> > as in the default for en.
> >
> > > If "en-Latn-US-boont" is a match for "en-boont", then so is
> > > "en-Ogam-MV-boont".
> >
> > Unless you have en-Ogam, then you need to know the default
> > script for en (which isn't Ogam).
> 
> Hang on. It's unclear what you're comparing. You need to start
> qualifying the argument structure here: you're comparing two tags: one
> is the what is being requested, the other is the attribute of the
> resource in the repository. You've got to specify which is which.
> 
> If the request is for "en-boont", then "en-Ogam" in the repository
> should not be a match (except in a fallback situation, when you must
> return something, haven't found a match and are looking for the best
> available alternative -- e.g. software UI resource loading). That's not
> because the script isn't Latn; it's because the request was specifically
> for Boontling, and "en-Ogam" is not constrained to that.
> 
> If the request is for "en-boont", then ideally you'd want
> "en-Latn-US-boont" (if that were actually used) in the repository to be
> a match.
> 
> 
> 
> Peter Constable
> 
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ltru

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 22 Mar 2005 13:25:44 -0800

Some languages are customarily written in multiple scripts, while others are
customarily written in a single script. In the case of English, the
overwhelming majority of English text is written with Latin, just as the
overwhelming majority of Arabic is written in the Arabic script. But there
are, of course, documents that do have ar-Latn, such as certain books
teaching Arabic. So to *always* map ar to ar-Arab is a mistake. And it gets
fuzzier for cases like Serbian, where while the predominant case of sr is
sr-Cyrl, but there are significant amounts of sr-Latn.

It really depends on what the matching process is *for*. There are different
usages of language tags that may have different matching strategies. For
example:

1. Matching. Use of 'sr' says that you want to match Serbian of
whatever form: doesn't depend on the country, doesn't depend on the script.
Use of, say, 'sr-Latn-IT' means that you want Serbian, but limited to
Latin script, and as used in Italy. If I want to have a document query, then
I could use one of the above to restrict the documents that match my query.
I could get back 1000 documents or none, according to what database of
documents I am searching and how restrictive my query is. I can also
probably structure queries with more capabilities, such as (sr AND (Latn OR
Cyrl)) OR (ru-Cyrl))

2. Lookup. When you lookup language data, for example, for display on a web
page, it is a different process. You typically don't have the choice of
displaying nothing if there is not an exact match. Instead, you fall back.
If you don't have data exactly matching 'sr-Latn-IT', you fallback to data
for 'sr-Latn'. If you don't have that either, you fallback to data for 'sr'.
Now, whatever data contents someone has associated with 'sr', it has to be a
single consistent type of data, so it will be in one of 'Latn' or 'Cyrl'.

Similarly, American English may be used in data associated with en for
lookup purposes. Any distinctions according to country would be stored
separately in an 'en-AU', 'en-CA', 'en-IE', etc., and if available, would be
found in a lookup. But if someone came in with 'en-JP', and there was no
separate data source for that, it would fall back to the data associated
with 'en', which would be American English.

Thus, I think it may be reasonable, in support of lookup fallback
strategies, to have a table of data designed for "default" lookup fallback,
structured along something like the following lines:

fallback | en | en-Latn
fallback | sr | sr-Cyrl
fallback | pt | pt-BR pt-Latn-BR

However, to me this is information that is much more closely associated with
matching than with the structure of language tags, and thus should be
discussed and elaborated in the context of the matching document, when we
start work on that.

‎Mark

----- Original Message ----- 
From: "Addison Phillips" <[email-masked]>
To: "Peter Constable" <[email-masked]>; "Frank Ellermann"
<[email-masked]>; <[email-masked]>
Sent: Tuesday, March 22, 2005 09:47
Subject: RE: [Ltru] Re: Preparations for smart matching

> That's an interesting suggestion worth some thought.

Um...that's what it does say, except that the draft doesn't say what Frank
says it says. It actually says:

<quote>
The field 'recommended prefix' is for use with registered variants and
contains a semicolon separated list of language-ranges considered most
appropriate for use with this subtag. Additional values can be added to this
field for variants only via additional registration. Other modification of
this field (such as removing or changing values) is not permitted.
</quote>

A language range is defined to allow inclusion of all of the examples Frank
gives. We can (and should) remove the restriction to registered variants in
the text.

Another way to accomplish what I think Frank is suggesting is to provide a
canonical mapping section in the registry. We could create specific "range"
entries with canonical mappings in this manner:

canonical | en-Latn | English written in Latin Script | en | | # use 'en'
prefix, not en-Latn

If users feel strongly about certain combinations they could register them.
Rules would have to be created to prevent abuse (for example, you couldn't
create mappings that create ambiguity, such as mapping both zh-Hant and
zh-Hans to zh). Note that these canonical mappings would be informational in
nature. Tags such as en-Latn-US would still be well-formed and valid. This
would merely allow tag processors in possession of the registry to ignore
the subtag Latn when matching the tag (by canonicalizing the tag to en-US
before matching).

Best Regards,

Addison

Addison P. Phillips

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Tue, 22 Mar 2005 23:02:08 +0100

>usages of language tags that may have different matching strategies. For

Is that not a good documentation of the reason why we need a semantic 
rather than a single universal format?
jfc

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 23 Mar 2005 10:37:08 +0100

> the draft doesn't say what Frank says it says

Oops, that was a quote of lines 1131 and 1132 in the file
draft-ietf-ltru-registry-00.txt.  What you quote is in lines
1192..1197.

> We can (and should) remove the restriction to registered
> variants in the text.

Yes, that could allow other uses for other types.  I've no
idea what that could be except from "default script" for a
language.  But maybe there are scripts which are only used
for a small set of languages, this set could be enumerated
in the "recommended_prefix" column of the script.

                            Bye, Frank

---

From: John Cowan <[email-masked]>
Date: Sat, 26 Mar 2005 22:11:15 -0500

Mark Davis scripsit:

> documents I am searching and how restrictive my query is. 

Something I didn't really understand about RFC 2616 matching until now
is that if a server has content tagged en-gb and en-us, but none tagged
just en, then it's conformant for the server to return either one in a
request for en without telling you about the other in any way.

> single consistent type of data, so it will be in one of 'Latn' or 'Cyrl'.

This algorithm should be added to the matching draft.

I also have a third algorithm to propose, which I have not worked out
in full detail.  It accepts a language pattern to match against and a
set of documents to be matched, each with a language tag.

1) The language pattern and the language tags are pre-normalized by
mapping grandfathered and obsolete tags into modern equivalents.

2) The language pattern and the language tags are normalized into
quadruples of the form (language, script, country, variant), where
extended language is considered part of language and x-private-codes
are considered part of the language if they are initial and part of
the language if not initial.  Missing components are set to "*".  An "*"
pattern becomes the quadruple ("*", "*", "*", "*").

3) Each document is assigned a figure of merit such that higher values
indicate better matches and lower values indicate worse ones.

	a) If the language matches, add 8 to the figure of merit.
	b) If the script matches, add 4 to the figure of merit.
	c) If the region matches, add 2 to the figure of merit.
	d) If the variant matches, add 1 to the figure of merit.

Elements of the quadruples are considered to match if they are the same
or if one of them is "*".

4) 15 is a perfect match, 0 is no match at all.  Any match less than 8 is
almost certainly not worth considering.

This algorithm may have bugs, and is certainly not expressed in the
necessary detail.  I invite people to fiddle with it.

-- 
Is not a patron, my Lord [Chesterfield],        John Cowan
one who looks with unconcern on a man           http://www.ccil.org/~cowan
struggling for life in the water, and when      http://www.reutershealth.com
he has reached ground encumbers him with help?          [email-masked]
        --Samuel Johnson