Subject: RE: [Ltru] wildcards in lookup

From: "McDonald, Ira" <[email-masked]>
Date: Mon, 5 Dec 2005 11:13:29 -0800

+1

A specification that depends heavily on the use of wildcards MUST
specify their behavior in a manner that is: (a) reasonable to a 
non-technical user; and (b) consistent across all implementations.

Since language tags are restricted to a subset of ASCII, saying
"numerically least" is easy to understand and consistent (though
not fully deterministic, of course).

Cheers,
- Ira

Ira McDonald (Musician / Software Architect)
Blue Roof Music / High North Inc
PO Box 221  Grand Marais, MI  49839
phone: +1-906-494-2434
email: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Mark Davis
> Sent: Monday, December 05, 2005 12:59 PM
> To: Addison Phillips
> Cc: LTRU Working Group
> Subject: Re: [Ltru] wildcards in lookup
> 
> 
> The problem is that your method subverts all user expectations about 
> what wildcards mean. If I supply the range "*-US", and the possible 
> content is
> 
> es
> es-MX
> en
> en-US
> fr
> fr-FR
> fr-BE
> [i-default]
> 
> then en-US is the obvious, clear choice. That is, after all, what is 
> being asked for: a language tag consisting of language code = any + 
> region code = 'US'.
> 
> The same goes for where the range is "en-*". Given the above tagged 
> content, it would be bizarre not to return "en-US".
> 
> The implementation, for any system I know of, is not rocket science. 
> Logically, you just enumerate through the available tags for 
> the content 
> you have, and see which (if any) matches.
> 
> The only problematic case is the one where the range matches 
> 2 or more 
> language tags. In that case, I don't think we have any choice but to 
> leave this implementation dependent. (Actually, we do; my preference 
> would be to have a fully specified specification that used 
> alphabetically-least, with the proviso that implementations 
> are free to 
> customize this area at will. But I can live with the incomplete 
> specification if you really want it.)
> 
> Mark
> 
> Addison Phillips wrote:
> 
> >Separately I sent an email responding to various small items 
> Mark/Martin suggested I change. This item, though, I think 
> needs its own thread.
> >
> >Mark suggested:
> >
> >=====
> >
> >    Wildcard values in an extended language range are 
> supposed to match
> >    any value that occurs in that position in a language 
> tag. Since only
> >    one item can be returned for any given lookup request, 
> the wildcards
> >    must be processed in a predictable manner (or the same 
> request might
> >    produce widely varying results). Thus, for each range in the
> >    language priority list, the following rules must be applied to
> >    produce a basic language range for use in the fallback mechanism
> >
> >    ...*
> >    Figure 10: Extended Language Range Fallback Example *
> >
> >=>
> >
> >    Wildcard values in an extended language range are 
> supposed to match
> >    any value that occurs in that position in a language 
> tag. Since only
> >    one item can be returned for any given lookup request, 
> the wildcards
> >    must be processed in a predictable manner (or the same 
> request might
> >    produce widely varying results). Where more than one piece of
> >    content would be returned, then the implementation MUST 
> specify the
> >    method used to determine the result.
> >    [The method supplied in the document doesn't match 
> reasonable user
> >    expections, as stated earlier. Since we can't agree, 
> leave it up to
> >    the implemention.]
> >
> >=====
> >
> >And later Martin opined:
> >
> >  
> >
> >>>>   Where more than one piece of
> >>>>   content would be returned, then the implementation 
> MUST specify the
> >>>>   method used to determine the result.
> >>>>        
> >>>>
> >>>'Where', 'would', and 'then' don't make a reasonable 
> sentence for me.
> >>>More general, the paragraph starts with "Since only one item can be
> >>>returned", so then speaking about more than one piece is confusing.
> >>>      
> >>>
> >>How about:
> >>
> >>Where the use of a wildcard would cause matches with two different
> >>pieces of content, the implementation MUST specify the 
> method used to
> >>choose between them. For example, the implementation could 
> specify that
> >>the content tagged with the alphabetically-least language 
> tag is returned.
> >>    
> >>
> >[Addison Phillips] 
> >
> >Mark and I don't agree about this and I dislike Martin's 
> suggestion even more. Here's the problem: when I implemented 
> a lookup scheme I didn't seek all of the possible matching 
> content first. I used the tag to construct directory or file 
> names, truncating until I found a matching bit of content.
> >
> >When I consider Java's ResourceBundle class (which finds all 
> matching content building out from root to the specified 
> value before, returning content when the lookup fails to find 
> a file), I see that the algorithm would have to change 
> appreciably. Notably, it would have to load some content by 
> name, with no way of knowing what that name actually was. 
> >
> >That is, most implementations today actually would be forced 
> to do what I propose: they rely on knowing the name tokens. A 
> wild card works better if it matches nothing in that case.
> >
> >I think it is problematic not to say anything about what to 
> do here. We know clearly what the problem is. Is there some 
> reason why we should not specify at least one mechanism (we 
> do not have to make it normative, as it is now)?
> >
> >To illustrate a bit more, if the range is "en-*-US", an 
> implementation can either:
> >
> >1. Try to enumerate all matching content. If content is 
> represented as directory or file names, the complete list of 
> file/directory names must be loaded from one or more 
> locations on disk and then processed to find all of the 
> matches. These can then be selected based on some algorithm.
> >
> >2. Work as lookup has historically worked, by searching 
> until there is a "miss" and return that item. If one works 
> out from root, then the tree must be fully populated. 
> >
> >So I don't favor making this change or I would prefer to 
> make changes that spell some of this out.
> >
> >Addison
> >
> >
> >Addison P. Phillips
> >Globalization Architect, Quest Software
> >http://www.quest.com
> >
> >Internationalization is not a feature.
> >It is an architecture. 
> >
> >  
> >
> >-------------------------------------------------------------
> -----------
> >
> >_______________________________________________
> >Ltru mailing list
> >[email-masked]
> >https://www1.ietf.org/mailman/listinfo/ltru
> >  
> >
> 
> 
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ltru
> 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Addison Phillips" <[email-masked]>
Date: Mon, 5 Dec 2005 11:32:30 -0800

It will produce silly results as often as ignoring the wildcards would. In any case, I don't care that much. I just wanted to point out that the proposed text may not fit very many extant implementations. Despite what Mark writes, my proposal for making extended language ranges deterministic in lookup DOES produce a result consistent with the user's request. It may be LESS consistent than searching all of the content, as Mark points out, but it is consistent.

I note (again) that lookup mechanisms I've personally written will not work with this: they return the first matching content they find and would be seriously slower if they had to enumerate all of the content first. As long as both are legal, though, I don't think it matters and don't actually care.

While you two were writing these responses, I wrote some proposed changes to the text, to wit:

--
<t>The special language range "*" matches any language tag. In the lookup scheme, this range does not convey enough information by itself to determine which content is most appropriate, since it matches everything. If the language range "*" is the only one in the language priority list, it matches the default content. If the language range "*" is followed by other language ranges, it should be skipped.</t>

<t>In some cases, the language priority list might contain one or more extended language ranges (as, for example, when the same language priority list is used as input for both lookup and filtering operations). Wildcard values in an extended language range normally match any value that occurs in that position in a language tag. Since only one item can be returned for any given lookup request, wildcards in a language range have to be processed in a consistent manner or the same request will produce widely varying results. Implementations that accept extended language ranges MUST define which content is returned when more than one item matches the extended language range.</t>

<t>For example, an implementation could return the matching content that is first in ASCII-order. For example, if the language range were "*-CH" and the set of content included "de-CH", "fr-CH", and "it-CH", then the content labeled "de-CH" would be returned.</t>

<t>Another way an implementation could address extended language ranges would be to map them to basic language ranges: if  the first subtag is a "*" then the entire range is treated as "*" (which matches the default content), otherwise the wildcard subtag is removed. For example, if the language range were "en-*-US", then the range would be mapped to "en-US".</t>
--

FWIW, I also tugged at the definition of "default content" too:

--
<t>The default content is implementation defined. It might be content with no language tag; might have an empty value, as with xml:lang in the XML specification <xref target="XML10"></xref>; might be a particular language designated for that bit of content; or it might be content that labeled with the tag "i-default" (see <xref target="RFC2277"></xref>). When performing lookup using a language priority list, the progressive search MUST proceed to consider each language range in the list before finding the default content or empty tag.</t>

<t>One common way for an application or implementation to provide for default content is to allow a specific language range to be set as the default for a specific type of request. This language range is then treated as if it were appended to the end of the language priority list as a whole, rather than after each item in the language priority list.</t>
--

Addison P. Phillips

> From: McDonald, Ira [mailto:[email-masked]]
> Sent: Monday, December 05, 2005 11:13 AM
> To: 'Mark Davis'; Addison Phillips
> Cc: LTRU Working Group
> Subject: RE: [Ltru] wildcards in lookup
> 
> +1
>