Subject: [Ltru] Re: extlang again (was: [psg.com #1011] add suppress-region)

From: "Doug Ewell" <[email-masked]>
Date: Mon, 30 May 2005 09:42:48 -0700

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

>> (I also don't see where this 17 limit came from.)
>
> 72 is the line length in the registry.  LWSP allows to fold
> behind the colon, Macro-Language:<CRLF><SP>lng-ex1-etc<CRLF>
>
> 72 minus one SP minus three "lng" is 68, and 68 / (1+3) = 17.

Wow, OK.  That's a lot of work, and ultimately it does lead to the
result you came up with: one primary language subtag plus 17
extended-language subtags.

But John is right too: such a tag is never going to be valid.  This
business with macrolanguages and the sub-languages they contain
currently involves two levels, and may involve three levels at some
point, but it is not ever going to involve 17 levels.  The hierarchy
exists because things like "Chinese" and "Arabic" and "Zapotec" have
been encoded in the past as one entity each, and the experts have
determined they should actually be broken into multiple entities.  But
it seems incredibly unlikely (I know, that's not good enough for some)
that this refinement could ever involve more than three or four levels.
The existing groupings are just not that coarse.

I suppose we could set an upper limit of, say, five extlangs, but I fail
to see how that qualifies as "keeping it simple."

> The fuss is that *("-" extlang) was discussed quite often,
> but I never got the idea that it's _redundant_ info.

I wouldn't have used the term "redundant" in the sense of "completely
unnecessary because already covered by other features."  In the tag
"ar-arb", one might consider the subtag 'ar' to be redundant because
'arb' already denotes a flavor of Arabic.  But it's also possible that a
user searching for "Arabic" may want to retrieve all flavors of Arabic,
in which case the generality implied by a search for "ar" (matched by
"ar-arb" and "ar-shu" and others) would be appropriate.

Note that at no point are we saying the extlang itself is redundant.
Rather, it is the primary language subtag in this scenario that could be
considered redundant.

> And we're supposed to prepare anything as far as possible
> for the extlang-s, and that would require a field name
> "Macro-Language:" (or similar), because adding new field
> names later could break some new 3066bis registry readers.

At the rate this group is inventing and renaming fields, it wouldn't
surprise me at all if some future version of the language tagging
document did introduce a new field name.  I would suggest that 3066bis
registry parsers that are intended to be forward compatible with future
versions should quietly ignore fields they don't understand, and would
further suggest (TICKET ALERT) that a passage to this effect should be
added to the draft.

> These tags might be used in any piece of MIME compatible
> software in some way, in any browser, in any Web server,
> in any XML lib, Ned mentioned IMAP, we talked about JAVA.

OK, each of these applications can set (and document) its own limit.

>> Memory leaks are always the fault of the developer (or
>> rarely, a buggy development environment) -- never the spec.
>
> The spec. should be KISS where possible.

"Keep It Simple, Stupid" (or bowdlerized to "Keep It Short and Simple")
makes a great lapel button or picket sign, but the sad fact is that not
all problems are simple, and not all problems are best served by the
simplest possible solution.

See also the Zero-One-Infinity Rule, which is certainly more KISS than
setting a scalar limit > 1.

And no matter how simple or complex the specification, memory leaks are
still the fault of the developer.

> Reserving 1 KB
> per tag is fine when you have it, but some devices might
> be forced to find a smaller buffer size still trying to
> support all "important" subtags (up to and including the
> "script").  And *("-" extlang) comes before the "script".

Then they can set (and document) their own limits.  Isn't Section 2.1.1
clear enough on this?  And I still don't believe it is necessary to
solve this problem, if such it is, within the current draft.

--
Doug Ewell
Fullerton, California
http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 30 May 2005 20:57:09 +0200

Doug Ewell wrote:
 
> I suppose we could set an upper limit of, say, five extlangs,
> but I fail to see how that qualifies as "keeping it simple."

For a developer reading the draft as is, without an explanation
for the Macro-Language plus last extlang idea, a clear limit of
five means 3 + 5 * (1+3) + (1+4) + (1+3) = 32 for any solution
guaranteeing to support at least the subtags up to the region.

With an 2047 encoded word <SP>=?charset*tag?B?content?=<CRLF>
you have 76-9 = 67 characters for the charset name plus the
tag plus the encoded content.  For the "minimal" 32 mentioned
above and "UTF-8" (5) that's 76-9-5-32 = 30 characters for the
encoded content.

For B (base64) 30*3/4 is about 22.5 source bytes, and in the
case of UTF-8 about 22.5/3 = 7.5 Unicode points.  Or in other
words, the hypothetical limit of five extlang-s with radical
truncation before any variant still allows to encode up to 7.5
Unicode points per folded 2231/2047 phrase (assuming UTF-8 and
Base 64).

If we'd consider one maximal variant (8) as important we'd get
21 instead of 30, or about five B64 UTF-8 characters.

Whatever we do, less than one worst case codepoint for 2231 is
impossible.  A proposal to limit the extlang-s to ten instead
of five would be utter dubious for this reason.

> Note that at no point are we saying the extlang itself is
> redundant.  Rather, it is the primary language subtag in
> this scenario that could be considered redundant.

Yes, looking from the "last" extlang to the "prefix" I got it
wrong, of course the Macro-Language is redundant, because it's
determined by the last extlang, the last extlang is required.

One vague idea I had, maybe restricting this stuff to at most
one extlang is the best solution.  So if they want to split
ar-shu later into ar-shu-sha and ar-shu-shb, we could say that
this won't be reflected in real tags but only in the registry.
Real tags would simply use ar-sha or ar-shb.

If a smart matching algorithm "knows" (because the registry
says so) that ar-shu "includes" ar-sha and ar-shb, it's fine,
and otherwise forget it.

> At the rate this group is inventing and renaming fields, it
> wouldn't surprise me at all if some future version of the
> language tagging document did introduce a new field name.

"Macro-Language" is only the second after "Suppress-Script",
and it's apparently necessary for the extlang-s.  Nobody liked
my idea to use one neutral name like "Associated" to combine
"Suppress-Script", "Macro-Laguage", and "Recomended-Prefix"
depending on the record type.
                              Bye, Frank

---

From: John Cowan <[email-masked]>
Date: Mon, 30 May 2005 21:56:42 -0400

> Real tags would simply use ar-sha or ar-shb.

The logic for not doing that is the same as the logic for 
doing macrolanguages in the first place: a macrolanguage covers
multiple languages, and a macrolanguage encompassed by a larger
macrolanguage does too.

> depending on the record type.

It would be too easy to get the semantics wrong.  Having separate
headers ensures that each is treated correctly, without having to
worry about what the type is.

-- 
The Unicode Standard does not encode            John Cowan
idiosyncratic, personal, novel, or private      http://www.ccil.org/~cowan
use characters, nor does it encode logos        http://www.reutershealth.com
or graphics.                                    [email-masked]