Subject: Re: draft-phillips-langtags-08, process, specifications, "stability", and extensions

From: Bruce Lilly <[email-masked]>
Date: Sat, 1 Jan 2005 10:57:58 -0500

>  Date: 2004-12-30 16:02
>  From: Tex Texin <[email-masked]>
>  To: 
>  CC: [email-masked], [email-masked]
>  
> As the number of question marks, exclamation marks, asterisks and other forms
> of expressing digital shock and awe increase with each mail, I would like to
> suggest a temporary cease and desist policy with respect to responding to JFC
> until the chair chimes in, presumably after the holidays.

What "chair"?; there is no WG associated with the ietf-languages
list, nor with the individual-effort draft under discussion.
There is of course an IETF/IESG chair, and he has in the past
(most recently about six months ago) participated in the ietf-
languages discussion of a predecessor of the draft under
discussion, but I see no reason to suggest an end to discussion
of issues arising out of a NEW Last Call for comments on that
draft.  I would agree that some of the rhetoric could be toned
down...

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Peter Constable" <[email-masked]>
Date: Sat, 1 Jan 2005 18:27:01 -0800

> From: [email-masked] [mailto:ietf-languages-
> [email-masked]] On Behalf Of Bruce Lilly

> > It was removed in the development of RFC 3066, which was appropriate
> because it was a particular application involving language tags;
> 
> We may be in danger of confusing terminology; the Content-Language
> field is a means of specifying the language of a (part of) a
> document, using a language-tag in the context of the MIME protocols.
> "The World Wide Web" is an application.

The *use* of RFC 3066 language tags in a particular field of some protocol is an application of RFC 3066. That is what I meant.

> Separating the specification
> of language via a field from registration procedure was entirely
> appropriate, as BCP documents are used for procedures and policies
> and not for technical specifications.

I'm not as deeply versed in the distinct purposes of distinct kinds of documents used for Internet specifications as you. It's my understanding that there is a general expectation that BCP documents are used for something like a registration procedure but not for a technical specification. Joel Halpern recently reported to me that he provide similar feedback on the draft to Harald Alvestrand some time ago, that Harald responded with reasons why things were mixed in this case, and that the two of them concluded that mixing these in a BCP document in this case was acceptable.

We already anticipate future revisions, and it would be a possibility to consider whether a division of the content into distinct documents of different types would be better. In view of the time already taken, the delays incurred, and the fact that there have been products in development that have been assuming the completion of this current round of revision, and think it makes best sense to allow the mixing that has existed since RFC 1766 to persist in a BCP for this round.

> So I take it that you agree that the technical specification of
> matching algorithms should also be separated from the tag registration
> procedure?

Again, apart from knowing the preferred divisions of content into different types of IETF documents, I don't think there's a problem in describing one type of matching algorithm in this document provided it is recognized that some applications may require different algorithms. And, again, I don't think it would be particularly helpful to delay completion of this revision any further to address an issue of mixing different kinds of content in a BCP that has existed through two versions already; if it's a serious problem, it can be remedied in the next, already-anticipated revision.

> > The *meaning* of any given language tag would be no more or less a
> problem under the proposed revision than it was for RFC 3066 or RFC 1766...

> That's a somewhat different take on the issue; certainly the ability
> to use a generative mechanism (i.e. w/o review/registration of an
> entire tag) can lead to a proliferation of incompatible uses by
> independent generators (and possibly loss of interoperability as a
> result). The draft under discussion would expand use of generative
> mechanisms to encompass all but private-use tags, and thereby expands
> the potential for such incompatibilities and loss of interoperability.

That was a issue I initially voiced when it was first suggested that the registry be a registry of subtags rather than tags. In practice, I'm not sure at this point that there's really a significant greater problem with the new level of generativity than there was before. The reason is that the new elements have quite specific semantic effects on the whole, whereas the semantic impact of a region ID on the whole is less certain: it may imply dialectal variants, spelling variants; it might actually reflect nothing but simply have been inserted because it could. In contrast, there is little question of what effect a script ID such as "Hans" has on the whole.

> > > Under the proposed draft, anybody may legally generate
> > > a tag such as
> > >   sr-Latn-CS-gaulish-boont-guoyu-i-enochian
> > > or
> > >   sr-Latn-CS-gaulish-boont-guoyu-i-enochian-x-foo
> > > with *no* specific registration requirements (i.e. all components
> > > are either registered or require no registration). In the latter
> > > case, a parser can only determine that it contains a private-use
> > > subtag after wading through the other subtags.  In either case,
> > > it is difficult (to say the least) for the recipient or his
> > > software to determine what the generator of that tag intended to
> > > convey.
> >
> > I've shown that this is no different in general that what already exists
> for RFC 3066 or RFC 1766.
> 
> It is certainly different; under RFC 3066 rules such a tag (as a whole)
> would be subject to review and registration.

You're describing a pathological case that is never going to occur, which I don't think is particularly helpful. If the meaning of "guoyu" and of "boont" are clearly documented, it will be evident that something like "sr-boont-guoyu" is basically as meaningless as "sr-guoyu" or as "pl-Hant-TH". 

You might respond that "pl-Hant-TH" is at least conceivable whereas "sr-guoyu" is oxymoronic, but it's just as useless as long as it has no correlation with the real world. I do not think it should be a requirement of a language tag specification to constrain combinations that are not useful, inconsistent with reality, or logically impossible. Likewise, it really isn't necessary IMO to insist on registration of complete tags vs. subtags just to avoid tags that are not useful, inconsistent with reality, or logically impossible.

> No, you seem to have missed the point; there exist RFC 3066
> implementations. Such implementations, using the RFC 3066 rules,
> could match something like "sr-CS-Latn" to "sr-CS", but could
> not match "sr-Latn-CS" to "sr-CS". By changing the definition of
> the interpretation of the second subtag, the proposed draft fails
> to be compatible with existing deployed implementations (which is
> what is meant by "backwards compatibility", which is a prime
> consideration for Internet protocols).

Ah, but RFC 3066 does not sanction use of tags like "sr-CS-Latn" without registration, and no such tags are registered. 

Because of the prevelance of implementations that use a left-prefix matching algorithm, it is more useful to combine elements in the order "sr-Latn-CS" rather than "sr-CS-Latn". If "sr-CS-Latn" were used, these implementations would fail to match "sr-CS-Latn" with "sr-Latn", which is actually a greater problem than failing to match "sr-Latn-CS" with "sr-CS".

> > At this point, I feel confident that it is not a problem to combine
> script IDs into "language" tags, and this is the consensus of the domain
> experts that have been discussing this proposed revision for the past year
> and more.
> 
> Evidently w/o considering the implications of and for core Internet
> protocols. 

You assume this is w/o such consideration. I think otherwise. I can't say that consideration has been given to every single individual protocol. But consideration has been given to many different protocols and usage scenarios. I think it's appropriate for the onus to be on someone to identify particular problems they feel would exist with protocols that concern them (which is precisely the kind of thing we have last-call announcements for).

> If script *can be* specified in a language tag *between*
> the language code and country code, then a parser must be able to
> recognize that case and deal with it appropriately (which, as noted
> above, existing RFC 3066 implementations in deployed use do not and
> cannot do) at *any* time and in any context (context may not be
> available when a Content-Language field is parsed). 

As described above, I think this argument is invalid.

> I don't have an
> issue with provision for specification of script where appropriate,
> but for crying out loud, at least do so in a compatible manner (e.g.
> a Content-Script field) rather than a) breaking compatibility with
> deployed protocols and b) burdening applications which need not be
> concerned with script from having to parse script information.

I've stated that the imputed back-compat problem is a non-issue. Lot's of consideration was given early on to this. If you want to press this argument, I think you need to show exactly how a problem would result in realistic usage scenarios.

Can you identify for us an Internet protocol that would not be concerned with script distinctions? 

Can you identify an Internet protocol for which matching algorithms imply that "sr-CS-Latn" makes better sense than "sr-Latn-CS"?

> > > > There is a clear need for script codes...
> >
> > > But none of that applies to an audio file of spoken material,
> > > where script would be superfluous...
> >
> > Not a problem: the proposed revision *allows* for the use of script IDs
> but does not require them.
> 
> Yes, it's a problem. Having allowed them, each parser must be able
> to handle them.

Look, they're already there in registered tags. This draft isn't doing anything new in that regard.

> > > and, as noted above, would
> > > lead to loss of backwards compatibility.
> >
> > But, as noted above, this is not an issue that is peculiar to the
> proposed revision -- it already existed in RFC 3066.
> 
> No, given a primary subtag which is a language code (and per RFCs
> 1766 and 3066, that's any primary subtag with 2 or more (RFC 3066
> only, more being limited to 3) characters), the second subtag --
> in either RFC 1766 or RFC 3066 language tags -- is always a country
> code and never a script code.

Go back and read RFC 3066 again. It does not impose that constraint:

<quote>
   The following rules apply to the second subtag:

   - All 2-letter subtags are interpreted as ISO 3166 alpha-2 country
     codes from [ISO 3166], or subsequently assigned by the ISO 3166
     maintenance agency or governing standardization bodies, denoting
     the area to which this language variant relates.

   - Tags with second subtags of 3 to 8 letters may be registered with
     IANA, according to the rules in chapter 5 of this document.
</quote>

It must be a country ID *if* it is two letters, but not otherwise.

> The proposed draft pulls the rug out
> from under existing parsers by changing that.

You are completely mistaken on this point -- the proposed draft does not change the constraint you assumed as that constraint never existed.

> Again you seem to be conflating established Internet Standards Track
> protocols with "applications"

I apparently am using "applications" in a sense you're not familiar with. I don't think it's that uncommon to refer to a specification A that makes use of another specification B as an application of B.

> and ignoring the critical importance of
> backwards compatibility.

As stated earlier, I quite disagree that back-compat issues have been ignored.

> > Note that there is nothing that prevents other applications from using
> other matching algorithms, including perhaps something that is able to
> recognize in "az-AZ" and "az-Latn-AZ" that both involve Azeri and used in
> Azerbaijan.
> 
> The issue at hand is the existing deployed base of RFC 3066
> implementations that depend on the matching algorithm specified
> therein (which doesn't work with a script tag interposed between
> language code and country code).

You say that these do not work; these implementations will still work, but they will match "sr-Latn" but not "sr-CS" with "sr-Latn-CS". If that is a problem, please explain why.

> > This is all a discussion we on the IETF-languages list went through five
> years ago, and in the intervening five years I think we have reached a
> consensus on these issues, that consensus being reflected in the proposed
> revision to RFC 3066. (Note that we made the relevant decisions over a
> year and a half ago when we reached a consensus to register az-Latn etc.
> The precedent was established then; the proposed revision adds nothing new
> in this regard.)
> 
> As previously noted, that is a danger recognized by RFC 2026 in
> activity that does not conform to IETF procedures; it is
> possible to reach good consensus on the wrong approach.

Well, that potential was created when RFC 1766 was first approved. Tags like az-Latn could have been registered under the terms of that RFC just as readily as RFC 3066.

But you are speaking as though it's a problem that these tags are registered. I have no idea why.

> > 7.1 says...

> > The proposed revision does not create Internet-specific versions of ISO
> standards...

> By cherry-picking, it effectively seeks to establish such a version.

I would not call what is done "cherry-picking". Any identifier defined in the source standard is valid for use, except in the case that the identifier was previously defined with a different meaning in that ISO standard. That isn't cherry-picking; that is a blindly-applied general principle, created with reasoned motivation: to provide stability.

But speaking of selective usage, have you noticed that RFC 3454 identifies specific characters from ISO/IEC 10646 as prohibited? Various space and control characters are not permitted, INVISIBLE TIMES isn't permitted, END OF AYAH isn't permitted, COMBINING GRAVE TONE MARK isn't permitted... How is what is proposed in this draft any more "cherry-picking" than that?

> > 10.1 states a general policy regarding IP...

> The ISO, as developers of ISO 639 and 3166, have rights. In particular,
> they have the right to determine what those standards specify -- in
> whole -- and they have the right to revise and amend those standards,
> and are the sole arbiters of what is (and what is not) "valid".

They certainly have and retain rights over standards for language, script and country identifiers. They do not, however, determine what is valid for use in Internet protocols. Just as it is appropriate for an IETF document RFC 3454 to specify for particular reasons that certain encoded entities of ISO/IEC 10646 are not valid for Stringprep output, so also it is appropriate for an IETF document to specify for particular reasons that certain encoded entities of an ISO standard are not valid for use in language tags used on the Internet.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Sun, 2 Jan 2005 23:09:31 -0800

> > Ah, but RFC 3066 does not sanction use of tags like "sr-CS-Latn" without
> registration, and no such tags are registered.
> 
> Precisely; an RFC 1766/3066 parser, based on the 1766 and
> 3066 specifications, can expect four classes of language tags:
> 1. ISO 639 language code as the primary subtag, optionally
>    followed by an ISO 3166 country code as the second tag
> 2. i as the primary tag; complete tag registered
> 3. x as primary tag; private-use
> 4. some other IANA-registered complete tag
> 
> "sr-CS-Latn" fits category 1. "sr-Latn-CS' fits none.

You are mistaken; "sr-Latn-CS" fits your category 4.

> > I've stated that the imputed back-compat problem is a non-issue.
> 
> You haven't convinced me of that.  Show me source code of an
> existing, deployed, RFC 3066 parser that handles "sr-Latn-CS".

It matches the RFC 3066 syntax, and so can be recognized; the notion of language-range is still applicable, and nothing about that tag would prevent language-range handling. In what way could a parser *not* handle it? Even if I had my finger on source code, I can't demonstrate that a fault doesn't exist if you only say "there's a problem in there somewhere".

> > If you want to press this argument, I think you need to show exactly how
> a problem would result in realistic usage scenarios.
> 
> I have explained the classes of tags described by RFCs
> 1766 and 3066, and how the proposed changed syntax permits
> tags which do not fit in any of those classes.

It has been shown that you have described such classes incorrectly.

> In the
> interest of interoperability, I believe the onus is on the
> proposers of the revised format to demonstrate that existing
> deployed implementations will be able to handle the revised
> syntax with no loss in functionality (meaning, e.g., that
> "sr-Latn-CS" must be recognizable by all such deployed
> implementations and be interpreted as equivalent to "sr-CS").

Why is it a requirement that a request for "sr-CS" must match "sr-Latn-CS"? That's quite unreasonable. It's like saying that a bunch of new characters are added to Unicode and existing implementations should recognize strings using the new characters as being equivalent to strings using only existing characters. *That tag represents _new_ functionality.*

> RFC 1766/3066 registered tags are integral tags, and can't
> be meaningfully (in the context of a parser) be said to
> contain a script subtag;

If one is registered with a script subtag, then they contain a script subtag.

> the entire tag needs to be recognized
> by a 1766/3066 parser and treated as a unit.

And nothing prevents that happening with a tag containing a script subtag.

>  The draft
> certainly changes that, in a way which an RFC 1766/3066
> parser cannot be expected to cope.

Not at all. RFC 1766/3066 need to be able to deal with tags that contain pieces they don't know about -- the only subtags they can know about are initial subtags of "i", "x" or ISO 639 IDs, or a second subtag consisting of an ISO 3166 code in case the first subtag is and ISO 639 ID. There are lots of other possible subtags permitted by RFC 1766/3066, including subtags that happen to be script IDs from ISO 15924. This draft does not change that in the slightest.

> Convince me by demonstrating that all deployed implementations
> handle "sr-Latn-CS" at least no differently than "sr-CS-Latn".

Why? They should not, be design.

> > You say that these do not work; these implementations will still work,
> but they will match "sr-Latn" but not "sr-CS" with "sr-Latn-CS". If that
> is a problem, please explain why.
> 
> No, unregistered "sr-Latn" is not a valid RFC 3066 language-tag. Nor
> is "sr-Latn-CS".  "sr-CS-Latn" is likely valid (the first two subtags
> are legal and have defined interpretation; RFC 3066 says that there
> are no requirements (implicitly including registration) other than
> syntax for third and subsequent subtags). "sr-CS" is clearly valid
> and in use. An RFC 1766/3066 parser/matcher has a chance of matching
> legal "sr-Cs-Latn" containing script designation with legal "sr-CS"
> (no script specified). 

In your comments here, you are being rather loose in your assessment of what is or isn't valid. The tag "sr-Latn" is a registered, valid RFC 3066 language tag. The tag "sr-Latn-CS" is not registered, but could be and would be valid if registered. The tag "sr-CS" is certainly valid; I have no idea how widely it is used. The tag "sr-CS-Latn" would be valid if registered, but is not registered (and it is unlikely that, if requested, a consensus could be obtained to register it, given the preference among those involved in reviewing requests for a different ordering of subtags).

*If* "sr-CS-Latn" were registered (it is not), then a language-range matcher *must* match a request of "sr-CS" with content tagged "sr-CS-Latn". In preceisely the same way, if "sr-Latn-CS" were registered, a language-range matcher would, and without modification could, match a request of "sr-Latn" with "sr-Latn-CS".

You cannot say that "sr-Latn-CS" has any less or more likelihood of being handled by existing language-range matchers than "sr-CS-Latn". Either the matchers work per the terms of RFC 3066 or they do not, and RFC 3066 does not indicate that either of these is any less valid than the other.

> The proposed draft would make "sr-CS-Latn"
> illegal and would instead require "sr-Latn-CS" which cannot be
> recognized as a valid language tag by an RFC 1766/3066 parser, let
> alone matching against "sr-CS".

There is no reason why an RFC 1766/3066 parser should not recognize "sr-Latn-CS" as valid since it conforms to the syntax specified.

A language-range matcher should match "sr-Latn-CS" against a request for "sr-Latn", but not "sr-CS". That is by design since a left-prefix matching algorithm is limited in what tags it can match, and it is considered more important to match for script than for regional variations.

> Registration of a complete tag is not itself a problem.  Registration
> of a complete tag which incorporates script information is not an
> ideal solution to the issue of conveying script information; that
> would be more appropriately done using an orthogonal mechanism to
> convey the orthogonal information...

That's one opinion; there are many who hold a different opinion.

> > But speaking of selective usage, have you noticed that RFC 3454
> identifies specific characters from ISO/IEC 10646 as prohibited? Various
> space and control characters are not permitted, INVISIBLE TIMES isn't
> permitted, END OF AYAH isn't permitted, COMBINING GRAVE TONE MARK isn't
> permitted... How is what is proposed in this draft any more "cherry-
> picking" than that?
> 
> 1. RFC 3454 is not BCP, and isn't being pushed through for immediate
>    Standards status without a phased roll-in. The draft under discussion
>    has been proposed as BCP which would lack phased roll-in.

So acceptability of selective usage depends upon whether the document is a BCP or a proposed standard? I cannot see anything in RFC 2026 that suggests that (and it seems pretty odd).

> 2. RFC 3454 does not declare any parts of ISO 10646 as not valid and
>    does not call for setting up an IANA registry of code points for the
>    purpose of effectively declaring ISO 10646 code points invalid.  The
>    draft under discussion explicitly seeks to set up a registry to
>    replace use of ISO standard list.

RFC 3454 does say that some parts of ISO 10646 are not valid in strings output by stringprep implementations. This draft is analogous. If new characters are added to ISO 10646, it is certainly possible that RFC 3454 could be updated to exclude some of those new characters as well; what is proposed in this draft is analogous; the only difference is that the values considered invalid for the given purpose are documented in the IANA registry rather than in an RFC -- which is certainly the easier way to maintain things, though perhaps it's not considered the preferred means of doing this in the IETF context.

> 3. RFC 3454 does not seek to redefine the meaning of any ISO 10646 code
>    points.  The draft under discussion does, as specifically noted in
>    the case of the ISO 3166 code "CS".

This draft would not change the meaning of an ISO identifier; it simply does not use the latest assigned meaning in case a prior ISO-assigned meaning in use on the Internet exists. 

(Note: the draft itself does not entail that CS in particular should be handled one way or another, and the question of the best handling of CS to provide stability on the Internet is open to comment as a separate issue from the draft itself.)

> So, hypothetically, if some other standards body, say W3c were to declare
> that "CS" used in a language-tag in an application profile of SGML (i.e.
> not an Internet protocol) meant something other than what the draft
> under discussion would have it mean while importing the meaning of other
> language tag components w/o change, you would have no issue with such
> cherry-picking?

Well, it would be a concern, though it is their prerogative to do what they want in their specifications. Since W3C has consistently referenced RFC 1766/3066, however, this no more than a purely hypothetical question -- I have no expectation of such a thing ever happening.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Mon, 3 Jan 2005 10:38:53 -0800

> > I don't think it's that uncommon to refer to a specification A that
> makes use of another specification B as an application of B.
> 
> Perhaps, but I think it's best to avoid misunderstanding in
> technical discussion by being precise in use of terminology.

I was being precise. Note that ISO 639 uses "application of language identifiers" in exactly the same sense in which I have used "application of RFC 3066".

Peter Constable