Subject: RE: draft-phillips-langtags-08, process, sp ecifications, "stability", and extensions

From: "Peter Constable" <[email-masked]>
Date: Tue, 4 Jan 2005 10:28:31 -0800

> From: Dave Singer [mailto:[email-masked]]

> The whole question of what is a language, a variant or dialect of a
> language, or a suitable substitute for a language, would benefit some
> thought in any tagging scheme, though I agree the problem is not
> generally soluble.

These are questions that have been given some thought. No time to delve
into it at the moment, however.

Peter Constable

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Peter Constable" <[email-masked]>
Date: Tue, 4 Jan 2005 10:38:05 -0800

> From: [email-masked] [mailto:ietf-languages-
> [email-masked]] On Behalf Of John Cowan

> See the editor's draft of ISO 639-3 at http://tinyurl.com/6kky2 ...

I would say that all of clause 4.2 is relevant; in addition to 4.2.1, I
would especially include 4.2.2, in relation to which I have presented
ideas that led to the inclusion of the Extensions subtag in the proposed
draft. (I originally thought of it as a way to capture some existing
registered tags as part of a consistent scheme rather than merely as
ad-hoc tags, but I think it may be more generally useful as well for
dealing with some of the issues regarding different perceptions of what
is a language.) I'm afraid I don't have time at the moment to elaborate
further.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Thu, 6 Jan 2005 14:44:53 -0800

> From: [email-masked] [mailto:[email-masked]]

> I notice two main types of arguments going on in this thread, where it
seems to me
> that there is frustration
> and "talking past each other" occurring due to fundamentally different
concerns and
> assumptions between different constituencies...

I have feet in both the "implementors" and "linguistic purists" camps,
and so think I understand both. But there are many points on which I
don't agree with your assessment.

> From [the implementors'] point of view, the most important aspect of
> language tags is being able to
> parse and match them -- exact linguistic purity and accuracy is a
secondary issue.

I would say as an implementor that it's important to find appropriate
ways to match tags that meet legitimate needs in realistic scenarios in
the best way we can, and to be aware of behaviour that will be
experienced when using existing implementations, making sure that any
degredation of behaviour is known and accepted to be offset by benefits,
and that there are no really bad behaviours that may result. I would
consider exact linguistic purity secondary since this system is not
intended to document linguistic realities but to provide useful
behaviours related to differences in language usage in information
systems.

> From [the implementors'] point of view, the
> addition of new tags, regardless of whether the new tags improve
language tagging
> "accuracy", may be actively
> harmful unless accompanied by improved matching rules.

Here, I disagree, unless this statement is to be understood in a
hypothetical way -- a priori, it would be possible to make changes that
are harmful, but I do not assume that addition of new tags is
necessarily harmful.

> To the extent that the adding of tags moves beyond
> simple registration of new tags, and instead into new forms of tags
and new rules for
> interpreting tags, that is, that
> the new tags bring up fundamental matching algorithm questions, that
becomes the
> main concern for this group.

There are no new forms of tags proposed! 

The draft would impose *restraints* on the forms that tags can take, and
define precisely what forms tags could take. 

This is a point where there may be some "talking past each other". Some
people are speaking from a position in which it is assumed that the part
of a tag that refers to country can be predicted to be in the
second-subtag position. Those supporting the draft are responding that
RFC 3066 does not assume this; it only implies that the only case in
which a country code can be reliably recognized as such is when it is
the second subtag. The former assume that we should continue to keep
country codes in second position because that's the place we've been
able up to now to recognize it. The latter respond that 

- existing implementations will still be able to recognize it when its
in that place

- RFC 3066 permits it to come in other places, but existing
implementations will never be able to recognize it more than
heuristically

- that the new draft would allow new implementations to *always*
recognize it in any tag, and

- *as implementors* it is thought that requiring that country codes only
ever come in that place is *not* what will provide the best behaviours
for users (specifically in cases where script and country subtags are
both used).

> For [linguistic purists], the most important aspect of language tags
is having
> them be accurate and precise...
> Any matching issue (and in particular issues of trying to fall back to
a more "generic"
> match when an exact match is not available) are secondary.

For the linguist, what matters is the functional behaviour of the
system, including matching, but not the implementation. The linguist,
per se, has no opinion on what the internal structure of tags should
look like; they only specify what the functional requirements of the
overall system should be, and which tradeoffs in functionality are
better or worse.

But maybe I haven't got the same picture of the distinction between the
"implementor" and the "linguistic purist" that you intend. 

> A second type of argument... seems to me to be more
linguistic/political in nature, which is
> what is the "correct" (linguistically correct? politically correct?)
way to name the tags: what sort of naming scheme corresponds to
linguistic reality,

The question of what the relationship between the naming scheme an
ontologies is important inasmuch as knowing the ontology informs us of
what kinds of distinctions need to be made and kinds of relationships
may exist between those kinds of distinctions, and that guides us in
determining functional requirements, which should be the basis of
implementations. (Once again, a pointer to a white paper on these issues
from a few years ago:
http://www.sil.org/silewp/abstract.asp?ref=2002-003.) 

> or what sort of naming scheme is politically acceptable, and is there
a conflict there.
> This does get back to the
> algorithmic matching issue in a sense though, which is that if one
wants some sort of
> hierarchical structure to
> the tags (to allow easier matching),

Insofar as tags are structures as linearly-sequenced elements and that
there are matching algorithms in use that are based on
left-prefix-trunctation, there is no debate over *wanting* a
hierarchical structure: it's a reality we must live with.

> or indeed [wants to] define any sort of matching rules (as an
> implementor wants), you're
> probably getting right into some political questions about how
matching "should work".
> So for those who wanted
> to stick just to linguistic accuracy and try to avoid political
issues, trying to avoid
> discussion of algorithmic matching
> may have seemed appealing (but then provides no help to what I've
termed the
> "implementors").

This seems to assume that those promoting an ordering of script and
country subtags as found in the draft are supporting that order for
reasons of linguistic purity and have no interest in discussion of
algorithmic matching, which is completely wrong: the reason for
supporting that order of subtags has everything to do with matching
behaviour in certain widely-deployed algorithms.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Thu, 6 Jan 2005 19:36:39 -0800

> From: [email-masked] [mailto:[email-masked]]

> > RFC 3066 left us with bigger problems: it doesn't give us any
> > way to identify pieces that we would be encountering in registered
tags
> > (apart from hard-coded tables compiled from versions of the registry
> > that pre-exist a given implementation).
> 
> With, as you point out below, one important exception: It did have a
way
> to
> reliably identify a country code in most cases (but not all).

If "in most cases" means from among tags in use today under the terms of
RFC 3066 (as John Cowan would say, "what is true"), then yes. But if "in
most cases" means trom among tags permitted by RFC 3066 (as John Cowan
might say, "what is the rule") -- including some that users have been
wanting to use but have delayed using pending a revision of RFC 3066--
then no: RFC 3066 allowed for reliable identification of a country code
in only a small portion of all possible cases: only if it occurred as
the second subtag following an ISO 639 code (it does not prohibit a
country code from occurring anywhere after the first subtag).

> And this ability
> to say "2 character subtag in the second position, most be a country
code"
> was
> quite useful even though it might miss other occurences of country
codes
> in some cases.

The draft would still grant the ability to make that statement, and
would permit new implementations never to miss *any* occurences of
country codes.

> 3066bis provides a reliable way to locate country codes in all cases,
but
> the
> algorithm is different. And this is a non-backwards-compatible change.

Surely this has been the point of greatest contention in this
discussion, and is clearly not obvious, for there are several who have
repeatedly indicated that they do not see any such backwards
non-compatibility. Please, anyone claiming there would be
incompatibility, be pedantic: define whatever terms, make explicit
whatever assumptions are required to support this claim. (I suspect the
root of this disagreement lies in unstated assumptions.) 

Those who claim backward compatibility do so on the basis that every
existing implementation conformant to RFC 3066 will continue to operate
precisely as designed and in conformance with RFC 3066 regardless
whether they encounter a tag presently well-formed and valid under the
terms of RFC 3066 or one that would be sanctioned by this draft. If
there is any term needing clarification in that statement or any
suspected assumption not made plain, please ask for clarification.

> Of course there's the option Dave Singer has raised: Reverse the
positions
> of
> script and country codes in 3066bis. I see two problems with this:
> 
> (1) Script codes are in general more important than country codes, and
>     therefore really should come first so that simple truncation
matches
>     work "better". (There are probably exceptions to this assertion
> lurking
>     out there somewhere, but I believe it is mostly true.)

Thank you for voicing support for this position.

> (2) I believe it increases the number of grandfathered codes that
won't
> conform
>     to the new format.

If I'm not mistaken, I think there would be no difference in this
regard.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Thu, 6 Jan 2005 21:02:54 -0800

> [email-masked]] On Behalf Of John C Klensin

> > This reflects a fundamental misunderstanding of what the draft
> > does compared to what RFC 3066 does.  It imposes *more*
> > restraints on language tags, not fewer.
> 
> It also very explicitly permits talking about scripts, not just
> languages and countries.    That, to me, is an extension,
> regardless of the additional constraints.

There may be a disagreement here due to a difference of perspective: one
could say that the grammar is more extensive, but that makes the formal
language less extensive. So, I suppose whether one considers such a
revision "an extension" depends on one's perspective. 

Note that while the draft permits "talking about" scripts, RFC 3066
permits "talking about" *anything*. More extensive grammar, less
extensive language (and vice versa). 

> And, as Ned as
> pointed out repeatedly, there are things that can be done in
> 3066 parsers/interpreters in practice that have to be done
> differently in this new system.

I think this claim can only be made on the basis of assumptions not
found in RFC 3066. Ned has most recently said, 

"3066bis provides a reliable way to locate country codes in all cases,
but the algorithm is different. And this is a non-backwards-compatible
change."

The fact that it can identify country codes in all cases but requires a
different algorithm does not imply a non-backwards-compatible change
since it is a new functionality -- it is doing something that wasn't
even possible in RFC 3066. 

Backwards compatibility cannot be measured in terms of whether new
processors require different algorithms to achieve new functionality. It
can only be measured in terms of whether new processors can perform
correct operations (correct according to the specification for those
processors -- the proposed draft) on existing tags, and whether existing
processors can perform correct operations (correct according to the
specification of those processors -- RFC 3066) on new tags. This draft
permits this.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Thu, 6 Jan 2005 21:13:31 -0800

John:

> Peter, just to clarify... In my opinion (which isn't necessarily
> worth much)

(I sincerely doubt that's the case.)

>, the procedures that were followed were perfectly
> reasonable.   Anyone can form a design team and put a document
> together, and there are no rules that bar such a design team
> from using and building on a mailing list set up for something
> else.  That may or may not be wise, but it is certainly
> permitted.  The only place this runs into a problem is if
> someone presumes that a document developed in the way this one
> was developed is equivalent to a WG product, or that it is
> entitled to the presumptions of relevancy and correctness that
> go with a WG product.

I can't speak for the authors. I was not familiar with those
distinctions when the process began, and I suspect that is true of
others on the IETF-languages list who contributed. In my mind we were
following a precident that implied not only a permitted procedure but an
entirely appropriate one. I think all of us now understand, at least in
part, that some distinctions exist that may have practical implications
on how something is received by the IETF community and processed by the
IESG.

> From that point of view, it is nothing
> more or less than an individual submission (or the output of a
> self-defined design team) and the comments Dave and I have been
> making apply.

I don't think I have questioned the applicability of your comments in
this regard at any point.

Peter Constable