Subject: Re: [newtrk] question 3 for newtrk (more)

From: John C Klensin <[email-masked]>
Date: Wed, 21 Jul 2004 19:03:20 -0400

Sorry.  I realized that I should have added something like 
"subject to all of the usual disclaimers and legal warnings" 
somewhere in the vicinity of "you can trust" below... both to 
increase accuracy and to avoid taking this thread down a 
too-familiar rathole.   But the point is that, today, someone 
can assume that the IETF community is applying its reasonable 
best efforts, or a bit more, to get things right.  If we start 
saying "you MAY use this" for things about which we have no clue 
as to whether they would actually work, and without any warning 
about that status, we are doing bad things to ourselves.

     john

--On Wednesday, July 21, 2004 10:36 PM +0200 Harald Tveit 
Alvestrand <[email-masked]> wrote:

>
>
> --On 21. juli 2004 10:15 +0200 Brian E Carpenter
> <[email-masked]> wrote:
>
>>>     * You MAY do this, but should exercise caution because
>>>     its ability to interoperate has not been demonstrated
>>>
>>>     * You MAY do this and, because of the maturity level of
>>>     the document in which it is described, you can
>>>     reasonably rely on its being safe and interoperable.
>>>
>>> The first seems ok to me.  The second is a showstopper and
>>> is, I think where we would end up if we relaxed the "show
>>> interoperability or get it out" requirement for MAY
>>> statements and features that are described without a
>>> requirement level.
>>
>> I've been hesitating to reply, partly to wait for other
>> people's opinions, and partly because I can't quite parse the
>> above. Do you mean that the if IETF purported to state the
>> second assertion it would be a showstopper, or that if the
>> IETF refused to state it would be a showstopper?
>> Also I would think that relaxing the "show interoperability
>> or get it out" requirement would lead to us making the first
>> assertion, not the second.
>
> I read John's mail as "if we make the second statement, either
> explicitly or implicitly, we HAVE to keep the requirement of
> 'show interoperability or remove it'".

Yes.  That was my intent.   If we don't do so, we take ourselves 
out of the realm of producing "interoperability specifications 
in which one can trust" (whether they are called by names like 
"standards" or not) and into the realm of publishing puff pieces 
whose content cannot be usefully relied on.

     john

.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 22 Jul 2004 11:01:12 +0200

OK, and now that your words are clear to me, it's my suggestion
that we should modify the "interoperability" requirement for DS
to specifically apply the first statement to MAY features,

>     reasonably rely on its being safe and interoperable.

      Brian

John C Klensin wrote:
> Sorry.  I realized that I should have added something like "subject to 
> all of the usual disclaimers and legal warnings" somewhere in the 
> vicinity of "you can trust" below... both to increase accuracy and to 
> avoid taking this thread down a too-familiar rathole.   But the point is 
> that, today, someone can assume that the IETF community is applying its 
> reasonable best efforts, or a bit more, to get things right.  If we 
> start saying "you MAY use this" for things about which we have no clue 
> as to whether they would actually work, and without any warning about 
> that status, we are doing bad things to ourselves.
> 
>     john
> 
> 
> --On Wednesday, July 21, 2004 10:36 PM +0200 Harald Tveit Alvestrand 
> <[email-masked]> wrote:
> 
>>
>>
> Yes.  That was my intent.   If we don't do so, we take ourselves out of 
> the realm of producing "interoperability specifications in which one can 
> trust" (whether they are called by names like "standards" or not) and 
> into the realm of publishing puff pieces whose content cannot be 
> usefully relied on.
> 
>     john
> 
> .
.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 23 Jul 2004 11:12:07 +0200

>>     reasonably rely on its being safe and interoperable.

seems strange to me - many specs have MAY features that are known to 
interoperate perfectly well..... how should we then inform people that this 
is the case?

I was facing a variant of the same problem when updating RFC 1766 (language 
tags), since there was a feature there (differences parameter to 
multipart/alternative) that had, to my knowledge, never been implemented.

One suggestion for what became RFC 3282 (content-language header) was to 
include the definition in an "informative annex", that would have a 
different status from the rest of the document. In the end, we didn't; 
instead, RFC 3282 says:

   The differences parameter to multipart/alternative is no
   longer part of this standard, because no implementations of the
   function were ever found.  Consult RFC 1766 if you need the
   information.

The tie-breaker was someone pointing out that RFC 1766 doesn't fall off the 
face of the planet even though it's obsolete....

My reading of draft-klensin-newtrk-std-repurposing is that this may be a 
place to put statements like this, without having to change the RFCs every 
time implementation status or experience with features changes.

I don't know if that's workable, but it's certainly tempting.

                     Harald

.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 23 Jul 2004 14:06:45 +0200

Harald Tveit Alvestrand wrote:
> interoperate perfectly well..... how should we then inform people that 
> this is the case?

That's exactly why I believe we should separate the status of an RFC
from statements about interoperability. There is nothing to prevent
the interoperability register also saying.

X has asserted that the following optional (MAY) features of RFC Y
have been demonstrated to interoperate between implementations A and B:

   feature 1
   feature 2
   ...

     Brian
> 
> I was facing a variant of the same problem when updating RFC 1766 
> (language tags), since there was a feature there (differences parameter 
> to multipart/alternative) that had, to my knowledge, never been 
> implemented.
> 
> The tie-breaker was someone pointing out that RFC 1766 doesn't fall off 
> the face of the planet even though it's obsolete....
> 
> place to put statements like this, without having to change the RFCs 
> every time implementation status or experience with features changes.
> 
.

---

From: John C Klensin <[email-masked]>
Date: Fri, 23 Jul 2004 21:07:29 -0400

--On Friday, July 23, 2004 11:12 AM +0200 Harald Tveit 
Alvestrand <[email-masked]> wrote:

>> OK, and now that your words are clear to me, it's my
>> suggestion that we should modify the "interoperability"
>> requirement for DS to specifically apply the first statement
>> to MAY features, and to specifically not apply the second:
>>
> seems strange to me - many specs have MAY features that are
> known to interoperate perfectly well..... how should we then
> inform people that this is the case?

Exactly.  The problem with any of these "the words always mean 
this and nothing else" solutions is that our world contains more 
variations.

> I was facing a variant of the same problem when updating RFC
> 1766 (language tags), since there was a feature there
> (differences parameter to multipart/alternative) that had, to
> my knowledge, never been implemented.
>
> One suggestion for what became RFC 3282 (content-language
> header) was to include the definition in an "informative
> annex", that would have a different status from the rest of
> the document. In the end, we didn't; instead, RFC 3282 says:
>
> The tie-breaker was someone pointing out that RFC 1766 doesn't
> fall off the face of the planet even though it's obsolete....

yes

> My reading of draft-klensin-newtrk-std-repurposing is that
> this may be a place to put statements like this, without
> having to change the RFCs every time implementation status or
> experience with features changes.
>

And that was clearly part of the intent.
      john

.

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 26 Jul 2004 13:54:07 +0200

John C Klensin wrote:
> 
> 
> --On Friday, July 23, 2004 11:12 AM +0200 Harald Tveit Alvestrand 
> Exactly.  The problem with any of these "the words always mean this and 
> nothing else" solutions is that our world contains more variations.
> 

Of my intent too, just to be clear. That's why I keep
harping on about an interoperability register, which would
be independent of the status of the RFCs concerned.

    Brian
.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 26 Jul 2004 10:40:27 -0600 (MDT)

> be independent of the status of the RFCs concerned.

Since nobody objected to the register, and many folks want it, would
it be possible to declare a rough consensus that

	Archived IETF specifications would benefit from being
	augmented with a register containing interoperability and
	compliance statements from implementors and other
	parties.  There needs to be an IETF mechanism to accept
	and manage such statements. If a multi-stage track is
	chosen, the register may be used for information on
	interoperability and implementation experience.

Thanks,

Alex.
.

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Jul 2004 14:56:11 -0400

--On Monday, 26 July, 2004 10:40 -0600 Alex Rousskov
<[email-masked]> wrote:

>> >> having to change the RFCs every time implementation status
>> >> or experience with features changes.
>> >> 
> Since nobody objected to the register, and many folks want it,
> would it be possible to declare a rough consensus that
> 

Sorry.  I must not have been sufficiently clear.  So let me try
short bullet points:

	*   I am in favor of better tools for pointing to and
	managing statements about interoperability better than
	republishing RFCs or trying to lump all interoperability
	statements into single-phrase maturity levels.
	
	* I believe that discarding the linkage of "actual
	experience with implementation, deployment, and
	interoperability" to the IETF Standards Track would be
	tantamount to discarding the IETF: anyone can write an
	unimplemented protocol description and publicize it, and
	there is a long and nasty history of various standards
	bodies and consortia doing just that.  Our claim to
	credibility is that our work is of proven
	implementability, and that we care about
	interoperability more than we care about
	often-artificial measures of conformance.
	
	* Consequently, I believe that some form of
	interoperability register would be a good idea, I think
	it should be added to the system only as part of a
	comprehensive approach to revising how we approach
	standards maturity levels and descriptions of what we
	are doing/ have done.
	draft-klensin-newtrk-std-repurposing is the beginning of
	one such approach (probably not the best one).  The
	paragraph quoted above is, IMO, not.

And you should consider that as an objection to "the register".
So, FWIW, the count of objections is up from "no one".

    john

.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 26 Jul 2004 13:16:21 -0600 (MDT)

> 	statements into single-phrase maturity levels.

The proposed "we need a register" wording includes that.

> 	tantamount to discarding the IETF

The proposed "we need a register" wording does not suggest discarding
anything (it's about creation of a register, not about deleting some
links). Whether those links currently exist in a meaningful shape and
how they should be used, is the subject of the current debate, outside
of the "we need a register" statement.

> 	standards maturity levels and descriptions of what we

Agreed. I do not think my paragraph contradicts that. It does not try
to establish consensus regarding the environment around the register;
we can debate that next, after we agree that some kind of register is
needed.

> 	paragraph quoted above is, IMO, not.

The "we need a register" is not an approach, it is just a simple
statement: this WGs agrees that some kind of register is needed (but
may disagree how that register is used). Currently, many folks post
saying that we need A and B (or we need B, assuming we have A). What I
am trying to do is to establish that A is a given, so that we can move
on and try to reach consensus regarding Bs assuming A.

> And you should consider that as an objection to "the register". So,
> FWIW, the count of objections is up from "no one".

I do not see any objection in your response. I cannot logically
interpret an "I believe that some form of interoperability register
would be a good idea" as an objection to the statement with the same
meaning! My paragraph was probably poorly worded (since it seems to
imply something it was not mean to imply). I hope this response
clarifies and helps to polish the wording to remove your objection to
"the register".

Thanks,

Alex.
.

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Jul 2004 15:32:20 -0400

Alex,

I am concerned about getting into race conditions with various
statements/ decisions from this WG.  With some possible outcomes
in terms of what we propose to do with the standards track, a
statement of support for an implementation registry would be a
great idea.  For others, it might be a poor idea, and having it
on record as WG consensus might be used to argue for blocking
those outcomes.

So I feel a need to see rather specific proposals (and that
usually means I-Ds, not comments on mailing lists), and their
context to the degree it is appropriate, before agreeing to any
statements about WG consensus.

    john

--On Monday, 26 July, 2004 13:16 -0600 Alex Rousskov
<[email-masked]> wrote:

>> > Since nobody objected to the register, and many folks want
>> > it, would it be possible to declare a rough consensus that
>> > 
>> Sorry.  I must not have been sufficiently clear.  So let me
>> try short bullet points:
>> 
> The proposed "we need a register" wording does not suggest
> discarding anything (it's about creation of a register, not
> about deleting some links). Whether those links currently
> exist in a meaningful shape and how they should be used, is
> the subject of the current debate, outside of the "we need a
> register" statement.
> 
> Agreed. I do not think my paragraph contradicts that. It does
> not try to establish consensus regarding the environment
> around the register; we can debate that next, after we agree
> that some kind of register is needed.
> 
> The "we need a register" is not an approach, it is just a
> simple statement: this WGs agrees that some kind of register
> is needed (but may disagree how that register is used).
> Currently, many folks post saying that we need A and B (or we
> need B, assuming we have A). What I am trying to do is to
> establish that A is a given, so that we can move on and try to
> reach consensus regarding Bs assuming A.
> 
>> And you should consider that as an objection to "the
>> register". So, FWIW, the count of objections is up from "no
>> one".
> 
> interpret an "I believe that some form of interoperability
> register would be a good idea" as an objection to the
> statement with the same meaning! My paragraph was probably
> poorly worded (since it seems to imply something it was not
> mean to imply). I hope this response clarifies and helps to
> polish the wording to remove your objection to "the register".
> 
> Thanks,
> 
> Alex.

.

---

From: Larry Masinter <[email-masked]>
Date: Mon, 26 Jul 2004 21:56:47 -0700

> 	interoperability and implementation experience.

Perhaps it's my fault, but this statement focuses 
on the mechanism, and not on the content. In truth,
I have no trouble finding interoperability statements; while
some of them are hard to read, in the end I've been able
to count the implementations and the features.

The problem is mainly what the interoperability statements
DON'T say.  Whether there's a register or a mechanism
or some management of the statements, if there isn't a
requirement for more accountibility in the interoperability
statements, we could have a spiffy new registry which contained
the same old (lack of) information.

I would rather focus first on the requirements for the content
of interoperability statements, and work on the registry or
mechanism for recording them later. The main criteria I'd
look for in the content is 

(a) clarity of what is being asserted; in particular:
  - which features were actually interoperable (and not
    just 'tested') against at least one other qualified
    implementations (preferably, the identities of the partner)
  - whether submitted implementations are independent implementations
  - whether submitted implementations represent independent exercise
    of any IPR license process (at least, in the cases where there has
    been notice of IPR and a licensing process, of the licensing
    process described in the notice)
(b) accountability of the individual(s) making the assertion
  - name, affiliation, current address & contact information.
  - If the person submitting the overall report is merely 
    reporting information that was relayed to them, then the name,
    affilitation, current address & contact information of
    their source.

I think these are still within the guidelines of
"self-certification".

This is within to the scope of this working group
because evidence of interoperability is the primary gating item
along the current standards track, and, as I noted in the
message that started this thread: "... if we're going to keep
interoperability statements as part of IETF process..."

Larry
-- 
http://larry.masinter.net

.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 27 Jul 2004 21:31:18 -0600 (MDT)

On Mon, 26 Jul 2004, John C Klensin wrote:

> So I feel a need to see rather specific proposals (and that usually
> means I-Ds, not comments on mailing lists), and their context to the
> degree it is appropriate, before agreeing to any statements about WG
> consensus.

On Mon, 26 Jul 2004, Larry Masinter wrote:

> I would rather focus first on the requirements for the content of
> interoperability statements, and work on the registry or mechanism
> for recording them later.

Understood. I was just hoping to make a baby step where there seems to
be consensus because we seemed to stop making any progress after
agreeing that published standards must be updated to reach acceptable
quality levels...

Thanks,

Alex.
.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 27 Jul 2004 21:46:15 -0600 (MDT)

On Mon, 26 Jul 2004, Larry Masinter wrote:

> The main criteria I'd look for in the content is
>

(c) Compliance: One of the essential design principle, IMO, is that
interoperability statements are combined with compliance statements.

Our primary goal in this context is to evaluate/estimate the quality
of a given IETF specification (or provide good info for others to
perform the evaluation). The fact that implementation A interoperates
with implementation B often says very little, if anything, about the
quality of the spec. Either A or B (or both) may not be implementing
the spec we are trying to evaluate.

There are factors that strengthen or weaken the link between
interoperability and compliance, but one cannot replace the other, and
one cannot be useful (for our purposes) without the other.

Alex.
.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 28 Jul 2004 10:19:11 +0200

>>    affilitation, current address & contact information of

Certainly we need to agree on the content, but we also need to agree
on the principle of whether or not to *separate* interoperability
information from the setps of the standard process. BTW, don't forget
that draft-loughney-what-standards-01.txt explores this topic.
> interoperability statements are combined with compliance statements.

But the IETF cannot get into the compliance business. The legal
implications are horrible. Of course, implementers can assert that
they comply, but the IETF can't cerify it.

> the spec we are trying to evaluate.

Well, the official theory is that if A, B and C have all read
the RFC, believe they comply, and do interoperate, that is a pretty
good proof that the spec is well written and unambiguous.

> one cannot be useful (for our purposes) without the other.

Disagree. Interoperability matters; compliance is a mathematical
abstraction.

    Brian
.

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 28 Jul 2004 09:42:04 -0600 (MDT)

> they comply, but the IETF can't cerify it.

Here we go again. I said "interoperability statements combined with
compliance statements". I did not say "IETF compliance certification".

We all agree that IETF cannot get into the interoperability
certification or compliance certification business. IETF cannot
certify implementation quality, regardless of the metric. IETF can
only certify the quality of its specifications, to an extent.

Now, there is a big gap between "implementers can assert that they
comply" and "IETF requires that implementers assert that they comply
before or while implementors are making assertions that they
interoperate". That's the gap I am trying to narrow here.

> > Our primary goal in this context is to evaluate/estimate the
> > quality of a given IETF specification (or provide good info for
> > others to perform the evaluation). The fact that implementation A
> > interoperates with implementation B often says very little, if
> > anything, about the quality of the spec. Either A or B (or both)
> > may not be implementing the spec we are trying to evaluate.
>
> Well, the official theory is that if A, B and C have all read the
> RFC, believe they comply, and do interoperate, that is a pretty good
> proof that the spec is well written and unambiguous.

Agreed. We need to put this theory into practice by requiring not only
interoperability statements (and assuming that implementations comply)
but requiring both interoperability and compliance statements (so that
vendors at least try to check that they comply).

> > interoperability and compliance, but one cannot replace the other,
> > and one cannot be useful (for our purposes) without the other.
>

I find it interesting that you use "mathematical abstraction" in an
apparently negative sense, but that is not important for this debate.
In my experience, true interoperability is no less abstract than
compliance, but, again, that is not a productive way of labeling
things.

Does interoperability matter? Yes, of course. How do we achieve
long-term interoperability? By writing specs that have a high chance
of producing interoperable implementations now and in the future. Do
first interoperability statements about a few selected
implementations, written just after the spec is released indicate high
spec quality? Usually not.

What happens is that first implementations both benefit from unwritten
knowledge still present in the community and from our inability to
check interoperability of all specified features. Later, when the
unwritten knowledge is forgotten and when implementations start
exercising more features, in more ways, we get interoperability
problems, pointing back to buggy specs. The spec is then replaced with
interoperability with popular implementations, with enormous amounts
of resources wasted chasing that moving target. I see it all the time
when we test real implementations and get complains/queries from real
customers.

There are many other factors that make a first set of interoperability
statements mostly irrelevant to specification quality (and IETF
currently does not collect interoperability data beyond that first
set).

If the word "compliance" is a tabu in IETF religion, I can still get
what I want using approved language, provided interoperability is
tested for all known features in all known ways to use those features.
The result will be the same as testing for compliance, of course.
Sorry if that sounds too mathematical for some, but it is true. It's
just not the approach I would recommend because it is more "natural"
for a vendor to test for "spec compliance and interoperability with
other vendors" that to test for "interoperability with all possible
spec implementations".

Alex.
.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 29 Jul 2004 09:40:49 +0200

> they comply, but the IETF can't cerify it.

I would think that the implementor's claim that he complies is a necessary 
and sufficient declaration. There are perfectly valid methods outside the 
IETF to deal with people who mislead their customers by making 
known-to-be-false statements.

Indeed, the statement from 2026 section 4.1.2: "A specification from which 
at least two independent and interoperable implementations from different 
code bases have been developed..." seems to regard the idea that an 
implementation of a standard will claim to conform to the standard as "too 
obvious to mention" - otherwise, what is it an implementation of?????

But there isn't anything too obvious to mention, is there?

                            Harald

.

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 29 Jul 2004 09:26:14 -0600 (MDT)

On Thu, 29 Jul 2004, Harald Tveit Alvestrand wrote:

> I would think that the implementor's claim that he complies is a
> necessary and sufficient declaration. There are perfectly valid
> methods outside the IETF to deal with people who mislead their
> customers by making known-to-be-false statements.

I agree.

There needs to be an IETF mechanism to cancel/remove a vendor claim if
a 3rd party can prove that the claim is false. And, if we end up with
using those claims for track stages, we have to be ready to go
backwards if claims that caused forward motion are removed.

It may also be useful to request some basis for the interoperability
and compliance claims. That is, what makes you think you are compliant
and interoperable? The answer may include, for example, a list of spec
features for which specific interoperability and compliance tests were
executed. If a vendor cannot provide this kind of information, their
claim is most likely false.

> Indeed, the statement from 2026 section 4.1.2: "A specification from
> which at least two independent and interoperable implementations
> from different code bases have been developed..." seems to regard
> the idea that an implementation of a standard will claim to conform
> to the standard as "too obvious to mention" - otherwise, what is it
> an implementation of?????

In theory, the compliance claim would indeed be an obvious assumption
or precondition!

In practice, what we often end up with is an implementation inspired
by the specification and made to interoperate with a few other
implementations on a limited set of features. It is the IETF
equivalent of "based on a true story" label used for movies. We can,
and, IMO, must do better because poor specifications cause
interoperability problems that cost the world a lot of resources
long-term. And specs are poor because the way IETF accepts
interoperability/compliance statements does not motivate vendors to
test the specs the right way.

A related issue is that many (most?) IETF specs do not even define
what it means to be compliant with them! Thus we are assuming
something that is undefined and most likely does not exist...

Alex.
.