Subject: [newtrk] question 3 for newtrk

From: [email-masked] (Scott Bradner)
Date: Fri, 9 Jul 2004 07:28:11 -0400 (EDT)

well, for what its worth I saw little consensus in the answers to my
questions to newtrk (including  a recognition that I may have
misunderstood the responses to the first question)

about the only consensus I saw was for the idea that standards get revised
and republished over time - I did not see any consensus over the number of
"named" stages the IETF standards track should have - not many people
expressed an opinion on the list so we still may see more consensus at the
session in San Diego

another question:

RFC 2026 says that:
   A Proposed Standard specification is generally stable, has resolved
   known design choices, is believed to be well-understood, has received
   significant community review, and appears to enjoy enough community
   interest to be considered valuable.

and
   A Proposed Standard should have no known technical omissions with
   respect to the requirements placed upon it.

RFC 2026 says that
   A specification from which at least two independent and interoperable
   implementations from different code bases have been developed, and
   for which sufficient successful operational experience has been
   obtained, may be elevated to the "Draft Standard" level.  For the
   purposes of this section, "interoperable" means to be functionally
   equivalent or interchangeable components of the system or process in
   which they are used.  If patented or otherwise controlled technology
   is required for implementation, the separate implementations must
   also have resulted from separate exercise of the licensing process.

and
   The requirement for at least two independent and interoperable
   implementations applies to all of the options and features of the
   specification.  In cases in which one or more options or features
   have not been demonstrated in at least two interoperable
   implementations, the specification may advance to the Draft Standard
   level only if those options or features are removed.

(I'm not including Internet Standard because its basically DS with
deployment)

independent of the number of named stages which of these requirements
should we continue to have?

a/ "resolved known design choices" 

b/  "no known technical omissions with respect to the requirements placed
   upon it"
 
c/ "at least two independent and interoperable implementations from
   different code bases"

d/ If patented or otherwise controlled technology
   is required for implementation, the separate implementations must
   also have resulted from separate exercise of the licensing process."

e/ implementation requirement "applies to all of the options and features
   of the specification" (including removal of options or features not
   meeting requirement)

Scott
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 09 Jul 2004 14:39:48 +0200

Scott Bradner wrote:
...

> a/ "resolved known design choices" 

yes

> b/  "no known technical omissions with respect to the requirements placed

yes

> c/ "at least two independent and interoperable implementations from

yes

>    also have resulted from separate exercise of the licensing process."

yes

>    of the specification" (including removal of options or features not

This is worth debating. We've certainly had cases where a MAY feature
can't demonstrate interoperability but was passionately desired by
some people. So I wonder whether e/ shouldn't be limited to MUST
and SHOULD features.

    Brian
.

---

From: Keith Moore <[email-masked]>
Date: Fri, 9 Jul 2004 12:12:20 -0400

>    of the specification" (including removal of options or features not

All of the above, with the possible exception that "e" might be relaxed
a bit.  At the least we need to develop a better understand as to what
is meant by an "option" or "feature" - i.e. what level of detail in
checking options and features best serves our purpose.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Larry Masinter <[email-masked]>
Date: Fri, 09 Jul 2004 09:02:35 -0700

> should we continue to have?

All of them (with some notes below). And whether or not these
remain part of named stages, we should have some mechanism for
formally noting when there is recognition (by the working group,
the IESG, after a 'last call') that these requirements have been met.

yes
 
> b/  "no known technical omissions with respect to the 
> requirements placed upon it"

yes, although it's often difficult to tell what constitutes
"the requirements placed upon it".

yes

>    also have resulted from separate exercise of the licensing process."

yes, except (as I recently discovered), there is apparently no
process for determining or evaluating whether this has happened,
and it's not clear that this requirement is well-understood.
So if there is an objection, the IESG disclaims any ability
to evaluate the objection.

We certainly should clarify what is meant by an "exercise of
the licensing process" or what a "licensing process" might
consist of.

> e/ implementation requirement "applies to all of the options 
> and features of the specification" (including removal of options or
> features not meeting requirement)

This is something that would benefit from better guidelines
about how to determine what might constitute a "feature" (or at least
some examples and best practices), and explicit independent
information about which features have been tested to be interoperable
with which other (independent) implementations.

I think we could gather this data in a standard form, and it
would be very useful to implementers and testers of new
implementations.

I think some amount of data gathering automation-- made available
to participants in interop testing -- would go a long way toward
helping out here.

Larry

.

---

From: David Meyer <[email-masked]>
Date: Fri, 9 Jul 2004 09:37:37 -0700

Scott,

>> a/ "resolved known design choices" 

	Yes.

>> b/  "no known technical omissions with respect to the requirements placed

	Yes.

>> c/ "at least two independent and interoperable implementations from

	Yes.

>>    also have resulted from separate exercise of the licensing process."

	Defintely

>>    of the specification" (including removal of options or features not

	Not sure on this one. Also, how does this one relate to
	RFC 2119 language (and if at all, what is appropriate use
	of such language?

	Dave
.

---

From: David Meyer <[email-masked]>
Date: Fri, 9 Jul 2004 09:41:37 -0700

Larry,

>> "the requirements placed upon it".

	I had a little trouble with this language as well, since
	(to draw a loose analogy) "no technical ommissions" isn't
	really recursively enumerable. That is, how do you know
	when you've considered all possible "technical ommissions"? 

	Dave
.

---

From: Larry Masinter <[email-masked]>
Date: Fri, 09 Jul 2004 09:54:15 -0700

> 	when you've considered all possible "technical ommissions"? 

I think the question comes down to whether the people who are
asked to make a judgment feel that they are able to make that
judgment. In this case, the question isn't whether there are
ANY technical omissions (which, as you point out, is unknowable),
but whether you know of any. If no one can point out a technical
omission in last call and you don't see any yourself, then it
passses.

In the case of "independent exercise of the license agreement",
if the IESG isn't capable of making this judgment, then perhaps
it shouldn't be part of standards track, or at least, part of
something the IESG is asked to judge.

Larry

.

---

From: David Meyer <[email-masked]>
Date: Fri, 9 Jul 2004 12:48:46 -0700

>> omission in last call and you don't see any yourself, then it

	Gotcha. The text seemed read otherwise to me, but I see
	your interpretation.

	Yes.

	Dave
.

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sat, 10 Jul 2004 07:15:58 -0500

This question did make me think...

Spencer

yes

> b/  "no known technical omissions with respect to the requirements
placed

for any stage past the first stage

> c/ "at least two independent and interoperable implementations from

for any stage past the first stage, but I'm wondering how valuable
this is

>    also have resulted from separate exercise of the licensing
process."

if there is value in two (or more) stages, this is probably most of
the value

>    of the specification" (including removal of options or features
not

this was the part that made me think. IETF old-timers talk a lot about
how important it has been, in the past, that we prune
options/features, because the alternative is a bunch of profiles that
all technically comply except for the part where they don't actually
interoperate.

The current procedure makes sense, if we actually used two or more
stages; if we were using one stage (so, please, no opinions on one
stage in general), would it make sense to prune at every
standards-track publication? If all we did was add tags that say "HTTP
PUSH has never been implemented", would that be enough?

And if you follow up to this question, please use the subject "Pruning
all standards", or some such, so we don't drive Scott insane...

Spencer

.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 13 Jul 2004 01:06:10 -0600 (MDT)

On Fri, 9 Jul 2004, Scott Bradner wrote:

> independent of the number of named stages which of these
> requirements should we continue to have?

Here is my rotten apple, destined to miss the rough consensus barrel.

As far as I can see, the value of any requirement R below can only be
judged in a "if the RFC satisfies R, then the RFC is FOO" context,
where FOO is "implementable" or some other RFC quality label. Thus,
the value of each requirement depends on the number/kind of named
stages (i.e., number/kind of different FOOs).

We seem to be trying to make independent decisions on several
variables, which is a bad idea when the variables are dependent.

> a/ "resolved known design choices"

Sounds nice, but might only be meaningful/practical if there is a list
of known and mandatory design choices to evaluate against.

> b/ "no known technical omissions with respect to the requirements
> placed upon it"

May only be meaningful/practical if there is a list of specific
requirements.

I am also not sure what "omission" means here. A technical
contradiction/bug in the spec? A missing feature that leaves some
requirements unaddressed? An extra feature that does not address any
requirements placed on the protocol?

Finally, isn't (a) a subset of (b)? That is, isn't "resolving known
design choices" a usual "requirement placed upon a spec"?

IMHO, the above two rules should be replaced with:

ab/ "technically correct"

where the notion of correctness is often spec-dependent, just like
"known design choices" or "requirements placed" are. There are some
universal prerequisites for correctness, of course. For example, a
spec that contains internal contradictions is not correct. A spec that
does not solve the problem it was supposed to solve is not correct
either. There are many other angles/areas of correctness.

> c/ "at least two independent and interoperable implementations from

No. Two is a meaningless (in isolation) magic number. Just like the
notion of "interoperability", the number of implementations that
should interoperate before we can label the spec as
good/stable/whatever is dependent on the specification. It is also
dependent on other factors like the intent of adding the above rule to
the standards track (one-stage track does not need it, two-stage track
might need a polished version of it).

For some specifications, no two implementations can test true
interoperability. For example, if a spec defines behavior of a client,
server, and proxy, then you need at least three implementations (one
of each class) just to run a meaningful test. Now add specifications
that define behavior of other specifications (e.g., Foo Core defining
how Foo extensions must be written), and the whole "two independent
and interoperable implementations" idea becomes completely
inapplicable/untestable!

Also, the notion of "independent" becomes blurry when independent
companies are employing protocol authors and/or getting advice from a
single source (like a WG mailing list). When the list becomes dormant
or more companies implement the spec without talking to anybody, "new"
problems with the spec arise.

>    of the specification" (including removal of options or features not

Depends on the definitions of "option" or "feature". Often meaningless
for micro-level definitions (e.g., those based on RFC 2119 keywords).
Its value also depends on the meaning of the track stage.

Thanks,

Alex.
.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 13 Jul 2004 01:25:28 -0600 (MDT)

On Sat, 10 Jul 2004, Spencer Dawkins wrote:

> > e/ implementation requirement "applies to all of the options and
> > features of the specification" (including removal of options or
> this was the part that made me think. IETF old-timers talk a lot
> about how important it has been, in the past, that we prune
> options/features, because the alternative is a bunch of profiles
> that all technically comply except for the part where they don't
> actually interoperate.

Are you talking about pruning unimplemented options or pruning of
options/features that cause interoperability problems? I assume the
former since the latter is just a simple indication of a technically
buggy spec (and we have a different rule for that).

> standards-track publication?

No. So-far unimplemented options with no known problems should only be
pruned if there is a major revision/rewrite of the spec. Simple
recycling to fix unrelated bugs should not cause features to be
removed. Ideally, we should only remove an option if we know that
virtually nobody is going to need it. We can only be relatively sure
of that if enough time passes by. "Every publication" may not be
"enough time"; it would be the wrong criteria.

> If all we did was add tags that say "HTTP PUSH has never been
> implemented", would that be enough?

Yes, IMO. Again, a major spec revision/rewrite may remove so-far
unimplemented features, most likely in conjunction with increasing
protocol version number, etc.

Alex.
.

---

From: John C Klensin <[email-masked]>
Date: Mon, 19 Jul 2004 17:51:58 -0400

sorry for the delay getting to this...

> 	when you've considered all possible "technical ommissions"?

Dave,

I think the keyword is "known", which is a lot different from 
proving the universal negative of "no omissions".   I don't know 
how to tune the language, or whether tuning is needed, but my 
expectation of what this means is that

* The WG or document author has applied a reasonable standard of 
diligence and care in trying to identify possible problems, 
working through the case analyses, etc.  I don't how to define 
"reasonable" in any precise way and don't want to try because 
some things are simply more important than others.  E.g., if a 
new document proposes a change to, or that might obviously 
impact, an existing and deployed protocol, I think its authors 
and the review process should be held to a much higher standard 
of analysis of possible problems and risks than something that 
does not appear to have that kind of impact.

* At whatever the first stage is, the document should not be 
required to _resolve_ any known deficiencies.  It ought to be 
sufficient to clearly identify them, letting readers --not the 
WG or the IESG-- decide whether to implement with those loose 
ends out there or to wait for a more complete version.   Again, 
a precise definition of the boundaries of what is appropriate in 
terms of problems that are described but not resolved is 
probably not either possible or desirable -- one has to look at 
risk, how the issue is presented, and whether the risks are so 
high that the document should not be published on the 
standards-track at all.

To take an example, it would be, IMO, perfectly rational to 
publish a first-stage standard (and maybe even a final state 
one) that describes, in a Security Considerations Section, the 
horrible problems that could occur if the proposed protocol is 
deployed without some (well-described) level of external/ linked 
protection.   But it is not, in general, reasonable to tie up or 
reject the document because it does not specify and include 
those mechanisms in the protocol itself -- that would defy 
everything we claim we know about modularization and layering.

--On Friday, July 09, 2004 2:39 PM +0200 Brian E Carpenter 
<[email-masked]> wrote:

>> and features of the specification" (including removal of
>>    options or features not meeting requirement)
>

Brian, here is where these questions intersect with the 
philosophy that drives some of my comments about and 
draft-klensin-newtrk-std-repurposing.  Whether it is inline or 
somewhere that we can be reasonably sure can be found, there is 
a huge difference, IMO, between statements equivalent to...

	* You MAY do this, but should exercise caution because
	its ability to interoperate has not been demonstrated
	
	* You MAY do this and, because of the maturity level of
	the document in which it is described, you can
	reasonably rely on its being safe and interoperable.

The first seems ok to me.  The second is a showstopper and is, I 
think where we would end up if we relaxed the "show 
interoperability or get it out" requirement for MAY statements 
and features that are described without a requirement level.

I believe that the most important thing we can do in all of 
these cases is to be absolutely clear about what we are doing, 
have checked, etc.  While consistency is desirable, I think we 
can, and should, give ourselves some latitude for the 
application of common sense ... but, again, the price of that is 
telling the truth, as explicitly as feasible.

      john

.

---

From: David Meyer <[email-masked]>
Date: Tue, 20 Jul 2004 00:42:09 -0700

John,

>> high that the document should not be published on the 

	Sure, and perhaps using some similar language would
	clarify the intention of the text (or minimally avoid 
	misinterpretation).  

>> everything we claim we know about modularization and layering.

	Yep. Completely reasonable.

	Dave
.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 21 Jul 2004 10:15:28 +0200

John C Klensin wrote:
...
> Brian, here is where these questions intersect with the philosophy that 
> drives some of my comments about and 
> draft-klensin-newtrk-std-repurposing.  Whether it is inline or somewhere 
> that we can be reasonably sure can be found, there is a huge difference, 
> IMO, between statements equivalent to...
> 
> The first seems ok to me.  The second is a showstopper and is, I think 
> where we would end up if we relaxed the "show interoperability or get it 
> out" requirement for MAY statements and features that are described 
> without a requirement level.

I've been hesitating to reply, partly to wait for other people's
opinions, and partly because I can't quite parse the above. Do you mean
that the if IETF purported to state the second assertion it would be a
showstopper, or that if the IETF refused to state it would be a showstopper?
Also I would think that relaxing the "show interoperability or get it out"
requirement would lead to us making the first assertion, not the second.

> 
> I believe that the most important thing we can do in all of these cases 
> is to be absolutely clear about what we are doing, have checked, etc.  
> While consistency is desirable, I think we can, and should, give 
> ourselves some latitude for the application of common sense ... but, 
> again, the price of that is telling the truth, as explicitly as feasible.

Agreed.

     Brian
.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 21 Jul 2004 22:36:53 +0200

> requirement would lead to us making the first assertion, not the second.

I read John's mail as "if we make the second statement, either explicitly 
or implicitly, we HAVE to keep the requirement of 'show interoperability or 
remove it'".

                    Harald

.

---

From: John C Klensin <[email-masked]>
Date: Wed, 21 Jul 2004 18:58:36 -0400

--On Wednesday, July 21, 2004 10:36 PM +0200 Harald Tveit 
Alvestrand <[email-masked]> wrote:

>>> The first seems ok to me.  The second is a showstopper and
>>> is, I think where we would end up if we relaxed the "show
>>> interoperability or get it out" requirement for MAY
>>> statements and features that are described without a
>>> requirement level.
>>
>> I've been hesitating to reply, partly to wait for other
>> people's opinions, and partly because I can't quite parse the
>> above. Do you mean that the if IETF purported to state the
>> second assertion it would be a showstopper, or that if the
>> IETF refused to state it would be a showstopper?
>> Also I would think that relaxing the "show interoperability
>> or get it out" requirement would lead to us making the first
>> assertion, not the second.
>
> I read John's mail as "if we make the second statement, either
> explicitly or implicitly, we HAVE to keep the requirement of
> 'show interoperability or remove it'".

Yes.  That was my intent.   If we don't do so, we take ourselves 
out of the realm of producing "interoperability specifications 
in which one can trust" (whether they are called by names like 
"standards" or not) and into the realm of publishing puff pieces 
whose content cannot be usefully relied on.

     john

.

---

From: Larry Masinter <[email-masked]>
Date: Thu, 22 Jul 2004 15:57:52 -0700

> whose content cannot be usefully relied on.

But we seem to be in a position where there is no effective
way of evaluating interoperability statements, or even
of requiring anyone to stand behind them.

I think if we're going to keep interoperability statements as
part of IETF process, we need to strengthen and normalize
the process around generating and evaluating them.

Larry
-- 
http://larry.masinter.net

.

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 22 Jul 2004 22:37:46 -0600 (MDT)

On Thu, 22 Jul 2004, Larry Masinter wrote:

> > out of the realm of producing "interoperability specifications in
> > which one can trust" (whether they are called by names like
> But we seem to be in a position where there is no effective way of
> evaluating interoperability statements, or even of requiring anyone
> to stand behind them.
>
> I think if we're going to keep interoperability statements as part
> of IETF process, we need to strengthen and normalize the process
> around generating and evaluating them.

I believe the situation is much worse. The link between
interoperability statements (necessarily covering a subset of
then-existing implementations) and interoperability among all other
implementations is very weak. An interoperability statement can only
prove that is possible to interoperate, not that all implementations
will interoperate and not even that all compliant implementations will
interoperate.

The above facts make me doubt that assigning a lot of weight/value to
interoperability statements (e.g., let them affect track stages)
and/or spending a lot of efforts on making them "stronger" is the
right direction.

Interoperability statements are often useful, but their utility
changes with their age and other context-dependent factors. Let's just
record them when we can and allow IETF users decide how/whether to use
them.

Alex.
.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 23 Jul 2004 11:14:34 +0200

> will interoperate and not even that all compliant implementations will

Remember that in the IETF process, the interoperability statement is taken 
as a test of the *specification*, not of the implementations.
Buggy implementations have many reasons for being buggy - their existence 
says very little about the specifications.

                      Harald

.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 23 Jul 2004 14:03:07 +0200

Harald Tveit Alvestrand wrote:
> Remember that in the IETF process, the interoperability statement is 
> taken as a test of the *specification*, not of the implementations.
> Buggy implementations have many reasons for being buggy - their 
> existence says very little about the specifications.

Indeed. Hence the idea of an interoperability register, which
would have carefully written content along the lines of

X has asserted that all mandatory (MUST) and strongly recommended
(SHOULD) features of RFC Y have been demonstrated to interoperate
between implementations A and B.

    Brian

.

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 23 Jul 2004 09:08:39 -0600 (MDT)

> as a test of the *specification*, not of the implementations.

Exactly. The above facts imply that the result of such a test means
little as far as specification quality is concerned, and the result
meaning changes with time and other context factors. Bad
specifications often result in some interoperable implementations,
especially short-term.

My point is that anything beyond an informative registry of
interoperability statements would be a misuse of the interoperability
concept in the context of IETF specs. For example, advancing a
specification to another track stage based on interoperability reports
is a bad idea. I tried to explain that in the paragraph following the
above facts.

Alex.
.

---

From: John C Klensin <[email-masked]>
Date: Fri, 23 Jul 2004 21:24:18 -0400

Larry,

There is an extremely long, and actually quite successful, 
history of self-certifications of both conformity and 
interoperability.  If an organization or individual makes such a 
certification, they are putting their reputation on the line 
and, in a number of environments, risking rather severe 
nastiness if they are found to be lying about it.  By  and 
large, when standards bodies have tried to evaluate, certify, or 
guarantee that an implementation conforms to a  specification in 
any way, or interoperates with other things that do, they, in 
general, have gotten themselves into big trouble.  But that 
doesn't mean one should bag it entirely.

     john

--On Thursday, July 22, 2004 3:57 PM -0700 Larry Masinter 
<[email-masked]> wrote:

>> > I read John's mail as "if we make the second statement,
>> > either explicitly or implicitly, we HAVE to keep the
>> > requirement of 'show interoperability or remove it'".
>>
>> Yes.  That was my intent.   If we don't do so, we take
>> ourselves  out of the realm of producing "interoperability
>> specifications  in which one can trust" (whether they are
>> called by names like  "standards" or not) and into the realm
>> of publishing puff pieces  whose content cannot be usefully
>> relied on.
>

.

---

From: Larry Masinter <[email-masked]>
Date: Fri, 23 Jul 2004 23:54:15 -0700

> nastiness if they are found to be lying about it. 

I have no trouble with self-certification for conformity
and interoperability. But even that doesn't seem to
be required. There is no requirement that the interoperability
statements have any particular name associated with them,
or a valid address or contact information, or that there be
any assertion of actual interoperability or independent
implementation, or any way to evaluate what might constitute
an "independent exercise of the license agreement".

Larry

.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 26 Jul 2004 10:23:21 -0600 (MDT)

On Fri, 23 Jul 2004, John C Klensin wrote:

> By and large, when standards bodies have tried to evaluate, certify,
> or guarantee that an implementation conforms to a specification in
> general, have gotten themselves into big trouble.

On Fri, 23 Jul 2004, Larry Masinter wrote:

> I have no trouble with self-certification for conformity and
> interoperability. But even that doesn't seem to be required. There
> is no requirement that the interoperability statements have any
> particular name associated with them, or a valid address or contact
> information, or that there be any assertion of actual
> interoperability or independent implementation, or any way to
> evaluate what might constitute an "independent exercise of the
> license agreement".

I agree with Larry. For some reason, whenever somebody mentions
compliance, somebody else posts a message that IETF should not be
doing compliance certification. I do not think that Larry or anybody
else here is proposing compliance or interoperability certification to
be performed by IETF.

What we need, IMO, is an IETF registry to collect compliance and
interoperability statements from implementors and 3rd parties, and
some simple but formal policy for accepting and managing those
statements.

I am not sure that discussing such a registry is within NEWTRK scope
though. It may be in scope since most (all?) folks who argue for
multi-stage track seem to assume that multiple stages are the only way
to get the interoperability message across. The registry is another
(and much better, IMO) way to disseminate interoperability and
compliance information.

Alex.
.