Subject: Re: Work effort? (Re: Proposed Standard and Perfection)

From: Keith Moore <[email-masked]>
Date: Mon, 8 Mar 2004 13:26:38 -0500

It's all well and good to try to retire Proposed Standard documents that
don't get implemented.  But I think it's even more important to make it
easier for documents that do meet the criteria to advance to Draft
Standard.  In my experience the hardest part of getting a document
advanced is to collect the implementation report.

Hence this modest proposal:

- For each standards-track document, create a web page that is used to
  keep track of bug reports, errata, implementation reports, and test
  reports.
  (yes, I know about the RFC Editor's errata page - this might be a 
  modification of that or it might be something else entirely)

- Allow implementors to submit reports via a form on that web page
  - An implementation report would name an implementation and specify
    what features it implemented
  - A test report would, for a given set of implementations, specify
    which features were tested and whether they interoperated

- Allow ADs to designate one or more people to review implementation
  reports  (to eliminate duplicates and cull out bogus reports)

- At adoption time + 2 years, every PS document would be Last Called
  for Draft Standard, for a period of 4 weeks.  This would serve as:

  - a final notice to submit implementation reports to the web site
  - a final notice to submit bug reports and errata to the web site

- At the end of the Last Call period the sheperding AD would review
  the implementation reports and bug reports and make a recommendation
  to IESG (similar to the AD writeup) to either:

  - approve document as-is
  - submit to author or WG for updates
  - recommend that the document be reclassified as historic, 
    experimental, or informational

Keith

p.s. The hardest part of this (and often, the hardest part of interop
testing) is defining exactly what tests are needed, especially when
features interact or when there are more than two parties participating
in a protocol at the same time.  Ideally each PS would specify what
implementation tests were needed to move the specification to DS, and
these would be published along with the specification.  But that will
have to wait awhile...

---

From: Graham Klyne <[email-masked]>
Date: Mon, 08 Mar 2004 20:11:53 +0000

Over the past couple of years, I've been involved in a W3C effort that 
might have some useful lessons for this discussion.

The working group concerned adopted a working practice of creating test 
cases for any significant decision that it was required to make.  One of 
the observations that underpinned this approach was that, given text 
specifying some awkward technical detail, it was usually possible for 
reasonable implementers to interpret it differently.  Not so for test cases.

The result was a process analogous to test-led software development, in 
which the specific test cases (concerning which it was far easier to 
unambiguously determine WG consensus) were used to drive the adoption of 
new or changed specification text.  A side effect of all this was that we 
ended up with a suite of test cases that could be used to judge the extent 
to which the specification was consistently implementable.

With these test cases available along with the (purported) complete 
specification, and compared with the long delay for IETF specs to move from 
proposed to draft, the process for moving from [equivalent of] Proposed to 
[equivalent of] Draft is relatively short ... typically a few months, it seems.

With this process, a substantial element of the particular "hardest part" 
that Keith notes is put together by the working group as the specification 
is being developed.

#g
--

At 13:26 08/03/04 -0500, Keith Moore wrote:
>_______________________________________________
>This message was passed through [email-masked], which 
>is a sublist of [email-masked]. Not all messages are passed. Decisions on 
>what to pass are made solely by IETF_CENSORED ML Administrator 
>([email-masked]).

------------
Graham Klyne
For email:
http://www.ninebynine.org/#Contact

---

From: Rick Stewart <[email-masked]>
Date: Mon, 8 Mar 2004 12:38:24 -0700

> Hence this modest proposal:

[clip]

I rather like the proposal. What's been lacking is any forum for further
development of standards outside of mailing lists and IETF meetings --
there's no tracking, and that's a major problem.

Rick

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 09 Mar 2004 19:48:07 -0800

> there's no tracking, and that's a major problem.

in draft-iesg-hardie-outline-01, the concept of a "maintenance team" 
(called "IANA Team" in that document) was floated. This didn't get much 
discussion. Is this something that's worth discussing as an idea?

                      Harald

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 9 Mar 2004 22:46:59 -0600

I don't KNOW that what I'm thinking is true, but I'm wondering to
myself if the target audience for protocol specification maintenance
is all in the IETF...

Spencer

----- Original Message ----- 
From: "Harald Tveit Alvestrand" <[email-masked]>
To: "Rick Stewart" <[email-masked]>; <[email-masked]>
Sent: Tuesday, March 09, 2004 9:48 PM
Subject: Re: Work effort? (Re: Proposed Standard and Perfection)

> This message was passed through [email-masked],
which is a sublist of [email-masked]. Not all messages are passed.
Decisions on what to pass are made solely by IETF_CENSORED ML
Administrator ([email-masked]).

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 10 Mar 2004 08:10:39 -0800

> myself if the target audience for protocol specification maintenance

not all the audience for protocol specification is in the IETF, so why 
should we expect the audience for protocol specification maintenance to 
be......?

the IETF works when it has *enough* of the audience, I think.
But we do have an answer for those who complain that they are not part of 
the process: "show up!".

(note - never said it was an easy answer..... but there is one....)

---

From: John C Klensin <[email-masked]>
Date: Fri, 12 Mar 2004 12:49:02 -0500

--On Monday, 08 March, 2004 13:26 -0500 Keith Moore 
<[email-masked]> wrote:

> It's all well and good to try to retire Proposed Standard
> documents that don't get implemented.  But I think it's even
> more important to make it easier for documents that do meet
> the criteria to advance to Draft Standard.  In my experience
> the hardest part of getting a document advanced is to collect
> the implementation report.

Interesting.  I would have described the hardest part as lying 
in the documentation process itself, in particular...

	* When a document has been very controversial in getting
	to Proposed, at least partially because of very loud and
	continuing objections about specific points from those
	who disagreed with them, it is often hard for authors,
	editors, [former?] WG Chairs, etc., to get up the energy
	to deal with what is likely to be a renewed version of
	the unpleasantness.
	
	* When one or more IESG members have engaged in
	extensive, time-consuming, and unpleasant nit-picking or
	bickering about provisions of the original, Proposed,
	document, authors/ editors/ WG Chairs may have the
	reasonable expectation that they would do an even
	"better" (more extensive) job on a draft coming up for
	Draft.

and

	* Documents that are, themselves, perfectly ready to
	advance get stalled indefinitely because they make
	normative references to documents or protocols that, for
	one reason or another, less ready.

The discussion below does not address the third point; I don't 
know quite what to do about it.

Note that none of these issues is intrinsically related to the 
protocol quality or deployment of the specification.  What the 
first two share with the "implementation report" issue is that 
the documentation issues are independent of the underlying 
reality.

> - For each standards-track document, create a web page that is
> used to   keep track of bug reports, errata, implementation
> reports, and test   reports.
>...

My not-very-modest proposal (many details would need filling in, 
but, as an idea...):

	(1) We decouple "change maturity levels" from "write and
	publish an RFC" (this means that the listing of maturity
	levels must be current and authoritative).  This also
	meshes nicely with another proposal that was floated a
	few years ago, but never really written up (see below)
	
	(2) When a document comes up for review for
	Proposed->Draft, we look for implementations, etc.,
	perhaps following Keith's proposal outline.  If the
	implementations are there, we issue a Last Call for
	identification of serious technical/definitional flaws
	in the document as published, where "serious" is defined
	as "problematic enough to interfere with independent
	interoperable implementations".  If none are found, we
	advance the maturity level of the document, in place --
	no new RFC publication required and hence little or no
	opportunity to reopen old wounds that lack demonstrated
	interoperability impact.  If the document is about to
	time out in grade, we issue a Last Call, wait a
	reasonable period for protests and volunteers, and then
	downgrade it in place.  The notion of having to write an
	RFC, following all of today's complex rules, and get
	formal consensus in order to declare a Protocol obsolete
	that isn't implemented and won't operate in today's
	environment is one of the more astonishing triumphs of
	bureaucracy over rationality.  Similar rules would apply
	to Draft->Full (or whatever formula newtrk comes up
	with).
	
	(3) If a document is judged defective (which is
	difficult from judging a protocol defective), we try to
	find someone to fix it.  If a plan (and volunteer(s))
	cannot be readily found, we publish a description of the
	defect(s), presumably as an RFC, but, if that is
	unworkable, in some other way.   Minimize the fuss -- if
	our customers don't think an updated document is worth
	the trouble --enough trouble to invest people, dollars
	for professional editing, or whatever else is required--
	then we should stop losing sleep over it.

Principle: If we are going to spend as much time and energy as 
we do getting something to Proposed, then moving it to Draft or 
Full should usually require only identification of 
implementations, deployment, and desirability, not extensive and 
time-consuming document rewriting

       john

--------------

Footnote -- that other proposal:  Some time ago, I noticed that 
the relationship between the STD definitions and the underlying 
RFCs had gotten a little too complicated for anyone to usefully 
understand.  By being reserved to full standards, the STDs also 
did not help with hints about things advancing along the 
standards track (e.g., a Proposed document that has fairly wide 
consensus and that is intended to ultimately replace a full 
standard is a very funny   state today).  That produced some 
informal discussion of an idea of actually turning STDs into 
documents, rather than references in an index.  A typical STD 
document, which could be issued for a Proposed Standard, under 
that proposal, would consist of

	* A title and abstract, which might differ from the
	underlying documents if there were several of them.
	
	* Any discussion of what made up the standard and why
	that was felt to be necessary.  If there is any useful
	content in, e.g., the IESG's Protocol Action notice, it
	could be included there as well.
	
	* Any short statements about applicability, maturity
	levels, recommendations (remember mandatory/
	recommended/ optional ?) that didn't justify publication
	elsewhere.  Note that this might be a place to correct
	significant errata and descriptions of deficiencies in
	the definition.
	
	* A change history, so that one could easily tell which
	RFCs were part of what standards at what times.

And, unlike RFCs, an STD document would change as standards 
evolved.

I don't know if it was a good idea, but it would mesh nicely 
with some of the other ideas that have been floating around.  I 
can write it up if there is serious interest, but have no plan 
to do so at present.

    john

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 12 Mar 2004 11:09:45 -0800

John,

I think the things you describe have very many of the same ideals and 
targets as draft-loghney-what-standards, currently being discussed in 
newtrk, which still needs work and significant input to be converted from 
an idea to a workable process - we may have a rare case of singing in 
harmony here!

                   Harald

---

From: Keith Moore <[email-masked]>
Date: Fri, 12 Mar 2004 20:19:22 -0500

John,

I cannot support the notion that the only technical flaws that should 
prevent a document from advancing from Proposed to Draft are those that 
"interfere with independent interoperable implementations".

A protocol may interoperate quite well with itself, and yet have a 
peripheral effect on a large number of other protocols. A protocol that 
does not share channel capacity fairly with TCP can have a disruptive 
effect on TCP-based protocols.  A protocol that defines a new kind of 
(IPv4 or IPv6) address space that behaves differently from existing 
address space can break applications even if that protocol does not 
directly interact with those applications.  A protocol which doesn't 
provide adequate authentication can cause its hosts to be compromised 
which in turn affects the operation and security of other services.

That, and it's too easy for technical flaws to be missed at Proposed 
Standard that show up after some implementation and/or deployment 
experience is gained - and these aren't at all limited to flaws that 
interfere with interoperation.

I do agree that we should not reopen noncontroversial documents that 
aren't found to have significant flaws.  I have occasionally suggested 
that rather than re-edit documents advancing in grade, we start out by 
making lists of things that need to be changed.  If that list ends up 
being less than N pages or X% of the length of the original 
specification, publish the list with a preface that says "RFC XXXX, 
with the following clarifications, changes, and corrections, is 
approved for Draft Standard".

OTOH, some documents that were controversial at Proposed probably do 
deserve special scrutiny at Draft.  That doesn't mean that we should 
automatically reopen them, but it might be that given the subsequent 
experience with the document it no longer meets the criteria for 
Proposed (e.g. "no known technical omissions").

> Principle: If we are going to spend as much time and energy as we do 
> getting something to Proposed, then moving it to Draft or Full should 
> usually require only identification of implementations, deployment, 
> and desirability, not extensive and time-consuming document rewriting

What if we applied the same "principle" to software?  What if once a 
program were written there were an expectation that it should never be 
substantially changed, never be re-evaluated in light of changed 
conditions or new understanding?

Keith

---

From: Donald Eastlake 3rd <[email-masked]>
Date: Sat, 13 Mar 2004 11:09:16 -0500 (EST)

Maybe I'm confused but, as I understand it, standards track level is
already, in principle, completely decoupled from "write and publish an
RFC" in that the standards level is not incorporated in the RFC
anywhere but listed separately.

In general, I agree with John Klensin as to what are considered the real
barriers to advancement, adding that the usual reason a new RFC is
required is that some small part/option of a Proposed Standards does not
have multiple implementations or if it does they don't interoperabe and
people have decided that part/option is of so little use this is not
worth fixing. Thus, under our current rules, a new RFC is needed to drop
out this more or less failed portion of the Proposed Standard.

Thanks,
Donald
======================================================================
 Donald E. Eastlake 3rd                       [email-masked]
 155 Beaver Street              +1-508-634-2066(h) +1-508-786-7554(w)
 Milford, MA 01757 USA                   [email-masked]

On Fri, 12 Mar 2004, John C Klensin wrote:

> Date: Fri, 12 Mar 2004 12:49:02 -0500
> From: John C Klensin <[email-masked]>
>
>...
>

As far a I know all of the above is true.

---

From: John C Klensin <[email-masked]>
Date: Sat, 13 Mar 2004 11:21:23 -0500

> I cannot support the notion that the only technical flaws that
> should prevent a document from advancing from Proposed to
> Draft are those that "interfere with independent interoperable
> implementations".
>
> A protocol may interoperate quite well with itself, and yet
> have a peripheral effect on a large number of other protocols.
> A protocol that does not share channel capacity fairly with
> TCP can have a disruptive effect on TCP-based protocols.  A
> protocol that defines a new kind of (IPv4 or IPv6) address
> space that behaves differently from existing address space can
> break applications even if that protocol does not directly
> interact with those applications.  A protocol which doesn't
> provide adequate authentication can cause its hosts to be
> compromised which in turn affects the operation and security
> of other services.
>...

Keith,

You are quite correct.  I oversimplified the situation 
considerably.  Clearly, any significant technical or 
interoperability difficulty that shows up should be considered 
and should be a showstopper for something going to Draft.  But 
consider:

(1) In your example above, I suggest that, with the amount to 
scrutiny now going into getting things to Proposed, the types of 
problems you identify above would be detected there and dealt 
with.  And, if they were not, we have a more serious problem 
than criteria for draft (in today's world, not an ideal one).

(2) We need to look at the system as it exists, not in how 
things might work if, e.g., everything got the perfect levels of 
review at the perfect time.  If we do that, our situation today 
is that documents go to Proposed, with whatever quality of 
(typically pre-implementation) review they get.  They get 
published.  Now, consider what might happen next, in an 
environment in which, for whatever reason (and I think both your 
observation about implementation reports and mine about 
document-fussing and general pain are part of the problem), we 
move almost nothing to Draft...

	* The Proposed Standard is implemented, causes no
	problems, turns out to be clear.  No one cares about
	moving it to Draft, so it stays at Proposed.
	
	* Flaws are discovered.  Our only real mechanism for
	documenting them is to reissue at Proposed or move the
	document to Draft (the RFC Editor's Errata are not
	IETF-authoritative and few people know where to find
	them).  But nothing goes to Draft, so the document sits
	there at Proposed, warts and all.
	
	* There are some other cases, but few result in the
	documents getting recycled, and fewer result in them
	going to Draft.

Not a good situation.  In fact, I think it is a sufficiently bad 
one that we should be looking at ways that would make things 
even slightly better in even a subset of cases than figuring 
what problems those changes would not fix in a perfect world.

> That, and it's too easy for technical flaws to be missed at
> Proposed Standard that show up after some implementation
> and/or deployment experience is gained - and these aren't at
> all limited to flaws that interfere with interoperation.

Sure.  Do you have a proposal as to how to get those flaws 
documented and the documents appropriately reissued?  If not, 
this is an "IETF is going downhill, news at 11" sort of 
observation.

> I do agree that we should not reopen noncontroversial
> documents that aren't found to have significant flaws.  I have
> occasionally suggested that rather than re-edit documents
> advancing in grade, we start out by making lists of things
> that need to be changed.  If that list ends up being less than
> N pages or X% of the length of the original specification,
> publish the list with a preface that says "RFC XXXX, with the
> following clarifications, changes, and corrections, is

We are in general agreement about this, and it is part of what 
my "turn STDs into real documents" idea was intended to address.

> OTOH, some documents that were controversial at Proposed
> probably do deserve special scrutiny at Draft.  That doesn't
> mean that we should automatically reopen them, but it might be
> that given the subsequent experience with the document it no
> longer meets the criteria for Proposed (e.g. "no known
> technical omissions").

Yes.  I was concerned about the ones about which there was no 
_real_ controversy (i.e., at the level at which a WG would have 
rough consensus on their being a controversy) but a large amount 
of noise... and no new facts arising during the interim period.

>> Principle: If we are going to spend as much time and energy
>> as we do  getting something to Proposed, then moving it to
>> Draft or Full should  usually require only identification of
>> implementations, deployment,  and desirability, not extensive
>> and time-consuming document rewriting
>
> What if we applied the same "principle" to software?  What if
> once a program were written there were an expectation that it
> should never be substantially changed, never be re-evaluated
> in light of changed conditions or new understanding?

Well, this isn't software.   And, if there is the incentive and 
energy to do more, I'm all in favor of it.   But I'm concerned 
about the cases where there isn't energy to do a lot more work 
but the implemented and deployed status of a specification (with 
or without some tuning and additional explanation) justifies its 
promotion.  If we can't solve that problem, discussions about 
retiring un-promoted standards will either turn out to be 
pointless or a mechanism for getting our decisions ignored in 
the marketplace.

Harald has pointed out that much of this is being discussed in 
newtrk.  I suggest that we move the discussion there if it is 
worth any further effort.

    john

---

From: "Conroy, Lawrence (SMTP)" <[email-masked]>
Date: Mon, 15 Mar 2004 12:40:52 +0000

On 13 Mar 2004, at 4:21 pm, John C Klensin wrote:

<snip>
> (1) In your example above, I suggest that, with the amount to scrutiny 
> now going into getting things to Proposed, the types of problems you 
> identify above would be detected there and dealt with.  And, if they 
> were not, we have a more serious problem than criteria for draft (in 
> today's world, not an ideal one).
>
> (2) We need to look at the system as it exists, not in how things 
> might work if, e.g., everything got the perfect levels of review at 
> the perfect time.  If we do that, our situation today is that 
> documents go to Proposed, with whatever quality of (typically 
> pre-implementation) review they get.  They get published.  Now, 
> consider what might happen next, in an environment in which, for 
> whatever reason (and I think both your observation about 
> implementation reports and mine about document-fussing and general 
> pain are part of the problem), we move almost nothing to Draft...
<snip>

To which I reply:

Hi John, folks,
  When I pass my driving test, the Examiner doesn't consider me to be a 
good driver; merely that
I'm unlikely to kill someone so readily.

I had thought that this was the analogy for Proposed, but this hasn't 
been my experience; protocol quality
is often considered before we get that far.

Given the amount of time we take to raise things to Proposed, *if 
people are serious about the protocol*
then there may well be implementations out there already - a number of 
protocols have had interoperation
tests whilst still I-Ds, simply due to the glacial nature of the 
process (i.e. people are overloaded)
and the requirement for quality (or "perfection", depending on who's 
saying it).

Thus the suggestion that implementation reports alone are the key to 
moves to Draft is interesting; do
we merely document the interoperation test results and (skip/fast-track 
past) Proposed altogether?

Unfortunately, describing the rationale and operation of a protocol is 
not easy, and IMHO one of the
main drivers for issuing a new RFC is to clarify things that were 
understood differently by different implementation teams - for example 
with ROHC, the first interoperation test exposed that just about
everyone interworked, but everyone had got the bit order wrong on the 
checksum as they interpreted
the text that way. Note - this isn't a mistake in the protocol, so 
putting this into the errata
collection is not appropriate. It is however a gotcha, so an 
Implementer's Guide (i.e. when it says
this, they mean that) is useful.

It's this documentation step that is needed for some poor sot who has 
to use this stuff, and it
isn't clear to me that it's the main driver for further steps "up the 
standards track".

A statement that this feature in section 12.3 and that feature in 
section 13.4 interworked between
implementation X and implementation Y may be true, but not necessarily 
complete if someone doesn't
understand what's in section 12.3 or 13.4 in the same way.

Didn't this organisation used to be about defining protocols *and* 
experience through using them?

all the best,
   Lawrence

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 15 Mar 2004 07:23:12 -0600

This really is NEWTRK material, but I think part of the question is
"how long is it SUPPOSED to take to get to Proposed?"

I happen to like the idea that you could write an I-D during one IETF
meeting cycle, talk to people about it at the meeting, rev it a couple
of times on a mailing list during the next meeting cycle, and forward
it for publication after discussion at the next meeting. That's a
six-month to eight-month interval, more or less.

People have told me I'm hopelessly optimistic. Note that my timeline
omits "update charter to include this work item", for instance.

Please followup to NEWTRK. It's been too quiet there anyway, and
that's the chartered WG for this topic...

Spencer

From: "Conroy, Lawrence (SMTP)" <[email-masked]>