Subject: [newtrk] Making the standards process more efficient

From: John C Klensin <[email-masked]>
Date: Sat, 06 Aug 2005 12:01:13 -0400

Hi.  I want to separate this out from my response to Jim about
Larry's proposal because the conclusion is really separate...

It is just my opinion, but I've come to the conclusion over the
last few months that the most important single thing we could do
to improve the speed and quality of the standards process is to
learn to say, in WGs, by ADs, and by the IESG and RFC Editor:

		"It is just Proposed, identify the issue if necessary
		and let it go"

Getting documents out much more quickly would serve us well,
even if they were editorially weaker than they might be.  It
would even serve us well if they were technically weaker than
they might be if we just annotated the areas of uncertainty and
moved on, rather than requiring ourselves to resolve all of
them.  Obviously, there are limits, and we should not aspire to
produce weak documents.  But, for Proposed, we need, IMO, to
strike the balance differently than we are doing now.

Such "weaker at Proposed" documents would also provide some real
incentive --other than hoping people pay attention to labels and
assigned semantics-- for doing partial rewrites for upgrading to
the next level.  A specification that contains warning notes,
flags unresolved issues, and otherwise more nearly reflects what
we have already claimed we meant by "Proposed Standard" would be
a lot harder to justify and use as the foundation for a
production-quality system than something that appears to be
finished and polished.

    john

.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sat, 6 Aug 2005 18:42:43 -0500

Dear John,

I won't respond to the details in your note, because none of us have 
caught up on our sleep, but I believe your note is very responsive to 
the comment I made about "IETF-approved specifications" in the Newtrk 
meeting (I hope I put that in the notes - I HATE trying to take notes 
when I'm talking at the microphone!).

> "It is just Proposed, identify the issue if necessary

I don't know what the right tradeoff is, and I know that names matter, 
but what I keep trying to come up with is

- we think this specification is solid enough to start coding - it may 
very well change, but the working group won't "start over" unless the 
specification just implodes in the face of running code, and

- we think this specification is solid enough to deploy in production 
networks, and if you have bad results, we really want to know about it

Charlie Perkins and I have had a long-running discussion about 
requiring implementations for any specification entering the standards 
track at all. I've spent two years disagreeing with this, but if we 
believe in rough consensus and running code, I don't know why this 
would be WRONG...

Spencer 

.

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Aug 2005 22:06:22 -0400

--On Saturday, 06 August, 2005 18:42 -0500 Spencer Dawkins
<[email-masked]> wrote:

> Dear John,
> 
> I won't respond to the details in your note, because none of
> us have caught up on our sleep, but I believe your note is
> very responsive to the comment I made about "IETF-approved
> specifications" in the Newtrk meeting (I hope I put that in
> the notes - I HATE trying to take notes when I'm talking at
> the microphone!).
>...
 
> I don't know what the right tradeoff is, and I know that names
> matter, but what I keep trying to come up with is
> 
> - we think this specification is solid enough to start coding
> - it may very well change, but the working group won't "start
> over" unless the specification just implodes in the face of
> running code, and

FWIW, this is a first order approximation of what I have
understood to be our traditional definition of Proposed Standard.

Note that it differs from experimental because, for
experimental, we need not have confidence that the spec is solid
enough to start over (the first phase of the experiment might be
to see if the thing could be coded), nor do we need to have an
implied commitment to avoid starting over if something equally
attractive comes along.

> - we think this specification is solid enough to deploy in
> production networks, and if you have bad results, we really
> want to know about it

And this is pretty close, operationally, to our traditional
definition of Draft Standard: we know that interoperable
implementations are possible because we have seen some, _and_
implementation, testing, and some additional thought has not
turned up additional (or any significant) problems.   If you
were to use the "just Proposed, identify the issue and let it
go", then one really has to have all of the loose ends and
unresolved issues sorted out before going to Draft too.

> requiring implementations for any specification entering the
> standards track at all. I've spent two years disagreeing with
> this, but if we believe in rough consensus and running code, I
> don't know why this would be WRONG...

Several years ago, I think I would have argued for "at least one
implementation before any sort of standardization".   In today's
environment, I think it is often a realistic requirement that we
take a "we think the spec is solid enough _and_ this is the
direction we are committed to going unless it clearly cannot be
made to work" position, and call it something standard-ish
before expecting people to code.

    john

.

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sat, 6 Aug 2005 21:29:25 -0500

Ah, and so we rise from the dead.

I was typing my trip report all the way back to Dallas today and was 
really proud of how much work I was getting done when the flight 
attendant came by and said, "you were asleep when we brought the pizza 
by. Want one?"

Now, I can't WAIT to read what I apparently wrote while snoring in a 
titanium tube...

> before expecting people to code.

We need to be really clear on the right adjectives :-)

"IETF Approved Proposal" and "IETF Approved Standard"? or "Implemented 
Standard" as the second stage?

Spencer 

.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 09 Aug 2005 16:18:33 +0200

John,

Not to ignore the subsequent comments on this thread, but...

> 		"It is just Proposed, identify the issue if necessary

It's a judgement call when to let it go, though.

Obvious unaddressed security threat? Let it go?

Totally ambiguous use of MAY statements? Let it go?

Sloppy language around X- headers? Let it go?

I'd be really interested to see specific examples of
currently open DISCUSS comments that you would put
in the "let it go" class. It is undoubtedly hard to
tune this correctly.

And when we do let something go, where do we log it for
the next revision?

    Brian

.

---

From: Spencer Dawkins <[email-masked]>
Date: Tue, 09 Aug 2005 10:36:34 -0500

Hi, Brian and JohnK,

I think I've now reached the point where I've had so many onlist, 
offlist, and hallway conversations about the standards track that I can 
no longer remember who I've said what to, or why. So, attempting to get 
back onto the Newtrk (and off of the Offtrk),

It seems to me that we have a pretty fundamental question to answer - as 
much as we wished we were actually using at least a couple of the steps 
in our documented process, I'm wondering if there is any reason to 
expect things to change from current practice as long as

- the first step is called a standard (no matter what adjective is 
attached to the word "standard",

- the first step exists as an RFC, and

- the first step requires IETF Last Call/IESG approval?

I have co-authored 1-step, 2-step, and 3-step proposals, so I am OK on 
trying to solidify 2-step (or even 3-step, as long as we actually have a 
process we can and will use). But we just don't have a lot of running 
code on using subsequent steps.

Some questions:

Would it help if a Standards Review Panel approved the first step, and 
someone else approved subsequent steps? or if someone else approved the 
first step, and a Standards Review Panel approved subsequent steps?

We didn't like the WGS/SSS answer, where a working group controls 
approval for a "first step" (even if this "first step" isn't a 
standards-track document). Under the current structure, there is no 
other group that could control approval for a first step. If Area Review 
Teams were formalized and asked to approve the first step, so that it 
required an Area to go into the weeds in order to publish a lousy 
document, would this help?

Would it help if the first step was not an RFC, but the second step was? 
If we believe there is ANY benefit in the periodic evaluation described 
in 2026, having a document that actually lives for the right period (not 
six months, and not forever) seems like table stakes. This would address 
Brian's question about "logging for subsequent revisions" - if the 
document "goes away" in the formal approval sense after two or three 
years, if there is no subsequent revision, there is no problem).

Spencer

.

---

From: John C Klensin <[email-masked]>
Date: Tue, 09 Aug 2005 17:10:23 -0400

--On Tuesday, 09 August, 2005 16:18 +0200 Brian E Carpenter
<[email-masked]> wrote:

>> It is just my opinion, but I've come to the conclusion over
>> the last few months that the most important single thing we
>> could do to improve the speed and quality of the standards
>> process is to learn to say, in WGs, by ADs, and by the IESG
>> and RFC Editor:
>> 

Of course.  And there are likely real showstoppers to which the
comments below do not apply.  My suggestion is based on the
assumption that we --and, in particular, the highly-skilled
people we put on the IESG and might put on a Review Panel--
really are capable of telling a showstopper from a problem /
risk area.   If we cannot, we are in deep trouble.  FWIW, part
of my definition of a real showstopper is that rough community
consensus (at least among the fraction of the community that is
competent about the relevant technology) around that
categorization would be pretty obvious.   As a corollary, if the
IESG cannot identify an issue as a showstopper by a quick hum,
it probably isn't one.

The comments below apply to potentially-serious, but not
showstopper, members of each of the categories you suggest in
your questions.

> Obvious unaddressed security threat? Let it go?

I think you are partially missing the "identify the issue" part.
At the risk of getting too far down into the details, the
picture in my mind is of a new section of documents, permitted
in Proposed Standards (or whatever they end up being called)
only, and with a title something like "Unresolved Critical
Issues" or "Considerations Restricting Applicability of
Protocol".  That section could be supplied, unilaterally, by the
IESG or, if we went that way, the Review Panel, based either on
their own expertise or input from the community.

If the security threat is "obvious", then I would expect that
either (i) the WG would adequately discuss it in a Security
Considerations section before the document was Last Called or
(ii) the new section would be attached and would say something
like "there are concerns that there is a major security threat
and vunerability in this protocol, and that it cannot be safely
deployed on the public Internet until that threat is analyzed
and controlled".

> Totally ambiguous use of MAY statements? Let it go?

Same comment: "Review of this specification identified concerns
that the conformance statements (MAY/SHOULD/MUST and so on) are
inappropriate and unbalanced.  Input is sought [to the
[email-masked] list) on the correct balance; early
implementations should use caution about interpretation of those
statements."

> Sloppy language around X- headers? Let it go?

Maybe.  Let's add to your list "language that is obviously so
sloppy as to make independent and interoperable implementations
either impossible or completely dependent on luck" and
"obviously breaks current infrastructure and widely-deployed
protocols without properly documenting that face".   Either of
those could be, and probably would be, a showstopper, but the
threshold is higher than "sloppy language".   Sloppy language
could be dealt with some sort of "danger warning" indicating
that we aren't sure we know what the spec means and the
implementers should vigorously apply the robustness principle
and report back.

> in the "let it go" class. It is undoubtedly hard to

Undoubtedly.   But these are exactly the sorts of judgments I'd
like to see the IESG (or the review panel) exerting on final
review, with whatever input can be obtained through Last Calls
and commissioned reviews.  I don't think they were take more
time than nit-picking, second-guessing, and negotiating.
Indeed, I think they would take less time if that "Unresolved
Critical Issues" section was clearly not a consensus statement
and avoiding saying things in words that implied otherwise.  The
net result would almost certainly be to get preliminary
documents out earlier and to lower the degree to which they were
polished up to look like final specifications.

> And when we do let something go, where do we log it for

In the RFC text.  In large letters.  See above.

     john

p.s. The sort of approach I'm advocating would require some
tuning to the text in 2026.  What it says about Proposed is "has
resolved known design choices".  This approach would require
that those words be modified to permit clearly identifying
design choices and technical loose ends that have not been
resolved but that are not considered showstoppers.

.

---

From: John Leslie <[email-masked]>
Date: Tue, 9 Aug 2005 21:22:43 -0400

> - the first step requires IETF Last Call/IESG approval?

   I'd like to echo Spencer's view.

   Most of this discussion has felt a lot like re-arranging the deck
chairs on the Titanic, specifically because:

1. I don't believe there is any word we can prefix to "Standard" which
   will trump the tendency to announce victory and go home;

2. People have come to believe that RFCs "are" standards;

3. The IESG feels a long-standing obligation to "protect" the Internet
   from bad standards.

   None of these are "bad" things: they're human nature.

   Changing none of them, IMHO, dooms us to no long-lasting change.

   Changing one of them, we might see some stable change.

   Changing two of them, I think we've got a good shot.

   Of course... YMMV.

--
John Leslie <[email-masked]>
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html