Subject: Re: draft-housley-two-maturity-levels

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 01 Aug 2011 09:18:59 -0600

On 7/30/11 11:05 AM, Joel M. Halpern wrote:
> It seems to me that this does two things, both small but useful.
> 1) It makes a minor change in the advancement procedures so that they
> are more reasonable.  They may still not be sufficiently reasonable to
> be used, but it improves them, and thereby improves the odds.
> 2) It is coupled to an intent to actually behave according to what the
> document says.  Such an intent is obviously not feasible without some
> change.  It is useful to have our behavior and our documented
> description of how we work match because the mismatch causes confusion,
> at least for new participants, and sometimes even for experienced
> participants.
> 
> It might be the case that it will improve the advancement percentage. It
> might not.  I can not imagine that it will make that even worse.

I'm with Joel here. Let's clean up our documentation so that it more
closely matches our current practice. We might even encourage more folks
to progress their work along the standards track, although I think that
failing is more cultural than procedural.

> So, it seems to me that this matches the description that Eric, Brian,
> and others have used of a baby step that is not harmful and may be helpful.

The "baby step" description is a metaphor. If we can't make even this
small change, how can we expect to complete more significant reforms?

BTW, the latest version more clearly describes the problem it is trying
to solve, so I think that my DISCUSS has been addressed. I'm clearing
the DISCUSS and now approve of publishing this I-D as an RFC.

Peter

--
Peter Saint-Andre
https://stpeter.im/

---

From: John C Klensin <[email-masked]>
Date: Tue, 02 Aug 2011 18:51:10 -0400

On 7/30/11 11:05 AM, Joel M. Halpern wrote:
> It seems to me that this does two things, both small but
> useful. 1) It makes a minor change in the advancement
> procedures so that they are more reasonable.  They may still
> not be sufficiently reasonable to be used, but it improves
> them, and thereby improves the odds.

Actually, there is no evidence that this is an improvement.  I'd
agree that it is a minor change, but there has been no serious
analysis of whether it is an improvement or not.  To the extent
to which approving this lowers the odds of considering and
making changes that might actually have significant effects
(e.g., really sorting out what maturity levels mean in a world
of increasingly complex standards), it is harmful.  See long
discussion below.

> 2) It is coupled to an
> intent to actually behave according to what the document
> says.  Such an intent is obviously not feasible without some
> change.

Well, yes and no.  My sense of the discussion over the last year
is that a significant fraction of the community believes that
the critical path change in this area is an adjustment to the
threshold for Proposed Standard.  That issue is addressed in
draft-bradner-restore-proposed, which requires no change to RFC
2026 or other formal procedures at all.  It is not addressed in
this document.

>  It is useful to have our behavior and our documented
> description of how we work match because the mismatch causes
> confusion, at least for new participants, and sometimes even
> for experienced participants.

I agree with this.  But this document doesn't make nearly enough
of a contribution in that area to justify approving it.  It
addresses exactly one provision of our processes in which
documentation and practice don't align, a provision about which
there is no subtlety or confusion within the community at all
(even though new participants may be confused).  If the issue is
important, then let's make a serious effort to update the places
where 2026, 2028, 2031, 2360, 3710, 4071, and probably several
other documents have fallen at least somewhat out of line with
reality.  If the particular issue of the annual review is really
more important than any of those other issues, I assume that a
stand-along document that changed it would rapidly achieve
community consensus (albeit with some complaints about wasted
time).  Certainly it should not be sufficient to justify
approving this document... the change is almost irrelevant to
it. 

> It might be the case that it will improve the advancement
> percentage. It might not.  I can not imagine that it will
> make that even worse.

I can because the effects of changes like this are actually very
hard to predict.  It increases the requirements for the second
level, however slightly.  Since we have trouble getting things
to the second level now, that increased requirement might reduce
incentives to bring things off Proposed at a least as much as a
theoretical "just one more level and then you are finished"
change would increase those incentives.  By changing Proposed
from "1/3 of the way" to "1/2 way or a bit more", it might
remove a small incentive to take the additional step.

In addition, the "publication without a new RFC" provision may
actually further increase the de facto requirements for Proposed
Standards by requiring that those documents be edited to a high
standard of clear English and specification precision.  While,
IMO, we have taken too little advantage of it, the current
provisions of RFC 2026 permit publishing a Proposed Standard as
soon as the WG understands it, leaving language cleanup (and
refined translation from the writing styles of many people in
the community whether native speakers of English or not) for
Draft Standard versions.

More important, for those of us who believe that a maturity
system based on rigid named categories that apply to entire
documents has outlived its usefulness as our specifications have
become more complex, approval of this document is almost certain
to cause consideration of such approaches to be postponed by
some years as people complain that the changes made in this
draft must have time to take effect and be evaluated.

   ---

A more extended analysis of other aspects of the situation with
this document follows.  Those who don't like extended analyses
might want to stop reading at some point.

A very long time ago, a then-professor of mine observed that one
of the most common sources of failures in engineering,
especially computer engineering, was to look at a problem that
seemed too large or too intractable, find some easily-changed
and tractable part of the problem, do something with it (almost
anything), and then claim that significant progress had been
made on the original/ real problem because one had started on
it.   In many cases, the approach actually makes the real
problem harder: systems become more complex, applying remedies
later turns out to be more complicated, and so on.  The narrow
"solution" may also use up energy and creates expectations that
make it harder than it would have been otherwise to take on the
real work when the narrow job fails to solve any significant
problems.

An even more insidious version of the same approach is to
identify a problem and claim it is serious enough that one needs
to do _something_.  One then proceeds to do something that has
nothing at all to do with the problem, just to demonstrate
motion (or in the hope that setting a butterfly in motion in one
part of the world will cause major change elsewhere).

These approaches are very different from taking a large design
issue and modularizing it into pieces that can be worked out
separately but with clear and well-understood interfaces.  Doing
that right requires that the overall design issue be understood
well enough to define all of the modules and interfaces, not
doing one piece, ignoring the poorly-understood rest, and hoping
that things will sort themselves out.

The IETF periodically falls into this trap.  We often see
protocols that are designed to handle the easy issues adopted
without consideration of the harder ones and edge cases, only to
discover that those protocols represent a framework into which
solutions for the other issues won't fit.  The largest symptoms
of this used to show up only with Security issues -- protocols
being designed without security provisions with the expectation
that security  would somehow be patched in later... and despite
repeated experiences that such patches are rarely completely
satisfactory.    Now we see it spreading: I've seen "we knew
about those issues but ignored them and now have to clean up the
mess somehow" comments in two separate WGs in recent weeks and
suspect that there are many more.  Many of the comments on this
list try to justify approving this draft on the same basis.

These "there is a problem so let's do something and hope it will
help", "even though we can demonstrate no logic that would
predict that the change being proposed will result in the
desired effect", and "let's fix something to show that we can"
approaches are _always_ justified on the basis of "baby steps"
or "incremental efforts".   Those justifications would be quite
reasonable if we actually had good evidence of the linkages.  In
the absence of such demonstrated linkages, we are just thrashing
around and wasting energy --energy that, under other
circumstances, could actually be used to address the problems. 

So, while I applaud the efforts of the authors of this draft to
remove controversial and largely extraneous material, it seems
to me that the core issue remains the one that many people have
commented on in much earlier versions: At present, we don't get
many documents from Proposed Standard to Draft Standard, despite
the requirements for Draft Standard being slightly lower than
those set forth for Internet Standard in this document.  We have
a possibly-serious problem with the norms for Proposed Standard
being too high in practice --far beyond what RFC 2026 requires.
It is noteworthy that draft-housley-two-maturity-levels no
longer makes any claim to solve that problem.  On the other
hand, there has been considerable discussion in the community,
previously supported by the strongest advocates for the
predecessors of the current draft, that the norms for getting to
Proposed Standard _are_ the keystone problem.  I don't
personally believe that -- I think things are more complicated--
but I do see it as a critical element (see
draft-bradner-restore-proposed (expired today but can be
reposted if that would be useful)).

So this draft now ignores the threshold problem for Proposed
Standard and argues that "Changing the Internet Standards
Process from three maturity levels to two is intended to create
an environment where lessons from implementation and deployment
experience are used to improve specifications".  But there is no
logic to support the belief that intention will be satisfied.
The purpose of the three-level system is to create an
environment where the same lessons can be learned, applied, and
documented.  This document does nothing to change or improve
that environment unless one believes that, magically, removing
the third level will increase motivations for getting things to
draft.  The problem, which the document still describes, is that
documents don't move off the first level, not that somehow the
existence of the third one is pressing things down.

In progressing by baby steps, actual babies fall down a lot.
Usually they learn while falling and keep trying until they get
it right.  Equally important, for their purposes, the important
target is learning how to walk more reliably, not reaching a
specific goal or solving a particular problem.  The IETF lacks
those advantages: First, we are focused on the goal of making a
change and have posited that change in terms of a choice between
this particular document and doing nothing --all other
possibilities have been dismissed without significant
consideration.  To quote one of the co-authors of this draft
from a different thread:

> Herein lies the real problem:  As with many process and
> structure discussions in the IETF, folk often see only a
> simplistic, binary choice between whatever they prefer, versus
> something akin to chaos.
> 
> The world is more nuanced than that, and the choices more rich.

I agree, but this document is being treated as that sort of
binary choice.

Second, the community is easily exhausted by process debates and
the debate about this particular document has now been going on
for 13+ months.  Even if the suggestion made at the plenary
about opening 2026 were serious, does anyone actually believe
the community will be able to take up and evaluate more
significant process changes in the near future?  Or that this
document, if approved, will not be used to justify deferring
discussion of other changes because we have to see how it works
out?

So this is not "baby steps".  It is one step, a step that makes
a change that isn't demonstrably connected to the problem it
purports to solve and that leads into a dead end.

   john

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 02 Aug 2011 20:23:28 -0400

With mild apologies, I have retained John's text below because, even 
though I come to a different conclusion, I thought it important to 
retain for now.  If folks choose to follow up on this, significant 
trimming is recommended.

John, as far as I can tell there are three problems which various folks 
have wanted this document or its predecessor to address:
1) That the bar for PS is too high
2) That not enough documents are advancing up the standards track
3) That what we do and what we say we do do not match.

Clearly, these problems are related.
We could try to adopt a change that would address all three.  I dobut we 
would accomplish anything.

As far as I can tell, this document sensible tries to address one of 
these, namely item 3.  It tries to align what we document with what we 
do.  In order to do so, it also makes a change which may help item 2. 
It may not.

Now, you can argue that it is taking up energy that should go to item 1. 
  That is not unreasonable.  Except that I consider all the proposals I 
have seen for item 1 to be failures.  They fail in different ways, but 
they all have appeared to me to be bad ideas.  (I think, based on 
conversations, some folks were supporting the previous version of this 
document in the mistaken believe that it did more to address that than 
it actually provided.)

As such, we can either do nothing, or take this step that in my personal 
opinion addresses item 3, might turn out to help item 2, and does not 
hurt item 1.
I believe I understand your point below that without understanding how 
we ought to address problem 1, it is hard to be confident we are not 
making it worse rather than better.

Yours,
Joel

---

From: John C Klensin <[email-masked]>
Date: Wed, 03 Aug 2011 09:36:37 -0400

--On Tuesday, August 02, 2011 20:23 -0400 "Joel M. Halpern"
<[email-masked]> wrote:

> With mild apologies, I have retained John's text below
> because, even though I come to a different conclusion, I
> thought it important to retain for now.  If folks choose to
> follow up on this, significant trimming is recommended.

Trimmed :-)

> John, as far as I can tell there are three problems which
> various folks have wanted this document or its predecessor to
> address:

I agree with your summary so far.

> Clearly, these problems are related.

That is less clear to me.  While I think that (1) is at least
part of the cause of (2), both of the following seem almost
equally plausible:

	(i) At the time the current model was assembled, IETF
	participation was much more heavily weighted toward
	researchers and others who had the flexibility to be
	able to concentrate on getting things right with
	multiple iterations and gradual advancement.  As the
	Internet has evolved toward an environment in which
	short-term product needs dominate, a larger percentage
	of IETF participants have needs to get a reasonable spec
	agreed on so it can be implemented and deployed, but
	little need to refine and revise unless flaws in the
	specs cause serious and user- or operator- visible
	interoperability problems.
	
	(ii) At the time the current model was assembled, IETF
	specifications tended to be for relatively simple
	protocols with either few options or orthogonal ones
	without complex interactions.  Today, we are seeing far
	more specs with optional variations, options that
	interact in complex ways, profiles for different
	purposes, etc.  We have whole working groups whose
	purpose it is to sort out the operational implications
	of protocols developed in other WGs (and sometimes still
	under development.   In that environment, having
	maturity categories that require an entire complex
	protocol, regardless of profiles or sets of options, to
	be classified into a small range of mutually-exclusive
	sets just makes no sense almost independent of how many
	categories we have.   Whether we can remove obstacles or
	not, we don't have the right incentives in place to get
	people to do things that make no sense to them (or to
	their sponsors).   Indeed, _increasing_ the number of
	categories might help: for example, perhaps we need a
	"base spec and 'mandatory to implement' pieces are fine,
	but we don't know about some of the option sets"
	category.

If some combination of those hypotheses is actually the driving
force behind few specs getting past Proposed, then nothing we do
about getting things into Proposed more easily will make any
difference.  Nor is fussing with the number of maturity levels
likely to have any useful effect.

In addition, (3) appears to me almost completely unconnected
with the other two.  Sure, we don't use annual review and never
have.  And it is definitely unattractive to have rules around
that we don't follow.  I would be a lot happier if we removed or
clarified _every_ rule in our specs that we don't follow (or if
we followed more of them).   But I haven't heard even a claim
that removing a rule that we haven't followed from 2026 would
make it easier to get documents approved as Proposed or that it
would increase, even slightly, the number of documents that are
advanced.

> We could try to adopt a change that would address all three.
> I dobut we would accomplish anything.

ok

> As far as I can tell, this document sensible tries to address
> one of these, namely item 3.  It tries to align what we
> document with what we do.  In order to do so, it also makes a
> change which may help item 2. It may not.

But there is no connection at all.  If we wanted to address item
3, then all it would take is a _really_ short RFC that updated
2026 and said "that provision has never been used and is now
eliminated".  As I indicated in my long note, I can't believe
there would be any controversy about such a document other than
complaints that we were wasting time that could be better spent
in other ways.

Might it help with item 2?  While I can't _prove_ it would not,
I don't recall seeing a single argument, either in the document
drafts or on the list, that justifies even a strong hypothesis
that the existence of a rule that we have never followed has
discouraged even a single specification.   If one thought there
was an interaction of that sort, it would make far more sense
logically to try applying the rule to see what effect that would
actually have.  I'm not suggesting that, if only because I
wonder whether moving RFC 3261, RFC 2616, or RFC 2460 to
Historic on the grounds of non-advancement would make us look
the most ridiculous.

> Now, you can argue that it is taking up energy that should go
> to item 1.   That is not unreasonable.  Except that I consider
> all the proposals I have seen for item 1 to be failures.  They
> fail in different ways, but they all have appeared to me to be
> bad ideas. 

Well, that is interesting. I'd be delighted to discuss some of
those proposals with you, but I don't believe that the community
has discussed any proposals that are intended to make
improvements on item 1 in at least the last half-dozen years.
Some attempts to have such discussions have been ruled out of
order on the grounds that they address a different topic than
draft-housley-two-maturity-levels.  Ones that merely would have
required the IESG to conduct a small-scale experiment have been
ignored.  Suggestions about others have been dealt with by
fairly clear statements that the IESG would not consider them as
individual submissions and that a WG would never be approved.

That may made all of them failures, but, if it does, there are,
IMO, some much more fundamental problems with our processes
than, e.g., whether the annual review rule is being used.

But, more important, I'm not making the argument that this is
"taking up energy that should go to item 1" (even though I
believe that to be true).  I think this document should stand
(or not) on its own content and what it proposes.   But it only
solves item 3 while making much more significant and noticeable
changes --look at its title for starters-- in the area of item
2... with no real justification for believing that it will have
any positive effect on the issue you identify as item 2
whatsoever.

I also believe that, if it is approved, it will be used as a
further excuse to not take up any proposals that might actually
solve blocking problems.  But I consider that an almost-separate
issue as well because, if this would really be effective against
_either_ issue 1 or 2, that would be worth the price.

> As such, we can either do nothing, or take this step that in
> my personal opinion addresses item 3, might turn out to help
> item 2, and does not hurt item 1.

See above.  I think that making a very significant change in the
standards process -- changing the number of maturity levels and
criteria for them certainly is such a change-- requires more
justification for believing it might be effective than "might
turn out to help".  The latter looks too much like
decision-making by throwing pasta at the wall... and using only
a single strand of pasta.

In addition, while I think it unlikely, there is a rational
argument (outlined in my earlier note and notes by others) for
believing that lowering the number of maturity levels and trying
to accomplish maturity level changes without new documents will
increase the pressure to get Proposed Standards just right.
Pressure to get Proposed Standards just right appears to be one
of the primary causes of the high threshold for those documents.
There is actually a stronger argument that this might make item
1 worse than there is that it will make item 2 better.

> I believe I understand your point below that without
> understanding how we ought to address problem 1, it is hard to
> be confident we are not making it worse rather than better.

Thanks,
   john

---

From: Jari Arkko <[email-masked]>
Date: Tue, 30 Aug 2011 23:17:52 +0300

I have reviewed the discussion from the last call on this document.

My conclusion as the sponsoring AD is that we have consensus to move forward. There was clearly a constituency who believed this is a good (albeit small) step forward. A number of other people did not care so much; did not believe there was either harm or benefit. I also saw a couple of opposing opinions, though some of them were more about a desire to do something else than specific objections about this proposal. I will be recommending that the IESG approve the draft.

There were a number of smaller details raised in the discussion. But I did not see an overwhelming consensus on any specific issue to make changes. But I will ask Russ to take a look at the issue raised by Scott, whether he wants to add an informative reference to RFC 5657.

Another issue that I wanted to highlight is the question of what kinds of discusses are desirable/acceptable for documents that move from PS to IS. It is outside the scope of Russ' document, but generated nevertheless some interest. The IESG has discussed this matter and drafted some suggested guidelines. Look for a different thread on this list, under "Discuss criteria for documents that advance on the standards track". Feedback on the guidelines would be appreciated.

Jari