Subject: Re: Last Call: <draft-housley-two-maturity-levels-08.txt> (Reducing the Standards Track to Two Maturity Levels) to BCP

From: Russ Housley <[email-masked]>
Date: Wed, 3 Aug 2011 15:28:28 -0400

SM:

> From Section 2.1:
> 
> "no existing published requirements are relaxed".
> 
> Are these published requirements BCPs?

Yes.

> From Section 2.2:
> 
>  'This maturity level is a merger of Draft Standard and Standard as
>   specified in RFC 2026 [1].  The chosen name avoids confusion between
>   "Draft Standard" and "Internet-Draft".'
> 
> Shouldn't that be "Internet Standard" instead of "Standard"?

Look at RFC 2026.  The three level defined there are Proposed Standard, Draft Standard, and Standard.

The draft-housley-two-maturity-levels proposes two levels: Proposed Standard and Internet Standard.

>   "The request for reclassification is sent to the IESG along with an
>    explanation of how the criteria have been met.
>    The criteria are:
> 
>   (1) There are at least two independent interoperating implementations
>       with widespread deployment and successful operational experience.
> 
>   (2) There are no errata against the specification that would cause a
>       new implementation to fail to interoperate with deployed ones.
> 
>   (3) There are no unused features in the specification that greatly
>       increase implementation complexity."
> 
> The document that has been the subject of innumerable messages highlights how difficult it can be to reclassify a RFC.  Moreover, it amplified the divide between application folks and operators.  The IESG could have used the review clause from RFC 2026 and:
> 
>  (i)  requested an implementation report from the people in favor of the
>       proposed standard; and
> 
>  (ii) a statement about deployment to determine whether there are operational
>       issues that have to be addressed.
> 
> I don't know whether application folks and operators agree that cross-area requires mutual understanding.
> 
> The creation of interoperable protocol implementations requires clear specifications.  Interoperability does not mean that the protocol would not have unintended effects on the network.  That is where operational experience comes in.  It can serve as valuable input to improve a specification.  For what it is worth, there approximately 75 implementation reports have been submitted since 1996.

I am well aware of the implementation reports.  The premise here is that the protocol specification is "good enough" there are at least two interoperable implementations and the protocol is deployed widely.  The implementation report would become optional.

> A two-step maturity level folds the two different classes of issues into one.  Quoting RFC 5657, which this draft unfortunately does not reference:
> 
>  "Moving documents along the standards track can be an important signal
>   to the user and implementor communities, and the process of
>   submitting a standard for advancement can help improve that standard
>   or the quality of implementations that participate."
> 
> During a discussion on another mailing list, it has been mentioned that such an effort has a cost.  Lumping all the issues together can only increase that cost.

People are not doing many implementation reports.  As you say above, there are only about 75 of them.  How many protocols are documented in RFCs?  That is a very low percentage in my view.

So, I see the cost quite differently.  Most protocols are published as Proposed Standards, and they are never advanced.  I'm seeking a process where implementation and deployment experience actually improves the protocol specifications.  Today, that rarely happens, and when it does, the document recycles at Proposed Standard and the reader does not know whether it happened or not.

> Strangely, this draft argues for measuring "interoperability through widespread deployment of multiple implementations from different code bases".  It will be more difficult to remove features once implementations are widely deployed.  Keeping the feature fight within the Draft Standard  discussion can reduce the level of controversy at the last step.  As a side note, it would be less costly if feature evaluation was based on implementations instead of what people would like to keep (only one implementation of a feature).

This is an argument for the status quo.  We have decades of experience with that not working.  That is essentially an argument for a single maturity level; that is how the process is really working today.

> Once a Proposed Standard is published, it is expected that people will go and write code, if they have not done so yet, and evaluate whether their implementation can interoperate with other implementations.  I don't see anything in this draft that encourages that.

Of course that is what we want.  Even better if the running code is used to make decisions before the protocol become a Proposed Standard.

> In the RFC 5000 range, there are 7 Internet Standards, 13 Draft Standards and 537 Proposed Standards.  One of the arguments for this draft is that it reduces the number of IESG evaluations, and other review cycles, from three to two.  Basically, this draft is to adjust the environment so that the IESG can review everything.  It does not reduce the barrier of "intended proposed standards" to the RFC 2026 level.  It does not offer any incentive to advance document along the Standards Track.

This document is not about IESG review time, except for the elimination of the requirement for annual reviews which are not done anyway.  If that is what you get from the document, then I have done a very poor job in writing it.  That is not the point at all.

Russ

---

From: Hector Santos <[email-masked]>
Date: Wed, 03 Aug 2011 17:19:48 -0400

I appreciate this exchange here. I have a better idea of the draft and 
your intention  I have a few comments.

What I have noticed of late are fast track RFCs are coming out of no 
where, very fast and sometimes are indirectly related to a WG but not 
a WG chartered work item, and it may have an unforeseen influence in 
the WG end results.  While not all WGs are the same, this is what I 
experienced in the DKIM WG.  The problem comes when there is little or 
isolating vetting to this external work.

For the most part, once work gets an RFC or even just an published 
I-D, it also comes with a complex to be labeled as a "standard" to 
"throw the book" at others; "you are not following the standard." 
While they are technically wrong, it happens and it puts others in 
defensive position. But it also shows why there are derivative work or 
why an RFC does not get followed 100%.

I guess, if anything, if we are going to allow for faster maturity, we 
probably need some guidelines (if not already in place) in how non-WG 
RFC productions could influence a current WG.

-- 
Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Russ Housley wrote:
> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
> 
>

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 04 Aug 2011 09:45:36 +1200

Hector,

On 2011-08-04 09:19, Hector Santos wrote:
> where, very fast and sometimes are indirectly related to a WG but not a
> WG chartered work item, and it may have an unforeseen influence in the
> WG end results.  While not all WGs are the same, this is what I

Can you be more specific? Are you talking about

a) drafts that appear in the WG with very mature text, so complete
the WG progress very quickly?

b) drafts that are direct submissions to the IESG, and go through
IETF Last Call and IESG review without coming near the WG?

c) drafts that are Independent Submissions to the RFC Editor,
so are only subject to minimal review by the IESG and are not
IETF documents at all?

And in case a) or b), are you talking about standards track and BCP
documents, or about informational or experimental ones?

> For the most part, once work gets an RFC or even just an published I-D,
> it also comes with a complex to be labeled as a "standard" to "throw the
> book" at others; "you are not following the standard." While they are
> technically wrong, it happens and it puts others in defensive position.
> But it also shows why there are derivative work or why an RFC does not
> get followed 100%.
> 

I don't understand that without understanding what kind of document
you are concerned about.

   Brian

---

From: Hector Santos <[email-masked]>
Date: Wed, 03 Aug 2011 22:35:01 -0400

> documents, or about informational or experimental ones?

Hi Brian,

 From what I have experienced, it doesn't matter what the fast track 
non-WG RFC status is - any status can have a surprising WG goal 
changing influence.

I don't really see that as the main concern until it is a competing or 
conflictive externally produced RFC, especially if it was originally 
deem out of scope in a WG charter. IMO, in such a case, it should be 
part of a WG charter change and a WG vetting process before it is made 
into an RFC and/or allowed to change the WG goals.

I am not an expert with IETF procedures and I don't wish to call that 
a "Loop Hole" but it appears to be one.  It seems to me a two-maturity 
level may inadvertently create more of these WG conflicts without the 
proper due diligence to watch for these type of situations.

Maybe the experience in the DKIM WG was an exception and not the rule 
typical of a WG, but if was a new reflection of an "IETF change" 
making it more possible to fast track RFCs, the review process needs 
to take into account the consequences it can have in any related WG. 
Is that a time problem? A WG Chair/AD issue?  Too much work for the AD 
reviews?  Not enough synergism of among all parties?  I can't say and 
quite maybe it was just an exceptional experience and not the norm. 
But I believe a watchdog for these type of possibilities will help.

---

From: SM <[email-masked]>
Date: Fri, 05 Aug 2011 02:45:10 -0700

Hi Russ,
At 12:28 PM 8/3/2011, Russ Housley wrote:
>I am well aware of the implementation reports.  The premise here is 
>that the protocol specification is "good enough" there are at least 
>two interoperable implementations and the protocol is deployed 
>widely.  The implementation report would become optional.

One of the advantages of an implementation report is that it provides 
a statement about interoperability between two or more known 
implementations.  If there is any dispute about that claim, it can be 
resolved in a non-controversial way.  Determining whether a protocol 
is widely deployed is not always a clear-cut decision.

>People are not doing many implementation reports.  As you say above, 
>there are only about 75 of them.  How many protocols are documented 
>in RFCs?  That is a very low percentage in my view.

Yes, it's a very low percentage.  I don't have the figure for the 
number of protocols documented.  Given the low barrier for such 
reports, I would have expected to see more reports.  After all, if 
the RFC has been published, the protocol has been widely deployed, it 
should simply have been a matter of filing the short report.

 From draft-housley-two-maturity-levels-08:

    "this document measures interoperability through widespread deployment
     of multiple implementations from different code bases, thus condensing
     the two separate metrics into one."

This change is expected to solve the problem.  I am not convinced 
that the metrics is the problem.

>So, I see the cost quite differently.  Most protocols are published 
>as Proposed Standards, and they are never advanced.  I'm seeking a 
>process where implementation and deployment experience actually 
>improves the protocol specifications.  Today, that rarely happens, 
>and when it does, the

Agreed.

I didn't find any incentive to inject implementation and deployment 
experience into the process.

>This is an argument for the status quo.  We have decades of 
>experience with that not working.  That is essentially an argument 
>for a single maturity level; that is how the process is really working today.

I am not arguing for a single maturity level (the status quo).  I do 
not agree with the conclusion that the decades of stagnation is due 
to the three maturity level.

>This document is not about IESG review time, except for the 
>elimination of the requirement for annual reviews which are not done 
>anyway.  If that is what you get from the document, then I have done 
>a very poor job in writing it.  That is not the point at all.

I don't think that you did a poor job.  A three maturity level 
requires three IESG Evaluations.  A two maturity level requires two 
IESG Evaluations.  If more documents moved from Proposed Standard to 
the next level, it would obviously take more IESG review time.

I presume that the IESG will only use the following criteria for advancement:

  - two independent interoperating implementations with widespread
    deployment and successful operational experience

  - no errata against the specification

  - no unused features in the specification

And there won't be any DISCUSSes along the lines of:

   "I don't think the implementation reports are adequate for me to meet the
    requirements of 2026. It does not clearly identify what software 
was used or
    show support of each of the individual options and features."

   "Examples througout the document make use of non-example domains."

   "The implementation report is woefully inadequate to document there are
    interoperable implementations of all the features from two different
    code bases."

   "My Discuss was not addressed at all - I believe that the WG ignored the
    spirit of the implementation report requirement - my Discuss said that
    we should know that there are multiple implementations that have
    handled the significant changes in the recycling of this Draft Standard.
    The group apparently refused to update its implementation report"

Regards,
-sm

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 06 Aug 2011 08:55:45 +1200

Hector,

On 2011-08-04 14:35, Hector Santos wrote:
> non-WG RFC status is - any status can have a surprising WG goal changing
> influence.
> 

I'm still not getting your point. External events of all kinds can
affect a WG; for example publication of a much better technique in
an academic journal, or publication of a patent that is essential
to the chosen solution. Are you saying that the existing review process
for direct submission or Independent Submission RFCs fails to detect
work that overlaps with WGs?

> I am not an expert with IETF procedures and I don't wish to call that a
> "Loop Hole" but it appears to be one.  It seems to me a two-maturity

How? It only concerns documents that have already gone through the
whole process of becoming Proposed Standard; essentially the changes
beyond that point in the standards track should always be quite minor.

> typical of a WG, but if was a new reflection of an "IETF change" making
> it more possible to fast track RFCs, the review process needs to take
> into account the consequences it can have in any related WG. Is that a
> time problem? A WG Chair/AD issue?  Too much work for the AD reviews? 
> Not enough synergism of among all parties?  I can't say and quite maybe
> it was just an exceptional experience and not the norm. But I believe a
> watchdog for these type of possibilities will help.

Which is exactly why we do have a review process for non-WG RFCs.

     Brian

---

From: Hector Santos <[email-masked]>
Date: Fri, 05 Aug 2011 22:52:44 -0400

Brian E Carpenter wrote:
> Are you saying that the existing review process

At least in one experience, I would not say it was a failure per se 
but more realistically, for many possible reasons, it simply fell 
through the cracks.  If interested, I can provide offlist summary 
outline example.

> Which is exactly why we do have a review process for non-WG RFCs.

Brian, no doubt. The IETF Review process and the procedures are in 
place for anyone to raise concerns and even appeal.

So in hindsight, it probably unrealistic to expect any IETF reviewer 
to catch consequential conflicts and in this case, I should of 
continued the appeal process as I had contemplated but several reasons 
I decided not to follow through.  Only then can concerns be raised for 
proper IETF review.

Maybe to help alleviate possible concerns with fast-track RFCs and 
perhaps reduce IETF reviewer time and provide some guidance, a short 
pre-review questionnaire asking relevant questions designed to 
highlight any possible conflict as known by the author and something 
the Reviewers might wish to know about, might be appropriate.

Example questions might be:

    1) Is this RFC directly or indirectly related to one or more existing
       IETF Working Groups?  If so, please list them below and continue
       with the additional questions.

    2) Is there any concept in your RFC which is deemed out of scope
       in the related Working Group? If so, which state the out of
       scope concept(s) in itemized format.

    3) Is there any concept in your RFC which competes and/or conflicts
       with the chartered goals of the related Working Group?

    4) Was this RFC vetted by the Working Group? If so, what was the
       conclusion?

Please note: In no way am I stating that these type of non-WG RFC work 
should not be written, put to a stop or not accepted by the IETF 
reviewers.  But with awareness, there be a reviewer problem solving 
suggestion to remove conflicts. Maybe it just a matter of 
word-smithing to make corrections. Again, if you wish, I can give you 
an example off-list to see why questions like these can help.

---

From: Bob Hinden <[email-masked]>
Date: Sat, 6 Aug 2011 07:15:25 -0700

Dave,

On Fri, Jul 29, 2011 at 9:53 AM, Dave CROCKER <[email-masked]> wrote:
>
>
> On 7/29/2011 11:13 AM, Russ Housley wrote:
>>>>
>>>> (2) At any time after two years from the approval of this document as a
>>>> BCP, the IESG may choose to reclassify any Draft Standard document as
>>>> Proposed Standard.
>>>
>>> I think this is unfair to the people who have done considerable work to
>>> get
>>> a document to Draft Standard.  I hope that the IESG would only do this
>>> after giving a lot of notice to the authors, appropriate working groups,
>>> and the IETF community to give them the opportunity to request
>>> advancement
>>> to Internet Standard.
>>
>> This was added after the discussion that Draft Standards could linger for
>> a
>> very long time.  Some people said that would not be a problem, and other
>> people said it would be harmful.  I conclude that no one knows, so we
>> should
>> include the powers necessary to resolve the problems if they emerge.  If
>> they
>> do not emerge, there is no requirement that the IESG do anything.
>
> If a document no longer has anyone watching it, there's a reasonable concern
> that it no longer has much constituency.  In that case, it's better to treat
> it as immature rather than mature.

In order to have reached Draft Standard, the w.g. must have shown
multiple implementations.  In at least one case I can think about the
protocol is very widely used operationally.  The working group is
disbanded, only the mailing list remains.  The protocol is very mature
and stable.  It's only the IETF activity that is inactive.

You can't measure maturity of a protocol at draft standard by looking
at the IETF activity.

Bob

>
>
>>> I think this Section of the document needs to provide additional detail
>>> on
>>> how this should work.
>>
>> I do not think we should add speculation about the potential problems to
>> this
>> document.
>
> +1
>
> d/
> --
>
>  Dave Crocker
>  Brandenburg InternetWorking
>  bbiw.net
>

---

From: John C Klensin <[email-masked]>
Date: Sun, 07 Aug 2011 12:31:33 -0400

--On Saturday, August 06, 2011 07:15 -0700 Bob Hinden
<[email-masked]> wrote:

>> If a document no longer has anyone watching it, there's a
>> reasonable concern that it no longer has much constituency.
>>  In that case, it's better to treat it as immature rather
>> than mature.
> 
> In order to have reached Draft Standard, the w.g. must have
> shown multiple implementations.  In at least one case I can
> think about the protocol is very widely used operationally.
> The working group is disbanded, only the mailing list remains.
> The protocol is very mature and stable.  It's only the IETF
> activity that is inactive.

Moreover, if the community that produced the Draft Standard came
to the IETF for the purpose of working on the particular
specification (and, presumably, its predecessors), decided they
were finished, and went away, perhaps after seeing it widely
deployed, there may be an even stronger argument for moving it
to Internet Standard than would be the case if there are still
people around and messing with it.
> 
> You can't measure maturity of a protocol at draft standard by
> looking at the IETF activity.

Right.

   john

---

From: "GT RAMIREZ, Medel G." <[email-masked]>
Date: Mon, 8 Aug 2011 08:57:38 +0800

Hi SM,
Pardon me;
1)Is there such thing as a "good enough" Criteria to handle this
concern?
2)Or as usual it passes "rough consensus" process?

Regards,
Medel
++++++++++++++++++++++++++++++++++++++++

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
SM
Sent: Friday, August 05, 2011 5:45 PM
To: Russ Housley
Cc: [email-masked]
Subject: Re: Last Call:
<draft-housley-two-maturity-levels-08.txt>(Reducing the Standards Track
to Two Maturity Levels) to BCP

Hi Russ,

    "this document measures interoperability through widespread
deployment
     of multiple implementations from different code bases, thus
condensing
     the two separate metrics into one."

Agreed.

I presume that the IESG will only use the following criteria for
advancement:

   "I don't think the implementation reports are adequate for me to meet
the
    requirements of 2026. It does not clearly identify what software 
was used or
    show support of each of the individual options and features."

   "The implementation report is woefully inadequate to document there
are
    interoperable implementations of all the features from two different
    code bases."

   "My Discuss was not addressed at all - I believe that the WG ignored
the
    spirit of the implementation report requirement - my Discuss said
that
    we should know that there are multiple implementations that have
    handled the significant changes in the recycling of this Draft
Standard.
    The group apparently refused to update its implementation report"

Regards,
-sm 

This e-mail message (including attachments, if any) is intended for the use of the individual or the entity to whom it is addressed and may contain information that is privileged, proprietary, confidential and exempt from disclosure. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender and delete this E-mail message immediately.

---

From: Hector Santos <[email-masked]>
Date: Sun, 07 Aug 2011 21:22:30 -0400

SM wrote:

>> there are only about 75 of them.  How many protocols are documented in 
>> RFCs?  That is a very low percentage in my view.
> 
> Yes, it's a very low percentage.  I don't have the figure for the number 
> of protocols documented.  Given the low barrier for such reports, I 
> would have expected to see more reports.  After all, if the RFC has been 
> published, the protocol has been widely deployed, it should simply have 
> been a matter of filing the short report.

Of course, when implementation reports are written, one has to 
watchful for the summarized analytical results that either attempt to 
add weight to an desired goal or mask the undesired goal and natural 
result.

In one such report, I was able to show the data collection indicated 
an undesired result was indeed the natural and dominant implementation 
mode.  The 2nd revision tweaked the semantics enough to hide the 
higher weight of the data collected for the undesirable dominant mode. 
  When that change was also seen and noted, it was added back in the 
3rd revision as a "oops" exclusion.

This is not an exercise we should have to go through.  Engineers must 
have complete faith in implementation reports.

---

From: John C Klensin <[email-masked]>
Date: Sun, 07 Aug 2011 12:45:18 -0400

To add one observation to SM's comment and other observations
that the scarcity of implementation reports implies that they
are somehow difficult...

--On Friday, August 05, 2011 02:45 -0700 SM <[email-masked]>
wrote:
 
>     requirements of 2026. It does not clearly identify what
> software was used or
>     interoperable implementations of all the features from two
> different code bases."
> 

An alternate hypothesis about the low numbers of implementation
reports on file is that people try them, get responses like the
ones SM lists above, and react with "why bother -- this is a
waste of time that I can better spend in other ways".  And that
is the end, not just of the report being commented on, but any
others that could have come from that author or participant.  

While it has not been raised as a specific argument for this
two-maturity-levels proposal, getting rid of formal
implementation reports might, for that reason, be useful in
getting documents advanced.  However, it does not improve the
case for two levels instead of three and won't help if IESG
members express the same thoughts --not in terms of
interoperability reports but in terms of lack of conviction the
interoperability had actually been demonstrated in the case of
that particular spec.  

I note that, when RFC 4846 and what became 5742 were under
discussion, a then-IESG member pointed out to me that, despite
the restrictions in those documents, if something in a draft
really offended him, he could always find a way to state his
objections in a way that would conform to the 4846/5742 rules.
The same comment presumably applies to presumed proofs of
independent interoperable implementations.

   john

---

From: SM <[email-masked]>
Date: Mon, 08 Aug 2011 15:33:39 -0700

Hi Medel,
>1)Is there such thing as a "good enough" Criteria to handle this

Glen Zorn asked a question during the last plenary and there was a 
discussion [1] about "criteria" on this mailing list.

To be fair, I'll quote a message from Warren Kumari [2]:

   "While not all ADs read all drafts, most read a large fraction
    of them (and read them carefully and thoughtfully enough to
    catch a number of large issues (and nits) *that were not caught
    in LC*) -- I think that they deserve recognition for performing
    a valuable and un-fun function..."

>2)Or as usual it passes "rough consensus" process?

The process is based on consensus and not "rough consensus".

Regards,
-sm

P.S. http://www.ietf.org/mail-archive/web/ietf/current/msg67494.html

1. http://www.ietf.org/mail-archive/web/ietf/current/msg68008.html
2. http://www.ietf.org/mail-archive/web/ietf/current/msg68008.html

---

From: SM <[email-masked]>
Date: Mon, 08 Aug 2011 16:06:24 -0700

Hi Hector,
At 18:22 07-08-2011, Hector Santos wrote:
>watchful for the summarized analytical results that either attempt 
>to add weight to an desired goal or mask the undesired goal and natural result.

An implementation report is written to further a desired goal, i.e. 
advancement of a specification along the Standards Track.

>This is not an exercise we should have to go through.  Engineers 
>must have complete faith in implementation reports.

Faith-based engineering and reality are mutually exclusive. :-)

Regards,
-sm

---

From: Hector Santos <[email-masked]>
Date: Mon, 08 Aug 2011 21:21:35 -0400

> Faith-based engineering and reality are mutually exclusive. :-)

Touche!

---

From: Peter Koch <[email-masked]>
Date: Tue, 9 Aug 2011 09:10:00 +0200

On Wed, Jul 27, 2011 at 07:02:07PM -0700, The IESG wrote:

> The IESG has received a request from an individual submitter to consider
> the following document:
> - 'Reducing the Standards Track to Two Maturity Levels'
>   <draft-housley-two-maturity-levels-08.txt> as a BCP

I have reviewed this version of the document and also the changes since -07.
While i am not sure that there is really a problem to solve, the current
version appears reasonable and an improvement over -07.  However, it would
benefit from a few clarifications before publication.  Also, even if it
will become part of BCP 9, i consider it a bit undesirable to distribute the core
IETF standards process across more than one document. For the short term,
that should be OK, though.

> 1.  Introduction

>    This document proposes several changes to the Internet Standards
>    Process defined in RFC 2026 [1].  In recent years, the Internet
>    Engineering Task Force (IETF) has witnessed difficulty in advancing
>    documents through the maturity levels: Proposed Standard, Draft
>    Standard, and finally Standard.  These changes are designed to
>    simplify the Standards Process and reduce impediments to standards
>    progression while preserving the most important benefits of the IETF
>    engineering approach.  In addition, the requirement for annual review
>    of standards-track documents that have not reached the top of the
>    maturity ladder is removed from the Internet Standards Process.

this informal language is a nice read, but consistency with RFC 2026
would suggest to say "standards track" instead of "maturity ladder".

> 2.  Two Maturity Levels

>    This document, once approved, replaces the three-tier maturity ladder
>    defined in RFC 2026 [1] with a two-tier maturity ladder.
>    Specifications become Internet Standards through a set of two
>    maturity levels known as the "standards track".  These maturity
>    levels are "Proposed Standard" and "Internet Standard".

>    A specification may be, and indeed, is likely to be, revised as it
>    advances from Proposed Standard to Internet Standard.  When a revised
>    specification is proposed for advancement to Internet Standard, the
>    IESG shall determine the scope and significance of the changes to the
>    specification, and, if necessary and appropriate, modify the
>    recommended action.  Minor revisions and the removal of unused
>    features are expected, but a significant revision may require that
>    the specification accumulate more experience at Proposed Standard
>    before progressing.

Suggest to add a reference to section 6.1.1 of RFC 2026 to clarify that
"recommended action" is actually the WG's request for advancement.

> 2.2.  The Second Maturity Level: Internet Standard

>    This maturity level is a merger of Draft Standard and Standard as
>    "Draft Standard" and "Internet-Draft".

>    The characterization of an Internet Standard remains as described in
>    RFC 2026 [1], which says:

>       An Internet Standard is characterized by a high degree of
>       technical maturity and by a generally held belief that the
>       specified protocol or service provides significant benefit
>       to the Internet community.

>    The criteria for advancing from Proposed Standard to Internet
>    Standard are confirmed by the IESG in an IETF-wide Last Call of at
>    least four weeks.  The request for reclassification is sent to the
>    IESG along with an explanation of how the criteria have been met.
>    The criteria are:

>        increase implementation complexity.

Operational complexity should be of equal concern.

>    (4) If patented or otherwise controlled technology is required for
>        implementation, the implementations demonstrate at least two
>        separate and successful uses of the licensing process.

>    After review and consideration of significant errata, the IESG will
>    perform an IETF-wide Last Call of at least four weeks on the
>    requested reclassification.

Is this another four weeks or the same as mentioned above?  In case of the
latter the first occurence could be removed to maintain chronological
ordering.

>    requested reclassification.  If there is consensus for
>    reclassification, the RFC will be reclassified without publication of
>    a new RFC.

So, how would a modified specification (say, with unused features not
passing criterion 3) be advanced?

>    As stated in RFC 2026 [1], in a timely fashion after the expiration
>    of the Last Call period, the IESG shall make its final determination
>    and notify the IETF of its decision via electronic mail to the IETF
>    Announce mailing list.  No changes are made to Section 6.1.2 of RFC
>    2026 [1].

Except for the extension of the LC period from two to four weeks, which
is good - but makes this an ambiguous statement.

> 2.3.  Transition to a Standards Track with Two Maturity Levels

>    Any protocol or service that is currently at the Proposed Standard
>    maturity level remains so.

>    Any protocol or service that is currently at the Standard maturity
>    level shall be immediately reclassified as an Internet Standard.

>    Any protocol or service that is currently at the abandoned Draft
>    Standard maturity level will retain that classification, absent
>    explicit actions.  Two possible actions are available:

>    (1) A Draft Standard may be reclassified as an Internet Standard as
>        soon as the criteria in Section 2.2 are satisfied.

>    (2) At any time after two years from the approval of this document as
>        a BCP, the IESG may choose to reclassify any Draft Standard
>        document as Proposed Standard.

> 3.  Removal of Requirement for Annual Review

>    In practice the annual review of Proposed Standard and Draft Standard
>    documents after two years called for in RFC 2026 [1] has not taken
>    place.  Lack of this review has not revealed any ill effects on the
>    Internet Standards Process.  As a result, the requirement for this
>    review is dropped.  No review cycle is imposed on standards track
>    documents at any maturity level.

I disagree with this analysis.  Lack of review (for whatever reason) has
lead to a perceived failure of the three-level standards track and thus
_had_ an effect on the standards process.  So, the reason to remove
this requirement is to bring it in line with reality and to deal with
the limited work force.

-Peter

---

From: [email-masked]
Date: Sat, 13 Aug 2011 11:29:22 -0700 (PDT)

> others that could have come from that author or participant.

That pretty much nails it. We've allowed exactly these sorts of vague and
nonspecific discusses to stand, with the result that the pain of getting to
draft or full standard is seen as far greater than the gain.

The really surprising thing, frankly, is that we have as many draft and full
standards as we do. 

> interoperability had actually been demonstrated in the case of

Actually it does in a way. Part of the problem with going to draft is that in
addition to it being painful and difficult, when it's over you're still not
done. And you have to wait for the final step, and when you get there it's not
all that well defined, either.

Really, this is Psych 101 stuff - it's been shown over and over that immediate
gratification wins out over long term satisfaction. Our three step process
sucks most if not all of the immediate gratification out of moving to draft.

And this is why I strongly support a move to a two step process. I think
the other changes that have been proposed are also important but I'll
take what I can get.

> independent interoperable implementations.

I really have to wonder if the entire yes/no-obj/discuss voting model is
appropriate for document advancement. For initial approval at proposed, sure,
having the ability to "discuss" the document makes all sorts of sense. But
for subsequent steps that virtue is a lot obvious, to me at least.

				Ned

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 14 Aug 2011 08:33:04 +1200

On 2011-08-14 06:29, [email-masked] wrote:
...
> for subsequent steps that virtue is a lot obvious, to me at least.

s/lot/lot less/

I pretty much agree, although one form of discuss might be
reasonable: "This document needs to be recycled at Proposed
Standard because of the following *observed* interoperability
problem:...".

In other words, once we have got this BCP out (soon, please),
the IESG should update the DISCUSS criteria specifically to
narrow them for the PS->IS transition.

    Brian

---

From: John Leslie <[email-masked]>
Date: Sat, 13 Aug 2011 16:43:15 -0400

[email-masked] <[email-masked]> wrote:
> 
> I really have to wonder if the entire yes/no-obj/discuss voting model
> is appropriate for document advancement. For initial approval at
> proposed, sure, having the ability to "discuss" the document makes
> all sorts of sense. But for subsequent steps that virtue is a lot
> obvious, to me at least.

   This, IMHO, is the right question: Does yes/no-obj/discuss resolve
the right issues when advancing from PS to DS?

   There is a perception, at least, that any AD can block progress to
DS (though in fact the process is more complicated, that's the way it's
perceived).

   I could somewhat-seriously suggest that the IESG shouldn't have to
vote at all on advancement to DS -- the right issues are questions of
fact which could perfectly well be delegated.

   (And wearing my Narrative-Scribe hat, I know how confusing it is
for IESG members to figure out what's expected of them when the question
is advancement of maturity level.)

--
John Leslie <[email-masked]>

---

From: [email-masked]
Date: Sat, 13 Aug 2011 15:03:46 -0700 (PDT)

> s/lot/lot less/

Yes, sorry.

+1

				Ned

---

From: Keith Moore <[email-masked]>
Date: Sat, 13 Aug 2011 18:08:02 -0400

> the right issues when advancing from PS to DS?

IMO, IESG members ought to be able to vote "no" if, after first voting "discuss" and not having the issue resolved within a well-defined amount of time, they still believe that the document should not progress.  That goes for any standards level.

(That's not to say that a single "no" vote should suffice to block progress of a document.)

"Discuss" is not the problem.  "Discuss" is actually a really good idea.  The problem is the lack of a "no" vote, which causes people to interpret "discuss" as if it were "no".  

If an IESG member sees something about a document that is suspect, it's entirely appropriate for him or her to raise questions about it.  That's their job.   And if the explanation is satisfactory, or the problem is fixed, it's then appropriate for the vote to be changed to "no objection" or "yes".   But if the problem isn't fixed, and the problem is substantial, the IESG member ought to be able to - IMO, has an obligation to - vote "no".  Anything else is dishonest and irresponsible.   The history of engineering is full of serious accidents that have been caused by technical reviews being ignored due to political pressure. 

Anybody can make a mistake, including IESG members.  It's okay to make an occasional bad review.   It's not okay to lie about that review.

Now there's a quite valid question about what kinds of objections IESG should be able to raise at PS vs DS or whatever.   And it's arguable that the criteria for PS and DS (or IS) should be different.   In particular, a revision of a document for DS is not an invitation to change the protocol, or even to fix deficiencies in the original design of the protocol.  I don't even think that advancement to DS should be considered an invitation to rewrite the document.  That's not to say that standards-track RFCs should never be revised, but that revising them should be done very conservatively (so as to make as few changes as possible to the text and especially organization of the document)  and that this should be treated separately from advancement in grade.

Keith

---

From: [email-masked]
Date: Sun, 14 Aug 2011 12:49:20 +0300

> narrow them for the PS->IS transition.

I also pretty much agree, and the IESG discuss criteria document
(http://www.ietf.org/iesg/statement/discuss-criteria.html) would certainly
benefit from a respin to provide guidelines for -bis documents and
documents advancing further.

However, as with most things I don't think there are hard and fast rules.
I can imagine a very old RFC being respin or advancing where I'd want some
rework. For instance, a vulnerability that was discovered after the
orginal RFC should be described, perhaps even dealt with somehow.

Jari

---

From: Keith Moore <[email-masked]>
Date: Sun, 14 Aug 2011 07:34:05 -0400

> orginal RFC should be described, perhaps even dealt with somehow.

I agree that such things need to be described, but I don't think this description should be gated on, or wait for, advancement in grade.  The errata mechanism can be used to report some kinds of vulnerabilities; separate RFCs for others.

Right now we have this expectation that we're going to rewrite the entire RFC just to declare something as DS.  That is a lot of work, opens up a huge can of worms, imposes a large barrier to the interop testing that we want to encourage, and sometimes introduces confusion by subtly changing the original specification.

Old standards do need maintenance.   Our process doesn't really account for that.   It has this fiction that once a protocol gets to Full Standard, it's good forever - or that it needs a complete and very tedious rewrite and recycle from Proposed.    
And the current process doesn't (in practice) document the shifting applicability of standards over time.  I'd like to see some sort of changes to the process that recognize the need to periodically review and occasionally maintain old standards, update their applicability without changing the text of the document, and that allows for incremental updating (with change bars etc.) without changing the basic form of the text.

Of course the changes do need to be reviewed, and IESG is already over-worked.  Maybe we need review groups that are analogous to working groups, and a separate supervisory board for reviews.   The review groups and board would be limited as to what they could do with a standard - e.g. clarify text where ambiguity had been identified, document new vulnerabilities (whether security, scaling, whatever), note limitations in applicability.   They'd have no power to add new features, or maybe only when the feature was narrowly tailored to fix a documented problem with the older spec.  This might even lighten IESG's load a bit because a few existing WGs could be changed to review groups.  And hopefully the review supervisory board's load would be lighter than IESG's so it wouldn't be so hard to recruit suitable candidates for it.

Keith

---

From: [email-masked]
Date: Sun, 14 Aug 2011 16:24:30 +0300

Keith,

> I agree that such things need to be described, but I don't think this
> description should be gated on, or wait for, advancement in grade.  The
> errata mechanism can be used to report some kinds of vulnerabilities;
> separate RFCs for others.
>
> Right now we have this expectation that we're going to rewrite the entire
> RFC just to declare something as DS.

I do not think we currently have that expectation. I do realize that we've
debated several cases and the need or lack thereof for additional changes.
But in general, a document that advanced is not expected to be rewritten.
At least not in my opinion. When I review -bis or -ds documents I usually
focus on the rfcdiff output between the old RFC and the new draft.

I also do acknowledge that we (the working groups and the IESG) have made
several errors in this respect, sometimes doing more than should really
have been done. But I do not think we have the expectation that all -ds
documents should be rewritten. That would be wrong.

What I tried to say above is that I dislike hard rules such as:

- We should never require a -ds document to say additional things
- We should always apply the most recent IETF approved rules (such as BCP
109 on key management) to all documents, including the -ds ones

But I do agree with you that whether a separate new RFC, editing this RFC,
errata or some other form of update should be decided on a case-by-case
basis.

Jari

---

From: Keith Moore <[email-masked]>
Date: Sun, 14 Aug 2011 09:37:54 -0400

> 109 on key management) to all documents, including the -ds ones

I'm not sure that I agree with that.  I think that when we have an old standard that's widely used, it's important to document newly discovered limitations.  But sometimes we live with the deficiencies, and sometimes we do what we can to fix the protocol without breaking compatibility.   We shouldn't necessarily insist that everything meet current criteria.

More generally, I don't think that advancing standards to DS should invite special scrutiny along those lines.  I think we should regularly apply such scrutiny to all IETF standards, regardless of their status.    If the standard no longer meets our current criteria for standardization, we should say so.  But that doesn't necessarily mean we should reclassify it as Historic or otherwise take away its status.   If people are still using the protocol, we can be of more service by maintaining the specification than by abandoning it.   We should enumerate the known problems, identify the known fixes or workarounds, identify opportunities for new fixes that haven't been developed yet.   

> errata or some other form of update should be decided on a case-by-case

I agree with that as you've stated it.  But I think there should be a presumption that writing a new RFC is done only in extreme cases, and that when this is done, the document has to recycle at Proposed.   And I'm okay in principle with editing existing RFCs and reissuing them with a new number, but I wish the new ones would have change bars.  (and maybe, when rendered in HTML, other indications of specifically which text is changed).

Keith

---

From: Hector Santos <[email-masked]>
Date: Sun, 14 Aug 2011 11:19:09 -0400

> documents should be rewritten. That would be wrong.

It is good to read this is recognized, especially for people who wish 
to remain on the sideline as online participants and depend on 
engineering common sense an faith among their peers and the IETF/IESG 
leaders and really don't have the time to be getting involved deeply 
in the IETF process.

I don't think anything is cut and dry, but I will note an increasing 
concern that in the name of fast tracking, politics and "who you 
know," there have been a perception of "rubber stamping" BIS work. 
Overall, not necessarily a bad thing, but serious care should be taken 
when existing base of protocol users are now asked to change again. 
When done at the expense of ignoring input as watered down concerns 
using "rough" consensus to exclude what is otherwise a less than 50%, 
but nonetheless significant weight of people, then the real losers is 
the IETF process and the IETF future. [I personally believe, the Rough 
Consensus model needs a serious review on how its applied today.]

I personally can recognize the need to move documents faster, but if 
only to serve as a reminder, it should not come at the expense of the 
long held guidelines the IETF/IESG has followed to make sure "errors" 
  are not made.  If anything, the two-maturity level proposal should 
increase the engineering expertise and scrutiny requirements for 
reviewers.

---

From: Russ Housley <[email-masked]>
Date: Sun, 14 Aug 2011 12:24:10 -0400

> "Discuss" is not the problem.  "Discuss" is actually a really good idea.  The problem is the lack of a "no" vote, which causes people to interpret "discuss" as if it were "no".  

We are a bit off topic here, at least for this subject line.

The IESG did make some changes to the voting procedures a couple of years ago.  The change was to make it clear that a single DISCUSS position could not block a document.  That is, the IESG believes in rough consensus too.  The current rules are available here:

   http://www.ietf.org/iesg/voting-procedures.html

Now, please return to the Last Call discussion of draft-housley-two-maturity-levels on this subject line.  If we want to discuss the IESG ballot process further, please start a new thread.

Russ

---

From: Keith Moore <[email-masked]>
Date: Sun, 14 Aug 2011 12:29:17 -0400

>   http://www.ietf.org/iesg/voting-procedures.html

Yes, I had read those procedures recently.   It's those very procedures that I have a problem with.

In particular this part is particularly heinous:

If an AD cannot get cooperation from the WG and cannot enter a ballot position that supports sending the document forward, then the AD should switch to "abstain."

That's completely inappropriate.   A document reviewer should never be expected to pretend like he doesn't have a problem with a document.   To expect an AD to change his vote to "abstain" is asking him to be dishonest and/or shirk responsibility. 

Keith

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 15 Aug 2011 09:09:36 +1200

Regards
   Brian Carpenter

> That's completely inappropriate.   A document reviewer should never be expected to pretend like he doesn't have a problem with a document.   To expect an AD to change his vote to "abstain" is askin…

iirc the IESG used to call this kind of abstention "holding
one's nose", but it can't be distinguished in the ballot from
an abstention "for cause" (conflict of interest). In theory,
you can find out which applies from the history in the tracker
or from the IESG minutes.

The point is to avoid blocking a document that the WG has
consciously chosen to support, and that has no concrete,
actionable defects, just because one or two ADs simply dislike
it.

    Brian

---

From: "Adrian Farrel" <[email-masked]>
Date: Sun, 14 Aug 2011 22:42:01 +0100

Both...

> > If an AD cannot get cooperation from the WG and cannot enter a ballot
position
> that supports sending the document forward, then the AD should switch to
> "abstain." 
> >
> > That's completely inappropriate.   A document reviewer should never be
> expected to pretend like he doesn't have a problem with a document.   To
expect
> an AD to change his vote to "abstain" is asking him to be dishonest and/or
shirk
> responsibility.

You can tell the difference because one is a "recuse" and the other is an
"abstain"

AFAIK the process that Keith calls out has never needed to be executed. It
exists to handle deadlock which might otherwise have to wait until the make-up
to the IESG changed or until NomCom was woken from its peaceful slumber. ADs are
not universal policemen and serve the will of the community. When an AD is
clearly in the rough against the consensus of the IESG they should not have the
power to individually block the progress of work for which there is clear IETF
consensus.

Note that when an AD does abstain he usually writes a strong comment in the
datatracker explaining why he is abstaining.

Adrian

---

From: Keith Moore <[email-masked]>
Date: Sun, 14 Aug 2011 17:42:59 -0400

On Aug 14, 2011, at 5:09 PM, Brian E Carpenter wrote:

>>   A document reviewer should never be expected to pretend like he doesn't have a problem with a document.   To expect an AD to change his vote to "abstain" is asking him to be dishonest and/or shirk responsibility. 
> 

To be fair, the current procedures do allow for a "recuse" vote, which handles the "conflict of interest" case.

> actionable defects, just because one or two ADs simply dislike

The problem with the current procedure is that sometimes a WG will be in denial about the existence of concrete, actionable defects, and other IESG members may not have the time to review the document thoroughly or be in a good position to do so.    Sometimes it's also the case that the WG is exercising poor judgment even if there aren't concrete defects, and sometimes there's no good and simple fix for the WG's output.  

When this happens, it is often the case that the WG represents a narrow set of interests and doesn't see (or want to see) how its work creates problems for other legitimate interests.  Though it does happen in other situations.

(Admittedly, some of these cases could be considered failures of management - the WG should not have been allowed to make a significant investment into a document without there being some cross-area checking and other sanity checking involved.  One hopes, of course, that such failures are rare.   But the proper remedy to a management failure is NOT to approve a seriously flawed document.)

If an IESG member finds a substantial defect with a document, I don't think it's responsible for him to hold his nose.  I don't think it's responsible to expect him to label his vote "abstain".  The responsible thing to do is to take that defect report seriously, even when the WG disagrees with the IESG member's opinion (as will often be the case when there's no simple fix for the document/protocol, or when the simple fix conflicts with the WG's ambition for the protocol).

The current IESG voting process presumes that the WG's opinion should trump that of its reviewers in the IESG.   This makes the whole review process a sham, as IESG review is reduced to essentially a rubber stamp.   

When I was on IESG, a single Discuss could block a document.  I'm not saying that was a good thing overall, but even under those circumstances a stubborn WG would often get away with refusing to fix technical defects in its documents.   With the current procedure, it's hard to imagine what would convince such a WG to fix things.   The only remaining available remedy would seem to be an appeal to IAB.

Keith

---

From: Keith Moore <[email-masked]>
Date: Sun, 14 Aug 2011 17:55:44 -0400

On Aug 14, 2011, at 5:42 PM, Adrian Farrel wrote:

> AFAIK the process that Keith calls out has never needed to be executed.

The process that I call out is any case where an IESG member is expected to vote abstain when he disagrees with the WG on the merits of a document or an appropriate fix.  Based on my own experience in IESG, I'd be very surprised if that never happened nowadays.

> datatracker explaining why he is abstaining.

I suppose my objections can be distilled into these points:

1) it's misleading to label this as an "abstain" vote, no matter how it's defined in IESG procedures
2) such objections should be treated more seriously than required by the current process
3) anything written in the data tracker is essentially buried to the audience of the RFCs  (though RFC errata might work somewhat better)

I'm not saying that a single AD should be able to block a document indefinitely, but neither should it be presumed that the WG should inherently prevail in a conflict.  

I actually think that the "rough consensus" model is not well suited for IESG, because IESG rarely has enough members with the kind of expertise needed to make that kind of judgment.   The number of IESG members who review a typical document and really understand its implications is probably around 3-4.    Most of those voting "no objection" have probably not read the document, at least not in depth.  So in a case where there's one sponsoring AD voting yes and one AD voting discuss, it's really close to one or two in favor against one opposed.  But the way IESG votes, distorts this and makes it look like many against one. 

Keith

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 15 Aug 2011 12:04:47 +1200

1. I'm glad that the ballot now has an explicit 'recuse' option.
Sorry about that red herring in my previous message.

2. (Excuse front posting). You can call it 'abstain' or you can
call it 'no'. What it means is that the AD concerned has
objections to the document that *cannot* be fixed (anything that
can be fixed is a 'discuss'). And I think in that case, there is
no way that one or two ADs should be able to block something
because they don't like it, *unless* they can convince the rest
of the IESG that the document is harmful.

Regards
   Brian

---

From: Keith Moore <[email-masked]>
Date: Sun, 14 Aug 2011 20:16:20 -0400

On Aug 14, 2011, at 8:04 PM, Brian E Carpenter wrote:

>  You can call it 'abstain' or you can
> can be fixed is a 'discuss').

That's not how I interpret the voting procedures:

...which seems to say that the question is not whether the objections can be fixed, but rather, whether the WG is willing to   fix them.

> And I think in that case, there is

Convincing the entire IESG is a very high barrier, especially when typically, most of the IESG just wants the issue to go away.    It might happen for a significant architectural issue, perhaps, but not for an area-specific technical flaw. 

(Though as I understand the rules, 1/3 of the non-recused IESG members can block a document by voting Abstain.  Still, that's a very high barrier given that in most cases 1/3 of IESG members won't even read the document.)

Why should a narrowly-focused working group get to hold sway unless a significant fraction of the IESG objects?   How does it benefit the Internet to gloss over technical problems found during IESG review?

(not to single you out, Brian, because I'm fairly confident that a significant plurality of IETF participants agree with you)

Keith

---

From: Barry Leiba <[email-masked]>
Date: Sun, 14 Aug 2011 20:40:08 -0400

> Convincing the entire IESG is a very high barrier, especially when
> typically, most of the IESG just wants the issue to go away.    It might
> happen for a significant architectural issue, perhaps, but not for an
> area-specific technical flaw.

Here's the point: if an AD can't get at least one or two other ADs to
read the document and agree to join in the blocking, then that AD MUST
NOT be allowed to block the document.  That's even the case if the AD
thinks she's found a serious flaw.  Because if, out of 14 others in
the IESG, not ONE other is willing to read the document, understand
the issue, and agree on it.

According to the current rules:

1. If ONE other AD places a second DISCUSS position OR enters a
comment that supports the first AD's DISCUSS, the document stays
blocked.  The "single discuss" procedure does NOT kick in.

2. If the WG still insists on pushing the document and the IESG goes
into the "alternate ballot" (which should be "alternative ballot", but
never mind that) procedure, the DISCUSSing AD still only needs TWO
other ADs to read the document and vote "no".  Three "no" votes will
maintain the block on the document.

Really, if you can't get two out of fourteen technical leaders to
agree with you, maybe you just have to accept that you're wrong.  If
things get down to such a rarely used (read: never) last-ditch
procedure, you can bet that essentially the entire IESG *will* read
the document (you might miss one or two ADs at the most, *maybe*).  If
it gets to that and you're *right* to block, you should be able to
convince two of fourteen to support you.

There is NO case in which it makes sense for a *single* person, with
no peer support or agreement, to be allowed to block a document
indefinitely, no matter HOW serious she THINKS the problem she's found
is.

Barry

---

From: Ronald Bonica <[email-masked]>
Date: Sun, 14 Aug 2011 20:46:56 -0400

> [snip]
> When an AD is clearly in the rough against the consensus of the IESG 
> they should not have the power to individually block the progress of 
> work for which there is clear IETF consensus.
>

+1

The current voting procedure is the only protection that the community has against an AD who has decided that his opinion trumps the expertise of the working group and the rest of the IESG. 

                                                                  Ron

---

From: Keith Moore <[email-masked]>
Date: Sun, 14 Aug 2011 21:01:40 -0400

> the issue, and agree on it.

That's also how I interpret the rules.  I just don't think that this is sufficient review.  I think that in practice it makes IESG more-or-less a rubber stamp for any issue that isn't easily fixed with small and often inconsequential changes to the document text.

The problem is, the ADs are very busy people, and their expertise has to cover a wide range of topics, so there will be few IESG members who can really understand a subtle issue.   Document reviews outside of one's subject area are very difficult and require considerable focus.   GIven that, even if only one AD catches a flaw in a document, there's a good chance (though not a certainty of course) that it's something that warrants more attention.   It's far more likely that no ADs will find the flaw because nobody really took the time to read the document thoroughly and to understand its implications of the document outside of the narrow subject area of the working group.

I understand (and agree with) the sentiment that, ultimately, one or two people shouldn't be able to block a document.  Nor do I want documents held up for trivialities as, unfortunately, sometimes happens.  But I've seen many cases where working groups failed to do an adequate level of review outside of their narrow areas of concern, and it appears that IESG's current rules and workload make it difficult for problems to get fixed after a document leaves the WG.   

(and people keep arguing to remove steps from our process so that there will be even less review after a document has progressed to Proposed...)

The only other formal level of review we have are the Last Call comments which, given the volume of documents that get Last Called, amounts to a fairly small and random chance that somebody outside the WG will happen to notice the proposed document action and give the document a thorough review.

To put the question another way:  What level of formal technical review, outside of a WG, best serves IETF's goals? 

Keith

---

From: Keith Moore <[email-masked]>
Date: Sun, 14 Aug 2011 21:03:35 -0400

>> work for which there is clear IETF consensus.

I don't disagree with the above.  What I disagree with is the notion that the rest of the overworked IESG provides adequate review.

> The current voting procedure is the only protection that the community has against an AD who has decided that his opinion trumps the expertise of the working group and the rest of the IESG. 

It's the only protection that exists now.  But it's not the only way to solve the problem.

Keith

---

From: Barry Leiba <[email-masked]>
Date: Sun, 14 Aug 2011 21:30:25 -0400

> The problem is, the ADs are very busy people, and their expertise has to
> cover a wide range of topics, so there will be few IESG members who can
> really understand a subtle issue.   Document reviews outside of one's
> subject area are very difficult and require considerable focus.   GIven
> that, even if only one AD catches a flaw in a document, there's a good
> chance (though not a certainty of course) that it's something that warrants
> more attention.   It's far more likely that no ADs will find the flaw

My point isn't that I expect multiple ADs to find the same problem
initially.  What I expect to happen is that, say, Keith finds a flaw
and lodges a DISCUSS position.  It's discussed, and the WG insists
that it's not a problem and shouldn't be changed.  Keith maintains the
DISCUSS, and it's gone over on the telechat.  At the end of that,
Keith still maintains the DISCUSS.  More discussion among Keith, the
responsible AD, and the WG ensues, and it's still not resolved -- the
WG still refuses to change the spec, and Keith still maintains the
DISCUSS position.

It's at THAT stage that at least one other AD should have read the
document, read Keith's DISCUSS position, and agreed with Keith.
That's not a very high bar.  And *that* is where I say that if there's
still no one backing Keith's DISCUSS, even with his *asking* other ADs
to have a look, Keith needs to back down.

Barry

---

From: Keith Moore <[email-masked]>
Date: Sun, 14 Aug 2011 21:36:30 -0400

> to have a look, Keith needs to back down.

Disagree in two aspects:

1. It can be very difficult to get other ADs to understand issues, especially in areas outside their expertise, when they're mired in other documents.  Getting something fixed shouldn't rely on other members of IESG to pay attention to it.

2. Even if the document is approved over the AD's objection, I think it's completely unacceptable to bury the AD's technical objection.  Expecting the AD to change his Discuss to an Abstain is dishonesty on the part of the organization.

But at this point I'm repeating myself, and I don't think we're covering any new ground.  So I'll either just hope that this made a useful impression on somebody, or I'll write up an Internet-Draft.

I just think it's sad to see the level of technical review in IETF, which was already insufficient at the time that I was an AD, steadily eroding still further.

Keith

---

From: Bill McQuillan <[email-masked]>
Date: Sun, 14 Aug 2011 22:21:58 -0700

On Sun, 2011-08-14, Keith Moore wrote:

> 2. Even if the document is approved over the AD's objection, I
> think it's completely unacceptable to bury the AD's technical
> objection.  Expecting the AD to change his Discuss to an
> Abstain is dishonesty on the part of the organization.

It sure sounds like if the status is renamed: 

   Disagree-But-Overruled

the annoyance becomes much smaller.

(Just my 2 cents)

-- 
Bill McQuillan <[email-masked]>

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 15 Aug 2011 11:50:39 +0200

Hi Keith,

Ø  The only other formal level of review we have are the Last Call comments which, given the volume of documents that get Last Called, amounts to a fairly small and random chance that somebody outside the WG will happen to notice the proposed document action and give the document a thorough review.

I do not know whether to call these 'formal' but currently all documents that go to Last Call also undergo three expert reviews - Gen-ART, Sec-DIR and OPS-DIR. According to the scope of the documents a few more expert reviews may be called as required by the WG, shepherd or AD - APPS-Dir, TSV-Dir, MIB-Doctors, DNS-Dir, etc. 

Regards,

Dan

From: [email-masked] [mailto:[email-masked]] On Behalf Of Keith Moore
Sent: Monday, August 15, 2011 4:02 AM
To: Barry Leiba
Cc: [email-masked]; IETF
Subject: Re: IESG voting procedures

Keith

---

From: Keith Moore <[email-masked]>
Date: Mon, 15 Aug 2011 07:52:36 -0400

Thanks for that info.  That does make me feel a bit better.

Keith

On Aug 15, 2011, at 5:50 AM, Romascanu, Dan (Dan) wrote:

> I do not know whether to call these ‘formal’ but currently all documents that go to Last Call also undergo three expert reviews – Gen-ART, Sec-DIR and OPS-DIR. According to the scope of the documents a few more expert reviews may be called as required by the WG, shepherd or AD – APPS-Dir, TSV-Dir, MIB-Doctors, DNS-Dir, etc.
>  
> Regards,
>  
> Dan
>  
>  
>

---

From: Thomas Narten <[email-masked]>
Date: Mon, 15 Aug 2011 08:48:25 -0400

+1.

One single AD should never be allowed to block a document for which
there appears to be community support. If a document really has
serious problems, it is the job of the AD to rally other ADs or other
impacted community members to oppose the document. Yes, this can be
more work for an AD, but that's what they get paid the big bucks for.

And if they can't do that, that really means the AD is an outlier and
needs to step aside, no matter how strongly they feel about a topic.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Mon, 15 Aug 2011 08:51:16 -0400

"Romascanu, Dan (Dan)" <[email-masked]> writes:

> The only other formal level of review we have are the Last Call
> comments which, given the volume of documents that get Last Called,
> amounts to a fairly small and random chance that somebody outside
> the WG will happen to notice the proposed document action and give
> the document a thorough review.

ADs can and should arm twist impacted parties to review documents that
appear to be troubled and for which approval has troubling
implications.

If an AD can't rally the community (and more specifically parties that
will be forced to implement or deploy a technology) to oppose a
problematical document, that speaks volumes to the issue.

Thoma

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 15 Aug 2011 14:53:51 +0200

Hi Thomas,

The paragraph below does not belong to me. In my message I was actually
answering it. 

Regards,

Dan

> From: Thomas Narten [mailto:[email-masked]]
> Sent: Monday, August 15, 2011 3:51 PM
> To: Romascanu, Dan (Dan)
> Cc: Keith Moore; Barry Leiba; [email-masked]; IETF

---

From: Thomas Narten <[email-masked]>
Date: Mon, 15 Aug 2011 09:03:57 -0400

> The paragraph below does not belong to me. In my message I was actually

Right. My mistake. It was Keith's comments I quoted.

Sorry 'bout that.

Thomas

---

From: Russ Housley <[email-masked]>
Date: Mon, 15 Aug 2011 14:18:34 -0400

> Convincing the entire IESG is a very high barrier, especially when typically, most of the IESG just wants the issue to go away.    It might happen for a significant architectural issue, perhaps, bu…

The current procedures do not require the AD that has found the flaw to convince the whole IESG that it is a flaw.  One other AD is sufficient,

Russ

---

From: Russ Housley <[email-masked]>
Date: Mon, 15 Aug 2011 14:28:18 -0400

> I understand (and agree with) the sentiment that, ultimately, one or two people shouldn't be able to block a document.  Nor do I want documents held up for trivialities as, unfortunately, sometimes…

My experience is that a technical flaw, even if it is a corner case, is acted upon by the WG.  There are rare cases where the WG has lost energy, but in general the WG wants to produce a quality output.  As a result, technical flaws are not the place where things get messy.  Rather, things get messy over issues that have a political component.

Russ

---

From: Keith Moore <[email-masked]>
Date: Mon, 15 Aug 2011 14:34:34 -0400

> My experience is that a technical flaw, even if it is a corner case, is acted upon by the WG.  There are rare cases where the WG has lost energy, but in general the WG wants to produce a quality ou…

What's the difference between "issues that have a political component" and disagreements about whether there is a technical flaw?   Seems to me that people who don't want to admit that there's a technical flaw (or who can't easily see it from their perspective) often dismiss the issue as a political problem.

Keith

---

From: Martin Rex <[email-masked]>
Date: Tue, 16 Aug 2011 17:08:42 +0200 (MEST)

Keith Moore wrote:
> 
> I actually think that the "rough consensus" model is not well suited for
> IESG, because IESG rarely has enough members with the kind of expertise
> needed to make that kind of judgment.   The number of IESG members who
> review a typical document and really understand its implications is
> probably around 3-4.    Most of those voting "no objection" have
> probably not read the document, at least not in depth.  So in a case
> where there's one sponsoring AD voting yes and one AD voting discuss,
> it's really close to one or two in favor against one opposed.  But the
> way IESG votes, distorts this and makes it look like many against one.

While I agree to your later observation that some IESG ballots are
more like 1:1 or 2:1 (because of the many "no objection"s), I still
think that "rough consensus" should work for the IESG as well, and
primarily because of the "issue resolution process" that it requires,
rather than every IESG member being or becoming a technical expert
on each particular subject.

I believe that a certain "technical leadership" is desirable
_and_ possible, and dismissing "rough consensus" and as well of
some other mentioned ideas would reduce the IESG function to a
mostly clerical one.

IMHO the current ballot rules, especially those for standards action,
are not very well suited for corner cases, and they can (and sometimes
do) fail in an unsafe fashion.  I would appreciate if they would be
adjusted to fail in a safe fashion.

One of the corner cases being discussed is when there is active
dissent within the IESG, and how to cope with it.  I'm not sure
that there really is a problem.  The assertion that has been made
is that publishing a document (or approving a standards action)
would be the "safe" failure mode, and I'm violently opposed to
such a concept.  If there is a problem, fix the issue resolution
process.  Adjusting the ballot rules to ignore dissent seems like
a bad idea to me.

A somewhat different issue, which I am actually more concerned about,
is the unsafe failure in the lack-of-support situation.  Why do we
make it so hard to IESG to let a document "die"?  One single "yes"
and otherwise just "recuded" and "no objection" does not make
a very good safeguard.  It would seem much more sensible to require
at least two ADs to provide a "yes"--which means that they should
have performed some level of independent review of the document
the document writeups and at least parts of controversial discussion
(if that exists).  Or alternatively, leave at 1 "yes", but make the
responsible AD always recused for documents for which they requested
an IETF LC.

The issue is, if there is strong dissent, put the burden to resolve
the dissent or to convince at least two IESG members that remaining
dissent after issue resultion it is unjustified, on the document
submittor(s) rather than leaving the burden on the IESG alone,
as a procedural safeguard.

-Martin