Subject: Discuss criteria for documents that advance on the standards track

From: Jari Arkko <[email-masked]>
Date: Tue, 30 Aug 2011 23:18:57 +0300

During the discussion of the two maturity levels change, a question was brought up about DISCUSSes appropriate for documents that advance on the standards track. We discussed this in the IESG and I drafted some suggested guidelines. Feedback on these suggestions would be welcome. The intent is to publish an IESG statement to complement the already existing general-purpose DISCUSS criteria IESG statement (http://www.ietf.org/iesg/statement/discuss-criteria.html).

Here are the suggested guidelines for documents that advance to IS:

http://www.arkko.com/ietf/iesg/discuss-criteria-advancing.txt

Comments appreciated. Please send comments either on this list or to the IESG ([email-masked]) in time before our next telechat (Thursday, September 8, 2011).

Jari

---

From: Keith Moore <[email-masked]>
Date: Tue, 30 Aug 2011 17:17:19 -0400

There's something inherently wrong with trying to establish criteria for voting DISCUSS.  

My understanding was always that DISCUSS was supposed to be an indication that, at a minimum, the AD needs to understand the situation better before casting a yea or nay vote.   The resolution of a DISCUSS might end up being a yes vote, a no objection vote, a request for clarification of the document.  It should also be possible for an AD to say "now that I've understood better, I can't support this going forward"   (for which ABSTAIN is not an appropriate label).

It should never be wrong for an AD to vote DISCUSS, though it's reasonable to limit the amount of time during which a DISCUSS can stand for a particular document.

It should also never be wrong for an AD to vote NO if the AD has initially voted DISCUSS, made a sincere effort to understand the issues, believes that he does so, and can state 2026 or other documented criteria for why the document is not suitable for approval.

So my recommendations are: 

1. Fix the broken IESG voting system before you try to establish more decision criteria.  I'm serious.  The system you have now is just too broken, and has been broken for a long time.  It places too much pressure on ADs to cave in when a WG produces flawed output that isn't easily fixed.  It is weighed too heavily in favor of approving a document - such that one AD can vote YES, another vote DISCUSS, no other ADs read the document and all vote NO OBJECTION, and the AD that votes DISCUSS will be expected to change that vote to ABSTAIN because nobody else felt like reading it.   And the idea that an AD should ever, ever, be compelled to change his vote to an ABSTAIN is completely unacceptable.  And the "votes" of ADs who haven't read the document should not count AT ALL.

Here is a stab at better voting criteria:

- The possible votes are: Yes, No, No Objection, Discuss, Abstain, Recuse
- Yes means "I've read it, I believe it meets applicable criteria (2026 or whatever else is applicable), that there's community-wide consensus to support the document, and that all issues raised by reviewers (including directorate, last call comments, etc.) have been adequately addressed."
- No means "I've read it, but one or more of the above criteria have failed to have been met."  The criteria must be documented.
- No Objection means "I've read it, and I think there are issues with it, but I don't think those issues are sufficient to block the document".   The criteria should be documented, but the AD isn't compelled to do so.
- Discuss means "I have read the document and see at least one potential issue which needs to be discussed."  Discuss should always be raised before voting No.  However, Discuss votes should time out and be replaced by Abstain (if not explicitly changed by the AD) after say 45 days.  (however, nothing should ever stop an AD from changing his vote to No.)  Note that the ADs, WG chairs, authors, etc. should attempt to resolve the issue offline (not during the telechat or other IESG meeting) but the discussion should be archived.
- Abstain means "I did not read this", and/or "I don't consider myself competent to review this"
- Recuse means "I have a conflict of interest with stating a position on this document"

All Discuss votes must be changed (whether explicitly by the AD or by automatic timeout) before a ballot is finished.
If there are two or more No votes, all ADs without a conflict of interest are expected to read the document, and the vote is taken again.
In order to approve a document or standards action, there must be twice as many Yes + No Objection votes as No votes.

2. Don't overconstrain the use of DISCUSS.  In particular, don't ever create a situation where a reviewer can't cite a problem with a document, regardless of whether that problem has previously been enumerated.    It's fine to have guidelines for the benefit of new ADs but they should be nonbinding.  You need to have other ways of dealing with the occasional stubborn AD who votes DISCUSS or whatever without a defensible reason.

This applies to all IESG voting actions, not just moving from PS->DS/IS.

3. I take serious issue with the statement in the draft that IESG reviews are "reviews of last resort" and the implication that WG reviews are sufficient.  In numerous situations this has not been the case.  I'm all for having IESG place greater reliance on directorates (especially if this is formalized to some degree), so as to lessen IESG's workload to to get individual ADs out of the hot seat.   But for IESG to presume by default that the WG has done due diligence is for IESG to shirk its responsibility.

4. When evaluating PS->DS/IS actions, IESG should avoid repeating the PS review.  Instead it should look at what has changed between PS and DS/IS, and what has been learned as a result of implementation and deployment experience.   The changes have to be minimal and "safe".    No new features can be added, and there needs to be reason to have confidence (ideally backed by analysis and implementation experience) that any changes made will not impair interoperability between both PS and DS/IS versions of the protocol.

Design flaws discovered since approval as PS are also fair game, and flaws for which the implications are better understood than at PS time should also be fair game.   However it's not fair to revisit an issue that was raised at PS approval time, unless new information or understanding makes that issue of worse consequence than understood at that time.  (e.g. say an issue was raised at PS time and it was decided it was too minor to hold up the document, but since then practical DoS attacks had been identified that exploited that problem...then it's fair to raise the issue.  It's always fair to raise an issue about something that actually causes problems.)

What's also not fair game is to "raise the bar" - to expect the document at DS to meet more stringent criteria than it was required to meet at the time of PS approval.  IESG needs strong justification (e.g. community consensus) to raise the bar in this way for standards that are actually deployed.  This includes but isn't limited to security issues.
(Though it might be reasonable to make an exception to this if more than N years had lapsed since PS approval.)

But IESG should not presume that there are no serious technical issues.   It shouldn't presume one way or the other.  And it shouldn't discourage ADs from raising DISCUSSes about technical issues at all.

Keith

On Aug 30, 2011, at 4:18 PM, Jari Arkko wrote:

> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf

---

From: Fred Baker <[email-masked]>
Date: Tue, 30 Aug 2011 14:51:01 -0700

> It should also never be wrong for an AD to vote NO if the AD has initially voted DISCUSS, made a sincere effort to understand the issues, believes that he does so, and can state 2026 or other docum…

I agree in part. My problem is that even some ADs tell me that ADs sometimes behave as if they haven't done their job if they haven't produced a "discuss". A few months ago, I got a "discuss" on a document that in essence said "what should I be writing a 'discuss' about?". I have no problem with valid concerns, and many of the concerns raised are valid. It should be possible for a working group to produce a quality document and have the IESG simply approve it, however. I'm not sure it is.

> Discuss votes should time out and be replaced by Abstain (if not explicitly changed by the AD) after say 45 days.  

There I disagree. If the AD raised a valid issue, the ball is in the author/wg's court to address it. They can game this rule by not responding until after 45 days.

Personally, I would rather see the document returned to the author/wg if review results in a request for major changes. In my working group, we recently had one draft completely rewritten in the response to "discuss"es, and I pulled it back for the WG to decide whether it still had consensus around the resulting draft. I would say the same about a "discuss" that is not adequately responded to in 45 days; the IESG should clean it off their plate.

> What's also not fair game is to "raise the bar" - to expect the document at DS to meet more stringent criteria than it was required to meet at the time of PS approval.

Hmmm, the "demonstrated interoperability" requirement of DS/IS is in fact a raising of the bar,and one that has served us well. We don't (although IMHO we should) require even an implementation to go to PS.

---

From: Keith Moore <[email-masked]>
Date: Tue, 30 Aug 2011 18:25:48 -0400

> I agree in part. My problem is that even some ADs tell me that ADs sometimes behave as if they haven't done their job if they haven't produced a "discuss". A few months ago, I got a "discuss" on a…

The first time I reviewed a document as an AD, I realized I was working very hard to find something wrong.  And I did - I found a bona fide (and uncontroversial) technical flaw - actually an incorrect value for a constant.  But after that I got over needing to find things that were wrong.   I found plenty of issues without trying to do anything more than read and understand the documents - and had quite enough work to do just reading and understanding them and writing up those issues without making more work for myself.  (Especially when other ADs would often demand that I not only identify the issues but also specify the fixes - something I always regarded as out-of-scope for an AD and even potentially "tampering" with WG output.)

So I don't disagree that ADs sometime believe that they need to vote Discuss or they're not doing their jobs.  Though I have a hard time believing that the drafts coming out of WGs these days are so good that ADs have to work hard to find things that are wrong with many of them.   If they're trying to find a reason to Discuss every document, well I guess that is a problem.   But overall, it seems like there are far more incentives to avoid finding reasons to object to a document, than there are to find them.

> There I disagree. If the AD raised a valid issue, the ball is in the author/wg's court to address it. They can game this rule by not responding until after 45 days.

I don't think I stated that rule very well.   If the author/wg fail to address the issue within 45 days, the AD should of course be able to change the vote to No.  The point is that the Discuss should inherently be a temporary state, a time during which the AD doesn't vote No (yet) but lets the author/WG know that he has issues with a document.  And the idea is to encourage those parties enough time to understand each other and work out their differences before bringing the issue to a firm vote.  But I don't think that the WG should be able to game the rule to force the AD's objection to go away, nor should the AD be able to singlehandedly block a document.

> Personally, I would rather see the document returned to the author/wg if review results in a request for major changes. In my working group, we recently had one draft completely rewritten in the re…

IMO, any time a document is significantly revised, IESG should consider it a separate item, requiring a new Last Call, writeup, ballot, etc.

> Hmmm, the "demonstrated interoperability" requirement of DS/IS is in fact a raising of the bar,and one that has served us well. We don't (although IMHO we should) require even an implementation to…

I should have been more specific.  By "raising the bar" I was thinking about things like design decisions, and document quality.   If feature FOO was judged to be of adequate design at PS, it's not fair to say it's not adequate on DS review unless there's some new information that demonstrates that it doesn't work well.  Or if the wording of a document was okay with IESG at PS, it's not fair for an AD to nitpick that wording at DS review time unless there's some reason to believe that the wording caused problems. For those cases the AD can always file an erratum.

Keith

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 31 Aug 2011 12:13:45 +1200

On 2011-08-31 09:51, Fred Baker wrote:

...
> If the AD raised a valid issue, the ball is in the author/wg's court to address it. They can game this rule by not responding until after 45 days.

Not if the draft has been updated and the AD doesn't either cancel the DISCUSS within
a reasonable time*, or explain why the update is insufficient. This happens, probably
as often as authors failing to address an issue within a reasonable time. Whichever
party is responsible for the delay should be subject to a timeout, surely.

That said, I disagree with Keith. The current DISCUSS criteria were written
for a very specific reason - to get rid of the type of DISCUSS listed in the
"non-criteria". There was a lot of analysis done of documents that were stuck
in the IESG for months in the 2004/2005 timeframe, and the non-criteria describe
the problem cases - DISCUSSes that were not actionable or simply expressed the
fact that the AD didn't like the WG's informed consensus.

Of course nothing's perfect and I'm sure the criteria could be improved. But
not having them was much worse.

* for example, is "IESG Evaluation::AD Followup (for 32 days)" a reasonable
  delay for a Discussing AD to review an updated draft [hint, hint]?

     Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 31 Aug 2011 12:30:09 +1200

On 2011-08-31 08:18, Jari Arkko wrote:
...
> Comments appreciated.

To answer Jari's original request: +1 to these new guidelines.

Not worth nit-picking until we have some experience.

   Brian

---

From: John C Klensin <[email-masked]>
Date: Wed, 31 Aug 2011 02:31:50 -0400

--On Tuesday, August 30, 2011 14:51 -0700 Fred Baker
<[email-masked]> wrote:

>> What's also not fair game is to "raise the bar" - to expect
>> the document at DS to meet more stringent criteria than it
>> was required to meet at the time of PS approval.
> 
> Hmmm, the "demonstrated interoperability" requirement of DS/IS
> is in fact a raising of the bar,and one that has served us
> well. We don't (although IMHO we should) require even an
> implementation to go to PS. 

I know it is controversial, but there is at least one other area
in which we should be raising the bar for DS/IS by dropping the
bar for Proposed.  If we really want to get PS specs out quickly
while the percentage of people who easily write very high
quality technical English in the IETF continues to go down, we
need to stop the behavior of various IESG members simulating
technical editors or translators to "fix" PS text (or insisting
that the author or WG do so, which, IMO, is less bad but still
often a problem).  Doing that will get documents out faster,
perhaps even a lot faster in some cases, but will inevitably
result in PS documents that need significant editorial work
before being approved at DS.  

I think that would actually be a good thing.  I think that
stuffing explicit placeholders to the effect of "this needs to
be rewritten to be completely clear to folks who haven't
participated in the WG" into PS documents would be a fine idea
-- it would get those documents finished and out while making
their preliminary nature very clear.   But it implies a higher
editorial quality standard --a higher bar-- for DS/IS than for
PS.

    john

---

From: Jari Arkko <[email-masked]>
Date: Wed, 31 Aug 2011 09:36:05 +0300

Keith, thank you for the feedback. Some responses inline:

> 1. Fix the broken IESG voting system before you try to establish more decision criteria.

I do agree with your general thinking here. The way that you describe the different positions is what I personally try to achieve in my IESG reviews.

But I think you are overemphasizing the role of the vote designations. Again, I try to do this already, though maybe not always succeeding. In general, if the Area Directors are not doing their job of stopping bad stuff and moving good stuff along, they don't need a new voting system, they need to grow a spine.

Elaborating a bit more about this:

We have plenty of cases where a DISCUSS has been left standing, and today that acts as your "NO" vote. (Obviously, in many cases this was an error, or lack of effort. I'm quilty of this for sure, even right now I have a queue of DISCUSSes and their proposed resolutions that I need to go check.) But I think a DISCUSS should stand, if there is a serious issue and it is not rectified.

There are corner cases where a single AD has an opinion that is not shared by the rest of the community. For those cases we have an override procedure. This has been never been invoked, but that's probably because it gets pretty ugly way before that -- there are often heated discussions between ADs about discusses.

We have the ABSTAIN vote, which some ADs use to vote "NO", often together with other ADs who feel the same. There's never been a case where this would have blocked a document from proceeding, as we've never collected the necessary 1/3 number of ADs to vote ABSTAIN for any single document. My conclusion is that ABSTAIN stands for NO-OBJECTION in practical terms. I don't recommend its use...

> 2. Don't overconstrain the use of DISCUSS.  In particular, don't ever create a situation where a reviewer can't cite a problem with a document, regardless of whether that problem has previously been enumerated.

I agree, and that's why the guidelines I posted are just that -- guidelines. They are not binding rules, they leave room for a judgment call.

> 3. I take serious issue with the statement in the draft that IESG reviews are "reviews of last resort" and the implication that WG reviews are sufficient.  In numerous situations this has not been the case.

Of course. But I don't see a conflict between a "review of last resort" and having the last resort find issues. I wish we'd find less issues, but at least I still view the IESG as the final check, that should catch issues if others have not. Its not a "last resort" in the sense that it would not be invoked; we do review all documents very carefully. Its a last resort only in the sense that there is an expectation that previous stages should have produced a quality result without issues.

Jari

---

From: Eric Burger <[email-masked]>
Date: Wed, 31 Aug 2011 03:13:55 -0400

Would having professional editors make a difference here?

---

From: Jari Arkko <[email-masked]>
Date: Wed, 31 Aug 2011 11:34:03 +0300

Eric, John,

>> often a problem).

I think the existing Discuss criteria already says very clearly that editorial comments cannot be blocking DISCUSSes.

I see a lot of language feedback from IESG and directorate reviews, but its rare to have them appear in the DISCUSSes. If they do, its inappropriate, you should push back. And I'm sure you will :-)

Besides, we pay the RFC Editor a large amount of money every year to do the editing. Documents need to be clear enough to be understood, but the RFC editor can handle most of the editorial problems.

(That being said, I've seen documents that were sent back because they really were not understandable. Obviously there is some bar under which you should not go, or the document cannot advance at all. This happens more in WG stages than in the IESG. But if you can't communicate your idea clearly then I think it should be up to you to hire co-workers/editors to  help clarify your idea... not the IETF's problem, IMHO.)

Jari

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Aug 2011 07:36:33 -0400

> before being approved at DS.  

Given that people commonly implement and deploy Proposed Standards in products, sometimes even prior to their approval as Proposed Standards, I think this is a very bad idea.  I think it's likely to create interoperability problems between PS and DS versions of products, and sometimes between implementations of the same PS version.  For better or worse, the widespread desire to implement at PS (recommendations in 2026 notwithstanding) pretty much forces IETF to try to produce high quality prose at PS.

I agree that IESG should not generally be acting as technical editors, so maybe IETF needs to find some way to provide technical editors who are proficient in technical English before documents are reviewed by IESG.  Documents written by native English speakers would also benefit.   Maybe the GEN-ART review could be expanded, or maybe the RFC Editor could assume some role here (awkward though that might be at first).

Keith

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Aug 2011 07:51:15 -0400

> We have the ABSTAIN vote, which some ADs use to vote "NO", often together with other ADs who feel the same. There's never been a case where this would have blocked a document from proceeding, as we…

The biggest problem with the current voting system (other than misleading labels, which do cause real problems of their own) is the presumption that the document should go forward no matter how few IESG members read the document.   So No Objection votes from ADs who didn't read the document count as Yes votes, but there's also a presumption in the rules (as well as pressure from other ADs who want to get documents off the agenda) to clear Discuss votes in favor of moving a document forward whether or not the identified issues have been adequately addressed.

(One thing that I didn't mention that also needs to be fixed if it's still the case is the presumption that the responsible AD votes Yes for the document.  I don't know what the tools do now, but this Yes vote used to be automatically filled-in.)

> Of course. But I don't see a conflict between a "review of last resort" and having the last resort find issues. I wish we'd find less issues, but at least I still view the IESG as the final check,…

That is not a valid expectation, in either theory or practice.  That's my point.  Arguably when a poor quality document gets to IESG, it's a failure on the part of the WG to do due diligence.  But the problem is actually deeper than that - it's partially structural (in that IETF partitions almost all work into narrowly focused WGs who don't represent a sufficiently broad spectrum of interests), and partially due to a failure to consistently apply good engineering principles across all of IETF.   

IESG's assuming that the WG has produced a quality result basically works to mask the other problems with IETF's way of doing work.   But even if WGs generally did produce high quality results without issues (which I don't think is the case now), IESG review should still not presume that they do.   There will always be some failures at the WG level, and the IESG's job is to try to catch those.

Keith

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Aug 2011 08:02:02 -0400

> I think the existing Discuss criteria already says very clearly that editorial comments cannot be blocking DISCUSSes.

So nobody has the job of making sure that the documents are well-written in clear English?

> Besides, we pay the RFC Editor a large amount of money every year to do the editing. Documents need to be clear enough to be understood, but the RFC editor can handle most of the editorial problems.

If the document is edited for clarity after final review, that's really a botch in the process.  It means that the review doesn't apply to the version of the document being published.   Of course the RFC Editor's resources shouldn't be used for documents that aren't otherwise deemed worthy of publication, and you'd like to avoid the RFC Editor having to do multiple reviews, so I admit that this is tricky to solve.

> (That being said, I've seen documents that were sent back because they really were not understandable. Obviously there is some bar under which you should not go, or the document cannot advance at a…

If writing clear specifications isn't the IETF's problem, I'm not sure what is.

Keith

---

From: John C Klensin <[email-masked]>
Date: Wed, 31 Aug 2011 10:15:50 -0400

--On Wednesday, August 31, 2011 11:34 +0300 Jari Arkko
<[email-masked]> wrote:

>>> I know it is controversial, but there is at least one other
>...

> I think the existing Discuss criteria already says very
> clearly that editorial comments cannot be blocking DISCUSSes.

This issue drifts quickly from DISCUSS criteria into how we
review, approve, and edit documents, but I think they are
completely intertwined... so please forgive the apparent
digression.

> I see a lot of language feedback from IESG and directorate
> reviews, but its rare to have them appear in the DISCUSSes. If
> they do, its inappropriate, you should push back. And I'm sure
> you will :-)

Yes, but... First of all, a long list of "editorial comments"
from one or more ADs can do fully as good a job of slowing a
document down as a DISCUSS or two.  And anyone smart enough to
be on the IESG is more than capable of disguising what is really
an editorial comment as a substantive issue that gets a DISCUSS.
Part of the problem is that there are a whole series of judgment
calls tangled up with this topic.  For example:

* If an AD says "this particular paragraph is really important
to the specification and, after several readings, I can't tell
what is actually means or whether a particular implementation
would be conforming", that is a problem, certainly worthy of a
"DISCUSS" (or, given Keith's typology, "NO until it is fixed".
On the other hand, "this particular paragraph is important and
it is much harder  to understand than it should be, even though
the intent is clear" is an editorial comment that should be
non-blocking.  I think we all know that the boundary between
those two can be very thin and even a matter for debate.

* I can't speak for others, but I'm often tired with a document
for which I've got responsibility by the time I get it past the
small revisions and nit-picking or WG LC and pre-IETF-LC
comments from ADs.  Faced with what I consider an inappropriate
editorial comment or even an inappropriate DISCUSS, I have a
choice between pushing back (thereby guaranteeing that I will
have to deal with the document for additional weeks or months)
or just making the requested changes and getting the document
off my plate.  I'm probably more stubborn about editorial
matters than most people, but even I usually just give in.

> Besides, we pay the RFC Editor a large amount of money every
> year to do the editing. Documents need to be clear enough to
> be understood, but the RFC editor can handle most of the
> editorial problems.

Yes, and I've been saying that, along with variations on "let
them do their job" for years.  But it introduces another set of
issues.  I don't think I'm giving away a secret by saying that
they believe that their contracts and the way their performance
is measured focus on pages of published output per month.  They
also believe that the IESG has told them things that amount to
"minimal editing" or "copy editing only" on many occasions.
Unless we can change things to shift from measuring what is easy
to include a focus on quality improvements, the RFC Editor ends
up in an impossible situation.

>From a standardization perspective, it also isn't desirable to
make lots of editorial changes post-approval.  AUTH48 is a
rather dangerous process because it is easy to make mistakes
there that no one catches or, again, for an exhausted author to
make "it is easier to let this go than to fight about it"
decisions.  One could, in principle, send all such documents
back to a producing WG for final, pre-publication review but, in
my experience, most of the people in most WGs get more exhausted
by the document end-game than most authors (and most other
participants want to reopen long-dead issues by questioning the
language used to describe them).

This is precisely why many SDOs use a two-stage vote in which
the first vote approves the general concepts and technical
content, the document then gets edited into final form by
profession staff, and then the first and only actual approval
vote occurs (possibly by a different body with a different
mission).  That wouldn't work well in the IETF... and not only
because we have a more-than-one-step standards process.   I
think we could adopt a process in which a WG, or an AD, could
look at a document before IETF LC and say "this is going to need
a lot of editorial work before approval but the technical
content looks ok, so let's hand it off to the RFC Editor now and
get it cleaned up before a final WG review and IETF LC".  But
that has a lot of procedural and budgetary implications and I
don't know if we are ready to deal with them.

> (That being said, I've seen documents that were sent back
> because they really were not understandable. Obviously there
> is some bar under which you should not go, or the document
> cannot advance at all. This happens more in WG stages than in
> the IESG. But if you can't communicate your idea clearly then
> I think it should be up to you to hire co-workers/editors to
> help clarify your idea... not the IETF's problem, IMHO.)

Agreed.  See above.

best,
   john

---

From: John C Klensin <[email-masked]>
Date: Wed, 31 Aug 2011 10:42:42 -0400

> well-written in clear English?

Keith, I don't think either Jari or I were saying anything
remotely close to that.

Think about it this way, moving away from the language in 2026
and back to the much-earlier assumptions on which that language,
and the multiple-stage standards process, were based.  With the
understanding that these are oversimplifications, we can have
three types of documents:

	(1) Complete spec, so complete and so clear that someone
	with reasonable familiarity with the technology but no
	contact with the authors, anyone who participated the
	relevant WG, or the WG mailing list can produce an
	implementation that will interoperate with other
	implementations --both those produced with the same
	level of knowledge and those produced by, or in
	coordination with, the authors.
	
	(2) Specs that are good and clear enough to act as aids
	to memory and documentation about agreement so that
	someone who participated in the WG and who can ask
	questions of a WG mailing list and/or the authors can
	produce an interoperable implementation.
	
	(3) A spec that just outlines an idea for others to try
	out.

Because I want to avoid losing the distinction, I would point
out that only the first is really suitable for use in a
procurement spec (although people often accept the second and
sometimes the third).   

>From a standardization standpoint, we should consider the third
as potentially appropriate for Experimental, especially if  the
purpose of the experiment is to help understand what the details
should be.  We've been trying to hold everything (even, in many
cases, experimental documents) up until we get to that first
criterion, but, if one looks at long-term history and reads
between the lines, only DS and above actually require
documentation at that level.  We ought to, IMO, be permitting
publication of PS documents at the second level as long as there
are no _obvious_ ambiguities that cannot be figured out (the
same way) by people of good will acting in good faith and with
help from WG lists and as long as there are no other "known
[technical] defects".

I suggest that we have called that second category "Proposed
Standard" and then interpreted that term in ways that have
caused us harm.  It is closer to what ISO has called a "Type II
Technical Report" (known in internal jargon as a "pre-standard"
or "sub-standard") and what IEEE calls (or used to call) a
"Draft Standard for Trial Use" -- a document that represents
consensus about the idea and about publication, including
publication as a standard when it is mature enough, and good
enough to form a basis for experimentation and serious
evaluation, but not expected to meet criterion (1) above.

>From that particular narrow perspective, if one wants PS
documents published quickly rather than killing them and the
standards process through delays and nit-picking that are almost
irrelevant to the technical content of the specification, then
we either need to make some really major changes in how we do
things or we need to be willing to publish things that,
explicitly or implicitly contain notes like "the following
paragraph is horribly written and barely intelligible and will
need to be fixed up for DS but, right now, everyone involved
knows what it means".  Insisting that all PS documents be
"well-written in clear English" may actually be our enemy.

That doesn't mean they shouldn't be.  And it would be perfectly
reasonable for some WGs and authors to decide the situation is
"do the work now (and accept the added delay) or do it later, so
better now".  But we should not be trying to force every PS
document to meet such a standard... at least if we want to get
them out quickly and have them treated as interim steps rather
than final products. 

> If the document is edited for clarity after final review,
> that's really a botch in the process.  It means that the
> review doesn't apply to the version of the document being
> published.   Of course the RFC Editor's resources shouldn't be
> used for documents that aren't otherwise deemed worthy of
> publication, and you'd like to avoid the RFC Editor having to
> do multiple reviews, so I admit that this is tricky to solve.

Yes.  See my previous note.

> If writing clear specifications isn't the IETF's problem, I'm

Well, see previous note.  But also remember that people evaluate
where to get work done based on what it "costs".  And SDOs like
ISO (and ISO/IEC JTC1) and ITU-T bundle professional editing,
pre-approval-review, into the system for "free".

      john

---

From: John Leslie <[email-masked]>
Date: Wed, 31 Aug 2011 11:06:49 -0400

Keith Moore <[email-masked]> wrote:
> 
> The biggest problem with the current voting system (other than
> misleading labels, which do cause real problems of their own) is the
> presumption that the document should go forward no matter how few
> IESG members read the document.

   Keith makes a good point here; but I wouldn't agree to any rule
that a particular number must "read" a document. Some ADs quite
properly defer actual reading to review teams.

> So No Objection votes from ADs who didn't read the document count as
> Yes votes,

   No, they don't. (But I can't ask folks posting to this thread to
actually _understand_ the difference.)

   Roughly, the rule says "enough" ADs must enter a "position" before
a document can be approved. Basically, "No-Objection" says the AD is
"somewhat" familiar with the document and actively consents to approving
it.

   These "positions" are _NOT_ "votes".

> but there's also a presumption in the rules (as well as pressure from
> other ADs who want to get documents off the agenda) to clear Discuss
> votes in favor of moving a document forward whether or not the
> identified issues have been adequately addressed.

   This is somewhat true, but the "pressure" is highly variable. The
agendas _are_ too crowded (IMHO); but in most cases sufficient progress
will have been made before the telechat that only a few seconds are
needed to agree to "AD-followup" status.

   When there is no progress between telechats (most often due to
unresponsive authors/editors), there _is_ some pressure to reduce
what a DISCUSS asks for. It might help for non-IESG folk to chime in
on whether this is good or bad...

   (There _are_ cases where the responsible AD puts quite a bit of
pressure on the DISCUSS-holder: that's really an internal issue which
I don't believe this list should be discussing.)

> (One thing that I didn't mention that also needs to be fixed if
> it's still the case is the presumption that the responsible AD
> votes Yes for the document. I don't know what the tools do now,
> but this Yes vote used to be automatically filled-in.)

   We're seeing a number of cases where the responsible AD holds a
DISCUSS at the time of the telechat -- generally because LastCall
hadn't ended yet when the document was placed on the agenda.

   IMHO, there's nothing there that needs "fixing".

> Arguably when a poor quality document gets to IESG, it's a failure
> on the part of the WG to do due diligence.

   IMHO, there _are_ poor-quality documents that get on the IESG agenda.
I'm not sure it helps to allocate blame...

   But there is a huge variance between WGs on diligence. I am alarmed
by the sheer number of COMMENTS saying in essence, "This document is
not specific enough to guide an implementor to an interoperable
implementation." To me, that's really-close to DISCUSS territory...

   However, we really don't have a process for improving situations
like that -- other than for it to be a DISCUSS and for authors to
actually be responsive (which would probably require repeating at
least one LastCall). :^(

   In the absence of such a process, I really can't blame ADs for
reducing such issues from DISCUSS to COMMENT, and entering ABSTAIN
if they think the issue is serious.

> But the problem is actually deeper than that - it's partially
> structural (in that IETF partitions almost all work into narrowly
> focused WGs who don't represent a sufficiently broad spectrum of
> interests), and partially due to a failure to consistently apply
> good engineering principles across all of IETF.   

   +1

   This problem, of course, is endemic in organizations that depend
on volunteers. And I really don't have suggestions on how to ensure
sufficient wide-area review, though review teams certainly help.

   I wonder if there's some room for process-improvement by formalizing
some role for review teams...

> IESG's assuming that the WG has produced a quality result basically
> works to mask the other problems with IETF's way of doing work.

   I don't agree that's what IESG members "assume" -- they IMHO instead
presume that documenting ideas (even not-fully-baked ideas) is a
mostly-unmitigated good-thing.

> But even if WGs generally did produce high quality results without
> issues (which I don't think is the case now), IESG review should
> still not presume that they do.   There will always be some failures
> at the WG level, and the IESG's job is to try to catch those.

   I don't think we have universal agreement on that as a goal.

   Of course, neither do we have universal agreement that anyone else
has the job to "catch these". Some of us quite plainly believe it
shouldn't be anyone's job to "catch these": that ideas should be
published and we should see whether "rough consensus" emerges later.

   (All of which brings us to the actual question: when advancing a
maturity-level, what constitutes sufficient "consensus"? Arguably,
folks will expect a higher maturity level to indicate that the
"standard" is ready to be handed to an implementor, and merely by
following it, sufficient interoperability is ensured. Alas, we
really don't have a process to address that expectation...)

--
John Leslie <[email-masked]>

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Aug 2011 11:08:07 -0400

On Aug 31, 2011, at 10:42 AM, John C Klensin wrote:

> We ought to, IMO, be permitting

That would be fine with me if we could somehow effectively discourage deployment of proposed standards in products.  But that's a lost cause, IMO.

Keith

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 31 Aug 2011 10:23:46 -0500

Dear Jari,

> During the discussion of the two maturity levels change, a question was 
> brought up about DISCUSSes appropriate for documents that advance on the 
> standards track. We discussed this in the IESG and I drafted some 
> suggested guidelines. Feedback on these suggestions would be welcome. The 
> intent is to publish an IESG statement to complement the already existing 
> general-purpose DISCUSS criteria IESG statement 
> (http://www.ietf.org/iesg/statement/discuss-criteria.html).
>
> Comments appreciated. Please send comments either on this list or to the 
> IESG ([email-masked]) in time before our next telechat (Thursday, September 
> 8, 2011).
>
> Jari

As with 2119bis, thank you guys for putting this together.

The text in http://www.arkko.com/ietf/iesg/discuss-criteria-advancing.txt
says

   IESG reviews should be considered as a review of "last resort".  Most
   documents reviewed by the IESG are produced and reviewed in the
   context of IETF working groups.  In those cases, the IESG cannot
   overrule working group consensus without good reason; informed
   community consensus should prevail.

I might have thought that the point for documents advancing on the standards
track is that they already have *IETF* consensus (especially if they are
being advanced without text changes), so the IESG cannot overrule *IETF*
consensus without good reason ... which seems like a higher bar (one can 
more easily assume the existence of a rogue working group producing a rogue 
Internet-Draft, than a rogue IETF that is still sane enough to NomCom-select 
non-rogue ADs who will push back on rogue standards-track documents 
advancing :-)

So perhaps s/informed working group consensus/informed IETF consensus/?

I note that this isn't quite tying the IESG's hands (I'm reading this 
descriptive text as the moral equivalent of "MUST NOT overrule IETF 
consensus without good reason", which leaves one hand free).

Spencer

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Aug 2011 11:29:38 -0400

> properly defer actual reading to review teams.

I disagree that that is "quite properly".  I believe that the AD's name goes on the vote, so an AD whose vote is actually counted as contributing toward yes or no has some responsibility to read the document.  Having said that, I'll readily admit that I read some documents more thoroughly than others when I was an AD.

> actually _understand_ the difference.)

In practice, they do, because the difference between Yes and NoObj under current voting rules is never significant.

>   These "positions" are _NOT_ "votes".

Call them what you will, they're still explicitly enumerated.

> needed to agree to "AD-followup" status.

That, of course, is how things should work.   Most drafts should not be controversial.   It's what happens when things are controversial that I'm concerned about.

>   IMHO, there's nothing there that needs "fixing".

Maybe that's been fixed since I was on IESG.  If so I'm glad to hear it.

> implementation." To me, that's really-close to DISCUSS territory...

To me, it's more than adequate grounds to reject a document.   It should be considered a duty to vote DISCUSS on such documents.

> if they think the issue is serious.

And again, the idea that an AD should be compelled to label such an objection as an abstention, is harmful.

> some role for review teams...

perhaps.

> mostly-unmitigated good-thing.

I don't think it's what individual IESG members assume; I think it's what the voting guidelines and other aspects of IESG's process assume.  And that process has been around without many changes since before I was on IESG, which is to say for 15 years or more.  It's not a valid assumption, and it hasn't been a valid assumption since the IETF grew to more than a few hundred active participants.  (When I first started working with IETF circa 1990, cross-area review was much less of an issue because the affected community was so much smaller, the protocols were much less complex, and the installed base was so small that it was relatively easy to fix mistakes.)

>   I don't think we have universal agreement on that as a goal.

Perhaps not, but I want to explicitly state that some form of area-independent review, independent of the WG, is an appropriate and necessary goal.  There are other ways to achieve that goal than having IESG do the review.  But the idea that nobody should do the review and the WG's output should always be considered adequate is ridiculous.

Keith

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Aug 2011 11:35:06 -0400

thanks Spencer for pointing this part out.

>  community consensus should prevail.

The idea that WG consensus should prevail is simply incorrect.  It biases IESG in an inappropriate way.

There are a number of very good reasons for overriding WG consensus, e.g.

- there is no evidence of broad community consensus or a clear lack of broad community consensus
- the document does not meet the criteria specified in 2026 (or other document when applicable)
- the document is ambiguous in such a way that it is likely to degrade interoperability

The WG DOES NOT represent the entire community.    Far too often, WGs are deliberately chartered in such a way as to marginalize parts of the community

Keith

---

From: John C Klensin <[email-masked]>
Date: Wed, 31 Aug 2011 11:36:20 -0400

--On Wednesday, August 31, 2011 11:08 -0400 Keith Moore
<[email-masked]> wrote:

>> publication of PS documents at the second level as long as
>> there are no _obvious_ ambiguities that cannot be figured out
>> (the same way) by people of good will acting in good faith
>> and with help from WG lists and as long as there are no other
>> "known [technical] defects".
> 
> That would be fine with me if we could somehow effectively
> discourage deployment of proposed standards in products.  But
> that's a lost cause, IMO.

Keith,

IMO, there are two possibilities here.  At this point, sadly,
both involve a chicken-and-egg problem.  Such is life.

(1) We proceed as if Proposed Standards are what 2026 (and the
earlier culture) claims they are and work on ways to reinforce
that notion in the community.  If that is our goal, than getting
documents out sooner and having them look a bit rough as well as
being a bit rough is A Good Thing and reinforces the "deploying
at PS, much less at I-D, is taking a big risk".  The idea isn't
foreign to the industry: to take a handy example, we saw
products identified as 801.11-draft-N floating around for years.
Every vendor who shipped one (and their more sophisticated
customers) knew that there could be serious incompatibilities
between their approximations to the drafts and the real 802.11n.

(2) We accept, and effectively encourage, deployment of proposed
standards in products, either because it is a lost cause or
because we think it is a good idea.  As part of that, we move
further down the path of ensuring that PS documents are complete
and polished (my group (1)) and of guaranteeing that there will
be no significant changes in the future (either in going to DS
or in grade).    If that is really our position, then we better
stop grumbling about how long it takes to get PS documents out
(and the partially-consequential deployment of technologies from
I-Ds), accept the fact that the SDOs who used to be very
concerned about how much faster the IETF was than they were have
now won and become faster than us (in large measure because we
have slowed down so much).  And we should stop wasting time
trying to figure out how many maturity levels we need or what
those criteria should be because the answer is "one level is
what we get and pretending anything else, e.g., by trying to
fine-tune procedures, is an embarrassing public act of
self-delusion" (stronger language occurs to me but would
probably get the message killed by spam filters).

Remember that, while ignoring procedures and category
definitions that we don't follow is not desirable, "fixing" them
to reflect a model that doesn't (and won't) exist either is a
public demonstration that we are disconnected from reality.  I'd
much rather leave that distinction to some other SDOs than join
them.  YMMD.

    john

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Aug 2011 11:47:00 -0400

On Aug 31, 2011, at 11:36 AM, John C Klensin wrote:

> that notion in the community. [...]

> because we think it is a good idea.  [...]

Agree with your description of the two possibilities, but I think the decision of possibility 2 has long since been forced on us by market expectation and habit.

We could fix it, perhaps, by drastically changing how we label our products (dropping the whole notion of "Proposed Standard", refusing to publish your category 2 documents as RFCs, etc.).   But we can't significantly change market perception of what "RFC" and "Proposed Standard" mean.  

(which, BTW, is why I think that moving to a two-step process, while probably Mostly Harmless, won't do a bit of good in the overall scheme of things)

> much rather leave that distinction to some other SDOs than join

And my assertion is that the model inherent in your possibility #1 above doesn't exist and won't exist absent from drastic change.   Insisting on high quality for proposed standards is recognizing current market reality.  

And I'm not one of those people who believes that market perceptions are inherently unchangeable.  But expecting the market to change how it interprets our documents without us bothering to significantly change how we present them to the public - THAT would indeed be a serious disconnect from reality.

Keith

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 31 Aug 2011 10:51:04 -0500

Keith,

Yes, to what you are saying, but I was pointing out that the text we're discussing isn't intended to apply to moving what a working group has consensus for onto the standards track, it's intended to apply to what the *IETF* already has consensus for, that's already on the standards track, further along the standards track.

I think there's a higher bar, especially when a document is advancing unchanged, because I don't think there's any meaningful distinction between a widely deployed PS that could be improved, and advancing it to be a widely deployed IS that could be improved. I don't see the value-add from DISCUSSing the advancement, because the same document is sitting there as a proposed standard, and widely deployed.

If ADs look at a document proposed for advancement and see real and meaningful opportunities to improve the specification, I think it makes just as much sense to advance the document, as is, and start looking for people who will produce an improved version, as to slow down the document for DISCUSSion in the hope you end up with an improved document, that can then advance. Finding those people, and chartering that work, falls well within the "S" in IESG, AFAICT.

Thanks,

Spencer
  ----- Original Message ----- 
  From: Keith Moore 
  To: Spencer Dawkins 
  Cc: Jari Arkko ; IETF Discussion 
  Sent: Wednesday, August 31, 2011 10:35 AM
  Subject: Re: Discuss criteria for documents that advance on the standards track

  Keith

---

From: John C Klensin <[email-masked]>
Date: Wed, 31 Aug 2011 12:19:20 -0400

--On Wednesday, August 31, 2011 11:47 -0400 Keith Moore
<[email-masked]> wrote:

>> (2) We accept, and effectively encourage, deployment of
>> proposed standards in products, either because it is a lost
>> cause or because we think it is a good idea.  [...]
> 
> Agree with your description of the two possibilities, but I
> think the decision of possibility 2 has long since been forced
> on us by market expectation and habit.
> 
> We could fix it, perhaps, by drastically changing how we label
> our products (dropping the whole notion of "Proposed
> Standard", refusing to publish your category 2 documents as
> RFCs, etc.).   But we can't significantly change market
> perception of what "RFC" and "Proposed Standard" mean.  

Whether we agree on how best to manage a perception change and
what it takes, I agree that there is no single magic bullet,
much less a painless one.   I would turn your comment around
slightly and say that a change in names is likely to have little
or no effect on perceptions unless we actually change what we
do.   As I said in and earlier note, there is something of a
chicken-and-egg problem here.
 
> while probably Mostly Harmless, won't do a bit of good in the
> overall scheme of things)

For reasons I've explained in previous notes, I don't think
procedural changes, especially from "not followed and doesn't
work but ancient" or "contemporary and [still] doesn't work or
reflect reality, are harmless ("mostly" or otherwise).

>> definitions that we don't follow is not desirable, "fixing"
>> them to reflect a model that doesn't (and won't) exist either
>> is a public demonstration that we are disconnected from
>> reality.  I'd much rather leave that distinction to some
>> other SDOs than join them.  YMMD.
> 
> And my assertion is that the model inherent in your
> possibility #1 above doesn't exist and won't exist absent from
> drastic change.   Insisting on high quality for proposed
> standards is recognizing current market reality.

If you are right, then fussing around with how many maturity
levels we have, what criteria we claim we are using, and even
how we name things, are dangerous as well as a waste of time and
resources.  That is where we are maybe in violent agreement --
we either ought to be identifying real problems and fixing them
or just staying with what we have until we have the knowledge
and will needed to make real changes.  In that regard, I applaud
one aspect of Jari's note, which is that it affects and IETF
administrative procedure and not a fundamental (and unnecessary)
procedural change.

By contrast, suppose draft-housley-two-maturity-levels were just
shelved for a while and replaced by an announcement by an AD or
two (ideally from areas that have really low advancement rates
even as compared to the already-low IETF average) that they
intended to

	(i) accept a public assertion of interoperability as an
	implementation report, issue an IETF LC for DS to see if
	anyone with real experience with the protocol objects
	loudly, and then move to advance the document, and 
	
	(ii) that they would construe the passage of the
	relevant number of month and public claims of deployment
	as sufficient incentive to issue an IETF LC for
	advancement to IS.

If the rest of the IESG were willing to go along with that, we'd
have a real experiment without permanent changes to the official
procedures.  People who saw a problem advancing a particular
protocol or document could object and it would presumably work
only for PS documents that were of high quality.  If it worked
out, we would have a basis for coming back and evaluating
whether there really was enough of a difference between DS and
IS to be worth the trouble and whether the practical differences
between that sort of implementation report and today's
interpretation of what is required was.  And then we could take
...two-maturity-levels back up with some confidence about both
utility and potential harm.

And, if you are right and this is all hopeless without really
major changes, then that experiment might give us the
information needed to finally come to grips with that and either
make the major change or accept the status quo and stop spending
energy trying to turn the knobs a degree or two.

> And I'm not one of those people who believes that market
> perceptions are inherently unchangeable.  But expecting the
> market to change how it interprets our documents without us
> bothering to significantly change how we present them to the
> public - THAT would indeed be a serious disconnect from
> reality.

I actually agree.

    john

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Aug 2011 12:25:08 -0400

On Aug 31, 2011, at 12:19 PM, John C Klensin wrote:

> and will needed to make real changes.

That would certainly be my preference.

Keith