Subject: Why the IESG needs to review everything...

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 Jul 2011 10:12:52 +1200

Responding to Glen Zorn's question in plenary:

Firstly, not all ADs review all drafts - that's why you will see
numerous "no objection" or missing ballot responses.

Secondly, the drafts are de facto reviewed by review teams
these days (gen-art, security area, etc.). This serves to alert
the ADs if a draft really needs careful review. The workload is
more reasonable than it used to be.

Thirdly, when I was in the IESG, I was surprised quite often by
*glaring* errors that had not been picked up before. Somebody has
to be responsible for catching these, and today it's the IESG.

Fourthly, because of the exact same discussion that Glen raised in
plenary, the IESG defined and published its criteria for DISCUSS
several years ago. Sometimes there are inappropriate DISCUSSes
and those need to be pointed out when they happen.

I hear the IESG members responding exactly right to this question.

-- 
Regards
   Brian Carpenter

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 27 Jul 2011 18:34:37 -0400

> numerous "no objection" or missing ballot responses.

Brian,

I've been repeatedly hearing from IESG folk for some year -- and seeing reports 
relating to Nomcom  -- that, in fact, ADs are expected (and maybe required) to 
read every draft.

> to be responsible for catching these, and today it's the IESG.

Offering no comment on any individuals, but looking at process and structure...

The problem with this sort of datum is that it is entirely out of essential 
context and therefore is not balanced against other factors.

It implies that the considerable cost of the AD review process -- including cost 
to authors, wg chair and shepherds for processing the AD's feedback -- is 
justified by this error-detection benefit.  But it doesn't really try to 
evaluate at relative cost against relative benefit.

Is all of this effort justified if exactly one minor error is found each year? 
Two?  One major?  What is that /actual/ improvement that this process produces? 
  What thresholds should apply?

That is, the presumption in your comment -- and your comment represents a broad 
constituency in the IETF -- is that reviewing only has positive effects of the 
review and it even implies a perfection to the process.

In reality, the current process produces documents with many errors, already. 
The world survives, as does the Internet.

It well might be true that omitting the AD reviews would increase the number. 
By how much?  To what effect?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Martin Rex <[email-masked]>
Date: Thu, 28 Jul 2011 01:13:22 +0200 (MEST)

> numerous "no objection" or missing ballot responses.

I can understand the resource contention when reading drafts
brought to the IESG.  I would not expect that more than 25% of
the IESG members have reviewed a draft (no average) when it is
up for ballot.  But for _standards_action_, I would probably
appreciate if there was a minimum requirement of "yes" ballots
(definitely more than just one) based on independently formed
opinions, which requires some level of review of a document.

According to rfc2026:

   4.2.2   Informational

   An "Informational" specification is published for the general
   information of the Internet community, and does not represent an
   Internet community consensus or recommendation.  [...]

   4.1.1   Proposed Standard

   [...]
   A Proposed Standard specification is generally stable, has resolved
   known design choices, is believed to be well-understood, has received
   significant community review, and appears to enjoy enough community
   interest to be considered valuable. [...]

While I'm OK seeing just a single "yes" on the IESG ballot for
Informational, I feel quite uncomfortable with a single "yes"
for a standards action.

-Martin

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 Jul 2011 12:37:34 +1200

> opinions, which requires some level of review of a document.

See http://www.ietf.org/iesg/voting-procedures.html

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 Jul 2011 12:50:24 +1200

On 2011-07-28 10:34, Dave CROCKER wrote:
> I've been repeatedly hearing from IESG folk for some year -- and seeing
> reports relating to Nomcom  -- that, in fact, ADs are expected (and
> maybe required) to read every draft.

Expected is one thing; but even the IESG's own rules do not *require* this.
http://www.ietf.org/iesg/voting-procedures.html

I certainly never like ballotting NO OBJECTION unless I had at least
skimmed the draft and read at least one positive review. I'm sure I
never ballotted YES without a careful reading.

...
>> to be responsible for catching these, and today it's the IESG.
...
> Is all of this effort justified if exactly one minor error is found each
> year? Two?  One major?  What is that /actual/ improvement that this
> process produces?  What thresholds should apply?

It depends on what you mean by minor or major, but my experience as an
author is that I get valuable comments from the IESG that definitely
improve the document, even after all the comments one gets from
WGLC and IETF LC and the various review teams.

Why? My guess is that it's because that the buck stops with the IESG - and
that automatically makes people more conscientious. Everybody else can
<shrug>; the final approving body can't.

In other words, if you "fix" this "bug" by moving the final responsibility
elswehere, you will just move the problem to the same place.

> 
> That is, the presumption in your comment -- and your comment represents
> a broad constituency in the IETF -- is that reviewing only has positive
> effects of the review 

Of course, there can be cases where that is not so - in fact, that's
the main reason that the IESG defined the DISCUSS criteria a few years
ago.

> and it even implies a perfection to the process.

Not in my mind.

> In reality, the current process produces documents with many errors,
> already. The world survives, as does the Internet.
> 
> It well might be true that omitting the AD reviews would increase the
> number. By how much?  To what effect?

Hard to tell. But it would amount to giving the IESG secretary a large
rubber stamp *unless* the final responsibility was explicitly moved
elsewhere.

    Brian

---

From: Hannes Tschofenig <[email-masked]>
Date: Wed, 27 Jul 2011 20:51:09 -0400

So, you arguing that all DISCUSSes by ADs are indeed justified and necessary.
It is great to hear that our leadership is completely unbiased with regard to technology, does not follow their own (or a company) agenda, misjudge their expertise in a certain area, showed long delays in responding, etc. 

As a document author I remember a couple of cases where certain ADs showed "interesting" behavior. 

As Jari said at the plenary it is difficult to talk about this topic without going into specific cases but on the other hand we don't want to upset individuals either. 
Hence, the story is difficult.  

My suggestion: Talk to the Nomcom if you think that certain ADs treated you in an unfair way. 

Ciao
Hannes

On Jul 27, 2011, at 6:12 PM, Brian E Carpenter wrote:

> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 Jul 2011 13:13:36 +1200

> So, you arguing that all DISCUSSes by ADs are indeed justified and necessary.

No. I said exactly the opposite: "Sometimes there are inappropriate DISCUSSes
and those need to be pointed out when they happen."

> It is great to hear that our leadership 

They are not our leadership; they are members of our community performing
a service role for the rest of us.

> is completely unbiased with regard to technology, 

I didn't say that; they are human.

> does not follow their own (or a company) agenda, 

I didn't say that. In my experience this is pretty rare in IESG discussions,
compared to the blatant company position-pushing I have often seen in
WG discussions. But again, they are human. That's why part of the NomCom's
job is balancing the membership as much as possible.

> misjudge their expertise in a certain area, 

I didn't say that; they are human.

> showed long delays in responding, 

I didn't say that; they are human.

> etc. 

So do I. But your name is on 44 RFCs - a couple of cases is not really surprising.

> My suggestion: Talk to the Nomcom if you think that certain ADs treated you in an unfair way. 

Absolutely agreed. The NomCom needs an overview of this.

   Brian

---

From: Hannes Tschofenig <[email-masked]>
Date: Wed, 27 Jul 2011 21:24:47 -0400

I believe we agree.

---

From: "GT RAMIREZ, Medel G." <[email-masked]>
Date: Thu, 28 Jul 2011 10:29:39 +0800

Likewise...

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Hannes Tschofenig
Sent: Thursday, July 28, 2011 9:25 AM
To: Brian E Carpenter
Cc: IETF discussion list
Subject: Re: Why the IESG needs to review everything...

I believe we agree. 

This e-mail message (including attachments, if any) is intended for the use of the individual or the entity to whom it is addressed and may contain information that is privileged, proprietary, confidential and exempt from disclosure. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender and delete this E-mail message immediately.

---

From: SM <[email-masked]>
Date: Wed, 27 Jul 2011 23:45:20 -0700

Hi Martin,
>    Internet community consensus or recommendation.  [...]

The above is incorrect.

Regards,
-sm

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 28 Jul 2011 09:18:35 -0400

Brian,

> http://www.ietf.org/iesg/voting-procedures.html

First, the written rules do not matter much, if actual behavior runs contrary.

Second, expectations constitute a real burden on people, and the discussion was 
about the actual burden.  The claim that ADs are expected to read all 
submissions has been consistent for some years, from many different folk 
including ADs.

> never ballotted YES without a careful reading.

On matters of human behavior, one of the continuing challenges in the IETF is 
the pervasive tendency for a sole speaker to cite their own behavior as if it 
were representative of a broader population.  (The formal term is "sampling 
error", I believe.)

There is a basic difference between a single exemplar versus the aggregated 
behavior of a population.  The former is useful to demonstrate an existence 
proof, such as an exception.  It says nothing about the behavior of others in a 
group.

In other words Brian, you are a great guy.  You were diligent, reasonable and 
balanced when you held these positions and asserted your authority.  No one 
should consider criticizing any aspect of how you did the job.

Forgive me, but none of that matters for this discussion.  Your own behavior, 
back then, is irrelevant to this discussion, since no one said that all ADs have 
always behaved a certain way.

What is relevant is the /current/ demands on ADs.

> WGLC and IETF LC and the various review teams.

Brian, you missed the point motivating my questions:  I'm suggesting that 
demanding the retention of massive demands on ADs and the imposition of 
significant procedural burden on working groups and authors warrant analysis.

Do the analysis, please.  Make it balanced. Look at pros and cons.  Answer your 
own question, while formulating a serious case for a situation that creates 
chronic problems.  The complaints about the current arrangement are not new or 
isolated.  Unfortunately, the defense of the arrangement is typically in the 
style you are giving.

Besides the skewed perspective -- oh, we caught one error, two years ago, so 
it's worth burdening everyone all the time -- the perspective ignores two key 
points, observed by many folk:

    1.  We have many sources of reviews; the reviewing capabilities of an AD are 
likely to be good, but not better than many others.

    2.  In spite of all the wonderful reviewing done by ADs, errors still get 
through; so the fact that some ADs, sometimes, might catch an important error 
ignores all the others they miss.

Again, Brian:

    Offer a cost/benefit formula and then factor in the known, considerable, 
current costs and demonstrate that we are deriving the requisite benefit.  Try 
to explore real-world behaviors, rather than theoretical hopes.

> Why? My guess is that it's because that the buck stops with the IESG - and

Except that it doesn't stop with the IESG.  (Actually, I don't really know what 
it means to claim it stops with the IESG, and I suspect there are valid claims 
that it stops in a variety of other places.)  Many conditions and actions, over 
a long period of time, are required to make a specification succeed.

It might be interesting for you to document examples of repercussions on the 
IESG for having approved a terrible specification.

At the least, please explain what your assertion means, exactly, and then 
explain what the repercussions are on the IESG when they make the egregious 
error of letting a serious error get published.

This sort of discussion calls for restraint from using comforting, Trumanesque 
catch-phrases.

There is, indeed, a current model that places the personal assessments of ADs on 
a pedestal of power.  It invites mere mortals to work under the myth that they, 
alone, carry the burden of keeping the Internet safe from bogus technologies. 
It's a silly myth, but it's the one that currently in force.

      A more tractable model is that ADs need a manageable workload and a modest 
set of responsibilities, balancing what is demanded of them with the practical 
benefit it provides and the costs to them (and us) that are incurred.

      Further, the model needs to be comfortable with having everyone make lots 
of errors, in spite of their being diligent.

> elswehere, you will just move the problem to the same place.

Perhaps you missed the trigger to this thread, which had to do with processing 
bottlenecks and inappropriate Discusses?

> the main reason that the IESG defined the DISCUSS criteria a few years

Have you seen a pattern of having a Discuss cite the criterion that justifies 
it?  I haven't.  It might be interesting to attempt an audit of Discusses, 
against the criteria...

> rubber stamp *unless* the final responsibility was explicitly moved

Herein lies the real problem:  As with many process and structure discussions in 
the IETF, folk often see only a simplistic, binary choice between whatever they 
prefer, versus something akin to chaos.

The world is more nuanced than that, and the choices more rich.

Here is a small counter-example to your sole alternatives of status quo or 
rubber stamp:

      Imagine a process which requires a range of reviews and requires ADs to 
take note of the reviews and the community support-vs-objection.

      Imagine that the job of the ADs is to assess these and to block proposals 
that have had major, unresolved problems uncovered or that lack support, and to 
approve ones that have support and lack known, major deficiencies as documented 
by the reviews.

That ain't a rubber stamp, Brian.

d/

-- 

   Dave Crocker

---

From: Warren Kumari <[email-masked]>
Date: Thu, 28 Jul 2011 11:22:39 -0400

[ Top posting - meta comment ]

Every now and then I get a bee in my bonnet and decide to carefully review each and every draft in LC. I hate to break it to y'all, but many drafts really poorly written, and, even if you have very broad interests, many of them are going to be really boring to you…

While not all ADs read all drafts, most read a large fraction of them (and read them carefully and thoughtfully enough to catch a number of large issues (and nits) *that were not caught in LC*) -- I think that they deserve recognition for performing a valuable and un-fun function...

Climbing off soapbox,
W

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 28 Jul 2011 08:45:41 -0700

I think the IESG, or its various delegates, do need to review everything, especially keeping in mind that "review" doesn't have to be some big heavyweight thing each time.  I share the same view as others that sometimes some really broken stuff manages to get up to that level.

And, although it can be annoying, I don't agree that the spurious DISCUSS problem is all that bad as long as the AD doing so is responsive to being called on it (and, as Barry mentioned during plenary, that's improved a lot lately).  There are only 15 people that can DISCUSS a document, but those of us producing broken documents seriously outnumber them.

Part of the problem with spurious DISCUSSes is psychological, and that's for a different conversation.  My only other remark there is that ADs could sometimes avoid a lot of heartache for themselves and their working groups with an inquisitive email prior to DISCUSSing something.
 
I have heard more than once during this meeting some hallway track chatter about ADs wishing they had invoked their available review resources (apps-review, the various directorates, etc.) earlier in the process.  So perhaps what's needed is an optional document state prior to Publication Requested called Review Requested, which triggers an early, informal review action by the AD and/or her/his review team prior to the formal start of the publication process.  Working group chairs would have to understand that it's an expensive request (i.e., not to be used frivolously), but still cheaper than letting a real disaster of a draft get up to the AD or the full IESG.

-MSK

---

From: Lou Berger <[email-masked]>
Date: Thu, 28 Jul 2011 12:48:54 -0400

+1

On 7/28/2011 11:22 AM, Warren Kumari wrote:
>...
> While not all ADs read all drafts, most read a large fraction of them
> (and read them carefully and thoughtfully enough to catch a number of
> large issues (and nits) *that were not caught in LC*) -- I think that
> they deserve recognition for performing a valuable and un-fun
> function...
> ...

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 29 Jul 2011 11:22:47 +1200

Dave, we are shouting past each other so I will not repeat myself
on all points. However,

...
> Have you seen a pattern of having a Discuss cite the criterion that
> justifies it?  I haven't.  It might be interesting to attempt an audit
> of Discusses, against the criteria...

It might, and at the time when the IESG had a large backlog of unresolved
DISCUSSes and the current criteria were being developed (I'm talking
about 2005/2006), the IESG did indeed end up looking at all the old
DISCUSSes against the criteria, and held dedicated conference calls
to talk through many of those DISCUSSes and in many cases persuade the AD
concerned to drop them, or rewrite them in an actionable format. In my
recollection, Allison Mankin was the leader of the charge on this.

But these are all judgment calls, so I don't think there can be an
objective audit. There could be an audit of how many DISCUSSes take
more than N months to clear, or something like that. There were tools
around for that some years ago, but I don't know if they exist for the
modern version of the tracker.

> Herein lies the real problem:  As with many process and structure
> discussions in the IETF, folk often see only a simplistic, binary choice
> between whatever they prefer, versus something akin to chaos.
> 
> Here is a small counter-example to your sole alternatives of status quo
> or rubber stamp:
> 
>      Imagine a process which requires a range of reviews and requires
> ADs to take note of the reviews and the community support-vs-objection.
> 
>      Imagine that the job of the ADs is to assess these and to block
> proposals that have had major, unresolved problems uncovered or that
> lack support, and to approve ones that have support and lack known,
> major deficiencies as documented by the reviews.

The only difference between that and what I see happening today is that
the ADs actually verify the reviews by looking at the drafts themselves.
And why do they do that? Because they aren't going to take the
responsibility for approving a document that they haven't read. Nobody
would, I hope.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 29 Jul 2011 11:24:49 +1200

>>    Internet community consensus or recommendation.  [...]

Er, no. By definition, it's correct until we update RFC 2026.

    Brian

---

From: SM <[email-masked]>
Date: Thu, 28 Jul 2011 16:54:45 -0700

Hi Brian,
>Er, no. By definition, it's correct until we update RFC 2026.

Quoting the Status of this memo section from RFC 6305, RFC 6308, RFC 
6319 and RFC 6331 which are Informational and from the IETF Stream:

   "This document is a product of the Internet Engineering Task Force
    (IETF).  It represents the consensus of the IETF community.  It has
    received public review and has been approved for publication by
    the Internet Engineering Steering Group (IESG)."

Although it is, by definition, correct until RFC 2026 is updated, RFC 
5741 is currently being used for the boilerplate in RFCs.

Regards,
-sm

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 28 Jul 2011 21:53:12 -0400

Brian,

I did not ask you to repeat anything -- and don't want you to.

Rather, I asked you to move beyond cliche's and personal anecdotes into 
consideration of tradeoffs.  That is, I pointedly asked you /not/ to repeat 
yourself.

Please engage in the substance of such balanced analysis comparing benefits 
against costs.  It's not as if that's an unusual approach for evaluating 
expensive activities of questioned value...

> DISCUSSes against the criteria, and held dedicated conference calls

I do not recall seeing this analysis made public.  The point behind my 
suggestion was to prmit transparent consideration of the use of Discuss. So 
whatever was done, it was not transparent to the community.

Further, my suggestion was for /current/ patterns, not in the past.

> objective audit.

I don't recall requiring it to be "objective".  In fact, audits are often 
subjective.  That's ok as long as:

    a) the auditor gives thought to using reasonable criteria

    b) the criteria are made public

    c) they are applied consistently

Let's try to refrain from throwing up artificial barriers, as an excuse not to 
hold the process accountable.

> There could be an audit of how many DISCUSSes take

That's nice, but not all that useful.  In contrast, looking at the substance of 
Discusses against the criteria that are supposed to be used for justifying them 
is directly relevant.

(Note that you replaced a focus on core substance and clear import, with 
something superficial and with a very ambiguous semantic.  In particular, 
longer-vs-shorter holding times have no obvious implication about the 
/appropriateness/ of the Discusses.)

> the ADs actually verify the reviews by looking at the drafts themselves.

You are factually wrong.  AD's do their own reviews.  ADs formulate their own 
lists of issues and requirements.  AD's assert them as the basis for a Discuss.

They often do pay attention to other reviews -- sometimes quite mechanically, 
rather than from an informed basis -- but the exemplar I put forward is a 
fundamentally different model of AD behavior and responsibility.  I can't guess 
how you can misunderstand the difference.

> responsibility for approving a document that they haven't read. Nobody

Therein lies a core problem with the model:  It hinges on personal investment by 
the AD, and a lack of trust in the community process, using the excuse that the 
process is not perfect, as if the AD's own evaluation process is...

d/
-- 

   Dave Crocker

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 29 Jul 2011 14:38:43 +1200

Dave,

On 2011-07-29 13:53, Dave CROCKER wrote:
> consideration of tradeoffs.  That is, I pointedly asked you /not/ to
> repeat yourself.
> 
> Please engage in the substance of such balanced analysis comparing
> benefits against costs.  It's not as if that's an unusual approach for
> evaluating expensive activities of questioned value...

May argument is that if we move the cost (that of final review prior
to committing the text to the RFC Editor), we do not remove the
problem (of the final review being exhaustive and time consuming),
unless we also lower the implied standard of quality.

To my mind it's implicit in RFC 3935 that the present quality standard
is not lowered - especially, that in addition to verifying that
a specification is valid in itself, we verify that it does not damage
other aspects of the Internet. That makes final review by a cross-area
team essential, and today that team is the IESG.

I wouldn't mind moving the responsibility to some other cross-area
team, but it seems like a zero-sum game to me.

> I do not recall seeing this analysis made public.

It was a dynamic process, but I wouldn't be surprised if Allison or
Bill Fenner or somebody did post data at some stage; but that's ancient
history now.

The point behind my
> suggestion was to prmit transparent consideration of the use of Discuss.
> So whatever was done, it was not transparent to the community.
> 

And all the data is in the tracker these days; it's very easy to
find the DISCUSSes against any draft.

> I don't recall requiring it to be "objective".  In fact, audits are
> often subjective.  That's ok as long as:
> 

That is the judgment part.

> 
> Let's try to refrain from throwing up artificial barriers, as an excuse
> not to hold the process accountable.
> 
> 
> That's nice, but not all that useful.  In contrast, looking at the
> substance of Discusses against the criteria that are supposed to be used
> for justifying them is directly relevant.
> 
> something superficial and with a very ambiguous semantic.  In
> particular, longer-vs-shorter holding times have no obvious implication
> about the /appropriateness/ of the Discusses.)

I think that a long holding time clearly implies one of two causes:

1. A fumble by the shepherd; in fact it was to avoid this sort of problem that
   the shepherding process was carefully defined. Again, credit to Allison.

2. A failed dialogue between the DISCUSSing AD and the authors and/or WG.
   I would assert that most of those are due to non-actionable DISCUSSes.
   There was a lot of analysis of real cases of serious delay behind the
   section on "DISCUSS non-criteria" in
   http://www.ietf.org/iesg/statement/discuss-criteria.html

The fact that we're having this conversation several years later
tells me that human nature hasn't changed much recently. ADs *do*
need to look at those criteria and it's absolutely in order for the
community to complain when they don't.

> You are factually wrong.  AD's do their own reviews.  ADs formulate
> their own lists of issues and requirements.  AD's assert them as the
> basis for a Discuss.

I didn't mean to imply that they (all) don't. But the criteria for
a comment being a DISCUSS are on public record.

> They often do pay attention to other reviews -- sometimes quite
> mechanically, rather than from an informed basis -- but the exemplar I
> put forward is a fundamentally different model of AD behavior and
> responsibility.  I can't guess how you can misunderstand the difference.

I don't think we differ on the responsibility that most ADs feel when
ballotting on a draft. I'm sure there is a significant variety of
"personal algorithms" by which ADs operate.

> Therein lies a core problem with the model:  It hinges on personal
> investment by the AD, 

 ... by whoever is making the final determination ...

> and a lack of trust in the community process,
> using the excuse that the process is not perfect, as if the AD's own
> evaluation process is...

Nobody, surely, is claiming perfection? What I am claiming is that
knowing one has final responsibility makes a person strive for
something closer to perfection - which means noticing defects in
a draft that nobody else has noticed.

Enough on this thread from me.

Regards

     Brian

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 28 Jul 2011 23:49:36 -0400

Dear colleagues,

This is not to pick on Murray, who was not making the point I am
trying to draw out of his remarks.  Sorry, Murray.

On Thu, Jul 28, 2011 at 08:45:41AM -0700, Murray S. Kucherawy wrote:
  
> the process.  So perhaps what's needed is an optional document state
> prior to Publication Requested called Review Requested,

I hereby propose that every single I-D adopted by every working group
go into Review Requested state.  Oh.  Right.

In my opinion, this thread would be a lot better motivated if we (and
I emphatically include myself) did a lot more review, and did it a lot
more carefully.  As a WG chair, I have faced a number of occasions in
which plain embarrassing flaws in documents made it past my own review
(which means "through WGLC") and up to the IESG.  As a participant, I
have noticed more than once that my failure to review something I
should have made it all the way to the IESG before someone said, "Uh,
hey, there's an issue."

Once frivolous DISCUSSes way outnumber serious ones, I'll feel like
there's a problem.  We're all busy, and by the grace of some power who
doesn't explain itself to me some employers are willing to subsidize
the Bulwark Panel of Final Review.  Given my own experience of what I
regard as mistaken or frivolous DISCUSSes, when every document has 5
or 6 really high quality reviews both in WGLC and IETF LC, I'll feel
that we can start throwing stones.  But today, not so much: most
DISCUSSes are in fact attempts to discuss unclear or -- sometimes --
seriously mistaken parts of drafts.  Maybe in others' corner of the
IETF it's different, but I feel that I need to pick up my end of the
log before telling other people to haul their end.

Best,

A
-- 
Andrew Sullivan
[email-masked]

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 28 Jul 2011 23:54:03 -0400

I think I understand your request taht we move beyond personal 
anecdotes.  In principle, I even agree with you.

However, when I try to act on that, I am stumped.  I do not see any way 
to evaluate say the last 2 years discusses to determine whether they 
were more harmful or more helpful.
Any such evaluation would have to icnlude the various area reviews which 
occur during last call, but which are supported by the possibility of a 
discuss if they are ignored and the AD decides they matter.
Yes, I have a gut feel on the matter.  So does everyone in the discussion.

But you asked that we go beyond that.  How?

Yours,
Joel

On 7/28/2011 9:53 PM, Dave CROCKER wrote:
> You are factually wrong. AD's do their own reviews. ADs formulate their
> own lists of issues and requirements. AD's assert them as the basis for
> a Discuss.
>
> investment by the AD, and a lack of trust in the community process,

---

From: Eliot Lear <[email-masked]>
Date: Fri, 29 Jul 2011 04:46:14 -0400

I don't have too much to say on whether the IESG is effective.  Our
standards production rate and the market uptake of same seems to speak
for itself.  I also don't have the numbers Dave is looking for either.

However, I would like to contribute my own anecdotal experience,
involving at least one draft from the person who kicked off this debate
(not you Dave or Brian).  I reviewed on behalf of the apps-area one of
the drafts he wrote and found a serious concern, and reported it.  He
ignored my review at his peril, and then complained when he got slapped
with a DISCUSS.  I don't know whether the AD based his DISCUSS on my
review or not, but clearly there was, at the very least, a substantial
disagreement.  What's particular egregious is that here we had a
cross-area expert review process that identified a problem that could
have been resolved without him wasting IESG time.  Instead, we are all
incompetent, and don't know what we're talking about, and the process is
broken.  It has not occurred to this guy that he might have made an
error in judgment.  In fact he made many.

Eliot

---

From: Dave CROCKER <[email-masked]>
Date: Fri, 29 Jul 2011 08:19:42 -0400

On 7/28/2011 7:54 PM, SM wrote:
> Quoting the Status of this memo section from RFC 6305, RFC 6308, RFC 6319 and
> RFC 6331 which are Informational and from the IETF Stream:
>
> (IETF). It represents the consensus of the IETF community.

SM is correctly detecting some problematic inconsistencies in our language.

We do not use adequately clear and distinctive language to distinguish between:

    *  IETF consensus used to approve standards/bcp,

    *  IETF consensus used to approve other documents through the IETF

    *  Independent Informational and Experimental submissions that are published 
/without/ IETF approval.

d/

-- 

   Dave Crocker

---

From: John Leslie <[email-masked]>
Date: Fri, 29 Jul 2011 08:22:07 -0400

> 5741 is currently being used for the boilerplate in RFCs.

   This actually caught me by surprise, but it is accurate.

   RFC 5741, status Informational, specifies boilerplate for
Informational RFCs. The operative text is in Section 3.2.2:
" 
" The text that follows is stream dependent -- these are suggested
" initial values and may be updated by stream definition document
" updates.
" 
" IETF Stream:
"    "This document is a product of the Internet Engineering Task Force
"    (IETF)."
" 
"    If there has been an IETF consensus call per IETF process, an
"    additional sentence should be added:
" 
"       "It represents the consensus of the IETF community.  It has
"       received public review and has been approved for publication by
"       the Internet Engineering Steering Group (IESG)."
" 
"    If there has not been such a consensus call, then this simply
"    reads:
" 
"       "It has been approved for publication by the Internet
"       Engineering Steering Group (IESG)."

   Frankly, I am not happy with this. An Informational track document
has effectively over-ruled RFC 2026, a BCP status document -- and
gone on to say that RFC 2026 may be further overruled by "stream
definition document updates" (whatever that may mean).

   I _much_ prefer the RFC 2026 definition of Informational (and I
don't believe they should require extensive IESG scrutiny).

   (BTW, I wonder to what extent our current repetition of the
argument about the IESG filing too many DISCUSSes is in reaction to
their scrutiny of Informational track documents.)

   I don't have time today to research to what extent Informational
track RFCs have actually received "an IETF consensus call per IETF
process". Perhaps somebody else would like to respond on that...

--
John Leslie <[email-masked]>

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 30 Jul 2011 08:17:44 -0400

Joel, et al,

On 7/28/2011 11:54 PM, Joel M. Halpern wrote:
> I think I understand your request taht we move beyond personal anecdotes. In
> principle, I even agree with you.
>
> However, when I try to act on that, I am stumped. I do not see any way to
> evaluate say the last 2 years discusses to determine whether they were more
> harmful or more helpful.

1. Pursue serious discussions that consider criteria for determining that.  A 
variety of obvious candidates for this list come to mind readily; for example:

    *  Does a review of the Discuss now produce a sense of its worth?  That is, 
do the rest of us, looking at the Discuss generally feel that it was clear and 
substantive and even important?

    *  Does the Discuss obviously align with the criteria for a Discuss?

    *  Does the AD who held the Discuss now feel that it was productive, and why?

    *  Do the authors and chairs now feel that it was productive, and why?

    *  How much delay did the Discuss introduce?  Was it timely?

    *  How much effort did it take to resolve the discuss?

    *  Did the Discuss produce a change in normative text?

    *  Does a review of the change in text, now, produce a sense of its worth?
    ...

It could well be worth trying to classify typical types of Discusses, such as:

    *  Curiosity questions.  "I'd like to understand..." where no actual problem 
is cited

    *  Review enforcement.  Ensuring that particular points from other reviews 
are resolved

    *  Personal preference.  AD dislike of a working group choice, without a 
clear basis the makes clear why the wg choice won't work or is otherwise impractical

    *  Fundamental problem.  Assertion of a basic limitation to the 
specification that renders it inherently problematic

    *  Scope.  The Discuss appears to be out of scope for the document.  For 
example, a document intended for Draft is being challenged on basic issues that 
were approved when it went to Proposed.

    *  Unclear.  The meaning of the Discuss is unclear and/or the path towards 
resolution cannot be understood because the Discuss offers no clear criteria for 
resolution.

       To illustrate this last item with examples that almost certainly have 
never happened:

       o  If a Discuss said "needs to be written in an active voice rather than 
passive", it would be clear what would resolve it.

       o  If a Discuss said "needs to be shorter", there's a clear direction for 
resolution, but no criteria for for knowing when it is short enough.

       o  If a Discuss said "Algorithm X in the specification will not work for 
the following reasons...; Algorithm Y is better for the following reasons" then 
it is clear what the problem is, why it is a problem and exactly what will 
satisfy the AD to resolve it.

    ...

It can also be worth looking at meta-issues.  For example, do documents 
typically get a Discuss?  That is, is it typical to block progress for a 
document?  Does it make sense to have working groups proceed for many months and 
then have presumably basic issues raised at the very last moment?

2. Review the datatracker histories over the last <x> years, sampling enough of 
the documents to get a sense of the overall pattern.  (The new datatracker now 
supports provides a complete history for the handling of drafts; massive kudos 
to the tools team!)

3. Talk with authors, chairs and ADs about previous Discusses.

> But you asked that we go beyond that. How?

The above is meant to prime a serious discussion, but the first requirement is a 
willingness to engage in that discussion and to pursue it seriously.

d/
-- 

   Dave Crocker