Subject: Purpose of IESG Review

From: Joe Touch <[email-masked]>
Date: Thu, 11 Apr 2013 10:54:05 -0700

Hi, all,

As an author who has had (and has) multiple documents in IESG review, 
I've noticed an increasing trend of this step to go beyond (IMO) its 
documented and original intent (BCP 9, currently RFC 2026):

    The IESG shall determine whether or not a specification submitted to
    it according to section 6.1.1 satisfies the applicable criteria for
    the recommended action (see sections 4.1 and 4.2), and shall in
    addition determine whether or not the technical quality and clarity
    of the specification is consistent with that expected for the
    maturity level to which the specification is recommended.

Although I appreciate that IESG members are often overloaded, and the 
IESG Review step is often the first time many see these documents, I 
believe they should be expected to more clearly differentiate their 
"IESG Review" (based on the above criteria) - and its accompanying 
Position ballot, with their personal review.

My concern is that by conflating their IESG position with their personal 
review, the document process is inappropriately delayed and that 
documents are modified to appease a small community that does not 
justify its position as representative.

How do others feel about this?

Joe

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 11 Apr 2013 11:55:07 -0700

On Apr 11, 2013, at 10:54 AM, Joe Touch <[email-masked]> wrote:

> As an author who has had (and has) multiple documents in IESG review, I've noticed an increasing trend of this step to go beyond (IMO) its documented and original intent (BCP 9, currently RFC 2026):
> 
> Although I appreciate that IESG members are often overloaded, and the IESG Review step is often the first time many see these documents, I believe they should be expected to more clearly differentiate their "IESG Review" (based on the above criteria) - and its accompanying Position ballot, with their personal review.
> 
> My concern is that by conflating their IESG position with their personal review, the document process is inappropriately delayed and that documents are modified to appease a small community that does not justify its position as representative.
> 

That it is too vague to comment on?

Please point to specific examples where you feel an IESG member's review went beyond determining the technical quality or clarity of the specification. That would help make the sure-to-be ensuing flamefest more light-filled.

--Paul Hoffman

---

From: <[email-masked]>
Date: Thu, 11 Apr 2013 21:39:48 +0100

+1 to Joe's comment.

Example: the existence of the extensibility bit in multipath tcp, which i understand came out of a review by the iesg member responsible for security.
In that context, that would be outside the scope of any security review, and the comments weren't raised in a personal capacity years earlier on the relevant mailing list.

Sure, getting past iesg only cost multipath tcp a bit. But iesg members exceeding their bounds as reviewers and leaving a personal mark seems commonplace. iesg members are there for expertise in their area and to provide that expertise in focused reviews, not to block until a protocol is redesigned to suit their personal tastes. (That transport expertise is lacking on iesg last I looked and everyone believes they're an expert in transport  - 'hey, why can't we just turn off udp checksums for sctp over udp? It's faster!' - another iesg redesign attempt overriding considered expertise of a workgroup - isn't helping here.)

There are two examples I know of, off the top of my head, telated to transport because that's my area of interest. Can others provide further examples?

Lloyd Wood
http://sat-net.com/L.Wood/

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Paul Hoffman [[email-masked]]
Sent: 11 April 2013 19:55
To: Joe Touch
Cc: IETF discussion list
Subject: Re: Purpose of IESG Review

--Paul Hoffman

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 11 Apr 2013 22:17:36 +0100

Hi Ian,

Examples are useful because they give the IESG something to chew on. If you
don't call us when we do "bad stuff" we might never know.

Examples can be dangerous because we can rat-hole into the specific rather than
the general, but i would like to use your example as data point to get some more
information that can possibly be generalised.

> Example: the existence of the extensibility bit in multipath tcp, which i
> understand came out of a review by the iesg member responsible for security.
> In that context, that would be outside the scope of any security review, and
the
> comments weren't raised in a personal capacity years earlier on the relevant
> mailing list.

Do people believe that the extensibility bit should not have been added?
I.e., is it a dumb or unnecessary thing that was forced on the community by the
IESG?
Maybe it is "harmless" so everyone said yes to get the document published.
Maybe the IESG had some "insight" suggesting that it was important to
future-proof the protocol even though the community didn't think it important.
Maybe it was a really cool idea that everyone missed.

I'm not asking in order to have a fight about whether the AD was right or wrong.
But I would like to understand more about the impact of this type of "blocking"
Discuss.

My point here is that sometimes it happens that ADs spot holes in protocols. In
those cases, I think we could live with publishing the RFCs and fixing them
later, but it is on the whole better to fix the issues at the time they are
spotted rather than later. The problem comes that an AD (especially out of
expertise) cannot tell between "I think there might be a problem here" and "I
know this is a serious bug".

I believe that the clue is in the word "Discuss" and I hope that I (at least) am
willing to listen when the response is "we thought about it, it's not a big
problem." Perhaps people will tell me I am not so good at listening! I know
other ADs also strive to that as an ideal, so perhaps we have something of a
communications breakdown...

Discuss is not meant to mean "and you shall not move unless you do exactly what
I say." It should mean "Please help me to be happy about publishing this
document because what I really want is you to publish the best possible document
as quickly as possible."

Thanks,
Adrian

> 
> Sure, getting past iesg only cost multipath tcp a bit. But iesg members
exceeding
> their bounds as reviewers and leaving a personal mark seems commonplace. iesg
> members are there for expertise in their area and to provide that expertise in
> focused reviews, not to block until a protocol is redesigned to suit their
personal
> tastes. (That transport expertise is lacking on iesg last I looked and
everyone
> believes they're an expert in transport  - 'hey, why can't we just turn off
udp
> checksums for sctp over udp? It's faster!' - another iesg redesign attempt
> overriding considered expertise of a workgroup - isn't helping here.)
> 
> There are two examples I know of, off the top of my head, telated to transport
> because that's my area of interest. Can others provide further examples?
> 
> Lloyd Wood
> > As an author who has had (and has) multiple documents in IESG review, I've
> noticed an increasing trend of this step to go beyond (IMO) its documented and
> original intent (BCP 9, currently RFC 2026):
> >
> > Although I appreciate that IESG members are often overloaded, and the IESG
> Review step is often the first time many see these documents, I believe they
> should be expected to more clearly differentiate their "IESG Review" (based on
> the above criteria) - and its accompanying Position ballot, with their
personal
> review.
review,
> the document process is inappropriately delayed and that documents are
> modified to appease a small community that does not justify its position as
> representative.
> >
> Please point to specific examples where you feel an IESG member's review went
> beyond determining the technical quality or clarity of the specification. That
> would help make the sure-to-be ensuing flamefest more light-filled.
> 
> --Paul Hoffman

---

From: John Leslie <[email-masked]>
Date: Thu, 11 Apr 2013 18:08:01 -0400

[email-masked] <[email-masked]> wrote:
> 
> +1 to Joe's comment.
> 
> Example: the existence of the extensibility bit in multipath tcp,
> which i understand came out of a review by the iesg member responsible
> for security.

   I assume you're talking RFC 6824. I recommend reading the Narrative
Minutes of September 13th:

http://www.ietf.org/iesg/minutes/2012/narrative-minutes-2012-09-13.html

There were DISCUSSes from Stephen Farrell (Security), Barry Leiba
(Applications), Robert Sparks (RAI), and Sean Turner (Security).
Stephen Farrell did complain about a negotiation scheme that only allowed
seven security algorithms; and asked "how you could practically extend
this design for stronger cryptographic security".

> In that context, that would be outside the scope of any security
> review,

   Hardly! Those are exactly what I would hope for in a Security review.

> and the comments weren't raised in a personal capacity years earlier
> on the relevant mailing list.

   I'm not going to research that; but it seems hardly relevant...

> Sure, getting past iesg only cost multipath tcp a bit.

   (which, BTW, I strongly endorse!)

> But iesg members exceeding their bounds as reviewers and leaving a
> personal mark seems commonplace.

   Perception is easily mistaken for reality. :^(

   But if you look at the datatracker history:

http://datatracker.ietf.org/doc/draft-ietf-mptcp-multiaddressed/history/

you'll see that all four DISCUSSes were cleared by October 22.

   Considering that this document started life in June of 2010, and
was a major enhancement of TCP, 40 days doesn't seem excessive, IMHO.

> iesg members are there for expertise in their area and to provide that
> expertise in focused reviews,

   Note that there's really a lot of overlap between areas: so "focused"
may not be the right criterion.

> not to block until a protocol is redesigned to suit their personal
> tastes.

   I am told this used to happen. I have not experienced it in the five
years I have been scribing.

====

   I really don't know how to change the perception -- but I strongly
recommend referring to the Narrative Minutes. Hopefully that history
will be preserved "forever".

--
John Leslie <[email-masked]>

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 11 Apr 2013 23:16:11 +0100

And that should, of course, have read "Hi Lloyd"
Sorry about that, Lloyd.
The rest of the message still stands.
Adrian

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Adrian
> Farrel
> Sent: 11 April 2013 22:18
> To: [email-masked]
> Cc: [email-masked]
> Subject: RE: Purpose of IESG Review
> 
> Hi Ian,
> 
> > From: [email-masked] [[email-masked]] On Behalf Of Paul
> Hoffman
> > [[email-masked]]

---

From: Joe Touch <[email-masked]>
Date: Thu, 11 Apr 2013 15:41:18 -0700

> Please point to specific examples where you feel an IESG member's review went beyond determining the technical quality or clarity of the specification. That would help make the sure-to-be ensuing f…

I've already done that within the IESG; the point of the message wasn't 
to have dozens of opinions of whether my feedback was beyond scope, but 
whether others were getting that feeling as well. At least first...

Joe

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Fri, 12 Apr 2013 00:27:42 +0000

In my opinion, some individual ADs seem to, from their behavior, feel that they have not done their jobs unless they have raised a "discuss". The one that took the cake for me personally was a "discuss" raised by a particular AD (who shall remain nameless) that in essence wondered what he should raise a "discuss" about in my document. There were a couple of errors in that; he told me later that what he had done was opened the comment tool and typed that question, and subsequently accidentally hit the equivalent of "send" - the question wasn't intended to go out. But the question itself is telling: the issue was not "does the document meet the requirements of BCP 9", it was "what comment shall I raise"?

Also, in my opinion, IESG review that raises a certain number of issues should not result in the document sitting in the IESG's queue for a few months while the authors go back and forth with the AD or the GEN-ART reviewer pounding the document into someone's idea of "shape" without working group involvement. I personally would prefer that simple matters get sorted out between the ADs and the author, but complex changes or additional content requested by the AD should result in the document being sent back to the working group.

---

From: John Leslie <[email-masked]>
Date: Fri, 12 Apr 2013 02:10:18 -0400

Fred Baker (fred) <[email-masked]> wrote:
> 
> Also, in my opinion, IESG review that raises a certain number of issues
> should not result in the document sitting in the IESG's queue for a
> few months while the authors go back and forth with the AD or the
> GEN-ART reviewer pounding the document into someone's idea of "shape"
> without working group involvement. I personally would prefer that
> simple matters get sorted out between the ADs and the author, but
> complex changes or additional content requested by the AD should
> result in the document being sent back to the working group.

   This is pretty much the call of Working Group Chairs, so perhaps you
should ask WGCs what they intend to do -- I don't think an IETF-wide
policy would work well here.

   Note also that when this is done, it seems to reinforce the attitude
that "We should humor the IESG!"

   (Even so, I personally prefer to hear about such changes. ;^)

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 12 Apr 2013 08:13:30 +0100

Seeing randomly selected drafts as a Gen-ART reviewer, I can
say that serious defects quite often survive WG review and
sometimes survive IETF Last Call review, so the final review
by the IESG does serve a purpose.

IMHO, if the IESG members sticks to their own criteria at
http://www.ietf.org/iesg/statement/discuss-criteria.html,
i.e. do not DISCUSS a document for spurious reasons, they
are doing just fine. If they don't stick to those criteria,
complaint is justified.

Of course this will always be a matter of judgment.

Regards
   Brian

---

From: Dave Crocker <[email-masked]>
Date: Fri, 12 Apr 2013 05:22:49 -0700

> by the IESG does serve a purpose.

Brian,

Of course it "serves" a purpose.  The lack of perfection in the 
specifications that reach the IESG has always been the justification for 
the current review model.

But what is tiring about this line of justification is its continuing 
failure to balance the analysis by looking at the costs and problems 
that come with it.  Does it provide regular and sufficient benefit to 
justify its considerable costs?

First, it is a phenomenally expensive step, with a very damaging 
organizational cost:  the time and effort burden put on ADs makes the 
pool of available ADs tiny, including dramatically reducing the range of 
companies that can afford to donate senior (strategic) staff to it.

Along with this is the general issue that has triggered Joe's query: 
the tendency of ADs to inject spontaneous, late-stage personal 
engineering preferences, which might have been reasonable to add to the 
original working group discussion but realistically have no place in a 
final review.  It's not that the preferences are necessarily 
inappropriate, it's that they typically are not essential to the success 
of the spec.

But in terms of the specific logic you've used, the presumption of the 
"it catches errors sometimes" language is that the final output that is 
the result is somehow perfect.  Of course, that's a silly assumption. 
But as soon as one acknowledges this, we are left with a model that must 
class this as merely one more review in an imperfect sequence, with the 
likelihood that an every new review will find more problems.  Or we are 
left with the view that pragmatics dictate accepting imperfections 
because each step of the quality assurance model -- of which this is a 
part -- really does need a strict cost/benefit analysis.  This needs to 
be done in terms of trade-offs, rather than an isolated approach that 
counts the detection of an occasional problem as somehow an inherent and 
controlling good.

An essential component to such a trade-off based model is recognizing 
that the ultimate quality assurance step always has been, and remains, 
the market.  No amount of IETF q/a guarantees success.  We have lots of 
failures and we won't ever get to zero.

If the IESG review step really is essential, then we should show a 
consistent pattern of its finding /essential/ errors that would have 
caused failure in the field.

But if we can find that -- which I believe we cannot -- we have deeper 
problems, of course, because that sort of shit needs to be found mch, 
much sooner.

At base, the IESG review seeks to compensate for seriously inadequate 
quality control during the development process...

d/
-- 
  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Fri, 12 Apr 2013 13:17:18 +0000

> by the IESG does serve a purpose.

I'm not saying it doesn't serve a purpose. I'm saying that I know of drafts that have been nearly rewritten during such back-and-forth, so what popped out was largely unrelated to what went in. In such cases, I think the document should have been returned to the working group with comments, not worked on privately.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 12 Apr 2013 14:50:57 +0100

> I'm not saying it doesn't serve a purpose. I'm saying that I know of drafts that have been nearly rewritten during such back-and-forth, so what popped out was largely unrelated to what went in. In…

I agree. That should be standard operating procedure.

   Brian

---

From: [email-masked] (Martin Rex)
Date: Fri, 12 Apr 2013 17:26:17 +0200 (CEST)

> by the IESG does serve a purpose.

I'm currently seeing a document with some serious defects in
IETF Last Call (rfc2560bis) and an apparent desire to have
it Rubberstamped by the IESG (recycling at Proposed Standard).

While that seems procedurally permitted for the IESG to do such
rubberstamping (aka "waive") for a proposed standard
(rfc2026, last paragraph on page 12):

   http://tools.ietf.org/html/rfc2026#page-12

   A Proposed Standard should have no known technical omissions with
   respect to the requirements placed upon it.  However, the IESG may
   waive this requirement in order to allow a specification to advance
   to the Proposed Standard state when it is considered to be useful and
   necessary (and timely) even with known technical omissions.

one should really consider the consequences of such a decision,
especially for -bis and -ter documents.

The original draft (and now full) standard requires that such defects
are removed _before_ the document is advanced.  So when "waiving" known
defects/omissions (in particular obvious and formally provable ones),
this means that there will have to be a ter document produced where
this defects are fixed and this document be recycled at Proposed
one more time before the standard can progress on the maturity level.

It turns out that for a non-negligible amount of the defects there
is a constituency that want the defect to be retained rather than
fixed, and they expect the IESG to waive defects on -bis, -ter etc.
documents whenever it was previously accepted for Proposed with
that defect.

> > justify its position as representative.

What do you want to see the IESG do instead?

Simply Rubberstamp WG consensus?  That would turn the whole IESG
diversity discussion into an absurd waste of time...

-Martin

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 12 Apr 2013 17:21:57 +0100

I have no interest in or knowledge of the technical details,
but there is a pretty complicated DISCUSS against this draft,
which doesn't look like rubber-stamping to me:
https://datatracker.ietf.org/doc/draft-ietf-pkix-rfc2560bis/ballot/

I assume you've already let the IESG know about the defects you're
concerned about.

  Brian

---

From: Ted Lemon <[email-masked]>
Date: Fri, 12 Apr 2013 16:31:09 +0000

> it Rubberstamped by the IESG (recycling at Proposed Standard).

FWIW, I raised the same question during IESG review.   It didn't seem like a "serious defect" to me, but it did seem like a strange design choice.   I ultimately withdrew my objection after we walked through the problem.   If we were doing a clean slate design, fixing the problem as you proposed would be a net win.   Given that there are substantial existing deployments, it doesn't look like a net win to me.

I think this is really a matter of judgment, not a matter of fact, so while I sympathize that it didn't come out your way, I don't agree with you that the wrong thing happened.

---

From: [email-masked] (Martin Rex)
Date: Fri, 12 Apr 2013 20:04:17 +0200 (CEST)

> I assume you've already let the IESG know about the defects you're

The concern I originally raised durint LC is primarily a defect of rfc5019
that rfc2560bis does not properly deal with, and is in theory fairly
trivial to fix.

A much more concerning defect has just recently been uncovered:
  http://www.ietf.org/mail-archive/web/pkix/current/msg32635.html

the complete absence of requirements to allow OCSP response caching
(and OCSP response cache updating),  which is going to become important
for the two work items that are currently active in TLS:
  - a TLS extension for multiple OCSP response stapling
  - a TLS caching extension

and I strongly believe that PKIX needs to fix that defect/omission
and clarify the OCSP response caching semantics in rfc2560bis
(and document requirements for "thisUpdate"), or this will result
in unexpected and undesirable behaviour (aka interop problems).

I have no implementation experience with OCSP, so a number of problems
will not be directly apparent to me.  But I'm really wondering why noone
who has implemented OCSP response caching so far (and I believe there
are implementations) has ever brought up this issue against rfc2560
so far.  This is close to impossible to miss _for_an_implementor_.  

-Martin

---

From: Arturo Servin <[email-masked]>
Date: Fri, 12 Apr 2013 16:28:54 -0300

Not answering any particular post. Just a comment.

	The IESG should be there to attest that the IETF procedure was followed
and the document reached consensus in the WG and in the IETF LC and it
was successfully reviewed by the Gen-ART. If it wasn't then this
particular process should be reviewed and take actions accordingly (e.g.
returning the document to the wg).

	But if a single individual of the IESG can technically challenge and
change the work of a whole WG and the IETF, then we have something wrong
in our process because that means that the document had a serious
problem and we didn't spot it in the process or an IESG member is using
its power to change a document according to its personal beliefs.

Just a thought,
as

---

From: Melinda Shore <[email-masked]>
Date: Fri, 12 Apr 2013 11:32:42 -0800

> its power to change a document according to its personal beliefs.

We've had that happen in a working group I chair and I do
think that it was the result of process failure in the
working group.  That said, there seemed to be broad
agreement across the IESG that the document had certain
specific flaws - I do think there's a problem if a single
IESG member can reject working group consensus and hold
up a document - it seems as if there should be wider
agreement that the document is too flawed to progress.

Melinda

---

From: Arturo Servin <[email-masked]>
Date: Fri, 12 Apr 2013 17:22:31 -0300

On 4/12/13 4:32 PM, Melinda Shore wrote:
> specific flaws 

	I think that exemplifies well what the IESG should do.

- I do think there's a problem if a single
> agreement that the document is too flawed to progress.
	Yes.

Best wishes,
as

---

From: t.p. <[email-masked]>
Date: Fri, 12 Apr 2013 21:52:20 +0100

----- Original Message -----
From: "Arturo Servin" <[email-masked]>
To: <[email-masked]>
Sent: Friday, April 12, 2013 8:28 PM
> its power to change a document according to its personal beliefs.

My experience has been the former, where the IESG has raised concerns
about arcane topics, such as security and congestion, of which the WG
had limited expertise.  This might be caught by a directorate review,
but those seem patchy; it might be caught by IETF Last Call, but if you
are an expert at an arcane topic then you are probably too busy to
follow them.

So I do see the IESG DISCUSSing, when it would have been lovely to have
had, but it is hard to see quite how, it resolved earlier.  We just do
not have the breadth of knowledge of arcane topics.

Tom Petch

---

From: Arturo Servin <[email-masked]>
Date: Fri, 12 Apr 2013 18:32:36 -0300

> not have the breadth of knowledge of arcane topics.

	Perhaps we need an "arcane topic reviewer" before the LC.

.as

---

From: <[email-masked]>
Date: Sat, 13 Apr 2013 03:46:38 +0100

If you think security and congestion are arcane, you have... problems.

This was an actual ietf working geoup, and not some e.g. W3c thing?

Lloyd Wood

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of t.p. [[email-masked]]
Sent: 12 April 2013 21:52
To: Arturo Servin; [email-masked]
Subject: Re: Purpose of IESG Review

----- Original Message -----
From: "Arturo Servin" <[email-masked]>
To: <[email-masked]>
Sent: Friday, April 12, 2013 8:28 PM

Tom Petch

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 13 Apr 2013 08:06:13 +0100

> If you think security and congestion are arcane, you have... problems.

I think this may be the point that Diana Raft was making in draft-draft-draft
about 12 days ago.

   Brian

---

From: Stewart Bryant <[email-masked]>
Date: Sat, 13 Apr 2013 09:13:08 +0100

> I'm not saying it doesn't serve a purpose. I'm saying that I know of drafts that have been nearly rewritten during such back-and-forth, so what popped out was largely unrelated to what went in. In…

Fred

The Discusses and Comments are public, all versions of the draft are 
public, the authors and chairs (who are usually the shepherds) see all 
the review emails. Now maybe the IESG need to be more pro-active in 
sending drafts back to the WG, although that may also be unpopular, but 
if an author, shepherd or chair made that request I can think of few 
circumstances where an AD that would likely refuse.

- Stewart

---

From: t.p. <[email-masked]>
Date: Mon, 15 Apr 2013 10:03:39 +0100

----- Original Message -----
From: <[email-masked]>
To: <[email-masked]>; <[email-masked]>; <[email-masked]>
Sent: Saturday, April 13, 2013 3:46 AM

<tp>
Lloyd

Yes, e.g. in Operations and Management.

I track ICCRG (and have some familiarity with congestive collapse,
having had to trouble shoot it) but I also track some transport groups
and see pressure to allow traffic to build up faster, coming I think
from a major 'vendor'.  I do not think that there is agreement on
whether or not we should speed up networks (e.g. initial window = 10)
thereby increasing the risk of congestion.

Equally, I track SSH and TLS and continue, even after many years, to be
surprised at the hostile actions that are devised against networks and
the measures that get taken to counter them.

In both areas, I find the IESG to be cautious (and wonder if they are
being too cautious); but I am no expert in those fields and continue to
be surprised by the IESG.  I think that for many outside Security or
Transport Areas, these topics are still arcane.

Tom Petch

p.s. the last response I made to Lloyd got a reply to the effect that
Lloyd was no longer at the University of Surrey, so I do not know if
this will reach you/him.

Lloyd Wood

From: [email-masked] [[email-masked]] On Behalf Of t.p.
[[email-masked]]

----- Original Message -----
From: "Arturo Servin" <[email-masked]>
To: <[email-masked]>
Sent: Friday, April 12, 2013 8:28 PM

Tom Petch

---

From: Jari Arkko <[email-masked]>
Date: Mon, 15 Apr 2013 16:09:30 +0300

Responding to various people in one e-mail.

To summarise, we have procedures that say what kinds of Discusses are appropriate, and personal engineering preferences are not. Legitimate issues should be raised, however, and in the case of most big issues, the right approach would be to send a document back to the working group for revision. But overall, my perception is that the ADs take their task very seriously, and try to do the right thing when performing their IESG reviews. I think they make a significant contribution to the quality of our documents.

That being said, a couple of other things are also obvious. 

First, we have a very tail-heavy process. I get a lot of comments about that, and you all know it too. Moving more of the problem finding to earlier parts of the process is necessary (but also not easy).

Secondly, I know from personal experience that feedback on what kinds of things get raised in the IESG review is very useful. Even if we are trying to do the right thing, we make mistake, and perhaps more importantly, our understanding of what kinds of discusses are useful keeps evolving. Please tell us when you think we are doing the wrong thing (be it not raising an issue or raising it in the wrong category).

Finally, today too many big revisions are handled as a discussion between the authors and ADs. Big discussions should be done publicly in the working group, and in some cases the right action is for a document to be sent to the working group so that it can be reprocesses and returned.

And then to the specific responses.

> In my opinion, some individual ADs seem to, from their behavior, feel that they have not done their jobs unless they have raised a "discuss". The one that took the cake for me personally was a "dis…

And this is of course completely against our rules in the Discuss criteria document. But accidents do happen. If there were an intentional "I owe you" Discuss, that is obviously something that should go away, and authors and other ADs should complain about it.

> I'm not saying it doesn't serve a purpose. I'm saying that I know of drafts that have been nearly rewritten during such back-and-forth, so what popped out was largely unrelated to what went in. In…

I agree 100% with this.

> don't call us when we do "bad stuff" we might never know.

Indeed. For what it is worth, my own idea of what kinds of things I should as an AD complain about has evolved quite a bit. As a new AD I was overly eager, ready to defend the Internet against… bad documents. In some cases filing Discusses that in retrospect I probably should have filed as Comments. Over the years I've come to the realisation that most things are additional Comments for the authors to take into account, but that real brokenness should still cause a Discuss to be raised. But for such development to occur, you need to gain some perspective. Part of that process is getting feedback.

> problem."

+1

> by the IESG does serve a purpose.

This is one of the big issues. I think we all agree that our process is tail-heavy. More review and more problem discoveries occur at the end. This is trivial to see, but the crux is how that could be changed.

Dave:

> the tendency of ADs to inject spontaneous, late-stage personal engineering preferences, which might have been reasonable to add to the original working group discussion but realistically have no place in a final review

And personal engineering preferences are against our rules for Discusses. But beauty is in the eye of the beholder, and so are bugs :-) Is a particular spec bug an legitimate, serious technical problem, otherwise somehow against agreed IETF consensus, or is it an engineering preference?

> But what is tiring about this line of justification is its continuing failure to balance the analysis by looking at the costs and problems that come with it.  Does it provide regular and sufficient benefit to justify its considerable costs?

Agreed. 

Jari

---

From: Andy Bierman <[email-masked]>
Date: Fri, 12 Apr 2013 19:47:08 -0700

> Also, in my opinion, IESG review that raises a certain number of issues should not result in the document sitting in the IESG's queue for a few months while the authors go back and forth with the A…

Many of us can change the acronyms and describe a similar experience.
I would say "We don't have the authority to make that decision
without asking the WG" a lot.

But this is a 2nd order problem.
The real issue is the way the IESG manages WGs.
Good management doesn't fall out for free.
It is yet another continuous integration progress.
The IESG is the overworked bottleneck in the system
and the review process is the main reason.

The ADs that own a WG should be the main IESG members
that get involved at the details for any possible issue,
and hopefully make these comments/changes as early
in the WG process as possible.

During IESG review, the ADs from other areas should
restrict their comments to issues related to their area.
The final review should avoid changes made
which are feature redesigns or feature enhancements,
and limit changes to bug fixes only.

Andy

---

From: Ted Lemon <[email-masked]>
Date: Mon, 15 Apr 2013 14:23:00 +0000

> and limit changes to bug fixes only.

I have some sympathy for this argument, since most of the comments I've made about drafts since I started reviewing them as an AD, where I felt the most unsure I should be making the comment, have indeed been out-of-area or process comments.   At the same time, I have expertise in quite a few protocols that aren't in my area, and I don't claim to be a perfect expert on every protocol that's being worked on by a working group for which I am responsible AD.

So in practice, although I feel great sympathy for this position, I think it's mistaken.   I want the other ADs to comment on anything that they notice that looks like a problem.   Then we can have a conversation about it.   What I've seen in the past two formal telechats, which are the only two I've been on as an AD, as opposed to a guest, is that out-of-area DISCUSSes and comments do get raised, and they get discussed, which is the whole point.  Typically either some adjustment is made to the spec to make it clearer (this has been the case for nearly all of my DISCUSSes), or the AD who raised the DISCUSS is satisfied by the discussion and clears the DISCUSS without any change being made to the document.

So it's hard to see the harm in this, although I know it's stressful for the authors and working group chairs who have to answer the questions.   I've been there, and I know what it's like, and I'm very conscious of that when I write a DISCUSS or a comment.

On the other hand, sometimes a document attracts a great deal of comment.   Sometimes someone raises a point that is impossible to refute, and that clearly matters.   I want that point to be raised whether the AD raising it is responsible for that area or not.

Reviewing documents as an AD is really hard work.   We had an easy telechat last Thursday—I think we had less than 150 pages of protocol specs to read for the telechat.   It would not surprise me if the number of careful reviews of those documents doubled last week, for most of the documents.   I think it's absurd to suggest that such a review won't catch any issues, and it's unreasonable to suggest that a reviewer keep silent on an issue he or she sees simply because it's "not their area."

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 15 Apr 2013 15:45:11 +0100

On 15/04/2013 15:23, Ted Lemon wrote:

...
> So in practice, although I feel great sympathy for this position, I think it's mistaken.   I want the other ADs to comment on anything that they notice that looks like a problem.

There's an important class of problem that can only be found by someone
who is *not* a specialist - that is to say wording that's perfectly
clear and unambiguous to someone very familiar with the topic, but is
quite unclear to someone who isn't. This matters because we (presumably)
want our specifications to be useful to people who are implementing or
deploying them without already being members of the inner circle.

Just to be specific, here is a piece of text that came out of a WG
not so long ago. I have bowdlerised it:

"The Foobar standards [RFCxxx], [RFCyyy] provide useful generic
functionality like blah, blah and blah for reducing the overhead in
Boofar networks.  This functionality can be partly applied to Bleep."

That was it - a third party implementing Bleep was apparently supposed
to guess which bits of those RFCs applied where.

This led to a DISCUSS and seven months of delay before that "partly"
was disambiguated. Was that inappropriate out-of-area review?

   Brian

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Mon, 15 Apr 2013 15:29:16 +0000

On Apr 15, 2013, at 7:45 AM, Brian E Carpenter <[email-masked]>
 wrote:

> not so long ago. I have bowdlerized it:
> 

No, it was not. But I would argue that "seven months" in an IESG review was too long. At some point, the responsible AD should have written a set of questions and sent it back to whatever working group it came from with "don't send it back until you have answered those questions". The ADs shouldn't have spent their time trying to have the conversation; they should only have ensured that they were OK with the result.

>From the working group side, it would *also* have been helpful to have the AD that raised the question get involved in the WG discussion, if only to ensure that s/he was OK with the outcome when it came.

---

From: Joe Touch <[email-masked]>
Date: Mon, 15 Apr 2013 08:36:14 -0700

On 4/15/2013 7:23 AM, Ted Lemon wrote:

> So it's hard to see the harm in [late non-area input by the IESG],

It gives the IESG an exemption to participating in WG and IESG last call 
processes, which then frustrates the rest of the community that does not 
have this opportunity.

It says that the opinion of the IESG is more important than that of the 
WG; we should be judging ideas on their own merit, not their origin.

Joe

---

From: Ted Lemon <[email-masked]>
Date: Mon, 15 Apr 2013 16:09:44 +0000

On Apr 15, 2013, at 11:36 AM, Joe Touch <[email-masked]> wrote:
> It gives the IESG an exemption to participating in WG and IESG last call processes, which then frustrates the rest of the community that does not have this opportunity.

You could equally say that the IETF last call frustrates the WG process, since a document can fail IETF last call, and this can be extremely frustrating for working groups.   Witness the fiasco in the MIF working group when they tried to advance a DHCP route option, for example.

The IETF last call process is important, but it's not a panacea.   Too many documents come through the last call process for each one to get thorough review by every IETF participant.   The IESG are effectively the sacrificial lambs of the IETF who have to read every single document on the IETF track that makes it through last call.

When you say that the IESG should not get special treatment, I think you are misunderstanding what is special about the treatment the IESG gets.   What is special is that we are expected to review every document.   We could impose that requirement on the IETF as a whole.   If all IETF participants were willing to do that, then IETF review might well be effective enough to eliminate the need for the IESG.   But I don't think that's a realistic expectation, and I do not mean that as a criticism.   What you would have accomplished if you did this would be to have turned every IETF participant into an AD, working 35 hours a week on document review.

---

From: Elwyn Davies <[email-masked]>
Date: Mon, 15 Apr 2013 17:18:34 +0100

On 15/04/13 14:09, Jari Arkko wrote:
> [Responding to Dave Crocker]
I would say it usually does justify its costs - skimping on the QA is 
almost always a bad idea.  Speaking also from a gen-art perspective, 
getting a level of clarity and ease of use into a document may take a 
little while  for a number of reviewers and authors, but the costs  and 
benefits need to include the effort of implementers, testers and future 
users who will not appreciate getting a product that doesn't 
interoperate because the spec was unclear or they missed a point that 
might have been 'obvious' to the original authors.  The size of the 
affected population is much bigger after publication (at least if 
anybody actually cares about the RFC).

I have to say that in my experience the seven months of delay that Brian 
cites is rather unusual.   For the vast majority of documents, the sorts 
of clarification needed can be sorted out in a couple of rounds of 
email, and the results are unlikely to require recycling back to the 
WG.  That being said, there certainly are documents where I wonder why 
anybody seriously thought the document was ready for publication; trying 
to fix the process 'tail heaviness' (as Jari describes it) would help if 
we could find a way.  It could be argued that the tail process is just 
too polite.  Once a document embarks on the publication process we 
seemingly inevitably send it through all those reviews resulting in 
multitudinous politely phrased DISCUSSes and comments when what it 
really needs is a blunt refusal followed by some competent editing.

/Elwyn

---

From: Michael Richardson <[email-masked]>
Date: Mon, 15 Apr 2013 12:23:38 -0400

>>>>> "Ted" == Ted Lemon <[email-masked]> writes:
    Ted> You could equally say that the IETF last call frustrates the WG
    Ted> process, since a document can fail IETF last call, and this can
    Ted> be extremely frustrating for working groups.   Witness the
    Ted> fiasco in the MIF working group when they tried to advance a
    Ted> DHCP route option, for example. 

Maybe we should have an IETF first call (for objections), rather than
last call. 

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Joe Touch <[email-masked]>
Date: Mon, 15 Apr 2013 09:26:35 -0700

> You could equally say that the IETF last call frustrates the WG process, since a document can fail IETF last call, and this can be extremely frustrating for working groups.   Witness the fiasco in…

IETF LC does not come with a list of constraints of what is in and not in scope.

> The IETF last call process is important, but it's not a panacea.   Too many documents come through the last call process for each one to get thorough review by every IETF participant.   The IESG ar…

If docs get out of WG LC with no review, then there's a failure of IETF process. Every such doc should be read at least by a few independent reviewers, by the WG chairs, and by the ADs in that area.

The same failure can - and does - happen at the IESG level. We can continue to appoint groups with additional rounds of review, but IMO, they are scoped (and the IESG review guidance appears to back up that point).

Joe

---

From: Ted Lemon <[email-masked]>
Date: Mon, 15 Apr 2013 16:28:10 +0000

> Maybe we should have an IETF first call (for objections), rather than

I think that would look a lot like a DoS attack on the IETF, but it would be nice if there were a way to make it work.

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 15 Apr 2013 17:34:58 +0100

On 04/15/2013 05:26 PM, Joe Touch wrote:
> We can continue to appoint groups with additional rounds of review, but IMO, they are scoped (and the IESG review guidance appears to back up that point).

I think Joe is correct there. Another data point is that we
asked secdir (who currently have an 80% review rate for
documents) if they could see a way to get more done earlier
and thus reduce the review-all requirement for ADs. There were
a significant number of secdir folks who thought that increasing
the expectations on them might well mean getting far less
than 80% of documents reviewed - basically, they're also very
busy folks and our current 20% dropped-review rate might
just shoot up if we ask for too much.

I do wish there were a good way to reduce the AD review
burden and am happy to chat on or offlist with anyone who
has ideas, but I for one am not seeing anything obvious.

S.