Subject: NomCom 2019 Call for Community Feedback

From: NomCom Chair 2019 <[email-masked]>
Date: Sun, 03 Nov 2019 07:13:18 -0800

Community Feedback Requested!

Feedback for IAB, IESG Area Directors, IETF Trust and LLC Board.

The 2019-20 Nominating Committee has collected a list of willing
candidates for the positions open this cycle.  You can see the list at 
https://datatracker.ietf.org/nomcom/2019/feedback/

You may provide feedback using the web form there; it is secure.  
All web feedback goes into the datatracker using asymmetric 
encryption, which is then decrypted by the NomCom members 
as they read it.  Your feedback can not be seen by the secretariat, the 
tools people, or any of the management.

Your feedback through the web form is not anonymous when shown to
NomCom members as you need an IETF login to provide it.

If you want to give truly anonymous feedback, please contact one 
of the NomCom members that you trust directly, and ask him/her to
relay the feedback anonymously to the NomCom.

You can also submit feedback via email to nomcom-chair-2019 at 
ietf.org and I will enter it in the datatracker. One email per 
candidate, please. Please indicate if I should share your identity
with the full NomCom.

We will announce office hours in Singapore  IETF 106 shortly.

The positions to be filled and the desired expertise are found here:

https://datatracker.ietf.org/nomcom/2019/

Thank you so much for your feedback - we really value it and 
appreciate your time taken to submit it.

regards,
Victor  

Victor Kuarsingh
victor at jvknet dot com
nomcom-chair-2019 at ietf dot org

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 4 Nov 2019 09:23:33 +0000

Am I alone in being both surprised and disappointed at the small number of candidates that Nomcom has for a number of the IESG positions?

5/2/2/1/1/2

Best regards

Stewart

> On 3 Nov 2019, at 15:13, NomCom Chair 2019 <[email-masked]> wrote:
> 
> _______________________________________________
> IETF-Announce mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-announce

---

From: "Salz, Rich" <[email-masked]>
Date: Mon, 4 Nov 2019 13:10:18 +0000

>    Am I alone in being both surprised and disappointed at the small number of candidates that Nomcom has for a number of the IESG positions?
  
Knowing how often an incumbent has *NOT* been chosen would help answer that question, but anecdotally why go through all that work if there are overwhelming odds that you're just going to lose?

---

From: Kyle Rose <[email-masked]>
Date: Mon, 4 Nov 2019 08:24:42 -0500

On Mon, Nov 4, 2019 at 4:24 AM Stewart Bryant <[email-masked]>
wrote:

> Am I alone in being both surprised and disappointed at the small number of
> candidates that Nomcom has for a number of the IESG positions?
>

Disappointed? No, I'm sure many IETFers feel the same way.

Surprised? Maybe. I'm personally not surprised: being AD is basically a
full-time job, and unless one is independently wealthy a candidate needs
support from his/her employer for a 2-year (and often 4-year) commitment.
Conservatively, that's a $600,000 investment for a 2-year term, and you
don't even get your logo in the credits for the trouble. It's hard to make
the business case for that.

---

From: Paul Wouters <[email-masked]>
Date: Mon, 4 Nov 2019 08:43:47 -0500 (EST)

On Mon, 4 Nov 2019, Kyle Rose wrote:

> Surprised? Maybe. I'm personally not surprised: being AD is basically a full-time job, and unless one is
> independently wealthy a candidate needs support from his/her employer for a 2-year (and often 4-year) commitment.
> Conservatively, that's a $600,000 investment for a 2-year term, and you don't even get your logo in the credits
> for the trouble. It's hard to make the business case for that.

This is indeed the core of the problem. And perhaps we need to think
about how it can be addressed in other ways.

Paul

---

From: Kathleen Moriarty <[email-masked]>
Date: Mon, 4 Nov 2019 10:20:48 -0500

On Mon, Nov 4, 2019 at 8:44 AM Paul Wouters <[email-masked]> wrote:

> full-time job, and unless one is
> > independently wealthy a candidate needs support from his/her employer
> for a 2-year (and often 4-year) commitment.
> don't even get your logo in the credits
For years we've had a good list for Security, so I am very disappointed to
see no one up except the incumbent.  I do see value in going through the
process even if you are not selected so that the NomCom has a choice and if
anything were to happen to either incumbent (new position that didn't
support them, lottery, etc.), there would be a place to start.  It's up to
the incumbents to invite others to put their names int he hat and that
works quite well.  I did nominate a few people when I saw there was no one
there, but the incumbents for all positions really should be reaching out
individually and discussing the pro/cons, offering to talk to management,
etc.

In terms of benefits, one is having the opportunity to see the work across
the IETF.  I found this quite helpful to be able to take a step back and
think about the work happening across areas and what the impact of that
would be collectively.  It's a really interesting exercise and could be
beneficial to your organization when you step back into your role with this
expanded view and knowledge.  I'm almost done writing a book from these
observations.  Benoit also published just after stepping down and it helped
him get promoted to the next level in Cisco as he clearly demonstrated
technical leadership across industry in one of his areas of expertise.
Alia moved on to a new type of role for her as she gained management
experience and it provided an opportunity to move in a new direction.
There are benefits to the individual and to the organization, but you have
to think about what that might be and how you will go about it.  This is
helpful when justifying the time that will be spent in the position.

Best regards,
Kathleen

Paul

-- 

Best regards,
Kathleen

---

From: "Livingood, Jason" <[email-masked]>
Date: Mon, 4 Nov 2019 19:52:10 +0000

+1 to what Kyle & Paul said. I agree this is a job scope / expectations challenge. Unless the expectations for the AD role meaningfully changes, we should not expect any great increase in the number of candidates.

Jason
(writing personally, not for IETF LLC)

On 11/4/19, 8:44 AM, "ietf on behalf of Paul Wouters" <[email-masked] on behalf of [email-masked]> wrote:

    Paul

---

From: "Salz, Rich" <[email-masked]>
Date: Mon, 4 Nov 2019 19:54:52 +0000

I wonder what people think would break if we moved to 5 AD's per area, and they could divide the WG's and IESG concalls amongst themselves?

---

From: Randy Bush <[email-masked]>
Date: Mon, 04 Nov 2019 12:22:43 -0800

> I wonder what people think would break if we moved to 5 AD's per area,
> and they could divide the WG's and IESG concalls amongst themselves?

and that might free up one of them to write more process docs about all
this.

perhaps it is not a problem of technical engineering load, but a problem
of bureaucratic, process, and politics load.

randy

---

From: Yoav Nir <[email-masked]>
Date: Mon, 4 Nov 2019 23:08:47 +0200

> On 4 Nov 2019, at 21:54, Salz, Rich <[email-masked]> wrote:
> 

Five?  We barely manage to get two.  Where will we find five?

This WG chair gig that you and I do?  We can sell it to our respective employers as something we do in our spare time.  An AD’s job?  Not so much.

Yes, having five of them would mean that each supervises 1-2 working groups. It might make scheduling sessions in a meeting a bit easier. But the ADs will still have to attend all three meetings plus the retrieat, and most of them are still going to be reading every draft.  I can see this cutting the AD job down to 25 hours a week, maybe even 20. But that’s still a sizeable chunk of time, and it makes you not available for crunch time for your employer. 

It might get Kyle’s $600,000 down to $400,000.  A decent chunk of change, and now you want 30 such chunks rather than 14.  

I don’t think adding more management is the solution.

---

From: Kathleen Moriarty <[email-masked]>
Date: Mon, 4 Nov 2019 16:12:02 -0500

On Mon, Nov 4, 2019 at 4:09 PM Yoav Nir <[email-masked]> wrote:

> This WG chair gig that you and I do?  We can sell it to our respective
> employers as something we do in our spare time.  An AD’s job?  Not so much.
>
> Yes, having five of them would mean that each supervises 1-2 working
> groups. It might make scheduling sessions in a meeting a bit easier. But
> the ADs will still have to attend all three meetings plus the retrieat, and
> most of them are still going to be reading every draft.  I can see this
> cutting the AD job down to 25 hours a week, maybe even 20. But that’s still
> a sizeable chunk of time, and it makes you not available for crunch time
> for your employer.
>
> It might get Kyle’s $600,000 down to $400,000.  A decent chunk of change,
> and now you want 30 such chunks rather than 14.
>

Right and then you have to coordinate among them all, which is time
consuming.

I, and others are willing to help talk to management to justify the role
and time spent.  I think there's a pretty compelling argument.

Best regards,
Kathleen

-- 

Best regards,
Kathleen

---

From: "Salz, Rich" <[email-masked]>
Date: Mon, 4 Nov 2019 21:13:52 +0000

> But the ADs will still have to attend all three meetings plus the retrieat, and most of them are still going to be reading every draft.

I would hope that is no longer necessary.

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 4 Nov 2019 21:25:19 +0000

On 04/11/2019 19:54, Salz, Rich wrote:
> I wonder what people think would break if we moved to 5 AD's per
> area, and they could divide the WG's and IESG concalls amongst
> themselves?
> 

I think that's essentially up to the IESG to deal with,
it's not really that easy to do from "outside" because
the current IESG will have to be the ones to do the work
for any change. (And it'll be a lot of work is my bet.)

FWIW, when I was on the IESG I tried arguing for there
to be only ~7 ADs total following the logic that that'd
force the IESG to delegate more. Despite trying, we never
found a practical way to get there though.

Cheers,
S.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 5 Nov 2019 10:29:13 +1300

> I wonder what people think would break if we moved to 5 AD's per area, and they could divide the WG's and IESG concalls amongst themselves?

Wouldn't work. An IESG of that size would be impossible to run like the current IESG, which is already too big to be cohesive. It would become more like a parliament and even less like a steering group.

If we want the IESG job to be more reasonably sized, we have to take work away from the ADs. As far as I can see, that means taking away their duty of acting as final reviewers. I don't want to name names because I don't think the ADs are to be blamed individually, but some of them spend *enormous* effort on detailed reviews. Either we have to trust the WGs more, or we have to trust the various review teams more, and allow the IESG to steer, which its name implies is its main job. And sign off on drafts that are good enough, rather than seeking perfection.

This is hardly a new discussion, but we just let it drift each year, after the NomCom season is over.

    Brian

---

From: "Salz, Rich" <[email-masked]>
Date: Mon, 4 Nov 2019 21:33:01 +0000

>    This is hardly a new discussion, but we just let it drift each year, after the NomCom season is over.
  
So something more lasting, like an IAB program?  YAKOBE?

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 4 Nov 2019 21:39:26 +0000

> So something more lasting, like an IAB program?  YAKOBE?

Heh:-)

To be clear: No, restructuring the IESG is so very
definitely not the IAB's job.

S.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 5 Nov 2019 11:21:42 +1300

> definitely not the IAB's job.

A draft in front of gendispatch might be the correct approach,
if there are any specific proposals. Although, reading
https://tools.ietf.org/html/rfc2026#section-6.1.2, it's not
clear to me that any process changes are needed, since
nowhere is it specified that IESG members should perform
detailed technical reviews themselves. As far as I can see,
they could stop doing that tomorrow without breaking any
process rules, and choose to rely entirely on the various
review teams.

   Brian

---

From: Kyle Rose <[email-masked]>
Date: Mon, 4 Nov 2019 17:55:56 -0500

On Mon, Nov 4, 2019 at 4:29 PM Brian E Carpenter <
[email-masked]> wrote:

> If we want the IESG job to be more reasonably sized, we have to take work
> away from the ADs. As far as I can see, that means taking away their duty
> of acting as final reviewers. I don't want to name names because I don't
> think the ADs are to be blamed individually, but some of them spend
> *enormous* effort on detailed reviews. Either we have to trust the WGs
> more, or we have to trust the various review teams more, and allow the IESG
> to steer, which its name implies is its main job. And sign off on drafts
> that are good enough, rather than seeking perfection.
>

Agreed, and with Stephen's advocacy of delegation as a path forward. But to
get there, the IETF (and the folks with dots, in particular) need to become
more comfortable with the idea of decentralizing authority to a greater
degree than they have been used to: that means empowering---and trusting,
and training, and reviewing the output of---the directorates to a greater
extent than happens today. The buck may still stop with the ADs, but they
should be willing and able to choose to delegate to a greater degree,
preserving their time and brain cycles for the truly critical stuff that
they either seek out or that is escalated to them.

We may also want to take a serious look at re-empowering the IAB's
historical role in architectural oversight, with an appropriate set of
checks and balances to prevent a recurrence of the kerfuffle that resulted
in the current structure.

Kyle

---

From: Keith Moore <[email-masked]>
Date: Mon, 4 Nov 2019 18:17:06 -0500

> I wonder what people think would break if we moved to 5 AD's per area, and they could divide the WG's and IESG concalls amongst themselves?

I suspect it would mean that there'd be more ADs looking for nits to 
pick in order to delay documents.

Note: I DO NOT think that ADs see their purpose in picking nits or 
delaying documents.   But I do think there's a tendency for people in 
any position to, perhaps subconsciously, try to "prove their worth", and 
I suspect that having a lot more ADs would create perverse incentives.

ADs do tend to be overloaded.   Sometimes, IMO, it's because there is a 
lack of political will (and/or community support) to push back against 
new WGs that lack the energy or the clue to be doing what they propose 
to do.    Sometimes, IMO, it's because IETF has a longstanding Bad Habit 
of trying to mask tussles by spinning up lots of narrowly focused WGs 
with non-overlapping scope.

But mostly, IMO, ADs are overloaded because we're trying to use 
volunteers to do what should be paid positions with limited terms.

Keith

---

From: Tim Wicinski <[email-masked]>
Date: Mon, 4 Nov 2019 18:37:59 -0500

A Working Group chair role also have very tangible results to employers.
They see the work we help shepherd and
it feels their corporate hearts with warmth.  I also like chairing working
groups - the interactions are much more personable
and enjoyable. I like to think I am OK at it.  But I also like working and
I can move in and out.

An AD job looks and feels like middle management. I feel I would do a
horrible job at it.
I've always been lucky to have great ADs but I feel my role is to make
their jobs easier however possible.

Tim

---

From: Rob Sayre <[email-masked]>
Date: Mon, 4 Nov 2019 21:32:23 -0800

On Mon, Nov 4, 2019 at 3:38 PM Tim Wicinski <[email-masked]> wrote:

>
> An AD job looks and feels like middle management.
>

Which of course raises the question: what do they actually do? The AD job
description is strikingly concise.

It seems to me that it's a lot of work, and requires knowing which people
need to be in which conversations. That takes both technical and political
savvy.

It might be that the desirability and responsibility of an AD role will ebb
and flow with the necessity for negotiation in a given area.

thanks,
Rob

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 5 Nov 2019 01:24:22 -0800

> process rules, and choose to rely entirely on the various

The past few generic IESG job descriptions (as sent to Nomcom) have had
some interesting text in this front (quoting from
https://datatracker.ietf.org/nomcom/2019/expertise/#pos-iesg-members):

% An AD should be able to personally review every Internet-Draft that they
% sponsor. For other Internet-Drafts an AD needs to be satisfied that
% adequate review has taken place, though many ADs personally review these
% documents as well. 

That does, of course, leave the question open of what constitutes "adequate
review", but it seems highly likely that the intent is to allow
directorates to satisfy it.

-Ben

---

From: Bob Hinden <[email-masked]>
Date: Tue, 5 Nov 2019 07:52:26 -0800

Rich,

> On Nov 4, 2019, at 11:54 AM, Salz, Rich <[email-masked]> wrote:
> 

Why stop there, how about one AD per working group?

I hope this helps makes it clear that more authority should go to the w.g. chairs, adding more AD won’t scale.   The NomCom needs to be selecting people who are not the best technical experts in an area, but who know enough to make sure that adequate review has been done in the working groups, verify directorate reviews, and that last call comments have been addressed.   They don’t need to do detailed technical reviews themselves.   They especially don’t need to do editorial reviews.

This might also allow the ADs to spend more time on working group charters to insure that the working group has the appropriate scope.

Bob

---

From: Donald Eastlake <[email-masked]>
Date: Tue, 5 Nov 2019 11:21:29 -0500

On Tue, Nov 5, 2019 at 4:24 AM Benjamin Kaduk <[email-masked]> wrote:
> ...

I find the quality of directorate reviews to be highly variable.
Sometimes they are quite deep and perceptive. Sometimes I have thought
a draft needs a review by the X Directorate because of X issues in it
and the resulting review has only a few comments on things found by
the nits checker -- and I mean not just that it has no substantive
comments on X issues, the review doesn't doesn't mention X at all,
except possibly is boilerplate at the beginning of the review, not
even to say that the reviewer didn't find any X issues in the draft
:-(

Thanks,
Donald
===============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 2386 Panoramic Circle, Apopka, FL 32703 USA
 [email-masked]

---

From: Kathleen Moriarty <[email-masked]>
Date: Tue, 5 Nov 2019 11:29:16 -0500

On Tue, Nov 5, 2019 at 10:52 AM Bob Hinden <[email-masked]> wrote:

> I hope this helps makes it clear that more authority should go to the w.g.
> chairs, adding more AD won’t scale.   The NomCom needs to be selecting
> people who are not the best technical experts in an area, but who know
> enough to make sure that adequate review has been done in the working
> groups, verify directorate reviews, and that last call comments have been
> addressed.   They don’t need to do detailed technical reviews themselves.
>  They especially don’t need to do editorial reviews.
>

For editorial problems, I sent some documents back to the working group to
figure out.  Sometimes with prolific authors, people get very used to the
writing style and are able to read it without problems.  When it gets to
the AD, the problems are sometimes more clear.  Just pointing this out and
sending it back was enough to resolve those issues, so yes, delegating. :-)

AD reviews may be more in-depth, but cross area should just be focused on
what the particular AD should be looking to see in documents from other
areas IMO. While I did read most documents, you figure out where skimming
is probably fine and learn where you may catch things of importance.  It
becomes less work if you do this after a few months in the job.  A few
calls of 800 pages to read really help you figure this out faster :-)

Kathleen

> This might also allow the ADs to spend more time on working group charters
> to insure that the working group has the appropriate scope.
>
> Bob
>
>
>
>
>

-- 

Best regards,
Kathleen

---

From: "Eric Vyncke (evyncke)" <[email-masked]>
Date: Tue, 5 Nov 2019 16:35:12 +0000

As a "junior" IESG member, I have been reading this open thread with great interest.

Replying to Bob's email because he nailed a lot of valid points:

1) AD are 'responsible' so they do need to check the intensive work of WG, shepherd, IETF last call and directorate
2) sometimes the directorate reviews come too late for the IESG ballot for approval
3) and the IESG review is an interesting step because it is a common decision by different ADs with different background and expertise (BTW, do not consider the IESG as monolithic => we tend to politely and respectfully disagree among ourselves but we always manage to come to a position)
4) the review is of course deeper when it is the WG where you are the responsible AD __, for others I look of the overall readability, some process oriented things (such as RFC 8174), conflicts with other drafts/WG/areas, and issues related to my area (internet)
5) indeed, the expertise coverage of an AD should aim for breadth and not for depth for the reviews (in addition to the
6) reviews are probably 50% of our time
7) several AD (including myself) are not English-speaking natives, how would we dare to have editorial comments ;-) but I am improving my English by reading those comments :-)

And as Kathleen wrote, while this requires a lot of time and energy, the reward is excellent: you learn a LOT and most of the time this is really interesting :-) 

Hope this helps

-éric	

On 05/11/2019, 16:53, "ietf on behalf of Bob Hinden" <[email-masked] on behalf of [email-masked]> wrote:

    Rich,

    

    Bob

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 5 Nov 2019 16:40:13 +0000

>    Why stop there, how about one AD per working group?
  
reductio ad absurdum
  
>    I hope this helps makes it clear

Far from it. If the job is too big (so that only full-sponsored people can be IESG members), then the choice is remove some of the work or add more people.

---

From: Keith Moore <[email-masked]>
Date: Tue, 5 Nov 2019 11:53:02 -0500

On 11/5/19 10:52 AM, Bob Hinden wrote:

> I hope this helps makes it clear that more authority should go to the w.g. chairs, adding more AD won’t scale.   The NomCom needs to be selecting people who are not the best technical experts in an area, but who know enough to make sure that adequate review has been done in the working groups, verify directorate reviews, and that last call comments have been addressed.

One problem with that idea is that WGs too often work to the benefit of 
a narrow interest and to the detriment of other interests.   The IESG is 
the only formal mechanism we have to keep WGs from getting out of hand 
in that way.   So IMO the ADs need to have a high level of technical 
expertise, but broad technical expertise rather than narrow technical 
expertise.

There's also a problem with relying on directorate reviews, which is 
that every reviewer approaches the situation from his or her own point 
of view, and sometimes reviewing a review is more work than reviewing 
the subject document.   Ideally an AD would have benefit of reviews from 
several reviewers across a spectrum of perspectives.

Keith

---

From: "John Levine" <[email-masked]>
Date: 5 Nov 2019 13:49:40 -0500

>Far from it. If the job is too big (so that only full-sponsored people can be IESG members), then the choice is remove some of the work or add more people. 

I wouldn't think that adding people would help much since a lot of the
work isn't divisible.  ADs traditionally read all of the drafts, not
just the ones in their area.  I suppose we might say that if there are
three or four ADs in an area, pick two or three at random for each
drafts, but there's a lot of other places where all the ADs are
involved.  The number of communication paths grows as the square of
the number of people.
 
The way to have less work is to publish fewer RFCs, or perhaps shorter
RFCs.  We might, for example, revisit our policy of not doing
versioned RFCs, and see if there's a way to do minor revisions without
having to reopen the rest of a document.

R's,
John

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 5 Nov 2019 18:59:01 +0000

work isn't divisible.  ADs traditionally read all of the drafts, not
    just the ones in their area.

Does every AD read all the drafts?  I don't know.  But changing the process to say one AD from each area would reduce the load.

> The number of communication paths grows as the square of
    the number of people.
   
Second time this has been mentioned.  There's a huge difference between information-theory and practice.
  
>    The way to have less work is to publish fewer RFCs, or perhaps shorter
    RFCs.

That is certainly *A* way, but not *THE* way.

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 5 Nov 2019 19:15:36 +0000

> On 5 Nov 2019, at 18:59, Salz, Rich <[email-masked]> wrote:
> 
> Does every AD read all the drafts?

That depends. I tried to read every draft. I mostly succeeded.

>  I don't know.  But changing the process to say one AD from each area would reduce the load.

That was tried. It did not entirely work.

> That is certainly *A* way, but not *THE* way.

Then people would look to take their work elsewhere which has other undesirable outcomes.

- Stewart

---

From: Kathleen Moriarty <[email-masked]>
Date: Tue, 5 Nov 2019 14:17:39 -0500

On Tue, Nov 5, 2019 at 2:00 PM Salz, Rich <[email-masked]> wrote:

> Does every AD read all the drafts?  I don't know.  But changing the
> process to say one AD from each area would reduce the load.
>
Some divide it up.  Pete & Barry had a split.  If there was a big load,
Stephen would start from one end of the list and I'd start from the other.
We both mostly wanted to 'read' them all though.

-- 

Best regards,
Kathleen

---

From: Michael Richardson <[email-masked]>
Date: Tue, 05 Nov 2019 16:50:57 -0500

Brian E Carpenter <[email-masked]> wrote:
    > If we want the IESG job to be more reasonably sized, we have to take
    > work away from the ADs. As far as I can see, that means taking away
    > their duty of acting as final reviewers. I don't want to name names
    > because I don't think the ADs are to be blamed individually, but some
    > of them spend *enormous* effort on detailed reviews.

+1

I think that there is a lack of trust by ADs of the various directorates.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Kyle Rose <[email-masked]>
Date: Tue, 5 Nov 2019 17:32:01 -0500

> I think that there is a lack of trust by ADs of the various directorates.

To be fair, output quality even within a particular directorate is highly
variable. If we were to go down this path, the directorates would need to
step up. That means:

* Better, more comprehensive area-specific reviews
* Completed on time
* Employing consistent standards, objectives, and issue coverage
* Possibly with explicit training and written resources guiding reviews
* Escalation of specific issues to the AD or to the wider directorate
* Most importantly, a feedback loop: reviews subjected to metrics and
success criteria evaluated on an on-going basis

Redundancy (multiple reviewers) and/or alignment with subject matter
expertise (rather than round-robin assignment) would probably help
immensely.

Kyle

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 6 Nov 2019 11:54:56 +1300

On 05-Nov-19 22:24, Benjamin Kaduk wrote:
...
> % adequate review has taken place, though many ADs personally review these

I'm sure the last clause is true, but maybe it's an error to include it in the job description almost as if it is a duty.

> 2) sometimes the directorate reviews come too late for the IESG ballot for approval

In that case, perhaps the response should be to defer the ballot automatically, and make it public that the reason is a late review.

Also, do some areas only request telechat reviews? In my experience in Gen-ART, most issues and fixes occur during Last Call reviews, so that the telechat review is often a formality.

On 06-Nov-19 08:17, Kathleen Moriarty wrote:

> Some divide it up.  Pete & Barry had a split.  If there was a big load, Stephen would start from one end of the list and I'd start from the other.  We both mostly wanted to 'read' them all though.

The word that stands out for me is "wanted". Similarly, it's because I *want* to track the technology that I've been a Gen-ART reviewer for many years. But if (as people say every year at about this time) we really need to reduce the AD workload to something more like part-time, some things have to change.

It seems to me that the IESG itself can make such changes, since this is a matter of procedures rather than our formal rules. It's too late for this year, but maybe next year's NomCom could have an easier job.

   Brian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 6 Nov 2019 13:22:59 +1100

+1 to the Brian's thoughts below.

I think this is one of the most important things we as a community need to address; we have a culture of expecting "heroes" to step up and take this job. That's going to bite us, sooner or later.

Cheers,

> On 6 Nov 2019, at 9:54 am, Brian E Carpenter <[email-masked]> wrote:
> 
> ....

--
Mark Nottingham   https://www.mnot.net/

---

From: Rob Sayre <[email-masked]>
Date: Tue, 5 Nov 2019 18:46:08 -0800

On Tue, Nov 5, 2019 at 6:23 PM Mark Nottingham <[email-masked]> wrote:

> I think this is one of the most important things we as a community need to
> address; we have a culture of expecting "heroes" to step up and take this
> job. That's going to bite us, sooner or later.
>

This message seems very correct to me.

A first step might be to evaluate the degree to which the IETF is viewed as
a spec printer. I've been on the IETF list for 15 years (mostly silently),
and I think most documents are pretty bad.

thanks,
Rob

---

From: Terry Manderson <[email-masked]>
Date: Wed, 6 Nov 2019 13:08:32 +1000

I’ve been watching this thread with an extreme level of interest.

And this, for better or worse, is my opinion. The IETF needs to want to change. The days, as Mark describes, of having that technical hero is gone. It’s time now to trust directorate reviews, and for WGs to make sure the WG does it’s own review. The days of a WG throwing a document over the wall to get AD acceptance really needs to be in the past. The “my AD will handle it” approach needs to be set aside. That document has to be proven before it gets to the rubber stamp process.

How you get there will require a culture change more significant than living through behaviours towards new participants, greater gender diversity involvement, cultural sensitives, and and understanding of the leadership/follower dynamic. The reason that it will be hard is that the extant culture (while I accept and appreciate my predecessors) is that it is built on the shoulders of legends. It’s now the time of the IETF to have a championship team, rather than a team of champions. Each member of that team needs to be responsible and accountable for their contribution. The world is changing around us, much faster than we can adapt. The IETF agility is, sadly, deserving of being turbo charged, and you can’t do that with a hand-full of brilliant people as the “fixers”. 

That is why the AD workload remains so high. Stop relying on the ADs to bless your “baby” (document). Make sure while in the WGs that WG-doc truely is ready to be published. Seek your own cross area and Directorate reviews. Be ruthless in what you write. Join Directorates en-masse. Learn how to review other’s documents and look out for your area or WG. Make sure you have thought about all scenarios. Work, above all, with your fellow contributors towards the best (not perfect) outcome that is deployable.

Understand that while the IETF has a huge reputation, there are other standards bodies that are doing this better, and faster, and resulting in bigger improvements to technology. 

This is not something one or two leaders can do for you. This you must all decide is appropriate and empower your leaders. Leaders can only exist with support.

You decide. The change comes from within first.

Cheers,
Terry
--
Mobile device, don't expect grammar.

> On 6 Nov 2019, at 12:23 pm, Mark Nottingham <[email-masked]> wrote:
>

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 6 Nov 2019 03:13:58 +0000

>    Understand that while the IETF has a huge reputation, there are other standards bodies that are doing this better, and faster, and resulting in bigger improvements to technology. 
  
Serious question: please name them, so we can learn from them.

---

From: Keith Moore <[email-masked]>
Date: Wed, 6 Nov 2019 01:22:51 -0500

On 11/5/19 10:08 PM, Terry Manderson wrote:

> And this, for better or worse, is my opinion. The IETF needs to want to change. The days, as Mark describes, of having that technical hero is gone. It’s time now to trust directorate reviews,

While I agree that we can't depend on having "heroes" in IESG, I also 
think that expecting directorates to do all of the detailed reviewing, 
without having to answer to anyone, will have the result of destroying 
what little quality control IETF has left.

Keith

---

From: Leif Johansson <[email-masked]>
Date: Wed, 6 Nov 2019 09:07:18 +0100

> Serious question: please name them, so we can learn from them.

openid foundation comes to mind

for agility its hard to beat the cray^H^H^H^H blockchain identity
people (I'm in no way biased against them) over at the distributed
identity foundation

	Cheers Leif

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 6 Nov 2019 10:34:42 +0000

Hi Michael,

> I think that there is a lack of trust by ADs of the various directorates.

Other ADs have commented on this, but I think I need to repeat what they
said and expand on it.

Results are vary varied. Some are quite good (e.g. Gen-Art) and others
really depend on reviewer. ADs responsible for Directorates are faces
with the choice of firing half of their Directorates (which has some
rather unfortunate consequences) and/or raise the bar on who should be
allowed to join. We already struggle to recruit people at all levels of
our organization.

Best Regards,
Alexey

---

From: Ralph Droms <[email-masked]>
Date: Wed, 6 Nov 2019 06:03:01 -0500

> On Nov 6, 2019, at 5:34 AM, Alexey Melnikov <[email-masked]> wrote:
> 
> Hi Michael,
> 

We shouldn't be depending on last-minute quality checks to maintain the quality of our output.  Working groups should be producing documents that are ready to publish, and develop trust that their documents are high quality.

- Ralph

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 6 Nov 2019 11:42:19 +0000

Perhaps RFCs could list (within the document) who reviewed/approved them, and in which role/capacity the review had been performed.

This could serve two purposes:
 - some minimal reward for those individuals taking the time to review the document,
 - encouragement for the reviewers to ensure that an adequate review has been performed based on the role/capacity in which they are acting. 

Rob

> -----Original Message-----
> From: ietf <[email-masked]> On Behalf Of Ralph Droms
> Sent: 06 November 2019 11:03
> To: Alexey Melnikov <[email-masked]>
> Cc: Michael Richardson <[email-masked]>; ietf <[email-masked]>
> Subject: Re: Quality of Directorate reviews
> 
> 
> 
> >>> because I don't think the ADs are to be blamed individually, but
> >>> some of them spend *enormous* effort on detailed reviews.
> >>
> >> +1
> >>
> > Other ADs have commented on this, but I think I need to repeat what
> > they said and expand on it.
> >
> > allowed to join. We already struggle to recruit people at all levels
> > of our organization.
> >
> > Best Regards,
> > Alexey
> 
> We shouldn't be depending on last-minute quality checks to maintain the
> quality of our output.  Working groups should be producing documents that
> are ready to publish, and develop trust that their documents are high
> quality.
> 
> - Ralph
> 
> >

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 6 Nov 2019 11:46:44 +0000

>   - encouragement for the reviewers to ensure that an adequate review has been performed based on the role/capacity in which they are acting.
I think acknowledging people is generally a good idea. I would be in 
favour of always acknowledging directorate reviews, shepherds and 
possibly ADs in documents.

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 6 Nov 2019 13:01:37 +0000

That is a good idea.

I would also like to see a statement from the responsible AD that they have reviewed the document in detail and believe that it is fully ready to go to for review by people other than the WG and their own directorate. In some cases I really wonder it a draft has been read end to end by anyone before it is passed on for review by the wider community. 

- Stewart

> On 6 Nov 2019, at 11:42, Rob Wilton (rwilton) <[email-masked]> wrote:
>

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 6 Nov 2019 13:28:30 +0000

>    openid foundation comes to mind

I wonder how many members they have?  https://openid.net/foundation/benefits-members/  Again, a serious question.

---

From: Paul Wouters <[email-masked]>
Date: Wed, 6 Nov 2019 09:38:07 -0500 (EST)

> - encouragement for the reviewers to ensure that an adequate review has been performed based on the role/capacity in which they are acting.

I think I like this idea, but also, speaking for myself I would refuse
to review a few more documents of things that are too far out of my
area of expertise (which might be a good or bad thing, depending on
whether anyone else steps up)

Paul

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 6 Nov 2019 15:45:57 +0100

> We shouldn't be depending on last-minute quality checks to maintain the quality of our output.  Working groups should be producing documents that are ready to publish, and develop trust that their…

Yes.

But please do keep in mind that after having worked a couple of years on a document, the WG might no longer see things that a fresh look will uncover.  I think getting that fresh look is one of the great features of the multi-step review process we have.  Also, there is the simple fact of serendipity!  (Which is not very structured, and someone schooled in quality processes will hate, but creating quality is not really a controlled process anyway…)

Grüße, Carsten

---

From: Ralph Droms <[email-masked]>
Date: Wed, 6 Nov 2019 09:50:24 -0500

> On Nov 6, 2019, at 8:01 AM, Stewart Bryant <[email-masked]> wrote:
> 
> That is a good idea.
> 

I would support this idea if you s/responsible AD/WG chairs/.  The first point of responsibility here is to get *uniformly* high-quality docs out of working groups.  It's been a while since I was an AD, but I recall a wide variability in the quality of docs coming out of WGs.  Many docs were ready for review and required little or no feedback from my AD review.  But there were enough docs that required more extensive revision that a significant amount of my time was spent on managing those documents.

Perhaps part of the problem is insufficient negative consequences when a WG forwards docs to the IESG that aren't ready?  

- Ralph

---

From: Keith Moore <[email-masked]>
Date: Wed, 6 Nov 2019 09:52:34 -0500

> We shouldn't be depending on last-minute quality checks to maintain the quality of our output.  Working groups should be producing documents that are ready to publish, and develop trust that their…

As long as working groups are as siloed as they are, this is unlikely 
any WG that affects interests that are outside of its narrow scope.   
And it's very dangerous to assume that WGs are competent to produce high 
quality work in isolation at least with our current processes.   Though 
I'd certainly be in favor of some kind of explicit extra-WG review 
earlier in a WG's or document's lifespan.

Keith

---

From: Ralph Droms <[email-masked]>
Date: Wed, 6 Nov 2019 10:19:48 -0500

> On Nov 6, 2019, at 9:52 AM, Keith Moore <[email-masked]> wrote:
> 
> As long as working groups are as siloed as they are, this is unlikely any WG that affects interests that are outside of its narrow scope.   And it's very dangerous to assume that WGs are competent to produce high quality work in isolation at least with our current processes.   Though I'd certainly be in favor of some kind of explicit extra-WG review earlier in a WG's or document's lifespan.

We don't want to assume WGs are competent to produce high quality work, but we should certainly expect that they are.  WGs that regularly don't produce high quality work, perhaps, should not be supported in the IETF?

---

From: Michael Richardson <[email-masked]>
Date: Wed, 06 Nov 2019 10:23:49 -0500

Terry Manderson <[email-masked]> wrote:
    > That document
    > has to be proven before it gets to the rubber stamp process.

really?  For a Proposed Standard?
At this point, our PS quality way exceeds Internet Standard in quality.
An ID-WG-08 massively exceeds RFC quality of 20 years ago.

My opinion is that the problem is that WGs get shutdown way too soon, and
they are not expected to deal with the consequences of (their) poor documents.

    > How you get there will require a culture change more significant than
    > living through behaviours towards new participants, greater gender
    > diversity involvement, cultural sensitives, and and understanding of
    > the leadership/follower dynamic. The reason that it will be hard is
    > that the extant culture (while I accept and appreciate my predecessors)
    > is that it is built on the shoulders of legends. It’s now the time of
    > the IETF to have a championship team, rather than a team of
    > champions. Each member of that team needs to be responsible and
    > accountable for their contribution. The world is changing around us,
    > much faster than we can adapt. The IETF agility is, sadly, deserving of
    > being turbo charged, and you can’t do that with a hand-full of
    > brilliant people as the “fixers”.

I agree with you here.

    > Understand that while the IETF has a huge reputation, there are other
    > standards bodies that are doing this better, and faster, and resulting
    > in bigger improvements to technology.

I'm actually not sure that I agree.  Can you give me an example?

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Michael Richardson <[email-masked]>
Date: Wed, 06 Nov 2019 10:27:49 -0500

> allowed to join. We already struggle to recruit people at all levels of

Firing people is not fun, but just doing the job of the directorate hurts the
organization a lot more.

Maybe this is where we need some confidential performance reviews. Or maybe
the make-up of the Directorate could be confidential, so that there is no
shame in being fired.

(Or maybe make the reviews blind, so that nobody knows who is being
fired. But there are lots of other issues here)

---

From: Michael Richardson <[email-masked]>
Date: Wed, 06 Nov 2019 10:28:48 -0500

Rob Wilton (rwilton) <[email-masked]> wrote:
    > Perhaps RFCs could list (within the document) who reviewed/approved
    > them, and in which role/capacity the review had been performed.

+10.

    > - encouragement for the reviewers to ensure that an adequate review has been performed based on the role/capacity in which they are acting.

I've also suggested that the XML for the acknowledgement section be
structured so that we could extract this information better.

---

From: Keith Moore <[email-masked]>
Date: Wed, 6 Nov 2019 10:32:10 -0500

On 11/6/19 10:19 AM, Ralph Droms wrote:

> We don't want to assume WGs are competent to produce high quality work, but we should certainly expect that they are.

I'm not even sure what that means.   And that seems like a tricky wire 
to balance on.

Let me state the problem in a different way:  Most WGs labor for years 
to produce a few documents, sometimes reaching "rough consensus" mostly 
by exhaustion, and the level of effort invested is such that it takes 
tremendous will to completely reject their output even if it's horrible. 
   Fortunately such output is usually not horrible, but it's not unusual 
for the result to fail to consider negative effects, and/or for the 
documents to be poorly written.    At that point the group is too burned 
out to consider significant changes.

But "expecting" that the work is high quality, to justify weakening the 
reviews that such drafts are subjected to, seems counterproductive.   
And I agree that we should not depend on "heroes" either inside or 
outside the IESG, to provide documents with thorough reviews at a point 
that it's too late to fix most problems anyway.   So to me that means we 
need to do thorough, IETF-wide reviews earlier.

> WGs that regularly don't produce high quality work, perhaps, should not be supported in the IETF?

IMO most WGs should have a short (2-3 years) lifetime.   So by the time 
it's clear that they do or don't produce high-quality work, it's time 
for them to be winding down anyway.

Keith

---

From: Michael Richardson <[email-masked]>
Date: Wed, 06 Nov 2019 10:33:50 -0500

> whether anyone else steps up)

I think that this is certainly wise, but I also think that having reviews
From people who are completely cold is often good.  It depends; I've been
asked to review some extension to a TE-RSVP (for instance, probably got that
TLA wrong) document that basically just added two new attributes, and I can't
even comprehend the thing being extended, as it's an extension of something
else I don't know. I agree with you: there is little I can add other than
saying, "paragraph four seems awkward".
But that in itself has value, but it's not a useful review to the AD.

But there are other things you and I could come to cold, like (to use my
example above), that might be:
"An Architecture for doing Traffic Engineering with RSVP" where we might
actually be able to say, "Yes, I understood the problem, and I agree that the
solution seems to be in scale with the problem, and the security posture
seems appropriate"

---

From: Michael Richardson <[email-masked]>
Date: Wed, 06 Nov 2019 10:39:15 -0500

Carsten Bormann <[email-masked]> wrote:
    > But please do keep in mind that after having worked a couple of years
    > on a document, the WG might no longer see things that a fresh look will
    > uncover.  I think getting that fresh look is one of the great features
    > of the multi-step review process we have.  Also, there is the simple
    > fact of serendipity!  (Which is not very structured, and someone
    > schooled in quality processes will hate, but creating quality is not
    > really a controlled process anyway…)

1) We need these things to happen significantly before WGLC, and the WG has
   to not regard these things as hostile; this means that the review has to help.
   The reviewer has to then become involved in the changes that result.
   (This argues for the "stable document" statement, but I'd way prefer that
   we published stable-document as PS, using some lighter weight process,
   even if it winds up in a new series)

2) The AD for that directorate has to be involved such that the relevant AD
   does not just repeat the entire conversation.

3) We need the time from beginning the document process to the end of this to
   occur a bit faster as authors of a document simply get exhausted.  That
   probably argues for shorter documents.  It also argues for more hands to
   be involved, which might mean to me, lifting the 5 author limit.

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 6 Nov 2019 16:00:10 +0000

> On 5 Nov 2019, at 22:54, Brian E Carpenter <[email-masked]> wrote:
> 

Maybe we need a larger more formal review team that aims to have a few people read every draft before it gets onto IETF LC?

As I have said before I am not entirely convinced the ADs are reading their own drafts before they go to the telecast. I have a simple barometer for this, how can a draft ever get to IETF LC or directorate review with serious nits errors? That is machine checkable so there should be none of those. 
- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 6 Nov 2019 16:02:33 +0000

> On 6 Nov 2019, at 14:38, Paul Wouters <[email-masked]> wrote:
> 

I learn a lot from the out of area drafts that I get asked to review.

- Stewart

---

From: Michael Richardson <[email-masked]>
Date: Wed, 06 Nov 2019 11:54:20 -0500

> IMO most WGs should have a short (2-3 years) lifetime.   So by the time it's
    > clear that they do or don't produce high-quality work, it's time for them to
    > be winding down anyway.

I would like to amend this.  WGs should have a short (1-2) year iteration time.
That is, time from milestone being added to the charter to IESG Publication as PS.
I really think that WGs need to remain in existence longer in order to deal
with errata, to look at interop results, write operational documents, and
promote to IS.
That doesn't mean they have to meet in person very often, in fact, I would
discourage that.  There does have to be a venue for new people to join and
get to meet the existing people.  We need that kind of social interaction to
form cohesion, and to renew our energy and enthusiasm.
(Insert image of old-timer drinking blood of newbies...)

----

Notice how milestones today say, "Submit Document to IESG for Publication",
and that's all for the WG.  But, it's just the beginning.

There could be four to eight months of IESG level review that follows, which the
WG has no control over.

Let me put this another way, as this started with dearth of candidates to NOMCOM.
The list of AD candidates is effectively limited to the current list of WG chairs.
(I've said this before, multiple times)

WG chairs that aren't actively engaged with their WG and the authors to get
at good result that passes external review and AD review easily aren't going
to have time/energy/ability to be good ADs... yet...

Our mistake, however, is that having picked good WGs chairs to become ADs, we
basically Peter-principle them, removing them from what they are good at, and
ask them now to: *manage* and *mentor* new WGs chairs to be their successors.

Since most of the ADs were awesome WG chairs, it's hardly surprising that
they revert to what they were good at, and basically just take on the WG
chair responsability for document quality.

Afterall: that's the skill that they demonstrated so well that it got them "promoted".

---

From: Bob Hinden <[email-masked]>
Date: Wed, 6 Nov 2019 09:04:45 -0800

Ralph,

> On Nov 6, 2019, at 3:03 AM, Ralph Droms <[email-masked]> wrote:
> 
> 
> 

I agree.  This pushes most of the responsibility to the w.g. chairs, where IMHO it belongs.   This does scale with the number of working groups.

Bob

---

From: Bob Hinden <[email-masked]>
Date: Wed, 6 Nov 2019 09:10:47 -0800

Hi,

> On Nov 6, 2019, at 7:28 AM, Michael Richardson <[email-masked]> wrote:
> 
> 

All of this (reviewers, Shepard, AD reviews, ballots, etc.) is captured in the datatracker currently.   I am not sure why it would also need to be in the document.

Bob

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 6 Nov 2019 17:26:03 +0000

>    All of this (reviewers, Shepard, AD reviews, ballots, etc.) is captured in the datatracker currently.   I am not sure why it would also need to be in the document.
  
As incentive to do the best possible job.

---

From: Bob Hinden <[email-masked]>
Date: Wed, 6 Nov 2019 09:47:01 -0800

Rich,

> On Nov 6, 2019, at 9:26 AM, Salz, Rich <[email-masked]> wrote:
> 

Might be better to give out gold stars in the plenary to people who do a lot of good reviews.

Bib

---

From: Jared Mauch <[email-masked]>
Date: Wed, 6 Nov 2019 12:49:03 -0500

> As long as working groups are as siloed as they are, this is unlikely any WG that affects interests that are outside of its narrow scope.   And it's very dangerous to assume that WGs are competent…

I have to agree here.  There’s things that happen where some better cross-area would be useful, but we also face the fact that with a growing community it’s not possible to have all the implications in mind.  A subtle change in one area with one protocol or how that transport happens may have ops impact for the protocol.

A great example is that the large providers are often well represented but people who are still building services themselves just end up dealing with the changes vs having the bandwidth to participate with people whose full time job it is to e-mail the lists.

The subject line of this is also possibly quite problematic as it may be about one or more ADs today or just an abstract problem.  We should be cautious to not have this turn into painting the aircraft carriers bike shed while it turns lest we go overboard. :-)

- Jared

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 6 Nov 2019 17:53:37 +0000

>    Might be better to give out gold stars in the plenary to people who do a lot of good reviews.
  
The intent is that if you know your name is going to "forever" be associated with an RFC, you will probably be incented to do good work on it.

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 6 Nov 2019 19:28:07 +0000

> On 6 Nov 2019, at 17:26, Salz, Rich <[email-masked]> wrote:
> 

.. and as a thank to the reviewer and their sponsor.

- Stewart

---

From: Leif Johansson <[email-masked]>
Date: Wed, 6 Nov 2019 21:35:50 +0100

Skickat från min iPhone

> 6 nov. 2019 kl. 14:28 skrev Salz, Rich <[email-masked]>:
> 
> 

Whats a member and how many does the IETF have and how is that metric related to (say) the impact of an SDO?

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 6 Nov 2019 20:40:48 +0000

>     Whats a member and how many does the IETF have and how is that metric related to (say) the impact of an SDO? 

Do you always answer a question with a question? :)

Comparing sizes might show some interesting things.  For example, if an SDO is dominated by three big companies and everyone else is there just to get early access and do rubber-stamping, that doesn't fit the IETF model very well.

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 6 Nov 2019 21:09:58 +0000

>    Do you always answer a question with a question? :)
  
Off-line someone said this was unfair and could be taken as a personal attack.  I never meant that, and I apologize to you, Leif, if I was out of line.

---

From: Leif Johansson <[email-masked]>
Date: Wed, 6 Nov 2019 22:35:33 +0100

Skickat från min iPhone

> 6 nov. 2019 kl. 21:42 skrev Salz, Rich <[email-masked]>:
> 
> 
>> 

Did it work :-) Yes!

> Comparing sizes might show some interesting things.  For example, if an SDO is dominated by three big companies and everyone else is there just to get early access and do rubber-stamping, that does…

I think uptake and impact are more important metrics.

Cheers Leif

---

From: Leif Johansson <[email-masked]>
Date: Wed, 6 Nov 2019 22:35:58 +0100

Skickat från min iPhone

> 6 nov. 2019 kl. 22:10 skrev Salz, Rich <[email-masked]>:
> 
> 
>> 

Oh ffs :-)

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 7 Nov 2019 11:54:59 +1300

Here's a thought experiment.

Update the standards process such that the approval of Proposed Standard
RFCs, after an IETF last call including some specified cross-area review
requirements, is done by the WG consensus process with the consent of the AD .

Why would that work? Because it now incents the WG chairs by making them,
in effect, where the buck stops. So the WG chairs and AD (typically
a committee of three) will feel the obligation to get everything
right. And it scales.

Regards
   Brian Carpenter

---

From: Keith Moore <[email-masked]>
Date: Wed, 6 Nov 2019 18:01:09 -0500

> requirements, is done by the WG consensus process with the consent of the AD .

I don't think a typical WG chair is in a good position to review things 
from a broad perspective.   The ADs are in a MUCH better position to do 
that, precisely because they are exposed to everything that IETF does.   
Also, the WG chairs are properly concerned with the specific perspective 
of their WGs; they know where the hard battles were fought.   Their WG 
needs them to be in a position to defend the WG's work.   To expect them 
to do both that and the broad review would put them in a conflicted 
position, and it's probably the broad review that would get shortchanged.

Keith

---

From: Keith Moore <[email-masked]>
Date: Wed, 6 Nov 2019 19:01:52 -0500

On 11/6/19 11:54 AM, Michael Richardson wrote:

> Keith Moore<[email-masked]>  wrote:

I can see pros and cons with this.   IMO one of the problems with 
long-lived WGs is that they can keep creating more work for themselves 
whether or not it is needed.   Sure, they need to update their charters 
to do so but I've rarely seen those efforts be rebuffed.

I also understand that the more WGs there are, the more load there is on 
the ADs, even if many of those WGs are dormant.

So I think it might make more sense to have a specific long-term WG or 
two per area to focus on maintenance, than to keep the narrowly-focused 
WGs around indefinitely.   Such a "maintenance" WG might also be tasked 
with periodically reviewing each of the current standards-track 
documents in that area, looking at the state of deployment and 
suitability for purpose, and making recommendations for updating, status 
change, new documents (e.g. usage profiles or applicability statements), 
or deprecation as appropriate.

> (I've said this before, multiple times)
I don't think this is actually true.   It's closer to the set of people 
who have chaired a WG at some time in the present or past. And I think 
it would be fair to consider directorate members in some cases.
> chair responsability for document quality.

While that might happen in some cases, I don't think that's a good 
explanation for ADs taking on detailed review.   ADs know that they're 
in a unique position to uphold the quality of IETF standards.   They see 
the widely varying quality that WGs produce, and they know how important 
it is to get some of that stuff right.   (Some of it is relatively 
inconsequential.)   And in practice, an AD actually has fairly limited 
power to make sure that a WG reliably produces good output.   The AD can 
in theory fire poor chairs, but this has the potential to be so 
disruptive as to make things worse, and there may be a dearth of good 
chairs to replace them.   It's all well and good (and politically wise) 
to get directorate members to do detailed reviews, but an AD doesn't 
always have that luxury, and sometimes an AD has to dig fairly deeply 
into a document in order to make sense of the (sometimes conflicting) 
reviews and Last Call comments anyway. The buck really does stop with 
the IESG and IMO they're the ones in the best positions to be making the 
final decisions - partly because they are likely to have broad views, 
and partly because about the only thing that gives them any "steering" 
capability is the fact that they're the ones doing the final signoff.

> Afterall: that's the skill that they demonstrated so well that it got them "promoted".

I believe that AD and WG chair require somewhat different skill sets.   
If I were on nomcom I'd be looking for ADs who have demonstrated cat 
herding expertise and are conversant in multiple technical subject 
areas.   A WG chair can be a specialist; an AD needs to be a generalist.

Keith

---

From: Keith Moore <[email-masked]>
Date: Wed, 6 Nov 2019 19:03:32 -0500

> The intent is that if you know your name is going to "forever" be associated with an RFC, you will probably be incented to do good work on it.

No matter how much care you take, if you're around long enough to see 
your work used for decades, there will probably be some embarrassing 
"gold stars" in your past.

Keith

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 7 Nov 2019 13:43:17 +1300

> that, precisely because they are exposed to everything that IETF does.

But it doesn't scale, or so the ADs are telling us. And my thought
experiment wouldn't take ADs out of the loop; it would take them out
of the detailed review work.
   
> position, and it's probably the broad review that would get shortchanged.

That's why the idea would be to make the WG chairs the *visible* approvers,
which IMHO would significantly change their incentives. I certainly agree
with the idea of naming the WG Chairs, the AD and the reviewers in the
published RFC: "Here are the people you should complain to: ..." just as
much as "Here are the people who share the glory: ...".

   Brian

---

From: Keith Moore <[email-masked]>
Date: Wed, 6 Nov 2019 20:20:15 -0500

On 11/6/19 7:43 PM, Brian E Carpenter wrote:

> But it doesn't scale, or so the ADs are telling us.

I believe them :)

Or to put it differently:  I don't think it scales so long as there is 
essentially no pushback on chartering any WG for which there appears to 
be community interest.

Note that there's not a contradiction there.  The ADs really are in the 
best position to review things from a broad perspective. AND with the 
current number of working groups and number of drafts that they're 
producing, there's too much work for the ADs to do.

But to me it looks like what doesn't scale is the community expectation 
that IETF should take on every WG for which there seems to be public 
interest, and those WGs should be able to produce as many drafts as they 
wish, with no page limit on those drafts.

(I'm not saying that there can't be some middle ground, some 
optimization of what's currently being done.   But I do believe that the 
organization needs to take a good hard look at its expectations for 
output and for "fairness" in what IETF takes on, and think about what 
the Internet needs from IETF rather than about what individuals and 
companies want IETF to do for them.)

It's long been clear to me that there's an inherent limit to the extent 
to which the IETF structure can scale, for several reasons other than 
just AD workload.   And to me that forces some hard questions about both 
the volume and scope of IETF's work.     IMO IETF should focus on 
producing fewer but higher quality and more relevant RFCs, rather than 
priding itself on the sheer number of RFCs produced.

> And my thought

It would also take away the tremendously valuable perspective that they 
acquire as the result of doing (some of) that work.   We would soon be 
left without anyone who could really do a good job at broad review.

Or maybe the thing to do is to make directorates more formal and more 
recognized, and expand their roles a bit.   Maybe we should explicitly 
look at (some?) directorates as the places to cultivate broad expertise 
(others might still have narrower focus).   Then have the directorates 
tasked with not just reviewing documents and making recommendations 
about them, but also other things - e.g. analyses of problem spaces and 
tussles - that would inform AD decisions and also feed-forward to WGs.   
The buck would still stop with IESG but they'd have more explicit support.

> which IMHO would significantly change their incentives.

Yes, but then they wouldn't be in a good position to support the WGs.   
A chair can't be both in the trenches in a WG AND be a reasonably 
objective reviewer from a broad perspective.   Either they're separate 
people, or we just give up on our notion of rough consensus because 
nobody would be in a good position to evaluate it.

Keith

---

From: Nico Williams <[email-masked]>
Date: Wed, 6 Nov 2019 19:48:49 -0600

> a committee of three) will feel the obligation to get everything

So, no more IESG review?  What would we need the IESG for anymore?  It
would be gone, I guess?

Sure, it will scale better.  But quality will suffer.

> [...]. So the WG chairs and AD (typically a committee of three) [...]

Typically one of the ADs is uninvolved with a WG for which the other is
responsible, so that would be a committee of two, not three.

Nico
--

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 7 Nov 2019 15:25:40 +1300

> So, no more IESG review?  What would we need the IESG for anymore?  It

Not at all. Firstly, I said *Proposed* Standard. The experiment would leave
the IESG in charge for Internet Standard. (Also for documents submitted
outside the WG process, which is rare of course.) And the criterion is
still IETF rough consensus following an IETF last call. Just trying
to eliminate a manifest bottleneck.

Secondly, they'd still be the authority for chartering, appointing
WG chairs, appointing directorates, and other aspects of overseeing
the process. Most of the duties described in RFC 2026 in fact.

> Sure, it will scale better.  But quality will suffer.

And maybe some documents will spend fewer than, say, 2 years in MISSREF.
Speed vs quality is indeed a trade-off. In the era of rapid prototyping
our timescale really looks pretty stupid.

> responsible, so that would be a committee of two, not three.

Two WG chairs is almost the norm these days. But yes, the essence of
the experiment is to reduce the number of people in the decision
process - and intentionally lower the bar for Proposed Standard closer
to how it's defined in RFC2026:
   A Proposed Standard specification is generally stable, has resolved
   known design choices, is believed to be well-understood, has received
   significant community review, and appears to enjoy enough community
   interest to be considered valuable.  However, further experience
   might result in a change or even retraction of the specification
   before it advances.
When I look at some the DISCUSSes I've seen in recent months, I really
don't think that definition is being applied.

I would also add to my thought experiment a mandatory Implementation
Status section. That's stronger than RFC2026.

 Brian

---

From: Keith Moore <[email-masked]>
Date: Wed, 6 Nov 2019 21:50:23 -0500

> to how it's defined in RFC2026:

But 2026 criteria assumed a three-stage standards track.   Now we're 
down to two stages in theory, and still usually just one stage in 
practice.   As far as I can tell, getting rid of Draft Standard hasn't 
done anything to make Proposed Standard less of a terminal state and 
more of a stepping-stone.

>     might result in a change or even retraction of the specification
RFC2026 also says:

    Implementors should treat Proposed Standards as immature
    specifications.  It is desirable to implement them in order to gain
    experience and to validate, test, and clarify the specification.
    However, since the content of Proposed Standards may be changed if
    problems are found or better solutions are identified,/*deploying implementations of such standards into a 
disruption-sensitive environment is not recommended.*/

(emphasis mine)

This is where the biggest disconnect between 2026 and reality is.   If 
the reality is that industry is going to deploy implementations at 
Proposed Standard or sooner (and as far as I can tell, that's been 
reality for as long as there's been an Internet "industry"), it makes 
sense for IETF to recognize that and react accordingly.

If we want there to be a prototype "just for testing" status, it should 
probably be called something other than Proposed - the name has come to 
mean something else in IETF context.   And we should deliberately change 
one or more protocol elements to make the standard incompatible with 
already-deployed implementations.   The trick would be to keep that 
information from leaking out beforehand.    Maybe give the RSE /N/ 
options for things to change that are selected at random, similarly to 
the way nomcoms are selected.

Also, the WG completion, IESG approval and RFC publication dates of a 
new standard should be determined well in advance so that vendors can 
tailor their product development schedules to those dates.   This would 
mean holding WGs, IESG, and the RSE to a schedule (gasp!) once there was 
confidence that the document would be standardized (another argument for 
broad review well prior to Last Call.)  I'm not saying that there could 
absolutely be no slippage, but reality should be close enough that 
developers can plan by it.

But my guess is that the only way to get buy-in for Prototype 
specifications would be to convincingly promise faster development.

Keith

---

From: Nico Williams <[email-masked]>
Date: Wed, 6 Nov 2019 23:02:22 -0600

> to eliminate a manifest bottleneck.

When most of our documents are PS -when PS is the new Standard!- this
really does mean doing away with IESG review.  Nothing obligates anyone
to do the work to move PS to DS and then S.

> the process. Most of the duties described in RFC 2026 in fact.

Sure, but picking good chairs now becomes essential, but finding chairs
as good as ADs will be difficult.

> our timescale really looks pretty stupid.

Perhaps so.  But then we'd better find a way to incentivize moving PS
docs to DS and S, or give up the fiction.

> don't think that definition is being applied.

That's a fair observation.  But, too, we could ask the IESG to take a
lighter approach to publication as PS.

> Status section. That's stronger than RFC2026.

Implementation Status for PS?  But that will drag us into the I-D
stability thread all over again -- it's a catch-22: how can we develop a
protocol, implement it, and test it but not deploy it (despite a very
strong market desire for it) without some sort of I-D stability
mechanism?

Nico
--

---

From: Leif Johansson <[email-masked]>
Date: Thu, 7 Nov 2019 06:34:28 +0100

Skickat från min iPhone

> 7 nov. 2019 kl. 02:49 skrev Nico Williams <[email-masked]>:
> 

Why do you think that would be the case?

---

From: Bob Hinden <[email-masked]>
Date: Thu, 7 Nov 2019 08:29:19 -0800

Nico,

> On Nov 6, 2019, at 9:02 PM, Nico Williams <[email-masked]> wrote:
> …..

I agree that picking good chairs is essential (it is now too), but a w.g. chair’s job is not full time.  They only have to manage and review the documents in their working group.   The biggest issue with finding qualified people to be ADs is that it is essentially a full time job.  Finding qualified people to be w.g. chairs is easier and there is a bigger pool of qualified people because it isn’t a full time job.

Bob

---

From: Bob Hinden <[email-masked]>
Date: Thu, 7 Nov 2019 08:42:59 -0800

Keith,

> On Nov 6, 2019, at 3:01 PM, Keith Moore <[email-masked]> wrote:
> 
> I don't think a typical WG chair is in a good position to review things from a broad perspective.   The ADs are in a MUCH better position to do that, precisely because they are exposed to everything that IETF does.   Also, the WG chairs are properly concerned with the specific perspective of their WGs; they know where the hard battles were fought.   Their WG needs them to be in a position to defend the WG's work.   To expect them to do both that and the broad review would put them in a conflicted position, and it's probably the broad review that would get shortchanged.

I suspect that for a lot of documents, broad cross area review beyond what Brian proposes is not necessary.   Not for all, but many things that working groups produce are very incremental.   We could have two tracks, one as Brian proposes, and another where the AD thinks more review is needed.

I suspect this would significantly reduce the load on the IESG and would not reduce quality.

Bob

---

From: Keith Moore <[email-masked]>
Date: Thu, 7 Nov 2019 11:56:57 -0500

> I suspect that for a lot of documents, broad cross area review beyond what Brian proposes is not necessary.   Not for all, but many things that working groups produce are very incremental.   We cou…

Sure, I agree that broad cross area review isn't necessary for every 
document.  Though I don't think the important factor is whether the work 
is "incremental", but rather, whether it affects interests beyond the 
narrow scope of the WG itself.   But even with the current process, I 
also don't think that an AD needs to read every document if it doesn't 
touch on his or her area.   So I don't think we need to change the 
process just to allow documents that don't need broad review to not be 
subjected to it.  I just assume that that fewer ADs will concern 
themselves with such documents, and for the documents that have less 
potential to wreak havoc, ADs will be more willing to trust reviews from 
directorate members and others in the community.    Voting "no 
objection" is pretty easy, after all, and ADs are busy enough that 
there's plenty of incentive to avoid unnecessary work.

> I suspect this would significantly reduce the load on the IESG and would not reduce quality.

I don't think it would reduce the load by much, since I assume the 
workload is already reduced for these documents.   And it should never 
be up to "the AD" (as in singular) to decide how broad review should 
be.   That judgment needs to be made by several people from different 
areas, even if the AD, or ADs for the area of the document in question, 
don't see the potential adverse impact.

Keith

---

From: tom petch <[email-masked]>
Date: Thu, 7 Nov 2019 17:08:25 +0000

<inline>

----- Original Message -----
From: "Bob Hinden" <[email-masked]>
To: "Nico Williams" <[email-masked]>
Cc: "IETF" <[email-masked]>; "Michael Richardson"
<[email-masked]>; "Bob Hinden" <[email-masked]>; "Alexey
Melnikov" <[email-masked]>
Sent: Thursday, November 07, 2019 4:29 PM

Nico,

> as good as ADs will be difficult.

I agree that picking good chairs is essential (it is now too), but a
w.g. chair’s job is not full time.  They only have to manage and review
the documents in their working group.   The biggest issue with finding
qualified people to be ADs is that it is essentially a full time job.
Finding qualified people to be w.g. chairs is easier and there is a
bigger pool of qualified people because it isn’t a full time job.

<tp>
And yet the quality of chairs seems lacking which then puts a greater
load on the ADs.

A general principle of doing anything is that the earlier you fix the
issue, the less expensive it is to fix so a defect that is fixed just
after WG adoption will cost less effort than one that gets fixed at IESG
review.

I see comments from myself and others not acted on even when they are
(to me) basic such as line lengths of 100+ characters which makes them
hard to read; or an absence of any normative references which means I
cannot tell if I should expect to understand this I-D; an IANA section
that registers 'n' YANG modules when the I-D contains 'm' YANG modules
where n>>m or m>>n.

These tend to get fixed before IETF Last Call but less basic ones may
not be; the issues were raised on the WG list and then the person
raising them says 'What about the issues I raised  two months ago and
which are still there?'

I am pointing a finger at WG Chairs who let unresolved issues run and
run; that costs, everyone.  And I see this in several of the WGs that I
track, as well as never seeing it in others; I believe that the
difference lies in the WG Chairs but all I can do is post comments to
the WG List - when they are not acted on, then I am ....ed

Tom Petch

Bob

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 Nov 2019 12:25:20 -0500

> All of this (reviewers, Shepard, AD reviews, ballots, etc.) is captured
    > in the datatracker currently.   I am not sure why it would also need to
    > be in the document.

Because the datatracker is an ephermeral (by historic scales of time)
database, while the RFCs are etched in stone.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 Nov 2019 12:26:22 -0500

> .. and as a thank to the reviewer and their sponsor.

Also, the RFC will capture the sponsor *at the time*, while the datatracker
will tell you their sponsor is *today*

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 7 Nov 2019 19:28:21 +0000

> On 7 Nov 2019, at 17:25, Michael Richardson <[email-masked]> wrote:
> 
> 
.. and fundamentally it is always nice to be thanked for, in some cases, hours of reading, and that in turn encourages reviewers to do more reviews, and demonstrates to their manager that they doing something the IETF appreciates.

- Stewart

---

From: Nico Williams <[email-masked]>
Date: Thu, 7 Nov 2019 13:33:15 -0600

> Why do you think that would be the case?

Well, it doesn't necessarily follow that fewer eyeballs / less review ==
poorer quality, but it is a fair assumption.

I don't know what the right thing to do here is, and we might have to
experiment somewhat.  Can we run an experiment where for some WG(s) we
don't require IESG review of their PS documents?

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 7 Nov 2019 13:44:08 -0600

On Wed, Nov 06, 2019 at 09:50:23PM -0500, Keith Moore wrote:
> But 2026 criteria assumed a three-stage standards track.   Now we're down to
> two stages in theory, and still usually just one stage in practice.   As far
> as I can tell, getting rid of Draft Standard hasn't done anything to make
> Proposed Standard less of a terminal state and more of a stepping-stone.
> 
> [...]
> 
> This is where the biggest disconnect between 2026 and reality is.   If the
> reality is that industry is going to deploy implementations at Proposed
> Standard or sooner (and as far as I can tell, that's been reality for as
> long as there's been an Internet "industry"), it makes sense for IETF to
> recognize that and react accordingly.

+1

> probably be called something other than Proposed - the name has come to mean
> something else in IETF context.   And we should deliberately change one or

We have it: it's Experimental.  But it's essentially seen as a negative,
thus underused.

> more protocol elements to make the standard incompatible with
> information from leaking out beforehand.    Maybe give the RSE /N/ options
> for things to change that are selected at random, similarly to the way
> nomcoms are selected.

Are you proposing that there be a required backwards-incompatible change
so as to force... what exactly?  That the old thing does not get
deployed?  I assume it will mostly discourage use of the new track.

The risk with deploying prototypes is that a backwards-incompatible
change will be needed, so perhaps early reviews should focus only on
identifying features at risk of such changes and requiring that the
experimental protocol cover upgrades or reduce the risk.

> Also, the WG completion, IESG approval and RFC publication dates of a new
> standard should be determined well in advance so that vendors can tailor
> their product development schedules to those dates.   This would mean
> holding WGs, IESG, and the RSE to a schedule (gasp!) once there was

Maybe, yeah, for high-profile protocols (e.g., TLS).

> But my guess is that the only way to get buy-in for Prototype specifications
> would be to convincingly promise faster development.

Yes.  That includes faster RFC-Editor turnarounds.  If we remove the
other bottlenecs, then RFC-Editor queue time will become the next
bottleneck to address.

Nico
--

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 Nov 2019 15:07:43 -0500

> a committee of three) will feel the obligation to get everything

I was not a fan of the proposal to designate an Internet-Draft as a stable
document, because I felt that it was basically creating a new level of
document status. (Another hurdle)

Your proposal is almost the same, but I like it because I feel that instead
of adding a hurdle, it is removing one.

Would you consider if these new documents are *RFC*s or, would you consider
if we could make a new document series for these documents? I would suggest
that it become the *Proposed Standard* series.  That is, we'd change our
first step to not be an *RFC*.

I believe that this would solve the problems that the stable-document
proponents were trying to create, while not adding a new step to the process.

I would want such RFCPS or WGPS, etc. to go through the RPC.
This fixes all the english, references, etc. issues, and of course causes
IANA assignments.

Upon being finished processing, the exact XML would be returned to the WG to
collect errata, interop experience and clarification, setting things up for
the "bis" document that would then be ready for Internet Standard (RFC).

This also synchronizes our notion of what an RFC is with what the industry
thinks is an RFC.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 Nov 2019 15:14:16 -0500

Keith Moore <[email-masked]> wrote:
    > essentially no pushback on chartering any WG for which there appears to be
    > community interest.

    > Note that there's not a contradiction there.  The ADs really are in the best
    > position to review things from a broad perspective. AND with the current
    > number of working groups and number of drafts that they're producing, there's
    > too much work for the ADs to do.

My impression from the regular plenary reports on attendance my impression is
that the number of people involved in the IETF is declining, yet the number
of RFCs being processed is going up.

And the number of conflicts seems to be going up because so many more people
are involved in quite a range of WGs.   How does this compare to ten years
ago?  I'm not even sure I know what number I want to ask for here.

Jari, can you tell if we have more unique authors from your stats system?
Are we individually producing more documents?  Are we collaborating across
communities more often, which is why we have more conflicts?

It seems like it must be that more of the people involved are writing
documents.  Did we have more people who were exclusively "tourists" before?

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 Nov 2019 15:16:50 -0500

> So, no more IESG review?  What would we need the IESG for anymore?  It

IESG review for Internet Standard, not PS.

    > Sure, it will scale better.  But quality will suffer.

More documents promoted to IS.
More time to recruit, manage and train WG chairs. I.e. MANAGING

    > responsible, so that would be a committee of two, not three.

Two chairs + 1 AD.
It would ideally also involve a few directorate reviewers.

---

From: Nico Williams <[email-masked]>
Date: Thu, 7 Nov 2019 14:16:42 -0600

> first step to not be an *RFC*.

Now this is appealing.  Perception counts for a lot.  "RFCxxxx" == "Standard"
in many minds.  So giving PS docs a PSRFCxxxx w/o a corresponding RFCxxxx
would help a lot.

> This also synchronizes our notion of what an RFC is with what the industry

+1.

Nico
--

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 Nov 2019 15:19:36 -0500

Keith Moore <[email-masked]> wrote:
    > Standard or sooner (and as far as I can tell, that's been reality for as long
    > as there's been an Internet "industry"), it makes sense for IETF to recognize
    > that and react accordingly.

You are saying this as if it's a bug.
It's not!  It's by design.
We deploy at PS in order to find out if there is interoperability.

    > more protocol elements to make the standard incompatible with

We do "just for testing" regularly at the internet-draft stage.

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 7 Nov 2019 20:19:36 +0000

Attendance is perhaps declining, but I don't recall any discussion of WG email traffic, for example.

---

From: Keith Moore <[email-masked]>
Date: Thu, 7 Nov 2019 16:14:54 -0500

> We have it: it's Experimental.  But it's essentially seen as a negative,

Experimental doesn't imply that there's an intention to standardize a 
protocol, though this does happen in rare cases.

Though now that I think about it, the right way to denote prototype 
status is probably to designate specific Internet-Drafts rather than to 
publish them as RFCs.   RFC publication isn't designed for temporary 
documents.

> deployed?  I assume it will mostly discourage use of the new track.

The purpose is to remove some of the incentives to deploy protocol 
implementations before they're approved, while still allowing 
implementations and meaningful interoperability tests. When things are 
prematurely deployed it puts pressure on the WG and/or IESG to accept 
specifications that have significant flaws. It can also harm IETF's 
reputation when implementations don't meet the specification or don't 
interoperate.

> experimental protocol cover upgrades or reduce the risk.
IMO one of the primary reasons for early reviews is to learn as early as 
possible whether an intended direction or protocol design choice would 
create or exacerbate tussles that could be addressed, and/or have 
adverse effects on other interests that might compel a different 
direction or choice.    It goes beyond "identifying features" because 
it's not always individual features that cause the conflicts.
> other bottlenecs, then RFC-Editor queue time will become the next

I was thinking earlier today that the biggest waster of time and energy 
might be when IETF invests years of volunteer effort on specifications 
that turn out to be not very useful.   Even worse might be when IETF 
increases complexity (of implementations, operations, etc.) that doesn't 
result in a tremendous benefit to the Internet user community.

(It's easy to pay more attention to things that are easily measured, 
than those that are not, even though the easily measured quantities 
might not be the most relevant ones.)

Keith

---

From: Keith Moore <[email-masked]>
Date: Thu, 7 Nov 2019 16:17:12 -0500

On 11/7/19 2:33 PM, Nico Williams wrote:

> Can we run an experiment where for some WG(s) we

Why in the world would we want to do that?

Keith

---

From: Keith Moore <[email-masked]>
Date: Thu, 7 Nov 2019 16:36:25 -0500

> We deploy at PS in order to find out if there is interoperability.

My first impression is that this is indeed a bug, a tremendous 
disservice to Internet users.   But I remind myself that automatic 
software update is becoming increasingly common.   So at least for 
products that are certain to be run on Internet-connected hosts (as 
opposed to, say, on air-gapped networks), and for which secure update 
can be provided, there might be cases for which premature deployment 
makes sense - especially if there's a strategy designed into the 
protocol for dealing with version skew in the endpoints' 
implementations.   (Assuming public internet connectivity isn't at all 
the general case, but it's an important corner case.)

Also, PS criteria were specifically not designed for deployment; the 
idea was that you need interop testing before deploying.   But if 
interop testing were incorporated earlier in the process (which does 
seem to be more common these days) then the sequencing assumed by 2026 
might be suboptimal.

I could imagine that rather than the initial RFC being at PS, it could 
be (for some cases, probably not all) at something akin to what Draft 
Standard used to be - interop testing already done by the time the 
initial RFC is published, with any changes made as a result of the 
testing incorporated into that RFC.

> We do "just for testing" regularly at the internet-draft stage.

yeah, and it doesn't make sense to go through the whole RFC publication 
process just to agree on a specification to test to.

Keith

---

From: Keith Moore <[email-masked]>
Date: Thu, 7 Nov 2019 16:41:20 -0500

On 11/7/19 3:07 PM, Michael Richardson wrote:

> that it become the*Proposed Standard*  series.  That is, we'd change our
> first step to not be an*RFC*.

At first glance I find this idea appealing.   I'd like to see it 
explored.   At lot depends on other factors - e.g. when, relative to 
interop testing, do we look for potential harm to other interests?

(though maybe we don't need a new document series - maybe we just need a 
way of designating certain Internet-Drafts as being suitable for interop 
testing and/or limited deployment)

Keith

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 Nov 2019 20:03:58 -0500

> At first glance I find this idea appealing.   I'd like to see it explored.  
    > At lot depends on other factors - e.g. when, relative to interop testing, do
    > we look for potential harm to other interests?

When do we do that now?  At IESG review, which is what causes such
significant delays.

That's just too late in the game for an *RFC*, which we can never really take
back.  The property that you like about internet-drafts is that they are
in principal ephermeral, or more to the point, version NN replaces NN-1,
and we can revise them.  That's the property you are looking for, I think.

    > (though maybe we don't need a new document series - maybe we just need a way
    > of designating certain Internet-Drafts as being suitable for interop testing
    > and/or limited deployment)

That was proposed a few months ago. See:
  https://mailarchive.ietf.org/arch/msg/ietf/zy2l7gWR8yGRIIt5mYY5WSorqGY

I prefer that we create a new series.  Maybe it shouldn't be hosted on *RFC-EDITOR.ORG*,
but I'd want the identical infrastructure used.

(I would name it after the WG involved.  WG-FOOBAR-1234, but I think this is
a detail, and I'm not stronly attached)

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 Nov 2019 20:05:09 -0500

> WG email traffic, for example.

So the number we need to know is what is the trend for RFCs being processed
by a set of authors who are exclusively remote.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 Nov 2019 20:07:10 -0500

> My first impression is that this is indeed a bug, a tremendous disservice to
    > Internet users.   But I remind myself that automatic software update is
    > becoming increasingly common.   So at least for products that are certain to
    > be run on Internet-connected hosts (as opposed to, say, on air-gapped
    > networks), and for which secure update can be provided, there might be
    > cases

An alternative view is that air-gapped systems should only run Internet
Standards.

    > I could imagine that rather than the initial RFC being at PS, it could be
    > (for some cases, probably not all) at something akin to what Draft Standard
    > used to be - interop testing already done by the time the initial RFC is
    > published, with any changes made as a result of the testing incorporated into
    > that RFC.

I think that we kept the wrong two steps!

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 8 Nov 2019 14:31:08 +1300

One more comment and then an alternative thought experiment:
On 08-Nov-19 08:44, Nico Williams wrote:
...
> other bottlenecs, then RFC-Editor queue time will become the next

In my experience (i.e. I have not run stats) the long delays are mainly
caused by missing normative references, i.e. are a result of IETF rules
and IETF slowness, not a problem caused in any way by the RFC service.
The quickest way to improve the RFC queue statistics would be to abolish
the rule that I-Ds cannot be used as normative references.

Anyway: my new experiment would be one that the IESG could decide to
start tomorrow. It's simply that the IESG would only ever issue one form
of DISCUSS ballot, which would look like this:

Pat Areadirector has entered the following ballot position for
draft-ietf-somewg-somedraft-99: Discuss
...
----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

There are still open issues from the following reviews:
<links to reviews>

In other words, the IESG simply busy-waits until all review issues
have been resolved, rather than finding and fixing the issues
personally.

(If an AD wishes to post a review for a given draft, that would be
a personal choice, not part of the IESG workload.)

This is a procedural change, and would not prevent a substantive
DISCUSS in unusual circumstances.

   Brian

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 Nov 2019 20:39:48 -0500

> There are still open issues from the following reviews:

okay, interesting idea.

    > have been resolved, rather than finding and fixing the issues

Let's be clear: that means that reviewers are expected to engage with authors
to get the issues resolved.  This might surprise some reviewers.

---

From: Bob Hinden <[email-masked]>
Date: Thu, 7 Nov 2019 17:42:19 -0800

Michael,

> On Nov 7, 2019, at 5:39 PM, Michael Richardson <[email-masked]> wrote:
> 
> 

Or better, the Document Shepard takes an active role to get the issues resolved.

Bob

---

From: Keith Moore <[email-masked]>
Date: Thu, 7 Nov 2019 20:43:14 -0500

> When do we do that now?  At IESG review, which is what causes such

I'm frankly not sure that cross-area review causes significant delays or 
even that IESG review causes significant delays.   I could argue instead 
that what causes those delays is working groups producing documents that 
are not suitable for the intended status.   When WGs produce broken 
documents that are hard to fix, this can be very time consuming for 
IESG.   In practice, IESG is expected to find some path to approving 
such documents. Sometimes there's just no good fix, and even poor fixes 
can be quite time consuming to come up with.

In my experience, well-written, thorough specifications are generally 
relatively easy to review; the documents that are overly long or poorly 
written (so it's hard to tell what is really intended, or whether the 
protocol will work reliably) or that seem to be potentially harmful are 
much harder and more time consuming to review.   So people shouldn't 
automatically assume that "delays in IESG" are IESG's fault.

But I will certainly agree that by the time a document gets to Last Call 
and after, it's usually too late to fix fundamental design flaws - and I 
would count failure to research and consider interests that might be 
harmed as a design flaw.    And yet, I don't think narrowly-focused WGs 
can be entrusted to consider a broad spectrum of interests - that really 
has to be an IETF-wide review.

> and we can revise them.  That's the property you are looking for, I think.

Close but maybe not quite.   Fundamentally, a specification that's being 
interoperability tested _should_ be an ephemeral version.   The whole 
purpose in doing the testing is the realization that it will likely need 
to be changed based on implementation and testing.   And the whole 
approval and RFC process takes too long to wait for it to be done before 
testing - it would basically cause a pipeline stall if people did it 
that way.   Testing at PS might have made sense long ago when there were 
fewer interests at stake, fewer procedures to follow, and the 
publication process was lightweight - but even then I don't think the 
testing that was anticipated between PS and DS happened very often.   
Also, external conditions have changed a lot since 2026 was written.  
Testing over the public Internet, or setting up a private testing VPN, 
are more practical than they used to be. Group collaboration tools can 
be used effectively to coordinate tests today whereas face-to-face 
meetings were once considered a practical necessity.    Computers are 
faster, have more memory, languages are better, and cloud resources can 
be used to simulate load.     And the general expectation is that 
product iteration cycles are much tighter than was once the case.   It 
makes sense to adapt IETF processes to suit these new realities.

Implementation concurrent with specification development, and testing 
_before_ IESG review make a LOT of sense, IMO.   I think it would shake 
out bugs that are difficult for IESG to see, and favorable testing 
results would give IESG more reason to have confidence that the protocol 
actually works.

To me it looks like that with a judicious reordering of constraints, we 
should be able to make time-to-completion more predictable AND produce 
higher quality documents.

>    https://mailarchive.ietf.org/arch/msg/ietf/zy2l7gWR8yGRIIt5mYY5WSorqGY

The devil is in the details.   I read that proposal as subtly different 
but in an important way.   Marking a draft as "stable" is to me a very 
different thing than marking it as "here's a version we're going to 
subject to interoperability tests".   It's perfectly reasonable in my 
mind to test portions of a protocol that there's general agreement on, 
even while knowing that some new features will need to be added before 
the protocol is ready for Last Call.

> but I'd want the identical infrastructure used.

I'm not sure we actually disagree on this aspect, but just in case we 
do:  I don't see why a process as heavyweight as RFC publication is 
needed for this.   It seems to me that except for the minor change of 
being able to mark internet-drafts with a few specific attributes, and 
the ability of the tools to search for such drafts by those attributes 
(maybe a few automatically generated pages), the existing I-D 
publication process is about right.   But I'd be interested in hearing 
specific reasons why this isn't the case.

Keith

---

From: Keith Moore <[email-masked]>
Date: Thu, 7 Nov 2019 20:54:20 -0500

> draft-ietf-somewg-somedraft-99: Discuss

To me the problems seem like: (a) getting sufficient reviews done, and 
(b) establishing review criteria well in advance, not only for the 
reviewers but also as feed-forward to the WG and document editors.  (not 
that reviewers couldn't identify issues outside of those criteria, but 
having a somewhat-shared view of those criteria would make it easier to 
compare different reviews for example)

I keep looking for ways to identify these criteria earlier in the 
process.   Not that there would never be last-minute showstoppers - 
we're always learning new things.   But if we could reduce the number of 
late surprises that would make everything easier IMO.

Keith

---

From: Nico Williams <[email-masked]>
Date: Thu, 7 Nov 2019 21:52:19 -0600

> draft-ietf-somewg-somedraft-99: Discuss

Reviewers would be responsible for making sure that ADs find their
reviews.  WGs/ADs/shepherds should help make sure reviewers know this.

I'm OK with this!

Keith's point about insufficient review can still be addressed by having
shepherds notice and report this (and ADs notice this independently),
thus triggering additional AD/directorate review.

+1

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 7 Nov 2019 21:54:09 -0600

> Or better, the Document Shepard takes an active role to get the issues resolved.

The shepherd already has to write a summary, including a summary of
review obtained among other things, so we already have the right process
for making sure reviews don't get dropped.

But sure, WG chairs, ADs, and shepherds can also make sure to educate
reviewers.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 7 Nov 2019 22:01:08 -0600

On Thu, Nov 07, 2019 at 04:36:25PM -0500, Keith Moore wrote:
> >      > This is where the biggest disconnect between 2026 and reality
> >      > is.   If the reality is that industry is going to deploy
> >      > implementations at Proposed Standard or sooner (and as far as
> >      > I can tell, that's been reality for as long as there's been
> >      > an Internet "industry"), it makes sense for IETF to recognize
> software update is becoming increasingly common.   [...]
> 

Progressing from PS->DS->Standard requires reporting on interop.
Publishing a DS does not.

IETF doesn't do anything to facilitate interop testing either.  Ah,
memories of Sun's old Connectathon events.  Maybe IETF could sponsor a
similar event.  It's essentially a lot like a weeklong hackathon, but
where the goal is to have implementors interop test rather implement.

Nico
--

---

From: Christian Huitema <[email-masked]>
Date: Thu, 7 Nov 2019 22:21:30 -0800

> other bottlenecs, then RFC-Editor queue time will become the next

That's not what the numbers say. I took a hard look at the delays in
https://tools.ietf.org/html/draft-huitema-rfc-eval-project-01, and the
RFC Editor queue time is by no means the largest part of the delay from
1st draft to RFC publication. The RFC Edition delay proper is about 2
months on average, plus about 1 month on average for the Auth48 period.
The edition delay does not vary much, but the Auth48 delay varies a lot,
and the main reason for this variability is the responsiveness of the
authors.

In the path from 1st draft to RFC, the bulk of the delays happen in the
working group. In my sample set, the average delay from start to finish
is 3 years and 3 months, of which on average 2 years and 8 months are
spent getting consensus in the working group.

-- Christian Huitema

---

From: Jari Arkko <[email-masked]>
Date: Fri, 8 Nov 2019 09:11:28 +0200

> of RFCs being processed is going up.

I think the reports tell you that persons in meetings is going down. I’m not sure we have data to say number of people involved is declining. Could be offset by people who are remote in the meetings or people who only use mailing lists but still participate effectively.

The number of RFCs is not going up, it went up (peaking at 2006 and 2011) and has been in somewhat of a decline since then. See https://arkko.com/tools/rfcstats/pubdistr.html <https://arkko.com/tools/rfcstats/pubdistr.html> and https://www.rfc-editor.org/report-summary/ <https://www.rfc-editor.org/report-summary/> (Note that on my stats the last year numbers are always a bit off, because the years aren’t complete. So don’t look at 2019 too much.)

> documents.  Did we have more people who were exclusively "tourists" before?

I don’t have an easy way to answer all your questions. I did pull the following stats from my server though. These are the unique authors (or explicitly named contributors) in RFCs published per year. So, the number of distinct people who published at least one RFC. (Note: again, don’t look at 2019, and I don’t have a good way of assessing how accurate the author identification is over all RFCs. Reader beware.)

2001	396
2002	408
2003	454
2004	448
2005	561
2006	713
2007	582
2008	509
2009	490
2010	630
2011	703
2012	599
2013	537
2014	562
2015	698
2016	711
2017	630
2018	541
2019	370

The highest levels of unique authors occurred in 2006, 2011, and 2016. The two of the first years co-incide with high RFC production levels, but 2016 did not. Eyeballing this, I’d say the number of unique authors seems to vary but I don’t see a recent drop-off. Pretty healthy numbers I’d say, even, if 600-700 people manage to publish RFCs and the meetings draw 900-1100 people.

I have anecdotal impression that conflicts are up because there’s a set of people who are key contributors across multiple areas and WGs. I could dig up some further stats on that, but not right now.

Jari

---

From: Keith Moore <[email-masked]>
Date: Fri, 8 Nov 2019 02:31:38 -0500

On 11/8/19 2:11 AM, Jari Arkko wrote:

> Pretty healthy numbers I’d say, even, if 600-700 people manage to 
> publish RFCs and the meetings draw 900-1100 people.

To me it sounds unhealthy.   If meeting attendance is an indicator of 
the number of active participants, it sounds like about half of the 
participants are publishing an RFC every year. With that many authors I 
have to wonder how many people have time to participate in discussion of 
others' documents.   Of course a person can author one document while 
discussing others, but it's easy for an author's attention to others' 
drafts to diminish while revising his own draft.

Keith

---

From: Eliot Lear <[email-masked]>
Date: Fri, 8 Nov 2019 10:28:43 +0100

> In that case, perhaps the response should be to defer the ballot automatically, and make it public that the reason is a late review.

Please not automatic.  It takes forever to get documents through the system as is.  It’s understandable that sometimes a deferral is appropriate.  But the AD needs to make that call, not a computer.

Eliot

---

From: Ole Troan <[email-masked]>
Date: Fri, 8 Nov 2019 10:49:54 +0100

I have for a long time been uncomfortable with the level of document changes that happens after the document leaves the working group.

From my perspective I would have preferred a document process, where the working group "owned" the document until publication.

For cross-area issues:
The responsible AD together with the chairs should identify cross-area conflicts and need for collboration earlier in the process. And as chairs/responsible ADs escalate cross area issues, those should be discussed in the IESG while document is in the working group.

If a document isn't good enough, the IESG should send it back to the working group. Not try to "fix" it themselves (often in a small group between chairs, authors and ADs).

Basically I want the IESG for leadership. I want the IESG for quality control. I do not want the IESG to have to read every document, nitpick or even (gasp) override working group consensus.

Executive summary: More leadership less document editing.

Cheers,
Ole

---

From: John C Klensin <[email-masked]>
Date: Fri, 08 Nov 2019 08:56:15 -0500

--On Friday, November 8, 2019 02:31 -0500 Keith Moore
<[email-masked]> wrote:

>> Pretty healthy numbers I'd say, even, if 600-700 people
>> manage to  publish RFCs and the meetings draw 900-1100 people.
> 
> To me it sounds unhealthy.   If meeting attendance is an
> indicator of the number of active participants, it sounds like
> about half of the participants are publishing an RFC every
> year. With that many authors I have to wonder how many people
> have time to participate in discussion of others'
> documents.   Of course a person can author one document
> while discussing others, but it's easy for an author's
> attention to others' drafts to diminish while revising his own
> draft.

Keith,

As Jari cautioned, those numbers have to be interpreted with
care.  For example, the number of documents with many authors
has has risen over the years since Jon Postel aggressively
pushed back against the practice.  We also have issued documents
that are revisions of older ones.  In general, original author
names have been retained even when newer authors have been added
on and the actual involvement of the original ones has been
minimal.   There may also be issues with listed contributors
because, at the discretion of a WG or whomever is holding the
pen, that category may include someone who contributed a concept
or a few critical paragraph, but who was not otherwise
significantly involved in the production of the document.

Even Jari's comment that you quote is problematic because, while
600-700 people may have their names on RFCs that are published,
sometimes it is accurate to say that they published them and
sometimes some of them are just names.  Even the definition of
"author" may differ from one document to the next.  We also
aren't careful about distinguishing between authors and editors.
For example, with a document that is really developed by a WG,
the "author" may be mostly a compiler of WG comments and text
rather than a contributor of and advocate for original ideas.
Either may involve a great deal of time and effort, but the kind
and level of involvement is different.  I even know of at least
a couple of often-cited RFCs in which there are several listed
authors but none of them, independent of their intellectual or
other contributions, actually provided any significant fraction
of the published text.

Similarly, comparing author counts to f2f meeting registrations
does not quite work and is probably getting worse over time as
meeting costs rise and active remote participation increases.
We should have at least approximate numbers by now for total
registered participants, but probably not reaching back to the
point that Jari is comfortable with the authorship numbers. 

I can imagine ways to estimate numbers and control, in the
statistical sense, for some (but not all) of those issues
without, e.g., interviewing authors or doing a careful
document-by-document evaluation.   They all would require a lot
of work.  Maybe others disagree, but I'd rather see the people
who would probably need to do that work spending their time on
substantive technical contributions instead.  But, without doing
that work (at least), inferences about how many people are
writing documents and therefore not evaluating those of others
seems fairly close to pure speculation.

best,
   john

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 8 Nov 2019 14:20:27 +0000

>    I have for a long time been uncomfortable with the level of document changes that happens after the document leaves the working group.

Me too.  In ACME, I try to pass any substantive changes to the WG to see if anyone has a problem.  Editorial clarifications, I don't bother with.

I agree that WGLC should not mean "let another smaller group get a whack at it"

---

From: Michael Richardson <[email-masked]>
Date: Fri, 08 Nov 2019 10:36:04 -0500

John C Klensin <[email-masked]> wrote:
    >>> Pretty healthy numbers I'd say, even, if 600-700 people manage to
    >> participants are publishing an RFC every year. With that many authors
    >> I have to wonder how many people have time to participate in
    >> discussion of others' documents.   Of course a person can author one
    >> document while discussing others, but it's easy for an author's
    >> attention to others' drafts to diminish while revising his own draft.

    > sometimes it is accurate to say that they published them and sometimes
    > some of them are just names.  Even the definition of "author" may
    > differ from one document to the next.  We also aren't careful about
    > distinguishing between authors and editors.  For example, with a
    > document that is really developed by a WG, the "author" may be mostly a
    > compiler of WG comments and text rather than a contributor of and
    > advocate for original ideas.  Either may involve a great deal of time
    > and effort, but the kind and level of involvement is different.  I even

So an alternate stat may be the number of unique people who upload to the
Datatracker.  So that would require, for each RFC published, getting the list
of people who uploaded.  That would be a lower bound on number active
authors.
The numbers Jari has would be the upper bound.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: "Andrew G. Malis" <[email-masked]>
Date: Fri, 8 Nov 2019 11:02:09 -0500

Michael,

> The numbers Jari has would be the upper bound.

That would include Internet Draft authors as well as RFC authors, but I
think that's a better metric of how many active authors we have in a given
year.

Except now we have authoring happening on GitHub as well, which is more
difficult to track. For example, how many active authors do we have in the
QUIC WG? Tough to say without looking at their GitHub document update
history.

Cheers,
Andy

---

From: Keith Moore <[email-masked]>
Date: Fri, 8 Nov 2019 11:25:52 -0500

> The responsible AD together with the chairs should identify cross-area conflicts and need for collboration earlier in the process. And as chairs/responsible ADs escalate cross area issues, those sh…

That's too few people, IMO.   Also there's a tendency to call them 
"cross-area issues" but I think that can be misleading. What I have seen 
time and time again was that a WG or document author would make 
decisions that would, for example, harm the ability of the Internet to 
support a certain kind of application.   But even a typical Applications 
AD wouldn't necessarily realize that, because "applications" is 
tremendously broad.   (These days the ART area is even broader.)

I see no substitute for a community-wide review.   A message is sent to 
an appropriate list (maybe lastcall, maybe not) that summarize the WG's 
charter and list each of the documents being worked on and describes how 
they fit into that charter.   The message states that the chairs and 
IESG are interested in knowing whether these solutions harm other 
interests, and more generally solicit community feedback on the approach 
(keeping in mind that Z1 and Z2 are probably not finished yet), and 
there's a special email address for the feedback that must be used.     
Either make those solicitations for review explicitly part of a WG's 
schedule, or automatically trigger them at one year intervals after a 
group has started.   The WG chairs would have the responsibility of 
writing those descriptions, any responses would be recorded and made 
publicly available, the WG chair would summarize the responses for the 
IESG, and the WG is expected to respond to substantive feedback.

> If a document isn't good enough, the IESG should send it back to the working group. Not try to "fix" it themselves (often in a small group between chairs, authors and ADs).
Agree with the latter part.   IESG is in a poor position to "fix" a 
document, especially so late in the process, though the same is true of 
a WG that has labored years to produce something before even being aware 
of potential harm that they're doing.   That's why the reviews need to 
be done well before a group thinks the document is complete.
> Basically I want the IESG for leadership. I want the IESG for quality control. I do not want the IESG to have to read every document, nitpick or even (gasp) override working group consensus.
Mostly agree, though I don't see how the IESG can provide meaningful 
quality control without each AD at least skimming the documents that are 
relevant to that AD's area.     But I have always thought that the IESG 
and other reviewers should point out the problems and the WG should 
figure out how to fix them.   But they need that feedback as early as 
possible so they can take that input into account when making their 
compromises.    "Last Call" and IESG review should be stopgap measures, 
not the only means by which a WG gets feedback from other parties.  Most 
of the time final IESG review should be more of a process review than a 
technical review.    But part of that process review should consider how 
the document addressed external feedback.

Keith

---

From: Bob Hinden <[email-masked]>
Date: Fri, 8 Nov 2019 08:40:19 -0800

Christian,

> On Nov 7, 2019, at 10:21 PM, Christian Huitema <[email-masked]> wrote:
> 
> 

That’s matches my experience as well.

Also, one of the significant causes for long delays in the RFC Editor queue is mis refs.  That is, when it has normative references to drafts that are not yet approved.   This can add considerable delay and can’t be speeded up by the RFC Editor.

Bob

---

From: Keith Moore <[email-masked]>
Date: Fri, 8 Nov 2019 11:49:23 -0500

On 11/8/19 8:56 AM, John C Klensin wrote:

> As Jari cautioned, those numbers have to be interpreted with care.

For what it's worth, I interpreted those numbers as the number of 
documents rather than the number of authors, because they seemed to be 
roughly consistent with the yearly numbers of RFCs produced elsewhere.

Keith

---

From: Michael Richardson <[email-masked]>
Date: Fri, 08 Nov 2019 12:06:04 -0500

Andrew G. Malis <[email-masked]> wrote:
    >> So an alternate stat may be the number of unique people who upload to
    >> the Datatracker.  So that would require, for each RFC published,
    >> getting the list of people who uploaded.  That would be a lower bound
    >> on number active authors.  The numbers Jari has would be the upper
    >> bound.

    > think that's a better metric of how many active authors we have in a
    > given year.

Well, the complexity is that one has to get the list of RFCs, then the list
of uploads for each RFC.

    > difficult to track. For example, how many active authors do we have in
    > the QUIC WG? Tough to say without looking at their GitHub document
    > update history.

Thus a reason why I'd like to have the list of contributors/acknowledgements
in a structured format in the document :-)

---

From: John Leslie <[email-masked]>
Date: Tue, 5 Nov 2019 10:57:59 -0500

> and they could divide the WG's and IESG concalls amongst themselves?

   The whole process would break. :^(

   (I don't know whether that's good or bad...)

   Beyond question, the workload has become oppressive.

   Different IETF-Chairs have different approaches. Adapting to these
changes, IMHO, has been challenging for IESG members.

   But the long-term trend has been to make it entirely too difficult
to say no to any new-group proposal. A pair of WG-chairs is appointed,
and the AD's don't have time to follow the actual process.

   Some WGCs listen very carefully to AD advice; others don't. Some ADs
give very good advice early; others don't.

   But there's an endemic problem: enough of the hoi-polloi see each WG
as the only possible way to "solve" their problem; and they develop
tunnel vision. Thus anyone other than the AD who points out a problem
is facing a cliff-like wall of resistance.

   This leads to problems entombed in published RFCs.

   It is rare for these problems to be solved -- ever.

   Beating your head against these entombed problems _seriously_ reduces
the enthusiasm of ordinary IETF-ers to devote full-time to our process.

   :^( :^( :^(

   (Having basically retired from my full-time job, I have perhaps enough
time available to work on this, but nowhere near enough money to cover
$50,000 per year of out-of pocket expenses.) (Also, I hate air travel!)

   But perhaps, somebody else will explore alternatives to selecting only
employer-sponsored folks for the IESG...

--
John Leslie <[email-masked]>

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 9 Nov 2019 08:22:46 +1300

On 09-Nov-19 05:25, Keith Moore wrote:
...
> I see no substitute for a community-wide review. 

...
> Executive summary: More leadership less document editing.

Violent agreement with all of the above. And again - this is all procedural, no change to IETF rules is needed. I begin to realise that the IESG's workload is in its own hands, and we (all of us who've served in the IESG over the last 20 years) have to share the blame for current practice.

   Brian

---

From: Keith Moore <[email-masked]>
Date: Fri, 8 Nov 2019 14:34:09 -0500

On 11/8/19 2:22 PM, Brian E Carpenter wrote:

> And again - this is all procedural, no change to IETF rules is needed. I begin to realise that the IESG's workload is in its own hands, and we (all of us who've served in the IESG over the last 20 years) have to share the blame for current practice.

Something I realized long ago is that IETF community expectations are at 
least as influential as the rules, probably moreso.

As long as the community expects that any working group that has support 
will be approved, that working groups can stay alive as long as they 
keep producing documents, and that any document that a working group 
produces should be approved, the workload problems will continue no 
matter how IESG manages its own process.

And every time someone points out that IESG is overloaded, most of the 
proposed "solutions" seem to have the intent, or at least effect, of 
further reducing document quality.

Maybe it's just me, but I keep thinking that the best way for IETF to 
serve the Internet is to produce fewer documents of higher quality and 
greater relevance.

Keith

---

From: Michael Richardson <[email-masked]>
Date: Fri, 08 Nov 2019 16:30:23 -0500

Keith Moore <[email-masked]> wrote:
    >> And again - this is all procedural, no change to IETF rules is
    >> needed. I begin to realise that the IESG's workload is in its own
    >> hands, and we (all of us who've served in the IESG over the last 20
    >> years) have to share the blame for current practice.

    > Something I realized long ago is that IETF community expectations are
    > at least as influential as the rules, probably moreso.

    > As long as the community expects that any working group that has
    > support will be approved, that working groups can stay alive as long as
    > they keep producing documents, and that any document that a working

I'd really like them to stay alive while not producing (new) documents.
So I feel that groups sometimes are eager to recharter because that is the
only way to stay alive.  I'd like to change that.

    > further reducing document quality.

But, the bar is already very very high.

    > serve the Internet is to produce fewer documents of higher quality and

Fewer RFCs, I agree.

---

From: Keith Moore <[email-masked]>
Date: Fri, 8 Nov 2019 16:40:02 -0500

On 11/8/19 4:30 PM, Michael Richardson wrote:

> only way to stay alive.  I'd like to change that.I

As far as I can tell, what makes people want to continue to participate 
in a working group is (usually) that they're working on new documents.

I don't think it should be a hard-and-fast rule, but I think that any 
group that tries to stay around long after its initial goals have been 
completed, should be regarded with extra scrutiny.

But I wouldn't object to having WG charters include interoperability 
testing as explicit milestones.

The goal of IETF should not be to produce as many documents as possible, 
it should be to produce the documents that the Internet needs most and 
that the IETF can best provide.

Perpetual WGs usually do not serve that goal, IMO.   Again, there are 
sometimes exceptions.

> But, the bar is already very very high.
Maybe the bar is not high in the best ways.    IMO the "bar" should be 
about document quality and relevance, more than about the willingness to 
work past the point of exhaustion.

Keith

---

From: John C Klensin <[email-masked]>
Date: Sat, 09 Nov 2019 08:54:52 -0500

--On Friday, November 8, 2019 12:06 -0500 Michael Richardson
<[email-masked]> wrote:

>     >> So an alternate stat may be the number of unique people
> who upload to     >> the Datatracker.  So that would require,
> for each RFC published,     >> getting the list of people who
> uploaded.  That would be a lower bound     >> on number active
> authors.  The numbers Jari has would be the upper     >> bound.
> 
>     > That would include Internet Draft authors as well as RFC
> authors, but I     > think that's a better metric of how many
> active authors we have in a     > given year.
> 
> Well, the complexity is that one has to get the list of RFCs,
> then the list of uploads for each RFC.
> 
>     > Except now we have authoring happening on GitHub as
> well, which is more     > difficult to track. For example, how
> many active authors do we have in     > the QUIC WG? Tough to
> say without looking at their GitHub document     > update
> history.
> 
> Thus a reason why I'd like to have the list of
> contributors/acknowledgements in a structured format in the
> document :-)

Structured or not, our acknowledgments have tended to include
people who made major contributions, people who proposed an
alternate approach that stimulated thinking but whose ideas
and/or text did not make it into the document, and people who
read the document as reviewers and picked a few editorial nits
without making any substantive contribution or giving evidence
that the understood its technical content.  I've occasionally
included people in acknowledgments (with appropriately
circuitous language) whose sole contribution was irritating me
enough that I didn't abandon the document or the underlying
work.  If you are going to structure such acknowledgement, you
will need a lot of categories,  willingness by authors/editors
to go to the extra trouble to properly identify them.   As an
extreme example, consider:

 <leadIrritant> Ima Fool </leadIrritant>  or
 <pwbdutgtow> Hes An Obstacle </pwbdutgtow>
(PersonWhoBlockedDocumentUntilTheyGotTheirOwnWay)

FWIW, this thread is leading down the path and into the rathole
that was a  significant cause of most other voluntary SDOs
concluding that finished documents should show only the
standards body as author, often accompanied by a list of WG
members and, in some bodies, identification of WG chairs and an
editor.  Perhaps not coincidently, it is also where Jon Postel
started, i.e., with the notion that the "author" or "editor"
should be the contact person and what we've sometimes called the
pen-holder, with everyone else acknowledged, perhaps in general
terms.

best,
   john

---

From: John C Klensin <[email-masked]>
Date: Sat, 09 Nov 2019 09:12:37 -0500

(top post)

Let me add one additional comment to John's description of the
problem, which which I largely agree. He suggested that it had
become "entirely too difficult to say no to any new-group
proposal".   I suggest that it has gotten even more difficult to
say no to anything emerging from a working group, no matter how
ill-conceived.  Statistics about how long documents were sitting
in IESG evaluation, waiting for AD, or waiting for new draft
states might be very illuminating about this.

best,
  john

--On Tuesday, November 5, 2019 10:57 -0500 John Leslie
<[email-masked]> wrote:

>> I wonder what people think would break if we moved to 5 AD's
>> per area, and they could divide the WG's and IESG concalls
>> amongst themselves?
> 
>    Different IETF-Chairs have different approaches. Adapting
> to these changes, IMHO, has been challenging for IESG members.
> 
>    But the long-term trend has been to make it entirely too
> difficult to say no to any new-group proposal. A pair of
> WG-chairs is appointed, and the AD's don't have time to follow
> the actual process.
> 
>    Some WGCs listen very carefully to AD advice; others don't.
> Some ADs give very good advice early; others don't.
> 
>    But there's an endemic problem: enough of the hoi-polloi
> see each WG as the only possible way to "solve" their problem;
> and they develop tunnel vision. Thus anyone other than the AD
> who points out a problem is facing a cliff-like wall of
> resistance.
> 
>    Beating your head against these entombed problems
> _seriously_ reduces the enthusiasm of ordinary IETF-ers to
> devote full-time to our process.
> 
>    :^( :^( :^(
> 
>    (Having basically retired from my full-time job, I have
> perhaps enough time available to work on this, but nowhere
> near enough money to cover $50,000 per year of out-of pocket
> expenses.) (Also, I hate air travel!)
> 
>    But perhaps, somebody else will explore alternatives to
> selecting only employer-sponsored folks for the IESG...
> 
> --

---

From: Mary B <[email-masked]>
Date: Sat, 9 Nov 2019 08:27:52 -0600

In DISPATCH WG, the whole point is to do a better job considering new work
before chartering or even progressing as AD sponsored.  We have told
several folks that there was no interest in their work at IETF.   Of
course, they then go the ISE route, which honestly I don't necessarily have
an issue with, but that work does come back to someone in the IETF with
expertise in that area to do a review and as I'm recalling ADs have to
consider whether there's any conflict.   So, yes, it's really hard, if not
impossible to tell people we're not going to spend any cycles on their pet
project.  And, unfortunately, it also depends upon who is bringing in the
work.  We are not the meritocracy that many think we are for several
reasons.

Regards,
Mary.

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 9 Nov 2019 15:36:21 +0100

>> 370*(365.0/Time.now.yday)
=> 431.4696485623003

More importantly, the RPC is constipated by the v3 switch, so those numbers aren’t very indicative.  Maybe looking at

>> 370*(365.0/Time.parse("2019-09-16").yday)
=> 521.4285714285714

gives a clearer indication that we are about where we were in 2018.  But really, there should be some metric run over the RFC editor queue as well.

OK, so much for playing scientist here.  Now for the process wonk in me (which I’m definitely only playing).

It seems to me all the proposals to add work for everyone to get better metadata for making these measurements are non-starters.  A little more information can be extracted out of the existing information in the datatracker, which would make a nice project for a student maybe, but the main contributors all have way more important things to do.

So what do we learn?  

> of RFCs being processed is going up.

There are lots of impressions here, which as we see often aren’t supported (or even are contradicted) by the indicators we have.
E.g., my “impression” also is that conflicts are going up, but maybe that is just a facet of the IoT work I’m engaged in going mainstream and impacting/being impacted by more WGs.

The original idea in some of the contributions to this thread, which in my words I’d summarize as lobotomizing the quality engineering we do so we can get more candidates for ADs, is a non-starter as well.  Most of the arguments in this thread for this are bolstered by statements about what specific participants of the process *should* be doing (also known as process confabulation in software engineering), not by observations about reality.

That we don’t have a good alternative to the current process doesn’t mean we can’t tune some knobs, and I’d prefer if we’d focus on what these knobs are.  Strengthening the directorate review system (with the objective to reduce variance in review quality and timing) is very well worth some effort.  Feeding more early cross-area review into the WG process that precedes the IESG review also would help; we have been trying to do this but probably need to be more consistent in these efforts.

All in all I think that making someone responsible for proposing tuning actions and observing and evaluating the results, preferably over a longer time than a single AD tenure, might help.  Not in the sense of “wenn ich nicht mehr weiter weiß, bilde ich ‘nen Arbeitskreis” (not sure that this aphorism about the tendency of German bureaucracy to shift responsibility to useless committees is translatable), but in the sense of creating actual responsibility.  Sweeping actions would still need to go through our consensus process, which makes them very expensive, but continued attention on smaller tweaks might already help a lot.

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Sun, 10 Nov 2019 10:05:08 +0800

On 2019-11-09 9:36 a.m., Carsten Bormann wrote:
> Strengthening the directorate review system (with the objective to
> reduce variance in review quality and timing) is very well worth some
effort.

I think that we are in agreement here.

My recent experience is that I could not get (timely) feedback from the
reviewers as to whether or not they were happy with the changes. I think
that this is a chronic situation, so the ADs have learnt that a negative
review doesn't mean much if there has been a revision to the document
since.

So they go ahead anyway, and then the AD winds up redoing the review,
and being uncertain if the WG actually thought about the review comments
and proceeded anyway.
This is a serious commitment by the reviewers; that's why I want better
acknowledgment for them.
(And I'm not worried about the way that John and John about having too
many categories)

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 13 Nov 2019 10:07:47 -0500

Maybe we have the wrong model for security reviews and this is in part the
result of the RFC series being kitchen sink.

Some reviews are so trivial as to be irrelevant. Other cases you get a
document where the security considerations is a link to another document
that is nothing but security considerations.

Maybe what we need is a structure that assigns multiple reviewers for some
projects and rubber stamps others.

Also note that I am a designer and my skill set is quite different to those
of a hacker. I can show people a way to do a job so that there is a very
small chance of getting it wrong. But thats not the same as spotting their
mistake if they decide to do it their way.

Another concern I have is formal methods and before folk start squawking
about how they are necessary, my doctoral thesis is on formal methods. My
college Tutor was Tony Hoare, I know what they are capable of. What worries
me is that systems we can prove to be secure seem to be turning out to be
fragile in the real world.

If folk want an extreme example, look at BTC, the cryptography of the
Blockchain is unbreakable. Yet I spent this morning talking to a friend
dealing with the aftermath of a relative who has literally lost everything
to a corrupt exchange scheme.

---

From: Keith Moore <[email-masked]>
Date: Wed, 13 Nov 2019 10:25:16 -0500

On 11/13/19 10:07 AM, Phillip Hallam-Baker wrote:

> Maybe what we need is a structure that assigns multiple reviewers for 
> some projects and rubber stamps others.

Seems like ADs already have a fair amount of discretion to ask for 
multiple in-depth reviewers vs. getting minimal review.   If having a 
human make such decisions isn't your idea of an appropriate "structure", 
I'd be curious to know what is.

Keith

---

From: Michael Richardson <[email-masked]>
Date: Fri, 15 Nov 2019 15:56:12 +0800

On 2019-11-13 11:25 p.m., Keith Moore wrote:
> human make such decisions isn't your idea of an appropriate
> "structure", I'd be curious to know what is.
>

The issue is that is only so much senior security clue to go around.
There is a non-trivial amount of effort for an-out-area reviewer to spin
up enough understanding about what a WG is doing.  There are a lot of
documents that simply allocate a new attribute from an existing registry
and then use it for something.  Determining if this has a trivial or
non-trivial security impact can be difficult.  If it turns out to be
trivial, then we've wasted the reviewers time (opportunity cost).  If it
turns out not to be trivial (and the reviewer missed that), then if we
are lucky, we catch it at IESG time, and then it might be a year later.

WGs are given security advisors, and most don't use them, and many of
them are AWOL.

---

From: Keith Moore <[email-masked]>
Date: Fri, 15 Nov 2019 09:08:50 -0500

> are lucky, we catch it at IESG time, and then it might be a year later.

I don't disagree with any of the above.  And yet, I don't see how it's 
responding to either of the above replies.

Keith

---

From: Julian Reschke <[email-masked]>
Date: Fri, 15 Nov 2019 19:54:37 +0100

> by a set of authors who are exclusively remote.

Why exclusively?

Best regards, Julian

---

From: Michael Richardson <[email-masked]>
Date: Sat, 16 Nov 2019 14:40:17 +0800

Julian Reschke <[email-masked]> wrote:
    >> Salz, Rich <[email-masked]> wrote: >> My impression from the regular
    >> plenary reports on attendance my impression is > that the number of
    >> people involved in the IETF is declining, yet the number > of RFCs
    >> being processed is going up.
    >>
    >> > Attendance is perhaps declining, but I don't recall any discussion
    >> of > WG email traffic, for example.
    >>
    >> So the number we need to know is what is the trend for RFCs being
    >> processed by a set of authors who are exclusively remote.  ...

    > Why exclusively?

There is a belief that documents which have authors that can travel to
meetings progress more easily than those that don't.
A document that has at least one author "shopping" the document around the
physical meeting could have five remote authors contributing.

---

From: Michael Richardson <[email-masked]>
Date: Sat, 16 Nov 2019 14:42:20 +0800

Keith Moore <[email-masked]> wrote:
    >>>> Maybe what we need is a structure that assigns multiple reviewers
    >>>> for some projects and rubber stamps others.
    >> There is a non-trivial amount of effort for an-out-area reviewer to
    >> spin up enough understanding about what a WG is doing.  There are a
    >> lot of documents that simply allocate a new attribute from an existing
    >> registry and then use it for something.  Determining if this has a
    >> trivial or non-trivial security impact can be difficult.  If it turns
    >> out to be trivial, then we've wasted the reviewers time (opportunity
    >> cost).  If it turns out not to be trivial (and the reviewer missed
    >> that), then if we are lucky, we catch it at IESG time, and then it
    >> might be a year later.

    > responding to either of the above replies.

The current system assigns the review prior to the AD determining if they
need an in-depth review or not.  So if we assign a senior (security) reviewer
to a document that didn't need in-depth senior experience, then that person
is unavailable (within the quantum of review assignment period) for the AD to
assign them to do something more in-depth.

---

From: Benjamin Kaduk <[email-masked]>
Date: Fri, 15 Nov 2019 23:08:02 -0800

> assign them to do something more in-depth.

My understanding is that most directorates have a secretary that does the
assignments (secdir does, at least).  By the time an AD is looking at the
review next to the document it might only be a few days before the telechat
where the document is up for approval, which is not really enough time to
get another review in without deferring the document.  Maybe we should go
get that extra review and try to remove the stigma against deferring
documents; I don't have a sense for how the community would feel about
that.

And yes, the AD should look at the directorate review when it arrives, but
looking only at the review and not the document being reviewed is not
always enough to tell whether additional review would be valuable.

-Ben

---

From: Keith Moore <[email-masked]>
Date: Sat, 16 Nov 2019 09:25:55 -0500

> documents; I don't have a sense for how the community would feel about

IMO there's always going to be some conflict between an AD doing his/her 
job and "what the community wants", because nobody thinks his/her own 
document is lowering the quality of IETF standards. So the pressure to 
approve documents despite serious flaws, and/or to approve documents 
without delay (even though some documents really do require more time to 
review and more eyeballs to look at them), will always be there.   There 
should not be a "stigma" associated with deferring documents if there's 
any reason at all to subject them to more scrutiny.

Neither IESG nor the RSE is the source of most delays, after all; most 
delays are in the working groups.   And yeah, people compile stats on 
the IESG and the RSE to see how well they "perform" but those stats 
don't measure the most relevant things, like the quality of IETF's 
output, or the extent to which IETF standards are utilized, or the 
extent to which they're followed (this is a different thing), or 
(especially) the degree to which IETF standards meet the Internet's 
needs.   There's an unfortunate tendency these days to pay more 
attention to quantities that are easily measured, than to those that 
actually matter.

> always enough to tell whether additional review would be valuable.

Entirely agree.   I don't think IESG should presume that directorate 
reviews will always be sufficient, and to do so IMO is tantamount to 
IESG abandoning its duty.

Keith

---

From: Michael Richardson <[email-masked]>
Date: Sun, 17 Nov 2019 10:58:43 +0800

Benjamin Kaduk <[email-masked]> wrote:
    >> Keith Moore <[email-masked]> wrote: >> On 2019-11-13 11:25
    >> p.m., Keith Moore wrote: >>> On 11/13/19 10:07 AM, Phillip
    >> Hallam-Baker wrote:

    >> >>>> for some projects and rubber stamps others.  >>> Seems like ADs
    >> already have a fair amount of discretion to ask for >>> multiple
    >> in-depth reviewers vs. getting minimal review.   If having a >>> human
    >> make such decisions isn't your idea of an appropriate >>> "structure",

    >> >> The issue is that is only so much senior security clue to go
    >> around.  >> There is a non-trivial amount of effort for an-out-area
    >> reviewer to >> spin up enough understanding about what a WG is doing. 
    >> There are a >> lot of documents that simply allocate a new attribute
    >> from an existing >> registry and then use it for something. 
    >> Determining if this has a >> trivial or non-trivial security impact
    >> can be difficult.  If it turns >> out to be trivial, then we've wasted
    >> the reviewers time (opportunity >> cost).  If it turns out not to be
    >> trivial (and the reviewer missed >> that), then if we are lucky, we
    >> catch it at IESG time, and then it >> might be a year later.
    >>
    >> > I don't disagree with any of the above.  And yet, I don't see how
    >> it's > responding to either of the above replies.
    >>
    >> The current system assigns the review prior to the AD determining if
    >> they need an in-depth review or not.  So if we assign a senior
    >> (security) reviewer to a document that didn't need in-depth senior
    >> experience, then that person is unavailable (within the quantum of
    >> review assignment period) for the AD to assign them to do something
    >> more in-depth.

    > My understanding is that most directorates have a secretary that does
    > the assignments (secdir does, at least).

yes, that's my understanding.

I'd like to see more coordination between ADs (particularly Sec-ADs) and
directorates so that the security review process can occur earlier, and so
that any loop with the SecADs can happen earlier.

In the case of draft-ietf-anima-bootstrapping-keyinfra, I'd have liked to get
more attention from Christian,Jari and Russ (reviewers) and the various ADs
earlier.  The significant reviews were done a year ago, and we are just
finishing now.
That's a big investment of time among the 6 or 7 people involved.

    > By the time an AD is looking
    > at the review next to the document it might only be a few days before
    > the telechat where the document is up for approval, which is not really
    > enough time to get another review in without deferring the document.

It seems that we doing these early secdir reviews, but someone this is not
feeding up to the ADs well enough, who then do their own review.  That's just
not leveraging the secdir well.

    > Maybe we should go get that extra review and try to remove the stigma
    > against deferring documents; I don't have a sense for how the community
    > would feel about that.

I'm okay with this, but maybe the sponsoring AD and WG chairs need to be more
active in chasing down reviewers.

Again, I'd like more offocial acknowledgement of the work reviewers do.

    > And yes, the AD should look at the directorate review when it arrives,
    > but looking only at the review and not the document being reviewed is
    > not always enough to tell whether additional review would be valuable.

Agreed.
What if the Shepherd write up was had more ways to flag things?

---

From: Benjamin Kaduk <[email-masked]>
Date: Mon, 18 Nov 2019 02:53:03 -0800

Hi Michael,

> That's a big investment of time among the 6 or 7 people involved.

I'm not sure that I understand what you're looking for here.
In the case of
https://datatracker.ietf.org/doc/review-ietf-anima-bootstrapping-keyinfra-16-secdir-lc-huitema-2018-09-29/,
my conclusion from looking at the review is that "this document is not
ready for publication as-is and should go back to the WG before it comes
before the IESG".  That is, I would essentially ignore the document until
the secdir reviewer is satisfied [or the authors' responses give some
indication that the reviewer is incorrect], to make more efficient use of
my time.  But it sounds like you're suggesting that I should see that
review and take it as a signal to get *more* involved with the document.
Am I missing something?

> not leveraging the secdir well.

In a similar vein, I'm not sure what you would see as "leveraging the
secdir well".  I don't see myself as beholden to accept the secdir review
as-is -- to me the secdir reviews are a tool that I can take advantage of
as I perform my AD duties, but it's not the only tool at my disposal, and
if I am concerned that a given document may have broad impact or contain
subtle security issues, I am generally going to do an in-depth review
myself, in addition to any other reviews that have been done.

> active in chasing down reviewers.

Just to check: this is "seeking enough reviews from the relevant area(s)"
as opposed to "ensuring that people who did reviews respond to the updates
made because of their reviews"?

>From a technical point of view, I think this may only be relevant when the
shepherd is not a WG chair for the WG in question -- my understanding is
that the WG chair can click a button in the datatracker to request a
directorate review at any time, which might be more helpful than just a
note in the writeup.

-Ben