Subject: AD time

From: Eliot Lear <[email-masked]>
Date: Tue, 31 Jul 2018 08:00:38 +0200

I'm just trying to summarize what I think I've read in this rather
MASSIVE thread:

 1. The AD job is Big.
 2. It can't be subdivided because there is a loss of context when one
    doesn't read all the drafts.
 3. We can't peel off meeting organization because it is directly tied
    to the drafts.
 4. We don't want to limit work the IETF takes on based on AD time
 5. We don't want only people in corporations holding leadership positions
 6. We don't want people without skin in the game holding leadership
    positions
 7. We don't want leaders who take the wrong incentive signals, such as
    # of documents processed

Do I have that about right?  If so, do people believe we are meeting
them today?  Also, is [5] markedly in conflict with [6]?

IMHO this isn't fatal to Ted's idea, with a tweek or two.  It seems to
me that having a small number of people (1-3?) appointed by the NOMCOM
specifically with a healthy stipend in mind would be an interesting
experiment.  The challenge in the experiment is identifying which areas
be involved.  Seems to me that people would apply as they do today, and
the best people would be chosen, and up to the limit, if there is
funding available, they'd get it.  If there are more highly qualified
people available than there is funding, we let the NOMCOM make a
decision to balance all those constraints.  It *is* more work for the
NOMCOM, and maybe we should be funding THEM (a joke! really!).

Eliot

---

From: Padma Pillay-Esnault <[email-masked]>
Date: Mon, 30 Jul 2018 23:32:41 -0700

Eliot

I would like to add the following

8. We want a healthy balance of people from the different type of
companies/orgs to avoid solution bias.

Padma

On Mon, Jul 30, 2018 at 11:00 PM, Eliot Lear <
lear=[email-masked]> wrote:

> I'm just trying to summarize what I think I've read in this rather MASSIVE
> thread:
>
>    1. The AD job is Big.
> Do I have that about right?  If so, do people believe we are meeting them
> today?  Also, is [5] markedly in conflict with [6]?
>
> IMHO this isn't fatal to Ted's idea, with a tweek or two.  It seems to me
> that having a small number of people (1-3?) appointed by the NOMCOM
> experiment.  The challenge in the experiment is identifying which areas be
> involved.  Seems to me that people would apply as they do today, and the
> best people would be chosen, and up to the limit, if there is funding
> available, they'd get it.  If there are more highly qualified people
> available than there is funding, we let the NOMCOM make a decision to
> balance all those constraints.  It *is* more work for the NOMCOM, and maybe
> we should be funding THEM (a joke! really!).
>
> Eliot
>

---

From: Melinda Shore <[email-masked]>
Date: Mon, 30 Jul 2018 22:42:34 -0800

>  4. We don't want to limit work the IETF takes on based on AD time

I think I'm actually okay with limiting the work the IETF takes
on based on {some set of resource constraints}.  I need to think
about that some more but that doesn't strike me as obviously a bad
idea (no worse, certainly, than professionalizing the IESG!).

What I'm not okay with is charging down this path, including
investigations, etc., when there's a universe of other ways to
address the problem that aren't being discussed.  I'm not sure
I've even seen a clear statement of "the problem."

I am astonished that this is receiving serious discussion.

Melinda

---

From: Eliot Lear <[email-masked]>
Date: Tue, 31 Jul 2018 08:59:45 +0200

Hi Melinda,

> idea (no worse, certainly, than professionalizing the IESG!).
We already seem to have some sort of scaling limit in effect on the
total number of working groups, given that we never quite seem to exceed
200.  It could be that the bound is "interesting stuff", but I have
heard prioritization arguments elsewhere.  Like the rfc++ discussion. 
But like that discussion, I don't see a need to rush into anything either.
> I'm not sure I've even seen a clear statement of "the problem."
>

Well, perhaps we are all assuming different problems.  The one I was
presuming was that we had an AD who might otherwise had stayed on had
she been able to be funded.  I didn't answer the problem statement in my
own message, but sometimes these discussions evolve to such a thing. 
I'm no more astonished about this conversation than I am about rfc++ or
the myriad of other topics that show up on this list.  One nice thing
about a crisp problem statement is that one can ask the question, “Is
that really a problem?”

Without repeating in their entirety the constraints I mentioned, here,
then, is a candidate problem statement:

> There are a group of highly qualified candidates for AD that are not
> volunteering because of funding concerns.

There are, of course, OTHER problems as well, and I may be constraining
the statement above to fit a particular class of solutions, but I think
it's fair to say we've all seen this one in action before.  It is at
least A problem statement.  Many of us know several IAB members who have
suffered this problem in the past to the point, which also leads to the
question of whether the scope of the problem should be limited to just
the IESG.

And as to this point:
> there's a universe of other ways to address the problem that aren't being discussed

Sure.  Discuss?

Eliot

---

From: Kathleen Moriarty <[email-masked]>
Date: Tue, 31 Jul 2018 08:55:31 -0400

Thanks for pulling this together.  Responding here as it’s to the list.  This list misses my initial point.

The split of work and time percentages between ADs needs to be considered by the Nomcom.  

Best regards,
Kathleen 

Sent from my mobile device

> On Jul 31, 2018, at 2:32 AM, Padma Pillay-Esnault <[email-masked]> wrote:
> 
> Eliot
> 
> 8. We want a healthy balance of people from the different type of companies/orgs to avoid solution bias. 
> 
> Padma
> 
>> On Mon, Jul 30, 2018 at 11:00 PM, Eliot Lear <lear=[email-masked]> wrote:
>> I'm just trying to summarize what I think I've read in this rather MASSIVE thread:
>> 
>> The AD job is Big.
>> It can't be subdivided because there is a loss of context when one doesn't read all the drafts.
>> We can't peel off meeting organization because it is directly tied to the drafts.
>> We don't want to limit work the IETF takes on based on AD time
>> We don't want only people in corporations holding leadership positions
>> We don't want people without skin in the game holding leadership positions
>> We don't want leaders who take the wrong incentive signals, such as # of documents processed
>> Do I have that about right?  If so, do people believe we are meeting them today?  Also, is [5] markedly in conflict with [6]?
>> 
>> IMHO this isn't fatal to Ted's idea, with a tweek or two.  It seems to me that having a small number of people (1-3?) appointed by the NOMCOM specifically with a healthy stipend in mind would be an interesting experiment.  The challenge in the experiment is identifying which areas be involved.  Seems to me that people would apply as they do today, and the best people would be chosen, and up to the limit, if there is funding available, they'd get       it.  If there are more highly qualified people available than there is funding, we let the NOMCOM make a decision to balance all those constraints.  It *is* more work for the NOMCOM, and maybe we should be funding THEM (a joke! really!).
>> 
>> Eliot
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 31 Jul 2018 09:27:03 -0500

Hi, Melinda,

On Tue, Jul 31, 2018 at 1:43 AM Melinda Shore <[email-masked]>
wrote:

> I've even seen a clear statement of "the problem."

I've actually been pointing people to
https://tools.ietf.org/html/rfc3774#section-2 in discussions recently, and
https://tools.ietf.org/html/rfc3774#section-2.6.2 seems especially relevant
to this thread.

If one needed a description of "the problem", that would be a useful place
to start. I'd assert that we have made changes in response to some of the
problems included in that document ("Three Stage Standards Hierarchy not
properly Utilized" is now a two-stage standards hierarchy, for better or
worse), but a lot of the problems identified fifteen years ago sound really
familiar to me now.

(Full disclosure - Melinda was one of the co-chairs of the working group
that produced this document, and I was ACKed in
https://tools.ietf.org/html/rfc3774#section-4)

Spencer

---

From: Ted Lemon <[email-masked]>
Date: Tue, 31 Jul 2018 10:31:20 -0400

> idea (no worse, certainly, than professionalizing the IESG!).

I (1) agree with this and (2) believe that it's already happening—that the
process is self-limiting.  It may be however that it self-limits at too
high a workload per AD.

> I've even seen a clear statement of "the problem."

Melinda, I would really appreciate it if you could articulate, first of
all, where the idea comes from that someone is charging down some path.
 We are having a discussion here.   Eliot has no power to do what he's
proposed.   I have no power to do what I've proposed.   There is no
charging down some path being done here.

As for other alternatives than figuring out how to fund ADs who do what ADs
do now, if there are ways to address the problem that haven't been
discussed, why not bring those up?   Padma made a good point, and we
discussed that; there wasn't so much emotion involved with that one, and so
it didn't turn into an exploding IETF thread of doom, but there was
discussion.

> I am astonished that this is receiving serious discussion.

It might help if you could really stop and reflect on why you are so
astonished.   I mean that sincerely—the discussion yesterday was really
puzzling to me.   You must have some kind of internal model of how it goes
wrong that's more detailed than just "if you pay people to be ADs, bad
things will happen."   But I didn't hear that yesterday—I heard a lot of
comparisons to other very different organizations with very different
appointment models in very different monetary ecosystems.   I saw no
reflection about the factors present there that produced the result *there,
*nor about whether those factors are also present here and would produce
the same result here.

I really don't want to burn another day on this discussion—if you don't
want to unpack that, that's fine.   But that's why the discussion went the
way it did yesterday, and TBF I'm astonished that you're astonished.   :)

---

From: Ted Lemon <[email-masked]>
Date: Tue, 31 Jul 2018 10:37:39 -0400

This is a great document, Spencer.   I don't remember it being on the
reading list for ADs when I joined the IESG.   Is it now?

On Tue, Jul 31, 2018 at 10:27 AM, Spencer Dawkins at IETF <
[email-masked]> wrote:

> I've actually been pointing people to https://tools.ietf.org/
> html/rfc3774#section-2 in discussions recently, and
> https://tools.ietf.org/html/rfc3774#section-2.6.2 seems especially
> relevant to this thread.
>
> that produced this document, and I was ACKed in https://tools.ietf.org/
> html/rfc3774#section-4)
>
> Spencer
>

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 31 Jul 2018 09:43:31 -0500

> reading list for ADs when I joined the IESG.   Is it now?

I don't remmber reading RFC 3774 during my onboarding.  This could be a
result of all the stuff I've read since pushing it out of my memory, me not
getting to that part of the onboarding checklist, it not being present on
the onboarding checklist, or something else...

-Ben

---

From: Ted Lemon <[email-masked]>
Date: Tue, 31 Jul 2018 10:52:26 -0400

Yeah, I don't remember it either, although I vaguely remember someone
suggesting that I read it during the period after I started ADing, when it
was one of many documents to read.   Sigh.   It seems like the fact that
ADs don't read this when they start is symptomatic of some of the problems
it describes—those who are unaware of history are doomed to repeat it,
etc.   :]

On Tue, Jul 31, 2018 at 10:43 AM, Benjamin Kaduk <[email-masked]> wrote:

> > > On Tue, Jul 31, 2018 at 1:43 AM Melinda Shore <[email-masked]
> >
> > > wrote:
> > >

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 31 Jul 2018 17:52:51 +0200

Spencer: Thanks for the pointer. It did answer my question to the
list if we have any form of accumulated process wisdom. The bad
news seems to be "Yes, we have (RFC3774), but we also ignore it".

Maybe the most simple action item for ADs from this thread would really
be to ask their directorate members to provide reviews that answer
to a set of questions derived from e.g.: the shepherd review list (removing
those that don't apply) but adding those points that Ted mentioned. 

As an attempt to give more breathing room to ADs instead of reviewing
everything themselves. And for those who do already heavily lean on the
directorate as a help to improve the quality of the reviews (instead of the
also mentioned common rfc-editor like spell checking review we unfortunately
see too much.).

Cheers
    Toerless

On Tue, Jul 31, 2018 at 10:52:26AM -0400, Ted Lemon wrote:
> it describes???those who are unaware of history are doomed to repeat it,
> etc.   :]
> 
> 

-- 
---
[email-masked]

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 31 Jul 2018 11:33:51 -0500

Hi, Ted,

> reading list for ADs when I joined the IESG.   Is it now?

Well, as of about 30 seconds ago, it is ... I added

 When you surface after joining the IESG long enough to think about
"steering", instead of just reviewing docs for telechats

   - https://tools.ietf.org/html/rfc3774 is a snapshot of what the
   community thought the IETF needed to fix in 2004. It is still relevant, and
   that is not a good sign ...

to the IESG wiki under "Getting Started as an AD". But third term ADs
aren't always that cynical ;-)

Spencer

---

From: Ted Lemon <[email-masked]>
Date: Tue, 31 Jul 2018 12:34:44 -0400

Excellent.   :)

---

From: Charlie Perkins <[email-masked]>
Date: Tue, 31 Jul 2018 10:03:58 -0700

Hello folks,

Please excuse if this is not relevant, or has already been suggested -- 
I didn't read the whole thread!  The heart of my suggestion could be 
summarized as reducing AD workload, but along the way making some 
improvements to the process of document review and progress.  The idea 
is motivated by comparison to IEEE task group management, which seems to 
go more smoothly.

First, I think that the IETF Last Call process is somewhat broken. It's 
not really Last Call; instead, it is an urgent plea for people to read 
the document.  It could be compared to a Letter Ballot in the IEEE.  But 
the implied urgency in the IETF context leads to a sense that the 
document is almost "done", when in fact the document may have major 
issues that need a lot of attention.  Letter Ballot isn't like that.  
All of the comments are tabulated, and subsequent task group meetings 
are dedicated to resolving the comments. Sometimes, there are 
*thousands* of comments (for documents in large task groups like 802.11ax).

Moreover, it sometimes happens in the IETF that WG Last Call ends for a 
document and no one in the working group has read it.  In contrast, at 
least in 802.15 Letter Ballots, it is very very likely that quite a few 
people will read and comment on the document. That's because, 
essentially, they have agreed to make document reviews as part of their 
willingness to participate in 802.15, which could be compared to an Area 
in the IETF (please note this is a very rough comparison).  How could 
the IETF find more assurance that a WG document would get sufficient review?

Perhaps the pool of readers should be identified before the WG adopts 
the document.  If that happened, then when the document is ready for 
wider review, the reviewers are known and have already expressed a 
willingness to review the document.  In the IEEE, the reviewers are 
partially motivated to retain their voting rights, but in the IETF we 
probably wouldn't want to institute such a system as that.  
Nevertheless, having a solid constituency for document review and 
comment resolution in the IETF would help a lot.

My hope would be that, with a better process for document review and 
comment resolution, the ADs would find their job to be a lot easier.

Lastly, I know that some working groups already try to do some of these 
things.  I suggest that we will know when the review process has been 
sufficiently improved, when a document almost never goes through Last 
Call without sufficient comment.

Regards,
Charlie P.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 31 Jul 2018 13:14:31 -0400

One of the things that's worked quite well in several working groups I participate in is (and I know this sounds radical) a culture of insisting on running code before publication.   I'm not convinced that what works in the IEEE would work in the IETF, unfortunately.   I think the process is a bit different—in the IEEE a lot of times what you are proposing is going to have implications in silicon, and that tends to concentrate the mind in a way that software doesn't.   Maybe that's a bug we should try to fix, because broken software can have some pretty severe consequences, as we've seen of late.   But how do we get there from here?

I think one of the differences between the IETF and IEEE is that at least in some parts of the IETF, what is being proposed is motivated by a grass-roots wish to improve things, rather than an industry push.   Yes, IETF members generally have employers, but our culture is more bottom-up than top-down.   A lot of the work that gets done here is work that the "corporate sponsor" might not even have a use for.

One way out of this is to say "only work on stuff that has major corporate buy-in," but I don't think we want that.   Consequently, I think the availability of reviewers is always going to be different in IETF than IEEE.   But I'm willing to be convinced otherwise.

And I agree with you that the IETF last call review process is badly broken.   The problem is that it becomes an N^2 algorithm when in principle every IETF participant is on the hook for every document.   That doesn't work, and so we don't do it—instead we rely on the directorates and on the ADs to serve the function that last call ought to serve.   I don't know how to address this, but I think it would be worthwhile to try to think about solutions.   I did some thinking about it late in my AD term, but when I didn't get renewed I dropped it.

---

From: Charlie Perkins <[email-masked]>
Date: Tue, 31 Jul 2018 10:24:03 -0700

Hello Ted,

One important clarification ...

On 7/31/2018 10:14 AM, Ted Lemon wrote:
>
> And I agree with you that the IETF last call review process is badly broken.   The problem is that it becomes an N^2 algorithm when in principle every IETF participant is on the hook for every document.

I really did not mean to suggest that.  A lot of IETF participants would 
not be on the hook for *any* documents.  But at the time a document is 
adopted by a WG, a group of reviewers should be known for that document 
(in addition to the document authors).

Regards,
Charlie P.

---

From: Stephen Farrell <[email-masked]>
Date: Tue, 31 Jul 2018 18:25:18 +0100

Hiya,

On 31/07/18 18:03, Charlie Perkins wrote:
> the document

I'd say that's worth a try, and could be a relatively
easy tracker improvement if it worked. It'd need the
WG chairs to know who is and isn't likely to do a good
and relatively objective job, but that'd be ok in many
cases I reckon.

On 31/07/18 18:14, Ted Lemon wrote:
> One of the things that's worked quite well in several working groups
> I participate in is (and I know this sounds radical) a culture of
> insisting on running code before publication.

I'm also keen on that, where it applies. I guess it
could maybe be handled similarly to the above - a WG
could adopt a document contingent on it having running
code before publication will be requested. A pointer
to the list archive where the relevant description of
what's expected in terms of running code could be added
at adoption-time just like adding the name of someone
who's promised a thorough review.

I'm not sure either thing would cut down on the time
requirement for ADs that much, but both seem like good
things to try out.

Cheers,
S.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 31 Jul 2018 13:26:52 -0400

On Jul 31, 2018, at 1:24 PM, Charlie Perkins <[email-masked]> wrote:
> I really did not mean to suggest that.  A lot of IETF participants would not be on the hook for *any* documents.  But at the time a document is adopted by a WG, a group of reviewers should be known for that document (in addition to the document authors).

I think we already do that informally.   This is what the call for adoption is supposed to do.   The problem with formalizing it is that over the lifetime of the document, who is interested in it changes.   So you can get committed reviewers, and then lose them.   And you can get people who agree to review, because they want the document adopted, and then don't.   I'm not saying "so don't do this," but that's the problem we'd have to overcome.

I suppose adding this to the tooling and requiring a certain number of committed reviewers before a document can advance might work, but I have a feeling there'd be a lot of resistance to this.

---

From: Charlie Perkins <[email-masked]>
Date: Tue, 31 Jul 2018 10:29:15 -0700

Hello folks,

In my note, I said that the IETF Last Call process was somewhat broken.  
I really meant to say that the WG Last Call process was somewhat 
broken.  Sorry about that!

Regards,
Charlie P.

On 7/31/2018 10:03 AM, Charlie Perkins wrote:
> Hello folks,
>
> Please excuse if this is not relevant, or has already been suggested 
> -- I didn't read the whole thread!  The heart of my suggestion could 
> be summarized as reducing AD workload, but along the way making some 
> is motivated by comparison to IEEE task group management, which seems 
> to go more smoothly.
>
> First, I think that the IETF Last Call process is somewhat broken. 
> It's not really Last Call; instead, it is an urgent plea for people to 
> read the document.  It could be compared to a Letter Ballot in the 
> IEEE.  But the implied urgency in the IETF context leads to a sense 
> that the document is almost "done", when in fact the document may have 
> major issues that need a lot of attention. Letter Ballot isn't like 
> that.  All of the comments are tabulated, and subsequent task group 
> meetings are dedicated to resolving the comments. Sometimes, there are 
> Moreover, it sometimes happens in the IETF that WG Last Call ends for 
> a document and no one in the working group has read it.  In contrast, 
> at least in 802.15 Letter Ballots, it is very very likely that quite a 
> few people will read and comment on the document. That's because, 
> essentially, they have agreed to make document reviews as part of 
> their willingness to participate in 802.15, which could be compared to 
> an Area in the IETF (please note this is a very rough comparison).  
> How could the IETF find more assurance that a WG document would get 
> sufficient review?
>
> Lastly, I know that some working groups already try to do some of 
> these things.  I suggest that we will know when the review process has 
> been sufficiently improved, when a document almost never goes through 
> Last Call without sufficient comment.
>
> Regards,
> Charlie P.
>
>

---

From: Kathleen Moriarty <[email-masked]>
Date: Tue, 31 Jul 2018 14:04:29 -0400

On Tue, Jul 31, 2018 at 1:29 PM, Charlie Perkins <
[email-masked]> wrote:

> Hello folks,
>
> In my note, I said that the IETF Last Call process was somewhat broken.  I
> really meant to say that the WG Last Call process was somewhat broken.
> Sorry about that!
>

Your suggestions makes a lot of sense.  While some WGs do this, not all
documents have adequate review prior to progressing documents.   When a
well reviewed document is on a telechat, it often has many positive
comments on it and it is usually quite helpful.  Cross area review from the
IETF last call is important in addition to WG review, so the directorate
reviews help to formalize that part of the problem.

Thanks,
Kathleen

>> Please excuse if this is not relevant, or has already been suggested -- I
>> didn't read the whole thread!  The heart of my suggestion could be
>> improvements to the process of document review and progress.  The idea is
>> motivated by comparison to IEEE task group management, which seems to go
>> more smoothly.
>>
>> not really Last Call; instead, it is an urgent plea for people to read the
>> document.  It could be compared to a Letter Ballot in the IEEE.  But the
>> implied urgency in the IETF context leads to a sense that the document is
>> almost "done", when in fact the document may have major issues that need a
>> lot of attention. Letter Ballot isn't like that.  All of the comments are
>> tabulated, and subsequent task group meetings are dedicated to resolving
>> the comments. Sometimes, there are *thousands* of comments (for documents
>> in large task groups like 802.11ax).
>>
>> people will read and comment on the document. That's because, essentially,
>> they have agreed to make document reviews as part of their willingness to
>> participate in 802.15, which could be compared to an Area in the IETF
>> (please note this is a very rough comparison).  How could the IETF find
>> more assurance that a WG document would get sufficient review?
>>
>> Perhaps the pool of readers should be identified before the WG adopts the
>> document.  If that happened, then when the document is ready for wider
>> review, the reviewers are known and have already expressed a willingness to
>> review the document.  In the IEEE, the reviewers are partially motivated to
>> retain their voting rights, but in the IETF we probably wouldn't want to
>> institute such a system as that.  Nevertheless, having a solid constituency
>> for document review and comment resolution in the IETF would help a lot.
>>
>> sufficiently improved, when a document almost never goes through Last Call
>> without sufficient comment.
>>
>> Regards,
>> Charlie P.
>>
>>
>>
>

-- 

Best regards,
Kathleen

---

From: [email-masked]
Date: Tue, 31 Jul 2018 21:50:14 +0000 (UTC)

onboarding is like waterboarding, but the flood is of documents to read...

Lloyd [email-masked]

On Wednesday, August 1, 2018, 12:43 am, Benjamin Kaduk <[email-masked]> wrote:

-Ben