Subject: [Icar] Summary of Discussion on Reforming IETF Quality Control Process

From: "James Kempf" <[email-masked]>
Date: Fri, 9 Jan 2004 16:40:01 -0800

I've written a draft summarizing the discussions on reforming the IETF
quality control process that started on mpowr in December and wandered to
solutions in January. It's here:

http://www.geocities.com/kempf42/draft-list-quality-control-00.txt

I am not exactly sure what to do with the draft at this point, but would be
interested in hearing feedback. There has been a preference expressed for
discussing the issue on solutions rather than on either mpowr or icar, but
I'm on all three lists and I'll try to respond as time allows. I'd also be
interested in hearing suggestions about how to proceed. If this doesn't look
like a valuable activity, we should apply the procedure described in the
draft and terminate it early. :-)

            jak

_______________________________________________
Icar mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/icar

---

From: Margaret Wasserman <[email-masked]>
Date: Fri, 09 Jan 2004 20:41:09 -0500

Hi James,

I just read your proposal.  Thanks for posting it.

Although I may have more detailed questions later, I have a
couple of quick clarification questions now...

What are the criteria for deciding whether a particular document
requires full IESG review?  And, when would this be decided -- at
charter time, when the document is accepted as a WG item, during
WG Last Call, when the shepherding AD reviews the status of the
document, or at some other time?

In dispute resolution, you seem to indicate that the shepherding
AD will take all disputes to the full IESG.  Currently, the
shepherding AD (or two ADs in an area) make an attempt to resolve
any dispute before it is brought to the full IESG.  Is it your
intention to change that?

Do you have any sort of transition plan for how we could move
to this approach?  Are there useful experiments that we could
run to see if it would work?

Margaret

At 04:40 PM 1/9/2004 -0800, James Kempf wrote:
>Solutions mailing list
>[email-masked]
>http://eikenes.alvestrand.no/mailman/listinfo/solutions

---

From: "James Kempf" <[email-masked]>
Date: Fri, 9 Jan 2004 21:05:24 -0800

Hi Margaret,

That was quick! Here are a couple of quick answers, but if you think this
draft might have legs, I'll get an issues page up on it.

> requires full IESG review?

It is up to the IESG to decide that. The WG can ask for a full review if
they think it might be necessary, but the IESG makes the decision. Also, the
shepherding AD can ask for a full review if s(he) thinks its necessary. In
general, I think the IESG would want to review drafts from WGs that are
proposing new technology or major changes in old technology, but I did not
think it would be such a good idea to be that explicit because it would
decrease the attractiveness of serving on the review board, and, as a
practical matter, the IESG may want at some point to have the review board
do a review on some draft that proposes new technology if it's a small
change or a major change in old technology if it isn't all the different or
if the WG is composed of people who are known to be reliable and world class
experts in the field.

>And, when would this be decided -- at

In the normal case, as soon as the draft showed up as a work item on the
WG's goals list, which is what the process draft should say because some WG
drafts are added after the charter has been written. A review would have to
be done before the draft became a WG draft in any case. The idea here is to
inject a note of predictability into the review process, so the chair, WG,
and shepherding AD know exactly what to do when the time comes for the
review.

However, in exceptional cases, typically when the shepherding AD has
detected that the WG is off track or has reason to believe that the
solutions being discussed by the WG may be impractical or otherwise flawed,
the shepherding AD can call for a review and include an IESG review of
documents, if talking with the WG fails to reorient the WG.

> intention to change that?

Yes, I believe that is probably a better idea and I will add that to the
document. No reason for full IESG intervention if the WG and reviewers can
come to an accommodation with the AD's help.

> run to see if it would work?

One way to start would be to pick a few WGs and give it a try. The list of
SIRS are obvious candidates for initial Reviewers. Obviously, if the process
is applied to some WGs that exist initially, they will not be able to set up
the review process when they are initially formed. I'd say picking groups
such as DHC where many of the drafts involve proposals for new options or
other small changes would be one way to test the review board process, I'm
not sure which ones would be appropriate for the new/deeply changed
technology case. The experiment could be run for, say, a year and closely
monitored to see how much work it really did eliminate for the IESG and
whether it really did free up more time for ADs to spend working with their
WGs, then tuned as appropriate or even discarded if it isn't working.

            jak

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 9 Jan 2004 22:43:57 -0700 (MST)

On Fri, 9 Jan 2004, James Kempf wrote:

> It is up to the IESG to decide that.

I want to clarify this because Margeret's question might be based on
the wrong assumption, possibly causing misinterpretation of the
correct answer.

There should be no "requires full IESG review" flag or state for any
document. IESG is not a special case when it comes to review. IESG or
any single AD can submit a review for any document that is up for
review, at any time. This is no different from any IETF participant
submitting a review. If IESG feels that a particular document needs
full IESG review, it is their internal business, invisible and
unpredictable to others (in general), until they submit a review. In
general, one does not know a priori whether a document will be
reviewed by IESG until the IESG submits the review.

Now, if there is a conflict that WG and a reviewer cannot resolve
despite all the negotiation efforts, then the document automatically
goes to IESG for the final conflict resolution. Such resolution may
require full IESG review, but it is internal IESG business how to
approach that. The final IESG decision is documented, of course. Note
that "a reviewer" above may be IESG (but it is not a special case).

Thus, IESG has special formal powers when it comes to conflict
resolution, but does not have (and does not need!) any special powers
or exceptions when it comes to document review. This scheme is both
simple and gives IESG full flexibility when it comes to selecting
"full IESG review" targets. We do not need to spend time documenting
what those targets might be.

Alex.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 10 Jan 2004 01:42:43 -0500

I may be missing something, but the problem I see reading this is that this 
gives the IESG no assistance in deciding which documents it needs to 
review.  There was another proposal which I read to say ~the IESG needs to 
review documents for which there is disagreement about the results of other 
reviews.~  While one can argue about whether that is good or not, it at 
least spells out when the IESG needs to review a document, and when it does 
not need to.
Obviously, if we can lighten the load enough that the IESG has time to 
review more than it needs to, that is great and they should do so.
Equally, some IESG members will review some documents over and above such a 
"requirement", since they will be interested.
But my understanding of one aspect of this discussion is to help replace 
the current ~the IESG must review everything~ with a useful alternative 
review procedure, and a useful definition of which things the IESG needs to 
review.
(Among other things, such a reduction should help enable the necessary IESG 
reviews to occur in a more timely fashion.)
The text below seems to say taht there are no documents the IESG needs to 
review.  While it would be nice to arrive in such a state, I doubt we can 
get there in one jump.

Yours,
Joel M. Halpern

PS: Can we pick one of the three lists for this?

---

From: Margaret Wasserman <[email-masked]>
Date: Sat, 10 Jan 2004 12:09:52 -0500

Hi James,

[I'm only responding on solutions and icar (not mpowr).  Most
of this discussion happened on solutions, and your proposal is
largely about review/approval structures, so I think it is
more applicable to the proposed ICAR effort than to the proposed
MPOWR effort.]

I've had some time to think about (and re-read) your draft, so I
now I will provide my feedback, rather than just clarifying
questions.  I do support the idea of a tiered review/approval
structure within the IETF, but I have many concerns with the
specifics of your proposal, several of which I also expressed
regarding the SIRS proposal.

IMO, any proposal in this area needs meet certain criteria
for scalability, consistency, cross-area coverage, efficiency,
manageability and accountability. I have some concerns with
your proposal in each of these areas and I have explained them
below.  I also hope that these criteria and my explanations may
be useful to assessing other review/approval proposals, as well.

These concerns are what lead me to originally prefer the type of
per-area review board that Alex Zinin has suggested, rather than
a single large (SIRS-like) review board as you suggest.  I
believe that it would be possible to give single ADs the ability
to approve non-critical documents (criteria TBD) based on
reviews by all of the per-area boards -- in other words, I think
that we can somewhat consider the structure/management of the review
board(s) to be orthogonal to whether some documents can be approved
without full IESG review.

So, on to the criteria...

SCALABILITY
===========

Like the SIRS proposal, I believe that this proposal is quite
naive about the scale of this particular problem.  Let's take
a few numbers:

We currently approve about 200 RFCs per year.  Each of these
RFCs receives (on average) ~2-1/2 review cycles from the IESG
plus a full AD Review.  So, let's assume that we will continue to
produce 200 documents per year, and that each will be subjected
to 3 cross-area review cycles.

BTW, if the same group of people reviews the document all three
times (also see section on consistency below), the later reviews
will take much less time than the earlier reviews.

The IESG consists of 13 people, not all of whom carefully review
each document during IESG review (for various reason).  So, let's
assume that we can get adequate cross-area coverage (see section
on cross-area coverage) by having each document reviewed by
8 properly selected members of the Quality Review Board.  That's
the number of ADs that it currently takes (today, with one slot
empty) to approve publication of a document.

So, the number of individual document reviews required would be
200 * 3 * 8 == 4800.

Let's assume that we can find 200 people who are willing and
qualified to serve on the Quality Review Board.  I don't know
that this is possible, and it means that someone will have to
manage a function that involves 200 people (see manageability
section below), but let's assume...  In that case, each member
of the review board would need to do an average of 24 reviews
per year -- so a minimum of 3 is misleadingly low.  Ideally,
this means that each member of the review board would do a
full 3-cycle review for 8 documents per year.

If we expect this system to result in a 50% improvement in
document throughput, we will need to handle 300 documents per
year, which requires 36 reviews/board member/year (or 12
documents).

Doable?  Maybe.  If we can find 200 people willing to do this,
figure out how to train and organize them (see sections on
preparation/training and management below), and sustain that
number over time.  The sign-up rate for SIRS does not make me
confident that this is possible, but we could try....

CONSISTENCY
===========

There are two types of consistency that are important for IETF
document reviews:

     (1) Consistency across the different reviews done for the
         same document (to avoid thrash).
     (2) Using a consistent set of acceptance criteria for each
         document that is reviewed/approved.

Although we don't always achieve both type of consistency today
(due to personnel changes on the IESG, for example), we usually
do quite well in this area.  The fact that the same group of people
reviews/approves every document is heavily optimized for consistency
(over efficiency, scalability, etc.), and because we don't suffer
too much is this area today, it is easy to underrate the benefits
and importance of consistency.

Type (1) consistency could be achieved by having the same set of
reviewers (to whatever extent possible) perform all of the
reviews for a given document.  In other words, each document
would have its own mini-board of reviewers (8 reviewers, in my
previous example) who would do the initial review and perform
whatever subsequent reviews are needed to determine that new
versions of the document correct problems found in earlier
reviews and do not introduce new problems.  So, this is fairly
easily handled, assuming that we have a manageable method for
assigning reviewers to a particular document and replacing them
as needed, etc. (see manageability section below).

Type (2) consistency is _much_ harder to achieve across a large
board (~200 people in my example above) than it is with a group
of 13 people who spend several hours per week on the phone with
each other, hold retreats and communicate daily...

In order to achieve even a reasonable level of consistency across
a large group (200 people?), I believe that we would need most or
all of the following things:

     - Written review criteria for each type of document.
     - Documented per-area review criteria (like the MIB nits) for
       every area or technical sub-area.
     - A record of architectural/policy decisions that were made
       that can be searched and used by subsequent reviewers.
     - A mandatory preparation/training program (in-person or
       on-line) for all new members of the review board.
     - Some type of mentoring or monitoring program for new
       members during the first NN reviews (most easily
       achieved, perhaps, by having some sort of structure
       within the board?).

Without a plan (including committed resources) to achieve these
things, I believe that a large review board would devolve into
chaos.

There may also be an issue with documents that are passed to the
IESG for final approval.  The IESG's review may not be consistent
with earlier reviews, perhaps resulting in "late surprises" or
demotivation of the WG.  This problem is most likely in a situation
where the IESG has no influence over what reviewers are chosen to
do the initial review, and where the IESG does not feel accountable
for the quality/correctness of the initial review (see section on
accountability below).

CROSS-AREA COVERAGE
===================

The IESG believes that Cross-Area Review is a core value of the
IETF, as well as one of the key properties that differentiates us
from other standards bodies.  Obviously the folks involved in
developing this plan think so, too, because otherwise we would
simply ship documents based on WG technical review.

However, this proposal doesn't seem to do much to support or
facilitate cross-area review.

In order for a document to receive sufficient cross-area review,
it is not sufficient to establish a pool of (perhaps 200)
reviewers and ensure that the document is reviewed by a certain
number (perhaps 8) of those reviewers.  We would need to make
sure that the document is reviewed by a range of people who,
together, have an appropriate breadth of expertise to provide
adequate cross-area coverage.

It isn't easy to put together a small group of people who can
cover all of the technology areas represented in the IETF (as
I'm sure any nomcom member could tell you), and it certainly
isn't possible to populate the Quality Review Team with people
who each have knowledge of the whole set of IETF technologies
and their possible interactions.  As far as I know, there isn't
anyone who could provide a full cross-area review alone (although
Allison can do a passable imitation) -- that's why we have documents
reviewed by IESG members from all areas before they are approved.

A lot of details are swept under the rug of the statement "The
Working Group Chair is responsible for ensuring that all work
done by the Working Group receives adequate and timely review".
How are you picturing that this would work?  How would the
WG chair identify the areas of expertise represented by each
reviewer and gauge their level of expertise in each area?
How would s/he make sure that the appropriate inter-area
issues are adequately covered by the chosen reviewers?

This proposal is largely predicated on the belief that WG chairs
should not be responsible for document quality because their
involvement in the technology may blind them to its technical
flaws.  However, it seems to assume that WG chairs will be
aware of all possible interactions that this technology may
have with other IETF areas and be able (and willing) to
solicit review feedback from the right sort of people to
uncover all of those problems, which IMO won't work.  For
example, if the WG is unaware that their work has major impact
on the application layer, the WG chair may not think it is
important that the work be reviewed by an applications-
knowledgeable reviewer! This seems, to me, to be a pretty big
flaw.

Per-area review boards have a major advantage in this area.  If
a document is passed by every per-area review board (particularly
if those boards are carefully chosen to cover their area well
and provide coverage for inter-area gaps), with the reviewers
chosen by the ADs or other area review board leaders who aren't
active in the WG, then it is more likely that a document will
have received adequate cross-area review than if it is reviewed
by a set of reviewers chosen from a large pool by the WG chair.

EFFICIENCY
==========

One goal of any new review/approval process should be to increase
our document throughput -- publishing documents faster than we
do today, without a reduction in quality.  In general, we should
try to avoid a situation where a document needs to go through
two (possibly inconsistent, see above) review/approval processes.

This proposal might achieve this goal for some documents,
particularly those that do not require IESG review.  However,
the most important documents might be subjected to two rounds
of review/approval -- one by the Quality Review Board and
another by the IESG.  This issue might be resolved by better
defining how/when a decision is made regarding whether a
document will be reviewed/approved by the Quality Review Board
or the IESG.

MANAGEABILITY
=============

In order to run an orderly review process, to assign reviewers
with appropriate (combinations of) expertise, to adquately
training reviewers and to maintain the quality of the review
process, the Quality Review Board would need to be managed.
This proposal does not indicate how the board would be managed,
who would manage it, how they would be chosen, etc.  This is
a major omission that makes it difficult to evaluate many
aspects of this proposal.

Passive voice is used throughout this document, and that tends
to obscure the fact that there is no management structure
defined.  For example, it says "Reviewers can be assigned to
particular Working Groups and have their names listed on the
Working Group Web page as technical advisors".  By whom?  By
the WG chair?

This document does not mention any preparation or training
for members of the Quality Review Board, nor does it
specify any period during which a member's initial
reviews will be monitored by experienced members, etc.
I think that some type of preparation/training is essential
to asking people to assume any role in the IETF process.

The per-area review board proposal relies on the ADs to select
reviewers and manage the process as appropriate to each area.
The AD would be responsible for ensuring that members of the
area boards are adequately trained and prepared, and would
be held accountable for the quality of their work.

ACCOUNTABILITY
==============

This document seems to be based on the concept that quality
review is performed for the WGs, and that the review process
should be accountable to the WGs.  Ultimately, reviewers report
to WG chairs, because they if they are not asked (by WG chairs)
to review at least 3 document per year, they are removed.
I believe that is wrong...

The current AD Review/IETF Last Call/IESG Review system is
not intended to be accountable to a WG.  The IESG (as a review
board) exists outside of the WG structure, as a group that is
accountable to the community.  AD/IESG reviews and IETF
Last Calls serve as a mechanism for the interests of the
IETF community (and perhaps the Internet as a whole) to be
balanced against the interests of a particular WG, and I don't
see this balance adequately represented in this proposal, at
least not for documents that don't receive full IESG review.

I realize that some members of this discussion would like
to move away from the hierarchical structure represented by
the current IESG.  I'm not sure I agree with that approach,
but I'm also not sure that I'm against it.  So far, though,
I have seen any proposal that gives the IETF/Internet
community a strong enough voice that doesn't involve
giving some authority to some selected representatives of
that community...

This doesn't mean that the IESG is the only group that can
review/approve documents, and I do strongly believe that we should
build a more scalable review/approval structure that does not
rely solely on the IESG.  What it does mean (to me) is that
reviewers should be chosen, managed, evaluated and held
accountable by someone other than WG chairs.  This leads to
two choices:

     - Have the IESG manage the review board(s) and the
       review/approval process.
     - Build a different group to manage this process that
       is selected by and accountable to the community.

The second choice has some known weaknesses.  In the past, we
have found that separating review/approval from the group that
was ultimately responsible for WG management lead to serious
problems with the review/approval group becoming out of touch
with the community, etc.  (at least that's a very short summary
of what I've been told about the pre-Kobe IESG/IAB split).
Perhaps we could do this in a way that wouldn't have those
flaws, but most people I've spoken to seem to be against the
idea of splitting review/approval and WG management at the highest
level, as the review/approval group has no motivation to help the
WGs to achieve their chartered goals...

So, perhaps the first choice makes more sense?  That is the
choice that is represented in the per-area review board proposal
that Alex has submitted.

In addition to properly balancing the interests of the WG
and the interests of the IETF/Internet community, the per-area
review board approach makes the IESG directly accountable to
the community for the results of the initial review/approval
process,  making it more likely that those reviews will be
consistent with a later IESG review, if one is deemed necessary.

---

Another advantage of the per-area review board proposal over
this proposal is that it builds upon, rather than replacing, the
work that has already been done to build per-area review teams
such as the MIB doctors, the ops-dir, Transport doctors, the
rtg-dir, etc.

At this point, I think that my comments on the proposal may be
almost as long as the proposal itself, so I'll stop now.

I hope that these comments will be taken as constructive input
to the effort of building a scalable IETF review/approval process,
even though I don't favor the specific type of review mechanism
proposed in this document.

Margaret

---

From: Dave Crocker <[email-masked]>
Date: Sat, 10 Jan 2004 16:33:53 -0800

Folks,

>  reviews

Joel is certainly correct.  If we expect to have the IESG treat working groups 
differentially, then there needs to be community-wide understanding of the 
differences they should filter on.

I think there are a number of factors that ought to be considered:

1. Infrastructure

	If the work is going to hit the core of the Internet, it needs to have 
high-level and broad-based review.  Period.  (In spite of being up in Apps 
space, something like imposing a system-wide spam control mechanism hits the 
email infrastructure and probably should count in this category.)

2. Risk

	Some work is clearly pretty straightforward.  The work, itself, is well 
understood, and/or the core team working on it are known to do this sort of work 
really well.  When the work is not straightforward and/or the team is not known 
to be experienced with this IETF territory, then the IESG should give it more 
attention.

3. Urgency

	When there is rough consensus that useful work is needed Real Soon, then the 
IESG should endeavor to help that work get done quickly.

This does not mean that working groups which do not qualify under any of these 
concerns should be ignored, but it does provide a meaningful basis for 
prioritizing oversight.

It also has the advantage of requiring working groups and the IESG to assess the 
factors at the outset (and perhaps on a continuing basis?)

d/
--
Dave Crocker <dcrocker-at-brandenburg-dot-com>
Brandenburg InternetWorking <http://brandenburg.com>

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 12 Jan 2004 12:23:05 -0700 (MST)

On Fri, 9 Jan 2004, James Kempf wrote:

> I've written a draft summarizing the discussions on reforming the
> IETF quality control process that started on mpowr in December and
> wandered to solutions in January. It's here:
>

James,

	Thank you for writing the draft. Here is my high-level
feedback. I assume we are not interested in low-level polishing at
this time.

1. "The ultimate responsibility for ensuring the quality
   of technical work done in the IETF lies with the Working Group..."

   I suspect you meant:

   "The ultimate responsibility for producing quality technical work
    in the IETF lies with the Working Group..."

   It is correct that WG/authors are responsible for producing quality
   output. It is wrong to imply that WG can ensure that. We need
   outside controls to ensure quality and your wording does not
   reflect that intent.

   Also, how does the above statement is affected by the fact that
   IESG can make WG-unapproved changes to WG documents? Do such
   changes make IESG responsible? Would it be more accurate to
   say that WG is responsible only up to a point where IESG takes
   control and does its conflict resolution thing?

2. "The ultimate responsibility for the quality of editorial content
   of the document describing the work lies with the authors and
   editors of the document."

   Sounds nice, but how does this reconcile with #1 above? The Author
   cannot overrule WG consensus. Are you implying that the author
   should resign if she disagrees with the WG consensus regarding
   her document?

3. "The IESG is responsible for deciding which Working Groups will
   require full IESG review on their documents"

   Wrong granularity, IMO. IESG decision should be made on document
   basis not WG basis, in general. That does not prevent IESG from
   reviewing every document of a given WG, of course, but it does
   not require that either.

4. "For documents that require IESG review or IETF management actions
   involving quality issues, the Area Director is responsible for
   ensuring that ...
   the [IESG] results are communicated to the Working Group, in as
   prompt a fashion as possible."

   A good review mechanism should not require manual involvement to
   communicate IESG (or any reviewer) actions on a document. We should
   avoid problems that manual forwarding of decisions always causes.

5. "Members of the board (called "Reviewers") may be
    subject experts with deep but fairly narrow expertise in one
    particular area, but more often they will have broad experience in
    a variety of areas involving Internet technology."

   I do not understand why deep expertise is worse than broad one.
   Why are you making this distinction? Is this sentence important?

6. Quality Review Board membership.

   I believe that any Board membership approach based on formal
   AD approval and/or popularity contest (IETF voting) is inferior
   to an "open pool of reviewers" approach. I hate to see IETF
   using such rigid mechanisms of "traditional" SDOs in a what is
   supposed to be an open volunteer organization.

   More specifically, I believe that the review pool/board should
   be open to anybody who volunteers and plays by the rules.

   There has to be a mechanism that identifies reviewers
   currently "trusted" by IESG, of course. Such mechanism
   will ensure consistency of reviews and IESG actions should the
   latter be necessary. Designing this mechanism is not trivial,
   but we can succeed if we focus on that rather than "membership
   rules".

   There should be no automatic inclusion or exclusion from the
   pool/board. We should not drag senior IETFers into the pool
   just because we can identify them by their IESG memberships
   or RFC counts. If they want to help, they will volunteer.
   We should not exclude idle reviewers from the pool because
   they did nothing wrong per se (rejecting review opportunities
   should not be counted _against_ anybody; it should be counted
   for solicitors to see and interpret).

7. "A determination of whether a full IESG review is necessary should
   be included in the Working Group Charter in order that the review
   process can proceed predictably"

   I do not understand the justification of this heavy requirement.
   Why is it important to know a priori that all WG documents will
   be fully reviewed by IESG? Will it make the WG work harder? Will
   it make the Chair look more important? Will it make other WG
   feel less important? What is the value to be gained from knowing
   that all WG documents will be reviewed by IESG, especially since
   IESG can review any document without a prior warning? This design
   also assumes that we are good at predicting what needs a
   full review even before the work has started.

   What problem does this WG flag solve??

8. Review Process

   The AD should not remove reviewers from the proposed list. The
   only thing an AD can do is to add reviewers if he thinks that
   proposed list does not provide enough trusted coverage.

   The review request should be sent to all reviewers accepting
   such requests, not just the ones on the AD-approved list.

   The whole design should make unknown but good reviewers known.
   This feature is essential for attracting always new/fresh
   reviewers and ensuring smooth scale.

   Full IESG review should not be a special case as far as
   notification and "handling" is concerned.

   All reviews must be archived and cross-referenced with
   documents and reviewers.

9. Disposition of Review Results

   The reviewers should be responsible for classifying their
   own comments and entering them into the review database.
   The Chair should not be involved at that stage. Again, we should
   not create a forwarding dependency/role when none is required.

   The above suggestion does create some extra work for some
   reviewers, but it is "good" work. It keeps reviews cleaner and
   more constructive. It also avoids problems with Chairs missing
   a comment or mis-classifying it. Finally, it actually saves
   reviewers time in the long run because they can check how their
   comments were addressed without mapping their comments with
   Chair's rendering of their comments.

   The review database/interface should be provided on IETF web
   site for everybody to use. The "issues page" is generated by
   IETF software, not Chairs. This provides everybody with a
   familiar interface and, again, solves "forwarding" problems
   like a forgotten, misrepresented, or even
   changed-after-final-reviewer-check issue on a Chair-controlled
   web page.

   IMO, we need an IETF-wide review management system to facilitate
   key review-related improvements we want.

Thank you,

Alex.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 12 Jan 2004 14:01:12 -0700 (MST)

On Sat, 10 Jan 2004, Joel M. Halpern wrote:

> I may be missing something, but the problem I see reading this is
> that this gives the IESG no assistance in deciding which documents
> it needs to review.

Why do you think IESG needs an assistance with this? Can IESG decide
on a case by case basis, as ADs see new documents popping up in a "now
soliciting reviews" state? What kind of assistance do you have in
mind?

> There was another proposal which I read to say ~the IESG needs to
> review documents for which there is disagreement about the results
> of other reviews.~ While one can argue about whether that is good or
> not, it at least spells out when the IESG needs to review a
> document, and when it does not need to.

You may be talking about the "IESG must resolve reviewer-WG conflicts"
rule I have been pushing for. That rule co-exists nicely with "IESG
may review any document as any IETF participant can" rule.  Put
together, the two rules essentially define how IESG interacts with WGs
as far as reviews are concerned: first as a normal reviewer, then as
the final conflict resolution authority if needed.

> ... my understanding of one aspect of this discussion is to help
> replace the current ~the IESG must review everything~ with a useful
> alternative review procedure, and a useful definition of which
> things the IESG needs to review.

I do not think we can pre-define what IESG needs to review without
being either too broad or too restrictive. I would rather avoid the
whole problem by letting IESG decide what warrants full review. Note
that there is some balance in this approach: since IESG is just a
"regular" reviewer, they are subject to the same deadlines and
timeouts. Thus, IESG (as a whole) should be careful not to overload
itself with non-essential reviews because they will start missing
deadlines on essential ones.

> The text below seems to say taht there are no documents the IESG
> needs to review.

Not exactly. The text below implies that we cannot define a priori
what IESG needs to review, but we give IESG an interface to review
anything that IESG thinks is important enough to be worth their
review.

Alex.

> >There should be no "requires full IESG review" flag or state for
> >any document. IESG is not a special case when it comes to review.
> >IESG or any single AD can submit a review for any document that is
> >up for review, at any time. This is no different from any IETF
> >participant submitting a review. If IESG feels that a particular
> >document needs full IESG review, it is their internal business,
> >invisible and unpredictable to others (in general), until they
> >submit a review. In general, one does not know a priori whether a
> >document will be reviewed by IESG until the IESG submits the
> >review.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 12 Jan 2004 15:13:23 -0700 (MST)

On Sat, 10 Jan 2004, Margaret Wasserman wrote:

> per-area review board that Alex Zinin has suggested, rather than a
> single large (SIRS-like) review board as you suggest.  I believe
> that it would be possible to give single ADs the ability to approve
> non-critical documents (criteria TBD) based on reviews by all of the
> per-area boards

I see no essential difference between a single board with members that
specialize in certain areas and multiple boards with members
specializing in board-specific area. From the point of view of the
important criteria you have been talking about, both designs are
essentially the same, IMO. I am not sure what concerns make one design
more attractive than another (on this level of abstraction). I suspect
the devil is in the details instead of high-level architectures.

For example, "what reviews are sufficient to publish a document" is a
question that can be answered equally well/poor with both single and
multiple boards structures.

If we concentrate on cross-area review and possibly lack volunteers, a
single board/pool sounds like a more natural framework.

> I think that we can somewhat consider the structure/management of
> the review board(s) to be orthogonal to whether some documents can
> be approved without full IESG review.

I agree.

> naive about the scale of this particular problem.

The draft-list-quality-control scalability depends on how many people
the ADs can approve as "certified reviewers" without approving bad
reviewers. I do not see how that process is essentially different from
any other process where there is a "trusted by final Authority" flag
assigned to a reviewer in one way or another.

If that flag is required for review consistency, then all approaches
would scale equally well/badly. It is not a problem we can "solve" by
using a process. We can only eliminate the scalability problem if we
remove the flag and, hence, require IESG to trust at least a certain
portion of unknown-to-IESG reviewers.

For example, we can say that any document that has no review conflicts
and has at least K reviews from each Area, with at least N reviews
from "trusted by AD/IESG" reviewers is published without IESG
involvement. Then you get a natural trade-off between scalability
(number of trusted reviewers required) and consistency/quality
(probability of IESG last-minute killing a document without a prior
review conflict and probability of IETF publishing a bad document
without a review conflict).

> CONSISTENCY

I do not understand what that means. Can you elaborate please?
Consistency in reviewer opinions during a round of reviews?
Consistency in reviewer opinions during the document lifetime?

>          document that is reviewed/approved.

This is important indeed.

> reviews for a given document.

This can be achieved in any proposal I have seen, including
draft-list-quality-control-00, right?

> board (~200 people in my example above) than it is with a group of
> 13 people who spend several hours per week on the phone with each
> other, hold retreats and communicate daily...

I do not see a difference. If you can define acceptance criteria well,
then 10, 100, or 1000 people can use them with good-enough
consistency. If your acceptance criteria are so vague that they can
only exist as an undocumented belief system of 13 close friends
(different from the belief system of the other 13 friends), then we
probably do not need that kind of consistency.

> things, I believe that a large review board would devolve into

How is N*13  reviewers groups would be any different?

I suspect there is either violation of the energy preservation laws in
these arguments or you are comparing apples to oranges.  The amount of
IETF review work that needs to be done should not change depending on
whether you use pre-area boards or one single pool. More precisely, if
the amount of work changes, then we are not comparing apples to apples
and we should decide on what the review system should accomplish
first.

> where the IESG has no influence over what reviewers are chosen to do
> the initial review, and where the IESG does not feel accountable for
> the quality/correctness of the initial review (see section on
> accountability below).

I assumed that consistency with IESG decisions is included in (2). If
it is not, I do not understand your classification scheme, but the
laws of physics should still apply.

> CROSS-AREA COVERAGE
> ===================
>
> this proposal doesn't seem to do much to support or

I agree. The proposal requires an AD to ensure that all areas are
covered. This is a poor design because AD is responsible for one area
only.

We should find a scheme where all areas are represented when it comes
to cross-area review. For example, each area can have a list of
"trusted area reviewers" (all members of the IETF review board/pool).
And at least one(?) review must come from each area. This will ensure
that all areas are represented without relying on one area AD to take
care of that.

> by a set of reviewers chosen from a large pool by the WG chair.

Indeed. However, what you describe as a "major advantage" can be
implemented equally well with a single pool design. Each reviewer has
a set of tags, including flags that mark that reviewer as a trusted
reviewer by an IETF Area Directorate. In fact, I can argue that a
single pool would work marginally better in some cases because Chairs
can easily find reviewers that are trusted by several areas and,
hence, reduce the number of reviews required. Common pool and
interface makes ensuring process rules easier as well.

> particularly those that do not require IESG review.  However, the
> most important documents might be subjected to two rounds of
> review/approval -- one by the Quality Review Board and another by
> the IESG.  This issue might be resolved by better defining how/when
> a decision is made regarding whether a document will be
> reviewed/approved by the Quality Review Board or the IESG.

I agree and proposed some specific interfaces to solve this problem
(to the extent it can be solved when IESG authority differs from the
Board authority).

> MANAGEABILITY

I tend to agree. However, I do note that the amount of required
management can be drastically reduced if we adopt a self-tuning open
pool of reviewers instead of going with a "traditional" managed
board(s) approach so typical to other SDOs and other non-volunteer
organizations.

> be held accountable for the quality of their work.

... which has exactly the same manageability problems and a cross-area
consistency problem added on top. IMO, IETF areas are large enough to
be comparable (in scale) with IETF as a whole when it comes to
manageability

> ACCOUNTABILITY

It would be wrong, but I disagree with your summary. While it is not
explicitly documented, the Review Board essentially works for and
accountable to IETF as a whole (just like a WG does/is). It should be
explicitly documented.

There are some minor mechanisms bugs (like requiring the Chair to
enter comments) that may create an impression that reviewers report to
WG chairs. I already commented on those; they can be fixed.

N.B. idle reviewers are _not_ removed. Only those that idle but also
rejecting review requests are.

> Another advantage of the per-area review board proposal over this
> proposal is that it builds upon, rather than replacing, the work
> that has already been done to build per-area review teams such as
> the MIB doctors, the ops-dir, Transport doctors, the rtg-dir, etc.

Members of those teams should be encouraged to volunteer to join the
open pool of reviewers, of course. We should not replace them, but
provide better access to and visibility for them!

Alex.

---

From: "Michael A. Patton" <[email-masked]>
Date: Tue, 13 Jan 2004 08:12:11 -0500 (EST)

Date: Mon, 12 Jan 2004 15:13:23 -0700 (MST)
   From: Alex Rousskov <[email-masked]>

   I see no essential difference between a single board with members that
   specialize in certain areas and multiple boards with members
   specializing in board-specific area.

I saw this from Alex (and realized a couple others had the same
concern, so this isn't really directed at just this remark, but using
it as a launching point).  Since I'm the one who stood up at the
plenary to point out an essential difference, I guess maybe I should
try and reinforce it.

I, personally, do not consider myself tied to any area.  There is no
area in the IETF in which more than a few of the WGs are interesting
to me and there is no area in the IETF in which I have not been an
active participant in at least one WG at some point.  There are a few
areas that I spend more time with than others, but I think of myself
more as a generalist than as any of these.  As such, I haven't been
tempted to join any of the area-specific oversight groups, but rather
have just offerred reviews on an ad-hoc basis...

I think one of the most important goals of this group is generalized
cross-fertilization, and it's just the sort of generalists who
wouldn't be part of an area-specific group, but might be part of a
single group, who can, possibly, offer the most in such an effort.
So, in that regard, there's probably a fair amount of difference
between those two organizational structures.  They will encourage
different classes of people to participate.

	-MAP

---

From: Dave Crocker <[email-masked]>
Date: Tue, 13 Jan 2004 10:34:05 -0800

Michael,

>  active participant in at least one WG at some point.  

I like this perspective a lot.  I've been concerned that reviewers be 
independent of ADs, but had not thought that an even deeper point is 
that senior IETF contributors nearly always are cross- (or multi-) area.

>  different classes of people to participate.

Nicely said, and many thanks for saying it.

d/

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 13 Jan 2004 11:55:13 -0700 (MST)

Michael,

	I agree with your observation. My no-difference comment was
referring specifically to Margaret's analysis.

	The per-area boards are better if we want to focus on internal
per-area review. A global pool is better if we want to emphasize
cross-area and support no-specific-area review. We will indeed attract
"different classes of people", especially if we also consider open
versus closed pool or AD-ruled versus self-managed board approaches.

Alex.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 13 Jan 2004 22:22:34 -0800

Michael,

let me just drop by to offer a contrarian view.....
in the current IESG review structure, it is important for me to know that 
(for instance) SOMEONE will have looked at the security issues, and that 
SOMEONE will have tried to compile the MIB if there is one.

So the idea behind area-specific reviews is kind of generalizing this 
concept - that after looking at the list of reviewers and the areas they 
are reviewing for, we're pretty certain that the important angles have at 
least been glanced at.

This, of course, presupposes that that some kind of (implicit or explicit) 
"question list" exists for each area review, and that the sum of all areas' 
questions comes reasonably close to "covering the map" - this is an 
untested hypothesis for the reviewer case, but hasn't been too far from the 
truth for the IESG review case.....

                    Harald

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 14 Jan 2004 11:55:32 -0700 (MST)

On Tue, 13 Jan 2004, Harald Tveit Alvestrand wrote:

> let me just drop by to offer a contrarian view..... in the current
> IESG review structure, it is important for me to know that (for
> instance) SOMEONE will have looked at the security issues, and that

Just for the record: The above is easy to ensure with both single pool
of reviewers and a collection of area-specific boards. It is not a
distinction point in the "One single group or per-area groups?"
debate.

In fact, a single pool architecture makes it somewhat easier to
support expertise areas that are not accurately covered by any
existing IETF area. With a single pool, one can "tag" individual
reviewers, and each reviewer may have several tags. The same can be
done in isolated boards, but tagging in a single pool is more
_natural_ than cross-area tagging in area-specific boards, IMO.

Alex.