Subject: Re: [Icar] Input based on SIRS experience

From: Dave Crocker <[email-masked]>
Date: Sun, 11 Jan 2004 18:50:20 -0800

(How embarrassing.  I've no idea why my previous posting wrapped so 
strangely, other than the quirkiness of a new mail client.  Here's an 
attempt to re-post, maybe without the problem... /d)

Adding my own thoughts to the archive, since the topic came up, and with 
the hope it can added to the discussion:

-    It was not much effort to get a core of reviewers signed up.

This suggests a good core of community motivation to help.  We have not 
needed more, so we do not know whether an effort like this can attract 
_enough_ qualified reviewers.

-    I am not sure how to interpret the low number of review requests.

It is easy to say that it is due to lack of community awareness or 
interest or trust in the SIRs effort.  Certainly we have not done as 
much marketing as we might have wished. However this view is confounded 
by the very slow metabolic rate of the IETF and the fact that the effort 
began at the beginning of a summer. 

No doubt there probably are other factors.  Cliches like 'tipping point' 
and 'crossing the chasm' were created for just this sort of issue in 
adopting change. For example, it may be that it is more difficult to 
change existing working group efforts than it is to get integrated in 
with new(er) ones. Certainly we have not done any serious, sustained 
effort to get working groups to request reviews.

On doing the administrative side of this effort, there are some things 
that I suspect are strategic, besides the relatively obvious fact that 
this really does take some ongoing, administrative effort, more than I 
had expected. (But, then, I always underestimate these things.)

-    It needs to be easy to have a record of the review process for a
     document and, by derivation, an ability to see the whole set of 
     review efforts.  

-    There should be a central point for tracking review efforts and a 
     central point for accessing reviews.

Besides the obvious, local and tactical benefit to the individual 
working group, this permits the community to get the ability to review 
the _set _of reviews and get a sense of the effort and result of doing 
reviews.  In other words, it has marketing value.  It makes it easy for 
others to see that there is review activity and it makes it easy for 
them to evaluate the nature and quality of those reviews.  This will 
make them more comfortable getting reviews for their own efforts and 
will help them decide on what reviewers to ask for. Whether this needs 
to be a permanent, formal archive, like RFCs, an ephemeral, formal 
archive, like I-Ds, or an unofficial, ephemeral archive like a WG 
mailing list, has not been resolved.

My own thought is that formal publication (RFC and even I-D) is far too
much overhead.  The only existing mechanism is the mailing list. 
Creating a new mechanism strikes me as too much effort, and added effort 
is a very large barrier to adoption of this process.

Opening up the IETF Tracker to working group chairs, and adding the 
review construct and/or allowing free-form entries would help this 
enormously. (Free-form means that the IETF does not have to formally 
adopt the thing being tracked.)  This means reviews -- and other 
activities -- could be tracked much sooner than would be possible if we 
first have to get IETF-wide approval of the review process.

-    I am not sure whether selection of reviewers, by requesters,  needs 
     to be different than we set up.

A web page with a list of bios, and a mailing list for sending requests
to is probably fine.

The public discussion about reviewing shows that the community is still 
working on clarifying what types of reviews are needed, when, and how to 
use them.

-    There is also the small matter of the SIRS reviews that have been 
     done.

Have they been they type that are needed?  Should they be done 
differently? We have not tried to assess this. Certainly we cannot 
expect regular, widespread use of a review process until working groups 
know what to ask for, or at least what to expect they will get and how 
they should use the results.

d/

--
Dave Crocker <dcrocker-at-brandenburg-dot-com>
Brandenburg InternetWorking <http://brandenburg.com>

_______________________________________________
Icar mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/icar

---

From: "James Kempf" <[email-masked]>
Date: Mon, 12 Jan 2004 10:01:27 -0800

Margaret Wasserman said:

>Like the SIRS proposal, I believe that this proposal is quite
>naive about the scale of this particular problem.  Let's take
>a few numbers:
>
>We currently approve about 200 RFCs per year.  Each of these
>RFCs receives (on average) ~2-1/2 review cycles from the IESG
>plus a full AD Review.  So, let's assume that we will continue to
>produce 200 documents per year, and that each will be subjected
>to 3 cross-area review cycles.
>
>BTW, if the same group of people reviews the document all three
>times (also see section on consistency below), the later reviews
>will take much less time than the earlier reviews.
>
>The IESG consists of 13 people, not all of whom carefully review
>each document during IESG review (for various reason).  So, let's
>assume that we can get adequate cross-area coverage (see section
>on cross-area coverage) by having each document reviewed by
>8 properly selected members of the Quality Review Board.  That's
>the number of ADs that it currently takes (today, with one slot
>empty) to approve publication of a document.
>
>So, the number of individual document reviews required would be
>200 * 3 * 8 == 4800.
>
>Let's assume that we can find 200 people who are willing and
>qualified to serve on the Quality Review Board.  I don't know
>that this is possible, and it means that someone will have to
>manage a function that involves 200 people (see manageability
>section below), but let's assume...  In that case, each member
>of the review board would need to do an average of 24 reviews
>per year -- so a minimum of 3 is misleadingly low.  Ideally,
>this means that each member of the review board would do a
>full 3-cycle review for 8 documents per year.
>
>If we expect this system to result in a 50% improvement in
>document throughput, we will need to handle 300 documents per
>year, which requires 36 reviews/board member/year (or 12
>documents).
>
>Doable?  Maybe.  If we can find 200 people willing to do this,
>figure out how to train and organize them (see sections on
>preparation/training and management below), and sustain that
>number over time.  The sign-up rate for SIRS does not make me
>confident that this is possible, but we could try....

and Spencer Dawkins said:

>  Yeah, Margaret is also coming up with 200-300 reviewers needed, and
>  we  still have no idea how many we can get. :-{

It seems to me that, in principle, the only difference between an IETF
review board and the program committee for a conference is that the members
of an IETF review board would be expected to stick with a draft for more
than just one review, and possibly in the number of reviewers per paper. So
it seems like we might be able to look at existing experience with
conference program committees for some idea about how to make this work.

As an example, I'm on the program committee for Mobihoc this year. There are
42 program committee members and 230 papers which works out to a ratio of
about 6 papers per reviewer. I was assigned 11 papers, from which I conclude
that the program chairs have decided to have about 2 reviews per paper.

If one accepts Margaret's numbers of about 24 reviews per year, that would
imply that the number of reviews for IETF drafts would be about twice what
Mobihoc members are expected to do. This doesn't seem all that bad to me
considering that I'll probably do at least 11 additional reviews this year,
of IETF drafts and papers for other conferences, etc. Working backwards from
the Mobihoc program committee, that would reduce the number of review board
members to maybe 90 (twice the Mobihoc number) rather than 200.

To do this, however, we would have to reduce the number of independent
_solicited_ reviewers to 2 rather than Margaret's suggestion of 8. Would
this cause a reduction of quality detection problems? Unclear. Many drafts
now receive unsolicited review as part of WG last call and IETF last call
and presumably that would not change. Margaret's number of 8 reviews was
based on an approximation of how many ADs review drafts, so YMMV. And, even
with 8 people in the IESG reviewing, there have been occasions when problems
have been caught later (MIPv6 binding update security comes to mind). Also,
with an issue tracking system in place, the reviewers may be able to
determine whether issues are resolved more easily without having to wade
through the text in the spec, so reviews might become less time consuming.

I think there is a different issue that is possibly more of a barrier,
getting people to volunteer by providing the right incentive. People
volunteer for a conference program commitee because they want to see what
other people in the area are up to from a reseach standpoint, and they want
to make sure that published results are of high quaility. The program
committee's comments are used to determine what papers are ultimately
published, that is, their reviews make a real difference in the conference
content.

If the review board's work is just advisory, as the ART and SIRS proposals
do, then I think we will have a hard time getting people to volunteer.
Imagine a program committee for a conference in which the reviews produced
by the program committee could be overridden by the program's organizing
committee, or even by the authors (this would be the equivalent of the IESG
or the WGs themselves, respectively, being allowed to override a review
board opinion). Who would volunteer?

There is another kind of conference program committee in which the authors
themselves are required to do reviews. This could be another way to solicit
reviews, people who are draft editors are required to provide reviews.
Obviously, since draft editors are less experienced, their reviews could not
be as highly rated as the review board's reviews. In such conferences, I
presume the organizing committee has the final say, and typically these
kinds of conferences are not first tier.

            jak

---

From: Pekka Savola <[email-masked]>
Date: Mon, 12 Jan 2004 20:52:32 +0200 (EET)

> conference program committees for some idea about how to make this work.

Note that there may be some unstated assumptions here.

For example, that such an IETF Review Board would replace some 
functions of the IESG, such as:
 - AD Evaluation
 - Review for RFC-Editor submissions
 - Review of WG submission of Info/Experimental documents
 - Review of individual submissions of --""--
 - Review of standards track documents as well

If you assume that (or all of these), you will undoubtedly arrive at a 
conclusion that any proposal will lack manpower.

However, if the idea is to improve cross-area or cross-functional 
review so that some of the above will go *easier*, requiring much 
smaller amount of time, we might be getting somewhere.

So, when proposing a review board(s), it's useful to explicitly state 
which functions we want to replace (or augment).

> do, then I think we will have a hard time getting people to volunteer.

I'm not sure if this is necessarily the case.  Speaking from personal
experience, there is several kinds of responses a reviewer will not
want to see:

- "it's too late to make reviews [especially on this specific issue
[which may have had consensus at X]], but thanks anyway", "

- (comments are received, but they're not taken into account)
  * at least in a timely fashion (following up within days is easier 
    than after months)

> committee, 

Organizing committee would not be a problem, at least to me -- you can 
consider it as "they know best" -- at least after discussion.

> or even by the authors (this would be the equivalent of the IESG

The authors and WGs are a different thing entirely.  If they failed to
see a problem in the first place, I would not count on them to judge 
whether a problem needs to solved or not.
 
> reviews, people who are draft editors are required to provide reviews.

Heh, an interesting idea :-)

Along the same lines, you could also require that each WG chair must
make N reviews every year.  An instant base of 200+ reviewers :-)

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings