Subject: [Icar] Notes from ICAR

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 4 Aug 2004 17:19:26 -0700

These are my notes from our session today (and I apologize in advance for HTML),

Spencr

Improved Cross-Area Review WG (icar)

Wednesday, August 4 at 1530-1730

================================

CHAIRS: Mark Allman <[email-masked]>

        Joel Halpern <[email-masked]>

AGENDA:

1.  intro/agenda bashing - co-chairs

2.  discussion of draft-ietf-icar-experiment-early-review-00.txt -

    David Partain

Draft to capture the state of WG discussions - supposed to capture WG mindset

Open issues:

- guidelines, if any, for the form of the review?

>From TSV - ask for important/crucial issues at the top of the review, as a summary of the review

>From ITU - major technical, minor technical, editorial categories

This isn't about binding, it's about demonstrating to ADs that "major technical" items have been considered

Working groups have to consider comments, can't just ignore them

What is "must respond"? Working group is free to say "we're right", not free to say "we've never heard this comment before"

- information about potential reviewers? How many requests, how many reviews ...

Number of reviews is also a recognition item for reviewers

Average time to complete a review per page of document

Need to be collecting raw information as we go

Need to avoid "top 10" lists of reviewers, etc.

Don't set the threshold to become a reviewer too high - need to be bringing in new reviewers (see later discussion)

Reviewer specializations? Mark will show what we have so far

What kind of review are we asking for? Very specific topics, or more general?

Need to accommodate specialist reviewers and generalist reviewers

A little history is a good thing...

- Subjective criteria for admittance to the reviewer pool?

Start by doing real work before you're admitted? AD can just decide? 

Requests are published publicly, so one could establish a track record

Sponsorship by an AD? Definition of "good" for an AD is "what I would have figured out anyway"?

Volunteering and accepting/rejecting should be in secret

If "n" (reviews) is sufficiently large, that flushes out a lot of the idiots

"Sorry, but your reviews don't seem useful" - as a teaching tool - but need to keep from turning ADs into fulltime mentors - are reviewers expected to be willing to serve as mentors? Could this be this an area directorate responsibility (defined broadly)?

EDU team should (obviously) be willing to help here

Could have a separate pool of mentors, set up independently

Are we doing early or late reviews? ICAR is focused on early review - the hope is that this helps with late reviews as well

WG needs some assurance that they are likely to get a good review from anyone in the pool

Nothing but early review can reduce late surprises - this needs to be part of the success criteria for ICAR

Some set of reviews plus AD sponsorship? 

ADs can sponsor anyone, but recommend looking at previous reviews? We trust ADs to do a lot more than make choices like this without adult supervision

(Objective - it's "two or more" RFCs, not "two")

- How do we measure success/failure of ICAR experiments?

Some types of failure are easy to detect, but success is harder - look for absence of failures and keep going?

Objective criteria for failure, subjective criteria for success

Helped or hindered in RFC production? Experiment will be over long before we get an RFC out through this mechanism

Ask the chairs, not the document editors, for success, but ask the document editors for feedback as well

- How are people removed from the pool, or is this process needed at all?

At least remove yourself!

Let bad reviewers rot on the vine?

Need a minimal "periodic survey" for self-removal? People should "time-out" after six months, or something

Removal on the basis of quality - one person's crank is another person's helpful reviewer

If you qualify and time out, you still qualify when you ask to be added back

ADs need a removal-for-cause mechanism - we do remove WG chairs from time to time

If reviewers don't provide reviews - performance criteria for remaining in the pool?

This is a voluntary process anyway - why do we need to throw people out of the pool? What's the worst case? Is it as bad as what a WG chair can do? We can throw them out now...

Are we talking one percent bad reviewers or ten percent bad reviewers? We don't need to create an expulsion mechanism for the experiments

We pretty much agree we need a timeout

What about a reviewer with an ax to grind? Is "serve at the pleasure" language good enough? Is "the grapevine" good enough? Bad reviewers are worse for new working groups with new chairs. 

Why is this worse than a bozo appearing at IETF last call? The pool is supposed to have more clue, and the working group is supposed to respond

What about reviewers that don't review? Just timing out is good enough

Underburden bad reviewers, ignore them, or overburden them? We already have three mechanisms

There is a difference between a process that's seldom used and one that doesn't exist

We don't have consensus here - take this question to the list

How does AD removal map to pre-qualified participants? To the list...

- What can we hope for from the IESG? How do reviewers, prospective reviewers, and working groups interact with IESG?

IESG has promised Harald they will provide sacrificial working groups for the experiment, and are concerned about providing names of people who don't have time to review documents for ADs now

"Perhaps your working group should get an ICAR review"

We're asking "help us find working groups, help us find reviewers, sponsor reviewers" - IESG is mumbling on the middle answer

- Other comments?

Non-performance will be a problem - need backup reviewers - yes, and WGs can handle this however they want to - we could learn this in an experiment, instead of noodling about it

We are depending on volunteers - dates and plans to follow

3.  the ICAR web page and soliciting for volunteers - Mark Allman

Web page available from ICAR charter page

Describes the proposal for an experiment

ADs can request reviews for non-WG documents, too

Pool members send reviews to icar mailing address

WG considers comments and suggestions included in the reviews

Reviewer attributes - expertise, reviews, ideal queue depth, current queue depth, plus arbitrary bio

Reviews and responses available from reviewer page

Could be sliced by WG, by draft, etc.

Logging time when information about requests and reviews are entered into the system

Feedback? Codify expertise? At least sort by expertise? Grade expertise? What about subjective grade inflation? What about "wireless", "XML", etc.

4.  experience with the general area review team - Harald Alvestrand

Gen-ART is late review - one week before IESG processing - and general review, including readability

IETF chair unable to get responsible and timely reviews, and other people had directorates and review teams - so asked for help

Called an experiment, getting experience with fast reviews, identify what turns out to be problems, and actually publish reviews

One-area pilot, so Harald didn't have to ask for agreement, and Harald already had a web server and mail server

Review guidelines stolen from SIR, driven by IESG mechanics

Reviewers assigned round-robin, reviews and names are public, started January 22

Reviewers recruited word-of-mouth, "known to the AD", but also asked for volunteers on ICAR, no reviewers have crumbled yet

One-person, one-document? Don't double up, because we have 20 documents per telechat and less than 40 reviewers

10 reviewers, 135 reviews, lots of concerns, 5 DISCUSSes (at least)

How did this compare to SIR experiment? This is review at a different level

Load on AD: down. Community involvement: Up

Lessons learned - one week is a short time, reviewers need a discussion forum, ten people isn't enough, it needs to be a team, procedures are necessary, IT support is optional but nice.

Trying to move standards-track review to IETF Last Call time

Would a single discussion forum scale to 200 reviewers?

Unresolved issues: Follow-up is important. Procedures for pointing at information is important. Getting the WG's attention for non-critical issues isn't a solved problem.

Not sure who to send reviews to (ADs, WG chairs, editors, or some combination).

Reviews need to be more visible to the community (you can find them now, if you're psychic).

5.   General Discussion

What sort of expectations for reviewer responsiveness on committing to a review?

Hard deadlines help us focus...

Short timeframes are good, when we're asking for a commitment - Dave will suggest a number on the mailing list
_______________________________________________
Icar mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/icar