Subject: [Icar] A modest proposal for moving forward with ICAR

From: "Spencer Dawkins" <[email-masked]>
Date: Fri, 12 Nov 2004 09:06:16 -0500

My apologies in advance for reporting what seems like a coherent idea
that popped out during discussions with Pekka and Joel last night, but
I'd like to suggest a next step for ICAR that takes into account some
realities we're facing here.

- we need experience with ICAR reviews before nailing down new process
  BCPs,
- we have some ICAR reviewers signed up, but probably not enough to do
  ICAR IETF-wide yet,
- we have a lot of reasons to do ICAR reviews, so I think we need to
  focus on one to get started
- I'm making structural suggestions during document reviews for
  Gen-ART ("late reviews"), that people agree are helpful but WAY too
  late in the process,
- we evolve our specifications from individual contributions under
  author control to working group documents under working group 
control,
  in most cases, and
- we don't even have a BCP that talks about what it means to "adopt
  a draft as a working group".

Howzabout we

- identify an AD who is willing to commit one or more working groups
  as experiment fodder,
- assign ICAR reviewers to any individual drafts that are adopted
  as WG items
- review for document structure and completeness THEN, so
  draft-ietf-WG-mumble-00.txt is always our best effort, and
- report the reviews and experiences with this experiment in
  Minneapolis?

This has the advantage of doing something now with the resources that
we have now ("contents may settle due to shipping").

Thoughts and comments?

Spencer 

_______________________________________________
Icar mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/icar

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sat, 13 Nov 2004 21:20:30 -0500

one further suggestion.....

why wait for permission, or to be asked? Why not perform the wg-00 reviews, 
archive them, and build experience that way?

If we can build on a body of experience that says "these wg-00 reviews were 
done, these wg-00 reviews were helpful", I think persuading WG chairs and 
ADs - AND potential reviewers - that this is a beneficial procedure to be 
followed will be easier.

Vigilante Reviewers Of Oncoming Material, anyone?
(VROOM!)

---

From: Alex Rousskov <[email-masked]>
Date: Sun, 14 Nov 2004 10:18:34 -0700 (MST)

On Sat, 13 Nov 2004, Harald Tveit Alvestrand wrote:

> why wait for permission, or to be asked? Why not perform the wg-00 
> reviews, archive them, and build experience that way?

While, in theory, every draft can be reviewed, reviewing a document 
that is not ready/meant for review can be a waste of time or even 
annoying to the WG. Since IETF process assigns special value to -00 
versions drafts with respect to F2F meeting deadlines, reviewing those 
would be especially risky.

If all drafts had a "State of this draft" notices or at least "please 
review" flags, we could select drafts that are ready/meant for 
external review. Until then, I would suggest at least asking the WG 
whether they think a review of a given WG draft version would be 
useful.

$0.02,

Alex.

---

From: Pekka Savola <[email-masked]>
Date: Tue, 16 Nov 2004 00:13:04 +0200 (EET)

> we have now ("contents may settle due to shipping").

One additional concern on ICAR: if there is no "ICAR manager" (such 
that I have been calling for), the likely result is either that the 
folks will try to get the reviews from the minority who are known to 
produce good (and critical) reviews, or from that class of reviewers, 
who are known to not be too critical about their business.

Either way is bad: the first for loading the best reviewers and not 
educating the others; the second for not actually delivering a good 
review and giving an impression that review has been conducted.

In any case, approaching people 1-on-1 is likely to result in longer 
timeouts if the first person(s) don't reply immediately, promise to 
review but don't deliver, requiring you to start from scratch, etc.

So, my thoughts are:
  1) motivation for being a reviewer is a bit lacking.  There was 
idea to create yet another 'dot'.  Maybe this would be enough. Did 
this get dropped?  This is probably needed at least after an 
experimental phase.
  2) an ICAR 'manager' might help in managing the pool, at least at a 
later phase.
  3) let's try to find a pool of 5-10 people: that should be enough to 
bootstrap an experiment with about 2 WGs per major areas (I'd hope).

That said, because I believe this is an important issue, I'm willing 
to volunteer though I'm too busy with other IETF activities as it is.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 15 Nov 2004 20:22:32 -0600

> versions drafts with respect to F2F meeting deadlines, reviewing 
> those would be especially risky.
>
> If all drafts had a "State of this draft" notices or at least 
> "please review" flags, we could select drafts that are ready/meant 
> for external review. Until then, I would suggest at least asking the 
> WG whether they think a review of a given WG draft version would be 
> useful.

I swear that somewhere between me and Alex there's a reasonable idea 
trying to get out...

>From my perspective, I'm late-reviewing documents on the IESG telechat 
agenda for Harald (with the rest of Gen-ART). Almost every telechat, I 
make a suggestion for a document change that, I honestly believe, 
would help implementers, but I'm making it WAY too late in the 
process.

In my favorite recent example, I reviewed the DNSSEC documents, a 
three-doc package that included an introduction (excellent idea!), a 
protocol spec, and a packet format spec.

I'm almost sure that a protocol spec and the packet format spec should 
have the same security considerations section, but these two documents 
didn't (interestingly, all three DID have the same acknowledgements 
section, by value in the introduction and by reference in the other 
two, so it's not like I'm inventing a concept no one ever thought of).

I suggested "do the same thing for security considerations, which 
matters a whole lot more to the Internet than the acknowledgements", 
but I was making a reasonable request for a change to text that had 
been last-called, AD-evaluated, and was being IESG-reviewed as I was 
typing - no reasonable AD would take the suggestion and reset the 
document back to "AD watching" (or worse). So all DNSSEC implementers 
get to read two security considerations sections and decide which one 
is right, if both are right, or if neither are right - from now on.

And I can't imagine that loose security considerations for secure DNS 
is a feature!

So - I was looking for a place to park suggestions like this, a lot 
earlier in the process. We have just about no definition of "becoming 
a WG draft" in our processes, so I was thinking that if we tied a 
review to "becoming a WG draft", that would make sense.

I hear what Alex is saying about making sure that WG 00 drafts don't 
get stalled because someone is waiting for a review. The mechanics I 
see in WGs today are

- somebody writes a draft, and may cycle it more than once,
- the workgroup adopts it as a WG draft, and this seems to be 
something we do most often at/near an IETF meeting,
- the draft gets resubmitted with a new name 
(draft-ietf-WGname-*-00.txt)

What I was thinking was, requesting a review of the individual draft 
before it's submitted as a WG draft, which gives a reviewer just about 
a full IETF meeting cycle (obviously sooner is better, especially if 
the draft is being actively worked between IETF meetings).

And the kind of review I'm thinking about here, is the kind that says 
"what's the relationship between this draft and other drafts? 
published RFCs? will there be IANA considerations? does the draft 
stand on its own? how much background can the editor assume?"

If I got Allman to do this kind of review for my TCP-ish draft, and he 
pointed out all the places I was confused about TCP, that would be a 
bonus - but I think getting an idiot like me to do this kind of review 
for a TCP-ish draft would still be an improvement.

And, obviously, I'm not thinking this is gating for a WG to have a 00 
draft, and obviously, I'm not thinking the reviewer is any kind of 
gatekeeper requiring specific changes. I'm thinking about a 
high-level, optional review...

Others may be solving other problems using other tools, of course :-)

Spencer