Subject: Re: [Sip] Proposals for review process to conclude WGLCs

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 03 Jul 2006 11:04:35 -0400

I really like the first idea - of using a conference call. I think it 
imposes a specific deadline that puts pressure on folks to get the 
reviews done in a timely way. Otherwise, you get embarassed by missing a 
call that you committed to be on. Its the same logic behind the IETF 
draft deadline.

A few more tweaks to what you propose - I'd suggest that reviewers have 
to show up to the call with specific comments on the document, which 
they can work through and raise.

Also, though the output of the review can be go/no-go, lets remember 
everything needs to come through the mailer ultimately.

Thanks,
Jonathan R.

Dean Willis wrote:

> 
> One of the questions we've struggled with is how to get sufficient  
> review on a document under WGLC to make sure it's really done.
> 
> 
> SIPPING is following a new procedure with designated reviewers, doing  
> review by email, and sending stuff back to the. Apparently this  suffers 
> from the expected issues, with designated reviewers sometimes  not quite 
> meeting their review commitments.
> 
> Cisco made me take a peer-review refresher class the other day, so  this 
> started me thinking about some alternatives.
> 
> In the model taught in this class, each peer review  includes 1) a  
> "moderator" or "reader" who frames the questions and keeps things on  
> track, 2) a "recorder" who records conclusions and issues raised  during 
> the review, 3) the author(s), who neither presents no  interprets the 
> work, but does respond to specific questions about it,  and 4) the 
> reviewers, who hopefully have expertise and have committed  time to the 
> review.
> 
> 
> I'd like to propose a couple of alternative models to what SPPING has  
> been doing and seeing if either resonates with our group.
> 
> 
> 1) Real-time review sessions via conference call: When we get ready  to 
> do a final review, the chairs assemble a review team. Presumably,  this 
> review team is assembled by polling the list for volunteers, and  by 
> pushing extra hard for stakeholders and other people with specific  
> expertise. The moderator builds up a checklist and set of basic  
> questions to consider, and distributes it to the review team. At a  tiem 
> agreed to by enough of the participants, we have a conference  call, not 
> to exceed two hours, and work through the checklist in a  moderated 
> fashion. At the conclusion of that two hours, we either  reach a 
> positive decision, a negative decision (with list of issues  and 
> actions), or decide we need more review.
> 
> 
> 2) Moderated email-review, using the entire list as the "review  team". 
> In this model, the chairs pick a moderator and recorder for  the review. 
> The moderator sets up a checklist and sends it to the  list, along with 
> an announcement of the review. The moderator then  goes through the 
> checklist item-by-item on list, polling for  comments. The recorder 
> records any issues or conclusions. Each item  might take two or three 
> days for primary comments, but could be  revisited during the review 
> process if needed. The author could  answer specific questions about the 
> draft, but not paraphrase or  interpret.  At the conclusion of the 
> review, the moderator and  reviewer would report to the chairs, who 
> would decide whether we have  issues, a conclusion, or need further review.
> 
> 
> What do y'all think?
> 
> -- 
> Dean
> 
> 
> _______________________________________________
> Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
> This list is for NEW development of the core SIP Protocol
> Use [email-masked] for questions on current sip
> Use [email-masked] for new developments on the application of sip
> 

-- 
Jonathan D. Rosenberg, Ph.D.                   600 Lanidex Plaza
Cisco Fellow                                   Parsippany, NJ 07054-2711
Cisco Systems
[email-masked]                              FAX:   (973) 952-5050
http://www.jdrosen.net                         PHONE: (973) 952-5000
http://www.cisco.com

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Mon, 03 Jul 2006 11:06:13 -0500

> reviewers, who hopefully have expertise and have committed  time to the 

Dean: This is probably the CMMI peer review (or a derivation
thereof).  I have been part of such reviews for software processes;
they tend to work well assuming the reviewers indeed know
what is going on with the code.  I have also observed that often
times, the reviews get bogged down on how many spaces of indentation
are permissible.  Hence the need for reviewers that are astute.

In the context of your proposals:

> 1) Real-time review sessions via conference call:
[...]
> 2) Moderated email-review, using the entire list as the "review  team". 
[...]

> What do y'all think?

(1) may be more productive, for the reasons that Jonathan mentioned
as well.

Also, I am mapping this to something else that happens in
the IETF -- the NomCom meeting, which is literally, a *peer's*
review instead of a peer review.  At these teleconference meetings,
it is expected that the voting members read the questionnaires/bios
in order to vote on the candidates.  Given that the voting members
are:

   (a) interested in the process, which is why they volunteered
   (b) prepared (by reading the bio and questionnaire)
   (c) on a real-time review session via conference call

things tend to go smoothly in so far as the review of a
candidate is concerned (whether or not everyone agrees that this
is the right candidate is another matter altogether).  Being on
the phone and preparing to verbalize your thoughts appears to
allow introspection of what is it that you are reviewing.

Substitute (a), (b), and (c) above for reviewing WGLC documents,
and I think you'll have a good working model to conclude WGLCs.

Thanks,

- vijay
-- 
Vijay K. Gurbani  vkg@{lucent.com,research.bell-labs.com,acm.org}
Bell Laboratories, Lucent Technologies, Inc.
2701 Lucent Lane, Rm. 9F-546, Lisle, Illinois 60532 (USA)

---

From: "Mary Barnes" <[email-masked]>
Date: Sat, 8 Jul 2006 21:10:04 -0500

You can add another vote in support of option 1), as I think it
increases the level of commitment for the reviewers, along with the
document editor/authors.  It might also prove quite useful in terms of
resolving some issues in real-time, thus reducing the amount of
thrashing on specific issues on the list due to misunderstandings via
email.  As you've already identified, a recorder would be imperative for
the process. 

I also support the idea of SIP doing something different than SIPPING
and at year end we can evaluate both options and see what seems to work
best. I'm beginning to think the SIPPING approach might be much better
applied early in the cycle (i.e, for the first few iterations), but that
the conference calls might be the most effective way to handle the
WGLCs.    

Obviously, any of the approaches requires committed and dedicated
reviewers for the documents.  The response in the SIPPING WG has been
pretty good, but I think it would be optimal to add the requirement that
a document isn't accepted as a WG document, unless you also have the
reviewers committed.    This might delay some things, but in my mind, if
we don't have WG members committed to the work, along with the
editor/authors, we don't have the resources to effectively complete the
work (in a timely manner).  

Mary

-----Original Message-----
From: Dean Willis [mailto:[email-masked]] 
Sent: Friday, June 30, 2006 3:26 PM
To: IETF SIP List
Subject: [Sip] Proposals for review process to conclude WGLCs

SIPPING is following a new procedure with designated reviewers, doing  
review by email, and sending stuff back to the. Apparently this  
suffers from the expected issues, with designated reviewers sometimes  
not quite meeting their review commitments.

Cisco made me take a peer-review refresher class the other day, so  
this started me thinking about some alternatives.

In the model taught in this class, each peer review  includes 1) a  
"moderator" or "reader" who frames the questions and keeps things on  
track, 2) a "recorder" who records conclusions and issues raised  
during the review, 3) the author(s), who neither presents no  
interprets the work, but does respond to specific questions about it,  
and 4) the reviewers, who hopefully have expertise and have committed  
time to the review.

1) Real-time review sessions via conference call: When we get ready  
to do a final review, the chairs assemble a review team. Presumably,  
this review team is assembled by polling the list for volunteers, and  
by pushing extra hard for stakeholders and other people with specific  
expertise. The moderator builds up a checklist and set of basic  
questions to consider, and distributes it to the review team. At a  
tiem agreed to by enough of the participants, we have a conference  
call, not to exceed two hours, and work through the checklist in a  
moderated fashion. At the conclusion of that two hours, we either  
reach a positive decision, a negative decision (with list of issues  
and actions), or decide we need more review.

2) Moderated email-review, using the entire list as the "review  
team". In this model, the chairs pick a moderator and recorder for  
the review. The moderator sets up a checklist and sends it to the  
list, along with an announcement of the review. The moderator then  
goes through the checklist item-by-item on list, polling for  
comments. The recorder records any issues or conclusions. Each item  
might take two or three days for primary comments, but could be  
revisited during the review process if needed. The author could  
answer specific questions about the draft, but not paraphrase or  
interpret.  At the conclusion of the review, the moderator and  
reviewer would report to the chairs, who would decide whether we have  
issues, a conclusion, or need further review.

What do y'all think?

--
Dean

---

From: Ben Campbell <[email-masked]>
Date: Sun, 9 Jul 2006 22:48:27 -0500

Time zones are going to be an issue with any real-time review  
process. Maybe it is an issue we can deal with, but I suspect some  
number people who would otherwise participate may be locked out, or  
at least discouraged, by the idea of doing conference calls in their  
normal sleep cycle.

---

From: Dean Willis <[email-masked]>
Date: Mon, 10 Jul 2006 00:22:28 -0500

On Jul 9, 2006, at 10:48 PM, Ben Campbell wrote:

> at least discouraged, by the idea of doing conference calls in  
> their normal sleep cycle.

You people sleep? No wonder we're missing milestones! Keith, we need  
to recalculate all the milestones to accommodate this sleep thing.

Seriously, one would have to schedule an individual review to meet  
the schedules of the selected review team. We'd have to be careful  
not to mix people in very bad configurations. This can usually be  
sorted out for groups of 3-5 people selected from a large pool,  
assuming they're willing to work between 7:00 AM and midnight in  
their home time zones. Of course, that does leave out at least one  
area director I know of . . .

--
Dean

---

From: Ben Campbell <[email-masked]>
Date: Mon, 10 Jul 2006 09:43:29 -0400

On Jul 10, 2006, at 1:22 AM, Dean Willis wrote:

>> number people who would otherwise participate may be locked out,  
>> or at least discouraged, by the idea of doing conference calls in  
> You people sleep? No wonder we're missing milestones! Keith, we  
> need to recalculate all the milestones to accommodate this sleep  
> thing.
>

For that group size, I agree. I had somehow assumed larger groups for  
no particular reasons.