Subject: RE: [Icar] Moving on with ICAR: WG formation and charter

From: "Robert Snively" <[email-masked]>
Date: Wed, 17 Dec 2003 12:47:50 -0800

I second Alex's concerns.  Given a separation of
technical review and process review and the formalization
of a self-consistent and monitored process like those
I have suggested in previous mails, the cross-functional
review is trivial and built in.  Without some sort of Big Picture
changes, cross-functional review will at best be spotty
and will probably overwhelm some sub-set of the participants.

Since ICAR has not yet seen my suggestions, I am enclosing
the most specific of them for you to target as you choose.

Bob Snively
408-333-8135

> -----Original Message-----
> From: Alex Rousskov [mailto:[email-masked]]
> Sent: Wednesday, December 17, 2003 9:08 AM
> To: [email-masked]
> Subject: Re: [Icar] Moving on with ICAR: WG formation and charter
> 
> 
> On Mon, 15 Dec 2003, Michael A. Patton wrote:
> 
> > 1. Do you believe that a WG should be formed to work on
> >    improving IETF cross-functional review?
> 
> No, such a WG should not be formed.
> Or, at least, such a WG should not be formed now.
> 
> We might need a WG to improve IETF reviews and review process. Such a
> WG can only succeed if it has clear understanding of what existing
> roles and rules it can change as a part of its "review improvement"
> proposals. At this time, there is no sufficient agreement with regard
> to Big Picture solutions to provide such a clear understanding in ICAR
> charter.
> 
> If chartered, the WG will either waste time on micro-level solutions
> that are likely to conflict with some Big Picture changes OR will
> immediately stumble on Big Picture questions such as "how we can
> enforce early reviews", "how we can make intermediate reviews
> consistent with the IESG final review without removing IESG from the
> final review loop", "who has the authority to resolve review
> conflicts". If the WG is to assume that there are no Big Picture
> changes, I question the ability of such WG to produce good solutions;
> many drafts quoted in the proposed charter imply Big Picture changes.
> 
> Once/if the Big Picture becomes a bit more clear, we can come back and
> write a good ICAR charter within that Big Picture.
> 
> [If we want a formal WG now, we can charter a Big Picture WG that will
> include review improvement as a possible motivation for the big
> changes. The purpose of that WG would be to engineer IETF Core changes
> or to conclude that no Core changes are needed.]
> 
> > 2. Would you be willing to actively participate in this WG (by
> >    contributing to or reviewing the documents, participating in the
> >    discussions, etc.)?
> 
> Yes, but given the above arguments, I suspect my contribution would be
> limited to a rather negative review/comments and, hence, may be
> perceived as unproductive by those who already decided on Big Picture
> changes or their absence.
> 
> > 3. Please review the part in [Ref1] that provides a preliminary
> >    description of the WG and send your comments.
> 
> a) The scope of the WG is unclear or wrong. "Cross-area" scope is too
>    narrow (assuming formal IETF areas are implied). "Cross-functional"
>    scope is undefined in the proposed charter. It is not clear
>    whether "cross-functional" predicate eliminates something useful.
> 
> b) The playing space for the WG is undefined: it is not clear whether
>    the WG is allowed to propose changes to core IETF processes, rules,
>    and roles and, if yes, which processes/rules/roles are in change
>    scope.
> 
> c) The charter assumes that "IESG review function" remains mostly
>    unchanged. There is currently no IETF consensus whether that
>    function needs to change, IMO. Even some of the quoted drafts
>    seem to imply significant changes to IESG review function.
> 
> d) The implied difference between "peer" and "structural" is unclear
>    to me. Define and contrast both terms better since your milestones
>    depend on the difference. Furthermore, an a priory assumption of
>    separation between "peer" and "structural"  review may hinder WG
>    creativity.
> 
> HTH,
> 
> Alex.
> 
> P.S. Note that rejection of ICAR WG does not imply that SIR and
>      similar trials should be killed. In fact, IESG should promote
>      and publicize such trials as hard as it can.
> 
> _______________________________________________
> Icar mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/icar
> 
> 

Subject: RE: [Solutions] lets back up a step
From: "Robert Snively" <[email-masked]>
To: "Keith Moore" <[email-masked]>, "Alex Rousskov" <[email-masked]>, "Pekka Savola" <[email-masked]>
Cc: <[email-masked]>, "Robert Snively" <[email-masked]>
Date: Wed, 26 Nov 2003 09:47:27 -0800

Keith and Alex,

Your thoughts are right on target.

One possible approach to this is to have a well understood
process executed by the working group during their 
collaboration.  Such a process should have clearly defined
and mandatory check points, with clear requirements to be 
met at each checkpoint.  At the same time it should allow
a maximum of flexibility in the development process and
working group culture, including the choice of guided or unguided
e-mail discussions, phone and video conferences, and interim 
face-to-face meetings.  Public archival of agendas, minutes, 
and talking-paper proposals not mature enough (or perhaps 
not in the right format) for internet drafts is also 
necessary.

As a possible example:

1)  Proposal for project:

	Collaborate to specify a project proposal that
	clearly scopes the activity and defines the
	external points of contact and expert liaison
	that must be achieved to complete the project.
	It also defines the compliance points that must
	be verified, including security and proper
	architectural layering with related documents.
	The proposal itself should be consented to
	in a manner similar to 3 below, so that everyone
	knows what is being worked on and agrees to it.  

2)  Develop the draft

	The working group should have great flexibility,
	facilitated by the wg chair, to develop this.

	Since the expert liaison and external points
	of contact have been identified and warned that
	such a project is going on, they are likely to
	provide at least some support during this
	development process.  The chair may have to add
	additional interaction points as the draft
	develops.

3)  Poll the working group and liaisons for consensus

	The last call process is kind of a pass at this,
	but there are often dozens of folks who participated
	in the development, but do not respond to last call.
	This may be because they trust that others will do
	it, or it may because their mailboxes were full that
	week.  There should be some kind of mechanism
	(some would call it "membership") that requires a
	response, like a balloting pool.  The personal
	responsibility for responding focuses attention
	on the document in a very constructive manner.

	This is really just formalizing the last call process.

4)  Resolve comments

	This is really a continuation of the development
	collaboration, but with the certainty that every
	known liaison and balloting pool member has done
	a useful review of the document.  This may
	take as long as the original document development.

5)  Approve the resolutions

	This is in the nature of a recirculation ballot.
	It may point to the need to go back and resolve
	more comments, but usually not.  The recirculation
	process should be similar to step 3.  Perhaps
	restricting the ballot pool to those who responded
	in step 3 would provide motivation for more
	complete review at step 3.

	At this point, you should have a pretty good document,
	one that the IESG or AD could probably not improve on.  If
	they were able to improve on it and were interested in
	the subject, they should have been part
	of the liaison and balloting pool.

6)  Wide-spread review process

	At this point another widely announced review process run
	by the central authority is useful, since the project 
	proposal's scope may have missed some interactions with 
	nominally unrelated technologies.  This should be announced
	widely (say in one of the Standards journals).  My experience
	is that comments at this stage are very unusual, but
	when they occur, they are almost always extremely important.

	This is the point at which the AD or IESG can say "halt, go
back"
	if real technical issues have crept through both the
	liaison and review processes.  They would be just 
	another of the public reviewers. The corrections of
	course would go back to step 4 in the working group,
	causing great embarrassment to the chair :-)

7)  Procedural review in parallel with public review process

	The central authority in parallel can go through the check boxes
	and if necessary the document archives (including documented
	comment resolutions and perhaps other documents) and see 
	that the process was completed correctly.  

Then presto, you have a fully collaborative and flexible development
and review environment and at the same time a well understood
and rigorous set of check points that force early and wide
oversight of the documents resulting in excellent standards.
Well, mostly, since stuff still happens.

What does this do for rough consensus and running code?

Running code often begins before the first balloting check
point, so problems discovered are fed back in that time frame.
Those few comments identified during the central authority's
public review process are also often part of the running
code verification.  People often do not identify all the 
weaknesses in a document's language until they do try to
implement it. 

The rough consensus is really made a bit stronger than before, 
since the flexible collaboration is performed within the
skeleton of formal review checkpoints with rigorous requirements. 

And of course, for any widely used document, new interpretations and
functions will be desired, so revisions will flow naturally
as new projects.

This process would produce nice clean standards track RFC's 
and provide a solid basis for their promotion to internet
standards.  Weaker procedures for other types of documents
could be created.

Bob Snively
[email-masked]
408-333-8135

> -----Original Message-----
> From: Alex Rousskov [mailto:[email-masked]]
>...
> I suspect that we cannot delete or move the function of a top-level
> enforcer.  There has to be some external control and it has to come
> from the top level. However, we can probably make it so that it takes
> relatively little effort and time for the top-level authority to make
> a determination:
> 
>   - The Authority must not review documents; instead,
>     it must make sure submitted documents have received
>     sufficient number of quality external reviews.
> 
>   - The Authority must not negotiate how reviewer
>     comments can be addressed; instead, it must check
>     how they were addressed.
> 
>   - The Authority must not spend endless cycles requesting
>     additional reviews and clarifications; instead,
>     it should reject any partial submission (i.e., submission
>     without sufficient number of quality external reviews or
>     without an easy-to-digest list of reviewer-identified
>     issues, how they were addressed by the WG, and what the
>     reviewer final reaction was after the WG addressed the
>     issues).
> 
>   - The Authority has a well-defined limit on the amount of
>     time it spends making a decision.
> 
>   - The Working Group is responsible for soliciting external
>     reviews and addressing reviewer comments, all prior to the final
>     submission to the Authority.
> 
>   - IETF provides tools and mechanisms to efficiently
>     solicit external reviews and track issues, among other
>     things.
> 
> In short, a working group does all the work, while the Authority has a
> straightforward function of looking at vital checkboxes and issuing a
> verdict. The Authority has a veto, but it is mostly a procedural-based
> veto; virtually all technical work is done before the Authority gets
> the document.

> -----Original Message-----
> From: Keith Moore [mailto:[email-masked]]
> ....
> We also need for such reviews to be made in a timely fashion 
> - i.e. well
> before the design is frozen.
> 
> > Today, IESG/ADs essentially decide what external (including 
> their own)
> > comments are valid and enforce their application. It often takes a
> > long time for IESG/ADs to review, negotiate, and to issue a verdict.
> > This creates a bottleneck and causes both WGs and IESG/ADs to be
> > sloppy.
> 
> Looking deeper, often the "bottleneck" results, at least 
> partially, from
> IESG trying to fix serious problems with a protocol so late in the
> process. In some sense this could be viewed as trying to use the wrong
> tool for the job.  Right now, IESG's main tool for 
> encouraging sanity is
> withholding final approval of the document.  This works okay if only
> small changes are needed to the protocol or the text, but often IESG
> ends up trying to fix big problems with small changes.  Even worse,
> often the AD is expected to  specify those changes - so the AD ends up
> having to second-guess large chunks of the design.  It's no surprise
> that this takes a long time, but somehow I doubt that giving WGs more
> autonomy is the answer.  Insisting that WGs do development in stages,
> and that they solicit and respond to widespread review (including from
> IESG) at several of those stages, might help.  
> ......
> > 
> >   - The Authority must not review documents; instead,
> >     it must make sure submitted documents have received
> >     sufficient number of quality external reviews.
> 
> I strongly disagree.  We will still need a way to fix bad designs that
> manage to get through the process.  Our goal needs to be to 
> minimize the
> number of bad designs that get to the final approval stage 
> (i.e. to fix
> them sooner) - or to put it another way, to ensure that when documents
> get to the final approval stage there is already a high level of
> confidence in those documents both inside and outside the WG.
> 
> >   - The Authority must not spend endless cycles requesting
> >     additional reviews and clarifications; instead,
> >     it should reject any partial submission (i.e., submission
> >     without sufficient number of quality external reviews or
> >     without an easy-to-digest list of reviewer-identified
> >     issues, how they were addressed by the WG, and what the
> >     reviewer final reaction was after the WG addressed the
> >     issues).
> 
> I certainly think that IESG should summarily reject any document that 
> doesn't have the required documentation, and that there should be 
> clear (or at least pre-established) criteria for such summary 
> rejection.
> On the other hand, if IESG identifies a problem at a late 
> date, it still
> has a duty to investigate that problem via additional reviews or 
> clarifications if necessary.
> 
> >   - The Authority has a well-defined limit on the amount of
> >     time it spends making a decision.
> 
> This is pretty much the case already.  The problem is, people 
> want IESG
> to decide "yes" within that amount of time (or at worst, "yes if you
> make these minor changes that we specify completely"), and that's not 
> always realistic.  
> 
> > In short, a working group does all the work, while the 
> Authority has a
> > straightforward function of looking at vital checkboxes and 
> issuing a
> > verdict. The Authority has a veto, but it is mostly a 
> procedural-based
> > veto; virtually all technical work is done before the Authority gets
> > the document.
> 
> I'd certainly like to see things get to the point that "virtually all
> technical work is done before the Authority gets the 
> document" for final
> approval, but the IESG still must be able to reject or remand 
> documents
> on technical grounds, rather than merely procedural grounds.  
> 
> As a practical matter it's already very difficult for the IESG to push
> back significantly on a WG effort; an effort with additional support 
> in the form of reviews and responses would make such pushback even
> more difficult.  That's fine.  I also believe that if such reviews
> became commonplace, in the (hopefully rare) occassion that IESG did 
> see the need to push back, it would be easier to "fix" the 
> protocol than
> it often is now, because the protocol would have been more thoroughly
> sanity-checked and less likely to completely overlook some important
> consideration.
> 
>