Subject: [Icar] independence of reviews; variability

From: Dave Crocker <[email-masked]>
Date: Mon, 8 Mar 2004 16:43:30 -0800

Folks,

At the icar meeting, last week, Harald and I had an exchange about the
independence of review effort. I was being channeled through jabber, so
I'm relying on the jabber log for the details. I think the exchange
highlights some very basic issues, both about reviewing and about IETF
culture and organization. It would be extremely helpful to pursue this
further.

Here are the relevant parts of the jabber logs. I'm hoping I have
retained all the relevant text, which was interspersed with other
threads.  I've also included some of the log from Harald's presentation,
which touches on variability of reviews.

This message is just the relevant text.  I'll post comments separately.

d/

--- (edited) JABBER LOG ---

dcrocker says: I strongly suggest that the review function be
independent of the IETF administrative management team. The reviewing
stuff is not about helping an "area" or dealing with any aspect of
working group process. It is strictly for technical commentary.

hta: every time dave says something, I have to say the opposite. if you
want to create a body with power and responsibility outside the IETF
management structure with power and responsibility, be careful what you
ask for

dcrocker says: what "power" does a review group have and what "benefit"
is there in making such a purely technical effort subject to the varied
goals of ietf management?

hta: if review is to make any sense, it has to influence the quality,
relevance, timeliness of ourput. the managemt exists to influence the
quality, relevance, timeliness of ietf output

<hartmans> Why. Of course I don't see the value of independence in the
review other than to get people who will see clarity problems.

[JJJ]: Are you saying we can already ignore reviewers?

hta: If you have reviewers who make a difference, they have power

dcrocker says: working group / iesg conflict is a fact of life. an
independent review process provides _independent_ input to such
conflict.

Next: harald alvestrand on draft-ietf-alvestrand-twolevel-00.txt (if I
got this right)

... note document proposes giving review groups ability to approve
documents [did I get this right]; that is out of scope for here

[Dave Thaler]: how is it determined which team a document is reviewed by?

hta: throw dice, except that when a group of docs have strong internal
refrerences, they should be treated by the same review team

[Dave Thaler]: concern I have with this is that not all review teams will be
equal, not just in terms of quality, but in terms of what direction
one team may say "this is a good idea" while another wants to go another
direction who has ultimate responsibility? central management important
can't "shop around" for a team that is friendly to what you want to
do.  need some way to counter that

hta: if you're going to have more than one group, you need to expect and
deal with variations and judgement. need to have someplace in the system
that can take a review team and say "you're out of line" i believe in
central authoritty (we have no kings, but we have central authority)

dcrocker says: shopping around is fine, if the reviews must be credible
on their own right, rather than by virtue of some organizational
affiliation. If a wg gets only friendly reviews, they have not done
their job.

[Dave Thaler]: if the team is chosen randomly, and there are differences between
teams, then WG may not know how to construct a document that will
pass. f it is not random, there may be forum shopping\

hta: if we have multiple teams, we may have more guidelines

greg daily(?): when doc goes to ietf last call, there can be a dynamic
selection of reviewers

<dcrocker>: differences between sets of reviews - this is goodness,
not badness. the job of a wg is to resolve differences and
considerations. if reviews do not differ, they probably are not adding
much benefit.

--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

_______________________________________________
Icar mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/icar

---

From: Dave Crocker <[email-masked]>
Date: Mon, 8 Mar 2004 17:45:08 -0800

So, now some comments and questions:

DC> hta: if you want to create a body with power and responsibility
DC> outside the IETF management structure with power and responsibility,
DC> be careful what you ask for

The caution is warranted. When organizing power centers, be very
careful. And fully centralized power is particularly subject to
problems, particularly for groups with highly diverse, volunteer
participation, like the IETF.

In fact, the history of the IETF has tended towards particularly
DE-centralized activities. This is exactly what has changed over recent
years and could reasonably be interpreted as one of the sources of
current problems. Note, for example, that the effort to exert arbitrary,
centralized power was what triggered the 1992 Kobe situation.

The real work of the organization is initiated by, and performed by,
self-motivated volunteers. At base, this is a grass-roots organization.
Highly centralized control works against a grass-roots sense of
empowerment and responsibility. (Lest one think this means that there
does not need to be coordination and quality control, it doesn't.)

Reviews are supposed to be technical analyses, rather than sources of
design recommendations. The latter always get added, because anyone good
enough to be a reviewer will also have thoughts on how to improve the
thing being reviewed. But the core benefit of a review is the
independent perspective on how well or poorly the thing being reviewed
will work.

DC> hta: if review is to make any sense, it has to influence the quality,
DC> relevance, timeliness of ourput. the managemt exists to influence the
DC> quality, relevance, timeliness of ietf output

The fact that this activity is being translated into a question of power
challenges the basic credibility of these volunteer contributions, since
it suggests that the reviewers need to be controlled. This is at odds
with the IETF's history of collaboration rather than of leverage through
power. Such a requirement deprives reviews of the independence needed to
ensure the essential variability of perspective that a reviews are
supposed to have.

If we view the IESG as "being in charge" of the IETF and really being
the "authority" for IETF decisions, then perhaps is it is useful to give
the IESG control over qualification and selection of reviewers.

If, instead, we view the IETF as being a collaborative organization that
seeks to build broad-based support for its decisions -- with the IESG
coordinating and assessing that support -- then we need to be organized
and operated in a way that encourages the diversity and independence
that make claims of broad-based support credible.

d/

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 8 Mar 2004 21:44:08 -0600

FWIW, my understanding of the current "IETF Last Call" mechanics is
that reviewers today are, in the final analysis, "uncontrolled". A
scathing review correctly pointing out technical flaws is always
admissible at any point in the review cycle, all the way through
publication by the RFC Editor, no matter who wrote it or why.

If we don't think anyone except area directors reads documents any
more, discussion about "reviewers inside/outside the management
structure" might make sense. I had hoped that we would not be starting
down that road.

Having a Internet Review Board that is entirely independent of current
management structures works for me. Having an Internet Review Board
that "reports" to the IAB works for me. We may end up with Area Review
Teams, but that is *not* the only reasonable answer.

Spencer

From: "Dave Crocker" <[email-masked]>

---

From: Mark Allman <[email-masked]>
Date: Mon, 08 Mar 2004 23:04:23 -0500

> structure" might make sense. I had hoped that we would not be starting

I believe the spirit of the ICAR charter is that we should think more
along the lines of *early* review -- which my mind thinks of as "way
before the IESG sees a document".  I was unclear about whether Dave's
comments were about early review or late review or just plain review.
But, I am pretty sure the above is about later review.  Am I off on the
spirit of ICAR?  And, if we're scoped on early review does that help the
independence issue?

allman
(w/o co-chair hat)

--
Mark Allman -- ICIR -- http://www.icir.org/mallman/

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 09 Mar 2004 10:30:09 -0800

[[[NOTE: This note is ONLY about LATE review. Early review is a different 
kettle of fish.....]]]

Spencer:

there are two ways to view reviews, I think:

- As input to further work ("early review")
- As a check that this is good enough to be declared finished
(what I usually refer to as "late review")

In the first category, I think getting serious thought applied is more 
important than consistency - and structures need to reflect that.
In the second category, I think consistency is important - and in two 
directions, not one:

- The "floor" - that at least SOME review is carried out in ALL cases
- The "ceiling" - that product not be held to an unreasonably high 
standard, that stuff that is really nits is not declared show-stoppers, and 
that one show respect for well justified technical choices, even when one 
fundamentally disagrees with them.

(The last one is actually something most IESG members have spent time 
learning - not what issues to find, but what issues are NOT worth raising, 
because the time lost is more costly than the value add of the improvement)

Adding more structured late reviews is, in a sense, "raising the floor".
(In -twolevel, it is also intended to get away from the idea that a single 
small group has to do *all* the reviews for the IETF. That's nonscalable.)

As was raised in the ICAR meeting, if we have multiple groups reviewing 
documents, consistency (of both "floor" and "ceiling") isn't automatic, and 
we should make sure we have mechanisms where someone sees if we have 
consistency, and can take steps to increase consistency if the divergence 
becomes big enough to be harmful. So - someone has to manage it.

If management of consistency of late reviews is completely independent from 
other management structures, we set ourselves up for another "late 
surprise" problem - that the standards groups have been told to apply 
during development is not the standards they will be held to in the final 
review. So - I think that having *one* management structure for the IETF 
makes sense.

I don't particularly care what the shape of that management structure is, 
or what it is called - I care that it works.

                    Harald

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 9 Mar 2004 14:11:07 -0500

Although Dave and Harald are both using the term "review" in this 
thread, I think that the they may be talking about two very different 
things...  In my mind, I would call these things review and 
evaluation.

IMO, review is a purely informative activity.  The reviewer reads the 
document and provides information about its quality, clarity, 
feasibility, etc.  A review generally includes both positive and 
negative feedback.  The WG can then use that review to inform their 
decisions about the document -- whether to accept it as a WG work 
item, how to modify it over time, whether or not to submit it for 
publication.  The reviewer is not making a decision about the 
document, but is instead providing information to the WG that can be 
used to make a decision.  I agree with Dave that this activity should 
be independent of the IESG/IAB leadership and that this can best be 
accomplished as a widely-distributed grass roots function.

There is also another concept that I will call "evaluation".  In 
today's process, there are two evaluations done on every IETF 
document:  an AD evaluation and an IESG evaluation.  These 
evaluations involves reading the document and _deciding_ whether or 
not it is suitable for publication.  In general, the AD and the IESG 
do not provide reviews (as I used the term above) as part of this 
evaluation process.  If we decide that a document is not suitable for 
publication in its current form, we do tell the WG why, but we do not 
attempt (nor, IMO, should we) to provide a complete, balanced review 
of the document.

IMO, both review and evaluation are necessary parts of the document 
process, but they don't need to be done by the same people, nor are 
they done at the same time in the process.

There are a number of things that I believe that this WG could do to 
support, encourage and improve reviews within the IETF, including:

    - Defining a mechanism for how WG chairs/editors can get reviewers assigned
      to their documents (perhaps building from experience with MIB 
Doctors, SIRs, etc.)

    - Setting guidelines for the type and quantity of review that 
should be obtained
      at each stage in the WG process (before accepting as WG work item, before
      sending to WG last call, before submitting for publication, etc.).

    - Establish review criteria for each stage in the WG process, 
perhaps including
      some type of review forms.

    - Provide input to the EDU team, so that we can educate IETF 
participants on how
      to provide useful and productive reviews.

This group might also look at how we can make better use of reviews 
to inform and accelerate the evaluation process.

Margaret

---

From: Dave Crocker <[email-masked]>
Date: Tue, 9 Mar 2004 13:02:31 -0800

Margaret,

MW> This group might also look at how we can make better use of reviews 
MW> to inform and accelerate the evaluation process.

 Very helpful -- and accurate -- distinction and goal.

 Thanks.

d/

---

From: Dave Crocker <[email-masked]>
Date: Wed, 10 Mar 2004 13:16:12 -0800

Harald,

HTA> If management of consistency of late reviews is completely independent from
HTA> other management structures, we set ourselves up for another "late 
HTA> surprise" problem - that the standards groups have been told to apply 
HTA> during development is not the standards they will be held to in the final 
HTA> review. So - I think that having *one* management structure for the IETF 
HTA> makes sense.

An interesting question is whether this potential disparity is a feature
or a bug? The sort of rigid control you are promoting can be its own
trap, for failing to detect problems.

If a collection of independent reviewers holds an assessment that is
markedly different that of one or more ADs, it might indicate a problem
with some of the reviewers or with the ADs holding objections.

Once again, diversity of perspectives is our friend... that is, if we
believe in community rough consensus, rather than hierarchical assertion
of authority.

The sort of homogeneity that you are espousing could easily encourage
production of specifications that are nicely in line with some AD's
desires, but not the desires of the community that must adopt and use
the specification.

d/

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 10 Mar 2004 16:42:28 -0500

>believe in community rough consensus, rather than hierarchical assertion

This touches on a thought I've had off-and-on for the past couple of years...

The current IETF structures, policies and processes are based on some 
fundamental assumptions two of which are (1) that making decisions by 
rough consensus, reached through an open process will achieve good 
results, and (2) we can select leaders (ADs, WG chairs, etc.) and 
trust them to fairly solicit, judge and act upon rough community 
consensus.  Sometimes, though, we seem to question those assumption, 
and we end up adding structures, policies or processes that are 
intended to protect us from the results of our own consensus-driven 
decisions.  I think that is (and has been) a mistake.

While there is no decision making process (hierarchical, democratic, 
I just get to decide...) that achieves good results all of the time, 
the consensus-driven process has worked well for the IETF over the 
years.  By failing to trust it, we don't actually move to another 
effective decision making process, we just break the one that we have.

I think that this same line of thinking applies to review...  We 
should put into place the mechanisms, tools, training, etc. to 
improve the community's capacity to provide quality review.  Perhaps 
we should provide mechanisms that help WGs' find and recruit 
reviewers.  We could even develop some guidelines about what type and 
quantity of review makes sense at each level.  But, ultimately, I 
think that we should trust that our WGs (and our WG chairs, document 
editors, etc.) actually _want_ to  produce good quality, 
well-reviewed work.  We can improve their ability to do that by 
giving them better tools, but we won't achieve anything by trying to 
enforce quality through "better" rules.

If we decide, as a community, that we can no longer trust the 
consensus process and/or the motivations of our ADs, WG chairs or 
document editors, we have a _much_ bigger problem than can be solved 
by changing our review processes.

Margaret

---

From: David Meyer <[email-masked]>
Date: Wed, 10 Mar 2004 14:58:39 -0800

Margaret,

>> decisions.  I think that is (and has been) a mistake.

	I couldn't really parse the last sentence in this
	paragraph, i.e., what do you think is the mistake,
	questioning the assumptions or adding structures,
	policies, or processes? 
	
	That being said, honestly questioning one's assumptions
	with an eye toward improvement is never wrong. However,
	the latter (adding structure, etc) clearly can. So maybe
	its not really so much about the questioning (that much
	is healthy for our or any organization); maybe it is more
	about what we do in few those cases (if any) that we
	might find in which our common assumptions don't hold. In
	the case of the IETF, it is pretty clear that there are
	at most a few cases in which our consensus based approach
	is less efficient than we might like; it clearly works
	well in the vast majority of cases (nuff said on that
	one; the record speaks for itself).    

>> effective decision making process, we just break the one that we have.

	Maybe there is be something more (less?) granular here,
	like trusting the "Rough consensus and running code"
	isn't a binary thing? That is, it works extremely well
	for us most of the time, however, we do need to be
	vigilant and watch out for those few instances in which
	it doesn't. So maybe it is less about trust, and possibly
	more about trying to make sure those processes that have
	been so successful for us continue to serve us well. That
	goal would seem (minimally) to require continual
	reevaluation as we, our technology, and our industry
	evolve. At least that is how I see this issue.      
	
>> enforce quality through "better" rules.

	Completely agree. 

>> by changing our review processes.

	Again, completely agree.

	Dave

---

From: David Meyer <[email-masked]>
Date: Wed, 10 Mar 2004 15:01:36 -0800

>> 	the latter (adding structure, etc) clearly can. So maybe

s/clearly can./clearly can be.

Sorry about that.

Dave

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 10 Mar 2004 15:53:32 -0800

Dave,

> with some of the reviewers or with the ADs holding objections.

you heard it from me:

The right way to deal with ADs that are markedly at odds with IETF 
consensus is to correct them, reeducate them or fire them.

Not building baroque structures of multiple incompatible control functions 
in order to minimize the damage they can do.

Nit: I never used the word "rigid". That was not an accident.

                   Harald

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 10 Mar 2004 19:26:23 -0500

At 2:58 PM -0800 3/10/04, David Meyer wrote:
>  >>  Sometimes, though, we seem to question those assumption,

I meant that I believe it is (and has been) a mistake to add new 
structures, policies and procedures at the end of the process that 
are intended to protect us from the less-than-perfect results of the 
WG consensus process, because I don't believe that is the most 
effective way to produce good work.

It's not that I think we should let small, insular WGs run amok and 
then blindly publish everything that they produce.  However, I think 
that we would do better to offer community guidance and input during 
the development of new work (through review, for instance) than to 
try to block that work at the end.

There is really nothing new in what I'm saying...  Most modern 
engineering organizations have reached the point where they know that 
more bugs can be found and corrected, with less schedule impact, 
through early review (such as design and code review) than by quality 
checks at the end of the process (such as traditional 
post-development QA processes).

IMO, we do need some type of evaluation at the end (as a final check 
to detect and correct failures earlier in the process), but the 
current situation where a large percentage of the work of the IETF is 
blocked and modified after the WG believes it is finished, is 
pathological.  IMO, we can't fix this problem at the end (by just 
publishing the work, for example) we need to fix it at the beginning 
(in the WGs). An important way to try to fix this problem is to get 
more community input/review, including review and guidance from IETF 
leaderhship, earlier in the process.

Margaret

---

From: David Meyer <[email-masked]>
Date: Wed, 10 Mar 2004 16:30:54 -0800

>> leaderhship, earlier in the process.

	I think we're in pretty close agreement.

	Dave

---

From: Eric Rosen <[email-masked]>
Date: Thu, 11 Mar 2004 10:40:55 -0500

Harald> The right way to deal with ADs that are markedly at odds with IETF 
Harald> consensus is to correct them, reeducate them or fire them. 

This view is hopelessly utopian.  

There really does  need to be something in the process  which prevents an AD
with an  agenda from blocking  the output of  any WGs which don't  share his
agenda.  

One advantage  of earlier, wider review  is that it may  make it politically
more difficult for  an AD to carry out his own  political agenda.  The trick
is to  find reviewers  outside the WG  who are  experts in the  WG's subject
matter and  who don't have  agendas of  their own.  If  we can't come  to an
understanding of how to do this, then ICAR itself may be too utopian.

---

From: Scott W Brim <[email-masked]>
Date: Thu, 11 Mar 2004 11:37:42 -0500

> effective decision making process, we just break the one that we have.

This whole exercise was triggered because enough people felt that
frequency of mistakes in the process had reached a threshold and needed
to be dealt with.  As the Internet has become more important, the level
of mistakes that it tolerable has shrunk to close to zero.  Also the
level of experience of the participants has decreased.  I agree in
principle, that we want to keep our basic processes, and I trust the
basic process in principle, but we must make some changes to ensure
quality.  We want to add to it, e.g. a few review processes, preferably
in parallel with other work so they don't introduce delay.

> enforce quality through "better" rules.

I don't doubt that WGs want to produce good work.  It's not a question
of motivation, but of capability (WGs and ADs both, and toss in the IAB
members as well).  We need to bring more minds to bear on decisions --
that is, review them more -- because the stakes are higher and our
ability is not.  Conceptually we are not giving the WGs better tools, we
are giving the IETF better tools.

swb

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 11 Mar 2004 10:47:08 -0800

--On 11. mars 2004 10:40 -0500 Eric Rosen <[email-masked]> wrote:

>
> Harald> The right way to deal with ADs that are markedly at odds with
> IETF  Harald> consensus is to correct them, reeducate them or fire them.
>
> There really does  need to be something in the process  which prevents an
> AD with an  agenda from blocking  the output of  any WGs which don't
> share his agenda.

I don't think we disagree. But in the case of the current process, I see 
resolving a blockage as having 3 elements:

- Detecting that this is happening on a specific issue
- Resolving the specific issue
- If it happens more than a few times, do something about the AD

I've sometimes had mail to [email-masked] that worried about the first, and 
have sometimes been able to accomplish the second informally; so far, I 
haven't explicitly done the third.

> One advantage  of earlier, wider review  is that it may  make it
> politically more difficult for  an AD to carry out his own  political
> agenda.  The trick is to  find reviewers  outside the WG  who are
> experts in the  WG's subject matter and  who don't have  agendas of
> their own.  If  we can't come  to an understanding of how to do this,
> then ICAR itself may be too utopian.

I agree. While I believe less in the power of political agendas than you 
do, I believe that getting more reviewers into the process makes it FAR 
easier to see the difference between an AD (or any process gatekeeper) 
raising a concern that is widely shared in the community and raising a 
concern that has no backing.

It is one reason why I believe archiving reviews is an *essential* part of 
the process.

                             Harald

---

From: [email-masked]
Date: Fri, 12 Mar 2004 07:36:38 +0900

On 12 mar 2004, at 00.40, Eric Rosen wrote:

> politically
> more difficult for  an AD to carry out his own  political agenda.

I wonder.  My hypothesis is that early reviews, especially if they are 
tightly linked with the with the IESG and AD, will be more likely to 
point out how to come in line with the AD's agenda.

I think it is like the trend we see with standardized testing - all the 
education tends toward passing the test, not with knowledge per se.

a.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 11 Mar 2004 18:49:35 -0800

avri,

aao> I wonder.  My hypothesis is that early reviews, especially if they are 
aao> tightly linked with the with the IESG and AD, will be more likely to 
aao> point out how to come in line with the AD's agenda.

     I think cognitive dissonance theory says that a process like this
     causes everyone to move towards each other...

d/

---

From: Eric Rosen <[email-masked]>
Date: Fri, 12 Mar 2004 10:30:08 -0500

Avri> My hypothesis  is that early  reviews, especially if they  are tightly
Avri> linked with the with the IESG and AD, will be more likely to point out
Avri> how to come in line with the AD's agenda. 

Excellent point.  I  had assumed that the "earlier,  wider reviews" would be
harder for the ADs to control very closely (for reasons of scale, if nothing
else), but perhaps that assumption is wrong.  If the effect of early reviews
is help  special interest  groups have  more impact on  the WGs  rather than
less, we won't have made much progress.

This  all  comes  back to  the  underlying  issue  of whether  "the  general
interest"  is best served  by applying  a variety  of perspectives  (which I
think means  a variety of people  with independent perspectives,  and by the
marketplace), or whether it is best  served by having all the decisions made
by a small  group of know-it-alls.  This  is one of the issues  on which the
problem-statement group failed to agree.

---

From: Dave Crocker <[email-masked]>
Date: Fri, 12 Mar 2004 08:12:56 -0800

Eric,

Avri>> My hypothesis  is that early  reviews, especially if they  are tightly
Avri>> linked with the with the IESG and AD, will be more likely to point out
Avri>> how to come in line with the AD's agenda. 
ER> Excellent point.  I  had assumed that the "earlier,  wider reviews" would be
ER> harder for the ADs to control very closely (for reasons of scale, if nothing
ER> else), but perhaps that assumption is wrong.

Early reviews wind up requiring _everyone_ to put their expectations and
concerns out on the table sooner.

Unless I missed something over the last 15 years, there is nothing that
gives ADs a unique insight into the creation of perfect protocols. They
are typically senior contributors, and that usually does give them
excellent insight, but not unique.  (For these discussions, I cannot
stress strongly enough just how important that distinction is.)

So, the idea that the rest of us all must always march to the desires of
particular individuals -- no matter their title -- goes exactly against
the grain of rough consensus.

By getting everyone's expectations and concerns out on the table early,
disparities can be reasonably negotiated openly and carefully.

ER>   If the effect of early reviews
ER> is help  special interest  groups have  more impact on  the WGs  rather than
ER> less, we won't have made much progress.

One of the areas in which the IETF has demonstrated a consistent lack of
skill is political/psychological gaming. We do best when we do what is
straightforward and constructive. Special Interests that involve
careful, intelligent folk, will almost always be better at the gaming
than the the IETF community. We hold sway over them by doing Good Work.

So let's not be too distracted by such concerns.  Early reviews are
pretty obviously a good thing, for all the reasons folks have been
saying.

ER> This  all  comes  back to  the  underlying  issue  of whether  "the  general
ER> interest"  is best served  by applying  a variety  of perspectives  (which I
ER> think means  a variety of people  with independent perspectives,  and by the
ER> marketplace), or whether it is best  served by having all the decisions made
ER> by a small  group of know-it-alls.  This  is one of the issues  on which the
ER> problem-statement group failed to agree.

The essence of rough consensus is the belief that paying attention to a
variety of perspectives ensures that there will, later, be both the
ability to use the output to satisfy real-world needs, and the general
support for pursuing it.

The problem with giving an individual or small group excessive control
is the likelihood that they will force an outcome that is not, later,
embraced by the folks who are needed to make the work successful (ie,
useful.)

d/

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 12 Mar 2004 08:21:40 -0800

--On 12. mars 2004 10:30 -0500 Eric Rosen <[email-masked]> wrote:

>
> Avri> My hypothesis  is that early  reviews, especially if they  are
> tightly Avri> linked with the with the IESG and AD, will be more likely
> to point out Avri> how to come in line with the AD's agenda.
>
> Excellent point.  I  had assumed that the "earlier,  wider reviews" would
> be harder for the ADs to control very closely (for reasons of scale, if
> nothing else), but perhaps that assumption is wrong.  If the effect of
> early reviews is help  special interest  groups have  more impact on  the
> WGs  rather than less, we won't have made much progress.

if early review works, it will certainly give more influence to the people 
who do the reviews.........

> This  all  comes  back to  the  underlying  issue  of whether  "the
> general interest"  is best served  by applying  a variety  of
> perspectives  (which I think means  a variety of people  with independent
> perspectives,  and by the marketplace), or whether it is best  served by
> having all the decisions made by a small  group of know-it-alls.  This
> is one of the issues  on which the problem-statement group failed to
> agree.

I don't know if there is a real conflict of theories here, or one of 
perspectives - in these discussions, I find that people seem to argue that 
the process they don't like (whether it's the IETF, the IESG or the WG) is 
a "small group of know-it-alls", while the process they like (whether it's 
the multiperspective review of the IESG or the review from a wider 
audience) is "a variety of people with independent perspectives".

Wide review seems to be "motherhood and apple pie" - the question we're 
debating is, I think, how to achieve it.

                    Harald