Subject: Reforming the BOF Process (was Declining the ifare bof for Chicago)

From: Bernard Aboba <[email-masked]>
Date: Tue, 12 Jun 2007 09:52:05 -0700

The recent discussion on the IFARE BOF has raised more fundamental issuesabout the IETF BOF process.  Rather than letting discussion continue on theSAAG list, it would seem better for this discussion to occur on the IETF list. > Speaking as a former AD, it can be a very tough call to say yes/no to> a BOF. Unfortunately, there is often interest, but interest is most  > definitely not enough. There needs to be more than interest. It should be understood that this is a feature of the IETF process that isnot necessarily held in common with other SDOs.For example, within IEEE 802 the initial meeting is termed a "Callfor Interest" because the determination of interest is the major focus;writing a charter/PAR is not.                                                                                   Assuming that sufficient interest exists, a study group is formed, whosesole purpose is to write a Project Authorization Request (PAR)(equivalent of a charter), and demonstrate that the proposed worksatisfies the "5 criteria":1. Broad Market Potential  a. Broad sets of applicability.  b. Multiple vendors and numerous users.  c. Balanced costs2. Compatibility with existing standards.3. Distinct Identity.4. Technical feasibility  a. Demonstrated system feasibility  b. Proven technology, reasonable testing  c. Confidence in reliability5. Economic Feasibility> There needs to be a reasonable chance of a positive, forward-moving> outcome.I believe that this ascribes more predictive value to the BOF process thanis warranted by experience.  Quite a few deployed technologies  haveoriginated from BOFs that the IESG judged to not have a likely "forward-movingoutcome", while many unproductive working groups arose from successful    BOFs.   The reality is that BOFs do not much have predictive value, if onlybecause the BOF process does not much resemble the WG process, so thatthe success dynamics cannot easily be ascertained as a result. > Yes, I* opinions are afforded special status. They are our chosen  > leadership, and with leadership comes responsibility. Responsibility> to be sure that if the work goes forward, it is well scoped, has a> reasonable likelihood of success, etc. And please remember, the IETF> is a meritocracy. So please don't raise the "I* has special status"> issue as if it were some kind of unfair or biased way of doing things.Again, the IESG role in the BOF process represents a choice on the part of the IETF.  It is possible to envisage other approaches that could yieldoutcomes as good or better while providing better accountability andtransparency. For example, by restricting the function of an initial BOF to a determination ofinterest and a decision to form/not form a study group,  the opportunitiesfor unfairness and bias can be reduced.  Once the study grouphad produced a charter and documentation of the formation criteria, the reviewof these documents could proceed with more information than is typicallyavailable as the result of a (potentially delayed) 2nd BOF.   Also, the review could utilize existing procedures for ensuring transparency andaccountability, such as an open review process and documentation ofDISCUSS comments. 

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Fri, 15 Jun 2007 12:22:01 +0200

Bernard,
 
Speaking as a participant in both the IETF and IEEE 802, there are many
things that I like in the CFI / Study Group process of IEEE. Your
proposal goes in the direction of solving one of the problems I perceive
in the IETF processes which is the lack of repeatability and
predictability (again speaking as a participant). I like it. Yet, there
are some differences:
 
- The five criteria in the IEEE would not apply as is. I am not sure
that 'broad market potential' should be there at all, or should be as
strong a factor as it is in the IEEE. Same with economic feasibility,
which in the IEEE often refers to the costs of hardware based
implementations
- 'Measuring interest' works differently in the IETF than in the IEEE
which is very much physical participation based, and where participants
and company votes are dully counted and registered in CFI meetings as
proof of interest.  
 
Dan

________________________________

	From: Bernard Aboba [mailto:[email-masked]] 
	Sent: Tuesday, June 12, 2007 7:52 PM
	To: [email-masked]
	Subject: Reforming the BOF Process (was Declining the ifare bof
for Chicago)

	The recent discussion on the IFARE BOF has raised more
fundamental issues
	about the IETF BOF process.  Rather than letting discussion
continue on the
	SAAG list, it would seem better for this discussion to occur on
the IETF list. 
	
	> Speaking as a former AD, it can be a very tough call to say
yes/no to
	> a BOF. Unfortunately, there is often interest, but interest is
most  
	> definitely not enough. There needs to be more than interest. 
	
	It should be understood that this is a feature of the IETF
process that is
	not necessarily held in common with other SDOs.
	
	For example, within IEEE 802 the initial meeting is termed a
"Call
	for Interest" because the determination of interest is the major
focus;
	writing a charter/PAR is not.

	Assuming that sufficient interest exists, a study group is
formed, whose
	sole purpose is to write a Project Authorization Request (PAR)
	(equivalent of a charter), and demonstrate that the proposed
work
	satisfies the "5 criteria":
	
	1. Broad Market Potential
	  a. Broad sets of applicability.
	  b. Multiple vendors and numerous users.
	  c. Balanced costs
	2. Compatibility with existing standards.
	3. Distinct Identity.
	4. Technical feasibility
	  a. Demonstrated system feasibility
	  b. Proven technology, reasonable testing
	  c. Confidence in reliability
	5. Economic Feasibility
	
	> There needs to be a reasonable chance of a positive,
forward-moving
	> outcome.
	
	I believe that this ascribes more predictive value to the BOF
process than
	is warranted by experience.  Quite a few deployed technologies
have
	originated from BOFs that the IESG judged to not have a likely
"forward-moving
	outcome", while many unproductive working groups arose from
successful    
	BOFs.   The reality is that BOFs do not much have predictive
value, if only
	because the BOF process does not much resemble the WG process,
so that
	the success dynamics cannot easily be ascertained as a result. 
	
	> Yes, I* opinions are afforded special status. They are our
chosen  
	> leadership, and with leadership comes responsibility.
Responsibility
	> to be sure that if the work goes forward, it is well scoped,
has a
	> reasonable likelihood of success, etc. And please remember,
the IETF
	> is a meritocracy. So please don't raise the "I* has special
status"
	> issue as if it were some kind of unfair or biased way of doing
things.
	
	Again, the IESG role in the BOF process represents a choice on
the part 
	of the IETF.  It is possible to envisage other approaches that
could yield
	outcomes as good or better while providing better accountability
and
	transparency. 
	
	For example, by restricting the function of an initial BOF to a
determination of
	interest and a decision to form/not form a study group,  the
opportunities
	for unfairness and bias can be reduced.  Once the study group
	had produced a charter and documentation of the formation
criteria, the review
	of these documents could proceed with more information than is
typically
	available as the result of a (potentially delayed) 2nd BOF.
Also, the 
	review could utilize existing procedures for ensuring
transparency and
	accountability, such as an open review process and documentation
of
	DISCUSS comments.

---

From: Jari Arkko <[email-masked]>
Date: Fri, 15 Jun 2007 13:53:31 +0300

Bernard,

I think your proposal is worth thinking about. The current BOF process
is very on/off in its nature. One of the problems that it is causing is that
when work is not far enough, a BOF or WG cannot be established. This
in turns leave the perception that the IETF is completely ignoring the
topic. In reality, a denied WG/BOF might mean anything ranging from
"go away with your stupid idea" to "this is very important and interesting,
but please do <X> first so that the WG can be chartered or BOF held".
We try to give the right perception, of course, but sometimes its hard to
convince people who can only observe the existence/non-existence
of an official activity.

Jari

Romascanu, Dan (Dan) kirjoitti:
> */Bernard,/*
> *//* 
> */Speaking as a participant in both the IETF and IEEE 802, there are
> many things that I like in the CFI / Study Group process of IEEE. Your
> proposal goes in the direction of solving one of the
> problems I perceive in the IETF processes which is the lack of
> repeatability and predictability (again speaking as a participant). I
> like it. Yet, there are some differences:/*
> *//* 
> */- The five criteria in the IEEE would not apply as is. I am not sure
> implementations/*
> */- 'Measuring interest' works differently in the IETF than in the
> IEEE which is very much physical participation based, and where
> participants and company votes are dully counted and registered in CFI
> meetings as proof of interest.  /*
> *//* 
> */Dan/*
>  
> *//* 
>
>     ------------------------------------------------------------------------
>     *From:* Bernard Aboba [mailto:[email-masked]]
>     *Sent:* Tuesday, June 12, 2007 7:52 PM
>     *To:* [email-masked]
>     *Subject:* Reforming the BOF Process (was Declining the ifare bof
>     for Chicago)
>
>     The recent discussion on the IFARE BOF has raised more fundamental
>     issues
>     It should be understood that this is a feature of the IETF process
>     that is
>     For example, within IEEE 802 the initial meeting is termed a "Call
>     Assuming that sufficient interest exists, a study group is formed,
>     whose
>     (equivalent of a charter), and demonstrate that the proposed work
>     > There needs to be a reasonable chance of a positive, forward-moving
>     > outcome.
>
>     is warranted by experience.  Quite a few deployed technologies  have
>     because the BOF process does not much resemble the WG process, so that
>     > Yes, I* opinions are afforded special status. They are our chosen 
>     > leadership, and with leadership comes responsibility. Responsibility
>     > to be sure that if the work goes forward, it is well scoped, has a
>     > reasonable likelihood of success, etc. And please remember, the IETF
>     > is a meritocracy. So please don't raise the "I* has special status"
>     Again, the IESG role in the BOF process represents a choice on the
>     part
>     outcomes as good or better while providing better accountability and
>     transparency.
>
>     review could utilize existing procedures for ensuring transparency and
>     accountability, such as an open review process and documentation of
>     DISCUSS comments.
>
> ------------------------------------------------------------------------
>

---

From: John C Klensin <[email-masked]>
Date: Sun, 17 Jun 2007 14:13:20 -0400

--On Tuesday, 12 June, 2007 09:52 -0700 Bernard Aboba
<[email-masked]> wrote:

>...
> For example, by restricting the function of an initial BOF to
> a determination of interest and a decision to form/not form a
> study group,  the opportunities for unfairness and bias can be
> reduced.  Once the study group had produced a charter and
> documentation of the formation criteria, the review of these
> documents could proceed with more information than is
> typically available as the result of a (potentially delayed)
> 2nd BOF. Also, the review could utilize existing procedures
> for ensuring transparency and accountability, such as an open
> review process and documentation of DISCUSS comments.

Bernard,

Some specific observations on shifting more toward an IEEE
model...

(1) Unless things have changed since I was last paying careful
attention, the process you describe is used to adding a new
project to an existing technical committee.  The process for
creating a new technical committee is somewhat more elaborate.
The IETF has no layer between the steering group (IESG, TSC (?))
and the WGs who actually do the technical work.  We also usually
try to charter WGs on a short-term, project basis rather than
assigning new projects to existing WGs.  Because of those
differences, we need to be careful about analogies unless we are
willing to rethink process models in much more fundamental ways
than tweaking the BOF process.

(2) I don't have statistics, but my impression is that most
technical standards work in the IEEE these days (not just in
802, but more broadly) starts with a proposal to standardize a
specific industry technology.   Those proposals are debated and
refined, but the assumption is that little fundamental
engineering or design work is done in the standardization
process.  We behave as if the IETF is still doing engineering
design work.  Maybe it is time to drop that as an inefficient
and ineffective fantasy, but, again, considering that involves
much broader issues than reforming the BOF process.

(3) Many of us are concerned about the length of time it takes
to move a well-thought-out idea that has significant support
forward from initial proposal to a functioning standards
development effort.  Perhaps we should be concentrating as much
on that question as on the one of how we cut bad, or
inadequately supported, ideas off as cleanly as possible.
Adding a study group creation process and a study group process
to the existing opportunities for delay would not contribute to
speeding things up.  Indeed, it would do much the contrary.

(4) I, and others, have said this before, but my belief is that
the single most effective change that could be made to the BOF
process would be for the IESG to stop taking its ability to
project the future so seriously.  As you and others have
commented, the track record on that isn't good anyway.  Suppose,
instead, we were to permit WGs to be set up on a provisional
basis, with intense review after some reasonable period of time
and cancellation if they were not performing adequately and
producing results the community seemed to care about.  This
would combine some of the advantages of IEEE-like study groups
with a streamlined process that focused on the one true measure
of whether a WG could produce useful work: starting it up and
seeing if it produced useful work.  

Since decisions as to whether to charter a WG are somewhat
subjective and the IESG has broad discretion, this is a change
the IESG could institute --experimentally or permanently-- on
its own initiative without our spending energy on changing the
processes.   Its success would, however, depend on a change of
attitude in the community: today, so much effort goes into the
charter process because it has become almost impossible, in
practice, to kill off a non-performing WG or to put a tight
leach on one that is wandering in the weeds.   ADs who try to do
so do not win popularity contests, to put it mildly.   If the
IESG saw clear and broad community support for chartering WGs
that were questionable but plausible and then tuning charters or
killing non-performing WGs if things didn't work out as
originally conceived, I believe we could get a great deal of the
nonsense, arbitrariness, and delays out of the early parts of
the process.  

But I don't think that support is there, at least yet.  Without
it, changes in forms or procedures probably do not produce
better results and, if delay is considered an important cost,
might produce worse ones.

regards,
    john

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 18 Jun 2007 08:34:50 +0200

See in-line.

Dan

> -----Original Message-----
> From: John C Klensin [mailto:[email-masked]] 
> Sent: Sunday, June 17, 2007 9:13 PM
> To: Bernard Aboba; [email-masked]
> Subject: Re: Reforming the BOF Process (was Declining the 
> ifare bof for Chicago)
> 
> 
> 
> >determination of interest and a decision to form/not form a  study 
> >group,  the opportunities for unfairness and bias can be  reduced.  
> >Once the study group had produced a charter and  
> documentation of the 
> >formation criteria, the review of these  documents could 
> proceed with 
> >more information than is  typically available as the result of a 
> >(potentially delayed)  2nd BOF. Also, the review could 
> utilize existing 
> >procedures  for ensuring transparency and accountability, such as an 
> >open  review process and documentation of DISCUSS comments.
> 
> Bernard,
> 
> Some specific observations on shifting more toward an IEEE model...
> 
> (1) Unless things have changed since I was last paying 
> careful attention, the process you describe is used to adding 
> a new project to an existing technical committee.  The 
> process for creating a new technical committee is somewhat 
> more elaborate.
> The IETF has no layer between the steering group (IESG, TSC 
> (?)) and the WGs who actually do the technical work.  We also 
> usually try to charter WGs on a short-term, project basis 
> rather than assigning new projects to existing WGs.  Because 
> of those differences, we need to be careful about analogies 
> unless we are willing to rethink process models in much more 
> fundamental ways than tweaking the BOF process.

Actually I believe that the IEEE have study groups both at the level of
technical committees (Working Groups) which end if successful with new
projects within existing Working Groups as well as at the level of the
whole IEEE which may end with new Working Groups being formed. Now a
IEEE Working Group vary very much in size and scope, but there are a few
like IEEE 802.1 (bridging, security QoS) and IEEE 802.11 (wireless LAN)
which are comparable in size, scope and number of projects with an IETF
area rather than with an IETF WG. 

> process. 

This also differs from case to case. In some IEEE projects work may
start with a given technology on the table, in some other there may be
more than one, or just an idea or several ideas about how to solve the
problem, and much debate happens until an initial proposal is baselined.
I do not believe that overall the distribution of cases is significantly
different to what happens in the IETF. 

> We behave as if the IETF is still doing engineering 
> and ineffective fantasy, but, again, considering that 
> involves much broader issues than reforming the BOF process.
> 
> (3) Many of us are concerned about the length of time it 
> takes to move a well-thought-out idea that has significant 
> support forward from initial proposal to a functioning 
> standards development effort.  Perhaps we should be 
> concentrating as much on that question as on the one of how 
> we cut bad, or inadequately supported, ideas off as cleanly 
> as possible.

I believe that the lack of some agreed criteria of approval for new work
is a problem. Maybe this is what you mean by 'how we cut bad'. 

> Adding a study group creation process and a study group 
> process to the existing opportunities for delay would not 
> contribute to speeding things up.  Indeed, it would do much 
> the contrary.
> 
> (4) I, and others, have said this before, but my belief is 
> that the single most effective change that could be made to 
> the BOF process would be for the IESG to stop taking its 
> ability to project the future so seriously.  As you and 
> others have commented, the track record on that isn't good 
> anyway.  Suppose, instead, we were to permit WGs to be set up 
> on a provisional basis, with intense review after some 
> reasonable period of time and cancellation if they were not 
> performing adequately and producing results the community 
> seemed to care about.  This would combine some of the 
> advantages of IEEE-like study groups with a streamlined 
> process that focused on the one true measure of whether a WG 
> could produce useful work: starting it up and seeing if it 
> produced useful work.  

What would be the differences between 'study groups' and 'WGs set up on
provisional basis'? 

> subjective and the IESG has broad discretion, this is a 
> change the IESG could institute --experimentally or 
> permanently-- on its own initiative without our spending 
> energy on changing the
> attitude in the community: today, so much effort goes into 
> the charter process because it has become almost impossible, 
> in practice, to kill off a non-performing WG or to put a tight
> that were questionable but plausible and then tuning charters 
> or killing non-performing WGs if things didn't work out as 
> originally conceived, I believe we could get a great deal of 
> the nonsense, arbitrariness, and delays out of the early 
> parts of the process.  
> 
> But I don't think that support is there, at least yet.  
> Without it, changes in forms or procedures probably do not 
> produce better results and, if delay is considered an 
> important cost, might produce worse ones.
> 
> regards,
>     john
> 
> 
> 
> 
>

---

From: Lakshminath Dondeti <[email-masked]>
Date: Mon, 18 Jun 2007 00:10:22 -0700

One of the things I have been doing and will continue to be doing is to 
bring work to the IETF that may be needed by other standards 
organizations and in some cases is needed by other organizations.  I 
have done this or tried to do this in the AD sponsored route as an 
author and/or document shepherd as well as through the BoF process.  I 
have had some successes and some failures.  Here is a meta-issue from 
that experience.

If a piece of work is proposed early, it may be dismissed in the 
category of "who needs this?"  It is also difficult for proponents to 
communicate constraints specific to other organizations.  Official 
communication channels are not possible since there may not be a work item.

If it is brought just in time (the interest may be communicated via an 
official LS), the time constraints involved are hard on everyone.

The consequence in either case is that people choose bad (in clear 
violation of an IETF BCP or RFC) alternatives in the other organizations 
with the simple reason of nothing better is available (from the IETF).

The Study Group concept that Bernard proposes (with Dan's 
qualifications; I am sure we will IETFy it before it is all said and 
done) might be helpful in addressing the first category, i.e., bringing 
work early.  As John notes it may introduce additional delay and that 
concerns me.  Perhaps we should have the same evaluation process as now 
and the result of the process, if the proponents are also up for it 
could be a Study Group instead of "go away."

thanks,
Lakshminath