Subject: When is a 3933 experiment necessary? [Was: Last Call: <draft-farrell-ft-03.txt> (A Fast-Track way to RFC with Running Code) to Experimental RFC]

From: "Adrian Farrel" <[email-masked]>
Date: Wed, 30 Jan 2013 09:15:19 -0000

Well, is that a meta-judgment call?

I took the view that the full process expressed in draft-farrell-ft could not be
done by the IESG at their discretion. That is, that some of the steps proposed
constituted a significant variation from documented processes or
well-established behavior. Thus, it was my opinion that if such changes were to
be made they would either need to be documented direct in a BCP or run as a 3933
experiment. It was also my opinion that consensus for a BCP would be improbable
(looks like discussion on this list has upheld that view), and so *if* this was
to go ahead it would need to be as a 3933 experiment.

Turns out, however, that the community is not willing to try this experiment as
currently documented. That's OK.

I do not take quite the same negative view as Stephen, but I do agree with
Spencer that getting consensus for a process change always looks like a
formidable task. Small changes never address enough of the problem or the right
piece of the problem. Large changes are too much in one go. :-) So, it seems to
be increasingly hard to make changes to our process.  

While IESG statements remain an option, I would not like to see them regularly
used for things that are definitively a change to consensus-based process and
especially for things that change the way the community is supposed to behave.
On the other hand, IESG statements are fine for declaring how the IESG will
behave, for advising on action within existing parameters, and for emergency
action.

So, my conclusion: it would be good to have more process experiments if people
feel the process needs to change. However, it would appear that such experiments
need:
- Thorough debate on an appropriate mailing list first
   Do we need a "process experiments" mailing list?
- Very clear expression of the purpose of the experiment
   and how it will be evaluated
- Very tight scoping to a portion of the IETF or of the IETF's
   work so that the risk of the experiment simply changing
   the IETF's process by default is removed

Cheers,
Adrian

> -----Original Message-----
> From: Spencer Dawkins [mailto:[email-masked]]
> Sent: 29 January 2013 21:35
> To: John C Klensin
> Cc: Thomas Narten; [email-masked]; [email-masked]
> Subject: Re: FW: Last Call: <draft-farrell-ft-03.txt> (A Fast-Track way to RFC
with
> Running Code) to Experimental RFC
> 
> Just as a follow-up here ...
> 
> I was John's co-author on RFC 3933 (http://tools.ietf.org/html/rfc3933).
> When we were working on the draft, the problem I thought we were
> solving, was that the IESG needs to update the IETF's BCP processes from
> time to time, but (1) it was like 32 simultaneous root canals to
> actually get community consensus to modify the IETF's process BCPs
> including untried changes that might be improvements, so (2) the IESG
> was using informal IESG statements developed with much less community
> involvement than was expected for process BCP changes, in order to get
> anything done at all.
> 
> I (and, I think, John as well) saw RFC 3933 process experiments as a
> middle path between lightweight IESG decisions and full process BCP
> revisions. That's what I thought Section 2 of RFC 3933 was trying to say.
> 
> I had assumed that if the IESG clearly had discretion to do something
> under the current process BCPs, and thought it was the right thing to
> do, they would just do it, rather than set up a process experiment.
> 
> For what that's worth.
> 
> Spencer
> 
> On 1/24/2013 12:34 PM, John C Klensin wrote:
> > --On Tuesday, January 22, 2013 16:31 -0500 Thomas Narten
> > <[email-masked]> wrote:
> 
> >> This document seems to have a bit of missing the forest for the
> >> trees. In the overall scheme of things, I don't believe the
> >> draft will materially help, and is at best a distraction from
> >> dealing with meaningful problems.
> >>
> >> The crux of the issue is that any attempt at "fast tracking" is
> >> fundamentally about short-circuiting some aspect of our review
> >> processes. We should only do that very carefully. In almost
> >> all cases, individual judgement is needed as to when it is
> >> appropriate to do that. ADs/WG chairs already have some
> >> flexibility here. e.g., a WG can skip WG LC if they think its
> >> ...
> >
> > Hi.
> > First of all, I am completely in agreement with Thomas and his
> > analysis.  Anything  that can reasonably and appropriately be
> > done by this sort of effort can be done by discretion without
> > adoption of a formal procedure and adoption of a formal
> > procedure creates additional and unnecessary risks.
> >
> > As co-author of the process experiment model, I also object to
> > its use when it is not demonstrably necessary, for example to
> > give extra status and force IETF-wide use where the actions
> > suggested can be carried out as a matter of normal discretion
> > (see Section 5 of the document).

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 30 Jan 2013 12:15:23 +0000

I believe that Adrian did right in this case. This was IMO one of the situations which in Spencer's language was 'middle path between lightweight IESG decisions and full process BCP revisions' and a 3933 experiment could have proved it right or wrong, useful or not. The community could not reach consensus to run the experiment and this is one possible outcomes when such questions are asked. 

Same as Adrian I do not believe that this lack of consensus invalidates process experiments. What I feel is that the key condition in the list drawn by Adrian is the second: 

> > - Very clear expression of the purpose of the experiment and how it will be evaluated

Almost in all cases when this condition was not met I noticed the discussions ran wild and became unfocussed, discussing not about the proposal to run the experiment, but about that subject of the experiment and its presumed results before it had even been started. 

Dan

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Adrian Farrel
> Sent: Wednesday, January 30, 2013 11:15 AM
> To: 'Spencer Dawkins'; 'John C Klensin'
> Cc: [email-masked]
> Subject: When is a 3933 experiment necessary? [Was: Last Call: <draft-
> farrell-ft-03.txt> (A Fast-Track way to RFC with Running Code) to
> Experimental RFC]
> 
> I took the view that the full process expressed in draft-farrell-ft
> could not be done by the IESG at their discretion. That is, that some of
> the steps proposed constituted a significant variation from documented
> processes or well-established behavior. Thus, it was my opinion that if
> such changes were to be made they would either need to be documented
> direct in a BCP or run as a 3933 experiment. It was also my opinion that
> consensus for a BCP would be improbable (looks like discussion on this
> list has upheld that view), and so *if* this was to go ahead it would
> need to be as a 3933 experiment.
> 
> Turns out, however, that the community is not willing to try this
> experiment as currently documented. That's OK.
> 
> I do not take quite the same negative view as Stephen, but I do agree
> with Spencer that getting consensus for a process change always looks
> like a formidable task. Small changes never address enough of the
> problem or the right piece of the problem. Large changes are too much in
> one go. :-) So, it seems to be increasingly hard to make changes to our
> process.
> 
> While IESG statements remain an option, I would not like to see them
> regularly used for things that are definitively a change to consensus-
> based process and especially for things that change the way the
> community is supposed to behave.
> On the other hand, IESG statements are fine for declaring how the IESG
> will behave, for advising on action within existing parameters, and for
> emergency action.
> 
> So, my conclusion: it would be good to have more process experiments if
> people feel the process needs to change. However, it would appear that
> such experiments
> need:
> > solving, was that the IESG needs to update the IETF's BCP processes
> > from time to time, but (1) it was like 32 simultaneous root canals to
> > >> trees. In the overall scheme of things, I don't believe the draft
> > >> will materially help, and is at best a distraction from dealing
> > >> with meaningful problems.
> > >>
> > >> processes. We should only do that very carefully. In almost all
> > >> cases, individual judgement is needed as to when it is appropriate
> > >> to do that. ADs/WG chairs already have some flexibility here. e.g.,
> > >> a WG can skip WG LC if they think its ...
> > >
> > > Hi.
> > > analysis.  Anything  that can reasonably and appropriately be done
> > > by this sort of effort can be done by discretion without adoption of
> > > a formal procedure and adoption of a formal procedure creates
> > > additional and unnecessary risks.
> > >
> > > As co-author of the process experiment model, I also object to its
> > > use when it is not demonstrably necessary, for example to give extra
> > > status and force IETF-wide use where the actions suggested can be
> > > carried out as a matter of normal discretion (see Section 5 of the
> > > document).

---

From: SM <[email-masked]>
Date: Wed, 30 Jan 2013 10:15:50 -0800

Hi Adrian,
>be increasingly hard to make changes to our process.

It is an unsurmountable task to change anything relating to 
process.  In a sense the process is built for incremental changes 
instead of drastic changes.

>behave, for advising on action within existing parameters, and for emergency

It is easier to wait for an emergency and take action.

>    the IETF's process by default is removed

It is good to see "process experiments" coming from IESG members.  I 
read it as meaning that those members don't expect the same old 
tricks to work forever.  "Process experiments" won't get me far 
unless I put in the effort to do the work.  I will probably sound 
cynical; it's unlikely that there will be thorough debate on an 
appropriate mailing list.  I am not against the idea of setting up a 
mailing list though.

Please do not read any of the above as implying that it won't 
work.  I cannot predict the future.  You are better placed than me to 
make the decision as you know more than me.

Regards,
-sm

---

From: Dave Crocker <[email-masked]>
Date: Wed, 30 Jan 2013 10:35:54 -0800

On 1/30/2013 1:15 AM, Adrian Farrel wrote:
>     I do agree with

I suspect it's not 'increasingly' but rather that it's always been 
extremely difficult...

Let me suggest a different possibility for the challenge in this topic:

      We are a diverse community.  Absent very, very strong consensus 
that a problem is serious enough to warrant a change, the community is 
not likely to line up automatically behind a proposal.  We will always 
have some people who prefer no change and some who offer their 
different, favorite approaches, or and some who offer a zillion tweaks. 
  In the aggregate, that makes for entropy, not consensus.

      What makes this process different from what we see in successful 
working groups?

I think there are two things:

    1. A wg has a committed core of participants who have agreed on a 
common goal.

    2. A wg process is managed.

On the average, proposals for IETF process change benefit from neither 
of these.

Hence I suggest that a proposal needs to recruit a committed core 
/before/ going public, and the discussion needs classic group 
facilitation, in terms of tracking issues, maintaining focus, and 
pursuing consensus.

d/

-- 
  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net

---

From: Barry Leiba <[email-masked]>
Date: Thu, 31 Jan 2013 19:25:51 -0500

>      We are a diverse community.  Absent very, very strong consensus that a
> problem is serious enough to warrant a change, the community is not likely
> to line up automatically behind a proposal.  We will always have some people
> who prefer no change and some who offer their different, favorite
> approaches, or and some who offer a zillion tweaks.  In the aggregate, that
> makes for entropy, not consensus.
...
>    1. A wg has a committed core of participants who have agreed on a common
> goal.
> On the average, proposals for IETF process change benefit from neither of
> these.

These are good observations; I agree with them, as well as with your
suggested approach to it.

I want to make a related point (which Dave's recommendation of classic
group facilitation can cut through if we allow it to):

We often pick on every suggested change and point out every possible
flaw, with different people holding out behind different flaws, and we
get stuck there.  There seems to be some assumption, when we do this,
that our current process doesn't also have significant flaws.  But the
very reason we're trying to change something is that we *already* have
a process with flaws.

We know that no process we ever come up with will be perfect, and
agreeing to change is a question of deciding what the flaw balance is.
 We need to allow ourselves to move from one (known, "comfortable")
set of flaws to a new set, *if* we can come to a reasonable agreement
that we've improved things in a useful way.

Barry

---

From: Donald Eastlake <[email-masked]>
Date: Thu, 31 Jan 2013 20:41:45 -0500

On Wed, Jan 30, 2013 at 1:35 PM, Dave Crocker <[email-masked]> wrote:
> I suspect it's not 'increasingly' but rather that it's always been extremely
> difficult...
>

  "It ought to be remembered that there is nothing more difficult to
take in hand, more perilous to conduct, or more uncertain in its
success, than to take the lead in the introduction of a new order of
things. Because the innovator has for enemies all those who have done
well under the old conditions, and lukewarm defenders in those who may
do well under the new."
                    Niccolo Machiavelli

Donald
=============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 155 Beaver Street, Milford, MA 01757 USA
 [email-masked]

> Hence I suggest that a proposal needs to recruit a committed core /before/
> going public, and the discussion needs classic group facilitation, in terms
> of tracking issues, maintaining focus, and pursuing consensus.
>
> d/
>
> --
>  Dave Crocker

---

From: Dave Crocker <[email-masked]>
Date: Thu, 31 Jan 2013 18:32:16 -0800

> well under the old conditions, and lukewarm defenders in those who may

Nice.

Might be worth adding that to the Intros for all IETF process and 
structure documents...

d/

---

From: Randy Bush <[email-masked]>
Date: Fri, 01 Feb 2013 14:14:53 +0900

> that we've improved things in a useful way.

perfection is the enemy of good

and adding complexity to overcome objections is an engineer's disease

randy