Subject: Re: [Gen-art] Re: Gen-art review of draft-hartman-mailinglist-experiment-01.txt

From: Sam Hartman <[email-masked]>
Date: Mon, 08 May 2006 14:00:06 -0400

>>>>> "Harald" == Harald Alvestrand <[email-masked]> writes:

    >> 
    >> The primary reason you want to encourage meta-experiments is that a
    >> lot of the hypotheses you want to test involve delegation.  For
    >> example I want to test the hypothesis that the right way to solve the
    >> mailing list mess is to delegate it to the IESG.  I could delegate it
    >> to the IESG as part of an experiment along with a initial procedure.
    >> But if I do that I'm testing a different hypothesis: is delegating
    >> something to the IESG with the whole IETF designing the initial
    >> conditions a way to solve the problem.  As you might imagine that's a
    >> different hypothesis.
    Harald> I do not believe that is a correct statement.
    Harald> An RFC 3933 experiment by its very nature involves three different
    Harald> sub-experiments:
    Harald> - Whether you can get the IETF to agree to running this experiment
    Harald> - Whether the bodies that administer the experiment can perform the
    Harald> experiment so that we have something to measure
    Harald> - Whether the experiment produces useful results

I agree with this characterization.

I don't see how this characterization is in conflict with mine.

    Harald> I think what you are proposing will lead to failure of the *first*
    Harald> experiment, which means that you won't get to the other two.
    Harald> If you say "give the IESG the power to produce procedures, we're not
    Harald> going to give you even a hint as to what these procedures might be",
    Harald> this is what I would characterize as selling a pig in a poke, and it's
    Harald> perfectly reasonable for the IETF community to ring up a "no sale" on
    Harald> the consensus register.

The IETF certainly may do that.  You clearly believe the IETF should
do that.  I believe the IETF should not do that.
Ultimately Brian will have to make a consensus call.

I think it is reasonable to delegate comuing up with procedures to the
IESG.  I think that if we're going to delegate that then the
procedures should not be discussed in this draft.
 If the community wants more oversight, here are some things I think would be reasonable to ask for:

* Add criteria the IESG will use for evaluating procedures or for
  creating procedures.
* Requiring proponents of this experiment to submit candidate procedures that demonstrate procedures are possible
* Adding explicit review requirements for procedures that the IESG proposes.
* Establishing principles the IESG should use

If you believe that one of these options would help, I invite you to
propose text or direction.

You could also propose an experiment to run with specific procedures.
I would not object to that experiment but I would also not be
interested in contributing to it.

    Harald> The theory that there is a different hypothesis being tested if the
    Harald> IESG sets the initial procedures than if the initial procedures are
    Harald> part of the experiment also seems doubtful to me; in both cases what I
    Harald> think will happen is that the IESG will start off with procedures
    Harald> designed by Sam Hartman, and the big difference is that the community
    Harald> will have seen the initial procedures before deciding to run the
    Harald> experiment.
    Harald> The whole IETF has never designed anything and never will; all design
    Harald> is done by specific people, and (hopefully) reviewed by some
    Harald> large-enough fraction of the IETF. That's how EVERYTHING in the IETF
    Harald> works.
    >> Since I'm on the IESG I'm actually in a
    >> reasonably good position to negotiate an initial procedure that the
    >> IESG will be happy with and that would be similar to a procedure the
    >> IESG would come up with on its own.  However we want 3933
    >> experiments--even experiments delegating things to the IESG--to be
    >> documents that anyone can write.  So we should require that authors of
    >> 3933 experiments demonstrate stakeholder buy-in for experiments but
    >> not require that they take actions as if they were the stakeholders.
    >> 
    Harald> I do not believe this theory either.

So, the main point of the paragraph here is that I don't believe only
IESG members should be able to propose experiments that change things
for the IESG.  I hope we can agree on that.

    Harald> RFC 3933 talks about the IESG and the community. The "stakeholder"
    Harald> concept is of your introduction, and I believe it serves no benefit.
    Harald> An RFC 3933 proposal author HAS to do three things:

    Harald> - Write up what he's proposing in enough detail that it can be evaluated
    Harald> - Convince the IESG that the experiment has enough merit to warrant a
    Harald> Last Call
    Harald> - Generate enough review and comment in the community that the IESG
    Harald> can confidently say that there's community consensus to run the
    Harald> experiment

I proposed the concept of stakeholder buy-in as an attempt to solve
the first criteria.  I think that the IESG should almost always last
call experiments when the proponets can demonstrate it is possible to
do the work and when those effected are interested in doing the work.
I think that using this criteria will help resolve long-standing
questions of where to focus process reform energy and of what people
need to do in order to move their proposals forward.  I think that the
IESG should focus on process reform where the stakeholders are
interested in the work over process reform where the stakeholders are
not interested.

I do agree that when the community really desires some reform and the
stakeholders are not interested, we need some resolution that allows
the reform to proceed possibly with resignations.  I think the bar in
that case needs to be much higher.

But my concept of stakeholders is more about establishing a criteria
for forward progress than about blocking things.

    Harald> Certainly, a community feedback that says "I'm one of the people
    Harald> required to do work under this experiment, and I'm not going to do it"
    Harald> is a strong statement that needs consideration when assessing whether
    Harald> there's IETF consensus to run the experiment. But sometimes that's
    Harald> just part of the "rough" in rough consensus, even when deciding to run
    Harald> the experiment will cause someone to resign his role. And sometimes
    Harald> it's not.
    >> The second reason that you want to allow meta-experiments is that we
    >> want to encourage RFC 3933 as the first step in process change.
    >> Process change often results in BCPs.  You want the 3933 experiment to
    >> be reasonably similar to a BCP so that when appropriate you can easily
    >> convert a successful experiment into a BCP.  You would probably
    >> replace any evaluation criteria with results of the evaluation,
    >> replace the sunset clause with something else.  However you want the
    >> operative language to remain the same.  A significant result of the
    >> mailing list discussion is the concern that our BCPs are too specific
    >> and encode operational details.  If you require that meta-experiments
    >> are not allowed you strongly push us in the direction of
    >> overly-specific BCPs.  I think that would be a very bad idea.
    >> 
    Harald> Here I agree, but disagree that this document does an appropriate job of it.
    Harald> I think the separation of "principle" and "process" is a good thing -
    Harald> the principle of this document being (I think) that the IESG sets
    Harald> mailing list management procedures, and the process being the process
    Harald> by which such procedures are set, and the third level being the
    Harald> procedures themselves.

    Harald> However, just stating the principle alone at an abstract level is not
    Harald> enough to either start or evaluate the experiment.
 I disagree.  If you'd provided more detail on why you don't think
that this document is sufficient, I could respond to it.  However I
think it might be useful to see what the consensus position is before
we spend a lot of time on this issue.

--Sam

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Margaret Wasserman" <[email-masked]>
Date: Tue, 9 May 2006 09:28:05 -0400

There are certainly some major weaknesses in the current IETF mailing list
management procedures, and those weaknesses are very well-described in this
document.  I agree that we need to address those weaknesses.  However, I am
not sure that I agree with the fix proposed in this document.

This document defines an RFC3933 experiment in which we would temporarily
give the IESG the authority to create new mailing list management procedures
and enact them.  The only hard limitations on this authority are that
posting suspensions cannot run beyond the timeframe of the experiment (18
months), nor can the procedures prevent anyone from reading an IETF mailing
list.  The document does not even limit the type of action that the IESG can
take -- while it only talks about posting rights suspensions, this document
would allow the IESG to define an enact other types of mailing list control
at their discretion.  It explicitly does not require that we use the same
procedures on all IETF mailing lists, as it explicitly allows the IESG to
define different procedures for different lists.  

This document does not define any principles that the IESG should follow in
determining mailing list management procedures, nor does it require any type
of community review or consensus to enact them.  Although it encourages an
IETF Last Call for new procedures, it does not require an IETF Last Call or
IETF consensus to establish a new procedure.  It doesn't even require that
the IESG send a new procedures for community review before enacting it
and/or that they explicitly notify the mailing lists to which the new
procedure would apply.  It merely requires that the new procedure be
published on a public web site.

At first, I thought it was the purpose of this document to allow the IESG to
try out different mailing list management procedures on different IETF
mailing lists for a short period of time, with the goal of picking some
successful procedures that would later be discussed by the community and
potentially reflected in our BCPs.  In other words, I thought that this was
a temporary measure to address the weaknesses in our current procedures and
get some experience with alternatives.  I still thought that the goal would
be to settle on well-defined, community-consensus-based procedures by then
end of this 18 months.  At that time, I supported this document, because I
saw it as a better alternative than living with our broken procedures until
the community could fix them.  I thought that the IESG could end this
experiment if/when we had community consensus on a new set of procedures.  

However, at the last Gen Area meeting, it became clear that if this
experiment is successful, the goal would be to permanently delegate this
level of discretion to the IESG, rather than publishing a specific set of
mailing list management procedures in BCPs.  Our mailing list management
procedures would permanently be defined in a web page that would be
maintained and modified by the IESG, rather than in BCPs that are determined
through a community consensus process.  

It is the nature of RFC 3933 experiments that they are intended to be
adopted as permanent procedural changes if they are found to be successful.
However, this document does not define any criteria for success.  RFC 3933
says:  "At or before the end of the "sunset" timeout, the IESG would either
revise (or cause to be revised) the document into a BCP RFC or the procedure
would expire..."  So, I _think_ what would happen at the end of this
18-month period is that the IESG would decide whether delegating this
authority to themselves was a useful procedural change and, if so, they
would publish a revision of this document as a BCP, permanently delegating
this authority to the IESG.

So, there are two questions I think we need to answer:  

(1) Is the community comfortable with the idea of giving the IESG such wide
discretion to establish and modify our mailing list management procedures? 

(2) Is the proposed experiment well-enough documented that we will be able
to clearly understand whether or not it was successful?

Personally, I would not be comfortable delegating this level of authority to
the IESG.  The former IESG's decisions in this area did not align with my
own principles, and I am not certain that the new IESG would balance
expediency and openness in a way that would be acceptable to me.  I am also
uncomfortalbe with the idea that something as important as our mailing list
management procedures would be documented on a web site that could be
modified without community review and approval.  I understand that I may not
be representative of the larger community in this regard, but I would much
prefer that we fix the problems with our current BCPs through a community
consensus process, resulting in updated BCPs that define our mailing list
management procedures.

I also have a smaller, but more concrete issue with this document.  IMO,
this document doesn't define a well-formed experiment.  In particular, it
doesn't define any criteria or method that that the IESG should use to
determine if the experiment has been successful.  Without such criteria, how
would the IESG decide whether or not to delegate this authority to
themselves on a permanent basis?  It has been recent IESG policy not to
publish experimental RFCs without well-defined success criteria, and I do
not think that this document meets that requirement.

Margaret 

> -----Original Message-----
> From: Sam Hartman [mailto:[email-masked]] 
> Sent: Monday, May 08, 2006 2:00 PM
> To: Harald Alvestrand
> Cc: IETF Discussion
> Subject: Re: [Gen-art] Re: Gen-art review 
> ofdraft-hartman-mailinglist-experiment-01.txt
> 
> The IETF certainly may do that.  You clearly believe the IETF 
> should do that.  I believe the IETF should not do that.
> I think it is reasonable to delegate comuing up with 
> procedures to the IESG.  I think that if we're going to 
> delegate that then the procedures should not be discussed in 
> this draft.
> * Requiring proponents of this experiment to submit candidate 
> procedures that demonstrate procedures are possible
> If you believe that one of these options would help, I invite 
> you to propose text or direction.
> 
> So, the main point of the paragraph here is that I don't 
> believe only IESG members should be able to propose 
> experiments that change things for the IESG.  I hope we can 
> agree on that.
> 
> I proposed the concept of stakeholder buy-in as an attempt to 
> solve the first criteria.  I think that the IESG should 
> almost always last call experiments when the proponets can 
> demonstrate it is possible to do the work and when those 
> effected are interested in doing the work.
> I think that using this criteria will help resolve 
> long-standing questions of where to focus process reform 
> energy and of what people need to do in order to move their 
> proposals forward.  I think that the IESG should focus on 
> process reform where the stakeholders are interested in the 
> work over process reform where the stakeholders are not interested.
> 
> I do agree that when the community really desires some reform 
> and the stakeholders are not interested, we need some 
> resolution that allows the reform to proceed possibly with 
> resignations.  I think the bar in that case needs to be much higher.
> 
> But my concept of stakeholders is more about establishing a 
> criteria for forward progress than about blocking things.
> 
>  I disagree.  If you'd provided more detail on why you don't 
> think that this document is sufficient, I could respond to 
> it.  However I think it might be useful to see what the 
> consensus position is before we spend a lot of time on this issue.
> 
> 
> 
> --Sam
>

---

From: "Margaret Wasserman" <[email-masked]>
Date: Tue, 9 May 2006 09:52:12 -0400

There is an interesting lesson to be learned from our mailing list
management situation...

The mailing list procedures draft currently under discussion
(draft-hartman-mailinglist-experiment-01.txt) contains the following correct
assessment of our current mailing list management situation:

   "RFC 3934 [RFC3934] amends RFC 2418 and grants the working group chair
   the ability to suspend a member's posting rights for 30 days.
   However it appears to remove the ability of the AD and IESG to
   approve longer suspensions or alternative procedures: "Other methods
   of mailing list control, including longer suspensions, must be
   carried out in accordance with other IETF-approved procedures."  An
   argument could be made that the amendment was not intended to remove
   the already-approved procedures in RFC 2418 although a perhaps
   stronger argument can be made that the actual textual changes  have
   the effect of removing these procedures."

Unfortunately, this problem was introduced during IESG and/or RFC Editor
processing of RFC 3934.  The last published I-D (the one circulated for IETF
LC) said:

    "Other
    methods of mailing list control, including longer suspensions, must
    be approved by the IESG or carried out in accordance with other
    IESG-approved procedures."

I was the author of this draft, but I no longer remember how/when this
change was introduced.  Someone from the IESG or RFC Editor might be able to
tell from checking their records, but the change isn't mentioned in the
public tracker.

BTW, I am not disavowing responsibility for this change.  I have every
reason to believe that I, as author of the RFC in question, was asked my
opinion...  

Margaret

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 09 May 2006 16:07:37 +0200

The tracker tracks....

the RFC Editor note was modified (by me) on July 24, 2004. The reason 
was a comment from Ted Hardie on July 21, augumenting a DISCUSS from 
Steve Bellovin:

Steve (DISCUSS):

 >he last paragraph of Section 2 should explain the relationship of this 
document to RFC 3683.

Ted (Comment):

 >I agree with Steve, and would have entered the same DISCUSS.  But I'll 
go one better and
 >suggest text.  Here's the current:
 >
 >However, further
 >    disruptive behavior by the same individual will be considered
 >   separately and may result in further warnings or suspensions.  Other
 >    IESG-approved procedures.
 >
 >
 >I suggest adding the following added sentence:  See BCP83 (RFC 3683)
 >for on set of procedures already defined and accepted by the community.
 >
 >I also think the sentence above should be changed a bit, to reflect the
 >idea that it is IETF approved procedures that result in suspensions, 
rather
 >than IESG approved suspensions.  New suggestion
 >
 >    Other methods of mailing list control, including longer 
suspensions, must
 >    be approved by the IESG and carried out in accordance with other
 >    IETF-approved procedures.  See BCP83 (RFC 3683 for one set of 
procedures
 >    already defined and accepted by the community.

Apparently Ted's proposal found favour with the IESG of that time.

                             Harald

---

From: Sam Hartman <[email-masked]>
Date: Tue, 09 May 2006 10:43:13 -0400

>>>>> "Margaret" == Margaret Wasserman <[email-masked]> writes:

    Margaret> This document defines an RFC3933 experiment in which we
    Margaret> would temporarily give the IESG the authority to create
    Margaret> new mailing list management procedures and enact them.
    Margaret> The only hard limitations on this authority are that
    Margaret> posting suspensions cannot run beyond the timeframe of
    Margaret> the experiment (18 months), nor can the procedures
    Margaret> prevent anyone from reading an IETF mailing list.  The
    Margaret> document does not even limit the type of action that the
    Margaret> IESG can take -- while it only talks about posting
    Margaret> rights suspensions, this document would allow the IESG
    Margaret> to define an enact other types of mailing list control
    Margaret> at their discretion.  It explicitly does not require
    Margaret> that we use the same procedures on all IETF mailing
    Margaret> lists, as it explicitly allows the IESG to define
    Margaret> different procedures for different lists.

Note that this is the same authority the IESG had for WG mailing lists
under RFC 2418.

However I agree that this is not where we want to be long-term.

    Margaret> This document does not define any principles that the
    Margaret> IESG should follow in determining mailing list
    Margaret> management procedures, nor does it require any type of
    Margaret> community review or consensus to enact them.  

I think these are the sorts of details--principles and review
requirements--that the community needs to decide on.  I think that
will take a while to do, and it is my hope that this experiment may
provide input to that process.  Long term though I agree with you that
the BCP for mailing list management must provide principles.

    Margaret> At first, I thought it was the purpose of this document
    Margaret> to allow the IESG to try out different mailing list
    Margaret> management procedures on different IETF mailing lists
    Margaret> for a short period of time, with the goal of picking
    Margaret> some successful procedures that would later be discussed
    Margaret> by the community and potentially reflected in our BCPs.
    Margaret> In other words, I thought that this was a temporary
    Margaret> measure to address the weaknesses in our current
    Margaret> procedures and get some experience with alternatives.  I
    Margaret> still thought that the goal would be to settle on
    Margaret> well-defined, community-consensus-based procedures by
    Margaret> then end of this 18 months.  

That is the goal of this experiment.
I propose adding text to clarify this fact.

I propose adding to the end of the last paragraph in the introduction:

  This experiment is successful if it gives the community useful input
  on how to design mailing list management process.  It is not expected
  that this experiment will be adopted in its current form as a
  permenant BCP.

    Margaret> At that time, I supported
    Margaret> this document, because I saw it as a better alternative
    Margaret> than living with our broken procedures until the
    Margaret> community could fix them.  I thought that the IESG could
    Margaret> end this experiment if/when we had community consensus
    Margaret> on a new set of procedures.

I still believe that to be true.

I'm sorry if my comments at the general area meeting were confusing.
I believe that as a general rule you want the operative part of an
experiment to look like a BCP.  I've been pushing fairly hard for this
even in this case because this is one of the first experiments we are
running.  I do not believe that this current experiment would make a
great BCP.

---

From: "Margaret Wasserman" <[email-masked]>
Date: Tue, 9 May 2006 11:29:58 -0400

Hi Sam,

With the change that you have proposed below, I would support publication of
this document (and the running of this experiment).

While there are a number of small things we could tweak, I think that would
be a waste of time.  This is good enough as a temporary measure to relieve
the current pressures, and I think that our efforts would be better spent on
working on a real BCP proposal along the lines you have described below.

Margaret 

> Sent: Tuesday, May 09, 2006 10:43 AM
> To: Margaret Wasserman
> Cc: 'IETF Discussion'
> Note that this is the same authority the IESG had for WG 
> mailing lists under RFC 2418.
> 
> requirements--that the community needs to decide on.  I think 
> that will take a while to do, and it is my hope that this 
> experiment may provide input to that process.  Long term 
> though I agree with you that the BCP for mailing list 
> management must provide principles.
> 
> 
> I believe that as a general rule you want the operative part 
> of an experiment to look like a BCP.  I've been pushing 
> fairly hard for this even in this case because this is one of 
> the first experiments we are running.  I do not believe that 
> this current experiment would make a great BCP.
>

---

From: "Margaret Wasserman" <[email-masked]>
Date: Tue, 9 May 2006 11:51:59 -0400

Hi Harald,

> The tracker tracks....

Sorry, I didn't realize where I would find RFC Editor notes in the public
tracker.  I have now been properly educated. 

> the RFC Editor note was modified (by me) on July 24, 2004. 
> The reason was a comment from Ted Hardie on July 21, 
> augumenting a DISCUSS from Steve Bellovin:

So, it appears that in July 2004 the IESG made a decision to revoke our own
authority to enforce longer suspensions or to approve new mailing list
management procedures, believing that such procedures should be approved by
the IETF community.  (I say "our", because this all happened in my first
week on the IESG).

Now Sam's document is proposing that we give that authority back to the
IESG, at least on a temporary basis.

Margaret

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 10 May 2006 10:09:15 +0200

Sam,

we have some differences of opinion on how these things work, and how 
they are supposed to work.
But I'll try to be constructive.

I think that in any experiment that involves giving someone the power to 
set procedures, there MUST be some words on how those procedures are set 
(the metaprocedures).

What your -01 draft currently says is:

The IESG MUST inform the community in a public statement of any
procedures for mailing list management approved under this
experiment. Such a statement should include the description of the
procedure and the description of mailing lists to which it applies or
an indication that it applies to all IETF mailing lists. The IESG is
encouraged to last call procedures it is considering approving under
this experiment. While such last calls are encouraged, they are not
required. The reason that the last call is not required is that
under RFC 2418, no last call is required; there seems to be no reason
to have a procedure more strict than that proposed in RFC 2418.

Sanctions made under this memo may be appealed using the procedures
outlined in [RFC2026].

I would add to this (suitably wordsmithed):

The IESG MUST make the procedures for mailing list management public
The IESG MUST make proposed changes public at least 14 days before 
enacting them.
The IESG is NOT required to assert that there is IETF consensus for a 
change in procedures.

(the last also means that I doubt that the term "last call" is useful 
here - a "request for input", like the one the IAB uses for its 
documents, is more in line with the procedures you are proposing)