Subject: RE: Last Call: <C> (On Consensus and Humming in the IETF) to Informational RFC

From: Larry Masinter <[email-masked]>
Date: Mon, 4 Nov 2013 07:52:08 -0800

Re draft-resnick-on-consensus-06:

my problem with this document is that it misses out on what I've always considered essential ... 

For me, "rough consensus" and "running code" should be taken together, not independently. I've always taken it as "rough consensus OF THOSE WITH running code".

I think we want to get the agreement of those who are going to implement and deploy systems which use the specification -- even if they're not in the room or not at the meeting or even not on the mailing list.  Those who attend the meetings and participate on the lists may act as representatives, or might just have opinions...

I don't think the draft helps clarify this point, at all, since it dwells at length on "consensus" and "rough consensus" in terms of numbers independent of the nature of the source.

Larry
--
http://larry.masinter.net

---

From: Dave Cridland <[email-masked]>
Date: Mon, 4 Nov 2013 16:00:19 +0000

On Mon, Nov 4, 2013 at 3:52 PM, Larry Masinter <[email-masked]> wrote:

> For me, "rough consensus" and "running code" should be taken together, not
> independently. I've always taken it as "rough consensus OF THOSE WITH
> running code".
>
> I think we want to get the agreement of those who are going to implement
> and deploy systems which use the specification -- even if they're not in
> the room or not at the meeting or even not on the mailing list.  Those who
> attend the meetings and participate on the lists may act as
> representatives, or might just have opinions...
>
> I don't think the draft helps clarify this point, at all, since it dwells
> at length on "consensus" and "rough consensus" in terms of numbers
> independent of the nature of the source.
>

That is an exceedingly good point.

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 04 Nov 2013 08:07:06 -0800

On 11/4/13 7:52 AM, Larry Masinter wrote:

> For me, "rough consensus" and "running code" should be taken
> together, not independently. I've always taken it as "rough consensus
> OF THOSE WITH running code".

Larry, that is awesome!

/psa

---

From: Pete Resnick <[email-masked]>
Date: Mon, 4 Nov 2013 08:46:13 -0800

> I don't think the draft helps clarify this point, at all, since it dwells at length on "consensus" and "rough consensus" in terms of numbers independent of the nature of the source.

I want to reserve judgment on the main point Larry makes, but I 
explicitly disagree with the above, and I'd like to hear his reaction to 
this before going back to the main question:

The draft explicitly *disavows* talking about numbers. It says that you 
*can't* determine rough consensus on the basis of numbers, but rather 
must look at the issues. And I think it says (or at least implies) that 

> For me, "rough consensus" and "running code" should be taken together, not independently. I've always taken it as "rough consensus OF THOSE WITH running code".

If you've got a real technical issue, I don't think it matters if you 
have running code.

Now, the reason I want to reserve judgment is because when we get to the 
"rough" part, the draft says that we do have to make engineering 
tradeoffs and might decide that we're going to dismiss a real technical 
issue based on some other reasons (quoting the document, e.g., "But we 
have a great deal of code already written that is similar to protocol X. 
While I agree that protocol Y is more elegant, the risks to 
interoperability with an untested solution is not worth it compared to 
the advantages of going with the well-understood protocol X."). Does the 
set of people saying "we need a different tradeoff" need to be folks who 
have running code? That I might be convinced by. But I'm not sure. I'd 
like to hear about some suggested text for sections 2 or 3 that talk 
about this. I certainly don't want to require that any issue requires 
someone with running code. But I'd like to hear more on this.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Ted Lemon <[email-masked]>
Date: Mon, 4 Nov 2013 19:28:42 +0000

>> OF THOSE WITH running code".

Unfortunately it's pretty easy to game this by writing some running code that isn't very usable, and then claim that you are right based on the fact that you have running code and the other folks don't.   I think this should certainly be considered, but it's not quite as strong an argument as is being claimed.

---

From: Dave Cridland <[email-masked]>
Date: Mon, 4 Nov 2013 21:01:56 +0000

On Mon, Nov 4, 2013 at 7:28 PM, Ted Lemon <[email-masked]> wrote:

> Unfortunately it's pretty easy to game this by writing some running code
> that isn't very usable, and then claim that you are right based on the fact
> that you have running code and the other folks don't.   I think this should
> certainly be considered, but it's not quite as strong an argument as is
> being claimed.
>
>
I think you're misunderstanding what "running code" means - it should not
be taken to mean "code that runs", in as much as it compiles, or whatever,
but "code that is running", meaning deployed and actively used. That is
quite hard to game, since if you manage to contrive some code that's in
deployment just to prove a point, you're really no longer gaming the system.

What Larry is saying - or at least, what I'm agreeing with, regardless of
whether Larry meant this or not - is that if you've a bunch of people
saying "I've implemented this in production, and X needs to be Y", then
that is a very hard argument to beat.

It's not insurmountable - various security-based arguments might hold up
well in particular - but it thoroughly raises the bar for counter-arguments.

Dave.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 5 Nov 2013 00:08:36 +0000

On Nov 4, 2013, at 1:01 PM, Dave Cridland <[email-masked]> wrote:
> What Larry is saying - or at least, what I'm agreeing with, regardless of whether Larry meant this or not - is that if you've a bunch of people saying "I've implemented this in production, and X needs to be Y", then that is a very hard argument to beat.

Historically we haven't just blessed existing implementations if there were things that needed to be fixed.   This interpretation of "running code" is a bit problematic because people with running code have a vested interest in the spec not changing.   Not always a bad thing, but it can be quite harmful.

---

From: [email-masked] (Martin Rex)
Date: Tue, 5 Nov 2013 05:18:01 +0100 (CET)

Ted Lemon wrote:
> > What Larry is saying - or at least, what I'm agreeing with,
> > regardless of whether Larry meant this or not - is that if
> > you've a bunch of people saying "I've implemented this in
> > production, and X needs to be Y", then that is a very hard
> > argument to beat.
> 
> Historically we haven't just blessed existing implementations
> if there were things that needed to be fixed.   This interpretation
> of "running code" is a bit problematic because people with running
> code have a vested interest in the spec not changing.
> Not always a bad thing, but it can be quite harmful.

It has proven to be quite harmful several times and in several ways.

It regularly causes bloat (functionality that is easily dispensible
to a huge amount of users/consumers) to stay in the specification with
a requirements tagged to it, rather than being a real option.

And the folks fighting for the bloat to stay in the spec are either
vendors that appreciate when its harder for competition to compete
in general, rather than for a limited subset of consumers.  The others
are organizational interest groups (often governmental) that, while
representing a minority among real consumers, would like specific bloat
in products without having to pay extra.  I don't mind a standard
offering them _options_.  But I strongly dislike the IETF pushing
requirements on bloat that the large majority doesn't need,
just to please a specific minority.

-Martin

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 5 Nov 2013 08:24:12 -0800

> certainly be considered, but it's not quite as strong an argument as is

Because of this, I would weight an open source implementation more heavily
because it can be viewed by the community.  I've found that developing an
open source implementation at the same time as the draft is under
development has been highly valuable to both; the running code can be
inspected for quality, and what's left after all of that gets tightened
thus deserves reflection in the draft revisions.

I realize it's easier to talk about open source in applications work versus
the lower layers, however.

Also valuable are interoperability events; even if the source code can't be
shared, bogus implementations become readily visible.

I think the output of both of those projects deserve serious consideration
when evaluating consensus.  If rough consensus alone is clearly in
contradiction to the data provided by running code, there's something
seriously wrong.

-MSK

---

From: Larry Masinter <[email-masked]>
Date: Tue, 5 Nov 2013 11:12:33 -0800

> > > production, and X needs to be Y", then that is a very hard

I didn't mean that. I think arguments need to be founded on technical justifications; claims of results  backed by open, reproducible data.

What I meant was that the "rough" in "rough consensus" is that non-implementor complaints shouldn't carry as much weight if all the implementors agree; you should listen to non-implementors and make sure that the agreement is based on technical judgment and not entirely on market or legacy considerations.

I have (unfortunately) been in working group meetings where none of the implementers were present.  Even if EVERYONE agrees, EVERYONE hums yes, I wouldn't call it "rough consensus", because the real implementors weren't there.

Larry

---

From: Abdussalam Baryun <[email-masked]>
Date: Wed, 6 Nov 2013 07:55:56 +0000

On Tuesday, November 5, 2013, Larry Masinter wrote:

>
>
> I have (unfortunately) been in working group meetings where none of the
> implementers were present.  Even if EVERYONE agrees, EVERYONE hums yes, I
> wouldn't call it "rough consensus", because the real implementors weren't
> there.

I agree for protocol standards we will need implementors comments and hums,
that is why remote hums are more important which the draft misses to
mention. We need more reviewers and implementors, not more management
members.

We may think to attract/pay implementors not managers. If IETF creates a
special hat/authority for implementors or a body, as each IETF Area needs
an implementor leader.

AB

---

From: Larry Masinter <[email-masked]>
Date: Sat, 9 Nov 2013 05:17:15 -0800

I wrote various things about rough consensus and the implementors and "those with implementations" in reaction to draft-resnick-on-consensus-06.

After discussion and more thought, I wanted to retract what I said, and offer some specific comments on the draft.

Retraction: 
"Running code" means it's the implementations (not necessarily the implementors) that should carry the weight. Yes, for protocols to be implemented the implementors (eventually) have to agree, because otherwise the protocol doesn't get implemented.  But "running code" is really the code, and not just the opinion of those who might write the code. 

To those who agreed with me:
Decisions should be based on technical arguments (including deployment considerations): "I implement X and I say Y" is not a technical argument, even if the speaker is the author of a popular implementation.
Managers of implementors, product managers who specify future implementations, consultants who advise implementors, those who install or deploy implementations, academics who have analyzed implementations, ...  those are all roles that can represent implementations credibly.

In the end, my main complaint about draft-resnick-on-consensus is about the introduction (and abstract), as the draft seems at times to redefine "rough consensus" even as it says it doesn't.

I offer Pete some specific edits as well as some comments. Some of the edits are minor editorial. The edits are offered just as examples, I've tried to include a rationale for each. Some things I don't know how to fix yet.

=== OLD ===
... (current section 1 Introduction) ... 
=== NEW ===
1. Introduction

[BCP 9] [BCP 11] and [BCP 25] ("The Internet Standards Process", "The
Organizations Involved in the IETF Standards Process" and
"Implementation Report Guidance", "IETF Working Group Guidelines and
Procedures") describe the overall standards process in the IETF.

[TAO] discusses some of the traditional operational procedures used in
IETF, including:

    Sometimes consensus is determined by "humming" -- if you agree with
    a proposal, you hum when prompted by the chair. Most "hum"
    questions come in two parts: you hum to the first part if you
    agree with the proposal, or you hum to the second part if you
    disagree with the proposal. Newcomers find it quite peculiar, but
    it works. It is up to the chair to decide when the Working Group
    has reached rough consensus.

Others have also written about ways to achieve (rough) consensus in
IETF and other organizations, (e.g.,
http://www.w3.org/2005/10/Process-20051014/policies.html#Consensus,
http://www.w3.org/Guide/chair-roles, [dussealt]
http://tools.ietf.org/html/draft-dusseault-consensus-00 )

Unfortunately, it seems recently that more and more of our actions are
now indistinguishable from voting, and quite often we are letting the
majority win the day without consideration of minority concerns.  This
document is a collection of thoughts how to get back to the core
values of the organization.

This document is quite consciously being put forward as Informational.
It does not propose to change any IETF processes and is therefore not
a BCP. It is simply a collection of principles, hopefully around which
the IETF can come to (at least rough) consensus.

2. Humming

As [TAO] notes: 

   One of the "founding beliefs" [of the IETF] is embodied in an early
   quote about the IETF from David Clark: "We reject kings, presidents
   and voting. We believe in rough consensus and running code".

That is, our credo is that we don't let a single individual dictate
decisions (kings and presidents), nor should decisions be made by a
simple vote of the majority, nor do we want decisions to be made in a
vacuum without practical experience.  Instead, we strive to make our
decisions by the consent of all participants, though allowing for some
dissent (rough consensus), and to have the actual products of
engineering trump theoretical designs (running code).
==== END NEW ===

Why:  It should be clear that this document doesn't override
existing BCPs, and that 'rough consensus' is already defined.
I like giving explicit reference to where it *is* defined, as well
as other guidelines and discussions of consensus.
I think the fact that this isn't a BCP should be in the document, 
not just a note.

In many ways this is part of the "TAO for the Experienced",
since TAO claims to be for novices.

I don't think the document is harmed by removing the
definition of "rough consensus" and just pointing to where
it is already elaborated.

===== OLD =====
              When, for example, the chair of the working group wants
   to get a "sense of the room" during a face-to-face meeting, instead
   of a show of hands, sometimes the chair will ask for each side to hum
   for or against a question.
===== NEW ====
              For example, when the chair of a working group wants
    to get a "sense of the room" during a face-to-face meeting, instead
   of a show of hands, the chair might ask for each side to hum
for or against a question.
===== END NEW =====
Why: just editorial. "when the chair" is part of the example.

==== OLD === 
   This document attempts to explain some features of rough consensus,
   explain what is not rough consensus, discuss some new ways to think
   about rough consensus, and suggest ways that we might achieve rough
   consensus and judge it in the IETF.
=== NEW ===
   This document gives some additional thoughts on the meaning of
  rough consensus,  notes some situations where rough consensus hasn't 
  been reached, discusses some new ways to think about rough consensus,
  and suggests ways that we might better achieve rough consensus
   and judge it in the IETF.  
=== END NEW ===
Why: editorial; "explaining" sounds more like "redefining". 

=== OLD ==
   These sorts of "capitulation" or "horse-trading" compromises have no
   place in consensus decision making.
=== END OLD ==
comment: I don't know exactly how to fix this, but "log rolling" (
http://en.wikipedia.org/wiki/Logrolling ) is common in all standards
organizations, and is a common failure mode of standards groups
that vote ("A camel is a horse designed by a committee.")  I think this
might need a whole other essay of why rough consensus is better
than voting. 

=== OLD  ===
   It is important to recognize that this view of rough consensus is a
   change from the way it has been traditionally characterized in the
   IETF. 
=== NEW ==
    This view of rough consensus differs from how recently
    it is often characterized in the IETF.
=== END NEW ===
Why: not redefining rough consensus but countering recent trends.

=== OLD ===
   Any finding of rough consensus needs, at some level, to provide a
   reasoned explanation to the person(s) raising the issue of why their
   concern is not going to be accommodated.
=== NEW ===
   Any finding of rough consensus needs, at some level, to provide,
   for each technical objection raised, a reasoned explanation why the
   concern is not going to be accommodated.
=== END NEW ===

Why: the explanation needs consensus too.

=== OLD ===
   Remember, if the objector feels that the issue is so essential that
   it must be attended to, they always have the option to file an
   appeal.
=== END OLD ===
comment: I don't know how to fix this. Asking someone to file an appeal is a
cop-out. Technical objections need responses that have rough consensus
as well, that the response is adequate.  You get this right later on,
but "they always have the option to file an appeal" may be true,
but more often lately "they always have the option to walk away saying
 'the IETF is broken'. 

=== OLD ===
4.  Humming should be the start of a conversation, not the end
=== NEW ===
4. Humming is only one way of helping a group come to real consensus.
=== END NEW ===
Why: humming is neither the start nor the end.

=== OLD ===
     ...    We've also decided that coming to
   consensus (albeit sometimes rough consensus) is an important thing to
   do. 
=== NEW ==
   The IETF's success in designing protocols that are effective, with
   secure, efficient, reliable, interoperable implementations widely
   deployed has rested on the reliance on rough consensus of the
   entire community.
=== END NEW ==
Why: Editorial: I wanted to emphasize that "rough consensus" is
a crucial factor in the success of the IETF protocol.