Subject: 6tsch BoF

From: Ralph Droms <[email-masked]>
Date: Thu, 1 Aug 2013 10:50:54 +0200

I found the process in the 6tsch BoF (Tue 1520) for asking about taking on the work discussed in the BoF to be thought-provoking.

Toward the end of the BoF, the chairs asked the question "1. Is this a topic that the IETF should address?"  First, the chairs asked for a hum.  From my vantage point (middle of the room), the hum was pretty close to equal, for/against.  I reviewed the audio (http://www.ietf.org/audio/ietf87/ietf87-bellevue-20130730-1520-pm2.mp3, starting about 1:22), and heard a slightly louder hum "for".  The BoF chairs decided they needed more information than they could extract from the hum, so they asked for a show of hands.  From the audio record, there were "a lot" for (which matches my recollection) and "a handful" against (my memory is that no hands showed against).  There was a request to ask for a show of hands for "how many don't know".  The question was asked, and the record shows "a dozen".

So, there was apparently a complete change in the answer to the question based on humming versus voting.  There may also have been some effect from asking, after the fact, for a show of hands for "don't know".

I'm really curious about the results, which indicate that, at least in this case, the response to the question is heavily dependent on the on the mode used to obtain the answers to the question (which we all know is possible).  In particular, the effect of humming versus show of hands was pretty obvious.  draft-resnick-on-consensus gives several reasons why humming is preferred over a show of hands.  From this example, it seems to me to be worth considering whether a more honest and accurate result is obtained through humming rather than a show of hands.

The other question raised in my mind is why the initial result from the hum, which did not have a consensus either way, was not sufficient.  "Roughly the same response" for/against the question would seem to me to be as valid a result as explicit consensus one way or the other, and the act of taking a show of hands to survey the appeared to treat the hum as irrelevant, rather than highly significant. 

- Ralph

---

From: Andy Bierman <[email-masked]>
Date: Thu, 1 Aug 2013 02:14:33 -0700

Hi,

Isn't it obvious why humming is flawed and raising hands works?
(Analog vs. digital).  A hand is either raised or it isn't.
The sum of all hands raised is comparable across tests.
The sum of the amplitude of all hums is not.

Andy

---

From: joel jaeggli <[email-masked]>
Date: Thu, 01 Aug 2013 11:29:20 +0200

> The sum of the amplitude of all hums is not.
Consensus for any particular outcome is in the end a judgment call.

---

From: Ralph Droms <[email-masked]>
Date: Thu, 1 Aug 2013 11:34:52 +0200

> The sum of all hands raised is comparable across tests.

The repeatable test gives *an* answer, but is not necessarily the answer that best reflects the sentiment of those answering the question.

A relatively imprecise thermometer that gives a reading close to the measured temperature is more useful than a digital thermometer that gives a precise but highly inaccurate reading.

- Ralph

---

From: Andy Bierman <[email-masked]>
Date: Thu, 1 Aug 2013 02:50:58 -0700

> A relatively imprecise thermometer that gives a reading close to the measured temperature is more useful than a digital thermometer that gives a precise but highly inaccurate reading.

I disagree.  Whether I raise my hand to ear level, 2 inches above my head,
or as high as I can reach, the chair will still count my raised hand as "1".
If I hum really load (and if everybody hums at a different volume) the
chair cannot possibly know how to quantify that result.

Quantifying the number of raised hands is not a judgement call,

Andy

---

From: manning bill <[email-masked]>
Date: Thu, 1 Aug 2013 03:04:02 -0700

we have never voted at IETFs.
"we believe in rough consensus and running code"

/bill

---

From: Joe Abley <[email-masked]>
Date: Thu, 1 Aug 2013 12:19:15 +0200

> "we believe in rough consensus and running code"

The enduring tautology in this is the use of the word "we".

"some of us believe in rough consensus and running code, probably enough that the mantra is worth repeating, although we have not measured support for this quantitatively, your mileage may vary"

:-)

Joe

---

From: SM <[email-masked]>
Date: Thu, 01 Aug 2013 03:29:03 -0700

Hi Ralph,
At 01:50 01-08-2013, Ralph Droms wrote:
>Toward the end of the BoF, the chairs asked the question "1. Is this 
>a topic that the IETF should address?"  First, the chairs asked for 
>a hum.  From my vantage point (middle of the room), the hum was 
>pretty close to equal, for/against.  I reviewed the audio 
>(http://www.ietf.org/audio/ietf87/ietf87-bellevue-20130730-1520-pm2.mp3, 
>starting about 1:22), and heard a slightly louder hum "for".  The 
>BoF chairs decided they needed more information than they could 
>extract from the hum, so they asked for a show of hands.  From the 
>audio record, there were "a lot" for (which matches my recollection) 
>and "a handful" against (my memory is that no hands showed 
>against).  There was a request to ask for a show of hands for "how 
>many don't know".  The question was asked, and the record shows "a dozen".

Switching from hums to votes is not a good idea.  The above could be because:

   (a) The question asked was incorrect.

   (b) The choices provided were incorrect.

>So, there was apparently a complete change in the answer to the 
>question based on humming versus voting.  There may also have been 
>some effect from asking, after the fact, for a show of hands for "don't know".
>
>I'm really curious about the results, which indicate that, at least 
>in this case, the response to the question is heavily dependent on 
>the on the mode used to obtain the answers to the question (which we 
>all know is possible).  In particular, the effect of humming versus 
>show of hands was pretty obvious.  draft-resnick-on-consensus gives 
>several reasons why humming is preferred over a show of hands.  From 
>this example, it seems to me to be worth considering whether a more 
>honest and accurate result is obtained through humming rather than a 
>show of hands.

Your message does not provide enough context.  The simple explanation 
is that the vote was used to change the answer.  Consensus is a 
process, i.e. there are steps which are carefully followed until 
everyone is okay with the choices which they are provided with.  In 
case of doubt, hum.

>The other question raised in my mind is why the initial result from 
>the hum, which did not have a consensus either way, was not 
>sufficient.  "Roughly the same response" for/against the question 
>would seem to me to be as valid a result as explicit consensus one 
>way or the other, and the act of taking a show of hands to survey 
>the appeared to treat the hum as irrelevant, rather than highly significant.

The first result was sufficient.  A second try, whether it is a hum 
or a vote, creates an appearance that the decision-maker did not like 
the result and is trying to change it.  A decision can be undermined 
by people conforming to what other people in the group think.

Regards,
-sm

---

From: Andy Bierman <[email-masked]>
Date: Thu, 1 Aug 2013 04:16:42 -0700

> "we believe in rough consensus and running code"

We are not voting.
We are expressing agreement with a specific assertion.
That is true whether the agreement is expressed via vocalization
or motion of limbs.

Humming allows the chairs to be totally biased and interpret the hum volume
in a way that aligns with their preference for the outcome.

However, I have seen many times where the chairs clearly miscounted
the number of raised hands in order to make the outcome align
with their preference.  However this is much harder to pull off than comparing
2 similar hum volumes.

The chairs can pick however they want to measure agreement.
Many chairs ask for a show of hands.  I prefer that method.
There is still a judgement call when the numbers for and against
are not significantly different.

Andy

---

From: Yoav Nir <[email-masked]>
Date: Thu, 1 Aug 2013 11:24:20 +0000

> The sum of the amplitude of all hums is not.

Hums are better as they give greater weight to people who are more vocally in support (or in opposition) to the assertion.

Research shows([1]), that the one humming loudly for acceptance, will also volunteer to review and contribute code. The one humming loudly against is going to jump up to the mike in all future meetings and tell the group that they're doing the wrong thing. Those who hum softly will go back to reading their email.

Yoav

[1] citation needed

---

From: Scott Brim <[email-masked]>
Date: Thu, 1 Aug 2013 13:38:28 +0200

See draft-resnick-on-consensus for the art of running a group using hums
and other tools.  With those nuances, I like hums.

---

From: Andy Bierman <[email-masked]>
Date: Thu, 1 Aug 2013 06:30:19 -0700

> Hums are better as they give greater weight to people who are more vocally in support (or in opposition) to the assertion.

Please provide some evidence that a loud hum means the person is more
committed to work on an item.

Favoring loud humming is subject to cultural bias.
Some cultures are more inclined to raise their voice than others.
Some people have naturally louder voices than others.
Measuring volume may introduce bias in favor of loud men and against
soft-spoken women, for example.

This cultural bias is not compatible with increased inclusiveness.

Andy

---

From: Ralph Droms <[email-masked]>
Date: Thu, 1 Aug 2013 16:15:07 +0200

> Favoring loud humming is subject to cultural bias.

As is identifying oneself with a position by raising one's hand.  I can imagine as much - my uninformed opinion would be more - cultural bias effect on a show of hands as on a hum.

> soft-spoken women, for example.

And a show of hands may introduce bias against women preferring anonymity.

- Ralph

---

From: Ted Lemon <[email-masked]>
Date: Thu, 1 Aug 2013 15:00:59 +0000

We actually had a talk about this amongst several IESG and former IESG members.   I am not going to report the results, because I might remember them wrong, but my thoughts on this are as follows:

- The hum is not a means of determining consensus; it is a means of determining the sense of the room.   If the hum is clearly many in favor, nobody against,then you have consensus, but that's unusual.   If the hum shows many in favor and some against, you need to go to the next step, because you do not yet know whether you have consensus.

- So since the hum was quite inconclusive, the next question is, "can somebody who objects please say why."   One way to approach that is to ask for a show of hands.   Here, the chairs could have asked for a show of hands against—the show of hands for was unnecessary.   But this is forgivable, I think, unless you think people were intimidated by the show of hands for.   I think that would be hard to argue, given that they had already heard the result of the hum.

- So the show of hands against elicited no hands.   I was there, that's what I remember seeing too.

So at this point, what do you do?   Nobody is willing to stand up and say "I think we shouldn't go forward with this because of X."   Many people have said "we think we should go forward with this, support doing this, and want to participate."

To me, that's consensus.

That wasn't the full outcome of the BoF; there was a lot of good feedback and the charter needs work.   But it's pretty clear to me that in principle at least, there was consensus to do this work.

If you disagree, now would be a fine time to say so, although if in fact we get a charter that the IESG decides is worth floating, you will get another chance during the last call on the charter.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 01 Aug 2013 17:32:29 +0200

Ralph, et al,

Perhaps I have missed relevant responses, but it appears that folk are 
missing what is significant here:

On 8/1/2013 10:50 AM, Ralph Droms wrote:
>  In particular, the effect of humming versus
> show of hands was pretty obvious.

The fact that the results were so profoundly different should get our 
attention, enough to get us to consider specifying how to measure 
consensus.  From the data you cited, it appears that raising hands 
carries some sort of social onus, at least for some people some times, 
that raising hands does not.

Perhaps that doesn't bother other folk very much but the differential 
result was so extreme -- as a single-event experiment -- it strongly 
suggests we should not call for hand-raising.  (The likely explanations 
for the difference are pretty straightforward.  Whether our community of 
engineers wants to believe the explanations or not doesn't matter  The 
data should be sufficiently compelling.)

> sufficient.

Exactly.  This is a point of working group management that should prompt 
some concern.

It's not appropriate to throw out results that were validly obtained but 
yield unpleasant results, and then repeat the same query.  (In fact a 
repetition of a survey on the same sample population is a rank violation 
of reasonable experimental methodology.)

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Barry Leiba <[email-masked]>
Date: Thu, 1 Aug 2013 12:09:58 -0400

> That is true whether the agreement is expressed via vocalization

Absolutely so.

> are not significantly different.

Just some somewhat arbitrary notes here:

There are times when I want to count hands, but it's not for a "vote".
 If I ask who will commit to working on something, I'm looking for a
critical mass that doesn't require counting... but I'm often *also*
looking for commitments, which requires both counting and
identification.

There are also times when I just want to see the relative balance.  As
Andy says, listening to hums can be dicey, even if we don't try to
second guess whether people who hum loudly have stronger opinions
and/or more enthusiasm, or are just demonstrating a cultural
difference.  But if I get a show of hands and do *not* count, I see
that relative balance better.

On the other hand, in perhaps the same situation, if it turns out that
I see only a few hands on one side of the question, I can see whom to
ask to stand up and tell us *why*, and that's often the real value of
the hums: we have a few people who object to X, and we need them to
tell us what their issues are so they can be discussed and addressed.

It drives me nuts [1] when chairs ask for a hum, hear a loud hum for A
and a weak hum for B, and then say, "OK, we have rough consensus,"
without pursuing the issue: What are the objections to A?  Do the B
people just *prefer* B, but they can live with A?  Or is A
fundamentally flawed, but only a few people understand that?

Barry

[1] Admittedly, a short drive.

---

From: Melinda Shore <[email-masked]>
Date: Thu, 01 Aug 2013 08:25:06 -0800

> Consensus for any particular outcome is in the end a judgment call.

Well, yes and no, but this situation strikes me as odd, and probably
a mistake on the part of the chairs.  If you can't tell whether or
not you've got consensus, you don't have consensus.  It sounds like
they were treating the hum as a vote in the first place (which we
do entirely too often).

Melinda

---

From: SM <[email-masked]>
Date: Thu, 01 Aug 2013 11:26:57 -0700

Hi Andy,
At 09:26 01-08-2013, Andy Bierman wrote:
>I meant that it is difficult to remain anonymous when the
>email to the WG has your email address in it.

Agreed.

>Perhaps you can point me to the RFC 2026 text (or other RFC)
>that says something about protecting the disclosure of the identity
>of the person expressing agreement or disagreement with
>some question asked in a WG meeting.

I read RFC 2026 quickly.  I did not find any text about protecting 
the disclosure of the identity of a person.  I read another RFC which 
was about a WG meeting.  It states that there are requirements for 
disclosure of conflicts of interest.  I did not find any text about 
protecting the disclosure of the identity of the person.

Regards,
-sm

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 02 Aug 2013 08:54:31 +1200

> committed to work on an item.

I spotted a number of virtual :-)s in Yoav's message.

The fact is that both methods are broken. A loud hum may indeed
represent strength of feeling, which in judging consensus is
valuable input. Or it may represent chance (some people naturally
hum louder) or a cultural split in opinion. So it's fairly
meaningless. A lot of hands up may indicate that a couple of
employers have loaded the room.

One can make a case that we shouldn't use either method: just
go by the arguments made in the room and on the list.

In the case of a WG-forming BOF, it seems to me that a nucleus
of people willing and competent to do the work, and a good set of
arguments why the work needs to be done and how it will make the
Internet better, are more important than any kind of numbers game.

  Brian

---

From: Melinda Shore <[email-masked]>
Date: Thu, 01 Aug 2013 16:38:30 -0800

> Internet better, are more important than any kind of numbers game.

Yes, and "Who here is planning on contributing to
this effort if we're chartered?" is an entirely different
question from "Who here thinks we should charter a
working group?"

Finding consensus is not simply a matter of asking
a particular question at a given point in time.  It's
a process.  I cannot recommend strongly enough that
people spend time with Pete's document:
http://datatracker.ietf.org/doc/draft-resnick-on-consensus/

Melinda

---

From: joel jaeggli <[email-masked]>
Date: Fri, 02 Aug 2013 08:55:53 +0200

On 8/1/13 6:25 PM, Melinda Shore wrote:
> not you've got consensus, you don't have consensus.
Yes agree The distinction of none or any variant of something else is a
judgement call.
>   It sounds like
I see nothing wrong with taking the temperature of the room.  further
exposition on the mailing list might be a discriminator that would be
easier to identify the positions and what action should therefore occur.

---

From: SM <[email-masked]>
Date: Fri, 02 Aug 2013 00:13:50 -0700

Hi Ted,
At 08:00 01-08-2013, Ted Lemon wrote:
>We actually had a talk about this amongst 
>several IESG and former IESG members.   I am not going to

Bernard Aboba once mentioned:

   To paraphrase Tilda Swinton's Oscar Acceptance Speech:

   "To the IESG, you know, the seriousness and the
    dedication to your art... you rock, man!"

>  report the results, because I might remember 
> them wrong, but my thoughts on this are as follows:
>
>- The hum is not a means of determining 
>consensus; it is a means of determining the 
>sense of the room.   If the hum is clearly many 
>in favor, nobody against,then you have 
>consensus, but that's unusual.   If the hum 
>shows many in favor and some against, you need 
>to go to the next step, because you do not yet know whether you have consensus.
>
>- So since the hum was quite inconclusive, the 
>next question is, "can somebody who objects 
>please say why."   One way to approach that is 
>to ask for a show of hands.   Here, the chairs 
>could have asked for a show of hands against—the 
>show of hands for was unnecessary.   But this is 
>forgivable, I think, unless you think people 
>were intimidated by the show of hands for.   I 
>think that would be hard to argue, given that 
>they had already heard the result of the hum.
>
>- So the show of hands against elicited no 
>hands.   I was there, that's what I remember seeing too.
>
>So at this point, what do you do?   Nobody is 
>willing to stand up and say "I think we 
>shouldn't go forward with this because of 
>X."   Many people have said "we think we should 
>go forward with this, support doing this, and want to participate."
>
>To me, that's consensus.

The above comment about consensus is an opinion. :-)

In my opinion part of the answer has been 
provided by Brian Carpenter.  The other part of 
the answer is the minutes.  The rest of the 
answer is in something mentioned in the Note Well.

Regards,
-sm

---

From: Ted Lemon <[email-masked]>
Date: Sat, 3 Aug 2013 12:47:27 +0000

On Aug 2, 2013, at 9:13 AM, SM <[email-masked]> wrote:
> In my opinion part of the answer has been provided by Brian Carpenter.  The other part of the answer is the minutes.  The rest of the answer is in something mentioned in the Note Well.

Do you think I said something that contradicts what Brian said?   I believe that I agree with him.   But he's talking about how you ultimately determine consensus, whereas I was talking about what hums and shows of hands mean, and in what way they are useful.

---

From: Ted Lemon <[email-masked]>
Date: Sat, 3 Aug 2013 12:47:34 +0000

On Aug 1, 2013, at 5:32 PM, Dave Crocker <[email-masked]> wrote:
> Perhaps that doesn't bother other folk very much but the differential result was so extreme -- as a single-event experiment -- it strongly suggests we should not call for hand-raising.  (The likely explanations for the difference are pretty straightforward.  Whether our community of engineers wants to believe the explanations or not doesn't matter  The data should be sufficiently compelling.)

I thought this was fascinating too.   And there may be some cultural issue at work here.   But letting the negative hums indicate a lack of consensus all by themselves would have been wrong: that's not how you determine consensus _either_.

So what do you think should have been done differently?

To be clear:

> Exactly.  This is a point of working group management that should prompt some concern.

If when the hum comes out indeterminate, you conclude that you do not have consensus, you are holding a vote.

---

From: SM <[email-masked]>
Date: Sat, 03 Aug 2013 09:34:43 -0700

Hi Ted,
At 05:47 03-08-2013, Ted Lemon wrote:
>Do you think I said something that contradicts what Brian said?   I 
>believe that I agree with him.   But he's talking about how you 
>ultimately determine consensus, whereas I was talking about what 
>hums and shows of hands mean, and in what way they are useful.

I don't think that Brian was talking about how you ultimately 
determine consensus.  Here is the last paragraph of Brian's message:

  "In the case of a WG-forming BOF, it seems to me that a nucleus
   of people willing and competent to do the work, and a good set of
   arguments why the work needs to be done and how it will make the
   Internet better, are more important than any kind of numbers game."

That one sentence covers all the points which are relevant.  It's an 
Area Director decision.  It does not require consensus or any kind of 
number game.

Regards,
-sm

---

From: Ofer Inbar <[email-masked]>
Date: Sat, 3 Aug 2013 13:27:15 -0400

> that raising hands does not.

Unless I missed something, it was not clear how much of the effect was
due to hands vs. hum, and how much of it was due to people being asked
a second time after they'd already seen (err, heard) the sense of the
room when it was asked just previously.
  -- Cos

---

From: Ted Lemon <[email-masked]>
Date: Sat, 3 Aug 2013 20:13:52 +0000

On Aug 3, 2013, at 6:34 PM, SM <[email-masked]> wrote:
> That one sentence covers all the points which are relevant.  It's an Area Director decision.  It does not require consensus or any kind of number game.

The working group charter explicitly requires IETF consensus.   But if you mean getting the charter out of the BoF, I think you are right.

---

From: Barry Leiba <[email-masked]>
Date: Sat, 3 Aug 2013 23:23:52 +0200

> That one sentence covers all the points which are relevant.  It's an Area
> Director decision.  It does not require consensus or any kind of number
> game.

Well, where the hum is useful here (and why I prefer to use raised
hands for this particular point) is that, while I don't much care what
the *numbers* are, I very much *do* care if there are objections, and
I need to hear what those objections are.  That's critical.  For that,
I need to have those with the objections willing to be identified and
to tell me what the issues are.

Barry, being an AD and all.

---

From: Scott Brim <[email-masked]>
Date: Sat, 3 Aug 2013 20:27:38 -0400

What did you think of Pete Resnick's draft about hums.

---

From: Barry Leiba <[email-masked]>
Date: Sun, 4 Aug 2013 06:55:00 +0200

> What did you think of Pete Resnick's draft about hums.

I thought well of it; I still do.  When Pete planned to write it, I
offered to co-author it.  But he said that once he got started, it all
just flowed out, and he wanted to present it as just his craziness, at
least at first.

It's not about hums, in particular, but about rough consensus, in
general, however it's reached.  The main point where Pete and I
disagree is whether raised hands are an appropriate replacement for
humming when you're "getting a sense of the room" -- I like hands, he
doesn't.  But we're completely in agreement about what to do with the
data.

Barry

---

From: <[email-masked]>
Date: Sun, 4 Aug 2013 06:32:49 +0100

Okay,

http://tools.ietf.org/id/draft-resnick-on-consensus-00.txt

can there be anything to discuss on not having consensus, developing two competing solutions in the same WG, and then having to downselect to one?

Lloyd Wood
http://sat-net.com/L.Wood/

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Barry Leiba [[email-masked]]
Sent: 04 August 2013 05:55
To: Scott Brim
Cc: SM; Andy Bierman; IETF discussion list
Subject: Re: 6tsch BoF

Barry

---

From: Randy Bush <[email-masked]>
Date: Mon, 05 Aug 2013 09:05:08 +0200

> What did you think of Pete Resnick's draft about hums.

i like it a lot and have used it in other fora which are somewhat loose
or confused about consensus.

randy