Subject: RE: Last Call: <C> (On Consensus and Humming in the IETF) to Informational RFC

From: Randy Presuhn <[email-masked]>
Date: Sat, 9 Nov 2013 08:40:20 -0800 (GMT-08:00)

Hi -

>From: Larry Masinter <[email-masked]>
>Sent: Nov 9, 2013 5:17 AM
>To: IETF Discussion <[email-masked]>
>Subject: RE: Last Call: <C> (On Consensus and Humming in the IETF) to	Informational RFC
>
>I wrote various things about rough consensus and the implementors
>and "those with implementations" in reaction to draft-resnick-on-consensus-06.
>
>After discussion and more thought, I wanted to retract what
>I said, and offer some specific comments on the draft.
>
>Retraction: 
>"Running code" means it's the implementations (not necessarily
>the implementors) that should carry the weight. Yes, for
>protocols to be implemented the implementors (eventually)
>have to agree, because otherwise the protocol doesn't get
>implemented.  But "running code" is really the code, and
>not just the opinion of those who might write the code. 
...

I have very mixed feelings about this.  Yes, it is the way
some working groups function.  However, implementors are
in the best position to comment on the how difficult it is
to implement the specification correctly, as well as how
elements of the specification contribute to (or reduce)
implementation complexity.  Both of these have very real
consequences for interoperability, security, and ensuring
a diverse ecosystem of independent implementations.  The
mere fact that someone was able to get something to run
as specified should not carry more weight than their
assessment of how much unnecessary pain or overhead (CPU,
memory, whatever) results from quirks of the specification.

Randy

---

From: Pete Resnick <[email-masked]>
Date: Sat, 9 Nov 2013 11:02:18 -0800

> memory, whatever) results from quirks of the specification.

Yup. I'll reply to the rest of Larry's message in a bit, but this part 
is complicated enough that I'm not sure I want to dive into it in the 
document. I agree with Randy that assessment of other factors (CPU, 
memory, whatever) is of course weighed against the "mere fact that 
someone was able to get something to run as specified", and if someone 
clueful calls this out, we tend to listen. But keep going with the 
example: If some smart person does call it out, the implementer might 
say, "Baloney! I got this implemented in 2K of code on an old 100Mhz 
embedded processor." And the smart person might respond, "Yeah, but on 
an embedded processor, you can play all sorts of games to get this 
running in that space; on a real processor on a general purpose piece of 
hardware, this is going to be a nightmare." And so on. At the end of the 
discussion, the chair (or whoever) is going to have to sort the wheat 
from the chaff: What's a showstopper and what's just a preference? Is 
the implementation cited just a one off and not representative of the 
type of implementation that the groups output is chartered to work on? 
Is the assessment that it's going to take too much CPU a wild guess or 
does it have some real backing? Do we need to look at some benchmarks? 
To figure these things out, hopefully the group will have come to its 
own consensus about the facts at issue, but if the consensus is going to 
be rough, the chair is going to have to make a call on what's legit. 
Blindly saying, "We're going with the running code" without seriously 
evaluating that running code is about as useful as saying "We're going 
with what Mary Smartimplementor claimed" without seriously evaluating 
the claim.

Engineering is hard. We've got to do that. Coming to consensus over the 
engineering tradeoffs is also hard; there's no one-liner like, "We 
always go with the running code". As the document says, the "running 
code" bit of the mantra is that we value actual practical experience 
over theoretical models. But running code is not itself some sort of 
magical thing. We still have to do the rest of engineering and come to 
consensus on whether we've gotten it "right". And I do think the 
document says that much. I don't think it should go much further.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Ted Lemon <[email-masked]>
Date: Sat, 9 Nov 2013 19:13:42 +0000

On Nov 9, 2013, at 2:02 PM, Pete Resnick <[email-masked]> wrote:
> Engineering is hard. We've got to do that. Coming to consensus over the engineering tradeoffs is also hard; there's no one-liner like, "We always go with the running code". As the document says, the "running code" bit of the mantra is that we value actual practical experience over theoretical models. But running code is not itself some sort of magical thing. We still have to do the rest of engineering and come to consensus on whether we've gotten it "right". And I do think the document says that much. I don't think it should go much further.

Nicely put, Pete (the rest, too).   This pretty much captures my discomfort with the "running code" line of discussion.

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 10 Nov 2013 12:25:55 +1300

> Nicely put, Pete (the rest, too).   This pretty much captures my discomfort with the "running code" line of discussion.

Yes. Depending on where a particular document or technology has got
to, we may seek rough consensus that the idea seems promising, or that
the specification seems complete and implementable, or that it has
reached widespread deployment and seems stable, just to mention three
out of many possibilities.

Rough consensus is our method, but running code is our target.

   Brian

---

From: Andy Bierman <[email-masked]>
Date: Sat, 9 Nov 2013 17:52:21 -0800

Hi Randy,

> memory, whatever) results from quirks of the specification.
+1

Early implementation and early design review both help improve
specifications.  People who have implemented similar solutions
but not the exact draft are qualified to gauge implementation pain.
IMO commentss about implementation complexity always
deserve careful attention.

Early implementation can be extremely expensive, so the people
who do them deserve lots of credit.  Often you have to prove
by writing the code "you really don't want to do it this way",
before the WG decides they messed up and over-engineered
an unworkable solution.

We just a long debate in NETMOD WG and decided to abandon
the SNMP ifType enumeration in favor of a YANG identity registry.
The key factor was the early implementation experience of
the ietf-interfaces module by 1 vendor.

Randy

Andy

---

From: Victor Kuarsingh <[email-masked]>
Date: Sun, 10 Nov 2013 21:24:44 -0800

A few comments.. Attached in line with previous comments.

On 2013-11-09 11:02 AM, "Pete Resnick" <[email-masked]> wrote:

[VK] I fully agree - engineering is hard, and goes much further then a
specific moment in time when we have our first or subsequent version of
code.  An important part of this is how the code/system runs in real
networks/environments and interoperates. Engineering is also a continuous
process in many cases.

I don't think we should enumerate the entire list of what trade-offs
should/can be made or what are all the considerations (list will likely be
very long).  Making clear note that such things are part of process should
be sufficient.  

On 2013-11-09 8:40 AM, "Randy Presuhn" <[email-masked]> wrote:
> "I have very mixed feelings about this.  Yes, it is the way
> memory, whatever) results from quirks of the specification."

[VK] I would agree that implementors are in a  good position to provide
feedback on various technologies and specifications.   When exposed to large
networks, we often see code/systems run against a wide range of other
systems and/or technologies which may result in validation or update
requirements to the running code.  I have seen many things which required
tweaking after it was in the wild.

From: Andy Bierman <[email-masked]>
"We just a long debate in NETMOD WG and decided to abandon
the SNMP ifType enumeration in favor of a YANG identity registry.
The key factor was the early implementation experience of
the ietf-interfaces module by 1 vendor."

[VK]  I think this is a great use case to exemplify the statements above.

[VK] Overall, I actually like the document and think it definitely helps the
reader (will often be a new person to the IETF) understand some of the
basics on how we reach consensus.  My take would be that someone reading
this draft, then experiencing the IETF process both on list and in person
(should they have that opportunity), would gain a good understanding of the
method (as run by capable contributors, working group chairs, ADs,
Directorates  etc)

Regards,

Victor K