Subject: Why have we gotten away from running code?

From: "Brian Rosen" <[email-masked]>
Date: Sat, 6 Aug 2005 19:07:19 -0400

I notice that we have stopped being interested in running code.

I think that is to our community's detriment.

If two groups are arguing with one another, and one has implemented code and
the other has not, I think we would give great weight to the running code.
I don't see that happening.  This happened in a session during this meeting
where I was present.  Running code was not considered significant in the
discussion; it was not even mentioned as a criterion in deciding the issue.

Probably more importantly, I think we should be VERY suspicious of new,
complex specifications before we have running code.  We are very clearly NOT
doing this.  We are willing to publish a proposed standard for an entirely
new protocol that has very significant complexity, where there are people
openly skeptical that it will work at all, with nothing but some sketchy
simulations and a (admittedly well reviewed) draft.  We are routinely
publishing complex protocols and significant changes/additions without even
simulations.

Our rules permit us to do such things.  We should rarely choose to.  We
don't know what we are getting into until we write code.  We don't know how
hard it is to implement, we don't know what works and what doesn't.

Perhaps there are a large number of you out there that are able to claim
reasonably complex things work based on reading a document and looking at
simulations.  I am not.  My experience is that getting something to actually
do what you want it to do is very illuminating.

I wonder if we should change our bias towards bestowing Experimental status
on drafts that ask to be published as RFCs without running code.

Clearly, there are specifications where the complexity is low, and we have
enough experience with the subject that we can be reasonably sure it works
without running code.  We should be able to bring such ideas out at Proposed
Standard.  Good judgment is always required to choose which side of a line a
particular draft falls on.  I assert we have pushed the line away from
running code quite too far.

We still do operate with rough consensus.  We ought to return to having
running code.

Brian

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sat, 6 Aug 2005 18:21:58 -0500

I agree with almost everything that Brian Rosen says in his note - the 
only thing that made me wonder, after Steve Bellovin talked at the IAB 
plenary about a crypto protocol that got blown twice in a 
specification that had only three message (message-equivalents? sorry 
if I misunderstood), that the definition of "complexity is low" may be 
a lot harder than we would have thought intuitively!

Spencer

> without running code.

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Sun, 07 Aug 2005 02:53:57 +0200

At 01:21 07/08/2005, Spencer Dawkins wrote:
>plenary about a crypto protocol that got blown twice in a specification 
>that had only three message (message-equivalents? sorry if I 
>misunderstood), that the definition of "complexity is low" may be a lot 
>harder than we would have thought intuitively!
>Spencer
>

Full agreement with both. With the additional remarks that "low complexity" 
may result from a lack of proper consideration of the charter (what should 
always be the first task of a WG) or from a lack of competence to see that 
complexity; and that great care should be given that the description of 
never ran code solutions are not labelled "BCP" just because their Draft 
claims to replace an RFC with a BCP number.
jfc

---

From: Scott W Brim <[email-masked]>
Date: Sun, 07 Aug 2005 07:23:19 -0400

> the other has not, I think we would give great weight to the running code.

Weight yes, but "great" weight?  Many things have been implemented
that only work in specific situations.  You're absolutely right that
running code should be considered, because it proves the idea
implemented can work, but it's just one factor.

> doing this.  

Yes

> We are willing to publish a proposed standard for an entirely

But that's specifically what "proposed" is for (currently).  "Here's
something we think we want to make a standard -- now test it".

> do what you want it to do is very illuminating.

See RFC 3439 :-).  Maybe you can't tell if something complex will
work, but at least you can reduce complexity as a factor in
determining whether it will work.

> on drafts that ask to be published as RFCs without running code.

Absolutely.

---

From: Henning Schulzrinne <[email-masked]>
Date: Sun, 07 Aug 2005 07:35:01 -0400

> something we think we want to make a standard -- now test it".

The problem with this notion is two-fold:

(1) Almost all protocols stay at "Proposed".

(2) The impact is particularly profound if there are multiple candidate 
protocols in a working group. If we had the model, "let's make all 
viable candidates Proposed Standards and then re-convene in a year to 
see which one worked best", there would be a basis to the deferred 
implementation experience.

In most cases, the chosen protocol works well enough, if necessary after 
an RFCxyz-bis round ("with enough thrust, anything flies"), but we don't 
seem to do a good job in using early implementation experience to guide 
our choice.

On the positive side, it should be noted that the NSIS working group 
just had a pre-IETF interop event with 5 implementations, even before 
working group last call on the main specs. From what I gather, these 
have been quite useful in confirming the implementability of the spec 
and in ferreting out some remaining issues. In that case, there were no 
competing protocol proposals, however.

Henning

---

From: John Leslie <[email-masked]>
Date: Sun, 7 Aug 2005 08:12:51 -0400

> I notice that we have stopped being interested in running code.

   Some of us, alas, seem to have lost interest in running code.

   :^( :^( :^(

> I think that is to our community's detriment.

   I could not agree more!

   (Of course, Brian is almost as old as I am; perhaps the latest
generation _believes_ that rough consensus can overrule running code...)

> If two groups are arguing with one another, and one has implemented
> code and the other has not, I think we would give great weight to the
> running code.

   There was a time when we'd tell the group without running code,
"Come back when you've got some code running!" That was, IMHO, a better
time.

> complex specifications before we have running code.  We are very
> clearly NOT doing this.  We are willing to publish a proposed standard
> for an entirely new protocol that has very significant complexity,
> where there are people openly skeptical that it will work at all,
> with nothing but some sketchy simulations and a (admittedly well
> reviewed) draft. 

   Unfortunately, we seem to have reached a situation in which there
is perceived to be no alternative starting point. :^(

   Here's another hint of what Ted Hardie talked about in NEWTRK: that
we must deal with ideas which are specifications, not standards; and
that "Proposed Standard" doesn't fit both categories very well.

> We are routinely publishing complex protocols and significant
> changes/additions without even simulations.

   I have no problem "publishing" such things; but I wish we wouldn't
publish them as if they were complete standards.

> Our rules permit us to do such things.  We should rarely choose to. 
> We don't know what we are getting into until we write code.  We don't
> know how hard it is to implement, we don't know what works and what
> doesn't.

   Actually, it's worse than that: often we don't know whether things
will work in practice until we have a year's experience of actual use.

> I wonder if we should change our bias towards bestowing Experimental
> status on drafts that ask to be published as RFCs without running code.

   I don't think there's much hope of changing the mindset that says
"Experimental" means "Don't use this!".

   But I believe we'd do well to establish a category for specifications
which may or may not be ready for large-scale trials, but do not qualify
for stable standards status.

   (I'll be happy to discuss this on NEWTRK, BTW, if anyone's interested.)

--
John Leslie <[email-masked]>

---

From: Melinda Shore <[email-masked]>
Date: Sun, 07 Aug 2005 08:46:56 -0400

Brian Rosen wrote:
> We still do operate with rough consensus. 

Probably only in the sense that some decisions are made
by a consensus process, but I'd guess that there's more
voting going on than not.  The lack of both rough
consensus and running code is something I've been wondering
about, too.  I think that it's probably more possible to
do something about the latter than the former.  Decision-
making when you've got a very large number of people involved
and in which there's sometimes radically different interest
in the outcome is a very, very difficult problem.

On the question of running code, I agree with you in
theory but we do have a problem with the timeliness of our
documents and I'm not sure that we want to make the process
even slower unless we're certain that there's a real
problem here that needs to be solved.

Melinda

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sun, 7 Aug 2005 07:53:39 -0500

>   (I'll be happy to discuss this on NEWTRK, BTW, if anyone's 

At least some are. The thread John Klensin started at 
http://darkwing.uoregon.edu/~llynch/newtrk/msg01313.html is exactly on 
this point, if I understand correctly.

Spencer

---

From: Jeroen Massar <[email-masked]>
Date: Sun, 07 Aug 2005 15:00:29 +0200

> I notice that we have stopped being interested in running code.

Not everywhere. For the IPFIX protocol, which is currently still in
draft status, there where 6 different implementations, both of collector
and meters, showing up at the Interop meeting, which took place just
before the IETF63. This helped a lot in finding a number of ambigueties,
which caused the document editors to revise/reword some parts of the
document and will also provide a lot of input, eg common
mistakes/misassumptions to the implementation guidelines.

Maybe there should be requirement that before having going to Last Call
there should at least be 2 separate implementations when a document is
created by a working group?

Greets,
 Jeroen

---

From: grenville armitage <[email-masked]>
Date: Sun, 07 Aug 2005 23:02:08 +1000

Melinda Shore wrote:

	[..]
> problem here that needs to be solved.

I wonder if absence of running code, and the apparently weakened
impact of running code on WG debate when there is some, is contributing
to drawn-out document development? Groups who do, or prefer, white-board
design will tend to spend more time tinkering with, or arguing over,
the theory of a design. Ironically, we might well speed things up
if design debates could, again, be swung by people who've actually got
implementation experiences to share.

cheers,
gja

---

From: Bill Fenner <[email-masked]>
Date: Sun, 7 Aug 2005 09:28:36 -0400

> created by a working group?

The Routing Area is debating having this rule.  Right now, the rules
laid down in RFC1264 include

   1) Documents specifying the Protocol and its Usage.  The
      specification for the routing protocol must be well written such
      that independent, interoperable implementations can be developed
      solely based on the specification.  For example, it should be
      possible to develop an interoperable implementation without
      consulting the original developers of the routing protocol.

The best evidence for "...possible to develop..." is, of course,
another implementation.

Our draft update for 1264 is draft-fenner-zinin-rtg-standard-reqts-00 .

  Bill

---

From: Melinda Shore <[email-masked]>
Date: Sun, 07 Aug 2005 13:43:08 -0400

grenville armitage wrote:
> to drawn-out document development? 

That's an excellent point.  To a great extent
we suffer from what the FreeBSD community calls
"bikeshed" 
(http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/misc.html#BIKESHED-PAINTING)
and while I think it's excellent that people are
providing input in areas not exactly their own
implementation experience can help filter discussion
and assist the decision-making process.  I'll note
that just because something's been implemented doesn't
mean it's a good design (or even a design at all), and
I wouldn't agree with "always prefer something that's
been implemented", but I do agree that there's no
substitute for implementation experience as one input
into the process.

Melinda

---

From: Mark Baugher <[email-masked]>
Date: Sun, 7 Aug 2005 12:07:45 -0700

I think there is much software publicly released by vendors for 
standards track protocols.  And there's a lot more protocol work being 
done by vendors than teams on public research grants.  I know 
personally that Brian Weis (RFC 3547) and David McGrew (RFC 3711) did 
outstanding implementations in recent years.  It takes a lot of 
patience to address legal and liability concerns.  Many people won't do 
it.  Small companies typically cannot afford to do it.

Mark
On Aug 6, 2005, at 4:07 PM, Brian Rosen wrote:

> code and
> clearly NOT
> status

---

From: Scott W Brim <[email-masked]>
Date: Sun, 07 Aug 2005 17:40:55 -0400

On 08/07/2005 13:43 PM, Melinda Shore allegedly wrote:
> and assist the decision-making process.  

Hi Melinda.  Are you saying that people shouldn't comment on an idea
unless they are implementing it?

swb

---

From: Melinda Shore <[email-masked]>
Date: Sun, 07 Aug 2005 17:58:59 -0400

> unless they are implementing it?

No, clearly (I hope) not.  Just that it seems likely
that maybe if we did more implementation it could
help end some of those round-and-round we go discussions
that can often be opinion-based.

Melinda

---

From: Scott W Brim <[email-masked]>
Date: Sun, 07 Aug 2005 18:12:41 -0400

> that can often be opinion-based.

For sure.  OK.  We always need more doers.

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Mon, 08 Aug 2005 00:32:38 +0200

Melinda,
Fully true. When you consider it, instead of talking of "running code" we 
should in fact talk of "used code". BCP are becoming the architectural key 
because they document the brainware: the way people use the technology to 
network together. Real experimentation needs to therefore be carried "live" 
on the Internet. ICANN called for this kind of experimentation by the 
global internet community, and defined its terms (ICP-3 document in 2001).

ICP-3 suggests that IETF shares in organising this experimentation. I 
repeatedly recalled it, but the comment I usually received was that it is 
not the role of IETF. We organised a test bed for two years with people 
from many places. Based upon this first experience, a new and more simply 
organised network test bed project is underway, open to all (Unitry). All 
the cooperations are welcome: the target is a mailing list of participating 
testing teams, with online resources and a charter accepted in common. It 
will be what the participants will make of it.
jfc

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 09 Aug 2005 15:16:15 +0200

> that can often be opinion-based.

Absolutely. "I tried to code this but couldn't understand
the spec" or "I coded this but it doesn't work as specified"
are two of the most powerful arguments and it would be good
to hear them (or their converse) more often.

The question on the table since RFC 3774 is: why don't we
execute the transition to Draft Standard more often,
otherwise known as: why are there so few implementation
reports at http://www.ietf.org/IESG/implementation.html
(three this year, I believe)?

Newtrk was expected to deal with this question.

     Brian

---

From: wayne <[email-masked]>
Date: Wed, 10 Aug 2005 02:46:57 -0500

> I think that is to our community's detriment.

I confess that while I've watched the IETF from afar for about a
decade, I am relatively new to actually doing anything in the IETF.  I
had always believed that "rough consensus and working code" were the
rules that the IETF lived by.

I was very surprised when participated in my first working group and
found that not only was working code was not required, but the
co-chairs and AD of the WG didn't think it was needed.

A few choice quotes from me about the lack of requiring working code:

   Without working code, we are just being a debating club.[1]

Another: [2]

    There doesn't appear to be a rough consensus [and the co-chair of the
    WG says likewise].  However, the long standing IETF mantra has, to the
    best of my knowledge, been "rough consensus *AND* working code".  What
    we lack with most proposals is working code.
    
    I think the lack of a rough consensus is in large part due to the lack
    of working code.  Working code quickly dispels both wishful-thinking
    and FUD.  Working code is a different way of expressing a proposal so
    disagreements and misunderstandings about a proposal can be cleared up
    by looking at the code and then either the code or the proposal can be
    fixed to make things clearer.
    
    The devil is always in the details.  As long as we continue to
    consider proposals that don't have working code, we allow people to
    nit-pick proposals that are complete, while glossing over the problems
    with proposals that exist on paper only.

The working group was shut down because no consensus could be
reached.  I think the lack of working code was one of the core causes
of the lack of consensus.

The list of messages that I could find where I stressed the importance
of working code: 

    Apr 06, 2004 http://www.imc.org/ietf-mxcomp/mail-archive/msg00762.html
    Apr 06, 2004 http://www.imc.org/ietf-mxcomp/mail-archive/msg00775.html

[1] Apr 22, 2004 http://www.imc.org/ietf-mxcomp/mail-archive/msg00994.html
       Without working code, we are just being a debating club.

    May 21, 2004 http://www.imc.org/ietf-mxcomp/mail-archive/msg01617.html
    
[2] Jun 15, 2004 http://www.imc.org/ietf-mxcomp/mail-archive/msg01982.html
    
    There doesn't appear to be a rough consensus [and the co-chair of the
    WG says likewise].  However, the long standing IETF mantra has, to the
    best of my knowledge, been "rough consensus *AND* working code".  What
    we lack with most proposals is working code.
    
    I think the lack of a rough consensus is in large part due to the lack
    of working code.  Working code quickly dispels both wishful-thinking
    and FUD.  Working code is a different way of expressing a proposal so
    disagreements and misunderstandings about a proposal can be cleared up
    by looking at the code and then either the code or the proposal can be
    fixed to make things clearer.
    
    The devil is always in the details.  As long as we continue to
    consider proposals that don't have working code, we allow people to
    nit-pick proposals that are complete, while glossing over the problems
    with proposals that exist on paper only.

    Jun 15, 2004 http://www.imc.org/ietf-mxcomp/mail-archive/msg01988.html
    
    A reply to the co-chair of the WG after he said that working code
    really isn't needed.
    
    Jul 01, 2004 http://www.imc.org/ietf-mxcomp/mail-archive/msg02476.html
    Jul 13, 2004 http://www.imc.org/ietf-mxcomp/mail-archive/msg02651.html

-wayne

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 10 Aug 2005 10:40:16 +0200

> I think that is to our community's detriment.

[...]

> complex specifications before we have running code.

I've been thinking about this on and off for a day, and I'm not  
convinced that having running code at the time a specification is  
first fleshed out would be all that helpful.

Can you point to any instance in recent IETF history (after 1995 or  
2000 or so) where having having running code early in the process  
would have made for a better _designed_ protocol?

Sure, trying to implement something brings out bugs in the  
specification, but those are usually relatively minor things that  
don't go to the design of the protocol. And wide deployment generally  
shows that a protocol could have been better in one regard or  
another, and that some parts of it don't turn out to be useful in  
practice.

However, waiting for implementations on account of the former only  
delays having a fixed spec even longer (we already see protocols  
_deployed_ before there is an RFC, which is very harmful IMO) while  
waiting for the latter leads to insanity such as RFC 1771 being stuck  
at "draft standard" even though it has dozens of interoperable  
implementations, a decade of operational experience and there  
wouldn't be an internet without it.

---

From: Love Hörnquist Åstrand <[email-masked]>
Date: Wed, 10 Aug 2005 11:14:30 +0200

> another, and that some parts of it don't turn out to be useful in

I don't agree, several IETF protocols that I've implemented while still
drafts have had major design changes done them because of an implementation
exposed serious flaws in them (secsh-gss, pk-init).

There are also specifications that would have been good to have
implementations before leaving the WG, because they are not
implemented-able as is (spkm).

Love

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 10 Aug 2005 12:34:41 +0200

> of the lack of consensus.

Don't be shy about naming names....

The MARID WG had one proposal (SPF) with running code and multiple 
implementations (I know nothing about whether the other proposals had 
implementations). It still failed to reach consensus.

It may say something about the value we place on running code, but I think 
it's hard to use it as an example of a situation with NO running code.

                     Harald

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 10 Aug 2005 15:53:07 +0200

> exposed serious flaws in them (secsh-gss, pk-init).

Hm, I'm not familiar with those. Still, for most of the protocols I'm  
familiar with I can't see how implementing them earlier would have  
changed the design. (Note that I haven't implemented any IETF  
protocols myself, but I did once do an implementation of a badly  
designed protocol.)

> implemented-able as is (spkm).

Is that because the designers did a bad job or because there was no  
way to anticipate the implementation difficulties?
_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Marc Manthey <[email-masked]>
Date: Wed, 10 Aug 2005 16:26:55 +0200

morning  experts,

> (Note that I haven't implemented any IETF protocols myself, but I  
> did once do an implementation of a badly designed protocol.)

ahhhh, is this why  you think that there is  no need  for any new  or  
old protocol at all ?

have a great day

marcM.

-- 
"Reality is what, when you stop believing in it, doesn't go away.
Failure is not an option. It is a privilege reserved for those who try."

European headoffice
of the  first in- official
cuseeme protocol
testing lounge
www.let.de

---

From: Jari Arkko <[email-masked]>
Date: Wed, 10 Aug 2005 17:49:39 +0300

> way to anticipate the implementation difficulties?

Protocol specs are like other pieces of engineering results:
they benefit greatly from testing and experience, particularly
if you've never created this specific type of entity before.

Sometimes what looks good on paper works but has nits and
missing details that need to filled out. In some other cases
there are more serious problems. In yet another cases there
are no issues.

I've seen many cases where supposedly well debated and
edited specifications had problems that could have been
detected by implementation experience. State transitions
or error conditions missed by specs written in prose rather
than in state machine form; security pixie dust that provides
a general solution but for which we cannot create a working
set of policy rules due to the nature of the application;
ambiguous requirements; lack of sufficient behaviour rules
to go with message formats; interactions between protocols
or layers that are hard to implement in commonly available
stack designs; optimizations that look good but turn out to
optimize an unimportant part of the problem; etc.

Obviously, I'm not advocating that we only listen to implementation
experience. A good spec combines input from multiple sources:
architetural insight, protocol design expertise, implementation
experience, marketing demand, different application areas, ...
You may also be able to trade one source to another one, but
not without limits.

--Jari

---

From: C Wegrzyn <[email-masked]>
Date: Wed, 10 Aug 2005 10:49:31 -0400

I think a big part of the issue is that the IETF has been taken over
little by little by corporate interests. Before it used to be for the
"love of doing it". Today it is more for "the benefit of one".

Chuck Wegrzyn

Marc Manthey wrote:

>------------------------------------------------------------------------
>

---

From: wayne <[email-masked]>
Date: Wed, 10 Aug 2005 10:23:41 -0500

> implementations). It still failed to reach consensus.

Of the half dozen or so input documents to MARID, both SPF and DMP had
running code.  The WG decided not to adopt either of those two
proposals and instead went with an untested system.  (Ok, at almost
the very last minute, some working code to support the PRA was
created, but no significant testing was done.)

> It may say something about the value we place on running code, but I
> think it's hard to use it as an example of a situation with NO running
> code.

There was NO running code for the PRA during most of MARID's
existence.

-wayne

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 10 Aug 2005 18:36:54 +0200

I think that is a good point.  A variation on that theme is that the
IETF is no longer run by people who actually implement protocols.  The
relevance and impact of the IETF on what is actually used on the
Internet is marginalized through that change of membership.  The
attitude of "That is not how we do things in the IETF" make people go
away.

Cheers,
Simon

---

From: Dave Singer <[email-masked]>
Date: Wed, 10 Aug 2005 16:55:30 +0000

I hear the opposite complaint enough to believe that the truth lies 
somewhere in between ("the ietf is dominated by academics who have no 
idea what it takes to design, deploy, and maintain large complex 
networks").  I only see a tiny portion of the ietf myself, agreed (I 
doubt many people see much more as it is so large), but I don't see 
reason to be excessively pejorative about the attendance I see.  It's 
mixed;  academics, industrial engineers, writers, thinkers, 
implementers, observers, dilettantes, all mixed in.  Just like other 
standards orgs.

-- 
David Singer
Apple Computer/QuickTime

---

From: Marc Manthey <[email-masked]>
Date: Wed, 10 Aug 2005 18:57:12 +0200

> attitude of "That is not how we do things in the IETF" make people go

hell o  , yes it is,

simon , all , see i am the  oppossite, a  i highly motivated  
undergraduated citizen.
with no money But i learn a lot in the last 2 year only by  reading  
lists and try to understand
what the real problem is or could be. I can´t do a lot,  just raise  
an issue
or have a meaning. But its  fantastic  to be  part of the whole thing.

just my 2 cents

marcM.

from old  germany;-P

-- 
" The mind, once expanded to the dimensions of larger ideas,  never   
returns to its original size."

Les Enfants Terribles
www.let.de

---

From: Lakshminath Dondeti <[email-masked]>
Date: Wed, 10 Aug 2005 10:23:52 -0700

My experience has been that implementations help improve the quality of 
the specifications, and formal security analyses help fix design 
errors.  I implemented two recent SEC area protocols, but unfortunately 
in both cases, my implementations were partial (due to lack of time, 
interest etc.,), and it appears that others' implementations were too 
(at least at the time we interoperated).  The other thing is that my 
counterparts and I were privy to some or all of the behind the scenes 
information.

The security analyses I am aware of (thanks to Cathy Meadows :-)) helped 
revise what is now RFC 3547 (and Russ H mentioned another example at the 
SAAG meeting in Paris).

The conclusion I draw is that a formal security analysis (or even 
protocol analysis) is very helpful once the protocol design has been 
completed to identify any errors, and implementations (preferably full 
implementations) would help improve the readability of specs (and that 
would happen if the implementer is basing the implementation only on a 
given I-D, and not on the history of it or behind the scenes info).  
Perhaps it might be worthwhile to identify "problem areas" in a spec and 
try and implement those parts to figure out whether we got them right.

best regards,
Lakshminath

---

From: C Wegrzyn <[email-masked]>
Date: Wed, 10 Aug 2005 14:25:09 -0400

>From my experience over the last 25 years I have seen the number go from
almost all "academics" (and some truly impressive geeks) to more a mix
like OSI The people that attend are there to represent the position of
their management (or manager) and their companies not look for the best
solution. The idea behind the code was that it would help flush away bad
ideas and help tighten up the language in the RFCs.

Hey, we not only had code that ran we also had "bake-offs" to make sure
all the stuff worked together. The idea was to work out the nuances (the
20% of the inaccuracies) and produce a damn good system. Today the idea
is to slap something together - damn the interop - and get out the door
for the "first mover advantage."  We also tend to not worry about the
experience of the user - we expect them to understand our "Gold" is more
like fool's gold than a well thought out and tested system.

Chuck

---

From: Jari Arkko <[email-masked]>
Date: Wed, 10 Aug 2005 21:54:50 +0300

>like fool's gold than a well thought out and tested system.
We still have bakeoffs. Or do you have some statistics that
show we have now less bakeoffs per RFC than we used to,
or that the bakeoffs are later than they used to be? ("We"
here is the internet community, the IETF doesn't hold
bakeoffs.)

--Jari

---

From: C Wegrzyn <[email-masked]>
Date: Wed, 10 Aug 2005 15:12:49 -0400

Perhaps they are more "regionalized". I know there are some "labs" like
at the UNH that hold them but the attendance isn't nearly as universal
as they once were.

As for statistics, no I don't have any. But I bet there aren't any more
-- in fact -- I would bet there are a lot less. I can't remember the
last SIP bake-off...

Chuck

---

From: Dave Singer <[email-masked]>
Date: Wed, 10 Aug 2005 19:16:56 +0000

Don't forget the organizations that adopt IETF specs.  ISMA has a 
regular interop and conformance program for RTSP + RTP + the codecs 
used, both 'virtual' over the internet and face to face at most 
meetings.  Likewise IMTC does testing of 3GPP SA4 multimedia specs, 
again using RTSP, RTP, codecs etc.  I'm sure there are more.

Not that I am opposed to running code, sample code, or more IETF 
bake-offs, mind you.  Just that the number of organizations filling 
the glass is more than it used to be.

-- 
David Singer
Apple Computer/QuickTime

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 10 Aug 2005 15:25:31 -0400

The next SIPit event is in about a month; see http://www.sipit.net/

There was a GIMPS (now GIST) + NSIS NSLP interop event just before the 
IETF meeting (pre-RFC).

I wish there were more, but there are some.

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 11 Aug 2005 10:33:52 +0200

> I wish there were more, but there are some.

The NSIS interop was co-located with IPFIX/NetFlow9 & NetConf, as can be
found at: http://www.ist-mome.org/events/interop/

The IPFIX interop resulted in a 30 min presentation/discussion during
the wg meeting and also solved quite a number of ambigues issues and
uncovered some things that many people might do wrong while making the
implementation. IPFIX is not in last call yet either, but having this
interop for sure improved the quality of the document a lot.

IMHO "Running Code" is a good thing, doing an interop before going to
last-call is even better. Thus, as mailed before, 2 seperate
interoperating implementations would be a good thing to have, not only
to take care of all the ambiguities, it also makes sure that the
document will actually be used and is not yet-another-paper...

Greets,
 Jeroen
 (who likes to actually implement things, not write about it ;)

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 11 Aug 2005 11:22:22 +0200

> implementers, observers, dilettantes, all mixed in.  Just like other 

I think that's right. However, what may well be missing in the mix
is input from people who actually deploy and operate our stuff, and
live with its limitations and quirks every day. We need to understand
the indirect consequences of our choices: not "can it be coded and will
it interoperate?" but "will it drive service providers and users crazy?"

     Brian

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 11 Aug 2005 12:23:32 +0200

On 11-aug-2005, at 11:22, Brian E Carpenter wrote:

> However, what may well be missing in the mix

Lack of a way to configure DNS resolvers automatically in IPv6  
continues to drive me crazy.

The arbitrary change from ip6.int to ip6.arpa did much the same.

I'm not convinced the IETF has learned anything here.

(Note that most of my income comes from keeping other people's  
networks running, but I also do some writing. The former isn't linked  
to my IETF activities, the latter is to some degree.)

---

From: C Wegrzyn <[email-masked]>
Date: Thu, 11 Aug 2005 06:31:27 -0400

That was one of the very best reasons for the bake-offs: you got to eat
your own dog food! There is nothing like trying to get your own software
(and hardware) to work! It tends to show all the bad decisions that were
made during the development.

Chuck Wegrzyn

---

From: Alexandru Petrescu <[email-masked]>
Date: Thu, 11 Aug 2005 16:09:54 +0200

> continues to drive me crazy.

I'm not sure I get it... DHCP distributes DNS resolvers, right?   I'm
not sure whether the DHCPv4 can distribute DNS IPv6 resolver addresses
or is that restricted to DHCPv6.  Would be nice if both did, just like
both dig/v4 and dig/v6 return both v4 and v6 addresses.  I think PPP
over IPv6 doesn't do it.

Otherwise RA could do it too, like this:

RA:
      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |     Type      |     Code      |          Checksum             |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     | Cur Hop Limit |M|O|  Reserved |       Router Lifetime         |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                         Reachable Time                        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                          Retrans Timer                        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |   Options ...
     +-+-+-+-+-+-+-+-+-+-+-+-

Type 6: DNS Recursive Name Server list

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |     Type      |    Length     |           Reserved            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                                                               +
      |                                                               |
      +                    DNS Server IPv6 addresses...               +
      |                                                               |
      +                                                               +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Type 7: Domain Search list

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |     Type      |    Length     |           Reserved            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                                                               +
      |                                                               |
      +     domain names to be searched, encoded as per RFC3315       +
      |                                                               |
      +                                                               +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Alex

---

From: Bill Manning <[email-masked]>
Date: Thu, 11 Aug 2005 14:06:39 -0700

> it interoperate?" but "will it drive service providers and users 
	thats -one- reason that DNSSEC has gestated these long months/years.
	operational feedback killed the first three attempts and may cripple 
the
	current version beyond repair.   Either we eat our own dog-food 
(presuming
	that the folks who design these things actual have networks entrusted 
to
	them to run this goop on (and no, an NS simulation does _NOT_ count), 
OR
	we have to find willing/gullible folks w/ the resources and interest 
in shaking
	down the crap that we pass off as implementation specs or $DIETY 
forbid,
	prototype code.

	brief summary - we should be the "service providers" and "users"
	of the stuff we design/build... if it drives you crazy or eats all your
  	remained time, then perhaps others less intimate w/ the ideas will
	be less than enthusiastic to embrace our ideas.

--bill

---

From: Bill Manning <[email-masked]>
Date: Thu, 11 Aug 2005 14:14:38 -0700

> both dig/v4 and dig/v6 return both v4 and v6 addresses.  I think PPP

	no you don't get it.   ask yourself, why is in-addr.arpa special?
	or, in the more modren wolrd...  where should the enum space
	be anchored,  e164.arpa,   e164.int,   e164.bti.gov.uk,
	or...  we.are.all.bozos.on.this.bus.

	this stuff is -HARDCODED- in the resolver libraries shipped with
	each end system.   the arbitrary change (after six years of deployed
	code) from ip6.int to ip6.arpa and -expecting- a mass change out of
	deployed code at zero cost to the folks "forcing" the change (guess who
	passed that change off on the unsuspecting?)  is enough to drive some
	endusers crazy... as well as service providers.

--bill

---

From: Alexandru Petrescu <[email-masked]>
Date: Thu, 11 Aug 2005 23:58:36 +0200

>     endusers crazy... as well as service providers.

Sounds like I've touched some sensitive issue on IPv6 and DNS
deployment... sorry about that.  Whoever passed that change off on the
unsuspecting - it wasn't me :-)

Alex

Alex

---

From: Ned Freed <[email-masked]>
Date: Thu, 11 Aug 2005 15:25:22 -0700 (PDT)

> it interoperate?" but "will it drive service providers and users crazy?"

I think there have been a lot of red herrings in this discussion, but this I
agree with completely.

I'm sure there are places in the IETF where implementation work prior to
standardization is wholly insufficient if not actually nonexistant. But this
doesn't generalize: There are also places where implementation work is the
norm. Consider the SIEVE working group as yet another example: This group has
10 active drafts, and I believe there are already multiple implementations of
all of them. (I have implemented 9 of them myself, as a matter of fact.)

While early implementation can help, it is no guarantee that the specification
is actually any good, especially if the people doing the implementing are also
the authors of the specification. In fact I would go so far as to say that
implementations by specification authors only protect against the most
egregious errors, errors that should be caught by any reasonable review. After
all, it is axiomatic that the authors know what they intended to say, which may
or may not be the same as what was actually said.

Moreover, as the world we develop our specifications for gets increasingly
complicated, our ability to predict how things will actually work out, which
was never all that great to begin with, is being eroded. This means we need to
look more and more at actual operational experience, not simple
implementability.

In short, I think bemoaning the lack of early implementations of our
specifications, whether true or not, misses the point almost completely. What
we need to be looking for is operational experience and better ways to
incorporate operational experience into our process.

And since most people are reluctant to try things operationally until there's
at least a published specification in place, one thing we need to be doing is
placing less emphasis on getting everything absolutely perfect for initial
publication and more emphasis on getting operational feedback and revising
things after that. And this, I regret to say, is pretty much the opposite of
where we seem to be heading.

				Ned

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 12 Aug 2005 10:16:42 +0900

Bill Manning wrote:

>     operational feedback killed the first three attempts and may cripple the
>     current version beyond repair.

Remember that the current DNSSEC protocol was, without much
discussion, chosen without running code, against a counter
proposal of mine with running code.

With the counter proposal, a lot of pitfalls not avoided by
DNSSEC was pointed out. There are a lot of subtlety in DNS
related to delegation, CNAME, wild cards and so on, none of
which was addressed by DNSSEC.

However, the pitfalls are ignored. Resulting implementations
were buggy, of course. The pitfalls are reconsidered and worked
around later only from operational experiences, which was a long
and painful experience.

With the demonstration of so miserable quality of the
specification and implementations, it is not surprising that
DNSSEC is not accepted at all by operators community.

But, I'm not saying running code is above all.

What's essential is not running code itself but acceptance
by the end users, imprecise proxy of which is acceptance by
operators, imprecise proxy of which is acceptance by
implementors, that is, running code, imprecise proxy of which
is IETF consensus, which means there is little point for IETF
to standardize protocols.

						Masataka Ohta

PS

It turns out that both the WG and I was wrong that DNSSEC is
not at all deployed is a good thing, because DNSSEC gives no
better security than so called weak security (If you can
trust CAs and their employees between you and your peer that
they won't sign forged public key of you unconsciously nor
maliciously, you can trust ISPs and their employees between
you and your peer that they won't route your packets to
someone else not having the destination IP addresses
unconsciously nor maliciously).

So, instead of introducing DNSSEC, just rely on ISPs and the
destination IP addresses and use 3 way handshakes with cookies
to securely confirm the source IP addresses are not forged.
ISPs are as reliable as CAs. If you think ISPs are not so
reliable, CAs neither.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 15 Aug 2005 09:55:36 +0200

Since nobody's mentioned the draft name yet, and we generally tell people 
"go read the drafts"....

  draft-ietf-dnsop-ipv6-dns-configuration-06.txt

Or - "there are 3 options. We can't pick one".

The document was approved by the IESG in July (but seems to be waiting for 
an IESG note).

                      Harald

--On torsdag, august 11, 2005 16:09:54 +0200 Alexandru Petrescu 
<[email-masked]> wrote:

> ljitsch van Beijnum wrote:

---

From: Margaret Wasserman <[email-masked]>
Date: Mon, 15 Aug 2005 08:57:09 -0400

People may also want to read RFC 3646 which defines DHCPv6 options to 
configure a DNS resolver.

We have considered _other_ ways to automatically configure a DNS 
resolver in IPv6, but we haven't managed to reach consensus on any of 
those proposals yet.

Margaret

At 9:55 AM +0200 8/15/05, Harald Tveit Alvestrand wrote:
>Since nobody's mentioned the draft name yet, and we generally tell 
>people "go read the drafts"....
>
>The document was approved by the IESG in July (but seems to be 
>waiting for an IESG note).
>
>                      Harald
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 15 Aug 2005 15:54:35 +0200

On 15-aug-2005, at 14:57, Margaret Wasserman wrote:

> People may also want to read RFC 3646 which defines DHCPv6 options  
> to configure a DNS resolver.

> resolver in IPv6, but we haven't managed to reach consensus on any  
> of those proposals

So you have reached consensus on forcing everyone who wants to look  
up DNS information over IPv6 transport to use DHCPv6?

> yet.

Ah.

---

From: Margaret Wasserman <[email-masked]>
Date: Mon, 15 Aug 2005 12:05:48 -0400

Hi Iljitsch,

>up DNS information over IPv6 transport to use DHCPv6?

As far as I know, nothing that we publish forces anyone to do 
anything (or not to do anything).

Margaret

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 15 Aug 2005 18:47:55 +0200

On 15-aug-2005, at 18:05, Margaret Wasserman wrote:

>> So you have reached consensus on forcing everyone who wants to  
>> look up DNS information over IPv6 transport to use DHCPv6?

> anything (or not to do anything).

But what's the alternative? If the only mechanism to discover IPv6  
DNS resolvers is DHCPv6, then people who want to query the DNS over  
IPv6 must either use DHCPv6 or configure those addresses manually.

Not reaching consensus is not an option in this case.

---

From: shogunx <[email-masked]>
Date: Mon, 15 Aug 2005 12:32:37 -0400 (EDT)

> anything (or not to do anything).

A nameserver should answer AAAA queries just as readily as A or MX's.

> Margaret

sleekfreak pirate broadcast
http://sleekfreak.ath.cx:81/

---

From: Lisa Dusseault <[email-masked]>
Date: Fri, 19 Aug 2005 17:32:06 -0700

> would have made for a better _designed_ protocol?

Yes -- in the IMAPEXT WG we run into this frequently.  In the past 
year, implementation experience has led people to make useful 
suggestions and find issues, as you can see on the mailing list Jun 1 
and Jan 10 for example.

Also the *lack* of server implementations of the IMAPEXT annotations 
draft is leading us to make better choices (I hope) about how 
annotations are designed and how much of the feature is required.  It's 
hard to notice a lack of implementations unless the regular situation 
is to have implementations. So I certainly appreciate early 
implementations.

Lisa

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Mon, 22 Aug 2005 00:18:42 +0200

At 02:32 20/08/2005, Lisa Dusseault wrote:
>suggestions and find issues, as you can see on the mailing list Jun 
>1 and Jan 10 for example.
>
>annotations are designed and how much of the feature is 
>required.  It's hard to notice a lack of implementations unless the 
>regular situation is to have implementations. So I certainly 
>appreciate early implementations.

May I suggest a slightly different formulation for a "running 
something culture": the needs seem to be architectural consistency 
and technical inclusiveness. Architectural consistency should be the 
respect of the Charter and technical inclusiveness is more a spirit, 
a culture to look what exist or is projected (as status of the future 
art) and try to accomodate it. This can be base on running code, 
current usage, common thinking, common analysis, etc.

This is why I insist for these two aspects to be included in the 
PROTO procedure. _every_ WG will have a problem with that two points 
because WG are not simple technical writers of IESG/IAB respecting 
100% of their Charter. This is precisely in the different between the 
intent and the delivery that one can measure the WG value-added 
(which as every change may be wrong) or possible bias - this may also 
occur (RFC 3774).

In this I presuppose that IAB is the ultimate reference: it review 
the Charter for consistency with the past and should ultimatey review 
the deliverable for consistency with the "future" (all what has been 
approved and is under practical implementation). I know some think 
they know better but I think this should be covered with the IAB 
itself. However I do not know the procedure.

jfc