Subject: Running Code

From: Marc Petit-Huguenin <[email-masked]>
Date: Tue, 03 Mar 2009 13:17:07 -0800

I would like to bring to your attention this proposal to put back
running code at the center of Internet protocol design by adding a
new Considerations Section in future Internet-Drafts and RFCs:

http://www.ietf.org/internet-drafts/draft-petithuguenin-running-code-considerations-00.txt

Thanks.

-- 
Marc Petit-Huguenin
Home: [email-masked]
Work: [email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 04 Mar 2009 10:54:42 +1300

Marc, and Henry,

I think adding any new mandatory section to all I-Ds is a bad idea.
It will quickly become bureaucratic. We've had proposals for mandatory
Management Considerations, IPv6 Considerations, and no doubt others
that I've forgotten, and they all have the same problem.

However, I think it's a very good idea to offer *guidelines* for what
should be in technical specifications in this area. In fact, my old
commentary on RFC2026 talked about related issues concerning
interoperability criteria for promotion to Draft Standard.
See the comments on "4.1.2 Draft Standard" in
http://tools.ietf.org/id/draft-carpenter-rfc2026-practice-00.txt
Obviously, the first stage in interoperability is interoperability
with yourself ;-).
(As far as I am concerned, you are welcome to use any of that
material under RFC5378 conditions.)

I encourage your draft to become purely a set of guidelines.
That would be useful and non-bureaucratic.

    Brian

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 03 Mar 2009 15:31:41 -0700

> http://www.ietf.org/internet-drafts/draft-petithuguenin-running-code-considerations-00.txt

It might be helpful to define what running code is. I've been meaning to
write an I-D about that but haven't gotten to it yet...

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Tue, 03 Mar 2009 14:43:25 -0800

> that I've forgotten, and they all have the same problem.

I agree that its sounds bad when presented like this.

The main motivation is to provide an incentive for early
implementations of a protocol, because I am convinced that this is a
very important factor on the quality of a protocol.  I had to
implement at least three times TURN from scratch during it's
development and this is an exhausting task.  This explain why a lot
of developers simply wait for the protocol to be stable (read: been
published as an RFC), and so deprive the protocol design of an
important feedback.  Giving to early implementers a guaranty that
their contributions will not be forgotten is a way to counterbalance
the time and effort spent in working on this contributions.

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 04 Mar 2009 00:42:02 +0100

> http://www.ietf.org/internet-drafts/draft-petithuguenin-running-code-considerations-00.txt
There used to be a term for those who attempted to manipulate the shape 
of the universe by manipulating the names in the Usenet hierarchy.

I get the same impression from people who want to manipulate IETF 
behaviour by manipulating the shape of Internet-Drafts.

(and - going into nits mode - the idea that only implementations that 
can be dowloaded from a public URL "count" as implementations flies into 
the face of long-term IETF policy.)

                       Harald

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Tue, 03 Mar 2009 17:19:17 -0800

> behaviour by manipulating the shape of Internet-Drafts.

I do not see how you can have this impression, as the I-D does not
try to make implementations mandatory for Internet-Drafts - _that_
would be changing the IETF behavior.  What the document says is that
early implementers efforts should be rewarded by listing their name,
sponsors and access to their code as a thank you for helping improve
the protocol.  That does not change the IETF behavior - at best that
will change the quality of IETF protocols.

---

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 3 Mar 2009 20:36:43 -0500

I admit that I'm no friend of additional I-D sections, as they easily  
generate into boilerplate and "make work" projects. If the goal, which  
does not seem stated, is to acknowledge the contributions of  
implementations in improving a standards document, we already have a  
mechanism for that, namely the customary acknowledgements found in  
most RFCs. I don't think anybody would object to saying something like

"The authors gratefully recognize the efforts of Joe Programmer, whose  
XYZ implementation of early versions of the draft helped to remove  
useless crud from the spec."

[well, maybe not quite verbatim like that.]

We can never hope to acknowledge all implementations in any event; for  
example, many are done by students in classes, and are never released  
(and that's a good thing...). It seems much more useful to capture  
implementations on WG-related web pages; after all, the value of  
implementations does not step when the I-D hits the RFC editor's desk.

We also certainly don't want to put yet more hurdles into the path of  
getting drafts published. Does the RFC editor have to verify the URLs  
and that they still exist? Do we worry about advertising pages and  
implementations that turn out to be malicious? I'd really rather not  
have to deal with an RFC where a domain of a fledgling open-source  
project was taken over by a malware distributor.

Henning

> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
>

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Tue, 3 Mar 2009 17:51:31 -0800

I don't see the value of running code quite as others do.

For me the value of running code is that the requirement to actually implement stuff does tend to reduce the scope for complexity, you have someone in the room pushing against something that will make work for them. And the other advantage is that there tends to be a closer relationship to actual real world needs.

But you do not have to do A to get B and doing A does not guarantee that you get B.

Another alternative is to require people to produce a proof of correctness for their protocol. That provides even greater encouragement to be concise and to get it right the first time. 

The running code strategy can also backfire. I have seen groups where one party has a large development team on call that allows them to drive the specification. And I have also seen groups where no progress can be made because the programmer who wrote the dufus code won't allow the dufus to be deleted from the spec. Coding too early can also be a problem.

-----Original Message-----
From: [email-masked] on behalf of Brian E Carpenter
Sent: Tue 3/3/2009 4:54 PM
To: Marc Petit-Huguenin
Cc: [email-masked]
Subject: Re: Running Code
 
Marc, and Henry,

    Brian

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 04 Mar 2009 11:15:39 +0900

> I don't see the value of running code quite as others do.

I agree. It was valuable in good old days, when implmenting a protocol
was purely voluntary with no budget.

Existence of multiple independent implementations, then, meant the
protocol was widely accepted by the society.

However, after the overly introduction of standardization engineering
to IETF, people satisfy requirements merely because they are required.

So, existence of required running code does not mean much.

						Masataka Ohta

---

From: "Gary E. Miller" <[email-masked]>
Date: Tue, 3 Mar 2009 18:35:25 -0800 (PST)

Yo Masataka!

> So, existence of required running code does not mean much.

Except a basic proof of real functionality and that is valuable.

RGDS
GARY
- ---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97701
	[email-masked]  Tel:+1(541)382-8588

---

From: Andy Bierman <[email-masked]>
Date: Tue, 03 Mar 2009 18:50:24 -0800

> So, existence of required running code does not mean much.

I disagree.
It means the specification is implementable.

Since the goal of our work is to produce specifications
that will allow multiple independent implementations to
inter-operate successfully, I can think of no more valuable
review input towards this goal than comments from implementers.

I think adequate procedures exist for gathering implementation
experience for the IESG to evaluate protocol interoperability.

> 						Masataka Ohta

Andy

---

From: [email-masked]
Date: Tue, 03 Mar 2009 19:35:28 -0800 (PST)

> would be changing the IETF behavior.

On the contrary, that's exactly what it does. Quoting from the draft:

   The "Running Code Considerations" section MUST be present in all
   Internet-Draft and SHOULD be inserted after the Security
   Considerations and IANA Considerations sections.  This section MUST
   be present even if the document does not describe an implementable
   protocol and should contain in this case a text explaining why this
   section is irrelevant.  The RFC Editor can remove this "Running Code
   Considerations" section before publication as RFC.

I'm already on record as oppossing the addition of such bureaucratic folderol
in other cases.

And while I'm a big believer in running code and always try to implement what's
described in my drafts before calling them complete, I think this proposal is
an absolutely terrible idea.

> What the document says is that

You need to read the draft again because that is not what it says.

				Ned

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 04 Mar 2009 13:08:09 +0900

> It means the specification is implementable.

If a protocol is so complex that its implementability is not
obvious, you have lost from the beginning.

> inter-operate successfully,

How can you define successful interoperation of implementations?

> experience for the IESG to evaluate protocol interoperability.

Such formalism has killed IETF.

To formally confirm that multiple implementations of a protocol
interoperate, which is required these days, you really need to
have a formal specification of a protocol, which, if any, is very
complex even if an informal specification of the protocol is simple.

If all you want is informal and vague feeling of interoperability,
it is not a very useful review.

							Masataka Ohta

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Tue, 03 Mar 2009 20:41:13 -0800

Henning Schulzrinne wrote:
> mechanism for that, namely the customary acknowledgements found in most
> RFCs. I don't think anybody would object to saying something like
> 

That's a good point.  The reference to the code is needed only until
the publication as RFC, so an URL does not need to be valid after
this - the subsequent need for two implementations for Draft
Standard is a completely different problem that have nothing to do
with my proposal.  So I will modify the draft to say that the URLs
have to be removed before publication as an RFC.

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Tue, 03 Mar 2009 20:54:44 -0800

>    Considerations" section before publication as RFC.

A "Running Code Considerations" section can be empty, this is the
reason of the last sentence (this is similar to what is done for the
IANA Consideration Section).  If a protocol described in an I-D has
no implementation then the section is empty, and people can decide
to invest in this protocol using this information.  This to say,
again, that the text does not implies that implementations are
mandatory, just that their existence must be documented in the I-D.

> You need to read the draft again because that is not what it says.

OK, I do not know what to say here.  I read again the text, and
nowhere it is said that having an implementation is mandatory.
Documenting eventual implementations would be.

---

From: "Hannes Tschofenig" <[email-masked]>
Date: Wed, 4 Mar 2009 09:53:04 +0200

>We also certainly don't want to put yet more hurdles into the 
>path of getting drafts published. Does the RFC editor have to 
>verify the URLs and that they still exist? Do we worry about 
>advertising pages and implementations that turn out to be 
>malicious? I'd really rather not have to deal with an RFC 
>where a domain of a fledgling open-source project was taken 
>over by a malware distributor.
>
>Henning

I would like to provide one recent example. In the EMU working group we
worked on a protocol, called EAP-GPSK http://www.ietf.org/rfc/rfc5433.txt. 
The work was done in a design team, it took a very long time (the first
design team draft dates back to May 2006). 

Jouni Malinen implemented the protocol in 8 hours! Jouni also provided a
number of suggestions and we put him into the acknowledgments section.
Before the draft got published as an RFC the reference to his implementation
was removed by the RFC Editor. The RFC Editor also verified the URL and it
was not correct anymore (but that's another topic). 

As mentioned in
http://www.ietf.org/internet-drafts/draft-tschofenig-rai-reducing-delays-00.
txt the problem with feeding experience with running code back into the
specification work is elsewhere.

There are three main problems: 

* Almost random changes to the specification make early implementation work
almost useless (if your goal is to produce code that aims having code for a
specific RFC after all). Since it takes so long to finish the
standardization work it is often not possible to wait till the RFC comes
out. 

* Nobody cares if you have running code. Requests to substantially change
the specification often come at a late stage. Even IESG members have already
re-written specifications during IETF Last Call. As a joke, I suggested to
just submit the table of contents to the IESG and then start the work there.

* Finally, because it takes so long to finish specifications the 3-level
Standards Track process is rarely utilized anymore. That process considers
interoperable code but what does it help? 

Ciao
Hannes

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 04 Mar 2009 10:15:15 +0100

> http://www.ietf.org/internet-drafts/draft-petithuguenin-running-code-considerations-00.txt

I like the idea. However, I'd put some comments:

* Acknowledgments already have their section, thus it would be 
preferable to use that to mention any authors and sponsors who 
contributed code for a specific protocol. There is no need to remove 
such acknowledgments upon publication.

* Obsolete code is useless. Anyone interested in knowing what code was 
available for version 00 can browse that version of the draft, or the 
historical sections at any referenced URLs.

* It may be worth mentioning development problems, implementation 
strategies, compatibility or integration issues, test suites, and the 
like. Should that section be the preferred place for discussing 
implementation issues, quoting code snippets while pointing to 
complete projects?

* IMHO, rather than being a throw-away section, Running Code 
Considerations should evolve into sections possibly published in the 
final RFCs, for two types of content: (i) mentions of established 
packages that implement the (final) protocol, and (ii) any guidelines 
and caveats for implementors.

Just my 2p

---

From: Lars Eggert <[email-masked]>
Date: Wed, 4 Mar 2009 12:02:53 +0100

Hi,

> that I've forgotten, and they all have the same problem.

agree with Brian.

Also, RFC4858 already recommends that the "Document Quality" section  
of the document announcement that is emailed out when the IESG  
approves a document describe implementation status. So it's not like  
we don't have a way in which to document implementation status; we  
simply don't do it very often.

Lars

---

From: Jari Arkko <[email-masked]>
Date: Wed, 04 Mar 2009 13:23:08 +0100

Hannes,

> Jouni Malinen implemented the protocol in 8 hours!

Not a good spec time / implementation time ratio! There are also 
examples of people starting and *finishing* their PhD on the topic of an 
RFC *before* the RFC comes out. Not a good sign of the effort required 
to publish an RFC...

Picture about the process for this particular draft can be found in 
http://www.arkko.com/tools/lifecycle/draft-ietf-emu-eap-gpsk-timing.html 
-- what can we do to make the process smoother?

> interoperable code but what does it help? 
These are issues. And my opinion is that we have to take implementation 
experience very seriously. I would like to disagree with you on the 
assertion "nobody cares if you have running code", however. I think all 
of us care. But running code does not necessarily override ALL other 
concerns. If there's a serious bug in the spec, it needs to be fixed, 
even if it is noticed late in the process. Personally, I find that we 
waste most of the time waiting (for reviews, for the author to revise a 
spec, for the AD to respond, etc). If we reduce that we can get the 
process much faster.

The entire value of the IETF specification effort is that you get 
comments and as a result the specification improves. That necessarily 
implies that there may be changes from your initial implementation. If 
the  goal is absolute minimum publication time and no changes to 
implementation, we should all just be publishing protocol specifications 
on our web pages or talking to the RFC Editor -- and this is what we do 
in many cases, but it is not the right approach for producing a standard.

Jari

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 4 Mar 2009 09:43:00 -0500

Dear colleagues,

> http://www.ietf.org/internet-drafts/draft-petithuguenin-running-code-considerations-00.txt

I oppose this draft, on the following grounds:

1.  It adds yet another required section to I-Ds.  If we have not
already passed it, we are certainly approaching the point at which the
required sections and boilerplate make up more of the document than
the substantive parts in a short draft.

2.  It imposes a requirement that is impossible to guarantee one has
satisfied: it requires that all implementations MUST be listed.  I am
personally familiar with at least one case where an early
implementation was completed "in house" and scrapped without telling
anyone it had been done.

3.  It implicitly requires that the running code be publicly
available.  This is contrary to the traditional IPR agnosticism of the
IETF.

I understand the point of the draft, and I think the goal is laudable.
But if we want to encourage early implementations, running code, and
interoperability tests, that goal will not be served by making the
drafting process more bureaucratic (or indeed by generally adding more
process rules).  I think others have made these points in their
remarks, too, so you can just add my voice to the chorus.

Best regards,

Andrew

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: [email-masked]
Date: Wed, 04 Mar 2009 06:42:21 -0800 (PST)

> mandatory, just that their existence must be documented in the I-D.

I'm talking about the mandatory nature of the section. What the seection says
is irrelevant. More mandatory sections are bad and that's exactly what this
proposal calls for.

If a draft author wants to describe implementation work and how it has helped
produce the document, that can be done in the acknowledgements section. So,
while I entirely support the development of codde in parallel with
specifications, I am totally opposed to ideas put forward in this draft. The
absolute last thing we need is ANYTHING that makes getting documents through
the process more difficult. We have too much piled on crap as it is.

				Ned

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 04 Mar 2009 07:52:37 -0700

> How can you define successful interoperation of implementations?

IMHO you define it by "running code" -- that is, code which is used to
run a functioning communications network. For me the canonical example
is the medium we're using right now: email. In general (there are always
exceptions!), you don't know or care what email clients people use, what
email servers they use, whether they retrieve their email using POP or
IMAP, etc. It just works, at least for the core use cases. And I think
that's why running code (not just compiling code or functioning code,
but a working network) is so important.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: "Hannes Tschofenig" <[email-masked]>
Date: Wed, 4 Mar 2009 16:58:43 +0200

Hi Jari, 

>Hannes,
>
>> The work was done in a design team, it took a very long time (the 
>> first design team draft dates back to May 2006).
>>
>Not a good spec time / implementation time ratio!
Nope. That's also what I thought. 

> There are 
>also examples of people starting and *finishing* their PhD on 
>the topic of an RFC *before* the RFC comes out. Not a good 
>sign of the effort required to publish an RFC...
>

2 years and 8 months -- super fast in comparison with other documents. 

>-- what can we do to make the process smoother?

Good that you ask (and this was not arranged). Here is the draft with
throughts from Henning, Markus and myself: 
http://www.ietf.org/internet-drafts/draft-tschofenig-rai-reducing-delays-00.
txt

>> * Almost random changes to the specification make early 
>implementation 
>> work almost useless (if your goal is to produce code that 
>aims having 
>> code for a specific RFC after all). Since it takes so long to finish 
>> the standardization work it is often not possible to wait 
>till the RFC 
>> comes out.
>>
>> * Nobody cares if you have running code. Requests to substantially 
>> change the specification often come at a late stage. Even 
>IESG members 
>> have already re-written specifications during IETF Last Call. As a 
>> joke, I suggested to just submit the table of contents to 
>the IESG and then start the work there.
>>
>> * Finally, because it takes so long to finish specifications the 
>> 3-level Standards Track process is rarely utilized anymore. That 
>> process considers interoperable code but what does it help?
>>   
>These are issues. And my opinion is that we have to take 
>implementation experience very seriously. I would like to 
>disagree with you on the assertion "nobody cares if you have 
>running code", however.

Maybe this is just reflecting a bit my frustration. 

My recent example: 
http://www.ietf.org/mail-archive/web/emu/current/msg01111.html

Timeline for that document: 
http://www.arkko.com/tools/lifecycle/draft-ietf-geopriv-radius-lo-timing.htm
l

> I think all of us care. But running 
>code does not necessarily override ALL other concerns. If 
>there's a serious bug in the spec, it needs to be fixed, even 
>if it is noticed late in the process.

I agree. It is certainly not black and white. 

> Personally, I find that 
>we waste most of the time waiting (for reviews, for the author 
>to revise a spec, for the AD to respond, etc). If we reduce 
>that we can get the process much faster.
>
>The entire value of the IETF specification effort is that you 
>get comments and as a result the specification improves. That 
>necessarily implies that there may be changes from your 
>initial implementation. If the  goal is absolute minimum 
>publication time and no changes to implementation, we should 
>all just be publishing protocol specifications on our web 
>pages or talking to the RFC Editor -- and this is what we do 
You are certainly correct. 

I am certainly not asking for instant publication of everything. I would,
however, like to reduce the publication delay from 5+ years down to
something more reasonable. 

Ciao
Hannes

---

From: Andy Bierman <[email-masked]>
Date: Wed, 04 Mar 2009 07:11:48 -0800

>> How can you define successful interoperation of implementations?

You gather implementation reports.
You conduct interoperability tests and bake-offs.
This used to happen a lot more, back when advancing
to Draft or Full Standard was considered important.

Andy

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Wed, 04 Mar 2009 08:09:18 -0800

Henry Sinnreich wrote:
>  Brian,
> 
> Having running code only as a guideline has not served the IETF well
> lately, since it is largely ignored.
> 
> I am still cringing during the IETF SIMPLE meetings when we use Jabber
> IM that has the code free and available.
> 
> Would the SIMPLE WG have had the mandatory requirement of running code,
> SIMPLE would be much further ahead.
> 

Which remind me of this quote from A. Lyman Chapin (in [1]):

"It didn't take long to recognize the basic irony of OSI standards
development: there we were, solemnly anointing international
standards for networking, and every time we needed to send
electronic mail or exchange files, we were using the TCP/IP-based
Internet!"

[1] Malkin, G., "Who's Who in the Internet: Biographies of
    IAB, IESG and IRSG Members", RFC 1336, May 1992.

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Wed, 04 Mar 2009 08:59:24 -0800

Hallam-Baker, Phillip wrote:
> For me the value of running code is that the requirement to actually
> implement stuff does tend to reduce the scope for complexity, you have
> someone in the room pushing against something that will make work for
> them. And the other advantage is that there tends to be a closer
> relationship to actual real world needs.
> 
> Another alternative is to require people to produce a proof of
> correctness for their protocol. That provides even greater encouragement
> to be concise and to get it right the first time.
> 
> 
> The running code strategy can also backfire. I have seen groups where
> one party has a large development team on call that allows them to drive
> the specification. And I have also seen groups where no progress can be
> made because the programmer who wrote the dufus code won't allow the
> dufus to be deleted from the spec. 

Yes, there is text in the document saying that having early
implementations should not prevent to break compatibility between
two versions of an Internet-Drafts

> Coding too early can also be a problem.

This is the main problem.  Nearly everybody agree that early
implementations are good for the development of a protocol.  It's a
fact that early implementations are frustrating for developers so
very few do it.

The proposed solution (mandatory acknowledgement in I-D and RFC) has
been rejected on every comment made in this mailing-list, so this
will not happen.  So what is the solution to encourage developers to
do early implementations?

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Wed, 04 Mar 2009 09:17:10 -0800

> the process more difficult. We have too much piled on crap as it is.

It seems that this is the consensus.

Now, I know by experience that even significant contributions to an
I-D does not guarantee you a place in the acknowledgement section.
So what is the incentive into developing code that 1) will probably
be obsoleted by the next version of the I-D and 2) will not be
acknowledged at all in contributing to the improvement of the protocol?

As I understand it, it is considered very offensive in the academic
world to not properly cite sources.  It should be the same for early
implementation when designing a protocol.

---

From: Henry Sinnreich <[email-masked]>
Date: Tue, 3 Mar 2009 17:41:27 -0800

Brian,

Having running code only as a guideline has not served the IETF well lately, since it is largely ignored.

I am still cringing during the IETF SIMPLE meetings when we use Jabber IM that has the code free and available.

Would the SIMPLE WG have had the mandatory requirement of running code, SIMPLE would be much further ahead.

Henry

On 3/3/09 4:43 PM, "Marc Petit-Huguenin" <[email-masked]> wrote:

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 4 Mar 2009 12:44:26 -0500

I think the developer should be acknowledged if they indeed  
contributed to the spec development. (Many implementations are done  
well outside the IETF, with essentially no feedback loop.) If they are  
not, this seems like a behavior for the WG chair to encourage.

We need to recognize that the value of implementations differs greatly  
depending on the nature of the specification. They are very valuable  
for new protocols and higher-risk efforts (e.g., new algorithms), but  
less so for the more routine maintenance activities, such as adding a  
DHCP option.

I don't see how making implementations mandatory helps, as Henry seems  
to be advocating. We can't force spec writers to code and we can't  
force those outside the IETF to implement our drafts. Thus, all we'd  
get is further delay, as we wait for this gating condition, or specs  
that are dropped in the early stages. It would, however, likely mean  
that only entities with more resources, who can more easily commandeer  
a developer for their draft to do a token implementation, get to write  
the specs.

We should encourage implementations, e.g., by having implementors  
speak during WG meetings or by hosting interoperability events, plug  
fests, code spurts and similar activities around the IETF, as well as  
by eating our own dog food.

Henning

---

From: Melinda Shore <[email-masked]>
Date: Wed, 04 Mar 2009 13:20:37 -0500

> acknowledged at all in contributing to the improvement of the protocol?

I tend to assume people will be interested in a protocol
for some other reasons than garnering acknowledgements
and fluffing their resumes, and those other reasons will
presumably be sufficient motivation.  Implementing something
will tend to be a pretty good way to find bugs, inefficiencies,
or other problems with a protocol specification.  If you're
interested in kudos, take the issues you find to the mailing
list rather than directly to the author.

I'm pretty surprised by this argument.

Melinda

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Wed, 04 Mar 2009 10:42:32 -0800

> I'm pretty surprised by this argument.

I assumed that acknowledgement would be a good enough incentive for
developers to contribute early implementations, but you seem to
think that there would be other reasons.  The fact is that feedback
from early implementations is rare, so what other reasons do you
think early implementers would have?  Cannot be money - early
implementations are very likely to become obsolete at the next
version of the I-D, and so have to be rewritten.

---

From: Melinda Shore <[email-masked]>
Date: Wed, 04 Mar 2009 14:28:20 -0500

On 3/4/09 1:42 PM, "Marc Petit-Huguenin" <[email-masked]> wrote:
> think that there would be other reasons.

Right.  My experience has been that people will work
on a protocol because it's necessary for work, or a
personal project (academic), or some other reason than
they're looking for acknowledgement.  I'm a little
startled, frankly, to hear that people are implementing
internet drafts to get their name out there, or that
they're doing early implementations without participating
in the working group.  How common is this?

I think this proposal is generally a very bad idea but
if there's a genuine problem with inadequate incentives
for implementation hopefully that's something that can be
dealt with as a process issue rather than a document
issue.  I'd be curious to see some evidence that there's
a more general problem here.

Melinda

---

From: "Hannes Tschofenig" <[email-masked]>
Date: Wed, 4 Mar 2009 21:32:49 +0200

Melinda, 

folks in the research community (e.g., students) need some publications and
being mentioned in documents is quite important for them. 

>> I-D does not guarantee you a place in the acknowledgement section.
If the authors don't like you then you don't end up there.
OR
If the authors are lazy then they just forget to add you. 

In general, I find it important to list those who have spent their time with
a specification. Sometimes it appears that reviewers spend more time with
the specification than the authors. 

Ciao
Hannes 

>From: [email-masked] [mailto:[email-masked]] On 
>Behalf Of Melinda Shore
>Sent: 04 March, 2009 20:21
>To: Marc Petit-Huguenin; Ned Freed
>Cc: Harald Alvestrand; [email-masked]
>> So what is the incentive into developing code that 1) will 
>probably be 
>> obsoleted by the next version of the I-D and 2) will not be 
>I tend to assume people will be interested in a protocol for 
>some other reasons than garnering acknowledgements and 
>fluffing their resumes, and those other reasons will 
>will tend to be a pretty good way to find bugs, 
>inefficiencies, or other problems with a protocol 
>specification.  If you're interested in kudos, take the issues 
>you find to the mailing list rather than directly to the author.
>

---

From: [email-masked]
Date: Wed, 04 Mar 2009 11:25:27 -0800 (PST)

> > I-D does not guarantee you a place in the acknowledgement section.

If that's indeed the case then that's a problem independent of this proposal.
Failure to properly acknowledge significant contributions is wrong no matter
how you slice it.

That said, I have on more than one occasion gotten requests from someone that
their name be removed from an acknowledgements section. In fact in one case I
was tempted in one case to replace the name with Alan Smithee, but on further
consideration I decided that was a bad idea...

> presumably be sufficient motivation.

And if you do an implementation of a protocol nothing prevents you from listing
that fact on a resume. I don't see why mention in an RFC would be a prequiisite
for this. And if authors are required to list all implementations per this
proposal independent of whether or not that work actually contributes to the
specification, the value of such citations drops to almost nothing anyway.

> Implementing something

+1

> I'm pretty surprised by this argument.

+1

				Ned

---

From: [email-masked]
Date: Wed, 04 Mar 2009 11:45:46 -0800 (PST)

> from early implementations is rare,

THat may be true in your neck of the woods, but not in mine. I try and do at
least a preliminary implementation for every specification I write (the notable
exception for me on this was RFC 2231, and boy do I wish I had done one in that
case), and I routinely receive notes from other implementors saying they've
iimplemented some draft of mine long before publication or even WG last call.

> so what other reasons do you

Again, if there's a problem with people not getting proper recognition for
having contributed to a specification, irrespective of whether that
contribution is based on implementation work or just reading the specification,
that needs to be addressed independently of this proposal. It is wrong to take
credit for someone else's work.

				Ned

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Wed, 04 Mar 2009 13:03:52 -0800

OK, so nearly everybody seems to think that I misunderstood the
motivations of early implementation contributors, so let's ask them
directly.

If you did contribute an early implementation or did think of
contributing but finally didn't, please respond to this email with
your story.  Interesting points are why you did (or not) the early
implementation, will you do more, what would motivate you to do more
early implementations, etc... You can send your responses directly
to me if you do not want to respond publicly - I will keep them
confidential and post just a summary of the responses.

For the purpose of this exercise, an early implementation is an
implementation of an IETF protocol under development as an
Internet-Draft.

Thanks.

---

From: Pete Resnick <[email-masked]>
Date: Wed, 4 Mar 2009 17:01:40 -0600

On 3/4/09 at 12:44 PM -0500, Henning Schulzrinne wrote:

>We can't force spec writers to code...

Ah, for the days where "spec writer" and "coder" were co-referent. 
That we have spec writers who don't write code is sad.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 4 Mar 2009 23:00:02 -0500

I would hope that most spec writers also write code (or at least have  
written code in the past), but maybe not for every draft that they  
prepare. Also, if we'd want to rule out any protocol contributions by  
individuals with management responsibilities that make writing  
production code difficult, it would probably eliminate a fair number  
of active contributors in the IETF. I'm making no value judgement  
whether that would be a good thing or not...

Henning

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 5 Mar 2009 06:42:51 -0700

Marc Petit-Huguenin <petithug at acm dot org> wrote:

> early implementations, etc... You can send your responses directly to 
> me if you do not want to respond publicly - I will keep them 

I did an early implementation of RFC 4646 while it was in draft form, 
and updated it for draft-4646bis.  Indeed, the work that ultimately 
became RFC 4645 and the initial IANA Language Subtag Registry started as 
a prototype.

I did this solely to help flesh out the ideas being discussed in the 
LTRU WG and as a smoke test against the group's output, not because of 
any expectation of monetary gain or tangible, widespread recognition --  
which is a darned good thing since I haven't received either one, though 
I did get an Acknowledgement in 4646.

Other people inside and outside the WG have built their own validators 
for both 4646 and 4646bis.  Having to list all of these implementations 
in the drafts would have made a slow review and approval process even 
slower.

--
Doug Ewell  *  Thornton, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

---

From: [email-masked]
Date: Fri, 06 Mar 2009 08:00:57 -0800 (PST)

> implementation of an IETF protocol under development as an

You asked..,. so here's my data.

In recent times I've mostly been involved with the Sieve WG. According to my
tally the Sieve WG has had a total of 35 drafts on its plate at one time or
other. Of these roughly 13 have become RFCs and 10 more are in the RFC Editor
queue (my numbers here may be slightly out of date).

Out of all of these I implemented 21 of them while they were still unapproved
drafts: 3028bis, relational, relational-bis, subaddress, subaddress-bis,
spamtest, spamtest-bis, copy, vacation, imapflags, variables, environment,
editheader, notify, notify-mailto, refuse-reject, date-index, ihave, regex,
external-lists, and dsn. When you take into account that at least 4 of these
drafts were never serious contenders for standardization, that's an early
implementaion rate of well over 60%.

I have also implemented body and mime-loops, but only after they were last
called. Implementation of body turned up no late-breaking issues in the draft
(although I now wish I had lobbied for one or two additional restrictions).
Implementation of mime-loops, OTOH, turned up a couple of omissions and errors,
one of which really needs to be addressed before publication. So that's a case
where earlier implementation would have helped a bit.

Sieve-in-xml is sort of a special case. In one sense I haven't implemented it,
but in another Sai and I wrote the XML schema, Relax NG grammar, and conversion
stylesheet that apppear in the draft, which collectively constitute at least a
partial implementation. Other folks here at Sun have also implemented an
earlier version of this draft in its entirety.

Additionally, at least two drafts, notify-sip and convert, are for various
reasons ones we are unlikely to ever implement. As such, there's no way I could
justify doing an early implementation of either one of them.

I'm the author or coauthor of six of these drafts. I implemented all but one of
these as I was writing the document (all six if you count sieve-in-xml). I
haven't checked to see if I'm listed as a constributor in most of the others
but I suspect I am.

As for "draft drift", i.e., things change a bunch after implementation,
requiring a lot of implementation retooling, that has not been a serious
problem - so far at least. (A bunch of these documents are still just drafts so
that could change.) The two drafts that changed quite significantly along the
way are imapflags and notify. Changes to imapflags required some recoding, but
nothing too onerous. So far I have only sketched out what's needed to get our
notify and notify-mailto support up to spec, and it looks a little tricky but
doable. In this case the bigger issue is that due to customer requirements we
had no choice but to deploy the original version of notify and notify-mailto.
We're now stuck with supporting the old notify in additional to the new, which
grots up the code a bit, and having to write a tool to up-convert existing
scripts to the new notify format.

Finally, refuse-reject proved to be an interesting case for us. One of the
provisions that got added to that draft along the way (production of SMTP-time
errors when possible) would have been very difficult if not impossible for us
to implement. But as the draft proved to be contentious and took a long time to
get through the process, some largely unrelated product enhancements changed
that and made implementation not just possible but fairly easy.

				Ned

---

From: "Scott Lawrence" <[email-masked]>
Date: Fri, 06 Mar 2009 23:30:02 -0500

> http://www.ietf.org/internet-drafts/draft-petithuguenin-running-code-considerations-00.txt

I oppose the addition of such a mandatory or formalized section, despite
the fact that I very much support measuring specification quality and
community support by looking for running code.

I oppose even more this part of the definition of "running code":

        The minimum rights that should be granted for this source code
        are the right to duplicate it for purpose of reading it and the
        right to execute it or generate the binary code to execute it.

I spend nearly all my time and energy these days on open source
software, so this would not be a barrier for me, but it would be for
many people whose contributions are important.

More importantly, I have no need (or desire) to look at the source code
for an implementation to determine whether or not it matches the
specification and interoperates with other implementations.  It is
behavior on the wire that we are concerned with in the IETF - we don't
specify how a protocol is implemented, we specify the behavior it
exhibits.

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Sat, 07 Mar 2009 10:12:51 -0800

> specify how a protocol is implemented, we specify the behavior it

The reason is to evaluate implementation complexity.  Protocol
behavior on the wire does not permit to do.  Sometimes a protocol or
feature are rejected because it is thought to be too complex to
implement - a good example would be the time-switch in CPL, where
IIRC an implementation permitted to have the calendar feature back
in the spec.  History-Info is an example of something that is
thought more more complex to implement that it is in reality - here
also showing some code would have helped to demonstrate that it is
not the case.  The opposite is probably true, i.e. some protocols or
features would have been abandoned when looking at the complexity of
implementing it (especially considering the correlation between code
complexity and security issues)

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Sat, 07 Mar 2009 10:42:32 -0800

> many people whose contributions are important.

It seems that there is a general misunderstanding that this draft
asks for mandatory source implementation.  This is not the case, and
this will be better explained in the next version of the draft.

What this draft ask for is to be mandatory to list in a specific
section the names, authors and sponsors of early implementations
available in source form.  Everything that does not fall under the
definition can still be acknowledged as it is now (or not) in the
normal Acknowledgement section.  And if there is no early source
implementation, then the section is empty.  The only burden for the
I-D author is to add an entry in the section when an implementer
sends a reference.  The burden for the RFC editor is to remove the
URLs and eventually the whole section if empty before publication as
RFC.  That's it.  On the other hand this give to reviewers an idea
of the complexity needed to implement it and a way to evaluate
corner cases, security issues and other "details" that will plague
the future production implementations, etc...

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 7 Mar 2009 12:01:54 -0700

Marc Petit-Huguenin <petithug at acm dot org> wrote:

> implementation, then the section is empty.

Why not simply make the section optional, then, and let authors and WGs 
include it if they think it will provide better insight into the 
specification -- or improve its chances of being approved -- and let 
others ignore it if they choose?

---

From: Cullen Jennings <[email-masked]>
Date: Sat, 7 Mar 2009 14:13:32 -0700

On Mar 3, 2009, at 3:43 PM, Marc Petit-Huguenin wrote:

>   Giving to early implementers a guaranty that

Marc, I feel that it is well worth thanking anyone in the  
acknowledgements that does a full or significant early implementation  
and sends comments to the list about it. Even if that comment is "we  
found no problems with the spec". It is a huge amount of work to  
"test" a spec by implementing it and a valuable contribution to the  
development.

If you, or others, see any RAI Area WG drafts that should really  
acknowledge an implementation, please bring it to my  attention. I  
suspect that many people at IETF would feel it is worth saying thanks  
for this kind of work, I'd encourage implementors to bring it up with  
WG draft editors and Chairs if they think someone should be added to  
the Acknowledgements section.

Thanks, Cullen <RAI AD>

PS - Thank you for your TURN implementations - as you know, they have  
helped find many problems with that spec. I'm off to read the TURN  
Acknowledgements :-)

---

From: [email-masked]
Date: Sat, 07 Mar 2009 16:14:51 -0800 (PST)

> asks for mandatory source implementation.

I have seen no evidence of any such misunderstanding. It certainly
isn't what Scott objected to above.

>  This is not the case, and

> available in source form. 

And that's the problem Scott is referring to, and which I agree is a
significant issue. There is no doubt that the process of implementing something
provides valuable insight into specification clarity, general implementabiity,
and protocol complexity. But the same insights accrue from *any*
implementation, irrespective of whether it is done as open source or not.

The only advantage an open source implementation has over closed source is that
others can look at it. But while this advantage is real, the bias it could
create in favor of open source as a means of assessing protocols is just not
worth it.

> Everything that does not fall under the

Right. More pointless boilerplate in all our drafts, more pointless
administrivia to deal with. The burden on authors are already too great; we
should be looking at ways to reduce them, not increase them.

> The only burden for the
> sends a reference.

Wrong. Any requirement such as this carries with it the need for enforcement,
with all that implies.

> The burden for the RFC editor is to remove the

And I continue to believe the costs are far greater than the benefits.

				Ned

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Sat, 07 Mar 2009 17:44:21 -0800

> create in favor of open source as a means of assessing protocols is just not

The proposal never asked for open source implementations.

[...]
> And I continue to believe the costs are far greater than the benefits.

OK.

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Sat, 07 Mar 2009 17:58:09 -0800

Cullen Jennings wrote:
> found no problems with the spec". It is a huge amount of work to "test"
> a spec by implementing it and a valuable contribution to the development.
> 
> for this kind of work, I'd encourage implementors to bring it up with WG
> draft editors and Chairs if they think someone should be added to the
> Acknowledgements section.
> 
> Thanks, Cullen <RAI AD>
> 

My contributions were acknowledged, it's not the issue.  The issue,
IMO, is that there is not enough early implementations and I was
just trying to find a way to improve this.  The solution proposed
was universally rejected, so the problem is still here.

---

From: [email-masked]
Date: Sun, 08 Mar 2009 12:15:59 -0700 (PDT)

> The proposal never asked for open source implementations.

Again, the proposal says the sorts of implementations that should be listed
are ones meeting this condition:

That may allow some implementations that don't fit your definition of "open
source", but the fact remains this proposal tries to limit the implementations
that are supposed to be listed in this new section. And Scott and I both object
to that.

Which do you think teaches us more about the scalabiliy of a given protocol -
an open source but hacked-together implementation done in great haste done by a
bunch of inexperienced students for an undergraduate class, or a proprietary
implementation done as part of the ongoing development of a high end product by
an expert implementation team with decades of experience?

And please don't try and tell that there are no early proprietary
implementations. I can easily list dozens of counterexamples.

One additional point. This proposal also imposes strong requirements that
implementation information in this section be kept up to the date:

   In the case a specific code implements
   multiple versions of the document, the URL MUST point to the latest
   version available but the text MUST contain the complete list of
   versions supported.

That alone places an unacceptable burden on draft authors. If there's a need to
track the status of various implementations, it can be, and often is, done on a
separate web page.

				Ned

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Sun, 08 Mar 2009 12:58:16 -0700

Ned Freed wrote:
> source", 

Here's the beginning of the Open Source Definition:

"Open source doesn't just mean access to the source code."

So please stop labeling it as open source.  It is not and never was.

Anyway, I already said in other thread that there is a consensus
against this proposal, so no need to add more.

---

From: Alexandru Petrescu <[email-masked]>
Date: Mon, 09 Mar 2009 09:51:38 +0100

> implementation of an IETF protocol under development as an

I did early implementations in the Mobile IPv6 space.  First, AH 
protection of MIP6 signalling.  Then an implementation of 'BAKE', a very 
early other-person proposal for what later became RR tests for RO.  That 
was a good exercice to understand the landscape, but unfortunately none 
got into an RFC.  I felt it a bit disappointing and I decided to never 
ever again implement anything until it's an RFC at least Proposed Standard.

I thus later did larger implementation effort of Mobile IPv6 RFC 
Proposed Standard, of MUST features but which were not used by anybody 
else... and even later suggested for deprecation.  That's even more 
disappointing.

Usually, when I do early implementation my motivation has to do with 
competitivity: first let self impressed by an IETF great new feature, 
then be there first before the others, claim ownership, etc. 
Unfortunately it can easily be _too_ early implementation :-)

Generally in the WGs where I participate, I find it very encouraging 
whenever implementers do talk.

Alex

---

From: Philip Matthews <[email-masked]>
Date: Sat, 21 Mar 2009 15:44:36 -0400

> was universally rejected, so the problem is still here.

In my opinion, what would be helpful is a gentle reminder to authors  
to review the acknowledgement section of their drafts regularly to  
ensure that it is up-to-date, perhaps with the suggestion that they  
make special mention of early implementors. If you are editing a  
specification that goes on for some time, at times you forget things  
like this. I am the editor of the above-mentioned TURN specification,  
and I found that I "forgot" about certain sections at times. Once, I  
even published a version with TWO open issues sections (thanks for  
catching that, Cullen!).

Perhaps idnits could be enhanced to suggest things like this? For  
example, perhaps idnits could be enhanced with the following warning:

    "You have not changed your acknowledgement section in the last  
three revisions. Should other people be acknowledged? For example, are  
there people who have done a preliminary implementation and provided  
useful feedback?"

Marc: Perhaps your draft should be rewritten to suggest this idnits  
change?

- Philip