Subject: Running code, take 2

From: Yaron Sheffer <[email-masked]>
Date: Wed, 12 Dec 2012 21:31:04 +0200

Hi,

I have just published a draft that proposes an alternative to Stephen's 
"fast track". My proposal simply allows authors to document, in a 
semi-standard way, whatever implementations exist for their protocol, as 
well as their interoperability.

http://www.ietf.org/id/draft-sheffer-running-code-00.txt

I believe this can achieve the same goal, which is to apply preferred 
treatment to protocols that have been implemented, and with a lot less 
process overhead. It also leaves much more leeway for individual working 
groups to apply their own procedures and customs.

Similarly to Stephen's proposal, we can use a process experiment (per 
RFC 3933) to gauge the effectiveness of this one.

I am looking forward to comments and discussion on this list.

     Yaron

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 13 Dec 2012 14:57:43 +0100

On Wed 12/Dec/2012 20:31:04 +0100 Yaron Sheffer wrote:
> 
> I have just published a draft that proposes an alternative to
> Stephen's "fast track". My proposal simply allows authors to document,
> in a semi-standard way, whatever implementations exist for their
> protocol, as well as their interoperability.
> 
> [...]

As an occasional I-D reader, I'd appreciate "Implementation Status"
sections, including IPR info.  I don't think anything forbids to add
such sections, if the authors wish.  I'd add a count of the number of
I-Ds that actually have it among the experiment's success criteria.

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 13 Dec 2012 14:16:24 -0000

I'm interested in this idea.

However, I note that an "implementation status" section of a document is frozen
in time when a document goes to RFC.

I wonder whether we could leverage our tools and do something similar to IPR
disclosures. That is, provide a semi-formal web page where implementation
details could be recorded and updated. These would then be searchable and linked
to from the tools page for the I-D / RFC. 

They could record the document version that has been implemented, and also allow
space for other notes.

Adrian (Just thinking aloud)

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Alessandro Vesely
> Sent: 13 December 2012 13:58
> To: [email-masked]
> Subject: Re: Running code, take 2
>

---

From: Marc Blanchet <[email-masked]>
Date: Thu, 13 Dec 2012 09:18:34 -0500

> They could record the document version that has been implemented, and also allow

I like that idea.

Marc.

---

From: Yaron Sheffer <[email-masked]>
Date: Thu, 13 Dec 2012 16:48:01 +0200

Thanks Alessandro. I couldn't think of a specific count (10? 100?) for 
the success criteria, which is why the criteria are rather fuzzy. The 
experiment's summary report does promise to list adoption statistics.

Best,
	Yaron

---

From: Yaron Sheffer <[email-masked]>
Date: Thu, 13 Dec 2012 16:52:35 +0200

Hi Adrian,

I would suggest to start with my proposal, because it requires zero 
implementation effort. If this catches on, I see a lot of value in your 
proposal.

Please also note that the "implementation status" section (according to 
my proposal) is not "frozen" when published as an RFC, rather it is 
deleted. RFCs are forever, and I think a point-in-time implementation 
status is not appropriate in an RFC.

Thanks,
	Yaron

---

From: Marc Blanchet <[email-masked]>
Date: Thu, 13 Dec 2012 09:55:22 -0500

Le 2012-12-13 à 09:52, Yaron Sheffer a écrit :

> Hi Adrian,
> 
> I would suggest to start with my proposal, because it requires zero implementation effort.

disagree. phase 1: use IETF wiki. phase 2: develop an widget within data tracker.

Marc.

> If this catches on, I see a lot of value in your proposal.
> 
> Please also note that the "implementation status" section (according to my proposal) is not "frozen" when published as an RFC, rather it is deleted. RFCs are forever, and I think a point-in-time implementation status is not appropriate in an RFC.
> 
> Thanks,
> 	Yaron
>

---

From: Yaron Sheffer <[email-masked]>
Date: Thu, 13 Dec 2012 17:00:11 +0200

Hi Marc,

I think it's critical that a person reading a draft (e.g. going to 
http://tools.ietf.org/html/draft-blanchet-iab-internetoverport443-01) 
will have a direct way to check out on the implementation status.

This is trivial if it's a section in the document. It's simple if it's 
linked from the Tools page. Otherwise, e.g. if you put it on the wiki, 
only IETF insiders will be aware of it.

Thanks,
	Yaron

---

From: Marc Blanchet <[email-masked]>
Date: Thu, 13 Dec 2012 10:04:42 -0500

Le 2012-12-13 à 10:00, Yaron Sheffer a écrit :

> Hi Marc,
> 
> I think it's critical that a person reading a draft (e.g. going to http://tools.ietf.org/html/draft-blanchet-iab-internetoverport443-01) will have a direct way to check out on the implementation status.
> 
> This is trivial if it's a section in the document. It's simple if it's linked from the Tools page. Otherwise, e.g. if you put it on the wiki, only IETF insiders will be aware of it.
> 

sure. Let me restart:
- I like Adrian proposal: instead of in RFC, put it online within our site
- but you wrote: requires implementation effort.
- I replied: well, phase 1 (of put it online within our site) can be done with almost zero implementation effort. phase 2 requires some work (I'd say not that big) for implementation/tools.

Regards, Marc.

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 13 Dec 2012 15:10:38 -0000

How about...

Start with Yaron's proposal to include in the I-D. This is easy as a starting
point. Duplicate documentation in wiki may be useful and provide a place to
track text for inclusion in the next revision.

When/if inclusion in the I-D gets messy, replace text in I-D with pointer to
wiki.

When/if experiment looks like a success, replace all above with data tracker
tool and allow it to persist for RFCs.

Adrian

> From: Marc Blanchet [mailto:[email-masked]]
> Sent: 13 December 2012 15:05
> To: Yaron Sheffer
> Cc: [email-masked]; [email-masked]; 'Alessandro Vesely'
> http://tools.ietf.org/html/draft-blanchet-iab-internetoverport443-01) will
have a
> direct way to check out on the implementation status.
> >
linked from the
> Tools page. Otherwise, e.g. if you put it on the wiki, only IETF insiders will
be
> aware of it.
> - I replied: well, phase 1 (of put it online within our site) can be done with
almost
> zero implementation effort. phase 2 requires some work (I'd say not that big)
for
> implementation/tools.
> implementation effort.
> >>> Please also note that the "implementation status" section (according to my
> proposal) is not "frozen" when published as an RFC, rather it is deleted. RFCs
are
> forever, and I think a point-in-time implementation status is not appropriate
in an
> RFC.

---

From: Yaron Sheffer <[email-masked]>
Date: Thu, 13 Dec 2012 17:11:54 +0200

+1.

	Yaron

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 13 Dec 2012 15:13:35 -0000

Yaron,

Would you be willing to add this to your I-D?

> From: Yaron Sheffer [mailto:[email-masked]]
> Sent: 13 December 2012 15:12
> To: [email-masked]
> Cc: 'Marc Blanchet'; [email-masked]; 'Alessandro Vesely'
> my
> >>>>>> However, I note that an "implementation status" section of a document
> is
> >> frozen
> >>>>>> I wonder whether we could leverage our tools and do something similar
> to
> >> IPR
> >>>>>> They could record the document version that has been implemented,
> and
> >> also allow
> >>>>>> space for other notes.
> >>>>>>

---

From: Marc Blanchet <[email-masked]>
Date: Thu, 13 Dec 2012 10:18:06 -0500

> tool and allow it to persist for RFCs.

makes sense to me.

Marc.

---

From: Loa Andersson <[email-masked]>
Date: Thu, 13 Dec 2012 16:22:42 +0100

Folks,

I agree that understanding the implementation status of a draft
sometimes is essential, but not for all drafts and not always.
Today wg chairs do this type of info collection at the shepherd
write-up.

Have anyone thought about how much work goes into compiling this type
of information. There are vendors that by policy decided not to disclose
implementation information, before the implementation is done and the
document has become an RFC.

Most of the time it is possible to to get some understanding, with a
promise not to make the info public and only use it for a rather
cryptic statement in the shepherd write-up - "we know of existing or
intended implementations of this draft".

A second, rather new problem, is that for some drafts the IANA
assignment is the singular most important part of the ID. We have heard
vendors say "We'll wait for the IANA assignment until we implement!"

/Loa

-- 

Loa Andersson                         email: [email-masked]
Sr Strategy and Standards Manager            [email-masked]
Ericsson Inc                          phone: +46 10 717 52 13
                                              +46 767 72 92 13

---

From: Marc Blanchet <[email-masked]>
Date: Thu, 13 Dec 2012 10:39:16 -0500

> vendors say "We'll wait for the IANA assignment until we implement!"

right. but that does not preclude doing the proposal. What you are saying is that maybe the info (in draft, wiki, tracker) may not be complete since some may not disclose. But the available information would be valuable.

Marc.

---

From: Yaron Sheffer <[email-masked]>
Date: Thu, 13 Dec 2012 17:35:08 +0200

Sure.

Adrian Farrel <[email-masked]> wrote:

>> > Start with Yaron's proposal to include in the I-D. This is easy as
>a
>starting
>will
>> >> Tools page. Otherwise, e.g. if you put it on the wiki, only IETF
>insiders
>will
>> > be
>> >> aware of it.
>> >>>
>> >>
>> >> sure. Let me restart:
>> >> - I replied: well, phase 1 (of put it online within our site) can
>be done
>with
>> > almost
>> >> zero implementation effort. phase 2 requires some work (I'd say
>not that
>big)
>> > for
>> >> implementation/tools.
>> >>
>> >> Regards, Marc.
>> >>
>> >>> Thanks,
>> >>> 	Yaron
>> >>>
>> >>>> disagree. phase 1: use IETF wiki. phase 2: develop an widget
>within data
>> >> tracker.
>> >>>>
>> >>>> Marc.
>> >>>>
>> >>>>
>> >> proposal) is not "frozen" when published as an RFC, rather it is
>deleted.
>RFCs
>> > are
>> >>>>>> details could be recorded and updated. These would then be
>searchable
>> >> and linked
>> >>>>>>> From: [email-masked] [mailto:[email-masked]] On
>Behalf
>> Of
>> >>>>>>> Alessandro Vesely

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

---

From: Ted Hardie <[email-masked]>
Date: Thu, 13 Dec 2012 08:01:57 -0800

Hi Yaron,

Reading through this draft, I note that you're suggesting inclusion of
information about the type of implementation license (open source,
proprietary, etc.).  While that may be useful, I'd be more interested in
knowing whether the implementation represents an exercise of any licenses
required by IPR disclosures for the relevant draft.  Since there are
implementations (even open source implementations) that might be submitted
by the same organization holding the IPR, this is not something you can
automatically infer.  There are also, of course, cases where an implementor
does not agree that an IPR statement's claim of coverage is correct.

Before we eliminated the requirements for demonstrating two interoperable
implementations at Draft, one piece of that requirement was that the two
being used represent different exercises of any license.  If we are going
to start tracking implementations, it might be useful to get that data back.

regards,

Ted Hardie

---

From: Loa Andersson <[email-masked]>
Date: Thu, 13 Dec 2012 17:10:16 +0100

Marc,

inline please!

> right. but that does not preclude doing the proposal. What you are saying is that maybe the info (in draft, wiki, tracker) may not be complete since some may not disclose. But the available informa…

that entirely depends on the policy of the 4-5 dominating vendors in the
area that the draft is applicable for. I know of people writing drafts
there their own companies say that they can disclose implementation
information in public. Why would anyone else want to supply their
information to that draft.

I think the key word here is "in public", a trustful relationship
between working group chairs gives you enough information to progress
the document through the IESG, I'm afraid that this will be much harder
if we have an empty (or single company) "Implementation Status"
section in the drafts.

/Loa

--

---

From: Melinda Shore <[email-masked]>
Date: Thu, 13 Dec 2012 09:04:14 -0900

> I like that idea.

Me too.

Melinda

---

From: Randy Bush <[email-masked]>
Date: Thu, 13 Dec 2012 10:13:54 -0800

don't we already have a way of doing this?  implementation reports, e.g.

    draft-ietf-sidr-rpki-rtr-impl-01.txt

a wiki can be more easily curated, though has authorization challenges.

could you clue me into how the different modes would facilitate progress
of the base document(s)?

randy

---

From: Melinda Shore <[email-masked]>
Date: Thu, 13 Dec 2012 09:19:55 -0900

> could you clue me into how the different modes would facilitate progress

The curation model is that somebody/somebodies take responsibility for
authoring a document and tracking down implementations, writing them
up, etc.  The wiki model is that somebody/somebodies say "Hey!  I've got
an implementation, let me tell you about it."  I don't see why they
can't co-exist, but in the meantime I don't think implementation reports
in the form of internet drafts are that common, and I'd be curious to
see which is more effective at getting information out.

What's the long-term plan for the RPKI implementation report -
publication?

Melinda

---

From: Randy Bush <[email-masked]>
Date: Thu, 13 Dec 2012 10:21:19 -0800

> What's the long-term plan for the RPKI implementation report -

yes.  it is traditional in the routing area.

randy

---

From: Yaron Sheffer <[email-masked]>
Date: Thu, 13 Dec 2012 23:51:54 +0200

Hi Randy,

the RPKI report is very impressive and probably very useful. But:

- Other areas (e.g. the Security Area, where I'm coming form) may not 
have this tradition.

- For "smaller" protocols the gain does not justify the effort of 
writing a separate implementation report. A section in the base document 
is much simpler to maintain.

- And as I noted earlier on this thread, a wiki is a good alternative, 
provided everyone reading the base draft is referred to the 
"implementation status" wiki. They should not have to guess that one 
exists, or to search for it.

Thanks,
	Yaron

---

From: John C Klensin <[email-masked]>
Date: Thu, 13 Dec 2012 17:47:36 -0500

--On Thursday, December 13, 2012 23:51 +0200 Yaron Sheffer
<[email-masked]> wrote:

> - Other areas (e.g. the Security Area, where I'm coming form)
> may not have this tradition.

If it is important, nothing prevents establishing it.

> - For "smaller" protocols the gain does not justify the effort
> of writing a separate implementation report. A section in the
> base document is much simpler to maintain.

I'm not at all clear why it is more effort to write a separate
report.  But, assuming it is, I am aware of nothing in any IETF
or RFC Editor procedure that prohibits including a discussion of
implementation experience in a standards track RFC.

> - And as I noted earlier on this thread, a wiki is a good
> alternative, provided everyone reading the base draft is
> referred to the "implementation status" wiki. They should not
> have to guess that one exists, or to search for it.

That is a more complicated problem because the RFC Editor is
somewhat reluctant (for good reasons, IMO) to accept references
to web pages whose permanency is not assured.  There are,
however, lots of other ways by which one could imagine creating
and maintaining such pointers.  For example, if you could
convince the community that this was important, and likely to be
done often enough, you might lobby the IESG to negotiate an
additional sentence in the "Status" section of standards track
RFCs that provided a stable pointer to an index of
implementation reports.

My concern remains that we not create new formal procedures to
do (or even experiment with) things that can be done under
existing rules either for the whole IETF or on an area by area
or even document by document basis,  Let me suggest, as
co-author of the "process experiment" doc, that it was intended
to deal to allow experiments that would otherwise be in
violation of existing procedures.  As far as I can see, that
situation doesn't apply with either of these proposals.  And,
given the number of experiments carried out with reference to
RFC 3933 that did not result in useful and formal reports back
to the community, I'm unconvinced that formally identifying any
particular exercise in identifying implementations or handling
some specifications with implementations differently in the IESG
review process than other specifications will increase the odds
of  the results being examined carefully and critically enough
to generate a report and, through it, permanently change
behavior.  YMMD but, if either of these ideas are interesting,
persuade a friendly AD to try it on whatever scale makes sense,
and then let the rest of us know how it worked out for you.
Since no existing rules would thereby be violated, IETF
consensus is not required for such an exercise... or any number
of such exercises.

best,
   john

---

From: Randy Bush <[email-masked]>
Date: Thu, 13 Dec 2012 16:11:02 -0800

> My concern remains that we not create new formal procedures to do (or
> even experiment with) things that can be done under existing rules
> either for the whole IETF or on an area by area or even document by
> document basis

<aol>

my apologies.  i did not mean that formal implementation reports should
be filed for everything, or even everything that wants to be on a fast
track, if there is a fast track.

i meant merely to illustrate that we already have flavors for filing
implementation reports.  and i was asking if/how melinda thought the
format of reporting might affect the speed and direction of the track.

randy

---

From: Melinda Shore <[email-masked]>
Date: Thu, 13 Dec 2012 15:43:58 -0900

> format of reporting might affect the speed and direction of the track.

Right, and I'm not really sure.  I don't think that implementation
reports should be a mandatory document or document section.  sidr
seems to be functioning well but I suspect that a separate
implementation report document is a consequence of it functioning
well (and within a specific context) rather than a causal factor.
Considering, for example, scim, where there are a bunch of
implementations based on the state of the protocol prior to its
adoption as an IETF working group, and it seems to me that there
are is a sufficiently large number of implementations in a sufficiently
dynamic set of states that a document, as opposed to a wiki, may be
more difficult to maintain and ultimately less useful because of
that.

I do agree that there's nothing stopping working groups from
experimenting with various implementation report formats now, and
it may be worth trying several things out.  I'll run this past
the scim guys.

Melinda

---

From: John C Klensin <[email-masked]>
Date: Thu, 13 Dec 2012 20:17:33 -0500

--On Thursday, December 13, 2012 16:11 -0800 Randy Bush
<[email-masked]> wrote:

>> My concern remains that we not create new formal procedures
>> to do (or even experiment with) things that can be done under
>> existing rules either for the whole IETF or on an area by
>> area or even document by document basis
> 
> <aol>
> 
> my apologies.  i did not mean that formal implementation
> reports should be filed for everything, or even everything
> that wants to be on a fast track, if there is a fast track.
> 
> i meant merely to illustrate that we already have flavors for
> filing implementation reports.  and i was asking if/how
> melinda thought the format of reporting might affect the speed
> and direction of the track.

I actually understood that.  Sorry for not being clear.  My
concern about creating new procedures, of even a formal
experiment, is almost orthogonal to your comments and
illustration, for which I'm grateful and in complete agreement.

    john

---

From: Yaron Sheffer <[email-masked]>
Date: Fri, 14 Dec 2012 10:49:30 +0200

Hi John,

to clarify, my proposal only applies to Internet Drafts, and clearly 
states that the implementation section should be removed from the 
document before it is published as RFC.

Formally, we don't want non-permanent stuff in RFCs. And realistically, 
even if we had an implementation wiki, it is unlikely to be kept up to 
date once the RFC is published.

Regarding the "experiment" question: I fully agree that there's nothing 
preventing us from adopting this proposal (or a variant of it, or 
Stephen's proposal) today.

The value in a 3933 experiment is in the Summary Report, otherwise I 
agree it's a waste of time. At the end of the period we will have a 
little bit of data to understand whether we have traction for this idea, 
and whether we should make it IETF-wide, allow it to quietly die or 
explicitly advise against it.

Thanks,
	Yaron

---

From: t.p. <[email-masked]>
Date: Fri, 14 Dec 2012 11:15:59 +0000

----- Original Message -----
From: "Yaron Sheffer" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>; "'Alessandro Vesely'" <[email-masked]>
Sent: Thursday, December 13, 2012 2:52 PM
> I would suggest to start with my proposal, because it requires zero

I am surprised at this.  Gathering information about implementations is
something that happens in some WGs and not in others, but it is always
the chair that is driving it, often as part of the write-up prior to
IETF Last Call.  This takes time, asking, chasing, clarifying, and
getting replies off-list because the implementor does not have
permission to reveal such matters to the world at large.

Since the WG Chair is a limited resource, and is on the critical path
for several actions in the progress of an I-D to RFC, asking them to do
more work can only delay the progress of RFC at large.  This is not a
zero sum game.

Tom Petch

>                             If this catches on, I see a lot of value
in your
> > details could be recorded and updated. These would then be
searchable and linked
> > They could record the document version that has been implemented,
and also allow
> >> From: [email-masked] [mailto:[email-masked]] On
Behalf Of

---

From: t.p. <[email-masked]>
Date: Fri, 14 Dec 2012 11:22:39 +0000

----- Original Message -----
From: "Marc Blanchet" <[email-masked]>
To: "Loa Andersson" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, December 13, 2012 3:39 PM

> vendors say "We'll wait for the IANA assignment until we implement!"

right. but that does not preclude doing the proposal. What you are
saying is that maybe the info (in draft, wiki, tracker) may not be
complete since some may not disclose. But the available information
would be valuable.

<tp>
And could be very misleading, because it is incomplete and people will
draw unwarranted inferences from the information that has been made
public.  They will likely not understand the nuances of different
implementors having different policies about when to make such details
public, whether for commercial reasons, IPR reasons or whatever.  Nor
may they realise that an I-D is a work in progress, so an implementation
at an early stage may be really bad news if there is a later change to
the technical details (something of which happened with MPLS-TP OAM,
with an ITU-T approach predating an IETF one).

Tom Petch
</tp>
Marc.

>>> Tools page. Otherwise, e.g. if you put it on the wiki, only IETF
insiders will
within data
>>> proposal) is not "frozen" when published as an RFC, rather it is
deleted. RFCs
>>>>>>> However, I note that an "implementation status" section of a
document is
>>>>>>> I wonder whether we could leverage our tools and do something
similar to
searchable
>>>>>>> They could record the document version that has been
implemented, and
Behalf Of

---

From: Randy Bush <[email-masked]>
Date: Fri, 14 Dec 2012 04:09:49 -0800

> date once the RFC is published.

so, we act on implementation and interoperability data which are not
kept?

randy

---

From: Randy Bush <[email-masked]>
Date: Fri, 14 Dec 2012 04:30:42 -0800

> IETF Last Call.

uh, not really.  in some wg cultures, it's just seen as part of the
drafters' or implementors' job, just as is doing a mib.  i suspect that
the implementation document (and the mib) i used as an example came as a
surprise to the chairs as they are more security than routing area.

randy

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 14 Dec 2012 13:36:47 +0100

> document before it is published as RFC.

One place where an Implementation Status may help is IETF LCs:  People
are asked to comment on I-Ds whose development they didn't track, on
topics they may have only a working knowledge of.  Code licensing and
IPR details could quickly convey whether a given standardization
process is being gamed, for example.

> Formally, we don't want non-permanent stuff in RFCs.

We have the Errata and the Outcomes, AFAIK.  The Errata is freely
writable, somewhat impractical, and doesn't address this purpose.

> And realistically, even if we had an implementation wiki, it is
> unlikely to be kept up to date once the RFC is published.

The Outcomes /is/ a wiki, http://trac.tools.ietf.org/misc/outcomes/.
No wonder it's not updated:  It is difficult to find it from the
relevant RFCs (let alone specific sections thereof), and it is not
freely writable.  Anyway, it addresses only a small subset of the
Implementation Status.

While we need no formal procedure to write the Implementation Status
section of an I-D, we probably need to specify how the content of that
section can be used to set off the wiki page for the new RFC,
and how that wiki should work.  Would that help running code?

---

From: Yaron Sheffer <[email-masked]>
Date: Fri, 14 Dec 2012 15:04:14 +0200

Hi Randy,

I don't know who's "we" in your question.

My proposal is aimed at an earlier stage in the process, when the WG 
needs to evaluate a draft before it becomes a WG document and later, 
during WGLC. During these stages, information about implementation 
status is very useful, even if (as some have stated), it may not always 
be available.

As to the later stage (off topic), let me just give one example. Earlier 
today I sent a SecDir review where I said that a certain 1997 RFC is 
probably not implemented by anybody today. I may be right or I may be 
wrong, I don't have any solid data. But do you expect the author of that 
RFC to maintain an up-to-date implementation status wiki for 15 years 
after the RFC had been published? I don't.

Thanks,
	Yaron

---

From: Riccardo Bernardini <[email-masked]>
Date: Fri, 14 Dec 2012 14:07:02 +0100

> and how that wiki should work.  Would that help running code?

<brainstorming_mode>
This just came to my mind... Why not doing a little step more and
making the wiki page a kind of "directory" to available
implementations?  Maybe each entry could have a "license type" field
(e.g. open source, commercial, ...) and "compliant flag" assuming
values like "fully compliant," "partially compliant," "unknown,"
"what kind of weed did they smoke before coding this?!?" :-)  If the
protocol allows it, the compliant flag could be assigned on the basis
of running a suitable set of test vectors (possibly described in the
RFC), otherwise it could be assigned after a community discussion
(maybe inside the WG that produced the protocol, it still existing).

This kind of "implementation directory" could be useful not only to
the IETF, but also to people who needs to implement a protocol and
would prefer to use something ready.  A new entry in the directory
could be written by anyone (or proposed via e-mail).  The author that
writes the entry cannot set the compliant field which will be
"unknown" by default.  If there is some evidence that the
implementation is compliant (e.g., the software is used inside a
widely used browser and it did not cause any meltdown so far:-), the
status could be set to "probably compliant." The upgrade to "fully
compliant" would require running tests and/or community discussion.

Finally, I would also add to the implementation page a reference to
any available test vectors (if the protocol allow them).  I find thme
are quite useful during the implementation phase.
</brainstorming_mode>

Riccardo

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 14 Dec 2012 13:18:53 +0000

Hi Yaron,

I'm glad to see more discussion of this general topic.

> I am looking forward to comments and discussion on this list.

Looks fine to me and I reckon that doing one or both of these
experiments in parallel or series could make sense depending on
what happens when/if they're last called. In contrast with
some others who've posted, I do think doing process experiments
is a good thing and we've not done enough to even know if 3933
can work or not - I only counted 4 or 5 RFCs that seem to refer
to 3933.

Cheers,
S.

PS: I just posted -02 of my proposal. [1]

[1] http://tools.ietf.org/html/draft-farrell-ft-02

---

From: Yaron Sheffer <[email-masked]>
Date: Fri, 14 Dec 2012 18:17:00 +0200

Hi Tom,

I am myself somewhat puzzled by the current process, where such 
information is gathered prior to IESG review, because (in my limited 
experience) I have never seen the IESG reject a draft merely because it 
is unimplemented.

In any case, my proposal aims to move the burden to authors, who can, if 
they choose, document extant implementations. This information, where 
available, can allow the WG to better judge the maturity of protocol 
proposals.

In cases when implementors choose not to reveal their implementations, 
we are back in the existing situation. Hopefully more people would be 
motivated to share such details publicly if it gives them some benefit 
during the early standardization stages.

Thanks,
	Yaron

On 12/14/2012 01:15 PM, t.p. wrote:
> document is frozen
> similar to IPR

---

From: John C Klensin <[email-masked]>
Date: Fri, 14 Dec 2012 18:09:51 -0500

Hi.

I've been trying to say out of this because I think most of the
suggestions are better carried out by AD-encouraged experiments
and reports to the rest of us on effectiveness rather than by
long discussions in the community about details and the costs of
an unnecessary consensus process. 

However, since I gather we are pushing (or being pushed) down
this path, let me suggest that approval of an IETF spec,
especially a standards-track one, has (or should have) elements
of all of the following:

(1) A conviction that the idea is implementable and that the
ideas expressed are consistent with implementation (and,
ideally, operational realities.

(2) Specification of sufficient quality to make
independently-developed interoperable implementations by people
who were not part of the WG or development process possible and
specifically that there are no ambiguities that could adversely
affect interoperable implementations.  This includes, but is not
limited to, editorial quality in terms of good technical
English, but does not include "good idea" criteria (see (4)).

(3) "No known technical defects" in the spec (the RFC 2026
requirement).  Note that, while an implementation might turn up
technical defects that might otherwise be unknown, it might
easily not turn up ones that could be identified in other ways.
It should imply a fairly comprehensive review that would have a
high likelihood of turning up any technical defects that are
present, but we all know that sometimes doesn't happen.

(4) Some level of IETF consensus that publishing the
specification has value for the community.  This might or might
not include a community belief that the document specifies a
good idea that should be implemented and deployed (the latter is
why we have Applicability Statements).

While I hope the above is obvious, the four categories are
nearly orthogonal.   One can have a good specification and a
worthwhile idea without an implementation, an implementation of
a bad idea and a lousy specification, and so on.   I believe...

(i) It is a bad idea, and could be a real disservice to the
community, to say that meeting a high standard on any one of
those dimensions should allow some of the others to be dealt
with in a perfunctory fashion.  That is exactly what a "fast
track" idea seems to be about and much of the debate seem to be
about how high a standard is relevant.

(ii) If it were really acceptable to decide that an
implementation (i.e., a limited demonstration of the first
criterion) justifies a "fast track", then demonstrations of the
others should to.  For example, if a WG could demonstrate that
the specification had been carefully and professionally
technically edited, perhaps that should justify "fast track"
treatment.  If there is already wide and successful deployment
of what is supposedly the specification, leaving aside the
question of whether those deployments and the specification
actually match, perhaps that should justify "fast track"
treatment.  If the review of the specification in a WG had been
particularly intense, including careful looks by many diverse
parties, perhaps that should justify "fast track" treatment.

I'm not certain what makes having an implementation more special
than any of these other categories and, to the extent to which
"fast track" means "less review" or a "higher bar to comments",
the idea makes me really, really, nervous, especially if the
requirement is not for a production-quality implementation that
has been tested both for interoperability and through deployment
to users.

   john

---

From: John C Klensin <[email-masked]>
Date: Fri, 14 Dec 2012 18:15:31 -0500

--On Friday, December 14, 2012 10:49 +0200 Yaron Sheffer
<[email-masked]> wrote:

>... 
> The value in a 3933 experiment is in the Summary Report,
> otherwise I agree it's a waste of time. At the end of the
> period we will have a little bit of data to understand whether
> we have traction for this idea, and whether we should make it
> IETF-wide, allow it to quietly die or explicitly advise
> against it.

I would encourage anyone who believes in "running code" wrt IETF
processes and who is willing to argue that one of the current
proposals should be adopted because there is real value in the
Summary Reports called for by 3933 to compile a list of 

	* 3933 Experiments
	* The published summary reports for each
	* How and where they found those reports.
	
I predict the results will be enlightening.  :-(

   john

---

From: Stephen Farrell <[email-masked]>
Date: Sat, 15 Dec 2012 14:58:34 +0000

Hiya,

> an unnecessary consensus process. 

That's a defensible opinion that I don't share. I'd say
s/most/many/ above would be better, but also think that we
ought try break this logjam where everything that touches
on process has to involve a gigantic debate with all the
usual suspects. (I don't foresee success in breaking that
logjam, but do think we ought not let it prevent us trying
things.)

I also think that since my proposal has a corner case where
a WG chair gets to put a draft on the IESG telechat agenda
that that ought only be done after some form of IETF LC.

> especially a standards-track one, has (or should have) elements

Luckily my draft says that: "All other criteria for Proposed
Standard or Experimental need to be met as usual." So I've no
idea how the text below is relevant.

Cheers,
S.

---

From: Yaron Sheffer <[email-masked]>
Date: Sat, 15 Dec 2012 18:03:19 +0200

Hi John,

According to Google, exactly one such report was issued: 
http://tools.ietf.org/html/draft-yevstifeyev-ion-report-06 (the 
published RFC omitted the results of the experiment, somehow). And this 
particular experiment is not even mentioned in 
http://www.ietf.org/iesg/process-experiment.html. Two other experiments 
are listed, and I was unable to find any reports summarizing them.

So I'm willing to concede that the "process experiment" experiment 
failed. But since I think it *could have been* a valuable process, and 
since I'm seeing an IESG member proposing to use it, I would request to 
hear from the IESG if they think RFC 3933 is still a management tool 
they'd like to use.

Thanks,
	Yaron

---

From: John C Klensin <[email-masked]>
Date: Sat, 15 Dec 2012 16:14:08 -0500

--On Saturday, December 15, 2012 18:03 +0200 Yaron Sheffer
<[email-masked]> wrote:

> http://tools.ietf.org/html/draft-yevstifeyev-ion-report-06
> (the published RFC omitted the results of the experiment,
> somehow). And this particular experiment is not even mentioned
> in http://www.ietf.org/iesg/process-experiment.html. Two other
> experiments are listed, and I was unable to find any reports
> summarizing them.
> 
> So I'm willing to concede that the "process experiment"
> experiment failed. But since I think it *could have been* a
> valuable process, and since I'm seeing an IESG member
> proposing to use it, I would request to hear from the IESG if
> they think RFC 3933 is still a management tool they'd like to
> use.

Yaron,

FWIW, _I_ still think it us a useful tool for situations in
which an experiment cannot otherwise be performed without
violating existing procedures.   In that regard, I've just been
trying to make two points:

(1) As far as I can tell, neither your proposal nor Stephen's
requires a violation of existing rules.   Both, again as far as
I can tell, can be performed by agreement between WG Chairs and
the relevant AD and at the discretion of the latter.   That
assumes that no one on the IESG would object strenuously and
that no one would find a decision to appeal on the grounds that
the faster procedure had the effect of suppressing important
input, but those risks would occur even if there were a formal
process experiment.

(2) One cannot justify a formal, 3933-style, process experiment
on the grounds that it would produce a report.  Reports are
possible without such experiments and, as you have seen, despite
the 3933 requirement the reports often do not appear in the
official record.

   john

---

From: Keith Moore <[email-masked]>
Date: Sun, 16 Dec 2012 15:08:38 -0500

> why we have Applicability Statements).
to this I'd add

(5) Widespread, examined belief that the specification has minimal 
impact on the Internet architecture.

I keep seeing IETF standardize protocols that seem likely to have 
seriously damaging architectural impact without much, if any, 
examination of that impact (the PCP and MIF work come most immediately 
to mind, but I could cite several others given a few minutes to think 
about it).  I'd hate to see a fast-tracking procedure used as a way to 
further circumvent such examination.

Keith

---

From: John C Klensin <[email-masked]>
Date: Sun, 16 Dec 2012 15:49:43 -0500

--On Sunday, December 16, 2012 15:08 -0500 Keith Moore
<[email-masked]> wrote:

> to this I'd add
> 
> (5) Widespread, examined belief that the specification has
> minimal impact on the Internet architecture.
> 
> I keep seeing IETF standardize protocols that seem likely to
> have seriously damaging architectural impact without much, if
> any, examination of that impact (the PCP and MIF work come
> most immediately to mind, but I could cite several others
> given a few minutes to think about it).  I'd hate to see a
> fast-tracking procedure used as a way to further circumvent
> such examination.

I think I would have said "impact on the Internet architecture
is well-understood and judged to be acceptable" instead, but we
agree about the general principle.

   john

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 16 Dec 2012 21:49:22 +0000

John, Keith,

I really have to say that you both seem to be talking about
things that have nothing whatsoever to do with my proposal.

ISTM that you are of the opinion that anything the IETF
does to go faster is bad in and of itself because its
scary. This also seems to me to be an example of the kind
of process discussion wandering onto everyone's favourite
hobby horse that I mentioned before.

So I don't see any change to the draft is required, but
if you do have a direct comment on the draft, that'd be
welcome and I'd be very happy to get such comments. Or
perhaps I'm missing something that is relevant to my
draft, but if so, I don't see it, so you need to bring
your concerns down to issues with the draft if you are
making a specific comment rather than being generally
worried.

Cheers,
S.

---

From: Keith Moore <[email-masked]>
Date: Sun, 16 Dec 2012 17:27:57 -0500

On 12/16/2012 04:49 PM, Stephen Farrell wrote:
> scary.
It's not that simple, at least in my opinion.

I'm generally fine with fast tracking a document that describes a simple 
protocol that is easily understood, easily implemented, 
noncontroversial, and has a limited impact.  In all other cases, I think 
that more deliberation is appropriate.   That's not to say that IETF 
currently makes good use of deliberation time.  But I'd rather see it 
make better use of that time, than to try to reduce that time without 
first improving protocol and document quality. For example, I'd like to 
see earlier and more explicit cross-area review.

Keith

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 16 Dec 2012 22:38:53 +0000

Hiya,

> see earlier and more explicit cross-area review.

All reasonable stuff and the draft says that this won't be suitable
for all cases. If you've better wording to suggest that'd be welcome.

Cheers,
S.