Subject: Idea for a process experiment to reward running code...

From: Stephen Farrell <[email-masked]>
Date: Sat, 01 Dec 2012 20:12:27 +0000

Hi all,

I've just posted an idea [1] for a small process improvement.
If it doesn't seem crazy I'll try pursue it with the IESG as
an RFC 3933 process experiment. If its universally hated then
that's fine, it can die.

The IESG have seen (more-or-less) this already but it hasn't
be discussed, so this is just a proposal from me and has no
"official" status whatsoever.

Any comments, suggestions or better ideas are very welcome.
Feel free to send me comments off list for now, or on this
list I guess. If there's loads of email (always possible,
this being a process thing;-) we can move to some other list.

Regards,
Stephen.

[1] http://tools.ietf.org/id/draft-farrell-ft

---

From: Dave Crocker <[email-masked]>
Date: Sat, 01 Dec 2012 12:36:21 -0800

> this being a process thing;-) we can move to some other list.

What actual problem is this trying to solve?  I see the reference to a 
'reward', but wasn't aware that there is a perceived problem needing 
incentive to solve.

In additional, the incentive you are proposing seems to be getting to 
Proposed perhaps a month or so quicker?  It's difficult to see that as 
being sufficient incentive for creating implementations that would not 
otherwise we created.

The tradeoff is, of course, less time for final review.

d/
-- 
  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net

---

From: Melinda Shore <[email-masked]>
Date: Sat, 01 Dec 2012 12:00:36 -0900

> 'reward', but wasn't aware that there is a perceived problem needing

I gather this is one of those "everybody knows" problems, where
"everybody knows" that it takes what's perceived as too long to
get documents through the post-wglc/pre-publication process.  I
can see some value in parallelizing some processes.  I'd really
like to see AD review happen in parallel with working group last
call but I'm not sure that's possible given the already considerable
area direct workload.

There's probably some sort of sympathetic vibe running between
this document and recent discussion of nearly-cooked work being
brought to the IETF for standardization.  I think some sort of
variation on what Stephen's proposing is probably reasonable.
If somebody hasn't already documented how long it takes to get
through the various steps once a document is into wglc, it
would be worthwhile to start taking notes.

Melinda

---

From: SM <[email-masked]>
Date: Sat, 01 Dec 2012 13:06:58 -0800

Hi Stephen,
>an RFC 3933 process experiment. If its universally hated then

In Section 1:

   "The idea here is not to save the universe, nor to boil
    any oceans."

It's better to boil an ocean than to kill a fish.  :-)

   "1.  Working group last call (WGLC), IETF last call (IETF-LC) and Area
        Director (AD) review all run in parallel over the same two-week
        period."

You could skip the WGLC as you have the Last Call.

In Section 3:

   "2.  Where there are two or more WG chairs, all need to agree to fast-
        track processing."

That can be rolled into item 1.

Could you ask an AD to sponsor this draft and generate the Last Call?

Regards,
-sm

P.S. Make the draft experimental.  Add a one-year timer.

---

From: Yoav Nir <[email-masked]>
Date: Sat, 1 Dec 2012 21:21:56 +0000

On Dec 1, 2012, at 10:36 PM, Dave Crocker <[email-masked]> wrote:

> 
> What actual problem is this trying to solve?  I see the reference to a 'reward', but wasn't aware that there is a perceived problem needing incentive to solve.
> 

I think the problem is in the subject line. Documents go through working groups and the IETF proposing standards that have never been implemented, and may never be implemented.

Why do they bother?  Maybe it looked like a good idea at the time. Maybe they want an RFC hoping to drive adoption. Maybe the want an RFC for their personal vanity, or to augment their LinkedIn profile.

Regardless of the reason, I agree that the IETF's resources are better spent on stuff that will make it into running code. Not sure this is the right incentive, though. Waiting two more months, and probably making one extra revision to the document seems to be a lower investment than actually implementing (incentives only apply to people who would otherwise not implement their proposal)

Yoav

---

From: Dave Crocker <[email-masked]>
Date: Sat, 01 Dec 2012 14:13:26 -0800

On 12/1/2012 1:00 PM, Melinda Shore wrote:
> get documents through the post-wglc/pre-publication process.

Yes.  Longstanding opinion held by many folk.  Might even be valid.

The problem is a failure to look carefully at wg lifecycle and consider 
where meaningful -- as opposed to 'appealing' -- improvements can be made.

At a minimum, any proposal for change should be expected to justify the 
specific problem it is claiming to solve -- that is, to establish the 
context that makes clear the problem is real and serious -- and that the 
proposed solution is also likely to have meaningful benefit.

I share the frustration about lengthy standardization, and particularly 
with delays at the end.  And certainly there is nothing wrong with 
adding parallelism where it makes sense.

However absent a consideration of the lifecycle, the current proposal is 
a random point change, quite possibly an example of looking for lost 
keys under a lamppost because that's where it's easiest to see.

> brought to the IETF for standardization.

rumblings of free-floating dis-ease, perhaps.  but are they really related?

> would be worthwhile to start taking notes.

If a wg takes 2 years to get into wglc, a difference of a month doesn't 
matter, does it?  That's why I mean about total lifecycle.  Otherwise 
we're committing the classic system engineering error of inappropriate 
local optimization.

d/

---

From: Barry Leiba <[email-masked]>
Date: Sat, 1 Dec 2012 17:22:08 -0500

> [1] http://tools.ietf.org/id/draft-farrell-ft

I have a serious problem with the premise of the proposal.

The short version is that if the process we're talking about is
useful, we should not shortcut it as a "reward" for anything.  And if
it's not useful, we should not be doing it, regardless of whether we
want to provide incentives or not.

In particular:

   1.  Working group last call (WGLC), IETF last call (IETF-LC) and Area
       Director (AD) review all run in parallel over the same two-week
       period.

Note that WGLC is not a part of our formal process, and that AD
Evaluation can take as little time as it takes for the responsible AD
to click the "Go directly to Last Call Requested" button in the
datatracker.  Therefore, making this step happen today is entirely up
to the working group chairs and the responsible AD.  Any chair who
thinks that a document has had enough review in the WG already can
send the document to the AD directly.  That chair can also ask the AD
to expedite her review and get Last Call started ASAP, and it's up to
the AD to decide whether that's appropriate.  We don't need a process
experiment for that.

That said, AD Evaluation often (VERY often) turns up things that would
otherwise come up in Last Call comments, including in directorate and
review team reviews.  It often corrects problems or gaps in the
protocol specification, issues with IANA requests, security issues,
and so on... which would take up a lot of other people's time as the
Last Call review goes on.  An AD who has been following the document
already might be happy going directly to Last Call Requested with just
a quick glance at the document (as I have with some documents), but
might have extensive comments about other documents, regardless of
whether there are implementations or not.

Kicking off Last Call is something that requires AD judgment, not some
sort of "reward" system.

   2.  Only comments that would be "DISCUSS-worthy" according to the
       IESG Discuss Criteria [DCRIT] need be handled during last call.
       Other comments can be handled or not, at the authors/editors
       discretion.

I'm puzzled by this.  First, a lot of things fall under the DISCUSS
criteria, but ADs decide not to give them DISCUSS status, depending
upon the situation.  I guess in item 3 you cover who decides this, but
as it's specified here it seems entirely too fluffy.  And it appears
to devalue the non-blocking comments more than they already are.
Again, if our process of making comments *of all sorts* has value,
then we shouldn't short-cut it.  It's already the case that the
responsible AD and the IESG as a whole get to decide which Last Call
comments "need to be handled" and which don't, so this really isn't a
process change either.  But I wouldn't want to say outright that
last-call comments that don't meet DISCUSS criterial will likely be
ignored in some "fast path" process that "rewards" a protocol for
having implementations.

   3.  After fast-track last-call, the document must either be returned
       to the WG, or else enter IESG review as soon as any changes
       required are made.  The relevant AD makes the decision as to
       whether changes are required and when those are completed
       sufficiently to move to the next stage.

This is no different to what we normally do: after Last Call ends, the
document goes into "Waiting for AD Go-Ahead" state, and the
responsible AD decides whether it's ready for IESG Evaluation or not.

I very strongly dislike the idea of saying that we'll "reward"
something by trimming (by a mere two or three weeks; what real value
does that have?) a process that we consider as adding value to our
documents.  And, again, if we think it doesn't add value, then let's
just do this as standard procedure, as we already can today, with no
process experiment at all.

Just tell all your WGCs that they can skip WGLC, and you'll send their
docs straight into Last Call as soon as you get them.

Barry

---

From: Stephen Farrell <[email-masked]>
Date: Sat, 01 Dec 2012 22:56:03 +0000

Hi Dave,

On 12/01/2012 10:13 PM, Dave Crocker wrote:
> specific problem it is claiming to solve -- 

Disagree. RFC 3933 says:

"A statement of the problem expected to be resolved is
      desirable but not required..."

There's a reason for that IMO - all proposed process changes seem
to generate *lots* of comment that there's a better problem to
solve elsewhere.

> that is, to establish the

You may be right, I don't make any claim that this is going to
be super-good. OTOH maybe this is worth trying to see if we like
it or not.

Cheers,
S.

---

From: Stephen Farrell <[email-masked]>
Date: Sat, 01 Dec 2012 23:04:33 +0000

On 12/01/2012 10:22 PM, Barry Leiba wrote:
> useful, we should not shortcut it as a "reward" for anything.  

I disagree. The process is neither useful nor just a PITA. Its both,
depending on what document and point of view you pick. So yours is
a false dichotomy I think. And even the process now were useful 100%
of the time, shortcutting it might still be an improvement. So I just
don't think you have a winning argument based on logic.

> And if
> Note that WGLC is not a part of our formal process, 

Whatever. Its a reality.

> and that AD
> datatracker.  

Or longer. Much longer sometimes if an AD is slow or
authors need to re-spin and are slow. Both happen.

I had meant to try measure that from Jari's data before shooting
the draft out but didn't get a chance. My bet is that if this
worked well, it might save about 6 weeks on average, for drafts
where its suitable. Or maybe more. We should see if we can get
some numbers though, I do agree with that.

But more blow-by-blow responses will only likely rathole us
here so I'll stop here for now. (But I'm happy to do send that
if you think its useful.)

What's wrong with doing the experiment and finding out if its
better, worse or not detectably different?

I, and I believe lots of us, do want to encourage running code
more than now. This is one attempt to help with that. Why not
try it and see?

S.

> Therefore, making this step happen today is entirely up

---

From: Dave Crocker <[email-masked]>
Date: Sat, 01 Dec 2012 15:13:58 -0800

Stephen,

>        desirable but not required..."

The IETF has become a curious place.  This is twice in two days I've 
cited pragmatics and been taken for citing formal requirements. Believe 
it or not "should be expected" is pretty lousy language to invoke for 
citing formal organizational requirements.

IMO a proposal that does not justify itself with a clear statement of 
the problem, the seriousness of the problem, and a basis for believing 
that the proposed solution will matter is frankly not a very serious 
proposal.  It has done none of the hard work of distinguishing itself 
from myriad other, appealing proposals that might randomly be generated.

I do realize that IETF culture has come to enjoy making and pursuing 
proposals that satisfy none of these criteria.  That's why I've taken to 
noting that we use the word "experiment" to justify a failure to do 
adequate homework or be accountable for the outcome.  (Or even be able 
to measure the outcome.)

> to generate *lots* of comment that there's a better problem to

Stephen, what you seem to be saying is that it isn't reasonable to 
expect serious analysis in the formulation of a proposal, when it is put 
forward, because other folk might/will prefer a different -- and equally 
ill-formulated proposal?

It certainly is a pain to have to provide meaningful justification.

On the other hand, it's pretty irresponsible not to.

> be super-good. OTOH maybe this is worth trying to see if we like

Maybe more cookies at breaks will produce better IETF work.

Maybe a posting limit of one message per day will produce better IETF work.

Maybe...

There's an infinite number of stray proposals any of us could formulate. 
  But they aren't free to pursue.

And relying on the whim of popular fashion -- oh, let's all follow 
today's pretty flashing proposal -- is mostly distracting from finding 
and fixing real, strategic issues.

d/

---

From: Stephen Farrell <[email-masked]>
Date: Sat, 01 Dec 2012 23:21:17 +0000

> adequate homework or be accountable for the outcome.  (Or even be able

IMO IETF culture has become too process-driven. And yes, I do
interpret your question (which is not unreasonable on its face)
to be saying that a justification is formally required.

My reluctance to get into this is based on an opinion that process
change proposals with more words attached tend to just not happen,
so fewer words is better. And I also think you and many others are
well placed to decide if you think this is a worthwhile change or
not, independent of any reasons for which I propose it. So I really
do think all the motivation text that'd be perfect fodder for
endless discussion is neither necessary nor useful here.

Cheers,
S.

---

From: Melinda Shore <[email-masked]>
Date: Sat, 01 Dec 2012 14:51:55 -0900

On 12/1/12 2:21 PM, Stephen Farrell wrote:
> so fewer words is better.

I think that's actually a pretty terrible reason.  The goal is
not to get the proposal through, the goal is to improve
something.  That said, while I don't like change just for the
sake of change I think that change for the sake of betterment is
a very good thing indeed.  The main thing is that I've been trying
to figure out where the harm would be here and I haven't been
able to identify anything substantive.  I'd be good with giving
this a limited run and seeing how it goes.

Melinda

---

From: Barry Leiba <[email-masked]>
Date: Sat, 1 Dec 2012 19:21:53 -0500

> don't think you have a winning argument based on logic.

Let me try saying it again, with different phrasing:
If, for any particular document, what you propose is the right way to
handle that document, that handling should not be handed out as a
"reward".  And if, for that document, this is *not* the right way to
handle it, it's a bad idea to do so anyway.

>> Note that WGLC is not a part of our formal process,

No, Stephen, this isn't a "whatever".  You're proposing an experiment
for changing process.  I'm reminding you that what you're proposing is
not a change in formal process, but something you can do *right now*,
today, if you and a WGC want to.  Please don't dismiss that with
"Whatever."

> authors need to re-spin and are slow. Both happen.

You're entirely missing the point: of *course* it *can* take longer.
My point is that it already doesn't have to, if the WGC and the
responsible AD agree that it shouldn't.

> more than now. This is one attempt to help with that. Why not

Because as a "reward" for claiming to have running code, I think it's
a terrible idea.  As a way of handling the process for documents where
it makes sense to, I think it's fine.  If you want the criteria for
your WGs to be "You have to have running code," knock yourself out.
Any other ADs who want to do it that way can join you.  If you want to
codify that for everyone, I think you need to do a lot more in the
line of criteria for adequate implementations, and so on.  If I spend
half a day knocking out some crappy untested code, is that good
enough?  If not, how much testing does it need?  I'm pretty sure we
don't want to go there.

Barry

---

From: Barry Leiba <[email-masked]>
Date: Sat, 1 Dec 2012 19:27:41 -0500

> your WGs to be "You have to have running code," knock yourself out.

Oh, and I did forget one other thing I'd meant to say:
What, actually is the experiment?  It seems to me the experiment is to
see if this actually encourages more "running code", which means that
to have a meaningful experiment you'll need to be able to compare how
many protocol documents represent running code (for some definition of
that) *today*, and then look at the same thing again after we've been
doing this for a while.

Do you have any baseline measurements?

Barry

---

From: Hector Santos <[email-masked]>
Date: Sat, 01 Dec 2012 19:47:20 -0500

This proposal sounds interesting but couldn't it run into conflicts 
when there are competition in running code?   Who's running code do 
you fast track?  How does it apply in the protocol updates area, i.e. 
BIS work?

This proposal and thread, similar to the recent others, all seem to be 
looking for endorsing methods for lack of a better term, "rubber 
stamping" and fast tracking work items, in particular when there are 
WG related barriers holding back the WG work item(s) production progress.

I personally do not have an issue with expediting work when the proper 
protocol engineering is done and the adequate engineering reviews are 
done, in fact, I depend on the IETF/IESG engineering do this work.  I 
depend on the long time wisdom and engineering judgment of the IETF 
leaders and reviewers to watch over sensitive engineering issues, 
including conflict of interest related matters.

In the end, we are talking about trusting the process.  If IETF 
participants, especially those who don't attend meetings and long 
participated remotely or via mailing list, lose faith in the WG 
process, these process change proposals may expedite IETF work, but 
they may also handicap the potential of a proposed standard, change 
and industry following.

--
HLS

-- 
HLS

---

From: Hector Santos <[email-masked]>
Date: Sat, 01 Dec 2012 21:19:21 -0500

> this a limited run and seeing how it goes.

I would be more comfortable with the proposal if its applied only to 
new protocols, ideas, methods, etc.  The conflicts come when applying 
it to established standards, methods or BIS work.  While its possible 
to fast track the obvious, where there is potential issues, the 
proposal needs to be sensitive to fast tracking problematic issues.

-- 
HLS

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 02 Dec 2012 08:41:11 +0000

Another condition for a fast track must be the absence of
unresolved IPR disclosures. I can see a big risk here - that
someone will use the fast track procedure to game the IPR
disclosure rule. First, release your open source code, using
an open source licence that doesn't assert the absence of IPR.
Then, post the 00 draft and persuade an AD to fast track it.
(The draft assumes there is a WG involved, but the IETF process
does not require a WG.) Then, "discover" the IPR and disclose it.
All within 3 weeks.

   Brian

On 01/12/2012 20:12, Stephen Farrell wrote:
...

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 02 Dec 2012 14:33:40 +0000

Hiya,

> Could you ask an AD to sponsor this draft and generate the Last Call?

Bit early yet. I'd like to know what folks think and hopefully
improve the thing via others' good ideas.

> P.S. Make the draft experimental.  Add a one-year timer.

Done.
Ta,
S.

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 02 Dec 2012 14:37:14 +0000

Hi Hector,

On 12/02/2012 12:47 AM, Hector Santos wrote:
> This proposal sounds interesting but couldn't it run into conflicts when
> there are competition in running code?   Who's running code do you fast
> track?  How does it apply in the protocol updates area, i.e. BIS work?

Good point. I clarified that its just drafts heading to PS and
-bis drafts are just fine. (Might actually be well-suited for those,
and I'd not thought about that.)

> stamping" and fast tracking work items, in particular when there are WG
> related barriers holding back the WG work item(s) production progress.

Well, I'm after something where fast-track != rubber-stamp, but I
do agree that that's a hard balance to achieve. I'm not claiming that
this draft does achieve that balance, but such a balance is a goal.

Cheers,
S.

> participated remotely or via mailing list, lose faith in the WG process,
> these process change proposals may expedite IETF work, but they may also
> handicap the potential of a proposed standard, change and industry
> following.
> 
> -- 
> HLS
> 
> Stephen Farrell wrote:
>> Hi all,
>>

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 02 Dec 2012 14:41:55 +0000

Hi Brian,

> does not require a WG.) Then, "discover" the IPR and disclose it.

Ah good point. This is only intended for WG documents, but the
late-IPR point is a good one nonetheless. I've added a sentence
saying any late-IPR declaration sends the draft back to the WG.
That's a potential DoS of course, but not a new one really.

Ta,
S.

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 02 Dec 2012 14:48:19 +0000

Hiya,

> it makes sense to, I think it's fine.  

I don't get that distinction to be honest. Maybe we interpret the
word reward differently here. I'm using it with only positive
connotations but the scare quotes above seem to imply you're not.

> If you want the criteria for
> codify that for everyone, 

I do. I want this to be something where WG chairs can take the
initiative (not ADs) and where IETF participants can hassle their
WG chairs to use this. So I don't want it to be an AD-driven
thing really.

> I think you need to do a lot more in the
> line of criteria for adequate implementations, and so on.  

Fair enough. Be interested in more on that.

> If I spend

Perhaps. As of now, I'm just saying that the WG chairs (all of 'em)
and the responsible AD need to accept that the code is ok for the
purpose. I think I can argue that that's enough, but would welcome
better ideas there or ideas about things they should take into
account that'd be worth including here.

Cheers,
S.

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 02 Dec 2012 15:00:31 +0000

On 12/01/2012 11:51 PM, Melinda Shore wrote:
> I think that's actually a pretty terrible reason.  

Well, I think its an ok reason for not fully answering Dave's
question in this case, given the latitude offered by RFC 3933.
Tactical reticence like this would not be a reason to adopt
anything, I agree.

> The goal is

Great.
S.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 03 Dec 2012 15:01:00 +0900

One of the advantages of a standards organization such as the IETF is 
cross-concern review. For the IETF, one very strong cross-concern is 
security. Another one (also for my personally) is internationalization. 
Another, more vague one, is general architecture. Early running code is 
very often (not always) characterized by the fact that such 
cross-concerns are actively or passively ignored.

I had a look at your draft and checked for "security" and 
"internationalization", but only found the former, and not not in a 
discussion about how this proposal would make sure that cross-concerns 
are adequately addressed.

Regards,   Martin.

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 03 Dec 2012 08:06:56 +0000

> [1] http://tools.ietf.org/id/draft-farrell-ft
I find this a worrying proposal.

In the just-in-time world that we live in, too much of the review is 
already tail driven. Reducing the time that people have to notice that a 
doc is up for final review and then clear enough time in their calendar 
against a myriad of other tasks makes it more likely that the quality of 
review will diminish and hence the quality of our documents will diminish.

I would hate for us to act like an SDO that regards publication 
milestones as crucial and ship the draft regardless of the state of the 
technical design.

I would also note that sometimes it just takes time during review to 
mull over the full implications of the design and to surface the issues. 
With the current scheme if you miss a problem in WGLC, you can raise it 
during IETF LC.

- Stewart

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 03 Dec 2012 10:48:58 +0000

Hi Stewart,

> I find this a worrying proposal.

I hope I can convince you that its not.

> already tail driven. 

I don't believe this touches on that at all (see below).

> Reducing the time that people have to notice that a

But this doesn't do that for IETF LC at all! Everyone
not involved in the WG gets just the same notice as now.

I guess I did assume that the chairs would not try surprise the
WG with this. I can add text along those lines.

So I don't think there's any bad effect here really.

> milestones as crucial and ship the draft regardless of the state of the

That's not at all what's proposed. Drafts still get worked
on in the WG and go through the same IESG evaluation and
IETF LC.

What I hope is different is that drafts taking this optional
approach are higher quality, being based on running code.

> With the current scheme if you miss a problem in WGLC, you can raise it

Sure. And everyone is free to keep using that of course. To try
this process you need to get the WG chairs and have a happy AD
and running code. Really, I don't see that this would lower the
quality of our output (considering some of the drafts we do see
that may never have running code), rather the opposite. And the
main danger here I suspect is that this never gets used.

Does that make you less worried?

Cheers,
S.

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 03 Dec 2012 11:02:15 +0000

> cross-concerns are actively or passively ignored.

An excellent point. The fact that a hack works, and can be implemented,
does not alter the fact that it's a hack. This is the sort of thing that
cross-area review is supposed to look for. As a gen-art reviewer, I am
sometimes surprised by what gets through to Last Call in the regular
process - if the whole review process is squeezed down to a couple
of weeks, we will definitely miss cross-area issues.

Encouraging running code is a Good Thing. Publishing sloppy specifications
is a Bad Thing.

The Interop show network used to be a Very Good Thing. We've lost that,
though I was delighted to see some actual running code at Bits-n-Bytes
in Atlanta. More please. Maybe a prize for Best Demo?

   Brian

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 03 Dec 2012 11:28:49 +0000

>> cross-concerns are actively or passively ignored.

(Odd, I've not seen Martin's mail yet.)

> of weeks, we will definitely miss cross-area issues.

Fair enough. Those reviews (at least secdir ones) are supposed to
be done during IETF LC, but that doesn't always happen and I'm not
sure if the same timing applies to 'em all. I've added this:

        The timers associated with fast-track processing do increase the
        burden on cross-area review teams.  At present such reviews are
        supposed to be done during IETF LC, but some useful reviews are
        not received until after the end of IETF LC.  As is currently
        the case, the responsible AD and IESG will have to deal with
        such reviews as they are received.  A part of the experiment
        here will be to see if fast-track processing significantly
        impacts on these reviews.

> Encouraging running code is a Good Thing. Publishing sloppy specifications

Sure. I guess I'd hope that we push back on sloppy specs as
usual, but that the running code might make that less likely,
or at least more likely to be just editorial.

Cheers,
S.

---

From: John C Klensin <[email-masked]>
Date: Mon, 03 Dec 2012 07:29:55 -0500

--On Monday, December 03, 2012 11:28 +0000 Stephen Farrell
<[email-masked]> wrote:

>> Encouraging running code is a Good Thing. Publishing sloppy
>> specifications is a Bad Thing.
> 

Stephen,

I agree with Brian, but want to reinforce thinking about this a
little differently and in a way that may draw several other
sub-threads together.  The IETF is functioning here as a
standards body, not an certification body for implementability.
Knowing that something can be implemented, and implemented at
production quality, has traditionally been an IETF hallmark (and
hence part of Dave Clark's slogan) relative to other bodies who
have standardized the un-implementable.  IMO, when we approve
something as a Proposed Standard, it suggests that we have
evaluated and approved it along three quite separate dimensions:

	(1) Utility of the idea as something that should be
	implemented (or at least as a decent and reasonable way
	to do something)
	
	(2) Quality of the protocol and whether the
	specification adequately describes it.
	
	(3) Whether the specification can be implemented

Failures are possible on any of the three.  It is possible for a
well-specified and implementable idea to be utterly useless.  It
is possible to have a useful and well-designed protocol, with
implementations by insiders, to be so badly described that it is
unlikely that anyone else could figure out how to implement it.
It is, as Brian pointed out in his recent note, possibly to have
a hack-level implementation and/or one that doesn't work in the
edge cases.

Using any of those three in a way that enables shortcuts around
the other two puts us at risk or creating bad standards.  Even a
few bad standards could be used to hold the IETF up to ridicule
in ways that would weaken everything else that we do for a long
time... and the fact that we saw them as the result of a process
experience would be no protection at all.  Worse from the
standpoint of a speed-up procedure, someone's discovering a
problem after a hasty IESG approval could easily lead to an
appeal on the grounds that the process used did not allow for
adequate review, a result that would cost far more time than the
difference between the current and proposed procedure.

As others have pointed out, there are lots of other ways to
speed things up, most of which fall within the discretion of
individual WGs, ADs, and the IESG without any changes at all.
At least a few of them would involve "changes" back to what RFC
2026 already specifies, including treating Proposed Standards as
Proposed Standards, and eliminating, e.g., days or weeks or
post-IETF-Last-Call AD nitpicking over text in favor of
practicing the intent of the "no known technical defects" rule
of 2026 and letting the RFC Editor do their job.  But "you claim
to have done consideration 3, therefore you get to shortcut
considerations 1 and 2" should not be one of them.

best,
   john

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 03 Dec 2012 13:42:48 +0000

Hi John,

> considerations 1 and 2" should not be one of them.

So are "hasty" and "shortcut" maybe pejorative ways of
saying that we cut out some of the "nitpicking" ;-)

Anyway, I'm not suggesting we forget #1 or #2 at all and
yes some or maybe all of what's proposed could already
be done but I think its notable that that is not being
done, so the proposal is a 3933 experiment to see if
putting some more emphasis on #3 will work or not.

So I do hope we run the experiment.

S.

---

From: Barry Leiba <[email-masked]>
Date: Mon, 3 Dec 2012 09:25:16 -0500

> not involved in the WG gets just the same notice as now.

This is true.

> approach are higher quality, being based on running code.

This is a stretch, and that *unprovable* assumption also bothers me.
We have lots of running code that implements specs badly.  We have
lots of running code that implements exactly what's in the spec and
misses everything that's missing (consider Martin's comment about
security and i18n).  We have lots of running code that doesn't
interoperate with other running code.  Your proposal specifically does
NOT require any testing of the running code, nor any interoperability
demonstrations (nor would I want it to).

Running code, when it's an organic part of the document development,
is undoubtedly a good thing -- it doesn't make everything right, but,
yes, it does do *some* spec validation and probably does help spec
quality.

But code that's written as part of a rote process, just to achieve
another check-box on the shepherd writeup and justify special handling
is not likely to provide any of those benefits.

Barry

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 03 Dec 2012 14:28:47 +0000

> yes, it does do *some* spec validation and probably does help spec

Fully agree. And this kind of experiment may encourage that
good thing some more. Or not. We'll not see if we don't try.
We may see if we do try. I think its worth trying. (That's
fairly obvious I guess:-)

S.

---

From: Barry Leiba <[email-masked]>
Date: Mon, 3 Dec 2012 09:36:23 -0500

> fairly obvious I guess:-)

Or we'll just waste time sticking in some side-process that isn't
necessary (all of this can already been done under current process,
with no experiment).  As I've said on the IESG list, I think this can
be far better done with an IESG statement that says that
implementation, testing, and deployment should be considered as we
(the community and the IESG) evaluate documents.  Then we just make
sure we facilitate the process instead of getting in the way of it.

That's a better promise than saying we'll cut out three or four weeks
of review time for a document that probably took 16 months to develop.

Barry

---

From: Elwyn Davies <[email-masked]>
Date: Mon, 03 Dec 2012 14:38:45 +0000

Hi.

> of weeks, we will definitely miss cross-area issues.

I also have the experience as a gen-art reviewer of seeing some pretty
awful pieces of work making through even to IESG review.  

However, I don't think that a short last call cycle need necessarily
compromise cross-area review. There has always been the possibility for
authors or wg chairs to request a early gen-art review with a view to
checking out whether something is in good shape cross-area and for
non-specialists.  This facility is not much used (I think I have done 3
in 8 years on the gen-art team) but it is there, and I guess the team
could cope with a few more since it doesn't drastically alter the total
workload. So it would be entirely possible for a draft that might be
fast-tracked to get some early review.

Given that there is also open source code, reviewers have the chance to
take a look at that and see the degree of hackiness involved.  

So I'd be for trying the experiment - and asking some cross-area
reviewers to take an early sniff.

Regards,
Elwyn

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 3 Dec 2012 15:39:55 +0100

> is not likely to provide any of those benefits.

+1.

As somebody who tends to think about security implications, the first thing that came to my mind when reading this was how this experiment would be used to game the system.  The results are so obvious I won't bother to write them down.

Society has a lot of experience with cases where, what used to be useful, but gameable indicators, are turned into measuring instruments with great effect.
Analogous to all these cases, I think the main effect of this proposal would be to destroy the remnant of usefulness that "running code" has in our current (living) process.

Grüße, Carsten

---

From: Elwyn Davies <[email-masked]>
Date: Mon, 03 Dec 2012 14:42:13 +0000

> fairly obvious I guess:-)

Another thing which it might help with is weeding out gratuitous
proliferation of options.

Regards,
Elwyn

---

From: Dave Crocker <[email-masked]>
Date: Mon, 03 Dec 2012 06:45:46 -0800

On 12/3/2012 6:36 AM, Barry Leiba wrote:
> with no experiment).
...
> of review time for a document that probably took 16 months to develop.

+10

d/

ps. given how creative human can be, in generating unintended 
consequences, an incentive to create running code /before/ going to 
Proposed could serve to /increase/ the barrier to standardization, by 
creating a perception that having running code is pragmatically required 
to get past the IESG.

d/

---

From: Barry Leiba <[email-masked]>
Date: Mon, 3 Dec 2012 09:50:38 -0500

Elwyn says...

> fast-tracked to get some early review.

Do you really think it's likely that a chair who's trying to
fast-track a document will likely go out asking for early GenART,
SecDir, AppDir, and OpsDir reviews?

> take a look at that and see the degree of hackiness involved.

This brings up something else that I've discussed with Stephen off list:

This proposal requires an *open source* implementation.  A robust set
of proprietary implementations, along with several interoperability
testing events, would not qualify.  But some untested open source that
someone knocked off in an afternoon would.  I find that odd.

I understand the desire to be able to inspect the implemented code,
but that would seem to rule out too many things, including stuff at
lower stack layers.  If the AD can look at the
implementations/deployments and talk with the implementors, the
openness of the source code is not an issue.  I'd be happy -- ecstatic
-- to have two or three proprietary, very-closed-source
implementations of most of the IMAP extensions, and have *no* need to
see the source code.  And I suspect that having shipping code in
routers is a good thing for validating new routing protocols, and also
suspect that we're not likely to be inspecting the source code of
those.

I'd really prefer if we'd talk about open source being desirable, but
not having it be necessary.

Barry

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 03 Dec 2012 14:53:56 +0000

> not having it be necessary.

Yep. I got another comment to that effect as well.
I'll try address that (but that's not done yet).

FWIW, a working copy is available [1] that has a
bunch of changes resulting from comments already
received.

Cheers,
S.

[1] http://down.dsg.cs.tcd.ie/misc/draft-farrell-ft.txt

---

From: Elwyn Davies <[email-masked]>
Date: Mon, 03 Dec 2012 15:15:38 +0000

Barry responded...

> SecDir, AppDir, and OpsDir reviews?
A few do already.  But seriously, if the wg chair(s) actually have an
interest in the technology and feel there is a valid reason for getting
something decent out into the world quickly then they could well be
motivated to do just that.  If our wg chairs are really just
bureaucratic process minders then it's time we found some different
ones, but I don't think they are. Maybe this is just a plea for a bit
more parallel processing and a few less silos.

/Elwyn

---

From: Barry Leiba <[email-masked]>
Date: Mon, 3 Dec 2012 10:20:39 -0500

> motivated to do just that.

I agree.  I'm concerned that when the focus is on "fast-track", such
that someone is trying to trim a few weeks off a many-month process,
this could be even less likely.  But this is speculation; I have no
data.

> If our wg chairs are really just

I don't think they are either, and I absolutely agree that we should
do more parallel processing and not put serial process in the way of
document handling.  That's never been at issue for me.

Barry

---

From: Sam Hartman <[email-masked]>
Date: Mon, 03 Dec 2012 11:21:12 -0500

>>>>> "Stephen" == Stephen Farrell <[email-masked]> writes:

    Stephen> On 12/03/2012 02:50 PM, Barry Leiba wrote:
    >> I'd really prefer if we'd talk about open source being desirable,
    >> but not having it be necessary.

    Stephen> Yep. I got another comment to that effect as well.  I'll
    Stephen> try address that (but that's not done yet).

For myself, I think the requirement for open-source is very good for
this experiment, or something like it.

People can examine the open-source implementation and consider the
question of whether the implementation explores enough of the edge cases
that a process short-cut is appropriate.  I think that's important in
this, so I'd be a lot more comfortable with this with an open-source
requirement.

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 03 Dec 2012 16:22:57 +0000

> this, so I'd be a lot more comfortable with this with an open-source

Me too. But I guess others are concerned about that. Anyway, if/when
this gets an IETF LC, we can see if the wording's ok for folks.

Cheers,
S.

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Mon, 03 Dec 2012 08:41:27 -0800

On 12/01/2012 12:12 PM, Stephen Farrell wrote:
> 
> Hi all,
> 
> I've just posted an idea [1] for a small process improvement. If it doesn't
> seem crazy I'll try pursue it with the IESG as an RFC 3933 process
> experiment. If its universally hated then that's fine, it can die.
> 
> The IESG have seen (more-or-less) this already but it hasn't be discussed,
> so this is just a proposal from me and has no "official" status
> whatsoever.
> 
> Any comments, suggestions or better ideas are very welcome. Feel free to
> send me comments off list for now, or on this list I guess. If there's
> loads of email (always possible, this being a process thing;-) we can move
> to some other list.
> 
> Regards, Stephen.
> 

I support this idea, but I think that free software should also be considered
as part of this experiment (free software and open source are not synonymous).
 Using the acronym FOSS and defining it as Free or Open Source Software in the
document would achieve this.

Thanks.

- -- 
Marc Petit-Huguenin
Email: [email-masked]
Blog: http://blog.marc.petit-huguenin.org
Profile: http://www.linkedin.com/in/petithug

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 03 Dec 2012 16:54:58 +0000

> document would achieve this.

Fair point. OTOH, there are folks who want to not require open-source
too. For now, I'd gone with "ideally open-source." As I said to Sam,
let's deal with that, if need be, at IETF LC.

In the meantime, I've posted -01 [1] with a bunch of changes. Do let
me know if I forgot to ack someone and as before all comments
(inevitable, this being a process thing;-) are very welcome.

Thanks,
S.

[1] http://tools.ietf.org/html/draft-farrell-ft-01

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Mon, 03 Dec 2012 09:06:06 -0800

On 12/03/2012 08:54 AM, Stephen Farrell wrote:
>> I support this idea, but I think that free software should also be
>> considered as part of this experiment (free software and open source are
>> not synonymous). Using the acronym FOSS and defining it as Free or Open
>> Source Software in the document would achieve this.
> 
> Fair point. OTOH, there are folks who want to not require open-source too.
> For now, I'd gone with "ideally open-source." As I said to Sam, let's deal
> with that, if need be, at IETF LC.

OK, but note that I will oppose for this experiment the use of software that
is not either available in source form or accompanied with a conformance test
suite.

> 
> In the meantime, I've posted -01 [1] with a bunch of changes. Do let me
> know if I forgot to ack someone and as before all comments (inevitable,
> this being a process thing;-) are very welcome.
> 
> Thanks, S.
> 

- -- 
Marc Petit-Huguenin

---

From: Jari Arkko <[email-masked]>
Date: Mon, 03 Dec 2012 23:16:04 +0200

Brian, Martin,

> of weeks, we will definitely miss cross-area issues.

True.

> Encouraging running code is a Good Thing. Publishing sloppy specifications

Yes. Of course, it is difficult tradeoff. No hard and fast rules here. You have to publish a timely spec to make an impact, but you also have to get it right. But how timely? How right?

> in Atlanta. More please. Maybe a prize for Best Demo?

I've been thinking about something along those lines. Following up in a second e-mail...

jari

---

From: Jari Arkko <[email-masked]>
Date: Mon, 03 Dec 2012 23:19:03 +0200

Barry,

> As I've said on the IESG list, I think this can

+1

Jari

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 03 Dec 2012 16:36:21 -0500

And I will strongly oppose any IETF policy that treats commercial or 
proprietary code differently from Open Source code.

Out mantra is "running code".  We try to stay out of people's business 
models.

The presence of a implementation is a useful measure.  The presence of 
interoperability between two independent implementations is clearly a 
much better measure.
But reading the code is not a good measure of the spec.  An implementer 
familiar with the discussions in the WG is likely to get right tings 
that are under-specified.  Such an implementation does not tell us 
anything about the quality of the spec.  Should we require an 
implementation done from the spec without talking to the authors or WG? 
  That would measure the spec better.  But would seem to be an 
unreasonable requiremet.

he IETF, to the degree it wants to encourage something, wants to 
encourage implementation.  It ought to be up to the implementors whether 
they see value in an open source, closed source, or some other form of 
implementation.

One could argue that it does not matter, since the experiment is not 
going to change anything.  But I will assume that it matters, and will 
change something.  If so, it should be "running code" that is the issue.

Yours,
Joel

PS: "Free" for whatever definition of "Free"distinct from Open Source 
the suggester meant, would seem to be completely irrelevant to the IETF.

On 12/3/2012 12:06 PM, Marc Petit-Huguenin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
>
> iQIcBAEBCAAGBQJQvNv6AAoJECnERZXWan7EV3QP/2Tv/rn7+GSwK2KkB3RsnS00
> ryB5FH0drUdMr7c+TreHDXh/uR6HLYxB1Cg9rDd5CZQaSBQlTtwIKVUxMPOW805k
> Uf3/xi8d3DpXVTmlesMtmbT/GeE7S0hJ3wf6LUwl4OkctX8GMDNuaONNudarl3Qt
> sR+AJw8OU8KOFMqEn6fOqpLwY7OiQG3M6tuMwfNIm1Z0bHb/WAssF9l+J2ZTqq8L
> AmJpHPql65WVSPck+jRnYyNtfnSkVCmCQ1y5cMeNoEnm5tkEt5oeinsvDwEAWBB3
> 5YzbUb+q72SCEyTbeVOrwTU1cBU/CKzOH+8r0ykK51KlZ1acYetwh31U4h1jDpHd
> tNrwu1Z5wvKCa4b4dvzkiEBE83gAr6IGMDlPWNkEtotB1ZZd/MUpRNsSCzyivtl5
> zr1rwpMY/5w6LljGO4jN4ZXKtXErFovEQpTc+bUz9K23WYyZMkARehXZLkbEvpb8
> /MY1zGlgRSrGHr5Wn+k9egCeWqYjGNQedYpDcqWw7oQ5V956W+/AvKHo6jPGGweC
> wpW3Hm+AWc3gPZ2pgYYt8dVScscWLo0qDWW4daHwAqu0rvge7PUB8aVagE72fQUn
> fPEPhETD2QqZfXDeCtfNqOVS1/v+SYsAYxcGFYwu7NR51OROBkhMpWT5KDuaPp02
> OPRv2MwPcsanVoQ1uNfY
> =30oW
> -----END PGP SIGNATURE-----
>

---

From: Jari Arkko <[email-masked]>
Date: Tue, 04 Dec 2012 00:03:53 +0200

Stephen,

Thanks for proposing this. You know that I agree with you that improving IETF's ability to publish specifications relating to real code out there is important. It is important to come up with ideas in this space. I think there has been situations where IETF has strayed from the "rough consensus and running code" model.

But of course following the running code is not always easy. I have some perspectives on this, both from the IESG side and as a document author who in some cases had running code. Obviously we want IETF's resources spent on things that result in running code, and we want experiences from that running code reflected back into specifications. And we want those fast-running implementation teams to bring their specs to the IETF rather than publish them somewhere else.

But on the other side, there are issues. It is sometimes difficult to distinguish useless junk with several implementations from useful and stable technology that is going to change the Internet and which also has several implementations. And it is difficult to guess whether an implementation is the beginning of something great or the by-product of someone's pet project that also produced the specs. In many cases it is very cheap to write an implementation. You can also easily build an implementation and fund a few other folks to build theirs. And even if you have independent implementations, I've seen cases where there are seven implementations that inter-operate but the spec is still unimplementable for outsiders.

If you want to support IETF work that has implementations better, you have to decide where your focus is in the overall process. I suspect that the last calls make up a very small part of IETF's overall end-to-end ("draft-00-to-RFC") delay. I do not have good statistics of this at the moment, http://www.arkko.com/tools/lifecycle/wgdocs.html has some numbers from buggy code that has not been run since last spring. Those numbers seems to indicate WG process times in the order of two years and LC/IESG times in the order of half a year, plus some RFC editor time. Intuitively, the numbers match my own experience, so maybe they are in the ballpark.

One of the issues is that it is very rare for document to sail through the LC/IESG without any changes. From the same crappy statistics, http://www.arkko.com/tools/admeasurements/stat/goodprocessingtime_4.html and http://www.arkko.com/tools/admeasurements/stat/processingtime_4.html that's maybe 10% of all documents; 20% in some cases but those drafts probably have some RFC Editor notes associated with them. A document that does not change goes through quite fast, maybe in a month. A document that needs change also needs discussion, debate, and waiting for a revision. Hence the half a year. Two weeks out of that? Not a big difference. Two weeks out of 2-3 years for the entire spec development? Not noticeable.

All decisions involving how you give support or priority to running code involve difficult judgment calls. There's running code. Does that trump someone's opinion that the bits should be ordered differently? That the design should be changed to work better through NATs? That no security is needed? There is no hard and fast rule for these questions, you have to evaluate what the situation is.

It is not easy to support running code as part of the IETF day-to-day process. People test obviously, on their own. Many people bring implementations to IETF meetings. Some organize back room meetings to do even proper interops. We saw some code in bits-n-bytes, and there's been a few other demos as well. But interops are not within IETF scope. It is even difficult to acquire resources for testing in all cases. In Atlanta, we asked for a room the hallway was no longer sufficient. We did this in the last minute, got some well-deserved pushback from the NOC crew (who did a wonderful job and supported us anyway - thanks). But we also got some unexpected pushback from the IESG, who among other things told us that they can't help us with a room in any way until the spec is officially a WG document. But to get to that state we'd need some testing to improve the spec and convince people that the stuff works. So catch 22. Luckily, we got a room from IPSO Alliance, who stepped up to help.

Anyway, all this puts me roughly in the same camp as Barry. How about doing this instead:

1) Write that IESG statement or new RFC that reminds everyone that they should seriously consider what running code tells us when they weigh issues, decide to send documents forward, etc. Obviously we should not follow code blindly. Perhaps the statement could highlight some of the above issues, or the issues from Stephen's draft to make people aware of the complexity of the matter.

2) Re-focus ourselves in the WG and IESG reviews on the stuff that actually matters. DISCUSSes on what's in the document header of the I-D... do not matter as much as whether the thing actually works, or is i18n capable or secure. Yes, I know I've erred on this myself several times.

3) Remember that some of the most successful IETF specifications were things that were brought to us from the outside, with implementations and first standards revisions already done. It is not a bad model. You do not always have to start with a committee. You can finish with a committee.

4) Find ways to support running code in all stages of the IETF process. Could we support unofficial interops better in our meetings, or work with more interop organizations? Could we use bits-n-bytes to do more demos, and would this be useful in promoting running code?

Some more comments on details on your draft:

* The IPR return-to-WG process is actually part of existing operations for all drafts. Perhaps you could clarify this.

* I'm not sure why you have taken the position that the sponsoring AD does not have to vote Yes in these. I think that should be necessary even more than with other drafts. Were you thinking that there's some other AD who knows the case better? There are certainly cases where another AD knows more about a particular case than the one officially in charge of the WG. In those cases it might actually make sense for that other AD to take on the sponsoring role. But that seems orthogonal to fast tracking, and is already happening.

* I think you have placed perhaps too much emphasis on verifying the software. I think these are judgment calls, you figure out what is available, what its functionality is, how it has been tested, and so on. It is hard to codify that in a set of rules. Strict rules may actually rule out some things that would be useful. For instance, would several proprietary implementations with well-documented interop success qualify or not?

Jari

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 03 Dec 2012 22:41:13 +0000

Hi Jari,

I agree with almost all of what you say. I think we only disagree
in two places, and perhaps more about tactics than anything else.

The first is whether or not its worthwhile addressing the specific
bit of process my draft tackles. I obviously do think it is, even
though you correctly note there's a very modest gain in time.

I think we also disagree as to whether or not an IESG statement
would have any impact - I don't think there would be any good
effect at all for any IESG statement on the topic I can envisage
that wouldn't require community consensus. And as we've seen in
the last couple of days, even my modest idea for an experiment has
generated lots of words and a hugely rough consensus if any.

That seems to be the fate of all process discussions. So my
conclusion is that the best way towards our shared goal is via
small steps such as are represented by my proposal.

I'll look at your specific comments at the end, thanks for those.

S

On 12/03/2012 10:03 PM, Jari Arkko wrote:
> Stephen,
> 
> Thanks for proposing this. You know that I agree with you that improving
> IETF's ability to publish specifications relating to real code out there
> is important. It is important to come up with ideas in this space. I
> think there has been situations where IETF has strayed from the "rough
> consensus and running code" model.
> 
> But of course following the running code is not always easy. I have some
> perspectives on this, both from the IESG side and as a document author
> who in some cases had running code. Obviously we want IETF's resources
> spent on things that result in running code, and we want experiences
> from that running code reflected back into specifications. And we want
> those fast-running implementation teams to bring their specs to the IETF
> rather than publish them somewhere else.
> 
> But on the other side, there are issues. It is sometimes difficult to
> distinguish useless junk with several implementations from useful and
> stable technology that is going to change the Internet and which also
> has several implementations. And it is difficult to guess whether an
> implementation is the beginning of something great or the by-product of
> someone's pet project that also produced the specs. In many cases it is
> very cheap to write an implementation. You can also easily build an
> implementation and fund a few other folks to build theirs. And even if
> you have independent implementations, I've seen cases where there are
> seven implementations that inter-operate but the spec is still
> unimplementable for outsiders.
> 
> If you want to support IETF work that has implementations better, you
> have to decide where your focus is in the overall process. I suspect
> that the last calls make up a very small part of IETF's overall
> end-to-end ("draft-00-to-RFC") delay. I do not have good statistics of
> this at the moment, http://www.arkko.com/tools/lifecycle/wgdocs.html has
> some numbers from buggy code that has not been run since last spring.
> Those numbers seems to indicate WG process times in the order of two
> years and LC/IESG times in the order of half a year, plus some RFC
> editor time. Intuitively, the numbers match my own experience, so maybe
> they are in the ballpark.
> 
> One of the issues is that it is very rare for document to sail through
> the LC/IESG without any changes. From the same crappy statistics,
> http://www.arkko.com/tools/admeasurements/stat/goodprocessingtime_4.html
> and http://www.arkko.com/tools/admeasurements/stat/processingtime_4.html
> that's maybe 10% of all documents; 20% in some cases but those drafts
> probably have some RFC Editor notes associated with them. A document
> that does not change goes through quite fast, maybe in a month. A
> document that needs change also needs discussion, debate, and waiting
> for a revision. Hence the half a year. Two weeks out of that? Not a big
> difference. Two weeks out of 2-3 years for the entire spec development?
> Not noticeable.
> 
> All decisions involving how you give support or priority to running code
> involve difficult judgment calls. There's running code. Does that trump
> someone's opinion that the bits should be ordered differently? That the
> design should be changed to work better through NATs? That no security
> is needed? There is no hard and fast rule for these questions, you have
> to evaluate what the situation is.
> 
> It is not easy to support running code as part of the IETF day-to-day
> process. People test obviously, on their own. Many people bring
> implementations to IETF meetings. Some organize back room meetings to do
> even proper interops. We saw some code in bits-n-bytes, and there's been
> a few other demos as well. But interops are not within IETF scope. It is
> even difficult to acquire resources for testing in all cases. In
> Atlanta, we asked for a room the hallway was no longer sufficient. We
> did this in the last minute, got some well-deserved pushback from the
> NOC crew (who did a wonderful job and supported us anyway - thanks). But
> we also got some unexpected pushback from the IESG, who among other
> things told us that they can't help us with a room in any way until the
> spec is officially a WG document. But to get to that state we'd need
> some testing to improve the spec and convince people that the stuff
> works. So catch 22. Luckily, we got a room from IPSO Alliance, who
> stepped up to help.
> 
> 1) Write that IESG statement or new RFC that reminds everyone that they
> should seriously consider what running code tells us when they weigh
> issues, decide to send documents forward, etc. Obviously we should not
> follow code blindly. Perhaps the statement could highlight some of the
> above issues, or the issues from Stephen's draft to make people aware of
> the complexity of the matter.
> 
> 2) Re-focus ourselves in the WG and IESG reviews on the stuff that
> actually matters. DISCUSSes on what's in the document header of the
> I-D... do not matter as much as whether the thing actually works, or is
> i18n capable or secure. Yes, I know I've erred on this myself several
> times.
> 
> 3) Remember that some of the most successful IETF specifications were
> things that were brought to us from the outside, with implementations
> and first standards revisions already done. It is not a bad model. You
> do not always have to start with a committee. You can finish with a
> committee.
> 
> 4) Find ways to support running code in all stages of the IETF process.
> Could we support unofficial interops better in our meetings, or work
> with more interop organizations? Could we use bits-n-bytes to do more
> demos, and would this be useful in promoting running code?
> 
> * The IPR return-to-WG process is actually part of existing operations
> for all drafts. Perhaps you could clarify this.
> 
> * I'm not sure why you have taken the position that the sponsoring AD
> does not have to vote Yes in these. I think that should be necessary
> even more than with other drafts. Were you thinking that there's some
> other AD who knows the case better? There are certainly cases where
> another AD knows more about a particular case than the one officially in
> charge of the WG. In those cases it might actually make sense for that
> other AD to take on the sponsoring role. But that seems orthogonal to
> fast tracking, and is already happening.
> 
> * I think you have placed perhaps too much emphasis on verifying the
> software. I think these are judgment calls, you figure out what is
> available, what its functionality is, how it has been tested, and so on.
> It is hard to codify that in a set of rules. Strict rules may actually
> rule out some things that would be useful. For instance, would several
> proprietary implementations with well-documented interop success qualify
> or not?
> 
> Jari
> 
> 
> 
>

---

From: David Morris <[email-masked]>
Date: Mon, 3 Dec 2012 14:50:10 -0800 (PST)

> document would achieve this.

By my understanding free is not applicable in that software can be free 
without the source being available. The key in my mind to the open source
sugestion is the ability of others to examine the actual code for coverage
of edge concerns or whatever. Open source is available today under many
license agreements. Free is not a requirement to achieve the review
objective.

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 03 Dec 2012 22:53:06 +0000

> license agreements. Free is not a requirement to achieve the review

Folks, can we *please* not rat-hole on this? If the proposal gets to
an IETF LC, then I'll make sure to try find some wording that works for
folks who've expressed divergent opinions on the topic before the
start of IETF LC. So if you care about this, I'd ask that you wait for
an IETF LC before diving in, since a) it doesn't matter until then and
b) I promise, I'll try get some wording that works for all before then.

Thanks,
Stephen.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 04 Dec 2012 14:47:12 +0900

> take a look at that and see the degree of hackiness involved.

Well, yes. It's easy enough to evaluate stuff such as non-descriptive 
variable names, messy indenting, and weird comments.

But there's a catch here: There are so many programming languages that 
it's very well possible that the code is written in a language you don't 
understand. It may be using techniques that are totally accepted for 
that specific language, but in a way that is considered very hacky in 
the relevant community. Or something else. As an outsider, you may have 
no way to evaluate that.

Specs are in English (also a problem for some people, but I hope we can 
leave that to another thread :-), which is at least a single target.

Regards,   Martin.

---

From: Randy Bush <[email-masked]>
Date: Tue, 04 Dec 2012 07:09:07 -0500

>> Given that there is also open source code, reviewers have the chance
>> to take a look at that and see the degree of hackiness involved.
> 
> understand. 

i think we're ratholing here.  the point i get is that, if there is open
source code that suppliments the drafts sufficiently to lend confidence
in the implementability and implementation, then progress might be
accelerated.

notice the if

randy

---

From: Barry Leiba <[email-masked]>
Date: Tue, 4 Dec 2012 09:46:59 -0500

> in the implementability and implementation, then progress might be

But the point of "running code" in our nice, catchy slogan, is that
"running" doesn't mean simply that it runs.  It means that it's
actually *in use*, possibly for real, but at least in a test lab where
it's getting real use.  *Real* running code probably does lend such
confidence.  Code that's just been written because it helps your draft
get an extra tick mark in a box almost certainly doesn't, whether we
can see and understand the source code or not.

Barry

---

From: Danny McPherson <[email-masked]>
Date: Tue, 4 Dec 2012 23:05:30 -0500

> can see and understand the source code or not.

But Barry - if you're in the consultancy business and wear "running code and run away" shirts to IETF meetings; well. perspectives diverge...

-danny