Subject: RE: are we willing to do change how we do discussions in IETF? (was: moving from hosts to sponsors)

From: "Burger, Eric" <[email-masked]>
Date: Fri, 23 Jun 2006 16:18:40 -0400

I would offer that in *some* groups the running code bar is reasonable.
For example, in SIPPING, the problem space is pretty well-defined, and
there are third-party specifications and requirements out there.  There
have been way too many half-baked ideas floated for consideration, and
that has sucked the life blood out of the work group.

As I have mentioned on the SIPPING list, this is NOT a prescription for
all proposals or work groups in the IETF.  However, the idea is
reasonable for *some* work groups in the IETF.  I think raising the bar
isn't going to hurt.  The "they'll take the idea to another standards
body" doesn't hold water if there is no code / prototype / product to
promote.

-----Original Message-----
From: Keith Moore [mailto:[email-masked]] 
Sent: Friday, March 24, 2006 2:50 PM
To: Dave Cridland
Cc: [email-masked]; [email-masked]
Subject: Re: are we willing to do change how we do discussions in IETF?
(was: moving from hosts to sponsors)

[snip]

It may be that we place too much emphasis on running code in IETF today.
In ARPAnet days, when the user community was small and homogeneous but
platforms were diverse (different word/character sizes, different
character sets, different limitations of operating systems and
networking hardware), and goals for protocols were modest, merely being
able to implement a protocol across different platforms was one of the
biggest barriers to adoption.  In that environment,  being able to
demonstrate running code on multiple platforms was nearly sufficient to
demonstrate the viability of a protocol.  Besides, since the net was
small, it wasn't terribly hard to make changes should they be found to
be necessary.

These days running code serves as proof-of-concept and also as a way to
validate the specification.  It doesn't say anything about the quality
of the design - not efficiency, nor usability, nor scalability, nor
security.  etc.

[snip]

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Keith Moore <[email-masked]>
Date: Fri, 23 Jun 2006 17:24:11 -0400

On Fri, 23 Jun 2006 16:18:40 -0400

> I would offer that in *some* groups the running code bar is reasonable.

I would have little objection to requiring running code as a test of feasibility of a new idea.  I would object strongly to an argument that just because someone has running code, means it's a good indication of adequacy of the protocol...DKIM is a great example of a poorly designed protocol that has been justified by running code.

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Fri, 23 Jun 2006 17:43:05 -0400

On Friday, June 23, 2006 05:24:11 PM -0400 Keith Moore <[email-masked]> 
wrote:

> I would have little objection to requiring running code as a test of
> feasibility of a new idea.  I would object strongly to an argument that
> just because someone has running code, means it's a good indication of
> adequacy of the protocol.

Specific examples aside, I agree.  Running code should be a necessary 
condition for something to progress, but not a sufficient one.

-- Jeffrey T. Hutzelman (N3NHS) <[email-masked]>
   Sr. Research Systems Programmer
   School of Computer Science - Research Computing Facility
   Carnegie Mellon University - Pittsburgh, PA

---

From: Keith Moore <[email-masked]>
Date: Fri, 23 Jun 2006 17:56:01 -0400

> condition for something to progress, but not a sufficient one.

I think we would do well to require a reference implementation as a condition for Proposed Standards from new working groups or individual submitters...but there are other conditions that we should impose that are far more important.  Such as, a requirement for formal cross-area review of the design goals document and of preliminary specifications as a prerequisite before producing a reference implementation.

---

From: Michael Thomas <[email-masked]>
Date: Fri, 23 Jun 2006 15:10:06 -0700

>I think we would do well to require a reference implementation as a condition for Proposed Standards from new working groups or individual submitters...but there are other conditions that we should…

How utterly sclerotic. And what is the IETF, the code police? If ever
there were a need to make the IETF utterly and completely beside the
point, this suggestion would be the perfect way.

       Mike

---

From: Michael Thomas <[email-masked]>
Date: Fri, 23 Jun 2006 15:17:14 -0700

>I would have little objection to requiring running code as a test of feasibility of a new idea.  I would object strongly to an argument that just because someone has running code, means it's a good…
This is simply untrue. The fact that we had running code across multiple
disparate code bases was offered only as an indication as to the maturity
of the draft. We never said that was the only reason or even the primary
reason that the working group should be formed and that it progress to
PS.  None of the participants that I've come across think that running
code trumps rough consensus -- both of which has been chugging alone
nicely, I'll add.
        
          Mike

---

From: Keith Moore <[email-masked]>
Date: Fri, 23 Jun 2006 22:26:19 -0400

> point, this suggestion would be the perfect way.

Heaven forbid the internet Engineering task force should actually do engineering.  Remember engineering?  That's the discipline of producing efficient designs that meet predetermined goals and requirements.

IETF is already plunging toward irrelevance at terminal velocity.  The only way to
arrest the descent is for it to start producing better quality and more relevant 
specifications.  A good start would be for it to actually pay some attention to the problem definition and rough specification phases and to conduct them in an environment where they can get meaningful review outside of a narrow community.

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 23 Jun 2006 22:27:37 -0400

> of the draft. 

Patently false.  And DKIM has only gained consensus by defining the problem so narrowly as to produce a useless solution.

Keith

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Sat, 24 Jun 2006 12:06:25 +0200

At 04:26 24/06/2006, Keith Moore wrote:
>Heaven forbid the internet Engineering task force should actually do 
>engineering.  Remember engineering?  That's the discipline of 
>producing efficient designs that meet predetermined goals and requirements.
>
>IETF is already plunging toward irrelevance at terminal 
>velocity.  The only way to arrest the descent is for it to start 
>producing better quality and more relevant specifications.  A good 
>start would be for it to actually pay some attention to the problem 
>definition and rough specification phases and to conduct them in an 
>environment where they can get meaningful review outside of a narrow community.

IETF deliverable user QA.

This is the main thing I am interested in (in my 
Multilingual/Multilateral Internet area). Sometimes it calls for some 
weak to strong strategy to obtain better (yet not respected) texts. 
This is probably because there are not so many interested yet in that 
area (what is another issue).

However the real problem I met is the now structural disinterest of 
the IESG when considering a WG document to know if it matches the WG 
Charter. Producing efficient designs is something one can only try, 
but striving to meet predetermined goal and requirements should be a 
demanded prerequisite.

I wandered why the Internet standard process degraded (IMHO) that 
way. I feel it is because the IESG/IAB are not assigning 
predetermined goals and specifying requirements when chartering. They 
indicate an area to be considered. So the WG is actually to consider 
the whole issue (not only engineering but also market study, etc). 
This leads to postdetermined goals and requirements. The IAB/IETF 
consider the "geography" of the proposed WG, not the "political" 
motivations of its proponents. Then the IESG only considers if the 
delivery fits the geography of the WG area, and if it has an inner 
logic. This is probably because they have not the time/resources to 
verify if that logic fits the logic of the Internet architecture 
(they made too complex to understand in proceeding this way).
jfc

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 24 Jun 2006 12:26:23 +0200

On 24-jun-2006, at 4:26, Keith Moore wrote:

> specifications.  A good start would be for it to actually pay some  
> attention to the problem definition and rough specification phases  
> and to conduct them in an environment where they can get meaningful  
> review outside of a narrow community.

I don't think the solution is more hoops to jump through. Unless I'm  
mistaken, the IESG already has significant lattitude in rejecting  
protocols or imposing additional requirements. Building in stuff to  
save the day in the cases where the IESG gets it wrong is a waste of  
everyone's time, IMO. Lightening the IESG's load so they can reflect  
on the errors of their ways sooner would be more helpful in those cases.

I don't know about "narrow community", but I agree that good reviews  
are essential. Reviewing is hard, especially with long documents /  
complex specifications (unfortunately it still seems some RFC writers  
are paid by the word) and also when there are many dependencies. And  
there's essentially nothing in it for the reviewer, so only people  
who are very much in favor or very much against something bother,  
with the former probably not being in the best position to uncover  
hidden problems.

In my opinion, if the IETF could make it worth someone's while in one  
way or another to do a thorough review, that would help a lot.

---

From: John Levine <[email-masked]>
Date: 24 Jun 2006 10:58:31 -0000

>  Such as, a requirement for formal cross-area review of the design
> goals document and of preliminary specifications as a prerequisite
> before producing a reference implementation.

The IETF standards process is already so slow and uncertain that
people throw up their hands in exasperation and go around it.

Changing the process to make it even slower and add yet more hoops to
jump through will solve the problem by ensuring that nobody bothers
any more.

R's,
John

---

From: Keith Moore <[email-masked]>
Date: Sat, 24 Jun 2006 08:58:15 -0400

On 24 Jun 2006 10:58:31 -0000

> people throw up their hands in exasperation and go around it.

True.  Which is why it's necessary to handle the reviews in a pipelined rather than a stop-and-wait fashion.  But part of the reason IETF's process is so slow is that the 
only meaningful checks we place are at the end - so a working group typically labors to the point of exhaustion without having received any external feedback, so when the feedback does arrive the working group is so dysfunctional that it's nearly incapable of fixing anything (and it is often in denial about what is wrong).  Providing more early feedback will speed up the process rather than slow it down.

Keith

---

From: Keith Moore <[email-masked]>
Date: Sat, 24 Jun 2006 09:05:00 -0400

On Sat, 24 Jun 2006 12:26:23 +0200
Iljitsch van Beijnum <[email-masked]> wrote:

> > IETF is already plunging toward irrelevance at terminal velocity.   The
> > only way to arrest the descent is for it to start producing better
> > quality and  more relevant specifications.  A good start would be for
> > it to actually pay some  attention to the problem definition and rough
> > specification phases  and to conduct them in an environment where they
> > can get meaningful review outside of a narrow community.
> 
> protocols or imposing additional requirements. 

By the time IESG gets around to reviewing things, more often than not
it's too late to fix whatever is wrong.   What we need to arrange is that the significant review happens earlier, when the WG is still functional.  By the time the final review rolls around it should mostly be a formality - assuming, of course, that the WG has not changed things significantly since earlier reviews.

Keith

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 24 Jun 2006 16:17:23 +0200

> it's too late to fix whatever is wrong.   What we need to arrange  
> is that the significant review happens earlier, when the WG is  
> still functional.  By the time the final review rolls around it  
> should mostly be a formality - assuming, of course, that the WG has  
> not changed things significantly since earlier reviews.

This is orthogonal to the earlier discussion about whether there  
should be additional requirements for running code and so on. (Or at  
least, it should be: if you think it's hard to get people to let go  
of specifications they hold dear, try getting them to part with  
running code.)

The IETF wgs work well when there is a clear "best" solution or set  
of more or less equivalent best solutions. As far as I can tell, the  
wg process and the IETF process in general have a hard time working  
well when there is no clear best solution so several interests must  
be traded off against each other. If a wg succeeds in doing this,  
there's always someone to cry faul because their favorite interest  
didn't get top billing as soon as the result leaves the wg.

Unfortunately, we don't seem to have a way to resolve this other than  
let the market decide, which isn't too good for the IETF because it  
means that some work done in the IETF is subsequently deemed a  
failure and people who don't understand this take a dim view of the  
IETF because they think it can't create good protocols. In reality,  
the IETF is successful because it allowed the market to do its work  
rather than to impose the choice of a small group. But then, the IETF  
operates under an American culteral bias and in American culture  
there is no appreciation for second place, only for winning.

---

From: Keith Moore <[email-masked]>
Date: Sat, 24 Jun 2006 10:28:40 -0400

> didn't get top billing as soon as the result leaves the wg.

agreed - but to me this is just an indicator that the current process by which we run WGs is severely broken.

> rather than to impose the choice of a small group. 

strongly disagree with this. the market is good at local optimization but is lousy at long-term foresight. it's even worse at technical foresight. and this has a lot to do with why the Internet is such a mess now.  IETF is the biggest repository of protocol design expertise, and the fact that IETF produces so many protocols which are poorly designed is not by any means an indicator of success.

Keith

---

From: "John R Levine" <[email-masked]>
Date: 24 Jun 2006 10:58:20 -0400

> True.  Which is why it's necessary to handle the reviews in a pipelined
> rather than a stop-and-wait fashion.

Maybe I'm unduly pessimistic, but I would have trouble working up a lot of
enthusiasm for doing work on a project if I knew that some gatekeeper
working in parallel might later tell me that I was wasting my time.

Although I completely disagree with your prediction that DKIM will be
useless, there are worse problems than standards that turn out not to be
used, so long as they are designed so they don't interfere with other
more useful work.  We seem to be managing that last issue reasonably well.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Information Superhighwayman wanna-be, http://www.johnlevine.com, Mayor
"I shook hands with Senators Dole and Inouye," said Tom, disarmingly.

---

From: Keith Moore <[email-masked]>
Date: Sat, 24 Jun 2006 11:21:19 -0400

> working in parallel might later tell me that I was wasting my time.

I have trouble working up a lot of enthusiasm for contributing to work that is going off in the weeds.  What I have in mind is not a "gatekeeper" but external review.  The external review would not be expected to tell a WG that it were wasting its time (though that is a possibility), but rather, how to make better use of its time by producing a specification that was more relevant.  

Apparently you think that an artist is the best judge of the relevance of his own work. 

> more useful work.  We seem to be managing that last issue reasonably well.

DKIM is a good counterexample of that also.  The existence of DKIM distracts attention from more useful work that could be done - partly because it's consuming energy from those who would work on more useful goals if they were chartered, partly because of the need for damage control, and partly because of the widespread assumption that since IETF has chartered DKIM that DKIM is "the" solution that will be promoted by IETF.

Keith

---

From: "John R Levine" <[email-masked]>
Date: 24 Jun 2006 11:35:02 -0400

> but rather, how to make better use of its time by producing a
> specification that was more relevant. ...

I find the assumption that external reviewers are better able to score a
project's "relevance", whatever that is, rather than the people who are
doing it and plan to use it, truly breathtaking.

> - partly because it's consuming energy from those who would work on more
> useful goals if they were chartered, partly because of the need for
> damage control,

This must be a different group of people from the ones who I find on the
DKIM list.  If we wanted to work on something else, we would be doing so,
and although I may overrate our collective wisdom, I don't think we're all
working on DKIM purely because we are too dim to imagine anything else.

> and partly because of the widespread assumption that since IETF has
> chartered DKIM that DKIM is "the" solution that will be promoted by
> IETF.

Seems to me that if it weren't so difficult to charter and complete WGs,
there would be more of them, and people would be less likely to
overestimate the importance of one or another of them.  But I wouldn't
generalize too much about DKIM, or MARID, or ASRG, because people have
been looking for a magic spam bullet for 10 years and nothing we tell them
is going to stop that.

Regards,

---

From: Michael Thomas <[email-masked]>
Date: Sat, 24 Jun 2006 08:45:12 -0700

>only meaningful checks we place are at the end - so a working group typically labors to the point of exhaustion without having received any external feedback, so when the feedback does arrive the wo…
There's already a means for "external reviewers" to do so: read the drafts,
make comments, add issues to the issue tracker. It's really  not rocket
science. Having some new sclerotic "pipeline" involved with the life 
blood of a
working group sounds like a recipe for working group infarction to me.

          Mike

---

From: Keith Moore <[email-masked]>
Date: Sat, 24 Jun 2006 11:46:22 -0400

> doing it and plan to use it, truly breathtaking.

As I said, apparently you think that an artist is the best judge of the relevance
of his own work.  I find this preposterous.

> overestimate the importance of one or another of them.

One reason that it's difficult to charter and complete WGs might be that WGs have demonstrated a huge potential to do more harm than good.

> But I wouldn't

True, the spam area is more difficult than most.  But that ought to compel us to rely even more heavily on engineering disciplines, rather than to invest a huge amount of effort in a dubious direction out of a naive belief that doing something, anything at all, is better than doing nothing.

Keth

---

From: Keith Moore <[email-masked]>
Date: Sat, 24 Jun 2006 11:55:56 -0400

> science.

That's not quite sufficient, because most WGs aren't proceeding according to good engineering discipline (e.g. they're doing things in the wrong order, like trying to define the protocol before the problem space is understood), there's little external visibility of what the WG is doing (so it's difficult to make timely input without actually following the entirety of the mailing list discussion), and often, nobody with any authority over the WG is checking to see whether the WG is actually responding appropriately to such comments.  A more formal process is necessary.

> Having some new sclerotic "pipeline" involved with the life 
> blood of a

In other words, we don't want to distract WGs with useful input ... better that they should keep their heads in the sand for the entire 2-3 years of their existence and then produce irrelevant or even harmful output.  And that way, maybe a few influential people within the WG can coerce the WG into producing something that favors their employers' short-term interest even if it harms other interests or glosses over important limitations.

---

From: Michael Thomas <[email-masked]>
Date: Sat, 24 Jun 2006 09:21:53 -0700

>That's not quite sufficient, because most WGs aren't proceeding according to good engineering discipline (e.g. they're doing things in the wrong order, like trying to define the protocol before the…

By whose standard? If you think that's going on, I'd think it would be
appropriate to take it up with the relevent WG chairs and AD's. Last
I heard, they're the stakeholders.

>make comments, add issues to the issue tracker. It's really  not rocket
What you seem to want is some sort of cabal of dilettants who
don't actually have to pay very much attention, but by their
rank alone get to exercise veto power. Yuck. I don't suppose 
you have anybody in mind for such an IETF version of landed gentry?

		Mike

---

From: marcelo bagnulo braun <[email-masked]>
Date: Sat, 24 Jun 2006 22:30:03 +0300

Hi Iljitsch,

> there's essentially nothing in it for the reviewer, so only people who 
> are very much in favor or very much against something bother, with the 
> former probably not being in the best position to uncover hidden 
> problems.
>

I very much agree with this

I think this is the way to improve quality
there have been some initiatives for this, see 
http://www.watersprings.org/pub/id/draft-handley-doc-market-00.txt but 
i guess it didn't progress...

we should try to figure out some mechanisms to foster reviews... do you 
have any ideas?

Regards, marcelo

---

From: Keith Moore <[email-masked]>
Date: Sat, 24 Jun 2006 19:15:04 -0400

Michael,

Merely calling something or someone a name like sclerotic or dillitante is unconvincing.  You haven't seen a proposal yet, you have no idea of how this would work (or apparently, how a review of an engineering effort should work), and yet you dismiss the very idea out of hand.  That's very similar the kind of failure that causes many working groups to produce poor or irrelevant output.  You haven't refuted anything, you've merely provided a case example of why working groups need adult supervision.

Keith

---

From: "John R Levine" <[email-masked]>
Date: 24 Jun 2006 20:20:33 -0400

> Merely calling something or someone a name like sclerotic or dillitante
> is unconvincing.  You haven't seen a proposal yet, you have no idea of
> how this would work

Good point.  The sooner you send around a proposal, the sooner we can
figure out how it might improve the process.

---

From: Tim Bray <[email-masked]>
Date: Sat, 24 Jun 2006 17:34:09 -0700

On Jun 24, 2006, at 8:55 AM, Keith Moore wrote:

> That's not quite sufficient, because most WGs aren't proceeding  
> according to good engineering discipline (e.g. they're doing things  
> in the wrong order, like trying to define the protocol before the  
> problem space is understood)

I'd generalize that.  I have never seen *any* standards org do a good  
job of inventing new technology.  I've been working with the soon-to- 
wind-down Atompub group for a couple of years and we got a pretty  
good result I think (if you can judge by implementations &  
deployments).  There were a few things in our favor - a high level of  
interest and energy, lots of experience on the WG, decent editors -  
but the key thing was there was a ton of hands-on experience in the  
space (syndication technology).  A whole lot of the key arguments  
could be resolved by appeal to example and experience.

When standards orgs go out to invent stuff in unexplored territory  
you get disasters like OSI networking, CORBA, and in the current  
landscape, WS-*.  I suppose there are exceptions but I don't know of  
any.

  -Tim

---

From: Keith Moore <[email-masked]>
Date: Sat, 24 Jun 2006 23:39:34 -0400

On Sat, 24 Jun 2006 17:34:09 -0700
Tim Bray <[email-masked]> wrote:

> job of inventing new technology. 

good point.  anytime a significant number of people participating in a WG are lacking some important set of expertise or experience, there's going to be a lot of time spent educating everyone and adopting a common language.  and in IETF, because we try to engineer things for the Internet as a whole, it's entirely normal for a significant plurality of WG participants to lack some fundamental kind of expertise that is required to make the protocol work well at Internet scale.  so we have our work cut out for us.

Keith

---

From: Keith Moore <[email-masked]>
Date: Sun, 25 Jun 2006 00:18:20 -0400

> figure out how it might improve the process.

Fair enough.  But I think Michael's note illustrates a somewhat different, but related, and very common phenomenon in IETF that is relevant to this problem.

All too often, we (and I include myself here) shoot down ideas before they've had a fair hearing.  We take an idea that is embryonic and - because we can _imagine_ that idea going somewhere that _might_ have a bad result - we do our best to kill it before it can breed.  In this way we discard many good ideas not because they have no merit, but because they bring with them the potential to change things.  The result is stagnation.

I think we do this because we lack confidence in the IETF working group process.  We all know that once something gets to the working group stage, it's very hard to kill.  No matter how badly it turns out, the IESG is likely to end up holding its nose - maybe demand a trivial text change or two or add a nasty disclaimer - but basically approve it.  By the time the WG has spent 2-3 years or more, exhausted itself and declared itself to be done, it takes a lot of Last Call feedback and IESG gumption to push back on it.  Simple fixes can of course be made, but significant design flaws can rarely be corrected.

At the same time, experienced WG participants know that even small, apparently trivial objections made late in a WG's lifetime have the potential to delay the WG's output for months or sometimes years.  So once there is a significant investment in a particular direction within a WG, there tends to be tremendous resistance on the part of core participants to feedback that would call that direction into question.  

Trouble is, in our current process, there's rarely any formal request for feedback, and little external visibility of a WG's output, until Last Call.  Which basically means that working groups get most of the external feedback about their fundamental design choices long after the designs are frozen and the working group is too exhausted to fix anything. Ideally our process should encourage these differences to be resolved early - before there is a significant investment in the WG's output that would encourage denial on the part of its proponents, and before the WG is exhausted to the point that it cannot respond quickly and flexibility to external input.  And if we had a process that made it likely that significant problems were discovered early, the latter stages of review should be less time consuming and less risky.

So when I'm saying that working groups need multiple stages of formal, external review, what I'm really saying is that we need a structure for working groups in which we can have confidence that sufficient feedback will be obtained early enough to put good ideas on the right track and to see that truly bad ideas get weeded out in due time, most of the time.    You (or Michael) and I might have somewhat different ideas about how to gain that confidence, or about the details for how to make it work without bogging down working groups, but the concept working groups need earlier external feedback is sound.   

Keith

p.s. as for additional review stages bogging things down, any good protocol engineer ought to understand that sliding windows produce better throughput than stop-and-wait, that a fin-wait state is unavoidable if you want a clean close, but the way to reduce time in fin-wait is to minimize rtt and packet loss...

---

From: Dave Crocker <[email-masked]>
Date: Mon, 26 Jun 2006 08:50:27 +0200

> working on DKIM purely because we are too dim to imagine anything else.

I think the silliness of this thread should be underscored with particulars,
rather than merely noting the (entirely correct) basic principle that you cite:

The IETF DKIM effort is version FOUR of a continuous line of effort that has
demonstrated substantial increases in number and variety of participants, number
and variety of implemtations, and amount of traffic supported, at each increment.

The concept of an increasing spiral of inclusiveness has generally been viewed
in the standards world as a key requirement for success.  Design-by-committee
does not work, but long-term control by a cabal does not either (most of the
time.)  The spiral solves both problems, as long as the effort is truly
incremental.  What the spiral does is to a) refine the work, and b) demonstrate
increasing support for it, among the community that is going to use it.

What has been interesting to note is that the IETF used to be a distinctively
useful place for moving through the spiral, but it is tending instead to be a
place for increasing indeterminacy and delay, both due to sclerotic -- thanks to
Michael for this apt term -- formal procedures and the single-person veto we
essentially allow.  (And, here, I am not referring to IESG issues, but rather
the way we fail to require critics to develop support before we attend to them,
no matter how silly their criticism.)

d/

---

From: Dave Crocker <[email-masked]>
Date: Mon, 26 Jun 2006 08:56:28 +0200

marcelo bagnulo braun wrote:
>> are essential. ...
> http://www.watersprings.org/pub/id/draft-handley-doc-market-00.txt but i
> guess it didn't progress...
> 

The first effort was the SIRS project, that was declared a failure after only 4
months (most of which were during the summer) in spite of recruiting a
substantial list of highly experienced participants.  The second was a working
group that took a year to develop a scheme that was essentially identical to
sirs, but did a nice job of dissipating community energy for the effort.

The "incentive" that something like SIRS offered was classification as a senior
contributor.  This was neither a small point nor a small benefit (IMO).

d/

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 26 Jun 2006 10:08:43 +0300

> sirs, but did a nice job of dissipating community energy for the 

too bad... perhaps we should try another effort in this area... but we 
should wait for the summer to end this time...

> contributor.  This was neither a small point nor a small benefit (IMO).

what was the benefit of becoming a senior contributor?

Thanks, marcelo

---

From: Dave Crocker <[email-masked]>
Date: Mon, 26 Jun 2006 09:49:45 +0200

marcelo bagnulo braun wrote:
>> The "incentive" that something like SIRS offered was classification as
>> a senior contributor.  This was neither a small point nor a small benefit (IMO).
> 

Idealistic answer:  The personal satisfaction of meaningful contribution.
(There is a rumor that active IETF participants are highly motivated by this
goal.  So, the incentive is provide explicit satisfaction of it beyond their
narrow areas of primary activity.)

Selfish:  Public recognition.  (Good for both one's ego and potentially one's
employment status.)

d/

ps. The ideal and the selfish are complementary and not at all negative, IMO.
But, then, I believe all idealism is driven by a form of self-satisfaction...

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 26 Jun 2006 11:10:21 +0300

> Selfish:  Public recognition.  (Good for both one's ego and 

ok, may make sense

how would this differ from the "technical advisor" title?

Regards, marcelo

---

From: Dave Crocker <[email-masked]>
Date: Mon, 26 Jun 2006 11:03:27 +0200

>>> what was the benefit of becoming a senior contributor?
...
> how would this differ from the "technical advisor" title?

In my model, the advisor is an on-going mentor.  They are an active participant
in the working group.

Reviewers are not (necessarily) participants.  There is an obvious -- and
probably quite appropriate -- view that a reviewer MUST NOT be a participant,
lest their review be too distorted by having too much context.

In both cases, I would think that neither has any sort of veto.  Rather, they
must sway by convincing rather than dictating.  This applies both to the
decision-making by the wg and decision-making by the IESG (about the wg output.)

d/

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 26 Jun 2006 11:50:27 +0200

On 25-jun-2006, at 6:18, Keith Moore wrote:

> All too often, we (and I include myself here) shoot down ideas  
> before they've had a fair hearing.  We take an idea that is  
> embryonic and - because we can _imagine_ that idea going somewhere  
> that _might_ have a bad result - we do our best to kill it before  
> it can breed.  In this way we discard many good ideas not because  
> they have no merit, but because they bring with them the potential  
> to change things.  The result is stagnation.

Are you the same Keith Moore as the one I was talking to ealier??

> Trouble is, in our current process, there's rarely any formal  
> request for feedback, and little external visibility of a WG's  
> output, until Last Call.

That's what charters are for, aren't they?

But I see your point. When I first attended an IETF meeting I was  
surprised to see how inward looking wgs are. There is very little, if  
any, effort to present what the working group is doing to people  
outside of the wg.

> So when I'm saying that working groups need multiple stages of  
> formal, external review, what I'm really saying is that we need a  
> structure for working groups in which we can have confidence that  
> sufficient feedback will be obtained early enough to put good ideas  
> on the right track and to see that truly bad ideas get weeded out  
> in due time, most of the time.

Hm, I think trying to kill bad ideas is largely a waste of time.  
(Saying this both as someone who came up with some ideas that others  
think are bad and someone who has tried to convince others that their  
ideas are bad.) Often, the fatal flaws will show up as the idea is  
developed, so a lot of them go away without doing anything anyway.

The trouble is, that if someone develops a presumably bad idea in a  
draft, that draft is going to be deleted after six months. So the  
only way to keep that work around is to put it on a private website  
where it's probably going to be lost between the billions of pages  
that make up the web, or push for publication as an RFC. And like it  
or not, this lends a lot of credibility to an idea as most people  
don't understand the informational/experimental/standards track  
classification. It might make sense to create a third class of  
published documents that sits somewhere between draft and RFC. This  
would avoid the ridiculous situation where a draft is implemented and  
the email announcing its existance is archived, but the draft itself  
is deleted after some time.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 26 Jun 2006 12:56:41 +0300

> lest their review be too distorted by having too much context.

isn't there already some "general area" reviewers that perform this 
type of function? I thought there were....

> decision-making by the wg and decision-making by the IESG (about the 

what do you think about these more aggresive forms of looking for 
feedback, like to one in Handley & Rescorla draft? maybe they could be 
tested wihtout enforcing them, but leaving the results public (i.e. a 
web page publishing the amount of credits that each participant has, so 
that the general public can see who does reviews and who doesn't...) 
(as oposed to enforcing it by not allowing submitting new draft if the 
person does not have enough credits...)

Regards, marcelo

---

From: Dave Crocker <[email-masked]>
Date: Mon, 26 Jun 2006 12:50:37 +0200

marcelo bagnulo braun wrote:
> isn't there already some "general area" reviewers that perform this type
> of function? I thought there were....

as I recall when that was established, they are related to IETF Last Call and
hence are useless for a working group development process.

> feedback, like to one in Handley & Rescorla draft? 

Review early and often (within reason.)  Specific templates for review schedules
must vary according to the nature of the work and the wg process that produces it.

The idea of a formal pseudo-currency strikes me as clever, complex and
unproductive. Extra overhead without extra beneift, other than in theory.

There is no demonstrated need for such an incentive/reinforcement mechanism and
I haven't seen any basis for believing it would work.  (On the average, IETF
discussions about human behavior tend to make a point of avoiding any empirical
basis. A bit like saying that fat pipes eliminate the need for queues in
routers, but if you have the queues, they must have room for lots of entries.
Two assertions that are empirically demonstrated to be incorrect, for the price
of one.)

By way of anecdotal observation, my experience is that any serious working group
effort that is able to develop a reasonable amount of community interest is able
to obtain meaningful review when it seeks it.  I'm sure the track record is not
perfect, but my impression is that it is quite good.

The problem is with getting groups to seek those reviews and heed their results
-- including the result of not being able to get reviews, since that
demonstrates a lack of serious community interest -- not with establishing yet
more complex formal process.

d/

ps. I'm sure I'm over my daily posting limit, so I'll stop on this point.  Let's
let the U.S. wake up. (Suggestion referring to time zones, not politics.)

---

From: Keith Moore <[email-masked]>
Date: Mon, 26 Jun 2006 09:50:08 -0400

> That's what charters are for, aren't they?

in practice, they rarely serve that purpose.  we're not in the habit of structuring our groups this way.

> Hm, I think trying to kill bad ideas is largely a waste of time.  

perhaps, but that doesn't mean we need to provide them with incubators, and that's what many groups end up doing.  

> Often, the fatal flaws will show up as the idea is  

there is one important class of bad ideas that doesn't go away in IETF -- the class of bad ideas that is obviously bad from a wider perspective but which looks good to a set of people who are focused on a narrow problem.  and in IETF what we often do with those ideas is to protect them and encourage development of them in isolation by giving them a working group.  we sometimes even write those groups' charters in such a way as to discourage clue donation or discussion of other ways of solving the problem.

Keith

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Mon, 26 Jun 2006 16:20:36 +0200

At 15:50 26/06/2006, Keith Moore wrote:
>in IETF what we often do with those ideas is to protect them and 
>encourage development of them in isolation by giving them a working 
>group.  we sometimes even write those groups' charters in such a way 
>as to discourage clue donation or discussion of other ways of 
>solving the problem

+1

The worst is when the bad idea get allocated that way a de facto 
exclusive control on a globally key IANA registry in an area where 
the IETF has neither expertise, nor interest. The IETF is then to 
assume this way, and due to the importance of the Internet, a leading 
worldwide responsibility (cf. RFC 3935) however it cannot assume its 
related (RFC 3935) duty of competence. Then its (RFC 3935) mission to 
"influence the way people design, use and manage" the Internet 
becomes a tool in the hands of the bad idea's supporters.

The worst of the worst, is when consensual protections against the 
bad-idea are delegated by consensus to the IESG, but the IESG 
violates them, and delegates them to the bad-idea's very afficionados.

jfc

---

From: Tony Finch <[email-masked]>
Date: Mon, 26 Jun 2006 15:51:24 +0100

On Mon, 26 Jun 2006, Iljitsch van Beijnum wrote:
> > Trouble is, in our current process, there's rarely any formal request
> > for feedback, and little external visibility of a WG's output, until
> > Last Call.
>
> any, effort to present what the working group is doing to people outside
> of the wg.

I guess that if people are interested in the WG then they will
participate, or at least read the drafts as they come up in the i-d
announcement list. You can't force volunteers to take an interest.

Perhaps requirements documents should be more like rationale documents:
"we chose to solve this problem in this way because of this"...

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOUTHEAST ICELAND: SOUTH VEERING SOUTHWEST 4 OR 5, OCCASIONALLY 6 IN NORTH.
RAIN AT TIMES. MODERATE OR GOOD.

---

From: Tony Finch <[email-masked]>
Date: Mon, 26 Jun 2006 15:51:24 +0100

Tony.

---

From: Keith Moore <[email-masked]>
Date: Mon, 26 Jun 2006 11:24:55 -0400

> announcement list. You can't force volunteers to take an interest.

the problem is the assumption that people are either interested enough
to participate in a WG or they're not interested at all in the
outcome.   for many WGs there are a significant number of people who
will potentially be adversely affected by the outcome but who only have
a peripheral interest in the design.   we need a better way for WGs to
get clues from those with peripheral interests than to expect those
people to read every new I-D that comes out and try to evaluate it
without benefit of mailing list context.

> "we chose to solve this problem in this way because of this"...

In general we need to discourage the meme "requirements documents".  We
need problem definition documents and design goal documents from the
early phases of the process, design rationale documents to explain particular
decisions made at later phases.

Keith

---

From: Dave Cridland <[email-masked]>
Date: Mon, 26 Jun 2006 16:32:06 +0100

On Mon Jun 26 15:51:24 2006, Tony Finch wrote:
> request
> outside
It's sometimes difficult to find the drafts you could comment on as 
they're produced, especially if you're not part of the WG, and it's 
also tricky to find the background to some of the decisions.

It's fustrating, too, to have issues which are brought up 
continuously in a WG by well-meaning outsiders, when the issues have 
been discussed to death.

But equally, it's virtually impossible to keep an active interest in 
more than a handful of WGs.

> "we chose to solve this problem in this way because of this"...

Perhaps one option would be if each WG maintained a problem-statement 
and rationale draft. This could reference the documents that provided 
the solutions to the problems found, and act as an overview to 
outsiders of the WG's previously discarded arguments, etc, as well as 
how they see the specifications used in practise.

This would act in part as Keith's "requirements document", and also 
act as a lure for external (to the WG, at least) reviewers.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 26 Jun 2006 17:33:37 +0200

> of function? I thought there were....

http://www.alvestrand.no/ietf/gen/art/review-guidelines.html

Invoked at IETF Last Call and/or IESG Evaluation.

For the IESG's latest discussion on early review, see
section 8 of
http://www.ietf.org/u/ietfchair/Spring06retreatNotes.txt

    Brian

---

From: Tony Finch <[email-masked]>
Date: Mon, 26 Jun 2006 15:46:50 +0100

On Sat, 24 Jun 2006, Tim Bray wrote:
>
> When standards orgs go out to invent stuff in unexplored territory you get
> disasters like OSI networking, CORBA, and in the current landscape, WS-*.  I
> suppose there are exceptions but I don't know of any.

Speaking of CORBA:

http://www.acmqueue.com/modules.php?name=Content&pa=showpage&pid=396

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
PORTPATRICK: ANOTHER AREA OF STRONG SOUTHERLY WINDS WILL SPREAD TO MOST
WESTERN AREAS, AND FASTNET, ON THURSDAY AND FRIDAY, WITH GALES POSSIBLE IN
ROCKALL AND BAILEY.

---

From: Keith Moore <[email-masked]>
Date: Mon, 26 Jun 2006 11:51:53 -0400

> act as a lure for external (to the WG, at least) reviewers.

please note: I am _not_ a proponent of requirements documents.  I think
WGs need to define their problems and refine their scopes beyond that
stated in their charters, write down their design goals, and perhaps to
try to characterize design tradeoffs.    A few of those design goals
thus identified might qualify as hard-and-fast requirements.  But
asking WGs to state things in terms of requirements is tantamount to
asking them to make those design tradeoffs before they are understood. 

Keith

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 26 Jun 2006 13:50:47 -0400

On Monday, June 26, 2006 11:24:55 AM -0400 Keith Moore <[email-masked]> 
wrote:

> In general we need to discourage the meme "requirements documents".

No, we don't.  We just need to discourage the practice of waiting to write 
such a document until after protocol design is complete, rather than doing 
it first.

> early phases of the process

Yes, absolutely.  That's what a requirements document is supposed to be.

---

From: Keith Moore <[email-masked]>
Date: Mon, 26 Jun 2006 14:12:42 -0400

> such a document until after protocol design is complete, rather than doing 

Disagree.  Unlike engineering of physical objects, very few things
that we do in IETF are tied to physical phenomena that can be measured
with precision and have inherent limits.    If you're designing
IP-over-foo, you might be dealing with physical characteristics of foo
(like maximum link propagation delay) but for anything higher on the
protocol stack you're probably not dealing with hard-and-fast
requirements so much as compromises between conflicting goals.  

How secure should a protocol be?  How scalable should a protocol be?
How flexible should a protocol be for different user communities or
operating environments? You usually can't answer these questions in
absolute terms.  It's not like designing a bridge where you can say
that it must hold X number of metric tons static load and withstand a
wind of Y km/hr.  That's not to say that you never nail down any
requirements of a protocol design, but the bulk of what must be
understood prior to the design phase is rarely so neat and tidy.

> Yes, absolutely.  That's what a requirements document is supposed to be.

The problem is that within IETF when we try to talk of "requirements
documents", people look at the word "requirement" and think of them
something that, once established, dictates the protocol design.  So
they try to define the requirements in such a way as to produce the
design they want to use.  That's backwards.  What needs to be done
instead is to develop a shared understanding of the design tradeoffs
within the working group.  The word "requirements" gets in the way of
doing that.

Keith

---

From: "Robert Sayre" <[email-masked]>
Date: Mon, 26 Jun 2006 15:10:02 -0400

On 6/24/06, Keith Moore <[email-masked]> wrote:
>
> In other words, we don't want to distract WGs with useful input ...
> better that they should keep their heads in the sand for the entire
> 2-3 years of their existence and then produce irrelevant or even
> harmful output.  And that way, maybe a few influential people within
> the WG can coerce the WG into producing something that favors their
> employers' short-term interest even if it harms other interests or
> glosses over important limitations.

If the errors are sufficiently grave, it is easy to fork the WG
documents and have them replaced or completely rewritten. That's
enough, IMHO. Disliking the existence of the WG itself is probably not
grounds for any sort of action.

The advice WGs need most often is probably "most of your goals are
false ones, and this design is 10x too complicated". In the apps area,
"stop using W3C XML Schema" is probably up there, too.

I don't see a way to require that WGs discard most of their goals and
designs when they're told to. Banning XML Schema should be possible,
though. ;)

-- 

Robert Sayre

---

From: Keith Moore <[email-masked]>
Date: Mon, 26 Jun 2006 15:30:20 -0400

> documents and have them replaced or completely rewritten.

it's not easy at all - because even if you replace the WG you'll have
most of the same individuals active in the new one as in the old one -
only they'll be angrier than the first time around, and there's a good
chance that any people you lose in the transition will include those
who had more clue.  I'm much more interested in trying to figure out
how to get WGs to stay on track in the first place and to accept useful
clue from elsewhere.

Keith

---

From: "Robert Sayre" <[email-masked]>
Date: Mon, 26 Jun 2006 15:47:04 -0400

On 6/26/06, Keith Moore <[email-masked]> wrote:
> only they'll be angrier than the first time around,

I meant replace the documents, rather than the WG. But you're right,
they do get angrier, at first. For example, in the somewhat-successful
Atompub group, this happened at least twice.

The protocol document still contains a lot of cruft that's only in
there so that a few vendors can call whatever it is they're building
"Atom". It might not be fatal stuff, so forking it won't work. The
resulting document will be easy to improve on, if someone wants to,
but it sure is a heck of a lot more practical than WS-*.

> I'm much more interested in trying to figure out

I maintain that no process will accomplish that. The only way to get a
WG to accept a clue is to demonstrate that their output is irrelevant
by concrete example.

-- 

Robert Sayre

---

From: Keith Moore <[email-masked]>
Date: Mon, 26 Jun 2006 19:38:45 -0400

> WG to accept a clue is to demonstrate that their output is irrelevant

no process can ensure that WGs stay on track, but we can certainly do better than what we have now.

Keith

---

From: "Tom.Petch" <[email-masked]>
Date: Tue, 27 Jun 2006 17:10:20 +0200

----- Original Message -----
From: "Keith Moore" <[email-masked]>
To: "Robert Sayre" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Tuesday, June 27, 2006 1:38 AM
Subject: Re: are we willing to do change how we do discussions in IETF? (was:
moving from hosts to sponsors)

> no process can ensure that WGs stay on track, but we can certainly do better
than what we have now.

I think that the single change most likely to keep WGs on track is to ensure
that they do not have a single dominant participant, eg one who is both chair
and
author of key I-Ds.  The WGs I see most at risk of going round in circles and/or
producing output that falls short of what is needed are ones such.

Some time ago, I did hear an IESG member talk of this in such a way as to make
me think that this was an understood problem, but nothing seems to have changed
in the two or so years since then.

And, of course, I believe that there is more to good engineering than just
engineering eg the right processes.

Tom Petch.

---

From: Eliot Lear <[email-masked]>
Date: Tue, 27 Jun 2006 18:50:31 +0200

Hi,
> in the two or so years since then.

Perhaps it is an unwritten rule, but I thought this battle was fought
and won years ago.  Perhaps you should discuss the specific problem with
the AD and or the IESG.  I can't recall the last time I was involved in
a group in which the chair played an active role in authoring.  And as
someone who did that way many years ago, I strongly advise against it.

> engineering eg the right processes.

Ding ding!

Eliot

---

From: Ned Freed <[email-masked]>
Date: Tue, 27 Jun 2006 09:55:28 -0700 (PDT)

> > that they do not have a single dominant participant, eg one who is both chair and

> someone who did that way many years ago, I strongly advise against it.

This is exactly my take as well. I've seen many cases where a chair has
refused to become a document author or editor in a group because of the
conflict it creates. I've also seen at least one case where a chair stepped
down in order to become a document author.

Of course there are exceptions. The obvious ones are that the conflict is much
more limited when there's a non-aothor co-chair, the authorship role is limited
to a small subset of the group's documents, or both. For example, I was asked
to co-chair the NNTPEXT group in large part because the other co-chair was also
the main document author. (I note in passing that in this case we ended up with
both roles being done by other people.)

In another case, I once tried to weasle out of writing a specification (now RFC
2034) because at the time I was co-chair of the NOTARY group. But the feedback
from all concerned was that it one small ancillary document and I was a only
the co-chair, and I was, um, persuaded to do the work.

> Ding ding!

Ditto.

				Ned

---

From: Eliot Lear <[email-masked]>
Date: Wed, 28 Jun 2006 15:49:47 +0200

Dave Cridland wrote:
> It's fustrating, too, to have issues which are brought up continuously
> in a WG by well-meaning outsiders, when the issues have been discussed
> to death.
>

Often times what you describe as "discussed to death" are those very
issues where there really isn't community consensus, and so what happens
is that people check out of working groups, and sometimes make a stink
toward the end of the process.  That's when the IESG has a
responsibility to make a judgment call as to whether or not a WG has
made the right technical decision, rough consensus be damned.

Eliot

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 28 Jun 2006 15:10:05 +0200

> down in order to become a document author.

Exactly.

> both roles being done by other people.)

Without commenting on the specific case, I think that co-chairs may be the
right solution when one of the chairs is an author of a fraction of the
documents, but not usually when s/he is *the* main author. In that situation,
the author+chair can never make a consensus call, so effectively is recused from
the most important part of chairing.

> the co-chair, and I was, um, persuaded to do the work.

Exactly.

    Brian