Subject: Re: IESG Success Stories

From: Harald Alvestrand <[email-masked]>
Date: Mon, 01 Jan 2007 15:30:56 +0100

--On 30. desember 2006 18:00 -0800 Michael Thomas <[email-masked]> wrote:

>> With regard to textual nit-picking and evaluation of worthiness of
>> prose, I tend to agree with what I think you are saying. However, if a
>> document is too badly written to permit interoperable implementations
>> to be constructed without clarifying conversations among implementers,
>> authors, and/or the WG, then the document is a failure and needs
>> pushback.  As with late surprises, somewhat more proactive effort on
>> the part of WGs could prevent many of the problems we see, but...
>
> I was using "wordsmithing" rather broadly. My probably idiosyncratic
> meaning
> of "wordsmithing" here was "will this DISCUSS change the mechanics of the
> protocol or not". If the answer is no, we're really just making the
> document
> more ready for publication IMO. Something that does bring that possibility
> is obviously a lot more serious. It's been my admittedly limited
> experience that
> my version of "wordsmithing" is a lot more common, and the source of a lot
> of delay to varying degrees of dubiousness.

One meaning of "wordsmithing" resulting from a DISCUSS that I think is an 
entirely worthwhile activity is where the mechanics of the protocol, as 
implemented by a WG participant, will not change one whit, but where the 
implementation by a non-participant changes from "improbable" to 
"possible", because it's clear what the words were intended to say.

Another example of "wordsmithing" that does not change the mechanics of the 
protocol, but is nevertheless important, is the IANA considerations stuff - 
while it does not change the protocol as such, it does change the 
"meta-protocol" of extending the protocol later. That's important.

            Harald

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: John C Klensin <[email-masked]>
Date: Mon, 01 Jan 2007 14:10:44 -0500

--On Monday, 01 January, 2007 15:30 +0100 Harald Alvestrand
<[email-masked]> wrote:

>> I was using "wordsmithing" rather broadly. My probably
>> idiosyncratic meaning
>> of "wordsmithing" here was "will this DISCUSS change the
>> mechanics of the protocol or not". If the answer is no, we're
>> really just making the document
>> more ready for publication IMO. Something that does bring
>> that possibility is obviously a lot more serious. It's been
>> my admittedly limited experience that
>> my version of "wordsmithing" is a lot more common, and the
>> source of a lot of delay to varying degrees of dubiousness.
> 
> One meaning of "wordsmithing" resulting from a DISCUSS that I
> think is an entirely worthwhile activity is where the
> mechanics of the protocol, as implemented by a WG participant,
> will not change one whit, but where the implementation by a
> non-participant changes from "improbable" to "possible",
> because it's clear what the words were intended to say.

> Another example of "wordsmithing" that does not change the
> mechanics of the protocol, but is nevertheless important, is
> the IANA considerations stuff - while it does not change the
> protocol as such, it does change the "meta-protocol" of
> extending the protocol later. That's important.

Harald, 

I could be wrong, but I believe that, if every instance of a
"discuss" for "wordsmithing" purposes fell into one of the
categories you have described above, we would not be having this
discussion.   I continue to believe that the document that
started the original thread is not the right place to fix the
problem.  However, I suggest that there is a perception in the
community --one that can be fairly well documented with specific
episodes-- of long approval and publication delays that started
with a "discuss" about issues of prose or presentation that do
not have either significant "this is wrong" protocol impact or
the level of impact you describe above.

That, of course, leads to another discussions: when evaluating
IESG or RFC Editor performance, we tend to assume that any delay
between when the token is passed back to a WG or author and when
it returns is a WG or author problem and doesn't count.   In
some cases, it clearly is a WG or author problem.  But, in
others, it is at least partially the result of a natural
reaction: "after all of the work that went into this, we just
don't have the energy to deal with that level of nit-picking".
And since, unfortunately, there is often no way for an outsider
to differentiate the two, we need to better understand and
appreciate the costs of "discuss" positions that do not have a
clear, substantive, basis appropriate to the maturity level of
the proposal, even ones whose intent is to get additional
explanations and understanding, to overall IETF productivity.

      john

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 02 Jan 2007 00:12:11 +0100

> the level of impact you describe above.
I think we're on the same page (or at least in the same chapter of the 
book).
My note was just intended to say that I don't agree with the reading of 
Michael's statement that says "a DISCUSS is just make-work if it doesn't 
require a protocol change". I don't think Michael agrees with that 
reading either, but it was possible to read his note that way.
> explanations and understanding, to overall IETF productivity.
The problem of working groups that emit documents as their last gasp 
before dying from lack of energy is, in my opinion, a much harder 
problem to tackle than that of browbeating the IESG into DISCUSS 
discipline - and one where, if the IESG's energy was suddenly magically 
to double, I'd definitely advise directing the windfall surplus.

                  Harald

---

From: John C Klensin <[email-masked]>
Date: Mon, 01 Jan 2007 19:42:29 -0500

--On Tuesday, 02 January, 2007 00:12 +0100 Harald Alvestrand
<[email-masked]> wrote:

> The problem of working groups that emit documents as their
> last gasp before dying from lack of energy is, in my opinion,
> a much harder problem to tackle than that of browbeating the
> IESG into DISCUSS discipline - and one where, if the IESG's
> energy was suddenly magically to double, I'd definitely advise
> directing the windfall surplus.

Precisely because this subtopic is a harder one, one
clarification.  I was not referring to "emit as WG last gasp"
situations, although those would clearly also apply.   I was
referring to the more common situation in which a WG has reached
the exhaustion point on a particular document, possibly because
it has been the result of very careful, but tiring, negotiations
over language as well as substance.  

For that class of document, a "please engage in fine tuning"
request, even if that request is accompanied by very specific
suggestions, can easily lead to an "oh no, we can't revisit
_that_ again" reaction that can lead to significant delays, even
if the WG and editors/authors are otherwise completely healthy
and committed.

>From my parochial point of view as someone who still believes in
a multi-step standards process, this is yet another argument as
to why the IESG should adopt and strengthen an "if it isn't
clearly harmful, let it go" attitude toward documents moving
toward Proposed Standard, regardless of what the current
"discuss" document does or does not permit.  It would, IMO, be
entirely appropriate for the IESG to include with the approval
of a document --possibly even as part of the protocol action
notice-- a clear message that there isn't a chance that the
document would be approved for advancement to Draft without
considerable improvements in editorial quality and that the time
to start working on those issues is probably "immediately"...
but not to hold a would-be Proposed Standard document up waiting.

     john

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 03 Jan 2007 15:57:44 +0100

Michael,

On 2006-12-31 03:00, Michael Thomas wrote:
> John C Klensin wrote:
>>
>> If an AD who was responsible for a WG came up with an issue about that 
>> WG's work and raised it only during or after Last Call, I'd expect 
>> either a really good explanation or a resignation.  I certainly would 
>> not expect it to happen often. But, IMO, we have an IESG and, indeed, 
>> an IETF rather than a collection of different organizations addressing 
>> per-area issues precisely to increase the odds of catching serious 
>> cross-area and cross-perspective issues before things go out.  Like it 
>> or not, unless the relevant WG makes an effort to get broad reviews at 
>> critical "early" points, there will always be a risk of late surprises 
>> as someone --in the community during Last Call or on the IESG-- 
>> suddenly wakes up and says "but how does this relate to XYZ".
> 
> I was obviously not clear enough on my first post because I wasn't 
> referring
> to the shepherding AD or the poor AD's who have paid no attention at all
> who are then tapped to do their job. It's really the in between AD's that I
> think really exasperate the wg participants -- if they've paid enough 
> attention
> to attend the wg meetings and perhaps even subscribe to the mailing list 
> and
> registered concerns, I really think they owe it to the working group to 
> either
> get them out when the working group is in active issue resolving mode, or
> to... well, just live with it. Or something. Frankly it does a lot of 
> damage to
> the IESG's reputation when you know that there's likely going to be 
> trouble,
> but there is nothing you can do to fend it off ahead of time because those
> AD's have only given vague allusions to their non-amusement. It promotes
> the vision of the unitary IESG which I think is a bad thing.

I think you're overlooking two or three facts that collide at a single
point:

1. ADs physically don't have time to read intermediate drafts oustide
their own Area. So while they may suspect that a WG is heading in
a worrisome direction, they aren't in a position to do much about it.

2. ADs are collectively instructed by our rules to act as the reviewers
of last resort - it's the IESG that takes the final responsibility
to say a document is good enough.

3. Unfortunately, WGs do sometimes agree on drafts that prove to
have signifcant defects when critically reviewed outside the WG.

Put these facts together and you *will* get a reasonable rate of
significant (i.e. hard to resolve) DISCUSSes, and in the nature of things
they will come from ADs outside the Area concerned.

However, the DISCUSS non-criteria are specifically intended to
exclude capricious "I don't like this" discusses.

BTW, I'm an author (draft-ietf-v6ops-nap-05.txt) trying to resolve
a very complex DISCUSS from an AD outside the Area. I believe the
document will be much better as a result, even though it's
a painful process. And it's largely a result of the WG having
a narrower view of the universe than is ideal.

      Brian

---

From: Michael Thomas <[email-masked]>
Date: Fri, 05 Jan 2007 10:03:58 -0800

> they will come from ADs outside the Area concerned.
Brian --

My focus was actually a lot more narrow: I wasn't trying to insist
that AD's be super-human, and I honestly believe that the job they
do is extremely difficult. My gripe is when an outside AD takes an
interest in the work, goes to the f2f meetings, maybe reads the drafts
but then waits to IESG evaluation time to DISCUSS their issues. If
they know they have a problem(s), it would be *far* better to air that
sooner rather than later for all parties concerned. Doing this leads to
the perception of a imperious and capricious IESG.

This is a very different situation where you get a DISCUSS where
it's a surprise to all parties from an AD not involved at all; the former
should have been resolved  before it got to last call, the latter is the
process working correctly. IMO.

       Mike

---

From: John C Klensin <[email-masked]>
Date: Fri, 05 Jan 2007 14:55:53 -0500

--On Friday, 05 January, 2007 10:03 -0800 Michael Thomas
<[email-masked]> wrote:

> My focus was actually a lot more narrow: I wasn't trying to
> insist that AD's be super-human, and I honestly believe that
> the job they do is extremely difficult. My gripe is when an
> outside AD takes an interest in the work, goes to the f2f
> meetings, maybe reads the drafts but then waits to IESG
> evaluation time to DISCUSS their issues. If they know they
> have a problem(s), it would be *far* better to air that sooner
> rather than later for all parties concerned. Doing this leads
> to the perception of a imperious and capricious IESG.

Michael,

I have two questions...

(1) Do you have evidence of actual situations in which an AD
behaved in this way, kept concerns to him or herself, and then
raised them only, and for the first time, via a DISCUSS after
Last Call?

(2) If the answer to (1) is "yes", why didn't you, and the other
people who were impacted, immediately file recall petitions?

The second question is not rhetorical.   We all understand that
recalls would be painful and destructive, that would they take
too long to have much practical impact, and so on.  However, the
behavior I think you are describing would be such an egregious
violation of the ways that the community and the IESG should be
interacting with each other that I believe a recall would be
appropriate even for someone whose term on the IESG had only a
few months remaining: it would be at least as important to
establish a clear message that the behavior is unacceptable,
ever, as it would be to get the person off the IESG.  No one is
valuable enough, hard-working enough, or smart enough that the
community should put up with the behavior I think you are
describing, even for a minute.    And that implies to me that
there should be no perception that an AD can get away with it.

      john

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sat, 6 Jan 2007 04:43:30 +0800

I strongly agree with John's reasoning here. But please keep reading...

From: "John C Klensin" <[email-masked]>

> there should be no perception that an AD can get away with it.

Now, backing off a few billion nanometers, When Michael kicked off this 
thread, his posting began:

> So what occurs to me is that a reasonable question to ask is whether
> there are some legitimate success stories where a DISCUSS has actually
> found big or reasonably big problems with a protocol that would have
> wreaked havoc had they not been caught. I ask because ...

My impression is that drafts that capture experience with IETF working group 
dynamics are painful and useful (thinking specifically of the case studies 
in the draft that became http://www.ietf.org/rfc/rfc3669.txt). I think 
Michael's question is important enough to answer. I think I can think of 
some examples where the answer is "yes", but this isn't about what *I* 
think, it's about what our experience has been.

Do other people think that collecting DISCUSS success stories is helpful?

I have two points to add -

- Michael asked specifically about SUCCESS stories. The level of cynicism in 
the circles I run being sufficiently high, I'm asking about the same thing 
Michael is asking about, where "success" matches the dictionary definition.

- if people want to collect DISCUSS "success" stories, of the more ironic 
nature, I ask that we do this, going forward, and not looking backward. 
Depending on how you count Jon Peterson, we seated new ADs for almost half 
the IESG last year, and we know that we will pick up at least a few more new 
ADs this year, because some people have already said they will not return. 
We don't need to replay 30 years of history, and even going back to the 
beginning of ID tracker use would be less than helpful.

If you have relevant deep dirt from the recent past, please feel free to 
file recalls and provide NomCom input, of course. But I hope everyone 
already knows that.

Thanks,

Spencer

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 5 Jan 2007 17:17:33 -0800

On Jan 5, 2007, at 10:03 AM, Michael Thomas wrote:

> My gripe is when an outside AD takes an
> sooner rather than later for all parties concerned.

I agree with the earlier is better than later for comments from  
anyone, AD or not.

A few interesting side cases on this. Some ADs (more than one  
actually) recently suggested to a WG that something there were doing  
was likely to result in in a DISCUSS when it reached the IESG. One of  
the WG members appealed the IESG trying to manipulate WG consensus.  
Sort of left me scratching my head on why the WG would not want to  
know that early but evidently some folks don't.

There have been other working groups where it was very hard to make  
an argument and get anyone to listen - lots of people just give up  
and decide to send their comments in IETF LC or not at all. That's  
really unfortunate but I understand that people have a limited time  
to go argue about stuff that is not even their work.

Cullen

---

From: Michael Thomas <[email-masked]>
Date: Fri, 05 Jan 2007 17:57:52 -0800

> I strongly agree with John's reasoning here. But please keep reading...

[]

I really didn't want to take this off on this well trod tangent because
it was... pretty much a tangent. And I definitely didn't mean this turn
into some sort of grudgathon  :)
> Now, backing off a few billion nanometers, When Michael kicked off 
> this thread, his posting began:
>
> My impression is that drafts that capture experience with IETF working 
> group dynamics are painful and useful (thinking specifically of the 
> case studies in the draft that became 
> http://www.ietf.org/rfc/rfc3669.txt). I think Michael's question is 
> important enough to answer. I think I can think of some examples where 
> the answer is "yes", but this isn't about what *I* think, it's about 
> what our experience has been.
>
> - Michael asked specifically about SUCCESS stories. The level of 
> cynicism in the circles I run being sufficiently high, I'm asking 
> about the same thing Michael is asking about, where "success" matches 
> the dictionary definition.
>
> - if people want to collect DISCUSS "success" stories, of the more 
> ironic nature, I ask that we do this, going forward, and not looking 
> backward. Depending on how you count Jon Peterson, we seated new ADs 
> for almost half the IESG last year, and we know that we will pick up 
> at least a few more new ADs this year, because some people have 
> already said they will not return. We don't need to replay 30 years of 
> history, and even going back to the beginning of ID tracker use would 
> be less than helpful.
>
> If you have relevant deep dirt from the recent past, please feel free 
> to file recalls and provide NomCom input,From [email-masked] Fri Jan 05 20:59:07 2007
From: "Kyse Faril" <[email-masked]>
To: [email-masked]
Bcc: 
Date: Fri, 05 Jan 2007 18:56:28 -0700
Subject: Copyright Notice: NRMEF4NYNOS

Great name for a rock-band or ...'wine label?' Designed in 
'text-messagise'...by yours truly!

         "N R M E F 4 N Y N O S"

I know!

How about a 'WIRE-LESS ROCK BAND's 'IMPORTED' PREMIUM WINE LABEL SKATE-BOARD 
STICKER?'

>From: Eadric Streona <[email-masked]>
>To: [email-masked]
>CC: [email-masked]
>Subject: Copyright Notice: NRMEF4NYNOS
>Date: Fri, 5 Jan 2007 17:44:42 -0800 (PST)
>
>
>--- [email-masked] wrote:
>
> > From: [email-masked]
> > To: [email-masked]
> > Subject: Copyright Notice: NRMEF4NYNOS
> > Date: Fri, 05 Jan 2007 17:43:06 -0800
> >
> > ALL RIGHTS RESERVED
> >
>
>
>__________________________________________________
>Do You Yahoo!?
>Tired of spam?  Yahoo! Mail has the best spam protection around
>http://mail.yahoo.com

_________________________________________________________________
Get live scores and news about your team: Add the Live.com Football Page 
www.live.com/?addtemplate=football&icid=T001MSN30A0701

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Fri, 5 Jan 2007 21:16:02 -0500

On Fri, 5 Jan 2007 17:17:33 -0800
Cullen Jennings <[email-masked]> wrote:

> > interest in the work, goes to the f2f meetings, maybe reads the
> > drafts but then waits to IESG evaluation time to DISCUSS their
> > issues. If they know they have a problem(s), it would be *far*
> > better to air that sooner rather than later for all parties
> > concerned.
> 
It's a delicate balance.  When I was an AD, I'd often post things saying

	<AD hat=off>
	...
	</AD>

and the like; I've seen other ADs do the same thing, both on lists and
in person.  It's not clear that people always believe the disclaimer.

Even more care is necessary when warning of a DISCUSS.  Is that an
accurate prediction of your own or someone else's liely views, or is it
an attempt to get one's way by threatening to block a legitimate choice
you don't like?  How will it be perceived?

On the other hand, I sometimes had long, detailed discussions with WG
chairs and document authors, often at the instigation of the
responsible AD, trying to convey my concerns ahead of time, trying to
get things resolved to my satisfaction.

Finally, there was one document I abstained on, because it was
terminally broken and not repairable, but was the product of too many
years work for me, as a relatively new AD (and hence as one who hadn't
seen it before) to block.  But I declined to say "no objection",
because I had very strenuous objections to it.

I'm glad there's a document being written to set forth DISCUSS
criteria.  I had lots of discussions with Harald about the number I
issued -- he wanted me to justify some of them with more than the text
I wrote.  "Is this issue serious enough to block publication of the
document as is?"  Obviously, I thought so, but not everyone on the IESG
agreed with me on some of those cases.

		--Steve Bellovin, http://www.cs.columbia.edu/~smb

---

From: Dave Crocker <[email-masked]>
Date: Sat, 06 Jan 2007 01:51:29 +0000

Folks,

> as implemented by a WG participant, will not change one whit, but where 
> the implementation by a non-participant changes from "improbable" to 
> Another example of "wordsmithing" that does not change the mechanics of 
> the protocol, but is nevertheless important, is the IANA considerations 

Let's be clear:

    A "Discuss" blocks the document.  It is an exercise of power.

    It is necessary and appropriate to use a Discuss when there is
    a fundamental problem with the document, or a strong indication
    of a fundamental problem.

    We should all think hard about the word "fundamental", because
    use of a Discuss for anything other than that is an abuse of
    power; it wastes resources and good will.

ADs make "comments" that do not block the document.  Presumably they would not
make comments unless they thought they would be heeded.

This is good, because my own experience is that working groups tend to be
mindful of *all* AD statements, Discuss or Comment, since most working groups
really do want a specification to be good quality.

The difference is that, as noted by someone else's post, the working group
worries about "clearing" the Discuss.  That's different from worrying about
quality.

Working groups either do or do not worry about (their definition of) quality. 
Alln exercise in IESG power or a failure to exercise power won't change that. It 
might salvage a marginal spec, but it won't make any difference for a spec that 
is solid or one that is garbage.

Getting a "discuss" vs. "comment" does not change this.

Using a Discuss will not make bad work good.  Using a Comment will not result in
a good working group's ignoring the input.

What we are caught in, I believe, is a failure to be careful, as in selective.

Working groups are careful or sloppy.  So are ADs.

We need to get into synch about both the exercise of power and the goal of
quality.  It needs to have more to do with satisfying real-world need than with
bureaucracy or abstract idealism about the purity of architecture.

Bureaucracy and idealism can be quite helpful, but only *in the service of*
satisfying real-world need.

We used to be pretty good at that.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: "Robert Sayre" <[email-masked]>
Date: Fri, 5 Jan 2007 20:28:42 -0800

> the WG members appealed the IESG trying to manipulate WG consensus.

That's completely inaccurate. It was appealed because the IESG engaged
in the behavior I'll quote yet again:

> Dave Crocker wrote:
> > There is often a failure to distinguish between new and peculiar problems
> > created by a particular specification, versus general problems that already exist.
> >
> > A classic example of this is citing basic DNS problems, for specifications that
> > are merely consumers of the DNS and, hence, are not creating any new problems.

Another classic example is citing a basic HTTP security trade-off, and
requiring implementations to make specific choices about
centralization, internationalization, graphic design, scalability, and
security in order to remain conformant. But I digress.

> know that early but evidently some folks don't.

Knowing early is good. Sometimes, the IESG happens to insist on
inaccurate and irrelevant requirements, and argument from authority
sometimes results in appeals. There's nothing wrong with that. But
let's face it--no one in the WG was worried about a "dicussion". They
were worried that the document would be blocked for ideological
reasons.

-- 

Robert Sayre

"I would have written a shorter letter, but I did not have the time."

---

From: Julian Reschke <[email-masked]>
Date: Sat, 06 Jan 2007 11:39:45 +0100

Cullen Jennings schrieb:
> I agree with the earlier is better than later for comments from anyone, 
> AD or not.
> 
> A few interesting side cases on this. Some ADs (more than one actually) 
> recently suggested to a WG that something there were doing was likely to 
> result in in a DISCUSS when it reached the IESG. One of the WG members 
> appealed the IESG trying to manipulate WG consensus. Sort of left me 
> scratching my head on why the WG would not want to know that early but 
> evidently some folks don't.
> ...

Cullen, in case you're talking about APP and HTTP authentication...: my 
recollection is that that WG member in fact tried to *defend* the WG 
consensus. The "new" (rough) WG consensus (after the announcement of a 
forthcoming DISCUSS) IMHO clearly falls into the category "we don't care 
enough to pursue the discussion". And that is really A Bad Thing.

Best regards, Julian

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 08 Jan 2007 11:21:45 +0100

On 2007-01-05 20:55, John C Klensin wrote:
...
> raised them only, and for the first time, via a DISCUSS after

How about a case where an AD had decided, before being an AD,
not to fight against something s/he thought was misguided,
and then found it on the IESG agenda two or three years later?

I could give you an example of that.

    Brian

---

From: John C Klensin <[email-masked]>
Date: Mon, 08 Jan 2007 10:42:26 -0500

> I could give you an example of that.

In case it wasn't clear, my comment was intended to try to 
separate what I consider the egregiously bad behavior of 
deliberately causing late surprises (an activity I have referred 
to in the past as the IESG, or some ADs, playing "gotcha" with 
the community) and situations in which late surprises occur 
through no one's particular fault.  The latter can happen 
because something is just discovered late in the game (an 
overall system failure which we should try to avoid or at least 
minimize, but that will sometimes happen) or for other reasons, 
but don't involve deliberate bad acts on the part of an AD.

The situation you describe is one that, I hope, would be flagged 
and identified to the WG as early as possible, but I can imagine 
an AD forgetting about the earlier concerns until the issue 
popped onto the IESG agenda and then feeling an obligation to 
object.   Too bad, but not, IMO, something that is sufficiently 
disastrous, or likely to be sufficiently frequent, that we 
should shape policy around it.

    john

---

From: Dave Crocker <[email-masked]>
Date: Mon, 08 Jan 2007 10:24:23 -0800

Brian E Carpenter wrote:
> 1. ADs physically don't have time to read intermediate drafts oustide their
> own Area. So while they may suspect that a WG is heading in a worrisome
> direction, they aren't in a position to do much about it.

Folks,

I hope no one doubts the basic truth of Brian's observation.  My own feeling, in 
fact, is that expecting all ADs to read even the final draft is an excessive 
burden.  Either way, it leads to the basic question of how ADs can be informed 
of the salient issues in a wg effort, long before an IESG vote?

Currently, wgs produce 4 things:  charter, email list archive, meeting 
notes/summary, and output documents (specifications or whatever).  None of these 
permits intelligent assessment of working group progress, by someone who is not 
significantly involved in a wg's on-going effort, without making a massive 
effort to review the archive and notes record.  At best, the meeting summary is 
good for incremental issues, not summarizing integrated design (or syntax, or 
operations, or...) decisions.

Given that working groups operate over many months or years, it seems like we 
need something that is a work-to-date design/decision summary.

If a working group's effort is solid and defensible, along the way, then it 
ought to be reasonable to request that it produce a summary of its work-to-date 
in a fashion that allows useful, critical review, without having to dive into 
the considerable detail of a specification or the wg record.

There is likely to be an added benefit for this, beyond giving ADs practical 
input for early- and middle-stage assessment:  The rest of the community gets 
it, too.  And by "the rest of the community", I mean the entire Internet 
community, not just IETF participants.

For work of any extended interest, being able to see a summary of on-going work 
that looks relevant to them has to be useful for deciding whether to participate 
but particularly useful for folks who cannot participate directly but are likely 
to be affected by the final output.

d/

-- 

   Dave Crocker

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Mon, 8 Jan 2007 14:16:18 -0500

On Mon, 08 Jan 2007 10:24:23 -0800
Dave Crocker <[email-masked]> wrote:

> I hope no one doubts the basic truth of Brian's observation.  My own
> feeling, in fact, is that expecting all ADs to read even the final
> draft is an excessive burden.  Either way, it leads to the basic
> question of how ADs can be informed of the salient issues in a wg
> effort, long before an IESG vote?
> 
> Currently, wgs produce 4 things:  charter, email list archive,
> meeting notes/summary, and output documents (specifications or
> whatever).  None of these permits intelligent assessment of working
> group progress, by someone who is not significantly involved in a
> wg's on-going effort, without making a massive effort to review the
> archive and notes record.  At best, the meeting summary is good for
> incremental issues, not summarizing integrated design (or syntax, or
> Given that working groups operate over many months or years, it seems
> like we need something that is a work-to-date design/decision summary.
> 
> If a working group's effort is solid and defensible, along the way,
> then it ought to be reasonable to request that it produce a summary
> of its work-to-date in a fashion that allows useful, critical review,
> without having to dive into the considerable detail of a
> specification or the wg record.
> 

Dave, a lot of this discussion has boiled down to a single topic, one
that's been talked about for a very long time: early, cross-area review.
Unfortunately, we've tried several schemes that haven't worked and we
don't really know how to do better.  All have had some successes; none
have scaled.  It seems to boil down to this: if enough time and effort
is invested, by ADs, WG participants, and reviewers, it can work.  But
it takes that kind of focus, and extra cycles by reviewers who aren't
necessarily interested in the subject area are few and far between.
Extra documents don't help much, either, because they're *strongly*
resented by WG members who see them as just more process imposed by the
IESG.

I tried several things myself when I was AD.  If I were still AD, I'd
keep trying.  But I don't have any new ideas, and I'm skeptical of
repeating old ones.

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 8 Jan 2007 15:02:01 -0500

I think it is helpful to distinguish at least three types of IETF  
work products:

(1) fully new protocols, at the level of (say) MPLS or NSIS

(2) extensions of existing protocols, such as a new DHCP option or a  
new RTP payload type (another huge fraction of our current activities)

(3) "bis" versions of existing protocols (a very large chunk of our  
current repertoire)

Some efforts will fall somewhere in-between, but I suspect that we're  
mainly worried about (1), and maybe (3) ["ten years hence, what have  
we learned and should we scrap the old stuff instead of adding a  
layer of paint"].

Fortunately or unfortunately, the number of concurrent (1) efforts is  
very small.

Currently, WG and IETF meetings could be a good opportunity to get  
people to do cross-area review, as they are in IETF mode. However, we  
seem to have intentionally or unintentionally made this difficult.

In WG meetings, we're encouraged to talk about open issues, which  
often means that the same three people that have been arguing some  
angels-on-a-pin point on the mailing list, having lost everybody  
else, take turns at the microphone, while the remaining 247 people in  
the audience are paying full attention - to their email. I would find  
it much more helpful to have the ability to walk into a WG as an  
interested party that's not on a mailing list and catch up quickly.  
Interim meetings and phone conferences often seem better suited to  
that level of detail, particularly for large WGs.

Similarly, in the plenaries, we rehash NAT and RFC-over-HTML  
arguments or learn about the number of enterprise numbers that have  
been assigned. That time might be better spent in giving not just the  
IRTF slot time, but also any (1) and significant (3) development,  
with an emphasis on the architectural issues being raised by the  
effort and maybe an attempt to highlight the big-ticket open issues.  
Right now, the only time this tends to happen, at best, is during the  
BOF, which is usually and hopefully too early.

Henning

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 9 Jan 2007 04:29:02 +0800

Hi, Dave,

> notes/summary, and output documents (specifications or whatever).  None of 
> these permits intelligent assessment of working group progress, by someone 
> who is not significantly involved in a wg's on-going effort, without 
> making a massive effort to review the archive and notes record.  At best, 
> the meeting summary is good for incremental issues, not summarizing 
> integrated design (or syntax, or operations, or...) decisions.
>
> Given that working groups operate over many months or years, it seems like 
> we need something that is a work-to-date design/decision summary.

When I was a WG chair, chairs were required to produce a SHORT summary of 
what the HECK was going on their working groups, each IETF week, due by 
Friday morning, for our ADs.

This wasn't a perfect plan (especially if your working group met on Friday 
morning after the summaries were due :-), but it was standard practice and 
apparently had been for several years before I became a chair.

The sitting IESG noticed in San Diego that this practice has been abandoned, 
without any explicit decision to do so.

(200 WG chairs just forgot to send summaries, and the ADs forgot to notice 
that they hadn't received them :-)

Perhaps we already know how to solve part of this problem, we just don't do 
what we know we should do? :-)

I am not the genius of prehistoric IETF meeting proceedings, but don't 
believe that these summaries were widely distributed. But that's another 
question - keep in mind that some of my experience pre-dated the ID tracker, 
so there were a LOT of other things that stopped at the IESG then, which we 
have made more transparent in the intervening years. So it's very possible 
that a 1-2 page summary, produced three times per year, might not be an 
excessive burden on the people who have to produce it, and might benefit the 
people who should be reading it.

> If a working group's effort is solid and defensible, along the way, then 
> it ought to be reasonable to request that it produce a summary of its 
> work-to-date in a fashion that allows useful, critical review, without 
> having to dive into the considerable detail of a specification or the wg 
> record.
>
> There is likely to be an added benefit for this, beyond giving ADs 
> practical input for early- and middle-stage assessment:  The rest of the 
> community gets it, too.  And by "the rest of the community", I mean the 
> entire Internet community, not just IETF participants.

Agree with these points as well.

Spencer

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 9 Jan 2007 05:39:05 +0800

Just in case I confused anyone else,

From: "Edward Lewis" <[email-masked]> said privately,

> At 4:29 +0800 1/9/07, Spencer Dawkins wrote:
>
>>the HECK was going on their working groups, each IETF week, due by Friday
>
> When I first read this, I wondered "how does an IETF week compare with a 
> New York minute?" ;)
>
> Then I figured out what you meant...

I really did mean "an IETF MEETING week", but the possibilities ARE pretty 
amusing...

Thanks,

Spencer

---

From: Dave Crocker <[email-masked]>
Date: Tue, 09 Jan 2007 08:31:48 -0800

Folks,

Steven M. Bellovin wrote:
> don't really know how to do better.  

Henning Schulzrinne wrote:
 > In WG meetings, we're encouraged to talk about open issues, which often
 > means that the same three people that have been arguing some
 > angels-on-a-pin point on the mailing list, having lost everybody else,
...
 > Similarly, in the plenaries, we rehash NAT and RFC-over-HTML arguments
 > or learn about the number of enterprise numbers that have been assigned.

Spencer Dawkins wrote:
 > When I was a WG chair, chairs were required to produce a SHORT summary
 > of what the HECK was going on their working groups, each IETF week, due
 > by Friday morning, for our ADs.

Each of the above refer to particular reporting or reviewing mechanisms that we 
currently have or claim to have.  (Spencer, the wgs that I participate in are 
quite diligent to produce summaries of their f2f meetings.)

Each of the above is for a particular purpose:  In-depth review by targeted 
experts, progress on specification details or reporting that progress, as well 
as community consideration (or haggling) of philosophical points. These are all 
worthy tasks.

However none cover the reason for Design Summaries (I'm changing the name, a 
bit.)  The nature of a Summary has the wg take a step back from daily details 
and create a snapshot of the basic design and specification decisions, to date, 
but not as a list of individual decisions (or open issues.)

Instead it should try to the provide salient details as a coherent set.  In 
other words, a summary of the design, architecture or the like.

The purpose that this serves is to enable interested parties to understand the 
salient aspects of working group effort, so far, in terms of that integrated 
design, rather than as a project activity report.  This includes workers in 
related efforts, content experts, those considering participation, and those 
likely to be consumers of the wg output.

Cross-area review is for a much more focused purpose, with targeted experts, and 
goes much deeper than a Design Summary, I believe. Certainly a cross-area 
reviewer might like to *start* with the latest DS. So might any reviewer, for 
that matter.

Side notes:

    1.  For a community that seeks to be open and inclusive, an on-going 
challenge is to incorporate new participants, after the working group has 
developed significant history, as Henning notes.  Currently either the new 
participant must review all the docs and records and then decide whether to 
participate, or they join in with too little background and burden the wg with 
tutoring them.  Pointing new folk at the latest Intermediate Summary could 
greatly reduce the burden both on new participants and current participants and 
greatly improve their understanding, when they start to participate.

    2.  It occurs to me that one small added bit of mechanism might facilitate 
things -- although it would not serve in the stead of a DS:  Have the issues 
ticketing system used by the wg includ a "major decision" flag. This is merely a 
way of noting decisions that are considered to be particular import, for 
whatever reason.  It does not cover a variety of decisions that are made 
implicitly but could help anyone trying to construct a summary.

    - - - -

Steve raises the larger question about making cross-area review (or, I believe, 
*any* tracking, reporting, or quality-assurance mechanism) work.  He views them 
as a failure.  My direct (and narrow) experience is quite the opposite.

I think this reduces to the very basic fact that we have some working groups 
that have mature participation and diligent performance, and others that don't.

It is striking to me that IESG review seems not to distinguish between these. 
(Perhaps the IESG believes it does.  I will leave it as an exercise to the 
reader as to why my own impression is that it does not.)

 From what I am seeing, some wgs have embraced cross-area review vigorously. 
Not merely to satisfy a bureaucratic requirement, but to ensure that possible 
issues with related content specialties get due consideration, early in the process.

And perhaps that is the deeper point that we ought to consider:  There are a 
variety of things a wg can do, during the many months or years of its work, to 
ensure competence and sufficiency of its output.  I believe that Design 
Summaries would fill a significant gap in the repertoire.

Whether a working group employs those tools and makes a diligent effort to 
produce quality work that has significant community vetting, prior to dumping 
its output into the IESG's lap, ought to be seriously considered.

d/

ps. As with many ideas that show up in these discussions, choosing to do a 
Design Summary does not require formal approval by the IESG or IETF. It merely 
requires the decision by a working group -- or for that matter, any wg 
participant -- to create it and make it available.  Given the popularity of 
non-IETF support web pages for working groups, it would of course be trivial to 
add the latest DS to that page...

-- 

   Dave Crocker

---

From: Bob Braden <[email-masked]>
Date: Tue, 09 Jan 2007 09:46:01 -0800

Steve Bellovin wrote:

>is invested, by ADs, WG participants, and reviewers, it can work.

Steve,

Perhaps working groups could be encouraged to do their own cross-area
review, preferably early if not often, by adding a new required section to
every standards-track RFC: Internet Considerations.  The purpose of
this section would be to position the current work within the larger Internet
protocol suite, listing cross-layer issues, dependencies on other protocols,
and architectural assumptions. Like Security Considerations, some would
turn out fairly useless, some would turn out to be important and profound,
and the rest would scatter in the middle.  But the effort would encourage
working groups to think less narrowly.

I suggest this as a scalable solution to the problem.

Bob Braden

>  But

---

From: John Leslie <[email-masked]>
Date: Tue, 9 Jan 2007 13:19:50 -0500

Dave Crocker <[email-masked]> wrote:
> 
> However none cover the reason for Design Summaries (I'm changing the name, 
> a bit.)  The nature of a Summary has the wg take a step back from daily 
> details and create a snapshot of the basic design and specification 
> decisions, to date, but not as a list of individual decisions (or open 
> issues.)

   This sounds as if it would be extremely helpful to folks sitting in
on WG meetings during IETF Weeks...

   Can Dave point out any examples?

--
John Leslie <[email-masked]>

---

From: Dave Crocker <[email-masked]>
Date: Tue, 09 Jan 2007 10:31:13 -0800

>>   The nature of a Summary has the wg take a step back from daily

No, he can't.

But he did realize that his making the suggestion obligated him to produce at 
least one...

d/

-- 

   Dave Crocker