Subject: Re: "Discuss" criteria

From: "Steven M. Bellovin" <[email-masked]>
Date: Mon, 1 Jan 2007 01:38:25 -0500

On Sun, 31 Dec 2006 19:11:33 -0800
Lisa Dusseault <[email-masked]> wrote:

> 
> On Dec 31, 2006, at 2:27 PM, Lakshminath Dondeti wrote:
> 
> > There is perhaps one more aspect to "Can somebody explain ..." that
> > > is worth considering.  In some cases, the AD simply does not have
> > > > the expertise or simply has incorrect/wrong understanding.  In
> > > > that > case, the burden is on the authors to help the AD
> > > > understand the > context of the work, provide references to
> > > > reading material and > such.  Until the AD understands at
> > > > his/her own pace, the work seems > to languish (sure the
> > > > authors do delay responses etc., but let us > work on one
> > > > problem at a time) in the IESG review stage.
> 
> Sure.  That could happen.  But it's not usually the case that an AD
> who knows that they don't understand something, holds a DISCUSS on a
> document for a long time, all the while getting useful responses and
> help from authors.  I sure wouldn't feel comfortable doing that.

Right.  The usual AD vote is "no objection", not "yes".  "No objection"
includes "I don't know enough to have a problem with this".  In such
cases, I looked at the parts I did understand -- that, of course,
included authentication and (if appropriate) confidentiality -- and
worried about those, and didn't worry nearly as much about arcana at
other levels.

		--Steve Bellovin, http://www.cs.columbia.edu/~smb

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Bill Fenner" <[email-masked]>
Date: Mon, 1 Jan 2007 11:08:49 -0500

John,

  Sorry to pick out a nugget from the middle of a very long message, but

On 12/30/06, John C Klensin <[email-masked]> wrote:
> ...   What we don't need is
> more rigid rules that either try to anticipate every
> circumstance ...

This is something that I feel very strongly about.  We write rules
with wiggle room, because it's impossible to anticipate every
situation and important not to overconstrain the leadership in order
to allow them to show ... leadership.  We then look at the rules and
say "Oh horrors!  Someone could use bad judgement and make something
bad happen!" and try to write more rules to prevent that.  I think
that instead we should try to select leadership that won't use bad
judgement or will be able to understand when it's pointed out that
they are.

> I think that long terms on the IESG tend to breed
> detachment from the community and a tendency to put IESG
> judgment ahead of that of the community and I don't think we can
> solve that with more rules about IESG behavior.

As the current IESG member with the longest term, I agree with this,
and that fed into my decision to not stand for the position again.  I
think a young [in terms-of-service] IESG is more likely to be a
healthy IESG.

  Bill

---

From: Keith Moore <[email-masked]>
Date: Mon, 01 Jan 2007 20:22:11 -0500

>      Something quite basic that is missing from the draft on
>      Discuss Criteria is a requirement that any Discuss not only
>      explain its precise normative basis, but that it give a clear
>      statement of what actions must be taken to clear the Discuss.
> 

I strongly disagree.  When a working group document fails to meet RFC 
2026 criteria for the intended status, it's not up to the AD voting 
Discuss to fix the problem.  The burden is on the WG to either convince 
the IESG that its document does indeed meet RFC 2026 criteria, or to 
bring the document in line with RFC 2026.

While there is nothing wrong with an AD suggesting a simple fix to a 
document problem if he or she can identify one, expecting the AD to fix 
nontrivial problems is unrealistic and also encourages micromanagement.

Keith

---

From: Ralph Droms <[email-masked]>
Date: Tue, 02 Jan 2007 07:04:24 -0500

I read Dave's words "clear statement of what actions must be taken to clear
the Discuss" not as requiring the specification of a complete fix, but
rather as an indication of what needs to happen to the draft.
Implementation details of meeting those requirements are left to the WG.

I agree with Dave on this point.

- Ralph

---

From: Keith Moore <[email-masked]>
Date: Tue, 02 Jan 2007 08:05:36 -0500

-------- Original Message --------

> Implementation details of meeting those requirements are left to the WG.

I don't know what Dave meant by his words, but I can say that similar 
words were being used when I was on IESG, and at that time they was 
taken to mean a bit more than you assume Dave meant.  It wasn't enough 
to say what was wrong with a draft, the AD was expected to tell the WG 
what to do.  The expectation was often that the AD should supply 
specific text for minor changes to a document that would allow the AD to 
clear his discuss.  This further implied that any problems that could 
not be fixed with small textual changes had to remain in the draft - the 
AD wasn't able to say, e.g. "the entire approach has serious and 
irreparable technical flaws", or "the protocol appears to be sound but 
the document is so poorly written that small changes will not save it" - 
even though one or the other were occasionally the case.  And this 
further led to ADs insisting on small textual changes that had no real 
effect on the document - mostly to make the ADs feel better for having 
to approve such poor documents.

Of course, such flaws should have been caught a long time before Last 
Call.  That's one of the big problems with our process - we don't have 
good mechanisms for catching those problems, particularly when they 
cross area boundaries.  We leave it to the supervising AD to implement 
any ad hoc mechanisms they choose, but the WGs are able to push back on 
those because there's no community consensus behind them.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Tue, 02 Jan 2007 20:45:46 +0000

> Implementation details of meeting those requirements are left to the WG.

You read my words correctly.

Thanks for the helpful re-phrasing/clarification.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 03 Jan 2007 15:30:55 +0100

On 2006-12-29 17:44, Dave Crocker wrote:
 >> Dave is probably correct that the specific criteria are of broader
 >> interest than just ADs, WG chairs, editors, and process wonks, and
 >> might become even more perfect with broader review, but that's another
 >> issue.
 >>
 >> And, since the criteria are public, I'm sure the IESG would be
 >> interested in feedback on the criteria,

Yes. That was indeed the point of publishing them in the first place.
Doesn't mean we have to agree of course ;-)

I think most of what I would say about your feedback has been
said by others. But a few follow-ups follow:

On 2006-12-30 22:32, John Leslie wrote:
...
 >    This is _not_ the general case. For the general case, I must agree
 > with Dave Crocker:
 > ]
 > ] This is perhaps the scariest of the criteria.  It says that a
 > ] knowledgeable, motivated constituency can spend months on solving a
 > ] problem that it needs to have solved, and then others who have not
 > ] participated in the work can come along and sabotage it.
 >
 >    Actual IETF consensus (not just the lack of adverse comments during
 > Last-Call) is very hard to achieve. Requiring it for ordinary WG work
 > is non-scalable. (We can't reasonably expect to accomplish it more
 > than perhaps a half-dozen times a year.)

You can't prove a negative (nobody disagrees) but you can prove
a positive (at least one individual disagrees).

 >    This DISCUSS criteria (alleging lack of IETF-wide consensus), as
 > written, allows any AD (hopefully not the one shepherding it!) to
 > effectively block the work of a WG. Sam's "solution" amounts to
 > negotiation among non-players over _how_ to judge the prejudices of
 > folks who mostly don't understand the issues.

There is that danger, but there is equally a danger of group-think
whereby a WG chooses to ignore a generic issue. And even if only
one person blows the whistle on that group-think during IETF Last
Call, that is something the IETF needs to deal with.

 >    Further Sam's "solution" tends to favor re-running the Last-Call
 > if you don't like the results. This is a very bad practice. It
 > causes everyone who pays attention to believe there must be problems
 > below the surface; and strongly tends to produce a situation in
 > which consensus looks impossible (because too many issues have been
 > raised).

I think that misrepresents actual practice, which is to refer the
issue back to the WG and make sure they actually discuss it.

 >    In the normal case, any DISCUSS should go back to the WG. With
 > technical issues named, addressing them is workable. With a demand
 > for review by particular experts, there's at least a place to start.
 > With "So-and-so thinks somebody somewhere wouldn't like this," there's
 > really no place to start. :^(

Agreed, with the qualification that if *big* changes are made
as a result, a second IETF Last Call may be appropriate.

On 2007-01-01 15:17, Robert Sayre wrote:
...
 > A WG can agree with the AD that there is a problem, but disagree that
 > it needs to be solved in their document. Too often, the compromise
 > ends up being the insertion of text that satisfies the AD's concerns,
 > but disenfranchises the WG. The WG either ignores the text in
 > practice, or the document author couches the text in so many
 > qualifiers that it becomes easy to explain any implementation in terms
 > that make it seem conformant. The result is inaccurate or misleading
 > documentation of Internet technology.

But it may be more harmful not to point out the issue at all, or
not to specify a better-than-nothing solution. You have to consider
the case of a relatively inexperienced implementer who may not know
something that we believe is common knowledge.

It's always open to the WG to propose a resolution of the DISCUSS
that is radically different from what the discussing AD suggests,
too.

 > Implementation details of meeting those requirements are left to the WG.

I would generalize that slightly: the action may not even involve the
draft itself. It's not unknown for a DISCUSS to be resolved by a promise
to cover the issue in another draft. But I think we agree that a DISCUSS
needs to be actionable in some way, and that is so obvious that we forgot
to write it down.

      Brian

---

From: Dave Crocker <[email-masked]>
Date: Wed, 03 Jan 2007 22:49:33 +0000

Seems like grouping criteria might be helpful, to try to understand the core 
reason for for including them in the list... or considering exclusion:

A.  BASIC FAILURE  (Won't Work / Major Document Problem)
--------------------------------------------------------

> * The specification is impossible to implement due to technical or clarity 
> issues.

This probably means that there has been a fundamental failure of the entire IETF
WG and oversight process.

> * The protocol has technical flaws that will prevent it from working 
> properly, or the description is unclear in such a way that the reader cannot 
> understand it without ambiguity.

This probably means that something was missed, but can be fixed, or that the
document has a -- presumably fixable -- flaw that is a show-stopper.

> * It is unlikely that multiple implementations of the specification would 
> interoperate, usually due to vagueness or incomplete specification.

This is probably a variant of the first concern -- or possibly entirely
redundant with it -- albeit stated differently.  If, in fact, it is different,
it would help to clarify in what way.

> * Widespread deployment would be damaging to the Internet or an enterprise 
> network for reasons of congestion control, scalability, or the like.

Again, a concern of this type, during a late-stage step, indicates a fundamental
failure of the IETF process.

> * The specification would create serious security holes, or the described 
> protocol has self-defeating security vulnerabilities (e.g. a protocol that 
> cannot fulfill its purpose without security properties it does not provide).

This seems to be a variant of the preceding point, but stated specifically in
terms of Security.  The obvious process question is why the working groupdid
not obtain an adequate security review earlier, and make changes accordingly?

> * It would present serious operational issues in widespread deployment,by 
> for example neglecting network management or configuration entirely.

There is often a failure to distinguish between new and peculiar problems that
are created by a particular specification, versus general problems that already
exist.

A classic example of this is citing basic DNS problems, for specifications that
are merely consumers of the DNS and, hence, are not creating any new problems.

B.  IMPURE
----------

(The title of this set might seem flippant, but I cannot think of a more useful
label.  This has to do with issues that likely are secondary, to not being able
to work.  Otherwise, one of the first set of criteria would be used.)

> * Failure to conform with IAB architecture (e.g., RFC1958 (Carpenter, B., 
> “Architectural Principles of the Internet,” June 1996.) [2], or UNSAF 
> (Daigle, L., “IAB Considerations for UNilateral Self-Address Fixing (UNSAF) 
> Across Network Address Translation,” November 2002.) [3]) in the absence of 
> any satisfactory text explaining this architectural decision.

Since IAB architecture documents have no formal, normative status, it's not
clear how to pursue this.  When the IAB speaks on a topic, formally, certainly
the rest of the IETF needs to listen.  Perhaps the specification needs tobe
required to explain any deviance from IAB recommendations.

However it seems clear this criterion is not claiming that the specification
"won't work" but that it has a problem more in the abstract. To raise this as a
Discuss it seems that the AD ought to explain what the negative impact ofthis
specification will be.

C.  PROCEDURAL BREAKAGE
-----------------------

Everything in this section represent tactical show-stoppers, in that the problem
is with the working group's not having been diligent enough in doing required
steps. Clearly the steps must be taken before the document can progress, pending
good outcomes from those steps, of course.  (And also of course, is the small
question of how the document got to the IESG before the required work hadbeen
done?)

> * The specification was not properly vetted against the I-D Checklist. 
> Symptoms include broken ABNF or XML, missing Security Considerations, and so 
> on.

> * The draft omits a normative reference necessary for its implementation, or 
> cites such a reference merely informatively rather than normatively.

> * The document does not meet criteria for advancement in its designated 
> standards track, for example because it is a document going to Full Standard 
> that contains 'down references' to RFCs at a lower position in the standards 
> track, or a Standards Track document that contains only informational 
> guidance.

> * IETF process related to document advancement was not carried out; e.g., 
> there are unresolved and substantive Last Call comments which the document 
> does not address, the document is outside the scope of the charter of the WG 
> which requested its publication, and so on.

D.  CONSTITUENCY MISMATCH
-------------------------

> * The IETF as a whole does not have consensus on the technical approachor 
> document. There are cases where individual working groups or areas have 
> forged rough consensus around a technical approach which does not garner IETF
>  consensus. An AD may DISCUSS a document where she or he believes this to be 
> the case. While the Area Director should describe the technical area where 
> consensus is flawed, the focus of the DISCUSS and its resolution shouldbe on
>  how to forge a cross-IETF consensus.

Presumably, the working group has been populated by people interested in using
the specification in the near-term. If no such people are active in the working 
group, then why has a standards effort been pursued?

The above criterion says that these motivated participants' concerns are not 
sufficient, when weighed against the more detached desires of the active IETF 
community.

Asserting this criterion seems to represent an IETF failure at so many different
levels, that it raises fundamental questions about the nature and purpose of the
IETF.  By implication, it says that we do not care as much about the people who 
are going to depend upon the protocol, as we do about those who won't.

There certainly are times that the nature of Internet architecture permits
fielding only one proposal.  However there are plenty of examples of fielding
multiple solutions, and letting the market choose among them.  So the above
criterion would seem to impose a universal requirement for unanimity that was
most definitely not part of the IETF model, for example, 10-15 years ago.

d/
-- 

   Dave Crocker

---

From: Sam Hartman <[email-masked]>
Date: Wed, 03 Jan 2007 23:35:54 -0500

Dave, I think you make a lot of interesting points here.  I
particularly like aspects of your categorization.

I tend to disagree with your analysis however.  At the current
juncture I think that you and I don't have a lot to gain by engaging
in a long mailing list discussion on this topic.  I'd be happy to
engage with you in a higher bandwith (in person/phone) chat on these
issues; I think we'd find that we do have a fundamental disagreement
although I'm not entirely sure where.  Others should feel free to jump
in; depending on what happens I may pop in later or not.

As always, thanks for the careful thought.

---

From: "Robert Sayre" <[email-masked]>
Date: Wed, 3 Jan 2007 22:56:09 -0800

> But it may be more harmful not to point out the issue at all, or

Maybe, but then it's an editorial issue, and tangential to my comment.
In more relevant cases, the AD asks for substantive technical changes.

> not to specify a better-than-nothing solution.

A decision for the WG, in the event that they haven't already
considered it. A DISCUSS is much more productive when it points out an
issue the WG hasn't thought of. That's extremely unlikely for issues

>> are merely consumers of the DNS and, hence, are not creating any
new problems.

I don't see anything in the ION that addresses this concern, and it's
more worrisome if the IESG is free to retroactively make any RFC a
BCP. Perhaps the ION should explicitly state which RFCs a draft must
obey. RFC2026 seems to be the least relevant, so I don't think it's
productive to pretend that's the one we follow.

> You have to consider

Again, an editorial issue.

> that is radically different from what the discussing AD suggests,

Yes, any group is free to try anything in the IETF process, but that's
not expedient for a WG that needs a document published. Let's face
it--DISCUSS is a fairly Orwellian euphemism.

-- 

Robert Sayre

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 04 Jan 2007 10:48:04 +0100

...
> not expedient for a WG that needs a document published.

I need to do some textual analysis on that sentence ;-)

"a WG that needs a document published"

This choice of words implies that you're thinking of a WG as
an autonomous body with its own objectives. But that isn't the
way I look at any IETF WG. An IETF WG is a component of the IETF,
and the IETF's first goal is "to make the Internet work better"
(RFC 3935). Its way of doing this is to publish "documents that
influence the way people design, use, and manage the Internet
in such a way as to make the Internet work better" (RFC 3935 again).

So, a WG "needing" a document published has to achieve this
within the goal of making "the Internet work better." I see
most of the DISCUSS criteria quite explicitly oriented towards
meeting this goal. Also this is where the whole discussion
of IETF-wide review outside the WG fits in.

> Let's face

Not if you accept that IETF WGs should operate to meet
the IETF's mission. But it does require that both the AD issuing
the DISCUSS, and the protagonists of the document, are willing
to actually discuss.

     Brian

---

From: "Robert Sayre" <[email-masked]>
Date: Thu, 4 Jan 2007 13:53:57 -0800

On 1/4/07, Brian E Carpenter <[email-masked]> wrote:
> an autonomous body with its own objectives.

It implies that WG members are humans with a limited amount of time,
so the goal quickly becomes "clear the DISCUSS", not "make the
Internet work better".

> But that isn't the
> meeting this goal.

A DISCUSS on an issue that the WG has considered, in the absence of
new information, does not help the Internet work better.

> the DISCUSS, and the protagonists of the document, are willing

Again, in the situation above, slapping a DISCUSS on a document is not
the start of a conversation. It's a directive. I don't see anything in
the guidelines that addresses this situation. I think the guidelines
should explicitly allow or disallow it. Why don't they?

-- 

Robert Sayre

"I would have written a shorter letter, but I did not have the time."

---

From: Keith Moore <[email-masked]>
Date: Thu, 04 Jan 2007 17:12:05 -0500

> so the goal quickly becomes "clear the DISCUSS", not "make the

The IETF's entire reason for existence is to make the Internet work 
better.  If WG participants aren't here for that reason, they don't 
belong in IETF.  IMHO.

However I do think that in practice there is often a disconnect along 
these lines.  Some WG participants may not see their goal as making the 
Internet work better, but rather to get some new technology standardized 
whether it works well or not, so that planned product ship dates will 
not slip.   There is a tussle between very-short-term economic interests 
and the long-term investment that technology buyers and users make in a 
standard, and we might do well to recognize that tussle and tweak our 
procedures accordingly.  But the best way I know to do get better 
standards out sooner is to actually employ the disciplines of 
engineering when developing protocols - which we rarely do.

> new information, does not help the Internet work better.

While it might be true in specific cases, that's not true as a 
generalization.  The judgment of WG participants as to what is best for 
the Internet might be colored by the narrow focus of the WG, whereas 
people with other perspectives (like participants in other WGs, or 
people with legitimate interests other than that of the WG), or broader 
perspectives (like those on IESG) might have a different opinion of the 
merit's of a WG's output.  If that turns out to be the case, the WG's 
work may not have earned rough consensus of the IETF and may therefore 
need further revision - or if the WG has erred too badly, a completely 
different approach.

> the start of a conversation. It's a directive. 

In my experience it's far too often taken as a directive, but that's not 
necessarily the fault of the AD who voted DISCUSS.

I generally seems that when a WG views IESG and the rest of IETF as 
collaborators toward the general goal of making the Internet work 
better, things go fairly smoothly.  OTOH, when a WG views IESG and the 
rest of IETF as potential adversaries, and external input as something 
to be discouraged or ignored, we tend to get train wrecks.

Keith

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 11 Jan 2007 21:07:12 -0500

On Wednesday, January 03, 2007 10:49:33 PM +0000 Dave Crocker 
<[email-masked]> wrote:

> C.  PROCEDURAL BREAKAGE
> -----------------------
>
>> * IETF process related to document advancement was not carried out;
>> e.g.,  there are unresolved and substantive Last Call comments which the
>> document  does not address...

IMHO, this particular situation is more than just "procedural breakage". 
If a document reaches this point with outstanding last call comments, then 
there is a more basic failure.  Such a document should not have reached the 
point where a DISCUSS is required to keep it from progressing long enough 
for the comment to be addressed.

>> * The IETF as a whole does not have consensus on the technical
>> approachor  document. There are cases where individual working groups or
>> areas have  forged rough consensus around a technical approach which
>> does not garner IETF consensus. An AD may DISCUSS a document where she
>>  or he believes this to be  the case. While the Area Director should
>> describe the technical area where  consensus is flawed, the focus of the
>> DISCUSS and its resolution shouldbe on how to forge a cross-IETF
>>  consensus.
>
> the specification in the near-term. If no such people are active in the
> working group, then why has a standards effort been pursued?
>
> The above criterion says that these motivated participants' concerns are
> not sufficient, when weighed against the more detached desires of the
> active IETF community.
>
> IETF.  By implication, it says that we do not care as much about the
> people who are going to depend upon the protocol, as we do about those
> who won't.

Not necessarily.  It may say that the members of the working group have 
consensus to break the Internet in order to sell more of their product. 
The people who invented a thing, developed it, and depend on its acceptance 
in the marketplace for their livelyhood may be tempted to overlook any 
number of significant problems.  Part of the job of the IESG is to prevent 
such people from using the IETF to make the Internet worse instead of 
better.

Now, I'll agree that an objection on this basis indicates a failure.  But 
the failure is often the working group's in not understanding the bigger 
picture, not asking for help in doing so, or not caring about it. 
Sometimes, the failure is not that help was not requested, but that it was 
not given, or not useful.  However...

The IETF process is a cooperative process toward a common goal, not an 
adversarial process.  Our common goal is to make the Internet better; if 
your goal is to rubber-stamp your pet proposal or market your product 
whether it makes the Internet better or not, you don't belong here.

Arguments like "no one warned my about this issue before, so it wouldn't be 
fair to hold back my document now until it's fixed" manage to completely 
miss the point.  If fixing the problem before the document is published 
advances the goal, then that is what we should do.  If publishing the 
document without the fix advances the goal, then _that_ is what we should 
do.  Sometimes, the right answer is to do both!

If an IESG member places a DISCUSS on a document, and your response is "you 
don't have sufficient grounds for a DISCUSS" or "please tell me what change 
to make to my document to make the DISCUSS go away", then you are totally 
missing the point.  The correct response is "please help us to understand 
your concern so that we can work toward a resolution".  Sometimes that 
resolution will in fact be a simple and obvious change to the document. 
Sometimes it will involve pointing out the obvious reason why there is not 
really a problem.  Usually it will be somewhere in between.  It may involve 
publishing a document which has a known problem because doing so advances 
the goal more than fixing the problem would.  It may involve scrapping the 
whole document and starting over.  Usually, it will be somewhere in 
between.  Often, the way forward will not be immediately obvious to anyone 
involved.  THIS IS NOT A FAILURE.  It is the system working the way it is 
supposed to, with people working together to produce high-quality standards 
that improve the Internet.

So, is "constituency mismatch" a failure of the IETF?  Quite often, yes.
Does it indicate a fundamental flaw in the process?  NO.
It means that the process did not get followed, and in particular that a 
group failed to work with the community to produce a document which the 
IETF as a whole could support.

That may be because someone doesn't "get it", and thus wasn't aware of who 
they needed to work with or what they needed to do or how to ask for help. 
We should work with such people, both to help them "get it" and to help 
them get their work to a state where it is ready for publication.  Some 
time may be lost, but everything has a learning curve, including working in 
the IETF.  The failure is when such people _don't_ learn and continue to 
make the same mistakes, and _that_ failure is on the part of the people who 
do "get it".

It may be because someone believes their proposal is so urgent that there's 
not time to get wider review, or to fix the problems.  Very occasionally, 
there is a real urgency _for the Internet_ to get something out sooner 
rather than later.  In other cases, well, you probably should have known 
better (but see above).

It may be because someone is acting in bad faith.  That may not be the 
people who developed the proposal in question; it may be the proponents of 
some competing proposal, or some other adversary.  Obviously, it does not 
advance the goal to allow someone who is acting in bad faith to force 
publication of a "bad" document, nor to prevent publication of a good one. 
Fortunately, we have piles of documents describing process for dealing with 
this when it comes up.  And yes, there are even ways to deal with a DISCUSS 
placed on a document by someone acting in bad faith.  Hopefully, that will 
never become a problem.

Finally, it may indicate there was a genuine attempt to work with the 
community, and that resulted in a technical disagreement that could not be 
resolved, even under the "rough consensus" model.  If that happens, there 
may well be a real problem, but it is not a failure of the process.  If we 
cannot reach consensus on something, then we should not publish it; after 
all, when all is said and done, our documents are supposed to represent the 
consensus of the IETF.  They should consist of the things we have agreed to 
say, not the things we haven't agreed not to say.

> fielding only one proposal.  However there are plenty of examples of
> fielding multiple solutions, and letting the market choose among them.

I fully agree with this.  I have seen some arguments that the IETF should 
only standardize one way of doing any particular thing.  While that is a 
good rule at some levels, it is unnecessary at others.  Often there are 
multiple approaches to doing something which make different but more or 
less equally valid tradeoffs, and any of which would improve the Internet. 
In such cases, we should not be adverse to standardizing multiple 
approaches, unless doing so would be contrary to the goal.  So, the 
required consensus is (or should be) that publication of a document is good 
for the Internet, not that it represents the _only_ approach that is good 
for the Internet.  If someone has another approach that they think would be 
better, but fails to convince the WG or anyone else, then they are in the 
rough.  It should be quite rare indeed for the IETF to come to consensus 
that an alternate approach is _so_ good that not only should it be done, 
but the approach sitting in last call should not be published.

Similarly, I don't think we should necessarily require a large display of 
support to charter a working group.  Certainly, there should be enough to 
suggest that, if chartered, the group would actually be productive.  But if 
20 people believe that something is worth doing, then they should not be 
told "no" for no better reason than that out of 100 people attending a BOF, 
only those 20 raised their hands and said they'd participate.  Of course, 
it's a different story if the work is out of our scope, or more suited to 
another SDO, or could never result in an IETF consensus (and sometimes we 
even charter those).

> So the
> above

I don't believe unanimity is required, but rough consensus is.  Of course, 
a large number of last calls go by with no comment at all.  That's OK, 
because to a certain extent "consensus" means that people were given an 
opportunity to object and did not do so.  It would still bother me if I 
thought we were publishing lots of documents without any review, but I 
don't think that's the case -- part of the IESG's job is to perform such 
review and/or find others to do so, so that the rest of us _don't_ have to 
read every document that goes by.

-- Jeff

---

From: Pekka Savola <[email-masked]>
Date: Fri, 12 Jan 2007 10:54:18 +0200 (EET)

Jeff, you wrote a good note.  I'll use this as an opportunity to 
expand on one topic a bit:

On Thu, 11 Jan 2007, Jeffrey Hutzelman wrote:
> IMHO, this particular situation is more than just "procedural breakage". If a 
> document reaches this point with outstanding last call comments, then there 
> is a more basic failure.  Such a document should not have reached the point 
> where a DISCUSS is required to keep it from progressing long enough for the 
> comment to be addressed.

Well, it seems rather common that IETF LC comments (especially if not 
copied to [email-masked] list) are not responded.  To reduce delay, it 
also seems common that IESG telechat is scheduled as soon as possible 
after IETF LC closes, and document is usually not taken out of the 
agenda if comments are received during the LC.  Also sometimes the 
document gets approved without there being any record (e.g., on IESG 
ballots) that some comments had been made but there was no response.

Therefore it is not clear to me whether such comment was "addressed" 
(I'd call this 'processed') but without public record [e.g., editor or 
chair] in essence rejecting the comment (possibly in good faith) or 
not received at all (maybe also in good faith, e.g. if WG mailing list 
discards non-subscriber posts or the moderator is asleep).

Maybe we should be clearer on what the expectation for processing IETF 
LC comments is.  Unless we do, it is not obvious how we could evaluate 
whether the procedure has been carried out properly or not.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 12 Jan 2007 08:17:05 -0500

It seems like most other SDOs use formalized issue trackers for the  
equivalent of last call ("ballot") comments, making it easy to see  
what has been going on. Some WG do this, but each usually picking  
their own peculiar tracker.

The problem with any substantial IETF LC or WGLC comments is that  
they are often multiple issues, from the trivial spelling errors to  
fundamental architectural issues. It is difficult to make sure that  
all have been addressed and that discussions don't become >>>>>  
emails with meaningless subject lines - unintentionally making sure  
that nobody beyond the authors (if you're lucky) pays attention.

Putting all comments, including DISCUSS, into a document-specific  
issue tracker would be most helpful. (It would be helpful even beyond  
publication of an RFC, as we have found for the SIP documents, as  
they can be used to gather issues that a future "bis" effort needs to  
address. From what I can tell, almost all of the non-trivial protocol  
documents these days generate an draft-ietf-rfcXYZbis, after all.)

Henning

On Jan 12, 2007, at 3:54 AM, Pekka Savola wrote:

>> IMHO, this particular situation is more than just "procedural  
>> breakage". If a document reaches this point with outstanding last  
>> call comments, then there is a more basic failure.  Such a  
>> document should not have reached the point where a DISCUSS is  
>> required to keep it from progressing long enough for the comment  
>> to be addressed.
>
> Well, it seems rather common that IETF LC comments (especially if  
> not copied to [email-masked] list) are not responded.  To reduce  
> delay, it also seems common that IESG telechat is scheduled as soon  
> as possible after IETF LC closes, and document is usually not taken  
> out of the agenda if comments are received during the LC.  Also  
> sometimes the document gets approved without there being any record  
> (e.g., on IESG ballots) that some comments had been made but there  
> was no response.
>
> Therefore it is not clear to me whether such comment was  
> "addressed" (I'd call this 'processed') but without public record  
> [e.g., editor or chair] in essence rejecting the comment (possibly  
> in good faith) or not received at all (maybe also in good faith,  
> e.g. if WG mailing list discards non-subscriber posts or the  
> moderator is asleep).
>
> Maybe we should be clearer on what the expectation for processing  
> IETF LC comments is.  Unless we do, it is not obvious how we could  
> evaluate whether the procedure has been carried out properly or not.
>
> --

---

From: Sam Hartman <[email-masked]>
Date: Fri, 12 Jan 2007 16:08:09 -0500

>>>>> "Henning" == Henning Schulzrinne <[email-masked]> writes:

    Henning> Putting all comments, including DISCUSS, into a
    Henning> document-specific issue tracker would be most
    Henning> helpful. (It would be helpful even beyond publication of
    Henning> an RFC, as we have found for the SIP documents, as they
    Henning> can be used to gather issues that a future "bis" effort

Agreed provided that it is simple for me to open a new issue from my
email client, turn an email discussion into an issue, resolve and
issue using email, etc.  Also, I need to be able to update multiple
issues in an email message, etc etc.  Basically, I'm OK with an issue
tracker provided that it does not increase the complexity for those
doing cross-area review who want to open up a bunch of issues and
manipulate them.  Especially using a web based issue tracker is a
non-starter from that standpoint.  (You do probably want a web
interface to any issue tracker; you just don't want to destroy the
existing interface.)

It is possible to meet these requirements in an issue tracker design.

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 14 Jan 2007 09:31:29 +0100

On 2007-01-12 09:54, Pekka Savola wrote:

> copied to [email-masked] list) are not responded.  

Firstly, this is the reason we recently made some minor changes
in the text of the IETF Last Call messages, and why you will see a
related IESG statement in the next few days. We'd rather that
LC comments and follow-ups are on a public list in most cases.

> To reduce delay, it 
> after IETF LC closes, 

Yes, since the community has asked us to reduce delays...

> and document is usually not taken out of the 
> agenda if comments are received during the LC.

That depends on the AD's judgement whether the comments are serious
enough to definitely require a new I-D. Quite often the AD will prefer
to get any DISCUSSes on the table at the same time, again to reduce
delay. It's highly unlikely that a document would get approved
in its first appearance on the agenda in the presence of
non-editorial LC comments.

> Also sometimes the 

Well, again, it is entirely possible for the IESG to make a decision
in voice discussion that the LC comments in question are editorial
(in which case the draft can be approved subject to the AD requesting
the necessary editorial fixes). And the IESG can also decide in voice
discussion that a LC comment is simply in the "rough" part of rough
consensus. I'd say this is fairly unusual, but I don't have numbers.

I agree that in such a case, it would be at least courteous for the
AD to tell the commenter and the PROTO shepherd this.
> chair] in essence rejecting the comment (possibly in good faith) or not 
> received at all (maybe also in good faith, e.g. if WG mailing list 

IETF LC comments are supposed to be sent to the IETF list, which has a
public archive (or exceptionally to the iesg).
> whether the procedure has been carried out properly or not.

I think RFC 2026 is fairly clear and the issue is transparency - again,
that is why the IESG now has a preference for IETF LC comments to go to
the IETF list. That's a first step towards better tracking.

     Brian

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sun, 14 Jan 2007 15:27:47 -0600

Just a minor followup here...

>From a Brian-o-gram:

>> Maybe we should be clearer on what the expectation for processing IETF LC 
>> comments is.  Unless we do, it is not obvious how we could evaluate 

Until a couple of months ago, I didn't think that IESG had a preference 
about where I sent Last Call comments (IESG list or IETF list), so I was 
sending comments to the IESG list, for two reasons: (1) I was confused, and 
(2) I preferred to send comments the list where they did not tend to set off 
long e-mail discussions with other Last Call commenters.

In discussions with ADs during IETF 67, it became obvious to me that (1) the 
IESG actually DID have a preference for comments on the IETF list, although 
you couldn't tell it from the old-style Last Call text, and (2) this was the 
result of a desire for transparency, so likely a good thing.

I haven't seen an announcement of the new-style Last Call text, only its use 
on specific recent last calls (I saw it on 12/22 Last Calls, so it's pretty 
recent). If you have also seen so many Last Call e-mails that you no longer 
actually read them, you might not have noticed the new text that Brian's 
referring to.

Thanks,

Spencer

---

From: David Kessens <[email-masked]>
Date: Sun, 14 Jan 2007 20:51:40 -0800

> non-editorial LC comments.

As an AD, I do expect other ADs to remove a document from the agenda
if the Last Call comments are substantial and don't have a completely
obvious resolution and/or really fall into the "rough" part of the
consensus as determined by the working group chair and AD.

Basically, that means I don't have a problem with documents on the
agenda that have known minor editorial issues or issues that have a
simple and straightforward solution that were brought up during the
Last Call.

Obviously, this involves a judgement call and I am happy to trust my
colleagues to make such decision. As with all human decision making,
we sometimes don't agree and have a discussion whether a document
should really be on the agenda or not. In addition, there is the
factor of time differences, vacation time and otherwise that sometimes
will result in a situation that the document is only removed at quite
a late stage.

Personnally, I rather have us occasionally fix a problem because we
are a bit too aggressive in getting a document on the agenda than
being so careful that all documents end up incurring more delays
during the IESG review phase.

David Kessens
---

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 15 Jan 2007 08:41:44 +0100

> I haven't seen an announcement of the new-style Last Call text, only its 
> use on specific recent last calls (I saw it on 12/22 Last Calls, so it's 
> pretty recent). If you have also seen so many Last Call e-mails that you 
> no longer actually read them, you might not have noticed the new text 
> that Brian's referring to.

That's why there is an IESG Statement in the secretariat's queue,
to be posted to the ietf-announce list.

    Brian

---

From: John C Klensin <[email-masked]>
Date: Mon, 15 Jan 2007 11:12:23 -0500

--On Sunday, 14 January, 2007 09:31 +0100 Brian E Carpenter
<[email-masked]> wrote:

>> Maybe we should be clearer on what the expectation for
>> processing IETF  LC comments is.  Unless we do, it is not
>> obvious how we could evaluate  whether the procedure has been
>> carried out properly or not.
> 
> I think RFC 2026 is fairly clear and the issue is transparency
> - again, that is why the IESG now has a preference for IETF LC
> comments to go to the IETF list. That's a first step towards
> better tracking.

Brian,

Especially in conjunction with the piece of text that says
"retain the first part of the subject line", it is also a first
step to more noise on the IETF list.  The IESG may well have
made the right decision here, but the recent explosion of the
usual IPR debate under the topic of draft-legg-xed-asd --
drowning out any possible substantive discussion on a document
for which some serious discussion is probably in order-- may
indicate that some further tuning is in order.  I note that
proposals to send comments to the WG list and discuss them there
don't work because the document is an individual submission.

Convincing more people that the S/N ratio on the IETF list is
such that it isn't worth following would not be a good
consequence of this decision.

     john

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 15 Jan 2007 19:32:09 +0100

John C Klensin wrote:

> The IESG may well have made the right decision here

s/may well have/certainly has/  I think it's a bug that
I-D.iab-publication-00.txt offers no list for public
Last Call comments.  IMO the rare Last Calls for I-Ds
published by the IAB could also use this list here for
public comments.

> the recent explosion of the usual IPR debate

13 articles so far, no big explosion.  If a Last Call
discussion here gets too specific it's still possible
to continue it on a more specific list - in that case
the IPR WG list.

Frank