Subject: RE: Appeal against IESG blocking DISCUSS on draft-klensin-rfc2821bis

From: Russ Housley <[email-masked]>
Date: Tue, 24 Jun 2008 16:35:46 -0400

Bernard:

The data is all public.  Jari has done a very good job of extracting 
the data from the I-D Tracker and making it accessible to 
everyone.  Of course, any requests for changes to additional graphs 
need to go to Jari.

http://www.arkko.com/tools/admeasurements/stat/base.html

Russ

At 11:32 AM 6/24/2008, Bernard Aboba wrote:
> > We have a way to count DISCUSS positions, but we do not have a way to
> > figure out what percentage of them are perceived as "late surprises"
> > by the community. So, while we are taking action in an attempt to
> > make things better, we do not have a way to measure our success or
> > failure beyond community perception. Suggestions on making this more
> > objective and less subjective are greatly appreciated.
> >
> > Russ
>
>I agree that it might be hard to measure the effect of particular 
>actions (e.g. changes in
>procedures). However, it is *not* hard to measure overall trends in 
>performance, and
>to break these trends down between areas and types of documents.
>
>My understanding is that the Simcoe dataset continues beyond 2001, 
>and that with some
>relatively modest effort, a detailed analysis could be produced 
>quantifying how the IETF
>as performed. This would give us a window into what the actual 
>results have been.

_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Jari Arkko <[email-masked]>
Date: Wed, 25 Jun 2008 14:37:15 +0300

Bernard, Russ,

I changed the subject line, I think the thread has continued long enough 
:-)  Indeed, I collect a set of measurements. These are based on pulling 
information from the tracker and the documents. The reason for setting 
this up was to try to better understand what is happening in the IETF 
process at various levels: for each document, my own AD work, how the 
IESG works, where is the time spent in the IETF process as a whole.

   http://www.arkko.com/tools/admeasurements/ (start page and disclaimers)
   http://www.arkko.com/tools/admeasurements/stat/base.html (main IESG 
measurements)
   
http://www.arkko.com/tools/lifecycle/draft-ietf-mobike-protocol-timing.html 
(exists for each approved draft)

The data is current and is being updated every week or so. But before 
everyone looks at the graphs and jumps to conclusions, I wanted to 
insert a few disclaimers. First, this is based on tracker data and it 
does not go into that far into the past -- and some of the older data 
might even not be all that accurate. Second, I recently added a lot of 
measurements and I don't yet have 100% confidence that my tool gets 
everything correct. This is particularly true of the items looking at 
various aspects of Discusses. Third, some of the measurements look at 
tracker substates, which are not used by all ADs and no one uses them 
completely accurately. And last but not least, I don't want anyone to 
blindly look at the numbers without considering what's behind them. 
There are significant variations between documents. Tracker status and 
real world situation may be different. Areas differ. There is limited 
amount of information about documents prior to them becoming WG 
documents. And its not necessarily the goal of the IETF to produce a 
large number of documents as fast as it can; I have no way to measure 
quality in these numbers. These are so hard issues that its not even 
clear that the statistics have more value than acting as entertaining 
trivia about the process.

That being said, it is beneficial to understand what is happening and 
what changes are occurring in the process. Or understand where 
improvements might have a negligible vs. visible impact.

One of the things that the IESG has been concerned about recently is 
that the number of outstanding Discusses is growing. We talked about 
this in our May retreat and identified some actions to help reduce this 
problem. For instance, better tool support so that the ADs would better 
see the different things that are waiting for their action, getting the 
document shepherds better involved in the resolution process, informing 
authors how they should respond to Discusses, using RFC Editor notes to 
make small changes when the authors are slow in producing a new version, 
better checks of documents before they come to telechats (e.g., to 
ensure that formal syntax in a document is free of errors), etc. These 
would be in addition to the usual things we'd do, like debate whether 
the Discuss was within the Discuss criteria, whether the issue is real, 
try to ensure that the AD and the authors are being responsive over 
e-mail, etc.

Another interesting area to think about is the time that our processes 
take. For instance, documents that go through me take on the average 
five months from publication request to approval. But there is a lot of 
variation. This time includes everything from AD review, IETF Last Call 
to IESG review and waiting for document revisions, etc. One interesting 
piece of information is that documents that require no revision during 
this process are very rare, but they go through the system about five 
times as fast. If we look at the (unreliable) substates, they appear to 
indicate that the IESG processing time is divided roughly 1:2:1 for 
waiting on AD, authors, and mandatory process waits like last call periods.

I am working to extend the analysis a little bit further by including 
individual draft and WG document stages. You see some of the results of 
this in the third URL above, but I'd like to understand what fraction of 
the overall draft-smith-00 to RFC time is taken by the different stages 
for all IETF work, and how the stages have developed over time.

Comments and suggestions on what would be useful to measure are welcome.

Jari

---

From: Marshall Eubanks <[email-masked]>
Date: Wed, 25 Jun 2008 09:56:04 -0400

Dear Jari;

On Jun 25, 2008, at 7:37 AM, Jari Arkko wrote:

> Bernard, Russ,
>
> I changed the subject line, I think the thread has continued long  
> enough :-)  Indeed, I collect a set of measurements. These are based  
> on pulling information from the tracker and the documents. The  
> reason for setting this up was to try to better understand what is  
> happening in the IETF process at various levels: for each document,  
> my own AD work, how the IESG works, where is the time spent in the  
> IETF process as a whole.
>
> The data is current and is being updated every week or so. But  
> before everyone looks at the graphs and jumps to conclusions, I  
> wanted to insert a few disclaimers. First, this is based on tracker  
> data and it does not go into that far into the past -- and some of  
> the older data might even not be all that accurate. Second, I  
> recently added a lot of measurements and I don't yet have 100%  
> confidence that my tool gets everything correct. This is  
> particularly true of the items looking at various aspects of  
> Discusses. Third, some of the measurements look at tracker  
> substates, which are not used by all ADs and no one uses them  
> completely accurately. And last but not least, I don't want anyone  
> to blindly look at the numbers without considering what's behind  
> them. There are significant variations between documents. Tracker  
> status and real world situation may be different. Areas differ.  
> There is limited amount of information about documents prior to them  
> becoming WG documents. And its not necessarily the goal of the IETF  
> to produce a large number of documents as fast as it can; I have no  
> way to measure quality in these numbers. These are so hard issues  
> that its not even clear that the statistics have more value than  
> acting as entertaining trivia about the process.

> That being said, it is beneficial to understand what is happening  
> and what changes are occurring in the process. Or understand where  
> this in our May retreat and identified some actions to help reduce  
> this problem. For instance, better tool support so that the ADs  
> would better see the different things that are waiting for their  
> action, getting the document shepherds better involved in the  
> resolution process, informing authors how they should respond to  
> Discusses, using RFC Editor notes to make small changes when the  
> authors are slow in producing a new version, better checks of  
> documents before they come to telechats (e.g., to ensure that formal  
> syntax in a document is free of errors), etc. These would be in  
> addition to the usual things we'd do, like debate whether the  
> Discuss was within the Discuss criteria, whether the issue is real,  
> Another interesting area to think about is the time that our  
> processes take. For instance, documents that go through me take on  
> the average five months from publication request to approval. But  
> there is a lot of variation. This time includes everything from AD  
> review, IETF Last Call to IESG review and waiting for document  
> revisions, etc. One interesting piece of information is that  
> documents that require no revision during this process are very  
> rare, but they go through the system about five times as fast. If we  
> look at the (unreliable) substates, they appear to indicate that the  
> IESG processing time is divided roughly 1:2:1 for waiting on AD,  
> authors, and mandatory process waits like last call periods.
>
> I am working to extend the analysis a little bit further by  
> including individual draft and WG document stages. You see some of  
> the results of this in the third URL above, but I'd like to  
> understand what fraction of the overall draft-smith-00 to RFC time  
> is taken by the different stages for all IETF work, and how the  
> stages have developed over time.
>

I, at least, always like looking at such statistics, so I thank you.

In my opinion, the real value of this come from trends. The IESG  
clearly should not have
Soviet style work norms, for a bunch of reasons, and it may not be  
possible to say what an approval rate should be,
but if something changes greatly over time it is generally worthwhile  
to figure out why.

To that end, I frequently find it useful to put means and standard  
deviations as horizontal lines on the plot, at least once enough data  
is available (say, 5 data points). (You can do the mean as a solid  
line and the mean + and - one or two standard deviations as dashes or  
dots or a fainter line).

The assumption is that there are underlying random (gaussian or  
poisson) processes that cause the numbers to fluctuate about the mean  
and that there is no other way to determine these means and S.D.  
except from the data. Under these assumptions, there should be fairly  
frequent fluctuations > 1 standard deviation from the mean, and  
occasional fluctuations > 2 standard deviations from the mean, but a  
string of fluctuations 2 + standard deviations from the mean probably  
should be looked into. (Of course, the underlying assumptions here may  
be faulty, and
these bounds may need to be adjusted based on experience.)

BTW, this page seems to have blank plots, which may be due to not  
enough data :

http://www.arkko.com/tools/admeasurements/stat/Eronen_Pasi-approved.html

Regards
Marshall

---

From: Lakshminath Dondeti <[email-masked]>
Date: Wed, 25 Jun 2008 08:44:32 -0700

Hi all,

I am concerned by the following trends:

* Number of outstanding Discusses is growing. (Thanks to Jari's data)

* The extent of text changes as part of Discuss Resolution is increasing 
(I have only anecdotal evidence on this; perhaps others have statistics).

* In some cases, members of the IESG are pretty much writing core parts 
of documents (or worse yet make authors iterate until the text is 
satisfactory), overruling WG consensus, based on personal opinions or 
citing solicited reviews.

There are a number of derivative trends as well:

* ADs who cannot convince working groups seem to use the threat of 
DISCUSS to get their way.

* I would have thought document shepherds represent and defend working 
group consensus and shepherding ADs defend the completeness, accuracy, 
relevance of the drafts and defend their assessment of the IETF 
consensus.  But, increasingly, it seems to fall on an AD holding a 
DISCUSS position, and authors to discuss and agree on text which becomes 
finalized without any other review.

(Note: I am a guilty party in some of these cases, as document author 
and document shepherd.  In all cases, I seem to have looked for an 
expedient way out rather than find a solution to what seems to be 
problematic).

* One of the new trends seems to be to use DISCUSS to include 
applicability statements in current drafts to avoid potential DISCUSS 
positions on future drafts.

That or some variation of that is status quo.  It should not be that 
way.  I would like to see a better definition of the roles of the WG, 
authors of a document, document shepherds, shepherding ADs and other ADs 
in our standards process.

I would like to hear others' opinions (I was going to put together a 
draft with some ideas on how we might define these roles, but I want to 
hear others' thoughts before I do that) on this topic.

thanks,
Lakshminath

On 6/25/2008 4:37 AM, Jari Arkko wrote:
<deleted text>

---

From: Melinda Shore <[email-masked]>
Date: Wed, 25 Jun 2008 12:19:18 -0400

> hear others' thoughts before I do that) on this topic.

I think your points are valid, but I'm not sure what the
effect would be if you controlled for quality coming out
of the working groups.  That is to say, I think that
occasionally working groups are coming to consensus on
bad documents or bad ideas, and that the incidence of that
is increasing.  If that's true it once again raises the
very familiar question of picking up quality problems
earlier in the process.  Actually, that latter question
applies regardless.

Melinda

---

From: "Ross Callon" <[email-masked]>
Date: Wed, 25 Jun 2008 13:19:19 -0400

I think that this is a very good partial summary of the trends, and I am
also concerned about these. 

My personal opinion only: I think that we need stronger guidelines on
what is a valid discuss, and what is not. We might need some mechanism
for the WG or the IETF or the IAB to override a DISCUSS that does not
represent consensus. As one example, personally I would prefer to not
allow DISCUSS's for the purpose of motivating future or ongoing work --
there are lots of people associated with the IETF who would like to get
their names on drafts, and if work is needed it should be possible to
motivate it without stopping progression of some different related
draft. Also, if work is being done motivated solely by the desire to get
past a DISCUSS comment and without any intention of ever deploying the
result, then there is a very real possibility that the work might not
result in the most practical solution.

I think that producing and coming to consensus on these guidelines might
be a significant challenge. However, I think that this is needed. 

Ross

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Lakshminath Dondeti
Sent: 25 June 2008 11:45
To: [email-masked]
Subject: Qualitative Analysis of IETF and IESG trends (Re: Measuring
IETFand IESG trends)

Hi all,

(I have only anecdotal evidence on this; perhaps others have
statistics).

thanks,
Lakshminath

> this in our May retreat and identified some actions to help reduce
this

---

From: Jari Arkko <[email-masked]>
Date: Wed, 25 Jun 2008 21:30:46 +0300

Lakshminath,

Better understanding of the type of behaviors in this space would 
certainly be useful. And I don't want to disagree with your assessment 
of the behaviors; many of them sound like something that appears in 
practice. In particular, the shepherds are far less involved in the 
Discuss resolutions than the authors. And we do not involve the WGs as 
much as we should. I think writing guidelines on what the role of the 
various persons in the process is would be very useful. And obviously we 
should start by better following of the existing documents, like the 
Proto Shepherd RFC or the Discuss criteria document.

However, with regards to blocking consensus of a WG, please remember 
that the WG is not necessarily the only place where consensus is tested. 
I recently had a case which had significant IETF Last Call discussion. I 
held a Discuss to ensure that the (fairly clear, IMO) conclusion from 
the discussion would be taken into the document. It did eventually, but 
only after significant back-and-forth with the authors. Overriding the 
original WG consensus? Yes. Right thing to do? I think so, not only was 
it right technically but it was something backed up by the Last Call. 
Did we get the details right, did the text go too far or did it fall 
short? I don't know, its a judgment call. The end result was somewhere 
between the LC guidance and authors' opinions. Painful for the WG? Sure.

On text that comes from the IESG: this is more common in recent years 
than it was before. I am one of the ADs who tends to do that, both for 
the documents that I sponsor and for resolving my Discusses. But I would 
rather not do it. But I often end up doing it if there is no progress 
otherwise; I want to get my sponsored documents approved and I want to 
reduce the list of my outstanding Discusses. If I can help my authors by 
proposing text, I will do it. But I would really like to see the 
document shepherds in active role here. Or at least the authors. The 
general guidance for authors whose document gets a Discuss is to first 
confirm whether the raised issue is a real one or not. If it is not, ask 
the Discuss to be cleared. Fight if needed! If it is real, work with 
your shepherd and WG to develop a proposal to fix the problem. Mail the 
proposal to the Discussing ADs in a timely manner. Address explicitly 
all components raised in a Discuss, either by explaining how they are 
not issues or providing a solution to resolve the issue.

Jari

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 26 Jun 2008 09:41:24 +1200

> not issues or providing a solution to resolve the issue.

Our fundamental collective job is defined in RFC 3935:

   The mission of the IETF is to produce high quality, relevant
   technical and engineering documents that influence the way people
   design, use, and manage the Internet in such a way as to make the
   Internet work better.

That means that it is *not* our collective job to ensure that a WG
consensus survives critical review by the IETF as a whole and by
the IESG, if there's reason to believe that the IETF as a whole
doesn't agree with the WG consensus. And it's clearly the IESG's
job to ensure that the critical review and final consensus (or lack
of consensus) occur.

That, IMHO, is the proper role of a DISCUSS and the proper reason
for delays in document approval. And if we see fluctuation in
these delays, and fluctuation in the amount of active intervention
by the ADs, it does not follow that the IESG is to blame. Maybe
there are external factors, maybe there are WGs that are forgetting
the IETF's mission, maybe our technology is getting harder and
more complex. So I'm very dubious about using either quantitative
*or* qualitative observations to point the finger at the IESG, or
at process issues in general, without digging much deeper.

Of course, the IESG needs to pay attention to delays,
so Jari's data (like the earlier data that Bill Fenner used to
produce) are very valuable. And of course, individual ADs
have to think carefully whether a given issue is or is not
worthy of a DISCUSS, and sometimes they get it wrong. But
that will always be true, however we tune the process and
procedures.

    Brian

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 25 Jun 2008 16:52:53 -0500

> And if we see fluctuation in

I'm remembering conversations with Allison and some other Proto types where 
Allison made the point that the tracker stages didn't actually map onto who 
held the token very cleanly - I'd agree with Brian's point here.

Thanks,

Spencer

---

From: John C Klensin <[email-masked]>
Date: Wed, 25 Jun 2008 19:28:34 -0400

--On Thursday, 26 June, 2008 09:41 +1200 Brian E Carpenter
<[email-masked]> wrote:

>...
> And of course, individual ADs

Brian,

While I agree with this, I also believe that there have to be
effective safeguards against bad judgments prevailing for too
long.  And I believe that those have largely slipped away from
us... unless we believe that making changes, unvalidated by WG
or mailing list consensus, simply to get a DISCUSS removed is
the right way to get to "high quality, relevant technical and
engineering documents...".

    john

---

From: Lakshminath Dondeti <[email-masked]>
Date: Thu, 26 Jun 2008 02:28:51 -0700

On 6/25/2008 9:19 AM, Melinda Shore wrote:
> is increasing.  

Well that is a disturbing trend as well.  A long while ago now, one of 
the then ADs mentioned that he needed to put a DISCUSS on a few 
successive MSEC documents that I was shepherding and mentioned that he 
wants to have a chat with the chairs on the quality of documents that we 
are forwarding.  I asked him to come to one of our meetings and explain 
the expectations directly to the WG.  That never happened.

But that is the kind of direction or steering an area director might do 
if working groups are indeed producing bad documents and advancing bad 
ideas.  Presumably they are several cases here, viz., going against 
charter or just being plain terrible at writing interoperable 
specifications.

Pushing a document back to the WG is actually a better thing, I am 
beginning to think.  Sure that may introduce delays initially as we 
learn how to operate in that mode, but the process of writing 
specifications is more transparent and more consensus based than in the 
current model of operation.

One of the problems with the current model is that comments toward the 
end of the process, either AD's or reviewers', are weighted more than 
comments during the working group discussions, at least in some cases.

> If that's true it once again raises the

Yes, I have heard it mentioned several times before, but I haven't seen 
any concrete steps to achieve that.  I am now making the case that if a 
document makes it beyond the shepherding AD and put on the IESG 
telechat, changes to the text should be relatively a big deal.  There 
can be changes, but they should be reviewed by the WG.

In some cases, there have been bitter disputes about taking change 
control from authors and putting it in the hands of a WG, but the 
current model toward the end of the process seems to put change control 
in too few hands.  That is what I am trying to highlight.

regards,
Lakshminath

---

From: Dave Crocker <[email-masked]>
Date: Wed, 25 Jun 2008 09:40:21 -0700

Melinda Shore wrote:
> of the working groups. 

The IETF works without any effort to measure quality or even uptake.

As a consequence, we have no way of determining whether our protocols succeed in 
the long-run, or whether AD Discusses are true diligence or mere whimsy.

What we are left with is isolated cases and reactions.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Lakshminath Dondeti <[email-masked]>
Date: Thu, 26 Jun 2008 04:35:21 -0700

Jari,

Thanks.  Some thoughts inline:

> Proto Shepherd RFC or the Discuss criteria document.

Of course.  I will try and see if I can put something of an initial 
draft coherently.

> between the LC guidance and authors' opinions. Painful for the WG? Sure.

So, here is where I am probably confused as to how consensus is 
determined.  I understand that the documents are eventually "IETF 
Consensus" with the IESG determining what the consensus is.  The 
sponsoring AD has the overall view of any given document better than 
anyone else, at least in the authoritative role (for this discussion, I 
am just considering WG documents).

Consider a hypothetical case: a large WG has strong consensus on one of 
their documents, they believe it is within the charter's scope and think 
that the document is in the best interest of the Internet.  The WG 
chairs assess the consensus, and forward the document to the shepherding 
AD.  The shepherding AD takes one last look, determines everything is in 
order and sends it to last call.  A few people on the IETF Discussion 
list think that the proposed specification is about to doom the 
Internet.  A few others who have not even read the document agree based 
on emails.  Most of the WG members are either not on the IETF list or 
choose to stay silent.

The shepherding AD considers those comments, thinks that those issues 
have been addressed already and puts the document on the IESG agenda. 
All other ADs (except one) think that everything is fine and vote No 
Objection.  One AD agrees with the few people on the IETF Discussion 
list and decides to put a DISCUSS and proceeds to hack the document.  In 
the current model, other than the very few exceptions cited recently, 
the AD gets what he or she wants for the most part.  It is plausible 
that AD may do this even if no one else identified a problem.

Responding to Brian here: Note that I am not pointing fingers at IESG 
alone, but yes, I am pointing fingers at our process in that the 
hypothetical situation described above is "allowed" to happen at the 
moment.  Brian suggests "external" factors, but I am sorry I am not 
convinced.  If there are WGs that have forgotten the IETF's mission, it 
is one of the primary roles of the ADs and the IESG to steer those WGs 
in the right direction.  If our technology is becoming harder and more 
complex, it calls for involving more people and increasing transparency 
and not the other way around.

> proposing text, I will do it. 

Jari, as I have noted before, if the status quo is considered the best 
way forward, I would rather you continue to propose text.  I as an 
author and document shepherd have appreciated that compared to the 
alternative.  That alternative, the iterative guess work, takes forever.

> But I would really like to see the 

What I am getting at is that the shepherding AD believes that the 
document is ready to move forward when he or she puts the document on 
the IESG agenda.  He or she should make a decision about the DISCUSS:

1.  Editorial changes are needed for clarification.  Suggests an RFC Ed 
note that the rest of the IESG is okay with as well, sends a 1-wk call 
to the WG, and then sends approved announcement if there are no objections.

2. Technical changes are needed.  The DISCUSS AD identified something 
everyone else missed or made a mistake about and the Shepherding AD (as 
well as the rest of the IESG) agrees with that assessment.  This should 
be a big deal.  The shepherding AD has been following the work for 
several years at some level (I understand the turnover in the AD 
position, but let's consider the general case) but still missed the 
issue that the DISCUSS AD identified after a quick read of the same 
document.

The Shepherding AD, document shepherd and the document authors 
understand the issue, prepare a short summary of the issue that every 
one agrees on, and a optionally possible solution and go back to the 
working group.  Clearly, this has to be a new issue or an old issue with 
substantial new information that will presumably convince many people to 
change their minds.  The process starts again.

3. No changes are needed.  Work as quickly as possible to convince the 
DISCUSS AD to remove the DISCUSS using all means at his or her disposal. 
  Note that this is not a discussion between two ADs at this stage in 
the process; the Shepherding AD represents years of work, and has 
considered most if not all of the information available.  The DISCUSS AD 
comparatively has less information.

This avoids coming up with compromise text between just the authors and 
one or more DISCUSS ADs.  There are many other people in the process and 
they have a huge say in how the document got to the shape it is in by 
the time it arrives at the IESG.

regards,
Lakshminath

---

From: Lakshminath Dondeti <[email-masked]>
Date: Thu, 26 Jun 2008 04:38:11 -0700

> engineering documents...".

Indeed that is the issue!

regards,
Lakshminath

---

From: Lakshminath Dondeti <[email-masked]>
Date: Thu, 26 Jun 2008 04:43:00 -0700

> job to ensure that the critical review and final consensus (or lack

But, surely the WG consensus counts as part of the overall IETF 
consensus process, doesn't it?  Please see the example in my response to 
Jari.  The shepherding AD (or at least the document shepherd) has an 
idea of the WG consensus as well as the IETF consensus.  We cannot 
simply weigh the latest opinions more than all the discussions that have 
happened as part of the WG consensus.

> that will always be true, however we tune the process and

I am suggesting that we make further progress along the lines of the 
definition of the role of the document shepherd and reassert (or clearly 
define and state the expectations) on the roles of the document shepherd 
and shepherding AD.

regards,
Lakshminath

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 27 Jun 2008 11:23:47 +1200

Lakshminath,

...
> happened as part of the WG consensus.

At one level I agree. But suppose that the set of people who are
active in the SXFG7M WG are so focused on the sxfg7m protocol that
they have all missed the fact that it's extremely damaging to
normal operations of the m7gfxs protocol? And this includes the
responsible AD, who has no deep knowledge of m7gfxs? This is the sort
of problem that IETF Last Call and IESG review is intended to find,
and it may well mean that the WG consensus ends up being irrelevant
to the IETF non-consensus. (I'm not in the least suggesting that
this applies to the draft that led to the appeal that led to this
thread.)

My conclusion, again, is that in the end this is the sort of
judgment call that we *expect* the IESG to make. And when we
feel they've misjudged, we appeal, and that tunes their judgment
for the future.

    Brian

---

From: John C Klensin <[email-masked]>
Date: Thu, 26 Jun 2008 19:36:17 -0400

--On Friday, 27 June, 2008 11:23 +1200 Brian E Carpenter
<[email-masked]> wrote:

>...
> At one level I agree. But suppose that the set of people who
> are active in the SXFG7M WG are so focused on the sxfg7m
> protocol that they have all missed the fact that it's
> extremely damaging to normal operations of the m7gfxs
> protocol? And this includes the responsible AD, who has no
> deep knowledge of m7gfxs? This is the sort of problem that
> IETF Last Call and IESG review is intended to find, and it may
> well mean that the WG consensus ends up being irrelevant to
> the IETF non-consensus. (I'm not in the least suggesting that
> this applies to the draft that led to the appeal that led to
> this thread.)
> 
> feel they've misjudged, we appeal, and that tunes their
> judgment for the future.

Brian,

Again, I agree.  And this sort of example is precisely one of
the major reasons why proposals to let a WG override an IESG
conclusion about consensus are problematic (the other involves a
WG that has been "captured" by a particular interest in order to
get a particular protocol output, but I'd expect an AD to notice
that and deal with it long before documents go into Last Call).

However, it means that we need to treat appeals about IESG
judgment calls as a completely normal and orderly part of the
process, rather than something sufficiently unusual that either 

	(i) the IESG treats an appeal as an attack on them
	collectively and starts circling the wagons rather than
	considering the appeal as a normal and formal request
	from the community for reconsideration, or that 
	
	(ii) IETF participants believe that they need to fear
	retaliation from one or more ADs if they generate appeals

independent of what might or might not be going on with recent
cases, I believe we have seen both phenomena in the last several
years.

This is not an easy problem.

     john

---

From: SM <[email-masked]>
Date: Thu, 26 Jun 2008 18:35:12 -0700

At 04:43 26-06-2008, Lakshminath Dondeti wrote:
>consensus process, doesn't it?  Please see the example in my 
>response to Jari.  The shepherding AD (or at least the document 
>shepherd) has an idea of the WG consensus as well as the IETF 
>consensus.  We cannot simply weigh the latest opinions more than all 
>the discussions that have happened as part of the WG consensus.

The document may be a product of WG consensus.  It still has to pass 
through the community and the IESG to be published as an IETF document.

The WG knows about the internals of the document and generally have a 
focused view.  The last call allows a wider range of input and to 
gauge the impact the proposal may have in other areas.  It is not 
about weighing the latest opinions more.  The author/shepard can 
always post an explanation.  The participants in the WG should be 
aware that there will be an IETF-wide last call.  You cannot blame 
the process if they choose to remain silent instead of taking part in 
the last call.  Note that letter-writing campaigns in a last call 
have been proven to be ineffective.

Regards,
-sm

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 27 Jun 2008 04:09:52 +0200

> This is not an easy problem.

Then let's do something easy first, add a section about
"shepherds and individual sheep" in chapter 8 of TAObis.

It is hard to find this info in Brian's marauder's map.
Some points:  Anybody can recommend an individual draft
for publication on standards track (RFC 2026 6.1.1).

By one of those rules that are no policy and hidden in
an ex-ION archive that recommendation has to include a
proposed write-up (roughly the same as for WG drafts),
and it is sent to the ADs of the corresponding area(s),
with a copy to iesg@, because it actually constitutes
a PubReq.  Informing the authors helps, the initiative
is doomed without cooperation from the author(s).  

After that the ADs can toss a coin who if anybody wants
to "sponsor" the draft, and appoint a shepherd for some
clerical tasks (watching the draft tracker, figure out
where the "token" is when it is MIA, and tickle author,
sponsor, or "discussing" ADs as needed).

The usual "those who propose" might apply, but ADs are
free to do this "shepherding" without help.  A proposed
write-up covers checked ABNF, checked nits, all required
reviews ready, etc., down to the "1F" question.

But unlike WG shepherds, who are typically co-Chairs, an
individual shepherd does not *decide* about consensus or
not, like an author does not *decide* this.  Only Chairs
and ADs are entitled to decide this, it's an appealable
decision.  Authors, editors, and shepherds are immune.

 Frank

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 26 Jun 2008 19:37:37 -0700

>"shepherds and individual sheep" in chapter 8 of TAObis.

Wearing my Tao hat: specific proposed text is appreciated. Taobis is 
still definitely an open document. (For IETF newcomers reading this 
thread: the latest version of the Tao can be found at 
<http://www.ietf.org/tao.html>.)

--Paul Hoffman, Director
--VPN Consortium

---

From: SM <[email-masked]>
Date: Fri, 27 Jun 2008 10:47:13 -0700

Hi Lakshminath,
At 07:11 27-06-2008, Lakshminath Dondeti wrote:
>Is it really necessary for all the battles to repeat on the IETF 
>list? Why can't the shepherding AD judge the overall consensus?

No, it is not necessary.  One could read the WG discussion of the 
topic instead of rehashing the same arguments on the IETF list.  The 
shepherd is there to gather information, get the document through the 
various stages and provide coordination.

The PACT I-D may be a useful read:

"An IETF effort is designed to resolve engineering choices for one 
issue and then move to a new issue.  It is not reasonable to permit 
arbitrary criticisms to be raised late in the process, derailing the 
incremental effort of a WG.

It is always reasonable to raise fundamental engineering problems, 
but it is essential to distinguish these from matters of engineering 
aesthetics.  In particular, the IETF Last Call and IESG review 
periods are not intended for second-guessing a WG's design choices -- 
the purpose of an IETF Last Call and IESG review is to focus on the 
overall viability of the document."

I'll also highlight a few points from RFC 3774:

Participants are frequently allowed to re-open previously closed 
issues just to replay parts of the previous discussion without 
introducing new material.  This may be either because the decision 
has not been clearly documented, or it may be a maneuver to try to 
get a decision changed because the participant did not concur with 
the consensus originally.

On the other hand, the decision making process must allow discussions 
to be re-opened if significant new information comes to light or 
additional experience is gained which appears to justify alternative 
conclusions for a closed issue.  One cause that can lead to 
legitimate attempts to re-open an apparently closed issue is the 
occurrence of 'consensus by exhaustion'.

The IETF culture of openness also tends to tolerate participants who, 
whilst understanding the principles of the IETF, disagree with them 
and actively ignore them.  This can be confusing for newer 
participants, but they need to be made aware that the IETF does not 
exclude such people.

Regards,
-sm 

Ietf mailing list
[email-masked]

---

From: Lakshminath Dondeti <[email-masked]>
Date: Fri, 27 Jun 2008 07:09:40 -0700

Brian,

Thanks for your response.  Please see inline:

> this applies to the draft that led to the appeal that led to this

For what it's worth, I am not talking about a specific draft or a 
specific WG at this point.  I am of the opinion that we are not 
discussing a one-off issue.

If protocol X disrupts protocol Y, we get into very interesting 
situations.  It is also going to get us into a rathole that I want to avoid.

My point was this: if a WG actually missed anything substantial and that 
comes out during an IETF last call, and the shepherding AD agrees, the 
document gets sent back to the WG.  If the shepherding AD also misses or 
misjudges, any member of the IESG can send it back to the WG for 
resolution.  What I think is not acceptable is for the author and one or 
more DISCUSS ADs to hack up the document and publish it.

If it so happens that the issue raised was considered and ruled out as a 
non-issue by the WG, then the shepherding AD knows the situation 
already.  Strong consensus in the working group damaging a protocol that 
matters to very few people (ok, that's a rathole) -- but here is where 
judgment is necessary.  And as you note, any of the judgment calls are 
appealable.

regards,
Lakshminath

---

From: Lakshminath Dondeti <[email-masked]>
Date: Fri, 27 Jun 2008 07:11:08 -0700

On 6/26/2008 6:35 PM, SM wrote:
>> consensus process, doesn't it?  Please see the example in my response 
>> to Jari.  The shepherding AD (or at least the document shepherd) has 
>> an idea of the WG consensus as well as the IETF consensus.  We cannot 
>> simply weigh the latest opinions more than all the discussions that 
>> have happened as part of the WG consensus.
> 
> focused view.  The last call allows a wider range of input and to gauge 
> the impact the proposal may have in other areas.  It is not about 
> weighing the latest opinions more.  The author/shepard can always post 
> an explanation.  The participants in the WG should be aware that there 
> will be an IETF-wide last call.  You cannot blame the process if they 
> choose to remain silent instead of taking part in the last call.  Note 
> that letter-writing campaigns in a last call have been proven to be 
> ineffective.

Is it really necessary for all the battles to repeat on the IETF list? 
Why can't the shepherding AD judge the overall consensus?

thanks,
Lakshminath

---

From: Russ Housley <[email-masked]>
Date: Fri, 27 Jun 2008 11:32:12 -0400

Lakshminath:

>Consider a hypothetical case: a large WG has strong consensus on one 
>of their documents, they believe it is within the charter's scope 
>and think that the document is in the best interest of the 
>Internet.  The WG chairs assess the consensus, and forward the 
>document to the shepherding AD.  The shepherding AD takes one last 
>look, determines everything is in order and sends it to last 
>call.  A few people on the IETF Discussion list think that the 
>proposed specification is about to doom the Internet.  A few others 
>who have not even read the document agree based on emails.  Most of 
>the WG members are either not on the IETF list or choose to stay silent.
>
>The shepherding AD considers those comments, thinks that those 
>issues have been addressed already and puts the document on the IESG 
>agenda. All other ADs (except one) think that everything is fine and 
>vote No Objection.  One AD agrees with the few people on the IETF 
>Discussion list and decides to put a DISCUSS and proceeds to hack 
>the document.  In the current model, other than the very few 
>exceptions cited recently, the AD gets what he or she wants for the 
>most part.  It is plausible that AD may do this even if no one else 
>identified a problem.

Actually, this sounds very similar to the case where an override vote 
was almost used.  Scheduling the override vote was sufficient for the 
DISCUSS-holding AD to ask for a strawpoll, and based on those 
results, the DISCUSS-holding AD cleared the DISCUSS position.

Russ

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 28 Jun 2008 10:08:33 +1200

Lakshminath,

On 2008-06-28 02:09, Lakshminath Dondeti wrote:
<snip>
> more DISCUSS ADs to hack up the document and publish it.

I completely agree. We can get into trouble if there's disagreement
whether the DISCUSS issue is only "important editorial" or substantive,
because if it's editorial, the WG really doesn't need to be involved.

Of course, sending a disputed issue back to the WG is 100% guaranteed
to cause delay, and sometimes a long delay if the dispute is profound.
That's what a quantitative analysis can't see, and neither can
a qualitative analysis if it only looks at the finished RFC.

    Brian

---

From: Jari Arkko <[email-masked]>
Date: Mon, 30 Jun 2008 02:52:11 +0300

Laksminath,

> My point was this: if a WG actually missed anything substantial and 
> that comes out during an IETF last call, and the shepherding AD 
> agrees, the document gets sent back to the WG.  If the shepherding AD 
> also misses or misjudges, any member of the IESG can send it back to 
> the WG for resolution.  What I think is not acceptable is for the 
> author and one or more DISCUSS ADs to hack up the document and publish it.

I think it would be useful to separate this discussion into different parts:

- what issues can be raised as a Discuss
- balancing the IETF opinion vs. WG opinion when there is a conflict
- how text proposals and draft revisions get created
- the role of shepherds, authors, and sponsors during this process
- how WGs become involved in changes from IETF Last Call and IESG reviews

The Discuss criteria document says quite a bit of the first topic. 
Earlier in this thread we talked about the second topic.

But I wanted to say a few words about changing text to resolve an issue. 
I said earlier that I would rather not be sending text proposals. I 
didn't want to imply that text coming from the IESG would be 
inappropriate. And certainly text coming from the author would not be 
inappropriate either. In fact, it is quite typical that the Discussing 
AD, possible directorate experts (if involved), and the author are the 
most likely suspects to come up with a way to resolve an issue. And most 
motived to get to a resolution. For instance, some of my documents have 
recently had issues with PMTU, and I asked the Discussing transport ADs 
to work with the authors to design a solution; this worked well.  In 
this sense "hacking the document" is not just OK but it may actually be 
the right thing.

The problems with the Discussing AD proposing text are more in the area 
of scalability. I prefer seeing the authors (or shepherds) be active and 
propose ways to resolve an issue. Or at least the initial proposal, 
review and suggestions from both sides may be needed to converge.

The big problem with any of the key players (author, Discussing AD, 
sponsoring AD) making changes relates mostly to the fifth item on my 
list. We are not very good at keeping the WGs in the loop. This is often 
done right, but far from always. I know I have problems in this area. 
One of the consequences is less control on the WG's side on 
controversial topics. Another one is reduced review of new text; errors 
might creep in. While the hacking part may have been OK, the publication 
is the problem.

I don't want to make excuses, but it may be helpful to understand some 
of the reasons behind this:

- Some issues are too small or obvious to warrant engaging the WG; 
getting the document approved on the given telechat day is seen as a 
higher priority. A fairly large number of Discusses get resolved on the 
day of the telechat, having been filed just days or hours before.

- The author - AD team works at a much faster pace in many cases than, 
say, the shepherd or the entire WG.

- Discussing AD is not on the WG list, does not know status of the 
document in other respects than his or hers own Discuss.

- Things falling in the cracks, e.g., Discussing AD thinks that the 
shepherd or the sponsoring AD is responsible for talking to the WG.

- No guidelines or processes relating to how the WG is actually involved 
in the discussion. In some cases we inform the WG what the Discusses are 
and what is being done about them. In other cases we actually run 
something resembling a WGLC or a consensus call. In yet other cases the 
new draft version announcement goes to the list as the sole 
announcement, and people are expected to look at the tracker for the 
Discusses.

- Desire to avoid lengthy discussions.

- Sponsoring AD trusts that the Discussing AD and author have made the 
right choices.

- WGs that for some reason have stopped caring about anything else than 
getting the document published. Not care about the particular hoop that 
they have to jump through to resolve a Discuss. (And by the same token, 
not care about Comment level review issues at all).

Some of these issues could be improved with a clearer definition of 
roles, and some additional guidelines on how to involve the WG.

Jari