Subject: Re: [newtrk] question two for newtrk

From: Alex Rousskov <[email-masked]>
Date: Wed, 30 Jun 2004 21:56:41 -0600 (MDT)

On Wed, 30 Jun 2004, Spencer Dawkins wrote:

> Seeing that we've had, functionally, one IESG-approved stage for
> almost all specifications we've published, I'd like to hear why
> people think we need two.
>
> I'd also like to hear why people think we'll start using more than
> one stage for most specifications, for the first time in history.
> Our history is actually a decent predictor of our future in a lot of
> cases...

And it would be ideal if arguments for multiple stages are given from
specs users point of view rather than just internal IETF processes
point of view. In other words, how would the second stage make the
life of a mainstream specs implementor, implementation tester, or
integrator different?

> Someone (Scott Brim?) said they liked rewarding specifications that
> were good enough to be stable ("these are the specs that have been
> stable for several years, and these are the specs that change every
> 9-18 months"). I understand this motivation; I'm just not sure we
> need to change the level label to accomplish it.

Moreover, the notion of "stability" is often user-dependent. 3 months
of no changes may be good enough for an ASP, while 9 months of no
changes might not be enough for those who have to implement the specs
in silicon. And 18 months may be too late for either of them to start
deploying because of market pressures!

IESG simply cannot make a reliable one-size-fits-all "this is stable"
declaration until it is too late for that declaration to matter. The
best we can do is to reject proposed standard publications with known
bugs, carefully document change history, and expect the users to make
the "stable enough" decision that is right for them.

Alex.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Melinda Shore <[email-masked]>
Date: Thu, 1 Jul 2004 11:33:51 -0400

On Wednesday, June 30, 2004, at 09:03 PM, Spencer Dawkins wrote:
> almost all specifications we've published, I'd like to hear why people
> think we need two.
>
> I'd also like to hear why people think we'll start using more than one
> stage for most specifications, for the first time in history. Our
> history is actually a decent predictor of our future in a lot of
> cases...

I don't think one is an awful answer, but I do think that having two
stages lends clarity about maturity that's otherwise missing.  Just
because the trade press can't tell the difference between an individual
internet draft and an internet standard doesn't mean that the people
who need to make decisions about which technologies to adopt can't,
either.

I'm also not sure that talking about our history is meaningful unless
we understand why most protocols never progress past proposed.  I 
suspect
that among the reasons is that the process is actually working.  We
produce an awful lot of protocols that are never deployed or that never
catch on, and there's no reason to progress those protocols to internet
standards.

Melinda

.

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 1 Jul 2004 10:10:05 -0600 (MDT)

On Thu, 1 Jul 2004, Melinda Shore wrote:

> stages lends clarity about maturity that's otherwise missing.

What would change, for a typical standards user, when IETF promotes
the standard to the second stage?

> Just because the trade press can't tell the difference between an
> individual internet draft and an internet standard doesn't mean that
> the people who need to make decisions about which technologies to
> adopt can't, either.

You may be right, but "people who need to make decisions" usually do
not care about the difference in labels if that's the only difference.
They may be able to tell the difference in labels, but we need to
offer something more than a new label for that difference to matter to
"people who need to make decisions".

That is why I am asking folks who argue for multi-stage track to tell
what the difference between stages would actually mean for a typical
standards user. "This is now a mature protocol" is too vague to mean
anything for a given user. Please be specific. Once we know what the
label actually means, we can see if it is possible and useful to
assign such a label.

A related and equally important question is whether the
"mature/stable" state is reached at about the same time for all major
user groups. I currently do not think so. Each standard user group has
their own notion of "stable/good enough to do X" because each group
does different kind of X. Thus, we would be assigning the label too
early or too late for some groups.

For example, today, we assign that "stable" label too late for most
user groups and so it means nothing for them.

N.B. I hope we can assume for this discussion that protocol fixes and
implementation records are available for any RFC on the new standards
track. Each potential RFC user can use that info to decide whether an
RFC is already "good enough" for them.

Thanks,

Alex.
.

---

From: Melinda Shore <[email-masked]>
Date: Thu, 1 Jul 2004 13:32:02 -0400

> the standard to the second stage?

I don't know who's "typical," but in my company the questions people ask
about IETF protocols usually center on whether or not anybody else is
using it and how stable it is.  The first question concerns not only
interoperability but also deployment experience - how the thing works
in practice, whether or not there are lurking gotchas, etc.  Advancing
to a second stage helps answer those questions, or should.  (A third
stage muddies the waters, in my experience.)

Melinda

.

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 1 Jul 2004 11:49:23 -0600 (MDT)

On Thu, 1 Jul 2004, Melinda Shore wrote:

> I don't know who's "typical,"

There are several classes of RFC users, of course. That's the problem!

> but in my company the questions people ask about IETF protocols
> usually center on whether or not anybody else is using it and how
> stable it is.  The first question concerns not only interoperability
> but also deployment experience - how the thing works in practice,
> whether or not there are lurking gotchas, etc.

The questions in most companies are indeed the same. It's not the
questions that usually differ, it's the answers (because the questions
are asked in a company-specific context that matters a lot)!  Yet,
with a multi-stage track, we would be implying that one answer works
for all classes of RFC users.

All of the above questions can be answered by each RFC user if
implementation, deployment records and errata are maintained for an
RFC. (and they have to be maintained for any number of stages)

> Advancing to a second stage helps answer those questions, or should.

Are you assuming that all companies will answer a

	"Is this RFC good enough for us to do X?"

question in about the same way at a given time? That seems highly
unlikely to me. In my experience, "stabile enough" or "deployed
enough" have very different meaning depending on what it is you are
doing with the RFC (e.g., prototyping, implementing in silicon, or
writing a test suite) and who you are (e.g., a startup, a small
player, a major vendor).

Even within one company like Cisco, different BUs have different
notions of "stability" and "deployment". Do you think a "this is
stable" label assigned without regard to their different needs is
worth the trouble? Would not we end up with most RFC users ignoring
the label again, because it does not reflect _their_ notion of
"stable" and "deployed"?

Thanks,

Alex.
.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 02 Jul 2004 11:50:50 +0200

(Co-author of the proposals draft hat is OFF)

I agree with Spenser. Since we seem to be agreed that on any
model, specs will be re-reviewed when they get updated, I see
no advantage in multiple labels for revs of the same thing.
It's bureaucracy is all.

    Brian

Spencer Dawkins wrote:
>>question two for newtrk is "how many IESG-approved stages?"
> 
> 
> Ignoring the part where I'm a co-author on a draft that says "two",
> I'm thinking "one" would be a pretty good answer, and it's probably
> what we'll end up with for most specs in practice anyway.
> 
> 9-18 months"). I understand this motivation; I'm just not sure we need
> to change the level label to accomplish it.
> 
> Dave Crocker's suggestion was that Full Standard is ONLY a mindshare
> consensus ("people think the spec is OK and are deploying it"). I'm
> not sure that would qualify as "IESG approval" in this thread.
> 
> Spencer
> 
> 
> .

.

---

From: Margaret Wasserman <[email-masked]>
Date: Fri, 2 Jul 2004 11:40:51 -0400

Another personal opinion...

>question two for newtrk is "how many IESG-approved stages?"

Stages in the standards track can be viewed as statements that we 
make about a document.  I think that there are two levels of 
statement that it makes sense to make:

(1) This document is reasonably complete, clear enough to implement, 
and we are not aware (using reasonable diligence) that deployment of 
this standard would cause damage to the Internet.  Some portion of 
the  IETF thinks that this standard is potentially useful, which is 
why we bothered to write it, but YMMV.  This document will almost 
definitely change (and may be abandoned) as we gain implementation 
and deployment experience (Stage 1).

(2) This standard is widely implemented, has proven useful in 
practice and is now widely deployed.  No non-backwards compatible 
changes are likely. (Stage 2).

So, I'd "vote" for two stages.

IMO, both of these stages would benefit from cross-area review and 
IESG approval.  Ideally, I'd also like both stages to be published as 
RFCs.

But, I see three problems that we'll need to overcome to make that happen:

- The IESG will need to adjust our approach and systems to allow for 
the approval of "good-enough" Stage 1 standards.  Some of our 
comments/discusses, for example, could transition to "Good-enough for 
Stage 1, but keep this in mind for when you want to move to Stage 2". 
If this change results in more documents being sent to the IESG, then 
we will need to find more scalable ways to do our work.

- The RFC publication process (post-IESG approval) currently takes an 
average of several months.  This will need to be corrected in order 
for RFC publication to be an effective publication path for Stage 1 
standards.

- We need to find a way to avoid the requirements creep that 
currently makes it difficult for existing standards to move up on the 
standards track.  Changing non-technical requirements (and defacto 
requirements) like new boilerplate, formatting requirements, RFC 2119 
language requirements, changing reference requirements, etc. are 
impediments to re-publishing widely implemented and deployed 
standards at a higher level on the standards track.  This would need 
to be addressed (by grandfathering?) for us to achieve an effective 
multi-level standards process.  We also need to figure out when (and 
when not) to require older standards to meet new technical 
requirements (new security algorithms, etc.).

Personally, I think that the bar that the IESG is setting for PS 
publication is currently too high, and that RF publication (our WG 
process, our IESG approval process, and our post-approval publication 
process) takes _far_ too long.  IMO, these are the factors that are 
driving the "need" for WGS publication.  But, I think that the better 
approach would be to fix these problems, so that we aren't 
encouraging the Internet community (directly, or indirectly through 
the non-responsiveness of our processes) to implement standards that 
haven't undergone appropriate cross-area review.

Other standards bodies get documents approved much faster than we do, 
and their documents are published almost immediately after approval. 
So, IMO, it _should_ be possible for us to publish documents at the 
first stage of the standards track quickly and predictably enough to 
obviate the need for a WG-approved WGS stage.  For that to happen, 
though, the whole community will have to make significant changes to 
our attitudes and expectations regarding the standards track process.

Margaret

.

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 2 Jul 2004 15:19:25 -0600 (MDT)

On Fri, 2 Jul 2004, Margaret Wasserman wrote:

> I think that there are two levels of statement that it makes sense
> to make:
>
> this standard would cause damage to the Internet.

I agree that the above makes sense, especially when compared to an ID
pre-stage where no IETF certification of completeness or correctness
is made.

> changes are likely. (Stage 2).

I agree that such a statement may make sense and can be made into a
stage. Since adding a stage requires some effort (compared to the
single-stage approach), what would be the benefit? It may make sense,
but is it essential?

I do not see much _value_ in the second stage because implementation
and deployment records should be available for each RFC in the first
stage. Those records will contain enough info to make statement (2),
for a given RFC user definition of "widely ...".

What are you motivations to have the second stage?

Alex.
.

---

From: John C Klensin <[email-masked]>
Date: Sat, 03 Jul 2004 04:28:29 -0400

--On Friday, 02 July, 2004 11:40 -0400 Margaret Wasserman
<[email-masked]> wrote:

>...
> Other standards bodies get documents approved much faster than
> we do, and their documents are published almost immediately
> after approval. So, IMO, it _should_ be possible for us to
> publish documents at the first stage of the standards track
> quickly and predictably enough to obviate the need for a
> WG-approved WGS stage.  For that to happen, though, the whole
> community will have to make significant changes to our
> attitudes and expectations regarding the standards track
> process.
>...

Margaret,

Without commenting at this point on the rest of your note, I
want to add some calibration on the point above.   Most of those
"other standards bodies" require almost-final editing of a
document prior to whatever the final vote is in their process.
ISO traditionally requires the equivalent of page proofs (the
analogy in our system of 48-hour RFC Editor Last Call) before
something goes out for international voting (which, in our
process, would correspond to IETF Last Call and IESG action).
Of course, if the documents go into the voting process at that
degree of quality and are approved, it doesn't take long to get
them out afterwards.  And, if they are not approved in the form
submitted, they are bounced back to the WG for modification and
re-editing, and then subjected to a new round of voting (or a
ballot/comment resolution process) before being finally approved
so, again, the editor-publisher has very little to do
post-approval.  It is also often the case that considerable
professional staff editing is significantly involved in document
preparation well before that document for final voting stage.

We, by contrast, often send things to the RFC Editor as approved
by the IESG that are badly formatted and just barely literate.
Sometimes the RFC Editor is completely successful at fixing
them, sometimes not, but the process would be time-consuming
under the very best of circumstances.

There are arguments for doing it our way; there are arguments
for doing it their way.  But let's not confuse differences in,
e.g., 

	* the point in the process at which the time is spent 
	
	* the mechanism and sequencing (what steps are taken, at
	what stage in processing, and who takes them) for
	identifying and correcting editorial problems (before or
	after final vote)

for overall differences in efficiency.

You will recall that there is at least one other standards body
that has been making a lot of statements in the last few years
about how efficient they have become in terms of getting
documents reviewed and out.   They actually have accomplished
quite a lot relative to their procedures of a decade ago and are
entitled to credit for those accomplishments.  At the same time,
a careful review of their statements and of the record indicates
that what is really being said is "when a document comes to us
complete, well-defined, with broad consensus, and without
controversy, it takes us little time to approve it and, if it is
complete enough and already ratified elsewhere, even less time
to rubber-stamp it.  And, if it doesn't meet those criteria, we
don't count it in reporting how fast we can be".  Well, yes,
but, given differences in norms, etc., comparing that to IETF
speed and process is a bit irrelevant.

regards,
     john

.

---

From: Margaret Wasserman <[email-masked]>
Date: Sun, 4 Jul 2004 08:51:19 -0400

Hi John,

>for overall differences in efficiency.

This is a very good point.  However, I am not sure that our current 
practice is always consistent with your comments.

We have a long list of editorial/formatting requirements  (parts of 
the I-D checklist document) that WG chairs are required to check 
before submitting documents to the IESG, ADs sometimes send documents 
back for editorial/formatting reasons during AD review, and there are 
even rare occasions where an AD issues a discuss regarding the 
editorial quality of a document that is being reviewed by the IESG.

Personally, I don't believe that there is always a clear line between 
editorial changes and technical clarity/correctness, so I believe 
that it would be better to engage professional editors earlier in the 
process, and to have the IESG approve the final document.  However, I 
would be willing to live with a situation where the IESG approves 
protocols and technical concepts in a somewhat raw form, with the 
expectation that the document will be professionally edited 
afterwards.

But, I don't see how either of these choices is consistent with what 
we are doing today.  Today, we seem to have two editing passes, one 
that is performed (perhaps poorly in some cases) by the document 
editor (often prompted by the WG, WG Chairs and ADs) before IESG 
approval, and a second that is performed by the RFC editor before RFC 
publication.  Do you think that this is the most efficient or 
effective alternative?

Margaret

.

---

From: John C Klensin <[email-masked]>
Date: Sun, 04 Jul 2004 09:33:00 -0400

--On Sunday, 04 July, 2004 08:51 -0400 Margaret Wasserman
<[email-masked]> wrote:

>> We, by contrast, often send things to the RFC Editor as
>> approved by the IESG that are badly formatted and just barely
>> literate. Sometimes the RFC Editor is completely successful
>> at fixing them, sometimes not, but the process would be
>> time-consuming under the very best of circumstances.
>> 
> This is a very good point.  However, I am not sure that our
> current practice is always consistent with your comments.

I was really trying to talk more about our intent.  Our
practice, in many cases, seems to be to send materials of poor
editorial and comprensibility-quality and to hope that they can
straighten that out.   That situation results in a lot of time
being spent post-IESG-approval (there are other factors in that
time as well) and may or may not result in adequate improvement. 

> We have a long list of editorial/formatting requirements
> (parts of the I-D checklist document) that WG chairs are
> required to check before submitting documents to the IESG, ADs
> sometimes send documents back for editorial/formatting reasons
> during AD review, and there are even rare occasions where an
> AD issues a discuss regarding the editorial quality of a
> document that is being reviewed by the IESG.

Having observed what comes out of the final editing process on
may occasions, I suggest that, if those steps are expected to
get things mostly right from technical, editorial, or
explanation quality standpoints, they are failing.   See below.

> Personally, I don't believe that there is always a clear line
> between editorial changes and technical clarity/correctness,
> so I believe that it would be better to engage professional
> editors earlier in the process, and to have the IESG approve
> the final document.  However, I would be willing to live with
> a situation where the IESG approves protocols and technical
> concepts in a somewhat raw form, with the expectation that the
> document will be professionally edited afterwards.

Well, personally, we agree.  And, because of the "unclear line"
you identify, I don't think anything else is really possible in
the general case.  It is a testimonial to the quality of the RFC
Editor process that we haven't had many document screwed up
technically in the process of trying to fix them editorially.
The institution of the "48 hour last call" has, I assumed,
caught many problems that might have slipped through earlier but
is clearly not adequate to assure that the _document_ has
community consensus.  I'd personally be much happier about our
process if they document that was Last Called was expected to be
the document that would be published, plus or minus boilerplate
changes and very minor editorial work.

But the other issues this, and some of the other discussions,
raise is that IESG behaviors --as distinct from whatever the
rules are supposed to be-- are key to this.   The decision about
how "finished" something needs to be before it is last-called is
up to AD discretion.  An AD can send a document back for
editorial work (including issues of clarity).  For a document in
a given quality range, sometimes it happens, sometimes it
doesn't, and I suggest that there is little consistency,
especially cross-area.

The community has a very poor track record with changing IESG
behavior: "you AD folks should stop doing this and do that
instead, despite the fact that both fall into the range you
consider to be part of you discretion" has a history comparable
to shouting into a very stiff wind (and ruder analogies).   My
perception is that usually nothing happens and sometimes the
person or group doing the shouting is punished for making noise.

And that is also why I've been very reluctant to spend much time
on this WG.  To the extent to which the proposals explicitly or
implicitly involve "IESG should handle documents differently",
e.g., adopt a different review standard for the first stage in
the process, there should be a "question zero" of "and how does
one expect to make that work, at least without replacing all of
the IESG at the same time and reinitializing the internal
culture"?

> But, I don't see how either of these choices is consistent
> with what we are doing today.  Today, we seem to have two
> editing passes, one that is performed (perhaps poorly in some
> cases) by the document editor (often prompted by the WG, WG
> Chairs and ADs) before IESG approval, and a second that is
> performed by the RFC editor before RFC publication.  Do you
> think that this is the most efficient or effective alternative?

Of course not.   But, unless we want quality to _really_
deteriorate, we had better avoid making comparisons to bodies
that have solid pre-approval editing models and then trying to
impose "go faster" on the RFC Editor on that basis.  As you may
be aware, I suggested at one stage that the RFC Editor do a
quality review pass prior to IETF Last Call and that _they_ have
the ability to prevent such a Last Call and bounce the document
if the quality didn't meet some standard (perhaps expressed as
the ability to get a document finished and out within some fixed
period).  Neither the IESG nor the RFC Editor liked that idea,
mostly for good reasons.  But asking that the RFC Editor go a
lot faster while holding the quality (or lack thereof) of what
it is getting constant would, IMO, fall into the "be careful
what you wish for" category and that is what caused my comment.

    john

.

---

From: Margaret Wasserman <[email-masked]>
Date: Sun, 4 Jul 2004 10:07:48 -0400

>changes and very minor editorial work.

I'm not quite sure why we're arguing, because I think that we
mostly agree about this...

>impose "go faster" on the RFC Editor on that basis.

My initial message didn't say anything about declaring that
the RFC editor should simply "go faster".   I don't think that
would work any better than telling the WGs or the IESG to
"go faster".

Instead, I think we need to examine the overall document
publication process and find ways to publish documents more
quickly.  One way to do that is to look for places where we are
duplicating effort or serializing steps that could/should be done
in parallel.

I don't think, for instance, that it makes sense to undergo a
lengthy, painful and ineffective process of having document
authors (who are not professional editors in most cases)
edit a document, only to throw it over the wall and have it
reformatted and re-edited by professionals.

But, I am _definitely_ not saying that we should skip the
professional editing step!

[Which brings us back to something pertinent to this WG...]

But, doesn't the WGS proposal involve publishing an archival
document without going through a professional editing step?  It also
skips the cross-area review and IESG approval steps.  Personally,
I  don't  think that any of these steps should be skipped.  Instead,
I think that we need to find a way to better integrate these steps
with the WG process, so that the final results are available
sooner.

This won't require less resources, especially in the area of
professional editing.  In fact it would probably require more.
But I believe that it should be  possible to parallelize some of
the many steps that we currently perform sequentially.

>mostly for good reasons.

I don't like this idea, either, because it adds another serial step.

>But asking that the RFC Editor go a

Agreed.

Margaret
.

---

From: "Spencer Dawkins" <[email-masked]>
Date: Fri, 9 Jul 2004 21:31:08 -0500

Sorry, I've been offline with a family emergency... just to follow up,

From: "Margaret Wasserman" <[email-masked]>
> document without going through a professional editing step?

Not in
http://www.ietf.org/internet-drafts/draft-dawkins-newtrk-wgs-00.txt...

     The document is subject to all normal I-D rules. When a WGS
     specification needs more time, active effort on it will
     usually result in changes to the specification being needed,
     with a resulting new version of the document, and (possibly)
     another working group decision to assign it WGS status.

     COMMENT:  This label augments the current situation with I-
               Ds. It imposes minimal delay and it permits
               participants, reviewers, vendors and others to
               be better . The label makes clear that the
               specification is a WG product, not an IETF
               product, and it does not imply long-term stability
               to the specification. The use of a non-archival
               version (I-D) should encourage folks to revise
               and progress the specification.

     COMMENT:  It is worth considering extending the I-D time-
               limit to 9 months. This will be more convenient
               for WGS documents -- for example, it provides two
               full IETF meeting cycles -- and will have no
               substantive effect on other I-Ds.

Of course, we haven't even tried to declare a WGS on what a WGS *is*,
so anyone can think it's anything, and anyone can submit a draft
saying it's anything...

Spencer

.