Subject: Proposed Standard and Perfection

From: [email-masked] (Sam Hartman)
Date: Wed, 3 Mar 2004 07:21:01 -0500 (EST)

Hi.  For the past few plenary meetings, people have gotten up to the
mic and complained that we've lost track of what proposed standard is
all about.  We take too long reviewing documents to make them perfect
and would be better off just throwing them out earlier without so much
review.

I disagree.

The point of proposed standard is not to throw a document out there
and get comments, although of course we're always willing to listen to
comments on our documents.

The point of proposed standard is to throw things out there and get
implementation experience.

If specs are unclear,  then we're not going to get implementation
experience; we are going to waste time.  

I've had a lot of experience with a rather unclear spec with some
significant problems that managed to make its way to proposed
standard: For the past 10 years I have been dealing with problems in
Kerberos (RFC 1510).  IThis leads me to believe very strongly that
catching problems before documents reach PS is worth a fairly high
price in time.

That said, I realize too much time can be spent on a review.  When
we're not sure we understand the implications of an issue well enough
to know whether it will a be a problem, letting a document go to PS
and getting implementation experience can be useful.

Similarly, if the review process will never successfully conclude,
then having the review early  is good.  

ALso, I am simply saying that waiting for complete reviews is good and
the pressure to get things out as PS faster with less review is
dangerous.  If there are ways to reduce time spent on the reviews
without reducing quality, I recognize the value of these efforts.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 4 Mar 2004 16:11:45 +0900

Sam,

SH> Hi.  For the past few plenary meetings, people have gotten up to the
...
SH> I disagree.
SH> The point of proposed standard is not to throw a document out there
SH> and get comments, although of course we're always willing to listen to
SH> comments on our documents.
SH> The point of proposed standard is to throw things out there and get
SH> implementation experience.

Your statements do not run contrary to the complaints that are being
lodged.

These statements do not propose that Proposed is for being to "get
comments".

SH> If specs are unclear,  then we're not going to get implementation
SH> experience; we are going to waste time.

Let's inspect RFC 2026's language:

   4.1.1  Proposed Standard
   ...
   A Proposed Standard specification is generally stable, has resolved
   known design choices, is believed to be well-understood, has received
   significant community review, and appears to enjoy enough community
   interest to be considered valuable.  However, further experience
   might result in a change or even retraction of the specification
   before it advances.

Let's take note of the non-social requirements:

     Stable.

     Resolved _known_ design choices.

This does not mean that the failure of the Security section to provide
a tutorial on security is required.

This does not mean that all of the uses and dynamics of the protocol
must already be understood.

It does not even mean that the writing in the specification must be
perfect.

And it most certainly does not mean that the design choices must
coincide with the design preferences of the cognizant AD.

Yet all of these have been reasons for delaying documents from
achieving Proposed.

SH> Similarly, if the review process will never successfully conclude,
SH> then having the review early  is good.  

Review early.  Review often.  Yes!

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

---

From: Eliot Lear <[email-masked]>
Date: Thu, 04 Mar 2004 10:59:26 -0800

Sam,

As the person who most recently complained, let me elaborate on my 
comments.  The problem I believe we all are facing is that the 
distinction between Proposed, Draft, and Internet Standard has been lost.

I agree with you 100% that...

> implementation experience.

But when it comes to...

> experience; we are going to waste time.

We disagree (slightly).  In my experience one needs to actually get the 
implementation experience to recognize when things are unclear.  And my 
understanding is that this is precisely why we have PS and DS.

> Kerberos (RFC 1510).  This leads me to believe very strongly that

We come to different conclusions here.  My conclusion is that no 
standard should remain at proposed for more than 2 years unless it's 
revised.  Either it goes up, it goes away, or it gets revised and goes 
around again.

Your fundamental problem with RFC 1510 is that it is too painful for 
people to go and fix the text.  And that's a problem that should be 
addressed as well.

Thus, let the IESG have a bias towards approval for PS, and let 
implementation experience guide them on DS and full standard.  But set a 
clock.

This has impact on the WG process of course.  People want to do their 
work and go home.  We like WGs to end.  Well, what really needs to 
happen is that either the WG hangs around to push the thing forward, or 
the doc needs to be assigned some sort of standing WG, akin to a an area 
directorate, who will take responsibility for moving it forward or 
killing it.

And moving it forward shouldn't be that hard EITHER.  Mostly in the 
editing of clarifications, removal of functions not found to be used, or 
perhaps changing a few SHOULDs to MUSTs and visa versa.

Let's take an example: COPS-PR; RFC 3084.  How many people actually 
implement it?  If we can't find anyone who is, won't it just cause 
confusion to leave it at PS?

And I'd like to know when someone plans to do the work to get Kerberos 
to DS.  Heck, at least it's used by people.  Consider HARPOON, RFC1496 
on the downgrading of X.400/88 -> X.400/84 with MIME.  Ya think Harald 
wants to take the time to update that one now?!  Well, why didn't it 
happen in some reasonable period of time, when perhaps it might have 
been more interesting?  Was it because nobody actually implemented it or 
was it simply because nobody felt the need to update it?

> and getting implementation experience can be useful.

Don't get me wrong.  Some review is definitely in order.   In as much as 
they are going to happen they should happen either prior to sending the 
doc to the IESG at all (remaining within the WG) or in parallel with 
IESG review.

> dangerous.

Only because today PS = Internet Standard, in reality.  And that's what 
needs to change.

Eliot

---

From: Sam Hartman <[email-masked]>
Date: Fri, 05 Mar 2004 03:48:58 -0500

>>>>> "Eliot" == Eliot Lear <[email-masked]> writes:

    Eliot> Sam, As the person who most recently complained, let me
    Eliot> elaborate on my comments.  The problem I believe we all are
    Eliot> facing is that the distinction between Proposed, Draft, and
    Eliot> Internet Standard has been lost.

    Eliot> I agree with you 100% that...

    >> The point of proposed standard is to throw things out there and
    >> get implementation experience.

    Eliot> But when it comes to...

    >> If specs are unclear, then we're not going to get
    >> implementation experience; we are going to waste time.

    Eliot> We disagree (slightly).  In my experience one needs to
    Eliot> actually get the implementation experience to recognize
    Eliot> when things are unclear.  And my understanding is that this
    Eliot> is precisely why we have PS and DS.

    >> I've had a lot of experience with a rather unclear spec with
    >> some significant problems that managed to make its way to
    >> proposed standard: For the past 10 years I have been dealing
    >> with problems in Kerberos (RFC 1510).  This leads me to believe
    >> very strongly that catching problems before documents reach PS
    >> is worth a fairly high price in time.

    Eliot> We come to different conclusions here.  My conclusion is
    Eliot> that no standard should remain at proposed for more than 2
    Eliot> years unless it's revised.  Either it goes up, it goes
    Eliot> away, or it gets revised and goes around again.

It's been under revision for all of that time.

    Eliot> Your fundamental problem with RFC 1510 is that it is too
    Eliot> painful for people to go and fix the text.  And that's a
    Eliot> problem that should be addressed as well.

nWell it certainly has been painful but because of a number of false
steps and to some extent because of WG management issues, it has taken
10 years, not 2 years to revise RFC 1510.

we might have gotten it down to 7 or 8 by fixing the WG management.

    Eliot> Thus, let the IESG have a bias towards approval for PS, and
    Eliot> let implementation experience guide them on DS and full
    Eliot> standard.  But set a clock.

It's in significant part because I think a lot of things may take more
than 2 years to revise understand and fix that I believe this approach
is wrong.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sat, 06 Mar 2004 23:41:40 -0800

it's me again.....

--On 4. mars 2004 10:59 -0800 Eliot Lear <[email-masked]> wrote:

> We come to different conclusions here.  My conclusion is that no standard
> should remain at proposed for more than 2 years unless it's revised.
> Either it goes up, it goes away, or it gets revised and goes around again.

I spent some time thinking about this comment from the plenary on my way 
home from Seoul. (An advantage of long flights....?)

I don't think obsoleting as a regular procedure is a bad idea. But it will 
take some work to get from here to there.

My musings came up with a guesstimate of 1/2 hour of work per document on 
the standards track that has been abandoned totally (due dilligence in 
figuring out that *nobody* is using it), and 4 hours of work per document 
on the standards track that is in use, but has no particularly interesting 
future and should be moved to informational - and a somewhat larger figure 
for the documents where there is in fact a reason to revive and revise them.

Some calculations using very round numbers.....

We have approximately 992 Proposed documents, of which I guesstimate that 
about 800 documents are ready for evaluation under the 2-year rule; if we 
assume a 12:3:1 distribution of the three categories above, we're looking 
at around 600x0.5 + 150x4 = 900 man-hours to get there, if we disregard the 
part about the standards that deserve updating, and disregard the documents 
that have gotten to Draft and the full standards that deserve honorable 
retirement.
Possibly quite a bit less once the team doing it gets into full swing, or 
if there are many standards for which it is easy to show that no 
usage/interest exists.

So if we can get 9 people to work at it, and want to be up-to-date in a 
year, we're looking at an investment of around 100 hours per volunteer - or 
about 2 hours each week for a year.

In the steady state (30 docs/month, currently), perhaps 30 man-hours/month 
could be enough - lower, because since the docs are newer, people who are 
asked actually remember.....

That doesn't look too awful.... people who want to volunteer for this work 
can contact me, and we'll see if we can figure out a way to do it....

                  Harald

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sun, 7 Mar 2004 07:59:51 -0600

I spent more time trying to capture what people were saying at the
plenary than trying to figure out who said what, but I would like to
figure out who said

[06:43:24] <anewton> too much time needed to take something out there
and take it back to historic.
[06:43:44] <anewton> suggests steps for things to automatically go
historic.
[06:43:48] <anewton> harald.
[06:43:55] --- AWGY has joined
[06:44:20] <anewton> perhaps have someone else beside IESG do leg
work.
[06:44:36] <anewton> ??.

on Thursday night - sound familiar to anyone? The last name mentioned
in the logs was John Loughney, then Harald replied, and then SOMEONE
said "too much time needed..." I'd love to find who who said this.

Thanks!

Spencer

---

From: Dave Crocker <[email-masked]>
Date: Sun, 7 Mar 2004 17:07:43 -0800

Harald,

HTA> In the steady state (30 docs/month, currently), perhaps 30 man-hours/month 

30 documents go to Proposed each month?

The steady-state rate of review is the average number of documents that
go to Proposed.  (well, ok, the average of the number that went to
proposed 2 years ago.

In any event, we can distinguish documents that newly come up to their
2-year limit, versus dusting out the closet of those that already hit 2
years, before this.

Keeping up with the new documents reaching their limit is the
critical-path activity. Dusting out the closet can take as long as we
want.

d/

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 9 Mar 2004 19:54:40 -0800

Hi -

> From: "Spencer Dawkins" <[email-masked]>
> To: <[email-masked]>
> Sent: Sunday, March 07, 2004 5:59 AM
> Subject: Re: Work effort? (Re: Proposed Standard and Perfection)
>

...

I made the comment that I thought we should apply RFC 2026 and force
things to either advance or go historic.  Our AD advised us in one case
that if our WG wanted one of its RFCs to go historic, we had to write
another RFC explaining why.  The procedure in RFC 2026 section 6.2
(last paragraph) seems very reasonable, and I like Harald's suggested
approach to cleaning up the cruft.

Randy

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 10 Mar 2004 08:07:46 -0800

> approach to cleaning up the cruft.

backpedalling somewhat.... I didn't intend to indicate that I had a 
"suggested approach" yet - there are lots of details that should be worked 
out before we start, such as:

- who should do it?
- what criteria should they use?
- which way should they be biased when in doubt?
- how much workload is created by doing this?
- who does something about the docs that do NOT go to Historic?
- how do we document the evaluations
- how do we challenge them?

The 2026 section 6.2 procedure, with no adjustments, says "IESG", "judgment 
about likelyhood of progressing on track", "downgrade", "don't care about 
workload", "WG if existing, otherwise don't care", "IETF-announce list 
message", "appeals procedure".

I don't think those are necessarily the answers that can be made to work.

that's just a start.... some thinking about these things can probably go a 
long way towards reducing the number of un-intended side effects we 
cause....

                   Harald