Subject: Re: [newtrk] My notes from Newtrk

From: Bruce Lilly <[email-masked]>
Date: Mon, 11 Apr 2005 09:29:13 -0400

On Tue March 29 2005 13:43, Brian E Carpenter wrote:
> Somewhat delayed reply...

So is mine; sorry.

> Bruce Lilly wrote:
> > On Tue March 15 2005 09:33, Brian E Carpenter wrote:
> > 
> > 
> >>These are good questions. For new or newish technologies, I understand
> >>that the idea is that writing & updating the ISD is simply part of the
> >>WG's job.
> > 
> > 
> > What about individual (i.e. not WG) submissions?
> 
> That's easy enough for new work I think - the same individual(s)
> get to do it, and it's quite straightforward for new work anyway.

Straightforward perhaps if it's just a matter of supplying some
suitable text; not so much if it means "learn XML including esoterica
from scratch".

> > What is the update mechanism?  Do you really want a half-dozen
> > WGs (chairs? Editors? Secretaries?) simultaneously editing a
> > master document?   Or do we need an XXXX Considerations section
> > in RFCs directing XXXX to update the document?
> 
> I think that the WG would have to appoint an ISD Editor. And
> for a non-WG or ex-WG topic, the relevant AD would have to appoint
> an ongoing ISD Editor. This is a problem we have anyway - how to
> arrange stewardship of specifications that need some maintenance
> but don't justify a WG. Whether it's tracking errata in RFCs
> or maintaining ISDs, we have this problem. And all of it is
> probably simpler than the profile maintenance that some other
> SDOs perform. (But of course, it isn't free.)

WG ISD Editor sounds like it warrants an update to BCP 25.

Agreed that there is a stewardship problem crying out for a solution
(though I think "become dormant" per BCP 25 sect. 4 might work).

IMO, the RFC Editor is doing a fine job at maintaining RFC errata.

There may still be a coordination issue if an ISD covers work
performed by multiple active WGs.

> >>Personal opinion: successful ISDs will show that the PS->DS->S progression
> >>is makework.
> > 
> > 
> > I couldn't disagree more strongly.  The progression through the
> > Standards Track levels is supposed to ensure that something that
> > reaches full Standard status really has had the bugs worked out.
> 
> Yes. Great theory. Lousy experimental results.

Lousy in what sense -- that there are buggy full standards?  That --
with nobody to do the work -- work needed for  progression doesn't get
done?
 
> > The failure in recent times of that process has been that things
> > advance to Draft without the requisite two *fully* interoperable
> > implementations, or to full Standard status after significant
> > substantive changes from Draft (e.g. RFC 2396 to 3986) without a
> > reset to Proposed.  I.e. the lack of quality of implementation
> > (of documents and the technologies they represent) and of the
> > review process are problems.  Possibly *because* progression is
> > wrongly viewed as makework.
> 
> It's certainly viewed as a lot of work with little reward.

The benefits accrue to the community, not specifically to those who do
the work.

> My 
> conclusion is that the users of our standards don't care,
> otherwise there would be real external pressure to progress
> from PS, and there isn't.

I would hesitate to jump to a conclusion; there are many reasons why a
particular PS doesn't advance -- at least one is the stewardship issue.
For that matter, I'm not convinced that there is a lack of pressure for
progression as much as a lack of a place to apply the pressure.
Personally, I'd like to see RFCs 2821 and 2822 (now 4 years at PS)
advance, but neither has a WG to do the work or to complain to.

> I'd rather put the limited effort 
> available on something that really does matter - interoperability
> statements. (And to be clear, although I think it is common
> ground in newtrk already, *revising* specs when they prove not
> to be interoperable is highly desirable - it's just the
> progression through exactly three levels that I find pointless.)

Interoperability statements would require substantial effort, probably
comparable to documenting the two fully interoperable implementations
(supposedly) required for advancement to Draft status.  However, the
provision of BCP 9 section 4.1.2, second paragraph is important.
Removing unused provisions enhances security, reliability, etc.
(components which aren't present can't be exploited and can't fail);
merely saying "feature X isn't widely used" or "feature Y has only
limited interoperability" doesn't have that effect.

Maybe three steps isn't the right number, but I believe that (if the
rules are enforced) the requirements associated with advancement to
Draft status provide substantial community benefit.

Returning to the stewardship issue, however, the requirement is that
the WG Chair is tasked with producing the required documentation for
advancement to Draft.  With the prevalent practice of disbanding WGs
once a PS is produced, that leads predictably to the observed
situation; many standards track specifications fail to advance.

> [If you want to share this on the list, feel free.]

Done.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 12 Apr 2005 13:43:59 +0200

(front-commenting this so that people don't have to go through
it all again)

>> with nobody to do the work -- work needed for  progression doesn't get

The second sense, as observed in RFC 3774.

>> Draft status provide substantial community benefit.

Fully agree about those benefits - the trick is to achieve them without
the process itself being a burden.

     Brian

.

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 12 Apr 2005 16:41:41 -0400

On Tue April 12 2005 07:43, Brian E Carpenter wrote:

> >> That --

I would have called that "expected results".

RFC 3774 claims that there is a perception that the quality bar for
IETF standards has been raised.  I find that incredible; here's a
quick, back-of-the-envelope, unscientific check:

The RFC Editor maintains errata.  Currently, the errata page is
roughly 260 kB of HTML.  The 50% point is at RFC 2731, issued
December 1999.  If the quality of recent IETF documents were higher
than in the past, one would expect the 50% point to be *before*
1/2 of the most recent RFC number (especially if allowing for the
fact that it takes time to find, report, verify, and publish errata,
which tends to cause errors in recent RFCs to be unpublished).  We
haven't reached RFC 5462 yet.  RFC 2731 was published just over 5 years
ago, compared to the 37-year RFC history.

So either by number of documents or time interval, there are apparently
more errors recently than in the past, and that seems to indicate a
lowering of the quality bar.

Weakening procedures designed to ensure quality (multiple stages of
review, specific requirements for advancement, etc.) would seem to be
counterproductive if the objective of high-quality standards is to be
maintained.

Below is a specific example (with all due respect to the RFC authors,
the IESG, and the RFC Editor) that shows that the quality bar certainly
hasn't been raised, and hasn't merely been lowered; it appears to have
been dropped:

The recently-issued RFC 4021 is categorized as Standards Track, but
itself specifies no protocol or format and has no requirements (indeed
*ALL* of the RFC 2119 keywords mentioned in RFC 4021 section 1.2 appear
*ONLY* in that section!).  It is unclear how such a document, which is
expected to have no "implementations" as such and contains no
requirement specifications, could advance to Draft (given the RFC 2026
requirement for such advancement for two independent and fully
interoperable *implementations*).  I suspect it should be reclassified
as Informational.  As a general procedural matter, perhaps the IESG
needs to have a checklist for RFCs, including (for those considered for
the Standards Track) such issues as "[how] can this document advance to
the next level?", "if the document cites RFC 2119, does it actually use
RFC 2119 language? if so, does it do so appropriately?".

If a document can't pass a simple mechanical check such as
egrep "MUST|REQUIRED|SHALL|SHOULD|RECOMMENDED|MAY|OPTIONAL"
showing either nothing (non-Standards Track status only!) or something
more than boilerplate merely copied from RFC 2119, one wonders how it
could pass IESG muster.

Maybe the burden needs to be raised and shifted back to authors, not
merely for the sake of doing so, but to make sure that authors are
generating documents of adequate quality to warrant consideration and
review which require the limited resources available.  E.g. perhaps
authors should be required to provide supplemental documentation when
review is requested, including such things as a list of each section
which makes use of an RFC 2119  keyword, a list of specific, objective,
testable criteria for determining conformance, etc. (such documentation
could be saved for the 24-month review of Proposed Standards).

Maybe the IESG needs a set of tools to perform some suitable mechanical
checks to weed out documents with "obvious" problems before expending
significant human effort on review.  Perhaps a suitable WG could be
tasked with that job.  As such tools might also be used to identify
documents at Proposed which have no hope of advancing (e.g. 4021),
those tools might be useful within NEWTRK to identify cruft.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: "C. M. Heard" <[email-masked]>
Date: Thu, 14 Apr 2005 09:04:19 -0700 (PDT)

Brief comments on two unrelated points:

On Tue, 12 Apr 2005, Bruce Lilly wrote:
> December 1999.  If the quality of recent IETF documents were
> higher than in the past, one would expect the 50% point to be
> *before* 1/2 of the most recent RFC number (especially if
> allowing for the fact that it takes time to find, report,
> verify, and publish errata, which tends to cause errors in
> recent RFCs to be unpublished).  We haven't reached RFC 5462
> yet.  RFC 2731 was published just over 5 years ago, compared to
> the 37-year RFC history.

One point that you have not considered is that the RFC Editor did
not maintain errata throughout the entire 37-year RFC history. That
is a relatively recent development.

> The recently-issued RFC 4021 is categorized as Standards Track,
> but itself specifies no protocol or format and has no
> requirements (indeed *ALL* of the RFC 2119 keywords mentioned in
> RFC 4021 section 1.2 appear *ONLY* in that section!).  It is
> unclear how such a document, which is expected to have no
> "implementations" as such and contains no requirement
> specifications, could advance to Draft (given the RFC 2026
> interoperable *implementations*).  I suspect it should be
> reclassified as Informational.  As a general procedural matter,
> perhaps the IESG needs to have a checklist for RFCs, including
> (for those considered for the Standards Track) such issues as
> "[how] can this document advance to the next level?", "if the
> document cites RFC 2119, does it actually use RFC 2119 language?
> if so, does it do so appropriately?".

I emphatically agree with you on this point.  This document and
others like it should be classified as BCPs, not as standards-track.
RFC 3737 and RFC 3575 are to other excellent examples of document
that should never have been categorized as BCPs rather than as
standards-track documents.  (I raised this issue in the IETF last
call for 3575 but was away on travel when 3737 was last-called)

I would, however, not call this a document quality control problem
so much as a process quality control problem.  It would be good to
speak up (in IETF last call, or sooner if possible) when such
mis-classification problems arise in the future.

Mike Heard

.