Subject: [newtrk] one size fits all vs two-step

From: Pekka Savola <[email-masked]>
Date: Mon, 1 Aug 2005 09:13:02 +0300 (EEST)

I read both draft-loughney-newtrk-one-size-fits-all-00.txt and
draft-carpenter-newtrk-twostep-00.txt.

I like twostep much more, because it still includes a notion that obtaining
interoperability information is important, and if a spec has
knobs which aren't used or or implemented, at least the fact should be
documented if those can't be reasonably removed.

In section 6, both state that downrefing an internet-draft would not be
allowed; I assume this also applies to Informational/Experimental
RFCs.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 1 Aug 2005 09:44:43 +0200

> documented if those can't be reasonably removed.

This would not be unreasonable, if we made more effort to obtain and 
document interoperability information.

My problem with the current process is that it is engineered for the 
case which is not the common case - count PS, count DS and count IS, 
for a proof of this.

We could change the current process in more than one way - I've 
proposed three versions so far, since 2003. Meanwhile, 2026 still says 
what it says.

I would really like to change it to what we are actually doing, 
whatever that is. If we change it to what we WILL be doing in the 
future, that could be lovely, too.

Spencer

.

---

From: Pekka Savola <[email-masked]>
Date: Mon, 1 Aug 2005 19:20:41 +0300 (EEST)

On Mon, 1 Aug 2005, Spencer Dawkins wrote:
> My problem with the current process is that it is engineered for the case 
> which is not the common case - count PS, count DS and count IS, for a proof 
> of this.

There may be a number of reasons for this, for example,

  * the current requirement of no stds-track downref is making it very 
difficult to progress documents becuase doing so is dependent on other 
specs also being progressed.

  * the current requirement of removing features which haven't been 
implemented (or interoperable) by at least two folks may require 
respinning the document, requiring more work.

Of course, one could rebut both of these and provide other 
considerations besides -- but the current process isn't exactly the 
same as that proposed in two-step.

---

From: Bruce Lilly <[email-masked]>
Date: Mon, 1 Aug 2005 12:42:35 -0400

> case which is not the common case - count PS, count DS and count IS, 

That's no proof; both RFCs 2821 and 2822 are at PS.  I believe it's
accurate to state that interoperability information is reasonably
well-known for SMTP and for the Internet Message Format.  Those two
RFCs -- and possibly others -- are the result of consolidating numerous
amendments and extensions into single documents, which caused a reset
to Proposed (from full Standards RFC 821 (STD 10) and 822 (STD 11).
Conversely, RFCs 2476 and 2234 have had successor Internet-Drafts
approved for elevation to DS, but the interoperability information
is either inadequate per 2026's "functionally equivalent or interchangeable
components of the system" criterion (2476 successor) or a sad joke
(2234 successor, which lists in the implementation report xx00 and xx99
RFCs as use of ABNF, none of which (obviously -- as in "sticks out like
a sore thumb") actually use ABNF, and neither of the two listed
implementations of ABNF parsers actually adhere to the full ABNF
specification).

Because of the former type of resets, PS status doesn't always indicate
lack of interoperability information (indeed, for both SMTP and the IMF,
which had achieved full standard status, the technologies have "significant
implementation and operational experience).  Because of the apparent
failure of the IESG to carefully review implementation reports (see the
disclaimer on the implementation report page, and consider the two
examples mentioned above), elevation to DS does *NOT* necessarily
indicate that interoperability (as defined in BCP 9) has been documented.
Taken together, the implication is that counting PS, DS, and full
Standard RFCs doesn't indicate anything about interoperability in
practice.  Unfortunately.

> whatever that is.

Personally, I'd rather see a change in actual process to match what the
community (via BCP 9) feels is necessary.  I'm not even sure it's
possible to characterize what the IESG is actually doing, other than
"inconsistency" (e.g. as opposed to the resets of RFCs 2821/2822 mentioned
above, consider the significant changes made between RFCs 1738, 2396,
and 3986 (a.k.a. STD 66), which have (obviously) not caused a reset,
instead having not deterred repeated elevation on the standards track.
Go figure.

Because the established process has not been adhered to, it is simply
not possible to infer reliable interoperability information from the
status of a standards track RFC -- that however is not (entirely [1]) a
problem with the process as designed; it is itself an implementation
issue.  People familiar with a specific technology may know what
interoperates well and what does not, but it is simply not possible in
general to reliably discern that information either from RFC status or
from implementation "reports" (some of which read more like marketing
blurbs than technical reports).

If we cannot find a way to get the IESG to follow the process, then we
might as well abandon having a standards track at all (because under the
status quo, status on the Standards Track is practically meaningless due
to failure to follow the process consistently), and just have Informational,
Experimental, Historic, and BCP, moving all of the existing Standards Track
RFCs to Informational (after the de-cruft process shifts some instead to
Historic).

What it boils down to is that if the IETF wants to truly be an engineering
SDO, then it needs to apply engineering rigor to standardization policies
in practice, not merely in on-paper process specifications which are
ignored.  If it's unable or unwilling to apply such rigor, then it ought
to abandon any pretense of being an SDO.   Part of the problem with the
failure to apply rigor may well be organizational; e.g. a quick comparison
of the standards-development structure of say the IEEE reveals a much
more intensive review effort at lower levels than is provided for in the
IETF.  IEEE work happens at the committee level, analogous to IETF WGs.
Standards documents generated in committee (and adhering to strict rules
checked by administrative staff and reviewing body) are reviewed by a
much larger group than the IESG (via a formal balloting process) and
upon passing that review are then reviewed by a formal Standards Board.
That process (http://standards.ieee.org/announcements/bkgnd_stdsprocess.html)
has:
1. more formal structure than IESG "Areas" (with membership)
2. a much larger base of (qualified volunteer) reviewers within each
   technical area than one or two ADs plus their designees, each of
   whom is involved in a formal balloting process
   a) IEEE ballots typically go to dozens or more technical experts in
      a field (but outside of the working group that developed the
      specification) for review, *before* going to a larger group of
      generalists (Standards Board)
   b) IESG review typically happens immediately after IETF Last Call (no
      formal ballots, no specific set of reviewers) and is intertwined
      with review of Last Call comments (as opposed to being contingent
      upon successful passing of a formal ballot) 
3. An additional level of review (committee, sponsoring society, and
   Standards Board vs. WG and IESG (with IETF Last Call input))
4. well-documented formal sets of rules, procedures, and guides
   (http://standards.ieee.org/resources/development/index.html)
   a) made available on a professionally-run web site
   b) administered by professional staff

What characterizes the current IETF/IESG process (few surprises, since
these issues have been mentioned before) as very different from IEEE is:
1. a haphazard web site
2. a loose set of guidelines, checklists, and documents in various states
   of disarray regarding document content, quality standards, and procedures
3. no formal balloting process or format at IETF Last Call, and consequently
   no formal objective mechanism for gauging Last Call comments
4. professional standards-development staff available at early stages of
   specification preparation (to ensure quality and consistency of
   specifications before outside review; IETF secretariat checks only a few
   things, quality of documents subjected to IETF Last Call varies enormously)
5. lack of formal fee-based membership requirements for participation

With the possible exception of the last item, these issues should be
addressed:
1. the web site needs an overhaul, and should be professionally redesigned
2. 1id-checklist and 1id-guidelines should probably be formalized as
   documents in some series (either a separate series or as (probably BCP)
   RFCs with some normative status)
3. the draft successor to RFC 2223 should probably move forward, and should
   probably have some normative status (most likely BCP)
4. improved use of existing tools (e.g. use of idnits and other tools such
   as MIB and ABNF validators by the Secretariat to reject drafts which do
   not meet the guidelines and checklist requirements) and development of
   new tools (e.g. a formal ballot form for IETF Last Calls)
5. Early professional help to authors in writing drafts:
   a) perhaps a team of volunteer reviewers to check for poor wording,
      bad grammar and/or usage, confusing punctuation, etc.
   b) publication of the RFC-Editor's IETF training presentation as another
      guideline document
   c) collecting a list of useful documents for authors/editors into a
      single resource (maybe an ISD-like or web-based list including
      glossaries (e.g. FYIs 18 & 36), how-to guides (e.g. BCPs 26 & 72),
      protocol requirements (e.g. BCP 18), architectural principles
      (e.g. RFC 1958), as well as the checklists, guidelines and RFCs
      listed above, plus others

Unfortunately, many of those seem to be outside of the scope of what this
WG can do.  Perhaps formalizing Last Calls with a specific ballot might
be appropriate, though it probably hasn't even been discussed until now.
Collecting a list of useful documents about RFC and draft preparation and
procedures is probably outside of what falls under ISDs as they have been
envisioned (many of the documents are not RFCs, those that are tend not to
be Standards Track). 
 
> If we change it to what we WILL be doing in the  

My crystal ball is broken.

----------
1. Some provision for downgrading from full Standard to DS in cases
   like 821->2821 and 822->2822, rather than a reset to PS, might be
   desirable.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 03 Aug 2005 14:23:10 +0200

Bruce Lilly wrote:
...
> If we cannot find a way to get the IESG to follow the process,

Let's point that finger also at the WGs, that in general don't
come up with requests for PS->DS and very rarely do so for DS->S.
When they do so, the IESG processes them.

The automatic review after two years in 2026 was unrealistic from
day one.

> then we
> to failure to follow the process consistently)

Disagree. When the PS->DS->S transitions are implemented, they occur
according to 2026, which does *not* include certification, only the
posting of an interop report.

>, and just have Informational,

I think this a false conclusion. otoh the facts do argue for simplification,
hence the drafts that this thread is comparing.

     Brian

.

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 3 Aug 2005 09:27:27 -0400

> When they do so, the IESG processes them.

Sure, and there may be some scope for minor reform of 2026 to assist
(see separate message in response to Sam's "in support of Larry's
proposal" message) WGs to plan for advancement and in so doing to
simplify review.  On the other hand, if the IESG disbands WGs before
they *can* request PS-DS status change (minimum 6 months after PS),
it's inappropriate to blame the WGs.  And the IESG has been disbanding
WGs.
 
> The automatic review after two years in 2026 was unrealistic from

Another possible minor tweak to 2026; some SDOs have a five-year
review cycle.
  
> posting of an interop report.

I disagree with that, based on discussions w.r.t. 2026 definition of
interoperability ("functional equivalence [...]") and what happened with
the 2476 successor draft and its implementation report.  Likewise for
the 2234 implementation report listing xx00 and xx99 RFCs as incorporating
ABNF.  Of course the letter of 2026, while it requires documentation of
interoperability to be submitted to the cognizant AD, doesn't specifically
state that the AD or IESG actually has to do anything with that
documentation, it is certainly the spirit of 2026 that advancement to DS
should entail adequate review for conformance to the 2026 criteria as
defined.  The discrepancy between handling of the 821->2821 reset to
proposed vs. lack of resets in the 1738->2396->3986 transition is
another example of at minimum inconsistency in application of 2026
procedures.

> hence the drafts that this thread is comparing.

A one-step process essentially would remove the distinction between Standards
Track and Informational or BCP or Experimental.

The argument, as I understand it, is that "the IESG doesn't have the cycles
to do the job".  From an engineering point of view, there are a number of
solutions other than "just don't do the job":
o Apply more cycles to the job by adding processors (presumes that the task
  can be at least approximately partitioned)
o Upgrade the processors to more powerful (i.e. having more cycles) units
  (presumes that more powerful processors are available)
o Offload the job to a separate set of processors dedicated to the task
o Improve the algorithm used to perform the task (w/o changing the task
  itself)
o Apply prioritization to tasks to raise the priority of the job in
  question
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html