Subject: Re: Review of: Characterization of Proposed Standards

From: Olaf Kolkman <[email-masked]>
Date: Fri, 1 Nov 2013 09:12:02 +0100

On 1 nov. 2013, at 01:00, SM <[email-masked]> wrote:

> 
>  "Hence IETF Proposed Standards are of such quality that they
>   are ready for the usual market-based product development and
>   deployment efforts into the Internet."
> 
> I am not comfortable with having that text in BCP 9.  The argument up to now has been running code and that that is the line which has been used for test of quality.  A better test of quality might be someone who has not followed the working group and who can implement the specification.  There is also the IPR test.  That is also one of the issues mentioned by the audience which the document targets.

Note that the text says that they are ready for market based product development and does not proscribe a test, in fact there is also no IPR test for proposed standards. The ‘IPR test’ what happens after market deployment: did market players have no problem getting (F)RAND licenses, hence it is an important, albeit somewhat implicit, criterium in going from Proposed to Internet Standard.

Frankly, if we develop standard track document that, in general, are not ready to be put in products hen it is going to be very very difficult to explain the relevance of the IETF. Not only to the policy people but also to the people who pay us to develop standards.

And off-course it is not black and white, hence the ‘in general’ in the sentence above. In the cases where we collectively think that we are not quite there there is the escape route that we’ve put in section 4.

—Olaf

---

From: Olaf Kolkman <[email-masked]>
Date: Fri, 1 Nov 2013 09:33:14 +0100

> 
> I think that's not really addressing the core of Dave's concern, which
> isn't that the repetition is unnecessary (which would be a matter of
> taste), but that it's actually harmful, in that future changes could
> leave us with two divergent definitions of "Internet Standard".
> 
> On this, I'll note that we're already somewhat along that path, as
> 6410 quoted text from 2026 in its definition of "Internet Standard".

This IMHO is an argument to consolidate. If I don’t get it right (with some high layer IETF experience) then how would an external consumer. Let’s put a consolidated  2026/6410 IETF Standards characterization in this document and create clarity.

(I would be helped with being pointed out those small things)

I want one document to point at when people ask me ‘what is the difference between proposed and Internet standards’.  For the people that ask for that difference it’s the characterization that matters. For those that want to know what the processes are… well, they can dive deeper. 

The target audience of this document is not only the IETF is what I am saying.

> Perhaps you have a suggestion for different wording, Dave, that will
> address your concern while still addressing mine?

I’d be happy to incorporate (also see my note to SM that contains an argument: we want to be able to generically say that Proposed standards are good enough for market, except when they are not).

—Olaf

---

From: SM <[email-masked]>
Date: Fri, 01 Nov 2013 15:26:58 -0700

Hi Olaf,
At 11:06 31-10-2013, Dave Crocker wrote:
>      2.  I also wonder whether we shouldn't circulate the draft 
> more broadly, outside of the IETF, to request the comments.  Will 
> it accomplish what we want it to accomplish?

I'll comment on the above point as I think that it is a good idea.

The problem (as discussed initially) was that it was difficult for 
the target audience to understand what is a Proposed Standard because 
of the way it was characterized in RFC 2026.  The fix was to have a 
draft to clarify that.

For that fix to accomplish what we [1] want it to accomplish, it is 
good to get feedback from the target audience.  That target audience 
is outside the IETF.  In my opinion it would be help if the draft was 
circulated [2] more broadly, i.e. outside the IETF.  It entails 
reaching out to the people and asking them for feedback.

Regards,
-sm

1. I am using the word "we" as it was easier.  It is not the royal "we". :-)

2. I understand that it may not be easy to do that.

---

From: Larry Masinter <[email-masked]>
Date: Fri, 1 Nov 2013 15:53:01 -0700

> Hence IETF Proposed Standards are of such quality that they
>   deployment efforts into the Internet.

There are too many IETF Proposed Standards for which
this assertion is questionable. Perhaps if we took the time
we could triage, recategorize them as Historic or Experimental.

There is some metadata the IETF wishes to publish about
the IETF's considered opinion as to the quality of a specification
and whether it is ready for market-based product and deployment
efforts. And *that* evaluation depends on operational considerations
such as whether widespread deployment would be disruptive
or benign, and a variety of other factors.

We only have a few bits of information going along with a 
published specification (its status as Standard or Proposed ).
If you have more to say than those few bits, we need to say
it in text. In the document itself or somewhere else.

Larry
--
http://larry.masinter.net

---

From: Barry Leiba <[email-masked]>
Date: Sat, 2 Nov 2013 05:11:09 -0400

On the Sec 3.2 point, I think Olaf is getting somewhere in his reply: make
this the definitive source, which should eliminate the issue of duplication
and also merge in the 6410 stuff.  I'll work up text on the plane (en route
to EWR now.

Barry

On Friday, November 1, 2013, Jari Arkko wrote:

> I largely agree with what Barry is saying. Additional review is always
> helpful, but I'm pessimistic about getting relevant additional review. And
> I'm mostly thinking of the kind of people where this work started from;
> folk who would have a problem using RFCs as reference standards in, say,
> government procurement, but would happily go on using their computers with
> all kinds of Internet technology in them after speaking :-) I think the
> crux of the matter is to get our own house in order and be consistent
> between reality and our description of the IETF process. And this is what
> we're doing.
>
> FWIW, my personal opinions on content:
>
> I am happy with the 3.2 as it stands, but also happy with Barry's
> suggested "by-reference" formulation.
>
> I agree with Barry's description of why Section 4 is necessary.
>
> Jari
>
>

---

From: Dave Crocker <[email-masked]>
Date: Sat, 02 Nov 2013 06:29:31 -0700

On 10/31/2013 1:53 PM, Barry Leiba wrote:
> 1. A general point: the document doesn't seem to have had the review
> and scrutiny that one would expect for a significant process change.
>
> 2. Simple duplication of text from 2026 (in Section 3.2) is not a
> good idea, as it invites future divergence.
>
> 3. Section 4 should be removed, as it weakens the statements made
> elsewhere in the document and invites criticism, and is in effect
> self evident anyway.
>
> To point 1... I believe that the light volume of comments is because (1) it
> *is* reflecting reality, and that's mostly not controversial and (2)
> a lot of this was discussed when RFC 6410 was being developed, and

That's a perfectly reasonable theory.  It might even correct.

But it's merely a guess and there are other possibilities, some of which 
could be problematic.

We should not have to guess.

Again, we are changing possibly THE core document for the IETF and we 
should not be making guesses about either what the IETF thinks of it nor 
what the community causing the change thinks about it.

We need to obtain explicit comment and establish an concrete record.

> these changes are really artifacts of that discussion (which did not
> lack in participation).  Further, the document has been out here for
> discussion; the opportunity has been here, and IETF folks aren't
> normally shy if they have things to say.

Actually, shy isn't the problem.  Complacent and inattentive is the 
problem.  We do that really well, also.

Again, this document is too important for the community to treat 
casually by staying silent about a change to core normative text.

>> 2.  I also wonder whether we shouldn't circulate the draft more
>> broadly, outside of the IETF, to request the comments.  Will it
accomplish
>> what we want it to accomplish?
>
> We could post it to the new-work mailing list, or explicitly send it
> out through our liaisons.  It's likely, I think, that it would get
> little attention that way, though we won't know unless we do it.  On
> the other hand, we don't normally send our process documents out
> that way -- we didn't do that with the draft that became 6410, for
> example. I have no objection to soliciting external reviews and
> comments, but I'm doubtful that it'll be useful.

Using existing means of public circulation is fine, but it's rather 
passive.  Here, too, my point is that we need to press for explicit 
feedback.

Again:  This round of modification is triggered by reports from Olaf of 
folk who are discounting the Proposed Standard label.  It makes little 
sense to change a foundational document for these folk without getting 
them to review it and agree that it assuages their concerns.

If the text does not satisfy these folk, are then going to modify the 
document again, and again do it by guessing what will work for them?

> a. 6410 tweaked the definition of Internet Standard, in order to
> merge in aspects of Draft Standard.  That this document quotes 2026
> without also bringing in the changes from 6410 somewhat misstates
> things (in a small way, but nonetheless...).
...
> I suggest that that's a better approach, and would suggest that
> Section 3.2 would be better done this way:

Well, re-using text from another update of RFC 2026 is appealing.  On 
reflection, I think we got this issue wrong for RFC 6410, for the reason 
I've stated about the current work.  That said, the cloning approach you 
are suggesting makes sense.

> On point 3, I'll note that Section 4 is in this document exactly in
> response to my comments.  I was concerned that as this document
> tightens the formal criteria for PS, it's likely to cause even more
> strictness in IESG approval.  We've always had the option (and have
> not infrequently exercised it) to approve a document at PS that
> *does* have known limitations.
...
> I believe that Section 4 is saying more than the things that apply
> to all technical specifications: it's not just saying that they're
> not perfect and that we might find problems later.  It's meant to say
> that we might, sometimes, approve specifications at Proposed Standard
> that we *know* to have imperfections and limitations, and we're
> approving them anyway.  And what it's adding to what 2026 says about
> that is that when we do that, we'll explicitly say so in the
> document.
>

Interesting.  I didn't read the text and get the meaning Barry is 
offering.  Barry's interpretation highlights an issue that makes sense 
to me and his language about the issue is simple and direct.  That makes 
it harder for the reader (including me) to miss the point.

So first let's drop discussion about what we do /not/ publish as 
Proposed.  It's distracting.

Second, let's have affirmative language about special cases that we /do/ 
publish as Proposed.  Again, Barry's language looks pretty reasonable 
for this, I think.

Perhaps:

    4.  Further Considerations

    Occasionally the IETF may choose to publish as Proposed Standard a
    document that contains areas of known limitations or challenges.  In
    such cases any known issues with the document will be clearly and
    prominently communicated in the document, for example in the
    abstract, the introduction, or a separate section or statement.

On 11/1/2013 1:33 AM, Olaf Kolkman wrote:
> This IMHO is an argument to consolidate. If I don’t get it right
> (with some high layer IETF experience) then how would an external
> consumer. Let’s put a consolidated  2026/6410 IETF Standards
> characterization in this document and create clarity.
>
> I want one document to point at when people ask me ‘what is the
> difference between proposed and Internet standards’.

I'm not sure what 'consolidated' means, but it sounds as if Olaf has
just argued for issuing an updated RFC 2026, folding in the previous and 
current changes.

That's probably the cleanest solution, of course.  The obvious downside 
is the risk of reviews that call for other changes in the document.  But 
given the low level of community interest in this foundational document, 
that's probably not a major risk...

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Olaf Kolkman <[email-masked]>
Date: Sat, 2 Nov 2013 15:45:59 +0100

On 2 nov. 2013, at 14:29, Dave Crocker <[email-masked]> wrote:

> 
> Again:  This round of modification is triggered by reports from Olaf of folk who are discounting the Proposed Standard label.  It makes little sense to change a foundational document for these folk without getting them to review it and agree that it assuages their concerns.

'Triggered by’ is the right characterization, not requested, wanted, or mandated by. As such those folk are not the direct consumers. 

That said and FWIW: I have socialized the draft with some of the folk I am working with that I consider consumers. The feedback that one gets is not at the level we are having the discussion now. 

>   abstract, the introduction, or a separate section or statement.

WFM.

—Olaf

---

From: Jari Arkko <[email-masked]>
Date: Sat, 2 Nov 2013 03:08:31 +0000

I largely agree with what Barry is saying. Additional review is always helpful, but I'm pessimistic about getting relevant additional review. And I'm mostly thinking of the kind of people where this work started from; folk who would have a problem using RFCs as reference standards in, say, government procurement, but would happily go on using their computers with all kinds of Internet technology in them after speaking :-) I think the crux of the matter is to get our own house in order and be consistent between reality and our description of the IETF process. And this is what we're doing.

I am happy with the 3.2 as it stands, but also happy with Barry's suggested "by-reference" formulation.

Jari

---

From: Jari Arkko <[email-masked]>
Date: Sat, 2 Nov 2013 14:45:22 +0000

>   abstract, the introduction, or a separate section or statement.

Seems reasonable to me.

jari

---

From: Barry Leiba <[email-masked]>
Date: Sat, 2 Nov 2013 18:18:19 -0400

> and also merge in the 6410 stuff.  I'll work up text on the plane (en route

OK, so...

Here's what I suggest, which has us explicitly replacing all of the
definitions of Proposed Standard and Internet Standard from both 2026
and 6410.  This leaves us with the new document being the definitive
place, and there's no duplication.

Abstract

OLD
   RFC 2026 describes the review performed by the IESG on IETF Proposed
   Standard RFCs and characterizes the maturity level of those
   documents.  This document updates RFC 2026 by providing a current and
   more accurate characterization of Proposed Standards.
NEW
   RFC 2026 describes the review performed by the IESG on IETF Standards
   Track RFCs and characterizes the maturity level of those documents.
   RFC 6410 updates those characterizations.  This document updates RFCs
   2026 and 6410 by providing a current and more accurate characterization
   of Proposed Standard, and merging the changes made to Internet Standard
   by RFC 6410.
END

1.  Introduction

OLD
   This document only updates the characterization of Proposed Standards
   from RFC2026 Section 4.1.1 and does not speak to or alter the
   procedures for the maintenance of Standards Track documents from RFC
   2026 and RFC 6410 [RFC6410].  For complete understanding of the
   requirements for standardization those documents should be read in
   conjunction with this document.
NEW
   This document only updates the characterization of Proposed Standards
   from RFC2026 Section 4.1.1 and merges the changes made to RFC 2026
   Section 4.1.3 by RFC 6410.  While this document now provides a complete
   current characterization of the Standards Track maturity levels, for a
   complete understanding of the requirements for standardization it is
   necessary to read RFCs 2026 and 6410 in conjunction with this document.
END

3.  Characterization of Specifications

OLD
  The text in the following section replaces RFC 2026 Section 4.1.1.
  Section 3.2 is a verbatim copy of the characterization of Internet
  Standards from RFC 2026 Section 4.1.3 and is provided for convenient
  reference.
NEW
  Internet specifications go through stages of development, testing,
  and acceptance.  Within the Internet Standards Process, there are
  two stages, formally labeled "maturity levels" in the "Standards
  Track".  These maturity levels are "Proposed Standard" and
  "Internet Standard".

  This section describes the maturity levels and the expected
  characteristics of specifications at each level.  It replaces
  Section 4.1 of RFC 2026, and its subsections, and Sections 2,
  2.1, and 2.2 of RFC 6410.

  A specification may be, and indeed, is likely to be, revised as it
  advances from Proposed Standard to Internet Standard.  When a revised
  specification is proposed for advancement to Internet Standard, the
  IESG shall determine the scope and significance of the changes to the
  specification, and, if necessary and appropriate, modify the
  recommended action.  Minor revisions and the removal of unused
  features are expected, but a significant revision may require that
  the specification accumulate more experience at Proposed Standard
  before progressing.
END

OLD
3.2.  Characteristics of Internet Standards

  A specification for which significant implementation and successful
  operational experience has been obtained may be elevated to the
  Internet Standard level.  An Internet Standard (which may simply be
  referred to as a Standard) is characterized by a high degree of
  technical maturity and by a generally held belief that the specified
  protocol or service provides significant benefit to the Internet
  community.
NEW
3.2.  Characterization of IETF Internet Standard Specifications

  A specification for which significant implementation and successful
  operational experience has been obtained may be elevated to the
  Internet Standard level.  An Internet Standard (which may simply be
  referred to as a Standard) is characterized by a high degree of
  technical maturity and by a generally held belief that the specified
  protocol or service provides significant benefit to the Internet
  community.

  The IESG, in an IETF-wide Last Call of at least four weeks, confirms
  that a document advances from Proposed Standard to Internet Standard.
  The request for reclassification is sent to the IESG along with an
  explanation of how the criteria have been met.  The criteria are:

  (1) There are at least two independent interoperating implementations
      with widespread deployment and successful operational experience.

  (2) There are no errata against the specification that would cause a
      new implementation to fail to interoperate with deployed ones.

  (3) There are no unused features in the specification that greatly
      increase implementation complexity.

  (4) If the technology required to implement the specification
      requires patented or otherwise controlled technology, then the
      set of implementations must demonstrate at least two independent,
      separate and successful uses of the licensing process.

  After review and consideration of significant errata, the IESG will
  perform an IETF-wide Last Call of at least four weeks on the
  requested reclassification.  If there is consensus for
  reclassification, the RFC will be reclassified without publication of
  a new RFC.

  In a timely fashion after the expiration of the Last Call period, the
  IESG shall make its final determination and notify the IETF of its
  decision via electronic mail to the IETF Announce mailing list.  See
  Section 6.1.2 of RFC 2026.

  A specification that reaches the status of Standard is assigned a
  number in the STD series while retaining its RFC number.
END

--
Barry

---

From: Barry Leiba <[email-masked]>
Date: Sat, 2 Nov 2013 18:19:11 -0400

>>   abstract, the introduction, or a separate section or statement.

I agree; that's a fine way to put it, and thanks.

Barry

---

From: Barry Leiba <[email-masked]>
Date: Sat, 2 Nov 2013 18:20:05 -0400

Oh, and, of course, if we use my suggestion below, this document will
have to "update" 6410 as well as 2026.

Barry

---

From: Dave Crocker <[email-masked]>
Date: Sat, 02 Nov 2013 15:25:07 -0700

> have to "update" 6410 as well as 2026.

It will "obsolete" both 6410 and 2026.  It's replacing them.

d/

---

From: Barry Leiba <[email-masked]>
Date: Sat, 2 Nov 2013 18:28:15 -0400

> It will "obsolete" both 6410 and 2026.  It's replacing them.

No, I don't think so.  It's replacing specific sections of them, but
not the whole documents.  I think we'd have a very hard time getting
consensus on a full replacement of 2026.

Barry

---

From: Dave Crocker <[email-masked]>
Date: Sat, 02 Nov 2013 15:44:29 -0700

> consensus on a full replacement of 2026.

sorry,  I clearly misunderstood what was being proposed.I'd like to 
blame it on Amtrak's impressive jossling of me today, but I suspect the 
flaws is genetic.

As nearly as I can tell, now, what's meant is to replace 6410 with the 
new document.  Please, please, please don't tell me that /portions/ of 
6410 are being replaced.

In any event, I've missed the logic for "updating", "replacing" or 
whatever 6410, if all of this isn't just being merged into 2026bis.

d/

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 03 Nov 2013 12:16:04 +1300

> number in the STD series while retaining its RFC number.

Unfortunately this is the entrance to a very enticing rat hole,
and a couple of small subsidiary rat holes as well.

First a small one: we should document the fact that an STD
is often a document set rather than a single RFC. Therefore,
a newly promoted Standard *either* gets a new STD number *or*
is associated with an existing STD number.

Second, the bigger one: a very confused situation is created
if part of a multi-RFC STD set is updated and thereby reverts
to Proposed Standard status. The only rational solution to
that is for the Proposed Standard document also to be associated
with the STD number. In practice, users don't care: they need to
know all the updates to a Standard, even if some of them are
only Proposed Standard.

Third, it needs to be specified that when multiple documents
form part of the same STD set, that is a decision made by the IETF,
not the RFC Editor.

(Barry - this is the downside of doing more than tinkering
with RFC 2026; you tend to drag in all its defects. In fact
a solution for this was spelled out in some detail in 2008
but didn't fly at that time:
http://tools.ietf.org/html/draft-carpenter-rfc2026-changes-02#section-3.2 )

Regards
   Brian

---

From: Barry Leiba <[email-masked]>
Date: Sat, 2 Nov 2013 19:31:02 -0400

> As nearly as I can tell, now, what's meant is to replace 6410 with the new
> document.  Please, please, please don't tell me that /portions/ of 6410 are
> being replaced.
>
> In any event, I've missed the logic for "updating", "replacing" or whatever
> 6410, if all of this isn't just being merged into 2026bis.

Sorry, yes: portions of 6410.

The point is that Section 4.1 of 2026 has become fragmented, with some
changes to it in 6410 and other changes to it here.  I'm proposing
addressing that fragmentation by making *this* document the definitive
place to get the information that used to be in 2026 Section 4.1.

Barry

---

From: Olaf Kolkman <[email-masked]>
Date: Sat, 2 Nov 2013 16:43:39 -0700

Barry,

[skipping to the core of the suggested changes]

>>  number in the STD series while retaining its RFC number.

I observe this is mixing characterization with process. I have been very careful to only touch the characterization in this document.

When I wrote the text in the draft I started from 6410 which says:

   The characterization of an Internet Standard remains as described in
   RFC 2026 [1], which says:

      An Internet Standard is characterized by a high degree of
      technical maturity and by a generally held belief that the
      specified protocol or service provides significant benefit to the
      Internet community.

That is the second sentence in the first paragraph from section 4.1.3 that talks about Internet Standards. The first sentence is an introduction sentence. Below is the full section from 2026 for reference:

4.1.3  Internet Standard

When I constructed the text in the draft I figured that the second paragraph was more of a process detail that could be left out. While that first sentence seems to be relevant for providing the context.

So really the question is whether the characterization is really out of sync with 2026/6410?

Personally I feel your suggestions are rescoping the drafts purpose.

—Olaf

---

From: Barry Leiba <[email-masked]>
Date: Sat, 2 Nov 2013 20:05:30 -0400

On Sat, Nov 2, 2013 at 7:43 PM, Olaf Kolkman <[email-masked]> wrote:
>
> I observe this is mixing characterization with process. I have been very
> careful to only touch the characterization in this document.
>
> That is the second sentence in the first paragraph from section 4.1.3 that
> talks about Internet Standards. The first sentence is an introduction
> sentence. Below is the full section from 2026 for reference:
>
> 4.1.3  Internet Standard
>
> When I constructed the text in the draft I figured that the second paragraph
> was more of a process detail that could be left out. While that first
> sentence seems to be relevant for providing the context.
>

OK, well...

Dave's point was that it's a bad idea to duplicate the text, and I
agree with that.

6410 changed 2026 Section 4.1.3 and nullified Section 4.1.2.  It made
no change to 4.1.1, and it said so (without duplicating text from it).

This document is changing Section 4.1.1.

This now means that in order to make sense of the Standards Track
maturity levels, which had previously been documented in RFC 2026
Section 4.1 and its subsections, one must now do this:

- Read 2026 and see that it's been updated.
- Replace Section 4.1.1 from this document.
- Get Section 4.1.3 by reading 6401 and 2026, and merging them.

If we want to leave it that way, we can... but if we do that, I agree
with Dave that we should NOT duplicate some of the text from 2026
Section 4.1.3, leaving out the changes that 6401 made.

Alternatively, my formulation (which isn't considering
characterization and process... it's consolidating the updates to a
particular section of 2026) means that one can do this:

- Read 2026 and see that it's been updated.
- Replace Sections 4.1.1 and 4.1.3 from this document.

That seems simpler.

I'm happy either way, though:
1. Using a change such as I've suggested.
2. Noting that this document does not change 2026 Section 4.1.3,
*without* copying text from it (much as 6410 does for Section 4.1.1).

Barry

---

From: t.p. <[email-masked]>
Date: Sun, 3 Nov 2013 11:38:06 +0000

Or perhaps the curse of Ericsson.

The e-mail to which I am replying causes my e-mail client to download
SEVEN Megabytes of CRL which, even over broadband, takes ninety five
seconds (I suspect that the PDU is minuscule and the round trip long).

Is it worth it just to know that the message is authenticated?  Not for
me:-(

Tom Petch

----- Original Message -----
From: "Jari Arkko" <[email-masked]>
To: "Barry Leiba" <[email-masked]>
Cc: <[email-masked]>;
"Dave Crocker" <[email-masked]>; "IETF Discussion" <[email-masked]>
Sent: Saturday, November 02, 2013 3:08 AM
Subject: Re: Review of: Characterization of Proposed Standards