Subject: RE: [newtrk] the problem

From: "Robert Snively" <[email-masked]>
Date: Mon, 6 Mar 2006 10:18:10 -0800

I would like to propose formally a one-step approach.

Using a one-step approach, one would create a series of 
Internet-Draft-Proposals that would be integrated into
a Standard.  During that integration period,
running code, examination, and other verification techniques would
be used to move toward correctness, completeness, 
winnowing unnecessary features, and clarity.  That work
would be considered "complete" when the Standard was approved
by a process still to be defined (probably involving 
relatively formal balloting and review procedures),
not when an RFC passed IETF last call or when
running code was (supposedly) interoperably demonstrated.

Future experience with a Standard might indicate
the requirement to change the Standard.  That would be 
a legitimate revision, performed as a separate project
illuminated by all the experience with the original
Standard.

It would argue for lengthening the life time or the recreation
of working groups as necessary for such revisions.  Things
like IPSec would essentially have a continuous working
group, much like today.

So that is simultaneously one step (Standard is the only
product), two steps (Standards are reviewed and
corrected before they become a Standard), and
three steps (revisions are made as reality forces
modifications or corrections in the technology).

This also reduces the cost of development by focusing the
work on  particular Standards that will only get additional
attention if they need it.

It also allows for sufficient specialization and
breadth of commercial interest in the standards
development process to allow more delegation of review,
at least partially off-loading some of our overloaded
technical reviewers.

For what it is worth, similar structures appear to be
fairly successful when properly managed in other standards 
organizations, ranging through IEC, ISO, and ANSI and
their derivative organizations.  At some point, we will
have to face up to the fact that the some of the IETF
legendary approach will have to become more formalized
to serve as the universal basis for communication over
the next century. 

And to agree with Eliot, these processes would have to
be documented, managed, and followed.

Bob Snively
408-835-4321

-----Original Message-----
From:  Eliot Lear
Sent: Monday, March 06, 2006 9:22 AM
Subject: Re: [newtrk] the problem

Ran,

You've suggested that I accept two step as perhaps an intermediate
step.  My support comes at what I hope is a   reasonable price: show me
an economic motivation that actually causes companies to want to spend
their employees' time to get specs to the 2nd step.  Put another way,
let's only have the second step if we can reasonably predict that people
will use it.

I think I wasn't clear in my earlier message.  It's not that customers
don't know the difference between an internet-draft and a standards
track document (some do - not all), but they clearly don't care about
the difference between a draft standard and a proposed standard - an RFC
# is sufficient most of the time.  And we know this to be generally true
because we'd at least see more attempts to get from proposed onward.

Ultimately the state I'd like to see us end up in is one where we're
using the practices as documented, as intended.  It seems we all agree
that three step isn't working.  Now we just have to sort between one and
two and how to encode them.  And I think you'd agree that we really
don't want to have to tinker again for a while, so let's try and get it
right.

Eliot

.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: <[email-masked]>
Date: Mon, 6 Mar 2006 22:11:45 +0200

Robert,

I have just resubmitted the "A Single-Stage Standards Process" draft
that Spencer and
I wrote. I'm sure it is stuck in the queue, so you can review it here:

http://www.geocities.com/j_loughney/ietf/draft-loughney-newtrk-one-size-
fits-all-01.txt 

I basically agree with your points below, fwiw.  However, I still have
the feeling
that multi-step process is more asethically pleasing to IETFers,
irrespective of it's
market impact.  Feel free to suggest changes or to add text where
appropriate.

thanks,
John

>From: [email-masked] 
>[mailto:[email-masked]] On Behalf Of ext Robert Snively
>Sent: 06 March, 2006 20:18
>To: Eliot Lear; RJ Atkinson
>Cc: [email-masked]
>Subject: RE: [newtrk] the problem
>
>Internet-Draft-Proposals that would be integrated into a 
>Standard.  During that integration period, running code, 
>examination, and other verification techniques would be used 
>to move toward correctness, completeness, winnowing 
>unnecessary features, and clarity.  That work would be 
>considered "complete" when the Standard was approved by a 
>process still to be defined (probably involving relatively 
>formal balloting and review procedures), not when an RFC 
>passed IETF last call or when running code was (supposedly) 
>interoperably demonstrated.
>
>Future experience with a Standard might indicate the 
>requirement to change the Standard.  That would be a 
>legitimate revision, performed as a separate project 
>illuminated by all the experience with the original Standard.
>
>like IPSec would essentially have a continuous working group, 
>much like today.
>
>product), two steps (Standards are reviewed and corrected 
>before they become a Standard), and three steps (revisions are 
>made as reality forces modifications or corrections in the technology).
>
>This also reduces the cost of development by focusing the work 
>on  particular Standards that will only get additional 
>It also allows for sufficient specialization and breadth of 
>commercial interest in the standards development process to 
>allow more delegation of review, at least partially 
>off-loading some of our overloaded technical reviewers.
>
>For what it is worth, similar structures appear to be fairly 
>successful when properly managed in other standards 
>organizations, ranging through IEC, ISO, and ANSI and their 
>derivative organizations.  At some point, we will have to face 
>up to the fact that the some of the IETF legendary approach 
>will have to become more formalized to serve as the universal 
>basis for communication over the next century. 
>
>And to agree with Eliot, these processes would have to be 
>documented, managed, and followed.
>
>Bob Snively
>408-835-4321
>
>
>an economic motivation that actually causes companies to want 
>to spend their employees' time to get specs to the 2nd step.  
>Put another way, let's only have the second step if we can 
>reasonably predict that people will use it.
>
>I think I wasn't clear in my earlier message.  It's not that 
>customers don't know the difference between an internet-draft 
>and a standards track document (some do - not all), but they 
>clearly don't care about the difference between a draft 
>standard and a proposed standard - an RFC # is sufficient most 
>of the time.  And we know this to be generally true because 
>we'd at least see more attempts to get from proposed onward.
>
>Ultimately the state I'd like to see us end up in is one where 
>we're using the practices as documented, as intended.  It 
>seems we all agree that three step isn't working.  Now we just 
>have to sort between one and two and how to encode them.  And 
>I think you'd agree that we really don't want to have to 
>tinker again for a while, so let's try and get it right.
>
>Eliot
>
>
>.

.

---

From: <[email-masked]>
Date: Mon, 6 Mar 2006 23:06:53 +0200

Actually, it is already out:
http://www.ietf.org/internet-drafts/draft-loughney-newtrk-one-size-fits-
all-01.txt 

John
>[mailto:[email-masked]] 
>Sent: 06 March, 2006 22:12
>To: [email-masked]
>I have just resubmitted the "A Single-Stage Standards Process" 
>draft that Spencer and I wrote. I'm sure it is stuck in the 
>queue, so you can review it here:
>
>I basically agree with your points below, fwiw.  However, I 
>still have the feeling that multi-step process is more 
>asethically pleasing to IETFers, irrespective of it's market 
>impact.  Feel free to suggest changes or to add text where appropriate.
>
>thanks,
>John
>
>>Internet-Draft-Proposals that would be integrated into a Standard.  
>>During that integration period, running code, examination, and other 
>>verification techniques would be used to move toward correctness, 
>>completeness, winnowing unnecessary features, and clarity.  That work 
>>would be considered "complete" when the Standard was approved by a 
>>process still to be defined (probably involving relatively formal 
>>balloting and review procedures), not when an RFC passed IETF 
>last call 
>>or when running code was (supposedly) interoperably demonstrated.
>>
>>Future experience with a Standard might indicate the requirement to 
>>change the Standard.  That would be a legitimate revision, 
>performed as 
>>a separate project illuminated by all the experience with the 
>original 
>>Standard.
>>
>>It would argue for lengthening the life time or the recreation of 
>>working groups as necessary for such revisions.  Things like IPSec 
>>would essentially have a continuous working group, much like today.
>>
>product), two 
>>steps (Standards are reviewed and corrected before they become a 
>>Standard), and three steps (revisions are made as reality forces 
>>This also reduces the cost of development by focusing the work on  
>>particular Standards that will only get additional attention if they 
>>need it.
>>
>commercial 
>>interest in the standards development process to allow more 
>delegation 
>>of review, at least partially off-loading some of our overloaded 
>>technical reviewers.
>>
>successful 
>>when properly managed in other standards organizations, 
>ranging through 
>>IEC, ISO, and ANSI and their derivative organizations.  At 
>some point, 
>>we will have to face up to the fact that the some of the IETF 
>legendary 
>>approach will have to become more formalized to serve as the 
>universal 
>>And to agree with Eliot, these processes would have to be documented, 
>>managed, and followed.
>>
>>Bob Snively
>>408-835-4321
>>
>>
>to spend 
>>their employees' time to get specs to the 2nd step.
>>Put another way, let's only have the second step if we can reasonably 
>>predict that people will use it.
>>
>customers 
>>the difference between a draft standard and a proposed standard - an 
>>RFC # is sufficient most of the time.  And we know this to be 
>generally 
>>true because we'd at least see more attempts to get from proposed 
>>onward.
>>
>>that three step isn't working.  Now we just have to sort between one 
>>and two and how to encode them.  And I think you'd agree that 
>we really 

.

---

From: Thomas Narten <[email-masked]>
Date: Mon, 06 Mar 2006 20:54:20 -0500

IMO, this  proposal throws out the baby with the bathwater.

While I agree that having the 3 levels isn't really working and may be
the wrong number of levels, having just one level isn't helpful
either.

When I step back and try to articulate what it is about the IETF
process that makes our specs good (when we achieve our ideal), a key
aspect is iteration. Multiple versions of the protocol spec, separated
by implementation/interoperability experience that is fed back into
the next revision of the spec.

We need to continue to encourage such iteration. With just one
standardization level, I worry that we lessen the incentive to iterate
and improve our specs.

There needs to be a step in which some sort of evaluation of the specs
quality is performed. While customers may not care so much about the
label associated with such a "certification", they are useful to the
IETF community in that they give us a desirable target to aim for, and
one that differentiates mature specs from others.

Also, just listing implemenation report isn't generally useful. The
real value of implementation reports and interoperability testing is
what they teach us about the quality of the specs themselves, and
having those lessons fed back into the next version of the spec.

Thomas
.

---

From: Eric Rosen <[email-masked]>
Date: Mon, 06 Mar 2006 21:45:20 -0500

Thomas> When I step back and try to articulate what it is about the IETF
Thomas> process that makes our specs good (when we achieve our ideal), a key
Thomas> aspect is iteration. Multiple versions of the protocol spec, separated
Thomas> by implementation/interoperability experience that is fed back into
Thomas> the next revision of the spec. 

Given the infrequency  with which the second and third  steps of the process
are invoked, I  don't see how it  can be claimed that having  those steps is
what makes our specs good. 

Obviously feedback from actual deployments over time is valuable, but as has
been  pointed out numerous  times over  the years  that newtrk  has existed,
revising a spec does not require additional stages.

Ran> I think that a two step process would actually get followed. 

It would be nice to see some actual evidence in favor of that view.  This is
a little example of why a WG like newtrk can never succeed -- anyone can say
anything, and no amount of technical analysis can resolve the disputes.

Ran> advancing documents from PS to  IS would address existing requests from
Ran> numerous  end users to  have higher  assurance of  interoperability and
Ran> numerous  stated  preferences for  specifications  that  are full  IETF
Ran> standards. 

While  the existing  process  is quite  a  nuisance, it  would certainly  be
invoked more often if this stuff  really mattered to enough people.  I think
the empirical facts  indicate that it doesn't really matter,  but as a group
we can't  seem to shake the feeling  that somehow it _ought_  to matter, and
that it _would_ matter if only we could get the details right ;-(

.

---

From: Thomas Narten <[email-masked]>
Date: Mon, 06 Mar 2006 21:50:58 -0500

> Given the infrequency with which the second and third steps of the
> process are invoked, I don't see how it can be claimed that having
> those steps is what makes our specs good.

You misread what I said. The benefit comes from the actual revisions
of specs, based on the experience of those that have implemented the
previous version of the spec. Whether the successive versions are PS
vs. DS are not the important factors.

Thomas

.

---

From: <[email-masked]>
Date: Tue, 7 Mar 2006 09:05:12 +0200

Hi Thomas,

>While I agree that having the 3 levels isn't really working 
>and may be the wrong number of levels, having just one level 
>isn't helpful either.
>
>When I step back and try to articulate what it is about the 
>IETF process that makes our specs good (when we achieve our 
>ideal), a key aspect is iteration. Multiple versions of the 
>protocol spec, separated by implementation/interoperability 
>experience that is fed back into the next revision of the spec.
>
>standardization level, I worry that we lessen the incentive to 
>iterate and improve our specs.
>
>There needs to be a step in which some sort of evaluation of 
>the specs quality is performed. While customers may not care 
>so much about the label associated with such a 
>"certification", they are useful to the IETF community in that 
>they give us a desirable target to aim for, and one that 
>differentiates mature specs from others.
>
>Also, just listing implemenation report isn't generally 
>useful. The real value of implementation reports and 
>interoperability testing is what they teach us about the 
>quality of the specs themselves, and having those lessons fed 
>back into the next version of the spec.

On paper, I agree with you, but in practice, reality differs from
what you list above.  I'm currently trying to see what should be
done with AAA protocols - namely Diameter & RADIUS.

Frankly, RADIUS (RFC 2865) is a mess, and I don't see that progressing
it up the Standards Track would be a useful thing.  Some rough stats
could be found here:

http://www.freeradius.org/rfc/
http://tools.ietf.org/wg/radext/

(RADIUS would be a protocol were an ISD would be useful, but that's
another story).

Now, we have a DiME working group - and I'm debating (mostly with myself
as temporary 
chair) whether trying to advance Diameter is a good thing, or should we
rather
do a bit of clean-up on it.  If we revise the spec enough, it'll get
recycled
at PS.  This is probably the correct thing to do IMO, but there's the
ideal
of moving it up. 

Likewise, during discussions on the IPv6 mailing list, there seemed to
be more
interest in moving the basic set of IPv6 RFCs to Full Standard, rather
than
clarifying issues if those clarifications would create potentially new 
normative behavior.

There will always be a tension between moving a spec forward or doing
the right 
thing, IMO, and I am not sure this is a good thing.

John

.

---

From: Thomas Narten <[email-masked]>
Date: Tue, 07 Mar 2006 08:50:47 -0500

> the wrong number of levels, having just one level isn't helpful

I am not arguing that the the current 3-level system is working. What
I am saying is that iteration - having multiple revisions of a spec -
is highly beneficial, and accounts for a lot of the value of our
specs. Initial versions of PS documents are rarely "perfect" or even
"great". They are full of ambiguities, etc. Doing revisions -- based
on feedback from implementations and interoperability testing -- is
what makes specs good. It is this aspect of our process that I think
is critical to preserve.

> it up the Standards Track would be a useful thing.  Some rough stats

One thing I'm not sure is written down (or widely understood) is that
there is a "best time" to do a revision, and there are other times
where one has essentially "missed the window" and it's probably too
late in practice. Radius is clearly in the latter category now.

The "best time" is usually within a year or two of the base spec, when
there are some initial implementations, the people who have done that
work are still around (and whose day jobs still relate to that spec),
and interoperability testing is still ongoing and/or fresh in people's
minds. At this time, it is much more likely that deployment experience
is still in the initial stages and people are more flexible about
being willing to upgrade implementations. If one waits too long,
vendors lose interest in upgrading implementations (plus there is real
concern that upgrading the installed base will never happen in
practice), the people who originally worked on the
specs/implementations have moved on and its really hard to get people
to rework a spec.

> Now, we have a DiME working group - and I'm debating (mostly with
> myself as temporary chair) whether trying to advance Diameter is a
> good thing, or should we rather do a bit of clean-up on it.

I'd argue that cleanup is much more valuable to the community than
advancement. What is the better _engineering_ direction to go in?
Which would have a more positive impact on the _operations_ associated
with a protocol? What is better for the Internet?

> If we revise the spec enough, it'll get recycled at PS.  This is
> probably the correct thing to do IMO, but there's the ideal of
> moving it up.

This is where the current 3-step process is a bit broken, IMO. It's a
no-win choice if we must chose between doing what's right vs. doing
something that has a better "cachet" (to the degree that is what it
is).

> Likewise, during discussions on the IPv6 mailing list, there seemed
> to be more interest in moving the basic set of IPv6 RFCs to Full
> Standard, rather than clarifying issues if those clarifications
> would create potentially new normative behavior.

I agree with this concern. I don't like it at all. And when I was AD,
I heard many discussions where this argument was made. It was even
made for excluding specific changes that arguably should have been
made.

I also heard many arguments along the the lines of "we don't want to
change our implementations at this point, so we don't want to see a
revised RFC that makes our current implementations uncompliant". This
goes back to an earlier point, that the timing of when to do a
revision has very practical implications, and one really can't wait
too long before doing a revision.

> There will always be a tension between moving a spec forward or
> doing the right thing, IMO, and I am not sure this is a good thing.

I believe that there would be value in retaining some sort of 2-tier
system, in which there was some incentive for reaching the higher
tier, and where that tier raises the bar w.r.t. other PS's, since not
all PS's are the same. I think the real discussion needs to be about
how high to set that bar, rather than whether there should be a bar at
all.

Thomas
.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 07 Mar 2006 16:41:33 +0100

...
> made for excluding specific changes that arguably should have been

That's why I prefer 2step. It provides a clear incentive (get
better status) and a clear rule (delete everything that
doesn't interoperate). If you can't meet this standard,
there is a strong incentive to recycle at PS to fix the
interop problems. And it isn't speculative; a fraction
of documents do advance or recycle in this way. It's
an improvement process.

Full Standard has a vague rule (widespread use) so
it has a vague meaning. So it's nothing but a label
process.

     Brian

.

---

From: Eric Rosen <[email-masked]>
Date: Tue, 07 Mar 2006 11:05:25 -0500

> specs/implementations have moved on and its really hard to get people

I agree  with this 100%.   However, one problem  I often face is  that often
this "best  time" can pass three, four,  or five years before  the spec ever
appears as an RFC.  When it takes six or seven years to guide a spec through
the process,  those of the  authors who are  still left standing  are rather
unlikely to say "now, let's start work on the next version". 

In  many such  cases, the  PS  document is  actually already  the result  of
iteration based on  implementation, deployment, and interoperability issues.
Of course, no one remembers all  the details of how things were tested years
ago.  

Issues that do come up with  PS docs are often dealt with through additional
PS  RFCs  that modify  and/or  clarify the  initial  doc.   This allows  the
additional docs to be "single issue"  docs that are less likely to catch the
attention  of trolls,  security ADs,  or  others who  specialize in  slowing
things down. 

To achieve your  goal, you should be proposing ways  to expedite the process
so  that it  is possible  to produce  an initial  RFC and  then  produce its
revision before the "best time" has  passed.  I don't think this has much to
do with the number of standardization  levels, it has more to do with things
like WG chartering and WG management.  
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: <[email-masked]>
Date: Sat, 18 Mar 2006 20:48:32 +0200

Thomas,

Catching up on mails ..
>I am not arguing that the the current 3-level system is 
>working. What I am saying is that iteration - having multiple 
>revisions of a spec - is highly beneficial, and accounts for a 
>lot of the value of our specs. Initial versions of PS 
>documents are rarely "perfect" or even "great". They are full 
>of ambiguities, etc. Doing revisions -- based on feedback from 
>implementations and interoperability testing -- is what makes 
>specs good. It is this aspect of our process that I think is 
>critical to preserve.

I do not disagree with you on this.  Ability to be able to fix
or modify protocols is a good thing.

>One thing I'm not sure is written down (or widely understood) 
>is that there is a "best time" to do a revision, and there are 
>other times where one has essentially "missed the window" and 
>it's probably too late in practice. Radius is clearly in the 
>latter category now.
>
>The "best time" is usually within a year or two of the base 
>spec, when there are some initial implementations, the people 
>who have done that work are still around (and whose day jobs 
>still relate to that spec), and interoperability testing is 
>still ongoing and/or fresh in people's minds. At this time, it 
>is much more likely that deployment experience is still in the 
>initial stages and people are more flexible about being 
>willing to upgrade implementations. If one waits too long, 
>vendors lose interest in upgrading implementations (plus there 
>is real concern that upgrading the installed base will never 
>happen in practice), the people who originally worked on the 
>specs/implementations have moved on and its really hard to get 
>people to rework a spec.

At a certain level, I agree, but in practice, this is hard. Getting
some protocols can be quite hard and many working groups run
out of gas.  I think timing is key, though.

Other comments cut, as I basically agree with you.

>> the right thing, IMO, and I am not sure this is a good thing.
>
>I believe that there would be value in retaining some sort of 
>2-tier system, in which there was some incentive for reaching 
>the higher tier, and where that tier raises the bar w.r.t. 
>other PS's, since not all PS's are the same. I think the real 
>discussion needs to be about how high to set that bar, rather 
>than whether there should be a bar at all.

I guess I am saying (IMO) is that tiers don't have value, but the
ability
to fix stuff that is broken has value.  

John

.