Subject: Reasons for delay

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 14 Jun 2005 15:15:10 -0400

The problem with capturing why WGs and drafts are delayed is that  
delayed drafts are like unhappy families: they are all unique in  
their misery. Here are some causes that I have seen:

(1) Waiting for others, particularly normative references. There are  
proposals to fix this problem; at least in my experience, this only  
becomes a problem if another draft is very late for other reasons, so  
fixing the other reasons will make this problem less likely.

(2) Sequential processing: There is very little parallel processing  
going on. Given that close to 100% of late-stage WG drafts eventually  
get published as an RFC, it would seem to be quite possible to do  
some post-IESG steps early. For example, the IANA review could be  
done during IETF LC. On occasion, changes in the draft will mean that  
IANA will have to redo work, but like branch prediction in a CPU, the  
average reduction in delay seems very likely to outweigh this  
incremental effort. It seems like this could easily be explored  
experimentally.

(3) Exhaustion: Far too many drafts linger years in 90%-completed  
state, while the authors or the WG has moved on to other things. It  
would be interesting to take a look at long-delayed drafts and see  
how much they have really changed as a function of time. My guess is  
that the change rate goes towards one-sentence-per-month or less.

(4) No comments: Many drafts receive no comments during WGLC or IETF  
LC. There are many reasons for that, but one reason is that there is  
very little motivation for reviewers, besides the feeling of doing  
something for the benefit of mankind. In most cases, there is no  
tracking of issues raised and the authors simply respond that "the  
new version addresses the issues raised during last call". The  
reviewer would have to go back and re-read the draft to make sure  
that has happened. Some WGs designate reviewers, but this seems to be  
done haphazardly and inconsistently. Most authors acknowledge LC  
reviewers, but this is again not always done.

(5) 6 work weeks: One sometimes has the impression that all work gets  
compressed into the two weeks before the IETF I-D deadline, with very  
little activity in-between. This applies to WG chairs and authors,  
unfortunately.

(6) General attitude: If it becomes custom that drafts takes three or  
five years to complete, nobody is in any particular hurry, after all  
it "always takes that long".

(7) We put a lot of freedom on document authors, essentially granting  
them a monopoly of the pen, but there is very little project  
management. In other words, we treat drafts as if they were science  
papers, but we're expecting results as if we were doing product  
development. Good project management would increase the  
responsibility of authors,  and make it easier to set expectations.

(8) WGLC limbo: After WGLC, there is often a long period of no  
action. There should be an expectation that within a very short time  
(say, one week unless major technical issues need to be revisited)  
the authors complete a revision and the chairs pass the draft to the  
AD. Drafts that have gone through WGLC and are not in AD hands after  
that period should be flagged so that they can receive particular  
attention.

(9) No WGLC schedule: The WGLC schedule is not published, so one has  
to dig through the mailing list archives to find such calls.  
Scheduling is often haphazard. Instead, WGLC should be used as a  
deadline to make things happen: Schedule the WGLC when it seems  
likely that the draft can be completed in, say, 3 or 6 months. Give  
the author and WG a deadline: either get it ready for the scheduled  
date or go back to the end of the queue. If a draft misses its WGLC  
target twice, escalate as appropriate, e.g., by swapping out editors,  
forcing author teleconferences, etc. Mark those drafts so that  
everybody can see those problem drafts.

Henning

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Spencer Dawkins" <[email-masked]>
Date: Thu, 16 Jun 2005 06:50:27 -0500

Henning,

Thanks again for the thoughtful set of suggestions.

Spencer

> that the change rate goes towards one-sentence-per-month or less.

Yes, and this interacts badly with document quality as community "best 
practices" change over time - what was recommended two years ago may 
now be deprecated, but most people don't notice the conflicts.

> that has happened. Some WGs designate reviewers, but this seems to 
> be  done haphazardly and inconsistently. Most authors acknowledge LC 

One of Keith's better suggestions was showing "external review" as a 
WG deliverable. I don't know that this is required in all cases, but 
if an AD says "and you need external review for this document", I 
would believe that.

> (5) 6 work weeks: One sometimes has the impression that all work 
> gets  compressed into the two weeks before the IETF I-D deadline, 
> with very  little activity in-between. This applies to WG chairs and 
> authors,  unfortunately.

This interacts badly with (4). A number of IETF plenary presentations 
have shown statistics like "half of the I-Ds were submitted within two 
weeks of the I-D cutoff(s)". The resulting blizzard of text makes it 
even less likely that any specific I-D will receive careful scrutiny 
outside the WG (and, maybe, outside the list of authors).

One possible fix, easy to try - WGs seem to set their milestones based 
on IETF meeting dates ("we'll do WGLC before Paris"). If WGs moved 
their milestones a month earlier, WG chairs and ADs would have more 
opportunities to notice problems and resolve them, and (one hopes) 
milestone I-Ds would not get lost in the general onslaught at I-D 
cutoff time.

> (6) General attitude: If it becomes custom that drafts takes three 
> or  five years to complete, nobody is in any particular hurry, after 
> all  it "always takes that long".

(a) I've seen reluctance to push back on "insanely old" drafts - my 
favorite example was a draft on the IESG telechat calendar in 2004 
that still had RFC 2119 referenced AS A DRAFT (2119 WAS a draft in 
1997, when it was referenced in the 00 version of the telechat 
specification). Delays don't always result in higher quality.

(b) It's likely that at least some of the document authors stop caring 
about their documents in any meaningful way, five years later, so this 
can lead to further delays and AUTH48 confusion (I participated in an 
AUTH48 process for a five-year-old Internet-Draft that generated 16 
"final versions" and 142 e-mails between authors and the RFC-Editor).

(c) the interaction between "it always takes that long" and WG 
charters that show 12-18 month schedules for document approval just 
leads to cynicism and demotivation. For a thought exercise, visualize 
the charter discussions if most WG charters showed the actual 
milestone dates achieved.

> (7) We put a lot of freedom on document authors, essentially 
> granting  them a monopoly of the pen, but there is very little 
> project  management. In other words, we treat drafts as if they were 
> science  papers, but we're expecting results as if we were doing 
> product  development. Good project management would increase the 

I could not agree with Henning's "science paper/product development" 
comment more.

> that period should be flagged so that they can receive particular 

Heck - make it a month. Drafts that can't be fixed in a month are 
likely to need attention anyway.

> target twice, escalate as appropriate, e.g., by swapping out 
> editors,  forcing author teleconferences, etc. Mark those drafts so 
> that  everybody can see those problem drafts.

(a) As Lars-Erik points out - almost every one does WGLCs, but WGLCs 
are not a mandatory part of the standards-track process. Do we 
actually need to give WGs this freedom, especially for specifications 
that are (theoretically) coming out of the WG?

(b) In the SIP community, the WG chairs are rate-limiting the number 
of active WGLCs at any given time, so something like Henning's 
suggestion is already happening in some parts of the IETF.

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 16 Jun 2005 08:57:52 -0400

Spencer Dawkins wrote:

>> would be interesting to take a look at long-delayed drafts and see how 
>> much they have really changed as a function of time. My guess is that 
>> the change rate goes towards one-sentence-per-month or less.
> 
> 
> practices" change over time - what was recommended two years ago may now 
> be deprecated, but most people don't notice the conflicts.

Plus, it becomes very difficult for authors and the WG to do context 
switches. Personally, I find it very difficult to work on a draft that I 
haven't looked at in a year - it takes a while to re-familiarize oneself 
with the issue, dig up old emails and not introduce new mistakes. This 
is exactly the same problem that some of us know from trying to debug 
our own code after a long absence.

This effect tends to be self-reinforcing: Knowing that it will be 
painful means that I'll be less likely to work on the draft.

> One of Keith's better suggestions was showing "external review" as a WG 
> deliverable. I don't know that this is required in all cases, but if an 
> AD says "and you need external review for this document", I would 
> believe that.

And this information should be captured somewhere.

> outside the WG (and, maybe, outside the list of authors).

On the other hand, I've heard from a WG chair that he doesn't schedule 
WGLCs except during that pre-cutoff time, as that's the only time people 
seem to be willing to read drafts and submit comments.

> on IETF meeting dates ("we'll do WGLC before Paris"). If WGs moved their 
> milestones a month earlier, WG chairs and ADs would have more 
> milestone I-Ds would not get lost in the general onslaught at I-D cutoff 
> time.

The pre-scheduled WGLCs I suggested may help as well.

> "final versions" and 142 e-mails between authors and the RFC-Editor).

Right - this then leads to tremendous pressure to "clean up" the draft 
in AUTH48, with very unsatisfying results for all parties.

In many cases, authors have changed companies and work fields in the 3-5 
year period. It's hard to get cycles out of an author that no longer 
works for a company that has any interest in the document topic.

> (a) As Lars-Erik points out - almost every one does WGLCs, but WGLCs are 
> not a mandatory part of the standards-track process. Do we actually need 
> to give WGs this freedom, especially for specifications that are 
> (theoretically) coming out of the WG?

I have yet to see a WG draft that did not receive a WGLC (or two...). I 
don't see the value of worrying too much about these oddities, or is 
there some philosophical objection to WGLC in some WGs?

> 
> (b) In the SIP community, the WG chairs are rate-limiting the number of 
> active WGLCs at any given time, so something like Henning's suggestion 
> is already happening in some parts of the IETF.

Let's just say that bursts of enthusiasm and scheduling are followed by 
years of no pre-scheduled WGLCs. In general, I've found that the biggest 
problem of trying project management tools in the IETF is the 
predictable cycle:

(1) WG looks at charter - "Ohmygod - we're three years behind schedule".

(2) A burst of energy erupts, with a couple of WGLCs scheduled.

(3) We pat ourselves on the back.

(4) We go back to the familiar ways since the mechanisms, culture, 
supervision and tools don't exist to make this part of the standard 
operating procedure.

Henning