Subject: Re: Guidelines for authors and reviewers

From: SM <[email-masked]>
Date: Sun, 01 Jun 2008 19:57:01 -0700

Hi Suresh,
At 14:42 30-05-2008, Suresh Krishnan wrote:
>I do share your sentiment, but this is not what we set out to do
>originally. We set out to document the review process(es) and provide
>some guidelines to effectively use the received reviews. I think the
>right document to weave the stuff together would be the Tao. But I do
>agree that adding references to the Tao and 2026 make sense. The
>dangling reference to 2119 is an editorial oversight :-).

I commend the authors for writing this document even though it will 
most likely attract the attention of the IRS^H^H^H IETF audit team. :-)

Quoting RFC 2555:

   "It has been ultimately fascinating to watch the transformation of the
    RFCs themselves from their earliest, tentative dialog form to today's
    much more structured character."

   "As the Internet has taken on greater economic importance, the standards
    documented in the RFCs have become more important and the RFCs 
more formal."

The Tao has an air of formality in some places.  It will be 
interesting to watch the transformation of this document as it moves forward.

An author needs to have both technical and social skills to see a 
document through the RFC process.  The Document Lifecycle tutorial 
covers the technical points required to produce a RFC.  Your I-D 
focuses on the second aspect.  One of the things I've learned is that 
it's not easy to apply rules there as we have to take the people 
element into account.  Even guidelines can have a detrimental affect 
as some people may read them as rules.

In Section 1:

   "An Internet Draft's life cycle begins when the author(s) submit the
    document as an individual submission"

That may lead to some confusion with independent submissions.  It may 
be better to have:

   "An Internet Draft's life cycle begins when the author(s) submit 
the document
    as a personal draft; it may become a Working Group draft, and usually ends
    when the draft is published as an RFC, or is abandoned and 
eventually expires."

Comments are generally sent either directly to the author(s) or to a 
mailing list.  There may be frustration on both sides as some 
reviewers may expect a personal reply.  The authors can be busy 
people and may not have the time to answer each individual 
email.  Sometimes it's easier for them to post a reply to a mailing 
list for issues about a Working Group document instead of responding 
to the same questions over and over again.

I suggest using "Internet-Draft" and "Last Call" throughout the document.

Section 2 (Sources of reviews) might benefit from some references to 
the Tao to expand on the review process as the document goes through 
the various stages.

In Section 3.1:

   "There are several persons who can respond to a received review.  If
    the document is an individual document"

I suggest using the same term as the RFC Editor.

    There are several persons who can respond to a received review.  If
    the document is an independent submission

In Section 3.3, you mention "every review merits a response."  In my 
personal experience, most authors send a response which may contain a 
question or a note.  Should I label the minute few who don't respond 
as lacking common courtesy? :-) I don't think so (see my comment 
about authors being busy).

In Section 3.3.1, you spell out a time frame for the authors to 
respond.  Some authors take an approach where they wait for the heat 
to turn down or wait for the usual suspects to respond before sending 
in a substantive response to cover the points raised.  It's difficult 
to set an acceptable time frame as some debates can be quite lengthy.

In Section 3.3.3.1:

    "It would be nice for the author to come up with a timeline on when he/
     she would be able to respond to the comments/issues contained in the
     review."

The author may be doing this work on a voluntary basis.  Although 
it's nice to have a timeline, it pressures the author to respond by that date.

In Section 3.3.3.3, there is:

   "However, often that explanation should really be in the draft itself,
    not just in emails to the authors, so that future readers will also
    understand the text."

The longer a document is, the less people are inclined to read it 
thoroughly.  It's a balance between what to put in the document while 
keeping it within an acceptable size.  An explanation is not put in 
because the reviewer did not understand some part of the draft; it is 
added to provide context or a rationale and make the specifications 
clear-cut.  Otherwise, the document would turn into a tutorial.

   "Soliciting text from the reviewer should be used sparingly."

The reviewer should also be part of the participative effort.  That's 
why there is "send text".  Else, the author has to second guess 
what's in the reviewer's mind which is no easy task.  It also makes 
it difficult to quell arguments as there may be people who disagree 
without proposing any resolution.

In Section 3.4, there is:

  "this is buy using"

In Section 3.5:

   Once the author feels that all the open issues have been addressed he/she
   can submit a new revision of the draft that incorporates the changes.

Section 3.6 mentions Acknowledgements.  The contributor is not only 
mentioned in a RFC out of politeness.  There may be an overlap with 
other IETF documents if you get into that.

Trivia, there was a recent I-D that contained a love note. :-)

I hope that this email is not a disheartening read.  Some of the 
ideas in the I-D are excellent.  Whether they are practical is 
another question.  Even though you are the authors, the onus of 
refining the document should not rest on you alone.  You raised 
several points which may be on the minds of the silent majority that 
follow the IETF mailing lists.

Regards,
-sm 

_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 2 Jun 2008 06:04:54 +0200

> Quoting RFC 2555:

Hopefully we'll get a 5555 next year :-)

> The Tao has an air of formality in some places.

Mostly limited to appendix B, though. 

> In Section 1:
> 
>    "An Internet Draft's life cycle begins when the author(s) 
>     submit the document as an individual submission"
 
> That may lead to some confusion with independent submissions.
> It may be better to have:
 
>    "An Internet Draft's life cycle begins when the author(s)
>     submit the document as a personal draft; it may become a
>     Working Group draft

Dunno, independent drafts are non-IETF by definition, using one
adjective "individual" everywhere at least doesn't add to any
existing confusion about "individual" vs. "independent" drafts.
Example:

> In Section 3.1:
> 
>    "There are several persons who can respond to a received
>     review.  If the document is an individual document"
 
>     There are several persons who can respond to a received
>     review.  If the document is an independent submission

But the guidelines are for IETF drafts, that's either WG draft
or individual, but not independent.  Ignoring IAB drafts and
IRTF drafts for the moment, anyway not the audience for these
guidelines.

 Frank

---

From: SM <[email-masked]>
Date: Sun, 01 Jun 2008 23:14:39 -0700

Hi Frank,
>Hopefully we'll get a 5555 next year :-)

I hope so too. :-)

>existing confusion about "individual" vs. "independent" drafts.

The topics discussed in this document are relevant to all I-Ds unless 
one views independent drafts as documents set in read-only mode only.

>or individual, but not independent.  Ignoring IAB drafts and

Good point.

Regards,
-sm

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 2 Jun 2008 11:54:21 +0200

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Joel M. Halpern
> Sent: Saturday, May 31, 2008 3:12 AM
> To: IETF Discussion
> Subject: Re: Guidelines for authors and reviewers
> 
> Comment inline, with most of the discussion elided.  I 
> believe that one particular question gets to the heart of 
> what is bothering me.
> 
> Ted Hardie wrote:
> > At 4:08 PM -0700 5/30/08, Joel M. Halpern wrote:
> ...
> >> On design decisions, there is an even more complex 
> tradeoff.  I have 
> >> reviewed several documents which had questionable design 
> decisions.  
> >> In one review I recently wrote that I did not expect the following 
> >> comment to change the WG consensus, but that I considered the 
> >> specific mechanism used by the document a bad idea.  If I had not 
> >> known that the particular mechanism had been discussed, I 
> might have put it more forcefully.
> >> On the other hand, a while back I reviewed a document which had a 
> >> mechanism which, although the working group had indeed 
> agreed on it, 
> >> fundamentally didn't work.  I don't care how much they agreed.  It 
> >> needed to be changed.  And they changed it.  (It was 
> incumbent upon 
> >> me to provide a clear and coherent explanation of why it 
> was broken.)
> > 
> > These both sound like excellent reviews:  you expressed 
> your personal 
> > design preferences in the first instance but did not try to 
> force it 
> > over the consensus of the working group, and pointed out a 
> showstopper 
> > in the second.
> > 
> > Now, show me in this draft how these two cases are distinguished, 
> > which is critical to getting a review done right?
> 
> The problem I have is that I do not know how to write text in 
> a draft that distinguishes those two.  The line between them 
> is very tricky, and possibly subjective.
> And part of the problem is to avoid turning it into a fight.  
> If all review comments get clear, reasonably timely 
> responses, there is room for the discussion without acrimony.
> 

Here is a key issue, and this is why ensuring that reviews are responded
in a timely manner is essential. As an AD I have been too often in the
situation to review a document which is on the IESG agenda for approval
(which means less than one week for me to review and get to an educated
opinion for my IESG balloting) and to find that expert reviews were
never answered by the authors or the document shepherd. 

Dan

---

From: Ted Hardie <[email-masked]>
Date: Mon, 2 Jun 2008 12:00:51 -0700

At 5:12 PM -0700 5/30/08, Joel M. Halpern wrote:
> >
>> These both sound like excellent reviews:  you expressed your personal
>> design preferences in the first instance but did not try to force it over
>> the consensus of the working group, and pointed out a showstopper
>> in the second.
>>
>The problem I have is that I do not know how to write text in a draft
>that distinguishes those two.  The line between them is very tricky, and
>possibly subjective.

Here is where I think we may have some room for some fruitful discussion.

A review that is raising a showstopper has a provable or disprovable statement
in it.  "This *will not work* in the following scenario" or even "This seems
to have poor results in networks with high rates of non-congestive loss" creates the
opportunity for the reviewer and working group to discuss the issue in terms
that can be tested.   The working group may argue (and, presumably, demonstrate)
that either statement is false, but there is a way to move forward that is not
just iteration of position.  To put that task to the working group, a reviewer
should be able to back their statement with a reasonably clear, worked set of
reasons for the assertion.   A working group can disprove the statement,
indicate why an applicability statement is a better response ( demonstrating
that the cases where it does work or does have reasonable results are useful
enough and distinguishable enough that the document should go forward),
or make relevant technical changes.

But review comments that do not contain testable assertions end up
being subjective.  "I think ZNK  ChillOut is better than ZNK BindMeTight,
for the following reasons" may contain a good set of reasons, but it
should never over-rule the consensus of a working group that has
agreed on ZNK BindMeTight unless one of those reasons amounts to
"it doesn't work".  Comments of the type "I think Section 4 is not
clear enough for an implementor to follow" are also subjective; they
are very valuable and may serve as a guide to the WG/author team,
but it is important for the reviewer to recognize that they may not
result in change.  

>And part of the problem is to avoid turning it into a fight.  If all
>review comments get clear, reasonably timely responses, there is room
>for the discussion without acrimony.

I don't think that helps unless there is a clear set of expectations
*on the part of reviewers* on what their responsibilities are in
producing an actionable review and in accepting that some of their
comments may result in no change.  I believe that the right way
to do that is to ground the description of the review process in
a strong understanding of the document production and participation
model of the IETF.  If you don't, you end up with an unbalanced
view in which the few hours of effort put in on a solicited review
have a very much larger effect than the effort of the participants
who produced the document and will use the protocol it
describes.

>Would it address your concern if the document said something like:
>    "Reviewers should be sensitive to the difference between
>    their personal opinions (and preferences) and issues
>    which will affect the correct operation or interoperation
>    of the documents under review"
>?

I don't think this is nearly enough.

					Ted

>I have no problem with pointing out that there are two different
>categories.  I have real problems with trying to define a hard line
>which distinguishes them.
>
>Yours,
>Joel
>
>PS: While there are other differences in our views, they seem to be
>questions on which reasonable folk may differ and we can let the
>community sort out how to write the wording.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 02 Jun 2008 15:37:44 -0400

Trying to paraphrase Ted's response to my comments, and then reply. 
(The paraphrase is to try to make sure that I understood the comment.)

I think that Ted is saying that review comments should have testable, 
actionable comments.  Further, if I am reading him right, he is saying 
that comments that are not testable are probably opinion.

That does seem a good place to start.  Further, Ted comments taht 
reviewers should understand that not all their comments are going to be 
accepted.  I agree.  Not even all of their comments based on proper 
understanding of the document are going to be accepted.

But, the question of where the boundary is between "that does not work" 
and "in my opinion that is a bad idea"  is, in my experience, frequently 
fuzzy.  To use a concrete example, we have plenty of experience to show 
that using a field length to determine what kind of field something has 
is a bad idea.  It fails, over time, in a myriad ways.  But no matter 
how many horror stories I put together, I was going to lose that 
particular debate with that particular document.
On the other hand, there are things which look perfect in theory, but we 
know turn out to be more fragile and are therefore bad ideas.  The 
assertion that such a thing is a bad idea is going to look like opinion, 
but is useful.

In fact, part of my concern is that sometimes opinions can be as 
important as facts.  Yes, reviewers need to be prepared to accept 
push-back.  But it should not be against the rules for a reviewer to say 
"I have seen exactly this before, and by the time they made it work it 
was a mess.  Please don't do this."

And even if it is opinion, it deserves a response.

TO use another example, when I said to a working group "Having worked 
with a lot of testers, I think that they will have trouble following the 
instructions as you have written them." I was clearly stating an 
opinion.  But it was an opinion that at least deserved a response.  (In 
fact, the working group did respond, and a good compromise was reached 
on that and a number of related issues.)

Part of this is that just as we expect authors to respond politely to 
reviewers, reviewers have an obligation of politeness towards the WG. 
Reviewers should assume that odd things are there for good reasons, even 
while raising the question as to whether they belong.

But trying to create a hard line between opinion and fact is I believe 
counter-productive.  If the issue is larger than it looks on reading the 
review, then the reviewer can add more explanation, examples, or 
justification in the dialog with the authors / WG.  Similarly, if the WG 
is overreacting to something but feels it is a big deal, the reviewer 
can say in followup "I had not realized that would be a big deal.  Let 
it be."

We are all presumably adults.  We are also human.

There is also a subtext that comes up periodically about the comments 
being actionable.  This has been raised for outside reviews and IESG 
reviews.  While I certainly agree that actionable comments are much 
better, it is not always possible.  If the problem is that the reader 
can not figure out what the text means, it is hard to suggest a fix.  It 
also gets laborious if the text could mean one of three different 
things.  Should the reviewer have to provide text for all three 
alternative meanings?  After all "clarify the text" while clear is not 
very specific as to what action is needed.  There are also other cases 
that come up where it may be impractical to require actionable comments.

Yours,
Joel M. Halpern

---

From: Scott Brim <[email-masked]>
Date: Tue, 03 Jun 2008 10:57:16 -0400

On 5/30/08 7:54 PM, Ted Hardie allegedly wrote:
>> There are several issues that get mixed up together in defining a late
>> external review process.  By definition, this is an external review.  So
>> it is not reasonable to say that the reviewer should think like a WG
>> member, or have the full email conversations available.
> 
> No, it is not.  But it is also not reasonable to assume that each document
> carry the full context of an area with it.  There is a balance there,
> and assuming that contextualizing information should be added to
> each document on the basis of reviewer request is a problem.  In addition
> to making each document very large, they run the great risk of getting
> out of synch, with the usual hilarity resulting.

If there is _written_ context in other drafts, that's fine.  If 
_cultural_ context is missing from the drafts, that's a problem that 
should be noted and fixed.  Sometimes the drafts where the context 
should be added are already out the door, and the only place to put the 
needed context is in the draft under review.

> But we're not writing our documents for reviewers; we're writing them for
> implementors, operators, and our successors, toiling in these protocol farms,
> right?  So there is some level of common knowledge among those working
> within a protocol area we can and should assume in our documents.

You can't assume that.  We get strange questions from implementors and 
operators all the time.  Far stranger than the ones from reviewers. 
Better to assume that reviewers understand the audience as well as you 
do and what their needs will be.

> We can and should provide citations (that's what those nifty normative
> and informative references are for), but that doesn't translate into providing
> tutorial text.   I was once a big fan of it, but I see people go wrong on the basis
> of thrown-in tutorial text often enough now that I think it is often
> less than helpful and sometimes actively harmful (by causing the reader
> to believe that it is sufficient context).

Personally I agree in separating out tutorial text, but all context 
required must be supplied somewhere.

On 6/2/08 3:00 PM, Ted Hardie allegedly wrote:
> that can be tested.   

Agreed, and reviewers with expertise do this.  Reviewers with less 
expertise may just explain their doubt and ask if it is justified, but 
even questions like that should be considered if they have not been 
already, because on occasion I've seen them save a group from disaster.

> "it doesn't work".  

In many WGs there are multiple designs that would work, and various 
tradeoffs are made in choosing one.  There is never (afaik) a 
requirement that one approach prove the other simply does not work. 
Similarly early reviews can be very helpful even in re-examining known 
tradeoffs or bringing up new ones so that the balance between the 
approaches changes.  It is rare for a late review to change a WG 
decision in this way -- it's hard for a WG to backtrack -- but it does 
happen occasionally, and there is never a requirement that a review 
prove that a design is utterly broken, just that there are serious 
disadvantages.

If a reviewer comment is opinion based on a great deal of experience, 
for example with system design, then the reviewer -- presenting 
him/herself as having special expertise -- then has an obligation to 
recommend a solution to the problem he/she sees, in detail.

 > Comments of the type "I think Section 4 is not

Yes.

Scott

---

From: Suresh Krishnan <[email-masked]>
Date: Tue, 03 Jun 2008 13:55:06 -0400

Hi Paul,

Paul Hoffman wrote:
>> Agree. And this topic (the recipient list of the review) is something 
>> I think hard about before I send out any review.
> 
> That's good to hear, but I didn't see it reflected in the document; 
> maybe your co-authors had a different slant. Regardless, my preference 
> is to encourage group communication during reviews for anything other 
> than editorial nits and "I was told to read this; I did; it was fine" 
> reviews. Group communication, in both directions for a review, helps 
> everyone. It also helps prevent a WG hearing that "I changed this thing 
> we had all agreed to because I was told to by { a security person | an 
> IAB member | an ex-AD | ... }". Those kinds of changes tend to make a 
> document weaker if they aren't agreed to and possibly modified by the WG 
> who worked on the document.

I will add the following text regarding this in a new section 4.2 before 
the current section 4.2.

***************************START OF TEXT********************************
4.2 Recipients of the review

The list of recipients of the review is tricky to get right. The main 
idea is to make sure all the relevant people receive the review. The 
recipient list is determined mainly by the following factors

* The timeframe of the review (early vs. late)
* The contents of the review (editorial vs. technical)

Early reviews are usually performed by active participants of a working 
group. The preferred destination for these reviews is the working group 
mailing list since it can be reasonably assumed that the persons 
interested in the document are subscribed to the mailing list. This 
applies for both technical and editorial issues. Alternately editorial 
issues can be resolved using a private mail to the author(s).

Late reviews are usually performed by persons who are not active 
participants of the working group and who usually review the draft from 
a different point of view than the working group. If the contents of the 
review are mainly editorial in nature, the reviews can be sent just to 
the authors, the working group chair(s), the document shepherd(s). If 
the review is of a more technical nature it is considered polite to 
include the working group mailing list and/or the IETF discussion list. 
As it is not reasonable to assume that the reviewer will subscribe to 
the working group mailing list just for discussing this issue, the 
working group chair(s) need to make sure that this review will get past 
any moderation controls imposed on non-subscribers by the working group 
mailing list.
****************************END OF TEXT*********************************

Would this resolve your concern?

Thanks
Suresh

---

From: Ted Hardie <[email-masked]>
Date: Tue, 3 Jun 2008 11:12:31 -0700

At 10:55 AM -0700 6/3/08, Suresh Krishnan wrote:
>group.

Suresh,
	Can you describe  the difference between "early reviews
performed by active participants" and just "participation"?   In my
mind, folks making suggestions and noting issues with a document during
the working group phase are WG contributors, not reviewers.  How do
you see this?
			Ted
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Suresh Krishnan <[email-masked]>
Date: Tue, 03 Jun 2008 15:11:22 -0400

Hi Ted,

> the working group phase are WG contributors, not reviewers.  How do

I see it just like you :-). The early reviewers are contributors and 
participants. The reason I made the distinction is because reviews are 
only one form of participation in the working group. There are other 
ways to participate in the working group

* Editing documents
* Keeping track of issues
* Contributing text to sections
etc.

Thanks
Suresh

---

From: Pekka Savola <[email-masked]>
Date: Wed, 4 Jun 2008 06:53:33 +0300 (EEST)

> the working group phase are WG contributors, not reviewers.  How do

If the reviewer is not subscribed to the WG list, and does not intend 
to become subscribed to the list, yet does a review and sends comments 
for a WG draft, then I'd say it's in the former camp.

Personally, I usually do that.  And it's a PITA that most WG mailing 
lists where I try to send mail unsubscribed just throw the mails to a 
black hole or a moderation queue that's never flushed. This is why I 
Cc: the WG chairs on these messages.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf