Subject: Comments on document shepherd writeups

From: Barry Leiba <[email-masked]>
Date: Sun, 15 Jul 2012 11:42:32 -0400

Having had some time, now, to read quite a lot of document shepherd
writeups (yes, we do read them, at least some of us), I want to give
my thoughts on what makes them more useful to the IESG -- and, so,
also, what doesn't help.  I'll start by saying that these are my
personal thoughts, and not any sort of consensus of the IESG.

First: the shepherd writeup has two major purposes.  It ensures that
certain things have been thought about and checked, and it provides
information to the IESG and input to the its evaluation of the
document.  It's tempting to answer as many of the questions on the
writeup questionnaire as possible with "yes" or "no", and that might
work for the first goal.  In most cases, it doesn't meet the second
goal well at all.

To be clear, I don't want to see things like "Yes" to question 13
replaced with "All references within this document have been
identified as either normative or informative."  That doesn't help,
and question 13 is one where "Yes" really is a fine answer (but note
that "No" is not; that would require some explanation).  I'm not
suggesting more wordy versions of "yes" and "no", but answers that
actually give us information, when there's information to give.  I'm
looking to avoid perfunctory writeups, of which there are now many.
(And I'll add that I've also read many excellent and informative
writeups; thanks for those!)

I also know that the questionnaire is imperfect.  Some of the
questions seem like they should be combined, and some (3 and 12 are
examples) were worded as they are specifically to avoid "yes" and
"no", but the result is often awkward or silly.

Two other small items: (a) Remember that the shepherd writeup will be
posted publicly; if there's anything you want to communicate privately
to the responsible AD or to the IESG as a whole, please do that
separately.  (b) Please use the tracker to post the writeup yourself,
and *also* send it to the responsible AD with CC to
<[email-masked]>.  To post it in the tracker, go to the
tracker page for the document, sign in (upper right), and click the
"Change stream writeup" link.  If you are a non-chair shepherd, ask
one of the chairs to do it.  (The AD or the IESG Secretary will also
post the writeup as a comment in the document history.)

Let me, then, run through the current questionnaire briefly, and
suggest the sorts of things I'd like to see conveyed in the answers.
You can follow along with the writeup template; I won't repeat the
questions here.
   https://www.ietf.org/iesg/template/doc-writeup.txt
I also won't go through every question -- only the ones I've found
often needing more by the shepherd.

--------------------------------

(1) There've been a number of cases where a draft was copied from
another as a starting point, and the author never changed the intended
status.  This is a reminder to the shepherd to check that, and to make
sure it's as intended.  But it's also looking for information about
the *reasons* for selecting the intended status.  There are plenty of
times when it's obvious to anyone that a document needs to be
Standards Track, or Informational, and just a brief answer is fine.
There are also many instances of the IESG quibbling about the status,
wondering why this document is Experimental, why that one is BCP
instead of Informational or Standards Track, why this is Standards
Track when it just registers a code point, and shouldn't it be
Informational, and so on.

Use this question to tell us something about the working group's
discussion of the status of this document.  For Experimental,
especially, explain why it's Experimental, and talk about how we'll
know if or when it might be revised to a different status.  If a
document doesn't specify a protocol but it's Standards Track, explain
that.  There aren't "right" or "wrong" answers; this is information
for the IESG.  Help us avoid discussion about things that were already
done to death in the working group.

(2) In the Working Group Summary section, please talk about the
discussions the document went through, and give specifics (you can use
questions (6) and (9) for this also).  I think it's best for a
document shepherd to be assigned as soon as a draft is proposed for
adoption by a working group, and the shepherd's job begins then and
runs through the publication as an RFC.  The shepherd should be
following the discussions, moving them forward, ensuring that the
authors/editors are responsive, and helping things converge.  Tell us
something about those discussions.  When you say, "There was some
controversy, but it was soon resolved," tell us what the controversy
was, tell us (briefly) what the two (or more) sides of the issue were,
and what compromises were made to come to rough consensus.

One reason we need to know this is so we can evaluate how the process
went, but it's also often the case that some of the same issues come
up during IESG evaluation.  If you tell us what happened in the
working group, we won't have to revisit it as we review the document.
A couple of times, I've put a DISCUSS position on documents that came
from not knowing the history.  The chairs explained the situation and
I cleared my DISCUSSes before the telechat... no terrible harm was
done, but some time was wasted for me, the responsible AD, the chairs,
and the authors.  Avoid that by explaining things up front.  Try to
anticipate what the IESG might need to know.

In the "Document Quality" section is a question about existing
implementations and plans to implement.  Those questions are often
misconstrued: as with much of the rest, there's not a "right" answer,
and the IESG is not requiring that there be implementations.  We're
looking for *information*.  So if there are no implementations, it's
OK: say so.  If there are neither implementations nor plans to
implement, and the document is going for Proposed Standard, it will be
useful to say a few words about why it's useful to proceed with the
protocol -- is there related work that has to finish first, or are
there other factors that are likely to spur implementations soon?

(4) (5) (12) I think these should be merged (and there's a small
overlap with (9) as well), but the point here is to talk about how the
reviews of the document went.  "No" might sometimes seem a fine answer
for question 4, but in general we're looking for something more
specific.  Tell us, in some rough sense, how many people reviewed it
in the working group, whether people from other working groups or
areas reviewed it (and how those reviews went), whether directorate
reviews have come in yet and how those went.  If your document
specifies an XML schema, defines a new HTTP header field, or uses
public keys or hash functions, tell us who reviewed those and what the
discussion was like.  If there was already some discussion with an AD,
discuss that.  You want to show us that we need worry little about
this, or alert us that we might need to take a closer look.  Again,
try to anticipate what we might ask about -- you know the Security ADs
will perk up if your document talks about key management, so
anticipate the questions in your explanation of the reviews.

(7) This question has been a bit problematic, not in its existence or
in the answers, but in the process of getting the answers.  Note that
the question is looking for assurance of compliance with BCP 79, and
note that BCP 79 specifically requires disclosure only for IPR that
the contributor personally knows about.  *Very* often, I'm seeing some
delays in getting documents to the responsible AD because one or more
of the authors has to go off and ask her company's legal department
before answering the query from the shepherd or chair.

In general, it should NOT be necessary to ask one's legal advisors
about this, and the question the shepherds ask should be worded so as
not to give that impression.  "Do you or your company have any IPR on
this document?" is a clear prompt for someone to check with the
lawyers.  "Do you personally know of any IPR related to this document
that has not already been disclosed?" is more likely to get the right
sense across.  If people want to voluntarily take the extra step of
checking with their companies, that's OK... but please don't phrase
your question in a way that makes them think they have to.

(8) (14) (15) These questions are here for a particular reason: our
process requires that the Last Call announcement informs the community
of these sorts of things, and if the shepherd fails to highlight
something that needs to be noted for Last Call, and the responsible AD
doesn't catch it, there will need to be a second Last Call to cover
it.  That will usually delay the document by at least two weeks, and
will cause unnecessary hassle.  Please be careful about getting these
right.  As with most of the other questions, it's not an issue of
"right" and "wrong" answers, but of accurate ones.

(11) I find this question a little funny.  On the one hand, I
personally think it's silly to worry about some of the nits that are
called out by the idnits check.  On the other hand, I think authors
should be required to run idnits and to either clean up or explain
every one of them before asking the shepherd to move forward with the
document.  The shepherd should then re-check it.  I've seen shepherd
writeups that said there were no nits called out, when there were.
I've seen ones that said "there are nits, but they're OK," when some
clearly *do* need to be fixed (and, it's true, some really don't --
some are false positives, some are the way they are for a reason, and
some would just happily be fixed by the RFC Editor).  But just know
that some ADs will run idnits themselves, and will comment on any
errors or warnings that come out of it.  Don't be easy game for the
IESG in that regard: make us *earn* our pay.

(12) Apart from what I said above (under (4) (5) (12)), if the
document is registering a value in a registry that requires expert
review, please make sure the proper process has been followed.  Part
of the shepherd's responsibility is to follow through with any
necessary expert reviews, and to make sure they get done.  If you
aren't sure what the process is, ask your AD for help.

(16) Related to that, one of the things that idnits calls out is
failure to say "This document updates RFC xyz" in the abstract when
the document header says "Updates: xyz".  Many of us think that's a
little silly, and it sometimes is, but here's the thing: Sometimes the
abstract is read on its own, separated from the rest of the document
and from the metadata (the stuff in the header).  And sometimes it's
not obvious *why* a document updates another.  More than just checking
mechanically that the abstract has a brief phrase in it that's
redundant with the document header, please make sure that someone
reading the abstract and introduction will *understand* what's going
on, and will know why things are updating or obsoleting other things.
"This document is a complete replacement for RFC xyz."  "This document
obsoletes RFC xyz by providing an alternative solution."  "This
document changes the definition of the BANANA field, and updates RFC
xyz in doing so."  That's the sort of thing that's of actual use in
the document; make sure it's there.

(17) Finally, one of my pet-peeve sections: I would say that *half* of
the IANA Considerations sections I read are lacking in one way or
another.  IANA are very good at figuring out what you mean, even when
you're not clear.  But don't make them have to figure it out -- they
do guess wrong sometimes, and, despite all the checks and
counter-checks, errors don't always get caught.  I look closely at
these during my review, and will call out problems, though these days
IANA will have tried to sort them out during Last Call.  I'm asking
that the shepherds address it *before* then.

- Make sure that each IANA action that's needed is clear and distinct.
 I like to put each action in its own subsection.  A numbered list
also works nicely, I think.  Some put each action in a separate
paragraph.  In any case, try to avoid things like "IANA are asked to
register item 1 in registry 1 and item 2 in registry 2," because it
just makes it harder to separate the actions.

- Make sure that all existing registries are *clearly* identified by
their *exact* names.  Don't guess; look them up.  The IANA registries
are here:
   https://www.iana.org/protocols/
I can't stress enough how important it is to use the *exact* name.  We
recently had a document that aimed to register something in the "vCard
Property Parameters" registry.  If you look at the link above, you'll
see that there's a "vCard Parameters" registry and a "vCard
Properties" registry.  IANA guessed the latter, but the former was the
correct one, and the result was several rounds of discussion with IANA
to make sure it was done right and that the document was correctly
updated.  Don't guess, and don't make them guess.  Look it up (and
here: if *you* can't find it, what makes you think IANA can?).

- It doesn't hurt to include the URL for the registry, as well.
Something like this:

   IANA is asked to make the following registration in the vCard
   Parameters registry.
   [https://www.iana.org/assignments/vcard-elements]

- If the document is creating a new registry, make sure it specifies
whether it should be a top-level registry, or a sub-registry of an
existing one.  Using vCard as an example again, something like this
works:

   IANA is asked to create a new sub-registry under vCard Elements.
   The new registry will be called "vCard Avatars", and registrations
   in it will be by the Specification Required policy [RFC5226].

- If the document is creating a new registry, make sure it specifies
the policy to be used for registrations in that registry.  We normally
specify one of the well-known policies defined in BCP 26 (currently
RFC 5226, though we're starting work on an update), and BCP 26 is a
normative reference (so check that too).

- Please include a brief explanation for the reason the selected
policy was chosen, and summarize working group discussion about that.
I would prefer to see that in the document itself, and you might ask
the authors to add it if there isn't already something there.  But if
there's a reason not to put it in the document, an explanation in the
shepherd writeup will do as a backup.  This doesn't have to be long,
but its importance increases with the strictness of the policy chosen:
"Standards Action", for example, is a heavyweight process that makes
new registrations difficult, and should include some justification for
that.  At the same time, though, "First Come First Served" could do
with a few words of assurance that no review is needed for this
registry.  I said "summarize working group discussion", and I'll
especially note that if there is no working group discussion there to
summarize, that should set warning bells off.  Registration policies
shouldn't just be picked out of a hat; there should be discussion,
even if it's brief, about whether the right one's been selected.  The
policy needs to balance the level of review and approval required with
the need to encourage registrations and the consequences of people's
using unregistered values.

--------------------------------

I'm sorry that this has been a bit long-winded, but these shepherd
writeups really do provide a lot of value when they're done with some
thought and care.  Thanks for reading, and I welcome feedback.  (And
here's a reminder that these are my thoughts, and don't necessarily
reflect the opinions of any other ADs.)

Barry, Applications AD

---

From: John C Klensin <[email-masked]>
Date: Tue, 24 Jul 2012 12:02:38 -0400

--On Sunday, July 15, 2012 11:42 -0400 Barry Leiba
<[email-masked]> wrote:

> Having had some time, now, to read quite a lot of document
> shepherd writeups (yes, we do read them, at least some of us),
> I want to give my thoughts on what makes them more useful to
> the IESG -- and, so, also, what doesn't help.  I'll start by
> saying that these are my personal thoughts, and not any sort
> of consensus of the IESG.
>...
 You can follow along with the writeup template; I
> won't repeat the questions here.

Having now tried to use it, that file is almost unusable as a
template.  As others pointed out over a month ago, it is a bit
hard to use from the standpoint of filling in information (quite
independent of the substantive issues addressed in your note).
When I retrieved it prior to today, I got a version with hard
double-spacing.

Since I've got several of these to do at once (and rather than
just complain), I've hacked together a XML template that
incorporates the document writeup material and that can be
easily filled in and used to generate readable output (or used
to generate a text file).

If the tools or xml2rfc teams think that such a thing might be
useful to others and want to post it somewhere, I'd be happy to
contribute it (I'll also accept individual email requests if
there aren't too many of them).

best,
   john

---

From: Barry Leiba <[email-masked]>
Date: Thu, 26 Jul 2012 01:51:28 -0400

> When I retrieved it prior to today, I got a version with hard

Ah.

The link from the IESG page gives you the text version, which is OK:
   http://www.ietf.org/iesg/template/doc-writeup.txt

The link from the WG Chairs page gives you the HTML version, which is bad:
   http://www.ietf.org/iesg/template/doc-writeup.html

I've put a ticket in to IT, asking them to fix the HTML page.

Apart from that, an XML version that could be processed with something
like XML2RFC might be nice, yes.

Barry

---

From: Leif Johansson <[email-masked]>
Date: Thu, 26 Jul 2012 11:44:59 +0200

> Apart from that, an XML version that could be processed with
> something like XML2RFC might be nice, yes.
> 

Or maybe just turn it into surveymonkey. Thats basically what it is: a
questionnaire.