Subject: Re: acknowledging reviewers better (was Re: Diversity considerations)

From: John C Klensin <[email-masked]>
Date: Tue, 02 Oct 2018 05:23:01 -0400

--On Monday, October 1, 2018 13:20 -0400 Michael Richardson
<[email-masked]> wrote:

>.,..
> My point is that we need to better acknowledge contributions
> in the form of review into our documents.  I pointed to the
> datatracker that keeps track of formal reviews, and that is
> definitely good data, but I feel that it needs to make into
> the document in a way that provides clear credit.
> 
> This is part of the diversity consideration because not
> everyone can contribute by writing documents, but they often
> can contribute through review.  Review is as important, maybe
> even more important, than the writing itself.
> 
> We do not, for instance, even have a clear tradition to
> acknowledge the WG chairs in the documents, which makes it
> hard for employers to agree to allocate valuable time that
> could (as Nico writes) be spend writing code, to instead
> manage process.
> 
> I'd like the acknowledgements to be structured (i.e. present
> in the XML as structured entries).

Michael,

Agreed, but note that, if you want such acknowledgments to be
useful for some of the purposes you and others have argued they
would help with, you may need to figure out how to differentiate
between a review that addresses the document substantively and
makes a contribution to its quality and one that has substantive
content equivalent to "I'm the designated Fubar team reviewer. I
know nothing about the subject matter of this document, but I
found a comma out of place in the middle of the second paragraph
of Section 6.6.6".

Speaking as someone who has always tried to acknowledge the
authors of review comments that are actually helpful and who
generally tries to resolve questions in which there is doubt
about whether an acknowledgment is needed by including it, I'm
not completely sure what the issue is.  As I read our IPR rules,
that behavior is just about required.  Or, put differently,
whether we need actual additional rules and procedures or just
more attention to general guidance.

Would you consider it useful if, following the example of many
journals, we simply posted a list of active "team" reviewers (or
all reviewers during IETF Last Call at regular (annual?)
intervals?  That at least would not require acknowledging
content-free reviews in the documents themselves, thereby
diminishing the perceived value of acknowledgements for actual
contributions.

   best,
     john

---

From: Michael Richardson <[email-masked]>
Date: Tue, 02 Oct 2018 09:01:46 -0400

John C Klensin <[email-masked]> wrote:
    > Agreed, but note that, if you want such acknowledgments to be useful
    > for some of the purposes you and others have argued they would help
    > with, you may need to figure out how to differentiate between a review
    > that addresses the document substantively and makes a contribution to
    > its quality and one that has substantive content equivalent to "I'm the
    > designated Fubar team reviewer. I know nothing about the subject matter
    > of this document, but I found a comma out of place in the middle of the
    > second paragraph of Section 6.6.6".

I agree that this is an issue, but I feel confident that good sense will
prevail.

    > Speaking as someone who has always tried to acknowledge the authors of
    > review comments that are actually helpful and who generally tries to
    > resolve questions in which there is doubt about whether an
    > acknowledgment is needed by including it, I'm not completely sure what
    > the issue is.  As I read our IPR rules, that behavior is just about
    > required.  Or, put differently, whether we need actual additional rules
    > and procedures or just more attention to general guidance.

I think that weneed more attention, and some additional proceedures that will
tweak people to pay more attention.

    > journals, we simply posted a list of active "team" reviewers (or all
    > reviewers during IETF Last Call at regular (annual?)  intervals?  That
    > at least would not require acknowledging content-free reviews in the
    > documents themselves, thereby diminishing the perceived value of
    > acknowledgements for actual contributions.

So, I think that many journals do this because the reviews are double blind.
They can not acknowledge reviewers in the document themselves, because that
would be unblinding one side.   But, I'm not sure that this really goes far
enough.  My impression is that tenure committees do not really look at the
list of people who have done reviews.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 2 Oct 2018 21:33:34 +0200

> found a comma out of place in the middle of the second paragraph

I got reviews about the quality of the language being bad, so i always
weelcome good language review feedback, and often there are more people
willing to do this in the broader IETF review than in a WG. I am curious
though how much of that language review should happen through the
formal IETF review or through RFC editor. The RFC editors are IMHO
(no offense to IETF community meant) degrees better on language
review than any IETF review i've seen. But of course i worry about what
is an appropriate low bar to pass on to them. I also felt bad about 
the language i passed on to them, so i compared the latest draft
version and final RFC from a couple of "native" english speakers RFCs,
and felt a lot better about my own documents afterwards (grin).

I think it will be quite common that specific areas will have no
reviewer available who is familiar with subject matter. I would
encourage those reviewers to not only help improve the language but also
not be afraid to ask "dumb question" when they do not understand the
document. In parenthesis, because i think there are no dumb questions.
Just badly written documents. Aka: non-experts can very well help
to improve the readability of documents by forcing authors/editors
to insert explanations that experts most likely will have thought to
be common sense. But IETF work gets more and more compartmentalized,
so that expectation is actually less and less true. 

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 2 Oct 2018 17:21:38 -0500

> and felt a lot better about my own documents afterwards (grin).

The RFC Editor is pretty cautious about making changes, certainly ones that
might change the technical meaning, but also for more broadly
rewording/reordering sentences or changing the organizational structure of
a document.  In some cases this broader sort of change might help a
document out, but even if the RPC staffer editing the document knows that
it would be better prose, the change will still not be proposed to the
authors as it's out of scope for their task.

It's also true that everyone suffers from "review fatigue" -- if there's a
lot of things to change, some will get missed in the time available.  More
nits will make it through to the final version if there are more in the
input to the process.  How important the final value is is a subjective
question, of course, and not necessarily even one that we need to achieve
IETF consensus on.

> so that expectation is actually less and less true. 

This is also true -- even in my IESG ballots I will frequently ask for
clarification on a point here and there.  I go in assuming that it's just
my confusion and lack of familiarity with the subject matter, but
occasionally the text is actually incorrect.  (And when I do ask for
clarification, most of the time I only want it in email, and not as a
change to the document.)

-Ben