Subject: what to do about missing cites and encouraging better citing

From: Randy Bush <[email-masked]>
Date: Sun, 24 Jun 2018 15:29:35 -0700

[ we need a new rat-hole as relief from pico-managing the selection of a
  board on which no one sane would want to be ]

this is meant to raise consciousness, not cause process-creation or
other complexity.
 
over in the research universe, (most; hi dan) folk are careful about
citing prior work in the area, especially if it is seminal.  we seem to
be somewhat sloppy over here in standards wonk land; and i am not
entirely comfortable with this.

but can we do anything when we find a missing cite?

to have an example, and not meaning to pick on anyone, rfc 7747 uses the
methods of, but does not cite, the 2001 seminal work in the area of
routing convergence measurement by shaikh and greenberg,
http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.910.8379

yes, it is somewhat painful when the cite would be to a non-rfc, as our
external citing mechanisms are clumsy.

i assume a missing cite is not an erratum.  so maybe nothing can be done
other than to encourage authors to be more careful.  in general, the
ietf has a weak culture of good citing.  and i guess i could/should have
flagged this particular example in a last call <blush>.

randy

---

From: Benjamin Kaduk <[email-masked]>
Date: Sun, 24 Jun 2018 17:57:01 -0500

> other than to encourage authors to be more careful.  in general, the

I tend to agree with that assumption, but am not sure that nothing at all
can be done.  There may be room for something analogous to an academic
review paper that covers the history of a given field and cites all the
known relevant work, as a way to tie them together.  Though this would not
qualify for standards-track (and would make using an Updates: header
difficult), one could perhaps make an argument for BCP.  Even Informational
would probably still have value, though.

-Ben

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 25 Jun 2018 11:16:52 +1200

On 25/06/2018 10:29, Randy Bush wrote:
...> i assume a missing cite is not an erratum.  so maybe nothing can be done
> ietf has a weak culture of good citing. 

Yes. I mentioned this briefly in
https://tools.ietf.org/html/draft-carpenter-whats-an-author-02#section-2
but didn't detect great enthusiasm in the community for dealing with
authorship ethics in general.

   Brian

---

From: Randy Bush <[email-masked]>
Date: Sun, 24 Jun 2018 17:07:28 -0700

> I tend to agree with that assumption, but am not sure that nothing at
> all can be done.  There may be room for something analogous to an
> academic review paper that covers the history of a given field and
> cites all the known relevant work, as a way to tie them together.
> Though this would not qualify for standards-track (and would make
> using an Updates: header difficult), one could perhaps make an
> argument for BCP.  Even Informational would probably still have value,
> though.

that would be an academic work, and there are no real world brownie
points for publishing an rfc.

randy

---

From: Randy Bush <[email-masked]>
Date: Sun, 24 Jun 2018 17:14:40 -0700

> authorship ethics in general.

how embarrassing and shameful

randy

---

From: John C Klensin <[email-masked]>
Date: Mon, 25 Jun 2018 03:52:16 -0400

--On Sunday, June 24, 2018 17:14 -0700 Randy Bush
<[email-masked]> wrote:

>> but didn't detect great enthusiasm in the community for
>> dealing with authorship ethics in general.
> 

Randy,

While I agree that this is embarrassing and shameful (and could
add other words), I question one aspect of your earlier note. 

At least until the IAB decides to micromanage that, an erratum
and its appropriateness is determined by the discretion of the
RFC Series Editor.  The term is arguably not precisely correct
for the way we use the things and I've seen many of them whose
subject matter or content would not be appropriate if the
classic publisher definition were used.  

So I would recommend a conversation with Heather (copied) and
use of the errata mechanism, at least unless she strongly
disagrees and wants to propose another mechanism.  That would at
least get citation on the record.  While I'm not optimistic, if
author(s) are shamed sufficiently to not omit such citations in
the future, that would be A Good Thing.  I think all of us
should be encouraged to watch for obvious omissions on Last Call
and would encourage document shepherds and responsible ADs (in
the IETF Stream) and reviewers (in others) to pay attention to
the issue.

I would suggest to Heather that a mention of this as the Style
Guide evolves would probably also be a good thing -- it isn't as
if adding relevant citations is difficult or costly.

We will probably never get everything right, but we can try
harder.

best,
   john

---

From: "Heather Flanagan (RFC Series Editor)" <[email-masked]>
Date: Tue, 26 Jun 2018 13:24:12 -0700

> the IETF Stream) and reviewers (in others) to pay attention to

Heather does not disagree - the errata system is the best mechanism we 
have at this time, and it does allow an author to respond as needed.

> if adding relevant citations is difficult or costly.

This is going to be a bit trickier. I'm supportive of adding something 
about author ethics, but it would be along the lines of "the RFC Editor 
will not be checking on this; you should do this because it's the right 
thing." We don't have the resources to do a full-on review of the 
literature; if the IETF wants that as a priority, we'll need to talk 
about costs and/or reprioritizing other RFC Editor activities.

I think a note on the IETF norms for referencing other RFCs to be a good 
thing to include in the Tao.

-Heather

---

From: "Eggert, Lars" <[email-masked]>
Date: Tue, 26 Jun 2018 21:18:23 +0000

On 2018-6-25, at 0:29, Randy Bush <[email-masked]> wrote:
> the 2001 seminal work in the area of

Actually, there is a way with some of the newer ways to write I-Ds to cite by DOI and have that pull in all the metadata, which makes this way less painful. (The details elude me at the moment, however.)

Lars

---

From: John C Klensin <[email-masked]>
Date: Tue, 26 Jun 2018 18:04:15 -0400

--On Tuesday, June 26, 2018 13:24 -0700 "Heather Flanagan (RFC
Series Editor)" <[email-masked]> wrote:

>> Guide evolves would probably also be a good thing -- it isn't
>> as if adding relevant citations is difficult or costly.
> 
> This is going to be a bit trickier. I'm supportive of adding
> something about author ethics, but it would be along the lines
> of "the RFC Editor will not be checking on this; you should do
> this because it's the right thing." We don't have the
> resources to do a full-on review of the literature; if the
> IETF wants that as a priority, we'll need to talk about costs
> and/or reprioritizing other RFC Editor activities.

That is consistent with what I had in mind.  On the other hand,
if a missing citation is noticed and commented on during Last
Call and the IESG decides to let the document reach you and the
Production Center without the issue being addressed, I think
that would be ample ground for an appeal of the document action.
 
> I think a note on the IETF norms for referencing other RFCs to
> be a good thing to include in the Tao.

Yes, although Randy's main point, at least as I understood it,
was that authors should be citing important papers in the
external literature that provide the foundations for whatever is
being done in proposed RFCs.   "Don't pretend, even by omission,
to have invented ideas that were not yours" would also be a good
comment for the Tao and/or assorted documents on appropriate
behavior.

best,
   john

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 27 Jun 2018 00:24:52 +0200

> external citing mechanisms are clumsy.

If the external paper to be cited has a DOI (as in the above example), the work reduces to adding a sentence:

  This approach was first described in {{?DOI.10.1145_637201.637236}}.

That is not that much work to do.  

In this case, finding the actual DOI — the citeseer thing is not a DOI — is a quick search engine search.

Adding a bibliography line of the form

  SHAIKH2002: DOI.10.1145_637201.637236

to the YAML header, so you can then simply reference it as [SHAIKH2002], is optional.

Grüße, Carsten

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 Jun 2018 10:33:07 +1200

> comment for the Tao and/or assorted documents on appropriate

Quite apart from the ethical aspect, such citations would help
anyone needing to do a prior art search.

    Brian

---

From: John C Klensin <[email-masked]>
Date: Tue, 26 Jun 2018 19:59:38 -0400

--On Wednesday, June 27, 2018 10:33 +1200 Brian E Carpenter
<[email-masked]> wrote:

>> external literature that provide the foundations for whatever
>> is being done in proposed RFCs.   "Don't pretend, even by
>> omission, to have invented ideas that were not yours" would
>> also be a good comment for the Tao and/or assorted documents
>> on appropriate behavior.
> 

And, of course, if we ever have a resurgence of the discussions
some years ago about making the RFC Series more academically
credible, getting those citations in is critical.   So, yes,
lots of reasons including, but not limited to, the ones that
Randy identified.

     john

---

From: Michael StJohns <[email-masked]>
Date: Tue, 26 Jun 2018 22:02:09 -0400

> lots of reasons including, but not limited to, the ones that

I don't disagree, but as Heather pointed out,  there's a cost to 
everything, either paid for in cash via the IETF/ISOC coffers, or in the 
sweat of volunteers.    As an adult organization (which is sort of 
different from an organization of adults), when identifying places where 
we want change, we really should be in the business of figuring how to 
pay for those changes (or at least what we're willing to pay for most) 
rather than just assuming there's zero cost.

To resolve the current identified issue on the cheap - perhaps one of 
the interested parties would care to author a BCP on Plagarism, Source 
Credits and Cites  that the IETF could adopt and refer to during 
document shepherding and last call?

Let me raise a slightly different but related point:  The IETF is 
currently in the business of creating standards, not academic 
publications.  If we want that to change, the IETF will fundamentally 
have to change.    I'm not sure about this, but it seems to me there is 
an upswing in the number of IDs and documents only peripherally related 
to the IETF's standardization process. Some are related to the IRTF 
process, some are semi-academic analyses of IETF related protocols and 
processes, some just seem to be dropped into the ID process just to have 
some place to put them.  It's still a pretty small set, but it *feels* 
like it's growing and I sometimes wonder if the trend will lead the IETF 
to a place the IETF is not prepared to go.

Later,  Mike

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 Jun 2018 14:35:09 +1200

> document shepherding and last call?

I feel I'm tooting my own trombone, but that is one of the points
made in my draft cited a few messages ago.

> have to change.

Another toot: Craig Partridge and I did tackle that topic
in CCR a few years ago. Our view was that the RFC review process
is at least as good as those of most journals and conferences,
so we weren't arguing for change, just for recognition.
(https://dl.acm.org/citation.cfm?id=1672315)

>    I'm not sure about this, but it seems to me there is 

I don't think you're referring to IETF stream documents, but certainly
IRTF and Independent stream documents might be of that nature. I don't
think the numbers are large, though, and the IETF has no exclusivity
over the RFC series.

Regards
   Brian

---

From: Randy Bush <[email-masked]>
Date: Tue, 26 Jun 2018 21:28:54 -0700

> about author ethics, but it would be along the lines of "the RFC
> Editor will not be checking on this; you should do this because it's
> the right thing."

seems sensible

> I think a note on the IETF norms for referencing other RFCs to be a
> good thing to include in the Tao.

it's not just cites to rfcs.  in the example i used, it was an academic
paper.

randy