Subject: Re: #1416

From: Russ Allbery <[email-masked]>
Date: Sun, 03 Aug 2008 16:37:43 -0700

"Charles Lindsey" <[email-masked]> writes:

> That left just one small worry. I wanted (version IC) Injection Agents
> to _always_ insert an Injection-Date (except when it was already
> present, of course). Russ argued (version IR) that this would sometimes
> cause existing implementations to behave oddly, and proposed a
> less-intuitive rule for when Injection Agents should insert it.

I think there's also a remaining point of disagreement over whether
injection agents SHOULD reject articles with a stale Date.

I just submitted draft-ietf-usefor-usepro-10 including the (I believe
non-controversial) new history section and corresponding rewordings in the
duties of a relaying and serving agent.  Below is the remaining diff for
my solution for #1416.  Please note if any of the below is uncontroversial
so that I can commit those sections and reduce the diff to only the
disputed portion.

--- usepro.xml	2008-08-03 16:12:51.000000000 -0700
+++ usepro-1416.xml	2008-08-03 16:34:50.000000000 -0700
@@ -18,6 +18,8 @@
     'http://xml.resource.org/public/rfc/bibxml/reference.RFC.2822.xml'>
   <!ENTITY rfc3629 PUBLIC '' 
     'http://xml.resource.org/public/rfc/bibxml/reference.RFC.3629.xml'>
+  <!ENTITY rfc3798 PUBLIC '' 
+    'http://xml.resource.org/public/rfc/bibxml/reference.RFC.3798.xml'>
   <!ENTITY rfc3977 PUBLIC '' 
     'http://xml.resource.org/public/rfc/bibxml/reference.RFC.3977.xml'>
   <!ENTITY rfc4234 PUBLIC '' 
@@ -165,8 +167,9 @@
 
         <t>"Injecting" an article is the processing of a proto-article by
         an injecting agent.  Normally this action is done once and only
-        once for a given article.  "Reinjecting" an article is passing an
-        already-injected article to an injection agent.</t>
+        once for a given article.  "Multiple injection" is passing the
+        same article to multiple injecting agents, either serially or in
+        parallel, by one or several posting agents.</t>
 
         <t>A "gateway" is software which receives news articles and
         converts them to messages of some other kind (such as <xref
@@ -519,9 +522,34 @@
 
         <t>Posting agents SHOULD ensure that proto-articles they create
         are valid according to <xref target="USEFOR" /> and any other
-        applicable policies.  They MUST NOT create any Injection-Date or
-        Injection-Info header fields; these headers will be added by the
-        injecting agent.</t>
+        applicable policies.  They MUST NOT create any Injection-Info
+        header field; this header field may only be added by the injecting
+        agent.</t>
+
+        <t>If the proto-article already contains both Message-ID and Date
+        header fields, posting agents MAY add an Injection-Date header
+        field to that proto-article immediately before passing that
+        proto-article to an injection agent.  They SHOULD do so if the
+        Date header field (representing the composition time of the
+        proto-article) is more than a day in the past at the time of
+        injection.  They MUST do so if the proto-article is being
+        submitted to more than one injecting agent; see <xref
+        target="multi-injection" />.</t>
+
+        <t>The Injection-Date header field is new in this revision of the
+        Netnews protocol and is designed to allow the Date header field to
+        hold the composition date (as recommended in section 3.6.1 of
+        <xref target="RFC2822" />), even if the proto-article is not to be
+        injected for some time after its composition.  However, note that
+        all implementations predating this specification ignore the
+        Injection-Date header field and use the Date header field in its
+        stead for rejecting articles older than their cutoff (see <xref
+        target="history" />), and injecting agents predating this
+        specification do not add an Injection-Date header.  Articles with
+        a Date header field substantially in the past will still be
+        rejected by implementations predating this specification,
+        regardless of the Injection-Date header field, and hence may
+        suffer poorer propagation.</t>
 
         <t>Contrary to <xref target="RFC2822" />, which implies that the
         mailbox or mailboxes in the From header field should be that of
@@ -544,48 +572,75 @@
           agent.</t>
 
           <t>A proto-article has the same format as a normal article
-          except that the Injection-Date, Injection-Info, and Xref header
-          fields MUST NOT be present; the Path header field MUST NOT
-          contain a "POSTED" &lt;diag-keyword>; and any of the following
-          mandatory header fields MAY be omitted: Message-ID, Date, and
-          Path.  In all other respects, a proto-article MUST be a valid
-          Netnews article.  In particular, the header fields which may be
-          omitted MUST NOT be present with invalid content.</t>
+          except that the Injection-Info and Xref header fields MUST NOT
+          be present; the Path header field MUST NOT contain a "POSTED"
+          &lt;diag-keyword>; and any of the following mandatory header
+          fields MAY be omitted: Message-ID, Date, and Path.  In all other
+          respects, a proto-article MUST be a valid Netnews article.  In
+          particular, the header fields which may be omitted MUST NOT be
+          present with invalid content.</t>
 
           <t>If a posting agent intends to offer the same proto-article to
-          multiple injecting agents, the header fields Message-ID and Date
-          MUST be present and identical in all copies of the
-          proto-article.</t>
+          multiple injecting agents, the header fields Message-ID, Date,
+          and Injection-Date MUST be present and identical in all copies
+          of the proto-article.  See <xref target="multi-injection" />.</t>
         </section>
 
-        <section anchor="reinjection" title="Reinjection of Articles">
-          <t>A given article SHOULD be processed by an injecting agent
-          once and only once.  The Injection-Date or Injection-Info
-          header fields are added by an injecting agent and are not
-          permitted in a proto-article.  Their presence (or the presence
-          of other unstandardized or obsolete trace headers such as
-          NNTP-Posting-Host, NNTP-Posting-Date, or X-Trace) indicates
-          that the proto-article is instead an article and has already
-          been processed by an injecting agent.  A posting agent SHOULD
-          normally reject such articles.</t>
-
-          <t>In the exceptional case that an article needs to be
-          reinjected for some reason (such as transferring an article from
-          one Netnews to another where those networks have no relaying
-          agreement), the posting agent doing the reinjection MUST convert
-          the article back into a proto-article before passing it to an
-          injecting agent (such as by renaming the Injection-Info and
-          Injection-Date header fields and removing any Xref header field)
-          and MUST perform the date checks on the existing Injection-Date
-          or Date header fields that would otherwise be done by the
-          injecting agent.</t>
-
-          <t>Reinjecting articles may cause loops, loss of trace
-          information, and other problems and should only be done with
-          care and when there is no available alternative.  A posting
-          agent that does reinjection is a limited type of gateway and as
-          such is subject to all of the requirements of an incoming
-          gateway in addition to the requirements of a posting agent.</t>
+        <section anchor="multi-injection"
+                 title="Multiple Injection of Articles">
+          <t>Under some circumstances (posting to multiple disjoint
+          networks, injecting agents with spotty connectivity, or for
+          redundancy, for example), a posting agent may wish to offer the
+          same article to multiple injecting agents.  In this unusual
+          case, the goal is to not create multiple independent articles
+          but rather to inject the same article at multiple points and let
+          the normal duplicate suppression facility of Netnews (see <xref
+          target="history" />) ensure that any given agent accepts the
+          article only once, even if supposedly disjoint networks have
+          unexpected links.</t>
+
+          <t>Whenever possible, multiple injection SHOULD be done by
+          offering the same proto-article to multiple injecting agents.
+          The posting agent MUST supply the Message-ID, Date, and
+          Injection-Date header fields, and the proto-article as offered
+          to each injecting agent MUST be identical.</t>
+
+          <t>In some cases, offering the same proto-article to all
+          injecting agents may not be possible (such as when gatewaying,
+          after the fact, articles found on one Netnews network to
+          another, supposedly unconnected one).  In this case, the posting
+          agent MUST convert the article back into a proto-article before
+          passing it to another injecting agent, but it MUST retain
+          unmodified the Message-ID, Date, and Injection-Date header
+          fields.  It MUST NOT add an Injection-Date header field if it is
+          missing from the existing article.  It MUST remove any Xref
+          header field and either rename or remove any Injection-Info
+          header field and other trace fields.
+            <list style="empty">
+              <t>NOTE: Multiple injection inherently risks duplicating
+              articles.  Multiple injection after the fact, by converting
+              an article back to a proto-article and injecting it again,
+              additionally risks loops, loss of trace information,
+              unintended repeat injection into the same network, and other
+              problems.  It should be done with care and only when there
+              is no alternative.  The requirement to retain Message-ID,
+              Date, and Injection-Date header fields minimizes the
+              possibility of a loop and ensures that the newly injected
+              article is not treated as a new, separate article.</t>
+            </list>
+          </t>
+
+          <t>Multiple injection of an article listing one or more
+          moderated newsgroups in its Newsgroups header field SHOULD only
+          be done by a moderator and MUST only be done after the
+          proto-article has been approved for all moderated groups to
+          which it is to be posted and has an Approved header field (see
+          <xref target="moderator" />).  Multiple injection of an
+          unapproved article intended for moderated newsgroups will
+          normally only result in the moderator receiving multiple copies,
+          and if the newsgroup status is not consistent across all
+          injecting agents, may result in duplication of the article or
+          other problems.</t>
         </section>
 
         <section anchor="followups" title="Followups">
@@ -710,23 +765,27 @@
 
             <t>It MUST reject any proto-article that does not have the
             proper mandatory header fields for a proto-article; that has
-            Injection-Date, Injection-Info, or Xref header fields; that
-            has a Path header field containing the "POSTED"
-            &lt;diag-keyword>; or that is not syntactically valid as
-            defined by <xref target="USEFOR" />.  It SHOULD reject any
-            proto-article which contains a header field deprecated for
-            Netnews.  It MAY reject any proto-article that contains trace
-            header fields indicating that it was already injected by an
-            injecting agent that did not add Injection-Info or
-            Injection-Date.</t>
-
-            <t>It SHOULD reject any article whose Date header field is
-            more than 24 hours into the future (and MAY use a margin less
-            than 24 hours).  It SHOULD reject any article whose Date
-            header appears to be stale (more than 72 hours into the past,
-            for example, or too old to still be recorded in the database
-            of a relaying agent the injecting agent will be using) since
-            not all news servers support Injection-Date.</t>
+            Injection-Info or Xref header fields; that has a Path header
+            field containing the "POSTED" &lt;diag-keyword>; or that is
+            not syntactically valid as defined by <xref target="USEFOR"
+            />.  It SHOULD reject any proto-article which contains a
+            header field deprecated for Netnews (see, for example, <xref
+            target="RFC3798" />).  It MAY reject any proto-article that
+            contains trace header fields (e.g., NNTP-Posting-Host)
+            indicating that it was already injected by an injecting agent
+            that did not add Injection-Info or Injection-Date.</t>
+
+            <t>It SHOULD reject any article whose Injection-Date or Date
+            header field is more than 24 hours into the future (and MAY
+            use a margin less than 24 hours).  It SHOULD reject any
+            article whose Injection-Date header field is too far in the
+            past (older than the cutoff interval of a relaying agent the
+            injecting agent is using, for example).  It SHOULD similarly
+            reject any article whose Date header field is too far in the
+            past, since not all news servers support Injection-Date and
+            only the injecting agent can provide a useful error message to
+            the posting agent.  In either case, this interval SHOULD NOT
+            be any shorter than 72 hours into the past.</t>
 
             <t>It SHOULD reject any proto-article whose Newsgroups header
             field does not contain at least one &lt;newsgroup-name> for a
@@ -770,8 +829,14 @@
             the source of the article and possibly other trace information
             as described in Section 3.2.8 of <xref target="USEFOR" />.</t>
 
-            <t>The injecting agent MUST then add an Injection-Date header
-            field containing the current date and time.</t>
+            <t>If the proto-article already had an Injection-Date header
+            field, it MUST NOT be modified or replaced.  If the
+            proto-article had both a Message-ID header field and a Date
+            header field, an Injection-Date header field MUST NOT be
+            added, since the proto-article may have been multiply injected
+            by a posting agent that predates this standard.  Otherwise,
+            the injecting agent MUST add an Injection-Date header field
+            containing the current date and time.</t>
 
             <t>Finally, the injecting agent forwards the article to one or
             more relaying agents, and the injection process is
@@ -1068,8 +1133,7 @@
             for reasons understood by the moderator (such as delays in the
             moderation process) in which case they MAY substitute the
             current date.  Any Injection-Date, Injection-Info, or Xref
-            header fields already present (though there should be none)
-            MUST be removed.</t>
+            header fields already present MUST be removed.</t>
 
             <t>Any Path header field MUST either be removed or truncated
             to only those entries following its "POSTED"
@@ -2109,6 +2173,7 @@
       &rfc1036;
       &rfc2045;
       &rfc2606;
+      &rfc3798;
       &rfc3977;
       <reference anchor="USEAGE">
         <front>

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 4 Aug 2008 14:08:49 GMT

>so that I can commit those sections and reduce the diff to only the

90% of what follows is agreed, and it might be helpful to get it into an
ID so that we can see it all in context, and then argue over the small
remaining difference. Such concerns as I have with the rest of it are all
in the "niggles" category.

>             field does not contain at least one &lt;newsgroup-name> for a

That is the principle point of difference between IR and IC. I would
prefer it to say

>+            field, it MUST NOT be modified or replaced.  Otherwise,

If that change is made, there are probably a few consequential changes
and niggles elsewhere, but that is the essence of it.

In that case, there is a small possibility that, until all agents have
been upgraded, a few users may occasionally see an article they have seen
before. See <http://www.imc.org/ietf-usefor/drafts/issue-1416> for the
somewhat complicated chain of events that can give rise to this.

OTOH, without my change, most articles will never receive an Injection-Date
at all, and that will continue to be the case undefinitely (unless Posting
agents routinely start to do it, which I consider unlikely).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 05 Aug 2008 12:00:53 +0100

> +            containing the current date and time.</t>
This ("if the prot-article had both a Message-ID header field and a Date 
header field") is the place that bothers me most, since it will mean 
that singly injected articles produced by "modern" UAs that generate 
their own message-IDs will never get an Injection-Date header field.

It seems to sacrifice a general goal (that messages should have 
Injection-Date) for a narrow benefit (that multiple copies of multiply 
injected articles that are injected over a large span of time won't be 
detected as duplicates by servers that check Injection-Date and ignore 
Date).

I'm not sure that's the right tradeoff.

                        Harald

---

From: Russ Allbery <[email-masked]>
Date: Wed, 06 Aug 2008 14:54:55 -0700

> their own message-IDs will never get an Injection-Date header field.

Yes, this is the point of disagreement.

> I'm not sure that's the right tradeoff.

This is the topic that we've gone back and forth on at some length.  To
summarize my opinion:

* I'm not sure that having all messages have Injection-Date header fields
  should be a general goal of this work.  The header is primarily useful
  when Date is well in the past and not that useful otherwise.  It's
  certainly fine if we get there, but I don't think getting there is a
  high priority.

* The Injection-Date feature will not realistically be able to be used to
  allow for articles with very old Date headers for the forseeable future,
  possibly ever.  It will require upgrading all servers to use
  Injection-Date rather than Date, which I think is very unlikely to
  happen.  Accordingly, the benefit of servers adding Injection-Date is
  fairly minor for the forseeable future; the only effect will be somewhat
  more correct behavior with articles with borderline Dates.

* Given the very slow development pace of netnews software, I think
  backward compatibility should be our highest goal.

I'm unlikely to change my mind in this area, but I'm happy to be
overridden by the working group if other people just don't agree with me.
If that happens, I'll gladly prepare a document that implements Charles's
approach.

I think a working group chair needs to decide if that's happened.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 7 Aug 2008 10:44:17 GMT

>  backward compatibility should be our highest goal.

Not surprisingly, I take an opposite view on all of the above.

>I think a working group chair needs to decide if that's happened.

Agreed. Clearly Russ's vote for IR is cancelled out by my vote for IC, so
it is the rest of the WG that gets to make the choice. But we need to get
that choice made so that we can move on.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Thu, 07 Aug 2008 08:20:57 -0400

> that choice made so that we can move on.

I'm with Russ, if that matters to the chair.

I have no idea why every vote counts equally here, for two reasons.

1. Russ has experience writing and maintaining news server software for a 
production environment.  I do too.  Others here do too.

2. Technical merit matters.  I agree completely with Russ's statements
about how rapidly software will get changed in production.  There is
lots of evidence for that in the last decade.  I believe that
the choices in what you call "IR" are appropriate for that.

---

From: Dan Schlitt <[email-masked]>
Date: Thu, 7 Aug 2008 09:48:30 -0400

I also think that Russ has the stronger position.

/dan

-- 

Dan Schlitt
[email-masked]

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 7 Aug 2008 18:18:17 +0200

Charles Lindsey wrote:

> Russ's vote for IR is cancelled out by my vote for IC

My enthusiasm for voting in technical questions is limited.

Harald noted elsewhere that the idea of humming is to find
rough consensus or its absence without counting.  Here with
the very limited participation it is anyway not compelling.

But I do think that the Injection-Date in the approved RFC
represents a *possible* and plausible concept.  Overruling
that older IETF consensus because the concept is actually
not as possible as the WG hoped after years of discussions
is IMO a bad idea, and nobody is going to modify this older
decision anytime soon, let alone in the next nineteen days.

Even if Injection-Date as intended turns out to be wishful
thinking in practice.  There is an "official" procedure to
remove features from standards:  This can be done when the
future implementation and interop reports for a promotion
on "standards track" show that this feature didn't get the
traction justifying to keep it as intended.

IMO usefor-usepro should reflect the usefor-usefor design,
and not try a roll-back.  After some time - not less than
six months, but likely years, if at all - folks with the
energy to promote these two documents to "draft standard"
can fix whatever needs fixing (*1).  

What is most important now is to get a "consistent story"
out (as RFCs with numbers, not rotting in an editor queue)
before the WG is terminated in about 19*24 hours.

 Frank
-- 
*1: There will be fixes, if anybody tries to promote the
    documents, e.g., a simplified <top-label> TBD "soon",
    and the simplified <msg-id-core> of 2822upd as "soon"
    as 2822upd can be approved and published.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 8 Aug 2008 12:00:25 GMT

>the very limited participation it is anyway not compelling.

Voting, or rather "counting heads" will give an indication of rough
consensus if there is a substantial majority one way or the other. If it
is evenly balanced, then you are still stuck of course.

But in this caase I think both Russ and I would be willing to accept a
majority view if it was clear we had one.

>represents a *possible* and plausible concept.

Sure, but nobody is suggesting abandoning Injection-Date totally - it is
just a matter of who is supposed to add it and when. As I just explained
to Lisa, even Russ's version of #1416 is better than no version at all.

It's just that I think Injection-Date will catch on more rapidly if we
allow Injecting Agents to add it more-or-less always, in spite of the very
rare risk that an occasional user might see the same article twice.

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 8 Aug 2008 19:40:41 +0200

Charles Lindsey wrote:

> in this caase I think both Russ and I would be willing
> to accept a majority view if it was clear we had one.

Good, if there is no clear trend, but most still agree
with "anything is better than nothing" we could flip a
coin.  

With two "candidates" (1 = Russ', 2 = your version) and
the NomCom 2008 entropy sources you'd get either 1 or 2.

Fortunately there is no time to revisit old rat-holes,
otherwise I'd wonder why Russ wrote that a Message-ID
is "more unique" when extended by a timestamp.

 Frank

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 08 Aug 2008 14:16:36 -0400

> is "more unique" when extended by a timestamp.

That is because the server's record of message IDs is a
finite list in practice.

No one should offer an opinion on how to resolve #1416
unless and until they understand this point.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 08 Aug 2008 11:19:33 -0700

> with "anything is better than nothing" we could flip a

I agree that anything is better than nothing.

> is "more unique" when extended by a timestamp.

I'm fairly sure that I didn't say that, and a quick grep of my e-mail
reveals that I have not used the phrase "more unique" in any e-mail I have
sent for the past month and a half.  I would explain (again) why the Date
is part of article uniqueness in the USEPRO protocol, but I've done so
multiple times before, I think the draft covers it, and I think you
actually understand it and are just needling me out of some sense of
personal amusement.  (That amusement, for the record, isn't shared.)

We've discussed how article uniqueness works in netnews repeatedly; please
don't take cheap shots at a simplified summary.  If you don't think we
have time for sort of nitpicking, then actually don't do it.

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 8 Aug 2008 21:47:16 +0200

Russ Allbery wrote:

> I'm fairly sure that I didn't say that

| Briefly: the Message-ID plus the Date is
| the unique identifier for the message

>From my POV you wrote this in a message
with a globally unique forever identifier
<[email-masked]>

That this message also had a
| Date: Thu, 07 Aug 2008 18:31:05 -0700
is important for a history of "seen unique
identifiers", but the identifier alone is
supposed to be unique.  

This identifier cannot be used for another
message, no matter what its Date might be.

Similar the Message-ID 
<[email-masked]>
is considered as "globally unique forever".

Using this identifier with a date different
from Date: 2 Dec 93 03:20:57 GMT for another
message is not allowed.

> I think you actually understand it

I'd hope that I understood this concept when
folks explained it to me about 13 years ago,
down to the s-o-1036 fine print of "forever".

> just needling me out of some sense of

The last time anything on this list amused me
was years ago when Bruce "forced" me to read
dozens of old RFCs I've never before heard of.

 Frank

---

From: Ralph Babel <[email-masked]>
Date: Sat, 9 Aug 2008 23:08:52 +0200 (CEST)

Charles Lindsey wrote:

> But in this caase I think [...] I would be willing

Accept a majority?

You?

Since when?

> nobody is suggesting abandoning Injection-Date totally

I am.

Usenet does not change because of an RFC. Never has,
never will. It may well die, but it's not gonna change,
regardless of how many RFCs will be published.

A single line of code written by Russ most likely has
a deeper impact on Usenet than anything this WG could
ever come up with.

It's dead, Jim.

And it doesn't even smell funny any more.

> It's just that I think Injection-Date
> will catch on more rapidly if [...]

You're naive.

Even Google abandoned their spaces-in-"Newsgroups:"
experiment just a few months ago, and that was a
relatively minor change proposed _fifteen_ years ago.