Subject: Commit in docs/usefor (usepro.xml)

From: [email-masked]
Date: Sun, 3 Aug 2008 16:02:51 -0700 (PDT)

Date: Sunday, August 3, 2008 @ 16:02:50
  Author: eagle
Revision: 4226

Add a section describing the history mechanism and algorithm.  Update
references in Duties of an Injecting Agent and Duties of a Serving
Agent accordingly, moving the specifics of date-based rejection from
the Duties sections to the history section and making clearer the
distinction between the protocol requirement and the implications of
specific implementations satisfying that requirement.

Modified:
  docs/usefor/usepro.xml

Modified: docs/usefor/usepro.xml
===================================================================
--- docs/usefor/usepro.xml	2008-08-03 21:07:34 UTC (rev 4225)
+++ docs/usefor/usepro.xml	2008-08-03 23:02:50 UTC (rev 4226)
@@ -452,6 +452,66 @@
         </section>
       </section>
 
+      <section anchor="history"
+               title="Article History and Duplicate Suppression">
+        <t>Netnews normally uses a flood-fill algorithm for propagation of
+        articles in which each news server offers the articles it accepts
+        to multiple peers and each news server may be offered the same
+        article from multiple other news servers.  Accordingly, duplicate
+        suppression is key; if a news server accepted every article it was
+        offered, it may needlessly accept (and then potentially
+        retransmit) dozens of copies of every article.</t>
+
+        <t>Relaying and serving agents therefore MUST keep a record of
+        articles they have already seen and use that record to reject
+        additional offers of the same article.  This record is called the
+        "history" file or database.</t>
+
+        <t>Each article is uniquely identified by its message identifier,
+        so a relaying or serving agent could satisfy this requirement by
+        storing a record of every message identifier that agent has ever
+        seen.  Such a history database would grow without bound, however,
+        so it is common and permitted to optimize based on the
+        Injection-Date or Date header field of an article as follows.  (In
+        the following discussion, the "date" of an article is defined to
+        be the date represented by its Injection-Date header field if
+        present, otherwise its Date header field.)
+          <list style="symbols">
+            <t>Agents MAY select a cutoff interval and reject any article
+            with a date farther in the past than that cutoff interval.  If
+            this interval is shorter than the time it takes for an article
+            to propagate through the network, the agent might reject an
+            article it had not yet seen, so it ought not be aggressively
+            short.  For Usenet, for example, a cutoff interval of no less
+            than seven days is conventional.</t>
+
+            <t>Agents that enforce such a cutoff MAY then drop records of
+            articles that had dates older than the cutoff from their
+            history databases.  If such an article were offered to the
+            agent again, it would be rejected due to the cutoff date, so
+            the history record is no longer required to suppress the
+            duplicate.</t>
+
+            <t>Alternatively, agents MAY drop history records according to
+            the date when the article was first seen by that agent rather
+            than the date of the article.  In this case, the history
+            retention interval MUST be at least 24 hours longer than the
+            cutoff interval to allow for articles dated in the future.
+            This interval matches the allowable error in the date of the
+            article (see <xref target="injecting" />).</t>
+          </list>
+        </t>
+
+        <t>These are just two implementation strategies for article
+        history, albeit the most common ones.  Relaying and serving agents
+        are not required to use these strategies, only to meet the
+        requirement of not accepting an article more than once.  However,
+        these strategies are safe and widely deployed and implementors are
+        encouraged to use one of them, especially if they do not have
+        extensive experience with Netnews and the subtle effects of its
+        flood-fill algorithm.</t>
+      </section>
+
       <section anchor="posting" title="Duties of a Posting Agent">
         <t>A posting agent is the component of a user agent that assists a
         poster in creating a valid proto-article and forwarding it to an
@@ -806,19 +866,19 @@
             field or Message-ID header field, or without either an
             Injection-Date or Date header field.</t>
 
-            <t>It MUST reject any article that has already been
-            successfully sent to it, based on the Message-ID header field
-            of the article.  To satisfy this requirement, a relaying agent
-            normally keeps a database of message identifiers it has
-            already accepted.</t>
-
             <t>It MUST examine the Injection-Date header field or, if
             absent, the Date header field, and reject the article if that
-            date predates the earliest articles of which it keeps record
-            or if that date is more than 24 hours into the future.  It MAY
-            reject articles with dates in the future with a smaller margin
-            than 24 hours.</t>
+            date is more than 24 hours into the future.  It MAY reject
+            articles with dates in the future with a smaller margin than
+            24 hours.</t>
 
+            <t>It MUST reject any article that has already been accepted.
+            If it implements one of the mechanisms described in <xref
+            target="history" />, this means that it MUST reject any
+            article whose date falls outside the cutoff interval since it
+            won't know whether such articles had been accepted previously
+            or not.</t>
+
             <t>It SHOULD reject any article that does not include all the
             mandatory header fields.  It MAY reject any article that
             contains header fields that do not have valid contents.</t>
@@ -891,16 +951,16 @@
 
             <t>It MUST examine the Injection-Date header field or, if
             absent, the Date header field, and reject the article if that
-            date predates the earliest articles of which it keeps record
-            or if that date is more than 24 hours into the future.  It MAY
-            reject articles with dates in the future with a smaller margin
-            than 24 hours.</t>
+            date is more than 24 hours into the future.  It MAY reject
+            articles with dates in the future with a smaller margin than
+            24 hours.</t>
 
-            <t>It MUST reject any article that has already been
-            successfully sent to it, based on the Message-ID header field
-            of the article.  To satisfy this requirement, a relaying agent
-            normally keeps a database of message identifiers it has
-            already accepted.</t>
+            <t>It MUST reject any article that has already been accepted.
+            If it implements one of the mechanisms described in <xref
+            target="history" />, this means that it MUST reject any
+            article whose date falls outside the cutoff interval since it
+            won't know whether such articles had been accepted previously
+            or not.</t>
 
             <t>It SHOULD reject any article that matches an
             already-received and honored cancel message or Supersedes