Subject: Re: Niggles in usepro-draft-11

From: Russ Allbery <[email-masked]>
Date: Sun, 21 Sep 2008 18:29:16 -0700

"Charles Lindsey" <[email-masked]> writes:

> 3.2.1.  Constructing the Path Header Field
>
>    ........  Note that the Path header
>    field content is constructed from right to left by prepending
>    elements.
>
>    1.  ........
>
>    2.  ........
>
>    3.  A relaying or serving agent SHOULD prepend a <path-diagnostic> to
>        the Path header field content, where the <path-diagnostic> is
>        chosen as follows:
>
>        *  If the expected <path-identity> of the source of the article
>           matches the leftmost <path-identity> of the Path header
>           field's content, use "!" (<diag-match>), resulting in two
>           consecutive "!"s.
>
>        *  If the expected <path-identity> of the source of the article
>           does not match, use "!.MISMATCH." followed by the expected
>           <path-identity> of the source or its IP address.
>
>        *  If the relaying or serving agent is not willing or able to
>           check the <path-identity>, use "!.SEEN." followed by the FQDN,
>           IP address, or expected <path-identity> of the source.
>
> The term "expected <path-identity>" has been used three times there. but
> has not been properly explained. I would suggest adding, at the end of
> these steps:
>
>    In the above, the "expected" <path-identity> refers to that
>    appropriate to the apparent source of the article (e.g. as determined
>    by the channel whence it arrived, its IP address, or some
>    authentication provided during connection setup).

How about this?

--- usepro.xml	(revision 5041)
+++ usepro.xml	(working copy)
@@ -405,7 +405,13 @@
                   followed by the FQDN, IP address, or expected
                   &lt;path-identity> of the source.</t>
                 </list>
-              Be aware that <xref target="RFC1036" /> did not include
+              The "expected &lt;path-identity> of the source of the
+              article" is a &lt;path-identity> for the injecting or
+              relaying agent that passed the article to this relaying or
+              serving agent, determined by properties of connection via
+              which the article was received (for example, an
+              authentication identity or a peer IP address).  Be aware
+              that <xref target="RFC1036" /> did not include
               &lt;path-diagnostic>.  Implementations which predate this
               specification will add only single "!" characters between
               &lt;path-identity> strings.</t>

"Source" wasn't really defined either, so I tried to tackle that.  It's
surprisingly hard to come up with good text for this.

> But then there is no explanation of why barbaz used that "!!" delimiter
> (all the other funnies in the example _do_ get accounted for in further
> on in the text). So I would suggest:
>
>    barbaz, being satisfied that it had indeed been recceived from
>    baz.isp.example, inserted a "!!" <path-delimiter>. It then relayed it
>    to old.site.example .........

Agreed.  Just committed a fix.

> 3.3.  Article History and Duplicate Suppression
>
>    o  ............  If this
>       interval is shorter than the time it takes for an article to
>       propagate through the network, the agent might reject an article
>       it had not yet seen, so it ought not be aggressively short. .....
>                                           ^
>                                           to

It's an unusual formulation in general, but I believe the current
formulation is more grammatically correct.  The subject of the dependent
clause is "it" and the verb is "be".  If "be" is converted to an
infinitive, the dependent clause no longer has a verb.

>    These are just two implementation strategies for article history,
>    albeit the most common ones.  Relaying and serving agents are not
>    required to use these strategies, only to meet the requirement of not
>    accepting an article more than once.  However, these strategies are
>    safe and widely deployed and implementors are encouraged to use one
>    of them, especially if they do not have extensive experience with
>    Netnews and the subtle effects of its flood-fill algorithm.
>                ^^^
>            with the more

Hm, that seems to me like more words and a more roundabout phrasing to say
the same thing.

> We are now allowing (sometimes even requiring) posting agents to insert
> an Injection Date.  Whereas clueful multiple injectors will probably do
> this correctly, I am worried that incompetent MUA implementors (or which
> there are many :-( ) will do it wrongly.
>
> On the other hand, now it seems to have been decided only to allow
> injection agents to insert it in limited situations, it is desirable
> that posting agents should be encouraged (somewhere between MAY and
> SHOULD) to do it routinely (otherwise this new header will never catch
> on). So I suggest:
>    
>    When the proto-article already contains both Message-ID and Date
>    header fields, posting agents MAY, as part of the injection process
>    (i.e. immediately before passing it to an injection agent), add an
>    Injection-Date header field to that proto-article (a useful practice,
>    since it provides diagnostic information and can imrove propagation).
>    Moreover, they SHOULD do so if the Date header field (representing
>    the composition time of the proto-article) is more than a day in the
>    past at the time of injection. and they MUST do so if the
>    proto-article is being submitted to more than one injecting agent;
>    see Section 3.4.2.

This is a protocol change, so I'd rather have you raise this one as a
separate issue so that we can go through more process on addressing it.
Personally, I'd prefer for most posting agents to not supply Message-ID,
since the news server can generally do a better job, but I know that isn't
always realistic.  Failing that, what you're saying does make sense to me.

> 3.4.1.  Proto-articles
>
>    A proto-article has the same format as a normal article except that
>    the Injection-Info and Xref header fields MUST NOT be present; the
>    Path header field MUST NOT contain a "POSTED" <diag-keyword>; and any
>    of the following mandatory header fields MAY be omitted: Message-ID,
>    Date, and Path.  In all other respects, a proto-article MUST be a
>    valid Netnews article.  In particular, the header fields which may be
>    omitted MUST NOT be present with invalid content.
>
> I am worried about that 'MUST NOT contain a "POSTED" <diag-keyword>'. It
> would cause no interoperability problems and cause no significant harm,
> and hence a "MUST" cannot be justified under RFC 2119. Nothing breaks if
> two "POSTEDs appear in one Path.  Indeed, it could arise legitimately in
> multiple injection "after the fact", and in other exotic gatewaying
> situations.
>
> Yes, some s[pc]ammers will preload the Path so as to disguise the true
> origin of an article (which is indeed why POSTED was invented), but that
> is a matter to be taken up by the netkops rather than the protocols. And
> I would prefer to preserve all evidence left over from previous Paths in
> order to facilitate bebugging of broken gateways, etc.
>
> So, at the most, it ought to be a SHOULD, and I would be happy to omit
> it altogether. Note that whatever change is made here, corresponding
> changes would be needed in 3.4.2 and 3.5.2 Step 2.

This is also a protocol change and should therefore be a separate issue.
I'm inclined to agree with you that MUST isn't justified here and SHOULD
is more appropriate.  I think we already talked about this before,
although I haven't gone back and looked.

> 3.4.2.  Multiple Injection of Articles
>
>    Under some circumstances (posting to multiple disjoint networks,
>                                                 ^
>                                              supposedly

Changed.

>    injecting agents with spotty connectivity, or for redundancy, for
>    example), ........
>
>    Whenever possible, multiple injection SHOULD be done by offering the
>    same proto-article to multiple injecting agents.  The posting agent
>    MUST supply the Message-ID, Date, and Injection-Date header fields,
>    and the proto-article as offered to each injecting agent MUST be
>            ^^^^^^^^^^^^^
>            proto-articles
>    identical.

Intentionally singular both here and in the first sentence of that
paragraph to emphasize that there's only one proto-article which is
offered identically to each injecting agent.

>    In some cases, offering the same proto-article to all injecting
>    agents may not be possible (such as when gatewaying, after the fact,
>    articles found on one Netnews network to another, supposedly
>    unconnected one).  In this case, the posting agent MUST convert the
>    article back into a proto-article before passing it to another
>    injecting agent, but it MUST retain unmodified the Message-ID, Date,
>    and Injection-Date header fields.  It MUST NOT add an Injection-Date
>    header field if it is missing from the existing article.  It MUST
>    remove any Xref header field and either rename or remove any
>    Injection-Info header field and other trace fields.
>
> I don't like the term "after the fact" (it has no intuitive meaning),

Changed to "after injection", which is more precise.

> and I don't think "converting back to a proto-article" is the best way
> to describe what needs to be done (since it can still contain various
> headers not present in the first-time-around proto-article). Here is an
> alternative wording:
>
>    In some cases, multiple injection arises (perhaps inadvertently) when
>    gatewaying articles found on one Netnews network to another,
>    supposedly unconnected one.  In this case, the (re-)posting agent
>    MUST remove any Xref header field and either rename or remove any
>    Injection-Info header field and other trace fields (thus converting
>    the article back into a proto-article) before passing it to another
>    injecting agent, but it MUST retain unmodified the Message-ID, Date,
>    and Injection-Date header fields.  It MUST NOT add an Injection-Date
>    header field if it is missing from the existing article.

Mmm, I see your point, but I don't prefer your wording.  Among other
things, it doesn't make any sense to me to insert a "perhaps
inadvertently" comment here, since if it's inadvertent, the agent has no
way of knowing that it should do this.  The current wording makes it clear
that this requirement applies to all cases of multiple injection after the
initial injection, regardless of whether the networks are disjoint.

> Note that wording might need further revision, depending on what we
> decide regarding allowing "POSTED" to appear twice in a Path.

Yes.

I now have the following, which makes the handling of Path consistent with
the current definition of a proto-article.  This is only provisional until
we resolve handling of the POSTED <diag-keyword> in Path as discussed
above.

    <t>In some cases, offering the same proto-article to all
    injecting agents may not be possible (such as when gatewaying,
    after injection, articles found on one Netnews network to
    another, supposedly unconnected one).  In this case, the posting
    agent MUST remove any Xref header field and rename or remove any
    Injection-Info, Path, and other trace header field before
    passing it to another injecting agent.  (This converts the
    article back into a proto-article.)  It MUST retain unmodified
    the Message-ID, Date, and Injection-Date header fields.  It MUST
    NOT add an Injection-Date header field if it is missing from the
    existing article.

>    Multiple injection of an article listing one or more moderated
>    newsgroups in its Newsgroups header field SHOULD only be done by a
>                                                     ^^^^
> 						    ONLY
>    moderator and MUST only be done after the proto-article has been
>                       ^^^^
> 		      ONLY
>    approved for all moderated groups to which it is to be posted and has
>    an Approved header field (see Section 3.9).  .....
>
> I think that is well within the spirit of RFC 2119, and will be much
> clearer.

I'd rather keep the capitalized words only at the RFC 2119 ones, and I
don't think this changes the clarity.  But if other people agree, I'll
change it.

> 3.4.4.  Construction of the References Header Field
>
>
>    The content of the new article's References header field MUST be
>    formed from the content of the parent's References header field if
>    present and the content of the Message-ID header field of the parent.
>            ^^^
>         followed by

Changed.

> 3.6.  Duties of a Relaying Agent
>
>    5.  It SHOULD reject any article that matches an already-received
>        cancel control message or the contents of the Supersedes header
>        field of an accepted article, provided that the relaying agent
>        chose (on the basis of local site policy) to honor that cancel
>        ^^^^^
>        has chosen
>        control message or Supersedes header field.

I'm not sure why the switch from past tense to present perfect tense here.
The decision to honor a cancel or Supersedes is normally made on receipt
of the cancel or Supersedes article.  I guess present perfect might be
more inclusive of different points at which that decision might be made,
and we don't actually care when the server makes the decision.

Okay, changed.

> 5.2.3.  The checkgroups Control Message
>
>    The body of the message is an entity of type application/
>                              ^
>        (or contains, as part of a multipart/mixed)

No, that would break compatibility with existing software.

> Cf. wording in 5.2.1.

They don't have the same legacy issues.  Legacy newgroup message
processing searches the body for the key string.  Legacy checkgroups
processing uses the entire article verbatim and hence would treat MIME
separators, headers, and other text as group names.

> 5.4.  The Supersedes Header Field
>
>    The presence of a Supersedes header field in an article requests that
>    the message identifier given in that header field be withdrawn in
>    exactly the same manner as if it were the target of a cancel control
>    message.  Accordingly, news servers SHOULD use the same
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>    authentication and authorization checks for deciding whether to honor
>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>    a Supersedes header field as they use for cancel control messages.
>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>    If the Supersedes header field is honored, the news server SHOULD
>    take the same actions as it would take when honoring a cancel control
>    message for the given target article.
>
> The bit I have marked with ^^^^^^^^ is all fine and dandy, except that
> the preceding section 5.3 on Cancel Messages make no mention of
> "authentication and authorization".

It's covered by section 5.1.

> However, what is currently said in our section 5.1 is fine (so far as it
> goes) for Group Control Messages, but has nothing to say of relevance to
> Cancels and Supersedes.

How so?  I think this pretty much covers it:

   Agents acting on control messages SHOULD take steps to authenticate
   control messages before acting on them, as determined by local
   authorization policy.  Whether this is done via the use of an
   unstandardized authentication protocol, by comparison with
   information obtained through another protocol, by human review, or by
   some other means is left unspecified by this document.  Further
   extensions or revisions of this protocol are expected to standardize
   a digital signature mechanism.

There isn't anything specific to group control messages there.  Those are,
stated generally, exactly the same provisions as should apply to cancel
messages and Supersedes.

> And also, as regards Supersedes headers, we do not actually say that, in
> addition to acting as a Cancel, they are also propagated and displayed
> like an ordinary article. I therefore suggest the following additional
> paragraph:
>
>    The article containing the Supersedes header field is itself still
>    intended to be made available to reading agents in the usual manner.

Agreed.  I tried to make the wording a bit more specific and now have:

        <t>The article containing the Supersedes header field, whether or
        not the Supersedes header field is honored, SHOULD be handled as a
        normal article and SHOULD NOT receive the special treatment of
        control messages described in <xref target="serving" />.</t>

This borders on a protocol change, but I think this just clarifies what
everyone's understanding was anyway, so I went ahead and changed it.

> Agreed as far as Cancels, but does there in fact exist such "widespread
> abuse" of Supersedes?

Yes.  It used to be significant.

>    Such articles intended to deny service, or other articles of an
>    inflammatory nature, may also have their From or Reply-To addresses
>    set to valid but incorrect email addresses, thus causing large
>    volumes of email to descend on the true owners of those addresses.
>                    ^
>               ("mailbombs")

Eh, I don't think that really adds anything.

>    Users and agents should always be aware that identifying information
>    in articles may be forged.
>
> Please add:
>
>    The inclusion of a Disposition-Notification-To header field could
>    also give rise to such mailbombs (hence its deprecation for Netnews
>    in [RFC3798]).

Wouldn't that be overkill?  No one has ever implemented that header for
netnews, no one has ever considered it so far as I've heard, and it's
explicitly deprecated for it.  I don't think it's enough risk to warrant
singling out specifically here.

> Authors' Addresses
>
>    Charles H. Lindsey
>    University of Manchester
>    5 Clerewood Avenue
>    Heald Green
>    Cheadle
>    Cheshire  SK8 3JU
>    United Kingdom
>
> I would suggest removing that "University of Manchester" line. I am long
> retired from the University, and I would not like to people to think
> that the address of the University was "5 Clerewood Avenue, Heald Green"
> :-) .

It took a little to figure out how to do that.  Apparently the trick is an
empty <organization> tag; you can't omit it entirely.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 24 Sep 2008 15:57:37 GMT

>although I haven't gone back and looked.

OK, here it is. I note that you agree that MUST is too strong.

However, I think SHOULD (and perhaps even MAY) is too strong. The protocol
use to prevent loops is not affected by this change. But the Path is also
important for diagnosing the cause of loops (not necessarily
non-terminating ones) such as those caused by peculiar gatewaying, and for
spotting trollers who try to disguise the source of their articles.

So seeing two 'POSTED's in a Path should immediately raise alarm bells
(even though it may occasionally turn out to have a benign explanation).
But if duplicate 'POSTED's are removed, then no alarm bells will be
heard. So leave them in, and let the Debuggers and the Netkops take care
of them.

In either case, there would be consequential changes elsewhere, as noted.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 24 Sep 2008 15:45:03 GMT

>always realistic.  Failing that, what you're saying does make sense to me.

OK, here it is, and I see that you partially agree.

My point is that many News User Agents do double duty as Email User
Agents. They routinely create Message-IDs for email messages, and hence
are likely to do the same for news articles. Granted they often do it
wrong (at one stage, my email was going out with Message-IDs based on
'localhost' :-( ), but that is the way it is, even if I might agree that
servers might indeed do the job better. So in that case, it is now likely
that the majority or articles will never receive an Injection-Date (which
is why I didn't like the IR formulation).

My proposal above is intended to mitigate this by encouraging ("MAY") user
agents to insert Injection-Date themselves if they are already doing
Message-ID.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 24 Sep 2008 16:27:01 GMT

>"Charles Lindsey" <[email-masked]> writes:

OK, you seem to have covered most of my suggestions, and I have created
Issues for the two that need discussion, as you suggested.

Here are a few remaining niggles.

>infinitive, the dependent clause no longer has a verb.

No, the verb in that dependent clause is "ought".

Consider the sentence "You ought to know better." Surely that is
gramatically correct, but it needs the infinitive "to know" to make it
work.

It would be similar in French: "Il vous faut savoir mieux."

>    another, supposedly unconnected one).  In this case, the posting
                                    ^
                                    ,
>    passing it to another injecting agent.  (This converts the
                                          ^^^^^^^^
                                           (this

>stated generally, exactly the same provisions as should apply to cancel

Yes, but we make a suggestion (explicitly 'pgpverify' now) in the case of
control messages, but remain devoid of suggestions for cancels.

Perhaps the following alternative wording would convey the sense better:

>>                                               ^^^^^^^^^^^^
                                                 apply whatever
>>    authentication and authorization checks for deciding whether to honor
                                      they already apply
>>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                 in the case of

which has the advantage of being shorter, and gives no implication that
such checks are normal for cancel messages (which, sadly, they are not).

---

From: Russ Allbery <[email-masked]>
Date: Wed, 24 Sep 2008 18:17:43 -0700

"Charles Lindsey" <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

> gramatically correct, but it needs the infinitive "to know" to make it

Ah, yes, thank you.  I missed that.  Fixed now.

>>    another, supposedly unconnected one).  In this case, the posting

I changed it to another supposedly-unconnected one instead.  Using them
both as adjectives rather than separating out the phrase reads better to
me.

>>    article back into a proto-article.)

The parenthetical is a complete sentence and therefore should begin with a
capital letter.

> control messages, but remain devoid of suggestions for cancels.

We can suggest cancel lock.  I almost did, except that it's not widely
deployed like pgpverify is.  There are some implementations, though.

> such checks are normal for cancel messages (which, sadly, they are not).

How about this:

      <section anchor="supersedes" title="The Supersedes Header Field">
        <t>The presence of a Supersedes header field in an article
        requests that the message identifier given in that header field be
        withdrawn in exactly the same manner as if it were the target 
        of a cancel control message.  Accordingly, news servers SHOULD
        apply to a Supersedes header field the same authentication and
        authorization checks as they would apply to cancel control
        messages.  If the Supersedes header field is honored, the news
        server SHOULD take the same actions as it would take when honoring
        a cancel control message for the given target article.</t>

I think inverting the sentence makes it clearer.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 25 Sep 2008 15:14:08 GMT

>The parenthetical is a complete sentence and therefore should begin with a

Yes, but my suggestion was to make it a continuation of the preceding
sentence (hence I removed the '.'). But it is no huge issue :-) .

>I think inverting the sentence makes it clearer.

Yes, but I would still like to see the word "whatever" in there somewhere,
just to make it clear that there might well be no checks on cancel
messages (and hence no action applicable to Supersedes). That is, in fact,
the common current practice, much as we might wish otherwise :-( .

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 25 Sep 2008 15:06:35 GMT

Sory, I omitted to change the Subject line to include "ISSUE" the first
time I posted this.

In <[email-masked]> "Charles Lindsey" <[email-masked]> writes:

---

From: Russ Allbery <[email-masked]>
Date: Thu, 25 Sep 2008 12:06:53 -0700

"Charles Lindsey" <[email-masked]> writes:

> Yes, but I would still like to see the word "whatever" in there
> somewhere, just to make it clear that there might well be no checks on
> cancel messages (and hence no action applicable to Supersedes). That is,
> in fact, the common current practice, much as we might wish otherwise
> :-( .

Does "the same authentication and authorization checks as they would
apply" versus "whatever authentication and authorization checks they would
apply" feel different in meaning to you?  That shading of meaning is too
fine for me to detect, and the word "whatever" seems less formal and
precise, which is why I used the former wording.

"The same" to me includes the possibility of "none."

---

From: Thorfinn <[email-masked]>
Date: Fri, 26 Sep 2008 10:23:22 +1000

> "The same" to me includes the possibility of "none."

:-) I have occasionally similar arguments with my wife regarding word  
usage, which is when she generally reminds me that I've been a  
computer programmer for too long, and I usually realise she's right.

"the same checks" does definitely have a small difference in  
implication to "whatever checks".  "the same checks" has a slightly  
stronger implication that there is at least one check.  It's not a  
hugely stronger implication, no.  And it is just in implication,  
rather than in explicit meaning.

However, I think that that is not a bad implication to have, and you  
are certainly correct that "the same" includes the possibility of  
"none".

Certainly implementors of news software (the target audience for this)  
should already understand that possibility. :-)

-- 
<a href="http://tertius.net.au/~thorfinn/">[email-masked]</a>
The world is coming to an end.  Please log off.  -- BSD fortune file

---

From: Julien ÉLIE <[email-masked]>
Date: Fri, 26 Sep 2008 07:57:04 +0200

Ho Thorfinn,

>>>>       Accordingly, news servers SHOULD
>>>>       messages.
>> apply" versus "whatever authentication and authorization checks they  would apply" feel different in meaning to you?
>
> "the same checks" does definitely have a small difference in  implication to "whatever checks".  "the same checks" has a slightly 
> stronger implication that there is at least one check.  It's not a  hugely stronger implication, no.  And it is just in 
> implication,  rather than in explicit meaning.

Why not say "the same authentication and authorization checks, if any,
as they would apply"?

-- 
Julien ÉLIE

« La vie ne vaut rien. Mais rien ne vaut la vie. » (André Malraux)

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 26 Sep 2008 14:27:42 GMT

>rather than in explicit meaning.

Yes, that is essentially the point I was trying to make. Or Julien's "if
any" would do as well.

>are certainly correct that "the same" includes the possibility of  

Indeed, but if you want to drop more/stronger hints that checks on cancel
messages are a Good Thing, then the text(s) discussing cancel messages is
the place to do it (but we already say as much as we reasonably can
there).

>should already understand that possibility. :-)

Implementors "should" do lots of things :-( .