Subject: Re: #1416

From: [email-masked]
Date: Fri, 8 Aug 2008 12:36:57 -0700 (PDT)

Russ Allbery <rra@xxxxxxxxxxxx>:

> Briefly: the Message-ID plus the Date is the unique identifier for the
> message.

I'm with Frank in being confused by this statement. According to USEFOR:

    A "message identifier" (Section 3.1.3) is a unique identifier for an
    article, usually supplied by the "user agent" that posted it or,
    failing that, by the "news server".  It distinguishes the article
    from every other article ever posted anywhere.  Articles with the
    same message identifier are treated as if they are the same article
    regardless of any differences in the body or header fields.

This clearly states that two articles with the same message id, even with 
different dates, are the same article. "Every article ... ever". If you 
say that "message id plus date" is the unique identifier, then you are 
saying that two messages with different dates but identical message ids 
are two different messages -- a violation of the definition of message id.

> If you allow an injecting agent to change one or the other (and
> adding an Injection-Date is effectively changing the Date),

It does not change the Date header.

> It requires, if I remember correctly, badly-timed multiple
> injection of the message such that it expires on the basis of its original
> Date and is then reaccepted on the basis of the modified Injection-Date.

If a message is expired based on the Date header, why would the server 
then accept the same article with the same Date header later? If it does,
it's just going to expire it again. This just doesn't make sense. It 
sounds too much like a bug and not a feature.

> Assuming that the message ever gets to you, yes.  However, unless enough
> other servers implement Injection-Date to allow flood-fill to reach you, a
> message that benefits from this feature won't get to you for you to be
> able to benefit.

So the benefit you see is that articles with really old dates will still 
be propogated, even though the date indicates that the message predates 
the start of a local history file and thus the article is assumed to have 
been seen already?

> Our protocol, as currently documented in -10, contains nothing about old
> article prevention except at the injecting agent point (see below).

By assuming "old date" (defined as "prior to oldest local history file 
entry") means "has already been seen", section 3.3 does implicitly deal
with old article rejection. Not an important point, but still true.

> Date or Injection-Date is a way to avoid having to keep perpetual history
> to satisfy this requirement.

Yes, by assuming that "old" means "already seen". Only then can you say 
that not finding a message id in the history means that you've already 
seen the article, when normally not finding the message id means you 
haven't seen it yet.

> The
> only protocol function (from a USEPRO, not a USEFOR, perspective) of the
> Date and Injection-Date header is to enable the normal history mechanism
> to function.

Some headers, sometimes, serve more than just a "protocol function from a 
USEPRO perspective". I see value in Injection-Date as a record of when the 
article was injected versus when it was written.

For that reason, I think I have to object to the exclusion statement and 
require injecting agents to put in an Injection-Date header unless that 
header is already there, without regard to existing message id or date 
headers.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 08 Aug 2008 14:21:32 -0700

[email-masked] writes:

> This clearly states that two articles with the same message id, even
> with different dates, are the same article. "Every article ... ever". If
> you say that "message id plus date" is the unique identifier, then you
> are saying that two messages with different dates but identical message
> ids are two different messages -- a violation of the definition of
> message id.

I really think the confusion here is just my unfortunate summary; there is
probably no simple way to talk about this.  From the rest of your message,
you understand what's going on, so we're not really disagreeing.  I think
the best thing to do at this point is to just ignore anything I write in
e-mail that's confusing in favor of paying attention to the language in
the current I-D, since that's all that really matters anyway.

If the language in the current I-D is still confusing, please someone
write something better and I'll incorporate it.  Or, for that matter, pick
up as editor; I'm happy to step down in favor of someone else who has more
time and energy to try to explain all of this.  I'm pretty burned out.

>> It requires, if I remember correctly, badly-timed multiple injection of
>> the message such that it expires on the basis of its original Date and
>> is then reaccepted on the basis of the modified Injection-Date.

> then accept the same article with the same Date header later?

Becuase of the added Injection-Date header with a newer Date, which
overrides the article's Date header for this purpose.

> If it does, it's just going to expire it again.

Most servers expire based on arrival time, not based on the Date header.
There are various internal data structure reasons why this is often
easier, plus it's arguably a more correct user experience.  See the third
bullet point in the history section in the current draft for the technical
details.

> the start of a local history file and thus the article is assumed to
> have been seen already?

Correct.  That's the intended benefit of Injection-Date.

>> Our protocol, as currently documented in -10, contains nothing about
>> old article prevention except at the injecting agent point (see below).

> with old article rejection. Not an important point, but still true.

Right.  But it only does so in a context of not accepting articles that
were already accepted, not in a broader sense of rejecting articles with
old dates only because they have old dates.

I agree that in practice with most servers using the typical history
mechanism, the distinction is purely academic.

>> Date or Injection-Date is a way to avoid having to keep perpetual
>> history to satisfy this requirement.

> seen the article, when normally not finding the message id means you

Exactly.  I don't think we're disagreeing here.

>> The only protocol function (from a USEPRO, not a USEFOR, perspective)
>> of the Date and Injection-Date header is to enable the normal history
>> mechanism to function.
>
> Some headers, sometimes, serve more than just a "protocol function from
> a USEPRO perspective". I see value in Injection-Date as a record of when
> the article was injected versus when it was written.

Right, that's why I worded it that way.  The header serves other purposes
outside of the USEPRO context.

> header is already there, without regard to existing message id or date

Okay.

You, Harald, and Charles feel that way by my current count, with Forrest,
myself, and Dan on the other side.  I'm not sure which way Frank is
leaning.

Given that the people who want Injection-Date to always be added feel
strongly about it and given that the circumstances when this fails are
obscure, maybe the best approach is to just go forward with always adding
Injection-Date.  As discussed in other threads, I'd really like to
finalize this and get it out the door before we all burn out completely
and never publish anything, and either is a significant improvement over
RFC 1036.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 14 Aug 2008 11:18:09 GMT

>    A "message identifier" (Section 3.1.3) is a unique identifier...

>are two different messages -- a violation of the definition of message id.

Sure, but in practice if you want to check whether a message is the same
as some other ancient message, it is much simpler to have a "date" as well,
so you can just say "this is so stale that I shall assume it is a
duplicate and ignore it anyway".

>sounds too much like a bug and not a feature.

If you now have both Date and Injection-Date headers, then which is the
true "date" for the purposes of the test above? "OLD" (i.e. unupgraded
software) with use Date, and "NEW" (upgraded sotware) will use
Injection-Date.

So if two copies of an article are circulating, one with just Date and the
other with both Date and Injection-Date (say 24 hours later), and the
first arrives at some NEW server, is stored and eventually expires (say
afyer 10 days) and then 23 hours later the second turns up, it will be
accepted as if it were new.

So how can this happen? it requires the following scenario:

1. The article was generated by an OLD User Agent, and multiply injected
   to *two* Injection Agents (one OLD and one NEW).
2. The two injections were 24 hours apart.
3. The 2nd copy got delayed by ~10 days somewhere, and then somehow got
   routed to that FINAL agent which had just expired the 1st copy.
4. That 2nd copy just "happened" to arrive at the FINAL agent within the
   critical 24 hours after the expiry.
5. The flooding algorithm had somewhere failed to prevent the propagation
   of the of the delayed copy (i.e. the FINAL agent must have been in
   some rather poorly connected part of the network).

So that is five conditions to be satisied, all somewhat improbable. And,
at the end of the day, all that happens is that the client users of that
FINAL agent get to see the article twice, which is no Big Deal IMHO.

>header is already there, without regard to existing message id or date 

Exactly!

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5