Subject: [Asrg] Race conditions at end of DATA

From: "David F. Skoll" <[email-masked]>
Date: Fri, 7 Mar 2003 15:30:05 -0500 (EST)

From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)

> David F. Skoll <[email-masked]> schrieb/wrote:
> > The scanning is done at the end of the DATA phase while the sender is
> > waiting for the final acknowledgement.  The recommended timeout here
> > according to RFC 2821 is 10 minutes, so adding under a second of
> > content-scanning shouldn't be a problem.

> The problem is that this phase is the one vulnerable to race conditions:

This is true.

> If the connection breaks between accepting the message and getting the
> response through to the sender, a dupe will be created. So it's
> desireable to keep that period of time as short as possible.

Right.  But the race condition is there whether you do scanning or not.
It's an inherent limitation of the protocol.

We have about 40 deployments of our commercial scanning solution, and
probably thousands of the GPL'd version, and I haven't heard of anyone
getting a dupe because of this race condition.  To cause a problem,
the scanner would have to take an unreasonably long time, and the
sending relay would have to have an unreasonably short timeout.

--
David.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "David F. Skoll" <[email-masked]>
Date: Fri, 7 Mar 2003 15:40:04 -0500 (EST)

Vernon Schryver wrote:

> > I think a mechanism similar to DCC could be good.  We need a way for
> > lots of sensors to dump information into the collection network.  We
> > need a way to decide what summaries of the data are useful.  And we need
> > a way to extract the data without compromising privacy.  (e.g., report
> > SHA1 hashes of addresses rather than addresses themselves.)
> > ...

> I think I'm qualified to comment on that idea.  The short version of
> my take is "great in theory but nearly hopeless in practice."

OK.

> The hopeless part is that in practice it is extremely difficult to
> build a network large enough to collect enough data to be other than
> a muddy pile of annecdotes.

Well, this is a research group; we consider blue-sky ideas.  If most
people start seeing the level of spam that striker.ottawa.on.ca does,
there will be huge incentive to collaborate.

> The DCC is more than 2 years old, but it
> still sees at most single-digit percentages of all mail in the network
> and perhaps less.

I'm not sure that you need a much higher percentage than that.  If
your sensors are well-dispersed, a couple of percent should be pretty
representative, or at least enough to start detecting trends.

> Even in the skewed population that does use the DCC, there is evidence
> that making generalizations is very hard.  There is 3X difference in
> spam load per user depending on organization type judging from
> http://www.dcc-servers.net/dcc/graphs/comp-rates

Yes, I see that also among my customers.  But we're looking for trends
across the whole Internet.

[...]
> Consider the dangers of being able to
> ask whether the system has seen a message with a sender of the hash of
> "Bill Gates" and at recipeint of the hash of "Steve Jobs" today.

You probably only want hashes of IP addresses, not e-mail addresses.
If the DCC collected not only message hashes, but also the number of
different IPs from which those messages originated, I bet we'd see
some interesting data.

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 7 Mar 2003 14:26:59 -0700 (MST)

> From: "David F. Skoll" <[email-masked]>

> different IPs from which those messages originated, I bet we'd see

Please think how you would utilize such a system if you were a bad
guy or just trolling for interesting information for buying and selling
stock.  Traffic analysis can be applied to more than just names.

The DCC can also collect hashes of IP addresses.  At Paul Vixie's
suggestion, early versions could answer questions like "how many messages
with this body checksum had this source IP address?" To mitigate the
obvious privacy worries even in the original business model where all
DCC servers would be run by a reasonable trustworty outfit as well as
to deal with performance issues, the results were imprecise.  I removed
those mechanisms as the model changed and to gain performance.

Please also consider the amount of data you are talking about.  You
will probably collect 500 or 1000 bytes per mail message.  (A dozen
16- byte checksums, pointers, counts, padding, etc.)  If you sample
1% of mail, and accept my guess that's 100 M msgs/day, you're talking
about collecting and reducing 100 GBytes of data/day.  You'd need to
repeat your measurements every day for a week, because spam varies
during the data and during the week.  You'd also want to repeat it
every few weeks to catch long term changes.  Arithmetic gives numbers
that would need a serious source funds.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg