Subject: [Asrg] DCC and IP checksums

From: "David F. Skoll" <[email-masked]>
Date: Fri, 7 Mar 2003 18:54:08 -0500 (EST)

> > Although it looks like the DCC servers have to collect the IP
> > information, they don't have to give it out.  So the potential for
> > abuse is limited to the people who run the DCC servers, not to anyone
> > who can query them.

> That's not an effective response to the privace issues.

Well, most of the other proposals on this forum are far more invasive
(witness the argument involving Hadmut and others.)

> The current, real life global
> network of ~120 DCC servers already involves far more people and
> organizations than I think can be trusted.  It's not that any single
> DCC server operator is untrustworthy, but that every group of 120 or
> more people is untrustworthy.

The privacy implications of leaking the fact that IP address x.y.z.w
sent the same message 1,000 times, or had 25 failed RCPT commands, are
not huge.

> > No; way less than this.  For a mail message, we collect the body
> > checksum, the sending-IP checksum and maybe a few flags indicating
> > failed RCPT commands.  At most 60-100 bytes/message.

> If I thought you knew how to make a single checksum that is fuzzy
> enough to ignore "hashbusters" but not so fuzzy that it has false
> positives, I'd ask you in private about it.

I use the following algorithm in CanIt.  It is by no means perfect,
but it's pretty good:

- Ignore the headers
- Read message lines as follows:
  - Delete leading and trailing whitespace
  - Ignore blank lines
  - Ignore MIME part delimiters
  - Skip a line starting "Dear "
  - Collapse multiple spaces into single spaces
  - Stop reading once you have 200 non-ignored lines or reach end of message

- If you have between 10 and 20 lines, ignore the first and last, else
- If you have between 20 and 40 lines, ignore the first and last 3 lines, else
- If you have more than 40 lines, ignore the first and last 4 lines

- Do a SHA1 hash on what's left.

> My imperfect current solution involves 3 checksums.

How are 3 checksums better than one, unless you're checksumming different
parts of the message?

> checksum would discover no two IP addresses ever send the same
> message, except for trivial cases such as some virus warnings,
> because its answers would all differ.

I have hard evidence to the contrary.  See, for example,
http://www.roaringpenguin.com/canit/showincident.php?id=3037

(You may have to authenticate as "demo/demo" and then enter
3037 as the incident ID.)

I have lots more messages like that, and that's just on my server.

--
David.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Hadmut Danisch <[email-masked]>
Date: Sat, 8 Mar 2003 10:21:18 +0100

> I use the following algorithm in CanIt.  It is by no means perfect,
...
>   - Skip a line starting "Dear "
...
> - Do a SHA1 hash on what's left.

This won't work. 

A significant amount of spam mails I receive slightly differs
for every recipient. E.g. many mails contain the recipients
name, even in lines not starting with "Dear". And several 
mails asking to watch a web page where the URL contains a
hash of the recipient's e-mail address in order to verify that e-mail
address and whether the recipient is willing to read such messages. 
Just that special URL is enough to break your method.

Once your system gets wideley deployed, spammers will start to 
include other random content. Especially in spam messages with 
HTML content it is extremely easy to hide some invisible 
information. Even in plain mail, that's easy. Just print a
random number anywhere. If it should look neat, just have 
something like "Welcome to the porn club, your member number is ####"
I've already received some messages assigning me some silly member
numbers. Maybe these numbers were supposed to spoil any hash
based spam protection.

Since you have to publish the details of your hash algorithm in 
order to deploy it, it's extremely easy for spammers to design
their message such that every single message has a different hash
value.

And, it is a leakage of information:

Let's assume a central server is sending standard messages with 
some kind of secret, e.g. 

  Dear David,

  welcome to this whatever-service, your password is
     12345

or something like

  Dear David,

  your current account balance is
     123.45

I can register to the same service to get the skelleton of the 
message. If I know that you have received a message from the
same IP address with a given hash value I can easily perform 
an exhaustive search through the secret's space to find your
secret.

Hadmut

---

From: "Chris Lewis" <[email-masked]>
Date: Sat, 08 Mar 2003 08:42:08 -0500

> This won't work. 

It works for certain classes of spammer. It can be made significantly 
better (eg: stripping HTML tags and comments, stripping 
non-alphanumerics, not only collapsing whitespace, but _removing_ it), 
but it's still trivially bypassable.

Good for the low rent spammers.  Not enough for Ralsky. Not enough for 
some of the ratware out there.

[ie: there are "pay services" which tune your email to get past 
SpamAssassin.  Not necessarily "bad", unless the intent is to spam as 
opposed to minimizing the possibility of interference with legit 
opt-in/confirmed bulk email.]

---

From: Matt Sergeant <[email-masked]>
Date: Sat, 8 Mar 2003 15:56:09 +0000 (GMT)

> opt-in/confirmed bulk email.]

SpamAssassin 2.50 includes a bayesian component. Thus all SpamAssassin 
installations are different, rendering this sort of service pretty much 
useless once people upgrade. We recognised the above as a major problem, 
so we fixed it.

---

From: "David F. Skoll" <[email-masked]>
Date: Sat, 8 Mar 2003 12:29:34 -0500 (EST)

> This won't work.

I know it won't.  As I wrote before, ANY checksum scheme can be defeated.