Subject: [Asrg] Nucleus of a draft BCP on filtering

From: "Chris Lewis" <[email-masked]>
Date: Fri, 07 Mar 2003 16:58:24 -0500

This is something I wrote as a general high level set of suggestions for 
spam filtering.  While it tends to stress current technologies (source 
and content filtering), it does have some applicability to other techniques.

May be useful as a starting point for a BCP on spam filtering.

Guiding General Principles: - spam is UBE (unsolicited bulk email)

- spam is a behaviour, not a content

- a spam filter is not an enforcer of your user's morals, but the
protection of your user's mailboxes from timewasting stuff they don't
want.

- in any but the smallest environment, some spam will get through
and some non-spam will be filtered. User expectation must be managed.
Provision must be made for detecting and dealing with false positives.

- spammers resort to the most extreme (sometimes criminal) measures to 
get around filtering as well as making the "message" as intrusive as 
possible.

- every spam block should be inline rejected with a contact point for
mistakes.

- spam should not be bounced - headers are almost always forged, thus
bounces go to the wrong places, and often harrass innocent third 
parties. Mail should be rejected during the SMTP transaction to provide 
feed back to legitimate senders.

- you need to come up with a guidance metric for what level of false
positives is acceptable.  Because there _will_ be false positives in
any but the most trivial and/or ineffective of spam filtering environments.

- Environments differ - filtering that may be okay for one may not
be acceptable for another.  Hence, purchased anti-spam services are
sometimes highly sub-optimal.

General techniques:

- spam is a behaviour, not a content. You're trying to identify the
sender behaving that way, not the material per-se.

- Blocking on simplistic body word matches, no matter how attractive 
they appear to be do not work effectively in any but the smallest 
environments. Even "fuck" appears in legitimate business email. 
Filtering techniques should attempt to detect only those things that 
ordinary users cannot do inadvertently, and spammers cannot bypass.

- sometimes simplistic content rules work if the rule is "unique" or
"unusual" enough.  FPs must be monitored within your false positive goals.

- header patterns (of spamware), and mail sources (IP ranges and rDNS
names) are generally more effective and less FP-prone than simplistic
content filters.

- Massive spammers are evolving towards "borrowing" innocent third 
parties for final delivery to your server.  Open SMTP relay, open SOCKS 
proxy, open HTTP proxy. It is not practical to test for these in 
real-time. You need to rely on third party blacklists.

- There are many blacklists.  Ranging from hyper-aggressive to totally
ineffective.  From professionally run to maintained out of spite and
hatred. Blacklists should be evaluated for effectiveness rate, FP rate,
and professionalism of maintenance.

- In some cases your policies will conflict with the blacklists, so you
need to be prepared to whitelist sources.

- You need the ability to whitelist on sender, recipient and source.

- Filtered email should be stored for a reasonable time.

- Filtered email should be available for viewing by the end users to
help detect and rectify false positives.

- Non-existant recipient addresses are valuable - you can use them to
detect spammers with little false positive risk (honeypot).

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Keith Moore <[email-masked]>
Date: Fri, 7 Mar 2003 18:52:46 -0500

> Guiding General Principles: - spam is UBE (unsolicited bulk email)

I don't think we're going to end up with a single uniform definition of spam.

to some, spam is simply unwanted mail.  presumably unwanted implies
unsolicited, but unsolicited does not imply unwanted.

to some, bulk is a necessary condition for spam.  I'm guessing that this is
based on a presumption that bulk mail is necessarily untargeted, or perhaps 
that we only need to be concerned about bulk unwanted mail since that is
the biggest part of the problem.

to some, commercial is a necessary condition for spam.  I'm guessing that this
is based on a desire to not censor non-commercial speech.

to some, spam is entirely a matter of content, not behavior.  to others,
it's exactly the opposite.  to others, it is some of each.

I could see where we might end up with a mechanism by which mail was tagged
according to each of several attributes: whether it was bulk, whether it was
commercial,  whether the sender could be reliably traced, whether the sender
had a good or bad or no reputation, etc.  a recipient's filter (or a filter
acting on consent of a recipient) could then treat different kinds of mail
differently based on the tags, or the absence of such tags.

that way, we wouldn't all have to agree on what spam was.

Keith

---

From: "Chris Lewis" <[email-masked]>
Date: Fri, 07 Mar 2003 22:41:54 -0500

> I don't think we're going to end up with a single uniform definition of spam.

I believe we have to, insofar as we need to limit the scope of what the 
"S" in ASRG means.  Of course, some people will call the nagging email 
from the SO, about forgetting their lunch, spam.  But that's just plain 
wrong.

If we don't limit our scope, we'll just be in an endless rathole.

The thing we're primarily concerned about is that activity/behaviour 
which endangers the future of email.  The behaviour that turned striker 
into a smoking crater.

Sending 1:1 email isn't it. Having a salesman for widgets pick your 
email address out of a mailing list about widgets and suggesting "you 
might be able to solve the problem you asked about by buying our XYZ 
widget" isn't it. Changing your mind about email you explicitly asked 
for isn't it.  Your neighbor's emailed porn from Germany isn't it.

Anti-spam lobby organizations, such as the various branches of CAUCE, 
have, I believe, now all fixated on "UBE" as the spam they're concerned 
about (regardless of the "C" in the name) - I'm a director of CAUCE 
Canada.  "Usenet spam" isn't content.  It's simply "the same thing 
posted many times" [I wrote the consensus standard...]

Perhaps it would be nice to exert effort on a classification system that 
provides more flexibility than a simple UBE definition. But you can 
forget about individuals classifying things properly. And we can forget 
about getting very far.  And if we include that, the pro-spam side, and 
the media will just continue on their merry way "they can't even define it".

Think about most the strategies that people are talking about here. 
Counting bounces correlated to senders. Establishing authorized "sending 
MTAs" for given "from" domains. DCC. Spamtrap analysis.  Bayesian, 
Razor, Brightmail recognizing messages that "look like stuff seen 
before" Etc.  That's _all_ about bulk.

The original direction of defining spam as "UCE" (actually, UBCE) 
derived directly out of a belief that C was easier to legislate given US 
jurisprudence. Thinking has shifted since then, to that believing a 
content-neutral definition of spam is (a) easier to legislate and (b) 
easier to implement (because you don't have to write AI engines to 
recognize whether something is commercial).

You can automate bulk detection a lot easier than commercial detection.

Yes, some people will call commercial email spam.  But why?  Why should 
they care if their neighbor is getting commercial email that they asked for?

Defining the "S" in ASRG as "UBE" helps us focus on solutions that may 
prevent more strikers.  Isn't that what all of us are primarily 
interested in?

Besides, I was hoping that the _rest_ of that original message would 
generate the majority of comments ;-)

---

From: "Jon Kyme" <[email-masked]>
Date: Sat, 08 Mar 2003 13:37:50 +0000

> "S" in ASRG means.  

Well, no, but I believe we must form a consensus on a
definition of "consent-based communication". 
That's what the RG is chartered for.

--

---

From: "Chris Lewis" <[email-masked]>
Date: Sat, 08 Mar 2003 09:22:56 -0500

> That's what the RG is chartered for.

Ie: from the charter:

The definition of spam messages is not clear and is not consistent 
across different individuals or organizations. Therefore, we generalize 
the problem into "consent-based communication". This means that an 
individual or organization should be able to express consent or lack of 
consent for certain communication and have the architecture support 
those desires.

I see that. On the other hand, as Paul's welcome to the list also 
comments, we should consider whether that's too confining.

I agree, being able to express consent and have the architecture support 
those desires is an interesting topic. But, unless you strain the 
definition of consent implementation to the limit (and build automatic 
classifiers, which we're a long way away from), you're faced with 
"merely" a technical infrastructure that implements something that 
spammers will ignore. It only works if the spammer "consent" to telling 
the truth.

Rule #1: spammers lie.

Like voluntary classification systems, if the incentive to get around it 
is enough, they will.  And they do. We have ample demonstration of that. 
Even relatively legitimate operations do things to ensure that their 
message gets through, because of measures we're taking to stop the 
totally spammish are believed by the more legitimate operations to be 
overbroad.

For example, you can consider SpamAssassin to be a consent 
implementation that automatically classifies (coarsely) the emails. 
There are already "services" available to tune your message against 
SpamAssassin - equally attractive to both spammers, and legitimate bulk 
mailers.

Which defeats the purpose of consent systems.

I think that one of the early, yet valuable, things that the ASRG can do 
is level-set the "anti-spam" effort: identify which definition of "spam" 
warrants the efforts (UBE, IMHO), classify existing solutions, make them 
play nice with each other, establish best-current-practises for various 
aspects of spam control - ie: BCPs on how anti-spam gateways should 
behave, BCPs on how DNSBLs should operate (ie: truth in advertising), 
BCPs on how false positives should be handled (ie: rejects, versus 
bouncing, versus silent blackholing), BCPs on privacy concerns of 
distributed bulk detection (hi Vernon! ;-), etc.

BCPs not necessarily as "musts", but guidance to administrators and 
filter designers when they're designing, building, deploying and 
operating stuff.

Through this, we also begin to understand what works better, as well as 
answering the age old question, "is it going to be enough to prevent 
striker meltdown of the entire Internet?" (IMHO, no, just a last ditch 
defence that's ultimately doomed without "help" from non-technical means).

---

From: "Alan DeKok" <[email-masked]>
Date: Sat, 08 Mar 2003 05:22:23 -0500

> sender behaving that way, not the material per-se.

  One glaring difference in behaviour is volume.  But the net isn't
currently set up to even be *aware* of volume.

  e.g. The "slashdot effect".  DDoS attacks.  Ping floods.

  e.g. The misconfigured open relay which tries to deliver spam 10x
per second, and ignores any reject messages.  The sysadmin of that
machine may not even notice any problem, if he's over-built his
system.

  While DDoS attacks aren't part of this groups charter, they are, in
my opinion, a large part of the spam problem.

  Is the IETF going to decide that traffic/protocol attacks on the
network infrastructure are outside of the scope of technical
solutions?  Are there not existing technical methods which can address
the problem?

  Alan DeKok.

---

From: "Chris Lewis" <[email-masked]>
Date: Sat, 08 Mar 2003 10:57:52 -0500

> currently set up to even be *aware* of volume.

That leads to an interesting question: can we extend the lower layers of 
the Internet protocols (eg: at TCP/IP) to allow someone to specify and 
enforce rate limiting in such a way as to seriously affect spam?

Unfortunately, I think not.  Consider striker - that's a broad-source 
attack.  Any one of the IPs spamming striker has less volume than some 
of our legitimate senders. Anyone disagree?

> solutions?

I think there's going to have to be.  As door locks are a good technical 
solution for certain classes of undesirable behaviour, you need ot 
resort to legislation and police enforcement to even begin to get a 
handle on the overall problem.

---

From: Vernon Schryver <[email-masked]>
Date: Sat, 8 Mar 2003 09:20:19 -0700 (MST)

> From: "Chris Lewis" <[email-masked]>

> ...

What about things like the DCC?  The fundamental purpose and idea of
the DCC is to measure the volume of mail messages.

> of our legitimate senders. Anyone disagree?

The lower layers have since practically the beginning been set up to
measure rates and some ISPs have since the 1980's been set u to "specify
and enforce rate limiting."  Recall the rates that said "buy a T1 and
pay for a 56K unless and until you use more."

You also said "in such a way as to seriously affect spam".  The fact
that all email including spam is a small part of the total bandwidth
used kills the idea of doing much about spam by counting raw bits.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Alan DeKok" <[email-masked]>
Date: Sat, 08 Mar 2003 07:08:26 -0500

> the DCC is to measure the volume of mail messages.

  That's great, but most of the net isn't running DCC.

  And notwithstanding my opinions about filters, if most of the net
*did* run something like DCC, we probably wouldn't be in this mess.

> pay for a 56K unless and until you use more."

  That's raw bandwidth, which isn't a good measure of network attacks,
or of end-user consent to being hammered by that bandwidth.

> used kills the idea of doing much about spam by counting raw bits.

  If you assume that you can't distinguish email from any other
traffic, then yes.  But I don't think that assumption is true.

  Alan DeKok.

---

From: "Chris Lewis" <[email-masked]>
Date: Sat, 08 Mar 2003 12:37:51 -0500

Vernon Schryver wrote:
>>Alan wrote:

> the DCC is to measure the volume of mail messages.

What about things like the DCC?

No, seriously, I think it's instructive to look at the experience with 
Usenet spam, because it's "farther along the learning curve".  There are 
fundamental differences, which I'll get into in a moment, but much of 
what I'm going to talk about mirrors the experience with email exactly - 
only more so.

And with all due respect, DCC isn't ubiquitous enough to really warrant 
much specific attention from the ratware authors.

In the beginning, there were rate detectors, such as the one I operated 
starting in 1994 which simply detected the same From: appearing in over 
X postings in Y minutes.

Human being checks, yup, it's the same stuff, take remedial action.  Few 
false detections, never FPs.

That worked well for quite a while (a couple of years). Never automated 
that tool to take remedial action on its own - because there _were_ FPs.

Then spammers got smart, and started minimally mutating from lines and 
subjects, yet, the bodies tended to remain pretty similar.

Then we started doing things like matching NNTP-Posting-Hosts (sound 
like the peerIP versus bounce rate proposal perhaps? ;-)

Then, Usenet's "DCC" came on the scene, in the guise of something called 
Cosmo Roadkill.  CR does virtually the same thing that DCC does - 
computes MD5 checksums of dehashed article bodies.  Only difference 
being there was only one of them - it wasn't distributed, but it didn't 
need to be because of the nature of Usenet.

My detector quickly became obsolete.  CR did a vastly better job. 
Overnight CR outstripped my stuff by an order of magnitude.  All 
unattended. No FPs.  So, I essentially switched roles to the small 
amount of stuff CR didn't handle.

Then came crackerbuster emulators, such as Hipcrime/NewsAgent, the 
pheromone spammer and such like.  Hashbusting galore.  As much as 
hundreds of lines of random words or sentences specifically designed to 
screw around with "volume detectors".  You should have seen some of the 
gyrations the manual filter designers went through to cope.  (ie: 
extremely complicated regexp analysis of message-id headers looking for 
artifacts in random number generators)

The changes were dramatic.  One day, I was it.  Next day CR was it.  A 
little while later, CR's effectiveness dropped to 20%, and Usenet 
anti-spam relied on (and still does) a small number of singularly 
_stubborn_ people eyeballing what was going on - which became less and 
less useful because Hipcrime was also attacking the "curative measures" 
themselves.

Not only were there hash busting, there were people searching for open 
NNTP relays (open smtp relays anyone?).  Later open proxies (Hipcrime 
may well have pioneered the technology that's now used to do open 
socks/http proxying - Hipcrime had his roots in writing some moderately 
ineffectual (by today's standards) email spamming tools). "Fake" ISPs. 
The works.

So after 10 years, where is the Usenet now?  Well, luckily, not bad (in 
the mainstream groups).  Is that because of the effectiveness of the 
tools?  For a time it was.  But now even the tools don't work that 
effectively for a variety of reasons.  The tools were a stopgap. 
Nowadays, most ISPs capable of running Usenet in a serious fashion have 
learned how to secure their systems, so abuse is much harder, and a lot 
more people see every incident.

That's the difference: Usenet is a broadcast medium, and people making 
bad on it are a lot more obvious.  The tools bought time.

But, rogues still occur (see David Ritz's proposed UDP of Telstra).

The main reason why I say this is, like CR, DCC will have a distinct 
lifetime during which it's useful.

Ask yourself, how well would DCC fair if someone with the experience of 
you or I started writing ratware with what we know?  The thought is 
appalling.

> pay for a 56K unless and until you use more."

IIRC, that wasn't so much a factor of detecting the rates, but simply 
not installing a full set of line cards.  At least that is what it was 
with our partial T3. That would be hard to automate ;-)

> used kills the idea of doing much about spam by counting raw bits.

Not even striker can much, even with 99.9999% of their bandwidth being 
connection attempt rejects...

That's what I thought (and said).  I'm hoping someone will prove the 
both of us wrong.  But I'm not holding my breath.

---

From: "Alan DeKok" <[email-masked]>
Date: Sat, 08 Mar 2003 07:32:51 -0500

> enforce rate limiting in such a way as to seriously affect spam?

  Current QoS methods may address this issue.  Do you really consent
to accepting more SMTP traffic than your MTA's can handle?

> of our legitimate senders. Anyone disagree?

  No, but that's not quite on-point.  I didn't consent to having 100
TCP SYN's per second sent to my MTA.  Even those syn's wouldn't be
much of a problem if I could inform my upstream provider that I do not
consent to that traffic.  But I can't.

  Look at it as a problem of the network not being aware of consent.
If there was a mechanism available for consent to flow back in the
network, that would result in back-pressure on the originator.  Near
the recipient, consent is easy to administer, because most people
consent to most traffic.  As we get closer in the network to the
sender, consent becomes more difficult to administer, because the
entire net is informing the sender of their non-consent.

  There's no problem with "who created the consent", either.  Someone
3 hops away from me doesn't know, or care, that I didn't consent to
some traffic.  But I have a consent relationship with my upstream hop,
to not send me traffic.  He can accept it from his upstream provider
and then not send it to me, or he can tell his upstream provider that
he doesn't consent to that traffic.  Either way, I don't care.  I
don't get the traffic.

  But notice the consent model: When my upstream provider tells his
provider he doesn't consent to the traffic, it's NOT my consent any
more.  It's his.  So there's no problem with tracking consent across
multiple hops.  Each hop already has a mutual business, technical,
and trust relationship which can be leveraged to administer consent.

  The end result is that if someone sends you traffic you indicated
non-consent for, it's incentive to stop talking to them at all.

  If a site hosts someone who gets non-consent from the everyone on
net, then they may decide that administering that consent is more
expensive than the money they get from that client.  If they *don't*
administer that consent, then everyone else cuts them off.

  The Usenet Death Penalty has been proven to work.  But it requires
administrator input, and isn't automated.

  For people who say that administering consent (most likely
implemented in the form of massive ACL's) is difficult, remember that
this is a *research* group.  Even with that, solutions exist today
which can do packet filtering at multi gigabit rates, with 10's of
1000's of ACL's.  It's not trivial, but it shouldn't prevent us from
examining consent enforcement in the network.

  Alan DeKok.

---

From: Vernon Schryver <[email-masked]>
Date: Sat, 8 Mar 2003 10:46:22 -0700 (MST)

> From: "Alan DeKok" <[email-masked]>

>   That's great, but most of the net isn't running DCC.

It's great for the fraction of the net that is running the DCC.  Even more
true than Brad Templeton's observations that spam solutions should not be
100% perfect is the fact that a significant part of the net will never
want or use good spam defenses, not to mention any particular defense.

> *did* run something like DCC, we probably wouldn't be in this mess.

Maybe so or not.  As others have said, should the DCC or something
similar become seriously popular instead of merely entirely usefully
popular, spammers would pay more attention and more work would be
needed to stay ahead of them.

> traffic, then yes.  But I don't think that assumption is true.

You can distinguish most mail from HTTP or other traffic by looking
at TCP headers (exceptions include VPNs), but that's not useful.  You
need to distinguish spam.  It's not exactly true that "one glaring
difference in [spam] behaviour is volume."  Sending 100,000 mail
messages/day is not necessarily spam or otherwise bad.  It's not even
remarkable for at least 1000's and probably 10,000's of entirely
reputable, very "anti-spam' outfits.

---

From: "Jon Kyme" <[email-masked]>
Date: Sat, 08 Mar 2003 17:47:35 +0000

> consent for certain communication and have the architecture support 

The charter for the RG is online :-)

> On the other hand, as Paul's welcome to the list also 

Indubitably. Of course the content (while deceiving some automatic
classifiers) can't. Which is why so many people favour these approaches.

However, either "consenting-ness" is only considered proximal to the
receiver (in the receivers MUA?) or it's propogated back toward the sender
in the transport chain. In the latter case some means of representing and
publicising "consent" is *required*
and so considerations of what consent is, what entities give and and seek
it, are not just "interesting" but merely acedemic exercises, but (I guess)
key to the definition of a framework which can utilise whatever particular
technology we might individually espouse.

> distributed bulk detection (hi Vernon! ;-), etc.

Yes, absolutely, for which a taxonomy / theory of "spam" and "spam
reduction" technologies is *required" or this push will become a morass of
narrowly empirical, anecdotal and disjoint initiatives.

---

From: Vernon Schryver <[email-masked]>
Date: Sat, 8 Mar 2003 11:02:51 -0700 (MST)

> you or I started writing ratware with what we know?  The thought is 

I see much the same, but from a different, perhaps optimistic angle.
Since I resumed email spam checksumming with the DCC, I've said that
the DCC is targeted against "mainsleaze" or junk such as the Dell
newsletters that have been rattling my traps.  I'm counting on the
cottage spammers being controlled by governments.  If my predictions
(not proposals!) about what Congress and the DMA will do come true
in time, then defenses like the DCC and even other defenses like
SpamAssassin and so called Bayesian filters will be remain effective.
I'm predicting that the genuinely professional advertisers will move
to save email advertising with fees or taxes, including capping the
spam per victim rate at about 1 dozen interesting commercial messages/day.
The professionals will not deign or even want to do things like hire
the anti-SpamAssassin groomers.

---

From: abuse <[email-masked]>
Date: Sat, 8 Mar 2003 18:44:01 -0800 (PST)

on "3-8-2003" "Jon Kyme" writ:

: Well, no, but I believe we must form a consensus on a
: definition of "consent-based communication".
: That's what the RG is chartered for.

... i'd like to see 'obscurity' considered as a element of it ...