Subject: [Asrg] Testing content filters

From: Matt Sergeant <[email-masked]>
Date: Tue, 18 Mar 2003 12:17:59 +0000

One thing the anti-virus people got right is the ability to test 
anti-virus engines by passing through a non-viral file that is by 
de-facto detected as a virus by the vendors engines. This file is 
called EICAR, and is a simple .COM executable that does nothing but 
trigger AV engines.

In order to provide something similar for spam content filters, I 
created something similar for spam. We put support for this into the 
SpamAssassin project and notified some other creators of spam content 
filters. Unfortunately they all ignored us.

I think this would be a good thing to support in content filters.

We called this thing "GTUBE" (Generic Test for UBE), and you can find 
the email that should be stopped at 
http://spamassassin.planetmirror.com/dist/t/data/spam/gtube.eml (note 
the headers are abitrary - it is the long string in the body of the 
message that should be detected).

The email is designed in such a way that it should not be munged by any 
gateways.

I urge other vendors of anti-spam content filters to support this. It's 
trivial, but it gives users the power to test that their filter is 
working.

I also think this would be something worth standardising through the 
IETF.

Matt.

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Kee Hinckley <[email-masked]>
Date: Tue, 18 Mar 2003 16:43:39 -0500

>I think this would be a good thing to support in content filters.

It's fine for content-filters.  What do you do about spam filters 
that don't look at the content?
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Matt Sergeant <[email-masked]>
Date: Tue, 18 Mar 2003 23:03:55 +0000 (GMT)

> that don't look at the content?

You don't test those with this test.

Matt.

---

From: Matt Sergeant <[email-masked]>
Date: Tue, 18 Mar 2003 23:09:50 +0000 (GMT)

> that don't look at the content?

Sorry, my last reply was a bit flippant...

What I really should have said is that there are probably 4 stages of 
filters in use right now:

1. IP (connection) filters.
2. Envelope and SMTP command stream based filters.
3. Header filters
4. Body filters (or header+body filters)

You can't create a test for 1. without spoofing the IP address, so we 
won't go there.

We could create a test for 2. (e.g. a fixed MAIL FROM address that has to 
be detected as spam, or a fixed HELO line or something like that).

We could create a test for 3.

We have a test for 4.

So please, feel free to add in suggestions for 2 and 3. I've started the 
groundwork - it won't be too hard to finish it off now.

This is a simple proposal, and doesn't get us anywhere closer to our goal, 
but it is important for end users to be able to check if something is 
working, and if they can make the same test on any bit of software that 
makes it easier for everyone.

Matt.

---

From: Vernon Schryver <[email-masked]>
Date: Tue, 18 Mar 2003 20:37:04 -0700 (MST)

> From: Matt Sergeant <[email-masked]>

> I also think this would be something worth standardising through the 

That sounds like a good thing to have, but nothing that should involve
the IETF (not to mention the IRTF).  The IETF cannot standardize anything
in less than 6 months and often neeeds a year or two.  By the time such
a test case were standardized, the spammers would have stopped some
behaviors and started others, making the RFC of only theoretic or historic
interest.

More fundamentally, you don't want to standardize the test case, but
to publish or exchange the current version of the test data.  IETF
standards are about ensuring interoperability, and are cast in stone.
It takes almost as much work to issue a replacement RFC as to create
the original, but you'd want at least quarterly updates for your test.

That the business models of anti-virus and anti-spam services differ
might be a more serious and fundamental problem.  Anti-virus protection
needs to be perfect,and so AV vendors can cooperate.  Anti-spam
protection is allowed and expected to be imperfect, so anti-spam
vendors are able to keep some of their patterns or whatever secret
to compete in customer tests.

This is not to say that you can't publish a non-working-group
Informational RFC, but that you should not be disappointed when it
joins the 95% of RFCs that are write-only/read-never.

As an asside, note that contrary one reading of the suggestion,
effective ilters need to deal with the modifications inflicted on mail
by gateways.  For example, somethings seem to encoded and decode
quoted-printable...never mind why or whether they should.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: [email-masked]
Date: Wed, 19 Mar 2003 00:38:08 -0500

> the original, but you'd want at least quarterly updates for your test.

The EICAR test "virus" is useful specifically *because* its standardized
and is *not* updated quarterly to track the latest virus trends.  It exists
for exactly ONE reason - to validate the following assertion:

*IF* something trips the scanner, an action is taken.

I've mailed myself copies of EICAR plenty of times, specifically to test
that things do what I want - I'd rather have an unquarantined copy of EICAR
than mail myself a copy of Klez and find out it didn't get quarantined.. ;)

An EICAR-SPAM test file would be equally useful for content-based antispam,
as you could mail yourself a copy to test that you bounced spam, rather than
have to wait to get on a spammer list to find out.

---

From: Kee Hinckley <[email-masked]>
Date: Wed, 19 Mar 2003 00:45:51 -0500

>groundwork - it won't be too hard to finish it off now.

Just to clarify.  The goal of this is to have a tag that indicates 
that your software is indeed installed and functioning?  If that's 
the case, couldn't we just establish an X-something header and use 
the same tag?
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: Matt Sergeant <[email-masked]>
Date: Wed, 19 Mar 2003 08:39:06 +0000 (GMT)

> the case, couldn't we just establish an X-something header and use 

No, the goal of this is to have a standard way to trip a spam filter that
is supported across the industry in a non-changing fashion. I don't care 
*how* the filter gets tripped (whether it bounces or quarantines or adds a 
header or munges the subject...), just that it does get tripped.

The idea is that when I install a new spam filter I want to be able to
check its working, and I want to be able to do that without having to use
a different email for each content filter, or without having to have
intimate knowledge of how this particular filter works.

Often a sysadmin would install a new spam filter on a test machine, so 
waiting for spam to come through wouldn't be an option.

Matt.

---

From: Vernon Schryver <[email-masked]>
Date: Wed, 19 Mar 2003 07:10:14 -0700 (MST)

> ...

Ok, I stand corrected about whether it might make sense to standardize
such a thing.  However, this does not sound useful enough for me to
put it into the DCC this year.  Spam and viruses differ.  It's far
more difficult and/or dangerous to give a Windows box a test infection
of a virus than to send test spam to a mail system.  To test a spam
filter that can be tested with a standardized test, you can instead
grab something from NANAS or whatever spam you hate and send it from
somewhere, such as a free provider or one of your own systems.  A
bigger problem is that anti-spam systems define spam in differing
ways.  For example, how would a standardized spam filter test check
that an SBL installation is working?  (The testing auto-responders
permanently listed i the RBL, DUL, and RSS should be but are not be
emulated by all DNS blacklists.)

---

From: wayne <[email-masked]>
Date: 19 Mar 2003 08:19:10 -0600

In <[email-masked]> Vernon Schryver <[email-masked]> writes:

> > de-facto detected as a virus by the vendors engines. 
> 
>                                                        By the time such

The idea isn't that this would be some sort of regression test to see
if a spam filter would detect certain kinds of spam.  Instead it is
just a simply file that all spam filters detect as spam, even though
it really isn't that spammy.

The idea is that if you want to test to make sure the spam filter that
you just installed/upgraded is working, you would email yourself this
tiny file and it should be treated as spam.  You could also test your
ISPs spam filters by sending this file, even though you don't know
what kind of filters they are using.  This would be the universally
accepted "this is spam" e-mail

This special email would be kind of like using ping to check if a
machine is working.

-wayne

---

From: Kee Hinckley <[email-masked]>
Date: Wed, 19 Mar 2003 12:00:25 -0500

>somewhere, such as a free provider or one of your own systems.  A

This is only true for content filters.  And even then it depends on 
the settings on your content filter.  I agree with Matt--there should 
be a standard string that is always flagged as spam.

You can forward all the spam you want at our spam filter--it won't 
get flagged as spam because we could care less what's in the message.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: Matt Sergeant <[email-masked]>
Date: Wed, 19 Mar 2003 18:30:47 +0000

On Wednesday, Mar 19, 2003, at 14:10 Europe/London, Vernon Schryver 
wrote:

> However, this does not sound useful enough for me to
> somewhere, such as a free provider or one of your own systems.

That would be fine if your content filter is 100% effective, but last 
time I looked the DCC wasn't. So you have no guarantee that a 
particular spam is going to trigger it. You're effectively asking your 
users to do the legwork, and that doesn't scale.

Matt.

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 20 Mar 2003 15:07:34 -0700 (MST)

> users to do the legwork, and that doesn't scale.

That the DCC is not 100% effective is a reason users (i.e. system operators)
should do their own leg work.

I don't see how "scaling" in any normal sense might be relevant.

But the real problem is that standardized test messages don't make
sense for some spam filters.  Spam is unlike viruses.  Viruses are
evil data but spam is behavior regardless of data.  I could twist
DCC client software to recognize some test messages, but that would
tell users nothing abut whether their DCC client is talking to a
DCC server that is getting current bulk checksums.  A better example
is what you would have a DNS blacklist client do about a standardized
test message.  A test message is just entirely wrong for DNS
blacklists..  The right test for something like the SBL is a
permanently listed auto-responder, not a test message.