Subject: Spam Filter Y2.1K Bug

From: Glen <[email-masked]>
Date: Sat, 02 Jan 2010 12:45:07 -0800

All -

For those of you who may have experienced alcohol-related blackouts in
the past 48 hours, please be advised that the year is now 2010.

This is relevant for many reasons, one of which is that there appears to
be a bug in "SpamAssassin", one of the spam filtering programs used on
our servers.

A particular rule that checks the date of messages with the purpose of
finding messages that are "far in the future" apparently does NOT know
that the year is 2010.

As a result, as of yesterday morning, this rule started to be applied to
all mail processed by SpamAssassin installations worldwide, resulting in
a significant increase in legitimate mail being incorrectly tagged as
Spam.  The rule in question was designed to find mail that was extremely
off-date, and, therefore, the rule had a very high default score, which
would push tagged mail well over all reasonable spam thresholds.

This bug affects everyone, and has been documented everywhere:
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6269 et al.

I found this problem in my own mail yesterday evening, and removed that
rule from all of our servers at that time.  However, from 00:01 until
about 19:30 Pacific time January 1, some legitimate mail sent through
our servers may have been incorrectly tagged as spam.

Please check your spam folders, mailing list moderation queues, and
other relevant locations to ensure that any held mail gets found and
released.

If you sent mail through our servers, and it did not go through, you may
wish to simply send it again.  The rule has been cleared, and mail is
flowing normally once again.  You may wish to notify your clients and/or
associates about this problem.  As this is a global problem, you should
consider checking all mail servers you have interaction with, worldwide,
until this gets fixed.

Happy new year.
Glen

Glen Barney
IT Director
AMS

---

From: Glen <[email-masked]>
Date: Mon, 4 Jan 2010 09:49:41 -0800

> the past 48 hours, please be advised that the year is now 2010.

Good morning/day:

I have received a surprising number of messages over the past 72 hours
pointing out that the problem was not a "Y2.1K" bug, but rather a "Y2.01K"
bug.

In my defense... 

The problem in question surrounded a specific rule: FH_DATE_PAST_20XX . As
you can see from the name of the rule, it is only supposed to hit email
that was dated past 2099.  However, it turns out that the rule was hitting
all email past 2009.  Because the rule had a base score of 20, which is 
well above the threshhold of 5 used by the IETF lists, almost every email
sent was being tagged as Spam.

... so I was beguiled by the name of the rule, which clearly clouded by
decimal-point-placement-judgement.

Only in this community could I expect such overwhelming dedication to 
such precision.  :-D  I bow to the wisdom of the collective, and humbly
acknowledge my decimal-point error. :-D

On a more serious note...

This problem exists globally, in all SpamAssassin installations worldwide,
and doubtless will take days or longer for every IT department in the world
to notice, research, figure out and fix.

Until every SpamAssassin installtion in a user's chain of email is fixed,
some email will continue to be incorrectly tagged as Spam for many of you.

However, THIS PROBLEM HAS BEEN FIXED AT AMS, and was fixed as of January 1.

What I did, on January 1, in the late evening, was to apply a new score rule:

SCORE FH_DATE_PAST_20XX 0

This effectively "disabled" that rule, so that 2010 mail would no longer be
tagged as Spam.

I've gotten several reports this morning that mail to some lists is still
being tagged as Spam.  I've taken extra steps, such as resetting the 
autowhitelist system built into SpamAssassin, to try to bring things back
to normal.  In the meantime, SpamAssassin may have additional
problems or interactions we haven't discovered yet, and I will be
working with list moderators to identify and analyze any other spam
filtering issues that may be present.

One thing that often happens is that people reply to a message, but
they don't end up removing the "SPAM" wording from the Subject. 
Therefore, that wording can be preserved through the thread and sent
back and forth through the list.  This does not indicate that the
spam system here is still broken, this only indicates that people
are resending the spam tagging in the Subject line, by hitting reply
and not editing the subject.  Please be mindful of this if it occurs.

A happy new year - which is NOT a new century! - to you all.

Glen