Subject: Re: [Asrg] "worm spam" and SPF

From: Florian Weimer <[email-masked]>
Date: Fri, 03 Dec 2004 17:50:21 +0100

* Gadi Evron:

> Seth Breidbart wrote:
>
>> Therefore, an ISP (or other sender) that wishes to retain a good
>> reputation will set its outgoing mailservers to check for the latest
>> worms fairly soon after they're discovered, and throttle mail from any
>> particular sender so not too many worms get out before they're learned
>> about.
>
> And you think that's easy?
>
> This is what the ENTIRE AV _industry_ is about...

I think the crucial difference is that you buy AV products to prevent
malware from *entering* your network.  Outbound filtering is mostly
accidental.

Detection of mail-spreading malware is pretty easy.  I suppose that
detecting infected customers, identifying them, and helping them to
clean their box simply doesn't pay off, from a single ISP POV.

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: [email-masked]
Date: Fri, 3 Dec 2004 19:48:10 +0000 (GMT)

>> Spammers are already sending out worms to their victims. Currently these
> worms install zombies which then send out mails using random sender
> addresses directly to the target MX. If that stops working, these
> zombies will start using the credentials and SMTP server found by
> reading the registry, tricking the user or sniffing traffic. Sure, that
> is a bit more difficult than what they do now, but what they do now is a
> lot more difficult than what they did a few years before, so you can
> expect that they will do it if they have to.

> Absolutely right, and as the guy who raised this point against blocking 
port 25, allow me to say this:

> If spammers send out spam from (usually) just one email address per 
compromised PC, spam will drop to tens or hundreds of messages a day per 
person, instead of hundreds or thousands.

> Still, there are enough drone out there to make up for it, but it is a 
step in the right direction.

> Unlike what was said here, using techniques that will force spammers to 
learn and adapt is not always a bad thing, when pressing them to the corner.

Fine, but if the goal is to force them into a smaller and smaller corner, then 
the fine-grained permissions list idea that I propose, combined with a good 
content filter, together does a LOT more to constrain them (and in a space 
that's MUCH harder to escape from) than SPF and the like do.

And in addition, it doesn't significantly harm legitimate users (the way SPF and 
other "authentication/reputation" schemes do), AND as a bonus, it virtually 
eliminates E-mail as an effective vector for the transmission of viruses and 
worms... thus accomplishing a MAJOR blow against zombie spambots, which create 
such an otherwise intractable problem in controlling spam.

Gordon Peterson                  http://personal.terabites.com/
1977-2002  Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections!  http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.

---

From: Gadi Evron <[email-masked]>
Date: Fri, 03 Dec 2004 21:59:38 +0200

> such an otherwise intractable problem in controlling spam.

Enlighten me with a link to your plan? You make it sound like FUSSP.

	Gadi.

---

From: Matthew Elvey <[email-masked]>
Date: Sat, 04 Dec 2004 17:37:46 -0800

On 11/27/2004 11:28 AM, Vipul Ved Prakash sent forth electrons to convey:

>Nod. This moves the content filtration problem to a different 
>place: on the outgoing servers. If the only way to send out spam 
>is through drones, then all spammers will move to using drones. 
>The conclusion that I draw from this is that content filtration 
>techniques used on outbound mail would have to be of the same 
>level of sophistication as inbound filtration today. 
>
Consider a filter that detects that a mail stream contains outbound spam 
only after the system has sent out, say 100 emails, but does so with 
99.99% reliability.
(SpamAssassin is at least an order of magnitude better than that. Your 
Razor2 scheme alone can do that, right? Note the word 'contains'; it's key.)
At that point, it blocks that mail stream until the system is cleaned.  
That's good enough to make end user mailboxes orders of magnitude 
cleaner than they are today, as zombies send out orders of magnitude 
more than 100 emails.

Have you noticed that several large ISPs have seen the writing on the 
wall and have started offering their users free (bundled) antivirus (and 
anti-zombie) ?  AOL, SBC, to name two huge ones off the top of my head.  
They see that they'll have to be able to disinfect their users if they 
want to ensure their mail keeps getting delivered.  This is the writing 
on the wall.

---

From: [email-masked]
Date: Sun, 5 Dec 2004 17:26:31 +0000 (GMT)

>> The gist of my proposal is that each recipient should be able to establish
> a
> fine-grained mesh which restricts (on a sender-by-sender basis) the types
> of
> mail the recipient is expecting (and willing) to receive from each sender.
>
> THE DEFAULT, for unknown/unspecified senders, would be to allow through
> the mesh
> filter only mails smaller than a specified size (say 25K or 50K bytes
> maybe)
> which contain no HTML and no attachments... thus simple, text E-mails not
> exceeding the specified maximum size.

> The average spam mail size is usually smaller than your proposed size.
which increases the false negative rate of ur proposal

The size limitation is NOT intended to differentiate spam/nospam.  Its purpose 
is to block "large" E-mails from unknown/untrusted people (and which might 
otherwise consume an undesirable amount of Inbox space) as well as to help avoid 
"large" mails which could contain viruses/worms/etc.  The default "no 
attachments" rule helps a lot with that, (as does the "no HTML" default rule) 
but putting the size limit (and setting it appropriately) also helps avoid 
spammers trying to evade that by using non-attachment "embedded attachments" of 
some sort.

As a recipient, I simply don't need (or want) to accept arbitrarily large 
unsolicited E-mails from just anybody.  If they want to send me something large 
or cumbersome, they can contact me FIRST and request permission for the bulkier 
or riskier mail..

> What about the sales and customer care dept. of organizations who receive
mails in bulk from customers, clients etc.

My permissions-list proposal would not limit that AT ALL, as long as those 
E-mails were below a certain size (which the receiving organization can set as 
appropriate for their case) and as long as they meet the attachments/HTML 
criteria they have set.  I think you'd have a hard time arguing, for example, 
that such "sales and customer care" departments would need to accept (at least 
not as a first contact!) E-mails containing Javascript decryptors, ActiveX, .SCR 
or .EXE files, mail of arbitrary sizes (say 2Mb?)... the list goes on and on.  
And that's one of the big differences between my proposal and a typical 
"whitelist" proposal... my proposal specifically provides for a significant, 
useful (while safe) type of E-mails for use in establishing initial contacts 
with people.  (And that E-mail still would need to satisfy the second-stage 
content filter, based on other criteria the recipient can control, to get 
through for delivery).  

> How user friendly will it be when the admin or user has to configure
settings for each sender.

That's an issue to be determined by the implementing software.  I would expect 
quite a diversity in dealing with those issues, depending on whether the 
software involved were intended for direct (fairly clueless) end-user use or for 
"serious-power-user" types.  

Anyhow, there doesn't HAVE to be any single standard for how to deal with that 
stuff (and nothing says the recipient couldn't change their settings over time, 
either).  You might T-can the objectionable stuff without a whimper, you might 
quarantine it with a periodic summary of questionable stuff being held, you 
might deliver it (especially during the initial phase of the implementation) 
with a way for the user to say "Yes, this sender is okay, let future stuff 
through from them if it looks 'like this' one."  

Likewise, the software might be clever enough to notice that when stuff is sent 
to quarantine, some senders are particularly persistent (and MAYBE think they're 
legitimate) and those might be something to call to the attention of the 
recipient, to verify whether it SHOULD in fact be getting delivered, or if maybe 
it's a known/persistent spammer which the recipient doesn't even need 
quarantined, where that stuff in the future can go straight into the trash.

> It sounds more like whitelisting with restrictions.

It certainly has whitelisting characteristics.  The major differences are:

  1)  there is an established (although NOT necessarily standardized) class of 
mail which can come from anybody and be delivered (subject, of course, to 
satisfying the second-stage content filter);  (those knowing they're sending an 
intial-contact E-mail would want to use 'good netiquette' in hopes of satisfying 
that 'politeness' standard);

  2)  the whitelisting would provide a fairly fine mesh, so that the recipient 
could fairly closely define the type of mails they expect (and are willing to 
accept) from each given sender;

  3)  the unsolicited stuff, from unknown/untrusted senders, would have to 
satisfy rules which would preclude use of most evasions, tricks, and subterfuges 
that spammers normally are able to employ to deceive or evade content filters;

  4)  the "no attachments/no HTML" (by default) rule would virtually eliminate 
E-mail as a viable propagation vector for viruses and worms (since almost nobody 
would in practice set ANYBODY with permission to send them executable, encrypted 
ZIP, or other such 'dangerous' attachments).  This would put a MAJOR crimp in 
the ease of building armies of zombie spambots.

> ...restricting mails based on size or format will only produce more False
Positives.

Again, you apparently just don't 'get it'.  

The fact is that (for example) I simply AM NOT INTERESTED in getting 
HTML-burdened E-mails from people I don't know and who haven't cleared those 
with me in advance.  There is VERY little I want or need to get by E-mail that 
can't be sent to me as (safer) plain ASCII text.

Once I *do* agree to accept formatted E-mail from somebody (and if they have a 
DAMNED good reason for it, which I concur on) then I *still* probably don't want 
them to send me ActiveX, or executable attachments, or cookies, or "big" mails, 
or (possibly malicious) scripting, and things of that sort.  I don't want 
obscured/misrepresented hyperlinks that are supposed to look like 
http://confirm.sunbank.com but which in fact point to some rogue system in China 
or Romania.  

Content filters can do a pretty decent job of detecting patterns in E-mail text, 
IF there is in fact text there to analyze.  Text-as-image, decrypting text, and 
so forth are among the tricks that spammers use to confound and evade content 
filters;  the solution to those evasions is simple... the presence of the 
evasion trick is itself prima facie evidence that the message IS IN FACT spam 
(and especially after you've had the ability to make exceptions for trusted 
senders who you might EXPECT to send you stuff of a particular kind).

---

From: [email-masked]
Date: Mon, 6 Dec 2004 04:54:37 +0000 (GMT)

>> OK, thanks.  :-)  It's pretty clear that my proposal really IS that simple, 
and really DOES go a LONG way to solving/reducing the problems of spams, worms, 
and viruses.

>> I *do* urge the use of a good antispam content filter IN CONJUNCTION WITH my 
proposal.  My proposal, however, *greatly* improves the efficacy of the content 
filter.

> Since your proposal can be usefully implemented on a small scale (unlike
SPF, which will only work if (almost) everybody uses it), a few
questions:

> 1) Have you implemented it? Is the implementation available?

I have implemented significant parts of it.  I haven't implemented more of it 
mainly because at the moment, the level of the problem that I personally have 
*here* is brought to a quite tolerable level (for now, at least) by the stuff I 
presently have in place.

I continue however to add more things to my incoming mail processing system, and 
some of those I would classify as "experimental".  Two recent additions are a 
pair of programs that I use to try to "immunize" me from recurring spam 
referencing known disreputable sites and domains.  

When spam does slip in past my present filters, I append it to a "spam.txt" 
file.  The first of these two recent additions locates, canonizes (basically, 
removing obfuscatory subdomains), and tallies domain names (and IP addresses) 
referenced in the spam E-mails.  These are deduped, sorted, and written to a 
file "spamdomn.txt" which I periodically can look at to see if I see "good" 
domains that got referenced somehow (the last good one was amazon.com, for 
example, which clearly I don't want blacklisted).

Once I've checked over the list in spamdomn.txt, the second of the two programs 
takes the file and adds the domains I wish to block (into the "right" places, 
based on various rules and conventions I've set) to the HOSTS file maintained 
and used by my mail server.  This essentially not only just blacklists the 
domain from E-mails, but prevents me (or other software on my system, with or 
without my permission) from accessing those disreputable domains, including 
using a Web browser (thus also blocking any popups, cookies, hyperlinks, 
embedded banner ads, or other such nuisances using those domains).  

My present E-mail processing system also blocks incoming E-mails which reference 
those "blacklisted", disreputable domains.

One change I'm going to add to my incoming mail processing system, for example, 
is to trash the spams which put the spam into the HTML part and where the 
text/alternative part of the message is just saying to "get a capable html 
mailer".  (AS IF!!!!!  :-) )

One of the things my current system here does is to simply discard *all* 
HTML-burdened alternative attachments in my incoming E-mails, and to furthermore 
eliminate a large percentage of any HTML tags that try to slip through in the 
"plain ASCII text" E-mail body as well.

> 2) Are you using it yourself? 

As I guess I just explained, I'm using some elements of the proposal, along with 
other proprietary filters and such (in part because I presently don't use a 
separate "premade" content filter ("Spam Assassin" or similar) so my system 
provides both functions, sort of...)

I hope I can be excused for not completing a full-blown, polished, 
ready-to-take-to-market professional application for my own, personal use.  (I 
*will* say for the record, though, that my present filter was architected 
internally with an eye to it being useful in a corporate setting, so it provides 
many of the hooks and internal structures to allow different users with multiple 
different POP3 servers and accounts (per user) and different mail accounts and 
different rules, etc etc).  So it WOULD be usable in a multiuser/multidepartment 
setting, with relatively little effort.  It presently runs as a background job 
on my Windows 2000 server, and accepts asynchronously issued commands (such as 
"exit", "reinitialize certain global tables", etc) issued from elsewhere on the 
network.  The system is primarily implemented in SPITBOL (a high-performance 
programming language with special talents for powerful pattern recognition and 
textual manipulation).

One of the features I'd class as "experimental" in my present system I'm using 
is a "gibberish detector" which is designed to detect and defeat spam that 
contains randomly generated gobbledygook (those spams designed to evade or 
defeat other "clever" types of adaptive filters).

My present system also identifies and removes most all cases of those annoying 
and repetitive "free mail provider" inline ads and such.  

> If so, how much much did it reduce the amount of spam, 

I don't really have good statistics on that, in part because it just hasn't been 
worth my time to collect all the statistics and compile the reports.  I have 
detailed (very!) logs of what the mail filtering system has done, but honestly 
have done relatively little to analyze those.

> ...how much time do you spend tweaking your permission
lists, how many people claim that they can't send mail to you?

At the moment, I have just under 2000 messages that have been sent automatically 
to my "spam/quarantine" folder, this covering about since the middle of August. 
 I've had a few isolated complaints about difficulty sending me mail, although 
in most cases investigation turns out that they either misaddressed it or the 
mail got improperly bounced by temporary ISP issues or the like.  

I spend relatively little time tweaking the filter and whitelist configurations, 
probably less than ten minutes a month.  I periodically sit down and devote some 
time to programming, such as this most recent program I wrote to add the 
disreputable domains into (at the suitable locations in) the HOSTS file, usually 
when I feel motivated to do that sort of thing.

Again, as long as it's strictly for my personal use, the payback simply isn't 
there for me to polish the thing as a serious, professional-grade "product" 
suitable for widespread installation.  Accordingly, the way this presently works 
is rather keyed to how my systems are configured here.  I do have a consulting 
client in Wisconsin who I've set up with an incoming mail processing "bot" type 
system, which in some ways is similar to what I have here for myself, although 
their needs were rather specialized and accordingly their system ended up being 
quite different in the end than what I use here.  (Their system is, however, 
like mine, implemented in SPITBOL and runs as a background job on one of their 
Windows 2000 Professional systems).

I would enjoy pursuing the polishing of the system, and elaborating it to full 
release quality, so if you know of someone interested in funding such a project, 
with an eye to producing a commercial product, definitely, send them my way.  
:-)

FWIW, by the way, over the past three or so days, my incoming mail filter has 
used a total of a little over 10 minutes of CPU time, this on an Athlon (2000+?) 
single-CPU Win2K server system.  This is without any particular effort being 
made to optimize or otherwise study the execution time the various components 
are using in detail.  I don't consider, at the moment, that scaleability would 
be a big problem.

> 3) Have you deployed it at a scale where you couldn't explain it to
everybody in person (E.g., a company with ~100 employees, a small ISP,
etc.)? Same questions as above, plus: How many support calls do you get?

At the moment, the incoming mail filtering system I use is strictly for my 
personal use in my home office.  The related incoming mail processingbot system 
I've installed at my consulting client is in daily use, in an office of about 15 
persons, and I don't remember the last time I got any kind of support call or 
question about it.  It just sort of sits there and runs.  :-)

---

From: [email-masked]
Date: Wed, 8 Dec 2004 02:29:29 +0000 (GMT)

>>  What my current mail filter here does is to first strip
> HTML-burdened alternative attachments, and then to further strip the
> majority of HTML tags that it finds in "plain text" parts of the
> message as well.

> How does the scheme deal with mislabeled attachments? Spam messages
don't play by the rules. 

The current implementation I use here basically handles that by:

  1)  IF there is a plain text version and an alternative HTML version of the 
message body, the alternative HTML version is simply removed;  the plain text 
version is retained.

  2)  HTML tags (most of them, at least;  I retain those that might be useful in 
determining spam origin or otherwise tracking back to the perps) are stripped 
from the message body, for essentially ALL types of message body.  So even if 
the part type is mislabelled, most HTML types will still be removed from it.

> ...And some popular mail programs virtually
ignore attachment types altogether and use the filename extension, or
rely on defaults known for the target mail reading software.

Sure, and presumably those will still work that way once they've picked up their 
(filtered, processed) mail.  The way they handle it doesn't really impact the 
filter.

> How does the scheme identify HTML in plain text? 

It looks for things that "resemble" HTML tags, given a list of common types and 
what those types look like.  It also uses a variety of rules to identify most 
cases of "bogus" HTML tags (that will end up being ignored when the mail is 
read) that are only there to break up keywords that otherwise would trigger 
content filters (and it removes those, too).

Although I don't think it makes a lot of sense to describe in detail what those 
rules are (and we must presume that spammers are monitoring our discussions 
here) it's fair to point out that SNOBOL/SPITBOL offers FAR more sophisticated 
and powerful pattern matching than the braindead reg-ex type pattern matching 
more typically found in Perl and similar.

> Does it correctly recognize a discussion about HTML tags from a text using 
HTML markup?

Not at present, although (again) if you had a correspondent or a mailing list 
where that might be expected (and desired) you could easily enough whitelist 
those specific senders to send you that sort of message content.  It's probably 
fair to presume that if you're sophisticated and enlightened enough to be 
discussing such points, you're less likely than the average Joe (or Gertrude!) 
to be confused or tricked by them.

>>> The only sure way to protect users against HTML attachments is to
> prohibit them from using mail software which displays HTML,

>> There's no point in "prohibiting users" from doing most ANYTHING.
> The whole point is that users should be able to do anything they CAN
> do now, but encouraging them to do it selectively, for those cases
> where they trust the senders.

> But as I understood it, your scheme is supposed to block the receipt
of email containing HTML from unknown/untrusted senders originally,
and only allow such email through after the sender is trusted. 

Right.  Now, what one DOES with such "untrusted mail" is something the receiving 
user ought to be able to configure... whether they blindly T-can it, or 
quarantine it, or flag it, or whatever.

> ...If you let HTML through to begin with (assuming you can always identify 
it), it's much less effective?

Well, not necessarily. 

For example, one thing that one can do (and again, no reason why this can't be 
configurable depending on how the recipient wishes to deal with such things) is 
to simply replace the pointy-brackets in HTML tags with curly-brackets before 
finishing with it.  This allows the recipient, IF desired, to examine the 
particular tags that the filter chose to pass through (and nothing says it can't 
still simply remove other stuff).  And there could be a suitable 'goback' 
program provided (or a retrievable, unmodified copy archived somewhere) that 
could allow the recipient to go back to the unmodified original copy (or 
whatever) should they want to do that.

Anyhow, by letting {a href="gotcha.phishing.ru"}security.sunbank.com{/a} through 
I think you'll agree that the user is pretty well protected.  Meanwhile, things 
like font sizes and colors and boldface and the like, while arguably interesting 
if you like stuff like that, can usually be simply stripped from E-mail messages 
without harming their actual information value.  And likewise, "junk" HTML tags 
like {qewhjwhfqeasdf} (using pointy brackets, of course) can of course be 
deleted from incoming mails without affecting their content, too.

>>> What you perhaps don't realize is that an attachment marked plain
> text but containing HTML tags is often displayed as HTML by mail
> reading software anyway. Some software reads plain text, looking for
> anything that resembles a web address and generates a clickable URL
> (thereby turning the plain text into HTML).

>> Fine, but at least in that case the URL will NOT be spoofed or
> misrepresented, right?  :-)

> The art of misrepresenting URLs to the public is called phishing, 

Well, that's one USE for it, but that's certainly not the ONLY use for it.  A 
more common use is simply to obscure the URL (using any of a whole variety of 
techniques) in an attempt to evade filters which try to block "disreputable" 
domains.  

> ...and is fairly well developed ;-)

Of course.  Anyhow, limiting the ability to HIDE such obfuscations makes it far 
more apparent that the URL is not trustworthy.

>> Clearly, but the point is to force the spammers into areas that are
> VERY, VERY gray (and which for most users, simply don't exist at
> all).

> The fact that most users aren't in the gray area is irrelevant. If
(your, any) scheme lets through too many spams, it is as useless as if
spams weren't being identified in the first place.

Not so.

First, spam blocking is only ONE of the functions performed by my approach, 
although it's certainly one of the notable ones.  As (even MORE?) important is 
the fact that it shields users from the GREAT majority (and perhaps virtually 
all) viruses and worms, INCLUDING those which set up armies of zombie spambots.

Second, "too many" is a very squishy number.  If it blocks 90% of spams and 99% 
of viruses and worms, is it "useless" because some spam still gets through?  I 
don't think so.  Likewise, it doesn't particularly bother me if a few relatively 
benign spams get through now and again.  (At SOME point, spams look a lot like 
minimal messages from friends that I *might* want to still get).

Third, remember that my approach is intended to be used ALONG WITH a good 
antispam content filter (and, it's IMPORTANT to note that antispam content 
filters will be a LOT more effective on the type of mails that have already 
passed through a filter such as mine, since most of the tricks used to evade and 
defeat them have already been blocked).

> However, the amount of spam passing through depends solely on the spam
filter and how clever the spammers are. 

Most of their "clever" depends on tricks based on attachments and HTML.  By 
slashing the utility of those, spammers are denied most of their favorite 
deceptions.

> So discussions naturally gravitate toward spammer tricks.

Of course.  The way we're going to defeat them, though, is by changing the 
rules, and I think we can do that in a way that's minimally (or not at all) 
destructive to legitimate users while still creating very difficult gauntlet for 
spammers to try to navigate their way through.

>>  I'm not naive enough to presume that ANY one approach will work for
> EVERYBODY or for ABSOLUTELY ALL possible (past/present/future)
> software.  I don't think that any other approach I've seen proposed
> will protect so many people, so automatically, with so little
> interference with their existing methods and systems, and with so
> many other advantageous effects.

> I'm not suggesting you are. However, current state of the art systems
achieve 99+ percent success, 

...I guess that depends on how you define "success"...

> ...and for organizations with very large userbases, even that isn't enough.  

In any case, my proposed fine-mesh permissions system would eliminate a LARGE 
amount of the stuff that still is getting through.

> ...Unless your scheme can claim these sorts of numbers for nearly all people 
to begin with, then it isn't worth trying to convince people here. 

I don't agree.  I don't think we have to use a SINGLE tool and a SINGLE approach 
(and said as much above in fact).  

And in particular, virtually NONE of the other approaches we've been discussing 
here have the effects of substantially reducing viruses/worms (and those are how 
these zombie spambots, that in fact prove SO problematical for these other 
non-starting approaches that so many people seem so focused on, typically 
propagate).

> Most people on this list are only interested in systems which can deal with 
vast numbers of messages with little extra work.

Most people here seem to be fascinated by (and fixated on) technologically 
complicated solutions that require a major transformation of the planet's E-mail 
systems.  While it's tempting to look at those, and there are admittedly 
advantages to trying to truncate spam as close to the entry point as possible, I 
personally think it's more important to squash it (at least) before the end user 
is greatly inconvenienced by it.  And, of course, at SOME point of success in 
preventing it from being seen by anybody, it will cease being profitable for 
spammers and they'll simply go off to pursue some other scam instead.

There's nothing in my proposal that prevents it from being used in dealing with 
"vast numbers of messages", especially if it's implemented at the recipient 
level (Outlook/Outlook Express/Eudora/Pegasus/whatever).  There's PLENTY of 
end-user computing resources, if used responsibly, to deal with that stuff.

[snip]

>>> Moreover, you might not realize that SpamAssassin is bundled with
> bulk mail sending software precisely so that spammers can design
> their emails against SpamAssassin and tweak them until they pass the
> tests performed by SA, and only then start spamming.

>> Of course.  Most of those deceptions [mis-]use HTML or attachments.
> That's one big reason for encouraging users to limit the use of such
> features only to trusted senders.

> I don't seem to remember whether you explained how to initially deal
with (untrusted) people who (unwittingly, incapably) send exclusively
HTML burdened messages.

Probably, during a transition period a recipient will choose to receive (at 
least SOMEWHAT more) HTML than they maybe will allow in the long run.  Most 
people (clueless AOL and HOTMAIL types) probably don't send (even INTENTIONALLY, 
let alone by default) anything more complicated than fonts, sizes, colors, bold, 
underline, and the like.  Happily, those are pretty benign, for the most part.  
As they get more familiar with the system, recipients will have whitelisted 
their familiar senders, and might choose to clamp the default "allow" filter 
somewhat tighter than they initially did.  

Also, as an approach like mine becomes more widespread, groups like AOL and 
HOTMAIL that presently casually and sloppily turn on HTML-burdening with nary a 
thought might be inclined to be more responsible in that regard, realizing that 
HTML-burdened mail is less likely to be delivered in a more spam-defensive 
world.

Again, my approach really is a FINE-MESH filter, where it doesn't have to be 
(and indeed, SHOULD NOT be) as gross as "HTML" or "no HTML".  (Although 
honestly, I _personally_ would be inclined to block or at least quarantine HTML 
mail overall, if from a non-trusted sender).  But that should be a recipient 
option.

>>  Well, it WOULD be sufficient to block "large" (as defined by the
> recipient) messages coming from unknown senders.

> It is debatable whether large attachments are in fact an indicator of 
friendly (ie nonspam) messages. 

It doesn't really matter.  The FACT is that most folks have limited Inbox space 
and are NOT willing to devote unlimited amounts of it to people they don't know. 
 ISPs tend to have 5-10Mb Inboxes (although we're seeing some recent 
improvements in that area, fortunately) but still often limit incoming 
individual E-mail messages to some smaller size.  If someone checks their E-mail 
every day or two, they obviously don't want an E-mail to arrive 30 seconds after 
they look, which fills their inbox and causes all their other mail to bounce 
until they get around to checking again.

But there are NUMEROUS reasons to not want big messages from unknown folks.  
Limited Inbox size, more likely virus/worm content, and so forth.  It's just 
another (useful!) tool that can and should be made available in the war against 
E-mail nuisances and annoyances.

> Spammers tend to send small messages, unless they're trying to be clever, 

Sure, and that's nice... at least they aren't then wasting huge(r) amounts of 
bandwidth.

> ...while worms are as you pointed out a bit larger. 

Usually a GOOD bit larger... generally 45K to 200K, by the time they're encoded 
for E-mail attachment.

> But the truly large messages are sent by clueless people who will
write two lines in Word, include a BMP image, then attach the whole
document to an empty email message and send the lot...

Fine, and if I have a correspondent I want to hear from that tends to be 
belligerant about stuff like that, I can whitelist them and continue to let them 
do that.  Hopefully, before simply sighing and accepting that, I can ask them to 
please be more responsible in the future.  Meanwhile, we don't do ANYBODY on the 
Net any favors to bend over backwards to support and continue that kind of 
irresponsible practice.

>>> There are two avenues for worms under your scheme. Worms can get
> smaller in size, or worms can stay the same size and spread via
> trusted senders, by looking for regular correspondents in address
> books etc. Fifteen years ago viruses were less than 1K in size,
> there is no reason beyond lazyness why worms need to be around 25K.

>> All this use of heuristics can be evaded by various strategies.  If
> worms shrink to less than 25K, it might be that a recipient will
> shift their "allowed maximum unsolicited E-mail size" to 15K, or
> 10K, or maybe even 5K or something.  THAT IS *THEIR* CHOICE, and
> they aren't constrained to maintain the SAME choice.

> None of these suggestions is preventative, only reactive. You're
suggesting that attacks on your proposed spam defense should be
best handled by an arms race between the spammer and the user.
The spammers still have the initiative.

Ultimately, if we do our job well, we'll force the spammers (and viruses/worms) 
OUT of the E-mail arena entirely (much the way that diskette boot sector viruses 
have virtually gone extinct);  I suspect they'll retreat to malicious Web sites. 
 But that is a different war, and we'll proceed to that one (undoubtedly) in due 
course.

But I disagree with your claim that my approach isn't "preventative".  For most 
users, who install the system with default rules and who aren't deceived into 
re-enabling executable attachment delivery, they will PREVENT the delivery of 
executable attachments, PERIOD, and that doesn't depend on constantly updating 
antivirus files (which ARE reactive and INVARIABLY lag the appearance of new 
viruses).

>>> The devil is in the details, because the high level concepts in
> which you describe the scheme do not map perfectly into the low
> level concepts required for implementation.

>> There is no one SINGLE set of details required.  There is a wide
> leeway in terms of how the various aspects get implemented, and in
> fact this is an advantage...  it allows different companies to
> differentiate their products, while at the same time creating
> distinct products which probably will not share common potential
> weaknesses that a spammer might exploit.

> Right, various elements of what you propose (scanning for HTML,
blocking attachments) are widely implemented in many different
anti-spam products. 

Right, although I'm not aware of ANY of those which use a fine-grained 
permissions approach and which is based on a sender-by-sender basis.

I'm sure that fact will change, over time, since I'm convinced that it's a VERY 
sound approach and eventually other companies, I'm quite sure, will come to the 
same conclusion.

> One could argue that companies already have several pieces of your puzzle and 
have mixed and matched them as they like.

And that's a good sign.  Hopefully they'll get the pieces together in the right 
way.  

One BAD sign I've seen is that Microsoft (sigh) seems determined to build their 
antispam defenses into Exchange rather than Outlook/Outlook Express, which is 
terribly clueless;  the spam problem (like the virus/worm problem) will NEVER be 
solved if we only work on it at the corporate/enterprise level.  It needs to 
work in the mass-market tools that clueless individual Net users can use and 
understand.

>>> Some problems with whitelists: 

>>> Once you've whitelisted all your friends and colleagues, you depend on
> how good *their* spam defenses are. If they receive a worm, it can
> travel to you freely. 

>> No, ABSOLUTELY NOT, since my proposed "whitelist" is NOT just a simple yes/no 
> whitelist.

> You've talked about examining a limited set of properties such as existence of
attachments and of HTML tags. 

That's good on defaults, but the goal is for a much finer set of permissions 
than that.  Executable/PIF/scripting attachments are clearly a LOT less safe 
than GIF or text attachments are.  Javascript or ActiveX HTML tags are clearly a 
lot less safe than boldface or italics markup tags are.  Again, I think the real 
goal is to whitelist correspondents based on what you EXPECT and TRUST them to 
send;  it's hardly as if there's NO choice other than "guest" versus "root".

> Your whitelisted friend may have a trojan or worm which will send plain text 
advertisements to all people in his address book.

Sure, and if that "advertisement" makes it past the additional content filter, 
it might actually get delivered.  Again, I don't think it's possible to 
eliminate ALL illegitimate mail.  (I have one friend who tends to come home 
drunk on his butt and send almost completely illegible E-mails at three or four 
in the morning;  I don't expect or require that my E-mail filter block things 
like that... it's part of his personal charm and personality, I suppose.  But as 
long as I can reduce the really egregious volume of wasteful and intellectually 
insulting spam/worms/viruses to an occasional message or two, I personally don't 
have a big problem with that.

> If some combination of tags and attachments can get through, it will
get through eventually.
 
Sure, and as long as the volume of that is negligible, I don't consider it a 
huge problem.  I'm going to receive, after all, some amount of LEGITIMATE mail 
which will be less interesting and useful.

>>   1)  Most of my "friends and colleagues" don't REQUIRE explicit whitelisting 
> because the stuff they send me does not exceed the default (very safe) rules.

> That's great for you, but these issues must be addressed when proposing a
spam solution for the masses.

I think that'll be true for the masses, too.  Most people do NOT receive EXE 
files or JavaScript or ActiveX or JavaScript decryptors and obscured URLs in 
legitimate mail.  One can, I feel, make some pretty general rules which block 
(or at least tag, whatever) a lot of stuff without greatly inconveniencing 
recipients (but that's why THEY must be in control of the filter).

>>   2)  The ones I *would* whitelist don't automatically receive "wide open" 
> permissions.  I would allow individual senders to send me the type of stuff 
that 
> I might expect THEM as an INDIVIDUAL to send.  NONE of them, for example, are 
> likely to be permitted to send me (say) PIF attachments.  :-)   Likewise, VERY 
> few (if any) would be whitelisted to send me (say) ActiveX or cookies or 
> scripting.

> How do you recognize a PIF attachment? 

Clearly, by extension... the same way the operating system does.

> ...What if it's inside a ZIP file?

Then it's a ZIP file (which might contain one or more files inside it).  I don't 
need (or WANT!) to receive unsolicited ZIP files, especially if they're 
encrypted.  If someone has a legitimate need to send me a ZIP file, they can 
arrange that with me in advance... and I'll whitelist them to allow me that 
specific type of content.

> Inside an LHA file? Inside a zipped HTML file containing an embedded
OBJECT tag with a particular class ID pointing to external data? 

Likewise.  I'm willing to simply DENY anything suspicious like that to anybody 
who doesn't bother to get my permission for such stuff in advance.

> There are innumerable ways of sending malicious code with various levels of
obfuscation. 

Sure, and by simply blocking attachments and HTML from unknown/untrusted senders 
you eliminate basically ALL of those (at least for E-mail!).

> Spammers don't use them because the simple stuff works
quite well for now.

Sure, but (again) the basic approach of denying HTML and attachments from 
unknown senders eliminates basically ALL of those, in ONE FELL SWOOP.  Maybe you 
can come up with some more devious scheme, but then my feeling is that one can 
deny THAT class of deceptions in unsolicited mail from unknown senders, too.

>>  One of the advantages of my approach is that, in fact, people using
> it would block VIRTUALLY ALL incoming worms and viruses, EVEN IF
> they came (ostensibly) from people who they otherwise know and
> trust.

>>> And worm writers realize this, so it is one of their priorities.

>> You greatly underestimate the difficulty of evading the type of
> permissions-list filter such as I'm proposing.  Perhaps you still
> don't understand my approach; I'll be glad to try to explain it
> further.

> I think I sort of understand your approach. Your filter scans an
email's MIME structure, looking for particular types of
attachments. It also scans for some simple tags such as <FONT>, <A>,
etc., in the plain text attachments.

Likewise, of course in decodable text attachments (another common spammer 
trick).

> [Not all email uses MIME, not all MIME using email is correctly
generated. Not all MIME attachments contain what they are labeled as
containing, not all mail software interprets attachments according to
the labels either, but others do]. 

Of course, and that's why I remove HTML tags also from "plain text" attachments 
as well.

> If the sender is unknown, the mail is blocked if it contains the tags
you scan for or attachments labeled as forbidden. 

Yes, although my CURRENT implementation simply REMOVES the HTML alternative 
attachments entirely, in addition to removing HTML tags from remaining portions 
(including supposedly "plain text" portions).

> If the sender is known, the mail is blocked if it contains tags or attachments 
which are prohibited for that sender.

OK, although I'd change that to "not allowed" rather than "prohibited", because 
in general I think it makes more sense to specify (perhaps automatically, for a 
future implementation level) ALLOWED types.

> [It blocks mail from unknown people who use HTML or attachments or
large mail bodies, which is a sizable proportion of the email sending
population. People need to know enough about email structure to maintain
complex sets of permissions.]

Currently, I don't block mail that contains HTML attachments... as you point 
out, virtually all AOL mail and most HOTMAIL messages have HTML parts.  (OTOH, 
those tend to at least have 'well formed' HTML and MIME types...).  Instead, my 
current implementation simply strips the HTML portions from those messages (and 
HTML from the remaining parts) and proceeds with delivery.

As for knowing about "complex sets of permissions", that would depend on how the 
implementing software is designed.  Obviously, one approach would be that the 
recipient user could see either a summary of the blocked mail(s) (who it's from, 
the subject line, who it was addressed to, how big it was, why the filter 
blocked it, etc etc) and be given an option to "allow messages like this one 
from this sender in the future" which could open the filter rules and 
permissions for that sender to allow (just) those features through for future 
mails from that sender.  That doesn't really mean that the recipient has to 
understand each of those individual permissions.

> Blocked mail can be inspected individually, possibly by grouping 
according to some heuristics based on header fields or message structure.

Right.

> [For spam, header fields are both faked and randomized to thwart easy
grouping, as that would also make filtering easy. 

Fine, but at least you can see if the E-mails LOOK like those things you EXPECT 
to receive... say, mailing lists you know you belong to.

> ...If you receive 500 spams a day and wait a week to check them at your 
leisure, you have to wade through 3500 messages looking for the elusive 
incorrectly blocked email - 

If you had to check all of them individually, yes.  Again, that's not what I'm 
seeing here... I've got about 2000 E-mails quarantined since mid-August.  And 
there are some types of detection rules that I'm willing to provide global T-can 
rules on... such as non-listed senders who send me mails containing URLs like 
the-rxsite.com or cyberbargain.biz or hotpersonalads.info.  Honestly, I don't 
need to look at those E-mails, I'm reasonably sure that they're not legitimate 
(or, at a minimum, not important).

> ...that's still a chore, once a week, and the unfortunate sender
has been waiting for a response for a week].

If something is IMPORTANT and urgent, they can pick up the phone and call me.  
:-)  Occasional E-mails can (and do) go astray.  E-mail has NEVER been a 
guaranteed-delivery medium.

> Based on the above objections, I'm not convinced your system is
competitive (accuracy wise) with other systems for large
organizations. For individual use by power users, it appears to me to
be less work than custom keyword filters, but more work than personal
Bayesian filters.

Well, for one, it works FAR better than Bayesian filters do, since so many 
spammers devote so much effort to messing up those types.

More to the point, I'm not EXPECTING that I will *ever* convince everyone, and 
maybe not even you.  I know what *I* think works, and I'm convinced that it will 
work for most people (and better overall, than competing approaches).

> This is just my opinion based on our discussion, of course. 
For a serious evalutation, nothing beats physical deployment at 
multiple locations, and you at least pass the test of using your own
system yourself.

Thanks (at least to the extent that my present experimental implementation 
models portions of the more complete and polished implementation that I envision 
and propose).

---

From: Laird Breyer <[email-masked]>
Date: Thu, 9 Dec 2004 12:05:46 +1000

Just a few points below, as the last email was becoming so long
that it's hard to follow.

Regarding HTML and MIME obfuscation tricks: 

I still don't agree that you are proposing things that aren't easily
evaded by determined spammers (explained below). When asked about
mislabeled attachment types, you suggest (unless I misunderstand) that
you can simply ignore the problem and scan the body directly for the kind of
regular expressions associated whith HTML tags and remove them.

Besides destroying (and in the process subtly breaking) the message
contents, which has serious user privacy issues, it won't stop all
HTML from bypassing the filter.

For example, message parts can be encoded in several formats
(UUENCODE, Base64, Quoted-Printable etc) with arbitrary levels of
nesting. (e.g. a message/rfc822 containing a message/rfc822 containing
a message/rfc822 containing a message/rfc822 containing a
message/rfc822 containing..., with each layer encoded differently. And
the very first layer might be labeled a GIF file.)

Whatever regular expression for an HTML tag you come up with, it can
easily be made unrecognizable. Even the interpretation of HTML tags
can be redefined on-the-fly if it comes to that. But say you keep up
to date with tricks designed to make a complex payload look innocuous
to simple minded filters, then you are on the losing side of such an
arms race, because a spammer need only change their email, while you
need to patch your software with new regular expressions and redeploy
it to all the customers every time.

Note also that it is straightforward for spammers to deduce the checks
made if they have access to your software (as they invariably will if
it becomes widely deployed), so there is little point in not
discussing specific parsing techniques publicly. It only makes
discussion imprecise and harder to see any flaws.

Some direct points:

You argue that perhaps the most important overall function is to block
the spread of viruses, worms and zombies, as these are the current
enabling technology. If so, you should address that problem directly,
as it has much wider scope than the "attachment" problem.  

Blocking attachments, if widesread, will only achieve that the payload
is moved from the email body to an external server. Users are then
tricked to open an external connection which downloads the malware in
any of a wide variety of ways, and still sends spam from then
on. Meanwhile, in the process you destroy the user's reasonable
expectation that their email is delivered as-is, unless they are in
some first class relationship with you.

Another issue is the use of your system in conjunction with a content
filter. If you remove/modify the mail content before passing it to a
content filter which is expected to handle the hard cases, you may be
shooting yourself in the foot. Modern content filters often have many
rules which are optimized to work together, but are not necessarily
optimized to work on mangled email.

A few points about "Bayesian" systems: 

To my knowledge, no successful attack has been performed on such
systems yet. There is a lot of garbage in mail to try to pass through
the statistical filtering, but just like you look for nonsense tokens
as an indicator of spam on a case by case basis, such nonsense tokens
if present easily tip the balance toward spam in a statistical filter,
automatically.

In some ways, these systems are a generalization of where you are headed.
For example, where you have code such as "if rule X is triggered or rule Y
is triggered" (with rules X and Y being statements about email structure or
presence of HTML etc), a Bayesian system will put weights on rule X and rule
Y, combining the weights to obtain a belief about the message. But that is
for another discussion.

-- 
Laird Breyer.

---

From: [email-masked]
Date: Thu, 9 Dec 2004 20:00:21 +0000 (GMT)

> I still don't agree that you are proposing things that aren't easily
evaded by determined spammers (explained below). 

I'll deal with those points to them as you raise them.

> When asked about mislabeled attachment types, you suggest (unless I 
misunderstand) that you can simply ignore the problem...

I'm not IGNORING anything.  But that doesn't mean that you necessarily have to 
believe everything that E-mail tells you, either.  :-)

> ...and scan the body directly for the kind of regular expressions associated 
with HTML tags and remove them.

Not quite.

First of all, I am *not* using "regular expressions" (a basically braindead 
excuse for pattern matching, which dates back to the days of early, primitive 
Unix systems and mechanical teletype terminals).  SNOBOL and SPITBOL 
(programming languages;  SPITBOL is to SNOBOL sort of what Turbo Pascal is to 
ordinary Pascal) allow for MUCH more sophisticated pattern matching than is 
possible using reg-ex type patterns.

> Besides destroying (and in the process subtly breaking) the message
contents, which has serious user privacy issues, 

Well, yes and no.

Agreed that if the mail is PGP-signed or something, then changing the mail 
contents will (of course) show up as a "changed" message body.  Frankly, that is 
a price that *I* am very willing to pay... a lot of the stuff that I get in a 
lot of my incoming E-mails is simply repetitive and annoying, and I'd rather 
simply have those kinds of things just disappear from the mail that arrives 
here.  Obviously, a different recipient might make different choices than I 
would, and that's part of why my system is designed for the RECIPIENT to be able 
to control, in a finegrained way and based on who the sender was, what it does 
and doesn't do.

> ...it won't stop all HTML from bypassing the filter.

Maybe, although:

  1)  it will prevent MANY kinds of (recognizable) HTML from being passed 
(including, hopefully, kinds that are likely to be dangerous or even just 
confusing);

  2)  it can transform the remainder so that ANY OTHER program is unlikely to 
recognize it as HTML and thus act on it (for example, by changing pointy 
brackets to curly brackets).

Of course, and there's all manner of ways to encode information (including 
steganographic encryption of information, such as for example sending a 
pornographic JPEG photo also containing on the nightstand table a clock where 
the hands point at 11:17:32 which time maybe has some secret meaning to the 
intended recipient).  You're NEVER going to be able to prevent the transfer of 
all kinds of information.  No point in even trying.

Fortunately, one doesn't HAVE to get that paranoid to essentially solve the 
problem, since MUAs aren't all that aggressive and suicidal in ferreting out 
dangerous HTML to interpret, either.  :-)

In my current experimental incoming mail filter, I use a recursive subroutine to 
deal with nested encodings and message parts.  I suppose I could even 
(trivially, in fact) limit that routine to only support a limited nesting depth, 
but I suspect that most mail clients would crash on such mails long before my 
mail filter would.  :-)

> Whatever regular expression for an HTML tag you come up with, it can
easily be made unrecognizable. 

Sure, but it can also in the process be made unrecognizable to MUAs, too.  

> ...Even the interpretation of HTML tags
can be redefined on-the-fly if it comes to that. 

Probably, but again:

  1)  the USE of such types of stuff is prima facie evidence of an E-mail having 
something to hide;

  2)  such tricks are of little value if they confuse or break MUAs too;

  3)  translating pointy brackets to curly brackets (or square brackets or 
something else) will also effectively "neuter" such HTML, such that MUAs won't 
try to process it;

  4)  it's relatively easy to (again, by default) simply say "NO HTML, period" 
and divert offending mail.

> But say you keep up

Well, "patch" isn't really necessary.  :-)  It's rather easy to add new stuff to 
SNOBOL/SPITBOL programs, including at run time.

But again, that's why one doesn't just look for FIXED limited number of specific 
things. 

If one simply bans (default case, for unknown senders) *all* attachments and 
*all* HTML, then it's pretty hard to argue that they'll figure out some new kind 
of HTML (but if and when they do, then one MIGHT have to tweak the filter a 
little bit).  If it doesn't look enough like HTML to be recognized by the MUA, 
then it clearly doesn't have to be recognized by the filter, either.

made if they have access to your software (as they invariably will if
it becomes widely deployed), so there is little point in not
discussing specific parsing techniques publicly. 

Fair enough, although it's pretty extraneous to discuss them publicly at this 
time.  As I've said, the current implementation is "experimental" and like all 
such software, a work in progress.... which I modify and improve from time to 
time as that seems necessary.

> ...It only makes discussion imprecise and harder to see any flaws.

The important thing is NOT whether there are "flaws" at the lowest level (and 
undoubtedly there are, since all nontrivial programs contain bugs or at least 
opportunities for improvement).  At this point we ought to be talking concepts 
and approaches, rather than getting bogged down in pointless minutae and detail 
which in any case is going to be implementation-dependent.

> You argue that perhaps the most important overall function... 

I don't know that I would characterize it that way, although I *do* believe that 
it's sorta silly to try to address the spam problem while ignoring the kinds of 
ridiculously suspect worm/virus stuff that clueless users naively click on.

> ...is to block the spread of viruses, worms and zombies, as these are the 
current enabling technology. 

Again, let's not say "the" (which implies one).

> If so, you should address that problem directly, as it has much wider scope 
than the "attachment" problem.  

I think it makes sense to attack the attachment problem DIRECTLY, and HEAD ON, 
since it is important NOT ONLY JUST to worms/viruses, but ALSO for spam evasion 
of content filters (e.g. text-as-image or even just content-as-image).

> Blocking attachments, if widesread, will only achieve that the payload
is moved from the email body to an external server. 

That's fine.

> Users are then tricked to open an external connection...

OK, but at least it's not going to be something that they just click on (again, 
by denying HTML, and hopefully by implementing suitably dire-sounding warnings 
when they try to follow any other links to external executables, whether EXE 
files or SCR files or DLL files or ActiveX or whatever).

Hey, we DO agree that "social engineering" can result in people doing pretty 
stupid things (like giving their secret passwords because someone calls on the 
phone and asks for them, etc etc) but at least we can offer REASONABLE 
safeguards and "are you SURE?" type things to at least make them have a second 
thought before proceeding with such things.

A person who is DETERMINED to sink their own ship, of course, CAN still do that, 
and at some point one simply has to cut the rope and let them go.

> ...which downloads the malware in any of a wide variety of ways, and still 
sends spam from then on. 

Right.  It's important to at least make that less "encouraged".  That's one good 
reason for also (by default, from unknown senders) getting rid of HTML, which 
tends to encourage (and conceal/misrepresent) external links.

> Meanwhile, in the process you destroy the user's reasonable

If you don't know me (and even if you DO) you do NOT have any right to expect 
that I will even receive or choose to read your mail AT ALL, let alone without 
my modifying it beforehand to suit MY tastes.  You lose your right to its 
absolute integrity as soon as you seal it up and send it to someone else.

Perhaps the key to your point, though, is the word "delivered".  And I suppose 
that your point is okay, since it _is_ "delivered" (to my incoming mail 
processing filter!), and that filter (AS I HAVE INSTRUCTED IT) then chooses to 
modify the incoming mail according to rules *I* have established to help make it 
acceptable to me, before I need to look at it.  Perhaps this is another good 
reason to implement the filter at the recipient end, rather than somewhere 
enroute.

Perhaps so, and what may end up happening is that content filters will be 
simplified and re-engineered to make them faster and more tailored to use within 
a framework such as I propose.  (Although some of those "hard cases" might still 
get through, from "somewhat-trusted" senders).  Current content filters usually 
presume that they are getting E-mail "raw" and therefore have to handle cases 
that might be filtered out already by the time mail would get to them through my 
filter.

> To my knowledge, no successful attack has been performed on such
systems yet. 

Depends on what you call an "attack", but certainly an awful lot of spam 
contains bogus (random or unrelated) stuff that's designed to confuse or evade 
such types of filters.

> There is a lot of garbage in mail to try to pass through

Perhaps, and we agree that a good program can detect certain types of such 
stuff. but at SOME point the spam E-mail in question is going to look EXACTLY 
like a regular E-mail that you want to get, except for the spam content (which 
might be JUST a URL or a phone number or who knows what?)   

Fine.  In any case, it is POSSIBLE to create spam E-mail that looks just like 
legitimate E-mail, at least within statistical uncertainties.  There are limits 
at what can be achieved going down that path (but that doesn't necessarily mean 
that it's not worth going there, if there's useful progress there nevertheless).

---

From: Laird Breyer <[email-masked]>
Date: Fri, 10 Dec 2004 13:16:34 +1000

> believe everything that E-mail tells you, either.  :-)

Do you fully parse the MIME structure to extract HTML or not?  Do you
decode + scan each attachment, identify its type, further decode its
contents according to identitied type, and extract things
 such as embedded HTML? Unless you respond yes to these questions, you
are ignoring the actual structure, and instead making assumptions
about certains string patterns. It doesn't matter if you use SNOBOL or
regular expressions, you make decisions without the full context,
which is risky in terms of content alterations.

I accept your point that you are willing to pay that price, but since
there are effective competing solutions available without this flaw (I
consider alteration a flaw), users already have alternative
options. May the best offering win, as they say.

> [snip steganography]
> You're NEVER going to be able to prevent the transfer of 

That's what spam filtering is. A competition between those who want to
pass along information and those who want to censor it. Spammers have
shown that they are willing to think laterally to get their message
across, and don't shy away from building complex edge cases (as well
as simple ones if that works). Dismissing such things now will only
mean more work for you later, imho.

> Sure, but it can also in the process be made unrecognizable to MUAs, too.  

As a filter, you take on the responsibility to censor for an unknown
class of MUAs with varying capabilities, unless you're building a
plugin for a single specific MUA. The latter problem is much easier,
but much more narrowly applicable, and certainly doesn't deserve to be
considered a general or universal solution to the spam problem.

>   1)  the USE of such types of stuff is prima facie evidence of an E-mail having 

If your (or any) filter becomes widely used, email will be crafted to fool
it if it has weaknesses. Being fooled literally means that filter won't recognize
such evidence, even though it might be obvious to other filters. So your
point 1) only applies as long as your filter is unpopular enough to not impact
spam distribution.

>   2)  such tricks are of little value if they confuse or break MUAs too;

Different MUAs break in different ways. There is no set of tricks which are
obviously going to break all MUAs and therefore can be dismissed a priori.

> and divert offending mail.

By claiming to modify content, you're only making your system less
attractive to a section of the user population who: want HTML, don't
want HTML but feel it is crucial to accept HTML in the event it is
sent bu customers, expect messages to be untampered for technical or
ethical reasons. That's a lot of people who have an interest in
preventing your system from getting critical mass.

> > But say you keep up to date with tricks designed to make a complex
> payload look innocuous to simple minded filters, then you are on the
> losing side of such an arms race, because a spammer need only change
> their email, while you need to patch your software with new regular
> expressions and redeploy it to all the customers every time.
> 
> Well, "patch" isn't really necessary.  :-) It's rather easy to add
> new stuff to SNOBOL/SPITBOL programs, including at run time.

Can you do it in an evening and have all your deployed systems fixed
up world wide by the next morning? The spammer needs an evening to
figure out a loophole and send millions of mails exploiting it the
next day.

Some commercial systems already have elaborate world wide networks
designed to propagate new email signatures in a matter of
seconds. That's the technology you're competing against today.

> Fair enough, although it's pretty extraneous to discuss them
> publicly at this time.  As I've said, the current implementation is
> "experimental" and like all such software, a work in
> progress.... which I modify and improve from time to time as that
> seems necessary.

Unless you have an argument that breaks the spam arms race, the
distinction between "experimental" and "release" is blurry, I think.
You can spend months or years polishing a user interface only to 
have your solution be outdated.

> The important thing is NOT whether there are "flaws" at the lowest
> level (and undoubtedly there are, since all nontrivial programs
> contain bugs or at least opportunities for improvement).  At this
> point we ought to be talking concepts and approaches, rather than
> getting bogged down in pointless minutae and detail which in any
> case is going to be implementation-dependent.

I'm not convinced that your high level description will handle the
rough seas of widespread deployment, but instead will get bogged down
in endless silly details.  Just my opinion ;-)

>  OK, but at least it's not going to be something that they just
> click on (again, by denying HTML, and hopefully by implementing
> suitably dire-sounding warnings when they try to follow any other
> links to external executables, whether EXE files or SCR files or DLL
> files or ActiveX or whatever).

Denying HTML is useless if the MUA generates it on the fly whenever it
sees something that looks like an url. People are inundated with
warning dialogs which they just click mindlessly. As is, I don't see
that your filter can change or address these issues (why should it?
those are some of your stated high level means of protection).

>  Perhaps so, and what may end up happening is that content filters
> will be simplified and re-engineered to make them faster and more
> tailored to use within a framework such as I propose.  (Although
> some of those "hard cases" might still get through, from
> "somewhat-trusted" senders).  Current content filters usually
> presume that they are getting E-mail "raw" and therefore have to
> handle cases that might be filtered out already by the time mail
> would get to them through my filter.

Either that, or users will decide that your system is hampering their
content filtering and remove it. Nobody knows...

> Depends on what you call an "attack", but certainly an awful lot of
> spam contains bogus (random or unrelated) stuff that's designed to
> confuse or evade such types of filters.

Most of the random stuff is designed to evade a different kind of filter,
namely those filters which keep a database of email hashes. It sticks
out to statistical filters because it contains novel tokens.

I call an attack a repeatable procedure which can consistently bypass
the spam verdict with sufficiently high success rate to be valuable.
An attack requires a modification of the filtering algorithm.

>  > There is a lot of garbage in mail to try to pass through the
> statistical filtering, but just like you look for nonsense tokens as
> an indicator of spam on a case by case basis, such nonsense tokens
> if present easily tip the balance toward spam in a statistical
> filter, automatically.
> 
> Perhaps, and we agree that a good program can detect certain types
> of such stuff. but at SOME point the spam E-mail in question is
> going to look EXACTLY like a regular E-mail that you want to get,
> except for the spam content (which might be JUST a URL or a phone
> number or who knows what?)

Not quite. The spam content is what looks statistically different
(unless the recipient is in the habit of discussing this kind of spam
content regularly), while the statistical commonalities between spam
and nonspam are automatically discounted if they appear in both types
of mail.  That and the fact that there is no need to do complex
parsing text is why it's difficult to evade. Any string added to a spam
message either

1) looks like some nonspam string, in which case it counts for
approximately nothing, or

2) doesn't look like some nonspam string, in which case it tips the
balance towards spam.

> Fine.  In any case, it is POSSIBLE to create spam E-mail that looks
> just like legitimate E-mail, at least within statistical
> uncertainties.  There are limits at what can be achieved going down
> that path (but that doesn't necessarily mean that it's not worth
> going there, if there's useful progress there nevertheless).

There are always limits, e.g. a single statistical system shared by
many people is vulnerable to contradictory inputs. As far as
advantages are concerned, the statistical systems have the simplest
user interface yet devised, as the user only needs to label the
message without analysis, and update automatically without programmer
intervention.

-- 
Laird Breyer.