Subject: Re: [Asrg] Re: bounces, and anti-spam principles

From: <[email-masked]>
Date: Fri, 26 Jan 2007 01:53:48 -0600

> Before trying to reply to this whole thing, you might 
>want to scroll
> forward to Comment #6.  You seem to be assuming various 
>things about how
> filters have to work that simply aren't the case.

We'll get there.... :-)

> You _can_ handle false positives 100% without the 
>recipient controlling
> the rules or even necessarily know that a false positive 
>has occured. In
> effect, reducing your false positive level to zero, 
>without any
> recipient involvement.
> 
> Our overall system is designed towards _zero_ false 
>positives.  In at
> least one way, we're theoretically (and near 
>effectively) there.  With
> DNSBLs and other techniques.

Obviously, I'd have to reserve judgement until I've 
experienced your "zero false positives" idea.  Let's just 
say that I've got a healthy degree of skepticism.

> In fact, in most ways we're far more aggressive than 
>other filtering
> schemes.  We can afford to have the filters misfire on 
>legitimate email.
> 
> Simply because we've paid as much attention to finding 
>out about
> misfires and remediating them as we've done with 
>filtering itself.

Certainly making the consequences of questioning a given 
E-mail "acceptable" from a user standpoint gets you a long 
way.

> What happens after being blocked is just as much part of 
>a well designed
> filtering strategy as the filters are.

Agreed.

>> [comment #1]
>> 
>>>> In any case, I still contend that simplistic blocking by 
>>>>IP address 
>>>> or domain name is a very poor approach, and for a whole 
>>>>variety of 
>>>> reasons.
>> 
>>> I will contend that there cannot be a content filter 
>>>that can 
>>> reliably separate spam from non spam. 
>> 
>> It doesn't NEED to be 100.000% accurate.
> 
> Nor does any other form of filtering then.

Right.  If the USER considers it okay (and that's to some 
degree a moving target!) then it's "okay".
  
>> The bulk of mail most people receive comes from people 
>>they are familiar
>> with, and which fits certain patterns. A given sender 
>>(mailing list etc)
>> will typically have a signature file, for instance.  I 
>>know that Aunt
>> Matilda is NOT going to send me an E-mail containing a 
>>JavaScript
>> decryption routine, or an ActiveX enclosure.  She also 
>>is not going to
>> send me an executable attachment.  If stuff like that 
>>arrives here, it
>> is safe to presume it is NOT from her, no matter what 
>>the From: address
>> says (and even if it WAS sent from her computer).
> 
> This does not work in the large scale beyond a limited 
>subset of users.

Why not?

> Not everyone has that small a set of correspondents to 
>cope with, and
> the "new correspondent" issue remains a big problem.

The total set of correspondents doesn't have to be small. 
 It just needs to have a relatively small number of NEW 
correspondents that NEED to use "advanced"/riskier 
features and therefore require whitelisting.  Ideally, 
each such correspondent only requires ONE click (one time) 
for the user to agree to allowing them to use the more 
advanced features.  That might be done following an 
initial negotiation E-mail where the sender introduces 
themself and requests the ability to send more elaborate 
mails.

Introductory E-mails should NOT automatically presume the 
desire or willingness of the recipient to receive 
HTML-burdened E-mails.

>> I guess that depends on what you call "bulk", and how 
>>you propose to
>> detect it.  Again, whatever rule you put into effect (on 
>>a global-type
>> basis) is going to be discovered by spammers and they 
>>will engineer
>> their sending patterns to avoid violating it.  That's 
>>why you need a
>> really narrow and twisty 'gauntlet' they must negotiate, 
>>with DIFFERENT
>> RULES for different recipients, where they don't know 
>>and basically can
>> not figure out what rules they would have to comply with 
>>to get a
>> message through to a particular person.
> 
> Having had almost 4 years of intimate operational 
>experience with what
> is probably the most effective single anti-spam 
>filtering method there's
> been (one specific DNSBL), I can assure you that it is 
>both more
> reliable in terms of FPs than any content filter I've 
>ever dealt with
> (we've always run hybrid content+source+other 
>techniques), I can also
> assure you that its effectiveness is _not_ declining - 
>in fact it's
> getting better.

The main problem with content filtering is caused by ruses 
based on HTML and attachments.  These techniques serve to 
obscure the content of the E-mail, and ALL BY THEMSELVES 
the presence of such content in E-mails (at least in 
E-mails from unfamiliar senders) can be a priori evidence 
of hostile intent, or spamming.
  
> Is it perfect?  No.  Does adding other filtering 
>techniques help?  Yes.
> But to claim that it's useless/trivially defeatable/too 
>many FPs/trivial
> to do better turns out to simply not be the case.

Certainly there are always special cases/special 
situations where just about anything can be made to look 
good.  That's like SPF, which is claimed to work well (but 
ONLY for the subset of the situations that it works for).

>>> The trick  is to stop accepting mail from that IP 
>>>address only until
>>> it has  cleaned up. 
> 
>> Again, when you have a LOT of users (and possibly MANY 
>>servers) behind a
>> NAT router, denying mail from that IP address results in 
>>simply too much
>> collateral damage. More to the point, it's a very blunt 
>>instrument for
>> the job, and it's relatively simple to do very much 
>>better.
> 
> So far, there's no indication of the latter being true. 

I do note you don't refute my objection on principle.  :-)

> Show me a
> methodology that (a) doesn't abuse innocent third 
>parties and/or (b)
> requires personal twiddling simply not achievable in the 
>large scale
> that's better than the CBL (or for that matter Zen), I 
>want to hear
> about it.

I don't consider that ANYBODY has the right to blindly 
haul off and presume that I'm willing to accept 
HTML-burdened E-mails from them.  So if YOU believe that 
such a restriction is "abusing innocent third parties" 
then we already disagree.

I also don't agree that my approach requires an 
unacceptable level of "personal twiddling" although it 
certainly DOES involve some degree of some recipient 
involvement, IF its capabilities are to be optimized.  If 
the software is well-designed, I don't believe that it 
will put off most users, especially if they have fought 
spam for any length of time before.

>>> Once the spam is gone there is no need to block the 
>>> address unless it
>>> has proven to be a repeat offender without an  effective 
>>>process for
>>> shutting the spammers down.
>> 
>> What about when the flow of spam is interleaved with all 
>>sorts of
>> good/important traffic as well?
> 
> You make a cost-benefit analysis and/or apply other 
>techniques.  Even
> content.
> 
> Why the insistance on choosing only one?  You don't have 
>to.

Agreed.  I never proposed my fine-grained whitelist all by 
itself.  It works best in conjunction with a good content 
filter (and in fact greatly improves the efficacy of the 
content filter).

> Effective spam filtering is best done with a hybrid 
>solution.  No one
> technique is complete on its own.

Agreed.

>> There is a lot of spam which is obvious.  That includes 
>>messages which
>> contain links to known-spam-promoted Web sites (at least 
>>in the absence
>> of contradicting factors, say being from a list 
>>discussing spam senders!)
> 
> SURBLs are blacklists too, and can be equally as blunt 
>as a source IP.
> Or are you contending that the user should be explicitly 
>entering them?

I don't really claim either way, but I WILL say that there 
are a lot fewer valid links to spam-promoted Web sites 
than there are randomly-generated counterfeit "from" 
domains and E-mail addresses.

> With _individual_ spammers using 1000 or more domain 
>names to advertise
> the same thing, how effective do you think that can be?

Again, it's only one of a variety of criteria that can 
identify spam.  It's clear to me, though, that just using 
From: and Subject: lines are by themselves NOWHERE NEAR 
adequate, particularly if you want to receive E-mails from 
previously unfamiliar senders.

>> It also includes, for example, messages which are 
>>identical to messages
>> that some number (dozens? hundreds?) of other recipients 
>>at the same ISP
>> have already reported as being "spam".
> 
> Have you done any work with checksumming/hashing spam, 
>ala DCC or Razor?

Not in that context, although I've used them in other 
contexts...

> Yes, they're useful.  But these days most high volume 
>spammers randomize
> content such that even highly developed de-hashbusting 
>techniques don't
> work very reliably.  Doesn't even work on graphical spam 
>anymore.

That's a good reason to simply make graphical spam no 
longer viable, by defaulting to not allowing images 
(whether attached or embedded) from unfamiliar senders.
  
>> One would think that ISPs could
>> locate and perhaps recategorize identical messages 
>>(again, perhaps
>> tempered by a specific recipient rule) which are still 
>>queued and have
>> not yet been delivered to their remaining customers.
> 
> Yes, it will yield some results, but the overall results 
>are highly
> disappointing.

A lot depends on how big the ISP in question is.  Within a 
company of 15 users, that won't help much.  For a big ISP 
like Google or Yahoo or Hotmail, flushing all remaining 
similar/identical unread E-mails which have been 
discovered to be spam (by user reporting, as Yahoo does) 
can quickly purge them from the Inboxes of other users who 
haven't yet downloaded or read them.

>> But let me state again (and this is part of what made me 
>>respond,
>> starting this sub-thread) is that it is virtually NEVER 
>>a good idea to
>> send a bounce message after-SMTP-time, because you can't 
>>be sure where
>> to send it, and most likely you are just harassing 
>>another innocent
>> victim.
> 
> That's something I'll certainly echo.

Thanks... it is VERY frustrating to me to have Yahoo start 
blocking all my outgoing Groups mail every month or so 
just because SOME virus-infected mail with a bogus return 
address gets bounced back to them as a "hard bounce". 
 :-((

Hard bounces should NEVER be sent based on content, 
especially when you can NOT be sure who to send the hard 
bounce to.  And they should NEVER be sent back based on 
detection of a virus/worm which is KNOWN to counterfeit 
return addresses!!!

>>> Being able to "slam the phone down" on miscreant IP 
>>>blocks at the
>>> accept() or helo is much, much, less processing than 
>>>going thru the
>>> entire SMTP interaction and whatever it takes to pass 
>>>processing off
>>> to an end-user.
>> 
>> It's true that it costs less, but it's also true that it 
>>blocks a lot of
>> innocent and legitimate mail that might be
>> originating from the same IP address (NAT router?).
> 
> This generally doesn't turn out to be a significant 
>issue in practise.

Tell that to the big company that will (eventually) find 
ALL their outgoing E-mails blocked (possibly for weeks) 
because of one or more infected systems behind their NAT 
router.  This kind of nonsense could literally put some 
kinds of companies out of business.

> If you spend the time you _should_ be spending to 
>research the DNSBLs
> (say) you plan on using.

I would not use that, because I consider IP-based blocking 
to be ultimately a fatally flawed approach.

I don't think it much matters HOW it is done.

>> There could be
>> dozens, hundreds, or even thousands of innocent users 
>>affected.
> 
> There could be, if the DNSBL is built in such a way that 
>it's
> susceptable to that.  But it doesn't have to be.  And 
>the good ones
> aren't in any meaningful way.

How can it NOT be, if a given IP address is sending 50% 
spam and 50% legitimate traffic, all interleaved?

Sorry, I don't agree.

Even 2% legitimate traffic is too much to block, if that 
2% is critically important stuff.

[incentives to clean up infected machines]

>> ...the problem is that after 
>>(first!) SMTP
>> time, the (intermediary, or final) recipient doesn't 
>>really know who
>> they ought to notify...! Notifying the wrong person, or 
>>someone who has
>> no control over the situation, probably does more harm 
>>than good.
> 
> You don't need to notify out-of-band.  Presuming that 
>recipient systems
> use DNSBLs in an appropriate fashion (inline rejects 
>with pointers to
> more information), legitimate senders find out that it's 
>blocked and
> why.  That's their notification.  

Again, I don't consider it acceptable to block good mail 
because someone ELSE using the same server or gateway is 
misbehaving.

>With decent DNSBLs, 
>that's sufficient
> to initiate resolution of the problem.  It sure is a lot 
>easier than
> tweaking Bayes or SpamAssassin (we specifically reject 
>whitelisting
> email addresses because of the forgery problem)

I consider a blanket "whitelist" as nowhere near adequate, 
because too many addresses are opened up.  That's why 
whitelisting must be done on a far more finegrained basis.

> Also, there's a remark somewhere on the CBL web site 
>that impressed me
> with its simplicity - something to the effect that "we 
>fully expect the
> vast majority of infected/blocked users _never_ notice 
>that they're
> listed, because they're using their provider's 
>smart-hosts as they
> should".  If an ISP wants to proactively scan for 
>infected IPs to see
> about getting them fixed, they can do that too.

You presume that spammers can't send out their spam using 
the legitimate user's "certifications" and "smart hosts" 
(and their "micropayment account" if pay-per-email were 
implemented).  I don't believe that's an accurate 
assessment.  (Even if they didn't today, they can change 
their strategy overnight to do so.)

>> [comment #4]
> 
> I'm skipping this one because it takes too long to 
>comment on ;-) other
> than:
> 
>>>  Executive summary...
>>>
>>>  - blocking email, because it meets some technical 
>>>criteria, is easier
>>>    on the technical side, but introduces legal problems
> 
> It may, perhaps depending on exactly what the technical 
>criteria _is_
> and the rationale for blocking it, but taking risks when 
>the business
> case/result justifies it, is what people do.

If you NEED to do that, sure.  But I consider it a poor 
bet if there are better approaches which don't leave you 
exposed like that.

> You can forestall a lot, for example, by simply saying 
>"it's our policy
> to reject email from IPs that appear to be dynamic". 
> That says nothing
> about the spammyness of an individual sender, and if 
>mistaken in
> "appearance" simply needs to be fixed.  Or not.

Obviously you can set your policy however you like, but 
when it gets too cavalier, expect angry customers to start 
bailing.

Again, there is good NO reason to do things that way. 
 (IMHO at least).

>>>  - blocking email, because the customer said so, may be 
>>>harder
>>>    technically, but avoids legal problems
> 
> The ISP could equally establish a infrastructure where 
>the customer
> explicitly delegates filtering decisions to the ISP.

Sure, but don't be surprised when customers decide they 
don't like the ISP's decisions, and it wouldn't take much 
for an unhappy sender to team up with an unhappy recipient 
customer and sue the ISP.

> And the protections in law for ISPs to be held harmless 
>for mistakes in
> good-faith filtering go a long way to shoot down any 
>attempt even where
> the customers haven't formally delegated.  It ain't easy 
>for a sender to
> prove bad-faith.  An ISP hasn't been sued in ages.

I wouldn't want to be the first one to break that record.

[snip]

>> ....knowing that all the classical ruses 
>>to avoid spam
>> classification (text as image, embedded links, 
>>attachments, scripting,
>> disguised HTML links, etc etc) are a priori denied 
>>them.... certainly
>> takes a major bite out of spammers.
> 
> It would if you could.  How can you tell that an image 
>is text?

It doesn't matter.  You simply block (by default) ALL 
images (whether embedded or attached) coming from 
unfamiliar/first time senders.  If they want to send 
images, they first negotiate that permission with the 
intended recipient.  (And that permission, of course, can 
be revoked by the recipient if the privilege is abused).

> Blocking spam with embedded links or attachments would 
>probably put us
> out of business.  More likely get me fired.

It will certainly put MANY spammers out of business, and 
put virtually ALL virus/worm authors out of business... at 
least as far as E-mail delivery goes.

I don't have a problem with a recipient ALLOWING a 
particular sender to send them that kind of stuff... as 
long as an introductory E-mail (in plain text) negotiates 
that permission with the recipient FIRST.

And, a given recipient might configure his setup so that 
he could look at E-mails that would have been blocked, to 
decide to allow them in the future... or he can just take 
a hard line and say "only if approved first!"  But in any 
case they could be marked as suspect/blockable to warn the 
recipient, before they open them.

There is NEVER the NEED to send HTML/attachments/scripting 
to someone in an INTRODUCTORY E-mail.  Not until you've 
established that they are willing and able to receive that 
kind of content from you.

>> And only allowing executable attachments, HTML, and 
>>"big" messages from
>> known/trusted senders basically eliminates E-mail as a 
>>vector for
>> virus/worm propagation,
  
> "known/trusted" senders by what measure?  Explicit 
>listing of them?

Sure.  Most users will not EVER need to approve ANYBODY to 
send them executable attachments.  ("listing" can be 
achieved by clicking an "allow this sender to send me 
content like this in the future" button.)

> Well, I can tell you about lots of viral attachments 
>from "known senders".

Absolutely, but MOST users don't need or want to accept 
EXECUTABLE attachments from ANYBODY (including "known" 
senders).  That's part of why it is a FINE-GRAINED 
whitelist and not a crude yes-or-no whitelist (which I 
agree completely is NOT adequate).  I might accept JPG 
attachments from Aunt Gertrude, but it's a safe bet that 
any EXECUTABLE attachment in an E-mail (claiming to be...) 
coming from her... well, I wouldn't touch THAT with a 
ten-foot pole.

I really don't understand why that's such a difficult 
concept for people to grasp.  That (VERY!) simple rule can 
basically eliminate viruses and worms (again, at least by 
E-mail) virtually overnight, if adopted net-wide.

Given that, it seems to me to be professionally 
IRRESPONSIBLE to NOT implement it.  That is the biggest, 
fastest thing that the IETF could do IMHO to cripple the 
recruitment of spambot zombie armies.

>> [comment #5]
> 
>> That's certainly true, and one advantage of fine-grained 
>>recipient
>> blocking is that it doesn't require any great worldwide 
>>consensus, nor
>> any re-engineering of Internet infrastructure.
> 
> Nor do DNSBLs ;-)

Again, I consider (IP-based, at least) blacklists to be 
unacceptably crude.  You're locking the door after the 
horse has left the barn.  A whitelist is better because a 
first-time spammer is still left facing a locked door.

>> What WOULD be helpful, though, would be a recognition by 
>>the IETF that:
>> 
>>  a) such fine-grained per-sender by-recipient blocking 
>>(and hopefully
>> augmented by subsequent content scanning) is an 
>>effective and desirable
>> approach to the problem, and
> 
> As I've been saying, that has yet to be established.

I believe in any case that it is (by FAR) the most 
promising approach.  NOT by itself, but in conjunction 
with content analysis and possibly other techniques.  But 
it is the primary key to make the other techniques more 
successful.
  
>>  b) in the general case, blocking of all non-whitelisted 
>>E-mails
>> containing HTML, scripting (probably covered under 
>>HTML... is it
>> possible to put in scripting without HTML?), or 
>>attachments is a "best
>> practice".  (It is probably a good idea to suggest 
>>including a maximum
>> message size, too, as a way of preventing "denial of 
>>service" attacks by
>> sending big E-mails to someone which would be expected 
>>to fill their
>> E-mail inbox to overflowing, blocking subsequent 
>>legitimate E-mails).
> 
> Obviously, you've not had to deal with the legitimate 
>mail traffic of a
> large corporation.  I mention those measures as comic 
>relief at
> meetings, because it always produces hysterical 
>laughter.  It'd shut us
> down.

Again, I don't consider anybody as having a legitimate 
need to send HTML or attachments in first-contact E-mails. 
 If they want to send that stuff, then they should 
negotiate that permission with the intended recipients 
first.  And they should be prepared for some recipients to 
say "No, thanks, I'd rather receive plain text."

If it's a question of offending those who "laugh 
hysterically" or shutting down spam, I'll opt for shutting 
down spam.

And I believe that others would agree.

So if you think that such a rule would shut your company 
down (given that a user COULD grant you the permission to 
send them such mails, after a preliminary meeting and 
negotiating that) then I propose that your company's 
long-term future is not bright...!  I suggest you get them 
thinking about how your company could adapt to survive in 
such a more-responsible world.

[snip]

>> Again, the problem is the degree of collateral damage 
>>that IP-based
>> blocking produces.
> 
> You haven't demonstrated what that degree _is_.  By long 
>exposure, I can
> assure you that the degree is surprisingly low.  If you 
>do your homework
> as you're supposed to.

If you block the mail from a big company's server(s) 
because one user within that company has an infected 
computer that's been recruited as a spambot, I don't think 
it's a great consolation that "gee, we hadn't seen that 
happen before."   Especially if the block thus established 
causes serious financial harm to said company.

> We're receiving 1-2 million emails per day.  80-90% of 
>that is spam.  We
> have less than 10 FPs per month against Spamhaus' 
>sbl-xbl (which is
> doing about 85% of our filtering).  We've arranged 
>things so that the
> sender finds out if they're blocked, and there's a well 
>established
> procedure by which they can notify us and we can 
>override listings.
> 
> If an email is blocked, they contact us, and we forward 
>their email and
> fix the listing, is it really a FP?  No.
> 
> If every sender correctly interpreted the error message 
>they got and
> followed through, then there'd be zero FPs.

Some VERY large ISPs (such as Yahoo) interpret a "hard 
bounce" message due to an infected E-mail as "this E-mail 
address is bouncing" and they disable sending all mail to 
the user involved.  Stupid?  Yes, of course.  But YOU try 
to convince them to change that policy.  They are totally 
non-responsive.

Worse, they only provide the now-disabled user with the 
SINGLE LINE saying the 5xx bounce reason message... saying 
what virus was found... thus NO clue as to WHO bounced the 
mail back, or where else it might have come from.  So, of 
course, this is totally useless.

> I wish our content filters even remotely approached 
>being _that_ good.

If you didn't have to contend with HTML (and the various 
ruses that allows to obscure content), and attachments, 
they would be FAR more effective.  Not allowing HTML from 
unapproved senders would also negate nearly all pfhishing 
exploits, since you couldn't have misrepresented links to 
click on, or buttons that take you to (hidden) 
disreputable Web sites using (invisibly-)obscured URLs.

>> [comment #6]
>> 
>> [how users configure their whitelist rules]
>> 
>>> The problem being that out of the 60,000 seats here, 
>>>perhaps less than
>>> 10 of them are able to competently configure a set of 
>>>rules like what
>>> you have.  
>> 
>> That's a software implementation issue, not an inherent 
>>problem in the
>> approach.  I envision a button to click on that simply 
>>says "allow
>> E-mails like this from the same sender in the future" 
>>and where the
>> software will open the keyway JUST enough to allow that 
>>type of message
>> if seen again from that sender.  How that recognition is 
>>accomplished,
>> whether by something crude like simple GREP-type 
>>scanning, or something
>> brain-damaged like RegEx pattern matching, or something 
>>still more
>> sophisticated like the pattern matching SNOBOL/SPITBOL 
>>offers, or even a
>> different sort of statistical ranking/rating approach 
>>like content
>> scanners use... will vary from one implementation to 
>>another.  The final
>> products will probably use a combination of techniques.
> 
> I'm sorry, this simply isn't a human interface issue. 
> No amount or
> technique of per-sender whitelisting comes remotely 
>close to the
> accuracy of our production filters, entirely aside from 
>the new
> correspondent issue.

It's not a simple "per sender" whitelist.  It is a 
fine-grained per-sender PERMISSIONS list, which enables 
some types of HTML tags and SOME types of attachments and 
SOME other characteristics, on a sender-by-sender basis.

I don't believe it is possible to come up with as high an 
accuracy on ANY "one-size-fits-all" rule, "production" or 
otherwise.  There are some mails which I would accept from 
familiar senders, where the SAME mail (same content, same 
subject line) would be spam if it came from anybody else. 
 You simply cannot possibly know who I would accept that 
mail from, and who I wouldn't, unless I tell you.

> You could give our users common filtering software (the 
>reader is
> already pretty much standardized) with every filtering 
>knob known to
> man, and perhaps three of our users could approach the 
>effectiveness of
> the production systems.  

I'm not suggesting that users have to "roll their own" 
from ground zero.  Agreed that 90% of users will have 90% 
of their senders operating on the defaults.  And your 
"production systems" can of course be tweaked and upgraded 
as spammers adopt new techniques.

That doesn't in any way demonstrate the lack of value in 
what I am proposing.

> I'm only including _me_ in that 
>list because
> it's me who built the production systems...  Over a 
>million decision
> items are being changed every day in our filters.
> 
> Our users simply don't know effective spam filtering 
>techniques.  They
> push the wrong button, twist the wrong knob, and they're 
>blocking
> something business critical.  Or trusting malicious 
>content with forged
> credentials.  Or simply trusting...

Agreed that it ought to default to "simple".  But the 
defaults can still be HIGHLY effective.

There is VERY little excuse for ANYBODY needing 
traditional antivirus scanners for incoming E-mail.  That 
kind of garbage ought to be, and can easily enough be, 
squashed outright by responsible ISPs.

> My favourite incident was the user who repeatedly 
>insisted that he
> needed to receive the "important information from the 
>FBI" that was
> sitting in his quarantine, and the quarantine forwarder 
>refused to
> forward to his mailbox.
> 
> Sorry, I said, but as much as you may want to see it, 
>forwarding the
> virus is a really bad idea ;-)

Obviously most phishing exploits work by "social 
engineering" to convince a user to let them in, or to 
provide them information they shouldn't provide.  And some 
users will always fall for the "Aggie Virus" approaches 
(like users who forward hoaxes).

I think we can agree, however, that by default delivering 
E-mails to clueless recipients that contains executable 
attachments is a Really Bad Idea too.  The trick is to 
decide how you can implement that restriction, while still 
allowing them to be sent in the 0.01% of the cases where 
the sender expects and trusts and WANTS them from a given 
sender.  I believe that my fine-grained whitelist answers 
that need, and does it (VERY) well.

>> Again, I consider IP-based blocking to be inherently 
>>flawed, to the
>> point where I consider it a dead-end.
> 
> It's a remarkably vigorous dead end ;-)

If the only tool you have is a hammer, you'll try to make 
every problem look like a nail.

Popular does NOT mean that it's good.

[snip]

>> Yeah, I admit that I usually at least cast a cursory 
>>eyeballing of the
>> Yahoo mail "spam" folder too, rather than just emptying 
>>it. 
>> Occasionally I -do- find a non-spam message there. 
>> (Although that
>> happens seldom, as I almost never give that E-mail 
>>address to anybody...
>> It's almost useful as a "personal honeypot" to see 
>>what's being spammed
>> out, before going to my more usual E-mail accounts and 
>>possibly
>> wondering if that curious E-mail just MIGHT be 
>>legitimate).
> 
> "Almost useful" is the key.  When you have users whose 
>spam load ranges
> from one or 2 per month, to 4000+ per day, you can see 
>that junk folders
> have only limited usefulness, and not to everyone. 

Agreed.  That's why users need to be able to choose what 
level of acceptance of the automated filters is most 
acceptable to them, and what level of time they are 
willing to devote to "questionable" mail.

> Nobody can find
> legitimate email in a 4000 spams/day feed, now matter 
>how the filters are implemented.

True enough, but if a person is willing to simply agree 
that they don't want to see HTML or attachments from 
unfamiliar senders, that instantly blocks a LOT of the 
garbage.  Simple, and HIGHLY effective.

AND it makes the remaining content-based filtering FAR 
more effective, by blocking stuff like 'text as image' 
(whether attached or embedded as a link... or even 
generated by a decrypting script).

And again, they can relieve that restriction on a 
sender-by-sender basis, if they need and want to, based on 
previous negotiation with such senders.

> Perhaps part of your problem is that you're not seeing 
>the big picture
> of how you can _use_ DNSBLs or any other filtering 
>technique.

I've been involved with BBSes and dealing with spam since 
the pre-Internet days, but please feel free to educate me 
if you think there's something worthwhile you can teach 
me.   :-)   I'm not too old a dog to learn new tricks.

> Your remarks seem to imply that DNSBLs necessitate no 
>notification
> anywhere, the email just disappears.  Or that filtering 
>in general is
> that way.

There are clearly all kinds of ways to deal with 
"questionable" mail.  That's something that is going to 
depend on how the software is implemented, and what 
options the recipient selects.  There is a lot of room in 
this area for programmer creativity.

> On the contrary, ours have never done that.  Indeed, 
>without anybody
> looking at quarantines, without anybody personally 
>twiddling filters,
> that 4000 spams/day user _does_ get the email that was 
>accidentally
> blocked.   Simply because we do inline rejects with 
>instructions on what
> to do, and problems get fixed fast and without harm.

Some antispam filtering obviously works better than 
others.

IMHO, if you don't have a finegrained permissions list on 
a per-sender basis, yours COULD work BETTER than it does 
today.

[snip]

>> The important thing is that the RECIPIENT
>> controls that, so they can decide the rule that 
>>determines what gets
>> blocked and what gets through.  That way they don't have 
>>to wonder what
>> SHOULD have been delivered to them and wasn't.
> 
> But what if you arrange things so that the recipient 
>doesn't have to
> control the rules, and still doesn't have to wonder?

I think that a per-sender rule can ALWAYS make things 
better and more accurate.  That doesn't negate the value 
of good pre-sets and defaults.

But again, if by default you let executable attachments 
from unfamiliar senders get delivered (or if you block 
them for ALL senders) then your software COULD do better.

> Our false positives are handled usually without the 
>recipient even
> knowing that something got blocked.

OK, so how do you handle them, specifically?

>> Spammers have gotten good at throwing enough random junk 
>>into E-mails to
>> confuse Bayesian filters.
> 
> And sender whitelisting and...

Again, don't confuse a simple sender-whitelist with a 
fine-grained per-sender PERMISSIONS list.  The difference 
is absolutely night and day.

Even if I allow Aunt Matilda to send me JPGS of her 
poodle, I don't care how many worms her infected computer 
sends to me with HER from: address and via HER mail 
server.  My approach will let me see ALL of her real mail, 
and rapidly and accurately discard all of the worms with 
executable attachments that her system sends me.

It is EASY to do discrimination like that.  And if I don't 
have ANYBODY set to allow them to send me executable 
attachments, or HTML to use links or scripting for that 
purpose, then it's hard to see how such a rule can be 
"confused" or bypassed by a spammer.  It's like trying to 
argue with a vending machine... you don't know until 
you've put your money in that it's not going to deliver 
your purchase, and it's absolutely pointless to try to 
argue with it after the fact.  :-)

[snip]

>> ....giving the recipient the ability to at 
>>least not see the
>> SAME kind of stuff over and over again, if they choose 
>>to use those
>> features, demonstrates the ISP's trying to give the user 
>>the tools to
>> reduce the frustration.
> 
> If you can _detect_ the "SAME kind of stuff" over and 
>over again.  

Sometimes that's very simple.  For example, even the very 
limited filtering the Webmail I'm presently using offers 
allows me the ability to tell it that I don't want to see 
anything more from "Fifth Third Bank" (in either the From 
or Subject fields), where I am not a customer.  Anything 
from them is clearly spam.

It is frustrating that I can't use a SNOBOL/SPITBOL-like 
pattern to describe spam rules, but I readily admit that 
most users would rather not do anything that advanced. 
 (That would CLEARLY however be my 'programming language 
of choice' to write such a filtering tool, though).  Even 
something relatively brain-damaged like RegEx pattern 
matching would be an improvement, although still 
frustrating.

> Even the best content techniques aren't very good at that 
>anymore.

Not automatically, no.  But if there is something the user 
considers adequate evidence of spam (say, a subject which 
contains both the words Penis and Enlarge) then it's 
satisfying to be able to at least block THAT case once and 
for all.  Lots of cases?  Sure, but every new rule at 
least saves you from seeing THAT one again.

Longer-term, more sophisticated rules would work better. 
 But that's why one needs per-sender rules which include 
not just From and Subject but also content within the 
E-mail message body too.

Or for another example, somehow an E-mail address of 
"errorstogep...@domain" got on a spammer E-mail targets 
list.  Any E-mail which contains that anywhere in the 
header is spam, even if it also contains my "good" address 
elsewhere.

> Correctly chosen and utilized DNSBLs do a vastly better 
>job.

They're certainly better than nothing.  But again, I don't 
believe that it's possible to do GOOD blocking, and 
minimize collateral damage, without per-sender 
fine-grained whitelisting which also takes into account 
the body content of the individual E-mail.

And as a recipient, it's very frustrating to see dozens or 
hundreds of similar E-mails continue to slip through an 
ISP's filters, especially when I can readily define a rule 
which IMHO adequately identifies them.  As a user, I want 
that ability, and to not continue to subsequently see 
those E-mails.

Gordon Peterson
http://personal.terabites.com
1977-2007  Thirty year anniversary of local area 
networking

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Bart Schaefer <[email-masked]>
Date: Fri, 26 Jan 2007 08:44:48 -0800

On Jan 26,  1:53am, <[email-masked]> wrote:
} Subject: Re: [Asrg] Re: bounces, and anti-spam principles
}
} I don't consider that ANYBODY has the right to blindly 
} haul off and presume that I'm willing to accept 
} HTML-burdened E-mails from them.

You're welcome to take that position, but it doesn't seem to fit with
the rest of your argument that what gets accepted should be controlled
on a per-user basis.  Elsewhere you wrote:

} Even 2% legitimate traffic is too much to block, if that 
} 2% is critically important stuff.

How is blocking legitmate email because it contains HTML less wrong
than blocking it based on point of origin?

} It doesn't matter.  You simply block (by default) ALL 
} images (whether embedded or attached) coming from 
} unfamiliar/first time senders.  If they want to send 
} images, they first negotiate that permission with the 
} intended recipient.  (And that permission, of course, can 
} be revoked by the recipient if the privilege is abused).

[...]

} There is NEVER the NEED to send HTML/attachments/scripting 
} to someone in an INTRODUCTORY E-mail.  Not until you've 
} established that they are willing and able to receive that 
} kind of content from you.

This presumes a whole lot of effort on the part of the recipient in a
variety of circumstances.

(1) I'm shopping at a major retailer's website and decide to sign up
for their newsletter.  What happens if the confirmation message uses
HTML content (not uncommon)?  But let's presume everyone has bought
into your scheme, so a plain text confirmation arrives.  I now have
to take the additional step of confirming (where? has my email client
but updated to include an interface for this?) that I'll accept HTML
from the same source in the future.  Assuming that the confirmation
and the newsletter come from the same source, that is, which also is
not always the case.

(2) I'm working with an HTML design firm.  They're doing a rush job
for me, which has to be confirmed on Saturday so it can be turned over
to my IT staff to be installed on the company website before Monday
morning; but I'm going to be out of town, so I want a copy of the work
sent to another address such as a freemail service so I can easily
review it.  The HTML contains images and scripts, but the design firm
has never sent mail to that freemail service before, and I don't know
which of their team of designers might be sending the final product.
How much effort must I go to up front to be able to tell them they can
send me an attachment that I know will reach me?

(3) My biggest customer is a giant corporation which requires its
employees to use Microsoft Outlook and to attach a vCard-like signature
to every email, using a background stationery and having an image of
the company's latest logo because they're engaged in a major rebranding
initiative.  Not surprisingly, they have massive turnover, so the
people there who send email to a variety of people at my company
changes every several weeks.  Who's responsible for upkeep of all of
those individual agreements to exhange HTML with each other?

---

From: "Chris Lewis" <[email-masked]>
Date: Sat, 27 Jan 2007 12:24:54 -0500

[email-masked] wrote:

>> Our overall system is designed towards _zero_ false positives.  In at
>> least one way, we're theoretically (and near effectively) there.  With
> Obviously, I'd have to reserve judgement until I've experienced your
> "zero false positives" idea.  Let's just say that I've got a healthy
> degree of skepticism.

I have 10+ years worth of exposure to this model (I built the thing)
with a user population of 60-120K and extreme FP aversion all the way up
to the CEO, you're welcome to come visit and I'll demonstrate it for you.

It only fails to be "zero false positive" when the legitimate sender
fails to read/follow the rejection notice.  Which happens a lot more
than I'd like, but if the message isn't worth that much to the sender,
it's probably not worth doing much about it on our end either.

>> Not everyone has that small a set of correspondents to cope with, and
> The total set of correspondents doesn't have to be small. It just needs
> to have a relatively small number of NEW correspondents that NEED to use
> "advanced"/riskier features and therefore require whitelisting.

The question is how do you even know of the new correspondents, when
you're being bombarded with dozens or hundreds of new correspondents per
day.

It's all well and good, if the spammers aren't forging names, to allow
them one bite before you null the sender, but today's reality isn't like
that.  The "one bite" on sender, or even on url or content, essentially
means you have to eat almost all spam.  Because spammers mutate their
content and senders that much.  We're tracking spammers who use in
excess of 1000 different domains in urls.  Can per-user techniques cope
via whitelisting or blacklisting?  Not a chance.

The only way your techniques can work effectively is if you assume that
the recipient is getting very little spam.  Well, 50% of our users are
being sent almost no spam.  But that doesn't help with the guy getting
sent 4000/day.  Or even the hundreds getting 50 or more.

The only way your technique works is if you're one user with a limited
variety and volume of spam.  But not everybody is in the same situation
as you.

> Ideally, each such correspondent only requires ONE click (one time) for
> the user to agree to allowing them to use the more advanced features. 
> That might be done following an initial negotiation E-mail where the
> sender introduces themself and requests the ability to send more
> elaborate mails.

You seem to be focussing on only blocking "advanced features".  A large
percentage of spammers don't use them anyway.  If all your spam consists
of mutating text-only spam with mutating headers, your technique has no
effect.

> Introductory E-mails should NOT automatically presume the desire or
> willingness of the recipient to receive HTML-burdened E-mails.

But if all spam were indistinguishable from "introductory e-mails",
where are you then?

And indeed, probably most spam isn't distinguishable from some vague
notion of "introductory e-mails" by your technique.

> The main problem with content filtering is caused by ruses based on HTML
> and attachments.  These techniques serve to obscure the content of the
> E-mail, and ALL BY THEMSELVES the presence of such content in E-mails
> (at least in E-mails from unfamiliar senders) can be a priori evidence

If you can _detect_ those ruses.  You and I may be able to detect it in
on a per-message basis with visual inspection, but computers generally
can't, say, identify sentences generated using random words.  Try
building a filter for hipcrime some day...  Plain text.  Random headers.
 No "advanced features".  Good luck.

Much of the recent evolution in spam content techniques has been to get
away from ruses that are detectable as ruses.  It used to be a common
technique to insert random invalid html tags in spam.  Defeatable by
simply looking for invalid html tags or meta (eg: tag)/content ratios.
But they don't do that anymore, do they?  And meta/content ratios can be
_extremely_ high in legitimate email (just look at the gunk that outlook
produces for simple emails some day).

Look for identical gifs?  Well, they stopped emitting identical gifs
months ago.

I had some luck with looking for gif geometry (scan for gif prefixes
containing identical image dimensions).  That stopped working too.

>>> Again, when you have a LOT of users (and possibly MANY servers) behind a
>>> NAT router, denying mail from that IP address results in simply too much
>>> collateral damage. More to the point, it's a very blunt instrument for
>>> the job, and it's relatively simple to do very much better.
>>

But in fact, I do.

Let's cut to the chase.  Your argument about source IP blocking being a
blunt instrument is well-taken and true.  It _is_ a blunt instrument.

So's a sledgehammer.  You wouldn't use a sledgehammer to install window
trim would you?  But a carpenter probably still has and uses a
sledgehammer for a different part of the same job of building a house.

Your argument assumes that IP-blocking is the ONLY technique being used
- - if all you have is IP-blocking, trying to discriminate between good
and bad content from a given IP (say an ISPs MTA) doesn't work.

Well, yeah, duh ;-)

Your argument assumes that the administrator hasn't made any effort to
make an intelligent choice on which DNSBL to use - and is hence at the
mercy of capricious listings of any old random IP.

Well, yeah, if you've chosen to use BLARS, you get what you deserve.

Yup both true.  But so what?

What's to prevent you from doing the intelligent thing and using
multiple complementing techniques simultaneously?  What's to prevent you
from making intelligent choices of _which_ techniques to use?

Nothing.

The XBL, for example, is _extremely_ reliable at detecting compromised
end-user machines.  All by itself it will block 70-85% of all spam.  By
it's very nature it doesn't list real MTAs (eg: ISP smarthosts).  It has
a false positive rate lower than virtually any content filter.
Certainly _way_ lower than any notion of trying to block spam based on
detecting "advanced features".

Does that work to stop Nigerian 419s coming from Hotmail's mail server?
 Well, short of blocking them entirely, no.  Does the XBL "fire" on the
Hotmail mail servers?  No.  So you add other techniques to the filters
that will do it.

Does that mean that the XBL is useless?  No.  Does that mean it has
excessive false positives?  No.  Does that mean that the XBL has
excessive false negatives - well, yeah, entirely by itself, yes - but so
_what_?  Just add more filtering techniques.  Do those techniques used
for blocking 419s from Hotmail make XBL unnecessary?  Because they're
not perfect either.

Why not let the users choose?  How many of your users know what the XBL
is?  About 5 of mine _might_.  I'm not even sure you do.  How many of
them have spent the manhours needing to learn what knobs do a good job?
 Less than that.  How many of them would do nearly as a good job at it?
 Probably almost none.  It simply doesn't scale.  Unless you're a spam
geek.  My management pays me to be the spam geek, so they don't have to
spend hundreds of times as much turning everybody into a spam geek.

Even I, with over 10 years of experience in this game, could not do an
acceptable job on email landing in my _own_ mailbox using your
techniques.  We have to resort to the leverage provided by techniques
only possible/practical thru server-based, and discussed in places like
here and other mailing lists.  I'm the only one of our users that is on
this list.  And most other anti-spam mailing lists.  That (mailing
lists) is an important technique too.  But it obviously doesn't scale to
everyone in the company ;-)