Subject: Re: [Asrg] Quarantines and block lists

From: <[email-masked]>
Date: Mon, 29 Jan 2007 03:50:59 -0600

[comment #1]

> Subject: Re: [Asrg] Quarantines and block lists

> Lost Email is any email that is sent but not presented 
>to the user  
> and does not generate a DSN to the sender. 

The problem is, in the case of spam as in the case of 
viruses or worms, determining WHO the TRUE sender actually 
is.  It is safe to presume that it often is NOT the person 
listed in the From: header.

Therefore, I think we should note under "good practices" 
that it is NOT "good practice" to return ANY indication 
(after the original SMTP-time, and thus it's pretty much 
limited anyhow to just going back 'one' level) regarding 
mail blocked either as spam, or mail blocked because it 
contains worms or viruses.  In both cases, I believe we 
MUST presume that the true sender will not be accurately 
determinable.  To send ANY kind of a bounce message back 
is likely to create more harm than good, probably just 
haranguing another innocent third party.

Spammers COULD even use the expectation of the generated 
'bounce' message to reflect the bounce message, along with 
the copy of the original message, on to the 
always-intended third party, but this time the E-mail 
"originating" at the system that bounced it.

> When 
>referring to a  
> receiving system loosing email it is assumed that the 
>sending system  
> is following standard procedures.

It is a safe bet that spam and viruses/worms seldom IF 
EVER "follow standard procedures".

>> While spamtraps and valid user lists are helpful 
>>sometimes, they  
>> will not trigger if the spammer is using "fresh" E-mail 
>>address  
>> lists which are also devoid of spamtrap E-mail 
>>addresses.
> 
> Is there evidence that spammers have 100% spamtrap free 
>lists? Where  
> are they getting their addresses from and why can't we 
>seed spamtrap  
> addresses there? 

Well, I believe that a lot of addresses are harvested from 
active Yahoogroups and other such mailing lists.  Trying 
to "seed" spamtrap addresses there (via postings) would 
probably also generate legitimate replies to such posts.

Certainly not all spammers use "fresh" and valid lists but 
IMHO it's poor form to imagine that NONE do.

>> IP address blocking, again, is simply too crude.  A 
>>further and  
>> more serious problem is that you have basically no way 
>>to be 100%  
>> sure of who the "sending IP" actually is.
>>
>> For just one example, Yahoogroups
> 
> That's obviously a non-starter. Yahoogroups is a 
>subscription based  
> service. A valid spamtrap would not be subscribed to any 
>group  
> (unless it was designed to catch spammers harvisting 
>group addresses  
> in which case it would take special precautions to 
>exclude group  
> related emails from the spamtrap reports).

The point is that NON-spamtrap addresses get mail through 
Yahoogroups, and where you can't tell much or anything 
about where the E-mail actually originated.  So do you 
block Yahoogroups servers by IP address?

And if you don't do anything about Yahoogroups-forwarded 
spam, then you're leaving a gaping hole through which a 
large quantity of spam can be sent.

Let's DO agree that it's probably not feasible to do much 
of anything about spam contained within a "digest" (and we 
ought to recognize, at least, the problem of antispam 
contact detection triggering based on one message within 
an otherwise good digest of messages).  But it would be 
desirable to try to handle effective spam detection for 
individually forwarded Yahoogroups messages.

Again, IP-based detection blocking is nowhere close to 
viable for such cases.

>> Or let's say that Aunt Gertrude's machine gets infected 
>>by a worm,  
>> which starts sending out E-mails using Aunt Gertrude's 
>>account at  
>> the ISP, via the ISP's mail server. Are you going to 
>>block  
>> everything coming from that mail server?  What if it is 
>>a VERY  
>> large ISP?  Obviously, it would have been nice if that 
>>ISP's  
>> outgoing mail rules had caught the messages, but in this 
>>case let's  
>> recognize that they didn't.
> 
> Spam has to be stopped somewhere. But read the full set 
>of  
> proposals...  If a site has never received email from 
>that IP before  
> it would quarantine the first emails from that IP as in 
>proposal #1.  
> The automatic spamtrap network would generate an 
>advisory that the IP  
> is sending UBE. Then depending on the reputation of that 
>IP from all  
> sources the site would choose to a: continue the 
>quarantine, b:  
> accept the email and possibly apply additional filtering 
>or c: reject  
> the email from that IP.

I know that a lot of folks here seem to have a strong 
emotional commitment to the basic concept of IP-based 
blocking, but as unpleasant as it is to accept, that is 
simply NOT a very good approach.

>>> If the source is  emitting a sufficient volume of spam 
>>>this  
>>> happens before the initial probation period expires.
>>
>> Again, that's part of what I said about spammers 
>>engineering their  
>> mails so they squeak under whatever threshold is 
>>established to  
>> trigger spam alarms.  That can be done on content, 
>>sending rate, or  
>> any other generally/widely used criteria.
> 
> So I've forced the spammer to send less spam. This I 
>call progress.

No, you've only forced them to send it at a lower rate 
from probably more different locations.

ULTIMATELY what convinces them to send less spam is 
because it becomes unprofitable to do it, because 
virtually none of it reaches its intended destination.  As 
long as it's profitable to try, they can be expected to 
continue.

That's one reason why it's SO critical to put the kibosh 
on viruses and worms that recruit the spambot armies... 
that makes it harder to proxy the spam transmissions and 
diversify the origin points.  Slamming the door on HTML 
and attachments from unknown senders puts a MAJOR brake on 
that, literally overnight, and in a way that is very 
difficult to circumvent in any widespread way.
  
>>> I think a form of flood distribution  through a mesh of 
>>>peer nodes  
>>> would be more robust. The distribution  would feed local 
>>>databases  
>>> that could be queried directly or through  a DNS front 
>>>end. The  
>>> mesh can handle data from multiple sources and  needs no 
>>>central  
>>> authority.
>>
>> And how do you propose to prevent worms from targeting 
>>and flooding  
>> or doing other Denial of Service attacks on that mesh?
> 
> What port and IP address is the worm going to target? 
>You won't know  
> unless you've talked with the admin of that node and 
>even then you  
> might have a private port that only you know about. And 
>even if you  
> do manage to take out one node (or a dozen) the mesh 
>survives and  
> routes around the damage. 

That depends, of course, on how widespread the 'damage' 
is.  Again, I think that blocking spambot army recruiting 
is critical to reducing the severity of DoS attacks like 
this.

[snip]

>2. Do you know what a response code in SMTP is? 

Yes.  But if the incoming SMTP transaction is coming from 
an intermediary MTA, existing intermediaries frequently 
will try to generate a bounce E-mail (to what they (often 
wrongly) believe was the sender) based on their inability 
to forward a mail message to its final destination, and 
that frequently does more harm than good.

> 3. I never "bounce" anything.

Good, I wish that (at least for spam/worm cases) nobody 
else did either.

> The determination to quarantine email while waiting for 
>additional  
> information to become available or the possibility that 
>the sending  
> site will clean up the spam first should be easy to 
>make. Are you  
> perhaps referring to the rules for implementing proposal 
>#2?

Again, I don't consider IP address (at least not by 
itself) a good basis for determining what is and is not 
spam.

>> Apart from (as previously noted) the problem of 
>>determining after  
>> the fact with any real accuracy just who the "sending 
>>server"  
>> actually is, particularly for multi-hop mail deliveries. 
>>  It's  
>> easy enough to forget that not all E-mail deliveries are 
>>simple  
>> user->ISP->ISP->user transfers.
> 
>First you have the intranet delivery on the sending side
> 
> 	user-> (users ISP internal servers) -> Sending MTA
> 
> Then you have the internet delivery
> 
> 	Sending MTA ----> Receiving MTA
> or
> 	Sending MTA ----> (open relay) ---> Receiving MTA
> 
> And finally the intranet delivery on the receiving side
> 
> 	Receiving MTA -> (receiving ISP internal servers) -> 
>destination
> 
> I am only concerned with the internet delivery. Anything 
>else the  
> ISPs can handle on their own. So all internet email 
>deliveries are  
> simple ISP->ISP transactions.

Let me give you an example of why that's overly 
simplistic.

   sender -> sender's ISP MTA -> recipient's vanity 
domain's mail forwarding MTA -> recipient's "real" ISP 
receiving MTA -> recipient

(and of course, there could be multiple such levels of 
intermediary forwarding, or things like Majordomo servers 
involved too).

>> Vanity domains that provide mail forwarding are just one 
>>example of  
>> a way that intermediaries enter into a mail delivery 
>>chain.   
>> Aggregators/forwarders like Yahoogroups are another.
> 
> If the vanity domain allows you to create hundreds or 
>thousands of  
> forwarders to addresses you don't own to make it look 
>like a spammer  
> site it deserves to be blocked.

That's not the question.  The issue is that it forwards 
the mail (and let's presume for now that there is only one 
vanity domain forwarding taking place) to the receipient's 
ISP's MTA, which doesn't like it for whatever reason and 
sends back an SMTP rejection.  Now, what does the vanity 
domain mail server do with that?  They can't (presumably) 
send it back to the MTA *they* got the mail from;  that 
SMTP transaction is most likely long over.  Generally what 
happens (unfortunately) is they try to send back the 
warning as an E-mail message, and that turns out to be a 
Really Bad Idea.

Second point is that if the recipient ISP sees that it got 
spam or a virus forwarded through the vanity domain 
forwarding MTA, IP address blocking probably tries to 
block the vanity domain forwarding agent, because the 
receiving ISP's MTA can't reliably determine who the 
original sender was.

>>> that can be done with a 5xx on the next  delivery 
>>>attempt instead  
>>> of sending a bounce.
>>
>> And what if the next E-mail coming from that IP address 
>>is  
>> legitimate, non-spam E-mail?
> 
> If the decision is to reject all mail coming from that 
>IP 

(which, again, I will insist is simply NOT practical).

> then  
> returning the 5xx response will cause the sending MTA to 
>generate a  
> DSN that will be returned to the legitimate sender.

And, again, THEY cannot be 100% sure who that was, once 
the original SMTP transaction they received is completed. 
 Most typically, they try to send an E-mail back to the 
From: address, but for spam/virus rejections, that address 
is almost never valid.

>>> If the sending ISP is  on their toes an offending user 
>>>or host  
>>> will be shut down quickly and  the remaining mail will 
>>>be  
>>> delivered when the block advisory is  lifted or expires.
>>
>> And what happens to their legitimate outgoing mail in 
>>the meantime?
> 
> It sits in the quarantine until the block advisory is 
>lifted and then  
> gets delivered normally. Nothing is lost except for some 
>sleep by the  
> admin on duty for the sending ISP.

If the "sending ISP" (or company) is huge, and the spam or 
virus mail they occasionally send is frequent enough, the 
result is that they have frequent major delays in mail 
delivery, or even suffer serious and relatively permanent 
"e-mail constipation".

>>> Local whitelists can be used to allow mail to  bypass 
>>>the quarantine.
>>
>> That's fine, but (again) I believe that traditional, 
>>simple  
>> "whitelists" are not nearly adequate.  I believe that a 
>>viable  
>> whitelist MUST comprise a recipient/sender pair of 
>>addresses, AND  
>> additional criteria regarding what the recipient expects 
>>to receive  
>> from the sender in question.
>>
>> More to the point, once THAT is done, I believe that IP 
>>address  
>> blocking becomes essentially superfluous (and indeed, 
>>COUNTER  
>> productive).
> 
> None of my proposals say "block" anything. I say issue 
>advisories in  
> proposal #2, some users might want that information. 

I claim that a user (or ISP) who did not actually send the 
mail that is attributed (wrongly) to them virtually never 
gets anything of value of receiving a lot of complaints 
about the counterfeit use of their E-mail address.

> I propose  
> quarantining email when it's looking suspicious. The 
>sending ISP  
> knows where the email came from which is something that 
>the receiver  
> cannot know in all cases. 

The sending ISP only knows where it CLAIMS to have come 
from, much of the time.  And nothing absolutely guarantees 
that that address is even still online.  It's one thing to 
catch a problem during the SMTP transaction, but much more 
problematical to try to report it back after the fact.

> If anybody is going to decide 
>that email is  
> spam it is the sending ISP that has the best information 
>to make that  
> decision. 

I disagree.  If the ultimate determination about whether 
the mail is unwanted or not is made by the RECIPIENT (and 
that's what I believe is the ONLY person who can decide 
that) then the sending ISP is NOT in the position to 
decide.

> Proposal #4 provides the one key piece that 
>the sending ISP  
> doesn't have to determine that their user is sending UBE 
>(aka spam).
> 
> If you as an ISP are offered these free tools to help 
>control spam  
> emanating from your network are you going to say "no 
>thanks, we'll  
> just send the spam and let everybody else clean it up"?

So what do you propose they do with things that say "this 
is spam, we won't accept it" that are bogus?  That's what 
I've seen happening here, where Yahoogroups gets a bogus 
5xx message that mail to "me" hard bounced (because it 
contained a virus, usually) and as a result, they block 
ALL mail from getting sent out to me until I realize what 
happened and manage to reset things.  Of course, I have NO 
control over the problem.  (And they don't either, 
really... but at least they shouldn't consider a mail 
account hard bouncing just because of the CONTENT of ONE 
message that 'hard' bounced).

[comment #2]

> Subject: Re: [Asrg] Quarantines and block lists

>> To begin with, crude IP blocks will ALWAYS lose 
>>legitimate 
>> mail, because a single IP address can send both 
>>legitimate 
>> and zombie-generated mail.
> 
> We've been using the spamhaus dnsbls for several years 
>and I can't
> remember a single false positive. IP adresses which send 
>legitimate
> mails are simply not listed there (or if they are, there 
>are so few of
> them that it's indetectable). 

You're trying to pretend that a zombie prevents all 
legitimate mail from being sent out from that same IP 
address?  So that it's a binary "spamming/not spamming" 
machine?

If your argument is that the zombies are not using the 
commandeered machine's habitual upstream MTA, then I'll 
claim that spammers could trivially change their approach 
to do that.

Then, IP-based blocking either blocks (or doesn't block) 
ALL the E-mails originating from that MTA.

> It is true that they aren't sufficient, but they reduce 
>spam by quite a
> bit (they're by far the most frequent reason for a 
>permanent error) with
> practically zero false positives.

I believe that a good sender/recipient pairing mapping to 
a fine-grained permissions list is FAR more effective and 
satisfactory, with FAR less collateral damage to 
legitimate mail.  And besides, it is FAR harder for 
spammers and viruses/worms to circumvent and evade such a 
robust restriction.  I believe that to be SO effective 
that IP-based blocking becomes (at least in most cases) 
superfluous.

> I don't know why you insist that they would have to list 
>a single IP
> address which sends both legitimate and zombie-generated 
>mail. Just look
> at spamhaus and you can see that this is false.

Are you claiming that both spam and non-spam can't be 
funnelled through the same IP address?

Just one excellent counter-example is a small company that 
has their inhouse outbound mail server behind a NAT router 
(this is the case, in fact, for one of my consulting 
client companies).  If one of their other inhouse 
workstations ends up being infected by a worm, that 
infected machine could start to send either spam, or virus 
propagation messages, and those messages will also carry 
the IP address of the company's NAT router.

[comment #3]

> Please don't answer mails from different people in 
>different subthreads
> in the same mail. Your mails are already too long, and 
>an argument is
> hard to follow if the answer the content of one mail is 
>in the answer to
> a different mail.

I agree that it's suboptimal, but otherwise it is 
cumbersome to split a large message digest into individual 
fragments to allow quoting those messages for separate 
replies.

I don't know of any GOOD solution for that problem, but am 
open to ideas.  (Approaches that I otherwise might find 
practical are limited by the Webmail system I'm presently 
using).

>> >(1) I'm shopping at a major retailer's website and 
>>decide to sign up
>> >for their newsletter.  What happens if the confirmation 
>>message uses
>> >HTML content (not uncommon)?  
>> 
>> Obviously, it SHOULD NOT use HTML content,
> 
> But it does. And if you're a small ISP or IT department 
>and your users
> can't get their newsletter, guess who they'll blame? 

Given a choice of controlling spam, or the irresponsible 
use of HTML being sent to users who might not wish to deal 
with it, I think that being a little more nuanced and 
disciplined in the use of HTML is a small price to pay.

> You or the major
> retailer? 

To some degree, it's obviously a problem of educating the 
user of what you're doing, and (more importantly) WHY you 
are doing it.

If enough users make the decision, the 'large retailer' 
will clearly learn that they have to change their behavior 
at least slightly.  But that's not a bad thing.

> And what's more likely to change? The way you 
>filter mail or
> the way the major retailer sends their newsletter?

I guess it depends on whether you want to control spam 
effectively, and how important that is to your users.

I happen to think that's pretty important to a large 
percentage of users.

> You can do that if you're the DoD. The DoD recently 
>banned HTML and they
> can get away with it, because almost everyone they 
>exchange mail with
> is more interested in talking to the DoD than the DoD is 
>talking to
> them. If the DoD doesn't get your mails, that's your 
>problem, not
> theirs.

I think that as more people become aware of the costs of 
HTML (bulk, risk, etc), a lot fewer people will be willing 
to pay the price of it.

>> for the simple reason that said retailer CAN NOT be 
>>certain that the
>> person asking to be on their newsletter list is able to 
>>handle (or
>> WANTS to receive) HTML-burdened mail messages.
> 
> He doesn't care. 99% of his prospective customers can 
>handle it, and
> they will be enticed by the pretty pictures. He can 
>afford to lose 1%
> (who are probably the more critical customers anyway).

I'm not suggesting that they can't use HTML in (at least 
the 'pretty' version of) their newsletter.  Just that they 
need to send at least one plain-ASCII-text confirmation 
mail to the user FIRST, to negotiate the agreement to 
accept HTML in subsequent E-mail messages.
  
>> >Assuming that the confirmation and the newsletter come 
>>from the same
>> >source, that is, which also is not always the case.
>> 
>> True, although if that is recognized as important to 
>> legitimate mailers, they can certainly correct such 
>> questionable behavior.
> 
> That "questionable behaviour" is pretty much standard, 
>and there are
> good reasons for it.

Well, there are some "pretty good reasons" for greater 
discipline.
  
If we are going to get a serious handle on spam, viruses, 
and worms... then SOMEONE is going to have to change 
behavior AT LEAST SOMEWHAT.  The objective should be to 
make that process as rational and painless as possible, 
commensurate with efficacy.

>> >(3) My biggest customer is a giant corporation which 
>>requires its
>> >employees to use Microsoft Outlook and to attach a 
>>vCard-like
>> >signature to every email, using a background stationery 
>>and having an
>> >image of the company's latest logo because they're 
>>engaged in a major
>> >rebranding initiative.  Not surprisingly, they have 
>>massive turnover,
>> >so the people there who send email to a variety of 
>>people at my
>> >company changes every several weeks.  Who's responsible 
>>for upkeep of
>> >all of those individual agreements to exhange HTML with 
>>each other?
>> 
>> Again, the recipient could presumably set up a 
>>"catch-all" 
>> whitelisting which would enable selected E-mail HTML 
>> features and selected attachment types from any user at 
>>a 
>> given company's domain.  For example, allow (hell, even 
>> REQUIRE) V-card attachments or JPGs for mail from that 
>> domain, but (for example) not allowing scripting, 
>> executable attachments, or ActiveX.
> 
> Yes, he could. But he doesn't want to. 

Well, ultimately, it's all about choices, isn't it?  You 
can give people an effective spam filter, or you can allow 
them to turn it completely off.  At least if you OFFER it, 
they can't then bitch about the spam load they're getting 
if they CHOSE to not use it.

I've had some spam filters at some ISPs that were SO 
terrible that I turned them off and decided to do the 
filtering myself instead.

But you know the old adage about leading a horse to water.

At least this proposal WOULD BE EFFECTIVE (and with less 
collateral damage than the IP-based solutions that so many 
folks here seem to like) AND would block nearly all worms 
and viruses (even new ones) which isn't the case for these 
IP-based approaches.  AND it's not got any huge "gotchas" 
that force people to turn it off, like so many other 
basically-unusable filtering schemes are afflicted with.

> It's not his job to worry about
> which types of content a certain user (which he not even 
>know) might
> send to him in the future.

Fair enough.  But spam has become annoying enough to 
enough people by now that I don't think most of them would 
be terribly averse to helping guide a "smart" filter so it 
would effectively recognize and block spam... especially 
if (as they do so) they see an immediate and very 
noticeable payback for the investment.

>> >The only way your techniques can work effectively is if 
>> >you assume that
>> >the recipient is getting very little spam.  
>> 
>> I don't think it's particularly volume-sensitive... 
>>other 
>> than the fact, of course, that ANY system could 
>> conceivably be flooded by a sufficiently determined DoS 
>> attack.
> 
> It's a DoS attack on the user, not on the system. The 
>user has to look
> at the mails, decide whether they are legitimate, which 
>of their
> features he wants to accept from that sender (and 
>whether "that sender"
> means only that particular address, or the whole domain 
>or something in
> between).

Again, good software can determine what features need to 
be accepted to allow "mail like this" to be added to the 
allowable characteristics for the current sender.

Some users will tweak things more precisely than others 
will.

A large subset of users will probably find that the 
default filtering rules are perfectly acceptable for the 
great majority of their incoming mail.

[users receiving large volumes of mail]

>> I don't see why my approach won't handle such cases with 
>> ease.
> 
> That guy has to scan through 4000 messages each day to 
>check if there
> wasn't one which he did want. Maybe not every day, but 
>at least every
> time he suspects that a mail might have been blocked.

To some degree, that's going to be true for ANY filtering 
scheme.  At least he has the mail to look at... if it was 
blocked by IP address, he can't go look for it, since it's 
not on his (recipient) system at all.

Some people will look for false positives, some won't. 
 Some people will go to the effort to "tune" their filters 
to the best and most effective degree of accuracy, and 
others will run with the defaults.  That's to be expected.

>> >>Introductory E-mails should NOT automatically presume 
>>the desire or
>> >>willingness of the recipient to receive HTML-burdened 
>>E-mails.
>> >
>> >But if all spam were indistinguishable from 
>> >"introductory e-mails",
>> >where are you then?
>> 
>> No worse than at present.
> 
> Depends on what you define as "better" and "worse". You 
>will get less
> spam (about 50% less, I would guess from looking through 
>my spambox),
> but you will also get less legitimate mail (I don't know 
>how common HTML
> mail is on average - that seems to vary a lot). 

Yes, it does.  And there's nothing inherently wrong with 
HTML mail (other than its 3-5x greater bulk) IF you're 
able and willing to deal with it.  But senders ought not 
to PRESUME that EVERYONE feels that way.

>> And BETTER, since the "no attachment, no HTML" default 
>> rule would seriously cripple zombie spambot army 
>> recruitment.
> 
> Only if it is widely adopted. Which it won't be, because 
>the early
> adopters will have to deal with severe problems.

I don't consider that a fair or accurate statement.  Early 
adopters can phase in the filtering, perhaps by starting 
with a relatively liberal default rule which is tightened 
as the whitelist is developed.  Again, that's an issue of 
the implementing software.  It doesn't invalidate the 
principle.

>> >The XBL, for example, is _extremely_ reliable at 
>> >detecting compromised
>> >end-user machines.  All by itself it will block 70-85% 
>> >of all spam.  
>> 
>> Fine.  But it also blocks LEGITIMATE mail coming from 
>> those machines, right?
> 
> There is no legitimate mail coming from those machines. 
>Maybe a message
> or two for every million messages being blocked. 
>Probably less.

I've already given a case earlier in this message of why 
that can't be counted on, or at least why we shouldn't 
base our antispam defenses on such 'lucky behavior' over 
the long term.  I won't repeat that example again here.

>> >By it's very nature it doesn't list real MTAs (eg: ISP 
>> >smarthosts).  
>> 
>> What if the spam is sent through ths "smarthost"?  You 
>> might claim that's not being done today, but nothing 
>> prevents spammers from doing that.
> 
> Then there is an ISP to complain to and the ISP knows 
>which of their
> users has a compromised machine and can do somthing 
>about it. 

And that's fine for small ISPs who can call a user 
(maybe).  Bigger ISPs with millions of users (especially 
where thousands or tens of thousands of their users could 
be infected) will find that kind of detailed 
followup/uninfection to be an extremely costly and 
time-consuming problem.

Big dot-com companies (Yahoo, Ebay/paypal, etc) almost 
universally have terrible customer service if you ever 
have to talk to a real HUMAN there.  And that's not only 
just demonstrating that they don't CARE... it's a huge 
problem for them.

> If the ISP
> doesn't do something about it, he may find that some may 
>refuse to
> accept mail from him or apply more stringent filters to 
>it. But that
> doesn't have anything to do with the XBL.

Again, it's a very blunt instrument to do IP-level 
blocking.  We can simply do a LOT better.

> You are assuming that IP-based reputation is used in an 
>all-or-nothing
> way. It isn't. There is more than one list, and just 
>because an IP
> address is or isn't on a specific list doesn't mean that 
>all mails from
> this address are rejected or accepted. It is one 
>criterion among many.

I consider that other criteria should carry more weight, 
then, and IP address carry much less weight.

>> >Subject: Re: [Asrg] Re: bounces, and anti-spam 
>> >principles
>> 
>> >>[how users configure their whitelist rules]
>> >>
>> >>>The problem being that out of the 60,000 seats here, 
>>perhaps less
>> >>>than 10 of them are able to competently configure a 
>>set of rules
>> >>>like what you have.  
>> >>
>> >>That's a software implementation issue, not an 
>>inherent 
>> >>problem in the approach.  I envision a button to click 
>> >>on 
>> > 
>>                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> >I guess that's the main reason why your ideas aren't 
>>met with wild
>> >enthusiasm here. You "envision" that this would be 
>>simple to build
>> >and to use, but it's not even clear that you have even 
>>built a
>> >prototype which you use yourself, much less deployed 
>>the system for
>> >"ordinary users", which aren't quite sure what an "HTML 
>>mail" is, and
>> >have never heard about an "ActiveX component".
>> 
>> I've built and have used portions of what I propose. 
>> Obviously it will work best if the proposed default 
>>rules 
>> are widely used.
>> 
>> As for "deploying it for ordinary users", I think I 
>>could 
>> make the same claim regarding all these DNS-based 
>> proposals that y'all are so fixated on.
> 
> No, absolutely not. Apart from not being fixated on 
>"DNS-based
> proposals" (if anything, I'm fixated on bayesian 
>filtering), I see two
> very important differences here: 
> 
> 1) DNSBLs like Spamhaus' SBL+XBL aren't "proposals". 
>They have been
> operational for years and their strengths and weaknesses 
>are well-known.

Straw man.  If existing approaches were satisfactory, then 
we don't have a problem and we're all wasting our time 
time.

If on the other hand we are going to do things DIFFERENTLY 
than they have been done before, then we should EXPECT 
that that new approach hasn't been implemented yet.

The fact that it hasn't been fully implemented yet doesn't 
mean that it's not promising.

Lots of things are designed and discussed and debated for 
a long time before the first working one is actually 
built.  Large passenger aircraft are one example.

I probably wouldn't go to the trouble to do a full 
customer-ready version of such software if it were just 
only me going to use it.  But Airbus wouldn't build the 
A380 if there were only going to be one of them, just as a 
Honda Accord wouldn't be practical to design and fully 
engineer if only one person were going to have one, 
either.
  
> 2) The user has absolutely nothing to do with them. I 
>can simply enable
> them on a system-wide basis and be confident that I will 
>maybe have to
> deal with one complaint every few years. 

By the same token, a large group of users will be content 
with default filtering and default content 
("SpamAssassin") filtering.  Some will care enough, and 
choose to tweak those.  Many will not.  At least the ones 
who CARE to do so CAN do so.

>> Again, good software implementation doesn't have to 
>>depend on users
>> understanding those details
> 
> I doubt that. In real life users get mails with 
>"advanced features" all
> the time and if they have to decide whether they can 
>safely enable them
> they need to understand them. 

Do you understand how your automatic transmission in your 
car works?  What percentage of drivers do you imagine do?

If enabling a relatively more dangerous feature (say, 
executable attachments), the software presumably should 
issue some pretty dire-sounding warnings about why the 
user maybe shouldn't do that... just to counter 
pfishing-type exploits where they try to convince people 
to release the safety.  (We saw some viruses/spam trying 
to do that, a few years back, telling users to please 
enable scripting and telling them how to do it... but I 
haven't seen any spam like that in quite a while now.).

> I predict that the average user will
> either have configured that system to accept anything 
>from anybody
> within a week or give up on email entirely.

If the software is well-designed, then I'd take on that 
bet with you.  :-)
  
>> I don't think where things fall on that particular 
>> equation are especially relevant to whether the approach 
>> would be a good one for more widespread adoption.  :-)
> 
> It is irrelevant whether it "would be good". User's 
>simply won't adopt
> it. Period. It will ask them questions they don't 
>understand twenty
> times a day, so they will resort to either clicking 
>"yes" all the time
> or "no" all the time. 

Again, that depends entirely on how well the 
human-engineering of the software is done.  Some companies 
traditionally do a bad job of stuff like that, but doesn't 
mean it can't be done right.

> As a result they won't get the 
>mail they want and
> they will still get mail they don't want. In addition 
>they will feel
> dumb because they don't understand all those questions, 
>and they will be
> angry at the computer (it should handle that stuff 
>without asking stupid
> questions) and the IT staff or ISP (it's their job to 
>handle that
> stuff). A week later the system will be shut off again 
>and the poor guy
> who thought "it would be good" will be fired.

I'm sure that's what the spammers would wish.

But the fact is that it will be MORE effective at blocking 
spam, nearly 100% effective at blocking worms and viruses 
(even previously unknown ones that no antivirus program 
identifies) and still give the user the ability to improve 
the detection to the degree they wish (or not, if they 
prefer).

Given the major improvement in spam reduction that is 
possible, the people who OUGHT to be fired are those 
naysayers who choose to instead pursue costly, compex and 
elaborate technical schemes which ultimately and 
predictably CAN NOT solve the problem as well.

>> >So Uncle Bob will see that a mail from Aunt Matilda was 
>>blocked
>> >because it contained an "executable attachment". Since 
>>he wants to
>> >get mail from Aunt Matilda (and Aunt Matilda is a nice 
>>lady, she
>> >wouldn't send him anything bad, would she?) he clicks 
>>on "allow
>> >E-mails like this from the same sender in the future". 
>>Oops!
>> 
>> Presumably he could tell from the rest of the content in 
>> the mail message that it really didn't look like E-mail 
>> from Aunt Matilda.
> 
> To do that he has to read it. What good is an anti-spam 
>scheme which
> requires me to read all my spam?

Well, to begin with, it will block nearly all HTML and 
attachment-containing E-mail from people you don't know.

I agree that some folks, especially initially, or those 
who get relatively small amounts of spam, will get a "warm 
and cozy" feeling by checking their quarantine/block 
folder to see what was filtered out (at least they CAN, 
which generally isn't the case in IP-type blocking).  As 
they develop more trust in the system, and get it more 
tuned, or as they get more spam end up in the reject bin, 
they'll probably come to trust its judgement and just dump 
that bin unread more and more often.

During the phase-in period, there will be more "training" 
taking place (both of the software, AND of the users!). 
 That's to be expected.

[comment #4]

> Subject: Re: [Asrg] Re: HTML-burdened email

>> (I don't know 
>>how common HTML
>> mail is on average - that seems to vary a lot). 
> 
> It's _very_ high here.  Corporate likes their HTML and 
>attachments.

That's fine, and I don't suggest that it never be used.

I do think it's impolite (at best) and presumptuous to 
send HTML E-mail to people without first determining that 
they are able and willing to accept mail in that format.

ESPECIALLY given the numerous ways that HTML hugely 
facilitates the task of spammers to get past antispam 
content filtering.

Accepting HTML ONLY AFTER negotiation from specific 
senders strikes me as a very small price to pay if it 
hugely reduces the spam problem (and it can).

Solving the spam problem DOES require some changes.  The 
trick is to make those changes both reasonable, and 
effective.

[snip]

>> 1) DNSBLs like Spamhaus' SBL+XBL aren't "proposals". 
>>They have been
>> operational for years and their strengths and weaknesses 
>>are well-known.
> 
> And default recommendations in many packages.  Such as 
>SpamAssassin.

In part, that's because SpamAssassin has a nearly 
impossible job to do if it has to deal with things like 
text-as-image (whether attachment or embedded), HTML, 
ActiveX, and such.  Once attachments, scripting, and HTML 
are excluded, SpamAssassin can be FAR more accurate and 
effective.

So it is understandable that they would suggest IP-based 
filtering to help for those more difficult cases.

It might even be the case that IP-based filtering MIGHT 
still be a valid component even with a first-stage 
fine-grained sender-based whitelisting.  Although I 
personally believe that by that point, IP-based filtering 
will be mostly redundant.

>> ...In real life users get mails with 
>>"advanced features" all
>> the time and if they have to decide whether they can 
>>safely enable them
>> they need to understand them. I predict that the average 
>>user will
>> either have configured that system to accept anything 
>>from anybody
>> within a week or give up on email entirely.
> 
> If we tried that here, the average user would be 
>screaming at me why
> their filters weren't working.  Or working too well.

I do expect that such filtering would be phased in during 
the initial period, rather than being turned on suddenly 
and abruptly.

> What good is an anti-spam scheme that, after you've seen 
>the one good
> html email from Aunt Matilda, blindly delivered you the 
>virus her
> machine sends the next day?  Or the spam forged in her 
>name?

That's why you don't just enable ALL HTML, or NO HTML. 
 Aunt Matilda isn't likely to do scripting, ActiveX, or 
other elaborate HTML features.  She might use fonts, 
color, boldface, stuff like that.  These all-or-nothing 
whitelists are, indeed, almost useless because they end up 
getting turned off.

It's unlikely that one would EVER allow receiving 
EXECUTABLE attachments from Aunt Matilda (indeed, most 
users would never enable receiving executable attachments 
from ANYBODY, at least if the software is sufficiently 
dissuasive).

The spam forged in Aunt Matilda's name would have to know 
WHOSE name to forge to get past MY filter, and MOREOVER 
would have to look "enough" like her mail that it didn't 
get filtered out (and it's likely not feasible to decide 
just what those criteria are for a given recipient).  It's 
a very winding, narrow gauntlet the spammers/virus authors 
would have to navigate through.

I believe that most of them won't be able to do that very 
successfully.

Gordon Peterson
http://personal.terabites.com
1977-2007  Thirty year anniversary of local area 
networking

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Dan Oetting <[email-masked]>
Date: Mon, 29 Jan 2007 10:03:33 -0700

> [comment #1]

<I'm just going to rant>
Your use of digests indicates that you haven't figured out how to use  
the mail filtering rules within you own mail reader yet you are  
pushing for user configured spam filtering?!? Try reading the whole  
post before responding. The answers to your objections are usually in  
the same post or have already been answered in the same thread. And  
speaking of threads, I started this thread without references to  
other threads so my mail reader would collect all the replies  
together. Your cross posting from other threads is causing the other  
replies to get commingled with replies to my thread. I've come to the  
conclusion that digests should be disbanded. You still haven't  
figured out that to be spam the email  must be not just unwanted but  
UNSOLICITED. If you don't want SPAM™ don't order it, there are other  
choices on the menu ("the ham, eggs sausage and SPAM doesn't have  
much SPAM in it"). Yahoogroups is a subscription based service. The  
email you get from Yahoogroups is what you ordered and is not spam.  
If you don't like the content you can cancel your subscription or  
close your forwarder. The email that Yahoo receives addressed to  
their groups might be spam. It is up to Yahoo to filter out the spam  
if they want to attract subscribers to their groups. It is not the  
job of any other mail system to edit what yahoo sends to it's  
subscribers.
<end of rant>

---

From: Chris Lewis <[email-masked]>
Date: Mon, 29 Jan 2007 18:00:26 -0500

[email-masked] wrote:
>> Lost Email is any email that is sent but not presented to the user 
> The problem is, in the case of spam as in the case of viruses or worms,
> determining WHO the TRUE sender actually is.  It is safe to presume that
> it often is NOT the person listed in the From: header.
> 
> Therefore, I think we should note under "good practices" that it is NOT
> "good practice" to return ANY indication (after the original SMTP-time,
> and thus it's pretty much limited anyhow to just going back 'one' level)
> regarding mail blocked either as spam, or mail blocked because it
> contains worms or viruses.  In both cases, I believe we MUST presume
> that the true sender will not be accurately determinable.  To send ANY
> kind of a bounce message back is likely to create more harm than good,
> probably just haranguing another innocent third party.

If you're doing your spam filtering at the front end, and ALWAYS do
filter hits as inline rejections, then this isn't an issue.

[Yes, there is a small amount of spam going thru "real" MTAs that
produce blowback in such a case.  But this is so rare these days to not
be a concern.  We're averaging less than a complaint a year about
misdirected NDRs (at 1.6M rejected per day).

But obviously, you can't do that in the UA.

Therefore, with UA-level filtering, you _cannot_ provide valid feedback
to legitimate senders unless you tolerate unacceptable levels of
collateral damage.

As sender feedback is critical as a counter-FP measure in most
environments, you're completely screwed if you insist on UA-level
filtering.  Sender feedback is fundamentally incompatible with UA-level
filtering.

> The point is that NON-spamtrap addresses get mail through Yahoogroups,
> and where you can't tell much or anything about where the E-mail
> actually originated.  So do you block Yahoogroups servers by IP address?

No, why would you have to?  But you can, if you wish, use
X-Originating-IP (or analogous headers).

> And if you don't do anything about Yahoogroups-forwarded spam, then
> you're leaving a gaping hole through which a large quantity of spam can
> be sent.

Using DNSBLs doesn't require you to "don't do anything about
Yahoogroups-forwarded" spam.

> Again, IP-based detection blocking is nowhere close to viable for such
> cases.

So what?

> I know that a lot of folks here seem to have a strong emotional
> commitment to the basic concept of IP-based blocking, but as unpleasant
> as it is to accept, that is simply NOT a very good approach.

It's an excellent approach when used correctly.  Using it _alone_ and
expecting it to work acceptably on _all_ circumstances is not using it
correctly.

As is any other filtering technique.

I have a strong commitment to _effective_ and reliable filtering.
DNSBLs provide part of that answer.  Other methods provide plug the
holes, complement DNSBLS, and deal with things that the DNSBLs can't (or
haven't yet) caught.

10 years ago we started with content filters only, and could only do
extremely limited amounts of IP-based blocking.  As of 2001, that had
simply become totally inadequate.  These days it's about 90% DNSBL, and
10% content.

> That's one reason why it's SO critical to put the kibosh on viruses and
> worms that recruit the spambot armies... that makes it harder to proxy
> the spam transmissions and diversify the origin points.  Slamming the
> door on HTML and attachments from unknown senders puts a MAJOR brake on
> that, literally overnight, and in a way that is very difficult to
> circumvent in any widespread way.

a) it really doesn't put the brakes on.  Think bare text with the
appropriate human engineering and links.

b) slamming the doors on html and attachments from even unknown senders
has a vastly higher FP rate than anything else with "normal" ISP flows.