Subject: Re: [Asrg] What are the IPs that sends mail for a domain?

From: Gordon Peterson <[email-masked]>
Date: Mon, 22 Jun 2009 07:19:04 -0500

And the circle goes round and round.

Periodically, I feel it necessary to point out some of the serious flaws in all 
of these IP-based "authentication" type schemes.

The first one has been pointed out, but perhaps not strongly enough.  IT IS 
STUPID AND COUNTERPRODUCTIVE TO BOUNCE NOTICE OF NON-DELIVERY TO RECOGNIZED SPAM 
E-MAILS.  It can double OR TRIPLE the bandwidth wasted by the original spam, in 
part because (at least in the case of spam which has been relayed one or more 
times) it is problematical to know WHO to send the bounce message to.

In my personal mailboxes I have (way) more than 50,000 archived bounceback 
messages to e-mails which I have never sent... just because they have a (forged, 
and generally invalid) From: address that is supposedly in one of my domains.

Since I haven't sent these messages (neither intentionally, nor by irresponsible 
management of my systems here) there is NOTHING I can do to prevent such 
messages.  Meanwhile, the handling of the (worthless) bounce messages multiply 
by perhaps several times the bandwidth wasted due to the original spam.

Barracuda spam blocking systems are particularly irresponsible by apparently not 
explaining to their users WHY bouncing spam back to the sender is a Bad Idea, 
resulting in their (even less clever) users often apparently leaving that option 
set.

Also let me reiterate (as was pointed out) that sending inquiry messages to try 
to authenticate a valid mail agent LIKEWISE multiplies the bandwidth already 
wasted by the original spam.

I believe that ultimately, the best way to deal with spam (okay, we're talking 
principle here, not necessarily given existing, insufficiently clever mail 
clients) is to simply deliver the spam to the recipient's system, and let their 
system decide which mail is wanted, and which is not, and to either simply 
delete or archive somewhere the mail which the recipient user's rules determine 
is not wanted.  I do not consider a bounceback message to be necessary or even 
desirable if a message is found to be spam/virus/phishing ... in part because 
you cannot reliably determine who the original, legitimate sender is (even if 
there IS one) to send the bounceback message to.

Furthermore, and I've mentioned this before, my domains that I use for my e-mail 
(including terabites.com) generally are handled by my domain provider, although 
if I am away from home (say, at an Internet cafe, perhaps onboard a cruise ship 
or airport kiosk or at a public library somewhere, to name several examples) I 
(a) still want to use my own From: address for reply or posting permission 
purposes, even though (b) I might not have ANY say at all regarding what 
outgoing mail server(s) are used for mail submitted from the location that I 
happen to be sending from.  The fact that I am sending outgoing mail though an 
unfamiliar and inhabitual location doesn't mean that it's not legitimate, and 
I'm certainly not going to switch my "From" address for such messages to some 
other "From" address just because it matches somehow the mail server that I 
happen to be sending through.

The fact that it's easier, or cheaper, or otherwise "more efficient" to do 
antispam blocking using some halfassed, braindead scheme which doesn't work 
reliably or well for (even some admittedly small) legitimate mail transmissions 
doesn't make that the right solution.

Another situation is where an accounting system at one of my consulting clients 
generates and sends electronic invoices, EFT notices, price updates, etc to 
their customers.  For these cases, it is VERY helpful for their own inhouse 
outgoing LAN mail server (which maybe doesn't try to handle incoming mail at 
all) is going to try to send outgoing mails... if for no other reason than to 
have a local, inhouse log that evidences the delivery of the e-mail not just to 
a relay somewhere, but actually to (usually) the mail server associated with the 
destination indicated for the e-mail message.  It is FAR less useful to only 
have the company's SMTP logs evidence to the delivery to an upstream ISP's 
outgoing mail server.

Yet another case is where a traveling salesperson connects via a prospect's WiFi 
connection during a sales call visit on-site to his customer, and where that 
host's corporate network policy blocks sending of port 25 messages other than 
to/through that company's own outgoing SMTP server.  The same situation occurs 
when a private individual is on holiday visiting (or staying with) a relative 
whose internet connection is provided by a different provider.  Again, sometimes 
legitimate mail must be routed through an inhabitual outgoing mail server.

Anyhow, these braindead schemes about trying to decide whether a mail server is 
or is not supposed to be sending mail for a given return address, or blocking 
all mail being forwarded by a widely shared mail relay (based on its IP address) 
just because ONE of the (even tens or hundreds of thousands of) users of that 
same relay happened to get infected, is just insane.  It's not sufficient that 
the scheme initially looks appealing because it works "much" of the time.

I still believe that a far better and more worthwhile direction for spam 
blocking involves a combination of tools, probably much of it performed at the 
receiving end, involving a finely grained discrimination tailored to familiar 
versus unfamiliar (to the recipient) e-mail senders.  This is not unlike the way 
locks work... they generally provide a series of grooves, chamfers and cuts (to 
the key BLANK!) which prevent the vast majority of presented keys from even 
being inserted INTO the lock, before the pins of the lock do the final 
determination based on how the individual key has been cut.

In the case of E-mail, the corresponding policy could screen incoming e-mail 
messages based upon the characteristics _expected_ in e-mail coming from 
specific individual (familiar) senders (this is not unlike the technique that an 
intelligent human reader would use, when they open an e-mail claiming to come 
from a company or friend and the e-mail upon opening not looking "right" based 
on what we would expect that company or friend to be sending.  An example is 
cases where companies like Grouply manage to convince naive users to provide 
Grouply with the user's e-mail credentials, and where Grouply then uses the 
user's e-mail identity to send Grouply's spam... and when the recipient opens 
the message, it clearly doesn't look like an e-mail that Aunt Martha typically 
sends).

E-mail coming from unfamiliar correspondents can be held to a (even much) 
higher-than-usual standard regarding the ground rules for what is acceptable and 
what is not.  As a recipient, for example, I would be willing to state that I 
don't want mail containing HTML or attachments (or more than, say, 50K or 100K) 
from unfamiliar senders.  That would block (and in a robust way!) misrepresented 
HTML links (a key part of most phishing exploits), malicious scripting, ActiveX 
exploits, infectious attachments, and the like.  It's also noteworthy that such 
a policy blocks in one fell swoop nearly all the ruses and tricks that spammers 
use to try to evade (SpamAssassin-like) antispam content filtering... meaning 
that such filters suddenly become a great deal more effective and reliable.

It seems best if such filtering is largely done at the recipient user's end, and 
preferably in conjunction with their mail software... so that if they are 
looking for an expected e-mail message, and find it in their spam folder, they 
can have (say) a simple dialog box pop up that offers the user to "allow mail 
like this in the future from this sender."

It makes it a significantly harder challenge for spammers and abusers to evade 
antispam protections when they don't even know what criteria are used by 
specific recipients, or what From addresses those recipients might have "cut 
their (individual) keyway" to accept.  (And again, note that this is NOT just a 
simple 'whitelist' scheme... since it will accept mail coming from unfamiliar 
senders, it only just holds such senders to a particularly strict standard for 
what can, and must not be, contained in the e-mails those unfamiliar senders 
send out.)

-- 

Gordon Peterson II
http://personal.terabites.com
1977-2007:  Thirty year anniversary of local area networking

---

From: John Levine <[email-masked]>
Date: 22 Jun 2009 12:51:24 -0000

>And the circle goes round and round.

My, we have a lot of dead horses here.

>STUPID AND COUNTERPRODUCTIVE TO BOUNCE NOTICE OF NON-DELIVERY TO RECOGNIZED SPAM 

Yes, we know.  It's been best practice for years to reject mail you're not
planning to deliver, not bounce it.  There are, of course, a lot of dusty
MTAs still doing worst practices, but our ability to fix them is limited.

>Also let me reiterate (as was pointed out) that sending inquiry
>messages to try to authenticate a valid mail agent LIKEWISE
>multiplies the bandwidth already wasted by the original spam.

Callbacks are widely discredited other than among a few small
filtering vendors who think they're the secret sauce to keep the users
paying.  I routinely block all connections from hosts where I see C/R
callbacks, and I doubt I'm the only one.

>to/through that company's own outgoing SMTP server. 

It's been best practice for a decade to use SUBMIT or a tunnel back to
your own host to send mail.  These days it's just laziness to do
anything else.  As someone else asked a few minutes ago, are there any
significant mail systems that still don't provide SUBMIT?

>higher-than-usual standard regarding the ground rules for what is
>acceptable and what is not.

Yes, that's why we've been working on mail authentication a la DKIM for
several years, to allow us to recognize known senders reliably.

R's,
John

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 22 Jun 2009 15:56:04 +0200

Gordon Peterson wrote:
> I believe that ultimately, the best way to deal with spam (okay, we're 
> talking principle here, not necessarily given existing, insufficiently 
> clever mail clients) is to simply deliver the spam to the recipient's 
> system, and let their system decide which mail is wanted, and which is 
> not, and to either simply delete or archive somewhere the mail which the 
> recipient user's rules determine is not wanted.  I do not consider a 
> bounceback message to be necessary or even desirable if a message is 
> found to be spam/virus/phishing ... in part because you cannot reliably 
> determine who the original, legitimate sender is (even if there IS one) 
> to send the bounceback message to.

Except that you cannot reliably determine whether a message is spam. 
Machines do not understand human language. They make spam/ham 
decisions based on indeterministic criteria. Why do we run journaled 
file system on RAID drives when we would want to kill random files for 
the sake of spam abatement?

> [...] I (a) still want to use my own 
> From: address for reply or posting permission purposes, even though (b) 
> I might not have ANY say at all regarding what outgoing mail server(s) 
> are used [...]
> 
> The fact that it's easier, or cheaper, or otherwise "more efficient" to 
> do antispam blocking using some halfassed, braindead scheme which 
> doesn't work reliably or well for (even some admittedly small) 
> legitimate mail transmissions doesn't make that the right solution.

The difference between content filtering and however braindead 
authentication schemes is that the latter ones are deterministic, 
rather than easier, or cheaper, or otherwise "more efficient". 
Deterministic results mean that messages sent correctly are delivered 
independently of their content, uniformly.

IMHO, the correct way to go is to have a (weak) authentication scheme 
that works well enough for most relevant cases, and at the same time 
keep the existing system as a compatibility option for the cases where 
it doesn't work.

If you want to use prehistoric appliances deploying obsolete mail 
transmission methods that have historically proven to be vulnerable to 
spammers, you have to accept that your mail will be handled via 
compatibility options, including unreliable spam filtering. If you 
expect your From: address to be treated specially, be aware that any 
spammer can use it just like you do (and wait until spammers decide to 
relate recipients to the From: addresses that they whitelist.) 
Anything that is capable to deterministically recognize that you are 
really you, is an authentication scheme.

> Another situation is where an accounting system at one of my consulting 
> clients generates and sends electronic invoices, EFT notices, price 
> updates, etc to their customers.  For these cases, it is VERY helpful 
> for their own inhouse outgoing LAN mail server (which maybe doesn't try 
> to handle incoming mail at all) is going to try to send outgoing 
> mails... if for no other reason than to have a local, inhouse log that 
> evidences the delivery of the e-mail not just to a relay somewhere, but 
> actually to (usually) the mail server associated with the destination 
> indicated for the e-mail message.

How come you consider that log line reliable, given that it doesn't 
tell you whether the corresponding message has been "simply deleted or 
archive somewhere"?

> Yet another case is where a traveling salesperson connects via a 
> prospect's WiFi connection during a sales call visit on-site to his 
> customer, and where that host's corporate network policy blocks sending 
> of port 25 messages other than to/through that company's own outgoing 
> SMTP server.

Try port 587.

> blocking involves a combination of tools, probably much of it performed 
> at the receiving end, involving a finely grained discrimination tailored 
> to familiar versus unfamiliar (to the recipient) e-mail senders.  This 
> is not unlike the way locks work... they generally provide a series of 
> grooves, chamfers and cuts (to the key BLANK!) which prevent the vast 
> majority of presented keys from even being inserted INTO the lock, 
> before the pins of the lock do the final determination based on how the 
> individual key has been cut.

Except that you want external unknown people to be able to contact you 
by email (otherwise you could just use an unofficial port different 
than 25 and relay mail privately within your closed group.) It would 
be also appreciable if newcomers could open brand new ESP shops and 
run their SMTP servers without having to spend years learning the 
intricacies of finely grained discrimination tools.

As those tools evolve, desperately seeking new ways to find needles in 
the growing haystack of spam, users experience more and more 
malfunctions. If you dig into the actual rules and data that your 
filters base their decisions upon, you may get surprised.

---

From: Ian Eiloart <[email-masked]>
Date: Mon, 22 Jun 2009 15:16:19 +0100

--On 22 June 2009 07:19:04 -0500 Gordon Peterson <[email-masked]> wrote:

> Periodically, I feel it necessary to point out some of the serious flaws
> in all of these IP-based "authentication" type schemes.
>
> The first one has been pointed out, but perhaps not strongly enough.  IT
> IS STUPID AND COUNTERPRODUCTIVE TO BOUNCE NOTICE OF NON-DELIVERY TO
> RECOGNIZED SPAM E-MAILS.  It can double OR TRIPLE the bandwidth wasted by
> the original spam, in part because (at least in the case of spam which
> has been relayed one or more times) it is problematical to know WHO to
> send the bounce message to.
>
> In my personal mailboxes I have (way) more than 50,000 archived
> bounceback messages to e-mails which I have never sent... just because
> they have a (forged, and generally invalid) From: address that is
> supposedly in one of my domains.
>
> Since I haven't sent these messages (neither intentionally, nor by
> irresponsible management of my systems here) there is NOTHING I can do to
> prevent such messages.

There is, actually. If you publish SPF records with a strong -all, then 
recipients can easily decide to reject (not bounce) messages. Add DKIM 
signatures, and they'll be able to tell when someone has forwarded your 
legitimate email.

>  Meanwhile, the handling of the (worthless) bounce
> messages multiply by perhaps several times the bandwidth wasted due to
> the original spam.
>
> Barracuda spam blocking systems are particularly irresponsible by
> apparently not explaining to their users WHY bouncing spam back to the
> sender is a Bad Idea, resulting in their (even less clever) users often
> apparently leaving that option set.
>
> Also let me reiterate (as was pointed out) that sending inquiry messages
> to try to authenticate a valid mail agent LIKEWISE multiplies the
> bandwidth already wasted by the original spam.
>
> Furthermore, and I've mentioned this before, my domains that I use for my
> e-mail (including terabites.com) generally are handled by my domain
> provider, although if I am away from home (say, at an Internet cafe,
> perhaps onboard a cruise ship or airport kiosk or at a public library
> somewhere, to name several examples) I (a) still want to use my own From:
> address for reply or posting permission purposes, even though (b) I might
> not have ANY say at all regarding what outgoing mail server(s) are used
> for mail submitted from the location that I happen to be sending from.
> The fact that I am sending outgoing mail though an unfamiliar and
> inhabitual location doesn't mean that it's not legitimate, and I'm
> certainly not going to switch my "From" address for such messages to some
> other "From" address just because it matches somehow the mail server that
> I happen to be sending through.
>
> do antispam blocking using some halfassed, braindead scheme which doesn't
> work reliably or well for (even some admittedly small) legitimate mail
> transmissions doesn't make that the right solution.
>
> updates, etc to their customers.  For these cases, it is VERY helpful for
> their own inhouse outgoing LAN mail server (which maybe doesn't try to
> handle incoming mail at all) is going to try to send outgoing mails... if
> for no other reason than to have a local, inhouse log that evidences the
> delivery of the e-mail not just to a relay somewhere, but actually to
> (usually) the mail server associated with the destination indicated for
> the e-mail message.  It is FAR less useful to only have the company's
> SMTP logs evidence to the delivery to an upstream ISP's outgoing mail
> server.
>
> SMTP server.  The same situation occurs when a private individual is on
> holiday visiting (or staying with) a relative whose internet connection
> is provided by a different provider.  Again, sometimes legitimate mail
> must be routed through an inhabitual outgoing mail server.
>
> Anyhow, these braindead schemes about trying to decide whether a mail
> server is or is not supposed to be sending mail for a given return
> address, or blocking all mail being forwarded by a widely shared mail
> relay (based on its IP address) just because ONE of the (even tens or
> hundreds of thousands of) users of that same relay happened to get
> infected, is just insane.  It's not sufficient that the scheme initially
> looks appealing because it works "much" of the time.
>
> to familiar versus unfamiliar (to the recipient) e-mail senders.  This is
> not unlike the way locks work... they generally provide a series of
> majority of presented keys from even being inserted INTO the lock, before
> the pins of the lock do the final determination based on how the
> In the case of E-mail, the corresponding policy could screen incoming
> e-mail messages based upon the characteristics _expected_ in e-mail
> coming from specific individual (familiar) senders (this is not unlike
> the technique that an intelligent human reader would use, when they open
> an e-mail claiming to come from a company or friend and the e-mail upon
> opening not looking "right" based on what we would expect that company or
> friend to be sending.  An example is cases where companies like Grouply
> manage to convince naive users to provide Grouply with the user's e-mail
> credentials, and where Grouply then uses the user's e-mail identity to
> send Grouply's spam... and when the recipient opens the message, it
> clearly doesn't look like an e-mail that Aunt Martha typically sends).
>
> acceptable and what is not.  As a recipient, for example, I would be
> willing to state that I don't want mail containing HTML or attachments
> (or more than, say, 50K or 100K) from unfamiliar senders.  That would
> block (and in a robust way!) misrepresented HTML links (a key part of
> most phishing exploits), malicious scripting, ActiveX exploits,
> infectious attachments, and the like.  It's also noteworthy that such a
> policy blocks in one fell swoop nearly all the ruses and tricks that
> spammers use to try to evade (SpamAssassin-like) antispam content
> filtering... meaning that such filters suddenly become a great deal more
> effective and reliable.
>
> It seems best if such filtering is largely done at the recipient user's
> end, and preferably in conjunction with their mail software... so that if
> they are looking for an expected e-mail message, and find it in their
> spam folder, they can have (say) a simple dialog box pop up that offers
> the user to "allow mail like this in the future from this sender."
>
> It makes it a significantly harder challenge for spammers and abusers to
> evade antispam protections when they don't even know what criteria are
> used by specific recipients, or what From addresses those recipients
> might have "cut their (individual) keyway" to accept.  (And again, note
> that this is NOT just a simple 'whitelist' scheme... since it will accept
> mail coming from unfamiliar senders, it only just holds such senders to a
> particularly strict standard for what can, and must not be, contained in
> the e-mails those unfamiliar senders send out.)

-- 
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/

---

From: Bill Cole <[email-masked]>
Date: Mon, 29 Jun 2009 13:36:00 -0400

> --On 22 June 2009 07:19:04 -0500 Gordon Peterson <[email-masked]>
[...]
> signatures, and they'll be able to tell when someone has forwarded your

Do you have any evidence that this actually works to any detectable degree?

I have solid proof that it is far from perfect, but I only have a handful of 
addresses that ever had significant bogus bounce flow in the one domain I 
could safely use in a SPF '-all' effectiveness test. The first 5 years of 
that test have shown a slow drop in the rate of bad bounces in general 
offered to that domain, but it isn't much more proportionally than the drop 
from a dribble to a trickle that I've seen for a domain with no SPF record. 
The noise in my minuscule and weakly controlled data makes it quantitatively 
worthless, but on a qualitative basis it makes clear that strong SPF records 
are not yet a strong universal tool for preventing blowback bounces.

If you are aware of SPF being any more useful than prayer at controlling 
blowback, please share it.

---

From: Ian Eiloart <[email-masked]>
Date: Tue, 30 Jun 2009 10:55:04 +0100

--On 29 June 2009 13:36:00 -0400 Bill Cole <[email-masked]> 
wrote:

> I have solid proof that it is far from perfect, but I only have a handful
> of addresses that ever had significant bogus bounce flow in the one
> domain I could safely use in a SPF '-all' effectiveness test. The first 5
> years of that test have shown a slow drop in the rate of bad bounces in
> general offered to that domain, but it isn't much more proportionally
> than the drop from a dribble to a trickle that I've seen for a domain
> with no SPF record. The noise in my minuscule and weakly controlled data
> makes it quantitatively worthless, but on a qualitative basis it makes
> clear that strong SPF records are not yet a strong universal tool for
> preventing blowback bounces.
>

Well, my claim is a theoretical one, and perhaps a hope for the future. It 
probably isn't yet effective. I do believe that it will be one day.

However, I do believe that people should take SPF records into account when 
deciding whether to generate bounce messages. It should not be a problem if 
there's a positive SPF or DKIM match. You shouldn't do it if there's an SPF 
fail. It's harder to decide what to do in the absence of SPF, or a neutral 
or softfail result.

I'd suggest that to drive up adoption of SPF, don't bounce for a neutral or 
softfail result (instead give a 5xx error), but (this will be 
controversial) feel free to bounce into domains that aren't trying to help 
- ie domains that don't publish SPF records at all. (ducks).

---

From: Rich Kulawiec <[email-masked]>
Date: Tue, 30 Jun 2009 07:11:05 -0400

On Tue, Jun 30, 2009 at 10:55:04AM +0100, Ian Eiloart wrote:
> However, I do believe that people should take SPF records into account 
> when deciding whether to generate bounce messages. 

Despite the ostentatious claims made by its originator ("Spam as a
technical problem is solved by SPF"), SPF has no anti-spam value.
Nor should it be used when deciding whether to generate a bounce:
the answer to that is always "no".  It's far better to reject (not
to mention far simpler, with any sane MTA) and thus greatly diminish
the possibility of outscatter/backscatter spam.

---Rsk

---

From: Dotzero <[email-masked]>
Date: Tue, 30 Jun 2009 13:17:31 -0400

> the possibility of outscatter/backscatter spam.

I'm going to agree with Rich that it's better to reject than to
bounce. I can't speak to whether SPF has anti-spam value generally but
my experience with publishing -all records for a number of well known
large sending sites is that it is useful in addressing phishing at the
risk of a small amount of breakage. (Different folks will have
different thresholds for this type of tradeoff) This is particularly
true if used in conjunction with DKIM signing all outbound mail and
making an assertion that one signs all mail. For other types of
senders YMMV.

As far as whether this is empirical or anecdotal, my statements are
based on a corpus of approximately 750 million sent emails with
analysis of outbound MTA logs for immediate rejects/bounces as well as
bounces/DSNs that come in later.