Subject: Re: [Asrg] What are the IPs that sends mail for a domain?

From: Ian Eiloart <[email-masked]>
Date: Wed, 01 Jul 2009 11:17:32 +0100

--On 30 June 2009 07:11:05 -0400 Rich Kulawiec <[email-masked]> wrote:

> On Tue, Jun 30, 2009 at 10:55:04AM +0100, Ian Eiloart wrote:
>> However, I do believe that people should take SPF records into account
>> when deciding whether to generate bounce messages.
>
> Despite the ostentatious claims made by its originator ("Spam as a
> technical problem is solved by SPF"), SPF has no anti-spam value.

Given that you don't have a precise definition of spam, that's a pretty 
strong claim in itself.

> Nor should it be used when deciding whether to generate a bounce:
> the answer to that is always "no".  It's far better to reject (not
> to mention far simpler, with any sane MTA) and thus greatly diminish
> the possibility of outscatter/backscatter spam.

The point of SPF is to authenticate the sending domain. If the IP address 
is authorised (by the domain owner) to send mail from the sender domain, 
then bouncing mail into that domain isn't going to be causing backscatter, 
unless the domain lacks internal controls over message submission. If it 
does lack those internal controls, then the users of the domain can blame 
the domain owner.

I guess there can also be issues where two distinct domains share the same 
outbound IP addresses, through an email service provider. In that case, the 
email service provider is the responsible party that needs to be held to 
account. They need to ensure either (a) separation of domains by outbound 
IP address combined with accurate SPF records, or (b) proper implementation 
of MSA on all the domains that they provide service for.

Backscatter is a problem, but bounce messages do have advantages over 5xx 
error codes when it comes to communicating with the sender. For example, 
you can't know what the sending MTA is going to do with a 5xx error code - 
they might just drop it. DSNs were invented for a reason, and it's a shame 
to lose them entirely - even when you have reason to believe that the 
return-path (or at least the return-path domain) isn't forged.

> ---Rsk
> _______________________________________________
> Asrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/asrg

-- 
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/

---

From: John Levine <[email-masked]>
Date: 1 Jul 2009 12:30:54 -0000

>return-path (or at least the return-path domain) isn't forged.

I don't think it's a very good idea to try to guess how other people's
MTAs might be broken.  It's true that some don't pass back 5xx codes
properly, but it's equally true that some mangle or discard DSNs.  The
reason that DSNs were invented is that SMTP is still store and
forward, and sometimes you can't tell during the SMTP session whether
a delivery will subsequently fail.  But if you can tell, which these
days is the vast majority of the time, 5xx is cheaper and less likely
to cause collateral damage.

R's,
John

---

From: Ian Eiloart <[email-masked]>
Date: Wed, 01 Jul 2009 14:34:43 +0100

--On 1 July 2009 12:30:54 +0000 John Levine <[email-masked]> wrote:

>  But if you can tell, which these

Absolutely true, and I certainly don't advocate sending DSNs when you know 
that its going to cause backscatter - eg, when you get an SPF fail.

But, what good reason is there for NOT using a DSN when you get an SPF 
pass?

---

From: John Leslie <[email-masked]>
Date: Wed, 1 Jul 2009 11:00:32 -0400

Ian Eiloart <[email-masked]> wrote:
> 
> The point of SPF is to authenticate the sending domain.

   I don't believe SPF does any such thing. Domains can publish SPF RRs,
but those can't reasonably be said to "authenticate" anything, least of
all the "sending domain."

> If the IP address is authorised (by the domain owner) to send mail from
> the sender domain,

   That's closer... But I'd argue that no SPF construct "authorizes"
sending email. In practice, I think it's quite clear that SPF constructs
merely express probabilities.

> unless the domain lacks internal controls over message submission.

   Of course, rather few domains other than corporate domains with
administrators more-than-average familiar with SMTP have reasonable
"internal controls over message submission". :^(

> If it does lack those internal controls, then the users of the domain
> can blame the domain owner.

   Indeed they can... does that actually accomplish anything?

> I guess there can also be issues where two distinct domains share the
> same outbound IP addresses, through an email service provider.

   Indeed, that is common...

> In that case, the email service provider is the responsible party that
> needs to be held to account.

   (which, BTW, is what CSV set out to do...)

> They need to ensure either (a) separation of domains by outbound IP
> address combined with accurate SPF records,

   Assuming they control either multiple IP addresses _or_ the SPF
records is risky. But even if they did, how would this lead to assigning
the responsibility correctly?

> or (b) proper implementation of MSA on all the domains that they
> provide service for.

   That is at least practial... But how does it lead to assigning the
responsibility correctly?

--
John Leslie <[email-masked]>

---

From: Dotzero <[email-masked]>
Date: Wed, 1 Jul 2009 11:12:13 -0400

> merely express probabilities.

What is the probability that you will receive legitimate email
originating from ibm.com?

ibm.com text = "v=spf1 -all"

---

From: Douglas Otis <[email-masked]>
Date: Wed, 1 Jul 2009 12:13:15 -0700

> The point of SPF is to authenticate the sending domain.

SPF(email-address)->pass provides authorization for Outbound MTAs.   
The rationale for offering SPF authorization might be to improve  
message acceptance or DSN rates, as recommended by AOL and MSN for  
example.  It would be risky to conclude SPF(email-address)->pass means  
other domains did not provide assess to the originator of the  
message.  It is fundamentally wrong to hold the wrong entity  
accountable.

> If the IP address is authorised (by the domain owner) to send mail  
> from the sender domain, then bouncing mail into that domain isn't  
> going to be causing backscatter, unless the domain lacks internal  
> controls over message submission. If it does lack those internal  
> controls, then the users of the domain can blame the domain owner.

SPF(email-address)->fail might only indicate a message had been  
forwarded.  Use of RFC 3464 and minimal DSN content with"multipart/ 
report" content types should occur irrespective of the SPF results  
when messages are returned post acceptance.  Nor should one assume SPF  
authorization fairly assigns blame for poor administration of Outbound  
MTAs.  How many ESPs even offer SLAs that ensure domain exclusivity  
when handling tens of thousands of domains?

> I guess there can also be issues where two distinct domains share  
> the same outbound IP addresses, through an email service provider.  
> In that case, the email service provider is the responsible party  
> that needs to be held to account. They need to ensure either (a)  
> separation of domains by outbound IP address combined with accurate  
> SPF records, or (b) proper implementation of MSA on all the domains  
> that they provide service for.

Use of verified EHLO IP address information should only be claimed by  
a _few_ domains over a period of time.  Seeing too many likely  
indicates the presence of a NAT, compromised systems, or both.  The  
domain providing stewardship over access to Outbound MTA should be  
assessed separately from domains that have purportedly originated the  
messages.  Even a cryptographically strong scheme like DKIM will not  
mitigate replay abuse, while it does help mitigate phishing.  On the  
other hand, SPF might enable convincing phish whenever SPF(email- 
address)->pass is assumed to authenticate domain sources.  It does not!

> Backscatter is a problem, but bounce messages do have advantages  
> over 5xx error codes when it comes to communicating with the sender.  
> For example, you can't know what the sending MTA is going to do with  
> a 5xx error code - they might just drop it. DSNs were invented for a  
> reason, and it's a shame to lose them entirely - even when you have  
> reason to believe that the return-path (or at least the return-path  
> domain) isn't forged.

Best practices should reduce DSNs, often by ensuring immediate  
rejection is enabled where possible.   Many of the DSNs from  
legitimate Inbound MTAs occur as a result of valid users not being  
known by border MTAs.  Those appear to be mostly from poorly  
integrated hybrid systems protecting an Exchange Server or offering  
stand-alone inbound filtering services.

-Doug

---

From: John Levine <[email-masked]>
Date: 1 Jul 2009 22:08:45 -0000

>But, what good reason is there for NOT using a DSN when you get an
>SPF pass?

Why should it do anything different from what it does the other 95% of
the time?  SPF is far from universal, and a whole lot of SPF lookups
end up saying "maybe".

R's,
John

---

From: Ian Eiloart <[email-masked]>
Date: Thu, 02 Jul 2009 10:17:59 +0100

> the time?  SPF is far from universal, and a whole lot of SPF lookups

You've avoided the question. SPF and DKIM are both growing quite rapidly. 
Some day, you'll get a definite answer for most of your mail. Or, perhaps 
it'll be some other sender domain authentication technology.

So, why should one NOT send a DSN when the domain is authenticated? You 
said "5xx is cheaper and less likely to cause collateral damage." Well, the 
cost is minimal, and there's some benefit when we're confident about the 
sender domain.

And, what's the risk collateral damage when the sender domain is 
authenticated? I guess there's a risk that the authentication mechanism has 
been compromised - but that's the responsibility of the domain owner, isn't 
it?

---

From: Ian Eiloart <[email-masked]>
Date: Thu, 02 Jul 2009 10:40:25 +0100

> merely express probabilities.

OK, so it's authorization rather than authentication. By "probabilities", 
are you referring to the probability of forwarding, or the use of some 
unauthorized MSA? Certainly, the latter need to be stamped out in the same 
way that open relays were. There are other ways to deal with the forwarding 
problem, and they'll be adopted in time as the value of SPF increases with 
take up.

>> then bouncing mail into that domain isn't going to be causing
>> backscatter,  unless the domain lacks internal controls over message
>> submission.
>

Yes, but they need to improve. They won't, unless they're held accountable.

>    Indeed they can... does that actually accomplish anything?

Yes. Not usually as fast as you'd like. But, companies (email service 
providers) go out of business if people stop using the service. For 
commercial users

> the responsibility correctly?

It prevents cross-domain abuse. When good.example shares IP addresses with 
bad.example on the outbound MTA, then users of bad.example can forge mail 
in the domain good.example and get themselves a free ride on the reputation 
of good.example - thus damaging the reputation of good.example

If the two domains have separate outbound IP addresses (very feasible with 
IPv6) then proper use of SPF can protect them from that abuse. Given that 
this is all in the control of a single ESP, that ESP can put in place the 
necessary protection for forwarding. This means that they can safely 
publish +ve records for a domain's allocted IP addresses, -ve records for 
the rest of the ESP's IP address space, and (for now) neutral or softfail 
for the rest of the world.

Assignment of responsibility should be to the domain owner, or the email 
address owner. It doesn't much matter which - they can sort that out 
themselves. This would be a huge improvement over the current situation, 
where you can only assign responsibility to an IP address. The main 
improvement is that non-technical people have some understanding of 
domains, but no understanding of IP addresses.

> responsibility correctly?

The point is that the ESP needs to prevent the sort of cross-domain abuse 
that I've referred to above. With proper implementation of MSA, they can 
detect or prevent the cross-domain abuse. That ensures that an SPF pass 
doesn't result in reputation being assigned to the wrong domain. It also 
ensures that bounce messages aren't delivered into the wrong domain.

---

From: Ian Eiloart <[email-masked]>
Date: Thu, 02 Jul 2009 11:23:53 +0100

> ibm.com text = "v=spf1 -all"

Nil. They don't use the domain for outbound email. They use country 
specific subdomains like @uk.ibm.com.

I'm not sure why they publish MX records for @ibm.com - perhaps they have 
some initial contact addresses @ibm.com, but don't reply using that domain.

It's very sensible of them to use the -all spf record, adding a little 
protection for their brand reputation.

Alternatively, this is a massive cock-up and a huge potential 
embarrassment. I don't think so, though. Our logs from June show no inbound 
email (either accepted or rejected) from the @ibm.com domain, but
a few dozen emails from @uk.ibm.com and some from the 'be', 'ca', 'us', 
'jp', 'hu' subdomains of ibm.com.

Exercise for the reader: why aren't spammers using the @ibm.com domain?
-- 
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/

---

From: John Levine <[email-masked]>
Date: 2 Jul 2009 13:54:09 -0000

>it'll be some other sender domain authentication technology.

That may be so at some time in the distant future, but for the
forseeable future there will always be a signficant number of maybes.

>So, why should one NOT send a DSN when the domain is authenticated?

Because rejects work better, and DSNs are a second rate substitute
when the recpient system can't tell during the SMTP session that the
delivery will fail.

You appear to be under the impression that a sender would obtain and
use valuable information from a DSN that can't be sent in a rejection,
which is completely contrary to my experience.  I don't spend an
enormous amount of time reading DSNs, but my mailing list software
does and all it really wants to know is the address that bounced,
which rejections reliably provide.  DSNs often give no hint what
address the bouncing message was sent to.  That's why we have to use
kludges like VERP.

R's,
John

---

From: Ian Eiloart <[email-masked]>
Date: Thu, 02 Jul 2009 15:10:34 +0100

> address the bouncing message was sent to.  That's why we have to use

I guess rejections are more useful for automated processes, but they're 
often converted to DSNs anyway. Certainly, our mailing list software only 
sees DSNs. VERP may be a kludge, but it is a solution.

DSNs have the potential to be more useful to humans if they're constructed 
by the MTA that has more information.  In fact, we never reject a message 
submission because most MUAs display little or nothing of the message. 
Instead, we accept all authenticated submissions, and generate a DSN if 
there's something we don't like (except unauthorised sender addresses, of 
course).

DSNs are often hard to understand, especially when the sending MTA is 
trying to wrap an error message. All too often, they discard the error 
message and insert completely misleading diagnostics.

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 2 Jul 2009 11:05:42 -0400

Ian Eiloart wrote:
> by the MTA that has more information.

You seem to be missing the distinction here.

Users (and most automated processes that defer to a MTA to initiate 
sending) _only_ see DSNs [+].

The question is _who_ constructs the DSN?

If it's the MX (or later) of the recipient, (IOW: _recipient_ generated 
DSNs) you get backscatter.  This is bad.  It's so bad that it's 
generally better to blackhole (possibly providing quarantine on the 
recipient side to find misfires) once the perimeter has accepted the 
email than to DSN later.

If its the sending MTA (the one that does the MX lookup and connect and 
is reacting to an inline-reject), this is good.  Good MTAs can generate 
quite readable messages if they care to.  Even if only to generate a 
reasonably well-formatted message that explains what the MX MTA actually 
said in its rejection.

I've seen a number of MTAs that do a good job (eg, postfix I think) on 
formatting DSNs, and others that do an abysmal job (eg, gag, Exchange).

> message and insert completely misleading diagnostics.

Eg: Exchange.

The work, if any, is to to encourage that sending MTAs generate 
reasonably readable DSNs from a reject and NOT to discard the text of 
the rejection.  Encouraging DSNs once you're past the MX's acceptance is 
a very bad idea.

[+] The only time a user will directly see an SMTP reject is during the 
SMTP SUBMIT phase with their own MTA.  The only time an application will 
see a reject "properly" is if it's the application's own SMTP client.

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 02 Jul 2009 17:23:44 +0200

Ian Eiloart wrote:
> DSNs have the potential to be more useful to humans if they're 
> constructed by the MTA that has more information.

If that happens after alias expansion, it may disclose sensible 
information. Some may regard that as being *too* useful. :-/

---

From: John Levine <[email-masked]>
Date: 2 Jul 2009 15:51:16 -0000

>constructed by the MTA that has more information. ...

Well, OK.  DSNs have been around for 13 years now.  How often do you see
one with more useful info that you'd find in a reject notice?  And if
you expect them to be better in the future, what's going to change in
the next 13 years to make that happen?

R's,
John

---

From: Bill Cole <[email-masked]>
Date: Thu, 02 Jul 2009 12:07:15 -0400

> specific subdomains like @uk.ibm.com.
[...]
> Exercise for the reader: why aren't spammers using the @ibm.com domain?

You provided the answer before the question.

Forged sender addresses are predominantly harvested rather than purely 
invented or recombinantly assembled. Forged sender spam is mostly the 
product of the blatantly criminal segment of spammers whose target lists are 
largely harvested from the web, Usenet, and the address books of compromised 
systems. In a world where there is a detectable fraction of sites making 
some effort to validate senders to the point of SMTP callbacks, the most 
economical approach for spammers forging the sender address is to just pull 
sender addresses from the same list as targets.

I see this most clearly in blowback like the bounce AOL sent me this 
morning. The original spam had been addressed to [email-masked]' with the 
sender [email-masked]'. That's an address I've used in very public ways 
for 15 years, making it a frequent spam target. 99%+ of the direct spam for 
it I never see, particularly the flavors using forged senders, but nearly 
all of the blowback I get for it is from spam aimed at alphabetically nearby 
targets.

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 2 Jul 2009 12:27:57 -0400

> You provided the answer before the question.

Somewhat.  Because spammers _are_ using @ibm.com too.  I got samples ;-)

Anybody saying "spammers don't do X" and "spammers do X" are wrong at 
least some of the time.  Except for the obvious tautology that "spammers 
spam".

> invented or recombinantly assembled.

IOW: the biggest asset spammers have is lists of potential spam victim's 
email addresses.

What better place to get the email addresses to forge as sender than 
from the exact same list?  Is it so hard to imagine that a bot might do 
this or some variation?

1) Read a bunch of addresses
2) Spam the bunch of addresses, forged with one of the bunch as sender
3) Goto step 1

Various corollaries:

- If you get spam, you're probably being forged as sender in other spam.

- If they're hitting valid addresses, then there will be blowback _to_ 
valid addresses.

---

From: Ian Eiloart <[email-masked]>
Date: Thu, 02 Jul 2009 17:44:03 +0100

> Somewhat.  Because spammers _are_ using @ibm.com too.  I got samples ;-)

Ok, but it's trivial to reject them after checking SPF.

> What better place to get the email addresses to forge as sender than from
> the exact same list?  Is it so hard to imagine that a bot might do this
> or some variation?
>

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 2 Jul 2009 13:03:12 -0400

> Ok, but it's trivial to reject them after checking SPF.

Don't need to.  They're all being rejected by either "no such user" or 
the spam filter rejects.

SPF isn't worth the cycles nor bandwidth (in this environment at least) 
to catch the rare SPF -all.

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 2 Jul 2009 13:20:11 -0400

> to catch the rare SPF -all.

I should add - _if_ spammers are using the "-all" to screen out bad 
senders to use, then the mere existance of SPF as a "standard" has some 
value to push spammers away from forging certain high-value-target 
domains literally and thus marginally reduce backscatter because of 
spammer-behaviour-modification.  Perhaps.

But it doesn't imply that implementing any SPF checking will make any 
noticeable difference.  Indeed, the only concrete numbers I've ever seen 
about SPF adoption were percentages of domains publishing SPF records 
due to noises being made by MSN/Hotmail, _not_ checking SPF.

Nobody has a handle on how many have actually implemented SPF checking.

The only stats I've seen about backscatter volume pre/post SPF 
publication don't show any compelling reason to believe SPF made any 
difference.  There's no particular reason to believe that it's going to 
get any better either.

We publish, but do not check simply because of the noises that 
MSN/Hotmail were making.  Publishing (and erroneous checking) has 
probably caused more problems (elsewhere) than it's solved.

---

From: Ian Eiloart <[email-masked]>
Date: Fri, 03 Jul 2009 10:22:21 +0100

> to catch the rare SPF -all.

<http://spf-all.com/>

There are a few prime domains that publish -all records.