Subject: [ietf-smtp] SPF DNS query limits

From: Paul Smith <[email-masked]>
Date: Mon, 23 May 2016 15:52:39 +0100

In RFC 7208 section 4.6.4 we're told that the SPF implementation should 
limit the number of include, a, mx, ptr and exists to 10, and if that 
limit is exceeded then we MUST return a 'permerror' result.

1) is this limit of 10 for the whole evaluation process, or does it 
restart for each 'include'/'redirect'? I've presumed it's for the whole 
process otherwise the total number of DNS queries is effectively 
unlimited if you have many nested includes, and the purpose of that 
section seems to be to limit them.

2) Is this limit generally being stuck to by SPF evaluation functions?

The reason I ask is that our SPF implementation is regularly hitting 
this limit (eg one or two per minute on a low volume server).

For instance, we got a spam message claiming to come from cncdost.com 
(chosen at random from our incoming messages). That results in:

- cncdost.com - contains "a mx ptr include:secureserver.net" - so that's 
4 already
- secureserver.net - contains "include:spf-ss1.domaincontrol.com" - so 
that's 5
- spf-ss1.domaincontrol.com - contains 
"include:spf-ss2.domaincontrol.com include:spf.messaging.microsoft.com" 
- so that's 7
- spf-ss2.domaincontrol.com - contains 
"include:spf-ss3.domaincontrol.com" - that's 8
- spf-ss3.domaincontrol.com - contains "a:..." - that's 9
- spf.messaging.microsoft.com - contains 
"include:spf.protection.outlook.com" - that's 10
- spf.protection.outlook.com - contains 
"include:spfa.protection.outlook.com" - that's 11

(there are more includes after that, but our evaluator has given up by 
now. In fact, the source IP address was not listed on any of the SPF 
records, so would have resulted in a FAIL if it had got that far).

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 23 May 2016 11:37:05 -0400

> section seems to be to limit them.

It is for the whole evaluation process.

> 2) Is this limit generally being stuck to by SPF evaluation functions?

This limit goes back to the experimental RFC 4408, so it's a decade old.  All 
of the open source implementations of which I'm aware follow it (although some 
provide configuration options to ignore it).  I don't know the status of any 
proprietary implementations.

> records, so would have resulted in a FAIL if it had got that far).

This is not atypical of records I've seen that are not well considered and 
looking at the guidance supplied by godaddy.com [1] not terribly surprising.  
Their guidance is not unusually bad.  I've yet to see any kind of a record 
generation wizard that can substitute for actually knowing what you're doing 
(at least a bit).

Scott K

[1] https://www.godaddy.com/help/add-an-spf-record-19218

---

From: "MH Michael Hammer (5304)" <[email-masked]>
Date: Mon, 23 May 2016 15:48:41 +0000

I'm going to jump in and top post because I only have one comment to add. 

ESPs (Email Service Providers) seem to be especially bad with lots of includes and lots of individual records inside the includes.

Mike

> -----Original Message-----
> From: ietf-smtp [mailto:[email-masked]] On Behalf Of Scott
> Kitterman
> Sent: Monday, May 23, 2016 11:37 AM
> To: [email-masked]
> Subject: [!!Mass Mail]Re: [ietf-smtp] SPF DNS query limits
> 
> > In RFC 7208 section 4.6.4 we're told that the SPF implementation
> > should limit the number of include, a, mx, ptr and exists to 10, and
> > if that limit is exceeded then we MUST return a 'permerror' result.
> >
> > restart for each 'include'/'redirect'? I've presumed it's for the
> > whole process otherwise the total number of DNS queries is effectively
> This limit goes back to the experimental RFC 4408, so it's a decade old.  All of
> the open source implementations of which I'm aware follow it (although
> some provide configuration options to ignore it).  I don't know the status of
> any proprietary implementations.
> 
> >
> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp

---

From: Steve Atkins <[email-masked]>
Date: Mon, 23 May 2016 09:44:42 -0700

> On May 23, 2016, at 7:52 AM, Paul Smith <[email-masked]> wrote:
> 
> In RFC 7208 section 4.6.4 we're told that the SPF implementation should limit the number of include, a, mx, ptr and exists to 10, and if that limit is exceeded then we MUST return a 'permerror' result.
> 
> 1) is this limit of 10 for the whole evaluation process, or does it restart for each 'include'/'redirect'? I've presumed it's for the whole process otherwise the total number of DNS queries is effectively unlimited if you have many nested includes, and the purpose of that section seems to be to limit them.

It's for the whole process, excluding A queries triggered by mx: entries, IIRC. (Pretty much, read the RFC if you want the gory details

> 2) Is this limit generally being stuck to by SPF evaluation functions?

It's hard to say overall, as SPF is almost never used on it's own to make delivery decisions.

http://tools.wordtothewise.com/spf/check/ebay.com suggests that it's not something that's causing spectacularly bad delivery problems or ebay would have fixed it.

It's exacerbated by many DNS records being less than half full - either because people (or "wizards") generating them don't know that you can have more than 250 characters in a record, or because they're using DNS management web interfaces that don't allow them to do so. That leads to more include records needed and so more queries. Running some commonly used included records through the (fairly crude) minimizer at http://tools.wordtothewise.com/spf/minimize suggests it's bloating by a factor of about two, on average.

And that's not the only common SPF mistake made that bloats records.

> 
> The reason I ask is that our SPF implementation is regularly hitting this limit (eg one or two per minute on a low volume server).
> 
> For instance, we got a spam message claiming to come from cncdost.com (chosen at random from our incoming messages). That results in:
> 
> - cncdost.com - contains "a mx ptr include:secureserver.net" - so that's 4 already
> - secureserver.net - contains "include:spf-ss1.domaincontrol.com" - so that's 5
> - spf-ss1.domaincontrol.com - contains "include:spf-ss2.domaincontrol.com include:spf.messaging.microsoft.com" - so that's 7
> - spf-ss2.domaincontrol.com - contains "include:spf-ss3.domaincontrol.com" - that's 8
> - spf.messaging.microsoft.com - contains "include:spf.protection.outlook.com" - that's 10
> - spf.protection.outlook.com - contains "include:spfa.protection.outlook.com" - that's 11
> 
> (there are more includes after that, but our evaluator has given up by now. In fact, the source IP address was not listed on any of the SPF records, so would have resulted in a FAIL if it had got that far).

The SPF spec says immediate FAIL with too many DNS queries, regardless of whether you found a match. I appreciate the intent, but given the number of includes that are required if you're outsourcing multiple services (and need the return path to point back at you) the limit of ten queries is a bit of a problem.

Given SPF only affects the return path, it's not actually required to include an outsourced providers SPF includes if the bounces go back to them (which they usually do). At least until you get into wanting custom return paths because you want DMARC-esque alignment, so as to get the delivery benefits of that...

Cheers,
  Steve

---

From: Paul Smith <[email-masked]>
Date: Tue, 24 May 2016 09:32:00 +0100

On 23/05/2016 17:44, Steve Atkins wrote:
> It's for the whole process, excluding A queries triggered by mx: entries, IIRC. (Pretty much, read the RFC if you want the gory details)
I have, but it's not 100% explicit that it's for the whole process. 
That's what I'd assumed, but then when we were getting so many failures 
I thought maybe I'd interpreted it incorrectly ;-)

> It's hard to say overall, as SPF is almost never used on it's own to make delivery decisions.
That seems to be changing. I've recently had several customers contact 
me because their outgoing mail was failing - turned out that it was 
because they had two SPF records for their domain, which is invalid. It 
had been working fine until a week or so ago. Once they fixed that, 
their mail was accepted again. (Various target domains, not sure if they 
used the same service provider)

> http://tools.wordtothewise.com/spf/check/ebay.com suggests that it's not something that's causing spectacularly bad delivery problems or ebay would have fixed it.
Or people have whitelisted ebay because they're a big company...

> It's exacerbated by many DNS records being less than half full - either because people (or "wizards") generating them don't know that you can have more than 250 characters in a record, or because t…

BTW - do these companies just add all the IP addresses they own to their 
SPF records? Do Google really have ~230,000 mail servers?

Just as a matter of interest - wouldn't it be worth them using macros in 
their includes? eg 'include:%{i4r}.spf.ebay.com'. This could cut down 
drastically on the level of nesting.

---

From: "John Levine" <[email-masked]>
Date: 24 May 2016 16:23:12 -0000

>SPF records? Do Google really have ~230,000 mail servers?

No, but they certainly have thousands, way too many to enumerate in SPF records.

>drastically on the level of nesting.

Yuck.  That feels really fragile.

R's,
John

---

From: Steve Atkins <[email-masked]>
Date: Tue, 24 May 2016 09:42:06 -0700

> On May 24, 2016, at 9:23 AM, John Levine <[email-masked]> wrote:
> 

There are (at least two?) companies trying to monetize SPF macros. It'll
be interesting to see what happens.

Cheers,
  Steve

---

From: "Carl S. Gutekunst" <[email-masked]>
Date: Tue, 24 May 2016 09:59:09 -0700

On 05/24/2016 01:32 AM, Paul Smith wrote:
> Just as a matter of interest - wouldn't it be worth them using macros 
> in their includes? eg 'include:%{i4r}.spf.ebay.com'. This could cut 
> down drastically on the level of nesting.

In my experience, SPF macros are not well/reliably supported across 
proprietary implementations. That dates back to the earliest days of 
SPF, when large scale Email consumers (notably AOL, but also Postini) 
were concerned about the costs of doing SPF queries in real time, and 
instead built dedicated SPF caches that were refreshed as background 
tasks. These did no DNS queries on message receipt, making it impossible 
to support most macros. AOL clearly documented this at the time, IIRC.

(Steve Atkins would have more visibility on this than I do.)

<csg>

---

From: "Carl S. Gutekunst" <[email-masked]>
Date: Tue, 24 May 2016 10:04:12 -0700

> proprietary implementations.

The four proprietary implementations with which I'm familiar all 
implement the 10-level limit. Two of those also explicitly checked for 
loops. It's in the receiver's own self interest to do so.

<csg>

---

From: Brandon Long <[email-masked]>
Date: Tue, 24 May 2016 14:38:26 -0700

On Tue, May 24, 2016 at 10:04 AM, Carl S. Gutekunst <[email-masked]> wrote:

> The four proprietary implementations with which I'm familiar all implement
> the 10-level limit. Two of those also explicitly checked for loops. It's in
> the receiver's own self interest to do so.

Gmail's implementation has a bunch of limits in it which may not be in the
spec, though they tend to be fairly generous.  Our DNS lookup limit is
definitely more generous than 10.

For us, we made the decision that enough mail had records requiring more
lookups that it was useful to go deeper.  And it doesn't help that our
current _spf.google.com record is already requiring 4 lookups.  And no, we
don't have >200k servers, we're also looking to reduce our SPF record to a
more reasonable set.

Also, we already cache DNS lookups on most of the internet, so almost all
of the increased usage is already cached.

We also (more recently) have a loop detector, and refuse to descend into
loops.

Brandon

---

From: Paul Smith <[email-masked]>
Date: Wed, 25 May 2016 09:33:22 +0100

> Yuck.  That feels really fragile.
Why?

Is it just that SPF macros aren't well implemented (despite them being a 
key part of SPF), or something else?

It seems to me that this is exactly the reason that macros are 
specifically allowed in include: mechanisms in RFC 7208.

A substantial proportion of the SPF checks our server does which include 
the Google/Outlook.com/etc SPF records go over the 10 query limit, so 
if/when people start failing on permerror results, it'll cause big issues.

---

From: Paul Smith <[email-masked]>
Date: Wed, 25 May 2016 09:38:17 +0100

> proprietary implementations.
Wow, really? That's like saying 'we support IMAP4, but not the SEARCH 
command' and not expecting any problems.

Interesting, (since Mandrill's Ts&Cs changed) we've started using 
SparkPost for transactional email. Their SPF record is (a) "bad" because 
it uses ptr which is now deprecated, and (b) uses macros (albeit in 
exists, not include):

v=spf1 exists:%{i}._spf.sparkpostmail.com ptr:sparkpostmail.com 
ptr:spmta.com ~all

---

From: "John R Levine" <[email-masked]>
Date: 25 May 2016 08:05:16 -0400

>>> 'include:%{i4r}.spf.ebay.com'. T
> Is it just that SPF macros aren't well implemented (despite them being a key 
> part of SPF), or something else?

Partly it's poor SPF implementations, partly that it's a much more complex 
set of records to publish.  Your example as written requires a record for 
every outbound IPv4 mail server, and probably still puts all of the IPv6 
servers into one record or for a really large provider one record per RIR 
from whom they get space.  Neither strike me as something that's likely to 
be maintained reliably.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.