Subject: [spfbis] Proposed spf TXT record change

From: "Roy A. Gilmore" <[email-masked]>
Date: Tue, 9 Feb 2016 15:33:47 -0800

I'm not sure how to go about this, but, I'll start here and maybe
somebody could point me in the right direction. I think placing the spf
information in a TXT record directly attached to the domain is a
mistake. I think the spf information should be placed in a TXT record
attached to a _spf selector (e.g. _spf.example.com). This behavior
already has a history of being used by other services (i.e.
_kerberos.example.com, _dmarc.example.com, etc.), and would make
retrieving the spf information much easier and much more robust. This
would also be a trivial change to implement. Any thoughts?

---

From: Mark Andrews <[email-masked]>
Date: Wed, 10 Feb 2016 11:36:05 +1100

> would also be a trivial change to implement. Any thoughts?

This group won't allow the double query and will introduce spurious
interoperablity arguments.  See the history of the SPF record type
which if the group hadn't abandoned would be well on the way to
being done by now as they abandoned the process just as the libraries
using the new code point were being deployed.

> _______________________________________________
> spfbis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spfbis
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: "Stuart D. Gathman" <[email-masked]>
Date: Tue, 9 Feb 2016 20:17:26 -0500 (EST)

> using the new code point were being deployed.

The big problem with the SPF record type was certain common DNS servers
that would completely ignore (read TIMEOUT) queries for an RR type they
didn't know about (instead of returning NXDOMAIN).

The _spf subdomain would not have that problem, and so would have some
legs for another try.  But I think having been burned by the SPF record
type, no one is anxious to try.

---

From: "John Levine" <[email-masked]>
Date: 10 Feb 2016 02:25:25 -0000

> I think the spf information should be placed in a TXT record
>attached to a _spf selector (e.g. _spf.example.com).

You're right, but you're also a decade too late.  Forget it.

R's,
John

---

From: "Roy A. Gilmore" <[email-masked]>
Date: Tue, 9 Feb 2016 19:24:33 -0800

I hope I don't come across as argumentative, that is not my intention. I
don't have any idea how these workgroups work. I'm curious about a
behavior that I THINK (I could be wrong) is incorrect with spf, and
trying to learn, so I'm asking what I think are good questions.

Double query? What if a single domain query returns multiple TXT
records. How long will it take to process multiple free form (i.e. RFC
1035: "The semantics of the text depends on the domain where it is
found.") TXT records to be sure you select the right one? Can you be
sure you selected the right one? If you query for a _spf TXT record, you
know you got the right one (well, I guess in the degenerate case where
DNS has been misconfigured, you could get multiple TXT records). I
suppose someone could say that only one TXT record should be returned.
RFC 1035 doesn't say whether TXT is a singleton resource record or not.
But, RFC 1035 has been updated by so many RFC's and BCP's that I'm not
sure (I'll have to look into that). I know bind allows multiple TXT
records per key (I'm not sure if "key" is the correct terminology here).
Anyway, assume for a minute that only one TXT record should to be
returned, should spfbis hijack that TXT record? I don't think so. What
about all the other services that query for a service specific TXT
record, are their workgroups "wrong", and the spfbis workgroup "right"?
I'd willing to bet the other workgroups have some pretty smart people
too (probably some of the same smart people).

The double query argument was used with SRV records (RFC 2782) not
allowing CNAMEs for targets, which effectively made the whole point of
CNAMEs useless. If I have a CNAME record pointing to a LDAP server, and
I replace the LDAP server, instead of just changing the CNAME record, I
have to change all the SRV records too, so why even bother with a CNAME
in the first place (rhetorical, don't bother answering)? I suppose it
doesn't really matter, does anybody actually use SRV records on the
internet (rhetorical again, bordering on sarcasm)? SRV records seem to
be used mainly on intranets, particularly ones with Microsoft Active
Directory Domains where a lot of the SRV record configuration happens
"under the hood" with little human input.

As to spurious interoperablity arguments, that's easy, set a transition
period where either record is acceptable with a preference for the _spf
TXT record (i.e. query for the _spf TXT record first, and fall back to
the domain TXT record if the _spf TXT record doesn't exist). Maybe, even
a "double" transition period where in the first transition period the
domain TXT record was the preferred record. POP3 didn't replace POP2
overnight. And IMAP4 still hasn't replaced POP3 even though it has many
advantages.

Sorry about being so long winded. I'll try to be more concise in the future.

---

From: "John Levine" <[email-masked]>
Date: 10 Feb 2016 03:31:04 -0000

>trying to learn, so I'm asking what I think are good questions.

Old joke:

Q. How could God create the world in six days?
A. He didn't have an installed base.

SPF works well enough that there is no incentive for anyone to change
the existing running code that is in tens of thousands of mail systems
all over the world.  If we were doing SPF from scratch, there's all
sorts of stuff we'd do differently, but like I said, it's a decade too
late.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 10 Feb 2016 03:35:13 -0000

PS:

>period ...

That was the plan with the type 99 SPF record.  See RFC 6686 to see
how well that worked.

There's huge pressure for people to move from IPv4 to IPv6 since the
price of IPv4 addresses has in the past few years increased from
roughly zero to about $10 per address while IPv6 is still nearly free
and always will be.  But it'll still be decades before IPv4 goes awy.

R's,
John

---

From: "Roy A. Gilmore" <[email-masked]>
Date: Tue, 9 Feb 2016 19:51:52 -0800

Why should it matter if I'm a decade too late? If I'm right (and I'm not
saying that I am), why shouldn't the behavior be changed. RFC's aren't
set in stone, they are updated and/or obsoleted all the time. This
proposed change is trivial to implement.

---

From: "Roy A. Gilmore" <[email-masked]>
Date: Tue, 9 Feb 2016 20:12:49 -0800

While I'm aware that there are "tens of thousands of mail systems all
over the world", there are only a few SMTP implementations that are run
on all those systems. You only have to get the SMTP developers on board.
This is a trivial change (maybe not a one-liner, but, close), would
simplify there code and make it more robust. This change could easily
rolled out as a bug fix. Bug fixes are released all the time, and SHOULD
be applied ASAP. If a server administrator is still using the same
version of his/her SMTP implementation next year that he/she is using
this year, he/she is not doing his/her job and should be fired.

---

From: Dotzero <[email-masked]>
Date: Tue, 9 Feb 2016 23:14:53 -0500

> proposed change is trivial to implement.
The fact that you believe this to be a trivial change to implement makes
clear that you don't understand what is actually involved. This horse has
been beaten so much it is only fit for the glue factory. You are absolutely
right that _spf would be a better approach... if there was not already a
large installed base to consider. SPF works (for some definition of works).
The change you are proposing sets up interoperability issues that we have
dealt with before.

---

From: Mark Andrews <[email-masked]>
Date: Wed, 10 Feb 2016 15:17:30 +1100

> proposed change is trivial to implement.

As is the change to use type99 which works with wildcard records.

_spf doesn't work with wildcard records.

Mark

---

From: Dave Crocker <[email-masked]>
Date: Tue, 9 Feb 2016 20:31:08 -0800

> You're right, but you're also a decade too late.  Forget it.

+1

Yes, it should have been done that way.  No it does not seem possible to 
get the installed base to move to a different model.

d/

---

From: Mark Andrews <[email-masked]>
Date: Wed, 10 Feb 2016 16:09:40 +1100

> > You're right, but you're also a decade too late.  Forget it.

-100000 _spf was never right.
 
> get the installed base to move to a different model.

No, you just needed a longer time frame to transition than this
group was willing to accept for a organic transition or to actually
publish a timeframe.  A simple instruction to nameserver developers
to "automatically publish a SPF record if a TXT spf records exist"
would have push the transition along enourmously by getting a large
amount of SPF records out there.  Actively sending out messages to
those only publishing TXT spf records would have also helped.

No one said "we want this transition done in N years" so it didn't
happen in that amount of time and there there where complaints that
the transition failed when the transition didn't happen overnight.

Expectations where not set so no one could meet them.

Mark

---

From: "John Levine" <[email-masked]>
Date: 10 Feb 2016 15:05:59 -0000

>on all those systems. You only have to get the SMTP developers on board.

Well, OK.  Why don't you start contacting them, encourange them to
change their SPF implementations with a suitable transition period,
and let us know when we can throw the switch.

One immutable rule of the standards and open source world is that
telling other people to do work that you're not willing to do yourself
is, ah, ineffective at best.

R's,
John

---

From: William Leibzon <[email-masked]>
Date: Wed, 10 Feb 2016 10:12:59 -0800

> is, ah, ineffective at best.

That's a double-edged sword - expecting implimentors to change first while
they are expecting a fully accepted standard that will tell them what they
need to do.

But yes this ship has sailed and I've been through _spf and SPF RR
discussions several times. _spf has a problem because of no wildcard
support and that's why it was discarded as an option after considerable
dicussion on spf-discuss mail list 10 years ago. I was from the start and
still am in favor of SPF RR. And DNS providers and vendors of email
software were given enough time (5+ years) to try to do "the rght thing"
and switch and it did not happen in big enough numbers. Go ahead and find
who to blame if you like.

And if you want to blame IETF, sure its your right. But it has no enforcing
power what so ever. It just gives out tech documents (with minimum
potential for conflict it can) and hope people find them useful. Like with
open-source software, some standards and documents survive and become very
popular and many don't survive even through they were or are a good idea
and well intended for benefits of everyone (or at least some).

William

---

From: "Stuart D. Gathman" <[email-masked]>
Date: Wed, 10 Feb 2016 14:25:42 -0500 (EST)

> This is a trivial change (maybe not a one-liner, but, close), would

Changing the SPD implementation is trivial.  I've done it, a long
time ago.  I voted to keep type99 for the production RFC.  Actually
working efficiently with braindead DNS servers is *not* trivial.
I've tried to do it, and gave up.  I still have TYPE99 code
in my SPF implementation, and I periodically turn it on to see
if a certain huge DNS server provider is still screwing up . . .

---

From: "Stuart D. Gathman" <[email-masked]>
Date: Wed, 10 Feb 2016 14:32:50 -0500 (EST)

> proposed change is trivial to implement.

As a pyspf developer, the _spf idea is *not* trivial to implement. 
The SPF RR is trivial to implement.  The problem was braindead DNS
servers.

I still publish both SPF and TXT records (no SPF police are stopping
me).  I still have the type99 code in my SPF implementation.
Type99 is only deprecated.  If you can convince DNS server providers
to operate correctly for unknown RR types (or failing that, to
recognize SPF RR), and convince lots of people to publish both,
then a switch could be reconsidered.  Especially if there is a new
backward incompatible SPF version.

---

From: Mark Andrews <[email-masked]>
Date: Thu, 11 Feb 2016 07:43:31 +1100

> The SPF RR is trivial to implement.  The problem was braindead DNS

No the problem is brain dead firewalls.  Apart from some load
balancers that get anything other than A (and now sometimes AAAA)
right, nameservers return NODATA or at worst NOTIMP when they don't
implement a type.  For many of the loadbalancer it is bad configuration
not that the load balancer is incapable of returning a correct
answer so you need to nag the operators of the load balancers to
correctly configure them.

The vast, vast, vast majority of nameservers have always done the
correct thing. RFC 1034 actually tells developers to return NODATA.

Nameservers put in SPF support in a timely manner.  It's still
there because there is no point in removing it.
 
> backward incompatible SPF version.

The biggest problem we had was not setting expectations.  I didn't
expect the transition to complete in 5 years (look at how long it
took before you could rely on just publishing a MX record for email)
yet we have had people saying 5 years was more than enough time in
this thread.  If you wanted a 5 year transition it required active
pushing of every to achieve that and I saw none of that.  It takes
about that long for new nameserver code to make it into OS's as the
OS vendors don't pickup the code in a timely manner.

I also expect everyone in this group to support the publication of
https://tools.ietf.org/html/draft-ietf-dnsop-no-response-issue-01
when it comes up as it is trying to get the remaining bad actors
weeded out of the ecosystem by actively looking for them and informing
the operators that they have a problem.  You have all seen the
issues cause by non RFC compliant code being deployed which is what
these tests are looking for.

It isn't impossible to test for whether servers answer correctly
for various types including unknown types.  See
https://ednscomp.isc.org/compliance/tld-typereport.txt
Note: there are false positives with timeouts as rate limiters kick
in.

Mark

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 10 Feb 2016 20:49:21 -0800

> using the new code point were being deployed.

I am mystified that the bitterness on this topic remains impervious to
things like evidence.

The history of the SPF record type is documented in RFC6686.   I'm happy to
let that document and the data it contains speak for itself.

I also contend that the record shows that the working group and the
community made an informed decision.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 10 Feb 2016 21:02:26 -0800

> this year, he/she is not doing his/her job and should be fired.

In an ideal world, you'd be right.  But in reality, a lot of things don't
get updated until there's pain happening and the only solution is to
update.  After trying only a few domains off the top of my head, I found a
university mail server running an installation of sendmail dating back to
2006, 11 releases back from latest.

If SPF is working fine as-is, there's little impetus for operators to
change it.

You're right that software and RFCs can be modified, but is it worth the
effort to do either?

What's the pain point you're trying to solve that makes this worth doing?

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 10 Feb 2016 21:26:43 -0800

On Tue, Feb 9, 2016 at 7:24 PM, Roy A. Gilmore <[email-masked]>
wrote:

> records.

Section 4.5 of RFC7208 covers that.

> How long will it take to process multiple free form (i.e. RFC
> found.") TXT records to be sure you select the right one?

I don't think processing time is the concern here.

> Can you be
> sure you selected the right one?

Yes, because after discarding things that aren't SPF records, you evaluate
what's left: if there's none, you get a "none" result; if one, you get the
real result; if more, you get an error.

> If you query for a _spf TXT record, you
> DNS has been misconfigured, you could get multiple TXT records).

I don't know that wildcard TXT records constitute a "degenerate" case.

> I

The rules for handling the DNS reply in RFC7208 cover all of these
possibilities, I believe.  They have to, given how much random stuff one
finds in the domain-level TXT reply sometimes.  Query for the TXT of "ut.edu",
for example.  And that's a bunch cleaner than what I found when I did the
RFC6686 surveys.  I found some that had valid SPF records mixed in with
junk like that.

SPF isn't making a claim to the domain level TXT record.  That's just where
it landed (again, see RFC6686 for the history), and it can easily coexist
with other things.

The spfbis archive, going back to early 2013 at least, documents the
extensive discussions we had on how we got here and why we should never do
it again, but fixing it now is simply not pragmatic.

> overnight. And IMAP4 still hasn't replaced POP3 even though it has many

Sure, transition periods could work if everyone is on board with making the
change.  Selling that is the challenge.

What problem are you really trying to solve here?

-MSK

---

From: Mark Andrews <[email-masked]>
Date: Thu, 11 Feb 2016 17:57:29 +1100

In message <[email-masked]>, "Murray S. Kucherawy" writes:
> --001a11437e362be933052b7746bc
> Content-Type: text/plain; charset=UTF-8
> 
 
Basically because the experiment's terms of reference appeared to
change.  I doubt anyone in the DNS community thought it was going
to be "can we migrate a established base to a new code point in a
couple of years" when the experiment was "a examination of whether
Sender ID or SPF was the best model".  The code point being used
for SPF was irrelevent to the examination of Sender ID vs SPF.

The DNS community was worried about getting to the right point for
the DNS in the end if SPF won the experiment.  If we thought we
were in a experiment to move the code point in a couple of years
lots of things would have been done differently.

The evidence actually showed the transition was on track.  Nameservers
support SPF were deployed.  Libraries supporting SPF as well as TXT
were being deployed.  None of this is captured in RFC6686.  But hey
lets look at records that have been entered at the begin of what
was expected to be a long transition process and declare defeat.

Mark

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 11 Feb 2016 06:03:51 -0800

On Wed, Feb 10, 2016 at 10:57 PM, Mark Andrews <[email-masked]> wrote:

> The evidence actually showed the transition was on track.

After six years, it didn't appear to the working group that a transition of
any kind was actually happening.

> Nameservers support SPF were deployed.

I thought it was pretty clear that this wasn't the problem.  The major
obstacles were poor provisioning systems and faulty firewalls (as you
pointed out, and as the experiments we did suggested).  The problem is that
they are widespread, and that's unlikely to change.

Libraries supporting SPF as well as TXT were being deployed.

We specifically looked for this (especially the "being deployed" part) when
preparing RFC6686, and found no evidence of it.  Exactly one source of type
99 queries was identified.  So, although there existed software support,
nobody was using it.  We asked around, and nobody was planning to use it,
either; many operators didn't even know what we were talking about.

> None of this is captured in RFC6686.

Because it wasn't supported by evidence.  If we had seen data to the
contrary, we'd have written a different report.

-MSK

---

From: Dotzero <[email-masked]>
Date: Thu, 11 Feb 2016 09:30:25 -0500

On Thu, Feb 11, 2016 at 9:03 AM, Murray S. Kucherawy <[email-masked]>
wrote:

> After six years, it didn't appear to the working group that a transition
> of any kind was actually happening.
>

To reinforce Murray's comment, there was quite a bit of discussion in the
working group before the consensus to drop support for type 99. And there
was incredibly little implementation in the wild after ~6 years.

> We specifically looked for this (especially the "being deployed" part)
> when preparing RFC6686, and found no evidence of it.  Exactly one source of
> type 99 queries was identified.  So, although there existed software
> support, nobody was using it.  We asked around, and nobody was planning to
> use it, either; many operators didn't even know what we were talking about.
>
>

It was captured in the working group discussions and the data that was
brought back to the working group clearly showed non significant
implementation in the wild.

Mike

---

From: "John Levine" <[email-masked]>
Date: 11 Feb 2016 15:56:40 -0000

>let that document and the data it contains speak for itself.

There is a belief in parts of the DNS community that adding new RRTYPEs
is trivially easy.  Provisioning system problems don't matter either
because they don't exist, or because any zone file of any importance
is created with vi.

A decade ago when 4408 was published, the belief was pervasive enough
that its adherents demanded that the SPF record be added at the last
minute or the draft wouldn't be published, and it was added so quickly
that the language was botched and describes implementations that don't
interoperate.

Since all existing SPF implementations already used TXT records, and
type 99 SPF records involved considerable pain and no operational
benefit (I know about having the server automatically add the SPF
records, my provisioning system did it), it's not surprising that
nobody used them.

And since that experience showed a treasured belief to be wrong, well,
the reactions are not surprising.

R's,
John

---

From: Hector Santos <[email-masked]>
Date: Thu, 11 Feb 2016 16:50:15 -0500

> -100000 _spf was never right.

+100000.

Its all wasted wasted overhead especially when relaxed policies (with 
no hard rejection capabilities).    If you are going to do this, 
please make it for hard policies.

-- 
HLS

---

From: Scott Kitterman <[email-masked]>
Date: Thu, 11 Feb 2016 18:42:48 -0500

>backward incompatible SPF version.

I think a backward incompatible update is exactly the right time to reintroduce use of the SPF RR.  I think I can even describe much of what an update should contain:

 - Combine a:, ip4:, and ip6: into a single mechanism (in retrospect there was no need to make them separate and combining would simplify things)

 - Drop the PTR mechanism entirely

 - Drop (or possibly radically simplify macros)

I personally doubt it would get a lot of deployment, but that kind of incompatible update is what the SPF RR type should be used for.

In any case, I think the new RR type is a better idea than _spf.

Scott K

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 12 Feb 2016 11:16:47 +0100

On Fri 12/Feb/2016 00:42:48 +0100 Scott Kitterman wrote:
>>> Why should it matter if I'm a decade too late? If I'm right (and I'm 
>>> not saying that I am), why shouldn't the behavior be changed.
>>
>> As a pyspf developer, the _spf idea is *not* trivial to implement. The SPF
>> RR is trivial to implement.
>>
>> If you can convince DNS server providers to operate correctly for unknown
>> RR types (or failing that, to recognize SPF RR), and convince lots of
>> people to publish both, then a switch could be reconsidered.  Especially
>> if there is a new backward incompatible SPF version.
> 
> I think a backward incompatible update is exactly the right time to
> reintroduce use of the SPF RR.  I think I can even describe much of what an
> update should contain:
> 
> - Combine a:, ip4:, and ip6: into a single mechanism (in retrospect there
>   was no need to make them separate and combining would simplify things)
> 

Those considerations don't have to be simultaneously valid for both the binary
on-the-wire format and the format used for declarations.  Records could be
compiled from high level directives.

> I personally doubt it would get a lot of deployment, but that kind of
> incompatible update is what the SPF RR type should be used for.

Whenever such an incompatible update will be needed, that is.  Paraphrasing
Roy, why should it matter if it will not arise within a decade?

> In any case, I think the new RR type is a better idea than _spf.

One doesn't preclude another.  Possibly related to _spf, looking up helo
records by administrative domain --à la DMARC-- would greatly simplify deployment.

Ale