Subject: The problem with Unified SPF

From: Roy Badami <[email-masked]>
Date: Wed, 30 Jun 2004 21:21:47 +0100

OK, here's what I see as the probems with Unified SPF (as I understand
the proposal) as compared with using SenderID in conjunction with
CSV/CSA.

1. Unified SPF encourages (but doesn't require) the same check to be
   made on the HELO as on other identities.  If this is how it is
   deployed in practice, it will lead to unnecessarily lax checks on
   the HELO, which will reduce the effectiveness of the HELO check.

   A complex SPF record may reference many different providers,
   whereas a particular HELO string will typically come from a
   specific MTA.  The big problem though is when an SPF record ends in
   ?all or ~all.  It is counterproductive to encourage people to use
   this same record for HELO checks.  You may not know all the MTAs
   that might sometimes originate mail for your domain, but HELO
   strings typically identify individual MTAs, and you almost
   certainly know the exact list of IP addresses in use by a specific
   MTA under your control.  Encouraging people to relax the HELO check
   just because they don't feel comfortable with a strict PRA (or MAIL
   FROM) check is undesirable.

2. It's not clear that unifying them (in the sense that I understand
   is intended by Meng Weng Wong) makes much sense, given the set of
   valid identities is typically disjoint.  Ignoring for a moment the
   fact that a HELO identity is a domain, whereas a PRA or MAIL FROM
   is a mailbox, even the domains used are typically disjoint.

   If example.com is a domain that occurs in the PRA and MAIL FROM, it
   typically won't be a valid HELO identity.  Conversely,
   mx1.example.com might be a valid HELO identity, but is unlikely to
   be valid as part of a mailbox.

   Unifying the proposals syntactically may make sense -- ie using a
   (subset of) SPF syntax for the CSV/CSA records.  I remain to be
   convinced that unifying them semantically is sensible.

3. The models of reputation and accreditation are different.  One is
   the reputation of a domain, the other is a reputation of a host.
   It's not clear to me that both of these reputation services will be
   provided by the same set of providers, so if you're going to try
   and use a single record for both purposes, you probably want to be
   able to select separate sets of reputation services for the HELO
   identity and the PRA/MAIL FROM identities, in order to avoid
   redundant quieries to the 'wrong' providers.  This _could_ be done
   with a single record, but given (2) above, this starts looking
   unnecessarily messy to me.

   -roy

---

From: Roy Badami <[email-masked]>
Date: Wed, 30 Jun 2004 22:19:36 +0100

And to followup:

1a. Even if using the same record for HELO as for other identities
    provides an acceptable level of assurance (which will only ever be
    the case when the record ends -all) sharing a complex SPF record
    will probably result in far more DNS queries to reject a forged
    HELO than would be necessary if a separate record (of whatever
    syntax) were used for this purpose.

    -roy

---

From: Meng Weng Wong <[email-masked]>
Date: Wed, 30 Jun 2004 18:03:53 -0400

On Wed, Jun 30, 2004 at 09:21:47PM +0100, Roy Badami wrote:
| 
| OK, here's what I see as the probems with Unified SPF (as I understand
| the proposal) as compared with using SenderID in conjunction with
| CSV/CSA.
| 
|    A complex SPF record may reference many different providers,
|    whereas a particular HELO string will typically come from a
|    specific MTA.  The big problem though is when an SPF record ends in
|    ?all or ~all.  It is counterproductive to encourage people to use
|    this same record for HELO checks.  You may not know all the MTAs
|    that might sometimes originate mail for your domain, but HELO
|    strings typically identify individual MTAs, and you almost
|    certainly know the exact list of IP addresses in use by a specific
|    MTA under your control.  Encouraging people to relax the HELO check
|    just because they don't feel comfortable with a strict PRA (or MAIL
|    FROM) check is undesirable.
| 

The problem statement seems to assume that domain names used
in MAIL FROM will overlap with domain names seen in HELO.
If I read this assumption wrongly, please correct me.

  HELO domain.com
  MAIL FROM:<[email-masked]>

In this situation, an SPF record could easily become
unnecessarily complex for HELO purposes.

  domain.com. TXT "v=spf1 include:this include:that a mx ?all"

If the domain admin changes the HELO string to be:

  HELO mta1.domain.com
  MAIL FROM:<[email-masked]>

Then the problem goes away.

  mta1.domain.com. TXT "v=spf1 a -all"
  domain.com.      TXT "v=spf1 include:this include:that a mx ?all"

Is that any better?

---

From: Roy Badami <[email-masked]>
Date: Wed, 30 Jun 2004 23:18:43 +0100

>>>>> "Meng" == Meng Weng Wong <[email-masked]> writes:

    Meng> The problem statement seems to assume that domain names used
    Meng> in MAIL FROM will overlap with domain names seen in HELO.
    Meng> If I read this assumption wrongly, please correct me.

    Meng>   HELO domain.com 
    Meng>   MAIL FROM:<[email-masked]>

Actually, I was assuming that the above case is uncommon.  Though we
probably need to address it.

    Meng> If the domain admin changes the HELO string to be:

    Meng>   HELO mta1.domain.com MAIL FROM:<[email-masked]>

    Meng> Then the problem goes away.

    Meng>   mta1.domain.com. TXT "v=spf1 a -all" 
    Meng>   domain.com.      TXT "v=spf1 include:this include:that a mx ?all"

Which (almost) boils down to CSV, but using SPF syntax instead of SRV
records.

I have to say that I hadn't realized this was the intent of Unified
SPF.

If you're expecting to have separate records for the HELO identitiy
and other identities, then I don't see the advantage of making them
both generic.  The first of your TXT records should be clearly for
HELO use only, and the second should be clearly for PRA and MAIL FROM.

As it stands, your example still allows anyone to forge a HELO domain.com

   -roy

---

From: Meng Weng Wong <[email-masked]>
Date: Wed, 30 Jun 2004 18:25:16 -0400

On Wed, Jun 30, 2004 at 11:18:43PM +0100, Roy Badami wrote:
| 
|     Meng>   mta1.domain.com. TXT "v=spf1 a -all" 
|     Meng>   domain.com.      TXT "v=spf1 include:this include:that a mx ?all"
| 
| Which (almost) boils down to CSV, but using SPF syntax instead of SRV
| records.
| 
| I have to say that I hadn't realized this was the intent of Unified
| SPF.

Yes, that is the intent of Unified SPF; use a single syntax
model to offer different semantics appropriate to each
identity context.

In the HELO context, you get CSV semantics.

In the PRA context, you get Caller ID semantics.

In the MAIL-FROM context, you get SPF Classic semantics.

In the PTR context, you get MTAMark semantics.

The "unification" does not mean all the above identities
need to use the same domain name.

---

From: Douglas Otis <[email-masked]>
Date: Wed, 30 Jun 2004 17:13:03 -0700

On Wed, 2004-06-30 at 15:03, Meng Weng Wong wrote:
<snip>
>   domain.com.      TXT "v=spf1 include:this include:that a mx ?all"

Is Sender-ID depreciated over rejections of XML?  Is there a push to
restore SPF?  It would be nice to see a draft that brings these ideas
(2822 identity schemes) into a "unified" document. : )

There is nothing in this SPF record that ensures this is intended to
define the MTA host name so it can be spoofed with helo strings
selecting the wrong record.  It still requires use of a parser to
understand the record as opposed to using existing libraries that
already include the SRV format.  This SPF string may require additional
queries to read any number of differing records.  The SRV record is able
to return a list of addresses, and where these hosts can be assigned
using existing tools.

CSV and SPF/Sender-ID records are requested at different points within
the process, so there may not be a need to spawn a SPF/Sender-ID parser
if the connection is rejected by the CSV process.  This is a
consideration when receiver side resources are considered.  If CSV is to
provide DDoS protection, using an SRV record improves upon this
protection.  As these lists may tend to look similar for some, there is
great risk of confusion or simply allowing clever spoofing made possible
by this overloading of SPF/Sender-ID/CSV.  Keep CSV separate using SRV.

I do not see these two distinct and separate functions as either
competitors or as incompatible. But again, there is no reason to join
these two mechanisms into a common record, and many reasons not to join
them.

-Doug

---

From: wayne <[email-masked]>
Date: Wed, 30 Jun 2004 19:42:42 -0500

> Which (almost) boils down to CSV, but using SPF syntax instead of SRV

Correctly, although the SPF syntax allows for more flexibility and
extensibility. 

> As it stands, your example still allows anyone to forge a HELO domain.com

Sort of.

For whitelisting purposes, the only people who can "forge" domain.com
in the HELO are those that are allowed by the domain.com SPF record.
While this SPF record is not as tight as it could/should be, in
practice I don't see much of a problem.  Anyone that you trust enough
to send email using your domain in the 2821.FROM or 2822.From: would
almost certainly be trusted not to forge your 2821.HELO.

However, you are right that because the domain.com SPF record ends in
?all, people can't immediately reject random zombie machines that
use HELO domain.com.

There are two ways to "fix" this:

1) domain.com can transition from using ?all to using ~all or -all

2) if the scope macro variable is allowed (it was suggested as part of
   the Unified-SPF), then you would need to have something like this:

mta1.domain.com. TXT "v=spf1 a -all" 
domain.com.      TXT "v=spf1 include:this include:that a mx redirect=%{scope}._spf.%{d}"
helo._spf.domain.com. TXT "v=spf -all"
*._spf.domain.com.    TXT "v=spf ?all"

The extra DNS lookup would only be required if the email didn't pass
the other tests.  For most domains, this will usually happen only in
the case of spam.

Actually, there has also been a third suggested way for Unified-SPF to
fix this.  Use a different version tag, such as:

domain.com.      TXT "v=spf1 include:this include:that a mx ?all"
domain.com.      TXT "v=spf1-helo -all"

Gain, the complicated cases are only needed for domains that are
waffling about their email policies, but want a tight email policy in
certain scopes.

Personally, I think something like the scope macro-variable needs to
be done in Sender-ID anyway since the current SPF records have all
been published under the assumption that they apply to 2821.FROM and
2821.HELO scopes and the 2822.From: scope is not always going to be
the same.

-wayne