Subject: Make CSV backwards compatible with SPF?

From: Matthew Elvey <[email-masked]>
Date: Mon, 20 Sep 2004 16:33:37 -0700

Here's the idea in a nutshell: CSV makes use of extant SPF records which 
already list tens of thousands of domains that are authorized for use in 
HELO.

Who supports/opposes this change?

Details:
Perhaps a CSV that was backwards compatible with a simple subset of SPF1 
records would be useful.  SPF records that expect to result in an 'SPF 
pass' already must list the names used in HELO, so why not make use of 
that fact?
There was a survey (I tried to find it, but failed) mentioned on list 
showing that ~95% of all SPF records were one of a dozen strings (-all, 
~all, +a +mx -all, ...) that were resolvable with one DNS query (a link 
would be appreciated). 
Who thinks compatibility with a subset of SPF records would be an 
improvement to CSV?  Domains could continue to publish one of these 
records, and would need do nothing else to be compatible with this 
revised CSV, or if they preferred more flexibility and/or wanted to be 
good neighbors, could publish normal CSV records.
The subset could be defined as
a)those that resolve to one of the n (e.g. n=12) most popular records, or
b)records that use only, e.g. a, ptr, mx, ip4, or ip6 and fit in UDP.

In either case, parsing issues, and the issues of TCP failover, TXT 
crowding and DNS providers that don't support CSV nearly disappear.

The algorithm would be that a check is done for a CSV record first (and 
if found, only it is used); if none is found, a check is made for a 
conforming SPF record.

Who would support this change?
Would you prefer a CSV compatible with a larger set of SPF records?  
(Keep in mind that domains that aren't used in HELO don't need to have 
these records at all, as far as CSV is concerned.  If you have a complex 
SPF record, that's OK - keep it.  Just add a simple SPF record for the 
hostname you announce in HELO, or add a CSV record.)

---

From: Douglas Otis <[email-masked]>
Date: Mon, 20 Sep 2004 17:21:48 -0700

> Who supports/opposes this change?

Ney.

This is a repeat of an older debate. This would not be an improvement
over the current CSV draft.  SPF often requires a series of lookups to
resolve an address list.  This list is often not fully defined as it is
commonly intended to associate mailbox domains with a much larger set of
MTA addresses.  It also may allow the EHLO domain to use SPF/Sender-ID
records as a means to spoof, as these records are often open-ended.  The
host name used within the EHLO domain will require a separate record
anyway, so why not use a record that achieves an answer within a single
lookup?  Why not use a different record which can not exploit a record
intended to associate mailbox domains with MTAs?  Making SPF/Sender-ID
records an optional fall-back will only cause problems.  SPF/Sender-ID
does not need to be an "all for one, and one for all" record.  : ) 

-Doug

---

From: Matthew Elvey <[email-masked]>
Date: Mon, 20 Sep 2004 18:50:54 -0700

On 9/20/2004 5:21 PM, Douglas Otis sent forth electrons to convey:

>This is a repeat of an older debate. 
>
No, I'm making a different suggestion.  Please read the whole post!
I'm eliminating some or all of the records that you mention having a 
problem with below.

I should have stated one thing clearly: only +type components of records 
would be relevant; ~type and ?type would be ignored.

I registered your Nay on Jabber...

>SPF often requires a series of lookups to
>records as a means to spoof, as these records are often open-ended. 
>
No open-ended records would be allowed.  I don't think the fact that, 
e.g. +0/0 is allowed is of much concern, as spammers have demonstrated 
they can create MARID records for their own domains quite well; A&R are 
required.

> The
>lookup?  
We should, as my post makes clear.  But there are reported to be what, 
~300,000 SPF records?  Why not use them to the extent that I suggest?

---

From: John Leslie <[email-masked]>
Date: Mon, 20 Sep 2004 22:08:06 -0400

Matthew Elvey <[email-masked]> wrote:
> 
> Perhaps a CSV that was backwards compatible with a simple subset of
> SPF1 records would be useful. 

   I don't wish to support or oppose this; but some discussion might
be helpful.

   Hopefully we can agree (for purposes of this discussion anyway)
that the subset should be small. At first blush, there are exactly
two useful cases:

- "+a -all"

- "-all"

   The first says that the exact list of address records (ipv4 or ipv6)
are all authorized to act as MTAs using this domain string as the
EHLO parameter; the second says that no host is authorized to use this
domain string as the EHLO parameter.

   In the first case, another DNS query for the address records would
be needed, which immediately raises two questions:

- is the response going to fit in a UDP reply?
  If not, we add a lot of complexity, and problems with (poorly configured)
  firewalls could be an issue.

- is the response going to return a complete list of address records,
  or are we facing a specialized DNS server which returns partial lists?
  If so, the situation is hopeless: we'd need to forbid this.

   IMHO, the "+a -all" case is a bad deal for CSV unless we define that
the response MUST be the complete set and MUST fit in a standard-size
UDP reply.

   The second case looks more useful: people _could_ publish wildcard
TXT records to forbid a wide variety of subdomains as EHLO parameters.
My personal thinking is that what we actually want is better accomplished
by wildcard suggested-service PTR records; but I don't want to rule out
other options.

   It's worth noting that CSV doesn't _need_ a way to publish prohibitions
in order to accomplish its purpose. There _will_ be many domains without
CSV policy published: if these domains only use IP address ranges with
no history of email abuse, there's no reason they _need_ CSV authentication,
authorization, and accreditation. CSV's great benefit is its ability to
quickly disseminate accreditation information which is both positive and
dependable.

   I agree there will be domains that want to make the prohibition of
use of any subdomains as EHLO strings; and it _might_ be conceptually
easier for managers of those domains to understand the use of SPF1 -all
records for that purpose.

   (My thinking for such domains has been to _not_ publish a CSV SRV
record, but publish a wildcard CSV PTR record with a target which will
recommend _against_ trusting MTAs using that EHLO string.)

--
John Leslie <[email-masked]>

---

From: Matt Sergeant <[email-masked]>
Date: Tue, 21 Sep 2004 09:00:56 +0100 (BST)

> anyway, so why not use a record that achieves an answer within a single

My argument for this idea is simply that we settled on the SPF format
simply because single lookups (an "exists" lookup) were not flexible
enough for people's full needs. I doubt CSV's single lookup will be
either. Of course there is nowhere near as much adoption of CSV as there
is of SPF so we can't possibly be certain of this yet.

Matt.

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

---

From: Douglas Otis <[email-masked]>
Date: Tue, 21 Sep 2004 11:57:17 -0700

> is of SPF so we can't possibly be certain of this yet.

You are right.  Much like a post-mark, the CSV records simply fix SMTP
EHLO authentication to lay a solid foundation upon which other
relationships can be asserted using additional records.  Importantly, an
authenticated MTA name is strong enough to provide reputation
assertions.  Building upon a validated list of MTA names allows a
relationship with some mailbox-domain and some vast mail channel to be
asserted using a simple name list.  There would be little motivation to
spoof the mailbox-domain/mail-channel (open) relationships, as this
identity would typically be unable to serve a gate-keeping function.  In
the few cases where it is used in this manner, the outbound mail-channel
must not be shared, and recipients must be alerted to not offer a
forwarded mail address.

By allowing the MTA to be validated, this identifies where the mail
originated, much like a post-mark.  Should you receive a letter from
Uncle Jon, but from a different city, you can write back and asked about
his trip.  If he says "What trip?" then the post-mark served a valuable
service.  If there was a crime, this electronic post-mark indicates
where to obtain logs to discover the true source of the message.  The
post-mark itself may be all that is needed should this be used by a
single organization.

At some point in the future, it is possible mail providers may request
customers to create CSV-CSA records to enable the use of customer's
domain within the EHLO name for messages being sent on their behalf. 
This assumes the CSV name reputation trumps any IP address black-list,
as it should.     

This additional record could be:
_marid.my-mailbox-domain.tld. PTR  my-isp.com.
_marid.my-mailbox-domain.tld. PTR  my-support.com.
_marid.my-mailbox-domain.tld. PTR  my-remote-office.com.
...

The scope of the assertions implied by this _marid record should be for
a single field, the RFC2822 From and nothing else in my view.  As I said
in another post, it could be for the REF2821 MAIL FROM or the PRA
selected field for that matter. 

The point is simple. By using a name list rather than an address list,
there is no need for highly risky and time consuming subsequent lookups
to just discover addresses.  A name is a much more concise means of
expressing this relationship which does not involve the same level of
maintenance and trouble.

The great thing about rehabilitating the EHLO name, is that this name is
provided early in every SMTP session.  The EHLO name is captured within
the message headers to allow tracing.  There is no need for submitter or
to invent new mail headers to discern where the message is originating
with the use of CSV applying the post-mark so to speak.                 

On Mon, 2004-09-20 at 18:50, Matthew Elvey wrote: 
> >> Here's the idea in a nutshell: CSV makes use of extant SPF records
> >> which already list tens of thousands of domains that are authorized
> >> for use in HELO.
> >>
> I should have stated one thing clearly: only +type components of
> records would be relevant; ~type and ?type would be ignored.

As I said, this will allow the use of other record types to spoof the
EHLO name.  The SPF type of record will normally be across an entire
array of providers and not for a single host.  One compromised system
within this vast region can then utilize one of these records to spoof
the EHLO name then.  If reputations are based upon the MTA name and not
the mailbox domain, then protecting the MTA name would be much more
important.  It MUST not be possible to mix these records out of this
concern.

> I registered your Nay on Jabber...

Thanks.

> > SPF often requires a series of lookups to resolve an address list.
> > This list is often not fully defined as it is commonly intended to
> > associate mailbox domains with a much larger set of MTA addresses.
> > It also may allow the EHLO domain to use SPF/Sender-ID records as a
> > means to spoof, as these records are often open-ended. 
> 
> > The host name used within the EHLO domain will require a separate
> > record anyway, so why not use a record that achieves an answer
> > within a single lookup?  
> 

These records are for a different purpose, and expose a different set of
machines.  The EHLO record set must be kept narrow in order to serve the
purpose of establishing accountability.  The broad and weak nature of
the identities obtained using SPF/Sender-ID are not useful for this
purpose.  SPF/Sender-ID should be viewed as informative for but a few
exceptions.  

-Doug

---

From: Matthew Elvey <[email-masked]>
Date: Tue, 21 Sep 2004 19:45:35 -0700

Doug, still hoping to hear back from you - on 9/14, I asked:

> Doug, did you get my email - of a possible broad, strong example 
> showing [SenderID] falling down. 

and

> If foo.dom's SPF/SenderID record lists just MX, and doesn't send spam, 
> how/why would a reputation service blacklist foo.dom?
> e.g .
> foo.dom.    IN    TXT      "v=spf1 mx -all" 

>with the use of CSV applying the post-mark so to speak.                 
Well put.

>EHLO name.  
I claim this is impossible.  Please provide an example of this (using a 
record of the kind that my proposal proposes using).

>The SPF type of record will normally be across an entire
>array of providers and not for a single host.  
>
I don't see how this is possible using a record of the kind that my 
proposal proposes using. 
+include:comcast.net is not allowed, for example! 
+ptr isn't allowed either (we don't want to allow the likes of 
dialup-pool.234.234.comcast.net...)

>One compromised system
I don't believe my proposal does this.  
<snippage>

>purpose.  SPF/Sender-ID should be viewed as informative for but a few
Let's nail down what exceptions are appropriate then! 
(Again, I'm NOT suggesting using the identities obtained using 
SPF/SenderID!)

Some refinements to my proposal based on stuff hashed out in private email:

99% of the time, an MX query will return the A records for all the 
servers listed in the MX record as add'l records, all in one UDP packet.
Let's say that's a requirement for our purposes, where there's a +MX in 
the SPF record.

"+a +mx -all" would
need 2 more DNS queries to resolve (unless a ALL query was done, which
is likely a bad idea). I think it's reasonable to accomodate a bit more
than 1 add'l DNS query.

Let's not worry about the <<1% of records where the DNS servers are known to return incomplete
lists of Address records
Let's say that's a requirement for our purposes that this not happen.

The fix in these cases would be make a 'proper' CSV record. 

Also, a pass relying on such records would carry slightly less weight than a pass based on a 
'proper' CSV record.  But let's keep the definition tight enough that the weight be only slightly less.

Again, the reason for my proposal is not to change or obsolete
CSV records, but to make CSV work in some reasonable cases where it
can work without them without being significantly detrimental.

---

From: Douglas Otis <[email-masked]>
Date: Wed, 22 Sep 2004 01:01:11 -0700

> > showing [SenderID] falling down. 

In general, shared MTA environments and open lists are the basic
ingredients for having a reputation damaged by SPF/PRA schemes which
only bolster the mailbox domain which would be the likely selected
identity for reputation assessment.  This would be a bad situation.

> > foo.dom.    IN    TXT      "v=spf1 mx -all" 

In a simple environment exchanging messages end-to-end, the risks would
be low.  There are risks accepting these types of records when a process
walks through redirects, includes and arbitrary labels for several
record types.  This invites a birthday attack where foo.dom can be
planted as a poison record in the cache.  The many possible records that
may be involved in the potential SPF matrix offers a risk where the
minimum accepted is 10 scripts and the record count could be in the
hundreds.  

> record of the kind that my proposal proposes using).

The label used to select the SPF record is normally based upon the
mailbox domain to select a full set of outbound MTAs.  The EHLO name can
be made to match any such mailbox domain provided one of the machines is
permitted in this SPF record set.  This would be bad.

Again, the intent of the SPF record is to grant a broad set of MTAs,
within many different domains, authority to send messages on behalf of
the mailbox domain.  The CSV record is limited to the single
administrative domain.  Only the domain that is actually administering
the domain should be included, actually just the machine hosting the
MTA.

> dialup-pool.234.234.comcast.net...)

One, you want to use existing SPF records that were not intended to
provide the narrow selection of machines administered by the MTA
operators.  Two, there is no means to restrict this list after the fact.

> > One compromised system  within this vast region can then utilize one
> > of these records to spoof the EHLO name then.  If reputations are
> > based upon the MTA name and not the mailbox domain, then protecting
> > the MTA name would be much more important.  It MUST not be possible
> > to mix these records out of this concern.
>

Your approach was to prune the record set used just for the EHLO
domain.  That pruning can not differentiate between machines within a
delegated domain or machines within the administered domain. This would
be bad.

> (Again, I'm NOT suggesting using the identities obtained using 

The exceptions would be with respect to those domains that feel it is
okay to close their lists.  Small organizations, able to manage the
support issues and don't care when some mail goes missing or gets
rejected, may opt to endure these problems.  There may be a trade-off
dealing with aggressive phishing attacks causing worse problems.  If you
look at the marid-mpr draft, you will see I have a setting to assert
that the name list is the exclusive source of mail for the mailbox
domain.  I described this mode being suitable for exigent situations,
but would not expect this to be the typically use.  

> is likely a bad idea). I think it's reasonable to accomodate a bit more

For both network and security concerns, running a parser that walks
through these records is not reasonable.  The CSV record can be obtained
in a single binary query, and the MPR list also can be obtain within a
single query.  Neither of these operations require a parser to perform
this operation.  My concern would be dedicating a thread to run the
parser will likely constrain the port used to do the queries.  Having
this parser forced to lookup one arbitrary record after the next on any
number of DNS servers enables a fairly lazy attack to be successful.

I suppose this would be a way to get people to speed up deployment of
DNSSEC, but I would rather stay on the safe side of the use of DNS for
now.  I see little about these TXT script records in DNS as being safe. 
> Let's not worry about the <<1% of records where the DNS servers are
> known to return incomplete lists of Address records. Let's say that's
> a requirement for our purposes that this not happen.

Once the IETF has expended their considerable efforts to endorse a
method that is well considered and robust, I doubt there will be any
trouble with the adoption of the standard.  The problem of spam is that
great.  Domains like AOL, as example, return incomplete lists.  I doubt
you could say their lists are actually incomplete, they are simply open
to indicate mail from their customers my emerge from machines outside
their networks.     

> The fix in these cases would be make a 'proper' CSV record. 

As I indicated, there are serious problem using scripts to spew a huge
list of addresses.  This should change to being just a list of names
used to define the potential sources of mail.  Leave the address list to
be within the natural scale of DNS records, the single host as provided
by the SRV record for example.

> Also, a pass relying on such records would carry slightly less weight
> than a pass based on a 'proper' CSV record.  But let's keep the
> definition tight enough that the weight be only slightly less.

I still insist the SPF record should never be allowed to validate the
EHLO domain.  It is a different data set for a different purpose.  

> can work without them without being significantly detrimental. 

Sorry, but I fail to see the point in using a TXT record and a parser to
obtain a list of addresses through a dangerous and expensive set of
lookups, when it can be done in one lookup safely.  Also where it will
not accidentally allow other domains to spoof the EHLO identity.  This
exclusion is needed to establish accountability.  

-Doug