Subject: MARID compatibility with SPF records

From: Roy Badami <[email-masked]>
Date: Sat, 17 Jul 2004 01:21:27 +0100

This is an issue that the WG was supposed to have decided upon some
time ago, but I've seen no discussion on this.

Currently MARID records (as defined by draft-ietf-marid-protocol-00)
and SPF records (as defined by draft-mengwong-spf-01) are not
distinguishable.  Given they are broadly compatible, this isn't a
major problem, but they are not semantically equivalent since they
check different identifies: the PRA in the case of MARID and the MAIL
FROM (usually; or the HELO in the case of <>) for SPF.  So I contend
that it _is_ still a problem.

My focus here is people publishing MARID records which they do not
wish to be evaluated by the deployed (and to be deployed [1]) base of
SPF checkers conforming to -spf-01.  I'm not considering the (IMHO
smaller) problem of existing records with -spf-01 semantics being
interpreted by a MARID-compliant checker.[2]

I made a proposal in
http://www.imc.org/ietf-mxcomp/mail-archive/msg02361.html of a way to
handle this, which no-one followed up on; I link to it here in case
anyone missed it.

I'd like to now make an alternative proposal:

Add a new mechanism 'null' to MARID, the null (or no-op) mechanism.
This mechanism is the converse of 'all'; it never matches.

A publisher who wishes to ensure that their records are not
interpreted by systems conforming to -spf-01 can easily use this as
the first mechanism in their MARID record.  Since 'null' is not a
valid mechanism in -spf-01, an SPF checker will see an unknown
mechanism and return 'unknown', essentially ignoring the MARID record.

I don't see most people caring.  Indeed, I suspect most people will be
happy for -spf-01 checks to be carried out on their MARID records.
But I do think that some people may have legitimate reasons for
caring, and that the issue should be addressed (or at least
considered)...

	-roy

[1] SpamAssassin 3.0, which is expected to be released in about a
month's time, supports SPF checks if Mail::SPF::Query is installed.
AIUI Mail::SPF::Query is fairly close to an implementation of the
check_host() function; however SpamAssassin 3.0 will pass it the MAIL
FROM identify, not the PRA, resulting in SPF semantics.

[2] A lesser problem because people who have already published SPF
records at this point in time are early adopters, who are likely to be
aware that they're implementing a mechanism that's subject to change,
and can be expected to be reasonably likely to review the records
they've published in the light of the MARID specifications.

---

From: Andrew Newton <[email-masked]>
Date: Sat, 17 Jul 2004 06:59:43 -0400

> time ago, but I've seen no discussion on this.

Agreed.  Nobody is talking about this issue, even though it is 
important and is a technical issue.

-andy

---

From: Margaret Olson <[email-masked]>
Date: Sat, 17 Jul 2004 09:57:13 -0400

> important and is a technical issue.

My understanding from reading draft-ietf-marid-protocol-00 was that 
"spf v1" was being redefined to use the PRD algorithm. I don't think 
this works, but perhaps I have missed something.  I'm a bit behind in 
my reading or I would have commented earlier.

There are two basic cases: the MAIL-FROM and the PRD match (SUBMITTER) 
match, and the MAIL-FROM and SUBMITTER don't match. For mail with a 
matching PRD and MAIL-FROM, the spf v1 and sender id records are the 
same and there is no need to make any changes. For domains without at 
matching MAIL-FROM and PRD, it is highly likely that the MAIL-FROM is a 
bounce manager of some sort, in which case the MAIL-FROM is never the 
SUBMITTER, and the spf v1 record for the MAIL-FROM domain is still 
valid. There either is or is not a record for the PRD, and the sender 
id algorithm proceeds on that PRD domain.

Problems occur with domains where the MAIL-FROM matches the PRD in one 
system (for example the mail sent by an ISP), and that domain has 
published an SPF v1 record, and that domain is used as the PRD (2822 
from) in another system with it's own error handler. In other words, in 
domains that have an ISP for personal mail and at least one outsourced 
service that sends specialized mail (for example a shopping cart or a 
email service provider for marketing messages). I think this situation 
is common for small businesses and far more common in general than this 
group has given it credit for.

In this situation, there are two basic cases: the first is an ISP 
customer is using an ISP shared domain (e.g. aol.com, hotmail.com) as 
the PRD to send mail through some other service, for example 
[email-masked] . In this case, bigisp.com may not wish to allow 
this, and almost certainly doesn't want to give a blanket permission 
for mail from any of their customers to come from outsource provider 
xyz just because one of their customers wishes to allow this. So I 
would call this case "not a problem" for MARID, although I'll predict 
that the use of consumer addresses for business is pretty common and 
that this will some level of unhappiness and turmoil. (I also firmly 
believe that at $5.00 for the low cost providers and $25.00 for the 
high cost providers, asking even the teeniest of businesses to get 
their own domain name is not the slightest bit unreasonable).

The second case is where the ISP customer has their own domain, managed 
by the ISP, e.g. [email-masked].  If the ISP has published spf 
classic records, this currently works in almost all cases: the non-ISP 
mail services use their own domain in the MAIL-FROM, and in many cases 
(about 70%) the customer domain is the PRD. This breaks the moment 
someone interprets the spf classic records as a sender id record. Here 
I think it behooves mail administrators to make sure they actually know 
how their customers send mail.  I do not have any good information on 
how many mail admins have published spf records on behalf of client 
domains (as opposed to their own domains). If this number is small to 
non-existent, than this is a non-problem.

Both these problems are fixed with either a separate SPF version for 
sender id or sub-domains assigned to specific outsourced providers. For 
people using their own domain, sub-domains are probably the best long 
term solution, but they do need to be set up.  My concern is that 
things that work now will break in chaotic ways; records will be valid 
at one receiving domain (implementing spf classic) but not at another 
(implementing sender id). No matter how much educating we do this will 
be a problem.

Since it is safe to interpret a v2 record as v1, v1 receive side 
implementations would at a minimum need to modified to accept v2 
records. Alternately, we could have a form of v2 record that said "go 
look at the v1 record" for those domains where the v1 and v2 records 
are the same.

Margaret.

---

From: Mark Lentczner <[email-masked]>
Date: Sat, 17 Jul 2004 09:45:52 -0700

On Jul 16, 2004, at 5:21 PM, Roy Badami wrote:
> FROM
Technically, But draft-ietf-marid-protocol-00 doesn't specify which 
identity to check.  Using the records to check PRA is defined in 
draft-ietf-marid-core-02.  But yes, there is a semantic difference 
here.

> Add a new mechanism 'null' to MARID, the null (or no-op) mechanism....
I think you correctly point out that for most situations, domains 
publishing records don't care, and what they publish to be checked 
against PRA, they can accept that it will be used to check MAIL FROM in 
older configurations.  For those situations where they do care, your 
'null' mechanism is a nice idea.

But the real question in my mind is, while I understand the semantic 
difference from a purely technical point of view, I'm not sure that I 
can find a practical difference.  Specifically, I'm not sure I can find 
a case where:

	There is a domain that identifies at least one host that it permits to 
be used as the PRA
	but not as envelope MAIL FROM.
or
	There is a domain that identifies at least one host that it does not 
permit to be used as
	PRA, but does permit as envelope MAIL FROM.

For simple mail, the envelope MAIL FROM and PRA are the same.  So any 
case must involve more complicated mail set ups: forwarding, lists, or 
bulk mail.

On Jul 17, 2004, at 6:57 AM, Margaret Olson wrote:
> from) in another system with it's own error handler. In other words, 
> in domains that have an ISP for personal mail and at least one 
> outsourced service that sends specialized mail (for example a shopping 
> cart or a email service provider for marketing messages).
Margaret gives two examples, the first of which she declares to be "not 
a problem", since it is solved by businesses having their own domain 
name.  I'm going to restate the second example to be sure I understand 
it, and to see if it fits the test above:

mybikeshop.com normally sends it's mail via its ISP's servers at 
bigisp.com, however, it sometimes sends bulk mailing to its customers 
via bulkmailer.com.  Hence, we see normal messages as:

	SMTP client:        mx.bigisp.com
	envelope MAIL FROM: [email-masked]
	header PRA:         [email-masked]

and we see bulk messages:

	SMTP client:        mx.bulkmailer.com
	envelope MAIL FROM: [email-masked]
	header PRA:         [email-masked]

So the records involved are the SPF records for bulkmailer.com and 
mybikeshop.com.  We'll take these in this order.

bulkmailer.com needs to publish a record that old, envelope MAIL FROM 
checking SPF implementations will pass when seeing the second type of 
e-mail.  So it publishes:

	bulkmailer.com.	IN TXT "v=spf1 +a:mx.bulkmailer.com -all"

If this record is used by a newer implementation that checks PRA, this 
almost certainly valid since bulkmailer.com could send its own mail 
(where it is in the PRA) from this machine.  Even if it didn't, it 
administratively controls this machine, and is already accepting 
accountability for it.  Such a declaration doesn't result in a loss of 
reputation or an opportunity for a spammer.

mybikeshop.com needs to publish a record that envelope MAIL FROM 
checking SPF implementations will pass when seeing the first type of 
mail, and newer implementations will pass with either.  So it 
publishes:

	mybikeshop.com.	IN TXT "v=spf1 +a:mx.bigisp.com +a:mx.bulkmailer.com 
-all"

This record is slightly more liberal than the requirements.  In 
particular, implementations checking envelope MAIL FROM against this 
record will Pass an envelope MAIL FROM with mybikeshop.com when coming 
from mx.bulkmailer.com.  Given that mybikeshop.com already trusts 
bullkmailer.com to use its name as PRA on e-mail, it seems unlikely 
that mybikeshop.com would be unwilling to trust that they could set up 
the envelope MAIL FROM correctly.

So while this case again shows that there could be a difference on 
purely technical grounds, (bulkmailer.com isn't expected to use 
mybikeshop.com in envelope MAIL FROM) as a practical matter it fails 
the tests: mybikeshop.com trusts bulkmailer.com with it's domain name 
for PRA and loses nothing by trusting them with it for envelope MAIL 
FROM, even if they agree not to use it that way.  (Though I suspect the 
vast majority of mybikeshop.com-s in the world don't even understand 
the difference and happily grant the bulkmailers of the world the 
rights to use the domain name in e-mail sent on their behalf in any way 
the bullkmailer sees fit!)

To recap --

1) There is concern that since the proposed use of SPF format records 
is changing from envelope MAIL FROM checking to PRA checking, that the 
difference in semantics might cause problems for some domains:  Older 
implementations of SPF will use records intended for PRA against 
envelope MAIL FROM; Newer implementations will use older records 
intended for envelope MAIL FROM against PRA.

2) The 'null' mechanism is a very simple and clean solution to the 
first of these to problems.

3) While one can construct cases that show the technical difference in 
the semantics, I still cannot find one that demonstrates a practical 
need for distinguishing a host between them.

Hence, I'm not convinced we need the 'null' mechanism, or two versions, 
or scoping declarations in the record, or multiple records at different 
DNS names (all proposed solutions I've reviewed to this issue), since 
I'm not convinced the issue is real enough to warrant the change.

	- Mark

Mark Lentczner
http://www.ozonehouse.com/mark/
[email-masked]

---

From: Margaret Olson <[email-masked]>
Date: Sat, 17 Jul 2004 17:18:53 -0400

On Jul 17, 2004, at 12:45 PM, Mark Lentczner wrote:
> can find a practical difference.  Specifically, I'm not sure I can 
> find a case where:
>
> 	There is a domain that identifies at least one host that it permits 
> to be used as the PRA
>> Problems occur with domains where the MAIL-FROM matches the PRD in 
>> one system (for example the mail sent by an ISP), and that domain has 
>> outsourced service that sends specialized mail (for example a 
>> shopping cart or a email service provider for marketing messages).
> Margaret gives two examples, the first of which she declares to be 
> "not a problem", since it is solved by businesses having their own 
> domain name.  I'm going to restate the second example to be sure I 
> understand it, and to see if it fits the test above:
>

I completely agree up to this point.

My question is: what mybikeshop.com's ISP has *already* published an 
SPF record for mybikeshop.com, without taking sender id into 
consideration? Given that SPF has been around longer than Caller-ID, 
and the syntax merger is extremely new, it is not unreasonable to 
expect that this situation exists. In this scenario, mybikeshop.com's 
SPF record probably looks like:

mybikeshop.com IN TEXT "v=spf1 +a:mx.bigism.com -all"

This is currently correct in an SPF classic context and produces the 
desired result of SPF classic: it protects against MAIL-FROM forgery. 
But it's not a correct SenderID record - the correct SenderID was shown 
in Mark's text above, and includes bulkmailer.com.

At what point is it safe for receivers to assume that mybikeshop.com's 
records have been updated and apply the SenderID PRD algorithm? (Here 
"updating" is meant to include the several approaches available, 
including literally updating the mybikeshop.com record and creating 
sub-domains for the transactional and bulk messages.)

I do not know how big this problem is, but given that there are 10s of 
thousands of domains publishing SPF records, and many 10s of thousands 
of domains (perhaps millions, depending on whose numbers you believe) 
using outsourced providers, it's reckless to assume that it is not a 
practical real world problem.

So I think we *do* need to express the record version. I am inclined to 
agree with Mark's point that there isn't any practical need for a 
permanent scoping mechanism.

Short of proof that there is no overlap between currently published v1 
records and clients of ESPs and clients of outsourced shopping carts 
record differentiation of some sort is mandatory. Otherwise we 
introduce a situation where publishing is  potentially more damaging 
that not publishing, which is a sure way to kill publication. I suppose 
the other possibility is that an authentication failure comes to have 
relatively little meaning, but that is hardly a desirable outcome 
either.

Margaret.

---

From: Dave Crocker <[email-masked]>
Date: Sun, 18 Jul 2004 12:07:20 -0700

Folks,

MO> At what point is it safe for receivers to assume that mybikeshop.com's 
...
MO> I do not know how big this problem is, but given that there are 10s of 

Networking standards that require "assumptions" do not work very well.
The point behind a standard is to specify things explicitly.

With respect to retaining syntax but changing semantics, here is a
simple example:

I offer service for "free".

Previously, everyone has used the word "free" to mean "there is no cost
to the consumer".

However for anyone offering services using the alice-in-wonderland
standard, the word "free" will mean that the consumer is obligated to
pay US$ 1,000.

I choose to conform to the alice-in-wonderland standard.

The question to the reader is:  How the consumer to know whether I
conform to this standard and, therefore, whether the consumer's use of
my service will cost them US$ 1,000?

Now apply the same question to new-vs-old semantics on a DNS record.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

---

From: "John Glube" <[email-masked]>
Date: Tue, 20 Jul 2004 07:24:24 -0400

A quick comment on the compatibility of MARID
with SPF records.

Let me use the example of mydomain.com. 

Mydomain.com sends email through its ISP and also uses the
services of a 3rd party email service provider to send out
its newsletter.

In setting up its SPF record, mydomain.com proposes
initially to publish a record:

mydomain.com.IN TXT "v=spf1 +a:mx.bigisp.com
+a:mx.bulkmailer.com -all"

(I am using the same record for ease of comparison.)

In discussions with bulkmailer.com, the owner says this is
not necessary as SPF authenticates the MAIL FROM address in
the envelope header and we use a root email address for our
domain. 

Since Sender-ID is not in the wild, by including
+a:mx.bulkmailer.com in the SPF record this will throw an
extra burden on our DNS server. Please don't do this. 

The response from mydomain.com is ok.

Now with Sender-ID a number of things happen:

* mydomain.com needs to amend its MARID record.

* Receiving MTA's may do a check on mydomain.com and
bulkmailer.com resulting in two queries of bulkmailer.com's
DNS server. (This may not be a problem if the query is
cached.)

* If mydomain.com wishes to be accredited for newsletter
sends, it can no longer rely on the accreditation of
bulkmailer.com, which is already included as part of
bulkmailer.com's service fee. 

Since the only mail sent through its ISP is response to
customer queries, it now has to pay an extra charge to have
its domain accredited, not having established a reputation
for mydomain.com, relying on the services of bulkmailer.com.

Since mydomain.com already follows best list management
practice of using closed loop verification through
bulkmailer.com, this is an extra financial cost.

Some may say, well this is the cost people will have to
bear if they want their domain protected against being used
for fraudulent purposes and to get through all the
intervening filters on an "express basis." 

I would argue this is an unnecessary cost for mydomain.com
since the business owner is already paying for
accreditation once through bulkmailer.com. Now, the
business owner has to pay twice.

As was pointed out, with SPF the authentication
cost for the business owner is perceived to be free.

However, with Sender-ID the mix has now changed and
although there is no cost in publishing a record, the cost
implications of the change on the accreditation side are
significant.

I simply ask these matters be considered as Sender-ID moves
forward. 

John Glube
Toronto, Canada

The FTC Calls For One Standard For Sender Authentication
http://www.learnsteps4profit.com/dne.html 

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.718 / Virus Database: 474 - Release Date: 09/07/2004

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 26 Jul 2004 05:46:39 +0200

> need for distinguishing a host between them.

Maybe I've found a case where this is very important: me ;-)

At the moment I use From: [email-masked] everywhere.
Including the trivial case where I submit my mail at clara.net
with a corresponding MAIL FROM:<[email-masked]>

Please note that xyzzy.claranet.de is a vanity host, it has a
simple "v=spf1 redirect=claranet.de" sender policy, and I'm
not in the position to modify this (wildcard) policy.

I also use other providers with the same From: address.  In
fact I and my MUA don't know which MSA I'll use at the moment
when I write a message.  The outbound mail waits in a file
"outbound" until I start "send now".  And then my MUA uses a
MAIL FROM corresponding to the selected provider, but it does
_not_ add a Sender: header.

Actually my 2nd SMTP provider insists on a correct MAIL FROM:
<[email-masked]> like any well behaved MSA.  It won't
modify my message in any way, it does _not_ add a Sender:.

Now if I've understood [Sender-Id] correctly the recipient
tries to find a Sender: in step 3, and a From: in step 4, and
this results in From: [email-masked] with an IP of
2nd.provider.example => Sender-Id FAIL.

In theory you could say that my MUA or the MSA should add a
Sender: header reflecting MAIL FROM:<[email-masked]>.

But in practice this won't happen (my MUA is old, and the MSA
isn't interested in MARID oddities, it's already good enough
that it enforces a MAIL FROM matching the identified user).

Why is the PRA determined by the 2822 From: address in step 4
instead of the existing and valid MAIL FROM ?  For recipients
trying to be compatible with classic SPF and RfC 2476 that
could be an obvious solution, use MAIL FROM as default Sender:

                              Bye, Frank