Subject: [Asrg] Yet another attempt to fix forwarding

From: Alessandro Vesely <[email-masked]>
Date: Wed, 30 Jan 2008 07:34:50 +0100

Hi all,
it is well known how SPF breaks forwarding. SPF is not itself an anti-spam 
mechanism, but it would help cleaning up the mail transport system by 
limiting domain abuse. If it were widely employed, that is. Thus, this 
post is not off topic, although not directly aimed at countering spam.

My tentative solution provides for a local forwarding policy. When a 
message is being forwarded (i.e. it already has a Return-Path), the mailer 
should look up the policy in order to determine what envelope sender 
replacement, if any, it should use.

Not replacing the envelope sender is fine if the mailer can be 
authenticated and/or whitelisted by the target MTA. Such whitelisting 
should be granted with per-recipient granularity: the target MTA 
whitelists only messages destined to specific recipients. The forwarder 
might use existing extensions to authenticate, e.g. the AUTH parameter of 
the MAIL FROM command. Either the mailer or the local policy 
implementation should be able to determine if the target MTA supports this 
kind of authorization, even if the current recipient has not been 
authorized yet. In the latter case the current message should remain in 
the queue (or be rejected with a 4xx response) until the required 
authorization is granted. If the target MTA does not support this 
solution, messages may still be forwarded using an SPF-compliant sender 
replacement, as ruled by the local policy. A sender replacement is needed 
anyway when the final recipient's addresses are not to be disclosed to the 
message originator.

Authorizations should be granted automatically by targets MTAs. Different 
sites may implement different strategies in order to check a specific 
relay is trustworthy, which may imply manual operations. After a relay has 
been registered, in case it needs to, it should be able to obtain 
authorizations for forwarding to each new user in a fully automated fashion.

A forwarder may provide some capabilities, such as querying some DNSBLs, 
performing SPF checks and similar. Zero or more of the available 
capabilities may be enabled for each specific recipient. After it has been 
whitelisted, a forwarder will not be deemed responsible for relayed spam. 
If both hosts support spam reporting, that activity may be coordinated by 
sending spam reports back to the forwarder when they belong there.

In return for a forwarding authorization, a forwarder grants the final 
recipient permissions to directly inquiry, update, and delete the relevant 
record of its local forwarding policy. To delete here means to not forward 
any further message, possibly responding "551 user not local". That way a 
recipient can control and recurse through a chain of forwarders, e.g. via 
web services. Update and delete actions should be handled differently 
according to the kind of alias to alias-expansion relationship (1-1 or 
1-many).

The target MTA also keeps track of the authorizations it granted. For each 
authorization, some properties may be relevant, e.g. if the forwarding 
happens after subscribing to a list, if the address has been 
double-checked, what was the requesting IP, if an alias is meant for 
concealing its expansion rather than as an address update, et cetera. The 
  access credentials gained from forwarders are also stored, keyed on both 
ip or helo name of the forwarder, and recipient address. That would 
provide an opportunity for verifying opt-in data and to enable end users 
to maintain their forwarding chains.

More or less, that's it. I've proposed the above description to a 
postmasters' mailing list, seeking for practical advice. I got almost no 
response. Why? Perhaps this solution is too simple, or too complicated, or 
there are obvious reasons why it won't fly. Would anyone on this list be 
able to spot them, please?

TIA
Ale

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Douglas Otis <[email-masked]>
Date: Wed, 30 Jan 2008 11:51:04 -0800

On Jan 29, 2008, at 10:34 PM, Alessandro Vesely wrote:

> Hi all,
> it is well known how SPF breaks forwarding. SPF is not itself an  
> anti-spam mechanism, but it would help cleaning up the mail  
> transport system by limiting domain abuse. If it were widely  
> employed, that is. Thus, this post is not off topic, although not  
> directly aimed at countering spam.

SPF's use of macros represents a serious security issue in itself.   
This allows traffic to be generated from cached records triggered by  
changes in a sender's email-address found within spam campaigns.   
Processing SPF against unknown domains increases the magnitude of this  
threat.

> message is being forwarded (i.e. it already has a Return-Path), the  
> mailer should look up the policy in order to determine what envelope  
> sender replacement, if any, it should use.

This reminds me of the Monty Python movie Jabberwocky central  
character Dennis Cooper's suggestion "It would increase your  
efficiency if you moved your box to here."  It resulted in chaos.  A  
chain of hashed local-parts translated locally into an original return- 
path address would need to include other elements of the message to  
prevent abuse.

Another solution using this concept and not depending upon a dangerous  
and unmanageable SPF path registration can be found here:
http://tools.ietf.org/wg/dkim/draft-otis-dkim-tpa-ssp-02.txt

This allows the RFC 2821 MAIL FROM (return-path) to reference a Third- 
Party Authorization against a specific domain in question.  In this  
case, it would be the signing domain.  In essence, this strategy  
overcomes the SPF forwarding issue and only require a single  
transaction.  SPF on the other hand might require orders of magnitude  
more (which could also be new queries emanating from cached records).

It seems rather ironic SPF's intended purpose was to direct  
culpability to the provider's customer (the email-address owner).  By  
limiting DKIM's delegation to being an exchange of keys, once again  
the customer rather than the provider is again held culpable.  The  
exchange of DKIM keys, especially where a common key might sign for  
thousands of different domains, creates a significant risk as well.

In addition, schemes directing culpability toward provider's customers  
are in conflict with the general protection of personal privacy.  Only  
the provider should be able to determine a message source, and  
therefore only the provider should be held responsible for controlling  
abuse.

P.S.
While spam is increasing in volume, the life span of dedicated anti- 
spam systems grow ever shorter, it seems pronouncements of "Mission  
Accomplished" might be a bit premature.  Nevertheless, it also  
apparent those able to influence the structure of today's  
communications are not interested in actually controlling abuse.   
Rather interests are focused on deflecting complaints and establishing  
elite services.  In other words, the market's hidden hand ensures a  
workable solution for controlling abuse remains unreachable.  So close  
the ASRG and cue the calliope, the circus rolls on. : )

Fatigue and "market" forces rather than accomplishment might be an  
epitaph for waning interest in ASRG.  At least John Levine's  
contributions to this noble cause can still be found elsewhere.

Although we may not always agree about what is possible, practical, or  
politically wise, John's perspective is always well worth  
consideration.  Thanks John.

-Doug

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 30 Jan 2008 21:41:48 +0100

Douglas Otis wrote:

> culpability to the provider's customer (the email-address owner).

The "owners" of a reverse path are the hops adding info to it,
today in essence limited to the envelope sender address as
accepted by the MSA.  
 
> In addition, schemes directing culpability toward provider's
> customers are in conflict with the general protection of personal
> privacy.

There is no such thing as "culpability" of senders in SPF.  If
folks want it they can arrange for a working envelope sender
address based on their Message-ID or using BATV, but that has
nothing at all to do with privacy.

> Only the provider should be able to determine a message source,
> and therefore only the provider should be held responsible for
> controlling abuse.

The provider is not responsible for forgeries by third parties.
SPF only allows to identify plausible (PASS) or forged (FAIL)
envelope sender addresses for domains publishing an SPF policy.

 Frank

---

From: Douglas Otis <[email-masked]>
Date: Wed, 30 Jan 2008 13:43:49 -0800

On Jan 30, 2008, at 12:41 PM, Frank Ellermann wrote:

> The "owners" of a reverse path are the hops adding info to it, today  
> in essence limited to the envelope sender address as accepted by the  
> MSA.

Owners of an email-address are not owners of the additive hops (the  
provider's addresses in the case of SPF).  While SPF might be applied  
against the envelope sender address (the return-path), these records  
may also be applied against the Purported Responsible Addresses  
representing another attempt at identifying the provider's customer.   
The difference between the provider and the provider's customer is  
extremely important.  When access depends upon an identity's indirect  
declaration of their authorized providers by way of address, privacy  
protection is clearly reduced.

> There is no such thing as "culpability" of senders in SPF.  If folks  
> want it they can arrange for a working envelope sender address based  
> on their Message-ID or using BATV, but that has nothing at all to do  
> with privacy.

When access depends upon an identity's declaration of authorized  
providers, the means for making this declaration resolves to the  
provider's customer, and not the provider.

>> Only the provider should be able to determine a message source, and  
>> therefore only the provider should be held responsible for  
>> controlling abuse.
>
> The provider is not responsible for forgeries by third parties. SPF  
> only allows to identify plausible (PASS) or forged (FAIL) envelope  
> sender addresses for domains publishing an SPF policy.

You just said that SPF does not hold senders culpable, and yet SPF  
senders are required to identify themselves by way of their  
declaration of authorized providers?  Why is the provider ignored?

There are perhaps a few hundred thousand major providers, and yet  
there are millions of individual's email domains in use.  SMTP client  
validation within a single transaction could eliminate far more abuse  
than SPF.  EHLO validation is yet another optional "feature" of SPF  
that _might_ be accomplished after a dozen or so DNS transactions.   
Unfortunately, SPF suffers from having too many "features" keeping  
this feature from being practical.  How convenient. : )

-Doug

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 31 Jan 2008 00:15:01 +0100

Douglas Otis wrote:
  
> these records may also be applied against the Purported Responsible
> Addresses

Receivers can also apply them to the Message-ID and use the results
as entropy source for /dev/random, but apart from a NOT RECOMMENDED
statement in RFC 4408 this is unrelated to SPF.

It's also unrelated to "forwarding" (see subject), and how RFC 821
made sure that "forwarders" must take the responsibility for their
actions (= by adding their host to the reverse path).

> extremely important.

Not for the reverse path, this was just a route to a mailbox of the
"sender", a sender@host mailbox for MAIL FROM this host.

> When access depends upon an identity's indirect declaration of
> their authorized providers by way of address, privacy protection
> is clearly reduced.

The envelope sender address is used for non-delivery reports, FWIW
it can be postmaster@host or postmaster+crypto@host or crypto@host,
this has nothing to do with "privacy protection".  If somebody can't
receive mail he has no business to send mail, that's as it always
was, and if you want real anonymity this needs considerably more
effort than using a forged envelope sender (or 2822 PRA) address.

> provider's customer, and not the provider.

Nobody is forced to declare which IPs they consider as permitted
or forbidden for mails sent by them.

And if they do you can still use "your" 2822-From address where it
pleases you as far as SPF is concerned.  This won't work for PRA
or SSP, but that's not my problem while talking about SPF and SMTP.
 
> there are millions of individual's email domains in use.

Yes, they can combine providers as they see fit in their policy,
or never care about it if they have no problems with backscatter.

> EHLO validation is yet another optional "feature" of SPF

Nope, it's RECOMMENDED, not "optional".  Does SPF really threaten
your commercial "anti-spam" interests so much ?  *Brilliant*

 Frank

---

From: Douglas Otis <[email-masked]>
Date: Wed, 30 Jan 2008 17:59:04 -0800

> actions (= by adding their host to the reverse path).

SPF advocates decided _not_ to adopt the revised record format that  
specifies scope.  Scope clarifies applicability without a separately  
published declaration specific to Sender-ID.  Most don't, as Sender-ID  
declares separate publication to be unnecessary.  Concerns seemed  
focused upon how adoption of revised records would be perceived, and  
whether Sender-ID might then assert a leadership role.  As it is now,  
Sender-ID declares older SPF versions to be a valid a means to  
indirectly authorize providers based upon the PRA, SPF recommendations  
not withstanding.

> "sender", a sender@host mailbox for MAIL FROM this host.

This still represents requirements for the provider's customers which  
might extend to the PRA, your assurances not withstanding.

>> their authorized providers by way of address, privacy protection is  
>> clearly reduced.
>

SPF records will never be revised to include scope, obsolete dangerous  
macros, or segregate IPv4 or IPv6 record sets.  Restrictions based  
upon the PRA and MAIL FROM are being done in lieu of requirements  
based upon a provider's stewardship.  Well managed MTA should permit  
their customer's latitude in the choice of addresses.  Call back  
validation represents a similar menace.  SPF specifically sets out to  
restrict choice.  Restrictions placed upon customers are better at  
diverting complaints than controlling abuse.

> Nobody is forced to declare which IPs they consider as permitted or  
> forbidden for mails sent by them.

This depends upon acceptance requirements.

> pleases you as far as SPF is concerned.  This won't work for PRA or  
> SSP, but that's not my problem while talking about SPF and SMTP.

This remains a problem that SPF failed to confront, your efforts to  
sweep this under the rug not withstanding.

> Yes, they can combine providers as they see fit in their policy, or  
> never care about it if they have no problems with backscatter.

This assumes SPF records are used in an innocuous manner.

> Nope, it's RECOMMENDED, not "optional".

Okay, but would there be a need to recommend something that is  
required?  SPF's sizeable overhead and dubious results doesn't help in  
this case either.

> Does SPF really threaten your commercial "anti-spam" interests so  
> much ?  *Brilliant*

Actually, SPF threatens all services and is not specific to "anti- 
spam".  DDoS remains a real concern imperilled by poorly considered  
protocols (SPF should not have been published, even as experimental  
without significant changes IMHO).

Making a difference is not impossible, and there is an effective means  
that affords greater freedom.  Hold providers accountable.  Require  
SMTP clients to confirm their domain's authorization.  This removes  
questions related to which individual is sending the message.  There  
is S/MIME and OpenPGP when identifying an individual is important.

Exchanging DKIM keys with providers represents another bad practice.   
Authorizations should be done by name, not address or key exchange.   
In addition, authorizations should not depend upon the identity used  
by the individual sending the message.  With respect to DKIM, the i=  
parameter should be allowed to be opaque or anything at all, even when  
a provider asserts "ALL" or "STRICT" signing policies.  Controlling  
abuse in an effective manner is the best way to eliminate a need for  
commercial anti-spam efforts.  There are better things to do, after  
all. : )

-Doug

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 31 Jan 2008 05:31:24 +0100

Douglas Otis wrote:

> SPF advocates decided _not_ to adopt the revised record format
> that specifies scope.

- There never was a scope in v=spf1, it was always about SMTP.
- There was apparent consensus that "spf2.0" cannot assimilate
  a different + existing + older v=spf1, and no MARID draft did.
- There was no consensus that PRA makes sense, therefore one of
  the MARID editors was asked to produce an "mfrom" draft.
- Five days after he posted it MARID was terminated without any
  discussion with the WG (in violence of RFC 2418 section 4). 
- After MARID the "mfrom" author returned to v=spf1 and updated
  it with the only tangible MARID result, an unambiguous ABNF.
- Later a version with more elaborated processing limits etc.
  by another author became RFC 4408.
- The PRA draft trying to assimilate v=spf1 (after MARID) was
  appealed twice at the IESG and once at the IAB, resulting in
  one of the longest "IESG notes" I have ever seen, stating
  among other things that PRA (as is) will be never a standard,
  because it is incompatible with existing Internet standards. 
- The IAB decided that conflicting experiments are no problem.
- PRA was never more than a footnote in history, their attempt
  to "steal" the installed v=spf1 base was a miserable failure.

http://www.openspf.org/blobs/spf-community-position.html
http://en.wikipedia.org/wiki/MARID
http://purl.net/xyzzy/home/test/senderid-appeal.htm

Of course the IETF was never asked what it thinks, as there
was never an IETF Last Call for SPF as requested.  The folks
pushing SenderID wanted *no* Last Call, and they always got
what they wanted.  Down to a clandestine attempt to remove
the critical NOT RECOMMENDED in 4408 by an RFC editor note.

Are we ready with this disgrace in the history of the IETF ?

> SPF records will never be revised to include scope

SPF does anything that's possible in SMTP given a connecting
IP, an EHLO, and a MAIL FROM, no "scopes" needed.  And FWIW
PRA does anything that is possible with an 2822 header after
stupidly ignoring the MAIL FROM, also no "scopes" needed.

A theoretically possible "Message-ID scope" is in practice
nonsense, nobody posted an spf2.0/mid (or similar) draft...

...go for it if you like the idea, maybe it is better than
the DKIM-SSP concept to bind author addresses (28822-From).  

SPF does not try to dictate a common (working for everybody)
interpretation of any header fields *in* the DATA, we'll see
how far DKIM-SSP gets with its attempt.

 Frank

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 31 Jan 2008 12:00:24 +0100

Douglas Otis wrote:
> SPF's use of macros represents a serious security issue in itself.  This 
> allows traffic to be generated from cached records triggered by changes 
> in a sender's email-address found within spam campaigns.  Processing SPF 
> against unknown domains increases the magnitude of this threat.

Still, from the limited point of view of running my own MTA, I find SPF 
checks cheaper than content filtering and verifying signatures, as the 
latter can only be done after the message body has been received. DNSBLs 
work much like it, but are one order of magnitude more efficient, for the 
time being.

> This reminds me of the Monty Python movie Jabberwocky central character 
> Dennis Cooper's suggestion "It would increase your efficiency if you 
> moved your box to here."  It resulted in chaos.  A chain of hashed 
> local-parts translated locally into an original return-path address 
> would need to include other elements of the message to prevent abuse.

That looks exactly like the kind of criticism I have been looking for, 
thanks! However, I don't think I've got it quite rightly (perhaps because 
I missed the movie.) I imagine a local forwarding policy as a database 
keyed on <forwarder, envelope recipient> pairs; where "forwarder" is 
either the IP address or its FQDN, provided it got an SPF "pass".

The receiving MTA already has a database of recipients, and deploying this 
technique is going to multiply that by a number of forwarders. Likewise, 
the sending MTA must have a similar database, and a forwarding directive 
attached to the address that triggered the relay. That looks manageable to 
me, if not flooded by spammers.

A spammer can pretend to forward a message. If the receiving MTA does not 
limit which hosts can receive an authorization for being whitelisted, the 
spammer can succeed. That is not much different than directly sending the 
spam, except that it adds one record to the local policy. If the receiving 
MTA can determine that the spammer is spamming directly rather than 
relying spam, then it may put a blacklist flag on that record, thereby 
nullifying the spammer's attempt to set up an acknowledged relay. However, 
  it is not easy for the receiving MTA to make that determination. Is that 
the abuse problem you mean?

> This allows the RFC 2821 MAIL FROM (return-path) to reference a 
> Third-Party Authorization against a specific domain in question.

The spammer probably won't sign its return-path. A forwarder advertising 
the ability to sign its return-paths could get authorizations more easily, 
though.

> It seems rather ironic SPF's intended purpose was to direct culpability 
> to the provider's customer (the email-address owner).

I don't want to delve into the discussion you had with Frank while I was 
sleeping. I just want to note that provider and customer must trust each 
other.

> epitaph for waning interest in ASRG.

We'll never have a better world if we don't try and design it...

>  At least John Levine's 

Any specific pointer? I just read some of his posts on the CircleId.

---

From: Douglas Otis <[email-masked]>
Date: Thu, 31 Jan 2008 11:33:54 -0800

On Jan 31, 2008, at 3:00 AM, Alessandro Vesely wrote:

>> This allows traffic to be generated from cached records triggered  
>> by changes in a sender's email-address found within spam  
>> campaigns.  Processing SPF against unknown domains increases the  
>> magnitude of this threat.
>
> Still, from the limited point of view of running my own MTA, I find  
> SPF checks cheaper than content filtering and verifying signatures,  
> as the latter can only be done after the message body has been  
> received. DNSBLs work much like it, but are one order of magnitude  
> more efficient, for the time being.

DDoS attack damages may run into the millions per event.  The concept  
of "cheaper" should be weight against this _very_ serious risk.   
Content filter is doomed, as bad actors always have the advantage.   
Something like CSV is far safer and likely just as effective.   
Adopting RFC 3464 would help with DSN as well, especially where  
original message content is redacted.

> That looks exactly like the kind of criticism I have been looking  
> for, thanks! However, I don't think I've got it quite rightly  
> (perhaps because I missed the movie.) I imagine a local forwarding  
> policy as a database keyed on <forwarder, envelope recipient> pairs;  
> where "forwarder" is either the IP address or its FQDN, provided it  
> got an SPF "pass".

Expecting a pass will be problematic.  A translation process will need  
to qualify DSNs, not the message being forwarded.  BATV is an example  
of how this might be done, but this imposes a requirement that the  
domain's email uses a common Return Path protection strategy.  That,  
in it self, can be a problem.

>> Another solution using this concept and not depending upon a  
>> dangerous and unmanageable SPF path registration can be found here:
> The spammer probably won't sign its return-path. A forwarder  
> advertising the ability to sign its return-paths could get  
> authorizations more easily, though.

The concept was to improve the integrity of email's delivery process  
that is currently suffering due to a high loss of DSNs.  DKIM in  
conjunction with TPA-SSP would allow signatures to be associated with  
the desired RFC 2821 MailFrom.  TPA-SSP allows this association to be  
made autonomously by just email-domain owner.  DKIM key exchanges  
would be far more complex, costly, and risky.

> I don't want to delve into the discussion you had with Frank while I  
> was sleeping. I just want to note that provider and customer must  
> trust each other.

Agreed.  But trusted implies specific expectations as well.

Using DKIM, a provider might sign email with a domain of  
"authenicated.provider.com" where any of their  
"authenticated" (control of the associated mailbox has been verified  
for example), could then autonomously use TPA-SSP to specifically  
authorize "authenicated.provider.com".  The provider could sign all  
their customers using a common signature, and apply the same criteria  
for all those permitted to access that specific domain.  The  
provider's own domain might authorize both  
"authenticated.provider.com" "not-authenticated.provider.com" and  
select the signing domain based upon the "authentication" status of  
the message.

> We'll never have a better world if we don't try and design it...

Agreed.  But there are also market forces that stand in the way of a  
good design.

>> At least John Levine's contributions to this noble cause can still  
>> be found elsewhere.
>

Here is one.

http://mipassoc.org/batv/draft-levine-batv-03.txt

-Doug

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 31 Jan 2008 21:18:53 +0100

> Any specific pointer? I just read some of his posts on the CircleId.

The ASRG WIKI, the DNSBL draft, abuse.net and its mailing list,
among many tons of other things.  John is the remaining Chair,
the co-Chair resigned back in 2004 one day after "Caller ID" was
added to the new IETF MARID WG agenda, and one day before I
submitted an "IPR report" about it. 

 I had not the faintest idea what I am doing, I tried to follow
 vague instructions in the boilerplate of the "Caller ID" draft,
 because I knew it's patented and that "XML over DNS" is weird ;-)

The, hm, "real fun" (?) on this list was before MARID, when they
had hot flamewars about RMX, SPF, ..., LMAP (a famous ASRG draft),
I missed all of it.  After some weeks here I found the SPF list,
and again some weeks later I figured that SPF is not really RMX,
but at that time SPF was already the only game in town.  <shrug />

 Frank

_______________________________________________
Asrg mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/asrg