Subject: Re: [Asrg] where the message originated (was: DKIM role?) (SM)

From: Ian Eiloart <[email-masked]>
Date: Thu, 15 Jan 2009 14:57:59 +0000

--On 12 January 2009 12:58:28 -0500 der Mouse <[email-masked]> 
wrote:

>> However, I do think that it's time that domain owners started taking
>> responsibility for the use of their domains.
>
> If I, say, were to forge mail from your domain to, say, some ibm.com
> user, would you take responsibility for it?  That's what you seem to me
> to be saying.

Not quite, but not far off. I'm not taking responsibility for your decision 
to forge the domain. However, I AM responsible for the fact that IBM have 
no easy way to detect that forgery. I SHOULD accept that IBM are therefore 
more likely to bounce that email back to me.

    When a spammer forges my domain, and the recipient bounces the message,
    they're doing what SMTP is designed to do. I shouldn't get pissed off 
with
    the recipient, if I've failed to publish records that allow them to
    check for forgeries.

What I should take responsibility for is publishing records that let IBM 
easily check whether the email is legitimate. For example, if I publish SPF 
records, and sign all my outgoing mail with DKIM, and require my users to 
use my Message Submission Service, then I think I've discharged that 
responsibility.

Having taken those actions (actually, I haven't but this is hypthetical), 
I'm quite happy to accept any bounce messages from IBM - provided they've 
checked for an SPF or DKIM record match before accepting the message.

Now, given that I haven't yet published SPF or DKIM records, I still get 
lots of spam bounced into my domain. Who do I blame? Well, in part it's my 
responsibility for failing to help the spam recipients. If I did publish 
those records, then recipient domains could more easily detect forgeries 
out of my domain.

SPF records would be useful because you can check them early, and cheaply. 
DKIM give senders the opportunity to get messages through forwarders.

Now, what I'm suggesting (not advocating yet, because I'm not certain that 
this is right), is that when there's no SPF record published, that we 
should not feel too bad about bouncing email because the domain 
administrator isn't taking adequate steps to protect the domain against 
spam blowback, and against phishing. Of course, I'm NOT suggesting that 
lack of an SPF record should score very high in any any spamicity measure, 
but it might count for something.

Now, an SPF or DKIM match gives us the huge gain that we can bounce 
messages selectively based on the content. Some recipients may not want 
certain message content, but by the time we've seen the content SMTP 
doesn't allow us to selectively reject. And, a pass allows us to 
confidently whitelist a domain or sender address without opening up a huge 
hole in our spam defences. For example, I've refused to whitelist UK 
academic research boards because they don't publish SPF records.

Now, the question is, what do we do when we see an SPF softfail, and no 
DKIM record? Well, at first I'd be reluctant to bounce the message, because 
I want to reward admins that are taking steps to protect their domains. 
However, that's not something that I'd be happy with on a permanent basis. 
Eventually, I want the world to be in a position where all forwarders check 
SPF on the way in, and rewrite sender domains. When we're sufficiently 
advanced in that project, then bouncing softfail would be more acceptable.

No SPF record:
SPF fail: reject the message at SMTP time.
SPF softfail: check DKIM
SPF pass: accept if you trust the domain, otherwise do other spam checks. 
You can reject, bounce or accept depending on other spam checks. 
Importantly, you can bounce PER RECIPIENT depending on the content, which 
you can't do with SMTP rejections.

>
>> In a world where you *can* prevent abuse of your domain, what's wrong
>> with bouncing email?
>
> Depends on what that prevention requires.  ("Sure you can prevent abuse
> of your domain.  You just need to enter into a bilateral agreement with
> every receiving mailhost on the net.")

No, you just need to sign your outgoing email, and publish SPF records.

I know, someone's going to tell me that SPF breaks forwarding. Well email 
is already broken. I get up to a million spam messages into my domain 
daily. Everything we do to prevent spam causes problems ranging from 
additional load on our DNS servers to loss of important emails. The 
question is, are we going to fix it? I think what I'm proposing would at 
least allow people to do safe whitelisting

-- 
Ian Eiloart
IT Services, University of Sussex
x3148
_______________________________________________
Asrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/asrg

---

From: David Wilson <[email-masked]>
Date: Thu, 15 Jan 2009 16:13:20 +0000

> No, you just need to sign your outgoing email, and publish SPF

And make sure that you send from hosts which are permitted by your SPF
info.

At one point I purchased on-line from Maplin, so they send me marketing
messages. maplin.co.uk has SPF information, but these messages are not
sent from one of the permitted hosts!

I have also seen an SPF failure on a message from a friend who has a
'vanity' domain for their family. The message came via their ISP's MTA,
and so failed as this is not permitted by their SPF info. In this case
the SPF info has probably been created by the domain registration
organization.

---

From: Rich Kulawiec <[email-masked]>
Date: Thu, 15 Jan 2009 11:17:16 -0500

On Thu, Jan 15, 2009 at 02:57:59PM +0000, Ian Eiloart wrote:
> lots of spam bounced into my domain. Who do I blame? 

Oh, that's easy.  You blame the people running badly-broken mail systems
that bounce instead of rejecting.  Their IP addresses are contained
in your logs and in the headers of the backscatter you're getting.

Fixing this problem is a straightforward exercise modulo a few edge
cases -- but the practical impact of those edge cases can be minimized
by judicious use of simple tools and procedures (e.g., cron, make, rsync).

(Example: one of the common problems of this type comes up in multi-tier
mail systems lacking LDAP or another directory service.  The problem
arises when an address becomes invalid on the internal system but is
still considered valid on the external MXs.  One easy solution to
this is to use a script to grab the current-list-of-valid-addresses
from the internal system at intervals, process it, and then rsync
it out to the external MX's.  This reduces possible backscatter due
to this problem to "the set of recently-removed addresses" and further
to "the subset of those receiving forged-sender mail during the
interval between updates".  It's thus possible to reduce the
probability of backscatter arbitrarily by reducing the interval.)

---Rsk

---

From: Rich Kulawiec <[email-masked]>
Date: Thu, 15 Jan 2009 18:45:18 -0500

On Thu, Jan 15, 2009 at 04:13:20PM +0000, David Wilson wrote:
> info:

This still isn't enough to prevent abuse of your domain.  Let's make
three assumptions, two of which (I think) you're advocating:

	1. Lots and lots of hosts that are backscattering today don't
	fix the fundamental problem (that is, they continue to allow
	their systems to bounce instead of rejecting).  Given how
	many mail system operators don't even know what backscatter,
	aka outscatter, is, and that some have openly declared themselves
	"required" to send it, this seems better-than-plausible.

	2. Lots and lots of hosts adopt something like SPF.

	3. You -- the keeper of example.com -- set up something like SPF.

	(Don't take any of this as an endorsement of SPF.)

Let's press on and have some fun with example.com, by way of
illustrating what I said above:

	A. I register throwaway domain a1s2d3f5g6h7.info.

	B. I arrange to have the domain hosted somewhere with, shall we
	say, less than stringent oversight.
	
	C. I set up DNS, including SPF, for a1s2d3f5g6h7.info.

	D. I send out a kazillion messages from [email-masked],
	all of which will of course pass SPF checks.  They're addressed
	to a kazillion addresses at a kazillion domains.  Maybe some
	have malware attached.  Maybe some have boilerplate spam content.
	Maybe some are addressed to probably-don't-exist addresses.

	E. Some of the receiving MX's will reject.  Some will try to bounce.

	F. And this is where step (C) comes in, because I didn't point
	the MX records for a1s2d3f5g6h7.info at my own server: I pointed
	them at example.com's MXs.  That's where the bounces will try to go.

	Oh, sure, the MX's for example.com are rather unlikely to accept
	any mail for a1s2d3f5g6h7.info, because of course they're not
	configured to, but how do you think those MX's will react when
	a kazillion mail systems (nearly) simultaneously all try to
	open up connections on port 25 and start shoving mail at them?

	G. Since I did all this using someone else's credit card, money
	is really no object.  Repeat steps (A) - (D) several hundred
	times using diverse registrars, DNS services, hosting services,
	outbound message targets, etc.

	I can make this more interesting with some optional steps:
	use of botnets instead of host service, use of fast-flux DNS,
	pre-selection of hosts known to backscatter for (D), and/or
	making the putative sender of all these messages
	[email-masked], which will confuse
	the hell out of a significant subset of people.

	Yes, this behavior will probably get each one of those domains
	and originating IP addresses of mine noticed and listed on various
	DNSBLs and RHSBLs relatively quickly, not that this will be
	much consolation to example.com while they're recovering
	from the DDoS attack.

This is one of the more benign possible scenarios.  And yes, there
are all kinds of ways to counter this particular one, but my general
point is that anything that lets third parties cause someone's mail
system to emit outbound SMTP traffic to arbitrary destinations is a
cause for concern...which is why I think one of the easiest and most
productive things we could and should be doing is getting our mail
systems set up to always reject and never bounce -- or as close as
we can, modulo edge cases I've discussed elsewhere in the thread.
That doesn't really "fix" anything, per se, but at least it minimizes
ensuing secondary and tertiary consequences, and avoids adding still
more SMTP traffic to the excess we already deal with.

---Rsk

---

From: David Wilson <[email-masked]>
Date: Fri, 16 Jan 2009 08:32:48 +0000

On Thu, 2009-01-15 at 18:45 -0500, Rich Kulawiec wrote:
> This still isn't enough to prevent abuse of your domain. 

My point was not that this would prevent abuse of your domain. Rather
that it is stupid to register information which is intended to go some
way to prevent domain abuse, and then send messages which appear to be
such abuse.

---

From: Rich Kulawiec <[email-masked]>
Date: Fri, 16 Jan 2009 09:14:18 -0500

> way to prevent domain abuse, and then send messages which appear to be

An SMTP reject is not a mail message.

But more broadly, you could have stopped after the first clause:
*nothing* you register will prevent domain abuse if someone (or something)
is determined to inflict it, since whatever you've done can simply be
ignored (and probably will be).  Besides, domain abuse is not a systemic,
persistent problem; spam, including backscatter/outscatter as one of
its variants, is, which is why (I believe) this RG exists.

That said, one of the best, easiest, and cheapest ways to gain free
assistance with abuse of one's domain is to maintain the "abuse" address,
per RFC 2142 (which dates from 1997).  I'm often struck by what byzantine
schemes are proposed (for abuse control) while this simple measure goes
almost entirely overlooked.  (Note for example yesterday's traffic on
Dave Farber's excellent "IP" list in re the difficulties of contacting
Facebook's abuse/fraud group.)

---Rsk

---

From: David Wilson <[email-masked]>
Date: Fri, 16 Jan 2009 16:34:04 +0000

> its variants, is, which is why (I believe) this RG exists.

I'm not sure where I was promoting anything. Ian wrote about SPF and
DKIM. I gave a couple of example of domain for which SPF information had
been registered, but the owners of the domain were going against their
own action. (In one case it was the domain owner's agent who probably
added the SPF information). I said nothing about the value or otherwise
of SPF for domain abuse.

Having said that, since we registered SPF information for our domain,
the amount of backscatter seems to have been dramatically reduced. I'm
talking about certainly a factor of 10, and perhaps much higher (it
would be some work to verify this). I don't know if this is because:

- receivers are using SPF information to reject at, say, MAIL FROM, or
otherwise 
- spammers avoid forging return paths with domains with SPF info

Roughly 15% of message from the outside world which we receive are
suffering SPF failures (about 3:1 soft fails to hard fails, and about
15% of the failures are for our own domain, typically as a result of
using the same address for originator and recipient).

So, I suspect that the first of the reasons is playing a significant
part in reducing backscatter.

---

From: Ian Eiloart <[email-masked]>
Date: Mon, 19 Jan 2009 11:16:19 +0000

> sent from one of the permitted hosts!

And, right now, they don't much care about that. Why? Because too few 
people respect SPF records. I'll bet that if, say, a couple of the major 
email service providers, or all .gov or .gov.uk addresses started to do so, 
then Maplin would start to get that right.

> the SPF info has probably been created by the domain registration

They'll need to get their SPF records right. We probably need better tools 
to enable this.

---

From: Ian Eiloart <[email-masked]>
Date: Mon, 19 Jan 2009 11:33:56 +0000

> 	"required" to send it, this seems better-than-plausible.

True. I think there's a real need for serious education in how to manage 
email systems. I don't know of any formal course or qualification that you 
can take. Peer pressure can be effective, though. We've seen, for example, 
the end of open relays. We've also seen a lot of providers take up MSA - 
which is also essential to the plan.

> 	2. Lots and lots of hosts adopt something like SPF.

Yep.

> 	3. You -- the keeper of example.com -- set up something like SPF.

Yep.

> 	(Don't take any of this as an endorsement of SPF.)

OK :)

> illustrating what I said above:

There's no point to what you describe below, except to create a DoS attack 
on the MX host. Why? Because the messages won't get delivered to sites that 
use sender verification callouts. The callouts should be acceptable when 
there's an SPF pass for domains that are set up properly. Callouts are 
typically cached, which should mitigate the attack. It could be further 
mitigated by use of 5.1.2 replies "Bad destination system address", which 
tells the caller that the domain is incorrect. That means that the DoS 
attack is only good for a single message from any given MTA, you don't have 
significant amplification. In fact, even without any of these measures, my 
server isn't going to amplify your attack, because I'll only bounce any 
message once.

So, spammers should not be able to use this mechanism to deliver email. The 
DoS attack is an issue. Perhaps "reverse" MX records should be published, 
too. So, I can look up an IP address and discover whether it will accept 
email for a particular domain.

Why go to all that trouble when you've already got a botnet that's capable 
of doing the damage directly?

---

From: Ian Eiloart <[email-masked]>
Date: Mon, 19 Jan 2009 11:40:13 +0000

> its variants, is, which is why (I believe) this RG exists.

The problem is with spam. However, the natural response of email users is 
to say "can you block (or whitelist) email from such and such a domain"? 
Currently there's little point because it's so hard to determine whether 
the sender address is a forgery.

I'd love to whitelist all .ac.uk addresses, because we do a lot of critical 
business with users of those domains, they can't be freely registered, and 
I know how to contact administrators of those domains. And, I've been asked 
to do so. Spammers would love me to do that, too.

> Facebook's abuse/fraud group.)

Agreed. Of course abuse and postmaster email addresses can be quite hard to 
monitor, since they can get a lot of spam.

---

From: Ian Eiloart <[email-masked]>
Date: Mon, 19 Jan 2009 11:52:20 +0000

> in your logs and in the headers of the backscatter you're getting.

Well, that's the traditional answer. However, I don't now think that it's 
reasonable. I *could* be publishing SPF records to tell them that the email 
isn't from me, but I'm not. I MUST shoulder some of the responsibility.

---

From: Rich Kulawiec <[email-masked]>
Date: Mon, 19 Jan 2009 07:06:11 -0500

On Mon, Jan 19, 2009 at 11:33:56AM +0000, Ian Eiloart wrote:
> There's no point to what you describe below, except to create a DoS 
> attack on the MX host. Why? Because the messages won't get delivered to 
> sites that use sender verification callouts. 

We can stop right there.  It's fairly well known that such callouts
are inherently abusive, and should never be used.  Please see the copious
discussion in the archives of spam-l, where this has been covered extensively.

---Rsk

---

From: Rich Kulawiec <[email-masked]>
Date: Mon, 19 Jan 2009 07:28:42 -0500

On Mon, Jan 19, 2009 at 11:16:19AM +0000, Ian Eiloart wrote:
> email service providers, or all .gov or .gov.uk addresses started to do 
> so, then Maplin would start to get that right.

You seem to think that SPF is a good idea.  It's not.  There's a reason
why the earliest and most prolific adopters were spammers, and it's
most certainly not because it stops spam.  There's also a reason why
it's been abandoned by some rather large operations who rolled it out at
some point in the past.

The days when sending bounces was acceptable are fading into the distance.
Attempts to preserve them are doomed by the ability of bad actors to
redirect that traffic to destinations of their choosing and thus to
enlist bystanders in DoS and other attacks.  I think in part this is
because the community is slowly realizing a general principle: anything
that allows third parties to cause them to generate outbound traffic to
arbitrary destinations is a bad idea.  (This applies to much more than
mail, but since that's our focus here...)

SPF belongs on the same scrap heap with C/R, SAV, e-postage, and other
pre-failed ideas that are one or more of (a) abusive (b) ineffective
(c) unworkable or (d) attack vectors.  And I think we've already extracted
from it all the lessons we're going to learn about *why* it's a failure,
so let's move on to something -- anything -- that might have a chance.
Whatever-it-is might also turn out to be worthless, but at least we may
learn something new from it.

---Rsk

---

From: Ian Eiloart <[email-masked]>
Date: Mon, 19 Jan 2009 13:02:49 +0000

--On 19 January 2009 07:28:42 -0500 Rich Kulawiec <[email-masked]> wrote:

> most certainly not because it stops spam.

It's because it increases deliverability of email, but that's 
indiscriminate and needs to be combined with an effective reputation 
service for email domains. The reputation service would make spf less 
attractive for spammers, and more attractive for the rest of us. SPF on 
it's own does little. SPF + DKIM a good reputation management system could 
achieve a lot.

>  There's also a reason why

> The days when sending bounces was acceptable are fading into the distance.

Yes, and that's a bad thing. It's bad that spammers have made an effective 
notification system virtually unusable. However, it's the spam that's the 
root problem. We shouldn't be throwing the baby out with the bathwater, we 
should be trying to reclaim SMTP so that we can use it as it was designed.

That means getting effective accountability into the system. Certainly, if 
we were starting from scratch, then SPF would be a part of the system, and 
forwarding mechanisms would have accounted for SPF. That's where we need to 
get to. The question is how do we get there from here?

---

From: Rich Kulawiec <[email-masked]>
Date: Mon, 19 Jan 2009 09:44:05 -0500

On Mon, Jan 19, 2009 at 01:02:49PM +0000, Ian Eiloart wrote:
> Yes, and that's a bad thing. It's bad that spammers have made an 
> effective notification system virtually unusable. 

I don't see a problem here that needs solving.  Rejects are far more
efficient, and have two huge advantages: (a) they don't generate
more largely-worthless SMTP traffic at a time when we're drowning in
largely-worthless SMTP traffic (b) they're much harder to co-opt
for abusive purposes. [1]

There are other areas where the damage spammers have done (or are likely
to do when it suits their purposes) is much more of an issue.  And it's
certainly not worth trying to salvage a fundamentally-broken technology
like SPF in order to fix a problem we don't have.

---Rsk

[1] And in most of the scenarios I've worked through where they *could*
be co-opted, they're a distant third or fourth choice, because attackers
in a position to carry those approaches out will quite likely already have
preferable means at their disposal.

---

From: Ian Eiloart <[email-masked]>
Date: Mon, 19 Jan 2009 15:38:39 +0000

> for abusive purposes. [1]

I agree, but bounces have two huge advantages,

1. You can bounce selectively. IE, you can accept a message for one 
recipient, while bouncing it for another, even after seeing the content. 
So, you can have different delivery policies for different people.

2. The content of the bounce message is generated by the MTA that made the 
decision. Therefore, it can contain content that's easier to understand.

So, you may not agree that the balance is in favour of bouncing, but it 
certainly would be better to bounce if you KNOW that the sender address is 
not forged - regardless of how you know that.

In fact, we adopt that policy when we know that the sender is local and 
authenticated. Because most mail clients are very unhelpful when you reject 
their mail submissions.

> like SPF in order to fix a problem we don't have.

It's not fundamentally broken. It's a tool that has some useful 
applications, even if you can use it to shoot yourself in the foot. For 
example, two organisations that want to improve their email communications 
could agree to deploy SPF and whitelist each other when SPF passes happen. 
It'll work better when they also ensure that their users use their message 
submission services, and when their networks are secure, and when they can 
easily contact each others administrators.

That works because they trust each other, and because they can now reliably 
associate the domain with the organisation under certain circumstances 
(with SPF passes). The trust model can be extended through domain 
reputation servers.

Is there any other way to do this, that doesn't involve publishing 
information that says where my email can originate, or signing outbound 
email?

---

From: Paul Russell <[email-masked]>
Date: Mon, 19 Jan 2009 16:53:58 -0500

> So, you can have different delivery policies for different people.

I know that sendmail can be configured to accept for one recipient and reject
for another recipient.  I assume there are other MTA products which also
support this functionality.  If yours does not, perhaps you should consider
using a different MTA.

> decision. Therefore, it can contain content that's easier to understand.

If your MTA cannot be configured to issue meaningful rejection messages,
perhaps you should consider using a different MTA.

In an earlier message, you noted that the MTA which receives the rejection
message may replace a meaningful error message with text with less meaningful
text.  There are mail servers which modify the body of any inbound DSN, so
generating a 'bounce' does not ensure that your meaningful message will
survive transit to the intended recipient.

-- 
Paul Russell, Senior Systems Administrator
OIT Messaging Services Team
University of Notre Dame
[email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/asrg

---

From: Daniel Feenberg <[email-masked]>
Date: Mon, 19 Jan 2009 19:06:34 -0500 (EST)

> support this functionality.  If yours does not, perhaps you should consider

If I may contribute...

Sendmail can selectively reject receipients while processing "RCPT TO:" 
commands, but not after the header and body are received. So anti-spam 
processing based on the sending IP address can be different for different 
users (and still use rejection rather than DSNs), but it isn't 
straightforward to selectively handle items based on content or headers 
without resorting to DSNs or spam folders.  That is one of the advantages 
DNSBLs have over content analysis. Or do you have a specific suggestion? 
As I understand it, this is a consequence of the SMTP protocol, and not 
something that sendmail can program around.

Dan Feenberg

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 20 Jan 2009 08:34:51 +0100

Daniel Feenberg wrote:
>>> 1. You can bounce selectively.
>>
>> I know that sendmail can be configured to accept for one recipient and 
>> reject for another recipient.
> 
> commands, but not after the header and body are received.
> [...]
> As I understand it, this is a consequence of the SMTP 
> protocol, and not something that sendmail can program around.

Given a set of recipients (r1, r2, ..., rn) the server can partition 
it in subsets that have homogeneous filtering recipes. It does that in 
steps. Each step consists in responding 250 for r1 and any other 
recipient with identical filtering, 4xx to the rest.
_______________________________________________
Asrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/asrg

---

From: Ian Eiloart <[email-masked]>
Date: Tue, 20 Jan 2009 11:30:33 +0000

--On 20 January 2009 08:34:51 +0100 Alessandro Vesely <[email-masked]> 
wrote:

> Given a set of recipients (r1, r2, ..., rn) the server can partition it
> in subsets that have homogeneous filtering recipes. It does that in

Yes, that's the scheme I was referring to. It can be implemented with my 
MTA - Exim. However, if any filter subset doesn't want the message, then 
you should give a 5yz response after seeing the body. RFC2821 says that the 
sending MTA "SHOULD not again attempt delivery to the same server without 
user review and intervention of the message". 
<http://www.apps.ietf.org/rfc/rfc2821.html#sec-4.2.5> I guess most MTA's 
aren't RFC2821 compliant in that respect.

RFC 5321 says the client "may either return it to the user or requeue it 
for a subsequent attempt". That's an improvement, but at best there's no 
guarantee that members of the second and subsequent subsets will ever see 
the message. If they do, it will be after a delay that many people find 
unacceptable. 2821 and 5321 both say retry intervals should be at least 30 
minutes, but that might be unusual. Exim's default policy is every 15 
minutes for the first two hours, for example, then at decreasing intervals. 

If the number of subsets is large enough, you may even exceed the retry 
timeout on the sending MTA, so a large site had better not have too many 
filter recipes. The limit would be about 30 different filter recipes when 
the sending server is using Exim's default retry policy. 
<http://www.exim.org/exim-html-current/doc/html/spec_html/ch07.html#SECID57>.

30 recipes lets you give your user four binary, or three ternary options. 
Even if the say "don't filter" to all of those options, they can still lose 
email if the first recipient has enabled filtering, and the sending server 
is rfc2821 compliant.

---

From: Rich Kulawiec <[email-masked]>
Date: Tue, 20 Jan 2009 07:06:17 -0500

> So, you can have different delivery policies for different people.

(a) I don't want to bounce at all, ever and (b) I don't see the value in this
approach.  If the message is from a spammer (or is classified as spam
by some mechanism), then it gets rejected [for all recipients] no matter
who wants it or why.  I don't permit users to selectively decide to
receive bogons; I don't permit them to selectively decide to receive
spam, and for the same reason: I think it's very poor security policy.

> 2. The content of the bounce message is generated by the MTA that made 
> the decision. Therefore, it can contain content that's easier to 
> understand.

This doesn't follow.  Reject messages, like bounces, are also generated by
the MTA that made the decision.  Any sufficiently-broken mail system
that mangles those is probably going to mangle bounces or do other
silly things as well.  If you need to transmit a long, detailed message,
then provide a static URL, as in:

	550 blah blah Your message rejected because your momma dresses
		you funny; see http://www.example.com/rejects/your-momma.html

Of course there's no guarantee that will survive transit either.  The bottom
line is that you can't do anything to fix other peoples' broken mail systems.

> certainly would be better to bounce if you KNOW that the sender address 
> is not forged - regardless of how you know that.

But I don't care if the sender address is forged or not.  It plays no role
in my anti-spam setup, and has no impact on what I do with such traffic,
since my intention is to either accept or reject it outright during the
SMTP connection.

---Rsk

---

From: Ian Eiloart <[email-masked]>
Date: Tue, 20 Jan 2009 13:26:48 +0000

--On 20 January 2009 07:06:17 -0500 Rich Kulawiec <[email-masked]> wrote:

> n Mon, Jan 19, 2009 at 03:38:39PM +0000, Ian Eiloart wrote:
> (a) I don't want to bounce at all, ever and (b) I don't see the value in
> this approach.  If the message is from a spammer (or is classified as spam

I work in an academic environment with a medical school, where a word like 
"viagra" or "penis" can mean spam to some people, or a vital research grant 
application to others.

I don't like content filtering at all, and only apply it quite lightly. I'd 
like to allow people to opt in to heaver filtering (eg, those that are 
particularly sensitive to obscenity), or opt out completely. While I'm not 
permitted to bounce messages (I don't) I can't do that well when there are 
multiple recipients.

Pre-data, of course, I can apply selective filtering, but only for policies 
that most people don't understand (source IP address reputation, and EHLO 
string). And not based on the sender address. Blacklisting specific senders 
is ineffective, whitelisting creates holes in my spam filter.

> Of course there's no guarantee that will survive transit either.  The
> bottom line is that you can't do anything to fix other peoples' broken
> mail systems.

Yes, I thought that's what I said. Not only might it not survive, but it 
might be replaced with a misleading error message. For example, if I reject 
a sender address after RCPT TO (in case the rcpt is postmaster), some MTA's 
will generate a bounce message which says the recipient doesn't exist.

Anyway, it's almost always embedded in some other message. Does anyone know 
of an MTA that doesn't add something to a reject message when generating a 
bounce message? Would such a bounce be of any value?

I do put URLs in my messages, usually on the first line because that seems 
more likely to survive. IIRC, there are MTAs that preserve only the first 
line of an error message in a bounce. I also use enhanced error codes, in 
the hope that some MTAs might generate better bounce messages.

I don't buy the argument that an MTA that mangles reject messages (that 
requires little or no administrative effort) will also mangle bounce 
messages (that requires more effort). Perhaps I'm wrong. Does anyone know 
of an MTA that mangles bounce messages in its default configuration?

---

From: "Peter J. Holzer" <[email-masked]>
Date: Tue, 20 Jan 2009 18:21:30 +0100

> aren't RFC2821 compliant in that respect.

I'm not sure how you interpret this sentence. Consider:

MAIL FROM:<[email-masked]>
250 ok
RCPT TO:<[email-masked]>
250 ok
RCPT TO:<[email-masked]>
450 try again later
DATA
354 send message
...
550 message rejected

Should the client assume a permanent failure for both recipients?
Or should it assume a permanent failure only for
<[email-masked]> and requeue the message for
<[email-masked]>? 

> for a subsequent attempt". That's an improvement,

Erm, that's for a 4xx reply. I am quite certain that RFC 5321 did not
intend any semantic changes relative to RFC 2821. All changes are only
meant to clarify.

> there's no guarantee that members of the second and subsequent subsets
> will ever see the message. If they do, it will be after a delay that
> many people find unacceptable.

> filter recipes.

Right. That's why I didn't consider looking at "filter recipies", when I
implemented a similar scheme for qpsmtpd. Instead I look at the result.
There are only 2 different results: Accept or reject, so unless the
result changes between delivery attempts (which can happen if e.g., a
blackist is updated or a recipient changes a rule), there are at most
three delivery attempts:

 1) The results of the content filters for the different recipients
    don't agree. Use the results to split the recipients into two sets.
 2) All the recipients from set1 get 2xx reply to RCPT, all the 
    recipients from set2 get a 4xx reply. Now the results of the content
    filters for all recipients agree (they are all from set1), so we can
    reply with either 2xx or 5xx.
 3) All the remaining recipients are now from set2, so the content
    filters agree again, and we can again reply with 2xx or 5xx.

(the actual implementation is a little bit more complicated, but that's
the basic idea).

For most mails, the content filters of all recipients agree on the
result, so the mail can be accepted or rejected at the first delivery
attempt.

It's a really ugly hack, but it works quite nicely.

	hp

-- 
   _  | Peter J. Holzer    | Openmoko has already embedded
|_|_) | Sysadmin WSR       | voting system.
| |   | [email-masked]         | Named "If you want it -- write it"
__/   | http://www.hjp.at/ |  -- Ilja O. on [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/asrg

---

From: der Mouse <[email-masked]>
Date: Tue, 20 Jan 2009 12:45:20 -0500 (EST)

> [Assuming no config changes], there are at most three delivery
> attempts:

This doesn't look right.

>    recipients from set2 get a 4xx reply.  Now the results of the
>    content filters for all recipients agree (they are all from set1),
>    so we can reply with either 2xx or 5xx.

Right so far.

> 3) All the remaining recipients are now from set2,

Yes.

> so the content filters agree again,

No - or rather, not necessarily, not unless you have only two possible
filtering setups.  All we know about set2 is that their filtering
choices differ from those of set1.  We have no particular reason to
think that set2 is uniform within itself (unless, as I said, there are
only two choices possible).

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		[email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B
_______________________________________________
Asrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/asrg

---

From: "Peter J. Holzer" <[email-masked]>
Date: Tue, 20 Jan 2009 20:01:59 +0100

> > [Assuming no config changes], there are at most three delivery

As I wrote, the actual algorithm is more complicated. If you are really
interested, you can get the perl source code from
http://www.hjp.at/projekte/qpsmtpd/cf_wrapper/. But I'll explain a
little more below.

> choices differ from those of set1.

Yup. So for this particular message (unless the setup has changed in the
mean time) if the result for set1 was "accept", for set2 it must be
"reject" and vice versa.

(If the setup does change, it can affect delivery attempt 2 just the
same as attempt 3 - if it does happen, new sets are created and a 4xx
reply is returned - theoretically that can continue until there is only
one recipient left)

> We have no particular reason to think that set2 is uniform within
> itself (unless, as I said, there are only two choices possible).

There are, if we are only interested acception/rejection of the message.
You cannot include specific messages, of course (currently I just
concatenate all the rejection reasons into a single message).

The real problem is in the phrase "this particular message". At the time
the server receives the first "RCPT TO:" command it has no way to
know that this is a new delivery attempt for a message which was
temporarily rejected earlier, much less which message (envid is
per-transaction, and you couldn't rely on the client sending it anyway).
So, as a weak approximation, I use sender/recipient pairs instead of
message/recipient pairs as elements of the sets. This works quite well
because a recipient usually wants to receive all messages from a
specific sender or none. But yes, theoretically it can happen that the
sets are split again and again until each recipient is in a set of their
own. In practice I haven't seen it yet (after more than 3 years, but
admittedly on a relatively small number of email addresses).

	hp

---

From: "Peter J. Holzer" <[email-masked]>
Date: Tue, 20 Jan 2009 20:40:37 +0100

---

From: "Chris Lewis" <[email-masked]>
Date: Tue, 20 Jan 2009 15:00:44 -0500

> I assume with "reject message" you mean the text in a 5xx reply, and
> with "mangle reject message" you mean "doesn't include the complete
> reject message in the DSN it generates, and/or adds a confusing
> "explanation" (like "Recipient doesn't exist" for "550 Message from
> 61.251.173.243 rejected, because it is listed in  XBL/CBL.").

I also see exchange spitting back "site didn't give a reason", and then
there's the full reason string (including SMTP extended error codes)
further down.

>> Perhaps I'm wrong. Does anyone know 
> Lotus Domino. It removes the first (human-readable) part of any RFC 3462
> conforming DSN and replaces it with a "user-friendly" explanation, which
> usually has nothing to do with the problem. It also completely mangles
> the Received lines in the third part (the headers of the bounced
> message). At least it leaves the second part alone so you usually know
> why the message was rejected, unless you are already confused by the 1st
> part (which most users are). [If anybody knows how that can be turned
> off, please tell me - it's driving me nuts]

We've also seen Exchange replacing site names in DSNs that were
_manually_forwarded_.  Same DSN, when manually forwarded to two
different places (one with Exchange, one without) are _different_.
Exchange (2007 methinks, not sure) substitutes in its own server name.
Makes you freak out when it's a DSN for an email that NEVER went
anywhere near your servers and had nothing to do with you.

[IOW: something like: user sends email from Yahoo to MSN, and gets a
bounce.  Forwards it to me for explanation, and the enclosed DSNs say
_our_ server issued the bounce.  WTF?]
_______________________________________________
Asrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/asrg

---

From: Ian Eiloart <[email-masked]>
Date: Wed, 21 Jan 2009 10:00:47 +0000

>> >>> I know that sendmail can be configured to accept for one recipient
>> >>> and reject for another recipient.
>> >>
>> >> Sendmail can selectively reject receipients while processing "RCPT
>> >> TO:" commands, but not after the header and body are received.
>> >> [...]
>> you should give a 5yz response after seeing the body. RFC2821 says that
>> the  sending MTA "SHOULD not again attempt delivery to the same server
>> without  user review and intervention of the message".

Well, to me it means that there's something wrong with recipient1's mailbox 
- perhaps currently over quota, but that the DATA isn't acceptable 
according to some site policy. I think the RFC's authors probably didn't 
have in mind the contortions of the selective filtering mechanism.

Really, autonegotiation of a switch to LMTP would be the solution here. Of 
course, spammers wouldn't adopt it. But, if it caught on then eventually 
people would start treating smtp clients which refused to switch to LMTP 
with suspicion.

> intend any semantic changes relative to RFC 2821. All changes are only

Sorry, you're quite right. I confused two fairly similar paragraphs. Both 
drafts say that you MAY retry after a 4xx post-data error, and that you 
SHOULD NOT retry after a 5xx error. So, the proposed mechanism shouldn't 
work.

However, I'm sure I've heard of sites that actually do implement it, so 
perhaps MTAs are mostly requeueing messages after they see the 4xx on RCPT 
TO.

---

From: Ian Eiloart <[email-masked]>
Date: Wed, 21 Jan 2009 10:21:02 +0000

> admittedly on a relatively small number of email addresses).

Oh, that's a pretty good solution. There are edge cases, and perhaps even 
exploits, but it's probably going to work most of the time - provided the 
client MTA ignores 4.2.5 of the RFC. It still delays messages for some 
recipients, but only up to an hour or so. It might cause problems with 
conversations on lists like this - but they could be mitigated by caching 
the "sender/recipient/source IP" triplet.

The exploit, of course, is that spammers could open channels by sending 
apparently innocent messages.

---

From: David Wilson <[email-masked]>
Date: Wed, 21 Jan 2009 10:40:13 +0000

> It still delays messages for some 
> recipients, but only up to an hour or so.

A few years back, when we were experimenting with greylisting, we had a
problem with someone we know sending from a site which only retried at
roughly 24 hour intervals, (and also from a different IP address each
time). The organization is a reasonably well known UK professional
institution.

The lesson was that you cannot necessarily rely upon reasonable retry
strategies from "legitimate" sources.

cheers

David

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 21 Jan 2009 16:14:05 +0100

> --On 20 January 2009 18:21:30 +0100 "Peter J. Holzer" <[email-masked]> 
> wrote:

"412 you are whitelisted" is also sometimes encountered.

> Well, to me it means that there's something wrong with recipient1's 
> mailbox - perhaps currently over quota, but that the DATA isn't 
> acceptable according to some site policy.

I assume you meant recipient2.

The server rejects a message, not a DATA block. In case the sender's 
MX resolves to the same server, any MTA would send the bounce there, 
notwithstanding that DATA block isn't acceptable according to some 
site policy.

The message that should not be retried is inclusive of the envelope, 
and recipient2 is not currently in the server's recipients buffer. So 
much so that the client should retry recipient2 even if the server 
accepted the data for recipient1.

---

From: Paul Russell <[email-masked]>
Date: Sun, 25 Jan 2009 22:12:23 -0500

> of an MTA that mangles bounce messages in its default configuration?

Exchange parses an inbound DSN to identify every address for which delivery
status information is provided, then pre-pends the entire DSN with boilerplate
text for every address in the DSN, regardless of the specific status code and
text for that address.

---

From: Graeme Fowler <[email-masked]>
Date: Mon, 26 Jan 2009 20:54:20 +0000

> text for every address in the DSN, regardless of the specific status code and

In my current experience, it's *Outlook* which does the
mangling/prepending/call-it-what-you-will.

In $dayjob we run Outlook as the supported desktop client (both 2k3 and
2k7) against a non-MS backend groupware server. When I see, using
Outlook 2k7, bounces arrive to our role account handling most of the
postmaster guff, I very briefly see the original MTA-provided bounce
message in my viewing pane. After a split second this is replaced by the
MS-ified version, rendering the item almost completely worthless.

I'd put money on Outlook having a DLL included which decodes the message
and does the metaphorical email equivalent of "It looks like you're
writing a letter!" for the DSN. If I view the messages in Thunderbird,
or Evolution, or any other sane(ish) MUA they display as expected - MIME
parts, in order or as attachments.

Unfortunately I know so little about Windows that debugging this
behaviour is all but impossible.

You can tar Exchange with many brushes, but apparently not this one - it
looks very much like client behaviour to me; that said it's quite rare
to see Outlook in "groupware" mode not using Exchange as the backend so
I can understand the confluence of data.

Graeme

---

From: "Chris Lewis" <[email-masked]>
Date: Mon, 26 Jan 2009 16:31:07 -0500

> mangling/prepending/call-it-what-you-will.

Unless this is a different "issue", in our case the DSN mangling _has_
to be being done in Exchange.  I'm running Thunderbird, and (some)
inbound DSNs get mangled.  I don't think the MUA knows enough about or
infrastructure to mangle it the way we see it ("reporting MTAs"
completely unrelated to us being replaced with internal servers).

I suspect it's simple "oh yeah, this is a DSN.  I'll report it as a DSN
- I'm the "reporter" now", not realizing that "Reporting MTA" _should_
be the server who reports the initial error, not umpty-ump intermediate
servers regenerating the same DSN.
_______________________________________________
Asrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/asrg