Subject: [Asrg] Greylisting BCP

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 18 Oct 2011 12:01:21 -0700

After some chatter inside MAAWG and on the ietf-smtp mailing list, I've started an outline for a BCP on the practice of greylisting.  The main purpose is to explain what it is, discuss the pros and cons of its variants, and give some recommendations for implementation and configuration for a few example installations and policies.

The draft (which is currently only an outline) is here: https://datatracker.ietf.org/doc/draft-kucherawy-greylisting-bcp/

Comments welcome.

-MSK

---

From: Daniel Feenberg <[email-masked]>
Date: Tue, 18 Oct 2011 15:42:24 -0400 (EDT)

On Tue, 18 Oct 2011, Murray S. Kucherawy wrote:

> 
> After some chatter inside MAAWG and on the ietf-smtp mailing list, I’ve started an
> outline for a BCP on the practice of greylisting.  The main purpose is to explain
> what it is, discuss the pros and cons of its variants, and give some recommendations
> for implementation and configuration for a few example installations and policies.
> 
>  
> 
> The draft (which is currently only an outline) is here:
> https://datatracker.ietf.org/doc/draft-kucherawy-greylisting-bcp/
> 
>  
> 
> Comments welcome.

Where should comments go? I have a question really, though it might be 
construed as a comment. Why do greylisters match on the (sender, 
receipient, MTA) triple rather on just the MTA? Isn't it nearly certain 
that if an MTA returns for one sender/receipient pair, it will return for 
any pair? So that keeping track of all three seems unnecessary and 
increases the probability of a message being delayed. What am I missing?

Daniel Feenberg
NBER

---

From: Chris Lewis <[email-masked]>
Date: Tue, 18 Oct 2011 19:12:18 -0400

On 11-10-18 03:42 PM, Daniel Feenberg wrote:

 > that if an MTA returns for one sender/receipient pair, it will return
 > for any pair? So that keeping track of all three seems unnecessary and

As I understand it, some grey-listing systems match on sender/recipient 
pairs (not MTA) so as to not penalize clustered outbounds that share queues.

There's all sorts of 'optimizations'/variations that you can apply for 
different behaviours.

You're right, _just_ the MTA would work just about as well for the main 
use case: bot armies.  But that's not the only potential use-case.

There is a danger in specifying the precise details/tuning values of a 
"standardized gray listing" mechanism.  If it's too predictable, you 
could probably come up with a simplistic mechanism for defeating it 
without requiring the complexity of queuing.  "Hybrid vigor" is a good 
thing.

---

From: Douglas Otis <[email-masked]>
Date: Tue, 18 Oct 2011 20:39:13 -0700

On 10/18/11 12:42 PM, Daniel Feenberg wrote:
> receipient, MTA) triple rather on just the MTA? Isn't it nearly 
> certain that if an MTA returns for one sender/receipient pair, it will 
> return for any pair? So that keeping track of all three seems 
> unnecessary and increases the probability of a message being delayed. 
> What am I missing?
Grey listing challenges stateful processing of the sender to test an 
often erroneous assumption that bots sending spam don't maintain state. 
Thanks to grey listing, many bots retry against the same recipients, 
just not always with the same message. Heuristic methods are quickly 
defeated once deployed on enough systems. Defining practices for failing 
concepts seems misguided since such heuristics are likely to make other 
erroneous assumptions related to the nature of the outbound MTA.

We employ temp errors to prioritize limited resources which may delay 
sources with lower reputations. When they are in the middle of some 
campaign, the delay may persist until their run completes. There would 
not be any way to predict how long that might take, nor would any hint 
likely affect the average bulk sender focused on completing their runs. 
We are about to publish a list that categorizes this type of behavior.

I agree it would make more sense to focus on knowing which domain 
_controls_ the outbound MTA without umpteen additional transactions. 
With outbound MTA authentication, receivers would have a sure and robust 
basis for avoiding abusive transactions, where grey listing would be 
considered wasted efforts.

-Doug

---

From: Jose-Marcio Martins da Cruz <[email-masked]>
Date: Wed, 19 Oct 2011 12:11:28 +0200

> without requiring the complexity of queuing. "Hybrid vigor" is a good

I think I agree with most you said. Maybe not all.

Greylisting is there and it will exist for some time. There are good and bad 
variations/optimisations, good and bad software configurations... This BCP should not generalize 
saying that all greylisting is bad, even if some people are tempted to say so.

It seems to me that this BCP mention only problems caused by the filters.

What's still lacking in the BCP proposal is a section about the behaviour of senders MTAs, which, 
IMHO, causes a lot of problems too.

Some examples of bad things which really causes problems to greylistings :

* Some MTAs from some (big) ISPs, which connects 3 to 5 times within some seconds and try again only 
after some hours (some equals something between 4 to 24 hours).

* Some ISPs using a pool of sending MTAs which IP addresses are randomly distributed over ranges 
without any common part. Something like 193.xxx.xxx.xxx followed by 67.xxx.xxx.xxx...

* Some sender MTAs which begins retrying each some seconds (1 to 10) during some days, and ends 
being catched by some rate limit mechanism.

* Some bad configured or misinterpreted 4xx reply code

* ...

--

---

From: Martijn Grooten <[email-masked]>
Date: Wed, 19 Oct 2011 16:05:35 +0100

> without requiring the complexity of queuing.  "Hybrid vigor" is a good

Good point.

I like the idea of a BCP though and I like the outline too.

Some quick comments:

Do people apply greylisting post-DATA in practise? Or is this really only something only performed in labs as it is the only way to determine whether causes false positives.

It is hard to reliably determine how much greylisting helps on a specific system, i.e. what difference it makes compared to the hypothetical situation greylisting wasn't used. May be an idea to include some caution about that.

"greylisting is more expensive than not greylisting"

Can you explain this?

"special actions to take if the same message is retried before the time limit expires"

I assume by "message" you mean the (sender, recipient, MTA) triplet?

Martijn.

Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Wed, 19 Oct 2011 17:58:59 +0200

On 2011-10-19 12:11:28 +0200, Jose-Marcio Martins da Cruz wrote:
> What's still lacking in the BCP proposal is a section about the behaviour 
> of senders MTAs, which, IMHO, causes a lot of problems too.
>
> * Some MTAs from some (big) ISPs, which connects 3 to 5 times within some 
> seconds and try again only after some hours (some equals something 
> between 4 to 24 hours).
>
> * Some ISPs using a pool of sending MTAs which IP addresses are randomly 
> distributed over ranges without any common part. Something like 
> 193.xxx.xxx.xxx followed by 67.xxx.xxx.xxx...
>
> * Some sender MTAs which begins retrying each some seconds (1 to 10) 
> during some days, and ends being catched by some rate limit mechanism.

* Some sender MTAs change the envelope sender with every delivery
  attempt (Yahoo Groups used to do this, but stopped several years ago.
  I don't know any recent example). This garantuees that the triple
  never matches ...

* Some ISPs use one machine for the first delivery and another for
  processing the queue. The "queue processing" IP will get whitelisted
  but the "first try" address wont (unless you are lucky), resulting in
  a constant delay for every mail.

> * Some bad configured or misinterpreted 4xx reply code

Some MTAs (e.g. old versions of Groupwise) don't understand 4xx at all
and treat it like 5xx.

	hp

-- 
   _  | Peter J. Holzer    | Web 2.0 könnte man also auch übersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         | 
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

---

From: Joe Sniderman <[email-masked]>
Date: Wed, 19 Oct 2011 12:19:52 -0400

On 10/19/2011 11:05 AM, Martijn Grooten wrote:
>> There is a danger in specifying the precise details/tuning values
>> of a "standardized gray listing" mechanism.  If it's too
>> predictable, you could probably come up with a simplistic mechanism
>> for defeating it without requiring the complexity of queuing.
>> "Hybrid vigor" is a good thing.
> 
> Good point.
> 

<aol> me too </aol>

> Some quick comments:
> 
> Do people apply greylisting post-DATA in practise?

milter-greylist supports greylisting during DATA (not sure if its at the
beginning of DATA or at the end, could try it easily enough and find out
though) by means of the dacl configuration directive (as opposed to racl
for greylisting during RCPT)

AFAIK the DCC also can perform greylisting during DATA.

Might be worth adding a section to for greylisting performed during DATA.

There's also some interesting archive discussion regarding it on the
Mimedefang list:
http://lists.roaringpenguin.com/pipermail/mimedefang/2009-May/034793.html

It would appear from that discussion that at least Roaring Penguin
probably does do so in practice.

> Or is this really
> only something only performed in labs as it is the only way to
> determine whether causes false positives.

AFAIK one of the supposed benefits of greylisting during DATA instead of
RCPT is reduction of false positives.

> It is hard to reliably determine how much greylisting helps on a
> specific system, i.e. what difference it makes compared to the
> hypothetical situation greylisting wasn't used. May be an idea to
> include some caution about that.

Log delivery attempts that are greylisted. Log greylisted delivery
attempts that subsequently pass. Compare the IPs in question against
whatever DNSBLs are in use to determine if the delivery attempts would
have been blocked anyway (without resorting to computationally expensive
content filtering)....

Might also make sense to log the DNSBL result at the time that a
delivery attempt is greylisted, and then log delivery attempts that pass
greylisting but are rejecting due to DNSBLs, and thereby determine if
greylisting is effectively giving a recently gone-bad host extra time to
get itself listed.

> I assume by "message" you mean the (sender, recipient, MTA) triplet?

Or if performed during DATA it could mean the same message data or at
least sufficiently similar message data, or even sender, recipient,
message-id triplet, or some other combo.

-- 
Joe Sniderman <[email-masked]>

---

From: Martijn Grooten <[email-masked]>
Date: Wed, 19 Oct 2011 19:42:34 +0100

> greylisting is effectively giving a recently gone-bad host extra time

Yes, that makes sense and, if you're running a test (and don't care too much about computational performance) could be extended to content scanning too.

But that assumes you are able to identify messages, which I doubt is possible to do in a reliable way unless you greylist during/post DATA. Which -- at least in theory -- does not need to give the same results as greylisting at an earlier stage.

I guess the only way to really test the effectiveness of (your version of) greylisting (on your mail stream) is if you have a reasonably large stream and you split that feed in two random groups and use greylisting one group only.

(I suppose most people don't care as much about measuring effectiveness as I do, nor should they, but including something about effectiveness may be a good idea as this is the main reason people use greylisting.)

Martijn.

---

From: Joe Sniderman <[email-masked]>
Date: Wed, 19 Oct 2011 15:22:34 -0400

On 10/19/2011 02:42 PM, Martijn Grooten wrote:
>>> hypothetical situation greylisting wasn't used. May be an idea
>>> to include some caution about that.
>> 
>> attempts that subsequently pass. Compare the IPs in question
>> against whatever DNSBLs are in use to determine if the delivery
>> attempts would have been blocked anyway (without resorting to
>> computationally expensive content filtering)....
>> 
>> delivery attempt is greylisted, and then log delivery attempts
>> that pass greylisting but are rejecting due to DNSBLs, and thereby
>> determine if greylisting is effectively giving a recently gone-bad
>> host extra time to get itself listed.
> 
> Yes, that makes sense and, if you're running a test (and don't care
> too much about computational performance) could be extended to
> content scanning too.

Right, and then log whether content filters would have blocked the
message anyway, had the sender eventually retried.  Glean how many (if
any) spams were stopped by greylisting that neither DNSBLs nor content
filters would have otherwise caught.

> But that assumes you are able to identify messages, which I doubt is
> possible to do in a reliable way unless you greylist during/post
> DATA. Which -- at least in theory -- does not need to give the same
> results as greylisting at an earlier stage.

Not necessarily..  Say one sees 1000 IPs that attempted to send 100,000
times, were greylisted, never passed greylisting, and weren't listed on
DNSBLs that the system normally checks - that gives one possible piece
of data on effectiveness. If one takes it a step further, find all the
counted greylistings where the IP attempted to send to at least one
spamtrap, and only count those incidents but exclude the actual
attempted trap deliveries, one might get a very conservative count of
how many attempted spam deliveries to actual user addresses were
prevented by greylisting that the DNSBLs in use would not have caught.
Since one isnt matching initial greylistings to later successful
retries, identifying the message isnt necessary to get useful stats.

Regarding the second method, simply logging
blacklisted-ip-was-greylisted, and non-blacklisted-ip-was-greylisted,
and then matching the second group of IPs (not messages) against later
DNSBL-based rejections would give a good indicator as to whether
greylisting is helping to mitigate the delay between when a host starts
spamming and when a host is blacklisted.

> I guess the only way to really test the effectiveness of (your
> version of) greylisting (on your mail stream) is if you have a
> reasonably large stream and you split that feed in two random groups
> and use greylisting one group only.

Yeah that would work.  If thats not feasible, one could also disable
greylisting periodically at random, and see if there is a spike in
uncaught spam.

> (I suppose most people don't care as much about measuring
> effectiveness as I do, nor should they, but including something about
> effectiveness may be a good idea as this is the main reason people
> use greylisting.)

Agreed. May also be a good idea to recommend some measuring techniques
in the BCP, since effectiveness might vary based on site-specific factors.

---

From: Jose-Marcio Martins da Cruz <[email-masked]>
Date: Wed, 19 Oct 2011 21:54:49 +0200

> (I suppose most people don't care as much about measuring effectiveness as I do, nor should they, but including something about effectiveness may be a good idea as this is the main reason people us…

I think that it's not serious to run any security tool without monitoring it. If you run a tool 
which may reject messages (or any kind of communication), you should monitor and know how it 
performs : how many connections it handles, how many connections it rejects, etc, etc, etc... If you 
do, and observe this numbers each day, you can have some idea about effectiveness. "Fire and forget" 
is never a good thing in security.

An unattended and interesting side effect of greylisting is that it blocks a lot of virus/worms.

José-Marcio

---

From: "Peter J. Holzer" <[email-masked]>
Date: Thu, 20 Oct 2011 09:22:41 +0200

On 2011-10-18 15:42:24 -0400, Daniel Feenberg wrote:
> any pair?

Yes, but how do you determine that the MTA returned for one
sender/receipient pair? For that you need the whole tripel. Once you
have established that the MTA does queue, the IP address is sufficient
(this wasn't in Harris' original proposal, but it is a common
optimization).

Actually, the tripel (ip address, envelope sender, envelope recipient)
is just an approximation. What greylisting really tries to establish is
whether a particular /message/ is queued. To do that right you would
have to compare the whole message, but that would be expensive. So
greylisting assumes that if there is a second delivery attempt from the same IP
address with the same sender and recipient within a certain time window,
then it is a second delivery attempt for the same message, not an
independent message. We all know that this assumption isn't always true
(and I've occasionally advised users to just send two mails within a few
minutes to get through greylisting) but its a reasonable heuristic.

There is also an argument for always using the whole tripel: It guards
against bots on the same IP address (e.g., an MTA and an infected
workstation behind a common NAT, or a reassigned IP address).

	hp

---

From: "Peter J. Holzer" <[email-masked]>
Date: Thu, 20 Oct 2011 09:43:48 +0200

On 2011-10-18 19:12:18 -0400, Chris Lewis wrote:
> use case: bot armies.

I don't think so. If the bot sends one spam from <[email-masked]> to
<[email-masked]> and some time later (within the greylisting windo)
another from <[email-masked]> to <[email-masked]> the second one
would get through if you use only the IP address. To establish that a
new MTA is "legitimate" you need to be fairly restrictive. After that
you just need to make sure that it's an MTA you already checked and the
IP address is (usually) sufficient.

> without requiring the complexity of queuing.  "Hybrid vigor" is a good  

ACK. However, for those who have to deal with greylisting (either on the
sending or receiving side) it's valuable to know what works well and
what doesn't. There's little sense in having everybody going through the
same learning curve. So, a BCP should not define the One True Way of
greylisting but list variants of greylisting as well as different
queueing strategies and discuss their pros and cons.

	hp

---

From: "t.petch" <[email-masked]>
Date: Fri, 21 Oct 2011 17:41:04 +0200

Well, I would prefer an alternative explanation, for why all my IRTF (and IETF)
e-mail now shows 'SPF fail' because it has come to my ISP (British Telecom) from
157.55.224.141,
and ARIN WHOIS lists 157.55.224.141 as an address allocated to Microsoft, which
is not listed as a permitted sender for irtf.org.

Any ideas?

Tom Petch

<snip>
Received-SPF: fail (mail9-am1: domain of irtf.org does not designate
157.55.224.141 as permitted sender) client-ip=157.55.224.141;
envelope-from=[email-masked];
helo=DB3PRD0702HT003.eurprd07.prod.outlook.com ;.outlook.com ;
<snip>

---

From: Scott Howard <[email-masked]>
Date: Fri, 21 Oct 2011 10:07:35 -0700

Perhaps because your "ISP" is using Microsoft?

$ dig +short mx btconnect.com
1 btconnect-com.mail.eo.outlook.com.

outlook.com (aka bigfish.com) is, fairly obviously, Microsoft.

Why it's failing SPF is probably something you'll need to take up with BT.

  Scott.

On Fri, Oct 21, 2011 at 8:41 AM, t.petch <[email-masked]> wrote:

> Received: from mail9-am1 (localhost.localdomain [127.0.0.1]) by mail9-am1
>  (MessageSwitch) id 1318964619898891_21731; Tue, 18 Oct 2011 19:03:39 +0000
>  (UTC)
> Received: from AM1EHSMHS011.bigfish.com (unknown [10.3.201.252]) by
>  mail9-am1.bigfish.com (Postfix) with ESMTP id D38CC1150054 for
>  <[email-masked]>; Tue, 18 Oct 2011 19:03:39 +0000 (UTC)
> Received: from DB3PRD0702HT003.eurprd07.prod.outlook.com (157.55.224.141)
> by
>  AM1EHSMHS011.bigfish.com (10.3.207.111) with Microsoft SMTP Server (TLS)
> id
>  14.1.225.22; Tue, 18 Oct 2011 19:03:39 +0000
> Received: from mail64-va3-R.bigfish.com (216.32.180.112) by
>  DB3PRD0702HT003.eurprd07.prod.outlook.com (10.3.4.151) with Microsoft
> SMTP
>  Server (TLS) id 14.1.225.71; Tue, 18 Oct 2011 19:03:38 +0000
> Received: from mail64-va3 (localhost.localdomain [127.0.0.1]) by
>  mail64-va3-R.bigfish.com (Postfix) with ESMTP id 209C74B8376 for
>  <[email-masked]>; Tue, 18 Oct 2011 19:03:37 +0000 (UTC)
> X-FB-SS: 13,
> Received: from mail64-va3 (localhost.localdomain [127.0.0.1]) by mail64-va3
>  (MessageSwitch) id 1318964522600467_28976; Tue, 18 Oct 2011 19:02:02 +0000
>  (UTC)
> Received: from VA3EHSMHS029.bigfish.com (unknown [10.7.14.249]) by
>  mail64-va3.bigfish.com (Postfix) with ESMTP id 7505A1640051 for
>  <[email-masked]>; Tue, 18 Oct 2011 19:02:02 +0000 (UTC)
> Received: from mail.ietf.org (12.22.58.30) by VA3EHSMHS029.bigfish.com
>  (10.7.99.39) with Microsoft SMTP Server id 14.1.225.22; Tue, 18 Oct 2011
>  19:01:55 +0000
> Received: from ietfa.amsl.com (localhost [127.0.0.1]) by ietfa.amsl.com
>  (Postfix) with ESMTP id 4106021F8B88; Tue, 18 Oct 2011 12:01:36 -0700
> (PDT)
> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ietf.org; s=ietf1;
> <snip>
> X-Original-To: [email-masked]
> Delivered-To: [email-masked]
> Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com(Postfix)
>  with ESMTP id A551021F8B74 for <[email-masked]>; Tue, 18 Oct 2011
>  12:01:35 -0700 (PDT)
> X-Virus-Scanned: amavisd-new at amsl.com
> X-Spam-Flag: NO
> X-Spam-Score: -102.712
> X-Spam-Level:
> X-Spam-Status: No, score=-102.712 tagged_above=-999 required=5
>  tests=[AWL=-0.114, BAYES_00=-2.599, HTML_MESSAGE=0.001,
>  USER_IN_WHITELIST=-100]
> Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com
>  [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3XSv6ncXIa5s for
>  <[email-masked]>; Tue, 18 Oct 2011 12:01:34 -0700 (PDT)
>
> _______________________________________________
> Asrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/asrg
>

---

From: Steve Atkins <[email-masked]>
Date: Fri, 21 Oct 2011 10:07:35 -0700

> e-mail now shows 'SPF fail' because it has come to my ISP (British Telecom) from

BT have outsourced their email handling (for btconnect.com) to Microsoft, and
SPF is being checked at an inappropriate point in the internal handoff.

The bigger issue is that you shouldn't care about SPF failing. An SPF pass
provides somewhat useful data, an SPF fail means absolutely nothing.

Cheers,
  Steve

---

From: David Romerstein <[email-masked]>
Date: Fri, 21 Oct 2011 13:19:26 -0400

> provides somewhat useful data, an SPF fail means absolutely nothing.

That is not ENTIRELY true. Under the proper circumstances, an SPF 
failure should indicate that NDRs should not be sent to the purported 
'Reply-To' or 'Return-Path' addresses.

But, yes, in an "is this spam or not" sense, SPF fail means nothing.

-- D

---

From: Steve Atkins <[email-masked]>
Date: Fri, 21 Oct 2011 10:28:43 -0700

On Oct 21, 2011, at 10:19 AM, David Romerstein wrote:

> That is not ENTIRELY true. Under the proper circumstances, an SPF failure should indicate that NDRs should not be sent to the purported 'Reply-To' or 'Return-Path' addresses.

Yes.

But if you remove the negative in that statement you end up with something like "Under the proper circumstances, an SPF pass should indicate that NDRs should be sent to the return-path address". :)

(Neither says what you should do with an SPF neutral response, but that's a whole other can of worms).

> But, yes, in an "is this spam or not" sense, SPF fail means nothing.

Yup. An SPF pass (or a DKIM pass) is a positive assertion - it means something solid and well defined.  An SPF or DKIM failure can be for any number of unrelated reasons, so it doesn't mean anything in particular.

Cheers,
  Steve

---

From: Douglas Otis <[email-masked]>
Date: Fri, 21 Oct 2011 10:47:17 -0700

> But, yes, in an "is this spam or not" sense, SPF fail means nothing.
The same problem may occur when traversing a carrier's LSN.  In which 
case, dropping NDRs and not accepting messages will negatively impact 
email integrity.  With a high level of shared MTAs, an SPF pass or fail 
should also be considered equally untrustworthy thanks to the efforts by 
malefactors.

There is an ongoing effort to combine SPF and DKIM to establish 
reputation.  DKIM excludes who sent the message and intended recipient.  
Will that mean DKIM signatures and SPF authorizations must be by same 
domain?  Either scheme may induce an inordinately large number of 
transactions.  Soon only the services of white-listed large providers 
will be a way out of this morass.

-Doug

---

From: Seth <[email-masked]>
Date: Fri, 21 Oct 2011 15:20:02 -0400 (EDT)

Steve Atkins <[email-masked]> wrote:

> But if you remove the negative in that statement you end up with
> something like "Under the proper circumstances, an SPF pass should
> indicate that NDRs should be sent to the return-path address". :)

Rather, you end up with "Under the proper circumstances, and SPF pass
would indicate that NDRs *may* be sent to the return-path address."

Seth

---

From: "t.petch" <[email-masked]>
Date: Sat, 22 Oct 2011 12:25:33 +0200

Many thanks to all who responded to my query.   I thought I could see what was
happening but did not want to believe it - I would have expected to have heard
something about this (which started over three weeks ago) in the general or
specialist press, or even direct from my ISP - but ....

Tom Petch

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 23 Oct 2011 16:39:11 +0200

> provides somewhat useful data, an SPF fail means absolutely nothing.

Wasn't it supposed to mean "reject"?  IIRC, that was the difference
between setting -all (as irtf.org does) rather than ~all or ?all.

---

From: Steve Atkins <[email-masked]>
Date: Sun, 23 Oct 2011 09:09:29 -0700

> between setting -all (as irtf.org does) rather than ~all or ?all.

That's what it's proponents used to claim, yes. But it's unfixably broken for
that purpose. The only reason you don't see serious mail problems for
anyone unwise enough to use -all is that almost nobody pays much attention
to SPF for rejecting email.

Cheers,
  Steve

---

From: Paul Smith <[email-masked]>
Date: Sun, 23 Oct 2011 22:16:53 +0100

> to SPF for rejecting email.
How is it 'unfixably' broken? Just interested.

it seems to me that the only reason it's 'unfixably broken' is that 
people are using it without really understanding what it's doing. If all 
MTAs rejected all mail that failed SPF checks, then they'd soon get the 
problems fixed. Remote users having to relay via a specified submission 
server is a trivial problem to fix (AFAICS), and incorrectly specified 
SPF records (as in the BT/MS case) deserve to cause messages to be 
blocked (IMHO)

(SPF is certainly not that useful as an 'anti-spam' mechanism, but it 
seems to have the potential to work reasonably well as an anti-spoofing 
mechanism, which makes it considerably harder for many spammers)

---

From: Steve Atkins <[email-masked]>
Date: Sun, 23 Oct 2011 14:43:00 -0700

On Oct 23, 2011, at 2:16 PM, Paul Smith wrote:

> it seems to me that the only reason it's 'unfixably broken' is that people are using it without really understanding what it's doing. If all MTAs rejected all mail that failed SPF checks, then they'd soon get the problems fixed. Remote users having to relay via a specified submission server is a trivial problem to fix (AFAICS), and incorrectly specified SPF records (as in the BT/MS case) deserve to cause messages to be blocked (IMHO)
> 
> (SPF is certainly not that useful as an 'anti-spam' mechanism, but it seems to have the potential to work reasonably well as an anti-spoofing mechanism, which makes it considerably harder for many spammers)

Mail forwarding is one common cause of failure. If you're sending mail to someone at @ieee.org and they forward it on to their @aol.com account, and you and AOL decide between you to enforce SPF -all then that mail will be undeliverable - and there's no way either you or AOL can predict that.

Mailing lists is another similar cause of SPF failures..

A third, which is a lot more common and complex to resolve than you'd expect, is organizations simply not knowing or not controlling where their mail is sent from. There've been cases both for SPF and ADSP ("SPF for DKIM") where one group within a company put out a restrictive policy based on their beliefs of where mail was sent from, and it all fell to pieces when employees got kicked off mailing lists when mail appearing to come from them was delivered from the mailing list manager and rejected due to auth failure, or where internal monitoring email sent from servers (cron, even) was silently discarded.

Combine that with SPF being pretty much ineffective for "anti-phishing" (which is what people actually mean when they say "anti-spoofing" or "brand protection") and you end up with it being not very useful as a negative assertion. 

It's interesting to compare SPF with DKIM and ADSP. In many ways DKIM+ADSP is the domain-based version of SPF. DKIM is the positive assertion half ("If this mail is DKIM signed / passes SPF then you know that you can trust that the sender is who they claim to be") while ADSP is the negative assertion half ("If this mail is not DKIM signed or fails SPF then you should reject / discard the mail"). 

The positive assertion is always valuable: it's difficult for "bad" mail - where by "bad" I mean mail that appears to have been sent someone you trust, but it isn't really from them -  to "pass" SPF or DKIM. The negative assertion much less so as it's pretty common for "good" mail to fail SPF or ADSP.

(I did some back of the envelope testing of ADSP as an anti-phishing / anti=spoofing measure, and based on my inbox it was worse than flipping a coin for each email, based on both false positives and false negatives).

Cheers,
  Steve

---

From: Paul Smith <[email-masked]>
Date: Mon, 24 Oct 2011 01:19:25 +0100

> Mail forwarding is one common cause of failure. If you're sending mail to someone at @ieee.org and they forward it on to their @aol.com account, and you and AOL decide between you to enforce SPF -a…
I thought most people did return-path rewriting for this case. If you 
don't do that, then you get undesirable behaviour anyway - eg if the AOL 
mailbox is full, you get a bounce message from AOL, even though you 
didn't send any messages to AOL at all.

> Mailing lists is another similar cause of SPF failures..
Similarly, I thought return-path rewriting was standard for mailing 
lists. Especially ones which do VERP, but usually even in ones which 
don't - eg the return path on the message I received from you was 
"<[email-masked]>", not your blighty.com address. Therefore, it 
correctly passed an SPF test for irtf.org.

> A third, which is a lot more common and complex to resolve than you'd expect, is organizations simply not knowing or not controlling where their mail is sent from. There've been cases both for SPF…
This is the problem of people who should know better not understanding 
what's going on. If the company had a policy that all mail from their 
domain had to go through certain servers, and employees were sending 
mail through other servers, then they deserved to get kicked off mailing 
lists... Just as they would deserve punishment for watching porn over 
the work Internet connection, etc. If they had a policy that mail had to 
go through certain servers, and configured SPF for different servers, 
then they shouldn't be managing mail systems.

If 'SPF' and/or DKIM was standardised and rigourously enforced, then 
return-path rewriting would be de rigeuer as would people learning how 
to configure the mechanisms properly.

The problem is that for years email was virtually unregulated (open 
relays were commonplace 15 years ago) which is why it is now so widely 
abused. The only chance of cutting back abuse is to tighten things up. 
Saying that attempts to tighten it up are "fatally flawed" just because 
people can't be bothered to do things properly is dooming the whole 
system to failure.

Extra authentication and authorisation is necessary in email. However 
these are done (SPF, DKIM, a 'web of trust', trusted authority, or 
whatever) will need more rigourous configuration and management than 
there has been in the past. If we can't accept a requirement for more 
rigourous configuration and management, we may as well all give up now.

I would say that giving a 5yz error in response to an SPF failure would 
be a good thing to do (silently discarding would not be). Accepting an 
SPF failed message just means that any bad configuration/behaviour goes 
"unpunished" and defeats the whole objective of having the SPF 
protection in the first place. In most cases, the sender should prefer 
to know that there's a problem with their SPF/DKIM configuration (so 
they can fix it), rather than it to be silently ignored all the time. 
IME, people set up SPF because they don't want their email addresses to 
be spoofed, not just as a fun way to spend half an hour.

> (I did some back of the envelope testing of ADSP as an anti-phishing / anti=spoofing measure, and based on my inbox it was worse than flipping a coin for each email, based on both false positives a…
Is that just because it's poorly configured in many places? If so, then 
rejecting any failed messages will surely, over time, increase the 
effectiveness as people fix their configurations. Accepting them anyway 
will just keep the bad configurations in place. (i.e. 'Tough love')

Big mail providers are happy rejecting mail for any and all spurious 
reasons, why can't others be willing to reject mail for badly configured 
senders?

---

From: Steve Atkins <[email-masked]>
Date: Sun, 23 Oct 2011 18:37:05 -0700

On Oct 23, 2011, at 5:19 PM, Paul Smith wrote:

> Is that just because it's poorly configured in many places? If so, then rejecting any failed messages will surely, over time, increase the effectiveness as people fix their configurations. Accepting them anyway will just keep the bad configurations in place. (i.e. 'Tough love')

That was based on modeled "perfect" ADSP-compliant behaviour by the recipient, and actual observed behaviour by paypal and phishers going after their brands. Paypal are the main folks backing ADSP, so if they can't get it right, nobody can. 

The change I saw them make in response to these problems was to have their employees use a domain other than paypal.com for paypal corporate email. That improved the false positives (in much the same way sending your email via Hotmail will prevent false positives based on your domains misconfiguration, which isn't really fixing the problem). It had no effect on the false negatives.

> Big mail providers are happy rejecting mail for any and all spurious reasons, why can't others be willing to reject mail for badly configured senders?

That's an entirely different issue - they, like all responsible email providers, want to deliver email their users want.

Cheers,
  Steve

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 24 Oct 2011 01:33:52 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Chris Lewis
> Sent: Tuesday, October 18, 2011 4:12 PM
> To: [email-masked]
> Subject: Re: [Asrg] Greylisting BCP
> 

Just to be clear, the intent of the BCP isn't to standardize greylisting, but rather to write down what it is in general, what the variants are, their respective advantages and disadvantages, and then recommend best practices for a few common environments.  Also valid for discussion would be why some sites think it's a bad idea overall.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 24 Oct 2011 01:33:53 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Martijn Grooten
> Sent: Wednesday, October 19, 2011 8:06 AM
> To: Anti-Spam Research Group - IRTF
> Do people apply greylisting post-DATA in practise? Or is this really

I plan to do a survey of popular implementations.  If none do, that section can be dropped.  But someone suggested including it in the outline which leads me to believe that some implementation does.

> include some caution about that.

Presumably a site using it turns it on because it perceives it has a problem, and notices a positive difference (or they'd turn it off).  It's subjective though, which can make it hard to measure.  So sure, a note about this might be reasonable.

> "greylisting is more expensive than not greylisting"

Put simply, any new filter has a cost to install, maintain, and operate.  For greylisting there has to be a local store of sources that you've seen that get to bypass greylisting, so anything not on that list gets caught.  That takes space, I/O, and processing time.  So "expensive" in the resources sense.

> I assume by "message" you mean the (sender, recipient, MTA) triplet?

This might get back to the DATA thing above.  I guess some greylisting takes into account whether or not a specific message is retried, not just the source.  It depends on whether or not I can find one that actually does this.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 24 Oct 2011 01:33:55 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Douglas Otis
> Sent: Tuesday, October 18, 2011 8:39 PM
> To: [email-masked]
> just not always with the same message.

That doesn't sound like a "retry" to me, in the MTA queueing sense.  For your claim to be true, it would mean bots institute MTA-style queue-and-retry systems, but that substantially increases the footprint on the infected machine.  It's been my impression that their reluctance to do this is precisely why greylisting is perceived to be effective.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 24 Oct 2011 01:33:56 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Peter J. Holzer
> Sent: Thursday, October 20, 2011 12:23 AM
> To: [email-masked]

The current outline will look at various tuples of data identifying the transaction.  This might include {IP, sender, recipient}, or any subset of those.

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 24 Oct 2011 11:40:29 +0200

On 24/Oct/11 02:19, Paul Smith wrote:
>> A third, which is a lot more common and complex to resolve than 
>> you'd expect, is organizations simply not knowing or not
>> controlling where their mail is sent from.

Forcing users to submit via official domain servers is one of the
points SPF is good at.

>> There've been cases both for SPF and ADSP ("SPF for DKIM") where
>> one group within a company put out a restrictive policy based on
>> their beliefs of where mail was sent from, and it all fell to
>> pieces when employees got kicked off mailing lists when mail
>> appearing to come from them was delivered from the mailing list
>> manager and rejected due to auth failure, or where internal
>> monitoring email sent from servers (cron, even) was silently
>> discarded.
>
> This is the problem of people who should know better not
> understanding what's going on.

Yes, if I've understood the OP's complaint correctly, someone should
have removed an SPF check altogether rather than leaving it in a
check-but-don't-enforce hybrid status.

> If the company had a policy that all mail from their domain had to
> go through certain servers, and employees were sending mail through
> other servers, then they deserved to get kicked off mailing
> lists... Just as they would deserve punishment for watching porn
> over the work Internet connection, etc.

For the record, you get kicked off a mailing list when you reject
mailing list messages.  This can happen when you enforce ADSP and thus
reject messages with broken author domain's signatures.  That is, the
duly compliant receiver gets punished in that case.

ADSP is more broken than SPF, with respect to forwarding.  SPF always
claimed to be valid for first hops only, and predicated to change the
envelope sender for further hops.  DKIM initially seemed to overcome
that limitation, but RFC 6377 finally clarified it cannot cope with
forwarding either (and nobody wants to change the value of From:).

> Saying that attempts to tighten it up are "fatally flawed" just
> because people can't be bothered to do things properly is dooming the
> whole system to failure.

Very much agreed.

With two policies out of two that choke on forwarding, I'd suspect the
culprit is the latter.  Indeed, forwarding implies that target
addresses are being kept on a server.  Hence, such server must have
some sort of authorization for using them.  Why don't we check that?

http://fixforwarding.org/

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 24 Oct 2011 07:16:25 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Jose-Marcio Martins da Cruz
> Sent: Wednesday, October 19, 2011 3:11 AM
> Cc: Chris Lewis

The point is not to make a general good/bad statement about greylisting.  The better approach, in my opinion, is to lay out the pros and cons and let the reader decide if it's right for his/her installation.

> It seems to me that this BCP mention only problems caused by the

That's not the intent.  It will also point out the benefits.

Good point.

> [...]

Thanks, I've added these to the outline.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 24 Oct 2011 07:17:59 -0700

> Sent: Wednesday, October 19, 2011 8:59 AM
> To: [email-masked]

Yuck.  OK, added.

>   a constant delay for every mail.

I think that's covered, in slightly more general terms.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 24 Oct 2011 07:34:10 -0700

Thanks to all for the feedback so far.  I've updated the draft to fold in the points people raised.  I'll start the work of actually adding text to the various sections after the next IETF meeting.

If any of you have specific text you'd like to see in there, please do send it along.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Murray S. Kucherawy
Sent: Tuesday, October 18, 2011 12:01 PM
To: [email-masked]
Subject: [Asrg] Greylisting BCP

Comments welcome.

-MSK

---

From: Chris Lewis <[email-masked]>
Date: Mon, 24 Oct 2011 18:11:58 +0200

> I plan to do a survey of popular implementations.  If none do, that section can be dropped.  But someone suggested including it in the outline which leads me to believe that some implementation doe…

As a FYI, some (perhaps only historical now) implementations behave 
badly with post-DATA rejects.  An old version of Exchange, in 
particular, treated even a post-data 5xx as "retry as quickly as possible".

But I've not seen one in a while.

---

From: Dave Warren <[email-masked]>
Date: Mon, 24 Oct 2011 16:38:37 -0700

> That doesn't sound like a "retry" to me, in the MTA queueing sense.  For your claim to be true, it would mean bots institute MTA-style queue-and-retry systems, but that substantially increases the…

Keep in mind that zombies don't necessarily need to create some copy of 
each message and "queue" it.  Rather, it's just as simple as keeping 
track of MAIL/RCPT pairs that should be retried and this doesn't need to 
incur much of a footprint at all.  Whether they have a DB entry to track 
which message they wanted to deliver or not is an implementation 
detail.  Either way, it's well within the capabilities of a zombie 
running on a midrange laptop/desjtop computer.

However, all of this misses a huge part of why greylisting can be a 
powerful tool: It's a happy accident that bots still don't retry 
properly.  I only greylist if I'm already suspicious based on rDNS 
patterns, mismatching reverse DNS, and similar.  I don't greylist 
because I care if they retry (although it's nice that so many still 
don't).  I greylist to let URIBLs and similar have a chance to catch up 
and detect a new zombie, to let my bayesian have time to learn, and to 
hope they hit one of my traps and get themselves blacklisted.

I don't advocate greylisting everything, I found it's too obnoxious for 
general purpose use here, but in cases where I'm already suspicious, 
it's a second chance for a poorly managed sender to get through some 
front-line filters.

-- 
Dave Warren, CEO
Hire A Hit Consulting Services
http://ca.linkedin.com/in/davejwarren

---

From: Jose-Marcio Martins da Cruz <[email-masked]>
Date: Tue, 25 Oct 2011 17:51:47 +0200

> That doesn't sound like a "retry" to me, in the MTA queueing sense.  For your claim to be true, it would mean bots institute MTA-style queue-and-retry systems, but that substantially increases the…

Retry or not retry... This just means that there are some possible ways to pass through greylisting 
and that greylisting isn't perfect... as any other filtering method, be an RBL, SPF, statistical 
filters or anything else.

The correct way to present it is something like :

"... to test a most of the time valid assumption that bots sending spam don't maintain state".

Maybe in the future this assumption won't be true anymore, but for the moment, it seems to me that 
it still holds.

--

---

From: Douglas Otis <[email-masked]>
Date: Tue, 25 Oct 2011 11:26:18 -0700

On 10/24/11 1:33 AM, Murray S. Kucherawy wrote:
>> On Tuesday, October 18, 2011 8:39 PM, Douglas Otis wrote:
>>
Not all RATs install simple SMTP proxies.  There is no reason to queue 
and retry messages.  They can receive a list of recipients separate from 
messages to reduce inbound traffic.  Marking transmission completion 
would allow repeated tuples compared by grey listing mechanisms.

-Doug

---

From: Douglas Otis <[email-masked]>
Date: Tue, 25 Oct 2011 13:41:09 -0700

> some sort of authorization for using them.  Why don't we check that?
Efforts aimed at determining an "IP address authorization" based upon 
some "message element" is flawed for two reasons:

1) Authorization is easily abused whenever reputation is applied against 
a "purported domain".  Outbound IP addresses are commonly shared, and 
this is becoming increasingly prevalent. The potential for abuse risks 
denial of service and legal responses as a singular recourse.

2) Even if magic happened and all specific "message elements" now 
attempt to offer "IP address authorizations", addresses seen by 
recipients may change at intervening SMTP proxies, LSNs, or CGNs.  This 
problem will become more pronounced as IPv6 sources increase and many 
MTAs fail to offer IPv6 connectivity.

Any effort to assign reputation to a range of IPv6 addresses immediately 
confronts a scale of active prefixes (ignoring lower 64 address bits) 
that is 65K larger than all possible IPv4 addresses.  This range is 
likely to increase another 6 fold in a few years.  A rate well beyond 
Moore's law.

A sensible solution is a cryptographic method to authenticate the domain 
of the outbound MTA.  DKIM does not authenticate the outbound MTA, and 
remains prone to abuse in a manner similar to IP address authorization.  
Perhaps a new type of SMTP needs to be developed, where the protocol 
remains unchanged with the exception of MTA authentication 
requirements.  Call it AMTP for Authenticated Mail Transport Protocol.

Once the MTA can be authenticated, the guesswork and mistakenly 
purported domain issues go away.  There would not be any need to grey 
list recipients once receivers are able to maintain and control who they 
permit to issue messages.  This may mean new domains need to solicit 
intended recipient domains to request inclusion.  Such a process 
predicated on authentication scales far better and would be less 
disruptive than reactive blocking of any "purported" abuse.

It also seems Kerberos offered by various third parties could help 
reduce the overhead related to MTA authentications.

-Doug

---

From: Jose-Marcio Martins da Cruz <[email-masked]>
Date: Wed, 26 Oct 2011 11:23:29 +0200

> RCPT is reduction of false positives.

In the case of MIMEDefang, the probable reason is to avoid the following situation : a bot try to 
send a message at some moment and some time later try to send a different message to the same 
recipient. So, reduction of false negatives.

Another reason to do greylisting during DATA is to apply greylisting only on messages which content 
score (evaluated by some kind of content filter) is greater than some threshold. In this case, not 
reduction of false positives, but applying greylisting only to messages already considered suspects.

Either way, one goal of original greylisting is precisely to avoid heavy content filtering checks. 
So, doing greylisting after DATA, may not be optimal.

---

From: Jose-Marcio Martins da Cruz <[email-masked]>
Date: Wed, 26 Oct 2011 11:46:34 +0200

> Put simply, any new filter has a cost to install, maintain, and operate.  For greylisting there has to be a local store of sources that you've seen that get to bypass greylisting, so anything not o…

You should take some care with this. Ressource usage varies a lot with different implementations : 
you'll find in memory chained lists, some RDBMS (e.g. MySQL) or ISAM/B-Tree/HASH (e.g. BerkeleyDB). 
Some implementations are smarter than others about ressources needes : e.g. how to expire old 
entries or how to aggregate them. And this makes a big difference on how expensive the filter is.

Put simply, when talking about greylisting, expensive or not is more related to the implementation 
and to some tuning (e.g. delays) than to the method itself. I'm not even talking about poor 
programming practices...

JM

--

---

From: Paul Smith <[email-masked]>
Date: Wed, 26 Oct 2011 10:51:10 +0100

Murray S. Kucherawy wrote:
>> Put simply, any new filter has a cost to install, maintain, and 
>> operate.  For greylisting there has to be a local store of sources 
>> that you've seen that get to bypass greylisting, so anything not on 
>> that list gets caught.  That takes space, I/O, and processing time.  
>> So "expensive" in the resources sense.
>>
>
On the other hand, if it stops a reasonable amount of spam, it reduces 
bandwidth, processing time and storage requirements, as well as human 
time & annoyance handling the spam.

It's up to the implementer to decide whether the cost of implementing 
greylisting is more or less than the cost of not implementing it.

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 26 Oct 2011 14:04:56 +0200

> some "message element" is flawed for two reasons:

I assume you mean SPF.  Flawed as it may be, it'd still be better than
nothing.

> A sensible solution is a cryptographic method to authenticate the
> domain of the outbound MTA.  DKIM does not authenticate the outbound
> MTA, and remains prone to abuse in a manner similar to IP address
> authorization.

Ditto.

> Perhaps a new type of SMTP needs to be developed, where the
> protocol remains unchanged with the exception of MTA authentication

Why not call it SMTP-AUTH?  Indeed, it's much stronger than SPF or DKIM.

> list recipients once receivers are able to maintain and control who
> they permit to issue messages.

Yep.

> This may mean new domains need to solicit intended recipient
> domains to request inclusion.

No, not only new domains.  Whenever someone prompts you for your email
address and your consent for using it, according to various laws, it
could/should store some form of auth token along with it.

That way, I'd address forwarded mail only.  That is, more or less,
what policies such as SPF and ADSP seem to be unable to handle
properly --I called it the culprit in the top quoted paragraph above.

Direct personal messages, where recipient addresses are set as part of
the message composition, need no authorization.  Of course, black
spammers will pretend to write personal messages, thereby breaking the
law, but grey spammers won't.

> Such a process predicated on authentication scales far better and
> would be less disruptive than reactive blocking of any "purported"
> abuse.

I concur.  The ability to mechanically tell legitimate messages will
then allow spam reporting to be a well defined activity.

---

From: Douglas Otis <[email-masked]>
Date: Fri, 28 Oct 2011 11:28:29 -0700

On 10/26/11 5:04 AM, Alessandro Vesely wrote:
> >> With two policies out of two that choke on forwarding, I'd
> >> suspect the culprit is the latter. Indeed, forwarding implies
> >> that target addresses are being kept on a server. Hence, such
> >> server must have some sort of authorization for using them. Why
> >> don't we check that?
>
> > Efforts aimed at determining an "IP address authorization" based
> > upon some "message element" is flawed for two reasons:
>
>  I assume you mean SPF. Flawed as it may be, it'd still be better
>  than nothing.

Not when SPF authorization is considered a basis in which to assess a 
domain's reputation.  This occurs at a time when large scale NATs (LSNs) 
are deployed by ISPs to support IPv6 transitions.  The inability to 
defend authorizations against reputation poisoning will foster 
dependence on providers protected by their "too big to block" status.  
The growing environment of shared outbound sources makes simple 
authorization fully unsuitable.

In addition, each email transaction demands DNS resolve all IPv4 and 
IPv6 hosts used by any globally recognized domain.  These DNS 
transactions may extend beyond 100 in the effort to authorize all 
outbound MTAs used by a single domain.  This exercise can lead to DDoS 
against domains not a party to either end of the specific email 
transaction.  SPF macro expansions defeat DNS caching and result in 
easily two orders of traffic amplification. :^(

There are many methods that might be used to authenticate outbound MTAs, 
such as SMTP Auth.  Ideally SMTP would include DANE extensions used in 
conjunction with Kerberos.  If it were not for DKIM ignoring prepended 
headers, it would have merit as an anti-phishing strategy.  Since DKIM 
does not verify who sent what to whom, it can only identify domains 
considered "too big to block" as well.

> > domain of the outbound MTA. DKIM does not authenticate the
> > outbound MTA, and remains prone to abuse in a manner similar to IP
> > address authorization.
>
>  Ditto.

What do you think is standing in the way of SMTP authentication?

> > protocol remains unchanged with the exception of MTA
> > authentication requirements. Call it AMTP for Authenticated Mail
> > Transport Protocol.
>

In the past, some of the objections had been cert costs.  DANE should 
remedy the cost issue, and ensure domains support DNSsec.  The increased 
overhead related with DNSsec demands greater attention given protocol 
dependent upon DNS that might lead to indirect high gain DDoS attacks.

> > purported domain issues go away. There would not be any need to
> > grey list recipients once receivers are able to maintain and
> > control who they permit to issue messages.
>
>  Yep.
>
>  No, not only new domains. Whenever someone prompts you for your
>  email address and your consent for using it, according to various
>  laws, it could/should store some form of auth token along with it.
>

Expectations any outbound MTA mitigate abusive accounts where reputation 
is based upon authentication of the MTA would have a profound outcome.   
Users in possession of compromised accounts/systems would be made aware 
and categorization of mail content (as you suggest) becomes moot.  Don't 
expect recipients are able to sort compromised accounts of large 
senders.  Placing that burden on recipients never scales.

>  Direct personal messages, where recipient addresses are set as part
>  of the message composition, need no authorization. Of course, black
>  spammers will pretend to write personal messages, thereby breaking
>  the law, but grey spammers won't.

Or allow new domains to quickly obtain the same status of any domain 
that quickly mitigates abuse.

>  then allow spam reporting to be a well defined activity.

Hopefully spam reporting will also play a reduced role as well.

-Doug

---

From: Paul Smith <[email-masked]>
Date: Sat, 29 Oct 2011 02:38:25 +0100

On 28/10/2011 19:28, Douglas Otis wrote:
> There are many methods that might be used to authenticate outbound 
> MTAs, such as SMTP Auth.  Ideally SMTP would include DANE extensions 
> used in conjunction with Kerberos.  If it were not for DKIM ignoring 
> prepended headers, it would have merit as an anti-phishing strategy.  
> Since DKIM does not verify who sent what to whom, it can only identify 
> domains considered "too big to block" as well.

Excuse me for being thick, I'm trying to understand your thoughts.

I can sort of see how you could AUTHENTICATE outbound MTAs, however 
that's not the problem as far as I can see. You can reliably 
authenticate an outbound MTA based on its IP address at the moment 
(although that would probably be less useful than authenticating a 
'group' of MTAs based on the owner, eg Yahoo, and will be less useful 
with IPv6). Some of the technicalities of authentication based on 
cryptographic means are a bit unclear, but I can see that it could 
probably be made to work.

However, once you have authenticated the MTA, you then need to decide 
whether to authorise it,and what to authorise it for. Are you talking 
about just authorising it to be able to send mail from a particular 
domain? (If so, I'm not sure how this fixes the 'forwarding problem'), 
or authorising it be able to send mail to me at all? (in which case, who 
would decide on that authorisation?)

(AFAICS Something like SRS should probably be able to fix the forwarding 
problem if the rest of the system is setup correctly)

You say "Ideally SMTP would include DANE extensions used in conjunction 
with Kerberos" - now, I'm not very familiar with DANE or Kerberos at the 
moment, but I think DANE lets you associate a certificate with a domain 
name using DNS - yes?. Since it wouldn't be possible for a sending MTA 
to authenticate using such a certificate based on the sender's email 
domain, I presume you mean it will use the certificate based on the 
MTA's reverse DNS entry? So, how would that link the sending MTA with 
the sender's email address?
And would Kerberos need a separate server? If so, where would that be, 
the sending or receiving end? (I'm not sure I like the idea of having to 
use Kerberos)

If you are just going to authenticate based on the MTA's reverse DNS, 
then why not just mandate TLS and authenticate the client certificate 
using DANE?

Could you not change SMTP to go something like:
EHLO sender
220 ENVSIGN CHALLENGE=RandomText

MAIL FROM: <[email-masked]>
220 OK
RCPT TO: <[email-masked]>
220 OK
RCPT TO:<[email-masked]>
220 OK
ENVSIGN: <PKI signature of 
[email-masked][email-masked][email-masked]>
220 OK

The sender signs the envelope data using the private key; the public key 
is exposed using DNS or something (DANE?) . Kerberus isn't needed, and 
the sending MTA can send from multiple different domains in a single 
session.

(Still doesn't solve forwarding without return path rewriting, or the 
general authorisation problem, but authenticates the sender MTA 
effectively, AFAICS)

---

From: Paul Smith <[email-masked]>
Date: Sat, 29 Oct 2011 10:16:27 +0100

On 29/10/2011 02:38, Paul Smith wrote:
>> Since DKIM does not verify who sent what to whom, it can only 
>> identify domains considered "too big to block" as well.
>
>

I've been thinking about forwarding

If you have A -> B, then server B forwards to server C, C can't do any 
authentication based on A, because A doesn't know about the forwarding 
(or it would, presumably, just send to C directly).

So, all sender domain authentication fails (without return path rewriting)

So, what you need is for C to be able to give B an authentication key 
for the forwarding. Then, B could pass that back to C with the message 
(possibly as a parameter to the RCPT command). The issue with this is 
that user intervention would be needed - so every time a user wants to 
subscribe to a mailing list,or set up a forwarding to their gmail 
account, they would need to go to the destination server, get a key from 
it, and give it to the forwarding server. This could show consent for 
any anti-spamming legislation, but could also be too complicated for 
many users to handle.

The authentication key would need to allow B to send *anything* to C for 
the relevant recipient, so ideally the key that C gives would be 
specific to B (to allow it to be revoked in the case of abuse)

Technically this wouldn't need to be hard at all, it's just the manual 
requirement for key exchange that would be an issue for many people.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sat, 29 Oct 2011 16:10:43 +0200

On 2011-10-23 14:43:00 -0700, Steve Atkins wrote:
> Mail forwarding is one common cause of failure. If you're sending mail
> to someone at @ieee.org and they forward it on to their @aol.com
> account, and you and AOL decide between you to enforce SPF -all then
> that mail will be undeliverable - and there's no way either you or AOL
> can predict that.

Actually, I can because I told ieee.org to forward my mail to aol.com
and I know that I enabled SPF for my aol.com address. There is nothing
unpredictable about that - I control both sides of the forwarding.

What AOL has to predict is that some of their customers may want to
forward from another mail address to their AOL mailbox and provide a way
to exempt specific sender IP addresses from SPF checks.

	hp

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 30 Oct 2011 13:11:26 +0100

> (or it would, presumably, just send to C directly).

Correct.  One may still be able to tell whether a message originated
at A, though.

> So, all sender domain authentication fails (without return path

That's the turn point.  We pine for domain authentication looking
forward to reckoning domain's reputation.  So far so good.  But a
simpler SMTP-AUTH suffices for authenticating a specific sender that
had been authorized beforehand.

> (possibly as a parameter to the RCPT command).

For SMTP AUTH, it is a parameter to the MAIL command.  After having
authenticated, B can use message-specific authorizations by

 MAIL FROM:<[email-masked]> AUTH=[email-masked]

C can check B is authorized, but must trust B uses authorizations
properly; that is, according to the message stream for which the
authorization was granted.

> The issue with this is that user intervention would be needed - so
> every time a user wants to subscribe to a mailing list,or set up a
> forwarding to their gmail account, they would need to go to the
> destination server, get a key from it, and give it to the
> forwarding server. This could show consent for any anti-spamming
> legislation, but could also be too complicated for many users to
> handle.

Exactly.  If law-makers had been clever, they would have provided for
a protocol to exchange the consent automatically.  Something similar
to paying with a credit card via a bank site.  Users would need to
tell only their domain name to the mailing list manager, or gmail
account.  The wannabe forwarder B would redirect them to their server
C where they can authenticate, configure any detail such as the target
folder, and grant their consent back to B.

> The authentication key would need to allow B to send *anything* to C
> for the relevant recipient, so ideally the key that C gives would be

Yes.  For non-transferable consent, the user's server C can pass an
opaque local-part to the wannabe forwarder B.  That way, B won't even
know the full email address of the user.

> requirement for key exchange that would be an issue for many people.

It is easy technically, but not politically.  In Europe, a
consent-exchange protocol would get some traction if it could replace
the tedious paperwork that Directive 95/46/EC requires.  For that,
we'd need the Art.29 Working Party to participate in an IETF Working
Group so as to standardize and bless that protocol at the same time.
Not an easy task.

---

From: Douglas Otis <[email-masked]>
Date: Mon, 31 Oct 2011 11:28:35 -0700

On 10/28/11 6:38 PM, Paul Smith wrote:
> > MTAs, such as SMTP Auth. Ideally SMTP would include DANE
> > extensions used in conjunction with Kerberos. If it were not for
> > DKIM ignoring prepended headers, it would have merit as an
> > anti-phishing strategy. Since DKIM does not verify who sent what
> > to whom, it can only identify domains considered "too big to block"
> > as well.
>  domain? (If so, I'm not sure how this fixes the 'forwarding
>  problem'), or authorising it be able to send mail to me at all? (in
>  which case, who would decide on that authorisation?)

David Black and myself authored a draft describing a method to authorize 
any number of domains using a single small DNS transaction.  Murray also 
authored a similar version once this draft expired.  Attempting to 
publish entire IP address sets for email domains is highly problematic, 
especially when misapplied as a basis for assessing accountability.

Use of IP address authorization based upon entire address sets will 
become highly disruptive from either a DDoS or rejection basis.  IP 
address translations occurring at providers that can not be anticipated 
by senders.  Any expectation that possible LSNs will be white-listed 
would permit any amount of abuse.

>  (AFAICS Something like SRS should probably be able to fix the
>  forwarding problem if the rest of the system is setup correctly)

Rather than expecting recipients to sort emails based upon message 
elements not authenticated with those who issued a message to specific 
recipients (which DKIM does not do), a more effective scheme holds 
outbound MTAs accountable for mitigating abuse.  MTAs failing to respond 
would be blocked.  Requiring outbound MTA to respond better informs 
individuals when accounts or systems are compromised and is the most 
effective method for controlling abuse.

>  You say "Ideally SMTP would include DANE extensions used in
>  conjunction with Kerberos" - now, I'm not very familiar with DANE or
>  Kerberos at the moment, but I think DANE lets you associate a
>  certificate with a domain name using DNS - yes?. Since it wouldn't be
>  possible for a sending MTA to authenticate using such a certificate
>  based on the sender's email domain, I presume you mean it will use
>  the certificate based on the MTA's reverse DNS entry? So, how would
>  that link the sending MTA with the sender's email address?

Some delivery issues occur when recipients are unaware their servers 
lack adequate resources where connections fail and yet these failure are 
not logged.  Doing any type of cryptographic authentication adds 
overhead, where a hybrid Kerberos/SMTP approach provides an efficient 
method with increased control.  Those wanting to obtain fast processing 
would first authenticate with a designated Kerberos server located by 
SRV records.  Once authenticated, a data-structure (perhaps published 
using APL RFC2133) could be accepted to open firewalls at the receivers, 
where individual SMTP transactions first exchange long-lived easy to 
process Kerberos tickets.

>  And would Kerberos need a separate server? If so, where would that
>  be, the sending or receiving end? (I'm not sure I like the idea of
>  having to use Kerberos)

If you own a Mac, Apple provides users Kerberos services to enhance 
security on otherwise insecure networks.  Any large domain could offer 
their own Kerberos server where senders would obtain a ticket once every 
10 hours for example.  It is also likely Kerberos services will be 
offered by various third-parties as well.

>  then why not just mandate TLS and authenticate the client certificate

rDNS is highly problematic because this involves separate 
administrations.  Those running a service are not necessarily delegated 
to publish rDNS records.  With the IP address ranges permitted by IPv6, 
rDNS is also unlikely to be useful for differentiating different 
categories of IP addresses.  It is not practical to expect SMTP to query 
rDNS against every IPv6 connection.  It is also not practical to expect 
a third-party will be publishing this information either.

>  Could you not change SMTP to go something like: EHLO sender 220
>  ENVSIGN CHALLENGE=RandomText
>
>  MAIL FROM: <[email-masked]> 220 OK RCPT TO: <[email-masked]> 220
>  OK RCPT TO:<[email-masked]> 220 OK ENVSIGN: <PKI signature of
>  [email-masked][email-masked][email-masked]> 220 OK
>
>  The sender signs the envelope data using the private key; the public
>  key is exposed using DNS or something (DANE?) . Kerberus isn't
>  needed, and the sending MTA can send from multiple different domains
>  in a single session.

Kerberos was suggested as a way to avoid much of the overhead related 
with processing certificates at each exchange.   It also provides a way 
to offer layered protections, such as selectively enabling firewall 
subsequent to completion of a Kerberos exchange.  Kerberos exchanges 
would be at much much lower rates than that demanded by SMTP.

>  general authorisation problem, but authenticates the sender MTA

IMHO, the IP address authorization scheme was poorly considered.  With 
more than 95% of SMTP transactions being abusive, expecting recipients 
to then perform as many as 100 additional SPF DNS transactions and any 
number of reputation transactions per message is neither practical nor 
safe.  In the face of IPv6, any IP address authorization requirement 
will also be highly disruptive.

There is a practical way for each domain to authorize other domains.  
Whether this would be needed after outbound MTAs have been held 
accountable is yet to be determined.

-Doug

---

From: John Leslie <[email-masked]>
Date: Mon, 31 Oct 2011 14:46:16 -0400

Douglas Otis <[email-masked]> wrote:
> 
> David Black and myself authored a draft describing a method to
> authorize any number of domains using a single small DNS transaction.

   Reference, please: I don't think I'm familiar with this.

> Murray also authored a similar version once this draft expired.

   Likewise...

> Attempting to publish entire IP address sets for email domains is
> highly problematic, especially when misapplied as a basis for
> assessing accountability.

   I quite agree.

--
John Leslie <[email-masked]>

---

From: Douglas Otis <[email-masked]>
Date: Mon, 31 Oct 2011 13:26:37 -0700

On 10/31/11 11:46 AM, John Leslie wrote:
> Douglas Otis<[email-masked]>  wrote:
http://tools.ietf.org/html/draft-otis-dkim-tpa-label-06
>> Murray also authored a similar version once this draft expired.
http://tools.ietf.org/html/draft-kucherawy-dkim-atps-10

The difference is tpa-label attempted to include methods to identify 
outbound MTAs, and was not limited to just authorized DKIM signature chains.

-Doug

---

From: Paul Smith <[email-masked]>
Date: Mon, 31 Oct 2011 22:06:31 +0000

On 31/10/2011 18:28, Douglas Otis wrote:
> with processing certificates at each exchange.   It also provides a 
> way to offer layered protections, such as selectively enabling 
> firewall subsequent to completion of a Kerberos exchange.  Kerberos 
> exchanges would be at much much lower rates than that demanded by SMTP. 
I'm struggling with the Kerberos idea

If you want to send a message to me, that means you need to authenticate 
with 'my' kerberos server? What authentication details do you use? Do I 
have to contact you to give you authentication details to my server? 
Surely that can't be the case, but how else would the authentication 
work?    (or would the sender authenticate *with itself* via the 
receiver? I can see that working in theory, but would be complicated and 
I can't see how it would work with a standard kerberos implementation)

Given that most of our customers are business with < 25 users, many of 
whom have their MX records pointing to their own MTA, how would this 
work? Most want to use their own MTA because their ISPs are useless, and 
switching to their own MTA gives them enhanced reliability and control.

I can't see them rushing to sign up to a third party kerberos server 
(probably with extra fees and unknown reliability). Does this mean we'll 
be needing to create our own kerberos server to run alongside our mail 
server software? Most of our customers use Windows desktop OSes (eg 
Windows XP, Windows 7 etc), which, AFAIAA don't give you kerberos services.

Using kerberos only makes sense if everyone who has a legitimate mail 
server has kerberos, which obviously isn't the case.

> with processing certificates at each exchange.
Does this mean that STARTTLS is undesirable as well?

Is the calculation of a signature that problematic? It's relatively CPU 
intensive, but low bandwidth. I'd have thought most mail systems would 
be I/O bound rather than CPU bound (unless they're doing 
antispam/antivirus, in which case a calculating/checking a signature is 
a relatively miniscule extra load). DKIM already generates/checks 
signatures, and with much more data & complexity.

---

From: Bill Cole <[email-masked]>
Date: Tue, 01 Nov 2011 02:41:18 -0400

Paul Smith wrote, On 10/23/11 8:19 PM:
>> Mail forwarding is one common cause of failure. If you're sending mail to
>> someone at @ieee.org and they forward it on to their @aol.com account, and
>> you and AOL decide between you to enforce SPF -all then that mail will be
>> undeliverable - and there's no way either you or AOL can predict that.
> I thought most people did return-path rewriting for this case. If you don't
> do that, then you get undesirable behaviour anyway - eg if the AOL mailbox
> is full, you get a bounce message from AOL, even though you didn't send any
> messages to AOL at all.

Yes, that happens. No, return-path rewriting is the exception, not the rule. 
People still use Sendmail aliases and ~/.forward mechanisms and functional 
work-alikes in other MTA's and it breaks the utility of explicitly 
derogatory SPF, both "-" and "~"

However, as often as the classical forwarding model is brought up as an 
example of SPF breakage, it really isn't the critical failure mode for 
simplistic SPF usage, since classical forwarding is mostly a tool of people 
who remember the days before the Eternal September. We're an influential 
bunch, yes, but there are not enough of us to break something that would 
otherwise work.

> Similarly, I thought return-path rewriting was standard for mailing lists.
> Especially ones which do VERP, but usually even in ones which don't - eg the
> return path on the message I received from you was

Yes, any mailing lists using reasonable list management tools rewrite 
messages with their own return-path and cause no trouble for SPF. There are 
some things called mailing lists that use 'exploder' features of various 
MTA's which don't use their own return-paths or any distinct software, but 
those sorts of lists were basically dysfunctional before spam was a 
significant problem. Like classical forwarding, that is also not the hardest 
obstacle for clean and simple use of derogatory SPF results.

>> A third, which is a lot more common and complex to resolve than you'd
>> expect, is organizations simply not knowing or not controlling where their
>> mail is sent from. There've been cases both for SPF and ADSP ("SPF for
>> DKIM") where one group within a company put out a restrictive policy based
>> on their beliefs of where mail was sent from, and it all fell to pieces
>> when employees got kicked off mailing lists when mail appearing to come
>> from them was delivered from the mailing list manager and rejected due to
>> auth failure, or where internal monitoring email sent from servers (cron,
>> even) was silently discarded.
> what's going on.

Yes. That is why this problem is the truly hard one for application of SPF 
in derogatory ways.

Deprecating technical mechanisms via "standards" (to whatever degree a 
low-grade RFC can be called that) and real-world bad outcomes has a 
successful track record of working to extinguish their use or encourage 
workarounds.

Deprecating human carelessness and ignorance has no successful track record.

NO SUCCESSFUL TRACK RECORD

In other words: THIS DOES NOT WORK!

> If the company had a policy that all mail from their domain
> had to go through certain servers, and employees were sending mail through
> other servers, then they deserved to get kicked off mailing lists... Just as
> they would deserve punishment for watching porn over the work Internet
> connection, etc. If they had a policy that mail had to go through certain
> servers, and configured SPF for different servers, then they shouldn't be
> managing mail systems.

That's nice. The fact is that sysadmins follow Sturgeon's Law just like any 
other large population. I'm very sorry about that. I really am.

Beyond that sad fact of life, there are other issues with "-all" (and even 
"~all") grounded in forms of dumb that are external to the administrators of 
any domain setting SPF records. For example, diverse online service 
providers including resume search systems, news clipping agencies, magazine 
publishers, and so on (and on and on) will send email on behalf of 
registered users with any envelope sender address that they have for the 
user. Whether it is a "mail a friend this article" link at the NYT or a 
"find me candidates for this job" form at a headhunter agency, there are a 
lot of ways that random 3rd parties can end up sending mail claiming to be 
from a user of a domain whose SPF maintainers cannot predict them. I don't 
like that, I think it is wrong and bad. However, it is my direct experience 
that fighting that practice is a consistently losing battle because the 
practice rarely fails, is easily accommodated, and is attractive to those 
who engage in it because it lets them completely abdicate responsibility for 
the mail they generate and send.

(I apologize for the overly complex sentences... )

> return-path rewriting would be de rigeuer as would people learning how to
> configure the mechanisms properly.

This would require universal deployment of people designing and running mail 
systems who are almost entirely not stupid, evil, irresponsible, lazy, or 
even overworked.

Good luck with that. If you figure out how to find clueful people in a 
reliable way, you can rule the world.

In the real world, the result is not (as some have claimed here) that SPF is 
completely useless. It's just not a tool you can use without paying 
attention to its many failure modes and particularly the ones where 
ignorance and hubris are likely to cause trouble. The canonical example is 
to counter phishing, where the sender domain is not used by random human 
users and legitimate sources are actually known. The next most commonly 
cited example is to recognize SPF records that consist of "-all" as the only 
element, stating that there is no such thing as legitimate mail from the 
domain. Stranger uses can be based on the recognition that there are 
spammers who know about SPF and about simple spam filtering techniques; 
being "too good" while being "very new" is a sign that a sender is a 
spammer, and an affirmative SPF result can be a "tell." (yes, I have SA 
meta-rules, no, I'm not sharing) If there is a general rule, it is that 
SPF's uses require a sender domain (good or bad) whose admins understand SPF 
well.

The bottom line is that SPF is actually useful as a small part of a spam
control system, but not in the simplistic way that it was designed to be
used or in ways that are practical for the giant providers of cheap/free
mailboxes. If you want to use SPF to control spam, then you need to have
someone skilled watching your mail stream. If you want to hand out cheap
mailboxes to anyone who wanders by, you will not find SPF to be useful.