Subject: [Asrg] misconception in SPF

From: Christian Grunfeld <[email-masked]>
Date: Thu, 6 Dec 2012 16:05:07 -0300

Hi,

Something I found about SPF. I don't know if it is new to you but it
is worth of explanation !

As in all the tutorials of SPF, one stays relaxed when lists hosts/ips
that are authorized to send for the domain and then finally closes
with -all. This is true only for that domain but if there are hosts or
subdomains with A records, they must enforce SPF policies ! (this is
not explicit in RFC or at least confusing).

A single host or subdomain that don't do it can lead to a sender spoofing attack
and pose as someone from our domain without the SPF to detect it.

This is because when we receive mail from [email-masked] the SPF
protocol seeks TXT records corresponding to bar.mydomain instead of
TXT record for mydomain. If there is an A record for bar.mydomain but
no TXT "v=spf1 -all" for it, SPF returns "none" although SPF exists
for mydomain !

Thus an attacker can inject an email from everywhere claiming that is
from [email-masked] !

As the default policy of SPF should not be "fail if not present" the
solution is to enforce with the records TXT "v = spf1-all" for each A
record that should not send emails!
This also can be made by means of wildcards but it is discourage in the RFC.

Cheers

---

From: Christian Grunfeld <[email-masked]>
Date: Thu, 6 Dec 2012 17:15:01 -0300

here is the proof of concept ! an email sent to John Levine claiming
to be from [email-masked] !!!

Could you please John paste the Received-SPF checks of your mailserver ?

Connected to mail1.iecc.com.
Escape character is '^]'.
220 mail1.iecc.com mailfront ESMTP
EHLO www.johnlevine.com
250-mail1.iecc.com
250-SIZE 0
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 PIPELINING
MAIL FROM: [email-masked]
250 2.1.0 Sender accepted.
RCPT TO: [email-masked]
250 2.1.5 Recipient accepted.
DATA
354 End your message with a period on a line by itself.
From: [email-masked]
To: [email-masked]
Subject: p o c

Hi John, just a proof of concept
.
250 2.6.0 Accepted message qp 34000 bytes 540
quit
221 2.0.0 Good bye.
Connection closed by foreign host.

---

From: Derek Diget <[email-masked]>
Date: Thu, 06 Dec 2012 15:34:52 -0500 (EST)

On Dec 6, 2012 at 16:05 -0300, Christian Grunfeld wrote:
=>Something I found about SPF. I don't know if it is new to you but it
=>is worth of explanation !
=>
=>As in all the tutorials of SPF, one stays relaxed when lists hosts/ips
=>that are authorized to send for the domain and then finally closes
=>with -all. This is true only for that domain but if there are hosts or
=>subdomains with A records, they must enforce SPF policies ! (this is
=>not explicit in RFC or at least confusing).

See <http://www.openspf.org/FAQ/The_demon_question> and
<http://www.openspf.org/FAQ/Common_mistakes#all-domains>

Note I have not followed the above at $WORKPLACE or my personal domains, 
because I would also want to follow RFC 6652 - Sender Policy Framework 
(SPF) Authentication Failure Reporting Using the Abuse Reporting Format.  
The problem is I don't want to publish MX records on all of the domains
and then configure the MX server to handle the thousands of new 
"domains" to support the ra modifier since it (ra) can't specify the 
domain part.  (I understand why it was done this way, but don't see how 
it scales with the this issue (SPF for non-mail sending domains).

-- 
***********************************************************************
Derek Diget                            Office of Information Technology
Western Michigan University - Kalamazoo  Michigan  USA - www.wmich.edu/
***********************************************************************

---

From: Martijn Grooten <[email-masked]>
Date: Thu, 6 Dec 2012 20:47:45 +0000

So, to use a real-world example, what you're saying is that if you want to spoof Twitter, to avoid your email being caught by the -all in Twitter's SPF record, you could use foo.twitter.com as the sending domain and your email wouldn't be blocked because of an SPF fail.

I think that's correct.

You could also use aimport dot no (as some spammer sending a fake Twitter email did an hour ago). That domain doesn't have an SPF record either.

As we're talking about the MAIL FROM in the SMTP envelope, which usually isn't shown to the user, I don't think this is a big problem.

Perhaps your MTA or spam-filter does use the MAIL FROM in its decision whether to deliver the email or not. If it decides to deliver the message because it claims to come from Twitter, uses a subdomain of twitter.com and didn't fail SPF than that's very wrong. But I don't think it's SPF's fault.

Martijn.

________________________________

Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.

---

From: "John Levine" <[email-masked]>
Date: 6 Dec 2012 21:21:16 -0000

>to be from [email-masked] !!!

Life is too short to publish SPF -all records for every host name that
doesn't send mail.

R's,
John

---

From: Christian Grunfeld <[email-masked]>
Date: Thu, 6 Dec 2012 23:09:44 -0300

> You could also use aimport dot no (as some spammer sending a fake Twitter email did an hour ago). That domain doesn't have an SPF record either.

simple users are more confident if the sender seems real !

> As we're talking about the MAIL FROM in the SMTP envelope, which usually isn't shown to the user, I don't think this is a big problem.

faking From: header is as simple as faking MAIL FROM envelope !

> Perhaps your MTA or spam-filter does use the MAIL FROM in its decision whether to deliver the email or not. If it decides to deliver the message because it claims to come from Twitter, uses a subdo…

I didn't say is SPF's fault. It's our fault

---

From: Paul Smith <[email-masked]>
Date: Fri, 07 Dec 2012 08:31:22 +0000

> Life is too short to publish SPF -all records for every host name that
While I agree totally, I understand the OP's point about semi-tech-savvy 
people being more trusting of Twitter mail coming from 
'bibble.twitter.com' than if it came from 'random.ru'

This problem is really due to the (IMHO horrible) allowance for an A 
record to be sufficient for mail delivery. However, it would be quite 
hard to remove that allowance nowadays. I don't know the stats for how 
many email addresses use A records for delivery rather than MX, but I'd 
guess its a significant number.

As a random thought, would there be the possibility to add some sort of 
marker on a parent domain to say 'we understand MX records, so we don't 
use A records for mail within this domain'? So, if you receive mail from 
'bibble.twitter.com', you check the TXT records for 'twitter.com' which 
tell you that subdomains/hosts without an MX record won't have mail, and 
since there isn't an MX record for 'bibble.twitter.com', you can reject 
it/treat it as spoofed.

The main problem I see with this is the increased DNS lookups - if you 
got a message from aa.bb.cc.dd.ee.ff.twitter.com (an excessive case), 
you'd have to do 6 or 7 extra DNS lookups before doing anything else, 
because you can't automatically tell where the 'parent' domain is. For 
small hosts this probably wouldn't be that big a problem. For big ones 
it may be, but then again, would any benefit this gave in reduced 
traffic outweigh the extra load, and would the DNS caches on the big 
hosts alleviate the problem?

(There'd also be a risk that a parent domain owned elsewhere (eg .com) 
would block A record mail for all its legacy subdomains, but you'd hope 
that wouldn't happen...)

Personally, I think I'd rather any effort went into some sort of 
'stronger' authentication being developed for prime phishing attack 
targets like Twitter, Facebook, banks etc.  eg a verification scheme 
which can verify specific messages, or a message collection system 
(rather than delivery) which can be automated by the recipient software, 
or something.

-

Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

---

From: Christian Grunfeld <[email-masked]>
Date: Fri, 7 Dec 2012 13:23:04 -0300

2012/12/7 Paul Smith <[email-masked]>:

> This problem is really due to the (IMHO horrible) allowance for an A record
> to be sufficient for mail delivery. However, it would be quite hard to
> remove that allowance nowadays. I don't know the stats for how many email
> addresses use A records for delivery rather than MX, but I'd guess its a
> significant number.

I think you are confused about the MX on the receiver side and the A
record of the sender !
Mails are sent to MX for a domain but the sender has not to have a MX
record to send !

The problem here is that the sender uses a forged address which has an
A record. What you said is that the recipient should not be an A
record and should be an MX. They are two different things !

> marker on a parent domain to say 'we understand MX records, so we don't use
> A records for mail within this domain'? So, if you receive mail from
> 'bibble.twitter.com', you check the TXT records for 'twitter.com' which tell
> you that subdomains/hosts without an MX record won't have mail, and since
> there isn't an MX record for 'bibble.twitter.com', you can reject it/treat
> it as spoofed.

same as above. MX for bibble.twitter.com is only to receive emails.
Nothing prevents [email-masked] to send unless you put a
TXT "v=spf1 -all" for it !

---

From: Paul Smith <[email-masked]>
Date: Fri, 07 Dec 2012 16:53:16 +0000

> Mails are sent to MX for a domain but the sender has not to have a MX
Oh, I know that. But a reasonable test is that if a message comes from 
an email address which you cannot reply to, then the sender email 
address is probably forged. (This is how call-back verification (CBV) works)

So, if you could ONLY send to MX addresses, the OP's issue wouldn't be a 
significant problem.

Currently, I could send a message from [email-masked]', and 
even though there are no MX records for 'www.microsoft.com', there is an 
'A' record, so a quick check cannot tell that the sender address is 
invalid. I can look for SPF records, but there is no SPF record for 
'www.microsoft.com', so I assume they don't use SPF, and let the message 
through.

(I could use CBV, but since that would also help with the OP's problem, 
I'll assume we aren't doing that)

If (in the past) it had been mandated that you can only send mail to an 
MX record server, then if a message comes from [email-masked]', 
I can quickly tell that it is probably a bad email address, since there 
is no MX record for www.microsoft.com. This would make it easier at the 
cost of 5 seconds more configuration time per domain. Unfortunately, 
it's too late to change that now.

> TXT "v=spf1 -all" for it !
But it would help tremendously, without needing to add SPF records for 
each host in a domain.

This is because there would be no MX record for 'bibble.twitter.com', so 
you could assume (because of this 'new rule') that that sender email 
address is invalid, because there is no way of replying to it.

Yes, the MX is for receiving mail only, according to the SMTP standard, 
BUT if you work on the assumption that you have to be able to reply to 
the sender (which is a common enough assumption), then it ALSO has to be 
valid for sending mail.

-

---

From: [email-masked]
Date: Fri, 7 Dec 2012 13:19:08 -0500

On 12/07, Paul Smith wrote:
> But it would help tremendously, without needing to add SPF records
> for each host in a domain.
> 
> This is because there would be no MX record for
> 'bibble.twitter.com', so you could assume (because of this 'new
> rule') that that sender email address is invalid, because there is
> no way of replying to it.
> 
> Yes, the MX is for receiving mail only, according to the SMTP
> standard, BUT if you work on the assumption that you have to be able
> to reply to the sender (which is a common enough assumption), then
> it ALSO has to be valid for sending mail.

I think this makes sense, but I think it would make more sense if there was
a way to just specify in the SPF record for, for example, twitter.com, that
all legit senders for all subdomains are included in the highest level SPF
record.  

I don't know, I'm not a huge fan of SPF at this point, I'm not sure it's
worth the work.  

-- 
"Forget not that the earth delights to feel your bare feet and the winds
long to play with your hair." - Kahlil Gibran
http://www.ChaosReigns.com

---

From: SM <[email-masked]>
Date: Fri, 07 Dec 2012 09:16:17 -0800

Hi Paul,
At 08:53 07-12-2012, Paul Smith wrote:
>to reply to the sender (which is a common

Actually no, or else people would not be using noreply@example.

Regards,
-sm

---

From: [email-masked]
Date: Fri, 7 Dec 2012 13:45:28 -0500

On 12/07, SM wrote:
> >standard, BUT if you work on the assumption that you have to be
> >able to reply to the sender (which is a common
> 

I think that behavior may be in violation of an RFC.  
(By which I do not mean to imply that it doesn't happen, or that blocking
emails which do it would work out for everybody.)

-- 
"Democracy is the theory that the common people know what they want,
and deserve to get it good and hard." - H. L. Mencken
http://www.ChaosReigns.com

---

From: Daniel Feenberg <[email-masked]>
Date: Fri, 7 Dec 2012 14:14:14 -0500 (EST)

On Fri, 7 Dec 2012, SM wrote:

>> the sender (which is a common
>

Anyone sending mail from [email-masked] would not be seriously 
inconvenienced if they were required to have a working MTA at example.com, 
even if they wanted to use a non-working username at that domain. In fact, 
that is what I generally see in such mail.

daniel feenberg

>
> Regards,
> -sm 
> _______________________________________________
> Asrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/asrg
>

---

From: Paul Smith <[email-masked]>
Date: Fri, 07 Dec 2012 19:33:58 +0000

> Actually no, or else people would not be using noreply@example.
Do people really send messages with a return path of 'noreply@example'? 
Or, do you mean that people use a return path of 'noreply@<their own 
domain>'?

With the former, I'd expect a large number of those messages to be 
blocked/discarded (it's common to do call back verification, or just 
simply check that the sender domain exists). I'd only expect that to be 
done by people who don't really understand what's going on. I know we 
have to deal with quite a few cases where our customers have errors or 
missing messages, which turn out to be because they've misspelled the 
domain part of their email address, and their mail is being 
blocked/discarded because the return address is invalid.

-

---

From: "John Levine" <[email-masked]>
Date: 7 Dec 2012 20:45:54 -0000

>all legit senders for all subdomains are included in the highest level SPF

This sort of thing has been proposed before.  It turns out that
anything in the DNS that starts "all names below this node ..." is
astonishingly hard to implement.

(Yes, I know about zone cuts.)

I do agree that it would have been really nice if the SMTP fallback
from MX to A records had been deprecated and killed back in the 1980s.
I tried to get RFC 5321 not to add fallback to AAAA but the powers
that be insisted that it was already too late.

---

From: "John Levine" <[email-masked]>
Date: 7 Dec 2012 20:50:03 -0000

>simply check that the sender domain exists).

Checking that the sender domain exists is quite common, cheap, and has
a very low failure rate.

SMTP callbacks are one of those bad ideas that just won't go away.
They're quite abusive (consider the 95% of mail that is spam with
someone else's return address) and don't work, since your idea of what
I should say in response to your MAIL FROM and RCPT TO commands is
probably not the same as my idea of what I actually do say.

R's,
John

PS: The wiki at http://wiki.asrg.sp.am has a taxonomy of anti-spam
techniques, both the good and the bad.  See if your favorite or least
favorite is listed there, and if not, please add it.  If you don't
have a login to edit, just drop me a note and I'll give you one.

---

From: Rich Kulawiec <[email-masked]>
Date: Fri, 7 Dec 2012 16:10:30 -0500

> probably not the same as my idea of what I actually do say.

+1

And:

Callbacks enable DDoS-by-proxy attacks. [1]

And they violate a fundamental principle of network abuse mitigation:
never deal with bad traffic by generating more traffic, it doesn't scale.

Moreover: even if they worked perfectly, they'd still have no anti-spam
value.  Knowing that an address is valid tells you nothing about the
intentions of either (a) its putative owner or (b) the current owner,
who, as we know, are often not the same entity.

---rsk

[1]  Consider, as one scenario out of many possibilities: the target
is example.net.  Attacker registers example[1-500].info.  Attacker sets
MX for all 500 of those domains to the MX for example.net.  Attacker uses
50K bots to open 10 simultaneous connections each to 500K distinct MX's
for other domains.  All 500K of those MX's look at putative sender (e.g.,
[email-masked]), look up MX for that domain (which is of course
the MX for example.net), open up a connection to it to do a callback...

---

From: SM <[email-masked]>
Date: Fri, 07 Dec 2012 13:32:00 -0800

Hi Paul,
At 11:33 07-12-2012, Paul Smith wrote:
>Do people really send messages with a return path of 
>'noreply@example'? Or, do you mean that people use a return path of 
>'noreply@<their own domain>'?

I used "example" as an example.  I misunderstood what you meant by 
reply.  I was referring to what's used as the author address.

I don't recall whether noreply@example is used in the envelope for 
delivery failures (assuming valid messages).  There are valid 
messages with an invalid Return-Path.  This may due to some 
assumptions about how email works.

>simply check that the sender domain

Call-back verification can be used to trigger a Denial of Service.

Regards,
-sm

---

From: "Bill Cole" <[email-masked]>
Date: Fri, 07 Dec 2012 17:07:24 -0500

On 6 Dec 2012, at 16:21, John Levine wrote:

> In article 
> <CAFduganiaDkf0jFsV7FYcAvi9JjA9G-iTj8XLVzcDNY=jo_m=[email-masked]> 
> you write:

Well, not really. It's not 1999: no one competent and sane manages zones 
of non-trivial size or significance by editing canonical-format zone 
files in a 24x80 text editor. As a matter of standard practice, many 
domains have long been managed with an awareness that the MX/A issue 
creates problems which are mitigated by never creating an A record 
without also creating an MX, with non-mailing hosts getting MX records 
that are easily detected as a declaration of that status. This is useful 
because there are spam exclusion tactics which detect such declarations 
and so reject spoofed addresses. It is a simple matter with many (most?) 
modern MTA's to reject mail in SMTP before DATA if the sender domain has 
a bogus MX record. The cleanest form of bogosity is the "nullmx" 
approach proposed and used by Yahoo, but it is also useful for receivers 
to detect MX records pointing to names without A records or with A 
records pointing to  unusable addresses (i.e. permanently reserved 
and/or otherwise special address ranges.) If a domain owner is already 
publishing SPF for mailing names (where it can be problematic) and 
already publishing anti-functional MX records, they should have no 
problem with publishing SPF records as well for their non-mailing names. 
There is a nuisance cost in doing this sort of thing for mid-sized 
domains which have long been able to function managing their DNS in ad 
hoc ways, but that condition is pretty much doomed anyway by IPv6 and 
DNSSEC. We all are destined for needing to think more and more carefully 
about DNS.

I should add that I don't think this issue is significant in the big 
picture of spam control and it certainly isn't news with respect to SPF. 
The days of hope for SPF as the FUSSP are long gone. Even with DMARC we 
will never have a world where SPF plays a primary role in sender 
authentication and repudiation. SPF has a few narrow applications where 
it is very useful, but anyone hoping for it to be the fix for all sender 
spoofing ought to have been disillusioned by real world experience some 
years ago.

---

From: Christian Grunfeld <[email-masked]>
Date: Fri, 7 Dec 2012 21:18:23 -0300

> have a login to edit, just drop me a note and I'll give you one.

John, I didn't find spamtraps or honeypots.....may be I didn't dig
enough the wiki

---

From: "John Levine" <[email-masked]>
Date: 8 Dec 2012 02:02:40 -0000

>John, I didn't find spamtraps or honeypots.....may be I didn't dig

That's because it's not done yet.  Where should I send your password
so you can add those sections?

---

From: "John Levine" <[email-masked]>
Date: 8 Dec 2012 02:05:57 -0000

>already publishing anti-functional MX records

I only publish a handful of anti-functional MX records, for a few
domains that get a buttload of spam due to having appeared in usenet
message IDs that were mis-scraped onto spam lists a long time ago.

I probably have two orders of magnitude more A records than MX records,
and I'm not inclined to add an MX . and SPF -all for all of them.

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 08 Dec 2012 19:43:03 +0100

> (Yes, I know about zone cuts.)

It is easy to do the opposite, though.  For example, having

  leila.iecc.com TXT "v=spf1 redirect=iecc.com"

  would enable receivers, at the cost of an extra lookup, to infer
that iecc.com is in the same administrative domain of that mailer, for
reputation or reporting purposes.

Aren't there generic (i.e. non spf-specific) DNS conventions for
publishing such relationships?

---

From: "Bill Cole" <[email-masked]>
Date: Sat, 08 Dec 2012 14:03:45 -0500

On 7 Dec 2012, at 16:32, SM wrote:

> delivery failures (assuming valid messages).

If that sentence is not mis-worded, it is a declaration of ignorance of 
important fundamental facts about how email works. RFC5321 is required 
reading for anyone who wants to be taken seriously in technical 
discussions of email.

> There are valid messages with an invalid Return-Path.

This could only be true when using a technically useless definition of 
"valid messages" which does not refer to any formal standard or even to 
informal best practice norms.

There are many strong arguments for mail systems to refrain from 
attempting to require strict compliance to many details of the SMTP and 
email format standards, but it is unhelpful to assume that historically 
or even currently widespread forbearance of operational enforcement of 
some facet of the standards grants any sort of validity to non-compliant 
messages.

> This may due to some assumptions about how email works.

Indeed.

---

From: "John Levine" <[email-masked]>
Date: 8 Dec 2012 20:36:54 -0000

>  leila.iecc.com TXT "v=spf1 redirect=iecc.com"

It's not easy if I have to add an extra TXT record for each of the
thousands of A records in my DNS zones.

>publishing such relationships?

There's CNAME, but that doesn't do what we want here.

---

From: Andrew Sullivan <[email-masked]>
Date: Sat, 8 Dec 2012 18:38:10 -0500

> reputation or reporting purposes.

I wrote a draft discussing some of these problems, and have talked
about it on the Applications Area list (and in one of the meetings).

> publishing such relationships?

No, but comments on
http://tools.ietf.org/html/draft-sullivan-domain-origin-assert-02
would be most welcome.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Paul Smith <[email-masked]>
Date: Sat, 08 Dec 2012 23:17:14 +0000

> astonishingly hard to implement.
Could we have a "direct parent" check rather than going up lots of levels?

Eg, if you are having to fall back to an A record, look at the direct 
parent for a TXT record with appropriate data in

So, if you are checking a message from [email-masked], you are 
doing your check back that bibble.twitter.com exists, and see that it 
only has an A record (if it has an MX record, that satisfies your 
checks). Now, you do a TXT lookup on 'twitter.com' and if the result has 
a record saying 'mx only' (or some such) you know that 
'bibble.twitter.com' isn't valid as an email domain.

This would be quicker than going up the tree, but would catch the most 
common case of having lots of hosts defined in a domain, any of which 
could be used for 'spoofing'. It would only be needed in the (hopefully, 
rare) case of having to fall back to the 'A' record.

Checking for the TXT record would be slower than not doing so, but would 
be a lot more efficient than attempting to connect to a non-existent 
mail server at the A record.
> that be insisted that it was already too late.
I wish it could be deprecated (but still required) now (ie, you MAY just 
have an A record to indicate where mail is to be delivered, but you 
SHOULD have an MX record instead). That way if we came across an A 
record only, we would have a stronger argument for persuading domain 
admins to add MX records. Currently some say 'well the standards say 
it's OK, so we're not changing anything'.

It could be worth logging how many times we have to fall back to an 
A/AAAA record, AND the delivery succeeds - I may have a go at that 
ourselves.

-

---

From: Andrew Sullivan <[email-masked]>
Date: Sat, 8 Dec 2012 19:35:33 -0500

> Could we have a "direct parent" check rather than going up lots of levels?

No.

> Eg, if you are having to fall back to an A record, look at the
> direct parent for a TXT record with appropriate data in

You have _no reason_ whatever to suppose that the parent is the
appropriate place to ask this.  Indeed, you don't even have a reason
to suppose that the parent isn't part of the same zone delegation
you're looking at (where's the SOA?  And why does the SOA tell you
anything anyway, since organizations can cheerfully introduce zone
cuts for administrative purposes that have nothing to do with
delegating outside the controlling organization).

> A/AAAA record, AND the delivery succeeds - I may have a go at that

That would indeed be useful empirical evidence.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: SM <[email-masked]>
Date: Sun, 09 Dec 2012 00:42:24 -0800

At 11:03 08-12-2012, Bill Cole wrote:
>If that sentence is not mis-worded, it is a declaration of ignorance 
>of important fundamental facts about how email works. RFC5321 is 
>required reading for anyone who wants

The sentence is likely incorrectly phrased.

>attempting to require strict compliance to many details of the SMTP 
>and email format standards, but it is unhelpful to assume that 
>historically or even currently widespread forbearance of operational 
>enforcement of some facet of the standards grants any sort of 
>validity to non-compliant messages.

Yes.

Regards,
-sm

---

From: Dave Crocker <[email-masked]>
Date: Sun, 09 Dec 2012 07:02:16 -0800

> 'bibble.twitter.com' than if it came from 'random.ru'

Strange thread.  Mostly seems to be based on using SPF in ways that it 
is not designed to be used.  It's always problematic to criticize a 
technology for failing to provide services that it is not intended to 
provide.

An essential point, above, is worrying about "semi-tech-savvy people".

First, SPF is not for end-users.  It is for receive-side operators and 
their filtering engines.

Second, embedded technologies, like SPF, cannot be mail fool-proof 
against poor use by operators who misunderstand one or another aspect of 
the technology.  Better software and better documentation are the best 
one can do for them.

This thread seems a bit like worrying about the inability of an 
automobile braking system's failure to prevent the driver from running a 
red light.

Absent a concise and compelling summary statement of the problem at 
issue here, I suggest terminating this fruitless thread.

d/

-- 
  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net

---

From: Christian Grunfeld <[email-masked]>
Date: Sun, 9 Dec 2012 12:43:10 -0300

2012/12/9 Dave Crocker <[email-masked]>:

> Strange thread.  Mostly seems to be based on using SPF in ways that it is
> not designed to be used.  It's always problematic to criticize a technology
> for failing to provide services that it is not intended to provide.
> First, SPF is not for end-users.  It is for receive-side operators and their
> filtering engines.

When operators believes that SPF and their filtering engines are well
configurated but a forged email pass anyway (by means of this thread),
it finally reaches a "semi-tech-savvy user" who is more trusting of
Twitter mail coming from 'bibble.twitter.com' than if it came from
'random.ru'. None of us are saying that simple users deals with SPF
directly !

> Second, embedded technologies, like SPF, cannot be mail fool-proof against
> poor use by operators who misunderstand one or another aspect of the
> technology.  Better software and better documentation are the best one can
> do for them.

yes, this is a misunderstanding and I believe this is a good place to
discuss, isn't it?....but I think is a misunderstanding of a huge part
of the operators..excluding you of course !

---

From: Dave Crocker <[email-masked]>
Date: Sun, 09 Dec 2012 08:09:11 -0800

> 'random.ru'. None of us are saying that simple users deals with SPF

Anyone who believes that SPF can 'guarantee' that no forged mail will 
get delivered is suffering a very basic failure in their understanding. 
  No changes to SPF can fix this.

The best line of action is to develop better statements about what SPF 
does and does not do, rather than discuss SPF 'failings'.  That is, try 
to help the understanding of operators.

d/

---

From: Martijn Grooten <[email-masked]>
Date: Sun, 9 Dec 2012 16:13:31 +0000

> a forged email pass anyway

A forged email using a subdomain for which no SPF record exists will pass the test checks against SPF fails, something which some spam-filters include, as it's a cheap way to get rid of a chunk of spam. Such a forged email will NOT "pass" SPF.

It is in theory possible that there is a incorrectly configured spam-filter out there that gives you an easy ride if you use an SPF-less subdomain in the SMTP envelope. I doubt this is the case, but if it does exist, please do tell its developers to fix their filter.

> I think is a misunderstanding of a huge part
> of the operators

Is it? Have you evidence, even if it is only anecdotal, that such a misunderstanding exists?

If so, as Dave said, we should consider writing better documentation.

Martijn.

---

From: "John Levine" <[email-masked]>
Date: 9 Dec 2012 19:16:14 -0000

>> a forged email pass anyway

No antiforgery scheme can defend against fakes that only sort of match
the forgery target.  We call this the paypai problem.  The only
approach I can see that has any hope of success is to figure out some
way to mark real mail from a category of targets (banks, say) in a way
that bad guys can't fake.  

But this is specific to each target group.  The fact that mail from me
doesn't have a seal saying that it's from a bank doesn't mean that
it's forged or otherwise bad.

>> I think is a misunderstanding of a huge part of the operators
>

I'm with Martijn.  Other than the test message you sent the other day,
I don't think I have ever seen a phish that used a subdomain of the
target.  Ever.

---

From: Seth <[email-masked]>
Date: Sun, 9 Dec 2012 16:33:07 -0500 (EST)

Martijn Grooten <[email-masked]> wrote:

> A forged email using a subdomain for which no SPF record exists will
> pass the test checks against SPF fails, something which some
> spam-filters include, as it's a cheap way to get rid of a chunk of
> spam. Such a forged email will NOT "pass" SPF.

There is no way for the owner of the overlying domain (who also owns
the subdomain) to force such email to FAIL.  There should be a way to
specify "all valid email from this domain and subdomains comes only
from this set of IPs and no others" and SPF fails to provide one.
That's a weakness in the structure of SPF which ought to be fixed.

Seth

---

From: Christian Grunfeld <[email-masked]>
Date: Sun, 9 Dec 2012 20:39:07 -0300

> That's a weakness in the structure of SPF which ought to be fixed.

there is ! you have to publish your sending IPs and also your not
sending IPs/hosts/subdomains.

> I don't think I have ever seen a phish that used a subdomain of the

...so the "vulneravility" exists ! may be spammers don't know it yet !
don't you believe that a phish with these characteristics could be
worse than other?

At my institution I have told my users to check mail headers whenever
possible. They are physicists ... the ones who invented the www time
ago, so they have some skill level to do it. I always told them: do
not believe in what you see in the From: also check the envelope and
any hop in between ! ....but what will I say them when they´ll see
mails "comming" from a subdomain of the real domain that the mail
claims to be from and no checks failed?
I personally think this kind of phishing worse than common ones!

For the ones that said we are treating SPF as the FUSSP or the best
solution...the answer is no ! I use SMTP level client checks, helo
checks, sender checks, recipient checks, rate limits, SPF, DKIM,
greylists, spamassassin, etc. SPF is only one of which we are
discussing now a misconception, misunderstanding or what you prefer.
We can also disscuss about any of the other methods.

---

From: "John Levine" <[email-masked]>
Date: 10 Dec 2012 00:11:57 -0000

>That's a weakness in the structure of SPF which ought to be fixed.

How do you propose to implement it, other than tree climbing and gross
hacks like the public suffix list?

---

From: "John Levine" <[email-masked]>
Date: 10 Dec 2012 00:16:52 -0000

>don't you believe that a phish with these characteristics could be

No, I don't see any reason to think that www.paypal.com is more likely
to fool users than paypaI.com.  Because of the way that Paypal uses
Akamai to distribute its load, adding a TXT record to www.paypal.com
would be rather difficult.

SPF has been around for about a decade.  If this hack were useful, I
think we can assume that spammers would use it.

R's,
John

---

From: Franck Martin <[email-masked]>
Date: Sun, 9 Dec 2012 22:34:35 -0600 (CST)

This issue is handled by DMARC, as it can work on all subdomains with only one TXT record.

If you try to do the same with SPF only, this may be complicated.

But then DMARC p=reject is not for all.

---

From: Martijn Grooten <[email-masked]>
Date: Mon, 10 Dec 2012 08:43:44 +0000

> but what will I say them when they´ll see

As others have pointed out, SPF is not for end-users. If you desperately want them to make sense of SPF checks, make sure you tell them that not failing SPF means exactly nothing; it definitely does not mean the emails passed SPF, or anything else that should give you extra reason to assume the domain is not forged.

If you believe they are really clever, you may consider telling them that in case of important emails, not passing SPF is a reason to be extra suspicious. But I personally wouldn't go there, even if the users had invented the Internet.

Martijn.

---

From: Paul Smith <[email-masked]>
Date: Mon, 10 Dec 2012 09:21:36 +0000

> If you believe they are really clever, you may consider telling them that in case of important emails, not passing SPF is a reason to be extra suspicious. But I personally wouldn't go there, even i…
Well, it generally doesn't actually matter who an email is from - if 
you're just going to read it. It only matters if you're going to click 
on links in it, or open attachments.

It doesn't really even matter who the email is from if you're going to 
click on links - all that matters is where those links go. If I get a 
forged email telling me to click on 'www.facebook.com', then what harm 
will it do - as long as the link REALLY is www.facebook.com.

So, what we really need (AFAICS) is not a way to detect forged messages, 
but a way to check that links in the message go to where the user would 
expect them to go.

So, two things:

1) I believe that any prime 'phishing target' (banks, Facebook, etc) 
should not put links in their emails - EVER. If you get an email from 
Facebook telling you you have new notifications, then the user should 
just go to Facebook independently. That way, you're pretty much 
protected from phishing emails. Unfortunately, this isn't going to 
happen, because users are generally too lazy, so want the emails they do 
get from Facebook etc to contain easy to click links/'buttons'.

2) A possible solution. Nothing amazingly clever, but AFAICS it would 
make it easier for users if it was implemented. I don't doubt it'll get 
shot down, but here goes...
(this isn't the "final" description of the idea, given that its likely 
to get shot down. it's just a rough outline)

  - The mail receiver (MUA or a spam checker) gets the message, and 
either looks at the return path domain, or some other 'marker' in the 
email to find the 'canonical domain' (doesn't have to be encrypted or 
secret)

  - The mail received software checks a DNS record at that domain to see 
if they support this facility. (as a method of protecting against a DDOS 
against random companies)

  - The mail receiver software checks a certificate at the domain it 
found in the step above (eg by going to https://www.<the domain>, or 
some other). It checks that the certificate is a valid, CA signed 
certificate, and can thus, reasonably confidently, give an organisation 
name where this message purports to be from. It can display this to the 
user.

  - The mail receiver software then checks the links in the message 
against something at that domain - eg download a list of valid link 
hostnames from the domain (using https), or use a surbl-like check using 
that domain as the database, or whatever

  - The mail receiver software now has a bit more confidence that the 
email only contains links which the purported originator would approve of.

Possible flaws I can see
- the DNS/website of the phish target could be hacked. If that is done, 
then there's a bigger problem...

- the certificate may show that the organisation is www.paypaI.com, 
because the phisher has got a certificate which only verifies the domain 
name, not the organisation. This may fool some users. Could we mandate 
certain types of certificate for this facility?

- extra work by the receiving software. But, given that things like 
Thunderbird already look at email links and suggest that they may be 
dodgy, the extra load may be considered acceptable

- possible DDOS vector against someone's DNS/web servers

- if the phish target uses unprotected redirection URLs, those could be 
used to get around it

- if the phish target doesn't know what domains they use, that could 
cause false positives

-

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 10 Dec 2012 12:13:40 +0100

> http://tools.ietf.org/html/draft-sullivan-domain-origin-assert-02

Those SOPA records, at least in the "names-only" strategy, would be an
easy way to lightheartedly use wildcard records so as to cover all the
subdomains.  I'm surprised there are no established hacks, such as
capturing part of the SOA, as it might be part of a negative reply to
a TXT query, e.g. RNAME =~ /hostmaster\.([-.a-z0-9]{2,63})/i.

While RNAMEs may get filled in unwittingly of such use, SOPA is sound
because of the clearness of its semantics.  However, it is also true
that its precise meaning depends on the kind of policy.  One then has
to turn to "port-and-scheme" strategy, and complicate the subject
beyond ease of reach.  In comparison, the overloaded-redirect hack has
some nice properties:

*semantics*:  the same mail abuse team looks after both domains.

*hierarchy*:  a loop entails the setup is broken.

*conciseness*:  the vertex must be reachable withing 10 queries.

*awareness*:  SPF never suggested to use redirect for helo records, so
anyone using it must be aware of it.  For a more stringent check, it
could be conceived as, say, redirect=_spf_policy_hack.master.tld.

That doesn't address the OP concern about non-mailing subdomains,
albeit in some cases it can.  It just reduces the number of entries in
a receiver's senders database.  Since each of those entries is likely
to cost some fraction of human operator's time, reducing them seems to
be worth an extra lookup on each message.

---

From: Martijn Grooten <[email-masked]>
Date: Mon, 10 Dec 2012 13:47:18 +0000

Paul Smith wrote:
> click on links - all that matters is where those links go.

While not all bad things that can happen as a consequence of taking a fake email to be real involve clicking links, I agree it would be very helpful if we could somehow determine the legitimacy of links at the MTA/MUA level.

However, we can't. There are simply too many examples of bad links that match the sender's address (paypai being just one of them) and even more examples of 'good' links that don't match, that in general you can't tell from just analysing the URL.

Martijn.

---

From: Dotzero <[email-masked]>
Date: Mon, 10 Dec 2012 09:22:22 -0500

On Sun, Dec 9, 2012 at 6:39 PM, Christian Grunfeld
<[email-masked]> wrote:
> possible.

Not a very scalable solution for the community at large even if it
were a good use of their time.

>They are physicists ... the ones who invented the www time

Is manually checking email headers a very good use of their time? How
much does a physicist at CERN get paid? Are they trained at
identifying forged headers?

Mike

---

From: Rich Kulawiec <[email-masked]>
Date: Mon, 10 Dec 2012 09:56:27 -0500

On Mon, Dec 10, 2012 at 01:47:18PM +0000, Martijn Grooten wrote:
> While not all bad things that can happen as a consequence of taking a
> fake email to be real involve clicking links, I agree it would be very
> helpful if we could somehow determine the legitimacy of links at the
> MTA/MUA level.
> 
> However, we can't. [...]

Precisely.  Moreover, there is no way to know that a link L, which is
good/legitimate/wholesome/nutritious at the moment at which some piece
of software examines it, will still be in that state at some later moment
when a user accesses it.  We see examples all day, every day, of sites
that have been hijacked by attackers and now host malicious content where
formerly there was something innocuous.

I think this best approach to solving this problem is not to solve this
problem.  Hmmm...perhaps "best" is a poor choice of words, as what I'm
about to suggest is, in practice, mediocre; but I'll stick with "best"
because I think it's the best available.

To wit: users should never follow "important" links in email.  They
should (for example) bookmark their bank's web site, and *always*
use the bookmark. [1]  Senders should never send "important" links
in email, e.g., banks should not include URLs in their messages.

I'll pause now while you all enjoy a hearty laugh at the prospect
of both of these things happening.

But the serious side is that the problem, as Martijn observes,
is unsolvable with software, so if we want to truly deal with it,
then (shudder) we need to deal with human behavior.

And I really do know how hard that is: I often quote Marcus Ranum's
Six Dumbest Ideas in Computer Security, and "Educating Users" is #5.
But I don't think we have any better alternatives.  Do we?

---rsk

[1] Of course if the bookmark is altered by an attacker, they're
already 0wned, so this is no worse.

---

From: Christian Grunfeld <[email-masked]>
Date: Mon, 10 Dec 2012 12:59:50 -0300

I'm really surprised to see how discussions diverges on this list !

The problem statement is very simple:

1) How many people on this list saying that SPF is bad have TXT
records published on their domains ? Why do you use it in real life
and deplore it here ?

2) How many people on this list that use SPF have only one TXT record
at the top domain and remain relaxed having [?~-]all, ignoring this
thread ?

I accept that some of you don't mind that a third party can use your
subdomains to send forged emails claiming that are from you. I
personally don't like that and applied the fix !

I repeat, this thread started discussing an SPF problem and then
derived in a lot of other problems. We can discuss any, but please
don't blame it on SPF of all of them !

over and out!

---

From: Dave Crocker <[email-masked]>
Date: Mon, 10 Dec 2012 08:35:31 -0800

> were a good use of their time.

It is much worse than merely being not scalable.  It doesn't work at 
all, for anyone.  Absent diligent and continuing training, human 
attention-related behavior decays rapidly, to a steady state that is, at 
best, highly erratic.  Expecting /anyone/ to be attentive in the way 
described above is not realistic.  Ever.  For anyone.  No matter their 
skill level.

> Is manually checking email headers a very good use of their time?

Right.  This highlights that even if people could be expected to perform 
the task properly and always, it would be more expensive than it is worth.

d/

---

From: Chris Lewis <[email-masked]>
Date: Mon, 10 Dec 2012 11:36:46 -0500

> records published on their domains ? Why do you use it in real life

Maybe they're experimenting with DMARC.  Maybe they're listening to PHBs
who insist on it.  Maybe they wanted to get email into hotmail.

Or simply that that it's useless, and it don't matter whether they have
one or not.

> at the top domain and remain relaxed having [?~-]all, ignoring this

See below

> personally don't like that and applied the fix !

You mean like this?

_spf.google.com.	300	IN	TXT	"v=spf1 include:_netblocks.google.com ?all"

---

From: Dave Crocker <[email-masked]>
Date: Mon, 10 Dec 2012 08:47:25 -0800

On 12/10/2012 6:56 AM, Rich Kulawiec wrote:
>  We see examples all day, every day, of sites
...
> use the bookmark.

There is the kernel of an implementable idea here:

    1.  Create a whitelist of links the user employes regularly through 
its browser.  For an extra measure of safety, query the user about how 
much they 'trust' the site associated with each link.  (The question 
needs to be put to them with better language than asking about trust.)

    2.  Have the email client distinguish between links that are 
whitelisted and those that aren't.

I don't have any idea how much incremental safety this actually would 
provide, but I think it's worthy of testing.

d/

---

From: Dave Crocker <[email-masked]>
Date: Mon, 10 Dec 2012 08:48:08 -0800

This posting demonstrates why the main line of this thread is useless.

d/

---

From: Christian Grunfeld <[email-masked]>
Date: Mon, 10 Dec 2012 14:00:50 -0300

> _spf.google.com.        300     IN      TXT     "v=spf1 include:_netblocks.google.com ?all"

google´s domain for sending mail is gmail.com.
gmail.com is a virtual domain with no subdomains (afaik) and is mapped
onto google, so if you check the senders no [email-masked] could pass
even using SPF or not.

as you said, if you dig:

dig -t txt gmail.com
;; ANSWER SECTION:
gmail.com.              244     IN      TXT     "v=spf1
redirect=_spf.google.com"

dig -t txt _spf.google.com
;; ANSWER SECTION:
_spf.google.com.        300     IN      TXT     "v=spf1
include:_netblocks.google.com ?all"

if you dig a bit more:

dig -t txt www.google.com

you get no answer because there is not SPF record for www.google.com,
so you can forge emails as if they come from www.google.com even if
there exists an SPF record for google.com !

---

From: Paul Smith <[email-masked]>
Date: Mon, 10 Dec 2012 17:10:58 +0000

> provide, but I think it's worthy of testing.
Surely this would be a browser feature (or 'Internet Security Software' 
feature) rather than an email client feature.

The email client will not necessarily have any access to web browser 
history.

The web browser should know that being called from an email client is 
'different' from the user clicking on a bookmark or typing in a URL in 
the browser. Then, the browser could say to the user 'You've never 
accessed this site before, are you sure you want to do it?', or whatever

The problem is that to have any idea of reputation you'd have to go on 
the hostname, not the full URL, as many email URLs will be 'unique' to 
have some tracking information in them (yes, I know it's bad, but you 
won't get banks to get rid of that, unfortunately), so each email will 
have different URLs in, even if the final destination is the same.

So, the question is, is having a hostname reputation for the user better 
than having no reputation, or not? I'd say yes because it would probably 
catch 99% of the bad links that I see in phishing/spam, others would say 
no because it won't catch 100%.

-

---

From: Martijn Grooten <[email-masked]>
Date: Mon, 10 Dec 2012 17:32:48 +0000

> there exists an SPF record for google.com !

You can also put go0gle[dot]com or googgle[dot]com in the envelope as neither have an SPF record.

Heck, you can even use google.com, the SPF record of which only tells the receiving MTA that it probably wasn't Google who sent the email, for it uses ~all rather than -all. And even if it did use -all, it doesn't necessarily mean that the email wouldn't get delivered as not all spam filters/MTAs block SPF fails.

So the 'problem' you've stated is ONE OF MANY WAYS in which a forged email can avoid being blocked by ONE OF MANY TESTS that is applied by SOME MTAs. Incidentally, it is ONE OF MANY WAYS that could confuse an end user who is looking at something most of us agree they shouldn't be looking in the first place and that BARELY ANY END USER looks at.

Unless you can finally come up with actual evidence that such emails are a) more likely to be delivered than other emails claiming to come from the target domain AND b) are more likely to be taken for real by the average end user, I think we're wasting our time here.

Martijn.

---

From: Martijn Grooten <[email-masked]>
Date: Mon, 10 Dec 2012 17:49:01 +0000

> whitelisted and those that aren't.

I'm with Paul in that I think the web browser and/or a web proxy is a more appropriate place to do that.

Other than the fact that they are simply better at anything web-related (they can, for instance, do some heuristics on the actual content that's being delivered by the web server), I'm worried that a "this link is fine" message displayed by a MUA is easily confused with "this email is fine", which I don't think is desirable.

Also, the overwhelming majority of websites are completely fine and harmless, yet most are probably too obscure to end up on some kind of whitelist, let alone a personalised one. So I think you're going to have a lot of collatoral damage.

Having said all this, maliciious links in email are a serious problem today. Whenever I need a link to the Blackhole exploit kit for research purposes, I simply dig into my spam folder and look for a fake Facebook/LinkedIn/Twitter/YouTube/UPS email. More often than not it contains such a link.

Martijn.

---

From: [email-masked]
Date: Mon, 10 Dec 2012 12:51:52 -0500

On 12/10, Dave Crocker wrote:
>    1.  Create a whitelist of links the user employes regularly
> through its browser.  For an extra measure of safety, query the user
> about how much they 'trust' the site associated with each link.

If we could get that kind of information from end users, reliably, I don't
think we'd have a spam problem.

-- 
'1960: "I have a great idea! lets have every person in the country carry
a radio tracking beacon!" "That'll never fly!"  2012: "I can has TWO
iphones??"' - http://bash.org/?952948
http://www.ChaosReigns.com

---

From: "Eggert, Lars" <[email-masked]>
Date: Mon, 10 Dec 2012 18:02:49 +0000

> at the top domain and remain relaxed having [?~-]all, ignoring this

Tangential point, but maybe it's useful. All the raw data behind http://eggert.org/meter/spf is available. For example, the latest snapshot for SPF use by the IETF author community domain set is http://eggert.org/meter/data/2012-01-01@21:00-ietf-spf.txt

Lars

---

From: Dave Crocker <[email-masked]>
Date: Mon, 10 Dec 2012 10:05:05 -0800

> The email client will not necessarily have any access to web browser

Sorry. I was too cryptic.  My suggestion was a whitelist that is shared 
with the browser and the MUA, vetted by the user.  It's not about one 
agent calling the other but of a shared whitelist.

(Bitdefender seems to have a feature that is related, which provides 
very distinctive controls over sites that are used for payment, like 
banks, based on a special list of such sites.)

> accessed this site before, are you sure you want to do it?', or whatever

Development of the list could include various kinds of user 
consultation, yes.

> the hostname, not the full URL,

right.

+1

d/

---

From: Martijn Grooten <[email-masked]>
Date: Mon, 10 Dec 2012 18:23:31 +0000

> I'd say yes because it would probably
> catch 99% of the bad links that I see in phishing/spam

No it would not, not even close to that. A lot of spam links to legitimate but compromised domains.

Note that just about any security vendors offers a web security product, which runs either inside the browser or at the corporate gateway. Most use a combination of blacklists, whitelists and content-inspection.

Martijn.

---

From: Michael Thomas <[email-masked]>
Date: Mon, 10 Dec 2012 10:50:16 -0800

> with the browser and the MUA, vetted by the user.  It's not about one 

We thought of this 5-6 years ago and even filed ipr on it as I recall. But
that was in a time when people mostly had one desktop/laptop. I would think
that the advent of phones, tablets, etc would significantly increase the complexity
and most likely false-positive rate.

Mike

---

From: "John Levine" <[email-masked]>
Date: 10 Dec 2012 19:24:04 -0000

>gmail.com is a virtual domain with no subdomains (afaik)

www.gmail.com at least, probably others.

Really, you know, there's a lot of expertise on this list, and if we
unanimously say that something is a bad idea, you might want to at
least momentarily contemplate the possibility that it really is a bad
idea.

---

From: Chris Lewis <[email-masked]>
Date: Mon, 10 Dec 2012 15:09:00 -0500

> there exists an SPF record for google.com !

The point was about your #2.  Those using ?all.  Eg: you.

People don't use !all, for the same reason gmail (you) uses ?all.

With that huge problem in the way, trying to solve <hostname>.<spf'd
name> doesn't seem very useful.

The "solution" of publishing SPF records for all A'd hosts works
sometimes, but sometimes it just doesn't scale.

Just imagine how many hostnames that Google and Gmail have between them.

---

From: Paul Smith <[email-masked]>
Date: Mon, 10 Dec 2012 20:24:22 +0000

> No it would not, not even close to that. A lot of spam links to legitimate but compromised domains.
Maybe - but most of those compromised domains are not domains which the 
user would normally go to.

So, even if the link is to a 'legitimate' domain, the vast majority 
would be to domains which the user does not recognise. So, it would 
catch those.

Remember, the idea wasn't to have a 'global' list of 'good domains', but 
ones which the *user* has whitelisted, so the user recognises them.

-

---

From: Steve Atkins <[email-masked]>
Date: Mon, 10 Dec 2012 12:41:44 -0800

On Dec 10, 2012, at 12:24 PM, Paul Smith <[email-masked]> wrote:

> Maybe - but most of those compromised domains are not domains which the user would normally go to.
> 
> So, even if the link is to a 'legitimate' domain, the vast majority would be to domains which the user does not recognise. So, it would catch those.
> 
> Remember, the idea wasn't to have a 'global' list of 'good domains', but ones which the *user* has whitelisted, so the user recognises them.

If it's user managed - rather than managed by a third party that actually keeps track of who is a bank and who isn't - I'd guess it'd just lead to meta-phishing, where the goal is to get the user to add a link to their whitelist, rather than having them click on a link.

Cheers,
  Steve

---

From: Martijn Grooten <[email-masked]>
Date: Mon, 10 Dec 2012 20:48:11 +0000

> ones which the *user* has whitelisted, so the user recognises them.

OK - I see what is meant now.

Still, how often does the average user visit a domain they've not visited before? For if they constantly have to approve 'new' websites, they're either going to turn the warnings off or they're going to ignore them, which defeats the point.

Note that it is not uncommon for example.com to use iframes that load stuff from the apparently unrelated example.org. This is also commonly used by malicious sites. If a user visits a 'new' site using iframes, should such a system warn against all the domains used by the iframes?

Again, there are a lot of web filters already out there. Can someone explain either why the problem we're trying to solve here is different than the one they attempt to solve or, if it isn't, why our solution will be better?

Martijn.

---

From: Dave Crocker <[email-masked]>
Date: Mon, 10 Dec 2012 13:36:07 -0800

On 12/10/2012 12:41 PM, Steve Atkins wrote:
>> Remember, the idea wasn't to have a 'global' list of 'good
>> domains', but ones which the *user* has whitelisted, so the user
>> recognises them.
>
> If it's user managed - rather than managed by a third party that
> actually keeps track of who is a bank and who isn't - I'd guess it'd
>  just lead to meta-phishing, where the goal is to get the user to add
>  a link to their whitelist, rather than having them click on a link.

Fair point.  that's probably why an independent list, such as
Bitdefender seems to use, is probably the right approach.  I could
imagine, however, some possible benefit in keeping track of which
entries in the list a particular user tends to go to.  Such as one or a
few banks versus others.

On 12/10/2012 12:48 PM, Martijn Grooten wrote:
> Again, there are a lot of web filters already out there. Can someone
> explain either why the problem we're trying to solve here is
> different than the one they attempt to solve or, if it isn't, why our
> solution will be better?

I'm not sure it is 'different' but possibly 'in addition'.  I'm 
suggesting integration with the MUA.  And to be clear, I'm not clear 
about the details.  I'm merely offering a very basic idea.  The precise 
details, nevermind determining actual efficacy, would be TBD.

d/

---

From: Martijn Grooten <[email-masked]>
Date: Mon, 10 Dec 2012 22:50:08 +0000

> I could

Ah yes, I can see that being useful in a scoring filter. I'm not sure many web-filters work that way; I think most are pretty black and white (or, rather, black and non-black).

> details, nevermind determining actual efficacy, would be TBD.

Thanks for the clarification.

I'm still sceptical about integration of MUA and browser/web-filter being of actual use, but I'll leave it for others to pursue and to prove me wrong.

Martijn.

---

From: Christian Grunfeld <[email-masked]>
Date: Tue, 11 Dec 2012 00:36:11 -0300

a few hours ago a dozen people kill me on the list because I told my
users to check hostnames on email headers. You said that end users
don't have the skill level or the time to do it.

Now, a "great idea" of making a whitelist of what users click on their
email and also asking them why they click what they click comes!

How do you treat shortened/obfuscated/tracking urls? You are
underestimating phishers as if they write links like
http://www.google.com. A link to the same location can be
http://bit.ly/someobfuscatedstring. And you are also overestimating
the end users as if they were "matrix operators". The user can simply
answer you: "I trust it because the link takes me to google"

What domain/hostname do you use in reputation? bit.ly ?

This "whitelist" can be poisoned by sending a first email pointing to
the good location. Then you add bit.ly to your list ....then I send a
second email pointing to an evil location !

C.

P.S for those with linear reasoning, stop before writing "people
should use www.google.com when they want to go to www.google.com". It
was an example !

2012/12/10 Paul Smith <[email-masked]>:
>>    1.  Create a whitelist of links the user employes regularly through its
>> browser.  For an extra measure of safety, query the user about how much they
>> 'trust' the site associated with each link.  (The question needs to be put
>> to them with better language than asking about trust.)
>>
> 'different' from the user clicking on a bookmark or typing in a URL in the
> browser. Then, the browser could say to the user 'You've never accessed this
> site before, are you sure you want to do it?', or whatever
>
> The problem is that to have any idea of reputation you'd have to go on the
> hostname, not the full URL, as many email URLs will be 'unique' to have some
> tracking information in them (yes, I know it's bad, but you won't get banks
> to get rid of that, unfortunately), so each email will have different URLs
> in, even if the final destination is the same.
>
> catch 99% of the bad links that I see in phishing/spam, others would say no
> because it won't catch 100%.
>
>
>
> -
>

---

From: Dave Crocker <[email-masked]>
Date: Mon, 10 Dec 2012 20:33:48 -0800

> email and also asking them why they click what they click comes!

Please re-read the thread more carefully.  You apparently misunderstood 
both lines of discussion.

d/

---

From: Chris Lewis <[email-masked]>
Date: Mon, 10 Dec 2012 23:59:30 -0500

> Note that it is not uncommon for example.com to use iframes that load stuff from the apparently unrelated example.org. This is also commonly used by malicious sites. If a user visits a 'new' site u…

Sounds somewhat like the "noscript" firefox plugin.  I think it works in
more than just firefox.

It won't execute javascript or iframes _unless_ you whitelist the site
they come from, either temporarily (for the current session) or
permanently.  There's a fairly well-developed mechanism for selecting
which sites you wish to whitelist.

It works pretty well.  It has the added advantage that you can, say,
visit websites and "choose" not to participate in things like
doubleclick tracking or advertisements from other sites by not approving
them.

It's truly remarkable how one URL can trigger javascript bits from
dozens of different places.

It does mean that some first-time visits to a web site can sometimes be
very noisy.   But it's less awkward than it sounds.

And you may get yourself somewhat habituated into recognizing "hey, it
threw alerts on a site that didn't before.  Oh gee, different spelling".

I use it.

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 11 Dec 2012 08:52:50 +0100

On Mon 10/Dec/2012 18:32:48 +0100 Martijn Grooten wrote:
>
> Heck, you can even use google.com, the SPF record of which only
> tells the receiving MTA that it probably wasn't Google who sent the
> email, for it uses ~all rather than -all. And even if it did use
> -all, it doesn't necessarily mean that the email wouldn't get
> delivered as not all spam filters/MTAs block SPF fails.
               ^^^^^^^
Does that imply that most, or even a relevant percentage of them, do?

---

From: Martijn Grooten <[email-masked]>
Date: Tue, 11 Dec 2012 08:36:10 +0000

> Sounds somewhat like the "noscript" firefox plugin.  I think it works in
[...]
> I use it.

I stopped using it after I found myself blindly allowing any JavaScript any website was running.

But still, it's a good point and I can see why it would work for some, like a whitelisting approach would work for some (possibly the same) users.

I think neither are suitable for use by the majority of web users though.

Martijn.

---

From: Martijn Grooten <[email-masked]>
Date: Tue, 11 Dec 2012 08:45:37 +0000

> Does that imply that most, or even a relevant percentage of them [block SPF fails]?

No. I know some do, but I think it's a minority. (Among commercial spam-filters used mainly by corporations, that is. It could be that some of the larger ISPs block SPF fails outright, which would skew the picture.)

What I know for sure is that failing SPF doesn't mean an email is blocked by all spam-filters, because I found several counter-examples.

Inspired by this thread, I'm going to do a little more research into this and will try to see if there is any correlation between the SPF status of a spam message and the likelihood of it being blocked. I'll share the results on this list.

Martijn.

---

From: Rich Kulawiec <[email-masked]>
Date: Tue, 11 Dec 2012 08:37:27 -0500

I think we're getting into a number of overlapping problems here, most
of which are caused by the use of worst practices (e.g., HTML in email,
outsourced email, use of URL shorteners, web sites that load Javascript
from a dozen different sources[1], financial institutions that are
training their customers to be phish victims, etc.)

I don't think we can solve or even capably attack any of these issues
at the MTA or MUA, so I'll suggest they may be out-of-scope here.

My suggestion that users bookmark their most-used/most-critical web
sites (in their web browser) is actually unrelated to email: that is,
I don't think anyone should ever read their email with a web browser or
with a mail client that has web browser features.  I regard doing so as
a catastrophic security failure, one that can't be mitigated no matter
how many layers of code are placed around or on top of it.

So to clarify: this is an entirely manual process, but since it need
only be done once per site and since it need only be done for "important"
sites (in the view of the user) I regard the effort as minimal.
Unfortunately, few users have the self-discipline required to always
use those bookmarks, doubly so given that they're going to receive
email containing links from the very institutions that they would be
most likely to bookmark.  (See "training customers to be phish victims".)

---rsk

[1] Using NoScript makes this problem highly visible.

---

From: Michael Thomas <[email-masked]>
Date: Tue, 11 Dec 2012 06:21:00 -0800

> training their customers to be phish victims, etc.)
Anybody who thinks that using HTML or outsourcers are "worst
practices" is part of the problem, not part of the solution.

Mike

---

From: "John Levine" <[email-masked]>
Date: 11 Dec 2012 14:46:55 -0000

>No. I know some do, but I think it's a minority. (Among commercial spam-filters used mainly
>by corporations, that is. It could be that some of the larger ISPs block SPF fails outright,
>which would skew the picture.)

No ISP I know does that.  The false positive rate is much too high,
since publication of a -all record is highly correlated with
assertions that all forwarding is evil, which is of course just wrong.

---

From: John Johnson <[email-masked]>
Date: Tue, 11 Dec 2012 08:52:55 -0600

> practices" is part of the problem, not part of the solution.

  I highly disagree.  A local bank just hired an outside firm to
  spam a "newsletter" to their customers in my area. It was quite
  difficult to tell if it was legitimate, as the bank had published
  SPF records, yet failed to provide the ip's of the outsourcers
  servers. And then used the banks domain name as the source.

  This should not be acceptable behavior, especially for a financial
  institution. It trains their customers to just accept anything
  and everything, they should be setting the bar, not lowering it.

-john

---

From: John Johnson <[email-masked]>
Date: Tue, 11 Dec 2012 08:57:05 -0600

> But then DMARC p=reject is not for all.

  +1, maybe +2

  I now get reports of hosts attempting to send using a subdomain that isn't in use.
  Franck, thanks for your article on DMARC, it truly is a good tool.

-john

---

From: Michael Thomas <[email-masked]>
Date: Tue, 11 Dec 2012 07:07:58 -0800

>    servers. And then used the banks domain name as the source.

Simply being an outsourcer doesn't preclude having first party
dkim signatures.

>    and everything, they should be setting the bar, not lowering it.

Customers aren't trained in anything to do with email headers of
any kind. What should and should not be acceptable behavior is
strictly between the sender's practices and the receiver's filtering.
If you want to complain, complain to the receiving MTA's. Legitimate
senders will adapt.

Mike

---

From: "John Levine" <[email-masked]>
Date: 11 Dec 2012 15:28:53 -0000

>  servers. And then used the banks domain name as the source.

I think we can all agree that being incompetent is a worst practice.

There are plenty of companies that oursource their advertising and
even a fair number who outsource transactional mail, but get the
domains and SPF and DKIM correct.

Running mail properly is hard, hiring people who know what they're
doing rather than trying to fake it yourself is often a good idea.

---

From: Dotzero <[email-masked]>
Date: Tue, 11 Dec 2012 10:29:46 -0500

> assertions that all forwarding is evil, which is of course just wrong.

I concur with John - The false positive rates, misconfigurations in
records, etc. are one of the reasons for the development of DMARC and
the buy-in by major mailbox providers (See announcement by MS
yesterday for outlook.com implementing DMARC and switching from SID to
SPF - http://blogs.office.com/b/microsoft-outlook/archive/2012/12/10/outlook-com-increases-security-with-support-for-dmarc-and-ev-certificates.aspx)

I'm not sure I agree with John that a -all record is somehow an
assertion that all forwarding is evil. I'll save that discussion for
drinks the next time I see him.

Mike

---

From: Martijn Grooten <[email-masked]>
Date: Tue, 11 Dec 2012 15:46:55 +0000

>  No ISP I know does that.  The false positive rate is much too high,

Thanks. I am aware of the latter, but didn't want to conclude the former based on that.

Martijn.

---

From: Paul Smith <[email-masked]>
Date: Tue, 11 Dec 2012 15:53:53 +0000

+1

Banks can use outsourcing for their mail, that's fine, but they should 
seriously consider the implications. It's fairly obvious that most banks 
don't.

There are things that an email outsource & bank can do together to keep 
things secure - and OBVIOUSLY secure, but it's clear that neither most 
banks nor email newsletter companies actually understand email enough to 
be using it in this way.

-

---

From: Christian Grunfeld <[email-masked]>
Date: Tue, 11 Dec 2012 13:07:04 -0300

yes, nice article !

If, for example, gmail/google is supporting DMARC why do they allow
multi-identities in From: header? this can really break DMARC tests.

Another point about SPF and forwardings... do ones allowing forwarding
and publishing -all must implement SRS to tackle the problem?

---

From: "John Levine" <[email-masked]>
Date: 11 Dec 2012 17:38:21 -0000

>and publishing -all must implement SRS to tackle the problem?

Gee, it seems to be dead horse time.

Forwarding isn't a problem, even though some people who think that
SPF is the FUSSP incorrectly imagine that it is.

SRS is basically saying that I have to change the way my mail system
works in order to work around the bugs in your anti-spam magic bullet.
Sorry, that's not how it works.

SPF certainly has its uses, but after a decade, I hope we all understand
that the world's mail systems are not going to rewrite themselves to
deal with SPF's well known limitations.  Ditto DMARC.

---

From: Laura Atkins <[email-masked]>
Date: Tue, 11 Dec 2012 09:51:01 -0800

> No. I know some do, but I think it's a minority. (Among commercial spam-filters used mainly by corporations, that is. It could be that some of the larger ISPs block SPF fails outright, which would…

I know no major ISP that reject a mail for a SPF fail. 

In fact, I'm getting multiple reports from clients and random senders this week that Hotmail is now sending out SPF failure emails to domains publishing DMARC records.

laura 

-- 
Laura Atkins   [email-masked]

   "If you do not choose to lead, you will forever be led by others.
   Find what  scares you and do it. And you *can* make a difference,
   if you choose to do so." JMS

---

From: "John Levine" <[email-masked]>
Date: 11 Dec 2012 19:35:07 -0000

>In fact, I'm getting multiple reports from clients and random senders this week that Hotmail
>is now sending out SPF failure emails to domains publishing DMARC records.

Yes, I'm getting them, too.

Unlike a lot of other DMARC reports, these all seem to include the
full unredacted message.  So far, all the ones they've sent have been
unsurprising, e.g., list mail, but I can't wait for some typo stuff to
arrive.