Subject: [Shutup] Levels of proposals

From: Brandon Long <[email-masked]>
Date: Thu, 3 Dec 2015 16:36:16 -0800

The WG proposal seems to imply taking all IPs out.  The discussion has
mostly been about submission.

It seems to me that there are at least three different IPs used, and some
of these are going to be visible regardless of intent.

Ie, there is the submission IPs, there are "internal" IPs, and external
ones.

Submission IPs seem like the largest level of risk, and from my gross
understanding of anti-spam, pretty minor.  I'm not sure what the current
source levels are, but submission IPs would be most useful in the case of
hijacked account spam or abusive account spam.  Presumably, if spam reports
about such are forwarded to the MSP, then the MSP can easily store the
information somewhere other than the easily forged headers and take the
appropriate action.  Only if the answer is "they don't take action" would
you need more.

Also, if the previous thread's list of large MSPs inclusion of submission
IPs is correct, then >2 out of the top 3 have already removed them (ie,
only a fraction of Gmail mail has them at this point).

Internal IPs, this hardly seems controversial.  If any mail system did
that, not sure if anyone would bat an eye.

External IPs, ie server to server... I guess one may learn "something" if
you can tell which submission server they talked to, we certainly have
servers across the world... but even with 20 odd locations, I doubt that
would be that specific.

So, I would recommend concentrating on submission IPs.  I might also
include a recommendation for submission servers to store the IPs for some
length of time to allow for abuse handling, or even to include an encrypted
version in the message.

Brandon

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 4 Dec 2015 01:19:50 +0000

This seems like another sensible opinion that doesn't make
overbroad claims about the goodness/badness of what's proposed
to be examined. More of us limiting our posting like this
would be good IMO.

S.

On 04/12/15 00:36, Brandon Long wrote:
> _______________________________________________
> Shutup mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/shutup
>

---

From: Chris Lewis <[email-masked]>
Date: Thu, 3 Dec 2015 21:00:01 -0500

> mostly been about submission.

The draft has been mostly about submission, but there have been some 
conversations about internals.  I've seen auditors bring up points about 
obscuring internals (corporate context, it's one of their checkoff 10 
commandments), but, we managed to convince them that ease of diagnosing 
problems outweighed the downsides.

Before getting too far into concrete proposals, we still need to decide 
whether it should be done, and what (if anything) should be done in 
mitigation.

That said:

> understanding of anti-spam, pretty minor.

Yes, and no...  It depends on how you use/acquire them, this is a fairly 
complex/esoteric end of spam filtering, and just doing it justice would 
take up pages of text.  I'll try to short cut - purely from a filtering 
but not-ML perspective:

Using them:

If you have a list of IPs known to be infected with AUTH-cracking 
spambots, it's of immediate/valuable use to both the MSAs themselves in 
detecting malicious injects, as well as the recipient's filtering, and 
header forgery is not an issue (certainly not to MSAs, and headers that 
forge collisions with the list you don't want anyway).

The potential yields are huge - ISPs (using a list we create of 2.2M IPs 
demonstrably capable of AUTH-cracking) can see botnet suppression rates 
at the MSA of 80% (of all submissions) or even higher, depending on the 
spam/botnet de-jour.  In just one case it's in the 1000+/second range. 
Some of these botnets are very very prolific per IP.

Which boils down to the MSAs being able to stop it at source, or the 
receivers being able to stop it at destination.  If a significant 
fraction of MSAs stopped it at source, then, maybe, the MSA Received 
line isn't important.  But where they don't (the majority), losing the 
MSA Received line is potentially a big hit.

Obtaining them:

Obviously, someone trying to authenticate to you and is demonstrably 
infected isn't spoofable.  That yields the vast majority of our list.

On a case-by-case basis with specialized knowledge it is possible to 
derive trustworthy infected submission addresses from received lines. 
Yes, actually, it really is.  Useful, but not terribly significant (or 
at least how far we've followed that rathole).

By eliminating MSA received clauses, it means that only the MSA can do 
blocking based on this technique, the receivers can't.

On one server, in a 40 minute interval, 774123 out of 1126763 
connections were AUTH spoofing from just one particular botnet.  That's 
775,000 spams that ordinary receivers couldn't filter via IP if the from 
clauses disappeared.

> Also, if the previous thread's list of large MSPs inclusion of
> submission IPs is correct, then >2 out of the top 3 have already removed
> them (ie, only a fraction of Gmail mail has them at this point).

The top 3 aren't generally large scale botnet spam sources, and botnets 
generally don't use real mail servers (except via AUTH crackers), and 
the top 3 are amply well instrumented with rate limiters etc.  So they 
really aren't relevant to this.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 04 Dec 2015 02:11:47 +0000

Thursday, Dec 3, 2015 9:00 PM Chris Lewis wrote:
> If you have a list of IPs known to be infected with AUTH-cracking spambots, it's of immediate/valuable use to both the MSAs themselves in detecting malicious injects, as well as the recipient's filtering, and header forgery is not an issue (certainly not to MSAs, and headers that forge collisions with the list you don't want anyway).

Can you unpack "AUTH-cracking spambots" for the greenhorns?   I have no idea what this means, and google unfortunately was unable to help.

--
Sent from Whiteout Mail - https://whiteout.io

My PGP key: https://keys.whiteout.io/[email-masked]

---

From: Ted Lemon <[email-masked]>
Date: Fri, 04 Dec 2015 02:39:35 +0000

Thursday, Dec 3, 2015 9:31 PM Russ Allbery wrote:
> Standard practice for attackers these days is to automate attacks on any
> sort of password-protected system, whether that be web pages,
> authentication providers, or anything else that takes a password.  Usually
> this is done by taking some list of common passwords and some list of
> account names and just brute-forcing combinations, although some attackers
> do more sophisticated things.
> 
> Obviously, that sort of brute force approach is easy to detect and
> throttle, so the next step in the arms race was for attackers to use large
> networks of compromised machines, usually home machines behind DSL and
> cable modem links, each of which tries a small number of passwords against
> a variety of targets to stay below the radar.  Those machines were
> generally compromised via malware of some kind and are part of a botnet,
> without the knowledge of the user of the machine.

Thanks for explaining!

I am still a bit puzzled: how does increasing the number of attackers help to bypass the throttling mechanism?   Why isn't the throttle per id/password pair, rather than per ip-address/password/id triple?

Secondarily, if distributed processing makes throttling per id/password pair difficult, why is it hard to do the botnet IP address matching at the authentication point?   This seems like it would avoid a _lot_ of extra processing.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 04 Dec 2015 04:22:46 +0000

Thursday, Dec 3, 2015 10:48 PM Russ Allbery wrote:
>> I am still a bit puzzled: how does increasing the number of attackers
>> help to bypass the throttling mechanism?  Why isn't the throttle per
>> id/password pair, rather than per ip-address/password/id triple?
> 
> Why would you throttle per id/password pair?  The attacker doesn't try the
> same pair more than once.  That would be pointless.

I think you missed the distinction I was making.   You are describing throttling per ip-address, irrespective of the password/id pair.   I am asking why you don't simply say "if more than 10 attempts are made on this username per hour, lock it out for a while."   This is a common practice on some web sites.   If you require the user to have a non-dictionary password, then 10 attempts per hour is few enough to prevent a botnet of any number of hosts from guessing that password.

> The TLDR in case something about that message was confusing is that only
> the authentication point can block the IP addresses at the authentication
> point, but you can analyze Received headers to do a bunch of other things,
> such as determine compromised botnet IP addresses that someone else
> *didn't* block but that you *do* want to block for *your* service.  It
> improves the scale and flexibility of what you can do by basically giving
> you more threat intelligence.

Yes, I have heard that before.  It makes sense.   However, it does make me wonder why you don't just stop accepting mail from sites that are this badly run until they shape up.   You seem to be suggesting that it's because you value the intelligence that you glean from their incompetence.   Did I misunderstand?

---

From: Russ Allbery <[email-masked]>
Date: Thu, 03 Dec 2015 18:31:53 -0800

> Can you unpack "AUTH-cracking spambots" for the greenhorns?  I have no
> idea what this means, and google unfortunately was unable to help.

This is used against SMTP AUTH just like it is against anything else on
the Internet that takes a password.  The usual goal is to send out spam
using other people's valid credentials to bypass spam filtering, or to
send phishing or stock pump and dump schemes, or what have you.

One useful tool in fighting this sort of attack is to be able to collect
and share information about currently compromised client IP addresses so
that you can detect them as being part of a bot net and use much more
aggressive rate limiting on these sorts of attempts, or block any email
that they successfully sent after cracking someone's SMTP AUTH password.

-- 
Russ Allbery ([email-masked])              <http://www.eyrie.org/~eagle/>

---

From: Russ Allbery <[email-masked]>
Date: Thu, 03 Dec 2015 19:48:52 -0800

Let me try explaining this from a different direction.  Suppose you are an
attacker.  You have a list of 10,000 common account names, and a list of
10,000 common passwords.  You want to try each of those passwords with
each of those account names and try to find a combination that works.

If you connect from your compromised host and start plowing through the
100 million combinations, after the first few thousand that you try from
that host any decent authentication system will go "hey, there are tons of
failed authentications from this one IP address" and block all further
attempts, successful or not, from that IP address.  That's pretty easy to
implement.

So what the attacker does instead is use their botnet of a million
compromised personal computers (that's sadly not really an exaggeration),
and has each one of those hosts try 100 combinations and then disconnect.
This is now below (or at least near) the threshold for an actual customer
typoing their address or password or something, and rate limiting becomes
fairly useless as a defense.

> Secondarily, if distributed processing makes throttling per id/password
> pair difficult, why is it hard to do the botnet IP address matching at
> the authentication point?  This seems like it would avoid a _lot_ of
> extra processing.

Chris addressed this quite well in his message.  I don't really have much
to add to what he already said.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 03 Dec 2015 20:53:32 -0800

>> Why would you throttle per id/password pair?  The attacker doesn't try
>> the same pair more than once.  That would be pointless.

> I think you missed the distinction I was making.  You are describing
> throttling per ip-address, irrespective of the password/id pair.  I am
> asking why you don't simply say "if more than 10 attempts are made on
> this username per hour, lock it out for a while."

Oh, you didn't mean throttle by id/password pair.  You meant throttle
purely by user ID.

There are two reasons (well, at least -- maybe more) why this doesn't help
as much as it sounds like it would, particularly in the case of SMTP AUTH.

One is that the attacker just changes the pattern of their brute force
search to distribute it across more known accounts or across more time, so
they just probe more accounts with fewer passwords.  It's all the same to
them; they're just doing a combinatoric search, and varying parameters
that way doesn't have too negative of an impact on their ability to
compromise accounts.

Another reason is more specific to SMTP AUTH: dozens of incorrect login
attempts for the same username is a common *legitimate* pattern for users
who have a single misconfigured device (a typo in their password, for
instance.  (And thanks to cell phones, those failed logins often come from
a huge variety of IP addresses.)  SMTP UAs often retry authentication
pretty aggressively and will happily look like an attacker.  If you lock
an account for that pattern, you can lock out that user's other legitimate
devices.  Web sites usually don't have to worry as much about automated
incorrect legitimate login attempts.

Now, you can do more complex things like checking whether the failures are
all the same password or are different passwords; I'm not saying there
aren't multiple approaches here.  My point, rather, is that figuring out
what IP addresses are probably part of botnets is really useful and not
something to be casually discarded as additional information for your
abuse prevention.

This example is obviously simplified in an attempt to answer your original
question, since you said you were wholly ignorant of this whole area of
abuse handling.  Real-world scenarios get more complicated and involve
more complex parameters to rate limiting and ways of figuring out what's
going on.  The point is that threat intelligence about compromised IPs is
really useful input into those systems, and that can be garnered from spam
email Received headers in some fairly effective ways, as touched on by
Chris in his original message.

Note that this sort of thing (login abuse detection) is something for
which many companies employ an entire team of full-time people devoted to
nothing but this.  Those people use all the relevant tools that they can
get their hands on because this is an arms race against quite
sophisticated attackers.

I was (mostly successfully) staying out of this thread, but one other
comment I'll make while I'm here: I suspect a lot of people who work on
email abuse detection and mail filtering would make the argument that they
care just as greatly about user privacy as you do, but consider phishing
attacks a much more serious threat to the privacy of the average user than
the information that leaks out in email headers.  It's one thing to get a
user's home address; it's quite another to get all of their bank
information and access to their personal email.  Obviously none of us want
those two goals to be in conflict, but please remember that spam filtering
is also *phishing* filtering, and therefore is *also* privacy defense for
users, and quite important privacy defense at that.

> Yes, I have heard that before.  It makes sense.  However, it does make
> me wonder why you don't just stop accepting mail from sites that are
> this badly run until they shape up.  You seem to be suggesting that it's
> because you value the intelligence that you glean from their
> incompetence.  Did I misunderstand?

Yes, in a couple of ways.

For one, just deciding to ban whole sending sites from your email system
and saying tough luck if someone at that site wants to mail you is the
sort of thing that you can do if you're running a hobbyist or personal
mail service.  When you have a large-scale email system for paying
customers, it turns out that those customers are unimpressed by your
decision to not let specific people talk to them because you don't like
how they run their mail system.  Or, similarly, if you're running a
large-scale mail system for a company, deciding that your sales people
don't get to hear from certain prospects, or that your business people
simply don't get to talk to certain domains, based on your opinions about
their email policy doesn't go over well.  You have to be pretty damn sure
that the sending site is 100% garbage before you can do things like that,
as opposed to something more sophisticated.

For another, I think you're missing that can garner useful information
from the Received headers of messages that you *reject*.  You can accept
the whole message body first and then reject it or throw it away.  If
you're pretty sure for other reasons that this collection of a few million
messages are all spam or phishing, analyzing the headers of those messages
to pull out patterns like, say, a list of submission IP addresses and then
using that IP addresses as an additional data point in analyzing more
borderline messages, or as a set of IP addresses for which to do
additional login throttling for your own authentication systems, can be
very helpful.  Chris had some concrete numbers in his original message.

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 08:10:58 -0500

> Can you unpack "AUTH-cracking spambots" for the greenhorns?   I have no idea what this means, and google unfortunately was unable to help.

Primarily botnets that connect to MSAs and use stolen (or potentially 
brute-forced) userid/password credentials in order to use the MSA/MTA 
combo as an open relay.

There's at least one major windows executable botnet that does this in 
very high volumes.  This is a particularly long-standing botnet that has 
many other nasty tricks at its disposal (DDOS, keystroke harvesting, 
account stealing etc).

There are several web server compromises that do the same thing, 
potentially using the same sources for compromised userid/passwords, 
tho, these aren't nearly the volume as the aforementioned botnet.

It's really quite surprising how successful that some spam campaigns are 
in doing this, and I can only imagine that at least part of it is  the 
botnet dredging out an infected user's MSA/userid/password triple from 
their mail reader and propagating it to the rest of the botnet (along 
with harvesting address books), and hence have enormous numbers of 
compromised accounts to exploit.  They may also be using more generic 
userid/password dumps from online site leaks.

We've seen ISPs subject to massive attacks where hundreds or thousands 
(or more) of different IPs do authenticated MSA submissions of 10s or 
100s of thousands of spams in fairly short (hours) intervals, resulting 
in traffic flows 100s of times above normal to the point of overloading 
the ISP's MSAs.  This is even with quite aggressive per-user and per-IP 
rate limiting.

AUTH-cracking to this extent is a relatively recent phenomena, and is 
clearly being used as an attempt to bypass normal direct-2-MX botnet 
blocking and hijack the reputation of the MTA instead of some random 
cracked PC.

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 08:26:57 -0500

> fairly useless as a defense.

Hi Russ, it's been a long time since we last were in the same place 
(physically or otherwise ;-)

One of the most depressing indicators I've seen is watching thousands of 
different IPs only attempt one or two submissions, and not trying again 
EVEN THO both submissions succeeded and none failed.

Secondly, we have full instrumentation, and the passwords they're trying 
are pretty obviously not "common password" dictionaries or brute-forcing 
attempts.

Clearly they're not lacking in compromised accounts.

So, the problem is much larger than "brute force or common password 
dictionaries"

> to add to what he already said.

Yeah, it's obvious that the authentication point is the best place to do 
it, but the reality is that most of them don't (we're working on that), 
so...

> you more threat intelligence.

you have to do it for them so that _you_ can block what they should have 
and didn't.

Furthermore, intelligent botnet operators will aim their cannons away 
from the sites successful at MSA-blocking to the ones who aren't.

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 08:46:51 -0500

> as much as it sounds like it would, particularly in the case of SMTP AUTH.

Third and most telling reason: few people know how many valid 
id/password pairs they already have, and coupled with the large numbers 
of IPs they also have, and the fact that userid/password pairs often 
work under multiple services, you can spread the spam across so many IPs 
and userids that throttling is only useful in some marginal edge cases 
(even when the throttle levers are in the hands of a certified BOFH).

Case in point: less than 10% of our list of auth-cracking IPs is derived 
from methods useable by MSAs, and those that are derived from MSAs 
utilize very specialized fingerprinting that has nothing to do with 
throttles.

---

From: Ned Freed <[email-masked]>
Date: Fri, 04 Dec 2015 07:17:55 -0800 (PST)

> Oh, you didn't mean throttle by id/password pair.  You meant throttle

An important reason not to have a pure user ID block is it provides an easy way
to block someone's access to their email: Simply set up a system to constantly
bang on their account with the wrong password, and they're effectively locked
out.

An alternative is a pure user ID slowdown (as opposed to a complete block),
but there are limits to how effective this can be. Bad guys can afford to be
patient whereas legitimate clients time out pretty quickly, turning a 
slowdown back into a block.

> incorrect legitimate login attempts.

I note that this also applies to IMAP and POP3. In fact the most common
case is probably going to be legitimate IMAP clients banging on the door.

Sometimes a single client even does it from multiple threads.

			Ned

---

From: Hector Santos <[email-masked]>
Date: Fri, 04 Dec 2015 10:59:00 -0500

On 12/4/2015 8:10 AM, Chris Lewis wrote:
>> Can you unpack "AUTH-cracking spambots" for the greenhorns?   I have
>> no idea what this means, and google unfortunately was unable to help.
>
> ...

Hi, I'm surprise to read you say this is "relatively recent."  Are you 
mean in months, years or one to several decades?

I have seen tons of "auth-cracking/brute force" auth attempts done at 
all the IP protocol servers. Generally, they do it all in one 
connection session where a "Maximum Failed Login Attempt" count may is 
used.  I always knew it was possible, but I suppose there are now 
increased distributed "botnet" forms of this where the Failed Login 
Attempt Count would be reset at the connection.  I have not seen the 
need to do much because I am not seeing damage. Everything (robust 
high scale servers) have been in place for many years, you know "Set 
it and forget it."

Overall, having robust 24x7x365 load balancing/control IP 
tracking/blocking server(s) is par for the course, so this is more 
about relaxing the STD10/RFC2821/RFC5321.Received SMTP receiver 
protocol requirement to help avoid mitigate negative correlation 
methods with ESMTP AUTH, right?

Incidentally, what I am seeing more of late, are single or "botnet" IP 
connections that do nothing, i.e. they sit there idle and they don't 
attempt to login. What is interesting is that if you (force) 
close/drop it, they immediately reconnect with the same IP.

-- 
HLS

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 11:31:36 -0500

> mean in months, years or one to several decades?

I should say that "back in the day", SMTP-auth from BOTs was 
sufficiently rare that it could safely be ignored.

SMTP-auth from bot started in a noticable fashion about 2-3 years ago 
and continuing to rise to extreme levels in the past 6-12 months.  To 
some MSAs, the impacts were obvious before that.

To me, this is "relatively recent".  Sorry, should have clarified.

As a MUCH more recent development, remember "open relay"?  That was 
obsolete 10 years ago, and except for a couple of low volume Chinese 
spammers, not seen at all.  Well, guess what?  One extremely prolific 
spambot started doing it in very high volumes less than a month ago. 
That's right, spambots attempting to open relay through MTAs.  Shipping 
almost exclusively malware at that.

Fun eh?

---

From: "John Levine" <[email-masked]>
Date: 4 Dec 2015 18:20:52 -0000

>understanding of anti-spam, pretty minor.  ...

If every mail provider had infrastructure and instrumentation as
sophisticated as Google's, that would be more or less true.  But it's
going to be a long time until we get there.

Also, as other people have noted, you can learn a lot from cross
correlating what you see, e.g., I see a fair amount of spam from AUTH
attackers and it's useful to know where the bot is and see whether the
same bots are attacking multiple sites or categories of sites.

We certainly have stuff to talk about, but the tradeoffs are a vastly more
difficult and subtle than the WG's proponents appear to understand.  Chris
Newman's counterproposal would be a good place to start.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Fri, 04 Dec 2015 11:18:21 -0800 (PST)

> some MSAs, the impacts were obvious before that.

FWIW, our customers are seeing this as well. It used to be that AUTH-cracking
on SUBMIT was a nonissue, now it's something you can't safely ignore.

I even see it on my home system. It kinds of amazes me that my little box is
seen as a target worth spending time banging on, but my logs show ~12,000
password guessing attempts in the last 12 hours. (It's all coming from Hong
Kong, the IPs doing it are in the SBL, and it seems to be driven on a generic
list of likely account names, not anything more targeted.)

> almost exclusively malware at that.

We're also getting reports of activities that look like attempts to trick
MTAs into relay through the use of oddball address formats, some legal,
some not. Not sure if this is what you're seeing or not.

				Ned

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 17:19:07 -0500

> some not. Not sure if this is what you're seeing or not.

What you're describing sounds like old-school open relay trickery. 
Indeed, what you could be seeing is decade or older open relay testers 
being used to scan for "buggy" open relay prevention.

What I'm seeing isn't that at all.  What I see is more like "for 
everybody I want to spam (address A), pick another address (address B), 
connect to address B's MX, forge the email to be From: address B, and 
attempt to get B's MX to relay to address A".

However, it's possible that what I'm seeing would shift gears to more 
obvious trickery if the first attempt failed.  It you see the same IP 
hitting more normal relay rejects before trying the oddball addresses, 
it could be.

If you sent me a log record or received string[s] (off-forum please) I 
can probably tell for sure.

---

From: Brandon Long <[email-masked]>
Date: Fri, 4 Dec 2015 14:51:19 -0800

On Thu, Dec 3, 2015 at 6:00 PM, Chris Lewis <[email-masked]> wrote:

> take up pages of text.  I'll try to short cut - purely from a filtering but
> not-ML perspective:
>
> Using them:
>
> demonstrably capable of AUTH-cracking) can see botnet suppression rates at
> the MSA of 80% (of all submissions) or even higher, depending on the
> spam/botnet de-jour.  In just one case it's in the 1000+/second range. Some
> of these botnets are very very prolific per IP.
>
> receivers being able to stop it at destination.  If a significant fraction
> of MSAs stopped it at source, then, maybe, the MSA Received line isn't
> important.  But where they don't (the majority), losing the MSA Received
> line is potentially a big hit.
>
> Obtaining them:
>
> derive trustworthy infected submission addresses from received lines. Yes,
> actually, it really is.  Useful, but not terribly significant (or at least
> how far we've followed that rathole).
>
> On one server, in a 40 minute interval, 774123 out of 1126763 connections
> were AUTH spoofing from just one particular botnet.  That's 775,000 spams
> that ordinary receivers couldn't filter via IP if the from clauses
> disappeared.
>
> generally don't use real mail servers (except via AUTH crackers), and the
> top 3 are amply well instrumented with rate limiters etc.  So they really
> aren't relevant to this.
>

We've had to deal with hijacked account spam for years at this point, and
have invested heavily against it, and why we are doing our best to sunset
"programmatic" password based auth, ie:

https://googleonlinesecurity.blogspot.com/2014/04/new-security-measures-will-affect-older.html

https://support.google.com/accounts/answer/6010255

Of course, oauth isn't quite ready for widespread adoption, due to issues
with discoverability of end-points and software registration.
And a botnet may be able to lift the oauth tokens out of the local
software, but it has some benefits.

Brandon