Subject: [Asrg] IPv6 Reputation / DNSBL

From: "Jan Griebsch" <[email-masked]>
Date: Wed, 12 Jan 2011 12:54:30 +0100

Hi,

having read through discussion threads here and on  
spamassassin-users@apache
regarding John Levine's  DNSBL implementation proposal, I have a related
question. I post separately, since my question is more general, and I dont
want to pollute these threads.

Motivation for IP-based reputation today is broader than SMTP traffic.
It is used as an resource-efficent first estimate,
wherever service providers decide on letting users use their resources
(create accounts etc) or on taking responsibility for user's content.

Even for the mail context, it seems to me short sighted, to only consider  
SMTP
traffic. For example, a lot of the private-user-to-private-user mail  
traffic is
generated by web-/freemailers. If their account creation process is  
over-run
(because no functioning IPv6 reputation solution + CAPTCHA is broken  
anyways),
outbound spam through their IPs will increase accordingly, and the DNSWL
approach discussed in aforementioned threads will tend to become  
meaningless
for private-to-private email.

If you accept the above premise, a more general approach
to the problem could be subdivided into 3 parts:
1) Which entities (/128s, /64, variable prefixes..) do we assign  
reputation to ?
2) What are suitable backend datastructures/implementations
    for efficient lookup/updates/caching for the assumed (number of)  
entities ?
3) What are suitable mechs/protocols for querying and updating/distributing
    the entity-reputation data ?

In my mind, the central, totally unresolved part is 1).
If we can come up with a reasonable (see below) solution for 1), then 2)
should be solvable as well. 3), as discussed in Levine's threads, I see in
this context, as a negotiation/convention issue (with tricky tech.  
problems).
Not to diminish it - but I want to make clear my point, that, if 1)
is not solved, 2) and 3) are moot - at least for blacklisting.

on 1): criteria for reasonable reputation entities
* the number of entities must be limited to something much smaller then  
2^64
   <- for computational cost
   <- for meaning; in the end, "reputation" means judging the history of  
behavoir
      of a real-world communication/inter-action partner
* corollary: creating entities must incur some type of cost
   <- e.g. simply rotating your hosts /64 suffix is for free
   <- e.g. money for registering a domain
   <- e.g. effort for an attacker to take over a host

on 1): a more concrete question

Can we manage to deduce (for traffic seen)

* the internal partitioning of ISP's (e.g) /32s into subnets
* classify each subnet into static vs. dynamic IP address assignment
* for static SNs, the end-customer prefix (length)
?

Then, I think, we'd have gone a long way to solving 1). We would assign  
reputation
to prefixes, that are tied to hosts/end-customers and could, deny  
dialup/dyn.IP
communication as desired (today via e.g. PBL).
2) would then amount to solving the LCP (longest common prefix) problem  
for a
reasonable (order of 10^8) number of prefixes. A lot o academic work
exists for that, and router do it, too.

Any ideas ?

Regards,
--jan

---

From: "Mark Scholten" <[email-masked]>
Date: Wed, 12 Jan 2011 14:50:31 +0100

Hello Jan,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Jan Griebsch
> Sent: Wednesday, January 12, 2011 12:55 PM
> 
> Hi,
> 
This is a problem that depends on how the web-/freemailers setup their
systems. As long as the traffic stays within the mail servers of that
company I think they could use some in-house tool to filter probably abusive
users. As soon as they send it to others their reputation is on the line and
they should act to defend it accordingly (by trying to prevent abusive users
to sign up/block email/filter outbound/etc.).
> 1) Which entities (/128s, /64, variable prefixes..) do we assign
If you ask me we should do it for multiple ranges. Let me explain it below:
- 2 spam messages per [time] per /128 could be "acceptable", with more block
that /128. For a good reputation a /128 should send at least XX messages per
[time] without X being marked as spam.
- 10 spam messages per [time] per /64 could be "acceptable", with more block
that /64. For a good reputation a /64 should send at least XX messages per
[time] without X being marked as spam.
- 50 spam messages per [time] per /32 could be "acceptable", with more block
that /32. For a good reputation a /32 should send at least XXX messages per
[time] without XX being marked as spam.

>     for efficient lookup/updates/caching for the assumed (number of)
PowerDNS with MySQL backend would work for DNS based lookups probably (it
does caching, it is efficient, it is easy to update, lookups are easy and
adding capacity is easy).

>     the entity-reputation data ?
Querying: DNS or some http/https form is probably the easiest solution (with
DNS you could cache answers more easily).
Distributing: Depending on the solution I'd look for AXFR/zone files/MySQL
master-slave replication/CSV files. I think we'll first have to decide what
is good for 1) and 2) and let 3) depend on that answer.

>    <- e.g. effort for an attacker to take over a host
That is why I'd look for some reputation based on multiple ranges to look
for the best action to take. If you've an ISP that has paying clients that
are hurting others with their bad reputation they are more likely to say
good bye to the clients that are hurting business. Or do you've a better
idea for this?

> * the internal partitioning of ISP's (e.g) /32s into subnets
Difficult if the ISP isn't cooperating/updating data sources.

> * classify each subnet into static vs. dynamic IP address assignment
Depends on the ISP and that makes it difficult to use I think.

> * for static SNs, the end-customer prefix (length)
Difficult if the ISP isn't cooperating/updating data sources.

> exists for that, and router do it, too.
With a good DNS based solution this shouldn't be a really big problem I
think.

Regards, Mark
> 
> Any ideas ?
> 
> Regards,
> --jan
> _______________________________________________
> Asrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/asrg

---

From: "Jan Griebsch" <[email-masked]>
Date: Thu, 13 Jan 2011 12:21:04 +0100

Hi Mark,

> to sign up/block email/filter outbound/etc.).

True, but you're essentially saying ESPs will have to cleanup spam(mers)
later, expensively, that they could have prevented earlier if they had
had working IP filters/reputation. True, IP filters do little/nothing
that one couldnt do without them. But the issue here is being efficient
(or: being overrun if you are not). Also, outgoing spam-filtering is just
more error prone (worse filter training, user-feedback, lack of viable
consequences if unsure of classification).

> [time] without XX being marked as spam.

2 problems:
* You dont want to punish whole groups for a few bad apples. If "group"
means another ESP (/32) this could hurt your bottom line outright.
* You dont want to give spammers freebies either. AFAIK, some ISPs even
give /48s to "end-customers". If your count onto /64 thats, according to
your example 10*2^16 freebies.

> good bye to the clients that are hurting business. Or do you've a better

What I was trying to say (in a rather overly generic way) is, that
it doesnt make sense to count & give *bad* reputation to entities
that are (almost) infinitely available (/128s and even /64s).
Even if you could, computationally. Because attackers would simply
keep creating new entities.

> Difficult if the ISP isn't cooperating/updating data sources.

.."Difficult".. Exactely. ** But somewhat/somehow doable ? **

If we cant aggregate /128s /64s etc. onto prefixes that are
tied to meaningful, costly entites (hosts, customers, companies..),
then, I think, (bad) IP blacklisting is dead, independently of all
clever solutions to 2) and 3).

Regards,
--jan

---

From: "Mark Scholten" <[email-masked]>
Date: Thu, 13 Jan 2011 14:23:49 +0100

> From: Jan Griebsch [mailto:[email-masked]]
> Sent: Thursday, January 13, 2011 12:21 PM
> Hi Mark,
> 
> > users. As soon as they send it to others their reputation is on the
> line
> > and
> > they should act to defend it accordingly (by trying to prevent
> abusive
> > users
I know that outgoing spam-filtering is more difficult. However I believe that it is at least required to filter some abuse that is clear for most filters (viruses/spam with high scores with default spam assassin setups and the like). Lists could also be used to limit number of email accounts per IP (IP range) or block bad users. In the beginning this will be difficult (lack of data).
> > - 2 spam messages per [time] per /128 could be "acceptable", with
> more
> > block
> > that /128. For a good reputation a /128 should send at least XX
> messages
> > per
> > - 10 spam messages per [time] per /64 could be "acceptable", with
> more
> > block
> > that /64. For a good reputation a /64 should send at least XX
> messages
> > per
> > - 50 spam messages per [time] per /32 could be "acceptable", with
> more
> > block
> > that /32. For a good reputation a /32 should send at least XXX
> messages
> > per
If blocking per /128 didn't work you should look for bigger ranges to block. Also limiting it to x bad /128 before calling a /64 bad is an idea. In that case you could increase the range you block/do extra checks on when multiple /64 ranges are considered bad (/56 or /48 or even /32). I don't prefer the last option, but if the ISP isn't providing valid whois data you really don't have another choice than consider the ISP as being bad.
> your example 10*2^16 freebies.
In my example I increase the range to block when there is more abuse coming from that range. For that you've of course multiple options.
> >>    <- for meaning; in the end, "reputation" means judging the
> history
> >> of
> >> behavoir
I'd increase the range when more IP's are used for bad behavior. That is also an option currently in use at some location for IPv4 and I don't see a reason to not do that again with IPv6. There are just too many /128s or /64s or /48s for that matter available in a /32.
> .."Difficult".. Exactely. ** But somewhat/somehow doable ? **
It is possible to aggregate it based on whois data, however if an ISP isn't providing the correct data (or has to many bad users) other action should be taken if you ask me.
> clever solutions to 2) and 3).
The option I did mention doesn't have to be extremely costly. You could apply some other rules before accepting the traffic (for example that it comes from space that is officially given out by a RIR). After that it is indeed difficult to block ranges without problems (if you don't also have a whitelist with known "normal" mailservers). This system could be used against botnets I think, for normal mailservers you need to look for some % of legit mail to never block that IP/range. For us this limit would probably be around 50% (or higher).

Regards, Mark

---

From: David Nicol <[email-masked]>
Date: Tue, 18 Jan 2011 08:45:55 -0600

On Wed, Jan 12, 2011 at 5:54 AM, Jan Griebsch <[email-masked]> wrote:

> 1) Which entities (/128s, /64, variable prefixes..) do we assign reputation
> to ?

Variable prefixes. Duh. Does this point need expansion?

---

From: John Levine <[email-masked]>
Date: 18 Jan 2011 15:43:31 -0000

>Variable prefixes. Duh. Does this point need expansion?

Depends on the application.  For a whitelist of mail hosts, probably
/128s.  For a blacklist of zombies, probably /64s.  For a blacklist of
evildoers, arbitrary ranges depending on their network sizes.

I agree that any technical approach to publishing BLs and WLs needs
to be able to handle different prefix lengths.

R's,
John