Subject: [Asrg] The "Human-Shield" effect; the need for end-user control

From: Walter Dnes <[email-masked]>
Date: Sat, 22 Mar 2003 18:23:33 -0500

Hello;

  I've reviewed the past few days' traffic, and haven't seen this item
mentioned (maybe I overlooked it).  ISPs can block the most obnoxious
and egregious spammers *ON AN ISP-WIDE BASIS*, and all their customers
are happy.  However, outfits like Topica and Shagmail, that run
unconfirmed lists are a much more difficult problem.  The basic reason
is that their lists (or their customers' lists) do contain *SOME*
willing subscribers.  Let's pick an example number of 10%.  Let's also
assume random distribution.

  An ISP wants to block their biggest sources of spam.  Block 1 source,
and 90% of your customers are happier, because they see less spam.  10%
lose emails they want.  Oops.  Block a second spamsource.  81% of your
customers are happier (less spam), 19% lose emails they want.  Etc, etc.
Keep this up, and you will eventually alienate most of your customers on
a piecemeal basis.  ISPs don't want to risk this, so they don't block
any but the most obnoxious spammers.  The big mailers and/or their
customers know about this "Human Shield" effect.  And they probably
have a good idea of the actual "pain threshold" at which ISPs will start
blocking over this.  Because it's 100% identical content, coming from
the same IP address, no ISP-wide filtering algorithm can possibly be
100% accurate.  So the spammers can get away with diluting their lists,
and bump up the head count for advertising purposes quite a bit without
being blocked.  Blocking countries which effectively mailbomb your
customers carries the additional risk of being called racism in today's
politically correct climate, in addition to alienating the usual small
number of your customers who might have friends/family/business contacts
in South Korea, Nigeria, China, and Taiwan.

  The obvious solution to this is to give end-users the final say on a
per-user basis.  This is usually done after-the-fact.  It has drawbacks
which all stem from the fact that the entire email has to be received...

  - broadband ISPs in Canada surcharge their customers for bandwidth in
    excess of a monthly quota.  I'm not into P2P, so I never get near my
    quota, however, for anyone who is already at or above their quota,
    receiving the body of the spam email costs real money.

  - files have to be stored somewhere.  This means more and/or bigger
    hard drives.  The resulting higher costs are ultimately passed on to
    the consumer.

  - filtering on DNSbls is difficult, and filtering on the user's choice
    of DNSbls even more so

  - filtering on lack of "in-addr.arpa" rDNS, a very good spamsign, is
    also difficult

  End-user-control at the SMTP stage is actually possible, but it's a
relatively new thing.  I'm a customer of a small ISP that runs a
modified qmail, which parses a filter configuration file in the user's
home directory just after the RCPT: stage of the SMTP transaction.  If
the filter indicates the email should be rejected, qmail sends a 550
reject before the DATA: stage begins.

  43 spams in 10 days from South Korea in Korean character set that I
can't read (and my computer can't display) at the beginning of November
2002 is now just a bad memory.  Ditto for the sob-stories allegedly
from Mrs. Miriam Booga, widow of the late dictator of Nigeria, General
Ooga Booga.  Thanks to the zz.countries.nerd.dk zone, South Korea,
Nigeria, China, and Taiwan have disappeared from my email universe.
Blocking the 200.0.0.0/8 CIDR blocks the latest spammer attack route of
open proxies and/or willing hosts in South America.

  The USA has its share of spams coming from it.  Between...
  - Osirusoft which also offers other zones like Spamaus.org's SBL zone
  - open-proxy zones
  - dynamic IP zones
  - a few CIDRs that I've hard-coded in
  - blocking MTAs with no rDNS
..the torrent of US-originated spam has dropped to a trickle.

  In addition to me being a happy customer, the ISP has not alienated
any of its other customers by blocking email addressed to them.  And
the ISP would obviously have less liklihood of being hit by frivolous
lawsuits like Cyberpromo (Sanford Wallace) demanding that ISP-wide
filters be taken down at AOL.

  There are a few drawbacks...
  - this will obviously *NOT* work with multi-hop MTAs that have the
    external-facing machine blindly accept all email and relay it to a
    second machine that does the final delivery.
  - spammers are persistent.  If you block them at your primary MX,
    they'll try secondaries.  Some spammers try secondary MX *BEFORE*
    trying the primary.  Either the backups are identically configured,
    or else you have to run with only one MX.
  - there are questions about scalability when it comes to spawning a
    process that does multiple DNSbl lookups for each incoming email.
    This is less of a problem on *nix-based systems, where unnecessary
    processes can be shut down by sysadmins; even excluded in a
    custom-compiled kernel if necessary.  Windows-based MTAs have
    overhead from a gratuitous gui and a whole bunch of processes that
    aren't really necessary on a dedicated MTA.  And you can *NOT*
    modify/re-compile the "Windows kernel".
  - instead of a sysadmin configuring a potentially complex filter, the
    average end-user has to configure it.  Imagine your Aunt Ethel...
    - logging in to a shell.
    - parsing email headers to determine the originating IP address.
    - running whois to get more info about the spammer.
    - based on the output from whois, deciding which CIDR to block.
    - editting the filter config file with vi to implement the block.
    menu-driven configurators can do basic stuff, but getting the most
    out of the system requires getting one's hands dirty editing the
    config file.
  - since I'm a customer, not an admin, I don't have access to the log
    files.  The ISP emails me a listing of rejects on a monthly basis.

-- 
Walter Dnes <[email-masked]>
An infinite number of monkeys pounding away on keyboards will
eventually produce a report showing that Windows is more secure,
and has a lower TCO, than linux.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: <[email-masked]>
Date: Sat, 22 Mar 2003 19:43:12 -0500

" broadband ISPs in Canada surcharge their customers for bandwidth in
excess of a monthly quota".

1 movie at 10 GB is the same as how many spam messages? Given that 100GB
will soon be the norm...

... why do people think that spam is equivalent to the end of the world
and that  we must give up free speech and be complete trackable?

The real danger on this list is that it attracts those who want to save
us from something and think it is the most important thing. I joined
because I believe we can do better with email and that spam is simply a
side-effect of simplistic protocols and, for now, I do deal with it on
the client side. ISPs may also choose to provide services for their
users.

But, relax, it's not the end of the world. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Walter Dnes
Sent: Saturday, March 22, 2003 18:24
To: ASRG list
Subject: [Asrg] The "Human-Shield" effect; the need for end-user control

Hello;

---

From: [email-masked]
Date: Sat, 22 Mar 2003 15:07:57 -0800 (PST)

I'v thought about all this before. I think it might be worth our time to 
define standard "mail-filter configuration" protocol. I imagine this being 
as something that would be located on some central database server and 
it would have "user configuration" update queries and "mail server check" 
read-only queries. 

All inbound mail servers (MX records) for domain would need to be able to 
interconnect to this database and parse filtering rules that they support. 
To make it easy for them to know where to connect to, best would be to 
define new SRV DNS record for mail filtering server, then if backup mx 
server supports it, it would lookup dns zone, pickup up one of such SRV 
records (more then one record/servers could exist for backup purposes, but 
all would have the same configuration, synchronization should be built as 
additional feature into protocol). 

For client side they could have special program just for filtering 
configuration, it could be built into mail client or could be web-based, 
but all would be able to access same configuraiton database. Also when 
users moves from one isp to another, he could retain his settings, which 
is very usefull.

Also as far as saying protocol, it would not necessarily be new protocol, 
but could be based on LDAP with defined scheme. But we would need to 
evaluate several protocols and create list of requirements like this is 
being done at CRISP working group working on new whois protocol.

What do you think of the above idea? I know there are lot of people on 
the list that represent anti-spam filtering software solutions, do you 
think you could work together to create standard for this?

---

From: Kee Hinckley <[email-masked]>
Date: Sat, 22 Mar 2003 20:34:54 -0500

>think you could work together to create standard for this?

There are some issues.  Your filter server's capabilities may not be 
the same as mine, for instance.  But in general, the idea of a common 
mechanism for expressing your filter desires (what countries, what 
languages, what's the white list, what's the blacklist, what IP's or 
RBLs...) is a good idea.  We have already done some preliminary work 
in this area with our product.

While you could standardize the database<->MTA communication layer, I 
suspect that's going to be pretty custom from product to product. 
However it definitely makes sense to have a standard mechanism for 
getting filter information from the MUA.  And (although I shudder to 
hear myself say so), some type of XML specification would probably 
make a lot of sense here.  It has the flexibility such that you could 
work with different filter products, but still have a common 
front-end for the settings.  The back-end filter can provide a 
capability list.  The MUA or filter client would let you specify the 
values--providing custom interfaces for the ones it understood, if 
necessary, and generic interfaces for the rest.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Kee Hinckley <[email-masked]>
Date: Sat, 22 Mar 2003 20:44:03 -0500

>and that  we must give up free speech and be complete trackable?

Because we don't live in a fantasy world where 100GB is the norm.

In urban areas, in developed countries, you *might* continue to see 
the increase in bandwidth that you are talking about.  But I know a 
lot of people who's internet connections in the past 10 years have 
increased by a stunning 2x.  (28.8 to 56).  And that's in the U.S.. 
Please think globally.

>But, relax, it's not the end of the world.

My daughter has a fifth grade classmate who spends 15-20 minutes 
during  lunch break cleaning the spam out of her in-box--every day. 
And she's not the only one doing it.  Do you think those email 
accounts are going to be useable by the time they graduate from high 
school?

Nobody is looking to make everyone completely trackable.  What we are 
aiming for is making people *accountable*.  I believe it is fully 
possible to make a system whereby people are accountable for the 
email they send (by domain, or by email address, or internet 
connection) without having that email linked (any more than it 
already is--which is pretty tight, all things considered) to a 
physical individual.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: [email-masked]
Date: Sat, 22 Mar 2003 18:24:59 -0800 (PST)

No, I was thinking of of more complex scenario then just standartized xml 
filter list. While I agree that current spam filters implementations often 
use different methods for representing and getting rule set, I do not 
think we're at the point yet that we can not create a standard 
distribution mechanism and having central database server can 
often be easy (it may not have to be a requirement, i.e. one can only 
implement MUA->server filter update mechanism but large company/network 
with many incoming mail server would really benefit from cental server and 
if we standard way to represent rules, going futher to distributing them 
is not really such hard work, many implementations for such distributions 
already exist, its just a metter of choosing right one)

And I was thinking futher along the line of creating standard header lines 
on what filter has been used (for client MUA to know in standard format) 
and in order to allow for distributed filtering systems and removing 
necessity of applying same filter twice.

---

From: Matt Sergeant <[email-masked]>
Date: Sun, 23 Mar 2003 10:42:01 +0000 (GMT)

> in this area with our product.

As have the IETF. See SEIVE.

Matt.

---

From: "Jon Kyme" <[email-masked]>
Date: Sun, 23 Mar 2003 11:07:40 +0000

> it would have "user configuration" update queries and "mail server check"

Yes, perhaps.
There was a (short) thread on 
"What would consent look like?"
started on
Mon, 10 Mar 2003 10:45:34 +0000

Which might be relevant here.
:-)

---

From: "Jon Kyme" <[email-masked]>
Date: Sun, 23 Mar 2003 11:31:20 +0000

> work with different filter products, 

This is why we need some kind of consent-expression framework
(which I'm still trying to come up with a rough draft of) I like the idea
of (optionally signed) XML (shudder also).

Filter Configuration is just Consent Expression
turned on it's head - kind of

Also needed - a classification of message classification methods, so that
the end user can mane (and constrain the employment of) generic or
(specific/proprietary) mechanisms.

You also need a way if passing parameters - I guess - e.g.
probablities/scores

I'm interested in this so that I can build in to my webmail system a UI to
the filtering system which I can distribute better than the current system
which has to dump application specific filter files into the right place in
my MTA filesystem (uggh).
But obviously more general applications would work.

I'd expect consent to be available from a per-domain service such as
"consent.merseymail.com" or "merseymail.com:consent" and be cacheable (this
requires some TTL field).

It's likely such a system would support legislation which requires proof of
consent in advance.

---

From: [email-masked]
Date: Sun, 23 Mar 2003 01:50:50 -0800 (PST)

Clear Consent is when end-user said he wants to receive email from you 
(whitelisting of email address or domain). Clear No Consent is when user 
specifically said he does not want to receive email from you (blacklisting).
Specifying of other mail filters are consent by user to use specific 
technology to filter out email. 

Anything that does not get stopped by the filter goes into final step 
which specifyies user's preference - either presumed consent or presumed 
denial. Denial may also be replaced by request for authorization which
maybe request to "buy" or calculate a stamp, go to specific webpage and 
look at the picture (or hear a sound) and aythorize there, send email to 
some other address, etc, etc.

And when user provided these settings for filtering this is alot better 
then presumed consent filtering by isp based on its own preferences and 
its opinion on what consent is.

---

From: "Jon Kyme" <[email-masked]>
Date: Sun, 23 Mar 2003 11:54:07 +0000

> > the idea of a common 

No. sieve is a filter language. I don't believe it's a general mechanism
for configuring disparate
"filter engines".

---

From: "Jon Kyme" <[email-masked]>
Date: Sun, 23 Mar 2003 11:57:53 +0000

> its opinion on what consent is.

Yes absolutely, that's why you need a consent expression framework.

---

From: Matt Sergeant <[email-masked]>
Date: Sun, 23 Mar 2003 13:38:59 +0000 (GMT)

> for configuring disparate

Yes, I realise that, but I was thinking of it as a starting point - it 
defines a common ground in the filter space, so it may be worth re-using 
its syntax.

Matt.

---

From: Kee Hinckley <[email-masked]>
Date: Sun, 23 Mar 2003 12:56:36 -0500

>technology to filter out email.

I think that defining consent in terms of filters is very confusing 
and inaccurate.

If I consent to receive a newsletter about books from Example, and 
they start sending me a newsletter about Toys, then they haven't 
listened to my request.  That has absolutely nothing to do with 
whether I've whitelisted [email-masked].

I'm not sure if you are really combining whitelisting with that 
concept of consent, or just using the same term in a different place, 
but I think it would be a good idea not to use the term in both 
places.

---

From: Kee Hinckley <[email-masked]>
Date: Sun, 23 Mar 2003 11:33:42 -0500

>use different methods for representing and getting rule set, I do not

I understand.  It's just that it's a harder job for filter software 
to integrate with it, and there's a tendency to not focus on things 
which make it easier for your clients to switch to a competitor :-). 
But yes.  I think it's possible, and a standard in that area would be 
good.

I would propose however, that if we're going to specify such a 
thing--let's start on the side of the architecture that everyone else 
seems to leave until last. Namely--how does it handle authentication 
and security.  Keep in mind that such a protocol would possibly also 
run over the WAN, as it would provide a way for third-party filtering 
services to manage your mail filtering, while you control the rules. 
(Or vice versa--you do the filtering, and the third party keeps your 
spam filtering rules up-to-date.)

>necessity of applying same filter twice.

That gets harder.  Products like ours and SpamAssassin may apply 
hundreds of rules, and what is far more important than which rule 
fires, is how the combination of results is weighted.  If you are 
specific to the point of "we checked for this string" then you have 
too much data.  And if you're general to the level of "we checked for 
strings" then you haven't really said anything of use to the next 
filter in the line.  (I'm also not a fan of adding lots of new 
headers--but I think this has problems even before we get to that.)
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: Kee Hinckley <[email-masked]>
Date: Sun, 23 Mar 2003 13:00:29 -0500

>As have the IETF. See SEIVE.

That looks like an attempt to write a programming language for 
filtering text in email format.  That is *way* more specific than 
simply configuring an existing filter.  Has anyone actually 
implemented it?
(http://www.ietf.org/rfc/rfc3028.txt?number=3028 for those wondering)
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: Matt Sergeant <[email-masked]>
Date: Sun, 23 Mar 2003 20:05:11 +0000 (GMT)

> simply configuring an existing filter.  Has anyone actually 

Yes, a few. http://www.cyrusoft.com/sieve/#implementations

As I said in another mail, I was thinking of this as something to build 
on, rather than inventing a new syntax (even a syntax of XML).

Matt.

---

From: Tony Finch <[email-masked]>
Date: Sun, 23 Mar 2003 20:02:50 +0000

Kee Hinckley <[email-masked]> wrote:
>Has anyone actually implemented it?

http://www.google.com/search?btnI&query=sieve ->
http://www.cyrusoft.com/sieve/#implementations

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
THE WASH TO NORTH FORELAND: SOUTHEAST 3 OR 4 VEERING SOUTHWEST 3 OR 4. MAINLY
FAIR; SOME MIST PATCHES; SMALL RISK OF A SHOWER MONDAY . MODERATE BUT POOR IN
MIST. SLIGHT.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: [email-masked]
Date: Sun, 23 Mar 2003 12:19:57 -0800 (PST)

> which make it easier for your clients to switch to a competitor :-). 
I was thinking more along the lines of allowing to use multiple products!

> spam filtering rules up-to-date.)
Standard user/password authentication for updating central repositatory.
Possibly TLS if desired.

SHA1 for reporting filtering in headers for confirmation with key at 
the central server. Or maybe even x.509 to be absolutly sure.

And those are just quick thoughts on that. Proper authentication can be 
done with no problem, we have many examples with other protocols, 
including SMTP.
 
> fires, is how the combination of results is weighted.
Actually spamassasin is quite good at reporting what has been checked as is.
I'v no problem with rules being weighted, but I'd like to know what each 
resulted in. And it really not 100s of rules (not yet...), it maybe 100s 
in terms of long filter list but actual top-level rules are about dozen.

> If you are 
> too much data. 
Don't want that. 

> And if you're general to the level of "we checked for 
> filter in the line
"Checked user-specific whitelists" would be good enough for me.

> (I'm also not a fan of adding lots of new 
Possibly. But for example mentioned spamassasin already adds headers and 
so do other filtering products. Each reports in its own way, though.

---

From: [email-masked]
Date: Sun, 23 Mar 2003 12:29:35 -0800 (PST)

I agree. SIEVE filter can be one of filters in the list but not the only 
one.

> simply configuring an existing filter.

---

From: [email-masked]
Date: Sun, 23 Mar 2003 12:28:15 -0800 (PST)

I see your point. Yes, possibly consent by whitelisting is not "clear 
consent" but its best somebody can express in technical terms. I do 
understand though that even after whitelisting somebody may want to run 
some extra filters, but I'd expect this to be rare and when they violate 
your "consent" and send you stock tips instead of anti-spam technology 
letter, then possibly this would mean end of presumed white-listing 
consent... At that point you would go and modify your preferences.

---

From: Kee Hinckley <[email-masked]>
Date: Sun, 23 Mar 2003 18:11:07 -0500

>in terms of long filter list but actual top-level rules are about dozen.

That's SpamAssassin.  The point is.  How are you going to standardize 
the names of rules when, virtually by definition, the rules are 
constantly changing?  You said the point was so that the next filter 
in line wouldn't run something redundantly.  I don't see how that 
works.  Filters don't work that way.  You give the example of having 
applied a whitelist--but that's about the easiest example.  And I 
don't see how you can put these all in a headers without ending up 
with a lot of junk in the headers.

>so do other filtering products. Each reports in its own way, though.

Yes.  That doesn't mean we should make the situation worse.  I've 
been real careful to keep our output to one single header--and that's 
primarily for reporting misfiled messages.

Now if you want to have a standard way for filters to report their 
results--that seems like a good idea.  It would certainly make the 
MUA developer's job easier.  I just don't see the point in listing 
what rules fired.  But a way for multiple filters to report their 
results in a standard way sounds good.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: "Jon Kyme" <[email-masked]>
Date: Mon, 24 Mar 2003 08:46:52 +0000

> on, rather than inventing a new syntax (even a syntax of XML).

I'd be really surprised if one could build on a language
which has the intention "...to make it impossible for users to do anything
more complex (and dangerous) than write simple mail filters"
so as to create a general mechanism for configuring arbitary filter
engines.