Subject: [Asrg] Lets Fix Mailing Lists

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Sat, 8 Mar 2003 16:42:32 -0800

Lets face it the management of mailing lists is a mess. A member of the IESG
told me that this is because the mailing lists and mail MUAs don't adhere to
the specs. An employee of a company that writes one of the most widely used
MTAs gave a reply that to use the J.K.Rowling technique would go
'horsepuckey, indecipherable, impractical, arrogant, not suprised'.

I think that we need to address mailing lists as a special case for action.
In particular it should be possible to communicate to MTAs in the
communication path the fact that a user has opted in to the list.

One reason for this is that if my mailer could detect mail from mailing
lists reliably I could then ditch the 50% of the spam I get that is not
addressed to me at all. [Yes I know that they would adapt, but their current
behaviour has a purpose and I want to block that purpose].

If we are going to get the mail infrastructure sorted out for the sake of
stopping spam we might as well sort out a bunch of the other pathologies it
currently suffers from if the cost is low. I believe that the way we will
eventually end up propagating whatever changes are required will be through
some form of branding, whether it be 'RFC6388 Compliant' or 'JamSpam
Compliant' or whatever.

My pet mailing list peves:

  1   Reply to sends the reply to the original author, not 
	the mailing list. Reply to all sends the reply to the 
	mailing list and to all prior authors, creating a shadow 
	mailing list

  2   Out of office responses getting sent to list members

  3   I have to tell the mail client to create a folder for each list

  4a  Delays introduced by anti-spam moderation
  4b  Time taken to do anti-spam moderation
  4c  Time spent reading idiotic posts from 'Prof' Bernstein 
	complaining that the anti-spam filtering is censorship.

  5   Spam on mailing lists

  6   Opt-in verification to prove its me when my original request
	had a digital signature and cert.

  7   Unsubscribe that does not work (often due to burster lists)

  8   Web archives that only show an index of 20 messages at a time
	so you have to backspace through 15 pages when catching up.
	[OK its only the IETF that configures MonHarc this stupid way]

  9   Maintenance messages that get sent to the list complaining about
	the above.

 10   Having the messages from a list get trapped in some dozy
	spam filter.

Lots of these are synchronization problems. There is one basic 
behavior that people expect from mailing lists. However the 
clients and mailing lists have slightly different configurations,
work arrounds etc. that causes chaos.

A header that identified mail as comming from a mailing list
would help a lot here. There have been proposals, but how do 
we get to DEPLOYMENT???

On the spam issue what we need is a means of communicating 
consent from the MUA to MTAs in the path. This can be done 
efficiently using either symmetric cryptography or public
key.

If we are using a symmetric scheme Alice generates a key and
sends the symmetric key to her receiving MTA. if we are using
a public key scheme Alice probably has her key already, 
the MTA can pull the public key from her XKMS service.

To subscribe to a list Alice goes to the 'subscribe to list'
option in the tools menu of her client. This generates a
subscription blob that contains Alice's email address, the
address of the list [possibly including a wildcard], date
and is signed with Alice's key. We might want to use DSA 
for this for compactness. 

The subscription blob is sent to the mailing list server 
manager. If the mailing list is compliant then it will 
subscribe Alice immediately.

The mailing lists stores Alice's subscription blob along 
with each message and sends it out with each mailing. This
allows Alice's MTA (and if public key is used all the 
intervening MTAs) to determine that the message was solicited.

Legacy mailing lists can also be supported with a bit of 
intelligence on the part of the subscriber's MTA. 

Basically it records the fact that Alice has subscribed 
and then records the fact she is subscribed locally.
It can also process the opt-in callback loop for her.

		Phill
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Vernon Schryver <[email-masked]>
Date: Sat, 8 Mar 2003 18:18:18 -0700 (MST)

> From: "Hallam-Baker, Phillip" <[email-masked]>

> ...

Those are bugs and errors by users. 
There's nothing the IETF can or should do about it.

>   2   Out of office responses getting sent to list members

Even worse is when out-of-office responses go to the list.
But both are well known bugs that the responsible parties refuse
to acknowledge or fix.

>   3   I have to tell the mail client to create a folder for each list

"Folders" are features of MTAs beyond the appropriate influence of the IETF
(or IRTF).  Just as the IETF doesn't presume to talk about GUIs, window
managers, and other stuff, it shouldn't talk about MTA user interface 
features or bugs.

> 	complaining that the anti-spam filtering is censorship.

The first two may have something to do with the IESG.  The entire IETF
including IAB and IESG has working on the more general case of 4c for
at leaset 10 years.  See http://www.google.com/search?q=tap+ident+Bernstein

>   5   Spam on mailing lists

In general, if you can stop spam to individual mailboxes, you can stop
it to mailing lists.  I think defenses such as the DCC and Pyzor/Razor are
particularly well suited for mailing lists, since bulk mail sent to
mailing lists is almost always spam.

> 	had a digital signature and cert.

That's mistaken.  Signatures and certs may prove (for various notions
of "prove"; recall the Microsoft cert saga) that you are you, but they
do not necessarily bind you to any particular RFC 2821 mailbox.  Your
cert may authenticate you, but it does not by itself authorize you to
subscribe a mailbox to a mailing list.  Outside Redmond and its
colonies, authentication differs from authorization.

>   7   Unsubscribe that does not work (often due to burster lists)

Those are bugs, often in the very existence of the burster.

> 	[OK its only the IETF that configures MonHarc this stupid way]

That's high on my list, but it's the domain of the IESG not the IRTF.

>  10   Having the messages from a list get trapped in some dozy

yes.

> ...

RFC 2369 is more than merely a proposal.

> efficiently using either symmetric cryptography or public

Cryptography might be a part of implementing that, but it's not
the whole thing.  Neither confidentiality, authentication, nor
non-repudiation are the same as authorization.   "Consent" sounds
more like authorization than the other crypto stuff.

> intervening MTAs) to determine that the message was solicited.

Where's the connection between Alice's key and the email address
she subscribed?  Again, authentication is not the same as authorization.

> It can also process the opt-in callback loop for her.

If you can teach MTA's to recognize mailing lists enough to do that,
then you do not need to send any keys, blobs, or other crypto stuff
to any mailing list.

The crypto stuff that is needed is not from Alice to the mailing list,
but only something that authenticates the mailing list so that Alice's
MTA or MUA can authorize its mail.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Henrik Levkowetz <[email-masked]>
Date: Sun, 9 Mar 2003 03:10:50 +0100

> would help a lot here. There have been proposals, but how do 

These two procmail rules sort 23 of the 37 lists I'm currently
subscribed to automatically, and my mail client picks up the new folders
without me having to create them specifically, today:

	:0
	* ^List-Id: .*<\/[^>@.]*
	mailing-lists/$MATCH/.

	:0
	* ^List-Owner: .*<mailto:owner-\/[^>@.]*
	mailing-lists/$MATCH/.

Mailman out of the box today is set up to add "List-Id:" to all mails passed on. 

Most lists that don't have List-Id: seems to be rather mature lists; 5 of my 14 are
actually ietf.org lists.

I think we can move this particular problem off the urgent list in this forum;
this *is* getting deployment.

	Henrik

-- 
Error message in Haiku form:

  Printer not ready.
  Could be a fatal error.
  Have a pen handy?

    -- Pat Davis

---

From: "Mark Delany" <[email-masked]>
Date: 9 Mar 2003 02:41:20 +0000

> > From: "Hallam-Baker, Phillip" <[email-masked]>

I completely agree with Phillip that mailing lists need to be
fixed. But to me, fixed means being able to distinguish them from spam
and currently there is no reliable way of doing that. As a
consequence, their bulkiness makes them very vulnerable to a lot of
the spam detection strategies and the need for lots of whitelist guff.

I also include in the "mailing list" category, all of the e-marketers
and e-commerce sites that routinely send communications to their
opt-in customers. A simple example might be ebay sending "out bid"
alerts (I'm pretty sure ebay only send such stuff to people who ask
for it, but it still counts as bulky just because so many people ask
for it).

I also agree that it shouldn't be that hard to "fix" mailing lists so
that bulk detection systems can let legitimate mailing list traffic
through.

But, as Vernon says, doing so doesn't need fancy crypto solutions.

It strikes me that mailing lists could be "fixed" by having MUAs
recognize the confirmation process initiated as part of making a
subscription request to the mailing list.

This could be as simple as a new header field that tells the MUA to
generate a unique token - let's call it a confirmation token - as part
of the confirmation reply. Heck, even the Message-ID might do.

If the mailing list s/w records the confirmation token and then uses
it in all subsequent outbound email to the subscriber, than the MUA
can reliably override any bulk detectors/indicators that might be set
by the ISP/MUA/whatever.

It doesn't strike me as especially hard for a mailing list system to
record the confirmation token along with the email address.

It also doesn't strike me as especially hard for an MUA to track the
confirmation tokens it has generated. After all, most of them track
UIDLs for POP, so they probably have the s/w infrastructure to deal
with this sort of keyed data. The confirmation tokens could also be
algorithmic checked of course, obviating the need for a database.

I probably being cavalier about the changes need, but they might be a
reasonable price to pay to forever exclude mailing lists from bulk
detectors and whitelist management hassles.

Confirmation tokens also get the ISP/service provider out of the loop
of guessing whether permission has really been granted for bulky
traffic.

The one down side is that all mailing lists have to send individual
mails to each recipient and we all know what sort of red flag that is
to some bulls.

I'm not trying to propose this as a specific solution, all I'm trying
to demonstrate is that some of our problems could be fixed with
relatively simple solutions, if it weren't for the huge inertia.

Regards.

---

From: Vernon Schryver <[email-masked]>
Date: Sat, 8 Mar 2003 20:22:46 -0700 (MST)

> From: "Mark Delany" <[email-masked]>

> mails to each recipient and we all know what sort of red flag that is

If you are willing to use some trivial crypto, you don't need to suffer
that hassle.  All that the target's MUA or MTA needs to do is recognize
or authenticate traffic from the mailing list.  For that it does not
need unique-per-target tokents.  Simple public key signatures using
a public key conveyed in the initial subscription confirmation would do.  

If you believe as I do that the reason genuine mail forgery (as opposed
to using a legitimately owned Hotmail dropbox) fell off dramatically
a year or three ago is related to the laws criminalizing header forger,
then you don't need any crypto.  Simply have the MTU or MTU choose
and record a suitable RFC 2369 List-whatever header from the confirmation
message.

Consider the common claims about spammers forging headers.  When was
the last time you saw spam supposedly from CERT?  Don't you think that
if spammers were willing to forge headers to get around simple
whitelists, at least some would use that envelope or header From value?

---

From: Jim Youll <[email-masked]>
Date: Sat, 8 Mar 2003 22:31:18 -0500

>whitelists, at least some would use that envelope or header From value?

You mean like this? I have tonnes of these...

 From [email-masked]  Wed Mar  5 14:06:40 2003
Received: from 200.203.76.54 ([200.203.76.54])
         by aleve.media.mit.edu (8.9.3/8.9.3/+ALEVE) with SMTP id OAA18641
         for (...); Wed, 5 Mar 2003 14:05:44 -0500 (EST)
Date: Wed, 5 Mar 2003 14:05:44 -0500 (EST)
From: "Efrain Franckowiak" <[email-masked]>
To: "Contact List" <[email-masked]>
Subject: Men, you have science in your corner

I see many forged headers. The biggest difference between "now" and 
the state of
things, oh, five years ago, is that the MTAs do their own corrections to the
headers within [ ] so we can see where things actually came from. The above
didn't attempt to lie, but I've got plenty that do.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Vernon Schryver <[email-masked]>
Date: Sat, 8 Mar 2003 20:43:45 -0700 (MST)

> From: Jim Youll <[email-masked]>

> Subject: Men, you have science in your corner

Where is cert.org among those headers? 

How do you know that [email-masked] is forged and not legitimately
owned by the spammer?  Are you sure Efrain Franckowiak is not the name
of the spammer?  That 200.203.76.54 is supposedly in Brazil and not
owned by Hotmail does not imply that message has forged headers.

Yes, some spammers have begun or resumed forging sender addresses,
often using addresses from the target list.  However, they're still
not doing blatent forging as they were a few years ago.  

> didn't attempt to lie, but I've got plenty that do.

I've also seen plenty of genuine forged headers.  However, most
mismatches between SMTP client IP address and sender mail address(es)
are not obvious forgeries, unless you use Hotmail's odd, self-serving
definition of "forgery" which coveres using not using your current
hotel room as your return address while travelling.

---

From: "Chris Lewis" <[email-masked]>
Date: Sat, 08 Mar 2003 22:50:44 -0500

Vernon Schryver wrote:

> then you don't need any crypto.

We still get a heck of a lot of spam that claims to be from our own 
servers and/or users.  Thousands of spam blowback (spam forged in our 
domains bouncing back) too.

---

From: Jim Youll <[email-masked]>
Date: Sat, 8 Mar 2003 22:51:37 -0500

>Where is cert.org among those headers?

You picked one domain out of an infinity of them... fine, they don't 
forget cert.org
any more. They forge all the others. Great. I feel good about 
cert.org mail now. But it's
the rest that's making a mess of the incoming traffic.

>owned by Hotmail does not imply that message has forged headers.

right. But your logic is broken.
If 200.203.76.54 were owned by Hotmail, the message would have come from a
Hotmail server. It didn't. I mean, sure, maybe, Hotmail is also known as
  brt-f0-0-0-cslce7003.brasiltelecom.net.br (200.180.192.27)

Forgive me if I have my doubts.

>not doing blatent forging as they were a few years ago.

oh baloney. It's saturday night and I should be out dancing but I have to
answer this message instead... no, there's plenty of forgery.
I'm really not sure what the issue is then, why are we discussing it?

>hotel room as your return address while travelling.

So what did I miss? You say there are plenty of forgeries, I say there
are plenty of forgeries, why are we discussing something about
forgeries not being relevant?
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Vernon Schryver <[email-masked]>
Date: Sat, 8 Mar 2003 20:58:53 -0700 (MST)

> From: "Chris Lewis" <[email-masked]>

> domains bouncing back) too.

Again, for the umpteenth time, of course there is some header forgery,
but is it as much as most people claime?  I don't think so.  My guess
is that perhaps 10% of what most people claim is "forged" spam really
is.  10% of all of the spam in the world implies a "heck of a lot of
blowback and other noise.  Do you have any evidence that your heck of
a lot of forged spam is the majority or even a large fraction of spam?

My jihad in this area is to get people to stop using "forged" to mean
"sender domain differs from SMTP client reverse DNS".  That sloppy
language thinking is like using "douple plus super duper opt-in" for
"(confirmed) opt-in."  People use both sorts of language to mislead
and to try to justify positions that have no honest justification.

---

From: Kee Hinckley <[email-masked]>
Date: Sat, 8 Mar 2003 20:29:12 -0500

>Lets face it the management of mailing lists is a mess. A member of the IESG

This depends on the software.  Mailing list software integrated with 
the MTA can work quite well (CommuniGate is an excellent example, 
although weak for moderated lists.)  But I'll agree, it's certainly 
very easy to run a badly managed mailing list, and the "standards" 
are all over the map.

>behaviour has a purpose and I want to block that purpose].

Requiring mailing lists to include verification of consent is an 
intriguing idea.  Although it suffers from the same problem as 
sending encrypted email--you either need to send an individual copy 
to every person, or include multiple consent verifications with each 
message (which, if you have good list software, is batched up on a 
per-domain basis into groups of 50 to 200 people per message).

However I'm not sure how this lets you ditch "bcc"d spam.  Bcc'ing 
recipients is becoming more and more common as people become more 
aware that cc'ing their friends and relatives with the latest joke is 
a great way to spread people's addresses to spammers and viruses.

>currently suffers from if the cost is low. I believe that the way we will

If the cost is low.  We don't want to pull a Mozilla here and spend 
three years developing a nifty new cross-platform toolkit and bug 
reporting tool when what people wanted was a browser.

>	mailing list and to all prior authors, creating a shadow

Given that most email clients have only two types of reply, neither 
of which map directly to the list model, it's not clear there's a 
right way to fix this, it would require an MUA option as well, but 
there could be a consistent list header to help it.  (Eudora has an 
option to make reply go to sender, reply-all go to list, but it 
doesn't work on all lists, including this one.)

>   2   Out of office responses getting sent to list members

That is 100% an MTA problem, and the finger for 99% of those points 
directly to Microsoft. Delivery problems are supposed to go the 
envelope from.  Out of office messages are delivery problems.  End of 
story.

>   3   I have to tell the mail client to create a folder for each list

What would you have the list do for this?  (Speaking as someone who 
filters some lists into shared folders, some into single folders, and 
splits some based on content into sub folders.)

The rest of the issues are pretty much addressed directly by the 
charter of this list.  (Well, not 4c.)

>option in the tools menu of her client. This generates a

And then we are back to the deployment problem.

But backing up.  If you want to block mailing lists that send to you 
without authorization, then you first need to be able to identify 
mailing lists.  So we need an authentication system for them.  And 
then you need a way to tell bulk mail from spam so you know that 
something is a mailing list when it's pretending not to be.

The one bright side on all of this, and the thing I really think we 
need to leverage, is that legitimate bulk mailers are currently in a 
position where they would be very receptive to spending money and 
time installing an authentication system.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Jim Youll <[email-masked]>
Date: Sat, 8 Mar 2003 23:13:25 -0500

>and to try to justify positions that have no honest justification.

10% would still be a pretty big number.

If you create a mechanism that doesn't account for forgeries, then
forgeries will increase.

I absolutely don't believe that spammers are not forging because
they have some fear of law enforcement over a small matter of
technology... considering that their "products" are often themselves
illegal scams. More likely, mailers that reject unmatched forward/reverse
DNS are responsible for any changes there.

---

From: [email-masked]
Date: Sat, 08 Mar 2003 22:49:39 -0600

> can reliably override any bulk detectors/indicators that might be set

Short of the protections offered by some "fancy crypto solutions", 
what is to keep a spammer from using the public algorithm used in 
generating tokens to generate his own?  Obviously it would result in 
a huge number of failures, but the fact that spammers now use 
dictionary attacks (and worse) suggests that they would be just as 
willing to try that approach.

> detectors and whitelist management hassles.

It doesn't sound that unreasonable to me (then again, I think HTML e-
mail should be eliminated so maybe I'm not the best person to define 
"reasonable"), but I think a crypto approach (preferably one that 
doesn't involve making annual rental payments to some "authority" for 
the use of a very large number; i.e., the list is it's own authority) 
might be more technically solid.

Ideally, what I would like to see is some approach that requires 
large CPU overhead for sending a large number of messages, because 
this should at least throttle the bulk mailers.  There are a couple 
of variations to this approach that I think might be workable, such 
as a mail protocol where the sender first sends a "i'm about to send 
a message" message, and the receiver generates a key pair, sending 
the arbitrary "public" key to the sender, who then uses it to encrypt 
the message, which is then sent to the receiver who can decode it 
with the private key.  One obvious problem with that approach is that 
the receiver does too much work (key generation), but that duty can't 
be given to the sender for obvious reasons (key reuse, etc., as well 
as the fact that if you're going to go to that much trouble you might 
as well get "secure" transmission out of it as well).  Thus, the only 
way it would be workable is to use an algorithm where key generation 
is fast, decryption is fast, but encoding is very CPU intensive, a 
combination that doesn't seem very likely.

---

From: Jim Youll <[email-masked]>
Date: Sun, 9 Mar 2003 00:05:22 -0500

At 20:58 -0700 3/8/03, Vernon Schryver wrote:
>but is it as much as most people claime?  I don't think so

As your message was arriving, so were three copies of this:

From: "STELLA KOFI" <[email-masked]>
Reply-To: [email-masked]
To: ....@........
Date: Sun, 9 Mar 2003 05:29:37 -0800
Subject: REPLY NEEDED

To answer your questions from the last go-round:
	afzhg1296.com does not resolve
		nslookup afzhg1296.com
		*** can't find afzhg1296.com: Non-existent host/domain

	[email-masked] is an unknown user
		rcpt to:<[email-masked]>
		550 5.2.1 <[email-masked]>... UseR UnKnowN

	this passed through MY filters because the previous hop 
(aleve.media) was trusted

	the '1' priority was probably not appropriate (this is a 
Nigerian 419 scam)

So, everything claimed about this message was forged by the sender, 
except the reply-to,
which, Nigerian authorities have made clear, will be received at 
[email-masked] by
someone whose names is certainly not "Stella Kofi"

---

From: Vernon Schryver <[email-masked]>
Date: Sun, 9 Mar 2003 00:39:49 -0700 (MST)

> 10% would still be a pretty big number.

That's what I said.

> If you create a mechanism that doesn't account for forgeries, then

If you mean real forgeries, then there are laws and there have
been convictions.  See http://news.com.com/2100-1023-249758.html 
http://www.spamlaws.com/
and http://archive.salon.com/tech/feature/2000/04/19/state_spam/ 

> technology... 

It certainly seems to me that forgery is involved in a far smaller
fraction of spam today than several years ago.  What's your explanation.

>               considering that their "products" are often themselves

There's an unsubtle difference between obvious, undeniable header forgery
on 30,000,000 copies of spam and selling spamware or herbal v***** that
for all anyone can readily prove might do what the spammer claims.

Obvious, undeniable header forgery would be putting your or CERT's
address on every copy of a 30,000,000 target spew.  That would create
bounce problems large enough for you or CERT to interest the courts,
as it did for Flowers.com.  See
http://www.google.com/search?q=%22flowers.%2Bcom%22+spam

A spammer putting your or CERT's address on a 10, 100, or even 1000
copies won't do more than make you or CERT angry.  It's very unlikely
to cause any real harm, and so the cops will have trouble justifying
spending the time and money to hire someone who can spell "email" or
tell the court that you were terribly damaged.  For better or worse,
courts tend to ignore some laws involving what they consider "victimless
crimes."  To get a flavor of that problem, talk to people with experience
in TCPA actions (junk fax or telephone solicitation).  It's common
for judges to ignore the law when no significant harm was done.
(I'm not talking about how things ought to be, but how they are.)

Flogging "untraceable" spamware or pills to make your "tool" bigger
is also petty enough to escape attention most of the time.

---

From: Vernon Schryver <[email-masked]>
Date: Sun, 9 Mar 2003 00:50:19 -0700 (MST)

> From: [email-masked]

> generating tokens to generate his own?

I don't think digital signatures involve fancy crypto solutions,
at least not if based on a public key published with the subscription
confirmation.  

If you know of a way that a spammer can generate its own authenticating
tokens in such a scheme, you should publish it and become famous.

>                                         Obviously it would result in 

Please read about digital signatures.

Brute force ictionary attacks involving a few 1000 or 1,000,000 names
are one thing.  Brute force attacks on public key signatures that
involve an SMTP transaction per test are something else entirely.
You couldn't begin to start a single attack before the human race has
disappeared.

> this should at least throttle the bulk mailers.

Why penalize legitimate bulk mailers such as the IETF?  
The IETF, CERT, and many other outfits are bulk mailers.

>                                                  There are a couple 
> the message,...

Please read about simple crypo-based authentication.

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 09 Mar 2003 05:34:59 -0500

> a lot of forged spam is the majority or even a large fraction of spam?

  My (admittedly unusual) traffic is 90%+ forged.  In my less moral
moments, I wonder what would happen if I set up the MX to lie, and
force open relays to send mail in an infinite loop.

  A looks up MX, gets B.  B looks up MX, gets C.  C looks up MX, gets
A.  Happiness and frivolity ensue...

  At least until the cops show up at my door, wanting to know why mail
for my domain took down N international links.

  Q: Is this a valid use of my MX?
  Q: Am I responsible for abuse resulting from my valid use of my MX?

> "sender domain differs from SMTP client reverse DNS".

  I understand.  But one of the guidelines I was taught in Physics was
"If you can't tell the difference between two things, then they're the
same."

  As you said in an earlier message, verifying consent for such header
re-writing is difficult.  So for the recipient, it's almost impossible
to tell the difference between consentual uses of such header
rewriting, and non-consentual uses.  Since the recipient can't tell
the difference between the two, he might as well lump them all into
the same "deceitful sender" bin.

  Alan DeKok.

---

From: Vernon Schryver <[email-masked]>
Date: Sun, 9 Mar 2003 09:28:27 -0700 (MST)

> From: "Alan DeKok" <[email-masked]>

> > is that perhaps 10% ...

>   My (admittedly unusual) traffic is 90%+ forged. 

How do you determine that a given message has forged headers?

Again, for the umpteenth time, you cannot compare the reverse DNS
dommain of the STMP client IP address with domains in the envelope or
the headers, with the possible exception of the HELO, to define forgery.
Mail from a Brazilian IP address carrying Hotmail sender is not
necessarily forged in any honest sense.

If you define a mismatch between reverse DNS and HELO domains as
forgery, you'll say that at least 1% and perhaps more than 10% of
all otherwise completely legitimate mail is "forged."

>                                                    In my less moral

That's not how MX RRs work.  Please read pages 3 and 4 of RFC 974.  The
answer to an MX query is a collection of A and CNAME RRs.  The value of
an A RR is an IP address and the value a CNAME RR cannot be an MX RR.

> "If you can't tell the difference between two things, then they're the

Did your physics teachers really say that that two things are the same
if you merely don't go to the trouble to distinguish them?  You can
often distinguish "header forgery" from "sender domain differs from
SMTP client reverse DNS" by serving checking records.  For example,
you could check Hotmail's records to see whether a given spammer ever
owned a Hotmail address.  That making such checks can require expensive
legal paperwork does not make them impossible or justify calling
mismatches between envelope or headers and IP address "forgery."

> the same "deceitful sender" bin.

"Deceitful sender" is not the same as "forgery."
The difference is important because one can carry significant penalites
while the other is merely a reason to reject mail.
See http://news.com.com/2100-1023-249758.html and
http://www.google.com/search?q=%22flowers.%2Bcom%22+spam

---

From: Jim Youll <[email-masked]>
Date: Sun, 9 Mar 2003 11:34:41 -0500

Can you state concisely in two sentences the purpose of this 
particular thread? This appears to be going nowhere and in any case I 
cannot see the relevance to the greater discussion. This may be one 
instance where content filters will be useful.

- jim

---

From: Vernon Schryver <[email-masked]>
Date: Sun, 9 Mar 2003 10:14:28 -0700 (MST)

> cannot see the relevance to the greater discussion. 

The common claims about header forgery are used to support claims
about the practicality of various sorts of evidence of consent.
For example, the claims of very common header forgery have been used
to support the demonstratably false claim that whitelisting cannot be
used to help determine consent to receive mail at the target's MUA or
MTAs.  Talk of header forgery has also been used to support claims of
the necessity of using encryption to prove consent.

Another evil of the false claims of forgery is that they are used to
support the claim that are no significant legal risks to header forgery,
and that implies things about technical spam defenses.

If you admit that sending from one ISP with return addresses pointing
to another is legitimate, then many anti-spam mechanisms are wrong,
others are more difficult, and still other become more interesting.

>                                                     This may be one 

Certainly.

---

From: wayne <[email-masked]>
Date: 09 Mar 2003 11:27:40 -0600

> force open relays to send mail in an infinite loop.

I don't think setting your MX to open relays is such a hot idea, but I
do like the suggestion given previously of setting it to 127.0.0.1.

-wayne

---

From: "Mark Delany" <[email-masked]>
Date: 9 Mar 2003 17:52:53 +0000

> generating tokens to generate his own?

But the public algorithm is merely a variant of rand()

The MUA can generate any sort of token it wants. All it has to do is
send the token back to the mailing list as part of the confirmation
email *and* keep a copy in some local database.

To: [email-masked]
From: [email-masked]
Confirmation-Token: 20700621ba1d4f92de29ffce0dc98b0d

Henceforth the list/e-commerce site includes that token in all email
addressed to [email-masked]

The MUA in turn, is told that the mail is bulky by the presence of,
eg, a "Precedence: bulk" header, thoughtfully placed their by your
bulk-aware inbound MTA.  That header triggers the MUA to check that
the "Confirmation-Token:" matches an entry in its local database. If
there is no match, punt.

To accumulate a spammer-load of victims a spammer has to:

o guess the address-token combinations on a large scale
o break into to MUAs database of a lot of recipients
o intercept traffic on a large scale
o break into a lot of mailing list databases
o acquire them from an unscrupulous mailing list/e-marketer

All of these are hard, excepting perhaps the last one, and even then
they are a one-time target for the spammer as one can readily imagine
an MUA interface that lets the user discard a compromised token.

As I said previously, this is not intended as a proposal, rather it's
meant to demonstrate that relatively modest changes are possible that
reliably disambiguate lists from spam.

But, whether our fragmented and fractious industry can even achieve
modest changes is possibly the first question this RG needs to answer.

Regards.

---

From: [email-masked]
Date: Sun, 09 Mar 2003 12:25:32 -0600

> at least not if based on a public key published with the subscription

In that case, I'm unsure what you meant by not wanting to use "fancy 
crypto solutions" (a quote from your earlier post) if this term does 
not include digital signature algorithms and public keys.  Were you 
just meaning no commercial solutions ("authorities") that involve 
paying money to maintain the key infrastructure?  Or do you not 
understand that "digital signatures" and "public keys" are 
applications of cryptography?

> Please read about digital signatures.

I already know quite a bit about digital signatures; but your 
original post (no "fancy crypto solutions") suggested that you were 
not using them.  Apparently you meant something besides what you 
seemed to mean (not surprising, since it appears that a lot of people 
can't even agree on the meaning of such terms as "spam").  Clearly, 
someone needs to start maintaining a glossary on this topic, since if 
people can't agree to the meanings of terms it is just going to 
create a lot of disagreements where no fundamental differences exist.

> involve an SMTP transaction per test are something else entirely. You
> couldn't begin to start a single attack before the human race has
> disappeared.

Again, if you're talking about public key signatures, you're talking 
about crypto (and if you're not, then you're talking about something 
that can be brute-forced.  You can't have it both ways).

> The IETF, CERT, and many other outfits are bulk mailers.

While the term "legitimate" bulk mailers is often misused (being only 
a tiny fraction of those that claim that title), such legitimate 
users could overcome the handicap via whitelisting by their recipient 
(of course, this brings up the issue of forging, which some ignorant 
parties believes doesn't happen to any extent).
 
> > the arbitrary "public" key to the sender, who then uses it to
> > encrypt the message,...
> 

If you had read the rest of the message, you would've realized that 
those won't work, since they place the calculation burden on the 
wrong party.

---

From: Vernon Schryver <[email-masked]>
Date: Sun, 9 Mar 2003 11:57:18 -0700 (MST)

> applications of cryptography?

I think signatures using nothing more than secret or public key encryption
are not fancy. By "fancy crypto solutions" I mean schemes such as PKI
which involve or should involve
  - networks of key servers or distributed lists of signature authorities,
  - certificate or key revocation machinery (which perhaps ought to be
     related to the previous item, but in real life seems to be utterly
     broken and useless),
  - private investigators checking claimed identities,
  - probably unexpired and possibly submarine patents
  - advertising, propaganda, and other "jockeying in the commercial space."

I suspect I'd find anything that relies on DNSSEC "fancy," particularly
given the history of DNSSEC.  I suspect but don't know that no crypto
mechanism can use DNS RRs without also requiring DNSSEC and so "fancy"
or even a fraud in real life like commercial PKI.

I suspect that some non-fancy crypto spam defenses can be designed and
(more important) deployed without global changes to MTAs or MUAs.
I think that should be a basic criterion for any recommendation by this
group.  However, I'm not yet convinced that any given non-fancy crypto
spam defense would be sufficient or even worthwhile.

> those won't work, since they place the calculation burden on the 

I tried to read the message, but the sense I got from it conflicted
with what I understand of public key encryption and signing.  It
suggested you knew less than nothing of the subject and I was trying
to say as much as politely as possible.  If that inference was wrong,
I appologize.  Unfortunately, rereading your quoted text above several
times does not disperse my original inference.  For example, I'm
troubled by the notion that a stranger could authentiate itself 
merely by preceding the mail message with a session key exchange.

---

From: Kee Hinckley <[email-masked]>
Date: Sun, 9 Mar 2003 14:05:51 -0500

>do like the suggestion given previously of setting it to 127.0.0.1.

I had to play the 127.0.0.1 game once with a domain.  My DNS server 
was getting slammed because someone sending spam claiming to be from 
ftp.somewhere.com, which didn't exist.  I wasn't getting the bounces. 
What I was getting were hundreds of DNS lookup requests every second 
from all the people trying to bounce the email.  Since the domain 
didn't exist nobody was cacheing--which meant that I got *all* of the 
DNS requests.  127.0.0.1 for ftp.somewhere.com solved that very 
nicely.  But it did make an interesting DoS attack.

Incidentally.  I've seen stats that indicate that a very large 
percentage of the lookups going to the root servers are for domains 
that don't exist.  It wouldn't surprise me if this is why.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: Kee Hinckley <[email-masked]>
Date: Sun, 9 Mar 2003 14:01:55 -0500

>and record a suitable RFC 2369 List-whatever header from the confirmation

I don't see that forgery itself has fallen off.  We detect most of 
our spam using forgery detection.  However it does seem that forgery 
of random commercial domains has fallen off.  I think they tend to 
use ISP addresses more, but I haven't done a really good analysis.  A 
quick scan of my 20,000 or so spams shows 4000 unique domains in the 
from lines.  The top ones are:

+---------------------------------+------+
| domain                          | cnt  |
+---------------------------------+------+
| yahoo.com                       | 1740 |
| hotmail.com                     | 1496 |
| aol.com                         |  973 |
| msn.com                         |  448 |
| Mail.com                        |  214 |
| recessionspecials.com           |  209 |
| lycos.com                       |  180 |
|                                 |  178 |
| excite.com                      |  170 |
| free-gift-offers.com            |  168 |
| juno.com                        |  148 |
| bluelightoffers.COM             |  145 |
| email.com                       |  125 |
| clickformail.com                |  122 |
| helpfuloffers.com               |  121 |
| 163.com                         |  118 |
| GreatDealsDepot.net             |  117 |
| earthlink.net                   |  108 |

That's all either ISPs or known spam domains.

On a personal note, I know that spam using @somewhere.com used to be 
a major problem (weekly blowbacks), but has now dropped off to almost 
nothing.  On the other hand, that may be because people started 
blocking the domain.

Then too.  There were a couple successful lawsuits against spammers 
by companies whose reputation had been damaged.  They may have 
decided to keep the forgeries to domains where that argument wouldn't 
result in huge monetary claims.

>not doing blatent forging as they were a few years ago.

The forging is there.  It's more subtle.  At least 90% of the spam we 
are filtering has something forged in the header, and I don't mean 
 From doesn't match sending IP--we don't count that--that's legit.  I 
don't really want to go into the details on-list.  But it is there.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: Hadmut Danisch <[email-masked]>
Date: Sun, 9 Mar 2003 20:50:18 +0100

> "sender domain differs from SMTP client reverse DNS". 

That's not "forged". That's "normal". Nobody calls such
a case forgery.

Maybe your jihad was a bad choice. Wouldn't it be better to 
choose a target that really exists for jihad?

Hadmut

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 09 Mar 2003 10:01:24 -0500

> do like the suggestion given previously of setting it to 127.0.0.1.

  There's at least one MTA which will happily "deliver" mail to itself
100's of times recursively in such a situation.  It will then
"discover" that the message has transited more than 20 hops, and
bounce the message to itself.

  Yes, it doesn't notice that it's delivering mail to itself.

  Yes, it takes 100's of hops before the MTA notices that the hop
count has exceeded 20.

  Yes, people use this garbage software to process mail.

  Alan DeKok.

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 09 Mar 2003 10:21:53 -0500

> How do you determine that a given message has forged headers?

  I admit it's difficult.

> necessarily forged in any honest sense.

  No, but I'm getting spam from 1000's of unique IP's across the
planet.  Each IP is sending spam with many different 'from' lines,
some of which are for non-existent domains.  Others are for well-known
domains.  The various 'from' lines overlap among the source IP's.

  The odds of *all* of these messages being consentual are next to
zero.  The odds of 90% of them being intentionally forged are pretty
good.

> all otherwise completely legitimate mail is "forged."

  Sure.  But the odds are that spam is 90-99% forged in that way.

  It's not a perfect descriminator, but it's better than chance.

> That's not how MX RRs work.  Please read pages 3 and 4 of RFC 974.

  Big deal.  Did I say I would use MX RR's?

  Let me re-phrase my explanation to make it a little simpler: I hack
my DNS server so that queries from one open relay get told that my MX
is another open relay.  The second open relay gets told my MX is the
first.  They bounce garbage back and forth until they decide they're
tired of it.

  Any protocol can be abused if you lie.  DNS & MX records are no
exception.

> if you merely don't go to the trouble to distinguish them?

  If the trouble is too expensive to be practical, yes.

  We can't tell the difference between many theories in high energy
particle physics, because we don't have the funds to build a
galaxy-sized particle accelerator.

  Similarly, if a header is "deceitful", and 90% of such traffic is
spam, and it's too expensive to verify the other 10%, the most
practical thing to do is often to lump it all in the same bin.

  Why should I pay for verifying that other peoples traffic is not
spam, when they've intentionally engaged in many of the same
behaviours as the spammers?

  Alan DeKok.

---

From: Justin Mason <[email-masked]>
Date: Sun, 9 Mar 2003 22:24:39 +0000

> whitelists, at least some would use that envelope or header From value?

FYI, spammers often forge some addresses that are seen as having
"filter evasion" value...  for example, Amazon.com is frequently
impersonated in the headers, since that's in the default SA whitelist.
I haven't seen CERT forgeries yet though.

--j.

---

From: Hadmut Danisch <[email-masked]>
Date: Sun, 9 Mar 2003 23:56:36 +0100

> whitelists, at least some would use that envelope or header From value?

I have tracked back around 150-200 spam mails and not in a single
case the header was not forged. This means that I received trustworthy
statements from the domain owner, IP owner or ISP that the given 
sender address never had existed or - in a few cases - that they
existed but the user was evidenced innocent. This is not an assumption 
or conclusion, it is a verified statement of the authorized owners. 

There are very few cases where the spam is just a bait to verify
the recipients address, asking the recipients to "opt out", and thus
verify their address and mark them as a "spam reader".

From my experience >99% of the spam contains forged sender addresses
(and wrong host names in the SMTP helo command, but that doesn't
really matter).

I definitely can't understand what makes you believe that spam 
messages do usually not contain forged headers. That's one of the 
most obvious and easiest to verify facts about spam.

Hadmut

---

From: Erez Zadok <[email-masked]>
Date: Sun, 9 Mar 2003 18:33:26 -0500

In message <[email-masked]>, Hadmut Danisch writes:
[...]
> (and wrong host names in the SMTP helo command, but that doesn't

My own experience is also that the vast majority of spam comes from forged
addresses that don't exist.  I personally am using ASK, a challenge-response
personal spam filter to be very effective in reducing the rate at which spam
sneaks in significantly.  ASK works well for me, but it does have
scalability problems if deployed widely.

Erez.

---

From: [email-masked]
Date: Sun, 09 Mar 2003 18:23:12 -0600

> I think signatures using nothing more than secret or public key
> encryption are not fancy. By "fancy crypto solutions" I mean schemes
> such as PKI which involve or should involve
>   - networks of key servers or distributed lists of signature
>   authorities, - certificate or key revocation machinery (which
>   perhaps ought to be
>      related to the previous item, but in real life seems to be
>      utterly broken and useless),

Ok; I think I understand what you mean by "fancy", and with that 
definition I agree with your position.

> > > > of variations to this approach that I think might be workable,
> > > > such as a mail protocol where the sender first sends a "i'm
> > > > about to send a message" message, and the receiver generates a
> > > > key pair, sending the arbitrary "public" key to the sender, who
> > > > then uses it to encrypt the message,...
> > > 
> with what I understand of public key encryption and signing.

That's because it didn't have anything to do with encryption / 
signing.  See below.

>  It
> I appologize.

Apology accepted.

>  Unfortunately, rereading your quoted text above several
> troubled by the notion that a stranger could authentiate itself merely
> by preceding the mail message with a session key exchange.

Ah; that's because I suppose I wasn't as clear as I thought I was 
about the purpose of what was proposed.  It has nothing to do with 
authentication, but instead is merely a mechanism to make the task of 
sending unsolicited messages very resource (specifically CPU) 
intensive, such that the current spammer strategy of sending millions 
or billions of messages is simply not technically viable due to the 
tremendous amount of resources that would be consumed.  Right now, 
the spammers are essentially parasites, in that the resources they 
are consuming mostly belong to someone else.  The purpose of the 
proposal was to provide a "minimum cost" to the act of sending the 
email.  In principle (but not quite in practice, for the obvious 
reasons), it could be as simply as passing the sender a product of  
primes, etc. and demanding that he factor them before accepting the 
message.  As you (or someone else, sorry; I've lost track) pointed 
out, this is particularly unfair to legitimate bulk mailers, but I 
think that problem could be solved by a whitelist approach (which 
itself implies the need of some authentication and authorization 
system, which is beyond the scope of what I was proposing).
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Vernon Schryver <[email-masked]>
Date: Sun, 9 Mar 2003 17:33:48 -0700 (MST)

> From: Erez Zadok <[email-masked]>

> scalability problems if deployed widely.

How do you know that the source addresses don't exist, because your
challenge-response system gets a bounce for its challenges?  That
would at most show that they don't exist when you challenge them, and
might in some cases only show that your challenges are unacceptable
(e.g. detected by a spam filter) and so rejected from perfectly valid
and legitimate addresses.  (For example, if your challenges are
substantially identical or bulk, then they are likely to be rejected
by spam filters like the DCC and Pyzor/Razor.)

Assuming that your vast majority of addresses in fact do not exist,
how do you know that none of the spammers owned the addresses they
are using in the reasonably recent past, perhaps even when they queued
the spam on the on the open relay or other SMTP client?  For example,
how do you distinguish "never existed" from "terminated for spamming"?
Do you consider both cases to be "forged?"  I hope not.

That distinction is significant, because contributors to this mailing
list have claimed that some forgeable spam defenses are impossible,
because they believe forgery is extremely common.  Others have claimed
that spam defenses based on validating sources would be sufficent
using similar reasoning.  Both are reasons are bogus if forged spam
is in fact relatively rare, although the claims might still be valid
for other reasons.

It's also worth quantifying the total from which that or any "vast
majority" is drawn.  For example, conclusions based on 5000 messages
sent toward a handful addresses should convince no one of much of
anything.

As others have said, numbers with complete disclosure of what
they mean would be good.

As I keep saying, I agree that plenty of spam has forged sources.
The question is whether that "plenty" is 1%, 10%, 50%, or 99%.  As I
said, my guess is ~10%.  Note that I used the word "guess."  I wish
that everyone else who is guessing would admit as much.  As far as I
can tell, everyone is guessing.

---

From: Vernon Schryver <[email-masked]>
Date: Sun, 9 Mar 2003 17:33:36 -0700 (MST)

> intensive, ...

That goal and mechanism needs no changes in protocols or MTAs.  You
need only configure your MTA to refuse all mail except when the SMTP
client uses SMTP-TLS.  That's easily done with modern sendmail.   In
many and probably cases your MTA won't have a cert for the SMTP client,
and so the two MTAs will exchange session keys and then do the crypto
CPU grinding required for confidentiality.  If you do have a valid
cert for the client, you'll also be able to authenticate the client.

This might work your CPU as hard as the spammer's, but that symmetry
or worse is unavoidable.  The speeds of CPUs running MTAs on the
Internet today vary by more than a factor of 200, and probably more
than a factor of 10,000 if you include systems with hardware help for
encryption.  It is impossible to burden the computers of spammers
without overtaxing some SMTP servers and some legitimate SMTP clients.

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 09 Mar 2003 14:48:56 -0500

Erez Zadok <[email-masked]> wrote:
> addresses that don't exist.

  Like the 100,000 fake addresses at my domain.  The annoying
"notifications" to "abuse@" about spam from my domain went away only
when I gave up, and firewalled port 25 for everyone but a tiny
whitelist.

  Before that, I had weekly fights with anti-spam people, who didn't
want to understand that most headers were forged.  Their experience
may have been different than mine, but it was a source of great
frustration that they refused to believe my experience.

  It's not just a few domains, either.  Look at the reports on this
list of people re-activating domains after 3 years, and immediately
receiving 10K or 100K spams.  There is *no* way that any legitimate
business would be sending that much mail to addresses which don't
exist, and *never* clean out bounces.

  Alan DeKok.

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 09 Mar 2003 15:01:09 -0500

Vernon Schryver <[email-masked]> wrote:
> For example, how do you distinguish "never existed" from "terminated
> for spamming"?  Do you consider both cases to be "forged?"  I hope
> not.

  To get back to the charter, this is an issue of consent.  Leaving
aside the question of how much of what kind of spam exists, having a
consent trail which can be validated helps enormously.

  For that reason, I'm not as worried about spam from throw-away
accounts, as I am about spam from open relays, or willing accomplices
hosting the spamming MTA.  I believe that the two kinds of UBE are
fundamentally different, and that the methods used to deal with them
should also be different.

> sent toward a handful addresses should convince no one of much of

  At the minimum, they're a known outlier, and a line drawn in the
sand.  They serve to give insight into the statistical distributions
in a way that the average cases cannot.

  Alan DeKok.

---

From: Erez Zadok <[email-masked]>
Date: Sun, 9 Mar 2003 20:09:58 -0500

Vernon, you have a number of good points here.  I'm CC'ing Marco Paganini,
ASK creator, who is currently running a user study for his upcoming FREENIX
paper on ASK.  Hopefully, he'll be able to respond much better than I could,
and perhaps quantify some of his experiences for this list.

Disclaimer: I'm shepherding Marco's paper for FREENIX, so I may be just a
tad biased. :-)

> by spam filters like the DCC and Pyzor/Razor.)

The challenges are not too similar.  They use a checksum to correlate their
original message w/ the response to the challenge.  A bounce is not used to
assume spam; a _lack_ of response to a (signed) challenge (after a period
of time), is considered instead.  The assumption here is that there are very
few cases where there are "real users" behind spam sources, who'd sift
through challenges and bother responding to them.  (Obviously there's more
details here, but I don't want to turn this email into a detailed
description of ASK.)

> Do you consider both cases to be "forged?"  I hope not.

From my personal perspective as a user, why should I care?  If the spammer's
account never existed or was turned off due to spamming, in both cases a
valid response to my challenge isn't likely to be returned, and so I won't
get to see that original spam message.

However, I do think that these challenge-response systems can overburden an
internet-wide SMTP system with all of these challenges.  That's why I said
in my original email that I think these systems suffer from scalability
problems.  I hope this list can come up with good and scalable solutions.

> is in fact relatively rare, although the claims might still be valid

Right.  My personal experience is that most (>99%) spam is from forged
addresses.  Marco is conducting a wider study (than just oh' lonely me :-)
and hopefully he can provide more accurate results.

> can tell, everyone is guessing.

Agreed.

I've read some articles in popular media that claim that over 50% of all
email is spam.  While I believe, from personal experience, that >99% of spam
is forged, I don't believe (again from personal experience), that 50% of all
email is spam.  All these anecdotal or personal results are not enough.

Perhaps what this list should do is prepare a Web based survey that
interested parties can complete.  That is exactly what I suggested that
Marco do before his paper is published, and he took on that daunting task of
a user survey.  I think a carefully designed study with as many possible
users filling it, would go a long way toward defining the scope of the spam
problem nowadays.

Cheers,
Erez.

---

From: Vernon Schryver <[email-masked]>
Date: Sun, 9 Mar 2003 18:22:33 -0700 (MST)

> original message w/ the response to the challenge.

That sounds "substantially identical" as far as the DCC and presumably
Pyzor/Razor see it.  Contrast your checksum with the "hashbusters"
common in spam.

> email is spam.  All these anecdotal or personal results are not enough.

As I think I've said here before, numbers from some of the servers in
the global network of DCC servers say that more than 50% of all email
is spam.  See http://www.dcc-servers.net/dcc/graphs/?resol=1week and
especially http://www.dcc-servers.net/dcc/graphs/comp-rates?resol=1week
Notice that those numbers are based on 10 Mmsgs/day.  The spam ratio
varies from <10% during the week for the surveyed government sites to
about 80% on weekends for some U.S. ISPs.

See also the charts on http://www.postini.com/

---

From: [email-masked]
Date: Sun, 09 Mar 2003 20:25:13 -0500

> exist, and *never* clean out bounces.

When I decomissioned our bitnet-internet gateway, the Saudi Arabian bitnet
nodes had already been off bitnet for close to 18 months, so ALL mail to those
nodes got bounced.  Some 90% of the traffic still using the gateway at the
time was a mailing list sending to 5 Saudi subscribers.  The mailing list
had been getting back well over 200 bounce messages a *DAY* for a year and
a half and not cleaned up.  And yes, I tried the list owner and postmaster
addresses as well.  I didn't even get any joy mailing to the WHOIS contacts
for the domain.

I won't identify the offending site other than to say that they probably
should have known better, as one of their research groups are mentioned
in some RFC's with single digits. ;)

---

From: Kee Hinckley <[email-masked]>
Date: Mon, 10 Mar 2003 01:30:10 -0500

>is spam.  See http://www.dcc-servers.net/dcc/graphs/?resol=1week and

We need to keep in mind that DCC and Postini are using self-selected 
samples.  If you don't have a spam problem you aren't likely to be 
using them.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: Kee Hinckley <[email-masked]>
Date: Mon, 10 Mar 2003 01:11:37 -0500

At 2:48 PM -0500 3/9/03, Alan DeKok wrote:
>There is *no* way that any legitimate

That much, no.  But certainly some don't bother cleaning things out. 
Here are a sampling of the addresses that were still sending to a 
former dotcom two years after the domain had been turned off.

Return-Path: <[email-masked]>
From: Schwab Alerts <[email-masked]>
Return-Path: 
<afterthebell-html-return-423-emallen=[email-masked]>
From: Adobe Systems Incorporated <[email-masked]>

---

From: Kee Hinckley <[email-masked]>
Date: Mon, 10 Mar 2003 01:37:02 -0500

>can tell, everyone is guessing.

We filter spam based solely on forged headers and private blacklists 
plus user preferences.  The number of those messages that have 
something faked is well over 50%.  We aren't currently logging all 
the triggers on every spam message, so I can't give you exact 
numbers.  But it certainly was the case last fall.  Remember though 
that there are lots of headers can you can badly fake in a 
message--not just the routing ones.  We're looking at the entire set 
of headers.

As another data point.  16% of the 100 million messages somewhere.com 
bounced last year were bounced because they had return-path's that 
didn't pass basic validation (e.g. no MX, invalid domain...).  That's 
before anything even *got* to the spam filters.

Now I will grant that it's possible that the guy who sent email from 
Russia through a server in Italy in order to spam me in the United 
States actually *did* have a Hotmail account at the time the first 
message was sent.  I don't know why he would have bothered, but it's 
possible.

Can you explain why you think that person would have set up an 
account just so that it could get shut down immediately?  And why you 
think this matters?
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: [email-masked]
Date: Mon, 10 Mar 2003 02:42:49 -0500

On Mon, 10 Mar 2003 01:37:02 EST, Kee Hinckley said:
> something faked is well over 50%. 

OK.. it's been 2 decades since I took statisics, and I've *mostly*
recovered from being a math/physics major, but...

All this says is that if you didn't filter on blacklists and user prefs,
that 100% of the mail that you tagged as having forged headers have forged
headers.  Addition of the other 2 criteria water it down to "over 50%".

Problem 1)  There's no indication of what percent of mail with forged
headers was actually spam - and without your definition of what counts
as "forged", it's hard to tell.  I know that *this* laptop has sent some
rather squirrelly headers when I'm travelling and borrowing some net access.

Problem 2) There's no indication of what percent of your total mail volume
got tagged as "forged headers".  If it's 1% of your total or 40% of your
total is important, as they have different implications for what solutions
have to be able to do...

Problem 3) There's no indication of what sorts of overlaps there are between
the forged, blacklist, and preferences.  If most of the mail that's forged
is *also* being hit by blacklist and/or preferences, then maybe checking for
forged headers isn't productive.  If it's catching lots of spam that your
blacklists and preferences aren't catching, maybe that's indicative of problems
with those two schemes...

(And no Kee, I'm not picking on you in particular - I didn't accept Vernon's
DCC numbers at face value either.  Nobody here uses the same methodology,
and it's hard to avoid comparing apples to oranges without asking what
sort of seeds were planted in both cases....

/Valdis

---

From: Hadmut Danisch <[email-masked]>
Date: Mon, 10 Mar 2003 12:01:29 +0100

On Sun, Mar 09, 2003 at 08:09:58PM -0500, Erez Zadok wrote:
> email is spam.

That completely depends on the use of email.

I find 5-15 spam mails in my mailbox every day, which is significantly
less than 10% of my email traffic (even without the  asrg list). 

On the other hand, I know some people who rarely use e-mail, who
don't subscribe to mailing lists, don't use e-mail for professional
just for private purposes. Their main use of the internet is
webbrowsing. Since they are not experienced and not security aware, 
they enter their e-mail address just on every web form that asks
to do so. They receive tons of spam and very few
non-spam. Significantly more than 90% of their incoming mail is spam.
Some of them even have capitulated and don't open their mailbox
anymore.

Another question is: How do you count the fraction of spam? Number
of mails or content size? 

The spam messages I do receive are getting bigger and bigger. Lot's of 
spam messages come with large pictures, powerpoint presentations,
PDF or Word documents. For someone who usually receives just a few simple
text messages from friends, a single spam message of that sort 
could be 98% of the monthly mail traffic.

Hadmut

---

From: Kee Hinckley <[email-masked]>
Date: Mon, 10 Mar 2003 10:58:27 -0500

>headers.  Addition of the other 2 criteria water it down to "over 50%".

I was trying not to go into the grody details.  Our false negative 
rate is <1%. Everytime a user adds someone to a blacklist, or reports 
a problem, that gets recorded as spam we missed.  I'm counting that 
in the numbers.

>rather squirrelly headers when I'm travelling and borrowing some net access.

We aren't deleting any of the mail.  The users scan the list and mark 
the false positives.  My numbers were based on the results *after* 
user confirmation.  As I said elsewhere, users do get sloppy, but not 
to an extent that's going to significantly change these numbers.

>total is important, as they have different implications for what solutions

We are currently running 26% spam across all users.

>with those two schemes...

Right.  As I said, we used to log that data, but we simply didn't 
have time to go over all of it.  Once things settle down a bit we'll 
be doing that sort of analysis to tune our filters.

>sort of seeds were planted in both cases....

No offence taken.  Talking about spam with people is very much like 
talking about user interfaces.  Everyone thinks they are an expert, 
because everyone deals with it every day.  In fact it's much more 
like the blind men and the elephant.  I don't claim to be immune to 
any of that.  I have two data samples.  One is somewhere.com, which 
is clearly an anomalous situation.  Like striker, it's a harbinger of 
things to come, but its sample information is wildly biased by 
mailbombs, forged mailing lists subscriptions and viruses.  My other 
sample is the PureMessaging spam filtering service (soon to be 
Messagefire... don't ask).  It's closer to an end-user analysis, but 
the sample size is still small and we simply don't have time to turn 
it into a research project right now.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: "Bruce Brown" <[email-masked]>
Date: Mon, 10 Mar 2003 10:20:30 -0800

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Vernon Schryver
> Sent: Monday, March 10, 2003 10:05 AM
> To: [email-masked]
> Subject: Re: [Asrg] Lets Fix Mailing Lists
> 
> 
> Other spammers know that their target lists tend to fill with garbage
> as addresses are abandoned and as target address harvesting mechanisms
> get confused (e.g. by message-IDs).  They know that some major ISPs
> by rumor including AOL automatically blacklist STMP clients by IP
> address after too many bad target addresses.  This provides a strong
> incentive for spammers to clean their target lists.  To do that, they
> need drop-boxes get delivery failure notifications.
> 
This seems to argue that there should be no feedback to spammer to
allow them easy means to clean their lists, since bad RcptTo may
trigger spammer detection.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Vernon Schryver <[email-masked]>
Date: Mon, 10 Mar 2003 11:05:03 -0700 (MST)

> From: Kee Hinckley <[email-masked]>

> account just so that it could get shut down immediately?  And why you 

I've repeatedly pointed out reasons why that is not just possible but
quite likely.  Contrary to spammer fighter religious dogma, many
(but certainly not all) spammers care very much about receiving
"bounces" and "removes."  Many spammers honestly think that letting
their targets opt-out is the right thing to do.  That's why you often
seen pleas in spam that you not ask ISPs to cancel "remove" drop-boxes.

Why do you think Yahoo or Hotmail (I've forgotten which) has deployed
mechanisms to prevent robots from creating free drop boxes?  Who do
you think might needs to create so many free drop boxes that they'd
use software?

---

From: Vernon Schryver <[email-masked]>
Date: Mon, 10 Mar 2003 11:29:02 -0700 (MST)

> From: "Bruce Brown" <[email-masked]>

> trigger spammer detection.

Those of us (and there are many) who use "traps" try to keep them
popular among spammers.  We never report spam received by our traps,
because we cannot know whether the address of the trap is hidden in
a Message-ID or Received queue-ID, hidden in the pattern of blanks,
capitalization, and punctuation in the body, or could be guessed by
checking IP addresses and dates in sender logs.  

On the other hand, it is even more important that every false positive
generate an SMTP error response or a "bounce."  Every spam defense
used on mailboxes intended to receive mail from strangers has some
false positives, if you define "false positive" broadly.

---

From: Hadmut Danisch <[email-masked]>
Date: Mon, 10 Mar 2003 19:38:16 +0100

On Mon, Mar 10, 2003 at 11:05:03AM -0700, Vernon Schryver wrote:

> Contrary to spammer fighter religious dogma, many
> their targets opt-out is the right thing to do.

Hooray! Yippee! We've got it. We have the final and ultimate solution. 
We can close the group, no need to develop DNS extensions, content
filters, and other witchcraft. All we need to do is just
send a "remove" reply to opt out. Why didn't you say this earlier, 
you could have saved me a whole week of following that group.

Hadmut

---

From: Justin Mason <[email-masked]>
Date: Mon, 10 Mar 2003 19:41:47 +0000

> their targets opt-out is the right thing to do.

In that case, things have changed a lot since April 2002. 

	http://www.ftc.gov/opa/2002/04/spam.htm:
	In another initiative, Netforce partners tested whether "remove
	me" or "unsubscribe" options in spam were being honored. From
	e-mail forwarded to the FTC's database, the agencies culled
	more than 200 e-mails that purported to allow recipients to
	remove their name from a spam list. The agencies set up dummy
	e-mail accounts to test the pledges, but discovered that the
	vast majority of addresses to which they sent the requests
	were invalid. Most of the "remove me" requests did not get
	through. Based on information gathered by the Netforce, the FTC
	has sent more than 75 letters warning spammers that deceptive
	"removal" claims in unsolicited e-mail are illegal.

--j.

---

From: Vernon Schryver <[email-masked]>
Date: Mon, 10 Mar 2003 13:17:51 -0700 (MST)

> From: Justin Mason <[email-masked]>

> 	"removal" claims in unsolicited e-mail are illegal.

Those who bother to read mine the FTC's words will see there is little
or no contradiction.  Furthermore, consider the implications on spammer
behavior of news of those 75 letters warning that bogus removal
addresses are illegal.

If you think that the FTC is smart enough to distinguish dropboxes
that are invalid because they've been terminated by Hotmail etc. for
being advertised in spam from dropboxes that were never valid, then
I have a deal on the profits from a bridge in Nigeria that will interest
you.  Or maybe the FTC is that smart but their PR flaks have odd uses
of English.  Do you think 75 out of 200 is "the vast majority?" 
If not, why didn't the FTC send letters to the vast majority?  Could it
be that most of the other 125 remove addresses were typical free
provider drop-boxes that had been terminated by the free providers?

---

From: Kee Hinckley <[email-masked]>
Date: Mon, 10 Mar 2003 17:12:52 -0500

>being advertised in spam from dropboxes that were never valid, then

Look.  Could we stop arguing about something that actually can be 
determined quite easily?

Could someone please contact Yahoo and Hotmail and ask them two very 
simple questions.

"Of the local email addresses at which you have seen excessively 
heavy bounces, what percentage used to exist and what percentage 
never existed?  And of those that used to exist, what percentage were 
closed do to spamming?"

Someone here must have a contact there?
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: Kee Hinckley <[email-masked]>
Date: Mon, 10 Mar 2003 17:06:59 -0500

>seen pleas in spam that you not ask ISPs to cancel "remove" drop-boxes.

Alternatively they are lying through their teeth.

What I do know is that my bounce rate is doubling every year.  My 
correspondence with some ISPs indicates that theirs is also 
drastically increasing.  I can get numbers from one if you like.

So, we know the number of messages sent to never-existent and 
non-existent accounts is increasing.  This could of course, just be 
an indication that the number of spammers is increasing.  (Note, 
that's number of spammers, not amount of spam.  If you are correct, 
then old spammers would stop targeting addresses after bounces.)

As I've mentioned before, somewhere.com is a weird case.  But if I 
look at the top 30% of my bouncers by IP address, they seem pretty 
stable.  That is the amount of bounces they are sending doesn't 
change much over time.  But that could be different spammers using 
the same IPs (and some aren't spammers, just idiots--if anyone has a 
contact at dvr1.suomen4g.fi please let me know--half a million 
bounces is ridiculous).

>need drop-boxes get delivery failure notifications.

So long as their is a plentiful list of open relays, open proxies, 
and spam friendly relays, why bother with the hassle of setting up a 
drop box that will be closed in just a few hours?  The number of 
bounces you'll get before it's closed is a tiny portion of the total.

On top of it.  We've heard from many people (and again, I've gotten 
this direct from ISPs) that blocking by IP address based on bounces 
does not have a significant impact on spam delivery--there are too 
many senders.

Finally.  I know for a fact that many major commercial mailers 
consider cleaning their lists too much of a hassle.  If they don't 
bother, why would the spammers?

>you think might needs to create so many free drop boxes that they'd

Because you aren't completely wrong.  BCC spam is definitely on the 
rise, especially with the Spanish Prisoner scams (give the Nigerian's 
a break, that's what it originally was).  In fact any significantly 
illegal scam needs an electronic drop box.  But they also need a much 
smaller response rate, so they can afford to get just a few responses 
before they are shut down.

But the fact that this is on the rise does not necessarily mean that 
the percentage of forged from addresses (and I mean "forged" as in 
never existed, or belonged to someone else) is significantly dropping.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: Vernon Schryver <[email-masked]>
Date: Mon, 10 Mar 2003 15:26:30 -0700 (MST)

> Someone here must have a contact there?

Many times over the last several years people known to be working there
have been asked in public for that information.  The response has been so
deafeningly silent that I assume that it reflects corporate policy.

I theorize is that if the public knew that a significant number of
what Hotmail calls "forged" addresses were not in any honest sense
forged, then even more people would refuse all mail from the free
providers and fewer people would sell their "eyeballs" for "free"
mailboxes.  This theory works whether you think (as I do) that most
envelope or header From values pointing to free providers are valid
at the start of a spew or if you think only 3% are valid.  In either
case there is so much spam with perfectly legitimate free provider
return addresses that anyone with any sense avoids the free providers
as sender or recipient.

---

From: Hadmut Danisch <[email-masked]>
Date: Tue, 11 Mar 2003 00:06:05 +0100

On Mon, Mar 10, 2003 at 03:26:30PM -0700, Vernon Schryver wrote: 
> there is so much spam with perfectly legitimate free provider
> return addresses 

Isn't that a contradiction in terms?

Free providers do not enjoy wasting money. They do provide
services for private persons, not spammers.

Even if the given address was real and the spammer would even
read all replies coming in, it would still be far from
"perfectly legitimate". That's "abuse".

Hadmut

---

From: "Chris Lewis" <[email-masked]>
Date: Tue, 11 Mar 2003 14:47:01 -0500

Kee Hinckley wrote:

> non-existent accounts is increasing.  This could of course, just be an 
> indication that the number of spammers is increasing.  (Note, that's 
> number of spammers, not amount of spam.  If you are correct, then old 
> spammers would stop targeting addresses after bounces.)

Some numbers: when we shut down some of our legacy domain names (removed 
them from DNS), they were getting 50K spams/day, peaking at 140K.

Two years later we resurrected the domains to see what was going on - 
500-800K/day continuous.

If spammers were cleaning lists, the number would have been _well_ below 
50K/day.  Instead, it was way above.

Spammers do not prune their lists. Neither do some mailing lists houses 
who should know better.