Subject: [Asrg] Re: Spam send/receive ratio

From: Jim Witte <[email-masked]>
Date: Wed, 19 May 2004 15:44:06 -0500

> One of the distinguishing characteristics of a spammer is the
> imbalance between mail sent and mail received.  Unfortunately I do not
> see a convenient way to penalize people who fall into this category.

   One idea about this (although I don't know how scalable it is) would 
be, if you could authenticate the sender ie DomainKeys or the like, 
then every sender would have a send/receive ratio (somewhere), which 
would be entered into the mail header in some way that it could not be 
spoofed (more protocol changes though).  Then if a mail is received by 
the end-user's inbox, it could check the ratio, and decide whether to 
accept/reject/quarantine it.

   But I sill don't see a way to penalize senders who are rejected too 
often.  Although perhaps it could be integrated into the 'delay' in 
creating the email-stamp/introduction-stamp (the more higher your 
send/receive ratio goes, the more it "costs" to send new stuff)  
Another possibility is of course law enforecement, but this would seem 
to require some centralization of the ratio information, and whose 
rejecting what.

   The send/receive ratio might also be a problem for mailing lists - 
maybe, I'd have to see statistics.

Jim

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Philip Miller <[email-masked]>
Date: Wed, 19 May 2004 19:20:33 -0400

Jim,
>> see a convenient way to penalize people who fall into this category.

I don't agree with that characteristic as generally defining, but with some 
large exceptions, it could be. I'll accept it for purposes of discussion.

> be, if you could authenticate the sender ie DomainKeys or the like, then 
> every sender would have a send/receive ratio (somewhere), which would be 
> entered into the mail header in some way that it could not be spoofed 
> (more protocol changes though).  Then if a mail is received by the 
> end-user's inbox, it could check the ratio, and decide whether to 

Not too many protocol changes, if the entire check were out-of-band. The 
only problem with it is that there either needs to be some trusted 3rd-party 
that would keep the counts (hah!) or some distributed system that could 
produce accurate, Internet-wide, statistics on demand (and double hah!). It 
would be nice and elegant, it's just a complete impossibility.

> send/receive ratio goes, the more it "costs" to send new stuff)  Another 
> possibility is of course law enforecement, but this would seem to 
> require some centralization of the ratio information, and whose 
> rejecting what.

You would either increase their costs/delays to send, or you would reject 
outright.

> maybe, I'd have to see statistics.

Mailing lists, in a hybrid system, could be easily white-listed by any 
number of means. The registration messages should, in theory, have a perfect 
1:1 send:receive ratio. Of course, if someone were trying to make 
subscribing to a particular list impossible, he could get it to send out 
numerous registration confirmation messages to skew the ratio.

Sincerely,
Philip Miller

---

From: "Chris" <[email-masked]>
Date: Thu, 20 May 2004 17:43:43 +0930

> > imbalance between mail sent and mail received.

Totally Unforgivable Rubbish (TUR for short)

I recieve maybe a hundred emails each day. but when I send out to my mail
list I send tens of thousands

under the above 'criteria' I automatically become a "spammer" when in fact I
simply have a lot of potential customers ALL on a double opt in list.

This is very blinkered reasoning.

Regards
Chris

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of Jim
> Witte
> Sent: Thursday, 20 May 2004 6:14 AM
> To: [email-masked]
> Subject: [Asrg] Re: Spam send/receive ratio
>
>

---

From: "Alan DeKok" <[email-masked]>
Date: Thu, 20 May 2004 10:33:21 -0400

> list I send tens of thousands

  Really?  That's not how mailing lists work, so far as I know.

  The *list* re-sends the message to tens of thousands of recipients.
You don't.  Since each recipient has opted-in to the list, this isn't
a problem.

> simply have a lot of potential customers ALL on a double opt in list.

  No.  You match a distinguishing characteristic of spammers.  This
doesn't mean you're a spammer.  It just means that your behavior
overlaps with that of spammers.

  This isn't news.  Everyone uses SMTP to send mail, and so do
spammers.  This doesn't make everyone using SMTP a spammer.

  Spammers have additional distinguishing characteristics which you
don't match.  Ergo, you're not a spammer.

  In other news, is everyone with a tattoo a criminal?  I've seen
"wanted" posters which list "tattos" as "distinguishing
characteristics" of a criminal.

> This is very blinkered reasoning.

  Please blink more often.  It might clarify your vision.

  Alan DeKok.

---

From: "Chris" <[email-masked]>
Date: Fri, 21 May 2004 11:00:33 +0930

> You don't.  Since each recipient has opted-in to the list, this isn't

HOW is a receiving MTA going to know this? it just sees the volume
discrepency and says 'Spammer'.
Just like some befuddled people see tattoos and think 'criminal'.

> characteristics" of a criminal.

Huh. So if I have a tattoo I am *more likely* to be a criminal then.

most criminals have hair. should that be listed as a distinguishing feature?

Ever heard of white collar crime... thats the stuff that costs billions of
dollars and ruins country economies. nary a tattoo in sight.

I stick to very blinkered. with no apology.

No doubt volume is an indicator that SHOULD be used. but only at the
injection point not at the receiving point. the receiver has no way of
knowing that volume may mean spam.

But the ISP who suddenly sees an upsurge in mail originating from a client
could spool that mail and confirm with the client that such volumes is in
fact intended.

Even this would catch me as I only do a mail out on average once a year. but
at least I could respond to the (automatically generated) mail with some
sort of verification that Yes I intended to send those emails please let
them pass.

A no would pobably mean I have been zombified. and no mail should be allowed
out until the problem is resolved.

A simple turing test would stop the zombified machine automatically
responding a yes to the validation email.

Chris

> From: [email-masked] [mailto:[email-masked]]On Behalf Of Alan
> DeKok
> Sent: Friday, 21 May 2004 12:03 AM
> To: [email-masked]
> Subject: Re: [Asrg] Re: Spam send/receive ratio
>
>

---

From: "Alan DeKok" <[email-masked]>
Date: Fri, 21 May 2004 11:15:26 -0400

"Chris" <[email-masked]> wrote:
> HOW is a receiving MTA going to know this?

  It sees the mail coming from the list, so any concerns about volume
are associated with the list, and not with the original submitter.

  That was your original point, I believe.  That you would be marked a
spammer because of the list exploding your traffic to tens of
thousands of people.  But it's not you, it's the list.

> it just sees the volume discrepency and says 'Spammer'. 

  Dumb MTA's may closely monitor the list, due to the volume of
traffic coming from it.  However, there are ways for the MTA to tell
that the traffic is NOT spam:

  - users at the MTA's domain send messages to the list
  - so the MTA communicates with the list's MX's
  - lists are long-lived, as opposed to spamvertized addresses
  - little spam comes from lists

> Huh. So if I have a tattoo I am *more likely* to be a criminal then.

  No.  The relationship is the inverse of what you said, which is an
important distinction: Criminals are more likely to have tattoos than
the average person.

  The distinction is critical to a proper understanding of the problem.

> most criminals have hair. should that be listed as a distinguishing feature?

  Most non-criminals have hair, too.  Therefore it's not a
distinguishing feature of criminals.

  This shouldn't be rocket science.

> dollars and ruins country economies. nary a tattoo in sight.

  You're misunderstanding statistics, correlation, causation,
overlapping sets, and inverse problems.  You're not alone.  These are
difficult problems for anyone.

  Criminals are more likely to have tattoos than the average person,
but not all criminals have tattoos, and not all tattooed people are
criminals.

> I stick to very blinkered. with no apology.

  Then I'll apologize for pointing out the flaws in your reasoning.

> knowing that volume may mean spam.

  The receiver doesn't have visibility into the injection point.  The
receiver sees only what it receives.

  And large volumes of traffic from someone the reciever has never
communicated with before is a strong indicator that the traffic may be
spam.  At the minimum, the receiver should treat the traffic with more
suspicion that low-volume traffic, or traffic from well-known sources.

> could spool that mail and confirm with the client that such volumes is in

  And the recipient can do the same thing.

> responding a yes to the validation email.

  Please describe such a test that would work in practice.

  Here, "work" means not only be practical as a turing test, but also
be something that the end user would respond to.

  Alan DeKok.

---

From: David Maxwell <[email-masked]>
Date: Fri, 21 May 2004 12:04:11 -0400

On Fri, 21 May 2004, Chris wrote:
> Someone that Chris wasn't polite enough to attribute, wrote:

No, you're only more likly to be _that_ criminal, which you probably
don't match the description of in a dozen other ways.

He said "of a criminal", not "of criminals". Tattoos are a way to
confirm that you've recognized that person, but probably not the best
way to go about finding him in a crowd.

-- 
David Maxwell, [email-masked][email-masked] -->
(About an Amiga rendering landscapes) It's not thinking, it's being artistic!
					      - Jamie Woods

---

From: "Chris" <[email-masked]>
Date: Sat, 22 May 2004 16:20:39 +0930

> are associated with the list, and not with the original submitter.

? perhaps confusion here over the meaning of "list" I am talking about a
mailing list of my customers
I am not talking about a list such as this ASRG list. on my mailing list I
am the submitter. I sent the mail.
But it makes no difference either way.

> difficult problems for anyone.

No what I was saying is that using the statistics of mail volume to try and
pinpoint a spammer is a bogus test.
Statistics can and are misused all the time. Mail Volume is just such a
statistic.

My example was of my mailing list so I will continue...

Say I send out twenty emails a day. The general number of service emails to
my clients.

I develop a new piece of software so I decide to pen a special invitation to
my clients and tell them thay can get the software at a special invitation
only price of $x dollars. I include the URL of the order form for
conveniance.

Now your good intentioned MTA say "hey I got mail from xyz, I'll check his
volumes". and it comes up with "normal volume = 20, last hours volume =
5000". "Ooops" the MTA says "Spammer alert" and tosses my email. Perhaps it
also looked for other statistically telltale clues such as an embedded URL
and/or a perhaps the words '/special.*price/' and combined with the sudden
high volume it decides 'spam!'.

The MTA has made a bad decision which the recipient may not have
appreciated, especially when he eventually gets to my site and is forced to
pay full price because the mail didn't get thru.

My argument is simple. Mail Volumes can *in no way* be used to indicate
spam. it is simply flawed logic.

Again to return to the Tattoo 'statistic'. Imagine a Judge in a murder trial
telling the jury that statistically more murderers have tattoos.  and as the
defendant has a tattoo he is more likely to be the murderer.

The statistic may be absolutely correct, BUT the inference cannot be used in
the decision making process. to do so is seriously flawed.

A judge who said that could and should be disbarred for incompetance.

> be something that the end user would respond to.

The end user is not relevant to this discussion. the only purposes of this
test is the sender's ISP confirming that *HIS* client in fact intended to
send out such volumes of mail. remembering that we are talking about an
unusual volume.

If I was to write such a volume check it would go like this...

Stats would be kept of the average volume of all mail sent from all clients.

As a client starts sending mail, a counter would be incremented. after a
period of time this count would be tested against the average for that
client for that time period. should the average exceed previous averages
then the mail would start to spool.

After another period of time the counts would again be tested. IF the number
of sends has dropped, the mail would be sent and it would be assumed it was
a simple spike, perhaps birthday invitations.

IF the count continued to exceed expectations then spooling would be
continued. an e-mail or other form of contact would be dispatched to the
client.

This email would contain a simple turing test something like numbers
embedded into an image or perhaps a stronger test. the recipient would have
to respond with the correct answer. This is to prevent a zombified machine
from intercepting the validation email and responding to it.

If the response was correct the spooled mail would be sent.

This is not designed to prevent a spammer. it is designed to prevent
someones account being misused by a third party.

Chris

> Sent: Saturday, 22 May 2004 12:45 AM
> To: [email-masked]

---

From: "Alan DeKok" <[email-masked]>
Date: Sat, 22 May 2004 08:38:56 -0400

> pinpoint a spammer is a bogus test.

  When used alone.  I never said it should be used alone.  In fact, I
said the exact opposite.

> 5000". "Ooops" the MTA says "Spammer alert" and tosses my email.

  Then it's not "my good intentioned MTA".  It's an MTA you invented
by explicitly ignoring my statements about how MTA's should deal with
high-volume spam.  I hate it when people read my messages, and
conclude that I believe the exact opposite of what I've said.

  Can you please explain why you're arguing that I believe high-volume
to be a near-perfect indicator of spam?  I just can't understand how
you come to that decision.

> Perhaps it also looked for other statistically telltale clues such
> as an embedded URL and/or a perhaps the words '/special.*price/' and
> combined with the sudden high volume it decides 'spam!'.

  If an MTA decides that a message is spam, that's it's perogative.

  In this (badly defined, hypothetical) case, I could agree that the
"spam" determination is probably not the bes thing to do.

> spam. it is simply flawed logic.

  By the same argument, the word "v**gr*" cannot be used to indicate
spam, because normal people use it in normal messages.

  The reality is different.  Keywords are strongly correlated with
"spamminess" of a message.  But they're not perfectly correlated.
Similarly, suddenly receiving a high volume of mail from a host which
usually sneds low-volume traffic is strongly correlated with spam from
zombied machines, or a new account buy a spammer, etc.  See the ASRG
archives for ISP admins discussion of this exact scenario.

> defendant has a tattoo he is more likely to be the murderer.

  The first statement may be correct.  The second does not follow from
the first.  You are arguing from flawed logic.  I tried to explain why
in my previous message, and it looks like I failed to communicate.

  Please read books by Marilyn Vos Savant on logic, or Jon Paulos
Allen.

  I suggest:

http://www.amazon.com/exec/obidos/tg/detail/-/0679726012/102-9321111-1748922?v=glance
http://www.alibris.com/search/search.cfm?qwork=5268032&matches=21&qsort=r

> The statistic may be absolutely correct,

  There is no statistic in your argument.  There is simply flawed
logic.  Correlation does not imply causation.

  Alan DeKok.

---

From: "Chris" <[email-masked]>
Date: Sun, 23 May 2004 11:03:10 +0930

>   When used alone.  I never said it should be used alone.  In fact, I

I never said you did. alone its useless, combined with other "triggers" its
still useless.
All I said is that no matter how it is used "Mail Volume" is unreliable to
the point it should not be considered.

Perhaps you could quieten this critic if you explain how Mail Volume can be
used to *reliably* detect spam.
I have tried my best to explain why it would fail. surely a counter point is
required at this stage.

Anti Spam systems that chuck out the baby with the bathwater should not be
considered as acceptable(IMHO).

As a mail user I would prefer to skim through a few spam mails KNOWING I
haven't lost anything important, than getting no spam and missing out on a
vital business/personal message.

You include another example V*gra

If you were a medical student discussing some of the finer points of V*gra
with a collegue, would you not wish for your mail to go through. or should
the word be censored from this moment forward.

Chris

> Sent: Saturday, 22 May 2004 10:09 PM
> To: [email-masked]

  Alan DeKok.

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 23 May 2004 10:24:20 -0400

> used to *reliably* detect spam.

  Did you read my previous messages?

> I have tried my best to explain why it would fail. surely a counter point is

  Did you read my previous messages?

> considered as acceptable(IMHO).

  Did you read my previous messages?

  Alan DeKok.

---

From: Bill Yurcik <[email-masked]>
Date: Sun, 23 May 2004 09:39:57 -0500 (CDT)

I am looking for Internet metrics on spam but with a security focus
such as number of open relays currently detected, current
number/type of malware propagating via spam, port activity
via spammers, or anything else that may be suggested as an indicator of
spam security.  I am especially interested in spam security trends.
I know this may be naive but I am new to the list and thought it wouldn't
hurt to ask as a newbie.

- Bill Yurcik
<[email-masked]>
NCSA Security Research

---

From: der Mouse <[email-masked]>
Date: Sun, 23 May 2004 12:28:59 -0400 (EDT)

>>> [...] the statistics of mail volume [...]
> alone its useless, combined with other "triggers" its still useless.

I disagree.  If I were setting up my ideal ISP, one thing it would
include is watching the outgoing email volume from each account.  A
sudden spike would cause throttling and flagging for human attention.
The human could then reopen the throttle, block the account, check with
the owner, whatever.

"Useless"?  Maybe.  I haven't had the luxury of being able to try it.
But I suspect it wouldn't be.

As Alan said, did you actually read the message you're responding to?
It looks more as though you picked out one point, carried it to an
extreme not supported by the message you took it from, and then argued
against that extreme.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: "Chris" <[email-masked]>
Date: Mon, 24 May 2004 11:25:41 +0930

Have you read the thread or just picked out something to comment on?

> The human could then reopen the throttle, block the account, check with

If you had read the thread you would have seen this is exactly what I said
it was at most useable for. and in fact went on to outline how this would be
applied (at Alans request) which included an automated turing test to allow
the sender to confirm whether or not they in fact intended to send that
quantity of mail.

This rather lengthy explanation was only two or three mailings ago.

unfortunately at the receiving end mail volumes is in fact *useless*.

It fails for the reasons I have stated in this thread. Regardles of Alans
protestations to the contrary he has not outlined how it would/could work.
only drawn references to statistics and not outlined how those statistics
would and could be applied.

> But I suspect it wouldn't be.

Read the thread you will at least see why I think it is useless. No-one has
yet come up with a counter point or example of how it could work.

> extreme not supported by the message you took it from, and then argued

Quite the opposite but I did leave out what had been gone over before. It
seems you only read Alans mail and took his word for it.

I can only respond on what needs to be responded about. WE ALL skip portions
of messages we don't feel are necessary to respond to. If we didnt we would
spend our days responding to longer and longer messages and nothing would be
achieved.

Not that I am saying anything is actually being achieved. Disappointingly it
is not.

Chris

> From: [email-masked] [mailto:[email-masked]]On Behalf Of der
> Mouse
> Sent: Monday, 24 May 2004 1:59 AM
> To: [email-masked]

---

From: "Chris" <[email-masked]>
Date: Mon, 24 May 2004 11:25:42 +0930

>   Did you read my previous messages?

Perhaps there is more to this thread than I have. the earliest copy of the
thread that I have is the one I originally responded to.

If you have an earlier item which outlined how mail volumes could be
adequately used I would appreciate a copy of it.

IN REVIEW...

> but not all criminals have tattoos, and not all tattooed people are

This is as close to an explanation of how it would work that I have seen
from you. apart from being condescending it does not explain how such
theories/methods could actually be applied.

THIS...

>>  - little spam comes from lists

Is your only other explanation. It seems to discuss mailing lists such as
the ASRG. as it was not relevant and I believe a misunderstanding as to what
I meant by the word "list" I pointed out this miscommunication and moved on.

But lets look at it anyway.

>> traffic coming from it.
Who's dumb MTA does this? should every ISP watch every mail list in case a
user joins it? If not why can't a spammer put up false "authoritive list
monitor" for the list they want to spam to.

>>  - users at the MTA's domain send messages to the list
So the MTA MUST TRACK every outgoing message and remember its destination.
for how long?

>>  - so the MTA communicates with the list's MX's
I assume you mean after a message is received. and it compares it to the
sent mail for this reciever to see if they sent to it. How would one ever
join a list. the initial mail would always be dropped because the user
hasn't sent to the list before.
(I joined the ASRG from a web page, Didn't you?)

>>  - lists are long-lived, as opposed to spamvertized addresses
Irrelevant. A new list is not long lived, its new.

>>  - little spam comes from lists
Very true. and easily blocked by the end user.
Hardly worth all the resources you are suggesting throwing at them and where
was the "volume watch" in all of this? remember we are talking about a
"Volume Spike" as the telltale sign most discussion lists (such as the
ASRG's) send out a fairly consistant volume roughly equal to the number of
members.

AND THEN THERE WAS THIS...

One more item I didnt respond to I didn't think it was a reasonable
statement so I didn't. But I will now.

>>  you come to that decision.

Where in any of my mailings did I say this? "near perfect indicator" I never
at any stage said any such thing, or even came close to it.

I only ever said that the use of mail volumes as any sort of indicator is
flawed. Even combined with as many other indicators as you want to throw at
it I don't believe it has any use.
I EVEN GAVE EXAMPLES OF WHEN IT WOULD FAIL eg: the special deal for software
to my clients. It has all the hallmarks of spam using many common indicators
and if you added the volume check (even if that only accounted for 10
percentage points of the test) it may push the filter into the spam range.
as the list is a double opt in list this is quite obviously a false
positive.

Do not attempt to twist my words into some sort of personal attack.

Chris

> Sent: Sunday, 23 May 2004 11:54 PM
> To: [email-masked]

---

From: "Alan DeKok" <[email-masked]>
Date: Mon, 24 May 2004 10:17:59 -0400

> at any stage said any such thing, or even came close to it.

  Sorry, you did.  You (paraphrased) said "in YOUR mta, it would see a
high volume of messages, and discard them as spam."  Any rational
person would conclude that that statement was meant to describe my
position, and was meant to describe an MTA which used volume as the
sole (i.e. perfect) indicator, for messages being spam.

  Can you explain why when I say "imperfect, but useful as part of a
system", you re-phrase it as "perfect, and usable only by itself"?

  Your prejudice against using mail volume is so high that it is
preventing you from understanding my messages.

> Do not attempt to twist my words into some sort of personal attack.

  Oh, I'm clear it's not personal.  You're careful to not make it
personal.  But it's still annoying when I respond to what you've
written, and your response is to re-define what you've said to mean
something else, and then attack my response as not addressing that new
topic.

  Alan DeKok.