Subject: Re: [Asrg] Some data on the validity of MAIL FROM addresses

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 18 May 2003 14:49:49 -0400

At 03:34 AM 5/18/2003 -0400, Kee Hinckley wrote:

>Vernon has regularly made the claim that a significant proportion of
>spam messages have valid MAIL FROM's.  That means that bounces will
>go the the spammer.  This has significant ramifications for C/R
>systems (especially auto-respond ones) since it means that should
>they have to, spammers could respond to challenges.
>
>To test this theory, I took a day's worth of bounce logs from
>somewhere.com (2003-05-15).  These should be fairly normal logs.
>There's been a bit of an upswing from a recent virus attack, but
>otherwise these are pretty normal bounce logs for somewhere.com.
>
>I ran a program which took each MAIL FROM address, parsed out the
>domain portion, looked up the MX record, and then connected to the
>SMTP port of the lowest numbered MX server.  I did a
>         HELO somewhere.com
>         MAIL FROM <[email-masked]>
>         RCPT TO <appropriate-address>
>         QUIT
>Note that a few sites bounced me at the HELO prompt (didn't like that
>I was on DSL, or that my name was somewhere.com).  A few bounced at
>the MAIL FROM (didn't like somewhere.com--and one claimed that +
>wasn't a legal email character).  But the number of either of those
>was pretty low (less than half a dozen).  I'll do a better job of
>recording those separately in the future.
>
>[.....]
>In general though, it appears that Vernon is correct.  If my sample
>is representative, a large percentage of spam is coming from real
>email addresses.

I see a problem with this testing strategy - an SMTP server is does not 
necessarily produce an error when receiving an RCPT TO command. See RFC 
2821, section 3.3:

----[snip]----
"However, in practice, some servers do not perform recipient verification 
until after the message text is received.  These servers    SHOULD treat a 
failure for one or more recipients as a "subsequent failure" and return a 
mail message as discussed in section 6.  Using a "550 mailbox not found" 
(or equivalent) reply code after the data are accepted makes it difficult 
or impossible for the client to   determine which recipients failed."
----[snip]----

And RFC 2821, Section 6.1.:

----[snip]----
"If there is a delivery failure after acceptance of a message, the 
receiver-SMTP MUST formulate and mail a notification message."
----[snip]----

Therefore, it is not possible to determine with certainty whether these 
accounts actually existed. A better testing strategy would actually send 
email to these accounts with the DATA command and watch for bounce 
messages. However, spammers can always choose to use a real email address 
as the return address and sending email to valid accounts in itself may be 
considered spam by the recipients.

Yakov

---------------------------------------------------------------------------------------------------
Yakov Shafranovich / <[email-masked]>
SolidMatrix Research, a division of SolidMatrix Technologies, Inc.
---------------------------------------------------------------------------------------------------
"One who watches the wind will never sow, and one who keeps his eyes on
the clouds will never reap" (Ecclesiastes 11:4)
---------------------------------------------------------------------------------------------------  

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Scott Nelson <[email-masked]>
Date: Sun, 18 May 2003 15:46:38 -0700

>they have to, spammers could respond to challenges.

There's a significant difference between "address is valid"
and "address belongs to spammer who could respond."

It's certainly possible for the Reply-To: (or any other sender supplied
addresses you test) to point to a valid address that does belong
to the sender (real), valid that doesn't belong to the sender (forged),
currently invalid address which belonged to the sender when they sent 
the message (revoked by isp) or invalid which never belonged to the 
sender (abandoned once the bounces started flooding in).

It would be nice to know how many addresses that tested valid 
when the spam was received, test invalid some time later 
(say, after 48 hours).  Of course, that data might not be much use, 
but at least it's collectable.

>
>There were 39595 entries in the log, with 34404 distinct SMTP sessions.
>There were 11559 unique MAIL FROM addresses.
>
>+---------+-------+------------+
>| errcode | total | percentage |
>+---------+-------+------------+
>|       0 |    99 |       0.86 |	???
>|     250 |  5796 |      50.14 |
>|     450 |     6 |       0.05 |
>|     451 |    12 |       0.10 |
>|     452 |     8 |       0.07 |
>|     473 |     4 |       0.03 |
>|     500 |     1 |       0.01 |
>|     501 |     1 |       0.01 |
>|     521 |     3 |       0.03 |
>|     530 |     1 |       0.01 |
>|     550 |  2341 |      20.25 |
>|     551 |     3 |       0.03 |
>|     552 |     2 |       0.02 |
>|     553 |   288 |       2.49 |
>|     554 |    48 |       0.42 |
>|     555 |     1 |       0.01 |
>|     556 |     1 |       0.01 |
>|     571 |     1 |       0.01 |
>|    1001 |  1880 |      16.26 |	No MX Record
>|    1003 |  1055 |       9.13 |	No SMTP Server
>|    1007 |     8 |       0.07 |	Invalid Email Format
>+---------+-------+------------+
>
>In aggregate.  51% of the addresses were valid.  49% were not.
>Of the ones that were not valid, 52% didn't have a reachable mail server.
>

Minor quibble:  Failing to have an MX record doesn't guarantee
an address is invalid.  But I'd bet that a fair number of the
"No MX Record" group don't have an A record either, and of the
ones that do, few if any point to a SMTP server.

At any rate, somewhere between 9% - 25% not only don't have
valid return addresses, the addresses given can't reasonably
be considered to ever have been valid.  We can at least say
with some certainty that those spams aren't impersonating
someone else.

Scott Nelson <[email-masked]>

---

From: Scott Nelson <[email-masked]>
Date: Sun, 18 May 2003 19:01:24 -0700

At 08:58 PM 5/18/03 -0400, Kee Hinckley wrote:
>At 10:53 AM +0100 5/18/03, Jon Kyme wrote:
>>  >
>>>  Interesting that the results vary so much by ISP.  Yahoo accounts are
>>>  pretty valid.  Hotmail accounts are pretty bad.  AOL is quite good.
>>>  Earthlink has a problem.  MSN's slightly better, but still negative.
>>
>>
>>Of course yahoo will say 250 to pretty much anything.
>>So these addresses are "valid" in what sense exactly?
>
>They said "no" to 16% of the messages I queried them on.  The 
>specific message they used was:
>
>553 VS10-RT Possible forgery or deactivated due to abuse (#5.1.1)
>
>Can you show instances in which they say yes to messages they cannot deliver?
>

I tried this:

% head /dev/urandom | md5sum
ce4c096d881440fffebbde3b64291d92  -

% telnet mx1.mail.yahoo.com 25
Trying 64.156.215.5...
Connected to mx1.mail.yahoo.com.
Escape character is '^]'.
220 YSmtp mta124.mail.scd.yahoo.com ESMTP service ready
helo spamwolf.com
250 mta124.mail.scd.yahoo.com
mail from:<>
250 null sender <> ok
rcpt to:<[email-masked]>
250 recipient <[email-masked]> ok
quit
221 mta124.mail.scd.yahoo.com
Connection closed by foreign host.

Saying that <[email-masked]> is 
unlikely to be a valid address is an understatement.

---

From: "Eric D. Williams" <[email-masked]>
Date: Tue, 20 May 2003 22:16:31 -0400

On Monday, May 19, 2003 9:07 AM, Alan DeKok [SMTP:[email-masked]] wrote:
> Scott Nelson <[email-masked]> wrote:
> > So what domains /aren't/ showing up in spam?
> >
> > I know striker.ottawa.on.ca has a lot of addresses on a lot of spam lists,
> > but checking the last 20,000 spams I received, not a single one
> > has the word "striker" in it anywhere.
>
>   I've talked to people with large spamtraps, and they see do see it.
> Not in the same volume as hotmail, though.

If you can get some data we could parse that would be a good thing.  Since we 
seem to have begun to establish some methodologies for scoping an area of 
concern.

-e

---

From: "Eric D. Williams" <[email-masked]>
Date: Tue, 20 May 2003 22:31:23 -0400

On Monday, May 19, 2003 11:48 AM, Vernon Schryver 
[SMTP:[email-masked]] wrote:
8<...>8
>   - some recent spam contains HTTP HREF URLs of the form
>      http://xxx.example.com__foobar
>     Does some HTML-aware MUA treat '__' like '/'?
>
>   - some recent spam contains several purely garbage URLs in HREFs or
>    two valid ones.  They're not even hidden (e.g. <A HREF="asdf"></A>)
>    What's the point of telling spam targets to "click on this" when it
>    is pure garbage?

These two are indeed interesting as they may not be spam at all but probes or 
perhaps even bugs of some sort.  It would be interesting to see whether your 
speculation on the first is true, and whether a BASE reference exists in the 
other.

-e

---

From: "Eric D. Williams" <[email-masked]>
Date: Tue, 20 May 2003 23:02:09 -0400

On Monday, May 19, 2003 5:50 PM, Vernon Schryver 
[SMTP:[email-masked]] wrote:
8<...>8
> That's not what I see.  About 192 or 0.6% of the last 27,972 spam
> caught by my traps had bogus sender domain names.  Since modern
> versions of sendmail and other MTAs usually require that the sender
> domain exist, the surprise is that even that small amount of spam
> has bogus sender domains.

I would like people to seriously consider this particular paragraph. It seems 
to touch on the vital issue of how 'spam' is produced in the wild and the 
automated means for its insertion into the MTS.  Whether an SMTP implementation 
is 'modern' or not is not relevant to the issue IMHO, but that is a factor in 
determining what the footprint is for the various tools and techniques are that 
are being utilized.

Has anyone done a forensic examination of 'spamware' and/or developed a 
taxonomy of its internals?
Will knowing the 'weapons' - to co-opt an analogy being proposed by some - aid 
in defeating the 'enemy'?
Are there any other 'artifacts' not directly related to the 'spam' problem, 
that can aid in determining methods to defeat 'it'?

-e

---

From: "Jon Kyme" <[email-masked]>
Date: Wed, 21 May 2003 13:27:06 +0100

>
>  b) The amount of horse power needed to process mail increases.  Our 
>     stats show that roughly 40% of our inbound SMTP connections refer
>     to multiple recipients.
>

And another thing (sorry) how much of this is spam? 

We're a pretty small mail system - I had a quick look at a 
(I hope representative) set of 50000 incoming messages.
We delivered these to 12843 mailboxes
774 (1.5%) had multiple recipients.

Of course many of these distinct messages will have come down the same
*connection*. There's no possiblity that there's confusion here is there?

This is so much at odds with the figures you report that I'm going to
really (no, really this time) check my script.

--

---

From: "Clayton, Nik [IT]" <[email-masked]>
Date: Wed, 21 May 2003 13:37:15 +0100

> >At least two workarounds exist for this.  One is to configure Sendmail to
> >only allow one recipient per SMTP transaction.  I.e., return 250 to the
> >first RCPT, and 550 to the others.
> 
> Oooh, surely not. 452 ? 

Yep.  Brain fart on my part.

N
-- 
1        1         2         3         4         5         6         7    7
         0         0         0         0         0         0         0    5
                                                    -- The 75 column-ometer
Global Messaging,                 A: Top posting
120 Cheapside, x83331             Q: What's the most annoying e-mail habit?
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Yakov Shafranovich <[email-masked]>
Date: Wed, 21 May 2003 15:28:44 -0400

At 11:32 AM 5/21/2003 -0700, you wrote:

>BTW, it would be instructive if someone who has access to one, and the
>know-how to decompile it, could examine its sending code and indicate
>whether it uses randomly-generated addresses, or a static list of existing
>ones loaded from a config file, for the addresses used in the MAIL FROM
>SMTP command and From: header.

I think it would me helpful if we can get all the information we can out of 
such tools. Is there anyone that can provide such information or perhaps 
have some "white-hat" friends that do?

Yakov

---

From: "Eric D. Williams" <[email-masked]>
Date: Wed, 21 May 2003 21:30:49 -0400

On Wednesday, May 21, 2003 10:41 AM, Vernon Schryver 
[SMTP:[email-masked]] wrote:
8<...>8
> Regardless of heat, this mailing list must address the issue of whether
> filtering SHOULD (in the special meaning of RFC 2119) be done during
> the SMTP transaction.  If we can't reach consensus on such a fundamental
> and simple issue, then this mailing list is distinctly worse than useless.

I agree.  I have been closely following this thread.  Though I won't at this 
point go into a diatribe concerning why, if anyone wants to discuss it hit me 
off-list. If there were a specific proposal associated with this concept, I 
certainly would be strongly leaning toward the idea that on this issue of where 
filtering as a mechanism to address RECEIVING spam is placed, is a SHOULD.

-e

---

From: "Eric D. Williams" <[email-masked]>
Date: Wed, 21 May 2003 22:01:44 -0400

On Wednesday, May 21, 2003 2:32 PM, Justin Mason [SMTP:[email-masked]] wrote:
8<...>8
> SMTP command and From: header.

You nailed it. That's where I am coming from exactly, I can do the forensics, 
or aid in decompilation and disassembly of it, however the key is to gain 
possession of some known working examples, and unfortunately (or fortunately 
for me) I have not had to 'deal with the zombies'.  In any event, if some code 
should become available I would be interested in developing the 'spamware' 
taxonomy.  Thanks for the search pointers J.

-e