Subject: [Asrg] 0. General: Some smtp statistics, analsys and suggestions

From: "Hector Santos" <[email-masked]>
Date: Sat, 29 Nov 2003 22:30:57 -0500

For your perusal, the following accumulated stats for the past 6-7 months
that you might find interesting:

             http://www.winserver.com/public/antispam/wcsmtpstats.htm

Only the past 3 months I had the logs turned on each day in anticipating of
getting some better analysis.

Engineering design notes,  Log analysis and suggestions:

The rejection stats in the logs are based on what happens at each state
point in the classic SMTP protocol state machine:

           http:://www.winserver.com/public/antispam/smtpstatemachine.jpg

Connect State:

The White/Black list and RBL checking is done at the connection level with
the rejection response delayed until the RCPT TO: state is reached.

This was done because we quickly learned that a certain group of spammers
(software) will continue to connect non-stop if the connection was
immediately and passively dropped.  This information was being logged and
accumulated in a very high volume.  In other words, it didn't seem like they
were 'learning" about the passive connection drops.  By providing some SMTP
feedback, it helped resolves their "scheduler."

I selected the RCPT TO: state rather than earlier states to provide
rejection feedback so that we can log the target email address to analyze
any possible any false negatives issues by the RBLs.

Suggestions:

SMTP guidelines may suggest server implementations performing IP connection
filtering should provide a SMTP rejection response at the greeting,
HELO/EHLO or the RCPT TO: state points rather than passively dropping the
connection to help reduce continue attack by "dumb spammer software."

HELO/EHLO State:

In our original implementation, we provided no logic to check the client
host issued by the remote MTA.  In 1999, we offered an optional strict
intranet client host domain checking for closed systems.

In August 21, 2003, we added new default logic to perform basic format
client domain format validation as defined by the RFC 2821 section 4.1.3
(Address Literal), this includes qualifying  the usage of a bracketed IP
address provided by the client and checking the IP against for fraud IPs
(using your IP for example).    As you can see beginning with the August
stats, this drastically helped reduce illegal entries at the HELO/EHLO
state.

The most common illegal usage is "HELO your.IP.address" without brackets.

Suggestion:

Obviously, the HELO/EHLO is probably the first of the most importantly state
point where technical compliance must be followed to help address the
illegal entries.  There are few proposals on this subject of HELO/EHLO DNS
lookup validations.

I would like to suggest that the SMTP specs be changed to suggest that a
bracket IP address be used in all cases for all MTA clients.

This will allow for a quick non-DNS comparison against the IP connect
address.  There is no need to provide a FQDN.  However, it does, it must
match the connecting IP address.

Legacy clients will be handled by defined the SMTP server implementation.
My only suggestion here is that the SMTP puts it in writing that SMTP server
are NOT required to support old non-compliance software.  Obviously, server
software will do a minimum here and allow the client to go next state, but
by adding the requirement to the specs, software vendors can begin to change
their client ware.   Once possibility to offer an new positive response
code:

            251  Warning: Illegal Client down. Future entry may be denied.

MAIL FROM: State:

Our new callerid verification method (WCSAP) was turned on November 23,
2003.  This is a new hook into our SMTP server that is called once the MAIL
FROM: return path is provided.

The connection RBL rejection flag is passed to WCSAP so that it can log it
and skip the callerid verification process. In this case, a MAIL FROM: 552
response is provided.

As you can see in the November 2003 stats, before the 23rd, RBL provided the
strongest rejection method before the RCPT TO: state point is reached.

However,  with WCSAP implemented,  Caller ID verification now provides an
even higher rejection rate, nearly making RBL obsolete.  This might suggest
that the RBL databases, I am using BL.SPAMCOP.NET,   is doing an excellent
job verifiable with return path validation logic.

In our WCSAP implementation,  it follows a standard SMTP sender process,
honoring MX logic, etc.  It looks for the response code for a RCPT TO:
<original return path> and if a positive result is provided, it goes a step
further to check for open relay sites by issues an unique and random RCPT
TO: email address.  Optionally, WCSAP can return a 552 for the open relay
site or allow it to go thru with a 250 response code.

The open relay checks works great!   However, it is problematic with
YAHOO.COM which is inherently for all intent and purpose an OPEN RELAY site!
I have my RejectOpenRelaySite option enabled in WCSAP, so sorry YAHOO, fix
your system.  Your system is NOT helping the internet email problem!     I
put more weight on an open relay site being a spam site.

Suggestions:

By far, the MAIL FROM: is the probably the focus point of most anti-spam
proposals.

I view WCSAP as a "HOOK" to validate the MAIL FROM: state point.  The
implementation is just one.  If a better approach comes along that provides
the same but optimized "verification" results,  fine!  WCSAP might be
changed to use it.

---

From: "Hector Santos" <[email-masked]>
Date: Sat, 29 Nov 2003 22:54:24 -0500

----- Original Message ----- 
From: "Hector Santos" <[email-masked]>
To: "Anti-Spam" <[email-masked]>
Sent: Saturday, November 29, 2003 10:30 PM
Subject: [Asrg] 0. General: Some smtp statistics, analsys and suggestions

>            http:://www.winserver.com/public/antispam/smtpstatemachine.jpg

This image can now be found in the above first URL.

---
Hector Santos, CTO
WINSERVER "Wildcat! Interactive Net Server"
support: http://www.winserver.com
sales: http://www.santronics.com

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 01:40:42 -0500

> connection to help reduce continue attack by "dumb spammer software."

Good material for a new BCP overhauling RFC 2505. Anybody wants to 
volunteer to help with this document?

> match the connecting IP address.

This is something that surely can be looked into, however what's the 
point of the HELO command in the first place if all it will do is state 
the IP address that is connecting? You know what that address is anyway. 
Therefore, before recommending this, we must find out *why* was the HELO 
command put in place in the first place.

RFC 2502 recommends the use of FQDN:

"When we suggest use of FQDNs rather than IP addresses this is because 
FQDNs are intuitively much easier to use."

> are NOT required to support old non-compliance software.

This needs to be looked into more, backwards compatability cannot be 
easily ignored.

> job verifiable with return path validation logic.

As per my other message, many RBLs are not reliable enough and do have 
false positive. My ISPs recently started using SpamCop and I received an 
email from a list member via a third party that was bounced because of 
that. The use of multiple RBLs or some other guidelines might be needed.

> put more weight on an open relay site being a spam site.

As per my other message, why is this an open relay if it does not relay 
email to outside domains? Plus what about domains that have "catch all" 
turned on?

Also, I do not think that *anyone* can afford to deny email from Yahoo, 
AOL, MSN, etc. who collectively have an extremely large number of users. 
They will not care that your customers are not getting their email, and 
it would be *your* customers who will care.

>
> If there is one possible SMTP command to add to the specs, it would be this
> idea:
> 
> non-server based MTA clients (local machine clients) use:
> 
>         MAIL FROM: <>
> 
 > server based MTA clients use:
 >
 >         RETURN PATH: <>
 >
 > This will give server software to decide if they want to VALIDATE the 
 > return path or not against DNS mx records.

There is an IETF SUBMIT protocol which is recommended to be used for 
local clients instead of SMTP, which makes this point moot.

Yakov

-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
"And this too shall come to pass"
-------

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 03:37:51 -0500

> HELO/EHLO

For logging and the matching the IP, but legacy server misconfigurations
makes this prohibitives. read below.

> FQDNs are intuitively much easier to use."

Legacy concept for TTY operations <g>   We are in the high speed, high
demand automation world.  Its one less lookup per million request is a
tremendous saving.

Yes, the need for HELO is probably not required, especially where EHLO is
available to provide extended capabilities.

The basic design is to use for servers to use the greeting to indentify an
ESMTP (Extended SMTP) system.
Not reliable though so system who issue an EHLO first are required to
fallback to HELO.

The SMTP specs should exploit this wonderful flexibility to the maximum
extent.

For example, make HELO optional forcing compliancy with EHLO extensions.
What I mean is that you need to give designer, software product the leverage
to offer advance security in compliance with the extended standards.  The
last thing I want to happen is offer something that is NOT compliant and
have some smoe sue it for tortious interference because we stopped his email
from spamming a site.   Again, I would like to stress the importance for the
"new" standards to offer technical comformity options.   My GUI
configuration for the Mail server may say:

        ---- Extended SMTP support ----

                [X] Support HELO
                [X] Validate Client Domain
                [X]  Strict Address Literal

etc, etc.  You must allow me to do this!!  Not that you can't stop me from
offering and I already offer strict extended smtp options, but I take pride
in RFC compliance. Its a selling point.  We don't tell me "We are 99.99%
compliant!"    We have this now:

            [X] Extended SMTP Authentication
                    [  ] Required for all connections (All users must log
in)
                    [  ] MAIL FROM must be local domain (Intranets, closed)

Anyway, once upon a time, we use the HELO/EHLO client doman for DNS lookups
to validate the IP against the connection IP.   Technically, I believe the
specs says this is want the desire was, however, legacy server do not allow
this to work.   Many systems use load balancing or "smart smtp" servers,
servers to receive and others to send.   What we saw is that many of these
setups do not configure their DNS requires correctly, so it was very
unreliable to validate the domain.    The logic was disabled.

So the suggestion is to "reinforce" the idea that server implementations may
now begin to offer strict connecting IP checking with the client domain in
the HELO/EHLO state.

I mean, look at it now when there is mismatch:

            230 Hello YYYYYY, why do you call yourself XXXXXX?

whats up with that?

If the server saw something is not right, give it the power to do something
about it! <g>

> This needs to be looked into more, backwards compatability cannot be

Oh I agree.  I for once will offer backward compatibility.  But I need to
offer strong connectivity too without someone (I will keep nameless) make a
claim my software is non-conformant and worst, allow this non-conformity be
used as evidence in a legal suit. :-)

> that. The use of multiple RBLs or some other guidelines might be needed.

Well, in my view, RBLS are a major contributor to the spam problem. I will
say that it isn't a coincidence the that grow of spam is directly
proportional with the growth of RBL databases.

Why?

Because if I wanted to become a bulk spammer tomorrow,  in just 1 hour of
operation, I can produce a list of thousands of open relays and open proxies
that are recorded in the databases.    I already proven this hypothesis with
real hard core results.   I downloaded the RBL database with over 2 million
or so entries and I wrote a program to double check which entires were FIXED
or still open for abuse.   I found on large percentage of the blacklisted
sites were stilll open in the first 30 days of being reported.  Over time
the systems were fixed, but it took awhile.

In other words, the RBL is a double-edge sword beause it also gives SPAMMER
information that would normally take weeks or more to gather in just 5
minutes!

However, I am in agreement that is everyone used it then it doesn't matter
what open relay site information is exposed.   Until that happens, its a
great source of open relay sites.

So in my technical,  if we can get away from RBL and similar systems, the
better!  I believe the commercial RBL sites make it different to get their
databases, but the freebies one are also good.

> email to outside domains? Plus what about domains that have "catch all"

I think we resolved this now.  Specifically YAHOO is delaying the validation
until the DATA state.   Initially, I didn't think so and with your
harvesting reason, I took it that is how it was.  In that mode,  it is an
open relay unless it bounces the message.

Independent of performance issues,  delaying validation increases the
problem.

We must remember spammers are in the volume business of selling email
addressss !   The more than can say "Hey this system accepts this address,
therefore it is valid" the better the marketing is for them when they are
selling their services to companies.  They can give a flying hoot what the
content of the mail is.  In the mail is accepted, that a problem.

But then again,  I am not proposing that we (SMTP) get involved in the
complicated mail analysis business.  All it can do is help the process using
strict technical specs.

> it would be *your* customers who will care.

I agree, but early on I personally announced for my own accounts to filter
all JUNO accounts. Lots of people followed suit and auto-magically spams was
reduced <g>

My only point, they have to comply like everyone else.  However, right,
there is no issue now with YAHOO.  They have a delayed validation which I
will add to WCSAP.   Solved :-)

> local clients instead of SMTP, which makes this point moot.

do you have something I can read?

Thanks

Hector Santos

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 03:49:28 -0500

Hector Santos wrote:
>>>If there is one possible SMTP command to add to the specs, it would be
> 
> this
> 
>>>idea:
>>>

RFC 2476, HTMLized copy here:

http://www.faqs.org/rfcs/rfc2476.html

Yakov
-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
"Power tends to corrupt, and absolute power corrupts absolutely" (Lord 
Acton)
-------