Subject: RE: [Asrg] Re: 6. Proposals - rDNS and rMX

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Thu, 27 Nov 2003 10:36:39 -0800

>Many people prefer the LMAP proposals that deal with regular "forward" 
>DNS than the MTA mark proposal that uses reverse DNS, because a usual 
>domain owner has control over regular DNS and does not have control over 
>reverse DNS.

Absolutely, reverse DNS has not been generally considered a critical
infrastructure, nobody uses it today except for debugging. 

The zones are prefectly stable, the problem is that the information in those
zones is not maintained by the IP block holders to a sufficiently high
quality. 

The other practical problem is that there are machines with several hundred
thousand email domains parked on one machine.

		Phill

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Jukka Inkeri <[email-masked]>
Date: Fri, 28 Nov 2003 11:13:51 +0200

> infrastructure, nobody uses it today except for debugging. 
Really ?
I know that lot of ex. finnish ISP, companies, servers, .... use reverse 
checking before accept connections. Also us. Maybe not in every 
services. Ex. PARANOID flag is very much used in many services, 
wrappers, ... same way like phone numbers, if somebody don't like to 
tell that this number belongs to them, then we have nothing to say. We 
are not so interesting handle anonymous trafic. Ofcource reverse is only 
little thing but that tell something if you have not reverse info.

> zones is not maintained by the IP block holders to a sufficiently high
Ex. in Europe I think that situation is not excellent, but good. Thanks 
for RIPE.
Also every reverse domain has soa and there is ...

> thousand email domains parked on one machine.
How it's problem ?
If we look server site, you can live just like today if you don't like 
to use this add-on posibilities. So this is not have to, this like all 
other security and so on, You can use it, if you like it.

If you are mail sender and like to make connection to server which, use 
this kind of add-on checking, then you must update reverse information.
Then this is have to properties for sender.

Only last MTA need to update, not all clients. Only those MTA have to 
register which make connection outsite from your net. Something same has 
private net - public net, between those you have ex. NAT, some 
controlling, ...,

I have thought something that we have some "routing" services
- ip-packet
- dns lookup
- smtp protocol
- ...

In dns services we have some rules, what is possible, who can/must do 
something.

Maybe we need same kind of rules also to other "routing" methods ?

Most of services is point-to-point services and more simple to make 
rules. You can make it yourself. Smtp is routed services, ofcource last 
step is point-to-point.

IP-network is structured, DNS is structured, smtp is not (yet).

Original doc:
http://www.awot.fi/cgi-bin/textdb/browser/showfile?cust=awkoulutus&subdir=dns&doc=reverse_mx

I wondering, if this kind of simple idea, not so much to do, it so hard 
to do, How we think about those drafts, ideas which are much more 
complicated, need software updates, ... Have those any posibilities to 
be accepted ?

Most comments has handled that "oh no, so much work". Why have more 
problems with our networks ? Hmmm ...

-jukka-

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 28 Nov 2003 05:30:34 -0500

Yakov,

I am seeing things or what?

We have the CallerID Verification system now in place for at least 4-5 days.
This method has drastically reduced the spam via unverified return paths,
but practically 100%.

Am I seeing things?   Why isn't the group homing in on this method which
basically says that the return path must be valid?   I read the pros and
cons in the PDF you provided to me. It didn't make sense to me (the CONS).
As I noted to you via EMAIL,   our system allows for programmable hooks at
each step of the protocol state machine:

    connect -->  connect IP validate hook
    HELO/EHLO ---> helo/ehlo validate hook
    MAIL FROM:  ---> mail from validate hook
    RCPT TO: ---> recipient validation hook
    DATA: ---> mail filter validation hook  (3rd party software hooks)

We offer internal support for RBL at the connect level,  restrict conformity
to HELO/ECHO,  local user validation at RCPT TO: and 3rd party hooks for the
DATA level.   Now we added WCSAP (Wildcat! Sender Authentication Protocol)
to the MAIL FROM: state where the callerid verification takes place.

I added additional logic to make it even stronger.

If the CallerID verification process receives a 250 positive response in the
RCPT TO: return path validation, the logic then goes a step further to check
if the site is an open relay (see if will it accept any random address).

For example, here is a snippet of a WCSAP callerid verification session for
an incoming return path.  This was spawned by our SMTP server once MAIL
FROM: <[email-masked]>  was issued by the remote MTA:

20031128 03:44 00006104 -------------------------------------
20031128 03:44 00006104 version: 1.34 / 1.3 (wcSAP)
20031128 03:44 00006104 uid    : 0
20031128 03:44 00006104 ip     : 64.200.120.224
20031128 03:44 00006104 cdn    : mx224.expts.biz
20031128 03:44 00006104 hdn    :
20031128 03:44 00006104 from   : <[email-masked]>
20031128 03:44 00006104 trying mx: mail.expts.biz ip: 64.119.222.7
20031128 03:44 00006104 # connecting to 64.119.222.7
20031128 03:44 00006104 S: 220 Simple Mail Transfer Service Ready
20031128 03:44 00006104 C: HELO localhost
20031128 03:44 00006104 S: 250 Hello, pleased to meet you.
20031128 03:44 00006104 C: MAIL FROM: <>
20031128 03:44 00006104 S: 250 Sender OK
20031128 03:44 00006104 C: RCPT TO: <[email-masked]>
20031128 03:44 00006104 S: 250 [email-masked]... Recipient
OK.
20031128 03:44 00006104 C: RCPT TO: <[email-masked]>
20031128 03:44 00006104 S: 250 [email-masked]... Recipient OK.
20031128 03:44 00006104 * Open Relay site: 64.119.222.7
20031128 03:44 00006104 C: QUIT
20031128 03:44 00006101 SMTP response code: 552
20031128 03:44 00006104 wcsap finish
20031128 03:44 00006105 -------------------------------------

Notice how we get a positive on the return path and a positive on a random
address.  This means this site is an open relay.  Without it,  the spammer
reaches the RCPT TO: state which in my opinion, is something you wish to
minimize allowing spammers to reach as it helps in thier learning logics.

Also,  I read little about JAMSPAM.

You probably discussed it already, but the technical standards should
indicate and emphasis restrictions to the transports protocol is purely
technical.  EMAIL SYSTEMS invalid handshaking is all the IETF should be
looking at to simplified this process.   To go any further, simply will
complicate things and give JAMSPAM proponents a reason to exist.   Their
layman argument is purely based on the legacy operations of the SMTP system.
Imposed stronger state point handshaking and these people have no basis for
argument other than "social."

Unlike others I read in the group, this is a TECHNICAL problem. Not a social
issue.

The idea of "filtering the data" once the transport system has done its job,
is purely a system level admistrative option and/or feature.  This is akin
to the FTP protocol.  In our system,  we offer a "Scan File" option once a
file is uploaded, but this is an admin/customer issue.  Has nothing to do
with the protocol itself.  The technical process must adhere to standards
otherwise it (the session) is rejected.

You want things to get moving,  get the proposals that concentrate on the
technical aspects of SMTP who do not have any association whatsoever with
mail content or links to "databases" that could make for questionable
non-technical restrictions.   That shouldn't be an SMTP issue.

The bottom line is that once a strict SMTP protocol is put in place, the
SPAMMERS will have no choice but to comply because those are the secured
technical rules.   In my view, this will clean up the spammer problem in
short order.  They must comply from a technical standpoint. Period.  It has
nothing to with SPAM, really.

In any case, we are going to finally WCSAP CallerId Verification in our
Wildcat! mail system and go gamma with it next week to thousands of
customers.   Ironically,  I already even have received a concern from a 3rd
party Anti-Spam author saying "This is going to hurt our sales!"     My
response was simple: "non-conforming anonymous systems into the mail
transport is now going to be a thing of the past.  Mail from conforming
systems will be allowed to pass thru which is where your product comes into
play with its mail filtering techniques optionally offer to the system admin
and for users of the system."

The thing is, unless I am seeing things, with WCSAP, we are blocking nearly
100% of the darn spam!  I have carefully looked at the looks for false
positives.  None!

I'm sure that spammers will eventually "conform" to technical standards. In
my view, from a SMTP standpoint, that is all the IETF and this working group
should care about.

As a mail processor/gateway developer for several networks over the last 25
years, I am firm believer in unaltered passthru of mail.  In my view, it is
unethical to a) restrict mail reaching its destination and b) modify it in
any way.  However, that has nothing to do with technical conformity.  The
handshaking process should be an identified process on both ends.

On a similar note,  the RMX/DNS solutions, etc, are essentially the same
idea, but it requires DNS administration which in my view is going to be
problematic in all aspects, technically, politically, including
capitalistically (or communistically depending on your point of view).
Because of this, it will be slow to catch on.

However, one way to help the process is to include as part of the SMTP
specification, a DNS registration process.  This will force (encourage)
developers to include a "registration" process as part of their SMTP
software installation or setup process.  This registration process can
include other attributes, besides RMX related, such as "type of system."  Is
it open, closed (private) system?  Does the system offer open spam,
restricted spam, or no spam?  etc.

Of course, this extended attribute concept be implemented using EHLO,
however, if we are going to require a modified MX DNS system, then we might
as well add other attributes to it as well.  A conforming client can use
this information and help save at least 1/2 of the bandwidth. A client can
also instantly inform a user or sender that the communication with a target
system will be restricted in one form or another.

Nonetheless, compared to a CallerID Verification concept,  the extended DNS
based solutions is going to be  a slow and costly endeavor. In my view,
Return Path callerid ID handshaking seems to be a very logical extension to
the SMTP client/server handshaking process.  In the current form of SMTP, it
works fantastic.   If we anticipate this is going to be a widely accepted
practice for all developers (big, including small, a major plus reason why
it will be accepted faster than a DNS based solution), then maybe we should
look at reducing handshaking requirements to accomplish a callerid
verification SMTP logic.

Anyway, what at the flaws in a CallerId Verification that I am not seeing?

Hector Santos, CTO
WINSERVER "Wildcat! Interactive Net Server"
support: http://www.winserver.com
sales: http://www.santronics.com

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 28 Nov 2003 06:57:00 -0500

Incidentally,  WCSAP is written using our wcBASIC ;anguage CLR system.  Our
Wildcat! SMTP server calls the WCSAP p-code image which is also accessible
as a WEB service from our Wildcat! web server.

I made it available for demonstation:

http://www.winserver.com/public/code/html-wcsap?&xml=1&from=[email-masked]

Just make sure XML=1 is set to get XML/SOAP output.

Other options:

&debug=1     - text output (essentially the logger dupes the output to the
virtual console)

---
Hector Santos/CTO

---

From: "Jon Kyme" <[email-masked]>
Date: Fri, 28 Nov 2003 14:47:10 +0000

> http://www.winserver.com/public/code/html-wcsap?&xml=1&from=
> 

Yow - that's your box blacked by AOL, when someone hits it a zillion times
as a joke.
I hope it doesn't need to send any real mail.

Incidentally, this is often called "Sender Callback Verification" and is
quite widely used - it's built into many MTAs (such as exim).

Or am I missing something?

--

---

From: "Alan DeKok" <[email-masked]>
Date: Fri, 28 Nov 2003 11:03:11 -0500

"Hector Santos" <[email-masked]> wrote:
> basically says that the return path must be valid?

  As noted in another message, a number of people *are* using this
method.

  The problem with it is that it's a band-aid.  The purpose of ASRG is
to see if there are other methods which will reduce the spam problem.
e.g. Reducing the amount of spam sent, which is very different than
reducing the amount of spam received.

  Alan DeKok.

---

From: Yakov Shafranovich <[email-masked]>
Date: Fri, 28 Nov 2003 12:59:34 -0500

> This method has drastically reduced the spam via unverified return paths,

The method you are refering is similar to what is more commonly known as 
"Challenge/Response" (C/R). Majority of C/R implementation actually send 
a message back to the sender asking him/her to click on a link to reply 
to the message in order to verify that the sender exists. Some go 
further and use reverse-Turing tests with images that supposedly only 
humans and not machines can read.

Your particular method does not actually send the email to the sender, 
but stops as soon as the RCPT TO: path has been found to be valid which 
varies from the C/R systems.

> cons in the PDF you provided to me. It didn't make sense to me (the CONS).

The simple reason why C/R works is because spammers don't care. At this 
point a very small minority of email systems in the world use any kind 
of C/R, and therefore spammers do not bother spending the time to fight 
them. Some C/R companies such as MailBlocks, etc. report virtually 99% 
success rate. However, if C/R usage spreads, then spammers will have an 
incentive to attack them, and they will. Also, spammers tend to share 
their results much easier then some of the anti-spam people.

There are problems with the the regular C/R approaches that apply here 
as well:
1. Nothing stops a spammer from using someone else's email address with 
that domain answering yes (unless you start matching the his email 
address with his IP which is what the LMAP proposals do). Especially 
this would be true if he uses YOUR domain. Also, as per RFC 2822 you 
must support MAIL FROM: <> which basically allows spammers to bypass 
everything at this step.
2. Even checking MAIL FROM does not prevent spammers from forging the 
"From" header inside the actual email message.

While there are numerous problems with regular C/R, your particular 
approach has some issues as well:
1. Nothing stops a spammer from using his own domain which answers "yes" 
to all RCPT TOs to that domain (BUT not an open relay).
2. ISPs tend to play around with RCPT TO: handling such as answering 
"yes" to all requests, or delaying responses in order to make sure 
spammers do not use RCPT TO: for harvesting email addresses.
3. Since your proposal operates at the MAIL FROM step, if the sender's 
receiving MTA is off-line this process will fail. This will also be a 
problem if the sender is using GreyListing 
(http://projects.puremagic.com/greylisting/). According to RFC 2821, 
section 4.5.4.1., the originator must be able to resend email several 
times when temporary failure occurs, which in practice happens sometimes.

>     connect -->  connect IP validate hook

Would this be an rDNS / RBL hook?

>     HELO/EHLO ---> helo/ehlo validate hook

What kind of hook are you using here? If a spammer uses a valid domain 
here, nothing stops him. Only something like LMAP/DRIP that ties IP 
addresses with domain names for SMTP can help against that.

>     MAIL FROM:  ---> mail from validate hook

What kind of hook aside from this proposal are you using here?

> if the site is an open relay (see if will it accept any random address).

What happens with 4xx codes? or if the MTA is off-line? In regular C/R, 
the challenge message will be resent multiple times as per RFC 2821, 
section 4.5.4.1 an that avoids the problems mentioned above.

[...]
> Also,  I read little about JAMSPAM.

JamSpam seems to be currently dead. The folks involved could not agree 
on anything so it fell apart.

> argument other than "social."

What kind of controls are we talking about? Something like LMAP?

> otherwise it (the session) is rejected.

RFC 2821 states  (sections 3.3, 4.1.1.4 and 6.1) that the SMTP server 
has two options: accept or reject the message, and has two options to 
send back OK or Failure (5xx). HOWEVER, once the message has been 
accepted as per this section (6.1):

"  When the receiver-SMTP accepts a piece of mail (by sending a "250 OK"
    message in response to DATA), it is accepting responsibility for
    delivering or relaying the message.  It must take this responsibility
    seriously.  It MUST NOT lose the message for frivolous reasons, such
    as because the host later crashes or because of a predictable
    resource shortage.

    If there is a delivery failure after acceptance of a message, the
    receiver-SMTP MUST formulate and mail a notification message.  This
    notification MUST be sent using a null ("<>") reverse path in the
    envelope.  The recipient of this notification MUST be the address
    from the envelope return path (or the Return-Path: line).  However,
    if this address is null ("<>"), the receiver-SMTP MUST NOT send a
    notification.  Obviously, nothing in this section can or should
    prohibit local decisions (i.e., as part of the same system
    environment as the receiver-SMTP) to log or otherwise transmit
    information about null address events locally if that is desired.  If
    the address is an explicit source route, it MUST be stripped down to
    its final hop.
"

Therefore, the RFC does dictate what should been done with the message 
once it has been filtered. Even though many anti-spam systems honor this 
"in breach", this is another illustration of how protocols affect other 
things, even filtering.

> nothing to with SPAM, really.

It will only work if most of the mail systems on the Internet comply 
with it. The IETF cannot force compliance with its standards. Therefore, 
partly what we are seeking is an incremental solution as stated in the 
technical considerations document 
(http://www.ietf.org/internet-drafts/draft-crocker-spam-techconsider-02.txt):

"    By contrast, some "edge" mechanisms provide utility to
      the first one, two or three adopters who interact with
      each other. No one else is needed for the adopters to
      gain some benefit. Each additional adopter makes the
      total system incrementally more useful. For example a
      filter can be useful to the first recipient to adopt
      it. A consent mechanism can be useful to the first two
      or three adopters, depending upon the design of the
      mechanism.
"

We are open to hearing how the existing SMTP protocol may be tightened 
but its must be proven that this will solve the problem.

> and for users of the system."

I am sure that you will allow for more than "4-5 days of testing" before 
deploying this.

> 100% of the darn spam!  I have carefully looked at the looks for false

IF you are rejecting email at the MAIL FROM stage, there is now way to 
check for false positives since you do not have the actual email 
message. Additionally, your specific test installation might not be 
large enough to extrapolate to the entire Internet. If you can provide 
additional statistical information, that would be useful.

> my view, from a SMTP standpoint, that is all the IETF and this working group

Spammers do not care - they will only conform to standards if they are 
forced to do so in order to deliver their garbage. Therefore, in order 
for any standard to work, existing email systems on the Internet must be 
willing to adapt them.

> Because of this, it will be slow to catch on.

Nevertheless, any existing methods do not provide a way during 
handshaking to tie in IP addresses to domain names for SMTP. This is 
something that must be introduced in order to verify senders, or some 
other method.

> look at reducing handshaking requirements to accomplish a callerid

The following sections from the technical considerations document are 
relevant:

"    The idea of replacing SMTP is appealing because it
      permits thinking in terms of creating an infrastructure
      that has accountability and restrictions built in.
      Unfortunately an installed base the size of the
      Internet is not likely to make such a change anytime
      soon.  It seems far more likely that successful spam
      control mechanisms will be introduced as increments to
      the existing Internet mail service.
"

This statement would apply to CHANGING the SMTP protocol as well. Any 
major changes would have the same deployment issues. See also the following:

"4.1. Adoption

      A critical barrier to the success of a new mechanism is
      the effort it takes to begin using it. It is essential
      to look carefully at the adoption process.

           1) Adoption    What is the effort for a new
           Effort         participant to start using the
                          proposed mechanism? This includes
                          installation, learning to use it
                          and performing initial
                          operations. This is also called
                          the "barrier to entry".

           2) Threshold   What is required before users get
           to benefit     some benefit from the mechanism?
                          Primarily, this looks for the
                          number of users who must adopt
                          the mechanism before the adopters
                          gain utility from it.

      A key construct to examination of adoption and benefit
      is "core-vs-edge".  Generally, adoption at the edge of
      a system is easier and quicker than adoption in the
      core. If a mechanism affects the core (infrastructure)
      then it usually must be adopted by most or all of the
      infrastructure before it provides meaningful utility.
      In something the scale of the Internet, it can take
      decades to reach that level of adoption, if it ever
      does.

      Remember that the Internet comprises a massive number
      of independent administrations, each with their own
      politics and funding. What is important and feasible to
      one might be neither to another. If the latter
      administration is in the handling path for a message,
      then it will not have implemented the necessary control
      mechanism. Worse, it well might not be possible to
      change this.  For example a proposal that requires a
      brand new mail service is not likely to gain much
      traction.

      By contrast, some "edge" mechanisms provide utility to
      the first one, two or three adopters who interact with
      each other. No one else is needed for the adopters to
      gain some benefit. Each additional adopter makes the
      total system incrementally more useful. For example a
      filter can be useful to the first recipient to adopt
      it. A consent mechanism can be useful to the first two
      or three adopters, depending upon the design of the
      mechanism.

           3) Impact on   What is the impact on the senders
           Participants   and receivers who adopt the
                          proposal? Senders and receivers
                          currently have certain styles of
                          operation.  How are those styles
                          changed?

           4) Impact on   What is the impact on the senders
           Others         and receivers who do *not* adopt
                          the proposal? What effect does it
                          have on legitimate users of
                          email? What effect does it have
                          on spammers? Is the nature of
                          Internet mail changed for
                          everyone, including non-
                          adopters?

      For example, a challenge-response system is irritating
      for the person being challenged, and it imposes extra
      delay on the desired communication. If the originator
      and the recipient both access the Internet only
      occasionally (such as through dial-up when mobile) a
      challenge-response model can impose days of delay. For
      some communications, this can be disastrous.
"

-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
"And this too shall come to pass"
-------

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 28 Nov 2003 15:34:40 -0500

----- Original Message ----- 
From: "Yakov Shafranovich" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Sent: Friday, November 28, 2003 1:03 PM
Subject: Re: [Asrg] 0. General - Inquiry about CallerID Verification
>
> Here is the illustration of why this method DOES NOT work:
>
>
http://www.winserver.com/public/code/html-wcsap?&xml=1&from=[email-masked]

This doesn't mean it DOES NOT work. It does work.   It has pruned a majority
of the false entries. Let me check my personal mail today......  3 out of
over 929 attempts got in since going to bed this morning and getting up now.
That is a 99.7% success rate!

> Note that [email-masked] surely does
> not exist. Nevertheless, Yahoo answers "250" to all RCPT TOs in order to
> avoid harvesting.

YAHOO.COM,  AOL.OCM and others accepts "billions" of mail therefore might
deem it necessary to delay any mail validation process.  Again, it does not
mean it doesn't work.    It means, these guys are the PROBLEM, but anyone in
the mail server business knows that.   You are comparing it against the
AOL.COM, YAHOO.COM, etc.  Yet, in the same breath,  I read how these guys
are trying to define or hijack your efforts.

---
Hector Santos

---

From: Yakov Shafranovich <[email-masked]>
Date: Fri, 28 Nov 2003 15:39:36 -0500

> are trying to define or hijack your efforts.

There is a reason to their madness - it discourages spammers from 
harvesting email addresses. Also, nothing prevents the spammers from 
doing the same thing for their domains.

As for their "hijacking efforts", we are in touch with them about it.

Yakov
-------

---

From: Bill Cole <[email-masked]>
Date: Fri, 28 Nov 2003 15:57:06 -0500

>doing the same thing for their domains.

Such behavior is also not restricted to large consumer/recreational 
ISP-like mail systems. Many corporate mail systems have exterior SMTP 
faces that live on machines with no knowledge of the interior 
environment except how to get mail to it. The outside bastion mail 
servers deal with basic filtering, but have no way to know whether a 
particular address is valid or not. This is a simple security issue: 
machines with exterior exposure are not allowed to carry or even 
query confidential databases like the list of valid user names. 
There are arguments on all sides of this sort of configuration, but 
no amount of recommendation by an I[RE]TF body will change it.

-- 
Bill Cole                                  
[email-masked]

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 28 Nov 2003 17:15:25 -0500

----- Original Message ----- 
From: "Yakov Shafranovich" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "ASRG" <[email-masked]>

> to the message in order to verify that the sender exists.

Not the same thing.   WCSAP checks the MAIL FROM: state point.    C/R
systems go to the RCPT TO: statement, and in some cases complete the
session.   In our early exploration of this almost two years
ago, it immediate apparent that it doesn't belong in the SMTP protocol, it
is intrusion for our business customers and it doesn't quite work to solve
the spammer problem.  It is easily broken with a little bit of automation.

> this would be true if he uses YOUR domain.

Our mail system does (optionally) check the return path domain against the
connection IP.  But more importantly, this puts the spammer into another
category, namely, outright fraud.

> Also, as per RFC 2822 you

True, but that puts it into another category handle by other technical SMTP
server designs.

> "From" header inside the actual email message.

Doesn't apply.  Again, the goal is no minimize reaching the RCPT TO: /DATA
state in the SMTP state machine.

> spammers do not use RCPT TO: for harvesting email addresses.

Again,  this is to verify the existence of the domain which is a good bit of
the problem, and in addition, helps address the fraud.

> times when temporary failure occurs, which in practice happens sometimes.

Handled with a  4xx response when the connect and/or welcome response
indicates so.

> Would this be an rDNS / RBL hook?

Yes.  As it is currently implemented, plus admins White/Black IP lists.

> addresses with domain names for SMTP can help against that.

As currently implemented, our server offers strict HELO requirements (for
intranets), but internally the server watches for fake domains, badly
formatted dot IP addresses, etc.   This has help eliminate a large
percentage of the illegal entry attempts.  Here is a log snippet:

**************************************************************************
Wildcat! SMTP Server v5.7.450.9b10
SMTP log started at Fri, 28 Nov 2003  01:21:31
Connection Time  : 20031128 01:21:30
Client IP Address: 24.92.200.21 (unknown)
SSL Enabled: NO
01:21:31 S: 220 winserver.com Wildcat! ESMTP Server v5.7.450.9b10 ready
01:21:31 C: HELO 208.247.131.9
01:21:31 S: 501 Invalid HELO client address.

Since implementing a HELO/EHLO check, we have noticed a drastic redundant of
certain type of spammers stop connecting to our system.   Their "software"
does not work here.

> What kind of hook aside from this proposal are you using here?

In our SMTP server design, just wants a response from an external source.
Currentlty, we have optional strong MAIL FROM: requirements (for intranets).

We implemented the WCSAP hook.   And I recently contacted you to see if how
fast I can test our a "wcRMX" hook.  But that C code was not "API" or coder
ready.  I started a small effort to make it useable as a API but then said,
"What about the DNS database?"    It am sure it will work, but this presents
an entirely different non-dynamic "administrative" non-technical issue.

>
> What happens with 4xx codes? or if the MTA is off-line?

Supported.   4xx is used for MX and/or MTA connection failures. WCSAP also
includes a "TryMXOnce" where per the RFC if there is no MX, it tries the
DOMAIN IP.  Just a standard mail sender.

> on anything so it fell apart.

Maybe so, but I am not too sure if they did not accomplish the goal of
allowing a legitimizing a classification of spam to be unrestricted at the
system level.   This "David" guy did attend the congressional hearings.  Was
IETG represented?

I have not read the ANTI-SPAM congress bills yet,  but from what I
understand so far,  it sounds like it may have opened up the opportunity for
spammers to claim tortious interference against an ISP or for that matter,
mail software vendor.   This is why I believe we need to make sure the SMTP
PROTOCOL and its "access" methodologies are purely technical in mind based
on security aspects of the system.  The SMTP protocol should have NO
business in the analysis of mail content, with the exception of possibility
checking headers for technical conformity.

> > indicate and emphasize restrictions to the transports protocol is purely

Any proposal that is based on authorizing the secured handshaking between
the client and server BEFORE the RCPT TO: is reached.

> RFC 2821 states  (sections 3.3, 4.1.1.4 and 6.1) that the SMTP server ....

I clearly understand this.

It doesn't apply.   This is after the client/server handshaking has been
negotiated.

> with it. The IETF cannot force compliance with its standards.

Right, but the market will.  Any current vendor (big and small) who is
serious about their mail server functionality in address the "spam problem"
will do something about it, especially the smaller vendors as it see it as a
market opportunity agains the bigger legacy systems.   Its a common trait in
many industries where change is first followed by newer or smaller vendors.

> Therefore, partly what we are seeking is an incremental solution as stated
in the
> technical considerations document

In understand and I agree.  This is why I like the RFC changes be made to
address stronger client/server negotiations at the TECHICAL level
elimination any notion that the protocol is in the business of many
"questionable" and "debateble"  logic towards the mail content.

In my view, if IETG does not do this,  you will force the "vendors" to move
towards a central authority entity concept, aka what the Verisign people are
proposing.

> but its must be proven that this will solve the problem.

I think you got all the technical proposals already.   Again, this is a
technical issue of resolving how a CLIENT and SERVER talk to each other.
You need to borrow technology has been in existence and worked in the past.
Such as Fidonet.  Randy Bush (if this is the same person that was involved
with Fidonet) clearly knows how this system does work, yet, at the same
time, understands there are administrative nightmares when trying to
maintain a "membership database."  This is probably why he is resistance to
DNSSEC and to a large degree why I don't particularity like the
ramifications of having a DNS membership database controlling access.

However, it can help the process to if this DNS database includes attributes
that augment the client/server negotiation process.   For example:

Type of system:  OPEN, CLOSE, SEMI-CLODE
Type of mail:  SpamAllowed, Commerical Spam,  No Spam
Hours:  0-24 SPAM: 1-3

etc, etc.   Use the opportunity to not just address SPAM but other technical
issues the current mail transport specs do not address.  Hours of operation
will HELP a growiing amount of the "small or home busines" operations with
limited bandwidth or non-full time mail operations.     In Fidonet, there
was a few requirements. One was a ZONE HOUR where all operations must be
open for mail.   We can extend this logic to include a "BOUNCE HOUR" or
"SPAM HOURS", etc.

In my view, a proposal that is based on a DNS modification will benefit
(more plausible) from extending the functionality of the system, not just ad
dresing spam.

> I am sure that you will allow for more than "4-5 days of testing" before

We are going gamma sometime next week.   Nothing better than getting excited
people to test it in real operations.   Obviously, between you, I and the
fencepost, the first wave will be "more beta."  <g>

> message.

Doesn't apply.  Again, I am from the SCHOOL that SMTP has no business in
analyzing mail content.  That is what is going to get you in trouble and
FEED the legal eagles.

The goal is CONFORMITY of the client/server session.   If a MAIL FROM:
FAILS, the client has no business going any futher.   If the MAIL FROM:
succeeds, then WCSAP has done its part and it up to the next stages, if any,
to do checking.

This is why the RFC needs to concentrate their efforts in the first 3
states:

    connect
    HELO/EHLO
    MAIL FROM:

The problems with AOL, YAHOO, etc with allowing all RCPT TO: is because of
the dearth of control at the above states.   Solve this and the major issues
will be addressed.   Then we might see the YAHOO, AOL, etc comply.

> Additionally, your specific test installation might not be

I got over 6 months of logs and collected mail that was accepted or filtered
at the DATA stage.  Most of the false positives exist at the DATA stage
because of the external mail filtering rules in use.  3rd party software
such as wcSpamGuard for the WIldcat System (http://www.wssoft.net)  hook
into the DATA stage and provide a suite of mail filtering methods for the
system and for users.

>       the existing Internet mail service.

Yet to borrow a quote:

    "The Internet was:  small, friendly, open, casual.
      The internet has become: Huge, Competitive, Closed/Open,
Casual/Formal."

                               An unaffiliated view of Electronic Commerce,
                                                                        Dave
Crocker,  1995

So you see from this that Mr. Crocker fully understood back in 1995 where we
were heading.  Corporations have ditched their NOVEL or X.400 system, or
Lotus Notes   system, etc.  The same vendors, including ourselves have
integrated internet technology into their software.

There is no REASON to believe it can't happen again.

Look, I have only one reason why we began efforts to add anti-spam solutions
into our system, my customers!  They are the ones who want a solution or one
that makes it less of a problem.  If I don't provide them, they LOOK else
where. Its that simple.   I am not going to wait.

Hector Santos

----

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 28 Nov 2003 17:41:08 -0500

----- Original Message ----- 
From: "Alan DeKok" <[email-masked]>
To: <[email-masked]>
Sent: Friday, November 28, 2003 11:03 AM
Subject: Re: [Asrg] 0. General - Inquiry about CallerID Verification

> As noted in another message, a number of people *are* using this

Yes, I understood this. Never claim it to be the first.  But we just
implemented and in my expert opinion, it is the best solution out there
today to eliminate a majority of the spam, yet at the same time, first into
the protocol and doesn't required a change the DNS records.

However, I clearly understand how having "extra" information at the DNS
level will help the process.  But that isn't there yet.

>  The problem with it is that it's a band-aid.

Most software solutions are "band-aids."   I don't consider this a BAND-AID
because it is LOGIC that naturally and logically fits into MAIL FROM: state
point in the SMTP protocol state machine.

Is it 100%?  No.  It is not, for the obvious and stated reasons.

> The purpose of ASRG is to see if there are other methods which will
> reduce the spam problem.  e.g. Reducing the amount of spam sent, which is
very different than
> reducing the amount of spam received.

ASRG should change their emphasis and concentrate on REDUCING unauthorized
MTA at the protocol level.   SPAM is only a symptom of the dearth of such
controls.  e.g.   The Caller ID methods prove that MAIL FROM: (and
HELO/EHLO) validation is (should be) a SMTP functional protocol (optional)
requirement.  How it is done is a technical and implementation issue.

In other words, there is NOTHING that should be stopping the IETF from
modifying the RFC to say:

    x.x.x  HELO/EHLO

    The client host domain may be subject to mail server implementation
validated with the following
    response codes:

    x.x.x  MAIL FROM:

    The client return path may be subject to mail server implementation
validated with the following
    response codes:

it has nothing to do with the technical implementations or how a mail server
will accomplish this.

On the other hand, if the ASRG/IETF is looking to add some "details" to the
specification:

    x.x.x  MAIL FROM:

    The client return path may be subject to mail server implementation
validated with the following
    response codes:

    ....

    The following RFC methods are recommended approaches.

then even better.  But Yakov has said, you can't force everyone to use any
particular method.  Others may find a better method or something that is
more internal to their operations.   The goal is that it should be a
functionally defined in the specification to help designers do a better job,
and in addition help eliminate any future LEGAL issues brought on by
spammers.  For systems to reject connecting MTA, it has to be because of
lack of technical compliance.

---
Hector Santos

---

From: Markus Stumpf <[email-masked]>
Date: Sat, 29 Nov 2003 03:36:42 +0100

> infrastructure, nobody uses it today except for debugging. 

This is not true.

> thousand email domains parked on one machine.

Did you read the MTAmark proposal at all?
Your above statement is neither a problem nor is it in any way related to
the MTAmark propsal. With MTAmark you "mark MTAs" as being such. If
there is a MTA running on the machine with several hundred thousand
email domains parked and the MTA should be a Internet wide sending MTA
it is marked as such. If the machine is only a webserver that should not
send out emails it is marked "MTA=no". It is irrelevant whether there is
one domain associated with that machine or 10000000.
MTAmark is a way to tell other MTAs "accept email from that IP" or "do
not accept email from that IP". This is the only criterion and given the
zillions of hacked, 0wned, misconfigured and virus contaminated machines
out there it is IMHO an important one.

	\Maex

-- 
SpaceNet AG            | Joseph-Dollinger-Bogen 14 | Fon: +49 (89) 32356-0
Research & Development |       D-80807 Muenchen    | Fax: +49 (89) 32356-299
"The security, stability and reliability of a computer system is reciprocally
 proportional to the amount of vacuity between the ears of the admin"

---

From: "Jon Kyme" <[email-masked]>
Date: Sat, 29 Nov 2003 09:10:32 +0000

>     delivering or relaying the message.  It must take this responsibility

[snip]

> "in breach", this is another illustration of how protocols affect other 

Absolutely, but of course it's not clear that any recipient or
administratively authorised disposition of the message can count as
"delivery failure". I'm not sure that silent failures due to filtering
*are* contrary to this recommendation.
Some say it's better for rejection to be earlier and explicit, but
successfully delivering to /dev/null isn't prohibited anywhere I can think
of.

--

---

From: "Jon Kyme" <[email-masked]>
Date: Sat, 29 Nov 2003 09:35:43 +0000

> no amount of recommendation by an I[RE]TF body will change it.

One of the arguments against it is that it's just plain dumb. It doesn't of
itself stop the confidential data leaking (I can send you a bunch of
messages and figure it out from the bounces), but it does make you have to
handle bounces to forged addresses in your domains (since remote systems
can't do sender verification against you). And of course you've got all the
spam addressed to former employees (etc) eating up disk and CPU. But like
you say, it's difficult to change the perception that
obscurity -> security.

--

---

From: Matthew Elvey <[email-masked]>
Date: Sat, 29 Nov 2003 11:41:00 -0800

On 11/28/2003 12:39 PM, Yakov Shafranovich sent forth electrons to convey:

> harvesting email addresses.

It's a poor excuse.  There are ISPs that detect harvesting attacks and 
when they do, deal with it - some then refuse even email to valid 
addresses.  They can refuse the email without stating or otherwise 
giving away the information that the reason for refusal is email to a 
nonexistent account. IMO, MAIL FROM: <> needs to be deprecated in favor 
of refusal during SMTP.  There is no compelling reason to delay the mail 
validation process.  It provides no benefit that cannot be met with 
immediate validation, barring the super-secure bastion host scenario 
Bill Cole mentions, which could be resolved with some work. (I should 
admit - I don't do what I preach, as the provider I choose to have host 
my domain does accept spam that I'd prefer it refuse during SMTP, but I 
have pushed them to fix the problem, and they do refuse a good fraction 
of the spam during SMTP.)

I forget whether SpamAssassin supports what we're currently calling 
CallerID Verification.  It has NO_DNS_FOR_FROM: Domain in From header 
has no MX or A DNS records, but I think it doesn't have the full 
CallerID Verification.

---

From: Yakov Shafranovich <[email-masked]>
Date: Sat, 29 Nov 2003 19:07:28 -0500

> category, namely, outright fraud.

There is nothing in the IETF standards that requires the return path 
domain to be the same as the IP. As a matter of fact, many systems use 
outside MTAs to deliver their email, so such check can have many false 
positives. The only way to check for this is to utilize some kind of 
LMAP or similar proposal, such as digital keys (MSFT's).

Additionally, spammers are *fraudsters* and this is the most common 
occurance - using someone else's domain as return address and someone 
else 0wned machine for sending the spam!

> system level.   This "David" guy did attend the congressional hearings.  Was

Nope, the IETF or IETF was not present.

> succeeds, then WCSAP has done its part and it up to the next stages, if any,

You still have not answered the question - how do you know there are no 
false positives? What method did you use to figure that out? And also, 
what is your definition of a "false positive"?

Yakov

-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
"Power tends to corrupt, and absolute power corrupts absolutely" (Lord 
Acton)
-------

---

From: Bill Cole <[email-masked]>
Date: Sat, 29 Nov 2003 20:32:30 -0500

At 11:41 AM -0800 11/29/03, Matthew Elvey wrote:
>It's a poor excuse.  There are ISPs that detect harvesting attacks 
>and when they do, deal with it - some then refuse even email to 
>valid addresses.  They can refuse the email without stating or 
>otherwise giving away the information that the reason for refusal is 
>email to a nonexistent account. IMO, MAIL FROM: <> needs to be 
>deprecated in favor of refusal during SMTP.

The null sender for system-generated messages about system-level 
issues should NOT be deprecated without providing some solution for 
everything that it is currently useful for. Bounces are just one 
variation on the broader theme of DSN's.

>  There is no compelling reason to delay the mail validation process.

Whether that is true really depends on how narrowly you want to view 
the email world. If you want to ordain that all mailable domains must 
have the full knowledge of their namespace in every host that they 
have an MX record pointing at, this is a very tall order for changes 
from how a significant fraction of the net works today.

>   It provides no benefit that cannot be met with immediate 
>validation, barring the super-secure bastion host scenario Bill Cole 
>mentions, which could be resolved with some work. (I should admit - 
>I don't do what I preach, as the provider I choose to have host my 
>domain does accept spam that I'd prefer it refuse during SMTP, but I 
>have pushed them to fix the problem, and they do refuse a good 
>fraction of the spam during SMTP.)

That example was ONE case. Without delaying the validation of the 
target address, every domain in which there are mail addresses would 
need published MX records pointing to machines with live access to 
the authoritative knowledge store for each domain it serves. We could 
kiss UUCP goodbye finally, and if you think no one still uses it you 
are quite wrong. Large organizations that now have single external 
faces on their mail systems for many possible subdomains which 
actually route internally in complex ways would need to be 
redesigned. As an example of this, I work with a major systems vendor 
with a global workforce whose individual email addresses reflect 
their geographic location, yet the exterior MX record set for all of 
the subdomains is one server farm. That server farm has to know how 
to route mail to many different subdomains handled on many different 
machines around the world. It DOES NOT have access to the full user 
namespace for all of those domains, it merely knows that it accepts 
mail for the various subdomains and hands it off internally to 
machines closer to the HR departments for the various branches. Even 
without exposed subdomains, such situations are problematic because 
companies beyond a certain size simply do not operate as single 
entities and cannot operate their email systems on single giant 
machines that know everything simultaneously. The fact of an address 
no longer being valid can easily take a day to propagate throughout a 
global operation, and it is a rare large company that can make the 
event of a user add or delete look instantaneous to their entire 
email system.

>CallerID Verification.  It has NO_DNS_FOR_FROM: Domain in From 
>header has no MX or A DNS records, but I think it doesn't have the 
>full CallerID Verification.

I admit to having only skimmed the stuff on 'CallerID' because it 
looks like the same unscaleable stuff that has been in in some MTA's 
for a while with essentially tries a RCPT verification of every MAIL 
FROM given, and there are probably enough of us managing mail servers 
who will address that sort of behavior as just another attack (null 
route the offending machine) so that the inherent scaling and 
indirect attack problems don't ever make it to visibility.

That said, the practice of a DNS check on any non-null MAIL FROM 
argument is reasonable and done by many modern MTA's as a basic bar 
to the most clueless spammers. Doing the same on From lines is a 
little more problematic, but it is not completely unreasonable.

-- 
Bill Cole                                  
[email-masked]

---

From: "Hector Santos" <[email-masked]>
Date: Sat, 29 Nov 2003 22:31:16 -0500

----- Original Message ----- 
From: "Matthew Elvey" <[email-masked]>
To: <[email-masked]>
Sent: Saturday, November 29, 2003 2:41 PM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

> > Yakov wrote:
> nonexistent account.

Matthew,

I agree.  sites such as Yahoo who allow all RCPT TO: are in my view, for all
intent open relay sites. It encourages and promotes the usages of obsfucated
email addresses across the board to all systems. In other words, it makes
the matter worst and adds overhead across the board to all systems.

There is another possible side to this, and that is to "collect" data to
perform anti-spam research.  I know that is what I did for a short period
where we accepted all mail in order to collect data and research the
problem.

>IMO, MAIL FROM: <> needs to be deprecated in favor

I would believe it does not.  I am only  saying this because an active 3rd
party anti-spam author who incorporates SpamAssasin in our beta team. His
only comment to me was that he was afraid we will taking away some of his
future business. :-)   We don't try to step of the toes of 3rd party
developers writing for our package. We normally concentrate providing the
API and HOOKS functionality into our client/server system as illustrated
here:

            http://www.winserver.com/public/antispam/AntiVirusSpam.htm

Our current release only offers the HOOK into the DATA state point.  We
quickly learned that while this offered anti-spam (and other special
processing capabilities),  from a anti-spam standpoint, it might have "help"
spammers because they have the most treasured state point, the "RCPT TO:"
stage to gain informaiton.  So from a anti-spam standpoint, the DATA hook
helps, but it might makes matters worst by increasing the frequency of spam.

But as with everyone else, the spam problem because so pervasive especially
with SOBIG exploiting everything that is wrong about the legacy SMTP
protocol, we simply could not wait for anyone to solve the problem at more
detail levels.   I have to admit I naively thought it was a waste of time as
eventually the issue would be solved using the new crime laws for spam.
After all, using your address for the MAIL FROM: is fraud.

In any case,  in the new current anti-spam design,  the design criteria is
based on SMTP technical compliance. Each SMTP command will a hook for
extended logic.  We will inherently supply  technical compliance logic with
the flexibility to allow future improvements by 3rd party vendors or an
official IETF method is established.

For now, I believe the Caller ID verification is the best method to prune
the majority of illegal entry into the system.

I have been saving our logs for the last 7 months.  I finally finish a SMTP
LOG statistics reporter for our system.   Check out some of these results:

    http://www.winserver.com/public/antispam/wcsmtpstats.htm

See my next message on this where I provide some analysis and possible new
SMTP suggestions.

---
Hector Santos, CTO

---

From: "Hector Santos" <[email-masked]>
Date: Sat, 29 Nov 2003 22:49:39 -0500

> false positives? > What method did you use to figure that out?

Sure I have. I said I have analyzing the logs.  I have been doing random
double checks where it might seen questionable.

> And also,  what is your definition of a "false positive"?

As a mail transport/gateway designer,   I am basing the design on technical
comformity.  Hence,  a false positiive is restricting what is otherwise a
legit client connecting into the system to deliver mail.

The WCSAP tuning process will be on going learning the behavior early day,
adding new administrative features, little bit of AI into it, etc, etc, etc.

Also, it isn't that hard.  If there was a REAL legit system or message not
getting here, we hear about it soon enough.

You should concentrate your efforts on technical conformity of the specs.
No solution will solve the spammer problem 100% and you will be beating a
dead horse trying to look for one.  In my view, your only option is to make
conformity in the future SMTP functional specification a requirement with
server-implementation backward compatibility.  That has nothing to do with
DNS, Mail Filters, SPAM vs NO SPAM, BULK vs NON-BULK,  non-server vs server
based MTA clients, etc.

---
Hector Santos, CTO

---

From: "Eric S. Raymond" <[email-masked]>
Date: Sat, 29 Nov 2003 23:05:24 -0500

> DNS, Mail Filters, SPAM vs NO SPAM, BULK vs NON-BULK,  non-server vs server

I think Hector has a point here.  Not that we should give up other approaches,
but that there are ways we can make life more difficult for spammers by
requiring strict technical conformity and failing closed rather than open.

This is is a viable approach because there are so many fewer semding servers
(concentrated at ISPs) than receiving servers.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

---

From: Bart Schaefer <[email-masked]>
Date: Sun, 30 Nov 2003 04:18:52 +0000

On Nov 28,  5:15pm, Hector Santos wrote:
} 
} ----- Original Message ----- 
} From: "Yakov Shafranovich" <[email-masked]>
} 
} > The method you are refering is similar to what is more commonly known as
} > "Challenge/Response" (C/R).
} 
} Not the same thing.  WCSAP checks the MAIL FROM: state point.  C/R
} systems go to the RCPT TO: statement, and in some cases complete the
} session.

It does share an important "feature" of C/R systems, though:  It's cost-
shifting.  If someone (or some worm) sends you 1000 messages with forged
senders in the brasslantern.com domain, you're going to hit my MX host
1000 times to refute them all.  Why is it *my* responsibility to expend
bandwidth and server cycles to help you reject the spam run?  How is what
you're doing to my server any less an abuse of resources than what the
spammer is doing to you?

Suppose a large ISP were to adopt the "caller ID" scheme (one already may
have, see below).  A spammer forges winserver.com MAIL FROM: on a couple of
million messages destined for that ISP, distributed across all its dozens
of MXs.  They all begin connecting to your MX to ID the caller.  Are you
prepared to handle that load, or have you just been DoS'd into oblivion?

On Nov 29,  8:32pm, Bill Cole wrote:
}
} I admit to having only skimmed the stuff on 'CallerID' because it 
} looks like the same unscaleable stuff that has been in in some MTA's 
} for a while with essentially tries a RCPT verification of every MAIL 
} FROM given

In fact -- although I don't have absolute evidence of this -- it appears
that the several days last month when verizon.net was rejecting all mail
from the ASRG list, occurred because verizon implemented a "caller ID"
check of some sort in their mail servers and the IETF mail system was
failing the check.  (The list is reaching me again, though I don't know
which end got fixed.)

More recently, I missed a message from another mailing list, and got a
bounce probe from the list software (which fortunately came through)
because my MX forwarder's DNS went down briefly, causing verizon's test
to fail.  This seems excessively brittle to me.

} and there are probably enough of us managing mail servers 
} who will address that sort of behavior as just another attack (null 
} route the offending machine) so that the inherent scaling and 
} indirect attack problems don't ever make it to visibility.

Exactly -- a "MAIL FROM: <>" followed by "RCPT TO:" an unknown address
is the signature of, for example, a system that is bouncing wormspew to
the worm-forged sender address.  How would the administrator of the MXs
that are being "caller ID" probed, know the difference?

---

From: Bill Cole <[email-masked]>
Date: Sat, 29 Nov 2003 23:50:04 -0500

>(concentrated at ISPs) than receiving servers.

What is your basis for this?

There are far more non-ISP servers (sending and receiving servers) 
than there are ISP servers. ISP's carry the most volume and have the 
most users, but in a head count of machines intentionally and 
actively running MTA's, they hold a minority. Except for ISP's, where 
the scaling makes it natural to end up with fewer sending machines 
than receiving machines, there isn't a big difference in those 
numbers, and at the low end of the scale the machines are often 
identical.

-- 
Bill Cole
[email-masked]

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 00:18:12 -0500

> you're doing to my server any less an abuse of resources than what the

Correct.  Optimization is surely next on the list for WCSAP.

> prepared to handle that load, or have you just been DoS'd into oblivion?

Doesn't this apply to any approach? A DNS based solution can also be
overloaded just as well.

But no, your right.  The process has to take this into account.  My main
design goal was to work out the interface into our SMTP state machine - a
black box concept.

Abuse, blacklisting,  optimization is next for this specific implementation.
I already have some ideas that address this.  A cached or database is
required, maybe timed based.  This could also be based on a network of
peer-to-peer systems.

> to fail.  This seems excessively brittle to me.

If I felt it was, trust me, I wouldn't be putting it in our system.   It
works.  The kinks need to be worked out.  By far, it is the #1 method to
drastic reduce spoofing spammers using current technology - TODAY!

> that are being "caller ID" probed, know the difference?

Good point.

What we did here was to make the HELO and MAIL FROM: definable.   So you can
define what you want to put in there:

        SapMailFrom    <[email-masked]>
        SapLocalHost  <winserver.com>

One of the beta testers needed to do this for his HOTMAIL account which is
violating the RFC by not accepting NULL addresses.

But it doesn't end there.  I am going to explore other ideas, such as use
"VRFY" option, which many systems disable but it can be tested, and other
ideas such as maybe using a new EHLO keyword:  XSAP

C:  EHLO winserver.com
S: 250-winserver.com, Pleased to meet you.
S: 250-XSAP
S: 250-AUTH CRAM-MD5 LOGIN PLAIN PLAIN-MD5 SHA-1
S: 250-AUTH=LOGIN
S: 250 HELP
C: XVALIDATE  [email-masked]

That way we can use MAIL FROM and also a loops.

---

From: "Eric S. Raymond" <[email-masked]>
Date: Sun, 30 Nov 2003 00:20:19 -0500

Bill Cole <[email-masked]>:
> >This is is a viable approach because there are so many fewer semding 
> >servers (concentrated at ISPs) than receiving servers.
> 

Well, for starters, the fact that people connected to AOL (which, last
I read, was still about 1/3rd of Internet users even America) never
originate their own SMTP transactions.  Also true of most Yahoo users
and most Speakeasy users.
 
> actively running MTA's, they hold a minority.

You may be right -- but even if you are, what I was really arguing is that is
we have a choice between technology that requires us to fix all the
world's senders and technology that requires us to fix all the world's
receivers,  fixing *senders* is a much easier deployment problem.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 00:51:33 -0500

Hi Bart

----- Original Message ----- 
From: "Bart Schaefer" <[email-masked]>
To: "ASRG" <[email-masked]>
> prepared to handle that load, or have you just been DoS'd into oblivion?

I was thinking more about this when it dawn on me,  this is non-issue.

This is already controlled by server access and availability.   The number
of receiver threads are definable by the sysop with a server 421 greeting
response of:

        421 domain, Service not available. Try again later

---
Hector Santos, CTO

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 00:50:58 -0500

> the matter worst and adds overhead across the board to all systems.

But they are not an open relay! According to the requirements document 
(hhttp://asrg.kavi.com/apps/group_public/download.php/7/draft-irtf-asrg-requirements-00.txt):

"1.3.26    Open Relay MTA

  An MTA which is configured to relay mail for local domains and
  foreign domains not associated with the MTA, without restricting
  access to the relay functions.  In essence an Open Relay MTA
  typically relays messages for known and unknown domains because
  of improper configuration."

In your definition of an open relay is a server that is allowed to say 
"250" to every RCPT TO command for a *local* domain! There is *nothing* 
in ANY RFC that states that an SMTP server is not allowed to do that. 
Therefore, your definition is inconsistent with the existing standards 
since you are expecting SMTP servers to do something that is not in the 
specs.

Yakov

-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
"I ate your Web page. / Forgive me. It was juicy / And tart on my 
tongue." (MIT's 404 Message)
-------

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 00:55:00 -0500

> requiring strict technical conformity and failing closed rather than open.

I have no problems with that, the problem that I have is the fact that 
some of things being discussed such as the "caller id" method of RCPT TO 
is not required by any existing specification.

Alan Dekok has previously volunteered to write up a document going 
through the entire SMTP model as it is today, and pointing out every 
single section of it which is currently being exploited by spammers. The 
next logical step would be checking which points in the model can be 
adjusted and tightened to reduce spam.

Yakov
-------

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 00:56:29 -0500

> failing the check.  (The list is reaching me again, though I don't know

There are existing issues with rDNS configuration on IETF's machines. We 
were not aware of any outstanding issues such as some kind of other 
"caller id" check aside from.

Yakov
-------

---

From: Bill Cole <[email-masked]>
Date: Sun, 30 Nov 2003 01:01:34 -0500

>and most Speakeasy users.

Those users never terminate their own inbound SMTP transactions. All 
the SMTP for recreational users and businesses who don't operate 
their own mail servers typically bottlenecks through ISP servers, 
although it is somewhat likely for the small businesses to also 
generate mail from senders that are not also receiving MTA's, i.e. 
things like mailing gadgets associated with websites.

the exception to that is that most end users in that class DO in fact 
initiate their own SMTP: to their providers' servers. If changes need 
to be made in sending agents like the myriad of MUA's that use SMTP 
to submit mail, that's a huge issue.

On a larger scale, there are a significant number of companies with 
far more senders than receivers. For example, I am currently working 
in an environment where there are approximately 150 different hosts 
which handle their own outbound SMTP to the world, and some of those 
have more than one SMTP sender implementation on them. Yet there are 
only 2 machines with very tightly administered identical MTA's that 
can accept SMTP connections from the outside world. That is not a 
particularly unusual corporate environment.

>You may be right --

Just think about it for a minute about all the non-ISP companies 
handling their own mail service. It is a practical endeavor at about 
the same scale where it becomes practical to have a full-time one-man 
IT department. It's a lot of mail systems of scales smaller than any 
serious ISP, but it is still a LOT of mail systems.

>but even if you are, what I was really arguing is that is

I don't think so.

I have spent much of the past year trying to evangelize a reduction 
in senders to one per host in the environment described above (i.e. 
to eliminate or trivialize things like  JavaMail in web apps and 
worse abominations that barely implement a working subset of SMTP) so 
that mail is auditable. Cutting down further is not going to happen 
because there's no compelling reason to do so and some tangible 
reasons not to. The result is 150 senders, 2 receivers.

In the end, I expect that there will need to be fixes to both senders 
and receivers,
-- 
Bill Cole
[email-masked]

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 01:17:43 -0500

Hector Santos wrote:
>>have, see below).  A spammer forges winserver.com MAIL FROM: on a couple
> 
> of
> 

DNS lookups are *much* more lightweight than SMTP sessions. According to 
RFC 1035, DNS queries can be done over UDP, with a limit of 512 bytes 
per query, with the usual ones much smaller than that. This has two 
advantages - very low transfer size, and also the usage of UDP does not 
require an acknowledgement to be sent back unlike TCP, reducing traffic. 
Also, DNS information tends to be cached all over the Internet.

That is why an DNS method is being pushed in LMAP, since it is more 
lightweight. As a matter of fact the "L" in "LMAP" stands for "lightweight".

Plus many domains have their DNS services outsourced to someone else 
like UltraDNS, whose systems are more than enough to withstand most DDOS 
attacks.

> ideas such as maybe using a new EHLO keyword:  XSAP

Use of VRFY should be done with caution, many systems may turn it off 
and a good reasond to do so, from RFC 2821:

"7.3 VRFY, EXPN, and Security

    As discussed in section 3.5, individual sites may want to disable
    either or both of VRFY or EXPN for security reasons.
"

 From RFC 2505:

"2.11. SMTP VRFY and EXPN

    Both SMTP VRFY and EXPN provide means for a potential spammer to test
    whether the addresses on his list are valid (VRFY) and even get more
    addresses (EXPN). Therefore, the MTA SHOULD control who is is allowed
    to issue these commands. This may be "on/off" or it may use access
    lists similar to those mentioned previously.

    Note that the "VRFY" command is required according to RFC821, [1].
    The response can, though, be "252 Argument not checked" to represent
    "off" or blocked via an access list. This should be the default.

    Default for the "EXPN" command should be "off".
"

ALSO, please do not take my comments as an attack on your approach, 
rather I think that what is missing here, is the fact that your approach 
is not 100% compliant with the standards. If you are proposing to change 
the standards, that's fine but I think that should be made clear. 
However, in my opinion as you are proposing it now, this proposal is not 
in compliance with existing email standards since it requires the 
receiving domains to answer 252 upon RCPT TO to valid addresses only. 
That is not a current requirement, and blacklisting Yahoo and others for 
doing that is incorrect.

I think that we need to evaluate the current SMTP model first to see how 
many holes there are, and THEN see what we can do about them. Since can 
help to put things in perspective better.

Yakov

-------

---

From: "Eric S. Raymond" <[email-masked]>
Date: Sun, 30 Nov 2003 01:26:58 -0500

> adjusted and tightened to reduce spam.

I will give Alan's draft careful and (constructively) critical attention.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 01:28:08 -0500

> legit client connecting into the system to deliver mail.

There is nothing in the current IETF email standards that requires (1) 
the sender's system to operate a valid MTA that responds to RCPT TOs for 
the email account that is sending the email, and (2) must not accept 
RCPT TOs to *any* address at its domain. For example, some domain owners 
use "catch alls" also to accept all incomings, which is perfectly legit, 
OR ISPs such as Yahoo that are doing this to stop harvesting and 
dictionary attacks.

> DNS, Mail Filters, SPAM vs NO SPAM, BULK vs NON-BULK,  non-server vs server

What I would like to do, is to go through the SMTP model step by step, 
analyzing loopholes, and see what can be tightened and improved. Then 
from there we can figure out which proposals can work.

Yakov

-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
  "Why are both drug addicts and computer aficionados both called 
users?" (Clifford Stoll)
-------

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 01:46:48 -0500

Side note;  Why do we cc direct users when we writing to the mailing list
too?   Just wondering.

----- Original Message ----- 
From: "Yakov Shafranovich" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "Matthew Elvey" <[email-masked]>; <[email-masked]>
Sent: Sunday, November 30, 2003 12:50 AM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

> since you are expecting SMTP servers to do something that is not in the

Please my mail better Yakov.

I am perfectly aware the SMTP server may accept RCPT TO: without validation
for local domains.  Geez, been designing SMTP ready software for quite a
years now and it ranks among the best.  So give me a break, huh?
Our software also offers a [X] Validate Local User option as well.  When
off, the validation is delayed until the gateway mail processor gets a hold
of the mail.  However, it must be delivered or a bounce notification is
immediately sent back.  That is the difference!  It can go into LA-LA land.

You stated YAHOO is accept all RCTP TO: to avoid "harvesting," which is not
quite the case as I will show.
What I had stated is that If YAHOO is accepted ALL addresses, including
accepting the open relay test against a totally random email domain, then it
is an OPEN RELAY:

Anyway,  lets test to see if YAHOO will send a BOUNCE notification to a fake
address but with a proper MAIL FROM: address.

OK, notice this transaction:

220 YSmtp mta239.mail.scd.yahoo.com ESMTP service ready
HELO LOCALHOST
250 mta239.mail.scd.yahoo.com
MAIL FROM: <[email-masked]>
250 sender <[email-masked]> ok
RCPT TO: <[email-masked]>
550 relaying denied for <[email-masked]>
RCPT TO: <[email-masked]>
250 recipient <[email-masked]> ok
DATA
354 go ahead
THIS IS A BOUNCE TEST TO SEE IF YAHOO WILL BOUNCE THE MAIL
BACK TO [email-masked]
.
554 delivery error: dd This user doesn't have a yahoo.com account
(asdasdas@yaho
o.com) [-5] - mta239.mail.scd.yahoo.com
quit
221 mta239.mail.scd.yahoo.com

Ok, so you were not quite right about Yahoo yourself.. They are not
addressing the harvesting problem. But instead delaying the user validation
handling until the DATA point is receive.    You either offer a dynamic
ACCEPT/REJECT or a delay BOUNCE, which many systems do to offload processing
issues.

YAHOO is doing Local User validation by delaying the process until the data
is received, which is probably done as a backend thread lookup while the
session thread was being receiving data.  Perfectly legit design to help in
the optimization process.

Nonethless, Yakov. This is good News!   This make WCSAP even better.  I will
now add this logic by detected the YAHOO MTA and go the DATA state.  This
will make WCSAP stronger now and work with YAHOO. I am going to explore the
other ISP as well to see if they also do delayed validation at the DATA
state.

Thanks YAKOV!

---
Hector Santos

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 01:58:36 -0500

> adjusted and tightened to reduce spam.

Thank you.  That's all I been saying.

Remember think!

connect ---> validate
helo/eho --> validate
mail from --> validate
rcpt to:---> validate
data---> validate
quit

Once you do this (properly) you will come to the logical technical
conclusions (I hope).

By way, what mail server has Alan written?  never mind, doesn't matter.

Have you seen my stats and analysis? Do you have any questions about it?

---
Hector Santos

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 02:00:22 -0500

> Side note;  Why do we cc direct users when we writing to the mailing list

That argument has been raised many times. Some people filter the main 
list traffic into a different folded unless it is also addressed to them 
directly.

>>since you are expecting SMTP servers to do something that is not in the
[..]
> ACCEPT/REJECT or a delay BOUNCE, which many systems do to offload processing

I believe that it has been mentioned before that the reason why they do 
it that way, is because spammers tend to do dictionary attacks using 
RCPT TO.

> the optimization process.

Correct.

> other ISP as well to see if they also do delayed validation at the DATA

What happens if this is a legit user? You will end up sending an email 
to him since at the DATA command, the MTA will accept the message? So 
therefore, for those MTAs which answer 250 to all RCPT TOs and do 
validation after the DATA command, your method of verification will 
result in an extra email message to the user which itself can be 
considered spam by the user (same argument as the C/R business).

What I would like to narrow down is what purpose does this proposal 
address? What exact forgery does it solve?

Yakov
-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
"Fight spam, but keep it in perspective" (Brad Templeton)
-------

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 02:02:51 -0500

> Once you do this (properly) you will come to the logical technical

Ok, I think we'll all be waiting for his document very eagerly. 
Hopefully he can provide a status update and perhaps let us know if he 
needs more help.

> Have you seen my stats and analysis? Do you have any questions about it?

My only concerns with the stats is that there is really no sure way to 
find out if any legimitate email was rejected through your process since 
you do not actually possess the actual piece of email.

Yakov
-------

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 02:19:10 -0500

----- Original Message ----- 
From: "Yakov Shafranovich" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "ASRG" <[email-masked]>
Sent: Sunday, November 30, 2003 1:17 AM
Subject: Re: [Asrg] 0. General - Inquiry about CallerID Verification

> require an acknowledgement to be sent back unlike TCP, reducing traffic.

With the good DNS system,  UPD is the first approach.  However, it has
provisions to switch to a TCP stream connection if UDP fails.

> Also, DNS information tends to be cached all over the Internet.

Thats the only good thing about it. :-)

In either case, that doesn't stop it from being overwhelmed.

> is not 100% compliant with the standards.

I don't think you are correct.   Everything I do is 100% compliant.
Everything else you stated is not worth even acknowledging.  Its nonsense.

PS: Do me a favor, quit dumping the RFC on me.  If you want to refer to an
section, please do.  But I know what the specifications are backwards and
forwards with real compliant products in the market place.

---
Hector Santos

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 02:25:04 -0500

[..]
> Everything else you stated is not worth even acknowledging.  Its nonsense.

You're are expecting the sender's system to operate a valid MTA that is 
connected to the Internet at the time of the incoming SMTP transaction 
from the sender and responds to an RCPT TO command for that sender. This 
is not a requirement in any current standard.

Yakov
-------

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 02:32:28 -0500

----- Original Message ----- 
From: "Yakov Shafranovich" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "Alan DeKok" <[email-masked]>; "ASRG" <[email-masked]>
Sent: Sunday, November 30, 2003 1:28 AM
Subject: Re: [Asrg] 0. General - Inquiry about CallerID Verification

> OR ISPs such as Yahoo that are doing this to stop harvesting and

Yakov, they are delaying the validation until the DATA state.   They are
probably using pareto's principle that suggest that most of their spammers
are TOO stupid to change their SMTP software to intepret that DATA results.
<g>

> from there we can figure out which proposals can work.

hope to see a good job. <g>

I'm available for technical review.

---
Hector Santos

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 03:43:08 -0500

----- Original Message ----- 
From: "Yakov Shafranovich" <[email-masked]>
To: "ASRG" <[email-masked]>

> considered spam by the user (same argument as the C/R business).

arrrrrrrrrrrrrrrrrrrgh.  Good point.   I having looked into yet. Busy
writing mail.

I'll explore it.    Is YAHOO around?????

> address? What exact forgery does it solve?

Valid Return Path.

The RFC says that all mail must be returnable (except for NULL address).

---
Hector Santos

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 03:50:23 -0500

> is not a requirement in any current standard.

I believe I already mention that WCSAP issues a 4xx code for down
connections.

If the remote MTA is down, a 4xx code is provide which tells the sender to
TRY again later.  If he is not following this, then the sender is
non-compliant.

---
Hector Santos

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 03:58:09 -0500

> I believe I already mention that WCSAP issues a 4xx code for down

That I got. My question is about the fact that you are requiring every 
sender to have an MTA with a RCPT TO - a valid return path. See my other 
message.

Yakov

-------

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 04:00:11 -0500

[...]
> The RFC says that all mail must be returnable (except for NULL address).

Ok, I believe we have narrowed it down :) Thanks for bearing with us 
folks :)

So what we are addressing is the valid return path. The current 
standards allow for invalid return addresses, which is the one of the 
root problems that led to spam. This has a side-effect of allowing 
anonymous email (which can still survive if the anonymous email server 
uses its own email address for MAIL FROM). That I believe is our point 
of contention, and I would be more than happy to see the exact section 
which requires all mail to be returnable.

Yakov
-------

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 04:28:59 -0500

----- Original Message ----- 

> which requires all mail to be returnable.

See RFC 2821 section 6, second paragraph:

   "If there is a delivery failure after acceptance of a message, the
   receiver-SMTP MUST formulate and mail a notification message.  This
   notification MUST be sent using a null ("<>") reverse path in the
   envelope.  The recipient of this notification MUST be the address
   from the envelope return path (or the Return-Path: line). "

Thats pretty straight forward to me.

If a return path MUST be available to delivery of failure notifications,
then it can only be accomplish with a valid return path.   Therefore, it is
only logical that a valid return path is provided in the first place in the
event a delivery failure notification is required.

Incidentally, section 7.1 last paragraph is obsolete and inconsistance with
the rest of the specs. Needs to be moved if you wish to move on with your
efforts.

  "This specification does not further address the authentication issues
   associated with SMTP other than to advocate that useful functionality
   not be disabled in the hope of providing some small margin of
   protection against an ignorant user who is trying to fake mail."

Obviously,   "small margin" is now LARGE and "an ignorant user" is now "an
ignorant industry"

<G>

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 30 Nov 2003 11:51:29 -0500

> connect ---> validate
...

  These checks are like firewall rules: they stop bad traffic, but do
nothing to prevent such traffic in the first place.

  I believe that there are minor modifications to deployment of SMTP
which may have a significant affect on the UBE problem.  The
modifications are more on the order of changing our assumptions about
how the mail system "should" be used, rather than adding filter rules.

> By way, what mail server has Alan written?  never mind, doesn't matter.

  It doesn't matter.  I'm not interested in how SMTP server are
written, and it's irrelevant to the problem at hand.  I'm interested
in the assumptions in the programmers mind, which is very different...

  Alan DeKok.

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 30 Nov 2003 11:55:25 -0500

> Hopefully he can provide a status update and perhaps let us know if he 

  More help would be appreciated.  I've got an outline, and a "todo"
list.  What would help is people willing to root through the details
of protocol X (RSS, HTTP, IRC, etc.), and describe how it differs from
SMTP (using the outline of SMTP's issues).  People willing to
contribute, please email me off-list.

  Alan DeKok.

---

From: Bart Schaefer <[email-masked]>
Date: Sun, 30 Nov 2003 16:51:10 +0000

On Nov 30,  4:28am, Hector Santos wrote:
} 
} ----- Original Message -----  (Yakov)
} 
} > > The RFC says that all mail must be returnable (except for NULL address).
} >
} > [...] I would be more than happy to see the exact section
} > which requires all mail to be returnable.
} 
} See RFC 2821 section 6, second paragraph:
} 
}    "If there is a delivery failure after acceptance of a message, the
}    receiver-SMTP MUST formulate and mail a notification message.  This
}    notification MUST be sent using a null ("<>") reverse path in the
}    envelope.  The recipient of this notification MUST be the address
}    from the envelope return path (or the Return-Path: line). "
} 
} Thats pretty straight forward to me.

That's a requirement that the receiving server send the delivery failure.
It is not a requirement that the original sender accept it.

---

From: Bart Schaefer <[email-masked]>
Date: Sun, 30 Nov 2003 17:03:05 +0000

On Nov 30, 12:51am, Hector Santos wrote:
} 
} ----- Original Message ----- 
} From: "Bart Schaefer" <[email-masked]>
} To: "ASRG" <[email-masked]>
} >
} > Suppose a large ISP were to adopt the "caller ID" scheme (one
} > already may have, see below). A spammer forges winserver.com MAIL
} > FROM: on a couple of million messages destined for that ISP,
} > distributed across all its dozens of MXs. They all begin connecting
} > to your MX to ID the caller. Are you prepared to handle that load,
} > or have you just been DoS'd into oblivion?
} 
} I was thinking more about this when it dawn on me,  this is non-issue.
} 
} This is already controlled by server access and availability.

You're missing the point.

You might believe it's a non-issue for you, but that doesn't make it a
non-issue for everyone who might become a victim of it.  Recommending an
approach such as your caller-ID technique is irresponsible, because it
can lead to indirect abuse of innocent third parties.

} This is already controlled by server access and availability.   The number
} of receiver threads are definable by the sysop with a server 421 greeting
} response of:
} 
}         421 domain, Service not available. Try again later

How is this a non-issue?  ALL of your mail is going to stop flowing with
this error until such time as the flood of caller-ID connections stops.
And assuming the hypothetical ISP in the example does wait and try again,
you'll have those interruptions repeatedly until all the forged messages
drain out of the ISP's queues.

This leads to a further question (of academic interest only, really):  You
have said that your system returns a 4xx to the sender when the caller-ID
callback is unable to connect.  What does it do when the callback gets a
4xx response itself?

---

From: Matthew Elvey <[email-masked]>
Date: Sun, 30 Nov 2003 10:11:39 -0800

Perhaps a better alternative to deprecating <> would be saying that MTAs 
SHOULD bitbucket DSNs about email they know (thanks to LMTP and/or 
Received/MessageID headers) weren't sent from authorized IPs/systems.  
Good/workable idea?

On 11/29/2003 5:32 PM, Bill Cole sent forth electrons to convey:

>> ... IMO, MAIL FROM: <> needs to be deprecated in favor of refusal 
>> during SMTP.
>

True.  By deprecated, I mean a spec that says that MTAs SHOULD not (not 
MUST not) use  <>.  Let's see if there is another solution for 
everything legit that it's currently useful for.  We don't want <> to be 
a way around LMTP, right?   For any failure DSN, I think the systems 
that send them can and should be modified so that all the failure status 
codes of rfc 1893 can be provided during SMTP.  Any intermediate system 
(offline UUCP excepted) that does so would need to be redesigned, as you 
point out below, if they wanted to avoid using <>.    For any success 
DSNs, perhaps they shouldn't be sent MAIL FROM: <>.  Locally generated 
and delivered mail shouldn't be bitbucketed.

>> mentions, which could be resolved with some work. (I should admit - I 
>> don't do what I preach, as the provider I choose to have host my 
> That example was ONE case. 

Thanks for providing more for discussion.

> Without delaying the validation of the target address, every domain in 
> which there are mail addresses would need published MX records 
> pointing to machines with live access to the authoritative knowledge 
> store for each domain it serves. We could kiss UUCP goodbye finally, 
> and if you think no one still uses it you are quite wrong. 

Valid point.  (except we're deprecating, not forbidding)

> Large organizations that now have single external faces on their mail 
> systems for many possible subdomains which actually route internally 
> in complex ways would need to be redesigned. As an example of this, I 
> work with a major systems vendor with a global workforce whose 
> individual email addresses reflect their geographic location, yet the 
> exterior MX record set for all of the subdomains is one server farm. 
> That server farm has to know how to route mail to many different 
> subdomains handled on many different machines around the world. It 
> DOES NOT have access to the full user namespace for all of those 
> domains, it merely knows that it accepts mail for the various 
> subdomains and hands it off internally to machines closer to the HR 
> departments for the various branches. Even without exposed subdomains, 
> such situations are problematic because companies beyond a certain 
> size simply do not operate as single entities and cannot operate their 
> email systems on single giant machines that know everything 
> simultaneously. 

An intermediate system has 10 minutes after the end-of-data indicator to 
contact the downstream system and find out if it will accept the 
message; it could be redesigned to do so before sending the final 250 
OK.   Do you think the idea in the first line of this email is more 
workable?

> The fact of an address no longer being valid can easily take a day to 
> propagate throughout a global operation, and it is a rare large 
> company that can make the event of a user add or delete look 
> instantaneous to their entire email system.
>
> route the offending machine) so that the inherent scaling and indirect 
> attack problems don't ever make it to visibility.

It would rarely look like an (address harvesting) attack, I think.

Hector still has to address someone (?Yakov's?) point that he 
isn't/can't provide accurate false positive data if he's rejecting mail 
after MAIL FROM!
Hector?  Until then, it's only a useful tool for a scoring system.

---

From: Walter Dnes <[email-masked]>
Date: Sun, 30 Nov 2003 13:59:41 -0500

On Sun, Nov 30, 2003 at 03:43:08AM -0500, Hector Santos wrote
> arrrrrrrrrrrrrrrrrrrgh.  Good point.   I having looked into yet. Busy

  Ever heard of the expressions "deadly embrace" or "vicious circle" or
"mail loop" ?  Assume that machine A, which uses your system, tries to
send to machine B, which also uses your system.  What protection is
there to prevent such problems ?

  Secondly... what exactly does your system do that "identd" hasn't been
doing for the past 30 years ?

-- 
Walter Dnes <[email-masked]>
Email users are divided into two classes;
1) Those who have effective spam-blocking
2) Those who wish they did

---

From: Claus Assmann <[email-masked]>
Date: Sun, 30 Nov 2003 11:07:57 -0800

> a way around LMTP, right?   For any failure DSN, I think the systems 

LMTP?

> and delivered mail shouldn't be bitbucketed.

What about systems that are temporarily offline? Do you want to get
rid of queueing?  Reminds me of the X.500 mailsystem we had: "the
system is currently unavailable, try again later" whenever it
encountered a problem... requiring the user to do the queueing (and
retrying) is a really bad idea.

> An intermediate system has 10 minutes after the end-of-data indicator to 

Some systems (esp. MUAs) don't wait 10 minutes.

PS: there are better ways to deal with MAIL From:<>
e.g., keep track of mails you sent and only accept DSNs for those.
This doesn't break anything in contrast to the proposals that are
coming up here again and again without considering why DSNs are
necessary and why <> has been chosen as sender address.

---

From: Scott A Crosby <[email-masked]>
Date: 30 Nov 2003 13:20:43 -0600

On Sun, 30 Nov 2003 10:11:39 -0800, Matthew Elvey <[email-masked]> writes:

> Perhaps a better alternative to deprecating <> would be saying that
> MTAs SHOULD bitbucket DSNs about email they know (thanks to LMTP
> and/or Received/MessageID headers) weren't sent from authorized
> IPs/systems.  Good/workable idea?

Probably, if the burden of proof is low and relatively easy.

> True.  By deprecated, I mean a spec that says that MTAs SHOULD not
> (not MUST not) use  <>.  Let's see if there is another solution for
> everything legit that it's currently useful for.  We don't want <> to
> be a way around LMTP, right?   For any failure DSN, I think the
> systems that send them can and should be modified so that all the
> failure status codes of rfc 1893 can be provided during SMTP.  Any
> intermediate system (offline UUCP excepted) that does so would need to
> be redesigned, as you point out below, if they wanted to avoid using
> <>.    For any success DSNs, perhaps they shouldn't be sent MAIL FROM:
> <>.  Locally generated and delivered mail shouldn't be bitbucketed.
> 

This seems to be putting the cart before the horse.

AFAIK, DSN's are for exceptional conditions. IE, in a well functioning
system, all messages get delivered and no DSN's need to be sent. Right
now we have a mis-functioning system where people send messages with
forged sending addresses, thus the DSN's are a DoS on the person
having their email address forged. Especially if they cannot be
programattically filtered.

But, perhaps they could?

How to do it is all sending systems attach a nonce (a random number)
to an email. All DSN's must include a nonce indicating to what this is
a response to. Users have the option to filter based on whether a
correct nonce is attached.

If a mailserver is changed to implement LMTP, it can be changed to
also implement the nonce scheme at the same time.

Scott

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 14:28:37 -0500

----- Original Message ----- 
From: "Walter Dnes" <[email-masked]>
To: "ASRG list" <[email-masked]>
Sent: Sunday, November 30, 2003 1:59 PM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

> send to machine B, which also uses your system.

Ever heard of "Paper hat?"

> What protection is there to prevent such problems ?

Null Address

> doing for the past 30 years ?

Apple and oranges.

---
Hector Santos

---

From: Claus Assmann <[email-masked]>
Date: Sun, 30 Nov 2003 11:33:33 -0800

> correct nonce is attached.

There's no need to invent something new. Use the Message-Id: header.

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 14:37:40 -0500

----- Original Message ----- 
From: "Claus Assmann" <[email-masked]>
To: "ASRG" <[email-masked]>
Sent: Sunday, November 30, 2003 2:07 PM
Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about CallerID
Verification)

> retrying) is a really bad idea.

Claus,

Queuing design is par for the course.  All software need to be queue ready
and all of them (worth their salt) are.  Otherwise you do expect the target
to be online at the time, which is unrealistic.  Also, it may not be that
its down, but maybe reached capacity.

Hector Santos

---

From: Claus Assmann <[email-masked]>
Date: Sun, 30 Nov 2003 11:52:22 -0800

...

Good to hear you agree.  Therefore this kills the proposal to not
use DSNs.

---

From: Bill Cole <[email-masked]>
Date: Sun, 30 Nov 2003 14:56:29 -0500

At 1:20 PM -0600 11/30/03, Scott A Crosby  imposed structure on a 
stream of electrons, yielding:

>system, all messages get delivered and no DSN's need to be sent.

No. DSN's are Delivery Status Notifications, not just bounces.

See RFC1891 and RFC3461.

-- 
Bill Cole
[email-masked]

---

From: Scott A Crosby <[email-masked]>
Date: 30 Nov 2003 14:06:37 -0600

> There's no need to invent something new. Use the Message-Id: header.

Not quite. This would be something added on by the email server, and
it doesn't need to be unique among all messages and all senders. By
using cryptography, for instance encoding a counter and a MAC over
just the counter, the mechanism can be made *MUCH* cheaper than
recording the message ID's of all sent messages.

With this implementation, the sender doesn't need to record the nonces
on all outgoing emails. They can accept a DSN, read the counter and
verify its MAC, and accept the DSN only if the counter is both fresh
(less than 10 days old) and has been used at most 5 times.

Storage for the server is one MAC key, one counter, and a list of all
DSN's accepted for delivery in the last 10 days.

Depending on the disclosure or abuse risks of DSN's, (which I don't
fully understand) this could be simplified. We remove the list of
accepted DSN's. This means that a given counter may now be attached to
an unlimited number of times in a 10 day period.  In exchange the
server now only needs to store two things, a counter and a MAC key.

Scott

[*] A MAC is a keyed cryptographic hash.

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 15:18:00 -0500

----- Original Message ----- 
From: "Alan DeKok" <[email-masked]>
To: <[email-masked]>
Sent: Sunday, November 30, 2003 11:55 AM
Subject: Re: [Asrg] 0. General - Inquiry about CallerID Verification

> contribute, please email me off-list.

Hmmmmmmmm,

I don't see why you would want to to confuse the issue with RSS, HTTP, IRC,
etc?

Look,   unless you are going to fund the development efforts for vendors to
change their software,  you are not going to define how they will REVAMP
their current systems.   Thats the bottom line.

The problem is the SMTP specification.   Don't confuse it with anything
else.  You need to highlight its problems,  and then make it technical
feasible for developers to create solutions in the spirit of Mr. Crocker
wishes.

Think Black Box Functional Engineering!   You don't need to be a programmer
to do this, it helps, but you do need to have experience as a system
engineer.

To drastically reduce the spammer problem,

A) Deal with access issues
B) Deal with technical comformity issues

thats it!!   "One small step for man,  one giant leap for mankind!"

Just imagine that you were writing a new "SMTP" protocol and you were going
to expose it specs to the world.   You have NO idea whats out there.  All
you can define the interface points in the specification.  That is all you
can do.

---
Hector Santos

---

From: Claus Assmann <[email-masked]>
Date: Sun, 30 Nov 2003 12:19:46 -0800

[please don't Cc: me]

> recording the message ID's of all sent messages.

That only works if all the MTAs that can receive the DSNs have
"knowledge" of all the mails that could cause DSNs to the domains
for which they are responsible. What if a mail is sent via a
different MTA? Well, maybe you want to forbid that?

Message-Id: works for the end user, it can be implemented in
the MUA; most MUAs allow for threading based on Message-Id:,
dealing with DSNs shouldn't be hard to add.

RFC 3461:
   (d)  preserves sufficient information to allow the sender to identify
        both the mail transaction and the recipient address which caused
        the notification, even when mail is forwarded or gatewayed to   
        foreign environments, and

That RFC also specifies ENVID (in case Message-Id: gets lost.)

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 15:22:00 -0500

> I don't see why you would want to to confuse the issue with RSS, HTTP, IRC,

There are security issues present in other protocols which have been 
dealt with in different ways. We can use some of those methods in SMTP.

Yakov
-------

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 15:35:28 -0500

----- Original Message ----- 
From: "Bart Schaefer" <[email-masked]>
To: "ASRG" <[email-masked]>
Sent: Sunday, November 30, 2003 12:03 PM
Subject: Re: [Asrg] 0. General - Inquiry about CallerID Verification

> can lead to indirect abuse of innocent third parties.

I fail to see how.   I would love to see an example.

> How is this a non-issue?

Retries are part of the functional specification of SMTP systems.   JAVA
beans, scripts with 1 shot approaches are asking for trouble, with or
without this.

> ALL of your mail is going to stop flowing with

No, its not.  Again, I fail to see this.   I have WCSAP running for nearly
5-6 days now with very little issues that is being worked out.

> And assuming the hypothetical ISP in the example does wait and try again,

You can't design software on the assumption that SMTP systems will not be
following specifications.  You will go nuts otherwise.

> drain out of the ISP's queues.

Not an issue.

> callback is unable to connect.  What does it do when the callback gets a

It returns the same thing back.

Look,  functionality is a IDEAL solution.

The implementation has to be worked out.   One implemention (WCSAP)  is to
implement it within the confines of the SMTP system in theory and in
practice.

The goal (at least for us) is to functionally engineer a "WCSAP" so that it
fits into our SMTP server, so that it does return a TRUE negative or TRUE
positive.    The day a more reliable "CallerID Verification" is worked out,
be it some other protocol, or lookup, etc,  we simply substitute it.   I
don't like WCSAP for one reason only, its OVERHEAD, but functionality, its
is a great!

I am going to make available the logs from WCSAP.  I think you will find
them interesting.  There are questionable issues that need manual checks.
There are also VERY interesting behaviors (as we learn about YAHOO delayed
validation).

But BY FAR, with EMPERICAL DATA on hand, that this is the best METHOD out
there to combat illegal spoofing by spammers.   I have provided 6 months of
emperical data of state point validations.

with these two alone:

strict HELO/EHLO format checking
MAIL FROM: validation

you will eliminate a large percentage of the spoofing spammers.

If real operations did not show this, we wouldn't be doing this.

----
Hector Santos

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sun, 30 Nov 2003 22:04:36 +0100

> I fail to see how.   I would love to see an example.

He already gave one.

To reiterate:

Someone sends out millions of messages with a forged winsite.com return
path to sites which implement the caller-ID technique. All of these
sites will connect to your mail server and ask it whether the return
path is valid.

Is your mail server prepared to handle this load and still accept
legitimate messages? Do you have enough bandwidth?

Of course this is not a new problem. They could also send lots of
messages with your domain in the return path to sites with an "accept
and bounce" policy and DDoS you with the bounces. Or DDoS you directly
with millions of messages, although a spammer probably wouldn't do that
unless you really annoy him.

> 5-6 days now with very little issues that is being worked out.

After 5-6 days on a single site it seems awfully optimistic to me when
you say "we haven't been DDoSed yet, so it will never happen".

> following specifications. 

You should. Because some systems out there won't follow specifications.
You don't have to provide service to those systems (although your
customers may have different opinions about that), but when designing
the software you must consider what happens if the the other breaks the
rules.

> You will go nuts otherwise.

That's always a possibility ;-)

> There are also VERY interesting behaviors (as we learn about YAHOO delayed

ACK. I deployed greylisting at two (rather different) sites 2-3 
months ago, and I've certainly learned a lot about "interesting"
behaviours.

	hp

-- 
   _  | Peter J. Holzer    | In this vale
|_|_) | Sysadmin WSR       | Of toil and sin
| |   | [email-masked]         | Your head grows bald
__/   | http://www.hjp.at/ | But not your chin.           -- Burma Shave

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 16:14:04 -0500

[..]
> the software you must consider what happens if the the other breaks the

As a matter of fact, many RFCs include sections on how to deal with 
violations and other "irregularities" in Internet protocols since they 
occur and occur pretty often. For an example, see RFC 2821, section 6.3.

Yakov

-------

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 16:32:26 -0500

----- Original Message ----- > > Hmmmmmmmm,
> dealt with in different ways. We can use some of those methods in SMTP.

Yeah, ok.  and what does this relate with that final conclusion that you
need "access security."  Are you going to exclusive lock into one method?
Are you going to now change the specs to fit a new model?   I gave you a new
proposal that changes the specs, if that is what you want.

I don't think you do.

You want to get this ball rolling using Mr. Crocker incremental approach?
He gave you an outline.  Lets use it.

Phase 1)  Functional Specifications

GOAL:

        Get the specs rewritten to clarify the new possible strict interface
points to give
        developers better insight and guidelines for new ideas based on the
SMTP or
        ESMTP protocol.

You might want to offer new interface points.   Among the current interface
points:

- connect
- helo/ehlo
- mail from:
- rcpt to:
- data
- quit

these can summarized as:

- client access/sender authentication
- recipient validation
- mail content validation ???

Mail content validation can be considered as an external functionality.
Again, in my engineering experience, no mail transport or gateway should
have no business in the "interpretation of data."   This puts ISP and
Software vendors at legal risk.   The only thing that can possible be done
here from a transport standpoint is a CRC32 data validation.   For example,
a sender can issue CRC32 that offers the server the ability to check data
integrity.

Recipient validation is a server implementation.   However,  final delivery
must be guaranteed or acknowledge in one form or another.  I believe there
is NOTHING wrong with dynamic validation especially when coupled with
trusted client access.    Having no validation delays notification. Which is
ok, but it needs to be acknowledge one way or another.

Client access and sender authentication is where it gets tricky and is the
heart of the problem.

All you can do now is get this acknowledge into the specs and make it
possible for restrictions to occur and/or they MIGHT occur depending on
extended SMTP server implementations.

At this point, you put a RFP,  "Request For Proposals"  for solutions using
a new, cleaner functional specification with new guidelines.

That's you first step.   This should be easy for you.  It is an
administrative task mostly but you do need some level of system design
experience to have you outline it without clouding external unrelated
issues.

Phase 2) Review of RFPs for SMTP Client/Access and Authentication Proposals

GOAL:

        Review, Discuss new RFP that work with the new SMTP/ESMTP functional
specifications.

In this phase, I see you dealing with two types of basic proposals:

a) designs to fit the legacy SMTP protocol,
b) designs to fit the new Extended SMTP protocol outlined in Phase 1.

This is where DEVELOPERS need help because it requires a consensus that is
independent of the SMTP design itself:

                 CLIENT --> SMTP ---> VALIDATE (HOW?)

Using these proposals, we now can revisit phase 1 to see if it needs
"tweaking" to better fit.

If you want to follow Mr. Crocker incremental design concept, it might help
if you produce a table of the SMTP state machine and for each STATE,
indicate the current proposals that touch base with each state.

for example:

             connect
                        proposals: RBL, RDNS, RMX?
             HELO
                        proposals: ???
             MAIL FROM:
                        proposals: C/R, SAP, MAP, etc
             RCPT TO:
                        proposals: C/R, ????
             DATA:
                        proposals: ???
             QUIT:
                        proposals: ???
etc, etc.

This will help ALL of us get a better insight on all this.

---
Hector Santos

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 17:00:51 -0500

Hi Peter

----- Original Message ----- 
From: "Peter J. Holzer" <[email-masked]>
To: "ASRG" <[email-masked]>
Sent: Sunday, November 30, 2003 4:04 PM
Subject: Re: [Asrg] 0. General - Inquiry about CallerID Verification

> To reiterate:

No difference from any other unhandled malicious attack.  You don't need
this method for something that is intent on crashing your system.   Of
course, you don't want it to begin on its own :-)

> Of course this is not a new problem.

Thank you. :-)

> you say "we haven't been DDoSed yet, so it will never happen".

I would multiply this with about 10+ alpha/beta testers.  I think it
significant at this preliminary level and with my vast engineering
experience in analyzing and solving problems,  I don't currently see
anything better that works today within the confines of the current
specification.

We are going gamma with our new Product Updates this week.  I might not
include WCSAP in this stage now. I need to work on it more - make it more
robust, faster and smarter.   The YAHOO delayed validation needs to be
worked out or NOT.      Hey,  its only a STEP in the total state machine
validation systems.   If can just say "Let him thru," and go the next step.

> You should. Because some systems out there won't follow specifications.

and I have dealt with broken software all the time.  It par for the course
when you are in business serving thousands of customers.  You might make
exceptions or workarounds for legacy situations or to deal with a broken BIG
GORILLA software. However,  it is exception to the rule, not the norm.   You
don't design software based on software NOT following specs. For one thing,
you don't know what is broken or could be broken.   I know of no
specification that outlines or exposes a spec and then adds:

            "If the client doesn't follow this, then you should work out the
problem with the client vendor"

Although, it will probably end up that way from a support standpoint when
you do cross that bridge.  But you don't design software with this
assumption.    The spec might say:

            "If the client doesn't follow this step , then you should break
the connection"

            "If the client doesn't follow this step , then you should go to
step 3"

but then this is a expected FUNCTIONAL design.  :-)

> customers may have different opinions about that),

Right.  As with most things, it is usually customer driven.  I am a firm
believer of pareto's principle.

> but when designing  the software you must consider what happens if the
> the other breaks the > rules.

But again, you don't design your software this way based on a well design
specification.   Fallbacks or sanity checks is something different,  time
engineered situations evolved, workarounds, are different matters
altogether.  It does make the difference from vendor to vendor. :-)

>  That's always a possibility ;-)

I'm already there <g>

---
Hector Santos

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 17:25:50 -0500

----- Original Message ----- 
From: "Yakov Shafranovich" <[email-masked]>
To: "Peter J. Holzer" <[email-masked]>
> occur and occur pretty often. For an example, see RFC 2821, section 6.3.

It is outlines these specifics, it its not unknown.  it becomes a functional
defined specification or guideline.

In practice,  most of the issues dealt with mail format conformity and
strict and proper usage of the MAIL FROM: (return path) for both transport
and gateways.

By far, most SMTP designers agree that if the MAIL FROM: is not intact, the
system will not WORK.

The mail format is more debated.  There is only ONE requirement (per RFC
guideline) which our software respect.   Each mail header must have:

    To:

and once of the following:

    Received:
    From:
    Date:

in order to be consider a valid RFC header.

Now, for our software, it does not mean the MAIL is reject, what it means is
that the it use the envelope information to fill it and add its own Message
Id, etc, etc, etc.  (PS: Please don't off on a tangent with the above,  the
complexity in the software logic engineered in our mail system to workaround
every piece of situation is not required to be discussed here)

Again,  most of the concerns in practice are with mail content, as the
statement in that section says:

     "These changes MUST NOT be applied by an SMTP server that  provides an
       intermediate relay function."

Is a very important for design criteria for making sure passthru operations
do not alter mail.

What we are talking about here specifically,   is making sure the interface
points are followed.

What if the MAIL FROM: is not followed correctly?

What if the HELO/EHLO is not followed correctly?

What if the RCPT TO: is not followed correctly?

I believe this is your part. To help changes the SPECS to clarify strict
compliance at the interface points.

Otherwise, we are all beating a dead horse here and we are not going to
solve any problem (as a group).   Trust me,  vendors will begin solve the
problem and when there is a divergence of solutions.  Bye Bye IETF!  I've
seen it before where change was not met with the market needs.  It will
happen again if you don't heed the advice.

If you don't clarify the specs,  you are not helping developers solve the
problem for you.

---
Hector Santos

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 18:20:16 -0500

Hi Bart,

----- Original Message ----- 
From: "Bart Schaefer" <[email-masked]>
To: "ASRG" <[email-masked]>

> It is not a requirement that the original sender accept it.

Thats a non-issue.  Unrelated really.    A Callerid Verification system is
not sending mail.  Half the battle is solve by being able to connect or not
connect and reaching the RCTO TO: stage.  It is functionally designed to
sended "mail."

But with that said,  the receiver must accept a NULL address.  Its a
requirement. See RFC 1123, specifically

 5.4  SMTP REQUIREMENTS SUMMARY
 5.3.3 Reliable Mail Receipt

---
Hector Santos

---

From: Matthew Elvey <[email-masked]>
Date: Sun, 30 Nov 2003 15:50:11 -0800

Err, I've said LMTP when I meant LMAP a couple times.

Mail servers already keep, and should keep Message-IDs. 
I proposed:

> MTAs SHOULD bitbucket DSNs about email they know (thanks to LMAP
> and/or Received/Message-ID headers) weren't sent from authorized

Expanding on this I see two schemes: A, B:
A)If we're going to use Message-IDs, then a new protocol is needed for 
the MXes for a domain to have access to Message-IDs from all the 
LMAP-authorized sending servers for the domains it handles. (a big hurdle!)
B)LMTP provides the list of all IPs that are authorized sources of email 
that can result in legitimate DSNs.  So if we can tell whether the DSN 
is the result of such an email, great!
To be confirmed:
It's currently requried that every DSN include the source IP of the 
initial email (including Received headers) that resulted in the DSN?
It's currently requried that it do so by including as an attachment the 
email, including Received headers, that resulted in the DSN?
All legit DSNs seem to have the property that in the raw body (including 
attachments) is a Received line containing the IP of an authorized 
source of the initial email.
DSNs that are bounces of email I never sent don't.
All legit DSNs seem to have the property that the first Received line in 
the attached email (or email header) is a Received line containing the 
IP of an authorized source of the initial email.
DSNs that are bounces of email I never sent don't.
How can B be enhanced to keep spammers from directly sending out 
DSN-mimicing spam with Received headers making it seem that the DSN was 
legit? 
What about the other uses for <>, beyond DSN (list needed!)

>>and all of them (worth their salt) are.  Otherwise you do expect the target
No, the source MTA (the first server to get the message) still queues in 
response to 4xx. 
What did I say that implies eliminating queuing, Claus?
If the origination MTA sends to an intermediate MTA that can't reach the 
destination MTA, the intermediate MTA would (in the scheme I'm 
proposing) send a 4xx to the origination MTA, where it would be queued.

><and also said>:
Sorry if I missed them.  

Hector - I think your definition of false positive is wrong.  WCSAP 
bounces (err... refuses) MAIL FROM
addresses like [email-masked], and [email-masked]!  
The former is arguably not a big deal, but the latter is a bit of a 
showstopper.
If I read you right, you think blocking MAIL FROM these addresses isn't 
a false positive.  It is in my book.

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 18:48:01 -0500

> connect and reaching the RCTO TO: stage.  It is functionally designed to

Allows me to rephrase this - This is a requirement that the receiver is 
required to send the delivery failure notice back to the sender. It is 
not a requirement that the address for that notice should exist and be 
functional, or in your case be able to reach the RCPT TO stage. 
Therefore, in practice this requirement is sometimes ignored since no 
one wants to be sending email to non-existant addresses anyway, and many 
filtering systems for spam will not send back a failure notice.

Yakov

-------

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 19:24:37 -0500

----- Original Message ----- 
> filtering systems for spam will not send back a failure notice.

Sorry, I disagree with your interpretation of the current specification.

A Mail From:  must be valid if all other RFC specifications is to fit.

Now, if it is a APPLIED in such a way that it "yields" incorrect results,
then it is NOW your job, to help developers use this failure in a functional
defined manner.

In Wildcat,  when bounce notifications fail to reach their destination after
the retries is exhausted (default is 72 attempts, 3 days, 1 per hour)  the
address is automatically BLACKLISTED (optional for admin).

The RETURN PATH: must exist, otherwise the system fails at all technical
levels.  It is for this reason, in practice, it works for compliant systems.
WCSAP is proven this.  For non-compliant systems, it is problematic.

I am failing to see why so much resistance?  All you are basically proving
is how the CURRENT specification needs to be tighten up.

If you are LOOKING for a solution that works with the current system with
all its idiosyncrasies, then you are LEFT with only one solution, and
EXTERNAL wrapper solution that is DNS based at stops entries into the
unsecured environment in the first place.

Why is so hard to see this?

---
Hector Santos

---

From: Bart Schaefer <[email-masked]>
Date: Mon, 1 Dec 2003 01:00:38 +0000

On Nov 30,  3:35pm, Hector Santos wrote:
}
} ----- Original Message ----- 
} From: "Bart Schaefer" <[email-masked]>
} 
} > You might believe it's a non-issue for you, but that doesn't make it a
} > non-issue for everyone who might become a victim of it.  Recommending an
} > approach such as your caller-ID technique is irresponsible, because it
} > can lead to indirect abuse of innocent third parties.
} 
} I fail to see how.   I would love to see an example.

As my original example doesn't seem to have gotten the point across, I'll
try it another way.

Let's suppose that your product with WSCAP included is wildly successful,
and 10,000 sites deploy it.  Collectively call these sites W.

Further suppose that some other vendor likes your suggestion and deploys
an equivalent system to another 10,000 sites.  Call these Q.

Now add spammer X and third-party site V.

X sends 1000 messages to each of the sites in W, forging V's domain in
the MAIL FROM:.  What happens to V's MX host?

Next X sends 1000 messages to each of the sites in Q, forging random
domains from W.  What happens?  (Remember that some fraction, possibly
all, of the hosts in both W and Q are performing the caller-id probes
with non-empty MAIL FROM: because they don't want to be mistaken for a
wormspew bounce.)

In the victim V case, an innocent third party has been DDoS'd by the
servers in set W.  That there currently exist other mechanisms by which
a similar DDoS could be caused is not justification for recommending
yet another one.

In the Q vs. W case, what stops all the systems involved from deadlocking,
as each site in W tries to caller-ID the non-empty MAIL FROM: generated by
the servers in Q, which then try to caller-ID the non-empty MAIL FROM:
generated by the servers in W, and so on around again?

} > } This is already controlled by server access and availability.
} > } The number of receiver threads are definable by the sysop with a
} > } server 421 greeting response of:
} > }
} > }         421 domain, Service not available. Try again later
} 
} > How is this a non-issue?
} 
} Retries are part of the functional specification of SMTP systems.

That's not what I mean.

How is it a non-issue _for the system that is issuing the 421 response_
that it has become so busy that it is _necessary_ for it to issue such
response?

Why is it a non-issue that timely receipt of email by your servers has
been interrupted, even if temporarily?

} > ALL of your mail is going to stop flowing with
} > this error until such time as the flood of caller-ID connections stops.
} 
} No, its not.  Again, I fail to see this.   I have WCSAP running for nearly
} 5-6 days now with very little issues that is being worked out.

That _you_ have WCSAP running is irrelevant.  The point is to consider
what happens when someone else has WCSAP running, and further that said
someone else (or maybe many someones) is hammering some unrelated server
(maybe yours, maybe mine, maybe the IETF's) with caller-ID callbacks for
mail that the unrelated server never originated.

} > you'll have those interruptions repeatedly until all the forged messages
} > drain out of the ISP's queues.
} 
} Not an issue.

Again, why not?

} But BY FAR, with EMPERICAL DATA on hand, that this is the best METHOD
} out there to combat illegal spoofing by spammers.

All challenge systems are highly effective when deployed in isolation.
The troubles begin when you consider what happens if they become widely
distributed.

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 20:12:34 -0500

Hector Santos wrote:
 >>one wants to be sending email to non-existent addresses anyway, and many
[..]
[..]

What I am trying to point out that your specific proposal is not 
compliant with the current architecture. What I do see is that the 
current SMTP specification has holes which need to be patched, and your 
proposal is addressing one of those holes.

Whether the RFCs require a valid return address, or not, in spirit or in 
letter of the law, is something Dave Crocker and Eric Raymond, and 
others, who worked on the original 821 and 2821 RFCs can tell us. But 
the fact today is that no one is expected to provide a valid address, 
and any system relying on this, will fail in some cases unless the 
existing RFCs are changed.

We understand that changes must be made. However, we need to justify 
these changes before imposing them on the entire Internet. For example, 
it is significantly more lightweight to verify domain/IP association via 
LMAP than do an RCPT TO callback. Both your proposal and LMAP address 
the same problem - forgery of the MAIL FROM address, except LMAP focuses 
on verifying the domain, while you are verifying the actual address. 
What we need to determine, is why should we go through the burden of 
verifying the actual address, when for the purposes of reducing forgery, 
verifying the domain is sufficient?

Let's say you have verified either the domain or the address, and the 
message in question turns out to be spam. In both cases, you are going 
to complain to the ISP of the domain, not the actual user! So why go 
through the trouble of verifying the actual email address, when a domain 
is sufficient?

Yakov

-------

---

From: Bart Schaefer <[email-masked]>
Date: Mon, 1 Dec 2003 01:38:26 +0000

On Nov 30,  6:20pm, Hector Santos wrote:
} 
} ----- Original Message ----- 
} From: "Bart Schaefer" <[email-masked]>
} 
} > }    receiver-SMTP MUST formulate and mail a notification message.  This
} > }    notification MUST be sent using a null ("<>") reverse path in the
} > }    envelope.  The recipient of this notification MUST be the address
} > }    from the envelope return path (or the Return-Path: line). "
} 
} > [That] is not a requirement that the original sender accept it.
} 
} Thats a non-issue.  Unrelated really.  A Callerid Verification system is
} not sending mail.  Half the battle is solve by being able to connect or
} not connect and reaching the RCTO TO: stage.

Another example:

As a SourceForge project administrator, I get notices of system upgrades
or downtimes and so on.  These messages are MAIL FROM: an address similar
to <[email-masked]> (I forget the exact address).

If such a message arrived at a host using your caller-ID system, your
system would connect to the sf.net MX, issue "MAIL FROM:<>" and then
issue "RCPT TO:<[email-masked]>".  Right so far?

The point that Yakov tries to make is that there is no RFC requirement
that sf.net respond "250 recipient OK" when presented with that RCPT TO:.
In fact, it probably responds "550 user unknown".  There is also no RFC
requirement that sf.net's sending server knows, at the time that it says
"MAIL FROM:<[email-masked]>", exactly what response sf.net's MX may
give when presented with that address as a RCPT TO:.

Your system would thus reject this perfectly legitimate message.  The
fragment of 2821 that you quoted above does not justify your use of this
RCPT TO: test for sender validity.  Is there some other standard that
you believe does justify it?

} But with that said,  the receiver must accept a NULL address.

In MAIL FROM:, yes.  That part truly _is_ unrelated to Yakov's point.

---

From: Matthew Elvey <[email-masked]>
Date: Sun, 30 Nov 2003 18:05:05 -0800

>a similar DDoS could be caused is not justification for recommending
Without WSCAP, wouldn't V likely receive tons of DSNs for email it never 
sent?  Replacing those with WSCAP probes seems a net reduction in the 
impact of the DDoS to me.

>generated by the servers in W, and so on around again?
Good example.  This is indicative of the Challenge-Response nature of 
WSCAP, as you say below.  But haven't some C-R systems solved this 
problem, or do they just have klugey partial solutions that only work if 
only their C-R system is used?  If not resolvable, we have a dealbreaker.

---

From: Bart Schaefer <[email-masked]>
Date: Mon, 1 Dec 2003 02:56:07 +0000

On Nov 30,  6:05pm, Matthew Elvey wrote:
}
} On 11/30/2003 5:00 PM, Bart Schaefer sent forth electrons to convey:
} 
} >In the victim V case, an innocent third party has been DDoS'd by the
} >servers in set W.  That there currently exist other mechanisms by which
} >a similar DDoS could be caused is not justification for recommending
} >yet another one.
}
} Without WSCAP, wouldn't V likely receive tons of DSNs for email it never 
} sent?  Replacing those with WSCAP probes seems a net reduction in the 
} impact of the DDoS to me.

DSNs are sent only when the spam's RCPT TO: is invalid -- and then only
when the border MX is not able to reject with a 5xx.  Caller-ID probes
occur for _all_ mail, whether destined for a valid recipient or not,
because Hector's stated purpose is to test at MAIL FROM: and prevent the
spammer from getting as far as RCPT TO:.

Unless I've missed something, the volume of WSCAP probes is going to be
significantly greater than the volume of DSNs would ever be.  And as I've
said already, the existence of another DDoS does not justify recommending
a new one.

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 30 Nov 2003 22:10:36 -0500

----- Original Message ----- 
From: "Bart Schaefer" <[email-masked]>
To: "ASRG" <[email-masked]>
Sent: Sunday, November 30, 2003 8:00 PM
Subject: Re: [Asrg] 0. General - Inquiry about CallerID Verification

> As my original example doesn't seem to have gotten the point across, I'll

Bart, I did not fail to see what you were saying nor fail to see the
possibility.   What I fail to see is how this is any different from any
other malicious system performing the same attack where there is ANY kind of
validation.

This is more of a scalarbility issue where there are solutions.

One possibility is use a client/server/farm framework, couples with IP
analytics. In short, a centralized tracking system.   But this applies to
EVERYTHING.  Thats my point.

let me summarize your thinking:

W = 10,000 sites with Callerid Verfication (WCSAP)
Q =  10,000 sites with Callerid Verfication (other)
V =  3rd party site.
X =  Spammer spoofing V site

> the MAIL FROM:.  What happens to V's MX host?

Assuming V will fail to validate the return path provided by X,   WCSAP will
track X IP address vs domain to stop any further validation against the V's
MX host.

If X uses a diifferent connecting IPs, this also can also be sensed.

> domains from W.  What happens?

Same as above.

> servers in set W.

Note if its not tracked.

> generated by the servers in W, and so on around again?

See commment above.

> that it has become so busy that it is _necessary_ for it to issue such

Like everything else, it can be attacked or consume for other reasons that a
WCSAP like system.  The server has to be prepared for it.   If not, then DNS
load balancing is required by the site.

Again, its not scalability question is what you are really asking, and from
that point of view, I fully agree. It doesn't have be because of a SPAMMER.
It could be a regular system sending "lots of mail" and even in this case,
tracking has be done to address redundancy.

> been interrupted, even if temporarily?

Again, it a non-issue because it CAN happen anyway and lets remember, this
is ONLY an issue for non-compliant systems and for ghost systems.   So if
the concept is WIDELY deployed,  senders and receivers will be probably
configured and spammers will be reduced to a bare minimum.

> Again, why not?

Already stated above.

> All challenge systems are highly effective when deployed in isolation.

(As a side note,  I would not catalogized this as a challenge system.
Technically, challenge concepts are dynamic in nature, where it is often
initiated by the server upon contact (or the initial stages) with a client
matching the challenge.)

> The troubles begin when you consider what happens if they become widely

Than its a scalability issue and it applies across the aboard.

Again, lets remember the main point here.  The SMTP state point, MAIL FROM:
is calling a black box validation function:

            MAIL FROM:   ----->  BLACKBOX VALIDATION.

WCSAP is just one method to do this.  It is highly effective, but present
scalibility/loading issues that need to be incorporated in its design.    As
a blackbox,  design a better MAIL FROM: validation concept, and it is easily
replaced.  No change to our SMTP server.  So from my point of view our SMTP
server is design to address a strong MAIL FROM validation concept.  That is
ALL I am proposing YAKOV et al to focus on. The PROTOCOL!   Then we can come
up with solutions and even then,  it is not going to be 1 single solution,
unless it something that really works 100% and addresses scalability issues.

With WCSAP (Source Code provided with our Compiler/SDK plus pack),
developers or companies can CHANGE it to comform to what they want it to do.

---
Hector Santos, CTO

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 30 Nov 2003 22:21:52 -0500

Hector Santos wrote:
[..]
> unless it something that really works 100% and addresses scalability issues.

Well my question here as per the other thread, is why do we need to do 
verification at the MAIL FROM point for the address itself? In the 
Internet realm shouldn't verifying the domain/IP be sufficient since the 
owner of that IP/domain is in responsible for it, not their individual 
users?

I have no problems with applying everything to the protocol itself. 
However, as the protocol stands right now, there is no need to rewrite 
it. Nothing steps *anyone* from doing checks at each state of the SMTP 
transaction, and many do so.

What I am getting from this and other related threads, is a need for the 
group as a whole to come up with a solid and concrete document which 
outlines the protocol and matches the proposals against the weak points 
in the protocol. That work has already begun with Alan, and we can 
perhaps shift our discussion to that topic on-list as well. We can start 
off with discussing what to do at the connect point of the SMTP 
transaction, and different proposals at that point. Therefore, I am 
chaning the subject and would like to continue the discussion on that topic.

We can start off with listing different approaches and their pros and 
cons here, as well as what possible tricks spammers use to get around them.

Yakov

-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
"All that is gold does not glitter" (J.R.R. Tolkien)
-------

---

From: "Eric S. Raymond" <[email-masked]>
Date: Sun, 30 Nov 2003 23:11:54 -0500

> existing RFCs are changed.

The spirit of the law, at least at the time I was involved with
RFC2822, was certainly that the return address was required to be
valid.  As for the letter...

Section 3.6.7 of RFC2822 says:

   The trace fields are a group of header fields consisting of an
   optional "Return-Path:" field, and one or more "Received:" fields.
   The "Return-Path:" header field contains a pair of angle brackets
   that enclose an optional addr-spec. [...]

The addr-spec is optional because it maty be omitted, as in daemon
bounces.  RFC2822 continues:

   A full discussion of the Internet mail use of trace fields is
   contained in [RFC2821].  For the purposes of this standard, the trace
   fields are strictly informational, and any formal interpretation of
   them is outside of the scope of this document.

Thus, RFC2822 does not mandate that the address be valid.  We are 
referred to RFC 2821.  Section 4.1.1.4 reads:

   When the delivery SMTP server makes the "final delivery" of a
   message, it inserts a return-path line at the beginning of the mail
   data.  This use of return-path is required; mail systems MUST support
   it.  The return-path line preserves the information in the <reverse-
   path> from the MAIL command.  Here, final delivery means the message
   has left the SMTP environment.  Normally, this would mean it had been
   delivered to the destination user or an associated mail drop, but in
   some cases it may be further processed and transmitted by another
   mail system.

   It is possible for the mailbox in the return path to be different
   from the actual sender's mailbox, for example, if error responses are
   to be delivered to a special error handling mailbox rather than to
   the message sender.  When mailing lists are involved, this
   arrangement is common and useful as a means of directing errors to
   the list maintainer rather than the message originator.

I read this to mean that the Return-Path must, if present, be a
valid mailbox for a responsible person (note the MUST).
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

---

From: "Chris" <[email-masked]>
Date: Mon, 1 Dec 2003 15:05:24 +1030

Doesn't all this discussion prove that the interpretation of a protocol can
vary?

Therefore any attempts to use a "feature" that is disputed will probably end
up breaking some software

Regards
Chris

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of Eric
> S. Raymond
> Sent: Monday, December 01, 2003 2:42 PM
> To: Yakov Shafranovich
> Cc: ASRG; Dave Crocker
> Subject: Re: [Asrg] 0. General - anti-harvesting (was Inquiry about
> CallerID Verification)
>
>