Subject: Re: MTAmark (was: Reality check please)

From: Markus Stumpf <[email-masked]>
Date: Sat, 12 Jun 2004 01:04:11 +0200

On Fri, Jun 11, 2004 at 07:42:38AM -0400, Hector Santos wrote:
> Nevertheless, one common thing I see around here is this "BIG" vs. "SMALL"
> thing, and I guess, the small end of the spectrum doesn't count as much for
> MARID.  I may be off based, but if this is true, this would be another
> mistake in the making as the early wide spread adoption will come from the
> larger pool of smaller systems.  I believe SPF has shown this very clearly.
> However, I will note that we jumped on board only when we saw AOL.COM, a
> major source of spam, began to support it.

I'd really like where all this urban legends come from. Isn't anybody
on this list runing a mailserver of some size?
Out of 500,000 messages with about 90% marked spam and viruses (this is
450,000 messages) I had
    18164 hotmail.com	(4.03%)
    15013 yahoo.com	(3.36%)
    11620 web.de
    10069 gmx.de
     8671 AOL		(1.92%)
     8172 msn.com
     2750 yahoo.de
AOL supporting SPF didn't change anything, besides some media hype.
Give me one site that really utilizes the client side of SPF? Serious
business companies CANNOT do it, as it will raise their false positive
rate and nobody will do that. The above domains account (+GMX.*) for
about 15% of all messages (and quite some messages from GMX and WEB.DE
are legal, as we are in DE).

So the main question is: Why do spammers use hotmail, yahoo, AOL?
Simple answer: you cannot block those domain totally, but if the
domain is "lamer.de" you don't sleep bad at night blocking it.
As soon as those "big" domains publish MARID records and the
technique is well established, spammers will immediately switch
and ONLY do what they already do now: abuse "small" domains. And these
are easy to find, as they don't publish MARID records.

23410 unique domains only injected one message.
5635  injected 2 messages
only about 15% of all messages already NOW account to "the big ones".

So which problem are we trying to solve?

	\Maex

-- 
SpaceNet AG            | Joseph-Dollinger-Bogen 14 | Fon: +49 (89) 32356-0
Research & Development |       D-80807 Muenchen    | Fax: +49 (89) 32356-299
"The security, stability and reliability of a computer system is reciprocally
 proportional to the amount of vacuity between the ears of the admin"

---

From: "Hector Santos" <[email-masked]>
Date: Sat, 12 Jun 2004 03:06:39 -0400

----- Original Message ----- 
From: "Markus Stumpf" <[email-masked]>
To: "IETF MARID WG" <[email-masked]>
Sent: Friday, June 11, 2004 7:04 PM
Subject: Re: MTAmark (was: Reality check please)

> 450,000 messages)

The rate is on par with industry average.

> AOL supporting SPF didn't change anything, besides some media hype.

It did for us.

> Give me one site that really utilizes the client side of SPF?

Not sure what this question ask.

> Serious business companies CANNOT do it, as it will raise their false
positive
> rate and nobody will do that.

Whats the difference between a serious business and a just plain anyone else
running a legitimate mail server servicing end-users, internal or otherwise?

> Simple answer: you cannot block those domain totally,

For hotmail,  MCEP works.

For yahoo, you have a problem as they go agains the grain of standard
operations contributing to the spamming problem.

For AOL, they are the best of the "large ISP" bunch.

Explain more below.

> are easy to find, as they don't publish MARID records.

Well, the harder (and complex) you make it publish,  you will find a barrier
to implementation.

In my view, as soon as the "big" domains publish MARID records, they will
spammers spoof them even more due to the "CodeRed Principle" that every
modern virus/hacker and now spammer now enjoys, and that is there will
always be a segment of market place that are still not updated with new
anti-spam logic.

> So which problem are we trying to solve?

For me, the focus is on solving the "anonymous mail abuse" using SMTP
technical compliancy with total disregard on the intent of the mail sender.

Hotmail publishes a _EP record, so this has help stop some of its spam.

But for our system, we use a CBV as the final analysis because in our view,
ultimately, the complete address must be verifible.   We use white/black
listing, RBL and LMAP methods as initial checks with one goal in mind -
eliminate the obvious and the need for the final CBV.

AOL is the best of the bunch because they support dynamic SMTP Local User
Validation. So the CBV works great in elimination all AOL spam that get pass
the initial checks.

YAHOO is problematic (doesn't help) because it validates ALL users. That is
why you got more spammers using YAHOO.

I forget, but HOTMAIL was also problematic, but not for the same reasons
YAHOO was.

Local User Validation is (should be) a important part of the anti-spam
effort/design.  Not only will it help eliminate the overhead in MARID lookup
requirements but it will also reduce your bounce requirements which is a
major part of the SORBIG-based virus dual-tier distribution logic.

Just consider this:

  Connecting IP
  HELO
  MAIL FROM
  RCPT TO:

If your RCPT TO is not valid, then you need not bother to check for MARID.

Sure, probably deemed an implementation issue, but for me:

    anti-spam is directly proportional and a function of (IP, HELO, MAIL
FROM, RCPT TO)

to ignore RCPT TO, well, you are just adding a major burden to your system
when the odds are very high the mail (about 40% bad RCPT) was going to be
non-deliverable in the first place.

Comment from customer:

"Just wanted to tell you what a great job the WcSAP piece is!

I had our network admin (From my work), help me set it up, and after about
30
minutes of learning/tweaking, it has been working great! Dozens of messages
blocked, and no new virus warnings in the last hour. -He is a die-hard Linux
freak and after he took a crack at studying the communications from multiple
mail servers, log files, and my machine's logs, he was totally impressed how
well it works. Our Linux mailservers use a similar type of verification, but
he mentioned your version was very efficient in how it performed the
verifications. That means alot to me since he hardly ever says anything
positive about anything that runs on Windows.

Great update!

Thanks!
-Chris Weis"

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

---

From: "Jon Kyme" <[email-masked]>
Date: Sat, 12 Jun 2004 09:49:48 +0100

> So which problem are we trying to solve?

There was a little discussion in a "measuring MARID" thread. It appears
that around 4% of sending domains currently publish SPF. As you point out,
"big domains" don't account for a large proportion of our messages. I
suspect that much SPF evangelism is faith-based.

---

From: Markus Stumpf <[email-masked]>
Date: Mon, 14 Jun 2004 21:10:45 +0200

> Not sure what this question ask.

Let me see one serious site that rejects mail from AOL.com because the IP
that sent the message was not on the SPF list.

> running a legitimate mail server servicing end-users, internal or otherwise?

Nothing. Neither of these can reject messages based on SPF information.
I, as a private person can do, if I don't care to piss off some friends.

> major part of the SORBIG-based virus dual-tier distribution logic.

For Sober use:
    if HELO.host == MAILFROM.username"."MAILFROM.tld
	if MAILFROM.username == MAILFROM.domain pass()
	else reject()
but I agree on the local user validation.
That's the reason why most spam engines use the reverse MX path. We have a
special setup with a special backup MX and catch about 40% of all spam that
way.

	\Maex

---

From: Matthew Elvey <[email-masked]>
Date: Mon, 14 Jun 2004 21:45:27 -0700

(Posted on 6/9 just to Markus, perhaps by mistake; haven't heard back; 
the minimum price of a domain is important!)

On 6/9/2004 2:58 PM, Markus Stumpf sent forth electrons to convey:

> <Agree with you up to this point.>
>
> And a point which is deliberately ignored is the problem of 0.10 USD
> throwaway domains and short-TTL bot networks.

Sorry, where can I get a domain for $0.10 US?  (And I don't mean 
foo.cjb.net or foo.cjb.co.uk)
If these are coming or here, as you and Gordon seem to be saying they 
are, then MARID won't work as well as I had thought it would.
You think an icann-authorized TLD will start selling domains at this 
price?  Please explain in more detail.

> Yeah, I know, this will be
> solved anytime later with accreditation services.
>
> CSV:
>  
>
>> Is an SMTP client authorized to use a particular domain name in its
>> SMTP EHLO command?  [CSV] attempts to answer this question.  It
>> suffers from the fact that the EHLO name has a tenuous relationship,
>> at best, with the contents of any mail message.
>>   
>
>
> So what? It is MTA authorization records not message authorization
> records, even if the group morphed to it.

Indeed!  Watch this point be studiously ignored.

> A EHLO identifier is the
> way one MTA tells another MTA about himself and his identity. It is
> an important fact to know if this identity is faked. However this
> could probably also partially accomplished by requiring the EHLO
> to match at least one PTR record and have the forward lookup of this
> record match the connecting IP (aka EHLO must match paranoid lookup).
>  
>
Some of the syntax and semantics of SPF records provide useful 
additional functionality without weakening the effectiveness with which 
an MTA is authorized by a domain.
The benefit is that it allows every legit MTA to pass MARID inspection, 
even the 1-10% of legit MTAs don't have control over rDNS, and hence 
don't pass the rDNS test.  And yet it ties that MTA to a domain that has 
a valuable reputation.  That being the goal of MARID.

> RMX/SPF:
>  
>
>> These suffer from the fact that the MAIL FROM address really 
>> describes where to send NDRs to, and in many third-party and 
>> forwarder situations this address is unrelated to the domain that is 
>> resending the message.   
>
>
> This has not more or less backdraws than to check a Resent-From:
> This provides no more or less authentication for a faked Resent-From:
> of a throwaway domain and in that case if the message is not checked
> during the SMTP transaction the bounce goes where to?
> The faked envelope sender? To the SUBMITTER, which is an address
> the originator of the message will never be able to check for mistyped
> addresses. What have we won?
>  
>
Nothing but big headaches, IMO.

> ... the XML hype ...
I think the vocal opponents of XML have RSI.
Or perhaps their jaws dropped open and they're speechless.

---

From: Douglas Otis <[email-masked]>
Date: Mon, 14 Jun 2004 23:04:19 -0700

> price?  Please explain in more detail.

To slip past a filter, new domains are simply the price of doing
business for those that abuse the system.  Unlike an IP address, a
history can be expected of a domain name that is not possible with just
an IP address.  New domain, offer little credit.  Tried and true, offer
full credit.  Tried and bad, no credit.  Using a domain name with a
reasonably good authentication practice is tremendously safer than
relying solely upon the address whether or not an accreditation service
helps administrate the lists.

> > Yeah, I know, this will be solved anytime later with accreditation
> > services.
> >
> > CSV:  
> >

Although the problem can be seen as forged headers, much of this junk
traffic comes from sources that can be tracked and this is made easier
through the use of names rather than IP addresses where accountability
is otherwise doubtful if not impossible.  

> > A EHLO identifier is the way one MTA tells another MTA about himself
> > and his identity. It is an important fact to know if this identity is
> > faked. However this could probably also partially accomplished by
> > requiring the EHLO to match at least one PTR record and have the
> > forward lookup of this record match the connecting IP (aka EHLO must
> > match paranoid lookup).
>   
> an MTA is authorized by a domain.  The benefit is that it allows every
> legit MTA to pass MARID inspection, even the 1-10% of legit MTAs don't
> have control over rDNS, and hence don't pass the rDNS test.  And yet it
> ties that MTA to a domain that has a valuable reputation.  That being
> the goal of MARID.

The use of an SRV record has the advantage of returning all the needed
information in a single query where both authorization is discovered and
the authentication is validated.

-Doug

---

From: Matthew Elvey <[email-masked]>
Date: Tue, 15 Jun 2004 00:38:09 -0700

>helps administrate the lists.
True, but I would like to know if domains will be avaiable for << $5.  
It'll impact reputation service / RHSRxL scalability, but MARID will 
still work.
IF the price approaches $0, all domains will need accredidation, and 
RHSBLs become unscalable, as they will take up more space than regular 
IP BLs.
We must assume spammers can warehouse domains and allow them to age, 
unused, until not new.

>the authentication is validated.
Yup, and CSV's CSA doesn't rely on rDNS.  Pretty nifty.  (Though all of 
hotmail's SRV records would take up much more cache space than their SPF 
record, even in XML format.)  I still need to mull over what SPF allows 
that CSA doesn't, and whether (IMO) CSA can or should be extended.  
Also, piggybacking on the deployed SPF records and creation tools is 
tempting. (It seems to have hooked Microsoft.)

---

From: "Hector Santos" <[email-masked]>
Date: Tue, 15 Jun 2004 05:06:28 -0400

----- Original Message ----- 
From: "Markus Stumpf" <[email-masked]>
To: "IETF MARID WG" <[email-masked]>
Sent: Monday, June 14, 2004 3:10 PM
Subject: Re: MTAmark (was: Reality check please)

> that sent the message was not on the SPF list.

The continued embedded usage of tthe word "serious" keeps me scratching my
head <smile>

The mere fact a mail server is made part of the common and world wide
network, makes it a "serious" operation with standards and BCP
responsibilities.  I don't mind saying (because I believe it very strongly),
putting up a mail server with the intent of running a mail operation but
failing to adhere to common practice and standards, well, is bordeline
unethical and malpractice that puts you at risk with your peers.   You seem
to put alot of weight on the "its my ball" syndrome with a heavy weight
placed on a system admin (you)  policy to decide what mail transactions are
acceptable.  However, from what I have seen and personally experienced, it
is all based on some criterias that in my view, "serious" mail server
operations do not usually typically use.  Auto-generated permanent blocks
based on what I believe are weak criterias is what I see on your system.
Oh well,  its your toy.  <g> As long as you keep your automated blocking
methods to your system only, thats fine.  But once you get into the game of
passing on this highly false "blocking IP database" information to a public
and general database, well, that is just plain "wrong."

Anyway,  AOL?

Markus, in this case, if the sender IP using a AOL sender domain is not
validated by the AOL SPF policy, then for this specific situation,  you have
AOL neutral result.  A no-decision based on a SPF neutral criteria.  This
alone can not be used for rejection per AOL's SPF policy.

But any "serious" anti-spam system is not going to rely solely on SPF or
MARID, period.  It doesn't solve or cover all possibilties, so you need to
incorporate other methods as well.

There would be several ways to reject this:

o Use popular RBL sites to cover the known reputation of the IP.
o Use Local Domain/IP spoof checking.  MARID/SPF not required.
o or use a CBV against AOL's local user validation which they support.

---

From: Matthew Elvey <[email-masked]>
Date: Tue, 15 Jun 2004 03:07:36 -0700

On 6/11/2004 11:29 AM, Markus Stumpf sent forth electrons to convey:

>Oups, hit send to early :/
>
>On Wed, Jun 09, 2004 at 04:49:26PM -0700, Matthew Elvey wrote:
>  
>
>Let's make a real life deployment test:
>How much people that use the Internet "for fun" (some web browsing, a
>bit of email and maybe chats) do you know?
>  
>
~100

>How many of them have their own domain?
>  
>
~20

>How many of them will be able to specify (even with software aid)
>correct SPF records that will not make it totally impossible for them
>to send eMails?
>
~10 (I live in San Francsico, else likely ~5)

> A big german hoster has registered - according to their
>website - more than 2 millions of domains for their users. A large
>portion of them are domains used like business cards or are the hobby
>of people like   joe-user.de.
>Do yopu expect them or anyone else for them to be able to specify
>correct SPF records?
>
No.
Also you forgot that in an SPF world, they'll have to configure their 
mail clients.  What about the ones that use many different From:'s
http://wiki.fastmail.fm/wiki/index.php/FromLine, e.g. with Mozilla's new 
Multiple Identity Support:
http://www.mozilla.org/projects/thunderbird/identities.html?  But I'm 
not advocating SPF in its entirety, just "some of the syntax and 
semantics"!

> Some folx which are my friends have such domains.
>They send out their emails through the mailserver of Internet on demand
>ISPs and DSL providers and and and. They change (with the aid of
>programs) their IPs, ISP and mailserver like other change their
>underware. Managing correct SPF records for them is hell.
>  
>
No, you misunderstand my proposal. (Thanks for setup though!) We still 
use EHLO to select the domain to authenticate; we ditch the PRL and 
SRS.  So our friends DON'T HAVE TO DO ANYTHING!  The big german hoster 
just has to make sure that the EHLO its server sends out is a domain 
that has an SPF record that validates its IP (and 
reputation/accredidation). SRS doesn't need to be deployed!
In my proposal, most domains DON'T NEED  new DNS records AT ALL.
(And none of this RFrom stuff is necessary either.)  Near lightning-fast 
deployment is feasible.  And we're still providing and using M.A.R.I.D. 
effectively.
How cool is that? 

>Do you know what hell it will be for a large ISP offering mail relay for
>their customers to change the IP of an outgoing SMTP server?
>That's why I wrote "reality check". It is one side to design a kewl
>and nitfy system and it is a completely other thing to make it work
>in the real world. How many systems fail in the real world you can read
>each day in the newspapers about.
>  
>
I agree.

On 6/11/2004 11:15 AM, Markus Stumpf sent forth electrons to convey:

>On Wed, Jun 09, 2004 at 04:49:26PM -0700, Matthew Elvey wrote:
>  
>
>What is the difference with regards to MARID and MARID records
>between a domain  example.com  and  foo.cjb.net or foo.cjb.co.uk?
>  
>
Big difference. Generally a cjb.net (a second level domain) will have a 
reputation (or get RHSBL'd), and cjb.co.uk (a third level domain) will 
too. 
the domains I listed generally won't have reputations separate from 
their parent domains.

>"Full featured" SLDs are as cheap as 5 USD (google for "cheap domains").
>  
>
(I know.  I said so tonight.)
You said I could get domains for $0.10.  I asked Where?  No answer.
I guess you're taking back your claim!  Good!

>Domains ARE cheap, most bundle it with disk space and webservers.
>  
>
Yes, but not so cheap that they make RHSBLs unscalable.  As was 
discussed (with stats) a while ago on the list.
1GB of RAM costs a lot less than 50GB of RAM.  (the $5 to $.10 ratio)