Subject: RE: Unified SPF: block versus factored records for HELO and MTAMA ark scopes

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Thu, 24 Jun 2004 21:30:01 -0700

> Most Unix MTAs including sendmail, exim, qmail and (I believe) postfix
> fork off a process per SMTP session which exits at the end of the
> session.  In this case, any effort beyond that to validate the single
> IP used in the session is wasted, and the local DNS cache is where the
> info will be remembered.  My impression is that there are a lot of
> large sites whose MTAs work this way.

This is certainly the case historicaly, but this approach is largely
due to the very poor performance and stability of early editions of 
pthreads, problems that are long since fixed.

I do not know what the situation is where dedicated edge servers 
such as spam filtering systems are concerned. I strongly suspect 
that these are based on very different architectures. Perhaps someone
could go and look at what some of these systems do.

Even on a UNIX box there are good reasons to prefer a per thread model
to a per process model, process creation and teardown in UNIX is only 
lightweight compared to other O/S. 

There are also other ways that a UNIX system may be organized. For 
example if I was operating a large cluster of mail filters I would 
hive off the task of fetching MARID data and resolving all forms
of reputation and accreditation data in a separate system that 
performed caching.

I would not want to store state in the mail server itself for the
reason you mention and also because that would load up my external
connections (which I pay for) rather than my internal 1Gb ethernet
connections which are essentially free.

So no, I do not accept this as much of an argument. Block records 
allow better efficiency in either case.

The only argument I see for factored records is on the maintenance
side where the records are being synthensized automatically via
some form of dynamic DNS. If you have block records you have to 
either define a custom update protocol (trivial via a web service but
must be done) or the dynamic DNS has to read the record, work out the 
update and write back the update.

Of course this could also be solved if the mailer in this case used
the sender header to give a unique machine address e.g.:

Sender: [email-masked]  
From:[email-masked]

So the MARID record for pbaker-pc.verisign.com would consist of a 
dynamic dns updated record for my laptop PC and a link to another 
record giving info like accreditation data for the domain as a whole.

OK we don't need factored records here either.

		Phill

---

From: "John R Levine" <[email-masked]>
Date: 25 Jun 2004 00:43:26 -0400

> lightweight compared to other O/S.

Depends how big the process is.  Forking sendmail takes forever, but
qmail's server is small and snappy.  I realize that one can do threads on
Unix boxes, but I don't think that many MTAs actually do so.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Information Superhighwayman wanna-be, http://iecc.com/johnl, Mayor
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: Theo Schlossnagle <[email-masked]>
Date: Fri, 25 Jun 2004 01:05:15 -0400

> could go and look at what some of these systems do.

Our Edge appliance (and Ecelerity MTA software) does not use a per 
process model.  It actually doesn't use a per thread model either.  It 
is a hybrid model where non-blocking ops are put into a small group of 
event engine threads and blocking ops are put into a larger group of 
worker threads.

As DNS queries have no good reason to block (there are a handlful of 
free nonblocking DNS resolving libraries out there) we can realize high 
concurrency and throughput with DNSBL, SPF, DomainKeys, etc. all 
performed inline.  100,000 concurrent SMTP sessions all performing DNS 
based lookups, MySQL lookups, ldap lookups, virus-checks and a slew of 
other things in real-time (during the SMTP session).

> lightweight compared to other O/S.

Threads have costs too.  Smaller than processes in most cases and 
almost negligible if they are user-space threads.  However, each has 
its pros and cons and you have to carry around that thread-stack.  So 
one thread-per connection can be a big big waste.

> of reputation and accreditation data in a separate system that

Yes... perhaps not "separate", but certainly cached.

> connections which are essentially free.

You can used a replicated data store.  Shared consistent replicated 
cache across the mail servers.  Much better and more reliable than a 
"backend store" as they usually have to be over-engineered to avoid a 
single point of failure.

// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
// Ecelerity: fastest MTA on Earth

---

From: Rand Wacker <[email-masked]>
Date: Fri, 25 Jun 2004 09:11:51 -0400

> Unix boxes, but I don't think that many MTAs actually do so.

I'm not going to disagree that forking on Unix takes longer than 
utilizing a long-running thread, so that's why we (Sendmail) have been 
implementing these checks in external milters, which are long-running 
threaded processes which the MTA can talk to.

Existing perceptions of architecture should not be a major burden on the 
protocol design process; most of those perceptions are out-of-date 
anyways. ;)

-Rand