Subject: SMTP Minimum Retry Period - Proposal To Modify Mx

From: "Sabahattin Gucukoglu" <[email-masked]>
Date: Fri, 09 Jan 2004 16:33:20 -0000

Hi people,

I have a friend whose on ADSL and who wants to run his own email service, 
but is periodically disconnected.  He likes his mail delivered straight to 
his primary host, but his mate is there across the other side of the 
country if ever he's out as a backup MX.  Now, every time he uses the 
connection, which is at least once every few hours and at most once a day, 
his connection automagically reestablishes itself.  While he's been down, 
though, any mail destined for him has gone to his mate, who only delivers 
mail once every four hours or so.  By misfortune, my friend has gone down 
just as a mail was being queued for sending, in a place far away.  Being 
down, he can't take the mail, so his mate takes it for him.  He must wait 
four whole hours, assuming he has come up after the delivery of that 
message to his mate, even if the originating client was willing to deliver 
it sooner - say, thirty minutes, if his mate hadn't been around.  There's 
just one more condition - his mate, though great as mates go, is an anti-
RBL purist.  He refuses to use RBLs.  His mate is picking up spam which he 
would have rejected, had he been responsible, and it isn't reasonable to 
parse the mail after the mail is accepted, in particular because he's 
serving other people and doesn't justify this kind of intrusive filtering.

Now, all that was mostly hypothetical - based on a true story.  If my 
friend is some poor soul on ADSL, and his mate (that is, friend, just in 
case - buddy for the Americans) is a dynamic service provider with backup 
MX facilities or some organisation who can't reasonably be expected to 
alter their mail service policies to be in line with that of my friends 
who are of my anti-spam, direct-to-destination kind with no wish to be at 
the mercy of our ISPs, we need some mechanism to tell the SMTP client to 
keep trying to deliver to a particular MX(es), for a particular time(s).

My proposal: an extension to the MX record in the DNS, which must be 
backward compatible with existing MX records - that is, non-conformant 
mailers must not be confused by the new form of the record.  The creation 
of a new record type must only be necessary if that record type may 
facilitate many such extensions, not just this one.  This use of the MX 
might look like:
example.org: mx 10 example.org 86400
(Or, if the record - new or existing - is used extensibly, something like 
"mx 10 example.org MRP86400")
Where 86400 is the minimum number of seconds (1 day) for which the MX must 
be tried before those of lower priority must be tried.  If other MX 
records exist with same preference, then the same rule applies to those 
hosts, except that the delivery timeouts occur in parallel, and no MXs of 
lower preferences are to be tried until all MXs at this preference have 
been tried - that is, if:
mx 10 mx1.example.org 60
mx 10 mx2.example.org 30
mx 20 mx3.example.org
Then mx1.example.org and mx2.example.org will be tried simultaneously 
(that is, one after another but immediately following as in normal MX 
selection and delivery attempt behaviour), with the client giving delivery 
attempts whenever it usually does at whatever intervals it normally uses 
until the timeout occurs on mx2, at which point only mx1 is continually 
assaulted with the same algorithms as before, until it to is deemed 
unusable - 30 seconds later (hypothetical numbers, we'd normally be 
talking many hours!).  Only at this point is mx3 tried, which may or may 
not have such a minimum time given, and the same algorithm procedure is 
used at this preference.  As you see here the mx3 is listed without the 
MRP, which may be away of avoiding compatibility issues.

Where local policy dictates differing delivery times and intervals, the 
time given in the DNS must always be respected - so, if you deliver first 
at 30 minutes, then at 60 following, then 2 hours, then 4 etc. in a 
typical backoff style, then the sum of these times preceding must be 
waited against the value in DNS (in seconds).  If the total time spent 
attempting delivery to this host is now exceeded, then - at discretion of 
administrator - the unreachable host may be tried once more.  In any 
event, it is now permissible to proceed to lower-priority MXs.

Since this is meant to be backwards-compatible, there's a possible 
argument that mandating the attempts for the MRP for compliant systems is 
a bit counter-intuitive; I'm hoping that enough systems could do this to 
make it worthwhile, and there's no other way of achieving the same thing 
without an MX knowing when another MX of higher preference was both 
available and not available (the latter obviously being very hard to 
advertise, because it aint available!) with which knowledge the MX could 
either accept the mail or refuse to accept mail on the grounds that the 
higher MX was already available.

Of course, this little flash of brilliance ( :-P ) is only going to work 
if we can bend the DNS without anyone noticing.  Which, chances are, they 
will.  RFC 2821 which replaced RFC 974 before it on this matter makes no 
reference to the exact way in which MX records should be parsed, so that 
it could be chancy.

Still, what do you all think?

Cheers,
Sabahattin
- -- 
Thought for the day:
    The only thing that hurts more than paying income tax
    is not having to pay income tax.

Latest PGP Public key blocks?  Send any mail to:
<[email-masked]>

Sabahattin Gucukoglu
Phone: +44 (0)20 7,502-1615
Mobile: +44 (0)7986 053399
http://www.sabahattin-gucukoglu.com/
Email/MSN: <[email-masked]>

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 09 Jan 2004 09:18:20 -0800

Why doesn't your friend use ETRN to trigger delivery of his queued mail 
from his mate whenever he gets online?

That way, the 4-hour delay is avoided without requiring global changes to 
the Internet infrastructure....

---

From: "Sabahattin Gucukoglu" <[email-masked]>
Date: Fri, 09 Jan 2004 18:45:57 -0000

> from his mate whenever he gets online?

He doesn't want his mate getting his mail while he's not available if he 
will be available shortly after.  The idea is to restrain clients from 
passing onto the next MX, and thereby let his mail fall into his own 
responsibility when the unavailability of his host is either known to be 
temporary or is simply not long enough to justify any resulting policy 
differences between hosts.  His mate could, if this were a dynamic DNS 
provider, be half-way across the world when the client only needed to wait 
a few minutes before it could deliver to the final destination.  This is 
just an example.  Also, etrn/similar don't solve the RBL problem.

> the Internet infrastructure....

The unilateral thought of all of us, no doubt. :-)  It is quite outrageous 
as far as requirements go for such a simple feature, but I honestly can't 
see a simple way otherwise that doesn't involve inter-host communication 
which, obviously, can't happen if the backed-up host is out, or other 
scheduled checking.  If his mate could somehow know that he (my friend) 
were down, his mate could serve him, and when he returned his mate could, 
upon instruction by him, give an appropriate number and message to the 
effect that the primary was up, go and deliver to him, when all clients 
tried sending to my friend through his mate.  Would implementation of such 
a system be worthwhile, even if it meant periodic checks by his mate that 
my friend was down?  The MX concept is very good, but very heavy-handed 
and served a time long ago when it really was necessary.  Also, the demand 
on the speed of mail delivery has gone up with the number of email users 
on the net and their expectations of it.  In terms of saved bandwidth, 
there's a slight argument for the cost of a DNS query.  I would be willing 
to implement it either way, though preferably the least destructive one.

Cheers,
Sabahattin
-- 
Thought for the day:
    Dictatorship (n): a form of government under which everything 
    which is not prohibited is compulsory.

Sabahattin Gucukoglu

---

From: Keith Moore <[email-masked]>
Date: Fri, 9 Jan 2004 16:29:52 -0500

> > Why doesn't your friend use ETRN to trigger delivery of his queued
> > mail from his mate whenever he gets online?
> 
> He doesn't want his mate getting his mail while he's not available if
> he will be available shortly after.  

So configure the secondary MX return to 4xx in response to RCPT whenever
the primary MX should be available "shortly after".  

> Also, etrn/similar don't solve the RBL problem.

For that matter, RBL doesn't solve any problem either.  But if you
really want to do RBL filtering on mail relayed to the secondary MX, you
should be able to filter on addresses in Received fields.  Sure, it's 
dodgy, but if you're trusting RBL you're already demonstrating you don't
care about reliable mail delivery.

---

From: "Theodore Ts'o" <[email-masked]>
Date: Fri, 9 Jan 2004 18:34:44 -0500

On Fri, Jan 09, 2004 at 06:45:57PM -0000, Sabahattin Gucukoglu wrote:
> differences between hosts.  

But the problem is that until vast quanties of people understand your
extended MX proposal, the mail will be received by the backup server
anyway.  And given past experience, it takes a long, long time for
support for new features to get propagated to enough of the global
internet infrastructure that it will really solve your friend's
problem.

Let me propose another possible solution, which doesn't require making
global infrastructural changes.  It will require changes to the SMTP
server on the backup MX machine, but that is something which is much
more likely to be doable.

First of all, on the backup MX server, you create a little daemon that
periodically checks to see if the primary mail server is up.  If it is
down for more than a predefined time perioud (a day in your example),
then it sets a flag "this host is down hard".

Secondly, you change the the SMTP server on the backup MX server that
by default, attempts to send mail to your friends domain return a 4xx
soft error, UNLESS the "this host is down hard" flag is set.  If that
flag is set, then the mail is accepted, and queued for transmission to
your friend's mail server.

As a side benefit, current spamming software will see the soft
failure, and not bother to retry the mail transmission while your
friend's host mail server is temporarily down.

						- Ted