Subject: [Asrg] Re: bounces, and anit-spam principles

From: <[email-masked]>
Date: Mon, 22 Jan 2007 18:05:50 -0600

Perhaps it's a good time to step back again and take a 
look at some principles that hopefully we can recognize 
(and some of these seemed controversial when I first 
proposed them, but hopefully by now my reasoning is a 
little more apparent):

1.  Spam is any mail the RECIPIENT does not want, 
regardless of how much the sender wants to send it.

2.  Accordingly, the definition of what they do and do not 
want MUST be such that the RECIPIENT defines it... not the 
IETF, not the sender's ISP, not the recipient's ISP, nor 
some governmental body, nor anybody else.

3.  Systems which rely on the "reputation" or 
"certifications" of the (supposed) sender are not very 
helpful, because a user's machine can be compromised by a 
worm or virus, or because a purported sender's credentials 
can be forged.

4.  Spammers, regardless of what they claim about not 
wanting to send unwanted mails, or only sending to 
"opt-in" addresses, specifically do everything in their 
power to bypass filters set up by recipients to block 
mails just like what the spammer wants to send.

5.  Most recipients do not really want to have their 
machine turned into a spambot zombie.

6.  Antispam/antiworm approaches based on traditional 
signatures are not effective in the initial hours of an 
infection wave, when all worms and viruses are at their 
most prolific and most dangerous.

7.  (And this point is particularly timely given the 
current discussion here).... it is almost NEVER really 
helpful to bounce an incoming message which contained a 
worm or a virus, or even which contains spam, unless you 
can determine (how??) who the legitimate original true 
sender really was.

For example, one incredibly annoying thing that happens to 
me every month or so is that Yahoo will stop delivering my 
Yahooogroups list mails to me, because they got back a 
"hard bounce" message implicating one of my E-mail 
addresses.  Almost invariably, this hard bounce message 
got sent out by some idiot mail software which found a 
worm or virus in an incoming message (and NONE of these 
infected mails have actually been sent out by ME, of 
course) and the idiot software decided to send the bounce 
back to some third party (in my case, Yahoo), which (also 
not-cleverly) decides that since they got back a 
(completely bogus!) hard bounce message (and to a message 
that Yahoo didn't really send, either), they'd better not 
try to send any more mail to me.  :-(  The result is that 
I typically lose anything from a half day's mail to as 
much as several days' mail from the various Yahoogroups 
I'm subscribed to.  :-((

Whether an E-mail contains spam, or a virus, or just about 
ANYTHING, the ONLY time it should be rejected is during 
the time of the original SMTP session when it is coming 
in... UNLESS the entity generating the bounce can 
(somehow) be ABSOLUTELY CERTAIN of who the guilty party 
REALLY is.

It's especially outrageous when a bounce message goes out 
specifying a contained virus which is KNOWN to forge a 
counterfeit return address...!  What is it supposed to 
accomplish to tell someone, whose system is NOT infected, 
that some random one else's system is?  And then, worse, 
to couch that as a "hard" bounce, suggesting that no more 
mail should be sent to the address in question??  (When, 
in fact, a legitimate mail would presumably sail right 
through...)

Critical aspects of solving the spam problem require that 
we deny spammers and abusers the techniques they have come 
to so effectively use to bypass effective content 
filtering.  That includes HTML, text-as-image, ActiveX, 
scripting (Java etc), and things of that sort.

Therefore, as an initial default (which would apply to 
"unfamiliar" senders) I still insist that mail from 
unaccepted senders be passed into my Inbox ONLY IF it 
contains NO HTML, and NO attachments, and does not exceed 
some specified size (10K, 50K, 100K, 200K, whatever the 
recipient decides to set as a threshold).  Assuming the 
mail message gets through that initial qualifying 
gauntlet, then Spam Assassin or whatever other antispam 
content filter should be able to decide whether it looks 
reasonable for delivery or not.

(And as an additional comment, based on a statement in 
today's ASRG digest, let me point out that brain-damaged 
"regular expressions" type pattern matching is IMHO 
nowhere close to adequate for this kind of good textual 
spam recognition, and something closer to SNOBOL4 or 
SPITBOL-type pattern matching is far more appropriate and 
useful).

The (initial filtering!) approach that I believe will work 
is based on this fine-grained "permissions list" (with a 
ruleset adjustable on a per-sender basis) which combined 
with a suitable default behavior denies in one fell swoop 
nearly all the tricks and subterfuges that spammers use to 
evade and deceive content filters.

All of these DNS-based things based on "sender reputation" 
and the like are doomed to failure because a well-reputed 
sender CAN be infected and caused to send out spam.  And 
just because they DO, doesn't mean that the LEGITIMATE 
mail they might still be sending out ought to now be 
t-canned.

It is FAR more helpful to allow recipient machines to 
identify that the mail message supposedly coming from a 
legitimate and familiar sender INDEED 'LOOKS' THE WAY that 
E-mail from that particular sender is EXPECTED AND ALLOWED 
to look.

And meanwhile, not allowing (by default) incoming E-mail 
that contains HTML, scripting, attachments and the like 
unless they are from someone that we TRUST to send such 
stuff (and have whitelisted) will virtually eliminate 
E-mail as a transmission vector for worms and viruses.

We keep dancing around with these elaborate technical 
schemes that folks keep claiming will solve (parts of) the 
problem, but which will predictably not solve it very much 
or very well.  Meanwhile, relatively straightforward 
approaches (as I'm proposing), and in conjunction with 
good content filtering, will go a long way towards solving 
the REAL problems, and still allow receiving (simple, 
safe) mail from previously unknown "first contact" 
senders.  AMD take a big chunk out of worms and viruses 
(today's AND tomorrow's!) at the same time.

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Steve Atkins <[email-masked]>
Date: Mon, 22 Jan 2007 16:48:08 -0800

On Jan 22, 2007, at 4:05 PM, <[email-masked]>  
<[email-masked]> wrote:

>
> Perhaps it's a good time to step back again and take a look at some  
> principles that hopefully we can recognize (and some of these  
> seemed controversial when I first proposed them, but hopefully by  
> now my reasoning is a little more apparent):
>
> 1.  Spam is any mail the RECIPIENT does not want, regardless of how  
> much the sender wants to send it.

Recipients often decide they don't want mail that they've explicitly  
asked to receive. That does not make it "spam" by any reasonable  
definition.

> 2.  Accordingly, the definition of what they do and do not want  
> MUST be such that the RECIPIENT defines it... not the IETF, not the  
> sender's ISP, not the recipient's ISP, nor some governmental body,  
> nor anybody else.

If spam is defined solely by the recipient then there is no  
reasonable standard, and no possible way for a sender to comply with  
it. So any discussion about "spam" becomes entirely irrelevant.

> 3.  Systems which rely on the "reputation" or "certifications" of  
> the (supposed) sender are not very helpful, because a user's  
> machine can be compromised by a worm or virus, or because a  
> purported sender's credentials can be forged.

Unless implemented pretty badly, credentials cannot realistically be  
forged. Even if a senders machine is compromised by a virus and is  
sending out mail with their credentials; it's still them sending out  
the mail, and those credentials prove the fact - making the  
credentials continue to be useful even though the mail is not mail  
you'd really want sent.

> 4.  Spammers, regardless of what they claim about not wanting to  
> send unwanted mails, or only sending to "opt-in" addresses,  
> specifically do everything in their power to bypass filters set up  
> by recipients to block mails just like what the spammer wants to send.

Everybody who sends email in any sort of volume, spammers or not,  
does everything in their power to bypass filters set up by  
recipients. That includes senders of squeaky clean closed-loop opt-in  
with a cherry on top email. It includes ISP postmaster groups, whose  
smarthosts are used for huge amounts of 1:1 email.

> 5.  Most recipients do not really want to have their machine turned  
> into a spambot zombie.

Empirical evidence suggests otherwise. Certainly most users are not  
inclined to use the free security products bundled with their PC or  
ISP account, let alone spend $20 on a hardware firewall of some sort.  
At best they're indifferent about it, until they're disconnected by  
their ISP, and then they're mad at the ISP.

> 6.  Antispam/antiworm approaches based on traditional signatures  
> are not effective in the initial hours of an infection wave, when  
> all worms and viruses are at their most prolific and most dangerous.

Not entirely true, despite the empirical evidence for it.

Antivirus vendors make their money out of subscriptions, so if their  
products were to detect new viruses even if the user hadn't paid for  
a subscription there'd be no reason at all for the users to pay for  
the subscription, and so no revenue for the antivirus vendor.

Because of that they tend implement their products so as to be very  
specific to each new virus, forcing users to pay for subscriptions,  
rather than implementing more general solutions.

(Personally I've found that a few signatures based on base64 encoded  
versions of the first few bytes of various win32 executable formats  
vastly more effective than typical windows antivirus products in  
preventing infectious payloads arriving via email.)

> 7.  (And this point is particularly timely given the current  
> discussion here).... it is almost NEVER really helpful to bounce an  
> incoming message which contained a worm or a virus, or even which  
> contains spam, unless you can determine (how??) who the legitimate  
> original true sender really was.

Yup. But again we're back to the whole commercial product problem.  
Rejection messages sent by antivirus software are both advertising  
material for the software and reassurance to the purchaser (and  
subscriber) to the software that it's doing something valuable for them.

> All of these DNS-based things based on "sender reputation" and the  
> like are doomed to failure because a well-reputed sender CAN be  
> infected and caused to send out spam.

No, they can't, for many reasons. One of those is logical - if they  
send out much spam, they'll no longer have a good reputation.

And operationally, legitimate senders do not send email from systems  
that can be infected by whatever virus-du-jour is going around. The  
only place this problem can really appear is when the legitimate  
sender is actually smarthosting for a large consumer userbase.

Identifying the sender reliably is the missing link in the current  
system.

Cheers,
   Steve

---

From: Dan Oetting <[email-masked]>
Date: Mon, 22 Jan 2007 18:59:59 -0700

> Identifying the sender reliably is the missing link in the current  

The missing link is really 2 different problems depending on which  
side of the link you are on. The receiver can't identify the actual  
sender and the smarthost can't reliably identify the email as  
abusive. Providing direct feedback when email is abusive (such as  
when unsolicited email hits a spamtrap) could bridge the gap and  
permit the smarthost to take appropriate action to stop the abuse.

-- Dan Oetting

---

From: Steve Atkins <[email-masked]>
Date: Mon, 22 Jan 2007 18:09:36 -0800

> permit the smarthost to take appropriate action to stop the abuse.

DKIM and ARF between them provide all the protocol-level solution  
needed for that. DKIM was the missing link I was thinking of.

Cheers,
   Steve

---

From: Steve Atkins <[email-masked]>
Date: Mon, 22 Jan 2007 18:09:36 -0800

Cheers,
   Steve

---

From: Tony Finch <[email-masked]>
Date: Tue, 23 Jan 2007 02:36:40 +0000

>some governmental body, nor anybody else.

Most users prefer to delegate this job. It's 10x more efficient
to do so.

>worm or virus, or because a purported sender's credentials

I'm quite happy with reputation systems that block email
in these situations, because you can't expect to let your
machine be compromised without consequences. They also deal
with the vast majority of spam very cheaply.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
ROCKALL MALIN HEBRIDES: NORTH BACKING NORTHWEST 5 OR 6, OCCASIONALLY 7 LATER.
ROUGH INCREASING VERY ROUGH. WINTRY SHOWERS OR RAIN. MODERATE OR GOOD.

---

From: Dan Oetting <[email-masked]>
Date: Mon, 22 Jan 2007 19:51:05 -0700

On Jan 22, 2007, at 7:09 PM, Steve Atkins wrote:

>>> Identifying the sender reliably is the missing link in the  
>>> current system.
>>
>> side of the link you are on. The receiver can't identify the  
>> actual sender and the smarthost can't reliably identify the email  
>> as abusive. Providing direct feedback when email is abusive (such  
>> as when unsolicited email hits a spamtrap) could bridge the gap  
>> and permit the smarthost to take appropriate action to stop the  
>> abuse.
>

That works if you don't mind taking the long route. I was thinking of  
a more direct path of sending a specific response code in the SMTP  
transaction. A really smart smarthost might be able to then fix the  
problem before the site gets blacklisted.

-- Dan Oetting

---

From: <[email-masked]>
Date: Mon, 22 Jan 2007 22:45:04 -0600

On Tue, 23 Jan 2007 02:36:40 +0000
> Most users prefer to delegate this job. It's 10x more 

I have no idea where you get that statistic, but looks 
sorta brown to me....  ;-)

I would be happy to have someone else adjust some of my 
spam filters for me, as long as the results are good, but 
my experience with such things is that they simply aren't 
as good as they need to be... to the point where I had to 
simply turn some ISP-provided spam filters off (they were 
more trouble than they were worth, and mis-categorized too 
many messages).

Another problem with centralized antispam filtering is 
that spammers get good at tweaking their messages so they 
manage to get through the widely used filters.  It's far 
harder to do that if individual recipients can set their 
own message size limits, rejection criteria, and so forth. 
 And of course, finely grained whitelists on a 
per-recipient/sender pair basis can really only be done by 
the recipient, since the list of approved senders (and 
what they are expected to be sending) probably won't be 
the same for any two recipients.

well-written software implementation can reduce the hassle 
factor of maintaining such finely-grained whitelists to 
(IMHO) very reasonable levels.

> machine be compromised without consequences. 

OK, so let's say Aunt Matilda's system gets a virus on it, 
and starts sending out spam (sooner or later, MOST 
machines will be infected at least once...!)  Now what? 
 Game over?  Aunt Matilda never manages to succesfully 
send another E-mail again as long as she lives?

And what about the case where Aunt Matilda's system IS NOT 
infected, never has been, but where her mail services is 
impacted (as mine has been) by SOMEONE ELSE's machine 
being infected, and forging HER return address on the 
e-mails?  (Rather like the bogus way that Yahoo disables 
valid e-mail addresses for forged mail that OTEHR people 
have sent?)

Clearly SHE doesn't deserve those "consequences", and 
since there is ABSOLUTELY NOTHING that she could possibly 
do about that situation, there is ABSOLUTELY NOTHING to be 
gained by sending "bounce" messages back to her, or 
otherwise harassing her for something she has NO control 
over.  You're only multiplying the nasty consequences of 
the spamming or worm infection.

> They also 
>deal

"Cheaply" isn't necessarily "well".  We've already 
discussed how I might occasionally send an E-mail message 
from an Internet cafe in a resort city, or for that matter 
onboard a cruise ship.  I will still want to use MY 
personal E-mail address, even though it will not be being 
sent out by anything remotely like the E-mail servers I 
would use from my systems here at home.  Blocking those 
messages as "spam" just because they aren't being sent 
through my habitual mail server (and I may not know in 
advance whose server that Internet cafe is using) isn't 
very helpful.

We programmers have the saying that "things should be made 
as simple as possible, BUT NO SIMPLER!"  A cheap solution 
that doesn't work (at all) for a significant number (even 
though small, percentage wise) of legitimate users isn't a 
viable or attractive solution, no matter how cheap it 
might be for those cases where it works.

> Tony.

Gordon Peterson
http://personal.terabites.com

---

From: "Chris Lewis" <[email-masked]>
Date: Tue, 23 Jan 2007 00:08:55 -0500

[email-masked] wrote:
> I have no idea where you get that statistic, but looks sorta brown to
> me....  ;-)
> 
> I would be happy to have someone else adjust some of my spam filters for
> me, as long as the results are good, but my experience with such things
> is that they simply aren't as good as they need to be... to the point
> where I had to simply turn some ISP-provided spam filters off (they were
> more trouble than they were worth, and mis-categorized too many messages).
> 
> Another problem with centralized antispam filtering is that spammers get
> good at tweaking their messages so they manage to get through the widely
> used filters.  It's far harder to do that if individual recipients can
> set their own message size limits, rejection criteria, and so forth. And
> of course, finely grained whitelists on a per-recipient/sender pair
> basis can really only be done by the recipient, since the list of
> approved senders (and what they are expected to be sending) probably
> won't be the same for any two recipients.

You would be hard pressed to find _any_ real world environment beyond a
trivial size where per-user filter tuning can even remotely approach the
effectiveness of a good centralized filtering system.  And furthermore,
most users simply don't want to waste time tuning filters.  They just
want the spam to stop.

Secondly, who said anything about centralized filtering having to use
"widely used filters"?  Personal filters are just as prone to do that,
or not as prone, as centralized filters.  And I can assure you, it's a
lot easier getting past Thunderbird or Outlook filtering than our front
ends.

In non-trivial real world environments, the only way you'd get good
filtering with a "recipient tweaks" model is if it was "block everybody
not in my whitelist".  Which is undesirable in many places.  Not the
least being that in many cases even:

> well-written software implementation can reduce the hassle factor of
> maintaining such finely-grained whitelists to (IMHO) very reasonable
> levels.

ain't easy.  Because what are you whitelisting?  From addresses?  IPs?
Which IPs?

Sorry, while I appreciate the notion of not letting, say, government
decide for me what spam is or isn't, users should be free to decide what
filters they want to use.  And that will almost always be provider-supplied.

And secondly, while some people think a "corporate entity" simply stands
in lieu of individual users in a corporate environment, it's not going
to be construed that way in the real world.  Sorry, nobody but my
management gets to decide how we run our filters.

> OK, so let's say Aunt Matilda's system gets a virus on it, and starts
> sending out spam (sooner or later, MOST machines will be infected at
> least once...!)  Now what? Game over?  Aunt Matilda never manages to
> succesfully send another E-mail again as long as she lives?

Many systems use timeouts for such detections.

> And what about the case where Aunt Matilda's system IS NOT infected,
> never has been, but where her mail services is impacted (as mine has
> been) by SOMEONE ELSE's machine being infected, and forging HER return
> address on the e-mails?

Few filtering systems actually work that way.

> (Rather like the bogus way that Yahoo disables
> valid e-mail addresses for forged mail that OTEHR people have sent?)

Are you _sure_ that's what you were seeing?  That Yahoo conflates
rejections not related at all to a given distribution list?

_All_ of the ones I've seen is where yahoogroups have accepted for group
distribution messages that the recipient's filtering system rejected.
Eg: spam and viruses.

Sorry, Yahoogroups doesn't get a free pass to distribute viruses and
spam.  The fact that that occasionally causes subscriptions to fail is
the price you have to pay to protect your users.

> "Cheaply" isn't necessarily "well".  We've already discussed how I might
> occasionally send an E-mail message from an Internet cafe in a resort
> city, or for that matter onboard a cruise ship.  I will still want to
> use MY personal E-mail address, even though it will not be being sent
> out by anything remotely like the E-mail servers I would use from my
> systems here at home.  Blocking those messages as "spam" just because
> they aren't being sent through my habitual mail server (and I may not
> know in advance whose server that Internet cafe is using) isn't very
> helpful.

Letting viruses in because they forge your email address isn't very
helpful either.

---

From: Dave Crocker <[email-masked]>
Date: Mon, 22 Jan 2007 21:22:14 -0800

> Most users prefer to delegate this job. It's 10x more efficient

Not to quibble, but I would have guessed several orders of magnitude greater.

Probably more.

Also, for most of the world's 1+ billion users, I'd have added "effective" to 
"efficient"...

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: John Levine <[email-masked]>
Date: 23 Jan 2007 05:27:04 -0000

>> more trouble than they were worth, and mis-categorized too many messages).

There are certainly lots of ISPs with crummy spam filters.  But my
experience agrees with Chris.  

When users are asking for knobs and dials on the spam filters, what it
really means is that the filters don't work.  A couple of years ago I
visited a large cable ISP who told me that with their old filters they
were getting constant requests for customization, then they installed
new filters that worked better and the requests completely stopped.

R's,
John

---

From: "David Nicol" <[email-masked]>
Date: Tue, 23 Jan 2007 00:25:02 -0600

> problem before the site gets blacklisted.

I for one caught that drift.  It wouldn't need to be its own ESMTP keyword,
just some specific text following the 550 rejected on policy grounds.  Perhaps
the word "spam."

So, a conformant sending domain, on seeing a /^550 .+spam/i response, would
do what exactly? Log the fact?  Page a sysadmin?

---

From: "Peter J. Holzer" <[email-masked]>
Date: Tue, 23 Jan 2007 10:01:26 +0100

On 2007-01-23 00:25:02 -0600, David Nicol wrote:
> I for one caught that drift.  It wouldn't need to be its own ESMTP
> keyword, just some specific text following the 550 rejected on policy
> grounds.  Perhaps the word "spam."
> 

Temporarily block emails from this user and generate a bounce message
explaining the policy (it's your user and he has authenticated himself
with SMTP AUTH or an equivalent mechanism, so you know who he is).
You probably still want to accept mail from the user to your own support
mail address, and you probably also want to start with a short (maybe
only a few minutes) blocking time and increase that with every offense.
If the user mistakenly has sent a mail to a spamtrap he will probably be
clean again when he's finished writing his next mail and because of the
bounce message he knows that he shouldn't send to that address again. If
his machine is a zombie which uses his credentials he will have
increasing trouble sending mail until he cleans up his machine. 

Sounds like a nice scheme to me. Has anybody ever tried that?

	hp

-- 
   _  | Peter J. Holzer    | I know I'd be respectful of a pirate 
|_|_) | Sysadmin WSR       | with an emu on his shoulder.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Sam in "Freefall"
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Tony Finch <[email-masked]>
Date: Tue, 23 Jan 2007 10:20:44 +0000

On Mon, 22 Jan 2007, Dave Crocker wrote:
> Not to quibble, but I would have guessed several orders of magnitude
> greater. Probably more. Also, for most of the world's 1+ billion users,
> I'd have added "effective" to "efficient"...

I was thinking in terms of the cost of accepting, storing, and retrieving
the user's email before passing it to the MUA filter and only then
discovering that 90% of it is spam, as opposed to rejecting it at SMTP
time.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOUTHEAST ICELAND: MAINLY NORTHWESTERLY 5 INCREASING 6 TO GALE 8. MODERATE
INCREASING ROUGH OR VERY ROUGH. SNOW SHOWERS. GOOD.

---

From: Tony Finch <[email-masked]>
Date: Tue, 23 Jan 2007 11:54:53 +0000

On Mon, 22 Jan 2007, [email-masked] wrote:
>
> OK, so let's say Aunt Matilda's system gets a virus on it, and starts sending
> out spam (sooner or later, MOST machines will be infected at least once...!)
> Now what?

Her machine gets quarantined and her ISP helps her (manually or via a
special web site) to fix her computer. This is not quite a pipe dream -
see http://wesii.econinfosec.org/draft.php?paper_id=47
(You can Google for an HTML version.)

> And what about the case where Aunt Matilda's system IS NOT infected, never has
> been, but where her mail services is impacted (as mine has been) by SOMEONE
> ELSE's machine being infected, and forging HER return address on the e-mails?

The blacklists I'm talking about work on IP addresses not email addresses.

> > They also deal with the vast majority of spam very cheaply.
>
> "Cheaply" isn't necessarily "well".

It is if you use the right blacklists, e.g. the Spamhaus ones.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SHANNON: NORTH 6 OR 7, OCCASIONALLY GALE 8. ROUGH OR VERY ROUGH. SHOWERS.
GOOD.

---

From: "Danny Angus" <[email-masked]>
Date: Tue, 23 Jan 2007 13:30:31 +0000

> Temporarily block emails from this user and generate a bounce message

...

> Sounds like a nice scheme to me. Has anybody ever tried that?

I don't know if anyone has tried it, but we discussed a proposal at
Apache James some years ago that proposed using a 4xx response to
allow recipient MTA's to spend time evaluating (or hand off evaluation
of) an mta's or sender's identity (possibly scoring it) before
accepting mail from them rather than perform expensive checks in real
time. I briefly mentioned it on this list in 2003[1] I believe that
this approach is also consistent with rfc2505 para 1.6[2].
The sending MTA would either retry after an agreed delay, or when
called with an enhanced ETRN, enhanced to specify allowable MTA's,
sender(s) and/or recipient(s) or listed message id's (not the ones in
the body, this would have to be a key agreed between MTA's).
In the event that the recipient decides not to allow it would return
another 4xx with an appropriate reason that might be resolved, or a
5xx with a reason which could not be resolved.
The main objection was that some felt the quantity of delayed mail and
the reconciliation of it against the allowable patterns would impose a
significant unwanted overhead on MTA's. We didn't do any analysis to
confirm or refute this.
A less crucial objection is that badly behaved sending MTA's wouldn't
abide by the rules and even though connections could be broken and 4xx
& 5xx re-issued it might not be possible to realise the perceived
benefits when challenged by a load of badly behaved spambots.

d.
[1] http://www1.ietf.org/mail-archive/web/asrg/current/msg06019.html
[2] http://www.ietf.org/rfc/rfc2505.txt

---

From: Daniel Feenberg <[email-masked]>
Date: Tue, 23 Jan 2007 09:14:30 -0500 (EST)

> Also, for most of the world's 1+ billion users, I'd have added "effective" to 

Among other problems with individually maintained spamblocks is the 
difficulty in rehabilitating a legitimate address (ip or domain) after
it has been widely blocked for prior bad behavior.

Forbidding cooperation or delegation is a spammer's trick.

Daniel Feenberg

---

From: <[email-masked]>
Date: Tue, 23 Jan 2007 09:55:45 -0600

On Tue, 23 Jan 2007 09:14:30 -0500 (EST)
>>>> some governmental body, nor anybody else.

[snip]

> Among other problems with individually maintained 
>spamblocks is the difficulty in rehabilitating a 
>legitimate address (ip or domain) after

Absolutely, and that's a good reason why blocking by 
either IP address or domain name is such a bad solution. 
 A fine-grained whitelist which specifies ALLOWED behavior 
on a per-sender basis, on the other hand, can easily allow 
or block messages from a given sender ON A 
MESSAGE-BY-MESSAGE basis, so that their legitimate 
messages get delivered but the (zombie) messages being 
sent by their same (infected)machine, using the same mail 
servers and same permissions/certifications but which do 
not look the way that sender's messages are expected to 
look (by the recipient!) are efficiently and accurately 
identified and blocked.

So "rehabilitation" isn't even an issue.

A suitable default set of criteria, combined with a 
suitable content filter ("Spam Assassin" or similar.... 
which can do a far more accurate job once HTML, scripting, 
and attachments are not allowed from unknown senders!) can 
handle the requirements fairly nicely for a large 
percentage of typical users.  So there you have your 
"centralized criteria" for both the first and second 
levels of filtering, while still allowing the recipients 
to open a "narrow and twisty gauntlet" to allow through 
the other messages that they expect from senders who they 
know and trust... (and because those individually-set 
criteria are not known to spammers, they will have a 
vanishingly small probability of getting such messages 
delivered).

I do agree that it would philosophically be better if 
there were some way to detect and block such messages 
earlier in the distribution path, but that's really a 
discussion for a later date.  Ultimately, that might not 
even be necessary... if the spammers realize that 
virtually none of their garbage will be delivered, at some 
point they will either go and do something else, or else 
just be more direct (virus-author-like) and unabashedly 
just abuse the network infrastructure without any pretense 
of expecting E-mails to get through at all, perhaps. 
 Again, that's a discussion for a later time.

>Forbidding cooperation or delegation is a spammer's 

The "spammers trick" is trying to make sure that everybody 
uses the same criteria, so they can engineer messages that 
will slip through the default criteria.  You'd better 
believe that all serious spammers have and use Spam 
Assassin and other such scanners, and test their intended 
messages to make sure they stay below the various spam 
thresholds.

Things like Spam Assassin can do a pretty good job, IF 
they aren't confronted with things like scripting, images 
(attached or embedded), ActiveX, HTML ruses of various 
kinds, and similar tricks that evade text-based content 
detection rules.

Y'all can sit back for years in philosophical and 
theoretical discussions about some elaborate technical 
"magic bullet" that you think is going to solve the spam 
problem, but when push comes to shove, what you're 
eventually going to figure out on this one is that I've 
been right all along, and that what I'm proposing really 
IS an effective and practical solution to the problem. 
 (And perhaps THE ONLY good approach).

Gordon Peterson

---

From: Dan Oetting <[email-masked]>
Date: Tue, 23 Jan 2007 10:00:05 -0700

On Jan 22, 2007, at 11:25 PM, David Nicol wrote:

> keyword,

Just like most blacklists wouldn't list a site for a single email  
hitting a spamtrap, the host would not be expected to take action  
against a user for sending the single email unless there were  
extenuating circumstances. If the host is on probation or a gray list  
for currently spamming it may be necessary to throttle the user to  
prevent escalation to a blacklist.

You don't want to reveal the spamtrap address to the user. If the  
users account has been taken over by a spammer it could wash the  
address from its lists and then you loose that method of identifying  
the spamers next identity. These traps should not be used for  
blacklisting because if they are identified by the spammers they  
could be abused to generate false listings.

The spamtraps should only be revealed between cooperating ISPs. And  
each trap should only reveal itself to a fraction of the connecting  
hosts to keep from revealing all the traps to a rogue ISP that joins  
the group. The response code given to reveal the trap should be an  
accept code (2xx) so that the text is not passed to the user by  
default if the smarthost MTA is non-conforming.

-- Dan Oetting
(sent to the list this time)

---

From: <[email-masked]>
Date: Tue, 23 Jan 2007 10:54:35 -0600

On Tue, 23 Jan 2007 11:54:53 +0000
> (You can Google for an HTML version.)

I agree that the zombie spambot problem is key (and it 
might even be the case that ISPs will someday be 
considered liable for allowing worm/virus messages to be 
delivered to their customers, given the relative ease of 
detecting those at their level...).  One of the advantages 
of my approach is that it virtually eliminates the 
viruses/worms which infect machines and produce these 
spambot armies (well, at least it virtually eliminates 
their propagation by E-mail...!)

The paper ignores the fact that a lot of spam messages are 
nearly impossible to detect using scanners, unless you 
block images and other attachments, scripting, and HTML.

Their contention that zombie spambot networks can be dealt 
with by their ISP help desks is ridiculous, given the 
scope of the problem and the general incompetence of those 
help desks.  Their so-called "automated" approach strikes 
me as little more than a nice fantasy, at least in the 
general case.

In any case, I still contend that simplistic blocking by 
IP address or domain name is a very poor approach, and for 
a whole variety of reasons.

> The blacklists I'm talking about work on IP addresses 

Dialup machines, for example (and there are still a lot of 
those) don't generally have fixed IP addresses.

> It is if you use the right blacklists, e.g. the Spamhaus 

I still consider the use of blacklists (either IP or 
domain name or E-mail address based) as a poor solution, 
due to the transient nature of dialup IP addresses, and 
the difficulty of rehabilitating addresses.  Also, very 
large networks of computers (say, an entire large company, 
or say a campus) operating using NAT could be compromised 
because just one machine (even a "visiting" portable 
notebook machine via a wireless connection) sent spam, 
causing their router's IP address or mail server's IP 
address to be blacklisted.

The problem with blacklists is that it is a wild goose 
chase... and you're trying to lock the door AFTER the 
horse has gotten out.  The person pulling the wagon (in 
this case the spammers) can pull it any direction they 
like, and everyone else is in a crazy keystone-cops-like 
race from behind.  Spammers can (and do) generate an 
infinite number of bogus E-mail addresses, user names, 
subject lines, and even message contents.  They can 
eventually infect a large percentage of every machine on 
the Net, worldwide.

The fact that a machine or IP address sent (once, twice, 
some number of times) spam (at some point in the past) is 
no guarantee that it will ever (or never!) send spam 
again.

Simply blocking (as a default behavior) attachments and 
HTML in messages from unfamiliar/untrusted senders (and on 
a fine-grained basis, such that one can allow say JPGs but 
block executable attachments, and different levels of HTML 
too, and on a sender-by-sender basis) goes a LONG, LONG 
way to stopping E-mail distribution of viruses and worms 
and trojans (and thus the recruitment of spambot nets), 
and at the same time is the single biggest step that can 
be made to allowing SpamAssassin and similar content 
filters to do their job far more effectively.

Gordon Peterson

---

From: Dave Crocker <[email-masked]>
Date: Tue, 23 Jan 2007 09:42:29 -0800

> discovering that 90% of it is spam, as opposed to rejecting it at SMTP

Ahh.  OK.

And, of course, I was thinking in terms of expecting/requiring users to do any
abuse-related processing.  Human factors, usability, and the like.  Yuch.

d/

-- 

   Dave Crocker

---

From: Dan Oetting <[email-masked]>
Date: Tue, 23 Jan 2007 11:38:54 -0700

On Jan 23, 2007, at 9:54 AM, [email-masked] wrote:

> In any case, I still contend that simplistic blocking by IP address  
> or domain name is a very poor approach, and for a whole variety of  
> reasons.

I will contend that there cannot be a content filter that can  
reliably separate spam from non spam. Any non-spam message received  
by one person would be spam if it were sent to 10 million harvested  
addresses. OTOH, taking the definition of spam as Unsolicited Bulk  
Email makes detecting a spamming IP address almost trivial. The trick  
is to stop accepting mail from that IP address only until it has  
cleaned up. Once the spam is gone there is no need to block the  
address unless it has proven to be a repeat offender without an  
effective process for shutting the spammers down.

-- Dan Oetting

---

From: Seth Breidbart <[email-masked]>
Date: Tue, 23 Jan 2007 14:45:23 -0500 (EST)

> So "rehabilitation" isn't even an issue.

So the zombie becomes unable to emit spam, but there's no incentive to
fix it so it's still available to the botmaster for use as a C&C
machine, web/DNS server, and DDoS participant.  I'd prefer that it get
uninfected.

Seth

---

From: Barry Shein <[email-masked]>
Date: Tue, 23 Jan 2007 14:44:32 -0500

Speaking as an ISP, what's unrealistic in these utopian end-user
filtering only arguments is costs.

Suffice it to say that we can add fast, capable mail servers and see
them flooded in a matter of hours.

Pushing all the filtering to the end-user would make that much
worse.

Being able to "slam the phone down" on miscreant IP blocks at the
accept() or helo is much, much, less processing than going thru the
entire SMTP interaction and whatever it takes to pass processing off
to an end-user.

Put another way, you can have almost unfiltered access and
near-perfect spam filtering!

Here's how to do it:

Get your own link to the backbone.

Set up your own mail servers etc.

Hire one or more secretaries to pre-screen your email according to
rules you have trained them in.

It might cost a few thousand a month, but surely in the face of all
this expressed urgency about the pitfalls of centralized filtering
it's a small price to pay.

Anyhow, as the old joke goes, we're not arguing about the principle,
we're arguing about the price.

-- 
        -Barry Shein

The World              | [email-masked]           | http://www.TheWorld.com
Purveyors to the Trade | Voice: 800-THE-WRLD        | Login: Nationwide
Software Tool & Die    | Public Access Internet     | SINCE 1989     *oo*

ssage-ID: <[email-masked]>
Date: Tue, 23 Jan 2007 14:44:32 -0500
To: [email-masked]
Subject: Re: [Asrg] Re: bounces, and anit-spam principles

Speaking as an ISP, what's unrealistic in these utopian end-user
filtering only arguments is costs.

Suffice it to say that we can add fast, capable mail servers and see
them flooded in a matter of hours.

Pushing all the filtering to the end-user would make that much
worse.

Being able to "slam the phone down" on miscreant IP blocks at the
accept() or helo is much, much, less processing than going thru the
entire SMTP interaction and whatever it takes to pass processing off
to an end-user.

Put another way, you can have almost unfiltered access and
near-perfect spam filtering!

Here's how to do it:

Get your own link to the backbone.

Set up your own mail servers etc.

Hire one or more secretaries to pre-screen your email according to
rules you have trained them in.

It might cost a few thousand a month, but surely in the face of all
this expressed urgency about the pitfalls of centralized filtering
it's a small price to pay.

Anyhow, as the old joke goes, we're not arguing about the principle,
we're arguing about the price.

-- 
        -Barry Shein

The World              | [email-masked]           | http://www.TheWorld.com
Purveyors to the Trade | Voice: 800-THE-WRLD        | Login: Nationwide
Software Tool & Die    | Public Access Internet     | SINCE 1989     *oo*

---

From: "David Nicol" <[email-masked]>
Date: Tue, 23 Jan 2007 14:05:37 -0600

> Pushing all the filtering to the end-user would make that much

not the filtering, just the filter configuration.  MTA-resident complex
content filtering with different per-recipient outcomes is not compatible
with recipient enumeration preceding the data block; therefore the
DATAFIRST ESMTP proposal I made some months ago.

> entire SMTP interaction and whatever it takes to pass processing off

even less is turning them off at the firewall.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Tue, 23 Jan 2007 22:00:57 +0100

On 2007-01-23 14:05:37 -0600, David Nicol wrote:
> with recipient enumeration preceding the data block;

It can still be done. It just needs a bit of trickery with 4xx replies.
It's ugly as hell, and it increases mail traffic (and CPU usage for
filtering), but it works.

> therefore the

I still like the LMTP model better, but having any method for returning
different results for different recipients based on the content would be
an improvement.

	hp

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 23 Jan 2007 16:27:12 -0500

> filtering), but it works.

see http://www.ehsco.com/misc/I-Ds/draft-hall-inline-dsn-01.txt which uses
per-recipient codes after the data block (the LMTP model)

It's not likely that spammers would use it however

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Douglas Otis <[email-masked]>
Date: Wed, 24 Jan 2007 03:22:21 -0800

> needed for that. DKIM was the missing link I was thinking of.

Problems remain even when DKIM provides a verifiable domain playing some
role in sending the message.  DKIM excludes the envelope.  A message
signed by Yahoo! can be replayed.  It remains to be seen how effective
ARF might be at suppressing abuse of a free account.  Looking at the
situation with .com, it seems even charging for a service might not
offer much of a reduction, due to a high level of fraud.

DKIM limitations on indicating on who's behalf the message is signed
seems aimed at providing a method to extort use of private keys of
customers as a means to avoid receiving the ARF information directly.
This does not bode well for DKIM and ARF working together. : (

Perhaps something as simple as an 'mx-' prefix on the SMTP client
hostname could be used to indicate compliance with address validation,
in addition to a convention where the domain below this label also
accepts ARF reports.  The prefix also implies the domain's authorization
for sending email.

-Doug

---

From: Claus Assmann <[email-masked]>
Date: Wed, 24 Jan 2007 10:10:37 -0800

> per-recipient codes after the data block (the LMTP model)

Are there any intention to make this into an RFC?

> It's not likely that spammers would use it however

For those you can still use the method of allowing only one recipient
per transaction (or at least only recipients with the same anti-spam
policies). Any conforming MTA however would not be subject to those
restrictions.

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 24 Jan 2007 14:36:54 -0500

> Are there any intention to make this into an RFC?

I didn't push it (the writeup was a thought experiment more than
anything), and there didn't seem to be a lot of interest from the WG
either. I imagine somebody would need to implement it in their MTA before
it advanced down the standardization track.

> policies). Any conforming MTA however would not be subject to those

all true

---

From: "David Nicol" <[email-masked]>
Date: Wed, 24 Jan 2007 17:30:34 -0600

> it advanced down the standardization track.

the DATAFIRST proposal I made on this mailing list (june 2006) performs the same
functions, semanticly, with less intermediate data and shorter times
during which
individual records would have to be locked.  I may get around to
implementing it later this
year or next; blocked by expression of plan and achieving funding for
central sender-pays
business concept.

---

From: Claus Assmann <[email-masked]>
Date: Wed, 24 Jan 2007 15:41:51 -0800

> functions, semanticly, with less intermediate data and shorter times

On one of my machines more than 95 per cent of the incoming mails
is to unknown recipients. I do not want to receive the message data
in that case at all.

---

From: "David Nicol" <[email-masked]>
Date: Wed, 24 Jan 2007 17:46:01 -0600

On 1/24/07, Claus Assmann <[email-masked]> wrote:
> > the DATAFIRST

> is to unknown recipients. I do not want to receive the message data

good point.  thanks.  that would have to be fixed first of course; for instance
DATAFIRST connections only allowed from reputable peers.

---

From: "Martin Hannigan" <[email-masked]>
Date: Wed, 24 Jan 2007 19:05:37 -0500

> DATAFIRST connections only allowed from reputable peers.

Account lookups already accomplish this. Most spam mechanisms can
handle checking for a valid account prior to accepting mail from a source.

-M<

---

From: "william(at)elan.net" <[email-masked]>
Date: Wed, 24 Jan 2007 16:14:22 -0800 (PST)

On Wed, 24 Jan 2007, David Nicol wrote:

> the DATAFIRST proposal I made on this mailing list (june 2006) performs the 
> same functions, semanticly, with less intermediate data and shorter times

Is what you're asking really HEADer data first or entier message data
content first?

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: "Martin Hannigan" <[email-masked]>
Date: Wed, 24 Jan 2007 22:33:14 -0500

> Is what you're asking really HEADer data first or entier message data

Even if, what would benefit "me" at the moment and for the foreseeable
future is having more capability to "drop connections" before they start
sending - based on recipient, then source. If I can shut down the millions
of incorrigable LDAP, DNS, and I/O operations that take place to accept,
review, then reject inbound spam, I'd be in the black operationally.

If I had to pick a technology priority to pique my interest for '07,
that I would
consider proposing as a spending item to "make mail better", it would
be centered around IP reputation.

-M<

---

From: Tony Finch <[email-masked]>
Date: Thu, 25 Jan 2007 12:33:15 +0000

>be centered around IP reputation.

DNS blacklists have been around for a decade.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOLE: NORTHEAST 6 OR 7 DECREASING 4. VERY ROUGH DECREASING SLIGHT OR MODERATE.
SHOWERS. GOOD.

---

From: Jose Marcio Martins da Cruz <[email-masked]>
Date: Thu, 25 Jan 2007 16:02:35 +0100

> DATAFIRST connections only allowed from reputable peers.

I agree with Claus. Me too, I don't want to receive messages for non
existent recipients.

A way to fix this may be a negotiation between the server and the
client, probably it's the server who shall ask for DATAFIRST. If the
server don't want it then the transaction fall down to current mode.

My two cents.

-- 
 ---------------------------------------------------------------
 Jose Marcio MARTINS DA CRUZ           Tel. :(33) 01.40.51.93.41
 Ecole des Mines de Paris              http://j-chkmail.ensmp.fr
 60, bd Saint Michel                http://www.ensmp.fr/~martins
 75272 - PARIS CEDEX 06      mailto:[email-masked]

---

From: "Peter J. Holzer" <[email-masked]>
Date: Thu, 25 Jan 2007 16:17:16 +0100

On 2007-01-25 16:02:35 +0100, Jose Marcio Martins da Cruz wrote:
> > good point.  thanks.  that would have to be fixed first of course;
> > for instance DATAFIRST connections only allowed from reputable
> > peers.
> 
> client, probably it's the server who shall ask for DATAFIRST.

It has to do that anyway in the EHLO reponse. It could advertise
DATAFIRST to "reputable" peers only. But I don't think that helps much,
because you specifically want it for the less reputable hosts: Where
some messages will be accepted by some users but rejected by others.

	hp

---

From: "Peter J. Holzer" <[email-masked]>
Date: Thu, 25 Jan 2007 17:06:17 +0100

On 2007-01-23 16:27:12 -0500, Eric A. Hall wrote:
> >> MTA-resident complex content filtering with different per-recipient
> >> outcomes is not compatible with recipient enumeration preceding the
> >> data block;
> > 

I meant "it can be done without extensions to SMTP". With extensions it
can be done in a much cleaner way, of course; but that needs support in
both the client and the server.

A few quibbles about your draft (which I'm tempted to implement in
qpsmtpd if someone implements the client side):

The name INLINE-DSN doesn't describe well what it does. To me, a DSN
is more than an SMTP reply, it's an entire message explaining what
happened to the mail, and your extension doesn't send that.
Something like DATA-RCPT-RESULT would be more descriptive (not that an
SMTP client will care much about the meaning of the bytes it sends).

I'm a bit dubious about the use of 3xx status codes. In the other cases
where they are used (DATA, SMTP AUTH, ...) they are used to indicate to
the client that it should send more data to complete the current
command, not that it should send another command. It's a clever way to
defer only some recipients until after data, though.

Finally a question: How is it determines which of the response codes
after DATA corresponds to which recipient? Do they have to be in the
same order or do they have to include the recipient address? Your
examples show both, but the text isn't clear.

	hp

---

From: "David Nicol" <[email-masked]>
Date: Thu, 25 Jan 2007 10:20:11 -0600

> some messages will be accepted by some users but rejected by others.

in the e-postage scenarios driving DATAFIRST, you would want to provide
the service only for peers participating in the e-postage system.  The advantage
over deferred deliviry status reports becomes clear if you are trying
to accept or
refuse a two-cent e-mail for a million customers whose individual
gatekeeper policies
have the option of accepting, refusing, or haggling.  Instead of
trying more than a million
times to send the message with one recipient at a time, DATAFIRST
provides a shortcut
to saying "same data as last time."

As such it is not a spam reduction technology in itself, but is an
optimization to make
an unimplemented e-postage scheme work more smoothly.

I brought it back up in response to Eric Hall's 2004 proposal, which
has come under
discussion again.

http://www1.ietf.org/mail-archive/web/asrg/current/msg12808.html was
the beginning
of the discussion of it, last summer.  I guess I've got just a few
more months to
try to patent that aspect of my particular e-postage scheme in order
to woo my angels :)

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 25 Jan 2007 14:37:48 -0500

> SMTP client will care much about the meaning of the bytes it sends).

The name isn't that important to me. I almost called it DEFERRALS (since
that's what it does) but I chose the DSN name to try and clarify it a bit.
Putting DEFER on the end of MAIL FROM makes some sense too.

> command, not that it should send another command.

Yeah, it usually means "need more input" although there's nothing else
that means "I'll tell you later" either. Waiting for the message body to
be received probably counts as "need more input" anyway.

> examples show both, but the text isn't clear.

Ordering is always preserved in SMTP (even where pipelining is enabled),
and text after the code is usually ignored for protocol purposes
(usually), so that's presumably what I meant but I'd have to find my brain
from 2004 to answer for sure. I suppose I should fix this in the next
draft, as well as rename the extensions.

---

From: "Martin Hannigan" <[email-masked]>
Date: Thu, 25 Jan 2007 18:19:49 -0500

> DNS blacklists have been around for a decade.

<delurk>

"IP reputation" is a different concept than black list. By tracking
the historical behavoir
of /32's or larger allocations, we could develop a profile of the
behaviors that will eventually
establish a baseline for those allocations as "good" "nuetral" or
"bad", to keep it kind of simple. A lot of this data is already out
there.

Take the case of 60 /8 (IIRC). It was allocated to APNIC by IANA and
almost immediately
spammers began unauthorized advertisements of prefixes high in this
block, "politely" staying ahead of the RIR allocations, but using them
as "fresh" address space. In a reputation system, those blocks would
likely be marked bad since they are unallocated
at the RIR level. Since they are allocated out of IANA, they would be
legitimate to most
RBL's until they were caught in the act, but by the time they are
observed, the allocations
are swapped.

The biggest problem I am having this year, so far,  is justifying
spending money on
anything related to mail that will not be a) very long life and b)
really make it better. The more knobs I have, the better, and I think
IP reputation is the place to be this year.

Account verification before data is "better" in my situation, and
technologies to block or drop connections en masse will have more of a
likelihood of getting funded internally.

The spam conference is coming up in Cambridge. I'm looking forward to it.

YMMV.

</lurk>

-M<

---

From: Tony Finch <[email-masked]>
Date: Thu, 25 Jan 2007 23:52:59 +0000

On Thu, 25 Jan 2007, Martin Hannigan wrote:
> "IP reputation" is a different concept than black list.

In what way? You seem to imply that because reputation is complicated,
that necessarily implies a complicated protocol. DNS blacklists
individually give you a binary good/bad result for each IP address or
domain name, but exactly what they mean by good or bad is subtle and
depends on the blacklist, not the protocol. It is wrong to assume that the
simplicity of the DNS blacklist protocol means that DNS blacklist
semantics are simple.

> By tracking the historical behavoir of /32's or larger allocations, we
> could develop a profile of the behaviors that will eventually establish
> a baseline for those allocations as "good" "nuetral" or "bad", to keep
> it kind of simple. A lot of this data is already out there.

Yes, blacklists do this already, though they are binary not ternary.

> almost immediately spammers began unauthorized advertisements of
> prefixes high in this block, "politely" staying ahead of the RIR
> allocations, but using them as "fresh" address space. In a reputation
> system, those blocks would likely be marked bad since they are
> unallocated at the RIR level. Since they are allocated out of IANA, they
> would be legitimate to most RBL's until they were caught in the act, but
> by the time they are observed, the allocations are swapped.

You are assuming that blacklists are necessarily based on post-hoc
observation the outgoing email behaviour of IP addresses, and that there
are no blacklists that are based on prior information such as allocations
and hijacking. However, you are wrong: an old example is the DUL, but for
something closer to what you have in mind, have a look at
http://www.completewhois.com/bogons/bogons_usage.htm

It's worth noting that the existing RHSBL support in MTAs and anti-spam
software is good enough to be used for reputation lookups for DKIM. What
is difficult is gathering the data to create a blacklist or whitelist for
domains in DKIM signatures. This is really hard: it is very difficult to
maintain an IP DNSBL that is both effective against spam and has a low
false-positive rate, and I expect it to take a long time before there is
enough operational experience to do the same for DKIM.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
ROCKALL MALIN: NORTHWEST 4 OR 5, OCCASIONALLY 6. MODERATE OR ROUGH. OCCASIONAL
RAIN. MODERATE OR GOOD.

---

From: "william(at)elan.net" <[email-masked]>
Date: Thu, 25 Jan 2007 16:16:36 -0800 (PST)

> "IP reputation" is a different concept than black list.

Not really.

> it kind of  simple. A lot of this data is already out there.

Blacklists are a form of reputation with single yes/no answer for
single type of question. More complex scoring similar or even same
as you describe is already in use by some companies spam products 
internally, they are just not making data available externally in
some general protocol (spamcop to a degree does). Others also do
sort of ip reputation by locally combining several blacklists & 
whitelists and calculating score based on that as well as based on 
bayesean filters. I also at some point did test run with calculating
score for an ip  and then block based on score for existing spamassasin
scores (mixed results due to large number of connections coming from
different zombied dsl/cable ips - I need to improve way of generating
list of ip blocks & assigning general reputation to it for use with
this system).

> at the RIR level.

These are bogon ips and some of us are rejecting them already, see
http://www.completewhois.com/bogons/ and blacklist for it is
bogons.dnsiplists.completewhois.com. BTW its interesting to note
that some did not believe me when I said 3+ years ago that spammers
would be doing it and it needs to be dealt with.

> Since they are allocated out of IANA, they would be legitimate to most
> observed, the allocations are swapped.

Possible I do not have enough details, but my understanding is that 
spammer[s] (I actually think its same one) were not staying close
enough ahead (need to be within 1-2 days of allocation and RIRs
actually do not always do them sequentially), but just taking some 
(somewhat random) higher-end ip range within unallocated space or
doing entire /8. Of course its possible they are even smarter in
the last few cases.

> spending money on anything related to mail that will not be
> a) very long life and b)
> really make it better.

Well, yes... but explain that with current internal technologies
"very long life" is still not achievable and you do need to
get some improvement to get cleaner inbox.

> The more knobs I have, the better, and I think

Number of folks are slowly working on it but its not really that
likely this year would make substantial difference I think.

> Account verification before data is "better" in my situation

To be able to do apply user policies when making decisions you
need to do it after RCPT TO. Using blacklist at the start is good
but for most lists (including reputation based ones) it would not
be appropriate (i.e. you must for example accept email to postmaster).

>, and technologies to block or drop connections en masse will have more
> of a likelihood of getting funded internally.

Really? The more "en masse" you do the more good mail would be lost
and reality is that there appears to be substantial enough difference
between users that that doing it "en mass" for ISP or for large enough 
enterprise is probably not ok.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 25 Jan 2007 20:49:51 -0500

> Yes, blacklists do this already, though they are binary not ternary.

That's not necessarily true.  Many DNSBLs are already multi-valued
(zen.spamhaus.org presently has 6 independent "flags"), and they can be
constructed to be more sophisticated with as many as 2^24 values on a
single return, and/or being used in scoring.  It's simply a matter of
deciding on the convention, publishing it, and having admins configure
to the intended interpretation (or not :-(.

>> ... Since they are e (a) small, and (b) not "dangerous" at least=20
reduces the impact of those.

And meanwhile, giving the recipient the ability to at=20
least not see the SAME kind of stuff over and over again,=20
if they choose to use those features, demonstrates the=20
ISP's trying to give the user the tools to reduce the=20
frustration.

Gordon Peterson
http://personal.terabites.com
1977-2007  Thirty year anniversary of local area=20
networking

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 25 Jan 2007 20:49:51 -0500

>> ... Since they are allocated out of IANA, they

I was going to mention that...

allocated out of IANA, they
> http://www.completewhois.com/bogons/bogons_usage.htm

I was going to mention that...

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 25 Jan 2007 21:16:55 -0500

> enterprise is probably not ok.

Most enterprises will have global policies in place that make it totally
unnecessary to have _any_ form of per-recipient policy. Our policy is
that it's not permitted to spam or harrass our users.  Certainly, we
allow both senders and recipients to question or challenge our decisions
in terms of that policy, but anyone continuing to insist on the ability
to receive something that we've reconfirmed to be blockable under the
policy is told that they should use their home account.

That happens less than once a month.

Only one in 10 years has asked for the names of my management.  It
didn't help them ;-)

There are some things that get past the filters, and 3 recipients that
have no filtering. But each exemption has a specific corporate business
case behind it.  Two of those recipients land in my mailbox <sigh>.

---

From: "Martin Hannigan" <[email-masked]>
Date: Thu, 25 Jan 2007 22:54:24 -0500

> On Thu, 25 Jan 2007, Martin Hannigan wrote:

[ snip, discussion to digest - thanks ]

> http://www.completewhois.com/bogons/ and blacklist for it is

They aren't bogons. They've been allocated to the RIR. They are unassigned
to ISP's or end users, but are eligible for assignment.

"Bogons is the name used to describe ip blocks not allocated by IANA and RIRs
to ISPs and organizations plus all other ip blocks that are reserved
for private or
special use by RFC"

Best,

Martin

---

From: "william(at)elan.net" <[email-masked]>
Date: Thu, 25 Jan 2007 20:14:31 -0800 (PST)

> to ISP's or end users, but are eligible for assignment.

And you quoted definition below under which such blocks would fit
since they are not allocated by IANA and RIRs to any ISP or end-user 
organization/company yet.

> to ISPs and organizations plus all other ip blocks that are reserved

Note that for those at ASRG there exist slight difference in ARIN 
terminology between assigned and allocated blocks. Assignment is 
considered to be when block is "given" (leased really) to final
end-user company where as allocation is when its given to the
entity which is ISP or alike and can assign ips to somebody else.
So it is said that IANA allocates blocks to RIR and RIRs then
allocate blocks to LIRs/ISPs and those then assigns them to
end-users. It is also possible for RIR to assign block directly
to end-user company if its big enough or small and they qualify
under some sort of PI (RIPE term) or micro-allocation/assignment
policies. Plus to that large legacy blocks are considered to have
been assigned directly from IANA to end-users when RIRs did not
exist yet. In addition to that the terms "reallocate" and "reassign"
are used although in practice only "reallocate" has meaning when
you talk about one ISP that reallocated the block to some other
ISP which would be making actual assignment to end-users (since
assignment is always final step there can not be "reassignment"
although term is also used).

In any case, in practice a lot (most) use allocation and assignment
terms interchangeably and do not make a distinction - this includes 
definition of bogons as well (it really means both allocation and/or
assignment there).

---
William Leibzon
Elan Networks
[email-masked]

---

From: "Martin Hannigan" <[email-masked]>
Date: Thu, 25 Jan 2007 23:31:25 -0500

> > to ISPs and organizations plus all other ip blocks that are reserved

Hmm. Understood. "And". I'm not sure I agree that this is a good
definition. I don't use CW so it doesn't affect me directly, but it
could.  But I'll take that off list.

> definition of bogons as well (it really means both allocation and/or

Good clarification.

-M<

---

From: Seth Breidbart <[email-masked]>
Date: Thu, 25 Jan 2007 23:52:03 -0500 (EST)

> They aren't bogons. They've been allocated to the RIR. They are
> unassigned to ISP's or end users, but are eligible for assignment.
>
> "Bogons is the name used to describe ip blocks not allocated by IANA
> and RIRs to ISPs and organizations

Note: assigned *by IANA and RIRs* *to* *ISPs and organizations*; those
blocks have not been assigned by the RIR to anybody, hence bogons.

Seth

---

From: [email-masked] (Justin Mason)
Date: Fri, 26 Jan 2007 10:20:48 +0000

> to the intended interpretation (or not :-(.

Yes -- in fact, IronPort's SenderBase at one point used TXT records,
containing a complex encoding of many attributes and ratings.
(Maybe they still do.)

A few of the commercial anti-spam vendors call this kind of thing
a "reputation service".

--j.

---

From: Tony Finch <[email-masked]>
Date: Fri, 26 Jan 2007 11:05:33 +0000

"Chris Lewis" <[email-masked]> wrote:
>single return, and/or being used in scoring.

AFAIK all the multi-value blacklists are combinations of multiple
underlying binary blacklists, and exist to reduce the number of
DNS lookups needed to do multiple blacklist lookups.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
BISCAY: NORTHEAST 4 OR 5. ROUGH BECOMING MODERATE OR SLIGHT. SHOWERS. GOOD.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 26 Jan 2007 12:03:04 +0100

Chris Lewis wrote:

> Many DNSBLs are already multi-valued (zen.spamhaus.org presently has
> 6 independent "flags"), and they can be constructed to be more
> sophisticated with as many as 2^24 values on a single return, and/or
> being used in scoring.

Thanks for info, I still had "sbl-xbl" instead of "zen" in a script.
I'd really love it if they'd take the same approach as SURBL (or the
former OPM) with sets represented by bits.   At the moment they have
apparently eight sets (2, 4,5,6,7,8, 10,11) encoded as integers.

Using bits 1..8 (127.0.0.2 up to 127.0.1.0) they could get a similar
effect, and even indicate if IPs belong to more than one set at the
same time.  That's what multi.surbl.org does (with 6 sets), and Jeff
asked implementors to support at least bits 1..15 for 15 sets.

The OPM scheme was also nice, use the 2nd byte for something that's
never zero (e.g. a version number for incompatible definitions of
the remaining two bytes).  That also avoids any potential trouble
with a "listed" 127.0.0.1, and supports up to 16 sets (using bit 0
is no problem if the second byte is never zero).

> It's simply a matter of deciding on the convention, publishing
> it, and having admins configure to the intended interpretation
> (or not :-(.

For the "or not" part it would be nice to publish the DNSBL draft
http://tools.ietf.org/html/draft-irtf-asrg-dnsbl as RFC, but maybe
that project still waits for an update of the IRTF RFC publication
procedure.

> I was going to mention that...

...does it work again ?  I've recently removed it because it didn't
pass the "127.0.0.2 listed" test anymore.  For similar reasons I've
removed wadb.isipp.com in 2005.

Frank

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 26 Jan 2007 10:38:35 -0500

> draft, as well as rename the extensions.

I resubmitted the I-D as draft-hall-deferrals-01.txt

The SMTP extension and the MAIL FROM command extension are both called
DEFERRALS now.

There was already a paragraph in section 6.2 about order of response being
preserved but I moved it into 6.3. Makes more sense there.

     When this response code is returned, absolute response codes for
     each of the recipients which were earlier acknowledged with a 352
     response code during the envelope exchange MUST be provided, and
     must be provided in the order in which they were originally
     received from the SMTP client.

A section head was renamed, but otherwise no other changes were made.

---

From: "Chris Lewis" <[email-masked]>
Date: Fri, 26 Jan 2007 12:02:50 -0500

> apparently eight sets (2, 4,5,6,7,8, 10,11) encoded as integers.

They've mentioned 8, but they're only _using_ 6.  2 are "reserved" ;-)

The items for XBL are somewhat in a state of flux, so they're basically
telling people that if you trust XBL, you can trust any of the other two
values if they become "real".

> asked implementors to support at least bits 1..15 for 15 sets.

I _used_ to prefer bitmapped DNSBLs, but I've been persuaded (and
learned thru personal experience) otherwise.

One main reason why they're not using bitmaps is the difficulty in
construction.  If you use bitmaps, you have to explicitly merge each
entry when you build your zone - completely reconstructing the zone
file.  With big DNSBLs that can be quite time-consuming just to do a
hack job.  There isn't any freeware code presently available to do that.
 Spfilter, for example, will die the "out of memory" death if you try to
combine, say, CBL and one or two other large DNSBLs, and doesn't have a
way to intelligently combine A records anyway (spfilter only merges
(part of) the TXT record, but not across ranges).

We used to use spfilter.  Remembering those days makes me cringe.  Was
taking over an hour to merge when I finally killed it.

In contrast, rbldnsd makes it trivial to merge.  Each zone retains its
separate identity, as a separate file if you wish.  Munging entries is
completely optional.  rbldnsd simply returns _all_ matching records in a
single DNSBL query.

By "hack job", I mean that most merging mechanisms (including spfilter)
trying to derive a single (eg: bitmapped) entry does not handle nesting
at all.  Eg: trying to coalesce a /24 PBL entry, a /22 SBL entry, and a
/32 XBL entry into a single bitmap is computationally difficult and
expensive.

This matters both to the DNSBL publisher, and the user of DNSBLs who
wishes to use local servers.  Like us.

We're currently using about 10 DNSBLs (5 of them locally constructed).
All of them are combined into local rbldnsd under a single query.  Aside
from some TXT mangling needed to make FP handling more consistent, and
remapping A values so that we have unique values across all subzones, we
don't actually have to _look_ at the zone contents when we merge - just
tell rbldnsd "publish these dnsbl files".

Remember it was simple for OPM to do this, because they only listed
/32s, and it was really only one (relatively small) database.

You _have_ to handle multiple A records if you want to deal with varying
range entries so that you get all hits (scoring for example).

In contrast, SORBS, NJABL and Spamhaus (using the combined query naming)
all use the multiple A return approach.  NJABL (mostly) publishes them
as a single file, but the entries are not merged.

There have been a few DNSBLs that have had somewhat different return
code schemes.  Like one which would return an integer "badness" score.
And another where you could encode specific questions (beyond just the
IP) in the query and get a numeric code back.  But they were only
experimental.

> removed wadb.isipp.com in 2005.

I stopped using it some time ago because it wasn't yielding much.  I
should check again.  I just turned off SORBSproxy and SORBSsocks (no
longer using SORBS at all), so, I can afford to add one ;-)

I'll direct you to this comment in the bogons zone file:

# This file does not include the following additional iana reserved blocks:
#   10.0.0.0 - 10.255.255.255     - reserved for intranet local networks
#   127.0.0.0 - 127.255.255.255   - reserved for local loop on each computer
#   172.16.0.0 - 172.31.255.255   - reserved for intranet local networks
#   192.168.0.0 - 192.168.255.255 - reserved for intranet local networks
#   224.0.0.0 - 239.255.255.255   - used for multicast routing

In other words, using 127.0.0.2 as a test entry for the bogon list is
more than a bit, shall we say, ambiguous.

If you use the zone file, you can tell whether it's "working" by the
size and datestamp on it.

---

From: "Chris Lewis" <[email-masked]>
Date: Fri, 26 Jan 2007 13:29:09 -0500

> DNS lookups needed to do multiple blacklist lookups.

There are exceptions.  One of the regional DNSBLs returns (encoded)
country codes in the A, rather than answering binary questions like "is
this IP in china?". I could build that myself[+].  I have seen
experimentations with DNSBLs that return message counts and other
non-binary info.  Several of my internal DNSBLs are computed from
scores.  I could just as easily publish the scores themselves rather
than a binary "did it exceed the threshold?".  The latter is
operationally easier for us.

Historically, most DNSBLs return binary values (or a set of binary
values).  But they don't have to, and some don't.

Secondly, obviously, you don't have to treat the binary values as
individually definitive.  SpamAssassin scores DNSBL hits - only block if
some combination of DNSBLs are hit.  We score DNSBL hits (all but one of
them is threshold level scores.  The one remaining is a bit pointless at
the moment, but we want the hit metrics and we'll add more sub-threshold
scoring DNSBLs in future).

[+] Come to think about it, I already do.  I internally serve up an
"asn" zone that contains the ASN, cidr allocation, country, state, owner
domain, abuse address as TXT records.  It's served as a subdomain via
our internal rbldnsd DNSBL servers. It's currently used for composing
metrics like "What country, ASN, domain is this crap coming from?", but
I could have the filters consulting it, and tell it to, say, nuke
particular ASNs, owner domains etc.

---

From: Claus Assmann <[email-masked]>
Date: Fri, 26 Jan 2007 20:22:37 -0800

[[Is this the appropriate list to discuss this topic?]]

1. the name is confusing and the new name isn't very helpful either.
Let's call it something like "Delayed-RCPT-Status" (DRS) or
"RCPT-Status-After-Dot" (RSAD).

2.  This is another confusing part:

     6.   The End-of-Data Response Codes
       6.1.  The 352 Response Code
       6.2.  The 353 Response Code
       6.3.  The Per-Recipient Response Codes
       6.4.  The Final Response Code

My understanding is:
352 is used for RCPT:
  6.1.    The 352 Response Code

     The 352 response code is used by a server to indicate that the
     mailbox address specified in the associated RCPT TO command
                                                 ^^^^

353 is used for end of data.

So it should be:

     6.   Additional RCPT Response Code
       6.1.  The 352 Response Code
     7.   The End-of-Data Response Codes
       7.1.  The 353 Response Code
       7.2.  The Per-Recipient Response Codes
       7.3.  The Final Response Code

BTW: it probably should be
     7.   The End of Mail Data Response Codes
to match RFC 2821.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 28 Jan 2007 13:04:32 +0100

> [[Is this the appropriate list to discuss this topic?]]

I expected the SMTP list to be flooded with 2821bis WG charter
debates, but in reality nothing happens.

> the name is confusing and the new name isn't very helpful either.

Maybe the complete technique could be simplified:

After the server offers "TBD" the client can accept it as "TBD"
keyword of MAIL FROM.  Then it sends the DATA first, afterwards
it sends RCPT TO and gets selective (per mailbox) responses as
usual, taking the known DATA into account.

The mail is selectively accepted with a 2xx for an RCPT TO.  Any
command that's not RCPT TO (like MAIL FROM or QUIT) terminates
the transmission of this mail.

Could "DATA-first mail" (DF) work ?

Frank

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sun, 28 Jan 2007 15:01:26 +0100

> usual, taking the known DATA into account.

That's basically David Nicols' DATAFIRST proposal.

> Could "DATA-first mail" (DF) work ?

It could but in reality quite a lot of mail is rejected before it even
hits the DATA stage (either because none of the recipients exists or
because of checks which don't need the content (DNSBL, greylisting,
...). With DATAFIRST all this mail will have to be transferred only to
be rejected. I suspect that this will sharply increase the bandwidth
requirements. (David proposed to offer DATAFIRST only to trusted
senders, which limits its usefulness)

	hp

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 28 Jan 2007 21:24:33 +0100

Peter J. Holzer wrote:
 
> David Nicols' DATAFIRST proposal.
[...]
 
> It could but in reality quite a lot of mail is rejected before
> it even hits the DATA stage (either because none of the recipients
> exists or because of checks which don't need the content (DNSBL,
> greylisting, ...).

Some of the checks (incl. DNSBL and SPF) are possible before RCPT,
unless you intend to apply them selectively per receiver.

If you want to get RCPT before DATA it's simple, just don't offer
"DATAFIRST".  A similar idea is Hector's HEAD proposal, splitting
DATA into HEAD and BODY somehow, for folks interested to check the
"header first" (roughly as with TOP in POP3).  Maybe he gave up on
it because it won't help with DKIM body hashes.

> With DATAFIRST all this mail will have to be transferred only to
> requirements.

It should be visible in the log files of major site.  If they check 
more than only the size between DATA and dot, otherwise they can't
be interested to get "DATAFIRST" anyway.

What I didn't like in draft-hall-inline-dsn was the sequence of
per recipient responses after DATA matching the sequence of RCPT
before DATA.  It would be more plausible if it implies PIPELINING.

Frank

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sun, 28 Jan 2007 23:06:10 +0100

> Some of the checks (incl. DNSBL and SPF) are possible before RCPT,

But rejecting because the recipient doesn't exist, isn't possible before
the recipient is known, and that is the most common cause for permanent
failures.

> unless you intend to apply them selectively per receiver.

In fact I do. SBL-XBL is currently used for all addresses except
postmaster, and I plan to enable PBL for a few addresses to test it. I
don't check SPF currently (doesn't seem worth it), but if I did I
certainly would only do it on a per-recipient basis. Greylisting needs
the recipient, and while having a part of the data (especially the
message-id) would help, I don't think it would offset the increase in
traffic caused by transferring the same message several times and
getting a temporary failure each time until the greylisting period is
over.

> "DATAFIRST".

But I want both. I want the ability to reject on a per-recipient basis
before I receive the data, and I want the ability to selectively apply
content filters for different recipients.

> more than only the size between DATA and dot,

We are not a major site, but we are running spamassassin between DATA
and dot, so we do have a reason to return different responses for
different recipients. We are using an ugly hack at the moment
(http://www.hjp.at/projekte/qpsmtpd/cf_wrapper/), but I would like
something cleaner. (I am fully aware of course that I would need to keep
the hack for many years, until INLINE-DSN (or DEFERRAL, or however it is
going to be called) is implemented by most of our neighbours)

> otherwise they can't be interested to get "DATAFIRST" anyway.

I'm not interested in DATAFIRST. I'm interested in returning
per-recipient results to DATA.

> before DATA.

That's what the proposal is about. It allows several return codes for
DATA. The exact mechanism might be different (for example, there could
be a multiline-response instead of multiple single-line responses, or
the return codes can be matched by address instead of order), but these
are minor changes.

> It would be more plausible if it implies PIPELINING.

The proposal suggests that a implementation offering INLINE-DSN also
offers PIPELINING, but frankly, I fail to see the connection.

	hp

---

From: Tony Finch <[email-masked]>
Date: Sun, 28 Jan 2007 22:53:00 +0000

>"RCPT-Status-After-Dot" (RSAD).

RCPT-status-after-data would be better - data sent using BTAT or BURL
doesn't end with a dot.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
BISCAY: EASTERLY 4 OR 5, OCCASIONALLY 6 IN SOUTHWEST. SLIGHT OR MODERATE.
FAIR. GOOD.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 29 Jan 2007 00:05:23 +0100

> The proposal suggests that a implementation offering INLINE-DSN
> also offers PIPELINING, but frankly, I fail to see the connection.

Anything else follows the model "one command gets one response",
with multi-line responses as worst case.  Multiple reponses is a
derivation from this model.  With PIPELINING the client is at
least used to the idea that it somehow must match responses to
pending commands.  Probably a matter of taste, of course clients
are not forced to use it if they don't like it.

Frank

---

From: Claus Assmann <[email-masked]>
Date: Sun, 28 Jan 2007 15:13:52 -0800

> with multi-line responses as worst case.  Multiple reponses is a

LMTP doesn't. SMTP clients are most likely able to deal with LMTP
too as it's just a "minor" change. In fact, I just implemented this
for the SMTP client in my MTA.

> are not forced to use it if they don't like it.

Let's make support for PIPELINING a MUST for this extension too
(as it is the case for LMTP).

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 29 Jan 2007 01:13:56 +0100

> (as it is the case for LMTP).

+1  (admittedly I never looked into any LMTP code or standard :-)

---

From: Claus Assmann <[email-masked]>
Date: Sun, 28 Jan 2007 18:24:17 -0800

Wrt section "6.4. The Final Response Code":

Why do we need a final reply code? Isn't that code the same as the
minimum over the RCPT reply codes (at least wrt the reply code
severity)? Hence it is not need to send this reply code as it can
be deduced from the existing reply codes.

We only get to this stage if there are at least two RCPTs with
different reply codes and hence 353 was used as reply for end of
mail data.  The final reply code is 250 iff at least one of the
RCPTs has a 2yz reply code. If none of the RCPTs has a 2yz reply
code then the final reply code must be 4yz; it can't be 5yz because
it that case all RCPTs would have a 5yz reply code, which means it
was useless to return individual reply codes (unless you want to
return different 5yz reply codes with different enhanced status
codes, however, even then the final reply code is still 5yz and
does not contain any additional information).

PS: "response code" should be replaced by "reply code" in the draft
to match RFC2821.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Mon, 29 Jan 2007 10:49:26 +0100

> be deduced from the existing reply codes.

As I understood it the final response code serves as a "commit"
indication. A final 2xx reply means that the server has accepted the
message and will try to deliver it to all the recipients which got a 2xx
reply individually. If the the server cannot accept the connection, for
example, because the queue is full, it can still return a 4xx or 5xx
code, although it has earlier returned 2xx codes for some or all
recipients. 

That should make implementing the server a bit easier, as the message
has to be queued only after all the recipients are known. Otherwise you
either need to queue the message for each recipient indivually or you
need to update the list of recipients after the message is already
queued.

It also may reduce the risk of resending a message which was already
accepted because the connection broke down, but I'm not sure that really
works:

If you send a message to 100 recipients, and the connection breaks after
43 replies, how many will get the message? It might be anything between
43 and 100, because you don't know how many reply codes got lost.
With the final reply code, the server only accepted the message after
sending *all* of the reply codes. The chance that 57 reply codes were
successfully sent from the server's perspective but never reached the
client aren't that big, so the client can assume that the server didn't
accept the message and resend it with all recipients.

(OTOH, especially with long recipient lists and lengthy and possibly
fragile content filters, maybe it is more robust to commit for each
recipient individually: If the client wants to send to 100 recipients,
and the server crashes after the 10th recipient every time, that way the
message will get through in 10 transactions. It will never get trough if
the server needs to scan the message for all recipients to queue it)

	hp

---

From: "David Nicol" <[email-masked]>
Date: Mon, 29 Jan 2007 16:35:02 -0600

On 1/28/07, Peter J. Holzer <[email-masked]> wrote:
> > > David Nicol's DATAFIRST proposal.
> > [...]
>

they aren't mutually exclusive. Nobody is saying they are, but someone
who does not realize that they aren't might think that they are, and
that there is some kind of dispute, where there isn't.

> the return codes can be matched by address instead of order), but these

the one request, one response model is crucial to SMTP servers -- you
can write a smtp server that essentially does

      smtp_connect;
      while (accept and fork){
           send_smtp("EHLO blah blah blah");
           send_smtp(craft_smtp_response(receive_smtp())) while 1;
      };

and multiple response lines makes that no longer work; or means that
the receive_smtp() has to have a lot more data in it than if we are reading
a data block or not.

so if the ordering is important, maintain it; how about a big response that
is a well-defined XML block?

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 30 Jan 2007 12:33:43 -0500

> [[Is this the appropriate list to discuss this topic?]]

I would imagine that [email-masked] is the right place to talk about
the protocol, particularly if its going to be advocated for standards
track, but I'll deal with questions where I find them

> "RCPT-Status-After-Dot" (RSAD).

deferred-rcpt-responses would be most descriptive but that's a bit greedy
given that command-line space is fixed. I don't really care what people
call it. Robots will be the 99.9999% use case and they won't care either.
Probably some kind of acronym like DRR or whatever would be most efficient
as far as that goes.

Best outcome I can see here is for a plurality of implementors to request
a specific name. Otherwise we are all going to be wanting our pet name so
any name is as bad as any other.

>      7.   The End of Mail Data Response Codes

I changed the section 6 heading in the current draft. The additional
breakdown looks reasonable and appropriate so I'll do that in the next
edition if there's any real interest in pursuing this further.

---

From: Claus Assmann <[email-masked]>
Date: Tue, 30 Jan 2007 09:48:31 -0800

> I would imagine that [email-masked] is the right place to talk about

If the current participants of this thread agree, we can move the
discussion over there.

> Probably some kind of acronym like DRR or whatever would be most efficient

"DRR" is fine, currently I call it "RSAD" in my implementation, but
that's trivial to change.

> edition if there's any real interest in pursuing this further.

I'm interested. I'm currently implementing it in my MTA, and I can
implement it in the (still?) most widely used MTA if we get this
to an RFC.

Please make also the change to require PIPELINING, as LMTP does:

5.  Implementation requirements

   A server implementation MUST implement the PIPELINING [PIPELINING]
   and ENHANCEDSTATUSCODES [ENHANCEDSTATUSCODES] ESMTP extensions.

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 30 Jan 2007 12:48:17 -0500

> be deduced from the existing reply codes.

The final code is where the server says that it is taking responsibility
for the message. The delayed RCPT codes just report on whether or not the
content is acceptable for the associated recipient.

It's possible that the recipients will all accept the message, but the
server is still unable to accept the message for processing for some
reason (server going down, whatever).

> PS: "response code" should be replaced by "reply code" in the draft

will add that as well as the clarification above to the next draft

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 30 Jan 2007 12:50:25 -0500

On 1/29/2007 4:49 AM, Peter J. Holzer wrote:

> 43 replies, how many will get the message?

Hopefully none of them, since the server was not able to fully accept the
message for delivery.

It might be worth putting a synchronization marker after the full list of
recipients has been acknowledged by the client in order to ensure that
both end points are at the same state when final code is issued.

---

From: Tony Finch <[email-masked]>
Date: Tue, 30 Jan 2007 18:13:07 +0000

>both end points are at the same state when final code is issued.

Ugh, no, that's horrible for performance. The right solution is:
http://www.ietf.org/internet-drafts/draft-fanf-smtp-rfc1845bis-00.txt

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
NORTH UTSIRE SOUTH UTSIRE: MAINLY NORTHWESTERLY 6 OR 7, OCCASIONALLY GALE 8,
BACKING SOUTHWESTERLY 5 FOR A TIME. ROUGH OR VERY ROUGH. SHOWERS THEN RAIN.
GOOD BECOMING MODERATE.

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 30 Jan 2007 13:34:15 -0500

> http://www.ietf.org/internet-drafts/draft-fanf-smtp-rfc1845bis-00.txt

That's a good idea but your I-D would have to get published as
standards-track before it could be linked in as a dependency.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Tue, 30 Jan 2007 20:47:38 +0100

> >Hopefully none of them, since the server was not able to fully accept the

That's what I understood to be the intention of the final code. If the
client receives the final code, it knows the server has taken
responsibility for the message and can forget about it (except for
recipients with 4xx replies, of course). If it doesn't receive the final
code, it must assume that the server hasn't accepted it and must resend
it for all recipients. I'm just not sure whether that really reduces the
number of duplicate messages. It it conceivable that the server sends
the final code before it notices that the connection has been lost. In
that case the client will resend the message to all recpients, while
otherwise it would resend the message only for those recipients for
which it hasn't received a reply code. So the final reply code reduces
the probability of duplicated messages, but if they happen, they are sent
to more users. I have no idea whether that results in more or less
duplicated messages overall.

> http://www.ietf.org/internet-drafts/draft-fanf-smtp-rfc1845bis-00.txt

That only lets you put checkpoints inside the data. It doesn't let the
client specify how many reply codes it received for data (there's no
need for that in unextended SMTP). Of course the server could save the
reply codes it has already sent and quickly replay those if the client
resumes at the end of data, so maybe that doesn't matter.

	hp

PS: I just subscribed to [email-masked], so as far as I am concerned
we can continue the discussion there.

---

From: Tony Finch <[email-masked]>
Date: Tue, 30 Jan 2007 20:09:39 +0000

>resumes at the end of data, so maybe that doesn't matter.

Right. The server's replies after data would be handled in a similar
way to the envelope before the data.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
ROCKALL MALIN HEBRIDES BAILEY: SOUTHWEST 7 TO SEVERE GALE 9, LATER DECREASING
5 OR 6. ROUGH OR VERY ROUGH, OCCASIONALLY HIGH. OCCASIONAL RAIN. MODERATE OR
GOOD.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Tue, 30 Jan 2007 21:25:30 +0100

On 2007-01-29 16:35:02 -0600, David Nicol wrote:
> they aren't mutually exclusive.

I didn't say they were. DATAFIRST includes the ability to return
per-recipient results to DATA. That's what I like about it.
Unfortunately it also removes the ability to return per-recipient
results before getting DATA, which I don't like.

So I prefer Eric's proposal, although it is admittedly more complex.

>           send_smtp(craft_smtp_response(receive_smtp())) while 1;

I think you mixed server and client a bit, here.

> the receive_smtp() has to have a lot more data in it than if we are reading

Yep, it gets more complicated. The server's implementation of the data
command has to be able to send intermediate reply codes (I think that's
how I would implement it in qpsmpd - simply write out the replies for
each recipient and return DONE instead of OK/DENY/whatever from the
data_post hook; the final reply would then be returned from the queue
hook as normal).

The client OTOH has to remember how many reply codes it is expecting and
match them up with the recipients.

> is a well-defined XML block?

I'm not sure I would want to introduce the complexities of XML into
SMTP. When I suggested a single multi-line response earlier in this
thread I was more thinking about nesting SMTP responses like this:

250-Message accepted by some recipients. Individual replies follow:
250-250 <[email-masked]> accepted message.
250-550-<[email-masked]> rejected message.
250-550 see http://www.example.com/mail-policy/ for details.
250 250 <[email-masked] accepted message.

which would be equivalent to

353 Message accepted by some recipients. Individual replies follow:
250 <[email-masked]> accepted message.
550-<[email-masked]> rejected message.
550 see http://www.example.com/mail-policy/ for details.
250 <[email-masked] accepted message.
250 Message queued.

in Eric's draft.

	hp