Subject: RE: [Asrg] A New Plan for No Spam / Velocity Indicator

From: "Tom Thomson" <[email-masked]>
Date: Thu, 1 May 2003 11:52:04 +0100

>Note that contrary that document, it generally makes no sense to talk
>about valid BCC fields in incoming legitimate mail.  See page 37 of
>RFC 2822.  I wouldn't be surprised if SpamAssassin treats a BCC
>header as evidence of evil, because I can't recall ever seeing one
>except in spam.  More important and also contrary to that document,
>RFC 822 and RFC 2822 do *NOT* require that every incoming message name
>the receipient in any SMTP header.  Most legitimate mailing list
>messags are cases in point.

Phill's document already points out that mailing lists are an exception
which require exceptional treatment, so your "case in point" appears to be
vacuous padding to try to fool us into thinking your message has more
substance than it has.
As for the bcc case, I used to use an MUA which generated n+1 copies of the
body - 1 for each bcc recipient containing a bcc line with only that
recipient, and one for all To and cc recipient containing no bcc line. These
were transmitted separately - multiple Rcpt To lines with one Data phase for
teh To and cc recipients, single Rcpt lines with a data phase for each bcc
recipient. Since Phill is as much of an antique as I am, maybe he too
remembers MUAs which did rather pleasant and intelligent things that didn't
conform to the modern desire for "efficiency" at all costs (even at the cost
of effectiveness) and maybe that's where theerrorneous statement comes from.
An easy mistake to make if you grew up with civilised MUAs.

>What discourages me most about this mailing list are self-described
>expert statements such as "the RFC822 message standard requires every
>message to have a valid To: CC: or BCC: field identifying the recipient,
>making adjustment where necessary to account for messages relayed
>through mailing lists."  There are obvious reasons why RFC 822 and
>RFC 2822 do not require mailing list exploders to anything of the sort.

Heavy protestations of knowledg and expertise which are clearly false bother
me too, but I also know the story about the pot and the kettle.

Tom

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 1 May 2003 07:26:18 -0600 (MDT)

> From: "Tom Thomson" <[email-masked]>

> vacuous padding to try to fool us into thinking your message has more

He did not say that mailing list software should be changed, but in
effect said that all major past and current SMTP mailing list software
violates words in RFC 822 that do not exist.  In fact RFC 2822 not
only does not require that recipients be listed, but allows an MTA to
remove BCCs.  He probably made the mistake common among end users who
complain about spam not containing their addresses, because they do
not know about or understand the significance of the SMTP envelope.
If you have not spent much time fighting spam or SMTP servers in more
than their external configuration spam, it is easy to not know about
the SMTP envelope.

> An easy mistake to make if you grew up with civilised MUAs.

I still use an MUA whose first version probably predates the MUA that
you are thinking of.  It sends a single copy with the BCC recipients
mentioned only in the envelope.  The only older UNIX MUA that I've used
of also sent only a single copy.  (Email systems I used before UNIX
were too different to mention.)  The MUA that I recall that did as you
describe was a fancy, "extensible," and "user friendly" package offered
as a replacement for those two older MUAs.  I've often wished that my
preferred MUA would do as you describe, because I've learned the hard
way the importance of preceding a BCC'ed message with another that says
something like "the following is a BCC so don't reply to all."

> me too, but I also know the story about the pot and the kettle.

Are you sure you see all of the relevant applications of the pot and
kettle story?  Do you see some of the obvious reasons why RFC 822,
RFC 821, RFC 2822, and RFC 2821 do not contain the words that the
other person thought they do?  Did you check RFC 2822 on BCC headers?

Requiring that every message list the recipient in a To, CC, or BCC
header would require that the MTA serving the MUA that is the mailing
list exploder send each copy to the next hop MTA by a separate SMTP
transaction.  The exploder MUA might do as it does now, sending messages
to its MTA, each addressed to from 10, 100s, or perhaps 1000s of
targets, but unlike now, the MUA would add to message BCC headers for
each addressee in the batch.  To honor the blindness required of a
BCC (and the "Security Considerations" in RFC 2822), the MTA must
convert those individual batch messages into sparate messages.  Each
new message would contain a single BCC header mentioning a single
addressee, and would be handled in a separate SMTP transaction.
Alternatively, the exploder MUA could generate the separate messages,
each with a single BCC header.

Have you watched an SMTP client responsible for passing a modest (few
1000 subscribers) or larger mailing list, particularly on hardware
and networks more than 10 years old?   Mailing list subscribers have
always tended to be clustered on SMTP servers, even before big ISPs
like AOL.  Requiring separate transactions for each addressee would
increase the load on the system by 10 times.  I suspect that is why
sendmail at least as early as the mid-1980s was willing to sort by
next-hop and use a single SMTP transaction when possible.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: J C Lawrence <[email-masked]>
Date: Thu, 01 May 2003 10:08:52 -0700

On Thu, 1 May 2003 07:26:18 -0600 (MDT) 
Vernon Schryver <[email-masked]> wrote:

> 1000 subscribers) or larger mailing list, particularly on hardware and
> networks more than 10 years old?  

Yup, did that for years.

> Mailing list subscribers have always tended to be clustered on SMTP
> servers, even before big ISPs like AOL.  

The current distribution of target MXs is a lot different than the
picture 10 years ago.  For one, the total number of target MXes for the
same size list population is more than an order of magnitude higher
today.

What may be more interesting is that the distribution curve is now and
was then a pretty standard power curve.

> Requiring separate transactions for each addressee would increase the
> load on the system by 10 times.

Some research has been done on this space, in particular with an
attention to increasing delivery performance for mailing lists to
variously standard distributions of destination addresses.  Chuq von
Rospach did the most recent survey here that I'm aware of -- you can
find the numbers etc in the mailman-developers list archives.

  Loose summary as I recall them: The optimal number of RCPT TOs per
  message for best performance outbound delivery is approximately 5, and
  is most definitely below 10.

My own testing with my list's MX distribution (known to be highly
unusual as AOL+Hotmail+MSN+Yahoo form less than 3% of my subscriber base
rather than the normal ~30% of most lists) confirms that 5 is pretty
close to the sweet spot for me as well.

> I suspect that is why sendmail at least as early as the mid-1980s was
> willing to sort by next-hop and use a single SMTP transaction when
> possible.

I wouldn't hold Sendmail's design or approach up as any sort of
performance model.  It has only very recently come back to being even
close to par in delivery performance against Exim, QMail and Postfix,
and then thru only very large and invasive changes to the delivery
architecture.

Given a $2,000US commodity machine its just not that hard to sustain
2,500 outbound deliveries a minute these days.

-- 
J C Lawrence                
---------(*)                Satan, oscillate my metallic sonatas. 
[email-masked]               He lived as a devil, eh?		  
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 1 May 2003 14:07:39 -0600 (MDT)

> From: J C Lawrence <[email-masked]>

> ...

> ...

Ok, requiring separate transactions would increase the load by 5 times.

> ...

Agreed, today the CPU and disk costs are far lower.  However, if you 
can run 2,500 outbound deliveries/minute at 5 addressees/transaction,
you'll only be able to run 500/minute at 1 addressee/transaction.

If each transaction involves a 5 KByte body and an ignored ~200 bytes
of SMTP commands and TCP/IP headers, then a T1 can carry about 38
deliveries/second or 2280 deliveries/minute.  That does not quite jibe
with 2500 deliveres/minute at 5 addressees/delivery, but it's the
right order of magnitude.  Perhaps the fact that I ignored DNS traffic
as as the TCP/IP headers and SMTP commands and responses is part of
the discrepancy.

---

From: J C Lawrence <[email-masked]>
Date: Thu, 01 May 2003 19:34:20 -0700

On Thu, 1 May 2003 14:07:39 -0600 (MDT) 

>> ...  Loose summary as I recall them: The optimal number of RCPT TOs
>> per message for best performance outbound delivery is approximately
>> 5, and is most definitely below 10.

> Ok, requiring separate transactions would increase the load by 5

No.  The arithmetic isn't that trivial, tho its close.  Think about it a
bit.  While it is a power curve, there's still a heck of a lot of MXes
in the thin end of the wedge, and even more tellingly, the majority of
MX collections are not modulo 5 (surprisingly its pretty close to 80%
aren't mod 5).  Without doing the full math (I'll leave that to the
enterprising reader), I'd say the actual factor should come out
somewhere between 2.5 and 3.  

My own non-scientific experiments with VERP with smaller lists support
that estimate.

>> ...  Given a $2,000US commodity machine its just not that hard to
>> sustain 2,500 outbound deliveries a minute these days.

> you'll only be able to run 500/minute at 1 addressee/transaction.

Nope.  The same math applies again.  2,500 deliveries a minute is not
quite 12,500 addresses delivered per minute due to the same power curve
and modulo problems.  Additionally it ignores the gain from chained SMTP
transactions over the same connection (TCP setup/teardown being a
non-trivial portion of the TCP overhead).  

> as as the TCP/IP headers and SMTP commands and responses is part of

Most of my systems are connected via multiple T3s to a local MAE, PAIX,
or other similarly well connected nexii.  Local bandwidth is not a
problem or significant factor for high sustained delivery rates.  A
solid localhost DNS cache with enforced longer minimum TLLs (if your
repeat rate is low) keeps DNS latencies and traffic down.  Add some
tuning of your spool strategies, slow MX domain routing and a few other
tricks and you can keep things running at capacity.

When you get to those sorts of ranges, and especially when you start
trying to push much last 3,500 deliveries per minute on a commodity box
(~$2K) its the design and management of the local IO chain that is your
killer.  You very quickly hit a point where throwing more RAM and CPU
into the mix doesn't gain much, but popping IO latency from your DASD is
the big gainer, and that tends to be expensive unless you play fast and
loose with the commit rules.  Wietse Venema demonstrated and charted
this space quite clearly a couple years back.

However, we're getting considerably off topic here.