Subject: Re: [Asrg] A New Plan for No Spam / Velocity Indicator

From: Scott Nelson <[email-masked]>
Date: Thu, 01 May 2003 11:20:30 -0700

At 10:08 AM 5/1/03 -0700, J C Lawrence wrote:
>On Thu, 1 May 2003 07:26:18 -0600 (MDT) 
>Vernon Schryver <[email-masked]> wrote:
>> Mailing list subscribers have always tended to be clustered on SMTP
>> servers, even before big ISPs like AOL.  
>
>The current distribution of target MXs is a lot different than the
>picture 10 years ago.  For one, the total number of target MXes for the
>same size list population is more than an order of magnitude higher
>today.
>
>What may be more interesting is that the distribution curve is now and
>was then a pretty standard power curve.
>

I would like to add that Moore's law has been quite accurate in
the last 10 years, adding approximately 2 orders of magnitude to
our ability to process mail in the same period.

The user base is not now, nor has it ever expanded at the same rate
as the technology.  (if it had, then SF lovers digest would still be
clobbering us today.)

In other words, I don't think we should rule out proposals that require 
greater network or computer time than was available 10 years ago.
Nor for that matter would a proposal that required more power than
we have /now/ be necessarily ruled out (at least, not by me) 
since it's going to take a long time to get people to switch over.

Of course, increasing bandwidth isn't good, so if a proposal requires
more bandwidth (or computing power or whatever), then it should give
us something worthwhile.

I haven't been keeping close track on this thread, could somebody
post a summary (or point me to an existing summary) of what the 
supposed benefits and penalties of this proposal are?

Scott Nelson <[email-masked]>
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Barry Shein <[email-masked]>
Date: Thu, 1 May 2003 17:44:27 -0400

>our ability to process mail in the same period.

Although the portions of the systems subject to Moore's law helps
there are critical components which have not improved much at all in
the same time period.

For example, disk speed isn't all that much faster today than it was
10 years ago. Where CPUs have improved by about 1000-fold, disk has
improved maybe 10-fold or so, and mechanically much less (1993,
3600RPM, 2003, 7200-10,000RPM, seek times have barely changed at all
remaining in the single-digit ms range, albeit cheaper.)

So, unless you're crunching numbers which all fit in memory, Moore's
law is not a reasonable gauge of e-mail processing improvement. And
often this sort of processing can only proceed as fast as its weakest
link, which is likely to be disk speed.

Cheaper RAM enabling much more aggressive and accurate caching of
course clouds this picture somewhat.

But to imply that because CPUs have gone from 1MHZ to 3GHZ that we can
now process 3000x as much mail is unsupportable and misleading.

-- 
        -Barry Shein

Software Tool & Die    | [email-masked]           | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202        | Login: 617-739-WRLD
The World              | Public Access Internet     | Since 1989     *oo*
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 1 May 2003 16:42:36 -0600 (MDT)

> From: Barry Shein <[email-masked]>

> ...

It's also in a sense true.  The volumes of mail are processed today
for reasonable costs are 3000X higher than the volumes of the era when
CPUs ran at single digit MHz.  Tolerable mail volumes have gone up by
100X since the more recent days when CPUs ran at 10s of MHz.  10,000,000
mail messages/day is a lot today but no more remarkable than 100,000 mail
messages/day were 10 years ago.

Of course, the change is related much less to CPU clock speeds than
file system speeds.  Disk drives have not become more than 50 or 100
times faster in that last 40 years (e.g. "big" and "fast" Data Products
drives of the early 1960's served ~50 MByte at 100-300 milliseconds
seek times), but file systems are running 1000 or 10,000 times faster
than they were 15 years ago.    In the late 1980's, 1 MByte/sec through
read() or write() was respectable.  Today you get yawns for 10
GBytes/sec.  The changes are not in seek (head motion) or search
(rotational delay) times, but in transfer times and in the number of
spindles.  Years ago very fancy "drums" had a few dozen heads.  Today
a fast storage system will have more than that many drives acting like
one super fast drive thanks to in part to 1000 times more bits in RAM
cache than ancient drives had in spinning iron oxide.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Barry Shein <[email-masked]>
Date: Thu, 1 May 2003 19:17:25 -0400

Moore's law describes the growth of transistor density over time.

What has increased the volume of mail volume processing is the
dramatic fall in price of processing. A single-MIPS machine used to be
half-million dollars, now a 3GHZ machine is around one thousand
dollars, and that includes all the system components (disk, ram, etc)
in both examples.

Since mail is reasonably distributed in nature that dramatic drop in
price allows one to have 100 mail servers all working in (relative)
unison and approximately additive capacity.

However, Moore's law per se would have been satisfied if that
half-million dollar machine were still $500K, but had increased from
1MHZ to 3GHZ.

So something else has occurred, albeit arguably related to Moore's Law
if you impose a price per computron per pound rule (thus higher
transistor density should come down in price.)

I guess the two, exponentially faster/denser hardware and price drops,
have become closely associated in our heads. But it's not necessarily
case.

In fact Cray (badly) bet it would not be the case and gambled their
business on the fastest/densest hardware being extremely expensive.

Ah well. But the point is it wasn't obvious to everyone, so 20/20
hindsight and all that.

And *MY* point was that faster CPUs are only part of the picture when
viewing servers, a problem people like myself have to grapple with all
the time when some breathtakingly fast CPU'd machine won't even barely
character echo because the mail load is backed up on disk I/O usually
from a spam attack. At that moment it just doesn't feel like Moore is
smiling on me.

-- 
        -Barry Shein

---

From: J C Lawrence <[email-masked]>
Date: Thu, 01 May 2003 19:37:55 -0700

On Thu, 1 May 2003 17:44:27 -0400 

> now process 3000x as much mail is unsupportable and misleading.

You have accurately stated why several large mail shops have invested so
heavily in solid state battery backed disks for their spools.  It allows
them to satisfy the SMTP commit rules while keeping IO latency damned
low.

-- 
J C Lawrence                
---------(*)                Satan, oscillate my metallic sonatas. 
[email-masked]               He lived as a devil, eh?		  
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Tom Thomson" <[email-masked]>
Date: Fri, 2 May 2003 10:39:51 +0100

>there are critical components which have not improved much at all in

Can't disagree with that.

>remaining in the single-digit ms range, albeit cheaper.)

I don't think this bit is quite accurate.

Lets look at discs:  First of all, the common discs in 1993 were 2400 rpm,
not 3600 - if think of 3600rmp and 11 ms seek time for 1993 you had better
compare it with 15k rpm and 4ms seek time for 2003, alternatively you can
talk about 2400rpm and 15ms for 1993 and compare with 10krpm and 8ms for
2003.  Second, in 2003 drives have substantial internal buffering and do
plenty of read-ahead; for a remailer/list exploder application this reduces
the number of seeks quite a bit and also reduces the impact of rotational
latency - the application is not a random access one, where perhaps 12 times
would be a reasonable disc speed up estimate, but one where the internal
drive buffering and read-ahead has a substantial effect so that there has
been a much greater improvement.  Third, head settling times are much lower
so that short seeks are now zero cost - in 1993 even a two track seek would
lose you a rev. Fourth, surface density is much higher, so that transfer
duration is much lower, and we have much larger mainstore so that we can
cache in the OS much more effectively store. Fifthly, with this particular
application I can use two discs in such a manner that I just about eliminate
seeks of more that 1 cylinder, and 1 cylinder seeks are zero cost on modern
discs.

Now lets look at processors: do we have anything like a 1000 times speed up?
Well, clock times do have a 1000 times speed up but I've never had much
faith in clock time as a measure of speed.  How much has FSB bandwidth
changed? Or main store bandwidth? - for some applications that will be a
much better measure.  So, unless your crunching numbers that fit in L1 cache
you have nothing like a 1000 times speed up.

>link, which is likely to be disk speed.

Much more likely to be main store bandwidth for the application in question,
I think, but thet too is obscured by caching and of course it depends on bus
width (but most systems use a store bus narrower than a complete L3 cache
line, so it's not as much obscured as it might be).  Of course that doesn't
affect the conclusion - Moores law is not a useful gauge of e-mail
processing improvement.

>course clouds this picture somewhat.

It certainly does. I think it clouds it enough so that the bottleneck will
not be where you think it is.

>now process 3000x as much mail is unsupportable and misleading.

Here we are in total agreement.

Tom Thomson