Subject: Re: [Asrg] FTC tells us what is SPAM (NO! what is commercial!)

From: Amir Herzberg <[email-masked]>
Date: Sun, 19 Dec 2004 10:52:51 +0200

>>FTC Issues Final Rule Defining What Constitutes a "Commercial Electronic 
>>Mail Message"

> This has vanishingly little to do with defining what spam is.

That's correct, and indeed I think the rule does not claim to define 
what is spam. Instead, it defines several classes of often-undesirable 
e-mail (commercial, etc.).

If e-mail belonging to such often-undesirable classes would have 
included an appropriate `warning` category label, this could provide 
very good mechanism for users and providers to filter out undesirable 
messages (based on user's and/or providers policies).

One possible (and imho very useful) definition of spam is `e-mail 
belonging ot one of the often-undesirable categories, but without an 
appropriate label`. The FCC rule may help define these categories, and 
technical mechanisms, and potentially complementary legal measures, can 
enforce appropriate labeling. See 
http://www.cs.biu.ac.il/~herzbea//Papers/ecommerce/Spam.htm for a simple 
  cryptographic protocol to enforce correct labeling for content 
selection (filtering).

Best, Amir Herzberg

   They're
> defining what messages are subject to the CAN SPAM act, which
> regulates all commercial e-mail, solicited and unsolicited.
> 
> I don't think much of the rules that CAN SPAM defines, but don't shoot
> the messenger.  The Congress directed the FTC to regulate some kinds
> of email, and the FTC is clarifying what mail that is.
> 
> Regards,
> John Levine, [email-masked], Taughannock Networks, Trumansburg NY
> http://www.taugh.com
> 
> 
> 
> 
> ------------------------------
> 
> Message: 3
> Date: Thu, 16 Dec 2004 13:13:29 -0800 (PST)
> From: William Leibzon <[email-masked]>
> Subject: Re: [Asrg] FTC tells us what is SPAM
> To: John Levine <[email-masked]>
> Cc: [email-masked]
> Message-ID:
> 	<[email-masked]>
> Content-Type: text/plain; charset="us-ascii"
> 
> On Thu, 16 Dec 2004, John Levine wrote:
> 
> 
>>>FTC Issues Final Rule Defining What Constitutes a "Commercial Electronic
>>>Mail Message"
>>>
>>> Notice Includes Criteria For Determining the "Primary Purpose" of an
>>> E-Mail Message
>>
>>This has vanishingly little to do with defining what spam is.  They're
>>defining what messages are subject to the CAN SPAM act, which
>>regulates all commercial e-mail, solicited and unsolicited.
>>
>>I don't think much of the rules that CAN SPAM defines, but don't shoot
>>the messenger.  The Congress directed the FTC to regulate some kinds
>>of email, and the FTC is clarifying what mail that is.
> 
> 
> I agree. I've very little faith in the "You can spam all you want" act by 
> congress. It seems to have allowed large spammers to tell ISPs that they 
> are not doing anything illegal because they are CAN-SPAM compliant (which 
> might or might not be true, but verifying takes long time) and if ISPs do 
> threated a disconnect then they begin to use lawyers to keep them up longer.
> 
> Nevertheless that Congress clarified what consititutes a COMMERCIAL EMAIL
> is very important because that directly related to UCE type of spam and
> commercial part is very often what makes a difference if deciding how to
> react to bulk sender on your net.
> 
> ---
> William Leibzon
>    mailto: [email-masked]
> Anti-Spam and Email Security Research Worksite:
>    http://www.elan.net/~william/emailsecurity/
> Whois & DNS Network Investigation Tools:
>    http://www.completewhois.com
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: smime.p7s
> Type: application/x-pkcs7-signature
> Size: 1118 bytes
> Desc: S/MIME Cryptographic Signature
> Url : http://www1.ietf.org/pipermail/asrg/attachments/20041216/8c673d47/smime-0001.bin
> 
> ------------------------------
> 
> _______________________________________________
> Asrg mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/asrg
> 
> 
> End of Asrg Digest, Vol 8, Issue 14
> ***********************************
> 
> .
> 

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Matthew Elvey <[email-masked]>
Date: Sun, 19 Dec 2004 22:48:27 -0800

> technical mechanisms, and potentially complementary legal measures, 
> can enforce appropriate labeling. See 
> http://www.cs.biu.ac.il/~herzbea//Papers/ecommerce/Spam.htm for a 
> simple  cryptographic protocol to enforce correct labeling for content 
> selection (filtering).

Proposals in the form of I-Ds (produced using xml2rfc, perhaps) are 
encouraged; please consider providing one.

The original RBL started out as a list of IPs from which to shun ALL 
traffic.  I think it was a mistake (given hindsight) to have switched to 
just shunning email traffic from IPs that were the source of abuse.  As 
we solve the email spam problem, spammers switch to other media (spim, 
wikispam,...) so we should be prepared.

---

From: John Levine <[email-masked]>
Date: 20 Dec 2004 18:13:36 -0000

>> One possible (and imho very useful) definition of spam ...

Please keep in mind that attempts to create a definition of spam are
specifically excluded from the ASRG's area of interest.

Taxonomies of objectionable mail might be interesting, though.

Regards,

---

From: Laird Breyer <[email-masked]>
Date: Tue, 21 Dec 2004 10:52:59 +1000

> specifically excluded from the ASRG's area of interest.

This is intriguing. While it isn't clear that having a specific
definition of spam is actually useful for blocking it, this policy does
have implications on the solution set.

Would it be accurate to say that ASRG is interested in Z-mail
prevention solutions where Z-mail is a label for some set of messages,
and Z-mail can be taken to be spam? 

If that's the case, then any solution of interest would also apply to
other labeling problems, e.g. ham-prevention. Furthermore, such a
restriction on the solution set would likely exclude some economic
solutions, as these solutions don't usually make sense for
ham-prevention say, and any solution the ASRG would be interested in
would have to work for Z-mail, whatever Z-mail turns out to be.

What am I missing here?

-- 
Laird Breyer.

---

From: John Levine <[email-masked]>
Date: 21 Dec 2004 02:28:14 -0000

>prevention solutions where Z-mail is a label for some set of
>messages, and Z-mail can be taken to be spam?

Not really.  That still implies that there's a fixed set of mail
that ASRG is trying to stamp out.  I presume you have looked at
the charter at http://www.irtf.org/charters/asrg.html.

My position is that any anti-spam project has to identify the flavor
of spam that the project is attempting to address, but I don't expect
projects all to address the same thing.

Pluralistically,
John

---

From: "Walter Dnes" <[email-masked]>
Date: Mon, 20 Dec 2004 23:46:52 -0500

On Tue, Dec 21, 2004 at 10:52:59AM +1000, Laird Breyer wrote

> would have to work for Z-mail, whatever Z-mail turns out to be.

  The thing that you may be missing is that "one size does not fit all".
One man's spam is another man's ham.  My idea of spam is different from
someone else's.  That's why end-user-control of some sort is so
desirable, so that end-users could stop (block/reject/filter/whatever)
the email that *THEY* don't want.  Rather than "blocking spam", the
problem is better stated as "blocking unwanted email".  Each person has
their own set of wants.

  There is another, legal, problem with "blocking spam".  The word
"spam" has a lot of negative baggage and spammers^H^H^H^H^H^H^H^H email
marketers with lawyers will hit back with lawsuits claiming "It's not
spam, it's ooga-booga" (substitute euphemism-du-jour for spam).  A good
parallel is the current spyware controversy, where Claria has sued anti-
spyware program publisherss, claiming to have been libelled/slandered by
being labelled as "spyware" rather than "adware".  Their lawsuit does
not currently affect "adware removal" programs.  A similar potential
problem exists in the spam arena.  Rather than a "spam-blocking" program
labelling something as spam and blocking it, we should have an
"unwanted-email-blocking program".  When the spammer's lawyer screams
"It's not spam, it's ooga-booga.  Stop blocking my client's emails with
your 'spam-blocking' program", we should respond that it's really an
'unwanted-email-blocking-program'.  And that our customer has indicated
that he doesn't want your email.

-- 
Walter Dnes <[email-masked]>
An infinite number of monkeys pounding away on keyboards will
eventually produce a report showing that Windows is more secure,
and has a lower TCO, than linux.

---

From: Laird Breyer <[email-masked]>
Date: Tue, 21 Dec 2004 15:15:51 +1000

> the charter at http://www.irtf.org/charters/asrg.html.

The charter does emphasise that the ASRG is looking for well
specified problems, and makes it clear that a multitude of
techniques to tackle them is desirable. As such, doesn't that
encourage attempts to define spam?

The original message you responded to suggested a possible
definition, followed by a link to some protocol which takes
that definition as a basis. I guess I don't see what prompted
you to warn against attempting to define spam. Could you explain?  

> projects all to address the same thing.

I agree with that position. Perhaps I read too much into your comment.

-- 
Laird Breyer.

---

From: Laird Breyer <[email-masked]>
Date: Tue, 21 Dec 2004 15:37:54 +1000

> problem is better stated as "blocking unwanted email".  Each person has

Not necessarily. Some people do not have the luxury of defining spam
as that which they don't want. For example, children whose mail is
monitored, or employees whose mail is filtered centrally.

There are some classes of solutions which don't allow end-user
control.  For example, in proposals where postage or a bond is
required for sending mail, the recipient is only indirectly involved
with the definition of spam. Instead, such a user can indicate a price
for accepting mail, but if the sender is rich or at least has access
to some other source of funding, then the recipient must accept the
spam together with some form of compensation. 

-- 
Laird Breyer.

---

From: Bill Cole <[email-masked]>
Date: Tue, 21 Dec 2004 08:45:19 -0500

>encourage attempts to define spam?

Probably, which is why it needs to be expressly excluded.

This is a relatively young forum for discussions about spam. One 
lesson from other places is that there is no point in defining what 
is or is not spam because even if there were to be an agreement of 
some sort to settle on some useful definition, there will be some 
people who want to shun other sorts of mail or some people who think 
some subset of spam should not be shunned or both.  If one defines 
'spam' in some way that is technically discernable with any sort of 
useful reliability, so much unsolicited bulk email will be excluded 
that practically no one will take that definition seriously. In the 
end, debates over what is or is not spam always devolve into 
scholastic hair-splitting over things like the nature of permission 
and the meaning of 'bulk' or 'broadcast' and so on.

It seems less hopeless to approach the issue of controlling spam in 
the same sort of ways that have already proven effective in the 
field, and attack subsets of spam that lend themselves to 
programmatic detection, in order to narrow the ranges of behavior 
that are useful for spammers. A great example is intentional 
selective slowing of mail, either by extended response delays in SMTP 
or explicit 4xx errors for a 'first try' of a particular piece of 
mail. Such tactics do not impede things like the Verisign spam I got 
last week, because Verisign hires a class of spammer that tries to 
operate above-ground and with some attention to the norms of SMTP and 
have their own facilities for sending mail. It DOES impede the 
Goodyear and Kraft spam, because those fine upstanding companies 
contract with spammers who act as if SMTP runs on a one-way channel. 
Simply requiring all SMTP clients to implement certain bits of SMTP 
sanely stops a lot of spam, but it doesn't stop all spam and once in 
a while it stops mail which no one would call spam. It is absurd to 
define spam as mail which is identified by some set of technical 
tricks, but existing practice makes it pretty clear that one can do a 
fairly effective and precise job of controlling spam by selecting the 
right set of technical tricks.

-- 
Bill Cole
[email-masked]

---

From: Barry Shein <[email-masked]>
Date: Tue, 21 Dec 2004 17:35:52 -0500

The idea of complete end-user control is a utopian ideal which died a
few years ago when spammers' zombie bots began hitting ISPs by the
thousands simultaneously.

At this point there are only two choices, either block at the ingress
(e.g., blocking network blocks and similar), or begin charging around
$1000/month for e-mail accounts to pay for the bandwidth and hardware
necessary to keep up with the unfettered flow (essentially, a
dedicated server with a dedicated T1 per mailbox.)

Anyone who says otherwise, notably EFF, is just ignorant.

-- 
        -Barry Shein

Software Tool & Die    | [email-masked]           | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202        | Login: 617-739-WRLD
The World              | Public Access Internet     | Since 1989     *oo*

---

From: Seth Breidbart <[email-masked]>
Date: Tue, 21 Dec 2004 18:21:07 -0500 (EST)

Barry Shein <[email-masked]> wrote:

> The idea of complete end-user control is a utopian ideal which died
> a few years ago when spammers' zombie bots began hitting ISPs by the
> At this point there are only two choices, either block at the
> ingress (e.g., blocking network blocks and similar), or begin
> charging around $1000/month for e-mail accounts to pay for the
> bandwidth and hardware necessary to keep up with the unfettered flow
> (essentially, a dedicated server with a dedicated T1 per mailbox.)
>

What about counterexamples, such as Panix?

Seth

---

From: Laird Breyer <[email-masked]>
Date: Wed, 22 Dec 2004 09:36:51 +1000

> dedicated server with a dedicated T1 per mailbox.)

Since you're an ISP, could you indicate roughly the numbers you are
talking about (per user if you like)?  How many inbound raw mails per
day? How much total mail data (megabytes) per day? How much total
traffic of all types per day? 

I don't know if the list already has discussed those kinds of numbers, 
but I'd like to get an idea of the current order of magnitude of the
problem, from the provider's POV. If others have numbers to share for
ISPs, let's hear them too.

-- 
Laird Breyer.

---

From: "william(at)elan.net" <[email-masked]>
Date: Tue, 21 Dec 2004 15:51:53 -0800 (PST)

It should be ISP's responsibility to help other ISPs and as such most 
appropriate way to deal with zombies is for ISPs to indicate which of
their ip blocks are used by broadband end-users and which are real mail
servers. If majority of ISPs did that, that would effectively cut the
zombie spam only to be within ISP's own network, which I'm sure they'd
deal with a lot more promptly (and if they don't its really their
problem and their users who have to suffer but nobody elses).

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: Barry Shein <[email-masked]>
Date: Tue, 21 Dec 2004 19:11:12 -0500

So what you're saying is that anyone with a broadband connection
should be prevented from delivering email directly.

I don't disagree, in the current milieu, but I'm sure there are some
who would disagree.

-- 
        -Barry Shein

---

From: Devdas Bhagat <[email-masked]>
Date: Wed, 22 Dec 2004 06:02:34 +0530

> ISPs, let's hear them too.

August numbers for work:
http://nixcartel.org/~devdas/minute.png

I'll try and get the new numbers out and displayed on the official
website.

Devdas Bhagat

---

From: "william(at)elan.net" <[email-masked]>
Date: Tue, 21 Dec 2004 17:18:43 -0800 (PST)

> I don't disagree, in the current milieu, but I'm sure there are some

Everyone who has not agree to be responsible for consequences equvalent to 
having user run its own mail server. I take it that those users with windows
who ended up with zombied machines don't have a clue about this and I
certainly don't want them anywhere near my mail system. A responsible 
tech user with his own linux box is another matter entirely but I believe 
those users can make arrangement with their ISPs to get static ip and have 
that ip included as valid smtp transmission initiating entity.
 
-- 
William Leibzon
Elan Networks
[email-masked]

---

From: Seth Breidbart <[email-masked]>
Date: Tue, 21 Dec 2004 20:05:15 -0500 (EST)

> should be prevented from delivering email directly.

Actually, he wrote that ISPs should indicate which IP addresses are in
use by customers, and which by mailservers.  It's up to anybody else
to use that information as they choose; it isn't a Port 25 block.

> I don't disagree, in the current milieu, but I'm sure there are some

Many disagree with blocking Port 25.  I haven't seen anyone disagree
with the providing of accurate information.

Seth

---

From: Barry Shein <[email-masked]>
Date: Tue, 21 Dec 2004 20:58:45 -0500

I don't know since we block so much at the ingress, such as vast
swaths of broadband pools which have been a problem, etc.

It's difficult to measure what you block.

I will say that another huge part of the problem is the stuff no one
ever sees which is the bazillions of user unknowns as the spammers use
various dictionary type attacks. There's far more user unknown
attempted deliveries than other deliveries.

So, even if you knew what went (or would go) into peoples' mailboxes,
opening the floodgates (AKA accepting EFF's point of view) would also
necessitate sifting through all that spew, increasing costs
significantly.

Someone would have to pay for that also, and removing those blocks is
the only way to allow what's called "end-user control" since you don't
know it's for a non-existant user until you let them begin delivery.

-- 
        -Barry Shein

---

From: Barry Shein <[email-masked]>
Date: Tue, 21 Dec 2004 20:59:47 -0500

I have no idea what panix does.

---

From: Seth Breidbart <[email-masked]>
Date: Tue, 21 Dec 2004 21:09:41 -0500 (EST)

> I have no idea what panix does.

In response to your claim that ISPs have to either do ingress blocking
or charge $1000/month, I offered panix as a counterexample.  Ingress
blocking is a user option.  Everything past it is, too; stuff is made
available to make things like spamassassin easy to configure, but it's
all up to the user to decide what to do.  Email accounts, with shell,
Usenet, etc. cost $100/year, two orders of magnitude below your claim.

Seth

---

From: Barry Shein <[email-masked]>
Date: Tue, 21 Dec 2004 21:11:37 -0500

> to use that information as they choose; it isn't a Port 25 block.

You're just detailing that distinction to us because you think it
makes you look intelligent? I don't understand why you bother.

 > with the providing of accurate information.

Are you trying to be helpful, or do you just like to type?

-- 
        -Barry Shein

---

From: Barry Shein <[email-masked]>
Date: Tue, 21 Dec 2004 21:23:56 -0500

So you're saying panix doesn't block any, e.g., broadband IP pools or
similar, at a system level?

Do you work at panix or otherwise have authoritative info on this? I'm
not asking that to be baiting, but oftentimes it's not the most public
information.

-- 
        -Barry Shein

---

From: Laird Breyer <[email-masked]>
Date: Wed, 22 Dec 2004 14:09:40 +1000

> It's difficult to measure what you block.

I think it's enough to discuss actual messages accepted for delivery or
filtering by the SMTP servers. See below.

> attempted deliveries than other deliveries.

The dictionary attacks use up incoming bandwidth, and processing or
storage.

Without knowing the numbers, it's hard to know if the bandwidth used
is a large fraction of incoming capacity, but I doubt it. 

However, the processing involved is in principle fully scalable.
There's only the need to exchange a request which can be denied by the
SMTP server for nonexistent users. So the full email is only
transmitted if the mail is accepted.

Figuring out whether a recipient address represents a real user is a
fast operation in principle, ie no matter how many valid user
addresses, a yes/no needs O(1) operations with the right algorithms.

So from a scalability POV, I'd expect the costs of spam to be mainly
proportional to those messages which are sent to a valid user.

> necessitate sifting through all that spew, increasing costs

Interestingly, Google started this fad of offering 1 gigabyte
mailboxes. I don't know if they crunched the numbers and figured it's
economically worthwhile, or whether they just have IPO money to burn.

-- 
Laird Breyer.

---

From: "Walter Dnes" <[email-masked]>
Date: Wed, 22 Dec 2004 00:21:33 -0500

On Tue, Dec 21, 2004 at 03:37:54PM +1000, Laird Breyer wrote

> monitored, or employees whose mail is filtered centrally.

  The "client" in this case is the entity that pays for the inbox.  In
the case of children, minors can't legally make contracts anyway.  It
has to be approved by their parent.  And the parent is the one paying
for the inbox as well.

  My email inbox at work is no more my property, than is the the
computer I use at work.  It is paid for, owned, and controlled by my
employer.  I use it in my capacity as an agent of my employer.

---

From: der Mouse <[email-masked]>
Date: Wed, 22 Dec 2004 00:27:14 -0500 (EST)

> talking about (per user if you like)?  [...]

I'm not bzs, and the numbers I have aren't quite the ones you ask for,
but they're harder numbers than I've usually seen cited. :-)

One of my tasks at $DAYJOB has been to deal with the crippling load
caused by spam blowback.  (Yes, crippling.  It had gotten to the point
where they were having to shut off all incoming SMTP during peak
periods just to keep the mailserver up at all.)  I wrote a small shim
layer whose job was to turn away unknown local-parts very cheaply.

The reason this is relevant is that I had it keep stats.  Thus, I know
what percentage of the RCPTs offered were rejected - ie, were totally
invalid local-parts.

At the peak of the blowback, when I first put it in place, it was
running about one in two hundred - 99.5% of the RCPTs offered were
invalid.  (These days, it's much better; it cruises along at only
something like 75% - I just now took a data sample, and while it was
too small to be really all that significant, I have no reason to think
it was unrepresentative, and it showed 93 bad and 32 good.  This
matches with my own memory of the sort of numbers I've seen when I've
looked at them in the past.)

That's just totally invalid local-parts.  Some amount of that is
doubtless non-spam (a very small amount - see below); there is also
doubtless a significant quantity of spam among the mail aimed at
existing local-parts.  I did arrange to capture some of the rejected
traffic, and when I looked, it was approximately all bounce blowback
from spam - other messages were notable by their rarity, and most of
*them* were non-bounced spam.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Seth Breidbart <[email-masked]>
Date: Wed, 22 Dec 2004 15:15:40 -0500 (EST)

(Responses to multiple messages included.)

>> and Z-mail can be taken to be spam? 
. . .
> One man's spam is another man's ham.

More important, I think, is that having a single definition of spam
works against solutions.

Somebody has a system that works well against Type X spam, and wants
to improve it.  Arguing that it isn't good because it ignores Type Y
spam just wastes effort.

Then somebody else comes up with a system that works OK against Type Y
spam.  If they both spend their time arguing over whether "spam" means
"Type X" or "Type Y" that's time they're not spending improving (or
combining) their systems.

> Rather than a "spam-blocking" program labelling something as spam
> and blocking it, we should have an "unwanted-email-blocking
> program".  When the spammer's lawyer screams "It's not spam, it's
> ooga-booga.  Stop blocking my client's emails with your
> 'spam-blocking' program", we should respond that it's really an
> 'unwanted-email-blocking-program'.  And that our customer has
> indicated that he doesn't want your email.

That doesn't work: the spammer will get a shill to sign up with your
ISP and claim he _wants_ the spam.

> encourage attempts to define spam?

No, that discourages them, because having one blessed definition works
against having multiple techniques each of which is most effective
against a different part of the problem.

> I agree with that position.

But having one definition works against that.

[email-masked] wrote:

> Of course, as far as the recipient, THEIR decision about what is
> spam and what isn't, and what they want to receive and what they
> don't, is the ONLY opinion which matters in the final analysis.

Providing your goal is to market your services to them, yes.

> This is yet another reason why a fine-grained permissions-based
> approach, where the *recipient* can decide what E-mails they do and
> do not want to receive (based on who the sender is, and what the
> mail contains) is so decidedly the right way to pursue this problem.

It is _a_ right way.  It is not the only one.

>  Especially with threats of attorneys et al, there is NO feasible
> way for a spammer to try to sue a recipient who has chosen to refuse
> to accept, or read, the mail a spammer has tried to jam into the
> recipient's mailbox.

You might have noticed that spammers don't limit themselves to
"feasible" lawsuits.

[email-masked] wrote:
> On 22 Dec 2004, John Levine <[email-masked]> wrote:

>>>This is yet another reason why a fine-grained permissions-based
>>>approach, where the *recipient* can decide what E-mails they do and
>>>do not want to receive (based on who the sender is, and what the mail
>>>contains) is so decidedly the right way to pursue this problem.
>
>>So long as you are willing to spend an unlimited amount of money on an
>>e-mail infrastructure that accepts and stores terabytes of spam,
>>almost all of which will be discarded unread, 
>
> You are presuming that (1) spamming will continue to be lucrative;

No, just that it will continue to be done.

> that (2) an approach such as I envision will reduce neither the
> numeric number of spam messages nor their average size;

or at least leave those numbers large.

> and (3) that spammers will continue to be able 
> to recruit zombie spambot armies to do their mailings for them.

That's certainly likely for the foreseeable future.

> First off, I believe that a fine-grained permissions list (with
> permissions based on who messages come from, along with what the
> nature of the contents of those messages are) and which by default
> will not allow either "large", HTML or attachments from untrusted
> senders, together will virtually eliminate E-mail as an effective
> vector for recruiting spambot zombie armies (it will in fact
> virtually eliminate the efficacy of sending worms and viruses in
> E-mail messages).

In order for it to eliminate zombies, it has to be implemented by
_everybody else_.  That's not going to happen.

> Secondly, making HTML-burdened E-mail acceptance CONTINGENT upon the
> sender being whitelisted by the recipient

You're assuming that you can tell whether or not the _recipient's_ MUA
will attempt to interpret a message as HTML.

> Third, making spam filtering more effective and harder to defeat or
> evade will dramatically reduce the payback to spammers, and the
> payback is what motivates spamming in the first place.

For some spammers, maybe.  Many others sell their services, and the
worldwide shortage of suckers is not expected soon.

>>and a fabuously complex
>>spam filter control panel that almost nobody will use, 
>
> Oh, that's TRULY rubbish.  While obviously it would be CONCEIVABLE
> to implement such a filter in a stupid and clumsy way, a reasonable
> implementation could make this VERY user-friendly (far more
> user-friendly, in fact, than typical "security permissions" for NTFS
> file systems).

Prove it.  Come up with a reasonable implementation that my mother can
handle.

>>ISPs tell me that when they have crummy filters that leak a lot of
>>spam, people are constantly asking to be able to tune the filters.
>
> The fact is that users who are able to simply and easily control
> THEIR OWN spam filtering, using techniques which are understandable
> and logical, are less likely to require as much ISP support.

As spammers learn to evade those controls, the ISP has to upgrade
them.

>>If you're in the United States, please consider them in relation to 47
>>USC 230 and section 8(c) of CAN SPAM.
>
> I don't think that's relevant to what we're talking about here (at
> least not what *I* am talking about).  I'm not talking about
> companies, individuals, or governments suing spammers, I'm talking
> about spammers (or those whose behavior might arguably look like
> spamming) suing (or threatening to sue) ISPs.

Did you read those?  They provide _immunity for ISPs_ for good-faith
spam filtering.

[email-masked] wrote:

>> The user has still made a clear decision - one to delegate their right
>> to accept/refuse to the ISP. As long as the ISP's contract makes this
>> clear, then I believe there's no difference in the 2 cases. 
>
> Sounds good, until the "spammer" can find a case (even just ONE)
> where the intended recipient actually WANTED the E-mail in question,
> and said user didn't feel they had in fact granted their ISP the
> 'right' to MIS-BLOCK mail that the user actually wanted.

Then the ISP show the contract the user agreed to, and the law
granting it immunity.

> Now, perhaps you can come up with a contract/TOS that has enough 
> waivers and disclaimers in it,

And every ISP already has; have you looked at any?  (I have, and they
all say the ISP can do what it wants.)

> but I still think that the "spammer"/"mailer" 
> will probably have adequate claim to bring it up in court, SOMEWHERE.

I don't care what the laws in Spamzania say.

>> The user can exercise choice by switching to an ISP which allows greater
>> control by its users (and as John has pointed out, probably charges a
>> premium to cover the costs of offering that control).
>
> I think you're being far, far too presumptuous about the
> practicality of switching to a different ISP.  Maybe you haven't
> been paying attention, but the ISP world has been consolidating (and
> especially if we're talking about broadband type services... okay,
> yes, dialup ISPs are more plentiful).

Mail Service Providers are growing.  They don't have to provide
Internet access; some people prefer buying unbundled services.

>  So if I didn't like Comcast's terms (and I don't, for example, like
> their policy of blocking port 25, which would enable me to send mail
> through my own domain provider's SMTP server) what, exactly, are my
> choices for switching to a different ISP?

$100/year for a Panix account with shell, mail, Usenet, etc.

$0/year and up for email accounts with Outblaze, gmail, Yahoo, . . .

> Meanwhile, even if one presumes that I'm therefore going to use
> Comcast for my connectivity, the fact that Comcast insists on
> blocking port 25 pretty well precludes me from signing up with a
> different and more-satisfactory "ISP", at least for outgoing
> SMTP-type mail service.

A decent one uses "SUBMIT".

Seth

---

From: Barry Shein <[email-masked]>
Date: Wed, 22 Dec 2004 18:04:13 -0500

> proportional to those messages which are sent to a valid user.

In my not inconsiderable experience as a system's load, measured in
requests per second, increases the response time per request tends to
follow an exponential.

So, your comments are only true if the hardware and software expands
at some rate to match the request rate. It doesn't have to be linearly
expanded, but when you hit a knee in the curve you either move that
knee (i.e., expand resources) or suffer the consequences (exponential
growth in response time.)

Where I'm using the word "exponential" colloquially to mean
"super-linear" if that bothers anyone, "hockey-sticks" might be a
better verb.

To bring this back to earth, spammers' behavior is such that, barring
response (blocking them, killing them, increasing resources, pulling
the plug on your internet connection, etc) they will find the knee in
that curve.

I am not speaking hypothetically, I am speaking from my not
inconsiderable...

-- 
        -Barry Shein

---

From: Barry Shein <[email-masked]>
Date: Wed, 22 Dec 2004 18:55:45 -0500

This is a good note: Read it again!

-- 
        -Barry Shein

---

From: "Walter Dnes" <[email-masked]>
Date: Wed, 22 Dec 2004 19:40:13 -0500

On Wed, Dec 22, 2004 at 03:15:40PM -0500, Seth Breidbart wrote

> ISP and claim he _wants_ the spam.

  I have a remote inbox at an ISP that has hacked up Qmail to allow
*END-USER-CONFIGURED* rulesets that block email *AT THE SMTP STAGE*.
Each user can have their own custom filters.  As a matter of fact, each
of the 10 email accounts that a user is allowed, can have different
rulesets.  I host my email domain with them.  My waltdnes ID is heavily
guarded.  My postmaster and abuse ID's are unfiltered, unless I run into
a really snotty spammer.

  As I said earlier, one size does not fit all.  The goal should be
end-user control, even if it's only an "on-off" switch for spam-blocking.
A serendipitous side-effect is that it defeats the shill-signs-up-at-ISP
stunt.

> You might have noticed that spammers don't limit themselves to

  In that case, why are you participating in this forum?  You might get
sued for contributing to tortious interference with a spammer's income.

---

From: Laird Breyer <[email-masked]>
Date: Thu, 23 Dec 2004 10:58:06 +1000

> growth in response time.)

That's true, but you aren't seriously suggesting that ISPs won't upgrade
their equipment over time? The internet hasn't finished growing yet,
and service providers are middle men, so their incoming traffic
is bound to grow.

It's perfectly reasonable (don't you think?) to expect an ISP to
deploy resources proportional to their number of users, which brings
us back to the question of the mail request rate per legitimate user
and how it can be eventually limited(*). Some real numbers have
already been offered on the list.

(*) why not (still) consider mail requests for non-users? Verifying
whether an address corresponds to a local user is cheap in principle
nevertheless. Yes, you'll be seeing very many incoming socket
connections, but you don't need to set up a full SMTP service
connection until you already know that the recipient is valid. Now
cheap connections together with Moore's law ought(comment?) to handle
the increase in requests over time at a constant upgrade cost for the
front end.

> the plug on your internet connection, etc) they will find the knee in

That's a lot of different responses, with different costs and
benefits. How can we get an overview without numbers? Or is the idea
that ASRG offers a grab bag of solutions and implementors silently
pick and choose what works for them without giving feedback? I don't
have a preference, just wondering how it's supposed to work.

-- 
Laird Breyer.

---

From: Laird Breyer <[email-masked]>
Date: Thu, 23 Dec 2004 11:26:00 +1000

>  > *them* were non-bounced spam.

It's particularly interesting because it suggests that the bulk of
those invalid requests are protocol related, not directly related to
the spammers' creativity.

-- 
Laird Breyer.

---

From: Barry Shein <[email-masked]>
Date: Wed, 22 Dec 2004 22:56:50 -0500

> and service providers are middle men, so their incoming traffic

Yes, but upgrading equipment just to handle spam load is a bad thing,
like any crime no one wants to pay for it yet it must be paid for
nonetheless.

The increased resource pressure from spammers isn't just "in the
noise", it's quite significant. Right now I think what we're seeing is
a mixture of upgrading anyhow and trying to adjust, and just putting
more bread crumbs in the hamburger (slower mail delivery, fooling
around with filters/blocks that are questionable, etc.)

 > already been offered on the list.

I don't think the "per legitimate user" is a realistic measure any
longer since some huge percentage, I'll guess more than half, is to
unknown users and other mischief like endless probes etc.

 > nevertheless.

It's not all that cheap. Remember there are aliases and mailing lists
and so on to deal with.

Cheap is a relative thing. Don't sell "cheaper" (than a full delivery)
as "cheap".

I've been hit by 1500 zombie pc's simultaneously pumping the same
exact spam at us.

How cheap does cheap have to be to deal with an attack like that?

 Yes, you'll be seeing very many incoming socket
 > connection until you already know that the recipient is valid.

I don't get you, SMTP is SMTP, until I talk some SMTP with them I
don't know who their intended recipient is.

 Now
 > the increase in requests over time at a constant upgrade cost for the

My experience sez that spammers seem to follow a parkinsonian law of
resource exploitation which means if you get more resources they'll
use more resources.

Otherwise how could they be such a nuisance to garageshopisp.com and
AOL simultaneously?

I don't think Moore's law covers it, not by a long shot.

Moore's law mostly applies to processing power (specifically,
transistor density.)

Disk speed and network bandwidth costs, two very critical factors in
mail server scaling, don't go up anything like Moore's law.

 > have a preference, just wondering how it's supposed to work.

ASRG is a research group.

Anxious as we may be for a solution I still think we're quite a ways
away from any meeting of the minds on what the problem is or, perhaps
put better, what a solution might look like.

-- 
        -Barry Shein

---

From: Laird Breyer <[email-masked]>
Date: Thu, 23 Dec 2004 16:04:39 +1000

> around with filters/blocks that are questionable, etc.)

I don't deny it's significant. I'm just trying to help clear up the issues.

> unknown users and other mischief like endless probes etc.

I've got a back of the envelope scheme below to address that specific
problem. Purely theoretical, but I really do think that issue can be 
discounted.

> Cheap is a relative thing. Don't sell "cheaper" (than a full delivery)

I'm talking about really cheap. I'll give a rough idea below of what I
have in mind. Note also that der Mouse already implemented something
of that nature as described in his post, but probably not the way I'm
going to describe it.

PROBLEM:
Say you have one million valid email addresses. Each address is between
10 and 20 characters long. The problem is to cheaply discard inbound 
SMTP connections with invalid addresses in the RCPT command. 

WHAT TO DO:
Use a dedicated proxy in front of your SMTP servers. This proxy keeps
all the addresses in memory, either hashed or set up as a trie. 
This guarantees that given an address string, figuring if it's valid
is literally a few machine instructions, independent of the number of
valid addresses. You can reload those lists once an hour say, and
you can precompute aliases if you like, whatever.

The proxy has these addresses in memory, but its real job is to accept
socket connections. An SMTP server, once the client has connected,
typically waits for HELO, MAIL, RCPT, DATA followed by the mail
message.

The proxy server is not going to be a full SMTP server. Conceptually,
it's going to wait for HELO, MAIL, RCPT, and at that point check if
the address is valid. If valid, the proxy opens a real SMTP connection
to your SMTP servers and replays the HELO, MAIL, RCPT commands, then
transparently forwards the SMTP conversation.  If the address was
invalid, there's no need to bother the SMTP servers, just return an
error and do whatever the protocol allows to close the connection.

So your proxy code really doesn't need to be very smart at all, and
the processing load for each attempted connection (up until the RCPT)
will be really cheap: buffer space for a couple of lines of input to
be replayed, and a very small number of instructions to check the
address, and after that it's mindless copying of bytes. So ten or
twenty megabytes for the full list of addresses, and a couple of K for
buffering each open socket, if you're paranoid about ultra-long SMTP
commands.
  
The bottleneck will be how many simultaneous open sockets your server
can handle over time. That'll depend on your kernel/hardware limits and
the time you have to wait until a socket is reusable.

Either way, let's take Devdas Bhagat's statistics 

I don't know if these numbers include invalid users or not, but let's
say 60K invalid requests per minute, and another 10K valid requests.
That's one thousand rejected requests per second, and less than 200
long lived requests. If you wait 30 seconds until you can reuse a
socket, then you'll need about 35,000 socket addresses in total in
this case. Or if you have fewer, you can use several proxies in
parallel.

> resource exploitation which means if you get more resources they'll

Probably. One problem at a time.

> mail server scaling, don't go up anything like Moore's law.

You'll note that the proxy scenario I painted needs no disk space, but
does need good I/O hardware and a typical PC's worth of RAM.

> put better, what a solution might look like.

Good point. However, I do think some real numbers once in a while help
focus the issues.

BTW, the proxy scenario I gave above isn't intended to be a ready
solution, rather it's an argument to prop up my claim that handling
the invalid requests can be handled scalably.

-- 
Laird Breyer.

---

From: der Mouse <[email-masked]>
Date: Thu, 23 Dec 2004 01:44:19 -0500 (EST)

> Note also that der Mouse already implemented something of that nature
> as described in his post, but probably not the way I'm going to
> describe it.

[much snippage -dM]
> Use a dedicated proxy in front of your SMTP servers.  This proxy
> keeps all the addresses in memory, either hashed or set up as a trie.

> The proxy server is not going to be a full SMTP server.
> Conceptually, it's going to wait for HELO, MAIL, RCPT, and at that
> point check if the address is valid. If valid, the proxy opens a real
> SMTP connection to your SMTP servers and replays the HELO, MAIL, RCPT
> commands, then transparently forwards the SMTP conversation.

This is..._almost exactly_ what I did.  The only ways in which it isn't
are (a) that it HELOs as itself, rather than relaying the HELO from the
client, and (b) it inserts a Received: header at the beginning of the
data (if the conversation gets to that point).

> the invalid requests can be handled scalably.

It certainly was a solution to a very real practical problem when I
wrote it; their mailserver went from tottering on the brink of total
collapse to fully usable in the length of time it took the MX record
changes to work their way through the world's DNS caches.  Not a really
large ISP - your transaction counts were high by orders of magnitude
compared to them - but not a mom-&-pop either.

Anyone who would like a copy of the code is welcome to it (minus a few
site-specific fragments that really need to be written per-site, like
checking that the domain portion of the address is right).  Drop me a
line and if I get lots of requests I'll put it up for ftp.  No
guarantees, of course, but I figure at worst someone may steal some
ideas from it.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Devdas Bhagat <[email-masked]>
Date: Thu, 23 Dec 2004 20:13:53 +0530

On 23/12/04 16:04 +1000, Laird Breyer wrote:
<snip>
> say 60K invalid requests per minute, and another 10K valid requests.

The breakup is very roughly 25% invalid users, 25% DNSBLs and 50% local
IP blocks.
And the invalid user query is a database hit (40 million+ users and
those keep changing very rapidly).

Devdas Bhagat

---

From: Barry Shein <[email-masked]>
Date: Thu, 23 Dec 2004 15:07:21 -0500

> the invalid requests can be handled scalably.

All I get from it is that an in-memory hash or other fast lookup
algorithm, if it's not being used already, might speed up response
time assuming that your model improves on the resources which are
actually strapped.

That doesn't address scalability, it just shifts the knee of the curve
much like a faster CPU or more memory might.

Scalability has to address:

a) The critical resource(s)
b) Permutational effects

Where (b) is often subtle and significant. For example, if you use N
servers you have to load balance between them. How does that
decision-making scale as N increases. It's often somewhere between
O(N^2) and O(N!).

At any rate, as fascinating as some might find it I sincerely don't
believe the problem with spam, even at the ingress, is going to be
solved or even ameliorated for very long by some improvement in
recipient validity processing.

It's like someone breaking your windows unchallenged and someone says
I know, let's find cheaper ways to make window panes!

It helps no doubt but somehow misses the point, including whether the
cost is in the panes or installing the panes, etc.

-- 
        -Barry Shein

---

From: Laird Breyer <[email-masked]>
Date: Fri, 24 Dec 2004 11:57:16 +1000

> time assuming that your model improves on the resources which are

I'm sure this sort of thing is being used already where needed.
However, it addresses scalability because each handled request uses up
a fixed amount of resources (bounded number of processing
instructions, fixed memory, I/O channels) independent of the number of
users or the number of queries. So there's not going to be a slow down
with size. Moreover, it's clearly a parallelizable concept.

> much like a faster CPU or more memory might.

Yes, it shifts the burden of spawning full-fledged SMTP sessions onto
the subset of valid requests. The question is whether the resulting
cost of handling an invalid request is so insignificant with off the
shelf hardware to allow orders of magnitude more requests than your
current full fledged SMTP. I believe it is in this case.

So we don't need to argue that invalid requests are as difficult to
address as spam sent to valid users, which is all I'm trying to
convince you of. Put differently, so long as (invalid user) requests
are a constant couple orders of magnitude more than (valid user)
requests, they are a pure optimization problem. If however the invalid
requests do grow exponentially compared to the valid requests, the
above won't work.

> decision-making scale as N increases. It's often somewhere between

Come on, for practical purposes you're not looking for the optimum,
just a local optimum so the big oh isn't as bad as you suggest. 

The load involved in censoring invalid user SMTP requests is going to
be purely I/O bound, since the picture I painted clearly won't tax the
CPU. Each system can handle so many simultaneous socket connections,
and each such connection is going to cost a nearly identical amount of
resources as any other connection. 

So I doubt you'll need complex load balancing. Just pick a server at
random for each request. This is different from full fledged SMTP
because with SMTP, the total resources needed will vary according to
the input, so the resultant load varies too. You will still need to load
balance your real SMTP servers.

> cost is in the panes or installing the panes, etc.

Well, if you're going to start with window analogies, I've got one too ;-) 

You're saying we can't just worry about someone breaking the windows,
we also need to worry about those who try and fail, because they leave
scratches. I'm saying make your windows scratch proof (cheaply) and then
only bother with people who actually break the windows. There's enough
window-breakers to keep us occupied.

-- 
Laird Breyer.

---

From: Laird Breyer <[email-masked]>
Date: Fri, 24 Dec 2004 19:14:58 +1000

> those keep changing very rapidly).

Right, 15K invalid database hits per minute. What I'm proposing needs a fairly
static list, as that's what is needed to build a quick lookup table (here
quick means you can find out your answer in n character comparisons and jumps
, where n is the number of characters in the email address, independent of the
total number of users. This would also work for IP addresses).

While your 40 million user database has constant activity, I presume the 
vast majority of usernames have a long lifetime, a few days at least. 
So if you build username snapshots at fixed intervals, then users would have
to wait a little for their new address to become "live" etc.

-- 
Laird Breyer.

---

From: Seth Breidbart <[email-masked]>
Date: Fri, 24 Dec 2004 11:05:48 -0500 (EST)

Laird Breyer <[email-masked]> wrote:

> While your 40 million user database has constant activity, I presume
> the vast majority of usernames have a long lifetime, a few days at
> least.  So if you build username snapshots at fixed intervals, then
> users would have to wait a little for their new address to become
> "live" etc.

If you're willing to suffer a _few_ false positives on the filtering,
and do a little more arithmetic, you can do better.  Construct some
number of hash functions whose output is an address that holds 1 bit.
Hash each email address with each function, and turn the pointed-at
bit on.  When email comes in, apply all the hash functions and check
the pointed-at bits.  If they're all on, pass the message through.

A new user gets turned on immediately, by turning on the necessary
bits in the table.  A deleted user doesn't go away until the next full
rebuild.  There's a small chance (depends on hash table size, number
of hash functions, number of valid email addresses) of a random
address being accepted due to collisions, but this can be made
acceptably small.  (Using a parameterized set of hash functions allows
them to be changed every rebuild, to avoid the chance that a commonly
attacked bad address passes.)

Seth

---

From: der Mouse <[email-masked]>
Date: Fri, 24 Dec 2004 15:44:56 -0500 (EST)

>> That doesn't address scalability, it just shifts the knee of the
>> curve much like [sic] a faster CPU or more memory might.

It certainly was so when I implemented such a thing.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Seth Breidbart <[email-masked]>
Date: Thu, 30 Dec 2004 21:54:08 -0500 (EST)

> makes you look intelligent? I don't understand why you bother.

No, I'm specifying the distinction because blocking Port 25 is
objected to, for what can be considered good reason.  (The customer
says he paid for Internet access, not blocked Internet access.)

> Are you trying to be helpful, or do you just like to type?

The former.  And you?

Seth

---

From: Seth Breidbart <[email-masked]>
Date: Thu, 30 Dec 2004 21:56:04 -0500 (EST)

> similar, at a system level?

As far as I know, yes.  It's a user option.

> Do you work at panix or otherwise have authoritative info on this?

No, I just have the information they provided all their users when
they created that option.  But I know I get email from IPs that are on
various blocklists, because I see that in the logs (spamassassin
reports which blocklists).

Seth

---

From: Barry Shein <[email-masked]>
Date: Thu, 30 Dec 2004 22:24:01 -0500

I'm sorry, this repartee' has exceeded its shelf expiration date.

    -b