Subject: [Asrg] Removing the incentive to spam

From: "Chui Tey" <[email-masked]>
Date: Thu, 15 May 2003 08:49:32 +1000

Unsolicited email has a high annoyance factor because they 
consume attention-time and they junk up the inbox. Some 
spam-filtering techniques generates false
positives, so the user ends up having to trawl through
their Junk folder looking for legitimate email. Any scheme
which makes the reader spend time reading the mail rewards
the spam sender.

Assuming we operate with client-side whitelists, what if 
non-white-listed email is quarantined (just left on the POP-server)
and the email client sent the user a list of quarantined messages
in a digest form? eg:

   Subject: Quarantined email
   
   Email from the following senders have been quarantined.
   You can preview the emails on the web by clicking 
   on these links or you may unquarantine the sender.

   Individually mailed messages (* indicates the number of
   computational tokens paid to receive your email)
   ---------------------------------------------------------
     ***** Sally Morten  <[email-masked]>                         
     *     E-bay         <[email-masked]> 

   Grey-listed email (email received from sites you have
   recently visited)
   ---------------------------------------------------------
     Weekly-newsletter   <[email-masked]>

   Bulk mailed messages (sent to multiple recipients, the emails
   more likely to be unsolicited advertisements appear at 
   the end of the list):
   ---------------------------------------------------------
     DeveloperWorks      <[email-masked]>
     .
     .
     John Black          <[email-masked]>
     .
     .
     .

The idea here is that if you are sending email for the
first time (to an old friend, when apply for a job, or 
a list-server notifying the user of mail-subscription) you have
to pay computational tokens to get the attention of the user.

An intelligent email client could even decide that 
<[email-masked]> shoudl be gray listed, since the user's
internet logs could be cross referenced.

The benefit of the scheme proposed are 

  o   does not require any changes to the 
      SMTP protocol as all the processing is done on the 
      client-side.

 o    the implementation could be achieved with a client-side
      proxy, leading to fast adoption.

 o    White-listed could be automated using the client's
      address book, achieving rapid training of the system.

 o    Quarantined email is left on the POP server, conserving
      a client's limited dial up bandwidth resources.

There is probably some scope for the extension of POP protocol 
to help the client identify spam emails, which may score emails
based on the probability that they are spam. 

On an aside: here's something to think about: Spam may include 
corporate information. If one operates in a large organisation, bulk
mailed corporate communications junk up the inbox as well. 
Client-side tools should make sorting out what is urgent and what 
isn't.

Chui Tey

---

From: Barry Shein <[email-masked]>
Date: Wed, 14 May 2003 22:01:25 -0400

One fly in the ointment with quarantining and similar, or two
actually, is that many people don't have infinite mailbox quotas to
store all this spam in quarantine.

And when they hit their quota the usual system response is to begin
bouncing email back to the recipient, a very negative (and not
uncommon) side-effect of spam.

The related, second, effect is for people who have limited bandwidth
to their mailbox, 56k still being quite common. Even a fairly modest
32MB mailbox (in terms of modern disk costs) can take over two hours
to download on a high-speed dial-up line (32MB @ 4KB/sec => ~8,000secs.)

That's painful, and I think the numbers I use are quite conservative
for a common profile of user, i.e., many tens of millions.

Since spam seems to be rising rapidly and, I think it's safe to say
tho I won't say I have data, spam msgs are expanding in size due to
the inclusion of graphics etc (often to by-pass spam filters!) the
outlook is bleak and worthwhile taking seriously.

Obviously one can use better tools to profile remotely etc but let's
not be too fast waving our hands making magic tools appear so proposed
solutions look better.

Spam really is an evil problem and one needs a multivariate sieve to
keep all the factors any approach needs to consider in view.

I'd hope as time goes on and the thinking matures we'd be able to
develop some sort of checklist to evaluate any proposed approach
against so it's not always oh yeah, dial-ups, damn, that won't work
very well at all for dial-ups (or whatever.)

-- 
        -Barry Shein

Software Tool & Die    | [email-masked]           | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202        | Login: 617-739-WRLD
The World              | Public Access Internet     | Since 1989     *oo*
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Tom Bartel <[email-masked]>
Date: Wed, 14 May 2003 21:59:56 -0600

> to download on a high-speed dial-up line (32MB @ 4KB/sec => ~8,000secs.)

I wonder if any ISP or email hosting providers could provide some
anonymous datapoint on the average inbox / pop file size for users over
a larger sample.

I am an email packrat (I suspect there are others like me) and one of my
webmail accounts currently has 1347 messages totaling 36.9MB in storage
space.  The messages go back several months.  I would guess this is
potentially the upper end of the bell curve, but don't have a clue. 
Again, maybe someone on the list possibly has some data points for this.
 
> outlook is bleak and worthwhile taking seriously.

I'm not sure how the inclusion of graphics increases the storage size of
mail.  It seems many spam messages now commonly use images to display
the "text" portion of their message, but the images are references to
external servers with fully qualified links.  Unless spammers are
encoding and embedding graphics in the message using multipart/related
MIME, I don't think HTML email adds to storage size.  It may infact
reduce storage size.

> keep all the factors any approach needs to consider in view.

Agreed.

> very well at all for dial-ups (or whatever.)

Very much agreed.  

Tom Bartel
303.642.4104

---

From: Barry Shein <[email-masked]>
Date: Thu, 15 May 2003 15:20:57 -0400

> Again, maybe someone on the list possibly has some data points for this.

A few percent of the mailboxes on World are >50MB, there are quite a
few >100MB, some as much as 300MB. Admittedly it drops off rapidly
with less than 10% > 32MB, but that's all way above what you suspected
I think.

 > MIME, I don't think HTML email adds to storage size.  It may infact

Yes, spammers encode the images in-line as base64 MIME parts.

-- 
        -Barry Shein

---

From: mathew <[email-masked]>
Date: Thu, 15 May 2003 21:26:18 -0400

On Wednesday, May 14, 2003, at 23:59 US/Eastern, Tom Bartel wrote:
> space.  The messages go back several months.

Hmm. My e-mail is 123MB, but the online folders are only 6MB. However, 
3MB of that is filtered spam from the last 3 days.

> MIME, I don't think HTML email adds to storage size.  It may infact

Graphics used to be included from referenced external servers, but (a) 
that was too easy to filter and (b) the web sites got closed down too 
quickly, making most of the sent spam retrospectively useless. So now a 
lot of spammers send images embedded.

Then there's all the Windoze virus spam...

mathew

---

From: Kee Hinckley <[email-masked]>
Date: Fri, 16 May 2003 20:15:40 -0400

At 9:26 PM -0400 5/15/03, mathew wrote:
>Graphics used to be included from referenced external servers, but 
>(a) that was too easy to filter and (b) the web sites got closed 
>down too quickly, making most of the sent spam retrospectively 
>useless. So now a lot of spammers send images embedded.

I suspect we'll see legitimate mailers sending more included graphics 
soon as well.  MSN and Hotmail are both providing (or about to) 
provide filters to turn off automatic inclusion of external graphics. 
And the next Outlook is supposedly shipping with the option off by 
default.  None of that bodes well for content-filters.

P.S. As for mailbox size.  Mine is just short of 1GB--not including 
attachments.
-- 
Kee Hinckley
http://www.messagefire.com/          Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 16 May 2003 19:33:07 -0600 (MDT)

> From: Kee Hinckley <[email-masked]>

> ...
> ...

That does not follow on two counts.

There's no law that says that "content-filters" can't recognize
identical pictures.  Spam pictures are likely to vary less than spam
URLs pointing to pictures.  That's not to say that spam pictures won't
vary, but that the easiest variations in URLs such as %-encoding and
user:password@ are lost if you send the picture instead of a pointer
to the picture.

Such changes will cause fewer total graphics in all mail, if you count
URLs pointing to pictures as "graphics," as I think you must.  Some
people who now include completely unneeded pictures, logos, icons,
and so forth will think twice when their mail bloats by 50 or 500
KBytes.  Note that I wrote "some people" instead of "everyone." Many
people won't notice or understand, at least not a first.  If at first
there are significantly more graphics, outfits like MSN and Hotmail
will have bloating disk space reasons figure out how to let their
users know.  For example, they might make the "Size" indications in
folders harder to ignore, or they might expire big messages sooner.

Of course, this is all speculation until not only do MSN and Hotmail
provide those filters, but many people use them. 

Note also that a filter that turns off included graphics is as easy as
a filter that turns off external graphics.

The biggest thing MSN, AOL, and Hotmail could do against spam is turn
off HTML by default.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Kee Hinckley <[email-masked]>
Date: Sat, 17 May 2003 02:42:41 -0400

>identical pictures.  Spam pictures are likely to vary less than spam

When I say "content-filtering" I'm talking about the stuff that 
happens on a solo desktop that may or may not be connected to the 
internet.  Any type of checksum comparison involves a centralized 
service.  I would normally put that in a different category.  (Time 
for me to go reread the last taxonomy of solutions document.) You're 
right.  That type of filtering may do well with  pictures.  Of course 
you may have to deal with random fuzz there too....

>URLs pointing to pictures as "graphics," as I think you must.  Some

I don't.  Because the conversation was about the size of email on the 
mail server.

>The biggest thing MSN, AOL, and Hotmail could do against spam is turn

Turning off email completely would work as well.  Outside of the 
technical community almost all the email I see is HTML.  I don't 
think it's going to happen--whether or not it should.
-- 
Kee Hinckley
http://www.messagefire.com/          Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: Vernon Schryver <[email-masked]>
Date: Sat, 17 May 2003 10:37:34 -0600 (MDT)

> think it's going to happen--whether or not it should.

Yes, but if AOL/Netscape and Microsoft would simply switch to a reasonable
default, the HTML mail problem would be fixed.  They could still send
HTML when the sending user asked for any sort of formatting that actually
requires some sort of mark-up language.  That would eliminate almost
all legitimate HTML mail within a single AOL/Microsoft update cycle,
and that would let many users turn off incoming HTML mail.

Of course, there is as much chance of that as Microsoft fixing the
years old bug that will cause some people to think that this messages
ends before it really does.
See http://www.google.com/search?q=microsoft+begin+uuencode

begin  but
You should read WORKAROUND" in  http://support.microsoft.com/?kbid=265230 
because you shouldn't believe me if I quoted it:

    To workaround this problem: 

      Do not start messages with the word "begin" followed by two spaces.
      Use only one space between the word "begin" and the following data.
      Capitalize the word "begin" so that it is reads "Begin."
      Use a different word such as "start" or "commence.