Subject: "Friends of IMAP" dinner in Stockholm on IETF Friday
From: Barry Leiba <[email-masked]>
To: [email-masked], IMAP Extensions WG <[email-masked]>, [email-masked]
Date: Wed, 29 Jul 2009 08:33:51 -0400
List-Id: <ietf-imapext.imc.org>

We'll be gathering for the traditional "Friends of IMAP" dinner, after
the IETF meeting, in Stockholm.  Because the only time we can get most
of the people together is Friday, we'll do it on Friday evening, 31
July.  We'll meet in the entrance lobby of the Clarion Sign Hotel at
19:00.

No plan, yet, about where to do.

Barry

Date: Tue, 21 Jul 2009 15:49:11 -0600 (MDT)
From: Lyndon Nerenberg <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: A survey of Spam-related keyword usage by IMAP clients.
Organization: The Frobozz Magic Homing Pigeon Company
Sender: [email-masked]

The response to the keyword survey has been ... underwhelming. Surely 
there is more than one client represented on the mailing lists?!?

Once again, the information we're looking for is described below. Please 
forward this to every client writer you know and ask them to respond. 
We're never going to solve this problem if these people don't start 
participating.

Please send responses directly to me.

>> It's time to get a current view of how IMAP clients are using keywords
>> to manage spam. If you are a client author I would appreciate it if you
>> could take a moment to send me a note describing which keywords your
>> client uses for spam-related processing, and what each of the keywords
>> signifies.

> I'd also really like to know if the client has "this is spam" or "this 
> is not spam" buttons in the UI, and if it does how those buttons 
> interact with the use of keywords.

> I ask this because users of any client that has such buttons are clearly 
> going to use those buttons preferentially over any other labelling 
> mechanism. And if a significant number of clients have such buttons that 
> do a particular thing - regardless of what that thing is - that's pretty 
> much going to be determinative in finding the most easily deployed 
> approach.

>> If your client does not use keywords for this purpose, that's
>> valuable information, too, as is a breakdown of functionality by
>> software version number if there are significant differences. (If you
>> use non-keyword facilities to manage spam feel free to include a short
>> description of your methods if you think the info might be relevant.)
>
> Yes please. I'm especially interested in how many clients do this by 
> resending the message to a special address.
>
>> I'm primarily interested in getting authoritative data from the client
>> authors, but others should feel free to chime in with deployment-based
>> experience of related keyword usage, but please indicate the source of
>> your comments (e.g. deployment experience, source code examination).

Date: Wed, 08 Jul 2009 22:50:22 -0400
From: Cyrus Daboo <[email-masked]>
To: Ned Freed <[email-masked]>, George Michaelson <[email-masked]>
cc: [email-masked], general discussion of application-layer protocols <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Hi Ned,

--On July 8, 2009 5:30:06 PM -0700 Ned Freed <[email-masked]> wrote:

> I note, however, that now that CALDAV is complete we see Apple,
> Thunderbird, and gmail all either supporting it or working on supporting
> it.

For a more comprehensive list of implementations see: 
<http://caldav.calconnect.org/implementations.html>.

> It's a little much to expect Outlook to shift from it's currrent
> proprietary way of doing all this, but even Microsoft may eventually have
> no choice but to support the emerging standards in this area.

In the meantime the folks at <http://zideone.com> have a plugin for Outlook 
that does CalDAV and CardDAV.

-- 
Cyrus Daboo

Date: Wed, 08 Jul 2009 17:30:06 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: George Michaelson <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> >>
> >
> > The place where I'd like to see white and black listing done is in
> > CARDDAV, not
> > IMAP.
> >
> > 				Ned

> so there's only one real question: if it was standardized, would the
> likes of google implement it? How about the client side?

This is actually one of the reasons why CARDDAV is a better choice to piggyback
on.

RIght now proprietary environments like Exchange/Outlook and Lotus Notes
have a significant advantage over standards-based solutions: They provide
a fully integrated mail/calendar/addressbook client experience.

One reason standards-based solutions have a problem with this has been the lack
of open protocols for accessing and manipulating calendar and addressbook
information.

But this is changing. CALDAV addresses the calendar access issue, CARDDAV does
the same for address books. Once all the pieces are in place, and assuming we
get the details right, this will allow standards-based clients and servers all
to interoperate.

Uptake isn't going to be fast because these are complex protocols and
getting the integration right isn't easy. But for once the incentives
appear to be woring for us, not against us.

> because, unless its in gmail AND in the Apple Mail client,
> Thunderbird, Outlook, its not useful.

That may be true now, but at some point given sufficient adoption things flip
around to  where not having support for this stuff makes your client or server
not useful.

I note, however, that now that CALDAV is complete we see Apple, Thunderbird,
and gmail all either supporting it or working on supporting it. It's a little
much to expect Outlook to shift from it's currrent proprietary way of doing
all this, but even Microsoft may eventually have no choice but to support
the emerging standards in this area.

				Ned

Cc: [email-masked], general discussion of application-layer protocols <[email-masked]>
From: George Michaelson <[email-masked]>
To: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Thu, 9 Jul 2009 08:27:39 +1000
Sender: [email-masked]

>>
>
> The place where I'd like to see white and black listing done is in  
> CARDDAV, not
> IMAP.
>
> 				Ned

so there's only one real question: if it was standardized, would the  
likes of google implement it? How about the client side?

because, unless its in gmail AND in the Apple Mail client,  
Thunderbird, Outlook, its not useful.

-G

Date: Wed, 08 Jul 2009 11:54:05 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Alexey Melnikov <[email-masked]>
Cc: George Michaelson <[email-masked]>, Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> Ned Freed wrote:

> >> I have no desire to argue against my own proposal! sorry if I
> >> misunderstood the tenor of some comments to this suggestion.
> >
> >> The original posting asked about two things
> >
> >> 1) spam/ham
> >
> >> 2) white/black
> >
> >> so far, people seem to be focussing on 1). I acknowledge problems with
> >> 1), I appreciate that people are trying to address 1) as a possibility.
> >
> >> I asked about 2). I asked you if, ignoring 1), it was possible or
> >> plausible to send white/black information about sender, if 1) was hard
> >> or intractable.
> >
> > Of course it's possible, but this has no business being in IMAP. IMAP
> > is a
> > message access protocol, not an address handling protocol. It is also
> > already
> > quite complex, and extending its function to include what amountss to a
> > specialized address book capability is a really bad idea IMO.
> >
> > In fact a pretty good case can be made that it's inappropriate to use
> > IMAP for spam/ham classification, but at the spam/ham classification is
> > intrinsicly tied to messages in the store. The same cannot be said for
> > address white and blacklisting.

> +1

> >> Does that make more sense? I do realize white/black listing is a
> >> different PROBLEM to spam/ham marking, but neither of them currently
> >> have support directly in imap, and I imagine both of them would use
> >> IMAP extensions to pass information client-server.
> >
> > The place where I'd like to see white and black listing done is in
> > CARDDAV, not
> > IMAP.

> Or LDAP :-).

That's actually how we do it now. We're hoping to move to CARDDAV in the
future.

> There is also a Sieve extension for making use of whitelists/blacklists
> during mail delivery. See draft-melnikov-sieve-external-lists-02.txt.
> It can work with CARDDAV, LDAP and other things.

Interestingly, this extension in its current form is not compatible with
at least one of the proposals that was made here. The issue is one I've
raised before - the fact that :list is separate argument and not a match type.

Where there is clearly some utility in being able to say stuff along the
lines of:

    header :contains :list "subject" "tag:dirty-word-list"

The problem is that in order for this to work the list has to be enumerable.
Not all lists are enumerable, and even some that are are so large even though 
oyu can enumerate them in theory you can't afford to do so in practice.

One use case where this matters is when the list is a set of hashed values. The
way you find if something is "on the list" is to hash it and see if that value
appears. And even if the input string is short enough that you can enumerate
all the unique substrings, how about :matches and :regex? Good luck with
those.

Hashed lists have in fact been proposed in the present discussion as a means of
avoiding giving your address whitelist to the mail server. I happen not to
think this is a useful thing to do for a variety of reasons, mostly having to
do with address canonicalization (or lack thereof), but there are other
use cases where hashed lists make more sense.

So, although it reduces functionality, I believe :list should be a match type
and the underlying comparison type that's done should be a property of the list
itself.

				Ned

Date: Wed, 08 Jul 2009 11:08:28 +0100
From: Alexey Melnikov <[email-masked]>
To: George Michaelson <[email-masked]>
CC: Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Ned Freed wrote:

>> I have no desire to argue against my own proposal! sorry if I
>> misunderstood the tenor of some comments to this suggestion.
>
>> The original posting asked about two things
>
>> 1) spam/ham
>
>> 2) white/black
>
>> so far, people seem to be focussing on 1). I acknowledge problems with
>> 1), I appreciate that people are trying to address 1) as a possibility.
>
>> I asked about 2). I asked you if, ignoring 1), it was possible or
>> plausible to send white/black information about sender, if 1) was hard
>> or intractable.
>
> Of course it's possible, but this has no business being in IMAP. IMAP 
> is a
> message access protocol, not an address handling protocol. It is also 
> already
> quite complex, and extending its function to include what amountss to a
> specialized address book capability is a really bad idea IMO.
>
> In fact a pretty good case can be made that it's inappropriate to use 
> IMAP for spam/ham classification, but at the spam/ham classification is
> intrinsicly tied to messages in the store. The same cannot be said for
> address white and blacklisting.

+1

>> Does that make more sense? I do realize white/black listing is a
>> different PROBLEM to spam/ham marking, but neither of them currently
>> have support directly in imap, and I imagine both of them would use
>> IMAP extensions to pass information client-server.
>
> The place where I'd like to see white and black listing done is in 
> CARDDAV, not
> IMAP.

Or LDAP :-).

There is also a Sieve extension for making use of whitelists/blacklists 
during mail delivery. See draft-melnikov-sieve-external-lists-02.txt.
It can work with CARDDAV, LDAP and other things.

Date: Tue, 07 Jul 2009 23:19:32 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Iljitsch van Beijnum <[email-masked]>
Cc: Ned Freed <[email-masked]>, George Michaelson <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> On 8 jul 2009, at 5:02, Ned Freed wrote:

> > The place where I'd like to see white and black listing done is in
> > CARDDAV, not
> > IMAP.

> I never felt comfortable uploading all my contact info to Google. (May
> be somewhat strange since they do have all the mail sent to one of my
> email addresses.)

That's fine, but this doesn't translate into a need to put an addressbook
capability in IMAP.

> So I think white/blacklists shouldn't be complete contact info, but
> rather, just lists of hashes of email addresses.

So use CARDDAV to only upload the stuff that's needed to Google. Problem
solved.

I'll also note that addressbook programs have had facilities for exporting
subsets of the information they contain pretty much from the start. So doing
this  is hardly entering new territory.

And most users want an authoritative address book with all their contact
information in one place. I know I do - it's much too confusing otherwise.

				Ned

Cc: George Michaelson <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
To: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Wed, 8 Jul 2009 08:08:05 +0200
Sender: [email-masked]

On 8 jul 2009, at 5:02, Ned Freed wrote:

> The place where I'd like to see white and black listing done is in  
> CARDDAV, not
> IMAP.

I never felt comfortable uploading all my contact info to Google. (May  
be somewhat strange since they do have all the mail sent to one of my  
email addresses.)

So I think white/blacklists shouldn't be complete contact info, but  
rather, just lists of hashes of email addresses.

Date: Tue, 07 Jul 2009 20:50:40 -0700 (PDT)
From: Anil SRIVASTAVA <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Ned Freed <[email-masked]>
Cc: George Michaelson <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Reply-to: [email-masked]
Organization: Sun Microsystems [http://www.sun.com]
Accept-Language: English; en
Sender: [email-masked]

On 2009-07-07/20:02 [-0700], [email-masked] [Ned Freed] wrote:

> 
> > I have no desire to argue against my own proposal! sorry if I
> > misunderstood the tenor of some comments to this suggestion.
> 
> > The original posting asked about two things
> 
> > 1) spam/ham
> 
> > 2) white/black
> 
> > so far, people seem to be focussing on 1). I acknowledge problems with
> > 1), I appreciate that people are trying to address 1) as a possibility.
> 
> > I asked about 2). I asked you if, ignoring 1), it was possible or
> > plausible to send white/black information about sender, if 1) was hard
> > or intractable.
> 
> Of course it's possible, but this has no business being in IMAP. IMAP is 
> a message access protocol, not an address handling protocol. It is also 
> already quite complex, and extending its function to include what 
> amountss to a specialized address book capability is a really bad idea 
> IMO.
> 
> In fact a pretty good case can be made that it's inappropriate to 
> useIMAP for spam/ham classification, but at the spam/ham classification 
> is intrinsicly tied to messages in the store. The same cannot be said 
> for address white and blacklisting.
> 
> > Does that make more sense? I do realize white/black listing is a 
> > different PROBLEM to spam/ham marking, but neither of them currently 
> > have support directly in imap, and I imagine both of them would use 
> > IMAP extensions to pass information client-server.
> 
> The place where I'd like to see white and black listing done is in 
> CARDDAV, not IMAP.
>

+1 

Anil
 
> 				Ned
> 

_______________
Anil SRIVASTAVA  |  [email-masked]  | Sun Microsystems, Inc

Date: Tue, 07 Jul 2009 20:02:51 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: George Michaelson <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> I have no desire to argue against my own proposal! sorry if I
> misunderstood the tenor of some comments to this suggestion.

> The original posting asked about two things

> 1) spam/ham

> 2) white/black

> so far, people seem to be focussing on 1). I acknowledge problems with
> 1), I appreciate that people are trying to address 1) as a possibility.

> I asked about 2). I asked you if, ignoring 1), it was possible or
> plausible to send white/black information about sender, if 1) was hard
> or intractable.

Of course it's possible, but this has no business being in IMAP. IMAP is a
message access protocol, not an address handling protocol. It is also already
quite complex, and extending its function to include what amountss to a
specialized address book capability is a really bad idea IMO.

In fact a pretty good case can be made that it's inappropriate to use 
IMAP for spam/ham classification, but at the spam/ham classification is
intrinsicly tied to messages in the store. The same cannot be said for
address white and blacklisting.

> Does that make more sense? I do realize white/black listing is a
> different PROBLEM to spam/ham marking, but neither of them currently
> have support directly in imap, and I imagine both of them would use
> IMAP extensions to pass information client-server.

The place where I'd like to see white and black listing done is in CARDDAV, not
IMAP.

				Ned

Date: Tue, 07 Jul 2009 14:32:14 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: A survey of Spam-related keyword usage by IMAP clients.
To: Lyndon Nerenberg <[email-masked]>
Cc: [email-masked], [email-masked]
Sender: [email-masked]

I was about to post something very much like this but you beat me to it.

> It's time to get a current view of how IMAP clients are using keywords
> to manage spam. If you are a client author I would appreciate it if you
> could take a moment to send me a note describing which keywords your
> client uses for spam-related processing, and what each of the keywords
> signifies.

I'd also really like to know if the client has "this is spam" or "this is not
spam" buttons in the UI, and if it does how those buttons interact with the use
of keywords.

I ask this because users of any client that has such buttons are clearly going
to use those buttons preferentially over any other labelling mechanism. And if
a significant number of clients have such buttons that do a particular thing -
regardless of what that thing is - that's pretty much going to be determinative
in finding the most easily deployed approach.

> If your client does not use keywords for this purpose, that's
> valuable information, too, as is a breakdown of functionality by
> software version number if there are significant differences. (If you
> use non-keyword facilities to manage spam feel free to include a short
> description of your methods if you think the info might be relevant.)

Yes please. I'm especially interested in how many clients do this by resending
the message to a special address.

> I'm primarily interested in getting authoritative data from the client
> authors, but others should feel free to chime in with deployment-based
> experience of related keyword usage, but please indicate the source of
> your comments (e.g. deployment experience, source code examination).

> I will summarize the responses to the imap-ext list in a couple of weeks
> (or sooner if the response rate dictates.)

Only other thing I'd add is that I'd like to hear from webmail authors in
addition to standalone client authors.

				Ned

Subject: A survey of Spam-related keyword usage by IMAP clients.
From: Lyndon Nerenberg <[email-masked]>
Reply-To: [email-masked]
To: [email-masked], [email-masked]
Date: Tue, 07 Jul 2009 15:18:27 -0600
Sender: [email-masked]

It's time to get a current view of how IMAP clients are using keywords
to manage spam. If you are a client author I would appreciate it if you
could take a moment to send me a note describing which keywords your
client uses for spam-related processing, and what each of the keywords
signifies. If your client does not use keywords for this purpose, that's
valuable information, too, as is a breakdown of functionality by
software version number if there are significant differences. (If you
use non-keyword facilities to manage spam feel free to include a short
description of your methods if you think the info might be relevant.)

I'm primarily interested in getting authoritative data from the client
authors, but others should feel free to chime in with deployment-based
experience of related keyword usage, but please indicate the source of
your comments (e.g. deployment experience, source code examination).

I will summarize the responses to the imap-ext list in a couple of weeks
(or sooner if the response rate dictates.)

--lyndon

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
From: Lyndon Nerenberg <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
Cc: [email-masked], [email-masked]
Date: Tue, 07 Jul 2009 11:31:13 -0600
Sender: [email-masked]

On Tue, 2009-07-07 at 09:21 +0200, Iljitsch van Beijnum wrote:
> 3. manually flagged as spam
> 4. manually flagged as non-spam
> 
> Where 1 and 2 / 3 and 4 are mutually exclusive but 1 + 4 and 2 + 3  
> should trigger retraining of baysian filters etc.

The keywords I'm describing are intended to tell the server to process
the flagged message through its filters (i.e. train on the message). It
isn't relevant to the proposal whether a human sets them, just what the
server's behaviour is in their presence.

--lyndon

Date: Tue, 07 Jul 2009 08:43:46 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Cyrus Daboo <[email-masked]>
Cc: Ned Freed <[email-masked]>, Lyndon Nerenberg <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> Hi Ned,

> --On July 7, 2009 7:22:42 AM -0700 Ned Freed <[email-masked]> wrote:

> > Now suppose I implement this with folder annotations. I'm in business
> > immediately. Things just work no matter what client people use.

> Actually you need to consider what the user experience is likely to be on
> the client side. There are several things to consider: for spam messages
> users will typically want to delete them from their regular mailbox. With
> ham messages they will typically want to leave the message in place (or
> file them away after processing). In each case I certainly would not want
> the spam messages or an extra copy of the ham messages to reside on my
> server consuming quota or other resources, or being indexed as part of a
> server-wide search etc.

> With that in mind lets look at how each solution might work client-side:

> 1) Keywords: for a spam message the user sets the spam keyword, then sets
> the deleted flag (and then may explicitly expunge if the client has that as
> an option as opposed to hide-deleted-messages with lazy expunge). For a ham
> message the user just sets the ham keyword. i.e. two user operations for
> spam, one for ham.

> 2) Mailbox (I am going to assume that the client has both a move and copy
> command available - the former implicitly setting the delete flag after the
> IMAP COPY command): for a spam message the user moves the message to the
> spam mailbox. For a ham message the user copies the message to the ham
> mailbox. NB both of these assume that the server does not actually store
> the spam/ham messages in the spam/ham mailboxes (or perhaps it garbage
> collects them).

> Things to note: typically clients don't offer shortcuts "move to mailbox X"
> or "copy to mailbox X" so the mailbox operations will involve some drag and
> drop or menu list selection operation. The same may be true for setting
> arbitrary keywords (though Mail.app and I suspect Thunderbird do have
> buttons for this). Frankly the best user experience would be for the client
> to have spam/ham buttons and commands that would allow the user to do the
> appropriate actions in one click or keyboard shortcut operation - that is
> certainly important if the level of spam is moderate to high.

The presence of spam/not-spam buttons in existing clients is an important
factor I hadn't really considered, but I think it has implications that go a
lot further than you let on here.

Simply put, if enough popular clients already have such buttons, and those
buttons do different things in different clients, I think we're pretty much
screwed. Given the presence of such buttons you're not going to be able to get
users to do something different to deal with spam, whether it's adding a
keyword or annotation or performing a folder move.

Our only recourse with such clients in the mix is to write a specification for
what such buttons should do in terms of signalling the server and then hope
that clients actually adopt whatever that specification says. And frankly, if
there's some degree of commonality in what those buttons do in existing clients
and it can be made to work, we probably should adopt that approach no matter
what it happens to be.

> So at the end of the day, for the best (and frankly simplest) user
> experience clients do have to change for both the keyword and move to
> mailbox approaches. Now for keywords that change does have to be immediate
> if arbitrary keyword support is not already present in the client, whereas
> for move to mailbox, or clients already supporting keywords, that change
> can be done over time - though as Ned points out client changes tend to
> happen on geological time-scales!

I agree completely that a keyword-based approach has the potential to offer
a better user experience. The question is whether the possibility of offering
a better user experience at some distant point down the road is worth giving
up on a more immediately deployable solution.

But as I indicated above, now that you've brought up the important point of
existing clients having spam classification buttons, that's the real gorilla in
the room we have to deal with.

> So other approaches to consider:

> 1) An extension with commands SPAM and HAM to explicitly tag one or more
> messages.

> 2) A "forward to spam/ham" option that specifies special email addresses
> for the user where spam/ham can be forwarded to trigger training.

I believe this is what the buttons I've seen on at least a couple of
clients already do. If that's indeed the case, it's worth considering.

> 3) An IMAP URL handed off to the SPAM system with URLAUTH to allow that to
> pull details of the message for training. (This could be tied to (2) in
> that forward-without-download could be used in (2) with URLAUTH etc).

> These of course all require new client/server behaviors and thus are
> probably non-starters, but they should not be dismissed entirely.

I'm actually not sure (2) does with all clients, but I agree regarding (1) and
(3).

				Ned

Date: Tue, 07 Jul 2009 11:15:04 -0400
From: Cyrus Daboo <[email-masked]>
To: Ned Freed <[email-masked]>, Lyndon Nerenberg <[email-masked]>
cc: [email-masked], [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Hi Ned,

--On July 7, 2009 7:22:42 AM -0700 Ned Freed <[email-masked]> wrote:

> Now suppose I implement this with folder annotations. I'm in business
> immediately. Things just work no matter what client people use.

Actually you need to consider what the user experience is likely to be on 
the client side. There are several things to consider: for spam messages 
users will typically want to delete them from their regular mailbox. With 
ham messages they will typically want to leave the message in place (or 
file them away after processing). In each case I certainly would not want 
the spam messages or an extra copy of the ham messages to reside on my 
server consuming quota or other resources, or being indexed as part of a 
server-wide search etc.

With that in mind lets look at how each solution might work client-side:

1) Keywords: for a spam message the user sets the spam keyword, then sets 
the deleted flag (and then may explicitly expunge if the client has that as 
an option as opposed to hide-deleted-messages with lazy expunge). For a ham 
message the user just sets the ham keyword. i.e. two user operations for 
spam, one for ham.

2) Mailbox (I am going to assume that the client has both a move and copy 
command available - the former implicitly setting the delete flag after the 
IMAP COPY command): for a spam message the user moves the message to the 
spam mailbox. For a ham message the user copies the message to the ham 
mailbox. NB both of these assume that the server does not actually store 
the spam/ham messages in the spam/ham mailboxes (or perhaps it garbage 
collects them).

Things to note: typically clients don't offer shortcuts "move to mailbox X" 
or "copy to mailbox X" so the mailbox operations will involve some drag and 
drop or menu list selection operation. The same may be true for setting 
arbitrary keywords (though Mail.app and I suspect Thunderbird do have 
buttons for this). Frankly the best user experience would be for the client 
to have spam/ham buttons and commands that would allow the user to do the 
appropriate actions in one click or keyboard shortcut operation - that is 
certainly important if the level of spam is moderate to high.

So at the end of the day, for the best (and frankly simplest) user 
experience clients do have to change for both the keyword and move to 
mailbox approaches. Now for keywords that change does have to be immediate 
if arbitrary keyword support is not already present in the client, whereas 
for move to mailbox, or clients already supporting keywords, that change 
can be done over time - though as Ned points out client changes tend to 
happen on geological time-scales!

So other approaches to consider:

1) An extension with commands SPAM and HAM to explicitly tag one or more 
messages.
2) A "forward to spam/ham" option that specifies special email addresses 
for the user where spam/ham can be forwarded to trigger training.
3) An IMAP URL handed off to the SPAM system with URLAUTH to allow that to 
pull details of the message for training. (This could be tied to (2) in 
that forward-without-download could be used in (2) with URLAUTH etc).

These of course all require new client/server behaviors and thus are 
probably non-starters, but they should not be dismissed entirely.

-- 
Cyrus Daboo

Date: Tue, 07 Jul 2009 07:42:42 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: George Michaelson <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> hang on.

> The MS has the message. It presumably has some held state of it,
> specific to its implementation.

Actually, that's highly unlikely. There is some amounts of state IMAP requires
that's associated with the message - flags/keywords, annotations, some remnants
of the envelope. But beyond that there's little point in keeping a bunch of
implementation-specific state. Why? Because there's no way to expose such state
through the IMAP protocol. So when messages are FETCHed the state stays behind.
And more to the point, when a client moves a message, implementation-specific
state tends to get lost, because the client doesn't know how to access it in
order to move it.

The result is that in most csaes IMAP servers don't store much beyond the
things the standard requires them to store. The differences between servers lie
in how the standard stuff is stored and to what extent metainformation, like
body structured, is cached instead of being recomputed.

> I don't want deep knowledge of that
> model, I really just want to be able to override it with my explicit
> knowledge.

> I simply want to pass the toggle:
	
> 	If you class this as spam, please class as ham
> or
> 	if you class this as ham, please class as spam

This is exactly what we're trying to accomplish. The question is what mechanism
to use to do it.

> what it means, semantically, is of course MS specific. I don't care
> how its held, or what the effect on the weighting is, in the re-class
> as spam. In the class as ham, I care quite a lot that its sender is
> removed from hard blocks of course.

I'm afraid you cannot count on that happening either. The problem is that
such blocks are often outside the control of the service provider to change.
In many cases the provider has subscribed to some sort of spam filtering
service.

Whitelisting may be an alternative in such cases, but you have to be careful
lest the amount of whitelisting information grow to the point where it is
unmanageable. (I've seen this happen.)

> Yes, I see that moving this state between servers requires the entire
> message flow. But, if we forget that for now, and just consider that I
> have some mail in my local client cache, which I feel is now worthy of
> a state change, why do I have to pass full-state baysian model
> information?

You appear to be arguing against a straw man of your own construction here,
because AFAIK nobody has suggested such a thing.

> I just want to tell an MS "change your own flags to match
> my flags" in the hard sense: yes-or-no, not some qualitative weighting
> model.

The only "weight" people have mentioned in any of this is storing a spam score
rather than a yes/no indicator. I've already responded that while there is a
place for such information in spam handling, it makes no sense in this
particular context.

> How it acts, is up to it. I don't see why I need a complex flag
> model to say what I think, as the consumer of the mail.

Again, you appear to be arguing against a proposal nobody has made. (Perhaps
some proposal was made in an earlier thread not on the imapext list.) I'll also
point out that flags are binary by definition in IMAP - you cannot implement
even a single-valued spam score using them. If you want to store a reviseable
per-message score, you have no choice but to use message-level annotations.

> If I moved off SPAM/HAM to white/black on sender, would that be easier
> for you to consider?

Whitelisting and blacklisting is actually  a fairly different problem.

				Ned

Date: Tue, 07 Jul 2009 07:22:42 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Lyndon Nerenberg <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> On Mon, 2009-07-06 at 11:00 -0700, Ned Freed wrote:
> > If there's a reason multiple folders can't be marked this way I sure
> > don't
> > know what it is.

> You're right -- I didn't parse what you were saying correctly the first
> time around.

> But if your concern is adoption rates, a keyword-only solution will get
> wider take-up than something that requires support for annotations. (And
> from a purely engineering standpoint, the simplest solution wins.)

My concern isn't adoption rate. Rather, my concern is, as someone who provides
server software, how I can offer a better user experience in regards to spam.

Suppose I implement spam filter retraining based on setting of flags. how well
this works, or whether it works at all, depends entirely on the client people
use. And past experience on client uptake of new features like this is best
described as "lousy".

Now suppose I implement this with folder annotations. I'm in business
immediately. Things just work no matter what client people use.

There is, of course, the issue of setting the annotation on the proper folder
and  having a client to do that. But this isn't as much of a problem as it
sounds. Since the annotation only has to be set once, it can be done through a
simple web page, or better still, make it an option in the webmail interface.
(Everyone has a webmail interface these days.) And in many cases even this is
unnecessary, because many service providers predesignate a folder as the spam
folder.

The only way I could get behind a flag-based solution is if there really was an
common set of ad-hoc flag choices already in use by all the major clients - not
a couple, all. But I don't believe that's the case.

We've had to choose between client and server side solutions many times in the
past, and in almost every case the conclusion has been server side solutions
are a deployment win. I really wish this weren't true - I wish clients would
pick up new capabilities in a more timely fashion - but that's just not how
things work.

				Ned

Date: Tue, 07 Jul 2009 06:54:49 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Iljitsch van Beijnum <[email-masked]>
Cc: Lyndon Nerenberg <[email-masked]>, Ned Freed <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> On 6 jul 2009, at 19:53, Lyndon Nerenberg wrote:

> > A message's 'state of spamminess' is an attribute of the
> > message, properly denoted by keywords that stick to (and with) the
> > message.

> That's nice in the abstract, but why do we have servers classify
> messages as spam to begin with? In order to not have to look at them,
> or to be able to look at them when it suits us to clean out the spam
> rather than in between mail that we actually wanted to receive.

Actually, the primary reason servers classify messages as spam is so they
can avoid having to deal with them. If the overhead could be ignored no
server operator would bother - spam filteringh is expensive and difficult.
The reason it's done is the alternate of receiving and dealing with the
those spam messages, with all the ramificaations, is even more expensive
and difficult.

This more than anything is why a score and not a yes/no verdict is important. A
score lets you do stuff like reject messages clearly found to be spam at the
SMTP level, while letting likely but less certain to be spam messages through
to specially designated spam folders or whatever.

> So moving spam to a separate folder is not something that happens
> because of lack of a better mechanism, but the thing that we want to
> happen 98% of the time anyway.

Sometimes yes, sometimes no.

> > $spam and $nospam

> There are more states than that:

> 1. automatically flagged as spam
> 2. automatically flagged as non-spam
> 3. manually flagged as spam
> 4. manually flagged as non-spam

> Where 1 and 2 / 3 and 4 are mutually exclusive but 1 + 4 and 2 + 3
> should trigger retraining of baysian filters etc.

It's not the flag state of the message that triggers retraining,  it's
the change of flag state (or folder locaation). Your proposal is unnecessatily
complicated.

				Ned

Cc: Ned Freed <[email-masked]>, [email-masked], [email-masked]
From: Iljitsch van Beijnum <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Tue, 7 Jul 2009 09:21:11 +0200
Sender: [email-masked]

On 6 jul 2009, at 19:53, Lyndon Nerenberg wrote:

> A message's 'state of spamminess' is an attribute of the
> message, properly denoted by keywords that stick to (and with) the
> message.

That's nice in the abstract, but why do we have servers classify  
messages as spam to begin with? In order to not have to look at them,  
or to be able to look at them when it suits us to clean out the spam  
rather than in between mail that we actually wanted to receive.

So moving spam to a separate folder is not something that happens  
because of lack of a better mechanism, but the thing that we want to  
happen 98% of the time anyway.

> $spam and $nospam

There are more states than that:

1. automatically flagged as spam
2. automatically flagged as non-spam
3. manually flagged as spam
4. manually flagged as non-spam

Where 1 and 2 / 3 and 4 are mutually exclusive but 1 + 4 and 2 + 3  
should trigger retraining of baysian filters etc.

Date: Mon, 06 Jul 2009 17:42:58 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: George Michaelson <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> >
> >
> > This is why I think folder annotations are a better bet. You mark
> > the spam
> > folder and use that marking to attach meaning to the movement of
> > messages into
> > and out of that folder. It's not perfect, but it has the key
> > advantage that it
> > works with any client (OK, any client that can use folders) once the
> > annotation
> > is in place.
> >
> > 				Ned

> I have two IMAP servers, one google, one not.

That's a problem in and of itself, because last time I looked gmail is rather
limited in its support of IMAP.

> how do I synchronize state between them.

By "state" you appear to be referring to spam filtering state. If so, the short
answer is you can't possibly do it.

> After all, If I (and I do)
> sometimes have to send from one, to reply to mails from the other
> (mail clients are bad at selecting sender account) it would be useful
> to have at least white and blacklist sender synchronization between
> them.

Sorry, that's simply not realistic. The number of different spam filtering
mechanisms in use is much too large, and the lack of semantic equivalents
across them all makes this an intractable problem.

> Or, I could be sharing explicit spam tagging information to
> improve BOTH of their tracking. Do I really have to pass the entire
> messages between imap servers?

If you want to train the server that doesn't have the message that you do,
or don't regard, it as spam, moving the entire message is the only option.

I will note that moving messages from one IMAP server to another without
having to download and upload them would be possible by relaxing the
relative URL restriction in the CATENATE extension defined in RFC 4469.

But sending misclassified messages to multiple servers is not going to get them
synchronized. Again, there are a myriad of approaches to spam filtering with
hugely different semantics. In fact it is entirely possible that you'll cause
their behavior to diverge instead of converging.

> An explicit extension/flag/keyword would be unquestionably more
> efficient use of the network.

This makes no sense at all. Setting a flag associated with a message stored
on one server is not going to be visible to the other in any way. So the
flagging approach offers no solution to the multiple server problem at all.

The folder annotation approach doesn't work any better, of course.

> And, removing the mails from the spam folder in your methodology
> implies saying 'its not spam' -so the server I move them FROM thinks
> they've been un-tagged.

No, not really. Since there's no facility in IMAP to move a message between
servers, the way you have to do it is to FETCH the message from one server and
APPEND it to the other. In constrast moving a message from one folder to
another on a single server is done with a COPY/SETFLAG sequence. Entirely
different.

> Again, I say what I said to Ijllitch: I don't like depending on side-
> effects. I think explicit messaging is better.

I don't especially like depending on side effects either, but when the choice
is between a mechanism that works with the vast array of currently deployed
clients and one that requires deployment of new client software everywhere to
stand even a chance of wortking, I'm strongly inclined to pursue the former.

Again, had we standardized flags for marking messages as spam 10 years ago, I'd
be fully in favor of using them for this additional purpose because there would
be plenty of deployed client support by now. But we didn't, there isn't, and I
don't hold out a lot of hope for _anything_ deploying with reasonable speed in
the IMAP client space right now.

				Ned

Date: Mon, 6 Jul 2009 15:31:43 -0500
From: Nicolas Williams <[email-masked]>
To: Tony Finch <[email-masked]>
Cc: Michael Kirkham <[email-masked]>, [email-masked], Ned Freed <[email-masked]>, George Michaelson <[email-masked]>, [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

On Mon, Jul 06, 2009 at 09:25:32PM +0100, Tony Finch wrote:
> On Mon, 6 Jul 2009, Michael Kirkham wrote:
> > On Sun, 5 Jul 2009, Ned Freed wrote:
> >
> > > Please explain how moving a message to a random, arbtirarily named folder
> > > tells the server anything useful.
> >
> > Seems to me this is up to the server configuration and software running on the
> > server.  I don't see why you need a specific contract between IMAP client and
> > server for flagging spam
> 
> So that clients can have user interfaces that expose the functionality
> in a more friendly manner.

More specifically: so that users and their clients don't need to know
specific details of the server's configuration.

Date: Mon, 06 Jul 2009 22:06:43 +0100
From: Alexey Melnikov <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Iljitsch van Beijnum wrote:
 [...]

> About folders for spam: Apple's Mail lets the user select "use this  
> folder for... {sent, drafts, junk, trash}, which is nice if you used 
> a  different client before that uses a different folder naming scheme 
> for  these things. Works well enough although a little more 
> automation  wouldn't hurt.
>
> The point that working with folders can be done with existing 
> software  (I later train spamassassin with what Mail put in the "junk" 
> folder,  even though SA doesn't know about IMAP) is an important one. 
> Requiring  keywords that are hardcoded in clients and servers means 
> deployment  will be extremely slow, if it ever happens.

I don't think your argument in favour of folders (mailboxes) is valid. 
An IMAP server don't need to treat $Junk/$NotJunk any specially, it just 
needs to be able to store arbitrary IMAP keywords. Most servers already do.
A recent Thunderbird can also be configured to set arbitrary IMAP 
keywords. (It also recognized $Junk/$NonJunk natively, but this is 
besides my point).

Date: Mon, 6 Jul 2009 22:56:19 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Cc: Lyndon Nerenberg <[email-masked]>, Ned Freed <[email-masked]>, [email-masked]
Sender: [email-masked]

Lyndon Nerenberg writes:
> But if your concern is adoption rates, a keyword-only solution will=20
> get wider take-up than something that requires support for=20
> annotations.

Given that two significant clients (say two of the top five) use the=20
keyword approach today, I think it's safe to say =ABhas received wider=20
take-up=BB, not =ABwill get wider=BB.

Arnt

Date: Mon, 6 Jul 2009 21:25:32 +0100
From: Tony Finch <[email-masked]>
To: Michael Kirkham <[email-masked]>
cc: Ned Freed <[email-masked]>, [email-masked], George Michaelson <[email-masked]>, [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

On Mon, 6 Jul 2009, Michael Kirkham wrote:
> On Sun, 5 Jul 2009, Ned Freed wrote:
>
> > Please explain how moving a message to a random, arbtirarily named folder
> > tells the server anything useful.
>
> Seems to me this is up to the server configuration and software running on the
> server.  I don't see why you need a specific contract between IMAP client and
> server for flagging spam

So that clients can have user interfaces that expose the functionality
in a more friendly manner.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Mon, 06 Jul 2009 21:16:30 +0100
From: Dave Cridland <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>, <[email-masked]>, general discussion of application-layer protocols <[email-masked]>, Ned Freed <[email-masked]>
Sender: [email-masked]

On Mon Jul  6 18:53:41 2009, Lyndon Nerenberg wrote:
> On Sun, 2009-07-05 at 17:40 -0700, Ned Freed wrote:
> > The obvious way to make this work in IMAP is with a standardized
> > folder
> > annotation saying "messages put in this folder were flagged as  
> spam".
> > Then
> > moves of messages to the folder with this annotation provide the
> > server with
> > the necessary information that this message is considered to be  
> spam
> > by the
> > user. And by the same token, moving a message from this folder  
> can be
> > construed
> > as the message not being considered spam after all.
> 
> This is just trying to turn a folder into a keyword, but without  
> full
> keyword semantics.

+1.

It's fundamentally no different from clients trying to replace the  
\Deleted keyword with a Trash mailbox, and has all the related  
issues, which we presumably all know. (Or else can find the attempt  
at documenting them I once tried).

This is compounded by the state being a tristate rather than binary -  
messages are "Ham", or "Spam", or else "The user has not expressed  
any decision". Should I move all my messages into another folder to  
mark them as good ham?

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: Michael Kirkham <[email-masked]>, [email-masked], George Michaelson <[email-masked]>, [email-masked]
From: Iljitsch van Beijnum <[email-masked]>
To: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Mon, 6 Jul 2009 21:55:39 +0200
Sender: [email-masked]

On 6 jul 2009, at 20:07, Ned Freed wrote:

>> it was pretty painless for me to set up cron jobs to suck in
>> whatever messages I save to certain folders (that could have been  
>> named
>> anything) for training my spam filter.

> Are you serious? Cron jobs? Do you have any idea how minute a  
> fraction of the
> users of email even know what a cron job is, let alone know how to  
> set one up?

Hey, everyone needs a hobby.

I have my procmail set up to check whether I have an IMAP session with  
the server. If so, I let the client do the mail filtering, if not, the  
server does it. That way, the unread counters for folders get updated  
when the client is connected, but the client doesn't have to move  
large volume mailing list traffic to the correct folder, costing me  
expensive wireless data traffic and/or slowing things down.

About folders for spam: Apple's Mail lets the user select "use this  
folder for... {sent, drafts, junk, trash}, which is nice if you used a  
different client before that uses a different folder naming scheme for  
these things. Works well enough although a little more automation  
wouldn't hurt.

The point that working with folders can be done with existing software  
(I later train spamassassin with what Mail put in the "junk" folder,  
even though SA doesn't know about IMAP) is an important one. Requiring  
keywords that are hardcoded in clients and servers means deployment  
will be extremely slow, if it ever happens.

Date: Mon, 06 Jul 2009 11:35:17 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Lyndon Nerenberg <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked]
Sender: [email-masked]

> On Mon, 2009-07-06 at 10:39 -0700, Ned Freed wrote:
> > The problem with using keywords for this is that you are now
> > dependence on your
> > client supporting that particular set of keywords and their associated
> > semantics. Had we standardize such a keyword set 10 or so years ago
> > this
> > approach miight have had a chance, but the client base is now too
> > large and
> > too difficult to update.

> Is it really too late? We're making an request to process a particular
> message through the site's "learning" filters, i.e. this is explicit
> request to have the server take action on the message. I don't that
> behaviour is implied by any of the current schemes floating around.

Well, you have two choices: Try to extend an existing ad-hoc keyword
usage scheme that has some deployment but doesn't have quite these semaantics
currently, or start from scratch with a fresh keyword set that has zero
deployment and is unlikely to displace any of the ad-hoc schemes.

This choice is why I view using keywords as at a minimum suboptimal.

> I don't see how it could hurt to standardize a pair of new keywords (as
> I suggested in my previous message) with this behaviour documented. It
> lets the user request specific identified behaviour from the server; the
> existing keywords are all private switches for client behaviour, in my
> experience, so the new keyword behaviour would be complimentary to
> what's already out there.

I on't think it would hurt either, the question is whether it will actually
accomplish something.

				Ned

Date: Mon, 06 Jul 2009 11:07:18 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Michael Kirkham <[email-masked]>
Cc: Ned Freed <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, [email-masked], George Michaelson <[email-masked]>, [email-masked]
Sender: [email-masked]

> On Sun, 5 Jul 2009, Ned Freed wrote:

> > Please explain how moving a message to a random, arbtirarily named
> > folder tells the server anything useful.

> Seems to me this is up to the server configuration and software running on
> the server.  I don't see why you need a specific contract between IMAP
> client and server for flagging spam,

The primary reason is so that the server can use the information to
automaticallly refine it's filtering policies on a per-user basis.

Other uses include, but are not limited to, setting expiration and arhival
policies.

> and it's of limited usefulness.

On the contrary, such a mechanism would be HUGELY useful. We have any number of
customers begging for it, as a matter of fact.

> For
> one thing, it was pretty painless for me to set up cron jobs to suck in
> whatever messages I save to certain folders (that could have been named
> anything) for training my spam filter.

Are you serious? Cron jobs? Do you have any idea how minute a fraction of the
users of email even know what a cron job is, let alone know how to set one up?

The minute you start talking about setting up your own cron jobs, it  pretty
much goes without saying that you're operating entirely outside the space where
99+% of the use of IMAP takes place.

> I also have other cron jobs set up
> to do other things with email completely unrelated to spam that also
> simply involve saving to a particular folder (e.g. a tickler file for
> email--I move messages to folders named for a month or day, and cron jobs
> automatiacally reinsert them back in my inbox later).  That kind of thing
> isn't addressed by having something spam-specific, but as far as my email
> client is concerned, they're flagged the same way.

And no doubt you also tweak SpamAssasin rules, set up milters, etc. etc. This
is far beyond the capability of most users.

				Ned

Date: Mon, 06 Jul 2009 19:29:20 +0100
From: Alexey Melnikov <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
CC: Ned Freed <[email-masked]>, [email-masked], [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Lyndon Nerenberg wrote:

>But if your concern is adoption rates, a keyword-only solution will get
>wider take-up than something that requires support for annotations. (And
>from a purely engineering standpoint, the simplest solution wins.)
>  
>
+1.

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
From: Lyndon Nerenberg <[email-masked]>
To: Ned Freed <[email-masked]>
Cc: [email-masked], [email-masked]
Date: Mon, 06 Jul 2009 12:26:01 -0600
Sender: [email-masked]

On Mon, 2009-07-06 at 11:00 -0700, Ned Freed wrote:
> If there's a reason multiple folders can't be marked this way I sure
> don't
> know what it is.

You're right -- I didn't parse what you were saying correctly the first
time around.

But if your concern is adoption rates, a keyword-only solution will get
wider take-up than something that requires support for annotations. (And
from a purely engineering standpoint, the simplest solution wins.)

--lyndon

Date: Mon, 06 Jul 2009 19:14:38 +0100
From: Alexey Melnikov <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
CC: general discussion of application-layer protocols <[email-masked]>, [email-masked], Ned Freed <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, George Michaelson <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> Ned Freed writes:
>
>> The obvious way to make this work in IMAP is with a standardized 
>> folder annotation saying "messages put in this folder were flagged as 
>> spam".
>
> I think what Thunderbird and Apple Mail do is to set specific message 
> flags on messages in whatever folders they happen to be in: $Junk if 
> the user said a particular message is spam, $autojunk if tb 
> autonomously decided that message is spam, $nonjunk and $autononjunk 
> for the opposite. (I may be wrong about the exact flag names.)

Something like that, yes.

> This was specified in some expired draft,

Which was mine.

> which IMO now has been frozen by deployment. Apple Mail and 
> Thunderbird add up to >10% for me. Time for an informational RFC, 
> perhaps. "Widely-used IMAP flag names"?

Yes. I think even more important to create the IANA registry for keywords.
Do you want to help finish my draft?

Date: Mon, 06 Jul 2009 11:00:24 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Lyndon Nerenberg <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> On Sun, 2009-07-05 at 17:40 -0700, Ned Freed wrote:
> > The obvious way to make this work in IMAP is with a standardized
> > folder
> > annotation saying "messages put in this folder were flagged as spam".
> > Then
> > moves of messages to the folder with this annotation provide the
> > server with
> > the necessary information that this message is considered to be spam
> > by the
> > user. And by the same token, moving a message from this folder can be
> > construed
> > as the message not being considered spam after all.

> This is just trying to turn a folder into a keyword, but without full
> keyword semantics.

Actually, it's a hack t work around the lack of client support, which is why
I didn't propose a keyword-based mechanism.

> The most obvious flaw is that it constrains the
> client to storing all the spam in one bucket.

If there's a reason multiple folders can't be marked this way I sure don't
know what it is.

> It doesn't allow for (say)
> sorting junk by type into different folders (perhaps on different
> servers). A message's 'state of spamminess' is an attribute of the
> message, properly denoted by keywords that stick to (and with) the
> message.

I see no reason why this couldn't be done, if you really want to.

> If you want this functionality (and I think it's a useful service for
> servers to provide) you need to implement two keywords: $spam and
> $nospam (these are just example names). Setting $spam on a message would
> instruct the server to pass the message off to the filtering component
> as spam; setting $nospam would instruct the server to pass the message
> off to the filtering component as valid (non-spam). The two keywords are
> exclusive, and it would be up to the server to decide what action to
> take if the client tries to light up both at once (e.g. unset the other
> flag and take the corresponding action, reject the command, or silently
> ignore the request).

Again, the problem with using keywords is now you have to get a significant
fraction of clients on board before it's useful. I'd be willing to go along
with a keyword-based approach if I could convinced that's going to happen, but
everything recentlts has supported exactly the opposite conclusion.

				Ned

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
From: Lyndon Nerenberg <[email-masked]>
To: Ned Freed <[email-masked]>
Cc: [email-masked]
Date: Mon, 06 Jul 2009 12:02:40 -0600
Sender: [email-masked]

On Mon, 2009-07-06 at 10:39 -0700, Ned Freed wrote:
> The problem with using keywords for this is that you are now
> dependence on your
> client supporting that particular set of keywords and their associated
> semantics. Had we standardize such a keyword set 10 or so years ago
> this
> approach miight have had a chance, but the client base is now too
> large and
> too difficult to update.

Is it really too late? We're making an request to process a particular
message through the site's "learning" filters, i.e. this is explicit
request to have the server take action on the message. I don't that
behaviour is implied by any of the current schemes floating around.

I don't see how it could hurt to standardize a pair of new keywords (as
I suggested in my previous message) with this behaviour documented. It
lets the user request specific identified behaviour from the server; the
existing keywords are all private switches for client behaviour, in my
experience, so the new keyword behaviour would be complimentary to
what's already out there.

--lyndon

Date: Mon, 6 Jul 2009 12:44:14 -0500
From: Nicolas Williams <[email-masked]>
To: Ned Freed <[email-masked]>
Cc: Iljitsch van Beijnum <[email-masked]>, [email-masked], George Michaelson <[email-masked]>, [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

On Sun, Jul 05, 2009 at 05:40:31PM -0700, Ned Freed wrote:
> 
> >On 3 jul 2009, at 1:05, George Michaelson wrote:
> 
> >> As an example, at the moment if I wish to inform google that I have
> >> known spam in local folders, I have to go to the web interface and
> >> manually tag. If there was an IMAP extension, I could review my
> >> local baysian junk folder, remove all non-spam (and flag the senders
> >> as white-listed if need be), and request the rest to be flagged as
> >> spam back on the IMAP backed MS.
> 
> >Doesn't moving the spam messages to the spam folder accomplish this
> >already? That's what my client does.
> 
> Please explain how moving a message to a random, arbtirarily named
> folder tells the server anything useful. People use all sorts of
> different naming schemes for folders, and it's common for clients to
> provide lots of flexibility in this area.

In particular the word 'spam' is really an English-language word.  L10N
would demand that the name of this folder not be 'spam' in all locales,
or at least that clients know how to map that folder name to the user's
language(s).  But even if we go for the latter, having special folder
names means denying or complicating the use of those names in other
languages for other purposes; it's bad enough we have one well-known
folder name (INBOX), let's have no more.

> The obvious way to make this work in IMAP is with a standardized
> folder annotation saying "messages put in this folder were flagged as
> spam". Then moves of messages to the folder with this annotation
> provide the server with the necessary information that this message is
> considered to be spam by the user. And by the same token, moving a
> message from this folder can be construed as the message not being
> considered spam after all.

+1

Or even just a message flag/annotation.  I understand the "move junk
mail to the junk mail folder" metaphor for flagging junk mail as such,
but one might want to flag junk mail and still move it to an "archived
mail" folder.  I don't find my argument for a per-message flag terribly
convincing, so I won't push it.

> It isn't perfect, but as long as this is taken as an indicator, not as
> an absolute statement, it should get the job done.
> 
> >But if you want this, I'd say that it needs to be a fractional thing,
> >not a binary spam/no spam indication. For instance, the server could
> >give something a spam score of 2 and the client also 2 and together
> >that would be 4 so the message is presumed to be spam (assuming the
> >spam threshold is 3), but in a binary system no spam OR no spam = no
> >spam.
> 
> There's a place for fractional scoring in spam filtering, but this isn't
> it. A user doesn't think "this message is 90% likely to be spam", they
> either think it's spam or it isn't.

+1

Fractional scoring by the server, or by software running on the client
can be useful, but the users themselves will generally make binary spam/
not-spam decisions.

Nico
-- 

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
From: Lyndon Nerenberg <[email-masked]>
To: Ned Freed <[email-masked]>
Cc: [email-masked], [email-masked]
Date: Mon, 06 Jul 2009 11:53:41 -0600
Sender: [email-masked]

On Sun, 2009-07-05 at 17:40 -0700, Ned Freed wrote:
> The obvious way to make this work in IMAP is with a standardized
> folder
> annotation saying "messages put in this folder were flagged as spam".
> Then
> moves of messages to the folder with this annotation provide the
> server with
> the necessary information that this message is considered to be spam
> by the
> user. And by the same token, moving a message from this folder can be
> construed
> as the message not being considered spam after all.

This is just trying to turn a folder into a keyword, but without full
keyword semantics. The most obvious flaw is that it constrains the
client to storing all the spam in one bucket. It doesn't allow for (say)
sorting junk by type into different folders (perhaps on different
servers). A message's 'state of spamminess' is an attribute of the
message, properly denoted by keywords that stick to (and with) the
message.

If you want this functionality (and I think it's a useful service for
servers to provide) you need to implement two keywords: $spam and
$nospam (these are just example names). Setting $spam on a message would
instruct the server to pass the message off to the filtering component
as spam; setting $nospam would instruct the server to pass the message
off to the filtering component as valid (non-spam). The two keywords are
exclusive, and it would be up to the server to decide what action to
take if the client tries to light up both at once (e.g. unset the other
flag and take the corresponding action, reject the command, or silently
ignore the request).

Date: Mon, 6 Jul 2009 19:50:08 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: general discussion of application-layer protocols <[email-masked]>, [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Cc: Ned Freed <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, George Michaelson <[email-masked]>
Sender: [email-masked]

Ned Freed writes:
> The obvious way to make this work in IMAP is with a standardized 
> folder annotation saying "messages put in this folder were flagged as 
> spam".

I think what Thunderbird and Apple Mail do is to set specific message 
flags on messages in whatever folders they happen to be in: $Junk if 
the user said a particular message is spam, $autojunk if tb 
autonomously decided that message is spam, $nonjunk and $autononjunk 
for the opposite. (I may be wrong about the exact flag names.)

This was specified in some expired draft, which IMO now has been frozen 
by deployment. Apple Mail and Thunderbird add up to >10% for me. Time 
for an informational RFC, perhaps. "Widely-used IMAP flag names"?

Arnt

Date: Mon, 06 Jul 2009 10:39:40 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Alexey Melnikov <[email-masked]>
Cc: Iljitsch van Beijnum <[email-masked]>, Dave Cridland <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>, George Michaelson <[email-masked]>
Sender: [email-masked]

> Dave Cridland wrote:

> > On Sun Jul  5 20:03:30 2009, Iljitsch van Beijnum wrote:
> >
> >> On 3 jul 2009, at 1:05, George Michaelson wrote:
> >>
> >>> As an example, at the moment if I wish to inform google that I
> >>> have  known spam in local folders, I have to go to the web
> >>> interface and  manually tag. If there was an IMAP extension, I
> >>> could review my  local baysian junk folder, remove all non-spam
> >>> (and flag the senders  as white-listed if need be), and request  the
> >>> rest to be flagged as  spam back on the IMAP backed MS.
> >>
> >> Doesn't moving the spam messages to the spam folder accomplish this
> >>  already? That's what my client does.
> >
> > There's also a proposal (expired?) to use keywords to signal  spamminess.

> To expand on this: I believe the proposal is implemented by several
> major email clients (Thunderbird is one of them).

> [For people not familiar with IMAP: IMAP keywords is a basic IMAP
> feature implemented by majority of IMAP servers I've seen.]

The problem with using keywords for this is that you are now dependence on your
client supporting that particular set of keywords and their associated
semantics. Had we standardize such a keyword set 10 or so years ago this
approach miight have had a chance, but the client base is now too large and
too difficult to update.

This is why I think folder annotations are a better bet. You mark the spam
folder and use that marking to attach meaning to the movement of messages into
and out of that folder. It's not perfect, but it has the key advantage that it
works with any client (OK, any client that can use folders) once the annotation
is in place.

				Ned

Date: Sun, 05 Jul 2009 17:40:31 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Iljitsch van Beijnum <[email-masked]>
Cc: George Michaelson <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> On 3 jul 2009, at 1:05, George Michaelson wrote:

> > As an example, at the moment if I wish to inform google that I have
> > known spam in local folders, I have to go to the web interface and
> > manually tag. If there was an IMAP extension, I could review my
> > local baysian junk folder, remove all non-spam (and flag the senders
> > as white-listed if need be), and request the rest to be flagged as
> > spam back on the IMAP backed MS.

> Doesn't moving the spam messages to the spam folder accomplish this
> already? That's what my client does.

Please explain how moving a message to a random, arbtirarily named folder tells
the server anything useful. People use all sorts of different naming schemes
for folders, and it's common for clients to provide lots of flexibility in this
area.

The obvious way to make this work in IMAP is with a standardized folder
annotation saying "messages put in this folder were flagged as spam". Then
moves of messages to the folder with this annotation provide the server with
the necessary information that this message is considered to be spam by the
user. And by the same token, moving a message from this folder can be construed
as the message not being considered spam after all.

It isn't perfect, but as long as this is taken as an indicator, not as an
absolute statement, it should get the job done.

> But if you want this, I'd say that it needs to be a fractional thing,
> not a binary spam/no spam indication. For instance, the server could
> give something a spam score of 2 and the client also 2 and together
> that would be 4 so the message is presumed to be spam (assuming the
> spam threshold is 3), but in a binary system no spam OR no spam = no
> spam.

There's a place for fractional scoring in spam filtering, but this isn't
it. A user doesn't think "this message is 90% likely to be spam", they
either think it's spam or it isn't.

				Ned

Date: Mon, 06 Jul 2009 11:27:45 +0100
From: Alexey Melnikov <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
CC: Dave Cridland <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>, George Michaelson <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Dave Cridland wrote:

> On Sun Jul  5 20:03:30 2009, Iljitsch van Beijnum wrote:
>
>> On 3 jul 2009, at 1:05, George Michaelson wrote:
>>
>>> As an example, at the moment if I wish to inform google that I  
>>> have  known spam in local folders, I have to go to the web  
>>> interface and  manually tag. If there was an IMAP extension, I  
>>> could review my  local baysian junk folder, remove all non-spam  
>>> (and flag the senders  as white-listed if need be), and request  the 
>>> rest to be flagged as  spam back on the IMAP backed MS.
>>
>> Doesn't moving the spam messages to the spam folder accomplish this  
>>  already? That's what my client does.
>
> There's also a proposal (expired?) to use keywords to signal  spamminess.

To expand on this: I believe the proposal is implemented by several 
major email clients (Thunderbird is one of them).

[For people not familiar with IMAP: IMAP keywords is a basic IMAP 
feature implemented by majority of IMAP servers I've seen.]

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Mon, 06 Jul 2009 10:05:41 +0100
From: Dave Cridland <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>, <[email-masked]>, general discussion of application-layer protocols <[email-masked]>, George Michaelson <[email-masked]>
Sender: [email-masked]

On Mon Jul  6 09:37:06 2009, Iljitsch van Beijnum wrote:
> =

> On 6 jul 2009, at 10:24, Dave Cridland wrote:
> =

>>> But if you want this, I'd say that it needs to be a fractional   =

>>> thing,  not a binary spam/no spam indication. For instance, the   =

>>> server could  give something a spam score of 2 and the client  =

>>> also  2 and together  that would be 4 so the message is presumed  =

>>> to be  spam (assuming the  spam threshold is 3), but in a binary  =

>>> system no  spam OR no spam =3D no  spam.
> =

>> The notion of using two spam filters in concert to attempt to make  =

>>  an overall decision is basically flawed, because it fits into one  =

>> or  other case above - if you do go to the effort of having a  =

>> range  based spamminess from one, you'd have to use it as mere  =

>> input into  the other's decision process, since combining them  =

>> na=EFvely would  produce poorly weighted results.
> =

> Right. The filtering would have to be complimentary. So perhaps  =

> there  should be an indication of the type of filtering done, too.  =

> The reason  to combine server and client side approaches is that  =

> servers can  easily do blacklists, and running any type of  =

> filtering on the server  gives it the opportunity to reject the  =

> message during the SMTP  session, creating a good error message for  =

> real senders but no useless  bounces to innocent third parties in  =

> the case of spoofed senders. On  the client is useful because the  =

> client can typically better afford to  run CPU intensive stuff like  =

> baysian filters and the client probably  has a better list of  =

> previous correspondents that it can use to  whitelist.

Whitelists should, in principle, be easily comminicable to the server  =

(and easily processed).

Bayesian filters on the client are only really useful if the same  =

client is always used - this is often not the case (and, in my  =

experience, is becoming rarer). Otherwise the quality of your spam  =

filtering radically changes depending on the client you happen to be  =

using at the time.

And basically, any filtering at all that can be done on the server  =

saves a vast amount of bandwidth, and makes use of email on smaller  =

devices (like mobile phones) instantly more useful.

The one massive advantage that a client has is direct contact with  =

the user, and the eyeball remains the best spam filtering technology  =

we have.

I'm not suggesting that clients must not do spam filtering -  =

obviously they do, and it's often useful - but I think we should aim  =

to make server-side filtering the best we can. A very simple uniform  =

feedback mechanism of whether the user thinks a message is spam or  =

not makes this very much easier.

Dave.
-- =

Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: <[email-masked]>, general discussion of application-layer protocols <[email-masked]>, George Michaelson <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
To: Dave Cridland <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Mon, 6 Jul 2009 10:37:06 +0200
Sender: [email-masked]

On 6 jul 2009, at 10:24, Dave Cridland wrote:

>> But if you want this, I'd say that it needs to be a fractional =20
>> thing,  not a binary spam/no spam indication. For instance, the =20
>> server could  give something a spam score of 2 and the client also =20=

>> 2 and together  that would be 4 so the message is presumed to be =20
>> spam (assuming the  spam threshold is 3), but in a binary system no =20=

>> spam OR no spam =3D no  spam.

> The notion of using two spam filters in concert to attempt to make =20
> an overall decision is basically flawed, because it fits into one or =20=

> other case above - if you do go to the effort of having a range =20
> based spamminess from one, you'd have to use it as mere input into =20
> the other's decision process, since combining them na=EFvely would =20
> produce poorly weighted results.

Right. The filtering would have to be complimentary. So perhaps there =20=

should be an indication of the type of filtering done, too. The reason =20=

to combine server and client side approaches is that servers can =20
easily do blacklists, and running any type of filtering on the server =20=

gives it the opportunity to reject the message during the SMTP =20
session, creating a good error message for real senders but no useless =20=

bounces to innocent third parties in the case of spoofed senders. On =20
the client is useful because the client can typically better afford to =20=

run CPU intensive stuff like baysian filters and the client probably =20
has a better list of previous correspondents that it can use to =20
whitelist.=

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Mon, 06 Jul 2009 09:24:42 +0100
From: Dave Cridland <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>, <[email-masked]>, general discussion of application-layer protocols <[email-masked]>, George Michaelson <[email-masked]>
Sender: [email-masked]

On Sun Jul  5 20:03:30 2009, Iljitsch van Beijnum wrote:
> On 3 jul 2009, at 1:05, George Michaelson wrote:
> =

>> As an example, at the moment if I wish to inform google that I  =

>> have  known spam in local folders, I have to go to the web  =

>> interface and  manually tag. If there was an IMAP extension, I  =

>> could review my  local baysian junk folder, remove all non-spam  =

>> (and flag the senders  as white-listed if need be), and request  =

>> the rest to be flagged as  spam back on the IMAP backed MS.
> =

> Doesn't moving the spam messages to the spam folder accomplish this  =

>  already? That's what my client does.
> =

> =

There's also a proposal (expired?) to use keywords to signal  =

spamminess.

> But if you want this, I'd say that it needs to be a fractional  =

> thing,  not a binary spam/no spam indication. For instance, the  =

> server could  give something a spam score of 2 and the client also  =

> 2 and together  that would be 4 so the message is presumed to be  =

> spam (assuming the  spam threshold is 3), but in a binary system no  =

> spam OR no spam =3D no  spam.

Of course, if both client and server use precisely the same criteria,  =

you've simply halved your threshold.

There's two cases:

1) The server has some feedback-based spam detection mechanism, like  =

a bayesian filter. You want to teach the server's filter about your  =

explicit decisions.

2) Your client has a spam filter (or some sort), you want the server  =

to tell your client about it's preliminary findings.

The notion of using two spam filters in concert to attempt to make an  =

overall decision is basically flawed, because it fits into one or  =

other case above - if you do go to the effort of having a range based  =

spamminess from one, you'd have to use it as mere input into the  =

other's decision process, since combining them na=EFvely would produce  =

poorly weighted results.

A simple keyword approach isn't quite as good as ranges, but it does  =

have the useful property that nearly every service already offers  =

arbitrary keywords, and so is quite likely to already offer it.

Ranges, on the other hand, require the use of annotations, which is a  =

rather more complex area.

Dave.
-- =

Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: [email-masked], [email-masked]
From: Iljitsch van Beijnum <[email-masked]>
To: George Michaelson <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Sun, 5 Jul 2009 21:03:30 +0200
Sender: [email-masked]

On 3 jul 2009, at 1:05, George Michaelson wrote:

> As an example, at the moment if I wish to inform google that I have  
> known spam in local folders, I have to go to the web interface and  
> manually tag. If there was an IMAP extension, I could review my  
> local baysian junk folder, remove all non-spam (and flag the senders  
> as white-listed if need be), and request the rest to be flagged as  
> spam back on the IMAP backed MS.

Doesn't moving the spam messages to the spam folder accomplish this  
already? That's what my client does.

But if you want this, I'd say that it needs to be a fractional thing,  
not a binary spam/no spam indication. For instance, the server could  
give something a spam score of 2 and the client also 2 and together  
that would be 4 so the message is presumed to be spam (assuming the  
spam threshold is 3), but in a binary system no spam OR no spam = no  
spam.

Reply-To: [email-masked]
Date: Wed, 29 Jul 2009 08:33:51 -0400
Subject: "Friends of IMAP" dinner in Stockholm on IETF Friday
From: Barry Leiba <[email-masked]>
To: [email-masked], IMAP Extensions WG <[email-masked]>, [email-masked]
Sender: [email-masked]

We'll be gathering for the traditional "Friends of IMAP" dinner, after
the IETF meeting, in Stockholm.  Because the only time we can get most
of the people together is Friday, we'll do it on Friday evening, 31
July.  We'll meet in the entrance lobby of the Clarion Sign Hotel at
19:00.

No plan, yet, about where to do.

Barry

Date: Tue, 21 Jul 2009 15:49:11 -0600 (MDT)
From: Lyndon Nerenberg <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: A survey of Spam-related keyword usage by IMAP clients.
Organization: The Frobozz Magic Homing Pigeon Company
Sender: [email-masked]

The response to the keyword survey has been ... underwhelming. Surely 
there is more than one client represented on the mailing lists?!?

Once again, the information we're looking for is described below. Please 
forward this to every client writer you know and ask them to respond. 
We're never going to solve this problem if these people don't start 
participating.

Please send responses directly to me.

>> It's time to get a current view of how IMAP clients are using keywords
>> to manage spam. If you are a client author I would appreciate it if you
>> could take a moment to send me a note describing which keywords your
>> client uses for spam-related processing, and what each of the keywords
>> signifies.

> I'd also really like to know if the client has "this is spam" or "this 
> is not spam" buttons in the UI, and if it does how those buttons 
> interact with the use of keywords.

> I ask this because users of any client that has such buttons are clearly 
> going to use those buttons preferentially over any other labelling 
> mechanism. And if a significant number of clients have such buttons that 
> do a particular thing - regardless of what that thing is - that's pretty 
> much going to be determinative in finding the most easily deployed 
> approach.

>> If your client does not use keywords for this purpose, that's
>> valuable information, too, as is a breakdown of functionality by
>> software version number if there are significant differences. (If you
>> use non-keyword facilities to manage spam feel free to include a short
>> description of your methods if you think the info might be relevant.)
>
> Yes please. I'm especially interested in how many clients do this by 
> resending the message to a special address.
>
>> I'm primarily interested in getting authoritative data from the client
>> authors, but others should feel free to chime in with deployment-based
>> experience of related keyword usage, but please indicate the source of
>> your comments (e.g. deployment experience, source code examination).

Date: Wed, 08 Jul 2009 22:50:22 -0400
From: Cyrus Daboo <[email-masked]>
To: Ned Freed <[email-masked]>, George Michaelson <[email-masked]>
cc: [email-masked], general discussion of application-layer protocols <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Hi Ned,

--On July 8, 2009 5:30:06 PM -0700 Ned Freed <[email-masked]> wrote:

> I note, however, that now that CALDAV is complete we see Apple,
> Thunderbird, and gmail all either supporting it or working on supporting
> it.

For a more comprehensive list of implementations see: 
<http://caldav.calconnect.org/implementations.html>.

> It's a little much to expect Outlook to shift from it's currrent
> proprietary way of doing all this, but even Microsoft may eventually have
> no choice but to support the emerging standards in this area.

In the meantime the folks at <http://zideone.com> have a plugin for Outlook 
that does CalDAV and CardDAV.

-- 
Cyrus Daboo

Date: Wed, 08 Jul 2009 17:30:06 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: George Michaelson <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> >>
> >
> > The place where I'd like to see white and black listing done is in
> > CARDDAV, not
> > IMAP.
> >
> > 				Ned

> so there's only one real question: if it was standardized, would the
> likes of google implement it? How about the client side?

This is actually one of the reasons why CARDDAV is a better choice to piggyback
on.

RIght now proprietary environments like Exchange/Outlook and Lotus Notes
have a significant advantage over standards-based solutions: They provide
a fully integrated mail/calendar/addressbook client experience.

One reason standards-based solutions have a problem with this has been the lack
of open protocols for accessing and manipulating calendar and addressbook
information.

But this is changing. CALDAV addresses the calendar access issue, CARDDAV does
the same for address books. Once all the pieces are in place, and assuming we
get the details right, this will allow standards-based clients and servers all
to interoperate.

Uptake isn't going to be fast because these are complex protocols and
getting the integration right isn't easy. But for once the incentives
appear to be woring for us, not against us.

> because, unless its in gmail AND in the Apple Mail client,
> Thunderbird, Outlook, its not useful.

That may be true now, but at some point given sufficient adoption things flip
around to  where not having support for this stuff makes your client or server
not useful.

I note, however, that now that CALDAV is complete we see Apple, Thunderbird,
and gmail all either supporting it or working on supporting it. It's a little
much to expect Outlook to shift from it's currrent proprietary way of doing
all this, but even Microsoft may eventually have no choice but to support
the emerging standards in this area.

				Ned

Cc: [email-masked], general discussion of application-layer protocols <[email-masked]>
From: George Michaelson <[email-masked]>
To: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Thu, 9 Jul 2009 08:27:39 +1000
Sender: [email-masked]

>>
>
> The place where I'd like to see white and black listing done is in  
> CARDDAV, not
> IMAP.
>
> 				Ned

so there's only one real question: if it was standardized, would the  
likes of google implement it? How about the client side?

because, unless its in gmail AND in the Apple Mail client,  
Thunderbird, Outlook, its not useful.

-G

Date: Wed, 08 Jul 2009 11:54:05 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Alexey Melnikov <[email-masked]>
Cc: George Michaelson <[email-masked]>, Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> Ned Freed wrote:

> >> I have no desire to argue against my own proposal! sorry if I
> >> misunderstood the tenor of some comments to this suggestion.
> >
> >> The original posting asked about two things
> >
> >> 1) spam/ham
> >
> >> 2) white/black
> >
> >> so far, people seem to be focussing on 1). I acknowledge problems with
> >> 1), I appreciate that people are trying to address 1) as a possibility.
> >
> >> I asked about 2). I asked you if, ignoring 1), it was possible or
> >> plausible to send white/black information about sender, if 1) was hard
> >> or intractable.
> >
> > Of course it's possible, but this has no business being in IMAP. IMAP
> > is a
> > message access protocol, not an address handling protocol. It is also
> > already
> > quite complex, and extending its function to include what amountss to a
> > specialized address book capability is a really bad idea IMO.
> >
> > In fact a pretty good case can be made that it's inappropriate to use
> > IMAP for spam/ham classification, but at the spam/ham classification is
> > intrinsicly tied to messages in the store. The same cannot be said for
> > address white and blacklisting.

> +1

> >> Does that make more sense? I do realize white/black listing is a
> >> different PROBLEM to spam/ham marking, but neither of them currently
> >> have support directly in imap, and I imagine both of them would use
> >> IMAP extensions to pass information client-server.
> >
> > The place where I'd like to see white and black listing done is in
> > CARDDAV, not
> > IMAP.

> Or LDAP :-).

That's actually how we do it now. We're hoping to move to CARDDAV in the
future.

> There is also a Sieve extension for making use of whitelists/blacklists
> during mail delivery. See draft-melnikov-sieve-external-lists-02.txt.
> It can work with CARDDAV, LDAP and other things.

Interestingly, this extension in its current form is not compatible with
at least one of the proposals that was made here. The issue is one I've
raised before - the fact that :list is separate argument and not a match type.

Where there is clearly some utility in being able to say stuff along the
lines of:

    header :contains :list "subject" "tag:dirty-word-list"

The problem is that in order for this to work the list has to be enumerable.
Not all lists are enumerable, and even some that are are so large even though 
oyu can enumerate them in theory you can't afford to do so in practice.

One use case where this matters is when the list is a set of hashed values. The
way you find if something is "on the list" is to hash it and see if that value
appears. And even if the input string is short enough that you can enumerate
all the unique substrings, how about :matches and :regex? Good luck with
those.

Hashed lists have in fact been proposed in the present discussion as a means of
avoiding giving your address whitelist to the mail server. I happen not to
think this is a useful thing to do for a variety of reasons, mostly having to
do with address canonicalization (or lack thereof), but there are other
use cases where hashed lists make more sense.

So, although it reduces functionality, I believe :list should be a match type
and the underlying comparison type that's done should be a property of the list
itself.

				Ned

Date: Wed, 08 Jul 2009 11:08:28 +0100
From: Alexey Melnikov <[email-masked]>
To: George Michaelson <[email-masked]>
CC: Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Ned Freed wrote:

>> I have no desire to argue against my own proposal! sorry if I
>> misunderstood the tenor of some comments to this suggestion.
>
>> The original posting asked about two things
>
>> 1) spam/ham
>
>> 2) white/black
>
>> so far, people seem to be focussing on 1). I acknowledge problems with
>> 1), I appreciate that people are trying to address 1) as a possibility.
>
>> I asked about 2). I asked you if, ignoring 1), it was possible or
>> plausible to send white/black information about sender, if 1) was hard
>> or intractable.
>
> Of course it's possible, but this has no business being in IMAP. IMAP 
> is a
> message access protocol, not an address handling protocol. It is also 
> already
> quite complex, and extending its function to include what amountss to a
> specialized address book capability is a really bad idea IMO.
>
> In fact a pretty good case can be made that it's inappropriate to use 
> IMAP for spam/ham classification, but at the spam/ham classification is
> intrinsicly tied to messages in the store. The same cannot be said for
> address white and blacklisting.

+1

>> Does that make more sense? I do realize white/black listing is a
>> different PROBLEM to spam/ham marking, but neither of them currently
>> have support directly in imap, and I imagine both of them would use
>> IMAP extensions to pass information client-server.
>
> The place where I'd like to see white and black listing done is in 
> CARDDAV, not
> IMAP.

Or LDAP :-).

There is also a Sieve extension for making use of whitelists/blacklists 
during mail delivery. See draft-melnikov-sieve-external-lists-02.txt.
It can work with CARDDAV, LDAP and other things.

Date: Tue, 07 Jul 2009 23:19:32 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Iljitsch van Beijnum <[email-masked]>
Cc: Ned Freed <[email-masked]>, George Michaelson <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> On 8 jul 2009, at 5:02, Ned Freed wrote:

> > The place where I'd like to see white and black listing done is in
> > CARDDAV, not
> > IMAP.

> I never felt comfortable uploading all my contact info to Google. (May
> be somewhat strange since they do have all the mail sent to one of my
> email addresses.)

That's fine, but this doesn't translate into a need to put an addressbook
capability in IMAP.

> So I think white/blacklists shouldn't be complete contact info, but
> rather, just lists of hashes of email addresses.

So use CARDDAV to only upload the stuff that's needed to Google. Problem
solved.

I'll also note that addressbook programs have had facilities for exporting
subsets of the information they contain pretty much from the start. So doing
this  is hardly entering new territory.

And most users want an authoritative address book with all their contact
information in one place. I know I do - it's much too confusing otherwise.

				Ned

Cc: George Michaelson <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
To: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Wed, 8 Jul 2009 08:08:05 +0200
Sender: [email-masked]

On 8 jul 2009, at 5:02, Ned Freed wrote:

> The place where I'd like to see white and black listing done is in  
> CARDDAV, not
> IMAP.

I never felt comfortable uploading all my contact info to Google. (May  
be somewhat strange since they do have all the mail sent to one of my  
email addresses.)

So I think white/blacklists shouldn't be complete contact info, but  
rather, just lists of hashes of email addresses.

Date: Tue, 07 Jul 2009 20:50:40 -0700 (PDT)
From: Anil SRIVASTAVA <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Ned Freed <[email-masked]>
Cc: George Michaelson <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Reply-to: [email-masked]
Organization: Sun Microsystems [http://www.sun.com]
Accept-Language: English; en
Sender: [email-masked]

On 2009-07-07/20:02 [-0700], [email-masked] [Ned Freed] wrote:

> 
> > I have no desire to argue against my own proposal! sorry if I
> > misunderstood the tenor of some comments to this suggestion.
> 
> > The original posting asked about two things
> 
> > 1) spam/ham
> 
> > 2) white/black
> 
> > so far, people seem to be focussing on 1). I acknowledge problems with
> > 1), I appreciate that people are trying to address 1) as a possibility.
> 
> > I asked about 2). I asked you if, ignoring 1), it was possible or
> > plausible to send white/black information about sender, if 1) was hard
> > or intractable.
> 
> Of course it's possible, but this has no business being in IMAP. IMAP is 
> a message access protocol, not an address handling protocol. It is also 
> already quite complex, and extending its function to include what 
> amountss to a specialized address book capability is a really bad idea 
> IMO.
> 
> In fact a pretty good case can be made that it's inappropriate to 
> useIMAP for spam/ham classification, but at the spam/ham classification 
> is intrinsicly tied to messages in the store. The same cannot be said 
> for address white and blacklisting.
> 
> > Does that make more sense? I do realize white/black listing is a 
> > different PROBLEM to spam/ham marking, but neither of them currently 
> > have support directly in imap, and I imagine both of them would use 
> > IMAP extensions to pass information client-server.
> 
> The place where I'd like to see white and black listing done is in 
> CARDDAV, not IMAP.
>

+1 

Anil
 
> 				Ned
> 

_______________
Anil SRIVASTAVA  |  [email-masked]  | Sun Microsystems, Inc

Date: Tue, 07 Jul 2009 20:02:51 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: George Michaelson <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> I have no desire to argue against my own proposal! sorry if I
> misunderstood the tenor of some comments to this suggestion.

> The original posting asked about two things

> 1) spam/ham

> 2) white/black

> so far, people seem to be focussing on 1). I acknowledge problems with
> 1), I appreciate that people are trying to address 1) as a possibility.

> I asked about 2). I asked you if, ignoring 1), it was possible or
> plausible to send white/black information about sender, if 1) was hard
> or intractable.

Of course it's possible, but this has no business being in IMAP. IMAP is a
message access protocol, not an address handling protocol. It is also already
quite complex, and extending its function to include what amountss to a
specialized address book capability is a really bad idea IMO.

In fact a pretty good case can be made that it's inappropriate to use 
IMAP for spam/ham classification, but at the spam/ham classification is
intrinsicly tied to messages in the store. The same cannot be said for
address white and blacklisting.

> Does that make more sense? I do realize white/black listing is a
> different PROBLEM to spam/ham marking, but neither of them currently
> have support directly in imap, and I imagine both of them would use
> IMAP extensions to pass information client-server.

The place where I'd like to see white and black listing done is in CARDDAV, not
IMAP.

				Ned

Date: Tue, 07 Jul 2009 14:32:14 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: A survey of Spam-related keyword usage by IMAP clients.
To: Lyndon Nerenberg <[email-masked]>
Cc: [email-masked], [email-masked]
Sender: [email-masked]

I was about to post something very much like this but you beat me to it.

> It's time to get a current view of how IMAP clients are using keywords
> to manage spam. If you are a client author I would appreciate it if you
> could take a moment to send me a note describing which keywords your
> client uses for spam-related processing, and what each of the keywords
> signifies.

I'd also really like to know if the client has "this is spam" or "this is not
spam" buttons in the UI, and if it does how those buttons interact with the use
of keywords.

I ask this because users of any client that has such buttons are clearly going
to use those buttons preferentially over any other labelling mechanism. And if
a significant number of clients have such buttons that do a particular thing -
regardless of what that thing is - that's pretty much going to be determinative
in finding the most easily deployed approach.

> If your client does not use keywords for this purpose, that's
> valuable information, too, as is a breakdown of functionality by
> software version number if there are significant differences. (If you
> use non-keyword facilities to manage spam feel free to include a short
> description of your methods if you think the info might be relevant.)

Yes please. I'm especially interested in how many clients do this by resending
the message to a special address.

> I'm primarily interested in getting authoritative data from the client
> authors, but others should feel free to chime in with deployment-based
> experience of related keyword usage, but please indicate the source of
> your comments (e.g. deployment experience, source code examination).

> I will summarize the responses to the imap-ext list in a couple of weeks
> (or sooner if the response rate dictates.)

Only other thing I'd add is that I'd like to hear from webmail authors in
addition to standalone client authors.

				Ned

Subject: A survey of Spam-related keyword usage by IMAP clients.
From: Lyndon Nerenberg <[email-masked]>
Reply-To: [email-masked]
To: [email-masked], [email-masked]
Date: Tue, 07 Jul 2009 15:18:27 -0600
Sender: [email-masked]

It's time to get a current view of how IMAP clients are using keywords
to manage spam. If you are a client author I would appreciate it if you
could take a moment to send me a note describing which keywords your
client uses for spam-related processing, and what each of the keywords
signifies. If your client does not use keywords for this purpose, that's
valuable information, too, as is a breakdown of functionality by
software version number if there are significant differences. (If you
use non-keyword facilities to manage spam feel free to include a short
description of your methods if you think the info might be relevant.)

I'm primarily interested in getting authoritative data from the client
authors, but others should feel free to chime in with deployment-based
experience of related keyword usage, but please indicate the source of
your comments (e.g. deployment experience, source code examination).

I will summarize the responses to the imap-ext list in a couple of weeks
(or sooner if the response rate dictates.)

--lyndon

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
From: Lyndon Nerenberg <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
Cc: [email-masked], [email-masked]
Date: Tue, 07 Jul 2009 11:31:13 -0600
Sender: [email-masked]

On Tue, 2009-07-07 at 09:21 +0200, Iljitsch van Beijnum wrote:
> 3. manually flagged as spam
> 4. manually flagged as non-spam
> 
> Where 1 and 2 / 3 and 4 are mutually exclusive but 1 + 4 and 2 + 3  
> should trigger retraining of baysian filters etc.

The keywords I'm describing are intended to tell the server to process
the flagged message through its filters (i.e. train on the message). It
isn't relevant to the proposal whether a human sets them, just what the
server's behaviour is in their presence.

--lyndon

Date: Tue, 07 Jul 2009 08:43:46 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Cyrus Daboo <[email-masked]>
Cc: Ned Freed <[email-masked]>, Lyndon Nerenberg <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> Hi Ned,

> --On July 7, 2009 7:22:42 AM -0700 Ned Freed <[email-masked]> wrote:

> > Now suppose I implement this with folder annotations. I'm in business
> > immediately. Things just work no matter what client people use.

> Actually you need to consider what the user experience is likely to be on
> the client side. There are several things to consider: for spam messages
> users will typically want to delete them from their regular mailbox. With
> ham messages they will typically want to leave the message in place (or
> file them away after processing). In each case I certainly would not want
> the spam messages or an extra copy of the ham messages to reside on my
> server consuming quota or other resources, or being indexed as part of a
> server-wide search etc.

> With that in mind lets look at how each solution might work client-side:

> 1) Keywords: for a spam message the user sets the spam keyword, then sets
> the deleted flag (and then may explicitly expunge if the client has that as
> an option as opposed to hide-deleted-messages with lazy expunge). For a ham
> message the user just sets the ham keyword. i.e. two user operations for
> spam, one for ham.

> 2) Mailbox (I am going to assume that the client has both a move and copy
> command available - the former implicitly setting the delete flag after the
> IMAP COPY command): for a spam message the user moves the message to the
> spam mailbox. For a ham message the user copies the message to the ham
> mailbox. NB both of these assume that the server does not actually store
> the spam/ham messages in the spam/ham mailboxes (or perhaps it garbage
> collects them).

> Things to note: typically clients don't offer shortcuts "move to mailbox X"
> or "copy to mailbox X" so the mailbox operations will involve some drag and
> drop or menu list selection operation. The same may be true for setting
> arbitrary keywords (though Mail.app and I suspect Thunderbird do have
> buttons for this). Frankly the best user experience would be for the client
> to have spam/ham buttons and commands that would allow the user to do the
> appropriate actions in one click or keyboard shortcut operation - that is
> certainly important if the level of spam is moderate to high.

The presence of spam/not-spam buttons in existing clients is an important
factor I hadn't really considered, but I think it has implications that go a
lot further than you let on here.

Simply put, if enough popular clients already have such buttons, and those
buttons do different things in different clients, I think we're pretty much
screwed. Given the presence of such buttons you're not going to be able to get
users to do something different to deal with spam, whether it's adding a
keyword or annotation or performing a folder move.

Our only recourse with such clients in the mix is to write a specification for
what such buttons should do in terms of signalling the server and then hope
that clients actually adopt whatever that specification says. And frankly, if
there's some degree of commonality in what those buttons do in existing clients
and it can be made to work, we probably should adopt that approach no matter
what it happens to be.

> So at the end of the day, for the best (and frankly simplest) user
> experience clients do have to change for both the keyword and move to
> mailbox approaches. Now for keywords that change does have to be immediate
> if arbitrary keyword support is not already present in the client, whereas
> for move to mailbox, or clients already supporting keywords, that change
> can be done over time - though as Ned points out client changes tend to
> happen on geological time-scales!

I agree completely that a keyword-based approach has the potential to offer
a better user experience. The question is whether the possibility of offering
a better user experience at some distant point down the road is worth giving
up on a more immediately deployable solution.

But as I indicated above, now that you've brought up the important point of
existing clients having spam classification buttons, that's the real gorilla in
the room we have to deal with.

> So other approaches to consider:

> 1) An extension with commands SPAM and HAM to explicitly tag one or more
> messages.

> 2) A "forward to spam/ham" option that specifies special email addresses
> for the user where spam/ham can be forwarded to trigger training.

I believe this is what the buttons I've seen on at least a couple of
clients already do. If that's indeed the case, it's worth considering.

> 3) An IMAP URL handed off to the SPAM system with URLAUTH to allow that to
> pull details of the message for training. (This could be tied to (2) in
> that forward-without-download could be used in (2) with URLAUTH etc).

> These of course all require new client/server behaviors and thus are
> probably non-starters, but they should not be dismissed entirely.

I'm actually not sure (2) does with all clients, but I agree regarding (1) and
(3).

				Ned

Date: Tue, 07 Jul 2009 11:15:04 -0400
From: Cyrus Daboo <[email-masked]>
To: Ned Freed <[email-masked]>, Lyndon Nerenberg <[email-masked]>
cc: [email-masked], [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Hi Ned,

--On July 7, 2009 7:22:42 AM -0700 Ned Freed <[email-masked]> wrote:

> Now suppose I implement this with folder annotations. I'm in business
> immediately. Things just work no matter what client people use.

Actually you need to consider what the user experience is likely to be on 
the client side. There are several things to consider: for spam messages 
users will typically want to delete them from their regular mailbox. With 
ham messages they will typically want to leave the message in place (or 
file them away after processing). In each case I certainly would not want 
the spam messages or an extra copy of the ham messages to reside on my 
server consuming quota or other resources, or being indexed as part of a 
server-wide search etc.

With that in mind lets look at how each solution might work client-side:

1) Keywords: for a spam message the user sets the spam keyword, then sets 
the deleted flag (and then may explicitly expunge if the client has that as 
an option as opposed to hide-deleted-messages with lazy expunge). For a ham 
message the user just sets the ham keyword. i.e. two user operations for 
spam, one for ham.

2) Mailbox (I am going to assume that the client has both a move and copy 
command available - the former implicitly setting the delete flag after the 
IMAP COPY command): for a spam message the user moves the message to the 
spam mailbox. For a ham message the user copies the message to the ham 
mailbox. NB both of these assume that the server does not actually store 
the spam/ham messages in the spam/ham mailboxes (or perhaps it garbage 
collects them).

Things to note: typically clients don't offer shortcuts "move to mailbox X" 
or "copy to mailbox X" so the mailbox operations will involve some drag and 
drop or menu list selection operation. The same may be true for setting 
arbitrary keywords (though Mail.app and I suspect Thunderbird do have 
buttons for this). Frankly the best user experience would be for the client 
to have spam/ham buttons and commands that would allow the user to do the 
appropriate actions in one click or keyboard shortcut operation - that is 
certainly important if the level of spam is moderate to high.

So at the end of the day, for the best (and frankly simplest) user 
experience clients do have to change for both the keyword and move to 
mailbox approaches. Now for keywords that change does have to be immediate 
if arbitrary keyword support is not already present in the client, whereas 
for move to mailbox, or clients already supporting keywords, that change 
can be done over time - though as Ned points out client changes tend to 
happen on geological time-scales!

So other approaches to consider:

1) An extension with commands SPAM and HAM to explicitly tag one or more 
messages.
2) A "forward to spam/ham" option that specifies special email addresses 
for the user where spam/ham can be forwarded to trigger training.
3) An IMAP URL handed off to the SPAM system with URLAUTH to allow that to 
pull details of the message for training. (This could be tied to (2) in 
that forward-without-download could be used in (2) with URLAUTH etc).

These of course all require new client/server behaviors and thus are 
probably non-starters, but they should not be dismissed entirely.

-- 
Cyrus Daboo

Date: Tue, 07 Jul 2009 07:42:42 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: George Michaelson <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> hang on.

> The MS has the message. It presumably has some held state of it,
> specific to its implementation.

Actually, that's highly unlikely. There is some amounts of state IMAP requires
that's associated with the message - flags/keywords, annotations, some remnants
of the envelope. But beyond that there's little point in keeping a bunch of
implementation-specific state. Why? Because there's no way to expose such state
through the IMAP protocol. So when messages are FETCHed the state stays behind.
And more to the point, when a client moves a message, implementation-specific
state tends to get lost, because the client doesn't know how to access it in
order to move it.

The result is that in most csaes IMAP servers don't store much beyond the
things the standard requires them to store. The differences between servers lie
in how the standard stuff is stored and to what extent metainformation, like
body structured, is cached instead of being recomputed.

> I don't want deep knowledge of that
> model, I really just want to be able to override it with my explicit
> knowledge.

> I simply want to pass the toggle:
	
> 	If you class this as spam, please class as ham
> or
> 	if you class this as ham, please class as spam

This is exactly what we're trying to accomplish. The question is what mechanism
to use to do it.

> what it means, semantically, is of course MS specific. I don't care
> how its held, or what the effect on the weighting is, in the re-class
> as spam. In the class as ham, I care quite a lot that its sender is
> removed from hard blocks of course.

I'm afraid you cannot count on that happening either. The problem is that
such blocks are often outside the control of the service provider to change.
In many cases the provider has subscribed to some sort of spam filtering
service.

Whitelisting may be an alternative in such cases, but you have to be careful
lest the amount of whitelisting information grow to the point where it is
unmanageable. (I've seen this happen.)

> Yes, I see that moving this state between servers requires the entire
> message flow. But, if we forget that for now, and just consider that I
> have some mail in my local client cache, which I feel is now worthy of
> a state change, why do I have to pass full-state baysian model
> information?

You appear to be arguing against a straw man of your own construction here,
because AFAIK nobody has suggested such a thing.

> I just want to tell an MS "change your own flags to match
> my flags" in the hard sense: yes-or-no, not some qualitative weighting
> model.

The only "weight" people have mentioned in any of this is storing a spam score
rather than a yes/no indicator. I've already responded that while there is a
place for such information in spam handling, it makes no sense in this
particular context.

> How it acts, is up to it. I don't see why I need a complex flag
> model to say what I think, as the consumer of the mail.

Again, you appear to be arguing against a proposal nobody has made. (Perhaps
some proposal was made in an earlier thread not on the imapext list.) I'll also
point out that flags are binary by definition in IMAP - you cannot implement
even a single-valued spam score using them. If you want to store a reviseable
per-message score, you have no choice but to use message-level annotations.

> If I moved off SPAM/HAM to white/black on sender, would that be easier
> for you to consider?

Whitelisting and blacklisting is actually  a fairly different problem.

				Ned

Date: Tue, 07 Jul 2009 07:22:42 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Lyndon Nerenberg <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> On Mon, 2009-07-06 at 11:00 -0700, Ned Freed wrote:
> > If there's a reason multiple folders can't be marked this way I sure
> > don't
> > know what it is.

> You're right -- I didn't parse what you were saying correctly the first
> time around.

> But if your concern is adoption rates, a keyword-only solution will get
> wider take-up than something that requires support for annotations. (And
> from a purely engineering standpoint, the simplest solution wins.)

My concern isn't adoption rate. Rather, my concern is, as someone who provides
server software, how I can offer a better user experience in regards to spam.

Suppose I implement spam filter retraining based on setting of flags. how well
this works, or whether it works at all, depends entirely on the client people
use. And past experience on client uptake of new features like this is best
described as "lousy".

Now suppose I implement this with folder annotations. I'm in business
immediately. Things just work no matter what client people use.

There is, of course, the issue of setting the annotation on the proper folder
and  having a client to do that. But this isn't as much of a problem as it
sounds. Since the annotation only has to be set once, it can be done through a
simple web page, or better still, make it an option in the webmail interface.
(Everyone has a webmail interface these days.) And in many cases even this is
unnecessary, because many service providers predesignate a folder as the spam
folder.

The only way I could get behind a flag-based solution is if there really was an
common set of ad-hoc flag choices already in use by all the major clients - not
a couple, all. But I don't believe that's the case.

We've had to choose between client and server side solutions many times in the
past, and in almost every case the conclusion has been server side solutions
are a deployment win. I really wish this weren't true - I wish clients would
pick up new capabilities in a more timely fashion - but that's just not how
things work.

				Ned

Date: Tue, 07 Jul 2009 06:54:49 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Iljitsch van Beijnum <[email-masked]>
Cc: Lyndon Nerenberg <[email-masked]>, Ned Freed <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> On 6 jul 2009, at 19:53, Lyndon Nerenberg wrote:

> > A message's 'state of spamminess' is an attribute of the
> > message, properly denoted by keywords that stick to (and with) the
> > message.

> That's nice in the abstract, but why do we have servers classify
> messages as spam to begin with? In order to not have to look at them,
> or to be able to look at them when it suits us to clean out the spam
> rather than in between mail that we actually wanted to receive.

Actually, the primary reason servers classify messages as spam is so they
can avoid having to deal with them. If the overhead could be ignored no
server operator would bother - spam filteringh is expensive and difficult.
The reason it's done is the alternate of receiving and dealing with the
those spam messages, with all the ramificaations, is even more expensive
and difficult.

This more than anything is why a score and not a yes/no verdict is important. A
score lets you do stuff like reject messages clearly found to be spam at the
SMTP level, while letting likely but less certain to be spam messages through
to specially designated spam folders or whatever.

> So moving spam to a separate folder is not something that happens
> because of lack of a better mechanism, but the thing that we want to
> happen 98% of the time anyway.

Sometimes yes, sometimes no.

> > $spam and $nospam

> There are more states than that:

> 1. automatically flagged as spam
> 2. automatically flagged as non-spam
> 3. manually flagged as spam
> 4. manually flagged as non-spam

> Where 1 and 2 / 3 and 4 are mutually exclusive but 1 + 4 and 2 + 3
> should trigger retraining of baysian filters etc.

It's not the flag state of the message that triggers retraining,  it's
the change of flag state (or folder locaation). Your proposal is unnecessatily
complicated.

				Ned

Cc: Ned Freed <[email-masked]>, [email-masked], [email-masked]
From: Iljitsch van Beijnum <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Tue, 7 Jul 2009 09:21:11 +0200
Sender: [email-masked]

On 6 jul 2009, at 19:53, Lyndon Nerenberg wrote:

> A message's 'state of spamminess' is an attribute of the
> message, properly denoted by keywords that stick to (and with) the
> message.

That's nice in the abstract, but why do we have servers classify  
messages as spam to begin with? In order to not have to look at them,  
or to be able to look at them when it suits us to clean out the spam  
rather than in between mail that we actually wanted to receive.

So moving spam to a separate folder is not something that happens  
because of lack of a better mechanism, but the thing that we want to  
happen 98% of the time anyway.

> $spam and $nospam

There are more states than that:

1. automatically flagged as spam
2. automatically flagged as non-spam
3. manually flagged as spam
4. manually flagged as non-spam

Where 1 and 2 / 3 and 4 are mutually exclusive but 1 + 4 and 2 + 3  
should trigger retraining of baysian filters etc.

Date: Mon, 06 Jul 2009 17:42:58 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: George Michaelson <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>
Sender: [email-masked]

> >
> >
> > This is why I think folder annotations are a better bet. You mark
> > the spam
> > folder and use that marking to attach meaning to the movement of
> > messages into
> > and out of that folder. It's not perfect, but it has the key
> > advantage that it
> > works with any client (OK, any client that can use folders) once the
> > annotation
> > is in place.
> >
> > 				Ned

> I have two IMAP servers, one google, one not.

That's a problem in and of itself, because last time I looked gmail is rather
limited in its support of IMAP.

> how do I synchronize state between them.

By "state" you appear to be referring to spam filtering state. If so, the short
answer is you can't possibly do it.

> After all, If I (and I do)
> sometimes have to send from one, to reply to mails from the other
> (mail clients are bad at selecting sender account) it would be useful
> to have at least white and blacklist sender synchronization between
> them.

Sorry, that's simply not realistic. The number of different spam filtering
mechanisms in use is much too large, and the lack of semantic equivalents
across them all makes this an intractable problem.

> Or, I could be sharing explicit spam tagging information to
> improve BOTH of their tracking. Do I really have to pass the entire
> messages between imap servers?

If you want to train the server that doesn't have the message that you do,
or don't regard, it as spam, moving the entire message is the only option.

I will note that moving messages from one IMAP server to another without
having to download and upload them would be possible by relaxing the
relative URL restriction in the CATENATE extension defined in RFC 4469.

But sending misclassified messages to multiple servers is not going to get them
synchronized. Again, there are a myriad of approaches to spam filtering with
hugely different semantics. In fact it is entirely possible that you'll cause
their behavior to diverge instead of converging.

> An explicit extension/flag/keyword would be unquestionably more
> efficient use of the network.

This makes no sense at all. Setting a flag associated with a message stored
on one server is not going to be visible to the other in any way. So the
flagging approach offers no solution to the multiple server problem at all.

The folder annotation approach doesn't work any better, of course.

> And, removing the mails from the spam folder in your methodology
> implies saying 'its not spam' -so the server I move them FROM thinks
> they've been un-tagged.

No, not really. Since there's no facility in IMAP to move a message between
servers, the way you have to do it is to FETCH the message from one server and
APPEND it to the other. In constrast moving a message from one folder to
another on a single server is done with a COPY/SETFLAG sequence. Entirely
different.

> Again, I say what I said to Ijllitch: I don't like depending on side-
> effects. I think explicit messaging is better.

I don't especially like depending on side effects either, but when the choice
is between a mechanism that works with the vast array of currently deployed
clients and one that requires deployment of new client software everywhere to
stand even a chance of wortking, I'm strongly inclined to pursue the former.

Again, had we standardized flags for marking messages as spam 10 years ago, I'd
be fully in favor of using them for this additional purpose because there would
be plenty of deployed client support by now. But we didn't, there isn't, and I
don't hold out a lot of hope for _anything_ deploying with reasonable speed in
the IMAP client space right now.

				Ned

Date: Mon, 6 Jul 2009 15:31:43 -0500
From: Nicolas Williams <[email-masked]>
To: Tony Finch <[email-masked]>
Cc: Michael Kirkham <[email-masked]>, [email-masked], Ned Freed <[email-masked]>, George Michaelson <[email-masked]>, [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

On Mon, Jul 06, 2009 at 09:25:32PM +0100, Tony Finch wrote:
> On Mon, 6 Jul 2009, Michael Kirkham wrote:
> > On Sun, 5 Jul 2009, Ned Freed wrote:
> >
> > > Please explain how moving a message to a random, arbtirarily named folder
> > > tells the server anything useful.
> >
> > Seems to me this is up to the server configuration and software running on the
> > server.  I don't see why you need a specific contract between IMAP client and
> > server for flagging spam
> 
> So that clients can have user interfaces that expose the functionality
> in a more friendly manner.

More specifically: so that users and their clients don't need to know
specific details of the server's configuration.

Date: Mon, 06 Jul 2009 22:06:43 +0100
From: Alexey Melnikov <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Iljitsch van Beijnum wrote:
 [...]

> About folders for spam: Apple's Mail lets the user select "use this  
> folder for... {sent, drafts, junk, trash}, which is nice if you used 
> a  different client before that uses a different folder naming scheme 
> for  these things. Works well enough although a little more 
> automation  wouldn't hurt.
>
> The point that working with folders can be done with existing 
> software  (I later train spamassassin with what Mail put in the "junk" 
> folder,  even though SA doesn't know about IMAP) is an important one. 
> Requiring  keywords that are hardcoded in clients and servers means 
> deployment  will be extremely slow, if it ever happens.

I don't think your argument in favour of folders (mailboxes) is valid. 
An IMAP server don't need to treat $Junk/$NotJunk any specially, it just 
needs to be able to store arbitrary IMAP keywords. Most servers already do.
A recent Thunderbird can also be configured to set arbitrary IMAP 
keywords. (It also recognized $Junk/$NonJunk natively, but this is 
besides my point).

Date: Mon, 6 Jul 2009 22:56:19 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Cc: Lyndon Nerenberg <[email-masked]>, Ned Freed <[email-masked]>, [email-masked]
Sender: [email-masked]

Lyndon Nerenberg writes:
> But if your concern is adoption rates, a keyword-only solution will=20
> get wider take-up than something that requires support for=20
> annotations.

Given that two significant clients (say two of the top five) use the=20
keyword approach today, I think it's safe to say =ABhas received wider=20
take-up=BB, not =ABwill get wider=BB.

Arnt

Date: Mon, 6 Jul 2009 21:25:32 +0100
From: Tony Finch <[email-masked]>
To: Michael Kirkham <[email-masked]>
cc: Ned Freed <[email-masked]>, [email-masked], George Michaelson <[email-masked]>, [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

On Mon, 6 Jul 2009, Michael Kirkham wrote:
> On Sun, 5 Jul 2009, Ned Freed wrote:
>
> > Please explain how moving a message to a random, arbtirarily named folder
> > tells the server anything useful.
>
> Seems to me this is up to the server configuration and software running on the
> server.  I don't see why you need a specific contract between IMAP client and
> server for flagging spam

So that clients can have user interfaces that expose the functionality
in a more friendly manner.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Mon, 06 Jul 2009 21:16:30 +0100
From: Dave Cridland <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>, <[email-masked]>, general discussion of application-layer protocols <[email-masked]>, Ned Freed <[email-masked]>
Sender: [email-masked]

On Mon Jul  6 18:53:41 2009, Lyndon Nerenberg wrote:
> On Sun, 2009-07-05 at 17:40 -0700, Ned Freed wrote:
> > The obvious way to make this work in IMAP is with a standardized
> > folder
> > annotation saying "messages put in this folder were flagged as  
> spam".
> > Then
> > moves of messages to the folder with this annotation provide the
> > server with
> > the necessary information that this message is considered to be  
> spam
> > by the
> > user. And by the same token, moving a message from this folder  
> can be
> > construed
> > as the message not being considered spam after all.
> 
> This is just trying to turn a folder into a keyword, but without  
> full
> keyword semantics.

+1.

It's fundamentally no different from clients trying to replace the  
\Deleted keyword with a Trash mailbox, and has all the related  
issues, which we presumably all know. (Or else can find the attempt  
at documenting them I once tried).

This is compounded by the state being a tristate rather than binary -  
messages are "Ham", or "Spam", or else "The user has not expressed  
any decision". Should I move all my messages into another folder to  
mark them as good ham?

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: Michael Kirkham <[email-masked]>, [email-masked], George Michaelson <[email-masked]>, [email-masked]
From: Iljitsch van Beijnum <[email-masked]>
To: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Mon, 6 Jul 2009 21:55:39 +0200
Sender: [email-masked]

On 6 jul 2009, at 20:07, Ned Freed wrote:

>> it was pretty painless for me to set up cron jobs to suck in
>> whatever messages I save to certain folders (that could have been  
>> named
>> anything) for training my spam filter.

> Are you serious? Cron jobs? Do you have any idea how minute a  
> fraction of the
> users of email even know what a cron job is, let alone know how to  
> set one up?

Hey, everyone needs a hobby.

I have my procmail set up to check whether I have an IMAP session with  
the server. If so, I let the client do the mail filtering, if not, the  
server does it. That way, the unread counters for folders get updated  
when the client is connected, but the client doesn't have to move  
large volume mailing list traffic to the correct folder, costing me  
expensive wireless data traffic and/or slowing things down.

About folders for spam: Apple's Mail lets the user select "use this  
folder for... {sent, drafts, junk, trash}, which is nice if you used a  
different client before that uses a different folder naming scheme for  
these things. Works well enough although a little more automation  
wouldn't hurt.

The point that working with folders can be done with existing software  
(I later train spamassassin with what Mail put in the "junk" folder,  
even though SA doesn't know about IMAP) is an important one. Requiring  
keywords that are hardcoded in clients and servers means deployment  
will be extremely slow, if it ever happens.

Date: Mon, 06 Jul 2009 11:35:17 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Lyndon Nerenberg <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked]
Sender: [email-masked]

> On Mon, 2009-07-06 at 10:39 -0700, Ned Freed wrote:
> > The problem with using keywords for this is that you are now
> > dependence on your
> > client supporting that particular set of keywords and their associated
> > semantics. Had we standardize such a keyword set 10 or so years ago
> > this
> > approach miight have had a chance, but the client base is now too
> > large and
> > too difficult to update.

> Is it really too late? We're making an request to process a particular
> message through the site's "learning" filters, i.e. this is explicit
> request to have the server take action on the message. I don't that
> behaviour is implied by any of the current schemes floating around.

Well, you have two choices: Try to extend an existing ad-hoc keyword
usage scheme that has some deployment but doesn't have quite these semaantics
currently, or start from scratch with a fresh keyword set that has zero
deployment and is unlikely to displace any of the ad-hoc schemes.

This choice is why I view using keywords as at a minimum suboptimal.

> I don't see how it could hurt to standardize a pair of new keywords (as
> I suggested in my previous message) with this behaviour documented. It
> lets the user request specific identified behaviour from the server; the
> existing keywords are all private switches for client behaviour, in my
> experience, so the new keyword behaviour would be complimentary to
> what's already out there.

I on't think it would hurt either, the question is whether it will actually
accomplish something.

				Ned

Date: Mon, 06 Jul 2009 11:07:18 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Michael Kirkham <[email-masked]>
Cc: Ned Freed <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, [email-masked], George Michaelson <[email-masked]>, [email-masked]
Sender: [email-masked]

> On Sun, 5 Jul 2009, Ned Freed wrote:

> > Please explain how moving a message to a random, arbtirarily named
> > folder tells the server anything useful.

> Seems to me this is up to the server configuration and software running on
> the server.  I don't see why you need a specific contract between IMAP
> client and server for flagging spam,

The primary reason is so that the server can use the information to
automaticallly refine it's filtering policies on a per-user basis.

Other uses include, but are not limited to, setting expiration and arhival
policies.

> and it's of limited usefulness.

On the contrary, such a mechanism would be HUGELY useful. We have any number of
customers begging for it, as a matter of fact.

> For
> one thing, it was pretty painless for me to set up cron jobs to suck in
> whatever messages I save to certain folders (that could have been named
> anything) for training my spam filter.

Are you serious? Cron jobs? Do you have any idea how minute a fraction of the
users of email even know what a cron job is, let alone know how to set one up?

The minute you start talking about setting up your own cron jobs, it  pretty
much goes without saying that you're operating entirely outside the space where
99+% of the use of IMAP takes place.

> I also have other cron jobs set up
> to do other things with email completely unrelated to spam that also
> simply involve saving to a particular folder (e.g. a tickler file for
> email--I move messages to folders named for a month or day, and cron jobs
> automatiacally reinsert them back in my inbox later).  That kind of thing
> isn't addressed by having something spam-specific, but as far as my email
> client is concerned, they're flagged the same way.

And no doubt you also tweak SpamAssasin rules, set up milters, etc. etc. This
is far beyond the capability of most users.

				Ned

Date: Mon, 06 Jul 2009 19:29:20 +0100
From: Alexey Melnikov <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>
CC: Ned Freed <[email-masked]>, [email-masked], [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Lyndon Nerenberg wrote:

>But if your concern is adoption rates, a keyword-only solution will get
>wider take-up than something that requires support for annotations. (And
>from a purely engineering standpoint, the simplest solution wins.)
>  
>
+1.

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
From: Lyndon Nerenberg <[email-masked]>
To: Ned Freed <[email-masked]>
Cc: [email-masked], [email-masked]
Date: Mon, 06 Jul 2009 12:26:01 -0600
Sender: [email-masked]

On Mon, 2009-07-06 at 11:00 -0700, Ned Freed wrote:
> If there's a reason multiple folders can't be marked this way I sure
> don't
> know what it is.

You're right -- I didn't parse what you were saying correctly the first
time around.

But if your concern is adoption rates, a keyword-only solution will get
wider take-up than something that requires support for annotations. (And
from a purely engineering standpoint, the simplest solution wins.)

--lyndon

Date: Mon, 06 Jul 2009 19:14:38 +0100
From: Alexey Melnikov <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
CC: general discussion of application-layer protocols <[email-masked]>, [email-masked], Ned Freed <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, George Michaelson <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> Ned Freed writes:
>
>> The obvious way to make this work in IMAP is with a standardized 
>> folder annotation saying "messages put in this folder were flagged as 
>> spam".
>
> I think what Thunderbird and Apple Mail do is to set specific message 
> flags on messages in whatever folders they happen to be in: $Junk if 
> the user said a particular message is spam, $autojunk if tb 
> autonomously decided that message is spam, $nonjunk and $autononjunk 
> for the opposite. (I may be wrong about the exact flag names.)

Something like that, yes.

> This was specified in some expired draft,

Which was mine.

> which IMO now has been frozen by deployment. Apple Mail and 
> Thunderbird add up to >10% for me. Time for an informational RFC, 
> perhaps. "Widely-used IMAP flag names"?

Yes. I think even more important to create the IANA registry for keywords.
Do you want to help finish my draft?

Date: Mon, 06 Jul 2009 11:00:24 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Lyndon Nerenberg <[email-masked]>
Cc: Ned Freed <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> On Sun, 2009-07-05 at 17:40 -0700, Ned Freed wrote:
> > The obvious way to make this work in IMAP is with a standardized
> > folder
> > annotation saying "messages put in this folder were flagged as spam".
> > Then
> > moves of messages to the folder with this annotation provide the
> > server with
> > the necessary information that this message is considered to be spam
> > by the
> > user. And by the same token, moving a message from this folder can be
> > construed
> > as the message not being considered spam after all.

> This is just trying to turn a folder into a keyword, but without full
> keyword semantics.

Actually, it's a hack t work around the lack of client support, which is why
I didn't propose a keyword-based mechanism.

> The most obvious flaw is that it constrains the
> client to storing all the spam in one bucket.

If there's a reason multiple folders can't be marked this way I sure don't
know what it is.

> It doesn't allow for (say)
> sorting junk by type into different folders (perhaps on different
> servers). A message's 'state of spamminess' is an attribute of the
> message, properly denoted by keywords that stick to (and with) the
> message.

I see no reason why this couldn't be done, if you really want to.

> If you want this functionality (and I think it's a useful service for
> servers to provide) you need to implement two keywords: $spam and
> $nospam (these are just example names). Setting $spam on a message would
> instruct the server to pass the message off to the filtering component
> as spam; setting $nospam would instruct the server to pass the message
> off to the filtering component as valid (non-spam). The two keywords are
> exclusive, and it would be up to the server to decide what action to
> take if the client tries to light up both at once (e.g. unset the other
> flag and take the corresponding action, reject the command, or silently
> ignore the request).

Again, the problem with using keywords is now you have to get a significant
fraction of clients on board before it's useful. I'd be willing to go along
with a keyword-based approach if I could convinced that's going to happen, but
everything recentlts has supported exactly the opposite conclusion.

				Ned

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
From: Lyndon Nerenberg <[email-masked]>
To: Ned Freed <[email-masked]>
Cc: [email-masked]
Date: Mon, 06 Jul 2009 12:02:40 -0600
Sender: [email-masked]

On Mon, 2009-07-06 at 10:39 -0700, Ned Freed wrote:
> The problem with using keywords for this is that you are now
> dependence on your
> client supporting that particular set of keywords and their associated
> semantics. Had we standardize such a keyword set 10 or so years ago
> this
> approach miight have had a chance, but the client base is now too
> large and
> too difficult to update.

Is it really too late? We're making an request to process a particular
message through the site's "learning" filters, i.e. this is explicit
request to have the server take action on the message. I don't that
behaviour is implied by any of the current schemes floating around.

I don't see how it could hurt to standardize a pair of new keywords (as
I suggested in my previous message) with this behaviour documented. It
lets the user request specific identified behaviour from the server; the
existing keywords are all private switches for client behaviour, in my
experience, so the new keyword behaviour would be complimentary to
what's already out there.

--lyndon

Date: Mon, 6 Jul 2009 12:44:14 -0500
From: Nicolas Williams <[email-masked]>
To: Ned Freed <[email-masked]>
Cc: Iljitsch van Beijnum <[email-masked]>, [email-masked], George Michaelson <[email-masked]>, [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

On Sun, Jul 05, 2009 at 05:40:31PM -0700, Ned Freed wrote:
> 
> >On 3 jul 2009, at 1:05, George Michaelson wrote:
> 
> >> As an example, at the moment if I wish to inform google that I have
> >> known spam in local folders, I have to go to the web interface and
> >> manually tag. If there was an IMAP extension, I could review my
> >> local baysian junk folder, remove all non-spam (and flag the senders
> >> as white-listed if need be), and request the rest to be flagged as
> >> spam back on the IMAP backed MS.
> 
> >Doesn't moving the spam messages to the spam folder accomplish this
> >already? That's what my client does.
> 
> Please explain how moving a message to a random, arbtirarily named
> folder tells the server anything useful. People use all sorts of
> different naming schemes for folders, and it's common for clients to
> provide lots of flexibility in this area.

In particular the word 'spam' is really an English-language word.  L10N
would demand that the name of this folder not be 'spam' in all locales,
or at least that clients know how to map that folder name to the user's
language(s).  But even if we go for the latter, having special folder
names means denying or complicating the use of those names in other
languages for other purposes; it's bad enough we have one well-known
folder name (INBOX), let's have no more.

> The obvious way to make this work in IMAP is with a standardized
> folder annotation saying "messages put in this folder were flagged as
> spam". Then moves of messages to the folder with this annotation
> provide the server with the necessary information that this message is
> considered to be spam by the user. And by the same token, moving a
> message from this folder can be construed as the message not being
> considered spam after all.

+1

Or even just a message flag/annotation.  I understand the "move junk
mail to the junk mail folder" metaphor for flagging junk mail as such,
but one might want to flag junk mail and still move it to an "archived
mail" folder.  I don't find my argument for a per-message flag terribly
convincing, so I won't push it.

> It isn't perfect, but as long as this is taken as an indicator, not as
> an absolute statement, it should get the job done.
> 
> >But if you want this, I'd say that it needs to be a fractional thing,
> >not a binary spam/no spam indication. For instance, the server could
> >give something a spam score of 2 and the client also 2 and together
> >that would be 4 so the message is presumed to be spam (assuming the
> >spam threshold is 3), but in a binary system no spam OR no spam = no
> >spam.
> 
> There's a place for fractional scoring in spam filtering, but this isn't
> it. A user doesn't think "this message is 90% likely to be spam", they
> either think it's spam or it isn't.

+1

Fractional scoring by the server, or by software running on the client
can be useful, but the users themselves will generally make binary spam/
not-spam decisions.

Nico
-- 

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
From: Lyndon Nerenberg <[email-masked]>
To: Ned Freed <[email-masked]>
Cc: [email-masked], [email-masked]
Date: Mon, 06 Jul 2009 11:53:41 -0600
Sender: [email-masked]

On Sun, 2009-07-05 at 17:40 -0700, Ned Freed wrote:
> The obvious way to make this work in IMAP is with a standardized
> folder
> annotation saying "messages put in this folder were flagged as spam".
> Then
> moves of messages to the folder with this annotation provide the
> server with
> the necessary information that this message is considered to be spam
> by the
> user. And by the same token, moving a message from this folder can be
> construed
> as the message not being considered spam after all.

This is just trying to turn a folder into a keyword, but without full
keyword semantics. The most obvious flaw is that it constrains the
client to storing all the spam in one bucket. It doesn't allow for (say)
sorting junk by type into different folders (perhaps on different
servers). A message's 'state of spamminess' is an attribute of the
message, properly denoted by keywords that stick to (and with) the
message.

If you want this functionality (and I think it's a useful service for
servers to provide) you need to implement two keywords: $spam and
$nospam (these are just example names). Setting $spam on a message would
instruct the server to pass the message off to the filtering component
as spam; setting $nospam would instruct the server to pass the message
off to the filtering component as valid (non-spam). The two keywords are
exclusive, and it would be up to the server to decide what action to
take if the client tries to light up both at once (e.g. unset the other
flag and take the corresponding action, reject the command, or silently
ignore the request).

Date: Mon, 6 Jul 2009 19:50:08 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: general discussion of application-layer protocols <[email-masked]>, [email-masked]
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Cc: Ned Freed <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, George Michaelson <[email-masked]>
Sender: [email-masked]

Ned Freed writes:
> The obvious way to make this work in IMAP is with a standardized 
> folder annotation saying "messages put in this folder were flagged as 
> spam".

I think what Thunderbird and Apple Mail do is to set specific message 
flags on messages in whatever folders they happen to be in: $Junk if 
the user said a particular message is spam, $autojunk if tb 
autonomously decided that message is spam, $nonjunk and $autononjunk 
for the opposite. (I may be wrong about the exact flag names.)

This was specified in some expired draft, which IMO now has been frozen 
by deployment. Apple Mail and Thunderbird add up to >10% for me. Time 
for an informational RFC, perhaps. "Widely-used IMAP flag names"?

Arnt

Date: Mon, 06 Jul 2009 10:39:40 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Alexey Melnikov <[email-masked]>
Cc: Iljitsch van Beijnum <[email-masked]>, Dave Cridland <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>, George Michaelson <[email-masked]>
Sender: [email-masked]

> Dave Cridland wrote:

> > On Sun Jul  5 20:03:30 2009, Iljitsch van Beijnum wrote:
> >
> >> On 3 jul 2009, at 1:05, George Michaelson wrote:
> >>
> >>> As an example, at the moment if I wish to inform google that I
> >>> have  known spam in local folders, I have to go to the web
> >>> interface and  manually tag. If there was an IMAP extension, I
> >>> could review my  local baysian junk folder, remove all non-spam
> >>> (and flag the senders  as white-listed if need be), and request  the
> >>> rest to be flagged as  spam back on the IMAP backed MS.
> >>
> >> Doesn't moving the spam messages to the spam folder accomplish this
> >>  already? That's what my client does.
> >
> > There's also a proposal (expired?) to use keywords to signal  spamminess.

> To expand on this: I believe the proposal is implemented by several
> major email clients (Thunderbird is one of them).

> [For people not familiar with IMAP: IMAP keywords is a basic IMAP
> feature implemented by majority of IMAP servers I've seen.]

The problem with using keywords for this is that you are now dependence on your
client supporting that particular set of keywords and their associated
semantics. Had we standardize such a keyword set 10 or so years ago this
approach miight have had a chance, but the client base is now too large and
too difficult to update.

This is why I think folder annotations are a better bet. You mark the spam
folder and use that marking to attach meaning to the movement of messages into
and out of that folder. It's not perfect, but it has the key advantage that it
works with any client (OK, any client that can use folders) once the annotation
is in place.

				Ned

Date: Sun, 05 Jul 2009 17:40:31 -0700 (PDT)
From: Ned Freed <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
To: Iljitsch van Beijnum <[email-masked]>
Cc: George Michaelson <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> On 3 jul 2009, at 1:05, George Michaelson wrote:

> > As an example, at the moment if I wish to inform google that I have
> > known spam in local folders, I have to go to the web interface and
> > manually tag. If there was an IMAP extension, I could review my
> > local baysian junk folder, remove all non-spam (and flag the senders
> > as white-listed if need be), and request the rest to be flagged as
> > spam back on the IMAP backed MS.

> Doesn't moving the spam messages to the spam folder accomplish this
> already? That's what my client does.

Please explain how moving a message to a random, arbtirarily named folder tells
the server anything useful. People use all sorts of different naming schemes
for folders, and it's common for clients to provide lots of flexibility in this
area.

The obvious way to make this work in IMAP is with a standardized folder
annotation saying "messages put in this folder were flagged as spam". Then
moves of messages to the folder with this annotation provide the server with
the necessary information that this message is considered to be spam by the
user. And by the same token, moving a message from this folder can be construed
as the message not being considered spam after all.

It isn't perfect, but as long as this is taken as an indicator, not as an
absolute statement, it should get the job done.

> But if you want this, I'd say that it needs to be a fractional thing,
> not a binary spam/no spam indication. For instance, the server could
> give something a spam score of 2 and the client also 2 and together
> that would be 4 so the message is presumed to be spam (assuming the
> spam threshold is 3), but in a binary system no spam OR no spam = no
> spam.

There's a place for fractional scoring in spam filtering, but this isn't
it. A user doesn't think "this message is 90% likely to be spam", they
either think it's spam or it isn't.

				Ned

Date: Mon, 06 Jul 2009 11:27:45 +0100
From: Alexey Melnikov <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
CC: Dave Cridland <[email-masked]>, [email-masked], general discussion of application-layer protocols <[email-masked]>, George Michaelson <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Sender: [email-masked]

Dave Cridland wrote:

> On Sun Jul  5 20:03:30 2009, Iljitsch van Beijnum wrote:
>
>> On 3 jul 2009, at 1:05, George Michaelson wrote:
>>
>>> As an example, at the moment if I wish to inform google that I  
>>> have  known spam in local folders, I have to go to the web  
>>> interface and  manually tag. If there was an IMAP extension, I  
>>> could review my  local baysian junk folder, remove all non-spam  
>>> (and flag the senders  as white-listed if need be), and request  the 
>>> rest to be flagged as  spam back on the IMAP backed MS.
>>
>> Doesn't moving the spam messages to the spam folder accomplish this  
>>  already? That's what my client does.
>
> There's also a proposal (expired?) to use keywords to signal  spamminess.

To expand on this: I believe the proposal is implemented by several 
major email clients (Thunderbird is one of them).

[For people not familiar with IMAP: IMAP keywords is a basic IMAP 
feature implemented by majority of IMAP servers I've seen.]

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Mon, 06 Jul 2009 10:05:41 +0100
From: Dave Cridland <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>, <[email-masked]>, general discussion of application-layer protocols <[email-masked]>, George Michaelson <[email-masked]>
Sender: [email-masked]

On Mon Jul  6 09:37:06 2009, Iljitsch van Beijnum wrote:
> =

> On 6 jul 2009, at 10:24, Dave Cridland wrote:
> =

>>> But if you want this, I'd say that it needs to be a fractional   =

>>> thing,  not a binary spam/no spam indication. For instance, the   =

>>> server could  give something a spam score of 2 and the client  =

>>> also  2 and together  that would be 4 so the message is presumed  =

>>> to be  spam (assuming the  spam threshold is 3), but in a binary  =

>>> system no  spam OR no spam =3D no  spam.
> =

>> The notion of using two spam filters in concert to attempt to make  =

>>  an overall decision is basically flawed, because it fits into one  =

>> or  other case above - if you do go to the effort of having a  =

>> range  based spamminess from one, you'd have to use it as mere  =

>> input into  the other's decision process, since combining them  =

>> na=EFvely would  produce poorly weighted results.
> =

> Right. The filtering would have to be complimentary. So perhaps  =

> there  should be an indication of the type of filtering done, too.  =

> The reason  to combine server and client side approaches is that  =

> servers can  easily do blacklists, and running any type of  =

> filtering on the server  gives it the opportunity to reject the  =

> message during the SMTP  session, creating a good error message for  =

> real senders but no useless  bounces to innocent third parties in  =

> the case of spoofed senders. On  the client is useful because the  =

> client can typically better afford to  run CPU intensive stuff like  =

> baysian filters and the client probably  has a better list of  =

> previous correspondents that it can use to  whitelist.

Whitelists should, in principle, be easily comminicable to the server  =

(and easily processed).

Bayesian filters on the client are only really useful if the same  =

client is always used - this is often not the case (and, in my  =

experience, is becoming rarer). Otherwise the quality of your spam  =

filtering radically changes depending on the client you happen to be  =

using at the time.

And basically, any filtering at all that can be done on the server  =

saves a vast amount of bandwidth, and makes use of email on smaller  =

devices (like mobile phones) instantly more useful.

The one massive advantage that a client has is direct contact with  =

the user, and the eyeball remains the best spam filtering technology  =

we have.

I'm not suggesting that clients must not do spam filtering -  =

obviously they do, and it's often useful - but I think we should aim  =

to make server-side filtering the best we can. A very simple uniform  =

feedback mechanism of whether the user thinks a message is spam or  =

not makes this very much easier.

Dave.
-- =

Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: <[email-masked]>, general discussion of application-layer protocols <[email-masked]>, George Michaelson <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
To: Dave Cridland <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Mon, 6 Jul 2009 10:37:06 +0200
Sender: [email-masked]

On 6 jul 2009, at 10:24, Dave Cridland wrote:

>> But if you want this, I'd say that it needs to be a fractional =20
>> thing,  not a binary spam/no spam indication. For instance, the =20
>> server could  give something a spam score of 2 and the client also =20=

>> 2 and together  that would be 4 so the message is presumed to be =20
>> spam (assuming the  spam threshold is 3), but in a binary system no =20=

>> spam OR no spam =3D no  spam.

> The notion of using two spam filters in concert to attempt to make =20
> an overall decision is basically flawed, because it fits into one or =20=

> other case above - if you do go to the effort of having a range =20
> based spamminess from one, you'd have to use it as mere input into =20
> the other's decision process, since combining them na=EFvely would =20
> produce poorly weighted results.

Right. The filtering would have to be complimentary. So perhaps there =20=

should be an indication of the type of filtering done, too. The reason =20=

to combine server and client side approaches is that servers can =20
easily do blacklists, and running any type of filtering on the server =20=

gives it the opportunity to reject the message during the SMTP =20
session, creating a good error message for real senders but no useless =20=

bounces to innocent third parties in the case of spoofed senders. On =20
the client is useful because the client can typically better afford to =20=

run CPU intensive stuff like baysian filters and the client probably =20
has a better list of previous correspondents that it can use to =20
whitelist.=

Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Mon, 06 Jul 2009 09:24:42 +0100
From: Dave Cridland <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>, <[email-masked]>, general discussion of application-layer protocols <[email-masked]>, George Michaelson <[email-masked]>
Sender: [email-masked]

On Sun Jul  5 20:03:30 2009, Iljitsch van Beijnum wrote:
> On 3 jul 2009, at 1:05, George Michaelson wrote:
> =

>> As an example, at the moment if I wish to inform google that I  =

>> have  known spam in local folders, I have to go to the web  =

>> interface and  manually tag. If there was an IMAP extension, I  =

>> could review my  local baysian junk folder, remove all non-spam  =

>> (and flag the senders  as white-listed if need be), and request  =

>> the rest to be flagged as  spam back on the IMAP backed MS.
> =

> Doesn't moving the spam messages to the spam folder accomplish this  =

>  already? That's what my client does.
> =

> =

There's also a proposal (expired?) to use keywords to signal  =

spamminess.

> But if you want this, I'd say that it needs to be a fractional  =

> thing,  not a binary spam/no spam indication. For instance, the  =

> server could  give something a spam score of 2 and the client also  =

> 2 and together  that would be 4 so the message is presumed to be  =

> spam (assuming the  spam threshold is 3), but in a binary system no  =

> spam OR no spam =3D no  spam.

Of course, if both client and server use precisely the same criteria,  =

you've simply halved your threshold.

There's two cases:

1) The server has some feedback-based spam detection mechanism, like  =

a bayesian filter. You want to teach the server's filter about your  =

explicit decisions.

2) Your client has a spam filter (or some sort), you want the server  =

to tell your client about it's preliminary findings.

The notion of using two spam filters in concert to attempt to make an  =

overall decision is basically flawed, because it fits into one or  =

other case above - if you do go to the effort of having a range based  =

spamminess from one, you'd have to use it as mere input into the  =

other's decision process, since combining them na=EFvely would produce  =

poorly weighted results.

A simple keyword approach isn't quite as good as ranges, but it does  =

have the useful property that nearly every service already offers  =

arbitrary keywords, and so is quite likely to already offer it.

Ranges, on the other hand, require the use of annotations, which is a  =

rather more complex area.

Dave.
-- =

Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Cc: [email-masked], [email-masked]
From: Iljitsch van Beijnum <[email-masked]>
To: George Michaelson <[email-masked]>
Subject: Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing
Date: Sun, 5 Jul 2009 21:03:30 +0200
Sender: [email-masked]

On 3 jul 2009, at 1:05, George Michaelson wrote:

> As an example, at the moment if I wish to inform google that I have  
> known spam in local folders, I have to go to the web interface and  
> manually tag. If there was an IMAP extension, I could review my  
> local baysian junk folder, remove all non-spam (and flag the senders  
> as white-listed if need be), and request the rest to be flagged as  
> spam back on the IMAP backed MS.

Doesn't moving the spam messages to the spam folder accomplish this  
already? That's what my client does.

But if you want this, I'd say that it needs to be a fractional thing,  
not a binary spam/no spam indication. For instance, the server could  
give something a spam score of 2 and the client also 2 and together  
that would be 4 so the message is presumed to be spam (assuming the  
spam threshold is 3), but in a binary system no spam OR no spam = no  
spam.