Subject: Re: Who will write filtering scripts?

From: John Mani <[email-masked]>
Date: Fri, 28 Mar 1997 09:48:00 -0800 (PST)

Now that we have some activity in this mailing list, I'd like to hear ideas
and opinions about the infrastructure for filtering:

+ Who runs the filter ? Choices:
 1. The delivery agent at the time of delivery to one's
    INBOX ? How does the delivery agent have access to the user's folders ?
    Thru IMAP ? (which means that the delivery agent needs an open connection
    to the server all the time ... and the server may need to support shared
    read-write access to folders ..)
 2. The IMAP server ? So that the filtering happens when the INBOX is selected ?
    Or should there be an extension to the server to start filtering ?
    
+ Where/how is the filter installed ?
    The filter has to be installed in a user-preference repository. ACAP/LDAP
    seems to be the obvious way to setup/access the filter.
    
Comments/Ideas ?

-john mani
    	
> From: Amos A Gouaux <[email-masked]>
> To: Tim Showalter <[email-masked]>
> cc: MTA Filters <[email-masked]>
> Subject: Re: Who will write filtering scripts?
> MIME-Version: 1.0
> 
> As I already mentioned to Tim, I personally think machine-readable is more
> important.  Even if it were to be human-writable, from what I've seen
> between Elm Filter and Procmail debates, folks are going to find it
> difficult to write correct filtering rules unless the language is
> simplistic to the point of being almost useless.  At my previous job, when
> faculty wanted their mail filtered, we ended up writing the procmail
> script ourselves (support staff).  So I would prefer to see this new
> filtering language machine-readable and then have a tk or even java app as
> the configuration tool.  Then eventually some kind of ACAP interface. 
> 
> Besides, with the movement towards client-server mail access with IMAP (or
> POP), I think server-side filtering is going to be an increasingly
> pressing issue.  It might even get to the point were editing local files
> like one's ~/.forward is a less common occurrence.  And even if folks
> still edited their own filter file, I would still prefer it to be
> machine-readable so that the mail delivery mechanism would be as fast and
> efficient as possible.
> 
> As for question of syntax, I'm certainly agreeable to it being LISP-like.
> Hmmm, I wonder if I could then plug it into GNUS..  ;-)
> 
> amos
> 
> 
> On Thu, 27 Mar 1997, Tim Showalter wrote:
> 
> > I'm working on a draft for a filtering language.  I have one fairly
> > fundamental question: should the language be geared to be machine-readable
> > or human-writable?  It would be far easier to write a prefix-oriented
> > language with lots of LISP-like syntax, but I believe most users find that
> > difficult to use (and it is my limited experience it's hard to balance
> > parens without an editor that helps out).
> > 
> > If it's machine readable, logical operators should be prefix.  Syntax should
> > not be overly important.
> > 
> > If it's human writable, logical operators should be infix.  Syntax should be
> > geared towards preventing syntax errors and readability.
> > 
> > Any comments will be appreciated.
> > 
> > -- 
> >                                            Tim Showalter [email-masked]
> > 
> > 
> 
> 
>

---

From: Tim Showalter <[email-masked]>
Date: Fri, 28 Mar 1997 13:52:58 -0500 (EST)

> + Who runs the filter ? Choices:

I would like to integrate filtering into the Cyrus deliver program, which
takes mail from sendmail (or whatever other MTA is in use) and puts it in an
IMAP mailbox and updates the cache files.

I see no reason why one language couldn't be used to support multiple
attitudes on when to filter -- for example, in Cyrus, we might not allow as
much flexibility as a user might like, but if his client supported
filtering, he might be able to get his client to do it.

>     read-write access to folders ..)

Shared read-write access to folders isn't a big problem for us, but it is
for others.  I believe delivery is the best answer, but that it can't be
implemented for everyhone.

>     Or should there be an extension to the server to start filtering ?

If I have a mailbox called "user.tjs.foo", and I want others to read it, I
need a way to submit messages to it.  By having filtering happen at
select-of-INBOX time, we have a number of serious problems with this model
(which several people have wanted to use under CMU's legacy system, but
haven't been able to).

a) Until I select my INBOX, no one else can read the messages.  I might
   go on vacation, or I might be lax and not read my mail for three days.
b) If it's a client extension, filtering requires client support -- I
   consider this just another obstacle.  Multiple folder support is enough
   to do filtering, and I'd rather not impose other requirements on it.

>     seems to be the obvious way to setup/access the filter.

Disclaimer: I'm biased towards ACAP.  In CMU's legacy system, a common
networked file system was used (AFS), and I suspect a shared file system
will be good enough for a lot of sites.

---

From: Randall Gellens <[email-masked]>
Date: Fri, 28 Mar 1997 10:53:49 -0800

>     Or should there be an extension to the server to start filtering ?

It seems to me filtering should be done during final delivery, before
placing the message in any mailbox.  This was the filter rules can file,
forward, trash, etc. the message before it has been delivered.  Which
software module actually runs the filter rules of course depends on the
implementation.

>     seems to be the obvious way to setup/access the filter.

I think ACAP is the logical place to store the filter rules.  In many ways,
filter rules look like a subset of an address book, and so would probably
go in the user's options, perhaps in a new dataset, which would be defined
and agreed to for interoperability.

The user would store/update the filters using ACAP.

The server could access the filters also via ACAP, or via some back-end
protocol or common back-end between it an ACAP.  That is
implementation-dependent.  There is a need for close integration between
IMAP and ACAP in general, but there are several ways an implementation can
do this.  For interoperability, all implementations should be able to fall
back on using ACAP in the normal way, but may be able to use local
optimizations as well.

---

From: Tim Showalter <[email-masked]>
Date: Fri, 28 Mar 1997 15:56:56 -0500 (EST)

> software module actually runs the filter rules of course depends on the

I agree with you, except this isn't possible in a POP3 environment where
folders are maintained on the client's end only.  I believe this is
important because there are a number of people who want to use a filtering
language cooperatively to stop spam, many of whom are not using IMAP.

> and agreed to for interoperability.

If a filter script looks like a simple set of "if A then do B" statements,
this works.  If a filtering script is more complicated -- nested
conditionals -- I don't think the mapping is so clear.

I also have concerns with extensions.  In the case of an IMAP server doing
filtering, the extensions supported will not be clear to the ACAP server
accepting scripts.  The ACAP server should not be expected to verify the
syntax of scripts; the store-this-script-client needs to do that, and I'm
not sure how it finds out that the IMAP server supports some random
extension.

The best I could come up with to solve this was to just demand that sites
advertise supported extensions and if filtering fails, messages get dumped
into INBOX.

---

From: Randall Gellens <[email-masked]>
Date: Fri, 28 Mar 1997 13:32:33 -0800

> language cooperatively to stop spam, many of whom are not using IMAP.

If the filters are being run by the client, then the client should run them
after receiving the message via POP and before storing the message in a
folder.  In this situation, "final delivery" means delivery via POP.

> not sure how it finds out that the IMAP server supports some random

ACAP does not verify syntax on options.  There really isn't any "mapping"
when storing filters in ACAP.  The way I see the filter dataset, it is very
simple: just a set of entries, each entry consists of an entry (name)
attribute and a filter attribute.  The entry attribute is any unique handle
assigned by the filter generator (probably the client), and the filter
attribute is just a text string.  For example:

\user\fred\filters
	dataset.entry = "make-money-fast"
	filter = "if = ( subject, "make-money-fast" ) then trash"

	dataset.entry = "peggy-sue"
	filter = "if = ( from, "[email-masked]" ) then file
			("personal")"

This way the dataset knows very little of the structure of the filters, and
it is easy to extend.

> advertise supported extensions and if filtering fails, messages get dumped

Yes, there needs to be a way for a site to advertise which filter
extensions it supports, and there needs to be a default action for
unrecognized filter clauses or actions.  I think filter extensions should
be advertised as part of advertising filtering support, in an IMAP
CAPABILITIES.  This does beg the question of how to advertise non-IMAP
support, but at least it is a start.  I agree that default action should be
to file in INBOX.

---

From: Tim Showalter <[email-masked]>
Date: Fri, 28 Mar 1997 16:53:00 -0500 (EST)

> folder.  In this situation, "final delivery" means delivery via POP.

Ok.

> 	filter = "if = ( from, "[email-masked]" ) then file

I'd like to allow nested conditionals -- this makes things a little bit more
complicated.  Also, guaranteeing the order of evaluation might be useful,
but might be difficult to work out with ACAP.

> support, but at least it is a start.  I agree that default action should be

Advertising filtering support in CAPABILITIES only answers the question for
IMAP, but it might be a workable approach.

---

From: Randall Gellens <[email-masked]>
Date: Fri, 28 Mar 1997 15:32:19 -0800

> but might be difficult to work out with ACAP.

I think we really need to focus on keeping the base spec simple, and
compatible with IMAP and ACAP.  The simplest way to store the filters in
ACAP would be as one attribute, in the user's options dataset.  That would
give you the explicit ordering you want, but would make it a bit difficult
for some clients to deal with very long sets of filters.  The other
approach I see is to store each filter rule in its own attribute.  Ordering
could be enforced by an extra attribute (execute-order) which would contain
a number.

---

From: Amos A Gouaux <[email-masked]>
Date: Fri, 28 Mar 1997 18:13:28 -0600 (CST)

> could be enforced by an extra attribute (execute-order) which would contain

I'm not that familiar with ACAP, but IMSP is used by one mail client I
know to store its config info.  It is able to keep its message groups in
the correct order by numbering each message group internally.  Perhaps
something similar could be used with filters, like you suggest.

amos

---

From: Chris Newman <[email-masked]>
Date: Fri, 28 Mar 1997 21:09:59 -0800 (PST)

> something similar could be used with filters, like you suggest.

ACAP is IMSP mark 2.  As you suggest, ordering isn't a big problem in this
model.