Subject: Re: MTA Filters BOF request, LA IETF; Proposed Charter

From: "Matthew Wall" <[email-masked]>
Date: Mon, 12 Jan 1998 17:44:21 -0500

--On Mon, Jan 12, 1998 16:51 -0500 "Keith Moore" <[email-masked]> wrote: 

> I'm very dubious about the desirability of standardizing an
> MTA-level mail filtering language.

> 
> in addition, the notion of a "general" filtering language,
> as opposed to one specifically designed for email, sounds
> like a rathole to me.

Maybe some language problems here on my part, specifically the intent of
"generalized". We don't mean the swiss-army-knife generalized; emphasis on
the use of the word to mean "simplified enough that it makes sense in
several related contexts".

I shall attempt to clarify and make a little more explicit what's going on.
There's a two-year history dating back to the January '96 IMAP meeting, and
the working SIEVE draft probably bears some examination to illuminate the
proposed WG's goals.

The proposed filtering language is only 'generalized' for use by delivery
agents tied to the MTA. I don't think we've ever been discussing anything
much more ambitious than that. It's desirable (saying this as a current
client guy) that this be 'fungible' between client and server sides, but not
a fundamental requirement. 

I can remember conversations about this as long ago as 1994 in the IMAP4
revisions penumbra. The specific impetus to go the current design route
originally came out of the early IMAP discussions as a particularly bad hole
in the architecture. Specifically, we had the first such meeting at the
first formal IMAP meeting in Seattle, and Terry Gray at that time proposed
dealing with it as a general issue rather than one tied specifically to
IMAP. It was identified specifically from the podium at the 2nd IMAP meeting
as among the most important architectural issues, and about 250 industry
people around the room nodded their heads. We've since continued to have
pretty broad participation, and we'd've proposed the WG a long time ago save
for the fact that many of the originators of the discussion have had other
windmills to joust at in the interim 8-). In fact, I was prodded to do this
after the dozenth time I'd answered the question at the DC IETF 'what's up
with SIEVE? We need this yesterday...'

I also count up 300 some posts from 50 or so domains on the list and related
discussions over the past 18 months. I believe this in and of itself
justifies consideration of pursuing this formally via the IETF. There's
clearly a large segment of the usual suspects who've considered this a
worthwhile area to spend time on.

Fo further clarify...the specific action issues that are a big problem here
are: vacation, auto-reply, auto-discard, and auto-file to multiple INBOX
locations (including resubmission, aka forward) according to logic. These
are traditionally MTA-delivery side things, are implemented in an incredible
variety of ways, and are completely obscure in a client/server architecture
-- ' things you or your server admin have to login and futz with to do', as
these were generally described at one of the meetings. I'd argue the basic
set of activities screams for a standardized approach. From the client side
of things, it's virtually impossible to talk about replacing proprietary
functionality with the internet architecture without a standards approach.

The last, the auto-file issue, is one that's become recently quite important
to certain IMAP implementations.  I thing we're generally agreed that
requiring delivery behavior via the access protocol is a bad thing and kind
of limited, and at the same time it's out of scope for the 821/822 revisions
and discussions. The others are all time-tested issues with just gobs of
current non-interop problems. Just do a search in the discussion groups for
getting filter agent X to work with SMTP agent Y and IMAP server Z -- and
this is a knowledgeable crowd, to boot.

The scope is purposefully quite limited here. Concentrating on a relatively
simple standard syntax is just about as minimal as you can get here. It's
location-neutral -- i.e. appropriate for stores ranging from local disks and
.rc files to more nuanced approaches like ACAP -- protocol-free, to avoid
adding more on-the-wire requirements -- and "generalized" only in the sense
that it's simple enough that both client and server implementers can build
on it.

I'm very aware of the rathole potentials here, ranging from current
commercial vendors getting enmired in retroactive compatibility issues (ala
Cal/Sched) to a misinterpretation of the problem area as merely anti-spam
(when this is only a small tool for that very large general area). Awareness
of potential ratholes is specifically why the scope of the language was
limited, it was designed to be a 'primitives' subset, and it's not a
Turing-complete language with all the "real" language and parsing issues
involved. It's a way of specifying a limited set of logical operations that
are common to MTA delivery.

Believe me, I have no love of having to face these potential ratholes. But
having been down the server side and client side both over the past few
years, a simple lingua franca would be enormously helpful. Many people are
going to implement their own chimerical versions regardless; it's a question
of whether the IETF will take a leadership stance, or we're going to have
permanent non-interoperability for some key functions that are missing from
the architecture.

> 
> with few exceptions, the mail transport system should not be 
> doing mail filtering.  the mail transport's job is to deliver
> mail, not to make filtering decisions on behalf of the user.
> if recipients want to filter their own mail, that's their 
> business, but then it's a UA-level thing.

I disagree. Filtering is something that *is* done at the delivery end of the
transport cycle. A "UA" relies on delivery and transport agents to make a
number of decisions. If it was merely a UA activity, we wouldn't have
vacation, procmail, .forward, IMAP BB+ -style delivery, auto-bouncing, or
any current filtering activities done completely without the participation
of a UA.

I personally hear from customers with increasing frequency about the need to
move and share filter-style activities. UAs obviously play a key role for
the individual user, but UA, location-, and even site-independence to be
able to move these guys about are really useful. As IMAP is to mail access,
maybe one could think of the MTA-filter proposal is to 'filter access',
minus the protocol overhead requirements.

Others, please chime in. Keith needs to be convinced... 8-)

- Matt

-----------------------------------------------------------------------
Matthew Wall                       mailto:[email-masked]
Cyrusoft International, Inc.       http://www.cyrusoft.com
Voice: 412 605 0499                Fax: 412 605 0705
-----------------------------------------------------------------------
     Proud Purveyors of the Mulberry IMAP Email Client
-----------------------------------------------------------------------

---

From: Keith Moore <[email-masked]>
Date: Mon, 12 Jan 1998 17:59:25 -0500

> any current filtering activities done completely without the participation

vacation and procmail *are* UAs, as are any programs to which mail is
forwarded for the purpose of filtering mail.

I can understand why a ubiquitous filtering language would be useful.
I can also understand how it could cause a great deal of harm, by making
the behavior of the mail transport system less predictable.  

Defining only the language also ignores the security issues.   How
are users going to specify such filtering without some means of 
authenticating themselves to the filter?

Keith

---

From: Lyndon Nerenberg <[email-masked]>
Date: Mon, 12 Jan 1998 16:28:19 -0700 (MST)

> forwarded for the purpose of filtering mail.

They aren't for a Windows 95 user running IMAP4 to a blackbox server 
that doesn't use UNIX accounts to provide IMAP account authentication.

> the behavior of the mail transport system less predictable. 

You'll get this with any filtering tool. Procmail is a very dangerous 
thing in the wrong hands. The fact that people use these dangerous 
tools regardless tells me there is a requirement for the functionality.
 
> authenticating themselves to the filter?

The point of SIEVE is to define the filtering language. How or where it 
gets stored is up to consenting clients and servers, although it's 
anticipated that ACAP will be the storage medium of choice. More than 
likely an individual user will have different datasets, residing in 
different locations,  providing different functionality.  E.g. ACAP 
will contain the rules I would like executed at the time the MTA hands 
off to the message store. The local disk on my workstation will contain 
a seperate set of rules for workstation-specific filtering I might want 
my local UA to perform. How the MTA, MDA, and UA find these rulesets is 
up to the implementation. What's important to me (and my customers) is 
that they be able to manipulate these rules in a consistent manner. 
This cries out for a standardized filtering language with UA support 
(GUI or whatever) for manipulating these rulesets.

Having a standard language means that I can reuse the same ruleset on 
different clients -- a *very* important criteria for mobile users who 
have no choice but to use disparate UA's when on the road vs. being at 
the office.

--lyndon

---

From: Keith Moore <[email-masked]>
Date: Mon, 12 Jan 1998 18:45:51 -0500

> that doesn't use UNIX accounts to provide IMAP account authentication.

If the filtering language is to be transmitted over the 'net between
a user and his mail service, then it's part of an on-the-wire protocol.
the filtering language might be thought of as a separate piece, but it's 
going to need some sort of transfer protocol with authentication to be 
useful in this scenario.

If the filtering language is not to be transmitted over the 'net,
then arguably it's system-private matter and not appropriate for
standardization.  (I don't entirely agree with this, but one
of the traditional rules of thumb is that system-private matters
don't belong in IETF)

> tools regardless tells me there is a requirement for the functionality.

Mail filtering is certainly useful.  My concerns are about whether
standarization of a mail filtering language is an appropriate activity
for IETF (given its tranditional scope and limited resources) and if so,
how to legitimize mail filtering without adversely impacting the mail 
transport system.

> anticipated that ACAP will be the storage medium of choice. 

Users will need a method to specify filtering to their ISPs.  If this 
method is not defined as part of the standard, different clients and
different ISPs will support different ways of specifying the filter,
and those users will have interoperability problems.  

I wonder if ISPs would really be willing to support ACAP as their 
storage medium of choice for mail filtering.

Keith

---

From: Lyndon Nerenberg <[email-masked]>
Date: Mon, 12 Jan 1998 17:39:05 -0700 (MST)

> going to need some sort of transfer protocol with authentication to be 

Well, RFC822 doesn't mandate RFC821. 822 is a very useful piece of 
work by itself; it doesn't require an on-the-wire protocol definition. 
(And I'm glad for that, having used 822 over more than one non-821 
transport-type beasty.)

> how to legitimize mail filtering without adversely impacting the mail 

I will admit it's a bit non-traditional in the IETF context. However, 
this initiative is being driven by many (most?) of the major players in 
the (IETF standards based) e-mail arena. We all agree that the only way 
to deliver functionality to our combined user community is by having a 
single, standard, light-weight filtering specification. That we also 
all agree that the IETF is the place to create this standard is a 
statement that the IETF directorate should pay attention to. (Many of 
the people who have been quietly pushing this initiative aren't 
newcomers to the IETF process.) If it's a *big* problem we could submit 
an informational RFC, but this is getting mired in politics and I'm not 
wanting to go there.  

[BTW, I'm using the "royal all" above. I'm sure not 100% of the 
SIEVE participants agree, but I haven't heard any of them speak up 
against this. ]

I'm not sure what you mean by "legitimize" above. The whole thing has 
been driven by customer demand. They *want* this, and they want it 
yesterday. There's no selling of the concept required. Now that ACAP is 
published I'm fielding  two to ten requests a week from people asking 
when we will have ACAP plus some sort of filtering language up and 
running. I'm sure other vendors can speak up as to numbers.

The bottom line here is that the user community wants it, and the 
vendor community agrees that an RFC is the only way to go. With that 
sort of support the IETF could be catching a lot of bad press from all 
directions if they adopt a stick-in-the-mud attitude on this. 

> storage medium of choice for mail filtering.

I can think of three or four large installations that would buy it 
*today* if we could deliver it (ACAP+SIEVE+IMAP4 server). The 
requirements are:

	* SIEVE ruleset manipulation support in the client, including 
	  fetch and store on an ACAP server,

	* An IMAP4 server that has the ability to filter a users 
	  incoming mail based on the SIEVE rules in that users
	  ACAP data set, and

	* An assurance that the ISP (and their customers) can switch 
	  components in and out (i.e. swap in a different vendor's 
	  IMAP4 server) without the rest of it breaking.

Now, if an RFC isn't the appropriate vehicle, can you suggest an 
alternative that would let us achieve the same thing (an open standard) 
in a similar timeframe? One thing that really concerns me is that if we 
*don't* get something out in a reasonable timeframe then vendors will 
simply define their own proprietary schemes. That's nothing but a lose 
for the everyone.

--lyndon

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 13 Jan 1998 09:22:25 +0100

Initial reactions:

1) I think a standard filter language is a Good Thing, exactly because
    the Right Place for the filtering is at delivery time, not reading time,
   and since all the rest of the UI/mailbox interface is standardized
   (IMAP, ACAP), the filtering language needs to be too.
   The alternative is that filters stay in the UI, and people have to wait
   minutes before reading their E-mail, not to mention the need to
   download everything for filtering before classifying most of it as
   "not worth reading now". BAD.
2) I will balk VERY strongly at having the letters MTA anywhere near
   the mailing list name (sorry!), the WG name or the charter text.
   The reason is that I believe the requirements for filtering at the UA
   and filtering in the message path to be VERY different; the UA must
   make a file/reply/drop/mark/call-attention decision; the MTA must
   make a relay/bounce/black-hole decision.
   These are differing requirements, I think aiming at both is likely to
   produce something that satisfies neither.
   I suggest calling it Mailbox Server Filtering Language - MSFL?
3) I will not object very strongly to an effort AFTER the UAFILTER effort
    to use the if-condition language for UA filtering to see if this can be
   applied to MTA filtering; as Keith said, it's doubtful that this will be
   anything but a short-term, somewhat-ineffective measure.

                             Harald A

who just spent about 3 days to get proper filter configurations into
Eudora, and is dreading the thought of moving mailbox yet again....

NOTE: New Email address: [email-masked]
I am working for Maxware (www.maxware.no) as of Dec 1, 1997

---

From: Tomas Fasth <[email-masked]>
Date: Tue, 13 Jan 1998 12:49:22 +0100

As Keith pointed out, there are certain aspects of MTA filtering that
IETF should be concerned about, more than just having a single filtering
language endorsed to become an IETF standard. The aspects I am thinking
of is interoperability, predictability and security. It might help if we
can give consenting answers to the following questions:

How should a MUA be able to transfer filtering rules to the MTA (or more
precisely; to the delivery agent) so that arbitrary brands of MUA and
MTA can interoperate?
And, how can this be done in a way preventing evil attacks to succeed?
How can the MTA be able to communicate filtering status and reports to a
MUA in an interoperable way between arbitrary brands?
And, what can we do to eliminate unpredictable results from running the
filter rules?

If we are going to endorse a standard on MTA filtering, we have to
address the interoperability issue as percieved from site deployment
point of view. Or do you suggest that we are going to say, "Hey, it's up
to you to find the pieces that fit together"? Seem not to adhere to the
general spirit of IETF, IMHO.

To be able to answer the questions above, I suggest we either discuss
the possibility to define a simple TCP based protocol to be used to
exchange information related to MTA filtering. Or, discuss the
possibility to define one or more MIME based formats suitable to
exchange information related to MTA filtering using the mail transport
itself as communication media between the MTA and the MUA.

These are the only alternatives that, IMO, can lead to an eventual IETF
standard on the issue of MTA filtering. SIEVE can of course be part of
that efford as the descriptive language of filtering rules, but it will
not do by itself.

In my own words,

--Tomas

---

From: Tim Showalter <[email-masked]>
Date: 13 Jan 1998 12:49:07 -0500

> From: Harald Tveit Alvestrand <[email-masked]>

>    the mailing list name (sorry!), the WG name or the charter text.

I think describing filtering taking place between the MTA and the message
store, or in the message store, as UA filtering is wrong, but taking MTA
out of the mailing list/WG names/charter text is fine with me.

>    make a relay/bounce/black-hole decision.

I see no reason a language can't do both of those.  The distinctions
between forward and reply, or reply and bounce, or between drop and black
hole, seem arbitrary.

It's been done on CMU's legacy system; the Sieve draft has mostly reached
concensus that the set of actions in Sieve are reasonable.  I think others
have filtering systems that do both sets of things, and they work.
(Procmail isn't the only one, but it's probably the example with the most
noteriety.)

>    I suggest calling it Mailbox Server Filtering Language - MSFL?

I named the language in the draft Sieve.  I like the name (and am sick of
acronyms) but if the name needs to be changed, so be it.

-- 
                                           Tim Showalter [email-masked]