Subject: Comments on draft-melnikov-sieve-external-lists-02.txt

From: Alexey Melnikov <[email-masked]>
Date: Sun, 19 Jul 2009 19:38:47 +0100

Hi Ned,
I've changed the subject and moving my reply to the Sieve mailing list, 
where I think it belongs.

Ned Freed wrote:
 [...]

>> There is also a Sieve extension for making use of whitelists/blacklists
>> during mail delivery. See draft-melnikov-sieve-external-lists-02.txt.
>> It can work with CARDDAV, LDAP and other things.
>
> Interestingly, this extension in its current form is not compatible with
> at least one of the proposals that was made here. The issue is one I've
> raised before - the fact that :list is separate argument and not a 
> match type.
>
> Where there is clearly some utility in being able to say stuff along the
> lines of:
>
>    header :contains :list "subject" "tag:dirty-word-list"
>
> The problem is that in order for this to work the list has to be 
> enumerable.

This wasn't the intent. I was certainly aware of the need not to 
retrieve the whole membership list.
But you are saying that my syntax wouldn't work in this case.

> Not all lists are enumerable, and even some that are are so large even 
> though oyu can enumerate them in theory you can't afford to do so in 
> practice.
>
> One use case where this matters is when the list is a set of hashed 
> values. The
> way you find if something is "on the list" is to hash it and see if 
> that value
> appears. And even if the input string is short enough that you can 
> enumerate
> all the unique substrings, how about :matches and :regex? Good luck with
> those.
>
> Hashed lists have in fact been proposed in the present discussion as a 
> means of
> avoiding giving your address whitelist to the mail server. I happen 
> not to
> think this is a useful thing to do for a variety of reasons, mostly 
> having to
> do with address canonicalization (or lack thereof), but there are other
> use cases where hashed lists make more sense.
>
> So, although it reduces functionality, I believe :list should be a 
> match type
> and the underlying comparison type that's done should be a property of 
> the list
> itself.

Hmm. You are probably right, :list as a modifier is a bit of a hack.
How would the new match type work with tests other than 
header/address/envelope?

---

From: Ned Freed <[email-masked]>
Date: Sun, 19 Jul 2009 11:46:53 -0700 (PDT)

> where I think it belongs.

Agreed.

> But you are saying that my syntax wouldn't work in this case.

It's not a question of syntax, but semantics. When a list isn't enumerable it
typically only supports some specific sort of matching that's intrinsic to the
list. In most case it's a :is match in sieve parlance, but not always.

So what you end up with are lists where some subset of
:is/:contains/:regex/:matches cannot be used. And which combinations are
and are nt allowed is going to vary from list to list.

There are basically two ways to address one. One is to say "some combinations
of :list and variaus MATCH-TYPES" are going to fail in some cases".
Alternately, we can simply say "the MATCH-TYPE is implied by the list itself,
and :list is therefore best thought of as a MATCH-TYPE".

I strongly prefer the latter. The former has least astonishment violation
written all over it. It also fails to allow for lists whose underlying match
criteria do not align with existing Sieve match types, and there are a LOT of
those; prefix match, suffix match, IP address pattern matches, etc. And before
anyone suggests defining new match types for these, allow me to point out that
in some cases the ynderlying match types allowed from some sources are open
ended and extensible.

> How would the new match type work with tests other than

Well, for example, something like

    string :list "target-string" "list-name"

would check to see if the list named "list-name" has "target-string" as
a member using whatever membership criteria the list is set up for.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 19 Jul 2009 20:25:35 +0100

Ned Freed wrote:
 [...]

> in some cases the ynderlying match types allowed from some sources are 

Would we need to have :list-is, :list-matches, :list-contains, or can we 
get away with just :list, which would check for the exact match?

---

From: Дилян Палаузов <[email-masked]>
Date: Sun, 19 Jul 2009 21:41:45 +0200

Hello,

> get away with just :list, which would check for the exact match?

As far as I understood Ned, it is the list who decides on the exact 
match type : it is not up to the script to prescribe the matching. So 
"just :list".

Any use cases where :list-is, :list-contains ... make sense and "just 
:list" would be insufficient?

Със здраве,
	дилян

---

From: Ned Freed <[email-masked]>
Date: Sun, 19 Jul 2009 13:13:56 -0700 (PDT)

> get away with just :list, which would check for the exact match?

Making the match type part of the list argument is just moving the information
around and not changing the underlying semantics. Having any specification at
aall of the match type other than "it's matching a list" is exactly what we
don't want and which cannot be supported. 

Perhaps some specific examples will make this clear. Suppose I have a list of
whitelisted addresses, but because of privacy concerns this list contains the 
SHA-1 hash of each address. The only match type such a list would support is
:is. And it's an inherent characteristic of the list that that's the only
match type.

Now suppose I  have a list of half a million or so strings I don't want to see
as part of a subject line. I've used some algorithm or other to turn this into
a data structure that will tell me if any of those substrings appear in a
single  pass over the subject. The implied match type for such a list is
:contains.

Now consider a list consisting of IP addresses and net masks. I want to check
and see if a given IP address matches anything on the list. (Assuming the list
content is properly normalized, this can be done for IPv4 with no more than 32
exact match comparisons, 128 for IPv6. Of course other, more sophisticated
approaches are also possible.) This list has an effective match type that
corresponds to nothing we currently have in sieve, yet it is a perfectly
reasonable sort of list to want to match against.

Now think for a moment what's involved with having a list that support :matches
or worse, :regex. You now have a list of patterns that have to be checked one
by one, and checking each one is pretty expensive. In other words,
enumerability is a requirement for such a list, and depending on how things are
stored, the match type may still be implied by the underlying storage
mechanism.

Again, there is no doubt that, for the subset of lists that are enumerable,
having the match type be separate makes sense and provides the most
flexibility. But many of the lists we're going to want to check aren't
enumerable, either because the underlying technology doesn't support it or
because enumeration is simply too expensive. And I really have to wonder
if allowing all the combinations when most of them are going to produce
a runtime error is the right way to design this.

And if we decide not to make :list a MATCH-TYPE, the text is going to have to
state that any particular combination of MATCH-TYPE and :list may produce a
runtime error for some subset of the available set of lists. This may even
extend to the default :is MATCH-TYPE, creating a situation where a MATCH-TYPE
has to be specified.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Sun, 19 Jul 2009 18:12:25 -0700 (PDT)

> :list" would be insufficient?

Sure, you can have lists that support more than one kind of matching, although
I expect such cases to be the exception rather than the rule. In such cases I
would propose that the type of matching be embedded in the list name somehow.

This then gets to the idea of using URLs for list names. A tag: URL is of
course completely capable of handling additional embedded material. Other
URL schemes, not so much. In fact I'm not entirely sure how this is supposed to
work with LDAP URLs with reasonable efficiency.

Consider the fairly typical case of whitelisting based on a personal address
book in LDAP. Assuming everyone is OK with users being able to construct
and perform their own LDAP searches and have no problem knowing and hard coding
the base DN of their entries (both very big ifs), this would call for something
like:

   if address :list "from"
      "ldap:///cn=me,o=pab?mail?sub?(objectclass=regularentry)" {
      ... whitelist ...
   }

The problem with this is we're again back to enumeration - that URL produces
a (potentially very large) list of addresses, which would have to be compared
one by one to the address(es) extracted from the "from" field.

But consider the URL:

  ldap:///cn=me,o=pab?mail?sub?(&(objectclass=regularentry)(mail=a@b))

This flips the problem around - now the address you're looking for is specified
in the URL itself. Now it's up to the LDAP server to find the match, and the
simple fact it returned a response is enough to conclude the address is on the
list.

This is much more efficient, especially if the LDAP server has proper indexing.
But now the URL is something that has to be constructed by knowing what you're
searching for.

I will also note that LDAP URLs can embed very complex matching criteria of
their own.

Now, I guess I have no problem with using tag: URLs that end up getting
converted to the more efficient LDAP URLs seen here. But I really have to
question whether anyone is actually going to allow, as the specification
implies, the direct use of LDAP URLs or other URLs with similar
characteristics.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 20 Jul 2009 10:41:50 +0100

Дилян Палаузов wrote:

> Hello,
>
>> Would we need to have :list-is, :list-matches, :list-contains, or can 
>> we get away with just :list, which would check for the exact match?
>

Ned has answered this nicely. I was thinking about substring matches 
(kind of simplified :matches), but this can be thought of as an 
implementation details. I.e. we can consider an external list to be 
black box.

---

From: Barry Leiba <[email-masked]>
Date: Tue, 21 Jul 2009 17:19:57 -0400

> Ned has answered this nicely. I was thinking about substring matches (kind
> of simplified :matches), but this can be thought of as an implementation
> details. I.e. we can consider an external list to be black box.

I agree with Ned completely, and Alexey sums it up perfectly: an
external list is a black box.  It was the lack of that semantic that
made me resistant to external lists before.  If :list is a match type,
that major issue goes away for me (though I still have others, but I
think they can be resolved).

Barry

---

From: Barry Leiba <[email-masked]>
Date: Wed, 22 Jul 2009 01:55:09 -0400

In my previous message I said that I had other issues with the
document.  Here they are:

Making :list a match type sorts out my issues with section 2.2, so that's fine.

But the same issues affect section 2.3.  If the list can't be
enumerated, you can't redirect to it.  Section 2.3 needs to make it
clear that whether an external list can do this or not depends upon
the list, and it needs to say what happens if it can't (ignore, or
throw runtime error seem to be the two sensible choices, and I think I
prefer the latter).

Section 2.4 says that the syntax of the list name is a URI, but it
doesn't talk about the semantics.  Along with the change to 2.2,
section 2.4 needs to change to (1) note that some URI types make sense
and some don't (Alexey already has this on his to-do list, and (2)
explain that the URI has to resolve to something that can be used as a
list in the manner specified in 2.2 and 2.3... and talk a little about
what that means.  Specifically, it has to be something that can be
given a string and can determine whether that string "is a member"...
and that it MAY also be something that can return its members as a
list or through an iterator.

Section 3, Security Considerations, needs some work, I think.  First,
I don't like that there's some normative language there that doesn't
appear elsewhere.  I prefer to have the normative parts of Security
Considerations be elsewhere in the document, in the parts that one
expects to be normative.

Second...
   Protocols/APIs used to retrieve/verify external list membership MUST
   provide at least the same level of confidentiality as protocols/APIs
   used to retrieve Sieve scripts.  For example, if Sieve scripts are
   retrieved using LDAP secured with Transport Layer Security (TLS)
   encryption, then the protocol used to retrieve external list
   membership must use a comparable mechanism for providing connection
   confidentiality.  In particular, the protocol used to retrieve
   external list membership must not be lacking encryption.

I don't think this is right.  The connection between the Sieve
processor and the Sieve script store might be very different to the
connection it has with the external list.  It's likely that it might
not be necessary to use TLS to retrieve Sieve scripts, but IS
necessary to use it to interact with the list, or vice-versa.  I'd
rather decouple them, and just say that the Sieve processor MUST use
"appropriate security" to interact with the list, and that that may
mean using TLS, proper authentication, and whatnot -- but that it also
might not... say, in a closed system where the Sieve processor acting
on behalf of a user has direct access to the user's address book,
without fear of interception.

The normative text in the "mail bombs" consideration should move to
section 2.3, though I think it's still a good idea to mention the
issue here as well.

Barry

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 22 Jul 2009 11:38:06 +0100

Barry Leiba wrote:

>enumerated, you can't redirect to it.
>
:list is not a match type in this case, again it is a black box.

>Section 2.3 needs to make it
We should discuss this in Stockholm (and on the mailing list). I wasn't 
thinking about being able to use the header test without being able to 
reference the same list in redirect.

>and some don't (Alexey already has this on his to-do list,
>
Right.

>and (2)
Ok.

>connection it has with the external list.
>
Agreed.

>It's likely that it might
>necessary to use it to interact with the list, or vice-versa.
>
Right.

>I'd rather decouple them, and just say that the Sieve processor MUST use
>"appropriate security" to interact with the list,
>
I think "appropriate security" is quite ambiguous.

>and that that may mean using TLS, proper authentication, and whatnot -- but that it also
The intent was to cover this case.
I think an internal API which doesn't use a TCP protocol underneath (for 
example) and can't be intercepted, then it is considered as secure.

But obviously your reading of this text doesn't agree with my intent, so 
I will think about rewording.

>section 2.3, though I think it's still a good idea to mention the
I am not sure this is needed. Similar text in the base Sieve spec is in 
the Security Considerations section.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 22 Jul 2009 11:42:42 +0100

> :list is not a match type in this case, again it is a black box.

So I would like to correct your statement. There is no need whatsoever 
to be able to enumerate list membership in Sieve in order to redirect.
The only requirement is that whoever does mailing list expansion needs 
to be able to do that. Such entity doesn't have to be the Sieve engine.

> We should discuss this in Stockholm (and on the mailing list). I 
> wasn't thinking about being able to use the header test without being 
> able to reference the same list in redirect.

---

From: Barry Leiba <[email-masked]>
Date: Wed, 22 Jul 2009 09:46:22 -0400

> :list is not a match type in this case, again it is a black box.
...
> So I would like to correct your statement. There is no need whatsoever to be
> able to enumerate list membership in Sieve in order to redirect.
> The only requirement is that whoever does mailing list expansion needs to be
> able to do that. Such entity doesn't have to be the Sieve engine.

Two things:
1. When I say that the list must be enumerable, I don't necessarily
mean "by the Sieve engine."  It's possible for the list not to be
enumerable at all, even by the list handler.  Examples are where the
list stores hashes of email addresses (to preserve privacy, perhaps)
and where the list stores mailbox patterns (all members of department
Z have email addresses that start with "dz", so the list of department
Z members is querieable ("Is X a member?"), but not enumerable).

2. Ah, so you're suggesting that the Sieve engine, which knows how to
do a redirect, should pass this on to the list handler, which now also
has to know how to do a redirect.  I'm not sure that's a good idea,
but I could perhaps be convinced.  I'm curious about what Ned thinks
here.

> reference the same list in redirect.

There's also the case where the list is not a list of email addresses.
 In that case, it can certainly be used for section 2.2, but not for
2.3.  2.3 still has to say what to do.  Repeating, I think it's a
"SHOULD [or maybe MUST] cause a runtime error" if the list can't be
used for redirect because of the type of list it is (which includes
the discussion above).

> I think "appropriate security" is quite ambiguous.

I don't think it's ambiguous, but, not, it's not well specified.  I
expected to put more words in to be clearer, but the point is that
it's ultimately the responsibility of the implementation to know
what's appropriate.  There's a difference between making an LDAP
request in a closed LAN that's only used for trusted servers (neither
encryption nor authentication may be needed), a firewalled LAN for a
particular company (it might be OK to skip encryption, depending upon
policy), or the open Internet (encryption and authentication probably
required), and whether the list being accessed is private or public
(no encryption or authentication needed for public data, even on the
Internet).

We can't dictate any of that in the spec... we can only give examples
and advice, and leave the final choice of what's "appropriate" to the
implementation.

> I am not sure this is needed. Similar text in the base Sieve spec is in the
> Security Considerations section.

OK.  I don't have a very strong opinion here.  I think it's harmless
and possible helpful to put the normative bits earlier in the
document.  But it's OK with me if consensus is that it doesn't matter.

Barry

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Jul 2009 07:24:52 -0700 (PDT)

> reference the same list in redirect.

Well, I can sort of see an argument that you might expect to be able to
reference any list that you used in an _address_ test in redirect, but
header? Who says the lists we'll be checking headers against are of
addresses or anything that correlates to addresses?

And in fact there are even lists of addresses (hashed lists are the obvious
example) where you won't be able to reference them in redirect.

The opposite is going to be true as well. There are plenty of use cases for a
list you want people to be able to redirect to but not test the content of in
any way.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Jul 2009 07:49:17 -0700 (PDT)

> Z members is querieable ("Is X a member?"), but not enumerable).

Exactly so. And by the same token, a list you can redirect to may not
necessarily be queryable in any other way.

> but I could perhaps be convinced.  I'm curious about what Ned thinks

Handling redirect :list by throwing the problem over the fence to a list
processors is an interesting approach. We actallly use a variant of it - the
sieve machinery performs variouos actions on the messeage but then invokes our
more general list expansion machinery to get the necessary address list.

				Ned

---

From: Barry Leiba <[email-masked]>
Date: Wed, 22 Jul 2009 12:18:32 -0400

Attached is my proposed -03 version (attached as "-03-pre01"), which
has NOT been posted as an I-D at this point.  I've sent Alexey my
edits, and the edit token is back with him.

I've made a bunch of changes, so forget DIFFs and please read the
whole thing (it's short) and comment.

Should this next version be renamed to
"draft-ietf-sieve-external-lists-00"?  In other words, should the WG
adopt it?  I think it should.

Barry

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 22 Jul 2009 22:18:06 +0200

Ned Freed wrote:
 [...]

> list you want people to be able to redirect to but not test the 

I was originally thinking about email only lists, but you and Barry 
convinced me that other types of lists are also useful.

So yes, I will update the draft.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 22 Jul 2009 22:50:49 +0200

Barry Leiba wrote:
 [...]

>adopt it?  I think it should.
Yes. This document is in our charter and my understanding is that the WG 
already made a decision to accept it.

---

From: [email-masked]
Date: Thu, 23 Jul 2009 22:06:07 +0100 (GMT)

(thanks for kicking this off: the timing is fortuitous since i planned to start implementing something similar soon. i'll start a new threads for any longer comments.) 

>> Barry Leiba wrote:

<snip>

> The opposite is going to be true as well. There are plenty of use cases for
> a list you want people to be able to redirect to but not test the content of
> in any way.

+1

- robert

---

From: Robert Burrell Donkin <[email-masked]>
Date: Thu, 23 Jul 2009 22:18:33 +0100

+1

> more general list expansion machinery to get the necessary address list.

i wonder whether the specification would be cleaner and clearer if it
allowed the sieve and list engines to collaborate in executing the
redirection in the most convenient fashion

- robert

---

From: Robert Burrell Donkin <[email-masked]>
Date: Thu, 23 Jul 2009 22:48:50 +0100

+1

the Sieve engine will face problems of semantics, resolution and
authentication with direct URLs

- robert

---

From: Ned Freed <[email-masked]>
Date: Fri, 24 Jul 2009 05:55:10 -0700 (PDT)

> authentication with direct URLs

The last item is an important point. Anyone who allows, say LDAP URls, is going
to have a fun time analyzing those URls to make sure they don't do  anything
naughty.

				Ned

---

From: Robert Burrell Donkin <[email-masked]>
Date: Fri, 24 Jul 2009 15:11:28 +0100

(apologies for messing up the reply-to foo first time 8-)

+1

i spent some time this morning thinking about the HTTP case - and it's
very interesting :-)

i have some comments related to this but i need some more thinking
time before i post them to the list (i've pushed them to github[1]
just in case anyone wants to see where they're going ATM)

- robert

[1] http://github.com/RobertBurrellDonkin/ietf-mta-filters/tree/master
follow links to ExternallyStoredLists/00