Subject: Re: notification options

From: "Cyrus Daboo" <[email-masked]>
Date: Tue, 24 Feb 1998 15:54:34 -0500

--On Tue, Feb 24, 1998 1:43 PM -0700 "Lyndon Nerenberg" <[email-masked]>
wrote: 

> A related problem that I've been mulling over is how to handle 
> notifications for a large set of folders (imagine someone monitoring 
> for new articles in several hundred folders mapping Usenet newsgroups). 
> It's impractical to think about opening an IMAP connection for each of 
> those n-hundred folders you want to moniter. Having a single (or a few) 
> IMAP connections poll through the folder list would be even worse, from 
> a server load standpoint.
> 
> One solution I've been considering is the creation of a new 
> light-weight protocol, sitting on a seperate port, to be used for 
> notifications. The client would issue a list of folder names it wanted 
> notifications for, then sit on the stream waiting for newmail 
> notifications from the server. Load on the server should be minimal 
> (vastly less than that of starting a couple of hundred IMAP sessions), 
> and the scaling would most likely be limited by the available network 
> bandwidth (i.e. don't monitor 2000 busy folders over a dialup PPP link).
> An added benefit is you get the notifications as fast as the server can 
> send them (ala IDLE) without the polling delay.
> 
> If there's interest in such a beast I can rough out a protocol spec and 
> mail it around.

Potentially a good idea but why not use 'MAKE CONTEXT' on a mailbox dataset
in ACAP and have automatic notification of changes? I guess this is
something we can discuss at the ACAP meeting in two days time!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Cyrus Daboo                                  mailto:[email-masked]
Cyrusoft International, Inc.                   http://www.cyrusoft.com
Voice: +1 412 605 0499                            fax: +1 412 605 0705
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     Proud Purveyors of Mulberry: Internet Mail from the Ground Up
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---

From: Lyndon Nerenberg <[email-masked]>
Date: Tue, 24 Feb 1998 14:41:13 -0700 (MST)

On Tue, 24 Feb 1998 15:54:34 -0500 Cyrus Daboo <[email-masked]> 
wrote:

> in ACAP and have automatic notification of changes?

Because I want something lightweight. For a simple single-purpose "new 
mail has arrived" notifier ACAP is overkill. (And the IMAP server may 
not support ACAP.)

Which is not to say the ACAP scheme is without merit, it's just a bit 
much for this specific application.

--lyndon

---

From: Mark Delany <[email-masked]>
Date: Wed, 25 Feb 1998 08:05:44 -1000

>much for this specific application.

If you want to go really lightweight and probablistic you could do something 
like this:

1. Have an array of counters - most likely in memory

2. hash the mailbox/user/whatever to form an index into this array

3. Whenever a mailbox changes, the server making the changes does the hash
and increments the counter. (A variation might be the last modification
time rather than a counter)

The notification protocol uses this to give the client a hint that the 
mailbox has probably changed. Hash the mailbox names and see if the counter 
has changed. If so, go to IMAP and check to be certain.

Whether the client initiates the check each time or presents a list of 
mailbox names to a notifier that periodically checks is irrelevant. Mind 
you, having a notifier maintain a potentially large state and large number 
of connections doesn't bode well.

The advantage of this strategy is that the overhead is very low regardless 
of the number of deliveries and whether a lot or a few clients are using the 
notifier mechanism. If the hash table is sufficiently sized, then false 
positives will be low.

Another advantage is the security implications. You almost certainly don't 
need to authenticate the client if they present a hash rather than a mailbox 
string. It does mean that you need to standardize the hash as part of the 
protocol of course, but that's easy.

It could then be a trivial UDP protocol. The request packet has a series of 
hashes, the reply packet has a series of corresponding date/time modified 
values.

Mind you it does imply that delivery and notification are centralized, which 
is not a fantastic assumption these days...

Regards.

---

From: Lyndon Nerenberg <[email-masked]>
Date: Tue, 24 Feb 1998 15:33:20 -0700 (MST)

> time rather than a counter)

Argh. This is a BIFF client. Simple. Simple. Simple. Keep it simple.

> has changed. If so, go to IMAP and check to be certain.

If you have to go to IMAP then there's no point in having the 
notification protocol. The object is to avoid IMAP (and it's associated 
overhead, in the client and on the server load) at all costs.

> of connections doesn't bode well.

The current mechanism requires an IMAP connection per mailbox being 
watched, or less connections doing mailbox polling. Both activities are 
resource intensive on the server. OTOH, providing the services for the 
notifier daemon could be accomplished using a single shared memory 
segment to maintain the requisite mailbox status. As things get 
delivered into mailboxes the corresponding entry in the SHM segment 
updates, and the notifier is given a poke. It runs the list and updates 
clients as appropriate. With reasonable care in coding a single 
multi-threaded daemon should be capable of servicing several hundred 
connections.

> notifier mechanism. If the hash table is sufficiently sized, then false 

In this case, false positives are not acceptable, for two reasons. The 
first is there can be no fallback to IMAP. The second is that you must 
know the exact mailbox name in order to apply ACL checks.

> string.

On the contrary. I *must* authenticate the user (and know the mailbox 
names) if I'm going to enforce ACLs.

> It could then be a trivial UDP protocol.

If it was unauthenticated, yes. Since we need authentication it's going 
to be a TCP channel.

> is not a fantastic assumption these days...

It's assumed that the notifier service runs on the same machine as the 
corresponding IMAP service (since one of the major goals of the 
notifier is to reduce the load on that IMAP server.)

--lyndon

---

From: Rob Earhart <[email-masked]>
Date: Mon, 2 Mar 1998 16:20:31 -0500 (EST)

> mail has arrived" notifier ACAP is overkill. (And the IMAP server may 

  The IMAP server may not support any particular special-purpose
notification protocol.  ACAP has the functionality, though, and it's
reasonably lightweight for what you're doing: after authenticating,
you issue a single search command, and let the notifications come in
(with an occasional NOOP if the server tears down idle connections).
What could be simpler?

  )Rob

---

From: Aidan Cully <[email-masked]>
Date: Mon, 2 Mar 1998 17:20:16 -0500

> overhead, in the client and on the server load) at all costs.

Note that I really hate the zephyr implementation, but isn't something
like zephyr what you're after?  Your biff client would subscribe itself
to MAIL:NOTIFICATION messages, or something ridiculous like that, and
your sendmail would invoke a zwrite-sort-of-thing to send notification
of this message to you.

> If it was unauthenticated, yes. Since we need authentication it's going 

We've hacked zhm/libzephyr.a here to authenticate over a TCP channel with
IDENT (for our shared servers), but by default it uses kerberos.

But I don't like zephyr.  The code is ridiculously modular.  I would
almost go as far as suggesting that you write your own version rather
than use the MIT.

--aidan
-- 
Aidan Cully       "You can lead a whore to culture, but you can't make her
Panix Staff        think."
[email-masked]        --Dorothy Parker