Subject: RE: [lemonade] Text on server to client notifications (per Paris AI).

From: [email-masked]
Date: Sat, 17 Sep 2005 12:42:16 -0500

There is an issue of scalabillity when it comes to notifications.

If we let a notification server do all of the filtering, the number of
messages that get sent ot the notification server from the mail server
can number in the thousands per hour for a large enterprise.
One of the reasons why email servers are so cost effective is that they
are idle storage most of the time.  Messages are delivered into the mail
server at a very low processing cost, but accessing the messsages with
the MUA is much more expensive (client operations are involved,
obviously).

Since most users check their mail about 4 times a day, each account is
active for only a few minutes a day.  Even in an enterprise setting
where the clients continuously poll the server, the polling is
aggregated and takes a few seconds, so it's probably still less than an
hour a day of traffic.
Now add the additional operation of sending events to a notification
server for messages and the email server will be active continusouly,
sucking up bandwidth for little or no return because the notification
server will do the filtering.

I would immagine that *some* filtering must be done in the MDA in order
to control the message flow.  SIEVE will get us most of the way there by
allowing us to control which folders we want to triger events from.
For example, have SIEVE filter messages I want to be notified about into
a "NOKIA 6260" folder that has a NOTIFY annotation and have that folder
trigger messages on a delta to my Nokia 6260 phone (shameless plug).
This will greatly reduce the traffic on the line for messages that will
most likely never make it out the notification server.

When thinking about notifications think about the super scalar problem.
For example, how would a super large scale system like Gmail with
millions of users implement something like this and still maintain
bandwidth at a reasonable cost.  If we can solve for this super scalar
problem, then we can solve for the small office as well.

Corby Wilson

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of ext Erev Ari
> Sent: Thursday, September 15, 2005 03:59 AM
> To: Mark Crispin; Stephane H. Maes
> Cc: [email-masked]
> Subject: RE: [lemonade] Text on server to client 
> notifications (per Paris AI).
> 
> 
> On Thu, 15 Sep 2005, Mark Crispin wrote:
> > 
> > IMAP should NOT be extended to be a notification service.  There is 
> > nothing in IMAP's scope that enables it to be a notification 
> > service.  It 
> > would be very strange to tell a POP3 or NNTP client that it 
> > must support 
> > IMAP in order to obtain notifications.
> > 
> > However, there is no reason why an IMAP server can not be a 
> > client of a 
> > notification service.  The effective result seen by an IMAP client 
> > programmer would be the same.
> > 
> > Put another way: you will get strong pushback from IMAP 
> > server vendors 
> > against any proposal that requires IMAP to be a notification 
> > service.  You 
> > will not get such negative pushback on a separate 
> > notification service 
> > that (among other things) happens to be accessible via IMAP.
> > 
> 
> I tend to agree with Mark. 
> 
> Actually, I even go further than that.... Which brings me to the
> following question:
> 
> Do we really need the "Server to Client" notification protocol be
> implemented *by the IMAP server* ???
> 
> I doubt it.
> Wouldn't it be better to have the IMAP server just implement a simple
> "Server to Server" protocol and have another entity ("the notification
> server/service") implement all the rest (which is not a trivial task,
> and is quite different from what IMAP servers usually do) ?
> 
> I think, that this is a very reasonable conclusion from 
> Mark's comment.
> 
> And in fact, (surprise, surprise...) this is the 
> "architecture" that was
> presented more than 3 years ago to Lemonade - where the IMAP 
> server just
> uses a "Server-to-Server" protocol and another entity is doing all the
> rest.
> 
> And this "rest" is not that obvious....
> 
> - Simple and complicated Filters (for example, event types 
> which are not
> specific to IMAP).
>   (complicated filters are actually logic applied to filters 
> such as "if
> the result of this filter is X - do Y)
> 
> - Delayed/Scheduled notifications 
>   (you do not necessarily want to be wakened in the middle of 
> the night
> just because a wrothless message arrived in your inbox).
> 
> - Logic for deativation rules.  
>   (deactivation is used to "remove" flashing icons on your 
> handset, and
> may involve finding out what is the
>   handset status (on, off, busy, reachable)
> 
> - Supports multiple transports 
>   (also outside the "IETF transport protocols") - such as SMS, and
> related non-IETF protocols which are very common in mobile ceullar
> communications nowdays).
> 
> In today's mobile/ceullar world all these features are percieved by
> Telecom service providers and users as basic requirements and are
> functioning in real networks.
> 
> Do we want all this logic to be handled by the IMAP server? 
> And if so - will part of these be (re)implemented by other messaging
> (and non-messaging) servers? 
> Are IMAP servers going to implement this?
> 
> On the other hand, the "server to server" architecture assumes a
> functional separation:
> 
> Messaging (and other) servers send rather-simple triggers to the
> notification server. 
> (they usually do *NOT* support a complex filter mechanism).
> 
> The notification server/service is the one responsible for processing
> the triggers, and deciding (based on filters, logic, etc.) 
> which type of
> notification should be sent, when, and on which channel.
> 
> This functioanl separation allows each server to develop in 
> its own pace
> and to concentrate on its own "problem".
> 
> >From a practical/maintenance point of view, I should also 
> mention, that
> combining messaging and notification in one server is a hard task to
> accomplish and *maintain* over time successfully.
> My real-life experience shows that although it is feasible, 
> it has many
> disadvantages compared to the "de-coupled" model.
> 
> Best regards,
> Ari Erev
> [email-masked]
> 
> _______________________________________________
> lemonade mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/lemonade
> 

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: "Stephane H. Maes" <[email-masked]>
Date: Sat, 17 Sep 2005 18:21:26 -0700

I am not sure why notification would have to be limited to what is captured in a sieve rule.

At least doing so would imply deciding from the only that the client is made aware of the event but no operational gains results from it.

I believe it is possible to provide such gains.

That said, it does not imply that the IMAP server would become a notification server. Just that if notification information (via another channel) is sent it may not have to be sent through the usual IMAP band.

Stephane

_____
Stephane H. Maes, PhD,
Director of Architecture - OCS & Mobile, Oracle Corporation.
Ph: +1-203-300-7786 (mobile/SMS); Fax / Office UM: +1-650-607-6296.
e-mail: [email-masked]
IM: shmaes (AIM/Y!/skype) or [email-masked] (MSN Messenger)

-----Original Message-----
From: Rob Siemborski [mailto:[email-masked]] 
Sent: Saturday, September 17, 2005 6:16 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [lemonade] Text on server to client notifications (per Paris AI).

> the MUA is much more expensive (client operations are involved,

Am I missing something?  Wouldn't the load on the notification server
be just the same as the load on the "mail server" from incoming
messages?  (are "notifications" being delivered for something else
that happens more frequently than incoming mail)?

> server will do the filtering.

If it did no filtering, the mail server would just be adding traffic
when a message is delivered, right?  So, if adding notifications makes
it active continuously, then that's only because it was already active
continuously from mail deliveries.

> most likely never make it out the notification server.

This basic approach I agree with.  Notification should be a job of the
MDA, not the IMAP server.  In fact, you shouldn't even need some magic
folder annotation to do this, just use the sieve notify extension (or
something similar) combined with a sieve filter in the MDA.

Or maybe I'm missing why we are trying to tie this notification
service to IMAP.  Are we looking to send notifications when someone
issues an APPEND into a folder too?  Is this really such a frequent
occurance that its a problem that users have?

> problem, then we can solve for the small office as well.

No comment. :)

-Rob

--
Rob Siemborski        |     PGP:0x5CE32FCC    |     [email-masked]
Software Engineer     |                       |            650-623-6925

---

From: Rob Siemborski <[email-masked]>
Date: Sat, 17 Sep 2005 22:27:21 -0700

> I am not sure why notification would have to be limited to what is captured in a sieve rule.

In the system Corby described, the sieve rule just decides when to
send a notification.  Presumably if the information contained in the
current notification systems isn't enough, that could be changed.  Ari
did describe a set of features that probably isn't handled well just
by filtering though.  Personally, I don't think this is something the
mail server should address (it should just send the necessary data to
the notification server which can deal with these things).

> At least doing so would imply deciding from the only that the client is made aware of the > event but no operational gains results from it.

I don't see why this is the case.

> That said, it does not imply that the IMAP server would become a notification server. Just > that if notification information (via another channel) is sent it may not have to be sent
> through the usual IMAP band.

Of course not.  We already have notifications that exist within the
IMAP band.  They're called EXISTS responses.

Am I misunderstanding?  Aren't we looking for a system to notify the
mobile device out of the IMAP (or POP or NNTP or whatever) band that
new messages have arrived?  I'm confused by Corby's message that the
number of notifications here is somehow difficult to deal with (given
that we already have the assumption that there is a mail server that
has to receive this volume of mail).

-Rob

---

From: "Stephane H. Maes" <[email-masked]>
Date: Sat, 17 Sep 2005 22:33:48 -0700

So why can't we consider some of the inband notifications to be made available outband?

Stephane

_____
Stephane H. Maes, PhD,

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Rob Siemborski
Sent: Saturday, September 17, 2005 10:27 PM
To: Stephane H. Maes
Cc: [email-masked]
Subject: Re: [lemonade] Text on server to client notifications (per Paris AI).

-Rob

---

From: Rob Siemborski <[email-masked]>
Date: Sat, 17 Sep 2005 23:13:42 -0700

> So why can't we consider some of the inband notifications to be made available outband?

Ok, so the goal is to handle notifications like flag changes and
expunges too then.  Seems like significant added complexity for
something that is unlikely to happen without action on the user's
part.

But at least I understand the goal now.

-Rob

---

From: "Stephane H. Maes" <[email-masked]>
Date: Sat, 17 Sep 2005 23:20:04 -0700

Rob,

I think the user action is not needed if the client is "programmed" to appropriately react.

I think one of the value for example of push mail is to have such processes optimized as well as handled without user's intervention.

Stephane

_____
Stephane H. Maes, PhD,

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Rob Siemborski
Sent: Saturday, September 17, 2005 11:14 PM
To: Stephane H. Maes
Cc: [email-masked]
Subject: Re: [lemonade] Text on server to client notifications (per Paris AI).

-Rob

---

From: Rob Siemborski <[email-masked]>
Date: Sat, 17 Sep 2005 23:35:16 -0700

> I think the user action is not needed if the client is "programmed" to appropriately react.

I think we're talking about different things.  I'm looking at the
*cause* of the event that triggered the notification:

If the events are:
- Mail arrives
- Mail flags or annotations change
- Mail is expunged

Only the first frequently happens without user intervention.  Most
people don't have bots managing their mail for them at random times
other than when mail arrives.  So, I contend that most of the "flags
change" and, to an only slightly lesser extent "mail is expunged"
events happen due to a direct user action by the user that owns the
folder (let's exclude shared folders which don't appear frequently in
practice).  Thus, the user knows about them already and doesn't really
need to be alerted to them.

Obviously a notification client or server receiving one of those
notifications is free to react however it chooses.

Is the use case then "maintain a folder index without keeping an IMAP
session up, knowing that the folder index may be out of sync with what
the client sees once the IMAP session is brought up"?

-Rob

---

From: "Stephane H. Maes" <[email-masked]>
Date: Sat, 17 Sep 2005 23:52:02 -0700

Rob,

Not only that way in the use cases and user experience requirements that we have seen for mobile. For example, this may also happen through another client (desktop) and the user expects to see it reflected ASAP on the mobile client (if preference / filters set as such).

Stephane

_____
Stephane H. Maes, PhD,

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Rob Siemborski
Sent: Saturday, September 17, 2005 11:35 PM
To: Stephane H. Maes
Cc: [email-masked]
Subject: Re: [lemonade] Text on server to client notifications (per Paris AI).

If the events are:
- Mail arrives

-Rob

---

From: Rob Siemborski <[email-masked]>
Date: Sat, 17 Sep 2005 18:15:51 -0700

No comment. :)

-Rob

---

From: Anil SRIVASTAVA <[email-masked]>
Date: Mon, 19 Sep 2005 08:11:39 -0700 (Pacific Daylight Time)

On 2005-09-17/18:15 [-0700], [email-masked] [Rob Siemborski] wrote:

> 
> > I would immagine that *some* filtering must be done in the MDA in 
> > order to control the message flow.  SIEVE will get us most of the 
> > way there by allowing us to control which folders we want to triger 
> > events from. For example, have SIEVE filter messages I want to be 
> > notified about into a "NOKIA 6260" folder that has a NOTIFY 
> > annotation and have that folder trigger messages on a delta to my 
> > Nokia 6260 phone (shameless plug). This will greatly reduce the 
> > traffic on the line for messages that will most likely never make it 
> > out the notification server.
> 

I believe the answer to above is yes.  The system should work of all 
kinds of system events and missing events generated as a result of an 
APPEND into a folder would miss the following use case:

 - exec has admin managing their account
 - admin triages the e-mail (much better than any pre-defined SIEVE rule 
   will ever do) and files away message admin believes exec needs to 
   respond to into folder call "urgent"
 - that must generate an event and eventually make the message available
   on the execs mobile device

Anil

> 
> > When thinking about notifications think about the super scalar 
> > problem. For example, how would a super large scale system like 
> > Gmail with millions of users implement something like this and still 
> > maintain bandwidth at a reasonable cost.  If we can solve for this 
> > super scalar problem, then we can solve for the small office as 
> > well.
> 
> No comment. :)
> 
> -Rob
> 
> --

-- 
_______________
Anil SRIVASTAVA
[email-masked]