Subject: [xmpp] 3921bis - outbound presence optimisation

From: Kevin Smith <[email-masked]>
Date: Wed, 21 Jul 2010 15:35:43 +0100

In 4.4.2, there's a SHOULD on the optimisation of not sending presence
unless you received an answer to the probe that showed the user is
online. This seems a sensible optimisation, but are we sure that we
don't have timing issues giving edge cases here (and can we have text
about how to deal with them).

Example:
1) User becomes available.
2) Server sends presence probes.
3) Server waits for presence probe results and sends on the original
stanza only if the probe result was positive?
4) User sends another broadcast.
5) Does server queue up the stanza from 1) for the contacts that
haven't shown they're available yet, in case they were online and the
probe response is just slow? Does the server bin the stanza from (1)
but keep (4) queued in case it receives a new response to the probe
showing they were online, and send if it receives an online presence?

One can come up with various silly scenarios here, and I think it'd be
good to lend some text to how we think they should be handled.

/K

---

From: Philipp Hancke <[email-masked]>
Date: Wed, 21 Jul 2010 18:12:26 +0200

> about how to deal with them).

It is good in theory, but may affect reliability in negative way 
(similar to the proposed handling of errors). At least the current 
version of that optimization does not break invisible anymore (see 4.5.2).

And why optimize? Is there any known problem with the "broadcast model"?
:-p

> stanza only if the probe result was positive?

The "optimization" does not apply to the initial broadcast (which is in 
4.2.2, not 4.4.2).

> good to lend some text to how we think they should be handled.

I do not see sufficient reason to modify the current behaviour unless 
someone explains the rationale, gains and risks of that optimization.

cheers

philipp

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 28 Jul 2010 00:06:07 +0200

[Typed while offline, messages will be delayed.

> stanza only if the probe result was positive?

No! This optimization is only for subsequent presence broadcast, not
initial presence. The server would *always* send initial presence
without waiting for or paying attention to the results of a presence
probe. The text in 4.4 ("Subsequent Presence Broadcast") says:
                         ^^^^^^^^^^

   As an optimization, if the subscription type is "both", then the
   server SHOULD send subsequent presence notifications to a contact
                      ^^^^^^^^^^
   only if the contact is online according to the user's server.  That
   is, if the user's server never received a positive indication that
   the contact is online in response to the presence probe it sent to
   the contact or if the last presence stanza it received from the
   contact during the user's presence session was of type "unavailable",
   the user's server SHOULD NOT send subsequent presence notifications
                                     ^^^^^^^^^^
   from the user to the contact.

> 4) User sends another broadcast.

How quickly? I suppose it is possible for the user to send initial
presence and then immediately send subsequent presence, without having
ever received an answer to the probe.

> 5) Does server queue up the stanza from 1) 

The server *always* sends initial presence to the contact.

> for the contacts that

I see two scenarios:

1. Contact is offline when user generates initial presence. User's
server sends initial presence to contact, user's server sends probe to
contact, contact's server sends nothing, user generates subsequent
presence, user's server does not send subsequent presence to contact,
contact comes online, contact's server probes for presence, user's
server returns subsequent presence to contact.

2. Contact is online when user generates initial presence but contact's
server is slow about responding to the presence probe. User's server
sends initial presence to contact, user's server sends probe to contact,
contact's server sends nothing (yet), user generates subsequent
presence, user's server does not send subsequent presence to contact,
contact does nothing further, contact's server finally answers presence
probe, contact's server does nothing further, user's server does nothing
further.

So you're right, in this scenario the contact will have received the
initial presence only and not the subsequent presence (thus the contact
will be out of date).

Given these considerations, we might want to change SHOULD to MAY in the
quoted text, as follows:

   As an optimization, if the subscription type is "both", then the
   server MAY send subsequent presence notifications to a contact only
   if the contact is online according to the user's server.  That is, if
   the user's server never received a positive indication that the
   contact is online in response to the presence probe it sent to the
   contact or if the last presence stanza it received from the contact
   during the user's presence session was of type "unavailable", the
   user's server MAY opt to ned send subsequent presence notifications
   from the user to the contact.  This optimization helps to save
   bandwidth, since most presence subscriptions are bidirectional and
   many contacts will not be online at any given time.

/psa

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 28 Jul 2010 00:16:47 +0200

[Typed while offline, delivery will be delayed.]

> version of that optimization does not break invisible anymore (see 4.5.2).

Invisible is evil.

> And why optimize? Is there any known problem with the "broadcast model"?

Why send all those subsequent presence updates if you don't have to?

> The "optimization" does not apply to the initial broadcast (which is in

Correct.

> someone explains the rationale, gains and risks of that optimization.

For a typical user, only something on the order of 10% - 20% of the
contacts are online at any one time. For example, I have ~2650  people
in my roster, but only ~350 of them are online at most times. Let's say
I have a long-lived presence session (e.g., 10 hours). For a simple
presence session (send initial, send subsequent, send unavailable), the
optimization would save ~2300 presence notifications. For a long and
more compliex presence session (send initial, change my presence 5
times, send unavailable), this presence optimization would save 11500
presence packets. That's starting to sound significant. But of course we
need to weigh the costs and benefits.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Philipp Hancke <[email-masked]>
Date: Wed, 28 Jul 2010 12:06:39 +0200

>> version of that optimization does not break invisible anymore (see 4.5.2).

Actually not. It is a result of a presence model (one group) which is 
too simplified, see
http://www.slideshare.net/padday/the-real-life-social-network-v2

> Why send all those subsequent presence updates if you don't have to?

Reliability. What about the scenario where two users are online but 
don't send each other presence updates because of that optimization?

> need to weigh the costs and benefits.

compression does not help? *scnr*

Running the models from the old presence scaling draft for non-optimized 
and optimized yields the following results (B3, total number of stanzas):

scenario   3921 / optimizized behaviour
#1:     2040000 /    1120000
#2:    10200000 /    5600000
#3:    15060000 /    2640000
#4:   150600000 /   26400000
#5: 19680000000 / 4640000000

This scales with the number of online contacts (obviously), which is
better than the number of contacts. However, remote fan-out ("smart
presence", "stanza repeaters", "view sharing" etc) scales with the 
number of contact domains, which is even smaller. This results in 
276000000 stanzas exchanged for scenario 5.

But we've had that discussion many times before :-)

philipp

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 28 Jul 2010 12:47:17 +0200

> http://www.slideshare.net/padday/the-real-life-social-network-v2

Actually, it is evil, but that's out of scope for this Internet-Draft.

> don't send each other presence updates because of that optimization?

The only example that's been provided so far seems like an edge case or
a fairly small-probability race condition. I grant that I might be wrong.

> scenario   3921 / optimizized behaviour

Indeed. :) I don't think that's a WGLC issue, though.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Philipp Hancke <[email-masked]>
Date: Wed, 28 Jul 2010 13:02:56 +0200

Peter Saint-Andre wrote:
[...]
> a fairly small-probability race condition. I grant that I might be wrong.

I think that is correct. It might be possible to avoid that condition by 
specifiying that the optimization should not be used if the initial 
probe bounced with type=error (i.e. because the s2s connection could not 
be established).

[snip]
> Indeed. :) I don't think that's a WGLC issue, though.

Ok, back to topic. Why is this a SHOULD if it is an optimization?

cheers

philipp

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 28 Jul 2010 14:18:32 +0200

> probe bounced with type=error (i.e. because the s2s connection could not

The issue that Kevin raised was a timing problem, not an error.

> Ok, back to topic. Why is this a SHOULD if it is an optimization?

I proposed changing SHOULD to MAY. I now think the optimization is truly
optional.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Philipp Hancke <[email-masked]>
Date: Wed, 28 Jul 2010 16:01:34 +0200

> The issue that Kevin raised was a timing problem, not an error.

I was thinking about a different problem when the s2s connection between 
two servers is not working when both users come online but is 
reestablished later. Since the initial presence and probe were bounced 
by the local servers, users will not include the contacts in their 
subsequent broadcasts. OTOH, they will not do so with 3921 behaviour 
either due to the old jabberd idea of not sending further updates if an 
error was received.

> I proposed changing SHOULD to MAY. I now think the optimization is truly

WFM.

---

From: Kevin Smith <[email-masked]>
Date: Wed, 28 Jul 2010 17:34:59 +0100

>>>>>> Why send all those subsequent presence updates if you don't have to?

Well, sure - what I'm asking is: Do we know for sure that we don't
have to? That we couldn't be failing to send the stanza we need to due
to either a timing issue, or a temporary loss of service.

>>>> a fairly small-probability race condition. I grant that I might be

It's certainly not the usual case - but do we know that it's an
unusual edge case or race condition? I'm concerned that by the time
you've got a few thousand servers all talking to each other, and a
healthy number of millions of users, these edge cases might be popping
up more often than one might like.

> I was thinking about a different problem when the s2s connection between two
> servers is not working when both users come online but is reestablished
> later. Since the initial presence and probe were bounced by the local
> servers, users will not include the contacts in their subsequent broadcasts.
> OTOH, they will not do so with 3921 behaviour either due to the old jabberd
> idea of not sending further updates if an error was received.
>
>>> [snip]
>>>>>

I'm not convinced yet, but I'm happy to bow to consensus if consensus
is that I've misunderstood, and these missing presence events
can't/won't happen. Is that the belief?

If these events are going to happen, but services can't afford the
bandwidth to avoid them, would a possible approach be to specify the
'correct' approach in 3921bis, and publish another document (here or
in the XSF) describing Lossy Bandwidth Saving Practices for XMPP

/K

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 28 Jul 2010 18:52:29 +0200

> to either a timing issue, or a temporary loss of service.

Unclear. I expect we'll discuss this in the meeting on Thursday, and of
course on the list as well.

> up more often than one might like.

Data would help, from those who have deployed servers that implement
this functionality.

> can't/won't happen. Is that the belief?

TBD.

> in the XSF) describing Lossy Bandwidth Saving Practices for XMPP

Quite possibly. We'll put that on the table during the meeting on Thursday.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/