Subject: Re: [xmpp] 3921bis - not responding to probes.

From: Jonas Lindberg <[email-masked]>
Date: Thu, 5 Aug 2010 14:12:11 +0200

On Thu, Jul 29, 2010 at 6:36 PM, Dave Cridland <[email-masked]> wrote:

> On Wed Jul 28 18:08:52 2010, Kevin Smith wrote:
>
>> On Mon, Jul 26, 2010 at 9:20 AM, Jonas Lindberg <[email-masked]>
>> wrote:
>> > Responding to presence probes for unavailable users would add an extra
>> > ~15% presence traffic in our use case, which is pretty significant.
>> > The percentage to be much larger if one also accounts for the "other
>> > optimization". (Disclaimer: this number is based on a quick back of
>> > the envelope calculation, so I reserve the right to have made
>> > errors..)
>> >
>> > Perhaps more interesting than that number is considering the cost of
>> > processing initial presence for users with large contact lists. Take
>> > the scenario where a user with a 1000 subscribed friends logs in. Say
>> > 20% of these friends are connected and present. With the optimization,
>> > servers responding to the probes need to process 200 probes to online
>> > friends and send responses. The other 800 probes can be dropped as
>> > soon as the receiver is identified as offline. But without the
>> > optimization, 400% more probes need to be responded to, which in
>> > addition to the cost of sending a response also may mean having to
>> > load (or keep) policy and roster data for each of these offline users
>> > in memory, to run business logic figuring out what the response should
>> > look like (unavailable/unsubscribed) etc.
>>
>> Thanks. I'm digesting.
>>
>
> Actually, you need to send those unsubscribed whether or not the user is
> online, it's the first step in 3921bis. This means you already have to load
> in the rosters. That's a MUST, incidentally - so you're also arguing against
> that.
>

That is a good point. I'd be interested in discussing that as well. It would
be much preferable from a scaling point of view if the protocol only
required loading state for users that are currently online, rather than all
users that have a friend that is online, since the latter set is likely much
larger.

The other point that was not brought up in this email thread is that making
this change likely would increase start-up time for existing clients. As an
anecdotal data point, the Xmpp desktop client I currently use (which doesn't
set any clever server side presence filters) takes ~5 seconds from login to
becoming useful, which seems to be when most of the initial presence storm
has subsided. I wouldn't be surprised if that time increased substantially
with also receiving a much larger set of unavailable presence on login.
Likely a bigger issue in scenarios with less available bandwidth, like cell
phone based xmpp clients (where of course battery is also a concern).

Also, thanks to PEP, servers currently maintain quite a substantial chunk of
> state for remote contacts which needs cleanup from time to time. If probes
> are unanswered, then we have to have timers per contact to decide if they're
> probably offline, and remove their subscriptions.
>
> There's similar cleanup in other places.
>
> For directed presence, as used in MUC, this is particularly important, as a
> directed probe is our sole means of seeing whether a full jid remains online
> or not. (Note that this is somewhat different - the response to the MUC
> probe is either unsubscribed or some presence). Again, this can be done with
> timers, but again, it feels very more more hacky.
>
> Essentially, a response to a probe allows consistent behaviour in a number
> of areas where otherwise heuristics, or degraded behaviour, have to be
> currently deployed.
>
>
> Dave.
> --
> Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]<[email-masked]>
>  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
>  - http://dave.cridland.net/
> Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
>

---

From: Kevin Smith <[email-masked]>
Date: Fri, 13 Aug 2010 14:15:42 +0100

So, I finally got around to replying to this...

On Wed, Jul 28, 2010 at 6:08 PM, Kevin Smith <[email-masked]> wrote:
> On Mon, Jul 26, 2010 at 9:20 AM, Jonas Lindberg <[email-masked]> wrote:
>> On Sat, Jul 24, 2010 at 11:59 AM, Peter Saint-Andre <[email-masked]> wrote:
>>> [Typed while in transit, delivery will be delayed.]
>>>
>>> On 7/21/10 10:33 AM, Kevin Smith wrote:
>>>> On Wed, Jul 21, 2010 at 5:01 PM, Peter Saint-Andre <[email-masked]> wrote:
>>>>> On 7/21/10 9:05 AM, Jonas Lindberg wrote:
>>>>>> I disagree per previous discussion on this subject. This is an extremely
>>>>>> useful optimization for large scale services.
>>>>>
>>>>> Yes, we had discussion about this some months ago and my sense of the
>>>>> list was SHOULD, not MUST. However, we might want to explain more fully
>>>>> why a service would not respond to probes.
>>>>
>>>> Ok - do we know how much this saves us, compared to the other
>>>> optimisation (not requesting presence when we already know it) that's
>>>> pretty much excluded by doing this? Given that you've already got the
>>>> s2s open by the point that you don't reply, you've already received a
>>>> probe, and already processed the probe, it doesn't seem to me that not
>>>> sending the reply if it's for an offline user gains much. Does anyone
>>>> have any metrics?
>>>
>>> I don't have metrics, but perhaps Jonas does.
>>

Right, so it's clear that it's important for at least Google. Given
this, I changed my stance to that of my suggestion from the meeting,
that bis stays saying you SHOULD reply to probes, and doesn't mention
the optimisation, and that we then write a spec (probably XEP) that
proposes the optimisation, and discusses the implications of it - as
it has knock-on-effects for correctness of state out in the wild.

/K

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 16 Aug 2010 13:57:15 -0600

On 8/13/10 7:15 AM, Kevin Smith wrote:
> the optimisation, 

I think that's what we have in -11 now, but if we are going to change
MUST to SHOULD then I think we want to do that here, too:

   1.  If the contact account does not exist or the user is in the
       contact's roster with a subscription state other than "From",
       "From + Pending Out", or "Both" (as defined under Appendix A) and
       the contact has not sent directed presence to the user (as
       defined under Section 4.6), then the contact's server MUST return
       a presence stanza of type "unsubscribed" in response to the
       presence probe....

That is, change this:

   MUST return a presence stanza of type "unsubscribed"

to:

   SHOULD return a presence stanza of type "unsubscribed"

> and that we then write a spec (probably XEP) that

Feel free. :)

Peter

-- 
Peter Saint-Andre
https://stpeter.im/