Subject: Re: [xmpp] "in-order processing"

From: Dave Cridland <[email-masked]>
Date: Fri, 01 Oct 2010 09:41:20 +0100

That was an excellent summary, thanks.

The only thing I'd add is that the approach I suggested appears (to  
me) to be compatible with both the "bare jid ordering" and the  
specific case of MUC traffic ordering which started the debate.

Essentially, MUC operates by having a service on a domain, bare jids  
used for rooms, and full jids used for occupants of those rooms.

Various traffic within a room (ie, from multiple different full jids)  
essentially relies on being ordered (at least relative to other  
traffic from the bare jid of the room). This in turn means that if  
ordering is specifically only reliable for full jid pairs, then MUC  
no longer works - this is obviously undesirable.

Ordering to a bare jid is quite tricky - this may result in trying to  
synchronize delivery between two streams, and if one fails (and a  
redelivery occurs, for example due to XEP-0198) then out-of-order  
delivery is fairly unavoidable, except by drastic action (such as  
awaiting a XEP-0198 ack from one stream prior to sending on another).

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Alexander Holler <[email-masked]>
Date: Fri, 01 Oct 2010 14:12:23 +0200

Hello,

Am 01.10.2010 08:42, schrieb Justin Karneges:
> Instead of ordering by bare JID pairs or full JID pairs, the above text
> suggests ordering by streams at each hop.  For example, if stanzas directed to
> [email-masked] and [email-masked]/resource would both be sent to the same
> client stream, then these stanzas should be delivered in order to that stream.
>
> The only critique of this approach has been that ordering by stream, instead
> of by bare JID, may end up producing unordered receipt if client connections
> disconnect and stanzas end up getting routed to alternate resources or offline
> storage.  Dave believes it is not practical to try solving this.

Ordering by bare JIDs isn't practical too if more than one server is 
serving a domain. E.g. if user@domain/res1 is connected to 
server1@domain and user@domain/res2 is connected to server2@domain.

Regards,

Alexander

---

From: Dave Cridland <[email-masked]>
Date: Fri, 01 Oct 2010 15:54:57 +0100

On Fri Oct  1 13:12:23 2010, Alexander Holler wrote:
> Ordering by bare JIDs isn't practical too if more than one server  
> is serving a domain. E.g. if user@domain/res1 is connected to  

No, that shouldn't have an effect on anyone but the server for  
"domain", surely? (And for that server, it's essentially the same  
problem as for two resources connected to the same server).

Dave.

---

From: Alexander Holler <[email-masked]>
Date: Fri, 01 Oct 2010 17:18:37 +0200

Am 01.10.2010 16:54, schrieb Dave Cridland:
> No, that shouldn't have an effect on anyone but the server for "domain",
> surely? (And for that server, it's essentially the same problem as for
> two resources connected to the same server).

I've tried to express that the promise of XCP "ordering by bare JID" is 
useless.
If two stanzas will get send out over two streams (e.g. two resources 
from the same user), nothing guarantees that those two stanzas are 
received by a server in the same chronological order as they are sent. 
I've used the example with two servers because I thought this will make 
it clearer.

Regards,

Alexander

---

From: Ben Schumacher <[email-masked]>
Date: Fri, 01 Oct 2010 09:20:48 -0600

> problem as for two resources connected to the same server).

Given that most stanzas intended for a given user are directed to a Bare 
JID (think <presence> and <message> stanzas here), how does an using 
this form of cluster ensure that the <presence> stanzas received from 
another users are delivered to each of the two clients in the order 
received. It would seem to me that you'd need some sort of queue (or a 
particle if you'd prefer) for handling stanzas to a Bare JID until you 
put those stanzas into a queue for the individual streams.

It would seem to me that what we want is in-order delivery to a Bare JID 
until the last branching point wherein a server determines that this 
stanza is destined to a specific stream. I believe this is consistent 
with the text you proposed, Dave, but I don't think the language is 
completely clear.

I don't, however, consider myself to be a technical wordsmith, so I'll 
leave revisions to the language to the more experienced.

Cheers,
Ben

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 01 Oct 2010 09:33:12 -0600

> leave revisions to the language to the more experienced.

Sure, we have some wordsmiths around here. :)

One pesky process point: Dave posted his proposed text to a non-IETF
list, so IETF Note Well rules did not apply...

http://www.ietf.org/about/note-well.html

Dave, if you have text to contribute, perhaps you could do so on this list.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Dave Cridland <[email-masked]>
Date: Fri, 01 Oct 2010 16:36:36 +0100

On Fri Oct  1 16:33:12 2010, Peter Saint-Andre wrote:
> >>>> suggests ordering by streams at each hop.  For example, if  
> stanzas
> >>>> directed to
> >>>> [email-masked] and [email-masked]/resource would both be  
> sent to
> >>>> the same
> >>>> client stream, then these stanzas should be delivered in order  
> to
> >>>> that stream.
> >>>>
> >>>> The only critique of this approach has been that ordering by  
> stream,
> >>>> instead
> >>>> of by bare JID, may end up producing unordered receipt if  
> client
> >>>> connections
> >>>> disconnect and stanzas end up getting routed to alternate  
> resources
> >>>> or offline

Ah. Interesting point. By posting it there, I assumed copyright could  
be assigned to the XSF as required, but still, for completeness:

Servers MUST ensure in-order processing of the stanzas received on a  
stream. Servers MUST resolve any ambiguity caused by processing  
requests (for example, those sent to the bare jid of a connected  
client); in the case where the result of a request could have an  
effect on subsequent stanzas, servers MUST suspend processing of  
stanzas until the request is completed. Servers MUST ensure that any  
stanzas forwarded from a stream to a particular remote domain are  
forwarded in order - in particular, this implies that for any given  
inbound stanza that is addressed to a remote domain, there SHOULD be  
at most one outbound stream, used consistently.

What this means in practical terms:

1) If you get a vCard request for a bare jid, and a message to that  
same bare jid, you needn't block the entire stream while you pootle  
off to your database (or, if you're as hip as us, your ACP-142  
compliant X.500 directory) to fetch the info. Just kick of the vCard  
fetching, and forward the stanza to a suitable resource. If a sender  
cares about what the result of the vCard is prior to delivery of the  
message, they'd wait. And you certainly don't care - whatever their  
family name is, you're going to pick the same resource, and even if  
you waited, there's no certainty you'd manage to delivery the result  
before the message anyway.

2) If you get a XEP-0191 change request from a client, and, in the  
same TCP packet, a presence broadcast - hence both to the same bare  
jid - then absolutely you need to fully process the XEP-0191 request  
before the presence broadcast, because the order clearly matters.

3) Once you've forwarded a message, that counts as processing. So if  
two sequential messages require forwarding to two different streams,  
then you don't need to care about the order they're delivered. If two  
sequential messages require forwarding to the same stream, you  
forward them in turn (such that they'll arrive in-order).

3) If a MUC service is - sensibly - using a single stream to  
communicate with you, it all Just Works™. If it doesn't use a  
single stream, it becomes their fault and not yours. (But note that a  
MUC service could use a single stream per-participant

4) If you use multiple streams to communicate with an entity (ie, two  
S2S streams which you randomly use either of to send to a remote full  
jid), then everything will break, and this is entirely your fault.  
You can, however, consistently use one of any number of streams to a  
remote domain for stanzas from a particular stream.

Dave.

---

From: Ben Schumacher <[email-masked]>
Date: Fri, 01 Oct 2010 09:51:28 -0600

On 10/1/10 9:18 AM, Alexander Holler wrote:
> I've tried to express that the promise of XCP "ordering by bare JID" 
> is useless.
> I've used the example with two servers because I thought this will 
> make it clearer.
>

Alexander-

This is obviously true.

The expectation should be *from* a given stream -- i.e. if one user is 
connected to the network with two resources (meaning two different XMPP 
streams) and each resource produces a stanza to the same Bare JID then 
there can be no guarantee that they will be delivered in the order sent 
(too many variables). This sort of ordering will have to be provided for 
by some application logic, assuming this sort of synchronization is 
required.

However, if the first resource sends a second stanza across the stream 
to the same Bare JID, the server should ensure that those are processed 
in the order received on that stream.

Cheers,
Ben

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 01 Oct 2010 10:26:35 -0600

On 10/1/10 9:36 AM, Dave Cridland wrote:

<snip/>

Dave, thanks for posting here.

The requirement for "in-order processing" comes from a very old layer of
XMPP archeology. Unfortunately, as these threads have uncovered, that
requirement has never been precisely specified. Here is my informal
understanding of the basic idea: we want to ensure that if Juliet is
having a conversation with Romeo, he will receive her messages in the
same order she sent them over the stream to her server (capulet.lit).

For the originating server, I think the rule means that if Juliet has
two devices that are connected to capulet.lit over two different streams
(identified by the "balcony" and "chamber" resources), her server is
*not* responsible for ensuring that outbound stanzas to
[email-masked] are ordered across those two originating c2s streams,
because those are two different conversations. However, if she sends
three messages msg1, msg2, and msg3 from the "balcony" resource to
[email-masked], then her server needs to (1) process those stanzas
in the order it received them over that single originating c2s stream
and (2) send them in that order when it routes them to Romeo's server
(montague.lit). My understanding of the intentions of Jeremie and the
early jabberd team is that this rule applied (on the originating server)
to destination addresses that were bare JIDs instead of full JIDs
([email-masked], not [email-masked]/foo), because the
originating entity could not know whether any outbound stanza would be
"forked" on the destination server. I further think this rule implies
that the originating server would try to do everything in its power to
preserve the order of stanzas sent by Juliet, including (if it has more
than one stream to montague.lit) making sure that it sends all stanzas
from [email-masked]/balcony to [email-masked] over a single s2s
stream.

For the destination server, I think the rule means that Romeo's server
needs to preserve the order of stanzas it receives over a single stream
from Juliet's server. If there are two streams from capulet.lit to
montague.lit then it does not need to preserve order across those
streams (e.g., if msg1, msg2, and msg3 comes across stream1 from
capulet.lit but msg4 comes across stream2 from capulet.lit then it is
*not* responsible for ensuring order across those two originating s2s
streams). However, it does need to preserve the order of stanzas that it
receives over stream1, so that it will deliver msg1, msg2, and msg3 to
Romeo in the order it received them from Juliet's server. It matters not
whether msg1 was addressed to Romeo's bare JID, msg2 to a full JID, and
msg3 to the bare JID again -- his server needs to deliver those messages
to any given resource of Romeo's in the order it received them over the
originating s2s stream. Thus here again ordering applies on a per-stream
basis, which for the destination server means both the originating s2s
stream and the destination c2s stream.

The same rules apply, mutatis mutandis, to the stanzas that a client
sends to its own server (with no "to" address, an address of the
account's bare JID, or an address of the server itself), which is why
the current text says:

   This includes stanzas sent by a client to its
   server for direct processing by the server (e.g., in-order processing
   of a roster get and initial presence as described in [XMPP-IM]).

At least, that's how I've always understood things. Do feel free to
disagree. :)

Now to wordsmithing...

> stream.

I'd prefer "given stream" or "particular stream" or somesuch.

> Servers MUST resolve any ambiguity 

The phrase "resolve any ambiguity" is ambiguous to me. :)

> caused by processing requests
> (for example, those sent to the bare jid of a connected client); in the
> case where the result of a request could have an effect on subsequent
> stanzas, servers MUST suspend processing of stanzas until the request is
> completed. Servers MUST ensure that any stanzas forwarded 

I'd prefer "routed" because that's the term we use for sending a stanza
from one domain to another -- we've tended to avoid the term
"forwarding" because it evokes ideas of .forward files and such.

> from a stream
> to a particular remote domain are forwarded in order - in particular,
> this implies that for any given inbound stanza that is addressed to a
> remote domain, there SHOULD be at most one outbound stream, used
> consistently.

As above, I think this applies to a pair of originating full JID and
destination bare JID, not originating server and destination server.

> 1) If you get a vCard request for a bare jid, and a message to that same
> bare jid, you needn't block the entire stream while you pootle off to
> your database (or, if you're as hip as us, your ACP-142 compliant X.500
> directory) to fetch the info. Just kick of the vCard fetching, and
> forward 

"deliver"

> the stanza to a suitable resource. If a sender cares about what
> the result of the vCard is prior to delivery of the message, they'd
> wait. And you certainly don't care - whatever their family name is,
> you're going to pick the same resource, and even if you waited, there's
> no certainty you'd manage to delivery the result before the message anyway.

Agreed.

> 2) If you get a XEP-0191 change request from a client, and, in the same
> TCP packet, a presence broadcast - hence both to the same bare jid -
> then absolutely you need to fully process the XEP-0191 request before
> the presence broadcast, because the order clearly matters.

Agreed.

> 3) Once you've forwarded a message, that counts as processing. So if two
> sequential messages require forwarding to two different streams, then
> you don't need to care about the order they're delivered. If two
> sequential messages require forwarding to the same stream, you forward
> them in turn (such that they'll arrive in-order).

Yes, with s/forwarded/routed or delivered/ and s/message/stanza/ etc.

> 3) If a MUC service is - sensibly - using a single stream to communicate
> with you, it all Just Works™. If it doesn't use a single stream, it
> becomes their fault and not yours. (But note that a MUC service could
> use a single stream per-participant

Agreed -- this is just an application of the general rule (process
in-order on a per-stream basis) to MUC.

> jid), then everything will break, and this is entirely your fault. You
> can, however, consistently use one of any number of streams to a remote
> domain for stanzas from a particular stream.

This seems in tension with the last sentence of your proposed text.
Perhaps it would help to clarify "inbound" and "outbound", or to use
different terms (because we often use those terms in relation to
stanzas, see for example the subscription sections in 3921bis).

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Dave Cridland <[email-masked]>
Date: Fri, 01 Oct 2010 21:13:47 +0100

> The phrase "resolve any ambiguity" is ambiguous to me. :)
That's for servers to resolve.

Dave.

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 04 Oct 2010 12:07:17 -0600

Proposed text at the end.

> destination bare JID, not originating server and destination server.

Taking all these comments together and incorporating Dave's earlier
suggestion, I propose the following text. This is much longer than what
Dave proposed, but it's more thorough and precise. Besides, you do know
that I get paid per word, right? ;-)

###

10.1.  In-Order Processing

   An XMPP server MUST ensure in-order processing of the stanzas and
   other XML elements it receives over a given stream from a connected
   client or server (for purposes of this section we describe such a
   stream as an "input stream", in contrast to an "output stream" that a
   server would use to deliver data to a connected client or to route
   data to a remote server).

   In-order processing applies (a) to any XML elements used to negotiate
   and manage XML streams, and (b) to all uses of XML stanzas, including
   but not limited to the following:

   1.  Stanzas sent by a client to its server or to its own bare JID for
       direct processing by the server (e.g., in-order processing of a
       roster get and initial presence as described in [XMPP-IM]).

   2.  Stanzas sent by a connected client and intended for delivery to
       another entity associated with a local domain (e.g., stanzas
       addressed from <[email-masked]> to
       <[email-masked]>).  The server MUST ensure that it delivers
       stanzas addressed to the intended recipient in the order it
       receives them over the input stream, treating stanzas addressed
       to the bare JID and the full JID of the intended recipient as
       equivalent for delivery purposes.

   3.  Stanzas sent by a connected client and intended for delivery to
       an entity located at a remote domain (e.g., stanzas addressed
       from <[email-masked]> to <[email-masked]>).  The server
       MUST ensure that it routes stanzas addressed to the intended
       recipient in the order it receives them over the input stream,
       treating stanzas addressed to the bare JID and the full JID of
       the intended recipient as equivalent for delivery purposes.  To
       help ensure in-order processing, the server MUST route such
       stanzas over a single "output stream" to the remote domain,
       rather than sending some stanzas over one server-to-server stream
       and other stanzas over another server-to-server stream.

   4.  Stanzas sent by one server to another server for direct
       processing by the server that is hosting the remote domain (e.g.,
       stanzas addressed from <im.example.com> to <example.net>).

   5.  Stanzas routed from one server to another server for delivery to
       an entity associated with the remote domain (e.g., stanzas
       addressed from <[email-masked]> to <[email-masked]> and
       routed by <im.example.com> over a server-to-server stream to
       <example.net>).  The routing server MUST ensure that it routes
       stanzas addressed to the intended recipient in the order it
       receives them over the input stream, treating stanzas addressed
       to the bare JID and the full JID of the intended recipient as
       equivalent for delivery purposes.  To help ensure in-order
       processing, the server MUST route such stanzas over a single
       "output stream" to the remote domain, rather than sending some
       stanzas over one server-to-server stream and other stanzas over
       another server-to-server stream.

   If the server's processing of a particular request could have an
   effect on its processing of subsequent data it might receive over
   that input stream (e.g., enforcement of communication policies), it
   MUST suspend processing of subsequent data until it has processed the
   request.

   In-order processing applies only to a single input stream.  Therefore
   a server is not responsible for ensuring the coherence of data it
   receives across multiple input streams associated with the same local
   account (e.g., stanzas received over two different input streams from
   <[email-masked]/balcony> and <[email-masked]/chamber>)
   or the same remote domain (e.g., two different input streams
   negotiated by a remote domain; however, a server MAY return a
   <conflict> stream error to a remote domain that attempts to negotiate
   more than one stream, as described under Section 4.8.3.3).

###

/psa

---

From: Justin Karneges <[email-masked]>
Date: Mon, 4 Oct 2010 11:35:55 -0700

> Proposed text at the end.

It's all so clear!  +1

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 04 Oct 2010 15:37:11 -0600

>> Proposed text at the end.

Super. I did notice one instance of redundant information, because the
following text needs to be only in bullet #5 and not also in bullet #3:

       The server

So I've removed that from bullet #3, leaving it only in bullet #5 (which
I've renumbered as #4). Thus:

###

   3.  Stanzas sent by a connected client and intended for delivery to
       an entity located at a remote domain (e.g., stanzas addressed
       from <[email-masked]> to <[email-masked]>).  See the
       next rule regarding routing of such stanzas to the remote domain.

   4.  Stanzas routed from one server to another server for delivery to

###

/psa

---

From: Justin Karneges <[email-masked]>
Date: Mon, 4 Oct 2010 15:11:28 -0700

> following text needs to be only in bullet #5 and not also in bullet #3:

I think you had it right the first time.  The bare jid rule needs to apply both 
at the s1->s2 (#3) hop as well as the s2->c2 hop (#5), since the second 
depends on the first being true.

-Justin

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 04 Oct 2010 16:38:10 -0600

> depends on the first being true.

I think we were both right. :) A few of the words were wrong in my
original version and it was unclear (even to me on re-reading what I'd
sent!) that #5 applied to the "delivering server" performing end
delivery to the intended recipient, not to the "routing server"
receiving a stanza from the sending client and routing it off to a
remote domain. And I do think there was one redundant sentence in #5.
Plus I prefer change the order of #4 and #5 so that the processing flow
is a bit clearer. Thus I've further tweaked the text as follows:

###

   2.  Stanzas sent by a connected client and intended for delivery to
       another entity associated with a local domain (e.g., stanzas
       addressed from <[email-masked]> to
       <[email-masked]>).  The server MUST ensure that it delivers
       stanzas addressed to the intended recipient in the order it
       receives them over the input stream from the sending client,
       treating stanzas addressed to the bare JID and the full JID of
       the intended recipient as equivalent for delivery purposes.

   3.  Stanzas sent by a connected client and intended for delivery to
       an entity located at a remote domain (e.g., stanzas addressed
       from <[email-masked]> to <[email-masked]>).  The
       routing server MUST ensure that it routes stanzas addressed to
       the intended recipient in the order it receives them over the
       input stream from the sending client, treating stanzas addressed
       to the bare JID and the full JID of the intended recipient as
       equivalent for routing purposes.  To help ensure in-order
       processing, the routing server MUST route such stanzas over a
       single output stream to the remote domain, rather than sending
       some stanzas over one server-to-server stream and other stanzas
       over another server-to-server stream.

   4.  Stanzas routed from one server to another server for delivery to
       an entity associated with the remote domain (e.g., stanzas
       addressed from <[email-masked]> to <[email-masked]> and
       routed by <im.example.com> over a server-to-server stream to
       <example.net>).  The delivering server MUST ensure that it
       delivers stanzas to the intended recipient in the order it
       receives them over the input stream from the routing server,
       treating stanzas addressed to the bare JID and the full JID of
       the intended recipient as equivalent for delivery purposes.

   5.  Stanzas sent by one server to another server for direct
       processing by the server that is hosting the remote domain (e.g.,
       stanzas addressed from <im.example.com> to <example.net>).

###

/psa