Subject: [dtn-users] Out of order bundles

From: "Mehta, Devanshu - 0665 - MITLL" <[email-masked]>
Date: Thu, 21 Nov 2013 14:49:38 -0500

Folks,
I'm using DTN 2.9.0 with the TCP convergence layer (opportunistic).

I'm using dtnsend/recv to periodically send bundles from directly connected nodes n1->n2.

When I introduce a link outage between n1 and n2, the bundles are stored on n1 as expected. When the link is restored, all the stored bundles are transmitted as expected.

However, the bundles are not transmitted in order. The newly arriving bundles are interspersed with the stored bundles. This causes out of order delivery. It's not a huge issue, but I was wondering if this is:

 1.  Expected behavior?
 2.  If there is a way to enforce ordering?

Thanks,
--
Devanshu Mehta
MIT Lincoln Laboratory

---

From: "Scott, Keith L." <[email-masked]>
Date: Thu, 21 Nov 2013 20:07:03 +0000

Devanshu,

It is the expected behavior, since the Bundle Protocol does not purport to provide in-order delivery.

There is a 'shim' that's been developed by the international space community (through the Consultative Committee for Space Data Systems, CCSDS) called Delay-Tolerant Payload Conditioning (DTPC).  DTPC provides a 'TCP-like' service on top of BP, in particular: reliable, in-order, no duplicates delivery.  DTPC support for DTN2 should be available 'real soon now'

DTPC is documented in the CCSDS extensions to the Bundle Protocol and is available from http://cwe.ccsds.org/sis/docs/SIS-DTN/Draft%20Documents/DTN%20Protocol/734x2r2_FC2_clean.docx

This IS NOT yet a full-fledged CCSDS Recommendation - IT IS STILL DRAFT (but it's been through one round of agency reviews and we HOPE it is pretty much final).

                        --keith

From: dtn-users [mailto:[email-masked]] On Behalf Of Mehta, Devanshu - 0665 - MITLL
Sent: Thursday, November 21, 2013 2:50 PM
To: [email-masked]
Subject: [dtn-users] Out of order bundles

Folks,

---

From: <[email-masked]>
Date: Fri, 22 Nov 2013 07:12:49 +0000

It's interesting that a shim to address some of the deficiencies in the Bundle Protocol is being developed outside the IRTF group that did the bundle protocol.

Still, CCSDS pushed bundling onto the IRTF, and CCSDS is the prime intended (only?) user of bundling with intent to deploy and activity around DTN, for its original purpose of the Interplanetary Internet.

As for the shim - in computing, anything can be fixed with a level of indirection; the end-to-end principle (or even ARQ) is an example of that. But embedded systems do not need the overhead of too many layers, and CFDP (for the API!) over DTCP over Bundle over Licklider over CCSDS on spacecraft might be a tad much, even before the infamously complex Bundle Security Protocol is added to the mix.

appendix D1 of the CCSDS bundle document has an appeal to the end-to-end principle - will it be a recommendation before testing and deployment?

Also interesting to read as context Scott Burleigh's arguments supporting the end-to-end principle and justifying the creation of DTCP:
http://www.intersys-lab.org/media/papers/2012/TR-DUTH-EE-2012-24_DTPC.pdf
Delay Tolerant Payload Conditioning Protocol, Burleigh et al, Technical Report: TR-DUTH-EE-2012-24

"To what extent does the design of the DTN architecture need to follow
 the end-to-end principle that has successfully served the Internet for
 almost 30 years? This question gains in importance when future large
 scale general use deployment of DTN is considered; experience in Internet
 evolution shows that small scale research had limited utility in predicting large
 scale behaviors. However, it has received little attention until today."

Good to see that someone is finally  -- FINALLY! -- asking if the DTN architecture needs to follow the end-to-end principle.

Lloyd Wood
http://sat-net.com/L.Wood/dtn/bundle.html

been there, done that.

________________________________________
From: dtn-users [[email-masked]] On Behalf Of Scott, Keith L. [[email-masked]]
Sent: 21 November 2013 20:07
To: Mehta, Devanshu - 0665 - MITLL; [email-masked]
Subject: Re: [dtn-users] Out of order bundles

Devanshu,

There is a ‘shim’ that’s been developed by the international space community (through the Consultative Committee for Space Data Systems, CCSDS) called Delay-Tolerant Payload Conditioning (DTPC).  DTPC provides a ‘TCP-like’ service on top of BP, in particular: reliable, in-order, no duplicates delivery.  DTPC support for DTN2 should be available ‘real soon now’

This IS NOT yet a full-fledged CCSDS Recommendation – IT IS STILL DRAFT (but it’s been through one round of agency reviews and we HOPE it is pretty much final).

                        --keith

Folks,

---

From: Caini Carlo <[email-masked]>
Date: Fri, 22 Nov 2013 10:26:57 +0100

Dear Devanshu,
        As already said by Keith Scott the BP does not assure any 
ordered delivery. Let me only add that disordered delivery can be 
present also in the absence of any interruptions if two TCP links are 
opened in parallel between the same couple of nodes. For instance, if 
you have one ALWAYSON "link" instruction in n1 pointing to n2, and 
analogously another one in n2 pointing to n1.
If you want just one TCP link, use ALWAYSON (or ONDEMAND) on n1 and 
OPPORTUNISTIC in n2. In recent DTN2 versions actually an 
opportunistic link is opened in an automatic way when a TCP 
connection is opened by another node. Of course, if you have two TCP 
links in parallel, disordered delivery would be exacerbated after a 
link disruption. Note that my considerations are based on experiments 
carried out on versions previous than 2.9, but I suppose that the 
behavior is the same.
Maybe you already knew everything. Sorry in this case.
Yours,
     Carlo

At 20.49 21/11/2013, Mehta, Devanshu - 0665 - MITLL wrote:
>Content-Language: en-US
>Content-Type: multipart/alternative;
>         boundary="_000_CEB3CE02F4CAmehtallmitedu_"
>
>Folks,
>When I introduce a link outage between n1 and n2, the bundles are 
>stored on n1 as expected. When the link is restored, all the stored 
>bundles are transmitted as expected.
>
>However, the bundles are not transmitted in order. The newly 
>arriving bundles are interspersed with the stored bundles. This 
>causes out of order delivery. It's not a huge issue, but I was 
>wondering if this is:
>    * Expected behavior?
>    * If there is a way to enforce ordering?
>Thanks,
>--
>Devanshu Mehta
>MIT Lincoln Laboratory
>
>_______________________________________________
>dtn-users mailing list
>[email-masked]
>https://www.irtf.org/mailman/listinfo/dtn-users

---

From: <[email-masked]>
Date: Fri, 22 Nov 2013 09:45:29 +0000

You'd think ordering  transmission according to bundle expiry time, or even creation time,would be easy enough to include.

But that's the sort of conditioning thing that's likely assigned to DTPC these days...

Lloyd Wood
http://sat-net.com/L.Wood/

________________________________________
From: dtn-users [[email-masked]] On Behalf Of Caini Carlo [[email-masked]]
Sent: 22 November 2013 09:26
To: Mehta, Devanshu - 0665 - MITLL; [email-masked]
Subject: Re: [dtn-users] Out of order bundles

Dear Devanshu,

---

From: Elwyn Davies <[email-masked]>
Date: Fri, 22 Nov 2013 10:50:16 +0000

Devanshu,

The semantics of the DTN architecture and the Bundle Protocol expect
bundles to be units of data that are complete in themselves.  So..

- it is up to the sending and receiving applications to make connections
between separate bundles (hence the NASA shim),

- your case of directly connected nodes is an exception to the general
case of DTN 'networks' - in a more general opportunistic networking
situation the bundles may well go by a totally separate sequence of
nodes resulting in delivery of bundles in essentially an arbitrary order
whatever order they were sent by the application - and potentially
whatever order they were transmitted in by the source node because there
is no guarantee that they will end up being handed to the same next hop
at any stage in their journey as it might have disconnected between
transmissions (or refused additional bundles 'cos it was overloaded,
etc.)  Accordingly there is no particular incentive for a node to try to
maintain any sort of ordering in its transmissions - indeed there might
be other considerations such as size or expiry time that could affect
scheduling of bundle dispatch; and

- the receiving node has no way to know that a bundle is part of a
sequence so it should just deliver them to the receiving application
rather than hang on to see if there are others coming.

It would be a legitimate policy for a DTN2 node to send bundles that it
has ready for transmission in order of their creation timestamps (which
are of course part of the bundle ID) either globally across all source
EIDs or relatively within each different source EID, but this is only
one of many different policies that has been considered as a way to
obtain the best delivery performance - another one is sending bundles
with the longest remaining time to live (i.e., most distant expiry time)
first.  That is effectively the policy you are seeing :-) 

Regards,
Elwyn

On Thu, 2013-11-21 at 20:07 +0000, Scott, Keith L. wrote:
> There is a ‘shim’ that’s been developed by the international space
> community (through the Consultative Committee for Space Data Systems,
> CCSDS) called Delay-Tolerant Payload Conditioning (DTPC).  DTPC
> provides a ‘TCP-like’ service on top of BP, in particular: reliable,
> in-order, no duplicates delivery.  DTPC support for DTN2 should be
> available ‘real soon now’ 
> 
>  
> 
> DTPC is documented in the CCSDS extensions to the Bundle Protocol and
> is available from http://cwe.ccsds.org/sis/docs/SIS-DTN/Draft%
> 20Documents/DTN%20Protocol/734x2r2_FC2_clean.docx
> 
>  
> 
> This IS NOT yet a full-fledged CCSDS Recommendation – IT IS STILL
> DRAFT (but it’s been through one round of agency reviews and we HOPE
> it is pretty much final).
> 
>  
> 
>                         --keith
> 
>  
> 
>  
> 
> However, the bundles are not transmitted in order. The newly arriving
> bundles are interspersed with the stored bundles. This causes out of
> order delivery. It's not a huge issue, but I was wondering if this is:
> 
> 
> 1.     Expected behavior?
>

---

From: "Ivancic, William D. (GRC-RHN0)" <[email-masked]>
Date: Fri, 22 Nov 2013 13:46:15 +0000

>bundles to be units of data that are complete in themselves.  So..

Well said.  Let us not forget that DTN is for disconnected, multi-hop
store, carry and forward networks - albeit you can used DTN in connected
networks. With that in mind, many things do not correlate well to
connected networks. For example, speed of delivery is somewhat
meaningless.  So long as the bundle arrives before it expires, that is
sufficient.  If I receive my "Do not open before Christmas" Christmas
present before Christmas, it really doesn't matter that I received it a
month before 30 seconds before.

Expected operation policy of forwarding agents has, to the best of my
knowledge, never been documented.  That is fine and probably appropriate
for a research item, but probably not so good for and operational
deployment.  For example (we had this discussion many years ago in some
internal NASA meetings) if I receive a bundle that has a lifetime of two
years and I currently have no route at this point in time, do I:  1) drop
it, 2) forward it to some default agent (like sending it to the master
post office),  3) hold it until it expires hoping that at some point I
will have a route, or 4) something else? Note, DTN2 way back during our
space testing operated as #3.  That was nice, because if it didn't our
experiment would have failed as we had a misconfigured route at the ground
station.  The bundle was sitting there unexpired, waiting for a route.
Hey!  I believe the ION code use to drop bundle if no know route was
available - even if the bundle was not going to expire.  I do not know if
this is still the operation.  Regardless, you probably want to know what
your networks policies.  There are probably very good reasons to have
different policies in different deployments.

Will

---

From: "Burleigh, Scott C (312G)" <[email-masked]>
Date: Fri, 22 Nov 2013 16:14:57 +0000

ION forwarding policy is set by whatever forwarder module you have plugged in to handle bundles for the endpoint ID scheme named in the bundle's destination EID.  For the "ipn" scheme, the off-the-shelf forwarder provided in ION will first try to compute a route for the bundle, from several sources of information including a managed schedule of future contacts; if that works, it inserts the bundle into the queue of bundles awaiting eventual transmission to the first waypoint on that route.  If that doesn't work but at least one of the "ducts" to neighboring nodes is currently "blocked" for some reason (e.g., there's normally a TCP connection but that connection is temporarily broken), the forwarder puts the bundle in a "limbo" list pending the unblocking of a duct, at which time it will try again to compute a route.  If that doesn't work -- that is, if there's no prospect of forwarding the bundle, based on the node's current configuration -- it discards the bundle to free up storage.

Scott

-----Original Message-----
From: dtn-users [mailto:[email-masked]] On Behalf Of Ivancic, William D. (GRC-RHN0)
Sent: Friday, November 22, 2013 5:46 AM
To: Elwyn Davies; Scott, Keith L.
Cc: [email-masked]
Subject: Re: [dtn-users] Out of order bundles

>bundles to be units of data that are complete in themselves.  So..

Well said.  Let us not forget that DTN is for disconnected, multi-hop store, carry and forward networks - albeit you can used DTN in connected networks. With that in mind, many things do not correlate well to connected networks. For example, speed of delivery is somewhat meaningless.  So long as the bundle arrives before it expires, that is sufficient.  If I receive my "Do not open before Christmas" Christmas present before Christmas, it really doesn't matter that I received it a month before 30 seconds before.

Expected operation policy of forwarding agents has, to the best of my knowledge, never been documented.  That is fine and probably appropriate for a research item, but probably not so good for and operational deployment.  For example (we had this discussion many years ago in some internal NASA meetings) if I receive a bundle that has a lifetime of two years and I currently have no route at this point in time, do I:  1) drop it, 2) forward it to some default agent (like sending it to the master post office),  3) hold it until it expires hoping that at some point I will have a route, or 4) something else? Note, DTN2 way back during our space testing operated as #3.  That was nice, because if it didn't our experiment would have failed as we had a misconfigured route at the ground station.  The bundle was sitting there unexpired, waiting for a route.
Hey!  I believe the ION code use to drop bundle if no know route was available - even if the bundle was not going to expire.  I do not know if this is still the operation.  Regardless, you probably want to know what your networks policies.  There are probably very good reasons to have different policies in different deployments.

Will

>
>- it is up to the sending and receiving applications to make 
>connections between separate bundles (hence the NASA shim),
>
>nodes resulting in delivery of bundles in essentially an arbitrary 
>order whatever order they were sent by the application - and 
>potentially whatever order they were transmitted in by the source node 
>because there is no guarantee that they will end up being handed to the 
>same next hop at any stage in their journey as it might have 
>disconnected between transmissions (or refused additional bundles 'cos 
>it was overloaded,
>etc.)  Accordingly there is no particular incentive for a node to try 
>to maintain any sort of ordering in its transmissions - indeed there 
>might be other considerations such as size or expiry time that could 
>affect scheduling of bundle dispatch; and
>
>with the longest remaining time to live (i.e., most distant expiry 
>time) first.  That is effectively the policy you are seeing :-)
>
>Regards,
>Elwyn
>