Subject: Re: [dtn-users] Improving performance in DTN2

From: "Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]" <[email-masked]>
Date: Mon, 2 Jul 2012 07:35:17 -0500

CGR scheduling is what I had heard about in reference to some upcoming ION testing so I'll just manually invoke some disconnects and provide some test results.

LTP is in our near future... Stephen, try to hold back on the excessive pride you take in having a functional implementation!  :-) 

David Zoller
COLSA Corporation
Marshall Space Flight Center

-----Original Message-----
From: Stephen Farrell [mailto:[email-masked]] 
Sent: Friday, June 29, 2012 4:41 PM
To: Ivancic, William D. (GRC-RHN0)
Cc: Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]; [email-masked]
Subject: Re: [dtn-users] Improving performance in DTN2

If you use the LTPCL via LTPlib you can also do that and we've gotten interop with ION working here.

(Having said that, my LTPlib code's pretty crap so might mess up any performance. OTOH, if you're in the mood for fixing things...:-)

S

On 06/29/2012 08:06 PM, Ivancic, William D. (GRC-RHN0) wrote:
> If your talking ION than contact graph routing (CGR) is there. As a first order approximation, you can of think of CGR as dynamic static routing (but it is a quite a bit more than that). In CGR you basically schedule contact times as would be done in predictive contacts such as space system (Orbital Dynamics).  With CGR  you could tell the system when to send and stop  as a way to emulate disconnection.
> 
> I do not believe CGR is implemented in DTN2.
> 
> - Will
> 
> On Jun 29, 2012, at 2:37 PM, Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] wrote:
> 
> Will,
> Good point. I've got my blinders on as I'm the ground receipt node for a downlink and the nodes I transmit to are almost always online. I'll have to expand my horizons and testing. A script should be no problem. I've heard tidbits of being able to schedule connect times for nodes and suspected that there might be a router type that implemented a schedule but haven't taken the time to research it yet and thought you might have been referring to that for the disconnects.
> Thanks,
> DZ
> 
> David Zoller
> COLSA Corporation
> Marshall Space Flight Center
> 
> From: Ivancic, William D. (GRC-RHN0)
> Sent: Friday, June 29, 2012 12:37 PM
> To: Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]
> Cc: [email-masked]<mailto:[email-masked]>
> Subject: Re: [dtn-users] Improving performance in DTN2
> 
> You can disconnect in a number of ways:
> netem
> pull cable (but not very repeatable timing wise) modify ip-table 
> script or link up/down script (probably the easiest and causes the least strain on the CPU).
> put a channel emulator in between DTN nodes.
> 
> The point is, you're streaming in a disconnected environment otherwise there is no reason to use DTN.  So you need to test the storage, not just the forwarding.
> 
> You probably should also put in some other bundle source/destination pairs with various lifetimes.
> 
> You need to see where it breaks and how fragile (or not) the system is.  Tuning a configuration to get something to work is just the start.
> 
> 
> - Will
> 
> On Jun 29, 2012, at 11:32 AM, Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] wrote:
> 
> 
> I'm sure this implementation will bog down during the disconnects as bundles accumulate in the pending queue which impacts the find_duplicate() mentioned below but I'll try to squeeze in some testing for you some time. What routing do you use for the disconnects or should I  just pull the network cable?
> DZ
> 
> 
> David Zoller
> COLSA Corporation
> Marshall Space Flight Center
> 
> From: Ivancic, William D. (GRC-RHN0)
> Sent: Friday, June 29, 2012 10:00 AM
> To: Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]
> Subject: Re: [dtn-users] Improving performance in DTN2
> 
> Add disconnection for 30 sec, 1 min, 5 min and 30 min and see what happens.
> 
> - Will
> 
> On Jun 29, 2012, at 10:55 AM, Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] wrote:
> 
> 
> 
> This is using static routes over IP with no disconnection. The Aggregate Custody Signals are configured for 30 seconds, so every 30 seconds the first node takes a hit while it is releasing about 5000 bundles from custody. Ingest stops for a second or so and then as it catches up it processes at about 400 bundles per second.
> 
> David Zoller
> COLSA Corporation
> Marshall Space Flight Center
> 
> From: Ivancic, William D. (GRC-RHN0)
> Sent: Friday, June 29, 2012 9:35 AM
> To: Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]
> Cc: [email-masked]<mailto:[email-masked]>; 
> [email-masked]<mailto:[email-masked]>
> Subject: Re: [dtn-users] Improving performance in DTN2
> 
> How long were your periods of disconnection while streaming?
> 
> How many bundle where buffered during disconnection?
> 
> Also, what routing protocol was used - or just static routes over IP?
> 
> - Will Ivancic
> 
> On Jun 29, 2012, at 10:25 AM, Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] wrote:
> 
> 
> 
> 
> Angela,
> I just joined the dtn-users mailing list yesterday and discovered that I've been working in a vacuum. We've been working on implementing Aggregate Custody Signals (ACS) in the DTN2 source code so that we can catch up with the ION version updated by the University of Colorado at Boulder and get back into the DTN network with their experiments. As I've been in the DTN2 source code, I have also been working on improving the throughput with the goal of streaming video via DTN. I am currently working on an ACS patch to submit for the reference implementation and am including some of the performance enhancements in it as a bonus.
> 
> I just concluded a 22 hour test piping 167 frames[bundles with a 1316 byte payload] per second through two DTN nodes with custody transfer and ACS processing. It processed 13,605,622 bundles and only lost 6 frames on the front end [missed UDP packets that never made it to the receiver that feeds the first DTN node] which were probably not related to the DTN node's performance. This is using the reference implementation with the ACS patches. In memory, the dtnd process has grown to 2.1GB due to timers that have been canceled but have not yet expired since I use a 24 hour expiration time. I have a reaper thread that periodically clears the canceled timers that I was not planning on including in the ACS patch since it requires a coordinated update to the oasys package but I think I'll go ahead and include it seeing that there is an interest.
> 
> On my development machine, our in-house version tops out at about 340 bundles/second and I haven't tested the patched reference implementation to that extent. Our in-house version has more extensive infrastructure changes that may not be suitable for a reference implementation since the goal was performance for our needs and some of the changes may have broken some of the nuances of the BP specification which I have not studied in depth. We'll check to see if there is a way to make it available to the community for evaluation as there may be some mods that can be cherry picked for adapting into the reference implementation.
> 
> In the meantime, here are some things we've run across:
> 
> * Configure and build oasys with the --disable-debug-locking option
> 
> * You might gain improvements using the--disable-debug option when 
> configuring both oasys and dtn2 but I haven't bothered to do that yet 
> as the debug is still useful to me
> 
> * BundleDaemon is a single thread. I've broken this into multiple threads: Input, Output, Storage, ACS and everything else in the original BD thread. The BD currently manages the oasys::TimerSystem which I changed to crank up the oasys::TimerThread to offload that work.
> 
> * BundleDaemon::find_duplicate() bogs down significantly as you approach 15,000 bundles in the pending_bundles_ list.
>      !!Ha!! My intent was to use a std::multimap for the pending bundles but I just saw that I used a std::map instead. I'll have to go back and change that. Anyway, std::map is better suited for this than the std::list and a change to std::multimap should be even better to allow detection and retention of duplicates if so configured.
> 
> * CustodyTimers do not release the reference to the bundle until they expire which keeps freed bundles from being removed as well as the timers. I use a reaper thread to periodically remove the canceled timers from the oasys::TimerSystem but I think adding an overridable onCancel() method to the oasys::Timer class would be good way to trigger processing that could free the bundle reference and allow the bundle to be freed quicker.
> 
> * I'm using the BerkeleyDB on a Solid State Drive.
> 
> * Payload storage to disk - the more files in the directory the longer it takes to add a bundle to the database. I changed the payload directory structure to a tiered layout with 10,000 bundle payload per directory.
> 
> * AdminRegistration and PingRegistration do not update the bundle 
> fwdlog to delivered so they don't get deleted until they expire. Minor 
> unless you have tons of custody signals hanging around but the leaner 
> the memory usage the better :)
> 
> Hope this helps,
> DZ
> 
> 
> David Zoller
> COLSA Corporation
> Marshall Space Flight Center
> _______________________________________________
> dtn-users mailing list
> [email-masked]<mailto:[email-masked]>
> https://www.irtf.org/mailman/listinfo/dtn-users
> 
> 
> 
> ******************************
> William D. Ivancic
> Phone 216-433-3494
> Fax 216-433-8705
> Networking Lab 216-433-2620
> Mobile 440-503-4892
> http://roland.grc.nasa.gov/~ivancic
> 
> 
> ******************************
> William D. Ivancic
> Phone 216-433-3494
> Fax 216-433-8705
> Networking Lab 216-433-2620
> Mobile 440-503-4892
> http://roland.grc.nasa.gov/~ivancic
> 
> 
> ******************************
> William D. Ivancic
> Phone 216-433-3494
> Fax 216-433-8705
> Networking Lab 216-433-2620
> Mobile 440-503-4892
> http://roland.grc.nasa.gov/~ivancic
> 
> 
> 
> 
> _______________________________________________
> dtn-users mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/dtn-users
>

---

From: "Pitts, Robert L. (MSFC-EO60)[HOSC SERVICES CONTRACT]" <[email-masked]>
Date: Mon, 2 Jul 2012 07:37:28 -0500

We may well be able to provide some improvements as we do some poking at it.  We are severely restricted on funds but we have a small window now.  WE are trying to ensure its compliance with the CCSDS direction.

	Lee

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]
Sent: Monday, July 02, 2012 7:35 AM
To: Stephen Farrell
Cc: [email-masked]
Subject: Re: [dtn-users] Improving performance in DTN2

David Zoller
COLSA Corporation

-----Original Message-----
From: Stephen Farrell [mailto:[email-masked]] 
Sent: Friday, June 29, 2012 4:41 PM
To: Ivancic, William D. (GRC-RHN0)
Cc: Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]; [email-masked]
Subject: Re: [dtn-users] Improving performance in DTN2

S

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 02 Jul 2012 14:51:47 +0100

On 07/02/2012 01:35 PM, Zoller, David A. (MSFC-EO60)[HOSC SERVICES
> CGR scheduling is what I had heard about in reference to some upcoming ION testing so I'll just manually invoke some disconnects and provide some test results.

CGR for DTN2? I'd be interested in that.

> LTP is in our near future... Stephen, try to hold back on the excessive pride you take in having a functional implementation!  :-) 

You obviously haven't yet looked:-)

S

---

From: "Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]" <[email-masked]>
Date: Mon, 2 Jul 2012 11:30:29 -0500

Change in plans...
As Will pointed out, DTN is meant to support a disconnected environment, so I believe it is worth postponing our checking off our ACS patch upload milestone by a week or so and make all of our modifications available for evaluation sooner. Maybe create a new branch for it until it can be vetted by the community?

Here are some stats from Thursday's version of the patched Sourceforge code which did not have the expired timer reaping:
VIRT        RES      SHR      %MEM      (statistics from top)
124m      29m     4636         0.8          shortly after start
1902m    1.7g     4636         43.5        16 hours
2428m    2.2g     4636         55.9        27 hours
2432m    2.2g     4704         56.2        39 hours
** 23489113 bundles processed 
** 60GB of transaction logs (I have a fix for this)
** memory usage stabilized after 24 hours which is when the expiration timers finally started expiring
** does not appear to be any memory leaks in the paths being tested

This weekend I adapted the BundleList changes to the Sourceforge version and included the expired timer reaping and the changed bundle storage directory structure. Here are some test results:

VIRT       RES      SHR      %MEM      (statistics from top)
51688    6024    4532          0.1         no bundles processed yet
201m     63m    4716          1.6         1 hour
201m     63m    4716          1.6         12 hours
201m     63m    4716          1.6         24 hours  14584000+ bundles processed
** much better memory usage than Thursday's version due to the expired timer reaping and no appearance of memory leaks.

At this point, I killed the receiving DTN node for 30+ minutes (input data continued flowing):
VIRT       RES      SHR      %MEM      (statistics from top)
1859m   1.7g    4788         43.5         30+ minutes storing bundles
** No problem keeping up with the 167 bundles per second being stored to disk for 30 minutes

At this point, I restarted the receiving DTN node and continued input data flowing:
* Input processing came to a complete halt for 40 minutes while the pending bundles were transferred to the link and transmitted
* Transmission to the receiving node during the 40 minutes averaged around 125 bundles/sec with occasional spikes to 190 and dips to 110
* After 40 minutes 456,391 pending bundles had been transmitted to the receiving node
* Over the next 8 minutes I saw 2 bundles processed on input and transmitted to the receiver - BundleDaemon was most likely processing all of the BundleTransmitted events during this phase
* Around minute 48, input processing resumed in earnest with bursts reaching 450+ bundles/second as it processed the UDP packets in the socket buffer (40MB) and output hit 900+ bundles/second at times
* About 50 minutes after restarting the receiver, processing settled back down to the 167 bundles per second
* 456,391 UDP packets were missed while the input processing was interrupted.

I believe that multi-threading the BundleDaemon will help alleviate some of this but I'm sure that there will be some bottlenecks we need to work on. I have not dug into the routers, contacts and convergence layers yet...

Would it be better to have one massive patch or a series of progressive patches? I am applying our changes in phases to the reference implementation and saving off the source code after each one so I could do it either way. These are the phases/ patches I am considering with a brief description. Any suggestions are welcome.

1) IPN Naming Scheme patch
* Fixes some ION interoperability issues
** This patch was our test to see what hoops we ("we" actually means Lee) have to jump through to get export approval and was uploaded to the Sourceforge Bugs Tracker last week. I think all of the patching below can be rolled into a single submission even if it is broken out into several patches.

2) ACS patch
* Aggregate Custody Signals processing/generating
* Custody Transfer Enhancement Block processing/generating
* Extended Class Of Service Block - partial implementation for compatibility with UC-Boulder ION experiment requirements.
	- Adds ECOS block to outgoing custody signals and processes incoming ECOS block but does not do anything with the incoming values
* Fixes for some other issues I ran across

3) BundleList patch
* BundleListBase - a base class for the bundle lists
* BundleList -  list<bundle*>   - the original descended from the new base
* BundleListIntMap - map<int, bundle*>   - used for the all_bundles_ list with the key being the bundle id
* BundleListStrMultiMap - multimap<string, bundle*> - used for the pending_bundles_ list with the key being the bundle GbofId string
* BundleListStrMap - map<string, bundle*> - used for the custody_bundles_ list with the key being the bundle GbofId string
* BundleMappings - changed to store any type of iterator 
* GbofId - now internally maintains its string representation
* Bundle - incorporated the GbofId object
* BundlePayload - changed the directory structure 
* BundleDaemon - crank up the oasys::TimerThread and TimerReaperThread
	- requires a new oasys patch

4) Oasys patch
* Timer - implementation of the expired timer reaping
* I thought a Shared Read/Exclusive Write lock would improve performance in some areas so I spent some time experimenting with the locks and I'll throw those in here while I'm at it.
	- Fixed the SXLock the way I think it should work - original did not guarantee the exclusiveness of the write lock
	- Created variations with atomic and mutex implementations
	- Unit test to compare performance

5) BundleDaemonInput thread patch
* this will break out the input related events to a separate thread
	- this should allow the input processing to continue during the 50 minute halt seen in the above test

6) BundleDaemonStorage thread patch
* this is a divergence from the current implementation and may technically break the spec??
	- GlobalStore updating remains the same so it updates immediately when new bundle ids, etc are issued
	- All other storage updating is queued to the BundleDaemonStorage thread
	`- Storage thread accumulates the updates to be applied and performs them on a configurable interval which I currently set to 100ms
	- The idea was that if a bundle was received and delivered fast enough then no actual write to the disk would need to take place for that particular bundle
	- Setting the interval to 0 should get results similar to the current implementation
	- Not sure if start and stop transaction is working. I added auto log removal for Berkeley DB and had to add calls to txn_checkpoint to get the logs to clear

7) BundleDaemonOutput thread patch
* this will break out the output related events to a separate thread
	- previously mentioned possible unintended consequences with the opportunistic contacts

8) Miscellaneous modifications that might be of interest
* dtn_recv_raw - new api method to get delivery of the entire bundle as it would be transmitted on the wire instead of just the payload
* udp2dtn, dtn2udp & udp repeater - utilities for reading UDP packets and sending the payload via DTN and back out
	- these currently use internal libraries so I would have to redo them as dtn apps
	- udp repeater is a java app that reads UDP packets on one interface and spits them out on another so that I can pipe a video from on one network through a firewall to another. It also can generate packets at configurable size and rate.
* AdminRegistrationIPN - set a local IPN style EID for the node and it takes custody of IPN bundles using that instead of the dtn:// EID so that the dtn node can take custody of the bundle and continue to use the CBHE compression
* configurable bundle id size 32 or 64 bit
* BundleInfoCache - this brings up a performance enhancement discussion:
	- basically this prevents a duplicate bundle from being delivered as long as it is within 1000 bundles of its doppelganger if I recall the code correctly
	- I remember seeing a comment in the code somewhere that it slows things down rather than speeds them up
	- I implemented a version on the DTN side that I think helped but did not  doing any timing testing to verify 
	- My real question:  Is this even necessary??
	- It seems to me that if the DTN node is configured to accept duplicates and someone is actually sending duplicates for some reason then should we assume they know what they are doing and pass them on? Maybe their clock got stuck but the data is actually different.
	- Is this an artifact left over from early development to compensate for repeated attempts to deliver the same bundle?

Now a confession: 
The BundleList adaptation above was one I planned on making in the future anyway to apply to our in-house version. I think that when this is complete the performance will be better than our in-house version even before the community has a chance to put their eyeballs on it and make additional improvements. My first iteration was an attempt to save on creating BundleRefs every time you turned around and was passing around and storing pointers to BundleRefs. This and the multi-threaded BundleDaemon lead to such fun as bundles starting the received processing and then being delivered and deleted before the input thread could complete its processing :).  As I worked through the various issues, I came to see the wisdom of the BundleRef and the current in-house version now stores a BundleRef. The drawback to storing the BundleRef in the lists is that every insertion and erasure results in something like 3 intermediate BundleRefs being created and the additional overhead that causes. So this brings me back full circle to the original implementation :) 

Anyway, my complements and thanks to all who have helped build a great foundation to work with and I'll quit rambling now.
DZ

David Zoller
COLSA Corporation

---

From: "Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]" <[email-masked]>
Date: Mon, 2 Jul 2012 11:32:39 -0500

Clarification - no CGR for DTN2 that I know of. The upcoming testing I mentioned was all ION nodes.

David Zoller
COLSA Corporation
HOSC / M103  
/Office: (256) 544-1820
(EMail: [email-masked]

-----Original Message-----
From: Stephen Farrell [mailto:[email-masked]] 
Sent: Monday, July 02, 2012 8:52 AM
To: Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]
Cc: [email-masked]
Subject: Re: [dtn-users] Improving performance in DTN2

> LTP is in our near future... Stephen, try to hold back on the 
> excessive pride you take in having a functional implementation!  :-)

S

> (Having said that, my LTPlib code's pretty crap so might mess up any 
> performance. OTOH, if you're in the mood for fixing things...:-)
> 
> S
> 
>> fwdlog to delivered so they don't get deleted until they expire. 
>> Minor unless you have tons of custody signals hanging around but the 
>> leaner the memory usage the better :)
>>
>> Hope this helps,
>> DZ
>>
>>
>> David Zoller
>> COLSA Corporation