Subject: [dtn-users] 1 Hour Data Flow Test Results - DTN2.8.0 Changeset 3550 plus InputThread patch

From: "Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]" <[email-masked]>
Date: Tue, 9 Oct 2012 12:51:24 -0500

Greetings all,
Priorities had me working on other projects but I managed to start sneaking in some "fun with DTN2" time again a couple of weeks ago and I've compiled some more test results of the patches I'm working on. I thought I would spare y'all a message and combined two changes into this first set of results.

While working with the Experimental TableBasedRouter patch uploaded earlier, I tracked down a slowdown to the dtn::Link class when two links are open for the same destination. In a dtn::Link all of the bundles to be sent a put on a deferred list and as they are transmitted they are removed from the list and put on the in-flight list. When two links are open, each link only removes bundles that it transmits from its deferred list and thus must cycle through an every growing set of bundles that were transmitted by the other link to find the next bundle to be transmitted. I did not want to change the fundamental behavior of the deferred list so I added a "working deferred" list for want of a better name. When bundles are added to the link they are placed on both lists and when they are transmitted by the link they are removed from both lists. The difference is that the "working deferred" list is used to find the next bundle to transmit and if it finds one that was transmitted by another link it removes it only from the "working deferred" list so that it will not have to keep checking it umpteen times. This probably improved the transmit rate a bit but did not help with the ingest performance I am currently interested in.

With a nod to Jacob, I modified my original implementation of the BundleDaemon Input Thread. When a bundle is received, the BundleReceivedEvent is first processed by the input thread up to the point where it is added to the database and inserted into the pending bundles list. The event is then passed on to the main BundleDaemon thread where it is processed for routing and should thus maintain the integrity the original main event Q design.

My results processing are now mostly automated so the data for this set of tests is collected on a 1 second cycle instead of a 10 second cycle to provide better insight into what is happening. The data collection is still within the main BundleDaemon thread so in this case the BunRcvPost count is the number of bundles that were processed by the input thread and added to the database in 1 second. The BunRcvProc count refers to the routing processing done by the main BundleDaemon thread. I'm also making sure that the timing is such that node1 connects to node2 shortly after it comes on line to maximize the impact of two links open with lots of bundles pending.

Best regards,
DZ

DTN2 - One Hour Data Flow with the Receiver (Node2) Brought Online After 30 Minutes

Configuration

*        DTN2 changeset 3550 plus patches up to and including the InputThread patch plus some statistics collection code.

*        OASYS changeset 2298 with all patches applied.

*        Computer (Node1)

o   HP xw4400 Workstation

o   Intel(R) Core(TM)2 CPU          6300  @ 1.86GHz   (dual core)

o   4GB memory

o   BerkeleyDB hosted on a Solid State Drive configured as an ext2 partition with the noatime and nodiratime options

*        Data flow is 160 non-custody bundles per second for 60 minutes

Bottom Line Results

*        576,000 UDP packets were transmitted (1316 byte payload)

*        561,769 packets were processed into bundles  (1316 byte payload)

*          14,231 packets were lost due to UDP socket buffer overflow

o   40MB socket buffer holds ~30k packets or ~187 seconds of data

*        Bundle ingest rate dropped to average 138 bundles/sec over a 1900 second interval

o   better than halting seen previously but not good enough to avoid losing data

o   delayed a lot of the transmission until after data flow stopped

Timeline Highlights
Seconds after Node1 started

10

Data flow started at 160 packets/sec

10 - 1817

Successfully received 289,363 bundles at 160 bundles/sec

1818

Node2 makes contact (link-0 created) - reroute_all_bundles entered - ingest (BunRcvPost) continues at 160 bundles/sec but BunRcvProc halts

1830

reroute_all_bundles finished - elapsed 12.3 seconds

1830 - 1835

Burst processing posted BundleReceived events, ingest at 160 bundles/sec and transmit averages 350

1836

Node1 establishes connection to Node2 - reroute_all_bundles entered - ingest (BunRcvPost) continues at 160 bundles/sec but BunRcvProc and Transmission halts

1852

reroute_all_bundles finished - elapsed 15.8 seconds

1852 - 3765

ingest avg=138 bundles/sec,  rcvd bundles routed avg=64 bundles/sec,   transmit avg=215 bundles/sec

3610

Data flow terminates

3766 - 4577

ingest avg=0,  rcvd bundles routed avg=180 bundles/sec,   transmit avg=180 bundles/sec, event queue slowly shrinks

4578 - 4666

ingest=0, routed=0, transmit=0, process transmitted events avg=1656 bundles/sec, process free events avg=2566 bundles/sec

4667 - 4697

ingest=0, routed=0, transmit=0, process transmitted=0, process free events avg=4725 bundles/sec

4697

Processing complete in 1.3 hours

David Zoller
COLSA Corporation
Marshall Space Flight Center