Subject: [dtn-users] On measuring the DTN2 performance

From: TSELIKIS.Christos <[email-masked]>
Date: Thu, 6 Sep 2012 11:05:45 +0300

Hi all,
I'm returning on the very interesting dtn2 performance issues/results that appeared in the list around July 2012 (seems that I let the thread sleeping for too long, but it's never too late).
Well, just a few simple things, how do you find the dtn_perf (v2) dtn2 application as a measuring the DTN network throughput tool? I suspect that since the results are reported at the client side of the appl, it can be considered for network throughput/delay estimations, yes especially versus the bundle payload sizes (ACKs are taken into account). Also can be used for API measurements and multi-hop scenarios, in my opinion. Finally, has anybody bumped into some newer version of it (v3)?
With very kind regards,
ctselikis

Tselikis Christos
Hellenic Aerospace Industry S.A.
Electronics Engineering Department
e-mail: [email-masked]<mailto:[email-masked]>
PO Box 23, GR-32009 Schimatari, Greece
Tel. : +30 22620 52561
Fax: +30 22620 52910

---

From: Michele Rodolfi <[email-masked]>
Date: Thu, 6 Sep 2012 13:48:12 +0200

2012/9/6 TSELIKIS.Christos <[email-masked]>

>  Hi all,

Hi Christos,

> ****
>
> I’m returning on the very interesting dtn2 performance issues/results that
> appeared in the list around July 2012 (seems that I let the thread sleeping
> for too long, but it’s never too late).****
>
> Well, just a few simple things, how do you find the dtn_perf (v2) dtn2
> application as a measuring the DTN network throughput tool? I suspect that
> since the results are reported at the client side of the appl, it can be
> considered for network throughput/delay estimations, yes especially versus
> the bundle payload sizes (ACKs are taken into account). Also can be used
> for API measurements and multi-hop scenarios, in my opinion. Finally, has
> anybody bumped into some newer version of it (v3)?
>

I'm actually working on the development of dtnperf (v3) for my thesis at
the university of Bologna (Italy).
There are many changes and improvements and i need one or two weeks to
ultimate the program and write some documentation.
Then I'll have to test the application to find bugs and fix them, but i
hope you can try the app and send me some feedback so i can consider
another point of view.

****
>
> With very kind regards,****
>
> ctselikis
>

Best regards
   Michele Rodolfi

> ****
> ** **
> *Tselikis Christos ***
> e-mail: [email-masked] ****
>
> PO Box 23, GR-32009 Schimatari, Greece****
>
> Tel. : +30 22620 52561
> Fax: +30 22620 52910****
>
> ****
>
> ** **
>
> _______________________________________________
> dtn-users mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/dtn-users
>
>

---

From: TSELIKIS.Christos <[email-masked]>
Date: Thu, 6 Sep 2012 14:59:58 +0300

Hi Michele,
Thanks, I look forward to sending you feedback on new dtn_perf features.
Whenever would be convenient for you, I remain tuned for further discussions.
With very kind regards,

Tselikis Christos

From: Michele Rodolfi [mailto:[email-masked]]
Sent: Thursday, September 06, 2012 2:48 PM
To: TSELIKIS.Christos
Cc: [email-masked]
Subject: Re: [dtn-users] On measuring the DTN2 performance

2012/9/6 TSELIKIS.Christos <[email-masked]<mailto:[email-masked]>>
Hi all,

Hi Christos,

I'm actually working on the development of dtnperf (v3) for my thesis at the university of Bologna (Italy).
There are many changes and improvements and i need one or two weeks to ultimate the program and write some documentation.
Then I'll have to test the application to find bugs and fix them, but i hope you can try the app and send me some feedback so i can consider another point of view.
With very kind regards,
ctselikis

Best regards
   Michele Rodolfi

Tselikis Christos
Hellenic Aerospace Industry S.A.
Electronics Engineering Department
e-mail: [email-masked]<mailto:[email-masked]>
PO Box 23, GR-32009 Schimatari, Greece
Tel. : +30 22620 52561<tel:%2B30%2022620%2052561>
Fax: +30 22620 52910<tel:%2B30%2022620%2052910>

_______________________________________________
dtn-users mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/dtn-users

---

From: "Ivancic, William D. (GRC-RHN0)" <[email-masked]>
Date: Thu, 6 Sep 2012 07:41:10 -0500

DTN perf and DTNtraceroute are interesting concepts.  Do you have a writeup on the operations concepts.  IMHO, performance needs consider this is a disconnected network.  Return path may be totally different than forward path.  It probably would be good to be able to have sender-to-receiver option without an ACK similar to the original ttcp and an option in nuttcp.  If you have a back channel (like in an experimental testbed), you can get one-way performance.  Basically, think about how the operation would work if none of the these as continuous connectivity and every bundle has to be stored, carried and forwarded over a multi-hop system and the forwarding agents are changing in the topology.  That is much different than a string-of-pearls  fully connected set of forwarding agents.

- Will

---

From: Razvan BEURAN <[email-masked]>
Date: Fri, 7 Sep 2012 09:00:47 +0900 (JST)

Hi all,

I think there can be a very long discussion as to what exactly is 
"performance" in DTNs, but I also believe that a tool such as dtnperf can 
be used for very useful purposes. For instance, one may imagine using the 
same network (with the same disconnections and mobility patterns -- which 
can readily be obtained through emulation) to compare how several routing 
protocols affect end-to-end performance. I believe that such kind of 
end-to-end performance is actually what matters for the users.

As a side note, last time I used dtnperf (in dtn-2.8) I was very surprised 
to discover that for the UDP convergence layer the test would stall 
indefinitely as soon as there was loss in the network. The reason seems 
to be that, as opposed to TCP, UDP doesn't recover from loss, and dtnperf 
keeps waiting forever for some kind of bundle receipt. I hope the new 
version mentioned by Michele takes this into account, because it is 
clearly unexpected for a tool which is supposed to evaluate DNT. Note that 
for the TCP convergence layer dtnperf behaves as expected over lossy 
links, but this is probably due to TCP' behaviour.

By all means though, this mailing list should be the perfect place to 
discuss about how to evaluate DTN performance, and I am very willing to 
contribute to this discussion. I know for instance that IBR-DTN doesn't 
include anything similar to dtnperf, which shows that its developers 
don't see much value in dtnperf either.

Best wishes,
Razvan

---

From: TSELIKIS.Christos <[email-masked]>
Date: Fri, 7 Sep 2012 12:48:36 +0300

Hi all,
I fully agree with Razvan, "what exactly is 
performance in DTNs", perhaps some other metrics (except from throughput and delay) might provide more accurate descriptions (one way metrics such as delivery rate, loss, etc, which have already been studied as well). On some tests with dtn-perf2/TCP I also recall the dtn_perf application was able to resume after reasonable link disruptions (for the sake of accuracy just to mention that "ACKs" in dtn-perf2 are actually bundle delivery reports, if requested).
Now for sure DTN performance has to be analyzed under adaptive routing conditions and it would be valuable to measure performance with two-way e2e metrics, since DTN/BP offers many interactive options.   
With very kind regards,
ctselikis

Tselikis Christos 
Hellenic Aerospace Industry S.A.
Electronics Engineering Department  
e-mail: [email-masked] 
PO Box 23, GR-32009 Schimatari, Greece
Tel. : +30 22620 52561
Fax: +30 22620 52910

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Razvan BEURAN
Sent: Friday, September 07, 2012 3:01 AM
To: Ivancic, William D. (GRC-RHN0)
Cc: [email-masked]
Subject: Re: [dtn-users] On measuring the DTN2 performance

Hi all,

Best wishes,
Razvan

---

From: ccaini <[email-masked]>
Date: Fri, 07 Sep 2012 14:10:14 +0200

Dear all,
     as telecommunication professor at the University of Bologna I am the
supervisor of the development of the DTNperf tool. I would like to thank
Razvan, William and other people for their comments and suggestions on
DTNperf.
Let me just summarized the development of DTNperf:
DTNperf 1.x (by Piero Cornice) was conceived as a tool to evaluate
perfromance on DTN continously connected networks, but with long delays,
like GEO satellites, and it design takes after the IPerf tool. It was
"stop
and wait": a bundle was sent and then we had to waith for the reception at
sender site of the delivered status reports before sending another bundle.
DTNperf 2.x (Piero Cornice and Marco Livini) aim was twofold

On Fri, 7 Sep 2012 09:00:47 +0900 (JST), Razvan BEURAN <[email-masked]>
wrote:
>> DTN perf and DTNtraceroute are interesting concepts.  Do you have a
>> writeup on the operations concepts.  IMHO, performance needs consider
>> this is a disconnected network.  Return path may be totally different
>> than forward path.  It probably would be good to be able to have
>> sender-to-receiver option without an ACK similar to the original ttcp
and
>> an option in nuttcp.  If you have a back channel (like in an
experimental
>> testbed), you can get one-way performance.  Basically, think about how
>> the operation would work if none of the these as continuous
connectivity
>> and every bundle has to be stored, carried and forwarded over a
multi-hop
>> system and the forwarding agents are changing in the topology.  That is
>> much different than a string-of-pearls  fully connected set of
forwarding
>> agents.

---

From: ccaini <[email-masked]>
Date: Fri, 07 Sep 2012 14:46:22 +0200

This time the text should be complete. Sorry for the previous e-mail.
Carlo Caini

Dear all,
     as telecommunication professor at the University of Bologna I am the
supervisor of the development of the DTNperf tool. I would like to thank
Razvan, William and other people for their comments and suggestions on
DTNperf.
Let me just summarized the development of DTNperf:
DTNperf 1.x (by Piero Cornice) was conceived as a tool to evaluate
perfromance on DTN continously connected networks, but with long delays,
like GEO satellites, and it design takes after the IPerf tool. It was
"stop
and wait": a bundle was sent and then we had to waith for the reception at
sender site of the delivered status reports before sending another bundle.
DTNperf 2.x (Piero Cornice and Marco Livini) aim is twofold: to evaluate
goodput at application layer, as before, and to collect status reports log.
This latter feature is very important in the presence of disruption and/or
network partitioning. For exemple, in a data mule experiment it is more
important to know if and when bundle have been delivered, if there was
fragmentation, and so on, than to know the actual goodput. Other features
of 2.x: it is possible to send "w" bundles in parallel, which is important
to fill the pipe in the presence of multiple hops, or to exploit two
parallel TCP connections both active on the same hop (e.g. in DTN2 with
ALWAYSON from node A to B and viceversa). Another freature is the
possibility of sending a file, which can be segmented into multiple bundles
of the wanted dimension, which is useful in DTN networks that do not
support fragmentation.
DTNperf 3.x (by Michele Rodolfi, supervised by me and by Piero Cornice)
want to be a significant step forward.
The most important novelties are:
1) new monitor mode; status reports can be sent to and logged by a third
node, acting as monitor. In a testbed DTN nodes can be connected to this
monitor node by means of links not affected by any impairment and used only
for management, not to send data. This is aligned with what suggested by
William Ivancic.
2) new additional "rate based" bundle transmission. Bundle can be sent at
a certain rate, independently from the reception of any ACKs, like UDP
packets. This feature should well match the comment by Razvan Beuran, who,
like me, seems not to happy to see, in case of no feedback, the dtnperf 2.x
sender "frozen", waiting for a status delivered report that maybe will
never arrive.
3) Both DTN2 and ION support. DTNperf 3.x has been conceived to support
both DTN2 and ION. It should be a unique application, able to recognize at
run time if on a given node either DTN2 or ION BP implementation is active,
and to call the appropriate DTN2 or ION APIs accordingly.
My and my students commitment is to provide with DTNperf 3.x the DTN
community with a useful evaluation/log tool, independent from the BP
implementation, and of course interoperable.
A preliminary version of DTNperf 3.x is going to be released in the next
weeks, unfortunately without the ION support yet. This will be added
immediately after, as it is the thesis work of another student, Anna
D'Amico, who is going to start in October.
I thank you all for your past and future comments and suggestions.
Yours,
  Carlo Caini

and
experimental
connectivity
multi-hop
forwarding

---

From: "Ivancic, William D. (GRC-RHN0)" <[email-masked]>
Date: Fri, 7 Sep 2012 08:01:13 -0500

On Sep 7, 2012, at 5:48 AM, TSELIKIS.Christos wrote:

> performance in DTNs", perhaps some other metrics (except from throughput and delay) might provide more accurate descriptions (one way metrics such as delivery rate, loss, etc, which have already been studied as well).

For RFC5050 DTNs where there is a bundle lifetime, than delay should be insignificant.  The metric is successful delivery of a bundle.  Think of it like this:  I send you a birthday gift. Your birthday is one month away.  The gift says do not open until your birthday.  Whether or not you receive that bundle tomorrow or in a month is meaningless.  So long as get it before your birthday.  Since you put a lifetime on the bundle, you are now allowing the network to decide when to deliver.  IMHO, the successful delivery, overall bandwidth utilization, overall storage utilization  and overall processing utilization are the metrics.  How much total network resources are you using to successfully deliver that bundle?  You may flood to improve chances of delivering a bundle, but that will use a lot of resources.  

Meaningful metrics are extremely deployment specific.  Is my network heavily used in which case flooding may congest and be worse?  Are my nodes power constrained in which case excessive transmissions my drain my batteries?  Being able to measure is most important.  What is good or bad may be different for different deployments.

---

From: Razvan BEURAN <[email-masked]>
Date: Tue, 11 Sep 2012 17:23:31 +0900 (JST)

Dear William,

I found your brief analysis very useful, and I totally agree with the fact 
that meaningful metric are deployment/application specific. Are you by any 
chance aware of any thorough study that looks at the appropriate metrics 
for different classes of DTNs?

A quick search didn't provide any useful results, and I believe that such 
a theoretical effort would be a great starting point for a discussion on 
DTN performance. If no such study exists it would be great to make one :-)

Best wishes,
Razvan

---

From: <[email-masked]>
Date: Tue, 11 Sep 2012 11:09:17 +0100

Razvan,

for thoughts on DTN metrics, look into Ioannis Psaras's work: his six years since dtn paper and accompanying tech report.

http://www.ee.ucl.ac.uk/~uceeips/UCL_site/About...html

Lloyd Wood
http://sat-net.com/L.Wood/

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Razvan BEURAN [[email-masked]]
Sent: 11 September 2012 09:23
To: Ivancic, William D. (GRC-RHN0)
Cc: [email-masked]
Subject: Re: [dtn-users] On measuring the DTN2 performance

Dear William,

Best wishes,
Razvan

---

From: "Ivancic, William D. (GRC-RHN0)" <[email-masked]>
Date: Tue, 11 Sep 2012 07:20:22 -0500

Lloyd,
This "Six Years"  paper is right on  target.

Razvan,

I know of know studies in this area.  I've seen little work done in actually moving any traffic around with DTNs.  I've also seen little in the way of running DTNs over disconnected networks.  Most of the work appears to be a sting-of-pearls space scenario with maybe three or four forwarding agents and no mix of traffic or multihoming and not much in the way of complex routing. CGR works fine in this scenario.

Military applications are probably the most complex scenarios to date and I have seen little done here.  Some simple demonstrations with little traffic.  My understanding is that the military is JUST starting to "really" evaluate DTN in real scenarios.  I am not use what the traffic mix is.

I would say, making your own scenario and use case would be a good start.  Perhaps first responders (but they may have little need for DTN as they are more of a single hop store and forward scenario).  Perhaps one of the generic scenarios presented in this draft may help.  But you will need to generate some type of traffic pattern.

http://ietfreport.isoc.org/ids/draft-ivancic-scf-problem-statement-00.txt

- Will

On Sep 11, 2012, at 6:09 AM, <[email-masked]<mailto:[email-masked]>> wrote:

Razvan,

Lloyd Wood

Dear William,

Best wishes,
Razvan

Hi all,

******************************
William D. Ivancic
Phone 216-433-3494
Fax 216-433-8705
Networking Lab 216-433-2620
Mobile 440-503-4892
http://roland.grc.nasa.gov/~ivancic

---

From: "Ivancic, William D. (GRC-RHN0)" <[email-masked]>
Date: Tue, 11 Sep 2012 07:32:48 -0500

Lloyd,

Razvan,

I know of NO studies in this area.  I've seen little work done in actually moving any traffic around with DTNs.  I've also seen little in the way of running DTNs over disconnected networks.  Most of the work appears to be a sting-of-pearls space scenario with maybe three or four forwarding agents and no mix of traffic or multihoming and not much in the way of complex routing. CGR works fine in this scenario.

- Will

Razvan,

Lloyd Wood

Dear William,

Best wishes,
Razvan

Hi all,

---

From: "Ivancic, William D. (GRC-RHN0)" <[email-masked]>
Date: Tue, 11 Sep 2012 08:05:53 -0500

Another thought on this topic.  Without meaningful measurable metrics, it will be very difficult to perform many aspects of network management.

- Will

---

From: TSELIKIS.Christos <[email-masked]>
Date: Wed, 12 Sep 2012 10:29:10 +0300

Hi there,

Dear Will, after your comments it is clear now that we have user-application-related metrics and network utilization metrics (dtnbone may be a source for information regarding the latter). Also, delay is not so vital for BP because of the bundle expiration time that you mentioned. Still, the study of how e2e delay varies below that limit with respect to some parameters should not be neglected. A good metric is bw utilization which has already been studied in "Implementing Delay Tolerant Networking", M. Demmer et al. New metrics could be defined taking into account the above. Finally, multiple large-volume flows injected in DTN infrastructure may reveal interesting performance-related / operational characteristics of DTN/BP configurations, see "DTN: An Architectural Retrospective", K. Fall, S. Farrell.

With very kind regards,

Tselikis Christos 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ivancic, William D. (GRC-RHN0)
Sent: Tuesday, September 11, 2012 4:06 PM
To: Razvan BEURAN
Cc: [email-masked]
Subject: Re: [dtn-users] On measuring the DTN2 performance

- Will