Subject: [Lwip] fragment forwarding implementation and performance report

From: Rahul Jadhav <[email-masked]>
Date: Mon, 8 Oct 2018 16:47:00 +0530

<sending to 6lo, lwig WGs because both have relevant drafts>

Hello All,

We tried experimenting with the virtual reassembly buffer and fragment
forwarding drafts.

One fundamental characteristic that has major implications on fragment
forwarding performance is its behavior with realistic 802.15.4 RF
(especially when a train of fragments are simultaneously received and
transmitted). This is something which was not evaluated in any other
experiment.

You ll find the details of the implementation, test setup details and
performance result here:

https://github.com/nyrahul/ietf-data/blob/rst/6lo-fragfwd-perf-report.rst

Results are quite interesting: Simultaneous send/recv of fragments with
fragment forwarding has major implications on PDR/Latency.

Feedback most welcome.

Thanks,

Rahul

---

From: Rahul Jadhav <[email-masked]>
Date: Mon, 8 Oct 2018 16:55:46 +0530

Sorry.
The earlier mail had old report. The latest one is here:
https://github.com/nyrahul/ietf-data/blob/master/6lo-fragfwd-perf-report.rst

---

From: "Chakrabarti, Samita" <[email-masked]>
Date: Mon, 8 Oct 2018 16:53:51 -0400

Hi Rahul,

Thanks very much for sharing the fragment forwarding implementation results.
The following drafts are implemented as per the link you provided:
Fragment Forwarding drafts

   1. Virtual reassembly buffers in 6LoWPAN
   <https://datatracker.ietf.org/doc/draft-ietf-lwig-6lowpan-virtual-reassembly/>
   2. LLN Minimal Fragment Forwarding
   <https://datatracker.ietf.org/doc/draft-watteyne-6lo-minimal-fragment/>

>From the results, I noticed an observation that send rate 80s, and 40s are
doing better than the 160s  send rate with 50s forwarding fragment spacing.
Send rate Xs means sending fragmented packets at X sec interval - right?
I thought, the performance would improve with higher X value,  but that is
not true - perhaps due to increased payload size.
A graph or tabular result with same payload size with increased send
interval rate might be useful to figure out the optimal pacing time for
that payload - just a thought.

In general, very interesting results!

Also, it shows that by controlling the pacing of forwarding the fragments
the performance can be improved to a great degree in a medium to small size
mesh. ( in this example, 50 nodes).

What happens when you increase the mesh size ( aka number of nodes)?

Cheers,
-Samita

On Mon, Oct 8, 2018 at 7:17 AM Rahul Jadhav <[email-masked]> wrote:

> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_nyrahul_ietf-2Ddata_blob_rst_6lo-2Dfragfwd-2Dperf-2Dreport.rst&d=DwMFaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=pWMzx7FsqijEJPyfMBfn-HJss-wVVTf0K5y-cxCTXL8&m=PeFHI-ltr748QRhWwqigY8iNFPw9EcyFDwOeSrv6KQc&s=Rtytc7AFwMLDcwFQOSojZZZ3hiXl-j78GKTwYRi8Nw0&e=>
>
>
>
> _______________________________________________
> 6lo mailing list
> [email-masked]
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_6lo&d=DwICAg&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=pWMzx7FsqijEJPyfMBfn-HJss-wVVTf0K5y-cxCTXL8&m=PeFHI-ltr748QRhWwqigY8iNFPw9EcyFDwOeSrv6KQc&s=ebzWBVEJyovVUcFHM2mByigGnDBv0aoTSm21fmwa5vU&e=
>

---

From: Rahul Jadhav <[email-masked]>
Date: Tue, 9 Oct 2018 14:50:02 +0530

Thank you Samita for the feedback.
Please find my response inline.

Regards,
Rahul

> Send rate Xs means sending fragmented packets at X sec interval - right?

The payload size before fragmentation is different for 40s, 80s, 160s,
which is 256B, 512B and 1024B respectively. So at 160s the payload size is
1024B which will result in more fragments per payload. Thus any loss of a
single fragment increases the failure rate of the complete payload.
There is no 50s fragment forwarding spacing .. There is __50ms & 100ms__
inter fragment spacing in case of fragment forwarding that was introduced
on the original sender (not forwarders).

The send logic is, we wait for 40s/80s/160s time duration and then randomly
choose a timer within 1-10s to start sending the UDP payload. After that in
case of fragment forwarding, we wait 50ms (pacing delay) before every
fragment is sent on the original sender.

> not true - perhaps due to increased payload size.

Yes, the increased payload size causes higher number of fragments per
payload which increases payload loss probability since a single fragment
loss will cause complete payload failure.

> that payload - just a thought.

Yes this can be attempted. But as of current observations it seems that
unless you have smart pacing algorithm the use of fragment forwarding can
backfire considering that under the same conditions the per hop reassembly
works much better. We tried pacing with 50ms and 100ms delay ... with 100ms
the PDR of fragment forwarding is almost same as per hop reassembly but
with much higher end to end latency.

> What happens when you increase the mesh size ( aka number of nodes)?

Yes we can increase the mesh size but then i do not think it will change
the inferrences much. We also wanted to try different (higher) node
densities which i feel might further cause problems for fragment
forwarding. Among other things we also want to experiment with fragment
acknowledgement mechanism. But we havent really validated all these points.

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 9 Oct 2018 11:47:31 +0000

Hello Rahul and Samita :

I thought, the performance would improve with higher X value,  but that is not true - perhaps due to increased payload size.

Yes, the increased payload size causes higher number of fragments per payload which increases payload loss probability since a single fragment loss will cause complete payload failure.

[PT>] Yep: this shows the chances of not only losing the packet but buffer bloat on the receiver, which entails not having memory left for receiving normal packets. IOW this justifies doing recoverable fragments.

Also, it shows that by controlling the pacing of forwarding the fragments the performance can be improved to a great degree in a medium to small size mesh. ( in this example, 50 nodes).

Yes we can increase the mesh size but then I do not think it will change the inferences much. We also wanted to try different (higher) node densities which I feel might further cause problems for fragment forwarding. Among other things we also want to experiment with fragment acknowledgement mechanism. But we haven’t really validated all these points.
[PT>] In a real network we can expect more than one flow. Here the one packet appears smooth but that’s because there is nothing to interfere with. What if you kept on sending packets continuously, and over multiple crossing paths?
There are really 2 effects that justify pacing the fragments. With a channel hopping technology like TSCH, it is just to left the next hop forward the fragment, so the wait can be short. When using a single frequency, there is also avoidance of hidden terminal between consecutive fragments, which means that we have to wait for the fragments to relay over multiple hops.
This tells us that the fragment forwarding technique is a lot more efficient when coupled with 6TiSCH. None of this is a surprised, all was described to the group when we started (see my slides from a few IETFS ago, see slide 32 of https://datatracker.ietf.org/meeting/101/materials/slides-101-6lo-fragmentation-design-team-formation-update-00 ).

I hope others continue the great work that Rahul has started, in particular with TSCH and multiple flows.

Take care,

Pascal

---

From: "Chakrabarti, Samita" <[email-masked]>
Date: Tue, 9 Oct 2018 11:04:05 -0400

Hi Pascal,Thanks for the explanation on the following points.

On Tue, Oct 9, 2018 at 7:48 AM Pascal Thubert (pthubert) <[email-masked]>
wrote:

> *[PT>] Yep: this shows the chances of not only losing the packet but
> buffer bloat on the receiver, which entails not having memory left for
> receiving normal packets. IOW this justifies doing recoverable fragments.*
>
>
>

It would be great to try out the intelligent fragment-forwarding draft
(Pascal's draft) on the same setup and show the difference in efficiency.

Just curious, in your estimate, does the forwarding issue show up more on
6tisch network compared to other technologies that don't have such reliable
L2?
This is something to think about for implementation recommendations-- for
example, intelligent fragment forwarding will be needed for efficient use
6tish network with payload range X and Y.
Similarly, if other technologies mostly use small payloads and can
reasonably work well with implementing minimal fragment forwarding, then
documenting that information will be very useful for deployment.

Thanks,
-Samita

> Yes we can increase the mesh size but then I do not think it will change
> the inferences much. We also wanted to try different (higher) node
> densities which I feel might further cause problems for fragment
> acknowledgement mechanism. But we haven’t really validated all these points.
>
> *[PT>] In a real network we can expect more than one flow. Here the one
> packet appears smooth but that’s because there is nothing to interfere
> with. What if you kept on sending packets continuously, and over multiple
> crossing paths?*
>
> *There are really 2 effects that justify pacing the fragments. With a
> channel hopping technology like TSCH, it is just to left the next hop
> forward the fragment, so the wait can be short. When using a single
> frequency, there is also avoidance of hidden terminal between consecutive
> fragments, which means that we have to wait for the fragments to relay over
> multiple hops.*
>
> *This tells us that the fragment forwarding technique is a lot more
> efficient when coupled with 6TiSCH. None of this is a surprised, all was
> described to the group when we started (see my slides from a few IETFS ago,
> see slide 32 of
> https://datatracker.ietf.org/meeting/101/materials/slides-101-6lo-fragmentation-design-team-formation-update-00
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_meeting_101_materials_slides-2D101-2D6lo-2Dfragmentation-2Ddesign-2Dteam-2Dformation-2Dupdate-2D00&d=DwMGaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=pWMzx7FsqijEJPyfMBfn-HJss-wVVTf0K5y-cxCTXL8&m=nVLkPPTF9g3I6X86AD76uwafjmV4U0IYkfZ0N8AQ25g&s=pTBiqO4WbYobDfj04cufLA9ev4-9M6XqKX-6pPxCCSQ&e=>
> ).*
>
>
>
> *I hope others continue the great work that Rahul has started, in
> particular with TSCH and multiple flows.*
>
>
>
> *Take care,*
>
>
>
> *Pascal*
>
>
>

---

From: "Chakrabarti, Samita" <[email-masked]>
Date: Tue, 9 Oct 2018 11:09:15 -0400

Hi Rahul,

Thanks for the clarification points.  If convenient, an analysis of optimum
payload recommendation for minimal fragment forwarding draft will be great!
In that case, the draft can recommend  that value or at least throw the
information in the appendix section for a guidance.

Great work! We  should do more such collaboration with lwip wg!

Cheers,
-Samita

On Tue, Oct 9, 2018 at 5:20 AM Rahul Jadhav <[email-masked]> wrote:

>> From the results, I noticed an observation that send rate 80s, and 40s
>> are doing better than the 160s  send rate with 50s forwarding fragment
>> spacing. Send rate Xs means sending fragmented packets at X sec interval -
>> right?
>>
>
> The send logic is, we wait for 40s/80s/160s time duration and then
> randomly choose a timer within 1-10s to start sending the UDP payload.
> After that in case of fragment forwarding, we wait 50ms (pacing delay)
> before every fragment is sent on the original sender.
>
>
>> I thought, the performance would improve with higher X value,  but that
>> is not true - perhaps due to increased payload size.
>>
>