Subject: Re: [6tisch] [6lo] Thomas' review of draft-thubert-6lo-forwarding-fragments-04

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 5 Apr 2017 15:18:45 +0000

Hello Thomas :

All in all, your points are well understood, and I will restructure a bit and add words. But, before I spend too much of my time on this draft, I’d like to see the WG interest clearly expressed. So far I saw you and Michel expressing clear support. Is the group behind us?

Please see below ;

point 1: clearer intro needed

from my understanding, this draft introduces two things: (1) a way of forwarding fragments so as to avoid to have to reassemble at every hop and (2) an end-to-end protocol to request lost fragments to be retransmitted. I think item (1) is really the main contribution of the draft, and (2) is a necessary corollary to making (1) work. The abstract and intro, however, seem to focus almost exclusively on (2), not clearly stating (1). A single sentence at the very beginning stressing the two contributions would go a long way IMO.

[Pascal] OK, I’ll improve that on the next revision. Note that the initial title of the draft was draft-thubert-6lowpan-simple-fragment-recovery<https://tools.ietf.org/html/draft-thubert-6lowpan-simple-fragment-recovery-00> ; since then, I moved the focus on the capability to streamli,ne the fragments over a mesh. But both are really important. Forwarding fragments is now the title, and the draft improves the throughput by streamlining while avoiding buffer bloat inside the network.
Yet, recovering the missing frags is important as well, avoids the bloat in the end point. Looking at the big picture, it’s very hard to do one without the other, the techniques benefit from one another and are not separable.

---

point 2: justification

Section 3 provides a rationale as to why this draft is needed. It focuses almost exclusively on reliability: if the link is 99% reliable, then the probability that 5 fragments, etc. IMO that doesn't make much sense as each fragment will be link-layer ACK'ed and retransmitted (at the link layer, i.e."transparently" to 6LoWPAN). There is of course a probability that all retries fail, but justifying the draft by assuming there are no link-layer ACKs make little sense. I may completely misunderstand, in which case, please point that out.

[Pascal] We want to make the transmission reliable over multihop. I agree with what you are saying over one hop. But in the multihop scenario, if a hop down the path temporarily fails, the retries there will exhaust, but the node that did the fragmentation will not be aware (no nack all the way back), as opposed to the single hop case. So it will keep sending the next fragments, contributing to the problem, wasting time and network resources, etc...

The major rationale IMO for this draft is that it doesn't require intermediary nodes to reassemble! Why not just say that? Reassembly at each hop yields several problems, the biggest one in my experience is that of memory management (if I'm missing one fragment from a packet, and a fragment from a second packet arrive, and I have only one assembly buffer, what should I do?).

[Pascal] Yes, this is one of the problems I illustrated the most at the 6lo meeting, as you saw. Now, the bloat may happen in the receive end node, even when fragments are streamlined. It happens as soon as one fragment is lost. So forwarding fragments protects the network, making the fragments reliable protects the packet and the end point.

I feel like I'm missing something, or even reading a draft that is not what I expected it to be...

[Pascal] Can’t say. But both are needed and help one another to the point that they cannot be separated. The draft addresses all the points in my presentation at 6lo (https://www.ietf.org/proceedings/98/slides/slides-98-6lo-fragmentation-forwarding-issues-in-6lowpan-00.pdf), but the hidden terminal which is not an issue for 6TiSCH and is only alleviated by the proposed flow control and the window of one fragment. The value that you focus on is discussed slides 9-12, and was a focus of the discussion.
---

point 3: Section 5 Overview is not an overview

Section 5 is called "Overview" and appears before the actual proposal is described. From the title, I was expecting a couple of paragraphs with the main idea, maybe a diagram or two to make the big picture clear. Instead, the section contains a discussion about very subtle points, including congestion control, interaction with link-layer protocols, timeout calculation and security.

To summarize, the text in this section is interesting and should certainly appear in the specification, but as discussion points AFTER the main idea is presented.

[Pascal] Ack
---

point 4: the confusing use of the word "acknowledgment"

While understandable, the use of acknowledgment which can refer to L2 ACKs or RFRAG-ACK messages is confused. I suggest to use the term "RFRAG-ACK" rather than the generic "Acknowledgement".

[Pascal] Ack

---

point 5: label switching

The label switching mechanism is elegant as the labels are locally significant only, with no need to maintain a network-wide label registry. The document should state so.

[Pascal] Ack

For the rest I answered in line, please see below.

Take care,

Pascal

===

6lo                                                      P. Thubert, Ed.
Internet-Draft                                             Cisco Systems
Intended status: Standards Track                                  J. Hui
Expires: July 14, 2017                                         Nest Labs
                                                        January 10, 2017

                  LLN Fragment Forwarding and Recovery
               draft-thubert-6lo-forwarding-fragments-04

Abstract

   In order to be routed, a fragmented 6LoWPAN packet must be
   reassembled at every hop of a multihop link where lower layer
   fragmentation occurs.  Considering that the IPv6 minimum MTU is 1280
   bytes, and that an 802.15.4 frame can have a payload limited to 74
   bytes in the worst case, a packet might end up fragmented into as
   many as 18 fragments by 6LoWPAN.
TW> not sure the term "shim layer" is widely used. I suggest to replace simply by "6LoWPAN" (done already)
   If a single one of
   those fragments is lost in transmission, all fragments must be
   resent, further contributing to the congestion that might have caused
   the initial packet loss.  This specification introduces a simple protocol to
   forward and recover individual fragments that might be lost over
   multiple hops between 6LoWPAN endpoints.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on July 14, 2017.

Copyright Notice

   Copyright (c) 2017 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents

Thubert & Hui             Expires July 14, 2017                 [Page 1]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Rationale . . . . . . . . . . . . . . . . . . . . . . . . . .   4
   4.  Requirements  . . . . . . . . . . . . . . . . . . . . . . . .   5
   5.  Overview  . . . . . . . . . . . . . . . . . . . . . . . . . .   6
   6.  New Dispatch types and headers  . . . . . . . . . . . . . . .   8
     6.1.  Recoverable Fragment Dispatch type and Header . . . . . .   8
     6.2.  Fragment acknowledgment Dispatch type and Header  . . . .   8
   7.  Fragments Recovery  . . . . . . . . . . . . . . . . . . . . .  10
   8.  Forwarding Fragments  . . . . . . . . . . . . . . . . . . . .  11
     8.1.  Upon the first fragment . . . . . . . . . . . . . . . . .  12
     8.2.  Upon the next fragments . . . . . . . . . . . . . . . . .  13
     8.3.  Upon the fragment acknowledgments . . . . . . . . . . . .  13
   9.  Security Considerations . . . . . . . . . . . . . . . . . . .  14
   10. IANA Considerations . . . . . . . . . . . . . . . . . . . . .  14
   11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .  14
   12. References  . . . . . . . . . . . . . . . . . . . . . . . . .  14
     12.1.  Normative References . . . . . . . . . . . . . . . . . .  14
     12.2.  Informative References . . . . . . . . . . . . . . . . .  15
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  16

1.  Introduction

   In most Low Power and Lossy Network (LLN) applications, the bulk of
   the traffic consists of small chunks of data (in the order few bytes
   to a few tens of bytes) at a time.  Given that an 802.15.4 frame can
   carry 74 bytes or more in all cases, fragmentation is usually not
   required.  However, and though this happens only occasionally, a
   number of mission-critical applications do require the capability to
   transfer larger chunks of data, for instance to support firmware
   update of the LLN nodes, or an extraction of logs from LLN nodes.
   In the former case, the large chunk of data is transferred to the LLN
   node, whereas in the latter, the large chunk flows away from the LLN
   node.  In both cases, the size can be on the order of 10 kB or
   more, and an end-to-end reliable transport is required.

   Mechanisms such as TCP or application-layer segmentation will be used
   to support end-to-end reliable transport.  One option to support bulk

Thubert & Hui             Expires July 14, 2017                 [Page 2]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

   data transfer over a frame-size-constrained LLN is to set the Maximum
   Segment Size to fit within the link maximum frame size.  Doing so,
   however, can add significant header overhead to each 802.15.4 frame.
   This causes the end-to-end transport to be intimately aware of the
   delivery properties of the underlaying LLN, which is a layer
   violation.

   An alternative mechanism uses 6LoWPAN fragmentation in
   addition to transport or application-layer segmentation.  Increasing
   the Maximum Segment Size reduces header overhead by the end-to-end
   transport protocol.  It also encourages the transport protocol to
   reduce the number of outstanding datagrams, ideally to a single
   datagram, thus reducing the need to support out-of-order delivery
   common to LLNs.

   [RFC4944] defines a datagram fragmentation mechanism for LLNs.
   However, because [RFC4944] does not define a mechanism for recovering
   fragments that are lost, datagram forwarding fails if even one
   fragment is not delivered properly to the next IP hop.  End-to-end
   transport mechanisms will require retransmission of all fragments,
   wasting resources in an already resource-constrained network.

   Past experience with fragmentation has shown that mis-associated or
   lost fragments can lead to poor network behavior and, eventually,
   trouble at application layer.  The reader is encouraged to read
   [RFC4963] and follow the references for more information.  That
   experience led to the definition of the Path MTU discovery [RFC1191]
   protocol that limits fragmentation over the Internet.

   For one-hop communications, a number of media propose a local
   acknowledgment mechanism that is enough to protect the fragments.  In
   a multihop environment, an end-to-end fragment recovery mechanism
   might be a good complement to a hop-by-hop MAC level recovery.  This
   draft introduces a simple protocol to recover individual fragments
   between 6LoWPAN endpoints.  Specifically in the case of UDP, valuable
   additional information can be found in UDP Usage Guidelines for
   Application Designers [RFC5405].

2.  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

   Readers are expected to be familiar with all the terms and concepts
   that are discussed in "IPv6 over Low-Power Wireless Personal Area
   Networks (6LoWPANs): Overview, Assumptions, Problem Statement, and

Thubert & Hui             Expires July 14, 2017                 [Page 3]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

   Goals" [RFC4919] and "Transmission of IPv6 Packets over IEEE 802.15.4
   Networks" [RFC4944].

   ERP

      Error Recovery Procedure.

   6LoWPAN endpoints

      The LLN nodes in charge of generating or expanding a 6LoWPAN
      header from/to a full IPv6 packet.  The 6LoWPAN endpoints are the
      points where fragmentation and reassembly take place.
TW> should we add a sentence that states that one of those point is NOT necessarily the LBR?

[Pascal]Not sure there is a need; this is not a ROLL document, there is not necessary such a thing as a 6LBR

3.  Rationale

   There are a number of use cases for large packets in Wireless Sensor
   Networks.  Such usages may not be the most typical or represent the
   largest amount of traffic over the LLN; however, the associated
   functionality can be critical enough to justify extra care for
   ensuring effective transport of large packets across the LLN.

   The list of those use cases includes:

   Towards the LLN node:

      Packages of Commands:  A number of commands or a full
         configuration can be packaged as a single message to ensure
         consistency and enable atomic execution or complete rollback.
         Until such commands are fully received and interpreted, the
         intended operation will not take effect.

      Firmware update:  For example, a new version of the LLN node
         software is downloaded from a system manager over unicast or
         multicast services.  Such a reflashing operation typically
         involves updating a large number of similar LLN nodes over a
         relatively short period of time.

   From the LLN node:

      Waveform captures:  A number of consecutive samples are measured
         at a high rate for a short time and then transferred from a
         sensor to a gateway or an edge server as a single large report.

      Data logs:  LLN nodes may generate large logs of sampled data for
         later extraction.  LLN nodes may also generate system logs to
         assist in diagnosing problems on the node or network.

Thubert & Hui             Expires July 14, 2017                 [Page 4]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

      Large data packets:  Rich data types might require more than one
         fragment.

   Uncontrolled firmware download or waveform upload can easily result
   in a massive increase of the traffic and saturate the network.

   When a fragment is lost in transmission, all fragments are resent,
   further contributing to the congestion that caused the initial loss,
   and potentially leading to congestion collapse.

   This saturation may lead to excessive radio interference, or random
   early discard (leaky bucket) in relaying nodes.  Additional queuing
   and memory congestion may result while waiting for a low power next
   hop to emerge from its sleeping state.

   To demonstrate the severity of the problem, consider a fairly
   reliable 802.15.4 frame delivery rate of 99.9% over a single 802.15.4
   hop.  The expected delivery rate of a 5-fragment datagram would be
   about 99.5% over a single 802.15.4 hop.  However, the expected
   delivery rate would drop to 95.1% over 10 hops, a reasonable network
   diameter for LLN applications.  The expected delivery rate for a
   1280-byte datagram is 98.4% over a single hop and 85.2% over 10 hops.
TW> I don't think this paragraph makes much sense.
TW> You would have link-layer ACKs and retries to ensure that all fragments reach the next hop.

[Pascal] In order to fit within transport time outs, you want to limit the number of retries to an acceptable time. In the 99.9% I assume 90% delivery, 3 tries, and complete independence of the 3 tries. All of these assumptions are completely wrong, but that gives an idea of how the problems adds up in a mesh. Still I can remove the paragraph, since, after all , it is not needed in a solution draft.

   Considering that [RFC4944] defines an MTU is 1280 bytes and that in
   most incarnations (but 802.15.4G) a 802.15.4 frame can limit the MAC
   payload to as few as 74 bytes, a packet might be fragmented into at
   least 18 fragments by 6LoWPAN.  Taking into account
   the worst-case header overhead for 6LoWPAN Fragmentation and Mesh
   Addressing headers will increase the number of required fragments to
   around 32.  This level of fragmentation is much higher than that
   traditionally experienced over the Internet with IPv4 fragments.  At
   the same time, the use of radios increases the probability of
   transmission loss and Mesh-Under techniques compound that risk over
   multiple hops.

4.  Requirements

   This specification proposes a method to recover individual fragments between
   LLN endpoints.
TW> add that endpoints can be multiple hops away from one another?

[Pascal] Exactly, I’ll add the words

   The method is designed to fit the following
   requirements of a LLN (with or without a Mesh-Under routing
   protocol):

   Number of fragments

      The recovery mechanism must support highly fragmented packets,
      with a maximum of 32 fragments per packet.
TW> resulting length of the packet? The abstract implies a 32*74=2368 byte limit. Is that correct?

[Pascal] Yes, depends on the security overhead which is variable. But I’d say that the reasonable goal of MTU=2048, which is the usual IEEE bar, is easily achieved.

   Minimum acknowledgment overhead

Thubert & Hui             Expires July 14, 2017                 [Page 5]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

      Because the radio is half duplex, and because of silent time spent
      in the various medium access mechanisms, an acknowledgment
      consumes roughly as many resources as data fragment.

      The recovery mechanism should be able to acknowledge multiple
      fragments in a single message and not require an acknowledgment at
      all if fragments are already protected at a lower layer.

TW> maybe specify here that we are talking about end-to-end ACKs between endpoints, not L2 ACKs

[Pascal] Will do

   Controlled latency

      The recovery mechanism must succeed or give up within the time
      boundary imposed by the recovery process of the Upper Layer
      Protocols.

   Support for out-of-order fragment delivery

      A Mesh-Under load balancing mechanism such as the ISA100 Data Link
      Layer can introduce out-of-sequence packets.

TW> I don't fully understand why we focus so much on mesh-under.
TW> If you have a fully mesh-under network, 4944 fragmentation results in fragment forwarding.
TW> I would remove the repeated mentioning on mesh-under (as well as ISA100)

[Pascal] I’ll move the whole section to appendix; the draft is more needed in route over but still beneficial to mesh under, for the benefits that are not the buffer bloat. These include flow control, loss detection, and recovery.

      The recovery mechanism must account for packets that appear lost
      but are actually only delayed over a different path.

   Optional congestion control

      The aggregation of multiple concurrent flows may lead to the
      saturation of the radio network and congestion collapse.

      The recovery mechanism should provide means for controlling the
      number of fragments in transit over the LLN.

5.  Overview

   Considering that a multi-hop LLN can be a very sensitive environment
   due to the limited queuing capabilities of a large population of its
   nodes, this specification recommends a simple and conservative approach to
   congestion control, based on TCP congestion avoidance.
TW> "based on" or "inspired by"

TW> The "overview" section delves immediately into subtle discussions about congestion control.
TW> At this point, the reader (e.g. me) have no idea what the basic principle is.
TW> Discussions about subtleties such as congestion control seem out of place here.

[Pascal] Understood, will restructure, add words; that’s a bit of work so please give me some time.

   Congestion on the forward path is assumed in case of packet loss, and
   packet loss is assumed upon time out.  This specification allows to control
   the number of outstanding fragments, that have been transmitted but
   for which an acknowledgment was not received yet.  It must be noted
   that the number of outstanding fragments should not exceed the number
   of hops in the network, but the way to figure the number of hops is
   out of scope for this document.

   Congestion on the forward path can also be indicated by an Explicit
   Congestion Notification (ECN) mechanism.  Though whether and how ECN
   [RFC3168] is carried out over the LoWPAN is out of scope, this draft

Thubert & Hui             Expires July 14, 2017                 [Page 6]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

   provides a way for the destination endpoint to echo an ECN indication
   back to the source endpoint in an acknowledgment message as
   represented in Figure 5 in Section 6.2.

TW> The following paragraph talks about how a link-layer transmits multiple frames in a row.
TW> The link-layer does so whether those frames are fragments or regular packets.
TW> I would remove this discussion, which seems to be link-layer only, and not specific to fragment forwarding.

[Pascal] I can remove the text, but the problem remains, and your point, while largely correct, is not fully so. The transport layer may do some flow control based on command response, windowing and ECN, and thus protect against bursts of packets, but it cannot do a thing against bursts of fragments. So essentially, we are echoing some of the transport functionality at L2+ to provide those service to the fragments.

   It must be noted that congestion and collision are different topics.
   In particular, when a mesh operates on a same channel over multiple
   hops, then the forwarding of a fragment over a certain hop may
   collide with the forwarding of a next fragment that is following over
   a previous hop but in a same interference domain.  This draft enables
   an end-to-end flow control, but leaves it to the sender stack to pace
   individual fragments within a transmit window, so that a given
   fragment is sent only when the previous fragment has had a chance to
   progress beyond the interference domain of this hop.  In the case of
   6TiSCH [I-D.ietf-6tisch-architecture], which operates over the
   TimeSlotted Channel Hopping [I-D.ietf-6tisch-tsch] (TSCH) mode of
   operation of IEEE802.14.5, a fragment is forwarded over a different
   channel at a different time and it make full sense to fire a next
   fragment as soon as the previous fragment has had its chance to be
   forwarded at the next hop, retry (ARQ) operations included.

   From the standpoint of a source 6LoWPAN endpoint, an outstanding
   fragment is a fragment that was sent but for which no explicit
   acknowledgment was received yet.  This means that the fragment might
   be on the way, received but not yet acknowledged, or the
   acknowledgment might be on the way back.  It is also possible that
   either the fragment or the acknowledgment was lost on the way.

   Because a meshed LLN might deliver frames out of order, it is
   virtually impossible to differentiate these situations.  In other
   words, from the sender's standpoint, all outstanding fragments might
   still be in the network and contribute to its congestion.  There is
   an assumption, though, that after a certain amount of time, a frame
   is either received or lost, so it is not causing congestion anymore.
   This amount of time can be estimated based on the round trip delay
   between the 6LoWPAN endpoints.  The method detailed in [RFC6298] is
   recommended for that computation.

   The reader is encouraged to read through "Congestion Control
   Principles" [RFC2914].  Additionally [RFC2309] and [RFC5681] provide
   deeper information on why this mechanism is needed and how TCP
   handles Congestion Control.  Basically, the goal here is to manage
   the amount of fragments present in the network; this is achieved by
   to reducing the number of outstanding fragments over a congested path
   by throttling the sources.

   Section 7 describes how the sender decides how many fragments are
   (re)sent before an acknowledgment is required, and how the sender
   adapts that number to the network conditions.

Thubert & Hui             Expires July 14, 2017                 [Page 7]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

6.  New Dispatch types and headers

   This specification extends "Transmission of IPv6 Packets over IEEE
   802.15.4 Networks" [RFC4944] with 4 new dispatch types, for
   Recoverable Fragments (RFRAG) headers with or without Acknowledgment
   Request, and for the Acknowledgment back, with or without ECN Echo.

            Pattern    Header Type
          +------------+-----------------------------------------------+
          | 11  101000 | RFRAG      - Recoverable Fragment             |
          | 11  101001 | RFRAG-AR   - RFRAG with Ack Request           |
          | 11  101010 | RFRAG-ACK  - RFRAG Acknowledgment             |
          | 11  101011 | RFRAG-AEC  - RFRAG Ack with ECN Echo          |
          +------------+-----------------------------------------------+

             Figure 1: Additional Dispatch Value Bit Patterns

   In the following sections, the semantics of "datagram_tag",
   "datagram_offset" and "datagram_size" and the reassembly process are
   changed from [RFC4944] Section 5.3.  "Fragmentation Type and Header".
   The size and offset are expressed on the compressed packet per
   [RFC6282] as opposed to the uncompressed - native packet - form.

6.1.  Recoverable Fragment Dispatch type and Header

                            1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |1 1 1 0 1 0 0 X|datagram_offset|         datagram_tag          |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Sequence |    datagram_size    |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                                  X set == Ack Requested

          Figure 2: Recoverable Fragment Dispatch type and Header

   X: 1 bit; When set, the sender requires an Acknowledgment from the
      receiver

   Sequence:  5 bits; The sequence number of the fragment.  Fragments
      are numbered [0..N] where N is in [0..31].

6.2.  Fragment acknowledgment Dispatch type and Header

   The specification also defines a 4-octet acknowledgment bitmap that
   is used to carry selective acknowledgments for the received
   fragments.  A given offset in the bitmap maps one to one with a given
   sequence number.

Thubert & Hui             Expires July 14, 2017                 [Page 8]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

   The offset of the bit in the bitmap indicates which fragment is
   acknowledged as follows:

                            1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |           Acknowledgment Bitmap                               |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        ^                   ^
        |                   |    bitmap indicating whether:
        |                   +--- Fragment with sequence 10 was received
        +----------------------- Fragment with sequence 00 was received

                 Figure 3: Acknowledgment bitmap encoding

TW> rewrote the paragraph below

[Pascal] Cool, thanks! Will use that text.

   Figure 4 shows an example Acknowledgment bitmap which indicates that all fragments from
   sequence 0 to 20 were received, except for fragments 1, 2 and 16 that were
   either lost or are still in the network over a slower path.

                            1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |1 0 0 1 1 1 1 1|1 1 1 1 1 1 1 1|0 1 1 1 1 0 0 0|0 0 0 0 0 0 0 0|
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                   Figure 4: Example acknowledgment bitmap

   The acknowledgment bitmap is carried in a Fragment Acknowledgment as
   follows:

                            1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                       |1 1 1 0 1 0 1 Y|         datagram_tag          |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |          Acknowledgment Bitmap (32 bits)                      |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

        Figure 5: Fragment Acknowledgment Dispatch type and Header

   Y: 1 bit; Explicit Congestion Notification (ECN) signaling

      When set, the sender indicates that at least one of the
      acknowledged fragments was received with an Explicit Congestion
      Notification, indicating that the path followed by the fragments
      is subject to congestion.

   Acknowledgment Bitmap

Thubert & Hui             Expires July 14, 2017                 [Page 9]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

      An acknowledgment bitmap, whereby but at offset x indicates that
      fragment x was received.

7.  Fragments Recovery

   The Recoverable Fragments header RFRAG and RFRAG-AR deprecate the
   original fragment headers from [RFC4944] and replace them in the
   fragmented packets.
TW> are we really deprecating 4944 fragmentation, or adding another option?

[Pascal] The goal is deprecation, which means a slow replacement. What could be the point of keeping the old way?

   The Fragment Acknowledgment RFRAG-ACK is
   introduced as a standalone header in a message that is sent back to the
   fragment source endpoint as known by its MAC address.  This assumes
   that the source MAC address in the fragment (if any) and datagram_tag
   are enough information to send the Fragment Acknowledgment back to
   the source fragmentation endpoint.

TW> I get confused. Isn't the source MAC address changed at each hop?

[Pascal] Hopefully yes. You send the ack to the MAC from which you got the fragment from. That’s the guy right before you in the path of that packet. And the datagram-tag is unique only when associated with that MAC address.

   The 6LoWPAN endpoint that fragments the packets at the 6LoWPAN level (the
   sender) controls the Fragment Acknowledgments.  If may do that at any
   fragment to implement its own policy or perform congestion control
   which is out of scope for this document.
TW> I don't understand the previous sentence.

[Pascal] Meant to say that the source of the fragments may ask for an ack on any fragment it likes per its own policy.

   When the sender of the
   fragment knows that an underlying mechanism protects
TW> "protects" -> "ensures delivery of"?

[Pascal] OK.

   the Fragments
   already it MAY refrain from using the Acknowledgment mechanism, and
   never set the ACK Requested bit.  The 6LoWPAN endpoint that
   recomposes the packets at 6LoWPAN level (the receiver) MUST
   acknowledge the fragments it has received when asked to, and MAY
   slightly defer that acknowledgment.

   The sender transfers a controlled number of fragments and MAY flag
   the last fragment of a series with an acknowledgment request.  The
   received MUST acknowledge a fragment with the acknowledgment request
   bit set.
TW> reword to "the receiver which receives a fragment with the ACK Request bit set MUST send back an acknowledgment"?
[Pascal] OK.

   If any fragment immediately preceding an acknowledgment
   request is still missing, the receiver MAY intentionally delay its
   acknowledgment to allow in-transit fragments to arrive. Delaying the
   acknowledgment might defeat the round trip delay computation so it
   should be configurable and not enabled by default.

   The receiver interacts with the sender using an Acknowledgment
   message with a bitmap that indicates which fragments were actually
   received.  The bitmap is a 32 bit SWORD
TW> why SWORD?

[Pascal] Typo, DWORD.  Maybe replace by bit field?

   , which accommodates up to 32
   fragments and is sufficient for the 6LoWPAN MTU.  For all n in
   [0..31], bit n is set to 1 in the bitmap to indicate that fragment
   with sequence n was received, otherwise the bit is set to 0.  All
   zeros is a NULL bitmap that indicates that the fragmentation process
   was canceled by the receiver for that datagram.

   The receiver MAY issue unsolicited acknowledgments.  An unsolicited
   acknowledgment enables the sender endpoint to resume sending if it
   had reached its maximum number of outstanding fragments or indicate
   that the receiver has canceled the process of an individual
   datagram.  Note that acknowledgments might consume precious resources

Thubert & Hui             Expires July 14, 2017                [Page 10]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

   so the use of unsolicited acknowledgments should be configurable and
   not enabled by default.

   The sender arms a retry timer to cover the fragment that carries the
   Acknowledgment request.  Upon time out, the sender assumes that all
   the fragments on the way are received or lost.  The process must have
   completed within an acceptable time that is within the boundaries of
   upper layer retries.  The method detailed in [RFC6298] is recommended
   for the computation of the retry timer.  It is expected that the
   upper layer retries obey the same or friendly rules in which case a
   single round of fragment recovery should fit within the upper layer
   recovery timers.

   Fragments are sent in a round robin fashion: the sender sends all the
   fragments for a first time before it retries any lost fragment; lost
   fragments are retried in sequence, oldest first.  This mechanism
   enables the receiver to acknowledge fragments that were delayed in
   the network before they are actually retried.

   When the sender decides that a packet should be dropped and the
   fragmentation process canceled, it sends a pseudo fragment with the
   datagram_offset, sequence and datagram_size all set to zero, and no
   data.  Upon reception of this message, the receiver should clean up
   all resources for the packet associated to the datagram_tag.  If an
   acknowledgment is requested, the receiver responds with a NULL
   bitmap.

   The receiver might need to cancel the process of a fragmented packet
   for internal reasons, for instance if it is out of recomposition
   buffers, or considers that this packet is already fully recomposed
   and passed to the upper layer.  In that case, the receiver SHOULD
   indicate so to the sender with a NULL bitmap.  Upon an acknowledgment
   with a NULL bitmap, the sender MUST drop the datagram.

8.  Forwarding Fragments

   This specification enables intermediate routers to forward fragments
   with no intermediate reconstruction of the entire packet.  Upon the
   first fragment, the routers lay a label along the path that is
   followed by that fragment (that is IP routed), and all further
   fragments are label switched along that path.  As a consequence,
   alternate routes are not possible for individual fragments.  The
   datagram_tag is used to carry the label, that is swapped at each hop.

Thubert & Hui             Expires July 14, 2017                [Page 11]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

8.1.  Upon the first fragment

   In route over the L2 source changes at each hop.
TW> something's wrong with the previous sentence

[Pascal] is this better:  In route over mode, the MAC address changes at each hop.

   The label that is
   formed and placed in the datagram_tag is associated to the source MAC
   and only valid (and unique) for that source MAC.  Say the first
   fragment has:

      Source IPv6 address = IP_A (may be multiple hops away)

      Destination IPv6 address = IP_B (maybe be multiple hops away)

      Source MAC = MAC_prv (prv as previous)

      Datagram_tag= DT_prv

   The intermediate router that forwards individual fragments does the
   following:

      a route lookup to get Next hop IPv6 towards IP_B, which resolves
      as IP_nxt (nxt as next)

      a MAC address resolution to get the MAC address associated to
      IP_nxt, which resolves as MAC_nxt

   Since it is a first fragment of a packet from that source MAC address
   MAC_prv for that tag DT_prv, the router:

      cleans up any leftover resource associated to the tuple (MAC_prv,
      DT_prv)

      allocates a new label for that flow, DT_nxt, from a Least Recently
      Used pool or some similar procedure.

      allocates a Label swap structure indexed by (MAC_prv, DT_prv) that
      contains (MAC_nxt, DT_nxt)

      allocates a Label swap structure indexed by (MAC_nxt, DT_nxt) that
      contains (MAC_prv, DT_prv)

      swaps the MAC info to from self to MAC_nxt

      Swaps the datagram_tag to DT_nxt

   At this point the router is all set and can forward the packet to
   nxt.

TW> this assumes the first fragment is long enough to contain the source/destination IP addresses, right?

[Pascal] Right. This could be said in a rule to drop the packet if not.

Thubert & Hui             Expires July 14, 2017                [Page 12]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

8.2.  Upon the next fragments

   Upon next fragments (that are not first fragment), the router expects
   to have already Label swap structure indexed by (MAC_prv, DT_prv).
   The router:

      looks up the Label swap entry for (MAC_prv, DT_prv), which
      resolves as (MAC_nxt, DT_nxt)

      swaps the MAC info to from self to MAC_nxt;

      Swaps the datagram_tag to DT_nxt

   At this point the router is all set and can forward the packet to
   nxt.

   if the Label swap entry for (MAC_src, DT_src) is not found, the
   router builds an RFRAG-ACK to indicate the error.  The acknowledgment
   message has the following information:

      MAC info set to from self to MAC_prv as found in the fragment
TW> "to from"?

[Pascal] the Label swap entry for (MAC_src, DT_src) -> the Label swap entry for (MAC_prv, DT_prv)

      Swaps the datagram_tag set to DT_prv

      Bitmap of all zeroes to indicate the error

   At this point the router is all set and can send the RFRAG-ACK back
   to the previous router.

8.3.  Upon the fragment acknowledgments

   Upon fragment acknowledgments next fragments
TW> reads strange
   (that are not first
   fragment), the router expects to have already Label swap structure
   indexed by (MAC_nxt, DT_nxt).  The router:

      looks up the Label swap entry for (MAC_nxt, DT_nxt), which
      resolves as (MAC_prv, DT_prv)

      swaps the MAC info to from self to MAC_prv;

      swaps the datagram_tag to DT_prv

   At this point the router is all set and can forward the RFRAG-ACK to
   prv.

   If the Label swap entry for (MAC_nxt, DT_nxt) is not found, it simply
   drops the packet.

Thubert & Hui             Expires July 14, 2017                [Page 13]
Internet-Draft    LLN Fragment Forwarding and Recovery      January 2017

   If the RFRAG-ACK indicates either an error or that the fragment was
   fully receive, the router schedules the Label swap entries for
   recycling.  If the RFRAG-ACK is lost on the way back, the source may
   retry the last fragments, which will result as an error RFRAG-ACK
   from the first router on the way that has already cleaned up.

9.  Security Considerations

   The process of recovering fragments does not appear to create any
   opening for new threat compared to "Transmission of IPv6 Packets over
   IEEE 802.15.4 Networks" [RFC4944].

10.  IANA Considerations

   Need extensions for formats defined in "Transmission of IPv6 Packets
   over IEEE 802.15.4 Networks" [RFC4944].

11.  Acknowledgments

   The author wishes to thank Jay Werb, Christos Polyzois, Soumitri
   Kolavennu, Pat Kinney, Margaret Wasserman, Richard Kelsey, Carsten
   Bormann and Harry Courtice for their contributions and review.

12.  References

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 5 Apr 2017 17:55:27 +0200

> The major rationale IMO for this draft is that it doesn't require intermediary nodes to reassemble!

As I said in the WG meeting (and pointed out in the 6LoWPAN book), this has not really been necessary even in the original 6LoWPAN.  However, to make “virtual reassembly buffers” work in multi-track environments, the first fragment (which provides the routing info) has to be sent on all tracks.

> The label switching mechanism is elegant as the labels are locally significant only, with no need to maintain a network-wide label registry. The document should state so.

Yes, that is the idea behind datagram tags in RFC 4944 — they are local to the hop.  Viewing the set of “virtual reassembly buffers” as a label switching table is certainly one way to describe it.  Still, this is an implementation technique for RFC 4944 6LoWPAN fragmentation, not a new protocol.

Grüße, Carsten

---

From: Ralph Droms <[email-masked]>
Date: Wed, 5 Apr 2017 13:20:50 -0400

Carsten...

> On Apr 5, 2017, at 11:55 AM, Carsten Bormann <[email-masked]> wrote:
> 
> 

I think I understand the technique you are describing.  Is there a written guideline or specification for the technique somewhere?

I don't recognize the term "multi-track environment".  Can you define it for me?

> Yes, that is the idea behind datagram tags in RFC 4944 — they are local to the hop.  Viewing the set of “virtual reassembly buffers” as a label switching table is certainly one way to describe it.…

In particular, I think I can infer how datagram tags would be used in a hop-scoped way with virtual reassembly buffers.  It would be good to have text to work through the details.  

- Ralph
 
> 
> Grüße, Carsten
> 
> _______________________________________________
> 6lo mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/6lo

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 6 Apr 2017 07:26:43 +0200

> I think I understand the technique you are describing.  Is there a written guideline or specification for the technique somewhere?

Apart from the brief description in section 2.5.2 of the 6LoWPAN book, I’m not aware of a writeup.
As I said at the meeting, maybe we should do that (food for lwig?).

> I don't recognize the term "multi-track environment".  Can you define it for me?

I was thinking about an environment where your routing protocol might cause different fragments to go different paths, or even to send copies of some fragments along multiple paths.

> In particular, I think I can infer how datagram tags would be used in a hop-scoped way with virtual reassembly buffers.  It would be good to have text to work through the details.  

Basically, a virtual reassembly buffer maps a incoming neighbor MAC address and an incoming datagram tag to an outgoing neighbor MAC address and an outgoing datagram tag.
It is created when an initial fragment is being processed and forwarded.  It could be deleted on forwarding the final fragment and/or on a timeout and/or on displacement.
(It could record more information, such as the target IP address that went into selecting the outgoing MAC address.)
It may also be useful to keep some management information (as in RFC 7388).
Finally, in certain cases it may keep fragments of a fragment in order to optimize re-fragmenting (but it generally is better to have the original fragmenter minimize the size of the *initial* fragment so no re-fragmenting is necessary even if header compression rates change).

Grüße, Carsten

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 6 Apr 2017 07:25:49 +0000

Hello Ralph:

A label switching technique to properly forward fragments to destination at L3 without recomposing the full packet at each hop is discussed in https://tools.ietf.org/html/draft-thubert-6lo-forwarding-fragments . 
Arguably, the technique that is proposed there could work with RFC 4944 datagram tags as well, and as Carsten says, that can be done by any 6LoPWAN node in the path independently of what other nodes do.

And IMHO keeping it at that and washing our hands would be incredibly short-sighted in the light of the other problems that we face with 6LoWPAN fragments as they exist today. Examples of such problems are handling of fragment loss and lack of capability to abort a transmission and clean up state in the network, but there's more, like handling of congestion in the mesh network and so on.

I presented the issues I'm aware of at 6lo at IETF 98, and hopefully the slides are self-explanatory enough so that people who did not follow the presentation can still derive enough information to make their own mind:  https://www.ietf.org/proceedings/98/slides/slides-98-6lo-fragmentation-forwarding-issues-in-6lowpan-00.pdf 

IEEE 802.5.4 understood and started addressing those issues in their protocols years ago, consider for instance how this is done at PHY layer in 802.15.4K LECIM - LP-WAN (cc'ing Pat). In the light of that art, the legacy 6LoWPAN technique is shamefully inadequate.

My suggestion to the group is to take a holistic view of these issues, and redesign or fragment support. This is what my draft does. 

Else we'd better be very clear to the world that our fragment support is only for the show, and that any link layer that expects a working IPv6 (leveraging 6LoWPAN) over it should have its own well-designed support for large frames and present MTU of 1280 or above to the upper layer. The IEEE appears to be taking that path anyway, generalizing MTUs of 2048.

Take care,

Pascal

-----Original Message-----
From: 6lo [mailto:[email-masked]] On Behalf Of Ralph Droms
Sent: mercredi 5 avril 2017 19:21
To: Carsten Bormann <[email-masked]>
Cc: [email-masked]; Thomas Watteyne <[email-masked]>; [email-masked]
Subject: Re: [6lo] [6tisch] Thomas' review of draft-thubert-6lo-forwarding-fragments-04

Carsten...

- Ralph

---

From: Rahul Jadhav <[email-masked]>
Date: Thu, 6 Apr 2017 03:57:53 -0500

Hi Pascal,

Since you checked regarding WG interest ... I for one, would definitely
like to see the fragmentation problem getting worked upon. You have already
presented the problem statement and few design considerations during IET98
and I fully agree with those slides. If there is some way to avoid buffer
bloat on intermediate and end nodes during frag & reassembly, then it will
be really great!

Fragmentation at 6lo layer as it is today is almost unusable for a
moderately sized network with lower MTUs such as 127B.
If required, I can help to work on these issues from design as well as
implementation point of view.
Secondly, just want to add that during the relevant poll in 6lo meeting it
was very evident that WG considers this as an important problem.

Regards,
Rahul

On 5 Apr 2017 8:49 p.m., "Pascal Thubert (pthubert)" <[email-masked]>
wrote:

> Hello Thomas :
>
>
>
> All in all, your points are well understood, and I will restructure a bit
> and add words. But, before I spend too much of my time on this draft, I’d
> like to see the WG interest clearly expressed. So far I saw you and Michel
> expressing clear support. Is the group behind us?
>
>
>
>
>
> Please see below ;
>
>
>
> from my understanding, this draft introduces two things: (1) a way of
> forwarding fragments so as to avoid to have to reassemble at every hop and
> (2) an end-to-end protocol to request lost fragments to be retransmitted. I
> think item (1) is really the main contribution of the draft, and (2) is a
> necessary corollary to making (1) work. The abstract and intro, however,
> seem to focus almost exclusively on (2), not clearly stating (1). A single
> sentence at the very beginning stressing the two contributions would go a
> long way IMO.
>
>
>
> [Pascal] OK, I’ll improve that on the next revision. Note that the initial
> title of the draft was draft-thubert-6lowpan-simple-fragment-recovery
> <https://tools.ietf.org/html/draft-thubert-6lowpan-simple-fragment-recovery-00>
> ; since then, I moved the focus on the capability to streamli,ne the
> fragments over a mesh. But both are really important. Forwarding
> fragments is now the title, and the draft improves the throughput by
> streamlining while avoiding buffer bloat inside the network.
>
> Yet, recovering the missing frags is important as well, avoids the bloat
> in the end point. Looking at the big picture, it’s very hard to do one
> without the other, the techniques benefit from one another and are not
> separable.
>
>
>
> ---
>
>
>
> point 2: justification
>
>
>
> Section 3 provides a rationale as to why this draft is needed. It focuses
> almost exclusively on reliability: if the link is 99% reliable, then the
> probability that 5 fragments, etc. IMO that doesn't make much sense as each
> fragment will be link-layer ACK'ed and retransmitted (at the link layer,
> i.e."transparently" to 6LoWPAN). There is of course a probability that all
> retries fail, but justifying the draft by assuming there are no link-layer
> ACKs make little sense. I may completely misunderstand, in which case,
> please point that out.
>
>
>
> [Pascal] We want to make the transmission reliable over multihop. I agree
> with what you are saying over one hop. But in the multihop scenario, if a
> hop down the path temporarily fails, the retries there will exhaust, but
> the node that did the fragmentation will not be aware (no nack all the way
> back), as opposed to the single hop case. So it will keep sending the next
> fragments, contributing to the problem, wasting time and network resources,
> etc...
>
>
>
> The major rationale IMO for this draft is that it doesn't require
> intermediary nodes to reassemble! Why not just say that? Reassembly at each
> hop yields several problems, the biggest one in my experience is that of
> memory management (if I'm missing one fragment from a packet, and a
> fragment from a second packet arrive, and I have only one assembly buffer,
> what should I do?).
>
>
>
> [Pascal] Yes, this is one of the problems I illustrated the most at the
> 6lo meeting, as you saw. Now, the bloat may happen in the receive end node,
> even when fragments are streamlined. It happens as soon as one fragment is
> lost. So forwarding fragments protects the network, making the fragments
> reliable protects the packet and the end point.
>
>
>
> [Pascal] Can’t say. But both are needed and help one another to the point
> that they cannot be separated. The draft addresses all the points in my
> presentation at 6lo (https://www.ietf.org/proceedings/98/slides/slides-
> 98-6lo-fragmentation-forwarding-issues-in-6lowpan-00.pdf), but the hidden
> terminal which is not an issue for 6TiSCH and is only alleviated by the
> proposed flow control and the window of one fragment. The value that you
> focus on is discussed slides 9-12, and was a focus of the discussion.
>
> ---
>
>
>
> Section 5 is called "Overview" and appears before the actual proposal is
> described. From the title, I was expecting a couple of paragraphs with the
> main idea, maybe a diagram or two to make the big picture clear. Instead,
> the section contains a discussion about very subtle points, including
> congestion control, interaction with link-layer protocols, timeout
> calculation and security.
>
>
>
> To summarize, the text in this section is interesting and should certainly
> appear in the specification, but as discussion points AFTER the main idea
> is presented.
>
>
>
> [Pascal] Ack
>
> ---
>
>
>
> While understandable, the use of acknowledgment which can refer to L2 ACKs
> or RFRAG-ACK messages is confused. I suggest to use the term "RFRAG-ACK"
> rather than the generic "Acknowledgement".
>
>
>
> [Pascal] Ack
>
>
>
>
>
> ---
>
>
>
> point 5: label switching
>
>
>
> The label switching mechanism is elegant as the labels are locally
> significant only, with no need to maintain a network-wide label registry.
> The document should state so.
>
>
>
> [Pascal] Ack
>
>
>
> [Pascal] In order to fit within transport time outs, you want to limit the
> number of retries to an acceptable time. In the 99.9% I assume 90%
> delivery, 3 tries, and complete independence of the 3 tries. All of these
> assumptions are completely wrong, but that gives an idea of how the
> problems adds up in a mesh. Still I can remove the paragraph, since, after
> all , it is not needed in a solution draft.
>
>
>
>
>
> [Pascal] Yes, depends on the security overhead which is variable. But I’d
> say that the reasonable goal of MTU=2048, which is the usual IEEE bar, is
> easily achieved.
>
>
>
>
>
> [Pascal] I’ll move the whole section to appendix; the draft is more needed
> in route over but still beneficial to mesh under, for the benefits that are
> not the buffer bloat. These include flow control, loss detection, and
> recovery.
>
>
>
>
>
> [Pascal] I can remove the text, but the problem remains, and your point,
> while largely correct, is not fully so. The transport layer may do some
> flow control based on command response, windowing and ECN, and thus protect
> against bursts of packets, but it cannot do a thing against bursts of
> fragments. So essentially, we are echoing some of the transport
> functionality at L2+ to provide those service to the fragments.
>
>
>
>
>
>    The specification also defines a 4-octet ackn
>
> 6tisch mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/6tisch
>
> ...

---

From: "Lijo Thomas" <[email-masked]>
Date: Thu, 6 Apr 2017 16:15:17 +0530

Hi Pascal,

I do agree the fragmentation problem is a real issue in 6LoWPAN, particularly when we play with bigger 802.15.4 networks . 

I extend my support to work on this issue and bring a good solution 

Thanks & Regards,

Lijo Thomas 

From: 6lo [mailto:[email-masked]] On Behalf Of Pascal Thubert (pthubert)
Sent: 05 April 2017 20:49
To: Thomas Watteyne; [email-masked]; [email-masked]
Subject: Re: [6lo] Thomas' review of draft-thubert-6lo-forwarding-fragments-04

Hello Thomas :

 

Please see below ;

[Pascal] OK, I’ll improve that on the next revision. Note that the initial title of the draft was  <https://tools.ietf.org/html/draft-thubert-6lowpan-simple-fragment-recovery-00> draft-thubert-6lowpan-simple-fragment-recovery ; since then, I moved the focus on the capability to streamli,ne the fragments over a mesh. But both are really important. Forwarding fragments is now the title, and the draft improves the throughput by streamlining while avoiding buffer bloat inside the network. 

---

point 2: justification

---

[Pascal] Ack

---

[Pascal] Ack

---

point 5: label switching

[Pascal] Ack

Take care,

Pascal

===

6lo                                                      P. Thubert, Ed.

                                                        January 10, 2017

Abstract

   If a single one of

Status of This Memo

   Drafts is at  <http://datatracker.ietf.org/drafts/current/> http://datatracker.ietf.org/drafts/current/.

Copyright Notice

 

 

   ( <http://trustee.ietf.org/license-info> http://trustee.ietf.org/license-info) in effect on the date of

Table of Contents

1.  Introduction

 

 

   violation.

   common to LLNs.

2.  Terminology

 

 

   Networks" [RFC4944].

   ERP

   6LoWPAN endpoints

 

 

3.  Rationale

   Towards the LLN node:

   From the LLN node:

 

 

         fragment.

 

 

   multiple hops.

4.  Requirements

   LLN endpoints.

 

   protocol):

   Number of fragments

 

 

 

 

[Pascal] Will do

   Controlled latency

      Protocols.

 

 

5.  Overview

 

 

 

 

   

 

 

            Pattern    Header Type

                            1                   2                   3

                                                  X set == Ack Requested

      receiver

   sequence number.

 

   acknowledged as follows:

                            1                   2                   3

        ^                   ^

 

                            1                   2                   3

   follows:

                            1                   2                   3

   Acknowledgment Bitmap

 

      fragment x was received.

7.  Fragments Recovery

   fragmented packets.

 

 

 

   When the sender of the

[Pascal] OK.

   the Fragments

   bit set.

[Pascal] OK.

TW> why SWORD?

 

 

   not enabled by default.

   recovery timers.

   bitmap.

8.  Forwarding Fragments

 

 

 

 

   The label that is

   fragment has:

      Datagram_tag= DT_prv

   following:

      as IP_nxt (nxt as next)

      DT_prv)

   nxt.

 

 

 

   The router:

   nxt.

TW> "to from"?

 

   to the previous router.

TW> reads strange

   (that are not first

   prv.

   drops the packet.

 

 

10.  IANA Considerations

11.  Acknowledgments

12.  References

-------------------------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at:
Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]

This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
-------------------------------------------------------------------------------------------------------------------------------

---

From: Tero Kivinen <[email-masked]>
Date: Thu, 6 Apr 2017 15:34:38 +0300

Pascal Thubert (pthubert) writes:
> IEEE 802.5.4 understood and started addressing those issues in their
> protocols years ago, consider for instance how this is done at PHY
> layer in 802.15.4K LECIM - LP-WAN (cc'ing Pat). In the light of that
> art, the legacy 6LoWPAN technique is shamefully inadequate.

802.15.4k is really only a phy level fragmentation, and the main issue
there is that the max packet size in that PHY is extremely small
(less than 30 octets).

When designing the IEEE std 802.15.9 we checked 4k that and realized
that it is not something that can be used in general case, and thats
why the MAC level fragmentation in the 802.15.9 is using different
method. Fragmentation was needed in the 802.15.9 as KMP payloads do
not fit to the small frames of 802.15.4. This fragmentation method
(and the multiplexing) from the 802.15.9 will most likely be adopted
in the 802.15.12 later...

Btw, the 802.15.9 is now available using the get IEEE 802 program, and
you can download it from

http://standards.ieee.org/getieee802/download/802.15.9-2016.pdf

The fragmentation wire formats are described in the section 7, and
state matchines are described in section 9.

> My suggestion to the group is to take a holistic view of these
> issues, and redesign or fragment support. This is what my draft
> does.

The 802.15.9 is hop by hop fragmentation, so if you want to be able to
forward fragments, you need to specify your own way of doing that... 
-- 
[email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 6 Apr 2017 14:54:10 +0200

On Apr 6, 2017, at 10:57, Rahul Jadhav <[email-masked]> wrote:
> 
> Fragmentation at 6lo layer as it is today is almost unusable for a moderately sized network with lower MTUs such as 127B.

Hi Rahul,

please explain some more.

What kind of problem do you experience?

Is the load on your network predominantly composed of packets > 127 B?

Are you using MAC-layer ACK?

Do the problems you are seeing occur with Unicast or with Multicast?

Are you already using fragment forwarding or are you reassembling hop by hop?

Grüße, Carsten

---

From: Muhammad Akbar <[email-masked]>
Date: Thu, 6 Apr 2017 13:36:22 +0000

Hi

What about IEEE 802.15.12?

 I attended a presentation in IETF98, where a person from IEEE mentioned about IEEE 802.15.12 with the packet size of 2048 (not sure the exact but its more than 1280 MTU), do we need fragmentation then?

Muhammad
________________________________________
From: 6tisch <[email-masked]> on behalf of Tero Kivinen <[email-masked]>
Sent: 06 April 2017 13:34
To: Pascal Thubert (pthubert)
Cc: PWK; Ralph Droms; Thomas Watteyne; [email-masked]; [email-masked]; Carsten Bormann
Subject: Re: [6tisch] [6lo] Thomas' review of draft-thubert-6lo-forwarding-fragments-04

_______________________________________________
6tisch mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/6tisch
BU is a Disability Two Ticks Employer and has signed up to the Mindful Employer charter. Information about the accessibility of University buildings can be found on the BU DisabledGo webpages. This email is intended only for the person to whom it is addressed and may contain confidential information. If you have received this email in error, please notify the sender and delete this email, which must not be copied, distributed or disclosed to any other person. Any views or opinions presented are solely those of the author and do not necessarily represent those of Bournemouth University or its subsidiary companies. Nor can any contract be formed on behalf of the University or its subsidiary companies via email.

---

From: Rahul Jadhav <[email-masked]>
Date: Thu, 6 Apr 2017 19:09:39 +0530

Hi Carsten,

Currently I m reassembling hop by hop.

Problem begins with initial auth session itself!! I use PANA for network
access in single channel (with no RDC) mesh and the payload in most PANA
messages crosses 127B (especially considering the relay header overhead),,,
its all unicast with MAC layer ACK. None of my mcast packets cross MTU
boundary. Also an imp point is that the buffer space is only for one IPv6
datagram (i.e. 1280B).
Some of the problems highlighted by Pascal such as, fragments getting
discarded prematurely for e.g. because of high node density causing
reception from different peers is hurting the end to end msg transmission
badly. The failure rate and thus the retransmission rate at app is very
high,, so much that we had to adopt a proprietary compressed PANA
signalling mechanism  to improve convergence time.

Regards,
Rahul

---

From: PWK <[email-masked]>
Date: Thu, 6 Apr 2017 08:44:39 -0500

Tero makes some good points but perhaps I have a different view on the Layer 1 fragmentation introduced with 802.15.4k.

Firstly, the Layer 1 fragmentation was early on intended to be a general function that could be used with any PHY, that was why it was in the MAC section for 802.15.4k.  During the 802.15.4-2015 revision effort we saw that there were some PHYs that would require additional effort on the mechanism (I cannot remember the specifics).  Rather than delay the revision further we chose the expedient way to move on which was to put it into the LECIM section, since they were the PHYs (DSSS in particular)  that absolutely needed Layer 1 fragmentation.  In the next revision I will propose that, once again, we revisit the concept of extending the Layer 1 fragmentation to more PHYs as well as change the moniker LECIM to LPWAN.  

One of the requirements that we placed on Layer 1 fragmentation was the ability to abort the fragmentation/reassembly process at any point in the process.  This was achieved with minimal impact to the overhead by setting the acknowledgment’s fragmentation number field and the acknowledgment content set to zero (ie. no packets were received).

While Layer 1 fragmentation has benefits over Layer 2 fragmentation, it does have some disadvantages.  Firstly, since the MAC will not accept datagrams that are larger than the size dictated by “aMaxPhyPacketSize” (a PHY constant that dictates the largest frame that the PHY can transmit, currently either 127 or 2047 octets), if this value is set to 127, a Layer 2 fragmentation method is required for datagrams larger than 127 octets.  The second that seems important for mesh networks is that a Layer 1 fragmentation is solely link-by-link/hop-by-hop, ie. the reassembly must occur at each receiver.  While this is no problem for star topologies, it is a concern for mesh or multi-hop networks.

Pat

Pat Kinney
Kinney Consulting LLC
IEEE 802.15 WG vice chair, SC chair
ISA100 co-chair, ISA100.20 chair
O: +1.847.960.3715
[email-masked] <mailto:[email-masked]>
On 6, Apr2017, at 7:34, Tero Kivinen <[email-masked]> wrote:

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 6 Apr 2017 15:57:00 +0200

Hi Rahul,

this sounds as if fragment forwarding would help already?
(Am I reading this right that the discards are buffer space discards, not retransmission count packet losses?)

Grüße, Carsten

> On Apr 6, 2017, at 15:39, Rahul Jadhav <[email-masked]> wrote:
> 
> Hi Carsten,
> 
> Problem begins with initial auth session itself!! I use PANA for network access in single channel (with no RDC) mesh and the payload in most PANA messages crosses 127B (especially considering the relay header overhead),,, its all unicast with MAC layer ACK. None of my mcast packets cross MTU boundary. Also an imp point is that the buffer space is only for one IPv6 datagram (i.e. 1280B).
> Some of the problems highlighted by Pascal such as, fragments getting discarded prematurely for e.g. because of high node density causing reception from different peers is hurting the end to end msg transmission badly. The failure rate and thus the retransmission rate at app is very high,, so much that we had to adopt a proprietary compressed PANA signalling mechanism  to improve convergence time.
> 
> Regards,
> Rahul
>

---

From: PWK <[email-masked]>
Date: Thu, 6 Apr 2017 09:22:40 -0500

Hi;

I made that IETF98 presentation.  Currently, the maximum frame size that IEEE 802.15.4 can handle is either 127 octets or 2047 octets.  The 2047 octet limit is for Higher Rate 2 Mb/s PHY (802.15.4t), Smart Utility Networks (SUN), Television white space (TVWS), railroad communications and control (RCC), and Low energy critical infrastructure monitors (LECIM) FSK PHYs and is 127 octets for all other PHYs.

Pat

Pat Kinney
Kinney Consulting LLC

On 6, Apr2017, at 8:36, Muhammad Akbar <[email-masked]> wrote:

Hi

Muhammad

---

From: Rahul Jadhav <[email-masked]>
Date: Thu, 6 Apr 2017 20:04:20 +0530

Hi Carsten,

Prima facie, I think fragment forwarding might help, having said i need to
still understand the implications of the revised design(and there are
reservations i have). Implementation-wise, it may not be difficult to try
out and would consider it ...
Regardless, my point was, working on fragmentation would be well worth the
time spent, be it fragment-forwarding or if there is a possibility of any
thing more improved!

Regards,
Rahul
p.s. pls chk inline as well.

> retransmission count packet losses?)
[RJ]: Yes buffer space discards, but it results in more retransmissions
causing more mac send failures.

---

From: Tero Kivinen <[email-masked]>
Date: Thu, 6 Apr 2017 17:53:50 +0300

> What about IEEE 802.15.12?

802.15.12 will bring fragmentation but I think it will be hop-by-hop
fragmentation based on the 802.15.9.

>  I attended a presentation in IETF98, where a person from IEEE
>  mentioned about IEEE 802.15.12 with the packet size of 2048 (not
>  sure the exact but its more than 1280 MTU), do we need
>  fragmentation then?

New PHY layers will usually support 2047 octet frames, and for example
the new 2Mbit/s phy for the 2.4GHz band which is backward compatible
with normal 2.4GHz phy will support 2047 octet frames. The old 2.4GHz
PHY will still be limited for 127 octets.
-- 
[email-masked]

---

From: Tero Kivinen <[email-masked]>
Date: Thu, 6 Apr 2017 18:05:13 +0300

PWK writes:
> Firstly, the Layer 1 fragmentation was early on intended to be a general
> function that could be used with any PHY, that was why it was in the MAC
> section for 802.15.4k.  During the 802.15.4-2015 revision effort we saw that
> there were some PHYs that would require additional effort on the mechanism (I
> cannot remember the specifics).

Like all other PHYs, as the LECIM PHY assumes that you know who is
talking based on the frequency they are using, thus the fragments do
not have addressing information in them. There is only two octet
fragment header having 7-bit transaction ID and fragment number. If
there are multiple senders sending fragments at the same time, they
will get mixed up unless you can know it from somewhere else who is
sending that fragment.

With TSCH it could be possible to use it as if you use dedicated slot,
then you know who is sending the frame. With normal 2.4GHz non-TSCH
frame this is not really going to happen, as if there is multiple
senders sending fragments at the same time, they will get mixed up.

If you are using security then you can verify the MIC-32 in the each
fragment, and if it does not match, then the fragment was corrupted,
or was part of some other fragment transaction...

> Rather than delay the revision further we chose the expedient way to
> move on which was to put it into the LECIM section, since they were
> the PHYs (DSSS in particular) that absolutely needed Layer 1
> fragmentation. In the next revision I will propose that, once again,
> we revisit the concept of extending the Layer 1 fragmentation to
> more PHYs as well as change the moniker LECIM to LPWAN.

The problem there is that max frame size in LECIM is so small that you
cannot include the things like source and destination addresses in
each fragment, and leaving them out will cause issues on other PHYs,
like normal 2.4GHz PHY, especially if there is lots of fragmented
traffic. 
-- 
[email-masked]

---

From: PWK <[email-masked]>
Date: Thu, 6 Apr 2017 10:27:28 -0500

Actually, with 802.15.4 Layer 1 fragmenting, the source device sends the destination device a message containing the Fragment Sequence Context Description which includes (among other information) the source and destination addresses and a 6-bit transaction identifier which uniquely defines this fragment sequence.  The destination device must ACK this message before the fragment sequence starts.  After that ACK, the source sends the fragments that include the transaction identifier but not the source and destination addresses.  Any device receiving a fragment checks to see if the transaction identifier belongs to that device, if it doesn’t, the fragment is discarded.  In summary, multiple fragment sequences will not get mixed up since the Fragment Sequence Context Description maps the transaction identifier to the source and destination addresses.

Pat

On 6, Apr2017, at 10:05, Tero Kivinen <[email-masked]> wrote:

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 6 Apr 2017 17:17:17 +0000

That ultra cool Rahul.

I published 05 that addresses Thomas’ comments ☺

And yes, there’s a lot more than forwarding fragments blindly.

Take care,

Pascal

From: 6tisch [mailto:[email-masked]] On Behalf Of Rahul Jadhav
Sent: jeudi 6 avril 2017 15:40
To: Carsten Bormann <[email-masked]>
Cc: [email-masked]; lo <[email-masked]>
Subject: Re: [6tisch] [6lo] Thomas' review of draft-thubert-6lo-forwarding-fragments-04

Hi Carsten,

Regards,
Rahul

On 6 April 2017 at 18:24, Carsten Bormann <[email-masked]<mailto:[email-masked]>> wrote:

Hi Rahul,

Grüße, Carsten

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 7 Apr 2017 12:48:55 +0300

PWK writes:
> Actually, with 802.15.4 Layer 1 fragmenting, the source device sends
> the destination device a message containing the Fragment Sequence
> Context Description which includes (among other information) the
> source and destination addresses and a 6-bit transaction identifier
> which uniquely defines this fragment sequence.  The destination
> device must ACK this message before the fragment sequence starts.
> After that ACK, the source sends the fragments that include the
> transaction identifier but not the source and destination addresses.
> Any device receiving a fragment checks to see if the transaction
> identifier belongs to that device, if it doesn’t, the fragment is
> discarded.  In summary, multiple fragment sequences will not get
> mixed up since the Fragment Sequence Context Description maps the
> transaction identifier to the source and destination addresses. 

That is only true if you have two nodes on the channel.

If node A sends fragment sequence context description IE and sets up
fragmentation context with node B with TID of 0x00, and at the same
time node C decides to send fragmented packet to node D, the node C
will send similar frame to setup the fragmentation context with D, but
as those fragment sequence context description IE frames do have
addressing fields, only the intended targets will see them, thus only
B receives the FSCD from A and only D receives it from C. Thus C does
not have any idea which TIDs are in use, and it might also pick TID of
0x00.

Then when A sends a fragment with TID of 0x00, and without addressing
fields, both B and D will receive it and they both think it belongs to
them.

Similar when C sends fragments with TID 0x00 both B and D receive it
and both will add it to their reassembly queue.

If security is enabled then the problem is not as bad, as the MIC-32
of the fragment is calculated using nonce, which includes the sender
extended address, thus MIC-32 will fail if the sender is not the one
device assumes it is, thus node B will only accept fragments from A,
and all fragments sent by the C will have invalid MIC-32, thus they
will fail the authentication and will be dropped. Similarly with node
D.

If no security is used then there is nothing protecting the fragments,
thus they will get mixed up. In LECIM this is not problem (at least
that was told to me), because there is only one sender in one channel,
thus the recipient can know that frame is from that sender by just
receiving it on that channel.

Of course the security of the fragments was broken in the 4k, and it
was fixed in the 802.15.4-2015 version. In 4k the fragment counters
and the frame counters could overlap, thus it could generate same
nonce twice in some cases (i.e. frame number 0x00000001, and fragment
0x01 of frame number 0x000000 had same nonce or so).
-- 
[email-masked]

---

From: PWK <[email-masked]>
Date: Fri, 7 Apr 2017 07:41:55 -0500

Tero; I understand your scenario, and I also believe that all WAN/LAN/PAN wireless networks should always use security.  

But to further analyze your scenario, since 802.15.4 LP-WAN is star topology based, in your example A<=>B would be one network and C<=>D would be another network.  Prior to staring they would have looked to see if there were other networks set up on the channel, therefore one would have used another channel (as you noted).  By the way, the scenario you choose is similar to two networks choosing the same PAN ID and allocating the same short addresses to nodes that are in range of both networks. 

Secondly, your scenario assumes that they both send the Fragment Sequence Context Description at almost the same time, which is very unlikely for two networks so close together.  If they were “out-of-sync” then B or D would have received a fragment out of sync, so if Frak policy zero was in use, reception of an out-of-order fragment would have resulted in termination of the transaction. The other two policies would have resulted in a condition where B or D would have received a fragment out of sequence that was already correctly received, so it would have discarded that fragment, while the other would have received a fragment number ahead of its time and would have accepted it.  

Finally, for the last case, I will also propose a text change in the next revision that when the frame is reassembled, the standard policy of checking the FCS and sending ACKs is used.

Pat

Pat Kinney
Kinney Consulting LLC

On 7, Apr2017, at 4:48, Tero Kivinen <[email-masked]> wrote:

---

From: Tero Kivinen <[email-masked]>
Date: Tue, 11 Apr 2017 17:31:00 +0300

PWK writes:
> Tero; I understand your scenario, and I also believe that all WAN/LAN/PAN
> wireless networks should always use security.  

Agree, but looking at the current usage that does not seem to be true,
and looking at things like issues found while making revision there
are not that many implementor who have implemented security, as they
would have noticed those issues if they actually would have
implemented them. Or they just implemented something, and didn't
really care that the specification was not specifying things exatly... 

> But to further analyze your scenario, since 802.15.4 LP-WAN is star
> topology based, in your example A<=>B would be one network and C<=>D
> would be another network.

Section 5.5 Network topologies of 802.15.4-2015 do say that IEEE
802.15.4 LR-WPAN operates in either of two topologies: the star
topology or the peer-to-peer topology.

So 802.15.4 is not only star topology...

Also this is still issue with star topology, as two nodes might send
fragments to the coordinator at the same time, and as TID is allocated
by the sender only, and coordinator does not have ability to reject
incoming FSCD IE. When the coordinator parses that FSCD IE it has most
likely already sent an Ack back to the frame, and if the TID it got in
the FSCD IE is overlapping with some other fragment exchange it has
already going on, there is nothing it can do. Even if it parses the
FSCD IE before sending the ACK, it is supposed to send ACK back
because it did receive the frame properly. 

> Prior to staring they would have looked to see if there were other
> networks set up on the channel, therefore one would have used
> another channel (as you noted).

Not really. They would look for free enough channel. If there is not
too much traffic on the channel they try when forming the network,
they can pick that channel. And some other networks could also end up
using the same channel. And there might even be some TSCH or similar
channel hopping networks using the same channel. 

> By the way, the scenario you choose is similar to two networks
> choosing the same PAN ID and allocating the same short addresses to
> nodes that are in range of both networks.

PAN ID conflicts are handled by the 802.15.4 using the PAN ID Conflict
notification command, which will cause the one of PANs to renumber to
use some other PAN ID. The PAN does not necessarely move to the other
channel, it can still stay on the same channel. 

> Secondly, your scenario assumes that they both send the Fragment Sequence
> Context Description at almost the same time, which is very unlikely for two
> networks so close together.

They do not need start at the same time, it is enough if one of them
is faster, and passes the other. I.e., A starts ending frames, but is
very slow and can only send one frame every 500 ms, and it has 20
frames to send, so total time of sending the fragment will be 10
seconds. C is must faster and can send back to back frames, meaning
it can send frames every 10 ms or so, so it can finish its 20 frame
transaction in 200 ms. If the C starts its transaction while A is
doing its fragment transaction, then both B and D will receive those
frame, and B will reject the frames having the fragment number smaller
than what A has already sent, but after C passes A it will accept rest
of the fragments from the C.

> If they were “out-of-sync” then B or D would have
> received a fragment out of sync, so if Frak policy zero was in use, reception
> of an out-of-order fragment would have resulted in termination of the
> transaction. The other two policies would have resulted in a condition where B
> or D would have received a fragment out of sequence that was already correctly
> received, so it would have discarded that fragment, while the other would have
> received a fragment number ahead of its time and would have accepted it.  
> 
> Finally, for the last case, I will also propose a text change in the next
> revision that when the frame is reassembled, the standard policy of checking
> the FCS and sending ACKs is used.

That might help somewhat, but I think there is too much issues with
the current LECIM fragmentation with the fact that it does not have
addressing fields, that fixing it is not really worth of the effort.
It is better to pick the fragmentation from the 802.15.9 and use that
instead. 
-- 
[email-masked]